regexx/README.md

regexx

A single-file C regular expression interpreter that reproduces the observable behavior of Python's re module, including its exact identifier names (Pattern, Match, re_compile, re_sub, IGNORECASE, and so on; see Section 9 of concept.md), and that additionally supports binary data (arbitrary byte streams, including embedded NUL) and UTF-8 text alongside plain ASCII.

The design rationale, the algorithmic trade-offs, and a full accounting of what is and is not carried over from Python's re and from POSIX's native regex.h are recorded in concept.md. This file documents the implementation that exists today at a glance; docs/API.md is the exhaustive reference (every type, every flag, every function's exact return-value and memory-ownership convention, checked against a real CPython interpreter, not against memory).

Implementation status

This is a v1 implementation. It is a complete, tested engine for the pattern syntax and operations listed below, executed by a single recursive backtracking engine (concept.md Section 7.3) over a fully materialized copy of the input.

It does not yet implement the streaming, bounded-memory regular engine of concept.md Section 7.2. Input (the abstraction over "a source of chunks", concept.md 9.2) is implemented, and Input_from_file reads a whole file into memory before matching. Every public function signature is already exactly what the streaming design in concept.md specifies, so the non-streaming implementation underneath a given call can be replaced later without changing any caller. Concretely, today:

• Pattern_match/Pattern_fullmatch (a single anchored attempt at a fixed position) run in time proportional to the length of that attempt, and, for the common case of a single character, class, or . repeated by a quantifier, in O(1) recursion depth regardless of input size (the OP_REPEAT1 fast path). A repeated compound sub-pattern (for example (ab)*) still recurses once per repetition, bounded by a configurable depth limit (MAX_DEPTH in regexx.c, currently 60000, not exposed through the public API, so changing it means editing regexx.c and rebuilding); past that limit, matching fails with a reported error rather than a stack overflow or a wrong answer.
• Pattern_search/Pattern_finditer/Pattern_split/Pattern_sub are quadratic, not linear, in the worst case, even for a pattern with no backreference and no unbounded-width lookahead. v1's search loop tries every start position in turn as an independent match attempt (concept.md Section 7.2's design shares work across start positions in one linear pass; that engine is not implemented yet, see above), so a pattern that ultimately does not match, or matches only very late, redoes the bulk of its work at every position it tries. Measured directly: a*b searched over n bytes of a with no b anywhere took 0.32s at n = 10,000 and 19.2s at n = 80,000, an approximately 4x slowdown per doubling, the signature of O(n^2). This is unrelated to, and in addition to, the backreference/lookahead ReDoS caveat below; it applies even to the simplest possible non-matching pattern, and is the most significant open gap between this build and the "gigabyte scale" objective in concept.md Section 1 for the search-family operations specifically (Pattern_match/ Pattern_fullmatch, which never try more than one position, are not affected).
• A pattern with a backreference or an unbounded-width lookahead can, like CPython's own _sre, take worst-case exponential time on an adversarial input (concept.md Section 5, 13.2); this is the same catastrophic backtracking (ReDoS) behavior CPython itself exhibits on such patterns, not a regression specific to this engine.

Pattern syntax supported

Literals; . (with DOTALL); character classes with ranges, negation, and \d \D \w \W \s \S; \b \B; anchors ^ $ \A \Z (with MULTILINE); quantifiers * + ? {m,n} {m,} {,n} {m}, greedy and lazy; possessive quantifiers *+ ++ ?+ {m,n}+; groups (...) (?:...) (?P<name>...); alternation |; backreferences \1-\99, (?P=name), \g<name>, \g<N>; lookahead (?=...) (?!...); fixed-width lookbehind (?<=...) (?<!...); atomic groups (?>...); comments (?#...); global inline flags (?aiLmsux) at the start of a pattern; escapes \n \r \t \f \v \a, octal \0-prefixed escapes, \xhh, \uxxxx, \Uxxxxxxxx; flags IGNORECASE, MULTILINE, DOTALL, VERBOSE, ASCII, LOCALE (all with observable effect; see docs/API.md Section 2 for exactly what each one does), plus UNICODE and DEBUG (accepted for source compatibility with Python, currently no-ops in this build).

Rejected at compile time with a clear PatternError, rather than mis-parsed: conditional groups (?(id)yes|no), scoped inline flags (?flags:...), \N{NAME} named code points, and POSIX bracket classes [:alpha:] (which are not part of Python re at all, concept.md 14.3). Variable-width lookbehind is also rejected at compile time, matching CPython.

Operations supported

Pattern_match/fullmatch/search/finditer/findall/split/sub/subn/free, Pattern_groupindex_lookup, Match_group/start/end/span/start_byte/end_byte/span_byte/free, re_compile/match/fullmatch/search/finditer/findall/split/sub/subn/escape/purge, PatternError_free. See regexx.h for exact signatures, docs/API.md for the full reference (return values, memory ownership, exact Python correspondence for each one), and concept.md Section 9 for the naming convention they follow.

Known deviations from concept.md and from CPython, beyond the items above

• \w, \s, IGNORECASE case folding, and \d in UTF8 mode are backed by glibc's wctype.h functions under the C.utf8 locale, not by a hand-generated Unicode table (concept.md 13.3 anticipated a reduced static table; using the C library's own tables turned out to be simpler and more complete, at the cost of depending on the platform's Unicode version rather than a pinned one).
• lastindex/lastgroup report the highest-numbered capturing group that participated in the match, which coincides with CPython's "most recently closed group" rule for straightforward patterns but can differ from it in pathological cases (nested alternation re-executing a lower-numbered group after a higher one). Not exercised by the test suite; documented here rather than silently accepted.
• Match_free, PatternError_free, Input_from_buffer, Input_from_file, and Input_free have no Python counterpart and are not mentioned in concept.md's API surface; they exist because C has no garbage collector. Pattern_sub/Pattern_subn take the replacement template and the callback as two separate parameters rather than one polymorphic argument, for the same reason (concept.md 9.4 already anticipates and justifies this one).
• Python's Match.start(group)/.end(group) raise IndexError for an invalid group number and return -1 only for a valid group that did not participate; Match_start/Match_end return -1 for both cases, since C has no exception to raise. Match_group does distinguish them (-1 for no such group, 0 for an unparticipated one), see docs/API.md Section 3.23.
• Pattern.groupindex has no enumeration function in this build, only Pattern_groupindex_lookup(pattern, name); there is no way to list every name a compiled pattern defines without already knowing what to look for.

Building

Requires a C11 compiler and, for the test suite, Python 3 (used only to generate ground truth from CPython's own re module, concept.md Section 11; the library itself has no runtime dependency beyond the C standard library and libc's wctype.h/locale.h).

make            # builds libregexx.a and the rxgrep example
make test       # regenerates tests/generated_tests.c from Python `re`
                # ground truth and runs the full suite
make check      # same, under AddressSanitizer + UndefinedBehaviorSanitizer
make clean

make install installs libregexx.a and regexx.h under PREFIX (default /usr/local).

Using the library

#include "regexx.h"
#include <string.h>

const char *pattern = "(\\w+)@(\\w+)";
Pattern *pat = re_compile(pattern, strlen(pattern), UTF8, NULL);
Input *in = Input_from_buffer((const uint8_t *)"user@host", strlen("user@host"));

Match m;
if (Pattern_search(pat, in, 0, -1, &m) == 1) {
    const char *g; size_t glen;
    Match_group(&m, NULL, 1, &g, &glen);   /* g/glen -> "user" */
    Match_free(&m);
}

char *out; size_t outlen;
Pattern_sub(pat, in, "\\2@\\1", NULL, NULL, 0, &out, &outlen); /* "host@user" */
free(out);

Pattern_free(pat);
Input_free(in);

flags to re_compile combine a data mode, exactly one of BINARY, ASCII, or UTF8 (concept.md 9.3), with any of the Python-named flags (IGNORECASE, MULTILINE, DOTALL, VERBOSE, ASCII as a flag also forces ASCII-only \w/\s/\d inside UTF8 mode, LOCALE, DEBUG).

Example: rxgrep

examples/rxgrep.c is a small grep-like program built on the library, demonstrating all three data modes and both the matching and substitution API:

./rxgrep -in 'hello' file.txt        # case-insensitive, line numbers
./rxgrep -m utf8 -o '\w+' file.txt   # print every UTF-8 word, one per line
./rxgrep -c 'error' log.txt          # count matching lines
./rxgrep -m binary 'a.c' data.bin    # match raw bytes, embedded NUL included
./rxgrep -m utf8 --sub 'REDACTED' '\d{3}-\d{4}' file.txt

Run ./rxgrep --help for the full option list.

Testing

tests/cases.py lists pattern/subject/operation triples. tests/gen.py computes each one's expected result with CPython's own re module and writes tests/generated_tests.c, which is then compiled against regexx.c and checked. This is a direct implementation of the strategy concept.md Section 11 describes: conformance is measured against what CPython actually does, not against a re-derived reading of its documentation. make check additionally runs the suite under AddressSanitizer and UndefinedBehaviorSanitizer.

License

MIT. See LICENSE.