retoor/regexx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add a Pike VM (regular engine, concept.md 7.2), dispatched automatically

Browse Source 
Researched first (Cox's regexp2 article, the submatch/tagged-NFA
follow-up crediting Laurikari, and rust-lang/regex's PikeVM source for
the exact leftmost-first mid-search Match handling), then planned in
concept.md 7.7 before writing any code, per the explicit instruction to
plan before implementing.

The existing bytecode compiler turned out to already produce valid
Thompson-construction NFA bytecode (OP_SPLIT/OP_JMP/OP_SAVE match
Cox's instruction set almost exactly), so no new compiler was needed:
Prog gained one field (no_repeat1) and compile_repeat one condition,
letting the same compile_node produce a second, NFA-only Prog from the
same AST for any pattern containing none of OP_BACKREF/OP_LOOKAHEAD/
OP_LOOKBEHIND/OP_ATOMIC (a possessive quantifier already desugars to
the last of these at parse time). That second Prog is matched by a new
Pike VM (pike_addthread/pike_step/pike_find): a breadth-first thread
list simulation with per-thread capture arrays, epsilon closure
implemented with an explicit heap stack rather than C recursion so a
pattern with many alternations cannot recurse the C stack, every
allocation checked and failing through the existing -1 error
convention rather than a NULL dereference. do_one, Pattern_finditer,
and Pattern_split each gained an impl->has_nfa branch to this engine,
sharing one small helper (find_next) for the "unanchored scan from a
position" versus "single anchored attempt at a position" distinction
finditer/split's empty-match retry needs.

Two real bugs, both found and precisely localized by the existing
3,252-case CPython-derived suite without writing a single test
specifically for this engine: OP_MATCH not writing group 0's end
position (it has no OP_SAVE; run()'s own OP_MATCH handler sets it
directly, and this engine's first version missed replicating that),
and an unconditional "thread list empty -> stop" early exit that is
wrong for unanchored search specifically, since a freshly injected
start thread can die immediately in its own epsilon closure (a
leading \b failing outright, repeatedly, inside a longer word like
"catalog" for \bcat\b) without that meaning every later position
would too. Both fixed; full suite passes, three clean AddressSanitizer/
UndefinedBehaviorSanitizer runs, plus hand-written whitebox checks
(a 20,000-branch alternation, UTF8 named groups, BINARY matching
across an embedded NUL, greedy/lazy and alternation priority).

Measured result: (a+)+b, this project's own running example of the
backtracking engine's remaining weak spot, is Pike VM eligible and now
measures as genuinely linear (0.0018s to 0.0308s, n=10,000 to
160,000), not merely improved. Measured cost: re-running
bench_vs_posix.c's three ordinary scenarios (all now Pike VM eligible
too) found the gap to POSIX <regex.h> widened, from roughly 2x-25x
before this engine existed to roughly 8x-70x now, the direct,
expected cost of this engine's performance axis being explicitly
deferred (no literal prefilter, no lazy DFA state caching, no
allocation pooling) in favor of correctness first, per the instruction
this was built under. Both results, and the reasoning behind
deferring the second, are recorded in concept.md 7.7/7.8.

README.md, docs/API.md, USAGE.md, and examples/redos_atomic.c and
bench_vs_posix.c are updated throughout to describe the new two-engine
dispatch accurately, including this real trade-off, rather than
leaving the previous single-engine description in place; redos_atomic.c
specifically now shows both that (a+)+b no longer needs an atomic group
at all and a backreference-forced variant where one still does.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY
This commit is contained in:
retoor
2026-09-14 11:38:44 +00:00
co-authored by Claude Sonnet 5
parent 02743cd959
commit fa07622671
10 changed files with 687 additions and 162 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/README.md
+2 -2
View File
@@ -20,10 +20,10 @@ directly, no build system required.
| [`binary_scan.c`](binary_scan.c) | `BINARY` mode: a byte-range character class (`[\x00-\xff]`) matching raw bytes 0-255, including an embedded `NUL` and an embedded `0x0A`, neither of which is a terminator or a line break in this mode the way it would be to a C string function or a line-oriented reader. This is the one thing `ASCII`/`UTF8` mode cannot do, since both introduce either a restricted classification or a decoding step. |
| [`utf8_scripts.c`](utf8_scripts.c) | `UTF8` mode: `\w` recognizing letters across Latin, Greek, Cyrillic, and CJK text (not only ASCII), and the resulting difference between `Match_span` (code point units) and `Match_span_byte` (byte units) once a match spans characters that are more than one byte wide. |
| [`ascii_logparse.c`](ascii_logparse.c) | `ASCII` mode doing what it is ordinarily used for: parsing structured, line-oriented text with named groups (`(?P<name>...)`) and reading fields back out by name via `Match_group`/`Pattern_groupindex_lookup`, not by a numeric position the caller has to remember. |
| [`redos_atomic.c`](redos_atomic.c) | Atomic groups (`(?>...)`) and possessive quantifiers (`a++`) as a real, pattern-author-controlled defense against catastrophic backtracking, timed directly against the plain, unprotected form of the textbook `(a+)+b` shape on a non-matching adversarial input. |
| [`redos_atomic.c`](redos_atomic.c) | Two-part: first, that the textbook `(a+)+b` ReDoS shape is now fixed automatically, with no atomic group, because it has no backreference and so runs on the Pike VM (README.md "The Pike VM"); second, a variant with a backreference added specifically to force it back onto the backtracking engine, where the same shape is exponential again and an atomic group (`(?>...)`) is still the pattern author's own necessary fix. |
| [`empty_match_rule.c`](empty_match_rule.c) | `Pattern_finditer`/`Pattern_split` reproducing a real CPython interpreter's undocumented empty-match retry rule exactly (`\d*?` against `"123abc456"` yields 16 matches, not 9), found by probing a real interpreter directly rather than by reading its documentation. |
| [`large_file_search.c`](large_file_search.c) | `Input_from_file`'s `mmap`-backed reading on a generated 100MB file, with the elapsed time and peak resident memory printed directly so they can be checked against the measured figures in `../README.md` "Memory footprint" rather than taken on faith. |
| [`bench_vs_posix.c`](bench_vs_posix.c) | A direct, timed comparison against the C standard library's own `<regex.h>` (POSIX `regcomp`/`regexec`) on six scenarios at multi-megabyte/multi-hundred-thousand-line scale, using only pattern syntax valid in both engines. Reported honestly: glibc's DFA-backed engine wins most scenarios by a wide margin, and the one adversarial pattern shape (`(a+)+b`) where this library's backtracking cost is most visible is shown both in its plain form and fixed with an atomic group, the latter having no POSIX ERE equivalent to compare against at all. |
| [`bench_vs_posix.c`](bench_vs_posix.c) | A direct, timed comparison against the C standard library's own `<regex.h>` (POSIX `regcomp`/`regexec`) on six scenarios at multi-megabyte/multi-hundred-thousand-line scale, using only pattern syntax valid in both engines, every one of them Pike VM eligible. Reported honestly, including a real regression: glibc wins the three ordinary scenarios by a wider margin than this build measured before the Pike VM existed (it has none of the backtracking engine's constant-factor optimizations yet), while the adversarial `(a+)+b` scenario inverts entirely and now beats glibc outright, with no atomic group needed. |
Every program below was compiled and actually run while writing it; the
claims in its top-of-file comment (what a real CPython interpreter does,
examples/bench_vs_posix.c
+31 -15
View File
@@ -426,21 +426,37 @@ int main(void) {
}
}
printf("\nWhat this measures and does not measure: regexx is a backtracking\n"
"interpreter built for Python `re` semantics and the three data modes\n"
"(concept.md Section 7.3); glibc's <regex.h> is a mature, DFA-backed\n"
"engine with decades of optimization behind it and a much smaller\n"
"feature set (no named groups, no lazy quantifiers, no lookaround, no\n"
"atomic groups, and, structurally, no way to search past an embedded\n"
"NUL byte at all, docs/API.md Section 1.3 vs examples/binary_scan.c).\n"
"Scenario D is the sharpest illustration of the trade-off this library\n"
"makes: memoized backtracking (README.md \"Implementation status\")\n"
"keeps a Python-compatible engine off exponential time, not off\n"
"quadratic time, for a pattern shape a DFA-backed engine with no\n"
"backreference support handles without either problem; the correct\n"
"response to that specific gap is the atomic group in row D2, the same\n"
"fix a Python `re` user would reach for, not a claim that this engine\n"
"matches a DFA's asymptotic behavior in general, which it does not.\n");
printf("\nWhat this measures and does not measure: regexx is Python `re`\n"
"compatible and dispatches every pattern here (all backreference-,\n"
"lookaround-, and atomic-group-free) to its Pike VM (README.md \"The\n"
"Pike VM\", concept.md 7.2/7.7), a Thompson-NFA simulation with no\n"
"literal prefilter, lazy DFA state caching, or allocation pooling yet\n"
"(concept.md 7.7's explicitly deferred performance axis); glibc's\n"
"<regex.h> is a mature, DFA-backed engine with decades of that exact\n"
"kind of optimization behind it and a much smaller feature set (no\n"
"named groups, no lazy quantifiers, no lookaround, no atomic groups,\n"
"and, structurally, no way to search past an embedded NUL byte at all,\n"
"docs/API.md Section 1.3 vs examples/binary_scan.c). Scenarios A-C\n"
"measure that missing optimization layer directly: this build was\n"
"previously measured on the same three scenarios, on this engine's\n"
"backtracking predecessor plus its own precomputed skip-ahead tables\n"
"(README.md \"Implementation status\"), at roughly 2-3x less of a gap\n"
"to POSIX than shown here, a real, expected regression for ordinary\n"
"patterns from moving to a v1 engine still missing that layer, not\n"
"a defect introduced by the new engine's correctness.\n\n"
"Scenario D inverts entirely, and is the actual point of building this\n"
"engine at all: (a+)+b has no backreference, so it is Pike VM eligible,\n"
"and a Thompson-NFA simulation has no notion of \"try one split, then\n"
"backtrack and try another\" to begin with, so the classic nested-\n"
"quantifier ambiguity this pattern is famous for causing simply does\n"
"not exist for this engine; row D1 now runs this pattern faster than\n"
"POSIX <regex.h> itself, with no atomic group, not merely fast enough\n"
"to stop being the slowest thing on this page. Row D2 (still with no\n"
"POSIX equivalent) is retained for comparison, not because D1 needs\n"
"fixing anymore: examples/redos_atomic.c's Part 2 shows a pattern this\n"
"specific improvement cannot reach (a backreference forces the older\n"
"engine regardless of shape), which is where an atomic group remains\n"
"the pattern author's own necessary tool, not something automatic.\n");
return 0;
}
examples/large_file_search.c
+4 -2
View File
@@ -82,8 +82,10 @@ int main(void) {
printf("Searching (Input_from_file mmaps this file rather than copying it,\n"
"so this process's own heap never holds a private copy of the 100MB\n"
"file content itself; it still allocates the search tables README.md\n"
"\"Memory footprint\" measures, roughly 8.4x the input length)...\n");
"file content itself; this pattern has no backreference, lookaround,\n"
"or atomic group, so it also runs on the Pike VM, README.md \"The\n"
"Pike VM\", which needs none of the backtracking engine's own search\n"
"tables, README.md \"Memory footprint\", at all)...\n");
Match m; memset(&m, 0, sizeof m);
double t0 = now_seconds();
int rc = Pattern_search(pat, in, 0, -1, &m);
examples/redos_atomic.c
+75 -39
View File
@@ -1,25 +1,28 @@
/* redos_atomic - demonstrates the one performance feature that is a
* genuine, user-facing correctness lever rather than an internal
* implementation detail: atomic groups (?>...) as a defense against
* catastrophic backtracking (ReDoS), on the textbook (a+)+b shape.
/* redos_atomic - demonstrates atomic groups (?>...) as a defense against
* catastrophic backtracking (ReDoS), on the textbook (a+)+b shape, and
* shows precisely when that defense is still needed versus when it no
* longer is.
*
* README.md "Implementation status" explains why this shape is not fixed
* by this build's memoization alone: (a+)+b has no backreference, so it
* is memoized and therefore no longer exponential, but the inner a+'s
* repeat is followed by the group's closing save, not by a simple atom,
* so the precomputed skip-ahead table that makes a*b linear does not
* apply to it either, leaving it empirically quadratic. Wrapping the
* inner repetition in an atomic group removes the ambiguity a+ was
* causing (once a+ has consumed as many a's as it can, an atomic group
* forbids ever giving any of them back to try a shorter run instead),
* which is a real, structural difference the pattern author controls,
* not a performance tweak this library makes automatically.
*
* This program compiles both forms of the same pattern and times a search
* against an adversarial subject (a long run of 'a' characters with no
* trailing 'b', the one input that forces the most backtracking) at a
* size chosen to make the quadratic form's cost visible in a few seconds
* without making it so large the program is unpleasant to run.
* (a+)+b by itself is no longer a useful demonstration of the problem:
* it has no backreference, lookahead, lookbehind, or atomic group, so it
* is eligible for and automatically matched by the Pike VM (concept.md
* 7.2/7.7), which runs it in genuinely linear time, not merely improved
* time, with no atomic group needed at all (README.md "The Pike VM").
* The backtracking engine's own remaining ceiling for this exact shape
* (empirically quadratic, not exponential, because the inner a+'s
* OP_REPEAT1 is followed by the group's closing save rather than a
* simple atom, so the skip-ahead table that makes a*b linear does not
* apply to it) is real and still documented in README.md "Implementation
* status", but only actually applies to a pattern that keeps a
* construct forcing it onto that engine. This program shows both halves
* directly: first, that the plain (a+)+b form is now fast on its own,
* with no atomic group; second, a variant with a trailing backreference
* added specifically to keep it on the backtracking engine, where the
* same nested-quantifier ambiguity is still exponential (memoization is
* unsound in the presence of a backreference and is disabled whenever
* one is present anywhere in the pattern, README.md "Implementation
* status"), and where the atomic group fix from this file's original
* version still matters exactly as much as it always did.
*/
#define _POSIX_C_SOURCE 199309L
#include "regexx.h"
@@ -47,7 +50,7 @@ static void run_case(const char *label, const char *pattern, const char *subject
double t0 = now_seconds();
int rc = Pattern_search(pat, in, 0, -1, &m);
double t1 = now_seconds();
printf("%-28s pattern=%-14s n=%-8zu rc=%-2d time=%.4fs\n",
printf("%-28s pattern=%-18s n=%-8zu rc=%-2d time=%.4fs\n",
label, pattern, subjlen, rc, t1 - t0);
if (rc == 1) Match_free(&m);
Input_free(in);
@@ -55,24 +58,57 @@ static void run_case(const char *label, const char *pattern, const char *subject
}
int main(void) {
size_t n = 20000;
char *subject = malloc(n + 1);
memset(subject, 'a', n);
subject[n] = '\0'; /* no trailing 'b' anywhere: the worst case for both patterns */
printf("=== Part 1: (a+)+b, backreference-free, now Pike VM eligible ===\n\n");
{
size_t n = 20000;
char *subject = malloc(n + 1);
memset(subject, 'a', n);
subject[n] = '\0';
printf("Searching %zu characters of 'a' with no trailing 'b' (never matches):\n\n", n);
run_case("plain nested quantifier", "(a+)+b", subject, n);
run_case("atomic inner group", "(?>a+)+b", subject, n);
run_case("possessive inner quant.", "(a++)+b", subject, n);
printf("\nAll three finish in comparable, close-to-instant time: the plain form\n"
"is no longer the slow one. It has no backreference, lookahead,\n"
"lookbehind, or atomic group, so it runs on the Pike VM (README.md\n"
"\"The Pike VM\"), which holds a bounded set of live parse states\n"
"instead of backtracking, and is linear in n regardless of this\n"
"pattern's nested-quantifier shape. The atomic and possessive forms\n"
"were not what fixed this; the engine dispatch did.\n\n");
free(subject);
}
printf("Searching %zu characters of 'a' with no trailing 'b' (never matches):\n\n", n);
run_case("plain nested quantifier", "(a+)+b", subject, n);
run_case("atomic inner group", "(?>a+)+b", subject, n);
run_case("possessive inner quant.", "(a++)+b", subject, n);
printf("=== Part 2: the same shape, forced onto the backtracking engine ===\n\n");
{
/* \2 (a backreference to the literal "b") is enough to make this
* pattern ineligible for the Pike VM (no backreference can ever
* be, README.md "Implementation status"), which is the only
* change from Part 1 that matters here: it puts the same
* (a+)+-shaped ambiguity back onto the engine that still pays
* for it. n is deliberately tiny (not the 20000 above): a
* backreference disables memoization entirely (it is unsound
* whenever one is present anywhere in the pattern, not only
* where it appears), so this shape is exponential here, not
* merely quadratic, and n = 24 already takes most of a second. */
size_t n = 24;
char *subject = malloc(n + 1);
memset(subject, 'a', n);
subject[n] = '\0';
printf("Searching %zu characters of 'a' with no trailing \"bb\" (never matches):\n\n", n);
run_case("plain, backreference", "(a+)+(b)\\2", subject, n);
run_case("atomic, backreference", "(?>(a+))+(b)\\2", subject, n);
printf("\nThe plain form is slow again (exponential, not merely quadratic,\n"
"since the backreference disables memoization outright), and the\n"
"atomic group fixes it again, exactly as it always did: wrapping the\n"
"capturing group itself, (?>(a+))+, keeps group 1 capturing (unlike\n"
"(?>a+)+, which would make the inner run non-capturing) while still\n"
"forbidding the atomic sub-program from ever giving back characters\n"
"it already consumed to try a shorter run instead. This is the\n"
"pattern author's own tool for the minority of patterns that still\n"
"need it, not something the engine can fix automatically the way it\n"
"now does for Part 1's plain, backreference-free shape.\n");
free(subject);
}
printf("\nAll three patterns are equivalent in what they can match (none of\n"
"them ever matches this subject). The atomic and possessive forms\n"
"finish close to instantly regardless of n because they never explore\n"
"more than one way of splitting the run of 'a' characters; the plain\n"
"form's cost grows with n because memoization alone (README.md\n"
"\"Implementation status\") makes this shape quadratic, not linear,\n"
"and not exponential only because it lacks a backreference.\n");
free(subject);
return 0;
}