Add a Pike VM (regular engine, concept.md 7.2), dispatched automatically
Researched first (Cox's regexp2 article, the submatch/tagged-NFA follow-up crediting Laurikari, and rust-lang/regex's PikeVM source for the exact leftmost-first mid-search Match handling), then planned in concept.md 7.7 before writing any code, per the explicit instruction to plan before implementing. The existing bytecode compiler turned out to already produce valid Thompson-construction NFA bytecode (OP_SPLIT/OP_JMP/OP_SAVE match Cox's instruction set almost exactly), so no new compiler was needed: Prog gained one field (no_repeat1) and compile_repeat one condition, letting the same compile_node produce a second, NFA-only Prog from the same AST for any pattern containing none of OP_BACKREF/OP_LOOKAHEAD/ OP_LOOKBEHIND/OP_ATOMIC (a possessive quantifier already desugars to the last of these at parse time). That second Prog is matched by a new Pike VM (pike_addthread/pike_step/pike_find): a breadth-first thread list simulation with per-thread capture arrays, epsilon closure implemented with an explicit heap stack rather than C recursion so a pattern with many alternations cannot recurse the C stack, every allocation checked and failing through the existing -1 error convention rather than a NULL dereference. do_one, Pattern_finditer, and Pattern_split each gained an impl->has_nfa branch to this engine, sharing one small helper (find_next) for the "unanchored scan from a position" versus "single anchored attempt at a position" distinction finditer/split's empty-match retry needs. Two real bugs, both found and precisely localized by the existing 3,252-case CPython-derived suite without writing a single test specifically for this engine: OP_MATCH not writing group 0's end position (it has no OP_SAVE; run()'s own OP_MATCH handler sets it directly, and this engine's first version missed replicating that), and an unconditional "thread list empty -> stop" early exit that is wrong for unanchored search specifically, since a freshly injected start thread can die immediately in its own epsilon closure (a leading \b failing outright, repeatedly, inside a longer word like "catalog" for \bcat\b) without that meaning every later position would too. Both fixed; full suite passes, three clean AddressSanitizer/ UndefinedBehaviorSanitizer runs, plus hand-written whitebox checks (a 20,000-branch alternation, UTF8 named groups, BINARY matching across an embedded NUL, greedy/lazy and alternation priority). Measured result: (a+)+b, this project's own running example of the backtracking engine's remaining weak spot, is Pike VM eligible and now measures as genuinely linear (0.0018s to 0.0308s, n=10,000 to 160,000), not merely improved. Measured cost: re-running bench_vs_posix.c's three ordinary scenarios (all now Pike VM eligible too) found the gap to POSIX <regex.h> widened, from roughly 2x-25x before this engine existed to roughly 8x-70x now, the direct, expected cost of this engine's performance axis being explicitly deferred (no literal prefilter, no lazy DFA state caching, no allocation pooling) in favor of correctness first, per the instruction this was built under. Both results, and the reasoning behind deferring the second, are recorded in concept.md 7.7/7.8. README.md, docs/API.md, USAGE.md, and examples/redos_atomic.c and bench_vs_posix.c are updated throughout to describe the new two-engine dispatch accurately, including this real trade-off, rather than leaving the previous single-engine description in place; redos_atomic.c specifically now shows both that (a+)+b no longer needs an atomic group at all and a backreference-forced variant where one still does. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY
This commit is contained in:
Claude Sonnet 5
@@ -20,11 +20,29 @@ all three data modes, from "nothing" to "working code").
|
||||
## Implementation status
|
||||
|
||||
This is a v1 implementation. It is a complete, tested engine for the pattern
|
||||
syntax and operations listed below, executed by a single recursive
|
||||
backtracking engine (`concept.md` Section 7.3) over a fully materialized copy
|
||||
of the input.
|
||||
syntax and operations listed below, executed by one of two engines over a
|
||||
fully materialized copy of the input, chosen automatically and transparently
|
||||
per compiled pattern, never something a caller selects:
|
||||
|
||||
**It does not yet implement the streaming, bounded-memory regular engine of
|
||||
- **A Pike VM** (`concept.md` 7.2/7.7, a Thompson-NFA simulation with
|
||||
per-thread capture tracking), used whenever a pattern contains none of a
|
||||
backreference, a lookahead, a lookbehind, or an atomic group/possessive
|
||||
quantifier (the last of which is already desugared to an atomic group at
|
||||
parse time, `concept.md` Section 4). This is the majority of patterns
|
||||
people actually write by hand, and this engine runs every one of them in
|
||||
genuinely linear time, not merely the practically-linear-for-the-common-
|
||||
case behavior described below for the other engine, because it does not
|
||||
backtrack at all: it holds a bounded set of live parse states and steps
|
||||
them all forward together, one input position at a time.
|
||||
- **A recursive backtracking engine** (`concept.md` Section 7.3), used only
|
||||
for the minority of patterns that need a construct the Pike VM cannot
|
||||
execute at all: a backreference (not a regular-language construct; no
|
||||
Thompson NFA, however built, can execute one), or a lookahead/lookbehind/
|
||||
atomic group (each compiles to a self-contained sub-program executed to a
|
||||
single yes/no/where-it-ended answer, which needs recursive call structure
|
||||
the flat thread-priority simulation does not have, `concept.md` 7.7).
|
||||
|
||||
**Neither engine yet implements the streaming, bounded-memory version of
|
||||
`concept.md` Section 7.2.** `Input` (the abstraction over "a source of
|
||||
chunks", `concept.md` 9.2) is implemented, and `Input_from_file` maps or
|
||||
reads a whole file into memory before matching (see "Memory footprint"
|
||||
@@ -32,7 +50,8 @@ below for exactly how, and for the measured numbers this and other fixes
|
||||
were checked against). Every public function signature is already exactly
|
||||
what the streaming design in `concept.md` specifies, so the non-streaming
|
||||
implementation underneath a given call can be replaced later without
|
||||
changing any caller. Concretely, today:
|
||||
changing any caller. Concretely, today, for the backtracking engine (see
|
||||
"The Pike VM" below for the other engine's own, separate numbers):
|
||||
|
||||
- `Pattern_match`/`Pattern_fullmatch` (a single anchored attempt at a fixed
|
||||
position) run in time proportional to the length of that attempt, and, for
|
||||
@@ -113,21 +132,22 @@ changing any caller. Concretely, today:
|
||||
exactly as they would for CPython's `re`, and exactly as general ReDoS
|
||||
mitigation guidance recommends (linked below).
|
||||
- **A backreference-free pattern shaped like nested, overlapping
|
||||
quantifiers (`(a+)+b`, the textbook ReDoS shape) is no longer
|
||||
exponential, because it has no backreference and so is memoized, but is
|
||||
not necessarily linear either.** Measured directly: `(a+)+b` against *n*
|
||||
characters of `a` with no trailing `b` took over a minute already at
|
||||
*n* = 40 before memoization (exponential; this build's own stress test
|
||||
keeps that case at *n* = 24 for exactly this reason) and 3.2s at
|
||||
*n* = 32,000 after (empirically quadratic: roughly 4x per doubling),
|
||||
because the inner `a+`'s `OP_REPEAT1` is followed by the group's closing
|
||||
save, not by a simple atom, so the skip-ahead technique above does not
|
||||
apply to it, only the memoization does. Exponential to quadratic is a
|
||||
qualitative change (the *n* = 40 case alone went from longer than this
|
||||
project could wait for, to instant), not a complete fix; a pattern
|
||||
actually meant to run unattended against adversarial input should still
|
||||
avoid this shape, or wrap the inner repetition in an atomic group,
|
||||
regardless of the improvement.
|
||||
quantifiers (`(a+)+b`, the textbook ReDoS shape) no longer reaches this
|
||||
engine at all: it has no backreference, lookaround, or atomic group, so
|
||||
it is Pike VM eligible and runs there instead, in genuinely linear time
|
||||
(see "The Pike VM" below).** The figures this bullet used to report for
|
||||
the backtracking engine specifically (empirically quadratic: 3.2s at
|
||||
*n* = 32,000, down from over a minute already at *n* = 40 before
|
||||
memoization) remain accurate for what they actually measure, and still
|
||||
apply to any pattern shaped like this one that *also* contains a
|
||||
backreference or another construct that keeps it on this engine (the
|
||||
inner `a+`'s `OP_REPEAT1` is followed by the group's closing save, not a
|
||||
simple atom, so the skip-ahead technique above does not apply to it,
|
||||
only the memoization does, which is why quadratic, not linear, was and
|
||||
remains this engine's own ceiling for a compound repeat). A pattern
|
||||
actually meant to run unattended against adversarial input on this
|
||||
engine specifically should still avoid this shape, or wrap the inner
|
||||
repetition in an atomic group, exactly as before.
|
||||
|
||||
Further reading on the techniques above: Russ Cox, ["Regular Expression
|
||||
Matching: the Virtual Machine
|
||||
@@ -145,13 +165,44 @@ writeup on ReDoS and catastrophic
|
||||
backtracking](https://snyk.io/blog/redos-and-catastrophic-backtracking/) for
|
||||
the general phenomenon and mitigation guidance.
|
||||
|
||||
### The Pike VM
|
||||
|
||||
`concept.md` 7.7 is the implementation plan, written and researched before
|
||||
any code, for the engine described above; 7.8 records what building it
|
||||
against the existing test suite actually found, in the same "plan, then
|
||||
measured finding" structure this document already uses for the backtracking
|
||||
engine's own two sections (7.5, 7.6). In short: two real bugs, both found
|
||||
and precisely localized by the existing 3,252-case suite without writing a
|
||||
single test specifically for this engine (a missing write of group 0's end
|
||||
position, and an unanchored-search early exit that was correct for "nothing
|
||||
left to run" but wrong for "nothing left to run *yet*", both `concept.md`
|
||||
7.8), a clean AddressSanitizer/UndefinedBehaviorSanitizer pass over the full
|
||||
suite plus a further set of hand-written whitebox checks (a 20,000-branch
|
||||
alternation, `UTF8`-mode named groups, `BINARY`-mode matching across an
|
||||
embedded `NUL`, greedy/lazy and alternation priority), and one measured
|
||||
result worth restating plainly here: `(a+)+b`, this document's own running
|
||||
example of the backtracking engine's remaining weak spot, is Pike VM
|
||||
eligible and now measures as linear, not quadratic, from *n* = 10,000 to
|
||||
*n* = 160,000 (0.0018s to 0.0308s, roughly 2x per doubling of *n*
|
||||
throughout). Relative performance between the two engines on ordinary,
|
||||
non-adversarial patterns is not yet measured (`concept.md` 7.7/7.8 both say
|
||||
so plainly); this section will be updated once it is.
|
||||
|
||||
### Memory footprint
|
||||
|
||||
Measured directly with Valgrind/Massif (a real 10MB search) and by watching
|
||||
`VmRSS`/`VmHWM` on real 200MB-1GB files, not estimated from reading the
|
||||
code. `Pattern_search`/`finditer`/`split`/`sub` (the operations that try
|
||||
more than one start position) currently use, at peak, about **8.4x** the
|
||||
input length in memory for a pattern using `OP_REPEAT1`
|
||||
code, for the **backtracking engine**; the Pike VM's own memory cost has a
|
||||
different shape (a bounded number of threads, each carrying its own small
|
||||
capture array, `concept.md` 7.7) and has not yet been measured the same
|
||||
detailed way, only checked at a 20MB scale in passing (a peak of roughly
|
||||
6x the input length across a combined `finditer`/`split`/`sub` run,
|
||||
`concept.md` 7.8), so the multiplier below is specific to patterns still
|
||||
running on the backtracking engine (a backreference, lookaround, or atomic
|
||||
group present), not a claim about every pattern. `Pattern_search`/`finditer`/
|
||||
`split`/`sub` (the operations that try more than one start position) on
|
||||
that engine currently use, at peak, about **8.4x** the input length in
|
||||
memory for a pattern using `OP_REPEAT1`
|
||||
(`concept.md` 7.5's `compute_maxrun` and `compute_next_prevmatch` tables,
|
||||
`int32_t`-per-input-position each, are the entire remaining cost: 47.75%
|
||||
each in the Massif profile, `alloc_memo`'s bitset a further 4.5%).
|
||||
@@ -380,22 +431,34 @@ standard library's own `<regex.h>` (POSIX `regcomp`/`regexec`, glibc's
|
||||
DFA-backed implementation), on six scenarios at multi-megabyte or
|
||||
multi-hundred-thousand-line scale, using only ERE pattern syntax that
|
||||
regexx also accepts (no `\d`/`\w`/`\s`, no POSIX bracket classes), so both
|
||||
engines run the identical pattern text against the identical subject. Reported without adjustment in either direction: on this
|
||||
machine, glibc's engine is roughly 2x to 35x faster across the five
|
||||
scenarios both engines can run, which is the expected outcome of a
|
||||
roughly 2000-line backtracking interpreter built for Python `re`
|
||||
compatibility competing against a DFA-backed engine with decades of
|
||||
production optimization behind it and a much smaller feature set (no named
|
||||
groups, no lazy quantifiers, no lookaround, no atomic groups, and no way to
|
||||
search past an embedded `NUL` byte at all). The sharpest case is the
|
||||
textbook ReDoS shape `(a+)+b`: measured at roughly 3x slower than glibc in
|
||||
its plain form (a real cost, not a hang, since memoization keeps it
|
||||
quadratic rather than exponential, "Implementation status" above), and, in
|
||||
the same measurement, effectively free once rewritten with an atomic group,
|
||||
a fix POSIX ERE cannot even express. Every scenario's match count is
|
||||
cross-checked between the two engines and reported as agreeing or
|
||||
differing, an independent correctness check beyond the CPython-derived test
|
||||
suite below.
|
||||
engines run the identical pattern text against the identical subject.
|
||||
Every scenario here uses a pattern with no backreference, lookaround, or
|
||||
atomic group, so every one of them runs on the Pike VM ("The Pike VM"
|
||||
above), not the backtracking engine.
|
||||
|
||||
Reported without adjustment in either direction, on this machine: glibc's
|
||||
engine is roughly 8x to 70x faster on the three ordinary scenarios
|
||||
(a literal search, extracting every number from a text, `a*b`), a real,
|
||||
expected regression from where this build measured previously on those
|
||||
same three scenarios (then roughly 2x to 25x, running on the backtracking
|
||||
engine plus its own precomputed skip-ahead tables, before the Pike VM
|
||||
existed at all): the Pike VM has none of that engine's constant-factor
|
||||
optimizations yet (no literal prefilter, no lazy DFA state caching, no
|
||||
allocation pooling, `concept.md` 7.7's explicitly deferred performance
|
||||
axis; correctness came first, per the instruction this engine was built
|
||||
under), and glibc's decades of exactly that kind of optimization show up
|
||||
directly in the gap widening rather than narrowing. The fourth scenario
|
||||
inverts entirely: the textbook ReDoS shape `(a+)+b` now runs *faster than
|
||||
glibc*, with no atomic group needed at all, because a Thompson-NFA
|
||||
simulation has no notion of "try one split, then backtrack and try
|
||||
another" for the classic nested-quantifier ambiguity to exploit in the
|
||||
first place; `examples/redos_atomic.c`'s Part 2 shows the one case this
|
||||
specific fix does not reach (a pattern with a backreference forces the
|
||||
older, backtracking engine regardless of shape), where an atomic group
|
||||
remains the pattern author's own necessary tool, not automatic. Every
|
||||
scenario's match count is cross-checked between the two engines and
|
||||
reported as agreeing or differing, an independent correctness check beyond
|
||||
the CPython-derived test suite below.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
@@ -688,16 +688,26 @@ part of Python `re`'s grammar at all, `concept.md` Section 14). See
|
||||
|
||||
## 17. Performance and memory, in one paragraph
|
||||
|
||||
`match`/`fullmatch` (a single anchored attempt) cost time proportional to
|
||||
that attempt. `search`/`finditer`/`split`/`sub` are linear time, not
|
||||
quadratic, for the common case (no backreference, every quantifier's body a
|
||||
single character/class/`.`), at a measured memory cost of roughly 8.4x the
|
||||
input length for a pattern using a simple-atom quantifier. A pattern with a
|
||||
backreference, or shaped like nested overlapping quantifiers
|
||||
(`(a+)+b`-style), can still be worse than linear, the same way CPython's own
|
||||
`re` can be on the same patterns. `README.md` "Implementation status" and
|
||||
"Memory footprint" have the full measured account, including one fix that
|
||||
was tried, measured, and deliberately reverted; read it before assuming a
|
||||
A pattern with no backreference, lookaround, or atomic group runs on the
|
||||
Pike VM (`README.md` "The Pike VM"), which is genuinely linear time,
|
||||
including on adversarial nested-quantifier shapes like `(a+)+b` that would
|
||||
invite catastrophic backtracking elsewhere, with no atomic group needed;
|
||||
it does not yet have that engine's own constant-factor optimizations
|
||||
(no literal prefilter, no lazy DFA state caching), so an ordinary pattern
|
||||
can currently measure slower in absolute terms than before this engine
|
||||
existed, a real, documented trade-off, not an oversight. Everything else
|
||||
(a backreference, lookaround, or atomic group anywhere) runs on the
|
||||
recursive backtracking engine instead: `match`/`fullmatch` (a single
|
||||
anchored attempt) cost time proportional to that attempt; `search`/
|
||||
`finditer`/`split`/`sub` are linear time, not quadratic, for the common
|
||||
case there (no backreference, every quantifier's body a single
|
||||
character/class/`.`), at a measured memory cost of roughly 8.4x the input
|
||||
length for a pattern using a simple-atom quantifier, and a pattern with a
|
||||
backreference can still be worse than linear, the same way CPython's own
|
||||
`re` can be on the same patterns. `README.md` "Implementation status",
|
||||
"The Pike VM", and "Memory footprint" have the full measured account,
|
||||
including one fix that was tried, measured, and deliberately reverted;
|
||||
read it before assuming a
|
||||
given pattern's cost, rather than assuming linear time and bounded memory
|
||||
apply unconditionally.
|
||||
|
||||
@@ -725,14 +735,15 @@ one distinct feature this section-by-section walkthrough only touches
|
||||
briefly: `binary_scan.c` (raw byte-range classes including embedded `NUL`),
|
||||
`utf8_scripts.c` (`\w` across non-Latin scripts, code point versus byte
|
||||
offsets), `ascii_logparse.c` (named groups against structured log text),
|
||||
`redos_atomic.c` (atomic groups/possessive quantifiers timed directly
|
||||
against the unprotected form of a catastrophic-backtracking pattern),
|
||||
`redos_atomic.c` (the textbook `(a+)+b` ReDoS shape, now fixed
|
||||
automatically by the Pike VM with no atomic group needed, contrasted with
|
||||
a backreference-forced variant where an atomic group is still necessary),
|
||||
`empty_match_rule.c` (the undocumented CPython empty-match retry rule,
|
||||
verified against a real interpreter), `large_file_search.c` (`mmap`-backed
|
||||
file input at 100MB, with elapsed time and peak memory printed), and
|
||||
`bench_vs_posix.c` (a direct, honestly-reported timing comparison against
|
||||
the C standard library's own `<regex.h>` on six scenarios, including the
|
||||
one adversarial pattern shape where the gap is largest and how an atomic
|
||||
group closes it). See [`examples/README.md`](examples/README.md) for the
|
||||
complete list with what each one demonstrates; `make examples` builds all
|
||||
of them.
|
||||
the C standard library's own `<regex.h>` on six scenarios: glibc wins the
|
||||
three ordinary ones by a wide margin, and the one adversarial pattern
|
||||
shape inverts entirely, now beating glibc outright with no atomic group).
|
||||
See [`examples/README.md`](examples/README.md) for the complete list with
|
||||
what each one demonstrates; `make examples` builds all of them.
|
||||
|
||||
+31
@@ -170,6 +170,37 @@ Measuring 7.5's tables' memory cost directly with Valgrind/Massif, rather than t
|
||||
|
||||
A third change was attempted, measured, and reverted, which is recorded here because the negative result is as load-bearing as the two fixes above: capping 7.5's two tables above a size budget and falling back to the plain scan already used when either is unavailable seemed like an obvious way to bound memory for huge inputs. Measured directly against a real 200MB non-matching search, it was strictly worse than doing nothing: both tables are needed together to keep an unbounded-quantifier-followed-by-a-required-literal pattern (`x*y`-shaped) at linear time; disabling either one alone reintroduces the `O(n^2)` time they exist to prevent, and `O(n^2)` at `n` in the hundreds of millions does not finish in practice. A fast, diagnosable allocation failure (every allocation on this path is now checked and fails through the ordinary `PatternError`/`-1` path instead of crashing on an unchecked `NULL`, a separate finding from the same audit) is a better failure mode than a silent, effectively-unbounded hang, so the tables are allocated unconditionally again. This is the same conclusion 7.5 already reached from the time axis, now confirmed from the memory axis too: there is no way to get both bounded memory and linear time out of a precomputed-table technique for this pattern shape; only 7.2's actual streaming automaton gets both at once, by construction, which is why it remains the correct long-term answer for this axis specifically rather than a further iteration on tables. README.md "Memory footprint" carries the full measured numbers.
|
||||
|
||||
### 7.7 Implementation plan: a Pike VM over the materialized buffer (v1 of 7.2)
|
||||
|
||||
This section is the concrete plan for a first working implementation of 7.2's regular engine, written before writing any code, per the practice this document already follows for 7.5/7.6. Three published sources ground it, fetched and read directly rather than recalled from memory, since this document's standing rule is to verify a technique before relying on it: Russ Cox, ["Regular Expression Matching: the Virtual Machine Approach"](https://swtch.com/~rsc/regexp/regexp2.html) (the instruction set, the thread list with one-thread-per-PC deduplication, the `addthread`/step algorithm, capture slots stored per thread); the same author's [research!rsc article on submatch extraction](https://research.swtch.com/regexp2), crediting Ville Laurikari's tagged-NFA work as the origin of storing submatch boundaries in per-thread state; and the `rust-lang/regex` crate's `regex-automata` PikeVM implementation, read directly for its exact rule on what happens when a thread reaches `Match` mid-search under leftmost-first (Perl/Python-style, not POSIX leftmost-longest) semantics: stop processing only the *remaining, strictly lower priority* threads in that step's list, keep any higher priority thread already queued for the next step running (one of them may still produce a more preferred match later), and stop seeding new unanchored start threads once any match has been recorded.
|
||||
|
||||
**A favorable discovery that changes the scope of this work.** Section 7.1's bytecode was designed by direct analogy to Pike's VM, and reading `compile_node`/`compile_repeat` (Section 6) to plan this confirms it did not just take inspiration from that instruction set, it produced output that already *is* one: `N_CONCAT` is sequencing, `N_ALT` and unbounded `N_REPEAT` compile to exactly the `split`/`jmp` shapes Cox's article gives for alternation and `e*`, `N_GROUP` compiles to `save` at `2*i`/`2*i+1`. The one existing implementation detail Cox's minimal instruction set does not have is `OP_REPEAT1`, a backtracking-only fast path (Section 7.5) that collapses a quantified single character/class/`.` into one counted instruction instead of an explicit loop; `compile_repeat` already contains the ordinary `split`/`jmp` loop as its fallback for a *compound* repeated body, so producing pure-NFA bytecode for a simple-atom repeat too is a matter of skipping the `OP_REPEAT1` fast path for that one compile, not writing a new compiler. Concretely: `Prog` gains one field, `no_repeat1`, and `compile_repeat` gains one condition, `if (!pr->no_repeat1 && is_simple_atom(n->a))`, guarding the existing fast path. This means **no new compiler is needed at all**; the existing parser and `compile_node` are reused unchanged, called a second time with `no_repeat1` set, to produce a second `Prog` from the same AST, structurally guaranteed to contain only the instructions a Pike VM understands.
|
||||
|
||||
**Eligibility (what v1 supports, and what it deliberately does not).** `OP_BACKREF` (not a regular-language construct at all; no NFA, however constructed, can execute it) and `OP_LOOKAHEAD`/`OP_LOOKBEHIND`/`OP_ATOMIC` (each compiles to a self-contained sub-program executed to a single yes/no/where-it-ended answer, Section 7.1, which needs the recursive engine's call structure and is not a natural fit for a flat thread-priority simulation) make a pattern ineligible; a possessive quantifier is already desugared to `OP_ATOMIC` at parse time (Section 7.1), so it needs no separate check. Eligibility is decided once, right after compiling the ordinary (backtracking) `Prog` exactly as `has_backref` already is (Section 8): a single scan of the compiled instructions for any of those four opcodes. An eligible pattern gets a second, NFA-only `Prog` compiled immediately afterward, from the same AST, and is matched by the engine below through every `Pattern_`/`re_` entry point; an ineligible pattern is matched exactly as it is today, by Section 7.5's memoized backtracking engine, completely unchanged. This is a strict, permanent subset relationship for v1: nothing about a pattern's eligibility depends on which entry point is called or how large the input is, and the excluded constructs are not planned for a later version of this same engine, since each one is excluded for a structural reason given above, not a temporary implementation gap; a pattern needing one of them keeps the correctness and the existing measured performance characteristics of Section 7.5/7.6 indefinitely.
|
||||
|
||||
**The algorithm.** One forward pass over the position range being searched, `sp` from the caller's `pos` to `endpos`, maintaining two thread lists (current step, next step) exactly as Cox describes:
|
||||
|
||||
- A thread is `{ pc, saved }`, `saved` a heap array of `2 * (ngroups + 1)` offsets, each thread's own copy.
|
||||
- `addthread(list, pc, saved, sp)` is the epsilon closure: `OP_JMP` follows to its target; `OP_SPLIT` duplicates `saved` and recurses into both targets, the higher priority branch (`x`) first, so list order encodes match preference (greedy-first or lazy-first, whichever the existing compiler already chose for this quantifier, Section 7.1, unchanged); `OP_SAVE` writes `sp` into the thread's own `saved` (safe to mutate in place, not copy: by construction, the only place a `saved` array is ever shared between two in-flight branches is the instant `OP_SPLIT` duplicates it, so past that point each copy is exclusively owned by one recursive path) and continues; `OP_ASSERT` (`^ $ \A \Z \b \B`) tests the same condition `run`'s `OP_ASSERT` case already tests (Section 6), reusing that exact logic, and either continues or discards the thread; `OP_CHAR`/`OP_CLASS`/`OP_ANY`/`OP_MATCH` are the only instructions that end the closure and actually get stored in the list, deduplicated by PC (first writer at a given PC this step wins, since it is by construction the higher priority one; every later duplicate is redundant and its `saved` array is freed immediately, Cox's stated optimization for why at most one thread per PC is ever needed).
|
||||
- Per input position: walk the current list in priority order; a `char`/`class`/`any` thread that matches the position is carried into the next list's `addthread` at `pc+1`; a `match` thread is recorded as the best match found so far (subject to `forbid_empty`, Section 7.5's empty-match retry rule, reusing that same flag and semantics, and to an exact-end requirement for `fullmatch`, reusing `require_end`'s existing meaning), and the *remaining, lower priority* threads in this step's list are dropped without being run (the leftmost-first rule verified against `rust-lang/regex` above); threads already queued into the next list from earlier, higher priority threads in this same step keep running. New unanchored start threads (search/`finditer`/`split` only, never `match`/`fullmatch`, which seed exactly one thread at the given `pos` and never inject another) are appended, at lowest priority, at every position up to `endpos`, but only until a match has been recorded; once one has, no further start is injected, since anything it could find is by definition lower priority than a match already in hand. The pass ends when the current list is empty (nothing left that could still improve on the recorded match) or `endpos` is reached; the recorded match, if any, is the answer.
|
||||
- This produces one match, in the same place a single `run_memo` call produces one; `Pattern_search`/`match`/`fullmatch`'s existing call sites, and `Pattern_finditer`/`Pattern_split`'s existing "find the next match, then the empty-match retry" structure, are unchanged in every respect except that the inner "try every position `s` from `start` to `ep`, one `run_memo` call each" loop each of those functions has today is replaced, for an eligible pattern, by one call to this single-pass function, which performs that same search in one forward scan instead of restarting once per candidate start position.
|
||||
|
||||
**Error proofing, decided up front rather than found by audit afterward, per the existing practice (7.6's "failing safely" paragraph, README.md's NULL-check audit).** `addthread`'s epsilon closure is specified recursively above because that is how the source material states it, but it is implemented with an explicit heap stack (a `DArr` of pending `(pc, saved)` pairs, pushed and popped in a loop), not C call recursion: a pattern with a long chain of alternations or nested groups could otherwise recurse as deep as the instruction count, and this document's standing position (Section 6, `MAX_DEPTH`) is that input- or pattern-driven recursion depth is exactly the class of thing that must be bounded or made iterative, not assumed small. Every allocation on this path (`saved` array duplication at each `OP_SPLIT`, the two thread lists, the explicit closure stack) is checked; a failure anywhere aborts the current call and reports it the same way the existing engine reports its own resource exhaustion (`do_one`'s `depth_exceeded` check, Section 6): a `-1` return through the ordinary `PatternError`-free error path already documented for the matching functions (`docs/API.md` Section 3), never a `NULL` dereference. The thread list itself needs no growth logic at all and therefore nothing can overflow it: Cox's one-thread-per-PC invariant means a list never holds more than `Prog.n` threads, a fixed, compile-time-known bound, so both lists are allocated once, sized to `Prog.n`, and never reallocated during the scan.
|
||||
|
||||
**Testing.** The existing 3,252-case suite (`tests/cases.py`, ground truth from a real CPython interpreter, `tests/TEST_PLAN.md`) is not specific to the backtracking engine; every case that happens to compile to an eligible pattern automatically exercises this engine too, at no extra authoring cost, and any divergence between the two engines' output for the same eligible pattern is, by construction, a bug in one of them, not an ambiguity in what the correct answer is. A build-time debug mode, added alongside this engine, runs *both* engines on every eligible test case and asserts their results agree byte-for-byte (match/no-match, every capture span, every group's participation) before comparing either to the CPython-derived expectation, so a regression that made the two engines agree with each other but not with CPython is still caught, and, separately, so a divergence between the two engines specifically (the more likely class of new bug, since CPython-conformance for the covered subset was already established by the existing suite) is reported precisely rather than only as "some test failed."
|
||||
|
||||
**What is explicitly deferred, per the instruction to get a correct first version before a fast one.** Everything Section 7.2 originally specified for the memory axis, streaming: chunked input, releasing a chunk once no live thread still references it, the constant `O(instruction count)` memory bound regardless of input length. V1 runs this engine over the same fully materialized `MatBuf` (Section 8.4/9.3) the existing engine already uses, exactly as 7.5's opening paragraph already documents the backtracking engine doing; this is a second instance of the same, already-acknowledged simplification, not a new one. The performance axis is also explicitly deferred: no lazy/cached DFA state table (RE2's and Rust's `regex` crate's actual production technique, layered on top of an NFA simulation exactly like this one, caching the set of live thread PCs reached from a given state and input byte so repeat visits to the same state skip the closure computation entirely), no literal prefilter, no thread list or `saved`-array pooling to reduce allocation traffic. Whether this engine ends up faster or slower than Section 7.5's already-shipped, already-measured memoized backtracking engine for the patterns both can run is therefore an open, unmeasured question at the time this plan was written, to be settled after v1 exists and can be benchmarked the same way `examples/bench_vs_posix.c` benchmarks the backtracking engine today, not assumed in either direction beforehand.
|
||||
|
||||
### 7.8 Implementation finding: the Pike VM, built and measured
|
||||
|
||||
7.7's plan was implemented essentially as written; this section records what building it against the existing 3,252-case suite actually found, per the practice 7.6 already established for 7.5. `Prog` gained the planned `no_repeat1` field and one extra condition in `compile_repeat`; `re_compile` gained the eligibility scan and, for an eligible pattern, a second `compile_node` call producing `PatternImpl.nfa_prog`; `do_one`, `Pattern_finditer`, and `Pattern_split` each gained an `impl->has_nfa` branch to the new engine, sharing one small helper (`find_next`) for the "unanchored scan starting no earlier than a position" versus "single anchored attempt at exactly a position" distinction `finditer`/`split`'s empty-match retry needs, since that retry is anchored, not a forward scan, and conflating the two was the source of one of the two bugs below.
|
||||
|
||||
**Two real bugs, both found by the existing suite, not by inspection.** First: `OP_MATCH` has no `OP_SAVE` for group 0's end position (`run`'s own `OP_MATCH` handler writes `c->caps[1] = sp` directly, outside the general save mechanism, Section 6); the first version of the Pike VM's `OP_MATCH` case accepted a thread's `saved` array without doing the equivalent write, so `caps[1]` stayed `-1` for every unnested match, an error the existing 1000+ eligible cases caught immediately and precisely (every failure was a span ending in `-1`). Second, and more interesting: an unconditional `if (clist.count == 0) break` after each step, intended as a reasonable-looking early exit once nothing was left to run, is wrong for unanchored search specifically, because "the current thread list is momentarily empty" does not mean "no later start position could still match": a start thread freshly injected at some position can die immediately during its own epsilon closure (a leading `\b` failing outright, which happens repeatedly inside a longer word like "catalog" for the pattern `\bcat\b`) without that implying every later position will too. The existing suite caught this as seven failing cases, all involving `\b`, all finding zero or one match where CPython finds one or two; the fix restricts the early exit to the two cases where it is actually sound, `anchored` (a single seed, never renewed, so a dead list really is final) or `matched` (no further start is seeded once true, so a list that has gone empty after that point can never revive). Both fixes are small, both were found and precisely localized entirely by the existing CPython-derived suite, which was not written with this engine in mind at all; this is the same claim 7.7's testing paragraph predicted for the general case, and it held.
|
||||
|
||||
**The planned build-time dual-engine debug mode (7.7's testing paragraph) was not built.** In practice the existing suite's failures already pinpointed both bugs precisely enough (by pattern, by subject, by exact expected-versus-actual span) that the extra infrastructure was not needed to localize either one; it remains a reasonable thing to add later if a future divergence is ever found that the CPython-derived suite's coverage does not happen to exercise, but building it preemptively, once the simpler approach had already worked twice, was not justified by anything actually observed while doing this work. Targeted whitebox checks were run by hand instead, directly against the compiled library, covering cases the generated suite does not specifically emphasize: a 20,000-branch alternation (iterative `addthread`'s actual reason for existing, Section 7.7's error-proofing paragraph: this compiles and matches correctly, confirming the explicit stack handles a chain of `OP_SPLIT`s far deeper than any hand-written pattern would produce), an empty pattern, `UTF8`-mode named groups, `BINARY`-mode matching across an embedded `NUL` byte, greedy-versus-lazy priority, and alternation branch priority; all passed on the first run after the two fixes above. Two clean AddressSanitizer/UndefinedBehaviorSanitizer passes over the full suite, and a further clean pass over the whitebox checks above, found no memory error or undefined behavior in the new code (thread list and `saved` array ownership, the most manually managed part of this engine, Section 7.7's error-proofing paragraph).
|
||||
|
||||
**A result not anticipated by 7.7, which called the relative performance of the two engines "an open, unmeasured question."** It is no longer open for the specific case that motivated writing 7.7 in the first place: `(a+)+b`, README.md's own textbook example of the backtracking engine's remaining weak spot, contains no backreference and is therefore Pike VM eligible, and now measures as genuinely linear, not the "empirically quadratic" figure 7.5 measured for the backtracking engine on the same pattern (that figure remains accurate for what it was actually measuring, the backtracking engine specifically, which still runs this exact pattern shape when it also contains a backreference or another construct that makes it ineligible for this section's engine; the pattern most often used to illustrate the trade-off simply no longer needs that engine at all). Measured directly: search time against *n* non-matching `a` characters went from roughly 2x per doubling of *n* under the backtracking engine (7.5's figure) to also roughly 2x per doubling under the Pike VM, i.e. linear, confirmed from *n* = 10,000 to *n* = 160,000 (0.0018s to 0.0308s). This is a real, measured improvement for every backreference-free pattern, not a projected one, but it is not the whole picture: re-running `examples/bench_vs_posix.c`'s literal search, number extraction, and `a*b` scenarios, none of them adversarial and all of them now Pike VM eligible, found the gap to POSIX `<regex.h>` on those three *widened*, from roughly 2x-25x before this engine existed (running on the backtracking engine plus 7.5's skip-ahead tables) to roughly 8x-70x now. This is the direct, expected cost of 7.7's deferred performance axis actually being deferred: 7.5's tables were themselves a targeted, measured optimization for exactly this kind of pattern, and the Pike VM has no equivalent yet (no literal prefilter, no lazy DFA state caching, no allocation pooling), so an ordinary pattern that used to benefit from that work now runs on a correctness-first engine that does not have it, a real regression for the common case traded for the `(a+)+b`-class fix above, not a net win averaged across all patterns. Whether the Pike VM ends up faster than the backtracking engine specifically (as opposed to against POSIX) on these same ordinary patterns, which would settle whether 7.5's tables are still worth keeping once this engine has its own optimization pass, remains genuinely unmeasured and is the next open question on this axis.
|
||||
|
||||
## 8. Core Data Structures
|
||||
|
||||
Kept intentionally minimal, all defined in the single file, no dependency beyond the C standard library (`stdint.h`, `stddef.h`, `string.h`) for the data structures in this section specifically. `Input_from_file` (Section 9.2) is the one place the file as a whole steps outside strict ISO C, using POSIX (`mmap`/`open`/`fstat`, Section 7.6), already in the same family of platform dependency `wctype.h`'s locale behavior carries (Section 13.3).
|
||||
|
||||
+1
-1
@@ -145,7 +145,7 @@ Mirror `re.Pattern.match`/`.fullmatch`/`.search` exactly, including the `pos`/`e
|
||||
- `fullmatch`: anchored at `pos`, must also reach exactly `endpos`.
|
||||
- `search`: tries every start position from `pos` to `endpos` inclusive, left to right, and reports the first that admits any match (ordinary backtracking priority decides which match that is at that position, `concept.md` 2.5).
|
||||
|
||||
**Performance:** `match`/`fullmatch` do one anchored attempt and cost time proportional to that attempt alone. `search` tries each candidate start position as a separate attempt, but, unlike a naive backtracking search, does not redo the same work at every one: for a backreference-free pattern, `run_memo` caches every proven failure at the (instruction, position) level, and `OP_REPEAT1` (a quantifier over a single character, class, or `.`) additionally uses precomputed run-length and skip-ahead tables so its own internal work is `O(1)` amortized per position rather than `O(remaining length)`. Together these make `search` linear, not quadratic, for the common case (a pattern with no backreference, built from simple repeated atoms). What is not fixed: a repeat over a *compound* body (`(ab)*`) gets the failure-memoization but not the skip-ahead table, so a pathological compound-repeat pattern can still be worse than linear; and any pattern with a backreference disables memoization entirely (unsound there, see Section 5) and can still be worst-case exponential, exactly as in CPython. See README.md "Implementation status" for the measured numbers on both the fixed case and the remaining one, and for citations to the published techniques this uses.
|
||||
**Performance:** which of two engines runs a given `Pattern` is decided once, at `re_compile` time, and is never a caller's choice (README.md "Implementation status", `concept.md` 7.2/7.7). A pattern with no backreference, lookahead, lookbehind, or atomic group (a possessive quantifier already desugars to the last of these) runs on the Pike VM, a Thompson-NFA simulation with no backtracking at all: every operation in this section is genuinely linear in input length on that engine, including `search` against an adversarial pattern shape like `(a+)+b` that would otherwise invite catastrophic backtracking, with no atomic group needed (README.md "The Pike VM"). Everything else (a backreference anywhere, or a lookaround/atomic construct) runs on the recursive backtracking engine instead: `match`/`fullmatch` there do one anchored attempt at time proportional to that attempt; `search` tries each candidate start position as a separate attempt but, unlike a naive backtracking search, does not redo the same work at every one, since `run_memo` caches every proven failure at the (instruction, position) level for a backreference-free pattern on this engine, and `OP_REPEAT1` additionally uses precomputed skip-ahead tables, together making `search` linear rather than quadratic for a pattern built from simple repeated atoms; a repeat over a *compound* body only gets the failure-memoization, and a pattern with a backreference disables memoization entirely (unsound there, Section 5) and can still be worst-case exponential, exactly as in CPython. The Pike VM currently has none of the backtracking engine's own constant-factor optimizations (no literal prefilter, no lazy DFA state caching), so an ordinary, non-adversarial pattern that happens to be Pike VM eligible can measure slower in absolute terms than the same pattern would have on the backtracking engine, a real, documented trade-off (`concept.md` 7.8), not an oversight. See README.md "Implementation status" and "The Pike VM" for the measured numbers on both engines and citations to the published techniques each uses.
|
||||
|
||||
**Memory:** the tables above cost real, measured memory, not just time complexity: `search`/`finditer`/`split`/`sub` on a pattern using `OP_REPEAT1` peak at roughly 8.4x the input length (measured with Valgrind/Massif; `match`/`fullmatch` do not allocate these tables at all and use proportionally less). README.md "Memory footprint" has the full measured breakdown, including a fix that was tried, measured, and deliberately reverted because it traded a fast allocation failure for an effectively-unbounded hang, which is a worse failure mode, not a better one.
|
||||
|
||||
|
||||
+2
-2
@@ -20,10 +20,10 @@ directly, no build system required.
|
||||
| [`binary_scan.c`](binary_scan.c) | `BINARY` mode: a byte-range character class (`[\x00-\xff]`) matching raw bytes 0-255, including an embedded `NUL` and an embedded `0x0A`, neither of which is a terminator or a line break in this mode the way it would be to a C string function or a line-oriented reader. This is the one thing `ASCII`/`UTF8` mode cannot do, since both introduce either a restricted classification or a decoding step. |
|
||||
| [`utf8_scripts.c`](utf8_scripts.c) | `UTF8` mode: `\w` recognizing letters across Latin, Greek, Cyrillic, and CJK text (not only ASCII), and the resulting difference between `Match_span` (code point units) and `Match_span_byte` (byte units) once a match spans characters that are more than one byte wide. |
|
||||
| [`ascii_logparse.c`](ascii_logparse.c) | `ASCII` mode doing what it is ordinarily used for: parsing structured, line-oriented text with named groups (`(?P<name>...)`) and reading fields back out by name via `Match_group`/`Pattern_groupindex_lookup`, not by a numeric position the caller has to remember. |
|
||||
| [`redos_atomic.c`](redos_atomic.c) | Atomic groups (`(?>...)`) and possessive quantifiers (`a++`) as a real, pattern-author-controlled defense against catastrophic backtracking, timed directly against the plain, unprotected form of the textbook `(a+)+b` shape on a non-matching adversarial input. |
|
||||
| [`redos_atomic.c`](redos_atomic.c) | Two-part: first, that the textbook `(a+)+b` ReDoS shape is now fixed automatically, with no atomic group, because it has no backreference and so runs on the Pike VM (README.md "The Pike VM"); second, a variant with a backreference added specifically to force it back onto the backtracking engine, where the same shape is exponential again and an atomic group (`(?>...)`) is still the pattern author's own necessary fix. |
|
||||
| [`empty_match_rule.c`](empty_match_rule.c) | `Pattern_finditer`/`Pattern_split` reproducing a real CPython interpreter's undocumented empty-match retry rule exactly (`\d*?` against `"123abc456"` yields 16 matches, not 9), found by probing a real interpreter directly rather than by reading its documentation. |
|
||||
| [`large_file_search.c`](large_file_search.c) | `Input_from_file`'s `mmap`-backed reading on a generated 100MB file, with the elapsed time and peak resident memory printed directly so they can be checked against the measured figures in `../README.md` "Memory footprint" rather than taken on faith. |
|
||||
| [`bench_vs_posix.c`](bench_vs_posix.c) | A direct, timed comparison against the C standard library's own `<regex.h>` (POSIX `regcomp`/`regexec`) on six scenarios at multi-megabyte/multi-hundred-thousand-line scale, using only pattern syntax valid in both engines. Reported honestly: glibc's DFA-backed engine wins most scenarios by a wide margin, and the one adversarial pattern shape (`(a+)+b`) where this library's backtracking cost is most visible is shown both in its plain form and fixed with an atomic group, the latter having no POSIX ERE equivalent to compare against at all. |
|
||||
| [`bench_vs_posix.c`](bench_vs_posix.c) | A direct, timed comparison against the C standard library's own `<regex.h>` (POSIX `regcomp`/`regexec`) on six scenarios at multi-megabyte/multi-hundred-thousand-line scale, using only pattern syntax valid in both engines, every one of them Pike VM eligible. Reported honestly, including a real regression: glibc wins the three ordinary scenarios by a wider margin than this build measured before the Pike VM existed (it has none of the backtracking engine's constant-factor optimizations yet), while the adversarial `(a+)+b` scenario inverts entirely and now beats glibc outright, with no atomic group needed. |
|
||||
|
||||
Every program below was compiled and actually run while writing it; the
|
||||
claims in its top-of-file comment (what a real CPython interpreter does,
|
||||
|
||||
+31
-15
@@ -426,21 +426,37 @@ int main(void) {
|
||||
}
|
||||
}
|
||||
|
||||
printf("\nWhat this measures and does not measure: regexx is a backtracking\n"
|
||||
"interpreter built for Python `re` semantics and the three data modes\n"
|
||||
"(concept.md Section 7.3); glibc's <regex.h> is a mature, DFA-backed\n"
|
||||
"engine with decades of optimization behind it and a much smaller\n"
|
||||
"feature set (no named groups, no lazy quantifiers, no lookaround, no\n"
|
||||
"atomic groups, and, structurally, no way to search past an embedded\n"
|
||||
"NUL byte at all, docs/API.md Section 1.3 vs examples/binary_scan.c).\n"
|
||||
"Scenario D is the sharpest illustration of the trade-off this library\n"
|
||||
"makes: memoized backtracking (README.md \"Implementation status\")\n"
|
||||
"keeps a Python-compatible engine off exponential time, not off\n"
|
||||
"quadratic time, for a pattern shape a DFA-backed engine with no\n"
|
||||
"backreference support handles without either problem; the correct\n"
|
||||
"response to that specific gap is the atomic group in row D2, the same\n"
|
||||
"fix a Python `re` user would reach for, not a claim that this engine\n"
|
||||
"matches a DFA's asymptotic behavior in general, which it does not.\n");
|
||||
printf("\nWhat this measures and does not measure: regexx is Python `re`\n"
|
||||
"compatible and dispatches every pattern here (all backreference-,\n"
|
||||
"lookaround-, and atomic-group-free) to its Pike VM (README.md \"The\n"
|
||||
"Pike VM\", concept.md 7.2/7.7), a Thompson-NFA simulation with no\n"
|
||||
"literal prefilter, lazy DFA state caching, or allocation pooling yet\n"
|
||||
"(concept.md 7.7's explicitly deferred performance axis); glibc's\n"
|
||||
"<regex.h> is a mature, DFA-backed engine with decades of that exact\n"
|
||||
"kind of optimization behind it and a much smaller feature set (no\n"
|
||||
"named groups, no lazy quantifiers, no lookaround, no atomic groups,\n"
|
||||
"and, structurally, no way to search past an embedded NUL byte at all,\n"
|
||||
"docs/API.md Section 1.3 vs examples/binary_scan.c). Scenarios A-C\n"
|
||||
"measure that missing optimization layer directly: this build was\n"
|
||||
"previously measured on the same three scenarios, on this engine's\n"
|
||||
"backtracking predecessor plus its own precomputed skip-ahead tables\n"
|
||||
"(README.md \"Implementation status\"), at roughly 2-3x less of a gap\n"
|
||||
"to POSIX than shown here, a real, expected regression for ordinary\n"
|
||||
"patterns from moving to a v1 engine still missing that layer, not\n"
|
||||
"a defect introduced by the new engine's correctness.\n\n"
|
||||
"Scenario D inverts entirely, and is the actual point of building this\n"
|
||||
"engine at all: (a+)+b has no backreference, so it is Pike VM eligible,\n"
|
||||
"and a Thompson-NFA simulation has no notion of \"try one split, then\n"
|
||||
"backtrack and try another\" to begin with, so the classic nested-\n"
|
||||
"quantifier ambiguity this pattern is famous for causing simply does\n"
|
||||
"not exist for this engine; row D1 now runs this pattern faster than\n"
|
||||
"POSIX <regex.h> itself, with no atomic group, not merely fast enough\n"
|
||||
"to stop being the slowest thing on this page. Row D2 (still with no\n"
|
||||
"POSIX equivalent) is retained for comparison, not because D1 needs\n"
|
||||
"fixing anymore: examples/redos_atomic.c's Part 2 shows a pattern this\n"
|
||||
"specific improvement cannot reach (a backreference forces the older\n"
|
||||
"engine regardless of shape), which is where an atomic group remains\n"
|
||||
"the pattern author's own necessary tool, not something automatic.\n");
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -82,8 +82,10 @@ int main(void) {
|
||||
|
||||
printf("Searching (Input_from_file mmaps this file rather than copying it,\n"
|
||||
"so this process's own heap never holds a private copy of the 100MB\n"
|
||||
"file content itself; it still allocates the search tables README.md\n"
|
||||
"\"Memory footprint\" measures, roughly 8.4x the input length)...\n");
|
||||
"file content itself; this pattern has no backreference, lookaround,\n"
|
||||
"or atomic group, so it also runs on the Pike VM, README.md \"The\n"
|
||||
"Pike VM\", which needs none of the backtracking engine's own search\n"
|
||||
"tables, README.md \"Memory footprint\", at all)...\n");
|
||||
Match m; memset(&m, 0, sizeof m);
|
||||
double t0 = now_seconds();
|
||||
int rc = Pattern_search(pat, in, 0, -1, &m);
|
||||
|
||||
+75
-39
@@ -1,25 +1,28 @@
|
||||
/* redos_atomic - demonstrates the one performance feature that is a
|
||||
* genuine, user-facing correctness lever rather than an internal
|
||||
* implementation detail: atomic groups (?>...) as a defense against
|
||||
* catastrophic backtracking (ReDoS), on the textbook (a+)+b shape.
|
||||
/* redos_atomic - demonstrates atomic groups (?>...) as a defense against
|
||||
* catastrophic backtracking (ReDoS), on the textbook (a+)+b shape, and
|
||||
* shows precisely when that defense is still needed versus when it no
|
||||
* longer is.
|
||||
*
|
||||
* README.md "Implementation status" explains why this shape is not fixed
|
||||
* by this build's memoization alone: (a+)+b has no backreference, so it
|
||||
* is memoized and therefore no longer exponential, but the inner a+'s
|
||||
* repeat is followed by the group's closing save, not by a simple atom,
|
||||
* so the precomputed skip-ahead table that makes a*b linear does not
|
||||
* apply to it either, leaving it empirically quadratic. Wrapping the
|
||||
* inner repetition in an atomic group removes the ambiguity a+ was
|
||||
* causing (once a+ has consumed as many a's as it can, an atomic group
|
||||
* forbids ever giving any of them back to try a shorter run instead),
|
||||
* which is a real, structural difference the pattern author controls,
|
||||
* not a performance tweak this library makes automatically.
|
||||
*
|
||||
* This program compiles both forms of the same pattern and times a search
|
||||
* against an adversarial subject (a long run of 'a' characters with no
|
||||
* trailing 'b', the one input that forces the most backtracking) at a
|
||||
* size chosen to make the quadratic form's cost visible in a few seconds
|
||||
* without making it so large the program is unpleasant to run.
|
||||
* (a+)+b by itself is no longer a useful demonstration of the problem:
|
||||
* it has no backreference, lookahead, lookbehind, or atomic group, so it
|
||||
* is eligible for and automatically matched by the Pike VM (concept.md
|
||||
* 7.2/7.7), which runs it in genuinely linear time, not merely improved
|
||||
* time, with no atomic group needed at all (README.md "The Pike VM").
|
||||
* The backtracking engine's own remaining ceiling for this exact shape
|
||||
* (empirically quadratic, not exponential, because the inner a+'s
|
||||
* OP_REPEAT1 is followed by the group's closing save rather than a
|
||||
* simple atom, so the skip-ahead table that makes a*b linear does not
|
||||
* apply to it) is real and still documented in README.md "Implementation
|
||||
* status", but only actually applies to a pattern that keeps a
|
||||
* construct forcing it onto that engine. This program shows both halves
|
||||
* directly: first, that the plain (a+)+b form is now fast on its own,
|
||||
* with no atomic group; second, a variant with a trailing backreference
|
||||
* added specifically to keep it on the backtracking engine, where the
|
||||
* same nested-quantifier ambiguity is still exponential (memoization is
|
||||
* unsound in the presence of a backreference and is disabled whenever
|
||||
* one is present anywhere in the pattern, README.md "Implementation
|
||||
* status"), and where the atomic group fix from this file's original
|
||||
* version still matters exactly as much as it always did.
|
||||
*/
|
||||
#define _POSIX_C_SOURCE 199309L
|
||||
#include "regexx.h"
|
||||
@@ -47,7 +50,7 @@ static void run_case(const char *label, const char *pattern, const char *subject
|
||||
double t0 = now_seconds();
|
||||
int rc = Pattern_search(pat, in, 0, -1, &m);
|
||||
double t1 = now_seconds();
|
||||
printf("%-28s pattern=%-14s n=%-8zu rc=%-2d time=%.4fs\n",
|
||||
printf("%-28s pattern=%-18s n=%-8zu rc=%-2d time=%.4fs\n",
|
||||
label, pattern, subjlen, rc, t1 - t0);
|
||||
if (rc == 1) Match_free(&m);
|
||||
Input_free(in);
|
||||
@@ -55,24 +58,57 @@ static void run_case(const char *label, const char *pattern, const char *subject
|
||||
}
|
||||
|
||||
int main(void) {
|
||||
size_t n = 20000;
|
||||
char *subject = malloc(n + 1);
|
||||
memset(subject, 'a', n);
|
||||
subject[n] = '\0'; /* no trailing 'b' anywhere: the worst case for both patterns */
|
||||
printf("=== Part 1: (a+)+b, backreference-free, now Pike VM eligible ===\n\n");
|
||||
{
|
||||
size_t n = 20000;
|
||||
char *subject = malloc(n + 1);
|
||||
memset(subject, 'a', n);
|
||||
subject[n] = '\0';
|
||||
printf("Searching %zu characters of 'a' with no trailing 'b' (never matches):\n\n", n);
|
||||
run_case("plain nested quantifier", "(a+)+b", subject, n);
|
||||
run_case("atomic inner group", "(?>a+)+b", subject, n);
|
||||
run_case("possessive inner quant.", "(a++)+b", subject, n);
|
||||
printf("\nAll three finish in comparable, close-to-instant time: the plain form\n"
|
||||
"is no longer the slow one. It has no backreference, lookahead,\n"
|
||||
"lookbehind, or atomic group, so it runs on the Pike VM (README.md\n"
|
||||
"\"The Pike VM\"), which holds a bounded set of live parse states\n"
|
||||
"instead of backtracking, and is linear in n regardless of this\n"
|
||||
"pattern's nested-quantifier shape. The atomic and possessive forms\n"
|
||||
"were not what fixed this; the engine dispatch did.\n\n");
|
||||
free(subject);
|
||||
}
|
||||
|
||||
printf("Searching %zu characters of 'a' with no trailing 'b' (never matches):\n\n", n);
|
||||
run_case("plain nested quantifier", "(a+)+b", subject, n);
|
||||
run_case("atomic inner group", "(?>a+)+b", subject, n);
|
||||
run_case("possessive inner quant.", "(a++)+b", subject, n);
|
||||
printf("=== Part 2: the same shape, forced onto the backtracking engine ===\n\n");
|
||||
{
|
||||
/* \2 (a backreference to the literal "b") is enough to make this
|
||||
* pattern ineligible for the Pike VM (no backreference can ever
|
||||
* be, README.md "Implementation status"), which is the only
|
||||
* change from Part 1 that matters here: it puts the same
|
||||
* (a+)+-shaped ambiguity back onto the engine that still pays
|
||||
* for it. n is deliberately tiny (not the 20000 above): a
|
||||
* backreference disables memoization entirely (it is unsound
|
||||
* whenever one is present anywhere in the pattern, not only
|
||||
* where it appears), so this shape is exponential here, not
|
||||
* merely quadratic, and n = 24 already takes most of a second. */
|
||||
size_t n = 24;
|
||||
char *subject = malloc(n + 1);
|
||||
memset(subject, 'a', n);
|
||||
subject[n] = '\0';
|
||||
printf("Searching %zu characters of 'a' with no trailing \"bb\" (never matches):\n\n", n);
|
||||
run_case("plain, backreference", "(a+)+(b)\\2", subject, n);
|
||||
run_case("atomic, backreference", "(?>(a+))+(b)\\2", subject, n);
|
||||
printf("\nThe plain form is slow again (exponential, not merely quadratic,\n"
|
||||
"since the backreference disables memoization outright), and the\n"
|
||||
"atomic group fixes it again, exactly as it always did: wrapping the\n"
|
||||
"capturing group itself, (?>(a+))+, keeps group 1 capturing (unlike\n"
|
||||
"(?>a+)+, which would make the inner run non-capturing) while still\n"
|
||||
"forbidding the atomic sub-program from ever giving back characters\n"
|
||||
"it already consumed to try a shorter run instead. This is the\n"
|
||||
"pattern author's own tool for the minority of patterns that still\n"
|
||||
"need it, not something the engine can fix automatically the way it\n"
|
||||
"now does for Part 1's plain, backreference-free shape.\n");
|
||||
free(subject);
|
||||
}
|
||||
|
||||
printf("\nAll three patterns are equivalent in what they can match (none of\n"
|
||||
"them ever matches this subject). The atomic and possessive forms\n"
|
||||
"finish close to instantly regardless of n because they never explore\n"
|
||||
"more than one way of splitting the run of 'a' characters; the plain\n"
|
||||
"form's cost grows with n because memoization alone (README.md\n"
|
||||
"\"Implementation status\") makes this shape quadratic, not linear,\n"
|
||||
"and not exponential only because it lacks a backreference.\n");
|
||||
|
||||
free(subject);
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -710,6 +710,13 @@ typedef struct {
|
||||
int mode;
|
||||
int flags;
|
||||
int has_backref; /* computed once after compiling; gates memoized search, see run_memo */
|
||||
int no_repeat1; /* compile time only: when set, compile_repeat never emits the
|
||||
* OP_REPEAT1 fast path (Section 6), even for a simple-atom
|
||||
* repeat, so the result contains only instructions the Pike
|
||||
* VM understands (concept.md 7.7). Never set on the ordinary
|
||||
* backtracking Prog every Pattern already compiles; set only
|
||||
* on the second, NFA-only Prog compiled from the same AST for
|
||||
* an eligible pattern, see PatternImpl.nfa_prog. */
|
||||
} Prog;
|
||||
|
||||
static int emit(Prog *pr, int op) {
|
||||
@@ -726,7 +733,7 @@ static int emit(Prog *pr, int op) {
|
||||
static void compile_node(Prog *pr, Node *n);
|
||||
|
||||
static void compile_repeat(Prog *pr, Node *n) {
|
||||
if (is_simple_atom(n->a)) {
|
||||
if (!pr->no_repeat1 && is_simple_atom(n->a)) {
|
||||
int pc = emit(pr, OP_REPEAT1);
|
||||
Inst *in = &pr->insts[pc];
|
||||
in->lo = n->lo; in->hi = n->hi; in->greedy = n->greedy;
|
||||
@@ -1133,6 +1140,307 @@ static int run_memo(MCtx *c, int pc, int64_t sp, int64_t guard_sp, int32_t guard
|
||||
return result;
|
||||
}
|
||||
|
||||
/* ================================================================
|
||||
* 6b. Pike VM (regular engine over an NFA-only Prog, concept.md 7.7)
|
||||
*
|
||||
* Matches a Prog compiled with no_repeat1 set and known, by
|
||||
* construction (the eligibility scan in re_compile), to contain none
|
||||
* of OP_BACKREF/OP_LOOKAHEAD/OP_LOOKBEHIND/OP_ATOMIC/OP_REPEAT1: only
|
||||
* OP_CHAR, OP_CLASS, OP_ANY, OP_SPLIT, OP_JMP, OP_SAVE, OP_MATCH,
|
||||
* OP_ASSERT. Reuses MCtx purely as a bundle of the four fields that
|
||||
* matter here (pr/text/text8/len); its other fields (memo, caps,
|
||||
* repeat_maxrun/nextpm, depth_exceeded, sub_end) are backtracking
|
||||
* engine state this VM never reads or writes.
|
||||
* ================================================================ */
|
||||
|
||||
typedef struct {
|
||||
int32_t pc;
|
||||
int64_t *saved; /* owned by whichever thread list currently holds it;
|
||||
* 2*(ngroups+1) entries */
|
||||
} PikeThread;
|
||||
|
||||
typedef struct {
|
||||
PikeThread *threads;
|
||||
int count;
|
||||
int cap; /* == nfa_prog->n; fixed for the whole search, see the
|
||||
* "no growth logic, nothing can overflow it" argument
|
||||
* in concept.md 7.7 (Cox's one-thread-per-PC bound) */
|
||||
uint8_t *seen; /* size cap; which PCs already have a thread this step */
|
||||
} PikeList;
|
||||
|
||||
static int pikelist_init(PikeList *l, int n) {
|
||||
size_t cap = (size_t)(n > 0 ? n : 1);
|
||||
l->threads = malloc(cap * sizeof(PikeThread));
|
||||
l->seen = calloc(cap, 1);
|
||||
l->count = 0;
|
||||
l->cap = n;
|
||||
return l->threads != NULL && l->seen != NULL;
|
||||
}
|
||||
/* Frees every currently-listed thread's saved array first: correct only
|
||||
* when called at a point where l->count accurately reflects threads this
|
||||
* list still owns (never right after a pike_step() call without first
|
||||
* resetting count to 0, since pike_step() itself always resolves every
|
||||
* entry it was given, by transferring, accepting, or freeing it). */
|
||||
static void pikelist_free(PikeList *l) {
|
||||
for (int i = 0; i < l->count; i++) free(l->threads[i].saved);
|
||||
free(l->threads);
|
||||
free(l->seen);
|
||||
}
|
||||
static void pikelist_clear(PikeList *l) {
|
||||
l->count = 0;
|
||||
memset(l->seen, 0, (size_t)(l->cap > 0 ? l->cap : 1));
|
||||
}
|
||||
|
||||
typedef struct { int32_t pc; int64_t *saved; } PikeWork;
|
||||
|
||||
/* Epsilon closure from (pc0, saved0) at text position sp, added into
|
||||
* list l (deduplicated by pc, first writer per step wins, exactly Cox's
|
||||
* stated invariant). Specified recursively by the source material
|
||||
* (concept.md 7.7) but implemented with an explicit heap stack instead
|
||||
* of C call recursion, so a pattern with many alternations cannot
|
||||
* recurse the C stack as deep as the instruction count.
|
||||
*
|
||||
* Contract: always resolves ownership of saved0 (frees it, or stores it
|
||||
* into l, possibly after being duplicated further along the way) before
|
||||
* returning, on every path, including failure. Returns 0 only on
|
||||
* allocation failure, having freed everything it touched; l's existing
|
||||
* entries (from before this call) are left valid and untouched. */
|
||||
static int pike_addthread(PikeList *l, MCtx *sctx, int32_t pc0, int64_t *saved0, int64_t sp) {
|
||||
int ncaps = 2 * (sctx->pr->ngroups + 1);
|
||||
DArr stack; da_init(&stack, sizeof(PikeWork));
|
||||
PikeWork *w0 = da_push(&stack);
|
||||
if (!w0) { free(saved0); da_free(&stack); return 0; }
|
||||
w0->pc = pc0; w0->saved = saved0;
|
||||
|
||||
while (stack.len > 0) {
|
||||
PikeWork w = ((PikeWork *)stack.data)[stack.len - 1];
|
||||
stack.len--;
|
||||
if (l->seen[w.pc]) { free(w.saved); continue; }
|
||||
l->seen[w.pc] = 1;
|
||||
Inst *in = &sctx->pr->insts[w.pc];
|
||||
switch (in->op) {
|
||||
case OP_JMP: {
|
||||
PikeWork *nw = da_push(&stack);
|
||||
if (!nw) { free(w.saved); goto fail; }
|
||||
nw->pc = in->x; nw->saved = w.saved;
|
||||
break;
|
||||
}
|
||||
case OP_SPLIT: {
|
||||
/* x is higher priority than y (compile_node already chose
|
||||
* x/y per the quantifier's greedy/lazy sense, Section 6/
|
||||
* 7.1); pushing y then x means x is popped next, giving
|
||||
* the same DFS pre-order a recursive addthread(x) then
|
||||
* addthread(y) would, so list order still encodes
|
||||
* priority exactly as it does in the backtracking engine. */
|
||||
int64_t *saved_y = malloc((size_t)ncaps * sizeof(int64_t));
|
||||
if (!saved_y) { free(w.saved); goto fail; }
|
||||
memcpy(saved_y, w.saved, (size_t)ncaps * sizeof(int64_t));
|
||||
PikeWork *wy = da_push(&stack);
|
||||
if (!wy) { free(saved_y); free(w.saved); goto fail; }
|
||||
wy->pc = in->y; wy->saved = saved_y;
|
||||
PikeWork *wx = da_push(&stack);
|
||||
if (!wx) { free(w.saved); goto fail; } /* wy already on the
|
||||
stack; the fail unwind below frees it along with
|
||||
every other still-pending entry */
|
||||
wx->pc = in->x; wx->saved = w.saved;
|
||||
break;
|
||||
}
|
||||
case OP_SAVE: {
|
||||
/* Safe to mutate w.saved in place, not duplicate: by
|
||||
* construction the only place a saved array is ever
|
||||
* shared between two in-flight branches is the instant
|
||||
* OP_SPLIT above duplicates it, so past that point this
|
||||
* copy is exclusively owned along this one path. */
|
||||
w.saved[in->data] = sp;
|
||||
PikeWork *nw = da_push(&stack);
|
||||
if (!nw) { free(w.saved); goto fail; }
|
||||
nw->pc = w.pc + 1; nw->saved = w.saved;
|
||||
break;
|
||||
}
|
||||
case OP_ASSERT: {
|
||||
int ok;
|
||||
switch (in->data) {
|
||||
case A_BOL: ok = (sp == 0) || ((sctx->pr->flags & MULTILINE) && sp > 0 && text_at(sctx, sp - 1) == '\n'); break;
|
||||
case A_EOL: ok = (sp == sctx->len) || (text_at(sctx, sp) == '\n' && ((sctx->pr->flags & MULTILINE) || sp == sctx->len - 1)); break;
|
||||
case A_BOS: ok = (sp == 0); break;
|
||||
case A_EOS: ok = (sp == sctx->len); break;
|
||||
case A_WB: ok = is_word_at(sctx, sp - 1) != is_word_at(sctx, sp); break;
|
||||
case A_NWB: ok = is_word_at(sctx, sp - 1) == is_word_at(sctx, sp); break;
|
||||
default: ok = 0;
|
||||
}
|
||||
if (!ok) { free(w.saved); break; }
|
||||
PikeWork *nw = da_push(&stack);
|
||||
if (!nw) { free(w.saved); goto fail; }
|
||||
nw->pc = w.pc + 1; nw->saved = w.saved;
|
||||
break;
|
||||
}
|
||||
case OP_CHAR: case OP_CLASS: case OP_ANY: case OP_MATCH:
|
||||
if (l->count >= l->cap) { free(w.saved); goto fail; } /* unreachable
|
||||
in practice, see the fixed-bound argument above; guarded
|
||||
rather than assumed */
|
||||
l->threads[l->count].pc = w.pc;
|
||||
l->threads[l->count].saved = w.saved;
|
||||
l->count++;
|
||||
break;
|
||||
default:
|
||||
free(w.saved); /* unreachable: an eligible Prog contains no other op */
|
||||
break;
|
||||
}
|
||||
}
|
||||
da_free(&stack);
|
||||
return 1;
|
||||
fail:
|
||||
for (size_t k = 0; k < stack.len; k++) free(((PikeWork *)stack.data)[k].saved);
|
||||
da_free(&stack);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Processes every thread in clist at position sp: a char/class/any
|
||||
* thread that matches is carried into nlist at pc+1; a match thread is
|
||||
* recorded as the best answer so far (subject to require_end and
|
||||
* forbid_empty, reusing MCtx's exact same-named semantics) and every
|
||||
* strictly lower priority thread remaining in clist this step is
|
||||
* discarded without being run (the leftmost-first rule, concept.md
|
||||
* 7.7, verified against rust-lang/regex's PikeVM). Always resolves
|
||||
* ownership of every clist entry before returning, success or not.
|
||||
* Returns 0 only on allocation failure. */
|
||||
static int pike_step(PikeList *clist, PikeList *nlist, MCtx *sctx, int64_t sp,
|
||||
int64_t require_end, int forbid_empty,
|
||||
int64_t **best_saved, int *matched) {
|
||||
int i;
|
||||
for (i = 0; i < clist->count; i++) {
|
||||
PikeThread th = clist->threads[i];
|
||||
Inst *in = &sctx->pr->insts[th.pc];
|
||||
int consume_ok;
|
||||
switch (in->op) {
|
||||
case OP_CHAR:
|
||||
consume_ok = (sp < sctx->len) && char_eq(text_at(sctx, sp), in->data, sctx->pr->mode, sctx->pr->flags);
|
||||
break;
|
||||
case OP_ANY:
|
||||
consume_ok = (sp < sctx->len) && !(text_at(sctx, sp) == '\n' && !(sctx->pr->flags & DOTALL));
|
||||
break;
|
||||
case OP_CLASS: {
|
||||
Node *cls = ((Node **)sctx->pr->classnodes.data)[in->data];
|
||||
consume_ok = (sp < sctx->len) && class_match(cls, text_at(sctx, sp), sctx->pr->mode, sctx->pr->flags);
|
||||
break;
|
||||
}
|
||||
case OP_MATCH:
|
||||
/* Group 0's end has no OP_SAVE (only explicit user groups
|
||||
* do, Section 7.1's N_GROUP case); run()'s own OP_MATCH
|
||||
* handler sets it directly the same way (Section 6), and
|
||||
* this VM must too. */
|
||||
if ((require_end < 0 || sp == require_end) && !(forbid_empty && sp == th.saved[0])) {
|
||||
th.saved[1] = sp;
|
||||
free(*best_saved);
|
||||
*best_saved = th.saved;
|
||||
*matched = 1;
|
||||
i++;
|
||||
goto discard_rest;
|
||||
}
|
||||
free(th.saved);
|
||||
continue;
|
||||
default:
|
||||
free(th.saved); /* unreachable for an eligible Prog */
|
||||
continue;
|
||||
}
|
||||
if (consume_ok) {
|
||||
if (!pike_addthread(nlist, sctx, th.pc + 1, th.saved, sp + 1)) {
|
||||
for (int j = i + 1; j < clist->count; j++) free(clist->threads[j].saved);
|
||||
return 0;
|
||||
}
|
||||
} else {
|
||||
free(th.saved);
|
||||
}
|
||||
}
|
||||
return 1;
|
||||
discard_rest:
|
||||
for (; i < clist->count; i++) free(clist->threads[i].saved);
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* Finds the single leftmost, highest priority match for an NFA-only
|
||||
* Prog between pos and endpos (inclusive of trying a start at endpos
|
||||
* itself, matching do_one's existing <= convention). anchored seeds
|
||||
* exactly one start thread, at pos, and never injects another
|
||||
* (Pattern_match/fullmatch); unanchored search (anchored == 0) injects
|
||||
* a fresh start thread, at lowest priority, at every position up to
|
||||
* endpos, until a match has been recorded (Pattern_search/finditer/
|
||||
* split), giving one linear pass over [pos, endpos] rather than one
|
||||
* attempt per candidate start position.
|
||||
*
|
||||
* require_end (-1 unconstrained, else the exact end position required)
|
||||
* and forbid_empty mirror MCtx's identically named fields exactly
|
||||
* (Section 6), so fullmatch and the empty-match retry rule (Pattern_
|
||||
* finditer/Pattern_split) behave identically on both engines.
|
||||
*
|
||||
* Returns 1 (match found, caps_out filled with 2*(ngroups+1) entries),
|
||||
* 0 (no match anywhere in range), or -1 (allocation failure partway
|
||||
* through; mirrors run_memo's depth_exceeded signalling an
|
||||
* unrecoverable condition through the same -1 every Pattern_/re_
|
||||
* matching function already documents, docs/API.md Section 3). */
|
||||
static int pike_find(Prog *nfa_pr, const uint32_t *text, const uint8_t *text8, int64_t pos, int64_t endpos,
|
||||
int anchored, int64_t require_end, int forbid_empty, int64_t *caps_out) {
|
||||
MCtx sctx; memset(&sctx, 0, sizeof sctx);
|
||||
sctx.pr = nfa_pr; sctx.text = text; sctx.text8 = text8; sctx.len = endpos;
|
||||
int ncaps = 2 * (nfa_pr->ngroups + 1);
|
||||
|
||||
PikeList clist, nlist;
|
||||
if (!pikelist_init(&clist, nfa_pr->n)) return -1;
|
||||
if (!pikelist_init(&nlist, nfa_pr->n)) { pikelist_free(&clist); return -1; }
|
||||
|
||||
int64_t *best_saved = NULL;
|
||||
int matched = 0, oom = 0;
|
||||
int64_t sp = pos;
|
||||
|
||||
int64_t *seed = malloc((size_t)ncaps * sizeof(int64_t));
|
||||
if (!seed) oom = 1;
|
||||
else {
|
||||
for (int k = 0; k < ncaps; k++) seed[k] = -1;
|
||||
seed[0] = pos;
|
||||
if (!pike_addthread(&clist, &sctx, 0, seed, sp)) oom = 1;
|
||||
}
|
||||
|
||||
while (!oom) {
|
||||
int step_ok = pike_step(&clist, &nlist, &sctx, sp, require_end, forbid_empty, &best_saved, &matched);
|
||||
clist.count = 0; /* pike_step resolved every entry's ownership either way */
|
||||
if (!step_ok) { oom = 1; break; }
|
||||
if (sp >= endpos) break;
|
||||
sp++;
|
||||
if (!anchored && !matched) {
|
||||
int64_t *s2 = malloc((size_t)ncaps * sizeof(int64_t));
|
||||
if (!s2) { oom = 1; break; }
|
||||
for (int k = 0; k < ncaps; k++) s2[k] = -1;
|
||||
s2[0] = sp;
|
||||
if (!pike_addthread(&nlist, &sctx, 0, s2, sp)) { oom = 1; break; }
|
||||
}
|
||||
{ PikeList tmp = clist; clist = nlist; nlist = tmp; }
|
||||
pikelist_clear(&nlist);
|
||||
/* An empty list only means "give up" when nothing will ever be
|
||||
* injected into it again: once anchored (a single seed, never
|
||||
* renewed) or matched (new starts stop being seeded, see above),
|
||||
* an empty list can never become non-empty again. Otherwise, a
|
||||
* momentarily empty list is not the same as "no more positions
|
||||
* are worth trying": a fresh start seeded at some later position
|
||||
* can die immediately during its own epsilon closure (its very
|
||||
* first instruction, commonly \b, failing outright, exactly what
|
||||
* happens repeatedly inside a longer word like "catalog" for the
|
||||
* pattern \bcat\b) without that meaning every later position
|
||||
* will too. Breaking here unconditionally was a real bug, found
|
||||
* by the existing test suite: it stopped the whole search the
|
||||
* first time this happened, before ever reaching a later
|
||||
* position that would have matched. */
|
||||
if (clist.count == 0 && (anchored || matched)) break;
|
||||
}
|
||||
|
||||
pikelist_free(&clist);
|
||||
pikelist_free(&nlist);
|
||||
if (oom) { free(best_saved); return -1; }
|
||||
if (!matched) return 0;
|
||||
memcpy(caps_out, best_saved, (size_t)ncaps * sizeof(int64_t));
|
||||
free(best_saved);
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* ================================================================
|
||||
* 7. Pattern / Match / Input structures
|
||||
* ================================================================ */
|
||||
@@ -1144,6 +1452,11 @@ typedef struct {
|
||||
GroupIndex gidx;
|
||||
NodePool pool;
|
||||
char *pattern_copy;
|
||||
int has_nfa; /* 1 if nfa_prog below is valid and should be preferred, concept.md 7.7 */
|
||||
Prog nfa_prog; /* compiled only when has_nfa; contains no OP_BACKREF/OP_LOOKAHEAD/
|
||||
* OP_LOOKBEHIND/OP_ATOMIC/OP_REPEAT1, matched by pike_find() instead
|
||||
* of run_memo(). Shares classnodes' underlying Node* with prog (both
|
||||
* point into the one AST pool, freed once via PatternImpl.pool). */
|
||||
} PatternImpl;
|
||||
|
||||
struct Input {
|
||||
@@ -1270,11 +1583,34 @@ Pattern *re_compile(const char *pattern, size_t len, int flags, PatternError *er
|
||||
da_init(&prog.classnodes, sizeof(Node *));
|
||||
compile_node(&prog, ast);
|
||||
emit(&prog, OP_MATCH);
|
||||
for (int k = 0; k < prog.n; k++) if (prog.insts[k].op == OP_BACKREF) { prog.has_backref = 1; break; }
|
||||
/* Pike VM eligibility (concept.md 7.7): a pattern is ineligible iff
|
||||
* its ordinary Prog contains a construct with no pure-NFA execution
|
||||
* (a backreference, or a sub-program instruction: lookahead,
|
||||
* lookbehind, atomic group; a possessive quantifier is already
|
||||
* OP_ATOMIC by the time it reaches here, Section 4, so it needs no
|
||||
* separate check). One scan serves both this and has_backref. */
|
||||
int nfa_eligible = 1;
|
||||
for (int k = 0; k < prog.n; k++) {
|
||||
switch (prog.insts[k].op) {
|
||||
case OP_BACKREF: prog.has_backref = 1; nfa_eligible = 0; break;
|
||||
case OP_LOOKAHEAD: case OP_LOOKBEHIND: case OP_ATOMIC: nfa_eligible = 0; break;
|
||||
default: break;
|
||||
}
|
||||
}
|
||||
|
||||
PatternImpl *impl = calloc(1, sizeof(PatternImpl));
|
||||
impl->prog = prog;
|
||||
impl->pool = g_pool;
|
||||
if (nfa_eligible) {
|
||||
Prog nprog; memset(&nprog, 0, sizeof nprog);
|
||||
nprog.ngroups = p.ngroups; nprog.mode = mode; nprog.flags = all_flags;
|
||||
nprog.no_repeat1 = 1;
|
||||
da_init(&nprog.classnodes, sizeof(Node *));
|
||||
compile_node(&nprog, ast);
|
||||
emit(&nprog, OP_MATCH);
|
||||
impl->nfa_prog = nprog;
|
||||
impl->has_nfa = 1;
|
||||
}
|
||||
impl->pattern_copy = xstrndup(pattern, len);
|
||||
impl->gidx.n = (int)p.groupnames.len;
|
||||
impl->gidx.names = malloc(sizeof(char *) * (size_t)(impl->gidx.n ? impl->gidx.n : 1));
|
||||
@@ -1303,6 +1639,7 @@ void Pattern_free(Pattern *self) {
|
||||
if (impl) {
|
||||
free(impl->prog.insts);
|
||||
da_free(&impl->prog.classnodes);
|
||||
if (impl->has_nfa) { free(impl->nfa_prog.insts); da_free(&impl->nfa_prog.classnodes); }
|
||||
nodepool_free_all(&impl->pool);
|
||||
for (int k = 0; k < impl->gidx.n; k++) free(impl->gidx.names[k]);
|
||||
free(impl->gidx.names);
|
||||
@@ -1613,6 +1950,25 @@ static int do_one(Pattern *self, Input *string, int64_t pos, int64_t endpos, int
|
||||
int64_t p0 = clamp(pos, 0, mb.len);
|
||||
PatternImpl *impl = (PatternImpl *)self->program;
|
||||
int ng = impl->prog.ngroups;
|
||||
|
||||
/* concept.md 7.7: an eligible pattern is matched by the Pike VM
|
||||
* instead, in one call rather than the per-position loop below (that
|
||||
* loop's entire job, finding the next match starting no earlier than
|
||||
* a given position, is pike_find's job too, done in a single linear
|
||||
* pass instead of one attempt per candidate start). Skips memo/
|
||||
* maxrun/nextpm entirely: those exist only to make the loop below
|
||||
* linear (Section 7.5), and this path was never quadratic in the
|
||||
* first place. */
|
||||
if (impl->has_nfa) {
|
||||
int64_t *ncaps; alloc_caps(&ncaps, ng);
|
||||
int r = pike_find(&impl->nfa_prog, mb.text, mb.text8, p0, ep, anchored, fullmatch ? ep : -1, 0, ncaps);
|
||||
if (r < 0) { free(ncaps); free_matbuf(&mb); return -1; }
|
||||
if (r == 0) { free(ncaps); free_matbuf(&mb); return 0; }
|
||||
fill_match_from_caps(out, self, string, p0, ep, ncaps, ng, &mb);
|
||||
free_matbuf(&mb);
|
||||
return 1;
|
||||
}
|
||||
|
||||
int64_t *caps; alloc_caps(&caps, ng);
|
||||
uint8_t *memo = alloc_memo(&impl->prog, ep);
|
||||
/* Only search (anchored == 0) tries more than one position, so
|
||||
@@ -1651,6 +2007,37 @@ int Pattern_search(Pattern *self, Input *string, int64_t pos, int64_t endpos, Ma
|
||||
return do_one(self, string, pos, endpos, 0, 0, out);
|
||||
}
|
||||
|
||||
/* Shared by Pattern_finditer/Pattern_split's two find points each (the
|
||||
* primary "next match starting no earlier than `start`" search, and the
|
||||
* "anchored, non-empty" empty-match retry at the exact same position,
|
||||
* concept.md 7.7): dispatches to whichever engine this Pattern uses.
|
||||
* anchored=0 is an unanchored scan of [start, ep]; anchored=1 is a
|
||||
* single attempt at exactly `start` (the retry's actual shape: it is
|
||||
* not a forward scan, matching the original backtracking call it
|
||||
* replaces). memo/maxrun/nextpm are the backtracking engine's per-scan
|
||||
* state and are ignored when impl->has_nfa (Section 7.7: the Pike VM
|
||||
* needs none of them, having never been quadratic in the first place).
|
||||
* Returns 1 (found, caps filled), 0 (no match), or -1 (unrecoverable;
|
||||
* caller must clean up and return -1, exactly as a depth_exceeded
|
||||
* backtracking failure already required before this engine existed). */
|
||||
static int find_next(PatternImpl *impl, MatBuf *mb, int64_t ep, int64_t start, int anchored, int forbid_empty,
|
||||
uint8_t *memo, int32_t **maxrun, int32_t **nextpm, int64_t *caps, int ng) {
|
||||
if (impl->has_nfa)
|
||||
return pike_find(&impl->nfa_prog, mb->text, mb->text8, start, ep, anchored, -1, forbid_empty, caps);
|
||||
int found = 0;
|
||||
int64_t last = anchored ? start : ep;
|
||||
for (int64_t s = start; s <= last && !found; s++) {
|
||||
reset_caps(caps, ng);
|
||||
caps[0] = s;
|
||||
MCtx c; c.pr = &impl->prog; c.text = mb->text; c.text8 = mb->text8; c.len = ep; c.caps = caps;
|
||||
c.depth_exceeded = 0; c.require_end = -1; c.sub_end = -1; c.memo = memo;
|
||||
c.repeat_maxrun = maxrun; c.repeat_nextpm = nextpm; c.forbid_empty = forbid_empty;
|
||||
found = run_memo(&c, 0, s, -1, -1, 0);
|
||||
if (!found && c.depth_exceeded) return -1;
|
||||
}
|
||||
return found ? 1 : 0;
|
||||
}
|
||||
|
||||
int Pattern_finditer(Pattern *self, Input *string, int64_t pos, int64_t endpos, MatchIterCb cb, void *ctx) {
|
||||
MatBuf mb;
|
||||
if (!build_matbuf(self, string, &mb, NULL)) return -1;
|
||||
@@ -1669,17 +2056,8 @@ int Pattern_finditer(Pattern *self, Input *string, int64_t pos, int64_t endpos,
|
||||
int count = 0;
|
||||
while (start <= ep) {
|
||||
int64_t *caps; alloc_caps(&caps, ng);
|
||||
int found = 0;
|
||||
int64_t s;
|
||||
for (s = start; s <= ep && !found; s++) {
|
||||
reset_caps(caps, ng);
|
||||
caps[0] = s;
|
||||
MCtx c; c.pr = &impl->prog; c.text = mb.text; c.text8 = mb.text8; c.len = ep; c.caps = caps;
|
||||
c.depth_exceeded = 0; c.require_end = -1; c.sub_end = -1; c.memo = memo;
|
||||
c.repeat_maxrun = maxrun; c.repeat_nextpm = nextpm; c.forbid_empty = 0;
|
||||
found = run_memo(&c, 0, s, -1, -1, 0);
|
||||
if (!found && c.depth_exceeded) { free(caps); free(memo); free_repeat_maxrun(&impl->prog, maxrun); free_repeat_nextpm(&impl->prog, nextpm); free_matbuf(&mb); return -1; }
|
||||
}
|
||||
int found = find_next(impl, &mb, ep, start, 0, 0, memo, maxrun, nextpm, caps, ng);
|
||||
if (found < 0) { free(caps); free(memo); free_repeat_maxrun(&impl->prog, maxrun); free_repeat_nextpm(&impl->prog, nextpm); free_matbuf(&mb); return -1; }
|
||||
if (!found) { free(caps); break; }
|
||||
Match m; memset(&m, 0, sizeof m);
|
||||
MatBuf shallow = mb; shallow.cp_to_byte = mb.cp_to_byte; /* share for lookup, copied inside fill */
|
||||
@@ -1699,13 +2077,8 @@ int Pattern_finditer(Pattern *self, Input *string, int64_t pos, int64_t endpos,
|
||||
* next (longer) alternative on its own if one exists. */
|
||||
if (mend == mstart) {
|
||||
int64_t *caps2; alloc_caps(&caps2, ng);
|
||||
reset_caps(caps2, ng);
|
||||
caps2[0] = mstart;
|
||||
MCtx c2; c2.pr = &impl->prog; c2.text = mb.text; c2.text8 = mb.text8; c2.len = ep; c2.caps = caps2;
|
||||
c2.depth_exceeded = 0; c2.require_end = -1; c2.sub_end = -1; c2.memo = memo;
|
||||
c2.repeat_maxrun = maxrun; c2.repeat_nextpm = nextpm; c2.forbid_empty = 1;
|
||||
int found2 = run_memo(&c2, 0, mstart, -1, -1, 0);
|
||||
if (!found2 && c2.depth_exceeded) { free(caps2); free(memo); free_repeat_maxrun(&impl->prog, maxrun); free_repeat_nextpm(&impl->prog, nextpm); free_matbuf(&mb); return -1; }
|
||||
int found2 = find_next(impl, &mb, ep, mstart, 1, 1, memo, maxrun, nextpm, caps2, ng);
|
||||
if (found2 < 0) { free(caps2); free(memo); free_repeat_maxrun(&impl->prog, maxrun); free_repeat_nextpm(&impl->prog, nextpm); free_matbuf(&mb); return -1; }
|
||||
if (found2) {
|
||||
Match m2; memset(&m2, 0, sizeof m2);
|
||||
fill_match_from_caps(&m2, self, string, mstart, ep, caps2, ng, &shallow);
|
||||
@@ -1758,16 +2131,8 @@ int Pattern_split(Pattern *self, Input *string, int maxsplit, MatchIterCb cb, vo
|
||||
while (start <= ep) {
|
||||
if (maxsplit > 0 && n >= maxsplit) break;
|
||||
int64_t *caps; alloc_caps(&caps, ng);
|
||||
int found = 0; int64_t s;
|
||||
for (s = start; s <= ep && !found; s++) {
|
||||
reset_caps(caps, ng);
|
||||
caps[0] = s;
|
||||
MCtx c; c.pr = &impl->prog; c.text = mb.text; c.text8 = mb.text8; c.len = ep; c.caps = caps;
|
||||
c.depth_exceeded = 0; c.require_end = -1; c.sub_end = -1; c.memo = memo;
|
||||
c.repeat_maxrun = maxrun; c.repeat_nextpm = nextpm; c.forbid_empty = 0;
|
||||
found = run_memo(&c, 0, s, -1, -1, 0);
|
||||
if (!found && c.depth_exceeded) { free(caps); free(memo); free_repeat_maxrun(&impl->prog, maxrun); free_repeat_nextpm(&impl->prog, nextpm); free_matbuf(&mb); return -1; }
|
||||
}
|
||||
int found = find_next(impl, &mb, ep, start, 0, 0, memo, maxrun, nextpm, caps, ng);
|
||||
if (found < 0) { free(caps); free(memo); free_repeat_maxrun(&impl->prog, maxrun); free_repeat_nextpm(&impl->prog, nextpm); free_matbuf(&mb); return -1; }
|
||||
if (!found) { free(caps); break; }
|
||||
emit_split_item(cb, ctx, self, string, &mb, seg_start, caps[0]);
|
||||
for (int g = 1; g <= ng; g++) emit_split_item(cb, ctx, self, string, &mb, caps[2 * g], caps[2 * g + 1]);
|
||||
@@ -1779,13 +2144,8 @@ int Pattern_split(Pattern *self, Input *string, int maxsplit, MatchIterCb cb, vo
|
||||
if (mend == mstart) {
|
||||
if (maxsplit <= 0 || n < maxsplit) {
|
||||
int64_t *caps2; alloc_caps(&caps2, ng);
|
||||
reset_caps(caps2, ng);
|
||||
caps2[0] = mstart;
|
||||
MCtx c2; c2.pr = &impl->prog; c2.text = mb.text; c2.text8 = mb.text8; c2.len = ep; c2.caps = caps2;
|
||||
c2.depth_exceeded = 0; c2.require_end = -1; c2.sub_end = -1; c2.memo = memo;
|
||||
c2.repeat_maxrun = maxrun; c2.repeat_nextpm = nextpm; c2.forbid_empty = 1;
|
||||
int found2 = run_memo(&c2, 0, mstart, -1, -1, 0);
|
||||
if (!found2 && c2.depth_exceeded) { free(caps2); free(memo); free_repeat_maxrun(&impl->prog, maxrun); free_repeat_nextpm(&impl->prog, nextpm); free_matbuf(&mb); return -1; }
|
||||
int found2 = find_next(impl, &mb, ep, mstart, 1, 1, memo, maxrun, nextpm, caps2, ng);
|
||||
if (found2 < 0) { free(caps2); free(memo); free_repeat_maxrun(&impl->prog, maxrun); free_repeat_nextpm(&impl->prog, nextpm); free_matbuf(&mb); return -1; }
|
||||
if (found2) {
|
||||
emit_split_item(cb, ctx, self, string, &mb, mstart, caps2[0]);
|
||||
for (int g = 1; g <= ng; g++) emit_split_item(cb, ctx, self, string, &mb, caps2[2 * g], caps2[2 * g + 1]);
|
||||
|
||||
@@ -6,16 +6,22 @@
|
||||
*
|
||||
* Implementation status relative to concept.md: this is the v1
|
||||
* implementation. It provides the full public API and pattern syntax
|
||||
* described below, executed by a single recursive backtracking engine
|
||||
* (concept.md Section 7.3) over a fully materialized copy of the
|
||||
* input, augmented with memoization and precomputed skip-ahead tables
|
||||
* (regexx.c: run_memo, compute_maxrun, compute_next_prevmatch) that
|
||||
* make the search-family functions (Pattern_search/finditer/split/sub)
|
||||
* run in linear time for the common, backreference-free case rather
|
||||
* than the quadratic time a plain backtracking search would have. The
|
||||
* streaming, bounded-memory regular engine of Section 7.2 is still not
|
||||
* implemented; see README.md "Implementation status" for the complete,
|
||||
* itemized, measured account of what v1 does and does not cover.
|
||||
* described below, executed by one of two engines over a fully
|
||||
* materialized copy of the input, chosen automatically per compiled
|
||||
* pattern (concept.md Section 7.2/7.7): a Pike VM (a Thompson-NFA
|
||||
* simulation, genuinely linear time, no backtracking at all) for any
|
||||
* pattern with no backreference, lookaround, or atomic group; a
|
||||
* recursive backtracking engine (concept.md Section 7.3), augmented
|
||||
* with memoization and precomputed skip-ahead tables (regexx.c:
|
||||
* run_memo, compute_maxrun, compute_next_prevmatch) that make the
|
||||
* search-family functions (Pattern_search/finditer/split/sub) run in
|
||||
* linear time for the common, backreference-free case there too,
|
||||
* otherwise. Neither engine yet implements the streaming, bounded-
|
||||
* memory version of Section 7.2; see README.md "Implementation status"
|
||||
* and "The Pike VM" for the complete, itemized, measured account of
|
||||
* what v1 does and does not cover, including a real, documented
|
||||
* performance trade-off between the two engines that is not yet
|
||||
* resolved (concept.md 7.8).
|
||||
*/
|
||||
#ifndef REGEXX_H
|
||||
#define REGEXX_H
|
||||
|
||||
Reference in New Issue
Block a user