concept.md described the bounded backtracking engine (7.3) as plain
recursive backtracking, with the Regular engine's linear-time guarantee
reserved for patterns the streaming Pike VM (7.2) handles. v1 has no
Pike VM yet, so every pattern actually runs on 7.3's engine; the
previous commit added memoization and precomputed skip-ahead tables to
that engine specifically because plain backtracking search was
quadratic even for ordinary patterns, not just exponential for
pathological ones.
Added Section 7.5 recording that finding precisely: what the two
techniques buy (linear time for the backreference-free majority,
quadratic instead of exponential for the textbook (a+)+b shape), what
they do not buy (7.2's bounded, input-length-independent memory
guarantee, still the correct target for that axis), and citations to
the published techniques they match. Added a short note to the Section
10 complexity table pointing at 7.5 so the table is read as the two
engines' designed guarantees, not conflated with what a specific
implementation measures. Updated regexx.c/regexx.h's top-of-file
comments, which still described a plain backtracking engine.
No code changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY
Root cause: do_one/Pattern_finditer/Pattern_split tried every start
position as an independent from-scratch backtracking attempt, so a
pattern that ultimately fails, or matches only very late, redid the
bulk of its work at every position. Measured: a*b over a run of a's
with no b took 19.2s at n=80,000 (quadratic, confirmed by the ~4x
slowdown per doubling).
Two techniques fix it, researched against and matching published
prior art rather than invented ad hoc:
- run_memo caches every proven match failure at the (instruction, text
position) level, never a success (so it cannot change which match is
found), and is only enabled when the compiled program has no
OP_BACKREF anywhere, since a backreference's outcome depends on
capture history, not on position alone. This is the same "memoize
only failures" scheme described in recent work on backtracking regex
matchers (Selective Memoization for Efficient Backtracking Regular
Expression Matching; Efficient Matching with Memoization for Regexes
with Look-around and Atomic Grouping).
- compute_maxrun and compute_next_prevmatch precompute, once per
Pattern_search/finditer/split call, how far a simple-atom quantifier
(OP_REPEAT1) can run from any position and, when it is immediately
followed by a single literal/class/., the rightmost position where
that next atom can match. This lets the backtrack loop jump straight
to candidates worth trying instead of visiting every position in
between, the same idea RE2 and Rust's regex crate call a literal
prefilter, implemented here with a precomputed array instead of a
SIMD memchr/memmem call, in keeping with concept.md's convenience
over performance priority.
Result: a*b at n=80,000 dropped from 19.2s to 0.0016s, now scaling
linearly. As a side effect, since it needs no backreference, the
textbook ReDoS pattern (a+)+b also went from exponential (already
impractical past n=40) to empirically quadratic (3.2s at n=32,000),
though not linear: the inner a+'s OP_REPEAT1 is followed by the
group's closing save rather than a simple atom, so the skip-ahead
table does not apply to it, only the failure memoization does.
README.md and docs/API.md are updated with the measured numbers, the
precise remaining limitations, and citations to the sources this was
checked against.
No test behavior changed: all 81 cases (checked against CPython's own
re module output) still pass, clean under AddressSanitizer/UBSan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY
Auditing the documentation against the actual code (not against what I
remembered writing) surfaced a real, previously undocumented defect:
README.md claimed a backreference-free, unbounded-lookahead-free
pattern "runs in linear time", but that is only true of Pattern_match/
Pattern_fullmatch (one anchored attempt). Pattern_search tries every
candidate start position as an independent from-scratch attempt, so it
is quadratic in the worst case even for the simplest pattern, since
concept.md Section 7.2's engine (which shares work across start
positions in one linear pass) is not implemented yet. Measured directly
with a*b over a run of plain a's: 0.32s at n=10,000, 19.2s at n=80,000,
an ~4x slowdown per doubling. Pattern_finditer/Pattern_split/
Pattern_sub all build on the same search loop and inherit it.
README.md and docs/API.md now state this precisely, with the measured
numbers, wherever the affected functions are documented, rather than
repeating the incorrect blanket "linear time" claim. No code changed;
this is a documentation correction only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY
concept.md is the full design document: the objective (Python re parity
plus binary/ASCII/UTF-8 modes, gigabyte-scale input, single C file), the
automata-theory argument for why unrestricted backreferences/lookaround
are incompatible with strict single-pass constant memory, the resulting
two-engine architecture, the exact Python-mirroring naming convention,
and a full comparison against POSIX regex.h for C-background readers.
regexx.c/regexx.h are the v1 implementation: parser, compiler to a
Pike/backtracking-style bytecode, and a single recursive backtracking
engine covering the pattern syntax and operations listed in README.md,
validated against CPython's own re module output (tests/), clean under
AddressSanitizer/UBSan, and stress-tested (50MB simple-quantifier match,
graceful failure rather than a crash on complex repeats over large
input, clean rejection of every intentionally unsupported construct).
Also included: examples/rxgrep.c (a small grep-like program exercising
all three data modes and the substitution API), the Makefile, the MIT
LICENSE, and docs/API.md, an exhaustive reference for every type, flag,
and function's exact return-value and memory-ownership convention,
checked against the current source and against a real CPython
interpreter rather than against memory.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY
retoorandClaude Sonnet 5