main
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  retoorandClaude Sonnet 5
|
e4fc067aa6 |
Eliminate the Pike VM's per-thread allocation, add a real BMH literal
prefilter, research and reject a lazy DFA cache Researched two production Pike-VM-family engines directly (RE2's nfa.cc, rust-lang/regex's regex-automata), fetched and read, not recalled, specifically for what explains the two weaknesses the last benchmark found (a*b's nullable loop, dense finditer): both avoid per-thread malloc/free, RE2 via a Thread free list, regex-automata via a flat SlotTable indexed directly by NFA state. Also researched Hyperscan's Teddy (SIMD literal matching) and set it aside: it needs platform-specific intrinsics, in tension with this project's single-file, portable-C, simplicity-over-performance priority: a portable Boyer-Moore-Horspool skip was judged proportionate where Teddy was not. Rewrote the Pike VM's memory model accordingly. PikeThread's owned int64_t* is gone; a committed thread's capture row now lives at a fixed offset in PikeList.table, indexed directly by instruction PC (Cox's one-thread-per-PC invariant already made this index unique, so no allocation is needed to store or discard one). Transient rows needed mid-closure, before the eventual terminal PC is known, come from ScratchPool, a fixed block with an explicit free list; a plain bump/decrement counter was tried first and proven incorrect by hand before being written into the file (an OP_SPLIT's second branch's row can outlive several non-branching pushes that reuse an *earlier* row without reallocating it, which only a real free list handles safely regardless of release order). The closure stack is pre-sized once instead of grown by realloc on demand. All three, plus a reusable best-match buffer and the prefilter below, bundle into one PikeEngine, built once per top level Pattern_/re_ call and reused across every match found within it, never cached on PatternImpl itself (that would make concurrent Pattern_search calls on the same compiled Pattern from different threads race on shared state, breaking the existing no-synchronization-needed guarantee for a Pattern nothing mutates). Extended the literal prefilter from a single leading character to the full mandatory literal prefix, with a real Boyer-Moore-Horspool bad-character skip table for BINARY/ASCII mode (UTF8 keeps a without-skip fallback: a byte-indexed table cannot cover code points past 0x10FFFF). The skip-ahead only ever applies to where a new unanchored start is injected, never to advancing sp itself while a thread from an earlier start position is still alive. Researched and did not build a lazy DFA state cache (memoizing a live-instruction-set-plus-byte transition). Not an omission: RE2's own lazy DFA cannot track submatch boundaries, the same structural reason applies here, since every call wants at least group 0's span, and a cached transition only answers whether a match is possible, not which path was taken; using one would need a two-phase architecture deserving its own research-and-plan pass. Checked empirically too: a*b, the case such a cache would help most, has at most two live instructions at any position for its whole run, so there is no repeated state worth caching in the first place. Verified: full 3,252-case suite (four clean runs), a whitebox dual-engine cross-check extended to also cover finditer/split/BINARY mode (30,000 + 3,334 + 2,500 comparisons, zero mismatches), a targeted suite for the new prefilter machinery including the classic Boyer-Moore-Horspool overlapping-suffix correctness trap (12/12), two clean AddressSanitizer/UndefinedBehaviorSanitizer passes on each (a third run of each hit the same pre-existing sandbox flake already documented, confirmed unrelated by retrying clean). Measured: literal search went from 0.64x of the backtracking engine's time (already ahead) to 0.03x (~33x faster), and against POSIX <regex.h> from roughly 11x slower to roughly 2x *faster* than glibc outright; a*b (no prefilter benefit at all) improved from 3.45x slower than backtracking to 1.97x, from the allocation fix alone; dense finditer over [0-9]+/\w+ improved from 3.2x/5.6x slower to 1.7x/2.8x. Full tables and citations in concept.md 7.10; README.md, docs/API.md, USAGE.md, and bench_vs_posix.c's own printed summary updated throughout with the corrected numbers and the full history, not just the final ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY |
||
|
retoorandClaude Sonnet 5
|
09718eaaf6 |
Add Pattern_groupindex_count/_at, closing the groupindex enumeration gap
README.md and docs/API.md both previously documented "no way to list every name a pattern defines without already knowing what to look for" as an accepted limitation of Pattern_groupindex_lookup being the only access to groupindex. It was not actually a hard constraint: the underlying GroupIndex struct (regexx.c) already stores every name and its group number in two parallel arrays, populated once at compile time; only a public accessor was missing. Pattern_groupindex_count returns the number of named groups; Pattern_groupindex_at(self, i, &name) for 0 <= i < count writes the i-th name and returns its 1-based group number, or returns -1 for an out-of-range i. Enumeration order is declaration order, verified against a real CPython 3.11 interpreter to match groupindex's own practical (insertion-order) iteration order, not just assumed. Verified directly (count/name/group-number correctness, matching the exact snippet now in USAGE.md's own output), full 3,252-case suite unaffected (3252/3252, this is a pure accessor addition touching no matching logic), clean AddressSanitizer/UndefinedBehaviorSanitizer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY |
||
|
retoorandClaude Sonnet 5
|
5ae68d4ea6 |
Add a literal prefilter to the Pike VM, profile its memory with Massif
Closed two of the three gaps the previous commit's honest self-review left open (the third, full streaming/bounded-memory input, remains out of scope for this pass and is still documented as such). Literal prefilter (regexx.c, pike_find): when the NFA-only Prog's first instruction is a mandatory OP_CHAR or OP_CLASS, a pattern beginning with a required literal or class rather than a nullable loop or a leading assertion, injecting a fresh unanchored start thread at a position that instruction would reject is certain to die on the very next pike_step call regardless; checking that identical condition before injecting rather than after changes nothing about which threads ever exist, only how much wasted work is done finding out. This targeted exactly examples/bench_vs_posix.c's worst regression: the literal-search scenario went from roughly 70x slower than POSIX <regex.h> (up from roughly 22x before the Pike VM existed) down to roughly 11x-13x, better than the original pre-Pike-VM number; number extraction (starts with a class) improved more modestly; a*b (starts with a nullable loop, structurally unhelped) is unchanged, as expected. Verified with the full 3,252-case suite, three clean AddressSanitizer/UndefinedBehaviorSanitizer passes, and a rerun of the whitebox dual-engine cross-check (24,000 match/fullmatch/search plus ~2,700 finditer comparisons between the two engines on the same compiled patterns, zero mismatches). Memory profiling (concept.md 7.9): Valgrind/Massif on the same adversarial, prefilter-proof pattern shape (a*b, nullable leading loop) used for the backtracking engine's own worst case, for a fair comparison. Peak heap was almost entirely the 10MB input buffer itself; the Pike VM's own contribution was roughly 12KB, confirming the design's O(instruction count x group count), input-length- independent memory bound actually holds for the v1 implementation, not only on paper. concept.md Section 10's table, which had only a "not yet profiled" caveat for this row before, is updated with the measured result. README.md, docs/API.md, USAGE.md, and bench_vs_posix.c's own printed summary are updated throughout with the corrected numbers, rather than left describing the pre-prefilter regression as current. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY |
||
|
retoorandClaude Sonnet 5
|
fa07622671 |
Add a Pike VM (regular engine, concept.md 7.2), dispatched automatically
Researched first (Cox's regexp2 article, the submatch/tagged-NFA follow-up crediting Laurikari, and rust-lang/regex's PikeVM source for the exact leftmost-first mid-search Match handling), then planned in concept.md 7.7 before writing any code, per the explicit instruction to plan before implementing. The existing bytecode compiler turned out to already produce valid Thompson-construction NFA bytecode (OP_SPLIT/OP_JMP/OP_SAVE match Cox's instruction set almost exactly), so no new compiler was needed: Prog gained one field (no_repeat1) and compile_repeat one condition, letting the same compile_node produce a second, NFA-only Prog from the same AST for any pattern containing none of OP_BACKREF/OP_LOOKAHEAD/ OP_LOOKBEHIND/OP_ATOMIC (a possessive quantifier already desugars to the last of these at parse time). That second Prog is matched by a new Pike VM (pike_addthread/pike_step/pike_find): a breadth-first thread list simulation with per-thread capture arrays, epsilon closure implemented with an explicit heap stack rather than C recursion so a pattern with many alternations cannot recurse the C stack, every allocation checked and failing through the existing -1 error convention rather than a NULL dereference. do_one, Pattern_finditer, and Pattern_split each gained an impl->has_nfa branch to this engine, sharing one small helper (find_next) for the "unanchored scan from a position" versus "single anchored attempt at a position" distinction finditer/split's empty-match retry needs. Two real bugs, both found and precisely localized by the existing 3,252-case CPython-derived suite without writing a single test specifically for this engine: OP_MATCH not writing group 0's end position (it has no OP_SAVE; run()'s own OP_MATCH handler sets it directly, and this engine's first version missed replicating that), and an unconditional "thread list empty -> stop" early exit that is wrong for unanchored search specifically, since a freshly injected start thread can die immediately in its own epsilon closure (a leading \b failing outright, repeatedly, inside a longer word like "catalog" for \bcat\b) without that meaning every later position would too. Both fixed; full suite passes, three clean AddressSanitizer/ UndefinedBehaviorSanitizer runs, plus hand-written whitebox checks (a 20,000-branch alternation, UTF8 named groups, BINARY matching across an embedded NUL, greedy/lazy and alternation priority). Measured result: (a+)+b, this project's own running example of the backtracking engine's remaining weak spot, is Pike VM eligible and now measures as genuinely linear (0.0018s to 0.0308s, n=10,000 to 160,000), not merely improved. Measured cost: re-running bench_vs_posix.c's three ordinary scenarios (all now Pike VM eligible too) found the gap to POSIX <regex.h> widened, from roughly 2x-25x before this engine existed to roughly 8x-70x now, the direct, expected cost of this engine's performance axis being explicitly deferred (no literal prefilter, no lazy DFA state caching, no allocation pooling) in favor of correctness first, per the instruction this was built under. Both results, and the reasoning behind deferring the second, are recorded in concept.md 7.7/7.8. README.md, docs/API.md, USAGE.md, and examples/redos_atomic.c and bench_vs_posix.c are updated throughout to describe the new two-engine dispatch accurately, including this real trade-off, rather than leaving the previous single-engine description in place; redos_atomic.c specifically now shows both that (a+)+b no longer needs an atomic group at all and a backreference-forced variant where one still does. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY |
||
|
retoorandClaude Sonnet 5
|
02743cd959 |
Add six feature examples and a direct benchmark against POSIX <regex.h>
Each new program under examples/ isolates one distinct feature rather than being a general purpose tool like the existing rxgrep.c: binary_scan.c (raw byte-range classes including an embedded NUL and an embedded 0x0A), utf8_scripts.c (\w across Latin/Greek/Cyrillic/CJK text, code point versus byte offsets), ascii_logparse.c (named groups against structured log text), redos_atomic.c (atomic groups and possessive quantifiers timed directly against the unprotected form of the textbook (a+)+b ReDoS shape), empty_match_rule.c (CPython's undocumented empty-match retry rule, verified: \d*? against "123abc456" gives 16 matches, not 9), and large_file_search.c (Input_from_file's mmap-backed reading on a generated 100MB file, with elapsed time and peak RSS printed). Every example was compiled and run while writing it; the claims in each file's top comment are checked against its own output, not written by hand and left unverified. Also adds examples/bench_vs_posix.c, a direct, honestly reported comparison against the C standard library's own <regex.h> (regcomp/regexec) on six scenarios at multi-megabyte or multi-hundred-thousand-line scale, using only pattern syntax valid for both engines so they run the identical pattern text. glibc's DFA-backed engine wins five of six scenarios by 2x-35x, which is the expected outcome of a roughly 2000-line backtracking interpreter built for Python `re` compatibility competing against a mature, heavily optimized engine with a much smaller feature set; the sixth scenario has no POSIX equivalent at all (an atomic group). Every scenario's match count is cross-checked between the two engines as an independent correctness signal beyond the existing CPython-derived test suite. Two real issues were found and fixed while building this benchmark, not left in: iterating regexec() over an advancing string pointer is quadratic in practice (no way to bound the search without an implicit NUL-scan on every call), fixed by using REG_STARTEND instead; and a signed integer overflow (undefined behavior, caught by UBSan) in the benchmark's own pseudo-random text generator, fixed by using an unsigned accumulator. README.md and USAGE.md gain pointers to examples/README.md (the new per-example index) and a "Benchmarks" section summarizing the POSIX comparison honestly, including where it loses. The Makefile gains a `make examples` target building all seven programs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY |
||
|
retoorandClaude Sonnet 5
|
71e8bdaaa2 |
Add USAGE.md, a task-oriented guide with compiled and verified examples
docs/API.md is the exhaustive reference; concept.md is the design rationale. Neither is the right document for "I have never used this library before, show me working code," so this adds one that is: compiling a pattern, building an Input from a buffer or a file, match/fullmatch/search, named groups, finditer/findall, split, sub/subn with both template and callback replacement, all six flags, error handling, the module level convenience functions and their cache, re_escape, and a pattern syntax quick reference, each with BINARY, ASCII, or UTF8 examples as appropriate (including the code-point-versus-byte-offset distinction UTF8 mode introduces). Every example was written as a real, compiled program linked against the current regexx.c and run; the output shown in the document is that program's actual output, not a hand-written guess, the same verification standard already used for README.md's own measured numbers. README.md gets a one-line pointer to it alongside the existing pointer to docs/API.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY |