Add a literal prefilter to the Pike VM, profile its memory with Massif

Closed two of the three gaps the previous commit's honest self-review
left open (the third, full streaming/bounded-memory input, remains
out of scope for this pass and is still documented as such).

Literal prefilter (regexx.c, pike_find): when the NFA-only Prog's
first instruction is a mandatory OP_CHAR or OP_CLASS, a pattern
beginning with a required literal or class rather than a nullable
loop or a leading assertion, injecting a fresh unanchored start
thread at a position that instruction would reject is certain to die
on the very next pike_step call regardless; checking that identical
condition before injecting rather than after changes nothing about
which threads ever exist, only how much wasted work is done finding
out. This targeted exactly examples/bench_vs_posix.c's worst
regression: the literal-search scenario went from roughly 70x slower
than POSIX <regex.h> (up from roughly 22x before the Pike VM existed)
down to roughly 11x-13x, better than the original pre-Pike-VM number;
number extraction (starts with a class) improved more modestly; a*b
(starts with a nullable loop, structurally unhelped) is unchanged, as
expected. Verified with the full 3,252-case suite, three clean
AddressSanitizer/UndefinedBehaviorSanitizer passes, and a rerun of
the whitebox dual-engine cross-check (24,000 match/fullmatch/search
plus ~2,700 finditer comparisons between the two engines on the same
compiled patterns, zero mismatches).

Memory profiling (concept.md 7.9): Valgrind/Massif on the same
adversarial, prefilter-proof pattern shape (a*b, nullable leading
loop) used for the backtracking engine's own worst case, for a fair
comparison. Peak heap was almost entirely the 10MB input buffer
itself; the Pike VM's own contribution was roughly 12KB, confirming
the design's O(instruction count x group count), input-length-
independent memory bound actually holds for the v1 implementation,
not only on paper. concept.md Section 10's table, which had only a
"not yet profiled" caveat for this row before, is updated with the
measured result.

README.md, docs/API.md, USAGE.md, and bench_vs_posix.c's own printed
summary are updated throughout with the corrected numbers, rather
than left describing the pre-prefilter regression as current.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY

concept.md

@@ -201,6 +201,16 @@ This section is the concrete plan for a first working implementation of 7.2's re
 
 **A result not anticipated by 7.7, which called the relative performance of the two engines "an open, unmeasured question."** It is no longer open for the specific case that motivated writing 7.7 in the first place: `(a+)+b`, README.md's own textbook example of the backtracking engine's remaining weak spot, contains no backreference and is therefore Pike VM eligible, and now measures as genuinely linear, not the "empirically quadratic" figure 7.5 measured for the backtracking engine on the same pattern (that figure remains accurate for what it was actually measuring, the backtracking engine specifically, which still runs this exact pattern shape when it also contains a backreference or another construct that makes it ineligible for this section's engine; the pattern most often used to illustrate the trade-off simply no longer needs that engine at all). Measured directly: search time against *n* non-matching `a` characters went from roughly 2x per doubling of *n* under the backtracking engine (7.5's figure) to also roughly 2x per doubling under the Pike VM, i.e. linear, confirmed from *n* = 10,000 to *n* = 160,000 (0.0018s to 0.0308s). This is a real, measured improvement for every backreference-free pattern, not a projected one, but it is not the whole picture: re-running `examples/bench_vs_posix.c`'s literal search, number extraction, and `a*b` scenarios, none of them adversarial and all of them now Pike VM eligible, found the gap to POSIX `<regex.h>` on those three *widened*, from roughly 2x-25x before this engine existed (running on the backtracking engine plus 7.5's skip-ahead tables) to roughly 8x-70x now. This is the direct, expected cost of 7.7's deferred performance axis actually being deferred: 7.5's tables were themselves a targeted, measured optimization for exactly this kind of pattern, and the Pike VM has no equivalent yet (no literal prefilter, no lazy DFA state caching, no allocation pooling), so an ordinary pattern that used to benefit from that work now runs on a correctness-first engine that does not have it, a real regression for the common case traded for the `(a+)+b`-class fix above, not a net win averaged across all patterns. Whether the Pike VM ends up faster than the backtracking engine specifically (as opposed to against POSIX) on these same ordinary patterns, which would settle whether 7.5's tables are still worth keeping once this engine has its own optimization pass, remains genuinely unmeasured and is the next open question on this axis.
 
+### 7.9 Implementation finding: the dual-engine cross-check, a literal prefilter, and a memory measurement
+
+Continued work after 7.8 closed three of the items it left open, in the order they were tackled.
+
+**The dual-engine debug mode 7.8 said was not built was built.** Not as a permanent build-time mode as 7.7 originally specified, but as an ad hoc whitebox harness (`#include "regexx.c"` for direct access to `PatternImpl.has_nfa`, toggled on the same compiled `Pattern` to run `match`/`fullmatch`/`search`/`finditer` through both engines and compare every result field by field), run against 6,000 randomly generated eligible patterns (literals, classes, anchors, quantifiers including lazy and bounded forms, alternation, capturing/non-capturing/named groups, nested combinations) across 20 subjects, all three data modes, several flag combinations: 24,000 `match`/`fullmatch`/`search` comparisons and roughly 2,700 `finditer` comparisons, zero mismatches, clean under AddressSanitizer/UndefinedBehaviorSanitizer. Not committed as a permanent test target, the same judgment 7.8 already made and still holds (the CPython-derived suite continues to be what actually finds and localizes divergences); this was additional, freshly gathered evidence on demand, not a change of strategy.
+
+**A literal prefilter was added to `pike_find`'s unanchored injection loop**, the single highest leverage item on 7.7/7.8's deferred performance list, chosen specifically because `examples/bench_vs_posix.c`'s literal-search scenario was the one 7.8 found had regressed the most (roughly 2x before this engine existed to roughly 70x after). When the very first instruction of the NFA-only `Prog` is a mandatory `OP_CHAR` or `OP_CLASS` (a pattern beginning with a required literal or class, not a nullable loop or a leading assertion), injecting a fresh start thread at a position that instruction would reject is certain, by construction, to die on the very next `pike_step` call regardless; checking that identical condition before injecting rather than after changes nothing about which threads ever exist, only how much work is spent finding out that most of them will not. This is the same literal-prefilter idea 7.5 already cites for the backtracking engine's own `compute_next_prevmatch`, applied here in its simplest form, a per-position check rather than a precomputed table, deliberately: `README.md` "Implementation status" already documents that a precomputed skip-ahead table traded for a correctness problem once (the reverted size-cap attempt, 7.6), and this narrower, unconditional check carries no equivalent risk, since it can only ever skip work a fuller simulation would also have discarded. Measured directly: the literal-search scenario dropped from roughly 70x slower than POSIX `<regex.h>` to roughly 11x-13x, better than this build's own pre-Pike-VM baseline (roughly 22x-35x, 7.8); the number-extraction scenario (`[0-9]+`, beginning with a mandatory class) improved from roughly 8x to roughly 7x; the `a*b` scenario, whose leading `a*` is a nullable loop the prefilter structurally cannot help, is unchanged, exactly as expected rather than as a gap. Verified against the dual-engine cross-check above (0 mismatches) and the full CPython-derived suite (3,252/3,252) both before and after.
+
+**The Pike VM's own memory cost, which Section 10's table stated as a design bound without measurement, was profiled with Valgrind/Massif** the same way 7.6 profiled the backtracking engine, on the same adversarial choice of pattern for a fair comparison: `a*b` against 10MB of non-matching `a` characters, the nullable-loop worst case a literal prefilter cannot reduce, so this number is not flattered by the fix above. Peak heap use was 10,497,824 bytes, of which 10,485,761 bytes, 99.89%, is the test harness's own 10MB input buffer, present regardless of which engine runs; the engine's own contribution, compiling the pattern and running the full search, was the remaining 12,063 bytes, an amount that does not grow with input length (the thread lists are cleared and reused every input position, never accumulating) and is small enough, relative to a multi-megabyte input, to be indistinguishable from a constant in practice. This confirms Section 10's `O(instruction count x group count)`, input-length-independent bound for the Pike VM actually holds for the v1 implementation, not only for the design (Section 10's table updated accordingly).
+
 ## 8. Core Data Structures
 
 Kept intentionally minimal, all defined in the single file, no dependency beyond the C standard library (`stdint.h`, `stddef.h`, `string.h`) for the data structures in this section specifically. `Input_from_file` (Section 9.2) is the one place the file as a whole steps outside strict ISO C, using POSIX (`mmap`/`open`/`fstat`, Section 7.6), already in the same family of platform dependency `wctype.h`'s locale behavior carries (Section 13.3).
@@ -343,7 +353,7 @@ A byte sequence in `UTF8` mode that is not valid UTF-8 at the point the decoder
 
 Both figures are stated relative to input length specifically because that is the axis the gigabyte scale requirement constrains; instruction count and group count are properties of the pattern, not the input, and are expected to stay small (tens to low hundreds) for realistically written patterns.
 
-This table describes the two engines as designed. Section 7.5 records that the bounded backtracking engine's v1 implementation, augmented with memoization and precomputed skip-ahead tables, in practice achieves the Regular engine's linear *time* bound for the backreference-free majority of patterns, at the cost of `O(instruction count x input length)` memory rather than the Regular engine's `O(instruction count x group count)`, so the two rows above remain the correct statement of each engine's guarantee, not of what a specific implementation happens to measure. The Regular row's own memory bound, for the Pike VM v1 implementation (7.7/7.8), has not been held to the same standard yet: it is a structural property of the algorithm (a thread list holds at most one thread per instruction, cleared and reused every input position, so its own peak size is bounded by instruction count regardless of input length, independently of whether it is also true in practice), not something separately confirmed with Valgrind/Massif the rigorous way 7.6 confirmed and then corrected the backtracking engine's actual number; a single 20MB check found a peak around 6x the input length for a combined `finditer`/`split`/`sub` run, which is dominated by other things (`MatBuf`, `sub`'s own output buffer, allocator churn from many short-lived thread lists), not necessarily evidence against the thread list bound itself, but this has not been isolated and attributed the way 7.5/7.6 attributed the backtracking engine's cost. Both engines, regardless of this row, still hold the entire input in memory in v1 (7.7's "what is explicitly deferred" paragraph), which this table's "independent of input length" language describes the *engine's own working set* as achieving, not the absence of that separate, already-documented cost.
+This table describes the two engines as designed. Section 7.5 records that the bounded backtracking engine's v1 implementation, augmented with memoization and precomputed skip-ahead tables, in practice achieves the Regular engine's linear *time* bound for the backreference-free majority of patterns, at the cost of `O(instruction count x input length)` memory rather than the Regular engine's `O(instruction count x group count)`, so the two rows above remain the correct statement of each engine's guarantee, not of what a specific implementation happens to measure. The Regular row's own memory bound, for the Pike VM v1 implementation, is now held to the same standard: 7.9 records a Valgrind/Massif profile (the same adversarial, nullable-loop pattern shape `a*b` this document uses for the backtracking engine's own worst case) finding the engine's own contribution, beyond holding the input itself, at roughly 12KB for a 10MB search, an amount that does not grow with input length because the thread lists are cleared and reused every input position rather than accumulated, confirming the design bound in this row holds for the v1 implementation, not only in theory. Both engines, regardless of this row, still hold the entire input in memory in v1 (7.7's "what is explicitly deferred" paragraph), which this table's "independent of input length" language describes the *engine's own working set* as achieving, not the absence of that separate, already-documented cost.
 
 ## 11. Testing Strategy