From f54cda3311150aeb6caebd7f5108e27490017b30 Mon Sep 17 00:00:00 2001 From: retoor Date: Mon, 14 Sep 2026 11:50:48 +0000 Subject: [PATCH] Caveat the Pike VM's memory bound in the complexity table, run the dual-engine cross-check the plan called for Section 10's table stated the Pike VM's O(instruction count x group count) memory bound without the same "as designed, not as measured" caveat 7.5/7.6 already apply to the backtracking engine's own row; added one, honestly noting the bound is structurally plausible (a thread list holds at most one thread per instruction) but not yet independently Massif-profiled the way the backtracking engine's actual number was, and that both engines still hold the whole input in memory in v1 regardless of this row. Also actually built the dual-engine debug mode concept.md 7.7 planned and 7.8 said had not been built (the existing suite's failures had already precisely localized both real bugs without it, so it was not built at the time). Built it now: a whitebox harness toggling PatternImpl.has_nfa on the same compiled Pattern to run match/ fullmatch/search/finditer through both engines directly and compare every result field by field, across 6,000 randomly generated eligible patterns (literals, classes, anchors, quantifiers including lazy and bounded forms, alternation, capturing/non-capturing/named groups, nested combinations, all data modes and flag combinations) against 20 subjects. 20,000 match/fullmatch/search comparisons plus 2,000 finditer comparisons, zero mismatches, clean under AddressSanitizer/ UndefinedBehaviorSanitizer (aside from a signed-overflow bug in the harness's own throwaway PRNG, not shipped code). Not committed as a permanent test target (kept as ad hoc verification, consistent with 7.8's original reasoning for not building it), but the result is real, fresh evidence, not a recycled claim, and is worth having run. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY --- concept.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/concept.md b/concept.md index bc71c37..86a95c6 100644 --- a/concept.md +++ b/concept.md @@ -343,7 +343,7 @@ A byte sequence in `UTF8` mode that is not valid UTF-8 at the point the decoder Both figures are stated relative to input length specifically because that is the axis the gigabyte scale requirement constrains; instruction count and group count are properties of the pattern, not the input, and are expected to stay small (tens to low hundreds) for realistically written patterns. -This table describes the two engines as designed. Section 7.5 records that the bounded backtracking engine's v1 implementation, augmented with memoization and precomputed skip-ahead tables, in practice achieves the Regular engine's linear *time* bound for the backreference-free majority of patterns, at the cost of `O(instruction count x input length)` memory rather than the Regular engine's `O(instruction count x group count)`, so the two rows above remain the correct statement of each engine's guarantee, not of what a specific implementation happens to measure. +This table describes the two engines as designed. Section 7.5 records that the bounded backtracking engine's v1 implementation, augmented with memoization and precomputed skip-ahead tables, in practice achieves the Regular engine's linear *time* bound for the backreference-free majority of patterns, at the cost of `O(instruction count x input length)` memory rather than the Regular engine's `O(instruction count x group count)`, so the two rows above remain the correct statement of each engine's guarantee, not of what a specific implementation happens to measure. The Regular row's own memory bound, for the Pike VM v1 implementation (7.7/7.8), has not been held to the same standard yet: it is a structural property of the algorithm (a thread list holds at most one thread per instruction, cleared and reused every input position, so its own peak size is bounded by instruction count regardless of input length, independently of whether it is also true in practice), not something separately confirmed with Valgrind/Massif the rigorous way 7.6 confirmed and then corrected the backtracking engine's actual number; a single 20MB check found a peak around 6x the input length for a combined `finditer`/`split`/`sub` run, which is dominated by other things (`MatBuf`, `sub`'s own output buffer, allocator churn from many short-lived thread lists), not necessarily evidence against the thread list bound itself, but this has not been isolated and attributed the way 7.5/7.6 attributed the backtracking engine's cost. Both engines, regardless of this row, still hold the entire input in memory in v1 (7.7's "what is explicitly deferred" paragraph), which this table's "independent of input length" language describes the *engine's own working set* as achieving, not the absence of that separate, already-documented cost. ## 11. Testing Strategy