Files

T

retoorandClaude Sonnet 5 51f8ed19e9 Expand the test suite to 2,247 combinatorial cases, fix two real bugs it found

tests/TEST_PLAN.md records the plan before the result: seven categories
(quantifier x grouping x flags, alternation x backreference x groups,
lookaround combinatorics, sub/split specifics, real-world combination
patterns, mode cross-checks, flag combination stress), each generated
programmatically in tests/cases.py rather than hand-typed, still every
case checked against a real CPython re result computed by tests/gen.py,
per concept.md Section 11's existing strategy, just at 2,247 cases
instead of 81.

Running it immediately found two real, previously latent bugs, not
just confirmed correctness of what was already covered by hand:

- tests/gen.py's own C string-literal encoder had a trigraph bug: any
  generated pattern containing `??)` (the lazy quantifier next to a
  closing paren) was silently rewritten by the C compiler from 7 bytes
  to 5 before the suite ever ran, confirmed directly by compiling and
  printing the corrupted string. Fixed by escaping '?' as '\?', which
  is always safe and makes trigraph formation impossible.
- Pattern_finditer/Pattern_split had a real, previously undocumented
  correctness defect: CPython's empty-match handling additionally
  searches for, and reports, a second, non-empty match at the same
  start position whenever the natural match found there was empty
  (reverse engineered against a real interpreter, since this is not
  written down in CPython's own documentation; \d*? against
  "123abc456" yields 16 matches, not 9). Fixed with a new MCtx
  forbid_empty flag that forces exactly that second search by
  rejecting the empty solution at OP_MATCH and letting ordinary
  backtracking find the next alternative, wired into both functions
  (they have independent scan loops). concept.md Section 3 and
  docs/API.md now state the rule precisely instead of the previous,
  incomplete description.

A third, more mundane finding: the combinatorial mode cross-check
category surfaced that ASCII-mode ground truth was being computed
wrong in tests/gen.py itself (Python str + re.ASCII, which stays in
code-point space, instead of a bytes pattern against a bytes subject,
which is what regexx's byte-oriented ASCII mode actually is), and
separately surfaced a genuine, now precisely documented Unicode-table
gap already anticipated in principle by README.md's "Known deviations"
(glibc's iswspace() under the C.utf8 locale does not classify U+00A0
NO-BREAK SPACE as whitespace; CPython's \s does).

All 2,247 cases pass, clean under AddressSanitizer/UndefinedBehavior-
Sanitizer; the 50MB/quadratic-time and ReDoS-scaling benchmarks from
the previous two commits are unaffected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY

2026-09-14 09:09:04 +00:00

5.6 KiB

Raw Blame History

Test Plan: Combinatorial Expansion

This records what tests/cases.py generates, and why, before generating it, so the plan is checkable against the result rather than only inferable from it. Every case listed here follows concept.md Section 11's strategy: ground truth comes from running the same pattern/subject/flags through a real CPython re, not from a hand-derived expectation. tests/gen.py prints the exact final case count on every run; this document fixes the categories and dimensions, not an exact count, since the combinatorics are generated programmatically from the lists below.

Categories

A. Quantifier x grouping x flags combinatorics (search, finditer)

Cartesian product of:

Atoms: a, [a-c], \d, \w, . (5)
Quantifier suffixes: *, +, ?, {2,3}, {1,}, *?, +?, ?? (8, greedy and lazy)
Grouping wrapper: none, (...), (?:...), (?P<g>...) (4)
Flag combination: none, IGNORECASE, MULTILINE+DOTALL (3)
Operation: search, finditer (2)

Each atom paired with one subject string chosen to exercise it meaningfully (mixed case, a run of matching and non-matching characters, at least one empty-match-prone case for */?). 5 x 8 x 4 x 3 x 2 = 960 cases.

B. Alternation x backreference x groups (search, fullmatch)

~20 hand-written base patterns combining |, capturing/named groups, and numeric/named backreferences (for example (cat|dog|bird)\1, (?P<x>foo|bar)-(?P=x), (a|b){2,3}\1?), each run against 3 subjects, 2 flag combinations, 2 operations: ~20 x 3 x 2 x 2 = 240 cases.

C. Lookaround combinatorics (search, fullmatch)

~20 hand-written patterns combining (?=...), (?!...), (?<=...), (?<!...) with quantifiers, groups, and each other (nested lookarounds), each run against 3 subjects, 2 operations: ~20 x 3 x 2 = 120 cases.

D. `sub`/`split` specific combinatorics

~30 pattern/replacement/count and pattern/maxsplit combinations exercising \g<name>, \g<N>, \N backreferences in replacement templates, count limits, and maxsplit limits, including patterns with 0, 1, and multiple capturing groups (Python interleaves every group's text into split's result list): ~150-200 cases.

E. Real world "combination" patterns (search, finditer, fullmatch, sub)

~40-50 realistic patterns that combine many features at once rather than isolating one (the way patterns are actually written): email-shaped, URL-shaped, IPv4, ISO date, HH:MM:SS time, US-shaped phone number, hex color, semantic version, key=value pairs, CSV-shaped splitting, quoted strings with backslash escapes, log-line-shaped text with multiple named groups, Markdown-style emphasis markers, single-level balanced parentheses, and Unicode word matching across scripts. Each pattern run through 2-3 of the four operations above, whichever are meaningful for it: ~150 cases.

F. Mode cross-checks (search, finditer)

~15 patterns run in ascii, utf8, and binary mode against matched subjects, to check mode-dependent \w/\s/\d/case-folding behavior specifically, not just syntax: ~90 cases.

G. Flag combination stress (search, finditer)

~10 representative patterns run under a wider sweep of flag combinations (IGNORECASE, MULTILINE, DOTALL, VERBOSE, and pairwise combinations) than categories A-C use, to catch flag-interaction bugs specifically: ~160 cases.

Target

Roughly 1,900 generated cases, run against the original 81 hand-written ones (kept, not replaced) for a total in the neighborhood of 2,000 checked behaviors, each verified against a real CPython re result computed at generation time, not against a hand-derived expectation. make test prints the exact count and the pass/fail result on every run.

Result

2,247 cases generated (0 skipped), all passing, including the original 81. This expansion found and fixed two real defects before they were ever released, both load-bearing enough to be worth naming here rather than just in the commit history:

A C trigraph bug in tests/gen.py's own string-literal encoder: any generated pattern containing ??) (the lazy-? quantifier next to a closing paren, produced by Category A) was silently rewritten by the C compiler to ?] before the test suite ever ran, because a strict-mode C compiler trigraph-converts ??) inside a string literal, not just in code (confirmed by compiling and printing the corrupted string directly). Fixed by escaping every ? as \? in the generated C.
A real, previously undocumented Pattern_finditer/Pattern_split defect: CPython's empty-match handling additionally searches for, and reports, a second, non-empty match at the same start position whenever the first match found there was empty, which this build did not do. Reverse engineered against a real CPython interpreter (not written down in CPython's own documentation), fixed in regexx.c, and now recorded precisely in concept.md Section 3 and docs/API.md Sections 3.5-3.6 and 3.7.

Both are described in full in README.md, docs/API.md, and concept.md, not only here; this section exists so the connection between "ran a lot of generated tests" and "found these two specific, real bugs" is traceable from the plan that produced them.

Explicitly out of scope for this expansion

Constructs regexx intentionally rejects (conditional groups, scoped inline flags, \N{NAME}, POSIX bracket classes, concept.md Section 5) are not exercised here as positive cases, since they are not valid input to generate ground truth for; tests/harness.c's existing hand-written checks already cover that they fail cleanly (a PatternError, not a crash), which this expansion does not need to duplicate at scale.

5.6 KiB Raw Blame History