regexx/tests/TEST_PLAN.md

Test Plan: Combinatorial Expansion

This records what tests/cases.py generates, and why, before generating it, so the plan is checkable against the result rather than only inferable from it. Every case listed here follows concept.md Section 11's strategy: ground truth comes from running the same pattern/subject/flags through a real CPython re, not from a hand-derived expectation. tests/gen.py prints the exact final case count on every run; this document fixes the categories and dimensions, not an exact count, since the combinatorics are generated programmatically from the lists below.

Categories

A. Quantifier x grouping x flags combinatorics (search, finditer)

Cartesian product of:

• Atoms: a, [a-c], \d, \w, . (5)
• Quantifier suffixes: *, +, ?, {2,3}, {1,}, *?, +?, ?? (8, greedy and lazy)
• Grouping wrapper: none, (...), (?:...), (?P<g>...) (4)
• Flag combination: none, IGNORECASE, MULTILINE+DOTALL (3)
• Operation: search, finditer (2)

Each atom paired with one subject string chosen to exercise it meaningfully (mixed case, a run of matching and non-matching characters, at least one empty-match-prone case for */?). 5 x 8 x 4 x 3 x 2 = 960 cases.

B. Alternation x backreference x groups (search, fullmatch)

~20 hand-written base patterns combining |, capturing/named groups, and numeric/named backreferences (for example (cat|dog|bird)\1, (?P<x>foo|bar)-(?P=x), (a|b){2,3}\1?), each run against 3 subjects, 2 flag combinations, 2 operations: ~20 x 3 x 2 x 2 = 240 cases.

C. Lookaround combinatorics (search, fullmatch)

~20 hand-written patterns combining (?=...), (?!...), (?<=...), (?<!...) with quantifiers, groups, and each other (nested lookarounds), each run against 3 subjects, 2 operations: ~20 x 3 x 2 = 120 cases.

D. sub/split specific combinatorics

~30 pattern/replacement/count and pattern/maxsplit combinations exercising \g<name>, \g<N>, \N backreferences in replacement templates, count limits, and maxsplit limits, including patterns with 0, 1, and multiple capturing groups (Python interleaves every group's text into split's result list): ~150-200 cases.

E. Real world "combination" patterns (search, finditer, fullmatch, sub)

~40-50 realistic patterns that combine many features at once rather than isolating one (the way patterns are actually written): email-shaped, URL-shaped, IPv4, ISO date, HH:MM:SS time, US-shaped phone number, hex color, semantic version, key=value pairs, CSV-shaped splitting, quoted strings with backslash escapes, log-line-shaped text with multiple named groups, Markdown-style emphasis markers, single-level balanced parentheses, and Unicode word matching across scripts. Each pattern run through 2-3 of the four operations above, whichever are meaningful for it: ~150 cases.

F. Mode cross-checks (search, finditer)

~15 patterns run in ascii, utf8, and binary mode against matched subjects, to check mode-dependent \w/\s/\d/case-folding behavior specifically, not just syntax: ~90 cases.

G. Flag combination stress (search, finditer)

~10 representative patterns run under a wider sweep of flag combinations (IGNORECASE, MULTILINE, DOTALL, VERBOSE, and pairwise combinations) than categories A-C use, to catch flag-interaction bugs specifically: ~160 cases.

H. BINARY mode with genuinely arbitrary raw bytes

Unlike Category F (which exercises BINARY/ASCII/UTF8 mode with ordinary UTF-8-encoded text), these cases use real Python bytes objects containing embedded NUL, high bytes (0x80-0xFF), and byte sequences that are not, and are not meant to be, valid UTF-8 at all: ~20 hand-written cases across search, finditer, sub, and split.

I. UTF8 mode edge cases

Real multi-byte content beyond simple accented Latin: 4-byte (astral plane) code points such as emoji, combining marks, right-to-left scripts (Arabic, Hebrew), CJK, mixed-width strings, and offset correctness for backreferences/groups/lookaround spanning multi-byte characters: ~20 hand-written cases across search, fullmatch, finditer, sub, and split.

J. Seeded random combinatorics

A fixed-seed (0xC0FFEE) random generator building 900 pattern/subject pairs from the same atom/quantifier/grouping vocabulary as Category A, but combined randomly rather than exhaustively, across all three modes and a sweep of flag combinations. The fixed seed makes this a deterministic, reproducible regression test, not a flaky fuzzer: the same 900 cases generate every run, so a failure here is exactly as reproducible and reportable as a hand-written one, while still exploring combinations no one sat down and thought to write by hand.

Target

Roughly 1,900 generated cases for Categories A-G, run against the original 81 hand-written ones (kept, not replaced); Categories H-J added later brought the total to just over 3,200. make test prints the exact current count and the pass/fail result on every run.

Result

3,252 cases generated (0 skipped), all passing, including the original 81. This expansion found and fixed four real defects before they were ever released, load-bearing enough to be worth naming here rather than only in the commit history:

• A C trigraph bug in tests/gen.py's own string-literal encoder: any generated pattern containing ??) (the lazy-? quantifier next to a closing paren, produced by Category A) was silently rewritten by the C compiler to ?] before the test suite ever ran, because a strict-mode C compiler trigraph-converts ??) inside a string literal, not just in code (confirmed by compiling and printing the corrupted string directly). Fixed by escaping every ? as \? in the generated C.
• A real, previously undocumented Pattern_finditer/Pattern_split defect: CPython's empty-match handling additionally searches for, and reports, a second, non-empty match at the same start position whenever the first match found there was empty, which this build did not do. Reverse engineered against a real CPython interpreter (not written down in CPython's own documentation), fixed in regexx.c, and now recorded precisely in concept.md Section 3 and docs/API.md Sections 3.5-3.6 and 3.7.
• tests/gen.py's own ASCII-mode ground truth was wrong (Python str + re.ASCII, which stays in code-point space, instead of a bytes pattern against a bytes subject, which is what regexx's byte-oriented ASCII mode actually is), caught by Category F's mode cross-checks against a subject containing a multi-byte character.
• A real, verified-wrong implementation of the LOCALE flag: an earlier revision treated bytes 0x80-0xFF as word characters under LOCALE in BINARY/ASCII mode, based on an unverified assumption about "C" locale behavior. Caught by Category H, checked directly against both the C standard's guarantee for isalnum() under the "C" locale and a real CPython interpreter with re.LOCALE and the "C" locale explicitly set, both of which classify only ASCII letters and digits. LOCALE is now documented as an accepted no-op in BINARY/ASCII mode, matching verified reality instead of a prior assumption (concept.md 13.4, docs/API.md Section 2, README.md "Known deviations").

Two further, smaller findings were concrete instances of an already-documented category (glibc's wctype.h Unicode tables not matching CPython's own bundled ones exactly, README.md "Known deviations"), not new defects: U+00A0 (NO-BREAK SPACE, Category F) and fullwidth digits U+FF10-U+FF19 (Category I) are both recognized by CPython's \s/\d but not by glibc's iswspace()/iswdigit() under the C.utf8 locale this build relies on. Recorded, not patched, for the same reason already given for the first instance: hand-patching individual code points would start down the path of maintaining an ad hoc table this design deliberately avoided.

After these fixes, the random Category J generator was additionally run against five more seeds (1, 42, 777, 99999, 123456) at 3,000 iterations each, 26,760 further checks beyond the 900 committed here, all passing; only the committed seed's 900 cases are part of the regular test suite, the rest was exploratory validation, not committed as a permanent fixture, to avoid the build slowing down for marginal additional coverage over what is already committed.

All of the above is described in full in README.md, docs/API.md, and concept.md, not only here; this section exists so the connection between "ran a lot of generated tests" and "found these specific, real bugs" is traceable from the plan that produced them.

Explicitly out of scope for this expansion

Constructs regexx intentionally rejects (conditional groups, scoped inline flags, \N{NAME}, POSIX bracket classes, concept.md Section 5) are not exercised here as positive cases, since they are not valid input to generate ground truth for; tests/harness.c's existing hand-written checks already cover that they fail cleanly (a PatternError, not a crash), which this expansion does not need to duplicate at scale.