diff --git a/README.md b/README.md index c3b85bb..3347937 100644 --- a/README.md +++ b/README.md @@ -30,14 +30,33 @@ already exactly what the streaming design in `concept.md` specifies, so the non-streaming implementation underneath a given call can be replaced later without changing any caller. Concretely, today: -- A pattern with no backreference and no unbounded-width lookahead runs in - linear time and, for the common case of a single character, class, or `.` - repeated by a quantifier, in *O(1)* recursion depth regardless of input - size (the `OP_REPEAT1` fast path). A repeated *compound* sub-pattern (for - example `(ab)*`) still recurses once per repetition, bounded by a - configurable depth limit (`MAX_DEPTH` in `regexx.c`, currently 60000); past - that limit, matching fails with a reported error rather than a stack - overflow or a wrong answer. +- `Pattern_match`/`Pattern_fullmatch` (a single anchored attempt at a fixed + position) run in time proportional to the length of that attempt, and, for + the common case of a single character, class, or `.` repeated by a + quantifier, in *O(1)* recursion depth regardless of input size (the + `OP_REPEAT1` fast path). A repeated *compound* sub-pattern (for example + `(ab)*`) still recurses once per repetition, bounded by a configurable + depth limit (`MAX_DEPTH` in `regexx.c`, currently 60000, not exposed + through the public API, so changing it means editing `regexx.c` and + rebuilding); past that limit, matching fails with a reported error rather + than a stack overflow or a wrong answer. +- **`Pattern_search`/`Pattern_finditer`/`Pattern_split`/`Pattern_sub` are + quadratic, not linear, in the worst case, even for a pattern with no + backreference and no unbounded-width lookahead.** v1's search loop tries + every start position in turn as an independent match attempt (`concept.md` + Section 7.2's design shares work across start positions in one linear + pass; that engine is not implemented yet, see above), so a pattern that + ultimately does not match, or matches only very late, redoes the bulk of + its work at every position it tries. Measured directly: `a*b` searched + over *n* bytes of `a` with no `b` anywhere took 0.32s at *n* = 10,000 and + 19.2s at *n* = 80,000, an approximately 4x slowdown per doubling, the + signature of `O(n^2)`. This is unrelated to, and in addition to, the + backreference/lookahead ReDoS caveat below; it applies even to the + simplest possible non-matching pattern, and is the most significant open + gap between this build and the "gigabyte scale" objective in `concept.md` + Section 1 for the search-family operations specifically (`Pattern_match`/ + `Pattern_fullmatch`, which never try more than one position, are not + affected). - A pattern with a backreference or an unbounded-width lookahead can, like CPython's own `_sre`, take worst-case exponential time on an adversarial input (`concept.md` Section 5, 13.2); this is the same catastrophic diff --git a/docs/API.md b/docs/API.md index 7fc554e..241db6e 100644 --- a/docs/API.md +++ b/docs/API.md @@ -145,6 +145,8 @@ Mirror `re.Pattern.match`/`.fullmatch`/`.search` exactly, including the `pos`/`e - `fullmatch`: anchored at `pos`, must also reach exactly `endpos`. - `search`: tries every start position from `pos` to `endpos` inclusive, left to right, and reports the first that admits any match (ordinary backtracking priority decides which match that is at that position, `concept.md` 2.5). +**Performance:** `match`/`fullmatch` do one anchored attempt and cost time proportional to that attempt alone. `search` tries each candidate start position as an *independent* attempt from scratch, so it is quadratic, not linear, in the worst case (a pattern that does not match, or matches only very late) even for a pattern with no backreference and no unbounded-width lookahead; see README.md "Implementation status" for a measured example and the reason (`concept.md` Section 7.2's engine, which shares work across start positions in one linear pass, is not implemented yet). + ### 3.5-3.6 `Pattern_finditer` / `Pattern_findall` ```c @@ -152,7 +154,7 @@ int Pattern_finditer(Pattern *self, Input *string, int64_t pos, int64_t endpos, int Pattern_findall(Pattern *self, Input *string, int64_t pos, int64_t endpos, MatchIterCb cb, void *ctx); ``` -`Pattern_findall` is defined as a call to `Pattern_finditer` with the same arguments; both invoke `cb(ctx, m)` once per non-overlapping match, left to right, applying CPython's own empty-match rule (an empty match advances one unit and is not merged with an adjacent non-empty match at the same start, `concept.md` 3). Returns the number of matches found, or `-1` on error. This build does not collapse a no-groups match down to "just the matched string" or a multi-group match to a Python tuple the way `re.findall` does at the Python level; the callback always receives a full `Match`, from which the caller reads whatever it needs via `Match_group`. This is a deliberate simplification: `findall`'s string/tuple collapsing is a Python-object-model convenience with no C equivalent to collapse into, so this build gives the caller the same, uniform `Match`-based access `finditer` does, and the two functions exist separately only for name-for-name parity with `re.findall`/`re.finditer`. +`Pattern_findall` is defined as a call to `Pattern_finditer` with the same arguments; both invoke `cb(ctx, m)` once per non-overlapping match, left to right, applying CPython's own empty-match rule (an empty match advances one unit and is not merged with an adjacent non-empty match at the same start, `concept.md` 3). Returns the number of matches found, or `-1` on error. Inherits `search`'s worst-case quadratic time, Section 3.2-3.4's performance note; each match found restarts the same from-scratch search for the next one. This build does not collapse a no-groups match down to "just the matched string" or a multi-group match to a Python tuple the way `re.findall` does at the Python level; the callback always receives a full `Match`, from which the caller reads whatever it needs via `Match_group`. This is a deliberate simplification: `findall`'s string/tuple collapsing is a Python-object-model convenience with no C equivalent to collapse into, so this build gives the caller the same, uniform `Match`-based access `finditer` does, and the two functions exist separately only for name-for-name parity with `re.findall`/`re.finditer`. ### 3.7 `Pattern_split` @@ -160,7 +162,7 @@ int Pattern_findall(Pattern *self, Input *string, int64_t pos, int64_t endpos, M int Pattern_split(Pattern *self, Input *string, int maxsplit, MatchIterCb cb, void *ctx); ``` -`cb` is invoked once per element of the list `re.split()` would return, **in order**: this is the entire contract, and it is unambiguous by construction (earlier drafts of this function called `cb` twice per match with the caller left to infer which call meant what; that design was replaced before release specifically because it was ambiguous). Each element's text is read via `Match_group(m, NULL, 0, &out, &outlen)`; a `0` return from that call means this element is Python's `None` (an unparticipated capturing group between two matches), matching how an unparticipated group reports on any other `Match`. `maxsplit` matches `re.split`'s parameter (`0` means unlimited). Returns the number of matches that were split on (not the number of list elements), or `-1` on error. +`cb` is invoked once per element of the list `re.split()` would return, **in order**: this is the entire contract, and it is unambiguous by construction (earlier drafts of this function called `cb` twice per match with the caller left to infer which call meant what; that design was replaced before release specifically because it was ambiguous). Each element's text is read via `Match_group(m, NULL, 0, &out, &outlen)`; a `0` return from that call means this element is Python's `None` (an unparticipated capturing group between two matches), matching how an unparticipated group reports on any other `Match`. `maxsplit` matches `re.split`'s parameter (`0` means unlimited). Returns the number of matches that were split on (not the number of list elements), or `-1` on error. Built on the same from-scratch-per-position search as `Pattern_search`; inherits its worst-case quadratic time, Section 3.2-3.4. ### 3.8-3.9 `Pattern_sub` / `Pattern_subn` @@ -175,7 +177,7 @@ Mirror `re.Pattern.sub`/`.subn`. Exactly one of `repl` (a template string) or `c `count` matches `re.sub`'s `count` parameter (`0` means unlimited; a positive `count` stops substituting after that many matches, leaving the rest of the subject, including any further matches within it, untouched, exactly as CPython leaves it). -`Pattern_sub` and `Pattern_subn` differ only in whether the number of substitutions actually made is reported back through `n` (mirroring `re.sub` returning just the string versus `re.subn` returning `(string, count)`); `Pattern_sub` is implemented as a call to `Pattern_subn` with a throwaway `n`. +`Pattern_sub` and `Pattern_subn` differ only in whether the number of substitutions actually made is reported back through `n` (mirroring `re.sub` returning just the string versus `re.subn` returning `(string, count)`); `Pattern_sub` is implemented as a call to `Pattern_subn` with a throwaway `n`. Both are implemented on top of `Pattern_finditer` and so inherit its worst-case quadratic time, Section 3.2-3.4. `*out` is a freshly `malloc`'d, `NUL`-terminated buffer of length `*outlen`; the caller must `free` it. On `-1` (error), `*out`/`*outlen` are left untouched.