tai/taicpp/docs/fuzzy.md

# FuzzyMatcher: "which candidate did the user mean"

Code: `src/util/fuzzy.h`, `src/util/fuzzy.cpp`. Tests: `tests/fuzzy_test.cpp` (`build/fuzzy_test`, step
`fuzzy.unit` of `tests/run_all.py`). Research: `research/fuzzy-matching-selectors/` (12 sources read,
53 listed, conclusion `generic-fuzzy-scorer-design`). This is a taicpp component; the Python app has no
equivalent. The code is the only definition of the scoring; this page states what it does today.

## 1. What it is for

One scorer, used wherever a short, typed, possibly wrong word has to pick one item out of a list of
identifiers. Today that is `@agent`, `@agent:model` and `@model` mentions ([cap-agents.md](cap-agents.md)).
It is deliberately generic and has no dependency on anything else in the application: pass a query and a
list of strings, get scores or the best index.

Design rules it follows (project rules, see `AGENTS.md`):

- Static ceilings, no auto tuning (section 6).
- Deterministic: the same inputs always give the same order, ties included.
- Loud limits: nothing is silently truncated beyond the ceilings listed in section 6.
- No dependency: it is a pure function over `std::string`.

## 2. API

```cpp
#include "fuzzy.h"
using detz::FuzzyMatcher;

double s = FuzzyMatcher::Score("opus", "claude-opus-4-5");               // 0..1
auto ranked = FuzzyMatcher::Rank("opus", {"claude-opus-4-5", "opencode"}); // vector<Ranked{index, score}>
long best  = FuzzyMatcher::Best("opus", candidates, /*floor=*/0.3, &score); // index or -1
```

| Function | Returns |
|---|---|
| `Score(query, text)` | 0 when nothing relates; 1.0 only for a case-insensitive exact match; otherwise the maximum of the four signals below, never above 0.999 |
| `Subsequence`, `Typo`, `TokenTypo`, `Trigram` | the individual signals, public for tests and tuning discussions |
| `Rank(query, texts)` | every candidate with score above 0, best first |
| `Best(query, texts, floor, &score)` | index of the top candidate, or -1 when there is none or its score is not strictly above `floor` |

Empty query or empty text scores 0. Comparison is case-insensitive for ASCII only (section 7).

## 3. The four signals

`Score` = max( `Subsequence`, `Typo`, 0.95 x `TokenTypo`, 0.9 x `Trigram` ), then capped at 0.999 unless the
folded strings are equal (then exactly 1.0).

### 3.1 Subsequence (letters in order, boundary aware)

The query bytes must appear in the text in order, case-insensitively, not necessarily next to each other.
The best alignment is found by dynamic programming and scored with the constants of fzf's
`FuzzyMatchV2` (`research/fuzzy-matching-selectors/github-fzf-algo-go.md`), using the two-matrix recurrence
of fzy (`github-fzy-match-c.md`):

- `D[i][j]` = best score of query[0..i] where query[i] is matched exactly at text[j].
- `M[i][j]` = best score of query[0..i] using text[0..j], matched or skipped.
- Matching text[j] for query[i]:
  - `i == 0`: `kScoreMatch + bonus[j] * kFirstMultiplier + kGap * j` (the skipped leading bytes cost `kGap` each).
  - `i > 0`: `max( M[i-1][j-1] + kScoreMatch + bonus[j],  D[i-1][j-1] + kScoreMatch + max(bonus[j], kBonusConsecutive) )`.
    The second branch is "continuing a run"; it always earns at least `kBonusConsecutive`.
- A skipped byte costs `kGap` (-1), at the start, inside and at the end.
- The raw result is `M[n-1][m-1]`; it is normalized by the perfect score `n * 16 + 10 * 2 + (n-1) * 4`
  (every byte matched, the first at a whitespace boundary, all consecutive) and clamped to [0, 1].

| Constant | Value | Applies when |
|---|---|---|
| `kScoreMatch` | 16 | each matched byte |
| `kGap` | -1 | each skipped byte |
| `kBonusWhite` | 10 | match at the start of the text or after whitespace |
| `kBonusBoundary` | 8 | match after a byte that is not a letter or digit (`-`, `_`, `/`, `:`, `.`) |
| `kBonusCamel` | 7 | lower to Upper, or a non-digit to a digit |
| `kBonusConsecutive` | 4 | minimum bonus for continuing a run |
| `kFirstMultiplier` | 2 | first query byte's bonus is doubled |

Consequences you can rely on: a query matches best at the start of a word; `fb` finds `FooBar` (camel
humps) better than `fxxxxbxx`; shorter texts lose fewer points to skipped bytes. A query longer than the
text cannot be a subsequence and scores 0 here.

### 3.2 Typo (restricted Damerau-Levenshtein)

`1 - d / max(len(query), len(text))` where `d` is the optimal-string-alignment distance: insert, delete,
substitute, or swap two adjacent bytes, each costing 1. It catches typos and transpositions
(`snonet` for `sonnet`) that are not subsequences. It is low for a short query against a long text, which
is what the next signal is for.

### 3.3 TokenTypo (typo against each word)

The text is split into runs of letters and digits (`claude-opus-4-5` gives `claude`, `opus`, `4`, `5`);
the result is the best `Typo` of the query against any one token, scaled by 0.95. It finds a misspelled
word inside a long identifier (`opuss` in `claude-opus-4-5`) and gives an exact word match (`opus`, `free`,
`pro`) a score of 0.95.

### 3.4 Trigram (pg_trgm)

Each alphanumeric word gets two leading spaces and one trailing space; the set of three-byte windows is
compared by Jaccard (shared / total distinct), the way PostgreSQL's `pg_trgm` does
(`postgresql-pg-trgm.md`: `word` against `words` is 4/7 = 0.571). Scaled by 0.9. It helps with scrambled or
partial words and is the weakest signal.

## 4. Ranking and ties

`Rank` keeps candidates with score above 0 and sorts by:

1. score, descending (differences under 1e-9 count as equal);
2. shorter text first (byte length), the default tie rule of fzf (`github-fzf-man-page.md`);
3. earlier position in the input list.

So two equal matches never produce an error or a coin flip: `free` against `zen-free` and `devplace-free`
(both 0.95) always picks `zen-free`.

## 5. Worked examples

Measured with the code (`Subsequence`, `Typo`, `TokenTypo` before the 0.95 scale, `Trigram` before the 0.9
scale, then the final `Score`):

| query | text | subseq | typo | token | trigram | score |
|---|---|---|---|---|---|---|
| `opus` | `claude-opus-4-5` | 0.84 | 0.27 | 1.00 | 0.31 | 0.95 |
| `opus` | `opencode` | 0.00 | 0.25 | 0.25 | 0.17 | 0.25 |
| `son` | `claude-sonnet-4-5` | 0.76 | 0.18 | 0.50 | 0.16 | 0.76 |
| `sonet` (dropped letter) | `claude-sonnet-4-5` | 0.83 | 0.29 | 0.83 | 0.26 | 0.83 |
| `snonet` (swap) | `claude-sonnet-4-5` | 0.00 | 0.29 | 0.83 | 0.14 | 0.79 |
| `opuss` (extra letter) | `claude-opus-4-5` | 0.00 | 0.27 | 0.80 | 0.22 | 0.76 |
| `g4m` (word starts) | `gpt-4o-mini` | 1.00 | 0.27 | 0.33 | 0.07 | 1.00 |
| `gpt4` | `gpt-4o` | 1.00 | 0.67 | 0.75 | 0.33 | 1.00 |
| `gpt4` | `gpt-4o-mini` | 0.97 | 0.36 | 0.75 | 0.21 | 0.97 |
| `free` | `zen-free` | 0.92 | 0.50 | 1.00 | 0.56 | 0.95 |
| `pro` | `gemini-2.5-pro` | 0.80 | 0.21 | 1.00 | 0.27 | 0.95 |
| `fb` | `FooBar` | 0.98 | 0.33 | 0.33 | 0.11 | 0.98 |
| `fb` | `fxxxxbxx` | 0.82 | 0.25 | 0.25 | 0.10 | 0.82 |
| `clde` | `claude` | 0.94 | 0.67 | 0.67 | 0.33 | 0.94 |
| `word` | `words` | 0.99 | 0.80 | 0.80 | 0.57 | 0.99 |
| `zzqq` | `zen-free` | 0.00 | 0.12 | 0.25 | 0.08 | 0.24 |

(`g4m` and `gpt4` show 1.00 because the subsequence is perfect after rounding; the function itself returns
at most 0.999 unless the strings are equal.) Reading the table: `opus` finds `claude-opus-4-5` mainly through
the exact word (token signal) and not at all through `opencode`; a swap is rescued by the token signal; a
nonsense query tops out around 0.25.

## 6. Ceilings

| Constant | Value | Effect |
|---|---|---|
| `kFuzzyQueryMax` | 64 bytes | longer queries are cut |
| `kFuzzyTextMax` | 256 bytes | longer candidates are cut |
| `kFuzzyCandidateMax` | 4096 | candidates beyond this are not ranked |
| `kTrigramScale` | 0.9 | |
| `kTokenScale` | 0.95 | |

Cost per score is at most 64 x 256 cells for the subsequence DP and for the edit distance, so ranking a
full catalog stays in the low milliseconds.

## 7. Known limits

- ASCII case folding only; bytes of 0x80 and above are treated as lowercase letters and compared as they
  are. Accented text is not normalized (fzf can).
- The constants are fzf's, tuned for file paths, not for model identifiers. They were not tuned on a real
  corpus of model ids; the examples above are the evidence. No claim of optimality.
- A subsequence can match scattered letters inside a long identifier; boundary bonuses make this rare, not
  impossible.
- The caller decides whether a score is good enough: `Best(..., floor)` takes the floor. The CAP catalog
  uses 0.3 (`CAP_FUZZY_FLOOR`); `zzqq` against everything scores at most 0.24 and is rejected.

## 8. Reusing it

Anything that picks from a list of names can call it: pass the user's text and the list, take `Best`
with a floor, show the chosen candidate to the user when it differs from what they typed (the catalog
prints `'tia:zne' resolved to tai:zen-free (fuzzy 0.67)`). Do not use it for security decisions or for
free text search; it is built for short identifiers.