# SQLite FTS5 Extension

## Metadata

| Field | Value |
|---|---|
| Institute | Molodetz Institute — Research & Retrieval Division |
| Subject | sqlite-rag-hybrid-search |
| Document type | Source Dossier |
| Tier | 1 |
| Source title | SQLite FTS5 Extension |
| Source URL | https://www.sqlite.org/fts5.html |
| Source publisher / author | SQLite (Hipp Wyrick and Co.) |
| Source type | vendor docs |
| Source date (published) | Unknown — not stated (living documentation page) |
| Date researched | 2026-10-18 |
| Researched by | tai (agent), model: Muse Spark, on behalf of retoor |
| Research method | direct page fetch |
| Confidence | High |
| Status | Active |

## Summary

The authoritative FTS5 reference, read for the BM25 tuning surface: k1 and b are hard-coded (1.2, 0.75), scores are negated so ascending sort puts best first, and per-column weights passed as extra bm25() arguments are the only ranking knob. It also documents prefix queries, NEAR, and boolean operators tai does not currently use.

## Key findings

- "k1 and b are both constants, hard-coded at 1.2 and 0.75 respectively."
- "By passing extra real value arguments to the bm25() SQL function, each column of the table may be assigned a different weight", with phrase frequency computed as the weight-weighted sum across columns; missing trailing weights default to 1.0.
- FTS5 "multiplies the result by -1 before returning it, ensuring that better matches are assigned numerically lower scores", so plain ORDER BY bm25() ascending is correct (tai does this).
- Query syntax supports prefix queries, initial-token queries, NEAR, column filters, and boolean operators; tai uses only quoted-phrase AND across the title and body columns.

## Pros / strengths

- Primary specification of the exact ranking function tai runs; settles what is tunable (column weights, query shape) and what is not (k1, b).
- Negation note confirms tai's sort direction is right and needs no change.

## Cons / limitations / caveats

- Reference documentation, not tuning guidance; it gives no recommended weights or query-shape advice for small corpora.
- No discussion of evaluation; measurement guidance comes from the IR sources in this subject.

## Relevance to tai

Defines the entire dependency-free tuning backlog: title-over-body column weights first, then OR-fallback and prefix queries for the strict-AND recall gap. Everything else (k1/b, stemming beyond porter) is fixed by the engine, which bounds the roadmap and kills any proposal to "tune BM25 parameters" in FTS5.

## Changelog

| Date | Change |
|---|---|
| 2026-10-18 | Initial dossier created. |