Files

T

retoor af080fe68e Strict goal verification, approval hardening, retrieval eval, work limits, sandbox caps (1.38.0)

- Approval: re-prompt unknown answers (max 3), shell-only affirmative
  guidance recovery; destructive actions stay strict
- GoalVerifier subagent with strict JSON verdicts, bounded follow-up
  rounds, nonzero exit plus open turn on unverified one-shots
- Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with
  title weighting, query/read event logging, --eval-retrieval CLI
- Supervisor caps (8 parallel, 64 records) and scheduler hardening
  (4 fires/tick, per-schedule lease, stated misfire policy)
- Sandbox box resource trio: 2g/2.0/256 at create, best-effort
  update for existing boxes, rm -f recovery pointer
- Research archive: 12 subjects, 667 sources, 27 conclusions;
  suites 979 + 16 green

2026-10-18 12:13:24 +02:00

2.6 KiB

Raw Blame History

SQLite FTS5 Extension

Metadata

Field	Value
Institute	Molodetz Institute — Research & Retrieval Division
Subject	sqlite-rag-hybrid-search
Document type	Source Dossier
Tier	1
Source title	SQLite FTS5 Extension
Source URL	https://www.sqlite.org/fts5.html
Source publisher / author	SQLite (Hipp Wyrick and Co.)
Source type	vendor docs
Source date (published)	Unknown — not stated (living documentation page)
Date researched	2026-10-18
Researched by	tai (agent), model: Muse Spark, on behalf of retoor
Research method	direct page fetch
Confidence	High
Status	Active

Summary

The authoritative FTS5 reference, read for the BM25 tuning surface: k1 and b are hard-coded (1.2, 0.75), scores are negated so ascending sort puts best first, and per-column weights passed as extra bm25() arguments are the only ranking knob. It also documents prefix queries, NEAR, and boolean operators tai does not currently use.

Key findings

"k1 and b are both constants, hard-coded at 1.2 and 0.75 respectively."
"By passing extra real value arguments to the bm25() SQL function, each column of the table may be assigned a different weight", with phrase frequency computed as the weight-weighted sum across columns; missing trailing weights default to 1.0.
FTS5 "multiplies the result by -1 before returning it, ensuring that better matches are assigned numerically lower scores", so plain ORDER BY bm25() ascending is correct (tai does this).
Query syntax supports prefix queries, initial-token queries, NEAR, column filters, and boolean operators; tai uses only quoted-phrase AND across the title and body columns.

Pros / strengths

Primary specification of the exact ranking function tai runs; settles what is tunable (column weights, query shape) and what is not (k1, b).
Negation note confirms tai's sort direction is right and needs no change.

Cons / limitations / caveats

Reference documentation, not tuning guidance; it gives no recommended weights or query-shape advice for small corpora.
No discussion of evaluation; measurement guidance comes from the IR sources in this subject.

Relevance to tai

Defines the entire dependency-free tuning backlog: title-over-body column weights first, then OR-fallback and prefix queries for the strict-AND recall gap. Everything else (k1/b, stemming beyond porter) is fixed by the engine, which bounds the roadmap and kills any proposal to "tune BM25 parameters" in FTS5.

Changelog

Date	Change
2026-10-18	Initial dossier created.

2.6 KiB Raw Blame History