tai/research/sqlite-rag-hybrid-search/conclusions/standing-retrieval-eval.md

Conclusion: Standing retrieval eval from goldens, rank metrics, and reads

Metadata

Field	Value
Institute	Molodetz Institute — Research & Retrieval Division
Subject	sqlite-rag-hybrid-search
Document type	Conclusion
Tier	N/A
Date decided	2026-10-18
Decided by	retoor, advised by tai (agent, Muse Spark)
Status	Active
Confidence	High

Question / problem statement

The original conclusion in this subject shipped BM25-only retrieval with an explicit open risk: the lexical-match tradeoff was never measured against real retrieval-quality metrics. The prompt-scaling research then made that risk load-bearing, because spilling giant prompts to records only works if retrieval finds the right slices. The question was how to measure retrieval quality with no labeled data, no new dependencies, and a corpus shaped like command output and logs.

Conclusion

Build a standing retrieval eval with three layers, in this order: (1) deterministic rank metrics (Hit@K, MRR, nDCG@K, Context Precision) over fts_search against golden query-plus-expected-id pairs generalized from the fingerprint method in tests/rag.py; (2) implicit pairwise preferences mined from search-then-read sequences using the existing reads counter, interpreted relatively per Joachims, never as absolute labels; (3) LLM-judged context relevance only for the ambiguous remainder the first two layers cannot grade. Close the logging gap first: run_search must log the query with returned ids, because results alone cannot be mined.

Supporting sources

• arxiv-2104-08663-beir: rank-aware metric rationale; nDCG covers binary and graded relevance, bare accuracy does not.
• ragas-context-precision-docs: the exact mean-precision@k ranking formula to reimplement natively.
• arxiv-2309-15217-ragas: reference-free context metrics when no labels exist; retriever-side only for tai's purposes.
• arxiv-2408-08067-ragchecker: claim recall versus context precision split, which tells tuning apart (missing chunk versus misranked chunk).
• deepeval-synthesizer-goldens: goldens-from-own-docs is production-standard; tai's chunker plus FTS index already supply the context-construction half.
• joachims-clickthrough-implicit-feedback: clicks informative but biased; relative preferences reasonably accurate, which is exactly how to read search-then-read sequences.
• radlinski-joachims-query-chains (Tier 2): preferences hold across reformulation chains, covering tai's retry-shaped search behavior.

Alternatives considered and rejected

• Adopting RAGAS/DeepEval as dependencies: rejected, dependency-hostile (langchain, chroma, API keys); reimplement the two retriever formulas natively instead.
• Human-labeled eval set: rejected as the starting point (expensive, slow); kept as the calibration backstop the log-mined signal gets spot-checked against.
• Judging retrieval from answer quality end-to-end: rejected, conflates retriever and generator; RAGAS/RAGChecker both insist on separating the two.
• Skipping measurement and tuning blind: rejected, this was the open risk being closed; no tuning knob moves without a before/after number after this conclusion.

Risks / open questions

• Synthesized queries can leak source phrasing and flatter lexical scores; pair every paraphrased golden with a fingerprint-exact one and report both.
• The reads signal reflects agent behavior, not human relevance; useful for reranking comparisons, never for certification.
• Revisit if: golden scores and real-task success diverge (the eval measures the wrong thing), or graded LLM judgments prove too noisy on free-tier backends.

Action taken

Built the same day the conclusion was recorded: tai.RetrievalEval (Hit@K, MRR, binary nDCG@K, mean precision@k; deterministic golden generator; implicit search-then-read miner), search-query and record-read event logging in tai.Tools.run_search / run_record_read, and the read-only tai.py --eval-retrieval CLI. Measured on the 5-record unit fixture (20 goldens): baseline Hit@5 0.500 with prefix and noisy at 0.000, post-tuning 1.000 in all four categories. Measured on the real vault (50 records, 200 goldens): Hit@5 0.890 overall (exact 0.92, noisy 0.94, phrase 0.96, prefix 0.74), MRR 0.708. The exact category needed rarest-term golden selection (first-token goldens scored 0.22, measuring golden noise, not the engine). Implicit mining is live but has no history yet (0 searches at ship time).

Changelog

Date	Change
2026-10-18	Initial conclusion recorded.
2026-10-18	Implemented as specified; baseline and post-tuning numbers recorded above.