Files

T

retoor af080fe68e Strict goal verification, approval hardening, retrieval eval, work limits, sandbox caps (1.38.0)

- Approval: re-prompt unknown answers (max 3), shell-only affirmative
  guidance recovery; destructive actions stay strict
- GoalVerifier subagent with strict JSON verdicts, bounded follow-up
  rounds, nonzero exit plus open turn on unverified one-shots
- Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with
  title weighting, query/read event logging, --eval-retrieval CLI
- Supervisor caps (8 parallel, 64 records) and scheduler hardening
  (4 fires/tick, per-schedule lease, stated misfire policy)
- Sandbox box resource trio: 2g/2.0/256 at create, best-effort
  update for existing boxes, rm -f recovery pointer
- Research archive: 12 subjects, 667 sources, 27 conclusions;
  suites 979 + 16 green

2026-10-18 12:13:24 +02:00

2.4 KiB

Raw Blame History

Context Precision - Ragas

Metadata

Field	Value
Institute	Molodetz Institute — Research & Retrieval Division
Subject	sqlite-rag-hybrid-search
Document type	Source Dossier
Tier	1
Source title	Context Precision - Ragas
Source URL	https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/
Source publisher / author	Ragas (Exploding Gradients)
Source type	vendor docs
Source date (published)	Unknown — not stated (living documentation page)
Date researched	2026-10-18
Researched by	tai (agent), model: Muse Spark, on behalf of retoor
Research method	both
Confidence	High
Status	Active

Summary

Ragas's Context Precision metric scores whether relevant retrieved chunks rank above irrelevant ones, formalized as relevance-weighted mean precision@k. A sibling metric, Context Utilization, covers the no-reference-answer case by comparing contexts against the generated response instead.

Key findings

"Context Precision is a metric that evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones for a given query in the retrieved context."
"It is calculated as the mean of the precision@k for each chunk in the context", with relevance indicators per rank and the standard precision@k ratio.
"The ContextUtilization metric evaluates whether retrieved contexts are useful by comparing each context against the generated response. Use this when you don't have a reference answer but have the response that was generated."

Pros / strengths

Gives tai the exact rank-aware formula to reimplement natively (mean precision@k over relevance judgments), with the reference-free variant spelled out.
Worked examples show the metric responding to rank order, not just set membership, which is what a tuning loop needs.

Cons / limitations / caveats

Relevance judgments still need a judge (LLM or human) per chunk; the formula is free, the labels are not.
Package docs assume the Ragas runtime; tai reimplements the arithmetic, not the dependency.

Relevance to tai

Supplies the rank-aware half of the standing eval's metric set: once goldens say which chunk answers a query, Context Precision (mean precision@k) measures ranking quality deterministically. Pairs with the fingerprint goldens, which supply the relevance labels for free.

Changelog

Date	Change
2026-10-18	Initial dossier created.

2.4 KiB Raw Blame History