retoor
af080fe68e
- Approval: re-prompt unknown answers (max 3), shell-only affirmative guidance recovery; destructive actions stay strict - GoalVerifier subagent with strict JSON verdicts, bounded follow-up rounds, nonzero exit plus open turn on unverified one-shots - Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with title weighting, query/read event logging, --eval-retrieval CLI - Supervisor caps (8 parallel, 64 records) and scheduler hardening (4 fires/tick, per-schedule lease, stated misfire policy) - Sandbox box resource trio: 2g/2.0/256 at create, best-effort update for existing boxes, rm -f recovery pointer - Research archive: 12 subjects, 667 sources, 27 conclusions; suites 979 + 16 green
51 lines
2.4 KiB
Markdown
51 lines
2.4 KiB
Markdown
# Context Precision - Ragas
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Institute | Molodetz Institute — Research & Retrieval Division |
|
|
| Subject | sqlite-rag-hybrid-search |
|
|
| Document type | Source Dossier |
|
|
| Tier | 1 |
|
|
| Source title | Context Precision - Ragas |
|
|
| Source URL | https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/ |
|
|
| Source publisher / author | Ragas (Exploding Gradients) |
|
|
| Source type | vendor docs |
|
|
| Source date (published) | Unknown — not stated (living documentation page) |
|
|
| Date researched | 2026-10-18 |
|
|
| Researched by | tai (agent), model: Muse Spark, on behalf of retoor |
|
|
| Research method | both |
|
|
| Confidence | High |
|
|
| Status | Active |
|
|
|
|
## Summary
|
|
|
|
Ragas's Context Precision metric scores whether relevant retrieved chunks rank above irrelevant ones, formalized as relevance-weighted mean precision@k. A sibling metric, Context Utilization, covers the no-reference-answer case by comparing contexts against the generated response instead.
|
|
|
|
## Key findings
|
|
|
|
- "Context Precision is a metric that evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones for a given query in the retrieved context."
|
|
- "It is calculated as the mean of the precision@k for each chunk in the context", with relevance indicators per rank and the standard precision@k ratio.
|
|
- "The ContextUtilization metric evaluates whether retrieved contexts are useful by comparing each context against the generated response. Use this when you don't have a reference answer but have the response that was generated."
|
|
|
|
## Pros / strengths
|
|
|
|
- Gives tai the exact rank-aware formula to reimplement natively (mean precision@k over relevance judgments), with the reference-free variant spelled out.
|
|
- Worked examples show the metric responding to rank order, not just set membership, which is what a tuning loop needs.
|
|
|
|
## Cons / limitations / caveats
|
|
|
|
- Relevance judgments still need a judge (LLM or human) per chunk; the formula is free, the labels are not.
|
|
- Package docs assume the Ragas runtime; tai reimplements the arithmetic, not the dependency.
|
|
|
|
## Relevance to tai
|
|
|
|
Supplies the rank-aware half of the standing eval's metric set: once goldens say which chunk answers a query, Context Precision (mean precision@k) measures ranking quality deterministically. Pairs with the fingerprint goldens, which supply the relevance labels for free.
|
|
|
|
## Changelog
|
|
|
|
| Date | Change |
|
|
|---|---|
|
|
| 2026-10-18 | Initial dossier created. |
|