Files

T

retoor af080fe68e Strict goal verification, approval hardening, retrieval eval, work limits, sandbox caps (1.38.0)

- Approval: re-prompt unknown answers (max 3), shell-only affirmative
  guidance recovery; destructive actions stay strict
- GoalVerifier subagent with strict JSON verdicts, bounded follow-up
  rounds, nonzero exit plus open turn on unverified one-shots
- Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with
  title weighting, query/read event logging, --eval-retrieval CLI
- Supervisor caps (8 parallel, 64 records) and scheduler hardening
  (4 fires/tick, per-schedule lease, stated misfire policy)
- Sandbox box resource trio: 2g/2.0/256 at create, best-effort
  update for existing boxes, rm -f recovery pointer
- Research archive: 12 subjects, 667 sources, 27 conclusions;
  suites 979 + 16 green

2026-10-18 12:13:24 +02:00

2.7 KiB

Raw Blame History

Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search

Metadata

Field	Value
Institute	Molodetz Institute — Research & Retrieval Division
Subject	sqlite-rag-hybrid-search
Document type	Source Dossier
Tier	1
Source title	Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search
Source URL	https://www.cs.cornell.edu/~tj/publications/joachims_etal_07a.pdf
Source publisher / author	Thorsten Joachims et al. (TOIS)
Source type	paper
Source date (published)	2007-01-01
Date researched	2026-10-18
Researched by	tai (agent), model: Muse Spark, on behalf of retoor
Research method	direct page fetch
Confidence	High
Status	Active

Summary

The founding result on clickthrough data as relevance signal: against eyetracking and manual judgments, clicks are informative but biased, unusable as absolute relevance judgments, while relative preferences derived from clicks (this result over that one for the same query) are reasonably accurate on average, including across query reformulation chains.

Key findings

"We conclude that clicks are informative but biased."
"While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average."
Relative preferences hold "not only between results from an individual query, but across multiple sets of results within chains of query reformulations."
Method: eyetracking of the users' decision process compared against manual relevance judgments.

Pros / strengths

Gives tai's already-collected reads counter plus search-then-read sequences a precise interpretation: pairwise preferences, not absolute labels.
TOIS publication with an experimental (not just log-correlational) design; the finding has held for nearly two decades of learning-to-rank practice.

Cons / limitations / caveats

Web-search clicks with SERP position bias; tai's analog (agent reads a record after searching) has different bias structure (agent, not human, doing the clicking), unstudied here.
Absolute judgments are explicitly unsupported, so implicit data can rerank and compare but never certify.

Relevance to tai

Authorizes the cheapest eval signal available: log search queries with returned ids (a gap today: only results are logged), join to subsequent record reads, and derive pairwise preferences for tuning. No new data collection beyond one log line; the reads counter already exists.

Changelog

Date	Change
2026-10-18	Initial dossier created.

2.7 KiB Raw Blame History