Files

T

retoor af080fe68e Strict goal verification, approval hardening, retrieval eval, work limits, sandbox caps (1.38.0)

- Approval: re-prompt unknown answers (max 3), shell-only affirmative
  guidance recovery; destructive actions stay strict
- GoalVerifier subagent with strict JSON verdicts, bounded follow-up
  rounds, nonzero exit plus open turn on unverified one-shots
- Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with
  title weighting, query/read event logging, --eval-retrieval CLI
- Supervisor caps (8 parallel, 64 records) and scheduler hardening
  (4 fires/tick, per-schedule lease, stated misfire policy)
- Sandbox box resource trio: 2g/2.0/256 at create, best-effort
  update for existing boxes, rm -f recovery pointer
- Research archive: 12 subjects, 667 sources, 27 conclusions;
  suites 979 + 16 green

2026-10-18 12:13:24 +02:00

2.6 KiB

Raw Blame History

SWE-bench Evaluation Guide

Metadata

Field	Value
Institute	Molodetz Institute — Research & Retrieval Division
Subject	agent-goal-verification
Document type	Source Dossier
Tier	1
Source title	SWE-bench Evaluation Guide
Source URL	https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/docs/guides/evaluation.md
Source publisher / author	SWE-bench (Princeton NLP)
Source type	repository
Source date (published)	Unknown — not stated (living project docs)
Date researched	2026-10-18
Researched by	tai (agent), model: Muse Spark, on behalf of retoor
Research method	both
Confidence	High
Status	Active

Summary

The reference implementation of test-based agent verification: candidate patches are applied to real repositories and the repositories' own tests decide whether the issue is resolved, all inside Docker for reproducibility. No LLM judges the outcome; execution is the judge.

Key findings

"SWE-bench evaluates models by applying their generated patches to real-world repositories and running the repository's tests to verify if the issue is resolved. The evaluation is performed in a containerized Docker environment to ensure consistent results across different platforms."
The harness entry point is swebench.harness.run_evaluation with worker parallelism, instance selection, Docker cache levels, and result caching by run id.
Predictions are JSONL rows of instance id, model name, and patch text; cloud execution via Modal or sb-cli is supported.

Pros / strengths

Ground truth by construction: the verdict is a test run, immune to judge bias, verbosity effects, and sycophancy.
Reproducible and automatable at scale, which is why it became the standard coding-agent benchmark.

Cons / limitations / caveats

Only works where executable checks exist (tests, builds, scripts). Most tai one-shot goals ("upgrade python", "research X") have no pre-written test suite, so pure SWE-bench-style verification is unavailable.
Heavyweight per check (containers, repos, dependency installs); overkill for judging a chat-length turn.

Relevance to tai

Sets the gold standard tai's verifier approximates: prefer checking machine state with tools over grading prose. tai's verifier prompt already instructs read-only tool inspection first and transcript judgment second; this source says that ordering is right and that any goal with an executable check should get one rather than relying on the judge alone.

Changelog

Date	Change
2026-10-18	Initial dossier created.

2.6 KiB Raw Blame History