retoor
af080fe68e
- Approval: re-prompt unknown answers (max 3), shell-only affirmative guidance recovery; destructive actions stay strict - GoalVerifier subagent with strict JSON verdicts, bounded follow-up rounds, nonzero exit plus open turn on unverified one-shots - Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with title weighting, query/read event logging, --eval-retrieval CLI - Supervisor caps (8 parallel, 64 records) and scheduler hardening (4 fires/tick, per-schedule lease, stated misfire policy) - Sandbox box resource trio: 2g/2.0/256 at create, best-effort update for existing boxes, rm -f recovery pointer - Research archive: 12 subjects, 667 sources, 27 conclusions; suites 979 + 16 green
51 lines
2.6 KiB
Markdown
51 lines
2.6 KiB
Markdown
# SWE-bench Evaluation Guide
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Institute | Molodetz Institute — Research & Retrieval Division |
|
|
| Subject | agent-goal-verification |
|
|
| Document type | Source Dossier |
|
|
| Tier | 1 |
|
|
| Source title | SWE-bench Evaluation Guide |
|
|
| Source URL | https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/docs/guides/evaluation.md |
|
|
| Source publisher / author | SWE-bench (Princeton NLP) |
|
|
| Source type | repository |
|
|
| Source date (published) | Unknown — not stated (living project docs) |
|
|
| Date researched | 2026-10-18 |
|
|
| Researched by | tai (agent), model: Muse Spark, on behalf of retoor |
|
|
| Research method | both |
|
|
| Confidence | High |
|
|
| Status | Active |
|
|
|
|
## Summary
|
|
|
|
The reference implementation of test-based agent verification: candidate patches are applied to real repositories and the repositories' own tests decide whether the issue is resolved, all inside Docker for reproducibility. No LLM judges the outcome; execution is the judge.
|
|
|
|
## Key findings
|
|
|
|
- "SWE-bench evaluates models by applying their generated patches to real-world repositories and running the repository's tests to verify if the issue is resolved. The evaluation is performed in a containerized Docker environment to ensure consistent results across different platforms."
|
|
- The harness entry point is `swebench.harness.run_evaluation` with worker parallelism, instance selection, Docker cache levels, and result caching by run id.
|
|
- Predictions are JSONL rows of instance id, model name, and patch text; cloud execution via Modal or sb-cli is supported.
|
|
|
|
## Pros / strengths
|
|
|
|
- Ground truth by construction: the verdict is a test run, immune to judge bias, verbosity effects, and sycophancy.
|
|
- Reproducible and automatable at scale, which is why it became the standard coding-agent benchmark.
|
|
|
|
## Cons / limitations / caveats
|
|
|
|
- Only works where executable checks exist (tests, builds, scripts). Most tai one-shot goals ("upgrade python", "research X") have no pre-written test suite, so pure SWE-bench-style verification is unavailable.
|
|
- Heavyweight per check (containers, repos, dependency installs); overkill for judging a chat-length turn.
|
|
|
|
## Relevance to tai
|
|
|
|
Sets the gold standard tai's verifier approximates: prefer checking machine state with tools over grading prose. tai's verifier prompt already instructs read-only tool inspection first and transcript judgment second; this source says that ordering is right and that any goal with an executable check should get one rather than relying on the judge alone.
|
|
|
|
## Changelog
|
|
|
|
| Date | Change |
|
|
|---|---|
|
|
| 2026-10-18 | Initial dossier created. |
|