Files

T

retoor af080fe68e Strict goal verification, approval hardening, retrieval eval, work limits, sandbox caps (1.38.0)

- Approval: re-prompt unknown answers (max 3), shell-only affirmative
  guidance recovery; destructive actions stay strict
- GoalVerifier subagent with strict JSON verdicts, bounded follow-up
  rounds, nonzero exit plus open turn on unverified one-shots
- Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with
  title weighting, query/read event logging, --eval-retrieval CLI
- Supervisor caps (8 parallel, 64 records) and scheduler hardening
  (4 fires/tick, per-schedule lease, stated misfire policy)
- Sandbox box resource trio: 2g/2.0/256 at create, best-effort
  update for existing boxes, rm -f recovery pointer
- Research archive: 12 subjects, 667 sources, 27 conclusions;
  suites 979 + 16 green

2026-10-18 12:13:24 +02:00

2.8 KiB

Raw Blame History

Structured-Output Reliability Eval

Metadata

Field	Value
Institute	Molodetz Institute — Research & Retrieval Division
Subject	agent-goal-verification
Document type	Source Dossier
Tier	1
Source title	Structured-Output Reliability Eval
Source URL	https://github.com/tkarim45/structured-output-eval
Source publisher / author	tkarim45
Source type	repository
Source date (published)	Unknown — not stated
Date researched	2026-10-18
Researched by	tai (agent), model: Muse Spark, on behalf of retoor
Research method	direct page fetch
Confidence	Medium
Status	Active

Summary

A runnable benchmark comparing four JSON strategies (plain prompt, schema-in-prompt, native constrained decoding, strict tool) on valid-JSON rate, schema adherence, and field accuracy, including adversarial inputs. On Claude Haiku 4.5, plain prompting hit only 25% schema adherence (33% adversarial) while schema-in-prompt hit 100%, tying constrained decoding on this task set.

Key findings

Measured table (Claude Haiku 4.5 on Bedrock, 8 tasks): plain schema adherence 0.250 (adversarial 0.333); schema_in_prompt, native, and strict_tool all 1.000 on every column.
"Malformed JSON was never the failure mode. Claude Haiku returned valid JSON 100% of the time on every strategy, plain included... The real failure is schema adherence."
"Just putting the schema in the prompt fixed it, no constrained decoding required... constrained decoding's real value is the guarantee at scale and on harder schemas, not a measured lift over schema-in-prompt on this set."
"Valid JSON is not equal to correct" (Valid ≠ correct): braces parse, shape wrong.

Pros / strengths

Real measured numbers on a real model with a reproducible harness, not vendor marketing; distinguishes the two failure modes (syntax vs schema) that most advice smears together.
Directly supports tai's position: for a small fixed schema like the verdict object, schema-in-prompt plus validation is empirically sufficient.

Cons / limitations / caveats

One model family (Claude Haiku), 8 tasks, one operator; tai's verifier runs on weaker free-tier models where plain-prompt adherence is likely worse than 25%.
Community repo, no peer review; the mock path ships deliberately different numbers as a teaching fixture, which a careless reader could quote instead of the real table.

Relevance to tai

Quantitative backing for the verdict contract: paste the schema into the verifier prompt (tai does), validate strictly on receipt (tai does), and treat constrained decoding as a future guarantee rather than a current requirement. Also warns that tai's weaker verifier backends need the strict-parse-plus-retry path more than Claude-class models do.

Changelog

Date	Change
2026-10-18	Initial dossier created.

2.8 KiB Raw Blame History