retoor
af080fe68e
- Approval: re-prompt unknown answers (max 3), shell-only affirmative guidance recovery; destructive actions stay strict - GoalVerifier subagent with strict JSON verdicts, bounded follow-up rounds, nonzero exit plus open turn on unverified one-shots - Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with title weighting, query/read event logging, --eval-retrieval CLI - Supervisor caps (8 parallel, 64 records) and scheduler hardening (4 fires/tick, per-schedule lease, stated misfire policy) - Sandbox box resource trio: 2g/2.0/256 at create, best-effort update for existing boxes, rm -f recovery pointer - Research archive: 12 subjects, 667 sources, 27 conclusions; suites 979 + 16 green
52 lines
2.8 KiB
Markdown
52 lines
2.8 KiB
Markdown
# Structured-Output Reliability Eval
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Institute | Molodetz Institute — Research & Retrieval Division |
|
|
| Subject | agent-goal-verification |
|
|
| Document type | Source Dossier |
|
|
| Tier | 1 |
|
|
| Source title | Structured-Output Reliability Eval |
|
|
| Source URL | https://github.com/tkarim45/structured-output-eval |
|
|
| Source publisher / author | tkarim45 |
|
|
| Source type | repository |
|
|
| Source date (published) | Unknown — not stated |
|
|
| Date researched | 2026-10-18 |
|
|
| Researched by | tai (agent), model: Muse Spark, on behalf of retoor |
|
|
| Research method | direct page fetch |
|
|
| Confidence | Medium |
|
|
| Status | Active |
|
|
|
|
## Summary
|
|
|
|
A runnable benchmark comparing four JSON strategies (plain prompt, schema-in-prompt, native constrained decoding, strict tool) on valid-JSON rate, schema adherence, and field accuracy, including adversarial inputs. On Claude Haiku 4.5, plain prompting hit only 25% schema adherence (33% adversarial) while schema-in-prompt hit 100%, tying constrained decoding on this task set.
|
|
|
|
## Key findings
|
|
|
|
- Measured table (Claude Haiku 4.5 on Bedrock, 8 tasks): plain schema adherence 0.250 (adversarial 0.333); schema_in_prompt, native, and strict_tool all 1.000 on every column.
|
|
- "Malformed JSON was never the failure mode. Claude Haiku returned valid JSON 100% of the time on every strategy, plain included... The real failure is schema adherence."
|
|
- "Just putting the schema in the prompt fixed it, no constrained decoding required... constrained decoding's real value is the guarantee at scale and on harder schemas, not a measured lift over schema-in-prompt on this set."
|
|
- "Valid JSON is not equal to correct" (Valid ≠ correct): braces parse, shape wrong.
|
|
|
|
## Pros / strengths
|
|
|
|
- Real measured numbers on a real model with a reproducible harness, not vendor marketing; distinguishes the two failure modes (syntax vs schema) that most advice smears together.
|
|
- Directly supports tai's position: for a small fixed schema like the verdict object, schema-in-prompt plus validation is empirically sufficient.
|
|
|
|
## Cons / limitations / caveats
|
|
|
|
- One model family (Claude Haiku), 8 tasks, one operator; tai's verifier runs on weaker free-tier models where plain-prompt adherence is likely worse than 25%.
|
|
- Community repo, no peer review; the mock path ships deliberately different numbers as a teaching fixture, which a careless reader could quote instead of the real table.
|
|
|
|
## Relevance to tai
|
|
|
|
Quantitative backing for the verdict contract: paste the schema into the verifier prompt (tai does), validate strictly on receipt (tai does), and treat constrained decoding as a future guarantee rather than a current requirement. Also warns that tai's weaker verifier backends need the strict-parse-plus-retry path more than Claude-class models do.
|
|
|
|
## Changelog
|
|
|
|
| Date | Change |
|
|
|---|---|
|
|
| 2026-10-18 | Initial dossier created. |
|