Files

T

retoor af080fe68e Strict goal verification, approval hardening, retrieval eval, work limits, sandbox caps (1.38.0)

- Approval: re-prompt unknown answers (max 3), shell-only affirmative
  guidance recovery; destructive actions stay strict
- GoalVerifier subagent with strict JSON verdicts, bounded follow-up
  rounds, nonzero exit plus open turn on unverified one-shots
- Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with
  title weighting, query/read event logging, --eval-retrieval CLI
- Supervisor caps (8 parallel, 64 records) and scheduler hardening
  (4 fires/tick, per-schedule lease, stated misfire policy)
- Sandbox box resource trio: 2g/2.0/256 at create, best-effort
  update for existing boxes, rm -f recovery pointer
- Research archive: 12 subjects, 667 sources, 27 conclusions;
  suites 979 + 16 green

2026-10-18 12:13:24 +02:00

3.1 KiB

Raw Blame History

Structured model outputs

Metadata

Field	Value
Institute	Molodetz Institute — Research & Retrieval Division
Subject	agent-goal-verification
Document type	Source Dossier
Tier	1
Source title	Structured model outputs
Source URL	https://developers.openai.com/api/docs/guides/structured-outputs
Source publisher / author	OpenAI
Source type	vendor docs
Source date (published)	Unknown — not stated (living documentation page)
Date researched	2026-10-18
Researched by	tai (agent), model: Muse Spark, on behalf of retoor
Research method	direct page fetch
Confidence	High
Status	Active

Summary

OpenAI's official guide draws a hard line between JSON mode (valid JSON guaranteed, schema adherence not guaranteed) and Structured Outputs (schema adherence enforced at generation time). For callers that cannot use Structured Outputs, the guide prescribes exactly the fallback tai uses: validate the output and retry.

Key findings

"Structured Outputs is the evolution of JSON mode. While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence."
"JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors. You should use Structured Outputs to ensure it matches your schema, or if that is not possible, you should use a validation library and potentially retries to ensure that the output matches your desired schema."
Benefits claimed for Structured Outputs: "Reliable type-safety: No need to validate or retry incorrectly formatted responses", explicit refusals, simpler prompting.
The comparison table states JSON Mode outputs valid JSON: Yes; adheres to schema: No.
When using JSON mode, the model must be explicitly instructed to produce JSON (the API errors if the string "JSON" does not appear in context), and applications "must detect and handle the edge cases that can result in the model output not being a complete JSON object".

Pros / strengths

Authoritative statement of the two-tier guarantee ladder from the vendor that defines both tiers.
Gives an explicit sanctioned fallback (validate plus retries) for callers without schema enforcement, which is tai's exact situation on multi-backend free tiers.

Cons / limitations / caveats

Vendor documentation for OpenAI's own API; says nothing about other providers' JSON modes or their adherence rates.
No numbers on how often prompt-only JSON fails; quantitative failure rates come from community evals, not this page.

Relevance to tai

Directly load-bearing for the GoalVerifier design: tai cannot use constrained decoding across its heterogeneous backends (CLI harnesses, free tiers), so prompt-only JSON plus strict schema validation plus verification rounds is the vendor-recommended fallback, not a hack. The "detect and handle edge cases" line justifies the forgiving extract plus strict parse split in GoalVerifier.extract_json / GoalVerdict.from_text.

Changelog

Date	Change
2026-10-18	Initial dossier created.

3.1 KiB Raw Blame History