Files

T

retoor af080fe68e Strict goal verification, approval hardening, retrieval eval, work limits, sandbox caps (1.38.0)

- Approval: re-prompt unknown answers (max 3), shell-only affirmative
  guidance recovery; destructive actions stay strict
- GoalVerifier subagent with strict JSON verdicts, bounded follow-up
  rounds, nonzero exit plus open turn on unverified one-shots
- Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with
  title weighting, query/read event logging, --eval-retrieval CLI
- Supervisor caps (8 parallel, 64 records) and scheduler hardening
  (4 fires/tick, per-schedule lease, stated misfire policy)
- Sandbox box resource trio: 2g/2.0/256 at create, best-effort
  update for existing boxes, rm -f recovery pointer
- Research archive: 12 subjects, 667 sources, 27 conclusions;
  suites 979 + 16 green

2026-10-18 12:13:24 +02:00

2.9 KiB

Raw Blame History

Human-in-the-loop - OpenAI Agents SDK

Metadata

Field	Value
Institute	Molodetz Institute — Research & Retrieval Division
Subject	agent-goal-verification
Document type	Source Dossier
Tier	1
Source title	Human-in-the-loop - OpenAI Agents SDK
Source URL	https://openai.github.io/openai-agents-python/human_in_the_loop/
Source publisher / author	OpenAI
Source type	vendor docs
Source date (published)	Unknown — not stated (living documentation page)
Date researched	2026-10-18
Researched by	tai (agent), model: Muse Spark, on behalf of retoor
Research method	direct page fetch
Confidence	High
Status	Active

Summary

The official Agents SDK guide describes the industry-standard approval loop: tools declare needs_approval, the run pauses with pending items in interruptions, the app approves or rejects (optionally with a custom rejection message fed back into the run), and the same run resumes from serialized state.

Key findings

"Set needs_approval to True to always require approval or provide an async function that decides per call."
"Callable approval rules fail closed when the SDK cannot safely inspect the arguments. If the arguments are missing, empty, contain only whitespace, are malformed JSON... the callable is not invoked and the call requires manual approval."
"By default, a rejected tool call returns the SDK's standard rejection text back into the run" with per-call rejection_message override taking precedence over the run-wide formatter.
"Convert the result to a RunState with result.to_state(), call state.approve(...) or state.reject(...), and then resume", and the resumed run continues where it left off.
Per-call approvals are scoped to the call ID; always_approve / always_reject persist the decision for the tool identity.

Pros / strengths

First-party specification of the dominant approval pattern, including the fail-closed rule and the denial-feedback mechanism.
The pause / serialize / resume shape matches tai's turn-state plus --resume design almost exactly.

Cons / limitations / caveats

SDK-specific mechanics (RunState, interruptions) that tai reimplements in its own vocabulary; no guidance on free-text approval prompts or typo handling at a CLI.
No discussion of approval fatigue or input validation; those come from practitioner sources, not this page.

Relevance to tai

Confirms three tai design choices as industry-standard: per-command approval with an always-allow escape hatch, denial text fed back into the run as corrective context (tai's Denied guidance), and resumable paused runs (tai's open turn state plus --resume). The fail-closed rule supports keeping destructive approvals strict while recovering only the shell case on unambiguous affirmation.

Changelog

Date	Change
2026-10-18	Initial dossier created.

2.9 KiB Raw Blame History