retoorandClaude Sonnet 5.5
ec9c471780
Feed panes in the multiplexer (vpane_* control commands, main-vertical layout, header row), src/orch (board, panel, formatter, transport, CAP adapter), Agent observer and cancel flag, supervision (idle warn/cancel, user close cancels, soft then hard stop), CAP answers collected through the spill path. Research archived under background-work-limits. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
3.8 KiB
Conclusion: Supervise sub agents with three clocks and collect results through the existing claim check
Metadata
| Field | Value |
|---|---|
| Institute | Molodetz Institute — Research & Retrieval Division |
| Subject | background-work-limits |
| Document type | Conclusion |
| Tier | N/A |
| Date decided | 2026-10-25 |
| Decided by | retoor (instructed the design and its integration), advised by tai (agent, Claude Sonnet 5) |
| Status | Active |
| Confidence | Medium |
Question / problem statement
How should running sub agents be scheduled, monitored, timed out and cancelled, and how should their results come back without flooding the parent's context.
Conclusion
Three clocks and a staged stop. (1) A wall-clock limit on every job: fork 30..3600 s, a CAP turn at CAP_TURN_TIMEOUT. (2) A heartbeat: every event is a heartbeat; 60 s without one marks the panel idle, 300 s cancels the job. (3) The user: closing the pane cancels. Stop is soft first (the worker stops at its next step; a CAP agent gets cap.cancel), then hard (a CAP session that ignores it is closed after 5 s). Parallelism stays capped at 8 live workers with a loud refusal. Supervision runs on the board's clock whether or not a panel is shown, and a failing display never stops supervision. Results use the claim-check path the system already has: a long answer (over 6000 characters) is stored as a record and the parent receives the head, the tail and the record id; poll does this for workers, Agent::run_agent for CAP answers. The panel is only a view.
Supporting sources
- temporal-blog-activity-timeouts: always set start-to-close; heartbeat finds dead workers early; bound retries by schedule-to-close.
- celery-workers-guide: soft limit before hard; cancelling a running job needs an explicit escalation.
- azure-claim-check-pattern: store the payload, pass a reference, conditional by size, write before reference, define retention.
- claude-code-agent-teams-docs: cooperative shutdown, token cost scales with agent count, failed teammates reported with the error.
- Existing in this subject: erlang-otp-supervisor, temporal-workflow-activity, go-semaphore (bounded parallelism, reject loud).
Alternatives considered and rejected
- Wall clock only: a stalled agent would burn the whole limit before anyone noticed.
- Automatic retries of sub agents: agents are not idempotent (they write files and run commands) and Temporal itself warns about unbounded retries; the parent decides.
- Hard kill first: loses the partial answer; the soft stage returns what the agent had.
- Putting results into the pane and reading them back from the screen: the screen is a view with a scrollback cap, not a store.
- Kubernetes-style restart back-off: not applicable, nothing restarts a sub agent; its page could not be read, so it was not relied on.
Risks / open questions
- The idle clock needs events: an agent that thinks silently for 300 s is cancelled although it works (the limit is a static guess, not a measurement).
- A hung model request inside an internal worker is only cut by its own wall-clock deadline; the cancel flag is read between steps.
- Output records of finished agents have no retention policy yet (the claim-check pattern asks for an owner of deletion).
- Cancel escalation closes the CAP session from a second thread; a race inside the driver on close is possible.
Action taken
Implemented the same session: OrchPanel watchdog and cancel hook, AgentRecord cancel flag and panel, CapMention timeout/cancel/escalation, Agent::run_agent spill; tests in orch_test. Spec: docs/orchestration.md.
Changelog
| Date | Change |
|---|---|
| 2026-10-25 | Initial conclusion recorded. |