tai/taicpp/docs/orchestration.md

Orchestration: sub agents on the multiplexer screen

Status: implemented in taicpp, not in the Python app. Code: src/mplex/mplex_vpane.*, src/orch/, src/agent/registry.*, src/agent/agent.*, src/cap/cap_mention.*. Tests: tests/orch_test.cpp (build/orch_test, step orch.unit), tests/orch_mplex_e2e.py with tests/orch_live.cpp (build/orch_live, step orch.mplex_e2e). Research: research/background-work-limits/ (agent teams, Temporal timeouts, claim check, Celery limits, tmux layouts, terminal streaming) and research/cap-protocol/. Related: cap-agents.md (what @agent runs), mplex.md (the multiplexer), fuzzy.md.

1. The rule

The main application orchestrates. Every sub agent it runs, whatever kind (an internal fork worker, a CAP session over any binding, an @agent mention), gets a panel on the main multiplexer session screen while it works, fed live, and is supervised by one clock whether it is shown or not.

 one sub agent             two sub agents            three
+-------+-------+        +-------+-------+        +-------+-------+
|       | agent |        |       | agent1|        |       | agent1|
| main  |   1   |        | main  +-------+        | main  +-------+
|       |       |        |       | agent2|        |       | agent2|
|       |       |        |       |       |        |       +-------+
+-------+-------+        +-------+-------+        |       | agent3|
                                                  +-------+-------+

The main pane stays on the left at full height, MPLEX_VPANE_MAIN_PERCENT (55) percent of the window wide. Sub agents are stacked equally on the right (tmux main-vertical). Panels appear when an agent starts and leave ORCH_LINGER seconds after it finishes; the column then rebalances. Outside a multiplexer (plain terminal, tmux) nothing is drawn and everything else works unchanged.

How the application gets into the multiplexer: the installed shell hook runs tai --plexer-wrap for interactive shells. A bare tai run is not wrapped automatically (changing how the program starts was not asked). Inside the multiplexer every pane has TAI_PLEXER_ACTIVE=1, TAI_PLEXER_SOCKET and TAI_PLEXER_PANE, and that is how the orchestrator finds its screen.

2. Architecture

Layer	Class	File	Job
server	MplexVpanes	src/mplex/mplex_vpane.*	feed panes: open, write, close, list; layout; linger; orphan lease
server	MplexPane::virtual_pane	src/mplex/mplex_pane.*	a pane without a process; row 0 is a header
server	MplexRenderer::frame	src/mplex/mplex_renderer.cpp	draws the header row and the screen below it
transport	OrchTransport, OrchMplexTransport, OrchNullTransport	src/orch/orch_transport.*	control-socket client, one round trip per write
core	OrchBoard	src/orch/orch_board.*	opens panels, coalesced flusher thread, watchdog clock
core	OrchPanel	src/orch/orch_panel.*	one agent's buffer, state, counters, cancel hook
view	OrchFormatter	src/orch/orch_formatter.*	events to sanitized, word-wrapped ANSI text
events	OrchEvent	src/orch/orch_event.h	what an agent tells its panel
adapter	OrchCapAdapter	src/orch/orch_cap_adapter.*	cap.* events to OrchEvent
adapter	Agent::observer	src/agent/agent.*	tool calls, results and streamed text of an internal worker
owner	spawn_agent / AgentRecord	src/agent/registry.*	opens a panel per fork worker, wires cancel, finishes it
owner	CapMention	src/cap/cap_mention.*	opens a panel per CAP turn, supervises, escalates cancel
policy	orch_policy.h	src/orch/orch_policy.h	every cap and floor (section 9)

Data flow for one streamed token:

agent thread --emit(OrchEvent)--> OrchPanel (format, append to buffer, mutex)
OrchBoard thread, every 25 ms: panel.collect(op) for every panel
      --> one "vpane_write" with all ops --> multiplexer server --> pane.feed() --> screen
server render loop, at most every 12 ms: diff by row --> attached terminal

Producers never wait for I/O. If the screen cannot keep up the buffer is bounded (section 4).

3. The multiplexer side: feed panes

A feed pane is an MplexPane with virtual_pane = true: no pty, no process, master_fd is -1, so the server loop never polls it and no reaper touches it. Its h - 1 rows are a normal MplexVtScreen (scrollback, copy mode, capture-pane, read all work); row 0 of its rectangle is a header line drawn by MplexRenderer::frame with the pane's header_style. Keys typed into it are dropped (MplexPane::write returns early for a feed pane).

3.1 Control protocol (JSON lines on the multiplexer socket)

All requests are one line, one reply line, one connection per request, like the existing send and read commands. Replies are {"ok": true, "data": ...} or {"ok": false, "error": "..."}.

vpane_open

{"cmd":"vpane_open","pane":"0","label":"agent #1","header":" ● agent #1 · running · 0s","state":"running","main_percent":55}
{"ok":true,"data":{"id":3,"cols":53,"rows":38}}

pane is the origin pane (default: the active pane of the latest session). label becomes the pane title, header the first line, state selects the header color. Errors: too many feed panes (limit 8), too many panes (limit 256), no space for another feed pane (the window is too short for another pane of at least MPLEX_VPANE_MIN_ROWS rows), window too narrow for a feed pane (under 41 columns).

vpane_write

{"cmd":"vpane_write","ops":[
  {"pane":3,"data":"\u001b[1mhello\u001b[0m\r\n","header":" ● agent #1 · running · 4s","state":"running"},
  {"pane":4,"data":"bye\r\n","state":"done","done":true,"linger":20}]}
{"ok":true,"data":{"closed":[],"cols":{"3":53,"4":53}}}

Per op: pane (required), data (text for the screen, at most MPLEX_VPANE_WRITE_MAX = 65536 bytes), header, state (running, idle, done, failed, timeout, cancelled), done (start the linger countdown) with linger seconds (clamped to MPLEX_VPANE_LINGER_MAX = 300). At most MPLEX_VPANE_OPS_MAX = 16 ops per request. Every op also refreshes the pane's lease. The reply lists in closed the pane ids that no longer exist (the user closed them or they timed out) and in cols the current width of every pane written to, which the client uses for word wrap. Errors: ops must be a list, too many ops in one write, op data too large.

vpane_close closes one pane now. vpane_list returns [{id, state, header, cols, rows, closing}].

3.2 Layout policy (MplexVpanes::arrange)

After every open and every close the window is rebuilt with MplexLayout::preset("main-vertical", panes, ...):

• the first element is the main pane (the first real, non-feed pane of the window);
• then any other real panes the user made;
• then the feed panes in creation order.

The main width is window width x percent / 100, clamped to [1, width - 2]; when the right column would be narrower than MPLEX_VPANE_MIN_COLS (20) the main pane gives up width to keep it. Heights are shared equally by the existing shares rule (the first pane gets the remainder), for example [13, 13, 12] for three agents in a 40-row window (the heights plus two border rows). The layout is re-applied on every open and close, so a layout the user set by hand does not survive a panel change.

3.3 Lifecycle on the server

Event	Behavior
done op	close_at = now + linger; the pane closes on the next tick after that and the column rebalances
user kills a feed pane (prefix x)	the pane is gone; the next write reports its id in closed; the orchestrator cancels the job
owner stops writing for MPLEX_VPANE_LEASE (30 s)	the header becomes owner gone: ..., state failed, closes after MPLEX_VPANE_ORPHAN_LINGER (10 s)
the main pane exits	MplexServer::kill_pane closes the whole window when only feed panes remain
window zoomed	opening a panel un-zooms it

The server tick runs once a second, so linger and lease have one-second granularity.

4. The orchestrator side

4.1 OrchBoard

One per process (OrchBoard::instance(), created on first use over OrchMplexTransport::from_env(), flushed at exit with a 2 s limit, never destroyed so detached worker threads cannot outlive it). open(label, task) never returns null: when there is no multiplexer (TAI_PLEXER_ACTIVE unset), TAI_ORCH_PANELS=0, no space, or ORCH_PANELS_MAX panels are already shown, it returns a panel with no pane behind it and records the reason (last_error()). Such a panel behaves identically except nothing is drawn.

The board thread runs every ORCH_FLUSH_INTERVAL (25 ms):

1. once per ORCH_MONITOR_INTERVAL (1 s) call watchdog on every panel;
2. call collect on every panel; combine the non-empty results into one vpane_write;
3. apply the reply: widths to the panels, closed ids to closed_by_user();
4. drop panels that are drained.

Three consecutive failed writes mark the transport broken (transport_ok() == false, reason in last_error()): drawing stops, supervision continues, new panels are hidden panels.

4.2 OrchPanel

Thread-safe. State: Queued, Running, Idle, Done, Failed, Timeout, Cancelled.

        emit()                       finish(state)
Running ------> Running   Running/Idle ----------> Done | Failed | Timeout | Cancelled
   | no event 60 s                 ^ emit() while Idle returns to Running
   v
 Idle --- no event 300 s ---> cancel hook fires once
 any --- pane closed by user ---> Cancelled, cancel hook fires

• emit(event) formats the event, appends to the buffer, counts tools and tokens, and is a heartbeat. After finish events are ignored.
• finish(state, summary) flushes the held word, appends the footer, and arms the done op, which is sent only after the buffer is drained.
• collect(op) takes up to ORCH_WRITE_MAX (32768) bytes without splitting a UTF-8 character, adds a changed header and state, and returns false when there is nothing to send. A word held back by the formatter is released when the text stream has paused for 0.1 s.
• The buffer is capped at ORCH_PENDING_MAX (262144). Over it, the oldest bytes are dropped at a line end and a visible line output skipped: the screen fell behind is inserted. Dropping is counted (dropped_bytes()).
• watchdog(now) is the idle logic above; set_cancel registers the owner's cancel function, called at most once.

Header text: <icon> <label> · <state> · <elapsed> · <n> tools · <tokens> tok, with ● running, ◌ idle, ✓ done, ✗ failed, ⏱ timeout, ⊘ cancelled; elapsed is Ns or MmSSs; tokens abbreviated (1.5k). Header colors (server side, by state): running blue, idle and timeout orange, done green, failed red, cancelled grey.

4.3 OrchFormatter

Turns events into bytes for a feed pane (CRLF line ends, because a feed pane has no tty to translate them).

Event	Rendering
(opening)	dim ▸ <task> on one line, at most ORCH_TASK_SHOW_MAX (300) characters, then a blank line
Text	word-wrapped to the pane width
Thought	the same, dim italic
ToolStart	on a new line: cyan ⏺ name, then dim (arguments) on one line, clipped to the pane width and ORCH_TOOL_ARGS_MAX (120)
ToolEnd	dim, red when failed: ⎿ the first ORCH_TOOL_RESULT_LINES (3) non-empty lines, each clipped to the width and ORCH_TOOL_RESULT_COLS (160), then · N chars
Plan	magenta ▤ and the plan lines
Note	yellow · text, at most 400 characters
Error	red ✗ text, at most 400 characters
Usage	nothing in the stream; shown in the header
(footer)	blank line, colored ✓ done / ✗ failed / ◌ timeout / ◌ cancelled, seconds, · summary (300)

Sanitizing: agent text may contain only SGR color sequences, newline and tab. Every other escape sequence (cursor movement, OSC titles, ...) and every other control byte is removed, so an agent cannot move the cursor, retitle the terminal or write outside its pane.

Word wrap: the width comes from the server (cols - 1, so a character in the last column cannot wrap by itself). A word that does not fit the rest of the line moves to the next line; a word longer than a line is broken; multi-byte characters count as one column; SGR sequences count as zero. The last, possibly incomplete word of a chunk is held until a space or newline arrives, because tokens often split a word.

4.4 Event sources

Internal worker (fork): Agent::observer is called from Agent::show_call (tool start, with secrets masked: store_secret arguments become [hidden], everything else goes through the vault redaction), from Agent::show_result (tool end, failure detected from the result text the way the main display does), from the stream sink of chat.complete (text pieces as they arrive), and after each reply (the whole text when the backend did not stream; reasoning when there is no text and no tool call).

CAP session: CapSession::addTap receives every event of the session; OrchCapAdapter::map turns them:

CAP event	OrchEvent
cap.text_chunk (channel thought)	Thought
cap.text_chunk, cap.thought	Text / Thought, newline appended when missing
cap.tool_call.start	ToolStart (name, input as JSON)
cap.tool_call.end	ToolEnd; the tool name comes from the remembered call id (at most 256 remembered)
cap.plan	Plan: [status] content per entry
cap.usage	Usage: input plus output tokens
cap.error	Error: code: message
cap.ask_user, cap.permission.request, cap.artifact	Note
anything else	ignored

5. Owners

5.1 fork workers (spawn_agent)

1. refuse loudly past AGENTS_MAX_PARALLEL (8) live workers (AgentsFull, existing behavior);
2. open a panel agent #<id> with the task; register a cancel function that raises AgentRecord::cancel;
3. in the worker thread default_runner sets worker.observer (events to the panel) and worker.cancel_flag;
4. the agent loop checks the flag before every step: a raised flag ends the turn with [cancelled];
5. on completion the record gets its status and the panel is finished: done to Done, the text [time limit reached] to Timeout, a raised cancel flag to Cancelled (record status error, result worker cancelled (<reason>); partial result: ...), any exception to Failed.

poll and the lanes display are unchanged. poll collects the result and spills a large one (section 7).

5.2 CAP turns (CapMention::ask)

1. open a panel cap <agent>[:<model>] with the prompt;
2. reuse or create the session for (agent, model, directory); a tap routes its events to the current panel;
3. register a cancel function: cap.cancel for the current turn at once, and after ORCH_CANCEL_GRACE (5 s) close() on the session if the turn is still running;
4. session.ask(prompt, timeout); a CapTimeoutError cancels and drops the session;
5. the caller (Agent::run_agent) collects the answer (section 7) and finishes the panel: Done, Timeout, Cancelled or Failed with a one-line summary.

6. Supervision

Concern	Rule	Constant
Parallelism	at most 8 live workers, refused loudly; at most 8 panels per process; the server allows 8 feed panes	ORCH_PANELS_MAX, MPLEX_VPANE_MAX
Space	a pane needs MPLEX_VPANE_MIN_ROWS (6, header plus five) and the window 41 columns; otherwise the agent runs without a panel and last_error says why	MPLEX_VPANE_MIN_ROWS, MPLEX_VPANE_MIN_COLS
Wall clock	fork: 30..3600 s, default 600, the worker ends with [time limit reached]; CAP turn: 600 s	existing
Heartbeat	every event is one; 60 s of silence shows idle	ORCH_IDLE_WARN
Stall	300 s of silence cancels the job	ORCH_IDLE_KILL
User	closing the pane cancels the job	none
Cancel	soft first (flag read between steps, cap.cancel); a CAP session that ignores it is closed after the grace	ORCH_CANCEL_GRACE
Display failure	three failed writes stop drawing; supervision continues	board

These follow the research: a limit on every job and a heartbeat to find a stalled worker earlier (Temporal), soft stop before hard stop (Celery, Kubernetes), a cap on useful team size (agent teams advice: 3 to 5, token cost scales with the number of agents). The idle limits are static judgments, not measurements.

Supervision does not depend on a panel being shown: the watchdog runs on every panel the board tracks.

7. Collecting results

Unchanged, shared and deliberately not the panel's job: the panel is a view with a scrollback cap, not a store.

• A result longer than SPILL_LIMIT (6000 characters) is stored as a record (mem: id, chunked and indexed) and the conversation receives the head, the tail and the record id (spilled_view); record_read pages the rest. This is the claim-check pattern (research/background-work-limits/azure-claim-check-pattern.md).
• poll does this for fork workers (once; the record id is cached on the agent record).
• Agent::run_agent does it for CAP answers with spill_or_truncate, tag agent-output.
• The panel footer shows the size and, for CAP answers, whether it was stored.

8. Switches and environment

Name	Effect
TAI_PLEXER_ACTIVE, TAI_PLEXER_SOCKET, TAI_PLEXER_PANE	set by the multiplexer in every pane; the orchestrator's way to find its screen
TAI_ORCH_PANELS=0	draw nothing (supervision stays on)

There is no configuration file for the caps; they are constants (section 9).

9. Caps and floors (all static)

Constant	Value
ORCH_PANELS_MAX	8
ORCH_FLUSH_INTERVAL	0.025 s
ORCH_MONITOR_INTERVAL	1.0 s
ORCH_PENDING_MAX	262144 bytes
ORCH_WRITE_MAX	32768 bytes
ORCH_LINGER	20 s
ORCH_IDLE_WARN	60 s
ORCH_IDLE_KILL	300 s
ORCH_CANCEL_GRACE	5 s
ORCH_TASK_SHOW_MAX	300 characters
ORCH_TOOL_ARGS_MAX	120 characters
ORCH_TOOL_RESULT_LINES	3
ORCH_TOOL_RESULT_COLS	160
ORCH_LABEL_MAX	60
ORCH_CALL_TIMEOUT_MS	2000
MPLEX_VPANE_MAX	8
MPLEX_VPANE_MIN_ROWS	6
MPLEX_VPANE_MIN_COLS	20
MPLEX_VPANE_MAIN_PERCENT	55
MPLEX_VPANE_OPS_MAX	16
MPLEX_VPANE_WRITE_MAX	65536 bytes
MPLEX_VPANE_HEADER_MAX	200 characters
MPLEX_VPANE_LEASE	30 s
MPLEX_VPANE_ORPHAN_LINGER	10 s
MPLEX_VPANE_LINGER_MAX	300 s

The feed panes and the vpane_* commands are outside the tables that mplex_spec_test compares with the Python mplex.md, because the Python app has none of them.

10. Measured behavior

Measured with the real binaries (an attached client on a pty, three agents each streaming 500 tokens/s for about 1.3 s):

Measure	Value
frames received by the client	85 in 1.33 s (about 64 per second)
interval between frames	median 12.9 ms, p90 25.3 ms (the single 295 ms gap is the scripted pause at the end of the stream)
bytes to the terminal	234 KB (row-diffed frames, three panes changing)
writes to the multiplexer	at most 40 per second however many agents stream (unit test: 200 events in far fewer than 120 writes)

Layout measured on a 120 x 40 window: one agent, main pane 66 columns and 40 rows, agent pane 53 x 40; three agents, right column heights [13, 13, 12]; six agents fit; a seventh gets no panel.

Live check: the real REPL as the main pane (detz --yolo), a real model asked to fork two workers: two panels appeared, streamed the tool calls (⏺ shell(...), ⎿ exit 0, output), changed to ✓ ... done, showed the footer, and left after the linger. A real @tai:zen CAP turn did the same (header cap tai:zen-free, the panel filled when the turn ended, section 11).

11. Verification

Test	Covers
tests/orch_test.cpp	sanitizing, clipping, CRLF, word wrap (inside a word never, across chunks rejoined, long words broken, multi-byte kept), tool and error rendering, the CAP adapter, panel buffering and the byte ceiling, never splitting a character, idle warn and idle cancel (once), cancel on user close, a hidden panel supervised like a shown one, the board over a fake transport (ordering, coalescing, done op, closed pane to cancel), no space, repeated write failure, the panel ceiling, and that this page lists every constant and command
tests/orch_mplex_e2e.py	a real multiplexer server and the real board: geometry for 1, 3 and 6 panels, content and footers, the space limit, linger and rebalance, a killed pane reported as closed, a failed footer, the main pane exiting
tests/run_all.py	steps orch.unit, orch.mplex_e2e, plus all mplex.* steps (the multiplexer itself is unchanged for existing behavior)

Not covered by an automated test: the orphan lease (30 s plus 10 s) and the live REPL runs above.

12. Gaps

• tmux and bare terminals get no panels (a tail-file transport would close this).
• Keys typed into a feed pane are ignored; steering an agent from its pane is not built.
• The idle watchdog needs events; a CAP agent that streams nothing for 300 s is cancelled even if it works.
• A hung model request inside an internal worker is bounded only by the worker's own wall clock; the cancel flag is read between steps.
• tai as a CAP agent (--cap-speak) sends its transcript after the turn, with generic tool names, so a tai sub agent over CAP fills its panel at the end of the turn; agents that stream (stream-json, ACP) fill it live. An internal fork worker is live.
• Panels are not restored after the orchestrator restarts; agent output records have no retention policy yet.
• A bare tai is not wrapped in the multiplexer automatically.
• Cancel escalation closes a CAP session from a second thread; a race inside the driver on close is possible.