tai/taicpp/docs/orchestration.md

# Orchestration: sub agents on the multiplexer screen

Status: implemented in taicpp, not in the Python app. Code: `src/mplex/mplex_vpane.*`, `src/orch/`,
`src/agent/registry.*`, `src/agent/agent.*`, `src/cap/cap_mention.*`. Tests: `tests/orch_test.cpp`
(`build/orch_test`, step `orch.unit`), `tests/orch_mplex_e2e.py` with `tests/orch_live.cpp`
(`build/orch_live`, step `orch.mplex_e2e`). Research: `research/background-work-limits/` (agent teams,
Temporal timeouts, claim check, Celery limits, tmux layouts, terminal streaming) and
`research/cap-protocol/`. Related: [cap-agents.md](cap-agents.md) (what `@agent` runs),
[mplex.md](mplex.md) (the multiplexer), [fuzzy.md](fuzzy.md).

## 1. The rule

The main application orchestrates. Every sub agent it runs, whatever kind (an internal `fork` worker, a CAP
session over any binding, an `@agent` mention), gets a panel on the main multiplexer session screen while it
works, fed live, and is supervised by one clock whether it is shown or not.

```
 one sub agent             two sub agents            three
+-------+-------+        +-------+-------+        +-------+-------+
|       | agent |        |       | agent1|        |       | agent1|
| main  |   1   |        | main  +-------+        | main  +-------+
|       |       |        |       | agent2|        |       | agent2|
|       |       |        |       |       |        |       +-------+
+-------+-------+        +-------+-------+        |       | agent3|
                                                  +-------+-------+
```

The main pane stays on the left at full height, `MPLEX_VPANE_MAIN_PERCENT` (55) percent of the window wide.
Sub agents are stacked equally on the right (tmux `main-vertical`). Panels appear when an agent starts and
leave `ORCH_LINGER` seconds after it finishes; the column then rebalances. Outside a multiplexer (plain
terminal, tmux) nothing is drawn and everything else works unchanged.

How the application gets into the multiplexer: the installed shell hook runs `tai --plexer-wrap` for interactive
shells. A bare `tai` run is not wrapped automatically (changing how the program starts was not asked).
Inside the multiplexer every pane has `TAI_PLEXER_ACTIVE=1`, `TAI_PLEXER_SOCKET` and `TAI_PLEXER_PANE`, and
that is how the orchestrator finds its screen.

## 2. Architecture

| Layer | Class | File | Job |
|---|---|---|---|
| server | `MplexVpanes` | `src/mplex/mplex_vpane.*` | feed panes: open, write, close, list; layout; linger; orphan lease |
| server | `MplexPane::virtual_pane` | `src/mplex/mplex_pane.*` | a pane without a process; row 0 is a header |
| server | `MplexRenderer::frame` | `src/mplex/mplex_renderer.cpp` | draws the header row and the screen below it |
| transport | `OrchTransport`, `OrchMplexTransport`, `OrchNullTransport` | `src/orch/orch_transport.*` | control-socket client, one round trip per write |
| core | `OrchBoard` | `src/orch/orch_board.*` | opens panels, coalesced flusher thread, watchdog clock |
| core | `OrchPanel` | `src/orch/orch_panel.*` | one agent's buffer, state, counters, cancel hook |
| view | `OrchFormatter` | `src/orch/orch_formatter.*` | events to sanitized, word-wrapped ANSI text |
| events | `OrchEvent` | `src/orch/orch_event.h` | what an agent tells its panel |
| adapter | `OrchCapAdapter` | `src/orch/orch_cap_adapter.*` | `cap.*` events to `OrchEvent` |
| adapter | `Agent::observer` | `src/agent/agent.*` | tool calls, results and streamed text of an internal worker |
| owner | `spawn_agent` / `AgentRecord` | `src/agent/registry.*` | opens a panel per `fork` worker, wires cancel, finishes it |
| owner | `CapMention` | `src/cap/cap_mention.*` | opens a panel per CAP turn, supervises, escalates cancel |
| policy | `orch_policy.h` | `src/orch/orch_policy.h` | every cap and floor (section 9) |

Data flow for one streamed token:

```
agent thread --emit(OrchEvent)--> OrchPanel (format, append to buffer, mutex)
OrchBoard thread, every 25 ms: panel.collect(op) for every panel
      --> one "vpane_write" with all ops --> multiplexer server --> pane.feed() --> screen
server render loop, at most every 12 ms: diff by row --> attached terminal
```

Producers never wait for I/O. If the screen cannot keep up the buffer is bounded (section 4).

## 3. The multiplexer side: feed panes

A feed pane is an `MplexPane` with `virtual_pane = true`: no pty, no process, `master_fd` is -1, so the
server loop never polls it and no reaper touches it. Its `h - 1` rows are a normal `MplexVtScreen` (scrollback,
copy mode, `capture-pane`, `read` all work); row 0 of its rectangle is a header line drawn by
`MplexRenderer::frame` with the pane's `header_style`. Keys typed into it are dropped (`MplexPane::write` returns
early for a feed pane).

### 3.1 Control protocol (JSON lines on the multiplexer socket)

All requests are one line, one reply line, one connection per request, like the existing `send` and `read`
commands. Replies are `{"ok": true, "data": ...}` or `{"ok": false, "error": "..."}`.

**`vpane_open`**

```json
{"cmd":"vpane_open","pane":"0","label":"agent #1","header":" ● agent #1 · running · 0s","state":"running","main_percent":55}
{"ok":true,"data":{"id":3,"cols":53,"rows":38}}
```

`pane` is the origin pane (default: the active pane of the latest session). `label` becomes the pane title,
`header` the first line, `state` selects the header color. Errors: `too many feed panes (limit 8)`,
`too many panes (limit 256)`, `no space for another feed pane` (the window is too short for another pane of
at least `MPLEX_VPANE_MIN_ROWS` rows), `window too narrow for a feed pane` (under 41 columns).

**`vpane_write`**

```json
{"cmd":"vpane_write","ops":[
  {"pane":3,"data":"\u001b[1mhello\u001b[0m\r\n","header":" ● agent #1 · running · 4s","state":"running"},
  {"pane":4,"data":"bye\r\n","state":"done","done":true,"linger":20}]}
{"ok":true,"data":{"closed":[],"cols":{"3":53,"4":53}}}
```

Per op: `pane` (required), `data` (text for the screen, at most `MPLEX_VPANE_WRITE_MAX` = 65536 bytes), `header`,
`state` (`running`, `idle`, `done`, `failed`, `timeout`, `cancelled`), `done` (start the linger countdown) with
`linger` seconds (clamped to `MPLEX_VPANE_LINGER_MAX` = 300). At most `MPLEX_VPANE_OPS_MAX` = 16 ops per request.
Every op also refreshes the pane's lease. The reply lists in `closed` the pane ids that no longer exist (the
user closed them or they timed out) and in `cols` the current width of every pane written to, which the client
uses for word wrap. Errors: `ops must be a list`, `too many ops in one write`, `op data too large`.

**`vpane_close`** closes one pane now. **`vpane_list`** returns
`[{id, state, header, cols, rows, closing}]`.

### 3.2 Layout policy (`MplexVpanes::arrange`)

After every open and every close the window is rebuilt with `MplexLayout::preset("main-vertical", panes, ...)`:

- the first element is the main pane (the first real, non-feed pane of the window);
- then any other real panes the user made;
- then the feed panes in creation order.

The main width is `window width x percent / 100`, clamped to [1, width - 2]; when the right column would be
narrower than `MPLEX_VPANE_MIN_COLS` (20) the main pane gives up width to keep it. Heights are shared equally
by the existing `shares` rule (the first pane gets the remainder), for example `[13, 13, 12]` for three agents
in a 40-row window (the heights plus two border rows). The layout is re-applied on every open and close, so a layout the user
set by hand does not survive a panel change.

### 3.3 Lifecycle on the server

| Event | Behavior |
|---|---|
| `done` op | `close_at = now + linger`; the pane closes on the next tick after that and the column rebalances |
| user kills a feed pane (`prefix x`) | the pane is gone; the next write reports its id in `closed`; the orchestrator cancels the job |
| owner stops writing for `MPLEX_VPANE_LEASE` (30 s) | the header becomes `owner gone: ...`, state `failed`, closes after `MPLEX_VPANE_ORPHAN_LINGER` (10 s) |
| the main pane exits | `MplexServer::kill_pane` closes the whole window when only feed panes remain |
| window zoomed | opening a panel un-zooms it |

The server tick runs once a second, so linger and lease have one-second granularity.

## 4. The orchestrator side

### 4.1 `OrchBoard`

One per process (`OrchBoard::instance()`, created on first use over `OrchMplexTransport::from_env()`, flushed
at exit with a 2 s limit, never destroyed so detached worker threads cannot outlive it). `open(label, task)`
never returns null: when there is no multiplexer (`TAI_PLEXER_ACTIVE` unset), `TAI_ORCH_PANELS=0`, no space,
or `ORCH_PANELS_MAX` panels are already shown, it returns a panel with no pane behind it and records the reason
(`last_error()`). Such a panel behaves identically except nothing is drawn.

The board thread runs every `ORCH_FLUSH_INTERVAL` (25 ms):

1. once per `ORCH_MONITOR_INTERVAL` (1 s) call `watchdog` on every panel;
2. call `collect` on every panel; combine the non-empty results into one `vpane_write`;
3. apply the reply: widths to the panels, `closed` ids to `closed_by_user()`;
4. drop panels that are drained.

Three consecutive failed writes mark the transport broken (`transport_ok() == false`, reason in
`last_error()`): drawing stops, supervision continues, new panels are hidden panels.

### 4.2 `OrchPanel`

Thread-safe. State: `Queued`, `Running`, `Idle`, `Done`, `Failed`, `Timeout`, `Cancelled`.

```
        emit()                       finish(state)
Running ------> Running   Running/Idle ----------> Done | Failed | Timeout | Cancelled
   | no event 60 s                 ^ emit() while Idle returns to Running
   v
 Idle --- no event 300 s ---> cancel hook fires once
 any --- pane closed by user ---> Cancelled, cancel hook fires
```

- `emit(event)` formats the event, appends to the buffer, counts tools and tokens, and is a heartbeat. After
  `finish` events are ignored.
- `finish(state, summary)` flushes the held word, appends the footer, and arms the `done` op, which is sent only
  after the buffer is drained.
- `collect(op)` takes up to `ORCH_WRITE_MAX` (32768) bytes without splitting a UTF-8 character, adds a changed
  header and state, and returns false when there is nothing to send. A word held back by the formatter is
  released when the text stream has paused for 0.1 s.
- The buffer is capped at `ORCH_PENDING_MAX` (262144). Over it, the oldest bytes are dropped at a line end and a
  visible line `output skipped: the screen fell behind` is inserted. Dropping is counted (`dropped_bytes()`).
- `watchdog(now)` is the idle logic above; `set_cancel` registers the owner's cancel function, called at most once.

Header text: ` <icon> <label> · <state> · <elapsed> · <n> tools · <tokens> tok`, with `●` running, `◌` idle,
`✓` done, `✗` failed, `⏱` timeout, `⊘` cancelled; elapsed is `Ns` or `MmSSs`; tokens abbreviated (`1.5k`).
Header colors (server side, by state): running blue, idle and timeout orange, done green, failed red, cancelled grey.

### 4.3 `OrchFormatter`

Turns events into bytes for a feed pane (CRLF line ends, because a feed pane has no tty to translate them).

| Event | Rendering |
|---|---|
| (opening) | dim `▸ <task>` on one line, at most `ORCH_TASK_SHOW_MAX` (300) characters, then a blank line |
| Text | word-wrapped to the pane width |
| Thought | the same, dim italic |
| ToolStart | on a new line: cyan `⏺ name`, then dim `(arguments)` on one line, clipped to the pane width and `ORCH_TOOL_ARGS_MAX` (120) |
| ToolEnd | dim, red when failed: `  ⎿` the first `ORCH_TOOL_RESULT_LINES` (3) non-empty lines, each clipped to the width and `ORCH_TOOL_RESULT_COLS` (160), then `· N chars` |
| Plan | magenta `▤` and the plan lines |
| Note | yellow `· text`, at most 400 characters |
| Error | red `✗ text`, at most 400 characters |
| Usage | nothing in the stream; shown in the header |
| (footer) | blank line, colored `✓ done` / `✗ failed` / `◌ timeout` / `◌ cancelled`, seconds, ` · summary` (300) |

Sanitizing: agent text may contain only SGR color sequences, newline and tab. Every other escape sequence
(cursor movement, OSC titles, ...) and every other control byte is removed, so an agent cannot move the
cursor, retitle the terminal or write outside its pane.

Word wrap: the width comes from the server (`cols - 1`, so a character in the last column cannot wrap by
itself). A word that does not fit the rest of the line moves to the next line; a word longer than a line is
broken; multi-byte characters count as one column; SGR sequences count as zero. The last, possibly incomplete word of a chunk is held until
a space or newline arrives, because tokens often split a word.

### 4.4 Event sources

**Internal worker (`fork`)**: `Agent::observer` is called from `Agent::show_call` (tool start, with secrets
masked: `store_secret` arguments become `[hidden]`, everything else goes through the vault redaction), from
`Agent::show_result` (tool end, failure detected from the result text the way the main display does), from the
stream sink of `chat.complete` (text pieces as they arrive), and after each reply (the whole text when the
backend did not stream; reasoning when there is no text and no tool call).

**CAP session**: `CapSession::addTap` receives every event of the session; `OrchCapAdapter::map` turns them:

| CAP event | OrchEvent |
|---|---|
| `cap.text_chunk` (channel `thought`) | Thought |
| `cap.text_chunk`, `cap.thought` | Text / Thought, newline appended when missing |
| `cap.tool_call.start` | ToolStart (name, input as JSON) |
| `cap.tool_call.end` | ToolEnd; the tool name comes from the remembered call id (at most 256 remembered) |
| `cap.plan` | Plan: `[status] content` per entry |
| `cap.usage` | Usage: input plus output tokens |
| `cap.error` | Error: `code: message` |
| `cap.ask_user`, `cap.permission.request`, `cap.artifact` | Note |
| anything else | ignored |

## 5. Owners

### 5.1 `fork` workers (`spawn_agent`)

1. refuse loudly past `AGENTS_MAX_PARALLEL` (8) live workers (`AgentsFull`, existing behavior);
2. open a panel `agent #<id>` with the task; register a cancel function that raises `AgentRecord::cancel`;
3. in the worker thread `default_runner` sets `worker.observer` (events to the panel) and `worker.cancel_flag`;
4. the agent loop checks the flag before every step: a raised flag ends the turn with `[cancelled]`;
5. on completion the record gets its status and the panel is finished: `done` to Done, the text
   `[time limit reached]` to Timeout, a raised cancel flag to Cancelled (record status `error`, result
   `worker cancelled (<reason>); partial result: ...`), any exception to Failed.

`poll` and the lanes display are unchanged. `poll` collects the result and spills a large one (section 7).

### 5.2 CAP turns (`CapMention::ask`)

1. open a panel `cap <agent>[:<model>]` with the prompt;
2. reuse or create the session for (agent, model, directory); a tap routes its events to the current panel;
3. register a cancel function: `cap.cancel` for the current turn at once, and after `ORCH_CANCEL_GRACE` (5 s)
   `close()` on the session if the turn is still running;
4. `session.ask(prompt, timeout)`; a `CapTimeoutError` cancels and drops the session;
5. the caller (`Agent::run_agent`) collects the answer (section 7) and finishes the panel: Done, Timeout,
   Cancelled or Failed with a one-line summary.

## 6. Supervision

| Concern | Rule | Constant |
|---|---|---|
| Parallelism | at most 8 live workers, refused loudly; at most 8 panels per process; the server allows 8 feed panes | `ORCH_PANELS_MAX`, `MPLEX_VPANE_MAX` |
| Space | a pane needs `MPLEX_VPANE_MIN_ROWS` (6, header plus five) and the window 41 columns; otherwise the agent runs without a panel and `last_error` says why | `MPLEX_VPANE_MIN_ROWS`, `MPLEX_VPANE_MIN_COLS` |
| Wall clock | `fork`: 30..3600 s, default 600, the worker ends with `[time limit reached]`; CAP turn: 600 s | existing |
| Heartbeat | every event is one; 60 s of silence shows `idle` | `ORCH_IDLE_WARN` |
| Stall | 300 s of silence cancels the job | `ORCH_IDLE_KILL` |
| User | closing the pane cancels the job | none |
| Cancel | soft first (flag read between steps, `cap.cancel`); a CAP session that ignores it is closed after the grace | `ORCH_CANCEL_GRACE` |
| Display failure | three failed writes stop drawing; supervision continues | board |

These follow the research: a limit on every job and a heartbeat to find a stalled worker earlier
(Temporal), soft stop before hard stop (Celery, Kubernetes), a cap on useful team size (agent teams advice: 3 to
5, token cost scales with the number of agents). The idle limits are static judgments, not measurements.

Supervision does not depend on a panel being shown: the watchdog runs on every panel the board tracks.

## 7. Collecting results

Unchanged, shared and deliberately not the panel's job: the panel is a view with a scrollback cap, not a store.

- A result longer than `SPILL_LIMIT` (6000 characters) is stored as a record (`mem:` id, chunked and indexed)
  and the conversation receives the head, the tail and the record id (`spilled_view`); `record_read` pages the rest.
  This is the claim-check pattern (`research/background-work-limits/azure-claim-check-pattern.md`).
- `poll` does this for `fork` workers (once; the record id is cached on the agent record).
- `Agent::run_agent` does it for CAP answers with `spill_or_truncate`, tag `agent-output`.
- The panel footer shows the size and, for CAP answers, whether it was stored.

## 8. Switches and environment

| Name | Effect |
|---|---|
| `TAI_PLEXER_ACTIVE`, `TAI_PLEXER_SOCKET`, `TAI_PLEXER_PANE` | set by the multiplexer in every pane; the orchestrator's way to find its screen |
| `TAI_ORCH_PANELS=0` | draw nothing (supervision stays on) |

There is no configuration file for the caps; they are constants (section 9).

## 9. Caps and floors (all static)

| Constant | Value |
|---|---|
| `ORCH_PANELS_MAX` | 8 |
| `ORCH_FLUSH_INTERVAL` | 0.025 s |
| `ORCH_MONITOR_INTERVAL` | 1.0 s |
| `ORCH_PENDING_MAX` | 262144 bytes |
| `ORCH_WRITE_MAX` | 32768 bytes |
| `ORCH_LINGER` | 20 s |
| `ORCH_IDLE_WARN` | 60 s |
| `ORCH_IDLE_KILL` | 300 s |
| `ORCH_CANCEL_GRACE` | 5 s |
| `ORCH_TASK_SHOW_MAX` | 300 characters |
| `ORCH_TOOL_ARGS_MAX` | 120 characters |
| `ORCH_TOOL_RESULT_LINES` | 3 |
| `ORCH_TOOL_RESULT_COLS` | 160 |
| `ORCH_LABEL_MAX` | 60 |
| `ORCH_CALL_TIMEOUT_MS` | 2000 |
| `MPLEX_VPANE_MAX` | 8 |
| `MPLEX_VPANE_MIN_ROWS` | 6 |
| `MPLEX_VPANE_MIN_COLS` | 20 |
| `MPLEX_VPANE_MAIN_PERCENT` | 55 |
| `MPLEX_VPANE_OPS_MAX` | 16 |
| `MPLEX_VPANE_WRITE_MAX` | 65536 bytes |
| `MPLEX_VPANE_HEADER_MAX` | 200 characters |
| `MPLEX_VPANE_LEASE` | 30 s |
| `MPLEX_VPANE_ORPHAN_LINGER` | 10 s |
| `MPLEX_VPANE_LINGER_MAX` | 300 s |

The feed panes and the `vpane_*` commands are outside the tables that `mplex_spec_test` compares with the
Python `mplex.md`, because the Python app has none of them.

## 10. Measured behavior

Measured with the real binaries (an attached client on a pty, three agents each streaming 500 tokens/s for
about 1.3 s):

| Measure | Value |
|---|---|
| frames received by the client | 85 in 1.33 s (about 64 per second) |
| interval between frames | median 12.9 ms, p90 25.3 ms (the single 295 ms gap is the scripted pause at the end of the stream) |
| bytes to the terminal | 234 KB (row-diffed frames, three panes changing) |
| writes to the multiplexer | at most 40 per second however many agents stream (unit test: 200 events in far fewer than 120 writes) |

Layout measured on a 120 x 40 window: one agent, main pane 66 columns and 40 rows, agent pane 53 x 40; three
agents, right column heights `[13, 13, 12]`; six agents fit; a seventh gets no panel.

Live check: the real REPL as the main pane (`detz --yolo`), a real model asked to fork two workers: two panels
appeared, streamed the tool calls (`⏺ shell(...)`, `⎿ exit 0`, output), changed to `✓ ... done`, showed the
footer, and left after the linger. A real `@tai:zen` CAP turn did the same (header `cap tai:zen-free`, the
panel filled when the turn ended, section 11).

## 11. Verification

| Test | Covers |
|---|---|
| `tests/orch_test.cpp` | sanitizing, clipping, CRLF, word wrap (inside a word never, across chunks rejoined, long words broken, multi-byte kept), tool and error rendering, the CAP adapter, panel buffering and the byte ceiling, never splitting a character, idle warn and idle cancel (once), cancel on user close, a hidden panel supervised like a shown one, the board over a fake transport (ordering, coalescing, done op, closed pane to cancel), no space, repeated write failure, the panel ceiling, and that this page lists every constant and command |
| `tests/orch_mplex_e2e.py` | a real multiplexer server and the real board: geometry for 1, 3 and 6 panels, content and footers, the space limit, linger and rebalance, a killed pane reported as closed, a failed footer, the main pane exiting |
| `tests/run_all.py` | steps `orch.unit`, `orch.mplex_e2e`, plus all `mplex.*` steps (the multiplexer itself is unchanged for existing behavior) |

Not covered by an automated test: the orphan lease (30 s plus 10 s) and the live REPL runs above.

## 12. Gaps

- tmux and bare terminals get no panels (a tail-file transport would close this).
- Keys typed into a feed pane are ignored; steering an agent from its pane is not built.
- The idle watchdog needs events; a CAP agent that streams nothing for 300 s is cancelled even if it works.
- A hung model request inside an internal worker is bounded only by the worker's own wall clock; the cancel
  flag is read between steps.
- tai as a CAP agent (`--cap-speak`) sends its transcript after the turn, with generic tool names, so a tai
  sub agent over CAP fills its panel at the end of the turn; agents that stream (stream-json, ACP) fill it
  live. An internal `fork` worker is live.
- Panels are not restored after the orchestrator restarts; agent output records have no retention policy yet.
- A bare `tai` is not wrapped in the multiplexer automatically.
- Cancel escalation closes a CAP session from a second thread; a race inside the driver on close is possible.