merge: worktree-agent-a7652b893d520b1a0

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

taicpp/docs/agent.md

@@ -0,0 +1,48 @@
+# agent: Agent core, Tools, goal verifier, RAG stats, scheduler
+
+Literal port of tai.py 1.44.2 (lines 9016-9335, 17071-20640, 40391-40460). Namespace `detz::agent`, sources in `src/agent`.
+
+## Layout
+
+| File | tai.py origin |
+|---|---|
+| `data.cpp` (generated) | `TOOL_SCHEMAS`, `TOOL_TAGS`, `CORE_TOOLS`, `LAZY_NAME_STOP`, prompt texts. Regenerate: `python3 -I src/agent/gen/gen_data.py > src/agent/data.cpp` |
+| `data_html.cpp` (generated) | `html.entities.html5` for `html.unescape` |
+| `agent.cpp/.h` | `Agent`, `select_tools`, `tool_catalog`, `compact_messages`, `resume_turn`, `update_models_command`, `default_runner`, mentions |
+| `tools_*.cpp`, `tools.h` | `Tools` (61 handlers, `dispatch`, secrets, audit, box helpers) |
+| `goal.cpp` | `Denied`, `AgentsFull`, `Approval`, `GoalFailure`, `GoalVerdict`, `GoalVerifier` |
+| `rag.cpp` | `RetrievalEval`, `RagStats` |
+| `registry.cpp` | `spawn_agent`, `poll_agent`, `list_agents`, `clear_agents`, `running_agents` |
+| `scheduler.cpp` | `note_schedule_result`, `watch_schedule`, `schedule_running`, `scheduler_tick`, `run_scheduler_loop`/`start_scheduler` (class `Scheduler`) |
+| `text.cpp` | `truncate`-family helpers, tags, `chunk_lines`, `spill_*`, tmux/history helpers, `html.unescape`, schedule helpers |
+| `tui_control.cpp` | `grr_control_path`, `grr_control_call` (client only) |
+| `zipfile.cpp` | `zipfile.extractall` (stored + deflate) for `install_skill` |
+| `ports.h` | the narrow interfaces to everything this module does not own |
+
+## Ports (declared here, implemented elsewhere)
+
+`StorePort` (mirrors Python `Store` method names), `RuntimePort` (worker store), `ShellPort` (run_live, LiveStream, LaneBoard, box), `UiPort` (Spinner, Glow, markdown, diff; optional, plain fallbacks exist), `McpPort`, `SkillsPort`, `PlexerPort`, `TuiControlPort` (optional, built-in client otherwise), `OpsPort` (install, release, sysinfo, speech), `SchedulerDbPort`. Unwired required ports raise `PortNotWired`, shown to the model as `error: port not wired: X`. Plexer unwired means "no session", exactly like no tmux.
+
+## Behaviour notes
+
+- All string lengths and slices are code points (`detz::py`, `src/llm/py.h`).
+- Integers beyond int64 clamp in `int()` conversions (Python is unbounded); JSON keeps big ints verbatim.
+- `html.unescape`, `str.lower/upper` (including final sigma) use tables generated from Python.
+
+## Verification (what was actually run)
+
+- `tests/agent_diff.py`: 7601 cases per seed (seeds 1-4 run, 0 mismatches). Includes the full `json.dumps(TOOL_SCHEMAS)` byte comparison, `select_tools` on 24 prompts plus fuzz, `tool_catalog`, all prompt texts, goal verdicts, CLI output parsers, history cleanup, tag helpers, html/url helpers, schedule helpers, candidate ordering of `ChatClient`.
+- `tests/agent_e2e.py`: 43 scripted conversations through the real Agent loop in both programs (messages, HTTP request bodies, store events, audit rows identical). Covers file tools, shell, secrets, schedules, bots/mentions, compaction, retries, 400 tool fallback, ask_user, approval deny, verify loop, remember, web_fetch/search, fork/poll, streaming quirks, deadline.
+- `tests/agent_sched.py`: `scheduler_tick` against a real sqlite vault.
+- Build for these tests: compile `src/llm/*.cpp src/agent/*.cpp tests/agent_*.cpp` with `-lcurl -lpthread` (see wiring doc).
+
+## Not verified / gaps (honest list)
+
+- Tools needing FTS or other modules were only checked on argument validation paths: `search`, `record_search`, `graph_*`, `tags`, `recall`, `audit` search, `export_terminal_log`.
+- Port-backed tools (MCP, skills install/create, plexer, install, release, sysinfo, speak, listen, sandbox `box_*` paths, LaneBoard) are ported but no test drives a real implementation: the ports are fakes or unwired.
+- `RagStats`/`RetrievalEval` compile and follow the Python line by line but have no differential test (needs a real vault).
+- Interactive tty paths (`ask_approval` console branch with `Denied` guidance, `shell_ask` getpass) are unexercised.
+- urllib error strings are approximated from libcurl codes (`<urlopen error [Errno 111] Connection refused>` etc.).
+- Muse session map/locking and CLI runners (`muse_run`, grok, claude, codex, gemini, opencode) are ported; only their output parsers are differentially tested, the process runners were not run against the real CLIs.
+- `zipfile`: bzip2/lzma members unsupported.
+- `str.lower` final-sigma uses derived property sets; `repr` printable test is approximated for exotic code points.

taicpp/docs/llm.md

@@ -0,0 +1,22 @@
+# llm: providers, ChatClient, JSON, Python semantics
+
+Port of tai.py 6957-8650 (backends, ModelHealth, ChatClient) plus the helpers they need. Namespace `detz::llm` (`detz::py` for Python semantics, `detz::json` for JSON). The old placeholder API (openai/anthropic providers) is removed: tai.py has no such providers.
+
+| File | tai.py origin |
+|---|---|
+| `json.cpp` | `json.dumps/loads/raw_decode` byte-exact (ensure_ascii, separators, float repr, JSONDecodeError text, big ints) |
+| `py.cpp` | code-point `len/slice`, `strip/lower/upper/split`, `str/repr/int/float`, datetime (`_fromisoformat`, fold-aware naive local), sha1/sha256, `utf8_replace` |
+| `helpers.cpp` | `BackendError`, `truncate`, `estimate_tokens`, `clean_history`, `tool_args_*`, `action_signature`, `conversation_tokens` |
+| `model_health.cpp` | `ModelHealth`, reward and weight functions |
+| `models.cpp` | `split_models`, `parse_backends`, `fetch_model_ids`, `fetch_pollinations_free`, `is_free_*`, `prefer_muse`, `openrouter_free_tier` |
+| `cli_backends.cpp` | opencode/grok/claude/codex/gemini prompts, parsers, runners, health, `probe_backend`, `opencode_headers` |
+| `muse.cpp` | muse session map, flock+thread locks, `muse_run` streaming runner |
+| `chat_client.cpp` | `ChatClient` (candidates, ordering, pins, retry rounds, single shot, streaming, `update_models`, `models_report`) |
+| `http_transport.cpp` | libcurl transport with an injectable fake (`set_transport`) |
+| `proc.cpp` | `subprocess.run/Popen`, `shutil.which` |
+| `ports.h` | `LlmConfig`, `LogSink`, `BoxPort` |
+| `unicode_data.cpp` (generated) | case maps, whitespace, digits from Python |
+
+`--update-models` is `agent::update_models_command(config)`.
+
+Gaps: see docs/agent.md. Unit tests never touch the network; the libcurl transport itself is only exercised by compile/link, not against a live server.

taicpp/docs/wiring/agent.md

@@ -0,0 +1,24 @@
+# Wiring for the agent core (integrator checklist)
+
+I did not touch main.cpp, Makefile, src/cli, src/store, src/config, PARITY.md.
+
+## Build
+- Add to the link line: `-lcurl -lpthread` (already `-lcurl`; `-lpthread` for threads). Static target needs nothing extra (zlib not used).
+- `src/llm/py.cpp` etc. include `"unicode_data.h"`, `"json.h"` relatively; no new include dirs needed beyond the existing `-I` per src dir. Note `consts.h`, `text.h` exist in several src dirs: my files include their own directory first.
+- Removed legacy files: `src/agent/{exec,loop}.{h,cpp}`, `src/agent/tools.cpp`, `src/llm/llm.{h,cpp}`. `main.cpp` (`ToolRegistry`, `RegisterAll`, `RunTurn`) and `tests/test_main.cpp` (`llm::ChatClient("openai", ...)`, `ModelHealth`, `Opencode`, `feed_sse`, `provider_info`) use that removed placeholder API: replace the `run` command with `agent::Agent` and drop those test sections. `src/agent/agent.mk` and `src/llm/llm.mk` are stale (unused by the Makefile).
+- Test targets (my files): `tests/agent_diff.cpp`, `agent_e2e.cpp`, `agent_sched.cpp` (+ `agent_memstore.h`) built with all of `src/llm/*.cpp src/agent/*.cpp`, driven by `tests/agent_diff.py`, `agent_e2e.py`, `agent_sched.py` (binaries expected at `build/agent_diff`, `build/agent_e2e`, `build/agent_sched`).
+
+## Hooks to apply at startup
+1. `agent::ports().store` is per-Agent (pass `StorePort*` to `Agent`); set the rest once: `ports().runtime, shell, ui, mcp, skills, plexer, tui (optional), ops, scheduler_db`.
+2. `llm::set_log_sink(&debuglog)` (DebugLog adapter: `enabled()`, `event(level, area, kind, message, profile, bot, meta)`).
+3. `llm::set_box_port(&sandbox)` for the opencode-in-box fallback.
+4. `llm::ChatClient::set_tool_catalog(natives, core)` with the 61 tool names and `data::CoreTools()` so `log_completion` counts match.
+5. Build `AgentConfig` from Config (fields in `agent_config.h`, `llm/ports.h::LlmConfig`); `ChatClient` and `Agent` keep a reference, so mutate that same object when settings change.
+6. `--update-models` -> `agent::update_models_command(cfg)`; `--resume` -> `agent::resume_turn(agent)`; one-shot -> `Agent::run_verified`; scheduler mode -> `agent::Scheduler(cfg, db).start()`.
+7. REPL/TUI set `agent.prompter`, `agent.stream_factory`, `agent.on_phase` as in Python; `Agent::total_step_cap()` is the live-patchable `Consts.TOTAL_STEP_CAP`.
+
+## Port contracts the other owners must satisfy (see `src/agent/ports.h`)
+- StorePort: same semantics as the Python Store methods of the same name; rows are `json::Value` objects with the Python key names; `get_chunks_by_ids` returns an object keyed by the id as a string; `sql_rows` is a read-only SELECT for RagStats.
+- ShellPort: `run_live_shell` = `run_live(command, shell=True, cwd, env=environ+extra_env, stream, ask)`; `shell_target_paths` = tai.py `shell_target_paths`.
+- McpPort, SkillsPort, OpsPort, PlexerPort: mirror the named tai.py functions (comments in ports.h).
+- UiPort is optional; without it replies print raw and the non-Glow `┌─`/`└─` call frames are used.