diff --git a/taicpp/docs/agent.md b/taicpp/docs/agent.md new file mode 100644 index 0000000..e44b886 --- /dev/null +++ b/taicpp/docs/agent.md @@ -0,0 +1,48 @@ +# agent: Agent core, Tools, goal verifier, RAG stats, scheduler + +Literal port of tai.py 1.44.2 (lines 9016-9335, 17071-20640, 40391-40460). Namespace `detz::agent`, sources in `src/agent`. + +## Layout + +| File | tai.py origin | +|---|---| +| `data.cpp` (generated) | `TOOL_SCHEMAS`, `TOOL_TAGS`, `CORE_TOOLS`, `LAZY_NAME_STOP`, prompt texts. Regenerate: `python3 -I src/agent/gen/gen_data.py > src/agent/data.cpp` | +| `data_html.cpp` (generated) | `html.entities.html5` for `html.unescape` | +| `agent.cpp/.h` | `Agent`, `select_tools`, `tool_catalog`, `compact_messages`, `resume_turn`, `update_models_command`, `default_runner`, mentions | +| `tools_*.cpp`, `tools.h` | `Tools` (61 handlers, `dispatch`, secrets, audit, box helpers) | +| `goal.cpp` | `Denied`, `AgentsFull`, `Approval`, `GoalFailure`, `GoalVerdict`, `GoalVerifier` | +| `rag.cpp` | `RetrievalEval`, `RagStats` | +| `registry.cpp` | `spawn_agent`, `poll_agent`, `list_agents`, `clear_agents`, `running_agents` | +| `scheduler.cpp` | `note_schedule_result`, `watch_schedule`, `schedule_running`, `scheduler_tick`, `run_scheduler_loop`/`start_scheduler` (class `Scheduler`) | +| `text.cpp` | `truncate`-family helpers, tags, `chunk_lines`, `spill_*`, tmux/history helpers, `html.unescape`, schedule helpers | +| `tui_control.cpp` | `grr_control_path`, `grr_control_call` (client only) | +| `zipfile.cpp` | `zipfile.extractall` (stored + deflate) for `install_skill` | +| `ports.h` | the narrow interfaces to everything this module does not own | + +## Ports (declared here, implemented elsewhere) + +`StorePort` (mirrors Python `Store` method names), `RuntimePort` (worker store), `ShellPort` (run_live, LiveStream, LaneBoard, box), `UiPort` (Spinner, Glow, markdown, diff; optional, plain fallbacks exist), `McpPort`, `SkillsPort`, `PlexerPort`, `TuiControlPort` (optional, built-in client otherwise), `OpsPort` (install, release, sysinfo, speech), `SchedulerDbPort`. Unwired required ports raise `PortNotWired`, shown to the model as `error: port not wired: X`. Plexer unwired means "no session", exactly like no tmux. + +## Behaviour notes + +- All string lengths and slices are code points (`detz::py`, `src/llm/py.h`). +- Integers beyond int64 clamp in `int()` conversions (Python is unbounded); JSON keeps big ints verbatim. +- `html.unescape`, `str.lower/upper` (including final sigma) use tables generated from Python. + +## Verification (what was actually run) + +- `tests/agent_diff.py`: 7601 cases per seed (seeds 1-4 run, 0 mismatches). Includes the full `json.dumps(TOOL_SCHEMAS)` byte comparison, `select_tools` on 24 prompts plus fuzz, `tool_catalog`, all prompt texts, goal verdicts, CLI output parsers, history cleanup, tag helpers, html/url helpers, schedule helpers, candidate ordering of `ChatClient`. +- `tests/agent_e2e.py`: 43 scripted conversations through the real Agent loop in both programs (messages, HTTP request bodies, store events, audit rows identical). Covers file tools, shell, secrets, schedules, bots/mentions, compaction, retries, 400 tool fallback, ask_user, approval deny, verify loop, remember, web_fetch/search, fork/poll, streaming quirks, deadline. +- `tests/agent_sched.py`: `scheduler_tick` against a real sqlite vault. +- Build for these tests: compile `src/llm/*.cpp src/agent/*.cpp tests/agent_*.cpp` with `-lcurl -lpthread` (see wiring doc). + +## Not verified / gaps (honest list) + +- Tools needing FTS or other modules were only checked on argument validation paths: `search`, `record_search`, `graph_*`, `tags`, `recall`, `audit` search, `export_terminal_log`. +- Port-backed tools (MCP, skills install/create, plexer, install, release, sysinfo, speak, listen, sandbox `box_*` paths, LaneBoard) are ported but no test drives a real implementation: the ports are fakes or unwired. +- `RagStats`/`RetrievalEval` compile and follow the Python line by line but have no differential test (needs a real vault). +- Interactive tty paths (`ask_approval` console branch with `Denied` guidance, `shell_ask` getpass) are unexercised. +- urllib error strings are approximated from libcurl codes (`` etc.). +- Muse session map/locking and CLI runners (`muse_run`, grok, claude, codex, gemini, opencode) are ported; only their output parsers are differentially tested, the process runners were not run against the real CLIs. +- `zipfile`: bzip2/lzma members unsupported. +- `str.lower` final-sigma uses derived property sets; `repr` printable test is approximated for exotic code points. diff --git a/taicpp/docs/llm.md b/taicpp/docs/llm.md new file mode 100644 index 0000000..1f7d261 --- /dev/null +++ b/taicpp/docs/llm.md @@ -0,0 +1,22 @@ +# llm: providers, ChatClient, JSON, Python semantics + +Port of tai.py 6957-8650 (backends, ModelHealth, ChatClient) plus the helpers they need. Namespace `detz::llm` (`detz::py` for Python semantics, `detz::json` for JSON). The old placeholder API (openai/anthropic providers) is removed: tai.py has no such providers. + +| File | tai.py origin | +|---|---| +| `json.cpp` | `json.dumps/loads/raw_decode` byte-exact (ensure_ascii, separators, float repr, JSONDecodeError text, big ints) | +| `py.cpp` | code-point `len/slice`, `strip/lower/upper/split`, `str/repr/int/float`, datetime (`_fromisoformat`, fold-aware naive local), sha1/sha256, `utf8_replace` | +| `helpers.cpp` | `BackendError`, `truncate`, `estimate_tokens`, `clean_history`, `tool_args_*`, `action_signature`, `conversation_tokens` | +| `model_health.cpp` | `ModelHealth`, reward and weight functions | +| `models.cpp` | `split_models`, `parse_backends`, `fetch_model_ids`, `fetch_pollinations_free`, `is_free_*`, `prefer_muse`, `openrouter_free_tier` | +| `cli_backends.cpp` | opencode/grok/claude/codex/gemini prompts, parsers, runners, health, `probe_backend`, `opencode_headers` | +| `muse.cpp` | muse session map, flock+thread locks, `muse_run` streaming runner | +| `chat_client.cpp` | `ChatClient` (candidates, ordering, pins, retry rounds, single shot, streaming, `update_models`, `models_report`) | +| `http_transport.cpp` | libcurl transport with an injectable fake (`set_transport`) | +| `proc.cpp` | `subprocess.run/Popen`, `shutil.which` | +| `ports.h` | `LlmConfig`, `LogSink`, `BoxPort` | +| `unicode_data.cpp` (generated) | case maps, whitespace, digits from Python | + +`--update-models` is `agent::update_models_command(config)`. + +Gaps: see docs/agent.md. Unit tests never touch the network; the libcurl transport itself is only exercised by compile/link, not against a live server. diff --git a/taicpp/docs/wiring/agent.md b/taicpp/docs/wiring/agent.md new file mode 100644 index 0000000..8e46c87 --- /dev/null +++ b/taicpp/docs/wiring/agent.md @@ -0,0 +1,24 @@ +# Wiring for the agent core (integrator checklist) + +I did not touch main.cpp, Makefile, src/cli, src/store, src/config, PARITY.md. + +## Build +- Add to the link line: `-lcurl -lpthread` (already `-lcurl`; `-lpthread` for threads). Static target needs nothing extra (zlib not used). +- `src/llm/py.cpp` etc. include `"unicode_data.h"`, `"json.h"` relatively; no new include dirs needed beyond the existing `-I` per src dir. Note `consts.h`, `text.h` exist in several src dirs: my files include their own directory first. +- Removed legacy files: `src/agent/{exec,loop}.{h,cpp}`, `src/agent/tools.cpp`, `src/llm/llm.{h,cpp}`. `main.cpp` (`ToolRegistry`, `RegisterAll`, `RunTurn`) and `tests/test_main.cpp` (`llm::ChatClient("openai", ...)`, `ModelHealth`, `Opencode`, `feed_sse`, `provider_info`) use that removed placeholder API: replace the `run` command with `agent::Agent` and drop those test sections. `src/agent/agent.mk` and `src/llm/llm.mk` are stale (unused by the Makefile). +- Test targets (my files): `tests/agent_diff.cpp`, `agent_e2e.cpp`, `agent_sched.cpp` (+ `agent_memstore.h`) built with all of `src/llm/*.cpp src/agent/*.cpp`, driven by `tests/agent_diff.py`, `agent_e2e.py`, `agent_sched.py` (binaries expected at `build/agent_diff`, `build/agent_e2e`, `build/agent_sched`). + +## Hooks to apply at startup +1. `agent::ports().store` is per-Agent (pass `StorePort*` to `Agent`); set the rest once: `ports().runtime, shell, ui, mcp, skills, plexer, tui (optional), ops, scheduler_db`. +2. `llm::set_log_sink(&debuglog)` (DebugLog adapter: `enabled()`, `event(level, area, kind, message, profile, bot, meta)`). +3. `llm::set_box_port(&sandbox)` for the opencode-in-box fallback. +4. `llm::ChatClient::set_tool_catalog(natives, core)` with the 61 tool names and `data::CoreTools()` so `log_completion` counts match. +5. Build `AgentConfig` from Config (fields in `agent_config.h`, `llm/ports.h::LlmConfig`); `ChatClient` and `Agent` keep a reference, so mutate that same object when settings change. +6. `--update-models` -> `agent::update_models_command(cfg)`; `--resume` -> `agent::resume_turn(agent)`; one-shot -> `Agent::run_verified`; scheduler mode -> `agent::Scheduler(cfg, db).start()`. +7. REPL/TUI set `agent.prompter`, `agent.stream_factory`, `agent.on_phase` as in Python; `Agent::total_step_cap()` is the live-patchable `Consts.TOTAL_STEP_CAP`. + +## Port contracts the other owners must satisfy (see `src/agent/ports.h`) +- StorePort: same semantics as the Python Store methods of the same name; rows are `json::Value` objects with the Python key names; `get_chunks_by_ids` returns an object keyed by the id as a string; `sql_rows` is a read-only SELECT for RagStats. +- ShellPort: `run_live_shell` = `run_live(command, shell=True, cwd, env=environ+extra_env, stream, ask)`; `shell_target_paths` = tai.py `shell_target_paths`. +- McpPort, SkillsPort, OpsPort, PlexerPort: mirror the named tai.py functions (comments in ports.h). +- UiPort is optional; without it replies print raw and the non-Glow `┌─`/`└─` call frames are used.