tai/taicpp/docs/llm.md

# llm: providers, ChatClient, JSON, Python semantics

Port of tai.py 6957-8650 (backends, ModelHealth, ChatClient) plus the helpers they need. Namespace `detz::llm` (`detz::py` for Python semantics, `detz::json` for JSON). The old placeholder API (openai/anthropic providers) is removed: tai.py has no such providers.

| File | tai.py origin |
|---|---|
| `json.cpp` | `json.dumps/loads/raw_decode` byte-exact (ensure_ascii, separators, float repr, JSONDecodeError text, big ints) |
| `py.cpp` | code-point `len/slice`, `strip/lower/upper/split`, `str/repr/int/float`, datetime (`_fromisoformat`, fold-aware naive local), sha1/sha256, `utf8_replace` |
| `helpers.cpp` | `BackendError`, `truncate`, `estimate_tokens`, `clean_history`, `tool_args_*`, `action_signature`, `conversation_tokens` |
| `model_health.cpp` | `ModelHealth`, reward and weight functions |
| `models.cpp` | `split_models`, `parse_backends`, `fetch_model_ids`, `fetch_pollinations_free`, `is_free_*`, `prefer_muse`, `openrouter_free_tier` |
| `cli_backends.cpp` | opencode/grok/claude/codex/gemini prompts, parsers, runners, health, `probe_backend`, `opencode_headers` |
| `muse.cpp` | muse session map, flock+thread locks, `muse_run` streaming runner |
| `chat_client.cpp` | `ChatClient` (candidates, ordering, pins, retry rounds, single shot, streaming, `update_models`, `models_report`) |
| `http_transport.cpp` | libcurl transport with an injectable fake (`set_transport`) |
| `proc.cpp` | `subprocess.run/Popen`, `shutil.which` |
| `ports.h` | `LlmConfig`, `LogSink`, `BoxPort` |
| `unicode_data.cpp` (generated) | case maps, whitespace, digits from Python |

`--update-models` is `agent::update_models_command(config)`.

Gaps: see docs/agent.md. Unit tests never touch the network; the libcurl transport itself is only exercised by compile/link, not against a live server.

## Failover visibility and CLI caps

- Each failed candidate prints `x provider/model failed after N.Ns: reason` on stderr immediately (`ChatClient::announce_failure`), in addition to the debug log event.
- `muse_argv` always passes `--reasoning-effort low` (`kMuseReasoningEffort`), `--preset empty`, `--no-foreign-personal-context`. Measured on a 16 KB tai prompt: 36.5 s default, 2.8 s low, 2.4 s low + empty preset. Gap: only one prompt was measured, on one machine.
- `ordered_candidates` drops CLI candidates whose circuit is open (3 consecutive failures, 300 s) unless none remain.
- opencode runs are capped at `kCliFallbackTimeout` (30 s).