tai/taicpp/docs/foundation.md

# foundation: store, config, session, repl, plexer

Port of the vault, settings, debug log, conversation persistence, chat REPL and
plexer controllers of `tai.py` (spec files: `settings.md`, `logging.md`,
`uiflow.md`). Behaviour is the contract: every operation below is replayed
through both implementations by `tests/foundation_parity.py` and diffed.

## Layout

| Directory | Classes (one header + source each) |
|---|---|
| `src/store` | `Store` (store.cpp, store_vault.cpp, store_files.cpp, store_settings.cpp, store_mcp.cpp), `SqlDb`, `Schema`, `DebugLog`, `Spill`, `Json`, `Fts`, `TagRules`, `Patterns`, `VaultFiles`, `Clock`, `Utf8`, `Sha256`, `SealPort`/`NullSeal`, errors |
| `src/config` | `Config`, `SettingsCatalog`, `SettingsRules`, `BootSettings`, `Prompts`, constants |
| `src/session` | `ShellContext` (tai.py `ShellSession`), `Conversation` (persistence half of `Agent`), `DirDefaults`, `Term`, `Md5` |
| `src/repl` | `Repl`, `ReplHost`, `ReplStyle`, `LineInput`, `Interrupt`, `Boot` |
| `src/plexer` | `PlexerController`, `NativePlexerController`, `TmuxPlexerController`, `PlexerControllerFactory`, `FlowValidator`, `PlexerFlow`, `PlexerError`, `Plexer` (constants), `ProcRunner` |

## Where history lives and how it is keyed (the pane question)

tai.py does not read `TMUX_PANE` or `TAI_PLEXER_PANE` to key a conversation.
The key is the **shell context id** (`ShellSession.current`), derived from the
nearest shell ancestor of the process:

    YYYYMMDD-HHMMSS-p<PID>-<BOOT8>     start time of that shell, its pid, 8 hex of the kernel boot id

A terminal pane is exactly one long-lived shell, so quitting and restarting tai
in the same pane (tmux, the native plexer, any terminal tab) resolves to the
same id and resumes the same messages; another pane has another shell and a
fresh history. `boot()` exports the id as `TAI_SESSION`; a plain word in
`TAI_SESSION` pins one shared context for every shell.

Files, all under `<TAI_HOME>/profiles`:

| File | Content |
|---|---|
| `<profile>.session.<sid>.json` | user/assistant/tool messages of the main bot, system messages never stored |
| `<profile>.bot.<bot>.session.<sid>.json` | same for another bot |
| `<profile>.turn.<sid>.json` | `{"open", "goal", "steps", "tools", "updated"}` of the last turn (`--continue`) |
| `<profile>.sys.md`, `<profile>.bots.json`, `<profile>.bot.<bot>.sys.md` | system messages and the bot registry |

At most `FILES_CAP` (20) scoped files per profile and bot are kept; the oldest
are pruned on save. Load: `Conversation::switch_profile` (also run by the
constructor) calls `Store::load_session(profile, "main", ctx_id)`. Save:
`Conversation::close_turn`/`checkpoint` and the REPL after every turn and on
exit. A session file that does not parse is renamed `*.corrupt-<epoch>`.

The `shell_sessions` table records every shell id with its parent chain
(`ShellContext::register_in`, `/ctx`).

## Symbol map

| tai.py | C++ |
|---|---|
| `SCHEMA`, `SCHEMA_INDEXES`, `ensure_schema`, `migrate_profiles`, `rebuild_with_profile`, `table_pk_columns`, `ensure_column`, `setup_fts` | `Schema` |
| `Store.*` (history, events, search, secrets, tags, graph, records, chunks, audit, schedules, files, bots, settings, mcp, dir defaults) | `Store` same method names |
| `fts_escape`, `fts_match*`, `fts_rank` | `Fts` |
| `normalize_tag(s)`, `singular_noun`, `tag_variants`, `parse_node_id` | `TagRules` |
| `PROFILE_RE`, `BOT_RE`, `SECRET_RE`, `MCP_SERVER_RE` | `Patterns` |
| `atomic_write`, `sealed_write`, `quarantine` | `VaultFiles` |
| `now_iso`, `uuid4().hex` | `Clock` |
| `spill_output`, `spilled_view`, `spill_or_truncate`, `truncate`, `chunk_lines` | `Spill` |
| `DebugLog`, `LOG`, `--debug`, `--debug-format` | `DebugLog`, `DebugLog::Global()`, `Boot::configure_debug` |
| `SETTINGS_CATALOG`, `CONFIG_SETTING_FIELDS`, `SETTING_EXPORT_KEYS` | `SettingsCatalog()`, `ConfigSettingFields()`, `SettingExportKeys()` |
| `normalize_setting_value`, `parse_setting_env`, `parse_setting_stored`, `validate_setting_scope` | `SettingsRules` |
| `get_or_create_settings`, `get_all_settings`, `upsert_settings` | `Store` |
| `export_settings_to_env`, `boot_settings`, `Config.apply_store_settings` | `BootSettings::ExportToEnv`, `BootSettings::Boot`, `Config::ApplyStoreSettings` |
| `Config`, `parse_backends`, `split_models`, `known_model_ids` | `Config::Build`/`FromEnv`, `ParseBackends`, `SplitModels`, `KnownModelIds` |
| `DEFAULT_SYSTEM` and the other prompt texts, `orientation_block`, `capability_block` | `Prompts` (generated byte for byte from tai.py) |
| `ShellSession` | `session::ShellContext` |
| `Agent.switch_profile`, `switch_bot`, `apply_system`, `reset_history`, `update_system`, `checkpoint`, `close_turn`, `mark_turn_open` | `session::Conversation` |
| `apply_dir_defaults` | `session::DirDefaults::apply` |
| `boot` (config, seal, store, settings, shell context, agent) | `repl::Boot::open` |
| `banner`, `resume_turn`, `repl`, one-shot tail of `main`, `handle_command` for `/quit /exit /help /profiles /bots /bot /profile /ctx /clear` | `repl::Repl` |
| `Glow.statusbar`, `Glow.prompt_text`, banner glow strings, `strip_ansi`, `truncate_ansi` | `repl::ReplStyle` |
| `PlexerError`, `plexer_dir`, `PLEXER_*` constants | `plexer::PlexerError`, `plexer::Plexer` |
| `PlexerController`, `NativePlexerController`, `TmuxPlexerController`, `PlexerControllerFactory.detect` | same names in `detz::plexer` |
| `validate_flow_*`, `PlexerFlow` | `plexer::FlowValidator`, `plexer::PlexerFlow` |

The python `Seal` is not part of this area: `Store` takes a `store::SealPort`
(`enabled`, `lock`, `unlock`, `safe_unlock`, `kPrefix = "tai1$"`). `NullSeal`
is the python "seal disabled" path.

## Deliberate differences

- Python dicts returned by `Store` are `store::Json` objects (insertion ordered); settings rows are `SettingRow`.
- `ShellSession` is named `ShellContext` because `detz::shell::ShellSession` is the command runner.
- A bare `/` at the prompt prints "unknown command, try /help" (tai.py raises IndexError).
- Without readline there is no line editing, arrow-key history or tab completion; `LineInput` is a plain line reader.
- `PlexerController::ask` takes the timeout as `Json` so an int stays an int on the wire like in python; a `double` overload exists.
- Unbounded python bigints in JSON become doubles.

## Gaps (stated, not hidden)

| Gap | Effect |
|---|---|
| No `Seal` implementation here (`SealPort` only) | `Boot::open` uses `NullSeal` when `TAI_PASSPHRASE` is unset (vault stored in plain text, which python seals on its next start); an explicit `TAI_PASSPHRASE` without a `seal_factory` is refused with a `SealError`; a vault sealed by python is refused ("memory is sealed"). `rotate_seal` and the default-key upgrade are not ported |
| readline (history recall, `complete_input`) | typed input is still recorded in `input_history` |
| `start_scheduler`, `running_agents`, `probe_backend` | host hooks (`ReplHost::on_start`, `running_agents`, `backend_status`) |
| `ensure_self_backup`, `snapshot_self`, `do_release` | script-specific, belong to release tooling |
| `DebugLog` uncaught-exception hook | `std::set_terminate` analog; logged line carries the C++ type name, python's carries the python one |
| Glow banner on a real tty | pieces are compared with python functions, the composed screen is not |

## Caps (all static)

`FILES_CAP` 20 scoped files, `ROWS_CAP` 1000 `shell_sessions`, `CHAIN_CAP` 64 process levels,
`INPUT_HISTORY_CAP` 1000 rows of 4000 chars, `LOG_DB_KEEP` 20000 log rows, `LOG_META_MAX` 16384 bytes,
`MEM_EVENTS_CAP` 10000 events in the in-memory index, `MCP_MAX_SERVERS` 32, `DIR_DEFAULTS_CAP` 512,
`SPILL_LIMIT` 6000 chars, `AUDIT_MAX_CHARS` 20000, `FILE_RECORD_MAX` 50000, flow limits in `Plexer`.

## Tests

Run from `taicpp/`: `make -f tests/foundation.mk test` (needs only g++, sqlite3, python3, tmux for the tmux cases).

| Test | What it proves |
|---|---|
| `tests/foundation_parity.py` + `foundation_probe` | one scenario replayed through tai.py and the C++ store: results, every table, the FTS index, the schema DDL and every profile file identical (timestamps and random ids masked). Pure functions (JSON, tags, settings rules, prompts, chunking, shell context against a fake /proc, flow validator and driver, Glow pieces), DebugLog sinks, REPL slash commands, native plexer against a scripted backend, tmux against a private server, and vault portability (each implementation reads the vault the other wrote) |
| `tests/repl_persist.py` + `repl_probe` | the history bug: the REPL started repeatedly under a long-lived shell per pane resumes the same session file, another pane stays separate (also in real tmux panes), Ctrl-C mid-turn, an uncaught exception reaching the debug log, a pinned `TAI_SESSION` is shared, an interrupted turn is resumed by `--continue`; every assertion reads the vault |
| `tests/tmux_ask.py` | tmux `ask` (menu, text popup, message, cancel, timeout) with a real client attached through a pty, both implementations, same replies |
| `tests/foundation_test.cpp` | hash vectors, UTF-8 helpers, seal-enabled store paths with a stand-in seal, boot settings, dir defaults, turn files, plexer detection, line input |

`repl_probe` runs the real `Repl`, `Conversation` and `Store` with a scripted
host in place of the model loop; no model is called.