tai/tests/README.md

# Integration tests

This is where **integration tests** live — tests that drive the real `tai.py` binary as a
subprocess and check its actual behavior. They are a different thing from the project's
**unit tests** (`test_tai.py`, `test_seal.py`, at the repository top level, `unittest`-based,
import `tai.py` as a module, fast, no subprocess). Both exist on purpose, side by side; see
[AGENTS.md](../AGENTS.md#testing)'s "Testing" section for which convention applies to which kind of test, and do
not move one style into the other's location or naming.

**This is the only way integration tests get written for this project from now on.** A new
integration test is a new file under `tests/`, built on [tai_test.py](../tai_test.py) (the framework, at the
repository root — read its module docstring before using it). Don't hand-roll a fresh
`subprocess`/`asyncio` harness per test file; extend `tai_test.TestCase` instead, so every
integration test behaves consistently (same timeout handling, same process cleanup, same
logging, same CLI flags) without each one having to get those details right from scratch.

## The core rule: test with the real AI, and accept the flakes

An integration test for this project is not done proving something until it has made the
real `tai.py` binary make a real decision through a real call to a real LLM backend. Not a
mock of the model, not a stub tool, not a direct write into the vault that *simulates* what
the agent would have done. If a test's "proof" could have been produced by a human directly
editing the database instead of the agent ever running, it isn't an integration test for an
AI agent — it's a database test wearing an integration test's name.

This isn't a style preference; it came directly from the first time this rule was followed
end to end, while building `tests/rag.py`:

- The CLI's own terminal rendering turned out to hide the actual tool-call return value
  behind a `└─ 78 chars · 1ms` summary. A test that stopped at the subprocess's stdout — the
  "deterministic-looking" layer — would have silently reported every real save as a failure.
  Checking the vault's actual state caught this; a mocked tool call would never have surfaced
  the discrepancy, because a mock doesn't render a UI.
- With a real model in the loop, `record_save` got called **twice for one file** and **three
  times for another** in the same run, once in a cross-call way that briefly made search
  results from an earlier file leak into a later one. That is a genuine, observed property of
  how the agent actually behaves under conversational history — not a bug anyone would have
  thought to inject into a scripted fake, and not something a test that never touches the real
  model could ever have found.
- The final, strong per-file proof (exact content fidelity, the fingerprint surviving
  chunking under the correct record, `/search` naming the correct `mem:` id) is only
  meaningful *because* the thing being measured — the model's own tool call — was real. A
  fidelity score of 1.00 against a value the test itself inserted would prove nothing.

So: **we do not write integration tests that fake the AI's part to get a clean, deterministic
result. We accept that free, rotating backends (see root [README.md](../README.md#backends)'s "Backends" section)
make LLM-driven steps genuinely flaky, and we treat that flakiness as the cost of actually
testing the thing we built, not as a defect to engineer around by testing something easier
instead.** The practical consequences of this — report ratios instead of demanding
all-or-nothing, prefer a deterministic *plumbing* check (a REPL slash command) only to prove
the harness itself works, never as a replacement for the AI-driven test of the actual feature
— are spelled out below in "LLM-driven steps are probabilistic."

## How to add a new integration test

1. Create `tests/<topic>.py` (lowercase, no `test_` prefix — that prefix is reserved for the
   top-level unit-test files so the two kinds are never confused by name alone).
2. `sys.path.insert(0, str(Path(__file__).resolve().parent.parent))` then `from tai_test import TestCase, ...` — see [tests/rag.py](rag.py) for the exact boilerplate.
3. Subclass `TestCase` for each distinct scenario. Required: `name` (a class attribute),
   `async def run(self)` (drive the process via `self.spawn(...)`, return whatever `verify()`
   needs), `async def verify(self, result)` (raise `TestAssertionFailed` with a clear message
   on failure). Optional: `async def setup(self)` / `async def teardown(self)`, `timeout`,
   `mode` ("pipe" or "pty" — see [tai_test.py](../tai_test.py)'s docstring for when each applies).
4. Collect your cases into a module-level `ALL_CASES = [...]` list and end the file with:
   ```python
   if __name__ == "__main__":
       sys.exit(main_cli(ALL_CASES, default_log_dir=str(Path(__file__).resolve().parent / "logs")))
   ```
5. Run it: `python3 tests/<topic>.py` (all cases), `--filter <substring>` (a subset),
   `--interactive` (also forward your own live typing into the running process — useful for
   watching a flaky step happen and nudging it by hand), `--timeout <seconds>` (override).

## Isolation — always use a throwaway `TAI_HOME`

Every test that touches the vault must set `TAI_HOME` (and `TAI_PASSPHRASE=""`, so the vault
stays plaintext and directly inspectable) to a fresh `tempfile.mkdtemp()` in `setup()`, and
remove it in `teardown()`. Never point a test at the real `~/.tai` — there is no "are you
sure" here, a test that forgets this will read and write the actual user's memory, secrets,
and schedules. `tests/rag.py`'s two test cases are the reference example.

## Verify against state, not against rendered text

tai's default terminal rendering shows a tool's result as a truncated summary (e.g.
`└─ 78 chars · 1ms`), not the tool's actual return string — this was discovered empirically
while building `tests/rag.py` (its first version tried to regex-match `"saved mem:..."` out of
one-shot stdout and silently failed every time, even when the save had genuinely succeeded).
**When a test needs to know whether something actually happened, check the thing that
actually changed** — read the vault's `memory.db` directly with `sqlite3` (safe once
`TAI_PASSPHRASE` is forced empty per the isolation rule above), check a file on disk, etc. —
rather than parsing CLI output that was only ever designed to be read by a human, not
asserted on. CLI output is still worth capturing (it's what the log file is for, and it's
exactly what a deterministic REPL slash-command reply like `/search` is reasonable to assert
on, since that output format is a stable, documented interface) — the rule is specifically
about not trusting *tool-call rendering* as a proxy for *tool-call success*.

## LLM-driven steps are probabilistic — say so, don't hide it

This is the practical follow-through on "test with the real AI" above, not an escape hatch
from it. The thing actually being validated — did the agent decide to use its tools
correctly — only exists as a real step if the model is really making the decision, so that
step is never faked out for convenience. What *can* and should vary is how much of the test
*around* that step needs the model too:

- A deterministic path (a REPL slash command, which dispatches a tool directly with no model
  call at all) is appropriate for checking that the harness and the plumbing work —
  `InteractiveReplSmokeTest` in `tests/rag.py` is exactly this, and it exists to isolate
  "is the pty transport broken" from "did the model behave today" when something fails. It is
  a smoke check *alongside* the AI-driven test, never a *replacement* for it when the thing
  under test is the agent's own tool-use behavior.
- tai's primary backend selects a free model per request (see the root [README.md](../README.md#backends)'s
  "Backends" section) — the exact model, and therefore exact tool-calling reliability, can
  differ between runs. Say this plainly in the test's docstring wherever a step is LLM-driven.
- Write `verify()` to fail only on a genuinely empty/broken result, not on "fewer than 100% of
  attempts worked" — report the actual ratio (or, better, a per-item breakdown, as
  `RagIndexAndSearchTest.verify()` does) in the failure message, so a flaky run is
  distinguishable from a real regression at a glance.

## Process hygiene

`tai_test.ManagedProcess` always starts children with `start_new_session=True` and tears them
down with an escalating, group-wide, always-reaping shutdown (`terminate()`), specifically so
a timed-out or crashed test never leaves an orphaned `tai.py` (or anything it spawned) running
in the background. If you ever spawn a process outside of `self.spawn()` for some reason, you
are responsible for giving it the same guarantee yourself — don't bypass it.