From 133eb5ef63633a3cf4ab53482dd5fa3ed6152a37 Mon Sep 17 00:00:00 2001 From: retoor Date: Sat, 10 Oct 2026 23:45:43 +0200 Subject: [PATCH] Track tests/README.md (integration test conventions) --- tests/README.md | 124 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 tests/README.md diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..0f925af --- /dev/null +++ b/tests/README.md @@ -0,0 +1,124 @@ +# Integration tests + +This is where **integration tests** live — tests that drive the real `tai.py` binary as a +subprocess and check its actual behavior. They are a different thing from the project's +**unit tests** (`test_tai.py`, `test_seal.py`, at the repository top level, `unittest`-based, +import `tai.py` as a module, fast, no subprocess). Both exist on purpose, side by side; see +`AGENTS.md`'s "Testing" section for which convention applies to which kind of test, and do +not move one style into the other's location or naming. + +**This is the only way integration tests get written for this project from now on.** A new +integration test is a new file under `tests/`, built on `tai_test.py` (the framework, at the +repository root — read its module docstring before using it). Don't hand-roll a fresh +`subprocess`/`asyncio` harness per test file; extend `tai_test.TestCase` instead, so every +integration test behaves consistently (same timeout handling, same process cleanup, same +logging, same CLI flags) without each one having to get those details right from scratch. + +## The core rule: test with the real AI, and accept the flakes + +An integration test for this project is not done proving something until it has made the +real `tai.py` binary make a real decision through a real call to a real LLM backend. Not a +mock of the model, not a stub tool, not a direct write into the vault that *simulates* what +the agent would have done. If a test's "proof" could have been produced by a human directly +editing the database instead of the agent ever running, it isn't an integration test for an +AI agent — it's a database test wearing an integration test's name. + +This isn't a style preference; it came directly from the first time this rule was followed +end to end, while building `tests/rag.py`: + +- The CLI's own terminal rendering turned out to hide the actual tool-call return value + behind a `└─ 78 chars · 1ms` summary. A test that stopped at the subprocess's stdout — the + "deterministic-looking" layer — would have silently reported every real save as a failure. + Checking the vault's actual state caught this; a mocked tool call would never have surfaced + the discrepancy, because a mock doesn't render a UI. +- With a real model in the loop, `record_save` got called **twice for one file** and **three + times for another** in the same run, once in a cross-call way that briefly made search + results from an earlier file leak into a later one. That is a genuine, observed property of + how the agent actually behaves under conversational history — not a bug anyone would have + thought to inject into a scripted fake, and not something a test that never touches the real + model could ever have found. +- The final, strong per-file proof (exact content fidelity, the fingerprint surviving + chunking under the correct record, `/search` naming the correct `mem:` id) is only + meaningful *because* the thing being measured — the model's own tool call — was real. A + fidelity score of 1.00 against a value the test itself inserted would prove nothing. + +So: **we do not write integration tests that fake the AI's part to get a clean, deterministic +result. We accept that free, rotating backends (see root `README.md`'s "Backends" section) +make LLM-driven steps genuinely flaky, and we treat that flakiness as the cost of actually +testing the thing we built, not as a defect to engineer around by testing something easier +instead.** The practical consequences of this — report ratios instead of demanding +all-or-nothing, prefer a deterministic *plumbing* check (a REPL slash command) only to prove +the harness itself works, never as a replacement for the AI-driven test of the actual feature +— are spelled out below in "LLM-driven steps are probabilistic." + +## How to add a new integration test + +1. Create `tests/.py` (lowercase, no `test_` prefix — that prefix is reserved for the + top-level unit-test files so the two kinds are never confused by name alone). +2. `sys.path.insert(0, str(Path(__file__).resolve().parent.parent))` then `from tai_test import TestCase, ...` — see `tests/rag.py` for the exact boilerplate. +3. Subclass `TestCase` for each distinct scenario. Required: `name` (a class attribute), + `async def run(self)` (drive the process via `self.spawn(...)`, return whatever `verify()` + needs), `async def verify(self, result)` (raise `TestAssertionFailed` with a clear message + on failure). Optional: `async def setup(self)` / `async def teardown(self)`, `timeout`, + `mode` ("pipe" or "pty" — see `tai_test.py`'s docstring for when each applies). +4. Collect your cases into a module-level `ALL_CASES = [...]` list and end the file with: + ```python + if __name__ == "__main__": + sys.exit(main_cli(ALL_CASES, default_log_dir=str(Path(__file__).resolve().parent / "logs"))) + ``` +5. Run it: `python3 tests/.py` (all cases), `--filter ` (a subset), + `--interactive` (also forward your own live typing into the running process — useful for + watching a flaky step happen and nudging it by hand), `--timeout ` (override). + +## Isolation — always use a throwaway `TAI_HOME` + +Every test that touches the vault must set `TAI_HOME` (and `TAI_PASSPHRASE=""`, so the vault +stays plaintext and directly inspectable) to a fresh `tempfile.mkdtemp()` in `setup()`, and +remove it in `teardown()`. Never point a test at the real `~/.tai` — there is no "are you +sure" here, a test that forgets this will read and write the actual user's memory, secrets, +and schedules. `tests/rag.py`'s two test cases are the reference example. + +## Verify against state, not against rendered text + +tai's default terminal rendering shows a tool's result as a truncated summary (e.g. +`└─ 78 chars · 1ms`), not the tool's actual return string — this was discovered empirically +while building `tests/rag.py` (its first version tried to regex-match `"saved mem:..."` out of +one-shot stdout and silently failed every time, even when the save had genuinely succeeded). +**When a test needs to know whether something actually happened, check the thing that +actually changed** — read the vault's `memory.db` directly with `sqlite3` (safe once +`TAI_PASSPHRASE` is forced empty per the isolation rule above), check a file on disk, etc. — +rather than parsing CLI output that was only ever designed to be read by a human, not +asserted on. CLI output is still worth capturing (it's what the log file is for, and it's +exactly what a deterministic REPL slash-command reply like `/search` is reasonable to assert +on, since that output format is a stable, documented interface) — the rule is specifically +about not trusting *tool-call rendering* as a proxy for *tool-call success*. + +## LLM-driven steps are probabilistic — say so, don't hide it + +This is the practical follow-through on "test with the real AI" above, not an escape hatch +from it. The thing actually being validated — did the agent decide to use its tools +correctly — only exists as a real step if the model is really making the decision, so that +step is never faked out for convenience. What *can* and should vary is how much of the test +*around* that step needs the model too: + +- A deterministic path (a REPL slash command, which dispatches a tool directly with no model + call at all) is appropriate for checking that the harness and the plumbing work — + `InteractiveReplSmokeTest` in `tests/rag.py` is exactly this, and it exists to isolate + "is the pty transport broken" from "did the model behave today" when something fails. It is + a smoke check *alongside* the AI-driven test, never a *replacement* for it when the thing + under test is the agent's own tool-use behavior. +- tai's primary backend selects a free model per request (see the root `README.md`'s + "Backends" section) — the exact model, and therefore exact tool-calling reliability, can + differ between runs. Say this plainly in the test's docstring wherever a step is LLM-driven. +- Write `verify()` to fail only on a genuinely empty/broken result, not on "fewer than 100% of + attempts worked" — report the actual ratio (or, better, a per-item breakdown, as + `RagIndexAndSearchTest.verify()` does) in the failure message, so a flaky run is + distinguishable from a real regression at a glance. + +## Process hygiene + +`tai_test.ManagedProcess` always starts children with `start_new_session=True` and tears them +down with an escalating, group-wide, always-reaping shutdown (`terminate()`), specifically so +a timed-out or crashed test never leaves an orphaned `tai.py` (or anything it spawned) running +in the background. If you ever spawn a process outside of `self.spawn()` for some reason, you +are responsible for giving it the same guarantee yourself — don't bypass it.