tai/tests/README.md

Integration tests

This is where integration tests live — tests that drive the real tai.py binary as a subprocess and check its actual behavior. They are a different thing from the project's unit tests (test_tai.py, test_seal.py, at the repository top level, unittest-based, import tai.py as a module, fast, no subprocess). Both exist on purpose, side by side; see AGENTS.md's "Testing" section for which convention applies to which kind of test, and do not move one style into the other's location or naming.

This is the only way integration tests get written for this project from now on. A new integration test is a new file under tests/, built on tai_test.py (the framework, at the repository root — read its module docstring before using it). Don't hand-roll a fresh subprocess/asyncio harness per test file; extend tai_test.TestCase instead, so every integration test behaves consistently (same timeout handling, same process cleanup, same logging, same CLI flags) without each one having to get those details right from scratch.

The core rule: test with the real AI, and accept the flakes

An integration test for this project is not done proving something until it has made the real tai.py binary make a real decision through a real call to a real LLM backend. Not a mock of the model, not a stub tool, not a direct write into the vault that simulates what the agent would have done. If a test's "proof" could have been produced by a human directly editing the database instead of the agent ever running, it isn't an integration test for an AI agent — it's a database test wearing an integration test's name.

This isn't a style preference; it came directly from the first time this rule was followed end to end, while building tests/rag.py:

• The CLI's own terminal rendering turned out to hide the actual tool-call return value behind a └─ 78 chars · 1ms summary. A test that stopped at the subprocess's stdout — the "deterministic-looking" layer — would have silently reported every real save as a failure. Checking the vault's actual state caught this; a mocked tool call would never have surfaced the discrepancy, because a mock doesn't render a UI.
• With a real model in the loop, record_save got called twice for one file and three times for another in the same run, once in a cross-call way that briefly made search results from an earlier file leak into a later one. That is a genuine, observed property of how the agent actually behaves under conversational history — not a bug anyone would have thought to inject into a scripted fake, and not something a test that never touches the real model could ever have found.
• The final, strong per-file proof (exact content fidelity, the fingerprint surviving chunking under the correct record, /search naming the correct mem: id) is only meaningful because the thing being measured — the model's own tool call — was real. A fidelity score of 1.00 against a value the test itself inserted would prove nothing.

So: we do not write integration tests that fake the AI's part to get a clean, deterministic result. We accept that free, rotating backends (see root README.md's "Backends" section) make LLM-driven steps genuinely flaky, and we treat that flakiness as the cost of actually testing the thing we built, not as a defect to engineer around by testing something easier instead. The practical consequences of this — report ratios instead of demanding all-or-nothing, prefer a deterministic plumbing check (a REPL slash command) only to prove the harness itself works, never as a replacement for the AI-driven test of the actual feature — are spelled out below in "LLM-driven steps are probabilistic."

How to add a new integration test

1. Create tests/<topic>.py (lowercase, no test_ prefix — that prefix is reserved for the top-level unit-test files so the two kinds are never confused by name alone).
2. sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) then from tai_test import TestCase, ... — see tests/rag.py for the exact boilerplate.
3. Subclass TestCase for each distinct scenario. Required: name (a class attribute), async def run(self) (drive the process via self.spawn(...), return whatever verify() needs), async def verify(self, result) (raise TestAssertionFailed with a clear message on failure). Optional: async def setup(self) / async def teardown(self), timeout, mode ("pipe" or "pty" — see tai_test.py's docstring for when each applies).
4. Collect your cases into a module-level ALL_CASES = [...] list and end the file with:

   if __name__ == "__main__":
       sys.exit(main_cli(ALL_CASES, default_log_dir=str(Path(__file__).resolve().parent / "logs")))
   

5. Run it: python3 tests/<topic>.py (all cases), --filter <substring> (a subset), --interactive (also forward your own live typing into the running process — useful for watching a flaky step happen and nudging it by hand), --timeout <seconds> (override).

Isolation — always use a throwaway TAI_HOME

Every test that touches the vault must set TAI_HOME (and TAI_PASSPHRASE="", so the vault stays plaintext and directly inspectable) to a fresh tempfile.mkdtemp() in setup(), and remove it in teardown(). Never point a test at the real ~/.tai — there is no "are you sure" here, a test that forgets this will read and write the actual user's memory, secrets, and schedules. tests/rag.py's two test cases are the reference example.

Verify against state, not against rendered text

tai's default terminal rendering shows a tool's result as a truncated summary (e.g. └─ 78 chars · 1ms), not the tool's actual return string — this was discovered empirically while building tests/rag.py (its first version tried to regex-match "saved mem:..." out of one-shot stdout and silently failed every time, even when the save had genuinely succeeded). When a test needs to know whether something actually happened, check the thing that actually changed — read the vault's memory.db directly with sqlite3 (safe once TAI_PASSPHRASE is forced empty per the isolation rule above), check a file on disk, etc. — rather than parsing CLI output that was only ever designed to be read by a human, not asserted on. CLI output is still worth capturing (it's what the log file is for, and it's exactly what a deterministic REPL slash-command reply like /search is reasonable to assert on, since that output format is a stable, documented interface) — the rule is specifically about not trusting tool-call rendering as a proxy for tool-call success.

LLM-driven steps are probabilistic — say so, don't hide it

This is the practical follow-through on "test with the real AI" above, not an escape hatch from it. The thing actually being validated — did the agent decide to use its tools correctly — only exists as a real step if the model is really making the decision, so that step is never faked out for convenience. What can and should vary is how much of the test around that step needs the model too:

• A deterministic path (a REPL slash command, which dispatches a tool directly with no model call at all) is appropriate for checking that the harness and the plumbing work — InteractiveReplSmokeTest in tests/rag.py is exactly this, and it exists to isolate "is the pty transport broken" from "did the model behave today" when something fails. It is a smoke check alongside the AI-driven test, never a replacement for it when the thing under test is the agent's own tool-use behavior.
• tai's primary backend selects a free model per request (see the root README.md's "Backends" section) — the exact model, and therefore exact tool-calling reliability, can differ between runs. Say this plainly in the test's docstring wherever a step is LLM-driven.
• Write verify() to fail only on a genuinely empty/broken result, not on "fewer than 100% of attempts worked" — report the actual ratio (or, better, a per-item breakdown, as RagIndexAndSearchTest.verify() does) in the failure message, so a flaky run is distinguishable from a real regression at a glance.

Process hygiene

tai_test.ManagedProcess always starts children with start_new_session=True and tears them down with an escalating, group-wide, always-reaping shutdown (terminate()), specifically so a timed-out or crashed test never leaves an orphaned tai.py (or anything it spawned) running in the background. If you ever spawn a process outside of self.spawn() for some reason, you are responsible for giving it the same guarantee yourself — don't bypass it.