Track tests/README.md (integration test conventions)
This commit is contained in:
+124
@@ -0,0 +1,124 @@
|
||||
# Integration tests
|
||||
|
||||
This is where **integration tests** live — tests that drive the real `tai.py` binary as a
|
||||
subprocess and check its actual behavior. They are a different thing from the project's
|
||||
**unit tests** (`test_tai.py`, `test_seal.py`, at the repository top level, `unittest`-based,
|
||||
import `tai.py` as a module, fast, no subprocess). Both exist on purpose, side by side; see
|
||||
`AGENTS.md`'s "Testing" section for which convention applies to which kind of test, and do
|
||||
not move one style into the other's location or naming.
|
||||
|
||||
**This is the only way integration tests get written for this project from now on.** A new
|
||||
integration test is a new file under `tests/`, built on `tai_test.py` (the framework, at the
|
||||
repository root — read its module docstring before using it). Don't hand-roll a fresh
|
||||
`subprocess`/`asyncio` harness per test file; extend `tai_test.TestCase` instead, so every
|
||||
integration test behaves consistently (same timeout handling, same process cleanup, same
|
||||
logging, same CLI flags) without each one having to get those details right from scratch.
|
||||
|
||||
## The core rule: test with the real AI, and accept the flakes
|
||||
|
||||
An integration test for this project is not done proving something until it has made the
|
||||
real `tai.py` binary make a real decision through a real call to a real LLM backend. Not a
|
||||
mock of the model, not a stub tool, not a direct write into the vault that *simulates* what
|
||||
the agent would have done. If a test's "proof" could have been produced by a human directly
|
||||
editing the database instead of the agent ever running, it isn't an integration test for an
|
||||
AI agent — it's a database test wearing an integration test's name.
|
||||
|
||||
This isn't a style preference; it came directly from the first time this rule was followed
|
||||
end to end, while building `tests/rag.py`:
|
||||
|
||||
- The CLI's own terminal rendering turned out to hide the actual tool-call return value
|
||||
behind a `└─ 78 chars · 1ms` summary. A test that stopped at the subprocess's stdout — the
|
||||
"deterministic-looking" layer — would have silently reported every real save as a failure.
|
||||
Checking the vault's actual state caught this; a mocked tool call would never have surfaced
|
||||
the discrepancy, because a mock doesn't render a UI.
|
||||
- With a real model in the loop, `record_save` got called **twice for one file** and **three
|
||||
times for another** in the same run, once in a cross-call way that briefly made search
|
||||
results from an earlier file leak into a later one. That is a genuine, observed property of
|
||||
how the agent actually behaves under conversational history — not a bug anyone would have
|
||||
thought to inject into a scripted fake, and not something a test that never touches the real
|
||||
model could ever have found.
|
||||
- The final, strong per-file proof (exact content fidelity, the fingerprint surviving
|
||||
chunking under the correct record, `/search` naming the correct `mem:` id) is only
|
||||
meaningful *because* the thing being measured — the model's own tool call — was real. A
|
||||
fidelity score of 1.00 against a value the test itself inserted would prove nothing.
|
||||
|
||||
So: **we do not write integration tests that fake the AI's part to get a clean, deterministic
|
||||
result. We accept that free, rotating backends (see root `README.md`'s "Backends" section)
|
||||
make LLM-driven steps genuinely flaky, and we treat that flakiness as the cost of actually
|
||||
testing the thing we built, not as a defect to engineer around by testing something easier
|
||||
instead.** The practical consequences of this — report ratios instead of demanding
|
||||
all-or-nothing, prefer a deterministic *plumbing* check (a REPL slash command) only to prove
|
||||
the harness itself works, never as a replacement for the AI-driven test of the actual feature
|
||||
— are spelled out below in "LLM-driven steps are probabilistic."
|
||||
|
||||
## How to add a new integration test
|
||||
|
||||
1. Create `tests/<topic>.py` (lowercase, no `test_` prefix — that prefix is reserved for the
|
||||
top-level unit-test files so the two kinds are never confused by name alone).
|
||||
2. `sys.path.insert(0, str(Path(__file__).resolve().parent.parent))` then `from tai_test import TestCase, ...` — see `tests/rag.py` for the exact boilerplate.
|
||||
3. Subclass `TestCase` for each distinct scenario. Required: `name` (a class attribute),
|
||||
`async def run(self)` (drive the process via `self.spawn(...)`, return whatever `verify()`
|
||||
needs), `async def verify(self, result)` (raise `TestAssertionFailed` with a clear message
|
||||
on failure). Optional: `async def setup(self)` / `async def teardown(self)`, `timeout`,
|
||||
`mode` ("pipe" or "pty" — see `tai_test.py`'s docstring for when each applies).
|
||||
4. Collect your cases into a module-level `ALL_CASES = [...]` list and end the file with:
|
||||
```python
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main_cli(ALL_CASES, default_log_dir=str(Path(__file__).resolve().parent / "logs")))
|
||||
```
|
||||
5. Run it: `python3 tests/<topic>.py` (all cases), `--filter <substring>` (a subset),
|
||||
`--interactive` (also forward your own live typing into the running process — useful for
|
||||
watching a flaky step happen and nudging it by hand), `--timeout <seconds>` (override).
|
||||
|
||||
## Isolation — always use a throwaway `TAI_HOME`
|
||||
|
||||
Every test that touches the vault must set `TAI_HOME` (and `TAI_PASSPHRASE=""`, so the vault
|
||||
stays plaintext and directly inspectable) to a fresh `tempfile.mkdtemp()` in `setup()`, and
|
||||
remove it in `teardown()`. Never point a test at the real `~/.tai` — there is no "are you
|
||||
sure" here, a test that forgets this will read and write the actual user's memory, secrets,
|
||||
and schedules. `tests/rag.py`'s two test cases are the reference example.
|
||||
|
||||
## Verify against state, not against rendered text
|
||||
|
||||
tai's default terminal rendering shows a tool's result as a truncated summary (e.g.
|
||||
`└─ 78 chars · 1ms`), not the tool's actual return string — this was discovered empirically
|
||||
while building `tests/rag.py` (its first version tried to regex-match `"saved mem:..."` out of
|
||||
one-shot stdout and silently failed every time, even when the save had genuinely succeeded).
|
||||
**When a test needs to know whether something actually happened, check the thing that
|
||||
actually changed** — read the vault's `memory.db` directly with `sqlite3` (safe once
|
||||
`TAI_PASSPHRASE` is forced empty per the isolation rule above), check a file on disk, etc. —
|
||||
rather than parsing CLI output that was only ever designed to be read by a human, not
|
||||
asserted on. CLI output is still worth capturing (it's what the log file is for, and it's
|
||||
exactly what a deterministic REPL slash-command reply like `/search` is reasonable to assert
|
||||
on, since that output format is a stable, documented interface) — the rule is specifically
|
||||
about not trusting *tool-call rendering* as a proxy for *tool-call success*.
|
||||
|
||||
## LLM-driven steps are probabilistic — say so, don't hide it
|
||||
|
||||
This is the practical follow-through on "test with the real AI" above, not an escape hatch
|
||||
from it. The thing actually being validated — did the agent decide to use its tools
|
||||
correctly — only exists as a real step if the model is really making the decision, so that
|
||||
step is never faked out for convenience. What *can* and should vary is how much of the test
|
||||
*around* that step needs the model too:
|
||||
|
||||
- A deterministic path (a REPL slash command, which dispatches a tool directly with no model
|
||||
call at all) is appropriate for checking that the harness and the plumbing work —
|
||||
`InteractiveReplSmokeTest` in `tests/rag.py` is exactly this, and it exists to isolate
|
||||
"is the pty transport broken" from "did the model behave today" when something fails. It is
|
||||
a smoke check *alongside* the AI-driven test, never a *replacement* for it when the thing
|
||||
under test is the agent's own tool-use behavior.
|
||||
- tai's primary backend selects a free model per request (see the root `README.md`'s
|
||||
"Backends" section) — the exact model, and therefore exact tool-calling reliability, can
|
||||
differ between runs. Say this plainly in the test's docstring wherever a step is LLM-driven.
|
||||
- Write `verify()` to fail only on a genuinely empty/broken result, not on "fewer than 100% of
|
||||
attempts worked" — report the actual ratio (or, better, a per-item breakdown, as
|
||||
`RagIndexAndSearchTest.verify()` does) in the failure message, so a flaky run is
|
||||
distinguishable from a real regression at a glance.
|
||||
|
||||
## Process hygiene
|
||||
|
||||
`tai_test.ManagedProcess` always starts children with `start_new_session=True` and tears them
|
||||
down with an escalating, group-wide, always-reaping shutdown (`terminate()`), specifically so
|
||||
a timed-out or crashed test never leaves an orphaned `tai.py` (or anything it spawned) running
|
||||
in the background. If you ever spawn a process outside of `self.spawn()` for some reason, you
|
||||
are responsible for giving it the same guarantee yourself — don't bypass it.
|
||||
Reference in New Issue
Block a user