Docs audit: fix stale claims, cross-link hub docs
- README: 42 tools (was 38), /demo row, TUI section, testing plus integration line, full layout map - tui.md: research link, tests pointer, soften shot-tested claim - AGENTS.md/tests/README.md/research hubs: cross-links between docs - Research: forward-correct tui conclusion plus INDEX activity row (19 failures fixed, tai_pkg deleted, 320/320 green); audit PASS
This commit is contained in:
+6
-6
@@ -4,11 +4,11 @@ This is where **integration tests** live — tests that drive the real `tai.py`
|
||||
subprocess and check its actual behavior. They are a different thing from the project's
|
||||
**unit tests** (`test_tai.py`, `test_seal.py`, at the repository top level, `unittest`-based,
|
||||
import `tai.py` as a module, fast, no subprocess). Both exist on purpose, side by side; see
|
||||
`AGENTS.md`'s "Testing" section for which convention applies to which kind of test, and do
|
||||
[AGENTS.md](../AGENTS.md#testing)'s "Testing" section for which convention applies to which kind of test, and do
|
||||
not move one style into the other's location or naming.
|
||||
|
||||
**This is the only way integration tests get written for this project from now on.** A new
|
||||
integration test is a new file under `tests/`, built on `tai_test.py` (the framework, at the
|
||||
integration test is a new file under `tests/`, built on [tai_test.py](../tai_test.py) (the framework, at the
|
||||
repository root — read its module docstring before using it). Don't hand-roll a fresh
|
||||
`subprocess`/`asyncio` harness per test file; extend `tai_test.TestCase` instead, so every
|
||||
integration test behaves consistently (same timeout handling, same process cleanup, same
|
||||
@@ -43,7 +43,7 @@ end to end, while building `tests/rag.py`:
|
||||
fidelity score of 1.00 against a value the test itself inserted would prove nothing.
|
||||
|
||||
So: **we do not write integration tests that fake the AI's part to get a clean, deterministic
|
||||
result. We accept that free, rotating backends (see root `README.md`'s "Backends" section)
|
||||
result. We accept that free, rotating backends (see root [README.md](../README.md#backends)'s "Backends" section)
|
||||
make LLM-driven steps genuinely flaky, and we treat that flakiness as the cost of actually
|
||||
testing the thing we built, not as a defect to engineer around by testing something easier
|
||||
instead.** The practical consequences of this — report ratios instead of demanding
|
||||
@@ -55,12 +55,12 @@ the harness itself works, never as a replacement for the AI-driven test of the a
|
||||
|
||||
1. Create `tests/<topic>.py` (lowercase, no `test_` prefix — that prefix is reserved for the
|
||||
top-level unit-test files so the two kinds are never confused by name alone).
|
||||
2. `sys.path.insert(0, str(Path(__file__).resolve().parent.parent))` then `from tai_test import TestCase, ...` — see `tests/rag.py` for the exact boilerplate.
|
||||
2. `sys.path.insert(0, str(Path(__file__).resolve().parent.parent))` then `from tai_test import TestCase, ...` — see [tests/rag.py](rag.py) for the exact boilerplate.
|
||||
3. Subclass `TestCase` for each distinct scenario. Required: `name` (a class attribute),
|
||||
`async def run(self)` (drive the process via `self.spawn(...)`, return whatever `verify()`
|
||||
needs), `async def verify(self, result)` (raise `TestAssertionFailed` with a clear message
|
||||
on failure). Optional: `async def setup(self)` / `async def teardown(self)`, `timeout`,
|
||||
`mode` ("pipe" or "pty" — see `tai_test.py`'s docstring for when each applies).
|
||||
`mode` ("pipe" or "pty" — see [tai_test.py](../tai_test.py)'s docstring for when each applies).
|
||||
4. Collect your cases into a module-level `ALL_CASES = [...]` list and end the file with:
|
||||
```python
|
||||
if __name__ == "__main__":
|
||||
@@ -107,7 +107,7 @@ step is never faked out for convenience. What *can* and should vary is how much
|
||||
"is the pty transport broken" from "did the model behave today" when something fails. It is
|
||||
a smoke check *alongside* the AI-driven test, never a *replacement* for it when the thing
|
||||
under test is the agent's own tool-use behavior.
|
||||
- tai's primary backend selects a free model per request (see the root `README.md`'s
|
||||
- tai's primary backend selects a free model per request (see the root [README.md](../README.md#backends)'s
|
||||
"Backends" section) — the exact model, and therefore exact tool-calling reliability, can
|
||||
differ between runs. Say this plainly in the test's docstring wherever a step is LLM-driven.
|
||||
- Write `verify()` to fail only on a genuinely empty/broken result, not on "fewer than 100% of
|
||||
|
||||
Reference in New Issue
Block a user