Add plexer subsystem, container test infra, and last-resort opencode box backend
- Plexer: always-active terminal multiplexing. NativePlexerController/ TmuxPlexerController share one interface, chosen by which env var is already set (TAI_PLEXER_SOCKET vs TMUX_PANE); NativePlexerBackend is the pty.fork()-based wrapper (raw mode, SIGWINCH resize forwarding, a Unix-socket control protocol, continuous session logging, a Ctrl+A prefix key distinct from tmux's own Ctrl+B). Auto-wraps new interactive shells via a .bashrc hook mirroring the existing command-not-found install pattern, gated on $TMUX/$TAI_PLEXER_ACTIVE/ a $HOME/.tai-no-plexer bypass. New plexer_send/read/type/status tools, env-gated on a detected controller. - tests/container.py + docker-compose.yml + tests/docker/Dockerfile: a disposable podman/docker image for the tests that must never touch a real machine (installing the plexer hook into a real ~/.bashrc, a live tmux session). Required suite (test_seal.py, test_tai.py, tests/plexer.py, tests/bashrc.py) gates the result; tests/rag.py runs as best-effort since its real-LLM dependency can't be satisfied inside a disposable container with no nested engine. - tests/bashrc.py: real end-to-end hook install + bash -i auto-wrap, anti-recursion, tmux-active skip, non-interactive skip, a live TmuxPlexerController test, and an install/uninstall round-trip. - tests/plexer.py: NativePlexerBackend under a real pty, plus a dedicated robustness pass (file permissions, >64KB payload framing, a malformed control request, a genuine self-deadlock fix in the write-to-child path). - tai_test.py: the shared async process-driving framework (pipe/pty transport, group-wide kill-and-reap, a StdinBridge for --interactive) both test files build on. - ChatClient last-resort backend: when there's no local opencode CLI but the sandbox box is already running, install opencode into it (official Node tarball, not apt -- apt's nodejs/npm pull in webpack/babel/jest as transitive deps) and route through it instead of giving up. Only reachable after the first retry round already failed everything else; never creates the box itself. TmuxPlexerController.status() fixed to use `list-panes` instead of `display-message -p`, which silently fell back to the default pane instead of erroring against a dead target. - Fix 4 pre-existing test_tai.py bugs surfaced by running it in a clean environment for the first time: two stale mock signatures (subprocess.run's stdout= kwarg, Chat.complete's deadline= kwarg), one $TMUX-dependent catalog assertion, one tty-dependent color assertion. - research/: a sourced research archive (Source Dossier/Citation Card/Conclusion tiers) backing the above design decisions, with its own audit.py consistency checker. - .gitignore: fix a blanket *.md rule that was about to silently drop the entire research/ archive from version control; add __pycache__/ and the one-off debug transcript this session started from. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 5
+389
@@ -0,0 +1,389 @@
|
||||
#!/usr/bin/env python3
|
||||
"""tests/bashrc.py — the tests that can only run honestly inside the throwaway
|
||||
container (see tests/docker/Dockerfile, docker-compose.yml, tests/container.py).
|
||||
|
||||
Everything here either installs a real hook into a real $HOME/.bashrc and
|
||||
spawns a real `bash -i` to watch it fire, or drives a real `tmux` session.
|
||||
Both are things no test should ever do against the machine actually running
|
||||
Claude Code -- so every test case that touches $HOME/.bashrc checks
|
||||
TAI_CONTAINER_TEST=1 in setup() and refuses outright if it's missing, rather
|
||||
than trusting whoever runs this file to remember to only run it in the
|
||||
container. tests/docker/Dockerfile sets that var; nothing else should.
|
||||
|
||||
Deterministic throughout, same as tests/plexer.py: no LLM involved, this is
|
||||
pty/shell/signal mechanics, not a model decision.
|
||||
"""
|
||||
import asyncio
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from tai_test import TestCase, TestAssertionFailed, ManagedProcess, main_cli # noqa: E402
|
||||
import tai # noqa: E402
|
||||
|
||||
|
||||
async def _drain_forever(proc):
|
||||
"""Same requirement as tests/plexer.py's helper of the same name (see
|
||||
that file for the full postmortem): only needed when the spawned process
|
||||
itself runs a second relay loop that would otherwise self-deadlock on its
|
||||
own unread output -- i.e. once bash execs into `tai.py --plexer-wrap`.
|
||||
Plain `bash -i` that never wraps doesn't need this; those tests read
|
||||
directly instead."""
|
||||
try:
|
||||
while True:
|
||||
chunk = await proc._read_chunk()
|
||||
if not chunk:
|
||||
await asyncio.sleep(0.05)
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
|
||||
class _BashrcHookTestBase(TestCase):
|
||||
"""Shared fixture for every test that installs the real plexer hook into
|
||||
a real .bashrc: a throwaway $HOME, the real tai.py binary copied into
|
||||
$HOME/.local/bin via tai.op_binary (exactly what `tai.py --install` does
|
||||
on a real machine), and the hook itself via tai.op_plexer_hook -- the
|
||||
same functions a real install runs, not a hand-rolled substitute."""
|
||||
|
||||
mode = "pty"
|
||||
timeout = 30.0
|
||||
|
||||
async def setup(self):
|
||||
if os.environ.get("TAI_CONTAINER_TEST") != "1":
|
||||
raise TestAssertionFailed(
|
||||
"refusing to run (TAI_CONTAINER_TEST is not set): this test installs a real "
|
||||
"hook into $HOME/.bashrc and must only ever run inside the disposable test "
|
||||
"container (see tests/docker/Dockerfile), never against a real machine's home"
|
||||
)
|
||||
self.tmp_home = tempfile.mkdtemp(prefix="tai-test-home-")
|
||||
os.makedirs(os.path.join(self.tmp_home, ".local", "bin"), exist_ok=True)
|
||||
real_home = os.environ.get("HOME")
|
||||
os.environ["HOME"] = self.tmp_home
|
||||
try:
|
||||
tai.op_binary("install", self.tmp_home)
|
||||
tai.op_plexer_hook("install", self.tmp_home)
|
||||
finally:
|
||||
if real_home is not None:
|
||||
os.environ["HOME"] = real_home
|
||||
else:
|
||||
os.environ.pop("HOME", None)
|
||||
self._drain_task = None
|
||||
|
||||
async def _spawn_bash(self, args, mode=None, env_overrides=None):
|
||||
env = self.build_env()
|
||||
env.update(env_overrides or {})
|
||||
proc = ManagedProcess(["bash"] + list(args), mode=mode or self.mode, env=env, on_line=self._on_line)
|
||||
await proc.start()
|
||||
self._processes.append(proc)
|
||||
return proc
|
||||
|
||||
def _sock_matches(self):
|
||||
return glob.glob(os.path.join(self.tmp_home, ".tai", "plexer", "*.sock"))
|
||||
|
||||
async def teardown(self):
|
||||
if self._drain_task is not None:
|
||||
self._drain_task.cancel()
|
||||
shutil.rmtree(self.tmp_home, ignore_errors=True)
|
||||
|
||||
|
||||
class BashrcInstallAndAutoWrapTest(_BashrcHookTestBase):
|
||||
"""The actual point of building a container for this: a real `bash -i`,
|
||||
with a real installed hook in its real $HOME/.bashrc, auto-wraps into a
|
||||
real NativePlexerBackend with no human and no flag telling it to."""
|
||||
|
||||
name = "bashrc-install-and-auto-wrap"
|
||||
|
||||
async def run(self):
|
||||
proc = await self._spawn_bash(["-i"], env_overrides={"HOME": self.tmp_home})
|
||||
self._drain_task = asyncio.ensure_future(_drain_forever(proc))
|
||||
|
||||
sock_path = None
|
||||
for _ in range(50):
|
||||
matches = self._sock_matches()
|
||||
if matches:
|
||||
sock_path = matches[0]
|
||||
break
|
||||
await asyncio.sleep(0.2)
|
||||
if not sock_path:
|
||||
raise TestAssertionFailed(
|
||||
"bash -i never auto-wrapped into a plexer backend via the installed .bashrc hook"
|
||||
)
|
||||
|
||||
controller = tai.NativePlexerController(sock_path)
|
||||
await asyncio.sleep(0.3)
|
||||
status = controller.status()
|
||||
|
||||
controller.send_keys("echo $TAI_PLEXER_SOCKET", enter=True)
|
||||
await asyncio.sleep(0.4)
|
||||
pane = controller.read_pane(50)
|
||||
|
||||
return {"sock_path": sock_path, "pane": pane, "status": status}
|
||||
|
||||
async def verify(self, result):
|
||||
if result["sock_path"] not in result["pane"]:
|
||||
raise TestAssertionFailed(
|
||||
"the auto-wrapped inner shell's $TAI_PLEXER_SOCKET did not match the backend's "
|
||||
"own socket path: %r" % result["pane"][-300:]
|
||||
)
|
||||
if not result["status"].get("pid") or result["status"]["pid"] <= 0:
|
||||
raise TestAssertionFailed("status() did not report a live child pid after auto-wrap")
|
||||
|
||||
|
||||
class AntiRecursionTest(_BashrcHookTestBase):
|
||||
"""Already-wrapped ($TAI_PLEXER_ACTIVE=1) must never wrap again."""
|
||||
|
||||
name = "bashrc-hook-anti-recursion"
|
||||
|
||||
async def run(self):
|
||||
proc = await self._spawn_bash(
|
||||
["-i"], env_overrides={"HOME": self.tmp_home, "TAI_PLEXER_ACTIVE": "1"}
|
||||
)
|
||||
await proc.send_line("echo anti_recursion_marker_ok")
|
||||
await proc.wait_for_pattern(re.compile(r"anti_recursion_marker_ok"), timeout=8.0)
|
||||
await asyncio.sleep(0.3)
|
||||
return {"sock_matches": self._sock_matches()}
|
||||
|
||||
async def verify(self, result):
|
||||
if result["sock_matches"]:
|
||||
raise TestAssertionFailed(
|
||||
"a plexer socket was created even though TAI_PLEXER_ACTIVE was already set -- "
|
||||
"recursive wrap was not prevented"
|
||||
)
|
||||
|
||||
|
||||
class TmuxActiveSkipTest(_BashrcHookTestBase):
|
||||
"""An already-active tmux pane ($TMUX set) must also never wrap."""
|
||||
|
||||
name = "bashrc-hook-tmux-active-skip"
|
||||
|
||||
async def run(self):
|
||||
proc = await self._spawn_bash(
|
||||
["-i"], env_overrides={"HOME": self.tmp_home, "TMUX": "dummy-session:0.0"}
|
||||
)
|
||||
await proc.send_line("echo tmux_skip_marker_ok")
|
||||
await proc.wait_for_pattern(re.compile(r"tmux_skip_marker_ok"), timeout=8.0)
|
||||
await asyncio.sleep(0.3)
|
||||
return {"sock_matches": self._sock_matches()}
|
||||
|
||||
async def verify(self, result):
|
||||
if result["sock_matches"]:
|
||||
raise TestAssertionFailed(
|
||||
"a plexer socket was created even though $TMUX was already set -- tmux should "
|
||||
"have been left in sole control of the session"
|
||||
)
|
||||
|
||||
|
||||
class NonInteractiveSkipTest(_BashrcHookTestBase):
|
||||
"""A non-interactive shell must never wrap, even if it explicitly sources
|
||||
the hook (isolates the hook's own `case $- in *i*)` guard from bash's
|
||||
separate, unrelated default of not reading ~/.bashrc for `-c` at all)."""
|
||||
|
||||
name = "bashrc-hook-non-interactive-skip"
|
||||
mode = "pipe"
|
||||
|
||||
async def run(self):
|
||||
proc = await self._spawn_bash(
|
||||
["-c", 'source "$HOME/.bashrc" 2>&1; echo marker_non_interactive_ok'],
|
||||
mode="pipe",
|
||||
env_overrides={"HOME": self.tmp_home},
|
||||
)
|
||||
output = await proc.drain_until_eof(timeout=10.0)
|
||||
returncode = await proc.wait(timeout=5.0)
|
||||
return {"output": output, "returncode": returncode}
|
||||
|
||||
async def verify(self, result):
|
||||
if "marker_non_interactive_ok" not in result["output"]:
|
||||
raise TestAssertionFailed(
|
||||
"a non-interactive `bash -c` that explicitly sourced the hook never reached past "
|
||||
"it: %r" % result["output"][-500:]
|
||||
)
|
||||
if result["returncode"] != 0:
|
||||
raise TestAssertionFailed(
|
||||
"non-interactive shell exited %r after sourcing the hook" % result["returncode"]
|
||||
)
|
||||
if self._sock_matches():
|
||||
raise TestAssertionFailed(
|
||||
"a plexer socket was created from a non-interactive shell -- the "
|
||||
"`case $- in *i*)` guard failed"
|
||||
)
|
||||
|
||||
|
||||
class TmuxControllerLiveTest(TestCase):
|
||||
"""Drives TmuxPlexerController against a real `tmux new-session -d`, not
|
||||
a mock -- the one TmuxPlexerController code path tests/plexer.py cannot
|
||||
cover, since that file only ever exercises NativePlexerBackend."""
|
||||
|
||||
name = "bashrc-tmux-controller-live"
|
||||
timeout = 20.0
|
||||
|
||||
async def setup(self):
|
||||
if not shutil.which("tmux"):
|
||||
raise TestAssertionFailed(
|
||||
"tmux is not installed -- required to test TmuxPlexerController against a real session"
|
||||
)
|
||||
self.session_name = "tai-test-%d" % os.getpid()
|
||||
|
||||
async def run(self):
|
||||
create = await asyncio.create_subprocess_exec(
|
||||
"tmux", "new-session", "-d", "-s", self.session_name, "-x", "80", "-y", "24",
|
||||
)
|
||||
await create.wait()
|
||||
if create.returncode != 0:
|
||||
raise TestAssertionFailed("tmux new-session -d failed (returncode %r)" % create.returncode)
|
||||
|
||||
pane = "%s:0.0" % self.session_name
|
||||
controller = tai.TmuxPlexerController(pane)
|
||||
|
||||
status = controller.status()
|
||||
if status.get("backend") != "tmux":
|
||||
raise TestAssertionFailed("status() did not report backend=tmux: %r" % status)
|
||||
|
||||
controller.send_keys("echo tmux_controller_marker_ok", enter=True)
|
||||
await asyncio.sleep(0.6)
|
||||
pane_text = controller.read_pane(50)
|
||||
if "tmux_controller_marker_ok" not in pane_text:
|
||||
raise TestAssertionFailed(
|
||||
"send_keys/read_pane against a real tmux session failed: %r" % pane_text[-300:]
|
||||
)
|
||||
|
||||
controller.type_text("echo tmux_typed_marker_ok", delay=0.01, enter=True)
|
||||
await asyncio.sleep(0.6)
|
||||
pane_text_2 = controller.read_pane(50)
|
||||
if "tmux_typed_marker_ok" not in pane_text_2:
|
||||
raise TestAssertionFailed(
|
||||
"type_text against a real tmux session failed: %r" % pane_text_2[-300:]
|
||||
)
|
||||
|
||||
bogus = tai.TmuxPlexerController("nonexistent-session-%d:99.99" % os.getpid())
|
||||
raised = False
|
||||
try:
|
||||
bogus.status()
|
||||
except tai.PlexerError:
|
||||
raised = True
|
||||
return {"raised": raised}
|
||||
|
||||
async def verify(self, result):
|
||||
if not result["raised"]:
|
||||
raise TestAssertionFailed(
|
||||
"TmuxPlexerController against a nonexistent pane should raise PlexerError, not "
|
||||
"silently succeed"
|
||||
)
|
||||
|
||||
async def teardown(self):
|
||||
kill = await asyncio.create_subprocess_exec(
|
||||
"tmux", "kill-session", "-t", self.session_name,
|
||||
stdout=asyncio.subprocess.DEVNULL, stderr=asyncio.subprocess.DEVNULL,
|
||||
)
|
||||
await kill.wait()
|
||||
|
||||
|
||||
class PlexerHookInstallUninstallRoundTripTest(TestCase):
|
||||
"""Purely deterministic, no subprocess, no real $HOME -- the install
|
||||
functions are tested directly against a throwaway file path, so this one
|
||||
is safe to run anywhere, not just in the container."""
|
||||
|
||||
name = "bashrc-hook-install-uninstall-roundtrip"
|
||||
timeout = 10.0
|
||||
|
||||
async def setup(self):
|
||||
self.tmp_dir = tempfile.mkdtemp(prefix="tai-test-bashrc-")
|
||||
self.bashrc_path = os.path.join(self.tmp_dir, ".bashrc")
|
||||
with open(self.bashrc_path, "w", encoding="utf-8") as handle:
|
||||
handle.write("# pre-existing user content\nexport FOO=bar\n")
|
||||
|
||||
async def run(self):
|
||||
with open(self.bashrc_path, encoding="utf-8") as handle:
|
||||
original = handle.read()
|
||||
|
||||
installed_first = tai.upsert_plexer_bashrc_block(self.bashrc_path)
|
||||
present_after_first = tai.plexer_hook_present(self.bashrc_path)
|
||||
with open(self.bashrc_path, encoding="utf-8") as handle:
|
||||
content_after_first = handle.read()
|
||||
|
||||
installed_second = tai.upsert_plexer_bashrc_block(self.bashrc_path)
|
||||
with open(self.bashrc_path, encoding="utf-8") as handle:
|
||||
content_after_second = handle.read()
|
||||
|
||||
backup_path = self.bashrc_path + ".bak-tai"
|
||||
backup_exists = os.path.exists(backup_path)
|
||||
backup_content = None
|
||||
if backup_exists:
|
||||
with open(backup_path, encoding="utf-8") as handle:
|
||||
backup_content = handle.read()
|
||||
|
||||
removed = tai.remove_plexer_bashrc_block(self.bashrc_path)
|
||||
present_after_remove = tai.plexer_hook_present(self.bashrc_path)
|
||||
with open(self.bashrc_path, encoding="utf-8") as handle:
|
||||
content_after_remove = handle.read()
|
||||
|
||||
reinstalled = tai.upsert_plexer_bashrc_block(self.bashrc_path)
|
||||
present_after_reinstall = tai.plexer_hook_present(self.bashrc_path)
|
||||
|
||||
return {
|
||||
"original": original,
|
||||
"installed_first": installed_first,
|
||||
"present_after_first": present_after_first,
|
||||
"content_after_first": content_after_first,
|
||||
"installed_second": installed_second,
|
||||
"content_after_second": content_after_second,
|
||||
"backup_exists": backup_exists,
|
||||
"backup_content": backup_content,
|
||||
"removed": removed,
|
||||
"present_after_remove": present_after_remove,
|
||||
"content_after_remove": content_after_remove,
|
||||
"reinstalled": reinstalled,
|
||||
"present_after_reinstall": present_after_reinstall,
|
||||
}
|
||||
|
||||
async def verify(self, result):
|
||||
if not result["installed_first"]:
|
||||
raise TestAssertionFailed("first upsert_plexer_bashrc_block() reported no change on a clean file")
|
||||
if not result["present_after_first"]:
|
||||
raise TestAssertionFailed("plexer_hook_present() returned false right after installing")
|
||||
if "export FOO=bar" not in result["content_after_first"]:
|
||||
raise TestAssertionFailed("installing the hook clobbered pre-existing .bashrc content")
|
||||
|
||||
if result["installed_second"]:
|
||||
raise TestAssertionFailed("upsert_plexer_bashrc_block() was not idempotent -- reported a change on a second call")
|
||||
if result["content_after_second"] != result["content_after_first"]:
|
||||
raise TestAssertionFailed("a second install duplicated the hook block instead of being a no-op")
|
||||
|
||||
if not result["backup_exists"]:
|
||||
raise TestAssertionFailed("no .bak-tai backup was created before modifying .bashrc")
|
||||
if result["backup_content"] != result["original"]:
|
||||
raise TestAssertionFailed("the .bak-tai backup does not match the original pre-install content")
|
||||
|
||||
if not result["removed"]:
|
||||
raise TestAssertionFailed("remove_plexer_bashrc_block() reported no change when the hook was present")
|
||||
if result["present_after_remove"]:
|
||||
raise TestAssertionFailed("plexer_hook_present() still true after removal")
|
||||
if "export FOO=bar" not in result["content_after_remove"]:
|
||||
raise TestAssertionFailed("uninstalling the hook lost pre-existing .bashrc content")
|
||||
if tai.PLEXER_BASHRC_MARK_BEGIN in result["content_after_remove"]:
|
||||
raise TestAssertionFailed("uninstall left hook marker text behind")
|
||||
|
||||
if not result["reinstalled"]:
|
||||
raise TestAssertionFailed("could not reinstall the hook after a clean uninstall")
|
||||
if not result["present_after_reinstall"]:
|
||||
raise TestAssertionFailed("hook not present after reinstalling post-uninstall")
|
||||
|
||||
async def teardown(self):
|
||||
shutil.rmtree(self.tmp_dir, ignore_errors=True)
|
||||
|
||||
|
||||
ALL_CASES = [
|
||||
BashrcInstallAndAutoWrapTest(),
|
||||
AntiRecursionTest(),
|
||||
TmuxActiveSkipTest(),
|
||||
NonInteractiveSkipTest(),
|
||||
TmuxControllerLiveTest(),
|
||||
PlexerHookInstallUninstallRoundTripTest(),
|
||||
]
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main_cli(ALL_CASES, default_log_dir=str(Path(__file__).resolve().parent / "logs")))
|
||||
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env python3
|
||||
"""tests/container.py — run the full test suite inside a throwaway container.
|
||||
|
||||
Two things can only be tested honestly inside a disposable environment, not
|
||||
on the machine actually running Claude Code: installing the plexer hook into
|
||||
a real ~/.bashrc and watching a real `bash -i` actually get wrapped by it
|
||||
(tests/bashrc.py), and driving TmuxPlexerController against a real tmux
|
||||
session. This is the harness for that -- not a replacement for test_seal.py,
|
||||
test_tai.py, or tests/plexer.py, which also run inside the same container
|
||||
first, so "did it work in a clean environment" is answered for everything,
|
||||
not just the two new container-only tests. tests/rag.py runs too, but as a
|
||||
best-effort check, not a required one -- see BEST_EFFORT_SUITE below for why.
|
||||
|
||||
Engine detection deliberately reuses tai.container_engine() (podman first,
|
||||
then docker, same as the project's own sandbox feature) rather than
|
||||
reimplementing "which one is installed" a second way. docker-compose.yml at
|
||||
the repo root exists for a human who wants `docker compose run --rm
|
||||
tai-test` / `podman compose run --rm tai-test`; this script drives plain
|
||||
build/run directly so the automated path never depends on whether a compose
|
||||
implementation happens to be installed on top of the engine itself.
|
||||
|
||||
Usage:
|
||||
python3 tests/container.py # build the image, run everything inside it
|
||||
python3 tests/container.py --inside # (used by the container itself) actually run the suite
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
import tai # noqa: E402
|
||||
|
||||
IMAGE_TAG = "tai-test:latest"
|
||||
|
||||
# Run in this order: cheapest/most-fundamental first, so a basic break fails
|
||||
# fast instead of burning time on tmux/bashrc tests that assume the basics
|
||||
# already work. These must all pass for the container run to count as green.
|
||||
REQUIRED_SUITE = [
|
||||
("test_seal.py", ["python3", "test_seal.py"]),
|
||||
("test_tai.py", ["python3", "-m", "unittest", "test_tai", "-v"]),
|
||||
("tests/plexer.py", ["python3", "tests/plexer.py"]),
|
||||
("tests/bashrc.py", ["python3", "tests/bashrc.py"]),
|
||||
]
|
||||
|
||||
# tests/rag.py's RagIndexAndSearchTest needs a real, working LLM backend.
|
||||
# Confirmed by direct investigation (not assumed): inside this disposable
|
||||
# container there is no opencode/claude/codex/gemini/grok CLI (the host's
|
||||
# own successful run used a locally authenticated `opencode` CLI that has
|
||||
# no business being baked into a throwaway test image), and the remaining
|
||||
# free HTTP backends fail for reasons that are about the shared sandbox
|
||||
# network, not this code -- openrouter's free tier returned "HTTP 401: No
|
||||
# cookie auth credentials found" for every model (anonymous free-tier auth
|
||||
# needs a browser cookie session this container doesn't have), and
|
||||
# pollinations returned "HTTP 429: Queue full for IP: ...: 1 requests
|
||||
# already queued" (its free tier's per-IP concurrency cap, already
|
||||
# exhausted by other traffic sharing this sandbox's egress address). Both
|
||||
# reproduced with a bare urllib POST outside of tai.py entirely, so this is
|
||||
# not a tai.py bug to fix. It's run and reported like everything else, but
|
||||
# never fails the overall container result -- tests/rag.py already runs
|
||||
# (and passes) against the host directly, which is where a real LLM
|
||||
# backend actually exists; see tests/README.md's "test with the real AI,
|
||||
# accept the flakes" rule, which this is really just a container-flavored
|
||||
# instance of.
|
||||
BEST_EFFORT_SUITE = [
|
||||
("tests/rag.py", ["python3", "tests/rag.py"]),
|
||||
]
|
||||
|
||||
|
||||
def run_outside():
|
||||
engine = tai.container_engine()
|
||||
if not engine:
|
||||
print("no container engine found (checked podman, then docker) -- refusing to run the "
|
||||
"container-only test suite. Install either one to run tests/bashrc.py and the "
|
||||
"TmuxPlexerController live test; everything else already ran on the host directly.")
|
||||
return 1
|
||||
|
||||
print("using container engine: %s" % engine, flush=True)
|
||||
build = subprocess.run([engine, "build", "-t", IMAGE_TAG, "-f", "tests/docker/Dockerfile", "."], cwd=str(ROOT))
|
||||
if build.returncode != 0:
|
||||
print("image build failed", flush=True)
|
||||
return build.returncode
|
||||
|
||||
run = subprocess.run([engine, "run", "--rm", IMAGE_TAG, "python3", "tests/container.py", "--inside"], cwd=str(ROOT))
|
||||
return run.returncode
|
||||
|
||||
|
||||
def _run_suite(suite):
|
||||
results = []
|
||||
for name, cmd in suite:
|
||||
print("\n=== %s ===" % name)
|
||||
done = subprocess.run(cmd, cwd=str(ROOT))
|
||||
results.append((name, done.returncode == 0))
|
||||
print("--- %s: %s ---" % (name, "PASS" if done.returncode == 0 else "FAIL"))
|
||||
return results
|
||||
|
||||
|
||||
def run_inside():
|
||||
required = _run_suite(REQUIRED_SUITE)
|
||||
best_effort = _run_suite(BEST_EFFORT_SUITE)
|
||||
|
||||
print("\n=== container test summary ===")
|
||||
for name, passed in required:
|
||||
print("%-6s %s" % ("PASS" if passed else "FAIL", name))
|
||||
for name, passed in best_effort:
|
||||
print("%-6s %s (best-effort: needs a real LLM backend, see comment in this file)" % ("PASS" if passed else "FAIL", name))
|
||||
|
||||
failed_required = [name for name, passed in required if not passed]
|
||||
failed_best_effort = [name for name, passed in best_effort if not passed]
|
||||
total = len(required) + len(best_effort)
|
||||
passed_count = total - len(failed_required) - len(failed_best_effort)
|
||||
print("%d/%d passed" % (passed_count, total), end="")
|
||||
if failed_best_effort:
|
||||
print(" (ignoring best-effort failures: %s)" % ", ".join(failed_best_effort))
|
||||
else:
|
||||
print("")
|
||||
|
||||
if failed_required:
|
||||
print("REQUIRED tests failed: %s" % ", ".join(failed_required))
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--inside", action="store_true", help="internal: actually run the suite (used by the container itself)")
|
||||
args = parser.parse_args()
|
||||
if args.inside:
|
||||
return run_inside()
|
||||
return run_outside()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,30 @@
|
||||
# Minimal, disposable environment for the tests that cannot safely or
|
||||
# meaningfully run against a real machine: anything that installs the
|
||||
# plexer hook into a real ~/.bashrc, or needs a real tmux session to test
|
||||
# TmuxPlexerController against. Built fresh per run (see tests/container.py);
|
||||
# nothing here is meant to persist, which is the entire point -- these tests
|
||||
# get to do things no test should ever do to the machine actually running
|
||||
# Claude Code.
|
||||
FROM python:3.11-slim
|
||||
|
||||
# bash: the thing being wrapped. tmux: TmuxPlexerController's real backend.
|
||||
# procps: `ps`, used by tests/plexer.py and tests/bashrc.py to prove a
|
||||
# process is actually gone, not just assumed gone. git: sysinfo/create_skill
|
||||
# paths touch it; harmless to have for a "simulate everything" image.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends bash tmux procps git ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
COPY . /app
|
||||
|
||||
# Safety marker, not a feature flag: tests/bashrc.py refuses to install a
|
||||
# real hook into $HOME/.bashrc unless this is set, so a copy-paste of this
|
||||
# test file running directly on a developer's machine fails loud instead of
|
||||
# quietly rewriting their actual shell startup.
|
||||
ENV TAI_CONTAINER_TEST=1
|
||||
|
||||
# No ENTRYPOINT/CMD here on purpose -- tests/container.py always passes an
|
||||
# explicit command (python3 tests/container.py --inside ...), so this image
|
||||
# has one job and docker-compose.yml's `command:` is the single source of
|
||||
# truth for what that job is, not two places that can drift apart.
|
||||
+289
@@ -0,0 +1,289 @@
|
||||
#!/usr/bin/env python3
|
||||
"""tests/plexer.py — integration tests for tai's Plexer subsystem (NativePlexerBackend).
|
||||
|
||||
Deterministic throughout: this drives tai.py --plexer-wrap, a plumbing mode
|
||||
that forks a real shell onto a real pty with no LLM involvement at all, so
|
||||
"test with the real AI" (tests/README.md) doesn't apply here the way it does
|
||||
to tests/rag.py — there is no model decision anywhere in this path to test.
|
||||
What's real instead is the pty/signal/socket mechanics themselves: the whole
|
||||
point of this file is that it exercises the actual `tai.py --plexer-wrap`
|
||||
binary under a real pty (via tai_test.py's own pty mode — testing a pty
|
||||
wrapper by driving it through another pty), not mocked termios/signal calls.
|
||||
|
||||
A real bug was caught by actually running this, not by reading the code: an
|
||||
external SIGTERM (and SIGHUP) does not terminate an interactive bash running
|
||||
as a pty.fork() session leader in this environment -- verified with a bare
|
||||
pty.fork()+bash+os.kill script, independent of any tai.py code.
|
||||
NativePlexerBackend's Ctrl+A,q shortcut uses SIGKILL specifically because of
|
||||
this, not out of caution -- SIGTERM was tried first and silently did nothing
|
||||
for ~10s in a real run.
|
||||
"""
|
||||
import glob
|
||||
import os
|
||||
import shutil
|
||||
import struct
|
||||
import sys
|
||||
import tempfile
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from tai_test import TestCase, TestAssertionFailed, main_cli # noqa: E402
|
||||
import tai # noqa: E402
|
||||
|
||||
|
||||
async def _drain_forever(proc):
|
||||
"""A real terminal emulator is always reading, continuously, for as long
|
||||
as it's open -- that's what keeps a pty's output buffer from ever
|
||||
backing up. This test harness has to replicate that explicitly: without
|
||||
a background task doing this, NativePlexerBackend's relay loop can block
|
||||
on its own os.write(stdout_fd, ...) the moment a command's echoed output
|
||||
is large enough to fill the (unread) outer pty buffer, which stalls the
|
||||
*entire* relay loop -- including replying to later control-socket calls,
|
||||
which is what made this look like a control-protocol deadlock the first
|
||||
time this was debugged, when the actual missing piece was this drain.
|
||||
|
||||
A second real bug, found immediately after fixing the first: once the
|
||||
wrapped shell exits, the stream hits EOF and read() returns b"" forever
|
||||
without blocking -- so a plain `while True: await proc._read_chunk()`
|
||||
busy-spins at that point instead of idling, and spins hard enough to
|
||||
starve the event loop so badly that even asyncio.wait_for's own timeout
|
||||
machinery stops getting a fair turn. Hence the explicit EOF check."""
|
||||
try:
|
||||
while True:
|
||||
chunk = await proc._read_chunk()
|
||||
if not chunk:
|
||||
await asyncio.sleep(0.05)
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
|
||||
class NativePlexerBackendTest(TestCase):
|
||||
"""Spawns `tai.py --plexer-wrap` under a real pty (tai_test's pty mode),
|
||||
then drives it as an external controller would: over its control socket,
|
||||
plus raw keystrokes written straight to the outer pty master to exercise
|
||||
the Ctrl+A prefix-key path exactly as a human's terminal would deliver it."""
|
||||
|
||||
name = "plexer-native-backend"
|
||||
mode = "pty"
|
||||
timeout = 30.0
|
||||
|
||||
async def setup(self):
|
||||
self.tmp_home = tempfile.mkdtemp(prefix="tai-test-home-")
|
||||
self.env_overrides = {"TAI_HOME": self.tmp_home, "TAI_PASSPHRASE": ""}
|
||||
|
||||
async def run(self):
|
||||
proc = await self.spawn(extra_args=["--plexer-wrap"])
|
||||
self._drain_task = asyncio.ensure_future(_drain_forever(proc))
|
||||
sock_path = None
|
||||
for _ in range(50):
|
||||
matches = glob.glob(os.path.join(self.tmp_home, "plexer", "*.sock"))
|
||||
if matches:
|
||||
sock_path = matches[0]
|
||||
break
|
||||
await asyncio.sleep(0.2)
|
||||
if not sock_path:
|
||||
raise TestAssertionFailed("NativePlexerBackend never created its control socket")
|
||||
|
||||
controller = tai.NativePlexerController(sock_path)
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
status = controller.status()
|
||||
child_pid = status.get("pid")
|
||||
if not child_pid or child_pid <= 0:
|
||||
raise TestAssertionFailed("status() did not report a valid child pid: %r" % status)
|
||||
|
||||
controller.send_keys("echo plexer_test_marker", enter=True)
|
||||
await asyncio.sleep(0.4)
|
||||
after_send = controller.read_pane(50)
|
||||
if "plexer_test_marker" not in after_send:
|
||||
raise TestAssertionFailed("send_keys/read_pane round trip failed: %r" % after_send[-300:])
|
||||
|
||||
controller.type_text("echo typed_marker_ok", delay=0.01, enter=True)
|
||||
await asyncio.sleep(0.4)
|
||||
after_type = controller.read_pane(50)
|
||||
if "typed_marker_ok" not in after_type:
|
||||
raise TestAssertionFailed("type_text did not reach the shell: %r" % after_type[-300:])
|
||||
|
||||
import fcntl
|
||||
import termios
|
||||
fcntl.ioctl(proc._master_fd, termios.TIOCSWINSZ, struct.pack("HHHH", 17, 66, 0, 0))
|
||||
os.kill(proc.proc.pid, tai.signal.SIGWINCH)
|
||||
await asyncio.sleep(0.4)
|
||||
controller.send_keys("stty size", enter=True)
|
||||
await asyncio.sleep(0.4)
|
||||
after_resize = controller.read_pane(50)
|
||||
if "17 66" not in after_resize:
|
||||
raise TestAssertionFailed("SIGWINCH resize was not forwarded to the inner shell: %r" % after_resize[-300:])
|
||||
|
||||
controller.send_keys("echo contains_a_d_character", enter=True)
|
||||
await asyncio.sleep(0.3)
|
||||
after_plain = controller.read_pane(50)
|
||||
if "contains_a_d_character" not in after_plain:
|
||||
raise TestAssertionFailed("a plain command containing the letter 'd' was mangled by prefix-key handling")
|
||||
|
||||
os.write(proc._master_fd, tai.PLEXER_PREFIX_KEY)
|
||||
os.write(proc._master_fd, b"q")
|
||||
await proc.wait(timeout=10.0)
|
||||
await asyncio.sleep(0.2)
|
||||
socket_survived = os.path.exists(sock_path)
|
||||
|
||||
import subprocess
|
||||
still_running = subprocess.run(["ps", "-p", str(child_pid)], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL).returncode == 0
|
||||
|
||||
log_matches = glob.glob(os.path.join(self.tmp_home, "plexer", "*.log"))
|
||||
log_has_marker = False
|
||||
if log_matches:
|
||||
with open(log_matches[0], "rb") as handle:
|
||||
log_has_marker = b"plexer_test_marker" in handle.read()
|
||||
|
||||
return {
|
||||
"socket_survived": socket_survived,
|
||||
"still_running": still_running,
|
||||
"log_has_marker": log_has_marker,
|
||||
}
|
||||
|
||||
async def verify(self, result):
|
||||
if result["socket_survived"]:
|
||||
raise TestAssertionFailed("control socket file was not cleaned up after Ctrl+A,q")
|
||||
if result["still_running"]:
|
||||
raise TestAssertionFailed("the wrapped shell process was still alive after Ctrl+A,q (leaked process)")
|
||||
if not result["log_has_marker"]:
|
||||
raise TestAssertionFailed("the continuous session log did not capture the session's actual output")
|
||||
|
||||
async def teardown(self):
|
||||
self._drain_task.cancel()
|
||||
shutil.rmtree(self.tmp_home, ignore_errors=True)
|
||||
|
||||
|
||||
class NativePlexerBackendRobustnessTest(TestCase):
|
||||
"""Targets three specific bugs found and fixed during a deliberate review
|
||||
pass after NativePlexerBackendTest first went green -- a happy-path test
|
||||
passing does not mean the code is reliable, and all three of these would
|
||||
have sailed through that test unnoticed:
|
||||
|
||||
1. File permissions: the control socket and the session log (which
|
||||
captures the ENTIRE terminal session verbatim -- anything typed,
|
||||
including passwords) were being created with umask-default
|
||||
permissions, often world-readable/connectable on a multi-user box.
|
||||
2. The control protocol's server side did a single recv() instead of
|
||||
looping like the client side already did, silently truncating any
|
||||
request too large for one read -- untestable with short demo strings,
|
||||
which is exactly why the first test never caught it.
|
||||
3. A malformed (non-JSON-object) request on the control socket was not
|
||||
defensively handled and would have propagated an uncaught exception
|
||||
out of the relay loop, crashing the user's entire live wrapped
|
||||
terminal session over one bad control message.
|
||||
4. A genuine deadlock: plexer_send used to write its text directly into
|
||||
the pty master from inside the control-request handler. For a large
|
||||
payload on an echo-enabled pty, that write can block waiting for
|
||||
buffer space that only frees up once something reads the echoed
|
||||
output back out -- which is this very relay loop's own job, and it
|
||||
was the one blocked. Fixed by queuing all writes-to-child and only
|
||||
ever draining them from the main select() loop, cooperatively with
|
||||
reading the child's output, never as a blocking write from inside
|
||||
request handling.
|
||||
"""
|
||||
|
||||
name = "plexer-native-backend-robustness"
|
||||
mode = "pty"
|
||||
timeout = 30.0
|
||||
|
||||
async def setup(self):
|
||||
self.tmp_home = tempfile.mkdtemp(prefix="tai-test-home-")
|
||||
self.env_overrides = {"TAI_HOME": self.tmp_home, "TAI_PASSPHRASE": ""}
|
||||
|
||||
async def run(self):
|
||||
proc = await self.spawn(extra_args=["--plexer-wrap"])
|
||||
self._drain_task = asyncio.ensure_future(_drain_forever(proc))
|
||||
sock_path = None
|
||||
for _ in range(50):
|
||||
matches = glob.glob(os.path.join(self.tmp_home, "plexer", "*.sock"))
|
||||
if matches:
|
||||
sock_path = matches[0]
|
||||
break
|
||||
await asyncio.sleep(0.2)
|
||||
if not sock_path:
|
||||
raise TestAssertionFailed("NativePlexerBackend never created its control socket")
|
||||
controller = tai.NativePlexerController(sock_path)
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
plexer_dir_path = os.path.dirname(sock_path)
|
||||
log_matches = glob.glob(os.path.join(plexer_dir_path, "*.log"))
|
||||
if not log_matches:
|
||||
raise TestAssertionFailed("no log file found to check permissions on")
|
||||
log_path = log_matches[0]
|
||||
|
||||
dir_mode = os.stat(plexer_dir_path).st_mode & 0o777
|
||||
sock_mode = os.stat(sock_path).st_mode & 0o777
|
||||
log_mode = os.stat(log_path).st_mode & 0o777
|
||||
|
||||
# Large payload: bigger than the control protocol's 65536-byte recv
|
||||
# chunk size, to actually exercise the server-side framing loop
|
||||
# rather than the common case that fits in one read.
|
||||
needle_head = "LARGE_PAYLOAD_START_"
|
||||
needle_tail = "_LARGE_PAYLOAD_END"
|
||||
padding = "x" * 90000
|
||||
large_text = "echo %s%s%s" % (needle_head, padding, needle_tail)
|
||||
controller.send_keys(large_text, enter=True)
|
||||
await asyncio.sleep(0.6)
|
||||
large_result = controller.read_pane(2000)
|
||||
|
||||
# Malformed control request: connect directly to the socket (bypassing
|
||||
# the well-formed NativePlexerController entirely) and send something
|
||||
# that is valid JSON but not an object, which the old code would have
|
||||
# handed straight to request.get("cmd") and crashed on.
|
||||
import socket as socket_module
|
||||
import json as json_module
|
||||
raw_sock = socket_module.socket(socket_module.AF_UNIX, socket_module.SOCK_STREAM)
|
||||
raw_sock.settimeout(5)
|
||||
raw_sock.connect(sock_path)
|
||||
raw_sock.sendall(b"[1, 2, 3]\n")
|
||||
malformed_reply_raw = raw_sock.recv(65536)
|
||||
raw_sock.close()
|
||||
malformed_reply = json_module.loads(malformed_reply_raw.decode("utf-8", "replace"))
|
||||
|
||||
# The backend must still be alive and responsive after that malformed
|
||||
# request -- this is the actual point of the test, not the error
|
||||
# shape above.
|
||||
status_after = controller.status()
|
||||
|
||||
controller.send_keys("exit", enter=True)
|
||||
await proc.wait(timeout=10.0)
|
||||
|
||||
return {
|
||||
"dir_mode": dir_mode,
|
||||
"sock_mode": sock_mode,
|
||||
"log_mode": log_mode,
|
||||
"large_result": large_result,
|
||||
"malformed_reply": malformed_reply,
|
||||
"status_after_pid": status_after.get("pid"),
|
||||
}
|
||||
|
||||
async def verify(self, result):
|
||||
if result["dir_mode"] != 0o700:
|
||||
raise TestAssertionFailed("plexer dir mode is %o, expected 0700 (owner-only)" % result["dir_mode"])
|
||||
if result["sock_mode"] != 0o600:
|
||||
raise TestAssertionFailed("control socket mode is %o, expected 0600 (owner-only)" % result["sock_mode"])
|
||||
if result["log_mode"] != 0o600:
|
||||
raise TestAssertionFailed("session log mode is %o, expected 0600 (owner-only) -- it captures everything typed" % result["log_mode"])
|
||||
|
||||
large = result["large_result"]
|
||||
if "LARGE_PAYLOAD_START_" not in large or "_LARGE_PAYLOAD_END" not in large:
|
||||
raise TestAssertionFailed("large (>65536 byte) plexer_send payload was truncated by the control protocol")
|
||||
|
||||
if result["malformed_reply"].get("ok") is not False:
|
||||
raise TestAssertionFailed("a malformed control request did not come back as a clean error: %r" % result["malformed_reply"])
|
||||
|
||||
if not result["status_after_pid"] or result["status_after_pid"] <= 0:
|
||||
raise TestAssertionFailed("backend did not survive the malformed request -- a bad control message must never crash the whole wrapped session")
|
||||
|
||||
async def teardown(self):
|
||||
self._drain_task.cancel()
|
||||
shutil.rmtree(self.tmp_home, ignore_errors=True)
|
||||
|
||||
|
||||
ALL_CASES = [NativePlexerBackendTest(), NativePlexerBackendRobustnessTest()]
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main_cli(ALL_CASES, default_log_dir=str(Path(__file__).resolve().parent / "logs")))
|
||||
+268
@@ -0,0 +1,268 @@
|
||||
#!/usr/bin/env python3
|
||||
"""tests/rag.py — integration tests for tai's chunked FTS5/BM25 RAG system.
|
||||
|
||||
Built on tai_test.py (see tests/README.md for the conventions this file
|
||||
follows). Two test cases, deliberately different in how deterministic they
|
||||
are:
|
||||
|
||||
- InteractiveReplSmokeTest drives a persistent REPL over a real pty with
|
||||
nothing but REPL slash commands (/skills, /search). Slash commands dispatch
|
||||
tools directly, with no LLM call involved, so this test is fully
|
||||
deterministic and fast — it exists to prove the pty transport, the
|
||||
prompt-detection turn boundary, and the REPL plumbing all actually work,
|
||||
independent of model behavior.
|
||||
- RagIndexAndSearchTest is the real ask: index a sample of this project's own
|
||||
research/ archive into a fresh, isolated vault, then verify it comes back
|
||||
through search. Indexing (asking the agent to call record_save) goes
|
||||
through the LLM, which is inherently probabilistic — a free, rotating
|
||||
backend (see README.md's "Backends" section) can pick a different model
|
||||
per request. This test accepts that honestly: it reports how many of the
|
||||
sampled files were actually indexed rather than demanding all-or-nothing,
|
||||
and only hard-fails if indexing came back empty or the deterministic
|
||||
search step (itself a REPL slash command, no LLM) doesn't find anything
|
||||
that was actually indexed.
|
||||
"""
|
||||
import difflib
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from tai_test import TestCase, TestAssertionFailed, main_cli # noqa: E402
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
RESEARCH_DIR = ROOT / "research"
|
||||
REPL_PROMPT_RE = re.compile(r"tai\[[^\]]*\]\s*[›>]")
|
||||
SAMPLE_SIZE = int(os.environ.get("RAG_TEST_SAMPLE_SIZE", "5"))
|
||||
SOURCE_URL_RE = re.compile(r"\|\s*Source URL\s*\|\s*(\S+)\s*\|")
|
||||
|
||||
|
||||
def pick_sample_files(limit=SAMPLE_SIZE):
|
||||
# Only source dossiers/citation cards: each has exactly one "Source URL"
|
||||
# metadata field, which is what gives us a per-file, globally-unique
|
||||
# fingerprint. Conclusions cite several URLs each, so they don't fit
|
||||
# this scheme and are excluded.
|
||||
files = sorted(
|
||||
p for p in RESEARCH_DIR.rglob("*.md")
|
||||
if p.name not in ("README.md", "INDEX.md") and "conclusions" not in p.parts
|
||||
)
|
||||
return files[:limit]
|
||||
|
||||
|
||||
class InteractiveReplSmokeTest(TestCase):
|
||||
"""Deterministic: pure REPL slash commands, no LLM call, proves the pty transport."""
|
||||
|
||||
name = "repl-pty-smoke"
|
||||
mode = "pty"
|
||||
timeout = 30.0
|
||||
|
||||
async def setup(self):
|
||||
self.tmp_home = tempfile.mkdtemp(prefix="tai-test-home-")
|
||||
self.env_overrides = {"TAI_HOME": self.tmp_home, "TAI_PASSPHRASE": ""}
|
||||
|
||||
async def run(self):
|
||||
proc = await self.spawn(extra_args=["--profile", "smoketest"])
|
||||
await proc.wait_for_pattern(REPL_PROMPT_RE, timeout=15.0)
|
||||
await proc.send_line("/skills")
|
||||
skills_reply = await proc.wait_for_pattern(REPL_PROMPT_RE, timeout=15.0)
|
||||
await proc.send_line("/search nonexistent-term-xyz")
|
||||
search_reply = await proc.wait_for_pattern(REPL_PROMPT_RE, timeout=15.0)
|
||||
await proc.send_line("/quit")
|
||||
await proc.wait(timeout=10.0)
|
||||
return {"skills_reply": skills_reply, "search_reply": search_reply}
|
||||
|
||||
async def verify(self, result):
|
||||
if "skills" not in result["skills_reply"] and "no skills" not in result["skills_reply"]:
|
||||
raise TestAssertionFailed("/skills produced no recognizable output: %r" % result["skills_reply"][:300])
|
||||
if "no matches" not in result["search_reply"] and "error" in result["search_reply"].lower():
|
||||
raise TestAssertionFailed("/search errored instead of reporting no matches: %r" % result["search_reply"][:300])
|
||||
|
||||
async def teardown(self):
|
||||
shutil.rmtree(self.tmp_home, ignore_errors=True)
|
||||
|
||||
|
||||
class RagIndexAndSearchTest(TestCase):
|
||||
"""The real ask, proven with per-file traceability, not just 'something came back'.
|
||||
|
||||
For each sampled file this test:
|
||||
1. extracts that file's Source URL as a fingerprint, and confirms it's
|
||||
not shared by any other file in the sample (a non-unique fingerprint
|
||||
would make a search hit ambiguous as "proof");
|
||||
2. asks the agent to record_save it, exactly as before;
|
||||
3. reads the saved record straight out of the vault DB and diffs it
|
||||
against what was actually sent (not the whole file — what we asked
|
||||
the model to save verbatim), catching silent paraphrasing/truncation;
|
||||
4. confirms the fingerprint survived chunking, scoped to that record's
|
||||
own chunks specifically (not "some chunk somewhere");
|
||||
5. runs a /search for that one file's fingerprint and requires the
|
||||
reply to name that exact record's mem: id.
|
||||
|
||||
Every step is reported per file (see verify()'s failure message and the
|
||||
log), so a partial result is auditable rather than a single opaque
|
||||
pass/fail.
|
||||
"""
|
||||
|
||||
name = "rag-index-and-search"
|
||||
mode = "pipe"
|
||||
timeout = 60.0 * 8 # two LLM round trips per sampled file now (save + per-file search)
|
||||
|
||||
async def setup(self):
|
||||
self.tmp_home = tempfile.mkdtemp(prefix="tai-test-home-")
|
||||
self.env_overrides = {"TAI_HOME": self.tmp_home, "TAI_PASSPHRASE": ""}
|
||||
self.files = pick_sample_files()
|
||||
if not self.files:
|
||||
raise TestAssertionFailed("no documents found under research/ to index — did research/README.md's archive get built?")
|
||||
|
||||
async def run(self):
|
||||
fingerprints = self._extract_and_verify_unique_fingerprints(self.files)
|
||||
|
||||
per_file = []
|
||||
for path, fingerprint in fingerprints:
|
||||
sent_content = path.read_text(encoding="utf-8", errors="replace")[:3000]
|
||||
title = path.relative_to(RESEARCH_DIR).as_posix()
|
||||
prompt = (
|
||||
"Call the record_save tool right now with kind=\"research\", "
|
||||
"title=\"%s\", and content set to exactly the text below, verbatim. "
|
||||
"Do not summarize it, do not explain yourself, just call the tool once.\n\n%s"
|
||||
% (title, sent_content)
|
||||
)
|
||||
proc = await self.spawn(extra_args=["--yes", "--profile", "ragtest", prompt])
|
||||
await proc.drain_until_eof(timeout=90.0)
|
||||
await proc.wait(timeout=10.0)
|
||||
|
||||
record = self._find_record_by_title(title)
|
||||
fidelity = self._fidelity_ratio(sent_content, record["content"]) if record else 0.0
|
||||
chunk_has_fingerprint = self._record_chunk_contains(record["id"], fingerprint) if record else False
|
||||
|
||||
per_file.append({
|
||||
"title": title,
|
||||
"fingerprint": fingerprint,
|
||||
"saved": record is not None,
|
||||
"record_id": record["id"] if record else None,
|
||||
"fidelity": fidelity,
|
||||
"chunk_has_fingerprint": chunk_has_fingerprint,
|
||||
})
|
||||
|
||||
search_results = []
|
||||
for entry in per_file:
|
||||
if not entry["saved"]:
|
||||
continue
|
||||
proc = await self.spawn(extra_args=["--profile", "ragtest"], mode="pty")
|
||||
await proc.wait_for_pattern(REPL_PROMPT_RE, timeout=15.0)
|
||||
await proc.send_line("/search %s" % entry["fingerprint"])
|
||||
reply = await proc.wait_for_pattern(REPL_PROMPT_RE, timeout=15.0)
|
||||
await proc.send_line("/quit")
|
||||
await proc.wait(timeout=10.0)
|
||||
found_own_record = entry["record_id"] in reply
|
||||
search_results.append({"title": entry["title"], "reply": reply, "found_own_record": found_own_record})
|
||||
entry["search_found_own_record"] = found_own_record
|
||||
|
||||
return {"attempted": len(self.files), "per_file": per_file}
|
||||
|
||||
def _extract_and_verify_unique_fingerprints(self, files):
|
||||
pairs = []
|
||||
seen = {}
|
||||
for path in files:
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
match = SOURCE_URL_RE.search(text)
|
||||
if not match:
|
||||
raise TestAssertionFailed("%s has no 'Source URL' metadata row — pick_sample_files() should have excluded it" % path)
|
||||
fingerprint = match.group(1)
|
||||
if fingerprint in seen:
|
||||
raise TestAssertionFailed(
|
||||
"fingerprint %r is not unique: shared by %s and %s — cannot use it as proof of traceability"
|
||||
% (fingerprint, seen[fingerprint], path)
|
||||
)
|
||||
seen[fingerprint] = path
|
||||
pairs.append((path, fingerprint))
|
||||
return pairs
|
||||
|
||||
def _connect(self):
|
||||
return sqlite3.connect(str(Path(self.tmp_home) / "memory.db"))
|
||||
|
||||
def _find_record_by_title(self, title):
|
||||
conn = self._connect()
|
||||
try:
|
||||
row = conn.execute("SELECT id, content FROM records WHERE kind = 'research' AND title = ?", (title,)).fetchone()
|
||||
finally:
|
||||
conn.close()
|
||||
return {"id": row[0], "content": row[1]} if row else None
|
||||
|
||||
def _record_chunk_contains(self, record_id, fingerprint):
|
||||
conn = self._connect()
|
||||
try:
|
||||
rows = conn.execute("SELECT text FROM chunks WHERE parent_kind = 'record' AND parent_id = ?", (record_id,)).fetchall()
|
||||
finally:
|
||||
conn.close()
|
||||
return any(fingerprint in row[0] for row in rows)
|
||||
|
||||
@staticmethod
|
||||
def _fidelity_ratio(sent, saved):
|
||||
if not saved:
|
||||
return 0.0
|
||||
return difflib.SequenceMatcher(None, sent, saved).ratio()
|
||||
|
||||
async def verify(self, result):
|
||||
per_file = result["per_file"]
|
||||
report_lines = [
|
||||
"%-60s saved=%-5s fidelity=%.2f chunked=%-5s found_by_search=%-5s"
|
||||
% (e["title"], e["saved"], e["fidelity"], e["chunk_has_fingerprint"], e.get("search_found_own_record"))
|
||||
for e in per_file
|
||||
]
|
||||
report = "per-file breakdown:\n " + "\n ".join(report_lines)
|
||||
|
||||
saved_count = sum(1 for e in per_file if e["saved"])
|
||||
if saved_count == 0:
|
||||
raise TestAssertionFailed(
|
||||
"indexed 0 of %d sampled files (checked the vault's records table directly) — this may be "
|
||||
"free-backend flakiness, not a RAG bug; re-run before treating as a regression\n%s"
|
||||
% (result["attempted"], report)
|
||||
)
|
||||
|
||||
fully_proven = [
|
||||
e for e in per_file
|
||||
if e["saved"] and e["fidelity"] >= 0.90 and e["chunk_has_fingerprint"] and e.get("search_found_own_record")
|
||||
]
|
||||
if not fully_proven:
|
||||
raise TestAssertionFailed(
|
||||
"saved %d/%d file(s), but none passed the full chain (fidelity>=0.90 AND fingerprint survived "
|
||||
"chunking AND /search for that file's own fingerprint named that exact record) — no file is "
|
||||
"provably traceable end-to-end\n%s" % (saved_count, result["attempted"], report)
|
||||
)
|
||||
|
||||
low_fidelity = [e for e in per_file if e["saved"] and e["fidelity"] < 0.90]
|
||||
if low_fidelity:
|
||||
raise TestAssertionFailed(
|
||||
"%d/%d saved file(s) had content fidelity below 0.90 against what was actually sent — the model "
|
||||
"paraphrased or truncated instead of saving verbatim: %s\n%s"
|
||||
% (len(low_fidelity), saved_count, [e["title"] for e in low_fidelity], report)
|
||||
)
|
||||
|
||||
not_chunked = [e for e in per_file if e["saved"] and not e["chunk_has_fingerprint"]]
|
||||
if not_chunked:
|
||||
raise TestAssertionFailed(
|
||||
"%d/%d saved file(s) lost their fingerprint during chunking (present in the record but not in any "
|
||||
"of that record's own chunks): %s\n%s" % (len(not_chunked), saved_count, [e["title"] for e in not_chunked], report)
|
||||
)
|
||||
|
||||
not_found = [e for e in per_file if e["saved"] and not e.get("search_found_own_record")]
|
||||
if not_found:
|
||||
raise TestAssertionFailed(
|
||||
"%d/%d saved file(s) did not come back when searching for their own unique fingerprint: %s\n%s"
|
||||
% (len(not_found), saved_count, [e["title"] for e in not_found], report)
|
||||
)
|
||||
|
||||
print(report)
|
||||
|
||||
async def teardown(self):
|
||||
shutil.rmtree(self.tmp_home, ignore_errors=True)
|
||||
|
||||
|
||||
ALL_CASES = [InteractiveReplSmokeTest(), RagIndexAndSearchTest()]
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main_cli(ALL_CASES, default_log_dir=str(Path(__file__).resolve().parent / "logs")))
|
||||
Reference in New Issue
Block a user