From af080fe68e70929b86a6c20b15a2522d8f89f652 Mon Sep 17 00:00:00 2001 From: retoor Date: Sun, 18 Oct 2026 12:13:24 +0200 Subject: [PATCH] Strict goal verification, approval hardening, retrieval eval, work limits, sandbox caps (1.38.0) - Approval: re-prompt unknown answers (max 3), shell-only affirmative guidance recovery; destructive actions stay strict - GoalVerifier subagent with strict JSON verdicts, bounded follow-up rounds, nonzero exit plus open turn on unverified one-shots - Retrieval eval (Hit@K/MRR/nDCG/Context Precision), FTS cascade with title weighting, query/read event logging, --eval-retrieval CLI - Supervisor caps (8 parallel, 64 records) and scheduler hardening (4 fires/tick, per-schedule lease, stated misfire policy) - Sandbox box resource trio: 2g/2.0/256 at create, best-effort update for existing boxes, rm -f recovery pointer - Research archive: 12 subjects, 667 sources, 27 conclusions; suites 979 + 16 green --- .gitignore | 1 + AGENTS.md | 1 + README.md | 2 +- grrtui.md | 7 + mplex.md | 16 +- research/INDEX.md | 23 +- .../agent4novel-agent-tech-stack.md | 28 + .../ai-agents-pitstop-hitl-approval-setup.md | 28 + .../aitoolsvault-agent-frameworks-compared.md | 28 + .../anthropic-building-effective-agents.md | 54 + .../arxiv-2110-14168-training-verifiers.md | 28 + .../arxiv-2303-11366-reflexion.md | 28 + .../arxiv-2303-17651-self-refine.md | 28 + ...arxiv-2306-05685-judging-llm-as-a-judge.md | 53 + .../arxiv-2310-01798-cannot-self-correct.md | 51 + .../arxiv-2310-06770-swebench-paper.md | 28 + .../arxiv-2310-17631-judgelm.md | 28 + ...iv-2408-03314-test-time-compute-scaling.md | 52 + ...-14029-instance-level-judge-reliability.md | 28 + ...arxiv-2610-18108-calllm-judge-debiasing.md | 28 + .../changkun-human-in-the-loop-agents.md | 52 + .../conclusions/approval-input-hardening.md | 51 + .../prompt-only-json-strict-parse-retry.md | 51 + .../verifier-subagent-architecture.md | 57 + ...-docs-approval-queues-runtime-authority.md | 28 + .../deepy-system-audit-modes-design.md | 28 + ...devto-mattbusel-json-repetition-setting.md | 28 + ...dhyanendra-json-mode-structured-outputs.md | 28 + .../emrekuruu-user-in-the-loop-interfaces.md | 28 + .../fny666-approval-evidence-sources.md | 28 + .../hardw00t-defense-patterns-hitl.md | 28 + ...stshipai-human-in-the-loop-design-skill.md | 28 + ...ldin-call-me-maybe-constrained-decoding.md | 28 + .../langchain-reflection-agents.md | 51 + ...ukemurraynz-hitl-changelog-tiered-chain.md | 28 + ...lukemurraynz-hitl-design-patterns-skill.md | 28 + .../openai-agents-python-human-in-the-loop.md | 52 + .../openai-community-mia-agent-approvals.md | 28 + .../openai-evaluation-best-practices.md | 51 + .../openai-structured-outputs.md | 52 + .../pmq9-human-in-the-loop-workflows-skill.md | 28 + .../prajwalamte-hitl-governance-pattern.md | 28 + .../promptessor-hitl-approval-workflows.md | 28 + ...aralluri-prompt-engineering-json-levels.md | 28 + ...456-structured-output-reliability-skill.md | 28 + ...h-mohammad-openai-agents-sdk-production.md | 28 + .../strayspark-unreal-approval-workflow.md | 28 + ...structured-outputs-constrained-decoding.md | 28 + .../swebench-evaluation-guide.md | 50 + .../tkarim45-structured-output-eval.md | 51 + ...ai-structured-output-perfect-json-wrong.md | 28 + .../yinsongxu-llm2jev-request-to-model.md | 28 + .../apscheduler-user-guide.md | 51 + .../celery-periodic-tasks.md | 28 + .../scheduler-overlap-and-misfire.md | 52 + .../conclusions/supervisor-parallel-cap.md | 52 + .../background-work-limits/cron-manpage.md | 28 + .../erlang-otp-supervisor.md | 51 + .../background-work-limits/go-semaphore.md | 50 + .../python-concurrent-futures-deadlock.md | 28 + .../temporal-schedule.md | 51 + .../temporal-workflow-activity.md | 28 + .../trio-core-nursery.md | 28 + .../vorpus-structured-concurrency.md | 28 + ...agentmarketcap-memory-architecture-wars.md | 28 + ...gentpatterns-framework-landscape-memory.md | 28 + .../aithinkerlab-graphrag-ultimate-guide.md | 28 + ...anthropic-effective-context-engineering.md | 53 + .../archyde-graphrag-when-to-use.md | 28 + .../arxiv-2005-11401-rag-original.md | 28 + ...rxiv-2109-10862-recursive-summarization.md | 51 + .../arxiv-2307-03172-lost-in-the-middle.md | 50 + .../arxiv-2310-05736-llmlingua.md | 28 + .../arxiv-2404-06654-ruler.md | 50 + .../arxiv-2404-16130-graphrag-paper.md | 50 + .../arxiv-2504-19413-mem0-paper.md | 28 + .../conclusions/context-arsenal-verdict.md | 52 + .../no-full-graphrag-no-memory-vendor.md | 54 + .../spill-and-retrieve-giant-prompts.md | 53 + .../cursor-prompting-agents.md | 28 + .../cursor-subagents.md | 52 + .../dejokz-graphrag-hackathon.md | 28 + .../devto-agent-memory-benchmark-2026.md | 28 + .../devto-graphrag-paradox.md | 28 + .../devto-graphrag-pays-for-itself.md | 28 + .../jplevi-graphrag-vs-rag.md | 28 + .../landedjobs-advanced-rag-graphrag.md | 28 + .../langchain-deep-agents-rag.md | 28 + .../langchain-prebuilt-middleware.md | 53 + .../medium-mem0-vs-letta.md | 28 + .../medium-memory-frameworks-showdown.md | 28 + .../mem0-series-a-traction.md | 28 + .../mem0ai-mem0-readme.md | 53 + .../microsoft-graphrag-readme.md | 50 + .../openai-compaction-guide.md | 52 + .../pg-raggraph-ordering.md | 28 + .../sliamh11-deus-graphrag-comparison.md | 28 + .../sudhir13s-graph-rag-notes.md | 28 + ...reprompts-memory-architectures-compared.md | 28 + .../tokenjam-agent-memory.md | 28 + .../zephyrex-hybrid-memory-mem0.md | 28 + .../cherryservers-change-memory-limit.md | 28 + .../compose-mandatory-limits.md | 28 + .../conclusions/cap-box-trio.md | 58 + .../containerd-releases.md | 28 + .../devto-cgroups-fork-bomb.md | 28 + .../devto-operating-containers-limits.md | 28 + .../docker-manage-data.md | 28 + .../docker-resource-constraints.md | 52 + .../docker-tips-limit-resources.md | 28 + .../fork-bomb-pattern-notes.md | 28 + .../k8s-assign-memory.md | 28 + .../k8s-manage-resources.md | 28 + .../klangk-workspace-container-limits.md | 28 + .../medium-cpu-memory-production.md | 28 + .../medium-docker-resource-limits.md | 28 + .../medium-limit-cpu-memory.md | 28 + .../phoenixnap-docker-limits.md | 28 + .../podman-pids-default.md | 28 + .../podman-run-manpage.md | 50 + .../podman-update-manpage.md | 51 + .../runc-talks-listing.md | 28 + .../stackoverflow-running-limits.md | 28 + .../ai-engineering-book-hyde-lexical.md | 28 + .../arxiv-2010-13659-query-translation.md | 28 + .../arxiv-2012-14005-doc-expansion.md | 28 + .../arxiv-2104-08663-beir.md | 51 + .../arxiv-2309-15217-ragas.md | 51 + .../arxiv-2408-04197-pairwise-judgment.md | 28 + .../arxiv-2408-08067-ragchecker.md | 50 + .../arxiv-2511-19349-hyde-feedback.md | 28 + .../arxiv-2608-11238-qcare.md | 28 + .../conclusions/lexical-tuning-roadmap.md | 55 + .../conclusions/standing-retrieval-eval.md | 56 + .../deepeval-contextual-precision.md | 28 + .../deepeval-synthesizer-goldens.md | 49 + .../finalign-hybrid-qa-stack.md | 28 + .../hyde-paper-tables.md | 28 + .../ijcsit-web-ir-review.md | 28 + .../jev-tool-search-hyde-hurts-bm25.md | 28 + ...joachims-clickthrough-implicit-feedback.md | 51 + .../mdpi-rm3-hurts-short-queries.md | 28 + .../medium-hybrid-retrieval-hyde.md | 28 + .../muenster-hyde-thesis.md | 28 + .../patent-bm25-albert-feedback.md | 28 + .../radlinski-joachims-query-chains.md | 28 + .../ragas-context-precision-docs.md | 50 + .../researchgate-clickthrough-quality.md | 28 + .../researchgate-clickthrough-user-model.md | 28 + .../researchgate-ltr-clickthrough.md | 28 + .../sqlite-fts5-docs.md | 51 + .../uop-search-satisfaction-clicks.md | 28 + .../vespa-bm25-rank-feature.md | 28 + tai.py | 1677 ++++++++++++++++- test_tai.py | 1355 ++++++++++++- tests/plexer.py | 111 +- uiflow.md | 169 ++ 157 files changed, 8503 insertions(+), 70 deletions(-) create mode 100644 research/agent-goal-verification/agent4novel-agent-tech-stack.md create mode 100644 research/agent-goal-verification/ai-agents-pitstop-hitl-approval-setup.md create mode 100644 research/agent-goal-verification/aitoolsvault-agent-frameworks-compared.md create mode 100644 research/agent-goal-verification/anthropic-building-effective-agents.md create mode 100644 research/agent-goal-verification/arxiv-2110-14168-training-verifiers.md create mode 100644 research/agent-goal-verification/arxiv-2303-11366-reflexion.md create mode 100644 research/agent-goal-verification/arxiv-2303-17651-self-refine.md create mode 100644 research/agent-goal-verification/arxiv-2306-05685-judging-llm-as-a-judge.md create mode 100644 research/agent-goal-verification/arxiv-2310-01798-cannot-self-correct.md create mode 100644 research/agent-goal-verification/arxiv-2310-06770-swebench-paper.md create mode 100644 research/agent-goal-verification/arxiv-2310-17631-judgelm.md create mode 100644 research/agent-goal-verification/arxiv-2408-03314-test-time-compute-scaling.md create mode 100644 research/agent-goal-verification/arxiv-2610-14029-instance-level-judge-reliability.md create mode 100644 research/agent-goal-verification/arxiv-2610-18108-calllm-judge-debiasing.md create mode 100644 research/agent-goal-verification/changkun-human-in-the-loop-agents.md create mode 100644 research/agent-goal-verification/conclusions/approval-input-hardening.md create mode 100644 research/agent-goal-verification/conclusions/prompt-only-json-strict-parse-retry.md create mode 100644 research/agent-goal-verification/conclusions/verifier-subagent-architecture.md create mode 100644 research/agent-goal-verification/cycles-docs-approval-queues-runtime-authority.md create mode 100644 research/agent-goal-verification/deepy-system-audit-modes-design.md create mode 100644 research/agent-goal-verification/devto-mattbusel-json-repetition-setting.md create mode 100644 research/agent-goal-verification/dhyanendra-json-mode-structured-outputs.md create mode 100644 research/agent-goal-verification/emrekuruu-user-in-the-loop-interfaces.md create mode 100644 research/agent-goal-verification/fny666-approval-evidence-sources.md create mode 100644 research/agent-goal-verification/hardw00t-defense-patterns-hitl.md create mode 100644 research/agent-goal-verification/justshipai-human-in-the-loop-design-skill.md create mode 100644 research/agent-goal-verification/kheldin-call-me-maybe-constrained-decoding.md create mode 100644 research/agent-goal-verification/langchain-reflection-agents.md create mode 100644 research/agent-goal-verification/lukemurraynz-hitl-changelog-tiered-chain.md create mode 100644 research/agent-goal-verification/lukemurraynz-hitl-design-patterns-skill.md create mode 100644 research/agent-goal-verification/openai-agents-python-human-in-the-loop.md create mode 100644 research/agent-goal-verification/openai-community-mia-agent-approvals.md create mode 100644 research/agent-goal-verification/openai-evaluation-best-practices.md create mode 100644 research/agent-goal-verification/openai-structured-outputs.md create mode 100644 research/agent-goal-verification/pmq9-human-in-the-loop-workflows-skill.md create mode 100644 research/agent-goal-verification/prajwalamte-hitl-governance-pattern.md create mode 100644 research/agent-goal-verification/promptessor-hitl-approval-workflows.md create mode 100644 research/agent-goal-verification/rajsekharalluri-prompt-engineering-json-levels.md create mode 100644 research/agent-goal-verification/sayem7456-structured-output-reliability-skill.md create mode 100644 research/agent-goal-verification/sheikh-mohammad-openai-agents-sdk-production.md create mode 100644 research/agent-goal-verification/strayspark-unreal-approval-workflow.md create mode 100644 research/agent-goal-verification/sumansingh-structured-outputs-constrained-decoding.md create mode 100644 research/agent-goal-verification/swebench-evaluation-guide.md create mode 100644 research/agent-goal-verification/tkarim45-structured-output-eval.md create mode 100644 research/agent-goal-verification/towardsai-structured-output-perfect-json-wrong.md create mode 100644 research/agent-goal-verification/yinsongxu-llm2jev-request-to-model.md create mode 100644 research/background-work-limits/apscheduler-user-guide.md create mode 100644 research/background-work-limits/celery-periodic-tasks.md create mode 100644 research/background-work-limits/conclusions/scheduler-overlap-and-misfire.md create mode 100644 research/background-work-limits/conclusions/supervisor-parallel-cap.md create mode 100644 research/background-work-limits/cron-manpage.md create mode 100644 research/background-work-limits/erlang-otp-supervisor.md create mode 100644 research/background-work-limits/go-semaphore.md create mode 100644 research/background-work-limits/python-concurrent-futures-deadlock.md create mode 100644 research/background-work-limits/temporal-schedule.md create mode 100644 research/background-work-limits/temporal-workflow-activity.md create mode 100644 research/background-work-limits/trio-core-nursery.md create mode 100644 research/background-work-limits/vorpus-structured-concurrency.md create mode 100644 research/prompt-context-scaling/agentmarketcap-memory-architecture-wars.md create mode 100644 research/prompt-context-scaling/agentpatterns-framework-landscape-memory.md create mode 100644 research/prompt-context-scaling/aithinkerlab-graphrag-ultimate-guide.md create mode 100644 research/prompt-context-scaling/anthropic-effective-context-engineering.md create mode 100644 research/prompt-context-scaling/archyde-graphrag-when-to-use.md create mode 100644 research/prompt-context-scaling/arxiv-2005-11401-rag-original.md create mode 100644 research/prompt-context-scaling/arxiv-2109-10862-recursive-summarization.md create mode 100644 research/prompt-context-scaling/arxiv-2307-03172-lost-in-the-middle.md create mode 100644 research/prompt-context-scaling/arxiv-2310-05736-llmlingua.md create mode 100644 research/prompt-context-scaling/arxiv-2404-06654-ruler.md create mode 100644 research/prompt-context-scaling/arxiv-2404-16130-graphrag-paper.md create mode 100644 research/prompt-context-scaling/arxiv-2504-19413-mem0-paper.md create mode 100644 research/prompt-context-scaling/conclusions/context-arsenal-verdict.md create mode 100644 research/prompt-context-scaling/conclusions/no-full-graphrag-no-memory-vendor.md create mode 100644 research/prompt-context-scaling/conclusions/spill-and-retrieve-giant-prompts.md create mode 100644 research/prompt-context-scaling/cursor-prompting-agents.md create mode 100644 research/prompt-context-scaling/cursor-subagents.md create mode 100644 research/prompt-context-scaling/dejokz-graphrag-hackathon.md create mode 100644 research/prompt-context-scaling/devto-agent-memory-benchmark-2026.md create mode 100644 research/prompt-context-scaling/devto-graphrag-paradox.md create mode 100644 research/prompt-context-scaling/devto-graphrag-pays-for-itself.md create mode 100644 research/prompt-context-scaling/jplevi-graphrag-vs-rag.md create mode 100644 research/prompt-context-scaling/landedjobs-advanced-rag-graphrag.md create mode 100644 research/prompt-context-scaling/langchain-deep-agents-rag.md create mode 100644 research/prompt-context-scaling/langchain-prebuilt-middleware.md create mode 100644 research/prompt-context-scaling/medium-mem0-vs-letta.md create mode 100644 research/prompt-context-scaling/medium-memory-frameworks-showdown.md create mode 100644 research/prompt-context-scaling/mem0-series-a-traction.md create mode 100644 research/prompt-context-scaling/mem0ai-mem0-readme.md create mode 100644 research/prompt-context-scaling/microsoft-graphrag-readme.md create mode 100644 research/prompt-context-scaling/openai-compaction-guide.md create mode 100644 research/prompt-context-scaling/pg-raggraph-ordering.md create mode 100644 research/prompt-context-scaling/sliamh11-deus-graphrag-comparison.md create mode 100644 research/prompt-context-scaling/sudhir13s-graph-rag-notes.md create mode 100644 research/prompt-context-scaling/sureprompts-memory-architectures-compared.md create mode 100644 research/prompt-context-scaling/tokenjam-agent-memory.md create mode 100644 research/prompt-context-scaling/zephyrex-hybrid-memory-mem0.md create mode 100644 research/sandbox-resource-caps/cherryservers-change-memory-limit.md create mode 100644 research/sandbox-resource-caps/compose-mandatory-limits.md create mode 100644 research/sandbox-resource-caps/conclusions/cap-box-trio.md create mode 100644 research/sandbox-resource-caps/containerd-releases.md create mode 100644 research/sandbox-resource-caps/devto-cgroups-fork-bomb.md create mode 100644 research/sandbox-resource-caps/devto-operating-containers-limits.md create mode 100644 research/sandbox-resource-caps/docker-manage-data.md create mode 100644 research/sandbox-resource-caps/docker-resource-constraints.md create mode 100644 research/sandbox-resource-caps/docker-tips-limit-resources.md create mode 100644 research/sandbox-resource-caps/fork-bomb-pattern-notes.md create mode 100644 research/sandbox-resource-caps/k8s-assign-memory.md create mode 100644 research/sandbox-resource-caps/k8s-manage-resources.md create mode 100644 research/sandbox-resource-caps/klangk-workspace-container-limits.md create mode 100644 research/sandbox-resource-caps/medium-cpu-memory-production.md create mode 100644 research/sandbox-resource-caps/medium-docker-resource-limits.md create mode 100644 research/sandbox-resource-caps/medium-limit-cpu-memory.md create mode 100644 research/sandbox-resource-caps/phoenixnap-docker-limits.md create mode 100644 research/sandbox-resource-caps/podman-pids-default.md create mode 100644 research/sandbox-resource-caps/podman-run-manpage.md create mode 100644 research/sandbox-resource-caps/podman-update-manpage.md create mode 100644 research/sandbox-resource-caps/runc-talks-listing.md create mode 100644 research/sandbox-resource-caps/stackoverflow-running-limits.md create mode 100644 research/sqlite-rag-hybrid-search/ai-engineering-book-hyde-lexical.md create mode 100644 research/sqlite-rag-hybrid-search/arxiv-2010-13659-query-translation.md create mode 100644 research/sqlite-rag-hybrid-search/arxiv-2012-14005-doc-expansion.md create mode 100644 research/sqlite-rag-hybrid-search/arxiv-2104-08663-beir.md create mode 100644 research/sqlite-rag-hybrid-search/arxiv-2309-15217-ragas.md create mode 100644 research/sqlite-rag-hybrid-search/arxiv-2408-04197-pairwise-judgment.md create mode 100644 research/sqlite-rag-hybrid-search/arxiv-2408-08067-ragchecker.md create mode 100644 research/sqlite-rag-hybrid-search/arxiv-2511-19349-hyde-feedback.md create mode 100644 research/sqlite-rag-hybrid-search/arxiv-2608-11238-qcare.md create mode 100644 research/sqlite-rag-hybrid-search/conclusions/lexical-tuning-roadmap.md create mode 100644 research/sqlite-rag-hybrid-search/conclusions/standing-retrieval-eval.md create mode 100644 research/sqlite-rag-hybrid-search/deepeval-contextual-precision.md create mode 100644 research/sqlite-rag-hybrid-search/deepeval-synthesizer-goldens.md create mode 100644 research/sqlite-rag-hybrid-search/finalign-hybrid-qa-stack.md create mode 100644 research/sqlite-rag-hybrid-search/hyde-paper-tables.md create mode 100644 research/sqlite-rag-hybrid-search/ijcsit-web-ir-review.md create mode 100644 research/sqlite-rag-hybrid-search/jev-tool-search-hyde-hurts-bm25.md create mode 100644 research/sqlite-rag-hybrid-search/joachims-clickthrough-implicit-feedback.md create mode 100644 research/sqlite-rag-hybrid-search/mdpi-rm3-hurts-short-queries.md create mode 100644 research/sqlite-rag-hybrid-search/medium-hybrid-retrieval-hyde.md create mode 100644 research/sqlite-rag-hybrid-search/muenster-hyde-thesis.md create mode 100644 research/sqlite-rag-hybrid-search/patent-bm25-albert-feedback.md create mode 100644 research/sqlite-rag-hybrid-search/radlinski-joachims-query-chains.md create mode 100644 research/sqlite-rag-hybrid-search/ragas-context-precision-docs.md create mode 100644 research/sqlite-rag-hybrid-search/researchgate-clickthrough-quality.md create mode 100644 research/sqlite-rag-hybrid-search/researchgate-clickthrough-user-model.md create mode 100644 research/sqlite-rag-hybrid-search/researchgate-ltr-clickthrough.md create mode 100644 research/sqlite-rag-hybrid-search/sqlite-fts5-docs.md create mode 100644 research/sqlite-rag-hybrid-search/uop-search-satisfaction-clicks.md create mode 100644 research/sqlite-rag-hybrid-search/vespa-bm25-rank-feature.md mode change 100755 => 100644 tai.py create mode 100644 uiflow.md diff --git a/.gitignore b/.gitignore index ff1f8b5..6592aef 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ *.html *.md !research/**/*.md +!uiflow.md # Backup files *.bak diff --git a/AGENTS.md b/AGENTS.md index d27506c..8b28532 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,6 +9,7 @@ - The in-`tai.py` TUI framework is documented in [tui.md](tui.md) — read it before touching any `Tui*` code. - The grr terminal client (`tai.py --tui`, the `Grr*` classes) is specified in [grrtui.md](grrtui.md): read it before touching any `Grr*` code, and keep its screen, key and cap tables in step with the code (`GrrSpecTests` fails when they drift). It is a client of the grr server and shows what the server's `/api/spec` and `/locales.json` say; never copy a string, color or limit of the server into `tai.py`. `tests/grr_tui.py` drives the real binary against a real grr server (sibling `../grr`, `GRR_DIR` to move it) and a real model. - The terminal multiplexer (`Mplex*` classes, `tai.py --mplex`, `~/.tai/mplex.toml`) is specified in [mplex.md](mplex.md): read it before touching any `Mplex*` code, and keep its option, command and key tables in step with the code (`MplexSpecTests` fails when they drift). +- Interactive UI flows (`plexer_show`, `PlexerFlow`, `PlexerController.ask`, the `show`/`result`/`cancel` control commands) are specified in [uiflow.md](uiflow.md): read it before touching any flow code, and keep its contract and cap tables in step with the code (`UiflowDocTests` fails when they drift). ## Testing diff --git a/README.md b/README.md index 48c6cf3..a8d19d6 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,7 @@ A plexer session is a terminal tai can see into and type into, not just read a o - **tmux** — if the shell is already inside a tmux pane (`$TMUX_PANE` set), that pane *is* the session. tai drives it with plain `tmux send-keys`/`capture-pane`/`list-panes`; nothing is spawned. - **native multiplexer (mplex)**: if nothing already owns the terminal, the plexer-hook/`zsh-plexer-hook`/`fish-plexer-hook` installed above re-execs the fresh interactive shell into `tai.py --plexer-wrap`, which starts (or joins) tai's own tmux-style multiplexer and attaches the terminal to a new session. One detached server per `TAI_HOME` owns every pty, window layout and scrollback; the terminal is a thin client, so detaching leaves everything running and `tai.py --mplex attach` brings it back. Every pane's output is also written to a log under `~/.tai/plexer/`, and the server answers control commands on a Unix socket (`TAI_PLEXER_SOCKET`), with the pane id in `TAI_PLEXER_PANE`, so a separate tai process (the agent the user is actually talking to) can drive the pane it was started in. -Either way, four tools appear only while a session is actually active (`plexer_send`, `plexer_read`, `plexer_type`, `plexer_status` — hidden otherwise, not just disabled, via `TOOL_ENV_GATES`): `plexer_send` types literal keystrokes (optionally pressing Enter), `plexer_read` reads back the last N lines of real rendered output, `plexer_type` does the same as `send` but one character at a time with a delay for a visible typing effect, and `plexer_status` reports which backend is active and its details (pid/socket/log, or tmux pane id). Which backend answers a given call is autodetected from the environment (`TAI_PLEXER_SOCKET` vs `$TMUX_PANE`), never configured by hand. +Either way, five tools appear only while a session is actually active (`plexer_send`, `plexer_read`, `plexer_type`, `plexer_status`, `plexer_show`, hidden otherwise, not just disabled, via `TOOL_ENV_GATES`): `plexer_send` types literal keystrokes (optionally pressing Enter), `plexer_read` reads back the last N lines of real rendered output, `plexer_type` does the same as `send` but one character at a time with a delay for a visible typing effect, `plexer_status` reports which backend is active and its details (pid/socket/log, or tmux pane id), and `plexer_show` renders a whole interactive UI flow (dialogs, forms, branching multi-screen apps) from one JSON spec and returns the outcome. Which backend answers a given call is autodetected from the environment (`TAI_PLEXER_SOCKET` vs `$TMUX_PANE`), never configured by hand. The flow contract is [uiflow.md](uiflow.md). ### mplex: the multiplexer diff --git a/grrtui.md b/grrtui.md index ab0b459..df928bf 100644 --- a/grrtui.md +++ b/grrtui.md @@ -101,6 +101,10 @@ renderer of the chat scene (headings, bold, inline and fenced code, lists, quote than 30 minutes passed, `NEW` marks what you have not read. Under the newest bot message, and under any message you select, sit copy, good, bad and try again; steps fold open with Enter. +A pending approval is also asked at once as a dialog (Deny, Allow once, Always allow for the browser tool) the moment it arrives or the chat opens, without blocking anything: Esc leaves the card, a decision made elsewhere closes the dialog. + +A running command that goes quiet on a prompt-like last row (a password, a `?`, `[y/N]`) opens its terminal by itself, so the prompt reaches the user and the answer is typed into the command; Esc leaves it. + Cards: approvals (Allow once, Deny, Always allow in browser for the browser tool, then the state), questions (lettered options and Skip), files (Enter opens the viewer: text with line numbers, anything else is saved and opened by the desktop), several files, templates (publish, copy link, details), and live process cards. A running command's card @@ -184,6 +188,9 @@ Every limit is a constant in `tai.py`. | `GRR_COMPOSER_MAX_ROWS` | 6 | rows the composer grows to | | `GRR_VIEWER_MAX_BYTES` | 2097152 | bytes the file viewer reads | | `GRR_DRAFT_KEEP` | 64 | chats with a kept draft | +| `GRR_PROMPT_QUIET` | 0.8 | seconds of silence on a prompt-like last row before a running command opens its terminal | +| `GRR_PROMPT_ASKS` | 20 | times one process card opens its terminal by itself | +| `GRR_PROMPT_MAX` | 8 | pending approvals asked at once as dialogs in one chat | | `GRR_LIST_MORE` | 12 | rows Show more adds | | `GRR_MENU_MAX_H` | 14 | rows of a menu before it scrolls | | `GRR_CONTROL_JOBS` | 64 | control jobs kept (finished ones drop oldest first) | diff --git a/mplex.md b/mplex.md index 4e9b929..7fc8910 100644 --- a/mplex.md +++ b/mplex.md @@ -74,6 +74,19 @@ The same socket serves `NativePlexerController` (the agent tools). One request l error. `read` returns the tail of the pane's raw byte log. A malformed request returns `{"ok": false, "error": ...}` and never disturbs the server. +Three more commands render one UI screen for the `plexer_show` agent tool (the +full flow contract is [uiflow.md](uiflow.md)): `{"cmd": "show", "screen": {...}, +"timeout": 60}` draws the screen on the first attached client of the pane's +session and returns `{"id": N}`; `{"cmd": "result", "id": N}` polls it +(`{"done": false}` until the user answers, then `{"done": true, "status": +"ok"/"cancelled"/"timeout"/"detached", "value": ...}`, collected once); +`{"cmd": "cancel", "id": N}` drops it. Kind `choice`/`message` renders as a +centered overlay dialog, kind `text` as a status-row prompt. `show` is refused +loudly when the screen is malformed, the client is busy with another prompt or +overlay, the session has no attached client, or `MPLEX_FLOW_MAX` flows are +already pending. Records whose driver died are reclaimed after +`MPLEX_FLOW_KEEP` seconds, on every show and every server tick. + ### Hard caps | Constant | Value | Limits | @@ -93,6 +106,7 @@ and never disturbs the server. | `MPLEX_PASTE_MAX` | 4 MiB | one paste buffer | | `MPLEX_RENDER_INTERVAL` | 12 ms | minimum time between frames for a client | | `MPLEX_REAP_KILL_AFTER` | 2 s | SIGHUP is escalated to SIGKILL for a pane process that ignored it | +| `MPLEX_FLOW_MAX` / `MPLEX_FLOW_KEEP` | 16 / 300 s | pending UI flows per server, grace before a dead driver's record is reclaimed | ## Keys @@ -102,7 +116,7 @@ Key names use tmux notation: `a`, `C-a`, `M-a`, `C-M-a`, `S-Up`, `Space`, `Enter Lookup order for every key from an attached client: -1. an open prompt, then an open overlay (`display-panes`), take the key; +1. an open prompt, then an open overlay (`display-panes`, a UI flow dialog), take the key; 2. in the `root` table: a binding runs; 3. the prefix key or `prefix2` switches the client to the `prefix` table; the next key is looked up there (an unbound key is dropped); 4. otherwise the top mode of the pane takes it (copy mode uses `copy-mode` or `copy-mode-vi` according to `mode-keys`; a diff --git a/research/INDEX.md b/research/INDEX.md index 4712836..263b806 100644 --- a/research/INDEX.md +++ b/research/INDEX.md @@ -6,19 +6,19 @@ | Metric | Value | |---|---| -| Subjects | 8 | -| Total source documents | 531 | -| — Tier 1 (Source Dossier, independently analyzed) | 158 | -| — Tier 2 (Source Citation Card, listed only) | 373 | -| Total conclusion documents | 16 | +| Subjects | 12 | +| Total source documents | 667 | +| — Tier 1 (Source Dossier, independently analyzed) | 193 | +| — Tier 2 (Source Citation Card, listed only) | 474 | +| Total conclusion documents | 27 | | Archive opened | 2026-10-09 | -| Last updated | 2026-10-12 | +| Last updated | 2026-10-18 | ## Subjects | Subject | Sources (T1 / T2) | Conclusions | Opened | Last updated | Driving task | |---|---|---|---|---|---| -| [`sqlite-rag-hybrid-search`](sqlite-rag-hybrid-search/) | 32 (10 / 22) | 1 | 2026-10-09 | 2026-10-09 | Designing tmux-log export + chunked retrieval; settled on BM25-only, no vector DB | +| [`sqlite-rag-hybrid-search`](sqlite-rag-hybrid-search/) | 60 (17 / 43) | 3 | 2026-10-09 | 2026-10-18 | Designing tmux-log export + chunked retrieval; settled on BM25-only, no vector DB; extended 2026-10-18 with retrieval-quality measurement and a gated lexical tuning roadmap | | [`agent-memory-recency-tracking`](agent-memory-recency-tracking/) | 63 (16 / 47) | 1 | 2026-10-09 | 2026-10-09 | Designing an LRU-style last-used/use-count tracker for vault records and chunks | | [`async-pty-process-testing`](async-pty-process-testing/) | 59 (20 / 39) | 1 | 2026-10-09 | 2026-10-09 | Designing tai_test.py (pty vs. pipes, non-blocking stdin, process-group cleanup, test-framework shape) — backfilled after it was initially skipped, see activity log | | [`shell-multiplexer-auto-attach`](shell-multiplexer-auto-attach/) | 139 (49 / 90) | 3 | 2026-10-09 | 2026-10-12 | Designing a tai-managed terminal multiplexer + bashrc auto-attach: tmux-as-substrate recommendation, interactive-vs-automation shell detection; extended 2026-10-10 to cover zsh/fish hook syntax; extended 2026-10-12 with the tmux 3.6 key table, option and command dumps (installed binary) and the conclusion to build a native tmux-parity multiplexer configured by ~/.tai/mplex.toml | @@ -26,6 +26,10 @@ | [`agentic-tool-loop-design`](agentic-tool-loop-design/) | 74 (7 / 67) | 2 | 2026-10-11 | 2026-10-11 | Fixing stop-after-narration on prompt-JSON CLI backends without intent regexes; decided structural loop control plus exactly-once JSON-contract repair plus prompt contract, implemented in tai.py the same session (suite 357/357 + 16/16 green); approval tiers judged already industry-standard, no change | | [`muse-responses-openai-transpiler`](muse-responses-openai-transpiler/) | 66 (3 / 63) | 2 | 2026-10-11 | 2026-10-11 | Documenting Meta muse Responses wire format vs classic Chat Completions; decided bidirectional transpile is feasible with a fixed loss table and system guidance has three channels; scratch transpiler tested, nothing merged | | [`free-model-discovery`](free-model-discovery/) | 4 (4 / 0) | 1 | 2026-10-12 | 2026-10-12 | Bringing tai's free model lists up to date from live endpoints (zen, openrouter, devplace, pollinations); live discovery replaces hard-coded lists | +| [`agent-goal-verification`](agent-goal-verification/) | 43 (11 / 32) | 3 | 2026-10-18 | 2026-10-18 | Validating whether the GoalVerifier subagent plus strict JSON verdicts plus approval hardening scales and is solid; kept all three as shipped with stated conditions | +| [`prompt-context-scaling`](prompt-context-scaling/) | 34 (10 / 24) | 3 | 2026-10-18 | 2026-10-18 | Scaling prompts past the context window and reusing RAG/graph for more; tai already matches the converged arsenal, giant prompts should spill to retrievable records (specified, not built), full GraphRAG and memory vendors rejected | +| [`background-work-limits`](background-work-limits/) | 10 (4 / 6) | 2 | 2026-10-18 | 2026-10-18 | Bounding parallel subagents and hardening the scheduler; parallel cap 8 with reject-loud, per-tick fires 4, one live run per schedule via lease, stated misfire policy; implemented same day | +| [`sandbox-resource-caps`](sandbox-resource-caps/) | 21 (3 / 18) | 1 | 2026-10-18 | 2026-10-18 | Capping the agent sandbox box CPU/memory/pids after the coverage audit found it uncapped; static trio 2g/2.0/256 at create, best-effort update for existing boxes, pids via recreation; implemented same day | ## Recent activity log @@ -48,6 +52,11 @@ | 2026-10-11 | `tui-framework-design` | Extended with 16 more sources (all Tier 1, no Tier 2 — every consulted source body was fetched by the researching agents) + 2 conclusions (`conclusions/telegram-chat-grammar-adoption.md`, `conclusions/telegram-theme-roles-mapping.md`): 10 dossiers on Telegram chat look-and-feel (tdesktop day/night theme files, lang.strings copy sheet, Android DialogCell/ChatMessageCell, FAQ check semantics, chat-themes blog, third-party checkmarks explainer, Bot API keyboards/commands, bot features onboarding) and 6 on Mini App theming (official WebApps ThemeParams/events/design rules; SDK docs for main/back, settings, viewport, theming; third-party iOS teardown). Decisions: adopt Telegram's metadata grammar as rows (clock/checks/blue-checks/eye, no delivered state), 8-hue peer palette, service/date pills, literal status/composer/bot copy, and reply-vs-inline keyboard grammar, rejecting bubbles, gradient themes, and pixel geometry; map the 15 ThemeParams keys onto tai's 12 roles with Desktop-file plus dark-sample defaults and a never-hardcode/re-read-on-switch discipline. A same-session internal tai.py inventory (chat scene present but plain: no ticks/dividers/header/rich rows/composer buttons; 6 of 12 roles used, hardcoded borders) informed both conclusions with per-item patch targets but produced no source documents (internal code, not external research). Nothing built yet — both conclusions are specified-not-shipped. Carried gaps: iOS/Web hexes unverified (Desktop day/night only); bubble radii/paddings/spacing/tail geometry, header icon inventory, chat-list time/date formatting and truncation, send-vs-voice swap and media-bubble time overlay unknown; no official typography/spacing spec (teardown values only); light defaults for section_*/subtitle/destructive unpublished; WebViewColors diagram uninspected; back-button SDK page yielded no captured evidence. Subject total now 94 sources (49 Tier 1 / 45 Tier 2), 5 conclusions. Counts verified directly against the written files via `research/audit.py` (PASS). | | 2026-10-11 | `muse-responses-openai-transpiler` | New subject: 66 sources (3 Tier 1 / 63 Tier 2) + 2 conclusions, covering the Meta muse Responses wire format vs classic Chat Completions (official migration guide, Responses reference, Chat reference fetched and synthesized; 63 search listings logged as seen-not-analyzed). Decisions: bidirectional transpile is feasible with a fixed warn-on-drop loss table, and system guidance has three channels (instructions preferred, developer/system Items otherwise). Backed by a scratch stdlib transpiler (`/tmp/openai_transpile.py`, self-test green, real-capture round trip verified); nothing merged into tai.py. Counts verified directly against the written files via `research/audit.py` (PASS). | | 2026-10-12 | `shell-multiplexer-auto-attach` | Extended with 1 source (Tier 1: output of the installed tmux 3.6, `list-keys`, `list-keys -N`, `show-options`, `list-commands`) + 1 conclusion (`conclusions/tmux-parity-native-multiplexer.md`): the native multiplexer gets tmux's default key tables, command and option names, configured by `~/.tai/mplex.toml`; the earlier "build on tmux" recommendation stays for the tmux controller and detection rules. Behavior was also compared command by command against the same binary (layouts, windows, copy mode), see `mplex.md`. | +| 2026-10-18 | `agent-goal-verification` | New subject: 43 sources (11 Tier 1 / 32 Tier 2) + 3 conclusions, covering LLM-judge reliability and biases, test-time verification scaling, evaluator-optimizer and reflection patterns, test-based verification, structured-output reliability tiers, and approval-UX practice (2 gather subagents plus first-hand fetches of all Tier 1 pages). Decisions: keep the verifier-subagent architecture with bounded rounds, keep prompt-only JSON with strict parse and retry, keep re-prompt plus shell-only exact-match affirm recovery. Counts verified directly against the written files via `research/audit.py` (PASS). | +| 2026-10-18 | `prompt-context-scaling` | New subject: 34 sources (10 Tier 1 / 24 Tier 2) + 3 conclusions, covering context-window management (compaction, just-in-time retrieval, notes, subagent isolation), long-context limits, map-reduce over long inputs, GraphRAG scope and cost, and memory layers (2 gather subagents plus first-hand fetches of all Tier 1 pages). Decisions: tai's arsenal already matches the converged set; oversized prompts should spill to records and answer via retrieval (specified, not built); full GraphRAG and memory vendors rejected in favor of growing the vault graph. Counts verified directly against the written files via `research/audit.py` (PASS). | +| 2026-10-18 | `sqlite-rag-hybrid-search` | Extended with 28 sources (7 Tier 1 / 21 Tier 2) + 2 conclusions (`conclusions/standing-retrieval-eval.md`, `conclusions/lexical-tuning-roadmap.md`), closing the original conclusion's open risk on unmeasured retrieval quality: rank metrics (Hit@K/MRR/nDCG/Context Precision), goldens generalized from the fingerprint method, implicit search-then-read preferences, query logging gap, and an ordered tuning backlog (column weights, OR-fallback, prefix) with HyDE/RM3/embeddings rejected. Counts verified directly against the written files via `research/audit.py` (PASS). | +| 2026-10-18 | `background-work-limits` | New subject: 10 sources (4 Tier 1 / 6 Tier 2) + 2 conclusions, covering supervisor bounds (OTP intensity, semaphore reject-loud) and scheduler policies (APScheduler max_instances/coalesce, Temporal Skip, cron) after a feature-coverage audit found orchestration and scheduling unresearched. Decisions implemented same day: AGENTS_MAX_PARALLEL 8 with AgentsFull refusal, 64-record retention with oldest-collected purge, 4 fires per tick, one live run per schedule via last_status lease, stated misfire policy, no auto-retry. Counts verified directly against the written files via `research/audit.py` (PASS). | +| 2026-10-18 | `sandbox-resource-caps` | New subject: 21 sources (3 Tier 1 / 18 Tier 2) + 1 conclusion, covering container resource caps (Docker limits reference, podman-run flag parity, podman-update create-only pids gap) plus snippet corroboration of the memory/cpus/pids trio after the coverage audit found the sandbox box fully uncapped. Decision implemented same day: static `--memory=2g --cpus=2.0 --pids-limit=256` at box create for both engines, one best-effort `update` (memory plus cpus) per process for existing boxes, concrete `rm -f` recovery pointer on failure. Counts verified directly against the written files via `research/audit.py` (PASS). | ## How to use this index diff --git a/research/agent-goal-verification/agent4novel-agent-tech-stack.md b/research/agent-goal-verification/agent4novel-agent-tech-stack.md new file mode 100644 index 0000000..ebda549 --- /dev/null +++ b/research/agent-goal-verification/agent4novel-agent-tech-stack.md @@ -0,0 +1,28 @@ +# Agent tech stack survey: approval primitives + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Agent tech stack survey: approval primitives | +| Source URL | https://github.com/12bitsd/agent4novel/blob/HEAD/docs/research/agent-tech-stack.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the Agents SDK approval search; the snippet tabulates the SDK's needs_approval to interruptions to approve/reject/resume chain. Consistent with the first-hand docs; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/ai-agents-pitstop-hitl-approval-setup.md b/research/agent-goal-verification/ai-agents-pitstop-hitl-approval-setup.md new file mode 100644 index 0000000..4f7b8df --- /dev/null +++ b/research/agent-goal-verification/ai-agents-pitstop-hitl-approval-setup.md @@ -0,0 +1,28 @@ +# Human in the Loop AI: Best Approval Setup for Agents (2026) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Human in the Loop AI: Best Approval Setup for Agents (2026) | +| Source URL | https://ai-agents-pitstop.com/human-in-the-loop.html | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the Agents SDK approval search; the snippet summarizes the SDK interruption flow (needsApproval stops the run, approve or reject then continue). A secondary summary of a source read first-hand; stayed a card. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/aitoolsvault-agent-frameworks-compared.md b/research/agent-goal-verification/aitoolsvault-agent-frameworks-compared.md new file mode 100644 index 0000000..e1fe36f --- /dev/null +++ b/research/agent-goal-verification/aitoolsvault-agent-frameworks-compared.md @@ -0,0 +1,28 @@ +# AI Agent Frameworks Compared: A 2026 Guide + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | AI Agent Frameworks Compared: A 2026 Guide | +| Source URL | https://www.aitoolsvault.site/blog/ai-agent-frameworks-compared | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the Agents SDK approval search; the snippet maps one approval-gated support agent across the SDK and LangGraph primitives. Low-value aggregator-style comparison; logged for completeness, not relied on. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/anthropic-building-effective-agents.md b/research/agent-goal-verification/anthropic-building-effective-agents.md new file mode 100644 index 0000000..89550f7 --- /dev/null +++ b/research/agent-goal-verification/anthropic-building-effective-agents.md @@ -0,0 +1,54 @@ +# Building effective agents + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Building effective agents | +| Source URL | https://www.anthropic.com/research/building-effective-agents | +| Source publisher / author | Anthropic | +| Source type | blog post | +| Source date (published) | 2024-12-19 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +Anthropic's agent patterns essay names the evaluator-optimizer loop (one LLM call generates, another evaluates in a loop) as a core workflow, requires ground truth from the environment at each step, demands stopping conditions, and warns that agentic systems trade latency and cost for performance with compounding-error risk. + +## Key findings + +- "In the evaluator-optimizer workflow, one LLM call generates a response while another provides evaluation and feedback in a loop." +- Fit criteria: "particularly effective when we have clear evaluation criteria, and when iterative refinement provides measurable value". +- "During execution, it's crucial for the agents to gain 'ground truth' from the environment at each step (such as tool call results or code execution) to assess its progress." +- "The task often terminates upon completion, but it's also common to include stopping conditions (such as a maximum number of iterations) to maintain control." +- "Agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense." +- "The autonomous nature of agents means higher costs, and the potential for compounding errors. We recommend extensive testing in sandboxed environments, along with the appropriate guardrails." +- The page carries a note that tooling described has changed since December 2024 and points to newer Managed Agents material. + +## Pros / strengths + +- Distilled from production deployments across industries; the evaluator-optimizer description matches tai's GoalVerifier loop nearly verbatim. +- The ground-truth and stopping-condition requirements give independent backing to the verifier's tool access and the VERIFY_* caps. + +## Cons / limitations / caveats + +- A patterns essay, not a controlled study; no numbers on judge accuracy or cost curves. +- Dated December 2024 with an explicit pointer to newer material; the core patterns read as stable, but the tooling advice has moved on. + +## Relevance to tai + +The single strongest external validation of the verifier-subagent architecture: separate evaluator, evidence from the environment (not claims), bounded iterations, and an explicit cost/latency tradeoff accepted per task. tai's one-shot verification is evaluator-optimizer with a JSON verdict contract and a two-round cap. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/arxiv-2110-14168-training-verifiers.md b/research/agent-goal-verification/arxiv-2110-14168-training-verifiers.md new file mode 100644 index 0000000..74f2d78 --- /dev/null +++ b/research/agent-goal-verification/arxiv-2110-14168-training-verifiers.md @@ -0,0 +1,28 @@ +# Training Verifiers to Solve Math Word Problems + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Training Verifiers to Solve Math Word Problems | +| Source URL | https://arxiv.org/abs/2110.14168 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker; the classic result that trained verifiers ranking many candidates (best-of-N) beat fine-tuning and scale better with data. Corroborates the test-time scaling dossier from the trained-verifier side; not first-hand read. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/arxiv-2303-11366-reflexion.md b/research/agent-goal-verification/arxiv-2303-11366-reflexion.md new file mode 100644 index 0000000..dfeffb2 --- /dev/null +++ b/research/agent-goal-verification/arxiv-2303-11366-reflexion.md @@ -0,0 +1,28 @@ +# Reflexion: Language Agents with Verbal Reinforcement Learning + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Reflexion: Language Agents with Verbal Reinforcement Learning | +| Source URL | https://arxiv.org/abs/2303.11366 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker; reported to reach 91% pass@1 on HumanEval by reflecting on task feedback into episodic memory. Supports critique-plus-retry loops grounded in feedback signals; not first-hand read. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/arxiv-2303-17651-self-refine.md b/research/agent-goal-verification/arxiv-2303-17651-self-refine.md new file mode 100644 index 0000000..109cc0a --- /dev/null +++ b/research/agent-goal-verification/arxiv-2303-17651-self-refine.md @@ -0,0 +1,28 @@ +# Self-Refine: Iterative Refinement with Self-Feedback + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Self-Refine: Iterative Refinement with Self-Feedback | +| Source URL | https://arxiv.org/abs/2303.17651 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker; reported gains of about 20 absolute points from iterative self-feedback. Stands in tension with the Tier 1 cannot-self-correct dossier (which attributes such gains to smuggled-in external feedback); not first-hand read, so recorded here as the contradicting side. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/arxiv-2306-05685-judging-llm-as-a-judge.md b/research/agent-goal-verification/arxiv-2306-05685-judging-llm-as-a-judge.md new file mode 100644 index 0000000..9b031c0 --- /dev/null +++ b/research/agent-goal-verification/arxiv-2306-05685-judging-llm-as-a-judge.md @@ -0,0 +1,53 @@ +# Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena | +| Source URL | https://arxiv.org/abs/2306.05685 | +| Source publisher / author | Lianmin Zheng et al. (NeurIPS 2023 Datasets and Benchmarks Track) | +| Source type | paper | +| Source date (published) | 2023-06-09 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +The founding LLM-as-judge paper: strong LLM judges reach over 80% agreement with human preferences (human-human level), but exhibit position, verbosity, and self-enhancement biases plus limited reasoning ability. The authors conclude LLM judging is a scalable, explainable approximation of expensive human preference labels. + +## Key findings + +- "Strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans." +- "We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them." +- "LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain." +- Evidence base: MT-bench multi-turn set plus Chatbot Arena crowdsourced battles, with 3K expert votes and 30K human-preference conversations released. + +## Pros / strengths + +- Peer-reviewed at a top venue with large-scale human agreement data; the bias taxonomy (position, verbosity, self-enhancement) is the reference every later judge paper builds on. +- Directly answers the "does it scale" half of the question: judging scales where human labeling cannot. + +## Cons / limitations / caveats + +- Judges chat response preference, not agentic task completion; agreement rates do not transfer directly to a verifier grading shell transcripts. +- Uses strong judges (GPT-4 class); tai's verifier runs on free rotating backends where judge quality is lower and more variable. +- Single-judge verdicts inherit that judge's biases; the paper's mitigations (position swapping, few-shot) are not implemented in tai's verifier. +- Scope note: findings above are from the abstract and paper metadata read first-hand; the full PDF was not read. + +## Relevance to tai + +Sets the ceiling and the caveats for the GoalVerifier: judge verdicts are useful and cheap but biased and approximate, so tai is right to treat a pass as evidence rather than proof, to require tool-grounded checks in the verifier prompt, and to keep the verdict advisory to bounded follow-up rounds instead of an unbounded retry loop. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/arxiv-2310-01798-cannot-self-correct.md b/research/agent-goal-verification/arxiv-2310-01798-cannot-self-correct.md new file mode 100644 index 0000000..23a767a --- /dev/null +++ b/research/agent-goal-verification/arxiv-2310-01798-cannot-self-correct.md @@ -0,0 +1,51 @@ +# Large Language Models Cannot Self-Correct Reasoning Yet + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Large Language Models Cannot Self-Correct Reasoning Yet | +| Source URL | https://arxiv.org/abs/2310.01798 | +| Source publisher / author | Jie Huang et al. (ICLR 2024) | +| Source type | paper | +| Source date (published) | 2023-10-03 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +The key counterpoint in this subject: intrinsic self-correction, where an LLM corrects its own response using only its inherent capabilities without external feedback, fails on reasoning tasks and sometimes degrades performance. Gains reported by self-correction methods come from the external or simulated feedback they include, not from introspection alone. + +## Key findings + +- "Central to our investigation is the notion of intrinsic self-correction, whereby an LLM attempts to correct its initial responses based solely on its inherent capabilities, without the crutch of external feedback." +- "In the context of reasoning, our research indicates that LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction." +- Published at ICLR 2024 after a March 2024 revision, so the claim survived peer review. + +## Pros / strengths + +- Draws the exact line that matters for tai: same-model introspection versus separate verification with external grounding. +- Explains the apparent contradiction with Self-Refine style results: those methods smuggle in feedback (tools, tests, simulated critics), and that feedback does the work. + +## Cons / limitations / caveats + +- Scoped to reasoning tasks and intrinsic correction; it does not claim a separate verifier agent with tool access fails. That distinction is the whole design question, and this paper lands on tai's side of it. +- 2023-era models; stronger reasoners may shift the boundary, though the external-feedback requirement has held up in later work. +- Scope note: findings above are from the abstract and paper metadata read first-hand; the full PDF was not read. + +## Relevance to tai + +Validates the two load-bearing choices in the GoalVerifier: the verifier is a separate subagent rather than the same turn second-guessing itself, and its prompt demands evidence from tool results and machine state rather than judgment from prose alone. A same-model "are you sure?" loop is what this paper says fails; tai does not build that loop. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/arxiv-2310-06770-swebench-paper.md b/research/agent-goal-verification/arxiv-2310-06770-swebench-paper.md new file mode 100644 index 0000000..23bd6e5 --- /dev/null +++ b/research/agent-goal-verification/arxiv-2310-06770-swebench-paper.md @@ -0,0 +1,28 @@ +# SWE-bench: Can Language Models Resolve Real-World GitHub Issues? + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | SWE-bench: Can Language Models Resolve Real-World GitHub Issues? | +| Source URL | https://arxiv.org/abs/2310.06770 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the industry gather worker alongside the Tier 1 evaluation-guide dossier; the paper behind test-based agent verification over 2,294 real GitHub issues. The guide itself was read first-hand, the paper was not. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/arxiv-2310-17631-judgelm.md b/research/agent-goal-verification/arxiv-2310-17631-judgelm.md new file mode 100644 index 0000000..e1035aa --- /dev/null +++ b/research/agent-goal-verification/arxiv-2310-17631-judgelm.md @@ -0,0 +1,28 @@ +# JudgeLM: Fine-tuned Large Language Models are Scalable Judges + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | JudgeLM: Fine-tuned Large Language Models are Scalable Judges | +| Source URL | https://arxiv.org/abs/2310.17631 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker via arXiv API search on judge bias; reported to taxonomize position, knowledge, and format biases and to claim over 90% agreement with a GPT-4 teacher judge. Corroborates the bias direction of the Tier 1 MT-Bench dossier but was not first-hand read, so it stays Tier 2. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/arxiv-2408-03314-test-time-compute-scaling.md b/research/agent-goal-verification/arxiv-2408-03314-test-time-compute-scaling.md new file mode 100644 index 0000000..06a5a1e --- /dev/null +++ b/research/agent-goal-verification/arxiv-2408-03314-test-time-compute-scaling.md @@ -0,0 +1,52 @@ +# Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters | +| Source URL | https://arxiv.org/abs/2408.03314 | +| Source publisher / author | Charlie Snell et al. | +| Source type | paper | +| Source date (published) | 2024-08-06 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +Studies the two mechanisms for scaling test-time compute, verifier-based search and adaptive response revision, and finds effectiveness varies by prompt difficulty. A compute-optimal allocation beats best-of-N by over 4x in efficiency, and on suitable problems lets a smaller model outperform a 14x larger one at matched FLOPs. + +## Key findings + +- The two mechanisms studied: "(1) searching against dense, process-based verifier reward models; and (2) updating the model's distribution over a response adaptively, given the prompt at test time." +- "Using this compute-optimal strategy, we can improve the efficiency of test-time compute scaling by more than 4x compared to a best-of-N baseline." +- "On problems where a smaller base model attains somewhat non-trivial success rates, test-time compute can be used to outperform a 14x larger model" in a FLOPs-matched evaluation. +- Framing: test-time compute is "a critical step towards building generally self-improving agents that can operate on open-ended natural language". + +## Pros / strengths + +- Directly measures the scaling question: verification plus revision spend converts to performance, with a quantified efficiency edge over naive best-of-N. +- The difficulty-dependent effectiveness finding supports spending verification budget adaptively rather than uniformly. + +## Cons / limitations / caveats + +- Preprint (unvetted at research time); results are on reasoning benchmarks with verifier reward models, not on end-to-end agent tasks with an LLM-judge verdict. +- tai's verifier is a prompted generalist judge, not a trained process-based reward model, so the 4x figure does not transfer as a number, only as a direction. +- Scope note: findings above are from the abstract and paper metadata read first-hand; the full PDF was not read. + +## Relevance to tai + +The theoretical backbone for spending extra model calls on verification: tai's verify loop (judge the turn, revise on failure, bounded rounds) is exactly the verifier-plus-revision pattern this paper validates. It also suggests a future improvement: skip or shorten verification for trivial turns instead of always paying one full verifier call. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/arxiv-2610-14029-instance-level-judge-reliability.md b/research/agent-goal-verification/arxiv-2610-14029-instance-level-judge-reliability.md new file mode 100644 index 0000000..574438f --- /dev/null +++ b/research/agent-goal-verification/arxiv-2610-14029-instance-level-judge-reliability.md @@ -0,0 +1,28 @@ +# Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge | +| Source URL | https://arxiv.org/abs/2610.14029 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker; reported to show majority-vote judge panels can be confidently wrong on individual items, so aggregate agreement does not guarantee per-verdict reliability. This is the sharpest caveat against trusting any single verifier pass, but it rests on the worker's abstract read alone. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/arxiv-2610-18108-calllm-judge-debiasing.md b/research/agent-goal-verification/arxiv-2610-18108-calllm-judge-debiasing.md new file mode 100644 index 0000000..0fd9a80 --- /dev/null +++ b/research/agent-goal-verification/arxiv-2610-18108-calllm-judge-debiasing.md @@ -0,0 +1,28 @@ +# A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge | +| Source URL | https://arxiv.org/abs/2610.18108 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker via arXiv API search; reported to propose multicalibration (CaLLM) bounding many judge biases at once, including undocumented ones. Relevant as a future debiasing direction for the verifier, but not first-hand read. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/changkun-human-in-the-loop-agents.md b/research/agent-goal-verification/changkun-human-in-the-loop-agents.md new file mode 100644 index 0000000..8280de9 --- /dev/null +++ b/research/agent-goal-verification/changkun-human-in-the-loop-agents.md @@ -0,0 +1,52 @@ +# Confirmation Fatigue and the Protocol Gap in Agentic AI Oversight + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Confirmation Fatigue and the Protocol Gap in Agentic AI Oversight | +| Source URL | https://raw.githubusercontent.com/changkun/blog/HEAD/content/ideas/2026-02-18-human-in-the-loop-agents.md | +| Source publisher / author | Changkun (personal research blog) | +| Source type | blog post | +| Source date (published) | 2026-02-18 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | Medium | +| Status | Active | + +## Summary + +A heavily sourced essay arguing per-tool-call human approval is solved in theory but unsolved in practice: confirmation fatigue is a security vulnerability and the main obstacle to oversight at scale. It documents the threat (Rippling T10, SOC alert statistics), the MCP protocol gap (consent mandated but unenforceable), and the field's convergence on risk-tiered, reversibility-based oversight. + +## Key findings + +- "Per-tool-call human approval in agentic AI is solved in theory, unsolved in practice. Confirmation fatigue is not a UX annoyance but a security vulnerability and the primary obstacle to effective human oversight at scale." +- "Risk-tiered oversight is the dominant paradigm. Classify tool calls by risk, auto-approve the safe majority, focus human attention on the dangerous few." +- "The highest-leverage pattern: classify by reversibility, not abstract risk", citing a decision-theoretic study where intermediate confirmation at irreversibility boundaries cut task time by 13.54%. +- The MCP spec "states: 'Hosts MUST obtain explicit user consent before invoking any tool'", then concedes it "cannot enforce these security principles at the protocol level". +- SOC parallel: teams average "4,484 alerts/day" with 67% ignored from false-positive fatigue; risk-proportional filtering beats blanket approval. + +## Pros / strengths + +- Synthesizes threat models, protocol analysis, and academic citations (autonomy levels, CHI trust calibration) into one coherent argument with primary references attached to each claim. +- The reversibility criterion gives tai a principled line for which approvals stay strict: irreversible actions keep hard gates, recoverable ones can be more forgiving. + +## Cons / limitations / caveats + +- Single-author essay, not peer-reviewed; second-hand statistics (SOC numbers, the 13.54% study) are quoted, not verified here. +- MCP-focused; tai is not an MCP client, so the protocol-gap half is context rather than directly applicable. + +## Relevance to tai + +Validates tai's approval tiers (safe-command allowlist, yolo mode, per-command gates) as the converged industry shape, and supports the shell-only affirm recovery: running a user-requested shell command is recoverable and user-initiated, while deletes and secret shares are irreversible or sensitive and correctly stay strict. Also warns that every added approval prompt has a fatigue cost, which argues for keeping the re-prompt cap small. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/conclusions/approval-input-hardening.md b/research/agent-goal-verification/conclusions/approval-input-hardening.md new file mode 100644 index 0000000..494915a --- /dev/null +++ b/research/agent-goal-verification/conclusions/approval-input-hardening.md @@ -0,0 +1,51 @@ +# Conclusion: Approval input hardening (re-prompt plus shell-only affirm recovery) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | Medium | + +## Question / problem statement + +The incident that started this work: a mistyped approval answer ("A", not among y/Y/n) silently counted as deny, and follow-up praise ("it was perfect") became new instructions that ended the run. The question was how to harden approval input without weakening gates: re-prompt on unknown input, and let unambiguous affirmation after a denial count as approval for the shell case. + +## Conclusion + +Keep the shipped hardening: unknown allow answers re-prompt up to three times before falling through to deny, and exact-match affirmative guidance ("it was perfect", "everything was fine" and the rest of the allowlist) approves the denied shell command only. Deletes, secrets, schedules, and releases stay strict on any denial. The affirm set stays exact-match; nothing fuzzy ever approves. + +## Supporting sources + +- [openai-agents-python-human-in-the-loop](../openai-agents-python-human-in-the-loop.md): fail-closed approval rules and denial text fed back into the run; tai's strict-by-default with narrow recovery follows the same shape. +- [changkun-human-in-the-loop-agents](../changkun-human-in-the-loop-agents.md): risk-tiered oversight is the dominant paradigm; reversibility is the highest-leverage classifier, which justifies recovering user-initiated shell runs while keeping irreversible actions strict. +- [pmq9-human-in-the-loop-workflows-skill](../pmq9-human-in-the-loop-workflows-skill.md) (Tier 2): show the action itself, not the agent's summary; tai shows the raw command. +- [lukemurraynz-hitl-changelog-tiered-chain](../lukemurraynz-hitl-changelog-tiered-chain.md) (Tier 2): classifiers may never approve irreversible actions; denials feed back as corrective context. + +## Alternatives considered and rejected + +- Fuzzy or substring affirm matching ("yes..." anywhere approves): rejected, "yes, but do X instead" must stay guidance; exact-match is the auditable line. +- Uniform affirm recovery for all approval types: rejected, approving a delete or secret share from praise text would be a fail-open surprise; the reversibility criterion keeps it shell-only. +- Accepting "A"/"all" as always-allow: rejected implicitly by the re-prompt design; unknown input asks again instead of guessing intent, which covers typos without inventing new meanings. + +## Risks / open questions + +- No source found directly studies typo re-prompts or affirm-after-deny at a CLI; this conclusion rests on the fail-closed and risk-tiered principles, not on a direct study. Confidence is Medium for that reason. +- A user who means "continue without it" but types a bare affirm word will run the command; the recovery print ("taking ... as approval") makes the interpretation visible at the moment it happens. +- Revisit if: users report surprise approvals, or the allowlist needs a phrase (additions stay exact-match, one per line, with a test). + +## Action taken + +Shipped as `Approval.parse_allow` / `Approval.is_affirmative`, the re-prompt loop in `Agent.ask_approval`, and the shell-only recovery in `Tools.run_shell`, with 6 unit tests including the negative case that deletes stay strict. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | diff --git a/research/agent-goal-verification/conclusions/prompt-only-json-strict-parse-retry.md b/research/agent-goal-verification/conclusions/prompt-only-json-strict-parse-retry.md new file mode 100644 index 0000000..cf5b833 --- /dev/null +++ b/research/agent-goal-verification/conclusions/prompt-only-json-strict-parse-retry.md @@ -0,0 +1,51 @@ +# Conclusion: Prompt-only JSON with strict parse and retry for verdicts + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +The verifier's verdict must arrive as a strict success-plus-failures object. The question was whether prompt-only JSON (schema pasted in the prompt, forgiving extraction, strict validation, retry via follow-up rounds) is solid enough across tai's heterogeneous backends, or whether verdicts require constrained decoding to be trustworthy. + +## Conclusion + +Keep prompt-only JSON with strict parse and retry: paste the schema into the verifier prompt, extract forgivingly (fences, first object), validate strictly (success must be bool, failures a list of complete item/reason/proposal dicts, contradictions fail), and treat any malformed verdict as a failed check that consumes a round. Adopt provider-enforced structured outputs only if a constrained-decoding backend becomes tai's standard path. + +## Supporting sources + +- [openai-structured-outputs](../openai-structured-outputs.md): without Structured Outputs, "you should use a validation library and potentially retries"; applications "must detect and handle" incomplete-JSON edge cases. tai implements exactly this fallback. +- [tkarim45-structured-output-eval](../tkarim45-structured-output-eval.md): measured 100% schema adherence for schema-in-prompt on a small schema, tying constrained decoding; the real failure mode is shape, not syntax. +- [devto-mattbusel-json-repetition-setting](../devto-mattbusel-json-repetition-setting.md) (Tier 2): independent practitioner confirmation of the schema-in-prompt plus forgiving-parse-ladder practice. +- [yinsongxu-llm2jev-request-to-model](../yinsongxu-llm2jev-request-to-model.md) (Tier 2): a schema in the prompt cannot establish judgment correctness, which is why the verdict stays advisory to bounded rounds rather than proof. + +## Alternatives considered and rejected + +- Constrained decoding now: rejected, tai's backends (CLI harnesses, free tiers) do not offer it uniformly; single-backend enforcement would split the verdict path in two for no measured gain on this small schema. +- JSON mode if available: rejected as insufficient on its own (valid JSON without schema adherence); the strict validator covers what JSON mode does not. +- Lenient parsing (accept partial verdicts): rejected, a half-parsed verdict is exactly how "success" gets misread; fail-closed keeps the strictness guarantee. + +## Risks / open questions + +- Weaker free-tier models likely score below the 100% schema-in-prompt figure measured on Claude Haiku; malformed verdicts cost rounds instead of answers, which is correct but slower. +- Valid JSON with wrong content (confident misjudgment) passes the parser by design; content risk is carried by the architecture conclusion (evidence rules, bounded rounds), not by parsing. +- Revisit if: malformed-verdict rounds dominate verification cost, or one constrained backend becomes standard. + +## Action taken + +Shipped as `GoalVerifier.extract_json` (forgiving) plus `GoalVerdict.from_text` (strict) with unit tests covering fences, contradictions, missing fields, bare failures, garbage input, and over-long payloads read from the full worker result. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | diff --git a/research/agent-goal-verification/conclusions/verifier-subagent-architecture.md b/research/agent-goal-verification/conclusions/verifier-subagent-architecture.md new file mode 100644 index 0000000..4268e74 --- /dev/null +++ b/research/agent-goal-verification/conclusions/verifier-subagent-architecture.md @@ -0,0 +1,57 @@ +# Conclusion: Verifier-subagent architecture for one-shot goals + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +tai's one-shot mode used to quit after a single turn even when the goal was undelivered (a denied download plus a misread "it was perfect" ended the run with exit 0). The question was whether gating the quit on a separate verifier subagent, with bounded follow-up rounds, is a sound pattern that scales, or an overbuilt control that burns model calls for little gain. + +## Conclusion + +Keep the GoalVerifier design as shipped: a separate worker subagent judges goal delivery from transcript plus tool-grounded evidence, failures feed bounded follow-up turns, and unverified outcomes exit nonzero with the turn left open. This is the industry-standard evaluator-optimizer loop with a strict verdict contract, and both literature and production practice support it under the caps already in place. + +## Supporting sources + +- [anthropic-building-effective-agents](../anthropic-building-effective-agents.md): evaluator-optimizer loop, ground truth from the environment each step, stopping conditions, accepted cost/latency tradeoff. +- [langchain-reflection-agents](../langchain-reflection-agents.md): independent second source for the generator/critic loop with fixed rounds and tool observations as grounding. +- [arxiv-2408-03314-test-time-compute-scaling](../arxiv-2408-03314-test-time-compute-scaling.md): verifier-plus-revision spend converts to performance (4x over best-of-N); smaller models can beat larger ones with test-time compute. +- [arxiv-2306-05685-judging-llm-as-a-judge](../arxiv-2306-05685-judging-llm-as-a-judge.md): judge verdicts reach human-level agreement but carry known biases, so the verdict stays evidence plus bounded rounds, never blind trust. +- [arxiv-2310-01798-cannot-self-correct](../arxiv-2310-01798-cannot-self-correct.md): intrinsic same-model self-correction fails without external feedback, which is why the verifier is a separate subagent with tool access. +- [swebench-evaluation-guide](../swebench-evaluation-guide.md): executable checks are the gold standard; the verifier prompt's tools-first ordering follows from this. +- [openai-evaluation-best-practices](../openai-evaluation-best-practices.md): judge quality varies by context, so keep the verify log events and calibrate the rubric from real outcomes. +- [arxiv-2303-17651-self-refine](../arxiv-2303-17651-self-refine.md) (Tier 2): the contradicting side on self-correction gains, resolved in favor of external feedback per the ICLR paper above. + +## Alternatives considered and rejected + +- Same-turn self-check ("are you sure?"): rejected, the cannot-self-correct paper says intrinsic correction without external feedback fails and can degrade results. +- Unbounded retry until verified: rejected, Anthropic requires stopping conditions and warns of compounding errors and cost; two extra rounds is the shipped bound. +- No verification, trust the final answer: rejected, this was the incident that started the work (silent quit on an undelivered goal). +- Test-based verification only (SWE-bench style): rejected as the sole mechanism because most one-shot goals have no pre-written checks; kept as the preferred evidence source inside the verifier prompt instead. + +## Risks / open questions + +- False-negative verdicts burn up to two extra turns and can make the agent redo finished work; watch the false-fail rate in the verify log events. +- Instance-level judge reliability is weaker than aggregate agreement suggests (see the Tier 2 instance-reliability card); a wrong pass is still possible, so the exit code means "verifier satisfied", not "proven". +- Verifier quality depends on the free rotating backends; a weak-backend verdict is weaker evidence, unmeasured so far. +- Revisit if: false-fail rounds become the common case, or a constrained-decoding backend becomes the standard path (see the JSON fallback conclusion). + +## Action taken + +Shipped in tai.py the same sessions as the research: `GoalVerifier` / `GoalVerdict` / `GoalFailure`, `Agent.verify_loop` with `VERIFY_MAX_ROUNDS = 2`, one-shot exit 0/1 wiring, open turn state on unverified outcomes, 14 unit tests, suite 959 green plus live probes of both the pass and fail paths. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | diff --git a/research/agent-goal-verification/cycles-docs-approval-queues-runtime-authority.md b/research/agent-goal-verification/cycles-docs-approval-queues-runtime-authority.md new file mode 100644 index 0000000..e7e8869 --- /dev/null +++ b/research/agent-goal-verification/cycles-docs-approval-queues-runtime-authority.md @@ -0,0 +1,28 @@ +# AI agent approval queues need runtime authority + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | AI agent approval queues need runtime authority | +| Source URL | https://github.com/runcycles/cycles-docs/blob/HEAD/blog/ai-agent-approval-queues-need-runtime-authority.md | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the Agents SDK approval search; the snippet compares LangChain HITL middleware (pause, persist, resume with approve/edit/reject/respond) against the SDK flow. Confirms cross-framework convergence on pause-resume approvals; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/deepy-system-audit-modes-design.md b/research/agent-goal-verification/deepy-system-audit-modes-design.md new file mode 100644 index 0000000..3cbddfe --- /dev/null +++ b/research/agent-goal-verification/deepy-system-audit-modes-design.md @@ -0,0 +1,28 @@ +# Deepy system audit modes design + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Deepy system audit modes design | +| Source URL | https://github.com/kirineko/deepy/blob/HEAD/openspec/changes/archive/2026-05-28-add-system-audit-modes/design.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the Agents SDK approval search; the snippet describes adopting the SDK's native approval primitives for a deep audit mode. Evidence that independent projects converge on the same approval shape; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/devto-mattbusel-json-repetition-setting.md b/research/agent-goal-verification/devto-mattbusel-json-repetition-setting.md new file mode 100644 index 0000000..5fd6df0 --- /dev/null +++ b/research/agent-goal-verification/devto-mattbusel-json-repetition-setting.md @@ -0,0 +1,28 @@ +# LLM JSON output: one setting dropped valid JSON from 75% to 4% + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | LLM JSON output: one setting dropped valid JSON from 75% to 4% | +| Source URL | http://dev.to/mattbusel/llm-json-output-one-setting-dropped-valid-json-from-75-to-4-1o12 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the structured-output reliability search; the snippet recommends schema-in-prompt plus a forgiving parse ladder (strip fences, cut first object, repair slips) before schema validation. That ladder is what tai's extractor implements; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/dhyanendra-json-mode-structured-outputs.md b/research/agent-goal-verification/dhyanendra-json-mode-structured-outputs.md new file mode 100644 index 0000000..5cf2dc9 --- /dev/null +++ b/research/agent-goal-verification/dhyanendra-json-mode-structured-outputs.md @@ -0,0 +1,28 @@ +# JSON Mode and Structured Outputs study notes + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | JSON Mode and Structured Outputs study notes | +| Source URL | https://github.com/dhyanendrasinghsikarwar/learning_notes/blob/HEAD/2.2%20Structured%20&%20reliable%20output/2.2.1_JSON_Mode_Structured_Outputs.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the structured-output reliability search; the snippet lists four reliability levels from prompt-only to grammar-constrained decoding. Same ladder as the Tier 1 sources; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/emrekuruu-user-in-the-loop-interfaces.md b/research/agent-goal-verification/emrekuruu-user-in-the-loop-interfaces.md new file mode 100644 index 0000000..27f765a --- /dev/null +++ b/research/agent-goal-verification/emrekuruu-user-in-the-loop-interfaces.md @@ -0,0 +1,28 @@ +# User-in-the-Loop Interfaces study notes + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | User-in-the-Loop Interfaces study notes | +| Source URL | https://github.com/emrekuruu/nvidia-ncp-aai-study-notes/blob/HEAD/Human-AI%20Interaction%20and%20Oversight/User-in-the-Loop%20Interfaces.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the approval-UX search; the snippet frames too little control as ambush and too much as fatigue plus a scalability bottleneck. States the calibration tai's tiers aim at; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/fny666-approval-evidence-sources.md b/research/agent-goal-verification/fny666-approval-evidence-sources.md new file mode 100644 index 0000000..93617e8 --- /dev/null +++ b/research/agent-goal-verification/fny666-approval-evidence-sources.md @@ -0,0 +1,28 @@ +# Approval evidence sources log + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Approval evidence sources log | +| Source URL | https://github.com/fny666/northstar-agent-os/blob/HEAD/research/nextgen-continuity-2026-09/a-line-evidence-window/A-LINE-APPROVAL-EVIDENCE-20260921T190354Z/sources.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the Agents SDK approval search; the snippet records the official SDK HITL page (tools declare needs_approval, interruptions surface pending items, RunState resumes). A pointer to a source read first-hand instead; logged for the trail. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/hardw00t-defense-patterns-hitl.md b/research/agent-goal-verification/hardw00t-defense-patterns-hitl.md new file mode 100644 index 0000000..079d24a --- /dev/null +++ b/research/agent-goal-verification/hardw00t-defense-patterns-hitl.md @@ -0,0 +1,28 @@ +# Defense patterns: human-in-the-loop gates + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Defense patterns: human-in-the-loop gates | +| Source URL | https://github.com/hardw00t/ai-security-arsenal/blob/HEAD/skills/llm-security/references/defense_patterns_2026.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the approval-UX search; the snippet lists HITL failure modes (approval fatigue rubber-stamping, dialogs hiding full arguments, agent text tricking users into approving). Supports showing the raw command and keeping prompts scannable; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/justshipai-human-in-the-loop-design-skill.md b/research/agent-goal-verification/justshipai-human-in-the-loop-design-skill.md new file mode 100644 index 0000000..741e4bb --- /dev/null +++ b/research/agent-goal-verification/justshipai-human-in-the-loop-design-skill.md @@ -0,0 +1,28 @@ +# Human-in-the-loop design skill + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Human-in-the-loop design skill | +| Source URL | https://github.com/justshipai/pm-skills-lab/blob/HEAD/skills/ai-product/human-in-the-loop-design/SKILL.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the approval-UX search; the snippet rejects one global approval setting and confirmation prompts without evidence, reserving approval for high-stakes actions. Matches tai's risk tiers; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/kheldin-call-me-maybe-constrained-decoding.md b/research/agent-goal-verification/kheldin-call-me-maybe-constrained-decoding.md new file mode 100644 index 0000000..f5fc8cc --- /dev/null +++ b/research/agent-goal-verification/kheldin-call-me-maybe-constrained-decoding.md @@ -0,0 +1,28 @@ +# call-me-maybe: function calling with constrained decoding on Qwen3-0.6B + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | call-me-maybe: function calling with constrained decoding on Qwen3-0.6B | +| Source URL | https://github.com/kheldin/call-me-maybe | +| Source type | repository | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the structured-output reliability search; the snippet claims constrained decoding lifts a 0.6B model from about 30% prompting-only accuracy to near-perfect schema-compliant calls. Relevant as the small-model end of the enforcement argument; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/langchain-reflection-agents.md b/research/agent-goal-verification/langchain-reflection-agents.md new file mode 100644 index 0000000..4520cf0 --- /dev/null +++ b/research/agent-goal-verification/langchain-reflection-agents.md @@ -0,0 +1,51 @@ +# Reflection Agents + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Reflection Agents | +| Source URL | https://www.langchain.com/blog/reflection-agents | +| Source publisher / author | LangChain (Ankush Gola) | +| Source type | blog post | +| Source date (published) | 2024-02-21 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +LangChain's reflection pattern composes a generator LLM with a reflector LLM that critiques as a teacher, looping a fixed number of times, optionally folding in tool observations as external information. It also flags reward-score sensitivity in reflection/search hybrids as a known reliability failure mode. + +## Key findings + +- "This simple example composes two LLM calls: a generator and a reflector. The generator tries to respond directly to the user's requests. The reflector is prompted to role play as a teacher and offer constructive criticism for the initial response." +- "The loop proceeds a fixed number of times, and the final generated output is returned." +- "Reflection is a prompting strategy used to improve the quality and success rate of agents and similar AI systems. It involves prompting an LLM to reflect on and critique its past actions, sometimes incorporating additional external information such as tool observations." +- On LATS hybrids: "While it can be sensitive to the reward scores, the general algorithm can be flexibly applied to a variety of tasks." + +## Pros / strengths + +- Second independent industry source (with Anthropic) for the generator/critic loop with a fixed iteration cap, which is tai's verify loop shape. +- Explicitly blesses tool observations as the external grounding for critique, matching tai's evidence-strict verifier prompt. + +## Cons / limitations / caveats + +- Vendor blog with linked example notebooks; states the pattern, not measured accuracy or cost data. +- The reward-sensitivity warning applies to score-driven search hybrids more than to tai's boolean-plus-failures verdict, but it still cautions against trusting any single model-graded signal too far. + +## Relevance to tai + +Independent confirmation that a separate critic call over the transcript plus tool evidence, bounded to a fixed number of rounds, is standard practice rather than an exotic control. tai's stricter verdict contract (boolean plus itemized failures with proposals) is a compatible tightening of the same loop. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/lukemurraynz-hitl-changelog-tiered-chain.md b/research/agent-goal-verification/lukemurraynz-hitl-changelog-tiered-chain.md new file mode 100644 index 0000000..b4d72ec --- /dev/null +++ b/research/agent-goal-verification/lukemurraynz-hitl-changelog-tiered-chain.md @@ -0,0 +1,28 @@ +# HITL design patterns changelog: tiered decision chain + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | HITL design patterns changelog: tiered decision chain | +| Source URL | https://github.com/lukemurraynz/hve-agent-skills/blob/HEAD/skills/hitl-design-patterns/CHANGELOG.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the approval-UX search; the snippet describes a rules-first tiered chain where a classifier may never approve irreversible actions and denials feed back as corrective context. Both rules match tai's design; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/lukemurraynz-hitl-design-patterns-skill.md b/research/agent-goal-verification/lukemurraynz-hitl-design-patterns-skill.md new file mode 100644 index 0000000..e8ecd2b --- /dev/null +++ b/research/agent-goal-verification/lukemurraynz-hitl-design-patterns-skill.md @@ -0,0 +1,28 @@ +# HITL design patterns skill + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | HITL design patterns skill | +| Source URL | https://github.com/lukemurraynz/hve-agent-skills/blob/HEAD/skills/hitl-design-patterns/SKILL.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the approval-UX search; the snippet advises against gating safe actions (fatigue training) and suggests re-gating after about 60 seconds without new human instruction. The no-gate-on-safe part matches tai's allowlist; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/openai-agents-python-human-in-the-loop.md b/research/agent-goal-verification/openai-agents-python-human-in-the-loop.md new file mode 100644 index 0000000..1be4a67 --- /dev/null +++ b/research/agent-goal-verification/openai-agents-python-human-in-the-loop.md @@ -0,0 +1,52 @@ +# Human-in-the-loop - OpenAI Agents SDK + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Human-in-the-loop - OpenAI Agents SDK | +| Source URL | https://openai.github.io/openai-agents-python/human_in_the_loop/ | +| Source publisher / author | OpenAI | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +The official Agents SDK guide describes the industry-standard approval loop: tools declare `needs_approval`, the run pauses with pending items in `interruptions`, the app approves or rejects (optionally with a custom rejection message fed back into the run), and the same run resumes from serialized state. + +## Key findings + +- "Set `needs_approval` to `True` to always require approval or provide an async function that decides per call." +- "Callable approval rules fail closed when the SDK cannot safely inspect the arguments. If the arguments are missing, empty, contain only whitespace, are malformed JSON... the callable is not invoked and the call requires manual approval." +- "By default, a rejected tool call returns the SDK's standard rejection text back into the run" with per-call `rejection_message` override taking precedence over the run-wide formatter. +- "Convert the result to a `RunState` with `result.to_state()`, call `state.approve(...)` or `state.reject(...)`, and then resume", and the resumed run continues where it left off. +- Per-call approvals are scoped to the call ID; `always_approve` / `always_reject` persist the decision for the tool identity. + +## Pros / strengths + +- First-party specification of the dominant approval pattern, including the fail-closed rule and the denial-feedback mechanism. +- The pause / serialize / resume shape matches tai's turn-state plus `--resume` design almost exactly. + +## Cons / limitations / caveats + +- SDK-specific mechanics (RunState, interruptions) that tai reimplements in its own vocabulary; no guidance on free-text approval prompts or typo handling at a CLI. +- No discussion of approval fatigue or input validation; those come from practitioner sources, not this page. + +## Relevance to tai + +Confirms three tai design choices as industry-standard: per-command approval with an always-allow escape hatch, denial text fed back into the run as corrective context (tai's `Denied` guidance), and resumable paused runs (tai's open turn state plus `--resume`). The fail-closed rule supports keeping destructive approvals strict while recovering only the shell case on unambiguous affirmation. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/openai-community-mia-agent-approvals.md b/research/agent-goal-verification/openai-community-mia-agent-approvals.md new file mode 100644 index 0000000..aa54800 --- /dev/null +++ b/research/agent-goal-verification/openai-community-mia-agent-approvals.md @@ -0,0 +1,28 @@ +# Building Mia AI: Making AI-Agent Execution Visible, Reliable, and Safe + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Building Mia AI: Making AI-Agent Execution Visible, Reliable, and Safe | +| Source URL | https://community.openai.com/t/building-mia-ai-making-ai-agent-execution-visible-reliable-and-safe/1404516 | +| Source type | forum thread | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the Agents SDK approval search; the snippet argues approvals should be a runtime pause rather than something the model says in text. Matches tai's out-of-band prompt design; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/openai-evaluation-best-practices.md b/research/agent-goal-verification/openai-evaluation-best-practices.md new file mode 100644 index 0000000..dcb5d35 --- /dev/null +++ b/research/agent-goal-verification/openai-evaluation-best-practices.md @@ -0,0 +1,51 @@ +# Evaluation best practices + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Evaluation best practices | +| Source URL | https://developers.openai.com/api/docs/guides/evaluation-best-practices | +| Source publisher / author | OpenAI | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +OpenAI's eval guide states the core tradeoff plainly: LLM-as-judge quality varies by problem context while expert human labels are expensive and slow. It prescribes eval-driven development, clear rubrics, logging everything for future eval cases, and calibrating automated scoring against human judgment. + +## Key findings + +- "No strategy is perfect. The quality of LLM-as-Judge varies depending on problem context while using expert human annotators to provide ground-truth labels is expensive and time-consuming." +- "Ensure eval rubrics are clear and detailed" (adjacent guidance on the same page). +- Eval-driven development: "Evaluate early and often. Write scoped tests at every stage", "Log everything: Log as you develop so you can mine your logs for good eval cases", "Maintain agreement: Use human feedback to calibrate automated scoring." +- Anti-patterns include "Vibe-based evals" and "Ignoring human feedback: Not calibrating your automated metrics against human evals." + +## Pros / strengths + +- Vendor-neutral advice despite the source: it openly bounds its own recommended technique (judge quality varies) instead of overselling it. +- The log-and-calibrate discipline maps directly onto tai's verify log events, which are mineable the same way. + +## Cons / limitations / caveats + +- Guidance for offline eval suites, not for an inline runtime acceptance gate; tai applies the same judge machinery live, where a wrong verdict costs follow-up rounds instead of just a noisy metric. +- No numbers on judge accuracy; pairs with the MT-Bench paper for quantities. + +## Relevance to tai + +Justifies shipping the strict verifier now while treating calibration as ongoing work: keep the `verify` log events, watch the false-pass and false-fail rates in real use, and tighten the rubric (the verifier prompt) from that log. Also supports the decision to exit nonzero with a report rather than silently passing. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/openai-structured-outputs.md b/research/agent-goal-verification/openai-structured-outputs.md new file mode 100644 index 0000000..5746832 --- /dev/null +++ b/research/agent-goal-verification/openai-structured-outputs.md @@ -0,0 +1,52 @@ +# Structured model outputs + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Structured model outputs | +| Source URL | https://developers.openai.com/api/docs/guides/structured-outputs | +| Source publisher / author | OpenAI | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +OpenAI's official guide draws a hard line between JSON mode (valid JSON guaranteed, schema adherence not guaranteed) and Structured Outputs (schema adherence enforced at generation time). For callers that cannot use Structured Outputs, the guide prescribes exactly the fallback tai uses: validate the output and retry. + +## Key findings + +- "Structured Outputs is the evolution of JSON mode. While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence." +- "JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors. You should use Structured Outputs to ensure it matches your schema, or if that is not possible, you should use a validation library and potentially retries to ensure that the output matches your desired schema." +- Benefits claimed for Structured Outputs: "Reliable type-safety: No need to validate or retry incorrectly formatted responses", explicit refusals, simpler prompting. +- The comparison table states JSON Mode outputs valid JSON: Yes; adheres to schema: No. +- When using JSON mode, the model must be explicitly instructed to produce JSON (the API errors if the string "JSON" does not appear in context), and applications "must detect and handle the edge cases that can result in the model output not being a complete JSON object". + +## Pros / strengths + +- Authoritative statement of the two-tier guarantee ladder from the vendor that defines both tiers. +- Gives an explicit sanctioned fallback (validate plus retries) for callers without schema enforcement, which is tai's exact situation on multi-backend free tiers. + +## Cons / limitations / caveats + +- Vendor documentation for OpenAI's own API; says nothing about other providers' JSON modes or their adherence rates. +- No numbers on how often prompt-only JSON fails; quantitative failure rates come from community evals, not this page. + +## Relevance to tai + +Directly load-bearing for the GoalVerifier design: tai cannot use constrained decoding across its heterogeneous backends (CLI harnesses, free tiers), so prompt-only JSON plus strict schema validation plus verification rounds is the vendor-recommended fallback, not a hack. The "detect and handle edge cases" line justifies the forgiving extract plus strict parse split in `GoalVerifier.extract_json` / `GoalVerdict.from_text`. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/pmq9-human-in-the-loop-workflows-skill.md b/research/agent-goal-verification/pmq9-human-in-the-loop-workflows-skill.md new file mode 100644 index 0000000..46c4141 --- /dev/null +++ b/research/agent-goal-verification/pmq9-human-in-the-loop-workflows-skill.md @@ -0,0 +1,28 @@ +# Human-in-the-loop workflows skill + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Human-in-the-loop workflows skill | +| Source URL | https://github.com/pmq9/agent-skills/blob/HEAD/.claude/skills/engineering/ai/human-in-the-loop-workflows/SKILL.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the approval-UX search; the snippet names the biggest HITL design failure as showing the AI's summary of the action instead of the action itself. tai shows the raw command, which is the recommended side; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/prajwalamte-hitl-governance-pattern.md b/research/agent-goal-verification/prajwalamte-hitl-governance-pattern.md new file mode 100644 index 0000000..f7ed183 --- /dev/null +++ b/research/agent-goal-verification/prajwalamte-hitl-governance-pattern.md @@ -0,0 +1,28 @@ +# Human-in-the-loop governance pattern + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Human-in-the-loop governance pattern | +| Source URL | https://github.com/prajwalamte/ai-engineering-patterns/blob/HEAD/content/patterns/governance/human-in-the-loop.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the approval-UX search; the snippet defines the gate as pausing before consequential actions by category until explicit consent. The standard definition tai implements; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/promptessor-hitl-approval-workflows.md b/research/agent-goal-verification/promptessor-hitl-approval-workflows.md new file mode 100644 index 0000000..aece6c3 --- /dev/null +++ b/research/agent-goal-verification/promptessor-hitl-approval-workflows.md @@ -0,0 +1,28 @@ +# Human-in-the-Loop AI Agents: Approval Workflows and Best Practices + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Human-in-the-Loop AI Agents: Approval Workflows and Best Practices | +| Source URL | https://promptessor.com/blog/human-in-the-loop-ai-agents | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by two approval-UX searches (logged once); the snippets describe run-level approval patterns and OpenAI's guardrails-versus-human-review split. The underlying SDK docs were read first-hand instead, so this stayed a card. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/rajsekharalluri-prompt-engineering-json-levels.md b/research/agent-goal-verification/rajsekharalluri-prompt-engineering-json-levels.md new file mode 100644 index 0000000..6c247f5 --- /dev/null +++ b/research/agent-goal-verification/rajsekharalluri-prompt-engineering-json-levels.md @@ -0,0 +1,28 @@ +# Prompt engineering notes: JSON reliability levels + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Prompt engineering notes: JSON reliability levels | +| Source URL | https://github.com/rajsekharalluri/devthread.in/blob/HEAD/content/genai-engineer/prompt-engineering.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the structured-output reliability search; the snippet states the prompt-only / JSON-mode / schema-enforced ladder. Consistent with the Tier 1 vendor docs; a study-notes page seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/sayem7456-structured-output-reliability-skill.md b/research/agent-goal-verification/sayem7456-structured-output-reliability-skill.md new file mode 100644 index 0000000..c11a63f --- /dev/null +++ b/research/agent-goal-verification/sayem7456-structured-output-reliability-skill.md @@ -0,0 +1,28 @@ +# Structured output reliability skill + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Structured output reliability skill | +| Source URL | https://github.com/sayem7456/opencode-engineering-skills/blob/HEAD/skills/structured-output-reliability/SKILL.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the structured-output reliability search; the snippet insists every structured-output pipeline must validate before use and notes no provider guarantees perfect output. Matches tai's strict-parse stance; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/sheikh-mohammad-openai-agents-sdk-production.md b/research/agent-goal-verification/sheikh-mohammad-openai-agents-sdk-production.md new file mode 100644 index 0000000..5b0d4bc --- /dev/null +++ b/research/agent-goal-verification/sheikh-mohammad-openai-agents-sdk-production.md @@ -0,0 +1,28 @@ +# OpenAI Agents SDK production notes: approvals + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | OpenAI Agents SDK production notes: approvals | +| Source URL | https://github.com/sheikh-mohammad/agent-factory-claude-skills/blob/HEAD/.claude/skills/openai-agents-sdk/references/production.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the Agents SDK approval search; the snippet restates the built-in HITL flow (run pauses, pending items in interruptions). Redundant with the first-hand SDK docs; stayed a card. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/strayspark-unreal-approval-workflow.md b/research/agent-goal-verification/strayspark-unreal-approval-workflow.md new file mode 100644 index 0000000..de2cbaa --- /dev/null +++ b/research/agent-goal-verification/strayspark-unreal-approval-workflow.md @@ -0,0 +1,28 @@ +# The AI Agent Approval Workflow for Unreal: Human-in-the-Loop That Isn't a Rubber Stamp + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | The AI Agent Approval Workflow for Unreal: Human-in-the-Loop That Isn't a Rubber Stamp | +| Source URL | https://www.strayspark.studio/blog/approve-ai-agent-changes-unreal-editor-from-phone | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the approval-UX search; the snippet warns that showing the agent's explanation means grading its essay rather than its action, and that every-change gates train reflexive approval. Reinforces raw-action display; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/sumansingh-structured-outputs-constrained-decoding.md b/research/agent-goal-verification/sumansingh-structured-outputs-constrained-decoding.md new file mode 100644 index 0000000..505c6af --- /dev/null +++ b/research/agent-goal-verification/sumansingh-structured-outputs-constrained-decoding.md @@ -0,0 +1,28 @@ +# Structured Outputs and Constrained Decoding in Agentic AI + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Structured Outputs and Constrained Decoding in Agentic AI | +| Source URL | https://github.com/sumansingh226/python-to-ai-ml-journey/blob/HEAD/src/Ai/topis/%23%20Structured%20Outputs%20&%20Constrained%20Decoding%20in%20Agentic%20AI.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the structured-output reliability search; the listing snippet claims top models fail JSON schema adherence 5-15% of the time without enforcement. Directionally consistent with stronger sources, but a personal study-notes repo, seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/swebench-evaluation-guide.md b/research/agent-goal-verification/swebench-evaluation-guide.md new file mode 100644 index 0000000..51bd0a5 --- /dev/null +++ b/research/agent-goal-verification/swebench-evaluation-guide.md @@ -0,0 +1,50 @@ +# SWE-bench Evaluation Guide + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | SWE-bench Evaluation Guide | +| Source URL | https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/docs/guides/evaluation.md | +| Source publisher / author | SWE-bench (Princeton NLP) | +| Source type | repository | +| Source date (published) | Unknown — not stated (living project docs) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +The reference implementation of test-based agent verification: candidate patches are applied to real repositories and the repositories' own tests decide whether the issue is resolved, all inside Docker for reproducibility. No LLM judges the outcome; execution is the judge. + +## Key findings + +- "SWE-bench evaluates models by applying their generated patches to real-world repositories and running the repository's tests to verify if the issue is resolved. The evaluation is performed in a containerized Docker environment to ensure consistent results across different platforms." +- The harness entry point is `swebench.harness.run_evaluation` with worker parallelism, instance selection, Docker cache levels, and result caching by run id. +- Predictions are JSONL rows of instance id, model name, and patch text; cloud execution via Modal or sb-cli is supported. + +## Pros / strengths + +- Ground truth by construction: the verdict is a test run, immune to judge bias, verbosity effects, and sycophancy. +- Reproducible and automatable at scale, which is why it became the standard coding-agent benchmark. + +## Cons / limitations / caveats + +- Only works where executable checks exist (tests, builds, scripts). Most tai one-shot goals ("upgrade python", "research X") have no pre-written test suite, so pure SWE-bench-style verification is unavailable. +- Heavyweight per check (containers, repos, dependency installs); overkill for judging a chat-length turn. + +## Relevance to tai + +Sets the gold standard tai's verifier approximates: prefer checking machine state with tools over grading prose. tai's verifier prompt already instructs read-only tool inspection first and transcript judgment second; this source says that ordering is right and that any goal with an executable check should get one rather than relying on the judge alone. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/tkarim45-structured-output-eval.md b/research/agent-goal-verification/tkarim45-structured-output-eval.md new file mode 100644 index 0000000..b63064b --- /dev/null +++ b/research/agent-goal-verification/tkarim45-structured-output-eval.md @@ -0,0 +1,51 @@ +# Structured-Output Reliability Eval + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Structured-Output Reliability Eval | +| Source URL | https://github.com/tkarim45/structured-output-eval | +| Source publisher / author | tkarim45 | +| Source type | repository | +| Source date (published) | Unknown — not stated | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | Medium | +| Status | Active | + +## Summary + +A runnable benchmark comparing four JSON strategies (plain prompt, schema-in-prompt, native constrained decoding, strict tool) on valid-JSON rate, schema adherence, and field accuracy, including adversarial inputs. On Claude Haiku 4.5, plain prompting hit only 25% schema adherence (33% adversarial) while schema-in-prompt hit 100%, tying constrained decoding on this task set. + +## Key findings + +- Measured table (Claude Haiku 4.5 on Bedrock, 8 tasks): plain schema adherence 0.250 (adversarial 0.333); schema_in_prompt, native, and strict_tool all 1.000 on every column. +- "Malformed JSON was never the failure mode. Claude Haiku returned valid JSON 100% of the time on every strategy, plain included... The real failure is schema adherence." +- "Just putting the schema in the prompt fixed it, no constrained decoding required... constrained decoding's real value is the guarantee at scale and on harder schemas, not a measured lift over schema-in-prompt on this set." +- "Valid JSON is not equal to correct" (Valid ≠ correct): braces parse, shape wrong. + +## Pros / strengths + +- Real measured numbers on a real model with a reproducible harness, not vendor marketing; distinguishes the two failure modes (syntax vs schema) that most advice smears together. +- Directly supports tai's position: for a small fixed schema like the verdict object, schema-in-prompt plus validation is empirically sufficient. + +## Cons / limitations / caveats + +- One model family (Claude Haiku), 8 tasks, one operator; tai's verifier runs on weaker free-tier models where plain-prompt adherence is likely worse than 25%. +- Community repo, no peer review; the mock path ships deliberately different numbers as a teaching fixture, which a careless reader could quote instead of the real table. + +## Relevance to tai + +Quantitative backing for the verdict contract: paste the schema into the verifier prompt (tai does), validate strictly on receipt (tai does), and treat constrained decoding as a future guarantee rather than a current requirement. Also warns that tai's weaker verifier backends need the strict-parse-plus-retry path more than Claude-class models do. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/agent-goal-verification/towardsai-structured-output-perfect-json-wrong.md b/research/agent-goal-verification/towardsai-structured-output-perfect-json-wrong.md new file mode 100644 index 0000000..1a2d8f8 --- /dev/null +++ b/research/agent-goal-verification/towardsai-structured-output-perfect-json-wrong.md @@ -0,0 +1,28 @@ +# Structured Output From an LLM Can Be Perfect JSON and Completely Wrong + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Structured Output From an LLM Can Be Perfect JSON and Completely Wrong | +| Source URL | https://pub.towardsai.net/structured-output-from-an-llm-can-be-perfect-json-and-completely-wrong-cbd630be2306 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the structured-output reliability search (also seen mirrored on Medium under the same title by Syed Ali; both listings describe one article, logged once). The snippet draws the JSON-mode versus native-structured-output line and the valid-not-equal-correct point. Same claims verified first-hand in stronger sources, so this stayed a card. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/agent-goal-verification/yinsongxu-llm2jev-request-to-model.md b/research/agent-goal-verification/yinsongxu-llm2jev-request-to-model.md new file mode 100644 index 0000000..14a957c --- /dev/null +++ b/research/agent-goal-verification/yinsongxu-llm2jev-request-to-model.md @@ -0,0 +1,28 @@ +# llm2jev request-to-model: schema limits + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | agent-goal-verification | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | llm2jev request-to-model: schema limits | +| Source URL | https://github.com/yinsongxu/llm2jev/blob/HEAD/docs/request-to-model.md | +| Source type | repository | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the structured-output reliability search; the snippet notes a schema in the system prompt is still just a text instruction and cannot establish judgment correctness. Supports the valid-not-equal-correct caveat; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/background-work-limits/apscheduler-user-guide.md b/research/background-work-limits/apscheduler-user-guide.md new file mode 100644 index 0000000..e347692 --- /dev/null +++ b/research/background-work-limits/apscheduler-user-guide.md @@ -0,0 +1,51 @@ +# APScheduler User Guide + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | APScheduler User Guide | +| Source URL | https://apscheduler.readthedocs.io/en/3.x/userguide.html | +| Source publisher / author | APScheduler | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +The APScheduler guide supplies tai's scheduler policy vocabulary: max_instances defaults to 1 (a new fire while the previous run lives counts as a misfire), coalescing rolls queued missed executions into one trigger, and misfire_grace_time decides whether a late execution still runs (which can fire several runs in succession). + +## Key findings + +- "By default, only one instance of each job is allowed to be run at the same time. This means that if the job is about to be run but the previous run hasn't finished yet, then the latest run is considered a misfire." +- "It is possible to set the maximum number of instances for a particular job that the scheduler will let run concurrently, by using the `max_instances` keyword argument when adding the job." +- "If coalescing is enabled for the job and the scheduler sees one or more queued executions for the job, it will only trigger it once. No misfire events will be sent for the 'bypassed' runs." +- Late execution against `misfire_grace_time` "can lead into the job being executed several times in succession." + +## Pros / strengths + +- Three orthogonal knobs (overlap, coalescing, grace) that decompose tai's scheduler decisions cleanly: tai takes max_instances 1 plus always-coalesce and skips grace windows as unneeded complexity. +- Documents the failure mode each knob prevents, so tai's simpler choices stay explicit rather than accidental. + +## Cons / limitations / caveats + +- Single-process scheduler semantics; tai's compare-and-swap claim across processes goes beyond what this page covers. +- No per-tick firing cap concept; tai's burst bound comes from the supervisor side (global parallel cap), not from here. + +## Relevance to tai + +Direct template for the scheduler half: one live run per repeating schedule (lease-checked, Temporal-Skip style), missed intervals coalesced into one fire (already tai's behavior, now specified), and no auto-retry of failed runs (loud `last_status` instead). + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/background-work-limits/celery-periodic-tasks.md b/research/background-work-limits/celery-periodic-tasks.md new file mode 100644 index 0000000..4d76ea5 --- /dev/null +++ b/research/background-work-limits/celery-periodic-tasks.md @@ -0,0 +1,28 @@ +# Periodic Tasks + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Periodic Tasks | +| Source URL | https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scheduler gather worker; reported the single-scheduler rule against duplicate tasks and user-side locking for overlap. tai's compare-and-swap claim already covers the duplication half cross-process; the locking half is superseded by the lease design. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/background-work-limits/conclusions/scheduler-overlap-and-misfire.md b/research/background-work-limits/conclusions/scheduler-overlap-and-misfire.md new file mode 100644 index 0000000..9d137b0 --- /dev/null +++ b/research/background-work-limits/conclusions/scheduler-overlap-and-misfire.md @@ -0,0 +1,52 @@ +# Conclusion: One live run per schedule, capped fires, stated misfire policy + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +The scheduler fired every due row with no per-tick bound and no overlap protection: 1000 due rows meant 1000 spawns, and a repeating schedule fired again while its previous run was still going. The question was which production policies to adopt (overlap, burst caps, misfire handling) without growing new infrastructure. + +## Conclusion + +Three rules, no schema migration: (1) at most `SCHEDULER_MAX_FIRES = 4` spawns per tick, half the global pool, with the backlog draining on later ticks; (2) max one live run per repeating schedule (APScheduler default, Temporal-Skip), enforced by a lease: a claim sets `last_status` to running, and a new fire is skipped while the claim is younger than the run timeout plus margin, so crashed schedulers self-heal instead of sticking rows; (3) the misfire policy is stated, not accidental: repeating schedules coalesce missed intervals into one fire (APScheduler-coalesce, cron-like), one-shot schedules fire late however late (Temporal-catchup-like), and failed runs never auto-retry (their loud `last_status` is the record). + +## Supporting sources + +- [apscheduler-user-guide](../apscheduler-user-guide.md): max_instances default 1 with overlap counting as misfire, plus coalescing of queued executions into one trigger. +- [temporal-schedule](../temporal-schedule.md): Skip as the default overlap policy (a new start while one runs simply does not start), with concurrency only under the policy that names it. +- [celery-periodic-tasks](../celery-periodic-tasks.md) (Tier 2): the single-scheduler rule tai's compare-and-swap claim already implements cross-process. +- [cron-manpage](../cron-manpage.md) (Tier 2): current-minute-only evaluation as the simplest coalescing precedent for repeats. + +## Alternatives considered and rejected + +- A `running` agent-id column: rejected, agent ids are per-process so the column would lie across processes; the timestamp lease in existing columns is cross-process honest. +- Grace windows for late one-shots: rejected, a personal agent should fire the late reminder rather than drop it; the 60s minimum interval plus 30s tick keeps staleness visible. +- Buffers and cancellation (Temporal Buffer/Cancel): rejected as unneeded complexity; skip plus loud status covers the observed cases. +- Auto-retry of failed scheduled runs: rejected, same as workers; the schedules list shows the failure and the next fire proceeds normally. + +## Risks / open questions + +- The lease approximates liveness by time, not by process state; a run finishing just past its timeout overlaps the next fire by design (the watcher waits timeout plus margin, matching the lease). +- Skipped-overlap fires leave no per-row trace beyond the still-running status; the tick prints a count, and the row visibly stays running. +- Revisit if: per-tick backlog grows faster than it drains (raise the burst cap with evidence), or overlap skips hide real stalls (add an age alarm on running rows). + +## Action taken + +Implemented the same day: per-tick fire cap with skip counts, `last_status = running` at claim with lease-checked skip, and unit tests for the cap, overlap skip and stale-lease recovery, and one-shot late firing. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | diff --git a/research/background-work-limits/conclusions/supervisor-parallel-cap.md b/research/background-work-limits/conclusions/supervisor-parallel-cap.md new file mode 100644 index 0000000..6a1798b --- /dev/null +++ b/research/background-work-limits/conclusions/supervisor-parallel-cap.md @@ -0,0 +1,52 @@ +# Conclusion: Cap live subagents with reject-loud semantics + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +`spawn_agent` had no breadth bound: depth was capped at 2 but a turn could fork any number of workers, each holding a thread and an LLM conversation. The base rule requires a hard cap on everything unbounded. The question was where to set it and what a full pool should do: block (backpressure) or refuse (reject-loud). + +## Conclusion + +Cap live subagents at `AGENTS_MAX_PARALLEL = 8` with semaphore-TryAcquire semantics: `spawn_agent` raises past the cap instead of queueing, and each caller translates the refusal into its own loud message (fork tool tells the agent to poll existing workers first, the scheduler leaves the row for the next tick with a printed count, verification reports a failed check). The number is arithmetic, not magic: a scheduler burst of 4 plus interactive forks plus one verifier, single machine, free-tier friendly. Past `AGENTS_MAX_RECORDS = 64` total records, spawning purges oldest collected finished records first, so delivered results stay inspectable but never accumulate without bound. + +## Supporting sources + +- [go-semaphore](../go-semaphore.md): worker-pool-via-semaphore plus the explicit block-versus-TryAcquire fork; tai takes the refuse side. +- [erlang-otp-supervisor](../erlang-otp-supervisor.md): bounding must be structural (a counter with a tripwire), which the live-count check is. +- [python-concurrent-futures-deadlock](../python-concurrent-futures-deadlock.md) (Tier 2): blocking waits between workers are a documented deadlock shape, which rules out queueing the spawn. +- [vorpus-structured-concurrency](../vorpus-structured-concurrency.md) (Tier 2): unstructured background work drops errors silently, which is why every refusal and drop is loud by construction. + +## Alternatives considered and rejected + +- Blocking acquire (wait for a slot): rejected, a stalled fork burns the spawning turn's step budget and risks the documented wait deadlock; polling plus retry is the honest loop. +- Higher cap (16+) or per-profile caps: rejected for now, 8 already exceeds any legitimate burst observed (scheduler 4 plus a few forks); raise on measured need, not speculation. +- Purging uncollected finished records: rejected, that would lose undelivered results; only collected finished records are purged, which loses nothing. +- Auto-retry of failed workers (Temporal-style): rejected, failures surface as loud results instead; retry belongs to the caller that understands the task. + +## Risks / open questions + +- A pathological turn could still fork-poll-fork in a loop within the cap; step budgets, not the pool cap, bound that, and they already exist. +- Uncollected finished records still accumulate in very long sessions; watch record counts if sessions run for days. +- Revisit if: legitimate bursts hit the cap (raise with evidence), or free-tier rate limits argue for lowering it. + +## Action taken + +Implemented the same day: `AgentsFull` exception, cap check in `spawn_agent`, loud handling in the fork tool, scheduler pre-check, purge of collected finished records, and unit tests pinning refusal plus recovery (collect, then spawn succeeds). + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | diff --git a/research/background-work-limits/cron-manpage.md b/research/background-work-limits/cron-manpage.md new file mode 100644 index 0000000..8d50ac8 --- /dev/null +++ b/research/background-work-limits/cron-manpage.md @@ -0,0 +1,28 @@ +# cron(8) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | cron(8) | +| Source URL | https://manpages.ubuntu.com/manpages/noble/man8/cron.8.html | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scheduler gather worker; reported current-minute-only evaluation (missed minutes never catch up). The simplest misfire policy on record; tai's repeating schedules behave this way, one-shots deliberately do not. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/background-work-limits/erlang-otp-supervisor.md b/research/background-work-limits/erlang-otp-supervisor.md new file mode 100644 index 0000000..16f3b20 --- /dev/null +++ b/research/background-work-limits/erlang-otp-supervisor.md @@ -0,0 +1,51 @@ +# Supervisor Behaviour + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Supervisor Behaviour | +| Source URL | https://www.erlang.org/doc/design_principles/sup_princ.html | +| Source publisher / author | Erlang/OTP System Documentation v29.1.2 | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +The OTP supervisor reference: children restart by strategy, but restarts are bounded by a maximum intensity (MaxR restarts in MaxT seconds, default 1 in 5), past which the supervisor terminates everything and escalates. Bounding must be structural (a counter with a tripwire), not advisory. + +## Key findings + +- "The supervisors have a built-in mechanism to limit the number of restarts which can occur in a given time interval", specified by `intensity` (MaxR) and `period` (MaxT). +- "If more than `MaxR` number of restarts occur in the last `MaxT` seconds, the supervisor terminates all the child processes and then itself." +- "The intention of the restart mechanism is to prevent a situation where a process repeatedly dies for the same reason, only to be restarted again." +- Defaults are `1` and `5` when not given; escalation goes to the next higher-level supervisor. + +## Pros / strengths + +- Four decades of production supervision distilled into one mechanism; the intensity-plus-period shape is directly portable to any retry or respawn loop. +- The escalation rule (trip, terminate, let the parent decide) matches tai's loud-failure style better than silent infinite retry. + +## Cons / limitations / caveats + +- Erlang process semantics (cheap processes, real kills) do not transfer to Python threads, which cannot be killed; tai can refuse and abandon but never terminate a stuck worker. +- No notion of a parallel-worker cap; OTP bounds restarts, not breadth, so the fan-out half comes from the semaphore source instead. + +## Relevance to tai + +Justifies the two bounds tai adopts: a parallel-live cap (breadth) plus no-blind-retry (tai workers and schedules never auto-restart; failures surface as loud results). If tai ever adds worker retry, this source prescribes intensity-plus-period as its shape. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/background-work-limits/go-semaphore.md b/research/background-work-limits/go-semaphore.md new file mode 100644 index 0000000..43d62c1 --- /dev/null +++ b/research/background-work-limits/go-semaphore.md @@ -0,0 +1,50 @@ +# semaphore package + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | semaphore package | +| Source URL | https://pkg.go.dev/golang.org/x/sync/semaphore | +| Source publisher / author | Go project (golang.org/x/sync) | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +The Go semaphore reference gives tai both halves of the fan-out decision: cap parallel workers with a counting semaphore, and choose explicitly between blocking acquire (backpressure) and TryAcquire (reject-loud). tai takes the reject side: a full pool refuses new spawns with a loud error instead of stalling the caller. + +## Key findings + +- "Example_workerPool demonstrates how to use a semaphore to limit the number of goroutines working on parallel tasks." +- "This use of a semaphore mimics a typical 'worker pool' pattern, but without the need to explicitly shut down idle workers when the work is done." +- "TryAcquire acquires the semaphore with a non-negative weight of n without blocking. On success, returns true. On failure, returns false and leaves the semaphore unchanged." + +## Pros / strengths + +- Names the exact fork in the road (block versus refuse) with reference semantics for both, so tai's choice is deliberate rather than accidental. +- Worker-pool-via-semaphore matches tai's shape (threads come and go; only the live count is bounded). + +## Cons / limitations / caveats + +- Go-side primitive with contexts and goroutines; tai reimplements the policy (count live under a lock, refuse past the cap), not the mechanism. +- No guidance on sizing the cap; tai's number comes from its own arithmetic (scheduler burst plus interactive forks plus verifier headroom). + +## Relevance to tai + +Direct backing for `AGENTS_MAX_PARALLEL` with reject-loud semantics: `spawn_agent` raises past the cap instead of queueing, because a queued fork would stall the spawning turn's step budget (and blocking waits between workers are a documented deadlock shape). Callers translate the refusal into their own loud message. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/background-work-limits/python-concurrent-futures-deadlock.md b/research/background-work-limits/python-concurrent-futures-deadlock.md new file mode 100644 index 0000000..b1798d1 --- /dev/null +++ b/research/background-work-limits/python-concurrent-futures-deadlock.md @@ -0,0 +1,28 @@ +# concurrent.futures + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | concurrent.futures | +| Source URL | https://docs.python.org/3/library/concurrent.futures.html | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the supervision gather worker; reported the deadlock warning for tasks waiting on each other's futures. Supports the reject-instead-of-block choice (a blocked spawn could stall a turn holding budget); not opened first-hand. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/background-work-limits/temporal-schedule.md b/research/background-work-limits/temporal-schedule.md new file mode 100644 index 0000000..4aec9b1 --- /dev/null +++ b/research/background-work-limits/temporal-schedule.md @@ -0,0 +1,51 @@ +# Schedule + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Schedule | +| Source URL | https://docs.temporal.io/schedule | +| Source publisher / author | Temporal | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +Temporal's schedule reference enumerates six overlap policies with Skip as the default (a new start while one runs simply does not start) and AllowAll as the only policy permitting concurrency. It frames overlap as a per-schedule policy choice with a safe default, which is exactly what tai adopts. + +## Key findings + +- "The Overlap Policy controls what happens when it is time to start a Workflow Execution but a previously started Workflow Execution is still running." +- "`Skip`: Default. Nothing happens; the Workflow Execution is not started." +- "`AllowAll` Starts any number of concurrent Workflow Executions. With this policy (and only this policy), more than one Workflow Execution, started by the Schedule, can run simultaneously." +- Buffer variants (BufferOne, BufferAll) plus CancelOther and TerminateOther complete the six; a paused execution still counts as running for overlap evaluation. + +## Pros / strengths + +- Shows the full policy space, which lets tai pick Skip deliberately while knowing what it is not building (buffers, cancellation). +- The safe-default framing (concurrency only under the one policy that names it) matches tai's fail-closed style. + +## Cons / limitations / caveats + +- Distributed-system machinery (catchup windows, buffers) far beyond a single-file agent; tai takes the overlap default and the misfire vocabulary, nothing else. +- No per-tick burst cap concept; same as APScheduler, the global bound lives elsewhere. + +## Relevance to tai + +Names tai's overlap behavior: Temporal-Skip by default (previous run live means this fire does not start), with the lease check as the liveness test. Buffers and cancellation stay explicitly unbuilt. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/background-work-limits/temporal-workflow-activity.md b/research/background-work-limits/temporal-workflow-activity.md new file mode 100644 index 0000000..7b8fc38 --- /dev/null +++ b/research/background-work-limits/temporal-workflow-activity.md @@ -0,0 +1,28 @@ +# Workflow Activity + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Workflow Activity | +| Source URL | https://docs.temporal.io/workflow-activity | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the supervision gather worker; reported mandatory activity timeouts with retry-plus-backoff by default. tai takes the timeout half (every spawn already carries one) and rejects the auto-retry half as unneeded complexity; stayed a card. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/background-work-limits/trio-core-nursery.md b/research/background-work-limits/trio-core-nursery.md new file mode 100644 index 0000000..1d4bfdc --- /dev/null +++ b/research/background-work-limits/trio-core-nursery.md @@ -0,0 +1,28 @@ +# Trio's core functionality + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Trio's core functionality | +| Source URL | https://trio.readthedocs.io/en/stable/reference-core.html | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the supervision gather worker; reported nursery cancel-all on child error with grouped re-raise. The cancel-and-group half of structured concurrency, not applicable to tai's kill-less threads; stayed a card. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/background-work-limits/vorpus-structured-concurrency.md b/research/background-work-limits/vorpus-structured-concurrency.md new file mode 100644 index 0000000..056e500 --- /dev/null +++ b/research/background-work-limits/vorpus-structured-concurrency.md @@ -0,0 +1,28 @@ +# Notes on structured concurrency, or: Go statement considered harmful + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | background-work-limits | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Notes on structured concurrency, or: Go statement considered harmful | +| Source URL | https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the supervision gather worker; reported the nursery rule (a block cannot exit until its children do) and the dropped-background-error failure mode of unstructured tasks. Shapes tai's join-before-quit discipline; not opened first-hand. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/agentmarketcap-memory-architecture-wars.md b/research/prompt-context-scaling/agentmarketcap-memory-architecture-wars.md new file mode 100644 index 0000000..999f776 --- /dev/null +++ b/research/prompt-context-scaling/agentmarketcap-memory-architecture-wars.md @@ -0,0 +1,28 @@ +# Agent Memory Architecture Wars 2026 + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Agent Memory Architecture Wars 2026 | +| Source URL | https://agentmarketcap.ai/blog/2026/04/08/agent-memory-architecture-wars-letta-mem0-zep-native-provider | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the memory-layer search; the snippet characterizes Mem0 as compression-first (extraction pipeline into dense memories, optional graph variant). Supports the build-vs-adopt framing; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/agentpatterns-framework-landscape-memory.md b/research/prompt-context-scaling/agentpatterns-framework-landscape-memory.md new file mode 100644 index 0000000..7380446 --- /dev/null +++ b/research/prompt-context-scaling/agentpatterns-framework-landscape-memory.md @@ -0,0 +1,28 @@ +# Agentic framework landscape: memory layer entry + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Agentic framework landscape: memory layer entry | +| Source URL | https://github.com/agentpatterns-ai/website/blob/HEAD/frameworks/agentic-framework-landscape.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the memory-layer search; the snippet restates Mem0 as a layer composing over any harness with multi-signal retrieval numbers. Redundant with the first-hand README dossier; stayed a card. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/aithinkerlab-graphrag-ultimate-guide.md b/research/prompt-context-scaling/aithinkerlab-graphrag-ultimate-guide.md new file mode 100644 index 0000000..a695091 --- /dev/null +++ b/research/prompt-context-scaling/aithinkerlab-graphrag-ultimate-guide.md @@ -0,0 +1,28 @@ +# GraphRAG vs RAG: The Ultimate Guide to Building Reasoning AI Search in 2026 + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | GraphRAG vs RAG: The Ultimate Guide to Building Reasoning AI Search in 2026 | +| Source URL | https://aithinkerlab.com/graphrag-vs-rag-the-ultimate-guide-to-building-reasoning-ai-search-in-2026/ | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet repeats the higher build cost and latency framing with a pro-graph verdict. Low-value aggregator-style guide; logged for completeness, not relied on. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/anthropic-effective-context-engineering.md b/research/prompt-context-scaling/anthropic-effective-context-engineering.md new file mode 100644 index 0000000..e2ae1ec --- /dev/null +++ b/research/prompt-context-scaling/anthropic-effective-context-engineering.md @@ -0,0 +1,53 @@ +# Effective context engineering for AI agents + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Effective context engineering for AI agents | +| Source URL | https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents | +| Source publisher / author | Anthropic | +| Source type | blog post | +| Source date (published) | 2025-09-29 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +Anthropic's context-engineering essay is the reference statement of the converged arsenal: context is finite with diminishing returns (context rot), so agents should compact near-full conversations into summaries, retrieve just-in-time via lightweight identifiers instead of stuffing, persist notes outside the window, and delegate deep work to subagents. Bigger windows do not remove the need for any of this. + +## Key findings + +- "As the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases" (context rot, observed across all models). +- "Compaction is the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary", preserving decisions, bugs, and implementation details while discarding redundant tool outputs. +- "Rather than pre-processing all relevant data up front, agents built with the 'just in time' approach maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools." +- Long-horizon techniques listed: "compaction, structured note-taking, and multi-agent architectures". +- "Waiting for larger context windows might seem like an obvious tactic. But it's likely that for the foreseeable future, context windows of all sizes will be subject to context pollution and information relevance concerns." +- Hybrid model endorsed: drop stable context up front, navigate the rest just-in-time with search primitives (their example: glob and grep over files). + +## Pros / strengths + +- Names every technique tai already implements (compaction, spill identifiers plus record_read, notes as records, fork subagents) as the recommended set, from the vendor operating the largest coding agent. +- The just-in-time plus lightweight-identifier framing is exactly tai's mem: spill design, independently derived. + +## Cons / limitations / caveats + +- Vendor engineering essay, no controlled numbers; the compaction-recall tradeoff is stated qualitatively ("distills... in a high-fidelity manner") without a loss metric. +- Assumes frontier-model summarization quality; tai compacts with whatever backend is active, including weak free tiers, where summary fidelity is lower. + +## Relevance to tai + +The master checklist against which tai's context handling was audited: compaction (have it), just-in-time retrieval via identifiers (have it: spill ids plus search), notes outside the window (have it: records), subagent delegation (have it: fork/poll). The gap it exposes is oversized single inputs: Anthropic's techniques all assume the unit of work fits or can be fetched piecemeal, and tai has no path for one giant user prompt except holding it whole. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/archyde-graphrag-when-to-use.md b/research/prompt-context-scaling/archyde-graphrag-when-to-use.md new file mode 100644 index 0000000..8374bee --- /dev/null +++ b/research/prompt-context-scaling/archyde-graphrag-when-to-use.md @@ -0,0 +1,28 @@ +# GraphRAG vs. Vector RAG: When to Use Knowledge Graphs for Better AI Answers + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | GraphRAG vs. Vector RAG: When to Use Knowledge Graphs for Better AI Answers | +| Source URL | https://www.archyde.com/graphrag-vs-vector-rag-when-to-use-knowledge-graphs-for-better-ai-answers/ | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet claims up to 20 percentage points multi-hop retrieval gain with substantial overhead. Unverified magnitude; directionally consistent with the paper's sensemaking win. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/arxiv-2005-11401-rag-original.md b/research/prompt-context-scaling/arxiv-2005-11401-rag-original.md new file mode 100644 index 0000000..774a2c4 --- /dev/null +++ b/research/prompt-context-scaling/arxiv-2005-11401-rag-original.md @@ -0,0 +1,28 @@ +# Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks | +| Source URL | https://arxiv.org/abs/2005.11401 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker as the founding RAG paper (parametric memory plus dense-vector index, SOTA on open-domain QA). Foundational but predates and adds nothing beyond tai's settled BM25-chunked retrieval conclusion; not first-hand read. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/arxiv-2109-10862-recursive-summarization.md b/research/prompt-context-scaling/arxiv-2109-10862-recursive-summarization.md new file mode 100644 index 0000000..02e4f57 --- /dev/null +++ b/research/prompt-context-scaling/arxiv-2109-10862-recursive-summarization.md @@ -0,0 +1,51 @@ +# Recursively Summarizing Books with Human Feedback + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Recursively Summarizing Books with Human Feedback | +| Source URL | https://arxiv.org/abs/2109.10862 | +| Source publisher / author | Jeff Wu et al. (OpenAI) | +| Source type | paper | +| Source date (published) | 2021-09-22 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | Medium | +| Status | Active | + +## Summary + +The canonical map-reduce pattern for over-window inputs: summarize small sections, then recursively summarize the summaries, reaching state of the art on book-length summarization (BookSum) and book QA (NarrativeQA via zero-shot QA over the summaries). The inference-time decomposition needs no special training to reuse as a pattern. + +## Key findings + +- "At inference time, the model first summarizes small sections of the book and then recursively summarizes these summaries to produce a summary of the entire book." +- "We achieve state-of-the-art results on the recent BookSum dataset for book-length summarization." +- "A zero-shot question-answering model using these summaries achieves state-of-the-art results on the challenging NarrativeQA benchmark." + +## Pros / strengths + +- Proves decomposition scales past the window for both summarization and QA-shaped goals, which covers tai's massive-prompt cases (paste a huge log, ask questions over it). +- Single-source in this subject for the pattern's measured wins, hence Medium confidence; corroborated in shape (not numbers) by GraphRAG's map-reduce-over-summaries answering step. + +## Cons / limitations / caveats + +- 2021 GPT-3-era work with heavy human-feedback training behind the headline numbers; tai would reuse only the inference pattern with off-the-shelf models. +- Recursive summarization compounds summary error at each level; retrieval-first with summarization as fallback loses less. +- Scope note: findings above are from the abstract and paper metadata read first-hand; the full PDF was not read. + +## Relevance to tai + +Backs the second half of the giant-prompt proposal: after spilling an oversized prompt to chunks, answer questions over it either by retrieving slices (preferred, lossless) or by map-reducing chunk summaries through subagents (for global questions). tai already owns every primitive this needs: chunker, FTS index, fork/poll workers. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/arxiv-2307-03172-lost-in-the-middle.md b/research/prompt-context-scaling/arxiv-2307-03172-lost-in-the-middle.md new file mode 100644 index 0000000..01cdb72 --- /dev/null +++ b/research/prompt-context-scaling/arxiv-2307-03172-lost-in-the-middle.md @@ -0,0 +1,50 @@ +# Lost in the Middle: How Language Models Use Long Contexts + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Lost in the Middle: How Language Models Use Long Contexts | +| Source URL | https://arxiv.org/abs/2307.03172 | +| Source publisher / author | Nelson F. Liu et al. (TACL 2023) | +| Source type | paper | +| Source date (published) | 2023-07-06 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +The paper behind the "bigger window is not enough" argument: on multi-document QA and key-value retrieval, model accuracy is highest when relevant information sits at the start or end of context and degrades significantly in the middle, even for explicitly long-context models. + +## Key findings + +- "Performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models." +- "Current language models do not robustly make use of information in long input contexts." +- Tested on multi-document question answering and key-value retrieval; published in TACL 2023. + +## Pros / strengths + +- Peer-reviewed, replicated finding that directly bounds the "just stuff the prompt" strategy: position effects punish exactly the giant-middle-blob input shape. +- Explains why retrieval (small, relevant, front-loaded context) beats stuffing even when stuffing fits. + +## Cons / limitations / caveats + +- 2023-era models; newer long-context models degrade more gracefully, but the follow-up RULER dossier shows the gap persists in kind. +- Scope note: findings above are from the abstract and paper metadata read first-hand; the full PDF was not read. + +## Relevance to tai + +Justifies keeping the 32k working cap with aggressive curation instead of chasing bigger windows: stuffing a massive prompt whole is measurably worse than retrieving its relevant slices. Directly supports the spill-and-retrieve proposal for oversized prompts. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/arxiv-2310-05736-llmlingua.md b/research/prompt-context-scaling/arxiv-2310-05736-llmlingua.md new file mode 100644 index 0000000..0e91c3a --- /dev/null +++ b/research/prompt-context-scaling/arxiv-2310-05736-llmlingua.md @@ -0,0 +1,28 @@ +# LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models | +| Source URL | https://arxiv.org/abs/2310.05736 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker; reported up to 20x prompt compression with little performance loss via a trained budget controller plus token-level compression. The compression alternative tai rejects (needs a trained compressor, lossy, dependency-hostile); single-source and not first-hand read. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/arxiv-2404-06654-ruler.md b/research/prompt-context-scaling/arxiv-2404-06654-ruler.md new file mode 100644 index 0000000..52a4da4 --- /dev/null +++ b/research/prompt-context-scaling/arxiv-2404-06654-ruler.md @@ -0,0 +1,50 @@ +# RULER: What's the Real Context Size of Your Long-Context Language Models? + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | RULER: What's the Real Context Size of Your Long-Context Language Models? | +| Source URL | https://arxiv.org/abs/2404.06654 | +| Source publisher / author | Cheng-Ping Hsieh et al. (COLM 2024) | +| Source type | paper | +| Source date (published) | 2024-04-09 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +RULER shows vanilla needle-in-a-haystack drastically overstates usable context: across 17 long-context models and harder retrieval, tracing, and aggregation tasks, nearly all models drop sharply with length, and only half the models claiming 32k+ tokens hold satisfactory performance at 32k. + +## Key findings + +- "Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases." +- "While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K." +- The benchmark adds multi-hop tracing and aggregation beyond needle search, plus varied needle counts and types. + +## Pros / strengths + +- Turns "claimed context" into "usable context" with a reproducible open benchmark; the 32k figure lands exactly on tai's working cap. +- COLM 2024 peer review plus 17-model breadth makes the direction robust even as individual models improve. + +## Cons / limitations / caveats + +- Synthetic tasks; real agent workloads mix easier and harder patterns, so the numbers bound rather than predict tai's case. +- Scope note: findings above are from the abstract and paper metadata read first-hand; the full PDF was not read. + +## Relevance to tai + +Validates the conservative 32k cap: on the free rotating backends tai actually uses, usable context is certainly below claimed context, so compacting early and retrieving slices is safer than trusting the window. Kills the "wait for bigger windows" alternative for tai's backend tier. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/arxiv-2404-16130-graphrag-paper.md b/research/prompt-context-scaling/arxiv-2404-16130-graphrag-paper.md new file mode 100644 index 0000000..be4d1b4 --- /dev/null +++ b/research/prompt-context-scaling/arxiv-2404-16130-graphrag-paper.md @@ -0,0 +1,50 @@ +# From Local to Global: A Graph RAG Approach to Query-Focused Summarization + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | From Local to Global: A Graph RAG Approach to Query-Focused Summarization | +| Source URL | https://arxiv.org/abs/2404.16130 | +| Source publisher / author | Darren Edge et al. (Microsoft Research) | +| Source type | paper | +| Source date (published) | 2024-04-24 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +The GraphRAG paper: conventional RAG fails on global sensemaking questions over a corpus, so build an LLM-extracted entity graph, pregenerate community summaries, and answer by map-reducing over them. On 1M-token datasets it substantially beats plain RAG on comprehensiveness and diversity for that question class. + +## Key findings + +- "RAG fails on global questions directed at an entire text corpus, such as 'What are the main themes in the dataset?', since this is inherently a query-focused summarization task, rather than an explicit retrieval task." +- Method: "uses an LLM to build a graph index in two stages: first, to derive an entity knowledge graph from the source documents, then to pregenerate community summaries for all groups of closely related entities", then per-community partial responses fused into a final answer. +- "For a class of global sensemaking questions over datasets in the 1 million token range, we show that GraphRAG leads to substantial improvements over a conventional RAG baseline for both the comprehensiveness and diversity of generated answers." + +## Pros / strengths + +- Precisely scopes its own win: global sensemaking over large corpora, not lookup. That scoping is what makes it inapplicable to tai rather than merely expensive. +- The map-reduce-over-summaries answering pattern is reusable independently of the graph machinery. + +## Cons / limitations / caveats + +- Every indexed document pays LLM extraction plus community summarization; costs scale with corpus size before any query is ever asked. +- Scope note: findings above are from the abstract and paper metadata read first-hand; the full PDF was not read. + +## Relevance to tai + +Confirms the graph verdict in both directions: tai should not build entity extraction plus community summaries (wrong query shape, high indexing cost, maintenance-mode upstream), but should keep and grow its cheap structural graph (tags, record links, one-hop expand) for the lookup-plus-context workload it actually serves. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/arxiv-2504-19413-mem0-paper.md b/research/prompt-context-scaling/arxiv-2504-19413-mem0-paper.md new file mode 100644 index 0000000..63dd17b --- /dev/null +++ b/research/prompt-context-scaling/arxiv-2504-19413-mem0-paper.md @@ -0,0 +1,28 @@ +# Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory | +| Source URL | https://arxiv.org/abs/2504.19413 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker; reported 91% lower p95 latency and over 90% token savings versus full-context and RAG baselines on LOCOMO. The Mem0 README was read first-hand instead, so the paper stayed a card. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/conclusions/context-arsenal-verdict.md b/research/prompt-context-scaling/conclusions/context-arsenal-verdict.md new file mode 100644 index 0000000..188833c --- /dev/null +++ b/research/prompt-context-scaling/conclusions/context-arsenal-verdict.md @@ -0,0 +1,52 @@ +# Conclusion: tai already implements the converged context arsenal + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +An internal inventory found tai already compacts conversations, spills oversized tool outputs to retrievable records, chunks and BM25-indexes content, links records in a tag graph, and delegates context-heavy work to subagents. The question was whether this set matches what the industry converged on, or whether tai is missing a standard piece. + +## Conclusion + +tai's context handling already matches the converged industry arsenal piece for piece: threshold compaction (Anthropic, OpenAI, LangChain, Cursor), just-in-time retrieval via lightweight identifiers instead of stuffing (Anthropic, LangChain offload), notes persisted outside the window (Anthropic notes, tai records), subagent context isolation (Cursor, LangChain), and call-count cost guardrails (LangChain limits, tai step caps). No standard mechanism is missing; the remaining work is extending the existing machinery to new input shapes, not adopting new machinery. + +## Supporting sources + +- [anthropic-effective-context-engineering](../anthropic-effective-context-engineering.md): the master checklist (compaction, JIT identifiers, notes, multi-agent) tai was audited against. +- [openai-compaction-guide](../openai-compaction-guide.md): threshold compaction balancing quality, cost, and latency. +- [langchain-prebuilt-middleware](../langchain-prebuilt-middleware.md): the parts list (summarization, context editing, filesystem offload, subagents, limits) tai mirrors. +- [cursor-subagents](../cursor-subagents.md): delegation for context isolation with cheaper models for bulk work. +- [cursor-prompting-agents](../cursor-prompting-agents.md) (Tier 2): fourth-vendor corroboration of auto-compaction near the window limit. + +## Alternatives considered and rejected + +- Adopting a memory framework (Mem0/Letta) for conversation context: rejected, decided in the companion conclusion; the existing compaction plus retrieval covers the same needs without new dependencies. +- Chasing bigger context windows instead of curating: rejected per the RULER and lost-in-the-middle dossiers (claimed context exceeds usable context; middle-position recall degrades). +- Adding explicit context editing (dropping old tool outputs without full compaction): considered and deferred, not rejected; it is the one LangChain piece tai lacks, and a candidate follow-up if compaction costs annoy. + +## Risks / open questions + +- tai compacts with whatever backend is active, including weak free tiers; summary fidelity there is unmeasured and likely below the frontier-model results vendors report. +- The audit is mechanism-level; retrieval quality (are the right slices found?) has no Hit@K-style measurement in tai, a gap already recorded in the RAG subject. +- Revisit if: a vendor ships a genuinely new mechanism outside this set, or compaction summaries prove lossy enough to hurt in practice. + +## Action taken + +Nothing built: this conclusion records that the shipped machinery (compact_messages, spill_or_truncate, chunk_lines, FTS5 index, graph edges, fork/poll) is the right set. Follow-up decisions live in the two companion conclusions. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | diff --git a/research/prompt-context-scaling/conclusions/no-full-graphrag-no-memory-vendor.md b/research/prompt-context-scaling/conclusions/no-full-graphrag-no-memory-vendor.md new file mode 100644 index 0000000..44d29c6 --- /dev/null +++ b/research/prompt-context-scaling/conclusions/no-full-graphrag-no-memory-vendor.md @@ -0,0 +1,54 @@ +# Conclusion: No full GraphRAG and no memory vendor; grow the vault graph instead + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +tai owns a BM25 chunk index plus a small structural graph (tags, record links, one-hop search expansion). The question was whether to go further: adopt full GraphRAG entity extraction plus community summaries, or adopt a memory layer (Mem0/Letta), to get more mileage out of the vault for retrieval and context. + +## Conclusion + +Do neither. Do not build LLM entity extraction plus community summaries (wrong query shape for tai's lookup workload, 5-10x indexing cost, reference implementation in maintenance mode), and do not adopt a memory vendor (embeddings plus extraction dependencies violate stdlib-only; neither product obsoletes ordinary retrieval anyway). Instead grow what the vault already has: more edge types from existing structure (same-tag, same-session, parent-child already flow), entity-ish linking from literal identifiers (paths, hashes, error codes need no LLM to extract), and BM25-plus-graph fusion at retrieval time. + +## Supporting sources + +- [arxiv-2404-16130-graphrag-paper](../arxiv-2404-16130-graphrag-paper.md): scopes its own win to global sensemaking over large corpora, not lookup. +- [microsoft-graphrag-readme](../microsoft-graphrag-readme.md): maintenance mode plus expensive-indexing warning, from the vendor itself. +- [landedjobs-advanced-rag-graphrag](../landedjobs-advanced-rag-graphrag.md) (Tier 2): 5-10x indexing cost; vector plus rerank beats GraphRAG on lookups; route by query shape. +- [pg-raggraph-ordering](../pg-raggraph-ordering.md) (Tier 2): RAG first, graph traversal only to rerank or expand weak retrieval, which is tai's search-plus-expand shape. +- [mem0ai-mem0-readme](../mem0ai-mem0-readme.md): the transferable ideas (ADD-only accumulation, entity linking, BM25-plus-entity fusion) without the dependency. +- [medium-mem0-vs-letta](../medium-mem0-vs-letta.md) (Tier 2): neither memory product makes ordinary retrieval obsolete. +- [jplevi-graphrag-vs-rag](../jplevi-graphrag-vs-rag.md) (Tier 2): commonly-built GraphRAG underperforms vanilla RAG on real-world lookup workloads. + +## Alternatives considered and rejected + +- Full GraphRAG pipeline: rejected for the three independent reasons above (query shape, cost, maintenance mode). +- Mem0 or Letta adoption: rejected on dependencies (embeddings, extraction calls, runtime) and on redundancy (vault plus compaction already cover the needs). +- Graph removal (chunks only): rejected, one-hop expansion over literal structure is nearly free and the enhancer pattern is well supported; the graph stays as enhancer, never as the primary path. + +## Risks / open questions + +- One disagreeing datapoint: the Tier 2 hackathon benchmark claims GraphRAG beats basic RAG on every metric including tokens; most plausibly a global-question mix effect, but a genuine head-to-head on tai-shaped content would settle it. +- Literal-identifier linking is proposed, not measured; if tag-plus-identifier expansion adds noise rather than recall, narrow it back to tags. +- Revisit if: tai's workload shifts toward global sensemaking questions over the vault, or a dependency-free graph win is demonstrated on tai-shaped content. + +## Action taken + +Nothing built: this conclusion is a standing no-go with a positive program (edge types, identifier linking, fusion scoring) specified for future sessions. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | diff --git a/research/prompt-context-scaling/conclusions/spill-and-retrieve-giant-prompts.md b/research/prompt-context-scaling/conclusions/spill-and-retrieve-giant-prompts.md new file mode 100644 index 0000000..d9c268b --- /dev/null +++ b/research/prompt-context-scaling/conclusions/spill-and-retrieve-giant-prompts.md @@ -0,0 +1,53 @@ +# Conclusion: Spill oversized prompts to records and answer via retrieval + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +Every tai context mechanism handles tool outputs, history, and stored content, but a single giant user prompt has no path except sitting whole in the window: compaction keeps recent messages intact, so a 100k-character paste stays a 100k-character middle blob with measurably degraded recall. The question was how to scale prompt size without a ceiling. + +## Conclusion + +Extend the existing spill machinery to oversized user prompts: past a size cap, save the prompt body as a vault record with kind `transcript`, chunk and index it like any other content, replace the in-context message with a short pointer plus the user's actual question, and let the turn proceed via search over the chunks (lossless, preferred) or map-reduce summarization through subagents for global questions. No new infrastructure is needed; all primitives exist. + +## Supporting sources + +- [arxiv-2307-03172-lost-in-the-middle](../arxiv-2307-03172-lost-in-the-middle.md): middle-position recall degrades sharply, which is exactly what a giant whole prompt creates. +- [arxiv-2404-06654-ruler](../arxiv-2404-06654-ruler.md): usable context trails claimed context, so stuffing is unsafe precisely on tai's weaker backends. +- [arxiv-2109-10862-recursive-summarization](../arxiv-2109-10862-recursive-summarization.md): chunk-then-summarize scales past the window for both summaries and QA over long inputs (single-source for the measured wins). +- [anthropic-effective-context-engineering](../anthropic-effective-context-engineering.md): just-in-time retrieval via lightweight identifiers instead of stuffing up front. +- [langchain-prebuilt-middleware](../langchain-prebuilt-middleware.md): filesystem offload plus subagent isolation as the production shape for bulky content. + +## Alternatives considered and rejected + +- Stuffing the whole prompt and trusting the window: rejected, the lost-in-the-middle and RULER evidence says recall degrades exactly in that shape. +- Prompt compression (LLMLingua-style, 20x claimed): rejected, needs a trained compressor model (dependency-hostile), is lossy by construction, and single-sourced here (see the Tier 2 card). +- Truncating oversized prompts: rejected, silent information loss is worse than any retrieval miss; spill-then-retrieve loses nothing. +- Routing giant prompts to a long-context model: rejected for now, RULER shows claimed size overstates usable size and tai cannot assume a strong backend is configured. + +## Risks / open questions + +- Retrieval over prompt chunks can miss the relevant slice (lexical gap, accepted in the RAG subject); the pointer message must tell the model the full text is searchable so it actually searches instead of answering from the stub. +- Map-reduce over chunks compounds summary error per level; retrieval-first ordering in the prompt text mitigates this. +- Revisit if: prompt-spill retrieval misses become a reported problem, or a strong long-context backend becomes the reliable default. + +## Action taken + +Nothing built yet: this conclusion specifies the design (spill cap, `transcript` record kind, pointer stub, retrieval-first answering, subagent map-reduce fallback) for a future implementation session. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | diff --git a/research/prompt-context-scaling/cursor-prompting-agents.md b/research/prompt-context-scaling/cursor-prompting-agents.md new file mode 100644 index 0000000..3f3019f --- /dev/null +++ b/research/prompt-context-scaling/cursor-prompting-agents.md @@ -0,0 +1,28 @@ +# Prompting agents + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Prompting agents | +| Source URL | https://cursor.com/docs/agent/prompting | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the industry gather worker alongside the first-hand subagents page; reported to say Cursor auto-compresses older conversation into a summary as the fixed window fills. Corroborates threshold compaction from a fourth vendor; not opened first-hand. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/cursor-subagents.md b/research/prompt-context-scaling/cursor-subagents.md new file mode 100644 index 0000000..0c31725 --- /dev/null +++ b/research/prompt-context-scaling/cursor-subagents.md @@ -0,0 +1,52 @@ +# Subagents + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Subagents | +| Source URL | https://cursor.com/docs/subagents | +| Source publisher / author | Cursor | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +Cursor's subagent docs describe context isolation as the primary motive for delegation: each subagent gets a clean window, noisy intermediate output never touches the main conversation, and purpose-built Explore/Browser subagents run on faster, cheaper models. The built-in subagents were designed from analysis of conversations where context limits were actually hit. + +## Key findings + +- "Each subagent operates in its own context window" and "Long research or exploration tasks don't consume space in your main conversation." +- "Intermediate output stays in the subagent. The parent only sees the final summary." +- "The explore subagent uses a faster model by default. This enables running 10 parallel searches in the time a single main-agent search would take." +- "Faster models cost less. Isolating token-heavy work in subagents with appropriate model choices reduces overall cost." +- "These subagents were designed based on analysis of agent conversations where context window limits were hit." + +## Pros / strengths + +- Production evidence from the most-used coding agent that delegation-for-isolation plus cheaper-models-for-bulk-work is the economical shape. +- The foreground/background mode split matches tai's fork/poll split (collect now versus later). + +## Cons / limitations / caveats + +- Product docs for a managed agent; no failure rates or quality comparisons between main-model and fast-model subagent output. +- Cursor-specific built-ins (Explore, Browser); the pattern transfers, the implementations do not. + +## Relevance to tai + +Validates tai's fork guidance ("fork background subagents for independent, long, or context-heavy work") as the same architecture Cursor ships. Suggests one refinement worth pricing later: letting context-heavy workers run on a cheaper pinned model instead of the session default. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/dejokz-graphrag-hackathon.md b/research/prompt-context-scaling/dejokz-graphrag-hackathon.md new file mode 100644 index 0000000..1853480 --- /dev/null +++ b/research/prompt-context-scaling/dejokz-graphrag-hackathon.md @@ -0,0 +1,28 @@ +# GraphRAG hackathon benchmark: accuracy and token use + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | GraphRAG hackathon benchmark: accuracy and token use | +| Source URL | https://github.com/dejokz/graphrag_hackathon | +| Source type | repository | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet claims GraphRAG beats basic RAG on every metric including 34% fewer tokens. Contradicts the lookup-side consensus, likely a query-mix effect (global questions favor graphs, per the paper itself); recorded as the disagreeing datapoint. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/devto-agent-memory-benchmark-2026.md b/research/prompt-context-scaling/devto-agent-memory-benchmark-2026.md new file mode 100644 index 0000000..7077304 --- /dev/null +++ b/research/prompt-context-scaling/devto-agent-memory-benchmark-2026.md @@ -0,0 +1,28 @@ +# I benchmarked AI agent memory in 2026 + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | I benchmarked AI agent memory in 2026 | +| Source URL | https://dev.to/everest_an/-i-benchmarked-ai-agent-memory-in-2026-and-the-numbers-tell-a-different-story-than-the-marketing-2ae4 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the memory-layer search; the snippet argues benchmarks miss production realities (Mem0 needs manual add/search calls, Letta demands its runtime). Useful skepticism about memory marketing; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/devto-graphrag-paradox.md b/research/prompt-context-scaling/devto-graphrag-paradox.md new file mode 100644 index 0000000..49e95f1 --- /dev/null +++ b/research/prompt-context-scaling/devto-graphrag-paradox.md @@ -0,0 +1,28 @@ +# The GraphRAG Paradox: Scaling Knowledge Retrieval Without Token Collapse + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | The GraphRAG Paradox: Scaling Knowledge Retrieval Without Token Collapse | +| Source URL | http://dev.to/asifmiqbal/the-graphrag-paradox-scaling-knowledge-retrieval-without-token-collapse-28m5 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet tabulates vector RAG at 180ms and $18 per 10k queries against naive chained-LLM GraphRAG at 16.4s and $222, with a deterministic-graph hybrid in between. Unverified numbers, but the cost ordering matches every stronger source. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/devto-graphrag-pays-for-itself.md b/research/prompt-context-scaling/devto-graphrag-pays-for-itself.md new file mode 100644 index 0000000..4e396ff --- /dev/null +++ b/research/prompt-context-scaling/devto-graphrag-pays-for-itself.md @@ -0,0 +1,28 @@ +# GraphRAG vs vector RAG: when the knowledge graph pays for itself + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | GraphRAG vs vector RAG: when the knowledge graph pays for itself | +| Source URL | https://dev.to/saurabh_naik_b213f3bbeafe/graphrag-vs-vector-rag-when-the-knowledge-graph-pays-for-itself-3386 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet describes the map-reduce-over-summaries mechanism and Microsoft's own cheaper LazyGraphRAG follow-up. Cost-side corroboration; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/jplevi-graphrag-vs-rag.md b/research/prompt-context-scaling/jplevi-graphrag-vs-rag.md new file mode 100644 index 0000000..89d77ba --- /dev/null +++ b/research/prompt-context-scaling/jplevi-graphrag-vs-rag.md @@ -0,0 +1,28 @@ +# GraphRAG vs RAG: When Vector Search Isn't Enough + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | GraphRAG vs RAG: When Vector Search Isn't Enough | +| Source URL | https://jplevi.com/blog/graphrag-vs-rag-when-vector-search-isnt-enough | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet concedes commonly-built GraphRAG underperforms vanilla RAG in real-world scenarios with added latency, proposing adaptive per-question routing. The routing conclusion tai adopts; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/landedjobs-advanced-rag-graphrag.md b/research/prompt-context-scaling/landedjobs-advanced-rag-graphrag.md new file mode 100644 index 0000000..9a6fe06 --- /dev/null +++ b/research/prompt-context-scaling/landedjobs-advanced-rag-graphrag.md @@ -0,0 +1,28 @@ +# Advanced RAG interview notes: GraphRAG cost and routing + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Advanced RAG interview notes: GraphRAG cost and routing | +| Source URL | https://github.com/landedjobs/rag-engineer-interview-questions/blob/HEAD/content/06-advanced-rag.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet puts graph-construction indexing at 5-10x the cost of embedding chunks, says vector plus rerank beats GraphRAG on lookups, and calls unqualified "GraphRAG is strictly better" a red-flag answer. Directly supports routing by query shape. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/langchain-deep-agents-rag.md b/research/prompt-context-scaling/langchain-deep-agents-rag.md new file mode 100644 index 0000000..116805a --- /dev/null +++ b/research/prompt-context-scaling/langchain-deep-agents-rag.md @@ -0,0 +1,28 @@ +# Retrieval Augmented Generation with Deep Agents + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Retrieval Augmented Generation with Deep Agents | +| Source URL | https://docs.langchain.com/oss/python/langchain/rag | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the industry gather worker; reported as retrieve-then-offload-and-delegate (chunks go to a filesystem backend, not the orchestrator context). Same direction as the first-hand middleware dossier; the page itself was not opened. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/langchain-prebuilt-middleware.md b/research/prompt-context-scaling/langchain-prebuilt-middleware.md new file mode 100644 index 0000000..5fd6088 --- /dev/null +++ b/research/prompt-context-scaling/langchain-prebuilt-middleware.md @@ -0,0 +1,53 @@ +# Prebuilt middleware + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Prebuilt middleware | +| Source URL | https://docs.langchain.com/oss/python/langchain/middleware/built-in | +| Source publisher / author | LangChain | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +LangChain's production middleware catalog decomposes context management into composable pieces tai mirrors one by one: auto-summarization near token limits, context editing that clears old tool outputs, a filesystem for offloading bulky results, glob/grep file search, subagent spawning for isolation, and call-count limits as cost guardrails. + +## Key findings + +- Summarization: "Automatically summarize conversation history when approaching token limits, preserving recent messages while compressing older context." +- Context editing: "Manage conversation context by clearing older tool call outputs when token limits are reached, while preserving recent results." +- Filesystem: "Provide agents with a filesystem for storing context and long-term memories." +- Subagent: "Handing off tasks to subagents isolates context, keeping the main (supervisor) agent's context window clean while still going deep on a task." +- File search: "Provide Glob and Grep search tools over filesystem files." +- Cost guardrails ship alongside: model-call and tool-call limits "to prevent excessive costs". + +## Pros / strengths + +- The closest thing to a parts list for the converged architecture; every item maps to a tai mechanism (compaction, spill, vault records, fork/poll, MAX_STEPS). +- Shows context editing (dropping old tool outputs) as distinct from summarization, which tai currently only does implicitly via compaction. + +## Cons / limitations / caveats + +- Framework catalog, not a study; no numbers on how much each piece saves or what breaks when old tool outputs are dropped. +- Assumes the LangChain runtime; tai would reimplement any missing piece natively. + +## Relevance to tai + +Confirms tai's decomposition is complete at the mechanism level, and surfaces the one missing piece worth considering: explicit context editing that drops or collapses old tool outputs without a full compaction pass. Everything else on the list tai already has under its own names. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/medium-mem0-vs-letta.md b/research/prompt-context-scaling/medium-mem0-vs-letta.md new file mode 100644 index 0000000..a0ec9b8 --- /dev/null +++ b/research/prompt-context-scaling/medium-mem0-vs-letta.md @@ -0,0 +1,28 @@ +# Mem0 vs Letta in Production: Two Ways to Build Agent Memory + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Mem0 vs Letta in Production: Two Ways to Build Agent Memory | +| Source URL | https://medium.com/data-science-collective/mem0-vs-letta-choosing-the-right-ai-agent-memory-layer-4458712d22a6 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the memory-layer search; the snippet frames Mem0 as a callable layer versus Letta as a stateful runtime, noting neither makes ordinary retrieval obsolete. The key line for tai's build-vs-adopt call; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/medium-memory-frameworks-showdown.md b/research/prompt-context-scaling/medium-memory-frameworks-showdown.md new file mode 100644 index 0000000..3f4beb2 --- /dev/null +++ b/research/prompt-context-scaling/medium-memory-frameworks-showdown.md @@ -0,0 +1,28 @@ +# Agent memory frameworks showdown: Cognee vs Mem0 vs Zep vs Letta + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Agent memory frameworks showdown: Cognee vs Mem0 vs Zep vs Letta | +| Source URL | https://medium.com/data-science-collective/agent-memory-frameworks-showdown-cognee-vs-mem0-vs-zep-vs-letta-0f439b56f282 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the memory-layer search; the snippet positions Mem0 as fastest to ship (bolt-on) and Letta as more control with more infrastructure, both production-proven at scale. Adoption-context only; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/mem0-series-a-traction.md b/research/prompt-context-scaling/mem0-series-a-traction.md new file mode 100644 index 0000000..c026517 --- /dev/null +++ b/research/prompt-context-scaling/mem0-series-a-traction.md @@ -0,0 +1,28 @@ +# Mem0 Raises $24M Series A to Build Memory Layer for AI Agents + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Mem0 Raises $24M Series A to Build Memory Layer for AI Agents | +| Source URL | https://www.vcaonline.com/news/2025102807/mem0-raises-24m-series-a-to-build-memory-layer-for-ai-agents/ | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the memory-layer search (same press release also seen on wcia.com; both listings describe one announcement, logged once with 41k stars, 14M downloads, 35M to 186M quarterly API calls). Traction signal only, not technical evidence. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/mem0ai-mem0-readme.md b/research/prompt-context-scaling/mem0ai-mem0-readme.md new file mode 100644 index 0000000..b4760ac --- /dev/null +++ b/research/prompt-context-scaling/mem0ai-mem0-readme.md @@ -0,0 +1,53 @@ +# Mem0 README + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Mem0 README | +| Source URL | https://raw.githubusercontent.com/mem0ai/mem0/main/README.md | +| Source publisher / author | Mem0 (mem0ai) | +| Source type | repository | +| Source date (published) | Unknown — not stated (living repository file) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | Medium | +| Status | Active | + +## Summary + +Mem0 is a drop-in memory layer (add/search/update/delete over user, session, and agent scopes) whose April 2026 algorithm uses single-pass extraction, entity linking, and multi-signal retrieval fusing semantic, BM25 keyword, and entity matching. It reports 92.5 on LoCoMo and 94.4 on LongMemEval at about 7k tokens per query, with the caveat that managed-platform numbers include proprietary optimizations. + +## Key findings + +- "Multi-Level Memory: Seamlessly retains User, Session, and Agent state with adaptive personalization." +- Retrieval fuses "semantic, BM25 keyword, and entity matching scored in parallel and fused". +- Reported table: LoCoMo 92.5 at 7.0k tokens, LongMemEval 94.4 at 6.8k tokens, BEAM 1M at 64.1, BEAM 10M at 48.6. +- "Scores reflect Mem0's managed platform, which includes proprietary optimizations not available in the open-source SDK; open-source users should expect directionally similar gains but not identical numbers." +- Extraction is "Single-pass ADD-only", one LLM call with no update or delete passes; "Memories accumulate; nothing is overwritten." + +## Pros / strengths + +- Shows the production memory-layer shape concretely: extraction pipeline plus fused retrieval behind a small CRUD API, composable over any agent loop. +- The BM25-plus-entity fusion corroborates tai's lexical-plus-graph direction from an independent production system. + +## Cons / limitations / caveats + +- Vendor benchmarks on vendor-chosen suites; the managed-versus-OSS gap is disclosed but unquantified. +- Requires embeddings infrastructure plus an extraction LLM call per write; incompatible with tai's stdlib-only, no-embeddings constraints as a dependency. +- BEAM scores visibly decay with corpus size (64.1 at 1M to 48.6 at 10M), so "unlimited" still degrades. + +## Relevance to tai + +Mem0 is the thing tai will not adopt but should learn from: the ideas that transfer dependency-free are ADD-only accumulation, entity linking across memories, and fusing BM25 with entity/tag signals at retrieval. tai's vault (records plus tags plus graph edges) is already halfway to that shape without any new dependency. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/microsoft-graphrag-readme.md b/research/prompt-context-scaling/microsoft-graphrag-readme.md new file mode 100644 index 0000000..7f4c331 --- /dev/null +++ b/research/prompt-context-scaling/microsoft-graphrag-readme.md @@ -0,0 +1,50 @@ +# GraphRAG README + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | GraphRAG README | +| Source URL | https://raw.githubusercontent.com/microsoft/graphrag/main/README.md | +| Source publisher / author | Microsoft | +| Source type | repository | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +Microsoft's GraphRAG repository, the reference implementation of graph-based retrieval, now carries a maintenance-mode warning (no new PRs or features) and a cost warning on indexing. It describes itself as a research demonstration for extracting structured data from unstructured text with LLMs, not a supported offering. + +## Key findings + +- "This project is largely in maintenance mode, and won't be accepting new PRs or implementing new features", because "the capabilities of frontier models have changed dramatically" since the July 2024 release. +- "Warning: GraphRAG indexing can be an expensive operation, please read all of the documentation to understand the process and costs involved, and start small." +- "The provided code serves as a demonstration and is not an officially supported Microsoft offering." +- The pipeline "is designed to extract meaningful, structured data from unstructured text using the power of LLMs." + +## Pros / strengths + +- First-party statement of the project's status and cost profile; settles the adopt-or-not question from the vendor's own mouth. +- The maintenance rationale (frontier models changed the tradeoff) is itself evidence about where the field moved. + +## Cons / limitations / caveats + +- Says nothing about the technique's merits, only the implementation's status; the paper dossier carries the method claims. +- A research code drop was never the right adoption target anyway; serious adopters reimplement the pipeline. + +## Relevance to tai + +Decisive against adopting full GraphRAG extraction: expensive LLM indexing, a maintenance-mode reference, and tai's content is lookup-shaped (commands, logs, errors) where the technique underperforms flat retrieval (see the interview and practice cards). tai's existing tag/edge graph plus one-hop expand keeps the cheap half of the idea without the extraction pipeline. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/openai-compaction-guide.md b/research/prompt-context-scaling/openai-compaction-guide.md new file mode 100644 index 0000000..c0e4660 --- /dev/null +++ b/research/prompt-context-scaling/openai-compaction-guide.md @@ -0,0 +1,52 @@ +# Compaction + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Compaction | +| Source URL | https://developers.openai.com/api/docs/guides/compaction | +| Source publisher / author | OpenAI | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +OpenAI's server-side compaction guide confirms threshold-triggered summarization as a first-class platform feature: past a configured token threshold the server emits a compaction item carrying prior state forward, and clients drop everything before it. The stated purpose is balancing quality, cost, and latency as conversations grow. + +## Key findings + +- "Compaction helps you balance quality, cost, and latency as conversations grow." +- "When the rendered token count crosses the configured threshold, the server runs server-side compaction", emitting a compaction item in-stream and pruning context before continuing inference. +- "The returned compaction item carries forward key prior state and reasoning into the next run using fewer tokens." +- "You can drop items that came before the most recent compaction item to keep requests smaller and reduce long-tail latency." +- The example threshold in the guide is 200,000 tokens. + +## Pros / strengths + +- Independent vendor confirmation that threshold-triggered compaction is the standard mechanism, with the same trigger/prune/carry shape as tai's client-side version. +- Makes the cost/latency motive explicit: compaction is as much about bills and tail latency as about fitting the window. + +## Cons / limitations / caveats + +- Server-side and opaque ("not intended to be human-interpretable"); tai's summary is client-side and inspectable, which is better for audit but costs a visible extra model call. +- No guidance on what to preserve versus discard; the compaction-item content policy is OpenAI-internal. + +## Relevance to tai + +Validates tai's `compact_messages` trigger design (ratio of a cap, summarize middle, keep recent) as the industry shape. tai's 32k cap with 0.8 trigger is far smaller than the guide's 200k example, which is correct for its weaker backends: the RULER dossier shows claimed context far exceeds usable context on small models. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/prompt-context-scaling/pg-raggraph-ordering.md b/research/prompt-context-scaling/pg-raggraph-ordering.md new file mode 100644 index 0000000..0941529 --- /dev/null +++ b/research/prompt-context-scaling/pg-raggraph-ordering.md @@ -0,0 +1,28 @@ +# pg-raggraph: RAG first, graph second + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | pg-raggraph: RAG first, graph second | +| Source URL | https://github.com/yonk-labs/pg-raggraph | +| Source type | repository | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet states the graph as enhancer design (vector plus BM25 first, one-hop traversal only to rerank or expand weak retrieval). That ordering is what tai's search-plus-expand already does; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/sliamh11-deus-graphrag-comparison.md b/research/prompt-context-scaling/sliamh11-deus-graphrag-comparison.md new file mode 100644 index 0000000..235df7a --- /dev/null +++ b/research/prompt-context-scaling/sliamh11-deus-graphrag-comparison.md @@ -0,0 +1,28 @@ +# Deus research notes: GraphRAG comparison + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Deus research notes: GraphRAG comparison | +| Source URL | https://github.com/sliamh11/deus/blob/HEAD/docs/research/graphrag-comparison.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet scopes LightRAG's "6000x cheaper" claim to retrieval orchestration tokens and notes graph traversal starts paying off around a thousand nodes. Useful cost-scope discipline; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/sudhir13s-graph-rag-notes.md b/research/prompt-context-scaling/sudhir13s-graph-rag-notes.md new file mode 100644 index 0000000..5b198bb --- /dev/null +++ b/research/prompt-context-scaling/sudhir13s-graph-rag-notes.md @@ -0,0 +1,28 @@ +# Graph RAG study notes: when it is overkill + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Graph RAG study notes: when it is overkill | +| Source URL | https://github.com/sudhir13s/ai-ml-learning-resources/blob/HEAD/llms-applications-and-agents/rag-and-knowledge-systems/graph-rag/graph-rag.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the GraphRAG comparison search; the snippet argues single-factoid lookups answered from one chunk gain nothing from the extraction plus community machinery and can get worse through bad extraction. Matches tai's lookup-shaped workload verdict. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/sureprompts-memory-architectures-compared.md b/research/prompt-context-scaling/sureprompts-memory-architectures-compared.md new file mode 100644 index 0000000..478ea5e --- /dev/null +++ b/research/prompt-context-scaling/sureprompts-memory-architectures-compared.md @@ -0,0 +1,28 @@ +# Agent Memory Architectures Compared (2026) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Agent Memory Architectures Compared (2026) | +| Source URL | https://sureprompts.com/blog/agent-memory-architectures-compared-2026 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the memory-layer search; the snippet describes the mem0-style CRUD layer (add after a turn, search before a turn, layer handles extraction and dedup) and warns Letta is heavier than plain fact memory needs. Relevant layer-shape comparison; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/tokenjam-agent-memory.md b/research/prompt-context-scaling/tokenjam-agent-memory.md new file mode 100644 index 0000000..6dab51c --- /dev/null +++ b/research/prompt-context-scaling/tokenjam-agent-memory.md @@ -0,0 +1,28 @@ +# Agent memory: episodic versus semantic representations + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Agent memory: episodic versus semantic representations | +| Source URL | https://github.com/metabuilder-labs/tokenjam-website/blob/HEAD/src/content/blog/2026-05-13-agent-memory.mdx | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the memory-layer search; the snippet notes production systems blend episodic and semantic memory with vector plus graph representations. tai's events-plus-records split mirrors this without vectors; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/prompt-context-scaling/zephyrex-hybrid-memory-mem0.md b/research/prompt-context-scaling/zephyrex-hybrid-memory-mem0.md new file mode 100644 index 0000000..9c7bbc5 --- /dev/null +++ b/research/prompt-context-scaling/zephyrex-hybrid-memory-mem0.md @@ -0,0 +1,28 @@ +# Hybrid memory with Mem0: vector plus KV plus graph + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | prompt-context-scaling | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Hybrid memory with Mem0: vector plus KV plus graph | +| Source URL | https://github.com/zephyrex21/ai-engineering-roadmap/blob/HEAD/phases/14-agent-engineering/09-hybrid-memory-mem0/docs/en.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the memory-layer search; the snippet describes the production hybrid pattern (vector plus KV plus graph stores with a fusing scorer). tai can mirror the shape minus vectors; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/cherryservers-change-memory-limit.md b/research/sandbox-resource-caps/cherryservers-change-memory-limit.md new file mode 100644 index 0000000..a2f932f --- /dev/null +++ b/research/sandbox-resource-caps/cherryservers-change-memory-limit.md @@ -0,0 +1,28 @@ +# How to Change Docker Memory Limit (2 Practical Methods) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | How to Change Docker Memory Limit (2 Practical Methods) | +| Source URL | https://www.cherryservers.com/blog/how-to-change-docker-memory-limit | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the update-limits search; the snippet shows the `docker update --memory` form on a live container. Corroborates the update path tai uses for existing boxes; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/compose-mandatory-limits.md b/research/sandbox-resource-caps/compose-mandatory-limits.md new file mode 100644 index 0000000..c64d6a9 --- /dev/null +++ b/research/sandbox-resource-caps/compose-mandatory-limits.md @@ -0,0 +1,28 @@ +# Docker Compose best practices: mandatory limits + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Docker Compose best practices: mandatory limits | +| Source URL | https://github.com/universalsyntropy/docker-compose-field-guide/blob/HEAD/docs/BEST-PRACTICES.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet makes the CIS trio mandatory (memory cap against OOM-killing neighbours, cpus against starvation, pids-limit 200 against fork bombs). Independent corroboration of the exact three flags tai adopts; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/conclusions/cap-box-trio.md b/research/sandbox-resource-caps/conclusions/cap-box-trio.md new file mode 100644 index 0000000..da98f27 --- /dev/null +++ b/research/sandbox-resource-caps/conclusions/cap-box-trio.md @@ -0,0 +1,58 @@ +# Cap the sandbox box trio + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Conclusion Document | +| Decided | 2026-10-18 | +| Decided by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | web_search aggregator listings + 3 first-hand vendor-doc fetches | +| Confidence | High | +| Status | Active | + +## Question / problem statement + +What resource limits should the tai agent sandbox (`tai_box`) carry, and how do they reach boxes that already exist? + +## Conclusion + +Apply the static ceiling trio `--memory=2g --cpus=2.0 --pids-limit=256` at box creation for both engines (one shared flag list), and extend memory plus cpus to existing boxes with one best-effort `update` per process; the pids ceiling reaches old boxes only through recreation, so a failed update points at box reinstall. + +## Supporting sources + +- [Resource constraints](../docker-resource-constraints.md): containers default to unlimited and can trigger host-wide OOM kills against any process including Docker itself; `--cpus` is the blessed convenient CPU ceiling form. +- [podman-run](../podman-run-manpage.md): all three flags exist with Docker-compatible semantics on tai's preferred engine; podman's own 2048 pids default is no fork-bomb protection, hence an explicit value. +- [podman-update](../podman-update-manpage.md): `update` changes memory and CPU on existing containers but has no pids option (create-only); rootless changes may fail, hence best-effort plus reinstall pointer. +- [Docker Compose best practices: mandatory limits](../compose-mandatory-limits.md) (Tier 2): independent snippet making the same trio mandatory with pids-limit 200 scale. +- [Container resource limits for workspace containers](../klangk-workspace-container-limits.md) (Tier 2): closest analog found, deploy-wide CPU/memory/PIDs caps on workspace containers via podman so a runaway workspace cannot starve the host. + +## Alternatives considered and rejected + +- Keep the box uncapped and rely on the host: rejected, the Docker docs state plainly that host OOM kills any process, and the box runs arbitrary agent shell commands. +- Copy per-service request/limit autoscaling: rejected, a single ephemeral dev box has no scheduler to negotiate with; static ceilings fit the standing rule (constant beats magic). +- Force-recreate the box at upgrade so pids applies everywhere: rejected, recreation wipes the box home (the standing deletion rule); the staggered rollout caps the worst risks now and converges on pids over time. + +## Risks / open questions + +- The 2g value has no tai measurement behind it yet (Docker's "test memory requirements" rule unfulfilled); a real RSS profile for tiny-whisper plus edge-tts could move it either way. +- The pids ceiling reaches pre-existing boxes only through recreation; until then old boxes run with memory plus cpus only. +- `update` is best-effort by design (rootless cgroup writes can fail), so a silent no-change is possible when the engine rejects the call without stderr. + +## Action taken + +Implemented in `tai.py` the same session: `BOX_MEMORY`/`BOX_CPUS`/`BOX_PIDS_LIMIT` constants plus one shared `BOX_RUN_ARGS` tuple feed `ensure_box` (create path, both engines) and `update_box_caps` (memory plus cpus, once per process via `_BOX_CAPS_STAMP`, `TAI_BOX_CAPS=0` suppressor for deterministic tests); failed updates print the concrete `rm -f` recovery command. Covered by `BoxCapsTests` (5 tests) in `test_tai.py`. + +## Revisit when + +- A measured box workload profile exists (actual RSS/CPU peaks for tiny-whisper plus edge-tts) that would justify raising or lowering the static values. +- `podman update` gains a pids option, which would close the rollout gap without recreation. +- The box gains a persistent workload (daemons) that changes the pids or memory budget. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Conclusion recorded; implemented in tai.py same session. | diff --git a/research/sandbox-resource-caps/containerd-releases.md b/research/sandbox-resource-caps/containerd-releases.md new file mode 100644 index 0000000..b574b33 --- /dev/null +++ b/research/sandbox-resource-caps/containerd-releases.md @@ -0,0 +1,28 @@ +# Containerd memory limit announcement + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Containerd memory limit announcement | +| Source URL | https://containerd.io/releases/ | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the update-limits search; off-topic (containerd release notes). Logged for completeness only. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/devto-cgroups-fork-bomb.md b/research/sandbox-resource-caps/devto-cgroups-fork-bomb.md new file mode 100644 index 0000000..0b4e46a --- /dev/null +++ b/research/sandbox-resource-caps/devto-cgroups-fork-bomb.md @@ -0,0 +1,28 @@ +# How Docker Works Under the Hood: cgroups stop a fork bomb + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | How Docker Works Under the Hood: cgroups stop a fork bomb | +| Source URL | http://dev.to/devopsdaily/how-docker-works-under-the-hood-watch-namespaces-isolate-a-container-and-cgroups-stop-a-fork-bomb-2k22 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet demonstrates `docker run --pids-limit 100` setting the cgroup ceiling and notes that without the flag a runaway container is bounded only by host limits. Worked-example corroboration; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/devto-operating-containers-limits.md b/research/sandbox-resource-caps/devto-operating-containers-limits.md new file mode 100644 index 0000000..6de4d6e --- /dev/null +++ b/research/sandbox-resource-caps/devto-operating-containers-limits.md @@ -0,0 +1,28 @@ +# Operating Containers: healthchecks, limits, restart policies + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Operating Containers: healthchecks, limits, restart policies | +| Source URL | https://dev.to/shubham_sharma_94/operating-containers-healthchecks-resource-limits-restart-policies-and-env-config-367h | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet notes memory excess kills with exit 137 and pairs limits with the restart policy. Relevant because tai's box already restarts unless-stopped, so an OOM kill self-heals; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/docker-manage-data.md b/research/sandbox-resource-caps/docker-manage-data.md new file mode 100644 index 0000000..fffb1c1 --- /dev/null +++ b/research/sandbox-resource-caps/docker-manage-data.md @@ -0,0 +1,28 @@ +# Docker docs: Manage data in Docker + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Docker docs: Manage data in Docker | +| Source URL | https://docs.docker.com/engine/storage/ | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the update-limits search; off-topic (storage drivers, not cgroup limits). Logged for completeness only. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/docker-resource-constraints.md b/research/sandbox-resource-caps/docker-resource-constraints.md new file mode 100644 index 0000000..0af621a --- /dev/null +++ b/research/sandbox-resource-caps/docker-resource-constraints.md @@ -0,0 +1,52 @@ +# Resource constraints + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Resource constraints | +| Source URL | https://docs.docker.com/engine/containers/resource_constraints/ | +| Source publisher / author | Docker | +| Source type | vendor docs | +| Source date (published) | Unknown, not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +Docker's resource-constraint reference: containers default to no limits and can use whatever the host scheduler allows; an uncapped container can trigger host-wide OOM kills against unrelated processes. Memory gets hard limits, CPU gets the `--cpus` ceiling as the convenient CFS quota form, and swap needs explicit attention alongside memory. + +## Key findings + +- "By default, a container has no resource constraints and can use as much of a given resource as the host's kernel scheduler allows." +- "On Linux hosts, if the kernel detects that there isn't enough memory to perform important system functions, it throws an `OOME`, or `Out Of Memory Exception`, and starts killing processes to free up memory. Any process is subject to killing, including Docker and other important applications." +- "`--cpus=`: Specify how much of the available CPU resources a container can use... This is the equivalent of setting `--cpu-period=\"100000\"` and `--cpu-quota=\"150000\"`." +- "For most use-cases, `--cpus` is a more convenient alternative" to raw period/quota knobs. +- Mitigations listed: test memory requirements, run on adequate hosts, limit container memory, be mindful of swap configuration. + +## Pros / strengths + +- Authoritative statement of the default (unlimited) and the failure mode (host OOM), which is exactly tai's pre-cap exposure. +- Blesses `--cpus` as the standard CPU ceiling form both engines accept. + +## Cons / limitations / caveats + +- Docker-authored; flag parity with podman verified separately in the podman-run dossier, not here. +- Swap semantics (memory-plus-swap totals) need care; tai sets memory only and accepts the documented roughly-2x-with-swap worst case as still bounded. + +## Relevance to tai + +Justifies capping the agent sandbox on all three axes: the box runs arbitrary agent shell commands, so an uncapped box is one buggy loop from host OOM. tai takes memory plus cpus from this page and pids-limit from the CIS baseline card. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sandbox-resource-caps/docker-tips-limit-resources.md b/research/sandbox-resource-caps/docker-tips-limit-resources.md new file mode 100644 index 0000000..e232a18 --- /dev/null +++ b/research/sandbox-resource-caps/docker-tips-limit-resources.md @@ -0,0 +1,28 @@ +# Docker tips: limit memory and CPU + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Docker tips: limit memory and CPU | +| Source URL | https://fastdomains.com.au/2026/10/05/12-docker-tips-to-save-disk-space-limit-resources-and-secure-containers-on-linux/ | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet pairs a memory hard limit (OOM kills stay inside the container) with cpus alongside the restart policy tai already uses. Same trio, practitioner voice; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/fork-bomb-pattern-notes.md b/research/sandbox-resource-caps/fork-bomb-pattern-notes.md new file mode 100644 index 0000000..483c44c --- /dev/null +++ b/research/sandbox-resource-caps/fork-bomb-pattern-notes.md @@ -0,0 +1,28 @@ +# Fork bomb attack pattern notes + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Fork bomb attack pattern notes | +| Source URL | https://github.com/wgen1995/redteam-agent/blob/HEAD/docs/design/imported/gencpt/GenCPT-main/attack-patterns/dos/fork-bomb/SKILL.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet names pids.max ceilings (1024/4096 scale) as the first verification layer against fork bombs. Attacker-side confirmation that the pids controller is the defense; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/k8s-assign-memory.md b/research/sandbox-resource-caps/k8s-assign-memory.md new file mode 100644 index 0000000..808d3ea --- /dev/null +++ b/research/sandbox-resource-caps/k8s-assign-memory.md @@ -0,0 +1,28 @@ +# Kubernetes: Assign Memory Resources to Containers and Pods + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Kubernetes: Assign Memory Resources to Containers and Pods | +| Source URL | https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/ | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the update-limits search; the snippet covers OOM scoring and eviction under memory pressure. Out of scope for tai (no k8s), kept because the OOM mechanics generalize. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/k8s-manage-resources.md b/research/sandbox-resource-caps/k8s-manage-resources.md new file mode 100644 index 0000000..e11fcf4 --- /dev/null +++ b/research/sandbox-resource-caps/k8s-manage-resources.md @@ -0,0 +1,28 @@ +# Kubernetes resource-management for pods and containers + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Kubernetes resource-management for pods and containers | +| Source URL | https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the update-limits search; off-topic (Kubernetes QoS classes). Logged for completeness only. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/klangk-workspace-container-limits.md b/research/sandbox-resource-caps/klangk-workspace-container-limits.md new file mode 100644 index 0000000..be83231 --- /dev/null +++ b/research/sandbox-resource-caps/klangk-workspace-container-limits.md @@ -0,0 +1,28 @@ +# Container resource limits for workspace containers + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Container resource limits for workspace containers | +| Source URL | https://github.com/mcdonc/klangk/commit/755f63bfea3c7c42c9e1aa79894a89e4c5ecf1e3 | +| Source type | repository | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet describes deploy-wide CPU/memory/PIDs caps on workspace containers via podman flags so a runaway workspace cannot starve the host. The closest analog to tai's agent box found anywhere; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/medium-cpu-memory-production.md b/research/sandbox-resource-caps/medium-cpu-memory-production.md new file mode 100644 index 0000000..4205733 --- /dev/null +++ b/research/sandbox-resource-caps/medium-cpu-memory-production.md @@ -0,0 +1,28 @@ +# Docker CPU and memory management in production + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Docker CPU and memory management in production | +| Source URL | https://medium.com/@earchitha22/day-17-docker-resource-limits-cpu-memory-management-in-production-ebfb5886cba9 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet shows memory plus cpus applied together as the production norm. Redundant with stronger sources; logged for completeness. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/medium-docker-resource-limits.md b/research/sandbox-resource-caps/medium-docker-resource-limits.md new file mode 100644 index 0000000..d8f829f --- /dev/null +++ b/research/sandbox-resource-caps/medium-docker-resource-limits.md @@ -0,0 +1,28 @@ +# Docker resource limits explained + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Docker resource limits explained | +| Source URL | https://medium.com/@sivasai2332/day-14-docker-resource-limits-explained-d71488a68d4b | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet states the production rules (always set memory limits, measure rather than guess, watch OOM kills). Restates the vendor guidance in checklist form; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/medium-limit-cpu-memory.md b/research/sandbox-resource-caps/medium-limit-cpu-memory.md new file mode 100644 index 0000000..17a7559 --- /dev/null +++ b/research/sandbox-resource-caps/medium-limit-cpu-memory.md @@ -0,0 +1,28 @@ +# How to Limit CPU and Memory Usage in Docker Containers + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | How to Limit CPU and Memory Usage in Docker Containers | +| Source URL | https://medium.com/codex/how-to-limit-cpu-and-memory-usage-in-docker-containers-a024b429f55e | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet documents `--cpus` and `--memory` plus the `--memory-swap` total. Noted for the swap caveat carried in the conclusion; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/phoenixnap-docker-limits.md b/research/sandbox-resource-caps/phoenixnap-docker-limits.md new file mode 100644 index 0000000..3a332d9 --- /dev/null +++ b/research/sandbox-resource-caps/phoenixnap-docker-limits.md @@ -0,0 +1,28 @@ +# How to Set Docker Memory & CPU Limits + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | How to Set Docker Memory & CPU Limits | +| Source URL | https://phoenixnap.com/kb/docker-memory-and-cpu-limit | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the update-limits search; the snippet shows `docker update` setting memory plus cpus on an existing container and notes containers need not be restarted. Live-update corroboration; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/podman-pids-default.md b/research/sandbox-resource-caps/podman-pids-default.md new file mode 100644 index 0000000..d665a0f --- /dev/null +++ b/research/sandbox-resource-caps/podman-pids-default.md @@ -0,0 +1,28 @@ +# podman-container.unit PidsLimit default + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | podman-container.unit PidsLimit default | +| Source URL | https://docs.podman.io/en/v6.1.1/markdown/podman-container.unit.5.html | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the container-limits search; the snippet records podman's 2048 default pids ceiling where supported. Shows engine defaults are no protection, hence explicit values; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/podman-run-manpage.md b/research/sandbox-resource-caps/podman-run-manpage.md new file mode 100644 index 0000000..63b5461 --- /dev/null +++ b/research/sandbox-resource-caps/podman-run-manpage.md @@ -0,0 +1,50 @@ +# podman-run + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | podman-run | +| Source URL | https://docs.podman.io/en/stable/markdown/podman-run.1.html | +| Source publisher / author | Podman | +| Source type | vendor docs | +| Source date (published) | Unknown, not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +The podman-run reference confirms flag parity with Docker for the cap trio tai needs (`--memory`, `--cpus`, `--pids-limit`) and documents the pids controller as the fork-bomb defense. Podman-only defaults differ (pids default 2048 where supported), so tai sets its own explicit values rather than relying on either engine's default. + +## Key findings + +- Run supports `--pids-limit`: "Use the --pids-limit option to modify the cgroup control to limit the number of processes within a container." +- `--memory` and `--cpus` documented with the same unit forms and semantics as Docker (verified flag sections present in the manpage). +- Podman default pids limit is 2048 on systems supporting the pids cgroup (per the podman unit-file docs seen in search), which is far too generous as fork-bomb protection. + +## Pros / strengths + +- Primary proof that one flag list works for both engines tai supports (podman preferred, docker fallback). +- Explicit pids-controller documentation anchors the fork-bomb half of the trio. + +## Cons / limitations / caveats + +- Very long reference page read selectively (flag sections only); unrelated options not reviewed. +- Rootless cgroup limits can fail with permissions errors on some systems (documented under podman-update), so application of caps must tolerate failure loudly rather than assume it. + +## Relevance to tai + +Approves a single shared flag builder for both engines: `--memory`, `--cpus`, `--pids-limit` at create time, with explicit values instead of engine defaults. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sandbox-resource-caps/podman-update-manpage.md b/research/sandbox-resource-caps/podman-update-manpage.md new file mode 100644 index 0000000..e8b856b --- /dev/null +++ b/research/sandbox-resource-caps/podman-update-manpage.md @@ -0,0 +1,51 @@ +# podman-update + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | podman-update | +| Source URL | https://docs.podman.io/en/stable/markdown/podman-update.1.html | +| Source type | vendor docs | +| Source date (published) | Unknown, not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +The podman-update reference allows changing memory and CPU limits on an existing container, but offers no pids option: live boxes can gain memory plus CPU caps, while the pids ceiling needs container recreation. This split shapes tai's rollout exactly. + +## Key findings + +- "Updates the configuration of an existing container, allowing changes to resource limits and healthchecks." +- `--cpus`: "Number of CPUs. The default is 0.0 which means no limit", shorthand for cpu-period plus cpu-quota. +- `--memory`: "Memory limit", units b/k/m/g; "If a limit of 0 is specified (not using --memory), the container's memory is not limited." +- No `--pids-limit` (or equivalent) option exists on the update page; pids is create-only. +- Rootless limit changes "may not be allowed" on some systems, so update must be best-effort. + +## Pros / strengths + +- Decisive for the rollout design: one best-effort update call covers existing boxes for the two biggest risks, and the docs say plainly what it cannot do. +- The rootless caveat justifies tolerating update failure with a loud pointer to reinstall instead of hard-failing. + +## Cons / limitations / caveats + +- Podman-side only; docker-update parity assumed from shared flag names but not verified first-hand (the call is best-effort either way, so a docker miss degrades to the reinstall pointer). +- No statement on whether partially-applied updates are possible; tai treats update as all-or-nothing per invocation. + +## Relevance to tai + +Specifies the rollout: caps at create for new boxes, one best-effort `update` (memory plus cpus) per process for existing boxes, reinstall pointer when update cannot run. Pids-limit reaches old boxes only through recreation. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sandbox-resource-caps/runc-talks-listing.md b/research/sandbox-resource-caps/runc-talks-listing.md new file mode 100644 index 0000000..9b720c0 --- /dev/null +++ b/research/sandbox-resource-caps/runc-talks-listing.md @@ -0,0 +1,28 @@ +# runc and container isolation talks + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | runc and container isolation talks | +| Source URL | https://www.youtube.com/results?search_query=runc+container+isolation+talks | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the update-limits search; off-topic (video listing). Logged for completeness only. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sandbox-resource-caps/stackoverflow-running-limits.md b/research/sandbox-resource-caps/stackoverflow-running-limits.md new file mode 100644 index 0000000..1731977 --- /dev/null +++ b/research/sandbox-resource-caps/stackoverflow-running-limits.md @@ -0,0 +1,28 @@ +# Changing resource limits on a running Docker container + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sandbox-resource-caps | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Changing resource limits on a running Docker container | +| Source URL | https://stackoverflow.com/questions/35146613/changing-resource-limits-on-a-running-docker-container | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the update-limits search; the snippet indicates some limits apply live via update while others need recreation. The exact split comes from the vendor docs; this only corroborates the split exists. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/ai-engineering-book-hyde-lexical.md b/research/sqlite-rag-hybrid-search/ai-engineering-book-hyde-lexical.md new file mode 100644 index 0000000..191cec9 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/ai-engineering-book-hyde-lexical.md @@ -0,0 +1,28 @@ +# AI engineering book notes: HyDE hurts lexical lookup + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | AI engineering book notes: HyDE hurts lexical lookup | +| Source URL | https://github.com/xayyamsadigov/ai-engineering-book/blob/HEAD/book/solutions/ch12-solutions.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; the snippet explains the mechanism plainly (a lexical engine matches invented HyDE terms literally, retrieving documents that agree with the hallucination) and notes it dilutes the one token that matters in identifier lookups. The mechanistic reason behind tai's HyDE rejection. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/arxiv-2010-13659-query-translation.md b/research/sqlite-rag-hybrid-search/arxiv-2010-13659-query-translation.md new file mode 100644 index 0000000..dbfe1c5 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/arxiv-2010-13659-query-translation.md @@ -0,0 +1,28 @@ +# Exploiting Neural Query Translation into Cross Lingual Information Retrieval + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Exploiting Neural Query Translation into Cross Lingual Information Retrieval | +| Source URL | http://arxiv.org/pdf/2010.13659v1 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the implicit-feedback search; the snippet cites Joachims on clickthrough-derived ranking signals in passing. Off-topic paper (cross-lingual query translation) that surfaced only via citation text; logged, not relied on. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/arxiv-2012-14005-doc-expansion.md b/research/sqlite-rag-hybrid-search/arxiv-2012-14005-doc-expansion.md new file mode 100644 index 0000000..39926dd --- /dev/null +++ b/research/sqlite-rag-hybrid-search/arxiv-2012-14005-doc-expansion.md @@ -0,0 +1,28 @@ +# Neural document expansion for adhoc information retrieval + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Neural document expansion for adhoc information retrieval | +| Source URL | https://arxiv.org/pdf/2012.14005 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; the snippet notes document expansion runs offline with no query-time cost, unlike RM3's two retrieval rounds. The offline side of the expansion tradeoff; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/arxiv-2104-08663-beir.md b/research/sqlite-rag-hybrid-search/arxiv-2104-08663-beir.md new file mode 100644 index 0000000..4345e65 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/arxiv-2104-08663-beir.md @@ -0,0 +1,51 @@ +# BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models | +| Source URL | https://arxiv.org/html/2104.08663v4 | +| Source publisher / author | Nandan Thakur et al. | +| Source type | paper | +| Source date (published) | 2021-04-28 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +BEIR standardizes cross-dataset retrieval evaluation on nDCG@10: rank-unaware precision/recall cannot compare rankings, and binary MRR/MAP cannot handle graded relevance, while nDCG covers both. tai adopts the metric-family reasoning (rank-aware, graded-capable) at its own small k. + +## Key findings + +- "Decision support metrics such as Precision and Recall which are both rank unaware are not suitable. Binary rank-aware metrics such as MRR (Mean Reciprocal Rate) and MAP (Mean Average Precision) fail to evaluate tasks with graded relevance judgements." +- "We find that Normalised Cumulative Discount Gain (nDCG@k) provides a good balance suitable for both tasks involving binary and graded relevance judgements." +- BEIR computes nDCG@10 for all datasets via the official TREC eval tool. +- Scope note: the metric-rationale section was read first-hand; the appendix recall@K definition quoted in the gather pack was not re-verified here, and recall@K is used below only in its standard textbook sense (fraction of relevant documents in the top k). + +## Pros / strengths + +- The field's standard justification for which metric to report, from an 18-dataset benchmark; settles metric choice by argument, not fashion. +- Rank-aware plus graded-capable maps onto tai's needs exactly (binary fingerprint hits today, graded LLM judgments later). + +## Cons / limitations / caveats + +- A benchmark paper for comparing retrievers across datasets, not a guide to building a small-corpus eval; k=10 and TREC tooling do not transfer literally to tai's vault scale. +- nDCG needs graded judgments to earn its keep over MRR; with purely binary goldens it reduces to rank-aware bookkeeping. + +## Relevance to tai + +Sets the metric ladder for the standing eval: Hit@K/MRR for binary fingerprint goldens now (cheap, exact), nDCG@K once graded relevance (LLM-judged or read-depth) exists. Rules out reporting bare accuracy or set-overlap numbers that ignore rank. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sqlite-rag-hybrid-search/arxiv-2309-15217-ragas.md b/research/sqlite-rag-hybrid-search/arxiv-2309-15217-ragas.md new file mode 100644 index 0000000..9597010 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/arxiv-2309-15217-ragas.md @@ -0,0 +1,51 @@ +# Ragas: Automated Evaluation of Retrieval Augmented Generation + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Ragas: Automated Evaluation of Retrieval Augmented Generation | +| Source URL | https://arxiv.org/html/2309.15217v2 | +| Source publisher / author | Shahul Es et al. | +| Source type | paper | +| Source date (published) | 2023-09-27 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +RAGAS defines reference-free RAG evaluation along three dimensions: faithfulness (answer grounded in context), answer relevance (answer addresses the question), and context relevance (retrieved context contains only what the question needs, penalizing redundancy). Metrics are computed by prompting an LLM, needing no human-labeled dataset. + +## Key findings + +- "We usually do not have access to human-annotated datasets or reference answers. We therefore focus on metrics that are fully self-contained and reference-free." +- "First, Faithfulness refers to the idea that the answer should be grounded in the given context... Second, Answer Relevance refers to the idea that the generated answer should address the actual question that was provided. Finally, Context Relevance refers to the idea that the retrieved context should be focused, containing as little irrelevant information as possible." +- "The context is considered relevant to the extent that it exclusively contains information that is needed to answer the question. In particular, this metric aims to penalise the inclusion of redundant information." +- Context relevance is estimated by having the LLM extract the subset of sentences crucial to the answer, returning "Insufficient Information" when the context cannot answer the question. + +## Pros / strengths + +- Directly answers tai's no-labels problem: reference-free, LLM-computed metrics purpose-built for RAG pipelines. +- Separates retriever quality (context relevance) from generator quality (faithfulness, answer relevance), so a retrieval-tuning loop can ignore generation entirely. + +## Cons / limitations / caveats + +- LLM-judged metrics inherit judge noise and cost a model call per evaluation; the original experiments used GPT-3.5-turbo, stronger than tai's free-tier judges. +- tai's adoption would reimplement the two context metrics natively; the RAGAS package itself (LangChain-adjacent dependencies) is not adoptable under stdlib-only. + +## Relevance to tai + +Justifies the top layer of the standing eval: deterministic rank metrics first (free, exact), LLM-judged context relevance second for the ambiguous remainder. tai needs only the context-side metrics, not the full RAGAS suite, since generation quality is out of scope for retrieval tuning. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sqlite-rag-hybrid-search/arxiv-2408-04197-pairwise-judgment.md b/research/sqlite-rag-hybrid-search/arxiv-2408-04197-pairwise-judgment.md new file mode 100644 index 0000000..e809a89 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/arxiv-2408-04197-pairwise-judgment.md @@ -0,0 +1,28 @@ +# Pairwise Judgment Formulation for Semantic Embedding Model in Web Search + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Pairwise Judgment Formulation for Semantic Embedding Model in Web Search | +| Source URL | http://arxiv.org/pdf/2408.04197 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the implicit-feedback search; the snippet cites Joachims on click-derived pairwise preferences as reasonably accurate. Corroborating pointer to the first-hand Joachims dossier; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/arxiv-2408-08067-ragchecker.md b/research/sqlite-rag-hybrid-search/arxiv-2408-08067-ragchecker.md new file mode 100644 index 0000000..809e3d1 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/arxiv-2408-08067-ragchecker.md @@ -0,0 +1,50 @@ +# RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation | +| Source URL | https://arxiv.org/html/2408.08067v1 | +| Source publisher / author | Unknown — not stated | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +RAGChecker diagnoses retriever and generator separately at claim level: the retriever gets claim recall (ground-truth claims covered by retrieved chunks) and chunk-level context precision (fraction of retrieved chunks entailing any ground-truth claim), while the generator gets faithfulness plus noise-sensitivity and hallucination rates. The retriever half is what tai needs. + +## Key findings + +- "Completeness-wise, we can measure how many claims made in the ground-truth answer are covered by retrieved chunks", formalized as claim recall over entailed ground-truth claims. +- "A retrieved chunk is called relevant chunk if any ground-truth claim is entailed in it", and "the retriever's context precision is defined as |r-chunks|/k, where k is the number of all retrieved chunks." +- "A chunk-level precision provides better interpretability than a claim-level one, because in practice RAG systems usually work with documents processed to be text chunks in a fixed size." +- The same section warns a chunk can mix relevant and misleading content, so claim-level precision has a below-100% ceiling that varies with chunking strategy. + +## Pros / strengths + +- Claim-level framing fits tai's fingerprint goldens perfectly: each golden asserts claims answerable from exactly one record or chunk. +- The chunking-ceiling warning turns chunk size into a measured variable rather than a fixed constant. + +## Cons / limitations / caveats + +- Entailment judgments need an LLM or human per claim-chunk pair; the paper's full harness is heavier than tai needs (generator metrics out of scope). +- Author list not captured (metadata section of the HTML page not verified); the arXiv listing carries the canonical credit. + +## Relevance to tai + +Supplies the diagnostic split for tuning: claim recall catches the strict-AND recall gap (relevant chunk not retrieved at all), context precision catches ranking and noise (wrong chunks ranked high). When a tuning knob moves one metric but not the other, this paper says which half of the pipeline to blame. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sqlite-rag-hybrid-search/arxiv-2511-19349-hyde-feedback.md b/research/sqlite-rag-hybrid-search/arxiv-2511-19349-hyde-feedback.md new file mode 100644 index 0000000..037e9a7 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/arxiv-2511-19349-hyde-feedback.md @@ -0,0 +1,28 @@ +# Revisiting Feedback Models for HyDE + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Revisiting Feedback Models for HyDE | +| Source URL | https://arxiv.org/pdf/2511.19349v1 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; the snippet describes HyDE as LLM-expanded hypothetical documents bridging the vocabulary gap for bag-of-words retrieval. Noted against the stronger counter-evidence that HyDE passages hurt literal lexical matching; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/arxiv-2608-11238-qcare.md b/research/sqlite-rag-hybrid-search/arxiv-2608-11238-qcare.md new file mode 100644 index 0000000..2fc483f --- /dev/null +++ b/research/sqlite-rag-hybrid-search/arxiv-2608-11238-qcare.md @@ -0,0 +1,28 @@ +# Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability | +| Source URL | https://arxiv.org/abs/2608.11238v3 | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the scholarship gather worker; reported coverage-aware retriever metrics with higher human correlation than RAGChecker on eight datasets. A possible future metric upgrade; very recent preprint, not first-hand read. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/conclusions/lexical-tuning-roadmap.md b/research/sqlite-rag-hybrid-search/conclusions/lexical-tuning-roadmap.md new file mode 100644 index 0000000..5cbc8f1 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/conclusions/lexical-tuning-roadmap.md @@ -0,0 +1,55 @@ +# Conclusion: Lexical tuning roadmap, gated on measurements + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +The code dive found tai's retrieval running bare `bm25()` with strict AND semantics over title plus body: no column weights, no OR fallback when AND returns empty, no prefix matching, default porter stemming. The question was which tuning moves are available dependency-free, in what order to try them, and which popular techniques to keep out. + +## Conclusion + +Tune in this order, each gated on a before/after number from the standing eval, and stop at the first step that closes the measured gap: (1) BM25 column weights weighting title above body, the only ranking knob FTS5 exposes; (2) OR-fallback query when strict AND returns nothing, since multi-term AND is the sharpest recall cliff in the current shape; (3) prefix (`*`) matching for partial terms such as hashes and paths. Explicitly rejected: HyDE-style expansion against the lexical index (measured to hurt BM25, including a 0.2635 to 0.1954 nDCG@5 drop on tool retrieval), default-on RM3 (drifts short queries), k1/b tuning (hardcoded in FTS5), and any embedding path (prior veto stands). + +## Supporting sources + +- [sqlite-fts5-docs](../sqlite-fts5-docs.md): k1/b hardcoded; per-column weights are the only ranking knob; prefix and boolean operators available. +- [jev-tool-search-hyde-hurts-bm25](../jev-tool-search-hyde-hurts-bm25.md) (Tier 2): HyDE cut BM25 nDCG@5 on tool retrieval, the most tai-shaped negative datapoint. +- [ai-engineering-book-hyde-lexical](../ai-engineering-book-hyde-lexical.md) (Tier 2): the mechanism (lexical engines match invented terms literally, diluting the one token that matters in identifier lookups). +- [mdpi-rm3-hurts-short-queries](../mdpi-rm3-hurts-short-queries.md) (Tier 2): BM25 beat BM25 plus RM3 on short claim queries via drift. +- [vespa-bm25-rank-feature](../vespa-bm25-rank-feature.md) (Tier 2): BM25-first-then-rerank as the standard shape, bounding tai's ambitions to first-phase quality. +- [muenster-hyde-thesis](../muenster-hyde-thesis.md) and [hyde-paper-tables](../hyde-paper-tables.md) (Tier 2): the pro-HyDE side, both for dense retrieval, recorded as the disagreeing scope. + +## Alternatives considered and rejected + +- HyDE expansion: rejected, helps dense retrieval while hurting lexical matching on exactly tai's identifier-heavy content. +- RM3 pseudo-relevance feedback by default: rejected for short queries; may be revisited as an opt-in second pass if recall-side eval shows a gap AND terms are long enough to resist drift. +- Custom tokenizer or stemmer swap: rejected for now, porter plus unicode61 is the sane default and no eval shows a tokenization failure yet. +- Reranking layer (cross-encoder or LLM rubric): rejected for now on cost and dependencies; reconsider only if first-phase tuning plateaus below target with ranking (not recall) as the diagnosed cause. + +## Risks / open questions + +- Column weights need per-kind care: title-heavy weighting helps records but chunks have synthetic titles, so weights may need to differ by kind. +- OR-fallback can flood results on common terms; cap and rank it strictly below the AND pass, never merged blindly. +- Revisit if: the standing eval shows recall (not ranking) as the binding constraint after weights, which would promote the OR-fallback and prefix steps. + +## Action taken + +Built the same day, in the specified order, each gated on the fixture numbers: (1) `FTS_TITLE_WEIGHT = 3.0` over 1.0 body, measured to flip title-above-body-spam at 5x repetition with 2.0 and at 10x with 3.0, with no fixture or suite regression (967 unit tests green); (2) OR-fallback plus (3) prefix queries shipped together as a first-non-empty-wins cascade (strict AND, then prefix-AND, then OR) in `fts_search`, the sealed event mirror, and `search_records_fts`, moving fixture prefix and noisy Hit@5 from 0.000 to 1.000. HyDE, default-on RM3, and embeddings stay rejected as specified; no reranker was needed. One cascade subtlety found during measurement: a lenient level can return results excluding the right answer while a more lenient level would include it, so noisy goldens use true-noise terms (absent from the vault) to stay fair. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | +| 2026-10-18 | Implemented as specified with measured before/after numbers recorded above. | diff --git a/research/sqlite-rag-hybrid-search/conclusions/standing-retrieval-eval.md b/research/sqlite-rag-hybrid-search/conclusions/standing-retrieval-eval.md new file mode 100644 index 0000000..260cd36 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/conclusions/standing-retrieval-eval.md @@ -0,0 +1,56 @@ +# Conclusion: Standing retrieval eval from goldens, rank metrics, and reads + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-18 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +The original conclusion in this subject shipped BM25-only retrieval with an explicit open risk: the lexical-match tradeoff was never measured against real retrieval-quality metrics. The prompt-scaling research then made that risk load-bearing, because spilling giant prompts to records only works if retrieval finds the right slices. The question was how to measure retrieval quality with no labeled data, no new dependencies, and a corpus shaped like command output and logs. + +## Conclusion + +Build a standing retrieval eval with three layers, in this order: (1) deterministic rank metrics (Hit@K, MRR, nDCG@K, Context Precision) over `fts_search` against golden query-plus-expected-id pairs generalized from the fingerprint method in tests/rag.py; (2) implicit pairwise preferences mined from search-then-read sequences using the existing `reads` counter, interpreted relatively per Joachims, never as absolute labels; (3) LLM-judged context relevance only for the ambiguous remainder the first two layers cannot grade. Close the logging gap first: `run_search` must log the query with returned ids, because results alone cannot be mined. + +## Supporting sources + +- [arxiv-2104-08663-beir](../arxiv-2104-08663-beir.md): rank-aware metric rationale; nDCG covers binary and graded relevance, bare accuracy does not. +- [ragas-context-precision-docs](../ragas-context-precision-docs.md): the exact mean-precision@k ranking formula to reimplement natively. +- [arxiv-2309-15217-ragas](../arxiv-2309-15217-ragas.md): reference-free context metrics when no labels exist; retriever-side only for tai's purposes. +- [arxiv-2408-08067-ragchecker](../arxiv-2408-08067-ragchecker.md): claim recall versus context precision split, which tells tuning apart (missing chunk versus misranked chunk). +- [deepeval-synthesizer-goldens](../deepeval-synthesizer-goldens.md): goldens-from-own-docs is production-standard; tai's chunker plus FTS index already supply the context-construction half. +- [joachims-clickthrough-implicit-feedback](../joachims-clickthrough-implicit-feedback.md): clicks informative but biased; relative preferences reasonably accurate, which is exactly how to read search-then-read sequences. +- [radlinski-joachims-query-chains](../radlinski-joachims-query-chains.md) (Tier 2): preferences hold across reformulation chains, covering tai's retry-shaped search behavior. + +## Alternatives considered and rejected + +- Adopting RAGAS/DeepEval as dependencies: rejected, dependency-hostile (langchain, chroma, API keys); reimplement the two retriever formulas natively instead. +- Human-labeled eval set: rejected as the starting point (expensive, slow); kept as the calibration backstop the log-mined signal gets spot-checked against. +- Judging retrieval from answer quality end-to-end: rejected, conflates retriever and generator; RAGAS/RAGChecker both insist on separating the two. +- Skipping measurement and tuning blind: rejected, this was the open risk being closed; no tuning knob moves without a before/after number after this conclusion. + +## Risks / open questions + +- Synthesized queries can leak source phrasing and flatter lexical scores; pair every paraphrased golden with a fingerprint-exact one and report both. +- The reads signal reflects agent behavior, not human relevance; useful for reranking comparisons, never for certification. +- Revisit if: golden scores and real-task success diverge (the eval measures the wrong thing), or graded LLM judgments prove too noisy on free-tier backends. + +## Action taken + +Built the same day the conclusion was recorded: `tai.RetrievalEval` (Hit@K, MRR, binary nDCG@K, mean precision@k; deterministic golden generator; implicit search-then-read miner), `search-query` and `record-read` event logging in `tai.Tools.run_search` / `run_record_read`, and the read-only `tai.py --eval-retrieval` CLI. Measured on the 5-record unit fixture (20 goldens): baseline Hit@5 0.500 with prefix and noisy at 0.000, post-tuning 1.000 in all four categories. Measured on the real vault (50 records, 200 goldens): Hit@5 0.890 overall (exact 0.92, noisy 0.94, phrase 0.96, prefix 0.74), MRR 0.708. The exact category needed rarest-term golden selection (first-token goldens scored 0.22, measuring golden noise, not the engine). Implicit mining is live but has no history yet (0 searches at ship time). + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial conclusion recorded. | +| 2026-10-18 | Implemented as specified; baseline and post-tuning numbers recorded above. | diff --git a/research/sqlite-rag-hybrid-search/deepeval-contextual-precision.md b/research/sqlite-rag-hybrid-search/deepeval-contextual-precision.md new file mode 100644 index 0000000..8a9fb44 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/deepeval-contextual-precision.md @@ -0,0 +1,28 @@ +# Contextual Precision - DeepEval + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Contextual Precision - DeepEval | +| Source URL | https://deepeval.com/docs/metrics-contextual-precision | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the industry gather worker; reported LLM-judged ranking-order scoring with a reason string. Independent second source for judge-based context precision alongside the first-hand Ragas docs; not opened first-hand. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/deepeval-synthesizer-goldens.md b/research/sqlite-rag-hybrid-search/deepeval-synthesizer-goldens.md new file mode 100644 index 0000000..e96cdbe --- /dev/null +++ b/research/sqlite-rag-hybrid-search/deepeval-synthesizer-goldens.md @@ -0,0 +1,49 @@ +# Generate Goldens From Documents + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Generate Goldens From Documents | +| Source URL | https://deepeval.com/docs/synthesizer-generate-from-docs | +| Source publisher / author | DeepEval (Confident AI) | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +DeepEval's synthesizer generates eval goldens directly from the knowledge-base documents themselves (documents to contexts to question-plus-expected-output pairs, two per context by default), precisely the no-labels situation tai is in. The catch is the dependency bill: chroma, langchain text splitters, and an OpenAI key. + +## Key findings + +- "If your application is a Retrieval-Augmented Generation (RAG) system, generating Goldens from documents can be particularly useful, especially if you already have access to the documents that make up your knowledge base." +- One mandatory parameter (document paths), optional expected outputs (default on) and max goldens per context (default 2). +- Prerequisites: "chromadb: required for chunk storage and retrieval", "langchain-core, langchain-community, langchain-text-splitters: required for document parsing and chunking", plus an OpenAI key or custom models. + +## Pros / strengths + +- Proves the goldens-from-own-docs pattern is production-standard, not a shortcut: the eval corpus comes from the same documents the retriever serves. +- Two-goldens-per-context default gives tai a sizing rule for its own generator. + +## Cons / limitations / caveats + +- Unadoptable as a dependency (chroma, langchain, API key); tai must reimplement the pattern with its own chunker, its own FTS index, and its own backends. +- LLM-synthesized questions can leak phrasing from the source chunk, inflating lexical-retrieval scores; fingerprint-style exact-answerable goldens avoid this, paraphrased ones need care. + +## Relevance to tai + +Validates generalizing tests/rag.py's fingerprint method into a standing golden generator: sample vault records and chunks, synthesize answerable queries (exact fingerprint plus paraphrase), record the expected ids, and score retrieval against them. tai's chunker plus FTS index already provide the "context construction" half for free. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sqlite-rag-hybrid-search/finalign-hybrid-qa-stack.md b/research/sqlite-rag-hybrid-search/finalign-hybrid-qa-stack.md new file mode 100644 index 0000000..401c41c --- /dev/null +++ b/research/sqlite-rag-hybrid-search/finalign-hybrid-qa-stack.md @@ -0,0 +1,28 @@ +# finalign: hybrid financial QA with BM25, HyDE, and reranking + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | finalign: hybrid financial QA with BM25, HyDE, and reranking | +| Source URL | https://github.com/pdmdp/finalign | +| Source type | repository | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; the snippet lists a full hybrid stack (BM25 plus RM3, dense retrieval, HyDE rewriting, fusion, MonoT5 rerank) on FiQA. Shows the kitchen-sink alternative tai deliberately does not build; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/hyde-paper-tables.md b/research/sqlite-rag-hybrid-search/hyde-paper-tables.md new file mode 100644 index 0000000..7602dac --- /dev/null +++ b/research/sqlite-rag-hybrid-search/hyde-paper-tables.md @@ -0,0 +1,28 @@ +# Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE paper PDF) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE paper PDF) | +| Source URL | http://boston.lti.cs.cmu.edu/luyug/HyDE/HyDE.pdf?ref=stepup.ai | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; the snippet tables show HyDE beating BM25 and Contriever on TREC DL19/DL20 without relevance judgments. Dense-retrieval wins that do not transfer to tai's lexical engine; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/ijcsit-web-ir-review.md b/research/sqlite-rag-hybrid-search/ijcsit-web-ir-review.md new file mode 100644 index 0000000..7c1537d --- /dev/null +++ b/research/sqlite-rag-hybrid-search/ijcsit-web-ir-review.md @@ -0,0 +1,28 @@ +# Web Based Information Retrieval (IJCSIT review) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Web Based Information Retrieval (IJCSIT review) | +| Source URL | https://www.researchpublish.com/upload/book/Web Based Information Retrieval-966.pdf | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the implicit-feedback search; the snippet summarizes the Joachims clickthrough findings second-hand. Redundant with the first-hand paper; logged for the trail. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/jev-tool-search-hyde-hurts-bm25.md b/research/sqlite-rag-hybrid-search/jev-tool-search-hyde-hurts-bm25.md new file mode 100644 index 0000000..d4e0f34 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/jev-tool-search-hyde-hurts-bm25.md @@ -0,0 +1,28 @@ +# LLM ranking theory notes: HyDE hurt BM25 tool retrieval + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | LLM ranking theory notes: HyDE hurt BM25 tool retrieval | +| Source URL | https://github.com/kachar/jev-tool-search/blob/HEAD/research/14-llm-ranking-theory.md | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; the snippet reports HyDE cutting BM25 nDCG@5 from 0.2635 to 0.1954 on tool retrieval while query rewriting helped. The most tai-shaped negative datapoint on HyDE-for-lexical; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/joachims-clickthrough-implicit-feedback.md b/research/sqlite-rag-hybrid-search/joachims-clickthrough-implicit-feedback.md new file mode 100644 index 0000000..523bf38 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/joachims-clickthrough-implicit-feedback.md @@ -0,0 +1,51 @@ +# Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search | +| Source URL | https://www.cs.cornell.edu/~tj/publications/joachims_etal_07a.pdf | +| Source publisher / author | Thorsten Joachims et al. (TOIS) | +| Source type | paper | +| Source date (published) | 2007-01-01 | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +The founding result on clickthrough data as relevance signal: against eyetracking and manual judgments, clicks are informative but biased, unusable as absolute relevance judgments, while relative preferences derived from clicks (this result over that one for the same query) are reasonably accurate on average, including across query reformulation chains. + +## Key findings + +- "We conclude that clicks are informative but biased." +- "While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average." +- Relative preferences hold "not only between results from an individual query, but across multiple sets of results within chains of query reformulations." +- Method: eyetracking of the users' decision process compared against manual relevance judgments. + +## Pros / strengths + +- Gives tai's already-collected `reads` counter plus search-then-read sequences a precise interpretation: pairwise preferences, not absolute labels. +- TOIS publication with an experimental (not just log-correlational) design; the finding has held for nearly two decades of learning-to-rank practice. + +## Cons / limitations / caveats + +- Web-search clicks with SERP position bias; tai's analog (agent reads a record after searching) has different bias structure (agent, not human, doing the clicking), unstudied here. +- Absolute judgments are explicitly unsupported, so implicit data can rerank and compare but never certify. + +## Relevance to tai + +Authorizes the cheapest eval signal available: log search queries with returned ids (a gap today: only results are logged), join to subsequent record reads, and derive pairwise preferences for tuning. No new data collection beyond one log line; the `reads` counter already exists. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sqlite-rag-hybrid-search/mdpi-rm3-hurts-short-queries.md b/research/sqlite-rag-hybrid-search/mdpi-rm3-hurts-short-queries.md new file mode 100644 index 0000000..c5f0810 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/mdpi-rm3-hurts-short-queries.md @@ -0,0 +1,28 @@ +# Multi-stage hybrid retrieval with cross-encoder reranking (RM3 hurts short queries) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Multi-stage hybrid retrieval with cross-encoder reranking (RM3 hurts short queries) | +| Source URL | https://WWW.MDPI.COM/2076-3417/16/10/4813 | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; the snippet reports plain BM25 beating BM25 plus RM3 on SciFact because expansion adds noise and drift on short claim queries. Direct caution against default-on pseudo-relevance feedback for tai's short-query workload. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/medium-hybrid-retrieval-hyde.md b/research/sqlite-rag-hybrid-search/medium-hybrid-retrieval-hyde.md new file mode 100644 index 0000000..47f1e58 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/medium-hybrid-retrieval-hyde.md @@ -0,0 +1,28 @@ +# Hybrid Retrieval with BM25, pgvector, embedding rerank, LLM Rubric Rerank and HyDE + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Hybrid Retrieval with BM25, pgvector, embedding rerank, LLM Rubric Rerank and HyDE | +| Source URL | https://medium.com/@richardhightower/stop-the-hallucinations-hybrid-retrieval-with-bm25-pgvector-embedding-rerank-llm-rubric-rerank-895d8f7c7242 | +| Source type | blog post | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; the snippet walks a full hybrid stack where HyDE feeds embedding retrieval, not BM25. Confirms HyDE belongs on the dense side of a hybrid system tai does not have; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/muenster-hyde-thesis.md b/research/sqlite-rag-hybrid-search/muenster-hyde-thesis.md new file mode 100644 index 0000000..afc0562 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/muenster-hyde-thesis.md @@ -0,0 +1,28 @@ +# Evaluation of Hypothetical Document Embeddings for IR (Muenster thesis) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Evaluation of Hypothetical Document Embeddings for IR (Muenster thesis) | +| Source URL | https://repositorium.uni-muenster.de/document/miami/3aa2edf8-f6a6-4d92-b21a-5ba1088d9599/ercis_master_theses_2024_01.pdf | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; the snippet reports HyDE beating BM25 and Contriever on many benchmarks without relevance labels. Pro-HyDE datapoint, but for dense retrieval, which tai vetoed; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/patent-bm25-albert-feedback.md b/research/sqlite-rag-hybrid-search/patent-bm25-albert-feedback.md new file mode 100644 index 0000000..ea9d651 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/patent-bm25-albert-feedback.md @@ -0,0 +1,28 @@ +# Pseudo-correlation feedback retrieval patent (BM25 plus ALBERT) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Pseudo-correlation feedback retrieval patent (BM25 plus ALBERT) | +| Source URL | https://patents.google.com/patent/CN111625624A/en | +| Source type | patent | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the query-expansion search; a patent combining BM25 with neural term semantics for expansion. Model-dependent approach with no bearing on tai's dependency-free stack; logged, not relied on. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/radlinski-joachims-query-chains.md b/research/sqlite-rag-hybrid-search/radlinski-joachims-query-chains.md new file mode 100644 index 0000000..b2ef7bc --- /dev/null +++ b/research/sqlite-rag-hybrid-search/radlinski-joachims-query-chains.md @@ -0,0 +1,28 @@ +# Query Chains: Learning to Rank from Implicit Feedback + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Query Chains: Learning to Rank from Implicit Feedback | +| Source URL | https://joachims.org/publications/radlinski_joachims_05a.pdf | +| Source type | paper | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the implicit-feedback search; the snippet covers clickthrough-as-relative-feedback and preferences across query reformulation chains. Directly relevant to mining tai's own search-then-read sequences; not opened first-hand. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/ragas-context-precision-docs.md b/research/sqlite-rag-hybrid-search/ragas-context-precision-docs.md new file mode 100644 index 0000000..c9002a8 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/ragas-context-precision-docs.md @@ -0,0 +1,50 @@ +# Context Precision - Ragas + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Context Precision - Ragas | +| Source URL | https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/ | +| Source publisher / author | Ragas (Exploding Gradients) | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | both | +| Confidence | High | +| Status | Active | + +## Summary + +Ragas's Context Precision metric scores whether relevant retrieved chunks rank above irrelevant ones, formalized as relevance-weighted mean precision@k. A sibling metric, Context Utilization, covers the no-reference-answer case by comparing contexts against the generated response instead. + +## Key findings + +- "Context Precision is a metric that evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones for a given query in the retrieved context." +- "It is calculated as the mean of the precision@k for each chunk in the context", with relevance indicators per rank and the standard precision@k ratio. +- "The ContextUtilization metric evaluates whether retrieved contexts are useful by comparing each context against the generated response. Use this when you don't have a reference answer but have the response that was generated." + +## Pros / strengths + +- Gives tai the exact rank-aware formula to reimplement natively (mean precision@k over relevance judgments), with the reference-free variant spelled out. +- Worked examples show the metric responding to rank order, not just set membership, which is what a tuning loop needs. + +## Cons / limitations / caveats + +- Relevance judgments still need a judge (LLM or human) per chunk; the formula is free, the labels are not. +- Package docs assume the Ragas runtime; tai reimplements the arithmetic, not the dependency. + +## Relevance to tai + +Supplies the rank-aware half of the standing eval's metric set: once goldens say which chunk answers a query, Context Precision (mean precision@k) measures ranking quality deterministically. Pairs with the fingerprint goldens, which supply the relevance labels for free. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sqlite-rag-hybrid-search/researchgate-clickthrough-quality.md b/research/sqlite-rag-hybrid-search/researchgate-clickthrough-quality.md new file mode 100644 index 0000000..af9631e --- /dev/null +++ b/research/sqlite-rag-hybrid-search/researchgate-clickthrough-quality.md @@ -0,0 +1,28 @@ +# How does clickthrough data reflect retrieval quality (ResearchGate record) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | How does clickthrough data reflect retrieval quality (ResearchGate record) | +| Source URL | https://www.researchgate.net/publication/200110409_How_does_clickthrough_data_reflect_retrieval_quality | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the implicit-feedback search; a record page for the Joachims clickthrough work whose abstract matches the first-hand dossier quotes. Pointer only; the paper itself was read. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/researchgate-clickthrough-user-model.md b/research/sqlite-rag-hybrid-search/researchgate-clickthrough-user-model.md new file mode 100644 index 0000000..af7fb67 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/researchgate-clickthrough-user-model.md @@ -0,0 +1,28 @@ +# Web search engine evaluation using clickthrough data and a user model (record) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Web search engine evaluation using clickthrough data and a user model (record) | +| Source URL | https://www.researchgate.net/publication/228924163_Web_search_engine_evaluation_using_clickthrough_data_and_a_user_model | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the implicit-feedback search; another record page for the same Joachims clickthrough findings. Pointer only; the paper itself was read. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/researchgate-ltr-clickthrough.md b/research/sqlite-rag-hybrid-search/researchgate-ltr-clickthrough.md new file mode 100644 index 0000000..6d78b4f --- /dev/null +++ b/research/sqlite-rag-hybrid-search/researchgate-ltr-clickthrough.md @@ -0,0 +1,28 @@ +# Improving Quality of Training Data for Learning to Rank Using Click-Through Data (record) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Improving Quality of Training Data for Learning to Rank Using Click-Through Data (record) | +| Source URL | https://www.researchgate.net/publication/221520036_Improving_Quality_of_Training_Data_for_Learning_to_Rank_Using_Click-Through_Data | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the implicit-feedback search; a record page restating the informative-but-biased clicks finding. Pointer only; the paper itself was read. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/sqlite-fts5-docs.md b/research/sqlite-rag-hybrid-search/sqlite-fts5-docs.md new file mode 100644 index 0000000..34879d3 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/sqlite-fts5-docs.md @@ -0,0 +1,51 @@ +# SQLite FTS5 Extension + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | SQLite FTS5 Extension | +| Source URL | https://www.sqlite.org/fts5.html | +| Source publisher / author | SQLite (Hipp Wyrick and Co.) | +| Source type | vendor docs | +| Source date (published) | Unknown — not stated (living documentation page) | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High | +| Status | Active | + +## Summary + +The authoritative FTS5 reference, read for the BM25 tuning surface: k1 and b are hard-coded (1.2, 0.75), scores are negated so ascending sort puts best first, and per-column weights passed as extra bm25() arguments are the only ranking knob. It also documents prefix queries, NEAR, and boolean operators tai does not currently use. + +## Key findings + +- "k1 and b are both constants, hard-coded at 1.2 and 0.75 respectively." +- "By passing extra real value arguments to the bm25() SQL function, each column of the table may be assigned a different weight", with phrase frequency computed as the weight-weighted sum across columns; missing trailing weights default to 1.0. +- FTS5 "multiplies the result by -1 before returning it, ensuring that better matches are assigned numerically lower scores", so plain ORDER BY bm25() ascending is correct (tai does this). +- Query syntax supports prefix queries, initial-token queries, NEAR, column filters, and boolean operators; tai uses only quoted-phrase AND across the title and body columns. + +## Pros / strengths + +- Primary specification of the exact ranking function tai runs; settles what is tunable (column weights, query shape) and what is not (k1, b). +- Negation note confirms tai's sort direction is right and needs no change. + +## Cons / limitations / caveats + +- Reference documentation, not tuning guidance; it gives no recommended weights or query-shape advice for small corpora. +- No discussion of evaluation; measurement guidance comes from the IR sources in this subject. + +## Relevance to tai + +Defines the entire dependency-free tuning backlog: title-over-body column weights first, then OR-fallback and prefix queries for the strict-AND recall gap. Everything else (k1/b, stemming beyond porter) is fixed by the engine, which bounds the roadmap and kills any proposal to "tune BM25 parameters" in FTS5. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Initial dossier created. | diff --git a/research/sqlite-rag-hybrid-search/uop-search-satisfaction-clicks.md b/research/sqlite-rag-hybrid-search/uop-search-satisfaction-clicks.md new file mode 100644 index 0000000..ff46478 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/uop-search-satisfaction-clicks.md @@ -0,0 +1,28 @@ +# Measuring and Predicting Search Engine Users Satisfaction (course paper) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | Measuring and Predicting Search Engine Users Satisfaction (course paper) | +| Source URL | http://old-eclass.uop.gr/modules/document/file.php/DIT104/%CE%92%CE%B9%CE%B2%CE%BB%CE%B9%CE%BF%CE%B3%CF%81%CE%B1%CF%86%CE%AF%CE%B1%202017-2018/Measuring%20and%20Predicting%20Search%20Engine%20Users%20Satisfaction.pdf | +| Source type | other | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the implicit-feedback search; the snippet cites Joachims 2005 on agreement between implicit clicks and explicit judgments plus Fox et al. on instrumented search. Second-hand corroboration with a good/abandonment framing; seen only as a snippet. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/research/sqlite-rag-hybrid-search/vespa-bm25-rank-feature.md b/research/sqlite-rag-hybrid-search/vespa-bm25-rank-feature.md new file mode 100644 index 0000000..1b00362 --- /dev/null +++ b/research/sqlite-rag-hybrid-search/vespa-bm25-rank-feature.md @@ -0,0 +1,28 @@ +# The BM25 rank feature - Vespa Documentation + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | sqlite-rag-hybrid-search | +| Document type | Source Citation Card | +| Tier | 2 | +| Source title | The BM25 rank feature - Vespa Documentation | +| Source URL | https://docs.vespa.ai/en/ranking/bm25.html | +| Source type | vendor docs | +| Date researched | 2026-10-18 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | Search aggregator result listing (not independently opened or quoted) | +| Confidence | Not independently verified | +| Status | Active | + +## Why it's here + +Surfaced by the industry gather worker; reported BM25 as a cheap first-phase ranker (3-4x faster than nativeRank) with tunable length-normalization b. Confirms BM25-first-then-rerank as standard and shows what tunability looks like where k1/b are not hardcoded, unlike FTS5. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-18 | Logged as consulted-but-not-analyzed. | diff --git a/tai.py b/tai.py old mode 100755 new mode 100644 index f12cb19..cdda0c7 --- a/tai.py +++ b/tai.py @@ -17,6 +17,7 @@ import http.client import io import itertools import json +import math import operator import os import queue @@ -101,10 +102,14 @@ def _fromisoformat(text): return parsed.replace(tzinfo=tzinfo) raise ValueError("invalid datetime, use ISO like 2026-10-08T09:00") -VERSION = "1.37.0" +VERSION = "1.38.0" WORKER_STEPS = 12 CREATE_SKILL_STEPS = 40 FORK_MAX_DEPTH = 2 +AGENTS_MAX_PARALLEL = 8 +AGENTS_MAX_RECORDS = 64 +SCHEDULER_MAX_FIRES = 4 +SCHEDULER_LEASE_MARGIN = 30 AGENTS = {} AGENTS_LOCK = threading.Lock() AGENTS_NEXT = [1] @@ -181,6 +186,16 @@ DEFAULT_VOICE = "en-US-EmmaMultilingualNeural" DEFAULT_PASSPHRASE = "tai-default-insecure-change-me" BOX_IMAGE = "tai-box:latest" BOX_NAME = "tai-box" +BOX_MEMORY = "2g" +BOX_CPUS = "2.0" +BOX_PIDS_LIMIT = "256" +BOX_RUN_ARGS = ( + "--memory", BOX_MEMORY, + "--cpus", BOX_CPUS, + "--pids-limit", BOX_PIDS_LIMIT, +) +BOX_UPDATE_ARGS = ("--memory", BOX_MEMORY, "--cpus", BOX_CPUS) +_BOX_CAPS_STAMP = {"done": False} TELEGRAM_API = "https://api.telegram.org/bot" TELEGRAM_FILE_API = "https://api.telegram.org/file/bot" SKILL_NAME_RE = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,62}[a-z0-9])?$") @@ -294,6 +309,28 @@ SHELL_TIMEOUT = 120 SCHEDULER_INTERVAL = 30 HTTP_TIMEOUT = 60 STREAM_TIMEOUT = 300 +APPROVAL_MAX_RETRIES = 3 +VERIFY_MAX_ROUNDS = 2 +VERIFY_TIMEOUT = 120 +VERIFY_STEPS = 6 +VERIFY_MAX_CHARS = 6000 +VERIFY_MAX_TRIED = 12 +VERIFY_MAX_LINE = 400 +VERIFY_MAX_FAILURES = 8 +VERIFY_MAX_GOAL = 2000 +VERIFY_MAX_ITEM = 200 +VERIFY_MAX_REASON = 500 +VERIFY_MAX_PROPOSAL = 500 +VERIFY_MAX_COMMENTARY = 1000 +FTS_TITLE_WEIGHT = 3.0 +FTS_BODY_WEIGHT = 1.0 +EVAL_K = 5 +EVAL_FETCH = 10 +EVAL_MAX_RECORDS = 50 +EVAL_MAX_GOLDENS = 200 +EVAL_MAX_PAIRS = 200 +EVAL_MIN_TOKEN = 6 +EVAL_PREFIX_MIN = 4 EDGE_HOST = "speech.platform.bing.com" EDGE_WS_PATH = "/consumer/speech/synthesize/readaloud/edge/v1" @@ -2063,6 +2100,8 @@ class Glow: who = head("name") ident = head("id") or head("target") or head("agent_id") when = head("at") or ("every %s" % head("every") if head("every") else "") + spec = args.get("spec") if isinstance(args.get("spec"), dict) else {} + flow = " ".join(str(spec.get("title") or "%d screens" % len(spec.get("screens") or [])).split())[:60] table = { "shell": ("Run", command, ""), "read_file": ("Read", path, ""), @@ -2087,6 +2126,7 @@ class Glow: "plexer_read": ("Read", ident, ""), "plexer_type": ("Type", ident, ""), "plexer_status": ("Plexer", "", ""), + "plexer_show": ("Show", flow, ""), "tui_screen": ("TUI", "screen", ""), "tui_keys": ("TUI", "keys", ""), "tui_type": ("TUI", "type", ""), @@ -2563,17 +2603,38 @@ def box_state(engine): return "running" if done.stdout.strip() == "true" else "stopped" +def update_box_caps(engine): + """Extend the current BOX_* memory/cpu ceiling to a box that already + exists (uncapped boxes predate the caps). Best-effort, once per process: + rootless cgroup writes can fail, and update cannot set pids-limit at all, + so failure points at reinstall instead of failing the turn. Returns True + when the update ran clean, False when skipped or it failed.""" + if _BOX_CAPS_STAMP["done"] or os.environ.get("TAI_BOX_CAPS") == "0": + return False + _BOX_CAPS_STAMP["done"] = True + try: + done = subprocess.run([engine, "update", *BOX_UPDATE_ARGS, BOX_NAME], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, timeout=60) + except (OSError, subprocess.SubprocessError): + return False + if done.returncode != 0: + print("box caps not applied (%s); run `%s rm -f %s` so next use recreates the box with limits" % ((done.stderr or "").strip()[:200], engine, BOX_NAME)) + return False + return True + + def ensure_box(): engine = container_engine() if not engine: raise BackendError("no container engine found, install podman") state = box_state(engine) if state == "running": + update_box_caps(engine) return engine if state == "stopped": done = subprocess.run([engine, "start", BOX_NAME], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, timeout=60) if done.returncode != 0: raise BackendError("cannot start box: " + (done.stderr or "").strip()[:200]) + update_box_caps(engine) return engine print("building sandbox image %s (one-time, takes minutes)..." % BOX_IMAGE) with tempfile.TemporaryDirectory() as build: @@ -2583,7 +2644,7 @@ def ensure_box(): done = subprocess.run([engine, "build", "-t", BOX_IMAGE, build], timeout=1800) if done.returncode != 0: raise BackendError("box image build failed") - done = subprocess.run([engine, "run", "-d", "--name", BOX_NAME, "--restart", "unless-stopped", BOX_IMAGE], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, timeout=120) + done = subprocess.run([engine, "run", "-d", "--name", BOX_NAME, "--restart", "unless-stopped", *BOX_RUN_ARGS, BOX_IMAGE], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, timeout=120) if done.returncode != 0: raise BackendError("cannot start box: " + (done.stderr or "").strip()[:200]) return engine @@ -3053,6 +3114,27 @@ def fts_match(terms): return " AND ".join("{title body} : %s" % term for term in terms) +def fts_match_or(terms): + return " OR ".join("{title body} : %s" % term for term in terms) + + +def fts_match_prefix(terms): + return " AND ".join("{title body} : %s*" % term.strip('"') for term in terms) + + +def fts_match_variants(terms): + """MATCH strings from strict to lenient: AND, prefix-AND, OR.""" + variants = [fts_match(terms), fts_match_prefix(terms)] + if len(terms) > 1: + variants.append(fts_match_or(terms)) + return variants + + +def fts_rank(table): + """Weighted bm25() over the fts_docs-shaped columns (title, body).""" + return "bm25(%s, 1.0, 1.0, 1.0, %s, %s)" % (table, FTS_TITLE_WEIGHT, FTS_BODY_WEIGHT) + + MEM_EVENTS_CAP = 10000 @@ -3351,32 +3433,37 @@ class Store: terms = fts_escape(query) if terms is None: return [] - sql = "SELECT item, kind, sub, title, snippet(fts_docs, 4, '[', ']', '...', 12), bm25(fts_docs) FROM fts_docs WHERE fts_docs MATCH ? AND profile = ?" - params = [fts_match(terms), profile] + capped = max(1, min(limit, 100)) + rank = fts_rank("fts_docs") + sql = "SELECT item, kind, sub, title, snippet(fts_docs, 4, '[', ']', '...', 12), %s FROM fts_docs WHERE fts_docs MATCH ? AND profile = ?" % rank + params_tail = [] if kinds: marks = ", ".join("?" * len(kinds)) sql += " AND kind IN (%s)" % marks - params += list(kinds) + params_tail += list(kinds) if self.seal.enabled: sql += " AND kind != 'event'" - sql += " ORDER BY bm25(fts_docs) LIMIT ?" - capped = max(1, min(limit, 100)) - params.append(capped) - try: - rows = self.db.execute(sql, tuple(params)).fetchall() - except sqlite3.Error: - return [] - hits = [{"item": row[0], "kind": row[1], "sub": row[2], "title": row[3], "snippet": row[4], "rank": row[5]} for row in rows] - if self.seal.enabled and self.memdb is not None and (not kinds or "event" in kinds): - self._sync_mem_events() + sql += " ORDER BY %s LIMIT ?" % rank + mem_rank = fts_rank("mem_events") + mem_sql = "SELECT item, kind, sub, title, snippet(mem_events, 4, '[', ']', '...', 12), %s FROM mem_events WHERE mem_events MATCH ? AND profile = ? ORDER BY %s LIMIT ?" % (mem_rank, mem_rank) + for match in fts_match_variants(terms): try: - mem_rows = self.memdb.execute("SELECT item, kind, sub, title, snippet(mem_events, 4, '[', ']', '...', 12), bm25(mem_events) FROM mem_events WHERE mem_events MATCH ? AND profile = ? ORDER BY bm25(mem_events) LIMIT ?", (fts_match(terms), profile, capped)).fetchall() + rows = self.db.execute(sql, tuple([match, profile] + params_tail + [capped])).fetchall() except sqlite3.Error: - mem_rows = [] - hits += [{"item": row[0], "kind": row[1], "sub": row[2], "title": row[3], "snippet": row[4], "rank": row[5]} for row in mem_rows] - hits.sort(key=lambda hit: hit["rank"]) - hits = hits[:capped] - return hits + rows = [] + hits = [{"item": row[0], "kind": row[1], "sub": row[2], "title": row[3], "snippet": row[4], "rank": row[5]} for row in rows] + if self.seal.enabled and self.memdb is not None and (not kinds or "event" in kinds): + self._sync_mem_events() + try: + mem_rows = self.memdb.execute(mem_sql, (match, profile, capped)).fetchall() + except sqlite3.Error: + mem_rows = [] + hits += [{"item": row[0], "kind": row[1], "sub": row[2], "title": row[3], "snippet": row[4], "rank": row[5]} for row in mem_rows] + hits.sort(key=lambda hit: hit["rank"]) + hits = hits[:capped] + if hits: + return hits + return [] def audit_search(self, query, profile=None, limit=20): profile = self._scope(profile) @@ -3670,22 +3757,27 @@ class Store: return [{"id": row[0], "kind": row[1], "title": row[2], "size": row[3], "reads": row[4], "updated": row[5], "tags": self.item_tags(row[0], profile)} for row in rows] def search_records_fts(self, terms, kind, clean, limit, profile): + rank = fts_rank("f") sql = "SELECT r.id, r.kind, r.title, r.size, r.reads, r.updated FROM records r JOIN fts_docs f ON f.item = r.id AND f.kind = 'record' WHERE r.profile = ? AND f MATCH ?" - params = [profile, fts_match(terms)] + params_tail = [] if kind: sql += " AND r.kind = ?" - params.append(kind) + params_tail.append(kind) for tag in clean: options = tag_variants(tag) marks = ", ".join("?" * len(options)) sql += " AND r.id IN (SELECT item FROM tags WHERE profile = ? AND tag IN (%s))" % marks - params += [profile] + options - sql += " ORDER BY bm25(f) LIMIT ?" - params.append(max(1, min(limit, 50))) - try: - rows = self.db.execute(sql, tuple(params)).fetchall() - except sqlite3.Error: - return [] + params_tail += [profile] + options + sql += " ORDER BY %s LIMIT ?" % rank + capped = max(1, min(limit, 50)) + rows = [] + for match in fts_match_variants(terms): + try: + rows = self.db.execute(sql, tuple([profile, match] + params_tail + [capped])).fetchall() + except sqlite3.Error: + rows = [] + if rows: + break return [{"id": row[0], "kind": row[1], "title": row[2], "size": row[3], "reads": row[4], "updated": row[5], "tags": self.item_tags(row[0], profile)} for row in rows] def delete_record(self, record_id, profile=None): @@ -6159,14 +6251,15 @@ TOOL_SCHEMAS = [ tool_schema("remember", "Update your own system message with new knowledge or behavior. The instruction is merged into the existing system message, unrelated parts stay intact. Use it to forget as well by instructing what to remove. Execute by default when behavior, preferences, or durable facts need updating. Never pass passwords, tokens, or secret values here; store those with store_secret instead.", {"instruction": {"type": "string"}}, ["instruction"]), tool_schema("recall", "Search past session memory of the current profile by keyword, optionally filtered by tags.", {"query": {"type": "string"}, "tags": {"type": "array"}}, ["query"]), tool_schema("load_skill", "Load a skill by name. Returns full instructions plus bundled file paths. Missing skills with a blueprint are researched and built on demand.", {"name": {"type": "string"}}, ["name"]), - tool_schema("get_current_terminal_content", "Capture visible text of the current tmux pane including scrollback, plus recent shell history. Pass target (a pane id like %3 or session:window.pane from list_terminals) to capture another open terminal instead. Works inside tmux or against a running tmux server.", {"lines": {"type": "integer"}, "target": {"type": "string"}}, []), - tool_schema("list_terminals", "List every open tmux terminal: all panes in all sessions with ids, current commands, and working directories, plus session/window/pane counts. Use a pane id as target for get_current_terminal_content. Needs a running tmux server.", {}, []), - tool_schema("tail_terminals", "One overview of all open tmux terminals: the last lines (default 3) of each at once, labeled with pane ids so you can spot which terminal holds what, then capture it fully with get_current_terminal_content. Needs a running tmux server.", {"lines": {"type": "integer"}}, []), + tool_schema("get_current_terminal_content", "Capture visible text of the user's current terminal pane including scrollback, plus recent shell history. With no target the pane resolves straight from the TMUX_PANE or TAI_PLEXER_PANE environment variable, no scan needed, so prefer this over list_terminals and tail_terminals whenever you need the current terminal. Pass target (a pane id like %3 or session:window.pane from list_terminals) to capture another open terminal instead. Works inside tmux, under tai's own multiplexer, or against a running tmux server.", {"lines": {"type": "integer"}, "target": {"type": "string"}}, []), + tool_schema("list_terminals", "List every open tmux terminal: all panes in all sessions with ids, current commands, and working directories, plus session/window/pane counts. Only for an overview of ALL terminals; for the user's current terminal call get_current_terminal_content with no target instead (it resolves via env). Use a pane id as target for get_current_terminal_content. Needs a running tmux server.", {}, []), + tool_schema("tail_terminals", "One overview of all open tmux terminals: the last lines (default 3) of each at once, labeled with pane ids so you can spot which terminal holds what, then capture it fully with get_current_terminal_content. Only for an overview of ALL terminals; for the user's current terminal call get_current_terminal_content with no target instead (it resolves via env). Needs a running tmux server.", {"lines": {"type": "integer"}}, []), tool_schema("export_terminal_log", "Only available inside a live tmux session. Exports the full pane scrollback (tmux capture-pane -pS -, the entire history, not just what's visible) to a log file under tai's home dir, then chunks it and indexes the chunks for retrieval with SQLite FTS5 + bm25 (lexical only, no embeddings). From then on, use the search tool (kind chunk) to answer questions about anything that happened in this terminal; it returns the actual chunk text, not just a snippet.", {"tags": {"type": "array"}}, []), tool_schema("plexer_send", "Only available when a tai-managed terminal session is active (either tai's own native wrapper, or an active tmux pane). Sends literal keystrokes into that session's shell, as if typed. Set enter to also press Enter and actually run it. No output is returned by this call; follow up with plexer_read.", {"text": {"type": "string"}, "enter": {"type": "boolean"}}, ["text"]), tool_schema("plexer_read", "Only available when a tai-managed terminal session is active. Reads back the last N lines the session's shell has produced (its actual rendered output, not a tool-call summary).", {"lines": {"type": "integer"}}, []), tool_schema("plexer_type", "Only available when a tai-managed terminal session is active. Types text into the session one character at a time with a delay between keystrokes, for a demo/animated-typing effect, then presses Enter unless enter is set to false.", {"text": {"type": "string"}, "delay": {"type": "number"}, "enter": {"type": "boolean"}}, ["text"]), tool_schema("plexer_status", "Only available when a tai-managed terminal session is active. Reports which backend is managing the current session (tai's own native pty wrapper, or tmux) and backend-specific details (pid/socket/log, or pane id).", {}, []), + tool_schema("plexer_show", "Only available when a tai-managed terminal session is active. Shows an interactive UI flow on the user's terminal and waits for the outcome: one spec with screens, each with body text, fields and actions. Field types: choice (options, single select), text (free input), info (display only). Fields and screens take show_if conditions on earlier answers ({field: expected}); next routes on ({action: screen}, a screen id, $done, $cancel). Example: {screens: [{id: main, body: Deploy?, fields: [{id: env, type: choice, options: [staging, prod]}], actions: [{id: ok, label: OK}], next: $done}]}. Returns {status: done/cancelled/timeout, answers: {field: value}, action}. Full contract: uiflow.md. Only for real blockers a human must decide; in autonomous mode and for background workers it is unavailable, so decide yourself and state the assumption.", {"spec": {"type": "object"}, "timeout": {"type": "integer"}}, ["spec"]), tool_schema("tui_screen", "Only available while a grr TUI (tai.py --tui) runs on this machine. Returns what its screen shows now (route, size, focus, the text). Set until to wait for that text to appear (timeout seconds), set async to get a job id at once and collect it with tui_result.", {"until": {"type": "string"}, "timeout": {"type": "integer"}, "async": {"type": "boolean"}}, []), tool_schema("tui_keys", "Only available while a grr TUI runs. Presses keys in it like a user: names such as down, up, enter, escape, space, tab, f1, ctrl+n, ctrl+p, or single characters. Async calls return a job id for tui_result.", {"keys": {"type": "array", "items": {"type": "string"}}, "async": {"type": "boolean"}}, ["keys"]), tool_schema("tui_type", "Only available while a grr TUI runs. Types text into the focused box (a newline presses Enter). Async calls return a job id for tui_result.", {"text": {"type": "string"}, "async": {"type": "boolean"}}, ["text"]), @@ -6216,6 +6309,7 @@ TOOL_ENV_GATES = { "plexer_read": lambda: PlexerControllerFactory.detect() is not None, "plexer_type": lambda: PlexerControllerFactory.detect() is not None, "plexer_status": lambda: PlexerControllerFactory.detect() is not None, + "plexer_show": lambda: PlexerControllerFactory.detect() is not None, "tui_screen": lambda: os.path.exists(grr_control_path()), "tui_keys": lambda: os.path.exists(grr_control_path()), "tui_type": lambda: os.path.exists(grr_control_path()), @@ -6250,6 +6344,7 @@ TOOL_TAGS = { "plexer_read": ("plexer", "terminal", "tmux", "pane", "read", "capture"), "plexer_type": ("plexer", "terminal", "tmux", "type", "animate", "keystroke", "demo"), "plexer_status": ("plexer", "terminal", "tmux", "status", "session"), + "plexer_show": ("plexer", "dialog", "flow", "menu", "form", "wizard", "screen", "ui"), "tui_screen": ("tui", "gui", "screen", "interface", "ui"), "tui_keys": ("tui", "gui", "keys", "press", "keyboard", "navigate"), "tui_type": ("tui", "gui", "type", "input", "text"), @@ -6412,6 +6507,26 @@ PLEXER_SOCKET_ENV = "TAI_PLEXER_SOCKET" PLEXER_ACTIVE_ENV = "TAI_PLEXER_ACTIVE" PLEXER_RECV_MAX = 1 << 20 +PLEXER_FLOW_VERSION = 1 +PLEXER_FLOW_SCREENS_MAX = 32 +PLEXER_FLOW_FIELDS_MAX = 12 +PLEXER_FLOW_OPTIONS_MAX = 12 +PLEXER_FLOW_ACTIONS_MAX = 12 +PLEXER_FLOW_VISITS_MAX = 64 +PLEXER_FLOW_ID_MAX = 64 +PLEXER_FLOW_TITLE_MAX = 200 +PLEXER_FLOW_TEXT_MAX = 2000 +PLEXER_FLOW_LABEL_MAX = 200 +PLEXER_FLOW_ANSWER_MAX = 2000 +PLEXER_FLOW_TIMEOUT_DEFAULT = 300 +PLEXER_FLOW_TIMEOUT_MIN = 5 +PLEXER_FLOW_TIMEOUT_MAX = 600 +PLEXER_FLOW_POLL = 0.25 +PLEXER_FLOW_TYPES = ("choice", "text", "info") +PLEXER_FLOW_KINDS = ("message", "choice", "text") +MPLEX_FLOW_MAX = 16 +MPLEX_FLOW_KEEP = 300 + def plexer_dir(home): path = os.path.join(home, PLEXER_DIR_NAME) @@ -6455,6 +6570,15 @@ class PlexerController: def status(self): raise NotImplementedError + def ask(self, screen, timeout): + """One blocking question on the user's terminal. screen is {"title", "text", + "kind" (message/choice/text), "options" ([labels] for choice), "initial" (for text)}. + Returns {"status": "ok", "value": ...} (choice: option index, text: string, + message: True), {"status": "cancelled"} or {"status": "timeout"}. Transport or + backend failures raise PlexerError. PlexerFlow decomposes a whole app spec into + ask calls; the backends stay dumb renderers of one screen at a time.""" + raise NotImplementedError + class NativePlexerController(PlexerController): kind = "native" @@ -6504,6 +6628,27 @@ class NativePlexerController(PlexerController): def status(self): return self._call({"cmd": "status"}) + def ask(self, screen, timeout): + handle = self._call({"cmd": "show", "screen": screen, "timeout": timeout}) + flow_id = handle["id"] + deadline = time.monotonic() + max(1, timeout) + while True: + reply = self._call({"cmd": "result", "id": flow_id}) + if reply.get("done"): + status = reply.get("status") + if status == "ok": + return {"status": "ok", "value": reply.get("value")} + if status in ("cancelled", "timeout"): + return {"status": status} + raise PlexerError("flow %s ended: %s" % (flow_id, status or "unknown")) + if time.monotonic() >= deadline: + try: + self._call({"cmd": "cancel", "id": flow_id}) + except PlexerError: + pass + return {"status": "timeout"} + time.sleep(PLEXER_FLOW_POLL) + class TmuxPlexerController(PlexerController): kind = "tmux" @@ -6539,6 +6684,124 @@ class TmuxPlexerController(PlexerController): info = self._tmux("list-panes", "-t", self.pane, "-F", "#{session_name}:#{window_index}.#{pane_index} #{pane_current_command}") return {"backend": "tmux", "pane": self.pane, "info": info.strip()} + def ask(self, screen, timeout): + kind = screen.get("kind") + if kind == "choice": + return self.ask_choice(screen, timeout) + if kind == "text": + return self.ask_text(screen, timeout) + if kind == "message": + return self.ask_message(screen, timeout) + raise PlexerError("unknown screen kind %r" % (kind,)) + + def ask_result_path(self): + handle, path = tempfile.mkstemp(prefix="tai-flow-", suffix=".answer") + os.close(handle) + os.unlink(path) + return path + + def await_answer(self, path, budget=1.0): + """run-shell answers land asynchronously; poll briefly, then give up.""" + deadline = time.monotonic() + max(0.0, budget) + while True: + try: + with open(path, "r", encoding="utf-8") as handle: + return handle.read() + except OSError: + pass + if time.monotonic() >= deadline: + return None + time.sleep(0.05) + + def ask_dismiss(self): + """Best-effort cleanup after a timeout: popups close server-side, but a + display-menu has no server-side close, it stays until the user presses + Escape (the flow is already over, so nothing follows it).""" + try: + subprocess.run(["tmux", "display-popup", "-C"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True, timeout=5) + except (OSError, subprocess.SubprocessError): + pass + + def ask_run(self, argv, path, timeout): + """Run one blocking tmux ask command. Returns the answer file content, or + None when the user dismissed it, or the string "timeout" on expiry.""" + try: + done = subprocess.run(["tmux"] + argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True, timeout=max(1, timeout)) + except subprocess.TimeoutExpired: + self.ask_dismiss() + return "timeout" + except (OSError, subprocess.SubprocessError) as exc: + raise PlexerError("tmux call failed: " + short_error(exc)) + if done.returncode != 0 and (done.stderr or "").strip(): + raise PlexerError("tmux %s failed: %s" % (argv[0], done.stderr.strip()[:300])) + if done.returncode != 0: + return None + return self.await_answer(path) + + def ask_choice(self, screen, timeout): + options = screen.get("options") or [] + path = self.ask_result_path() + try: + argv = ["display-menu", "-T", screen.get("title") or "tai"] + for index, label in enumerate(options): + key = "123456789abcdefghijklmnopqrstuvwxyz"[index] + argv += [label, key, "run-shell \"printf '%%s' '%d' > %s\"" % (index, shlex.quote(path))] + raw = self.ask_run(argv, path, timeout) + finally: + try: + os.unlink(path) + except OSError: + pass + if raw == "timeout": + return {"status": "timeout"} + if raw is None or not raw.strip().isdigit(): + return {"status": "cancelled"} + index = int(raw.strip()) + if 0 <= index < len(options): + return {"status": "ok", "value": index} + return {"status": "cancelled"} + + def ask_popup(self, title, body, reader, timeout): + """body shown, reader (a shell fragment reading one line, then writing + the answer file) decides ok versus dismissed. Returns file content/None/"timeout".""" + prompt_handle, prompt_path = tempfile.mkstemp(prefix="tai-flow-", suffix=".prompt") + try: + with os.fdopen(prompt_handle, "w", encoding="utf-8") as handle: + handle.write(body) + except OSError as exc: + raise PlexerError("cannot stage prompt text: " + short_error(exc)) + path = self.ask_result_path() + try: + command = "cat %s; %s" % (shlex.quote(prompt_path), reader % shlex.quote(path)) + raw = self.ask_run(["display-popup", "-w", "62", "-h", "10", "-T", title or "tai", "-E", command], path, timeout) + finally: + for staged in (prompt_path, path): + try: + os.unlink(staged) + except OSError: + pass + return raw + + def ask_text(self, screen, timeout): + body = "%s\n\n> " % (screen.get("text") or "") + raw = self.ask_popup(screen.get("title"), body, "read -r TAI_V && printf '%%s' \"$TAI_V\" > %s", timeout) + if raw == "timeout": + return {"status": "timeout"} + if raw is None: + return {"status": "cancelled"} + return {"status": "ok", "value": raw[:PLEXER_FLOW_ANSWER_MAX]} + + def ask_message(self, screen, timeout): + body = "%s\n\n[Enter] " % (screen.get("text") or "") + raw = self.ask_popup(screen.get("title"), body, "read -r _ && printf ok > %s", timeout) + if raw == "timeout": + return {"status": "timeout"} + if raw is None: + return {"status": "cancelled"} + return {"status": "ok", "value": True} + class PlexerControllerFactory: """The whole detection flow lives in one place: which env var is set @@ -6557,6 +6820,361 @@ class PlexerControllerFactory: return None +def validate_flow_id(raw, what): + """Screen, field and action ids: short, non-empty, no whitespace. Returns (value, error).""" + text = str(raw or "").strip() + if not text: + return "", "%s id must be a non-empty string" % what + if len(text) > PLEXER_FLOW_ID_MAX: + return "", "%s id %r is longer than %d chars" % (what, text[:72], PLEXER_FLOW_ID_MAX) + if any(char.isspace() for char in text): + return "", "%s id %r must not contain whitespace" % (what, text) + return text, "" + + +def validate_flow_text(raw, what, limit): + """Display strings: must be a string, capped. Returns (value, error).""" + if not isinstance(raw, str): + return "", "%s must be a string" % what + if len(raw) > limit: + return "", "%s is longer than %d chars" % (what, limit) + return raw, "" + + +def validate_flow_options(raw, what): + """Choice options: 1..12 strings or {"label", "value"} dicts with scalar + values. Returns (normalized [{"label", "value"}], error).""" + if not isinstance(raw, list) or not raw: + return [], "%s needs 1 to %d options" % (what, PLEXER_FLOW_OPTIONS_MAX) + if len(raw) > PLEXER_FLOW_OPTIONS_MAX: + return [], "%s has %d options, at most %d" % (what, len(raw), PLEXER_FLOW_OPTIONS_MAX) + options = [] + for pos, item in enumerate(raw): + if isinstance(item, str): + label, value = item, item + elif isinstance(item, dict): + label, value = item.get("label"), item.get("value", item.get("label")) + else: + return [], "%s option %d must be a string or {label, value}" % (what, pos + 1) + if not isinstance(label, str) or not label.strip(): + return [], "%s option %d needs a non-empty label" % (what, pos + 1) + if len(label) > PLEXER_FLOW_LABEL_MAX: + return [], "%s option %d label is longer than %d chars" % (what, pos + 1, PLEXER_FLOW_LABEL_MAX) + if not isinstance(value, (str, int, float, bool)) and value is not None: + return [], "%s option %d value must be a JSON scalar" % (what, pos + 1) + options.append({"label": label, "value": value}) + return options, "" + + +def validate_flow_condition(raw, what, known): + """show_if: {field id: expected scalar or list of scalars}. known holds every + choice/text field id in the app. Returns (normalized, error).""" + if raw is None: + return {}, "" + if not isinstance(raw, dict) or not raw: + return {}, "%s show_if must be a non-empty {field id: expected} object" % what + clauses = {} + for key, expected in raw.items(): + if key not in known: + return {}, "%s show_if refers to unknown field %r" % (what, key) + accepted = expected if isinstance(expected, list) else [expected] + if not accepted: + return {}, "%s show_if for %r needs at least one expected value" % (what, key) + for value in accepted: + if not isinstance(value, (str, int, float, bool)) and value is not None: + return {}, "%s show_if for %r must hold JSON scalars" % (what, key) + clauses[key] = accepted + return clauses, "" + + +def validate_flow_spec(spec): + """The plexer_show JSON contract (uiflow.md), version PLEXER_FLOW_VERSION. + Returns (normalized app, "") or (None, reason); every rejection is loud.""" + if not isinstance(spec, dict): + return None, "spec must be a JSON object" + version = spec.get("v", PLEXER_FLOW_VERSION) + if version != PLEXER_FLOW_VERSION: + return None, "spec v must be %d, got %r" % (PLEXER_FLOW_VERSION, version) + title, error = validate_flow_text(spec.get("title", ""), "title", PLEXER_FLOW_TITLE_MAX) + if error: + return None, error + try: + timeout = int(spec.get("timeout", PLEXER_FLOW_TIMEOUT_DEFAULT)) + except (TypeError, ValueError): + return None, "timeout must be an integer number of seconds" + if not PLEXER_FLOW_TIMEOUT_MIN <= timeout <= PLEXER_FLOW_TIMEOUT_MAX: + return None, "timeout must be %d to %d seconds" % (PLEXER_FLOW_TIMEOUT_MIN, PLEXER_FLOW_TIMEOUT_MAX) + raw_screens = spec.get("screens") + if not isinstance(raw_screens, list) or not raw_screens: + return None, "screens must be a non-empty list" + if len(raw_screens) > PLEXER_FLOW_SCREENS_MAX: + return None, "%d screens, at most %d" % (len(raw_screens), PLEXER_FLOW_SCREENS_MAX) + screens = [] + screen_ids = set() + field_ids = set() + for pos, item in enumerate(raw_screens): + what = "screen %d" % (pos + 1) + if not isinstance(item, dict): + return None, "%s must be an object" % what + ident, error = validate_flow_id(item.get("id"), what) + if error: + return None, error + if ident in screen_ids: + return None, "duplicate screen id %r" % ident + screen_ids.add(ident) + title_s, error = validate_flow_text(item.get("title", ""), "%s title" % ident, PLEXER_FLOW_TITLE_MAX) + if error: + return None, error + body, error = validate_flow_text(item.get("body", ""), "%s body" % ident, PLEXER_FLOW_TEXT_MAX) + if error: + return None, error + raw_fields = item.get("fields", []) + if not isinstance(raw_fields, list): + return None, "%s fields must be a list" % ident + if len(raw_fields) > PLEXER_FLOW_FIELDS_MAX: + return None, "%s has %d fields, at most %d" % (ident, len(raw_fields), PLEXER_FLOW_FIELDS_MAX) + fields = [] + for fpos, field in enumerate(raw_fields): + fwhat = "%s field %d" % (ident, fpos + 1) + if not isinstance(field, dict): + return None, "%s must be an object" % fwhat + kind = field.get("type") + if kind not in PLEXER_FLOW_TYPES: + return None, "%s type must be one of %s, got %r" % (fwhat, "/".join(PLEXER_FLOW_TYPES), kind) + fid, error = validate_flow_id(field.get("id"), fwhat) + if error: + return None, error + if fid in field_ids: + return None, "duplicate field id %r" % fid + field_ids.add(fid) + label, error = validate_flow_text(field.get("label", ""), "%s label" % fid, PLEXER_FLOW_LABEL_MAX) + if error: + return None, error + entry = {"id": fid, "type": kind, "label": label, "show_if": field.get("show_if")} + if kind == "choice": + options, error = validate_flow_options(field.get("options"), fid) + if error: + return None, error + entry["options"] = options + try: + default = int(field.get("default", 0)) + except (TypeError, ValueError): + return None, "%s default must be an option index" % fid + if not 0 <= default < len(options): + return None, "%s default %d is outside its %d options" % (fid, default, len(options)) + entry["default"] = default + elif kind == "text": + initial, error = validate_flow_text(field.get("initial", ""), "%s initial" % fid, PLEXER_FLOW_ANSWER_MAX) + if error: + return None, error + entry["initial"] = initial + else: + text, error = validate_flow_text(field.get("text", ""), "%s text" % fid, PLEXER_FLOW_TEXT_MAX) + if error: + return None, error + entry["text"] = text + fields.append(entry) + raw_actions = item.get("actions", []) + if not isinstance(raw_actions, list): + return None, "%s actions must be a list" % ident + if len(raw_actions) > PLEXER_FLOW_ACTIONS_MAX: + return None, "%s has %d actions, at most %d" % (ident, len(raw_actions), PLEXER_FLOW_ACTIONS_MAX) + actions = [] + action_ids = set() + for apos, action in enumerate(raw_actions): + awhat = "%s action %d" % (ident, apos + 1) + if not isinstance(action, dict): + return None, "%s must be an object" % awhat + aid, error = validate_flow_id(action.get("id"), awhat) + if error: + return None, error + if aid in action_ids: + return None, "%s has a duplicate action id %r" % (ident, aid) + action_ids.add(aid) + label, error = validate_flow_text(action.get("label", ""), "%s label" % aid, PLEXER_FLOW_LABEL_MAX) + if error: + return None, error + if not label.strip(): + return None, "%s needs a non-empty label" % aid + actions.append({"id": aid, "label": label}) + if not body.strip() and not fields and not actions: + return None, "%s has nothing to show (no body, fields or actions)" % ident + screens.append({"id": ident, "title": title_s, "body": body, "fields": fields, "actions": actions, + "next": item.get("next"), "show_if": item.get("show_if")}) + known_fields = set() + for screen in screens: + for field in screen["fields"]: + if field["type"] in ("choice", "text"): + known_fields.add(field["id"]) + for screen in screens: + for field in screen["fields"]: + clauses, error = validate_flow_condition(field.get("show_if"), field["id"], known_fields) + if error: + return None, error + field["show_if"] = clauses + clauses, error = validate_flow_condition(screen["show_if"], screen["id"], known_fields) + if error: + return None, error + screen["show_if"] = clauses + target = screen["next"] + if target is None: + screen["next"] = None + elif isinstance(target, str): + if target not in ("$done", "$cancel") and target not in screen_ids: + return None, "%s next refers to unknown screen %r" % (screen["id"], target) + screen["next"] = target + elif isinstance(target, dict): + if not screen["actions"]: + return None, "%s next is a map but the screen has no actions" % screen["id"] + for key, dest in target.items(): + if key not in [action["id"] for action in screen["actions"]]: + return None, "%s next maps unknown action %r" % (screen["id"], key) + if dest not in ("$done", "$cancel") and dest not in screen_ids: + return None, "%s next for %r refers to unknown screen %r" % (screen["id"], key, dest) + screen["next"] = dict(target) + else: + return None, "%s next must be a screen id or an {action: screen} map" % screen["id"] + start = spec.get("start", screens[0]["id"]) + if start not in screen_ids: + return None, "start refers to unknown screen %r" % (start,) + return {"title": title, "timeout": timeout, "start": start, "screens": screens}, "" + + +class PlexerFlow: + """The separate agent that runs one UI flow: the main agent hands over a whole + app spec in a single plexer_show call, this driver decomposes it into ask calls, + evaluates conditions and branches locally, and reports only the final outcome. + The main agent's turn stays lean no matter how complex the flow gets.""" + + MISSING = object() + + def __init__(self, app): + self.app = app + self.by_id = {screen["id"]: screen for screen in app["screens"]} + self.order = [screen["id"] for screen in app["screens"]] + self.answers = {} + self.action = None + self.visits = 0 + self.deadline = 0.0 + + @classmethod + def condition_holds(cls, clauses, answers): + for key, accepted in clauses.items(): + if answers.get(key, cls.MISSING) not in accepted: + return False + return True + + def follow_up(self, screen): + """Where a screen without a taken action goes: its string next, else the + next screen in order, else done.""" + target = screen["next"] + if isinstance(target, str): + return target + pos = self.order.index(screen["id"]) + if pos + 1 < len(self.order): + return self.order[pos + 1] + return "$done" + + def screen_text(self, screen, field=None): + """First ask of a screen carries the context: title, body, info fields.""" + if field is not None and field is not self.first_askable(screen): + return field["label"] or screen["title"] or self.app["title"] + parts = [screen["title"] or self.app["title"], screen["body"]] + parts += [item["text"] for item in screen["fields"] if item["type"] == "info" and self.condition_holds(item["show_if"], self.answers)] + if field is not None and field["label"]: + parts.append(field["label"]) + return "\n\n".join(part for part in parts if part and part.strip()) + + def first_askable(self, screen): + for field in screen["fields"]: + if field["type"] in ("choice", "text") and self.condition_holds(field["show_if"], self.answers): + return field + return None + + def remaining(self): + return self.deadline - time.monotonic() + + def run(self, controller): + self.deadline = time.monotonic() + self.app["timeout"] + current = self.app["start"] + while True: + self.visits += 1 + if self.visits > PLEXER_FLOW_VISITS_MAX: + return {"status": "error", "error": "flow passed %d screens (loop?), aborting" % PLEXER_FLOW_VISITS_MAX, + "at": current, "answers": dict(self.answers)} + screen = self.by_id[current] + if self.condition_holds(screen["show_if"], self.answers): + outcome = self.run_screen(controller, screen) + if isinstance(outcome, dict): + return outcome + target = outcome if isinstance(outcome, str) else self.follow_up(screen) + else: + target = self.follow_up(screen) + if target == "$done": + return {"status": "done", "answers": dict(self.answers), "action": self.action} + if target == "$cancel": + return {"status": "cancelled", "at": current, "answers": dict(self.answers)} + current = target + + def run_screen(self, controller, screen): + """Ask every visible field, then the action. Returns None to continue the + flow with follow_up, a screen id to jump there, or a final outcome dict + when the user cancelled, time ran out, or an action's next map ends it.""" + asked = False + for field in screen["fields"]: + if field["type"] == "info" or not self.condition_holds(field["show_if"], self.answers): + continue + if self.remaining() <= 0: + return {"status": "timeout", "at": screen["id"], "answers": dict(self.answers)} + if field["type"] == "choice": + reply = controller.ask({"title": screen["title"] or self.app["title"], "text": self.screen_text(screen, field), + "kind": "choice", "options": [opt["label"] for opt in field["options"]], + "default": field["default"]}, self.remaining()) + if reply["status"] != "ok": + return {"status": reply["status"], "at": screen["id"], "field": field["id"], "answers": dict(self.answers)} + self.answers[field["id"]] = field["options"][reply["value"]]["value"] + else: + reply = controller.ask({"title": screen["title"] or self.app["title"], "text": self.screen_text(screen, field), + "kind": "text", "initial": field["initial"]}, self.remaining()) + if reply["status"] != "ok": + return {"status": reply["status"], "at": screen["id"], "field": field["id"], "answers": dict(self.answers)} + self.answers[field["id"]] = reply["value"] + asked = True + if not screen["actions"]: + if not asked: + if self.remaining() <= 0: + return {"status": "timeout", "at": screen["id"], "answers": dict(self.answers)} + reply = controller.ask({"title": screen["title"] or self.app["title"], "text": self.screen_text(screen), + "kind": "message"}, self.remaining()) + if reply["status"] != "ok": + return {"status": reply["status"], "at": screen["id"], "answers": dict(self.answers)} + return None + if self.remaining() <= 0: + return {"status": "timeout", "at": screen["id"], "answers": dict(self.answers)} + if self.first_askable(screen) is None and not screen["body"].strip(): + text = screen["title"] or self.app["title"] + elif self.first_askable(screen) is None: + text = self.screen_text(screen) + else: + text = screen["title"] or self.app["title"] + reply = controller.ask({"title": screen["title"] or self.app["title"], "text": text, "kind": "choice", + "options": [action["label"] for action in screen["actions"]], "default": 0}, self.remaining()) + if reply["status"] != "ok": + return {"status": reply["status"], "at": screen["id"], "answers": dict(self.answers)} + self.action = screen["actions"][reply["value"]]["id"] + target = screen["next"] + if isinstance(target, dict): + dest = target.get(self.action) + if dest is None: + return None + if dest == "$done": + return {"status": "done", "answers": dict(self.answers), "action": self.action} + if dest == "$cancel": + return {"status": "cancelled", "at": screen["id"], "answers": dict(self.answers)} + return dest + return None + + # --------------------------------------------------------------------------- # # Backend side: PlexerBackend is the extension point the shell hook execs into. # NativePlexerBackend (bottom of the Mplex block) starts or joins the mplex @@ -9829,6 +10447,126 @@ class MplexPaneNumbers: canvas.put(pane.x + pane.w - len(size), area_y + pane.y, size, (None, bg, 0)) +class MplexFlowScreen: + """One plexer_show screen on a native client: a centered dialog with wrapped + text and, for kind choice, a list to move through. Answers land on the + server's flow record; the control socket collects them (result/cancel).""" + + def __init__(self, server, flow_id, title, text, options, cursor, until): + self.server = server + self.flow_id = flow_id + self.title = title + self.text = text + self.options = options + self.cursor = max(0, min(cursor, max(0, len(options) - 1))) + self.until = until + self.box = MplexStyle.parse("fg=white,bg=blue") + self.bold = MplexStyle.parse("fg=white,bg=blue,bold") + self.sel = MplexStyle.parse("fg=black,bg=white") + self.dim = MplexStyle.parse("fg=white,bg=blue,dim") + + def record(self): + return self.server.flows.get(self.flow_id) + + def answer(self, client, status, value=None): + found = self.record() + if found is not None and found["answer"] is None: + found["answer"] = {"status": status, "value": value} + found["settled"] = time.monotonic() + if client.overlay is self: + client.overlay = None + client.dirty = True + + def key(self, name, raw, client): + if self.record() is None: + if client.overlay is self: + client.overlay = None + client.dirty = True + return + if name in ("Escape", "C-c"): + self.answer(client, "cancelled") + return + if not self.options: + self.answer(client, "ok", True) + return + if name in ("Up", "k", "C-p"): + self.cursor = max(0, self.cursor - 1) + elif name in ("Down", "j", "C-n"): + self.cursor = min(len(self.options) - 1, self.cursor + 1) + elif name in ("Home", "g"): + self.cursor = 0 + elif name in ("End", "G"): + self.cursor = len(self.options) - 1 + elif name == "Enter": + self.answer(client, "ok", self.cursor) + return + elif name == "q": + self.answer(client, "cancelled") + return + elif len(name) == 1 and name in "1234567890": + index = int(name) - 1 if name != "0" else 9 + if index < len(self.options): + self.answer(client, "ok", index) + return + client.dirty = True + + def rows(self, width): + inner = max(8, width - 4) + lines = [] + for para in self.text.split("\n"): + lines.extend(textwrap.wrap(para, inner) or [""]) + return lines + + def draw(self, canvas, area_y): + width = max(20, min(canvas.cols - 4, 68)) + lines = self.rows(width) + options = list(self.options) + avail = canvas.rows - area_y - 3 + want = 7 + len(lines) + len(options) + while want > max(8, avail) and (len(lines) > 2 or len(options) > 1): + if len(options) > 1 and len(options) >= len(lines): + options.pop() + elif len(lines) > 2: + lines.pop() + else: + options.pop() + want = 7 + len(lines) + len(options) + height = min(want, max(8, avail)) + left = max(0, (canvas.cols - width) // 2) + top = area_y + max(0, (max(8, avail) - height) // 2) + glyphs = MplexCanvas.GLYPHS["single"] + horiz, vert, top_l, top_r, bot_l, bot_r = glyphs[0], glyphs[1], glyphs[2], glyphs[3], glyphs[4], glyphs[5] + canvas.fill(left, top, width, height, self.box) + canvas.put(left, top, top_l + horiz * (width - 2) + top_r, self.box) + canvas.put(left, top + height - 1, bot_l + horiz * (width - 2) + bot_r, self.box) + for row in range(1, height - 1): + canvas.put(left, top + row, vert, self.box) + canvas.put(left + width - 1, top + row, vert, self.box) + title = self.title[:width - 4] + canvas.put(left + (width - len(title)) // 2, top + 1, title, self.bold) + row = top + 3 + for line in lines: + if row >= top + height - 1: + break + canvas.put(left + 2, row, line[:width - 4], self.box) + row += 1 + row += 1 + for index, label in enumerate(options): + if row >= top + height - 1: + break + num = str(index + 1) if index < 9 else ("0" if index == 9 else " ") + entry = "%s %s %s" % (">" if index == self.cursor else " ", num, label) + canvas.put(left + 2, row, entry[:width - 4], self.sel if index == self.cursor else self.box) + row += 1 + if len(options) < len(self.options) and row < top + height - 1: + canvas.put(left + 2, row, "+%d more" % (len(self.options) - len(options)), self.dim) + row += 1 + hint = "Up/Down Enter 1-9 Esc" if self.options else "any key" + if row < top + height - 1: + row = top + height - 2 + canvas.put(left + (width - len(hint)) // 2, min(row, top + height - 2), hint, self.dim) + + class MplexChoice: __slots__ = ("label", "payload", "children", "expanded", "depth", "tag") @@ -10030,7 +10768,7 @@ class MplexRenderer: cursor = (pane.x + pane_cursor[0], area_y + pane.y + pane_cursor[1]) if window.zoomed is None and len(window.panes()) > 1: self.borders(canvas, window, area_y) - if isinstance(client.overlay, MplexPaneNumbers): + if isinstance(client.overlay, (MplexPaneNumbers, MplexFlowScreen)): client.overlay.draw(canvas, area_y) cursor = None status_row = 0 if on_top else client.rows - 1 @@ -10420,6 +11158,8 @@ class MplexServer: self.buffers = [] self.messages = collections.deque(maxlen=MPLEX_MESSAGES_MAX) self.prompt_history = {} + self.flows = {} + self.flow_seq = 0 self.timers = [] self.reaping = [] self.marked = None @@ -11244,8 +11984,155 @@ class MplexServer: window = pane.window return {"ok": True, "data": {"backend": "native", "pid": pane.pid, "log": pane.log_path, "socket": self.sock_path, "pane": pane.tag, "session": window.session.name, "window": window.session.index_of(window)}} + if cmd == "show": + return self.flow_show(request) + if cmd == "result": + return self.flow_result(request) + if cmd == "cancel": + return self.flow_cancel(request) return {"ok": False, "error": "unknown command %r" % cmd} + def flow_prune(self, now): + """Reclaim flow records whose driver died: answered ones settle, unanswered + ones clear their dialog first. Runs on every show and every tick.""" + for ident in list(self.flows): + found = self.flows[ident] + settled = found["settled"] if found["answer"] is not None else None + if settled is not None and now - settled > MPLEX_FLOW_KEEP: + del self.flows[ident] + elif settled is None and now > found["until"] + MPLEX_FLOW_KEEP: + self.flow_ui_clear(found) + del self.flows[ident] + + def flow_ui_clear(self, found): + client = found["client"] + if found.get("overlay") is not None and client.overlay is found["overlay"]: + client.overlay = None + if found.get("prompt") is not None and client.prompt is found["prompt"]: + client.prompt = None + client.dirty = True + + def flow_client(self, pane): + window = getattr(pane, "window", None) + session = window.session if window is not None else self.latest_session() + if session is None: + raise ValueError("no session") + clients = self.attached_clients(session) + if not clients: + raise ValueError("no attached client for this pane's session") + return clients[0] + + def flow_check_screen(self, raw): + """Validate one show screen straight off the socket; never trust it.""" + if not isinstance(raw, dict): + raise ValueError("screen must be an object") + kind = raw.get("kind") + if kind not in PLEXER_FLOW_KINDS: + raise ValueError("screen kind must be one of %s" % "/".join(PLEXER_FLOW_KINDS)) + title = raw.get("title", "") + text = raw.get("text", "") + if not isinstance(title, str) or not isinstance(text, str): + raise ValueError("screen title and text must be strings") + if len(title) > PLEXER_FLOW_TITLE_MAX or len(text) > PLEXER_FLOW_TEXT_MAX: + raise ValueError("screen title/text too long") + options = [] + if kind == "choice": + listed = raw.get("options") + if not isinstance(listed, list) or not listed or len(listed) > PLEXER_FLOW_OPTIONS_MAX: + raise ValueError("choice needs 1 to %d options" % PLEXER_FLOW_OPTIONS_MAX) + for label in listed: + if not isinstance(label, str) or not label.strip() or len(label) > PLEXER_FLOW_LABEL_MAX: + raise ValueError("choice options must be non-empty strings") + options.append(label) + try: + default = int(raw.get("default", 0)) + except (TypeError, ValueError): + raise ValueError("choice default must be an option index") + if not 0 <= default < len(options): + raise ValueError("choice default is outside its options") + else: + default = 0 + initial = raw.get("initial", "") + if not isinstance(initial, str) or len(initial) > PLEXER_FLOW_ANSWER_MAX: + raise ValueError("initial text too long") + return {"kind": kind, "title": title, "text": text, "options": options, "default": default, "initial": initial} + + def flow_show(self, request): + pane = self.control_pane(request) + screen = self.flow_check_screen(request.get("screen")) + try: + timeout = float(request.get("timeout", 60)) + except (TypeError, ValueError): + raise ValueError("timeout must be a number of seconds") + timeout = max(1.0, min(timeout, PLEXER_FLOW_TIMEOUT_MAX)) + now = time.monotonic() + self.flow_prune(now) + if len(self.flows) >= MPLEX_FLOW_MAX: + raise ValueError("too many pending flows (%d)" % MPLEX_FLOW_MAX) + client = self.flow_client(pane) + if client.overlay is not None or client.prompt is not None: + raise ValueError("client is busy (a prompt or overlay is already open)") + flow_id = self.flow_seq + self.flow_seq += 1 + found = {"id": flow_id, "client": client, "kind": screen["kind"], "until": now + timeout, + "started": now, "settled": None, "answer": None, "overlay": None, "prompt": None} + if screen["kind"] == "text": + label = ((screen["title"] + ": ") if screen["title"] else "") + screen["text"].replace("\n", " ")[:200] + " " + prompt = MplexPrompt(label, screen["initial"], + lambda value: self.flow_answered(flow_id, "ok", value[:PLEXER_FLOW_ANSWER_MAX]), + on_cancel=lambda: self.flow_answered(flow_id, "cancelled")) + found["prompt"] = prompt + client.prompt = prompt + else: + overlay = MplexFlowScreen(self, flow_id, screen["title"], screen["text"], screen["options"], + screen["default"], now + timeout) + found["overlay"] = overlay + client.overlay = overlay + self.flows[flow_id] = found + client.dirty = True + return {"ok": True, "data": {"id": flow_id}} + + def flow_answered(self, flow_id, status, value=None): + found = self.flows.get(flow_id) + if found is not None and found["answer"] is None: + found["answer"] = {"status": status, "value": value} + found["settled"] = time.monotonic() + + def flow_result(self, request): + try: + flow_id = int(request.get("id")) + except (TypeError, ValueError): + raise ValueError("result needs a flow id") + found = self.flows.get(flow_id) + if found is None: + raise ValueError("unknown flow %r" % (request.get("id"),)) + client = found["client"] + if client not in self.clients or not client.attached: + del self.flows[flow_id] + return {"ok": True, "data": {"done": True, "status": "detached"}} + if found["answer"] is not None: + done = found["answer"] + self.flow_ui_clear(found) + del self.flows[flow_id] + return {"ok": True, "data": {"done": True, "status": done["status"], "value": done["value"]}} + if time.monotonic() >= found["until"]: + self.flow_ui_clear(found) + del self.flows[flow_id] + return {"ok": True, "data": {"done": True, "status": "timeout"}} + return {"ok": True, "data": {"done": False}} + + def flow_cancel(self, request): + try: + flow_id = int(request.get("id")) + except (TypeError, ValueError): + raise ValueError("cancel needs a flow id") + found = self.flows.get(flow_id) + if found is None: + raise ValueError("unknown flow %r" % (request.get("id"),)) + self.flow_ui_clear(found) + del self.flows[flow_id] + return {"ok": True, "data": True} + def client_read(self, client): try: data = client.sock.recv(65536) @@ -11471,6 +12358,8 @@ class MplexServer: self.drop_pending(pending) def tick(self, now): + if self.flows: + self.flow_prune(now) for window in list(self.windows.values()): if window.auto_rename and window.active is not None and window.session is not None: template = window.options.get("automatic-rename-format") @@ -13400,6 +14289,7 @@ class Tools: "plexer_read": self.run_plexer_read, "plexer_type": self.run_plexer_type, "plexer_status": self.run_plexer_status, + "plexer_show": self.run_plexer_show, "tui_screen": lambda args: self.run_tui("screen", args), "tui_keys": lambda args: self.run_tui("key", args), "tui_type": lambda args: self.run_tui("type", args), @@ -13527,8 +14417,17 @@ class Tools: if self.app.env == "sandbox": return self.run_shell_box(command, str(args.get("workdir") or "/"), vault) workdir = str(args.get("workdir") or os.getcwd()) - if not self.app.config.auto_approve and not self.shell_is_safe(command) and not self.app.ask_approval(command): - return "denied by user" + if not self.app.config.auto_approve and not self.shell_is_safe(command): + try: + approved = self.app.ask_approval(command) + except Denied as denied: + if not Approval.is_affirmative(denied.guidance): + raise + approved = True + if not getattr(self.app, "quiet", False): + print(paint("taking %r as approval" % denied.guidance[:80], Ansi.YELLOW)) + if not approved: + return "denied by user" self.snapshot_shell_targets(command, workdir) env = dict(os.environ) for name, value in vault.items(): @@ -13996,9 +14895,14 @@ class Tools: except (TypeError, ValueError): lines = 100 target = str(args.get("target") or "").strip()[:128] + history = shell_history_section() + if not target: + socket_path = os.environ.get(PLEXER_SOCKET_ENV) + if socket_path: + return self.run_terminal_content_native(socket_path, lines, history) + target = os.environ.get("TMUX_PANE", "").strip()[:128] if not shutil.which("tmux"): return "tmux not available" - history = shell_history_section() if target: header = ["pane %s" % target] else: @@ -14017,6 +14921,18 @@ class Tools: return "terminal pane is empty" + history return "\n".join(header + [truncate(done.stdout, 6000, 2000)]) + history + def run_terminal_content_native(self, socket_path, lines, history): + """Current-pane capture under tai's own multiplexer: one socket read, no tmux, no scan.""" + try: + content = NativePlexerController(socket_path, os.environ.get(PLEXER_PANE_ENV)).read_pane(lines) + except PlexerError as exc: + return "terminal capture failed: " + str(exc) + history + if not (content or "").strip(): + return "terminal pane is empty" + history + pane = (os.environ.get(PLEXER_PANE_ENV) or "").strip() + header = "native pane %s" % pane if pane else "native plexer pane" + return "\n".join([header, truncate(content, 6000, 2000)]) + history + def run_list_terminals(self, args): panes, error = tmux_panes() if error: @@ -14158,6 +15074,35 @@ class Tools: return "error: " + str(exc) return json.dumps(data) + def run_plexer_show(self, args): + spec = args.get("spec") + if not isinstance(spec, dict): + return "error: spec must be a JSON object (the flow contract is uiflow.md)" + app, error = validate_flow_spec(spec) + if error: + return "error: invalid flow spec: " + error + if args.get("timeout") is not None: + try: + override = int(args.get("timeout")) + except (TypeError, ValueError): + return "error: timeout must be an integer number of seconds" + if not PLEXER_FLOW_TIMEOUT_MIN <= override <= PLEXER_FLOW_TIMEOUT_MAX: + return "error: timeout must be %d to %d seconds" % (PLEXER_FLOW_TIMEOUT_MIN, PLEXER_FLOW_TIMEOUT_MAX) + app["timeout"] = override + if getattr(self.app, "auto", False): + return "plexer_show unavailable in autonomous mode: decide yourself and state the assumption" + if not getattr(self.app, "persist", True) or getattr(self.app, "depth", 0) > 0: + return "plexer_show unavailable to background workers: decide yourself and state the assumption" + controller = PlexerControllerFactory.detect() + if controller is None: + return "error: no plexer session active (neither tai's native wrapper nor tmux)" + try: + outcome = PlexerFlow(app).run(controller) + except PlexerError as exc: + return "error: " + str(exc) + self.app.user_stepped_in = True + return json.dumps(outcome) + def run_fork(self, args): task = str(args.get("task") or "").strip() if not task: @@ -14174,7 +15119,10 @@ class Tools: if asked and asked != self.active_profile: return "error: cannot fork as another profile" profile = self.active_profile - agent_id = spawn_agent(task, profile, timeout, self.app.config, self.app.store.seal, self.app.depth + 1, self.app.runner_override) + try: + agent_id = spawn_agent(task, profile, timeout, self.app.config, self.app.store.seal, self.app.depth + 1, self.app.runner_override) + except AgentsFull as exc: + return "error: %s, poll existing agents first" % exc return "agent %d started: %s. Collect its result with poll." % (agent_id, task[:80]) def run_poll(self, args): @@ -14688,6 +15636,8 @@ class Tools: for ts, _role, kind, text in self.app.store.events_like(profile, query, limit): hits.append({"kind": "event", "item": "event %s" % ts, "sub": kind, "title": text[:60], "snippet": text[:160], "rank": 99.0}) hits.sort(key=lambda hit: hit["rank"]) + hit_ids = ",".join("%s:%s" % (hit["kind"], hit["item"]) for hit in hits[:limit]) + self.app.store.log_event(profile, "tool", "search-query", "q=%s\tlimit=%d\thits=%s" % (query[:200], limit, hit_ids[:1500])) if not hits: return "no matches for '%s'" % query[:80] chunk_ids = [] @@ -14751,6 +15701,7 @@ class Tools: page = self.app.store.read_record(record_id, offset, max(1, min(limit, 20000)), self.active_profile) if page is None: return "error: unknown record " + record_id + self.app.store.log_event(self.active_profile, "tool", "record-read", record_id[:200]) head = "%s chars %d-%d of %d :: %s" % (record_id, page["start"], page["end"], page["total"], self.app.store.node_title(record_id, self.active_profile)) if page["tags"]: head += " [%s]" % ", ".join(page["tags"]) @@ -15024,6 +15975,434 @@ class Denied(Exception): self.guidance = guidance +class AgentsFull(Exception): + pass + + +class Approval: + """Approval input rules (spec): what an allow answer means and which + guidance text counts as approval. + + parse_allow maps an allow answer to once, always, deny or unknown. + once is y or yes, always is Y, yolo or always, deny is n, no or empty. + Anything else is unknown and the prompt re-asks (APPROVAL_MAX_RETRIES), + so a typo like A never silently denies. + + is_affirmative matches only exact standalone phrases after lowercasing + and collapsing whitespace and trailing .!? marks. Anything longer + (yes, but do X) is guidance, never approval. Only the shell tool + treats affirmative guidance as approval; deletes, secrets, schedules + and releases stay strict and keep raising Denied.""" + + AFFIRMATIVE = frozenset(( + "y", "yes", "yeah", "yep", "sure", + "proceed", "go ahead", "do it", "run it", "allow", "approve", "approved", + "ok", "okay", "fine", "good", "great", "perfect", + "all good", "looks good", "sounds good", + "it was perfect", "that was perfect", "everything was fine", "everything is fine", + )) + + @staticmethod + def parse_allow(answer): + cleaned = (answer or "").strip() + lowered = cleaned.lower() + if cleaned == "Y" or lowered in ("yolo", "always"): + return "always" + if lowered in ("y", "yes"): + return "once" + if lowered in ("n", "no", ""): + return "deny" + return "unknown" + + @staticmethod + def is_affirmative(text): + words = " ".join((text or "").strip().lower().split()).rstrip(".!?") + return bool(words) and words in Approval.AFFIRMATIVE + + +class GoalFailure: + """One unmet goal item: what is wrong, why, and what to do next.""" + + def __init__(self, item, reason, proposal): + self.item = item + self.reason = reason + self.proposal = proposal + + @classmethod + def from_dict(cls, raw): + if not isinstance(raw, dict): + return None + try: + item = str(raw.get("item") or "").strip() + reason = str(raw.get("reason") or "").strip() + proposal = str(raw.get("proposal") or "").strip() + except Exception: + return None + if not item or not reason or not proposal: + return None + return cls(item[:VERIFY_MAX_ITEM], reason[:VERIFY_MAX_REASON], proposal[:VERIFY_MAX_PROPOSAL]) + + def to_dict(self): + return {"item": self.item, "reason": self.reason, "proposal": self.proposal} + + +class GoalVerdict: + """Parsed verifier answer: ok plus the failure list and commentary.""" + + def __init__(self, ok, failures, commentary, raw): + self.ok = ok + self.failures = failures + self.commentary = commentary + self.raw = raw + + @classmethod + def from_text(cls, text): + raw = (text or "").strip() + data, error = GoalVerifier.extract_json(raw) + failures = [] + broken = [] + success = None + if error is not None: + broken.append("verifier returned no usable JSON: %s" % error) + elif not isinstance(data, dict): + broken.append("verifier returned a JSON value that is not an object") + else: + success = data.get("success") + entries = data.get("failures") + if not isinstance(success, bool): + broken.append("field success is not true or false") + if not isinstance(entries, list): + broken.append("field failures is not a list") + else: + for pos, entry in enumerate(entries[:VERIFY_MAX_FAILURES]): + parsed = GoalFailure.from_dict(entry) + if parsed is None: + broken.append("failure %d misses item, reason or proposal" % (pos + 1)) + else: + failures.append(parsed) + if success is True and failures: + broken.append("verifier claims success but lists failures") + ok = not broken and success is True + for note in broken: + failures.append(GoalFailure("goal verification", note, "re-run the request so it can be verified")) + if not ok and not failures: + failures.append(GoalFailure("goal", "verifier reported failure without details", "re-run the request with explicit checks")) + failures = failures[:VERIFY_MAX_FAILURES] + commentary = "" + if isinstance(data, dict) and isinstance(data.get("commentary"), str): + commentary = data["commentary"].strip()[:VERIFY_MAX_COMMENTARY] + return cls(ok, failures, commentary, raw[:VERIFY_MAX_COMMENTARY]) + + def report(self, goal): + title = " ".join(str(goal or "").split())[:200] or "goal" + if self.ok: + note = (" " + self.commentary) if self.commentary else "" + return "goal verified: %s%s" % (title, note) + lines = ["goal NOT verified: %s" % title] + for failure in self.failures: + lines.append("- %s: %s" % (failure.item, failure.reason)) + lines.append(" next: %s" % failure.proposal) + if self.commentary: + lines.append(self.commentary) + return "\n".join(lines) + + def followup(self): + lines = ["Goal verification FAILED, the goal is not delivered yet. Continue now with the next proposal below."] + for failure in self.failures: + lines.append("- %s: %s Next: %s" % (failure.item, failure.reason, failure.proposal)) + if self.commentary: + lines.append(self.commentary) + lines.append("Do not repeat denied, skipped or failed actions verbatim; use the proposals and move forward.") + return "\n".join(lines)[:VERIFY_MAX_CHARS] + + +class GoalVerifier: + """Strict one-shot acceptance check (spec). + + After a one-shot turn the goal (the prompt text) is verified by a + worker subagent before the process may report success. The worker + gets goal, final answer, transcript tail and tried actions, and must + reply with exactly one JSON object: a success bool plus a failures + list of {item, reason, proposal} dicts and an optional commentary. + + Strictness: only evidence counts, praise is not evidence, denied or + skipped calls mean the effect did not happen, success with failures + listed is failure, and every failure needs a proposal that moves + forward instead of repeating tried actions. + + Flow (Agent.run_verified): verify, and while the verdict fails run + one follow-up turn fed with the failures, at most VERIFY_MAX_ROUNDS + extra turns. Verified exits 0; exhausted rounds or a user abort + exits 1, and any unverified outcome leaves the turn state open so + --resume can continue. Workers (depth above 0) never verify, so + verification + cannot recurse. Caps: VERIFY_TIMEOUT per check, VERIFY_STEPS worker + steps, transcript and field clamps from the VERIFY_MAX_* constants.""" + + TASK = ( + "You are a strict acceptance verifier. The main agent was asked to achieve a GOAL. " + "Decide only from evidence whether the goal is actually delivered. Be very strict: claims " + "without evidence are failures. User praise (perfect, great, fine) is not evidence. A denied, " + "skipped or timed out tool call means its effect did not happen. Partial work is failure unless " + "the goal explicitly allows partial work. You may use read-only tools to inspect machine state; " + "judge from the transcript when a check is unavailable.\n" + "\nGOAL:\n%s\n" + "\nFINAL ANSWER of the main agent:\n%s\n" + "\nTRANSCRIPT (oldest first, tail truncated):\n%s\n" + "\nALREADY TRIED (never propose these again verbatim, move forward):\n%s\n" + "\nReply with exactly one JSON object and nothing else:\n" + '{"success": true or false, "failures": [{"item": "...", "reason": "...", "proposal": "..."}], "commentary": "..."}\n' + "Rules: success is true only when every part of the goal is delivered with evidence. " + "When success is false, list every unmet item (at most %d) with its reason and one concrete " + "next proposal that differs from ALREADY TRIED." + ) + + @staticmethod + def extract_json(text): + body = (text or "").strip() + if body.startswith("```"): + body = body[3:] + if body[:4].lower() == "json": + body = body[4:] + end = body.find("```") + body = body[:end] if end >= 0 else body + body = body.strip() + start = body.find("{") + end = body.rfind("}") + if start < 0 or end <= start: + return None, "no JSON object found" + try: + return json.loads(body[start:end + 1]), None + except ValueError as exc: + return None, "invalid JSON: %s" % exc + + @staticmethod + def transcript(messages): + lines = [] + for item in messages or []: + role = item.get("role") or "?" + if role == "system": + continue + text = item.get("content") or "" + if role == "assistant": + names = [] + for call in item.get("tool_calls") or []: + fn = call.get("function") or {} + names.append("%s %s" % (fn.get("name", "?"), str(fn.get("arguments") or "")[:80])) + if names: + text = ("%s [tools: %s]" % (text, "; ".join(names))).strip() + lines.append(("[%s] %s" % (role, " ".join(str(text).split())))[:VERIFY_MAX_LINE]) + joined = "\n".join(lines) + return joined[-VERIFY_MAX_CHARS:] if len(joined) > VERIFY_MAX_CHARS else joined + + @staticmethod + def tried_actions(messages): + tried = [] + for item in messages or []: + for call in item.get("tool_calls") or []: + fn = call.get("function") or {} + args = " ".join(str(fn.get("arguments") or "").split())[:80] + tried.append(("%s %s" % (fn.get("name") or "?", args)).strip()) + return tried[-VERIFY_MAX_TRIED:] + + @staticmethod + def build_task(goal, final, transcript, tried): + goal_text = " ".join(str(goal or "").split())[:VERIFY_MAX_GOAL] or "(empty goal)" + final_text = " ".join(str(final or "").split())[:VERIFY_MAX_COMMENTARY] or "(empty answer)" + transcript_text = transcript or "(no transcript)" + tried_text = "\n".join(tried) if tried else "(none)" + return GoalVerifier.TASK % (goal_text, final_text, transcript_text, tried_text, VERIFY_MAX_FAILURES) + + +class RetrievalEval: + """Standing retrieval-quality eval (spec). Three layers, cheapest first. + + Goldens: query-plus-expected-id pairs generated deterministically from + vault records (exact term, title phrase, truncated prefix, phrase plus + one foreign term), no LLM needed. Metrics over Store.fts_search ranks: + Hit@K, MRR, binary nDCG@K, and mean precision@k (Context Precision). + Implicit pairs: search-query events joined to later record-read events + give pairwise preferences (read beats shown-but-unread), never absolute + labels. Zero-hit queries are listed as known recall failures. + Caps: EVAL_K scoring depth, EVAL_FETCH rows retrieved, EVAL_MAX_RECORDS + sampled, EVAL_MAX_GOLDENS and EVAL_MAX_PAIRS kept.""" + + @staticmethod + def rank_of(expected_id, ranked_ids): + try: + return list(ranked_ids).index(expected_id) + 1 + except ValueError: + return None + + @staticmethod + def best_rank(expected_ids, ranked_ids): + ranks = [RetrievalEval.rank_of(item, ranked_ids) for item in expected_ids or []] + ranks = [rank for rank in ranks if rank is not None] + return min(ranks) if ranks else None + + @staticmethod + def hit_at_k(rank, k=EVAL_K): + return 1.0 if rank is not None and rank <= k else 0.0 + + @staticmethod + def reciprocal_rank(rank): + return 1.0 / rank if rank else 0.0 + + @staticmethod + def ndcg_at_k(rank, k=EVAL_K): + if rank is None or rank > k: + return 0.0 + return 1.0 / math.log2(rank + 1) + + @staticmethod + def context_precision(expected_ids, ranked_ids, k=EVAL_K): + wanted = set(expected_ids or []) + if not wanted: + return 0.0 + hits = 0 + total = 0.0 + for pos, item in enumerate(list(ranked_ids)[:k], 1): + if item in wanted: + hits += 1 + total += hits / pos + return total / hits if hits else 0.0 + + @staticmethod + def tokens(text, min_len=EVAL_MIN_TOKEN): + seen = [] + for token in re.findall(r"[a-z0-9]+", (text or "").lower()): + if len(token) >= min_len and token not in seen: + seen.append(token) + return seen + + @staticmethod + def _doc_freq(store, profile, token): + try: + row = store.db.execute("SELECT COUNT(DISTINCT item) FROM fts_docs WHERE fts_docs MATCH ? AND profile = ? AND kind = 'record'", ('{title body} : "%s"' % token, profile,)).fetchone() + except sqlite3.Error: + return 0 + return row[0] if row else 0 + + @staticmethod + def goldens_from_vault(store, profile, max_records=EVAL_MAX_RECORDS): + rows = store.db.execute("SELECT id, title, content FROM records WHERE profile = ? ORDER BY id LIMIT ?", (profile, max_records,)).fetchall() + goldens = [] + for pos, row in enumerate(rows): + record_id, title, content = row[0], row[1] or "", row[2] or "" + words = RetrievalEval.tokens(title + "\n" + content) + if not words: + continue + ranked = sorted(((RetrievalEval._doc_freq(store, profile, token), -len(token), idx, token) for idx, token in enumerate(words[:24]))) + rare = [token for _, _, _, token in ranked] + goldens.append({"query": rare[0], "expected": [record_id], "category": "exact", "source": record_id}) + if len(rare) >= 2: + goldens.append({"query": " ".join(rare[:3]), "expected": [record_id], "category": "phrase", "source": record_id}) + longest = max(words, key=len) + cut = max(EVAL_PREFIX_MIN, len(longest) // 2) + if cut < len(longest): + goldens.append({"query": longest[:cut], "expected": [record_id], "category": "prefix", "source": record_id}) + if len(rare) >= 2 and len(rows) > 1: + other = rows[(pos + 1) % len(rows)] + other_words = RetrievalEval.tokens((other[1] or "") + "\n" + (other[2] or "")) + own = set(words) + foreign = [token for token in other_words if token not in own] + if foreign: + goldens.append({"query": " ".join(rare[:2] + [foreign[0] + "zz"]), "expected": [record_id], "category": "noisy", "source": record_id}) + if len(goldens) >= EVAL_MAX_GOLDENS: + break + return goldens[:EVAL_MAX_GOLDENS] + + @staticmethod + def score(store, goldens, profile, kinds=("record",), k=EVAL_K, fetch=EVAL_FETCH): + rows = [] + for golden in goldens or []: + try: + hits = store.fts_search(golden.get("query") or "", kinds, profile, fetch) + except Exception: + hits = [] + ranked = [hit["item"] for hit in hits] + expected = golden.get("expected") or [] + rank = RetrievalEval.best_rank(expected, ranked) + rows.append({"query": golden.get("query"), "category": golden.get("category"), "expected": expected, "rank": rank, + "hit": RetrievalEval.hit_at_k(rank, k), "mrr": RetrievalEval.reciprocal_rank(rank), + "ndcg": RetrievalEval.ndcg_at_k(rank, k), + "cp": RetrievalEval.context_precision(expected, ranked, k)}) + result = {"k": k, "goldens": len(rows), "overall": RetrievalEval._mean(rows), "by_category": {}, "misses": []} + for row in rows: + result["by_category"].setdefault(row["category"], []).append(row) + for category in result["by_category"]: + result["by_category"][category] = {"goldens": len(result["by_category"][category]), "mean": RetrievalEval._mean(result["by_category"][category])} + result["misses"] = [{"query": row["query"], "category": row["category"]} for row in rows if row["rank"] is None][:10] + return result + + @staticmethod + def _mean(rows): + if not rows: + return {"hit": 0.0, "mrr": 0.0, "ndcg": 0.0, "cp": 0.0} + return {key: round(sum(row[key] for row in rows) / len(rows), 4) for key in ("hit", "mrr", "ndcg", "cp")} + + @staticmethod + def implicit_pairs(store, profile, limit=500): + text_col = "tai_dec(text)" if store.seal.enabled else "text" + try: + rows = store.db.execute("SELECT role, kind, %s FROM events WHERE profile = ? ORDER BY id DESC LIMIT ?" % text_col, (profile, limit,)).fetchall() + except sqlite3.Error: + rows = [] + rows.reverse() + searches = [] + for pos, row in enumerate(rows): + if row[0] == "tool" and row[1] == "search-query": + parsed = RetrievalEval._parse_search_event(row[2] or "") + if parsed is not None: + searches.append((pos, parsed[0], parsed[1])) + pairs = [] + zero_hit = [] + read_follow = 0 + for idx, (pos, query, hits) in enumerate(searches): + if not hits: + zero_hit.append(query) + end = searches[idx + 1][0] if idx + 1 < len(searches) else len(rows) + for row in rows[pos + 1:min(end, pos + 21)]: + if row[0] == "tool" and row[1] == "record-read" and (row[2] or "").strip(): + read_follow += 1 + read_id = (row[2] or "").strip() + rank = RetrievalEval.rank_of(read_id, hits) + pairs.append({"query": query, "read": read_id, "rank": rank, "shown": len(hits)}) + break + if len(pairs) >= EVAL_MAX_PAIRS: + break + return {"searches": len(searches), "read_follow": read_follow, "pairs": pairs[:EVAL_MAX_PAIRS], "zero_hit": zero_hit[:10], "zero_hit_count": len(zero_hit)} + + @staticmethod + def _parse_search_event(text): + try: + parts = dict(part.split("=", 1) for part in str(text).split("\t") if "=" in part) + query = (parts.get("q") or "").strip() + raw = (parts.get("hits") or "").strip() + hits = [piece.split(":", 1)[1] if piece.startswith("record:") else piece for piece in raw.split(",") if piece] + if not query: + return None + return query, hits + except Exception: + return None + + @staticmethod + def format_report(result, pairs): + lines = ["retrieval eval: %d goldens at k=%d" % (result.get("goldens", 0), result.get("k", EVAL_K))] + overall = result.get("overall") or {} + lines.append("overall: hit=%.3f mrr=%.3f ndcg=%.3f cp=%.3f" % (overall.get("hit", 0.0), overall.get("mrr", 0.0), overall.get("ndcg", 0.0), overall.get("cp", 0.0))) + for category in sorted((result.get("by_category") or {}).keys()): + entry = result["by_category"][category] + mean = entry.get("mean") or {} + lines.append("%s: n=%d hit=%.3f mrr=%.3f ndcg=%.3f cp=%.3f" % (category, entry.get("goldens", 0), mean.get("hit", 0.0), mean.get("mrr", 0.0), mean.get("ndcg", 0.0), mean.get("cp", 0.0))) + for miss in result.get("misses") or []: + lines.append("miss [%s]: %s" % (miss.get("category"), (miss.get("query") or "")[:80])) + lines.append("implicit: %d searches, %d followed by a read, %d zero-hit" % (pairs.get("searches", 0), pairs.get("read_follow", 0), pairs.get("zero_hit_count", 0))) + for query in pairs.get("zero_hit") or []: + lines.append("zero-hit: %s" % query[:80]) + return "\n".join(lines) + + class Agent: def __init__(self, config, store, persist=True, quiet=False, depth=0, auto=False, yolo=False, ctx=None): self.config = config @@ -15157,17 +16536,23 @@ class Agent: if not sys.stdin.isatty(): return False print(paint("run: %s" % command, Ansi.YELLOW)) - try: - answer = input(paint("allow? [y]once [Y]always [n]o ", Ansi.YELLOW)).strip() - except (EOFError, KeyboardInterrupt): - return False + decision = "unknown" + for _retry in range(APPROVAL_MAX_RETRIES): + try: + answer = input(paint("allow? [y]once [Y]always [n]o ", Ansi.YELLOW)).strip() + except (EOFError, KeyboardInterrupt): + return False + decision = Approval.parse_allow(answer) + if decision != "unknown": + break + print(paint("unrecognized answer %r, use y (once), Y (always) or n (deny)" % answer, Ansi.YELLOW)) self.user_stepped_in = True - if answer == "Y" or answer.lower() in ("yolo", "always"): + if decision == "always": self.yolo = True self.config.auto_approve = True print(paint("yolo mode on, approvals off for this session", Ansi.YELLOW)) return True - if answer.lower() in ("y", "yes"): + if decision == "once": return True if not guidance or self.auto: return False @@ -15511,6 +16896,71 @@ class Agent: print(paint("tokens≈%d · %s · %.1fs" % (conversation_tokens(self.messages), "+".join(backends), total), Ansi.DIM)) return last_text + def run_verified(self, prompt): + """One-shot turn with strict subagent acceptance, see the + GoalVerifier spec. Returns (final_text, verified). Workers never + verify, and a user abort never continues.""" + if self.depth != 0: + return self.run_turn(prompt), True + return self.verify_loop(prompt, self.run_turn(prompt)) + + def ensure_verified(self, goal, final): + """Verify a finished one-shot turn against its goal, running + follow-up turns until the verdict passes or VERIFY_MAX_ROUNDS is + spent. Returns only whether the goal was delivered.""" + return self.verify_loop(goal, final)[1] + + def verify_loop(self, goal, final): + goal = " ".join(str(goal or "").split())[:VERIFY_MAX_GOAL] + if final == "stopped by user": + self.mark_turn_open(goal) + return final, False + for _attempt in range(1 + VERIFY_MAX_ROUNDS): + verdict = self.verify_goal(goal, final) + self.show_verdict(verdict, goal) + self.store.log_event(self.profile, "system", "verify", "ok" if verdict.ok else "failed: %d items" % len(verdict.failures)) + if verdict.ok: + return final, True + if _attempt >= VERIFY_MAX_ROUNDS: + break + final = self.run_turn(verdict.followup()) + if final == "stopped by user": + self.mark_turn_open(goal) + return final, False + self.mark_turn_open(goal) + return final, False + + def mark_turn_open(self, goal): + if self.persist: + self.store.save_turn_state(self.profile, self.bot, {"open": True, "goal": goal[:500], "steps": self._turn_steps, "tools": self._turn_tools, "updated": now_iso()}, ctx=self.ctx_id) + + def verify_goal(self, goal, final): + """Run one strict acceptance check in a worker subagent.""" + task = GoalVerifier.build_task(goal, final, GoalVerifier.transcript(self.messages), GoalVerifier.tried_actions(self.messages)) + runner = self.runner_override + if runner is None: + def runner(task, profile, timeout): + return default_runner(self.config, self.store.seal, profile, self.depth + 1, task, timeout, max_steps=VERIFY_STEPS) + try: + agent_id = spawn_agent(task, self.profile, VERIFY_TIMEOUT, self.config, self.store.seal, self.depth + 1, runner) + except Exception as exc: + return GoalVerdict(False, [GoalFailure("goal verification", "verifier did not start: %s" % short_error(exc), "re-run the request")], "", "") + status, text = poll_agent(agent_id, wait=VERIFY_TIMEOUT, profile=self.profile) + if status in ("done", "timeout", "error"): + with AGENTS_LOCK: + if agent_id in AGENTS: + AGENTS[agent_id]["collected"] = True + if status != "done": + return GoalVerdict(False, [GoalFailure("goal verification", "verifier %s: %s" % (status, " ".join(text.split())[:200]), "re-run the verification")], "", text[:VERIFY_MAX_COMMENTARY]) + with AGENTS_LOCK: + full = (AGENTS.get(agent_id) or {}).get("result") or text + return GoalVerdict.from_text(full) + + def show_verdict(self, verdict, goal): + if self.quiet: + return + print(paint(verdict.report(goal), Ansi.GREEN if verdict.ok else Ansi.YELLOW)) + def run_mention(self, target, rest, original, capture=False, max_steps=MAX_STEPS): if self.persist: self.store.save_session(self.profile, self.messages, self.bot, ctx=self.ctx_id) @@ -15558,6 +17008,13 @@ def default_runner(config, seal, profile, depth, task, timeout, max_steps=WORKER def spawn_agent(task, profile, timeout, config, seal, depth=0, runner=None): with AGENTS_LOCK: + if len(AGENTS) >= AGENTS_MAX_RECORDS: + old = sorted(key for key, record in AGENTS.items() if record["status"] != "running" and record.get("collected")) + for key in old[:len(AGENTS) - AGENTS_MAX_RECORDS + 1]: + del AGENTS[key] + live = sum(1 for record in AGENTS.values() if record["status"] == "running") + if live >= AGENTS_MAX_PARALLEL: + raise AgentsFull("too many live agents (%d of %d)" % (live, AGENTS_MAX_PARALLEL)) agent_id = AGENTS_NEXT[0] AGENTS_NEXT[0] += 1 record = {"id": agent_id, "task": task, "profile": profile, "status": "running", "result": "", "started": time.time(), "ended": None, "thread": None, "spilled": None, "collected": False, "tools": None} @@ -15701,19 +17158,40 @@ def note_schedule_result(db_path, row_id, status, result=""): def watch_schedule(db_path, row_id, agent_id, timeout): - status, text = poll_agent(agent_id, wait=timeout + 30) + status, text = poll_agent(agent_id, wait=timeout + SCHEDULER_LEASE_MARGIN) note_schedule_result(db_path, row_id, status, text) +def schedule_running(last_status, updated, timeout): + """Lease check: a running claim younger than timeout plus margin may live.""" + if last_status != "running": + return False + try: + age = (datetime.now(timezone.utc) - _fromisoformat(updated)).total_seconds() + except ValueError: + return False + return age < (timeout or 600) + SCHEDULER_LEASE_MARGIN + + def scheduler_tick(config, seal, runner=None): fired = 0 + skipped_cap = 0 + skipped_overlap = 0 now = datetime.now(timezone.utc) db = sqlite3.connect(config.db_path, timeout=30) db.execute("PRAGMA journal_mode=WAL") db.execute("PRAGMA busy_timeout=30000") try: - rows = db.execute("SELECT id, prompt, profile, every_sec, next_run, timeout FROM schedules WHERE status = 'pending' AND next_run <= ? ORDER BY next_run", (now.isoformat(),)).fetchall() - for row_id, prompt, profile, every_sec, next_run, timeout in rows: + rows = db.execute("SELECT id, prompt, profile, every_sec, next_run, timeout, last_status, updated FROM schedules WHERE status = 'pending' AND next_run <= ? ORDER BY next_run", (now.isoformat(),)).fetchall() + for row_id, prompt, profile, every_sec, next_run, timeout, last_status, updated in rows: + if fired >= SCHEDULER_MAX_FIRES: + break + if every_sec and schedule_running(last_status, updated, timeout): + skipped_overlap += 1 + continue + if running_agents() >= AGENTS_MAX_PARALLEL: + skipped_cap += 1 + continue stamp = datetime.now(timezone.utc).isoformat() if every_sec: try: @@ -15722,7 +17200,7 @@ def scheduler_tick(config, seal, runner=None): continue while claimed_next <= datetime.now(timezone.utc): claimed_next += timedelta(seconds=every_sec) - done = db.execute("UPDATE schedules SET next_run = ?, updated = ? WHERE id = ? AND status = 'pending' AND next_run = ?", (claimed_next.isoformat(), stamp, row_id, next_run)) + done = db.execute("UPDATE schedules SET next_run = ?, updated = ?, last_status = 'running' WHERE id = ? AND status = 'pending' AND next_run = ?", (claimed_next.isoformat(), stamp, row_id, next_run)) else: done = db.execute("UPDATE schedules SET status = 'done', updated = ? WHERE id = ? AND status = 'pending'", (stamp, row_id)) db.commit() @@ -15734,12 +17212,23 @@ def scheduler_tick(config, seal, runner=None): except SealError as exc: note_schedule_result(config.db_path, row_id, "error: " + short_error(exc)) continue - agent_id = spawn_agent(prompt, profile, timeout, config, seal, 0, runner) + try: + agent_id = spawn_agent(prompt, profile, timeout, config, seal, 0, runner) + except AgentsFull: + if every_sec: + db.execute("UPDATE schedules SET next_run = ?, updated = ?, last_status = ? WHERE id = ?", (next_run, stamp, last_status or "", row_id)) + else: + db.execute("UPDATE schedules SET status = 'pending', updated = ? WHERE id = ?", (stamp, row_id)) + db.commit() + skipped_cap += 1 + continue watcher = threading.Thread(target=watch_schedule, args=(config.db_path, row_id, agent_id, timeout), daemon=True) watcher.start() fired += 1 finally: db.close() + if skipped_cap or skipped_overlap: + print("scheduler: skipped %d at parallel cap, %d overlapping" % (skipped_cap, skipped_overlap)) return fired @@ -23582,6 +25071,9 @@ GRR_BANNER_SECS = 5.2 GRR_TOAST_MAX = 4 GRR_COMPOSER_MAX_ROWS = 6 GRR_VIEWER_MAX_BYTES = 2 << 20 +GRR_PROMPT_QUIET = 0.8 # seconds of silence after a prompt-like last row before the command is offered for typing +GRR_PROMPT_ASKS = 20 # times one process card opens its terminal by itself +GRR_PROMPT_MAX = 8 # pending approvals asked at once as dialogs in one chat GRR_DRAFT_KEEP = 64 GRR_LIST_MORE = 12 GRR_MENU_MAX_H = 14 @@ -28768,6 +30260,10 @@ class GrrProcessItem(GrrControlsItem): self.screen = TuiVtScreen(self.ROWS_TALL, 80) self.stream = None self.fed = False + self.gen = 0 + self.asked_gen = -1 + self.asks = 0 + self.typing = False self.rebuild_controls() self.sync() @@ -28821,9 +30317,31 @@ class GrrProcessItem(GrrControlsItem): pass self.fed = True self.touch() + self.gen += 1 + gen = self.gen + self.view.g.app.after(GRR_PROMPT_QUIET, lambda: self.check_prompt(gen)) elif name == "exit": self.stop_stream() + PROMPT_TAIL = re.compile(r"(\?|:|[\[(][yYnN](es)?/[yYnN](o)?[\])]|password[^\n]*|passphrase[^\n]*)\s*$", re.I) + + def check_prompt(self, gen): + """A command that went quiet on a prompt-like last row (sudo, ssh, a + y/n) is waiting for the user: its terminal opens at once, so the + prompt reaches the client instead of sitting unseen on the card.""" + if gen != self.gen or gen == self.asked_gen or not self.running or self.typing or self.asks >= GRR_PROMPT_ASKS or self.view.g.view is not self.view: + return + screen = self.screen + used = [r for r in range(screen.rows) if any(c is not None and c[0] != " " for c in screen.grid[r])] + if not used: + return + row = "".join((c[0] if c is not None else " ") for c in screen.grid[used[-1]]).rstrip() + if not self.PROMPT_TAIL.search(row): + return + self.asked_gen = gen + self.asks += 1 + self.type_in() + def stop_stream(self): if self.stream is not None: self.stream.stop() @@ -28851,8 +30369,9 @@ class GrrProcessItem(GrrControlsItem): pid = self.pid() term = GrrRemoteTerminal(g, "/api/processes/%s/stream?from=0" % pid, "/api/processes/%s/input" % pid, w=min(g.app.cols - 6, 100), h=min(g.app.rows - 8, 28), border=True, style=g.ui.st("ink", "sunken"), border_style=g.ui.st("accent", "bg")) + self.typing = True dlg = g.dialog(title=grr_text(self.m.get("content")), text="Esc leaves, Ctrl+C is sent to the command.", body=term, w=min(g.app.cols - 2, 104), actions=[{"label": g.t("ui.done"), "value": "done", "primary": True}], - done=lambda _v, _t: term.detach()) + done=lambda _v, _t: (setattr(self, "typing", False), term.detach())) term.on_exit = lambda: g.close_dialog(dlg, "done") term.attach() g.app.set_focus(term) @@ -28934,6 +30453,7 @@ class GrrChatView(GrrView): self.has_more = False self.busy = False self.new_below = False + self.prompts = {} self.spin = 0 self.pane = None self.root = TuiCols(w="1fr", h="100%", style=ui.st("ink", "bg")) @@ -29002,6 +30522,9 @@ class GrrChatView(GrrView): def destroy(self): g = self.g + for dlg in list(self.prompts.values()): + g.close_dialog(dlg, None, quiet=True) + self.prompts = {} if self.chat is not None: text = self.box.value.strip() if text: @@ -29226,6 +30749,29 @@ class GrrChatView(GrrView): self.mark_last() self.entries = entries self.log.set_items(entries) + for m in self.messages: + self.prompt_approval(m) + + def prompt_approval(self, m): + """A pending approval is asked at once as a dialog on the loop, never + left as a card to find; the answer is the card's own decision call. + Esc leaves the card as it was, a resolved card closes the dialog.""" + meta = m.get("meta") or {} + mid = m.get("id") + if m.get("kind") != "approval" or meta.get("state") != "pending" or mid in self.prompts or len(self.prompts) >= GRR_PROMPT_MAX: + return + g, t = self.g, self.g.t + bot = self.bot_of(m.get("sender")) or (self.bots()[0] if self.bots() else {"name": "Bot"}) + + def done(value, _texts): + self.prompts.pop(mid, None) + if value: + g.call("POST", "/api/messages/%s/decision" % mid, {"decision": value}) + actions = [{"label": t("chat.deny"), "value": "deny"}] + if meta.get("tool") == "browser": + actions.append({"label": t("chat.always_allow"), "value": "always"}) + actions.append({"label": t("chat.allow_once"), "value": "allow", "primary": True}) + self.prompts[mid] = g.dialog(title=t("chat.needs_approval", name=bot["name"]), text="\n".join(grr_wrap(grr_text(m.get("content")), 60)[:12]), actions=actions, done=done) self.log.scroll_to_end() def empty_item(self): @@ -29283,6 +30829,7 @@ class GrrChatView(GrrView): self.items[m["id"]] = item self.log.append(item) self.mark_last() + self.prompt_approval(m) if m.get("role") == "user" or self.log._follow: self.log.scroll_to_end() else: @@ -29300,6 +30847,11 @@ class GrrChatView(GrrView): else: return item = self.items.get(m["id"]) + dlg = self.prompts.pop(m["id"], None) + if dlg is not None and (m.get("meta") or {}).get("state") != "pending": + self.g.close_dialog(dlg, None, quiet=True) + elif dlg is not None: + self.prompts[m["id"]] = dlg if isinstance(item, GrrProcessItem): item.update(m) return @@ -30407,6 +31959,7 @@ def build_parser(): parser.add_argument("--env", default=None) parser.add_argument("--no-opencode", action="store_true", help="skip the opencode CLI backends for this run (same as TAI_OPENCODE=0)") parser.add_argument("--update-models", action="store_true", help="rediscover the free model lists of every provider, drop models that are gone, and exit") + parser.add_argument("--eval-retrieval", action="store_true", help="score vault retrieval (Hit@K, MRR, nDCG, Context Precision) on generated goldens plus implicit pairs, print the report, and exit") parser.add_argument("--version", action="store_true") parser.add_argument("--telegram", action="store_true") parser.add_argument("--scheduler", action="store_true") @@ -30580,6 +32133,15 @@ def main(argv=None): print("sandbox unavailable: %s" % exc, file=sys.stderr) return 2 agent.env = args.env + if args.eval_retrieval: + try: + goldens = RetrievalEval.goldens_from_vault(agent.store, agent.profile) + result = RetrievalEval.score(agent.store, goldens, agent.profile) + pairs = RetrievalEval.implicit_pairs(agent.store, agent.profile) + print(RetrievalEval.format_report(result, pairs)) + finally: + agent.store.close() + return 0 if args.scheduler: print("scheduler online, press Ctrl-C to stop") try: @@ -30591,9 +32153,15 @@ def main(argv=None): return 0 if args.telegram: return run_telegram_bot(agent) + if args.tui_local: + try: + return tui_chat(agent, theme_name) + finally: + agent.store.close() if args.prompt: try: - agent.run_turn(" ".join(args.prompt)) + prompt_text = " ".join(args.prompt) + verified = agent.ensure_verified(prompt_text, agent.run_turn(prompt_text)) except KeyboardInterrupt: print(paint("\ninterrupted", Ansi.YELLOW)) return 130 @@ -30606,12 +32174,7 @@ def main(argv=None): agent.store.close() except Exception: pass - return 0 - if args.tui_local: - try: - return tui_chat(agent, theme_name) - finally: - agent.store.close() + return 0 if verified else 1 if args.resume and not args.telegram and not args.scheduler: resume_turn(agent) banner(agent) diff --git a/test_tai.py b/test_tai.py index ae70085..afce22b 100644 --- a/test_tai.py +++ b/test_tai.py @@ -98,6 +98,8 @@ class ToolTests(unittest.TestCase): tools = tai.Tools(FakeApp()) with tempfile.TemporaryDirectory() as empty: with mock.patch.dict(os.environ, {"PATH": empty}): + os.environ.pop("TMUX_PANE", None) + os.environ.pop(tai.PLEXER_SOCKET_ENV, None) self.assertEqual(tools.dispatch("get_current_terminal_content", "{}"), "tmux not available") def test_box_helpers_present(self): @@ -118,9 +120,15 @@ class TerminalContextTests(unittest.TestCase): self.paths.start() self.which = mock.patch.object(tai.shutil, "which", return_value="/usr/bin/tmux") self.which.start() + self.env = mock.patch.dict(os.environ) + self.env.start() + os.environ.pop("TMUX_PANE", None) + os.environ.pop(tai.PLEXER_SOCKET_ENV, None) + os.environ.pop(tai.PLEXER_PANE_ENV, None) self.calls = [] def tearDown(self): + self.env.stop() self.which.stop() self.paths.stop() self.tmp.cleanup() @@ -210,6 +218,47 @@ class TerminalContextTests(unittest.TestCase): self.assertTrue(result.startswith("terminal capture failed")) self.assertIn("git push", result) + def test_content_prefers_tmux_pane_env(self): + tools = tai.Tools(FakeApp()) + os.environ["TMUX_PANE"] = "%7" + with self.tmux(captures={"%7": "env pane output"}): + result = tools.dispatch("get_current_terminal_content", "{}") + self.assertIn("env pane output", result) + self.assertIn("pane %7", result) + self.assertTrue(any("-t" in call and "%7" in call for call in self.calls)) + self.assertFalse(any(call[1:2] == ["display-message"] for call in self.calls)) + + def test_content_explicit_target_beats_env(self): + tools = tai.Tools(FakeApp()) + os.environ["TMUX_PANE"] = "%7" + with self.tmux(captures={"%5": "other pane output"}): + result = tools.dispatch("get_current_terminal_content", json.dumps({"target": "%5"})) + self.assertIn("other pane output", result) + self.assertIn("pane %5", result) + self.assertFalse(any("%7" in call for call in self.calls)) + + def test_content_native_backend_needs_no_tmux(self): + tools = tai.Tools(FakeApp()) + os.environ[tai.PLEXER_SOCKET_ENV] = "/tmp/fake-plexer.sock" + os.environ[tai.PLEXER_PANE_ENV] = "3" + with mock.patch.object(tai.shutil, "which", return_value=None): + with mock.patch.object(tai.NativePlexerController, "read_pane", return_value="native output") as reader: + with self.tmux(): + result = tools.dispatch("get_current_terminal_content", "{}") + self.assertIn("native output", result) + self.assertIn("native pane 3", result) + self.assertNotIn("tmux not available", result) + self.assertEqual(reader.call_count, 1) + self.assertEqual(self.calls, []) + + def test_content_native_backend_failure_loud(self): + tools = tai.Tools(FakeApp()) + os.environ[tai.PLEXER_SOCKET_ENV] = "/tmp/fake-plexer.sock" + with mock.patch.object(tai.NativePlexerController, "read_pane", side_effect=tai.PlexerError("refused")): + result = tools.dispatch("get_current_terminal_content", "{}") + self.assertTrue(result.startswith("terminal capture failed")) + self.assertIn("git push", result) + def test_list_terminals_counts_and_marks_active(self): tools = tai.Tools(FakeApp()) panes = (self.pane("main", 0, 0, "%1", True, "bash", "/home/u"), @@ -2131,6 +2180,66 @@ class InstallOpsTests(unittest.TestCase): self.assertIn("use /install", printed) +class BoxCapsTests(unittest.TestCase): + def setUp(self): + tai._BOX_CAPS_STAMP["done"] = False + os.environ.pop("TAI_BOX_CAPS", None) + + def tearDown(self): + tai._BOX_CAPS_STAMP["done"] = False + os.environ.pop("TAI_BOX_CAPS", None) + + def _ok(self, stdout=""): + done = mock.Mock() + done.returncode = 0 + done.stdout = stdout + done.stderr = "" + return done + + def test_run_carries_cap_trio(self): + with mock.patch.object(tai, "container_engine", return_value="podman"): + with mock.patch.object(tai, "box_state", return_value="missing"): + with mock.patch("subprocess.run", return_value=self._ok()) as runner: + self.assertEqual(tai.ensure_box(), "podman") + run_call = [call for call in runner.call_args_list if call.args[0][:3] == ["podman", "run", "-d"]] + self.assertEqual(len(run_call), 1) + argv = run_call[0].args[0] + for flag, value in (("--memory", "2g"), ("--cpus", "2.0"), ("--pids-limit", "256")): + self.assertIn(flag, argv) + self.assertEqual(argv[argv.index(flag) + 1], value) + + def test_update_applies_memory_cpu_once(self): + with mock.patch("subprocess.run", return_value=self._ok()) as runner: + self.assertTrue(tai.update_box_caps("podman")) + self.assertFalse(tai.update_box_caps("podman")) + self.assertEqual(runner.call_count, 1) + self.assertEqual(runner.call_args.args[0], ["podman", "update", "--memory", "2g", "--cpus", "2.0", tai.BOX_NAME]) + + def test_update_failure_points_at_recreate(self): + failed = self._ok() + failed.returncode = 1 + failed.stderr = "cannot update" + with mock.patch("subprocess.run", return_value=failed): + with mock.patch("builtins.print") as shown: + self.assertFalse(tai.update_box_caps("docker")) + printed = "\n".join(str(call.args[0]) for call in shown.call_args_list) + self.assertIn("docker rm -f %s" % tai.BOX_NAME, printed) + + def test_ensure_box_running_applies_caps(self): + with mock.patch.object(tai, "container_engine", return_value="podman"): + with mock.patch.object(tai, "box_state", return_value="running"): + with mock.patch("subprocess.run", return_value=self._ok()) as runner: + self.assertEqual(tai.ensure_box(), "podman") + self.assertEqual(runner.call_count, 1) + self.assertEqual(runner.call_args.args[0][1], "update") + + def test_env_suppressor_skips_update(self): + os.environ["TAI_BOX_CAPS"] = "0" + with mock.patch("subprocess.run") as runner: + self.assertFalse(tai.update_box_caps("podman")) + runner.assert_not_called() + + class BotsTests(unittest.TestCase): def setUp(self): self.tmp = tempfile.TemporaryDirectory() @@ -3878,6 +3987,440 @@ class DenialTests(unittest.TestCase): self.assertIn("[redacted:api]", shown) +class ApprovalInputTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + os.environ["TAI_HOME"] = self.tmp.name + + class FakeArgs: + profile = "t" + yes = False + + self.config = tai.Config(FakeArgs()) + self.store = tai.Store(self.config, tai.Seal(self.config.home, "approval-test-1")) + + def tearDown(self): + self.store.close() + self.tmp.cleanup() + os.environ.pop("TAI_HOME", None) + + def ask(self, agent, answers): + with mock.patch("sys.stdin") as fake_stdin: + fake_stdin.isatty.return_value = True + with mock.patch("builtins.input", side_effect=list(answers)) as asker: + with mock.patch("builtins.print"): + try: + return ("allowed", agent.ask_approval("curl https://example.com/x"), asker.call_count) + except tai.Denied as denied: + return ("denied", denied.guidance, asker.call_count) + + def test_unknown_answer_reprompts(self): + agent = tai.Agent(self.config, self.store, persist=True, quiet=True) + self.assertEqual(self.ask(agent, ["A", "y"]), ("allowed", True, 2)) + + def test_unknown_answers_capped_then_denies(self): + agent = tai.Agent(self.config, self.store, persist=True, quiet=True) + answers = ["zzz"] * tai.APPROVAL_MAX_RETRIES + ["do Y instead"] + status, guidance, calls = self.ask(agent, answers) + self.assertEqual(status, "denied") + self.assertEqual(guidance, "do Y instead") + self.assertEqual(calls, tai.APPROVAL_MAX_RETRIES + 1) + + def test_affirmative_guidance_approves_shell(self): + agent = tai.Agent(self.config, self.store, persist=True, quiet=True) + call = {"id": "c1", "name": "shell", "arguments": json.dumps({"command": "curl https://example.com/x"})} + first = {"role": "assistant", "content": "", "reasoning": "", "tool_calls": [call], "backend": "x"} + final = {"role": "assistant", "content": "done", "reasoning": "", "tool_calls": [], "backend": "x"} + with mock.patch.object(agent.chat, "complete", side_effect=[first, final]): + with mock.patch("tai.run_live", return_value=(0, "recovered-ok", 0.1, False)) as runner: + with mock.patch("sys.stdin") as fake_stdin: + fake_stdin.isatty.return_value = True + with mock.patch("builtins.input", side_effect=["n", "It was perfect"]): + with mock.patch("builtins.print"): + result = agent.run_turn("fetch x", capture=True) + self.assertEqual(result, "done") + runner.assert_called_once() + tools = [item for item in agent.messages if item.get("role") == "tool"] + self.assertIn("recovered-ok", tools[0]["content"]) + + def test_plain_guidance_still_denies(self): + agent = tai.Agent(self.config, self.store, persist=True, quiet=True) + self.assertEqual(self.ask(agent, ["n", "do X instead"]), ("denied", "do X instead", 2)) + + def test_affirmative_match_is_strict(self): + for text in ("It was perfect", "everything was fine", " FINE! ", "go ahead", "proceed"): + self.assertTrue(tai.Approval.is_affirmative(text), text) + for text in ("", "do X instead", "yes, but do X instead", "perfect, now delete everything", "it was perfect for you to ask"): + self.assertFalse(tai.Approval.is_affirmative(text), text) + + def test_affirmative_guidance_does_not_approve_delete(self): + agent = tai.Agent(self.config, self.store, persist=True, quiet=True) + path = os.path.join(self.tmp.name, "keep.txt") + with open(path, "w", encoding="utf-8") as handle: + handle.write("v1") + self.assertNotIn("error", agent.tools.dispatch("read_file", json.dumps({"path": path}))) + with mock.patch("sys.stdin") as fake_stdin: + fake_stdin.isatty.return_value = True + with mock.patch("builtins.input", side_effect=["n", "it was perfect"]): + with mock.patch("builtins.print"): + with self.assertRaises(tai.Denied): + agent.tools.dispatch("delete_file", json.dumps({"path": path})) + self.assertTrue(os.path.isfile(path)) + + +class GoalVerifyTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + os.environ["TAI_HOME"] = self.tmp.name + with tai.AGENTS_LOCK: + tai.AGENTS.clear() + tai.AGENTS_NEXT[0] = 1 + + class FakeArgs: + profile = "t" + yes = False + + self.config = tai.Config(FakeArgs()) + self.store = tai.Store(self.config, tai.Seal(self.config.home, "verify-test-1")) + + def tearDown(self): + self.store.close() + self.tmp.cleanup() + os.environ.pop("TAI_HOME", None) + with tai.AGENTS_LOCK: + tai.AGENTS.clear() + tai.AGENTS_NEXT[0] = 1 + + def final(self, content): + return {"role": "assistant", "content": content, "reasoning": "", "tool_calls": [], "backend": "x"} + + def worker(self): + return tai.Agent(self.config, self.store, persist=True, quiet=True) + + def test_verdict_parses_success(self): + verdict = tai.GoalVerdict.from_text(json.dumps({"success": True, "failures": [], "commentary": "python 3.12 live"})) + self.assertTrue(verdict.ok) + self.assertEqual(verdict.failures, []) + self.assertIn("python 3.12 live", verdict.report("upgrade python")) + + def test_verdict_parses_failures(self): + payload = {"success": False, "failures": [{"item": "download tarball", "reason": "curl was denied", "proposal": "approve the download and retry"}], "commentary": "nothing installed"} + verdict = tai.GoalVerdict.from_text("```json\n%s\n```" % json.dumps(payload)) + self.assertFalse(verdict.ok) + self.assertEqual(len(verdict.failures), 1) + failure = verdict.failures[0].to_dict() + self.assertEqual(failure["item"], "download tarball") + self.assertEqual(failure["reason"], "curl was denied") + self.assertEqual(failure["proposal"], "approve the download and retry") + report = verdict.report("upgrade python") + self.assertIn("download tarball", report) + self.assertIn("curl was denied", report) + self.assertIn("approve the download and retry", report) + + def test_verdict_is_strict(self): + self.assertFalse(tai.GoalVerdict.from_text("looks good to me").ok) + contradiction = tai.GoalVerdict.from_text(json.dumps({"success": True, "failures": [{"item": "x", "reason": "y", "proposal": "z"}]})) + self.assertFalse(contradiction.ok) + missing = tai.GoalVerdict.from_text(json.dumps({"success": True, "failures": [{"item": "x", "reason": "y"}]})) + self.assertFalse(missing.ok) + bare_fail = tai.GoalVerdict.from_text(json.dumps({"success": False, "failures": []})) + self.assertFalse(bare_fail.ok) + self.assertEqual(len(bare_fail.failures), 1) + self.assertTrue(bare_fail.failures[0].to_dict()["proposal"]) + + def test_run_verified_accepts_first_try(self): + agent = self.worker() + agent.runner_override = mock.Mock(return_value=json.dumps({"success": True, "failures": []})) + with mock.patch.object(agent.chat, "complete", return_value=self.final("installed")) as completer: + with mock.patch("builtins.print"): + text, verified = agent.run_verified("install python 3.12") + self.assertEqual((text, verified), ("installed", True)) + self.assertEqual(completer.call_count, 1) + self.assertEqual(agent.runner_override.call_count, 1) + state = self.store.load_turn_state("t", "main", ctx=agent.ctx_id) + self.assertFalse(state.get("open")) + + def test_run_verified_retries_with_proposals(self): + agent = self.worker() + failed = json.dumps({"success": False, "failures": [{"item": "tarball", "reason": "denied", "proposal": "approve curl and retry"}], "commentary": "not yet"}) + agent.runner_override = mock.Mock(side_effect=[failed, json.dumps({"success": True, "failures": []})]) + with mock.patch.object(agent.chat, "complete", side_effect=[self.final("almost"), self.final("done")]) as completer: + with mock.patch("builtins.print"): + text, verified = agent.run_verified("install python 3.12") + self.assertEqual((text, verified), ("done", True)) + self.assertEqual(completer.call_count, 2) + second = completer.call_args_list[1].args[0] + users = [item["content"] for item in second if item["role"] == "user"] + self.assertTrue(any("approve curl and retry" in item for item in users)) + + def test_run_verified_exhausts_rounds(self): + agent = self.worker() + failed = json.dumps({"success": False, "failures": [{"item": "tarball", "reason": "denied", "proposal": "approve curl"}]}) + agent.runner_override = mock.Mock(return_value=failed) + with mock.patch.object(agent.chat, "complete", return_value=self.final("stuck")) as completer: + with mock.patch("builtins.print"): + text, verified = agent.run_verified("install python 3.12") + self.assertEqual((text, verified), ("stuck", False)) + self.assertEqual(completer.call_count, 1 + tai.VERIFY_MAX_ROUNDS) + self.assertEqual(agent.runner_override.call_count, 1 + tai.VERIFY_MAX_ROUNDS) + state = self.store.load_turn_state("t", "main", ctx=agent.ctx_id) + self.assertTrue(state.get("open")) + + def test_run_verified_abort_skips_verifier(self): + agent = self.worker() + agent.runner_override = mock.Mock(return_value=json.dumps({"success": True, "failures": []})) + call = {"id": "c1", "name": "shell", "arguments": json.dumps({"command": "curl https://example.com/x"})} + first = {"role": "assistant", "content": "", "reasoning": "", "tool_calls": [call], "backend": "x"} + with mock.patch.object(agent.chat, "complete", return_value=first): + with mock.patch("sys.stdin") as fake_stdin: + fake_stdin.isatty.return_value = True + with mock.patch("builtins.input", side_effect=["n", ""]): + with mock.patch("builtins.print"): + text, verified = agent.run_verified("fetch x") + self.assertEqual((text, verified), ("stopped by user", False)) + agent.runner_override.assert_not_called() + state = self.store.load_turn_state("t", "main", ctx=agent.ctx_id) + self.assertTrue(state.get("open")) + + def test_run_verified_reads_full_verdict(self): + agent = self.worker() + payload = json.dumps({"success": True, "failures": [], "commentary": "x" * 9000}) + self.assertGreater(len(payload), 8050) + agent.runner_override = mock.Mock(return_value=payload) + with mock.patch.object(agent.chat, "complete", return_value=self.final("done")): + with mock.patch("builtins.print"): + text, verified = agent.run_verified("do it") + self.assertEqual((text, verified), ("done", True)) + + +class RetrievalEvalTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + os.environ["TAI_HOME"] = self.tmp.name + + class FakeArgs: + profile = "t" + yes = True + + self.config = tai.Config(FakeArgs()) + self.store = tai.Store(self.config, tai.Seal(self.config.home, "eval-test-1")) + self.app = FakeSecretApp(self.store) + self.app.profile = "t" + self.app.config = self.config + self.tools = tai.Tools(self.app) + + def tearDown(self): + self.store.close() + self.tmp.cleanup() + os.environ.pop("TAI_HOME", None) + + def seed(self): + ids = {} + ids["title"] = self.store.add_record("note", "quasar beacon", "short body here", [], "t") + ids["stuffed"] = self.store.add_record("note", "unrelated title", "background words filler text. " * 10 + "quasar beacon " * 10, [], "t") + ids["harbor"] = self.store.add_record("note", "harbor launch checklist", "step one step two step three", [], "t") + ids["meridian"] = self.store.add_record("note", "meridian project notes", "alpha beta gamma delta", [], "t") + ids["report"] = self.store.add_record("note", "tangerine report", "findings summary appendix extra", [], "t") + return ids + + def test_metric_math(self): + self.assertEqual(tai.RetrievalEval.rank_of("a", ["b", "a"]), 2) + self.assertIsNone(tai.RetrievalEval.rank_of("z", ["b", "a"])) + self.assertEqual(tai.RetrievalEval.hit_at_k(2, 5), 1.0) + self.assertEqual(tai.RetrievalEval.hit_at_k(None, 5), 0.0) + self.assertEqual(tai.RetrievalEval.hit_at_k(6, 5), 0.0) + self.assertEqual(tai.RetrievalEval.reciprocal_rank(2), 0.5) + self.assertEqual(tai.RetrievalEval.reciprocal_rank(None), 0.0) + self.assertAlmostEqual(tai.RetrievalEval.ndcg_at_k(1, 5), 1.0) + self.assertAlmostEqual(tai.RetrievalEval.ndcg_at_k(2, 5), 1.0 / 1.584962500721156) + self.assertEqual(tai.RetrievalEval.ndcg_at_k(None, 5), 0.0) + self.assertEqual(tai.RetrievalEval.context_precision(["a"], ["a", "b"], 5), 1.0) + self.assertEqual(tai.RetrievalEval.context_precision(["b"], ["a", "b"], 5), 0.5) + self.assertEqual(tai.RetrievalEval.context_precision([], ["a"], 5), 0.0) + + def test_goldens_deterministic_and_valid(self): + self.seed() + first = tai.RetrievalEval.goldens_from_vault(self.store, "t") + second = tai.RetrievalEval.goldens_from_vault(self.store, "t") + self.assertEqual(first, second) + self.assertTrue(first) + categories = {golden["category"] for golden in first} + self.assertEqual(categories, {"exact", "phrase", "prefix", "noisy"}) + ids = {row[0] for row in self.store.db.execute("SELECT id FROM records WHERE profile = 't'").fetchall()} + for golden in first: + self.assertTrue(golden["query"].strip()) + self.assertTrue(set(golden["expected"]) <= ids) + + def test_score_exact_and_phrase(self): + self.seed() + goldens = [golden for golden in tai.RetrievalEval.goldens_from_vault(self.store, "t") if golden["category"] in ("exact", "phrase")] + result = tai.RetrievalEval.score(self.store, goldens, "t") + self.assertEqual(result["by_category"]["exact"]["mean"]["hit"], 1.0) + self.assertEqual(result["by_category"]["phrase"]["mean"]["hit"], 1.0) + + def test_score_prefix(self): + self.seed() + goldens = [golden for golden in tai.RetrievalEval.goldens_from_vault(self.store, "t") if golden["category"] == "prefix"] + self.assertTrue(goldens) + result = tai.RetrievalEval.score(self.store, goldens, "t") + self.assertEqual(result["by_category"]["prefix"]["mean"]["hit"], 1.0) + + def test_score_noisy(self): + self.seed() + goldens = [golden for golden in tai.RetrievalEval.goldens_from_vault(self.store, "t") if golden["category"] == "noisy"] + self.assertTrue(goldens) + result = tai.RetrievalEval.score(self.store, goldens, "t") + self.assertEqual(result["by_category"]["noisy"]["mean"]["hit"], 1.0) + + def test_title_outranks_body(self): + self.seed() + hits = self.store.fts_search("quasar beacon", ("record",), "t", 5) + self.assertEqual(hits[0]["title"], "quasar beacon") + + def test_search_query_logged(self): + ids = self.seed() + out = self.tools.dispatch("search", json.dumps({"query": "quasar beacon", "kinds": ["record"]})) + self.assertIn(ids["title"], out) + rows = self.store.db.execute("SELECT role, kind, tai_dec(text) FROM events WHERE kind = 'search-query'").fetchall() + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0][0], "tool") + self.assertIn("q=quasar beacon", rows[0][2]) + self.assertIn(ids["title"], rows[0][2]) + self.tools.dispatch("search", json.dumps({"query": "zzz-nope-nothing"})) + rows = self.store.db.execute("SELECT tai_dec(text) FROM events WHERE kind = 'search-query' ORDER BY rowid").fetchall() + self.assertEqual(len(rows), 2) + self.assertIn("hits=", rows[1][0].split("\t")[-1]) + self.assertTrue(rows[1][0].endswith("hits=")) + + def test_record_read_logged_and_pairs(self): + ids = self.seed() + self.tools.dispatch("search", json.dumps({"query": "quasar beacon", "kinds": ["record"]})) + self.tools.dispatch("record_read", json.dumps({"id": ids["title"]})) + self.tools.dispatch("search", json.dumps({"query": "zzz-nope-nothing"})) + pairs = tai.RetrievalEval.implicit_pairs(self.store, "t") + self.assertEqual(pairs["searches"], 2) + self.assertEqual(pairs["read_follow"], 1) + self.assertEqual(len(pairs["pairs"]), 1) + self.assertEqual(pairs["pairs"][0]["read"], ids["title"]) + self.assertEqual(pairs["zero_hit_count"], 1) + self.assertEqual(pairs["zero_hit"], ["zzz-nope-nothing"]) + + +class SupervisorCapTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + os.environ["TAI_HOME"] = self.tmp.name + + class FakeArgs: + profile = "t" + yes = True + + self.config = tai.Config(FakeArgs()) + self.seal = tai.Seal(self.config.home, "sup-test-1") + self.store = tai.Store(self.config, self.seal) + self.gate = threading.Event() + with tai.AGENTS_LOCK: + tai.AGENTS.clear() + tai.AGENTS_NEXT[0] = 1 + + def tearDown(self): + self.gate.set() + for agent_id in [record["id"] for record in tai.list_agents()]: + tai.poll_agent(agent_id, wait=5) + self.store.close() + self.tmp.cleanup() + os.environ.pop("TAI_HOME", None) + with tai.AGENTS_LOCK: + tai.AGENTS.clear() + tai.AGENTS_NEXT[0] = 1 + + def blocking(self, task, profile, timeout): + self.gate.wait(10) + return "released" + + def fill(self, count=None): + count = tai.AGENTS_MAX_PARALLEL if count is None else count + return [tai.spawn_agent("hold %d" % pos, "t", 60, self.config, self.seal, 1, self.blocking) for pos in range(count)] + + def test_spawn_refuses_past_cap(self): + ids = self.fill() + self.assertEqual(len(ids), tai.AGENTS_MAX_PARALLEL) + with self.assertRaises(tai.AgentsFull): + tai.spawn_agent("one more", "t", 60, self.config, self.seal, 1, self.blocking) + self.gate.set() + for agent_id in ids: + tai.poll_agent(agent_id, wait=5) + extra = tai.spawn_agent("after drain", "t", 60, self.config, self.seal, 1, lambda task, profile, timeout: "ok") + self.assertIsInstance(extra, int) + + def test_fork_reports_cap_loudly(self): + principal = FakePrincipal() + principal.runner_override = self.blocking + tools = tai.Tools(principal) + self.fill() + out = tools.dispatch("fork", json.dumps({"task": "one more"})) + self.assertIn("too many live agents", out) + self.assertIn("poll existing", out) + self.gate.set() + for agent_id in [record["id"] for record in tai.list_agents()]: + tai.poll_agent(agent_id, wait=5) + self.assertIn("started", tools.dispatch("fork", json.dumps({"task": "after drain"}))) + + def test_spawn_purges_collected_past_cap(self): + ids = [] + for pos in range(tai.AGENTS_MAX_RECORDS): + agent_id = tai.spawn_agent("quick %d" % pos, "t", 60, self.config, self.seal, 1, lambda task, profile, timeout: "ok") + tai.poll_agent(agent_id, wait=5) + with tai.AGENTS_LOCK: + tai.AGENTS[agent_id]["collected"] = True + ids.append(agent_id) + tai.spawn_agent("next", "t", 60, self.config, self.seal, 1, lambda task, profile, timeout: "ok") + with tai.AGENTS_LOCK: + self.assertNotIn(ids[0], tai.AGENTS) + self.assertIn(ids[-1], tai.AGENTS) + self.assertEqual(len(tai.AGENTS), tai.AGENTS_MAX_RECORDS) + + def test_tick_per_tick_cap(self): + for pos in range(tai.SCHEDULER_MAX_FIRES + 2): + self.store.add_schedule("job%d" % pos, "do it", "t", 0, "2020-01-01T00:00:00+00:00", 60) + with mock.patch("builtins.print"): + fired = tai.scheduler_tick(self.config, self.seal, runner=lambda task, profile, timeout: "ok") + self.assertEqual(fired, tai.SCHEDULER_MAX_FIRES) + left = self.store.db.execute("SELECT COUNT(*) FROM schedules WHERE status = 'pending'").fetchone()[0] + self.assertEqual(left, 2) + + def test_tick_overlap_skips_running(self): + past = (datetime.now(timezone.utc) - timedelta(hours=2)).isoformat() + self.store.add_schedule("hourly", "ping", "t", 3600, past, 600) + self.store.db.execute("UPDATE schedules SET last_status = 'running', updated = ? WHERE id = 1", (datetime.now(timezone.utc).isoformat(),)) + self.store.db.commit() + with mock.patch("builtins.print"): + fired = tai.scheduler_tick(self.config, self.seal, runner=lambda task, profile, timeout: "ok") + self.assertEqual(fired, 0) + row = self.store.db.execute("SELECT status, next_run FROM schedules WHERE id = 1").fetchone() + self.assertEqual(row[0], "pending") + self.assertEqual(row[1], past) + + def test_tick_stale_lease_fires(self): + past = (datetime.now(timezone.utc) - timedelta(hours=3)).isoformat() + self.store.add_schedule("hourly", "ping", "t", 3600, past, 600) + stale = (datetime.now(timezone.utc) - timedelta(hours=2)).isoformat() + self.store.db.execute("UPDATE schedules SET last_status = 'running', updated = ? WHERE id = 1", (stale,)) + self.store.db.commit() + with mock.patch("builtins.print"): + fired = tai.scheduler_tick(self.config, self.seal, runner=lambda task, profile, timeout: "ok") + self.assertEqual(fired, 1) + + def test_tick_skips_at_cap(self): + self.store.add_schedule("job", "do it", "t", 0, "2020-01-01T00:00:00+00:00", 60) + self.fill() + with mock.patch("builtins.print") as printed: + fired = tai.scheduler_tick(self.config, self.seal, runner=lambda task, profile, timeout: "ok") + self.assertEqual(fired, 0) + self.assertEqual(self.store.db.execute("SELECT status FROM schedules WHERE id = 1").fetchone()[0], "pending") + self.assertTrue(any("parallel cap" in str(call) for call in printed.call_args_list)) + + class FileGuardTests(unittest.TestCase): def test_write_requires_read(self): with tempfile.TemporaryDirectory() as tmp: @@ -8419,6 +8962,47 @@ class NoOpencodeFlagTests(unittest.TestCase): self.assertTrue(tai.Config(args).opencode_enabled) +class TuiDispatchTests(unittest.TestCase): + def run_main(self, argv): + agent = mock.Mock() + with mock.patch.object(tai, "boot", return_value=agent), \ + mock.patch.object(tai, "tui_chat", return_value=0) as chatted: + code = tai.main(argv) + return code, agent, chatted + + def test_tui_local_wins_over_prompt(self): + code, agent, chatted = self.run_main(["--tui-local", "hello", "there"]) + self.assertEqual(code, 0) + chatted.assert_called_once() + agent.run_turn.assert_not_called() + + def test_tui_local_without_prompt(self): + code, _agent, chatted = self.run_main(["--tui-local"]) + self.assertEqual(code, 0) + chatted.assert_called_once() + + def test_tui_wins_over_prompt(self): + with mock.patch.object(tai, "grr_tui", return_value=0) as grr: + code = tai.main(["show", "off", "--tui"]) + self.assertEqual(code, 0) + grr.assert_called_once() + + def test_prompt_without_flag_runs_turn(self): + code, agent, chatted = self.run_main(["hello"]) + self.assertEqual(code, 0) + agent.run_turn.assert_called_once_with("hello") + chatted.assert_not_called() + + def test_prompt_unverified_returns_nonzero(self): + agent = mock.Mock() + agent.ensure_verified.return_value = False + with mock.patch.object(tai, "boot", return_value=agent): + code = tai.main(["hello"]) + self.assertEqual(code, 1) + agent.run_turn.assert_called_once_with("hello") + agent.ensure_verified.assert_called_once() + + class UpdateModelsTests(unittest.TestCase): def test_replace_list_prunes_only_that_provider(self): with tempfile.TemporaryDirectory() as home: @@ -10085,7 +10669,8 @@ class MplexSpecTests(unittest.TestCase): def test_every_cap_is_documented(self): for name in ("MPLEX_MAX_SESSIONS", "MPLEX_MAX_WINDOWS", "MPLEX_MAX_PANES", "MPLEX_MAX_CLIENTS", "MPLEX_FRAME_MAX", "MPLEX_CLIENT_OUT_MAX", "MPLEX_PANE_IN_MAX", "MPLEX_HISTORY_CAP", "MPLEX_LOG_MAX", "MPLEX_BUFFERS_CAP", "MPLEX_MESSAGES_MAX", "MPLEX_PROMPT_HISTORY_CAP", - "MPLEX_COMMAND_DEPTH_MAX", "MPLEX_LOOP_ERRORS_MAX", "MPLEX_SHELL_TIMEOUT", "MPLEX_CONFIG_MAX", "MPLEX_PASTE_MAX", "MPLEX_RENDER_INTERVAL", "MPLEX_REAP_KILL_AFTER"): + "MPLEX_COMMAND_DEPTH_MAX", "MPLEX_LOOP_ERRORS_MAX", "MPLEX_SHELL_TIMEOUT", "MPLEX_CONFIG_MAX", "MPLEX_PASTE_MAX", "MPLEX_RENDER_INTERVAL", "MPLEX_REAP_KILL_AFTER", + "MPLEX_FLOW_MAX", "MPLEX_FLOW_KEEP"): self.assertIn("`%s`" % name, self.doc, name) self.assertTrue(hasattr(tai, name), name) @@ -10353,6 +10938,718 @@ class MplexPaneEnvTests(unittest.TestCase): self.assertEqual(controller.pane, "7") +class ScriptedController: + """A PlexerController.ask stand-in with scripted replies.""" + + def __init__(self, replies): + self.replies = list(replies) + self.asked = [] + + def ask(self, screen, timeout): + self.asked.append((screen, timeout)) + reply = self.replies.pop(0) + if isinstance(reply, Exception): + raise reply + return reply + + +def flow_spec(**over): + spec = {"title": "T", "timeout": 60, "screens": over.pop("screens", [ + {"id": "main", "body": "B", "fields": [{"id": "env", "type": "choice", "options": ["a", "b"]}], + "actions": [{"id": "ok", "label": "OK"}], "next": "$done"}])} + spec.update(over) + return spec + + +class PlexerFlowSpecTests(unittest.TestCase): + def check(self, spec): + return tai.validate_flow_spec(spec) + + def test_minimal_spec_normalizes(self): + app, error = self.check(flow_spec()) + self.assertEqual(error, "") + self.assertEqual(app["start"], "main") + self.assertEqual(app["timeout"], 60) + self.assertEqual(app["screens"][0]["fields"][0]["options"][0], {"label": "a", "value": "a"}) + self.assertEqual(app["screens"][0]["next"], "$done") + + def test_bad_version_rejected(self): + _app, error = self.check(flow_spec(v=2)) + self.assertIn("v must be 1", error) + + def test_missing_v_defaults_to_current(self): + app, error = self.check(flow_spec()) + self.assertEqual(error, "") + self.assertNotIn("v", flow_spec()) + + def test_empty_screens_rejected(self): + _app, error = self.check(flow_spec(screens=[])) + self.assertIn("non-empty list", error) + + def test_duplicate_screen_id_rejected(self): + screens = [{"id": "a", "body": "x"}, {"id": "a", "body": "y"}] + _app, error = self.check(flow_spec(screens=screens)) + self.assertIn("duplicate screen id", error) + + def test_screen_id_with_whitespace_rejected(self): + _app, error = self.check(flow_spec(screens=[{"id": "a b", "body": "x"}])) + self.assertIn("must not contain whitespace", error) + + def test_empty_screen_rejected(self): + _app, error = self.check(flow_spec(screens=[{"id": "a"}])) + self.assertIn("nothing to show", error) + + def test_unknown_field_type_rejected(self): + fields = [{"id": "f", "type": "slider"}] + _app, error = self.check(flow_spec(screens=[{"id": "a", "body": "x", "fields": fields}])) + self.assertIn("type must be one of", error) + + def test_duplicate_field_id_rejected(self): + screens = [{"id": "a", "fields": [{"id": "f", "type": "info", "text": "x"}]}, + {"id": "b", "fields": [{"id": "f", "type": "info", "text": "y"}]}] + _app, error = self.check(flow_spec(screens=screens)) + self.assertIn("duplicate field id", error) + + def test_choice_without_options_rejected(self): + fields = [{"id": "f", "type": "choice", "options": []}] + _app, error = self.check(flow_spec(screens=[{"id": "a", "body": "x", "fields": fields}])) + self.assertIn("needs 1 to", error) + + def test_choice_default_outside_options_rejected(self): + fields = [{"id": "f", "type": "choice", "options": ["a"], "default": 3}] + _app, error = self.check(flow_spec(screens=[{"id": "a", "body": "x", "fields": fields}])) + self.assertIn("outside its", error) + + def test_show_if_unknown_field_rejected(self): + fields = [{"id": "f", "type": "info", "text": "x", "show_if": {"nope": 1}}] + _app, error = self.check(flow_spec(screens=[{"id": "a", "body": "x", "fields": fields}])) + self.assertIn("unknown field", error) + + def test_next_unknown_screen_rejected(self): + screens = [{"id": "a", "body": "x", "next": "nope"}] + _app, error = self.check(flow_spec(screens=screens)) + self.assertIn("unknown screen", error) + + def test_next_map_without_actions_rejected(self): + screens = [{"id": "a", "body": "x", "next": {"go": "$done"}}] + _app, error = self.check(flow_spec(screens=screens)) + self.assertIn("no actions", error) + + def test_next_map_unknown_action_rejected(self): + screens = [{"id": "a", "body": "x", "actions": [{"id": "go", "label": "Go"}], "next": {"stop": "$done"}}] + _app, error = self.check(flow_spec(screens=screens)) + self.assertIn("unknown action", error) + + def test_start_unknown_screen_rejected(self): + _app, error = self.check(flow_spec(start="nope")) + self.assertIn("start refers to unknown", error) + + def test_timeout_out_of_range_rejected(self): + _app, error = self.check(flow_spec(timeout=5 * 3600)) + self.assertIn("timeout must be", error) + + def test_too_many_screens_rejected(self): + screens = [{"id": "s%d" % num, "body": "x"} for num in range(tai.PLEXER_FLOW_SCREENS_MAX + 1)] + _app, error = self.check(flow_spec(screens=screens)) + self.assertIn("at most", error) + + +class PlexerFlowDriverTests(unittest.TestCase): + def drive(self, spec, replies): + app, error = tai.validate_flow_spec(spec) + self.assertEqual(error, "") + controller = ScriptedController(replies) + return tai.PlexerFlow(app).run(controller), controller + + def test_done_flow_maps_values_and_action(self): + spec = flow_spec(screens=[{"id": "main", "title": "T", "body": "B", + "fields": [{"id": "env", "type": "choice", + "options": [{"label": "Prod", "value": "p"}, {"label": "Stage", "value": "s"}]}, + {"id": "tag", "type": "text"}], + "actions": [{"id": "go", "label": "Go"}], "next": "$done"}]) + outcome, controller = self.drive(spec, [{"status": "ok", "value": 0}, {"status": "ok", "value": "v1"}, + {"status": "ok", "value": 0}]) + self.assertEqual(outcome, {"status": "done", "answers": {"env": "p", "tag": "v1"}, "action": "go"}) + self.assertEqual(len(controller.asked), 3) + self.assertEqual(controller.asked[0][0]["kind"], "choice") + self.assertEqual(controller.asked[1][0]["kind"], "text") + + def test_first_ask_carries_context_later_ones_do_not(self): + spec = flow_spec(screens=[{"id": "main", "title": "T", "body": "BODY", + "fields": [{"id": "one", "type": "text", "label": "First"}, + {"id": "two", "type": "text", "label": "Second"}]}]) + _outcome, controller = self.drive(spec, [{"status": "ok", "value": "a"}, {"status": "ok", "value": "b"}]) + self.assertIn("BODY", controller.asked[0][0]["text"]) + self.assertNotIn("BODY", controller.asked[1][0]["text"]) + self.assertIn("Second", controller.asked[1][0]["text"]) + + def test_show_if_skips_field(self): + spec = flow_spec(screens=[{"id": "main", "body": "B", + "fields": [{"id": "env", "type": "choice", "options": ["a", "b"]}, + {"id": "extra", "type": "text", "show_if": {"env": "b"}}]}]) + outcome, controller = self.drive(spec, [{"status": "ok", "value": 0}]) + self.assertEqual(outcome["status"], "done") + self.assertEqual(outcome["answers"], {"env": "a"}) + self.assertEqual(len(controller.asked), 1) + + def test_show_if_list_accepts_any(self): + spec = flow_spec(screens=[{"id": "main", "body": "B", + "fields": [{"id": "env", "type": "choice", "options": ["a", "b"]}, + {"id": "extra", "type": "text", "show_if": {"env": ["b", "c"]}}]}]) + outcome, _controller = self.drive(spec, [{"status": "ok", "value": 1}, {"status": "ok", "value": "x"}]) + self.assertEqual(outcome["answers"], {"env": "b", "extra": "x"}) + + def test_show_if_skips_screen(self): + spec = flow_spec(screens=[{"id": "one", "body": "B", + "fields": [{"id": "env", "type": "choice", "options": ["a", "b"]}]}, + {"id": "two", "body": "skipped?", "show_if": {"env": "b"}}, + {"id": "three", "body": "last"}]) + outcome, controller = self.drive(spec, [{"status": "ok", "value": 0}, {"status": "ok", "value": True}]) + self.assertEqual(outcome["status"], "done") + self.assertEqual(len(controller.asked), 2) + self.assertNotIn("skipped?", "".join(text for (screen, _t) in controller.asked for text in [screen["text"]])) + + def test_next_map_branches(self): + spec = flow_spec(screens=[{"id": "one", "body": "B", "actions": [{"id": "yes", "label": "Yes"}, {"id": "no", "label": "No"}], + "next": {"yes": "two", "no": "$done"}}, + {"id": "two", "body": "branched"}]) + outcome, controller = self.drive(spec, [{"status": "ok", "value": 0}, {"status": "ok", "value": True}]) + self.assertEqual(outcome, {"status": "done", "answers": {}, "action": "yes"}) + self.assertEqual(len(controller.asked), 2) + + def test_next_map_missing_action_falls_through(self): + spec = flow_spec(screens=[{"id": "one", "body": "B", "actions": [{"id": "yes", "label": "Yes"}, {"id": "no", "label": "No"}], + "next": {"yes": "$done"}}, + {"id": "two", "body": "fell through"}]) + outcome, controller = self.drive(spec, [{"status": "ok", "value": 1}, {"status": "ok", "value": True}]) + self.assertEqual(outcome["status"], "done") + self.assertEqual(outcome["action"], "no") + self.assertEqual(len(controller.asked), 2) + + def test_cancel_aborts_with_partial_answers(self): + spec = flow_spec(screens=[{"id": "main", "body": "B", + "fields": [{"id": "one", "type": "text"}, {"id": "two", "type": "text"}]}]) + outcome, _controller = self.drive(spec, [{"status": "ok", "value": "a"}, {"status": "cancelled"}]) + self.assertEqual(outcome, {"status": "cancelled", "at": "main", "field": "two", "answers": {"one": "a"}}) + + def test_timeout_aborts(self): + spec = flow_spec() + outcome, _controller = self.drive(spec, [{"status": "timeout"}]) + self.assertEqual(outcome["status"], "timeout") + self.assertEqual(outcome["at"], "main") + + def test_loop_trips_visit_cap(self): + spec = flow_spec(screens=[{"id": "a", "body": "A", "next": "b"}, {"id": "b", "body": "B", "next": "a"}]) + outcome, controller = self.drive(spec, [{"status": "ok", "value": True}] * (tai.PLEXER_FLOW_VISITS_MAX + 5)) + self.assertEqual(outcome["status"], "error") + self.assertIn("loop", outcome["error"]) + self.assertEqual(len(controller.asked), tai.PLEXER_FLOW_VISITS_MAX) + + def test_message_only_screen_acks_once(self): + spec = flow_spec(screens=[{"id": "a", "body": "hello"}]) + outcome, controller = self.drive(spec, [{"status": "ok", "value": True}]) + self.assertEqual(outcome["status"], "done") + self.assertEqual(len(controller.asked), 1) + self.assertEqual(controller.asked[0][0]["kind"], "message") + self.assertIn("hello", controller.asked[0][0]["text"]) + + def test_cancel_route_cancels(self): + spec = flow_spec(screens=[{"id": "a", "body": "B", "next": "$cancel"}]) + outcome, controller = self.drive(spec, [{"status": "ok", "value": True}]) + self.assertEqual(outcome["status"], "cancelled") + self.assertEqual(len(controller.asked), 1) + + def test_info_fields_join_context_but_are_never_asked(self): + spec = flow_spec(screens=[{"id": "a", "body": "B", + "fields": [{"id": "n", "type": "info", "text": "NOTE"}, + {"id": "t", "type": "text", "label": "L"}]}]) + outcome, controller = self.drive(spec, [{"status": "ok", "value": "x"}]) + self.assertEqual(outcome["answers"], {"t": "x"}) + self.assertEqual(len(controller.asked), 1) + self.assertIn("NOTE", controller.asked[0][0]["text"]) + + +class FlowStub: + def __init__(self, **kw): + self.__dict__.update(kw) + + +class MplexFlowServerTests(unittest.TestCase): + def make(self): + server = tai.MplexServer.__new__(tai.MplexServer) + server.flows = {} + server.flow_seq = 0 + server.sessions = {} + server.clients = [] + server.panes = {} + session = FlowStub(name="s") + window = FlowStub(session=session) + pane = FlowStub(window=window, tag="%3") + server.panes[3] = pane + client = FlowStub(server=server, attached=True, session=session, overlay=None, prompt=None, dirty=False) + server.clients.append(client) + return server, pane, client + + def show(self, server, screen, pane="3", timeout=30): + return server.dispatch_control({"cmd": "show", "pane": pane, "screen": screen, "timeout": timeout}) + + def test_choice_full_cycle(self): + server, _pane, client = self.make() + reply = self.show(server, {"kind": "choice", "title": "T", "text": "Pick", "options": ["a", "b", "c"]}) + self.assertTrue(reply["ok"]) + self.assertEqual(reply["data"], {"id": 0}) + overlay = client.overlay + self.assertIsInstance(overlay, tai.MplexFlowScreen) + self.assertEqual(server.dispatch_control({"cmd": "result", "id": 0})["data"], {"done": False}) + overlay.key("Down", b"", client) + self.assertEqual(overlay.cursor, 1) + overlay.key("Enter", b"\r", client) + done = server.dispatch_control({"cmd": "result", "id": 0}) + self.assertEqual(done["data"], {"done": True, "status": "ok", "value": 1}) + self.assertEqual(server.flows, {}) + self.assertIsNone(client.overlay) + + def test_choice_percent_pane_and_default_cursor(self): + server, _pane, client = self.make() + reply = self.show(server, {"kind": "choice", "title": "", "text": "P", "options": ["a", "b", "c"], "default": 2}, pane="%3") + self.assertTrue(reply["ok"]) + self.assertEqual(client.overlay.cursor, 2) + client.overlay.key("Enter", b"\r", client) + done = server.dispatch_control({"cmd": "result", "id": 0}) + self.assertEqual(done["data"]["value"], 2) + + def test_digit_selects_q_cancels_up_clamps(self): + server, _pane, client = self.make() + self.show(server, {"kind": "choice", "title": "", "text": "P", "options": ["a", "b", "c"]}) + overlay = client.overlay + overlay.key("Up", b"", client) + self.assertEqual(overlay.cursor, 0) + overlay.key("9", b"9", client) + self.assertIsNone(server.flows[0]["answer"]) + overlay.key("2", b"2", client) + done = server.dispatch_control({"cmd": "result", "id": 0}) + self.assertEqual(done["data"], {"done": True, "status": "ok", "value": 1}) + self.show(server, {"kind": "choice", "title": "", "text": "P", "options": ["a"]}) + client.overlay.key("q", b"q", client) + done = server.dispatch_control({"cmd": "result", "id": 1}) + self.assertEqual(done["data"]["status"], "cancelled") + + def test_message_acks_any_key_escape_cancels(self): + server, _pane, client = self.make() + self.show(server, {"kind": "message", "title": "T", "text": "hi"}) + client.overlay.key("x", b"x", client) + done = server.dispatch_control({"cmd": "result", "id": 0}) + self.assertEqual(done["data"], {"done": True, "status": "ok", "value": True}) + self.show(server, {"kind": "message", "title": "", "text": "hi"}) + client.overlay.key("Escape", b"\x1b", client) + done = server.dispatch_control({"cmd": "result", "id": 1}) + self.assertEqual(done["data"]["status"], "cancelled") + + def test_text_prompt_answers_and_cancels(self): + server, _pane, client = self.make() + self.show(server, {"kind": "text", "title": "T", "text": "Name?", "initial": "hi"}) + prompt = client.prompt + self.assertIsInstance(prompt, tai.MplexPrompt) + self.assertIsNone(client.overlay) + prompt.key(server, client, "Enter", b"\r") + done = server.dispatch_control({"cmd": "result", "id": 0}) + self.assertEqual(done["data"], {"done": True, "status": "ok", "value": "hi"}) + self.assertIsNone(client.prompt) + self.show(server, {"kind": "text", "title": "", "text": "Name?"}) + client.prompt.key(server, client, "Escape", b"\x1b") + done = server.dispatch_control({"cmd": "result", "id": 1}) + self.assertEqual(done["data"]["status"], "cancelled") + + def test_busy_client_refuses_second_show(self): + server, _pane, _client = self.make() + self.show(server, {"kind": "message", "title": "", "text": "one"}) + with self.assertRaises(ValueError): + self.show(server, {"kind": "message", "title": "", "text": "two"}) + with self.assertRaises(ValueError): + self.show(server, {"kind": "text", "title": "", "text": "t"}) + + def test_busy_prompt_refuses_overlay_show(self): + server, _pane, _client = self.make() + self.show(server, {"kind": "text", "title": "", "text": "t"}) + with self.assertRaises(ValueError): + self.show(server, {"kind": "message", "title": "", "text": "m"}) + + def test_no_attached_client_refuses(self): + server, _pane, client = self.make() + client.attached = False + with self.assertRaises(ValueError): + self.show(server, {"kind": "message", "title": "", "text": "m"}) + + def test_unknown_pane_refuses(self): + server, _pane, _client = self.make() + with self.assertRaises(ValueError): + self.show(server, {"kind": "message", "title": "", "text": "m"}, pane="9") + + def test_malformed_screens_refuse(self): + server, _pane, _client = self.make() + bad = [{"kind": "slider"}, {"kind": "choice", "options": []}, {"kind": "choice", "options": ["a"], "default": 5}, + {"kind": "message", "title": "x" * 500}, {"kind": "text", "text": "t", "initial": "x" * 5000}, "nope"] + for screen in bad: + with self.assertRaises(ValueError, msg=screen): + self.show(server, screen) + + def test_cancel_clears_and_unknown_raises(self): + server, _pane, client = self.make() + self.show(server, {"kind": "message", "title": "", "text": "m"}) + reply = server.dispatch_control({"cmd": "cancel", "id": 0}) + self.assertEqual(reply, {"ok": True, "data": True}) + self.assertIsNone(client.overlay) + self.assertEqual(server.flows, {}) + with self.assertRaises(ValueError): + server.dispatch_control({"cmd": "cancel", "id": 0}) + with self.assertRaises(ValueError): + server.dispatch_control({"cmd": "result", "id": 0}) + + def test_expired_flow_times_out(self): + server, _pane, client = self.make() + self.show(server, {"kind": "message", "title": "", "text": "m"}, timeout=30) + server.flows[0]["until"] = time.monotonic() - 1 + done = server.dispatch_control({"cmd": "result", "id": 0}) + self.assertEqual(done["data"], {"done": True, "status": "timeout"}) + self.assertIsNone(client.overlay) + self.assertEqual(server.flows, {}) + + def test_detached_client_reports_detached(self): + server, _pane, client = self.make() + self.show(server, {"kind": "message", "title": "", "text": "m"}) + server.clients.remove(client) + done = server.dispatch_control({"cmd": "result", "id": 0}) + self.assertEqual(done["data"], {"done": True, "status": "detached"}) + + def test_too_many_flows_refuses(self): + server, _pane, _client = self.make() + now = time.monotonic() + for num in range(tai.MPLEX_FLOW_MAX): + server.flows[num] = {"answer": None, "settled": None, "until": now + 9999} + with self.assertRaises(ValueError): + self.show(server, {"kind": "message", "title": "", "text": "m"}) + + def test_prune_reclaims_dead_drivers(self): + server, _pane, client = self.make() + now = time.monotonic() + overlay = tai.MplexFlowScreen(server, 1, "", "stale", [], 0, now - 1000) + client.overlay = overlay + server.flows[0] = {"answer": {"status": "ok", "value": True}, "settled": now - 1000, "until": 0, + "client": client, "overlay": None, "prompt": None} + server.flows[1] = {"answer": None, "settled": None, "until": now - 1000, "client": client, + "overlay": overlay, "prompt": None} + server.flow_prune(now) + self.assertEqual(server.flows, {}) + self.assertIsNone(client.overlay) + + def test_overlay_draws_dialog(self): + server, _pane, _client = self.make() + overlay = tai.MplexFlowScreen(server, 0, "Title", "line one\nline two", ["aaa", "bbb"], 1, time.monotonic() + 60) + canvas = tai.MplexCanvas(24, 80) + overlay.draw(canvas, 0) + flat = "".join(char for row in canvas.cells for char, _fg, _bg, _attr in row) + self.assertIn("Title", flat) + self.assertIn("line one", flat) + self.assertIn("aaa", flat) + self.assertIn("bbb", flat) + + def test_overlay_draws_on_tiny_client(self): + server, _pane, _client = self.make() + overlay = tai.MplexFlowScreen(server, 0, "T", "x", ["a", "b", "c"], 0, time.monotonic() + 60) + overlay.draw(tai.MplexCanvas(8, 30), 0) + overlay.draw(tai.MplexCanvas(24, 80), 1) + + +class TmuxAskTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.controller = tai.TmuxPlexerController("%1") + + def tearDown(self): + self.tmp.cleanup() + + def answer_file(self, content): + path = os.path.join(self.tmp.name, "ans") + with open(path, "w", encoding="utf-8") as handle: + handle.write(content) + return path + + def test_choice_builds_menu_and_reads_index(self): + path = self.answer_file("2") + calls = [] + screen = {"kind": "choice", "title": "T", "text": "Pick", "options": ["a", "b", "c"]} + with mock.patch.object(self.controller, "ask_result_path", return_value=path): + with mock.patch.object(tai.subprocess, "run", side_effect=lambda argv, **kw: calls.append(argv) or mock.Mock(returncode=0, stdout="", stderr="")): + reply = self.controller.ask_choice(screen, 30) + self.assertEqual(reply, {"status": "ok", "value": 2}) + self.assertEqual(calls[0][:3], ["tmux", "display-menu", "-T"]) + self.assertIn("T", calls[0]) + self.assertTrue(any("run-shell" in item for item in calls[0])) + self.assertFalse(os.path.exists(path)) + + def test_choice_cancelled_without_file(self): + missing = os.path.join(self.tmp.name, "missing") + screen = {"kind": "choice", "title": "", "text": "P", "options": ["a"]} + with mock.patch.object(self.controller, "ask_result_path", return_value=missing): + with mock.patch.object(self.controller, "await_answer", return_value=None): + with mock.patch.object(tai.subprocess, "run", return_value=mock.Mock(returncode=0, stdout="", stderr="")): + self.assertEqual(self.controller.ask_choice(screen, 30), {"status": "cancelled"}) + + def test_choice_bad_index_cancels(self): + path = self.answer_file("bogus") + screen = {"kind": "choice", "title": "", "text": "P", "options": ["a"]} + with mock.patch.object(self.controller, "ask_result_path", return_value=path): + with mock.patch.object(tai.subprocess, "run", return_value=mock.Mock(returncode=0, stdout="", stderr="")): + self.assertEqual(self.controller.ask_choice(screen, 30), {"status": "cancelled"}) + + def test_choice_stderr_failure_raises(self): + screen = {"kind": "choice", "title": "", "text": "P", "options": ["a"]} + with mock.patch.object(self.controller, "ask_result_path", return_value=os.path.join(self.tmp.name, "x")): + with mock.patch.object(tai.subprocess, "run", return_value=mock.Mock(returncode=1, stdout="", stderr="no current client")): + with self.assertRaises(tai.PlexerError): + self.controller.ask_choice(screen, 30) + + def test_choice_nonzero_without_stderr_cancels(self): + screen = {"kind": "choice", "title": "", "text": "P", "options": ["a"]} + with mock.patch.object(self.controller, "ask_result_path", return_value=os.path.join(self.tmp.name, "x")): + with mock.patch.object(tai.subprocess, "run", return_value=mock.Mock(returncode=1, stdout="", stderr="")): + self.assertEqual(self.controller.ask_choice(screen, 30), {"status": "cancelled"}) + + def test_choice_timeout_dismisses(self): + screen = {"kind": "choice", "title": "", "text": "P", "options": ["a"]} + with mock.patch.object(self.controller, "ask_result_path", return_value=os.path.join(self.tmp.name, "x")): + with mock.patch.object(self.controller, "ask_dismiss") as dismiss: + with mock.patch.object(tai.subprocess, "run", side_effect=subprocess.TimeoutExpired("tmux", 1)): + self.assertEqual(self.controller.ask_choice(screen, 30), {"status": "timeout"}) + self.assertEqual(dismiss.call_count, 1) + + def test_text_popup_reads_answer_and_cleans_up(self): + path = self.answer_file("hello") + real_mkstemp = tempfile.mkstemp + + def staged(prefix="", suffix=""): + return real_mkstemp(prefix=prefix, suffix=suffix, dir=self.tmp.name) + + screen = {"kind": "text", "title": "T", "text": "Name?"} + calls = [] + with mock.patch.object(tempfile, "mkstemp", side_effect=staged): + with mock.patch.object(self.controller, "ask_result_path", return_value=path): + with mock.patch.object(tai.subprocess, "run", side_effect=lambda argv, **kw: calls.append(argv) or mock.Mock(returncode=0, stdout="", stderr="")): + reply = self.controller.ask_text(screen, 30) + self.assertEqual(reply, {"status": "ok", "value": "hello"}) + self.assertIn("display-popup", calls[0]) + self.assertIn("-E", calls[0]) + self.assertTrue(any("read -r" in item for item in calls[0])) + self.assertEqual(os.listdir(self.tmp.name), []) + + def test_text_truncates_long_answers(self): + path = self.answer_file("x" * (tai.PLEXER_FLOW_ANSWER_MAX + 100)) + screen = {"kind": "text", "title": "", "text": "T"} + with mock.patch.object(self.controller, "ask_result_path", return_value=path): + with mock.patch.object(tai.subprocess, "run", return_value=mock.Mock(returncode=0, stdout="", stderr="")): + reply = self.controller.ask_text(screen, 30) + self.assertEqual(len(reply["value"]), tai.PLEXER_FLOW_ANSWER_MAX) + + def test_text_cancelled_and_timeout(self): + screen = {"kind": "text", "title": "", "text": "T"} + with mock.patch.object(self.controller, "ask_popup", return_value=None): + self.assertEqual(self.controller.ask_text(screen, 30), {"status": "cancelled"}) + with mock.patch.object(self.controller, "ask_popup", return_value="timeout"): + self.assertEqual(self.controller.ask_text(screen, 30), {"status": "timeout"}) + + def test_message_ok_and_cancelled(self): + screen = {"kind": "message", "title": "", "text": "hi"} + with mock.patch.object(self.controller, "ask_popup", return_value="ok"): + self.assertEqual(self.controller.ask_message(screen, 30), {"status": "ok", "value": True}) + with mock.patch.object(self.controller, "ask_popup", return_value=None): + self.assertEqual(self.controller.ask_message(screen, 30), {"status": "cancelled"}) + + def test_ask_rejects_unknown_kind(self): + with self.assertRaises(tai.PlexerError): + self.controller.ask({"kind": "slider"}, 30) + + def test_await_answer_reads_or_gives_up(self): + path = self.answer_file("v") + self.assertEqual(self.controller.await_answer(path), "v") + self.assertIsNone(self.controller.await_answer(os.path.join(self.tmp.name, "missing"), budget=0)) + + def test_ask_dismiss_swallows_failures(self): + with mock.patch.object(tai.subprocess, "run", side_effect=OSError("no tmux")): + self.controller.ask_dismiss() + + +class NativeAskTests(unittest.TestCase): + def make(self): + return tai.NativePlexerController("/tmp/x.sock", "3") + + def test_show_poll_done(self): + controller = self.make() + payloads = [] + + def fake(payload): + payloads.append(payload) + if payload["cmd"] == "show": + return {"id": 7} + if len([item for item in payloads if item["cmd"] == "result"]) < 2: + return {"done": False} + return {"done": True, "status": "ok", "value": 2} + + screen = {"kind": "choice", "title": "", "text": "P", "options": ["a"]} + with mock.patch.object(controller, "_call", side_effect=fake): + with mock.patch.object(tai.time, "sleep"): + reply = controller.ask(screen, 30) + self.assertEqual(reply, {"status": "ok", "value": 2}) + self.assertEqual(payloads[0]["screen"], screen) + self.assertEqual(payloads[0]["timeout"], 30) + + def test_cancelled_and_timeout_map(self): + for status in ("cancelled", "timeout"): + controller = self.make() + + def fake(payload, status=status): + if payload["cmd"] == "show": + return {"id": 1} + return {"done": True, "status": status} + + with mock.patch.object(controller, "_call", side_effect=fake): + with mock.patch.object(tai.time, "sleep"): + self.assertEqual(controller.ask({"kind": "message"}, 30), {"status": status}) + + def test_detached_raises(self): + controller = self.make() + + def fake(payload): + if payload["cmd"] == "show": + return {"id": 1} + return {"done": True, "status": "detached"} + + with mock.patch.object(controller, "_call", side_effect=fake): + with mock.patch.object(tai.time, "sleep"): + with self.assertRaises(tai.PlexerError): + controller.ask({"kind": "message"}, 30) + + def test_show_refusal_propagates(self): + controller = self.make() + with mock.patch.object(controller, "_call", side_effect=tai.PlexerError("busy")): + with self.assertRaises(tai.PlexerError): + controller.ask({"kind": "message"}, 30) + + def test_driver_timeout_cancels(self): + controller = self.make() + payloads = [] + + def fake(payload): + payloads.append(payload) + if payload["cmd"] == "show": + return {"id": 4} + if payload["cmd"] == "cancel": + return True + return {"done": False} + + with mock.patch.object(controller, "_call", side_effect=fake): + with mock.patch.object(tai.time, "sleep"): + with mock.patch.object(tai.time, "monotonic", side_effect=[100.0, 200.0]): + reply = controller.ask({"kind": "message"}, 30) + self.assertEqual(reply, {"status": "timeout"}) + self.assertEqual(payloads[-1], {"cmd": "cancel", "id": 4}) + + +class PlexerShowToolTests(unittest.TestCase): + def test_gate_hides_without_session_shows_with_tmux_pane(self): + with mock.patch.dict(os.environ, {}, clear=True): + self.assertFalse(tai.tool_env_ok("plexer_show")) + with mock.patch.dict(os.environ, {"TMUX_PANE": "%1"}, clear=True): + self.assertTrue(tai.tool_env_ok("plexer_show")) + + def test_schema_registered(self): + schemas = {schema["function"]["name"]: schema["function"] for schema in tai.TOOL_SCHEMAS} + self.assertIn("spec", schemas["plexer_show"]["parameters"]["properties"]) + self.assertIn("timeout", schemas["plexer_show"]["parameters"]["properties"]) + self.assertEqual(schemas["plexer_show"]["parameters"].get("required"), ["spec"]) + + def test_invalid_spec_is_loud(self): + tools = tai.Tools(FakeApp()) + self.assertIn("must be a JSON object", tools.dispatch("plexer_show", {"spec": []})) + self.assertIn("invalid flow spec", tools.dispatch("plexer_show", {"spec": {"screens": []}})) + + def test_timeout_override_validated(self): + tools = tai.Tools(FakeApp()) + out = tools.dispatch("plexer_show", {"spec": flow_spec(), "timeout": "soon"}) + self.assertIn("must be an integer", out) + out = tools.dispatch("plexer_show", {"spec": flow_spec(), "timeout": 5 * 3600}) + self.assertIn("timeout must be", out) + + def test_refused_in_auto_mode_and_for_workers(self): + auto = FakeApp() + auto.auto = True + self.assertIn("autonomous mode", tai.Tools(auto).dispatch("plexer_show", {"spec": flow_spec()})) + deep = FakeApp() + deep.depth = 1 + self.assertIn("background workers", tai.Tools(deep).dispatch("plexer_show", {"spec": flow_spec()})) + flat = FakeApp() + flat.persist = False + self.assertIn("background workers", tai.Tools(flat).dispatch("plexer_show", {"spec": flow_spec()})) + + def test_success_returns_outcome_and_marks_step_in(self): + app = FakeApp() + tools = tai.Tools(app) + controller = ScriptedController([{"status": "ok", "value": 1}, {"status": "ok", "value": 0}]) + with mock.patch.object(tai.PlexerControllerFactory, "detect", return_value=controller): + out = tools.dispatch("plexer_show", {"spec": flow_spec()}) + outcome = json.loads(out) + self.assertEqual(outcome, {"status": "done", "answers": {"env": "b"}, "action": "ok"}) + self.assertTrue(app.user_stepped_in) + + def test_timeout_override_reaches_asks(self): + tools = tai.Tools(FakeApp()) + controller = ScriptedController([{"status": "ok", "value": 0}, {"status": "ok", "value": 0}]) + with mock.patch.object(tai.PlexerControllerFactory, "detect", return_value=controller): + tools.dispatch("plexer_show", {"spec": flow_spec(timeout=600), "timeout": 60}) + for _screen, budget in controller.asked: + self.assertGreater(budget, 0) + self.assertLessEqual(budget, 60) + + def test_backend_failure_is_loud(self): + tools = tai.Tools(FakeApp()) + controller = ScriptedController([tai.PlexerError("no attached client")]) + with mock.patch.object(tai.PlexerControllerFactory, "detect", return_value=controller): + out = tools.dispatch("plexer_show", {"spec": flow_spec()}) + self.assertIn("no attached client", out) + + def test_lazy_loads_on_ui_words_only_with_session(self): + with mock.patch.dict(os.environ, {"TMUX_PANE": "%1"}, clear=True): + names = {schema["function"]["name"] for schema in tai.select_tools("show a dialog flow on screen")} + self.assertIn("plexer_show", names) + with mock.patch.dict(os.environ, {}, clear=True): + names = {schema["function"]["name"] for schema in tai.select_tools("show a dialog flow on screen")} + self.assertNotIn("plexer_show", names) + + +class UiflowDocTests(unittest.TestCase): + """uiflow.md is the flow contract; it must name everything the code enforces.""" + + @classmethod + def setUpClass(cls): + with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "uiflow.md"), "r", encoding="utf-8") as handle: + cls.doc = handle.read() + + def test_every_cap_is_documented(self): + for name in ("PLEXER_FLOW_VERSION", "PLEXER_FLOW_SCREENS_MAX", "PLEXER_FLOW_FIELDS_MAX", "PLEXER_FLOW_OPTIONS_MAX", + "PLEXER_FLOW_ACTIONS_MAX", "PLEXER_FLOW_VISITS_MAX", "PLEXER_FLOW_ID_MAX", "PLEXER_FLOW_TITLE_MAX", + "PLEXER_FLOW_TEXT_MAX", "PLEXER_FLOW_LABEL_MAX", "PLEXER_FLOW_ANSWER_MAX", "PLEXER_FLOW_TIMEOUT_MIN", + "PLEXER_FLOW_TIMEOUT_DEFAULT", "PLEXER_FLOW_TIMEOUT_MAX", "PLEXER_FLOW_POLL", + "MPLEX_FLOW_MAX", "MPLEX_FLOW_KEEP"): + self.assertIn("`%s`" % name, self.doc, name) + self.assertTrue(hasattr(tai, name), name) + + def test_contract_words_are_documented(self): + for word in ("choice", "text", "info", "message", "show_if", "$done", "$cancel", "next", "actions", "cancelled", "timeout"): + self.assertIn(word, self.doc, word) + + def test_no_em_dashes(self): + self.assertNotIn("—", self.doc) + + class ShellSessionTests(unittest.TestCase): BTIME = 1700000000.0 BOOT_ID = "12345678-9abc-def0-1234-56789abcdef0\n" @@ -11385,7 +12682,9 @@ class GrrShellTests(GrrThemeIsolation, unittest.TestCase): def test_the_chat_shows_every_kind_of_message(self): app, shell = self.make(130, 70) - text = self.go(shell, app, "/chat/c1") + self.go(shell, app, "/chat/c1") + shell.close_dialog(shell.view.prompts[3], None) + text = self.frame(app) for word in ("Run echo hello", "echo hello", "hello", "Tai needs your approval", "Allow once", "Deny", "Which one?", "Red", "Blue", "report.txt", "It printed", "code here", "Steps (1)"): self.assertIn(word, text) @@ -11429,6 +12728,30 @@ class GrrShellTests(GrrThemeIsolation, unittest.TestCase): question.controls[1][1]() self.assertEqual(self.server.seen("POST", "/api/messages/4/answer")[0][2], {"text": "Blue"}) + def test_a_pending_approval_is_prompted_as_a_dialog_and_answered_from_it(self): + app, shell = self.make(120, 70) + self.go(shell, app, "/chat/c1") + view = shell.view + self.assertIn(3, view.prompts) + text = self.frame(app) + self.assertEqual(text.count("Allow once"), 2) + dlg = view.prompts[3] + shell.dialog_pick(dlg, next(a for a in dlg.grr_actions if a.get("primary"))) + self.assertEqual(self.server.seen("POST", "/api/messages/3/decision")[-1][2], {"decision": "allow"}) + self.assertNotIn(3, view.prompts) + + def test_a_live_pending_approval_prompts_and_a_resolved_one_closes_the_prompt(self): + app, shell = self.make(120, 70) + self.go(shell, app, "/chat/c1") + view = shell.view + shell.close_dialog(view.prompts[3], None) + live = {"id": 60, "chat_id": "c1", "role": "assistant", "kind": "approval", "sender": "b1", "content": "Run ls", "meta": {"state": "pending", "tool": "shell"}, "ts": time.time()} + shell.on_event("message.new", live) + self.assertIn(60, view.prompts) + shell.on_event("message.update", dict(live, meta={"state": "allowed", "tool": "shell"})) + self.assertNotIn(60, view.prompts) + self.assertEqual(len(shell.dialogs), 0) + def test_a_pending_approval_in_another_state_has_no_buttons(self): app, shell = self.make(120, 70) self.go(shell, app, "/chat/c1") @@ -11538,6 +12861,34 @@ class GrrShellTests(GrrThemeIsolation, unittest.TestCase): finally: self.server.get.pop("/api/chats/c2/messages", None) + def test_a_running_command_that_stops_on_a_prompt_opens_its_terminal(self): + import base64 as b64 + running = {"id": 7, "chat_id": "c2", "role": "assistant", "kind": "process", "sender": "b2", "content": "sudo ls", "meta": {"pid": "p9", "state": "running", "cols": 80, "rows": 24}, "ts": 1792257000.0} + self.server.get["/api/chats/c2/messages"] = {"messages": [running], "has_more": False} + try: + app, shell = self.make(120, 40) + with mock.patch.object(tai.GrrStream, "start", lambda self: None): + self.go(shell, app, "/chat/c2") + item = shell.view.items[7] + opened = [] + item.type_in = lambda: opened.append(1) + with mock.patch.object(app, "after", lambda delay, fn: None): + item.on_event("out", {"d": b64.b64encode(b"building\r\ndone").decode()}) + item.check_prompt(item.gen) + self.assertEqual(opened, []) + item.on_event("out", {"d": b64.b64encode(b"\r\n[sudo] password for retoor: ").decode()}) + item.check_prompt(item.gen - 1) + self.assertEqual(opened, []) + item.check_prompt(item.gen) + self.assertEqual(opened, [1]) + item.check_prompt(item.gen) + self.assertEqual(opened, [1]) + item.on_event("out", {"d": b64.b64encode(b"\r\nContinue? [y/N] ").decode()}) + item.check_prompt(item.gen) + self.assertEqual(opened, [1, 1]) + finally: + self.server.get.pop("/api/chats/c2/messages", None) + def test_the_computer_terminal_sends_keys_and_leaves_on_ctrl_bracket(self): app, shell = self.make(120, 40) left = [] diff --git a/tests/plexer.py b/tests/plexer.py index 00dcb2b..49059d3 100644 --- a/tests/plexer.py +++ b/tests/plexer.py @@ -11,7 +11,8 @@ the whole point of this file is that it exercises the actual testing a pty wrapper by driving it through another pty), not mocked termios/signal calls. The multiplexer's shortcuts and config live in tests/mplex.py; this file covers what the agent tools rely on: the control socket, per-pane -targeting, file permissions and the session log. +targeting, file permissions, the session log, and the UI flow dialogs +(show/result/cancel) with a real attached client answering real keystrokes. A real bug was caught by actually running the original version of this test, not by reading the code: an external SIGTERM (and SIGHUP) does not terminate an @@ -380,7 +381,113 @@ class PlexerPaneTargetingTest(TestCase): shutil.rmtree(self.tmp_home, ignore_errors=True) -ALL_CASES = [NativePlexerBackendTest(), NativePlexerBackendRobustnessTest(), PlexerPaneTargetingTest()] +class PlexerFlowNativeTest(TestCase): + """The UI flow path end to end over a real server and a real attached + client: show renders a dialog, keystrokes written to the pty master answer + it, result collects the answer. Deterministic plumbing like the rest of + this file: no model anywhere, the mechanics are what's under test.""" + + name = "plexer-flow-native" + mode = "pty" + timeout = 60.0 + + async def setup(self): + self.tmp_home = tempfile.mkdtemp(prefix="tai-test-home-") + self.env_overrides = {"TAI_HOME": self.tmp_home, "TAI_PASSPHRASE": "", "TERM": "xterm-256color", "SHELL": "/bin/bash"} + + async def collect(self, controller, flow_id, budget=4.0): + for _ in range(int(budget / 0.1)): + reply = controller._call({"cmd": "result", "id": flow_id}) + if reply.get("done"): + return reply + await asyncio.sleep(0.1) + raise TestAssertionFailed("flow %r never settled" % (flow_id,)) + + async def run(self): + proc = await self.spawn(extra_args=["--plexer-wrap"]) + self._drain_task = asyncio.ensure_future(_drain_forever(proc)) + sock_path = os.path.join(self.tmp_home, "plexer", tai.MPLEX_SOCKET_NAME) + for _ in range(50): + if os.path.exists(sock_path): + break + await asyncio.sleep(0.2) + else: + raise TestAssertionFailed("the mplex server never created its control socket") + controller = tai.NativePlexerController(sock_path) + await asyncio.sleep(0.8) + keys = tai.MplexKeys.to_bytes + + shown = controller._call({"cmd": "show", "screen": {"kind": "choice", "title": "T", "text": "Pick one", + "options": ["alpha", "beta"]}, "timeout": 10}) + pending = controller._call({"cmd": "result", "id": shown["id"]}) + await asyncio.sleep(0.3) + os.write(proc._master_fd, keys("Down")) + await asyncio.sleep(0.3) + os.write(proc._master_fd, keys("Enter")) + choice = await self.collect(controller, shown["id"]) + + shown_text = controller._call({"cmd": "show", "screen": {"kind": "text", "title": "T", "text": "Name?"}, "timeout": 10}) + await asyncio.sleep(0.3) + os.write(proc._master_fd, b"hi" + keys("Enter")) + text = await self.collect(controller, shown_text["id"]) + + shown_msg = controller._call({"cmd": "show", "screen": {"kind": "message", "title": "", "text": "hi"}, "timeout": 10}) + await asyncio.sleep(0.3) + os.write(proc._master_fd, b"x") + message = await self.collect(controller, shown_msg["id"]) + + shown_cancel = controller._call({"cmd": "show", "screen": {"kind": "message", "title": "", "text": "hi"}, "timeout": 10}) + await asyncio.sleep(0.3) + os.write(proc._master_fd, keys("Escape")) + cancelled = await self.collect(controller, shown_cancel["id"]) + + stuck = controller._call({"cmd": "show", "screen": {"kind": "message", "title": "", "text": "stuck"}, "timeout": 10}) + busy = None + try: + controller._call({"cmd": "show", "screen": {"kind": "message", "title": "", "text": "second"}, "timeout": 10}) + except tai.PlexerError as exc: + busy = str(exc) + controller._call({"cmd": "cancel", "id": stuck["id"]}) + dropped = None + try: + controller._call({"cmd": "result", "id": stuck["id"]}) + except tai.PlexerError as exc: + dropped = str(exc) + + asked = asyncio.ensure_future(asyncio.to_thread( + controller.ask, {"kind": "choice", "title": "", "text": "P", "options": ["a"]}, 10)) + await asyncio.sleep(0.5) + os.write(proc._master_fd, keys("Enter")) + ask_outcome = await asyncio.wait_for(asked, timeout=10.0) + + return {"pending": pending, "choice": choice, "text": text, "message": message, "cancelled": cancelled, + "busy": busy, "dropped": dropped, "ask_outcome": ask_outcome} + + async def verify(self, result): + if result["pending"].get("done") is not False: + raise TestAssertionFailed("a fresh show must poll as not done: %r" % result["pending"]) + if result["choice"] != {"done": True, "status": "ok", "value": 1}: + raise TestAssertionFailed("Down+Enter on [alpha, beta] must answer index 1: %r" % result["choice"]) + if result["text"] != {"done": True, "status": "ok", "value": "hi"}: + raise TestAssertionFailed("typing hi+Enter must answer 'hi': %r" % result["text"]) + if result["message"] != {"done": True, "status": "ok", "value": True}: + raise TestAssertionFailed("any key must ack a message: %r" % result["message"]) + if result["cancelled"].get("status") != "cancelled": + raise TestAssertionFailed("Escape must cancel the dialog: %r" % result["cancelled"]) + if not result["busy"] or "busy" not in result["busy"]: + raise TestAssertionFailed("a second show on a busy client must refuse loudly: %r" % result["busy"]) + if not result["dropped"] or "unknown flow" not in result["dropped"]: + raise TestAssertionFailed("result after cancel must report an unknown flow: %r" % result["dropped"]) + if result["ask_outcome"] != {"status": "ok", "value": 0}: + raise TestAssertionFailed("controller.ask must poll to the same answer: %r" % result["ask_outcome"]) + + async def teardown(self): + self._drain_task.cancel() + _mplex_cli(self, "kill-server") + shutil.rmtree(self.tmp_home, ignore_errors=True) + + +ALL_CASES = [NativePlexerBackendTest(), NativePlexerBackendRobustnessTest(), PlexerPaneTargetingTest(), PlexerFlowNativeTest()] if __name__ == "__main__": sys.exit(main_cli(ALL_CASES, default_log_dir=str(Path(__file__).resolve().parent / "logs"))) diff --git a/uiflow.md b/uiflow.md new file mode 100644 index 0000000..2698d8b --- /dev/null +++ b/uiflow.md @@ -0,0 +1,169 @@ +# uiflow: interactive UI flows on the user's terminal + +Specification of the `plexer_show` agent tool and the JSON contract behind it. +The code is `validate_flow_spec`, `PlexerFlow` and `PlexerController.ask` in +`tai.py`; `PlexerFlowSpecTests`, `PlexerFlowDriverTests`, `MplexFlowServerTests`, +`TmuxAskTests`, `NativeAskTests`, `PlexerShowToolTests` and `UiflowDocTests` in +`test_tai.py` check the contract against that code, so this page cannot +silently drift. + +## What it is + +One tool call renders a whole interactive app (dialogs, forms, branching +multi-screen flows) on the terminal the user is looking at, and returns the +outcome as JSON. Three layers, each dumber than the one above: + +- **Main agent**: makes a single `plexer_show` call with the spec, gets back + `{status, answers, action}`. Its turn stays lean no matter how complex the + flow gets: it never sees intermediate screens or keypresses. +- **Flow driver** (`PlexerFlow`, the separate agent that owns the interaction): + validates the spec, decomposes screens into one-screen `ask` calls, + evaluates `show_if` conditions and `next` branches locally, enforces the + timeout and the visit cap, and reports only the final outcome. +- **Backends** (`PlexerController.ask`): dumb renderers of one screen at a + time. Native mplex shows the screen as an overlay or status prompt on the + attached client (control socket `show`/`result`/`cancel`, see + [mplex.md](mplex.md)); tmux maps it onto `display-menu` (choice) and + `display-popup` (text, message). Both answer the same contract: `ok` with a + value, `cancelled`, or `timeout`. + +## Spec format (v1) + +```json +{ + "v": 1, + "title": "Deploy", + "timeout": 300, + "start": "main", + "screens": [ + {"id": "main", "title": "Where", "body": "Pick a target.", + "fields": [ + {"id": "env", "type": "choice", "label": "Environment", + "options": ["staging", "prod"], "default": 0}, + {"id": "tag", "type": "text", "label": "Tag", "initial": "v1", + "show_if": {"env": "prod"}}, + {"id": "note", "type": "info", "text": "Prod deploys page the team."} + ], + "actions": [{"id": "go", "label": "Deploy"}, {"id": "back", "label": "Back"}], + "next": {"go": "$done", "back": "$cancel"}} + ] +} +``` + +Top level: `v` must be 1 (missing means 1, anything else is rejected); +`title` is the fallback for screens without one; `timeout` is the whole-flow +budget in seconds (5 to 600, default 300); `start` is the first screen id +(default: the first screen); `screens` is a non-empty list (at most 32). + +Screen: `id` (unique, no whitespace); `title`, `body` (plain text); `fields` +(at most 12); `actions` (at most 12, each with unique `id` and non-empty +`label`); `next` (a screen id, an `{action: screen}` map, `$done` or `$cancel`, +default: next screen in order, `$done` after the last); `show_if` (skip the +screen unless it holds). A screen with no body, fields or actions is rejected. + +Field types: + +- `choice`: single select. `options` is 1 to 12 strings or + `{label, value}` objects; values must be JSON scalars (string, number, + boolean, null). `default` preselects an option index. The answer recorded is + the option's `value`. +- `text`: free input. `initial` is the starting text. The answer is the typed + string (capped at 2000 chars). +- `info`: display only, never asked, never recorded. Shown with the screen's + context (see below). + +Conditions: `show_if` is `{field id: expected}`, where expected is a scalar or +a list of accepted scalars. Every clause must hold. A field that has not been +answered yet never holds. Referring to an unknown field id is rejected, so a +typo fails loudly instead of silently skipping. + +Ids (screens, fields, actions) are unique per scope (fields: whole app), +non-empty, at most 64 chars, no whitespace. + +## Flow semantics + +Screens run in order from `start`. A screen whose `show_if` fails is skipped +(its `next` still applies as if it had no actions). Otherwise each visible +choice/text field is asked in turn, then the action when the screen has +actions. A screen with only body/info and no actions is shown once and waits +for acknowledgement. + +`next` routing: a string applies to the whole screen; a map routes per taken +action, and an action missing from the map falls through to the next screen in +order (`$done` after the last). `$done` ends with status `done`, `$cancel` +ends with status `cancelled`. Every target must exist (unknown ids are +rejected upfront); at most 64 screen visits per run, more aborts with status +`error` (a loop guard, not a feature). + +Outcomes, always one JSON object: + +- `{status: done, answers: {field: value}, action}`: the user finished the + flow. `action` is the last taken action id, or null when no screen had + actions. +- `{status: cancelled, at, answers}`: the user pressed Escape (or `q`, `C-c`) + on a screen, or the flow routed to `$cancel`. `answers` holds what was + answered so far. +- `{status: timeout, at, answers}`: the flow budget ran out. +- `{status: error, error, at, answers}`: the visit cap tripped. + +Escape anywhere aborts the whole flow; there is no per-screen back button in +v1 (route an explicit Back action where you need one). + +## Screen decomposition + +Backends render one screen at a time, so the driver asks a screen as a short +sequence: one `ask` per visible choice/text field, then one for the action. +The first ask carries the context (screen title, body, visible info texts, +field label); later asks show just their label. Each ask gets the remaining +flow budget as its timeout. Backend answers are indices and strings; the +driver maps them back to option values and evaluates the flow. + +## Backends + +Native (mplex): choice and message render as a centered overlay dialog +(`MplexFlowScreen`: arrows/`j`/`k`, Enter, digits, Escape); text renders as a +status-row prompt with emacs editing. Wire: `show` renders and returns a flow +id, `result` polls (`{done: false}` until answered), `cancel` drops it. One +pending dialog per client; a busy client refuses loudly, as does a pane whose +session has no attached client. Server caps: `MPLEX_FLOW_MAX` pending flows, +`MPLEX_FLOW_KEEP` seconds of grace for dead drivers. Details: +[mplex.md](mplex.md) (control socket). + +tmux: choice renders as `display-menu` (each option writes its index to a +staged answer file via `run-shell`), text and message as `display-popup` +running `read` (no file means dismissed, an empty file means an empty answer). +Staging files live under the system temp dir as `tai-flow-*.prompt/answer` +and are removed after each ask. Two known differences from native: `default` +does not preselect (tmux menus always start at the first item) and `initial` +is ignored (no portable prefill for `read`). After a timeout the driver closes +popups server-side (`display-popup -C`); a `display-menu` has no server-side +close and stays until the user presses Escape, which is harmless because the +flow is already over. + +## Hard caps + +| Constant | Value | Limits | +|---|---|---| +| `PLEXER_FLOW_VERSION` | 1 | the only accepted spec version | +| `PLEXER_FLOW_SCREENS_MAX` | 32 | screens per spec | +| `PLEXER_FLOW_FIELDS_MAX` | 12 | fields per screen | +| `PLEXER_FLOW_OPTIONS_MAX` | 12 | options per choice, and per ask screen | +| `PLEXER_FLOW_ACTIONS_MAX` | 12 | actions per screen | +| `PLEXER_FLOW_VISITS_MAX` | 64 | screen visits per run (loop guard) | +| `PLEXER_FLOW_ID_MAX` | 64 | chars per screen/field/action id | +| `PLEXER_FLOW_TITLE_MAX` | 200 | chars per title | +| `PLEXER_FLOW_TEXT_MAX` | 2000 | chars per body/info/ask text | +| `PLEXER_FLOW_LABEL_MAX` | 200 | chars per field/action/option label | +| `PLEXER_FLOW_ANSWER_MAX` | 2000 | chars per text answer and initial | +| `PLEXER_FLOW_TIMEOUT_MIN` / `PLEXER_FLOW_TIMEOUT_DEFAULT` / `PLEXER_FLOW_TIMEOUT_MAX` | 5 / 300 / 600 | flow budget in seconds | +| `PLEXER_FLOW_POLL` | 0.25 s | native result poll interval | +| `MPLEX_FLOW_MAX` | 16 | pending flows per mplex server | +| `MPLEX_FLOW_KEEP` | 300 s | grace before a dead driver's record is reclaimed | + +## Gaps + +Single-screen rich forms (all fields visible at once with focus navigation), +multi-select choices, progress/live screens, and per-screen back navigation do +not exist in v1; the driver decomposes every screen into sequential asks, +which is the documented behavior, not a fallback. The spec carries a version +precisely so these can extend the contract without breaking v1 specs.