retoorandClaude Sonnet 5
9dbd1fe667
- Plexer: always-active terminal multiplexing. NativePlexerController/ TmuxPlexerController share one interface, chosen by which env var is already set (TAI_PLEXER_SOCKET vs TMUX_PANE); NativePlexerBackend is the pty.fork()-based wrapper (raw mode, SIGWINCH resize forwarding, a Unix-socket control protocol, continuous session logging, a Ctrl+A prefix key distinct from tmux's own Ctrl+B). Auto-wraps new interactive shells via a .bashrc hook mirroring the existing command-not-found install pattern, gated on $TMUX/$TAI_PLEXER_ACTIVE/ a $HOME/.tai-no-plexer bypass. New plexer_send/read/type/status tools, env-gated on a detected controller. - tests/container.py + docker-compose.yml + tests/docker/Dockerfile: a disposable podman/docker image for the tests that must never touch a real machine (installing the plexer hook into a real ~/.bashrc, a live tmux session). Required suite (test_seal.py, test_tai.py, tests/plexer.py, tests/bashrc.py) gates the result; tests/rag.py runs as best-effort since its real-LLM dependency can't be satisfied inside a disposable container with no nested engine. - tests/bashrc.py: real end-to-end hook install + bash -i auto-wrap, anti-recursion, tmux-active skip, non-interactive skip, a live TmuxPlexerController test, and an install/uninstall round-trip. - tests/plexer.py: NativePlexerBackend under a real pty, plus a dedicated robustness pass (file permissions, >64KB payload framing, a malformed control request, a genuine self-deadlock fix in the write-to-child path). - tai_test.py: the shared async process-driving framework (pipe/pty transport, group-wide kill-and-reap, a StdinBridge for --interactive) both test files build on. - ChatClient last-resort backend: when there's no local opencode CLI but the sandbox box is already running, install opencode into it (official Node tarball, not apt -- apt's nodejs/npm pull in webpack/babel/jest as transitive deps) and route through it instead of giving up. Only reachable after the first retry round already failed everything else; never creates the box itself. TmuxPlexerController.status() fixed to use `list-panes` instead of `display-message -p`, which silently fell back to the default pane instead of erroring against a dead target. - Fix 4 pre-existing test_tai.py bugs surfaced by running it in a clean environment for the first time: two stale mock signatures (subprocess.run's stdout= kwarg, Chat.complete's deadline= kwarg), one $TMUX-dependent catalog assertion, one tty-dependent color assertion. - research/: a sourced research archive (Source Dossier/Citation Card/Conclusion tiers) backing the above design decisions, with its own audit.py consistency checker. - .gitignore: fix a blanket *.md rule that was about to silently drop the entire research/ archive from version control; add __pycache__/ and the one-off debug transcript this session started from. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
52 lines
4.0 KiB
Markdown
52 lines
4.0 KiB
Markdown
# S16 follow-up: three open surfaces of the last_downloaded_at write-amplification argument
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Institute | Molodetz Institute — Research & Retrieval Division |
|
|
| Subject | agent-memory-recency-tracking |
|
|
| Document type | Source Dossier |
|
|
| Tier | 1 |
|
|
| Source title | S16 follow-up: three open surfaces of the last_downloaded_at write-amplification argument |
|
|
| Source URL | https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/680 |
|
|
| Source publisher / author | GitLab (internal engineering work item, publicly visible) |
|
|
| Source type | other (production engineering issue tracker entry) |
|
|
| Source date (published) | Unknown — not stated (exact work item date not captured by this research pass) |
|
|
| Date researched | 2026-10-09 |
|
|
| Researched by | tai (agent), model: Claude Sonnet 5, on behalf of retoor |
|
|
| Research method | web search synthesis |
|
|
| Confidence | High |
|
|
| Status | Active |
|
|
|
|
## Summary
|
|
|
|
A real, named production incident write-up from GitLab's own artifact registry engineering work, documenting the exact failure mode this subject's conclusion is designed to avoid: an unconditional "touch" update on a hot read path causing serious database write amplification — with a concrete, quantified load estimate and GitLab's own fix.
|
|
|
|
## Key findings
|
|
|
|
- The `last_downloaded_at` column in question is part of a btree index, so updating it cannot be satisfied by a heap-only-tuple (HOT) update — every bump therefore creates a new row version *and* a new entry in every index the row belongs to.
|
|
- Concrete load estimate given: a 10-layer container image pulled 100 times a minute, with every layer resolving to the same underlying row, works out to roughly 17 row versions per second plus 51 index entries per second on that one row — from what looks like "just incrementing a timestamp."
|
|
- GitLab's actual fix was a throttled update: the write is skipped based on the freshness of the row's own existing timestamp, so repeated reads within the throttle window are no-ops at the database layer.
|
|
- The work item notes one of the two write routes was throttled this way while a second (blob reads, which have no manifest row to check against) was not yet fixed at time of writing — i.e. even GitLab's own fix was partial/in-progress, not a universally complete solution captured in this source.
|
|
|
|
## Pros / strengths
|
|
|
|
- This is a real, quantified, named production incident — not a theoretical cache paper — making it the single most concrete evidence in this entire subject that "just add a last_used column and UPDATE it on every read" is a genuine, not hypothetical, performance risk.
|
|
- The fix GitLab actually applied (throttle based on the row's own existing timestamp) is structurally identical to the pattern this subject's conclusion recommends for tai (`UPDATE ... WHERE last_used IS NULL OR last_used < ?`).
|
|
|
|
## Cons / limitations / caveats
|
|
|
|
- This is PostgreSQL-specific in its mechanics (HOT updates, btree index versioning) — SQLite (tai's actual database) has different internals, so the exact row-version/index-entry multiplication described here does not transfer number-for-number. The general lesson (indexed, frequently-touched timestamp columns cause real write amplification) does transfer; the specific "17 row versions/sec, 51 index entries/sec" figures are PostgreSQL-specific illustrations, not a universal constant.
|
|
- The source itself documents an incomplete, in-progress fix (one route throttled, one not) — it should not be read as "GitLab solved this cleanly," only as "GitLab hit this problem and applied a partial, same-shaped fix."
|
|
|
|
## Relevance to tai
|
|
|
|
This is the primary real-world justification, cited directly in this subject's conclusion, for why tai's `last_used` column must never be updated unconditionally on every read/hit, and for why the recommended fix is a conditional/throttled UPDATE rather than a naive touch-on-every-access implementation.
|
|
|
|
## Changelog
|
|
|
|
| Date | Change |
|
|
|---|---|
|
|
| 2026-10-09 | Initial dossier created. |
|