docs: commit markdown previously hidden by the *.md ignore

2026-10-28 13:19:03 +01:00
parent 4eff597be7
commit 1ebb7c637d
13 changed files with 1617 additions and 0 deletions
@@ -0,0 +1,3 @@
+App name is model_sequence.
+
+It is an app based on the 
@@ -0,0 +1 @@
+These are all terminal apps that are convenient to use with the lead24 aopplication...
@@ -0,0 +1,21 @@
+# tai examples
+
+Complete, runnable, stdlib-only examples in tai-spirit: single file, zero deps, Python 3.6+.
+
+Elke example is direct draaibaar, geen pip, geen build. Kopieer het bestand, run het.
+
+| # | Example | Run | Wat laat het zien |
+|---|---------|-----|-------------------|
+| 01 | [WebDAV server](webdav-server.py) | `python3 webdav-server.py --root ./share --port 8080` | Complete WebDAV Class 1: PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, Range, ETag, Basic-auth, jail. Werkt met cadaver, Nautilus, rclone, curl. |
+| 02 | [Todo REST API](todo-rest-api.py) | `python3 todo-rest-api.py --port 8081` | REST JSON API + SQLite + embedded UI. Filtering, paging, validation, concurrency-safe. |
+| 03 | [SSE chat](sse-chat.py) | `python3 sse-chat.py --port 8082` | Multi-room chat met Server-Sent Events. Geen JS-deps, history, heartbeat, graceful reconnect. |
+| 04 | [Reverse proxy](reverse-proxy.py) | `python3 reverse-proxy.py --backends http://127.0.0.1:9001,http://127.0.0.1:9002` | Round-robin proxy met healthchecks, retries, hop-header hygiene, X-Forwarded-*. |
+| 05 | [FTS search](fts-search.py) | `python3 fts-search.py index ./docs && python3 fts-search.py serve --port 8083` | SQLite FTS5 full-text search over een directory. BM25 ranking, snippet highlights, HTTP UI + CLI. |
+
+Conventies (zoals tai.py):
+
+- Alleen stdlib imports.
+- Werkt op Python 3.6+.
+- `--help` op elk bestand.
+- Harde caps: request size, upload size, history lengte.
+- Veilige paden: geen traversal buiten root.
@@ -0,0 +1,6 @@
+# SPEC — detz v0.1.0
+
+moloapp pin: 12e45e8
+264 classes in ../tai.py (grep ^class). Full parity table per docs/<module>.md.
+Target tree: src/{cli,config,util,shell,seal,store,llm,agent,comms,plexer,tui,mcp,cap}.
+Build: -std=c++23 -Wall -Wextra, outputs only in build/.
@@ -0,0 +1,14 @@
+# config parity (M2)
+
+| python (tai.py) | C++ (src/config/) | notes |
+|---|---|---|
+| Consts | consts.h Consts | same values |
+| TOTAL_STEP_CAP/SYSINFO_TIMEOUT/LOOP_*/TIMEOUTs | Consts::* | — |
+| DEFAULT_MODEL/DEFAULT_VOICE/OPENROUTER_MODEL | kDefaultModel/... | — |
+| ZEN_FREE_MODELS/POLLINATIONS_MODELS/DEVPLACE_FREE_ROUTES | ZenFreeModels()/... | — |
+| Config.__init__ env mapping | Config::FromEnv | subset: home/model/keys/models/voice |
+| SETTINGS_CATALOG keys | SettingsCatalog() | 11 keys ported (subset) |
+| Config.Get/settings_row | Config::Get | scalar lookup only |
+| split_models | SplitModels | comma/semicolon split |
+| known_model_ids | KnownModelIds | config lists + extra (no health/backend) |
+| resolve_tag | ResolveTag | bots map + models + dir; no store/agent |
@@ -0,0 +1,9 @@
+# Embedding detz: research notes
+
+Options compared:
+1. **Dynamic .so plugin** (`dlopen`): flexible versioning, but host must ship/resolve the .so at runtime; symbol drift risk; packaging burden.
+2. **Subprocess/IPC** (spawn detz binary): simplest isolation, but per-call latency + serialization cost; no in-process context hook.
+3. **Static archive link (chosen)**: `build/libdetz.a` + C ABI header `src/glue/detz_embed.h` (`detz_set_context`, `detz_context`, `detz_version`) plus C++ hook `detz::embed::apply_context()`. Single artifact, no runtime deps beyond system libs, version pinned at build time. `make static` additionally yields fully static `build/detz-static` (verified `file`: statically linked, `ldd`: not a dynamic executable) using vendored `third_party/{curl,sqlite,jitterentropy}` archives. Re-verified 2026-10-27 at 1.44.3: `make static` exits 0, the binary is 29 MB, `file` says statically linked, `--version` prints `tai 1.44.3`, and the link shows only the standard glibc static warnings (`dlopen`, `getaddrinfo`, `gethostbyname`, `getpwuid`).
+4. **Header-only**: rejected — codebase too large, would explode host build times.
+
+Why this API: minimal C surface (stable ABI, callable from C/Rust/Python-ctypes) with a thin C++ `apply_context()` injection point wired into `GlueApp::one_shot`, so hosts prepend context without touching agent internals. Proven: third-party TU links against `libdetz.a` and `apply_context("ping")` → `[ctx] STATIC-PROOF\nping`, version `1.44.3`.
@@ -0,0 +1,120 @@
+# termshot (taicpp): PNG pictures of the current terminal
+
+Namespace `detz::termshot`, sources in `src/termshot/`. detz-only: tai.py has no
+equivalent. Research and decisions:
+[`../../research/terminal-screenshot-rendering/`](../../research/terminal-screenshot-rendering/).
+
+## What it does
+
+Makes a PNG of the terminal the user is looking at, on a headless server too.
+No desktop, X server, system font, browser or external binary is involved.
+
+```
+tmux capture-pane -e -p -N      (or the native multiplexer pane log tail)
+        |
+TuiVtScreen (src/tui)            replay the bytes into a rows x cols cell grid
+        |
+TermshotRenderer                 background, glyph, bold, dim, reverse, underline, strike, cursor
+        |
+TermshotPng                      8-bit RGB, filter 0, zlib deflate, zlib crc32
+        |
+<home>/shots/terminal-YYYYmmdd-HHMMSS-mmm.png
+```
+
+## Classes
+
+| Class | File | Role |
+|---|---|---|
+| `TermshotFont` | `termshot_font.*`, `termshot_font_data.cpp` | embedded Spleen 8x16 glyphs, binary search, fallback U+FFFD then `?` |
+| `TermshotPalette` | `termshot_palette.*` | SGR fragment to 0xRRGGBB: 16 xterm colors, 256 cube and gray ramp, truecolor, bold brightens 0..7 |
+| `TermshotCanvas` | `termshot_canvas.*` | RGB buffer: fill, glyph (bold doubles lit pixels right), pixel |
+| `TermshotPng` | `termshot_png.*` | encoder and IHDR size reader |
+| `TermshotFrame`, `TermshotError` | `termshot_frame.h` | captured bytes plus size, cursor, source label; failure with a tool-ready reason |
+| `TermshotCapture` | `termshot_capture.*` | `tmux(target)`, `native(raw, pane, rows, cols)`, `current(target, reader)` |
+| `TermshotRenderer` | `termshot_renderer.*` | frame to canvas |
+| `TermshotShooter`, `TermshotShot` | `termshot_shooter.*` | frame to PNG bytes to file, cap check, one-line summary |
+
+## Capture rules
+
+`TermshotCapture::current(target, reader)` resolves the pane exactly like
+`get_current_terminal_content`:
+
+1. a `target` (tmux pane id `%3` or `session:window.pane`) always means tmux;
+2. no target and `TAI_PLEXER_SOCKET` set: the native multiplexer. The agent passes
+   `ports().plexer->native_read_pane` as the reader; the last 200 lines of the raw pane
+   log are replayed into a `LINES` x `COLUMNS` screen, default 50 x 160;
+3. otherwise tmux with `TMUX_PANE`, or tmux's current pane when that is unset.
+
+tmux calls: `display-message -p '#{pane_width} #{pane_height} #{cursor_x} #{cursor_y}
+#{cursor_flag} #{pane_id}'`, then `capture-pane -e -p -N [-t target]` (visible screen,
+escapes kept, trailing spaces kept so background colors reach the right edge). Line
+ends become `\r\n` before the replay.
+
+## Font
+
+Spleen 8x16 2.2.0 by Frederic Cambus, BSD 2-Clause. The BDF and the license are
+vendored in `src/termshot/gen/`; `python3 src/termshot/gen/gen_font.py` regenerates
+`termshot_font_data.cpp` (1001 glyphs, 16 bytes each, license text in the header).
+Covers Basic Latin, Latin-1, Latin Extended-A, box drawing, block elements, braille and
+CP437. Bold is synthesized (each lit pixel doubled one to the right).
+
+## Caps
+
+All in `termshot_consts.h`.
+
+| Constant | Value | Why |
+|---|---|---|
+| `kMaxCols` | 400 | wider panes are cut at the right |
+| `kMaxRows` | 200 | taller panes are cut |
+| `kNativeDefaultCols` x `kNativeDefaultRows` | 160 x 50 | the native log has no pane size |
+| `kMaxCaptureBytes` | 4 MiB | bytes fed to the VT parser (native keeps the tail) |
+| `kMaxPngBytes` | 16 MiB | refuse larger files, also the WhatsApp image cap |
+| `kCaptureTimeout` | 10 s | per tmux call |
+| `kCellWidth` x `kCellHeight` | 8 x 16 px | font cell |
+| `kPadding` | 12 px | border around the grid |
+| `kDefaultFg` / `kDefaultBg` / `kCursorColor` | `#D0D0D0` / `#1C1C1C` / `#D0D0D0` | look |
+
+Measured: a 192 x 35 tmux pane gives a 1560 x 584 PNG of 22 to 26 KB.
+
+## Tool
+
+`terminal_screenshot {target?: string, path?: string}`, registered in
+`src/agent/detz_tools.cpp` (detz-only tools; `data.cpp` stays a byte-exact copy of
+tai.py). Trigger tags: terminal, screenshot, picture, screen, image, snapshot.
+Result: `terminal screenshot <path> (<bytes> bytes, <w>x<h> px, <rows> rows x <cols>
+cols, <source>, image/png)`, or `error: terminal screenshot failed: <reason>`.
+
+For a caller in C++ (for example a WhatsApp send):
+
+```cpp
+#include "../termshot/termshot_capture.h"
+#include "../termshot/termshot_shooter.h"
+using namespace detz::termshot;
+TermshotFrame frame = TermshotCapture::current(target, reader);   // reader may be empty
+TermshotShot shot = TermshotShooter(home).shoot(frame);           // writes the file
+std::string png = TermshotShooter::png(frame);                     // or bytes only
+```
+
+WhatsApp: send the bytes as an imageMessage with mimetype `image/png`; no JPEG
+conversion is done (see the research conclusion `whatsapp-image-as-png`).
+
+## Tests
+
+- `build/termshot_test` (`tests/termshot_test.cpp`, step `termshot.unit`): font coverage
+  and fallback, palette values, PNG signature and size, decoded pixel colors for
+  background, SGR background, full block foreground, reverse, line feed, cursor, caps,
+  native tail cut, file write under `<home>/shots`.
+- `agent.diff` and `agent.e2e` compare against tai.py, so both call
+  `DetzTools::set_enabled(false)`; with detz tools on they stay on in every real run.
+- Live: panes rendered from tmux 3.6 and viewed; a real model (devplace) called the tool
+  through `build/detz` and returned the written path.
+
+## Gaps
+
+- Glyphs Spleen lacks (CJK, Greek, Cyrillic beyond Latin Extended-A, symbols such as
+  U+276F or U+23F5) draw as `?`.
+- Wide characters take one column (the same `TuiVtScreen` gap).
+- The native multiplexer path replays a log tail into a guessed size; not verified live.
+- Only the visible screen; no scrollback option.
+- No JPEG thumbnail is produced for WhatsApp; the chat preview may stay empty until the
+  image is downloaded.
@@ -0,0 +1,2 @@
+# versioning
+VERSION file; tools/version-bump.sh; tools/release.sh; CHANGELOG.md.
@@ -0,0 +1,2 @@
+# workflow
+agent -> approval -> tools -> goal verify; tui renders; mcp serves; cap gates.
@@ -0,0 +1,3 @@
+I said this with my big mouth: ```Ik ga overmorgen maken dat ik kan zeggrn: "Create screenshot and explain what is genius about it and send to the Niggel on WhatsApp" en dat het gewoon werkt.```.
+
+So this statement is about this application. Iti s written in C++ and it is an agent. So on the cli i want to achieve what is stated above. THat means, we should be able to communciate with freaking whatsapp using C++. My god my ggod my god. FIne. Let's just find out what the best python lbirarires are to control whatsapp with using a normal pessant phone number and automate whatever we want. I imagine it ccould be a very painful process. Well, our goal is to make it as user friendly possible with configuring everything and hwatnot. We will isolate all this functionality when it comes to appending it to our awesome agent project. Anyway, also please find out if browser useage is required to achieve our goals, if it is, don't worry, just mention it and i have solutions but just try to avoid it. All i want for now is research and perfectly documented on hoew to achieve this using C++. And yes, that will include cloning python directorys of github and transpile them completely without shame to something we can use. We do this by deciding up front when we need to build and when we need to deep research. Everything is plain simple. YOu will use / execute yourself for literally every thask making everything possible. THe most important thing is, KISS, keep it isimpe. then dry, and then consistency and fuck performance. I do performance, you do code. Ok. lets go. Deliver what i've asked. 
@@ -0,0 +1,118 @@
+# detz — C++23 port of `tai.py` — Plan (v1)
+
+> App name: `detz`. Future: subcommand of `molo` (moloapp framework).
+> Source of truth to port: `../tai.py` (~40k lines) + tests `../test_tai.py`, `../tai_test.py`.
+> Original must remain untouched — no downgrades in functionality.
+> Coding standard / reuse: `moloapp` (`/home/retoor/projects/moloapp`) — components, providers, clients (HTTP client w/ streaming), header-only style, per-part `.mk` files, root Makefile, `build/` outputs only.
+
+## 1. Goals & non-goals
+
+Goals:
+- Full-featured structured C++23 port of tai.py as `detz` binary in this dir (`taicpp/`).
+- Max depth/layering in file structure; maximally object-oriented.
+- argparse/CLI done well so it later plugs in as `molo detz ...` subcommand.
+- Reuse moloapp: HTTP client (+SSE streaming), LLM provider types (`src/llm`, `src/clients`), app errors/middleware patterns where applicable.
+- Docs: README.md, AGENTS.md, SPEC per module, workflow + versioning/tagging/git automation via AGENTS.md + Makefile targets.
+
+Non-goals: modifying `../tai.py`; rewriting moloapp itself (vendor/link only).
+
+## 2. Source inventory (from `../tai.py`)
+
+~40k lines. Key classes/functions to port (grouped):
+
+| Group | Symbols |
+|---|---|
+| Constants/config | `Consts`, `Config`, settings fns (`validate_setting_scope`, `normalize_setting_value`, `parse_setting_env/stored`, `export_settings_to_env`, `boot_settings`), `dir_key`, `orientation_block`, `capability_block` |
+| Text/ANSI/utils | `Ansi`, `paint`, `now_iso`, `short_error`, `truncate`, `capped_text`, `read_capped`, `chunk_lines`, `collapse_repeated_lines`, `split_leading_tags`, `expand_file_mentions`, `complete_files`, `known_model_ids`, `resolve_tag` |
+| Shell/sandbox | `Sandbox`, `ShellSession`, `shell_target_paths`, `segment_targets`, `curl_hint`, `speak`, `Spinner`, `LiveStream`, `PtyProc`, `LaneBoard`, `Glow` |
+| Crypto/vault | `SealError`, `Seal` |
+| Persistence | `Store` (sqlite+FTS5), `spill_output`, `spilled_view`, `spill_or_truncate`, `DebugLog`, `ensure_column`, `fts_*` |
+| LLM backends | `BackendError`, `Opencode`, `ModelHealth`, `ChatClient`, providers (openai/anthropic/google/deepseek/xai/openrouter via moloapp), streaming SSE |
+| Agents/tools | `Tools`, `Agent`, `Approval`, `Denied`, `AgentsFull`, `GoalFailure`, `GoalVerdict`, `GoalVerifier`, `RetrievalEval`, `RagStats` |
+| Comms/ops | `Telegram`, `Install`, `SysInfo`, plexer (`PlexerError`, `PlexerController`, `Native/TmuxPlexerController`), TUI (`TuiTheme/Watched/Style/Span/Event/Key/Click`, see `tui.md`), MCP server (`mcp.md`, `--serve-mcp`), cap orchestrator (`cap.md`) |
+| CLI | argparse tree (~`--cap`, `--serve-mcp`, `--mcp-port`, …) → port to structured CLI layer |
+
+Full class list: extract with `grep -n "^class " ../../tai.py` during milestone M0 and pin counts in SPEC.
+
+## 3. Target structure (deep, layered)
+
+```
+taicpp/
+  plan.md  README.md  AGENTS.md  SPEC.md  CHANGELOG.md  VERSION
+  Makefile  (includes src/*/[*.mk])  .clang-format
+  src/
+    cli/        arg parser tree, detz_main, future molo_subcommand.{h,cpp}
+    config/     consts, settings, env, boot
+    util/       ansi, text, time, truncate, files_mention, complete
+    shell/      sandbox, session, pty, spinner, livestream, lanaboard, glow
+    seal/       seal (vault crypto)
+    store/      sqlite store, fts, spill, debuglog, migrations
+    llm/        thin adapters over moloapp src/llm + src/clients (streaming); model_health, chat_client, backends
+    agent/      tools registry, approvals, agent loop, goal_verifier, retrieval_eval, rag
+    comms/      telegram, install, sysinfo, speak
+    plexer/     controller iface, native, tmux
+    tui/        theme/style/span/events/keys/render (per tui.md)
+    mcp/        serve-mcp http (per mcp.md), auth bearer
+    cap/        orchestrator (per cap.md)
+  tests/  README.md + per-module *_test.cpp mirroring ../test_tai.py + ../tai_test.py cases
+  docs/   specs per module (cli.md, store.md, llm.md, agent.md, tui.md, mcp.md, cap.md), workflow.md, versioning.md
+  tools/  version-bump.sh, tag.sh, changelog.sh
+  build/  (generated only)
+```
+
+Rules: headers `.h` + impl `.cpp`; one class per file pair where feasible; no outputs outside `build/`; `-std=c++23 -Wall -Wextra`, warning-free.
+
+## 4. moloapp reuse map
+
+- HTTP/SSE streaming: `moloapp/src/clients/*` (`client.h`, `sse.h`, transports) — use directly, do not reimplement.
+- Provider/model types: `moloapp/src/llm/*`, `src/openai|anthropic|google|deepseek|xai|openrouter/*`.
+- App patterns (errors, middleware, routing) for MCP HTTP serve: `moloapp/src/app/**/*`.
+- CLI conventions: `moloapp` CLI part (`tests/...cli`, `src/cli`) + `audit/MOLO-CLI-001`.
+- Build pattern: per-part `*.mk` + root Makefile; `example/example.mk` as template.
+
+## 5. CLI design (molo-ready)
+
+- `detz` binary with same flags as tai.py argparse tree (parity table in `docs/cli.md`).
+- Implementation: `src/cli::{Parser, Command, DetzCli}` returning a `Command` value object; `main.cpp` only wires DI + dispatches. Future `molo detz` = register `DetzCommand` as subcommand (adapter header `molo_subcommand.h` stubbed in M1).
+- Golden test: `--help` output diff vs `python ../tai.py --help`.
+
+## 6. Milestones & acceptance
+
+- M0 scaffold: dirs, root Makefile + `src/*/[*.mk]`, `main.cpp` printing version, `make`, `make test` green, AGENTS.md workflow live. Accept: warning-free build, outputs only in `build/`.
+- M1 CLI parity: full argparse tree ported + `--help` golden test + molo subcommand stub. Accept: flag-for-flag table complete.
+- M2 util+config: consts/settings/ansi/text/files. Accept: unit tests mirror python edge cases.
+- M3 store: sqlite+FTS5, spill, debuglog, migrations. Accept: CRUD/FTS tests pass.
+- M4 shell: sandbox/session/pty/speak. Accept: sandboxed exec tests.
+- M5 llm: adapters over moloapp clients + streaming + health. Accept: mocked-transport chat/stream tests, no real network.
+- M6 agent/tools: registry, approvals, loop, goal verifier, rag eval. Accept: ported `test_tai.py` scenarios (mocked backends).
+- M7 plexer/tui/comms: native+tmux controllers, TUI render, telegram/install/sysinfo. Accept: headless render + controller tests.
+- M8 mcp+cap: serve-mcp (bearer), cap orchestrator. Accept: route-level tests through real stack.
+- M9 docs/release automation: README, SPECs, CHANGELOG, version bump/tag targets. Accept: `make release` path documented & dry-runnable.
+
+Order per AGENTS.md workflow: spec delta → code → tests → changelog, per milestone.
+
+## 7. Build/test/lint (mirrors moloapp AGENTS.md)
+
+```sh
+g++ --version   # need C++23 (g++ ≥ 13, moloapp expects 15.2)
+make            # whole repo
+make test
+make lint       # -Wall -Wextra clean; zero findings
+make release PART=patch|minor|major MSG="..."
+```
+
+## 8. Versioning/tagging/git automation
+
+- `VERSION` file (`0.1.0` start, semver); `src/cli/version.hpp` generated from it at build.
+- Targets: `make version-bump PART=x`, `make tag` (annotated `detz-vX.Y.Z`), `make changelog` (prepend CHANGELOG entry), `make release PART=x MSG=...` = bump+changelog+commit+tag. Never push unless asked.
+- First: write root `AGENTS.md` (workflow above) before M1 code — per original request.
+
+## 9. Risks
+
+- tai.py size (~40k lines): mitigate by module-by-module parity tables + ported test counts per milestone.
+- moloapp API drift: pin moloapp commit in SPEC; thin adapter layer in `src/llm/`.
+- sqlite/FTS5 + pty in C++: reuse system sqlite3, forkpty; isolate in `store/` + `shell/`.
+
+## 10. Next action
+
+Write `AGENTS.md` (workflow + commands), then M0 scaffold.
@@ -0,0 +1,12 @@
+# WhatsApp golden test vectors
+
+These four files are copied verbatim from jfelipesjc/wa-go (MIT licensed),
+`internal/.../testdata`, and are used by `tests/whatsapp_test.cpp` to validate
+detz's native WhatsApp wire/crypto/Signal implementation byte-for-byte. They
+were generated by wa-go's capture harness from the real `@whiskeysockets/baileys`
+client; see docs/whatsapp.md. Do not edit.
+
+- `codec_battery.jsonl`   binary-XML node encode/decode battery (19 cases)
+- `client_payload.json`   a registration ClientPayload with its exact bytes
+- `noise.json`            a full Noise XX handshake transcript with keys
+- `session_ab.json`       a Signal X3DH + Double Ratchet session (4 exchanges)
				`@@ -0,0 +1 @@`
				`These are all terminal apps that are convenient to use with the lead24 aopplication...`