docs: split plan into numbered detailed files with research

Ten sequential spec files plus index covering runtime,
observability, security, data, routing, realtime, supervisor,
clients, CLI ops, and workflow, each with best practices,
structure, rules, guidelines, specs, and cross-references.

plans/01-runtime.md

@@ -0,0 +1,211 @@
+# Core Runtime and HTTP Fundamentals
+
+File: plans/01-runtime.md
+Scope: plan.md sections 2-3, 10 step 1 partial. Topics: event loop, lifecycle, HTTP/1.1 + HTTP/2, compression, cookies, static.
+
+## 1. Purpose and philosophy
+
+Provide the single non-blocking foundation every other component builds on: one in-house reactor, one coroutine model, one connection lifecycle, one HTTP/1.1 plus HTTP/2 stack, one compression layer, one cookie transport, one static file path. No feature reimplements I/O, parsing, framing, or file delivery.
+
+Philosophy:
+
+- Boring core, strict caps. Every knob has a static ceiling. No autotuning.
+- Correctness from standards, not folklore. HTTP behavior follows RFC 9110/9112/9113, cookies follow RFC 6265, compression formats follow RFC 1952/7932.
+- Transport only here. Sessions, auth, guards live in 03-security.md. Routing, views, middleware, errors live in 05-request-routing.md. Metrics and tracing hooks live in 02-observability.md but cost near zero when disabled.
+- Deep structure: one class or concern per file, namespaces mirror directories.
+
+## 2. Best practices (from research)
+
+1. HTTP/1.1: persistent connections are the default; framing comes from Content-Length or chunked transfer coding; a recipient MUST parse chunked; unknown transfer codings SHOULD get 501; keep-alive plus correct length framing enables connection reuse (Source: https://www.rfc-editor.org/rfc/rfc9112, retrieved 2026-10-23).
+2. HTTP/1.1 request smuggling defense: never accept both Content-Length and Transfer-Encoding on one message; treat conflicts as 400 and close. This triangulates with RFC 9112 framing rules and current CVE practice around chunked plus keep-alive handling.
+3. HTTP/2: binary framing, interleaved multiplexed streams on one connection, HPACK field compression, connection-level plus stream-level flow control, request prioritization (Source: https://www.rfc-editor.org/rfc/rfc9113, retrieved 2026-10-23). Independent H2 implementations confirm the same trio: multiplexing, HPACK, dual flow control.
+4. HTTP/2 server push is gone in RFC 9113 (obsoletes RFC 7540/8740). Do not implement push. Use 103 Early Hints, normal preload, or SSE from 06-realtime.md instead (Source: https://www.rfc-editor.org/rfc/rfc9113, retrieved 2026-10-23).
+5. HTTP semantics are version independent and defined once in RFC 9110: methods, status codes, fields, content negotiation, validators, ranges, caching (Source: https://www.rfc-editor.org/rfc/rfc9110, retrieved 2026-10-23). MoloApp therefore shares one Request/Response semantic layer across H1 and H2.
+6. Static validators: send both ETag and Last-Modified on 200 when cheap; If-None-Match takes precedence over If-Modified-Since; support If-Range, single byte ranges with 206, and 416 for unsatisfiable ranges (Source: https://www.rfc-editor.org/rfc/rfc9110, retrieved 2026-10-23; corroborated by https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching, retrieved 2026-10-23).
+7. Cookies: servers store state via Set-Cookie and Cookie fields; RFC 6265 defines syntax, domain/path scoping, and expiry semantics (Source: https://www.rfc-editor.org/rfc/rfc6265, retrieved 2026-10-23). H2 MAY split one logical Cookie into several fields to help HPACK; a server MUST join them back with "; " before parsing (Source: https://www.rfc-editor.org/rfc/rfc9113, retrieved 2026-10-23). Set-Cookie MUST never be joined.
+8. Compression: Brotli combines LZ77 plus Huffman coding with quality comparable to best general-purpose methods (Source: https://www.rfc-editor.org/rfc/rfc7932, retrieved 2026-10-23). gzip wraps DEFLATE with header, CRC32, and original size (Source: https://www.rfc-editor.org/rfc/rfc1952, retrieved 2026-10-23). Both are negotiated via Accept-Encoding/Content-Encoding; Vary: Accept-Encoding is required on negotiated responses.
+9. Reactor: epoll scales to many FDs via epoll_create/ctl/wait and supports level-triggered (default) plus edge-triggered (EPOLLET) modes (Source: https://man7.org/linux/man-pages/man7/epoll.7.html, retrieved 2026-10-23). kqueue (BSD/macOS) and IOCP with GetQueuedCompletionStatus/PostQueuedCompletionStatus (Windows, see https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o, retrieved 2026-10-23) are the native peers. MoloApp hides all three behind one Poller interface; default is level-triggered for safety.
+10. Coroutines: a C++20/23 coroutine is promise plus handle plus awaitable; initial_suspend decides lazy vs eager start; co_await suspends without blocking the thread (Source: https://en.cppreference.com/cpp/language/coroutines, retrieved 2026-10-23). All handlers are coroutines returning Task<T>; blocking work never runs on loop threads.
+
+## 3. Structure
+
+Directories mirror namespaces. One concern per file. Tests mirror routes per 10-workflow-example.md.
+
+```
+src/molo/runtime/
+  event_loop.h/.cpp        molo::runtime::EventLoop
+  poller.h                 molo::runtime::IPoller (interface)
+  poller_epoll.h/.cpp      molo::runtime::EpollPoller (Linux)
+  poller_kqueue.h/.cpp     molo::runtime::KqueuePoller (BSD/macOS)
+  poller_iocp.h/.cpp       molo::runtime::IocpPoller (Windows)
+  task.h                   molo::runtime::Task<T>, molo::runtime::Generator<T>
+  executor.h/.cpp          molo::runtime::ThreadPool (bounded blocking pool)
+  timer_wheel.h/.cpp       molo::runtime::TimerWheel
+  signal_handler.h/.cpp    molo::runtime::SignalHandler
+  strand.h                 molo::runtime::Strand (per-connection serial queue)
+  channel.h                molo::runtime::Channel<T> (inter-loop messages)
+src/molo/http/
+  message.h/.cpp           molo::http::RequestHead, ResponseHead, Headers
+  connection.h/.cpp        molo::http::Connection (H1/H2 dispatch, lifecycle)
+  listener.h/.cpp          molo::http::Listener (accept, SO_REUSEPORT)
+  protocol_detect.h/.cpp   molo::http::ProtocolDetector (ALPN, prior knowledge)
+  v1/parser.h/.cpp         molo::http::v1::Parser (strict RFC 9112)
+  v1/serializer.h/.cpp     molo::http::v1::Serializer
+  v2/connection.h/.cpp     molo::http::v2::H2Connection (framing, settings)
+  v2/stream.h/.cpp         molo::http::v2::H2Stream (state machine)
+  v2/hpack.h/.cpp          molo::http::v2::HpackEncoder, HpackDecoder
+  v2/flow_control.h/.cpp   molo::http::v2::FlowController
+  v2/settings.h            molo::http::v2::Settings (caps negotiation)
+src/molo/compress/
+  compressor.h             molo::compress::ICompressor (interface)
+  gzip.h/.cpp              molo::compress::GzipCompressor (RFC 1952)
+  brotli.h/.cpp            molo::compress::BrotliCompressor (RFC 7932)
+  negotiate.h/.cpp         molo::compress::Negotiator (Accept-Encoding)
+src/molo/cookie/
+  cookie.h/.cpp            molo::cookie::Cookie (parse, serialize)
+  jar.h/.cpp               molo::cookie::CookieJar (request join, response list)
+  signer.h                 molo::cookie::ISigner (HMAC interface, impl in 03-security.md)
+src/molo/static/
+  file_server.h/.cpp       molo::static_::FileServer (conditional, ranges)
+  asset_provider.h         molo::static_::IAssetProvider (disk + embedded)
+  etag.h/.cpp              molo::static_::Etag (strong/weak generation)
+```
+
+Namespace rule: `molo::runtime`, `molo::http`, `molo::http::v1`, `molo::http::v2`, `molo::compress`, `molo::cookie`, `molo::static_`. No cross-namespace includes except through declared interfaces.
+
+## 4. Rules (hard, numbered)
+
+- R1. No external loop, HTTP, compression, or cookie library. Only stdlib, OS syscalls, and wolfSSL (TLS, owned by 03-security.md).
+- R2. Non-blocking everywhere on loop threads. Blocking syscalls, DNS, file hashing, or compression above threshold go to ThreadPool or use async FDs.
+- R3. One EventLoop per thread; connections are pinned to their loop. Cross-loop contact only via Channel<T>. No shared mutable connection state.
+- R4. Every limit in section 6 is a compile-time default plus runtime config with a static ceiling. No unbounded queue, header, body, stream, or decompressed size.
+- R5. H1 parser is strict: reject ambiguous framing with 400 and close; require Host; accept chunked; answer unknown transfer coding with 501.
+- R6. H2 MUST implement settings, flow control, headers/data/priority (ignored)/rst/ping/goaway/window-update; MUST NOT implement server push.
+- R7. Compression never changes semantics: always send Vary: Accept-Encoding when negotiating; never compress 204/304, HEAD, or already-encoded bodies.
+- R8. Static serving defaults to safe: directory indexing off, dotfiles denied, symlinks not followed unless explicitly allowed, traversal outside root impossible.
+- R9. Cookies: parse per RFC 6265, join split H2 Cookie fields, never join Set-Cookie, default SameSite=Lax plus HttpOnly; Secure forced on https.
+- R10. Graceful shutdown is mandatory: stop accept, drain with deadline, GOAWAY on H2, Connection: close on H1, then hard close and report.
+- R11. No H2 or H1 knowledge inside handlers. Handlers see one semantic Request/Response; version differences stay inside src/molo/http.
+- R12. HTTP/3 is a designed path, not v1 scope: keep a Transport interface QUIC can implement later; ship H1 + H2 only.
+
+## 5. Development guidelines
+
+- Write the interface header first, then the strictest parser or state machine, then the serializer, then integration. Each file gets a spec comment naming its RFC section.
+- Prefer state machines with explicit states and transition tables for H1 parsing, H2 streams, and HPACK. Fuzz each with truncated, oversized, and interleaved inputs.
+- Coroutine style: `Task<Response> handle(Request)`; co_await I/O, timers, pool work; honor CancellationToken; never hold a lock across co_await.
+- Threading: N loops equal min(CPU count, configured workers), each with own Poller; Listener uses SO_REUSEPORT where available, else single acceptor handing FDs round-robin.
+- Backpressure: stop reading when the write buffer passes high watermark; resume at low watermark; H2 additionally obeys stream and connection windows.
+- Errors are structured: parser errors map to 400/413/431/501 with connection action (keep or close); internal errors emit 500 via 05-request-routing.md error hierarchy.
+- Observability hooks (02-observability.md) wrap accept, parse, compress, static hit/miss; disabled builds compile them out.
+- Keep gzip and Brotli streaming: bounded input chunks, bounded output chunks, flush points; never buffer a full passthrough body to compress it.
+
+## 6. Specifications
+
+### 6.1 Event loop and threading
+
+- Pollers: epoll (Linux), kqueue (BSD/macOS), IOCP (Windows). Default level-triggered; edge-triggered only inside Poller with drain-to-EAGAIN.
+- Task<T>: lazy start (initial_suspend_always), executor affinity to owning loop, cancellation via token checked at each await.
+- ThreadPool for blocking work: default 4 threads, queue cap 1024 jobs, overflow returns 503 with Retry-After, never grows silently.
+- TimerWheel: 10 ms granularity, cap 65536 timers per loop.
+
+### 6.2 Connection and request lifecycle
+
+- Stages: accept, TLS handshake (03-security.md), protocol detect, head parse, body read, dispatch, handler, serialize, keep-alive or close.
+- Timeouts (defaults, all configurable down to floor, up to ceiling): header 10 s, body 30 s, idle keep-alive 75 s, handler soft budget 30 s then 503, shutdown drain 30 s.
+- Caps: max connections 16384 per process (ceiling 65536); max requests per H1 connection 1000; H2 max concurrent streams 100 per connection; header block max 64 KB and 100 fields; URI max 8 KB; default body max 10 MB (routes may lower; raising above 100 MB needs explicit config and streams to disk).
+- Keep-alive: H1 persistent by default; `Connection: close` or protocol error closes. H2 uses GOAWAY with last-stream-id on drain.
+- Signals: SIGINT/SIGTERM start drain; SIGHUP reloads config where safe; SIGTERM during drain shortens deadline to 5 s.
+
+### 6.3 HTTP/1.1
+
+- Strict start-line and field parsing per RFC 9112; obs-fold rejected; Host required; 400 on bad version, bad chunk, or Content-Length plus Transfer-Encoding conflict.
+- Transfer codings: must parse chunked including extensions (ignored); unknown coding gets 501. Trailers parsed and exposed, capped by same field limits.
+- Pipelining: accepted but processed sequentially in order; no parallel handler execution per connection.
+- Responses: automatic Content-Length or chunked; HEAD sends head only; 204/304 never carry a body.
+
+### 6.4 HTTP/2
+
+- Entry: h2 over TLS via ALPN `h2`; h2c via prior knowledge (PRI preface) or Upgrade from H1. Cleartext default in dev, TLS default in prod (TLS owned by 03-security.md).
+- Preface and Settings handshake mandatory; bad preface is connection error PROTOCOL_ERROR.
+- HPACK: static plus dynamic table, decoder cap 4096 bytes default (SETTINGS_HEADER_TABLE_SIZE ceiling 16384); oversized header list is COMPRESSION_ERROR.
+- Flow control: initial window 65535 both directions; max frame 16384 (ceiling 16777215); WINDOW_UPDATE honored; writes block on window via backpressure, never buffer unbounded.
+- Streams: states idle, reserved, open, half-closed, closed per RFC 9113; max 100 concurrent; RST_STREAM on handler cancel or caps; PRIORITY frames accepted and ignored.
+- No server push. No CONNECT except plain tunnels if later specified; v1 returns 405 for exotic methods unless a sibling spec claims them.
+
+### 6.5 Compression
+
+- Encodings: `gzip` and `br`, both in-house. Negotiation picks br over gzip when q-values tie and type is compressible.
+- Threshold: only bodies >= 1024 bytes and content-type in allowlist (html, json, xml, text, css, js, svg, wasm except precompressed). Never compress images, video, audio, zip, pdf, or responses with Content-Encoding already set.
+- Levels: gzip 6 dynamic; brotli quality 4 dynamic, 11 for precompressed static assets at build time.
+- Static precompression: build may store `.gz` and `.br` siblings; server selects by Accept-Encoding and sends Vary plus correct Content-Encoding and ETag variant suffix (`-gz`, `-br`).
+- Bomb guard: decompression of request bodies capped at 10x compressed size and 10 MB absolute; overflow is 413.
+
+### 6.6 Cookies (transport only)
+
+- Request: parse Cookie per RFC 6265; join H2 split fields with "; "; cap 50 cookies and 8 KB total; overflow is 431.
+- Response: emit one Set-Cookie per cookie; attributes Domain, Path, Max-Age/Expires, Secure, HttpOnly, SameSite (Lax default, None requires Secure), Partitioned passthrough.
+- Signed and encrypted values: this file defines Cookie and ISigner interfaces; crypto and key hierarchy live in 03-security.md. Format: `v1.<base64url(payload)>.<base64url(tag)>`.
+- Size: single Set-Cookie max 4 KB; larger is a handler error, never silently truncated.
+
+### 6.7 Static files
+
+- Providers: DiskProvider (root jailed) and EmbeddedProvider (from 09-cli-ops.md embedder). Same FileServer logic over both.
+- Safety: lexical normalize then verify prefix is inside root; reject NUL, backslash on Windows, `..` escapes; symlinks not followed unless `follow_symlinks=true` and target stays inside root; dotfiles denied by default.
+- Validators: strong ETag `"size-mtimehex"` for disk, `"len-hashhex"` for embedded; Last-Modified from mtime; send both; honor If-None-Match then If-Modified-Since; 304 with validators and Cache-Control.
+- Ranges: Accept-Ranges: bytes; single range yields 206 with Content-Range; multiple ranges yield 416 in v1 (documented gap, multipart later); If-Range honored; bad range yields 416 with Content-Range: bytes */len.
+- Defaults: Cache-Control `public, max-age=3600` for versioned assets, `no-cache` for index HTML (overridable); directory index off (403); only GET, HEAD, OPTIONS; others 405.
+- Performance: sendfile or TransmitFile on disk when no compression or encryption transform; otherwise bounded streaming (64 KB chunks).
+
+### 6.8 HTTP/3 path (design only)
+
+- Connection owns a `Transport` abstraction (TCP today). H2 logic never touches sockets directly. A future QUIC transport can reuse semantic Request/Response, HPACK successor QPACK stays isolated in a new v3 directory. No QUIC code in v1.
+
+## 7. Execution steps (in build order)
+
+1. Reactor skeleton: IPoller, EpollPoller, Task<T>, EventLoop with timers, ThreadPool, echo TCP server, signal handling.
+2. Lifecycle core: Listener, Connection, timeouts, watermarks, graceful drain, structured accept/close logging hooks.
+3. HTTP/1.1: strict parser, serializer, keep-alive, chunked, pipelining queue, 400/413/431/501 mapping.
+4. H2 framing: preface, settings, HPACK codec, stream states, headers/data flow, flow control, GOAWAY/RST/PING.
+5. Protocol detection: ALPN wiring point, h2c prior knowledge and Upgrade, version-independent dispatch.
+6. Compression: gzip encoder/decoder, brotli encoder/decoder, negotiator, thresholds, Vary, bomb guards.
+7. Cookies: parser, serializer, jar join/split, signer interface with stub, attribute defaults.
+8. Static: providers, traversal guard, ETag/Last-Modified, conditional, single range, cache headers, sendfile path.
+9. Hardening pass: all caps enforced, fuzz harnesses for H1/H2/HPACK/cookies, slowloris and large-body tests.
+10. Integration seam: hand semantic Request/Response to 05-request-routing.md, export counters to 02-observability.md, accept TLS handle from 03-security.md.
+
+## 8. Validation (how to verify each step)
+
+- Step 1: echo server holds 10k idle connections under 512 MB; kill -INT drains in under 30 s; blocking job overflow returns 503 not hang.
+- Step 2: header/body/idle timeouts fire within 10 percent; high/low watermarks pause and resume reads; drain closes listeners first then connections.
+- Step 3: RFC 9112 vectors pass (chunked, trailers, bad framing gives 400+close, unknown TE gives 501); keep-alive reuses connection for 1000 requests; pipelined requests answer in order.
+- Step 4: h2spec-style checks pass for settings, window updates, stream reset, GOAWAY; HPACK round-trips interop vectors; oversized header list triggers COMPRESSION_ERROR.
+- Step 5: ALPN h2 negotiates H2, http/1.1 negotiates H1; prior-knowledge h2c works; H1 Upgrade to h2c works; wrong preface closes with PROTOCOL_ERROR.
+- Step 6: Accept-Encoding br/gzip/identity matrix returns right encoding plus Vary; bodies under 1024 bytes stay identity; bomb input stops at cap with 413.
+- Step 7: RFC 6265 cookie vectors parse; H2 split Cookie joins correctly; Set-Cookie never merged; oversized jar returns 431.
+- Step 8: traversal attempts (`..`, encoded, NUL, symlink escape) all fail closed; conditional requests return 304; single range returns 206 and bad range 416; embedded and disk providers return byte-identical bodies.
+- Step 9: fuzzers run 1M mutated inputs with zero crashes or hangs; slowloris with 1000 trickling connections stays under caps.
+- Step 10: integration suite from 10-workflow-example.md passes end to end through real sockets for H1 and H2; disabled observability changes throughput by under 2 percent.
+
+## 9. References
+
+- Depends on: 02-observability.md (event, metric, trace hooks and zero-cost macros); 03-security.md (TLS handle, cookie signing and encryption, body and header ceilings alignment).
+- Feeds: 05-request-routing.md (semantic Request/Response, app and router dispatch); 06-realtime.md (H1 Upgrade path for WebSocket, H2 streams for SSE); 08-clients-openapi.md (framing and HPACK reuse for test and REST clients); 09-cli-ops.md (embedded asset format, dev server listener reuse, health endpoints).
+- Parallel: 04-data.md and 07-supervisor-replication.md consume lifecycle and Task<T> but define their own pooling and durability.
+
+## 10. Open questions
+
+1. Default loop count: thread-per-core with SO_REUSEPORT, or single acceptor plus worker loops for simpler Windows parity?
+2. H2 PRIORITY: fully ignore, or keep a minimal scheduling hint for SSE vs API multiplexing?
+3. Multipart ranges in v1 or documented 416 gap until demand appears?
+4. Brotli static dictionary: ship full default dictionary or trimmed web subset for binary size?
+5. Cookie chunking for >4 KB signed values: reject, or support `name_0..N` reassembly?
+
+## 11. Sources
+
+- RFC 9112 HTTP/1.1, R. Fielding et al., Internet Standard, https://www.rfc-editor.org/rfc/rfc9112 (retrieved 2026-10-23).
+- RFC 9113 HTTP/2, M. Thomson et al., Proposed Standard obsoleting RFC 7540/8740, https://www.rfc-editor.org/rfc/rfc9113 (retrieved 2026-10-23).
+- RFC 9110 HTTP Semantics, R. Fielding et al., Internet Standard, https://www.rfc-editor.org/rfc/rfc9110 (retrieved 2026-10-23).
+- RFC 6265 HTTP State Management Mechanism, A. Barth, https://www.rfc-editor.org/rfc/rfc6265 (retrieved 2026-10-23).
+- RFC 7932 Brotli Compressed Data Format, https://www.rfc-editor.org/rfc/rfc7932 (retrieved 2026-10-23); RFC 1952 GZIP File Format 4.3, https://www.rfc-editor.org/rfc/rfc1952 (retrieved 2026-10-23).
+- C++ coroutines reference, https://en.cppreference.com/cpp/language/coroutines (retrieved 2026-10-23).
+- epoll man page man7/epoll.7, https://man7.org/linux/man-pages/man7/epoll.7.html (retrieved 2026-10-23); Windows async I/O and IOCP, https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o (retrieved 2026-10-23).

plans/02-observability.md

@@ -0,0 +1,159 @@
+# 02 - Observability Backbone
+
+Title: Observability Backbone
+Scope: plan.md sections 2, 8 partial, 10 step 1 partial. Topics: logging, events, metrics, telemetry, zero-cost.
+
+## 1. Purpose and philosophy
+
+This file specifies the four observability pillars of MoloApp (logging, typed events, metrics, distributed tracing) plus the correlation plumbing that ties them together. Everything is implemented in-house in C++23 with zero third-party dependencies, but every wire format stays compatible with the surrounding ecosystem: Prometheus text exposition for metrics, W3C Trace Context for propagation, OTel-compatible span model and OTLP/HTTP JSON export for traces.
+
+Guiding rule from plan.md: instrumentation stays everywhere and collects like a maniac, yet compiles to near-zero overhead when disabled. That means two switches for every pillar: a compile-time flag that removes code entirely, and a runtime flag that reduces a disabled call to one predictable branch. Disabled overhead is measured by benchmark, not assumed.
+
+Correlation is the backbone inside the backbone: one request ID and one trace/span context flow through middleware, ORM, replication, tasks and all clients, and every log record, event, metric label set and span carries them.
+
+## 2. Best practices (from research)
+
+- Separate API from SDK and exporters. OpenTelemetry C++ structures telemetry as an API plus SDKs plus exporters, with traces, metrics and logs all Stable, which confirms the split used in section 4 below.
+- Metric names use snake_case with a single-word application prefix relevant to the domain, counters end in `_total`, and units belong in the name (for example `http_request_duration_seconds`). Label sets stay low-cardinality.
+- Prometheus exposition is line-based text served over HTTP: `# HELP` and `# TYPE` comment lines followed by `name{label="value"} value [timestamp]` samples. Our `/metrics` renderer emits exactly this shape.
+- Distributed context propagates via the W3C `traceparent` header with `version-format = trace-id "-" parent-id "-" trace-flags` (trace ID 16 bytes hex, parent/span ID 8 bytes hex, 1-byte flags), plus `tracestate` for vendor extras. Inbound headers are parsed, outbound requests re-emit them.
+- HTTP server span names follow the OTel semantic convention `{method} {target}` where target is a low-cardinality route template (never a raw path), and `{method}` equals the `http.request.method` attribute. OTel HTTP span conventions are Stable.
+- Compile-time log-level removal follows the spdlog `SPDLOG_ACTIVE_LEVEL` pattern: calls below the active level compile to nothing, so debug and trace logging can stay in hot paths. Async logging follows the bounded-queue plus dedicated-writer-thread pattern with a drop policy and a dropped counter instead of blocking producers.
+
+## 3. Structure
+
+Directory root: `src/observability/`. Public headers mirror it under `include/molo/observability/`. One class or concern per file. Namespaces: `molo::log`, `molo::events`, `molo::metrics`, `molo::trace`, `molo::corr`.
+
+Logging (`src/observability/logging/`):
+
+- `level.h`: `enum class Level { Trace, Debug, Info, Warn, Error, Critical }` plus `to_string`, `from_string`.
+- `record.h`: `struct Record` with timestamp, level, logger name, message, source location, request ID, trace/span IDs, structured fields.
+- `logger.h/.cpp`: `class Logger`, named loggers, per-logger level, child loggers inheriting context.
+- `sink.h`: `class Sink` interface (`write(const Record&)`); `console_sink.h`, `file_sink.h/.cpp` (size rotation), `null_sink.h`, `ring_sink.h` (in-memory tail for tests and `logs` inspection).
+- `async_queue.h/.cpp`: bounded MPSC queue plus writer thread, overflow policy, dropped-record counter.
+- `json_format.h/.cpp`: fixed JSON schema renderer for records.
+- `registry.h/.cpp`: `class Registry` holding named loggers, global level, sink list.
+- `macros.h`: `MOLO_LOG_*` macros with compile-time level gating and source location capture.
+
+Events (`src/observability/events/`):
+
+- `event.h`: `class Event` base with type ID, timestamp, request/trace linkage.
+- `listener.h`: sync and async listener interfaces.
+- `dispatcher.h/.cpp`: `class Dispatcher` built on the pubsub core, listener registration, zero-listener fast path.
+- `lifecycle_events.h`: server start/stop, connection open/close, request start/finish, task enqueue/start/finish/fail.
+
+Metrics (`src/observability/metrics/`):
+
+- `counter.h`, `gauge.h`, `histogram.h`, `timer.h` (RAII timer recording into a histogram).
+- `labels.h`: label set type with static cardinality caps.
+- `registry.h/.cpp`: `class Registry` owning all series, per-app instance with a process default.
+- `prometheus.h/.cpp`: text exposition renderer (`# HELP`, `# TYPE`, samples).
+
+Tracing (`src/observability/tracing/`):
+
+- `ids.h`: `TraceId` (16 bytes), `SpanId` (8 bytes), `TraceFlags`, hex parse/format.
+- `span_context.h`: `struct SpanContext` (IDs plus flags plus tracestate).
+- `span.h/.cpp`: `class Span` with name, kind (server, client, internal), attributes, events, status, RAII end.
+- `tracer.h/.cpp`: `class Tracer` creating spans, null-span fast path when disabled or unsampled.
+- `sampler.h`: head-based samplers (always on/off, ratio).
+- `propagator.h/.cpp`: W3C `traceparent`/`tracestate` parse and inject for HTTP headers.
+- `batch_processor.h/.cpp`: bounded span buffer plus background batch export.
+- `otlp_http.h/.cpp`: OTLP/HTTP JSON exporter over the framework HTTP client.
+
+Correlation (`src/observability/correlation/`):
+
+- `request_context.h/.cpp`: `class RequestContext` holding request ID plus current `SpanContext`, created per request, passed explicitly (never a global) through middleware, handlers, ORM and replication.
+
+## 4. Rules (hard, numbered)
+
+1. Every log call, event emission, metric update and span operation must compile to near-zero overhead when its pillar is disabled at compile time, and to at most one branch plus argument skip when disabled at runtime.
+2. No observability call may block a request handler: logging and span export go through bounded queues with drop-on-full and a visible dropped counter.
+3. Every log record emitted inside a request carries `request_id`; every record inside a trace also carries `trace_id` and `span_id`.
+4. Metric label values must come from bounded sets only (route templates, methods, status classes, error kinds). Raw paths, user IDs, query strings and full error text are forbidden as label values.
+5. Counters are named `*_total`, use snake_case, and framework metrics carry the `molo_` prefix while application metrics carry the application prefix.
+6. Histograms use cumulative buckets and always export `_bucket`, `_sum` and `_count` series.
+7. Server span names are `{METHOD} {route_template}` per OTel HTTP conventions; raw paths in span names are forbidden.
+8. Inbound `traceparent` is parsed and continued; malformed headers start a new trace and increment `molo_trace_propagator_errors_total`.
+9. Outbound requests from framework clients (HTTP, WebSocket, Socket.IO, SSE, WebDAV) always inject current trace context when tracing is enabled.
+10. Sampling decisions are head-based and immutable for the lifetime of a trace.
+11. All buffers, series counts, label cardinalities, attribute counts and export rates have static ceilings declared in section 6; exceeding a ceiling drops data and counts the drop, never grows memory.
+12. Secrets, passwords, tokens, session contents and request bodies are never logged, never set as span attributes, and never appear in metric labels. A shared redaction helper enforces this.
+
+## 5. Development guidelines
+
+- Instrument first at every new call site: log line at the right level, one span for each cross-boundary operation, one counter or histogram for each countable outcome. Reviews reject uninstrumented handlers.
+- Prefer structured fields over string interpolation: `MOLO_LOG_INFO(logger, "db.query", {{"table", t}, {"rows", n}})` instead of building sentences.
+- Levels: Trace for per-byte or per-frame detail, Debug for developer diagnostics, Info for lifecycle and business milestones, Warn for degraded but handled states, Error for failed operations needing attention, Critical for process-threatening states.
+- Keep hot-path instrumentation allocation-free: static format strings, integer IDs, reused label sets.
+- Tests assert on the `ring_sink`, the metrics registry snapshot and the in-memory span exporter, never on files or network.
+- Exporters and sinks are the only components allowed to touch I/O; pillars never call the network or filesystem directly.
+
+## 6. Specifications (concrete behaviors, caps, formats)
+
+Log record JSON schema (fixed keys, UTC RFC 3339 with millis in `ts`):
+
+`{"ts":"...","level":"info","logger":"molo.http","msg":"request.finished","request_id":"...","trace_id":"...","span_id":"...","fields":{...}}`
+
+Compile-time flags: `MOLO_ACTIVE_LEVEL` (default `Debug` in dev, `Info` in prod), `MOLO_ENABLE_METRICS` (default 1), `MOLO_ENABLE_TRACING` (default 1), `MOLO_OBSERVABILITY` master switch (0 removes all pillars). Runtime toggles mirror each flag and are read once per call site through a cached atomic.
+
+Static ceilings (defaults, overridable via typed config only):
+
+- Async log queue: 8192 slots; max log line 8192 bytes (truncate plus `truncated:true`); file sink rotation 10 x 10 MB; ring sink 1024 records.
+- Metrics: max 65536 series per registry; max 64 distinct values per label key (65th maps to `"other"` and counts `molo_metrics_cardinality_overflows_total`); max 16 labels per metric.
+- Tracing: span buffer 4096; export batch 512 spans or 5 s interval, whichever comes first; max 64 attributes and 32 events per span; max attribute value 1024 bytes.
+- Default HTTP duration histogram buckets in seconds: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10.
+
+Framework metrics shipped by default: `molo_http_requests_total{method,route,code}`, `molo_http_request_duration_seconds{method,route}`, `molo_http_requests_in_flight`, `molo_db_queries_total{op,table}`, `molo_db_query_duration_seconds{op}`, `molo_log_records_total{level}`, `molo_log_dropped_total`, `molo_spans_total{kind}`, `molo_spans_dropped_total`, `molo_trace_propagator_errors_total`, `molo_events_total{type}`.
+
+Exposition endpoint contract (wired by 09-cli-ops.md): `GET /metrics` returns `text/plain; version=0.0.4`, status 200, full registry render under 100 ms for 10k series. Trace export contract: OTLP/HTTP POST of JSON `ExportTraceServiceRequest` to the configured collector URL with gzip, timeout 10 s, at most 3 retries with backoff.
+
+## 7. Execution steps (in build order)
+
+1. Correlation: `RequestContext` plus request-ID generation and middleware hook point (hook wired in 05-request-routing.md, context type lives here).
+2. Logging core: levels, records, macros with compile-time gating, registry, console and null sinks, JSON format.
+3. Logging async path: bounded queue, writer thread, file sink with rotation, ring sink, dropped counters.
+4. Events: base event, dispatcher on pubsub core, sync/async listeners, lifecycle event set.
+5. Metrics core: counter, gauge, histogram, timer, labels with cardinality caps, registry.
+6. Metrics exposition: Prometheus text renderer plus render benchmarks.
+7. Tracing core: IDs, span context, tracer, span lifecycle, samplers, null-span fast path.
+8. Propagation: W3C parse/inject helpers plus client/server integration points (wiring in 01-runtime.md, 05-request-routing.md, 08-clients-openapi.md).
+9. Export: batch processor plus OTLP/HTTP JSON exporter with retry and backoff.
+10. Zero-cost pass: compile-time removal checks and disabled-overhead benchmarks for all four pillars.
+
+## 8. Validation (how to verify each step)
+
+- Steps 1-2: integration test asserting every response carries `X-Request-ID` and every request-scoped log record contains that ID; macro test asserting calls below `MOLO_ACTIVE_LEVEL` emit no code (object-size or disassembly check on a probe TU).
+- Step 3: flood test pushing 100k records showing bounded memory, a nonzero `molo_log_dropped_total` under saturation, and no handler blocking beyond 1 ms p99.
+- Step 4: lifecycle test asserting request start/finish events fire exactly once per request; zero-listener benchmark showing dispatch under 5 ns per call.
+- Steps 5-6: exposition test comparing renderer output byte-shape against the Prometheus format (HELP/TYPE/sample lines, cumulative buckets); cardinality test asserting overflow maps to `"other"` and counts the overflow metric.
+- Step 7: span lifecycle test (parent/child linkage, attribute caps, truncation flags); sampler test asserting ratio sampler honors its rate within 5 points over 100k traces.
+- Step 8: W3C round-trip test using the spec vectors (`00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01` style) plus malformed-header tests; client test asserting outbound injection.
+- Step 9: exporter test against a stub collector asserting valid OTLP JSON, batching behavior and retry/backoff timing.
+- Step 10: benchmark gate failing the build if any disabled pillar adds more than 1 percent overhead versus a no-instrumentation baseline on the hot path.
+
+## 9. References (sibling files this depends on or feeds)
+
+- Depends on 01-runtime.md: request lifecycle hook points, event-loop threading rules for background writer/exporter threads.
+- Feeds 04-data.md: per-query metrics, ORM span conventions, correlation passing into transactions.
+- Feeds 05-request-routing.md: middleware correlation hook, per-route label values, error-to-log mapping.
+- Feeds 06-realtime.md: WebSocket and Socket.IO span kinds and connection metrics.
+- Feeds 07-supervisor-replication.md: task lifecycle events and metrics, replication trace continuation.
+- Feeds 08-clients-openapi.md: client-side span creation and propagation injection, test-client observability inspection hooks.
+- Feeds 09-cli-ops.md: `/metrics` exposition renderer, health/readiness metric inputs, `logs` command tail source, event stream inspection.
+- Feeds 10-workflow-example.md: living-example dashboards and the observability demonstration checklist.
+
+## 10. Open questions
+
+- OTLP/HTTP JSON is chosen over protobuf to avoid an in-house protobuf codec; revisit if a collector strictly requires protobuf.
+- Default trace sampling ratio in production is undecided (proposal: 0.1 with always-on errors); needs a measurement run first.
+- Per-app versus process-global metrics registry default needs confirmation with 05-request-routing.md once the application class shape lands.
+- Log file rotation count and size defaults are placeholders pending the operations profile in 09-cli-ops.md.
+
+## 11. Sources (exact URLs with retrieval dates)
+
+- OpenTelemetry C++ documentation, API/SDK structure and Stable status for traces, metrics, logs: https://opentelemetry.io/docs/instrumentation/cpp/ (retrieved 2026-10-23). Main repo: https://github.com/open-telemetry/opentelemetry-cpp (retrieved 2026-10-23).
+- Prometheus metric and label naming (single-word application prefix): https://prometheus.io/docs/practices/naming/ (retrieved 2026-10-23).
+- Prometheus exposition formats (text format, HELP/TYPE/sample lines): https://prometheus.io/docs/instrumenting/exposition_formats/ (retrieved 2026-10-23).
+- W3C Trace Context Recommendation, traceparent version-format (REC 2021-11-23): https://www.w3.org/TR/trace-context/ (retrieved 2026-10-23).
+- spdlog repository, SPDLOG_ACTIVE_LEVEL compile-time gating and async logging design: https://github.com/gabime/spdlog (retrieved 2026-10-23).
+- OpenTelemetry semantic conventions for HTTP spans, Stable, `{method} {target}` naming: https://opentelemetry.io/docs/specs/semconv/http/http-spans/ (retrieved 2026-10-23).

plans/03-security.md

@@ -0,0 +1,135 @@
+# 03 - Security
+
+Scope: plan.md sections 2, 4, 10 step 2. Topics: TLS wolfSSL, auth, hashing and crypto, sessions, guards, upload safety. Also covers CSRF, CORS, security headers, rate limiting, input caps, secrets, XSS-safe defaults, path traversal, ORM query safety contract.
+
+## Purpose and philosophy
+
+Secure by default, specified not folkloric. Every guard ships on with safe defaults and an explicit override path. Deny by default: closed CORS, strict headers, capped inputs, authenticated sessions. One shared core owns each primitive (hashing, encryption, sessions, TLS); features consume it through declared interfaces, never reimplement. Static ceilings everywhere, no autotuning. Loud rejection with structured reason and counter, never silent downgrade.
+
+## Best practices (from research)
+
+- Passwords: Argon2id first choice, minimum m=19456 (19 MiB), t=2, p=1; bcrypt work factor 10 or more for legacy; PBKDF2 600,000+ with HMAC-SHA-256 only for FIPS-140; unique random salt per password; optional pepper stored outside the DB in a vault or HSM; hash verification under one second; re-hash on login when work factor rises (OWASP Password Storage).
+- Argon2 standard: Argon2id MUST be supported; salt 16 bytes RECOMMENDED, unique per password; FIRST RECOMMENDED option t=1, p=4, m=2 GiB; SECOND for constrained hosts t=3, p=4, m=64 MiB; OWASP minimum (19 MiB, t=2, p=1) is the interactive-login tuning of the same primitive, RFC values are the max-security anchors (RFC 9106, triangulated with OWASP).
+- Sessions: ID minimum 64 bits entropy, framework uses 128-bit CSPRNG IDs; value meaningless, logic server-side; cookies Secure, HttpOnly, SameSite Strict or Lax; renew ID on any privilege change; idle plus absolute plus renewal timeouts; validate IDs as untrusted input (OWASP Session Management).
+- CSRF: stateful apps use synchronizer token pattern; stateless or API fallback is signed double-submit cookie with HMAC bound to session data, naive unsigned double-submit is discouraged (cookie injection via sibling subdomain); Fetch Metadata Sec-Fetch-Site as lightweight cross-site block; custom headers for AJAX APIs (OWASP CSRF Prevention).
+- Uploads: allow-list business-critical extensions only; never trust Content-Type header, verify magic bytes and content; generate server-side filenames; cap size and name length; store outside webroot or on separate host with ID mapping handler; AV or sandbox plus CDR hook; authorized users only (OWASP File Upload).
+- Headers: X-Frame-Options DENY (frame-ancestors in CSP for modern browsers), X-Content-Type-Options nosniff, Referrer-Policy strict-origin-when-cross-origin set explicitly, HSTS max-age=63072000 with includeSubDomains and preload, restrictive CSP, minimal Permissions-Policy (OWASP HTTP Headers).
+- TLS: wolfSSL small fast portable TLS up to 1.3 plus DTLS 1.3, wolfCrypt core, FIPS-validated builds available, OpenSSL-compat API for porting, dual GPL or commercial license; MoloApp uses commercial license for closed-source use, TLS 1.3 only on the server path (wolfSSL repo).
+
+## Structure
+
+Directory: `src/security/`, one class or concern per file. Namespace root `molo::security`, sub-namespaces `tls`, `auth`, `crypto`, `session`, `guard`, `upload`, `secret`.
+
+- `tls/TlsConfig.h`: cipher suites, ALPN list (h2, http/1.1), cert paths, min version 1.3.
+- `tls/TlsContext.h`: owns wolfSSL CTX lifecycle, loads certs, applies config, no per-connection state.
+- `tls/TlsAcceptor.h`: non-blocking handshake on the event loop, SNI dispatch, failure counters.
+- `crypto/PasswordHasher.h`: Argon2id verify and hash, params struct, needs-rehash check.
+- `crypto/CryptoBox.h`: AES-GCM plus ChaCha20-Poly1305 encrypt and decrypt with key IDs.
+- `crypto/KeyHierarchy.h`: master key, per-purpose subkeys (cookies, sessions, CSRF, uploads), rotation epochs.
+- `crypto/SecureRandom.h`: CSPRNG bytes, token generation, constant-time compare.
+- `auth/Authenticator.h`: Basic auth parsing, credential check, login and logout, re-auth gate.
+- `auth/Permission.h` and `auth/Guard.h`: permission objects, route guards, deny-by-default checks.
+- `session/Session.h`: session value object, expiry fields, privilege level.
+- `session/SessionStore.h`: interface (get, put, rotate, destroy); `session/MemoryStore.h`, `session/DbStore.h` implement it.
+- `session/SessionManager.h`: cookie issue, rotation, idle and absolute enforcement.
+- `guard/CsrfGuard.h`: synchronizer tokens plus signed double-submit fallback, Fetch Metadata check.
+- `guard/CorsPolicy.h`: closed default, explicit origin and method allow-lists, preflight handler.
+- `guard/SecurityHeaders.h`: HSTS, framing, nosniff, referrer, permissions, CSP builder hook.
+- `guard/RateLimiter.h`: token bucket per IP, user, route; headers Retry-After.
+- `guard/BodyLimits.h`: header, body, fields, JSON depth enforcement before parsing completes.
+- `upload/UploadValidator.h`: extension allow-list, magic bytes, size and name caps, temp file handle.
+- `upload/UploadStore.h`: quarantine dir, final store outside webroot, AV hook interface.
+- `secret/SecretsProvider.h`: .env.json plus env overrides, typed validated secrets, rotation API.
+
+## Rules (hard, numbered)
+
+1. TLS 1.3 only on all server listeners; no TLS 1.2 fallback unless a major version spec reopens it.
+2. wolfSSL is the sole third-party dependency for TLS; crypto (Argon2id, AES-GCM, ChaCha20-Poly1305) is in-house, never per-feature forks.
+3. Parameterized queries only; raw SQL string concatenation with user input is a build-breaking violation (contract owned here, enforced in 04-data.md).
+4. Passwords stored as Argon2id with unique 16-byte salt; plaintext or fast hashes (MD5, SHA-1, raw SHA-256) never accepted.
+5. Session IDs are 128-bit CSPRNG, meaningless, rotated on login, privilege change, and re-auth.
+6. Cookies carrying sessions or CSRF state are Secure, HttpOnly, SameSite (Strict default, Lax opt-in).
+7. CSRF protection on every state-changing cookie route; exempted CORS or webhook endpoints must be listed explicitly with auth plus logging.
+8. CORS default deny; wildcard origin with credentials is rejected at config load.
+9. Security headers middleware is always in the default chain; removal requires explicit per-route opt-out recorded in config.
+10. All inputs capped before allocation: headers, body, fields, JSON depth, uploads; over-cap requests fail closed with 413 or 431.
+11. Uploaded files never execute: stored outside webroot, served only through an ID handler with nosniff and safe Content-Disposition.
+12. Secrets never in code, logs, or error bodies; rotation revokes old keys within one absolute session lifetime.
+
+## Development guidelines
+
+- Extremely object-oriented: one small class per file, deep namespaces, shared behavior in exactly one base.
+- Guards are middleware or router wrappers with a single `check(request)` entry; no inline security logic in handlers.
+- Every denial emits a structured log plus metric with reason code, never the secret or full token.
+- Constant-time compare for hashes, tokens, HMACs; zeroize key material on drop.
+- Blocking crypto (Argon2id verify) runs off the event loop via the blocking-work path from 01-runtime.md.
+- Config is typed and validated at startup; invalid security config aborts boot with a named error.
+- Mobile-first has no security shortcut: same guards on small-screen and desktop template paths.
+
+## Specifications (concrete behaviors, caps, formats)
+
+- Argon2id: default m=19456 KiB, t=2, p=1; stored format `$argon2id$v=19$m=...,t=...,p=...$salt$hash`; verify budget under 1 s; `needs_rehash` true when stored params are weaker.
+- Data encryption: AES-GCM with 96-bit nonce or ChaCha20-Poly1305 with 96-bit nonce; envelope `v1:key_id:nonce:ct:tag`; key rotation keeps previous epoch readable for 24 h.
+- Sessions: cookie name `__Host-molo_sid`; 128-bit ID, base64url 22 chars min; idle timeout 30 min, absolute 12 h, renewal cap 24 h; rotation on privilege change; store interface supports memory and DB backends.
+- CSRF: header `X-CSRF-Token` or form field `csrf_token` must match session-bound value; signed fallback token is `HMAC_SHA256(session_id, nonce)`; Fetch Metadata deny when `Sec-Fetch-Site: cross-site` on state-changing methods unless route is exempted.
+- CORS: default no `Access-Control-Allow-Origin`; preflight cache max 600 s; allowed methods and headers explicit lists; credentials require exact origin match.
+- Headers defaults: `Strict-Transport-Security: max-age=63072000; includeSubDomains; preload`, `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: strict-origin-when-cross-origin`, `Permissions-Policy: geolocation=(), camera=(), microphone=()`; CSP builder default `default-src 'self'; object-src 'none'; frame-ancestors 'none'`.
+- Rate limits: per-IP 100 req/min, per-user 300 req/min, per-route override; burst 2x for 10 s; 429 with `Retry-After`; counters in memory with static cap 100k buckets, LRU evict.
+- Input caps: headers 16 KB total, single header 8 KB; body 1 MB default (route override to 10 MB); form fields 1000; JSON depth 32, JSON size bound by body cap; upload default 10 MB per file, 4 files per request; filename 255 chars, generated storage name `uuid.ext`.
+- Upload checks: extension allow-list per route (default images: jpg, png, webp; docs: pdf, docx); magic-byte sniff for declared type; temp dir with 0600 perms, swept after 1 h; AV hook `scan(path) -> clean or infected`, infected goes to quarantine, never to store.
+- Auth: Basic `Authorization: Basic base64(user:pass)` parsed once, constant-time user lookup path; login rate limited 10 attempts per min per IP; high-risk actions require re-auth within last 10 min.
+- Path traversal: static and upload paths canonicalized, `..` and absolute and `:` rejected, symlink escape rejected, serve root jailed (implements plan.md static route rule).
+- TLS: min TLS 1.3, ALPN `h2,http/1.1`, H2 over TLS plus h2c cleartext per 01-runtime.md; cert reload without restart; handshake timeout 5 s; failed-handshake counter with per-IP backoff.
+
+## Execution steps (in build order)
+
+1. SecureRandom, KeyHierarchy, CryptoBox with RFC test vectors and round-trip tests.
+2. PasswordHasher Argon2id with OWASP params, salt, pepper hook, needs-rehash.
+3. TlsConfig, TlsContext, TlsAcceptor on wolfSSL; ALPN, SNI, reload, handshake caps.
+4. Session, SessionStore (memory first), SessionManager with rotation and timeouts.
+5. Authenticator plus Permission and Guard; Basic auth, login, re-auth, deny-by-default.
+6. CsrfGuard (synchronizer first, signed double-submit second, Fetch Metadata third).
+7. CorsPolicy and SecurityHeaders middleware in the default chain.
+8. RateLimiter and BodyLimits wired before routing and parsing.
+9. UploadValidator and UploadStore with quarantine and AV hook.
+10. SecretsProvider with .env.json, env overrides, rotation; audit pass over all denials and logs.
+
+## Validation (how to verify each step)
+
+- Steps 1 to 2: RFC 9106 Argon2id test vectors pass; OWASP params confirmed in config dump; wrong-password, wrong-pepper, weak-params cases covered; verify latency under 1 s on reference host; integration test through a login URL, not a unit-only hash call.
+- Step 3: real TLS 1.3 handshake against the server with valid and expired and wrong-SNI certs; ALPN negotiates h2 and http/1.1; TLS 1.2 client fails closed; cert reload keeps existing connections alive.
+- Steps 4 to 5: login rotates session ID; privilege change rotates again; idle expiry at 31 min and absolute at 12 h enforced through real requests; Basic auth good and bad credentials; guarded route denies anonymous with 401 or 403 and structured log.
+- Step 6: POST without token fails 403; token from another session fails; signed double-submit passes with matching cookie and header; cross-site Fetch Metadata POST blocked; exempted webhook passes with auth and audit entry.
+- Step 7: default responses carry all six header families; CORS preflight from unlisted origin gets no allow header; wildcard-with-credentials config refuses to boot.
+- Step 8: burst over per-IP cap yields 429 with Retry-After; oversized body yields 413 before handler runs; JSON depth 33 rejected; header flood yields 431.
+- Step 9: double extension `x.jpg.php`, null byte, case trick `x.pHp`, and spoofed Content-Type all rejected; valid file stored outside webroot under generated name and served via ID handler; infected fixture quarantined via stub AV hook; quota overage rejected.
+- Step 10: secrets load from .env.json with env override winning; rotation invalidates old cookie crypto epoch; no secret appears in logs or error pages under failure injection; full integration suite green in the same session.
+
+## References
+
+- Depends on 01-runtime.md for event loop off-loop work, cookies, static serving, H2 and ALPN, lifecycle caps.
+- Depends on 02-observability.md for structured denial logs, auth and TLS metrics, trace linkage.
+- Feeds 04-data.md the parameterized-query rule and DB session store backend.
+- Feeds 05-request-routing.md guards, validation caps, error mapping, CSP builder for templates.
+- Feeds 06-realtime.md auth and session behavior reused by WebSocket, SSE, Socket.IO.
+- Feeds 07-supervisor-replication.md session sharing across instances and secret distribution.
+- Feeds 09-cli-ops.md health endpoints that must bypass auth safely and i18n-safe error pages.
+- Demonstrated end to end by 10-workflow-example.md living example and its integration suite.
+
+## Open questions
+
+1. Pepper management: single global pepper with forced reset on rotation, or per-epoch peppers with dual-verify window? Leaning to per-epoch with 24 h overlap.
+2. Session store default in production: DB-backed sessions or a Redis-like external store? DB first keeps zero extra deps; external store may follow in 04-data.md.
+3. AV hook shape: synchronous `scan(path)` blocking upload response, or async quarantine-then-release with event? Async fits non-blocking rule but complicates UX.
+4. Rate limit sharing across instances: local buckets only for v1, or replicated counters in step 6? Local first, document divergence.
+5. Commercial wolfSSL licensing integration into CI builds: vendored tarball or system package? Needs a build-owner decision.
+
+## Sources
+
+- OWASP Password Storage Cheat Sheet, https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html (retrieved 2026-10-23)
+- OWASP Session Management Cheat Sheet, https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html (retrieved 2026-10-23)
+- OWASP Cross-Site Request Forgery Prevention Cheat Sheet, https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html (retrieved 2026-10-23)
+- OWASP File Upload Cheat Sheet, https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html (retrieved 2026-10-23)
+- OWASP HTTP Headers Cheat Sheet, https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html (retrieved 2026-10-23)
+- RFC 9106 Argon2 Memory-Hard Function, https://www.rfc-editor.org/rfc/rfc9106.html (retrieved 2026-10-23)
+- wolfSSL repository (TLS 1.3, wolfCrypt, licensing), https://github.com/wolfSSL/wolfssl (retrieved 2026-10-23)

plans/04-data.md

@@ -0,0 +1,130 @@
+# 04 - Data Layer
+
+Scope: plan.md sections 2, 5, 10 step 3. Topics: ORM dual backends, self-healing, pooling, transactions, KV, cache, pubsub.
+
+## Purpose and philosophy
+
+One data layer serves every feature: ORM models, sessions, KV, cache, pubsub, uploads metadata, task queues. Two SQL backends (Postgres, SQLite) hide behind one declared interface so application code never branches on the driver. The framework owns both drivers dependency-free: its own Postgres wire-protocol client and its own SQLite handling, no external driver libraries.
+
+Philosophy: parameterized always, blocking never, healing automatic, audit preserved. Every query goes through the extended protocol path with bound parameters. Every I/O operation is non-blocking on the event loop (see 01-runtime.md). Schema mismatches heal themselves at runtime and emit a migration record for review, so dev and production converge without manual DDL. Caching is explicit: reads declare their source, writes invalidate through hooks, nothing is magically stale.
+
+## Best practices (from research)
+
+1. Postgres wire protocol: frontend/backend exchange typed messages (Startup, Parse, Bind, Describe, Execute, Sync); the extended query flow separates SQL text from parameters, which is the structural defense against SQL injection (Source 1).
+2. Pooling modes matter: session pooling supports all Postgres features; transaction pooling reassigns the server connection after each transaction but breaks session state (SET, LISTEN, advisory locks, SQL-level PREPARE); protocol-level prepared plans still work under transaction pooling (Source 6). Default to transaction pooling and forbid session-state features unless a session-pinned checkout is requested.
+3. MVCC: each statement sees a snapshot; readers never block writers and writers never block readers, even at serializable via SSI (Source 5). Design transactions around short snapshots, optimistic conflict handling, and explicit isolation levels per transaction.
+4. SQLite WAL: readers do not block writers and writers do not block readers; I/O is more sequential with fewer fsync calls; all processes must share one host (no network filesystem); SQLITE_BUSY can still occur; checkpoints bound WAL growth (Source 2). Run WAL mode always, one writer at a time, busy timeout plus retry, scheduled checkpoints.
+5. SQLite schema change is intentionally narrow: ADD COLUMN and DROP COLUMN exist, but most alterations require the documented table-rebuild procedure (create new table, copy, drop, rename, rebuild indexes) inside a transaction (Source 3). The self-healer follows exactly this procedure for SQLite widen operations.
+6. Pubsub delivery is at-most-once with no persistence: a message published with no subscribers is lost; ordering holds per publisher-to-channel; pattern subscriptions and sharded channels scale fan-out (Source 4). In-process pubsub mirrors these semantics; durability is a separate opt-in owned by 07-supervisor-replication.md, never silently implied here.
+7. Triangulated decision: Postgres ADD COLUMN without volatile default is metadata-only and safe to auto-apply; rewriting ALTER TABLE forms are not MVCC-safe under concurrency (Source 5 context, Postgres ALTER TABLE docs). The healer therefore prefers additive DDL and schedules rewrites behind an advisory lock with retry.
+
+## Structure
+
+Root: `molo/data/`, namespace `molo::data`. One class or concern per file, deep over flat.
+
+- `connection.hpp`: `Connection` abstract base (exec, query, prepare, ping, backend id, escaping off-limits).
+- `pool.hpp`, `pool_config.hpp`: `Pool`, `PoolConfig` (min/max size, acquire timeout, idle eviction, health probe query).
+- `transaction.hpp`: `Transaction` RAII guard (begin, commit, rollback, isolation level, nesting via savepoints).
+- `value.hpp`, `row.hpp`, `result.hpp`: `Value` variant (null, int64, double, string, bytes, bool, time), `Row`, `Result` (rows, affected count, last insert id).
+- `error.hpp`: `DbError` with backend code, SQLSTATE for Postgres, extended result code for SQLite, healable flag.
+- `drivers/pg/conn.hpp`, `.../message.hpp`, `.../auth.hpp`, `.../extended.hpp`, `.../codecs.hpp`: wire client (startup, SASLprep SCRAM-SHA-256 plus cleartext fallback refused by default, simple query, Parse/Bind/Execute/Sync, binary and text codecs for core types).
+- `drivers/sqlite/db.hpp`, `.../wal.hpp`, `.../busy.hpp`: SQLite handling (open flags, WAL activation, busy timeout, checkpoint scheduler, single-writer gate).
+- `orm/model.hpp`: `Model` CRTP base (table name, primary key, field list, hooks).
+- `orm/field.hpp`: `Field<T>` descriptors (type, length, null, default, unique, index, check).
+- `orm/query.hpp`: `Query<T>` builder (filter, order, limit, offset, join, select columns, count, exists).
+- `orm/relations.hpp`: `BelongsTo`, `HasMany`, `ManyToMany` with lazy and eager (`preload`) loading.
+- `orm/schema.hpp`: `Schema` introspection (tables, columns, indexes) per backend.
+- `orm/sync.hpp`: `Syncer` (create missing tables/columns/indexes from models).
+- `orm/healer.hpp`: `Healer` (classify error, plan DDL, apply, retry once, emit migration record).
+- `orm/migration.hpp`, `orm/migrator.hpp`: `Migration` record (id, timestamp, SQL up, reason, checksum) and applier.
+- `kv/store.hpp`, `kv/memory.hpp`: `KvStore` interface (get, set with TTL, del, exists, scan prefix) and in-memory backend.
+- `cache/cache.hpp`, `cache/policy.hpp`, `cache/singleflight.hpp`: `Cache` (get-or-compute with TTL, explicit invalidate by key/tag, LRU with cap, stampede guard).
+- `pubsub/broker.hpp`, `pubsub/channel.hpp`, `pubsub/subscription.hpp`: `Broker` (publish, subscribe exact and pattern, unsubscribe), fan-out on loop, backpressure drops counted, never blocking.
+
+## Rules (hard, numbered)
+
+1. No raw SQL string concatenation with user input anywhere. All values travel as bound parameters via the extended protocol (Postgres) or bound host parameters (SQLite). Raw SQL is allowed only as constant fragments with bound values, reviewed and flagged `raw()`.
+2. No blocking I/O on the event loop. Drivers are async over the reactor from 01-runtime.md; SQLite file I/O uses the documented off-loop worker with a bounded queue.
+3. One writer at a time per SQLite database file. Enforced by an async mutex in the driver, plus `busy_timeout` as second line; SQLITE_BUSY after retries is a loud error with a counter, never silent.
+4. Self-heal retries a failed statement at most once per statement execution. If the retry fails, the error propagates with the migration record attached.
+5. The healer only widens, never narrows: add tables/columns/indexes, widen types, extend lengths, drop NOT NULL with default backfill. Narrowing, renames, and drops require an explicit human-authored migration.
+6. Every heal action emits a migration record (SQL text, reason, triggering error code, timestamp, checksum) to the migration journal before retry. No journal write, no retry.
+7. Transactions default to READ COMMITTED; SERIALIZABLE is opt-in per transaction. Nested scopes use savepoints with unique names. A Transaction that goes out of scope without commit rolls back.
+8. Pool checkouts have timeouts and every pool has static caps (size, queue, lifetime). Exhaustion fails fast with a structured error and metric, never waits forever.
+9. Cache entries always carry TTL and a size cap; invalidation is explicit through ORM write hooks and tag purges. No unbounded maps, no infinite TTL.
+10. Pubsub is at-most-once and non-persistent by default. Any code needing delivery guarantees must use the durable channel from 07-supervisor-replication.md; this layer must not pretend to persist.
+11. All drivers handle credentials as opaque secrets handed from 03-security.md. Connection strings are redacted in logs; auth failures emit one structured event without the secret.
+12. Every public data API takes a `Context` carrying deadline, correlation id, and tenant/database selector from 02-observability.md. No hidden globals for the current connection.
+
+## Development guidelines
+
+- Build the driver interface first, then Postgres wire client, then SQLite handling, then pooling, then ORM core, then sync/healer, then KV/cache/pubsub. Each step compiles and passes integration tests before the next starts.
+- Types map explicitly: C++ int64, double, string, bytes, bool, chrono time map to Postgres BIGINT, DOUBLE PRECISION, TEXT/VARCHAR, BYTEA, BOOLEAN, TIMESTAMPTZ and to SQLite INTEGER, REAL, TEXT, BLOB, INTEGER 0/1, TEXT ISO-8601. The mapping table lives in `orm/types.md` next to the code and is the single source of truth.
+- Error classification is a pure function `classify(backend, code) -> HealPlan | Fatal`, unit-covered through the real stack per plan.md section 8 (integration tests that trigger real 42P01, 42703, 22001 on Postgres and real "no such table/column" on SQLite).
+- Migrations are append-only files `migrations/NNNN_name.sql` plus a `schema_migrations` table recording applied checksums. Generated heals land in `migrations/auto/` for review.
+- KV and cache share the `KvStore` interface; cache adds TTL enforcement, tags, and singleflight. New backends implement the interface, never fork callers.
+- Pubsub channels are plain strings `topic:sub` with `*` single-segment wildcard for patterns. Payload is bytes plus content-type hint; JSON is a helper, not the wire format.
+- Observe everything through 02-observability.md: per-query latency histogram, pool saturation gauge, heal counter by plan kind, cache hit ratio, pubsub publish/drop counters. All zero-cost when disabled.
+
+## Specifications (concrete behaviors, caps, formats)
+
+- Pool defaults: Postgres min 2, max 32 connections per process; SQLite max 1 writer plus 8 readers; acquire timeout 5 s; idle eviction 60 s; max connection lifetime 30 min; max queued waiters 256, overflow fails fast. All ceilings static in `PoolConfig`, tunable via validated config only.
+- Query limits: statement timeout 30 s default (per-Context override, hard ceiling 300 s); max bound parameters 1024; max identifier length 63 chars (Postgres limit); max row payload 64 MiB; max rows per `Query::all()` 10,000 with mandatory cursor/pagination beyond.
+- Transaction semantics: `co_await txn.commit()` flushes on the owning connection; the connection stays pinned for the transaction lifetime (session-pinned checkout); savepoint names `molo_sp_<depth>_<counter>`; serialization failures (SQLSTATE 40001) surface as retryable `DbError`, the caller decides, the layer never auto-retries user transactions.
+- Sync behavior: at startup (and on demand) the Syncer diffs models against introspected schema and applies additive DDL inside one transaction per table; failures abort startup in production unless `data.sync_on_boot=false`, in which case a loud warning plus health-check downgrade results.
+- Heal matrix: undefined_table -> CREATE TABLE; undefined_column -> ADD COLUMN; string truncation (22001) -> extend VARCHAR length x4 up to TEXT promotion; type mismatch int vs bigint -> widen; SQLite equivalents via ADD COLUMN or table-rebuild procedure. Anything else -> fatal, no DDL attempted.
+- SQLite specifics: `PRAGMA journal_mode=WAL`, `synchronous=NORMAL`, `foreign_keys=ON`, `busy_timeout=5000`; auto-checkpoint every 1000 pages; startup verifies same-host file (refuses network paths); checkpoint failures counted and surfaced.
+- KV format: keys UTF-8, max 512 bytes, max 1,048,576 keys per store (LRU eviction beyond); values max 16 MiB; TTL seconds, max 30 days; `scan(prefix)` paginated, max 1000 keys per page.
+- Cache: default TTL 60 s; max 131,072 entries and 512 MiB total (whichever hits first evicts LRU); tags max 16 per entry; negative caching opt-in with max 10 s TTL; singleflight coalesces concurrent computes per key.
+- Pubsub: channel name max 256 chars; payload max 1 MiB; per-subscriber queue 1024 messages then oldest-drop with `pubsub.dropped` counter; pattern subscriptions capped at 1024 per broker; publish never awaits subscribers.
+- Replication hooks: every ORM write emits an invalidation event (table, pk, op, tx id) on the internal bus consumed by cache and by 07-supervisor-replication.md; ordering key is `(tx_commit_seq, statement_seq)`.
+
+## Execution steps (in build order)
+
+1. Value/Row/Result/Error types plus DbError classification table skeleton.
+2. Postgres wire client: startup, auth (SCRAM-SHA-256), simple query, then Parse/Bind/Execute/Sync with text codecs, then binary codecs.
+3. SQLite handling: open, pragmas, WAL activation, busy handler, single-writer gate, checkpoint scheduler.
+4. Pool and Transaction: pooling modes (transaction default, session-pinned), savepoints, timeouts, health probes.
+5. ORM core: Model, Field, Query builder, relations, schema introspection for both backends.
+6. Syncer plus migration journal and migrator (apply, checksum verify, audit list).
+7. Healer: classify, plan, apply DDL, retry once, emit record; heal matrix covered per backend.
+8. KV store interface plus memory backend; cache with TTL, tags, LRU, singleflight; ORM write-through invalidation hooks.
+9. Pubsub broker: exact and pattern subscriptions, fan-out, drop counters, backpressure behavior.
+10. Hardening pass: caps audit, timeout audit, redaction audit, metrics wiring, docs and living-example models.
+
+## Validation (how to verify each step)
+
+- Steps 1-4: integration tests through real servers (ephemeral Postgres via loopback, temp-file SQLite): connect, auth failure redaction, 10k sequential and 128 concurrent queries, pool exhaustion fails fast, transaction commit/rollback/savepoint behavior, statement timeout fires, idle eviction observed.
+- Step 5: CRUD plus relations plus eager loading against both backends with identical assertions; parameterized-query proof (injection strings stored literally, never executed); join and pagination correctness.
+- Step 6: fresh database boots to full schema on both backends; checksum mismatch aborts with loud error; migration list reproducible.
+- Step 7: fault-injection suite drops a column/table, shortens a VARCHAR, narrows an int on each backend, then asserts auto-heal plus one retry success plus journal record; narrowing request refused; second failure propagates with record attached.
+- Step 8: TTL expiry, tag purge, LRU eviction at cap, stampede test (100 concurrent computes collapse to 1), ORM write invalidates cached read across both backends.
+- Step 9: 10k messages fan-out to 100 subscribers in order; slow subscriber drops counted, publisher never blocks; pattern delivery correct; no-subscriber publish is a counted no-op.
+- Step 10: caps review checklist signed, metrics present in Prometheus export (see 09-cli-ops.md), living example exercises every behavior above in its suite.
+
+## References (sibling files this depends on or feeds)
+
+- Depends on 01-runtime.md: event loop, timeouts, off-loop worker scheduling, connection lifecycle primitives.
+- Depends on 02-observability.md: log format, correlation ids, metrics and tracing macros, per-query instrumentation.
+- Depends on 03-security.md: credential handling, redaction rules, query-safety policy (parameterized-only), encryption for secrets at rest.
+- Feeds 05-request-routing.md: validation binding to models, DI accessors for DB/cache, error-to-status mapping for DbError.
+- Feeds 06-realtime.md: pubsub transport under WebSocket/SSE fan-out.
+- Feeds 07-supervisor-replication.md: invalidation event stream, durable pubsub extension point, replication conflict input (row versions, tx ids).
+- Feeds 08-clients-openapi.md: model-to-schema derivation for OpenAPI, test-client DB fixtures.
+- Feeds 10-workflow-example.md: integration-test templates for data behaviors, living-example models and seed data.
+
+## Open questions
+
+1. Should the Postgres wire client support SCRAM channel binding (tls-server-end-point) in v1, or is plain SCRAM-SHA-256 acceptable with TLS 1.3 underneath?
+2. Do we need logical-replication output-plugin consumption in this layer for v1, or is the ORM write-event stream sufficient input for 07-supervisor-replication.md replication?
+3. SQLite BEGIN CONCURRENT (multiple writers in WAL2) is experimental upstream; do we track it or keep single-writer permanently?
+4. Cache backend choice for multi-instance v1: shared SQLite file, or a networked memory protocol owned by 07-supervisor-replication.md? This layer defines the interface either way.
+
+## Sources
+
+1. PostgreSQL 18 documentation, Chapter 54 Frontend/Backend Protocol (message framing, startup, simple and extended query flows). https://www.postgresql.org/docs/18/protocol.html (retrieved 2026-10-23).
+2. SQLite documentation, Write-Ahead Logging (concurrency, checkpointing, SQLITE_BUSY, same-host requirement). https://www.sqlite.org/wal.html (retrieved 2026-10-23).
+3. SQLite documentation, ALTER TABLE (ADD/DROP COLUMN, table-rebuild procedure for other changes). https://www.sqlite.org/lang_altertable.html (retrieved 2026-10-23).
+4. Redis documentation, Pub/sub (at-most-once delivery, ordering, pattern subscriptions, sharded pubsub). https://redis.io/docs/latest/develop/pubsub/ (retrieved 2026-10-23).
+5. PostgreSQL 18 documentation, 13.1 Concurrency Control Introduction (MVCC snapshots, readers never block writers, SSI). https://www.postgresql.org/docs/current/mvcc-intro.html (retrieved 2026-10-23).
+6. PgBouncer Features (session vs transaction vs statement pooling, SQL feature compatibility map). https://www.pgbouncer.org/features.html (retrieved 2026-10-23).

plans/05-request-routing.md

@@ -0,0 +1,302 @@
+# 05 - Request/Response, Routing, Views, Templates
+
+Scope: plan.md sections 2, 6 partial, 10 step 5 partial.
+Topics: app, routers, views, validation/forms, negotiation, templates, middleware, DI, errors.
+
+## 1. Purpose and philosophy
+
+This file owns everything between "bytes parsed by the runtime" and "value returned by user code": the
+`Application` composition root, FastAPI-style routers, aiohttp-style class-based views, the single
+validation framework (Pydantic-style models plus HTML/JSON forms), the response hierarchy with content
+negotiation, the Jinja-like template engine, the middleware onion, dependency injection without globals,
+and the typed error hierarchy.
+
+Philosophy:
+
+- One request object, one response hierarchy. Handlers never touch sockets; they take a `Request` and
+  return a `Response`. Streaming, SSE and realtime build on the same pair (see 06-realtime.md).
+- Declarative over imperative. Routes, validation models, negotiation and DI bindings are declared;
+  the framework binds, validates, negotiates and injects before user code runs.
+- A single validation framework feeds path, query, header, cookie and body binding, forms, and OpenAPI
+  schema generation (see 08-clients-openapi.md). No second ad-hoc parser anywhere.
+- Templates are logic-light text renderers with secure defaults: auto-escape on for HTML/XML, explicit
+  `safe` opt-out only.
+- Errors are typed values, not strings. Every framework exception maps to exactly one HTTP status and
+  renders as JSON for API clients or HTML for browsers, chosen by negotiation.
+
+## 2. Best practices (from research)
+
+- Routers group related endpoints and mount them on the app with a shared prefix, tags and dependencies.
+  FastAPI documents `APIRouter` plus `app.include_router(router)` as the scaling pattern for bigger
+  applications, where the include call applies prefix, tags, dependencies and responses to every route
+  in the router (Source: https://fastapi.tiangolo.com/tutorial/bigger-applications/, retrieved 2026-10-23).
+- Class-based views derive from a base `View` holding the request and implement one coroutine per HTTP
+  method (`get`, `post`, ...). aiohttp documents `class MyView(web.View)` with `async def get(self)`
+  returning a response, registered on the router like a function handler
+  (Source: https://docs.aiohttp.org/en/stable/web_quickstart.html, retrieved 2026-10-23).
+- Validation guarantees the output, not the input: untrusted data is parsed and coerced into a model
+  instance whose fields conform to declared types and constraints, else a single error listing all
+  failures is raised. Pydantic v2 states models this way: fields are annotated attributes on a
+  `BaseModel` subclass, and a `ValidationError` is raised when data cannot be parsed into an instance
+  (Source: https://pydantic.dev/docs/validation/latest/concepts/models/, retrieved 2026-10-23).
+- Templates center on one reusable `Environment` holding loaders, filters, tests and globals; syntax is
+  `{{ ... }}` for expressions, `{% ... %}` for statements, `{# ... #}` for comments, with inheritance
+  (`extends`, `blocks`), includes, `for`/`if`, and autoescaping selectable per template name/extension
+  (Source: https://jinja.palletsprojects.com/en/stable/templates/, retrieved 2026-10-23).
+- Content negotiation is server-driven: the client sends `Accept` with media ranges and optional `q`
+  weights, the server picks one proposal and answers with `Content-Type`, or `406 Not Acceptable`
+  when nothing matches (Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Content_negotiation,
+  retrieved 2026-10-23; triangulated by RFC 9110, Section 12, which obsoletes RFC 7231,
+  Source: https://www.rfc-editor.org/rfc/rfc9110.html, retrieved 2026-10-23).
+- Middleware forms an onion: each layer sees the request before the endpoint and the response after it,
+  and may short-circuit. Ordering rule derived from plan.md layering: exception mapping outermost,
+  then observability, security, session/auth, routing-adjacent concerns innermost.
+
+## 3. Structure
+
+Directories (one class or concern per file, deep over flat):
+
+```
+src/molo/app/          Application, sub-app mounting, lifespan, plugin registry
+src/molo/routing/      Router, Route, path compiler, matchers, url_for
+src/molo/http_msg/     Request, Response hierarchy, Headers, Query, Cookies view
+src/molo/views/        View base, method dispatch, handler adapters
+src/molo/validate/     Model, Field, validators, ValidationError, coercions
+src/molo/forms/        Form, JSON binding, urlencoded/multipart binding
+src/molo/negotiate/    Accept parser, q-sorting, format selection
+src/molo/templates/    Environment, Loader, Lexer, Parser, Renderer, Filters, Tests
+src/molo/middleware/   Middleware base, Chain, built-in middlewares owned here
+src/molo/di/           Container, Scope, Provider, LuckilyNoGlobals
+src/molo/errors/       HttpError hierarchy, handler, renderers (JSON, HTML)
+```
+
+Namespaces and key classes:
+
+- `molo::app::Application`: owns routers, middleware chain, DI container, template env, error handler.
+  `include_router(prefix, router)`, `mount(path, sub_app)`, `on_startup/on_shutdown` hooks.
+- `molo::routing::Router`: `get/post/put/patch/delete/head/options(path, handler)`; nested
+  `include_router`; `Route` record (methods, compiled path, handler, name, validation models, tags).
+- `molo::http_msg::Request`: method, path, query, headers, cookies, `get<T>(key)` for query/form,
+  `json<T>()`, `form<T>()`, body access with caps from 01-runtime.md/03-security.md; `Response` base plus `HtmlResponse`,
+  `JsonResponse`, `XmlResponse`, `TextResponse`, `RedirectResponse`, `FileResponse`,
+  `StreamingResponse` (streaming mechanics live in 06-realtime.md, the class shell lives here).
+- `molo::views::View`: constructed with `Request`; virtual `get/post/...` default to 405; `dispatch()`
+  selects by method. Function handlers adapt to the same interface.
+- `molo::validate::Model`: field descriptors with type, constraints, default, required; `validate()`
+  returns model or throws `ValidationError` listing every field failure. Used for path/query/body.
+- `molo::forms::Form`: binds `application/json`, `application/x-www-form-urlencoded`,
+  `multipart/form-data` (file parts delegate to upload handling in 03-security.md) into a `Model`.
+- `molo::negotiate::Negotiator`: parses `Accept`, sorts by q, matches handler-declared formats.
+- `molo::templates::Environment`: loaders, filters, tests, globals, autoescape predicate, cache.
+- `molo::middleware::Middleware`: `handle(Request, Next) -> Response`; `Chain` composes in order.
+- `molo::di::Container`: request-scoped and app-scoped providers, constructor injection for views.
+- `molo::errors::HttpError(status, code, message, details)`; `ValidationHttpError` (422);
+  `ErrorHandler` maps exception to status plus negotiated JSON/HTML body.
+
+## 4. Rules (hard, numbered)
+
+- R1: Every route handler is a coroutine taking `Request` (or a `View` holding one) and returning a
+  `Response`. No handler touches sockets, the event loop, or globals.
+- R2: All input binding (path, query, header, cookie, JSON body, form body) goes through
+  `molo::validate`. A second parser for the same input is a defect.
+- R3: `ValidationError` lists ALL field failures in one response, never fail-fast on the first field.
+- R4: Responses are chosen from the `Response` hierarchy only. Raw string returns from handlers are
+  rejected at compile time (no implicit conversion to 200 text).
+- R5: Template autoescaping is ON for `.html`, `.htm`, `.xml` and for all streamed template output.
+  `safe` bypass requires naming the value safe at the call site; no global disable flag.
+- R6: Content negotiation runs for every response with more than one declared format. Explicit
+  `?format=` or path suffix overrides `Accept` only when the route declares `allow_format_override`.
+  No match yields `406` with a JSON or HTML body listing available types.
+- R7: Middleware order is fixed by category (errors outer, then observability, security, session/auth,
+  user middleware, router innermost). Reordering across categories is a compile-time or startup error.
+- R8: DI provides DB, cache, auth, templates and settings. No service singletons, no file-static
+  mutable state reachable from handlers. Request scope ends at response completion.
+- R9: Every framework exception maps to exactly one HTTP status via the error table in section 6.
+  Unmapped exceptions become 500 and are logged with correlation ID; stack details never leak to clients.
+- R10: Route names are unique per application (including mounted sub-apps after path merge). Duplicate
+  names fail at startup with the colliding paths quoted.
+- R11: All caps in section 6 are static constants with documented defaults. No autotuning, no silent
+  growth.
+- R12: No feature in this file duplicates scope owned by a sibling: no TLS/auth/session logic (03-security.md),
+  no ORM/KV/cache (04-data.md), no WebSocket/SSE/Socket.IO mechanics (06-realtime.md), no OpenAPI emission (08-clients-openapi.md), no CLI
+  (09-cli-ops.md). This file exposes the hooks those siblings consume.
+
+## 5. Development guidelines
+
+- Build in the order of section 7; each step ends green with integration tests through real HTTP.
+- Prefer small composable classes: one concern per file, behavior in exactly one base class, features
+  composed through declared interfaces (plan.md section 9).
+- Headers follow the runtime: case-insensitive lookup, size and count caps enforced in 01-runtime.md before this
+  layer sees them.
+- Keep the happy path allocation-light: route match without heap allocation per request where feasible;
+  template rendering reuses the compiled template cache.
+- Every public API gets a usage example in the living example project (see 10-workflow-example.md)
+  in the same change that introduces it.
+- Docs state current behavior with gaps listed next to the code that closes them; no aspirational APIs
+  in headers.
+
+## 6. Specifications (concrete behaviors, caps, formats)
+
+Route matching:
+
+- Path syntax: `/users/{id:int}` with converters `int`, `float`, `str` (single segment), `path`
+  (remainder, must be last). Unknown converter is a startup error naming the route.
+- Matching order: static segments first, then converters in registration order, then `path` catch-alls.
+  Method mismatch on an existing path yields 405 with an `Allow` header, not 404.
+- Trailing slash: strict. `/users` and `/users/` are different routes unless the route sets
+  `slash_insensitive=true`, which registers both and canonical-redirects (308) to the declared form.
+- Sub-app mount merges paths: mounting app B at `/app1` inside A prefixes all B routes with `/app1`;
+  triple nesting yields `/app1/app1/app1` automatically. Mounted apps keep their own DI scope but share
+  the parent middleware chain plus their own appended layers.
+
+Validation and forms:
+
+- Field constraints: `min/max` (numbers), `min_len/max_len` (strings/collections), `pattern` (regex),
+  `choices`, `gt/ge/lt/le`, `required`, `default`, `nullable`. Coercion: strict for path params
+  (no `"1abc"` to int), lenient-but-documented for query/form strings.
+- `ValidationError` JSON shape: `{"detail":[{"loc":["query","age"],"msg":"...","type":"..."}]}`,
+  status 422. HTML rendering shows the same list in a 422 page.
+- Form caps: max fields 1000, max field name 256 chars, max non-file value 1 MiB, max JSON depth 32,
+  max JSON document 10 MiB default (overridable per route down to 0 = reject bodies). File parts use
+  the upload quotas from 03-security.md.
+
+Negotiation:
+
+- Parse `Accept` into (range, q, params); sort by q desc, then specificity (exact beat type star beat
+  star star), then handler declaration order. Missing `Accept` means `*/*`.
+- Handler declares `produces = {HTML, JSON}` etc. First negotiated match wins. No match: 406 with
+  `{"detail":{"available":["text/html","application/json"]}}` or the HTML equivalent.
+- `Content-Type` of requests must match a route-declared `consumes` set when declared, else 415.
+
+Templates (Jinja-compatible subset, in-house engine):
+
+- Delimiters: `{{ }}`, `{% %}`, `{# #}`. Constructs: `extends`, `block`/`endblock`, `include`, `if`/
+  `elif`/`else`/`endif`, `for`/`else`/`endfor` with `loop` variable, `set`, `macro`, filters with `|`,
+  tests with `is`, `autoescape` blocks. Undefined variables render per policy: `chain` default renders
+  empty string and logs at debug; `strict` mode (dev default) throws `UndefinedError` (500 in prod
+  rendering, startup warning listing the template).
+- Filters shipped: `escape`, `safe`, `upper`, `lower`, `trim`, `striptags`, `urlencode`, `length`,
+  `default`, `join`, `date`, `tojson`. Tests shipped: `defined`, `none`, `string`, `number`,
+  `sequence`, `mapping`, `even`, `odd`.
+- Caps: max template size 1 MiB, max include depth 16, max extends chain 8, max loop iterations 100000,
+  max render output 32 MiB, compiled-template cache max 512 entries with LRU eviction. Recursion beyond
+  depth aborts with 500 and a metric increment, never a stack overflow.
+- Loaders: filesystem directory, embedded assets (from 09-cli-ops.md embedder), in-memory dict. Dev reloads from
+  disk per render when `dev_reload=true`; prod serves compiled cache only.
+
+Middleware and DI:
+
+- Signature: `Task<Response> handle(Request req, Next next)`. `Next` invokes the remainder exactly once;
+  calling twice is a runtime error. Middleware may replace request/response but must preserve the
+  correlation ID (owned by 02-observability.md).
+- Built-ins owned here: `ExceptionMappingMiddleware` (outermost), `NegotiationMiddleware`,
+  `CORSPassthrough` is NOT here (03-security.md owns CORS policy; this file only preserves its headers),
+  `RequestIdEcho` helpers defer to 02-observability.md. User middleware registers with an explicit category tag.
+- DI scopes: `app` (one per Application), `request` (one per request, destroyed after response).
+  Providers are factories or values; circular dependencies fail at startup with the cycle quoted.
+
+Errors:
+
+- Mapping table: 400 `BadRequest`, 401 `Unauthorized`, 403 `Forbidden`, 404 `NotFound` (includes
+  `url_for` miss at render time), 405 `MethodNotAllowed` (sets `Allow`), 406 `NotAcceptable`,
+  408/413/414/431 from runtime caps pass through, 415 `UnsupportedMediaType`, 422 `ValidationFailed`,
+  429 from 03-security.md passes through, 500 `InternalError`, 503 `Unavailable`.
+- JSON error shape: `{"error":{"code":"not_found","message":"...","request_id":"..."}}`.
+  HTML error shape: responsive error page with the same fields, no internals.
+- Redirects: `RedirectResponse(url, permanent)` emits 308/307 (method-preserving). Flash messages ride
+  the session (03-security.md) via `flash(request, level, message)` and render once.
+
+Global caps (static, overridable by config, never autotuned):
+
+- max routes per app 100000; max path length 2048 chars; max path params per route 32;
+  max middleware layers 64; max sub-app nesting depth 8; max DI providers 4096.
+
+## 7. Execution steps (in build order)
+
+- E1: `Request`, `Headers`, `Query`, cookie view, `Response` base plus Text/JSON/HTML/XML/Redirect.
+  No routing yet; handlers invoked directly in tests.
+- E2: Path compiler plus `Router` (static, converters, 404/405, Allow header), `url_for` by name.
+- E3: `Application`, `include_router` with prefix/tags/dependencies, sub-app `mount` with path merge,
+  startup validation (duplicate names, unknown converters, cycles).
+- E4: `molo::validate` (Model, Field, coercions, full-error `ValidationError`) bound to path, query,
+  headers, cookies.
+- E5: `molo::errors` (hierarchy, mapping table, JSON plus HTML renderers, global handler).
+- E6: `molo::negotiate` plus `NegotiationMiddleware`, `consumes/produces` declarations, 406/415 paths.
+- E7: `molo::views::View` with method dispatch, function-handler adapter, redirects, flash helpers.
+- E8: Middleware `Chain` with category ordering enforcement and `Next`-once guard.
+- E9: `molo::di` (app and request scopes, cycle detection) wired into views and handlers.
+- E10: `molo::forms` (JSON, urlencoded, multipart field parts) on top of validation.
+- E11: Template engine: lexer, parser, renderer, Environment, loaders, filters/tests, autoescape,
+  caps and compiled cache. `TemplateResponse(template, context)` as a `Response` subclass.
+- E12: Plugin/extension registry on `Application` (named hooks: route_added, app_started, error_render)
+  with capability declarations so plugins cannot reach undeclared internals.
+
+## 8. Validation (how to verify each step)
+
+Integration tests only, through the real stack (real HTTP, routing, middleware), per plan.md section 8.
+Test files mirror routes: `tests/routes/<router>/<route>_test.cpp`.
+
+- V1: GET/POST round-trips return correct status, headers and bodies for each response subclass.
+- V2: static, converter and catch-all matches; 404 body shape; 405 with exact `Allow` value;
+  `url_for` reverses every named route including converted params.
+- V3: prefixed includes; triple-nested mount produces `/app1/app1/app1`; duplicate names and unknown
+  converters fail startup with quoted diagnostics.
+- V4: valid input binds; invalid input yields one 422 listing every bad field (path, query, header,
+  cookie cases each covered).
+- V5: each mapped exception renders the exact JSON shape and the HTML page; unknown exception yields
+  500 JSON/HTML without internals; `request_id` present in both.
+- V6: q-weighted `Accept` fixtures pick the expected format; no-match yields 406 listing available
+  types; wrong `Content-Type` yields 415.
+- V7: View dispatches each method; unimplemented method yields 405; redirects preserve method (307/308);
+  flash message appears exactly once across two requests.
+- V8: middleware order test asserts onion sequence; double-`Next` call fails loudly; short-circuit
+  returns without reaching the handler.
+- V9: request scope isolation across two concurrent requests; app scope shared; cycle fixture fails
+  startup quoting the cycle.
+- V10: JSON, urlencoded and multipart forms bind to the same model; cap fixtures (1001 fields, depth
+  33, oversized value) are rejected with the documented status.
+- V11: template fixtures cover extends/blocks, include, if/for/else, filters, tests, macros; XSS
+  fixture asserts escaping; `safe` fixture asserts opt-out; cap fixtures (depth 17, 100001 iterations)
+  abort safely; dev reload picks up an edited file, prod serves cache.
+- V12: sample plugin receives hooks in order and is denied undeclared access.
+
+## 9. References (sibling files this depends on or feeds)
+
+- Depends on 01-runtime.md: timeouts, body/header caps, cookies, compression, static serving. This
+  file never re-enforces byte-level caps.
+- Depends on 02-observability.md: correlation IDs, log format, per-route metrics, tracing spans.
+- Depends on 03-security.md: CSRF, CORS policy, rate limits, session/flash storage, upload quotas,
+  auth guards consumed by views and DI.
+- Depends on 04-data.md: ORM/KV/cache handles provided through DI; explicit invalidation hooks.
+- Feeds 06-realtime.md: `Request`/`Response` shells, router mounting and middleware/DI participation
+  for WebSocket/SSE/Socket.IO views and streaming responses.
+- Feeds 07-supervisor-replication.md: request scope teardown points safe for background handoff.
+- Feeds 08-clients-openapi.md: routers plus validation models as the OpenAPI source; response formats
+  as client expectations; test client drives `Application` without network.
+- Feeds 09-cli-ops.md: dev server hosts `Application`; embedder output plugs into template loaders;
+  health endpoints are plain routes defined here.
+- Follows 10-workflow-example.md: specs, test layout, living example coverage for every API above.
+
+## 10. Open questions
+
+- Q1: Exact lenient coercion table for query/form strings (e.g. `"yes"` to bool, empty string to
+  null). Proposal: strict booleans (`true/false/1/0` only), empty string never coerces to null.
+- Q2: Whether `?format=` override defaults on or off. Proposal: off; routes opt in.
+- Q3: Template `strict` undefined policy in production: render empty plus warn, or 500. Proposal:
+  empty plus warn and metric, 500 only in dev/test.
+- Q4: Plugin capability model granularity: coarse hook names now, fine-grained review before v1.
+
+## 11. Sources (exact URLs with retrieval dates)
+
+- FastAPI tutorial, Bigger Applications (APIRouter, include_router):
+  https://fastapi.tiangolo.com/tutorial/bigger-applications/ (retrieved 2026-10-23)
+- aiohttp Web Server Quickstart (Application, handlers, class-based views):
+  https://docs.aiohttp.org/en/stable/web_quickstart.html (retrieved 2026-10-23)
+- Pydantic v2 concepts, Models (validation guarantees, ValidationError):
+  https://pydantic.dev/docs/validation/latest/concepts/models/ (retrieved 2026-10-23)
+- Jinja Template Designer Documentation (syntax, inheritance, autoescape):
+  https://jinja.palletsprojects.com/en/stable/templates/ (retrieved 2026-10-23)
+- MDN, Content negotiation (Accept, q weights, 406):
+  https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Content_negotiation (retrieved 2026-10-23)
+- RFC 9110, HTTP Semantics, Section 12 (content negotiation, obsoletes RFC 7231):
+  https://www.rfc-editor.org/rfc/rfc9110.html (retrieved 2026-10-23)

plans/06-realtime.md

@@ -0,0 +1,368 @@
+# 06 - Realtime
+
+Title: Realtime (WebSocket, SSE, Socket.IO, Streaming, WebDAV, Client Parity)
+Scope: plan.md sections 2, 6-7 partial, 10 step 4 partial.
+
+## Purpose and philosophy
+
+Realtime turns MoloApp from a request/response framework into a live
+system: bidirectional WebSocket channels, one-way server push over SSE,
+multiplexed Socket.IO namespaces with rooms, unbounded streaming
+responses, and WebDAV resource management, every one of them usable
+through the same middleware, auth, session, logging and metrics path as
+plain HTTP handlers.
+
+Philosophy:
+
+- One connection model. A realtime connection is a long-lived request
+  context, not a parallel universe. It enters through routing,
+  middleware, auth and sessions exactly like any handler, then upgrades.
+- Protocol correctness first. Wire formats follow the governing
+  standard byte for byte (RFC 6455 framing, WHATWG event-stream
+  grammar, Socket.IO v5 packet encoding, RFC 4918 method semantics),
+  verified against third-party clients, not only our own.
+- Client parity is a release gate. Per plan.md section 2, everything
+  served ships a matching client (WebSocket, SSE, Socket.IO, WebDAV),
+  sharing config, auth, reconnect and logging behavior with the server
+  side. A protocol without its client is not done.
+- Backpressure everywhere. Slow consumers must slow producers or be
+  shed by rule, never buffer unboundedly until the process dies.
+- Layered, not duplicated. Socket.IO sits on Engine.IO, Engine.IO sits
+  on the WebSocket/long-polling transports, streaming sits on the
+  response hierarchy from 05-request-routing.md. Shared behavior lives
+  once in a base class.
+
+## Best practices (from research)
+
+1. WebSocket handshake: validate `Upgrade: websocket` and
+   `Connection: Upgrade`, require a `Sec-WebSocket-Key`, answer
+   `101 Switching Protocols` with `Sec-WebSocket-Accept` computed as
+   base64(SHA-1(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11")).
+   (RFC 6455)
+2. WebSocket framing: opcodes 0x0 continuation, 0x1 text (must be valid
+   UTF-8), 0x2 binary, 0x8 close, 0x9 ping, 0xA pong; control frames
+   max 125 bytes payload and never fragmented; server MUST fail
+   connections whose client frames are unmasked, and MUST NOT mask
+   server-to-client frames. (RFC 6455)
+3. Liveness: answer every ping with a pong carrying identical payload,
+   run an application heartbeat (ping interval plus pong timeout) and
+   close dead peers with a close frame and status code. (RFC 6455)
+4. SSE wire format: `Content-Type: text/event-stream`, UTF-8 only,
+   fields `event:`, `data:`, `id:`, `retry:`, blank line dispatches;
+   default event type is `message`; multiline data joins with LF.
+   (WHATWG HTML 9.2)
+5. SSE resumability: server assigns monotonically increasing `id:` per
+   stream; on reconnect the client sends `Last-Event-ID` and the
+   server replays missed events; HTTP 204 tells a client to stop
+   reconnecting; initial `retry:` sets reconnect delay. (WHATWG HTML)
+6. Engine.IO v4.1: transports polling then WebSocket upgrade
+   (WebTransport out of scope for v1); packet types open 0, close 1,
+   ping 2, pong 3, message 4, upgrade 5, noop 6; handshake returns
+   JSON `{sid, upgrades, pingInterval, pingTimeout, maxPayload}`;
+   default path `/engine.io/` with `EIO=4` query; at most one active
+   polling GET and one WebSocket per session. (Engine.IO protocol)
+7. Socket.IO v5: packet types CONNECT 0, DISCONNECT 1, EVENT 2,
+   ACK 3, CONNECT_ERROR 4, BINARY_EVENT 5, BINARY_ACK 6; text
+   encoding `<type>[<attachments>-][<nsp>,][<ackId>]<json>`, each
+   packet wrapped as one Engine.IO message (wire prefix `4`);
+   namespace omitted when `/`; CONNECT handshake carries session id
+   or refusal reason; binary payloads extracted as attachments with
+   placeholder objects. (Socket.IO protocol)
+8. WebDAV: implement PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK,
+   UNLOCK over the RFC 4918 data model; multistatus 207 with
+   `DAV:` XML bodies; honor `Depth: 0,1,infinity` (infinity capped,
+   see Specifications); advertise compliance via the `DAV:` header;
+   locks are advisory with tokens, timeouts and refresh. (RFC 4918)
+9. Streaming: chunked transfer with explicit flush points, SSE and
+   file/upstream passthrough stream with bounded buffers and pause
+   signals; compression never buffers a whole stream. Corroborated by
+   HTTP semantics practice and the WHATWG line-by-line processing
+   model.
+
+## Structure
+
+Directories (one class or concern per file, deep structure):
+
+- `molo/realtime/websocket/`: `server/`
+  (`WebSocketView`, `WebSocketResponse`, `WebSocketHandler`,
+  `FrameParser`, `FrameSerializer`, `HandshakeValidator`,
+  `HeartbeatManager`), `client/` (`WebSocketClient`,
+  `ClientHandshake`, `ReconnectPolicy` shared with all clients).
+- `molo/realtime/sse/`: `SseResponse`, `EventEncoder`,
+  `EventStore` (replay buffer interface), `SseClient`
+  (EventSource-compatible: readyState CONNECTING/OPEN/CLOSED,
+  Last-Event-ID tracking).
+- `molo/realtime/engineio/`: `EngineSession`, `PollingTransport`,
+  `WebSocketTransport`, `UpgradeManager`, `HeartbeatScheduler`,
+  `PacketCodec` (types 0-6, polling payload framing).
+- `molo/realtime/socketio/`: `SocketIoServer`, `Namespace`,
+  `RoomManager`, `EventDispatcher`, `AckTracker`, `PacketCodec`
+  (types 0-6, binary placeholder extraction), `client/`
+  (`SocketIoClient` with identical namespace/room/event API).
+- `molo/realtime/streaming/`: `ChunkedBody`, `PassthroughBody`,
+  `BackpressureValve`, `FlushPolicy` (used by SSE, streaming
+  responses, WebDAV passthrough).
+- `molo/realtime/webdav/`: `server/` (`DavView`, `PropfindHandler`,
+  `ProppatchHandler`, `MkcolHandler`, `CopyMoveHandler`,
+  `LockManager`, `MultistatusBuilder`, `QuotaGuard`), `client/`
+  (`WebDavClient` with PROPFIND/GET/PUT/DELETE passthrough,
+  response cache, quota accounting).
+- `molo/realtime/shared/`: `ConnectionContext` (auth/session/logging
+  handle shared by every realtime connection), `SendQueue`
+  (bounded per-connection queue), `RateLimiter` hook.
+
+Namespaces: `molo::realtime::websocket`, `::sse`, `::engineio`,
+`::socketio`, `::streaming`, `::webdav`. All connection types derive
+from `ConnectionContext`; all clients derive from `BaseRealtimeClient`
+(auth, reconnect, backoff, logging in one place).
+
+## Rules (hard, numbered)
+
+1. Every realtime endpoint mounts on a router and passes through the
+   full middleware, auth and session chain before upgrade. No
+   side-channel listeners.
+2. Server MUST reject client WebSocket frames without the mask bit by
+   failing the connection; server MUST NOT mask its own frames.
+3. Text frames MUST contain valid UTF-8; invalid UTF-8 fails the
+   connection with close code 1007.
+4. Control frames MUST have payload length at most 125 bytes and FIN
+   set; fragmented control frames are a protocol error (1002).
+5. Max message size, max frame count per message, max connections per
+   IP/user/route, send-queue depth and idle timeout are static caps
+   from config; exceeding them closes or sheds by rule, never grows.
+6. SSE responses MUST send `Content-Type: text/event-stream`,
+   `Cache-Control: no-store`, and MUST flush headers immediately so
+   intermediaries do not buffer the stream.
+7. Socket.IO and Engine.IO packet codecs MUST round-trip the official
+   test vectors (packet type tables in Specifications) before any
+   feature work builds on them.
+8. Client parity: no realtime server feature merges without its
+   matching client supporting the same auth, reconnect/backoff and
+   logging behavior in the same change.
+9. Slow consumers MUST be shed: when a send queue is full, apply the
+   route policy (drop oldest, drop newest, or close with code) and
+   count it in metrics. Unbounded queues are forbidden.
+10. WebDAV `Depth: infinity` MUST be rewritten to the configured max
+    depth; recursive operations MUST enforce quotas and per-request
+    resource ceilings.
+11. Locks MUST have finite timeouts and MUST auto-expire; lock tokens
+    MUST be unguessable random values.
+12. Streaming and passthrough paths MUST propagate backpressure to the
+    producer (pause reads) and MUST support client disconnect cleanup
+    within one heartbeat interval.
+
+## Development guidelines
+
+- Build bottom-up: handshake plus framing codecs first with byte-level
+  tests against captured third-party traffic (browsers, `websocat`
+  equivalents, official Socket.IO client), then views, then rooms and
+  namespaces, then clients, then WebDAV last.
+- Every codec gets a fuzzer-fed unit harness plus golden vectors from
+  the official protocol test suites; every view gets integration tests
+  through the real stack per 10-workflow-example.md.
+- Reuse, never fork: heartbeat, reconnect with jittered exponential
+  backoff, send queues and logging context are implemented once in
+  `shared/` and composed into each protocol.
+- Coroutine handlers (`co_await`) for message loops; blocking work
+  (XML parsing for large PROPFIND, disk-backed event replay) goes to
+  the off-loop executor defined in 01-runtime.md.
+- Log connection lifecycle (open, upgrade, heartbeat loss, close code
+  and reason) at fixed levels with correlation IDs per
+  02-observability.md; per-connection debug logging is gated and
+  capped.
+- Document every wire-visible behavior in the component spec before
+  coding it; docs state what the system does today, gaps listed next
+  to the code that closes them.
+
+## Specifications
+
+- WebSocket handshake: `GET` with `Upgrade: websocket`,
+  `Connection: Upgrade`, `Sec-WebSocket-Key` (16 random bytes,
+  base64), optional `Sec-WebSocket-Protocol` subprotocol
+  negotiation (server selects at most one, echoes it, else fails);
+  response `101` with `Sec-WebSocket-Accept`. Origin checked
+  against CORS policy from 03-security.md.
+- WebSocket opcodes: 0x0 cont, 0x1 text, 0x2 binary, 0x8 close
+  (0-125 byte body: 2-byte code plus UTF-8 reason), 0x9 ping,
+  0xA pong. Close codes used: 1000 normal, 1001 going away,
+  1002 protocol error, 1003 unsupported data, 1007 bad UTF-8,
+  1008 policy violation, 1009 message too big, 1011 internal
+  error. Fragmented messages reassembled up to max message size.
+- Caps (defaults, all overridable): max message 1 MiB, max frame
+  payload 256 KiB, max frames per message 64, send queue 256
+  messages, connections per IP 128, per route 100k, idle timeout
+  60 s, ping interval 25 s, pong timeout 20 s (Engine.IO defaults
+  aligned), close handshake timeout 5 s.
+- SSE event grammar: lines `field: value`, fields `event`, `data`,
+  `id`, `retry`, `:` comment (keep-alive); blank line dispatches;
+  `retry:` is milliseconds, clamped to 1 s..300 s; server sends
+  comment keep-alive every 15 s; `Last-Event-ID` replay window
+  holds last 1000 events or 5 minutes per stream, whichever is
+  smaller.
+- `SseClient` states CONNECTING 0, OPEN 1, CLOSED 2; fails the
+  connection on non-200 status or wrong content type; honors
+  301/307 redirect and 204 stop; sends stored last event id on
+  reconnect.
+- Engine.IO: path `/engine.io/`, query `EIO=4`,
+  `transport=polling|websocket`, `sid` after handshake; open
+  packet JSON keys exactly `sid, upgrades, pingInterval,
+  pingTimeout, maxPayload`; polling GET long-holds up to 25 s;
+  POST bodies carry length-prefixed packet payloads per the
+  polling encoding; upgrade probe `ping/probe` then `upgrade`.
+- Socket.IO: namespaces start with `/`, no commas; rooms are
+  server-side sets with join/leave, broadcast excludes sender by
+  default with explicit include flag; EVENT payload is a JSON
+  array whose first element is the event name; ack id is a
+  monotonically increasing integer per connection with 30 s
+  default ack timeout; binary attachments use
+  `{"_placeholder":true,"num":N}` markers, max 16 attachments
+  and 8 MiB total per packet by default.
+- Rooms and broadcast fan-out MUST complete in O(subscribers)
+  with per-recipient queue caps; broadcast to 0 subscribers is a
+  no-op counted in metrics, not an error.
+- Streaming: chunked bodies flush at least every 64 KiB or 100 ms
+  during activity; passthrough (proxy/file/WebDAV GET) uses a
+  256 KiB ring buffer with producer pause; range and conditional
+  requests supported on file passthrough.
+- WebDAV server: resource tree rooted at a configured collection;
+  methods GET, PUT, DELETE plus PROPFIND (allprop, propname,
+  prop selections), PROPPATCH (set/remove with 207 per-property
+  status), MKCOL, COPY/MOVE (with `Overwrite`, `Destination`,
+  `Depth` headers), LOCK (exclusive and shared, `Timeout`,
+  `Depth: 0`), UNLOCK (matching `Lock-Token`); `DAV: 1` header
+  on OPTIONS and responses; XML bodies use `DAV:` namespace,
+  max XML body 1 MiB, max depth 32.
+- WebDAV client: streams GET directly into a caller response body
+  (no full buffering), conditional GET with ETag cache (max 128
+  entries, 64 MiB), quota accounting hooks per transfer.
+- Replication hooks: connection open/close, room join/leave and
+  broadcast envelopes are emitted as typed events for
+  07-supervisor-replication.md; v1 guarantees at-most-once
+  delivery across instances, at-least-once only inside one
+  process via the send queue.
+
+## Execution steps (in build order)
+
+1. `shared/`: ConnectionContext, SendQueue, BaseRealtimeClient
+   (auth, jittered backoff, logging), BackpressureValve.
+2. WebSocket: HandshakeValidator, FrameParser/Serializer,
+   WebSocketView/Handler/Response, HeartbeatManager.
+3. `WebSocketClient` plus conformance tests against a third-party
+   peer.
+4. SSE: EventEncoder, SseResponse, EventStore replay,
+   `SseClient`.
+5. Streaming: ChunkedBody, PassthroughBody, FlushPolicy; wire
+   through SSE and file download.
+6. Engine.IO: PacketCodec, EngineSession, PollingTransport,
+   WebSocketTransport, UpgradeManager, HeartbeatScheduler.
+7. Socket.IO: PacketCodec (text plus binary), Namespace,
+   RoomManager, EventDispatcher, AckTracker, SocketIoServer.
+8. `SocketIoClient` with namespace/room/event parity and
+   long-polling fallback.
+9. WebDAV server: MultistatusBuilder, PROPFIND/PROPPATCH/MKCOL,
+   COPY/MOVE, LockManager, QuotaGuard, DavView.
+10. `WebDavClient` with streaming GET passthrough, cache, quota.
+11. Cross-cutting: metrics per connection/room/route, trace spans
+    for upgrade and broadcast, living-example realtime pages.
+
+## Validation (how to verify each step)
+
+1. Shared: unit tests prove queue shed policies and backoff
+   timings; no unbounded growth under a blocked consumer test.
+2. WebSocket: golden handshake vectors (key/accept pairs);
+   byte-level frame round-trips including fragmentation,
+   masking rejection, UTF-8 rejection, control-frame limits;
+   interop test against an independent client implementation.
+3. Client: connects to a third-party echo server, survives
+   restart with backoff, exchanges text plus binary.
+4. SSE: byte-exact event encoding tests; browser/EventSource
+   interop including reconnect with Last-Event-ID replay and
+   204 stop; wrong content-type fails cleanly.
+5. Streaming: 1 GiB passthrough with flat memory profile;
+   disconnect mid-stream releases all resources within one
+   heartbeat interval; flush latency within policy.
+6. Engine.IO: open/handshake JSON shape, ping/pong heartbeat
+   timing, polling-to-WebSocket upgrade sequence, duplicate
+   transport rejection, unknown sid returns 400.
+7. Socket.IO: official packet vectors for all 7 types including
+   binary placeholders; namespace connect/refuse; rooms
+   join/leave/broadcast; ack round-trip and timeout.
+8. Client parity: scripted session where our client drives the
+   official server behaviors (namespaces, rooms, acks, binary)
+   and vice versa; long-polling-only mode passes with
+   WebSocket disabled.
+9. WebDAV: PROPFIND/MKCOL/COPY/MOVE/LOCK/UNLOCK sequences
+   against litmus-style assertions; Depth infinity capped;
+   quota exceeded returns 507; lock expiry releases.
+10. Client: PROPFIND listings, streaming GET into a response
+    with bounded memory, ETag cache hits, quota accounting.
+11. Observability: connection counts, message rates, shed
+    counts, room sizes and broadcast fan-out exported per
+    02-observability.md; full integration suite green through
+    the real stack.
+
+## References
+
+- Depends on 01-runtime.md: event loop, lifecycle, timeouts,
+  HTTP/1.1 framing and upgrade path, compression interplay.
+- Depends on 02-observability.md: log format, correlation IDs,
+  metrics and tracing primitives, zero-cost gating.
+- Depends on 03-security.md: origin/CORS checks, rate limits,
+  body caps, auth/session resolution before upgrade.
+- Depends on 05-request-routing.md: routers, view base classes,
+  response hierarchy, middleware ordering, DI access to shared
+  services.
+- Feeds 04-data.md: pubsub bridge for cross-instance broadcast
+  and event replay storage interfaces.
+- Feeds 07-supervisor-replication.md: connection/room events,
+  broadcast envelopes, delivery guarantees, multi-instance
+  session sharing.
+- Feeds 08-clients-openapi.md: shared client transport behavior
+  (timeouts, retries, TLS, test doubles) reused by realtime
+  clients.
+- Feeds 10-workflow-example.md: integration test templates for
+  realtime routes and living-example chat, feed and DAV pages.
+
+## Open questions
+
+1. WebTransport (Engine.IO transport) needs HTTP/3, which
+   01-runtime.md lists as a future path. Stay out of v1: yes,
+   unless runtime lands H3 early.
+2. Should Socket.IO v2/v3 protocol compat be offered? Proposal:
+   no, v5 plus Engine.IO v4 only; document the refusal clearly.
+3. Cross-instance room broadcast transport: reuse pubsub from
+   04-data.md or a dedicated replication channel in
+   07-supervisor-replication.md? Needs a joint decision.
+4. SSE replay durability: in-memory ring per process for v1, or
+   persistent store from the start? Proposal: in-memory with a
+   documented interface for a persistent EventStore later.
+5. WebDAV ACL (RFC 3744) and versioning (DeltaV): explicitly out
+   of v1; confirm no v1 consumer needs them.
+
+## Sources
+
+1. IETF RFC 6455, The WebSocket Protocol (Fette, Melnikov,
+   Dec 2011, Standards Track). Handshake, framing, opcodes,
+   masking, close codes. URL:
+   https://www.rfc-editor.org/rfc/rfc6455 (retrieved
+   2026-10-23).
+2. WHATWG HTML Living Standard, section 9.2 Server-sent events
+   (EventSource interface, processing model, event-stream
+   parsing and interpretation). Wire grammar, reconnect,
+   Last-Event-ID, 204 stop. URL:
+   https://html.spec.whatwg.org/multipage/server-sent-events.html
+   (retrieved 2026-10-23).
+3. Socket.IO Protocol v5, official docs (packet types 0-6,
+   encoding format, binary placeholders, sample session).
+   URL: https://socket.io/docs/v4/socket-io-protocol/
+   (retrieved 2026-10-23).
+4. Engine.IO Protocol v4.1, official docs (transports,
+   handshake, heartbeat, upgrade, packet encoding). URL:
+   https://socket.io/docs/v4/engine-io-protocol/ (retrieved
+   2026-10-23).
+5. IETF RFC 4918, HTTP Extensions for WebDAV (Dusseault, ed.,
+   Jun 2007, Standards Track, obsoletes RFC 2518). Methods
+   PROPFIND/PROPPATCH/MKCOL/COPY/MOVE/LOCK/UNLOCK, 207
+   Multistatus, Depth/DAV headers. URL:
+   https://www.rfc-editor.org/rfc/rfc4918 (retrieved
+   2026-10-23).

plans/07-supervisor-replication.md

@@ -0,0 +1,143 @@
+# 07 - Supervisor, Tasks, Replication
+
+Scope: plan.md sections 2, 7, 10 steps 4+6. Topics: process supervisor, background tasks, pubsub durability, replication and conflict handling, multi-instance deployment.
+
+## Purpose and philosophy
+
+Request handlers stay fast and non-blocking. Anything slow, delayed, periodic, or retried runs as a supervised task outside the request path. Anything that must survive a crash or reach another instance goes through durable pubsub or the replication log, never through local memory assumed to be shared.
+
+Philosophy, in one line per pillar:
+
+- Supervision: workers are disposable, the supervisor is not. Restarts follow a declared strategy with a hard intensity cap, so a crashing child can never wedge the host.
+- Tasks: delivery is at-least-once, so every handler is idempotent and every outcome is acknowledged, retried, or dead-lettered, never silently dropped.
+- Replication: each model declares its conflict policy up front (optimistic locking, last-writer-wins, or conflict hook). Callers never resolve conflicts ad hoc.
+- Multi-instance: instances are stateless. Sessions, locks, broadcasts, and queues live in shared backends, and the load balancer pins realtime connections to one node.
+
+## Best practices (from research)
+
+BP-1: Restart strategies are named and explicit. Erlang/OTP defines one_for_one (restart only the failed child), one_for_all (restart all children), and rest_for_one (restart the failed child plus children started after it), with the supervisors purpose stated as keeping child processes alive by restarting them when necessary (https://www.erlang.org/doc/design_principles/sup_princ, retrieved 2026-10-23). Molo adopts the same three strategy names for worker groups.
+
+BP-2: Restarts always carry an intensity cap. OTP pairs every supervisor with intensity and period (max restarts in a time window), and supervisord pairs autorestart policies (false, unexpected, true) with startretries plus BACKOFF states before a process is declared FATAL (https://www.erlang.org/doc/design_principles/sup_princ, retrieved 2026-10-23; https://supervisord.org/configuration.html and https://supervisord.org/subprocess.html, retrieved 2026-10-23). Molo requires max_restarts plus window on every supervised program, with exponential backoff between attempts.
+
+BP-3: Task queues acknowledge after success and redeliver otherwise. Celery keeps a task message queued until a worker acknowledges it, redelivers reserved messages when a worker dies, and recommends idempotent task functions with acks_late so a started task is retried after failure (https://docs.celeryq.dev/en/stable/userguide/tasks.html, retrieved 2026-10-23). Molo tasks use ack-after-success with redelivery on worker loss, and handlers must be idempotent.
+
+BP-4: Retries use bounded exponential backoff and tasks carry time limits. Celery documents retry with backoff, max retries, and worker time limits that force-fail runaway tasks, warning that a task blocking forever starves the worker (https://docs.celeryq.dev/en/stable/userguide/tasks.html, retrieved 2026-10-23). Molo sets per-task max_retries, backoff base and cap, soft timeout (cancel requested) and hard timeout (worker recycled).
+
+BP-5: Pubsub durability is tiered, not assumed. Core NATS delivers at-most-once to currently connected subscribers, while JetStream adds persisted streams with server-tracked consumers and acknowledgments, giving at-least-once delivery with redelivery until acked (https://docs.nats.io/nats-concepts/jetstream, retrieved 2026-10-23). Molo mirrors this: ephemeral topics (at-most-once, no storage) and durable streams (persisted, cursor-tracked, acked).
+
+BP-6: Realtime fan-out across nodes needs a shared bus plus connection pinning. Socket.IO multi-node deployment requires sticky sessions so all requests for a session ID reach the originating node, plus an adapter that forwards broadcasts between servers over pubsub (https://socket.io/docs/v4/using-multiple-nodes/, retrieved 2026-10-23). Molo specifies the same topology: sticky load balancing plus a replication bus that carries room and broadcast events to every node.
+
+BP-7: Convergent replication needs either commutative data types or an explicit conflict rule. Shapiro et al. define Conflict-free Replicated Data Types whose concurrent updates converge without coordination (Shapiro, Preguica, Baquero, Zawirski, Conflict-free Replicated Data Types, https://inria.hal.science/hal-00932836, retrieved 2026-10-23). Full CRDTs are out of scope for v1, so Molo uses the pragmatic subset: LWW with hybrid timestamps for cache and KV entries, optimistic locking with version compare for ORM records, and typed conflict hooks where neither fits.
+
+Triangulation note: OTP and supervisord agree on named strategies plus intensity caps (BP-1, BP-2). Celery and JetStream agree on at-least-once plus ack-driven redelivery (BP-3, BP-5). Socket.IO practice and the CRDT literature agree that cross-node state needs an explicit bus and an explicit conflict rule, never implicit sharing (BP-6, BP-7).
+
+## Structure
+
+Directories (all under src/molo/, headers in include/molo/ mirroring the same tree):
+
+- supervisor/: Supervisor, WorkerGroup, WorkerProcess, RestartPolicy, HealthCheck, ShutdownCoordinator.
+- tasks/: TaskQueue, TaskEnvelope, TaskHandler (base), Scheduler (delayed and periodic), RetryPolicy, DeadLetterQueue, TaskRegistry.
+- replication/: Replicator, OplogWriter, OplogReader, ConflictResolver, VersionVector, SyncSession.
+- bus/: StreamBus (durable streams), TopicBus (ephemeral topics), ConsumerCursor, AckTracker. Built on the 04-data.md pubsub core, adding only durability here.
+
+Namespaces: molo::supervisor, molo::tasks, molo::replication, molo::bus. One class or concern per file pair (.hpp plus .cpp), no umbrella headers outside per-directory forward headers.
+
+Key classes:
+
+- supervisor::Supervisor: owns worker groups, applies restart strategies, enforces intensity caps, exposes status snapshot for CLI and health endpoints.
+- supervisor::WorkerProcess: one supervised child (in-process thread worker in v1, OS process in v2 behind the same interface).
+- tasks::TaskEnvelope: immutable job record (id, name, payload, attempts, schedule, deadlines). Only type that crosses queue boundaries.
+- tasks::Scheduler: delayed run_at and periodic cron or interval entries, backed by a durable stream so schedules survive restarts.
+- tasks::RetryPolicy: max retries, backoff computation, dead-letter routing.
+- replication::OplogWriter and OplogReader: append and tail the per-instance operation log used for sync.
+- replication::ConflictResolver: dispatches per-model policy (optimistic, LWW, hook).
+- bus::StreamBus: durable persisted streams with named consumers, cursors, and ack deadlines.
+
+## Rules (hard, numbered)
+
+R7-1: Request handlers never run background work inline. Slow, delayed, periodic, or retryable work enters tasks::TaskQueue only.
+R7-2: Every task handler is idempotent. Redelivery is normal under at-least-once semantics, and handlers must tolerate repeat execution with the same task id.
+R7-3: Every supervised program declares a restart policy (never, on-failure, always) and an intensity cap (max restarts per window). Unbounded restart loops are forbidden.
+R7-4: Every queue depth, payload size, retry count, backoff bound, ack deadline, cursor count, and replication buffer has a static cap from Specifications. No unbounded growth.
+R7-5: Task execution and replication I/O never run on the HTTP event loop threads. Workers run on dedicated threads in v1 and OS processes in v2.
+R7-6: Every worker lifecycle event, task state change, and replication conflict emits events, metrics, and trace spans per 02-observability.md. No silent work.
+R7-7: Replication conflicts resolve only through the model-declared policy in ConflictResolver. Callers never compare timestamps or versions directly.
+R7-8: Anything needed by more than one instance lives in a shared backend (durable stream, shared KV, shared session store). Local memory is never treated as global.
+R7-9: Shutdown is graceful with a deadline: stop accepting new tasks, drain in-flight tasks until the deadline, then stop. Dropped tasks go to the dead-letter queue with a reason, never vanish.
+R7-10: The CLI and HTTP admin surface manage supervisor and tasks only through the declared Supervisor and TaskQueue APIs. No poking internals, no side-channel signals.
+
+## Development guidelines
+
+- Build the minimal in-process supervisor plus task runner first (plan.md step 4), then durable streams, then replication and conflict handling (plan.md step 6). Each stage is independently shippable.
+- Keep the WorkerProcess interface transport-agnostic so v2 OS-process workers slot in without touching TaskQueue or Scheduler.
+- Prefer polling cursors over callbacks for stream consumers: simpler backpressure, easier testing, no reentrancy.
+- Use hybrid logical timestamps (wall clock plus counter) for LWW so clock skew cannot silently invert ordering; record source instance id in every oplog entry for deterministic tie-breaks.
+- Write integration tests through the real stack per 10-workflow-example.md: enqueue over HTTP, observe execution via status API, kill a worker mid-task, assert redelivery.
+- Document every new cap in Specifications the day it is introduced, with the constant name and default.
+
+## Specifications
+
+S7-1 Task envelope (JSON, also the on-disk format for the durable queue):
+{"id": "uuid7", "name": "billing.charge", "payload": {}, "attempts": 0, "max_retries": 5, "run_at": "iso8601|null", "timeout_soft_ms": 30000, "timeout_hard_ms": 60000, "idempotency_key": "string", "enqueued_at": "iso8601"}. Payload max 1 MiB. Queue depth max 100000 envelopes per queue. Duplicate idempotency_key within 24h returns the existing task id without re-enqueue.
+
+S7-2 Retry schedule: delay(n) = min(backoff_base_ms * 2^n + jitter_ms, backoff_cap_ms), defaults base 1000ms, cap 300000ms, jitter uniform 0..1000ms. After max_retries the envelope moves to the dead-letter queue with reason code and stays 7 days, max 10000 entries, then evicted oldest-first with an event emitted.
+
+S7-3 Task states: queued, leased, running, succeeded, failed-retryable, failed-permanent, dead-lettered, cancelled. Lease = ack deadline, default 60000ms, renewable by heartbeat every 15000ms. Lease expiry without ack returns the task to queued with attempts incremented.
+
+S7-4 Supervisor worker states: stopped, starting, running, backoff, stopping, exited, fatal. Fatal entered after max_restarts (default 5) within window (default 60s). Backoff delay doubles from 1s to a 60s cap. Health checks: TCP or command probe every 10s, 3 failures mark unhealthy and trigger policy action.
+
+S7-5 Periodic tasks: cron expression (5-field, UTC) or fixed interval with minimum 60s. Scheduler tick 1s. At most one scheduler leader per deployment (lease in shared KV, 10s TTL, 3s renewal); followers stay passive. Missed-fire policy per entry: skip or run-once, default skip.
+
+S7-6 Pubsub delivery tiers. Ephemeral topics: at-most-once, no persistence, no replay, subscriber must be connected. Durable streams: persisted append-only log, per-message sequence number, named consumers with server-side cursors, ack-required, redelivery until acked or retention (default 24h or 1 GiB per stream, whichever first). Retention and redelivery counters are observable metrics.
+
+S7-7 Replication oplog entry: {"seq": n, "instance": "id", "ts": "hybrid-clock", "model": "name", "key": "pk", "op": "upsert|delete", "version": n, "data": {}}. Transport: each instance tails peers over the bus (push) with catch-up pull on reconnect. Batch max 500 entries or 1 MiB per sync round. Sync lag (entries behind) is a gauge metric with alert threshold 10000.
+
+S7-8 Conflict policies per model, declared once: optimistic (compare version, reject stale with ConflictError carrying winner version; caller retries), LWW (higher hybrid timestamp wins, ties break on instance id; loser kept in conflict log 7 days), hook (typed ConflictResolver callback runs, must return within 5s or LWW fallback applies). Default: optimistic for ORM models, LWW for cache and KV.
+
+S7-9 Multi-instance topology: N stateless app instances behind a load balancer with sticky sessions for WebSocket and Socket.IO long-polling transports (plain WS and SSE may pin or re-resolve, documented per route in 06-realtime.md); one shared durable bus; one shared session and KV store; scheduler leader lease in shared KV. Broadcast emit publishes once to the bus; every node re-emits to its local sockets.
+
+S7-10 Static caps summary: workers per group max 64; groups per supervisor max 32; task payload 1 MiB; queue depth 100000; dead-letter 10000; stream retention 24h or 1 GiB; sync batch 500 entries or 1 MiB; conflict log 10000 entries; supervisor status snapshot 1 MiB; all timeouts listed above.
+
+## Execution steps (in build order)
+
+E7-1: Minimal Supervisor with in-process thread workers, start/stop/restart/status, restart policies never/on-failure/always, intensity cap, graceful shutdown with drain deadline. (plan.md step 4 slice)
+E7-2: Minimal TaskQueue: enqueue, lease, ack, in-memory queue, one worker pool off the event loop, CLI enqueue plus status via the Supervisor API.
+E7-3: RetryPolicy with bounded exponential backoff, dead-letter queue, soft and hard timeouts, idempotency keys.
+E7-4: Scheduler with run_at delays plus interval and cron periodic entries, single-node first, then shared-KV leader lease.
+E7-5: Durable streams on the 04-data.md pubsub core: persisted log, named consumers, cursors, ack deadlines, redelivery, retention enforcement, ephemeral tier kept separate.
+E7-6: Back the task queue and scheduler on durable streams so queued and scheduled work survives restarts; add catch-up replay on boot.
+E7-7: Replication oplog, push plus catch-up pull transport, per-model policy declarations, optimistic and LWW resolvers, conflict log. (plan.md step 6 slice)
+E7-8: Multi-instance wiring: shared bus broadcasts for realtime rooms (with 06-realtime.md), shared session and KV usage (with 04-data.md), scheduler leadership, documented sticky-session LB config and health endpoints.
+
+## Validation (how to verify each step)
+
+V7-1: Start 3 workers, kill -9 one twice, assert automatic restart and fatal after cap; measure restart latency under 2s; status API reflects states.
+V7-2: Enqueue 1000 tasks over HTTP, assert all succeed without blocking a concurrent latency probe (p99 within 2x baseline).
+V7-3: Inject a handler failing twice then succeeding, assert success with attempts=2 and backoff delays within 20 percent of schedule; inject permanent failure, assert dead-letter entry with reason.
+V7-4: Schedule a 60s periodic task, advance frozen test clock 5 ticks, assert 5 runs; kill leader, assert follower takes over within 15s with no duplicate run.
+V7-5: Publish 10000 stream messages, disconnect a consumer mid-way, reconnect, assert zero loss and zero gaps in sequence numbers; verify ephemeral subscribers get nothing sent while offline.
+V7-6: Fill queue, hard-kill the process, reboot, assert all unacked tasks replay and complete, with duplicates tolerated by idempotent handlers.
+V7-7: Run two instances with conflicting writes to one ORM record, assert exactly one winner plus ConflictError to the loser; conflicting KV writes resolve by LWW with loser in conflict log.
+V7-8: Run two instances behind a sticky LB, open sockets on both, broadcast once, assert delivery on both; kill one node, assert its sockets reconnect and catch up via the bus.
+
+## References (sibling files this depends on or feeds)
+
+Depends on: 01-runtime.md (event loop threads workers must avoid, graceful shutdown hooks); 02-observability.md (event, metric, trace APIs for every state change); 04-data.md (pubsub core, KV, cache invalidation, ORM versions for optimistic locking); 06-realtime.md (room and broadcast semantics replicated across nodes).
+Feeds: 09-cli-ops.md (supervisor and task operations the CLI drives: start, stop, restart, scale, status, logs, enqueue, retry, cancel, inspect); 10-workflow-example.md (living example must demo tasks, schedules, failover, and two-node replication).
+
+## Open questions
+
+O7-1: Optional external backends: should v1 ship a Redis-protocol-compatible queue backend, or stay in-house only per the zero-dependency rule? Leaning in-house only, external adapters as v2 plugins.
+O7-2: OS-process workers in v1 or v2? v1 thread workers are simpler and testable; process isolation waits for v2 unless a reviewer demands it.
+O7-3: Exactly-once needs: any v1 feature truly requiring exactly-once (billing capture)? Current answer is no; idempotent at-least-once covers all known cases.
+O7-4: Network partition behavior for scheduler leadership and LWW: accept brief dual-run risk with documented bounds, or add fencing tokens in v1?
+O7-5: Durable stream storage engine: reuse the 04-data.md SQLite WAL path for streams, or a dedicated append-only segment file? Needs a joint decision with 04-data.md.
+
+## Sources
+
+S1: Erlang/OTP Supervisor Behaviour, official docs, restart strategies and intensity: https://www.erlang.org/doc/design_principles/sup_princ (retrieved 2026-10-23).
+S2: supervisord Configuration File and Subprocesses, autorestart policies, startretries, BACKOFF and FATAL states: https://supervisord.org/configuration.html and https://supervisord.org/subprocess.html (retrieved 2026-10-23).
+S3: Celery Tasks user guide, acks_late, idempotency, retry, time limits: https://docs.celeryq.dev/en/stable/userguide/tasks.html (retrieved 2026-10-23).
+S4: NATS JetStream concepts, streams, consumers, at-least-once delivery and acknowledgment: https://docs.nats.io/nats-concepts/jetstream (retrieved 2026-10-23).
+S5: Socket.IO Using Multiple Nodes, sticky sessions and adapter broadcast: https://socket.io/docs/v4/using-multiple-nodes/ (retrieved 2026-10-23).
+S6: Shapiro, Preguica, Baquero, Zawirski, Conflict-free Replicated Data Types, INRIA HAL record: https://inria.hal.science/hal-00932836 (retrieved 2026-10-23).

plans/08-clients-openapi.md

@@ -0,0 +1,331 @@
+# 08 - Clients, OpenAPI, Test Client
+
+Title: Clients, OpenAPI, Test Client
+Scope: plan.md sections 2, 6, 8 partial, 10 step 5 partial.
+Topics: HTTP/REST client, REST standards, OpenAPI generation, test client.
+
+## 1. Purpose and philosophy
+
+This file plans the outbound and introspection side of MoloApp: the modern
+async HTTP/API/REST client, the REST conventions every app follows, automatic
+OpenAPI document generation from routers plus validation models, and the test
+client that drives the real stack without real network I/O.
+
+Philosophy:
+
+- One client core. `Client` owns pooling, timeouts, retries, auth, cookies,
+  proxy, TLS, logging. Convenience free functions exist only for one-off
+  scripts and never pool.
+- Client parity. Everything the framework serves ships a client. This file
+  owns HTTP/REST; realtime clients live in 06-realtime.md and reuse the
+  timeout, retry, auth, logging and backoff behavior defined here.
+- Transport injection. Real network, in-process app dispatch, and scripted
+  mocks are all `Transport` implementations behind one interface. Tests swap
+  the transport, never the calling code.
+- Spec from code. Routers plus validation models are the single source of
+  truth; the OpenAPI document is derived, never hand-maintained.
+- Tests through the real stack. The test client runs routing, middleware,
+  validation, views and DB for real; only the socket is fake.
+
+## 2. Best practices (from research)
+
+1. Reuse connections via a long-lived client. Top-level one-shot request APIs
+   open a new connection per call; a `Client` instance pools and reuses TCP
+   connections to the same host, cutting latency, CPU and congestion. Client
+   instances additionally carry cookie persistence, shared config, proxying
+   and HTTP/2 support (httpx Clients doc).
+2. Separate timeout phases. Configure connect, read, write and pool timeouts
+   independently with a sane default total, so slow servers and slow networks
+   fail fast in the right place (httpx Timeouts/Clients docs).
+3. Retry only what is safe. Retry connect errors and idempotent methods on
+   429/502/503/504 with exponential backoff and jitter; never auto-retry
+   non-idempotent bodies unless explicitly enabled (httpx Transports doc on
+   connection retries; Microsoft REST guidance on throttling with Retry-After).
+4. Design resources, not RPC. Nouns in URIs, verbs in HTTP methods, plural
+   collection names, stateless requests, standard status codes, versioning by
+   path or media type, pagination/filtering/sorting as query parameters, and
+   `Retry-After` plus problem-detail bodies on errors (Microsoft API design
+   best practices).
+5. Generate OpenAPI from code annotations. FastAPI derives paths, parameters,
+   request/response schemas (JSON Schema), security schemes and examples from
+   route decorators plus Pydantic models, serves `openapi.json` plus Swagger
+   UI/ReDoc, and allows app-level overrides of the generated document
+   (FastAPI OpenAPI docs).
+6. Emit OpenAPI 3.1. The 3.1.x line aligns schemas fully with JSON Schema
+   (draft 2020-12), uses `openapi: 3.1.x`, `info`, `paths`, `components`,
+   `tags`, `servers` and `security` top-level objects, and path templating
+   with `{param}` expressions (OpenAPI Specification v3.1.0).
+7. Test through the app object. Starlette/FastAPI `TestClient` wraps an HTTP
+   client whose transport calls the ASGI app in-process: same routing,
+   middleware, validation and responses as production, no socket. Lifespan
+   events run via context-manager entry (FastAPI Testing tutorial; httpx
+   ASGI/WSGI/Mock transports doc).
+
+## 3. Structure
+
+Directories (one class or concern per file, deep over flat):
+
+```
+src/client/
+  client.h, client.cpp            # molo::client::Client, config + verbs
+  request.h, request.cpp          # ClientRequest builder (method, URL, headers, body)
+  response.h, response.cpp        # ClientResponse (status, headers, body, json/xml/form)
+  url.h, url.cpp                  # URL parse, join, query encode
+  headers.h                       # shared header map helpers (thin, delegates to runtime)
+  cookies.h, cookies.cpp          # CookieJar, persistence across requests
+  auth.h, auth.cpp                # Basic, Bearer, ApiKey, custom Auth base
+  timeouts.h                      # Timeout phases struct
+  retry.h, retry.cpp              # RetryPolicy, backoff with jitter
+  pool.h, pool.cpp                # connection pool (delegates sockets to 01-runtime.md)
+  proxy.h, proxy.cpp              # proxy selection incl. NO_PROXY matching
+  transport.h                     # Transport interface
+  transports/
+    http_transport.h, .cpp        # real network via runtime event loop
+    mock_transport.h, .cpp        # scripted request->response double
+    app_transport.h, .cpp         # in-process dispatch into Application (test client base)
+  body/
+    json_body.h, .cpp
+    form_body.h, .cpp             # urlencoded + multipart, file upload parts
+    xml_body.h, .cpp
+    stream_body.h, .cpp           # chunked upload/download with backpressure
+  rest/
+    resource.h, .cpp              # Resource fluent helper (CRUD over a base path)
+    paginator.h, .cpp             # page/limit/offset/cursor iteration
+    errors.h, .cpp                # raise-for-status, ProblemDetails parse
+src/openapi/
+  document.h, .cpp                # OpenAPI document model, serialize JSON (+YAML)
+  builder.h, .cpp                 # builds document from routers + validation models
+  schema.h, .cpp                  # validation-model to JSON Schema converter
+  routes.h                        # serves /openapi.json, /docs (Swagger UI), /redoc
+  extensions.h                    # x- vendor extensions, app overrides hook
+src/testing/
+  test_client.h, .cpp             # TestClient: Client preset with AppTransport
+  fixtures.h, .cpp                # DB/ORM fixtures, seeded app factory
+  time.h, .cpp                    # frozen/manual clock injected into runtime + ORM
+  assertions.h, .cpp              # status/json/header/schema assertion helpers
+```
+
+Namespaces: `molo::client`, `molo::client::transports`, `molo::client::rest`,
+`molo::openapi`, `molo::testing`.
+
+Key classes:
+
+- `client::Client`: coroutine verbs `get/post/put/patch/delete/head/options`,
+  `send(ClientRequest)`, shared `ClientConfig`, owns pool + jar + transport.
+- `client::Transport`: `send(request) -> response` coroutine interface.
+- `client::RetryPolicy`: max attempts, backoff base/cap, jitter, retryable
+  status set, idempotency gate.
+- `rest::Resource`: `collection("users").get(id)`, `.list(query)`,
+  `.create(doc)`, `.update(id, doc)`, `.remove(id)`.
+- `openapi::Builder`: `from_app(const App&) -> Document`, deterministic
+  output, app override hook `extend(Document&)`.
+- `testing::TestClient`: `Client` subclass fixed to `AppTransport`,
+  lifespan helpers, fixture + clock wiring.
+
+## 4. Rules (hard, numbered)
+
+1. All outbound HTTP from handlers, tasks and CLI goes through
+   `client::Client`; raw socket HTTP is forbidden outside `http_transport`.
+2. Every request has an explicit effective timeout (phases plus total); no
+   request may wait unbounded.
+3. Retries apply only to idempotent methods (GET, HEAD, OPTIONS, PUT, DELETE)
+   plus POST explicitly marked idempotent (Idempotency-Key header or policy
+   flag); anything else retries only connect-phase failures, never a sent
+   body.
+4. Backoff is exponential with jitter and a static cap; max attempts is a
+   static ceiling (default 3 total tries).
+5. Redirects are followed only for GET/HEAD by default, max 5 hops, and never
+   forward Authorization across hosts.
+6. Cookie jar is per-Client, opt-in persistent, honors Secure/HttpOnly/
+   SameSite semantics on read-back.
+7. REST surface rules: plural noun collections, no verbs in paths, correct
+   method semantics (POST create, GET read, PUT full replace, PATCH partial,
+   DELETE remove), standard status codes (200/201/204/400/401/403/404/409/
+   422/429/500), errors as RFC 9457 problem details (`application/problem+json`).
+8. OpenAPI output is OpenAPI 3.1.x only; no 2.0/3.0 emitters.
+9. Generated OpenAPI is deterministic: stable key order, stable schema names,
+   byte-identical output for identical app input.
+10. Test client runs the real stack (routing, middleware, auth, validation,
+    views, DB). Tests that bypass routing to call views directly are
+    forbidden except for documented error-path probes.
+11. All caps in section 6 are static constants, never autotuned.
+
+## 5. Development guidelines
+
+- Build order inside this file: URL + headers + bodies, then Transport +
+  Client + pool + timeouts, then retry/auth/cookies/proxy, then REST helpers,
+  then OpenAPI model + builder + schema converter + doc routes, then test
+  client + fixtures + clock.
+- Reuse, do not reimplement: sockets, TLS, HTTP/1.1+H2 framing and
+  compression come from 01-runtime.md; auth/session semantics from
+  03-security.md; validation models from 05-request-routing.md; logging,
+  metrics, traces from 02-observability.md.
+- Every public client API has sync (blocking on internal loop bridge for CLI
+  use) and coroutine forms sharing one implementation; the coroutine form is
+  primary.
+- Streaming is first-class: uploads and downloads support bounded buffers
+  with backpressure; files over 1 MB must be able to stream without full
+  buffering.
+- Mock transport matches on method + path template + headers/body predicates
+  and asserts unconsumed expectations at teardown.
+- OpenAPI builder reads route metadata (method, path template, summary,
+  tags, auth guards, validation models for params/body/response, status
+  codes). Anything not declared in code appears as `description: TBD`, never
+  as invented detail.
+- Serve Swagger UI and ReDoc from embedded assets (see 09-cli-ops.md
+  embedder); docs routes are disabled by default in production config.
+- Test client API mirrors the real client verb-for-verb so tests read like
+  production call sites.
+
+## 6. Specifications (concrete behaviors, caps, formats)
+
+Client:
+
+- Protocols: HTTP/1.1 and HTTP/2 (via 01-runtime.md). H2 enabled per-client
+  flag, default on for https URLs.
+- Timeout defaults: connect 5s, read 30s, write 30s, pool acquire 5s, total
+  60s. All overridable per request.
+- Pool caps: max 100 connections per Client, max 20 per host, max keepalive
+  60s, max idle 10 per host. Queue wait bounded by pool-acquire timeout.
+- Retry defaults: 3 total attempts, base 200ms, cap 5s, full jitter,
+  retryable statuses {429, 502, 503, 504}, honor `Retry-After` (seconds and
+  HTTP-date) up to cap.
+- Redirects: max 5, GET/HEAD auto, 307/308 preserve method+body, strip
+  Authorization on cross-host.
+- Body caps: in-memory body max 10 MB default; larger requires streaming
+  API. Multipart file part max 512 MB, total request max 1 GB (static caps,
+  lowered by server policy in 03-security.md).
+- Content helpers: `json<T>()` (serialize/parse via validation framework),
+  `form()`, `multipart()`, `xml()`, `bytes()`, `stream()`.
+- Proxy: `HTTP_PROXY/HTTPS_PROXY/NO_PROXY` honored, per-client override,
+  wildcard + suffix + CIDR NO_PROXY match.
+- Observability: every request emits span + metrics (duration, status,
+  retries, bytes) and debug log with correlation ID; disabled paths compile
+  near-zero per 02-observability.md.
+
+REST standards:
+
+- URI style: `/v1/users/{id}`, lowercase, hyphenated, plural collections,
+  query for filter/sort/page (`?page=&per_page=&sort=&filter=`).
+- Pagination: default page+per_page (max per_page 100, default 20); cursor
+  alternative for large sets. Envelope: `{data, meta:{page,per_page,total}, links}`.
+- Errors: RFC 9457 `{type,title,status,detail,instance}` plus optional
+  `errors[]` for field failures (422 with field paths from validation).
+- Versioning: path prefix `/v{n}`; breaking change requires new version.
+- Idempotency: `Idempotency-Key` header supported on POST create endpoints;
+  server stores keyed result 24h (see 04-data.md KV).
+
+OpenAPI:
+
+- Version string `3.1.0` minimum; generator stamps `x-molo-framework-version`.
+- Served at `/openapi.json` (JSON, canonical) and `/openapi.yaml`; UI at
+  `/docs` (Swagger) and `/redoc`.
+- Schemas: every validation model becomes `#/components/schemas/<Name>`;
+  path/query/header/cookie params from binding declarations; security
+  schemes from auth guards (http/bearer, http/basic, apiKey, oauth2 as
+  configured); examples from model examples.
+- Generation cap: max 5000 paths, max 2000 schemas, document max 10 MB;
+  beyond that the builder errors with a clear message.
+
+Test client:
+
+- `TestClient(App)` issues requests through `AppTransport`: no TCP, real
+  everything else. Lifespan via `test_client.lifespan()` guard.
+- Fixtures: per-test DB reset (SQLite in-memory or isolated Postgres
+  schema), seeded objects factory, ORM session bound to request scope.
+- Clock: frozen time default in tests (`testing::Clock::freeze(t)`), manual
+  advance `clock.advance(d)`; runtime timers and ORM timestamps read it.
+- Assertions: `expect_status`, `expect_json`, `expect_json_schema(name)`
+  validated against the generated OpenAPI schemas, `expect_header`.
+
+## 7. Execution steps (in build order)
+
+1. URL parser, header map, request/response value types with JSON/form/XML
+   bodies (unit-covered via test client loopback only).
+2. Transport interface plus mock transport; Client verbs over injected
+   transport; timeout phases enforced.
+3. Real HTTP transport on the 01-runtime.md loop: HTTP/1.1 first, then H2;
+   pooling honoring caps; cookie jar; proxy incl. NO_PROXY.
+4. Retry policy with backoff/jitter/Retry-After; auth providers; redirect
+   handling; streaming upload/download.
+5. REST helpers: Resource fluent API, paginator, problem-details errors,
+   idempotency-key sender.
+6. OpenAPI document model + JSON/YAML serializers; schema converter from
+   validation models; builder from routers; doc routes + embedded UI.
+7. Test client on AppTransport with lifespan; fixtures; frozen clock;
+   assertion helpers; OpenAPI schema-conformance assertion.
+8. Living-example coverage: REST client flows, OpenAPI endpoints, and one
+   full test-client suite per 10-workflow-example.md.
+
+## 8. Validation (how to verify each step)
+
+1. URL/body round-trips: parse/serialize vectors incl. encoding edges;
+   loopback test-client echo returns identical bodies.
+2. Mock-transport suite: scripted responses, predicate matching, timeout
+   injection fires the right phase error, unconsumed-expectation failure.
+3. Real-transport suite against local test server: keep-alive reuse proven
+   by connection counter, pool caps enforced under burst, H2 negotiated on
+   TLS fixture, proxy + NO_PROXY routing matrix.
+4. Retry matrix: each retryable status retried then succeeding; POST without
+   idempotency key not retried after body sent; Retry-After honored; redirect
+   Authorization-strip test; 50 MB streamed download under 16 MB RSS growth.
+5. REST conformance: golden problem-details bodies; paginator walks a 250
+   item set; idempotent POST replay returns original result.
+6. OpenAPI: generated document validates against the official 3.1 JSON
+   schema; byte-determinism check (generate twice, diff empty); snapshot of
+   example app reviewed; UI routes return 200 in dev, 404 in prod default.
+7. Test client: full suite runs with loopback transport only (assert zero
+   real sockets via pool counter); lifespan startup/shutdown observed;
+   frozen-clock test proves identical timestamps; schema-conformance
+   assertion passes for every example route.
+
+## 9. References
+
+Depends on:
+
+- 01-runtime.md: event loop, HTTP/1.1+H2 framing, TLS hooks, compression,
+  connection primitives the pool and HTTP transport build on.
+- 02-observability.md: spans, metrics, log format, correlation IDs, zero-cost
+  macros used by every request.
+- 03-security.md: auth/session semantics, TLS via wolfSSL, body/upload caps,
+  redirect Authorization rules.
+- 04-data.md: KV for idempotency keys, DB fixtures base for test client.
+- 05-request-routing.md: routers, validation models, response hierarchy,
+  error mapping; OpenAPI builder inputs.
+
+Feeds / defers to:
+
+- 06-realtime.md: owns WebSocket/SSE/Socket.IO/WebDAV clients; reuses this
+  file's timeout/retry/auth/logging/backoff contracts.
+- 09-cli-ops.md: AI harness CLI is built on this HTTP client; embedder packs
+  Swagger/ReDoc UI assets; dev server hosts doc routes.
+- 10-workflow-example.md: owns test-file layout/naming and living-example
+  gates; this file provides the TestClient mechanism it standardizes on.
+
+## 10. Open questions
+
+1. H2 for cleartext (h2c) in the client: prior-knowledge only, or upgrade
+   handshake too? Proposal: prior-knowledge with opt-in upgrade.
+2. Should the OpenAPI builder also emit AsyncAPI fragments for WS/SSE
+   routes, or stay HTTP-only and let 06-realtime.md define its own export?
+   Proposal: HTTP-only here, cross-link.
+3. Idempotency-key storage backend default: in-memory with documented loss
+   on restart, or KV-backed always? Proposal: KV-backed, in-memory only for
+   single-process dev.
+
+## 11. Sources
+
+1. OpenAPI Initiative - OpenAPI Specification v3.1.0 (2021-02-15):
+   https://spec.openapis.org/oas/v3.1.0 (retrieved 2026-10-23).
+2. httpx documentation - Clients (Client pooling, config sharing):
+   https://www.python-httpx.org/advanced/clients/ (retrieved 2026-10-23).
+3. httpx documentation - QuickStart (verbs, bodies, timeouts, auth):
+   https://www.python-httpx.org/quickstart/ (retrieved 2026-10-23).
+4. httpx documentation - Transports (HTTP/ASGI/WSGI/Mock, retries, mounts):
+   https://www.python-httpx.org/advanced/transports/ (retrieved 2026-10-23).
+5. FastAPI documentation - Testing (TestClient through app):
+   https://fastapi.tiangolo.com/tutorial/testing/ (retrieved 2026-10-23).
+6. FastAPI documentation - Extending OpenAPI (code-first generation):
+   https://fastapi.tiangolo.com/how-to/extending-openapi/ (retrieved 2026-10-23).
+7. Microsoft Learn - Web API design best practices (REST guidance):
+   https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design (retrieved 2026-10-23).

plans/09-cli-ops.md

@@ -0,0 +1,158 @@
+# 09 - CLI Tooling, Dev Server, Operations
+
+Scope: plan.md sections 2, 7-8, 10 step 5 partial. Topics: CLI, dev server, embedder, AI harness, health/metrics, i18n/CSP.
+
+## Purpose and philosophy
+
+One binary, `molo`, is the only project tool: it scaffolds, serves, embeds, supervises, and vibe-codes. Operators get health, readiness, and metrics endpoints that work with stock Prometheus and Kubernetes probes with zero glue code. Developers get a dev server with fast restart and a strict dev/prod settings split. Templates get i18n hooks and a CSP builder by default, so every generated page is localizable and locked down. Philosophy: human-first output that degrades gracefully to machine-readable output, static ceilings on everything watched or exported, and dev/prod parity except where explicitly split.
+
+## Best practices (from research)
+
+- CLI design follows the Command Line Interface Guidelines: zero exit code on success and non-zero on failure, primary output to stdout, diagnostics to stderr, concise help by default with extensive help on request, and machine-readable output (JSON lines or `--format json`) where it does not hurt usability (clig.dev).
+- Metrics exposition follows the Prometheus text format 0.0.4: line oriented, `\n` terminated with trailing newline on the last line, `# HELP` and `# TYPE` comment lines before samples, `Content-Type: text/plain; version=0.0.4`, optional gzip encoding, counter/gauge/histogram/summary/untyped primitives (Prometheus docs).
+- Health semantics follow the Kubernetes probe model: liveness detects a broken state that only a restart fixes (kubelet kills and restarts the container on failure), readiness gates traffic, startup protects slow-starting containers with its own window (Kubernetes docs).
+- CSP starts from `default-src 'none'` and opens only what is needed; `default-src` is the fallback for all other fetch directives; rollout begins in report-only mode (`Content-Security-Policy-Report-Only`) before enforcing (MDN CSP, field practice).
+- Streaming uses Server-Sent Events: `Content-Type: text/event-stream`, `Cache-Control: no-cache`, notifications as text blocks terminated by blank lines, `data:` payload lines (MDN SSE). OpenAI-compatible chat streaming sends data-only SSE chunks terminated by a `data: [DONE]` message with `stream: true` in the request (Intel OpenVINO OpenAI-API docs; OpenAI first-party docs unreachable, 403, so this is single-vendor confirmed plus community triangulation).
+- i18n follows GNU gettext practice: full sentences as message keys (never concatenated fragments), translator comments in source, PO-file based catalogs with context and plural forms, locale selected per request with a compiled-in fallback chain (GNU gettext manual).
+
+## Structure
+
+Directories (deep, one concern per file):
+
+```
+tools/molo/                 main() entry, links framework as a library
+include/molo/cli/           argument parsing, command tree, help, output
+src/cli/
+include/molo/dev/           dev server, file watcher, reloader
+src/dev/
+include/molo/embed/         asset embedder (scanner, emitter)
+src/embed/
+include/molo/ai/            AI harness client, prompts, session
+src/ai/
+include/molo/ops/           health, readiness, metrics export, probe routes
+src/ops/
+include/molo/i18n/          locale, catalog, translator, PO loader
+src/i18n/
+include/molo/csp/           policy model, builder, nonce manager
+src/csp/
+```
+
+Namespaces and classes (one class per header/source pair):
+
+- `molo::cli`: `Command`, `CommandTree`, `Flag`, `FlagSet`, `ArgParser`, `HelpRenderer`, `TableWriter`, `JsonWriter`, `ExitCode` (enum: Ok=0, Usage=2, Runtime=1, Unavailable=69).
+- `molo::dev`: `DevServer`, `FileWatcher` (inotify/kqueue/ReadDirectoryChangesW backends behind one interface), `Reloader`, `DevConfig` vs `ProdConfig` split type.
+- `molo::embed`: `AssetScanner`, `EmbedEmitter` (directory to `.cpp`/`.hpp`), `EmbeddedBundle` (runtime reader used by static routes in 01-runtime.md).
+- `molo::ai`: `AiClient` (built on the 08-clients-openapi.md HTTP client), `AiSession`, `PromptPack` (framework feature prompts), `SseStreamParser` (shared with 06-realtime.md SSE client, not duplicated).
+- `molo::ops`: `HealthRegistry`, `HealthCheck` (base), `LivenessCheck`, `ReadinessCheck`, `StartupGate`, `PrometheusExporter`, `OtlpBridge` (thin; transport owned by 02-observability.md).
+- `molo::i18n`: `Locale`, `Catalog`, `PoLoader`, `Translator`, `PluralRule`.
+- `molo::csp`: `CspPolicy`, `CspBuilder`, `Nonce` (per-request, from 03-security.md crypto).
+
+## Rules (hard, numbered)
+
+1. The CLI never performs a destructive or history-rewriting action autonomously beyond its spec: `molo new` refuses to write into a non-empty directory without `--force`, and nothing here touches git history (history rules live in 10-workflow-example.md).
+2. Every command returns 0 on success, 2 on usage errors, 1 on runtime failures, 69 when a required service is unavailable; scripts must be able to rely on this mapping.
+3. Primary command output goes to stdout; logs, progress, and errors go to stderr; `--format json` output on stdout must be parseable JSON with no log interleaving.
+4. `--help` on every command and subcommand; `-h` prints concise help, `help <cmd>` prints the full page with examples; no command without documented usage.
+5. Dev server is dev-only: it refuses to start with production config unless `--allow-prod` is passed, and it never enables the reloader against a production database URL.
+6. Health endpoints never require auth, never allocate unbounded memory, and must answer in under 50 ms p99 with all checks short-circuited on timeout (each check capped at 200 ms).
+7. Metrics export has static cardinality caps: max 10000 series per process, max 64 label pairs per metric, per-route labels use route templates (`/users/{id}`), never raw paths.
+8. Embedded builds are byte-identical reproducible: same input directory plus same tool version yields identical generated bytes (sorted file order, no timestamps in output).
+9. The AI harness never sends secrets: `.env.json` values, key files, and anything matching `*_KEY`, `*_SECRET`, `*_TOKEN` are redacted before prompt construction; redaction failures abort the request.
+10. Default CSP is deny-first: generated pages start at `default-src 'none'` plus explicit allows, and `report-only` mode is the default until the app opts into enforcement.
+11. All user-visible CLI and template strings go through the translator; no hardcoded English outside the fallback catalog.
+
+## Development guidelines
+
+- Build the CLI as data first: declare the command tree (names, flags, help) in one place, then attach handlers; help and shell completion generate from the same tree.
+- Reuse, never reimplement: the AI harness uses the 08-clients-openapi.md HTTP client and the 06-realtime.md SSE parser; metrics use the 02-observability.md registry; nonces and hashing use 03-security.md; static serving of embedded bundles uses 01-runtime.md.
+- Keep the dev server dumb: watch, rebuild via `make`, restart the child process, preserve no state; stateful niceties (state-preserving reload) are explicitly out of scope for v1.
+- Prefer `std::expected` or an equivalent result type over exceptions across CLI/ops boundaries so exit codes map cleanly.
+- Every new command ships with its spec entry, its help text, one integration test through the real binary, and a living-example mention where user-visible.
+
+## Specifications (concrete behaviors, caps, formats)
+
+CLI command tree (v1):
+
+```
+molo new <name> [--backend postgres|sqlite] [--force]
+molo serve [--port 8000] [--reload] [--config .env.json]
+molo build [--embed assets/] [--out build/]
+molo embed <dir> --out <file.cpp> [--namespace molo::assets] [--gzip]
+molo migrate [--check | --apply]                 (executes 04-data.md plans)
+molo supervisor start|stop|restart|status|logs   (drives 07-supervisor-replication.md)
+molo task enqueue|retry|cancel|inspect           (drives 07-supervisor-replication.md)
+molo ai new|continue [--stream] [--base-url URL] [--model NAME]
+molo health [--format json|text]
+molo version [--format json|text]
+```
+
+Global flags: `-h/--help`, `--version`, `--format json|text` (default text, TTY-aware: no color when piped), `--verbose/-v` (repeatable, max 3), `--quiet/-q`.
+
+Exit codes: 0 ok, 1 runtime failure, 2 usage error, 69 service unavailable (matches SYSEXITS EX_UNAVAILABLE).
+
+Dev server: `molo serve --reload` watches `src/`, `templates/`, `assets/` (max 20000 files, debounce 150 ms, recursive); on change it runs `make -j$(nproc)` with a 300 s cap, kills the child with SIGTERM then SIGKILL after 5 s, and restarts; crash loops back off 1 s, 2 s, 4 s up to 30 s, then hold with a loud message. Dev/prod split: `.env.json` plus `MOLO_ENV=dev|prod`; prod requires TLS config and refuses `--reload` without `--allow-prod`.
+
+Embedder: input max 512 MB total, max 50000 files, max 64 MB per file; skips dotfiles and symlinks escaping the root; emits one `.cpp` plus one `.hpp` with `constexpr` byte arrays (chunked at 1 MB per array), a sorted manifest (path, size, sha256, mime), and optional precompressed gzip variants for text types over 1 KB; output includes tool version string for audit.
+
+AI harness: base URL default `https://devplace.net/openai/v1`, model default `molodetz`, key from `DEVPLACE_API_KEY` env only (never a flag, never stored); request `POST {base}/chat/completions` with `{"model","messages","stream":true}`; response parsed as SSE (`text/event-stream`), JSON per `data:` line, end on `data: [DONE]`; caps: 30 s connect timeout, 300 s session cap, max 8 MB transcript, max 50 tool/file actions per session, human confirms every file write (plan-then-apply, no silent writes).
+
+Health/metrics endpoints (mounted by `molo::ops`, paths fixed):
+
+- `GET /healthz` liveness: 200 `{"status":"ok"}` when the loop serves; fails only when shutdown is stuck.
+- `GET /readyz` readiness: 200 when DB, KV, and supervisor heartbeats pass; 503 `{"status":"degraded","failed":[...]}` otherwise.
+- `GET /startupz` startup gate: 503 until boot completes, then 200.
+- `GET /metrics` Prometheus text 0.0.4 (`text/plain; version=0.0.4`), gzip if accepted; OpenTelemetry OTLP export optional via 02-observability.md bridge.
+- Budgets: scrape render under 100 ms p99 at 10k series; export interval floor 5 s; series cap 10000 with `molo_metrics_dropped_total` counter on overflow.
+
+i18n: locale format BCP 47 (`en`, `nl-NL`); resolution order: route prefix (`/nl/...`), cookie `molo_locale`, `Accept-Language` (q-values honored), config default; catalogs are PO files under `locales/<locale>/LC_MESSAGES/*.po`, compiled to embedded bundles for prod; plural rules per CLDR cardinal set for shipped locales; missing key falls back through `nl-NL` to `nl` to `en` to msgid, and logs once per key per process.
+
+CSP builder: `CspBuilder` fluent API (`default_src()`, `script_src()`, `style_src()`, `img_src()`, `connect_src()`, `frame_ancestors()`, `report_to()`); default policy `default-src 'none'; script-src 'self'; style-src 'self'; img-src 'self' data:; connect-src 'self'; frame-ancestors 'none'`; per-request nonces for inline scripts (`'nonce-<base64>'`, 16 bytes from 03-security.md RNG); report endpoint `POST /csp-report` (JSON, capped 64 KB per report, 100 reports/min/IP); `Content-Security-Policy-Report-Only` until `enforce()` is called.
+
+## Execution steps (in build order)
+
+1. CLI skeleton: `Command`, `ArgParser`, `HelpRenderer`, exit-code mapping, `--format`, TTY detection, `molo version` and `molo health` (stub checks).
+2. `molo new` scaffolding: Makefile, AGENTS.md stub, directory layout, `.env.json`, `.gitignore`, `git init`, plug-and-play example files.
+3. Health/readiness/startup endpoints plus registry; wire DB/KV/supervisor checks as those components land.
+4. Prometheus exporter on the 02-observability.md registry with series caps and gzip.
+5. Dev server with watcher backends, rebuild/restart loop, dev/prod config split enforcement.
+6. Embedder scanner plus emitter plus runtime bundle reader; static-route integration test.
+7. CSP builder, nonce plumbing into templates (with 05-request-routing.md), report endpoint, report-only default.
+8. i18n core: locale resolution, PO loader, translator, template filter/function hooks (with 05-request-routing.md).
+9. AI harness: client on 08-clients-openapi.md HTTP, SSE parsing via 06-realtime.md, redaction, plan-then-apply loop, `molo ai` commands.
+10. Supervisor/task CLI surface on top of 07-supervisor-replication.md APIs; structured JSON output for scripts.
+
+## Validation (how to verify each step)
+
+1. Run `molo version`, `molo --help`, `molo nosuch`; assert exit codes 0/0/2 and JSON parseability of `--format json`; pipe stdout and confirm no color codes.
+2. Scaffold into a temp dir; assert all files exist, `git status` clean after init, `make` succeeds, and re-run without `--force` fails with exit 1.
+3. Start app, curl `/healthz`, `/readyz`, `/startupz`; kill the DB and assert `/readyz` flips to 503 within 2 s while `/healthz` stays 200; measure p99 under 50 ms over 1000 requests.
+4. Curl `/metrics`, validate with `promtool check metrics`; assert trailing newline, HELP/TYPE lines, gzip variant identical after decode; flood labels and assert the 10000 cap plus dropped counter.
+5. Touch a template, assert rebuild plus restart under 5 s; assert `--reload` with prod config exits 2 without `--allow-prod`; kill child 5 times, assert backoff holds.
+6. Embed a fixture tree twice, assert byte-identical output; serve from the bundle and diff bodies plus ETags against disk serving; feed a 65 MB file and assert rejection.
+7. Render a page, assert CSP header present with nonce matching the inline script tag; post a violation report and assert 202 plus storage; assert report-only header by default and enforcing header after `enforce()`.
+8. Request `/nl/` and assert translated strings; remove one key and assert fallback chain plus single log line; load a PO with plurals and assert all plural forms render.
+9. Run `molo ai new` against a local stub SSE server returning two chunks plus `[DONE]`; assert transcript assembly, redaction of a planted `API_KEY` value, and that no file is written without confirmation; point at the real DevPlace endpoint only in a manual opt-in test.
+10. Enqueue/retry/cancel a fixture task via CLI with `--format json`; assert state transitions match `07-supervisor-replication.md` semantics and output parses.
+
+## References (sibling files this depends on or feeds)
+
+- Depends on 01-runtime.md (static serving of bundles, lifecycle), 02-observability.md (registry, logging, OTLP), 03-security.md (nonces, redaction-adjacent crypto, TLS posture), 04-data.md (migrate targets, readiness checks), 05-request-routing.md (template hooks for i18n/CSP, middleware mount points), 06-realtime.md (SSE parser reuse), 07-supervisor-replication.md (CLI surface targets), 08-clients-openapi.md (HTTP client for AI harness).
+- Feeds 10-workflow-example.md (CLI-verified commands, example pages proving i18n/CSP/health, `molo new` template source).
+
+## Open questions
+
+1. Should `molo` be one static binary or a thin driver plus per-project plugin commands? Leaning single binary for v1.
+2. Do we ship shell completion scripts (bash/zsh/fish) generated from the command tree in v1, or defer to v1.1?
+3. Should `/metrics` require auth in production by default ( risk: scraping complexity) or stay open with IP allow-listing?
+4. Which locales ship compiled-in for v1 beyond `en`: `nl` only, or a wider starter set?
+5. Does the AI harness need a non-streaming fallback for proxies that buffer SSE, or is streaming mandatory?
+
+## Sources (exact URLs with retrieval dates)
+
+- Command Line Interface Guidelines, https://clig.dev/ (retrieved 2026-10-23): exit codes, stdout/stderr split, help conventions, machine-readable output.
+- Prometheus exposition formats, https://prometheus.io/docs/instrumenting/exposition_formats/ (retrieved 2026-10-23): text 0.0.4 line format, HELP/TYPE, content type, primitives.
+- Kubernetes probes task guide, https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ (retrieved 2026-10-23): liveness restart semantics, probe timing fields.
+- MDN Content-Security-Policy, https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy (retrieved 2026-10-23): default-src fallback, report-only header, reporting directives.
+- MDN Using server-sent events, https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events (retrieved 2026-10-23): text/event-stream, data blocks, blank-line framing.
+- Intel OpenVINO OpenAI chat API docs, https://docs.openvino.ai/2024/ovms_docs_rest_api_chat.html (found 2026-10-23 via search snippet): stream flag, data-only SSE, `data: [DONE]` terminator. Note: OpenAI first-party streaming reference (https://platform.openai.com/docs/api-reference/chat-streaming) returned HTTP 403 on 2026-10-23; convention triangulated with MDN SSE and multiple compatible-server docs.
+- GNU gettext manual, https://www.gnu.org/software/gettext/manual/gettext.html (retrieved 2026-10-23): msgid practice, PO format, plural and context entries, locale selection.

plans/10-workflow-example.md

@@ -0,0 +1,171 @@
+# 10 - Workflow, Testing Discipline, Living Example
+
+Scope: plan.md sections 8-9, 10 step 7. Topics: specs, AGENTS.md, autonomous git, integration testing, living example.
+
+## 1. Purpose and philosophy
+
+This file defines how MoloApp is built, not what it does. Feature scope lives in siblings 01-runtime.md through 09-cli-ops.md; this file owns the process that keeps those features consistent: structure-driven development (every component has a written spec), an agent-executable workflow (`AGENTS.md`), autonomous versioning and release, integration-only testing through the real stack, and one living example project that proves every feature works end to end.
+
+Philosophy, from plan.md section 9:
+
+- Structure first: no code without a spec declaration somewhere; docs state what the system does today, gaps listed next to the code that closes them.
+- Isolation with DRY integration: components depend only on declared interfaces, shared behavior lives in exactly one base class or component.
+- Autonomous by default, human on floors: the AI agent builds, tests, versions, commits, tags, deploys and verifies in one session; humans intervene only on conflicts, protected paths, retention windows, destructive ops, and release failures.
+- A change that is not running is not done: commit, deploy, verify, same session.
+- User-facing simplicity is a release gate; responsive-by-default is a ship blocker for every template and example UI.
+
+## 2. Best practices (from research)
+
+1. AGENTS.md is a README for agents: a dedicated predictable place for build steps, tests, and conventions, kept separate so READMEs stay human-focused (https://agents.md/, retrieved 2026-10-23). MoloApp adopts this split literally: root `AGENTS.md` is normative for the agent; `README.md` is human marketing plus quickstart.
+2. Conventional Commits gives the subject line a machine-parseable structure `<type>[optional scope]: <description>` with optional body and footers, where `fix:` maps to PATCH, `feat:` to MINOR, and `BREAKING CHANGE:` footer or `!` marker to MAJOR (https://www.conventionalcommits.org/en/v1.0.0/, retrieved 2026-10-23). MoloApp uses exactly this mapping for autonomous bumps.
+3. Semantic Versioning 2.0.0 defines MAJOR for incompatible API changes, MINOR for backward-compatible additions, PATCH for backward-compatible fixes (https://semver.org/, retrieved 2026-10-23). Triangulated with (2) and (5): commit type is the sole version signal, no manual version choice.
+4. Keep a Changelog requires one `CHANGELOG.md`, Unreleased section at top, `Added/Changed/Fixed/Removed` groups per release, ISO 8601 dates, and commit-log dumps explicitly rejected (https://keepachangelog.com/en/1.0.0/, retrieved 2026-10-23). MoloApp generates the changelog from parsed commits but edits entries into human-readable form before tagging.
+5. semantic-release automates determining the next version, generating release notes, and publishing, removing the connection between human emotions and version numbers and running only after successful CI (https://semantic-release.gitbook.io/semantic-release/, retrieved 2026-10-23; canonical docs now at https://semantic-release.org/). Triangulated with (2)+(3): MoloApp release pipeline runs only on a green suite and derives version purely from commits since last tag.
+6. FastAPI documents testing via `TestClient(app)` exercising real routes with `client.get/post` and status plus body assertions (https://fastapi.tiangolo.com/tutorial/testing/, retrieved 2026-10-23). Django documents simulating requests through the full stack (URL, view, middleware, form, template) with its test Client (https://docs.djangoproject.com/en/5.0/topics/testing/, retrieved 2026-10-23). Triangulated: MoloApp integration tests follow the same black-box route-level pattern, but stricter (no isolated unit tests at all; every test hits a URL end to end, see 08-clients-openapi.md for the test client itself).
+
+## 3. Structure
+
+Deep structure, one class or concern per file, namespaces mirror directories.
+
+```
+MoloApp/
+  AGENTS.md                  # normative agent workflow (this file specifies its content)
+  README.md                  # human quickstart (no agent instructions)
+  CHANGELOG.md               # Keep a Changelog format, agent-maintained
+  VERSION                    # single line MAJOR.MINOR.PATCH, agent-maintained
+  Makefile                   # canonical targets: build, test, lint, docs, release, verify
+  specs/
+    SPEC-TEMPLATE.md         # normative template (see section 6.1)
+    runtime.md               # one spec per sibling area, mirrors 01-runtime.md through 09-cli-ops.md
+    observability.md
+    security.md
+    data.md
+    request-routing.md
+    realtime.md
+    supervisor-replication.md
+    clients-openapi.md
+    cli-ops.md
+    workflow-example.md      # this plan, once accepted
+  tools/
+    release/                 # autonomous git/version/changelog pipeline (behavior here, CLI UX in 09-cli-ops.md)
+      ReleasePlanner.hpp     # molo::workflow::ReleasePlanner: commits since tag -> bump decision
+      ChangelogWriter.hpp    # molo::workflow::ChangelogWriter: commits -> CHANGELOG entries
+      ReleaseVerifier.hpp    # molo::workflow::ReleaseVerifier: green-suite + live checks before tag
+  tests/
+    integration/             # ONLY test location; mirrors app routes (see 6.3)
+      CASE-TEMPLATE.md       # normative test-case template
+      health_test.cpp
+      <router>/
+        <route>_test.cpp
+  examples/
+    living/                  # the big living example project (see 6.4)
+      App.cpp                # mounts every demo sub-app (application chaining demo)
+      routers/
+      views/
+      templates/
+      static/
+      tests/                 # example suite, same templates as tests/
+      README.md              # feature coverage matrix with links to routes
+```
+
+Namespaces: `molo::workflow` for ReleasePlanner, ChangelogWriter, ReleaseVerifier. No other framework namespace may contain workflow logic. Test files use `molo::test` helpers only. Example app uses `molo::example::` sub-namespaces per demo area.
+
+## 4. Rules (hard, numbered)
+
+1. No feature is done until: spec updated, docs updated, living example demonstrates it, full suite green, all in the same session.
+2. Integration tests only: every test exercises a URL end to end through real HTTP, routing, middleware, and DB. Isolated unit tests are forbidden.
+3. Test placement mirrors the app: test file path and name derive from the URL/router under test; exactly one structured place per route.
+4. Conventional Commits format is mandatory for every commit; malformed commits are rejected by the pipeline before version analysis.
+5. Version numbers are never hand-chosen; the bump derives solely from commit types since the last tag (feat -> minor, fix -> patch, BREAKING CHANGE or `!` -> major).
+6. Tagging requires a green suite plus live verification after deploy, same session; failed verification blocks the tag.
+7. Destructive git ops (history rewrite, force push, hard reset) are never autonomous: dry-run plus explicit human confirmation, no exceptions.
+8. Protected floors the agent must never cross alone: `specs/`, `CHANGELOG.md`, `VERSION`, tags, retention windows, destructive file deletes; on a floor hit, stop and page a human.
+9. Every shipped template, example UI, and generated page works on small phones; big button groups collapse to dropdowns or equivalent, verified in the example suite.
+10. Specs are normative: on any conflict between code and spec, the spec wins until formally amended in the same change.
+
+## 5. Development guidelines
+
+- Write the spec delta first, then code, then example coverage, then tests, then changelog entry, then commit. Never commit code whose spec is stale.
+- Keep changes small and reviewable: one router or one behavior per commit; squash agent thrash before tagging.
+- Prefer static ceilings over autotuning everywhere (timeouts, caps, intervals are constants in specs, visible in `AGENTS.md`).
+- Loud skips: if input, fixture, or environment cannot be verified, fail the test with a clear reason and counter, never silently pass.
+- Reproduce bugs through the real stack first (failing route-level test), then fix at root cause; the reproduction test stays in the suite.
+- Docs use fixed templates; every doc names what the system does today and lists gaps next to the code that will close them.
+- `AGENTS.md` is the single entry point for autonomous work: any workflow an agent must run (build, test, lint, docs, bump, commit, tag, deploy, verify) is invocable from there with exact commands.
+
+## 6. Specifications
+
+### 6.1 Spec template (specs/SPEC-TEMPLATE.md)
+
+Every spec file contains, in order: Status (draft/accepted/deprecated) and date; Behaviors (numbered, testable); Caps (every rate, size, count, interval, memory ceiling as literal constants); Formats (wire, file, log, error shapes); Errors (exception type per failure, status mapping); Isolation (declared interfaces it may depend on, explicit non-dependencies); Gaps (what is not true yet, with owning file). Max 500 lines per spec; split by sub-concern beyond that.
+
+### 6.2 AGENTS.md content and autonomous git pipeline
+
+`AGENTS.md` sections, in order: Setup commands; Build; Test; Lint; Docs update; Version bump; Commit; Tag; Deploy; Verify live; Floors and escalation. Each section gives one copy-paste command (Makefile target) plus expected output and failure action.
+
+Commit format: `<type>[(scope)][!]: <description>` plus optional body plus `BREAKING CHANGE:` footer, per Conventional Commits 1.0.0. Allowed types: feat, fix, docs, style, refactor, perf, test, build, ci, chore. Scope is the sibling area slug (runtime, security, data, routing, realtime, supervisor, clients, cli, workflow).
+
+Bump mapping: any `!` or `BREAKING CHANGE` -> MAJOR; else any `feat` -> MINOR; else any `fix` -> PATCH; else no release. `VERSION` file and annotated git tag `vMAJOR.MINOR.PATCH` update atomically with the changelog entry.
+
+Changelog: Keep a Changelog groups (Added, Changed, Fixed, Removed) under `## [x.y.z] - YYYY-MM-DD`, Unreleased section always present, ISO dates only.
+
+Pipeline order: lint -> full integration suite -> bump decision -> changelog -> commit -> tag -> deploy -> live verify. Any red step halts the pipeline; the agent reports, never force-pushes past red. CLI command surface for these steps is specified in 09-cli-ops.md; this file owns the ordering, gating, and format rules.
+
+Caps: max 1 release per green suite run; changelog entry max 20 lines per version; commit subject max 100 chars; pipeline timeout 30 min, then human escalation.
+
+### 6.3 Integration test discipline
+
+Location: `tests/integration/<router-path>/<route>_test.cpp`. Naming: route `/users/:id` -> `users/id_test.cpp`. Fixtures: DB/ORM fixtures plus frozen time via the test client from 08-clients-openapi.md; no real network except loopback server under test.
+
+Case template (CASE-TEMPLATE.md): Arrange (seed via app API, never direct DB writes unless testing self-healing); Act (one HTTP request through the test client); Assert (status code, content-type, body shape, plus one side-effect check: DB row, metric emitted, log line). Every route needs at minimum: happy path, auth failure, validation failure, and one limit case (body too large, rate limited, or not found).
+
+Caps: full suite max 10 min on reference hardware; single test max 30 s; fixture DBs reset per file, never shared across files; flaky test policy is zero tolerance (quarantine file plus spec gap entry, fix within one session).
+
+### 6.4 Living example project (examples/living/)
+
+Must demonstrate, each reachable via a documented URL and covered by `examples/living/tests/`: routing, views, routers, ORM both backends, auth, sessions, forms, validation, templates, all response formats, streaming/passthrough, WebSocket plus Socket.IO, pubsub, events, metrics, telemetry, KV, cache, uploads, WebDAV, replication, supervisor plus background tasks, REST client self-call, OpenAPI page, embedded static assets, AI harness CLI sample, structured integration tests.
+
+Coverage matrix in `examples/living/README.md`: table of feature -> route -> test file -> spec link. Matrix row count equals feature count; CI fails on missing row. Example must compile warning-free and stay green as part of the normal suite; it is also the manual post-deploy verification target (open index, click through matrix top 10 routes on a small-phone viewport).
+
+## 7. Execution steps (in build order)
+
+1. Write `specs/SPEC-TEMPLATE.md` and stub specs for all areas.
+2. Write root `AGENTS.md` with build/test/lint/docs/bump/commit/tag/deploy/verify commands wired to the Makefile.
+3. Implement `molo::workflow` release pipeline (ReleasePlanner, ChangelogWriter, ReleaseVerifier) with dry-run mode.
+4. Write `tests/integration/CASE-TEMPLATE.md` plus first health-route test proving the harness works.
+5. Scaffold `examples/living/` with index, coverage matrix, and one route per area stub.
+6. Fill example coverage feature by feature alongside siblings 01-runtime.md through 09-cli-ops.md; each feature lands with spec, example route, and tests in one session.
+7. Harden pipeline: floor guards, protected paths, destructive-op blocks, timeout and escalation paths.
+8. First autonomous end-to-end run: green suite -> bump -> changelog -> tag -> deploy -> live verify -> matrix check.
+
+## 8. Validation (how to verify each step)
+
+1. Template check: every spec file parses against the section order in 6.1; reject on missing Caps or Gaps.
+2. Agent replay: a fresh agent following only `AGENTS.md` reaches a green build plus green suite with zero questions.
+3. Pipeline dry-run: on a fixture repo with known commits, bump decision and changelog output match expected version and groups exactly.
+4. Harness proof: health-route test passes and fails correctly when the route is broken (mutation check).
+5. Matrix audit: every matrix row resolves to a live route, a test file, and a spec anchor; script fails on dangling links.
+6. Per-feature gate: rule 1 checklist enforced by CI (spec diff plus example diff plus tests diff present in the same release).
+7. Floor drill: attempt a protected-path write and a force push in dry-run; both must refuse and request human confirmation.
+8. Live verify: after deploy, automated health plus metrics plus trace checks pass and the example index loads on a 360px viewport.
+
+## 9. References
+
+Depends on: 09-cli-ops.md (CLI command surface the pipeline invokes, dev server, Makefile targets); 08-clients-openapi.md (test client, fixtures, time freezing used by every test); 02-observability.md (log format, metrics, trace linkage asserted in tests). Feeds: all siblings 01-runtime.md through 09-cli-ops.md (spec template, test template, release gates, and example coverage apply to every area). Never duplicates: CLI flags and UX (09-cli-ops.md), test-client API (08-clients-openapi.md), metric/log schemas (02-observability.md).
+
+## 10. Open questions
+
+1. Reference hardware for the 10-minute suite cap: which CI runner SKU is normative?
+2. Deploy target for live verification: single binary on loopback, or staged multi-instance per 07-supervisor-replication.md?
+3. Should `BREAKING CHANGE` commits require a human co-sign before MAJOR tagging, or stay fully autonomous?
+4. Example backend default: SQLite for speed with Postgres nightly, or both on every run?
+
+## 11. Sources
+
+- AGENTS.md open format, https://agents.md/ (retrieved 2026-10-23)
+- Conventional Commits 1.0.0, https://www.conventionalcommits.org/en/v1.0.0/ (retrieved 2026-10-23)
+- Semantic Versioning 2.0.0, https://semver.org/ (retrieved 2026-10-23)
+- Keep a Changelog 1.0.0, https://keepachangelog.com/en/1.0.0/ (retrieved 2026-10-23)
+- semantic-release docs, https://semantic-release.gitbook.io/semantic-release/ (retrieved 2026-10-23)
+- FastAPI testing tutorial, https://fastapi.tiangolo.com/tutorial/testing/ (retrieved 2026-10-23)
+- Django testing docs, https://docs.djangoproject.com/en/5.0/topics/testing/ (retrieved 2026-10-23)

plans/README.md

@@ -0,0 +1,48 @@
+# MoloApp Detailed Plans
+
+Index for the ten detailed plan files. Each file is self-contained with purpose, best practices, structure, rules, guidelines, specifications, execution steps, validation, references, open questions, and sources.
+
+## Index
+
+| # | File | Title | Scope (one line) |
+|---|------|-------|------------------|
+| 1 | 01-runtime.md | Core Runtime and HTTP Fundamentals | Event loop, lifecycle, HTTP/1.1 + HTTP/2, compression, cookies, static. |
+| 2 | 02-observability.md | Observability Backbone | Logging, typed events, metrics, tracing, correlation, zero-cost gating. |
+| 3 | 03-security.md | Security | TLS wolfSSL, auth, hashing/crypto, sessions, guards, upload safety. |
+| 4 | 04-data.md | Data Layer | ORM dual backends, self-healing, pooling, transactions, KV, cache, pubsub. |
+| 5 | 05-request-routing.md | Request/Response, Routing, Views, Templates | App, routers, views, validation/forms, negotiation, templates, middleware, DI, errors. |
+| 6 | 06-realtime.md | Realtime (WebSocket, SSE, Socket.IO, Streaming, WebDAV, Client Parity) | Live protocols plus streaming plus WebDAV, every one with client parity. |
+| 7 | 07-supervisor-replication.md | Supervisor, Tasks, Replication | Process supervisor, background tasks, pubsub durability, replication, multi-instance. |
+| 8 | 08-clients-openapi.md | Clients, OpenAPI, Test Client | HTTP/REST client, REST standards, OpenAPI generation, test client. |
+| 9 | 09-cli-ops.md | CLI Tooling, Dev Server, Operations | CLI, dev server, embedder, AI harness, health/metrics, i18n/CSP. |
+| 10 | 10-workflow-example.md | Workflow, Testing Discipline, Living Example | Specs, AGENTS.md, autonomous git, integration testing, living example. |
+
+## How files reference each other
+
+- References always use exact sibling filenames (`01-runtime.md` through `10-workflow-example.md`).
+- Each file has a References section naming Depends on (interfaces it consumes) and Feeds (interfaces it provides).
+- Dependency direction (consumes, upstream first):
+  - `01-runtime.md` depends on `02-observability.md` (hooks) and `03-security.md` (TLS handle); feeds `05-request-routing.md`, `06-realtime.md`, `08-clients-openapi.md`, `09-cli-ops.md`.
+  - `02-observability.md` depends on `01-runtime.md` (lifecycle hooks); feeds `04-data.md`, `05-request-routing.md`, `06-realtime.md`, `07-supervisor-replication.md`, `08-clients-openapi.md`, `09-cli-ops.md`, `10-workflow-example.md`.
+  - `03-security.md` depends on `01-runtime.md` and `02-observability.md`; feeds `04-data.md`, `05-request-routing.md`, `06-realtime.md`, `07-supervisor-replication.md`, `09-cli-ops.md`, `10-workflow-example.md`.
+  - `04-data.md` depends on `01-runtime.md`, `02-observability.md`, `03-security.md`; feeds `05-request-routing.md`, `06-realtime.md`, `07-supervisor-replication.md`, `08-clients-openapi.md`, `10-workflow-example.md`.
+  - `05-request-routing.md` depends on `01-runtime.md`, `02-observability.md`, `03-security.md`, `04-data.md`; feeds `06-realtime.md`, `07-supervisor-replication.md`, `08-clients-openapi.md`, `09-cli-ops.md`; follows `10-workflow-example.md`.
+  - `06-realtime.md` depends on `01-runtime.md`, `02-observability.md`, `03-security.md`, `05-request-routing.md`; feeds `04-data.md` (pubsub bridge), `07-supervisor-replication.md`, `08-clients-openapi.md`, `10-workflow-example.md`.
+  - `07-supervisor-replication.md` depends on `01-runtime.md`, `02-observability.md`, `04-data.md`, `06-realtime.md`; feeds `09-cli-ops.md`, `10-workflow-example.md`.
+  - `08-clients-openapi.md` depends on `01-runtime.md`, `02-observability.md`, `03-security.md`, `04-data.md`, `05-request-routing.md`; defers realtime clients to `06-realtime.md`; feeds `09-cli-ops.md`, `10-workflow-example.md`.
+  - `09-cli-ops.md` depends on `01-runtime.md` through `08-clients-openapi.md`; feeds `10-workflow-example.md`.
+  - `10-workflow-example.md` depends on `02-observability.md`, `08-clients-openapi.md`, `09-cli-ops.md`; feeds `01-runtime.md` through `09-cli-ops.md` (spec template, test discipline, release gates).
+
+## Overall execution order
+
+Build order follows plan.md section 10, with the file sequence matching dependency order:
+
+1. `01-runtime.md` steps 1-10 plus `02-observability.md` steps 1-10 (plan step 1: foundation and instrumentation).
+2. `03-security.md` steps 1-10 (plan step 2: TLS, auth, guards, uploads).
+3. `04-data.md` steps 1-10 (plan step 3: drivers, ORM, sync/heal, KV/cache/pubsub).
+4. `06-realtime.md` steps 1-5 plus `07-supervisor-replication.md` steps E7-1 through E7-4 (plan step 4: realtime base, supervisor and task runner).
+5. `05-request-routing.md` steps E1-E12 plus `08-clients-openapi.md` steps 1-8 plus `09-cli-ops.md` steps 1-10 (plan step 5: app layer, clients, OpenAPI, CLI/ops).
+6. `07-supervisor-replication.md` steps E7-5 through E7-8 (plan step 6: durable streams, replication, multi-instance).
+7. `10-workflow-example.md` steps 1-8 throughout and finally (plan step 7: specs, AGENTS.md, pipeline, living example, autonomous end-to-end run).
+
+Rule: no file is done until its validation section passes through the real stack and `10-workflow-example.md` rule 1 holds (spec updated, docs updated, living example demonstrates it, full suite green, same session).