moloapp/plans/09-cli-ops.md

# 09 - CLI Tooling, Dev Server, Operations

Scope: plan.md sections 2, 7-8, 10 step 5 partial. Topics: CLI, dev server, embedder, AI harness, health/metrics, i18n/CSP.

## Purpose and philosophy

One binary, `molo`, is the only project tool: it scaffolds, serves, embeds, supervises, and vibe-codes. Operators get health, readiness, and metrics endpoints that work with stock Prometheus and Kubernetes probes with zero glue code. Developers get a dev server with fast restart and a strict dev/prod settings split. Templates get i18n hooks and a CSP builder by default, so every generated page is localizable and locked down. Philosophy: human-first output that degrades gracefully to machine-readable output, static ceilings on everything watched or exported, and dev/prod parity except where explicitly split.

## Best practices (from research)

- CLI design follows the Command Line Interface Guidelines: zero exit code on success and non-zero on failure, primary output to stdout, diagnostics to stderr, concise help by default with extensive help on request, and machine-readable output (JSON lines or `--format json`) where it does not hurt usability (clig.dev).
- Metrics exposition follows the Prometheus text format 0.0.4: line oriented, `\n` terminated with trailing newline on the last line, `# HELP` and `# TYPE` comment lines before samples, `Content-Type: text/plain; version=0.0.4`, optional gzip encoding, counter/gauge/histogram/summary/untyped primitives (Prometheus docs).
- Health semantics follow the Kubernetes probe model: liveness detects a broken state that only a restart fixes (kubelet kills and restarts the container on failure), readiness gates traffic, startup protects slow-starting containers with its own window (Kubernetes docs).
- CSP starts from `default-src 'none'` and opens only what is needed; `default-src` is the fallback for all other fetch directives; rollout begins in report-only mode (`Content-Security-Policy-Report-Only`) before enforcing (MDN CSP, field practice).
- Streaming uses Server-Sent Events: `Content-Type: text/event-stream`, `Cache-Control: no-cache`, notifications as text blocks terminated by blank lines, `data:` payload lines (MDN SSE). OpenAI-compatible chat streaming sends data-only SSE chunks terminated by a `data: [DONE]` message with `stream: true` in the request (Intel OpenVINO OpenAI-API docs; OpenAI first-party docs unreachable, 403, so this is single-vendor confirmed plus community triangulation).
- i18n follows GNU gettext practice: full sentences as message keys (never concatenated fragments), translator comments in source, PO-file based catalogs with context and plural forms, locale selected per request with a compiled-in fallback chain (GNU gettext manual).

## Structure

Directories (deep, one concern per file):

```
tools/molo/                 main() entry, links framework as a library
include/molo/cli/           argument parsing, command tree, help, output
src/cli/
include/molo/dev/           dev server, file watcher, reloader
src/dev/
include/molo/embed/         asset embedder (scanner, emitter)
src/embed/
include/molo/ai/            AI harness client, prompts, session
src/ai/
include/molo/ops/           health, readiness, metrics export, probe routes
src/ops/
include/molo/i18n/          locale, catalog, translator, PO loader
src/i18n/
include/molo/csp/           policy model, builder, nonce manager
src/csp/
```

Namespaces and classes (one class per header/source pair):

- `molo::cli`: `Command`, `CommandTree`, `Flag`, `FlagSet`, `ArgParser`, `HelpRenderer`, `TableWriter`, `JsonWriter`, `ExitCode` (enum: Ok=0, Usage=2, Runtime=1, Unavailable=69).
- `molo::dev`: `DevServer`, `FileWatcher` (inotify/kqueue/ReadDirectoryChangesW backends behind one interface), `Reloader`, `DevConfig` vs `ProdConfig` split type.
- `molo::embed`: `AssetScanner`, `EmbedEmitter` (directory to `.cpp`/`.hpp`), `EmbeddedBundle` (runtime reader used by static routes in 01-runtime.md).
- `molo::ai`: `AiClient` (built on the 08-clients-openapi.md HTTP client), `AiSession`, `PromptPack` (framework feature prompts), `SseStreamParser` (shared with 06-realtime.md SSE client, not duplicated).
- `molo::ops`: `HealthRegistry`, `HealthCheck` (base), `LivenessCheck`, `ReadinessCheck`, `StartupGate`, `PrometheusExporter`, `OtlpBridge` (thin; transport owned by 02-observability.md).
- `molo::i18n`: `Locale`, `Catalog`, `PoLoader`, `Translator`, `PluralRule`.
- `molo::csp`: `CspPolicy`, `CspBuilder`, `Nonce` (per-request, from 03-security.md crypto).

## Rules (hard, numbered)

1. The CLI never performs a destructive or history-rewriting action autonomously beyond its spec: `molo new` refuses to write into a non-empty directory without `--force`, and nothing here touches git history (history rules live in 10-workflow-example.md).
2. Every command returns 0 on success, 2 on usage errors, 1 on runtime failures, 69 when a required service is unavailable; scripts must be able to rely on this mapping.
3. Primary command output goes to stdout; logs, progress, and errors go to stderr; `--format json` output on stdout must be parseable JSON with no log interleaving.
4. `--help` on every command and subcommand; `-h` prints concise help, `help <cmd>` prints the full page with examples; no command without documented usage.
5. Dev server is dev-only: it refuses to start with production config unless `--allow-prod` is passed, and it never enables the reloader against a production database URL.
6. Health endpoints never require auth, never allocate unbounded memory, and must answer in under 50 ms p99 with all checks short-circuited on timeout (each check capped at 200 ms).
7. Metrics export has static cardinality caps: max 10000 series per process, max 64 label pairs per metric, per-route labels use route templates (`/users/{id}`), never raw paths.
8. Embedded builds are byte-identical reproducible: same input directory plus same tool version yields identical generated bytes (sorted file order, no timestamps in output).
9. The AI harness never sends secrets: `.env.json` values, key files, and anything matching `*_KEY`, `*_SECRET`, `*_TOKEN` are redacted before prompt construction; redaction failures abort the request.
10. Default CSP is deny-first: generated pages start at `default-src 'none'` plus explicit allows, and `report-only` mode is the default until the app opts into enforcement.
11. All user-visible CLI and template strings go through the translator; no hardcoded English outside the fallback catalog.

## Development guidelines

- Build the CLI as data first: declare the command tree (names, flags, help) in one place, then attach handlers; help and shell completion generate from the same tree.
- Reuse, never reimplement: the AI harness uses the 08-clients-openapi.md HTTP client and the 06-realtime.md SSE parser; metrics use the 02-observability.md registry; nonces and hashing use 03-security.md; static serving of embedded bundles uses 01-runtime.md.
- Keep the dev server dumb: watch, rebuild via `make`, restart the child process, preserve no state; stateful niceties (state-preserving reload) are explicitly out of scope for v1.
- Prefer `std::expected` or an equivalent result type over exceptions across CLI/ops boundaries so exit codes map cleanly.
- Every new command ships with its spec entry, its help text, one integration test through the real binary, and a living-example mention where user-visible.

## Specifications (concrete behaviors, caps, formats)

CLI command tree (v1):

```
molo new <name> [--backend postgres|sqlite] [--force]
molo serve [--port 8000] [--reload] [--config .env.json]
molo build [--embed assets/] [--out build/]
molo embed <dir> --out <file.cpp> [--namespace molo::assets] [--gzip]
molo migrate [--check | --apply]                 (executes 04-data.md plans)
molo supervisor start|stop|restart|status|logs   (drives 07-supervisor-replication.md)
molo task enqueue|retry|cancel|inspect           (drives 07-supervisor-replication.md)
molo ai new|continue [--stream] [--base-url URL] [--model NAME]
molo health [--format json|text]
molo version [--format json|text]
```

Global flags: `-h/--help`, `--version`, `--format json|text` (default text, TTY-aware: no color when piped), `--verbose/-v` (repeatable, max 3), `--quiet/-q`.

Exit codes: 0 ok, 1 runtime failure, 2 usage error, 69 service unavailable (matches SYSEXITS EX_UNAVAILABLE).

Dev server: `molo serve --reload` watches `src/`, `templates/`, `assets/` (max 20000 files, debounce 150 ms, recursive); on change it runs `make -j$(nproc)` with a 300 s cap, kills the child with SIGTERM then SIGKILL after 5 s, and restarts; crash loops back off 1 s, 2 s, 4 s up to 30 s, then hold with a loud message. Dev/prod split: `.env.json` plus `MOLO_ENV=dev|prod`; prod requires TLS config and refuses `--reload` without `--allow-prod`.

Embedder: input max 512 MB total, max 50000 files, max 64 MB per file; skips dotfiles and symlinks escaping the root; emits one `.cpp` plus one `.hpp` with `constexpr` byte arrays (chunked at 1 MB per array), a sorted manifest (path, size, sha256, mime), and optional precompressed gzip variants for text types over 1 KB; output includes tool version string for audit.

AI harness: base URL default `https://devplace.net/openai/v1`, model default `molodetz`, key from `DEVPLACE_API_KEY` env only (never a flag, never stored); request `POST {base}/chat/completions` with `{"model","messages","stream":true}`; response parsed as SSE (`text/event-stream`), JSON per `data:` line, end on `data: [DONE]`; caps: 30 s connect timeout, 300 s session cap, max 8 MB transcript, max 50 tool/file actions per session, human confirms every file write (plan-then-apply, no silent writes).

Health/metrics endpoints (mounted by `molo::ops`, paths fixed):

- `GET /healthz` liveness: 200 `{"status":"ok"}` when the loop serves; fails only when shutdown is stuck.
- `GET /readyz` readiness: 200 when DB, KV, and supervisor heartbeats pass; 503 `{"status":"degraded","failed":[...]}` otherwise.
- `GET /startupz` startup gate: 503 until boot completes, then 200.
- `GET /metrics` Prometheus text 0.0.4 (`text/plain; version=0.0.4`), gzip if accepted; OpenTelemetry OTLP export optional via 02-observability.md bridge.
- Budgets: scrape render under 100 ms p99 at 10k series; export interval floor 5 s; series cap 10000 with `molo_metrics_dropped_total` counter on overflow.

i18n: locale format BCP 47 (`en`, `nl-NL`); resolution order: route prefix (`/nl/...`), cookie `molo_locale`, `Accept-Language` (q-values honored), config default; catalogs are PO files under `locales/<locale>/LC_MESSAGES/*.po`, compiled to embedded bundles for prod; plural rules per CLDR cardinal set for shipped locales; missing key falls back through `nl-NL` to `nl` to `en` to msgid, and logs once per key per process.

CSP builder: `CspBuilder` fluent API (`default_src()`, `script_src()`, `style_src()`, `img_src()`, `connect_src()`, `frame_ancestors()`, `report_to()`); default policy `default-src 'none'; script-src 'self'; style-src 'self'; img-src 'self' data:; connect-src 'self'; frame-ancestors 'none'`; per-request nonces for inline scripts (`'nonce-<base64>'`, 16 bytes from 03-security.md RNG); report endpoint `POST /csp-report` (JSON, capped 64 KB per report, 100 reports/min/IP); `Content-Security-Policy-Report-Only` until `enforce()` is called.

## Execution steps (in build order)

1. CLI skeleton: `Command`, `ArgParser`, `HelpRenderer`, exit-code mapping, `--format`, TTY detection, `molo version` and `molo health` (stub checks).
2. `molo new` scaffolding: Makefile, AGENTS.md stub, directory layout, `.env.json`, `.gitignore`, `git init`, plug-and-play example files.
3. Health/readiness/startup endpoints plus registry; wire DB/KV/supervisor checks as those components land.
4. Prometheus exporter on the 02-observability.md registry with series caps and gzip.
5. Dev server with watcher backends, rebuild/restart loop, dev/prod config split enforcement.
6. Embedder scanner plus emitter plus runtime bundle reader; static-route integration test.
7. CSP builder, nonce plumbing into templates (with 05-request-routing.md), report endpoint, report-only default.
8. i18n core: locale resolution, PO loader, translator, template filter/function hooks (with 05-request-routing.md).
9. AI harness: client on 08-clients-openapi.md HTTP, SSE parsing via 06-realtime.md, redaction, plan-then-apply loop, `molo ai` commands.
10. Supervisor/task CLI surface on top of 07-supervisor-replication.md APIs; structured JSON output for scripts.

## Validation (how to verify each step)

1. Run `molo version`, `molo --help`, `molo nosuch`; assert exit codes 0/0/2 and JSON parseability of `--format json`; pipe stdout and confirm no color codes.
2. Scaffold into a temp dir; assert all files exist, `git status` clean after init, `make` succeeds, and re-run without `--force` fails with exit 1.
3. Start app, curl `/healthz`, `/readyz`, `/startupz`; kill the DB and assert `/readyz` flips to 503 within 2 s while `/healthz` stays 200; measure p99 under 50 ms over 1000 requests.
4. Curl `/metrics`, validate with `promtool check metrics`; assert trailing newline, HELP/TYPE lines, gzip variant identical after decode; flood labels and assert the 10000 cap plus dropped counter.
5. Touch a template, assert rebuild plus restart under 5 s; assert `--reload` with prod config exits 2 without `--allow-prod`; kill child 5 times, assert backoff holds.
6. Embed a fixture tree twice, assert byte-identical output; serve from the bundle and diff bodies plus ETags against disk serving; feed a 65 MB file and assert rejection.
7. Render a page, assert CSP header present with nonce matching the inline script tag; post a violation report and assert 202 plus storage; assert report-only header by default and enforcing header after `enforce()`.
8. Request `/nl/` and assert translated strings; remove one key and assert fallback chain plus single log line; load a PO with plurals and assert all plural forms render.
9. Run `molo ai new` against a local stub SSE server returning two chunks plus `[DONE]`; assert transcript assembly, redaction of a planted `API_KEY` value, and that no file is written without confirmation; point at the real DevPlace endpoint only in a manual opt-in test.
10. Enqueue/retry/cancel a fixture task via CLI with `--format json`; assert state transitions match `07-supervisor-replication.md` semantics and output parses.

## References (sibling files this depends on or feeds)

- Depends on 01-runtime.md (static serving of bundles, lifecycle), 02-observability.md (registry, logging, OTLP), 03-security.md (nonces, redaction-adjacent crypto, TLS posture), 04-data.md (migrate targets, readiness checks), 05-request-routing.md (template hooks for i18n/CSP, middleware mount points), 06-realtime.md (SSE parser reuse), 07-supervisor-replication.md (CLI surface targets), 08-clients-openapi.md (HTTP client for AI harness).
- Feeds 10-workflow-example.md (CLI-verified commands, example pages proving i18n/CSP/health, `molo new` template source).

## Open questions

1. Should `molo` be one static binary or a thin driver plus per-project plugin commands? Leaning single binary for v1.
2. Do we ship shell completion scripts (bash/zsh/fish) generated from the command tree in v1, or defer to v1.1?
3. Should `/metrics` require auth in production by default ( risk: scraping complexity) or stay open with IP allow-listing?
4. Which locales ship compiled-in for v1 beyond `en`: `nl` only, or a wider starter set?
5. Does the AI harness need a non-streaming fallback for proxies that buffer SSE, or is streaming mandatory?

## Sources (exact URLs with retrieval dates)

- Command Line Interface Guidelines, https://clig.dev/ (retrieved 2026-10-23): exit codes, stdout/stderr split, help conventions, machine-readable output.
- Prometheus exposition formats, https://prometheus.io/docs/instrumenting/exposition_formats/ (retrieved 2026-10-23): text 0.0.4 line format, HELP/TYPE, content type, primitives.
- Kubernetes probes task guide, https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ (retrieved 2026-10-23): liveness restart semantics, probe timing fields.
- MDN Content-Security-Policy, https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy (retrieved 2026-10-23): default-src fallback, report-only header, reporting directives.
- MDN Using server-sent events, https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events (retrieved 2026-10-23): text/event-stream, data blocks, blank-line framing.
- Intel OpenVINO OpenAI chat API docs, https://docs.openvino.ai/2024/ovms_docs_rest_api_chat.html (found 2026-10-23 via search snippet): stream flag, data-only SSE, `data: [DONE]` terminator. Note: OpenAI first-party streaming reference (https://platform.openai.com/docs/api-reference/chat-streaming) returned HTTP 403 on 2026-10-23; convention triangulated with MDN SSE and multiple compatible-server docs.
- GNU gettext manual, https://www.gnu.org/software/gettext/manual/gettext.html (retrieved 2026-10-23): msgid practice, PO format, plural and context entries, locale selection.