moloapp/plans/09-cli-ops.md

09 - CLI Tooling, Dev Server, Operations

Scope: plan.md sections 2, 7-8, 10 step 5 partial. Topics: CLI, dev server, embedder, AI harness, health/metrics, i18n/CSP.

Purpose and philosophy

One binary, molo, is the only project tool: it scaffolds, serves, embeds, supervises, and vibe-codes. Operators get health, readiness, and metrics endpoints that work with stock Prometheus and Kubernetes probes with zero glue code. Developers get a dev server with fast restart and a strict dev/prod settings split. Templates get i18n hooks and a CSP builder by default, so every generated page is localizable and locked down. Philosophy: human-first output that degrades gracefully to machine-readable output, static ceilings on everything watched or exported, and dev/prod parity except where explicitly split.

Best practices (from research)

• CLI design follows the Command Line Interface Guidelines: zero exit code on success and non-zero on failure, primary output to stdout, diagnostics to stderr, concise help by default with extensive help on request, and machine-readable output (JSON lines or --format json) where it does not hurt usability (clig.dev).
• Metrics exposition follows the Prometheus text format 0.0.4: line oriented, \n terminated with trailing newline on the last line, # HELP and # TYPE comment lines before samples, Content-Type: text/plain; version=0.0.4, optional gzip encoding, counter/gauge/histogram/summary/untyped primitives (Prometheus docs).
• Health semantics follow the Kubernetes probe model: liveness detects a broken state that only a restart fixes (kubelet kills and restarts the container on failure), readiness gates traffic, startup protects slow-starting containers with its own window (Kubernetes docs).
• CSP starts from default-src 'none' and opens only what is needed; default-src is the fallback for all other fetch directives; rollout begins in report-only mode (Content-Security-Policy-Report-Only) before enforcing (MDN CSP, field practice).
• Streaming uses Server-Sent Events: Content-Type: text/event-stream, Cache-Control: no-cache, notifications as text blocks terminated by blank lines, data: payload lines (MDN SSE). OpenAI-compatible chat streaming sends data-only SSE chunks terminated by a data: [DONE] message with stream: true in the request (Intel OpenVINO OpenAI-API docs; OpenAI first-party docs unreachable, 403, so this is single-vendor confirmed plus community triangulation).
• i18n follows GNU gettext practice: full sentences as message keys (never concatenated fragments), translator comments in source, PO-file based catalogs with context and plural forms, locale selected per request with a compiled-in fallback chain (GNU gettext manual).

Structure

Directories (deep, one concern per file):

tools/molo/                 main() entry, links framework as a library
include/molo/cli/           argument parsing, command tree, help, output
src/cli/
include/molo/dev/           dev server, file watcher, reloader
src/dev/
include/molo/embed/         asset embedder (scanner, emitter)
src/embed/
include/molo/ai/            AI harness client, prompts, session
src/ai/
include/molo/ops/           health, readiness, metrics export, probe routes
src/ops/
include/molo/i18n/          locale, catalog, translator, PO loader
src/i18n/
include/molo/csp/           policy model, builder, nonce manager
src/csp/

Namespaces and classes (one class per header/source pair):

• molo::cli: Command, CommandTree, Flag, FlagSet, ArgParser, HelpRenderer, TableWriter, JsonWriter, ExitCode (enum: Ok=0, Usage=2, Runtime=1, Unavailable=69).
• molo::dev: DevServer, FileWatcher (inotify/kqueue/ReadDirectoryChangesW backends behind one interface), Reloader, DevConfig vs ProdConfig split type.
• molo::embed: AssetScanner, EmbedEmitter (directory to .cpp/.hpp), EmbeddedBundle (runtime reader used by static routes in 01-runtime.md).
• molo::ai: AiClient (built on the 08-clients-openapi.md HTTP client), AiSession, PromptPack (framework feature prompts), SseStreamParser (shared with 06-realtime.md SSE client, not duplicated).
• molo::ops: HealthRegistry, HealthCheck (base), LivenessCheck, ReadinessCheck, StartupGate, PrometheusExporter, OtlpBridge (thin; transport owned by 02-observability.md).
• molo::i18n: Locale, Catalog, PoLoader, Translator, PluralRule.
• molo::csp: CspPolicy, CspBuilder, Nonce (per-request, from 03-security.md crypto).

Rules (hard, numbered)

1. The CLI never performs a destructive or history-rewriting action autonomously beyond its spec: molo new refuses to write into a non-empty directory without --force, and nothing here touches git history (history rules live in 10-workflow-example.md).
2. Every command returns 0 on success, 2 on usage errors, 1 on runtime failures, 69 when a required service is unavailable; scripts must be able to rely on this mapping.
3. Primary command output goes to stdout; logs, progress, and errors go to stderr; --format json output on stdout must be parseable JSON with no log interleaving.
4. --help on every command and subcommand; -h prints concise help, help <cmd> prints the full page with examples; no command without documented usage.
5. Dev server is dev-only: it refuses to start with production config unless --allow-prod is passed, and it never enables the reloader against a production database URL.
6. Health endpoints never require auth, never allocate unbounded memory, and must answer in under 50 ms p99 with all checks short-circuited on timeout (each check capped at 200 ms).
7. Metrics export has static cardinality caps: max 10000 series per process, max 64 label pairs per metric, per-route labels use route templates (/users/{id}), never raw paths.
8. Embedded builds are byte-identical reproducible: same input directory plus same tool version yields identical generated bytes (sorted file order, no timestamps in output).
9. The AI harness never sends secrets: .env.json values, key files, and anything matching *_KEY, *_SECRET, *_TOKEN are redacted before prompt construction; redaction failures abort the request.
10. Default CSP is deny-first: generated pages start at default-src 'none' plus explicit allows, and report-only mode is the default until the app opts into enforcement.
11. All user-visible CLI and template strings go through the translator; no hardcoded English outside the fallback catalog.

Development guidelines

• Build the CLI as data first: declare the command tree (names, flags, help) in one place, then attach handlers; help and shell completion generate from the same tree.
• Reuse, never reimplement: the AI harness uses the 08-clients-openapi.md HTTP client and the 06-realtime.md SSE parser; metrics use the 02-observability.md registry; nonces and hashing use 03-security.md; static serving of embedded bundles uses 01-runtime.md.
• Keep the dev server dumb: watch, rebuild via make, restart the child process, preserve no state; stateful niceties (state-preserving reload) are explicitly out of scope for v1.
• Prefer std::expected or an equivalent result type over exceptions across CLI/ops boundaries so exit codes map cleanly.
• Every new command ships with its spec entry, its help text, one integration test through the real binary, and a living-example mention where user-visible.

Specifications (concrete behaviors, caps, formats)

CLI command tree (v1):

molo new <name> [--backend postgres|sqlite] [--force]
molo serve [--port 8000] [--reload] [--config .env.json]
molo build [--embed assets/] [--out build/]
molo embed <dir> --out <file.cpp> [--namespace molo::assets] [--gzip]
molo migrate [--check | --apply]                 (executes 04-data.md plans)
molo supervisor start|stop|restart|status|logs   (drives 07-supervisor-replication.md)
molo task enqueue|retry|cancel|inspect           (drives 07-supervisor-replication.md)
molo ai new|continue [--stream] [--base-url URL] [--model NAME]
molo health [--format json|text]
molo version [--format json|text]

Global flags: -h/--help, --version, --format json|text (default text, TTY-aware: no color when piped), --verbose/-v (repeatable, max 3), --quiet/-q.

Exit codes: 0 ok, 1 runtime failure, 2 usage error, 69 service unavailable (matches SYSEXITS EX_UNAVAILABLE).

Dev server: molo serve --reload watches src/, templates/, assets/ (max 20000 files, debounce 150 ms, recursive); on change it runs make -j$(nproc) with a 300 s cap, kills the child with SIGTERM then SIGKILL after 5 s, and restarts; crash loops back off 1 s, 2 s, 4 s up to 30 s, then hold with a loud message. Dev/prod split: .env.json plus MOLO_ENV=dev|prod; prod requires TLS config and refuses --reload without --allow-prod.

Embedder: input max 512 MB total, max 50000 files, max 64 MB per file; skips dotfiles and symlinks escaping the root; emits one .cpp plus one .hpp with constexpr byte arrays (chunked at 1 MB per array), a sorted manifest (path, size, sha256, mime), and optional precompressed gzip variants for text types over 1 KB; output includes tool version string for audit.

AI harness: base URL default https://devplace.net/openai/v1, model default molodetz, key from DEVPLACE_API_KEY env only (never a flag, never stored); request POST {base}/chat/completions with {"model","messages","stream":true}; response parsed as SSE (text/event-stream), JSON per data: line, end on data: [DONE]; caps: 30 s connect timeout, 300 s session cap, max 8 MB transcript, max 50 tool/file actions per session, human confirms every file write (plan-then-apply, no silent writes).

Health/metrics endpoints (mounted by molo::ops, paths fixed):

• GET /healthz liveness: 200 {"status":"ok"} when the loop serves; fails only when shutdown is stuck.
• GET /readyz readiness: 200 when DB, KV, and supervisor heartbeats pass; 503 {"status":"degraded","failed":[...]} otherwise.
• GET /startupz startup gate: 503 until boot completes, then 200.
• GET /metrics Prometheus text 0.0.4 (text/plain; version=0.0.4), gzip if accepted; OpenTelemetry OTLP export optional via 02-observability.md bridge.
• Budgets: scrape render under 100 ms p99 at 10k series; export interval floor 5 s; series cap 10000 with molo_metrics_dropped_total counter on overflow.

i18n: locale format BCP 47 (en, nl-NL); resolution order: route prefix (/nl/...), cookie molo_locale, Accept-Language (q-values honored), config default; catalogs are PO files under locales/<locale>/LC_MESSAGES/*.po, compiled to embedded bundles for prod; plural rules per CLDR cardinal set for shipped locales; missing key falls back through nl-NL to nl to en to msgid, and logs once per key per process.

CSP builder: CspBuilder fluent API (default_src(), script_src(), style_src(), img_src(), connect_src(), frame_ancestors(), report_to()); default policy default-src 'none'; script-src 'self'; style-src 'self'; img-src 'self' data:; connect-src 'self'; frame-ancestors 'none'; per-request nonces for inline scripts ('nonce-<base64>', 16 bytes from 03-security.md RNG); report endpoint POST /csp-report (JSON, capped 64 KB per report, 100 reports/min/IP); Content-Security-Policy-Report-Only until enforce() is called.

Execution steps (in build order)

1. CLI skeleton: Command, ArgParser, HelpRenderer, exit-code mapping, --format, TTY detection, molo version and molo health (stub checks).
2. molo new scaffolding: Makefile, AGENTS.md stub, directory layout, .env.json, .gitignore, git init, plug-and-play example files.
3. Health/readiness/startup endpoints plus registry; wire DB/KV/supervisor checks as those components land.
4. Prometheus exporter on the 02-observability.md registry with series caps and gzip.
5. Dev server with watcher backends, rebuild/restart loop, dev/prod config split enforcement.
6. Embedder scanner plus emitter plus runtime bundle reader; static-route integration test.
7. CSP builder, nonce plumbing into templates (with 05-request-routing.md), report endpoint, report-only default.
8. i18n core: locale resolution, PO loader, translator, template filter/function hooks (with 05-request-routing.md).
9. AI harness: client on 08-clients-openapi.md HTTP, SSE parsing via 06-realtime.md, redaction, plan-then-apply loop, molo ai commands.
10. Supervisor/task CLI surface on top of 07-supervisor-replication.md APIs; structured JSON output for scripts.

Validation (how to verify each step)

1. Run molo version, molo --help, molo nosuch; assert exit codes 0/0/2 and JSON parseability of --format json; pipe stdout and confirm no color codes.
2. Scaffold into a temp dir; assert all files exist, git status clean after init, make succeeds, and re-run without --force fails with exit 1.
3. Start app, curl /healthz, /readyz, /startupz; kill the DB and assert /readyz flips to 503 within 2 s while /healthz stays 200; measure p99 under 50 ms over 1000 requests.
4. Curl /metrics, validate with promtool check metrics; assert trailing newline, HELP/TYPE lines, gzip variant identical after decode; flood labels and assert the 10000 cap plus dropped counter.
5. Touch a template, assert rebuild plus restart under 5 s; assert --reload with prod config exits 2 without --allow-prod; kill child 5 times, assert backoff holds.
6. Embed a fixture tree twice, assert byte-identical output; serve from the bundle and diff bodies plus ETags against disk serving; feed a 65 MB file and assert rejection.
7. Render a page, assert CSP header present with nonce matching the inline script tag; post a violation report and assert 202 plus storage; assert report-only header by default and enforcing header after enforce().
8. Request /nl/ and assert translated strings; remove one key and assert fallback chain plus single log line; load a PO with plurals and assert all plural forms render.
9. Run molo ai new against a local stub SSE server returning two chunks plus [DONE]; assert transcript assembly, redaction of a planted API_KEY value, and that no file is written without confirmation; point at the real DevPlace endpoint only in a manual opt-in test.
10. Enqueue/retry/cancel a fixture task via CLI with --format json; assert state transitions match 07-supervisor-replication.md semantics and output parses.

References (sibling files this depends on or feeds)

• Depends on 01-runtime.md (static serving of bundles, lifecycle), 02-observability.md (registry, logging, OTLP), 03-security.md (nonces, redaction-adjacent crypto, TLS posture), 04-data.md (migrate targets, readiness checks), 05-request-routing.md (template hooks for i18n/CSP, middleware mount points), 06-realtime.md (SSE parser reuse), 07-supervisor-replication.md (CLI surface targets), 08-clients-openapi.md (HTTP client for AI harness).
• Feeds 10-workflow-example.md (CLI-verified commands, example pages proving i18n/CSP/health, molo new template source).

Open questions

1. Should molo be one static binary or a thin driver plus per-project plugin commands? Leaning single binary for v1.
2. Do we ship shell completion scripts (bash/zsh/fish) generated from the command tree in v1, or defer to v1.1?
3. Should /metrics require auth in production by default ( risk: scraping complexity) or stay open with IP allow-listing?
4. Which locales ship compiled-in for v1 beyond en: nl only, or a wider starter set?
5. Does the AI harness need a non-streaming fallback for proxies that buffer SSE, or is streaming mandatory?

Sources (exact URLs with retrieval dates)

• Command Line Interface Guidelines, https://clig.dev/ (retrieved 2026-10-23): exit codes, stdout/stderr split, help conventions, machine-readable output.
• Prometheus exposition formats, https://prometheus.io/docs/instrumenting/exposition_formats/ (retrieved 2026-10-23): text 0.0.4 line format, HELP/TYPE, content type, primitives.
• Kubernetes probes task guide, https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ (retrieved 2026-10-23): liveness restart semantics, probe timing fields.
• MDN Content-Security-Policy, https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy (retrieved 2026-10-23): default-src fallback, report-only header, reporting directives.
• MDN Using server-sent events, https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events (retrieved 2026-10-23): text/event-stream, data blocks, blank-line framing.
• Intel OpenVINO OpenAI chat API docs, https://docs.openvino.ai/2024/ovms_docs_rest_api_chat.html (found 2026-10-23 via search snippet): stream flag, data-only SSE, data: [DONE] terminator. Note: OpenAI first-party streaming reference (https://platform.openai.com/docs/api-reference/chat-streaming) returned HTTP 403 on 2026-10-23; convention triangulated with MDN SSE and multiple compatible-server docs.
• GNU gettext manual, https://www.gnu.org/software/gettext/manual/gettext.html (retrieved 2026-10-23): msgid practice, PO format, plural and context entries, locale selection.