Files
moloapp/specs/example-site.md
retoor 0595284fe3 feat(doos): add container runtime, audit cases, example demo and tests
- New rootless doos container runtime: sandbox, cgroup, network, images, hardening, davmount, CLI plus molo integration
- 18 CVE audit reproductions with READMEs and Makefiles
- Example DoosDemo routes, IndexCatalog, static assets, spec update
- CLI serve on 0.0.0.0:8000, arg parser, dev server improvements
- Runtime HTTP fixes, supervisor isolation backend
- Tests for doos, example doos routes, CLI serve; suite count 261 to 265
2026-11-06 04:36:00 +01:00

247 lines
25 KiB
Markdown

<!-- doc: specs/example-site.md | 0.1.0-draft | 2026-10-29 -->
# Example-Site Specification Draft
Status: draft for review. Source: synthesis of workflow evidence P1-P6 (structure, tests, OpenAPI drift, reference-site reuse, error/UX audit, falsification gaps). No live probes were run; counts and bodies cited below are source/grep-level and need the falsification reruns in Section 11 before this becomes normative.
Conventions: MUST/SHOULD/MAY per RFC 2119. This spec wins over code on conflict until amended in the same change.
## 1. Scope, goals, non-goals
Goals:
- One example site that demonstrates every framework module through a consistent shell, nav, page, form, error, and JS contract.
- Commercial and scientific tone: direct, factual, user friendly; no complexity pushed to the visitor.
- Responsive at 360px viewport minimum; narrow screens get dropdowns/collapse, never hidden nav with no fallback.
- Fully specified: every shared component, envelope, tag, and automation gate is declared here.
Non-goals:
- Production auth hardening and threat modeling (out of scope for now).
- Live vendor calls in default suite (paid/live routes stay dry-run or skipped without keys/network).
- Rewriting framework modules; the example consumes them via the documented demo mounts.
## 2. Quality standards, coverage targets, caps, floors
### 2.1 Test standard
- Example suite is integration-only; isolated unit tests are forbidden. Arrange/Act/Assert with Fixture plus TestClient, PASS:/FAIL: protocol (tests/README.md).
- Target: `example suite: 265 passed, 0 failed` (AGENTS.md). Current inventory: 265 files under tests/example, 285 under tests/src. Any count change MUST update this spec plus AGENTS.md in the same change.
- Per-route minimum (4 cases): happy path, auth-fail or 404/405, validation/422, limit/edge. Applies to every mounted route including /readyz, /startupz (currently misplaced in probes_test.cpp; MUST move to tests/example/readyz and tests/example/startupz).
- Matrix test: GET every index href, require 2xx/3xx (401 allowed only for /auth/me), minimum row count declared (today >=37; index catalog carries ~40 features in 16 groups).
- Loud skips: unverifiable input is rejected with reason plus counter, never silently stored.
### 2.2 Build and lint gates
- Warning-free build with -std=c++23 -Wall -Wextra everywhere. -Werror MUST be universal; today it is missing in example, app, data, cli, observability, realtime, tenancy, audit, governance, supplychain .mk files. Closing that gap is required.
- `make lint` = warning-free build plus repo-wide em-dash grep. No tidy/sanitizer/format/coverage gate exists today; until one lands, the lint target is the gate and MUST stay green.
- Pipeline order: lint -> suite -> bump -> changelog -> commit -> tag -> deploy -> verify. Timeout 30 min, then escalate.
### 2.3 Coverage targets
- Route coverage: 100 percent of mounted router registrations reachable from index matrix or docs table, or explicitly listed as excluded (paid/live prefixes /voice, /google, /agent, /ai excluded from locust by design).
- OpenAPI coverage: 100 percent of mounted routes appear in served spec (today 3 manual adds vs ~490 registrations; see Section 9).
- Component coverage: every named component in Section 5 has at least one route-level test rendering it.
- JS coverage: every shipped asset JS file has a route test that loads its page plus a simulated POST exercising its err/out path.
### 2.4 Caps (static ceilings, no autotuning)
- Suite: 10 min total, 30 s single test, zero flakes (tests/README.md).
- HTTP demo payloads: 64 KB request body, 256 KB response body, 100 items per list page default.
- Rate: 60 req/min per demo write endpoint when loopback deployed; 429 with Retry-After on breach.
- Embedded bundle: 5 MB total under example/assets; single asset 512 KB max.
- Gate output panels: 200 lines max with truncation notice; stats tiles: 12 max.
- Locust headless run: 60 s, 10 users default.
### 2.5 Floors (never crossed alone; dry-run, stop, page human)
- Protected: specs/, CHANGELOG.md, VERSION, git tags, retention windows, destructive file deletes, history rewrite, force push, hard reset.
- Docs/off behavior: doc routes disabled by default in prod (prod 404 test required).
- Destructive/paid demo calls: dry-run by default, explicit confirm to execute; no auto-spend.
- Release: max 1 release per green run; failed live verification blocks tag.
## 3. Whole-site structure
### 3.1 Information architecture
- Single shell (base.html) with sticky masthead, main, footer. One shell template only; merge page.html, ent_page.html, platform_page.html into one.
- Global topbar MUST cover all 16 index groups (today only 7 links: Index/Templates/Forms/Stats/Voice/Docs/Health). Groups: routing, requests, data, security, static, realtime, observability, operations, doos, tenancy, audit, governance, google(+voice), agent, vendors(/ai), tools, plus Other fallback. Narrow screens use the existing details-dropdown pattern (keep; it beats the reference-site nav-hide).
- Index page: grouped matrix via IndexCatalog render, filter input (no-JS safe), needs-API-key badges on google/agent/vendors groups.
- Every detail page: section-head (numeral + lede), breadcrumbs (Index / Group / Page), back-link, section nav where page has 2+ sections.
- Docs page: filterable register table (see Section 5), not a plain list.
- Static vs embedded: example/static holds shell CSS/JS (app.css, app.js, index.js); example/assets holds per-demo JS bundled to build/example/gen/assets.cpp and served under /embedded/{name}.
### 3.2 Route map (mount contract)
- Core: / (index), /health, /readyz, /startupz, /metrics, /telemetry, /routing*, /orm/users, /kv/items, /cache/lookup, /pubsub/publish, /auth/*, /sessions/*, /security/*, /security-extra/*, /formats/data, /templates/page, /forms/contact, /validate/user, /ws, /socketio, /sse, /streaming, /webdav, /expose, /supervisor/*, /restclient, /clients/*, /operability/*, /cli/*, /workflow/release, /supplychain/sbom.json, /openapi.json, /docs, /stats, /stats/data, /voice, /voice/say|hear|turn|reset, /google/status, /doos, /doos/status, /doos/create, /doos/delete, /tenancy/*, /audit/*, /governance/*, /v1|v2/items.
- Agent (9, /agent/<m>): models, runtime, sessions, sandbox, registry, armor, optimize, adk, live.
- AI (20, /ai/<v>): xai, xaiplatform, anthropic, anthropicadmin, meta, deepseek, openai, openaiplatform, openaiadmin, openrouter, compat, mistral, cohere, perplexity, bedrock, azure, huggingface, audio, media, retrieval.
- Google (7, /google/<pkg>): discovery, cloudai, enterprise, data, maps, workspace, ops.
- Contract for agent/ai/google: GET page plus status returns 2xx always; POST/DELETE without key returns 503 with machine-readable not_configured/google_not_configured; validation failures return 422; paid/destructive runs dry-run unless confirm:true.
- Doos: GET page, GET status JSON, POST create, POST delete with confirm=yes (else dry-run). Demo store: DOOS_HOME else build/doos-demo.
### 3.3 Build, deploy, verify-live
- Standalone: make -f example/example.mk example|example-test|example-clean|example-locust. Whole repo: make, make test, make lint via root Makefile aggregation.
- Deploy: loopback living-server on EXAMPLE_PORT (default 8000), pid build/example/run/server.pid, health-gated; verify via tools/verify_deploy.py.
- Verify-live gate (manual today, MUST stay manual until wired): health probe, /metrics scrape, index 360px markup checks (viewport meta, .nav-narrow collapse), crawl first 10 matrix links. Any red step halts the pipeline.
## 4. Per-module look (page contract)
Every demo page MUST follow this skeleton in order:
1. Breadcrumbs plus section-head (numeral, title, one-line lede, status pill if the module has state).
2. Config/status strip: molo-config-status element (configured/not-configured/live/dry-run) for key-gated modules.
3. How-it-works block: steps or tabs plus callout for paid/dry-run notes.
4. Interactive form(s): shared FormView builder only (Section 7); one primary submit; confirm control per Section 7 for paid/destructive.
5. Output area: molo-json-out (codeblock in panel-dark) plus molo-alert region; run-button triggers fetch; output capped at 200 lines.
6. Reference row: link to /docs entry plus OpenAPI operation id.
Look rules:
- One section class vocabulary from Section 5; no per-vendor invented section classes or id prefixes.
- No page-local <style> blocks and no inline style attributes; all styling lives in app.css tokens/components.
- Global .primary button token works on every page (today dead except voice page; MUST fix in app.css).
- Vendor pages share one JS module plus per-page config; no more ~30 copy-paste fetch/err/out files (see Section 8).
- Delete or generate example/assets/app.css stub from the same source as example/static/app.css; two diverging stylesheets are forbidden.
- CSP page MUST use the shared shell (nav + stylesheet), not a standalone document.
## 5. Reference-site component minimum map
Reference: seriousdevelopment single-page brief (static.molodetz.nl/seriousdevelopment). Minimum portable set, mapped to example targets:
- REUSE-01 tokens + type trio: paper/wash/ink/hairline/oxblood/navy/panel vars; serif display, system sans body 17px/1.65, mono labels. Target: :root in example/static/app.css.
- REUSE-02 dark masthead/footer bars with 3px accent rule. Target: kBase header/footer in Templates.cpp. Keep details-dropdown narrow nav.
- REUSE-03 section-head numeral + lede. Target: index group heads and demo page heads.
- REUSE-04 register table (mono ID / rule / check). Target: /docs route table and index group tables.
- REUSE-05 status pills + stamp badges. Target: /health, /doos/status, supervisor/worker states, config strips.
- REUSE-06 tabs + steps + callout. Target: how-it-works/mode blocks. ARIA tabs with arrow-key step, portable as IIFE.
- REUSE-07 panel-dark + tree/codeblock + gate-output + run-button. Target: scaffold/API/code display and run-a-gate demos (/docs, /formats, /workflow/release). Timed line reveal with reduced-motion instant fallback is the model for live demo output.
- REUSE-08 hero-stats strip. Target: stats tiles on index hero and /stats.
- REUSE-09 data-component one-class-per-file JS with Application register/start. Target: organization of example/assets plus static JS; no-JS-safe defaults required.
- REUSE-10 single-column collapse + overflow-x:auto code panels at <=760px. Target: app.css responsive rules checklist.
Named class vocabulary to adopt (minimum): masthead, nav, hero, hero-stats, figure-frame, titleblock, section, section-head, section-numeral, finding, stamp, pillar, tabs, tablist, tabpanel, steps, callout, terminal-head, run-button, gate-output, track-controls, status + modifiers, stack-row, register, panel-dark, tree, codeblock, split, footer, footer-grid, footer-base.
Explicit non-reuse: reference nav-hide at <=760px (display:none, no fallback) is rejected; example keeps its dropdown.
## 6. Visual bar: rich and fancy standard
Floor (must pass):
- 360px viewport: no horizontal scroll; single-column collapse per REUSE-10; code panels scroll internally.
- Serif display headline with clamp(), mono kickers/IDs, hairline rules, squared corners, 3px accent bars.
- Dark masthead/hero-figure/footer-panels for contrast rhythm; light paper body.
Rich bar (expected on index, /docs, /stats, /voice, and one page per agent/ai/google family):
- Hero with kicker, headline, standfirst, dual buttons, 4-stat strip, framed figure with mono caption.
- Status pills that reflect live state (not static text); stamp badges for HALT/DRIFT/dry-run states.
- Tabs or steps with keyboard support; callouts for paid/key-gated notes.
- Panel-dark code/output blocks with run-button and reveal animation (instant when prefers-reduced-motion).
- Stats tiles fed by live /stats/data with empty-state and error-state designs (no bare hint spans).
Proof: every visual-bar claim needs a route-level test asserting the markup plus a 360px markup check in verify_deploy.py. Rendered browser screenshots are recommended but the markup gate is the normative one until a screenshot tool is granted.
## 7. Error, message, dialog, form standards
### 7.1 Error envelope (one standard)
- Canonical JSON: {error, message} for all API errors (SPEC.md contract). Validation: {error: validation_error, failures: [{field, message}]} with 422.
- Retire or map: bare {error: string}, {errors: [...]}, 3-key {error, field, message} variant, and all plain-text error branches (405 text, static-file text, HTTP-layer text, platform text). HTML clients get error.html rendered with the same fields; error.html MUST actually be rendered (today registered but never used).
- problem+json {type, title, status, detail, instance}: everywhere or nowhere. Today enterprise-only; decide in implementation plan and apply uniformly.
- Status codes: bad credentials uniformly 401 (today 403 in SecurityDemo vs 401 in SecurityExtraDemo; SPEC pins 403 and MUST be amended to 401 in the same change if 401 is chosen). Missing key: 503 with not_configured code. Unknown route: 404 JSON for API, error.html for browsers. Wrong method: 405 JSON with Allow header.
- Content negotiation: one wants_html() helper everywhere; retire the Accept-only fork in /validate/user.
### 7.2 Messages and notices
- One pattern: notice/error regions with role=status/alert plus aria-live on every page including /stats. Retire the three divergent success patterns (done-flag hide-form, lab-page notice paragraph, PRG 303 flash) in favor of: inline molo-alert region update for fetch flows; PRG flash only for full-page form posts, rendered through the same molo-alert component.
- Async JS results use one toast component (stacked, dismissible, role=status/alert) replacing per-file err/out drift.
### 7.3 Dialogs and confirms
- Zero native dialogs today; standard is <dialog>-based molo-confirm: two-step destructive flow (arm + confirm), focus trap, Esc to cancel, return focus to invoker.
- One confirm vocabulary: checkbox label Confirm (run paid or destructive calls; otherwise dry-run) plus dialog copy naming the spend/destruction. Retire the five divergent labels (spend money / really send non-read calls / dry run if paid / confirm=yes variants).
- Dry-run default holds: unconfirmed paid/destructive POST returns the dry-run plan with 200, never 4xx, so clients can preview.
### 7.4 Forms
- One builder: FormView plus form_page.html partial, novalidate, summary box plus inline per-field errors. Retire form_html() (no error display) and lab toolbar inline-style forms.
- Invalid input re-renders the same template (no lab_page to form_page layout jump).
- Binding: shared Form binder (query merged) everywhere; retire hand-parsing in /validate/user.
- Every form has error summary region, per-field messages, and preserved values on failure; pinned by contact_test pattern extended to all forms.
### 7.5 JS error handling
- Shared client module reads message with fallback chain: j.message else j.error (string) else j.detail else j.title else HTTP status. No more j.message-only degradation to HTTP <status>.
- No generic Request failed discards; no errors written into output <pre> without an alert region (ADK page MUST gain one; today it is the only aria-live in the repo and still misplaces errors).
- Uniform output: JSON codeblock plus optional text preamble; xAI text+JSON vs siblings JSON-only divergence is resolved in favor of the shared renderer with a text slot.
## 8. ECMAScript custom-element tag strategy
Strategy: molo-* autonomous custom elements defined in one shared module (molo.js), used directly in server-rendered templates. Progressive enhancement: every tag has useful server-rendered fallback content; JS upgrades behavior.
Required tags (minimum set):
- molo-topnav: global nav with wide links plus narrow dropdown; data-groups JSON; current-group highlight.
- molo-breadcrumbs: Index / Group / Page trail from data attributes.
- molo-alert: notice/error region; attrs type (ok/error/hint), role auto (status/alert); method show(text)/clear().
- molo-field-errors: summary plus inline error binding; attrs errors JSON.
- molo-confirm: <dialog>-based confirm; attrs action-label, dry-run-note; events molo:confirmed/molo:cancelled.
- molo-config-status: key/config pill; attrs state (configured/not-configured/live/dry-run), service label.
- molo-json-out: codeblock output with line cap, copy button, truncation notice; attrs max-lines default 200.
- molo-toast-stack: toast host; method push({type, text}).
- molo-tabs: ARIA tabs wrapper (arrow-key nav) for how-it-works blocks.
- molo-gate-output: reveal-into-output panel honoring prefers-reduced-motion.
Rules:
- Tag names kebab-case with molo- prefix; one class per tag in molo.js (or one file per tag under static/components/ if molo.js exceeds 500 lines).
- Templates use the tags as plain HTML; no framework-specific template syntax beyond existing ctx substitution.
- No customElements polyfill required for evergreen; fallback content MUST remain readable with JS off.
- Migration: replace ~30 copy-paste asset files with molo.js plus small per-page config scripts (data attributes preferred over new files).
## 9. openapi.json auto-sync
Problem (DRIFT-1..8): no committed artifact; served spec is a 3-operation paths-only template at 3.0.0 while the real builder mandates 3.1.0; tests assert both versions against different generators; ~490 routes unrepresented; DELETE/methods/schemas/status/security absent; /openapi.yaml and /redoc declared but never mounted; governance/S1-S10 and version wiring dead.
Target contract:
- Served spec is OpenAPI 3.1.0, derived at startup from mounted routes, never hand-maintained.
- One generator: the real Builder (src/clients/openapi) fed by a Route to RouteDecl adapter. The example/fw/Net.cpp template emitter is retired or reduced to a thin delegate.
- Coverage: every router.get/post/del registration appears with path, method, summary, tags, parameters (query/header/cookie from route models), requestBody/response schemas where route declares models, and status codes observed from handlers (minimum: 200/201 success plus documented 400/401/404/422/503/500 where applicable).
- Routes: /openapi.json (JSON), /openapi.yaml (YAML), /docs (Swagger UI), /redoc (ReDoc). Real UI shells replace the static placeholder divs; the part-09 embedder packs them or they load from vendored assets.
- Versioning: info.version reads VERSION file at startup (today hardcoded in 3 places: Net.cpp, document.h ApiInfo default, caps.h kFrameworkVersion). No hardcoded doc versions.
- Prod default: doc routes return 404 in production unless explicitly enabled; covered by a prod-404 test.
Automation:
- Startup derivation: from_app-style walk over the mounted app router (build the missing adapter; plan name from_app does not exist yet).
- Determinism/snapshot test: canonical JSON snapshot of the derived spec checked into tests/example/openapi/; any route change without snapshot update fails with a diff.
- Lint: RouteLinter S1-S10 runs over real example routes in the suite (today unit-tested only, never executed over real routes). Example toy linter (path casing/verbs textarea) is retired or relabeled as a teaching demo that delegates to S1-S10.
- Governance: Builder::use_governance_declarations wired to route gov_* fields; dead-code grep becomes a passing integration test.
## 10. Whole-site auto-maintenance
Beyond OpenAPI, these artifacts MUST be derived or checked, never hand-synced:
- Index matrix: rendered from the same route registry that feeds OpenAPI; matrix_test cross-checks that every registry route is linked or explicitly excluded.
- Docs table: same registry; columns route/methods/group/summary/status.
- Locust coverage: check_locust_coverage.py stays a gate (static route-vs-locust check; fails on gaps, paid-route leaks, missing methods) and MUST be wired into `test` (today standalone only).
- Embedded bundle: gen/assets.cpp regenerated on asset change; bundle-list test fails on unlisted files.
- Nav groups: topnav data-groups generated from IndexCatalog groups; test asserts topbar link set equals catalog group set.
- Confirm/error vocabulary: grep-based consistency test over templates and demo .cpps asserting only the canonical labels/envelopes exist (with an allowlist file for intentional exceptions).
- Release: tools/release.py bump flow extended to regenerate the OpenAPI snapshot and fail on uncommitted spec drift before tagging.
## 11. Refactor automation plan
Order of work (each step: spec delta first if behavior changes, then code, then example route, then tests, then changelog):
1. Falsification reruns (no code): make -n all/test/lint; find/grep reruns for counts (~490 routes, 265/285 files, ~35 .primary hits, 40 assets); live GET /openapi.json plus POST/DELETE 503 matrix plus HTML-vs-JSON error matrix; run matrix_test, check_locust_coverage.py, verify_deploy.py, cve-audit; 360px render checks. Close every gap named in the critic evidence before normative edits.
2. Envelope unification: single renderer + wants_html everywhere; map legacy shapes; wire error.html; fix 403-vs-401 with SPEC amendment; add content-negotiated error matrix tests.
3. Shell/nav consolidation: merge 3 shell templates; expand topbar to 16 groups; add breadcrumbs; make /docs filterable; fix doos column misuse; bring CSP page into shell.
4. Token/component CSS: global .primary, dialog/toast/empty-state/skeleton/confirm styles; adopt REUSE-01..10 vocabulary; resolve assets/app.css stub; page-local styles deleted.
5. molo-* elements + molo.js: implement Section 8 tag set; migrate vendor pages in batches (agent 9, google 7, ai 20) with per-batch route tests; delete migrated copy-paste files in the same change (delete reclaims; no orphan assets).
6. OpenAPI auto-sync: Route to RouteDecl adapter; 3.1.0 builder wiring; yaml/redoc mounts; VERSION read; snapshot + determinism + prod-404 + S1-S10 tests; retire toy linter/manual adds.
7. Whole-site maintenance wiring: registry-driven index/docs/nav; locust gate into test; bundle-list, vocabulary-grep, release-drift checks.
8. -Werror universal + missing placements: add -Werror to the ~10 .mk files lacking it; move readyz/startupz tests; wire zero_cost_probe.cpp or delete it; remove stray empty tests/src/security subdirs; resolve doos SUP_LIB standalone skip.
9. Visual-bar pass: hero/stats/tabs/panels per Section 6 on the required pages; 360px gate green; per-component render tests.
10. Docs/version close-out: CHANGELOG entry without lossy ...and N more summaries; VERSION bump per Conventional-Commit rule (any ! or BREAKING CHANGE -> MAJOR; else feat -> MINOR; else fix -> PATCH; else no release); annotated tag only after green suite plus live verification.
Each step MUST keep make, make test, make lint green or stop and report; never force-push past red.
## 12. Spec storage path, versioning, docs discipline
- Logical path: specs/example-site.md is the normative home for this draft (specs/ is a protected floor; changes need the spec-delta-first workflow). Until specs/ landing is approved, this draft lives as the review artifact at /tmp/example-site-spec-draft.md and the working reference at example/SPEC.md stays descriptive of today only.
- Supporting docs: tests/README.md (suite contract), plans/08-clients-openapi.md (builder rules; rule 8 pins 3.1.0), audit/README.md (30-check CVE audit), docs/ for runbooks. Docs state what the system does today; gaps list the owning file next to the code.
- Versioning: VERSION plus annotated tag vMAJOR.MINOR.PATCH update atomically with the changelog entry. Demo: POST /workflow/release. Malformed commits (non-Conventional) are rejected before version analysis.
- Commit scope for this work: workflow(example-site) for spec, feat(example) for site changes, fix(example) for drift closures, test(example) for gate additions.
## Appendix A. Section list (for workflow evidence)
1 Scope-goals-non-goals; 2 Quality-standards-coverage-caps-floors; 3 Whole-site-structure; 4 Per-module-look; 5 Reference-component-minimum-map; 6 Visual-bar; 7 Error-message-dialog-form-standards; 8 Custom-element-tag-strategy; 9 OpenAPI-auto-sync; 10 Whole-site-auto-maintenance; 11 Refactor-automation-plan; 12 Spec-storage-versioning-docs.
## Appendix B. Open falsification gaps carried forward
- Live probes never executed (spec bodies, 503 matrix, error matrix, suite runs, 360px renders, make -n reruns).
- Full /ai|/google|/agent sub-route table beyond grep head/tail; Ops/Platform/Resilience/SecurityExtra/Formats/Realtime/Routing/Security .cpp bodies; 39 of 40 asset JS bodies; tools/release.py body beyond bump header; audit/ and plans/*.md bodies beyond cited excerpts.
- Full 4-case-per-route audit beyond 3 sampled test files; release.py tag-side gating below line 163.
- Vendor check_spec.py beyond OpenAI header; matrix_test OpenAPI cross-check; prod app-tree mount check beyond grep.
- Rendered 360px browser evidence; hero.svg artwork; IndexCatalog.cpp render body; /embedded/app.css reference trace.