Production cutover, public test, tmp-shots galleries, development flow

DEVELOPMENT_FLOW.md

@@ -0,0 +1,84 @@
+<!-- retoor <retoor@molodetz.nl> -->
+# Development flow
+
+The complete blueprint for generating and running projects. This is the evolution of the frozen DPP template Grok originally followed (not present on this machine): everything below was learned by shipping Molodetz to production and must be applied at once, from the first commit, on every future project of any type.
+
+## Storage law
+
+All application data lives under `./data/<env>` inside the project. No exceptions, no second locations.
+
+- `development` for local runs, `staging` and `production` as container file mounts, `test` for persistent test services.
+- Ephemeral runs use dotted subdirectories that clean themselves: `./data/.tmp-pytest-<pid>`, `./data/.tmp-locust`, `./data/.tmp-shots-<pid>`.
+- Each env dir holds the database file plus one subdirectory per kind (`uploads`, `backups`, `keys`, `locks`, `seo_reports`, ...). Database and files are separated by directory, never by system location.
+- Enforcement beats convention: the application refuses to start when its data dir resolves outside `./data` or its sqlite file outside the data dir. Prove it with a maintained test that runs the real import in a subprocess against inside and outside paths.
+- Named docker volumes are forbidden. Every mount is a file mount. Containers run as uid/gid 1000 so files stay human-owned; bind host directories must pre-exist with correct ownership because docker creates missing ones as root.
+
+## Environments
+
+Four environments, one shape each, fixed localhost ports (example offsets: staging 19847, production 19848, test 19849).
+
+- Production owns the holy database. Nothing writes to it except production itself.
+- Staging is refreshed from production snapshots only: online sqlite backup API while prod keeps running, integrity check, stop, swap, start, verify health. Never the reverse direction.
+- Test is hermetic: persistent service data for the public test site, wiped or temp dirs for gallery and suite runs.
+- Snapshots flow one way: production to staging. Test never touches production data.
+
+## Docker law
+
+One compose file per environment, each with its own top-level `name:` so projects can never steal each other's containers.
+
+- Dev and test run the dev server (`uvicorn --reload`) with the package, the manifest (`pyproject.toml`), and `./data/<env>` mounted: every saved change is live within seconds.
+- Production runs workers, no reload, no source mount except the data dir.
+- Production secrets use dedicated `PROD_`-prefixed variables with `:?` guards, so the dev `.env` can never silently satisfy them. Missing secrets fail loudly at startup, never with defaults.
+
+## Environment files law
+
+`.env.example` lists every variable the code reads, no more, no less. A maintained contract test asserts the exact key set plus non-empty values for every path-like key, and fails on the old file before passing on the new one.
+
+- Never leave a key empty when empty overrides a good default (paths, binaries, URLs): use safe literals (`./data/development`, `rclone`).
+- The private `.env` is generated from the example with fresh random secrets, mode 600, never committed. The admin password applies at first bootstrap only and is reported to the owner once.
+
+## Serving law (molohttp)
+
+One front door on 80/443, all local reverse-proxy upstreams.
+
+- Back up before every edit, one format only: `<file>.bak-<reason>-<YYYYMMDDHHMMSS>`, `root:root 640`, reason slug required, never deleted.
+- Assert unique site ids AND unique hostnames before appending: on reload molohttp keeps the first site per hostname and silently drops later duplicates. Re-read the file after reload and assert the new site survived.
+- Change order: backup, edit, restore `molohttp:molohttp 640` ownership, validate JSON, reload, verify the site over HTTPS. Rollback is the same steps with the previous values; the old backend stays running until rollback is no longer wanted.
+- The admin surface is never exposed publicly. Local CLI only.
+
+## Gallery law
+
+Every environment gets a screenshot gallery, generated the same way every time.
+
+- Boot mode builds a throwaway server on a fresh data dir and seeds one fixed record; base-url mode shoots a live environment and never writes seed data.
+- Full-page desktop shots plus a responsive matrix (down to 320px) with a horizontal-overflow check per cell; overflows are reported and exit nonzero.
+- One gallery per environment plus a hub page that rebuilds from whichever environments exist. Same script, same selectors, same waits for all.
+
+## Test law
+
+Mirror layout (`tests/unit`, `tests/api`, `tests/e2e`), full suite green before done.
+
+- Browser tests wait up to 60 seconds per step and retry a stalled page load once through one shared helper. Filter only proven environmental console noise, never application errors.
+- Static gates run on every change: imports, linter, syntax, schema lint, UI invariants. Text gates (for example the em-dash ban) apply to docs too.
+- Every behavior change ships the smallest maintained test exercising it, proven red before the fix and green after. Never weaken a failing test to get green.
+
+## Git law
+
+Stage named files only, never broad-add a dirty tree. Commit and push only on explicit ask, never as a side effect of finishing.
+
+- `.gitignore` contains exactly: `.venv/`, `__pycache__/`, `*.pyc`, `.pytest_cache/`, `.coverage*`, `data/`, `out/`, `.env`, `*.egg-info/`, `build/`, `dist/`, `temp_*`, `*.swp`, `*~`, `.DS_Store`. Never ignore source, tests, docs, compose files, Dockerfiles, `.env.example`, Makefiles, or CI configs.
+- Any scratch, probe, or temporary file must be prefixed `temp_` and is never committed. `temp_*` in `.gitignore` enforces it.
+- Markdown files are never committed without explicit user confirmation. Docs are reviewed prose, not byproducts.
+- A pre-commit hook owns the patch version bump; nothing else edits the version.
+
+## Code law
+
+Explicit over implicit, flat over nested, sparse over dense, readable above all. No comments anywhere except the author line `retoor <retoor@molodetz.nl>` at the top of every file; code that needs a comment is rewritten until it does not. Defensive everywhere, errors never pass silently, no invented names with suffixes. JavaScript is ES6 modules, one class per file, one `Application` instantiated as `app` on `window`. Storage is dataset plus SQLite. The frontend is served through the backend. No emojis, no hype language, professional and scientific.
+
+## Docs law
+
+`README.md` is the living entry point (running, testing, docker, screenshots, data, operations) and stays true to the repo. Each deployment gets an operator guide with access facts, file owners, backup specification, and toggle procedures. Document only inspected facts, mark dated observations, and record anomalies instead of hiding them.
+
+## Handover law
+
+Secrets are generated with a cryptographic RNG, stored only in their target (`.env` 600, container env), and reported to the owner once in chat, never written to the repo, docs, or logs beyond that handoff.