ad/DEVELOPMENT_FLOW.md

Development flow lessons

Purpose

This file is the shared handover of teachings and knowledge that may feed the DPP project template. Agents record generic, project-agnostic lessons here so no good preference or idea is lost. Grok alone folds accepted lessons into the normative template for future projects.

Normative fold-in: lessons below are reflected in /workspace/dpp/dptemplate.md where Grok has folded them. Where a lesson could not override an existing template MUST or MUST NOT, the template wins and the residual is recorded in /workspace/dpp/CONFLICTS.md. This file remains the lesson source; the template remains the conformance authority.

Who may edit what

Surface	Who	Rule
/workspace/dpp/dptemplate.md	Grok Bot only (grok-bot / AbsurdDevelopmentProject)	Only Grok updates the frozen template, and only on explicit user order.
/workspace/dpp/CONFLICTS.md	Grok Bot only	Written when a lesson conflicts with an existing template MUST.
DEVELOPMENT_FLOW.md (this file)	Any agent on the project	Append new lessons. Do not delete existing lesson content. Reformat only when the user orders a format change.
Project code and docs	Agents per retoor's instructions	Continue the project from available knowledge plus user direction.

Tone for every entry: generic, consistent, dry. No product names in the lesson body when the rule is meant for the template. Name a product only when stating a concrete observation that cannot be generalized yet.

Lesson format

Every lesson is one subsection under Lessons. Fields are fixed. Body text is prose plus bullets. Status values:

• proposed: recorded, not yet folded into the template
• folded: reflected in dptemplate.md
• conflict: folded where possible; residual conflict kept in CONFLICTS.md (template rule wins)

Skeleton

### L-YYYY-MM-DD-NN: Short title

| Field  | Value |
|--------|-------|
| date   | YYYY-MM-DD |
| agent  | grok-bot \| muse-spark \| <other-id> |
| status | proposed \| folded \| conflict |

Body starts here. Project-agnostic. Dry. Complete enough that Grok can
turn it into MUST / MUST NOT text without asking again.

Example entry

### L-2026-10-05-00: Example (format only)

| Field  | Value |
|--------|-------|
| date   | 2026-10-05 |
| agent  | grok-bot |
| status | proposed |

Applications MUST refuse to start when the configured data directory
resolves outside `./data`. Prove it with a maintained subprocess test
against an inside path (allowed) and an outside path (refused).

Append new lessons at the end of Lessons. Never renumber or delete prior entries. If a later lesson revises an earlier one, add a new entry that cites the earlier id.

Lessons

L-2026-10-05-01: Storage law

Field	Value
date	2026-10-05
agent	muse-spark
status	conflict

All application data lives under ./data/<env> inside the project. No exceptions, no second locations.

• development for local runs, staging and production as container file mounts, test for persistent test services.
• Ephemeral runs use dotted subdirectories that clean themselves: ./data/.tmp-pytest-<pid>, ./data/.tmp-locust, ./data/.tmp-shots-<pid>.
• Each env dir holds the database file plus one subdirectory per kind (uploads, backups, keys, locks, seo_reports, ...). Database and files are separated by directory, never by system location.
• Enforcement beats convention: the application refuses to start when its data dir resolves outside ./data or its sqlite file outside the data dir. Prove it with a maintained test that runs the real import in a subprocess against inside and outside paths.
• Named docker volumes are forbidden. Every mount is a file mount. Containers run as uid/gid 1000 so files stay human-owned; bind host directories must pre-exist with correct ownership because docker creates missing ones as root.

Source commits (approx.): 44d0c08, 4268b1e, 60dd597. Residual conflict with template bare-metal shared-DB and "data volume" wording: see CONFLICTS.md items 1 and 4.

L-2026-10-05-02: Environments

Field	Value
date	2026-10-05
agent	muse-spark
status	conflict

Four environments, one shape each, fixed localhost ports (example offsets: staging 19847, production 19848, test 19849).

• Production owns the holy database. Nothing writes to it except production itself.
• Staging is refreshed from production snapshots only: online sqlite backup API while prod keeps running, integrity check, stop, swap, start, verify health. Never the reverse direction.
• Test is hermetic: persistent service data for the public test site, wiped or temp dirs for gallery and suite runs.
• Snapshots flow one way: production to staging. Test never touches production data.

Source commits (approx.): 4268b1e, 60dd597. Residual conflict with template section 28.1 (make dev / make prod share the production database): see CONFLICTS.md item 1.

L-2026-10-05-03: Docker law

Field	Value
date	2026-10-05
agent	muse-spark
status	conflict

One compose file per environment, each with its own top-level name: so projects can never steal each other's containers.

• Dev and test run the dev server (uvicorn --reload) with the package, the manifest (pyproject.toml), and ./data/<env> mounted: every saved change is live within seconds.
• Production runs workers, no reload, no source mount except the data dir.
• Production secrets use dedicated PROD_-prefixed variables with :? guards, so the dev .env can never silently satisfy them. Missing secrets fail loudly at startup, never with defaults.

Source commits (approx.): 4268b1e, 60dd597. Residual conflict with template zero-config boot and SECRET_KEY hardcoded fallback: see CONFLICTS.md item 2.

L-2026-10-05-04: Environment files law

Field	Value
date	2026-10-05
agent	muse-spark
status	conflict

.env.example lists every variable the code reads, no more, no less. A maintained contract test asserts the exact key set plus non-empty values for every path-like key, and fails on the old file before passing on the new one.

• Never leave a key empty when empty overrides a good default (paths, binaries, URLs): use safe literals (./data/development, rclone).
• The private .env is generated from the example with fresh random secrets, mode 600, never committed. The admin password applies at first bootstrap only and is reported to the owner once.

Source commit (approx.): 60dd597. Tied to CONFLICTS.md item 2 for secret defaults versus loud failure.

L-2026-10-05-05: Serving law (molohttp)

Field	Value
date	2026-10-05
agent	muse-spark
status	folded

One front door on 80/443, all local reverse-proxy upstreams.

• Back up before every edit, one format only: <file>.bak-<reason>-<YYYYMMDDHHMMSS>, root:root 640, reason slug required, never deleted.
• Assert unique site ids AND unique hostnames before appending: on reload molohttp keeps the first site per hostname and silently drops later duplicates. Re-read the file after reload and assert the new site survived.
• Change order: backup, edit, restore molohttp:molohttp 640 ownership, validate JSON, reload, verify the site over HTTPS. Rollback is the same steps with the previous values; the old backend stays running until rollback is no longer wanted.
• The admin surface is never exposed publicly. Local CLI only.

Source commits (approx.): 60dd597 (operator notes), production cutover work. Folded into template section 28.4 in product-agnostic form (front-door reverse proxy); the molohttp-specific ownership and JSON validate steps remain the lesson source here.

L-2026-10-05-06: Gallery law

Field	Value
date	2026-10-05
agent	muse-spark
status	folded

Every environment gets a screenshot gallery, generated the same way every time.

• Boot mode builds a throwaway server on a fresh data dir and seeds one fixed record; base-url mode shoots a live environment and never writes seed data.
• Full-page desktop shots plus a responsive matrix (down to 320px) with a horizontal-overflow check per cell; overflows are reported and exit nonzero.
• One gallery per environment plus a hub page that rebuilds from whichever environments exist. Same script, same selectors, same waits for all.

Source commits (approx.): 4268b1e, 60dd597. Folded into template section 28.5.

L-2026-10-05-07: Test law

Field	Value
date	2026-10-05
agent	muse-spark
status	conflict

Mirror layout (tests/unit, tests/api, tests/e2e), full suite green before done.

• Browser tests wait up to 60 seconds per step and retry a stalled page load once through one shared helper. Filter only proven environmental console noise, never application errors.
• Static gates run on every change: imports, linter, syntax, schema lint, UI invariants. Text gates (for example the em-dash ban) apply to docs too.
• Every behavior change ships the smallest maintained test exercising it, proven red before the fix and green after. Never weaken a failing test to get green.

Source commits (approx.): 94ae750, 60dd597. Residual conflict with template 15s page-fixture default timeout: see CONFLICTS.md item 3. Shared retry helper and related gates were folded.

L-2026-10-05-08: Git law

Field	Value
date	2026-10-05
agent	muse-spark
status	folded

Stage named files only, never broad-add a dirty tree. Commit and push only on explicit ask, never as a side effect of finishing.

• .gitignore contains exactly: .venv/, __pycache__/, *.pyc, .pytest_cache/, .coverage*, data/, out/, .env, *.egg-info/, build/, dist/, temp_*, *.swp, *~, .DS_Store. Never ignore source, tests, docs, compose files, Dockerfiles, .env.example, Makefiles, or CI configs.
• Any scratch, probe, or temporary file must be prefixed temp_ and is never committed. temp_* in .gitignore enforces it.
• Markdown files are never committed without explicit user confirmation. Docs are reviewed prose, not byproducts.
• A pre-commit hook owns the patch version bump; nothing else edits the version.

Source commit (approx.): 60dd597. Folded into template sections 2 and 36.

L-2026-10-05-09: Code law

Field	Value
date	2026-10-05
agent	muse-spark
status	folded

Explicit over implicit, flat over nested, sparse over dense, readable above all. No comments anywhere except the author line retoor <retoor@molodetz.nl> at the top of every file; code that needs a comment is rewritten until it does not. Defensive everywhere, errors never pass silently, no invented names with suffixes. JavaScript is ES6 modules, one class per file, one Application instantiated as app on window. Storage is dataset plus SQLite. The frontend is served through the backend. No emojis, no hype language, professional and scientific.

Source commit (approx.): 60dd597. Already largely present in the template; kept here as the lesson source.

L-2026-10-05-10: Docs law

Field	Value
date	2026-10-05
agent	muse-spark
status	folded

README.md is the living entry point (running, testing, docker, screenshots, data, operations) and stays true to the repo. Each deployment gets an operator guide with access facts, file owners, backup specification, and toggle procedures. Document only inspected facts, mark dated observations, and record anomalies instead of hiding them.

Source commits (approx.): 4268b1e, 60dd597. Folded into template section 32.

L-2026-10-05-11: Handover law

Field	Value
date	2026-10-05
agent	muse-spark
status	folded

Secrets are generated with a cryptographic RNG, stored only in their target (.env 600, container env), and reported to the owner once in chat, never written to the repo, docs, or logs beyond that handoff.

Source commit (approx.): 60dd597. Folded into template section 37.

L-2026-10-05-12: Multi-agent lesson format for DPP feed

Field	Value
date	2026-10-05
agent	grok-bot
status	folded

This file is reformatted as the shared multi-agent lesson log. Grok Bot (AbsurdDevelopmentProject) alone updates /workspace/dpp/dptemplate.md. Other agents append generic lessons here. Existing storage, environments, docker, env-files, serving, gallery, test, git, code, docs, and handover laws are preserved as dated muse-spark entries above. New teachings use the skeleton under Lesson format. No lesson content may be deleted when the format evolves.