Reformat DEVELOPMENT_FLOW.md as multi-agent DPP lesson log
Preserve every storage, env, docker, env-file, serving, gallery, test, git, code, docs, and handover rule as dated muse-spark entries. Add the shared lesson format and a grok-bot entry for this rewrite.
This commit is contained in:
+315
-50
@@ -1,84 +1,349 @@
|
||||
<!-- retoor <retoor@molodetz.nl> -->
|
||||
# Development flow
|
||||
# Development flow lessons
|
||||
|
||||
The complete blueprint for generating and running projects. This is the evolution of the frozen DPP template Grok originally followed (not present on this machine): everything below was learned by shipping Molodetz to production and must be applied at once, from the first commit, on every future project of any type.
|
||||
## Purpose
|
||||
|
||||
## Storage law
|
||||
This file is the shared handover of teachings and knowledge that may feed
|
||||
the DPP project template. Agents record generic, project-agnostic lessons
|
||||
here so no good preference or idea is lost. Grok alone folds accepted
|
||||
lessons into the normative template for future projects.
|
||||
|
||||
All application data lives under `./data/<env>` inside the project. No exceptions, no second locations.
|
||||
Normative fold-in: lessons below are reflected in
|
||||
`/workspace/dpp/dptemplate.md` where Grok has folded them. Where a lesson
|
||||
could not override an existing template MUST or MUST NOT, the template
|
||||
wins and the residual is recorded in `/workspace/dpp/CONFLICTS.md`. This
|
||||
file remains the lesson source; the template remains the conformance
|
||||
authority.
|
||||
|
||||
- `development` for local runs, `staging` and `production` as container file mounts, `test` for persistent test services.
|
||||
- Ephemeral runs use dotted subdirectories that clean themselves: `./data/.tmp-pytest-<pid>`, `./data/.tmp-locust`, `./data/.tmp-shots-<pid>`.
|
||||
- Each env dir holds the database file plus one subdirectory per kind (`uploads`, `backups`, `keys`, `locks`, `seo_reports`, ...). Database and files are separated by directory, never by system location.
|
||||
- Enforcement beats convention: the application refuses to start when its data dir resolves outside `./data` or its sqlite file outside the data dir. Prove it with a maintained test that runs the real import in a subprocess against inside and outside paths.
|
||||
- Named docker volumes are forbidden. Every mount is a file mount. Containers run as uid/gid 1000 so files stay human-owned; bind host directories must pre-exist with correct ownership because docker creates missing ones as root.
|
||||
## Who may edit what
|
||||
|
||||
## Environments
|
||||
| Surface | Who | Rule |
|
||||
|---------|-----|------|
|
||||
| `/workspace/dpp/dptemplate.md` | Grok Bot only (`grok-bot` / AbsurdDevelopmentProject) | Only Grok updates the frozen template, and only on explicit user order. |
|
||||
| `/workspace/dpp/CONFLICTS.md` | Grok Bot only | Written when a lesson conflicts with an existing template MUST. |
|
||||
| `DEVELOPMENT_FLOW.md` (this file) | Any agent on the project | Append new lessons. Do not delete existing lesson content. Reformat only when the user orders a format change. |
|
||||
| Project code and docs | Agents per retoor's instructions | Continue the project from available knowledge plus user direction. |
|
||||
|
||||
Four environments, one shape each, fixed localhost ports (example offsets: staging 19847, production 19848, test 19849).
|
||||
Tone for every entry: generic, consistent, dry. No product names in the
|
||||
lesson body when the rule is meant for the template. Name a product only
|
||||
when stating a concrete observation that cannot be generalized yet.
|
||||
|
||||
- Production owns the holy database. Nothing writes to it except production itself.
|
||||
- Staging is refreshed from production snapshots only: online sqlite backup API while prod keeps running, integrity check, stop, swap, start, verify health. Never the reverse direction.
|
||||
- Test is hermetic: persistent service data for the public test site, wiped or temp dirs for gallery and suite runs.
|
||||
- Snapshots flow one way: production to staging. Test never touches production data.
|
||||
## Lesson format
|
||||
|
||||
## Docker law
|
||||
Every lesson is one subsection under [Lessons](#lessons). Fields are
|
||||
fixed. Body text is prose plus bullets. Status values:
|
||||
|
||||
One compose file per environment, each with its own top-level `name:` so projects can never steal each other's containers.
|
||||
- `proposed`: recorded, not yet folded into the template
|
||||
- `folded`: reflected in `dptemplate.md`
|
||||
- `conflict`: folded where possible; residual conflict kept in
|
||||
`CONFLICTS.md` (template rule wins)
|
||||
|
||||
- Dev and test run the dev server (`uvicorn --reload`) with the package, the manifest (`pyproject.toml`), and `./data/<env>` mounted: every saved change is live within seconds.
|
||||
- Production runs workers, no reload, no source mount except the data dir.
|
||||
- Production secrets use dedicated `PROD_`-prefixed variables with `:?` guards, so the dev `.env` can never silently satisfy them. Missing secrets fail loudly at startup, never with defaults.
|
||||
### Skeleton
|
||||
|
||||
## Environment files law
|
||||
```markdown
|
||||
### L-YYYY-MM-DD-NN: Short title
|
||||
|
||||
`.env.example` lists every variable the code reads, no more, no less. A maintained contract test asserts the exact key set plus non-empty values for every path-like key, and fails on the old file before passing on the new one.
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | YYYY-MM-DD |
|
||||
| agent | grok-bot \| muse-spark \| <other-id> |
|
||||
| status | proposed \| folded \| conflict |
|
||||
|
||||
- Never leave a key empty when empty overrides a good default (paths, binaries, URLs): use safe literals (`./data/development`, `rclone`).
|
||||
- The private `.env` is generated from the example with fresh random secrets, mode 600, never committed. The admin password applies at first bootstrap only and is reported to the owner once.
|
||||
Body starts here. Project-agnostic. Dry. Complete enough that Grok can
|
||||
turn it into MUST / MUST NOT text without asking again.
|
||||
```
|
||||
|
||||
## Serving law (molohttp)
|
||||
### Example entry
|
||||
|
||||
```markdown
|
||||
### L-2026-10-05-00: Example (format only)
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | grok-bot |
|
||||
| status | proposed |
|
||||
|
||||
Applications MUST refuse to start when the configured data directory
|
||||
resolves outside `./data`. Prove it with a maintained subprocess test
|
||||
against an inside path (allowed) and an outside path (refused).
|
||||
```
|
||||
|
||||
Append new lessons at the end of [Lessons](#lessons). Never renumber or
|
||||
delete prior entries. If a later lesson revises an earlier one, add a new
|
||||
entry that cites the earlier id.
|
||||
|
||||
## Lessons
|
||||
|
||||
### L-2026-10-05-01: Storage law
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | conflict |
|
||||
|
||||
All application data lives under `./data/<env>` inside the project. No
|
||||
exceptions, no second locations.
|
||||
|
||||
- `development` for local runs, `staging` and `production` as container
|
||||
file mounts, `test` for persistent test services.
|
||||
- Ephemeral runs use dotted subdirectories that clean themselves:
|
||||
`./data/.tmp-pytest-<pid>`, `./data/.tmp-locust`,
|
||||
`./data/.tmp-shots-<pid>`.
|
||||
- Each env dir holds the database file plus one subdirectory per kind
|
||||
(`uploads`, `backups`, `keys`, `locks`, `seo_reports`, ...). Database
|
||||
and files are separated by directory, never by system location.
|
||||
- Enforcement beats convention: the application refuses to start when its
|
||||
data dir resolves outside `./data` or its sqlite file outside the data
|
||||
dir. Prove it with a maintained test that runs the real import in a
|
||||
subprocess against inside and outside paths.
|
||||
- Named docker volumes are forbidden. Every mount is a file mount.
|
||||
Containers run as uid/gid 1000 so files stay human-owned; bind host
|
||||
directories must pre-exist with correct ownership because docker
|
||||
creates missing ones as root.
|
||||
|
||||
Source commits (approx.): `44d0c08`, `4268b1e`, `60dd597`. Residual
|
||||
conflict with template bare-metal shared-DB and "data volume" wording:
|
||||
see `CONFLICTS.md` items 1 and 4.
|
||||
|
||||
### L-2026-10-05-02: Environments
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | conflict |
|
||||
|
||||
Four environments, one shape each, fixed localhost ports (example
|
||||
offsets: staging 19847, production 19848, test 19849).
|
||||
|
||||
- Production owns the holy database. Nothing writes to it except
|
||||
production itself.
|
||||
- Staging is refreshed from production snapshots only: online sqlite
|
||||
backup API while prod keeps running, integrity check, stop, swap,
|
||||
start, verify health. Never the reverse direction.
|
||||
- Test is hermetic: persistent service data for the public test site,
|
||||
wiped or temp dirs for gallery and suite runs.
|
||||
- Snapshots flow one way: production to staging. Test never touches
|
||||
production data.
|
||||
|
||||
Source commits (approx.): `4268b1e`, `60dd597`. Residual conflict with
|
||||
template section 28.1 (`make dev` / `make prod` share the production
|
||||
database): see `CONFLICTS.md` item 1.
|
||||
|
||||
### L-2026-10-05-03: Docker law
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | conflict |
|
||||
|
||||
One compose file per environment, each with its own top-level `name:` so
|
||||
projects can never steal each other's containers.
|
||||
|
||||
- Dev and test run the dev server (`uvicorn --reload`) with the package,
|
||||
the manifest (`pyproject.toml`), and `./data/<env>` mounted: every
|
||||
saved change is live within seconds.
|
||||
- Production runs workers, no reload, no source mount except the data
|
||||
dir.
|
||||
- Production secrets use dedicated `PROD_`-prefixed variables with `:?`
|
||||
guards, so the dev `.env` can never silently satisfy them. Missing
|
||||
secrets fail loudly at startup, never with defaults.
|
||||
|
||||
Source commits (approx.): `4268b1e`, `60dd597`. Residual conflict with
|
||||
template zero-config boot and `SECRET_KEY` hardcoded fallback: see
|
||||
`CONFLICTS.md` item 2.
|
||||
|
||||
### L-2026-10-05-04: Environment files law
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | conflict |
|
||||
|
||||
`.env.example` lists every variable the code reads, no more, no less. A
|
||||
maintained contract test asserts the exact key set plus non-empty values
|
||||
for every path-like key, and fails on the old file before passing on the
|
||||
new one.
|
||||
|
||||
- Never leave a key empty when empty overrides a good default (paths,
|
||||
binaries, URLs): use safe literals (`./data/development`, `rclone`).
|
||||
- The private `.env` is generated from the example with fresh random
|
||||
secrets, mode 600, never committed. The admin password applies at
|
||||
first bootstrap only and is reported to the owner once.
|
||||
|
||||
Source commit (approx.): `60dd597`. Tied to `CONFLICTS.md` item 2 for
|
||||
secret defaults versus loud failure.
|
||||
|
||||
### L-2026-10-05-05: Serving law (molohttp)
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | folded |
|
||||
|
||||
One front door on 80/443, all local reverse-proxy upstreams.
|
||||
|
||||
- Back up before every edit, one format only: `<file>.bak-<reason>-<YYYYMMDDHHMMSS>`, `root:root 640`, reason slug required, never deleted.
|
||||
- Assert unique site ids AND unique hostnames before appending: on reload molohttp keeps the first site per hostname and silently drops later duplicates. Re-read the file after reload and assert the new site survived.
|
||||
- Change order: backup, edit, restore `molohttp:molohttp 640` ownership, validate JSON, reload, verify the site over HTTPS. Rollback is the same steps with the previous values; the old backend stays running until rollback is no longer wanted.
|
||||
- Back up before every edit, one format only:
|
||||
`<file>.bak-<reason>-<YYYYMMDDHHMMSS>`, `root:root 640`, reason slug
|
||||
required, never deleted.
|
||||
- Assert unique site ids AND unique hostnames before appending: on
|
||||
reload molohttp keeps the first site per hostname and silently drops
|
||||
later duplicates. Re-read the file after reload and assert the new
|
||||
site survived.
|
||||
- Change order: backup, edit, restore `molohttp:molohttp 640` ownership,
|
||||
validate JSON, reload, verify the site over HTTPS. Rollback is the
|
||||
same steps with the previous values; the old backend stays running
|
||||
until rollback is no longer wanted.
|
||||
- The admin surface is never exposed publicly. Local CLI only.
|
||||
|
||||
## Gallery law
|
||||
Source commits (approx.): `60dd597` (operator notes), production
|
||||
cutover work. Folded into template section 28.4 in product-agnostic
|
||||
form (front-door reverse proxy); the molohttp-specific ownership and
|
||||
JSON validate steps remain the lesson source here.
|
||||
|
||||
Every environment gets a screenshot gallery, generated the same way every time.
|
||||
### L-2026-10-05-06: Gallery law
|
||||
|
||||
- Boot mode builds a throwaway server on a fresh data dir and seeds one fixed record; base-url mode shoots a live environment and never writes seed data.
|
||||
- Full-page desktop shots plus a responsive matrix (down to 320px) with a horizontal-overflow check per cell; overflows are reported and exit nonzero.
|
||||
- One gallery per environment plus a hub page that rebuilds from whichever environments exist. Same script, same selectors, same waits for all.
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | folded |
|
||||
|
||||
## Test law
|
||||
Every environment gets a screenshot gallery, generated the same way
|
||||
every time.
|
||||
|
||||
Mirror layout (`tests/unit`, `tests/api`, `tests/e2e`), full suite green before done.
|
||||
- Boot mode builds a throwaway server on a fresh data dir and seeds one
|
||||
fixed record; base-url mode shoots a live environment and never writes
|
||||
seed data.
|
||||
- Full-page desktop shots plus a responsive matrix (down to 320px) with
|
||||
a horizontal-overflow check per cell; overflows are reported and exit
|
||||
nonzero.
|
||||
- One gallery per environment plus a hub page that rebuilds from
|
||||
whichever environments exist. Same script, same selectors, same waits
|
||||
for all.
|
||||
|
||||
- Browser tests wait up to 60 seconds per step and retry a stalled page load once through one shared helper. Filter only proven environmental console noise, never application errors.
|
||||
- Static gates run on every change: imports, linter, syntax, schema lint, UI invariants. Text gates (for example the em-dash ban) apply to docs too.
|
||||
- Every behavior change ships the smallest maintained test exercising it, proven red before the fix and green after. Never weaken a failing test to get green.
|
||||
Source commits (approx.): `4268b1e`, `60dd597`. Folded into template
|
||||
section 28.5.
|
||||
|
||||
## Git law
|
||||
### L-2026-10-05-07: Test law
|
||||
|
||||
Stage named files only, never broad-add a dirty tree. Commit and push only on explicit ask, never as a side effect of finishing.
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | conflict |
|
||||
|
||||
- `.gitignore` contains exactly: `.venv/`, `__pycache__/`, `*.pyc`, `.pytest_cache/`, `.coverage*`, `data/`, `out/`, `.env`, `*.egg-info/`, `build/`, `dist/`, `temp_*`, `*.swp`, `*~`, `.DS_Store`. Never ignore source, tests, docs, compose files, Dockerfiles, `.env.example`, Makefiles, or CI configs.
|
||||
- Any scratch, probe, or temporary file must be prefixed `temp_` and is never committed. `temp_*` in `.gitignore` enforces it.
|
||||
- Markdown files are never committed without explicit user confirmation. Docs are reviewed prose, not byproducts.
|
||||
- A pre-commit hook owns the patch version bump; nothing else edits the version.
|
||||
Mirror layout (`tests/unit`, `tests/api`, `tests/e2e`), full suite green
|
||||
before done.
|
||||
|
||||
## Code law
|
||||
- Browser tests wait up to 60 seconds per step and retry a stalled page
|
||||
load once through one shared helper. Filter only proven environmental
|
||||
console noise, never application errors.
|
||||
- Static gates run on every change: imports, linter, syntax, schema
|
||||
lint, UI invariants. Text gates (for example the em-dash ban) apply to
|
||||
docs too.
|
||||
- Every behavior change ships the smallest maintained test exercising
|
||||
it, proven red before the fix and green after. Never weaken a failing
|
||||
test to get green.
|
||||
|
||||
Explicit over implicit, flat over nested, sparse over dense, readable above all. No comments anywhere except the author line `retoor <retoor@molodetz.nl>` at the top of every file; code that needs a comment is rewritten until it does not. Defensive everywhere, errors never pass silently, no invented names with suffixes. JavaScript is ES6 modules, one class per file, one `Application` instantiated as `app` on `window`. Storage is dataset plus SQLite. The frontend is served through the backend. No emojis, no hype language, professional and scientific.
|
||||
Source commits (approx.): `94ae750`, `60dd597`. Residual conflict with
|
||||
template 15s page-fixture default timeout: see `CONFLICTS.md` item 3.
|
||||
Shared retry helper and related gates were folded.
|
||||
|
||||
## Docs law
|
||||
### L-2026-10-05-08: Git law
|
||||
|
||||
`README.md` is the living entry point (running, testing, docker, screenshots, data, operations) and stays true to the repo. Each deployment gets an operator guide with access facts, file owners, backup specification, and toggle procedures. Document only inspected facts, mark dated observations, and record anomalies instead of hiding them.
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | folded |
|
||||
|
||||
## Handover law
|
||||
Stage named files only, never broad-add a dirty tree. Commit and push
|
||||
only on explicit ask, never as a side effect of finishing.
|
||||
|
||||
Secrets are generated with a cryptographic RNG, stored only in their target (`.env` 600, container env), and reported to the owner once in chat, never written to the repo, docs, or logs beyond that handoff.
|
||||
- `.gitignore` contains exactly: `.venv/`, `__pycache__/`, `*.pyc`,
|
||||
`.pytest_cache/`, `.coverage*`, `data/`, `out/`, `.env`,
|
||||
`*.egg-info/`, `build/`, `dist/`, `temp_*`, `*.swp`, `*~`,
|
||||
`.DS_Store`. Never ignore source, tests, docs, compose files,
|
||||
Dockerfiles, `.env.example`, Makefiles, or CI configs.
|
||||
- Any scratch, probe, or temporary file must be prefixed `temp_` and is
|
||||
never committed. `temp_*` in `.gitignore` enforces it.
|
||||
- Markdown files are never committed without explicit user confirmation.
|
||||
Docs are reviewed prose, not byproducts.
|
||||
- A pre-commit hook owns the patch version bump; nothing else edits the
|
||||
version.
|
||||
|
||||
Source commit (approx.): `60dd597`. Folded into template sections 2 and
|
||||
36.
|
||||
|
||||
### L-2026-10-05-09: Code law
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | folded |
|
||||
|
||||
Explicit over implicit, flat over nested, sparse over dense, readable
|
||||
above all. No comments anywhere except the author line
|
||||
`retoor <retoor@molodetz.nl>` at the top of every file; code that needs
|
||||
a comment is rewritten until it does not. Defensive everywhere, errors
|
||||
never pass silently, no invented names with suffixes. JavaScript is ES6
|
||||
modules, one class per file, one `Application` instantiated as `app` on
|
||||
`window`. Storage is dataset plus SQLite. The frontend is served through
|
||||
the backend. No emojis, no hype language, professional and scientific.
|
||||
|
||||
Source commit (approx.): `60dd597`. Already largely present in the
|
||||
template; kept here as the lesson source.
|
||||
|
||||
### L-2026-10-05-10: Docs law
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | folded |
|
||||
|
||||
`README.md` is the living entry point (running, testing, docker,
|
||||
screenshots, data, operations) and stays true to the repo. Each
|
||||
deployment gets an operator guide with access facts, file owners, backup
|
||||
specification, and toggle procedures. Document only inspected facts,
|
||||
mark dated observations, and record anomalies instead of hiding them.
|
||||
|
||||
Source commits (approx.): `4268b1e`, `60dd597`. Folded into template
|
||||
section 32.
|
||||
|
||||
### L-2026-10-05-11: Handover law
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | muse-spark |
|
||||
| status | folded |
|
||||
|
||||
Secrets are generated with a cryptographic RNG, stored only in their
|
||||
target (`.env` 600, container env), and reported to the owner once in
|
||||
chat, never written to the repo, docs, or logs beyond that handoff.
|
||||
|
||||
Source commit (approx.): `60dd597`. Folded into template section 37.
|
||||
|
||||
### L-2026-10-05-12: Multi-agent lesson format for DPP feed
|
||||
|
||||
| Field | Value |
|
||||
|--------|-------|
|
||||
| date | 2026-10-05 |
|
||||
| agent | grok-bot |
|
||||
| status | folded |
|
||||
|
||||
This file is reformatted as the shared multi-agent lesson log. Grok Bot
|
||||
(AbsurdDevelopmentProject) alone updates `/workspace/dpp/dptemplate.md`.
|
||||
Other agents append generic lessons here. Existing storage,
|
||||
environments, docker, env-files, serving, gallery, test, git, code,
|
||||
docs, and handover laws are preserved as dated `muse-spark` entries
|
||||
above. New teachings use the skeleton under [Lesson format](#lesson-format).
|
||||
No lesson content may be deleted when the format evolves.
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "molodetz"
|
||||
version = "1.0.13"
|
||||
version = "1.0.14"
|
||||
description = "Molodetz, a calm community blog roll."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
Reference in New Issue
Block a user