devplacepy/quality_software.md

<!-- doc: quality_software.md | version: 1.2.1 | updated: 2026-10-11 -->
# Quality Software Playbook: How to Build a Site as Good as DevPlace

This is a generic, reusable method distilled from how DevPlace was built.
It is written so you can apply it to a completely different site, for
example a hosting company (accounts, products, orders, provisioning,
support tickets, status pages), and get the same stability and quality.

It is not about copying features. It is about copying structure,
constraints, and verification habits.

## 1. Core philosophy

1. Boring technology, smart edges. One backend language, one database
   file, server-rendered HTML, plain JavaScript. Put complexity where
   you can see it (request handlers, templates), keep storage dumb.
2. Fewer moving parts. Every dependency is a way to fail. Prefer the
   standard library, one process type, one data root.
3. Server-rendered first. The page works without JavaScript. JavaScript
   only upgrades live behavior (polling, websockets, optimistic UI).
4. One data source, many faces. Every domain object is served as HTML
   for browsers, JSON for API clients, an agent/tool action for
   automation, and a documented endpoint. Design the data once, then
   expose all four.
5. Exact, minimal diffs. Edit existing files, reuse existing helpers,
   match existing visual structure. Zero tolerance for a second way
   of doing something that already has one way.
6. Measure, do not theorize. Performance claims come from runs on real
   machines. If there is no measurement, say so.
7. Loud skips beat silent corruption. Reject unverifiable input with a
   clear reason and a counter. Never store what you cannot vouch for.

Hosting example: a VPS order is one row in one table. The same handler
renders the order page (HTML), returns the order (JSON), exposes a
`order_create` automation action, and is documented in your API docs.

## 2. Reference stack (proven shape)

You can swap brands, but keep the shape:

- Backend: one typed web framework (DevPlace: Python + FastAPI).
- Templates: server-side templates with one shared instance
  (DevPlace: Jinja2, single `templates` object imported everywhere).
- Frontend: no framework, no build step. Plain ES modules, one class
  per file, small shared utilities (`Http`, `Poller`, modal, toast).
- Database: one embedded relational file with WAL mode for small to
  medium scale (DevPlace: SQLite via `dataset`, WAL, busy timeout,
  mmap). Synchronous local calls are fine when they are
  sub-millisecond; do not make the data layer async for fashion.
- Auth: session cookie for browsers plus API key and Basic for
  machines, resolved in one `get_current_user` choke point. No JWT
  unless you need third-party verification.
- Static assets: content-addressed or boot-versioned URLs served
  immutable (`/static/v<ts>/...`), one helper builds every URL.
- Background work: one in-process queue with inline fallback so tests
  stay deterministic and production stays fast.

Hosting example: keep customers, services, invoices, tickets in one
SQLite/Postgres schema behind one data module. Serve the control
panel as HTML, the reseller API as JSON from the same handlers.

## 3. Repository layout that scales

Mirror URLs with code paths, one segment per directory:

- `routers/` mirrors the URL tree. Flat file for one resource
  (`feed.py`), package directory for sub-resources (`projects/`).
  Each leaf owns an `APIRouter`, the package `__init__.py` aggregates.
- `templates/` mirrors pages, extends one `base.html`. Page CSS in
  `extra_head`, page JS in `extra_js`. Shared partials for repeated
  UI (avatar link, user link, pagination, report button).
- `static/js/` mirrors behavior, one class per file, instantiated as
  `app`. Shared utilities live once and are reused, never forked.
- `static/css/` uses design tokens, no literal colors in page files,
  one closed breakpoint set, one z-index scale.
- `services/` holds background services behind one base class and one
  manager (start/stop/status/logs). Each service owns its tables and
  its admin controls.
- `tests/` mirrors the same tree by path: `tests/api/...` for HTTP,
  `tests/e2e/...` for browser, `tests/unit/...` for pure logic.

Rule: if you cannot guess the file from the URL, the layout is wrong.

## 4. Single data root and single config module

All runtime artifacts live under one root (`DATA_DIR`), every path
derived from it in one config module, created at boot:

- Database file, uploads, attachments, generated archives, keys,
  locks, backups, service state.
- Stored DB values are relative (`directory` + `stored_name`), so a
  layout move never needs a DB rewrite.
- One `DATA_PATHS` registry lists every path; one `ensure_data_dirs()`
  creates them before anything writes.

Hosting example: `data/` holds `app.db`, `uploads/`, `invoices/`,
`backups/`, `locks/`, `keys/`. Point `DATA_DIR` at a volume in prod.

## 5. Request pipeline (same for every route)

1. Single app factory: mount `/static`, register routers, install
   middleware in fixed order (timing outermost, then security
   headers, rate limit, maintenance gate, error handlers).
2. One auth resolution: cookie, API key header, Bearer, Basic,
   all in `get_current_user`. Guards: public read
   (`get_current_user`), member write (`require_user`), admin
   (`require_admin`).
3. One response helper (`respond()`): same context renders HTML for
   browsers and JSON for `Accept: application/json` via a response
   schema. A context key missing from the schema is dropped from
   JSON by design, so schemas are explicit contracts.
4. One validation style: Pydantic form models bound from forms,
   one global validation-error handler (re-render or redirect).
5. One timing mechanism: outermost middleware stamps start time and
   sets `X-Response-Time`. Never re-time per route.
6. Rate limit only mutations (`POST/PUT/PATCH/DELETE`), keyed by
   real IP, tunable from settings. Reads stay fast and exempt.

## 6. Data conventions that prevent entire bug classes

- Primary keys are opaque string IDs (`uid`), slugs accept slug or
  bare ID via one resolver. Slugs embed the random tail of the ID,
  never a timestamp prefix (prefix sharding funnels writes).
- Soft delete everywhere user content is removable: `deleted_at` +
  `deleted_by`, live rows are `deleted_at IS NULL`. Every insert
  writes both columns, every list filters them. Only garbage
  collection hard-deletes. One admin Trash restores/purges by event.
- Ownership rule: edits are owner-only, deletes are owner-or-admin,
  checked by one helper. Admin seniority: junior admins cannot act
  on senior admins.
- No N+1: batch helpers (`get_users_by_uids`, counts by ID lists,
  paginators) are the only way to load related rows in lists.
- Idempotent schema init: ensure full column set before indexes,
  because auto-schema tools create partial tables from first insert.
- Toggles revive, never duplicate: look up ignoring deleted flag,
  stamp on off, clear on on, under one shared stamp for cascades.

Hosting example: canceling a server is a soft delete with one stamp
covering service, DNS rows, and scheduled jobs, restorable from Trash
until GC purges it.

## 7. Frontend rules that keep UI consistent

- One base template owns nav, head, modals, toasts, timing badge.
  Nav active state is one global, never inline path checks.
- Never hardcode `/static/...`. One helper builds versioned URLs in
  templates and one in JS.
- Server renders content that exists at request time (SEO, no-JS,
  speed). Client pipeline is only for live content (chat bubbles,
  streamed results), always sanitized fail-closed.
- Dates stored as UTC ISO, rendered as `<time data-dt>` and
  reformatted to viewer timezone client-side, one format for users
  (DevPlace: DD/MM/YYYY).
- Modals, lightbox, toasts, floating windows, scroll memory: one
  implementation each, reused by class name. No inline JS.

## 8. Background jobs and services (one pattern)

- One generic fire-and-forget queue: `submit(fn, *args)`. When the
  consumer runs, it defers; when it does not (tests, full queue),
  it runs inline so behavior stays deterministic.
- One base service class: `run` loop, supervised start/stop, status,
  logs, settings. One manager registers all services.
- One async-job pattern for slow work (archives, audits, reports):
  enqueue (`POST .../run`), poll (`GET .../{uid}`), read report
  (`GET .../{uid}/report`), expire and prune by CLI.
- Multi-worker safety: schema work behind one init lock, background
  services elected by one service lock so exactly one worker owns
  them. Never assume single-process in job code.

Hosting example: provisioning, backups, SSL renewal, and usage
metering are all job services with run/poll/report plus
`cli <domain> prune` retention commands.

## 9. Testing: three tiers, one tree, no skips

- Tiers by fixture, not by folder debate:
  - `unit`: pure in-process, no HTTP, no browser.
  - `api`: HTTP integration against a live test server + temp DB.
  - `e2e`: Playwright browser, real user flows.
- Test server runs on its own port with its own temp DB and data
  dir, background services disabled. One serial process, no
  parallelism, so results are stable.
- Every run reports every failure (`-rf`), triage cheapest first:
  unit+api (~minutes), then browser tier, then `--last-failed`
  loop until empty.
- Required browser habits: explicit `domcontentloaded` on
  navigations, visible-state waits over sleeps, scoped selectors,
  global settings restored in `try/finally`.
- Gate before suite: clean app import, per-language syntax check,
  linter on touched files (catches missing imports that compile
  checks miss), no forbidden characters or patterns.

Quality bar: the full suite (all three tiers, every test) passes
before a change is done. A red test covering your change is the
requirement, not a stale artifact. Never narrow a run to make it
pass.

## 10. Extra verification for money, state, and races

Example-based tests are blind to wrong formulas at untried inputs,
long-sequence invariant breaks, and races. For spendable resources
(credits, coins, quotas), bounded state machines (tiers, levels),
or read-then-write mutations reachable from two requests, add four
disposable layers against a temp DB (not committed tests unless
asked):

1. Property checks over the full input domain (monotonicity,
   bounds, non-negativity) across hundreds of values.
2. Stateful fuzzing: randomized action sequences across many
   entities, invariants re-checked after every action.
3. Real-process concurrency: separate OS processes (not threads),
   fresh production-like rows (never pre-zero NULL columns),
   exact success counts and exact final totals. Fix races with one
   atomic conditional `UPDATE ... WHERE <precondition>` checked by
   real `rowcount` at the chokepoint.
4. Static analysis on every touched file (`pyflakes`/`ruff`), which
   catches runtime `NameError` paths the other layers may miss.

Hosting example: apply this to credit spend, plan upgrades,
affiliate payouts, and concurrent provisioning of the last IP or
stock-limited node.

## 11. Operations: deploy, verify, prune

- One-command environments: `install`, `dev` (reload, one worker),
  `prod` (multi-worker, no template reload, real backlog), `test*`,
  `coverage`, load test. Every Python target uses the project venv.
- No bytecode in repo: `PYTHONDONTWRITEBYTECODE=1` for all targets,
  `clean` removes venv and stray artifacts.
- CI runs the full suite serially on every push/PR, publishes
  coverage and failure screenshots. CI green before merge.
- Production DB is never touched without explicit written
  confirmation plus a two-factor token; disposable scripts always
  target a scratch DB via env override.
- Retention is a feature: every expirable domain gets
  `prune` (expired only) and `clear` (everything) CLI commands,
  wired to the same GC the server uses.
- Backups are first-class: targets, worker, schedules, offload,
  download restricted to the primary admin, prune missing files.

## 12. Security, moderation, and trust checklist

- Passwords hashed with a slow KDF (PBKDF2-SHA256), session tokens
  random 256-bit, per-process TTL cache keyed by token.
- Ownership checked server-side on every mutation; deletes confirm
  explicitly and audit; destructive automation actions require a
  two-step confirm flag declared in their schema.
- Every user-visible content table is reportable by construction:
  registry entry, object URL resolver, report button in the action
  bar. Private-only tables are listed as unreportable with a reason,
  enforced by a test that fails on unclassified tables.
- Outbound HTTP goes through one stealth/guarded client (consistent
  TLS + headers + proxy + SSRF guard), never bare clients per call.
- Audit log is append-only, never raises into the caller, with one
  catalogue of event keys. Admin-only read.
- Privacy docs and store declarations update in the same change as
  any new personal-data collection or third-party transmission.

Hosting example: tickets, reviews, and public server comments are
reportable; invoices and API tokens are private-by-design with
written reasons; provisioning calls go through the one guarded
client with proxy support.

## 13. Docs, SEO, and API surface

- Every public/auth endpoint gets prose docs plus an interactive
  API entry (params, auth, sample response). Public pages get SEO
  context and JSON-LD; sitemap and robots are served, not static.
- One docs registry (`DOCS_PAGES`) owns prose pages; API docs own
  endpoint entries. New route without docs is incomplete.
- README stays product-facing (routes, config, env vars, features).
  Nested `CLAUDE.md` files hold subsystem mechanics next to the code
  so the always-loaded root doc stays small.
- Version static assets, canonical URLs per listing, breadcrumbs,
  related content, and one default OG image.

## 14. Development flow (the daily loop)

Every feature follows the same eight steps in data-flow order, and a
failed step blocks the rest (never skip ahead):

1. Understand. Read the router, template, matching tests, and the
   subsystem doc. Trace input model to router to data helper to
   response (HTML and JSON). Find a relatable existing example and
   match it structurally and visually; if none exists, stop and agree
   on the new pattern before building.
2. Data layer. Query/batch helpers first (never inline N+1), input
   model, JSON response schema (every exposed key declared or it is
   silently dropped), idempotent schema/index changes.
3. Server layer. Handler with the right guard (public/member/admin),
   specific paths before catch-alls, HTML+JSON via one responder.
4. View layer. Extend base, reuse partials and globals, page CSS/JS
   in the standard blocks, one ES class per file.
5. Safety layer. Reportable/private classification, ownership checks,
   confirm gates, audit events, privacy-doc update if new personal
   data or third-party transmission appears.
6. Agent + docs layer. Tool action per user-askable operation,
   endpoint doc entry, prose page if user-facing, SEO context and
   sitemap entry if indexable. This step is the most-forgotten; do
   not skip it.
7. Document. Product README for user-visible change, subsystem doc
   for new mechanics or pitfalls, root doc only for new global rules.
8. Validate. Clean import, per-language syntax check, linter on
   touched files, full three-tier suite green (see section 9), plus
   the money/state/race procedure where it applies.

Failure triage is cheapest-first: fast tiers (unit+api, minutes)
before browser tier, then re-run only failures (`--last-failed`)
until empty. One pass reports every failure; fix the list, not one
bug per pass.

Session rule: commit, deploy, verify in the same session, in that
order. A change that is not running in production is not done. After
deploys, verify live; green locally means nothing until prod
confirms it.

## 15. Workflows and automation

- One command surface. Every repeated action is a named target or CLI
  command (`dev`, `prod`, `test`, `test-fast`, `test-failed`,
  `coverage`, load test, `prune`, backups). No tribal shell
  one-liners; if it is run twice, it gets a name.
- DTAP pipeline. Development (reload, one worker) to Test (CI serial
  suite + coverage + failure screenshots, green before merge) to
  production (multi-worker, no template reload, real backlog).
- Scheduled hygiene inside the system, not in heads: retention,
  cleanup, quota resets, and health checks run on loops or CLI
  commands with dry-run defaults and narrow scopes for destructive
  ops. Destructive operations are dry-run first, explicitly
  confirmed, date-guarded or allowlisted.
- Load testing is a target, not an event: scripted Locust run
  against a scratch DB with fixed users, spawn rate, and run time,
  producing an HTML report.
- Production guardrails are enforced, not remembered: a hook denies
  commands reaching the live DB without explicit written
  confirmation plus a second approval; disposable scripts always
  target scratch databases via env override.

Hosting example: `provision`, `suspend`, `backup-run`,
`ssl-renew`, `usage-meter`, `invoice-prune` are all named commands
with `--dry-run`; nightly loops handle expiry, renewal, and status
checks with human-readable pressure states.

## 16. Agents: in-product assistant, services, and coding help

DevPlace quality comes partly from treating agents as first-class
consumers of the same routes humans use:

- In-product assistant (DevPlace: Devii). A conversational operator
  over the real API: per-owner sessions and channels (web, terminal,
  chat bridge), a tool catalog where each tool mirrors one route
  with matching auth, confirmation gates on destructive tools (first
  call refused, agent shows exact target, second call passes
  `confirm=true`), per-user cost attribution and quotas through one
  AI gateway, and usage/ledger pages per user. New feature rule: if
  a user could ask the assistant to do it, it gets a tool.
- Gateway pattern. Every AI/model call flows through one internal
  gateway (usage ledger, provider/model routing, per-call cost
  headers), so spend is attributable and every consumer (correction,
  modifier, grading, chat) shares quotas and audit.
- Background service agents. Crawlers, importers, reconcilers, and
  persona bots run as supervised services under one manager with
  start/stop/status/logs and admin pages (DevPlace: news importer,
  bot fleet with live monitor, container reconciler, audit writer).
  Exactly one worker owns them via a service lock.
- Coding-agent support. Nested subsystem docs next to the code so an
  agent loads only what it touches; strict conventions (no second
  implementation, named helpers, closed token/breakpoint/z-index
  sets) so generated code matches; tool catalog and docs registry
  so the agent can act and document through the same contracts.
- Human authority. Agents propose and execute reversible work;
  destructive, cross-user, or policy-crossing actions need explicit
  human confirmation, seniority checks where roles exist, and an
  audit trail. When a floor is hit (retention minimum, protected
  path), the agent holds and pages a human instead of deleting.

Hosting example: a support assistant with tools like `server_reboot`
(public read tools open, mutating tools auth-gated, destructive
tools confirm-gated), all spend metered per customer through your
gateway, provisioning and renewal running as supervised services.

## 17. Documenting: what lives where, and when

Documentation is layered by audience and distance from the code, and
every layer updates in the same change as the code it describes:

- Source code: no comments, no docstrings. Code is self-documenting
  through names, small helpers, and one obvious way per task. A
  decision worth recording goes in the change description or the
  subsystem doc, never as deliberation text in the source.
- Subsystem docs (DevPlace: nested `CLAUDE.md` next to the code).
  Each directory that owns a mechanic holds its own doc: helpers,
  pitfalls, cross-layer wiring, worked examples. An agent or human
  loads only the docs for the area being touched, so the always-
  loaded cost stays flat as the project grows.
- Root doc (DevPlace: `CLAUDE.md`, `ARCHITECTURE.md`). Global rules,
  commands, and architecture only. A per-feature detail never lands
  here; a new global rule always does. The architecture atlas is
  measured from the running app and schema (counts, diagrams), not
  written from memory, and disagreements between docs and code are
  recorded explicitly.
- Product README. User-visible surface: routes, config, env vars,
  features, quick start. Updated when behavior a user can see
  changes.
- User docs site (DevPlace: `/docs` with a `DOCS_PAGES` registry).
  Prose pages per audience tier plus an interactive API reference
  where every endpoint entry carries params, auth, and a sample
  response. A new public or auth endpoint without a doc entry is
  incomplete by definition.
- Privacy and store declarations. Any new personal-data category or
  third-party transmission updates the privacy doc and platform
  declarations in the same change, never as a follow-up.
- Event and reason registries. Audit event keys, report reasons, and
  similar closed sets live in one catalogue each, so every client
  and test renders the same list.

Rule: docs state what the system does today, with gaps listed next
to the code that will close them. No fiction, no roadmap prose.

Hosting example: `billing/CLAUDE.md` documents proration and retry
rules next to the billing code; the root doc only says billing
exists and which commands run it; `/docs` gains a prose page plus
one API entry per new endpoint in the same change.

## 18. Roles, permissions, and ownership

One small role set, enforced server-side at every surface (web UI,
REST API, and agent tools alike):

- Roles (DevPlace: guest, member, admin, primary admin). Stored as
  exact strings, tested through one `is_admin()` helper, never
  hand-rolled compares. Any write to a role invalidates the cached
  role lookups.
- Guests read public pages; members write their own content;
  admins moderate and operate. Public reads resolve the viewer but
  never require login; every write names its guard explicitly.
- Ownership first: edits are owner-only, deletes are owner-or-admin,
  checked by one helper comparing resource owner to actor. Admins
  can remove abuse but cannot rewrite someone else's words.
- Seniority among admins: a junior admin (later registration) cannot
  mutate a more senior admin's account or content. Every per-user
  admin mutation is gated by one seniority predicate, denials are
  audited, and the gate is server-side so it also covers agent
  tools. One primary admin (the first admin account) is the single
  exception with full visibility and full control, including
  private-by-admin content and restricted downloads.
- Visibility predicates are single functions, never inline checks:
  e.g. `can_view(project, user)` decides private-content access in
  one place (owner sees own, admins see members' private rows, but
  an admin's own private rows stay visible only to that admin
  except for the primary admin). Container and workspace isolation
  get their own stricter predicates on the same pattern.
- Machines authenticate as users: every account holds an API key,
  plus issuable scoped tokens; the same guards apply regardless of
  credential type, so the REST API cannot do what the UI forbids.
- Agents act as the asking user: tool auth mirrors route guards,
  destructive tools add confirmation gates, and every admin tool
  passes through the same seniority and audit path as the web UI.

Rule: if a permission check appears twice in different words, one
of them is a bug. Hoist it into the single predicate and call it
from both places.

Hosting example: guest (pricing, status), customer (own servers and
invoices), support (view customers' servers, restart but not delete;
junior support cannot touch senior staff accounts), owner (full
access plus backup downloads and billing refunds). One
`can_view_server` / `can_manage_server` pair gates panel, API, and
assistant tools together.

## 19. Replication checklist for your hosting site

Use this order; it is the data-flow order DevPlace follows:

1. Scaffold: app factory, config with one data root, shared
   templates, response helper, auth choke point, middleware stack.
2. Data: users, sessions, settings, then one domain (products).
   Add soft-delete columns, batch helpers, idempotent init.
3. Server: public reads first (HTML+JSON via one handler), then
   member writes with guards, then admin views.
4. View: base template, tokens, partials, one JS utility set.
   Server-render listings and detail pages.
5. Safety: reportable/private classification, ownership checks,
   confirm gates, audit events.
6. Automation + docs: tool action per user-askable operation,
   API doc entry per endpoint, SEO context per public page.
7. Verification: import gate, lint touched files, full three-tier
   suite green, plus the four-layer money/state/race procedure
   where it applies.
8. Ops: venv targets, CI serial suite, prune/clear CLIs, backups,
   production-DB guard, deploy then verify live.

## 20. What to copy literally vs adapt

Copy literally: single data root, soft-delete columns and filters,
batch loaders, one response helper, one auth resolver, test tiers
and temp-DB server fixture, prune/clear per domain, audit-never-
raises, versioned static URLs.

Adapt to hosting: table list (customers, plans, servers, invoices,
tickets, status incidents), provisioning job services, payment and
registrar integrations behind the one outbound client, usage
metering and quota rules, status page and notifications.

If you keep the constraints and the verification loop, the domain
can change completely and the quality transfers with it.

## Doc history

| Version | Date | Change |
|---------|------|--------|
| 1.0.0 | 2026-10-11 | Created: philosophy, stack, layout, data, pipeline, frontend, jobs, testing, verification, ops, security, docs, checklist. |
| 1.1.0 | 2026-10-11 | Added development flow, workflows/automation, and agents sections. |
| 1.2.0 | 2026-10-11 | Added documenting and roles/permissions sections. |
| 1.2.1 | 2026-10-11 | Added doc version header and history per quality_versioning.md. |