devplacepy/quality_software.md

Quality Software Playbook: How to Build a Site as Good as DevPlace

This is a generic, reusable method distilled from how DevPlace was built. It is written so you can apply it to a completely different site, for example a hosting company (accounts, products, orders, provisioning, support tickets, status pages), and get the same stability and quality.

It is not about copying features. It is about copying structure, constraints, and verification habits.

1. Core philosophy

1. Boring technology, smart edges. One backend language, one database file, server-rendered HTML, plain JavaScript. Put complexity where you can see it (request handlers, templates), keep storage dumb.
2. Fewer moving parts. Every dependency is a way to fail. Prefer the standard library, one process type, one data root.
3. Server-rendered first. The page works without JavaScript. JavaScript only upgrades live behavior (polling, websockets, optimistic UI).
4. One data source, many faces. Every domain object is served as HTML for browsers, JSON for API clients, an agent/tool action for automation, and a documented endpoint. Design the data once, then expose all four.
5. Exact, minimal diffs. Edit existing files, reuse existing helpers, match existing visual structure. Zero tolerance for a second way of doing something that already has one way.
6. Measure, do not theorize. Performance claims come from runs on real machines. If there is no measurement, say so.
7. Loud skips beat silent corruption. Reject unverifiable input with a clear reason and a counter. Never store what you cannot vouch for.

Hosting example: a VPS order is one row in one table. The same handler renders the order page (HTML), returns the order (JSON), exposes a order_create automation action, and is documented in your API docs.

2. Reference stack (proven shape)

You can swap brands, but keep the shape:

• Backend: one typed web framework (DevPlace: Python + FastAPI).
• Templates: server-side templates with one shared instance (DevPlace: Jinja2, single templates object imported everywhere).
• Frontend: no framework, no build step. Plain ES modules, one class per file, small shared utilities (Http, Poller, modal, toast).
• Database: one embedded relational file with WAL mode for small to medium scale (DevPlace: SQLite via dataset, WAL, busy timeout, mmap). Synchronous local calls are fine when they are sub-millisecond; do not make the data layer async for fashion.
• Auth: session cookie for browsers plus API key and Basic for machines, resolved in one get_current_user choke point. No JWT unless you need third-party verification.
• Static assets: content-addressed or boot-versioned URLs served immutable (/static/v<ts>/...), one helper builds every URL.
• Background work: one in-process queue with inline fallback so tests stay deterministic and production stays fast.

Hosting example: keep customers, services, invoices, tickets in one SQLite/Postgres schema behind one data module. Serve the control panel as HTML, the reseller API as JSON from the same handlers.

3. Repository layout that scales

Mirror URLs with code paths, one segment per directory:

• routers/ mirrors the URL tree. Flat file for one resource (feed.py), package directory for sub-resources (projects/). Each leaf owns an APIRouter, the package __init__.py aggregates.
• templates/ mirrors pages, extends one base.html. Page CSS in extra_head, page JS in extra_js. Shared partials for repeated UI (avatar link, user link, pagination, report button).
• static/js/ mirrors behavior, one class per file, instantiated as app. Shared utilities live once and are reused, never forked.
• static/css/ uses design tokens, no literal colors in page files, one closed breakpoint set, one z-index scale.
• services/ holds background services behind one base class and one manager (start/stop/status/logs). Each service owns its tables and its admin controls.
• tests/ mirrors the same tree by path: tests/api/... for HTTP, tests/e2e/... for browser, tests/unit/... for pure logic.

Rule: if you cannot guess the file from the URL, the layout is wrong.

4. Single data root and single config module

All runtime artifacts live under one root (DATA_DIR), every path derived from it in one config module, created at boot:

• Database file, uploads, attachments, generated archives, keys, locks, backups, service state.
• Stored DB values are relative (directory + stored_name), so a layout move never needs a DB rewrite.
• One DATA_PATHS registry lists every path; one ensure_data_dirs() creates them before anything writes.

Hosting example: data/ holds app.db, uploads/, invoices/, backups/, locks/, keys/. Point DATA_DIR at a volume in prod.

5. Request pipeline (same for every route)

1. Single app factory: mount /static, register routers, install middleware in fixed order (timing outermost, then security headers, rate limit, maintenance gate, error handlers).
2. One auth resolution: cookie, API key header, Bearer, Basic, all in get_current_user. Guards: public read (get_current_user), member write (require_user), admin (require_admin).
3. One response helper (respond()): same context renders HTML for browsers and JSON for Accept: application/json via a response schema. A context key missing from the schema is dropped from JSON by design, so schemas are explicit contracts.
4. One validation style: Pydantic form models bound from forms, one global validation-error handler (re-render or redirect).
5. One timing mechanism: outermost middleware stamps start time and sets X-Response-Time. Never re-time per route.
6. Rate limit only mutations (POST/PUT/PATCH/DELETE), keyed by real IP, tunable from settings. Reads stay fast and exempt.

6. Data conventions that prevent entire bug classes

• Primary keys are opaque string IDs (uid), slugs accept slug or bare ID via one resolver. Slugs embed the random tail of the ID, never a timestamp prefix (prefix sharding funnels writes).
• Soft delete everywhere user content is removable: deleted_at + deleted_by, live rows are deleted_at IS NULL. Every insert writes both columns, every list filters them. Only garbage collection hard-deletes. One admin Trash restores/purges by event.
• Ownership rule: edits are owner-only, deletes are owner-or-admin, checked by one helper. Admin seniority: junior admins cannot act on senior admins.
• No N+1: batch helpers (get_users_by_uids, counts by ID lists, paginators) are the only way to load related rows in lists.
• Idempotent schema init: ensure full column set before indexes, because auto-schema tools create partial tables from first insert.
• Toggles revive, never duplicate: look up ignoring deleted flag, stamp on off, clear on on, under one shared stamp for cascades.

Hosting example: canceling a server is a soft delete with one stamp covering service, DNS rows, and scheduled jobs, restorable from Trash until GC purges it.

7. Frontend rules that keep UI consistent

• One base template owns nav, head, modals, toasts, timing badge. Nav active state is one global, never inline path checks.
• Never hardcode /static/.... One helper builds versioned URLs in templates and one in JS.
• Server renders content that exists at request time (SEO, no-JS, speed). Client pipeline is only for live content (chat bubbles, streamed results), always sanitized fail-closed.
• Dates stored as UTC ISO, rendered as <time data-dt> and reformatted to viewer timezone client-side, one format for users (DevPlace: DD/MM/YYYY).
• Modals, lightbox, toasts, floating windows, scroll memory: one implementation each, reused by class name. No inline JS.

8. Background jobs and services (one pattern)

• One generic fire-and-forget queue: submit(fn, *args). When the consumer runs, it defers; when it does not (tests, full queue), it runs inline so behavior stays deterministic.
• One base service class: run loop, supervised start/stop, status, logs, settings. One manager registers all services.
• One async-job pattern for slow work (archives, audits, reports): enqueue (POST .../run), poll (GET .../{uid}), read report (GET .../{uid}/report), expire and prune by CLI.
• Multi-worker safety: schema work behind one init lock, background services elected by one service lock so exactly one worker owns them. Never assume single-process in job code.

Hosting example: provisioning, backups, SSL renewal, and usage metering are all job services with run/poll/report plus cli <domain> prune retention commands.

9. Testing: three tiers, one tree, no skips

• Tiers by fixture, not by folder debate:
  • unit: pure in-process, no HTTP, no browser.
  • api: HTTP integration against a live test server + temp DB.
  • e2e: Playwright browser, real user flows.
• Test server runs on its own port with its own temp DB and data dir, background services disabled. One serial process, no parallelism, so results are stable.
• Every run reports every failure (-rf), triage cheapest first: unit+api (~minutes), then browser tier, then --last-failed loop until empty.
• Required browser habits: explicit domcontentloaded on navigations, visible-state waits over sleeps, scoped selectors, global settings restored in try/finally.
• Gate before suite: clean app import, per-language syntax check, linter on touched files (catches missing imports that compile checks miss), no forbidden characters or patterns.

Quality bar: the full suite (all three tiers, every test) passes before a change is done. A red test covering your change is the requirement, not a stale artifact. Never narrow a run to make it pass.

10. Extra verification for money, state, and races

Example-based tests are blind to wrong formulas at untried inputs, long-sequence invariant breaks, and races. For spendable resources (credits, coins, quotas), bounded state machines (tiers, levels), or read-then-write mutations reachable from two requests, add four disposable layers against a temp DB (not committed tests unless asked):

1. Property checks over the full input domain (monotonicity, bounds, non-negativity) across hundreds of values.
2. Stateful fuzzing: randomized action sequences across many entities, invariants re-checked after every action.
3. Real-process concurrency: separate OS processes (not threads), fresh production-like rows (never pre-zero NULL columns), exact success counts and exact final totals. Fix races with one atomic conditional UPDATE ... WHERE <precondition> checked by real rowcount at the chokepoint.
4. Static analysis on every touched file (pyflakes/ruff), which catches runtime NameError paths the other layers may miss.

Hosting example: apply this to credit spend, plan upgrades, affiliate payouts, and concurrent provisioning of the last IP or stock-limited node.

11. Operations: deploy, verify, prune

• One-command environments: install, dev (reload, one worker), prod (multi-worker, no template reload, real backlog), test*, coverage, load test. Every Python target uses the project venv.
• No bytecode in repo: PYTHONDONTWRITEBYTECODE=1 for all targets, clean removes venv and stray artifacts.
• CI runs the full suite serially on every push/PR, publishes coverage and failure screenshots. CI green before merge.
• Production DB is never touched without explicit written confirmation plus a two-factor token; disposable scripts always target a scratch DB via env override.
• Retention is a feature: every expirable domain gets prune (expired only) and clear (everything) CLI commands, wired to the same GC the server uses.
• Backups are first-class: targets, worker, schedules, offload, download restricted to the primary admin, prune missing files.

12. Security, moderation, and trust checklist

• Passwords hashed with a slow KDF (PBKDF2-SHA256), session tokens random 256-bit, per-process TTL cache keyed by token.
• Ownership checked server-side on every mutation; deletes confirm explicitly and audit; destructive automation actions require a two-step confirm flag declared in their schema.
• Every user-visible content table is reportable by construction: registry entry, object URL resolver, report button in the action bar. Private-only tables are listed as unreportable with a reason, enforced by a test that fails on unclassified tables.
• Outbound HTTP goes through one stealth/guarded client (consistent TLS + headers + proxy + SSRF guard), never bare clients per call.
• Audit log is append-only, never raises into the caller, with one catalogue of event keys. Admin-only read.
• Privacy docs and store declarations update in the same change as any new personal-data collection or third-party transmission.

Hosting example: tickets, reviews, and public server comments are reportable; invoices and API tokens are private-by-design with written reasons; provisioning calls go through the one guarded client with proxy support.

13. Docs, SEO, and API surface

• Every public/auth endpoint gets prose docs plus an interactive API entry (params, auth, sample response). Public pages get SEO context and JSON-LD; sitemap and robots are served, not static.
• One docs registry (DOCS_PAGES) owns prose pages; API docs own endpoint entries. New route without docs is incomplete.
• README stays product-facing (routes, config, env vars, features). Nested CLAUDE.md files hold subsystem mechanics next to the code so the always-loaded root doc stays small.
• Version static assets, canonical URLs per listing, breadcrumbs, related content, and one default OG image.

14. Development flow (the daily loop)

Every feature follows the same eight steps in data-flow order, and a failed step blocks the rest (never skip ahead):

1. Understand. Read the router, template, matching tests, and the subsystem doc. Trace input model to router to data helper to response (HTML and JSON). Find a relatable existing example and match it structurally and visually; if none exists, stop and agree on the new pattern before building.
2. Data layer. Query/batch helpers first (never inline N+1), input model, JSON response schema (every exposed key declared or it is silently dropped), idempotent schema/index changes.
3. Server layer. Handler with the right guard (public/member/admin), specific paths before catch-alls, HTML+JSON via one responder.
4. View layer. Extend base, reuse partials and globals, page CSS/JS in the standard blocks, one ES class per file.
5. Safety layer. Reportable/private classification, ownership checks, confirm gates, audit events, privacy-doc update if new personal data or third-party transmission appears.
6. Agent + docs layer. Tool action per user-askable operation, endpoint doc entry, prose page if user-facing, SEO context and sitemap entry if indexable. This step is the most-forgotten; do not skip it.
7. Document. Product README for user-visible change, subsystem doc for new mechanics or pitfalls, root doc only for new global rules.
8. Validate. Clean import, per-language syntax check, linter on touched files, full three-tier suite green (see section 9), plus the money/state/race procedure where it applies.

Failure triage is cheapest-first: fast tiers (unit+api, minutes) before browser tier, then re-run only failures (--last-failed) until empty. One pass reports every failure; fix the list, not one bug per pass.

Session rule: commit, deploy, verify in the same session, in that order. A change that is not running in production is not done. After deploys, verify live; green locally means nothing until prod confirms it.

15. Workflows and automation

• One command surface. Every repeated action is a named target or CLI command (dev, prod, test, test-fast, test-failed, coverage, load test, prune, backups). No tribal shell one-liners; if it is run twice, it gets a name.
• DTAP pipeline. Development (reload, one worker) to Test (CI serial suite + coverage + failure screenshots, green before merge) to production (multi-worker, no template reload, real backlog).
• Scheduled hygiene inside the system, not in heads: retention, cleanup, quota resets, and health checks run on loops or CLI commands with dry-run defaults and narrow scopes for destructive ops. Destructive operations are dry-run first, explicitly confirmed, date-guarded or allowlisted.
• Load testing is a target, not an event: scripted Locust run against a scratch DB with fixed users, spawn rate, and run time, producing an HTML report.
• Production guardrails are enforced, not remembered: a hook denies commands reaching the live DB without explicit written confirmation plus a second approval; disposable scripts always target scratch databases via env override.

Hosting example: provision, suspend, backup-run, ssl-renew, usage-meter, invoice-prune are all named commands with --dry-run; nightly loops handle expiry, renewal, and status checks with human-readable pressure states.

16. Agents: in-product assistant, services, and coding help

DevPlace quality comes partly from treating agents as first-class consumers of the same routes humans use:

• In-product assistant (DevPlace: Devii). A conversational operator over the real API: per-owner sessions and channels (web, terminal, chat bridge), a tool catalog where each tool mirrors one route with matching auth, confirmation gates on destructive tools (first call refused, agent shows exact target, second call passes confirm=true), per-user cost attribution and quotas through one AI gateway, and usage/ledger pages per user. New feature rule: if a user could ask the assistant to do it, it gets a tool.
• Gateway pattern. Every AI/model call flows through one internal gateway (usage ledger, provider/model routing, per-call cost headers), so spend is attributable and every consumer (correction, modifier, grading, chat) shares quotas and audit.
• Background service agents. Crawlers, importers, reconcilers, and persona bots run as supervised services under one manager with start/stop/status/logs and admin pages (DevPlace: news importer, bot fleet with live monitor, container reconciler, audit writer). Exactly one worker owns them via a service lock.
• Coding-agent support. Nested subsystem docs next to the code so an agent loads only what it touches; strict conventions (no second implementation, named helpers, closed token/breakpoint/z-index sets) so generated code matches; tool catalog and docs registry so the agent can act and document through the same contracts.
• Human authority. Agents propose and execute reversible work; destructive, cross-user, or policy-crossing actions need explicit human confirmation, seniority checks where roles exist, and an audit trail. When a floor is hit (retention minimum, protected path), the agent holds and pages a human instead of deleting.

Hosting example: a support assistant with tools like server_reboot (public read tools open, mutating tools auth-gated, destructive tools confirm-gated), all spend metered per customer through your gateway, provisioning and renewal running as supervised services.

17. Documenting: what lives where, and when

Documentation is layered by audience and distance from the code, and every layer updates in the same change as the code it describes:

• Source code: no comments, no docstrings. Code is self-documenting through names, small helpers, and one obvious way per task. A decision worth recording goes in the change description or the subsystem doc, never as deliberation text in the source.
• Subsystem docs (DevPlace: nested CLAUDE.md next to the code). Each directory that owns a mechanic holds its own doc: helpers, pitfalls, cross-layer wiring, worked examples. An agent or human loads only the docs for the area being touched, so the always- loaded cost stays flat as the project grows.
• Root doc (DevPlace: CLAUDE.md, ARCHITECTURE.md). Global rules, commands, and architecture only. A per-feature detail never lands here; a new global rule always does. The architecture atlas is measured from the running app and schema (counts, diagrams), not written from memory, and disagreements between docs and code are recorded explicitly.
• Product README. User-visible surface: routes, config, env vars, features, quick start. Updated when behavior a user can see changes.
• User docs site (DevPlace: /docs with a DOCS_PAGES registry). Prose pages per audience tier plus an interactive API reference where every endpoint entry carries params, auth, and a sample response. A new public or auth endpoint without a doc entry is incomplete by definition.
• Privacy and store declarations. Any new personal-data category or third-party transmission updates the privacy doc and platform declarations in the same change, never as a follow-up.
• Event and reason registries. Audit event keys, report reasons, and similar closed sets live in one catalogue each, so every client and test renders the same list.

Rule: docs state what the system does today, with gaps listed next to the code that will close them. No fiction, no roadmap prose.

Hosting example: billing/CLAUDE.md documents proration and retry rules next to the billing code; the root doc only says billing exists and which commands run it; /docs gains a prose page plus one API entry per new endpoint in the same change.

18. Roles, permissions, and ownership

One small role set, enforced server-side at every surface (web UI, REST API, and agent tools alike):

• Roles (DevPlace: guest, member, admin, primary admin). Stored as exact strings, tested through one is_admin() helper, never hand-rolled compares. Any write to a role invalidates the cached role lookups.
• Guests read public pages; members write their own content; admins moderate and operate. Public reads resolve the viewer but never require login; every write names its guard explicitly.
• Ownership first: edits are owner-only, deletes are owner-or-admin, checked by one helper comparing resource owner to actor. Admins can remove abuse but cannot rewrite someone else's words.
• Seniority among admins: a junior admin (later registration) cannot mutate a more senior admin's account or content. Every per-user admin mutation is gated by one seniority predicate, denials are audited, and the gate is server-side so it also covers agent tools. One primary admin (the first admin account) is the single exception with full visibility and full control, including private-by-admin content and restricted downloads.
• Visibility predicates are single functions, never inline checks: e.g. can_view(project, user) decides private-content access in one place (owner sees own, admins see members' private rows, but an admin's own private rows stay visible only to that admin except for the primary admin). Container and workspace isolation get their own stricter predicates on the same pattern.
• Machines authenticate as users: every account holds an API key, plus issuable scoped tokens; the same guards apply regardless of credential type, so the REST API cannot do what the UI forbids.
• Agents act as the asking user: tool auth mirrors route guards, destructive tools add confirmation gates, and every admin tool passes through the same seniority and audit path as the web UI.

Rule: if a permission check appears twice in different words, one of them is a bug. Hoist it into the single predicate and call it from both places.

Hosting example: guest (pricing, status), customer (own servers and invoices), support (view customers' servers, restart but not delete; junior support cannot touch senior staff accounts), owner (full access plus backup downloads and billing refunds). One can_view_server / can_manage_server pair gates panel, API, and assistant tools together.

19. Replication checklist for your hosting site

Use this order; it is the data-flow order DevPlace follows:

1. Scaffold: app factory, config with one data root, shared templates, response helper, auth choke point, middleware stack.
2. Data: users, sessions, settings, then one domain (products). Add soft-delete columns, batch helpers, idempotent init.
3. Server: public reads first (HTML+JSON via one handler), then member writes with guards, then admin views.
4. View: base template, tokens, partials, one JS utility set. Server-render listings and detail pages.
5. Safety: reportable/private classification, ownership checks, confirm gates, audit events.
6. Automation + docs: tool action per user-askable operation, API doc entry per endpoint, SEO context per public page.
7. Verification: import gate, lint touched files, full three-tier suite green, plus the four-layer money/state/race procedure where it applies.
8. Ops: venv targets, CI serial suite, prune/clear CLIs, backups, production-DB guard, deploy then verify live.

20. What to copy literally vs adapt

Copy literally: single data root, soft-delete columns and filters, batch loaders, one response helper, one auth resolver, test tiers and temp-DB server fixture, prune/clear per domain, audit-never- raises, versioned static URLs.

Adapt to hosting: table list (customers, plans, servers, invoices, tickets, status incidents), provisioning job services, payment and registrar integrations behind the one outbound client, usage metering and quota rules, status page and notifications.

If you keep the constraints and the verification loop, the domain can change completely and the quality transfers with it.

Doc history

Version	Date	Change
1.0.0	2026-10-11	Created: philosophy, stack, layout, data, pipeline, frontend, jobs, testing, verification, ops, security, docs, checklist.
1.1.0	2026-10-11	Added development flow, workflows/automation, and agents sections.
1.2.0	2026-10-11	Added documenting and roles/permissions sections.
1.2.1	2026-10-11	Added doc version header and history per quality_versioning.md.