devplacepy/devplacepy/templates/docs/architecture-backend.html

<div class="docs-content" data-render>
# Backend and request pipeline

How a request flows through middleware to a router, how the database layer is built and why it is synchronous, how caches stay correct across workers, and how the response, auth, and service layers fit together. See also [Architecture overview](/docs/architecture.html) and [Conventions and rules](/docs/architecture-conventions.html).

> Audience: administrators and maintainers. These pages are hidden from members and guests, in the sidebar, the search index, and the documentation export.

## Application setup

`main.py` builds the FastAPI app, mounts `/static`, registers every router with its prefix, installs the middlewares, and registers global 404, 500, and `RequestValidationError` handlers that render `error.html` through the shared `templates` instance.

## Middleware

Seven HTTP middlewares run as a stack around every request, listed outermost first. `response_timing` is the outermost, `refresh_db_snapshot` the innermost, and a `GZipMiddleware` (responses over 512 bytes) wraps the whole stack on top:

| Middleware (outermost first) | Responsibility |
|------------|----------------|
| `response_timing` | Stamps `request.state.request_start` (a `perf_counter()` reading) and sets an `X-Response-Time: <ms>ms` header on every response (the full request total). |
| `track_presence` | For the resolved current user on every non-asset request, calls `presence.touch(uid)` (a throttled `last_seen` write). |
| `maintenance_middleware` | When `maintenance_mode="1"`, returns a 503 `error.html` for non-admins, but always allows `/static`, `/avatar`, `/auth`, `/admin`, `/openai`, and admin users, so an operator can never lock themselves out. |
| `rate_limit_middleware` | Per-IP limit for mutating methods (POST/PUT/DELETE/PATCH), held in an in-process `defaultdict`. Reads `rate_limit_per_minute` / `rate_limit_window_seconds` from `site_settings`, floored to `max(1, ...)`, and is worker-count-aware. `/openai` and `/xmlrpc` are excluded. |
| `add_security_headers` | Sets `X-Content-Type-Options: nosniff` always, `X-Robots-Tag: index, follow` unless the handler already set it (so pages can opt out of indexing), plus HSTS, `Referrer-Policy`, and a CSP whose `frame-ancestors` allows `'self'` and the workspace tunnel domain (except the `/p/` ingress, which sets no CSP at all), and no-store cache headers on `/admin`. Framing is controlled by `frame-ancestors` alone - no `X-Frame-Options` is sent, because it cannot express an allow-list and browsers honour the stricter of the two. |
| `await_pending_corrections` | After the handler runs, awaits any pending AI correction/modifier futures parked on `request.scope` (sync apply mode). |
| `refresh_db_snapshot` | Calls `refresh_snapshot()` so each request sees committed data. |

## Routing

Routers in `routers/` form a **directory tree that mirrors the endpoint (URL) path**, the same convention as the test suite. A single-resource domain is one flat file (`feed.py`, `posts.py`, ...); a multi-sub-resource domain is a **package directory** split one file per sub-resource (a distinct noun under the domain), never one file per endpoint. Each leaf exports its own `router = APIRouter()` keeping the exact path strings, and the package `__init__.py` aggregates them with `include_router`. A package exposes `.router` like a module, so `main.py` mounts each domain at its prefix unchanged:

```
/auth   /feed   /posts   /comments   /projects   /profile
/messages   /notifications   /votes   /reactions   /bookmarks
/polls   /avatar   /follow   /leaderboard   /admin   /issues
/gists   /news   /uploads   /openai   /devii
```

Multi-sub-resource domains are packages (`auth/`, `profile/`, `issues/`, `docs/`, `admin/`, `projects/`, `tools/`, `devrant/`, `dbapi/`, `game/`); single-resource domains stay flat files. The whole `/projects` tree (project CRUD + `files.py` + the `containers/` subpackage) and the whole `/admin` tree (every admin sub-resource plus the folded-in `services` and `containers`) each mount from a single package. `seo.py`, `push.py`, `relations.py`, and the `docs/` package mount without a prefix.

**Aggregation rules.** A leaf that owns the domain's collection-root (`""`) route is the package's base router (FastAPI rejects an empty path included under an empty prefix), so the package `__init__` imports that leaf's `router` and includes the rest onto it. A router folded in from a deeper mount keeps its own paths and is included with a sub-prefix (`admin/__init__` includes `services.router` with `prefix="/services"`). Package-private helpers shared by leaves live in `_shared.py`. Within any router, **specific paths must be declared before catch-alls**: `/{username}/followers` before `/{username}`.

## Lifecycle and services

The app uses a `lifespan` async context manager (not the deprecated `@app.on_event` hooks). On startup it runs `init_db()` and certificate setup under an exclusive init lock (`fcntl.flock` on `devplace-init.lock`) so multiple workers serialize first-boot side effects. It then registers every background service (`NewsService`, `GatewayService`, `DeviiService`, the job services, the pub/sub and relay services, `ContainerService`, and more). On shutdown it calls `service_manager.shutdown_all()` and stops the background queue.

Background services follow a **single-owner model**. Each worker tries to acquire an exclusive non-blocking lock on `devplace-services.lock`; exactly one wins, becomes the lock owner, and calls `service_manager.supervise()`. The others run no services. This is why `/devii/ws` is served only by the lock-owning worker (a non-owner worker closes with the application code 4013, which the client treats as a silent fast retry until it lands on the owner; the disabled-service path uses the standard 1013), and why news is never fetched twice. Setting `DEVPLACE_DISABLE_SERVICES=1` skips all of this (the test conftest does so).

## Database

SQLite via `dataset.connect`, tuned on every connection:

```
PRAGMA journal_mode=WAL
PRAGMA synchronous=NORMAL
PRAGMA busy_timeout=30000      -- 30s
PRAGMA cache_size=-8000        -- 8MB
PRAGMA temp_store=MEMORY
PRAGMA mmap_size=268435456     -- 256MB
```

**SQLite is synchronous by design and will never be made async. This is a hard rule.** The synchronous `dataset`/SQLite layer is called directly inside the async route handlers. With a local file in WAL mode, a 30s busy timeout, an 8MB page cache, and a 256MB mmap, queries are sub-millisecond and never meaningfully stall the event loop. Do not convert the database layer to async, do not wrap DB calls in a threadpool / `run_in_executor` / `to_thread`, and do not raise synchronous DB access as a performance or blocking-I/O concern.

Other database rules:

- **`init_db()` is idempotent.** Every index is `CREATE INDEX IF NOT EXISTS` wrapped in try/except, and seed defaults insert only when the key is absent.
- **`dataset.find()` does not accept raw SQL.** Use kwargs for equality, dict syntax (`{">=": value}`) for comparisons, or `table.table.columns.<col>.in_([...])` for IN clauses. The raw-SQL escape hatch is `db.query("... :p", p=val)` with named params as kwargs.
- **Guard raw SQL with `if "table" in db.tables:`** because `dataset` only auto-creates a table when something is first inserted.
- **Pagination is cursor-based** (`paginate`), fetching one extra row to detect `has_more`, not offset-based.
- **Slug + UUID lookup** via `resolve_by_slug(table, slug)`: resources accept either their slug or the bare UUID.
- **Avoid N+1.** Use the batch helpers (`get_users_by_uids`, `get_comment_counts_by_post_uids`, `get_vote_counts`, `load_comments`, `get_attachments_by_type`) in listing routes instead of per-row lookups.

## Caching and multi-worker consistency

In-process TTL caches accelerate hot reads but are **never authoritative**: `_settings_cache` (60s) for `site_settings` and `_user_cache` (300s) for authenticated users. Because the app runs multiple workers, caches are invalidated cross-worker through the `cache_state` version table. A read calls `sync_local_cache(name, cache)` to compare its local version against the database version, clearing the local cache if it is stale; a write calls `bump_cache_version(name)`. Settings use the `settings` name, the user cache the `auth` name. Authoritative state lives in the database; memory is only a short-TTL accelerator.

## Models, schemas, and responses

- **`models.py`** holds the Pydantic `Form` input models. Routers declare `data: Annotated[SomeForm, Form()]`; FastAPI validates, and the global `RequestValidationError` handler re-renders auth pages with messages (400) or redirects other routes.
- **`schemas/`** (a package) holds the `*Out` response projections. They ignore extra keys, so the full context dict can be passed through, but only declared fields reach the JSON. `*Out` models never expose `password_hash`, `email`, or `api_key`.
- **`responses.py`** provides content negotiation. `wants_json(request)` is true for `Accept: application/json` or `Content-Type: application/json`. Page GETs return `respond(request, "x.html", ctx, model=XOut)` (HTML or validated JSON); action POSTs return `action_result(request, url, data=...)` (a 302 redirect or `{ok, redirect, data}` JSON).

## Auth

Session cookies named `session` hold a 64-char hex token from `secrets.token_hex(32)`. Passwords are hashed with `pbkdf2_sha256` via passlib. A request is resolved to a user through multiple paths tried in order: the session cookie, `X-API-KEY` / `Authorization: Bearer` headers (matched against the user's permanent API key, then DevRant tokens, then expiring access tokens from `POST /auth/token`), and HTTP Basic auth. `get_current_user(request)` returns a user dict or `None` (public reads); `require_user(request)` redirects guests; `require_admin(request)` redirects non-admins. All POSTs are guarded. A per-request memoization plus the `_user_cache` TTL cache keep auth resolution cheap.

## Services framework

`BaseService` provides the async run loop, typed `ConfigField` definitions, state persisted to `service_state` (status, last/next run, log deque, metrics), and admin commands (run, clear). `ServiceManager` is a singleton that registers, supervises, and stops services and tracks lock ownership. New services extend `BaseService`, override `async def run_once(self)`, and register in `main.py`; they then appear on `/admin/services`. `GatewayService` is the single source of truth for AI: every other AI consumer routes through it, so usage is attributed and limitable per owner.
</div>