# Multi-worker and concurrency DevPlace runs safely under multiple Uvicorn workers. This page covers the model, what is safe by design, and the rules that keep it safe. See also [Production overview](/docs/production.html), [Deploy and update](/docs/production-deploy.html), and [nginx and networking](/docs/production-nginx.html). ## The model `uvicorn --workers N` starts **N independent OS processes**. They do not share memory - only the filesystem (the SQLite database and lock files). Anything held in a module-global (a cache, a counter) exists separately in each worker, so an in-process `clear()` is invisible to the others. Every cross-worker mechanism below is therefore built on the shared database or a file lock. ## Safe by design - **Background services** (news, bots, Devii hub) run in exactly one worker, chosen by an exclusive `fcntl.flock` on `devplace-services.lock`. The others skip them. News is never fetched twice; bots never double-run. - **Devii WebSocket** is served only by the lock-owning worker; other workers close with code 1013 and the auto-reconnecting client lands on the owner. The 24h cost cap reads the database ledger, not memory. See [Architecture and sessions](/docs/devii-architecture.html) for the hub and session model behind this. - **Sessions, API keys, cost ledger, tasks, lessons** are all database-backed and shared. - **SQLite** runs in WAL mode with a 30s busy timeout, allowing concurrent readers and a serialized writer across processes. ## Cross-worker cache invalidation The in-process caches (`_user_cache` in `utils.py`, `_settings_cache` in `database.py`) are fast but per-process. To keep them correct across workers, invalidation is coordinated through a `cache_state` table of named integer versions: - A mutation that must invalidate a cache calls `bump_cache_version(name)` - an atomic `UPDATE ... version + 1` in the shared table. - Before serving from a cache, the worker calls `sync_local_cache(name, cache)`, which reads the table's version and compares it to the version its local cache was built against; if they differ, it clears and rebuilds the local cache from the database. The version read is an uncached indexed primary-key lookup on the tiny `cache_state` table, so invalidation is immediate across workers. A logout, ban, role change, or settings save in one worker is therefore honored by every other worker on its next request, while the common path stays a cheap in-memory hit. The named versions in use are `auth` (covers `_user_cache`: logout, deactivation, role and password changes), `settings` (`_settings_cache`: every `set_setting` and the admin settings save), `relations` (block/mute lists), `customizations` (per-user CSS/JS), `notif_prefs` (notification preferences), `gateway_routing` (AI provider/model routes), and `admins` (the cached admin-user set behind `get_admin_uids`/`get_primary_admin_uid`, invalidated by `invalidate_admins_cache()` on every role change). Caches whose value is deterministic or only cosmetically stale do not use a version: the docs prose markdown is memoized on its static source, and the guest home page blocks use a short 60-second time-to-live. ## Rate limiting The rate limiter counts per IP in a per-process structure, so N workers would otherwise allow N times the configured limit. Each worker instead enforces `ceil(limit / DEVPLACE_WEB_WORKERS)`, so the aggregate across workers matches the admin-configured value without a database write on every request. `DEVPLACE_WEB_WORKERS` must equal the actual `--workers` count; `make prod` derives both from the box core count (`nproc`, override with `make prod WEB_WORKERS=N`), while the Docker image pins both to 2; it defaults to 1 for single-worker development. ## First-boot serialization Two startup actions are unsafe to run concurrently from a cold state, so both are serialized with a file lock: - **Database initialization.** Every worker runs `init_db()` at startup; the seed-if-missing inserts have a check-then-insert race that could duplicate rows on a fresh database. Startup wraps `init_db()` in a blocking `flock` on `devplace-init.lock`, so one worker initializes while the others wait, then run the idempotent path. - **VAPID key generation.** If the push keys are absent, concurrent workers could each generate a different keypair and overwrite each other, breaking push. `ensure_certificates()` early-returns when the keys exist, and otherwise generates them under a `.vapid.lock` `flock`, so exactly one worker creates the keypair. ## Preferred rules 1. **Never trust an in-process cache to be authoritative across workers.** A module-global cache is per-process; design for N copies. 2. **Invalidate shared caches through `cache_state`,** not a local `clear()`. Add a `bump_cache_version(name)` at every write path and a `sync_local_cache(name, cache)` before every read path. Reuse the existing `auth` and `settings` names or add a new one. 3. **Keep authoritative state in the database** (sessions, caps, counters that must be exact), and treat memory only as a short-TTL accelerator. 4. **Make any rate or quota that must hold globally either database-backed or worker-count-aware.** If it is worker-count-aware, read `DEVPLACE_WEB_WORKERS` and divide; keep that variable in lockstep with `--workers`. 5. **Serialize one-time, first-boot side effects with an `flock`,** and make the work itself idempotent (exists-checks, `INSERT OR IGNORE`), so a second worker is always a safe no-op. 6. **A single background-services owner is the rule, enforced by `devplace-services.lock`.** Anything that must run once (schedulers, fetchers, hubs) goes behind `service_manager.owns_lock()`, never per-worker.