This file documents devplacepy/database/ - the dataset/SQLite data layer, indexing rules, and the project-wide soft-delete model. Claude Code loads it automatically whenever a file under this directory is read or edited.

## Database engine and dataset library

SQLite via `dataset` with these pragmas on every connection:

```python
PRAGMA journal_mode=WAL;      -- concurrent readers + writers
PRAGMA synchronous=NORMAL;     -- safe with WAL mode
PRAGMA busy_timeout=30000;     -- wait 30s instead of failing on lock
PRAGMA cache_size=-8000;       -- 8MB page cache
PRAGMA temp_store=MEMORY;      -- temp tables in memory
```

Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}, "poolclass": NullPool}, on_connect_statements=[...])`. In addition to the pragmas above, the connection is tuned with a 30s busy timeout, an 8MB page cache, and a 256MB mmap.

**`poolclass=NullPool` is load-bearing - never revert it to SQLAlchemy's default `QueuePool` (caused a production outage).** `dataset.Database.executable` caches ONE DBAPI connection per OS thread ID **forever** and never returns it to the pool except via `db.close()`, which nothing in this codebase calls (`dataset/database.py`: `self.connections[tid] = self.engine.connect()`). That is fine as long as the same handful of threads ever touch the DB - but FastAPI runs every sync route dependency (`get_setting` and friends, hit on nearly every request) through `anyio.to_thread.run_sync`, whose worker pool scales up and recycles threads elastically under load, and container sync (`asyncio.to_thread`) adds more. Each new thread's first query permanently claims one pool slot. With the default bounded `QueuePool` (`pool_size=5, max_overflow=10` = 15 total), a burst of concurrent load creates enough new threads that the pool fills for good within minutes, and every request thereafter - including the Docker healthcheck's own probe - blocks the full 30s pool timeout and then raises `sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached`, wedging the whole app (nginx waits on an app that is waiting on itself; every external caller sees a bare connection timeout, not an HTTP error). `NullPool` removes the artificial ceiling: each `engine.connect()` opens a real, unpooled SQLite connection, so the existing "one connection cached per thread forever" behavior just works, exactly as WAL mode is designed to support. Never pass `pool_size`/`max_overflow` alongside `NullPool` (SQLAlchemy rejects them). Do not "fix" the underlying thread churn instead - that means touching the sync-dependency/threadpool model, which the hard rule below forbids.

`init_db()` is idempotent - every index is created via `CREATE INDEX IF NOT EXISTS` (through the `_index()` helper, wrapped in try/except so it is safe to run on every startup regardless of table state), and seed defaults for `site_settings` only insert when the key doesn't already exist.

**`init_db()` also ensures the full column set of any table that code filters on** (`news`, `news_sync`, `attachments` use `get_table(name)` + `create_column_by_example` before their indexes). This is mandatory, not optional: `dataset` gives a lazily-created table ONLY the columns of its first insert, so a partial insert from a CLI tool/test/maintenance script would otherwise create the table with a reduced schema, and the long-running server caches that stale schema - making later queries on the missing column 500 with `no such column`. When adding a new filtered/indexed column, add it to the matching `init_db()` ensure-block:

```python
news = get_table("news")
for column, example in (("uid", ""), ("status", ""), ("synced_at", ""), ("external_id", ""), ...):
    if not news.has_column(column):
        news.create_column_by_example(column, example)
_index(db, "news", "idx_news_status", ["status"])   # now safe - column exists
```

Do this with `get_table(name)` (NOT `if name in db.tables`): `get_table` + `create_column_by_example` creates the table if it is absent, so the schema is guaranteed regardless of which process or which insert touches the table first. Symptom to watch for: a page renders fine in isolation but shows empty data or 500s only after a CLI/maintenance test runs earlier in the suite (it created the table partially).

**This applies to `users` too, and an `if "users" not in db.tables: return` guard silently defeats it.** `backfill_api_keys()` is the `users` ensure-block (every non-signup column: `api_key`, `last_seen`, the AI correction/modifier settings, `avatar_seed`, `award_count`/`last_award_at`/`last_award_slug`/`last_award_uid`, ...). It used to early-return when `users` was absent, which is exactly the state on a **brand-new database**: `init_db()` runs before the first signup, so the ensure-block was skipped entirely, and the first `/auth/signup` then created `users` with only the columns of that INSERT. Every ensured-but-unwritten column was therefore missing from the long-running server's reflected metadata, so `users.find_one(...)` returned rows without them **for the whole process lifetime** - the feature reading them looked simply switched off (this is what made the awards tab, the prominent-award banner, and the avatar award badge invisible on a fresh DB, and it is invisible in production only because the columns happen to exist from an older boot). It now calls `get_table("users")` unconditionally, so the table is born with the full ensured column set. Never reintroduce a `db.tables` guard in front of a column-ensure block.

The `_index(...)` helper supports `where=` (partial) and `unique=` indexes; every table with a `uid` column gets a UNIQUE `idx_<table>_uid`, soft-delete tables get a PARTIAL `idx_<table>_trash` (`WHERE deleted_at IS NOT NULL`) and NEVER a bare `deleted_at` index (it mis-steers the planner on live reads), and "live newest-first" listings need a composite/live-partial index that carries the sort column (see "Indexing conventions" below). `init_db()` finishes with `ANALYZE`/`PRAGMA optimize`. Verify any index change with `EXPLAIN QUERY PLAN`.

**SQLite is synchronous by design and will never be made async (hard rule).** The `dataset`/SQLite layer is synchronous and is called directly inside the async route handlers. This is intentional and more than fast enough for this platform: the database is a local file tuned with WAL, `synchronous=NORMAL`, a 30s busy timeout, an 8MB page cache, and a 256MB mmap, so queries are sub-millisecond and never meaningfully stall the event loop. Do NOT convert the database layer to async, do NOT wrap DB calls in a threadpool/`run_in_executor`/`to_thread`, and do NOT raise the synchronous DB access as a performance, blocking-I/O, or "requests aren't fully async" concern. This is a settled, sophisticated design choice and is not open for revisiting.

Runtime config lives in `site_settings`, read via `get_setting(key, default)` / `get_int_setting(key, default)` (60s TTL cache, invalidated cross-worker via the `cache_state` version table - `get_setting` calls `sync_local_cache("settings", ...)`, writes call `bump_cache_version("settings")`; the `_user_cache` in `utils.py` uses the same primitive under the `auth` name). Consumers always pass the production default to `get_setting`, so behavior is correct even before the row exists. Numeric operational values are floored at the call site so an invalid `0` can't lock out writes or stall a service. Booleans are stored as `"0"`/`"1"` and rendered as `<select>` (not checkboxes) because the settings save handler skips empty form values - an unchecked checkbox could never be turned off. See "Site settings" and "Operational settings" below for the full key registry.

Batch helpers (`get_users_by_uids`, `get_comment_counts_by_post_uids`, `get_vote_counts`, `load_comments`, `get_attachments_by_type`) exist specifically to avoid N+1 queries - use them in feed/listing routes instead of per-row lookups.

## Dataset rules (hard-learned)

**`find()` does NOT accept raw SQL strings.** It takes keyword arguments for equality filters, dict comparison operators, or SQLAlchemy column expressions.

```python
# WRONG - causes 500 Internal Server Error:
table.find("created_at >= :start", {"start": today})
table.find(text("created_at >= :start"), start=today)

# CORRECT - dict comparison syntax:
table.find(created_at={">=": today})

# CORRECT - keyword equality:
table.find(country="France")

# CORRECT - SQLAlchemy column expression for IN clause:
table.find(table.table.columns.user_uid.in_(["uid1", "uid2"]))

# CORRECT - multiple equality filters combined:
table.find(topic="devlog", user_uid=some_uid)
```

**`update()` requires a key column list as second argument.** The first dict contains all fields including the key column.

```python
table.update({"uid": user_uid, "bio": "new bio"}, ["uid"])
```

**`db.query()` accepts raw SQL with named params as keyword arguments:**

```python
db.query("SELECT * FROM posts WHERE topic = :t", t="devlog")
# NOT: db.query("...", {"t": "devlog"})
```

**`db.query()` WRITES DO NOT AUTO-COMMIT - wrap any `db.query` INSERT/UPDATE/DELETE in `with db:` (load-bearing, caused a production deadlock).** The dataset table API (`table.insert`/`update`/`delete`) calls `db._auto_commit()` internally, but `db.query()` does NOT. SQLAlchemy 2.x autobegins a transaction on first `execute`, so a raw `db.query` write leaves an open transaction holding the SQLite write lock until that thread's connection next commits. On the request/loop thread this is masked (the next table op's `_auto_commit` flushes it), but on a **background-queue or `run_in_executor` worker thread** the thread goes idle still holding the lock, and EVERY subsequent write app-wide blocks for the 30s busy-timeout then fails `database is locked` - a full deadlock. Always commit raw writes:

```python
with db:                                   # commits + releases the write lock on exit
    db.query("INSERT INTO t (...) VALUES (:a) ON CONFLICT(...) DO UPDATE SET ...", a=1)
```

Atomic counters (e.g. `add_correction_usage`) must use raw `ON CONFLICT DO UPDATE SET col = col + excluded.col` (the table API cannot increment), so they MUST use the `with db:` wrapper. Prefer the table API whenever an atomic SQL increment is not required.

**Always check `tables` list before raw SQL queries:**

```python
if "comments" not in db.tables:
    return {}  # table doesn't exist yet
```

**Batch queries eliminate N+1 problems.** Use `get_users_by_uids()`, `get_comment_counts_by_post_uids()`, and `get_vote_counts()` from `database.py` instead of per-row lookups in loops.

**`init_db()` MUST create every queried column for any table that code filters on, even if the table is created lazily.** dataset creates a table on its FIRST insert and gives it ONLY the columns in that insert. If any code path can insert a *partial* row before the full schema exists (a CLI tool, a test fixture, a maintenance script), the table is born with a reduced schema and every later query against a missing column throws `sqlite3.OperationalError: no such column: X` (a 500), or - for an indexed column - logs a `Could not create index ... no such column` warning at startup. This is worsened by **cross-process metadata staleness**: the long-running uvicorn server reflects a table's columns once and caches them, so a column another process adds afterward is invisible to the server until it reconnects. The defence is to make `init_db()` ensure the complete column set up front, exactly like the existing `news`, `news_sync`, and `attachments` blocks (see the code example under "Database engine and dataset library" above). When you add a NEW column that any query filters/indexes, add it to the `init_db()` ensure-block too - never rely on the first insert to define it.

## Indexing conventions (the soft-delete planner trap)

`init_db()` owns every index. The `_index(db, table, name, columns, *, where=None, unique=False)` helper builds the DDL; it supports **partial** indexes (`where=`) and **unique** indexes, and wraps each `CREATE`/`DROP` in `with db:` (DDL via `db.query` does not auto-commit - see "Dataset rules" above). Three load-bearing rules learned from an `EXPLAIN QUERY PLAN` audit against production data:

- **Every table with a `uid` column gets `idx_<table>_uid` (UNIQUE).** `dataset` makes its own `id` autoincrement PK and does NOT key `uid`, so `find_one(uid=...)`, `resolve_by_slug`, `soft_delete`, and `table.update({...}, ["uid"])` full-SCAN without it. `init_db()` loops `for table in db.tables: _uid_index(db, table)` (falls back to a non-unique index if a UNIQUE build ever fails on legacy duplicate data). New tables are covered automatically.

- **NEVER index the bare `deleted_at` column - use a PARTIAL trash index `WHERE deleted_at IS NOT NULL`.** `ensure_soft_delete_columns` creates `idx_<table>_trash ON (deleted_at) WHERE deleted_at IS NOT NULL` (and drops any legacy full `idx_<table>_deleted`). A full `deleted_at` index is a planner hazard: the column is one giant `NULL` bucket plus many unique delete-timestamps, so `sqlite_stat1` mis-estimates `deleted_at IS NULL` as returning ~2 rows and the planner picks that index for live reads, then `USE TEMP B-TREE FOR ORDER BY` to sort the whole live set (the global feed was doing exactly this, with 82% of posts soft-deleted). The partial index serves the admin Trash view (`deleted_at IS NOT NULL`) cheaply and stops poisoning live `IS NULL` queries.

- **For "live, newest-first" listings add a composite or live-partial index that includes the sort column.** A `WHERE deleted_at IS NULL ORDER BY created_at` query needs the ordering in the index or it filesorts. Posts use a partial `idx_posts_live_created ON (created_at) WHERE deleted_at IS NULL` (feed) plus `idx_posts_user_created (user_uid, created_at)` (profile); comments use `idx_comments_target_created (target_type, target_uid, created_at)`; votes use `idx_votes_user_target (user_uid, target_uid)` (the per-user "my_vote" check on every card); notifications/gists/projects use `(user_uid, created_at)`; follows use `idx_follows_follower_created (follower_uid, created_at)` + `idx_follows_following_created (following_uid, created_at)` (the followers/following tabs sort newest-first; the legacy single-column follower/following indexes were dropped as redundant prefixes). All were verified to drop the `USE TEMP B-TREE FOR ORDER BY` step.

- **Index the non-`uid` lookup keys too, not just the sort/owner columns.** A demand-vs-supply audit added the last missing single-key lookups: the `resolve_by_slug` hot path filters `slug` on content detail pages, so posts/gists/news/projects each get `idx_<table>_slug (slug)`; `get_setting`/`set_setting` filter `key`, so `idx_site_settings_key (key)`; the container store's `find_one(slug=)`/`find_one(name=)` fallbacks get `idx_instances_slug`/`idx_instances_name`. The DM thread load `find(sender_uid=, receiver_uid=)` gets the covering composites `idx_messages_conversation (sender_uid, receiver_uid)` + `idx_messages_conversation_rev (receiver_uid, sender_uid)` (the read-flag `UPDATE` uses the reverse); the badge-has check gets `idx_badges_user_name (user_uid, badge_name)`; the admin user list `ORDER BY -created_at` gets `idx_users_created_at (created_at)` (the existing `(role, created_at)` cannot serve a full-table created_at sort). All are non-unique so `_index` always creates them even if legacy duplicate data exists. Column sets already resolved to ~1 row by an existing prefix index (votes `+target_type`, game_quests `+kind`, poll_options `position`) are intentionally left uncovered - a trailing column there only adds write cost.

`init_db()` ends with `_refresh_query_planner_stats()`: a one-time `ANALYZE` when `sqlite_stat1` is absent, then `PRAGMA optimize` every boot (both inside `with db:`). Validate any index change with `EXPLAIN QUERY PLAN` against `data/devplace.db` (or a copy) - a correct plan reads `SEARCH ... USING INDEX <name>` with no temp b-tree.

## Startup backfills must converge (hard rule)

`init_db()` runs inside `lifespan` under an exclusive `init_lock()`, **before the worker accepts a single request**, and every uvicorn worker runs it in turn. Time-to-serving is therefore `workers x init_db`, so anything added there is paid N times on every boot and every deploy. Two rules follow:

- **A backfill must be able to finish.** A backfill selects the rows that still need migrating and must leave them *not* selected afterwards. If a row can stay in the candidate set after a successful pass, the "one-time migration" is really a permanent per-boot scan that grows with the table. `_backfill_gamification` had exactly this bug: it selected `users.find(xp=0)` and then ran the full milestone sweep over *every* one of them, but a user with no content is awarded no XP, so they stayed at `xp=0` and were re-swept forever. At 7814 such users that was 36s of the 37.5s boot - about 140k queries that provably could not award anything, on every worker, on every restart.
- **Never fan a per-row query out over a whole table at boot.** Compute the candidate set with a few set-based `GROUP BY`/`DISTINCT` queries first, then do per-row work only for rows that survive. `_milestone_candidates()` is the pattern: one `SELECT DISTINCT` per milestone source table (`MILESTONE_SOURCES`), unioned into a set, intersected with the pending users. A user absent from all of those tables scores 0 on every milestone metric and the lowest threshold is 5, so skipping them cannot change any award - verified by diffing the full `badges` table between the full sweep and the narrowed one (identical, 9800 rows, 22.6s -> 0.9s).

Watch for the same shape in `dataset` internals: `db.tables` is a live SQLAlchemy reflection, not a cached attribute. `get_user_stars` does one `in db.tables` check per `STAR_TARGETS` entry, so a per-user loop calling it re-reflects the whole table list on every iteration - 78667 reflections costing 17.7s in the profile above. Hoist `db.tables` into a local when looping.

Profile with the real database before and after any change here (`cProfile` around `init_db()` against a copy of `data/devplace.db`); a synthetic or empty DB hides every one of these costs.

## Project-wide soft delete (hard rule)

Attachments (see "Profile media gallery and soft-deleted attachments" below) are one instance of a platform-wide model: **every removal is a soft delete; only garbage collection is a hard delete.** Every removable row carries two columns, `deleted_at` (ISO timestamp) and `deleted_by` (actor uid, or `system`). A live row has `deleted_at = NULL`, and every list/count read filters `deleted_at IS NULL`.

- **Tables (`database.SOFT_DELETE_TABLES`):** posts, comments, gists, projects, news, news_images, project_files, attachments, votes, reactions, bookmarks, follows, poll_votes, polls, poll_options, sessions, instances, instance_schedules, devii_conversations, devii_tasks, devii_lessons, devii_virtual_tools, user_customizations, project_forks. `init_db` loops `ensure_soft_delete_columns(t)` for each (adds both columns + `idx_<t>_deleted` / the partial trash index described above). The Devii task/lesson/virtual-tool stores run on their own DB handles, so their `_ensure_indexes` adds the columns there too.
- **Every INSERT into a soft-deletable table MUST write `deleted_at: None, deleted_by: None`.** `dataset.find(deleted_at=None)` on a table that lacks the column matches NOTHING (a false predicate), silently hiding all rows - the born-live insert is what creates the column. Add the pair to any new insert.
- **Central helpers (`database/`):** `soft_delete(table, deleted_by, *, stamp=None, **criteria)` (equality), `soft_delete_in(table, column, uids, deleted_by, *, stamp=None, **extra)` (IN-clause cascade), `restore(table, **criteria)`, `purge(table, **criteria)` (real delete), `list_deleted(table, page)` / `count_deleted(table)` (trash listings), and the event helpers `restore_event(stamp)` / `purge_event(stamp)` that act across ALL tables sharing one `deleted_at` stamp.
- **Two generic chokepoints are conditionally filtered:** `resolve_by_slug(table, slug, include_deleted=False)` (detail-page lookups; restore passes `include_deleted=True`) and `paginate(table, ...)` (auto-appends `deleted_at IS NULL` when the table has the column and the caller did not pass `deleted_at`). `seo._collect` does the same for the sitemap. Read filters were threaded through every batch helper, analytics/activity/leaderboard UNION, feed/profile/listing route, and store; never re-introduce an unfiltered read of a soft-deletable table.
- **Any new read** (find/count/query) of a soft-deletable table MUST filter `deleted_at IS NULL`. Use the central helpers/chokepoints instead of inline deletes.
- **Toggles revive, they do not duplicate.** votes/reactions/bookmarks/follows/poll_votes look up the physical row regardless of `deleted_at`: toggle-off stamps `deleted_at`; re-toggle clears it on the same row. Counts/state reads filter `deleted_at IS NULL`.
- **Cascades share one stamp.** `content.delete_content_item` soft-deletes the item plus its comments, votes, engagement, project files, fork relations, and attachments with one shared `stamp` and `deleted_by = actor`. That timestamp identifies the whole event, so `restore_event`/`purge_event` reverse or finalize it atomically.
- **Delete authorization is owner-OR-admin, enforced on the endpoint** (`is_owner(...) or is_admin(user)`): posts/gists/projects (`content.delete_content_item`, also rejects a missing item), `comments.delete_comment`, `project_files.project_file_delete`, `media.delete_media`, `uploads.delete_attachment_route`; news (`admin_news_delete`) is admin-only. Because the check is on the endpoint, one rule covers the human UI and **Devii** at once - Devii only ever calls the platform API, authenticated as the signed-in user, so an admin's Devii may soft-delete any member's content and a member's is refused with no agent-side logic. The matching `delete_*` Devii catalog tools stay `requires_auth` (not `requires_admin`) so a member can still delete their own, and every one is in the dispatcher's confirmation gate (`CONFIRM_REQUIRED`) so a delete only runs on a repeat call with `confirm=true`. Standalone `comment` and uploaded-`attachment` deletes are soft like the rest (`soft_delete` cascade / `soft_delete_attachment`); the only attachment hard delete is the admin `/admin/media/{uid}/purge` and the CLI prune. Any NEW content delete path must reuse this guard, soft-delete, and (for the Devii tool) be added to `dispatcher.CONFIRM_REQUIRED`.
- **What stays HARD (GC / the empty-trash stage):** the async-job sweep + CLI prune/clear, the container metrics ring trim, gateway and Devii usage-ledger retention prunes and quota resets, the expired-session cleanup branch in `utils._user_from_session`, fork-rollback of a half-created project, the news-sync image replacement, and the admin **Purge** action. Logout is a soft delete (auditable via `deleted_by`); only expiry GC is hard.
- **Admin Trash surface:** `/admin/trash` (sidebar **Trash**, `routers/admin/` package, `admin_trash.html`, `AdminTrashOut`) lists soft-deleted rows per table with restore/purge per row. Restore calls `restore_event(row.deleted_at)`; Purge calls `purge_event(...)` and unlinks attachment files / project-file blobs. The attachment-specific `/admin/media` view is unchanged. Admin-only docs: `docs/soft-delete.html` (`admin: True`, Administration section).

## Profile media gallery and soft-deleted attachments

The profile **Media** tab (`/profile/{username}?tab=media`, public) is a paginated grid of every attachment a user uploaded, newest first, across all `target_type`s. Attachments support a **soft delete** so a user can remove one upload without affecting its parent object.

- **One `deleted_at` column** on `attachments` (ISO string, mirrors the `push.py` precedent), ensured idempotently in `init_db` via `create_column_by_example("deleted_at", "")` plus the `idx_attachments_user_created` index. `store_attachment` writes `deleted_at: None`.
- **Soft delete preserves the relation and the file.** `attachments.soft_delete_attachment(uid)` only stamps `deleted_at`; it never touches `target_type`/`target_uid` and never unlinks the file. `restore_attachment(uid)` clears `deleted_at`, so the item reappears on its parent object and in the gallery with zero extra bookkeeping.
- **Three read paths filter `deleted_at IS NULL`** so a soft-deleted item vanishes everywhere (the gallery AND its parent post/project/etc.): `get_attachments`, `get_attachments_batch` (both in `attachments.py`), and `database.get_user_media`. **The hard-delete cascades (`delete_attachments_for`, `delete_target_attachments`) stay UNFILTERED** so permanently deleting a parent object still removes ALL its attachment files, including soft-deleted ones - never add the filter there.
- **Queries:** `database.get_user_media(user_uid, page)` (linked, non-deleted, newest first; each item gets a `target_url` via `resolve_object_url`) and `database.get_deleted_media(page)` (the admin trash, joined to uploader username).
- **Authorization:** `POST /media/{uid}/delete` (`routers/media.py`) is owner-or-admin (`attachment["user_uid"] == user["uid"] or is_admin(user)`); `POST /media/{uid}/restore` and `POST /admin/media/{uid}/purge` (the only hard delete, via `delete_attachment`) are admin-only (`routers/admin/` package, sidebar **Media** -> `/admin/media`). The tab itself is public.
- **Frontend:** `_media_gallery.html` reuses the `_attachment_display.html` type branches and the `dp-lightbox` contract (`data-lightbox`/`data-full`). The delete button carries `data-media-delete` + `data-confirm`; `ModalManager.initConfirmations` shows the confirm and `MediaGallery.js` (`app.mediaGallery`) does the optimistic `Http.send` delete, fades the tile, and toasts. A `<noscript>` form is the no-JS fallback. Grid styling is `static/css/media.css`.
- **Devii:** `list_media` (public) and `delete_media` (auth, in `CONFIRM_REQUIRED`) in the catalog.
- **Docs visibility (deliberate):** members and guests must never be told this is a *soft* delete. The public prose page `docs/media-gallery.html` (General) and the member-facing `media-delete` API endpoint (Profiles group) describe deletion as a plain "remove" - no soft-delete, restore, trash, or purge language. All moderation mechanics live on the admin-only `docs/media-moderation` prose page (`admin: True`) and in the admin API group (`media-restore`, `admin-media`, `admin-media-purge`, all `auth="admin"`), which `docs_search` excludes from member results and `routers/docs/` package 404s for non-admins. Because `docs_search._strip` keeps the text *inside* `{% if %}` blocks, admin content must live on a separate `admin: True` page, never inline-gated on a public page (a public page may only carry an admin-gated *link*). The member `MediaItemOut` schema omits `deleted_at`; the admin-only `AdminMediaItemOut` adds it.

## Moderation, consent and maturity tables (`database/moderation.py`)

Four soft-deletable tables carry the trust-and-safety layer; the full subsystem is documented in `devplacepy/services/moderation/CLAUDE.md`.

| Table | Shape | Notes |
|---|---|---|
| `content_reports` | `(target_type, target_uid)` + `reporter_uid` + `owner_uid` | The one queue. `owner_uid` is denormalised at insert so the admin list never N+1s. Indexes `(status, created_at)` for the queue and SLA scan, `(target_type, target_uid)` for duplicate detection, `(reporter_uid)`, `(owner_uid)` |
| `moderation_actions` | one row per moderator decision, linked to its report | Queryable moderation state with its own lifecycle - deliberately separate from the append-only audit log, the same way `workspace_flags` is |
| `content_maturity` | `(target_type, target_uid)` -> `level` | Polymorphic age label. Read through the batch helper `get_maturity_by_targets`, never per row. **Absence of a row means `general`**, so nothing needed backfilling |
| `user_consents` | `(owner_kind, owner_id, kind)` | Append-only in effect: withdrawing stamps `withdrawn_at` on the current row and inserts a new one, so the history is provable |

`REPORTABLE_TARGETS` (target type -> table) is the registry every consumer reads, exactly like `VOTABLE_TARGETS`. `UNREPORTABLE_TABLES` is its explicit counterpart: each entry names a soft-deletable table and **why** it carries no reportable content. `tests/unit/database/moderation.py` asserts the two partition `SOFT_DELETE_TABLES`, so a new user-generated table cannot be added without classifying it.

The `users` columns added alongside are ensured in `backfill_api_keys()` like every other non-signup column: `terms_version`, `terms_accepted_at`, `age_band`, `age_declared_at`, `mature_opt_in`, `suspended_until`, `suspension_reason`, `deletion_requested_at`. `mature_opt_in` is normalised from NULL to `0` in the same `with db:` block that fixes the AI-modifier defaults, because it is read as a flag. **No date of birth is ever stored** - only the derived `age_band`.

Two atomic conditional updates protect this data and must never become read-then-write: `queue.claim_open` (report resolution) and `deletion.claim_deletion` (the account-deletion cascade). The latter's precondition is `COALESCE(deletion_requested_at, '') = ''` because the column is SQL `NULL` on rows that predate it - the exact `NULL = 0` trap recorded above.

## Role-based visibility (generic + DRY)

- **One source of truth for role/visibility checks**, registered as Jinja globals in `templating.py` - never hand-roll `user.get('role') == 'Admin'` or `user['uid'] == x['user_uid']` in a template again:
  - `is_admin(user)` (also `utils.is_admin`, reused by `require_admin` and `docs.py`) - admin-only UI.
  - `owns(item, user)` (= `content.is_owner`) - per-item ownership (e.g. each comment). Page-level detail templates keep using the `is_owner` **bool** passed in their context (post/gist/project/profile); do not call `is_owner(...)` as a function - that name is a context bool and shadows globals.
  - `is_self(user, uid)` - "is this me" (profile follow vs edit, leaderboard highlight).
  - `guest_disabled(user)` -> emits ` disabled aria-disabled="true" title="Log in to participate"` for guests (empty for members); `login_hint(user)` -> a small login link. Both return `Markup`.
- **Account enabled/disabled is `database.is_account_active(row)` - never read `is_active` inline.** `is_active` is a nullable column, so a row written before it existed (or by any insert that omits it) holds SQL `NULL`, and the obvious `bool(row.get("is_active"))` reads that as *disabled*. `.get("is_active", True)` is no better: the default only applies when the key is **absent**, and a `SELECT *` row always has the key with value `None`. The predicate is `is_active is None or bool(is_active)` - unknown means active, only an explicit `0`/`False` disables. This is the same NULL-vs-0 trap as the `COALESCE` rule for atomic updates. It gates session auth, API-key auth, Basic auth, the login and access-token routes, the devRant token/auth paths, the admin enable/disable toggle, and `_can_hold_primary_admin`; every one of them goes through this single function. A NULL row previously could not log in at all, and the admin toggle could not disable it. (The `is_active` on a `gateway_models` row is a different table with its own semantics and is deliberately not routed through this.)
- **Role values are stored capitalized:** `users.role` is exactly `"Admin"` or `"Member"` (first registered user is `"Admin"`, `auth.py`); `is_admin` compares `== "Admin"` case-sensitively. The CLI is the only lowercase surface (`devplace role set ... <member|admin>` writes `role.capitalize()`; `role get` prints `.lower()`). A lowercase role in the DB silently defeats every admin check - never write a raw lowercase role.
- **The shadow rule generalizes beyond `is_owner` to ANY Jinja global** (`is_admin`, `avatar_url`, `format_date`, `is_self`, `owns`, `guest_disabled`): `respond(req, tmpl, ctx, model=XOut)` hands the **same** `ctx` to the Pydantic model and the template, and a context key shadows the same-named global across the whole `base.html` chain. A bool named `is_admin` in the context makes `base.html`'s `{% if is_admin(user) %}` raise `TypeError: 'bool' object is not callable` - a 500 that only fires for the branch invoking the global (logged-in users, not guests, which is why guest-only smoke tests miss it). Name viewer/permission flags distinctly (`viewer_is_admin`) in both schema and context. Real issue fixed on `/issues/{number}`; regression-guarded by `tests/api/issues/create.py::test_issue_detail_renders_for_{member,admin}` (they render the page as an authenticated Member/Admin and assert 200 + the admin-only control).
- **Policy enforced everywhere:** guests see all non-admin content read-only with action controls **shown but disabled** (`guest_disabled` on vote/star/react/poll/bookmark/follow/comment submit; create FABs become `/auth/login` links via `.feed-fab.login-required`); members get full member actions; **role badges render only to admin viewers** (`{% if is_admin(user) %}` around every `*.role` label). Backend stays the real gate (`require_user`/`require_admin`).
- Docs admin gating is unchanged behaviourally but now uses `is_admin` (`docs_base.html` `DEVPLACE_DOCS.isAdmin`, `docs/index.html`, `docs.py`).
- **Tests:** the role-gating e2e tests across `tests/e2e/` (guest/member/admin via `page`/`bob`/`alice`) are the UI enforcement; `tests/api/auth/matrix.py` is the backend companion. Guest action controls are asserted **disabled** (not absent) - don't reintroduce `count() == 0` assertions for them.

## Database tables

| Table | Purpose |
|-------|---------|
| `news` | All synced articles with `status` (published/draft), `grade`, `slug`, `show_on_landing` |
| `news_images` | Images extracted from article URLs |
| `news_sync` | Sync state per article `guid` - tracks grading history |

The platform-wide soft-delete table set (`database.SOFT_DELETE_TABLES`) is listed in full under "Project-wide soft delete (hard rule)" above.

## Site settings

Site settings are seeded on startup (`site_settings` table):

| Key | Default | Purpose |
|-----|---------|---------|
| `site_name` / `site_description` / `site_tagline` | DevPlace branding | General site metadata |
| `news_grade_threshold` | `"7"` | Minimum AI grade for auto-publish |
| `news_api_url` | `"https://news.app.molodetz.nl/api"` | News source API |
| `news_ai_url` | `"https://aquality.cloud.pravda.education/v1/chat/completions"` | AI grading endpoint - the free, local aquality quality model by default (see `devplacepy/services/news/CLAUDE.md`) |
| `news_ai_model` | `"aquality"` | AI model identifier |
| `max_upload_size_mb` / `allowed_file_types` / `max_attachments_per_resource` | `"10"` / `""` / `"10"` | Upload limits |
| `rate_limit_per_minute` | `"60"` | Mutating requests per IP per window (`main.py` middleware); a `429` carries a `Retry-After: <window>` header |
| `rate_limit_window_seconds` | `"60"` | Rolling window for the rate limit (also the `Retry-After` value) |
| `news_service_interval` | `"3600"` | Seconds between news fetch cycles (`NewsService.run_once` re-reads each cycle) |
| `session_max_age_days` | `"7"` | Standard session cookie + DB session lifetime |
| `session_remember_days` | `"30"` | Remember-me session lifetime |
| `registration_open` | `"1"` | When `"0"`, signup GET shows a closed notice and POST is rejected (`auth.py`) |
| `maintenance_mode` | `"0"` | When `"1"`, non-admins get a 503 (`main.py` maintenance middleware) |
| `maintenance_message` | scheduled-maintenance text | Body shown on the maintenance 503 page |
| `customization_enabled` | `"1"` | When `"0"`, `custom_css_tag`/`custom_js_tag` inject nothing (feature off) |
| `customization_js_enabled` | `"1"` | When `"0"`, custom CSS still serves but custom JS is suppressed |
| `moderation_sla_hours` | `"24"` | The published moderation response window; the admin queue badge turns red past it |
| `moderation_filter_mode` | `"review"` | `off`/`label`/`review`/`block` - how the content filter acts on a match |
| `moderation_filter_review_score` | `"2"` | Rule score at which a match becomes a report rather than a label |
| `moderation_minimum_age` | `"16"` | Signup floor; only the derived age band is stored |
| `moderation_mature_default_hidden` | `"1"` | Hide mature-labelled content behind an interstitial by default |
| `account_deletion_grace_hours` | `"24"` | Reversible window before a deleted account is purged |
| `contact_email` / `contact_phone` / `contact_address` | `""` | Published contact details, rendered on `/docs/contact.html` |
| `ios_app_url` | `"https://apps.apple.com/app/devplace/id6797215143"` | Official iOS app listing; the App Store badges in the footer, topnav and mobile menu link here and hide when the value is empty |
| `terms_version` / `privacy_version` / `guidelines_version` | `"1"` | Bumping `terms_version` forces re-acceptance before the next write. **Every reader uses `get_setting(key, "1") or "1"`** - an empty stored value must read as the default or the gate 403s every write |
| `ai_third_party_provider` | `""` | Named in the consent copy and the privacy policy |
| `extra_head` | `""` | Raw HTML emitted verbatim into every page `<head>` by `templating.extra_head_tag()`; site-wide trusted-admin input, not sanitized |

Besides the site/news/upload keys above, the **Operational** group is admin-editable at `/admin/settings`: `site_url` (public origin for absolute links incl. container ingress; resolved by `seo.public_base_url()` = setting -> `DEVPLACE_SITE_URL` env -> request origin), `rate_limit_per_minute`, `rate_limit_window_seconds`, `news_service_interval`, `session_max_age_days`, `session_remember_days`, `registration_open`, `maintenance_mode`, `maintenance_message`, `docs_search_mode` (`agent`|`bm25`, default `agent` - picks the `/docs/search.html` surface: the in-page Devii chat or the classic BM25 list; a viewer over their daily AI limit, or a guest when Devii is disabled, auto-falls-back to BM25 via `routers/docs/views.py` `_agent_search_state`), `outbound_proxy_url` (empty by default - when set, every `stealth.stealth_async_client`/`stealth_sync_client` call across the whole app routes through it via `stealth.configured_proxy_url()`; validated as `http(s)://`/`socks5(h)://` with a host in `AdminSettingsForm`; falls back to `DEVPLACE_OUTBOUND_PROXY_URL` when unset - see the "Outbound HTTP" note in the root `CLAUDE.md`). The **Custom Code** key `extra_head` and the `ios_app_url` badge link are the keys in the settings handler's `CLEARABLE_SETTINGS` set, so saving an empty textarea/field removes them (the default loop skips empty values).

The seed block in `database.py` is guarded by `if "site_settings" in tables:` - on a brand-new DB the table does not exist yet (dataset creates tables lazily on first insert), so none of these rows are written until the table exists. Correct runtime behavior therefore relies on every consumer passing the production default to `get_setting`/`get_int_setting`, not on the seed.

## Operational settings

Operational settings - read sites and rules:

| Setting(s) | Read at | Notes |
|-----------|---------|-------|
| `rate_limit_*` | `rate_limit_middleware` in `main.py` | `max(1, get_int_setting(...))` so `0` can't block all writes |
| `maintenance_mode` / `maintenance_message` | `maintenance_middleware` in `main.py` | Allows `/static`, `/avatar`, `/auth`, `/admin` and admins; everyone else gets `error.html` at 503 |
| `news_service_interval` | `BaseService` reconciling loop via `current_interval()` | `max(60, ...)`; edited on the Services tab (not `/admin/settings`); a change applies on the next cycle |
| `service_<name>_enabled` / `service_<name>_command` / `service_<name>_log_size` | `BaseService` reconciling loop | Generic per-service controls written by the Services tab; the loop reconciles within ~1s |
| `session_max_age_days` / `session_remember_days` | `auth.py` signup + login | Multiplied by `SECONDS_PER_DAY`; passed to `create_session(uid, max_age)` so the cookie and the DB session row expire together |
| `registration_open` | `auth.py` `signup_page` (GET) and `signup` (POST) | POST returns before any DB write when closed |

**Booleans are `<select>`, never checkboxes.** The settings save handler (`admin.py`) skips empty form values so empty fields don't clobber existing rows. An unchecked checkbox submits nothing, so it could never be turned off - `registration_open` and `maintenance_mode` use `<option value="1">`/`<option value="0">` so a value is always submitted.


