This file documents devplacepy/database/ - the dataset/SQLite data layer, indexing rules, and the project-wide soft-delete model. Claude Code loads it automatically whenever a file under this directory is read or edited.

Database engine and dataset library

SQLite via dataset with these pragmas on every connection:

PRAGMA journal_mode=WAL;      -- concurrent readers + writers
PRAGMA synchronous=NORMAL;     -- safe with WAL mode
PRAGMA busy_timeout=30000;     -- wait 30s instead of failing on lock
PRAGMA cache_size=-8000;       -- 8MB page cache
PRAGMA temp_store=MEMORY;      -- temp tables in memory

Configured via dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}}, on_connect_statements=[...]). In addition to the pragmas above, the connection is tuned with a 30s busy timeout, an 8MB page cache, and a 256MB mmap.

init_db() is idempotent - every index is created via CREATE INDEX IF NOT EXISTS (through the _index() helper, wrapped in try/except so it is safe to run on every startup regardless of table state), and seed defaults for site_settings only insert when the key doesn't already exist.

init_db() also ensures the full column set of any table that code filters on (news, news_sync, attachments use get_table(name) + create_column_by_example before their indexes). This is mandatory, not optional: dataset gives a lazily-created table ONLY the columns of its first insert, so a partial insert from a CLI tool/test/maintenance script would otherwise create the table with a reduced schema, and the long-running server caches that stale schema - making later queries on the missing column 500 with no such column. When adding a new filtered/indexed column, add it to the matching init_db() ensure-block:

news = get_table("news")
for column, example in (("uid", ""), ("status", ""), ("synced_at", ""), ("external_id", ""), ...):
    if not news.has_column(column):
        news.create_column_by_example(column, example)
_index(db, "news", "idx_news_status", ["status"])   # now safe - column exists

Do this with get_table(name) (NOT if name in db.tables): get_table + create_column_by_example creates the table if it is absent, so the schema is guaranteed regardless of which process or which insert touches the table first. Symptom to watch for: a page renders fine in isolation but shows empty data or 500s only after a CLI/maintenance test runs earlier in the suite (it created the table partially).

This applies to users too, and an if "users" not in db.tables: return guard silently defeats it. backfill_api_keys() is the users ensure-block (every non-signup column: api_key, last_seen, the AI correction/modifier settings, avatar_seed, award_count/last_award_at/last_award_slug/last_award_uid, ...). It used to early-return when users was absent, which is exactly the state on a brand-new database: init_db() runs before the first signup, so the ensure-block was skipped entirely, and the first /auth/signup then created users with only the columns of that INSERT. Every ensured-but-unwritten column was therefore missing from the long-running server's reflected metadata, so users.find_one(...) returned rows without them for the whole process lifetime - the feature reading them looked simply switched off (this is what made the awards tab, the prominent-award banner, and the avatar award badge invisible on a fresh DB, and it is invisible in production only because the columns happen to exist from an older boot). It now calls get_table("users") unconditionally, so the table is born with the full ensured column set. Never reintroduce a db.tables guard in front of a column-ensure block.

The _index(...) helper supports where= (partial) and unique= indexes; every table with a uid column gets a UNIQUE idx_<table>_uid, soft-delete tables get a PARTIAL idx_<table>_trash (WHERE deleted_at IS NOT NULL) and NEVER a bare deleted_at index (it mis-steers the planner on live reads), and "live newest-first" listings need a composite/live-partial index that carries the sort column (see "Indexing conventions" below). init_db() finishes with ANALYZE/PRAGMA optimize. Verify any index change with EXPLAIN QUERY PLAN.

SQLite is synchronous by design and will never be made async (hard rule). The dataset/SQLite layer is synchronous and is called directly inside the async route handlers. This is intentional and more than fast enough for this platform: the database is a local file tuned with WAL, synchronous=NORMAL, a 30s busy timeout, an 8MB page cache, and a 256MB mmap, so queries are sub-millisecond and never meaningfully stall the event loop. Do NOT convert the database layer to async, do NOT wrap DB calls in a threadpool/run_in_executor/to_thread, and do NOT raise the synchronous DB access as a performance, blocking-I/O, or "requests aren't fully async" concern. This is a settled, sophisticated design choice and is not open for revisiting.

Runtime config lives in site_settings, read via get_setting(key, default) / get_int_setting(key, default) (60s TTL cache, invalidated cross-worker via the cache_state version table - get_setting calls sync_local_cache("settings", ...), writes call bump_cache_version("settings"); the _user_cache in utils.py uses the same primitive under the auth name). Consumers always pass the production default to get_setting, so behavior is correct even before the row exists. Numeric operational values are floored at the call site so an invalid 0 can't lock out writes or stall a service. Booleans are stored as "0"/"1" and rendered as <select> (not checkboxes) because the settings save handler skips empty form values - an unchecked checkbox could never be turned off. See "Site settings" and "Operational settings" below for the full key registry.

Batch helpers (get_users_by_uids, get_comment_counts_by_post_uids, get_vote_counts, load_comments, get_attachments_by_type) exist specifically to avoid N+1 queries - use them in feed/listing routes instead of per-row lookups.

Dataset rules (hard-learned)

find() does NOT accept raw SQL strings. It takes keyword arguments for equality filters, dict comparison operators, or SQLAlchemy column expressions.

# WRONG - causes 500 Internal Server Error:
table.find("created_at >= :start", {"start": today})
table.find(text("created_at >= :start"), start=today)

# CORRECT - dict comparison syntax:
table.find(created_at={">=": today})

# CORRECT - keyword equality:
table.find(country="France")

# CORRECT - SQLAlchemy column expression for IN clause:
table.find(table.table.columns.user_uid.in_(["uid1", "uid2"]))

# CORRECT - multiple equality filters combined:
table.find(topic="devlog", user_uid=some_uid)

update() requires a key column list as second argument. The first dict contains all fields including the key column.

table.update({"uid": user_uid, "bio": "new bio"}, ["uid"])

db.query() accepts raw SQL with named params as keyword arguments:

db.query("SELECT * FROM posts WHERE topic = :t", t="devlog")
# NOT: db.query("...", {"t": "devlog"})

db.query() WRITES DO NOT AUTO-COMMIT - wrap any db.query INSERT/UPDATE/DELETE in with db: (load-bearing, caused a production deadlock). The dataset table API (table.insert/update/delete) calls db._auto_commit() internally, but db.query() does NOT. SQLAlchemy 2.x autobegins a transaction on first execute, so a raw db.query write leaves an open transaction holding the SQLite write lock until that thread's connection next commits. On the request/loop thread this is masked (the next table op's _auto_commit flushes it), but on a background-queue or run_in_executor worker thread the thread goes idle still holding the lock, and EVERY subsequent write app-wide blocks for the 30s busy-timeout then fails database is locked - a full deadlock. Always commit raw writes:

with db:                                   # commits + releases the write lock on exit
    db.query("INSERT INTO t (...) VALUES (:a) ON CONFLICT(...) DO UPDATE SET ...", a=1)

Atomic counters (e.g. add_correction_usage) must use raw ON CONFLICT DO UPDATE SET col = col + excluded.col (the table API cannot increment), so they MUST use the with db: wrapper. Prefer the table API whenever an atomic SQL increment is not required.

Always check tables list before raw SQL queries:

if "comments" not in db.tables:
    return {}  # table doesn't exist yet

Batch queries eliminate N+1 problems. Use get_users_by_uids(), get_comment_counts_by_post_uids(), and get_vote_counts() from database.py instead of per-row lookups in loops.

init_db() MUST create every queried column for any table that code filters on, even if the table is created lazily. dataset creates a table on its FIRST insert and gives it ONLY the columns in that insert. If any code path can insert a partial row before the full schema exists (a CLI tool, a test fixture, a maintenance script), the table is born with a reduced schema and every later query against a missing column throws sqlite3.OperationalError: no such column: X (a 500), or - for an indexed column - logs a Could not create index ... no such column warning at startup. This is worsened by cross-process metadata staleness: the long-running uvicorn server reflects a table's columns once and caches them, so a column another process adds afterward is invisible to the server until it reconnects. The defence is to make init_db() ensure the complete column set up front, exactly like the existing news, news_sync, and attachments blocks (see the code example under "Database engine and dataset library" above). When you add a NEW column that any query filters/indexes, add it to the init_db() ensure-block too - never rely on the first insert to define it.

Indexing conventions (the soft-delete planner trap)

init_db() owns every index. The _index(db, table, name, columns, *, where=None, unique=False) helper builds the DDL; it supports partial indexes (where=) and unique indexes, and wraps each CREATE/DROP in with db: (DDL via db.query does not auto-commit - see "Dataset rules" above). Three load-bearing rules learned from an EXPLAIN QUERY PLAN audit against production data:

  • Every table with a uid column gets idx_<table>_uid (UNIQUE). dataset makes its own id autoincrement PK and does NOT key uid, so find_one(uid=...), resolve_by_slug, soft_delete, and table.update({...}, ["uid"]) full-SCAN without it. init_db() loops for table in db.tables: _uid_index(db, table) (falls back to a non-unique index if a UNIQUE build ever fails on legacy duplicate data). New tables are covered automatically.

  • NEVER index the bare deleted_at column - use a PARTIAL trash index WHERE deleted_at IS NOT NULL. ensure_soft_delete_columns creates idx_<table>_trash ON (deleted_at) WHERE deleted_at IS NOT NULL (and drops any legacy full idx_<table>_deleted). A full deleted_at index is a planner hazard: the column is one giant NULL bucket plus many unique delete-timestamps, so sqlite_stat1 mis-estimates deleted_at IS NULL as returning ~2 rows and the planner picks that index for live reads, then USE TEMP B-TREE FOR ORDER BY to sort the whole live set (the global feed was doing exactly this, with 82% of posts soft-deleted). The partial index serves the admin Trash view (deleted_at IS NOT NULL) cheaply and stops poisoning live IS NULL queries.

  • For "live, newest-first" listings add a composite or live-partial index that includes the sort column. A WHERE deleted_at IS NULL ORDER BY created_at query needs the ordering in the index or it filesorts. Posts use a partial idx_posts_live_created ON (created_at) WHERE deleted_at IS NULL (feed) plus idx_posts_user_created (user_uid, created_at) (profile); comments use idx_comments_target_created (target_type, target_uid, created_at); votes use idx_votes_user_target (user_uid, target_uid) (the per-user "my_vote" check on every card); notifications/gists/projects use (user_uid, created_at); follows use idx_follows_follower_created (follower_uid, created_at) + idx_follows_following_created (following_uid, created_at) (the followers/following tabs sort newest-first; the legacy single-column follower/following indexes were dropped as redundant prefixes). All were verified to drop the USE TEMP B-TREE FOR ORDER BY step.

  • Index the non-uid lookup keys too, not just the sort/owner columns. A demand-vs-supply audit added the last missing single-key lookups: the resolve_by_slug hot path filters slug on content detail pages, so posts/gists/news/projects each get idx_<table>_slug (slug); get_setting/set_setting filter key, so idx_site_settings_key (key); the container store's find_one(slug=)/find_one(name=) fallbacks get idx_instances_slug/idx_instances_name. The DM thread load find(sender_uid=, receiver_uid=) gets the covering composites idx_messages_conversation (sender_uid, receiver_uid) + idx_messages_conversation_rev (receiver_uid, sender_uid) (the read-flag UPDATE uses the reverse); the badge-has check gets idx_badges_user_name (user_uid, badge_name); the admin user list ORDER BY -created_at gets idx_users_created_at (created_at) (the existing (role, created_at) cannot serve a full-table created_at sort). All are non-unique so _index always creates them even if legacy duplicate data exists. Column sets already resolved to ~1 row by an existing prefix index (votes +target_type, game_quests +kind, poll_options position) are intentionally left uncovered - a trailing column there only adds write cost.

init_db() ends with _refresh_query_planner_stats(): a one-time ANALYZE when sqlite_stat1 is absent, then PRAGMA optimize every boot (both inside with db:). Validate any index change with EXPLAIN QUERY PLAN against data/devplace.db (or a copy) - a correct plan reads SEARCH ... USING INDEX <name> with no temp b-tree.

Startup backfills must converge (hard rule)

init_db() runs inside lifespan under an exclusive init_lock(), before the worker accepts a single request, and every uvicorn worker runs it in turn. Time-to-serving is therefore workers x init_db, so anything added there is paid N times on every boot and every deploy. Two rules follow:

  • A backfill must be able to finish. A backfill selects the rows that still need migrating and must leave them not selected afterwards. If a row can stay in the candidate set after a successful pass, the "one-time migration" is really a permanent per-boot scan that grows with the table. _backfill_gamification had exactly this bug: it selected users.find(xp=0) and then ran the full milestone sweep over every one of them, but a user with no content is awarded no XP, so they stayed at xp=0 and were re-swept forever. At 7814 such users that was 36s of the 37.5s boot - about 140k queries that provably could not award anything, on every worker, on every restart.
  • Never fan a per-row query out over a whole table at boot. Compute the candidate set with a few set-based GROUP BY/DISTINCT queries first, then do per-row work only for rows that survive. _milestone_candidates() is the pattern: one SELECT DISTINCT per milestone source table (MILESTONE_SOURCES), unioned into a set, intersected with the pending users. A user absent from all of those tables scores 0 on every milestone metric and the lowest threshold is 5, so skipping them cannot change any award - verified by diffing the full badges table between the full sweep and the narrowed one (identical, 9800 rows, 22.6s -> 0.9s).

Watch for the same shape in dataset internals: db.tables is a live SQLAlchemy reflection, not a cached attribute. get_user_stars does one in db.tables check per STAR_TARGETS entry, so a per-user loop calling it re-reflects the whole table list on every iteration - 78667 reflections costing 17.7s in the profile above. Hoist db.tables into a local when looping.

Profile with the real database before and after any change here (cProfile around init_db() against a copy of data/devplace.db); a synthetic or empty DB hides every one of these costs.

Project-wide soft delete (hard rule)

Attachments (see "Profile media gallery and soft-deleted attachments" below) are one instance of a platform-wide model: every removal is a soft delete; only garbage collection is a hard delete. Every removable row carries two columns, deleted_at (ISO timestamp) and deleted_by (actor uid, or system). A live row has deleted_at = NULL, and every list/count read filters deleted_at IS NULL.

  • Tables (database.SOFT_DELETE_TABLES): posts, comments, gists, projects, news, news_images, project_files, attachments, votes, reactions, bookmarks, follows, poll_votes, polls, poll_options, sessions, instances, instance_schedules, devii_conversations, devii_tasks, devii_lessons, devii_virtual_tools, user_customizations, project_forks. init_db loops ensure_soft_delete_columns(t) for each (adds both columns + idx_<t>_deleted / the partial trash index described above). The Devii task/lesson/virtual-tool stores run on their own DB handles, so their _ensure_indexes adds the columns there too.
  • Every INSERT into a soft-deletable table MUST write deleted_at: None, deleted_by: None. dataset.find(deleted_at=None) on a table that lacks the column matches NOTHING (a false predicate), silently hiding all rows - the born-live insert is what creates the column. Add the pair to any new insert.
  • Central helpers (database/): soft_delete(table, deleted_by, *, stamp=None, **criteria) (equality), soft_delete_in(table, column, uids, deleted_by, *, stamp=None, **extra) (IN-clause cascade), restore(table, **criteria), purge(table, **criteria) (real delete), list_deleted(table, page) / count_deleted(table) (trash listings), and the event helpers restore_event(stamp) / purge_event(stamp) that act across ALL tables sharing one deleted_at stamp.
  • Two generic chokepoints are conditionally filtered: resolve_by_slug(table, slug, include_deleted=False) (detail-page lookups; restore passes include_deleted=True) and paginate(table, ...) (auto-appends deleted_at IS NULL when the table has the column and the caller did not pass deleted_at). seo._collect does the same for the sitemap. Read filters were threaded through every batch helper, analytics/activity/leaderboard UNION, feed/profile/listing route, and store; never re-introduce an unfiltered read of a soft-deletable table.
  • Any new read (find/count/query) of a soft-deletable table MUST filter deleted_at IS NULL. Use the central helpers/chokepoints instead of inline deletes.
  • Toggles revive, they do not duplicate. votes/reactions/bookmarks/follows/poll_votes look up the physical row regardless of deleted_at: toggle-off stamps deleted_at; re-toggle clears it on the same row. Counts/state reads filter deleted_at IS NULL.
  • Cascades share one stamp. content.delete_content_item soft-deletes the item plus its comments, votes, engagement, project files, fork relations, and attachments with one shared stamp and deleted_by = actor. That timestamp identifies the whole event, so restore_event/purge_event reverse or finalize it atomically.
  • Delete authorization is owner-OR-admin, enforced on the endpoint (is_owner(...) or is_admin(user)): posts/gists/projects (content.delete_content_item, also rejects a missing item), comments.delete_comment, project_files.project_file_delete, media.delete_media, uploads.delete_attachment_route; news (admin_news_delete) is admin-only. Because the check is on the endpoint, one rule covers the human UI and Devii at once - Devii only ever calls the platform API, authenticated as the signed-in user, so an admin's Devii may soft-delete any member's content and a member's is refused with no agent-side logic. The matching delete_* Devii catalog tools stay requires_auth (not requires_admin) so a member can still delete their own, and every one is in the dispatcher's confirmation gate (CONFIRM_REQUIRED) so a delete only runs on a repeat call with confirm=true. Standalone comment and uploaded-attachment deletes are soft like the rest (soft_delete cascade / soft_delete_attachment); the only attachment hard delete is the admin /admin/media/{uid}/purge and the CLI prune. Any NEW content delete path must reuse this guard, soft-delete, and (for the Devii tool) be added to dispatcher.CONFIRM_REQUIRED.
  • What stays HARD (GC / the empty-trash stage): the async-job sweep + CLI prune/clear, the container metrics ring trim, gateway and Devii usage-ledger retention prunes and quota resets, the expired-session cleanup branch in utils._user_from_session, fork-rollback of a half-created project, the news-sync image replacement, and the admin Purge action. Logout is a soft delete (auditable via deleted_by); only expiry GC is hard.
  • Admin Trash surface: /admin/trash (sidebar Trash, routers/admin/ package, admin_trash.html, AdminTrashOut) lists soft-deleted rows per table with restore/purge per row. Restore calls restore_event(row.deleted_at); Purge calls purge_event(...) and unlinks attachment files / project-file blobs. The attachment-specific /admin/media view is unchanged. Admin-only docs: docs/soft-delete.html (admin: True, Administration section).

Profile media gallery and soft-deleted attachments

The profile Media tab (/profile/{username}?tab=media, public) is a paginated grid of every attachment a user uploaded, newest first, across all target_types. Attachments support a soft delete so a user can remove one upload without affecting its parent object.

  • One deleted_at column on attachments (ISO string, mirrors the push.py precedent), ensured idempotently in init_db via create_column_by_example("deleted_at", "") plus the idx_attachments_user_created index. store_attachment writes deleted_at: None.
  • Soft delete preserves the relation and the file. attachments.soft_delete_attachment(uid) only stamps deleted_at; it never touches target_type/target_uid and never unlinks the file. restore_attachment(uid) clears deleted_at, so the item reappears on its parent object and in the gallery with zero extra bookkeeping.
  • Three read paths filter deleted_at IS NULL so a soft-deleted item vanishes everywhere (the gallery AND its parent post/project/etc.): get_attachments, get_attachments_batch (both in attachments.py), and database.get_user_media. The hard-delete cascades (delete_attachments_for, delete_target_attachments) stay UNFILTERED so permanently deleting a parent object still removes ALL its attachment files, including soft-deleted ones - never add the filter there.
  • Queries: database.get_user_media(user_uid, page) (linked, non-deleted, newest first; each item gets a target_url via resolve_object_url) and database.get_deleted_media(page) (the admin trash, joined to uploader username).
  • Authorization: POST /media/{uid}/delete (routers/media.py) is owner-or-admin (attachment["user_uid"] == user["uid"] or is_admin(user)); POST /media/{uid}/restore and POST /admin/media/{uid}/purge (the only hard delete, via delete_attachment) are admin-only (routers/admin/ package, sidebar Media -> /admin/media). The tab itself is public.
  • Frontend: _media_gallery.html reuses the _attachment_display.html type branches and the dp-lightbox contract (data-lightbox/data-full). The delete button carries data-media-delete + data-confirm; ModalManager.initConfirmations shows the confirm and MediaGallery.js (app.mediaGallery) does the optimistic Http.send delete, fades the tile, and toasts. A <noscript> form is the no-JS fallback. Grid styling is static/css/media.css.
  • Devii: list_media (public) and delete_media (auth, in CONFIRM_REQUIRED) in the catalog.
  • Docs visibility (deliberate): members and guests must never be told this is a soft delete. The public prose page docs/media-gallery.html (General) and the member-facing media-delete API endpoint (Profiles group) describe deletion as a plain "remove" - no soft-delete, restore, trash, or purge language. All moderation mechanics live on the admin-only docs/media-moderation prose page (admin: True) and in the admin API group (media-restore, admin-media, admin-media-purge, all auth="admin"), which docs_search excludes from member results and routers/docs/ package 404s for non-admins. Because docs_search._strip keeps the text inside {% if %} blocks, admin content must live on a separate admin: True page, never inline-gated on a public page (a public page may only carry an admin-gated link). The member MediaItemOut schema omits deleted_at; the admin-only AdminMediaItemOut adds it.

Moderation, consent and maturity tables (database/moderation.py)

Four soft-deletable tables carry the trust-and-safety layer; the full subsystem is documented in devplacepy/services/moderation/CLAUDE.md.

Table Shape Notes
content_reports (target_type, target_uid) + reporter_uid + owner_uid The one queue. owner_uid is denormalised at insert so the admin list never N+1s. Indexes (status, created_at) for the queue and SLA scan, (target_type, target_uid) for duplicate detection, (reporter_uid), (owner_uid)
moderation_actions one row per moderator decision, linked to its report Queryable moderation state with its own lifecycle - deliberately separate from the append-only audit log, the same way workspace_flags is
content_maturity (target_type, target_uid) -> level Polymorphic age label. Read through the batch helper get_maturity_by_targets, never per row. Absence of a row means general, so nothing needed backfilling
user_consents (owner_kind, owner_id, kind) Append-only in effect: withdrawing stamps withdrawn_at on the current row and inserts a new one, so the history is provable

REPORTABLE_TARGETS (target type -> table) is the registry every consumer reads, exactly like VOTABLE_TARGETS. UNREPORTABLE_TABLES is its explicit counterpart: each entry names a soft-deletable table and why it carries no reportable content. tests/unit/database/moderation.py asserts the two partition SOFT_DELETE_TABLES, so a new user-generated table cannot be added without classifying it.

The users columns added alongside are ensured in backfill_api_keys() like every other non-signup column: terms_version, terms_accepted_at, age_band, age_declared_at, mature_opt_in, suspended_until, suspension_reason, deletion_requested_at. mature_opt_in is normalised from NULL to 0 in the same with db: block that fixes the AI-modifier defaults, because it is read as a flag. No date of birth is ever stored - only the derived age_band.

Two atomic conditional updates protect this data and must never become read-then-write: queue.claim_open (report resolution) and deletion.claim_deletion (the account-deletion cascade). The latter's precondition is COALESCE(deletion_requested_at, '') = '' because the column is SQL NULL on rows that predate it - the exact NULL = 0 trap recorded above.

Role-based visibility (generic + DRY)

  • One source of truth for role/visibility checks, registered as Jinja globals in templating.py - never hand-roll user.get('role') == 'Admin' or user['uid'] == x['user_uid'] in a template again:
    • is_admin(user) (also utils.is_admin, reused by require_admin and docs.py) - admin-only UI.
    • owns(item, user) (= content.is_owner) - per-item ownership (e.g. each comment). Page-level detail templates keep using the is_owner bool passed in their context (post/gist/project/profile); do not call is_owner(...) as a function - that name is a context bool and shadows globals.
    • is_self(user, uid) - "is this me" (profile follow vs edit, leaderboard highlight).
    • guest_disabled(user) -> emits disabled aria-disabled="true" title="Log in to participate" for guests (empty for members); login_hint(user) -> a small login link. Both return Markup.
  • Account enabled/disabled is database.is_account_active(row) - never read is_active inline. is_active is a nullable column, so a row written before it existed (or by any insert that omits it) holds SQL NULL, and the obvious bool(row.get("is_active")) reads that as disabled. .get("is_active", True) is no better: the default only applies when the key is absent, and a SELECT * row always has the key with value None. The predicate is is_active is None or bool(is_active) - unknown means active, only an explicit 0/False disables. This is the same NULL-vs-0 trap as the COALESCE rule for atomic updates. It gates session auth, API-key auth, Basic auth, the login and access-token routes, the devRant token/auth paths, the admin enable/disable toggle, and _can_hold_primary_admin; every one of them goes through this single function. A NULL row previously could not log in at all, and the admin toggle could not disable it. (The is_active on a gateway_models row is a different table with its own semantics and is deliberately not routed through this.)
  • Role values are stored capitalized: users.role is exactly "Admin" or "Member" (first registered user is "Admin", auth.py); is_admin compares == "Admin" case-sensitively. The CLI is the only lowercase surface (devplace role set ... <member|admin> writes role.capitalize(); role get prints .lower()). A lowercase role in the DB silently defeats every admin check - never write a raw lowercase role.
  • The shadow rule generalizes beyond is_owner to ANY Jinja global (is_admin, avatar_url, format_date, is_self, owns, guest_disabled): respond(req, tmpl, ctx, model=XOut) hands the same ctx to the Pydantic model and the template, and a context key shadows the same-named global across the whole base.html chain. A bool named is_admin in the context makes base.html's {% if is_admin(user) %} raise TypeError: 'bool' object is not callable - a 500 that only fires for the branch invoking the global (logged-in users, not guests, which is why guest-only smoke tests miss it). Name viewer/permission flags distinctly (viewer_is_admin) in both schema and context. Real issue fixed on /issues/{number}; regression-guarded by tests/api/issues/create.py::test_issue_detail_renders_for_{member,admin} (they render the page as an authenticated Member/Admin and assert 200 + the admin-only control).
  • Policy enforced everywhere: guests see all non-admin content read-only with action controls shown but disabled (guest_disabled on vote/star/react/poll/bookmark/follow/comment submit; create FABs become /auth/login links via .feed-fab.login-required); members get full member actions; role badges render only to admin viewers ({% if is_admin(user) %} around every *.role label). Backend stays the real gate (require_user/require_admin).
  • Docs admin gating is unchanged behaviourally but now uses is_admin (docs_base.html DEVPLACE_DOCS.isAdmin, docs/index.html, docs.py).
  • Tests: the role-gating e2e tests across tests/e2e/ (guest/member/admin via page/bob/alice) are the UI enforcement; tests/api/auth/matrix.py is the backend companion. Guest action controls are asserted disabled (not absent) - don't reintroduce count() == 0 assertions for them.

Database tables

Table Purpose
news All synced articles with status (published/draft), grade, slug, show_on_landing
news_images Images extracted from article URLs
news_sync Sync state per article guid - tracks grading history

The platform-wide soft-delete table set (database.SOFT_DELETE_TABLES) is listed in full under "Project-wide soft delete (hard rule)" above.

Site settings

Site settings are seeded on startup (site_settings table):

Key Default Purpose
site_name / site_description / site_tagline DevPlace branding General site metadata
news_grade_threshold "7" Minimum AI grade for auto-publish
news_api_url "https://news.app.molodetz.nl/api" News source API
news_ai_url "https://openai.app.molodetz.nl/v1/chat/completions" AI grading endpoint
news_ai_model "molodetz" AI model identifier
max_upload_size_mb / allowed_file_types / max_attachments_per_resource "10" / "" / "10" Upload limits
rate_limit_per_minute "60" Mutating requests per IP per window (main.py middleware); a 429 carries a Retry-After: <window> header
rate_limit_window_seconds "60" Rolling window for the rate limit (also the Retry-After value)
news_service_interval "3600" Seconds between news fetch cycles (NewsService.run_once re-reads each cycle)
session_max_age_days "7" Standard session cookie + DB session lifetime
session_remember_days "30" Remember-me session lifetime
registration_open "1" When "0", signup GET shows a closed notice and POST is rejected (auth.py)
maintenance_mode "0" When "1", non-admins get a 503 (main.py maintenance middleware)
maintenance_message scheduled-maintenance text Body shown on the maintenance 503 page
customization_enabled "1" When "0", custom_css_tag/custom_js_tag inject nothing (feature off)
customization_js_enabled "1" When "0", custom CSS still serves but custom JS is suppressed
moderation_sla_hours "24" The published moderation response window; the admin queue badge turns red past it
moderation_filter_mode "review" off/label/review/block - how the content filter acts on a match
moderation_filter_review_score "2" Rule score at which a match becomes a report rather than a label
moderation_minimum_age "16" Signup floor; only the derived age band is stored
moderation_mature_default_hidden "1" Hide mature-labelled content behind an interstitial by default
account_deletion_grace_hours "24" Reversible window before a deleted account is purged
contact_email / contact_phone / contact_address "" Published contact details, rendered on /docs/contact.html
terms_version / privacy_version / guidelines_version "1" Bumping terms_version forces re-acceptance before the next write. Every reader uses get_setting(key, "1") or "1" - an empty stored value must read as the default or the gate 403s every write
ai_third_party_provider "" Named in the consent copy and the privacy policy
extra_head "" Raw HTML emitted verbatim into every page <head> by templating.extra_head_tag(); site-wide trusted-admin input, not sanitized

Besides the site/news/upload keys above, the Operational group is admin-editable at /admin/settings: site_url (public origin for absolute links incl. container ingress; resolved by seo.public_base_url() = setting -> DEVPLACE_SITE_URL env -> request origin), rate_limit_per_minute, rate_limit_window_seconds, news_service_interval, session_max_age_days, session_remember_days, registration_open, maintenance_mode, maintenance_message, docs_search_mode (agent|bm25, default agent - picks the /docs/search.html surface: the in-page Devii chat or the classic BM25 list; a viewer over their daily AI limit, or a guest when Devii is disabled, auto-falls-back to BM25 via routers/docs/views.py _agent_search_state), outbound_proxy_url (empty by default - when set, every stealth.stealth_async_client/stealth_sync_client call across the whole app routes through it via stealth.configured_proxy_url(); validated as http(s):///socks5(h):// with a host in AdminSettingsForm; falls back to DEVPLACE_OUTBOUND_PROXY_URL when unset - see the "Outbound HTTP" note in the root CLAUDE.md). The Custom Code key extra_head is the sole key in the settings handler's CLEARABLE_SETTINGS set, so saving an empty textarea removes it (the default loop skips empty values).

The seed block in database.py is guarded by if "site_settings" in tables: - on a brand-new DB the table does not exist yet (dataset creates tables lazily on first insert), so none of these rows are written until the table exists. Correct runtime behavior therefore relies on every consumer passing the production default to get_setting/get_int_setting, not on the seed.

Operational settings

Operational settings - read sites and rules:

Setting(s) Read at Notes
rate_limit_* rate_limit_middleware in main.py max(1, get_int_setting(...)) so 0 can't block all writes
maintenance_mode / maintenance_message maintenance_middleware in main.py Allows /static, /avatar, /auth, /admin and admins; everyone else gets error.html at 503
news_service_interval BaseService reconciling loop via current_interval() max(60, ...); edited on the Services tab (not /admin/settings); a change applies on the next cycle
service_<name>_enabled / service_<name>_command / service_<name>_log_size BaseService reconciling loop Generic per-service controls written by the Services tab; the loop reconciles within ~1s
session_max_age_days / session_remember_days auth.py signup + login Multiplied by SECONDS_PER_DAY; passed to create_session(uid, max_age) so the cookie and the DB session row expire together
registration_open auth.py signup_page (GET) and signup (POST) POST returns before any DB write when closed

Booleans are <select>, never checkboxes. The settings save handler (admin.py) skips empty form values so empty fields don't clobber existing rows. An unchecked checkbox submits nothing, so it could never be turned off - registration_open and maintenance_mode use <option value="1">/<option value="0"> so a value is always submitted.