This file documents the service files that live directly in devplacepy/services/ (background task queue, AI correction/modifier, presence, live view relay, and the BaseService/ServiceManager base machinery). Claude Code loads it automatically whenever a file directly under devplacepy/services/ (or any of its subdirectories) is read or edited - domain-specific services (Devii, containers, gateway, jobs, audit, telegram, email, gitea, news, bot, messaging, xmlrpc, dbapi, pubsub, game) have their own more specific nested CLAUDE.md files.

Background task queue (services/background.py)

Generic, lightweight, fire-and-forget offload for non-critical side-effects so they leave the request path. The singleton background (from devplacepy.services.background import background) wraps one in-process asyncio.Queue drained by a single consumer task. The whole API is one call: background.submit(fn, *args, **kwargs) enqueues a synchronous callable and returns immediately (put_nowait). The consumer runs each callable inside its own try/except, so a failing task is logged and never kills the loop; FIFO order is preserved.

  • Per-worker, not lock-gated (the reason it is NOT a JobService/BaseService). Producers run in every uvicorn worker, and process memory is not shared, so the drain must run wherever requests are handled. main.py startup() calls await background.start() for every worker (inside the if not DEVPLACE_DISABLE_SERVICES guard, outside the acquire_service_lock() branch); shutdown() calls await background.stop(). A BaseService supervisor only runs in the lock owner, which would strand records produced by the other worker. The JobService queue is also wrong here: it is DB-backed, so deferring a tiny audit insert would add writes instead of removing them.
  • Inline fallback (load-bearing). When the consumer is not running - tests with DEVPLACE_DISABLE_SERVICES=1, unit tests, request-less bootstrap, or a full queue - submit runs fn inline and synchronously. This keeps audit/notification/XP writes deterministic and immediately visible to the test suite (which asserts audit rows right after an action) while production defers them. No test changes are needed.
  • Best-effort durability (intentional). In-memory only; stop() drains whatever remains synchronously so a graceful shutdown loses nothing, but a hard crash drops unflushed items. This matches audit/notifications already being best-effort (the recorder never raises). Do not put response-critical or money-touching work on it.
  • Capture plain data, never the Request. Its lifecycle ends with the response - build the row/payload synchronously on the request thread and submit only the resulting dict/scalars.
  • Consumers today (deferred at their canonical choke points, so callers need no change):
    • Audit - services/audit/record.py _write builds the row + links synchronously, generates the uid/created_at eagerly so record() still returns the real uid, then background.submit(_persist, row, links) does the two store inserts off-thread.
    • XP/rewards - utils.award_rewards defers its body via background.submit(_apply_rewards, ...), so every XP award (posts/comments/projects/gists/follow/votes) leaves the request path.
    • Notifications - utils.create_notification (the single notification funnel) defers via background.submit(_deliver_notification, ...), so every in-app insert + push schedule + audit for a notification (vote/follow/comment/mention/message/badge/level) runs on the consumer.
    • Mention fan-out - utils.create_mention_notifications defers its whole body (_deliver_mention_notifications): the extract_mentions regex + the @-username lookup query + the per-user loop all run off-thread (it is called on every post/gist/project/comment/message create).
    • Issue-comment admin fan-out - routers/issues/comment.py defers the _notify_admins loop (admin lookup + per-admin notify) via background.submit, after the synchronous Gitea call.
  • Because the reward and notification funnels self-defer, request handlers just call award_rewards/create_notification directly - do NOT wrap them in background.submit (that double-queues). The pattern for any NEW side-effect: do the user-visible write inline, then call the self-deferring funnel (or background.submit(...) a one-off).
  • Feature/first-use badges ride the same self-deferring model: call utils.track_action(user_uid, action[, target]) inline at a feature's success point (do NOT wrap it - it background.submits itself), where action is a key in utils.ACHIEVEMENTS (action -> [(threshold, badge), ...], threshold 1 = first use; UNIQUE_ACTIONS use record_unique_activity for "N distinct things" like docs.read). Source-derivable milestones (counts of existing tables) instead go in check_milestone_badges/_COUNT_MILESTONES. Badge metadata + group live in BADGE_CATALOG; the profile shows a grouped earned/locked Achievements showcase via build_achievements.
  • Deliberately NOT deferred (would break correctness): cache invalidations (clear_user_cache/clear_unread_cache/clear_messages_cache/bump_cache_version) must run before the response so the next read is fresh (and they are microsecond version bumps); the vote/reaction count aggregation feeds the AJAX response body; and synchronous external calls whose result the response needs or must surface on failure (the Gitea comment/status calls; file/thumbnail writes whose returned URL must already exist) belong in an async JobService (durable + retryable), not this fire-and-forget queue.

AI content correction (services/correction.py)

Opt-in, default off, server-side. When a user turns it on (profile AI content correction block, owner-only, saved at POST /profile/{username}/ai-correction), the prose they author is rewritten by the AI gateway. The per-user ai_correction_sync flag (0 = background default, 1 = sync) picks the apply mode: background rewrites the stored fields a moment after the write (never slows the request); sync makes the HTTP response wait until the correction is applied.

  • Sync must never block the event loop (the self-deadlock trap). The correction POSTs to the in-process gateway (INTERNAL_GATEWAY_URL = localhost:{PORT}), and the hooked content helpers are synchronous on the loop thread, so a blocking inline call self-deadlocks the (single) worker against its own gateway request - on a single worker that loop is the only thing that can serve the gateway request it is waiting on, so a blocking inline call deadlocks the worker until the httpx timeout, then fail-softs (server hangs, no correction). The fix: sync runs _run_correction on the dedicated AI_APPLY_EXECUTOR thread pool (correction.py, 4 workers) via loop.run_in_executor (loop stays free, and the default executor - shared with asyncio.to_thread password hashing at login/signup - is never occupied by blocking AI HTTP calls), stashes the future on request.scope[PENDING_SCOPE_KEY], and the await_pending_corrections middleware in main.py awaits it after the handler. Never reintroduce a blocking inline correction on the loop thread.
  • Field registry is the single source of truth. CORRECTABLE_FIELDS: dict[str, tuple[str, ...]] maps each correctable table to its prose columns: posts -> (title, content), projects/gists -> (title, description), comments/messages -> (content,), users -> (bio,). gists.source_code, project files, and Gitea issues are intentionally excluded - code and external systems are never corrected.
  • Choke entrypoint. schedule_correction(user, table, uid, request=None) is the only thing handlers call (every hook passes request). It is a no-op unless: a user dict is present, table is in the registry, user["ai_correction_enabled"] is truthy, and the user has a non-empty api_key. In sync mode (user["ai_correction_sync"]) it calls _run_inline_awaited, which - only when a request.scope and a running event loop exist - submits _run_correction to loop.run_in_executor(AI_APPLY_EXECUTOR, ...) (off the loop thread, on the dedicated AI pool) and stashes the future on request.scope[PENDING_SCOPE_KEY] for the middleware to await; if there is no request/loop it returns False and falls back to background. In background mode (default) it background.submits _run_correction (per-worker queue; also runs inline under DEVPLACE_DISABLE_SERVICES=1). The same _run_correction worker function runs in all cases.
  • The hooks (covers UI + REST + Devii + devRant in one place): content.create_content_item (posts/projects/gists), content.edit_content_item, content.create_comment_record, content.edit_comment_record, services/messaging/persist.persist_message (DMs - sender is the user), routers/profile/index.update_profile (bio), and the two devRant direct-update edit paths (routers/devrant/rants.edit_rant, routers/devrant/comments.edit_comment). devRant create paths route through the shared content cores, so they are already hooked. Code-only and external paths (project files, Gitea) are deliberately not hooked.
  • The gateway call is fail-soft. correct_text(api_key, prompt, text) is synchronous (runs on the background worker thread), POSTs to INTERNAL_GATEWAY_URL with model=INTERNAL_MODEL via stealth.stealth_sync_client, authenticated with the user's own Bearer api_key (per-user attribution). It returns the ORIGINAL text on any error, empty output, or suspiciously large output (len > len(text) * MAX_GROWTH_FACTOR + 200, rejecting hallucinated expansion). _run_correction reads the row back, corrects each registry field that has non-blank text, writes only changed fields via table.update(updates, ["uid"]) without touching updated_at (auto-correction is not a user edit) or the slug (slugs are permanent), and clear_user_cache(user_uid) when table == "users" so the corrected bio re-caches.
  • Per-user usage aggregation (only on success). correct_text returns (text, usage); usage is parsed from the gateway's X-Gateway-* response headers (_usage_from_headers) whenever the upstream call returned 200 (cost was incurred, even if the corrected output was rejected), else None on any failure. _usage_from_headers captures the token and cost headers PLUS the timing headers X-Gateway-Upstream-Latency-Ms and X-Gateway-Total-Latency-Ms (as upstream_latency_ms/total_latency_ms), so each call's timing is metered. _run_correction accumulates the per-field usage into one totals dict and, when totals["calls"] > 0, makes ONE call to database.add_correction_usage(user_uid, totals) (a single totals dict, not positional args) - so a 2-field content item is a single aggregated write, and a failed/empty correction records nothing. add_correction_usage (and add_modifier_usage) delegate to the shared database._add_usage(usage_table, user_uid, totals): a single atomic INSERT ... ON CONFLICT(user_uid) DO UPDATE SET col = col + excluded.col upsert against the correction_usage table (per-user running SUMS: calls/prompt_tokens/completion_tokens/total_tokens/cost_usd/upstream_latency_ms/total_latency_ms/updated_at, unique index idx_correction_usage_user, the two latency columns REAL default 0.0, all ensured in init_db). It is a derived counter table (NOT in SOFT_DELETE_TABLES, like gateway_usage_ledger) and is deliberately separate from users so accumulating never invalidates the auth/user cache. database.get_correction_usage(user_uid) (via _get_usage) returns the stored sums PLUS computed averages: avg_tokens (total_tokens/calls), avg_upstream_latency_ms, avg_total_latency_ms, avg_tokens_per_second (completion_tokens over total upstream seconds), and avg_cost_usd.
  • Profile display: routers/profile/usage._correction_usage(uid, include_cost) shapes it via the shared _usage_view(data, include_cost) (mirrors _ai_quota); profile/index.py builds it only for is_owner or viewer_is_admin and passes include_cost=viewer_is_admin, exposed as the correction_usage dict on the context and ProfileOut. The view surfaces the sums plus averages - avg_tokens (avg tokens/call), avg_latency_ms (avg upstream latency), avg_total_latency_ms, avg_tokens_per_second (avg speed), and total_time_s (total upstream seconds) - rendered as extra tiles on the card. Financial gating: tokens/call-count and the performance tiles show to the owner and admins; the dollar cost_usd and avg_cost_usd keys are present ONLY when viewer_is_admin, in BOTH the HTML card (templates/profile.html, .correction-usage-*) and the respond(..., model=ProfileOut) JSON (same rule as _ai_quota's spent_usd - hiding it in the template alone would leak it to a member fetching their own profile as JSON). The card renders only when correction_usage.calls > 0.
  • Import-cycle discipline. correction.py imports only stealth, config, database.get_table/add_correction_usage, and services.background.background at module top; clear_user_cache is imported lazily inside _run_correction. Never import content or utils at module top.
  • Settings live on users: three columns ai_correction_enabled (0/1), ai_correction_sync (0/1, default 0 = background), and ai_correction_prompt (text, default config.DEFAULT_CORRECTION_PROMPT), ensured in database.backfill_api_keys() (the user column-ensure block run by init_db) and seeded born-live in utils._create_account. The edit route is the owner-or-admin leaf POST /profile/{username}/ai-correction (routers/profile/ai_correction.py, AiCorrectionForm{enabled, sync, prompt}, audit key profile.ai_correction). The owner-only values are exposed on the profile page context and ProfileOut (ai_correction_enabled/ai_correction_sync/ai_correction_prompt, gated by is_owner), the UI block lives in profile.html (owner-only: enable checkbox, Apply mode select, prompt textarea) wired by static/js/AiCorrection.js (app.aiCorrection), and Devii drives it via the owner-scoped ai_correction_get/ai_correction_set tools (services/devii/ai_correction/, handler="ai_correction", requires_auth=True, not confirm-gated - it is a reversible per-user toggle; ai_correction_set accepts enabled, optional sync, optional prompt).

AI modifier (services/ai_modifier.py, services/ai_context.py)

A sibling of AI content correction that runs only on an explicit inline directive. The engine reuses the correction plumbing wholesale (CORRECTABLE_FIELDS, PENDING_SCOPE_KEY, gateway_complete, the sync/background apply-mode mechanics, the per-user usage upsert) and differs only in the trigger and the apply-mode/enabled defaults: it is enabled by default and synchronous by default, and it runs ONLY where an authored prose field contains an inline @ai <instruction> directive.

  • The @ai gate is the whole difference. has_ai_directive(text) matches the regex @ai\s+\S (case-insensitive). schedule_modification(user, table, uid, request=None) is a no-op unless a user is present, table is in CORRECTABLE_FIELDS, user["ai_modifier_enabled"] is truthy, the user has an api_key, AND at least one of the table's registry fields actually contains an @ai directive. _run_modification re-checks the gate per field, so untriggered fields are never sent to the gateway and never metered. Triggerless writes cost nothing. The configured prompt (default config.DEFAULT_MODIFIER_PROMPT = "Execute what is behind @ai (the prompt) and replace that part including @ai") tells the model to execute the instruction and replace the marked part including the @ai marker.
  • Total reuse of the correction layer. CORRECTABLE_FIELDS, PENDING_SCOPE_KEY, gateway_complete, the sync/background apply-mode mechanics (_run_inline_awaited -> loop.run_in_executor + request.scope[PENDING_SCOPE_KEY] awaited by the await_pending_corrections middleware in main.py, the same self-deadlock-avoiding path), and the per-user usage upsert pattern are all imported from / mirror services/correction.py. modify_text(api_key, prompt, text, context="") composes a modifier system message and calls the shared gateway_complete. The same hooks fire it: profile/index.update_profile calls both schedule_correction and schedule_modification, and the content/comment/messaging cores invoke it alongside correction, so it covers the web UI, REST, devRant, and Devii in one place. Code and source files are never modified (same CORRECTABLE_FIELDS registry, gists.source_code/project files/Gitea excluded). schedule_modification is hooked alongside schedule_correction at the same content/comment/messaging/profile entrypoints.
  • Context-aware (modifier only, not correction). Unlike correction, the modifier gives the model a grounding context block so an @ai instruction can reason about who is asking and what it is attached to. services/ai_context.py build_context(table, uid, row, user_uid) -> str assembles it; _run_modification builds it lazily once per row (only after a field is confirmed to contain @ai, so triggerless writes do no extra queries) and passes it to every field's modify_text, which appends it to the system message under a # Context (use it to inform the result; never echo this block) header. The block (fail-soft, length-capped, each part wrapped in try/except so a failed lookup never blocks the modification) has three parts:
    1. date - Today is DD/MM/YYYY on the DevPlace developer network.
    2. author/stats - the author's username, role, level, stars, post count (get_user_post_count), leaderboard rank (get_user_rank), follower count (get_follow_counts), member-since date, and bio (capped MAX_BIO).
    3. location/thread - table-specific: a comment gets the target post/project/gist/news title + excerpt and the parent comment it replies to; a post gets its topic and attached project; a project/gist gets its sibling title/description and, for gists, the language + source_code (truncated, marked "reference only" - this is read context, the modifier still never WRITES source_code); a direct message gets the recipient and the last MAX_THREAD_MESSAGES messages of the conversation (each truncated). All excerpts are length-capped (MAX_* constants) to bound token cost; the added context tokens are billed and metered like any other input via the usage stats. So @ai answer the question above in a comment, @ai write my bio from my stats, or @ai reply to this in a DM all work. The message-thread query binds a param named :current (NOT :self, which collides with db.query's bound-method argument).
  • Defaults differ: enabled ON by default and apply mode defaults to synchronous (ai_modifier_sync column defaults to 1), the inverse of correction.
  • Per-user usage (only on success). Same plumbing as correction: _usage_from_headers captures the token, cost, and timing headers (X-Gateway-Upstream-Latency-Ms/X-Gateway-Total-Latency-Ms); _run_modification accumulates per-field gateway usage into a totals dict and, when totals["calls"] > 0, makes ONE database.add_modifier_usage(user_uid, totals) call (a single totals dict) that delegates to the shared database._add_usage atomic INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col upsert into the modifier_usage table (running SUMS calls/token/cost_usd/upstream_latency_ms/total_latency_ms totals, the two latency columns REAL default 0.0, separate from users so it never busts the auth cache, NOT in SOFT_DELETE_TABLES, ensured in init_db). database.get_modifier_usage(user_uid) (via _get_usage) returns the sums plus the same computed averages as correction (avg_tokens, avg_upstream_latency_ms, avg_total_latency_ms, avg_tokens_per_second, avg_cost_usd).
  • Settings live on users: three columns ai_modifier_enabled (0/1, default 1), ai_modifier_sync (0/1, default 1 = sync), and ai_modifier_prompt (text, default config.DEFAULT_MODIFIER_PROMPT), ensured in database.backfill_api_keys() (the user column-ensure block run by init_db) and seeded born-live in utils._create_account. Because the feature is on by default, backfill_api_keys() ALSO runs a with db: UPDATE users SET ... WHERE ... IS NULL so EXISTING accounts inherit the on/sync defaults (a freshly create_column_by_example-added column is NULL on existing rows, which would read as disabled - the backfill is what makes "on by default" true for everyone, not just new accounts). The with db: wrapper is load-bearing: a raw db.query write that does not commit holds the SQLite write lock. The edit route is the owner-or-admin leaf POST /profile/{username}/ai-modifier (routers/profile/ai_modifier.py, AiModifierForm{enabled, sync, prompt}, audit key profile.ai_modifier). The owner-only values are exposed on the profile page context and ProfileOut (ai_modifier_enabled/ai_modifier_sync/ai_modifier_prompt, gated by is_owner); modifier_usage is built by routers/profile/usage._modifier_usage(uid, include_cost=viewer_is_admin) (via the shared _usage_view) for is_owner or viewer_is_admin, surfacing the sums plus the performance averages (avg tokens/call, avg latency, avg tokens/sec, total time) like correction_usage; the dollar cost_usd and avg_cost_usd are present only when viewer_is_admin, in both HTML and JSON. The UI block lives in profile.html (owner-only: enable checkbox, Apply mode select defaulting to sync, prompt textarea, plus the usage card reusing the .correction-usage-* classes with the extra performance tiles) wired by static/js/AiModifier.js (app.aiModifier). Devii drives it via the owner-scoped ai_modifier_get/ai_modifier_set tools (services/devii/ai_modifier/, handler="ai_modifier", requires_auth=True, not confirm-gated; ai_modifier_set accepts enabled, optional sync, optional prompt).

Background services base machinery (services/base.py, services/manager.py)

devplacepy/services/ provides a generic framework for running background async services alongside the FastAPI server. BaseService provides the async run loop, a deque(maxlen=20) log buffer, and graceful cancellation; ServiceManager is a singleton that registers, starts, and stops services. Services are fully managed from the Services admin tab (/admin/services): start/stop, enable-on-boot, run-now, edit parameters, clear logs, adjustable log buffer size, with live status. All of this is generic - a new service gets it for free by declaring its config and implementing run_once.

This section covers only the shared machinery. The individual services built on top of it live in their own subdirectories with their own nested CLAUDE.md: NewsService (services/news/, see devplacepy/services/news/CLAUDE.md), GatewayService and provider/model routing (services/openai_gateway/, see devplacepy/services/openai_gateway/CLAUDE.md), DeviiService (services/devii/, see devplacepy/services/devii/CLAUDE.md), and the bot fleet service (services/bot/, see devplacepy/services/bot/CLAUDE.md).

DB-backed state (correct across workers)

In production (make prod, 2 workers) only the lock-holding worker runs services, but an admin request can hit either worker. State therefore lives in the DB, not process memory:

  • Desired/config state in site_settings (via get_setting/set_setting): service_<name>_enabled ("0"/"1" - also the boot flag), service_<name>_command ("<verb>:<counter>", verbs run/clear), service_<name>_log_size, plus each declared config field's own key.
  • Observed state in the service_state table (one row per service): status, last_run, next_run, started_at, heartbeat, logs (JSON), updated_at. Written by the supervising worker, read by any worker. The heartbeat persists every PERSIST_SECONDS (8s, under the 15s STALE_SECONDS liveness window) and caches the row id so a persist is one UPDATE, not a find_one + UPDATE. collect_metrics() runs on its own slower METRICS_SECONDS cadence (15s default, per-class overridable - AuditService uses 300s because its metric is a COUNT(*) over the ever-growing audit table); between refreshes the last snapshot is re-persisted, and a force persist (transitions, run-now) always recomputes. Keep expensive aggregates out of the 1s tick: put them in collect_metrics and, if still heavy, raise the subclass METRICS_SECONDS.

main.py registers every service in all workers (so describe_all() works anywhere) but only the lock worker calls service_manager.supervise().

ConfigField (services/base.py)

Declarative parameter spec. type in int/str/url/text/password/bool/select. coerce(raw) validates and types (raises ValueError, never silent); read() is the lenient runtime reader (falls back to default); spec() renders the field for the template/JSON (secrets masked, never sent). Optional minimum/maximum/options/help/secret. group="..." lets the detail page render fields as sections (built-in fields are grouped "General"/"Advanced").

BaseService (services/base.py)

Abstract class for all services:

  • name, interval_key (default service_<name>_interval; a subclass may point it at an existing key), min_interval.
  • config_fields - class-level list of ConfigField. The framework prepends built-in enabled and interval fields and appends log_size; all_fields() returns the full set.
  • get_config() returns a typed dict from settings; is_enabled(), current_interval() (floored to min_interval).
  • log(message) - writes to log_buffer and standard logging.
  • run_once() - abstract; override with actual work. Read parameters via self.get_config().
  • Reconciling loop (_run_loop/_tick, ~1s tick): honors enabled, runs run_once() when due or on a run command, handles clear, rebuilds the log deque if log_size changed, and persists observed state (throttled, force on transitions). Start/stop/run/clear from any worker are just DB writes the loop reacts to within ~1s - services are never hard-restarted.
  • Lifecycle hooks async on_enable() / async on_disable() - called on transition to enabled / disabled (and on_disable on shutdown). Override for long-running resources that outlive one run_once (e.g. the bot fleet); run_once then becomes a periodic reconcile/metrics tick.
  • collect_metrics() -> dict - generic live metrics. Return {"stats": [{"label", "value"}], "table": {"columns": [...], "rows": [[...]]}}. Persisted to service_state.metrics each tick and rendered live in the Services tab (stat cards + table) by ServiceMonitor.js. Default returns {}.
  • default_enabled class flag (default True) - set False for opt-in services so they never auto-start on boot (the bots service uses this).
  • title / description class attrs - shown in the admin UI; every service should set a description of what it does.

Admin UI (routers/admin/services.py): /admin/services is a slim index (services.html, an admin-table of name/description/status/uptime/last-run/next-run/interval) where each row links to /admin/services/{name}. The detail page (service_detail.html) holds everything for one service - header with controls (start/stop/run/clear), and [data-tabs] tabs Overview (meta + metrics), Configuration (the config form rendered as a <fieldset> per field_groups entry), Logs. ServiceMonitor.js is dual-mode (polls /admin/services/data for the index, /admin/services/{name}/data for the detail, via the shared Poller; its config-save POST goes through Http.sendForm) and shares one update(root, svc) over data-* hooks; tabs use the generic reusable static/js/Tabs.js ([data-tabs]/[data-tab]/[data-tab-pane], wired in Application.js). Adding a service still needs zero UI code.

  • describe() - builds the dict the UI consumes: title, description, derived display status (stopped when disabled, running with fresh heartbeat, stalled when enabled but heartbeat older than STALE_SECONDS), meta, logs, metrics, flat fields, and field_groups (ordered {name, fields} sections) read from service_state + settings.

ServiceManager (services/manager.py)

Singleton that manages all registered services:

  • register(service) - add a service
  • describe_all() -> list[dict] (per-service describe())
  • set_enabled(name, bool) - Start/Stop (writes the enabled flag)
  • send_command(name, "run"|"clear") - one-shot command channel
  • save_config(name, form) - strict validation via each ConfigField; blank secret = keep existing; returns {ok, errors}
  • supervise() - start the reconciling task per service (lock worker only)
  • shutdown_all() - cancel tasks on server shutdown

Adding a new service

  1. Create devplacepy/services/your_service.py with a class extending BaseService. Declare config_fields (and set interval_key/min_interval if reusing an existing key); read them in run_once via self.get_config().
  2. Override async def run_once(self) -> None.
  3. Register in main.py startup event (outside the lock block, so all workers know the roster): 
    from devplacepy.services.your_service import YourService
service_manager.register(YourService())
  4. The service appears automatically on /admin/services with start/stop, enable-on-boot, run-now, a generic config form, clear-logs, live status, and the log tail - no template, JS, or router changes needed.

CLI

devplace news clear                     # Delete all news from local database
devplace devii reset-quota <username>   # Reset one user's rolling 24h AI quota
devplace devii reset-quota --guests     # Reset every guest quota
devplace devii reset-quota --all        # Reset every quota (users and guests)

Multi-worker concurrency (preferred rules)

uvicorn --workers N = N independent processes sharing only the filesystem and SQLite DB. Module-global caches/counters are per-process, so a local clear() is invisible to siblings. Full reference: admin docs Production -> Multi-worker and concurrency (templates/docs/production-concurrency.html). Enforce these:

  • In-process caches are never authoritative across workers. Coordinate invalidation through the cache_state version table in database.py: call bump_cache_version(name) at every write path and sync_local_cache(name, cache) before every read path. Existing names: auth (guards _user_cache; bumped by clear_user_cache/clear_session_cache on logout, ban, role/password change - a version bump makes EVERY worker clear its WHOLE _user_cache, so display-only refreshes (XP/level in award_xp, AI-corrected bio) call clear_user_cache(uid, propagate=False) for a local pop without the global bump; identity/authz changes (logout, ban, role, password, api-key/token revoke) MUST keep the default propagate=True), settings (guards _settings_cache; bumped by set_setting/clear_settings_cache), relations (_relations_cache), customizations (_customizations_cache), notif_prefs (_notification_prefs_cache), gateway_routing (_ROUTING_CACHE), and admins (guards _admins_cache, which memoizes get_admin_uids() / get_primary_admin_uid()). The _cache_version_cache (ttl=1) caches the version reads themselves, so the bump is seen by other workers within ~1s on their next read (the originating worker is immediate); the version read/bump fail open (logged, never raise). When you add a per-process cache whose staleness matters, give it a name and wire both calls - do not invent a second invalidation mechanism.
  • Admin-set cache invariant (load-bearing). _admins_cache makes get_admin_uids() / get_primary_admin_uid() (hit on the projects-listing visibility filter, _owner_is_admin, and every primary-admin gate: /dbapi, backup-archive download) a dict lookup instead of a per-call SELECT. It is NOT the authorization gate - is_admin(user)/require_admin read the role off the user object (independently invalidated via clear_user_cache), so a stale admin set cannot grant access. But any code that writes users.role MUST call database.invalidate_admins_cache() (clears local + bumps the admins version), exactly like the soft-delete and with db: rules. Current role-write sites all do: routers/admin/users.py (role change), cli.py (role set), and utils._create_account (first user -> Admin). Omitting the call leaves the admin set stale for up to the 300s TTL.
  • Deterministic / eventual caches skip version-sync on purpose. Two caches need no cache_state name because correctness does not depend on cross-worker freshness: (1) docs_prose._render_markdown is an @lru_cache on the static prose source string (pure function of template content, so identical on every worker; changes only on deploy), and (2) main._home_cache (TTLCache ttl=60) memoizes the guest / blocks - the featured-news block (identical for everyone) and the latest-posts block only for the no-block case; a viewer with a non-empty block set bypasses the cache and is computed fresh, so the original block-filter semantics are preserved byte-for-byte and there is zero cross-user leakage. Worst case is a soft-deleted/edited public post lingering on / for <=60s. Use a plain TTLCache/lru_cache (no version name) ONLY when the value is deterministic or its staleness is purely cosmetic; anything whose staleness affects correctness or permissions MUST use the version-sync pattern above.
  • Authoritative state lives in the DB, memory is only a short-TTL accelerator (sessions, the Devii 24h cap via devii_usage_ledger, cost counters).
  • Global rates/quotas are worker-count-aware or DB-backed. The rate limiter enforces ceil(limit / DEVPLACE_WEB_WORKERS) per process so the aggregate matches the configured value with no per-request write. DEVPLACE_WEB_WORKERS MUST equal the real --workers (set in make prod=2 and the Dockerfile=2; default 1 for dev). Update it whenever you change worker count.
  • First-boot side effects are flock-serialized and idempotent. init_db() runs under a blocking flock on devplace-init.lock (the seed-if-missing inserts would otherwise race to duplicate rows); ensure_certificates() early-returns when keys exist and otherwise generates under .vapid.lock. Any new run-once-at-startup work follows the same pattern (flock + exists-check / INSERT OR IGNORE).
  • Run-once background work stays behind the services lock, never per-worker: gate it on service_manager.owns_lock() (held via devplace-services.lock), as news/bots/Devii hub do.

Performance caches (request-path)

Read-mostly aggregates that were recomputed per request or per vote sit behind short per-worker TTLCaches (devplacepy/cache.py). These are DISPLAY caches: staleness up to the TTL is accepted by design, so none of them uses the cross-worker cache_state versioning (that is reserved for correctness-critical caches like auth/settings/admins). The registry:

Cache Where Key TTL Invalidation
_authors_cache database/ranking.py ranked/rank_map 60s TTL only - update_target_stars deliberately does NOT clear it per vote anymore (the old per-vote clear() forced a full UNION-JOIN ranking recompute on the next feed/leaderboard/landing request and never propagated cross-worker anyway)
_stars_cache database/ranking.py user uid 15s TTL + clear_user_stars(owner_uid) from content.apply_vote, so a user's own total updates immediately on the voting worker
_leaderboard_cache services/game/store/farm.py top:{limit} 15s TTL only (the farm scan + Python farm_score sort runs at most once per 15s per worker)
_projects_cache templating.py user uid 10s TTL + clear_user_projects_cache(uid) from the content.py project create/delete choke points (in-function import - templating imports content at module level). jinja_user_projects also filters deleted_at=None now (the composer dropdown previously listed soft-deleted projects)

Rules: a new hot read-path aggregate follows this exact pattern (module-level TTLCache, 10-15s, clear_* helper only when a same-worker write must be visible immediately); never reintroduce a whole-cache clear() on a per-event write path; profile projects/gists loads are tab-gated in routers/profile/index.py (tab == X or wants_json(request) - the JSON ProfileOut keeps serializing both, HTML tabs that do not render them get []).

Live view relay (services/live_view_relay.py)

LiveViewRelayService is a second lock-owner pub/sub bridge (default-enabled, 1s interval, registered in main.py after NotificationRelayService) that replaces the per-client HTTP polling on the admin live views with server push, computing a snapshot only for topics that currently have subscribers. Each tick it reads pubsub.topics() (the hub's live subscription set, which converges on the lock owner), matches each concrete subscribed topic against the VIEWS registry (a list of (compiled_regex, compute_callable, min_interval_seconds)), throttles per topic via a time.monotonic() map, computes the payload, and publishes it on the same topic. The compute callables (async, lazy-importing to avoid cycles, defensive try/except -> skip) reproduce the exact payload shape the matching HTTP endpoint already returns, so the frontend render code is unchanged - only the data-arrival path is added. Registry:

Topic Cadence Payload (same as endpoint)
container.list 4s {instances} (admin all-instances, _decorate)
project.{slug}.containers 3s {instances} (per-project, slug resolved via resolve_by_slug)
container.{uid}.detail 4s {instance, events, schedules, stats, runtime}
container.{uid}.logs 3s {logs} (backend log tail 400)
fleet.bots 2s bot frames payload (routers/admin/bots._frames_payload)
admin.services 5s {services} (service_manager.describe_all())
admin.services.{name} 5s {service}
admin.ai-usage.{hours} 15s build_analytics(hours) (hours parsed from the topic)
admin.backups 8s routers/admin/backups._dashboard(can_download=False) (storage, backups, schedules, metrics)

All these topics are admin-only by pub/sub policy (non-public, non-user.{uid} -> privileged required), matching the admin-only pages. Frontend monitors (ContainerInstance, ContainerList, ContainerManager, BotMonitor, ServiceMonitor, AiUsageMonitor, BackupMonitor) each window.app.pubsub.subscribe(topic, render) in their init and keep a lengthened HTTP poll (15-30s) as initial-load + fallback - the relay drives liveness at the cadence above. AiUsageMonitor re-subscribes (unsubscribe old, subscribe new) when the window-hours selector changes, since hours is in the topic.

Container topics never broadcast private-project instances: container.list publishes only public-project rows with partial: true (ContainerList.merge updates by uid, never removes, so private rows from the authoritative HTTP poll survive), and project.{slug}.containers / container.{uid}.detail / container.{uid}.logs skip private-project targets entirely (owners fall back to their HTTP polls).

admin.backups is the one view with a per-viewer field: the backup download_url is restricted to the primary administrator at every endpoint, so the relay publishes the snapshot with can_download=False (the bus broadcasts one payload to every admin subscriber, and is therefore treated as another endpoint that must withhold it). BackupMonitor derives canDownload from its authoritative per-viewer HTTP poll only (never from a pushed frame) and builds the /admin/backups/{uid}/download URL client-side; the download route itself stays primary-admin-gated.

To add a new admin live view: add one (regex, compute, interval) row to VIEWS whose compute returns the endpoint payload, and have the frontend subscribe to the topic while keeping a fallback poll. Reuse this subscriber-gated snapshot pattern for any future "several admins watch a periodically-recomputed server snapshot" surface; keep durable/ordered/stateful/raw-I/O channels (messages, Devii, SEO/DeepSearch progress, container PTY) OFF pub/sub. See pubreport.md for the full migration analysis.

Notification relay (services/notification_relay.py)

Live toasts ride the in_app channel (no new channel). NotificationRelayService is a lock-owner BaseService (registered in main.py, default-enabled, 1s interval) modeled on MessageRelay: it primes a watermark to MAX(notifications.id) on first tick, then each tick selects notifications WHERE id > watermark and publishes {uid, type, message, target_url} to the per-recipient pub/sub topic user.{user_uid}.notifications. It runs only on the service lock owner, which is exactly where every pub/sub WS subscriber converges (non-owner /pubsub/ws closes 4013), so the in-process services.pubsub.publish reaches the recipient's browser. Because the notifications row exists only when the in_app channel is enabled, a live toast fires exactly for the notifications a user has enabled - the relay needs no preference lookup of its own. The watermark always advances (boot-priming + every-tick) so a newly-connected subscriber never gets a backlog flood.

Frontend: base.html exposes the viewer uid as <body data-user-uid>, static/js/LiveNotifications.js (app.liveNotifications, constructed with this.pubsub/this.toast) subscribes to that topic and calls app.toast.show(message, {type:"info", ms, url}). The toast click reproduces a bell-item click exactly: it targets GET /notifications/open/{uid} (which marks the row read and redirects to its target_url), not the bare target_url - the relay publishes uid for this. AppToast gained a generic click action via _action(options): options.onClick (a function) or options.url (navigate); either adds the .dp-toast-link pointer cursor, so any caller can attach a click action. DMs toast too (type message is not excluded). Reuse this relay-on-the-lock-owner pattern for any future "live mirror of a DB-persisted, per-user event."

Unread-count badges ride the same relay. The same NotificationRelayService tick, after publishing the toast rows, also publishes the recipient's fresh unread counts {notifications, messages} to user.{uid}.counts (computed with a direct DB query on the lock owner, never the per-worker _unread_cache, which could be stale there). Because a new DM also inserts a message-type notification row through persist_message -> create_notification, this one publish point covers both new notifications and new messages, so the header badge bumps instantly. static/js/CounterManager.js (app.counters, constructed with this.pubsub) subscribes to user.{uid}.counts and applies the pushed counts; its HTTP poll of /notifications/counts stays as a 60s reconciliation fallback (covers decrements on read and any missed frame). Read events still clear the per-worker cache as before.

Online presence (services/presence.py, services/presence_relay.py)

Online status is a single users.last_seen UTC-ISO column (ensured in database.backfill_api_keys), never a new table and never a per-request insert. The design is dictated by the multi-worker architecture: a profile of user X renders on any worker, so "is X online" needs cross-worker state, and pub/sub is in-process only (a non-owner worker cannot publish to the lock owner), leaving SQLite as the sole shared medium. So presence is a heavily-throttled in-place UPDATE.

Write path (all workers): main.py's track_presence HTTP middleware resolves the cached current user on every non-/static, non-/avatar request and calls presence.touch(uid). touch keeps a per-worker in-memory _last_write: dict[uid -> monotonic] and writes users.last_seen (via database.set_last_seen) only when the last write for that uid is older than config.PRESENCE_WRITE_SECONDS (= PRESENCE_TIMEOUT_SECONDS // 2). So continuous browsing is a dict lookup; a write happens at most ~once per half-window per active user per worker, and the row is updated in place (zero growth). It deliberately does not call clear_user_cache (that would defeat the 300s auth cache; the stale cached self-row is irrelevant since presence of other users is always read from a fresh row).

Read path (any worker): presence.is_online(user_row) = now - last_seen < PRESENCE_TIMEOUT_SECONDS (env DEVPLACE_PRESENCE_TIMEOUT_SECONDS, default 60). Profile (routers/profile/index.py -> profile_online) and messages (routers/messages.py seed) read last_seen off the user row they already loaded - no extra query. Exposed as the Jinja global is_online(user) (templating.py), on UserOut.last_seen and ProfileOut.profile_online. This is the only cross-worker-correct approach here because pub/sub is in-process.

Live path (lock owner only), change-only + hysteresis: PresenceRelayService (services/presence_relay.py, BaseService, default-enabled, 2s tick, registered in main.py) is a sibling of NotificationRelayService/LiveViewRelayService. Each tick it recomputes ONE global online set self._online (dot-subscribed uids batch-read via get_users_by_uids + roster candidates) and drives BOTH the per-user dots and the feed roster from that one set, so they can never disagree. The set uses hysteresis via presence.stays_online(elapsed, was_online): a user becomes online at PRESENCE_TIMEOUT_SECONDS but only drops after + PRESENCE_ONLINE_MARGIN_SECONDS (env DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS, default 20) - quick to go online, slow (grace margin) to go offline - which kills boundary flicker for a user hovering near the timeout. It publishes {online, last_seen} to public.presence.{uid} only when a topic's online bool changed OR the topic is newly subscribed (first-seen) - never on a fixed interval, so a steady page emits nothing after the initial frame (self._published[topic] -> bool, pruned to active topics). public.presence.{uid} is subscribable by any logged-in user (pubsub/policy.py allows public.*); guests fall back to the server-rendered initial state. The one-directional grace also means dots and roster stay consistent across viewers (the shared self._online is the single authority). To keep it lightweight the relay reads all due users in one batched get_users_by_uids per tick.

Frontend: static/js/PresenceManager.js (app.presence, constructed with this.pubsub in Application.js, mirroring LocalTime/CounterManager) scans [data-presence-uid] elements, subscribes each uid to public.presence.{uid} (deduped per uid, so a repeated author is one subscription), and treats each frame's online flag as authoritative (entry.online), toggling the online class + (for data-presence-label elements) the "online / last seen X / offline" text. Because the relay is change-only and authoritative, a live-subscribed dot is not expired by the client clock (no false-offline flicker for an active user whose last_seen the client cannot see advancing); the 20s last_seen staleness timer only applies to entries that never received a frame (guests / degraded). The window is read from <body data-presence-timeout>. Both the profile .profile-presence dot and the messages #messages-presence span carry data-presence-uid/data-presence-last-seen.

Online-now roster (feed): the same relay maintains ONE shared topic public.presence.roster, republished only when the SET of online uids changes (a frozenset compare, so pure reordering never republishes). services/presence.py online_users(limit) (strict, feed initial render) and online_candidates(limit) (grace window, relay hysteresis) both go through database.get_online_users(cutoff_iso, limit), which reads users with last_seen >= cutoff via the idx_users_last_seen index (the one place presence is queried by last_seen; config.PRESENCE_ONLINE_LIMIT, env DEVPLACE_PRESENCE_ONLINE_LIMIT, default 30). The list is ordered ALPHABETICALLY by username (get_online_users order_by=["username"] + case-insensitive presence.sort_by_username), NOT by recency, so avatars keep a stable position and do not needlessly reshuffle as people's last_seen ticks. routers/feed.py puts online_users on the context (FeedOut.online_users) and feed.html renders the initial Online now panel as a .sidebar-section at the bottom of the left feed sidebar (aside.sidebar-card); static/js/OnlineUsers.js (app.onlineUsers) subscribes to public.presence.roster and re-renders the avatar list + count live. Roster avatars use a plain green .presence-dot with NO data-presence-uid (list membership IS the presence, so no per-user subscription - the relay drops a user from the roster when they go offline).

Avatar presence dot (sitewide, DRY): a small corner dot on every user avatar (green online, muted grey offline) comes from ONE reusable partial templates/_presence_dot.html - <span class="presence-dot" data-presence-uid data-presence-last-seen> guarded on _user.get('uid') (a partial-dict author, e.g. the issues includes, renders no dot). It carries no data-presence-label, so PresenceManager colours it with zero extra JS. It is included by the shared avatar partial templates/_avatar_link.html (its .user-avatar-link anchor is the positioning host, covering ~19 sites) and by the handful of raw-<img class="avatar-img"> sites wrapped in a positioned <span class="avatar-badge"> (the two base.html nav avatars, the profile.html hero + followers list, the messages.html conversation list). CSS in static/css/base.css (.user-avatar-link/.avatar-badge position:relative;display:inline-flex, .presence-dot sized 30% of the avatar clamped 8-14px with a --bg-card ring, .online -> --success), so it is proportional and responsive at every avatar size with no per-size class. database/follows.py get_follow_list now carries last_seen in its trimmed dict so the followers/following dots resolve (all other author dicts are full get_users_by_uids rows). dp-avatar (AppAvatar.js) is docs-demo only (no real user avatars) and is intentionally out of scope. Reuse _presence_dot.html + the .avatar-badge wrapper for any new avatar surface - never hand-roll a presence dot.

Messaging refactor: the old presence was per-worker and WS-connect-based (message_hub.is_online/last_seen, _announce_presence, the WS presence frame) and broke with >1 worker. That display path was removed; message_hub keeps only its socket connection tracking for message delivery. The messages header presence is now the shared PresenceManager, so chat presence is finally cross-worker correct. Never re-implement WS-connect presence - reuse presence.is_online, the public.presence.{uid} topic, and PresenceManager.