This file documents the service files that live directly in devplacepy/services/ (background task queue, AI correction/modifier, presence, live view relay, and the BaseService/ServiceManager base machinery). Claude Code loads it automatically whenever a file directly under devplacepy/services/ (or any of its subdirectories) is read or edited - domain-specific services (Devii, containers, gateway, jobs, audit, telegram, email, gitea, news, bot, messaging, xmlrpc, dbapi, pubsub, game) have their own more specific nested CLAUDE.md files.
## Background task queue (`services/background.py`)
Generic, lightweight, **fire-and-forget** offload for non-critical side-effects so they leave the request path. The singleton `background` (`from devplacepy.services.background import background`) wraps one in-process `asyncio.Queue` drained by a single consumer task. The whole API is one call: `background.submit(fn, *args, **kwargs)` enqueues a **synchronous** callable and returns immediately (`put_nowait`). The consumer runs each callable inside its own `try/except`, so a failing task is logged and never kills the loop; FIFO order is preserved.
- **Per-worker, not lock-gated (the reason it is NOT a `JobService`/`BaseService`).** Producers run in every uvicorn worker, and process memory is not shared, so the drain must run wherever requests are handled. `main.py` `startup()` calls `await background.start()` for every worker (inside the `if not DEVPLACE_DISABLE_SERVICES` guard, **outside** the `acquire_service_lock()` branch); `shutdown()` calls `await background.stop()`. A `BaseService` supervisor only runs in the lock owner, which would strand records produced by the other worker. The `JobService` queue is also wrong here: it is DB-backed, so deferring a tiny audit insert would *add* writes instead of removing them.
- **Inline fallback (load-bearing).** When the consumer is not running - tests with `DEVPLACE_DISABLE_SERVICES=1`, unit tests, request-less bootstrap, or a full queue - `submit` runs `fn` **inline and synchronously**. This keeps audit/notification/XP writes deterministic and immediately visible to the test suite (which asserts audit rows right after an action) while production defers them. No test changes are needed.
- **Best-effort durability (intentional).** In-memory only; `stop()` drains whatever remains synchronously so a **graceful** shutdown loses nothing, but a hard crash drops unflushed items. This matches audit/notifications already being best-effort (the recorder never raises). Do not put response-critical or money-touching work on it.
- **Capture plain data, never the `Request`.** Its lifecycle ends with the response - build the row/payload synchronously on the request thread and submit only the resulting dict/scalars.
- **Consumers today** (deferred at their canonical choke points, so callers need no change):
- **Audit** - `services/audit/record.py` `_write` builds the row + links synchronously, generates the `uid`/`created_at` eagerly so `record()` still returns the real uid, then `background.submit(_persist, row, links)` does the two `store` inserts off-thread.
- **XP/rewards** - `utils.award_rewards` defers its body via `background.submit(_apply_rewards, ...)`, so every XP award (posts/comments/projects/gists/follow/votes) leaves the request path.
- **Notifications** - `utils.create_notification` (the single notification funnel) defers via `background.submit(_deliver_notification, ...)`, so every in-app insert + push schedule + audit for a notification (vote/follow/comment/mention/message/badge/level) runs on the consumer.
- **Mention fan-out** - `utils.create_mention_notifications` defers its whole body (`_deliver_mention_notifications`): the `extract_mentions` regex + the `@`-username lookup query + the per-user loop all run off-thread (it is called on every post/gist/project/comment/message create).
- **Issue-comment admin fan-out** - `routers/issues/comment.py` defers the `_notify_admins` loop (admin lookup + per-admin notify) via `background.submit`, after the synchronous Gitea call.
- Because the reward and notification funnels self-defer, request handlers just **call `award_rewards`/`create_notification` directly** - do NOT wrap them in `background.submit` (that double-queues). The pattern for any NEW side-effect: do the user-visible write inline, then call the self-deferring funnel (or `background.submit(...)` a one-off).
- **Feature/first-use badges** ride the same self-deferring model: call `utils.track_action(user_uid, action[, target])` inline at a feature's success point (do NOT wrap it - it `background.submit`s itself), where `action` is a key in `utils.ACHIEVEMENTS` (`action -> [(threshold, badge), ...]`, threshold 1 = first use; `UNIQUE_ACTIONS` use `record_unique_activity` for "N distinct things" like `docs.read`). Source-derivable milestones (counts of existing tables) instead go in `check_milestone_badges`/`_COUNT_MILESTONES`. Badge metadata + `group` live in `BADGE_CATALOG`; the profile shows a grouped earned/locked **Achievements** showcase via `build_achievements`.
- **Deliberately NOT deferred (would break correctness):** cache invalidations (`clear_user_cache`/`clear_unread_cache`/`clear_messages_cache`/`bump_cache_version`) must run before the response so the next read is fresh (and they are microsecond version bumps); the vote/reaction count aggregation feeds the AJAX response body; and synchronous **external** calls whose result the response needs or must surface on failure (the Gitea comment/status calls; file/thumbnail writes whose returned URL must already exist) belong in an async **JobService** (durable + retryable), not this fire-and-forget queue.
## AI content correction (`services/correction.py`)
**Opt-in, default off, server-side.** When a user turns it on (profile **AI content correction** block, owner-only, saved at `POST /profile/{username}/ai-correction`), the prose they author is rewritten by the AI gateway. The per-user `ai_correction_sync` flag (0 = background default, 1 = sync) picks the **apply mode**: background rewrites the stored fields a moment after the write (never slows the request); sync makes the HTTP response wait until the correction is applied.
- **Sync must never block the event loop (the self-deadlock trap).** The correction POSTs to the in-process gateway (`INTERNAL_GATEWAY_URL` = `localhost:{PORT}`), and the hooked content helpers are synchronous on the loop thread, so a blocking inline call self-deadlocks the (single) worker against its own gateway request - on a single worker that loop is the only thing that can serve the gateway request it is waiting on, so a blocking inline call deadlocks the worker until the httpx timeout, then fail-softs (server hangs, no correction). The fix: sync runs `_run_correction` on the dedicated `AI_APPLY_EXECUTOR` thread pool (`correction.py`, 4 workers) via `loop.run_in_executor` (loop stays free, and the default executor - shared with `asyncio.to_thread` password hashing at login/signup - is never occupied by blocking AI HTTP calls), stashes the future on `request.scope[PENDING_SCOPE_KEY]`, and the `await_pending_corrections` middleware in `main.py` awaits it after the handler. Never reintroduce a blocking inline correction on the loop thread.
- **Field registry is the single source of truth.** `CORRECTABLE_FIELDS: dict[str, tuple[str, ...]]` maps each correctable table to its prose columns: `posts` -> `(title, content)`, `projects`/`gists` -> `(title, description)`, `comments`/`messages` -> `(content,)`, `users` -> `(bio,)`. `gists.source_code`, project files, and Gitea issues are intentionally excluded - code and external systems are never corrected.
- **Choke entrypoint.** `schedule_correction(user, table, uid, request=None)` is the only thing handlers call (every hook passes `request`). It is a no-op unless: a user dict is present, `table` is in the registry, `user["ai_correction_enabled"]` is truthy, and the user has a non-empty `api_key`. In **sync** mode (`user["ai_correction_sync"]`) it calls `_run_inline_awaited`, which - only when a `request.scope` and a running event loop exist - submits `_run_correction` to `loop.run_in_executor(AI_APPLY_EXECUTOR, ...)` (off the loop thread, on the dedicated AI pool) and stashes the future on `request.scope[PENDING_SCOPE_KEY]` for the middleware to await; if there is no request/loop it returns False and falls back to background. In **background** mode (default) it `background.submit`s `_run_correction` (per-worker queue; also runs inline under `DEVPLACE_DISABLE_SERVICES=1`). The same `_run_correction` worker function runs in all cases.
- **The hooks (covers UI + REST + Devii + devRant in one place):** `content.create_content_item` (posts/projects/gists), `content.edit_content_item`, `content.create_comment_record`, `content.edit_comment_record`, `services/messaging/persist.persist_message` (DMs - sender is the user), `routers/profile/index.update_profile` (bio), and the two devRant direct-update edit paths (`routers/devrant/rants.edit_rant`, `routers/devrant/comments.edit_comment`). devRant create paths route through the shared content cores, so they are already hooked. Code-only and external paths (project files, Gitea) are deliberately not hooked.
- **The gateway call is fail-soft.** `correct_text(api_key, prompt, text)` is synchronous (runs on the background worker thread), POSTs to `INTERNAL_GATEWAY_URL` with `model=INTERNAL_MODEL` via `stealth.stealth_sync_client`, authenticated with the user's own `Bearer` api_key (per-user attribution). It returns the ORIGINAL text on any error, empty output, or suspiciously large output (`len > len(text) * MAX_GROWTH_FACTOR + 200`, rejecting hallucinated expansion). `_run_correction` reads the row back, corrects each registry field that has non-blank text, writes only changed fields via `table.update(updates, ["uid"])` without touching `updated_at` (auto-correction is not a user edit) or the slug (slugs are permanent), and `clear_user_cache(user_uid)` when `table == "users"` so the corrected bio re-caches.
- **Per-user usage aggregation (only on success).** `correct_text` returns `(text, usage)`; `usage` is parsed from the gateway's `X-Gateway-*` response headers (`_usage_from_headers`) whenever the upstream call returned 200 (cost was incurred, even if the corrected output was rejected), else `None` on any failure. `_usage_from_headers` captures the token and cost headers PLUS the timing headers `X-Gateway-Upstream-Latency-Ms` and `X-Gateway-Total-Latency-Ms` (as `upstream_latency_ms`/`total_latency_ms`), so each call's timing is metered. `_run_correction` accumulates the per-field `usage` into one `totals` dict and, when `totals["calls"] > 0`, makes ONE call to `database.add_correction_usage(user_uid, totals)` (a single `totals` dict, not positional args) - so a 2-field content item is a single aggregated write, and a failed/empty correction records nothing. `add_correction_usage` (and `add_modifier_usage`) delegate to the shared `database._add_usage(usage_table, user_uid, totals)`: a single atomic `INSERT ... ON CONFLICT(user_uid) DO UPDATE SET col = col + excluded.col` upsert against the `correction_usage` table (per-user running SUMS: `calls`/`prompt_tokens`/`completion_tokens`/`total_tokens`/`cost_usd`/`upstream_latency_ms`/`total_latency_ms`/`updated_at`, unique index `idx_correction_usage_user`, the two latency columns REAL default 0.0, all ensured in `init_db`). It is a derived counter table (NOT in `SOFT_DELETE_TABLES`, like `gateway_usage_ledger`) and is deliberately separate from `users` so accumulating never invalidates the auth/user cache. `database.get_correction_usage(user_uid)` (via `_get_usage`) returns the stored sums PLUS computed averages: `avg_tokens` (total_tokens/calls), `avg_upstream_latency_ms`, `avg_total_latency_ms`, `avg_tokens_per_second` (completion_tokens over total upstream seconds), and `avg_cost_usd`.
- **Profile display:** `routers/profile/usage._correction_usage(uid, include_cost)` shapes it via the shared `_usage_view(data, include_cost)` (mirrors `_ai_quota`); `profile/index.py` builds it only for `is_owner or viewer_is_admin` and passes `include_cost=viewer_is_admin`, exposed as the `correction_usage` dict on the context and `ProfileOut`. The view surfaces the sums plus averages - `avg_tokens` (avg tokens/call), `avg_latency_ms` (avg upstream latency), `avg_total_latency_ms`, `avg_tokens_per_second` (avg speed), and `total_time_s` (total upstream seconds) - rendered as extra tiles on the card. **Financial gating:** tokens/call-count and the performance tiles show to the owner and admins; the dollar `cost_usd` and `avg_cost_usd` keys are present ONLY when `viewer_is_admin`, in BOTH the HTML card (`templates/profile.html`, `.correction-usage-*`) and the `respond(..., model=ProfileOut)` JSON (same rule as `_ai_quota`'s `spent_usd` - hiding it in the template alone would leak it to a member fetching their own profile as JSON). The card renders only when `correction_usage.calls` > 0.
- **Import-cycle discipline.** `correction.py` imports only `stealth`, `config`, `database.get_table`/`add_correction_usage`, and `services.background.background` at module top; `clear_user_cache` is imported lazily inside `_run_correction`. Never import `content` or `utils` at module top.
- **Settings live on `users`:** three columns `ai_correction_enabled` (0/1), `ai_correction_sync` (0/1, default 0 = background), and `ai_correction_prompt` (text, default `config.DEFAULT_CORRECTION_PROMPT`), ensured in `database.backfill_api_keys()` (the user column-ensure block run by `init_db`) and seeded born-live in `utils._create_account`. The edit route is the owner-or-admin leaf `POST /profile/{username}/ai-correction` (`routers/profile/ai_correction.py`, `AiCorrectionForm{enabled, sync, prompt}`, audit key `profile.ai_correction`). The owner-only values are exposed on the profile page context and `ProfileOut` (`ai_correction_enabled`/`ai_correction_sync`/`ai_correction_prompt`, gated by `is_owner`), the UI block lives in `profile.html` (owner-only: enable checkbox, **Apply mode** select, prompt textarea) wired by `static/js/AiCorrection.js` (`app.aiCorrection`), and Devii drives it via the owner-scoped `ai_correction_get`/`ai_correction_set` tools (`services/devii/ai_correction/`, `handler="ai_correction"`, `requires_auth=True`, not confirm-gated - it is a reversible per-user toggle; `ai_correction_set` accepts `enabled`, optional `sync`, optional `prompt`).
## AI modifier (`services/ai_modifier.py`, `services/ai_context.py`)
**A sibling of AI content correction that runs only on an explicit inline directive.** The engine reuses the correction plumbing wholesale (`CORRECTABLE_FIELDS`, `PENDING_SCOPE_KEY`, `gateway_complete`, the sync/background apply-mode mechanics, the per-user usage upsert) and differs only in the trigger and the apply-mode/enabled defaults: it is **enabled by default** and **synchronous by default**, and it runs ONLY where an authored prose field contains an inline `@ai <instruction>` directive.
- **The `@ai` gate is the whole difference.** `has_ai_directive(text)` matches the regex `@ai\s+\S` (case-insensitive). `schedule_modification(user, table, uid, request=None)` is a no-op unless a user is present, `table` is in `CORRECTABLE_FIELDS`, `user["ai_modifier_enabled"]` is truthy, the user has an `api_key`, AND at least one of the table's registry fields actually contains an `@ai` directive. `_run_modification` re-checks the gate per field, so untriggered fields are never sent to the gateway and never metered. Triggerless writes cost nothing. The configured prompt (default `config.DEFAULT_MODIFIER_PROMPT` = "Execute what is behind `@ai` (the prompt) and replace that part including `@ai`") tells the model to execute the instruction and replace the marked part including the `@ai` marker.
- **Total reuse of the correction layer.** `CORRECTABLE_FIELDS`, `PENDING_SCOPE_KEY`, `gateway_complete`, the sync/background apply-mode mechanics (`_run_inline_awaited` -> `loop.run_in_executor` + `request.scope[PENDING_SCOPE_KEY]` awaited by the `await_pending_corrections` middleware in `main.py`, the same self-deadlock-avoiding path), and the per-user usage upsert pattern are all imported from / mirror `services/correction.py`. `modify_text(api_key, prompt, text, context="")` composes a modifier system message and calls the shared `gateway_complete`. The same hooks fire it: `profile/index.update_profile` calls both `schedule_correction` and `schedule_modification`, and the content/comment/messaging cores invoke it alongside correction, so it covers the web UI, REST, devRant, and Devii in one place. Code and source files are never modified (same `CORRECTABLE_FIELDS` registry, `gists.source_code`/project files/Gitea excluded). `schedule_modification` is hooked alongside `schedule_correction` at the same content/comment/messaging/profile entrypoints.
- **Context-aware (modifier only, not correction).** Unlike correction, the modifier gives the model a grounding **context block** so an `@ai` instruction can reason about who is asking and what it is attached to. `services/ai_context.py` `build_context(table, uid, row, user_uid) -> str` assembles it; `_run_modification` builds it **lazily once per row** (only after a field is confirmed to contain `@ai`, so triggerless writes do no extra queries) and passes it to every field's `modify_text`, which appends it to the system message under a `# Context (use it to inform the result; never echo this block)` header. The block (fail-soft, length-capped, each part wrapped in try/except so a failed lookup never blocks the modification) has three parts:
1. **date** - `Today is DD/MM/YYYY on the DevPlace developer network.`
2. **author/stats** - the author's username, role, level, stars, post count (`get_user_post_count`), leaderboard rank (`get_user_rank`), follower count (`get_follow_counts`), member-since date, and bio (capped `MAX_BIO`).
3. **location/thread** - table-specific: a comment gets the target post/project/gist/news title + excerpt and the parent comment it replies to; a post gets its topic and attached project; a project/gist gets its sibling title/description and, for gists, the language + `source_code` (truncated, marked "reference only" - this is read context, the modifier still never WRITES `source_code`); a direct message gets the recipient and the last `MAX_THREAD_MESSAGES` messages of the conversation (each truncated).
All excerpts are length-capped (`MAX_*` constants) to bound token cost; the added context tokens are billed and metered like any other input via the usage stats. So `@ai answer the question above` in a comment, `@ai write my bio from my stats`, or `@ai reply to this` in a DM all work. The message-thread query binds a param named `:current` (NOT `:self`, which collides with `db.query`'s bound-method argument).
- **Defaults differ:** enabled **ON** by default and apply mode defaults to **synchronous** (`ai_modifier_sync` column defaults to 1), the inverse of correction.
- **Per-user usage (only on success).** Same plumbing as correction: `_usage_from_headers` captures the token, cost, and timing headers (`X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`); `_run_modification` accumulates per-field gateway usage into a `totals` dict and, when `totals["calls"] > 0`, makes ONE `database.add_modifier_usage(user_uid, totals)` call (a single `totals` dict) that delegates to the shared `database._add_usage` atomic `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` upsert into the `modifier_usage` table (running SUMS `calls`/token/`cost_usd`/`upstream_latency_ms`/`total_latency_ms` totals, the two latency columns REAL default 0.0, separate from `users` so it never busts the auth cache, NOT in `SOFT_DELETE_TABLES`, ensured in `init_db`). `database.get_modifier_usage(user_uid)` (via `_get_usage`) returns the sums plus the same computed averages as correction (`avg_tokens`, `avg_upstream_latency_ms`, `avg_total_latency_ms`, `avg_tokens_per_second`, `avg_cost_usd`).
- **Settings live on `users`:** three columns `ai_modifier_enabled` (0/1, default 1), `ai_modifier_sync` (0/1, default 1 = sync), and `ai_modifier_prompt` (text, default `config.DEFAULT_MODIFIER_PROMPT`), ensured in `database.backfill_api_keys()` (the user column-ensure block run by `init_db`) and seeded born-live in `utils._create_account`. **Because the feature is on by default, `backfill_api_keys()` ALSO runs a `with db:` `UPDATE users SET ... WHERE ... IS NULL` so EXISTING accounts inherit the on/sync defaults** (a freshly `create_column_by_example`-added column is NULL on existing rows, which would read as disabled - the backfill is what makes "on by default" true for everyone, not just new accounts). The `with db:` wrapper is load-bearing: a raw `db.query` write that does not commit holds the SQLite write lock. The edit route is the owner-or-admin leaf `POST /profile/{username}/ai-modifier` (`routers/profile/ai_modifier.py`, `AiModifierForm{enabled, sync, prompt}`, audit key `profile.ai_modifier`). The owner-only values are exposed on the profile page context and `ProfileOut` (`ai_modifier_enabled`/`ai_modifier_sync`/`ai_modifier_prompt`, gated by `is_owner`); `modifier_usage` is built by `routers/profile/usage._modifier_usage(uid, include_cost=viewer_is_admin)` (via the shared `_usage_view`) for `is_owner or viewer_is_admin`, surfacing the sums plus the performance averages (avg tokens/call, avg latency, avg tokens/sec, total time) like `correction_usage`; the dollar `cost_usd` and `avg_cost_usd` are present only when `viewer_is_admin`, in both HTML and JSON. The UI block lives in `profile.html` (owner-only: enable checkbox, **Apply mode** select defaulting to sync, prompt textarea, plus the usage card reusing the `.correction-usage-*` classes with the extra performance tiles) wired by `static/js/AiModifier.js` (`app.aiModifier`). Devii drives it via the owner-scoped `ai_modifier_get`/`ai_modifier_set` tools (`services/devii/ai_modifier/`, `handler="ai_modifier"`, `requires_auth=True`, not confirm-gated; `ai_modifier_set` accepts `enabled`, optional `sync`, optional `prompt`).
## Background services base machinery (`services/base.py`, `services/manager.py`)
`devplacepy/services/` provides a generic framework for running background async services alongside the FastAPI server. `BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation; `ServiceManager` is a singleton that registers, starts, and stops services. Services are fully managed from the **Services admin tab** (`/admin/services`): start/stop, enable-on-boot, run-now, edit parameters, clear logs, adjustable log buffer size, with live status. All of this is generic - a new service gets it for free by declaring its config and implementing `run_once`.
This section covers only the shared machinery. The individual services built on top of it live in their own subdirectories with their own nested CLAUDE.md: `NewsService` (`services/news/`, see `devplacepy/services/news/CLAUDE.md`), `GatewayService` and provider/model routing (`services/openai_gateway/`, see `devplacepy/services/openai_gateway/CLAUDE.md`), `DeviiService` (`services/devii/`, see `devplacepy/services/devii/CLAUDE.md`), and the bot fleet service (`services/bot/`, see `devplacepy/services/bot/CLAUDE.md`). `PushService` (`services/push/`) is the thinnest example of the pattern: it owns no loop work beyond pruning dead subscriptions, and exists mainly so every push provider's configuration is edited through the same `ConfigField` surface as every other subsystem - its `config_fields` are assembled from `devplacepy.push.providers.admin_fields()`, so a new provider appears at `/admin/services/push` with no edit to the service. Push delivery does NOT depend on that service running (see `devplacepy/push/CLAUDE.md`).
### DB-backed state (correct across workers)
In production (`make prod`, 2 workers) only the lock-holding worker runs services, but an admin request can hit either worker. State therefore lives in the DB, not process memory:
- **Desired/config state** in `site_settings` (via `get_setting`/`set_setting`): `service_<name>_enabled` (`"0"`/`"1"` - also the boot flag), `service_<name>_command` (`"<verb>:<counter>"`, verbs `run`/`clear`), `service_<name>_log_size`, plus each declared config field's own key.
- **Observed state** in the `service_state` table (one row per service): `status`, `last_run`, `next_run`, `started_at`, `heartbeat`, `logs` (JSON), `updated_at`. Written by the supervising worker, read by any worker. The heartbeat persists every `PERSIST_SECONDS` (8s, under the 15s `STALE_SECONDS` liveness window) and caches the row id so a persist is one UPDATE, not a `find_one` + UPDATE. `collect_metrics()` runs on its own slower `METRICS_SECONDS` cadence (15s default, per-class overridable - `AuditService` uses 300s because its metric is a `COUNT(*)` over the ever-growing audit table); between refreshes the last snapshot is re-persisted, and a `force` persist (transitions, run-now) always recomputes. Keep expensive aggregates out of the 1s tick: put them in `collect_metrics` and, if still heavy, raise the subclass `METRICS_SECONDS`.
`main.py` registers every service in **all** workers (so `describe_all()` works anywhere) but only the lock worker calls `service_manager.supervise()`.
### `ConfigField` (`services/base.py`)
Declarative parameter spec. `type` in `int`/`str`/`url`/`text`/`password`/`bool`/`select`. `coerce(raw)` validates and types (raises `ValueError`, never silent); `read()` is the lenient runtime reader (falls back to `default`); `spec()` renders the field for the template/JSON (secrets masked, never sent). Optional `minimum`/`maximum`/`options`/`help`/`secret`. `group="..."` lets the detail page render fields as sections (built-in fields are grouped "General"/"Advanced").
### `BaseService` (`services/base.py`)
Abstract class for all services:
- **`name`**, **`interval_key`** (default `service_<name>_interval`; a subclass may point it at an existing key), **`min_interval`**.
- **`config_fields`** - class-level list of `ConfigField`. The framework prepends built-in `enabled` and interval fields and appends `log_size`; `all_fields()` returns the full set.
- **`get_config()`** returns a typed dict from settings; **`is_enabled()`**, **`current_interval()`** (floored to `min_interval`).
- **`log(message)`** - writes to `log_buffer` and standard `logging`.
- **`run_once()`** - abstract; override with actual work. Read parameters via `self.get_config()`.
- **Reconciling loop** (`_run_loop`/`_tick`, ~1s tick): honors `enabled`, runs `run_once()` when due or on a `run` command, handles `clear`, rebuilds the log deque if `log_size` changed, and persists observed state (throttled, force on transitions). Start/stop/run/clear from any worker are just DB writes the loop reacts to within ~1s - services are never hard-restarted.
- **Lifecycle hooks** `async on_enable()` / `async on_disable()` - called on transition to enabled / disabled (and `on_disable` on shutdown). Override for long-running resources that outlive one `run_once` (e.g. the bot fleet); `run_once` then becomes a periodic reconcile/metrics tick.
- **`collect_metrics() -> dict`** - generic live metrics. Return `{"stats": [{"label", "value"}], "table": {"columns": [...], "rows": [[...]]}}`. Persisted to `service_state.metrics` each tick and rendered live in the Services tab (stat cards + table) by `ServiceMonitor.js`. Default returns `{}`.
- **`default_enabled`** class flag (default `True`) - set `False` for opt-in services so they never auto-start on boot (the bots service uses this).
- **`title` / `description`** class attrs - shown in the admin UI; every service should set a `description` of what it does.
**Admin UI** (`routers/admin/services.py`): `/admin/services` is a slim **index** (`services.html`, an `admin-table` of name/description/status/uptime/last-run/next-run/interval) where each row links to `/admin/services/{name}`. The **detail** page (`service_detail.html`) holds everything for one service - header with controls (start/stop/run/clear), and `[data-tabs]` tabs **Overview** (meta + metrics), **Configuration** (the config form rendered as a `<fieldset>` per `field_groups` entry), **Logs**. `ServiceMonitor.js` is dual-mode (polls `/admin/services/data` for the index, `/admin/services/{name}/data` for the detail, via the shared `Poller`; its config-save POST goes through `Http.sendForm`) and shares one `update(root, svc)` over `data-*` hooks; tabs use the generic reusable `static/js/Tabs.js` (`[data-tabs]`/`[data-tab]`/`[data-tab-pane]`, wired in `Application.js`). Adding a service still needs zero UI code.
- **`describe()`** - builds the dict the UI consumes: `title`, `description`, derived display status (`stopped` when disabled, `running` with fresh heartbeat, `stalled` when enabled but heartbeat older than `STALE_SECONDS`), meta, logs, metrics, flat `fields`, and `field_groups` (ordered `{name, fields}` sections) read from `service_state` + settings.
### `ServiceManager` (`services/manager.py`)
Singleton that manages all registered services:
- `register(service)` - add a service
- `describe_all()` -> `list[dict]` (per-service `describe()`)
- `set_enabled(name, bool)` - Start/Stop (writes the enabled flag)
- `send_command(name, "run"|"clear")` - one-shot command channel
- `save_config(name, form)` - strict validation via each `ConfigField`; blank secret = keep existing; returns `{ok, errors}`
- `supervise()` - start the reconciling task per service (lock worker only)
- `shutdown_all()` - cancel tasks on server shutdown
### Adding a new service
1. Create `devplacepy/services/your_service.py` with a class extending `BaseService`. Declare `config_fields` (and set `interval_key`/`min_interval` if reusing an existing key); read them in `run_once` via `self.get_config()`.
2. Override `async def run_once(self) -> None`.
3. Register in `main.py` startup event (outside the lock block, so all workers know the roster):
```python
from devplacepy.services.your_service import YourService
service_manager.register(YourService())
```
4. The service appears automatically on `/admin/services` with start/stop, enable-on-boot, run-now, a generic config form, clear-logs, live status, and the log tail - no template, JS, or router changes needed.
### CLI
```bash
devplace news clear # Delete all news from local database
devplace devii reset-quota <username> # Reset one user's rolling 24h AI quota
devplace devii reset-quota --guests # Reset every guest quota
devplace devii reset-quota --all # Reset every quota (users and guests)
```
## Multi-worker concurrency (preferred rules)
`uvicorn --workers N` = N independent processes sharing only the filesystem and SQLite DB. Module-global caches/counters are per-process, so a local `clear()` is invisible to siblings. Full reference: admin docs `Production -> Multi-worker and concurrency` (`templates/docs/production-concurrency.html`). Enforce these:
- **In-process caches are never authoritative across workers.** Coordinate invalidation through the `cache_state` version table in `database.py`: call `bump_cache_version(name)` at every write path and `sync_local_cache(name, cache)` before every read path. Existing names: `auth` (guards `_user_cache`; bumped by `clear_user_cache`/`clear_session_cache` on logout, ban, role/password change - a version bump makes EVERY worker clear its WHOLE `_user_cache`, so display-only refreshes (XP/level in `award_xp`, AI-corrected bio) call `clear_user_cache(uid, propagate=False)` for a local pop without the global bump; identity/authz changes (logout, ban, role, password, api-key/token revoke) MUST keep the default propagate=True), `settings` (guards `_settings_cache`; bumped by `set_setting`/`clear_settings_cache`), `relations` (`_relations_cache`), `customizations` (`_customizations_cache`), `notif_prefs` (`_notification_prefs_cache`), `gateway_routing` (`_ROUTING_CACHE`), and **`admins`** (guards `_admins_cache`, which memoizes `get_admin_uids()` / `get_primary_admin_uid()`). The `_cache_version_cache` (`ttl=1`) caches the version reads themselves, so the bump is seen by other workers within ~1s on their next read (the originating worker is immediate); the version read/bump fail open (logged, never raise). When you add a per-process cache whose staleness matters, give it a name and wire both calls - do not invent a second invalidation mechanism.
- **Admin-set cache invariant (load-bearing).** `_admins_cache` makes `get_admin_uids()` / `get_primary_admin_uid()` (hit on the projects-listing visibility filter, `_owner_is_admin`, and every primary-admin gate: `/dbapi`, backup-archive download) a dict lookup instead of a per-call `SELECT`. It is NOT the authorization gate - `is_admin(user)`/`require_admin` read the role off the user object (independently invalidated via `clear_user_cache`), so a stale admin set cannot grant access. But **any code that writes `users.role` MUST call `database.invalidate_admins_cache()`** (clears local + bumps the `admins` version), exactly like the soft-delete and `with db:` rules. Current role-write sites all do: `routers/admin/users.py` (role change), `cli.py` (`role set`), and `utils._create_account` (first user -> Admin). Omitting the call leaves the admin set stale for up to the 300s TTL.
- **Deterministic / eventual caches skip version-sync on purpose.** Two caches need no `cache_state` name because correctness does not depend on cross-worker freshness: (1) `docs_prose._render_markdown` is an `@lru_cache` on the **static** prose source string (pure function of template content, so identical on every worker; changes only on deploy), and (2) `main._home_cache` (`TTLCache ttl=60`) memoizes the guest `/` blocks - the featured-news block (identical for everyone) and the latest-posts block **only for the no-block case**; a viewer with a non-empty block set bypasses the cache and is computed fresh, so the original block-filter semantics are preserved byte-for-byte and there is zero cross-user leakage. Worst case is a soft-deleted/edited public post lingering on `/` for <=60s. Use a plain `TTLCache`/`lru_cache` (no version name) ONLY when the value is deterministic or its staleness is purely cosmetic; anything whose staleness affects correctness or permissions MUST use the version-sync pattern above.
- **Authoritative state lives in the DB**, memory is only a short-TTL accelerator (sessions, the Devii 24h cap via `devii_usage_ledger`, cost counters).
- **Global rates/quotas are worker-count-aware or DB-backed.** The rate limiter enforces `ceil(limit / DEVPLACE_WEB_WORKERS)` per process so the aggregate matches the configured value with no per-request write. `DEVPLACE_WEB_WORKERS` MUST equal the real `--workers` (set in `make prod`=2 and the Dockerfile=2; default 1 for dev). Update it whenever you change worker count.
- **First-boot side effects are flock-serialized and idempotent.** `init_db()` runs under a blocking `flock` on `devplace-init.lock` (the seed-if-missing inserts would otherwise race to duplicate rows); `ensure_certificates()` early-returns when keys exist and otherwise generates under `.vapid.lock`. Any new run-once-at-startup work follows the same pattern (flock + exists-check / `INSERT OR IGNORE`).
- **Run-once background work stays behind the services lock**, never per-worker: gate it on `service_manager.owns_lock()` (held via `devplace-services.lock`), as news/bots/Devii hub do.
## Performance caches (request-path)
Read-mostly aggregates that were recomputed per request or per vote sit behind short per-worker `TTLCache`s (`devplacepy/cache.py`). These are DISPLAY caches: staleness up to the TTL is accepted by design, so none of them uses the cross-worker `cache_state` versioning (that is reserved for correctness-critical caches like auth/settings/admins). The registry:
| Cache | Where | Key | TTL | Invalidation |
|---|---|---|---|---|
| `_authors_cache` | `database/ranking.py` | `ranked`/`rank_map` | 60s | TTL only - `update_target_stars` deliberately does NOT clear it per vote anymore (the old per-vote `clear()` forced a full UNION-JOIN ranking recompute on the next feed/leaderboard/landing request and never propagated cross-worker anyway) |
| `_stars_cache` | `database/ranking.py` | user uid | 15s | TTL + `clear_user_stars(owner_uid)` from `content.apply_vote`, so a user's own total updates immediately on the voting worker |
| `_leaderboard_cache` | `services/game/store/farm.py` | `top:{limit}` | 15s | TTL only (the farm scan + Python `farm_score` sort runs at most once per 15s per worker) |
| `_projects_cache` | `templating.py` | user uid | 10s | TTL + `clear_user_projects_cache(uid)` from the `content.py` project create/delete choke points (in-function import - `templating` imports `content` at module level). `jinja_user_projects` also filters `deleted_at=None` now (the composer dropdown previously listed soft-deleted projects) |
Rules: a new hot read-path aggregate follows this exact pattern (module-level `TTLCache`, 10-15s, `clear_*` helper only when a same-worker write must be visible immediately); never reintroduce a whole-cache `clear()` on a per-event write path; profile `projects`/`gists` loads are tab-gated in `routers/profile/index.py` (`tab == X or wants_json(request)` - the JSON `ProfileOut` keeps serializing both, HTML tabs that do not render them get `[]`).
## Live view relay (`services/live_view_relay.py`)
`LiveViewRelayService` is a second lock-owner pub/sub bridge (default-enabled, 1s interval, registered in `main.py` after `NotificationRelayService`) that replaces the per-client HTTP polling on the admin live views with server push, **computing a snapshot only for topics that currently have subscribers**. Each tick it reads `pubsub.topics()` (the hub's live subscription set, which converges on the lock owner), matches each concrete subscribed topic against the `VIEWS` registry (a list of `(compiled_regex, compute_callable, min_interval_seconds)`), throttles per topic via a `time.monotonic()` map, computes the payload, and publishes it on the same topic. The compute callables (async, lazy-importing to avoid cycles, defensive try/except -> skip) reproduce the **exact payload shape the matching HTTP endpoint already returns**, so the frontend render code is unchanged - only the data-arrival path is added. Registry:
| Topic | Cadence | Payload (same as endpoint) |
|-------|---------|----------------------------|
| `container.list` | 4s | `{instances}` (admin all-instances, `_decorate`) |
| `project.{slug}.containers` | 3s | `{instances}` (per-project, slug resolved via `resolve_by_slug`) |
| `container.{uid}.detail` | 4s | `{instance, events, schedules, stats, runtime}` |
| `container.{uid}.logs` | 3s | `{logs}` (backend log tail 400) |
| `fleet.bots` | 2s | bot frames payload (`routers/admin/bots._frames_payload`) |
| `admin.services` | 5s | `{services}` (`service_manager.describe_all()`) |
| `admin.services.{name}` | 5s | `{service}` |
| `admin.ai-usage.{hours}` | 15s | `build_analytics(hours)` (hours parsed from the topic) |
| `admin.backups` | 8s | `routers/admin/backups._dashboard(can_download=False)` (storage, backups, schedules, metrics) |
All these topics are admin-only by pub/sub policy (non-`public`, non-`user.{uid}` -> `privileged` required), matching the admin-only pages. Frontend monitors (`ContainerInstance`, `ContainerList`, `ContainerManager`, `BotMonitor`, `ServiceMonitor`, `AiUsageMonitor`, `BackupMonitor`) each `window.app.pubsub.subscribe(topic, render)` in their init and keep a **lengthened HTTP poll (15-30s) as initial-load + fallback** - the relay drives liveness at the cadence above. `AiUsageMonitor` re-subscribes (unsubscribe old, subscribe new) when the window-hours selector changes, since hours is in the topic.
**Container topics never broadcast private-project instances**: `container.list` publishes only public-project rows with `partial: true` (`ContainerList.merge` updates by uid, never removes, so private rows from the authoritative HTTP poll survive), and `project.{slug}.containers` / `container.{uid}.detail` / `container.{uid}.logs` skip private-project targets entirely (owners fall back to their HTTP polls).
**`admin.backups` is the one view with a per-viewer field:** the backup `download_url` is restricted to the primary administrator at every endpoint, so the relay publishes the snapshot with `can_download=False` (the bus broadcasts one payload to every admin subscriber, and is therefore treated as another endpoint that must withhold it). `BackupMonitor` derives `canDownload` from its authoritative per-viewer HTTP poll only (never from a pushed frame) and builds the `/admin/backups/{uid}/download` URL client-side; the download route itself stays primary-admin-gated.
To add a new admin live view: add one `(regex, compute, interval)` row to `VIEWS` whose compute returns the endpoint payload, and have the frontend subscribe to the topic while keeping a fallback poll. Reuse this subscriber-gated snapshot pattern for any future "several admins watch a periodically-recomputed server snapshot" surface; keep durable/ordered/stateful/raw-I/O channels (messages, Devii, SEO/DeepSearch progress, container PTY) OFF pub/sub. See `pubreport.md` for the full migration analysis.
## Notification relay (`services/notification_relay.py`)
**Live toasts ride the `in_app` channel** (no new channel). `NotificationRelayService` is a lock-owner `BaseService` (registered in `main.py`, default-enabled, 1s interval) modeled on `MessageRelay`: it primes a watermark to `MAX(notifications.id)` on first tick, then each tick selects `notifications WHERE id > watermark` and publishes `{uid, type, message, target_url}` to the per-recipient pub/sub topic `user.{user_uid}.notifications`. It runs **only on the service lock owner**, which is exactly where every pub/sub WS subscriber converges (non-owner `/pubsub/ws` closes `4013`), so the in-process `services.pubsub.publish` reaches the recipient's browser. Because the `notifications` row exists only when the `in_app` channel is enabled, a live toast fires **exactly** for the notifications a user has enabled - the relay needs no preference lookup of its own. The watermark always advances (boot-priming + every-tick) so a newly-connected subscriber never gets a backlog flood.
**Frontend:** `base.html` exposes the viewer uid as `<body data-user-uid>`, `static/js/LiveNotifications.js` (`app.liveNotifications`, constructed with `this.pubsub`/`this.toast`) subscribes to that topic and calls `app.toast.show(message, {type:"info", ms, url})`. **The toast click reproduces a bell-item click exactly:** it targets `GET /notifications/open/{uid}` (which marks the row read and redirects to its `target_url`), not the bare `target_url` - the relay publishes `uid` for this. `AppToast` gained a generic click action via `_action(options)`: `options.onClick` (a function) or `options.url` (navigate); either adds the `.dp-toast-link` pointer cursor, so any caller can attach a click action. DMs toast too (type `message` is not excluded). Reuse this relay-on-the-lock-owner pattern for any future "live mirror of a DB-persisted, per-user event."
**Unread-count badges ride the same relay.** The same `NotificationRelayService` tick, after publishing the toast rows, also publishes the recipient's fresh unread counts `{notifications, messages}` to `user.{uid}.counts` (computed with a direct DB query on the lock owner, never the per-worker `_unread_cache`, which could be stale there). Because a new DM also inserts a `message`-type notification row through `persist_message -> create_notification`, this one publish point covers both new notifications and new messages, so the header badge bumps instantly. `static/js/CounterManager.js` (`app.counters`, constructed with `this.pubsub`) subscribes to `user.{uid}.counts` and applies the pushed counts; its HTTP poll of `/notifications/counts` stays as a 60s reconciliation fallback (covers decrements on read and any missed frame). Read events still clear the per-worker cache as before.
## Online presence (`services/presence.py`, `services/presence_relay.py`)
Online status is a single **`users.last_seen`** UTC-ISO column (ensured in `database.backfill_api_keys`), never a new table and never a per-request insert. The design is dictated by the multi-worker architecture: a profile of user X renders on **any** worker, so "is X online" needs cross-worker state, and pub/sub is in-process only (a non-owner worker cannot publish to the lock owner), leaving SQLite as the sole shared medium. So presence is a heavily-throttled in-place UPDATE.
**Write path (all workers):** `main.py`'s `track_presence` HTTP middleware resolves the cached current user on every non-`/static`, non-`/avatar` request and calls `presence.touch(uid)`. `touch` keeps a per-worker in-memory `_last_write: dict[uid -> monotonic]` and writes `users.last_seen` (via `database.set_last_seen`) only when the last write for that uid is older than `config.PRESENCE_WRITE_SECONDS` (= `PRESENCE_TIMEOUT_SECONDS // 2`). So continuous browsing is a dict lookup; a write happens at most ~once per half-window per active user per worker, and the row is updated in place (zero growth). It deliberately does **not** call `clear_user_cache` (that would defeat the 300s auth cache; the stale cached self-row is irrelevant since presence of *other* users is always read from a fresh row).
**Read path (any worker):** `presence.is_online(user_row)` = `now - last_seen < PRESENCE_TIMEOUT_SECONDS` (env `DEVPLACE_PRESENCE_TIMEOUT_SECONDS`, default 60). Profile (`routers/profile/index.py` -> `profile_online`) and messages (`routers/messages.py` seed) read `last_seen` off the user row they already loaded - no extra query. Exposed as the Jinja global `is_online(user)` (`templating.py`), on `UserOut.last_seen` and `ProfileOut.profile_online`. This is the **only** cross-worker-correct approach here because pub/sub is in-process.
**Live path (lock owner only), change-only + hysteresis:** `PresenceRelayService` (`services/presence_relay.py`, `BaseService`, default-enabled, 2s tick, registered in `main.py`) is a sibling of `NotificationRelayService`/`LiveViewRelayService`. Each tick it recomputes ONE global online set `self._online` (dot-subscribed uids batch-read via `get_users_by_uids` + roster candidates) and drives BOTH the per-user dots and the feed roster from that one set, so they can never disagree. The set uses **hysteresis** via `presence.stays_online(elapsed, was_online)`: a user becomes online at `PRESENCE_TIMEOUT_SECONDS` but only drops after `+ PRESENCE_ONLINE_MARGIN_SECONDS` (env `DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS`, default 20) - **quick to go online, slow (grace margin) to go offline** - which kills boundary flicker for a user hovering near the timeout. It publishes `{online, last_seen}` to `public.presence.{uid}` **only when a topic's `online` bool changed OR the topic is newly subscribed (first-seen)** - never on a fixed interval, so a steady page emits nothing after the initial frame (`self._published[topic] -> bool`, pruned to active topics). `public.presence.{uid}` is subscribable by any logged-in user (`pubsub/policy.py` allows `public.*`); guests fall back to the server-rendered initial state. The one-directional grace also means dots and roster stay consistent across viewers (the shared `self._online` is the single authority). To keep it lightweight the relay reads all due users in **one batched `get_users_by_uids`** per tick.
**Frontend:** `static/js/PresenceManager.js` (`app.presence`, constructed with `this.pubsub` in `Application.js`, mirroring `LocalTime`/`CounterManager`) scans `[data-presence-uid]` elements, subscribes each uid to `public.presence.{uid}` (deduped per uid, so a repeated author is one subscription), and treats each frame's `online` flag as **authoritative** (`entry.online`), toggling the `online` class + (for `data-presence-label` elements) the "online / last seen X / offline" text. Because the relay is change-only and authoritative, a live-subscribed dot is **not** expired by the client clock (no false-offline flicker for an active user whose `last_seen` the client cannot see advancing); the 20s `last_seen` staleness timer only applies to entries that never received a frame (guests / degraded). The window is read from `<body data-presence-timeout>`. Both the profile `.profile-presence` dot and the messages `#messages-presence` span carry `data-presence-uid`/`data-presence-last-seen`.
**Online-now roster (feed):** the same relay maintains ONE shared topic `public.presence.roster`, republished **only when the SET of online uids changes** (a `frozenset` compare, so pure reordering never republishes). `services/presence.py` `online_users(limit)` (strict, feed initial render) and `online_candidates(limit)` (grace window, relay hysteresis) both go through `database.get_online_users(cutoff_iso, limit)`, which reads users with `last_seen >= cutoff` via the `idx_users_last_seen` index (the one place presence is queried by `last_seen`; `config.PRESENCE_ONLINE_LIMIT`, env `DEVPLACE_PRESENCE_ONLINE_LIMIT`, default 30). **The list is ordered ALPHABETICALLY by username** (`get_online_users` `order_by=["username"]` + case-insensitive `presence.sort_by_username`), NOT by recency, so avatars keep a stable position and do not needlessly reshuffle as people's `last_seen` ticks. `routers/feed.py` puts `online_users` on the context (`FeedOut.online_users`) and `feed.html` renders the initial **Online now** panel as a `.sidebar-section` at the bottom of the left feed sidebar (`aside.sidebar-card`); `static/js/OnlineUsers.js` (`app.onlineUsers`) subscribes to `public.presence.roster` and re-renders the avatar list + count live. Roster avatars use a plain green `.presence-dot` with NO `data-presence-uid` (list membership IS the presence, so no per-user subscription - the relay drops a user from the roster when they go offline).
**Avatar presence dot (sitewide, DRY):** a small corner dot on **every** user avatar (green online, muted grey offline) comes from ONE reusable partial `templates/_presence_dot.html` - `<span class="presence-dot" data-presence-uid data-presence-last-seen>` guarded on `_user.get('uid')` (a partial-dict author, e.g. the issues includes, renders no dot). It carries **no** `data-presence-label`, so `PresenceManager` colours it with zero extra JS. It is included by the shared avatar partial `templates/_avatar_link.html` (its `.user-avatar-link` anchor is the positioning host, covering ~19 sites) and by the handful of raw-`<img class="avatar-img">` sites wrapped in a positioned `<span class="avatar-badge">` (the two `base.html` nav avatars, the `profile.html` hero + followers list, the `messages.html` conversation list). CSS in `static/css/base.css` (`.user-avatar-link`/`.avatar-badge` `position:relative;display:inline-flex`, `.presence-dot` sized `30%` of the avatar clamped 8-14px with a `--bg-card` ring, `.online` -> `--success`), so it is proportional and responsive at every avatar size with no per-size class. `database/follows.py` `get_follow_list` now carries `last_seen` in its trimmed dict so the followers/following dots resolve (all other author dicts are full `get_users_by_uids` rows). `dp-avatar` (`AppAvatar.js`) is docs-demo only (no real user avatars) and is intentionally out of scope. Reuse `_presence_dot.html` + the `.avatar-badge` wrapper for any new avatar surface - never hand-roll a presence dot.
**Messaging refactor:** the old presence was per-worker and WS-connect-based (`message_hub.is_online`/`last_seen`, `_announce_presence`, the WS `presence` frame) and broke with >1 worker. That display path was **removed**; `message_hub` keeps only its socket connection tracking for message delivery. The messages header presence is now the shared `PresenceManager`, so chat presence is finally cross-worker correct. **Never re-implement WS-connect presence** - reuse `presence.is_online`, the `public.presence.{uid}` topic, and `PresenceManager`.
Reference in New Issue View Git Blame Copy Permalink