# DevPlace - Agent Guide

## Quick start

```bash
make install      # pip install -e .
make dev          # uvicorn --reload on port 10500, backlog 4096
make prod         # uvicorn with 2 workers, backlog 8192 (production)
make test         # Playwright integration + unit tests, serial (one at a time), fail-fast -x
make test-headed  # same tests in a visible browser (single window)
make locust       # Locust load test (interactive web UI)
make locust-headless  # Locust in headless CLI mode (for CI)
```

**Env vars:** `DEVPLACE_DISABLE_SERVICES=1` prevents the NewsService (and future background services) from starting. Automatically set during tests.

## Architecture

- **FastAPI** backend serving **Jinja2 templates** (SSR). Pure ES6 JS for interactivity.
- **Database:** `dataset` (auto-syncs schema, uses `uid` for PKs). SQLite.
- **Auth:** `session` cookie, plus `X-API-KEY` / `Authorization: Bearer <api_key>` / HTTP Basic (username-or-email:password) - all resolved in `get_current_user`. PBKDF2-SHA256 via passlib. No JWT. Every user has an `api_key` (uuid7), set at signup and backfilled in `init_db`/`devplace apikey backfill`. Docs site at `/docs` (`routers/docs/ package`, `DOCS_PAGES` registry; FastAPI's Swagger is moved to `/swagger` so `/docs` is free).
- **Static:** `devplacepy/static/` mounted at `/static`; URLs are emitted boot-versioned (`/static/v<ts>/...`) via the `static_url`/`assetUrl` helpers and served immutable for a year (see "Static asset caching and versioning")
- **Templates:** `devplacepy/templates/`. Shared `templates` instance from `devplacepy.templating` - all routers import from there, do NOT create their own.
- **Ports:** 10500 (dev), 10501 (tests; the serial suite uses a single uvicorn subprocess)
- **Username:** letters, numbers, hyphens, underscores only. 3-32 chars.
- **Password:** minimum 6 chars.
- **Avatars:** Multiavatar-based (local SVG generation, zero network). URL: `/avatar/multiavatar/{seed}`. Generation takes <5ms. Fallback to initial-based SVG on error. Cache is in-memory (cleared on restart).

## Routing

Routers in `devplacepy/routers/` form a **directory tree that mirrors the endpoint (URL) path**, the same convention as `tests/`. A single-resource domain is one flat file; a multi-sub-resource domain is a **package directory** split **one file per sub-resource** (not per endpoint). Each leaf has its own `router = APIRouter()` keeping the exact path strings; the package `__init__.py` aggregates them with `include_router`, and `main.py` mounts the domain at its prefix unchanged (a package exposes `.router` like a module). The leaf that owns the collection-root (`""`) route is the package base router (FastAPI rejects empty-path-under-empty-prefix), so `__init__` imports that leaf's `router` and includes the rest onto it; a router folded in from a deeper mount is included with a sub-prefix (`admin/__init__` includes `services.router` with `prefix="/services"` and `containers.router` with `prefix="/containers"`). Package-private helpers shared by leaves live in `_shared.py`. The `/projects` tree (project CRUD + `files.py` + `containers/`) and the `/admin` tree (every admin sub-resource + folded-in `services` and `containers`) are each mounted from a single package.

| Prefix | Router |
|--------|-------------|
| `/auth` | `routers/auth/` package (`signup`, `login`, `logout`, `forgotpassword`, `resetpassword`) |
| `/feed` | `routers/feed.py` |
| `/news` | `routers/news.py` |
| `/posts` | `routers/posts.py` |
| `/comments` | `routers/comments.py` |
| `/projects` | `routers/projects/` package (`index.py`, `files.py`, `containers/` subpackage) |
| `/profile` | `routers/profile/` package (`index`, `customization`, `notifications`, `usage`) |
| `/messages` | `routers/messages.py` |
| `/notifications` | `routers/notifications.py` |
| `/votes` | `routers/votes.py` |
| `/reactions` | `routers/reactions.py` |
| `/bookmarks` | `routers/bookmarks.py` |
| `/polls` | `routers/polls.py` |
| `/avatar` | `routers/avatar.py` |
| `/follow` | `routers/follow.py` |
| `/leaderboard` | `routers/leaderboard.py` |
| `/admin` | `routers/admin/` package (`index`, `users`, `aiusage`, `aiquota`, `media`, `trash`, `settings`, `notifications`, `news`, `auditlog`, `services`, `containers`) |
| `/issues` | `routers/issues/` package (`index`, `create`, `comment`, `status`) |
| `/gists` | `routers/gists.py` |
| `/admin/services` | `routers/admin/services.py` (folded into the `/admin` package) |
| `(none)` | `routers/seo.py` (`/robots.txt`, `/sitemap.xml`) |

## Content Rendering Pipeline

`ContentRenderer.js` processes all user-generated text in this exact order:

1. **Emoji shortcodes** → Unicode emoji (`:fire:` → 🔥, 80+ shortcodes)
2. **Markdown parse** → via `marked` with GFM tables, line breaks
3. **Sanitize** → `DOMPurify.sanitize` strips script/event-handler/iframe/`javascript:` payloads from the marked output
4. **Code syntax highlight** → `highlight.js` on all `<pre><code>` blocks
5. **Image URLs** → standalone `.jpg/.png/.gif` URLs become `<img>` tags
6. **YouTube URLs** → `youtube.com/watch?v=` or `youtu.be/` become embedded iframe players
7. **All URLs** → become `<a>` links with `target="_blank"` and `rel="noopener"`

**Sanitization is the XSS control.** Content is rendered client-side from `element.textContent`, so Jinja autoescaping does not protect it. `DOMPurify.sanitize` (vendored at `static/vendor/purify.min.js`, loaded `defer` in `base.html`) runs on the raw `marked` output before `processMedia` injects the trusted YouTube iframes, so user payloads are removed while our embeds survive. It is fail-closed: `render()` throws if `DOMPurify` is missing rather than emitting unsanitized HTML - never relax this into a `typeof` skip.

**Code blocks are protected** - `NodeIterator` skips `CODE`, `PRE`, `SCRIPT`, `STYLE` elements during URL/media processing, so source code in markdown code blocks is never touched.

Elements with `data-render` attribute are auto-rendered by `Application.js` on page load. The `.rendered-content` CSS class provides table styles, code block backgrounds, and image sizing.

## API documentation site (`/docs`)

`routers/docs/ package` serves the docs. `DOCS_PAGES` keeps curated prose pages (`kind: "prose"`, each with its own template under `templates/docs/<slug>.html`) and spreads `api_doc_pages()` from `devplacepy/docs_api.py` for every API reference page. Prose pages grouped by `section`: `General` (`index`, `getting-started`, `devii`, `dashboard`, `media-gallery`, `notification-settings`), admin-only `Devii internals` (`devii-*`), admin-only `Services` (`services-*`, documenting the `BaseService` framework and every background service including the live data at `GET /admin/services/data`), and admin-only `Production` (`production-*`, documenting the deployment below). A page is admin-gated by `"admin": True`; search/export pick it up automatically by slug and respect the admin flag - no extra wiring.

The sidebar groups `section`s under four ordered **audience tiers** (`AUDIENCES` in `routers/docs/pages.py`): `Start here` (General), `Build with the API` (API, Components, Styles), `Contribute and internals` (Architecture, Services, Devii internals, Bots internals, Testing, Claude Code), and `Operate` (Administration, Production). `nav_groups(pages)` builds the `[(audience, [(section, [pages])])]` tree from the flat visible-page list (so a section's pages collect under one heading regardless of `DOCS_PAGES` order or the API/Administration interleave from `api_doc_pages()`); `views.py` passes it as `nav`, and `docs_base.html` renders an audience super-header (`.sidebar-tier`) above each section subheading (`.sidebar-subheading`). `DOCS_PAGES` stays the canonical list for search/export/routing - the tiering is sidebar-only. The public `getting-started` page is the new-contributor on-ramp (install/run, the four-faces workflow, validation); its deep-internals links are gated `{% if is_admin(user) %}` so guests get no 404s. Each concept has one canonical home: the `auth` API group intro in `docs_api.py` defers method detail to the `authentication` prose page rather than re-listing the four methods.

- **Single source of truth:** `docs_api.py` `API_GROUPS` defines each reference page and its endpoints (method, path, auth, params, notes, sample_response). Add an endpoint there and the page, sidebar link, code examples, and interactive runner appear automatically.
- **Downloads** (`devplacepy/docs_export.py`): `/docs/download.md` returns the entire docs as one Markdown file; `/docs/download.html` returns a single **self-contained** HTML (vendored `marked` + `highlight.js` + theme inlined, the Markdown embedded base64 and rendered on open - works offline). Both are admin-filtered like the rest of the docs and reuse the same group/prose sources. Routes are declared before `/docs/{slug}.html` so `download.html` isn't caught by the slug pattern.
- **Docs search** (`/docs/search.html`): a backend-rendered BM25 search over all docs pages, in `devplacepy/docs_search.py`. The corpus = prose page text (template stripped of Jinja/HTML) + each API group's intro/endpoints + the live services group; the inverted index (postings + idf) is built **once** lazily (`get_index()`, cached module-global) so a query is just postings lookups (~120µs). `docs.py` special-cases `slug == "search"` (before the page registry) and renders `docs/_search.html` inside `docs_base.html` (`kind == 'search'`). Admin-only pages are filtered from results for non-admins, exactly like the sidebar. Snippets are HTML-escaped then wrapped in `<mark>` (XSS-safe). `_strip` removes `<script>`/`<style>` blocks and runs `_demarkdown` (drops headings, list/quote markers, table pipes, code fences, link/image syntax, and `` ` `` `*` `~` emphasis, but keeps `_` so identifiers like `owner_kind` stay searchable), so both the index and the snippets read as clean prose rather than raw markdown. The former `search` API group (user/recipient lookups) was renamed to slug **`lookups`** ("Search & Lookups") to free the `search` slug - keep that in mind if adding endpoints there.
- **Token substitution:** intros/notes/examples may use `{{ base }}`, `{{ username }}`, `{{ api_key }}` - these are substituted server-side by `render_group()` (plain string replace, not Jinja), because the registry is data, not template source.
- **`kind` branch:** `docs_base.html` renders api pages via `_api_page.html` → `_endpoint.html` per endpoint. Prose pages are rendered **server-side**: `docs.py` calls `docs_prose.render_prose(slug, ctx)` (`devplacepy/docs_prose.py`, mistune GFM with tables/strikethrough/url + `hard_wrap` to mirror the client `marked` config), which renders the template, converts the single `<div class="docs-content" data-render>` markdown block to HTML (after `html.unescape`, matching the client's `textContent` read), and **strips `data-render`**. `docs_base.html` outputs the result via `{{ prose_html|safe }}`. This eliminates the client markdown-to-HTML flicker and improves first paint. Anything outside that block (component live-demo blocks + their `<script type="module">`, the `devii.html` hero/CTA) is passed through verbatim. `hljs` highlighting and `CodeCopy` still run client-side over the now-server-rendered `<pre><code>` (text is already present, only colors/copy button appear after JS).
- **`data-render` destroys inner HTML** (it renders `textContent`). Only group-intro / prose markdown lives inside a `data-render` block; every endpoint card, `[data-api-tester]` mount, and component live-demo lives OUTSIDE it. Example markup shown inside a block must be HTML-escaped (`&lt;dp-...&gt;`) so neither the browser parser nor the server `html.unescape`+markdown step instantiates it.
- **`data-config` must be single-quoted:** `_endpoint.html` emits `data-config='{{ endpoint|tojson }}'`. `tojson` escapes `'`→`'`, so single quotes are safe; double quotes would break.
- **The reusable widget** is `static/js/ApiTester.js` (one instance per `[data-api-tester]`, wired by `ApiDocs.js` loaded in the docs `extra_js` block). It builds the param form, a **response-format picker**, live cURL/JS/Python tabs, a Send button that runs the real call, and a two-tab response area. It reads `window.DEVPLACE_DOCS` (`base`, `loggedIn`, `username`, `apiKey`, `isAdmin`) injected in `docs_base.html`. **Code blocks** are decorated by the shared `static/js/CodeBlock.js` (highlight via `hljs` + a line-number gutter + an always-visible Copy button); the widget calls `CodeBlock.refresh(pre)` on every tab switch (re-highlights - it clears `data-highlighted` first, the fix for stale tabs), the response panes use `CodeBlock.enhance(pre, {lineNumbers:false})`, and `CodeCopy.js` runs `CodeBlock.enhance` over prose `.docs-content pre`. Styling lives in `docs.css` (`.code-pre`/`.code-gutter`/`.code-has-copy`, `.format-picker`/`.format-option`, `.response-tabs`/`.response-pane`).
- **Response-format negotiation in the panel:** every endpoint dict carries a `negotiation` field, set by `_classify()` in `docs_api.py` (not hand-written): `"negotiable"` (page GETs + action POSTs - toggle JSON/HTML via the `Accept` header), `"ajax"` (votes/reactions/bookmarks/polls - JSON via `X-Requested-With: fetch`, HTML shows the redirect), `"json"` (always JSON), `"none"` (avatar/proxy/redirect - no body). The picker **defaults to JSON** everywhere; `ApiTester.headerPairs()` adds `Accept: application/json|text/html` accordingly and only sends `X-Requested-With` for ajax endpoints in JSON mode, so the live request **and** the generated snippets stay in sync. The **Expected** tab (always visible, default) renders the endpoint's `sample_response`; the **Live response** tab fills in after Send.
- **Runnable scope:** page GETs are now `interactive: true` (they get a Send button, gated by `ctx.loggedIn`/`ctx.isAdmin` for user/admin endpoints). Mutations use `destructive: true` (confirm dialog). Only `avatar`, `gateway-passthrough`, `notifications-open`, `push-register`, and `profile-regenerate-key` stay `interactive: false` (image/proxy/redirect/side-effecting) - they still show the Expected tab. Admin endpoints (`auth: "admin"`) only run for admins.
- **Enum params: pass `options`, never hardcode allowed values in prose.** A `field(... type="enum", options=[...])` auto-renders an "Allowed: a, b, c" line under the control in the live tester (`ApiTester.js` `buildParams`) and appends `Allowed: ...` to the Description column in the Markdown/HTML export (`docs_export.py` `_params_table`). Source enum lists from `devplacepy/constants.py` (`TOPICS`, `REACTION_EMOJI`) or the model `Literal`s (`PROJECT_TYPES`, `VOTE_TARGETS`) so docs stay in sync. Do NOT spell the values into the description - it would duplicate and drift.
- **Minimal role is documented and validated:** the `endpoint()` factory derives `min_role` from `auth` (`public`→Public, `user`→Member, `admin`→Admin); it is rendered as the "Minimal role:" badge in `_endpoint.html` and as `*Minimal role:*` in the Markdown/HTML export. The `auth` value MUST reflect the real enforcement: `tests/api/auth/matrix.py` drives every documented endpoint (incl. the dynamic services group) as anonymous, member, and admin and asserts the enforcement matches the doc - public allows anonymous, `user` rejects anonymous (401/login-redirect), `admin` rejects a non-admin member (403 /feed-redirect). It forces explicit roles to survive the `is_first`-becomes-Admin rule. If you add/relax a route's auth, update its `auth` in `docs_api.py` or this test fails. (Example caught by it: `/notifications/counts` uses `get_current_user` and returns zeros to guests, so it is documented `public`, not `user`.)
- **Admin-only pages:** a group with `"admin": True` (currently `services`, `admin`) is hidden from the sidebar (`docs.py` filters `visible_pages`) and returns 404 for non-admins (`page.get("admin") and not is_admin → not_found`, same message as unknown slugs). `index.html` wraps the operator links in `{% if user and user.get('role') == 'Admin' %}`.
- **Background Services page is dynamic:** the `services` group is a placeholder (`"dynamic": True`, empty endpoints). `docs.py` branches on `dynamic` and calls `build_services_group(service_manager.describe_all(), base)` to generate it live from the registered services and their `ConfigField` specs (one `POST /admin/services/{name}/config` card per service, `*_enabled` fields excluded). Add a service to `main.py` startup and it documents itself - keep `docs_api.py` import-pure (no `describe_all()` at import time).

## Runtime data layout and single source of truth (`config.DATA_PATHS`)

Every runtime/user-generated artifact lives under one root, `config.DATA_DIR` (`DEVPLACE_DATA_DIR`, default `<repo>/data`), **outside the package and never under `/static`**. `config.py` is the single source of truth: it derives every runtime path from `DATA_DIR` and lists them in the `DATA_PATHS` registry, and `ensure_data_dirs()` creates the whole tree. Layout: `data/devplace.db` (+ `-wal`/`-shm`), `data/devii_tasks.db`, `data/devii_lessons.db`, `data/uploads/{attachments,project_files}/`, `data/container_workspaces/`, `data/zips/`, `data/zip_staging/`, `data/fork_staging/`, `data/keys/` (the three VAPID PEMs + `.vapid.lock`), `data/bot/` (`state_slot*.json` + `article_registry.json`), `data/locks/` (`devplace-services.lock`, `devplace-init.lock`).

Rules when touching this area:
- **Never compute a runtime path from scratch.** Import the constant from `config` (`UPLOADS_DIR`, `ATTACHMENTS_DIR`, `PROJECT_FILES_DIR`, `ZIPS_DIR`, `ZIP_STAGING_DIR`, `FORK_STAGING_DIR`, `KEYS_DIR`, `BOT_DIR`, `LOCKS_DIR`, `CONTAINER_WORKSPACES_DIR`, `DEVII_TASKS_DB`, `DEVII_LESSONS_DB`, `VAPID_*_FILE`, `SERVICE_LOCK_FILE`, `INIT_LOCK_FILE`). A new runtime dir goes in `DATA_PATHS` (it is then auto-created), nowhere else.
- **`ensure_data_dirs()` runs early.** It is called at the top of `main.startup()` (before the init/service `flock`s, which now live under `data/locks/`) and at the top of `database.init_db()`; `database.py` also calls it plus mkdirs the resolved DB parent **at import time, before `dataset.connect`**, because the default DB now lives under `DATA_DIR` and SQLite will not create the parent dir. Do not move the DB connect ahead of that guard.
- **Uploads moved physically only.** They live in `data/uploads/` but are still served at the unchanged `/static/uploads/` URL: the FastAPI `UploadStaticFiles` mount points at `config.UPLOADS_DIR`, and nginx aliases `/static/uploads/` to a mounted `/data/uploads/` (base compose mounts `${DEVPLACE_DATA_DIR:-./data}/uploads`; the containers overlay overrides it to the overlay's data dir - compose merges service volumes by container target path, last file wins). The URL-builder string literals (`/static/uploads/...`) and the DB-stored relative values (`directory` + `stored_name`) are unchanged, so **no DB rewrite is needed**.
- **Relocating an old install:** `devplace migrate-data` (`cli.py`) relocates legacy artifacts (repo-root DBs/keys/locks, `devplacepy/static/uploads/`, `var/`, `~/.devplace_bots`, `~/.dpbot_article_registry.json`) into `data/`. It is idempotent and per-file safe (copy -> fsync -> verify size+CRC -> atomic `os.replace` -> unlink source); DBs are checkpointed (`wal_checkpoint(TRUNCATE)`) and refused if a writer holds the lock (`BEGIN IMMEDIATE` probe) - stop the app first. `--dry-run` previews. The main DB is only moved when `DATABASE_URL` is still the default; the devii DBs only when their env vars are unset.

## Production deployment

Single-host Docker Compose (`docker-compose.yml`): an **app** container (Uvicorn, 2 workers) behind an **nginx** container. Full operator reference is the admin-only `Production` docs section (`templates/docs/production*.html`); the conventions that must not regress:

- **Shared DB and files = same as dev.** The app container bind-mounts the host project root (`.:/app`) and runs as `${DEVPLACE_UID}:${DEVPLACE_GID}` (default `1000`), so it reads/writes the same `data/devplace.db`, `data/uploads/`, `data/devii_*.db`, `data/keys/` (VAPID), and `data/locks/devplace-services.lock` as `make dev`. No `DEVPLACE_DATABASE_URL` override - `config.py` resolves an absolute path under the project's `data/` dir. WAL + the `flock` on `devplace-services.lock` make concurrent dev/prod safe and keep a single background-services owner. SQLite is local-file, so prod and dev must be the **same host**.
- **Code updates need no rebuild** (bind-mounted source); rebuild only on `pyproject.toml` dependency changes. `.env` is git-ignored; `.env.example` is the committed template. The app reads `DEVPLACE_DATABASE_URL`, never `DATABASE_URL`.
- **nginx parity rules** (`nginx/nginx.conf.template`, rendered by `start.sh` via `envsubst` with an allow-list that preserves `$http_upgrade`): `/static/uploads/` must re-apply `nosniff` + a `Content-Disposition` via the `map $uri $upload_disposition` block (`inline` for safe image/video/audio extensions, `attachment` otherwise), mirroring `UploadStaticFiles.INLINE_MEDIA_EXTENSIONS` - an XSS control nginx would otherwise bypass, and the inline branch is what lets video play in production; `/devii/ws` needs the `map $http_upgrade $connection_upgrade` block and `Upgrade`/`Connection` headers or the Devii terminal cannot connect; `client_max_body_size` comes from `NGINX_MAX_BODY_SIZE` (default `50m`) and must be `>= max_upload_size_mb` or uploads 413. nginx serves `devplacepy/static` via a read-only bind mount, so assets stay current without an image rebuild.

## Timezone-aware date display (`local_dt`/`dt_ago` + `static/js/LocalTime.js`)

Every user-facing **instant** is rendered in the **viewer's own browser timezone**, client-side. Timestamps stay stored and emitted as UTC ISO; localization happens in the browser (browsers expose their zone via `Intl`, so no timezone cookie/header is needed and it is automatic per user).

- **Server side:** the Jinja globals `local_dt(iso, mode="datetime")` and `dt_ago(iso)` (`templating.py`) emit `<time datetime="{normalized UTC ISO}" data-dt data-dt-mode="date|datetime|ago">{server-fallback}</time>` Markup. The `<time>` text content is the server-rendered fallback (`format_date`/`time_ago`) that no-JS clients see; `local_dt` normalizes a naive ISO to UTC, returns `""` for empty input, and returns the escaped raw string for an unparseable value. `mode`: `date` (DD/MM/YYYY), `datetime` (DD/MM/YYYY HH:MM), `ago` (relative).
- **Client side:** `static/js/LocalTime.js` (`app.localTime`, instantiated in `Application.js`) reformats every `[data-dt]` to the browser-local string via `Intl` (`toLocaleString`/`toLocaleDateString`, `ago` computed against `Date.now()`), sets a full-datetime `title`, runs a `MutationObserver` on `document.body` so dynamically-inserted dates localize automatically, and refreshes `data-dt-mode="ago"` elements every 60s. JS that injects dates (e.g. `MessagesLayout` bubbles) builds a `<time ... data-dt data-dt-mode="ago">` element and either calls `app.localTime.apply(el)` or relies on the observer.
- **`format_date()`/`time_ago()` stay plain-text helpers** (`utils.py`) for JSON responses, the no-JS fallback inside `<time>`, and the dates that must NOT be localized: **non-timestamp** user-entered dates (project `release_date`/`demo_date` are DD/MM/YYYY strings, not instants) and **attribute contexts** (`value="..."`, `title="..."`, where injecting a `<time>` tag would break the markup). Do NOT re-point the `format_date` global to emit `<time>` - it is used in those attribute/non-instant spots.
- **Conversion pattern** for a precomputed `time_ago` string display: `{{ dt_ago(x.created_at) if x.created_at else x.time_ago }}` - localizes when the raw ISO is present, falls back to the original precomputed string otherwise (so a wrong/missing field never blanks the date). New user-facing timestamp displays use `local_dt`/`dt_ago` directly. Converted surfaces include the shared `_post_header.html`/`_post_card.html` (feed + profile posts), `_comment.html` (all comments), messages (template + `MessagesLayout.js` live bubbles), gists/news/notifications/saved/landing/leaderboard lists, the post/news/gist detail pages, and the static admin/issue timestamp displays; the JS-live admin service tables keep server-time until their poller is converted too.

## Static asset caching and versioning (`config.STATIC_VERSION`, `static_url`, `assetUrl`)

Static assets are served `public, immutable, max-age=31536000` (1 year) for the Lighthouse "efficient cache policy" / SEO score, while a restart still busts every browser. The reconciler is a **boot-time version path segment**: `config.STATIC_VERSION` (env `DEVPLACE_STATIC_VERSION`, else `int(time.time())` at process start) is injected after `/static`, so `/static/css/base.css` is emitted as `/static/v<ts>/css/base.css`. A deploy = a restart = a new timestamp = a new URL for every asset.

- **Path segment, never a query string.** The frontend is unbundled ES6 with ~130 **relative** imports (`./Http.js`) and zero absolute ones. A `?v=` only versions the entry `<script>`; its transitive relative imports would resolve to unversioned URLs and stay frozen under `immutable`, so a deploy would not propagate JS for a year. A path segment is inherited automatically by the entire module graph and by relative CSS `url()`, so everything busts together. Do NOT switch this to a query param.
- **Helpers (reuse, never hardcode a `/static/` literal):** templates use the `static_url(path)` Jinja global (`templating.py`); runtime JS that builds an absolute static URL uses `assetUrl(path)` from `static/js/assetVersion.js`, which reads `<meta name="asset-version">` rendered in `base.html` (also a `static_version` Jinja global). Both no-op for non-`/static/` paths and for `/static/uploads/`, so they are safe to wrap around dynamic values (`static_url(og_image or '/static/og-default.png')`). Relative ES6 imports and relative CSS `url()` need neither - they inherit the version from the importing URL.
- **Serving:** `main.py` mounts a `CachedStaticFiles` at `/static/v{STATIC_VERSION}` (immutable 1y; `service-worker.js` -> `no-cache`) plus the plain `/static` mount as the unversioned fallback; nginx mirrors this with `location ~ ^/static/v\d+/` (immutable 1y) and a short `max-age=3600` on the unversioned `/static/`.
- **Exclusions:** `/static/uploads/` (user content, stable DB paths) is never versioned; `service-worker.js` keeps a stable unversioned URL with `no-cache` (its `/service-worker.js` route already sets this) so its registration scope is stable and a new worker always deploys.
- **Multi-worker:** the version is captured **once at launch** and shared via `DEVPLACE_STATIC_VERSION` (set in `make prod` = `$(date +%s)` and the Dockerfile's `sh -c` CMD), otherwise two workers could compute timestamps a second apart and serve mismatched versioned mounts (a 404 on the other worker). Unset in dev (`--reload`) so each reload refreshes it. Docs: `/docs/static-caching.html`.

## Stealth HTTP client (`devplacepy/stealth.py` + `devplacepy/curl_transport.py`) - the ONLY outbound HTTP client

Every outbound HTTP request the server makes goes through the **stealth client**, which reproduces a real Google Chrome desktop across all three fingerprint layers an anti-bot vendor checks - and they must agree, or a cross-layer mismatch is itself the tell:

- **TLS (JA3/JA4)** and **HTTP/2 (Akamai fingerprint)** come from **`curl_cffi`** (a Python binding to curl-impersonate, BoringSSL). `curl_transport.CurlTransport` is an `httpx.AsyncBaseTransport` that routes the actual bytes through `curl_cffi.requests.AsyncSession(impersonate="chrome146")`, so the TLS ClientHello (extension order, supported-groups, sig-algs, GREASE) and the HTTP/2 SETTINGS/WINDOW_UPDATE/pseudo-header-order/PRIORITY behaviour are **byte-identical to Chrome 146** (verified live: JA4 `t13d1516h2_...`, Akamai H2 hash `52d84b11737d980aef856699f885ca86`). Pure `httpx`/`h2` could never match these (OpenSSL can't reorder TLS extensions; `h2` doesn't expose the SETTINGS/window/pseudo-header knobs).
- **Application headers** (`user-agent`, `sec-ch-ua*`, `accept`, `accept-encoding`, `sec-fetch-*`) are owned by the same `impersonate` profile, so they match the TLS/H2 layer exactly.
- **Cleartext `http://` is forced to HTTP/1.1** (`curl_transport.http_version_for`): the `chrome146` profile defaults to HTTP/2, but a plaintext origin has no TLS/ALPN to negotiate h2, and real Chrome uses HTTP/1.1 for `http://` anyway. This is **load-bearing for the internal gateway hop**: every internal AI consumer (Devii, news, DeepSearch, SEO, the dbapi NL designer) calls `config.INTERNAL_GATEWAY_URL` = `http://localhost:<port>/openai/v1/...`, which is served by **uvicorn (HTTP/1.1 only)**. curl's cleartext HTTP/2 attempt makes uvicorn return `400 "Invalid HTTP request received."` once the body crosses ~128KB (e.g. a large Devii system-prompt + tool list), so without the override a big Devii turn fails on every request. `https://` keeps the impersonation default (h2 over TLS), so the external fingerprint is unchanged. Regression: `tests/unit/curl_transport.py`.

**The whole thing hides behind one factory that still returns an `httpx.AsyncClient`, so every call site uses the ordinary httpx API unchanged** (`get`/`post`/`stream`/`build_request`+`send`, `aiter_bytes`, `raise_for_status`, `response.json()`, `httpx.HTTPError`/`TimeoutException`). The adapter maps curl_cffi <-> httpx: it strips `user-agent`/`accept-encoding`/hop headers from the outbound request (curl's `impersonate` owns them) while forwarding per-call `Authorization`/`Content-Type`/custom headers; it strips `content-encoding`/`content-length` from the response (libcurl already decompressed); it forces `allow_redirects=False` so **httpx drives redirects** (which keeps the SSRF guard re-checking every hop); and it maps curl timeouts/errors back to `httpx.ConnectTimeout`/`httpx.ConnectError`. If `curl_cffi` is unavailable the factory **falls back** to a pure-httpx Chrome-aligned transport (`stealth.chrome_ssl_context()` + HTTP/2 + `chrome_base_headers()`); `stealth.curl_active()` reports which path is live.

**Hard rule: never instantiate a bare `httpx.AsyncClient(...)` / `httpx.Client(...)` for outbound traffic.** Build it through the factory. It accepts the **same kwargs** as the httpx client it wraps (`timeout`, `limits`, `follow_redirects`, `base_url`, `headers`, `proxy`, ...); `headers=` merge so per-call `Authorization`/`Content-Type` win.

- `stealth.stealth_async_client(**kwargs) -> httpx.AsyncClient` - the default; curl-backed when available. Use everywhere an async client was created.
- `stealth.stealth_sync_client(**kwargs) -> httpx.Client` - synchronous call sites only (the container `_http_probe` diagnostic, the optional `services/bot/` subsystem). **This path stays pure-httpx Chrome-aligned** (no curl_cffi sync adapter yet); it is low-value internal/optional traffic. Extend with a sync `CurlTransport` if those ever need full parity.
- `stealth.stealth_transport()` - the active async transport (curl or httpx fallback), used by the SSRF guard as its inner transport; `stealth.chrome_ssl_context()` / `chrome_base_headers()` back the fallback path.
- `stealth.ChromeStealthClient` - the rich pure-httpx per-request-type client (navigation vs sub-resource vs API `Sec-Fetch-*`, streaming `download_files`/`download_bytes`); a self-contained reference variant, not curl-backed.
- `curl_transport.IMPERSONATE_TARGET` (`"chrome146"`) - bump this string when curl_cffi ships a newer Chrome target (`curl_cffi.requests.BrowserType`).

**SSRF guard composes with stealth automatically.** `net_guard.guarded_async_client(**kwargs)` wraps `stealth.stealth_transport()` in `_GuardedTransport` (composition, not inheritance), so every guarded external fetch (DeepSearch crawl, SEO crawler, Devii fetch) gets the curl fingerprint AND the per-hop `guard_public_url` check (redirects are httpx-driven, so each hop is re-guarded).

**The one deliberate exclusion is `routers/proxy.py`** (the `/p/<slug>` public ingress reverse proxy). It relays the **end user's own** request headers verbatim to a private container address; injecting a Chrome fingerprint would corrupt the forwarded request and is pointless against internal infrastructure. Playwright (the SEO/DeepSearch headless-browser crawls) is a real browser and is out of scope - stealth only governs the httpx/curl path. Dependencies: `curl_cffi`, `httpx[http2]`, `brotli`, `zstandard` (declared in `pyproject.toml`).

## Multi-worker concurrency (preferred rules)

`uvicorn --workers N` = N independent processes sharing only the filesystem and SQLite DB. Module-global caches/counters are per-process, so a local `clear()` is invisible to siblings. Full reference: admin docs `Production -> Multi-worker and concurrency` (`templates/docs/production-concurrency.html`). Enforce these:

- **In-process caches are never authoritative across workers.** Coordinate invalidation through the `cache_state` version table in `database.py`: call `bump_cache_version(name)` at every write path and `sync_local_cache(name, cache)` before every read path. Existing names: `auth` (guards `_user_cache`; bumped by `clear_user_cache`/`clear_session_cache` on logout, ban, role/password change) and `settings` (guards `_settings_cache`; bumped by `set_setting`/`clear_settings_cache`). The version itself is read uncached (a single indexed primary-key lookup on the tiny `cache_state` table), so a bump in one worker is seen by every other worker on its next read - invalidation is immediate, not eventually consistent. The version read/bump fail open (logged, never raise). When you add a per-process cache whose staleness matters, give it a name and wire both calls - do not invent a second invalidation mechanism.
- **Authoritative state lives in the DB**, memory is only a short-TTL accelerator (sessions, the Devii 24h cap via `devii_usage_ledger`, cost counters).
- **Global rates/quotas are worker-count-aware or DB-backed.** The rate limiter enforces `ceil(limit / DEVPLACE_WEB_WORKERS)` per process so the aggregate matches the configured value with no per-request write. `DEVPLACE_WEB_WORKERS` MUST equal the real `--workers` (set in `make prod`=2 and the Dockerfile=2; default 1 for dev). Update it whenever you change worker count.
- **First-boot side effects are flock-serialized and idempotent.** `init_db()` runs under a blocking `flock` on `devplace-init.lock` (the seed-if-missing inserts would otherwise race to duplicate rows); `ensure_certificates()` early-returns when keys exist and otherwise generates under `.vapid.lock`. Any new run-once-at-startup work follows the same pattern (flock + exists-check / `INSERT OR IGNORE`).
- **Run-once background work stays behind the services lock**, never per-worker: gate it on `service_manager.owns_lock()` (held via `devplace-services.lock`), as news/bots/Devii hub do.

## Async job services (`services/jobs/`)

The standard for heavy, blocking work that must run off the request path and hand back a result URL. Reference: admin docs `Architecture -> Async job services` and `Services -> ZipService`.

- **The DB row is the queue.** `services/jobs/queue.py` (`enqueue`, `get_job`, `touch_job`, `list_jobs`) is pure DB and callable from any worker's handler - so enqueue is a fast, non-blocking insert, status reads work from any worker, and only the lock owner processes. All kinds share one `jobs` table discriminated by `kind`; kind-specific fields live in the `result` JSON, common stats in `bytes_in`/`bytes_out`/`item_count` columns.
- **`JobService(BaseService)`** runs only in the lock owner. Each `run_once`: reap finished in-flight tasks -> recover orphaned `running` rows (a previous owner's, since there is only one processor) back to `pending` with bounded `retry_count` -> refill up to `max_concurrent` oldest `pending` jobs (uuid7 sorts FIFO) as `asyncio` tasks -> sweep rows past `expires_at` via the per-kind `cleanup()` hook. In-flight tasks span ticks (kept in an instance map); the loop returns immediately each tick, so keep the interval short (default 2s).
- **No atomic claim is needed** (single processor by construction) and **no separate reaper** exists (retention is built into every job service; default 7 days, admin-configurable; downloading extends `expires_at` via `touch_job`).
- **Add a kind:** subclass `JobService`, set `kind`, implement `async process(self, job) -> dict` (return the `result` dict incl. stat keys) and `cleanup(self, job)`; register in `main.py`; add enqueue endpoints that own authz, a status route, a download route, a Devii tool, and docs.
- **`ZipService`** (`zip_service.py`) materializes a project subtree with `project_files.export_to_dir` into `config.DATA_DIR/zip_staging/{uid}`, compresses it in a stdlib-only subprocess (`zip_worker.py`, prints stats JSON incl. crc32), writes `{crc32}.{slug}.zip` under `config.DATA_DIR/zips/`, and removes staging. Artifacts live in `DATA_DIR` (`data/`, outside the package, not served by `/static`); the download is a `FileResponse` route, not the static mount. Downloads are **capability URLs** (`/zips/{uid}/download`), scoped only by the unguessable uuid7 - matching publicly-viewable projects; the owner is stored for attribution, not access control. Frontend `app.zipDownloader` auto-wires `data-zip-download` elements and the files context menu; its status poll + download trigger run through the shared `JobPoller` (see "Shared frontend utilities").
- **`PlanningReportService`** (`planning_service.py`, kind `planning`) is **admin-only** and builds a grouped, ordered markdown planning report of all open Gitea tickets, off the request path via the same async-job pattern as zip. `process`: pages `runtime.get_client().list_issues(state="open", limit=50)` until exhausted (cap 50), builds the markdown via `services/gitea/planning.py` `generate_plan(issues, config)` (an AI-or-fallback helper mirroring `enhance.py`: a `SYSTEM_PROMPT` that groups tickets into level-2-heading workstreams with a numbered, rationale-bearing list plus a flat `## Suggested Order`, posted to the internal gateway via `stealth.stealth_async_client`; fail-soft `_fallback` groups by primary label then ticket number when `ai_enhance` is off or any `httpx.HTTPError/ValueError/KeyError/IndexError` occurs), writes the markdown to `config.PLANNING_REPORTS_DIR/_directory_for(uid)/{crc32}.{slug}.md` (sharded on the uuid7 random tail, crc32 over the markdown bytes), and returns `{download_url, local_path, final_name, markdown, ai_used, item_count, bytes_out}`. The `markdown` string is carried in the result so the status route can hand it to the renderer without re-reading disk. `cleanup` unlinks the file (retention prunes only the artifact + job row). Audit `issue.planning.request` (enqueue) and `issue.planning.generate` (success/failure) via `record`/`record_system` with an `audit.job(uid)` link. Routes (all `require_admin`): `POST /issues/planning` (enqueue, `503` when Gitea unconfigured, returns `{uid, status_url}`), `GET /issues/planning/{uid}` (`PlanningJobOut`: status, `markdown`, `download_url`, `issue_count`, `ai_used`), `GET /issues/planning/{uid}/download` (`FileResponse`, `text/markdown`, traversal-guarded against `PLANNING_REPORTS_DIR`, extends retention via `touch_job`). The `/issues/{number}` detail route uses the `{number:int}` path converter so `/issues/planning` is never captured by it. Admin entry point: `GET /admin/issues/planning` (`routers/admin/issues.py`, `admin_issues_planning.html`) hosts the **Generate planning** button, the `JobPoller` status panel, the `<dp-content>` render target, and a **Download** anchor; the issues listing shows an admin-only **Generate planning** link (`viewer_is_admin` in `issues_page`). Frontend driver `static/js/PlanningGenerator.js` (`app.planningGenerator`): POST -> `JobPoller.run` -> replace the `<dp-content>` with a fresh one holding `status.markdown` so it re-renders, set the download href. Devii tool `planning_report_generate` (`requires_admin=True`). The renderer's built-in copy button (the `dp-content` **Copy** button styled in `markdown.css`, identical to the per-code-block copy button) satisfies the copy-source requirement and is opt-out via the `no-copy` boolean attribute on `<dp-content>`.

## Project fork (`services/jobs/fork_service.py`)

Forking copies a source project into a brand-new project owned by the forking user, off the request path via the **same async-job pattern as the zip flow**.

- **Enqueue:** `POST /projects/{slug}/fork` (`routers/projects/index.py`) - `require_user` then `can_view_project(source, user)` (any logged-in user may fork any project they can view: public projects and their own private ones). Body is `ForkForm{title}` (the destination project name). It enqueues a `fork` job (`{source_project_uid, title, forked_by_uid}`, owner `("user", uid)`) and returns `{uid, status_url}`. No work happens on the request path.
- **`ForkService`** (kind `fork`) `process`: looks up the source project row and the forking user row (raises -> job `failed` if either is missing), creates the destination project with `content.create_content_item("projects", "project", user, fields, ...)` (so slug/XP/logging match a normal create; metadata - `description`, `project_type`, `platforms`, `status`, dates, `is_private` - is copied from the source, `read_only` reset to 0), copies the whole virtual FS off-thread (`export_to_dir(source, "", staging)` -> `import_from_dir(new_uid, staging, user, skip_names=set())`, an **exact** copy including binaries, staging under `config.DATA_DIR/fork_staging/{uid}`), then records the relation with `database.record_fork(source_uid, new_uid, forked_by_uid)`. Result: `{project_uid, project_url, source_project_uid, item_count}`.
- **Rollback:** any failure after the project is created triggers `_rollback(new_uid)` (`project_files.delete_all_project_files` + `delete_fork_relations` + delete the `projects` row) before re-raising, so a failed fork leaves no orphan project.
- **Cleanup is a no-op on the project.** Unlike zip's disposable archive, a fork's artifact is a **permanent project** that must outlive the job. `cleanup()` only removes any leftover staging dir; the retention sweep deletes the job tracking row, never the forked project. `devplace forks prune|clear` likewise deletes job rows only.
- **Relation model:** the `project_forks` table (`uid`, `source_project_uid`, `forked_project_uid`, `forked_by_uid`, `created_at`) records direction explicitly (source -> forked), indexed on both project columns. Helpers in `database.py`: `record_fork`, `get_fork_parent(forked_uid)` (the source project row, for the "Forked from X" link on the project detail page), `count_forks(source_uid)`, and `delete_fork_relations(project_uid)` (called on project delete in `content.delete_content_item`, and in fork rollback).
- **Frontend:** `app.projectForker` (`static/js/ProjectForker.js`) wires `data-fork-project` (a Fork button on the project detail page, visible to any logged-in user): prompt for a name via `app.dialog.prompt` -> `Http.sendForm` POST -> `JobPoller.run("/forks/{uid}", ...)` -> on `done` redirect to `project_url`.
- **Status route** `GET /forks/{uid}` (`routers/forks.py`, `ForkJobOut`) exposes `project_uid`/`project_url`/`source_project_uid` only once `status == done`. Devii tools `fork_project`/`fork_status`; docs `projects-fork`/`forks-status`.

## SEO Diagnostics tool (`services/jobs/seo/`, `routers/tools/`)

The public **Tools -> SEO Diagnostics** auditor crawls a URL or sitemap with a headless browser and runs a broad battery of SEO checks, on the **same async-job pattern as zip/fork** plus a live websocket. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap, and the shared SSRF guard.

- **Surface:** a collapsible **Tools** dropdown in `base.html` (desktop center nav + a mobile section, visible to everyone) toggled by `MobileNav.initToolsDropdown`. `GET /tools` lists tools; `GET /tools/seo` is the auditor page (`static/js/SeoDiagnostics.js` -> `app.seoDiagnostics`, instantiated page-side in the template, not in `Application.js`).
- **Enqueue:** `POST /tools/seo/run` (`routers/tools/seo.py`, body `SeoRunForm{url, mode: url|sitemap, max_pages 1-50}`). Owner is `("user", uid)` or `("guest", X-Real-IP)`. It rejects with `429` if the owner already has a pending/running `seo` job, then enqueues `{url, mode, max_pages, allow_private:False}` and returns `{uid, status_url, ws_url}`.
- **`SeoService`** (kind `seo`, `service.py`) `process`: writes the payload to `config.SEO_REPORTS_DIR/{uid}/payload.json`, launches `python -m devplacepy.services.jobs.seo.worker <payload_json> <output_dir>` via `create_subprocess_exec` (high `limit=` so big lines never overflow the StreamReader), reads **NDJSON frames from stdout** line by line, forwards each into the in-process **`ProgressHub`** (`progress.py`), and on completion loads `output_dir/report.json` as the job result. `cleanup()` clears the hub buffer and removes the report dir.
- **Worker** (`worker.py`, subprocess): `crawler.crawl_target` resolves the target (single URL, or sitemap `<loc>` URLs capped at `max_pages`) and fetches `robots.txt`/`sitemap.xml`/`llms.txt` with `httpx`. The audited target host is guarded once with `net_guard.guard_public_url`; candidate URLs **sharing that host are pre-approved** (no redundant per-URL `getaddrinfo` - a transient DNS failure or a self-hosted server resolving its own domain must not blank the whole crawl), and only cross-host sitemap entries are re-guarded. **In sitemap mode the crawler never falls back to auditing the sitemap document itself**: if no page URLs survive it raises a clear error (a stray `or [target]` fallback previously rendered the sitemap XML as one 56k-node "page" with no title/H1). For each page it launches one Playwright navigation: a single `page.evaluate(EXTRACT_SCRIPT)` returns the whole DOM contract (title/metas/canonical/headings/images/links/jsonld/og/twitter/semantic/mixed-content/word-count), an injected `PerformanceObserver` (`add_init_script(INIT_SCRIPT)`) captures LCP/CLS, navigation timing gives TTFB/FCP/transfer/protocol, a mobile-viewport pass measures overflow/tap-targets, a screenshot is saved, and a raw `httpx` GET supplies the SSR HTML for the rendered-vs-server parity check. It emits `stage|target|progress|page_loaded|page|site_checks|report_ready` frames and writes the full `report.json`.
- **Check registry** (`checks/`): `base.py` defines the `Check`/`PageContext`/`SiteContext` dataclasses and the `@page_check`/`@site_check` decorators (collected into `PAGE_CHECKS`/`SITE_CHECKS`); one module per category (`crawl`, `meta`, `headings`, `links`, `structured_data`, `social`, `performance`, `mobile_a11y`, `security`, `ai_readiness`, `crosspage`). `registry.run_page_checks`/`run_site_checks` run them defensively (one failing check never aborts a page), and `compute_score` produces a severity-weighted overall score + grade and per-category subscores. **To add a check:** write a function decorated `@page_check`/`@site_check` in the right category module and import that module in `registry.py`.
- **Live progress WS:** `WS /tools/seo/{uid}/ws` is served **only by the service-lock owner** (closes `4013` for a fast retry on a non-owner worker, like `/devii/ws`); it replays `hub.snapshot(uid)` then streams `hub.subscribe(uid)`. The frontend `SeoProgressSocket` mirrors the `DeviiSocket` 4013/reconnect pattern. **The `done` frame carries the full report inline** (and the router's late-join terminal branch reads it from `job.result`); the client renders from `frame.report` directly. This is load-bearing: `service.process` publishes `done` from inside `process()`, BEFORE the JobService base persists the result on the next reap tick (~2s later), so a client that fetched `/tools/seo/{uid}/report` on the `done` signal would race the DB write and get an empty (all-zero) report. Do not "simplify" this back into a fetch-on-done. After publishing `done`, `process()` calls `hub.clear(uid)` to bound the in-memory buffer (late reconnects fall back to the DB-backed terminal branch).
- **Status/report routes:** `GET /tools/seo/{uid}` (`SeoJobOut`), `GET /tools/seo/{uid}/report` (`respond(..., SeoReportOut)`, HTML or JSON), `GET /tools/seo/{uid}/screenshot/{n}` (FileResponse from `SEO_REPORTS_DIR`, path-guarded). All are **capability URLs** scoped by the unguessable uuid7. Devii tools `seo_diagnostics`/`seo_status`/`seo_report` (public); docs `tools-seo-*`; CLI `devplace seo prune|clear`; audit `seo.run.request|complete|failed` (category `tools`).
- **SSRF guard is shared:** `devplacepy/net_guard.py` (`guard_public_url`, `is_blocked_address`, `effective_address`) was extracted from the Devii fetch controller, which now imports it; the crawler reuses it. **`playwright` is a core dependency** (Chromium installed by `make install` and the Docker image).
- **Production nginx needs a dedicated WS location.** In `nginx/nginx.conf.template` the catch-all `location /` sets `Connection ""` (no upgrade) and a 60s timeout, so any websocket route that falls through to it fails the handshake (browser: `WebSocket connection failed`, no close code). The progress socket has its own `location ~ ^/tools/seo/[^/]+/ws$` block forwarding `Upgrade`/`Connection` with a 1h timeout, mirroring `/devii/ws`. **Any new websocket path must add its own upgrade `location` above `location /`** (see [[nginx-websocket-location]] and docs `Production -> nginx`).

## DeepSearch tool (`services/jobs/deepsearch/`, `services/deepsearch/`, `routers/tools/deepsearch.py`)

The public **Tools -> DeepSearch** researcher is a multi-agent deep web researcher built on the **same async-job + ProgressHub + 4013-WS pattern as the SEO tool**, plus a per-session vector store and a grounded RAG chat. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap (1-30), depth cap (1-4), and the shared SSRF guard. Reuse the SEO tool as the template for any new Tools async job.

- **Owner helper is shared:** `routers/tools/_shared.py` `owner_for(request)` returns `("user", uid)` or `("guest", X-Real-IP)`; both `seo.py` and `deepsearch.py` import it (do not re-inline the owner derivation).
- **Enqueue:** `POST /tools/deepsearch/run` (body `DeepsearchRunForm{query, depth 1-4, max_pages 1-30}`). It rejects with `429` if the owner already has a pending/running `deepsearch` job. It resolves the **logged-in user's `users.api_key`** (guests use `database.internal_gateway_key()`) into the job payload for per-user embedding/LLM spend attribution, generates the uid up front, writes a `deepsearch_sessions` row (`create_deepsearch_session`), enqueues the job carrying `{query, depth, max_pages, api_key, collection}`, and returns `{uid, status_url, ws_url}`. The enqueue uses a local `_enqueue` (not `queue.enqueue`) so the session uid and the job uid match.
- **`DeepsearchService`** (kind `deepsearch`, `service.py`) `process`: writes `control.json` (state `running`) + `payload.json` (augmented with the cross-session `cached_hashes`) under `config.DEEPSEARCH_DIR/{uid}`, launches `python -m devplacepy.services.jobs.deepsearch.worker <payload_json> <output_dir>` via `create_subprocess_exec` (high `limit=`), pumps **NDJSON stdout frames** into the in-process `ProgressHub` (`progress.py`), and on completion loads `output_dir/report.json`, persists the URL cache (`upsert_deepsearch_url_cache`), and updates the session row. `cleanup()` **drops the ChromaDB collection** (`VectorStore.drop`) and removes the session dir.
- **Worker pipeline** (`worker.py`, stdlib + httpx + playwright, importable subprocess): `enhance.plan_queries` (gateway -> JSON sub-queries, deterministic fallback) -> `crawl.search_queries` (rsearch via standalone httpx, never `PlatformClient`) -> `crawl.crawl` (httpx fetch then playwright render fallback, `guard_public_url` on the URL and every redirect, content-hash + URL-hash dedup) -> `chunking.chunk_text` -> `embeddings.embed_texts` (gateway, **local hashing fallback** when unavailable) -> `store.VectorStore.add` (Chroma) -> `orchestrate.orchestrate` (summarizer/critic/linker agents producing findings+confidence+gaps+score+source_diversity, heuristic fallback). The worker writes `report.json` and `url_cache.json` and emits a `report_ready` frame.
- **PDF ingestion (`crawl.py` `fetch_page` + `pdf.py`):** a crawled candidate is treated as a PDF when its `content-type` is `application/pdf`/`application/x-pdf`, its URL path ends in `.pdf`, or its first bytes match the `%PDF-` magic (`pdf.is_pdf`). `fetch_page` streams the body and caps it at `MAX_PDF_BYTES` (15 MB); for a PDF it calls `pdf.extract_pdf_text`, which writes the raw binary to a `tempfile` temp location (cleaned up via `Path.unlink(missing_ok=True)` in `finally`), parses it with `pypdf` (`PdfReader`, capped at `MAX_PDF_PAGES` = 50, title pulled from metadata), and normalizes whitespace. The resulting `CrawledPage` carries `source="pdf"`; PDFs skip the Playwright fallback. Everything downstream is source-agnostic (chunking/embedding/orchestration read `page.text`/`page.source` unchanged), so no other module changes. New unpinned dep `pypdf` (pure-python, no system deps).
- **Pause/resume/cancel:** `POST /tools/deepsearch/{uid}/{pause|resume|cancel}` (owner-gated) rewrite `control.json`; the worker's `should_stop` callback polls it between source fetches (paused = sleep-loop, cancelled = stop). State lives in a file, not the job row, so the running-in-a-subprocess worker can read it without a DB round-trip.
- **Vector store (`services/deepsearch/store.py`):** `VectorStore` wraps `chromadb.PersistentClient(path=config.DEEPSEARCH_CHROMA_DIR)`, one collection per session (`ds_<uid>`). `Chunk` is the dataclass. `hybrid_search` blends cosine vector similarity with a BM25 keyword score (weights `HYBRID_VECTOR_WEIGHT`/`HYBRID_KEYWORD_WEIGHT`) over the candidate set, with optional metadata `where` filters. `embeddings.py` `embed_texts` calls the gateway embeddings endpoint and **falls back to a deterministic local hashing vector** on any failure (so the tool degrades, never breaks).
- **RAG chat (`services/deepsearch/chat.py` + `WS /tools/deepsearch/{uid}/chat`):** a dedicated lightweight loop (NOT the Devii hub), served **only by the service-lock owner** (closes `4013` for fast retry). Answers are grounded ONLY in the session collection via `hybrid_search`, cited inline, rendered client-side via `dp-content`. Turns persist to `deepsearch_messages` and audit `deepsearch.chat`. Frontend component `<dp-deepsearch-chat>` (`static/js/components/AppDeepsearchChat.js`) clones `AppDocsChat`'s framing but uses its own WebSocket to the chat path.
- **Status/report/export routes:** `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (`respond(..., DeepsearchSessionOut)`, HTML or JSON), `GET /tools/deepsearch/{uid}/export.{md,json,pdf}` (`services/deepsearch/export.py`; PDF via weasyprint). All are **capability URLs** scoped by the unguessable uuid7. **Viewer-flag discipline:** the session schema/context use `viewer_is_admin`/`viewer_owns` (never `is_admin`/`owns`) so a `respond()` context key never shadows a Jinja global (the issues/`{number}` issue class). `tests/api/tools/deepsearch/session.py` guards the HTML render.
- **Tables (`deepsearch_sessions`, `deepsearch_messages` soft-deletable + in `SOFT_DELETE_TABLES`; `deepsearch_url_cache` GC-only):** columns are ensured in `init_db()` (every queried column) with indexes. Every insert writes `deleted_at:None/deleted_by:None`; every read filters `deleted_at IS NULL`.
- **Frontend** (do not hand-roll): `DeepsearchTool.js` (`app.deepsearchTool`) drives the form via `Http.send`, watches `DeepsearchProgressSocket` (cloned from `SeoProgressSocket`, 4013 retry), and wires pause/resume/cancel. `static/css/deepsearch.css` uses the design tokens and is mobile-responsive.
- Devii tools `deepsearch`/`deepsearch_status`/`deepsearch_session` (public); docs `tools-deepsearch`; CLI `devplace deepsearch prune|clear`; audit `deepsearch.run.request|complete|failed` + `deepsearch.chat` (category `tools`). **New dependencies:** `chromadb`, `weasyprint`, `pypdf` (all unpinned). New runtime dirs `config.DEEPSEARCH_DIR`/`DEEPSEARCH_CHROMA_DIR` are registered in `DATA_PATHS`. **Add a dedicated nginx WS `location` for `/tools/deepsearch/{uid}/ws` and `/chat`** above `location /` for production, like the SEO and Devii sockets.

## Backup service (`services/backup/`, `services/jobs/backup_worker.py`, `routers/admin/backups.py`)

Admin-only, enterprise-grade backups built on the **same async-job pattern as zip/fork**, so nothing runs on the request path. `BackupService(JobService)` (kind `backup`) is registered in `main.py` and overrides `run_once` to call `super().run_once()` (reap/refill/sweep) and then `_fire_due_schedules()`.

- **Targets** (`store.BACKUP_TARGETS`): `database` (consistent SQLite snapshot of the main DB + `devii_tasks.db` + `devii_lessons.db` via the `sqlite3` online backup API, so it is consistent under WAL - never a raw file copy), `uploads` (`UPLOADS_DIR`), `keys` (`KEYS_DIR`), `full` (database snapshot + uploads + keys; regenerable/volatile dirs - staging, locks, zips, container workspaces, chroma - are intentionally excluded). `service._materialize(target, staging)` returns a list of `{root, path}` sources; DB targets are snapshotted into `staging/db_snapshot` first.
- **Worker** (`services/jobs/backup_worker.py`, stdlib only): reads a JSON spec `{sources:[{root,path}]}` and an output path, builds a deterministic `tar.gz` (uid/gid 0, symlinks skipped) rooted at each `root/`, and returns `{bytes_in, bytes_out, file_count, dir_count, sha256}`. Invoked via `asyncio.create_subprocess_exec` like `zip_worker`.
- **Storage:** archives go under `config.BACKUPS_DIR` (`data/backups/`, in `DATA_PATHS`) sharded with `attachments._directory_for` on the **random uuid tail** (same load-bearing reason as zips/blobs), named `{target}-{YYYYMMDD-HHMMSS}-{tail}.tar.gz`. Staging is `config.BACKUP_STAGING_DIR` (`data/backup_staging/`), removed in `process` `finally`.
- **Data model** (`store.py`, ensured in `database.init_db` via `backup_store.ensure_tables()`): `backups` (NOT soft-deletable - an archive is a reclaimable operational artifact, hard-deleted like zips) and `backup_schedules` (in `SOFT_DELETE_TABLES`, born-live `deleted_at:None`). `store` holds all CRUD plus `compute_storage_stats()` (du of every major data area + `shutil.disk_usage`, run in `asyncio.to_thread` from the route, 30s in-process TTL cache so the walk never blocks).
- **Permanent artifact:** `cleanup(job)` only removes leftover staging, NEVER the archive. Job retention prunes the `jobs` row; the archive and `backups` row persist until an admin deletes it, a schedule rotates it out (`keep_last`), or `devplace backups clear`. Deleting a backup is a HARD delete (unlink file + delete row) - correct because backups are GC artifacts, the documented exception to the soft-delete rule.
- **Schedules:** `backup_schedules` carry `kind` (`interval`|`cron`), `every_seconds`/`cron`, `enabled`, `keep_last`, `next_run_at`, run bookkeeping. `_fire_due_schedules` (lock-owner only, so each fires once) compares `next_run_at <= to_iso(now_utc())` and enqueues a `backup` job + a `backups` record, then advances `next_run_at` via `schedule.next_run`. **Timestamp format is load-bearing:** schedule `next_run_at` uses the devii `schedule.to_iso` format (`%Y-%m-%dT%H:%M:%S`, no tz/micros) on BOTH sides of the comparison so lexicographic compare equals chronological - do not mix it with `datetime.isoformat()`.
- **Routes** (`routers/admin/backups.py`, all `require_admin`, mounted via `admin/__init__`): `GET /admin/backups` (dashboard, `respond(..., model=BackupDashboardOut)`), `GET /admin/backups/data` (JSON dashboard for the monitor + Devii), `POST /admin/backups/run`, `GET /admin/backups/jobs/{uid}` (`BackupJobOut` for `JobPoller`), `POST /admin/backups/{uid}/delete`, `GET /admin/backups/{uid}/download` (`FileResponse`, path-guarded against `BACKUPS_DIR`), `GET /admin/backups/{uid}` (`BackupOut`), and schedule CRUD `schedules/create|{uid}/edit|toggle|run|delete`. **Route order:** every literal sub-path (`data`, `run`, `jobs/{uid}`, `schedules/...`) is declared BEFORE the `{uid}` catch-alls.
- **Frontend:** `static/js/BackupMonitor.js` (`window.BackupMonitor`, started inline from `admin_backups.html` like `AiUsageMonitor`) polls `/admin/backups/data` every 8s and renders storage/backup/schedule sections; run uses `Http.send` + `JobPoller`; delete/toggle/run/schedule actions use delegated clicks + `Http.send`; the schedule modal (create/edit) reuses the shared `.modal-overlay`/`data-modal` pattern. CSS `static/css/backups.css`. Sidebar link in `admin_base.html` (`admin_section == 'backups'`).
- **Devii** (all `requires_admin=True`, `handler="http"`): `backups_overview`, `backup_run`, `backup_status`, `backup_delete` (+confirm, in `CONFIRM_REQUIRED`), `backup_schedule_create`, `backup_schedule_delete` (+confirm). **CLI:** `devplace backups list|run <target>|prune|clear`. **Audit:** `job.backup.complete|failed` (category `backup`), `admin.backup.run|delete`, `admin.backup_schedule.create|update|toggle|delete` (category `admin`). **Docs:** admin API group endpoints + admin prose page `backups`.
- **No restore into a live server** (by design - overwriting the DB/uploads while running risks corruption). Restore is a manual ops procedure documented on the `backups` docs page.

## Container manager (`services/containers/`)

Admin-only. Supervises container instances, all running one shared prebuilt image. Reference: admin docs `Services -> Container Manager` and the `containers` API group.

- **Backend seam.** `backend/base.py` `Backend` ABC; `DockerCliBackend` drives `docker` via `asyncio.create_subprocess_exec` (streaming logs through `_stream`), `FakeBackend` is the in-memory test double. `runtime.get_backend()`/`set_backend(b)` selects it - tests call `set_backend(FakeBackend())`. A future Kubernetes/remote backend implements the same ABC.
- **Security (load-bearing).** This needs the docker socket = host root. Everything is gated `require_admin`/`requires_admin=True`; `--privileged` is never passed; user input is never shell-interpolated (arg-list subprocesses only); resource limits via `--cpus`/`--memory`.
- **One shared image, no in-app builds (load-bearing).** Every instance runs `config.CONTAINER_IMAGE` (default `ppy:latest`, override `DEVPLACE_CONTAINER_IMAGE`), built ONCE from `ppy.Dockerfile` via `make ppy` (context `devplacepy/services/containers/files`). There are no per-project Dockerfiles, builds, `ContainerBuildService`, or build UI - all removed. `service._launch` calls `api.run_spec_for(inst, config.CONTAINER_IMAGE)`; `api.create_instance(project, *, name, ...)` fails fast with `ContainerError` if `backend.image_exists(CONTAINER_IMAGE)` is false ("run 'make ppy'"). `backend.image_exists(ref)` (`docker image inspect`) was added to the `Backend` ABC (`FakeBackend` returns True). Migrating off the old model: `devplace containers prune-builds` removes the legacy per-project images (`backend.remove_image`) and clears the `dockerfiles`/`dockerfile_versions`/`builds` tables; existing instances auto-repoint to `ppy` (their stored `build_uid` is ignored). **Default idle:** `run_spec_for` runs `["sleep", "infinity"]` when an instance has no `boot_command`, so a bare instance stays up instead of exiting (the image `CMD` is the same, but the explicit command makes it independent of the image).
- **Data model** (`store.py`, indexed in `init_db`): `instances`, `instance_events` (audit), `instance_metrics` (ring buffer, sweep keeps `METRICS_RING`), `instance_schedules`.
- **Reconciler** (`service.py`, `BaseService`): the model is desired-vs-actual, NOT a task per container. Each tick: `backend.ps(label=devplace.instance)` -> converge each instance to `desired_state`, apply restart policy, reap orphan containers (labeled but no DB row), fire due `instance_schedules` (reuse `devii/tasks/schedule.py` `cron_next`/`next_run`), sample `docker stats`. Only the lock owner reconciles; HTTP routes only flip `desired_state`/write events. `docker run --name <slug>` collision is the double-launch guard.
- **Workspace** = `/app`: `project_files.export_to_dir` materializes the project to `config.CONTAINER_WORKSPACES_DIR/<project>` once, bind-mounted RW; `project_files.import_from_dir` (the inverse) syncs it back. Both run via `asyncio.to_thread`. **All runtime data lives OUTSIDE the `devplacepy/` package and NOT under `/static`**: workspaces + zips in `config.DATA_DIR` (`data/`, configurable via `DEVPLACE_DATA_DIR`). Never put generated data under `STATIC_DIR` - it is served publicly AND watched by dev `--reload` (scoped to `--reload-dir devplacepy`). The docker daemon must be able to bind-mount `DATA_DIR` for `/app`. The mount point is the single constant `backend/base.py` `WORKSPACE_MOUNT` (`/app`); the `ppy` image sets `WORKDIR /app` and **every exec passes `-w /app` explicitly** (`DockerCliBackend.exec` and the PTY exec in `routers/projects/containers/instances.py`), so one-shot `container_exec`, the interactive shell, and tmux sessions all start in the project workspace. The agent is told this (system prompt + `container_exec` tool summary) so it never prefixes a command with `cd /app`.
- **Pravda image (load-bearing, workspace ownership).** The hotpatch that used to run per build is now baked into `ppy.Dockerfile` once. The final stage `COPY`s the **sudo superclone** (`files/sudo`) over `/usr/local/bin/sudo` (+ symlink `/usr/bin/sudo`; the real `sudo` package is not installed) and **`pagent`** (`files/pagent`, the stdlib AI agent; reads `PRAVDA_OPENAI_URL`+`PRAVDA_API_KEY`, falling back to its public endpoint + `DEEPSEEK_API_KEY`) to `/usr/bin/pagent.py` (and `files/.vimrc` to `/home/pravda/.vimrc`, whose `AiEditSelection` helper hits the same gateway as pagent via `PRAVDA_OPENAI_URL`/`PRAVDA_API_KEY` with a public fallback, not `api.openai.com`), evicts any pre-existing uid-1000 user, creates user **`pravda` at `1000:1000`**, hands pravda **ownership of the toolchain and the OS package trees** (`chown -R pravda` over `/usr/local/lib`, `/usr/local/bin`, `/usr/lib/python3`, `/opt`, `/app`, `/home/pravda`, plus `/usr/lib`, `/usr/bin`, `/usr/sbin`, `/usr/share`, `/usr/include`, `/etc`, `/var/lib`, `/var/cache`, `/var/log`, `/srv`; `~/.local/bin` on `PATH`), and ends on `USER pravda`. **Why uid 1000:** `/app` is bind-mounted from the host, and under DooD a container's UID maps 1:1 to the host. A container running as **root** wrote root-owned files into the workspace; the app process (host `retoor`, uid 1000) then hit `[Errno 13] Permission denied` in `export_to_dir` on the next instance create - a permanent per-project brick. Pinning the container UID to `1000` makes every write land as `retoor`, and pravda owning the global site-packages means runtime `pip install` just succeeds as pravda with no elevation. **The sudo superclone** (`files/sudo`, POSIX `sh`) is sudo-CLI-compatible but **never swaps user**: it parses the full flag interface (`-u/-g/-E/-H/-n/-S/-i/-s/--`, leading `VAR=value` env assignments, `-V/-l/-v/-k/-K` short-circuits) and `exec`s the command **as the current user (pravda)**, exporting `SUDO_USER`/`SUDO_UID`/`SUDO_GID`/`SUDO_COMMAND`. So even a blind `sudo apt ...` / `sudo -u root ...` runs as uid 1000 and **cannot create a root-owned file** - the residual is gone by construction. **Rootless apt (`files/aptroot`):** pravda installs system packages directly (`apt install <pkg>`, no sudo). `aptroot` is symlinked over `apt`/`apt-get`/`dpkg` in `/usr/local/bin` (ahead of `/usr/bin`) and execs the real tool under **`fakeroot`** with `APT::Sandbox::User=root`; dpkg's chown-to-root is virtualized while files actually land owned by pravda (uid 1000), so installs succeed with no real euid 0 and still cannot brick the bind mount. Pravda owning the system trees makes the writes themselves succeed. Run `apt update` first (the image clears `/var/lib/apt/lists`). **Trade-off (intentional):** the only genuinely-root op left that does NOT escalate is binding a port < 1024 - use a high port + `/p/<slug>` ingress. Packages whose maintainer scripts need real privileged syscalls still belong in `ppy.Dockerfile` (its `RUN apt-get` runs as root before `USER pravda`) then `make ppy`. Enforcement lives in the Dockerfile (no `--user` on `docker run`). The `export_to_dir` unlink-before-write fix remains as belt-and-suspenders (the app owns the workspace dir, so it may delete any stale file in it regardless of owner before rewriting it).
- **`PRAVDA_*` runtime env injection.** `api.run_spec_for` merges `api.pravda_env(instance)` over the instance's own `env_json` (PRAVDA keys win) so every running container gets six platform vars: `PRAVDA_BASE_URL` (the `site_url` setting via `seo.public_base_url()`), `PRAVDA_OPENAI_URL` (`{base}/openai/v1`, the OpenAI-compatible gateway base), `PRAVDA_API_KEY` (the **creating** user's `api_key`, resolved live), `PRAVDA_USER_UID` (the project **owner**'s uid), `PRAVDA_CONTAINER_NAME` (instance name), `PRAVDA_CONTAINER_UID` (instance uid), `PRAVDA_INGRESS_URL` (the instance's absolute public ingress URL `{base}/p/{ingress_slug}` when published and `site_url` is set, relative `/p/{slug}` if not, empty if the instance has no `ingress_slug`). They let code inside a container call back into the platform and the AI gateway authenticated as the user. **Injected at run (docker run `-e`), not baked into the image:** the image is shared by every instance, but name/uid/api-key are per-instance and base-url/key must stay fresh, so they are resolved each launch and never stored as secrets in the DB. Resolution needs two new instance columns set at `create_instance`: `created_by` (= `actor[1]` when the actor is a user) feeds `PRAVDA_API_KEY`, and `owner_uid` (= `project["user_uid"]`) feeds `PRAVDA_USER_UID`; instances created before this change have neither and degrade to an empty key/uid (base-url and container name/uid still resolve) until recreated. `PRAVDA_BASE_URL`/`PRAVDA_OPENAI_URL` are empty when `site_url` is unset, so set the admin `site_url` for container-to-platform calls to work (same requirement as ingress URLs).
- **Shared ops** in `api.py` back BOTH `routers/projects/containers/instances.py` and the Devii `ContainerController` (`services/devii/container/`, `handler="container"`, 7 instance tools). The reconciler service defaults disabled (needs docker).
- **UI is two surfaces over one API set, both discoverable** (the old `templates/containers.html` page had zero links to it - that issue is fixed). (1) Per-project manager: `templates/containers.html` + `static/js/ContainerManager.js` handles instance creation through an app **modal form** (the `_macros.html` `modal()` macro + `ModalManager` `.visible` toggle); its instance list links out to the shared detail page. Reached from the project detail page's admin-only **Containers** button (`is_admin(user)` gate) and now passes breadcrumbs so content clears the fixed nav. (2) Admin **Containers** section: `routers/admin/containers.py` (mounted `/admin/containers`, sidebar link in `admin_base.html`, `admin_section="containers"`) - `GET /admin/containers` lists every instance via `store.all_instances()` (decorated with project title/slug from one `projects` lookup) in an `.admin-table`; `GET /admin/containers/data` is the poll JSON; `GET /admin/containers/{uid}` renders `templates/containers_instance.html` + `static/js/ContainerInstance.js`, a dedicated detail page (lifecycle, poll logs/metrics, schedules add/delete, ingress, sync, interactive exec over a PTY WebSocket gated on the lock owner). **Key reuse rule:** `containers_admin.py` adds NO lifecycle/logs/exec endpoints - the instance carries its `project_uid`, so the detail route resolves the project and `ContainerInstance.js` targets the existing `/projects/{slug}/containers/instances/{uid}/...` routes. Add new instance operations to `routers/projects/containers/instances.py` only; the admin detail page picks them up for free. All container CSS (`static/css/containers.css`) uses app design tokens (`--bg-card`, `--text-primary`, `--success`/`--danger`/`--warning`, `--radius`) and the shared `.card` recipe.
- **Interactive shell = floating xterm.js window.** An instance's shell is NOT inline; it is a floating `<container-terminal>` (`static/js/components/ContainerTerminal.js`) over the existing exec WS (`/projects/{slug}/containers/instances/{uid}/exec/ws`, `pty.openpty()` + `docker exec -it`, admin + `service_manager.owns_lock()` gated). xterm + fit addon are vendored under `static/vendor/xterm/` and lazy-loaded. It extends the generic **`FloatingWindow`** base (`static/js/components/FloatingWindow.js` + `static/css/floating-window.css`) which owns the Devii-style chrome (drag, native `resize:both`, maximize/fullscreen, geometry persistence). **`app.windows`** (`WindowManager`, `components/WindowManager.js`) assigns z-index on `pointerdown` so the last-clicked window is on top. **The Devii terminal (`devii/devii-terminal.js`) now also `extends FloatingWindow`** and reuses the base drag/geometry/preset/resize/persist plus `_window`/`_registerWindow` (z-order + context-menu) machinery; it overrides only the parts that genuinely differ - `_setState` (Devii adds a `closed` state + launcher FAB), `_defaultGeometry` (bottom-right 760x600), `_changeFont` (per-user `--devii-font-size` CSS var), `_contextItems`, and `_persistGeometry` (delegates to its richer `{state,geometry,focused}` blob under `devii-terminal-state`). Devii keeps native `resize:both` (no `.fw-resize` grip) and **`devii.css` is unchanged**: the only base change required was a `get _dragClass()` getter (default `fw-dragging`) that Devii overrides to `devii-dragging`, so the base drag handlers toggle Devii's own class and its existing CSS drives everything. Open it two ways: the instance-page **Open terminal** button (`app.containerTerminals.open(slug, uid, name)`, `ContainerTerminalManager`) or Devii "attach &lt;container&gt;" -> the admin-only `open_terminal` **client** action (`client_actions.py`, `requires_admin=True`) -> `DeviiClient._openTerminal`. **Resize protocol:** the exec WS treats a brace-leading JSON frame `{"type":"resize","cols","rows"}` as a control message - it `TIOCSWINSZ`-es the host pty (`fcntl.ioctl`) **and** `proc.send_signal(SIGWINCH)` to the `docker exec -it` client so the resize forwards into the container tty (the SIGWINCH is required: with `start_new_session=True` the host slave is not the client's controlling terminal, so the kernel does not auto-deliver it; the pair shares its winsize, so the client reads the new size off its slave stdio and calls Docker's exec-resize API). Everything else is raw stdin, so keystrokes are untouched. Client-side the **fit addon** drives it and the window has a dedicated `.fw-resize` grip (xterm covers the native CSS resize corner, so the base `FloatingWindow` adds a manual bottom-right grip instead of relying on `resize:both`). **Persistence (tmux):** with `?session=<name>` (validated `^[A-Za-z0-9_-]{1,64}$`) the WS runs `tmux new-session -A -s <name>` (falls back to bash if tmux missing), so the shell/server survives WS death - `proc.kill` on disconnect kills the tmux *client*, the server daemon + session live on (proven: a detached `devplace` session keeps its process running across exec clients). The frontend keeps exactly ONE persistent terminal (the last opened, `ContainerTerminalManager`): it attaches to session `devplace`, **auto-reconnects** with capped backoff on unexpected WS close (not on code 1008), and is remembered per user in `localStorage` (`ct-persistent:<scope>`); `app.containerTerminals.restore()` (one call in `Application.js` after `window.app`) reopens it on the next page load. Opening another terminal calls `setPersistent(false)` on the previous one (its tmux session lingers in the container, but the frontend stops auto-reconnecting/remembering it); explicit window-close (`_onClose`) detaches + forgets (session still survives, reopen re-attaches). **Context menu:** every window wires `app.contextMenu.attach(this.win, () => this._contextItems())` (right-click + long-press). The base `FloatingWindow._contextItems` is Minimize/Normalize/Close; `ContainerTerminal` overrides it with Copy / Paste / Sync files / Restart / Terminate (`dp-dialog` confirm, then `delete` + close) / Minimize / Normalize / Project / Files / Close - the lifecycle items POST to the existing `instances/{uid}/{sync,restart,delete}` routes and Project/Files `window.location.assign` to `/projects/{slug}` and `/projects/{slug}/files`. Devii's `_contextItems` override returns Copy / Paste / Minimize/Normalize/Close (no container/project actions, since it is bound to neither). **Copy/Paste:** both use the async Clipboard API with a hidden-`textarea` + `execCommand("copy")` write fallback. Container Copy reads `term.getSelection()` (the item is `disabled` when `!term.hasSelection()`) and Paste sends `navigator.clipboard.readText()` straight down the exec WS as stdin. Devii Copy takes the live `window.getSelection()` or, when empty, the current input line; Paste inserts clipboard text into the `.devii-input` at the caret (`selectionStart/End` splice) and refocuses. **Minimize/Normalize** are `_presetGeometry(w,h)` size presets (smallest-usable / comfortable), available as both titlebar buttons (`data-win="minimize|normalize"`) and menu items. xterm renders ANSI natively (no `cleanTerm` stripping for the interactive shell; the one-shot exec box still strips, since `docker exec` without `-t` is non-TTY).
- **No wildcard verb route (load-bearing).** `routers/projects/containers/instances.py` registers every instance verb as a **literal** route - `start`/`stop`/`pause`/`resume`/`restart` (stacked `@router.post` decorators on one `instance_action` handler that reads the verb from `request.url.path`), plus `delete`/`exec`/`sync`/`schedules`. There is deliberately NO `POST /instances/{uid}/{action}` catch-all: a wildcard segment matches its literal siblings too (Starlette matches first-declared), so a catch-all silently swallowed `exec`/`sync`/`delete`/`schedules` as `{"error": "unknown action: exec"}` whenever it sat above them. Add any new instance verb as a literal route (and to the `instance_action` decorator stack if it just flips desired state) - never reintroduce a `{action}` wildcard.
- **Host ports are auto-allocated and globally unique.** `parse_ports` accepts a bare container port (`8899`, host side `0` = auto) or a pinned `host:container`. `api.assign_host_ports` (called in `create_instance`) resolves every `host=0` to the lowest free port in `[HOST_PORT_MIN=20001, 65535]` that is neither in `used_host_ports()` (the union of every instance's published host ports from `ports_json`) nor currently bound on the host (`_host_port_free` test-binds `0.0.0.0:port`). A pinned host port already published by another instance is rejected up front. This guarantees no two instances ever collide on a host port, which was the silent `docker run` "port is already allocated" failure. The resolved host port is what gets stored in `ports_json` and what the ingress proxy reads.
- **Ingress (`/p/<slug>`)** = `routers/proxy.py` (registered at `/p`). An instance opts in with `ingress_slug` + `ingress_port` (must be one of its mapped container ports; slug unique, validated in `api.validate_ingress`). `_resolve(slug)` -> `store.find_instance_by_ingress` -> `api.proxy_target(instance)` returning `(gateway, host_port)` (the instance's recorded docker bridge gateway + published host port); the route reverse-proxies HTTP (httpx) and WebSocket (`websockets` client) to `http://{host}:{port}/{path}`, stripping the `/p/<slug>` prefix. The reconciler records `container_ip`/`container_gateway` from `docker inspect` each running tick. Gateway+published-port is used instead of loopback (broken where docker's `nat OUTPUT` excludes `127.0.0.0/8` from DNAT) and instead of the container's own bridge IP (droppable by docker's `DOCKER ! -i docker0 -o docker0 -j DROP` isolation rule); `DEVPLACE_CONTAINER_PROXY_HOST` overrides the host (still with the published port). **Public** (no auth) but SSRF-safe (host/port come from the instance row, never user input). nginx needs a `/p/` location with WS upgrade + long timeouts (added to `nginx/nginx.conf.template`).
- **Production wiring** (the manager drives the host docker daemon): opt-in overlay `docker-compose.containers.yml` (socket mount, `INSTALL_DOCKER_CLI=true` build arg, `group_add` docker gid). `make docker-build`/`make docker-up` always apply this overlay and self-derive its inputs (`DOCKER_GID` from `stat -c '%g' /var/run/docker.sock`, `DEVPLACE_DATA_DIR=$(CURDIR)/data`), so it works with no `sudo`/`/srv`/`.env` edits; a plain `docker compose up -d` drops the overlay and re-breaks it. **DooD bind-mount gotcha**: `docker run -v <path>:/app` resolves `<path>` on the HOST, so `DEVPLACE_DATA_DIR` must sit at an identical host+container path (make uses `$(CURDIR)/data`; manual compose defaults to `/srv/devplace-data`); build contexts ship via the docker API tarball so the container temp dir is fine. Ingress reaches host ports via each instance's recorded docker bridge gateway (`api.proxy_target` -> `gateway:host_port`); set `DEVPLACE_CONTAINER_PROXY_HOST=host.docker.internal` only when a containerized app cannot route to that gateway. See README "Container Manager wiring".
- **Admin CRUD manager.** `/admin/containers` is a full manager, not a read-only list. `routers/admin/containers.py` adds (all `require_admin`, all reuse `api.*` directly - no docker/exec backend duplicated): `POST /create` (`ContainerAdminCreateForm` -> `api.create_instance`, project search-select + run-as user + boot fields + restart policy + start_on_boot + env/ports/limits/ingress), `GET/POST /{uid}/edit` (`ContainerEditForm` -> `api.update_instance_config`, edits run_as_uid/boot_language/boot_script/boot_command/restart_policy/start_on_boot/cpu_limit/mem_limit), stacked literal lifecycle routes `POST /{uid}/{start,stop,restart,pause,resume}` (the no-wildcard rule), `POST /{uid}/sync` (bidirectional), `POST /{uid}/delete` (soft, owner-or-admin via admin guard), and the two JSON search endpoints `GET /{projects,users}/search` (back the create/edit search-selects via `db.query LIKE`). **Route order:** `/{uid}/edit` and the literal verb routes are declared BEFORE the `/{uid}` catch-all. Frontend: `static/js/ContainerList.js` (poll-render rows with inline action buttons + the create modal with two debounced search-select widgets and a boot-language toggle) and `static/js/ContainerEdit.js` (the edit page). Both route through `Http.send`/`Http.getJson` and surface errors via `app.toast`; the Terminal button reuses `app.containerTerminals.open(slug, uid, name)`. The instance detail page (`containers_instance.html`) gained a config summary (boot language, run-as uid, start-on-boot) + a **Status history** card rendering the server-side `instance_events`. Devii: `container_create_instance` gained the boot/run-as/start_on_boot args, a new `container_configure_instance` action (`controller._configure_instance` -> `api.update_instance_config`) edits the same fields, both admin-only, audited via `_DEVII_CONTAINER_EVENTS`.
- **Bidirectional newer-wins sync.** `project_files.sync_dir_bidirectional(project_uid, workspace, user)` reconciles the project FS against the instance workspace per file by timestamp (project `updated_at` epoch via `datetime.fromisoformat(...).timestamp()` vs filesystem `st_mtime`, 1s skew favouring export on ties); only-one-side files propagate; it NEVER deletes, only creates/overwrites the older side. A read-only project exports only (the read-only guard direction). `api.sync_workspace` (HTTP/Devii sync action, returns `{exported, imported}`) and `api.sync_bidirectional_sync` (reconciler, system actor, `record_event` only when non-zero) both call it. The reconciler runs it before every `_launch` and on a ~60s wall-clock cadence (`SYNC_EVERY_SECONDS`, gated on `time.monotonic()` independent of the 5s tick). The boot-script files are in `SYNC_SKIP_NAMES`.
- **Run-as user, boot source, start_on_boot columns.** `run_as_uid` (identity + `api_key` only; the container OS user stays `pravda` uid 1000 for the bind mount - `api.pravda_env` resolves it ahead of `created_by`/`owner_uid`), `boot_language`/`boot_script` (precedence `boot_script` by language > `boot_command` > image CMD in `api.run_spec_for`; `api.materialize_boot_script` writes `.devplace_boot.{py,sh}` into the workspace before launch; `api.validate_boot` enforces the language set + 100k cap), and `start_on_boot` (per-container; `service._boot_pass` forces `desired_state=running` for flagged instances once on first `run_once`). All four are ensured in `init_db`'s `instances` block and defaulted in `store.create_instance`.
- **Status-change logging choke point.** `service._set_status(inst, changes, reason=)` diffs old vs new `status`, and on change writes an `instance_events` `status_change` row plus an audit `container.instance.status` event (`record_system`, `old_value`/`new_value`). Every reconciler status mutation (`_reconcile`, `_launch`, `_handle_exit` terminal + policy_restart) flows through it, so previously-silent transitions are now logged and visible on the detail page's Status history.
- **Testing without Docker:** `FakeBackend` (its `image_exists` returns True) + `runtime.set_backend`, monkeypatch `config.CONTAINER_WORKSPACES_DIR` to tmp. See `tests/unit/services/containers.py` and `tests/api/containers.py` (argv, instance creation on `config.CONTAINER_IMAGE`, image-not-built guard, reconcile matrices, schedule firing, ingress validation + live HTTP proxy, HTTP admin gate).

## Audit log (`services/audit/`)

Append-only, admin-only record of every state-changing action. The full event catalogue (199 keys) is `events.md`; the catalogue's storage model, relation vocabulary, and per-event specs are authoritative.

- **Package layout.** `store.py` (the `audit_log` + `audit_log_links` tables, `ensure_tables`, `insert_event`/`insert_links`/`get_event`/`get_links`/`sweep`), `categories.py` (`category_for(event_key)` longest-prefix map), `record.py` (the recorder + link builders), `query.py` (`list_events`/`filter_options`/`get_event_with_links` for the admin UI), `service.py` (`AuditService` retention sweep). `ensure_tables()` + the 11 indexes are wired into `database.py` `init_db()` (local import to avoid the cycle: audit modules import `database`/`utils`). `AuditService` is registered in `main.py`.
- **Two recorder entrypoints (the reuse keystone).** `record(request, event_key, *, user=_UNSET, actor_kind=None, target_type/uid/label, old_value, new_value, summary, metadata, result="success", origin, via_agent, links, category)` is for HTTP/WebSocket handlers - `request` may be a Starlette `Request` OR `WebSocket` (both expose `.headers`/`.url`/`.client`). It auto-derives the actor from `get_current_user(request)` (or an explicit `user=`), the request fields, and the **`X-Devii-Agent`** header (-> `origin=devii`, `via_agent=1`), and auto-appends the `actor` link. `record_system(event_key, *, actor_kind, actor_uid, actor_username, actor_role, origin, via_agent, ...)` is for request-less contexts (rewards, notifications, the AI gateway ledger, the news/container services, Devii turns/tasks, job services, the CLI). Both are wrapped in try/except and **never raise into the caller**; `summary` is HTML-stripped and capped at 140 chars; `metadata` is JSON-serialised. Link builders (`audit.target`, `audit.parent`, `audit.author`, `audit.recipient`, `audit.project`, `audit.instance`, `audit.setting`, `audit.job`, `audit.poll`, `audit.option`, `audit.task`, ...) keep call sites terse.
- **Emission convention.** Explicit `audit.record(...)` / `record_system(...)` calls placed after the mutation succeeds (or on the guard branch with `result="denied"`/`"failure"`). DRY choke points: posts/projects/gists create/edit/delete record inside `content.py` (`create_content_item`/`edit_content_item`/`delete_content_item`, the latter two derive the key from `target_type`); project file ops record via the `_audit_file`/`_edit`/`_fail` helpers in `project_files.py` (read-only guard -> `result="denied"`); services via `_audit_service`; containers via `audit_instance` (in `routers/projects/containers/_shared.py`). Container ops are recorded in `routers/projects/containers/instances.py` for the HTTP path and in the Devii **dispatcher** (`actions/dispatcher.py` `_audit_mechanic`, after a successful `_run`) for the agent path - the two paths are disjoint (Devii calls `api.py` directly, never the routes), so there is no double counting. The dispatcher hook also emits the `devii.*` self-config mechanics (behavior/tools/tasks/lessons/customization) from one place.
- **Failures and denials are events.** `auth.login.failure`, `auth.password.forgot_request` (unknown email), `security.rate_limit.block`, `security.maintenance.block`, `security.authz.denied` (emitted inside `require_user`/`require_admin`), self-role-change and self-disable denials, read-only write attempts, and `ai.quota.exceeded` all record with `result` set to `failure`/`denied`.
- **Admin UI.** `GET /admin/audit-log` (paginated list + filter bar, `admin_section="audit-log"`) and `GET /admin/audit-log/{uid}` (detail + links) in `routers/admin/ package`, both `require_admin`, both negotiate via `respond(..., model=AuditLogOut|AuditEventOut)`. Templates `admin_audit_log.html` / `admin_audit_event.html` reuse `admin.css` + a small `audit.css`; `_pagination.html` gained an optional `pagination_query` prefix (default empty, backward-compatible) so filters survive paging. Sidebar link sits last, before Settings.
- **Devii access.** The admin-only Devii tools `audit_log` (GET `/admin/audit-log`) and `audit_event` (GET `/admin/audit-log/{uid}`) (`services/devii/actions/catalog.py`, `handler="http"`, `requires_admin=True`) expose the same two routes to the agent - no new endpoint, since the `respond(...)` routes already serve JSON and Devii's client always sends `Accept: application/json` (the `admin_list_users` action follows the same pattern). `audit_log` takes every filter as a query param (`page`, `event_key`, `category`, `actor_role`, `actor_uid`, `origin`, `result`, `q`, `date_from`, `date_to`) and returns `AuditLogOut`, whose `options` object lists the valid values for each filter so the agent can discover them in one call; `audit_event` returns `AuditEventOut` (the row + related links). A system-prompt steer in `agent.py` (the AGGREGATES block) routes audit/history/"who did X" questions to these tools.
- **Deferred persistence.** `record`/`record_system` do all the cheap prep (actor resolve, sanitize, `json.dumps`, link assembly) on the request thread, then hand the two DB inserts to the **background task queue** (`services/background.py`, `background.submit(_persist, row, links)`) so the request returns without waiting on SQLite. The `uid`/`created_at` are generated eagerly at record time (so the return value and the audit timestamp reflect the action, not the flush). Under `DEVPLACE_DISABLE_SERVICES=1` (tests) the queue runs inline, so audit rows are visible immediately. See AGENTS.md -> "Background task queue".
- **Retention.** `AuditService` (`audit` service, daily) prunes rows older than `audit_log_retention_days` (default 90; `0` disables) via `store.sweep`. Configurable on the Services page like any other service.
- **Adding an event:** pick/extend a key in `events.md`, add the `category_for` prefix if it is a new domain, then call `audit.record`/`record_system` at the mutation point with the right links and `result`. Never gate the audited action on the recording.

## CDN Libraries

Loaded via `<script>` tags in `base.html`. ALL must use `defer` to avoid blocking DOMContentLoaded:

```html
<script defer src="https://cdn.jsdelivr.net/npm/marked/lib/marked.umd.js"></script>
<script defer src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.11.1/build/highlight.min.js"></script>
<script defer src="/static/vendor/purify.min.js"></script>
<script type="module" src="https://cdn.jsdelivr.net/npm/emoji-picker-element@^1/index.js"></script>
<script defer src="/static/js/ContentRenderer.js"></script>
<script defer src="/static/js/EmojiPicker.js"></script>
<script type="module" src="/static/js/Application.js"></script>
```

**Never use `<script>` without `defer` for CDN libraries** - they block HTML parsing and cause `wait_until="domcontentloaded"` to timeout in Playwright tests.

## Emoji Picker

Uses `emoji-picker-element` web component (Discord-style, searchable, skin tones):
- `EmojiPicker.js` wraps it with a toggle button and inserts unicode at cursor position
- Added to all `.comment-form textarea` and `.emoji-picker-target` elements
- The old `&#x1F600;` emoji button has been removed from all templates

## Modal System

`Application.js` `initModals()` toggles the `.visible` CSS class on the modal overlay. The CSS rule `.modal-overlay.visible { display: flex; }` handles visibility:

```javascript
// CORRECT - toggle the .visible class on the modal:
modal.classList.add("visible");    // show
modal.classList.remove("visible"); // hide
```

Always call `e.preventDefault()` on `[data-modal]` click handlers since trigger elements often have `href="#"`:

```javascript
trigger.addEventListener("click", (e) => {
    e.preventDefault();
    modal.style.display = "flex";
});
```

The `modal-close` class is handled by `Application.js` - no inline JS needed in templates for basic modals.

Do NOT hand-write the overlay/header markup. Use the shared macro in `templates/_macros.html`:

```jinja
{% from "_macros.html" import modal %}
{% call modal('create-post-modal', 'Create New Post') %}{# wide=true for modal-card-wide #}
    <form ...> ... <div class="modal-footer">...</div> </form>
{% endcall %}
```

## Shared template partials

Reuse these via `{% set _x = ... %}{% include %}` (the `_avatar_link.html` convention) instead of copy-pasting markup:

- `_post_votes.html` - post +/- vote bar. Locals: `_uid`, `_my_vote`, `_count`.
- `_star_vote.html` - project/gist star button. Locals: `_type` (`project`|`gist`), `_uid`, `_my_vote`, `_count`, `_btn_class`, optional `_stop` (adds `data-stop-propagation`). The star glyph (`☆`→`★` when `.voted`) comes from the `vote-star` CSS class via `::before` (`base.css`) - do not put a literal star in markup.
- `_post_header.html` - post author/avatar/time header (`.post-header`). Locals: `_author`, `_time`.
- `_topic_selector.html` - topic radio group. Locals: `_topics`, `_selected`.

Vote button styles live ONCE in `feed.css` (`.post-action-btn`) - never redefine them in `post.css`. Page-specific CSS goes in a `static/css/*.css` file referenced from `{% block extra_head %}`, never an inline `<style>` block.

## Custom web components (`static/js/components/`)

Self-contained, presentational UI is built as custom elements with the `dp-` prefix:
`dp-avatar`, `dp-code`, `dp-content`, `dp-toast`, `dp-dialog`, `dp-context-menu`, `dp-lightbox`.
Conventions:

- Each extends `Component` (`static/js/components/Component.js`), a thin `HTMLElement` base
  with `attr(name, fallback)`, `boolAttr(name)`, `intAttr(name, fallback)` helpers.
- **Light DOM only** (no `attachShadow`), so the site's global CSS applies - matching the
  existing `devii-*` elements. `dp-dialog`/`dp-context-menu` reuse the existing
  `.dialog-*`/`.context-menu*` styles in `components.css`; `dp-toast` styles are in
  `components.css` (`.dp-toast-host`/`.dp-toast`).
- Each file self-registers via `customElements.define` at the bottom. `components/index.js`
  imports them all and is imported by `Application.js`, so every element is defined on every
  page (including `/docs`).
- The behavioural singletons are created once in `Application.js` and exposed as `app.dialog`,
  `app.contextMenu`, `app.toast`. The `dp-dialog`/`dp-context-menu` elements keep the exact
  pre-existing `Dialog`/`ContextMenu` APIs (`confirm`/`prompt`/`alert`/`open`,
  `attach`/`open`/`close`) so existing callers (`ProjectFiles`, `ApiKeyManager`, `ModalManager`,
  `PushManager`, `ApiTester`) are unchanged. `Avatar`/`CodeBlock`/`Toast`/`ContentRenderer`
  remain as static/service modules (widely imported); the matching elements (`dp-avatar`,
  `dp-code`, `dp-content`) reuse them. `dp-content` wraps the shared `contentRenderer` engine
  that also powers the `data-render` attribute (driven by `ContentEnhancer`) - the engine stays
  a reusable service, the element is its component face. `dp-content` captures its raw markdown
  source before `contentRenderer.applyTo` overwrites `textContent`, then mounts a hover/focus
  reveal `.content-copy-btn` in its top-right corner that copies that original source to the
  clipboard (mirroring `CodeBlock.copyButton`); CSS lives beside `.code-copy-btn` in `docs.css`.
- **What is NOT a component (by design):** partial-bound controllers (`ReactionBar`,
  `VoteManager`, `BookmarkManager`, `PollManager`, `CommentManager`, etc.) enhance
  server-rendered `_*.html` markup rather than render standalone UI; pure utilities (`Http`,
  `DomUtils`, `FormManager`, `Poller`, `JobPoller`, `OptimisticAction`) have no UI. These stay
  plain ES6 modules. The four optimistic engagement controllers extend the shared
  `OptimisticAction` base - see "Shared frontend utilities" below.
- Documented in the public docs **Components** section (`templates/docs/component-*.html`,
  registered under `SECTION_COMPONENTS` in `routers/docs/ package`). Each page has a `data-render`
  markdown block (with example markup HTML-escaped as `&lt;dp-...&gt;`, since `data-render`
  re-parses `textContent` and would otherwise instantiate the example) plus a live demo in a
  separate block whose `<script type="module">` imports its own component module (it runs
  before `Application.js`).
- **`dp-lightbox` (`AppLightbox.js`, `app.lightbox`, `static/css/lightbox.css`)** is the single
  image lightbox. It is **attribute-wired, not call-wired**: the singleton installs ONE delegated
  `document` click listener and opens any clicked `img[data-lightbox]` (showing `data-full` when
  present, else the image's own src; `alt` as caption; ignored when the image sits inside an
  `<a>`). To make a thumbnail open it, just add `data-lightbox` - never attach a click handler.
  Attachment thumbnails carry `data-lightbox`+`data-full` (`_attachment_display.html`); markdown
  content images are marked centrally in `ContentEnhancer.initContentRenderer` (one place covers
  every `data-render` surface), so embedded images are always clickable. The old hand-rolled
  `.attachment-lightbox` markup/CSS was dead (never wired) and has been removed.

## Shared frontend utilities (DRY)

A small set of plain ES6 modules under `static/js/` own the cross-cutting patterns so feature
code stays tiny. Reach for these instead of hand-rolling a loop, a fetch, or a click handler.

- **`Http` (`static/js/Http.js`, global `window.Http`).** The single HTTP helper. `getJson(url)`
  (GET -> JSON, throws on non-2xx); `sendForm(url, params)` (POST form-encoded, follows the
  `/auth/login` redirect via `Http.toLogin()`, throws a bare status on failure, returns JSON);
  `send(url, params)` (POST form-encoded that throws `data.error.message` on `!ok` **or** a 200
  body with `ok:false` - the manager-style error the container/admin UIs surface in a toast);
  `postJson`/`postForm`/`toLogin`. Container files (`ContainerManager`, `ContainerList`,
  `ContainerInstance`, `ContainerTerminal`), `ServiceMonitor`, and `ProjectFiles` all route
  through it - none re-implement `fetch`.
- **`Poller` (`static/js/Poller.js`).** `new Poller(fn, intervalMs, { immediate = true, pauseHidden = false })`
  runs `fn` on an interval with `start()`/`stop()`/`tick()`; `tick()` swallows errors so one
  failed poll never kills the loop, and `pauseHidden` skips the tick while `document.hidden`.
  Used by every live-update loop: `CounterManager` (30s, `pauseHidden`), `ContainerManager` (3s),
  `ContainerList` (4s), `AiUsageMonitor`, `ServiceMonitor`, and `ContainerInstance`'s detail (4s)
  + logs (3s). Store the `Poller`, not a raw interval id.
- **`JobPoller` (`static/js/JobPoller.js`).** `JobPoller.run(statusUrl, { onDone, onFailed, onTimeout, intervalMs = 1500, maxAttempts = 200 })`
  returns a Promise; it polls `Http.getJson(statusUrl)`, swallows transient fetch errors, and
  fires the matching callback on `status === "done"|"failed"` or timeout. This is the one place
  the async-job status-poll lives - `ProjectForker` and `ZipDownloader` both call it with their
  own navigate/download/toast callbacks.
- **`OptimisticAction` (`static/js/OptimisticAction.js`).** Base with one method,
  `submit(url, params, errorTarget, render)`: `Http.sendForm` -> `render(result)` on success ->
  `console.error` + (when `errorTarget` is given) `Toast.flash(errorTarget, "Error", 1500)` on
  failure. `VoteManager`/`ReactionBar`/`BookmarkManager`/`PollManager` `extend` it and call
  `this.submit(...)` for their POST, **keeping their own event wiring** (so `ReactionBar`'s
  palette toggle and `PollManager`'s multi-action handlers and `VoteManager`'s per-button
  `stopPropagation` are untouched). Pass `errorTarget` only where the old code toasted
  (`VoteManager`); the others pass `null` to keep their console-only behaviour.
- **`FloatingWindow` / `WindowManager` (`static/js/components/`).** The draggable window base and
  shared z-order manager - documented in the Container manager section; both the container
  terminals and the Devii terminal extend `FloatingWindow`.
- **On-screen keyboard reflow (mobile).** `FloatingWindow` listens on `window.visualViewport`
  `resize`/`scroll` and, while fullscreen/maximized, sets the window inline `height`/`top` to the
  visual viewport (the area NOT covered by the phone keyboard) instead of letting the keyboard push
  the window off-screen; it also toggles the `fw-keyboard-visible` class whose CSS drops the fixed
  `bottom` so the inline size wins. The Devii terminal mirrors this in its own `_setState` /
  `_onVisualViewport2` (`devii-keyboard-visible`) and `_ensureInputVisible` keeps the focused
  `.devii-input` pinned just above the keyboard; `ContainerTerminal._onResize` re-fits xterm against
  the shrunken viewport and `scrollToBottom`s. `AppContextMenu.open` clamps into the visual-viewport
  rect (`offsetLeft/Top` + `width/height`) so Paste stays on-screen above the keyboard, and the
  menu no longer closes on a visualViewport-only height change (keyboard appearing) - only a true
  `window.innerWidth` change closes it. All paths fall back to `window.inner*` when `visualViewport`
  is absent, so desktop and old browsers are untouched.

## Profile media gallery and soft-deleted attachments

The profile **Media** tab (`/profile/{username}?tab=media`, public) is a paginated grid of every
attachment a user uploaded, newest first, across all `target_type`s. Attachments support a
**soft delete** so a user can remove one upload without affecting its parent object.

- **One `deleted_at` column** on `attachments` (ISO string, mirrors the `push.py` precedent),
  ensured idempotently in `init_db` via `create_column_by_example("deleted_at", "")` plus the
  `idx_attachments_user_created` index. `store_attachment` writes `deleted_at: None`.
- **Soft delete preserves the relation and the file.** `attachments.soft_delete_attachment(uid)`
  only stamps `deleted_at`; it never touches `target_type`/`target_uid` and never unlinks the
  file. `restore_attachment(uid)` clears `deleted_at`, so the item reappears on its parent object
  and in the gallery with zero extra bookkeeping.
- **Three read paths filter `deleted_at IS NULL`** so a soft-deleted item vanishes everywhere
  (the gallery AND its parent post/project/etc.): `get_attachments`, `get_attachments_batch`
  (both in `attachments.py`), and `database.get_user_media`. **The hard-delete cascades
  (`delete_attachments_for`, `delete_target_attachments`) stay UNFILTERED** so permanently
  deleting a parent object still removes ALL its attachment files, including soft-deleted ones -
  never add the filter there.
- **Queries:** `database.get_user_media(user_uid, page)` (linked, non-deleted, newest first; each
  item gets a `target_url` via `resolve_object_url`) and `database.get_deleted_media(page)` (the
  admin trash, joined to uploader username).
- **Authorization:** `POST /media/{uid}/delete` (`routers/media.py`) is owner-or-admin
  (`attachment["user_uid"] == user["uid"] or is_admin(user)`); `POST /media/{uid}/restore` and
  `POST /admin/media/{uid}/purge` (the only hard delete, via `delete_attachment`) are admin-only
  (`routers/admin/ package`, sidebar **Media** -> `/admin/media`). The tab itself is public.
- **Frontend:** `_media_gallery.html` reuses the `_attachment_display.html` type branches and the
  `dp-lightbox` contract (`data-lightbox`/`data-full`). The delete button carries
  `data-media-delete` + `data-confirm`; `ModalManager.initConfirmations` shows the confirm and
  `MediaGallery.js` (`app.mediaGallery`) does the optimistic `Http.send` delete, fades the tile,
  and toasts. A `<noscript>` form is the no-JS fallback. Grid styling is `static/css/media.css`.
- **Devii:** `list_media` (public) and `delete_media` (auth, in `CONFIRM_REQUIRED`) in the catalog.
- **Docs visibility (deliberate):** members and guests must never be told this is a *soft* delete.
  The public prose page `docs/media-gallery.html` (General) and the member-facing `media-delete`
  API endpoint (Profiles group) describe deletion as a plain "remove" - no soft-delete, restore,
  trash, or purge language. All moderation mechanics live on the admin-only `docs/media-moderation`
  prose page (`admin: True`) and in the admin API group (`media-restore`, `admin-media`,
  `admin-media-purge`, all `auth="admin"`), which `docs_search` excludes from member results and
  `routers/docs/ package` 404s for non-admins. Because `docs_search._strip` keeps the text *inside*
  `{% if %}` blocks, admin content must live on a separate `admin: True` page, never inline-gated on
  a public page (a public page may only carry an admin-gated *link*). The member `MediaItemOut`
  schema omits `deleted_at`; the admin-only `AdminMediaItemOut` adds it.

## Project-wide soft delete (deleted_at + deleted_by)

The attachment soft delete above is one instance of a platform-wide model: **removals are soft
deletes everywhere; only garbage collection stays a hard delete.** Every removable row carries two
columns, `deleted_at` (ISO timestamp) and `deleted_by` (actor uid, or `system`). A live row has
`deleted_at = NULL`, and every list/count read filters `deleted_at IS NULL`.

- **Tables (`database.SOFT_DELETE_TABLES`):** posts, comments, gists, projects, news, news_images,
  project_files, attachments, votes, reactions, bookmarks, follows, poll_votes, polls, poll_options,
  sessions, instances, instance_schedules, devii_conversations, devii_tasks, devii_lessons,
  devii_virtual_tools, user_customizations, project_forks. `init_db` loops
  `ensure_soft_delete_columns(t)` for each (adds both columns + `idx_<t>_deleted`). The Devii
  task/lesson/virtual-tool stores run on their own DB handles, so their `_ensure_indexes` adds the
  columns there too. **Every INSERT into a soft-deletable table writes `deleted_at: None,
  deleted_by: None`** so the column is born with the table on a fresh DB - this is mandatory, because
  `dataset.find(deleted_at=None)` on a table that lacks the column matches NOTHING (a false
  predicate), silently hiding all rows. Add the pair to any new insert.
- **Central helpers (`database.py`):** `soft_delete(table, deleted_by, *, stamp=None, **criteria)`
  (equality), `soft_delete_in(table, column, uids, deleted_by, *, stamp=None, **extra)` (IN-clause
  cascade), `restore(table, **criteria)`, `purge(table, **criteria)` (real delete),
  `list_deleted(table, page)` / `count_deleted(table)` (trash listings), and the event helpers
  `restore_event(stamp)` / `purge_event(stamp)` that act across ALL tables sharing one `deleted_at`.
- **Two generic chokepoints are conditionally filtered:** `resolve_by_slug(table, slug,
  include_deleted=False)` (detail-page lookups; restore passes `include_deleted=True`) and
  `paginate(table, ...)` (auto-appends `deleted_at IS NULL` when the table has the column and the
  caller did not pass `deleted_at`). `seo._collect` does the same for the sitemap. Read filters were
  threaded through every batch helper, analytics/activity/leaderboard UNION, feed/profile/listing
  route, and store; never re-introduce an unfiltered read of a soft-deletable table.
- **Toggles revive, they do not duplicate.** votes/reactions/bookmarks/follows/poll_votes look up the
  physical row regardless of `deleted_at`: toggle-off stamps `deleted_at`; re-toggle clears it on the
  same row. Counts/state reads filter `deleted_at IS NULL`.
- **Cascades share one stamp.** `content.delete_content_item` soft-deletes the item plus its
  comments, votes, engagement, project files, fork relations, and attachments with one shared
  `stamp` and `deleted_by = actor`. That timestamp identifies the whole event, so
  `restore_event`/`purge_event` reverse or finalize it atomically.
- **Authorization is owner-OR-admin, enforced on the endpoint.** Every content delete accepts the
  resource owner OR any administrator, guarded with `is_owner(...) or is_admin(user)` at the route:
  `content.delete_content_item` (posts/gists/projects, also rejects a missing item), `comments`
  `delete_comment`, `project_files` `project_file_delete`, `media` `delete_media`, and `uploads`
  `delete_attachment_route`; `admin_news_delete` is admin-only. Because the check is on the endpoint,
  one rule covers the human UI and **Devii** at once - Devii only ever calls the platform API,
  authenticated as the signed-in user, so an admin's Devii may soft-delete any member's content and a
  member's is refused with no agent-side logic. The matching `delete_*` Devii catalog tools stay
  `requires_auth` (not `requires_admin`) so a member can still delete their own, and every one is in
  the dispatcher's confirmation gate (`CONFIRM_REQUIRED`) so a delete only runs on a repeat call with
  `confirm=true`. Standalone `comment` and uploaded-`attachment` deletes are now soft like the rest
  (`soft_delete` cascade / `soft_delete_attachment`); the only attachment hard delete is the admin
  `/admin/media/{uid}/purge` and the CLI prune.
- **What stays HARD (GC / the empty-trash stage):** the async-job sweep + CLI prune/clear, the
  container metrics ring trim, gateway and Devii usage-ledger retention prunes and quota resets, the
  expired-session cleanup branch in `utils._user_from_session`, fork-rollback of a half-created
  project, the news-sync image replacement, and the admin **Purge** action. Logout is a soft delete
  (auditable via `deleted_by`); only expiry GC is hard.
- **Admin Trash surface:** `/admin/trash` (sidebar **Trash**, `routers/admin/ package`, `admin_trash.html`,
  `AdminTrashOut`) lists soft-deleted rows per table with restore/purge per row. Restore calls
  `restore_event(row.deleted_at)`; Purge calls `purge_event(...)` and unlinks attachment files /
  project-file blobs. The attachment-specific `/admin/media` view is unchanged. Admin-only docs:
  `docs/soft-delete.html` (`admin: True`, Administration section).

## Database

SQLite via `dataset` with these pragmas on every connection:

```python
PRAGMA journal_mode=WAL;      -- concurrent readers + writers
PRAGMA synchronous=NORMAL;     -- safe with WAL mode
PRAGMA busy_timeout=30000;     -- wait 30s instead of failing on lock
PRAGMA cache_size=-8000;       -- 8MB page cache
PRAGMA temp_store=MEMORY;      -- temp tables in memory
```

Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}}, on_connect_statements=[...])`.

All indexes are created via `_index()` helper wrapped in try/except - safe to run on every startup regardless of table state.

**SQLite is synchronous by design and will never be made async (hard rule).** The `dataset`/SQLite layer is synchronous and is called directly inside the async handlers. This is intentional and more than fast enough for this platform: a local-file DB tuned with WAL, `synchronous=NORMAL`, a 30s busy timeout, an 8MB page cache, and a 256MB mmap, so queries are sub-millisecond and never meaningfully stall the event loop. Do NOT convert the DB to async, do NOT offload DB calls to a threadpool/`run_in_executor`/`to_thread`, and do NOT report the synchronous DB access as a blocking-I/O or "not fully async" problem in reviews, audits, or refactors. It is a settled, sophisticated design choice.

## Dataset rules (hard-learned)

**`find()` does NOT accept raw SQL strings.** It takes keyword arguments for equality filters, dict comparison operators, or SQLAlchemy column expressions.

```python
# WRONG - causes 500 Internal Server Error:
table.find("created_at >= :start", {"start": today})
table.find(text("created_at >= :start"), start=today)

# CORRECT - dict comparison syntax:
table.find(created_at={">=": today})

# CORRECT - keyword equality:
table.find(country="France")

# CORRECT - SQLAlchemy column expression for IN clause:
table.find(table.table.columns.user_uid.in_(["uid1", "uid2"]))

# CORRECT - multiple equality filters combined:
table.find(topic="devlog", user_uid=some_uid)
```

**`update()` requires a key column list as second argument.** The first dict contains all fields including the key column.

```python
table.update({"uid": user_uid, "bio": "new bio"}, ["uid"])
```

**`db.query()` accepts raw SQL with named params as keyword arguments:**

```python
db.query("SELECT * FROM posts WHERE topic = :t", t="devlog")
# NOT: db.query("...", {"t": "devlog"})
```

**`db.query()` WRITES DO NOT AUTO-COMMIT - wrap any `db.query` INSERT/UPDATE/DELETE in `with db:` (load-bearing, caused a production deadlock).** The dataset table API (`table.insert`/`update`/`delete`) calls `db._auto_commit()` internally, but `db.query()` does NOT. SQLAlchemy 2.x autobegins a transaction on first `execute`, so a raw `db.query` write leaves an open transaction holding the SQLite write lock until that thread's connection next commits. On the request/loop thread this is masked (the next table op's `_auto_commit` flushes it), but on a **background-queue or `run_in_executor` worker thread** the thread goes idle still holding the lock, and EVERY subsequent write app-wide blocks for the 30s busy-timeout then fails `database is locked` - a full deadlock. Always commit raw writes:

```python
with db:                                   # commits + releases the write lock on exit
    db.query("INSERT INTO t (...) VALUES (:a) ON CONFLICT(...) DO UPDATE SET ...", a=1)
```

Atomic counters (e.g. `add_correction_usage`) must use raw `ON CONFLICT DO UPDATE SET col = col + excluded.col` (the table API cannot increment), so they MUST use the `with db:` wrapper. Prefer the table API whenever an atomic SQL increment is not required.

**Always check `tables` list before raw SQL queries:**

```python
if "comments" not in db.tables:
    return {}  # table doesn't exist yet
```

**Batch queries eliminate N+1 problems.** Use `get_users_by_uids()`, `get_comment_counts_by_post_uids()`, and `get_vote_counts()` from `database.py` instead of per-row lookups in loops.

**`init_db()` MUST create every queried column for any table that code filters on, even if the table is created lazily.** dataset creates a table on its FIRST insert and gives it ONLY the columns in that insert. If any code path can insert a *partial* row before the full schema exists (a CLI tool, a test fixture, a maintenance script), the table is born with a reduced schema and every later query against a missing column throws `sqlite3.OperationalError: no such column: X` (a 500), or - for an indexed column - logs a `Could not create index ... no such column` warning at startup. This is worsened by **cross-process metadata staleness**: the long-running uvicorn server reflects a table's columns once and caches them, so a column another process adds afterward is invisible to the server until it reconnects. The defence is to make `init_db()` ensure the complete column set up front, exactly like the existing `news`, `news_sync`, and `attachments` blocks:

```python
news = get_table("news")
for column, example in (("uid", ""), ("status", ""), ("synced_at", ""), ("external_id", ""), ...):
    if not news.has_column(column):
        news.create_column_by_example(column, example)
_index(db, "news", "idx_news_status", ["status"])   # now safe - column exists
```

Do this with `get_table(name)` (NOT `if name in db.tables`): `get_table` + `create_column_by_example` creates the table if it is absent, so the schema is guaranteed regardless of which process or which insert touches the table first. When you add a NEW column that any query filters/indexes, add it to the `init_db()` ensure-block too - never rely on the first insert to define it. Symptom to watch for: a page renders fine in isolation but shows empty data or 500s only after a CLI/maintenance test runs earlier in the suite (it created the table partially).

## HTML/JSON content negotiation

Every page/redirect endpoint also returns JSON when the client asks. Core in
`devplacepy/responses.py`: `wants_json(request)` (true for `Accept: application/json` or
`Content-Type: application/json`; browser `text/html` → HTML, so existing behaviour is
unchanged). **`X-Requested-With: fetch` is deliberately NOT a trigger** - the frontend sends
it on form/engagement fetches expecting the old redirect, and the four legacy engagement
endpoints (votes/reactions/bookmarks/polls) handle that header themselves. Two helpers replace the direct
returns:

- **Page GETs:** `return respond(request, "x.html", context, model=XOut)` - HTML renders the
  template; JSON does `XOut.model_validate(context).model_dump()`. One context, two renderings.
- **Action POSTs:** `return action_result(request, url, data=<resource|None>)` - HTML 302
  redirects; JSON returns `{ok, redirect, data}`. (Set cookies on the returned response after
  calling it, as `auth.py` login/signup do.)

Response models live in `devplacepy/schemas.py` (Pydantic v2, `extra="ignore"`, all-Optional so
they validate the existing context dicts directly). **Always project users through `UserOut`**
(and `AdminUserOut`) - the raw user rows contain `email`/`api_key`/`password_hash`, and the
models drop them; never serialize a raw user row. List item shapes vary (feed/gists/news/admin-
news are wrapped `{post|gist|article: …}`; projects are flat rows with `author_name`/`my_vote`)
- match the context exactly. Errors negotiate centrally: `main.py` 404/500/validation handlers
and the rate-limit/maintenance middleware, plus `utils.require_user`/`require_admin` (401/403
for JSON, 303 redirect for browsers). The four legacy AJAX endpoints (votes/reactions/bookmarks/
polls) keep their original flat JSON shapes and are left untouched. Documented in
`docs_api.py`'s Conventions group.

## devRant compatibility API (`routers/devrant/`, `services/devrant/`)

A second REST protocol mounted at `/api` that reproduces the public devRant API shape on
DevPlace data, so legacy devRant clients (the `./devranta` reference client is the spec; real
captured responses are in `./devranta/api_test_results.json`) run unchanged. It is a pure
**translation layer** over the existing domain - it adds no new content model.

**Layout.** `routers/devrant/` is the thin endpoint package: `__init__.py` builds the aggregate
router with `dependencies=[Depends(ensure_enabled)]` (gated by the `devrant_api_enabled` setting,
default on) and includes `auth.py` (login/register/profile/edit/avatar), `rants.py` (feed,
CRUD, vote, favorite, comment, search), `comments.py` (read/edit/delete/vote), `notifs.py`
(feed/clear); `_shared.py` holds the `dr_ok`/`dr_error` envelope helpers + the enable gate.
`services/devrant/` is the logic: `params.merge_params` (merges query string + body, accepting
BOTH form-encoded and JSON since devRant clients use both, GET sends auth as query params),
`tokens` (issue/validate/revoke the `devrant_tokens` triple), `ids` (`as_int`, `to_unix`,
`post_by_id`/`comment_by_id`/`user_by_id`), `serializers` (`serialize_rant`/`serialize_comment`,
`encode_tags`/`decode_tags`), `feed` (`list_rants`/`search_rants`/`load_rant_detail`, offset
pagination via `find(_limit, _offset)`), `profile` (`build_profile`), `notifications`
(`build_notif_feed`/`clear_notifications`), `avatar` (`avatar_payload` + `render_png` via
`cairosvg`).

**ID mapping (load-bearing).** devRant integer ids ARE the auto-increment `id` PK every
`dataset` table already has: `rant_id`=`posts.id`, `comment_id`=`comments.id`,
`user_id`=`users.id`, `token_id`=`devrant_tokens.id`. No translation table. Serialization
converts ISO `created_at` to unix via `ids.to_unix`.

**Auth.** `POST /api/users/auth-token` accepts username OR email, verifies with passlib, and
inserts a `devrant_tokens` row (in `SOFT_DELETE_TABLES`; born-live; `key`=`secrets.token_hex`,
`expire_time` from `session_max_age_days`). Every later call re-validates `(token_id, token_key,
user_id)` with `tokens.resolve_user(params)`. Read endpoints take an OPTIONAL viewer
(`resolve_user` may return None); write endpoints return `_shared.unauthorized()` (401) when it
does.

**Writes reuse the audited native cores - never duplicate.** Implementing this drove four DRY
extractions in `content.py` (`apply_vote`, `create_comment_record`, `delete_comment_record`,
`set_bookmark`) and one in `utils.py` (`register_account`); the native `routers/votes.py`,
`routers/comments.py`, and `routers/auth/signup.py` were refactored onto the SAME functions. So a
devRant rant/comment/vote awards XP, fires notifications, writes the audit row, and soft-deletes
exactly like the UI path. Rant create calls `content.create_content_item` directly; rant delete
calls `content.delete_content_item` (full cascade) and returns the devRant envelope.

**Field mappings.** Rant `text` = `title\n\n content` (inbound rants have no title, `topic`
forced to `"rant"`). devRant `tags` round-trip verbatim through a new `posts.tags` JSON column
(`init_db` ensures it; `decode_tags` falls back to `[topic]`). `profile_skills` is derived from
`bio` (`skills_from_bio`, no native skills field). `favorite`/`unfavorite` map to bookmarks via
`set_bookmark`. Avatars are real PNGs from the multiavatar engine at
`GET /api/avatars/u/{username}.png`; `user_avatar.i` points at that path with a deterministic
`b` background colour. The feed envelope includes the auxiliary devRant keys (`settings`, `set`,
`wrw`, `dpp`, `num_notifs`, `unread`, `news`) with safe values so clients parse cleanly.

**Envelope.** `dr_ok(**f)` -> `{success:true, **f}`; `dr_error(msg, status, **extra)` ->
`{success:false, error:msg, ...}`. Logical failures stay `200` except bad login -> `400`
(matches the captured devRant behaviour). Host routing (legacy clients hard-code devrant.com) is
an infra/DNS concern, out of app scope.

## XML-RPC bridge (`services/xmlrpc/`, `routers/xmlrpc.py`)

The whole REST surface is also exposed over XML-RPC at `/xmlrpc`, generated **automatically
from `docs_api.API_GROUPS`** so it never drifts: add a documented endpoint and it becomes an
XML-RPC method for free. Pieces:

- **`registry.py`** walks `docs_api.API_GROUPS` and yields one `Method` per endpoint. Method
  name = the endpoint `id` with `-`->`.` (`posts-create` -> `posts.create`). It carries the HTTP
  method, path, auth, and the `param` list split by `location` (path/query/form). It also
  builds the `system.methodHelp` text and `system.methodSignature` from the same data.
- **`bridge.py`** translates ONE XML-RPC call into an internal REST request. The single struct
  argument is a dict of named params; the bridge substitutes `{name}` path segments, routes the
  rest to query (GET) or form (write methods), sets auth headers, and calls
  `config.INTERNAL_BASE_URL` with a **synchronous** `stealth.stealth_sync_client` (the forking
  server is sync). It returns the decoded JSON; any REST `status >= 400` becomes an
  `xmlrpc.client.Fault` whose `faultCode` is the HTTP status. A missing required param faults
  before the call.
- **`server.py`** is `ForkingXMLRPCServer(ForkingMixIn, BridgeDispatcher, SimpleXMLRPCServer)`,
  `allow_none=True`, with `register_introspection_functions()` + `register_multicall_functions()`
  and one registered function per generated method. The custom request handler captures auth
  from the POST into `server.current_auth` (safe because each request runs in its own forked
  child) and the bridge forwards it: `X-API-KEY` and `Bearer` become an api-key header; a raw
  `Authorization: Basic` header is **forwarded verbatim** so the REST layer does its normal
  `username/email:password` login (`utils._user_from_basic`) - this is what makes
  `xmlrpc.client.ServerProxy("http://user:pass@host/xmlrpc")` work. It also forwards
  `X-Real-IP`/`X-Forwarded-For` so **rate limiting is attributed to the real caller** on the
  internal REST hop. Run standalone with `python -m devplacepy.services.xmlrpc.server` or the
  `devplace-xmlrpc` console script.
- **`XmlrpcService(BaseService)`** (`services/xmlrpc/__init__.py`) supervises the server as a
  **subprocess** (never fork the asyncio/uvicorn process): `on_enable` spawns, `run_once`
  health-checks and respawns, `on_disable` terminates. `default_enabled=True`, but like every
  service only the lock-owner worker runs it, so exactly one process binds the port.
- **`routers/xmlrpc.py`** reverse-proxies `/xmlrpc` (POST = RPC, GET = usage banner) to the
  forking server on `config.XMLRPC_PORT`, forwarding the auth + real-IP headers. `/xmlrpc` is
  **exempt from the request rate limiter** in `main.py` (enforcement happens on the forwarded
  internal hop). nginx forwards `/xmlrpc` in production with longer timeouts.

Auth has three equivalent forms: `api_key` inside the param struct (works with a vanilla
`ServerProxy`), an `X-API-KEY`/`Bearer` header on the transport, or HTTP Basic via a
credentialed URL `http://username:password@host/xmlrpc`. Public endpoints need none. Config:
`config.XMLRPC_BIND`/`XMLRPC_PORT` (loopback, `10550`). Documented at `/docs/xmlrpc.html`.

## FastAPI patterns

- **All routes are async.** Form data is validated via a typed Pydantic body param: `data: Annotated[SomeForm, Form()]` (models in `models.py`). Read raw `await request.form()` only when also handling an uploaded file (a separate `File()` param would embed the model under its parameter name).
- **Return `RedirectResponse(url=..., status_code=302)`** for redirects.
- **Return `templates.TemplateResponse("name.html", {...})`** from `devplacepy.templating` to render.
- **Never create your own `Jinja2Templates` instance.** Import the shared one: `from devplacepy.templating import templates`.
- **Register new routers in `main.py`:** `app.include_router(router_instance, prefix="/{path}")`
- **`require_user(request)`** redirects guests (303) to login, but raises **401** when credentials *were* supplied yet invalid (so API clients get a clear error). Only post/comment/vote/etc. routes use it - the feed is public. `require_admin` and `require_user_api` (401-only) build on it. None of these changed signatures, so all auth schemes work through existing call sites.
- **For a missing detail resource, `raise not_found("X not found")`** (`utils.py`) - it returns an `HTTPException(404)` that the global handler renders as `error.html`. Do not return a bare `HTMLResponse(..., status_code=404)`.
- **Detail pages reuse `load_detail(table, target_type, slug, user)`** (`content.py`) for item+author+comments+attachments+`star_count`+`my_vote`; list pages reuse `enrich_items(items, key, authors, extra_maps, user=...)`. Prefer these over manual per-row loading (posts/projects/gists detail and feed/gists/profile lists already do).
- **`get_current_user(request)` resolves ALL auth schemes** (`utils.py`), in order: `session` cookie → `X-API-KEY` header → `Authorization: Bearer <api_key>` → `Authorization: Basic base64(username-or-email:password)`. It memoizes the result on `request.state._auth_user` (resolved once per request - it is called by both the maintenance middleware and the route) and caches users in `_user_cache` (by session token, `"k:"+api_key`, or a hash of the Basic header). Because every router already routes through this one function, API-key/Bearer/Basic auth work on every page/action with no per-route code. Use it for pages viewable by guests too (feed, news detail, projects).
- **Post deletion must cascade:** delete comments and votes first, then the post. Always check ownership: `post["user_uid"] == user["uid"]`.
- **Message deduplication needed** when `sender_uid == receiver_uid` (messaging yourself): `seen = set()` of message UIDs before appending to result list.

## Real-time messaging (`devplacepy/services/messaging/`)

The `/messages` feature is a live WebSocket chat layered over the SAME `messages` table and audit/notification cores as before; there is no parallel chat store. The WS only adds live delivery, presence, typing, and read receipts on top of the existing persistence.

- **WS endpoint `/messages/ws`** lives on the existing messages router (no new router/prefix; already mounted at `/messages`). It is **NOT lock-owner gated** (deliberately - unlike `/devii/ws`): `await websocket.accept()`, resolve the owner via session cookie / api-key (`_resolve_ws_user` reuses `utils._user_from_session`/`_user_from_api_key`), and accept on EVERY worker. Gating on `service_manager.owns_lock()` was the original design and was wrong here: when no worker holds the service lock (services disabled, or the lock not yet acquired) every socket got closed `4013` and the client fast-retried every 200ms forever (a connection storm with no stream, and the send form fell back to a full-page POST reload). Chat must not depend on the background-service lock. Messaging requires an authenticated user - guests are closed with `1008`. The receive loop is wrapped in `try/except WebSocketDisconnect` + a broad `except` (never crash the worker) and ALWAYS unregisters the socket in `finally`.
- **Connection registry** is the `ConnectionManager` singleton `message_hub` (`services/messaging/hub.py`), **one per worker**: `user_uid -> set[WebSocket]`, plus an in-process `last_seen` map and a bounded `delivered` LRU set (`mark_delivered`/`was_delivered`, cap 4000) used to dedupe direct vs relayed delivery. `register`/`unregister` return whether the user just came online / went offline (for presence announces). `send_to_user`/`send_to_users` fan a frame out to all of a user's sockets (multi-tab) and swallow dead-socket errors. Presence and last-seen are in-process per worker (no DB column).
- **Cross-worker delivery (the relay).** Because the WS accepts on every worker, a message persisted on worker A must still reach a recipient whose socket lives on worker B. `services/messaging/relay.py` `message_relay` (a per-worker singleton asyncio loop, started on the first socket connect, self-stops when `message_hub.has_connections()` is false) polls `SELECT * FROM messages WHERE id > :watermark` (uuid7-backed autoincrement `id`) every ~1s and pushes any row whose sender/receiver has a LOCAL socket and that was not already `was_delivered` to those local sockets, advancing the watermark to the max row seen. Same-worker sends are delivered INSTANTLY by `broadcast_message` (which calls `message_hub.mark_delivered` so the relay skips them); the relay only fills the cross-worker gap, so same-worker latency is zero and cross-worker latency is bounded by the poll interval. The relay primes its watermark to the current `MAX(id)` on first start so it never replays history. **Trade-off:** typing/read/presence frames are in-process per worker only - across the two `make prod` workers those ephemeral signals reach only same-worker peers (message delivery is always correct). On a single dev worker everything is instant and complete.
- **DRY persist choke point:** `services/messaging/persist.py` `persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side. When `request` is a `Request` it audits via `audit.record`; on the WS path it now passes `request=websocket` (auditing via `audit.record_system` with `actor_kind="user"` + `origin="websocket"`, same `message.send` key/category - no new event invented; presence/typing/read are ephemeral and NOT audited).
- **AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content.** `"messages": ("content",)` is in the `CORRECTABLE_FIELDS` registry and `persist_message` invokes `schedule_correction`/`schedule_modification` (sender = the user), so typing `@ai <instruction>` in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through `routers/messages._finalize_and_broadcast(sender, message, request, client_id=None)`: it awaits any pending SYNC correction/modification futures stashed on `request.scope[PENDING_SCOPE_KEY]` (the same `PENDING_SCOPE_KEY`/`loop.run_in_executor` mechanism the HTTP `await_pending_corrections` middleware uses), RE-READS the message row, and broadcasts the FINAL (corrected/modified) content via `broadcast_message`. The HTTP `POST /send` and the WS `/ws` send path both call it; the WS path passes `request=websocket` to `persist_message` so the modifier stashes its executor future on the websocket scope and the handler awaits it directly (HTTP middleware does not run for websockets). A message with no `@ai` directive and correction off has nothing pending, so the broadcast is immediate with zero added latency. Frontend `static/js/MessagesLayout.js` reconciles the sender's own echo by `client_id` and replaces the optimistic bubble's `.rendered-content` with the server's authoritative `frame.content`, re-rendered - so the sender sees their `@ai` directive resolved in-place (and corrections become visible to the sender too). See "AI content correction" and "AI modifier".
- **WS protocol frames.** Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"presence", with_uid}` (one-shot presence query). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, receiver_uid, content, created_at, time_ago, client_id}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"presence", user_uid, online, last_seen}` (announced to everyone in an open conversation with the user on connect/disconnect, and as a direct reply to a `presence` query). The WS `read` path reuses `mark_conversation_read` (the same `UPDATE messages SET read=1` SQL the page uses) and `clear_messages_cache`.
- **Renderer integration (XSS control).** Live/echoed message bubbles are NEVER injected as raw HTML. `MessagesLayout.buildBubble` creates a `<div class="rendered-content" data-render>`, sets its `textContent` to the message body, then runs `contentRenderer.applyTo(el)` + marks `img:not(.avatar-img)` with `data-lightbox` + `contentRenderer.highlightAll()` - exactly the `ContentEnhancer.initContentRenderer` sequence (emoji shortcode -> marked -> DOMPurify.sanitize -> highlight.js -> image/YouTube/autolink). This gives Discord-style emoji, image and YouTube embeds, autolinking, and sanitization for free.
- **Frontend.** `MessagesSocket.js` (mirrors `DeviiSocket.js`: reconnect with backoff, `4013` silent fast-retry, `onReady` fires on the first server frame not raw open) points at `/messages/ws`. `MessagesLayout.js` (instantiated on `app` in `Application.js`) opens the socket on the messages page, sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, appends incoming messages live through the renderer, auto-scrolls, throttles + shows the typing indicator, flips the read-receipt double-check, renders the presence dot / last-seen in the header, and live-updates the conversation-list preview + unread dot + ordering. If the socket is not open the send `<form method=POST>` submits normally (no-JS fallback). The template carries `data-self-uid`/`data-other-uid`/`data-other-online`/`data-other-last-seen` on `.messages-layout`. Styling (presence dot, typing dots, read-receipt, unread dot, pending opacity) is in `messages.css` using design tokens only, responsive down to 360px.
- **Attachments stream live.** The single `message_frame` builder (`services/messaging/persist.py`, used by BOTH `broadcast_message` and the relay) fetches `get_attachments("message", uid)` and includes a slimmed list (`uid`/`url`/`thumbnail_url`/`is_image`/`is_video`/`original_filename`/`file_size`/`mime_type`) on every frame. `MessagesLayout.renderAttachments` mirrors `_attachment_display.html` in the DOM (image -> `img.gallery-thumb` marked `data-lightbox`+`data-full`, video -> `<video>`, else a download link) - no `innerHTML`, so it stays XSS-safe. The sender's optimistic bubble shows an "Uploading attachment..." placeholder until its own echo arrives and swaps in the real gallery. **Attachment-only messages (empty caption) are allowed**: the content input dropped `required`, `sendViaSocket` sends when content OR attachments are present, and `persist_message` accepts empty content when `attachment_uids` is non-empty (`if not content and not attachment_uids: return None`). `dp-upload` (mode `attachment`) uploads each file to `/uploads/upload` and writes the uids into its hidden `attachment_uids` input; `collectAttachments()` reads that before the send, then `dp-upload.clear()` resets it. The chat uploader sets `hide-chips` so only the `(N)` count badge shows (no filename chips) - a generic `dp-upload` attribute. The send button shows a spinner and is disabled while busy: `MessagesLayout.refreshSendButton()` ORs `_uploading` (driven by the new `dp-upload:busy` event) with `_pendingSends` (a set of in-flight send `client_id`s, cleared on the echo or an 8s safety timeout), toggling `.is-sending` + `disabled` on `.messages-send-btn`.

## Role-based visibility (generic + DRY)

- **One source of truth for role/visibility checks**, registered as Jinja globals in `templating.py` - never hand-roll `user.get('role') == 'Admin'` or `user['uid'] == x['user_uid']` in a template again:
  - `is_admin(user)` (also `utils.is_admin`, reused by `require_admin` and `docs.py`) - admin-only UI.
  - `owns(item, user)` (= `content.is_owner`) - per-item ownership (e.g. each comment). Page-level detail templates keep using the `is_owner` **bool** passed in their context (post/gist/project/profile); do not call `is_owner(...)` as a function - that name is a context bool and shadows globals.
  - `is_self(user, uid)` - "is this me" (profile follow vs edit, leaderboard highlight).
  - `guest_disabled(user)` → emits ` disabled aria-disabled="true" title="Log in to participate"` for guests (empty for members); `login_hint(user)` → a small login link. Both return `Markup`.
- **Role values are stored capitalized:** `users.role` is exactly `"Admin"` or `"Member"` (first registered user is `"Admin"`, `auth.py`); `is_admin` compares `== "Admin"` case-sensitively. The CLI is the only lowercase surface (`devplace role set ... <member|admin>` writes `role.capitalize()`; `role get` prints `.lower()`). A lowercase role in the DB silently defeats every admin check - never write a raw lowercase role.
- **The shadow rule generalizes beyond `is_owner` to ANY Jinja global** (`is_admin`, `avatar_url`, `format_date`, `is_self`, `owns`, `guest_disabled`): `respond(req, tmpl, ctx, model=XOut)` hands the **same** `ctx` to the Pydantic model and the template, and a context key shadows the same-named global across the whole `base.html` chain. A bool named `is_admin` in the context makes `base.html`'s `{% if is_admin(user) %}` raise `TypeError: 'bool' object is not callable` - a 500 that only fires for the branch invoking the global (logged-in users, not guests, which is why guest-only smoke tests miss it). Name viewer/permission flags distinctly (`viewer_is_admin`) in both schema and context. Real issue fixed on `/issues/{number}`; regression-guarded by `tests/api/issues/create.py::test_issue_detail_renders_for_{member,admin}` (they render the page as an authenticated Member/Admin and assert 200 + the admin-only control).
- **Policy enforced everywhere:** guests see all non-admin content read-only with action controls **shown but disabled** (`guest_disabled` on vote/star/react/poll/bookmark/follow/comment submit; create FABs become `/auth/login` links via `.feed-fab.login-required`); members get full member actions; **role badges render only to admin viewers** (`{% if is_admin(user) %}` around every `*.role` label). Backend stays the real gate (`require_user`/`require_admin`).
- Docs admin gating is unchanged behaviourally but now uses `is_admin` (`docs_base.html` `DEVPLACE_DOCS.isAdmin`, `docs/index.html`, `docs.py`).
- **Tests:** the role-gating e2e tests across `tests/e2e/` (guest/member/admin via `page`/`bob`/`alice`) are the UI enforcement; `tests/api/auth/matrix.py` is the backend companion. Guest action controls are asserted **disabled** (not absent) - don't reintroduce `count() == 0` assertions for them.

## Key conventions

- No comments/docstrings in source - code is self-documenting.
- Forbidden variable name patterns: `_new`, `_old`, `_temp`, `_v2`, `better_`, `my_`, `the_` (see CLAUDE.md for full list).
- Form validation uses Pydantic models in `models.py` via `Annotated[Model, Form()]` params; invalid input is caught by the global `RequestValidationError` handler in `main.py` (auth pages re-render with messages at 400, other routes redirect).
- Template globals: `get_unread_count(user_uid)`, `get_unread_messages(user_uid)`, `get_user_projects(user_uid)`, `avatar_url(style, seed, size)`, `format_date(dt_str, include_time=False)` (ISO → `DD/MM/YYYY` or `DD/MM/YYYY HH:MM`).
- `DEVPLACE_DATABASE_URL` env var overrides the SQLite path (used by tests).
- All `RedirectResponse` must use `status_code=302` (integer, not `status` module).
- All `dataset` operations are synchronous and run in the async event loop - keep them fast. No external HTTP calls in request handlers. The synchronous DB is a deliberate design choice and will never be made async; do not flag it (see the Database section).
- For ownership-sensitive operations (delete, edit), always check `user["uid"]` against the resource's `user_uid`.

## Clickable Avatars & Usernames

Every avatar and username in the UI links to the user's profile page. Use the `_avatar_link.html` and `_user_link.html` include components:

```html
{% set _user = item.author %}
{% set _size = 32 %}
{% set _size_class = "sm" %}
{% include "_avatar_link.html" %}

<a href="/profile/{{ user['username'] }}" class="post-author-link">{{ user['username'] }}</a>
```

The include files expect: `_user` (dict), `_size` (pixels), `_size_class` ("sm"|"md"|"lg").

Affected templates: `base.html`, `feed.html`, `post.html`, `profile.html`, `messages.html`, `notifications.html`.

## Image Upload

When a user uploads an image during post creation, the markdown `![](/static/uploads/{filename})` is appended to the post content. The ContentRenderer then renders it as an `<img>`. All URLs are relative.

```python
content += f"\n\n![](/static/uploads/{image_filename})"
```

File validation: max 5MB, allowed extensions: `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.svg`.

## Attachments and media

`attachments.py` is the single store for files referenced via `attachment_uids` (posts, comments, projects, gists, messages, issues). One gate, `is_extension_allowed(ext)`, decides what may be uploaded and is used by both `routers/uploads.py` and `store_attachment()` - do not reintroduce a second check. Its semantics: when the admin `allowed_file_types` setting is non-empty it is the **authoritative** allowlist (mime auto-detected via `mimetypes` for extensions outside the built-in `ALLOWED_UPLOAD_TYPES` map); when empty it falls back to the built-in map, which includes images plus the browser-playable video formats `.mp4/.webm/.ogv/.mov/.m4v`. (Before this, `allowed_file_types` could only *narrow* the hardcoded list, never add to it - which is why adding a video extension there had no effect.)

**Ingesting a file from a URL.** `store_attachment_from_url(url, user_uid, filename=None)` (async, in `attachments.py`) is the remote counterpart to `store_attachment`: it downloads the URL on the server through `fetch_remote_file()` - SSRF-guarded (`_guard_public_url` resolves the host and refuses private/loopback/reserved/multicast addresses, mirroring the Devii fetch guard) and size-capped (streams, aborting once `_get_max_upload_bytes()` is exceeded) - resolves a filename from the URL path or the response `Content-Type` (`MIME_TO_EXT`), then calls `store_attachment()` so the bytes land in the **exact same** pipeline (validation, thumbnailing, DB row). It raises `RemoteFetchError(message, status)` which the route maps to an HTTP status. It is exposed at `POST /uploads/upload-url` (`UploadUrlForm{url, filename?}`, `require_user_api`) and as the Devii catalog action `attach_url` (handler `http`, `requires_auth=True`); both return the same record as `/uploads/upload`. The returned `uid` binds to a resource the same way as any upload - via `attachment_uids` at create/edit time - so attaching a remote image is just `attach_url` then `create_post`/`create_project`/etc. with that uid. Do not re-download remote files in a router; reuse this helper so the guard and size cap stay in one place.

`_row_to_attachment()` / `store_attachment()` expose `is_image` and `is_video` (derived from the mime prefix). The shared partial `templates/_attachment_display.html` branches image -> `<img>`, video -> `<video controls preload="metadata" class="gallery-video">`, else download link; rendering through this one partial is what makes video work across every feature at once. `AttachmentOut` (`schemas.py`) carries both flags - add new display keys there too or JSON drops them.

Media is served **inline** (not forced-download) for known-safe types only. The set `INLINE_MEDIA_EXTENSIONS` in `main.py` (`UploadStaticFiles`) and the matching `map $uri $upload_disposition` in `nginx/nginx.conf.template` must stay in sync: images/video/audio -> `inline` (so `<video>` plays and seeks via Range), everything else -> `attachment`. SVG is deliberately excluded from both (stored-XSS defense). `ContentRenderer.js` embeds direct video URLs typed into content via `videoExtRe`, mirroring its image handling.

### The upload button (`dp-upload`)
Every file-upload UI is the one custom element `dp-upload` (`static/js/components/AppUpload.js`); it
replaced `AttachmentUploader.js` (deleted). It shows a clean button with a count badge + removable
filename chips (no thumbnail grid), and runs in one of three `mode`s:
- `attachment` (default, the 7 forms via `_attachment_form.html` / inline in `_comment_form.html` and
  `messages.html`): uploads each file to `/uploads/upload` on select and keeps the hidden
  `<input name="attachment_uids">` the routers already link - **server contract unchanged**. Limits
  come from `max-size`/`max-files`/`allowed-types` attributes (fed by the Jinja globals).
- `direct` (the project file browser, `project_files.html`): uploads to a custom `endpoint` with
  `field-name` plus a settable `extraFields` (e.g. `{path}`) and emits `dp-upload:uploaded` /
  `dp-upload:done` / `dp-upload:error`. `ProjectFiles.uploadTo(dir)` sets `extraFields` then calls
  `widget.open()`, and refreshes the tree on `done`.
- `field` (create-post image, `feed.html`): wraps a real `<input type="file" name="image">` that
  submits with the form - no AJAX, inline-image flow unchanged.
The component validates size/type/count and reports errors via `app.toast`. CSS is `.dp-upload-*` in
`components.css`; the old `.attachment-upload-*` upload-widget styles were removed, but the
`.attachment-gallery`/`.attachment-lightbox` display styles (for already-saved attachments) remain.

## Testing patterns

### Layout (three tiers, a directory tree that mirrors the path)

Tests live in three category directories under `tests/`, chosen by *what they exercise* (decided by fixtures: browser fixture `page`/`alice`/`bob` = e2e; `app_server`/`seeded_db` or any HTTP call = api; `local_db`-only or no fixture = unit):

- `tests/api/` - HTTP integration against the live uvicorn subprocess (`app_server`/`seeded_db`, `requests`/`httpx` vs `BASE_URL`), no browser. **Organised by endpoint path.**
- `tests/e2e/` - Playwright (`page`/`alice`/`bob`). **Organised by endpoint path.**
- `tests/unit/` - pure in-process (`local_db`/no fixture), library functions directly, no server. **Organised by the SOURCE module path** of the code under test.

**The directory tree mirrors the path - one segment per directory, last segment is the file, NOT a flat `_`-joined name.**

- **api/e2e:** each route segment is a directory, the final segment is the file; `{param}` segments dropped, each segment lowercased + non-alphanumerics stripped. `GET /admin/ai-usage` -> `tests/e2e/admin/aiusage.py`, `GET /projects/{slug}/files/lines` -> `tests/api/projects/files/lines.py`. A bare/collection path that is also a parent uses `index.py` in its own directory (`/posts` -> `tests/e2e/posts/index.py`, `/posts/create` -> `tests/e2e/posts/create.py`); `GET /` -> `tests/<tier>/root.py`. Multiple verbs of one path share a file.
- **unit:** mirror the `devplacepy.` module path (dots -> directories): `devplacepy/utils.py` -> `tests/unit/utils.py`, `devplacepy/services/audit/store.py` -> `tests/unit/services/audit/store.py`.

Every directory is a package (`__init__.py`). `pyproject.toml` has `testpaths = ["tests/unit","tests/api","tests/e2e"]` + `python_files = ["*.py"]`; `conftest.py` stays at `tests/` root (fixtures inherit into all tiers). Tier targets: `make test-unit` / `make test-api` / `make test-e2e`. When adding a test, derive the directory path from the route (api/e2e) or the source module (unit) and append to the matching file (create it + its package dirs if absent). Shared helpers imported across files point at the canonical new module path (e.g. `from tests.e2e.post import create_post`), never a removed `tests.test_*` module.

### General

- **932 tests across `tests/{unit,api,e2e}/`.** Playwright integration + HTTP + unit tests. All must pass before any merge.
- **Tests use `-x` (fail-fast).** The suite stops at the first failure. Fix that test, then re-run.
- **NEVER run tests unless specifically asked by user.** Not the full suite, not a single file - do not run any tests unless the user explicitly requests it.
- **Validate each touched language manually:** Python (compile + import), JS (parse / bracket matching), CSS (brace matching), HTML (tag matching). Zero tolerance.

### Playwright navigation

- **Every `page.goto()` must use `wait_until="domcontentloaded"`**, never the default `"load"`. CDN scripts and avatar images cause `load` to timeout.
- **Every `page.wait_for_url()` must also use `wait_until="domcontentloaded"`** for the same reason.
- **Prefer `page.locator(...).wait_for(state="visible")`** over bare `wait_for_selector` - it gives better error messages.
- **Default timeout is 15 seconds** (increased from 10s for CDN script loading).

```python
page.goto(f"{BASE_URL}/feed", wait_until="domcontentloaded")
page.locator(".feed-fab").first.wait_for(state="visible", timeout=10000)
```

### Delete button locator scoping

When both a post Delete and comment Delete button exist, always scope to the comment:

```python
# CORRECT - scoped to comment:
page.locator(".comment-action-btn:has-text('Delete')")

# WRONG - matches both post and comment Delete:
page.locator("button:has-text('Delete')")
```

### Confirm dialogs

Delete buttons carry `data-confirm`, which `ModalManager` turns into a native `confirm()`
dialog. Playwright auto-dismisses dialogs (treated as Cancel), so the form never submits.
Accept the dialog **before** clicking, or the delete silently does nothing:

```python
page.once("dialog", lambda d: d.accept())
page.locator(".comment-action-btn:has-text('Delete')").click()
```

### Comment hash navigation

Opening a URL with a `#comment-<uid>` fragment scrolls the comment into view and adds the
`comment-highlight` class (`NotificationManager.initHashScroll`). Tests assert on
`#comment-<uid>.comment-highlight` after the page loads.

### Browser context

- **`browser_context` is session-scoped** (one per test session, shared by all tests in all files).
- **Cookies are cleared per test via `browser_context.clear_cookies()`** in the `page` fixture.
- **Each test gets a fresh `page`** from the shared context.
- **`bob` fixture creates its own context** from the session `browser` - necessary for multi-user tests.
- **Never share a page between two logged-in users** in the same test - use separate contexts.

### Test users

- **`alice_test` / `bob_test` are seeded once at session level** via HTTP POST to `/auth/signup`.
- **`alice` fixture logs in alice_test** via the login form.
- **`bob` fixture logs in bob_test** in a separate Playwright context.
- **Use `alice` for single-user tests.** It returns `(page, user_dict)`.

### Global `site_settings` tests

- **The uvicorn server is shared for the whole session.** A test that flips a global setting (maintenance mode, registration, rate limit, session length) changes behavior for every later test, so **restore the value in a `try/finally`** - see `tests/e2e/operational.py`.
- **Saving through the admin form is the reliable mutation path**, not a direct DB write: the form handler runs in the server process and calls `clear_settings_cache()`, so the change takes effect immediately (a direct DB write waits out the 60s server-side cache).
- **`conftest.py`'s `app_server` seeds `rate_limit_per_minute="1000000"`** so the suite is never throttled and the settings form round-trips a safe value instead of the template's `"60"` default.
- **Rate-limit unit tests patch `m.get_int_setting`**, not the `RATE_LIMIT` constant - the constant is now only the fallback default passed to `get_int_setting`.
- **Service enabled/disabled state is global state too - restore it.** Stopping a service through `POST /admin/services/{name}/stop` sets its `service_{name}_enabled` setting to `"0"` for the rest of the session. A test that stops a service (e.g. `test_gateway_disabled_returns_503` stops `openai` to assert the gateway returns 503) MUST restart it in a `try/finally` (`POST /admin/services/{name}/start`), and any `finally` that ends in `stop` should end in `start` instead - the default, expected state is enabled. Leaving the gateway stopped makes a later file's request hit `is_enabled()` first and get a `503` where it expected a `401`/`200`. The `openai` gateway's `handle()` checks `is_enabled()` (503) BEFORE `authorize()` (401), so a stopped gateway masks the auth result entirely.
- **The 1,000,000/min rate-limit seed only applies after the server's 60s settings cache expires.** `conftest.py` seeds `rate_limit_per_minute` with a raw DB insert (no `clear_settings_cache()`), so for the first ~60s of server uptime the server still uses the `60`/min default. A high-volume request loop (e.g. `test_auth_matrix`) run in the FIRST minute can therefore trip a spurious `429`; in the full suite that test runs well past the window so it never does. If you reproduce a `429` only in a tiny isolated run, it is this warm-up artifact, not a real auth/rate-limit issue.

### Failure handling

- **Failure screenshots auto-save** to `/tmp/devplace_test_screenshots/`.
- **Tests stop at first failure** (`-x` flag in Makefile). No cascading failures.
- **If the server won't start, kill leftover processes:** `kill -9 $(pgrep -f "uvicorn")`

### Common pitfalls

| Pitfall | Fix |
|---------|------|
| `goto`/`wait_for_url` times out | Add `wait_until="domcontentloaded"` |
| CDN scripts block page load | Use `defer` on all `<script>` tags |
| 500 on dataset `find()` | Use dict comparison syntax, not raw SQL |
| N+1 query slowness | Use batch helpers: `get_users_by_uids()`, `get_comment_counts_by_post_uids()` |
| Modal not opening/closing | Use `style.display`, not `classList.add/remove` |
| Dual Delete buttons match | Scope to `.comment-action-btn` in tests |
| Dual Post buttons match (feed inline comment) | Scope to `#create-post-modal button.btn-primary:has-text('Post')` in tests |
| Edit modal textarea conflicts with comment textarea | Scope to `.comment-form textarea[name='content']` for comments |
| Tests fail in sequence | Session-scoped context + `clear_cookies()` per test |
| Double messages in chat | Deduplicate by message UID with `seen` set |
| Avatar generation fails | Falls back to initial-based SVG - check multiavatar import |
| `Http.getJson` returns HTML / `JSON.parse` throws | The helper sends `Accept: application/json`; a content-negotiated route (`respond(..., model=...)`) needs it. Don't hand-roll `fetch()` without that header against a negotiated route |
| Page renders but data empty / 500 only after a CLI test ran earlier | A partial insert created the table with a reduced schema. Ensure the full column set in `init_db()` (see "Dataset rules -> `init_db()` MUST create every queried column") |
| `no such column: X` 500 in the server log | Same root cause: a queried/indexed column is missing because the table was created lazily by a partial insert. Add the column to the `init_db()` ensure-block |
| Action button click times out ("element is not visible") | The button moved into the `.project-actions-overflow` menu. Open it first: click `.project-actions-more`, then `.context-menu-item:has-text('<Label>')` (see `tests/e2e/projects/index.py`) |
| `ImportError: cannot import name X from devplacepy.utils` in a test | The util was renamed (e.g. `badge_info` -> `get_badge`). Update the test import to the current name; grep `templating.py` globals for the new name |
| Admin page heading assertion fails | Admin pages use `<h2>` inside `.admin-toolbar`, not `<h1>` - assert `text=` or `.admin-toolbar h2:has-text(...)` |
| `test_auth_matrix` reports `422` instead of `401`/`403` | A POST route's form params are undocumented in `docs_api.py`, so the matrix sends an empty body and FastAPI validation (422) fires before the auth guard. Document every required form param so auth is reached and tested |
| `test_auth_matrix` reports `503` on a gateway endpoint | A prior test left the `openai` service stopped. Restore it in `finally` (see "Global `site_settings` tests -> Service enabled/disabled state") |

## Anatomy of a feature (the relationship map)

DevPlace has no isolated change. A feature is **one data source consumed by four faces**, plus the layers that feed and document it. Forgetting a face is the recurring defect, so internalize the graph before editing.

### The dependency graph (data flows downward; each arrow is a contract)

```
  models.py (input)          schemas.py (output *Out)
        |                            |
        v                            v
  database.py  ----helpers---->  routers/{area}.py  --register-->  main.py
        ^                       /     |       \        \
        |                      /      |        \        \
   (no N+1: use batch     HTML       JSON      Devii     SEO
    helpers, _in_clause)   |          |       catalog    seo.py
                           v          v         |         |
                  templates/{area}    (consumed  v         v
                  + static/css        by API    docs_api  robots/
                  + static/js         clients)  .py       sitemap
                                                 |
                                                 v
                                         routers/docs/ package (prose)

  Then ALWAYS: README.md (product)  AGENTS.md (mechanics)  CLAUDE.md (rules)
```

### The four faces of one route

A single handler in `routers/{area}.py` is simultaneously:

1. **HTML** - `respond()` returns a rendered template for browsers.
2. **JSON** - the SAME `respond(..., model=XOut)` returns JSON when `Accept: application/json`. The `*Out` schema in `schemas.py` is the gate: a context key not declared on `*Out` is dropped from JSON even though the template still sees it. (See "HTML/JSON content negotiation".)
3. **Agent tool** - `services/devii/actions/catalog.py` exposes the route as an `Action` so Devii can call it on the user's behalf. `requires_auth` mirrors the route guard.
4. **Documented endpoint** - `docs_api.py` `endpoint()` describes it (params, auth, `sample_response`) in the right group, and it renders at `/docs/{group}.html`.

If you touch the route, ask of each face: *does this one need updating too?* A new public read almost always needs all four.

### What feeds the route

- **`models.py`** - typed `Form` model for input; FastAPI validates it and `main.py`'s `RequestValidationError` handler renders/redirects on failure.
- **`schemas.py`** - the `*Out` model for the JSON face.
- **`database.py`** - all data access. Add a helper (guard `if "table" in db.tables`, paginate with `build_pagination`, batch with `get_users_by_uids` / `_in_clause`); never write an N+1 loop in the router. Indexes go in `init_db()` as `CREATE INDEX IF NOT EXISTS`.
- **`main.py`** - prefix registration for new routers; startup wiring for new services.

### What the route feeds

- **View**: `templates/{area}.html` (shared `templates` instance, partials, globals), `static/css/{area}.css` (via `{% block extra_head %}`), `static/js/*` ES6 module on `app` (via `{% block extra_js %}`).
- **Agent**: the `catalog.py` `Action`.
- **SEO**: `base_seo_context` + JSON-LD in `seo.py`; sitemap/robots in `routers/seo.py` for indexable public pages.
- **Docs**: `docs_api.py` endpoint entry and/or `routers/docs/pages.py` `DOCS_PAGES` prose.

### Always-update trio (the "never forget" rule)

After ANY behavioural change, re-check all three before declaring done:

- **`README.md`** - product-facing; update for new routes, env/config, dependencies, or user-visible features. Professional, no dev process, no emoticons, no em-dash.
- **`AGENTS.md`** - this file; add a domain section for the new mechanic/helper/pitfall and note the cross-layer wiring.
- **`CLAUDE.md`** - only when a new architectural rule/convention/workflow step is introduced (not a log of every feature).

### Worked example: followers/following listing (one feature, every layer)

| Layer | File(s) | What was added |
|-------|---------|----------------|
| Data helpers | `database.py` | `get_follow_counts`, `get_follow_list` (paginated, newest-first), `get_following_among` (single IN-clause, no N+1) |
| Output schema | `schemas.py` | `ProfileOut` gained `people`, `follow_pagination`, `followers_count`, `following_count` |
| Route (HTML+JSON) | `routers/profile/ package` | `?tab=followers`/`?tab=following` on the profile page; public JSON `GET /profile/{username}/followers` and `/following` - declared **before** `/{username}` |
| View | `profile.html`, `profile.css` | Two tabs with counts, paginated people rows with follow/unfollow control, empty states |
| Agent tool | `services/devii/actions/catalog.py` | `list_followers` / `list_following` (`requires_auth=False`) |
| API docs | `docs_api.py` | Two `endpoint()` entries in the `profiles` group + `followers`/`following` added to the tab enum |
| Docs trio | `README.md`, `AGENTS.md` | Social-graph listings (product) + "Follow graph listings" section (mechanics) |

Nine files for one conceptual feature. That count is the norm, not the exception - if a change touches only one file, confirm none of the four faces or the trio were missed.

## Feature Workflow (for automated agents)

### Step 1: Understand

- Read the router file for the feature area (`routers/{area}.py`)
- Read the template (`templates/{area}.html`)
- Read the existing tests (`tests/{unit,api,e2e}/<path>.py` - api/e2e by endpoint, unit by source module)
- Identify what data flows through: form fields → router → template → response
- Map which of the four faces (HTML, JSON, Devii tool, API docs) the change touches - see "Anatomy of a feature" above

## Notification System

Notifications are created server-side in the route handlers and stored in the `notifications` table. Unread counts are cached per-process in `_unread_cache` (notifications) and `_messages_cache` (messages), both 10s TTL in `templating.py`. Mutating routes call `clear_unread_cache(uid)` / `clear_messages_cache(uid)` to invalidate.

### Live counter badges

The top-nav bell and Messages link carry `data-counter="notifications"` / `data-counter="messages"` with a child `<span data-counter-badge hidden>`. `CounterManager.js` polls `GET /notifications/counts` (returns `{"notifications", "messages"}`, zeros for guests) every 30s (via the shared `Poller`, `pauseHidden`) and on tab focus, then sets each badge's text and toggles its `hidden` attribute. Server-rendered counts seed the initial state, so the badges are correct on first paint and self-heal without a reload. Absolute badges use `.nav-badge`; inline badges (next to text links) add `.nav-badge-inline`.

### Notification types and trigger points

| Type | Trigger | Location | Condition |
|------|---------|----------|-----------|
| `comment` | Top-level comment on post | `comments.py` | `post["user_uid"] != user["uid"]` |
| `reply` | Reply to a comment | `comments.py` | `parent["user_uid"] != user["uid"]` |
| `vote` | Upvote on post or comment | `votes.py` | `value == 1` AND voter != owner |
| `follow` | Follow another user | `follow.py` | Always (self-follow blocked upstream) |
| `message` | Send a message | `messages.py` | Always (different user) |
| `mention` | `@username` in content | `utils.py` `create_mention_notifications` | Mentioned user != actor |
| `level` | Reach a new level | `utils.py` `award_xp` | `new_level > current_level` |
| `badge` | Earn any badge | `utils.py` `notify_badge` | First grant only (`award_badge` returns `True`) |
| `issue` | Issue filed/replied/status | `issues.py`, `services/gitea`, `services/jobs/issue_create_service.py` | Reporter or admins |

`level`/`badge` notifications have `related_uid == user_uid` (self), so the notifications template renders them with the recipient's own avatar; the template is type-agnostic (driven by `message`/`actor`), so no per-type handling is needed.

### Time-grouped display

Notifications are grouped by time period in `notifications.py` `_group_label()`:
- Same day → **Today**
- Previous day → **Yesterday**
- Within 7 days → **This week**
- Older → **Older**

The template uses `notification_groups` (list of `{label, entries}` dicts). Jinja2 note: avoid `.items` as a dict key - it clashes with Python's `dict.items()` method.

### Vote notification messages

```python
f"{user['username']} ++'d your post"
f"{user['username']} ++'d your comment"
```

### Notification preferences (per-user, per-channel)

Every notification type is independently toggleable on two channels - **in-app** and **push** - per user, with admin-set platform defaults. This is enforced at the single funnel, so there is no per-call-site work: a new `create_notification(user_uid, type, ...)` call is automatically subject to preferences.

- **Canonical types** = `database.NOTIFICATION_TYPES` (the nine listed above; `key`/`label`/`description`). This list drives the profile tab and the admin page; it is the single source of truth. The `test` type is intentionally absent, so it always delivers (the resolver defaults unknown types to on).
- **Resolution** (`database.notification_enabled(user_uid, type, channel)`): a live `notification_preferences` row for `(user_uid, type)` wins; else the global default `get_int_setting("notif_default_{type}_{channel}", 1)`; else on. So a user who never touched a setting inherits the admin default; a user who customized keeps their choice. Channels are `in_app` / `push` (`NOTIFICATION_CHANNELS`).
- **Storage**: table `notification_preferences` (`uid`, `user_uid`, `notification_type`, `in_app_enabled`, `push_enabled`, timestamps, soft-delete columns), in `SOFT_DELETE_TABLES`, columns+indexes ensured in `init_db`. Cached under the `"notif_prefs"` cache-version name (`sync_local_cache`/`bump_cache_version`), keyed by `user_uid`. Helpers: `get_notification_prefs(user_uid)` (full resolved list for the UI/JSON), `set_notification_pref(user_uid, type, channel, enabled)` (upsert one channel, leaving the other at its resolved value; revives a soft-deleted row), `reset_notification_prefs(user_uid)` (soft-delete the user's overrides -> fall back to defaults), `get_notification_default`/`set_notification_default` (the `notif_default_*` `site_settings` keys; defaults are NOT seeded so an untouched type reads `1`).
- **Enforcement** is in `utils.create_notification`: the in-app `notifications.insert` + `clear_unread_cache` run only when `notification_enabled(..., "in_app")`, and `_schedule_push` runs only when `notification_enabled(..., "push")`. The `notification.create` audit row is unconditional and records which channels fired (`metadata.in_app`/`push`). **Because the DM path in `messages.py` now routes through `create_notification` (type `message`)** instead of inserting directly, direct messages are gated like everything else and gained a (default-on) push channel. **`create_notification` is a thin wrapper that defers the whole body** (preference reads + insert + push + audit) to the **background task queue** via `background.submit(_deliver_notification, ...)`, so it never blocks the request (inline under tests); call it directly, never wrap it in `background.submit`. See AGENTS.md -> "Background task queue".
- **Live toasts ride the `in_app` channel** (no new channel). `services/notification_relay.py` `NotificationRelayService` is a lock-owner `BaseService` (registered in `main.py`, default-enabled, 1s interval) modeled on `MessageRelay`: it primes a watermark to `MAX(notifications.id)` on first tick, then each tick selects `notifications WHERE id > watermark` and publishes `{uid, type, message, target_url}` to the per-recipient pub/sub topic `user.{user_uid}.notifications`. It runs **only on the service lock owner**, which is exactly where every pub/sub WS subscriber converges (non-owner `/pubsub/ws` closes `4013`), so the in-process `services.pubsub.publish` reaches the recipient's browser. Because the `notifications` row exists only when the `in_app` channel is enabled (the insert above), a live toast fires **exactly** for the notifications a user has enabled - the relay needs no preference lookup of its own. The watermark always advances (boot-priming + every-tick) so a newly-connected subscriber never gets a backlog flood. Frontend: `base.html` exposes the viewer uid as `<body data-user-uid>`, `static/js/LiveNotifications.js` (`app.liveNotifications`, constructed with `this.pubsub`/`this.toast`) subscribes to that topic and calls `app.toast.show(message, {type:"info", ms, url})`. **The toast click reproduces a bell-item click exactly:** it targets `GET /notifications/open/{uid}` (which marks the row read and redirects to its `target_url`), not the bare `target_url` - the relay publishes `uid` for this. `AppToast` gained a generic click action via `_action(options)`: `options.onClick` (a function) or `options.url` (navigate); either adds the `.dp-toast-link` pointer cursor, so any caller can attach a click action. DMs toast too (type `message` is not excluded). Reuse this relay-on-the-lock-owner pattern for any future "live mirror of a DB-persisted, per-user event."
- **Unread-count badges ride the same relay.** The same `NotificationRelayService` tick, after publishing the toast rows, also publishes the recipient's fresh unread counts `{notifications, messages}` to `user.{uid}.counts` (computed with a direct DB query on the lock owner, never the per-worker `_unread_cache`, which could be stale there). Because a new DM also inserts a `message`-type notification row through `persist_message -> create_notification`, this one publish point covers both new notifications and new messages, so the header badge bumps instantly. `static/js/CounterManager.js` (`app.counters`, constructed with `this.pubsub`) subscribes to `user.{uid}.counts` and applies the pushed counts; its HTTP poll of `/notifications/counts` stays as a 60s reconciliation fallback (covers decrements on read and any missed frame). Read events still clear the per-worker cache as before.

## Live view relay (`devplacepy/services/live_view_relay.py`)

`LiveViewRelayService` is the second pub/sub bridge (lock-owner `BaseService`, default-enabled, 1s interval, registered in `main.py` after `NotificationRelayService`). It replaces the per-client HTTP polling on the admin live views with server push, **computing a snapshot only for topics that currently have subscribers**. Each tick it reads `pubsub.topics()` (the hub's live subscription set, which converges on the lock owner), matches each concrete subscribed topic against the `VIEWS` registry (a list of `(compiled_regex, compute_callable, min_interval_seconds)`), throttles per topic via a `time.monotonic()` map, computes the payload, and publishes it on the same topic. The compute callables (async, lazy-importing to avoid cycles, defensive try/except -> skip) reproduce the **exact payload shape the matching HTTP endpoint already returns**, so the frontend render code is unchanged - only the data-arrival path is added. Registry:

| Topic | Cadence | Payload (same as endpoint) |
|-------|---------|----------------------------|
| `container.list` | 4s | `{instances}` (admin all-instances, `_decorate`) |
| `project.{slug}.containers` | 3s | `{instances}` (per-project, slug resolved via `resolve_by_slug`) |
| `container.{uid}.detail` | 4s | `{instance, events, schedules, stats, runtime}` |
| `container.{uid}.logs` | 3s | `{logs}` (backend log tail 400) |
| `fleet.bots` | 2s | bot frames payload (`routers/admin/bots._frames_payload`) |
| `admin.services` | 5s | `{services}` (`service_manager.describe_all()`) |
| `admin.services.{name}` | 5s | `{service}` |
| `admin.ai-usage.{hours}` | 15s | `build_analytics(hours)` (hours parsed from the topic) |

All these topics are admin-only by pub/sub policy (non-`public`, non-`user.{uid}` -> `privileged` required), matching the admin-only pages. Frontend monitors (`ContainerInstance`, `ContainerList`, `ContainerManager`, `BotMonitor`, `ServiceMonitor`, `AiUsageMonitor`) each `window.app.pubsub.subscribe(topic, render)` in their init and keep a **lengthened HTTP poll (15-20s) as initial-load + fallback** - the relay drives liveness at the cadence above. `AiUsageMonitor` re-subscribes (unsubscribe old, subscribe new) when the window-hours selector changes, since hours is in the topic. To add a new admin live view: add one `(regex, compute, interval)` row to `VIEWS` whose compute returns the endpoint payload, and have the frontend subscribe to the topic while keeping a fallback poll. Reuse this subscriber-gated snapshot pattern for any future "several admins watch a periodically-recomputed server snapshot" surface; keep durable/ordered/stateful/raw-I/O channels (messages, Devii, SEO/DeepSearch progress, container PTY) off it. See `pubreport.md` for the full migration analysis.

- **UI** is the profile **Notifications** tab (`/profile/{username}?tab=notifications`, owner-or-admin, reusing `resolve_customization_target` for authz): `templates/profile.html` renders the type x channel checkbox grid from `notification_prefs`, `static/js/NotificationPrefs.js` (`app.notificationPrefs`) optimistically `Http.send`s each change to `POST /profile/{username}/notifications` (`NotificationPrefForm{notification_type, channel, value}`) and the reset button to `POST /profile/{username}/notifications/reset`. The admin global defaults page is `/admin/notifications` (`templates/admin_notifications.html`, `static/js/AdminNotificationDefaults.js`, sidebar link in `admin_base.html`) -> `POST /admin/notifications` (`NotificationDefaultForm`). **Devii** (`services/devii/notification/`, `handler="notification"`, `requires_auth=True`): `notification_list`/`notification_set`/`notification_reset`, owner-scoped via `NotificationController(owner_kind, owner_id)` built in `Dispatcher.__init__` (the same way as `CustomizationController`); `notification_reset` is in `CONFIRM_REQUIRED` (it clears all overrides) but `notification_set` is not (one reversible, unambiguous toggle). Both mutations emit `devii.notification.*` audit events via the dispatcher's `_audit_mechanic`. **Docs:** the public prose page is `templates/docs/notification-settings.html` (slug `notification-settings` in `routers/docs/pages.py` `DOCS_PAGES`, `SECTION_GENERAL`; the slug is deliberately not `notifications`, which is already the API-reference group). Reuse this exact pattern (canonical-list + resolver + cached override table + single-funnel enforcement) for any future per-user per-channel preference.

## Gamification (XP, levels, badges, leaderboard)

The progression engine lives in `utils.py` and is wired into the existing content-creation, vote, and follow hooks. Do NOT scatter XP/badge logic - go through these helpers.

### XP and levels (`utils.py`)

- `award_xp(user_uid, amount)` - adds XP (clamped to `>= 0`), recomputes and stores `level`, invalidates the per-process user cache (`clear_user_cache`), and on level-up fires a `level` notification plus any level-milestone badge. Returns `{"xp", "level", "leveled_up"}`.
- `level_for_xp(xp)` → `1 + max(0, xp) // LEVEL_XP` (`LEVEL_XP = 100`). `level` is stored on the user (not derived in the template) so `profile.html`'s `xp % 100` progress bar keeps working.
- XP amounts are constants in `utils.py`: `XP_POST=10`, `XP_COMMENT=2`, `XP_PROJECT=15`, `XP_GIST=5`, `XP_UPVOTE=5`, `XP_FOLLOW=5`. Awards for received upvotes/followers go to the content owner / followed user and live **inside** the existing `owner_uid != user["uid"]` guards (`votes.py`, `follow.py`).

### Badges (`utils.py`)

- `award_badge(user_uid, name)` is idempotent (returns `True` only on first grant) - reuse it; never insert into `badges` directly.
- `check_milestone_badges(user_uid)` recomputes **source-derived** count thresholds (counts of existing tables, so always accurate, no tracking needed) and awards + notifies any newly crossed badge. Call it after content creation and after an upvote/follow that changes the recipient's totals. The thresholds are the data-driven `_COUNT_MILESTONES` list `(badge, metric_key, threshold)` over metric keys `posts`/`comments`/`projects`/`gists`/`stars`/`followers`/`following`/`streak`, resolved lazily by `_milestone_metrics(user_uid)` (a memoized resolver that only queries a metric when an unheld badge needs it - badges already held are skipped before any query). To add a source-derived milestone, add a row to `_COUNT_MILESTONES` and the badge to `BADGE_CATALOG`. Streak badges additionally emit a `reward.streak.milestone` audit event.
- `award_rewards(user_uid, amount, first_badge=None)` is the single entry point for the create/upvote/follow reward sequence: it awards the optional first-time badge, calls `award_xp`, then `check_milestone_badges`. Use it instead of calling the three separately (posts/projects/gists/comments/follow/votes all go through it) so milestone checks are never skipped. **`award_rewards` defers its whole body to the background task queue** (`background.submit(_apply_rewards, ...)`), so EVERY XP award on the platform leaves the request path - the badge/XP/milestone writes (and the reward-triggered level/badge notifications nested inside `award_xp`/`check_milestone_badges`) all run on the consumer. In tests the queue runs inline, so XP is awarded synchronously and assertions hold. Never re-wrap an `award_rewards` call in `background.submit` - it self-defers. See AGENTS.md -> "Background task queue".
- Badge catalog + display metadata is `BADGE_CATALOG` in `utils.py` (55 badges), exposed to templates as the `badge_info(name)` global. Each entry has `icon`, `description`, and a `group` (one of `BADGE_GROUPS`: First steps, Explorer, Engagement, Content, Community, Reputation, Dedication, Levels, Milestones). Unknown names fall back to a default icon and the name as description. `LEVEL_BADGES` maps levels 5/10/25/50/100 to Level badges, awarded inside `award_xp` on level-up.

#### Activity tracking and feature-first-use badges

- For achievements that are **not** derivable from an existing table (first time a user uses a feature, "read N docs", number of DeepSearches, etc.), there is a generic per-user activity counter. `database.record_activity(user_uid, action) -> int` upserts a `user_activity` counter (unique `(user_uid, action)`, raw SQL `ON CONFLICT` inside `with db:`) and returns the new count; `database.record_unique_activity(user_uid, action, target) -> int | None` records a distinct `(user_uid, action, target)` in `user_activity_seen` (returns the new distinct count, or `None` if the target was already seen) for "N distinct things" achievements. Both tables + their unique indexes are ensured in `init_db()`.
- `utils.ACHIEVEMENTS` maps `action -> [(threshold, badge_name), ...]` (threshold 1 = first use). `utils.UNIQUE_ACTIONS` is the set of actions counted distinctly (currently `docs.read`). `utils.track_action(user_uid, action, target=None)` is the single public hook: it returns immediately if the action is unknown, else `background.submit`s `_apply_achievements`, which records the activity (counter or unique), then awards + notifies every crossed threshold via `award_badge`/`notify_badge`. It is fully deferred (off the request path) and idempotent, so call it inline at the success point of any feature.
- Wired actions and their hook sites: `vote`/`bookmark` (`content.py`), `reaction` (`routers/reactions.py`), `follow` (`routers/follow.py`, follower side), `message` (`services/messaging/persist.py`), `docs.read` (`routers/docs/views.py`, unique per slug, authenticated only), `fork`/`zip` (`routers/projects/index.py`), `zip`/`project_file` (`routers/projects/files.py`), `seo` (`routers/tools/seo.py`), `deepsearch` (`routers/tools/deepsearch.py`), `container` (`services/containers/api.create_instance`, user actor), `upload` (`routers/uploads.py`), `issue` (`routers/issues/create.py`), `poll` (`routers/polls.py`), `profile` (`routers/profile/index.py`, when a field is filled), `devii` (`services/devii/session.py` `_run_turn`, user owner on the non-docs channel only). To add a feature badge: add the badge to `BADGE_CATALOG`, an `ACHIEVEMENTS` entry, and a `track_action(...)` call at the feature's success point.
- **Profile showcase:** `utils.build_achievements(held_names)` returns the full catalog grouped by `BADGE_GROUPS` with an `earned` flag per badge and `earned`/`total` per group. The profile route passes it as `achievements` (+ `badge_earned`/`badge_total`) and `profile.html` renders a collapsible **Achievements** showcase (`.profile-achievements`, styles in `profile.css`) above the tabs, so every badge - earned and locked - is discoverable.

### Rank and leaderboard (`database.py`)

- `_ranked_authors()` builds (and 60s-caches in `_authors_cache`) the full list of authors with positive total stars, ordered desc, enriched via `get_users_by_uids`. `get_top_authors(limit)`, `get_leaderboard(limit, offset)` (adds a 1-based `rank`), and `get_user_rank(user_uid)` all slice/scan this one list - keep them consistent.
- `update_target_stars()` clears `_authors_cache` so a vote is reflected on the leaderboard and profile rank immediately (also keeps integration tests deterministic).
- The leaderboard page (`/leaderboard`, `routers/leaderboard.py`) shows the **top 50 only - no pagination** (`TOP_LIMIT = 50`). It is public (`get_current_user`), highlights the current user's row (`.leaderboard-row-self`), and is listed in `sitemap.xml`.

### Backfill

`_backfill_gamification()` runs at the end of `init_db()`. It computes XP from prior activity (same amounts as above) for users still at the default `xp=0`, sets `level`, then runs `check_milestone_badges` per user. Guarded on `xp=0` so it is idempotent across restarts.

## Inline Comment on Feed Cards

Every post card on the feed now has an inline comment form (`.feed-comment-form`) beneath the post actions. It posts to `/comments/create` like the detail page comment form. Tests must scope Post button clicks to `#create-post-modal button.btn-primary:has-text('Post')` to avoid matching the inline comment's Post button.

## Comment Editing

A comment's owner (only the owner, never an admin) sees an inline "Edit" button (`data-action='edit'`) in `_comment.html`. `CommentManager.toggleEditForm` swaps the `.comment-text` for a textarea seeded from its `data-raw` attribute (the raw markdown, since `contentRenderer.applyTo` overwrites `textContent` on first render), posts via `Http.send` to `POST /comments/edit/{comment_uid}`, then re-renders the new body in place with `contentRenderer.applyTo`. The route (`content.edit_comment_record`) is `is_owner`-only, writes `content` + `updated_at`, records the `comment.edit` audit event, and branches on `wants_json`: JSON clients get `CommentEditOut{uid, content, url, updated_at}`, the no-JS form falls back to a redirect to the comment anchor. Edits are NOT soft-delete related (the body is overwritten in place). Devii tool: `edit_comment` (owner-only, no confirm). Scope test Edit clicks to `.comment-action-btn:has-text('Edit')`.

## Post Editing

Post owners see an "Edit" button on the post detail page that opens `#edit-post-modal`. The edit form allows changing title, content, and topic. The POST route is `/posts/edit/{post_uid}` with ownership check. The edit modal's textarea has `id="edit-content"` - tests must scope to `.comment-form textarea[name='content']` for comment operations.

## Project Detail Page

Each project card links to `/projects/{project_uid}` showing full project details, author info, platforms, star count, delete-for-owner, and (for the owner) Private/Read-only toggle buttons plus badges (see **Project visibility and read-only**). The route is `GET /projects/{project_uid}` in `routers/projects/index.py` and 404s when the viewer cannot see a private project. The sitemap generator links to this URL (not the old `?user_uid=` query param). The detail page also links to the project filesystem at `/projects/{slug}/files`.

**Action row overflow.** The detail page has more actions than fit one line, so `project_detail.html` keeps the engagement actions inline (Files, Share, star vote, bookmark, reactions) and collapses the rest behind a single **More** button (`.project-actions-more`) that opens the shared `app.contextMenu`. The secondary actions (Containers, Download zip, Fork, the owner Edit/Private/Read-only/Delete controls) live as real elements inside a hidden `.project-actions-overflow` container, each tagged `data-menu-action` plus `data-menu-icon`/`data-menu-label`. `static/js/ProjectActionsMenu.js` builds the menu from those elements and each item's `onSelect` simply `.click()`s the real element, so all existing wiring is reused unchanged - `app.zipDownloader` (`data-zip-download`), `app.projectForker` (`data-fork-project`), `data-share`, the `data-modal` Edit trigger, and the delegated `data-confirm`/`data-confirm-danger` dialog on the owner forms. The open handler must `stopPropagation()` because `app.contextMenu`'s document-level close listener would otherwise dismiss it on the same click (every other caller opens it from a right-click `attach`, not a left-click). Reuse this pattern - a `More` trigger over `[data-menu-action]` real elements - for any future action row that overflows; do not duplicate controller logic into menu callbacks.

**Owner editing.** Mirrors post editing exactly: an owner-only **Edit** menu item (`data-modal="edit-project-modal"`) opens the `modal()` macro's `edit-project-modal`, a plain `POST` form to `/projects/edit/{slug}` (route `edit_project` in `routers/projects/index.py`, body `ProjectEditForm`, owner-gated through the shared `content.edit_content_item` which returns 403 JSON / redirect for non-owners and stamps `updated_at`). The modal is the create modal pre-filled from the `project` row (title, description, type/status radios pre-checked, dates via `format_date()` back to DD/MM/YYYY). `is_private`/`read_only` are NOT edited here - they stay on their dedicated toggles. The platforms tag widget reuses the create modal's `platforms-input`/`platforms`/`platforms-tags` ids; `ProfileEditor.initPlatformTags` now **seeds existing tags** from the hidden `#platforms` value on load, so both the empty create form and the pre-filled edit form work from the same code. Devii tool `edit_project`; documented in `docs_api.py` (`projects-edit`).

## Project Filesystem

Each project carries a virtual filesystem so it can hold a whole software project. Logic lives in `devplacepy/project_files.py` (mirrors `attachments.py`); routes in `routers/projects/files.py` (registered at prefix `/projects`, after `projects.router`). It is **public read, owner write** - reads use `get_current_user`, mutations use `require_user` + `is_owner(project, user)`. Two project flags layer on top (see **Project visibility and read-only**): a private project's reads are restricted to owner/admin, and a read-only project refuses every mutation at the data layer.

- **Model.** One `project_files` table, each row a node keyed by a normalized POSIX `path` unique per project: `uid, project_uid, user_uid, path, name, parent_path, type` (`file`/`dir`), `content` (text in DB), `is_binary`, `stored_name`, `directory`, `mime_type`, `size`, timestamps. Indexes `(project_uid, path)` and `(project_uid, parent_path)` in `init_db`.
- **Text vs binary.** Text files store `content` in the DB (editable inline). Binary uploads write bytes to `config.PROJECT_FILES_DIR/<shard>/<stored_name>` (= `data/uploads/project_files/...`, reusing `attachments._directory_for`/`_detect_mime`) and are served via the existing `/static/uploads` mount. The `<shard>` is the canonical two-level `xx/yy` tree taken from the **random tail** of the uuid7 (`{tail[-2:]}/{tail[-4:-2]}`), never its time-ordered head - see CLAUDE.md "Blob sharding" for why the tail is mandatory. `store_upload` decodes UTF-8 texty files into editable DB content; everything else is binary.
- **Path is the security control.** `normalize_path` rejects `..`, null bytes, control chars, and empty/over-long paths; a leading `/` is normalized to relative. The logical `path` is never used as a real FS path (blobs are uid-hashed), so traversal is structurally impossible. `write`/`upload`/`mkdir` create parent dirs recursively (`ensure_dirs`). Caps: `MAX_TEXT_CHARS` 400000, `MAX_FILES_PER_PROJECT` 5000.
- **Routes (all POST mutations, JSON via `respond`/`action_result`; file path travels in query/body `path`, not a path param):** `GET .../files` (page or `ProjectFilesOut`), `GET .../files/raw?path=`, `POST .../files/{write,upload,mkdir,move,delete}`. `move` rewrites a dir's descendants by path prefix; `delete` is recursive and unlinks blobs.
- **Line-range editing (large text files).** `GET .../files/lines?path=&start=&end=` returns `{path, start, end, total_lines, lines, content}`; `POST .../files/{replace-lines,insert-lines,delete-lines,append}` mutate a text file surgically without resending the whole thing. Helpers in `project_files.py` (`read_lines`, `replace_lines`, `insert_lines`, `delete_lines`, `append_lines`) work on a `(lines, trailing_newline)` model via `_split_lines`/`_join_lines`, are 1-indexed inclusive, enforce `MAX_TEXT_CHARS`, and reject binary/dir/missing paths. These are the preferred way to edit existing files; `write` replaces the whole file. They never create parents (the file must exist).
- **Cascade.** `content.py` `delete_content_item` calls `delete_all_project_files(uid)` when `target_type == "project"`.
- **Frontend.** `templates/project_files.html` (IDE layout: tree + CodeMirror editor / media preview, same CodeMirror vendor as gists), `static/js/ProjectFiles.js` (tree from the flat list, open/save/new/upload/rename/delete over JSON), `static/css/project_files.css`. The CodeMirror mode map is shared via `static/js/codemirrorModes.js` (used by `GistEditor.js` too).
- **Devii + API.** `http` catalog actions cover the full filesystem: `project_list_files`/`read_file`/`read_lines`/`write_file`/`replace_lines`/`insert_lines`/`delete_lines`/`append_file`/`upload_file`/`make_dir`/`move_file`/`delete_file`, plus the `project-files` `docs_api.py` group. Because `PlatformClient` sends `Accept: application/json`, the same routes serve the agent and the API.
- **Overwrite-protection guard (agent only).** `Dispatcher._run_http` blocks `project_write_file` for a `(slug, path)` only when the file **already exists** and the agent has not read it this session - it must call `project_read_file` first. Existence is probed live via `_file_exists` (`GET /projects/{slug}/files/raw`, 200 = exists, 404 = new), so creating a brand-new file is never blocked. Reads of `project_read_file`/`project_read_lines` (and a successful write) record the path in the per-session `Dispatcher._read_files` set; the key is `normalize_path`d so read and write paths match. This enforces "look before you overwrite" and steers the agent to the surgical line tools for existing files; it does not affect the human UI or the public HTTP API (the guard lives in the Devii dispatcher, not the route). The system prompt's `WRITING AND EDITING FILES` rule mirrors this.

## Project visibility and read-only

Two owner-controlled flags on the `projects` row, both integer `0`/`1` (default `0`, set on the create insert so the columns always exist):

- **`is_private`.** When `1`, the project is visible only to its owner and to administrators. `content.can_view_project(project, user)` is the single predicate (`not is_private` OR `is_owner` OR `is_admin`). It gates every read surface: `project_detail` (404 otherwise), the project filesystem GET routes via `_load_viewable_project` (`/files`, `/files/raw`, `/files/lines`, `/files/zip`), `zip_project`, the `/projects` listing (`get_projects_list` adds a SQLAlchemy clause excluding others' private rows unless the viewer is admin), the profile projects list (`routers/profile/ package` filters when the viewer is not owner/admin), and the sitemap (`seo._build_sitemap` skips `is_private`). Because Devii's `project_list_files`/`project_read_file`/`project_read_lines` are `requires_auth=False` GETs that hit those same routes, a guest/non-owner Devii session gets a 404 for free; a signed-in owner's Devii authenticates with the owner's key and sees it.
- **`read_only`.** When `1`, the project's files are **fully immutable** - every mutation is refused for everyone (owner included). Enforcement is a single data-layer guard `project_files._guard_writable(project_uid)` (raises `ProjectFileError("project is read-only; ...")`) called at the top of every write entrypoint: `write_text_file`, `store_upload`, `make_dir`, `move_node`, `delete_node`, `replace_lines`, `insert_lines`, `delete_lines`, `append_lines`, and `import_from_dir`. Placing it in `project_files.py` (not the route) covers ALL callers in one place: the HTTP routes, Devii (which goes through the routes), and container workspace **sync** (`services/containers/api.py sync_workspace` -> `import_from_dir`). `delete_all_project_files` is intentionally NOT guarded so deleting the whole project still cascades. `export_to_dir`/reads are never guarded. The owner unsets read-only to edit again.

**Toggle routes** (owner-only, `routers/projects/index.py`): `POST /projects/{slug}/private` and `POST /projects/{slug}/readonly`, each taking `ProjectFlagForm{value: bool}` and routed through `_set_project_flag`. The create form carries an `is_private` checkbox (`ProjectForm.is_private`); `project_detail.html` shows Private/Read-only badges and owner toggle buttons (**both** the private and read-only buttons carry `data-confirm` so `ModalManager.initConfirmations` gates them through `app.dialog`), and `project_files.html` hides the editing toolbar + shows a banner when read-only. Schemas: `is_private`/`read_only` on `ProjectOut` and `ProjectDetailOut`, `read_only`/`is_private` on `ProjectFilesOut`.

**Devii.** Actions `project_set_private` and `project_set_readonly` (catalog). **Both** are gated by an explicit-confirmation requirement: `dispatcher.confirmation_error(name, arguments)` (driven by the `CONFIRM_REQUIRED` set, which lists both action names) is checked in `Dispatcher.dispatch` BEFORE any HTTP call and returns a `ToolInputError` telling the agent to obtain user confirmation unless `confirm` is truthy (the message for `project_set_private` adapts to the `value` argument: making private vs. making public). Each action declares `confirm` as a required body param, and the `PROJECT PRIVACY AND READ-ONLY` system-prompt rule tells the agent to ask first and only then pass `confirm=true`. This mirrors the overwrite-protection guard pattern: a Devii-only gate that the human UI and HTTP API do not see.

### Deletion confirmation gate

`confirmation_error` also blocks every Devii deletion until `confirm` is truthy. Unconditional entries in `CONFIRM_REQUIRED`: every content delete - `delete_post`, `delete_comment`, `delete_gist`, `delete_project`, `project_delete_file`, `delete_media`, `delete_attachment`, `admin_delete_news` - plus `project_set_readonly` and the customization mutations; a generic fallback message covers any name in the set that lacks a bespoke message. **Load-bearing invariant: every gated tool MUST declare a `confirm` body param** (the `confirm()` helper in `catalog.py`, or `arg("confirm", ..., kind="boolean")` for container actions). Tool schemas are `additionalProperties: false`, so the model can only send arguments that are declared properties - a gated tool with NO `confirm` param can never be confirmed, so `confirmation_error` refuses every call and the agent loops forever asking for a confirmation it has no way to express. This was a real production issue: an admin asked Devii to delete a user's 27 posts, confirmed repeatedly, and all 27 `delete_post` calls were rejected because `delete_post` (and `delete_project`/`delete_media`/`project_delete_file`/`container_*`) never exposed `confirm`. The param is harmlessly forwarded to the HTTP endpoint (routes do not declare it, FastAPI drops undeclared form fields) and ignored by the LOCAL container/customization controllers. Conditional entries (gated by inspecting arguments, not membership): `container_instance_action` only when `action="delete"` (start/stop/restart/pause/resume/sync are unaffected), and `container_exec` only when its `command` matches `dispatcher.DESTRUCTIVE_COMMAND` - a token-aware regex for `rm`/`rmdir`/`unlink`/`shred`/`truncate`/`dd`/`mkfs*`/`wipefs` (with or without a leading `sudo`), `find ... -delete`, redirection over a file (`> /...`), and SQL `drop table|database` / `delete from`. Benign commands (`pip install`, `ls`, `grep -rm`) are not matched. The error message echoes the exact path/command so the agent shows the user what will be removed; after a clear yes it retries with `confirm=true`. The `DELETING IS ALWAYS CONFIRMED` system-prompt section in `agent.py` mirrors the rule. This exists because an unconfirmed `container_exec rm -f` once deleted a live project database mid-"test the site"; the gate is the data-loss backstop. The regex is a heuristic (it cannot catch `python -c "os.remove(...)"` and similar), so it is defense-in-depth, not a sandbox - the system-prompt rule is the primary control.

## Issue Reports (Gitea integration)

The issue tracker is a full integration with a Gitea repository (default `retoor/devplacepy` on `retoor.molodetz.nl`); there is **no local issue store**, the listing and detail views read issues straight from Gitea with live status. All connection settings live in `site_settings` and are admin-editable on the `IssueTrackerService` config at `/admin/services`.

**Package `services/gitea/`:**
- `config.py` - the `CONFIG_FIELDS` (Gitea base URL/owner/repo/token, AI enhance toggle/model/key) plus `gitea_config()` -> frozen `GiteaConfig` (with `api_base`/`repo_base`/`is_configured`) and `is_configured()`. The token is one shared bot account; per-user attribution is done by storing names in DevPlace, not by per-user Gitea accounts.
- `client.py` - async `GiteaClient` over the Gitea REST API (`Authorization: token`): `create_issue`, `list_issues(state,page,limit)` returning `(issues, total)` from `X-Total-Count` and filtering out pull requests, `get_issue`, `list_comments`, `create_comment`, `set_state`. Raises `GiteaError(message, status)` on any non-2xx or transport error.
- `fake.py` - in-memory `FakeGiteaClient` mirroring the same async interface (plus `add_external_comment` to simulate a developer reply); the test backend.
- `runtime.py` - `get_client()` / `set_client()` swap (tests inject the fake). `get_client()` builds a fresh `GiteaClient` from current config unless overridden.
- `enhance.py` - `enhance_ticket(title, description, config)` calls the **internal AI gateway** (`INTERNAL_GATEWAY_URL`, key defaults to the gateway internal key) to rewrite a raw report into a consistent ticket: a level-2-heading markdown body (`Summary` / `Steps to Reproduce` / `Expected Behaviour` / `Actual Behaviour` / `Environment`) and a tightened title. It is fail-soft: any error or unparseable response falls back to a deterministic template built from the original text (`EnhancedTicket.enhanced=False`). This is the AI attachment point - the gateway, like every other AI consumer.
- `store.py` - the **thin local mapping** (the only DB state): `issue_tickets` (gitea_number -> author_uid + original text + cached `last_status`/`last_comment_count` for update detection) and `issue_comment_authors` (gitea_comment_id -> author_uid, for display attribution and to tell DevPlace-authored comments from developer replies). Helpers: `record_ticket`, `get_ticket`, `author_uid_for_issue`, `author_map`, `record_comment_author`, `comment_author_map`, `local_comment_ids`, `tracked_tickets`, `update_ticket_cache`. Both tables are in `SOFT_DELETE_TABLES`; `init_db` ensures their full column set + indexes (so queries are safe on a fresh DB before any insert).
- `service.py` - `IssueTrackerService(BaseService)` (`default_enabled=False`, holds `CONFIG_FIELDS`). Each tick it polls every tracked ticket: when the Gitea comment count grew, it loads comments and notifies the reporter if any of the new tail comments are **not** in `local_comment_ids` (a developer reply); when the state changed it notifies the reporter (closed/reopened). It then updates the cache. This is the single source of update notifications to the author (`issue.sync.reply` / `issue.sync.status`).

**Filing is an async job** (`services/jobs/issue_create_service.py` `IssueCreateService`, kind `issue_create`): `process` loads the user, enhances the report, appends a `Reported by **user** via DevPlace` footer, creates the Gitea issue, records the local mapping, notifies the reporter (`Your issue report was filed as #N`), and audits `issue.create`. `cleanup` is a no-op (the issue is permanent; only the job row is swept). `POST /issues/create` enqueues and returns `{uid, status_url}`; the frontend `app.issueReporter` (`static/js/IssueReporter.js`) submits the create form, polls `/issues/jobs/{uid}` via `JobPoller`, and redirects to `/issues/{number}`.

**Routes (`routers/issues/ package`, prefix `/issues`):** `GET /issues` (Gitea list, `?state=open|closed|all&page=`, admin-style `build_pagination` + `_pagination.html` with a `state=` prefix; renders an empty-state notice when unconfigured/unreachable), `POST /issues/create` (enqueue), `GET /issues/jobs/{uid}` (`IssueJobOut`), `GET /issues/{number}` (`issue_detail.html`: issue body + comments rendered as markdown via `data-render`, decorated with DevPlace authors or a `dev` badge), `POST /issues/{number}/comment` (synchronous - posts the comment to Gitea attributed to the user, records the author mapping, notifies **all admins**, audits `issue.comment`), `POST /issues/{number}/status` (admin-only open/closed via Gitea PATCH, audits `issue.status`; non-admins get a `denied` audit + 403). Comments and status changes are deliberately synchronous (one fast Gitea call, immediate redirect); only the AI-heavy create is a job. Status-change notifications to the author are NOT emitted by the route - the poller detects the diff and notifies, so the admin (or external developer) closing an issue both flow through one path.

**Content rendering and mentions (frontend reuse, no new modules):** the issue body and every comment render through the platform content pipeline like posts/comments - each is a `<div class="...rendered-content" data-render>{{ body|comment.body }}</div>` (issue_detail.html), so the globally-instantiated `ContentEnhancer` (`app.content`) runs `contentRenderer.applyTo` (marked -> DOMPurify -> highlight.js -> media/autolink) over Gitea-sourced markdown client-side from `element.textContent`. Gitea content is NEVER marked `| safe` into raw HTML; Jinja autoescapes it and the DOMPurify step inside `ContentRenderer` is the XSS control, identical to user posts. The issue authoring textareas (`#issue-description` in the create modal on issues.html, and the comment textarea on issue_detail.html) carry the `data-mention` attribute, so the same `ContentEnhancer.initMentionInputs()` -> `MentionInput` flow that wires post `@`-mention autocomplete (debounced `GET /profile/search?q=` dropdown) attaches to them with no new code. There is no local issue-edit form (issues are AI-created then commented), so "edit" reuses nothing further. `mention.css` is loaded globally in `base.html`.

Schemas: `IssueItemOut`/`IssueCommentOut`/`IssueDetailOut`/`IssuesOut`/`IssueJobOut` (`schemas.py`). Forms: `IssueForm`/`IssueCommentForm`/`IssueStatusForm` (`models.py`). Devii tools: `list_issues`, `create_issue`, `view_issue`, `comment_issue`, `set_issue_status` (admin). Docs: the `issues` API group in `docs_api.py`. Footer link in `base.html` under `.site-footer`.

## Gists

A dedicated `/gists` page for sharing code snippets. Uses `gists` table (auto-created by `dataset`).

### Database columns

| Column | Type | Notes |
|--------|------|-------|
| `uid` | text | UUID |
| `user_uid` | text | FK → users.uid |
| `title` | text | Required, max 200 |
| `description` | text | Optional, max 5000, markdown (rendered by ContentRenderer) |
| `source_code` | text | Required, max 50000 |
| `language` | text | One of 27 supported languages |
| `slug` | text | `make_combined_slug(title, uid)` |
| `stars` | int | Net vote count (via `/votes/gist/{uid}`) |
| `created_at` | text | ISO datetime |

### Routes

| Method | Path | Handler | Auth |
|--------|------|---------|------|
| GET | `/gists` | `gists_page` | No |
| GET | `/gists/{slug}` | `gist_detail` | No |
| POST | `/gists/create` | `create_gist` | Yes |
| POST | `/gists/delete/{slug}` | `delete_gist` | Yes (owner) |

### Polymorphic reuse

- **Comments**: Uses `_comment_section.html` with `target_type="gist"` - same component as posts/projects
- **Voting**: Uses existing `/votes/gist/{uid}` route - updates `gists.stars`
- **Content rendering**: Description rendered via `ContentRenderer.js` (.rendered-content[data-render])
- **Profile tab**: "Gists" tab between Projects and Activity on profile pages

### CodeMirror editor

- CodeMirror 5 loaded from CDN in `gists.html` via `{% block extra_js %}`
- 22 language modes pre-loaded (Python, JS, TS, HTML, CSS, C, C++, Java, Go, Rust, SQL, Bash, YAML, Markdown, Swift, PHP, Ruby, Kotlin, Haskell, Lua, Perl, R, Dart, Scala)
- `GistEditor.js` initializes CodeMirror on `#gist-source-editor` textarea
- Language selector dropdown dynamically switches CodeMirror mode
- `Ctrl+S` shortcut saves and submits the form
- On form submit, `editor.save()` syncs CodeMirror content back to the hidden textarea

### Display

- Source code rendered in `<pre><code class="language-xxx">` block on detail page
- Syntax highlighting handled by existing `highlight.js` loaded globally in `base.html`
- Copy button uses `navigator.clipboard.writeText()`
- Cards in listing show language badge, title, truncated description, author, star count

### Sitemap

- Latest 500 gists included in sitemap, `changefreq="weekly"`, `priority="0.6"`

## Background task queue (`services/background.py`)

Generic, lightweight, **fire-and-forget** offload for non-critical side-effects so they leave the request path. The singleton `background` (`from devplacepy.services.background import background`) wraps one in-process `asyncio.Queue` drained by a single consumer task. The whole API is one call: `background.submit(fn, *args, **kwargs)` enqueues a **synchronous** callable and returns immediately (`put_nowait`). The consumer runs each callable inside its own `try/except`, so a failing task is logged and never kills the loop; FIFO order is preserved.

- **Per-worker, not lock-gated (the reason it is NOT a `JobService`/`BaseService`).** Producers run in every uvicorn worker, and process memory is not shared, so the drain must run wherever requests are handled. `main.py` `startup()` calls `await background.start()` for every worker (inside the `if not DEVPLACE_DISABLE_SERVICES` guard, **outside** the `acquire_service_lock()` branch); `shutdown()` calls `await background.stop()`. A `BaseService` supervisor only runs in the lock owner, which would strand records produced by the other worker. The `JobService` queue is also wrong here: it is DB-backed, so deferring a tiny audit insert would *add* writes instead of removing them.
- **Inline fallback (load-bearing).** When the consumer is not running - tests with `DEVPLACE_DISABLE_SERVICES=1`, unit tests, request-less bootstrap, or a full queue - `submit` runs `fn` **inline and synchronously**. This keeps audit/notification writes deterministic and immediately visible to the test suite (which asserts audit rows right after an action) while production defers them. No test changes are needed.
- **Best-effort durability (intentional).** In-memory only; `stop()` drains whatever remains synchronously so a **graceful** shutdown loses nothing, but a hard crash drops unflushed items. This matches audit/notifications already being best-effort (the recorder never raises). Do not put response-critical or money-touching work on it.
- **Capture plain data, never the `Request`.** Its lifecycle ends with the response - build the row/payload synchronously on the request thread and submit only the resulting dict/scalars.
- **Consumers today (deferred at their canonical choke points, so callers need no change):**
  - **Audit** - `services/audit/record.py` `_write` builds the row + links synchronously, generates the `uid`/`created_at` eagerly so `record()` still returns the real uid, then `background.submit(_persist, row, links)` does the two `store` inserts off-thread.
  - **XP/rewards** - `utils.award_rewards` defers its body via `background.submit(_apply_rewards, ...)`, so every XP award (posts/comments/projects/gists/follow/votes) leaves the request path.
  - **Notifications** - `utils.create_notification` (the single notification funnel) defers via `background.submit(_deliver_notification, ...)`, so every in-app insert + push schedule + audit for a notification (vote/follow/comment/mention/message/badge/level) runs on the consumer.
  - **Mention fan-out** - `utils.create_mention_notifications` defers its whole body (`_deliver_mention_notifications`): the `extract_mentions` regex + the `@`-username lookup query + the per-user loop all run off-thread (it is called on every post/gist/project/comment/message create).
  - **Issue-comment admin fan-out** - `routers/issues/comment.py` defers the `_notify_admins` loop (admin lookup + per-admin notify) via `background.submit`, after the synchronous Gitea call.
- Because the reward and notification funnels self-defer, request handlers just **call `award_rewards`/`create_notification` directly** - do NOT wrap them in `background.submit` (that double-queues). The pattern for any NEW side-effect: do the user-visible write inline, then call the self-deferring funnel (or `background.submit(...)` a one-off).
- **Deliberately NOT deferred (would break correctness):** cache invalidations (`clear_user_cache`/`clear_unread_cache`/`clear_messages_cache`/`bump_cache_version`) must run before the response so the next read is fresh (and they are microsecond version bumps); the vote/reaction count aggregation feeds the AJAX response body; and synchronous **external** calls whose result the response needs or must surface on failure (the Gitea comment/status calls; file/thumbnail writes whose returned URL must already exist) belong in an async **JobService** (durable + retryable), not this fire-and-forget queue.

## AI content correction (`services/correction.py`)

**Opt-in, default off, server-side.** When a user enables it, every piece of prose they author is automatically rewritten by the AI gateway according to a user-defined instruction. The user chooses the **apply mode** (per-user `ai_correction_sync` flag, 0 = background, 1 = sync): **background** (default) saves the content exactly as written and rewrites the stored prose fields a moment later on the background consumer; **synchronous** makes the HTTP response wait until the correction has been applied, so the redirect/next read already shows the corrected text.

**Critical (the self-deadlock trap):** the correction POSTs to the in-process gateway (`INTERNAL_GATEWAY_URL` = `localhost:{PORT}`), and the hooked content helpers are SYNCHRONOUS, called on the worker's event-loop thread. So sync mode must NEVER run the blocking gateway call on the event loop - on a single worker that loop is the only thing that can serve the gateway request it is waiting on, so a blocking inline call deadlocks the worker until the httpx timeout, then fail-softs (server hangs, no correction). The fix: sync mode runs `_run_correction` on a worker thread via `loop.run_in_executor` (the loop stays free to serve the gateway request), stashes the returned future on `request.scope[PENDING_SCOPE_KEY]`, and the `await_pending_corrections` HTTP middleware in `main.py` awaits it AFTER the handler - so the response waits for the correction while the event loop stays free. Never reintroduce a blocking inline correction on the loop thread.

- **Field registry is the single source of truth.** `CORRECTABLE_FIELDS: dict[str, tuple[str, ...]]` maps each correctable table to its prose columns: `posts` -> `(title, content)`, `projects`/`gists` -> `(title, description)`, `comments`/`messages` -> `(content,)`, `users` -> `(bio,)`. `gists.source_code`, project files, and Gitea issues are intentionally absent - code and external systems are never corrected.
- **Choke entrypoint.** `schedule_correction(user, table, uid, request=None)` is the only thing handlers call (every hook passes `request`). It is a no-op unless: a user dict is present, `table` is in the registry, `user["ai_correction_enabled"]` is truthy, and the user has a non-empty `api_key`. In **sync** mode (`user["ai_correction_sync"]`) it calls `_run_inline_awaited`, which - only when a `request.scope` and a running event loop exist - submits `_run_correction` to `loop.run_in_executor` (off the loop thread) and stashes the future on `request.scope[PENDING_SCOPE_KEY]` for the middleware to await; if there is no request/loop it returns False and falls back to background. In **background** mode (default) it `background.submit`s `_run_correction` (per-worker queue; also runs inline under `DEVPLACE_DISABLE_SERVICES=1`). Same `_run_correction` worker function in all cases.
- **The hooks (covers UI + REST + Devii + devRant in one place):** `content.create_content_item` (posts/projects/gists), `content.edit_content_item`, `content.create_comment_record`, `content.edit_comment_record`, `services/messaging/persist.persist_message` (DMs - sender is the user), `routers/profile/index.update_profile` (bio), and the two devRant direct-update edit paths (`routers/devrant/rants.edit_rant`, `routers/devrant/comments.edit_comment`). devRant create paths route through the shared content cores, so they are already hooked. Code-only and external paths (project files, Gitea) are deliberately not hooked.
- **The gateway call is fail-soft.** `correct_text(api_key, prompt, text)` is synchronous (runs on the background worker thread), POSTs to `INTERNAL_GATEWAY_URL` with `model=INTERNAL_MODEL` via `stealth.stealth_sync_client`, authenticated with the user's own `Bearer` api_key (per-user attribution). It returns the ORIGINAL text on any error, empty output, or suspiciously large output (`len > len(text) * MAX_GROWTH_FACTOR + 200`, rejecting hallucinated expansion). `_run_correction` reads the row back, corrects each registry field that has non-blank text, writes only changed fields via `table.update(updates, ["uid"])` without touching `updated_at` (auto-correction is not a user edit) or the slug (slugs are permanent), and `clear_user_cache(user_uid)` when `table == "users"` so the corrected bio re-caches.
- **Per-user usage aggregation (only on success).** `correct_text` returns `(text, usage)`; `usage` is parsed from the gateway's `X-Gateway-*` response headers (`_usage_from_headers`) whenever the upstream call returned 200 (cost was incurred, even if the corrected output was rejected), else `None` on any failure. `_usage_from_headers` captures the token and cost headers PLUS the timing headers `X-Gateway-Upstream-Latency-Ms` and `X-Gateway-Total-Latency-Ms` (as `upstream_latency_ms`/`total_latency_ms`), so each call's timing is metered. `_run_correction` accumulates the per-field `usage` into one `totals` dict and, when `totals["calls"] > 0`, makes ONE call to `database.add_correction_usage(user_uid, totals)` (a single `totals` dict, not positional args) - so a 2-field content item is a single aggregated write, and a failed/empty correction records nothing. `add_correction_usage` (and `add_modifier_usage`) delegate to the shared `database._add_usage(usage_table, user_uid, totals)`: a single atomic `INSERT ... ON CONFLICT(user_uid) DO UPDATE SET col = col + excluded.col` upsert against the `correction_usage` table (per-user running SUMS: `calls`/`prompt_tokens`/`completion_tokens`/`total_tokens`/`cost_usd`/`upstream_latency_ms`/`total_latency_ms`/`updated_at`, unique index `idx_correction_usage_user`, the two latency columns REAL default 0.0, all ensured in `init_db`). It is a derived counter table (NOT in `SOFT_DELETE_TABLES`, like `gateway_usage_ledger`) and is deliberately separate from `users` so accumulating never invalidates the auth/user cache. `database.get_correction_usage(user_uid)` (via `_get_usage`) returns the stored sums PLUS computed averages: `avg_tokens` (total_tokens/calls), `avg_upstream_latency_ms`, `avg_total_latency_ms`, `avg_tokens_per_second` (completion_tokens over total upstream seconds), and `avg_cost_usd`. **Profile display:** `routers/profile/usage._correction_usage(uid, include_cost)` shapes it via the shared `_usage_view(data, include_cost)` (mirrors `_ai_quota`); `profile/index.py` builds it only for `is_owner or viewer_is_admin` and passes `include_cost=viewer_is_admin`, exposed as the `correction_usage` dict on the context and `ProfileOut`. The view surfaces the sums plus averages - `avg_tokens` (avg tokens/call), `avg_latency_ms` (avg upstream latency), `avg_total_latency_ms`, `avg_tokens_per_second` (avg speed), and `total_time_s` (total upstream seconds) - rendered as extra tiles on the card. **Financial gating:** tokens/call-count and the performance tiles show to the owner and admins; the dollar `cost_usd` and `avg_cost_usd` keys are present ONLY when `viewer_is_admin`, in BOTH the HTML card (`templates/profile.html`, `.correction-usage-*`) and the `respond(..., model=ProfileOut)` JSON (same rule as `_ai_quota`'s `spent_usd` - hiding it in the template alone would leak it to a member fetching their own profile as JSON). The card renders only when `correction_usage.calls` > 0.
- **Import-cycle discipline.** `correction.py` imports only `stealth`, `config`, `database.get_table`/`add_correction_usage`, and `services.background.background` at module top; `clear_user_cache` is imported lazily inside `_run_correction`. Never import `content` or `utils` at module top.
- **Settings live on `users`:** three columns `ai_correction_enabled` (0/1), `ai_correction_sync` (0/1, default 0 = background), and `ai_correction_prompt` (text, default `config.DEFAULT_CORRECTION_PROMPT`), ensured in `database.backfill_api_keys()` (the user column-ensure block run by `init_db`) and seeded born-live in `utils._create_account`. The edit route is the owner-or-admin leaf `POST /profile/{username}/ai-correction` (`routers/profile/ai_correction.py`, `AiCorrectionForm{enabled, sync, prompt}`, audit key `profile.ai_correction`). The owner-only values are exposed on the profile page context and `ProfileOut` (`ai_correction_enabled`/`ai_correction_sync`/`ai_correction_prompt`, gated by `is_owner`), the UI block lives in `profile.html` (owner-only: enable checkbox, **Apply mode** select, prompt textarea) wired by `static/js/AiCorrection.js` (`app.aiCorrection`), and Devii drives it via the owner-scoped `ai_correction_get`/`ai_correction_set` tools (`services/devii/ai_correction/`, `handler="ai_correction"`, `requires_auth=True`, not confirm-gated - it is a reversible per-user toggle; `ai_correction_set` accepts `enabled`, optional `sync`, optional `prompt`).

## AI modifier (`services/ai_modifier.py`)

**A sibling of AI content correction that runs only on an explicit inline directive.** The engine reuses the correction plumbing wholesale and differs only in the trigger and the apply-mode/enabled defaults. When a user enables it (default ON), any prose field they author is rewritten by the AI gateway **only where the text contains an inline `@ai <instruction>` directive**; the configured prompt acts as a system message telling the model to execute the instruction behind `@ai` and replace that marked part, including the `@ai` marker itself.

- **The `@ai` gate is the whole difference.** `has_ai_directive(text)` matches the regex `@ai\s+\S` (case-insensitive). `schedule_modification(user, table, uid, request=None)` is a no-op unless a user is present, `table` is in `CORRECTABLE_FIELDS`, `user["ai_modifier_enabled"]` is truthy, the user has an `api_key`, AND at least one of the table's registry fields actually contains an `@ai` directive. `_run_modification` re-checks the gate per field, so untriggered fields are never sent to the gateway and never metered. Triggerless writes cost nothing.
- **Total reuse of the correction layer.** `CORRECTABLE_FIELDS`, `PENDING_SCOPE_KEY`, `gateway_complete`, the sync/background apply-mode mechanics (`_run_inline_awaited` -> `loop.run_in_executor` + `request.scope[PENDING_SCOPE_KEY]` awaited by the `await_pending_corrections` middleware in `main.py`, the same self-deadlock-avoiding path), and the per-user usage upsert pattern are all imported from / mirror `services/correction.py`. `modify_text(api_key, prompt, text, context="")` composes a modifier system message and calls the shared `gateway_complete`. The same hooks fire it: `profile/index.update_profile` calls both `schedule_correction` and `schedule_modification`, and the content/comment/messaging cores invoke it alongside correction, so it covers the web UI, REST, devRant, and Devii in one place. Code and source files are never modified (same `CORRECTABLE_FIELDS` registry, `gists.source_code`/project files/Gitea excluded).
- **Context-aware (modifier only, not correction).** Unlike correction, the modifier gives the model a grounding **context block** so an `@ai` instruction can reason about who is asking and what it is attached to. `services/ai_context.py` `build_context(table, uid, row, user_uid) -> str` assembles it; `_run_modification` builds it **lazily once per row** (only after a field is confirmed to contain `@ai`, so triggerless writes do no extra queries) and passes it to every field's `modify_text`, which appends it to the system message under a `# Context (use it to inform the result; never echo this block)` header. The block has three parts, each wrapped in try/except so a failed lookup never blocks the modification: (1) **date** - `Today is DD/MM/YYYY on the DevPlace developer network.`; (2) **author/stats** - the author's username, role, level, stars, post count (`get_user_post_count`), leaderboard rank (`get_user_rank`), follower count (`get_follow_counts`), member-since date, and bio (capped `MAX_BIO`); (3) **location/thread** - table-specific: a comment gets the target post/project/gist/news title + excerpt and the parent comment it replies to; a post gets its topic and attached project; a project/gist gets its sibling title/description and, for gists, the language + `source_code` (truncated, marked "reference only" - this is read context, the modifier still never WRITES `source_code`); a direct message gets the recipient and the last `MAX_THREAD_MESSAGES` messages of the conversation (each truncated). All excerpts are length-capped (`MAX_*` constants) to bound token cost; the added context tokens are billed and metered like any other input via the usage stats. So `@ai answer the question above` in a comment, `@ai write my bio from my stats`, or `@ai reply to this` in a DM all work. The message-thread query binds a param named `:current` (NOT `:self`, which collides with `db.query`'s bound-method argument).
- **Defaults differ:** enabled **ON** by default and apply mode defaults to **synchronous** (`ai_modifier_sync` column defaults to 1), the inverse of correction.
- **Per-user usage (only on success).** Same plumbing as correction: `_usage_from_headers` captures the token, cost, and timing headers (`X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`); `_run_modification` accumulates per-field gateway usage into a `totals` dict and, when `totals["calls"] > 0`, makes ONE `database.add_modifier_usage(user_uid, totals)` call (a single `totals` dict) that delegates to the shared `database._add_usage` atomic `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` upsert into the `modifier_usage` table (running SUMS `calls`/token/`cost_usd`/`upstream_latency_ms`/`total_latency_ms` totals, the two latency columns REAL default 0.0, separate from `users` so it never busts the auth cache, NOT in `SOFT_DELETE_TABLES`, ensured in `init_db`). `database.get_modifier_usage(user_uid)` (via `_get_usage`) returns the sums plus the same computed averages as correction (`avg_tokens`, `avg_upstream_latency_ms`, `avg_total_latency_ms`, `avg_tokens_per_second`, `avg_cost_usd`).
- **Settings live on `users`:** three columns `ai_modifier_enabled` (0/1, default 1), `ai_modifier_sync` (0/1, default 1 = sync), and `ai_modifier_prompt` (text, default `config.DEFAULT_MODIFIER_PROMPT` = "Execute what is behind `@ai` (the prompt) and replace that part including `@ai`"), ensured in `database.backfill_api_keys()` (the user column-ensure block run by `init_db`) and seeded born-live in `utils._create_account`. **Because the feature is on by default, `backfill_api_keys()` ALSO runs a `with db:` `UPDATE users SET ... WHERE ... IS NULL` so EXISTING accounts inherit the on/sync defaults** (a freshly `create_column_by_example`-added column is NULL on existing rows, which would read as disabled - the backfill is what makes "on by default" true for everyone, not just new accounts). The `with db:` wrapper is load-bearing: a raw `db.query` write that does not commit holds the SQLite write lock (see "Dataset rules"). The edit route is the owner-or-admin leaf `POST /profile/{username}/ai-modifier` (`routers/profile/ai_modifier.py`, `AiModifierForm{enabled, sync, prompt}`, audit key `profile.ai_modifier`). The owner-only values are exposed on the profile page context and `ProfileOut` (`ai_modifier_enabled`/`ai_modifier_sync`/`ai_modifier_prompt`, gated by `is_owner`); `modifier_usage` is built by `routers/profile/usage._modifier_usage(uid, include_cost=viewer_is_admin)` (via the shared `_usage_view`) for `is_owner or viewer_is_admin`, surfacing the sums plus the performance averages (avg tokens/call, avg latency, avg tokens/sec, total time) like `correction_usage`; the dollar `cost_usd` and `avg_cost_usd` are present only when `viewer_is_admin`, in both HTML and JSON. The UI block lives in `profile.html` (owner-only: enable checkbox, **Apply mode** select defaulting to sync, prompt textarea, plus the usage card reusing the `.correction-usage-*` classes with the extra performance tiles) wired by `static/js/AiModifier.js` (`app.aiModifier`). Devii drives it via the owner-scoped `ai_modifier_get`/`ai_modifier_set` tools (`services/devii/ai_modifier/`, `handler="ai_modifier"`, `requires_auth=True`, not confirm-gated; `ai_modifier_set` accepts `enabled`, optional `sync`, optional `prompt`).

## Background Services

The `devplacepy/services/` package provides a generic framework for running background async services alongside the FastAPI server. Architecture:

Services are fully managed from the **Services admin tab** (`/admin/services`): start/stop, enable-on-boot, run-now, edit parameters, clear logs, adjustable log buffer size, with live status. All of this is generic - a new service gets it for free by declaring its config and implementing `run_once`.

### DB-backed state (correct across workers)
In production (`make prod`, 2 workers) only the lock-holding worker runs services, but an admin request can hit either worker. State therefore lives in the DB, not process memory:
- **Desired/config state** in `site_settings` (via `get_setting`/`set_setting`): `service_<name>_enabled` (`"0"`/`"1"` - also the boot flag), `service_<name>_command` (`"<verb>:<counter>"`, verbs `run`/`clear`), `service_<name>_log_size`, plus each declared config field's own key.
- **Observed state** in the `service_state` table (one row per service): `status`, `last_run`, `next_run`, `started_at`, `heartbeat`, `logs` (JSON), `updated_at`. Written by the supervising worker, read by any worker.

`main.py` registers every service in **all** workers (so `describe_all()` works anywhere) but only the lock worker calls `service_manager.supervise()`.

### `ConfigField` (`services/base.py`)
Declarative parameter spec. `type` ∈ `int`/`str`/`url`/`text`/`password`/`bool`/`select`. `coerce(raw)` validates and types (raises `ValueError`, never silent); `read()` is the lenient runtime reader (falls back to `default`); `spec()` renders the field for the template/JSON (secrets masked, never sent). Optional `minimum`/`maximum`/`options`/`help`/`secret`.

### `BaseService` (`services/base.py`)
Abstract class for all services:
- **`name`**, **`interval_key`** (default `service_<name>_interval`; a subclass may point it at an existing key), **`min_interval`**.
- **`config_fields`** - class-level list of `ConfigField`. The framework prepends built-in `enabled` and interval fields and appends `log_size`; `all_fields()` returns the full set.
- **`get_config()`** → typed dict from settings; **`is_enabled()`**, **`current_interval()`** (floored to `min_interval`).
- **`log(message)`** - writes to `log_buffer` and standard `logging`.
- **`run_once()`** - abstract; override with actual work. Read parameters via `self.get_config()`.
- **Reconciling loop** (`_run_loop`/`_tick`, ~1s tick): honors `enabled`, runs `run_once()` when due or on a `run` command, handles `clear`, rebuilds the log deque if `log_size` changed, and persists observed state (throttled, force on transitions). Start/stop/run/clear from any worker are just DB writes the loop reacts to within ~1s - services are never hard-restarted.
- **Lifecycle hooks** `async on_enable()` / `async on_disable()` - called on transition to enabled / disabled (and `on_disable` on shutdown). Override for long-running resources that outlive one `run_once` (e.g. the bot fleet); `run_once` then becomes a periodic reconcile/metrics tick.
- **`collect_metrics() -> dict`** - generic live metrics. Return `{"stats": [{"label", "value"}], "table": {"columns": [...], "rows": [[...]]}}`. Persisted to `service_state.metrics` each tick and rendered live in the Services tab (stat cards + table) by `ServiceMonitor.js`. Default returns `{}`.
- **`default_enabled`** class flag (default `True`) - set `False` for opt-in services so they never auto-start on boot (the bots service uses this).
- **`title` / `description`** class attrs - shown in the admin UI; every service should set a `description` of what it does. `ConfigField` takes `group="…"` so the detail page can render fields as sections (built-in fields are grouped "General"/"Advanced").
- `ConfigField` supports `int`, `float`, `str`, `url`, `text`, `password`, `bool`, `select`.
- **`describe()`** - builds the dict the UI consumes: `title`, `description`, derived display status (`stopped` when disabled, `running` with fresh heartbeat, `stalled` when enabled but heartbeat older than `STALE_SECONDS`), meta, logs, metrics, flat `fields`, and `field_groups` (ordered `{name, fields}` sections) read from `service_state` + settings.

**Admin UI** (`routers/admin/services.py`): `/admin/services` is a slim **index** (`services.html`, an `admin-table` of name/description/status/uptime/last-run/next-run/interval) where each row links to `/admin/services/{name}`. The **detail** page (`service_detail.html`) holds everything for one service - header with controls (start/stop/run/clear), and `[data-tabs]` tabs **Overview** (meta + metrics), **Configuration** (the config form rendered as a `<fieldset>` per `field_groups` entry), **Logs**. `ServiceMonitor.js` is dual-mode (polls `/admin/services/data` for the index, `/admin/services/{name}/data` for the detail, via the shared `Poller`; its config-save POST goes through `Http.sendForm`) and shares one `update(root, svc)` over `data-*` hooks; tabs use the generic reusable `static/js/Tabs.js` (`[data-tabs]`/`[data-tab]`/`[data-tab-pane]`, wired in `Application.js`). Adding a service still needs zero UI code.

### `ServiceManager` (`services/manager.py`)
Singleton that manages all registered services:
- `register(service)` - add a service
- `describe_all()` → `list[dict]` (per-service `describe()`)
- `set_enabled(name, bool)` - Start/Stop (writes the enabled flag)
- `send_command(name, "run"|"clear")` - one-shot command channel
- `save_config(name, form)` - strict validation via each `ConfigField`; blank secret = keep existing; returns `{ok, errors}`
- `supervise()` - start the reconciling task per service (lock worker only)
- `shutdown_all()` - cancel tasks on server shutdown

### `NewsService` (`services/news.py`)
Implements `BaseService`:
- Fetches `GET {news_api_url}` → `{"articles": [...]}`
- Every article is graded via AI: `POST {news_ai_url}` with model `{news_ai_model}`. Both default to the internal gateway (`INTERNAL_GATEWAY_URL` / `molodetz`); the key falls back to `internal_gateway_key()` when `news_ai_key`/`NEWS_AI_KEY` is unset
- ALL articles are inserted into `news` table regardless of grade (never silently skipped)
- Each article gets a `status` field: `"published"` if grade >= threshold, `"draft"` otherwise
- All parameters (`news_api_url`, `news_ai_url`, `news_ai_model`, `news_grade_threshold`, `news_ai_key`, interval) are declared as `config_fields` and edited on the Services tab
- Articles re-synced each run (upsert by `external_id`) - grade, status, images updated on every cycle
- Slugs generated via `make_combined_slug(title, uid)` - same format as posts/projects

### `BotsService` (`services/bot/`)

A Playwright-driven fleet of AI personas that browse and interact with a DevPlace instance (posts, comments, votes, gists, projects, issues, follows, messages). It is the former standalone `dpbot.py` refactored into a package and wired into the service framework. Opt-in (`default_enabled = False`).

Module layout (one responsibility per file):

| File | Purpose |
|------|---------|
| `config.py` | Static constants + defaults (URLs, model, costs, personas, categories, paths) |
| `runtime.py` | `BotRuntimeConfig` dataclass - per-run settings threaded into a bot |
| `state.py` | `BotState` dataclass + JSON persistence (incl. the per-bot `identity` card) |
| `llm.py` | `LLMClient` (parameterized: key/url/model/costs); content generation + quality checks + the `decide`/`generate_identity` decision engine |
| `news_fetcher.py` | `NewsFetcher` - TTL-cached article source |
| `browser.py` | `BotBrowser` - Playwright wrapper (human-like typing/clicking/scrolling) |
| `registry.py` | `ArticleRegistry` - flock-based cross-bot article dedupe |
| `bot.py` | `DevPlaceBot` - the orchestrator (sessions, action cycle, `run_forever`) |
| `service.py` | `BotsService(BaseService)` - fleet manager + metrics |

Mechanics:
- `run_once` is a **reconcile tick**: it ensures `bot_fleet_size` bot tasks are alive (relaunching dead ones), scales the fleet up/down, and is a no-op heavy work otherwise. Each bot is an `asyncio` task running `DevPlaceBot.run_forever` in the lock worker's loop. `on_disable` cancels the fleet (each bot closes its browser on `CancelledError`).
- All "startup parameters" of the old script are `config_fields`: `bot_fleet_size` (was `--bots`), `bot_headless` (was `--headed`), `bot_max_actions` (was `--actions`), plus `bot_base_url`, `bot_api_url` (defaults to the internal gateway), `bot_news_api`, `bot_model` (defaults to `molodetz`), `bot_api_key` (secret; falls back to `internal_gateway_key()`), `bot_input_cost_per_1m`, `bot_output_cost_per_1m`.
- **A bot uses its own account's `api_key` for gateway calls once it has an account.** `bot_api_key` (-> `cfg.api_key`, the shared `internal_gateway_key()` fallback) is only the bootstrap credential used until the bot registers/logs in. After `_ensure_auth()` succeeds in `run_forever`, `DevPlaceBot._adopt_account_api_key()` fetches the bot's own `users.api_key` from its profile JSON (`fetch('/profile/{username}', {Accept: 'application/json'})` over the authenticated browser session, where `ProfileOut.api_key` is exposed to the owner) and switches `LLMClient.api_key` to it, persisting it in `BotState.account_api_key`. The key is read once and reused on every later session (the `LLMClient` is constructed with `state.account_api_key or cfg.api_key`, and `_raw_call` reads `self.api_key` per request so a mid-session swap takes effect on the next call). This routes each bot's gateway spend through its own user attribution (`resolve_owner()` -> `(owner_kind="user", owner_id=uid)`) instead of the shared internal key. Adoption is best-effort: a fetch failure leaves the bot on the fallback key.
- **Behavior and pacing tuning are admin `config_fields` too**, threaded through `BotRuntimeConfig` into the bot (never read module constants directly for these). The `config.py` constants (`MAX_BOTS_PER_ARTICLE`, `ARTICLE_TTL_DAYS`, `GIST_MIN_LINES`, `ACTION_PAUSE_MIN_SECONDS`/`ACTION_PAUSE_MAX_SECONDS`, `BREAK_SCALE_DEFAULT`) are only the **defaults**; the live values come from the settings: **Behavior** group - `bot_max_per_article` (-> `ArticleRegistry.max_per_article`), `bot_article_ttl_days` (-> `ArticleRegistry.ttl_days`), `bot_gist_min_lines` (-> `LLMClient.gist_min_lines`); **Pacing** group - `bot_action_pause_min_seconds`/`bot_action_pause_max_seconds` (the intra-session pause, clamped lo/hi in `DevPlaceBot.__init__` so an inverted pair is harmless) and `bot_break_scale` (a float multiplier on the between-session break, floored at 0.1 and the result floored at 5s); **Decisions** group - `bot_ai_decisions` (-> `BotRuntimeConfig.ai_decisions`, the AI-decision kill switch) and `bot_decision_temperature` (-> `BotRuntimeConfig.decision_temperature`). To add a new tuning knob: add the default to `config.py`, a `ConfigField` to `BotsService.config_fields` (its `group` creates/joins a UI section automatically), a field to `BotRuntimeConfig`, the `cfg[...]` mapping in `_launch_slot`, and consume it via the runtime config - do not reach for the module constant at the call site.
- **Live pricing** is published via `collect_metrics()`: fleet stat cards (bots running, fleet cost, LLM calls, tokens, posts/comments/votes) plus a per-bot table. `Fleet cost` is cumulative across restarts because each bot seeds its `LLMClient` totals from its saved `BotState`; it is also an *estimate* from the configurable `bot_input_cost_per_1m`/`bot_output_cost_per_1m`, not the gateway's recorded spend. `Cost rate` and `Projected 24h cost` come from a **sliding window**, not lifetime: `_sample_cost()` appends `(now, total_cost)` each tick into `_cost_samples`, evicts samples older than `config.COST_WINDOW_SECONDS` (600s), and measures the cost delta over the retained span; the projection is shown only once the window reaches `config.COST_WARMUP_SECONDS` (120s), otherwise both cards read "warming up". Samples reset when `_started_at` changes (`_cost_anchor`). Never project from lifetime `Fleet cost`/uptime, and never anchor the rate at startup - the boot burst (all bots ramping at once) then dominates the average and massively overstates a 24h projection extrapolated 274x from a few minutes.
- Heavy deps (`playwright`, `faker`) are **lazily imported** inside `run_once`/`_launch_slot` (install the `bots` extra). The service registers and shows in the admin tab even when they are absent; it logs and stays idle. Per-bot state lives in `~/.devplace_bots/state_slot{N}.json`; the article registry in `~/.dpbot_article_registry.json`.
- **Direct messages stay a dialog, never a monologue.** A bot may open a conversation, but `_compose_message` first calls `_spoke_last_in_thread()` and refuses to send when the last `.message-bubble` in `.messages-thread` is `.mine` (the bot itself sent the most recent message). So a bot only replies after the other person has spoken since its last message; it never sends two messages in a row. An empty thread (no bubbles) is allowed, so a bot can still initiate contact. This guards both reply paths (`_check_messages` and `_send_message`), mirroring how a real person waits for a reply before writing again.

#### Live screenshot monitor (`services/bot/monitor.py`, `/admin/bots`)

The admin **Bot Monitor** (`/admin/bots`, sidebar link in `admin_base.html`, `require_admin`, `noindex`) shows the **latest low-quality screenshot per bot** with its username, persona, and current action/status, auto-refreshing every 2s. It is the constructive counterpart to the cost metrics: cost tells you what the fleet spent, the monitor tells you what each bot is looking at right now.

- **Capture is a `BotBrowser` concern** because the browser owns the Playwright `page`. `BotBrowser.bind_monitor(slot, username, persona)` (called from `DevPlaceBot.run_forever` once auth completes), `BotBrowser.note(action=, status=)`, and `BotBrowser.capture(reason, force=)` are the surface. `capture` takes a JPEG via `page.screenshot(type="jpeg", quality=MONITOR_JPEG_QUALITY=35, scale="css")`, throttled to one every `MONITOR_MIN_INTERVAL_SECONDS` (1s) unless `force=True`. **Capture is best-effort and never raises into the bot loop.** It fires at the meaningful-event choke points already centralized in `BotBrowser`: `goto` (page load, forced), `scroll`, `fill` (field input), `reload` (forced), plus `DevPlaceBot._action(tag, detail)` (every post/comment/vote/react/gist - an action-labelled forced capture via `asyncio.create_task`).
- **Storage is in-memory + one file per bot, never accumulating.** `monitor` (the `BotMonitor` singleton) holds a `slot -> BotFrame` dataclass map of tiny metadata, and writes the JPEG bytes to `BOT_DIR/monitor/slot{N}.jpg` (`config.DATA_DIR`, OUTSIDE the package, NOT under `/static`), **overwritten atomically in place** (`.tmp` then `replace`). One frame per slot, capped at the fleet size (`bot_fleet_size`, 0 to 20). `_stop_slot` calls `monitor.drop(slot)` to evict the row + unlink the file. No DB table, no soft-delete (nothing is persisted as a row), no shard tree (one file per bot is already bounded). This is what keeps it cheap at full fleet: at most 20 small JPEGs in RAM-backed files, refreshed in place.
- **Transport is polling, not WebSocket - deliberately.** Frames live only in the worker running the Bots service (the service lock owner), exactly like container logs/metrics. Rather than a lock-owner-gated WS with the 4013 retry, the monitor reuses the **existing `service_state` cross-worker bridge**: the lock owner publishes the frame metadata in `collect_metrics()["frames"]` (persisted to the `service_state` DB row by `BaseService._persist_state`), and ANY worker reads it back via `service_manager.get_service("bots").describe()["metrics"]["frames"]`. The page polls `GET /admin/bots/data` (`Poller`, 2s, `pauseHidden`) for that metadata and renders `<img>` tags at `GET /admin/bots/{slot}/frame.jpg` (raw bytes from the shared `BOT_DIR/monitor/` file, `Cache-Control: no-store`, served by any worker since it is a file under `DATA_DIR`). The `frame_url` carries a `?t=captured_at` cache-buster so a new frame is fetched only when it changed. This is the lightest correct option for the multi-worker model: no extra socket, no handshake, no 4013 bounce, and it reuses the same DB-backed metrics path the services page already polls. JSON shape is `schemas.AdminBotsOut`/`BotFrameOut`; Devii reads the same `/admin/bots/data` via the admin `bot_monitor` action; docs in `docs_api.py` (`admin-bots-*`, the frame endpoint in `NON_BODY_ENDPOINTS`).
- **Frontend** is `static/js/BotMonitor.js` (`app.botMonitor`, one class, `Http` + `Poller`) rendering the `static/css/bots.css` responsive grid (`bot-active`/`bot-idle` by frame age, design tokens only). Add a new capture point by calling `self.b.capture("reason")` (or `note(...)` then capture) at the new event; do not screenshot outside `BotBrowser` (it owns the page and the throttle).

#### Realism: titles, topic spread, gist quality, inter-bot threads, thread-aware distinct opinions

Five content/behaviour mechanics keep the fleet from reading as machine-generated. Treat them as the realism contract; do not regress them.

- **Post titles are rewritten, never the raw headline.** A post body reacts to a news article, but the title is generated separately by `llm.generate_post_title(headline, persona, category)` - a short (3 to 8 word) human title in the persona's voice, not the verbatim article headline. The raw headline is still the registry dedupe key (`clean_title`), but the typed title is `type_title` (the rewritten one, `strip_md`-capped at 120). Same applies to projects: `generate_project_title`/`generate_project_desc` take the persona and run through `strip_label`. **`LLMClient.strip_label`** removes LLM label echoes (`Title:`, `Project Name:`, `Snippet name:`, and a trailing ` Concept:`/` Description:` clause) so a title can never be `Project Name: X Concept: Y`. Run every generated title through `strip_label`.
- **Personality drives topic and category, not just voice.** Category is chosen by `config.pick_category(persona)` (a persona-weighted `random.choices` over `PERSONA_CATEGORY_WEIGHTS`), replacing the old content classifier - a grumpy_senior rants, an enthusiastic_junior shows off and asks questions. Article selection is persona-scored: `config.persona_article_score(article, persona)` counts `SEARCH_TERMS` keyword hits, and both `_pick_article` (cache path) and `ArticleRegistry.reserve_unused` (direct path) rank articles by score plus a `random()` jitter, so different personas gravitate to different news rather than all reacting to the same headline.
- **Gists are quality-gated and non-trivial.** `generate_gist` now writes the code first (8 to 20 lines, persona-flavoured via `PERSONA_GIST_FLAVOR`, language drawn from `PERSONA_LANGUAGES`), then titles/describes it from the code. `_create_gist` runs a two-attempt loop through `llm.gist_quality_check(title, code, language)`: a `TRIVIAL_GIST_TERMS` blocklist on the title, a minimum non-empty line count (`LLMClient.gist_min_lines`, admin `bot_gist_min_lines`, default 6), and a strict LLM judge that rejects 101-tutorial snippets. Mirrors the post/comment `quality_check` loop; a reject regenerates once then skips.
- **Every comment is thread-aware and must hold a distinct opinion.** A bot never comments blind to what others already said. Before generating, `_comment_on_post` / `_reply_to_random_comment` call `DevPlaceBot._existing_comments()`, which scrapes the last `MAX_SIBLING_COMMENTS` (8) visible `.comment-text` bodies with their `/profile/` author (own comments excluded, each capped at `SIBLING_SNIPPET_LEN`=240). The block is threaded into `llm.generate_comment(..., existing_comments=)`, whose system prompt then forces a point none of them made - a different angle, a missed caveat, or respectful disagreement with one of them by name - and forbids repeating any opinion, framing, example, or question already raised. The mechanical backstop is in `llm.quality_check(..., siblings=)`: a candidate whose `_overlap_ratio` against the joined sibling text exceeds `SIBLING_OVERLAP_THRESHOLD` (0.55) is rejected as `echoes another comment` and regenerated (same two-attempt loop as the source-restatement gate at `RESTATEMENT_OVERLAP_THRESHOLD`=0.6). This is what stops a salient post from drawing a dozen near-identical "tipping point / feedback loop / cooldown" replies. Any new comment-generation path MUST gather siblings and pass them to both `generate_comment` and `quality_check`.
- **Usernames read like nerd handles, not real names.** Bots do NOT sign up with faker person names; they pick devRant/Hacker-News-style handles. Two layers, LLM-first with an offline fallback: (1) `llm.generate_handle_candidates(persona)` asks the model for ~8 distinct handles grounded in the persona and its `SEARCH_TERMS` interests (tech nouns, leetspeak, adjective+noun, creatures, short word+number), each run through `handles.sanitize_handle`; the pool is cached on `BotState.handle_candidates` and consumed by `_next_handle()`. (2) `handles.make_handle(interests)` is the pure, offline algorithmic generator (curated word banks + probabilistic leetspeak + number/separator/casing decoration, persona-seeded from `SEARCH_TERMS`), used as the fallback whenever the LLM pool is empty or a signup collides. Both layers emit only `[A-Za-z0-9_-]`, 3 to 20 chars (the old `first.last` styles silently failed signup validation, which rejects dots). On a third signup retry a random number is appended to bust collisions. Word banks and tuning constants live in `services/bot/handles.py`; never reintroduce faker person-name handles.
- **Bots engage each other, and threads deepen.** `ArticleRegistry` allows up to `max_per_article` holders per article (admin `bot_max_per_article`, default 2), each with a **distinct category** (stored as `holders: [{bot, category, time}]`; old `{bot, time}` rows are normalized on load), so two bots can post different angles on the same trending news - which gives them each other's posts to discuss. A holder ages out after `ttl_days` (admin `bot_article_ttl_days`, default 7). `_engage_community()` (called once per normal/deep session after notifications) goes to `/feed?tab=recent|trending`, ranks non-own posts with a preference for `known_users` authors, opens one, comments, and replies into the comment thread. The on-post reply-to-comment probability is also raised so multi-bot threads actually form. `reserve(title, owner, category)` is idempotent per owner and enforces the same distinct-angle / max-holders rules as `reserve_unused`.

#### AI-driven decisions and identity cards (`bot_ai_decisions`)

Opt-in mechanic (off by default; full design and cost model in `aibots.md`). When `bot_ai_decisions` is on, a bot replaces the procedural action cascade with one LLM decision call per page, driven by a unique AI-generated identity. **Do not regress the grounding and the kill-switch fallback.**

- **Two LLM entrypoints, both on `LLMClient`.** `generate_identity(archetype)` runs once per bot (seeded from the persona archetype) and returns the identity card: `name`, `backstory`, `interests`, `dislikes`, `temperament`, and four 0.0-1.0 scalars (`verbosity`, `contrarianness`, `generosity`, `curiosity`) plus `rhythm`. `_normalize_identity` clamps every scalar and defaults `interests` to the archetype's `SEARCH_TERMS`, so a malformed model reply can never produce a broken card. The card persists on `BotState.identity` (serialised to `state_slot{N}.json`); old state files load with an empty dict and regenerate on first AI session. `decide(identity, page_state, menu, history, temperature)` returns `{plan, energy, stop_after}` (strict JSON), the per-page ordered action plan.
- **JSON is parsed raw, never `clean()`-ed.** `_call` runs the output through `clean()` (which strips `*`/`_` and collapses whitespace, corrupting JSON keys like `stop_after`); the decision engine instead uses `_raw_call` (the extracted HTTP+cost-accounting core, no `clean`) and `_parse_json` (tolerates code fences and prose wrappers by slicing the first `{`..`}`). When adding any JSON-returning LLM method, use `_raw_call`, not `_call`.
- **The harness supplies the options; the model only picks.** `DevPlaceBot._build_menu(page_state)` builds the action menu from the same observable-page predicates the random `_cycle` uses, listing only actions that are physically possible and policy-allowed (throttles like `can_post`/`can_project`/own-post-no-mention are enforced by *omitting* the action). `decide` filters the returned plan against that menu (`_filter_plan` drops any unknown action), re-asks once on malformed/empty output, and the executor `_run_plan` falls back to a single `navigate` to feed when the plan is still empty - **never a `random` draw**. `_dispatch_action` maps each action string to the existing per-action method (`_comment_on_post`, `_vote_for_page`, `_react_to_object`, `_navigate_to`, ...); content actions still run their `quality_check`/`gist_quality_check` gates unchanged, so quality cannot degrade.
- **`_cycle_ai` mirrors `_cycle`'s signature and return tuple** so `run_forever` swaps between them with `(self._cycle_ai if use_ai else self._cycle)(...)`. `energy` and `stop_after` replace `_session_type` mood and `_fatigue_check`: the session ends when `session_actions >= stop_after`, the intra-action pause is `_scaled_pause()` (scaled by `energy`), and the between-session break is scaled by `energy` too. Timing jitter (`_read_like_human`, `_type_like_human`, `_idle`) stays procedural - the model decides *what*, not the millisecond cadence. `use_ai` is `self._ai_decisions and bool(self.state.identity)`; identity generation and the flag are gated so a disabled fleet pays nothing and behaves exactly as before.

### `GatewayService` (`services/openai_gateway/`)

OpenAI-compatible LLM gateway (the ported `openai5.py`), mounted at `/openai/v1/*`
(`routers/openai_gateway.py`, prefix `/openai`). It is a **service that serves an HTTP
endpoint** rather than a loop:

- The router is thin: it calls `service_manager.get_service("openai").handle(request, subpath)`.
  `POST /v1/chat/completions` runs the full gateway (vision augment → model override →
  forward → optional SSE re-emit via `_fake_stream`); `POST /v1/embeddings` forwards to the
  configured embeddings upstream (`handle_embeddings`, no vision/streaming); `/v1/{path:path}`
  is a transparent passthrough to the upstream base. Disabled → 503, unauthorized → 401,
  upstream connection failure → 502.
- **Per-call cost/token response headers.** Every gateway response (chat, embeddings, passthrough; success and error) carries `X-Gateway-*` headers describing that single call: `Model`, `Backend`, `Prompt-Tokens`, `Completion-Tokens`, `Total-Tokens`, `Cache-Hit-Tokens`, `Cache-Miss-Tokens`, `Reasoning-Tokens`, `Cost-USD`/`Input-Cost-USD`/`Output-Cost-USD` (dollars), `Cost-Native` (1 if the upstream returned a native cost), `Tokens-Per-Second`, `Upstream-Latency-Ms`, and `Context-Window`/`Context-Utilization` when the model's window is known. `GatewayUsageLedger.record(...)` now RETURNS the computed ledger row (or None on failure); each handler's `finalize` closure maps it through `usage.usage_response_headers(row)` and attaches it to the returned `Response` (`resp_headers`). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads `X-Gateway-Cost-USD`/token headers off its own correction call to accumulate per-user totals (see "AI content correction").
- **Serves in every worker.** Config/enabled come from `site_settings` (via the
  service's `get_config()`/`is_enabled()`), so any uvicorn worker answers - not just the
  supervisor worker. Per-worker runtime (`GatewayRuntime`: `httpx.AsyncClient` pool +
  `asyncio.Semaphore` sized to `gateway_instances`, plus counters and a `VisionCache`)
  is created lazily on first request and rebuilt when `instances`/`timeout`/cache size
  change. `gateway_instances` is the **scaling knob** (concurrency per worker); process
  scaling is uvicorn workers.
- **Auth** (`authorize`): reuses `get_current_user` (cookie/X-API-KEY/Bearer/Basic).
  Allowed if `gateway_require_auth` is off, the presented X-API-KEY/Bearer equals the
  static `gateway_access_key` **or the auto-generated `gateway_internal_key`**, or the
  resolved user is an admin (`gateway_allow_admins`) or any user (`gateway_allow_users`).
  Enabled by default (`default_enabled=True`) so internal callers work out of the box.
- All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's
  `gateway_system_preamble` (see below), timeout,
  instances, vision url/model/key/cache/toggle, the Embeddings group
  (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal
  key, plus the Pricing/Reliability/Tracking groups below); live metrics
  (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success
  rollups) via `collect_metrics`.
- `main.py` registers it and exempts `/openai` from the rate-limit middleware and the
  maintenance gate. No new dependency (`httpx` already required).

#### System-message composition (date awareness + operator preamble)
`system_message.py` (`apply_system_directives`) runs in `handle_chat` only (NOT
`handle_embeddings`/`handle_passthrough`), after vision augmentation and before the upstream
payload is assembled, so it covers every chat consumer (news, bots, Devii guests, per-user
direct API calls). It composes ONE system message in a fixed, cache-friendly order:
**operator preamble (`gateway_system_preamble`) -> date line (`Current date: DD/MM/YYYY`) ->
the client's original system content**. Two behaviors:
- **Date awareness (EU, no time).** The current date is injected only when the composed text
  (preamble + client system content) contains NO date in any common format. Detection lives in
  `contains_date` via the `_DATE_PATTERNS` regex set: `YYYY-MM-DD` (ISO), `D/M/YYYY` and
  `DD/MM/YYYY` (also matches `MM/DD/YYYY`), `DD-MM-YYYY`, `DD.MM.YYYY`, and textual months
  (`14 June 2026`, `June 14, 2026`, `14 Jun 2026`, with abbreviations and optional ordinal/comma).
  The date is `datetime.now().strftime("%d/%m/%Y")` - **date only, never time**: omitting the
  time keeps the system message byte-stable for the whole day so upstream prompt/token caching
  stays effective.
- **Operator preamble.** `gateway_system_preamble` (Prompt group, `type="text"` textarea,
  default empty) is prepended ahead of the client's system content on every chat call so all
  calls carry operator preferences. When the client sent NO system message, the composed content
  (preamble and/or date) becomes a brand-new leading system message; when the client DID send one,
  its content is replaced in place (preamble + date + original text). A blank/whitespace-only
  preamble with an already-dated request is a no-op (no empty system message is created). The date
  detection runs over the preamble too, so an operator preamble that already states a date
  suppresses the injected date line.

#### Usage, cost, latency, and reliability tracking
The gateway records one row per upstream call (chat, vision, passthrough) and surfaces
per-hour and 24h analytics. The pieces:
- **`usage.py`** - `GatewayUsageLedger` writes to `gateway_usage_ledger` (one row per call,
  success and failure) and samples `gateway_concurrency_samples` each 30s tick. `normalize_usage`
  handles BOTH upstream shapes: DeepSeek (`prompt_cache_hit_tokens`/`prompt_cache_miss_tokens`,
  `completion_tokens_details.reasoning_tokens`) and OpenRouter (`prompt_tokens_details.cached_tokens`,
  native `cost`, `cost_details`). `compute_cost` PREFERS the upstream native `cost` (OpenRouter)
  and FALLS BACK to per-1M pricing for the chat backend (DeepSeek reports no cost); it also returns
  the input/output split (native cost is split by token ratio). `record()` is wrapped in
  try/except so a tracking failure never breaks the proxy.
- **`gateway.py`** - `_send` is the gated/timed/retried path: a `CircuitBreaker` (`reliability.py`)
  short-circuits when open; `retry_send` retries timeouts/connection errors/5xx with linear backoff;
  semaphore acquire time is the `queue_wait_ms`; an httpx per-request `trace` extension captures
  `connect_ms` (0 on pooled reuse). `handle_chat`/`handle_passthrough` record every outcome with
  timings, tokens, cost, params, and the resolved owner. Vision calls record from
  `VisionAugmenter._describe_one`.
- **Owner attribution** - `GatewayService.resolve_owner` maps the caller to `(owner_kind, owner_id)`:
  `internal:devii` (internal key), `key:access` (static key), `user:<uid>`/`admin:<uid>` (session or
  API key), else `anonymous`. `gateway_allow_users` (default **on**) is what lets a real user's
  `api_key` through `authorize()`; with it off, only internal/access/admin keys work and per-user
  attribution never happens. Devii operating a signed-in user sends that user's `api_key` as the LLM
  auth key (see `build_settings(..., owner_kind)` below), so Devii turns land under `user:<uid>` in the
  ledger, not `internal:devii`.
- **`analytics.py`** - `build_analytics(hours, top_n, pricing)` pulls the bounded window once (clamped
  to `MAX_WINDOW_HOURS=168`) and computes everything in Python: volume/throughput, tokens (sums + avg +
  p50/p90/p95/p99 + max via `reliability.percentile`), latency (upstream/overhead/queue/connect/total +
  tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching
  savings), behavior, and an hourly breakdown. `summary_metrics()` runs cheap SQL aggregates for the
  live service panel. TTFT/inter-token are returned `null` (gateway forwards non-streaming).
- **Admin** - `/admin/ai-usage` (HTML, `templates/ai_usage.html` + `static/js/AiUsageMonitor.js`) and
  `/admin/ai-usage/data?hours=&top_n=` (JSON) in `routers/admin/ package`, admin-gated. Schema
  `GatewayUsageOut`. Devii action `ai_usage` and docs `admin-ai-usage` expose the same endpoint.
- **Per-user usage** - `analytics.build_user_usage(owner_id, hours=24, pricing)` filters the ledger by
  `owner_id` (a uid matches both `user:` and `admin:` rows) and returns requests, success/error %,
  token totals, cost (window / per-hour / per-request / 30-day projection = 24h spend x 30), avg
  latency + tps, per-model breakdown, and an hourly cost series. Served admin-only at
  `GET /admin/users/{uid}/ai-usage` (schema `UserAiUsageOut`, docs `admin-user-ai-usage`) and rendered
  on the profile page by `static/js/UserAiUsage.js` (the `[data-ai-usage]` card, gated `{% if is_admin(user) %}`
  in `templates/profile.html`). The 24h window is a sound projection basis (unlike a startup burst),
  so `cost_24h * 30` is honest.
- **Profile visibility split** - `routers/profile/ package` `_ai_quota(uid, include_cost=False)` builds the
  profile quota bar. **The displayed spend is the user's REAL total gateway spend, not the Devii-turn
  subtotal:** `spent_usd`/`used_pct`/`requests`/`last_used` come from `analytics.user_spend_24h(uid)`,
  the SAME `gateway_usage_ledger` source (summed by `owner_id`, matching both `user:` and `admin:` rows)
  that feeds the stat grid's `cost.window_usd`. So the server-rendered `spent_usd` is byte-for-byte the
  card's "Cost 24h" and the two halves of the card reconcile. `turns` is still the `devii_usage_ledger`
  `turns_24h` count (Devii conversation turns) and is labeled **"Devii turns"** in the template precisely
  because it is a *different metric* from the grid's total **API requests** (one Devii turn fans out into
  several gateway requests; direct API-key calls add requests with no turn, which is why `0 Devii turns`
  can sit beside `18 API requests`). The `limit`/`turns` still come from the Devii service
  (`daily_limit_for`/`turns_24h`); an **unlimited** limit (`<= 0`, the admin default) sets
  `unlimited=True` so the template renders "no cap" / "No daily spend cap" instead of a misleading
  `/ $0.00`. **This is display-only: the Devii daily-cap ENFORCEMENT and `/devii/usage` still read the
  Devii-scoped `devii_usage_ledger.spent_24h`** (per-turn, resettable, and the only per-guest spend
  record since guest Devii calls hit the gateway under the shared internal key, not the guest cookie).
  The profile page renders three ways: **admin** sees the full gateway stats card *plus* a
  server-rendered quota bar (with spend/limit/Devii turns); the **owner** (non-admin, `is_self`) sees a
  quota-**only** card showing just the percentage; everyone else sees nothing AI-related. `_ai_quota`
  is fail-safe (returns `None` on any error or when the Devii service is unregistered) and the result
  rides on `ProfileOut.ai_quota`. **Dollar figures (`spent_usd`/`limit_usd`) are included only when
  `include_cost=viewer_is_admin`** - never for a non-admin owner. This matters because the route serves
  the same context as JSON via `respond(..., model=ProfileOut)` and `ProfileOut.ai_quota` is a
  free-form dict, so hiding dollars in the HTML branch alone would still leak them to a member
  requesting their own profile with `Accept: application/json`; the server-side omission is the real
  control. Non-admins (HTML and JSON) get only `used_pct`/`turns`/`requests`.
- **Retention** - `run_once` prunes both tables past `gateway_usage_retention_hours` (default 720 = 30 days).

#### Embeddings
`POST /openai/v1/embeddings` exposes an OpenAI-compatible text-embeddings model. Clients send the
generic model `molodetz~embed` (`config.INTERNAL_EMBED_MODEL`), which `handle_embeddings` remaps to
`gateway_embed_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when
`gateway_force_model` is on or the model is empty). It defaults to OpenRouter's
`qwen/qwen3-embedding-8b` at `https://openrouter.ai/api/v1/embeddings` (`config.EMBED_*_DEFAULT`).
`handle_embeddings` mirrors `handle_chat` but is simpler: no vision augmentation and no streaming -
build the payload, forward via `_send`, and record one ledger row through the same `finalize(...)`
closure. The config fields are the **Embeddings** group (`gateway_embed_enabled` default on,
`gateway_embed_url`, `gateway_embed_model`, `gateway_embed_key`) plus the Pricing-group
`gateway_embed_price_input_per_m` (default `$0.01` per 1M input tokens). `effective_config()` falls
the embed key back to `gateway_vision_key` then `OPENROUTER_API_KEY` (NOT `gateway_api_key`: that is
the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with
**`backend="embed"`**; `usage.compute_cost` adds an `embed` branch (input-only, completion always 0,
native OpenRouter `cost` still preferred) and `Pricing` gained `embed_input_per_m`. `analytics.py`
groups by `backend` generically, so embed rows roll up automatically; `caching_savings` stays
chat-only. When `gateway_embed_enabled` is off the endpoint returns 503 with no ledger row.

#### Single point of truth for AI
The gateway is the only place that holds real provider URLs/models/keys. Every other AI
consumer points at it by default and never touches a provider key:
- Defaults live in `config.py`: `INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`,
  override with `DEVPLACE_INTERNAL_BASE_URL`) and `INTERNAL_MODEL` (`molodetz`). `news_ai_url`,
  `bot_api_url`, `devii_ai_url` default to `INTERNAL_GATEWAY_URL`; their model defaults to `molodetz`.
- Each consumer's key falls back to `database.internal_gateway_key()` (reads the
  `gateway_internal_key` setting) when its own key field/env is unset - the provider-key
  fallbacks (`DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY`) were removed from news and bots.
- `gateway_force_model` (default on) and a `molodetz`/empty alias in `handle_chat` make the
  upstream always receive `gateway_model`, so `molodetz` is a stable generic alias.
- `database.migrate_ai_gateway_settings()` (called at the end of `init_db()`, under the startup
  `init_lock`): generates `gateway_internal_key` (uuid4) if missing; migrates `DEEPSEEK_API_KEY`/
  `OPENROUTER_API_KEY` env into `gateway_api_key`/`gateway_vision_key` when the db value is empty;
  and rewrites any consumer AI URL still equal to the old `openai.app.molodetz.nl` default to the
  gateway, plus `bot_model` `deepseek-chat` -> `molodetz` (only uncustomized values).
- The gateway's `gateway_api_key`/`gateway_vision_key`/`gateway_internal_key` fields are
  **non-secret** so the admin services page shows the value actually in use, editable.
- Bot LLM calls are synchronous `urllib` but already run via `asyncio.to_thread`
  (`bot/bot.py`), so the local round-trip never blocks the event loop.

### `DeviiService` (`services/devii/`)

In-platform agentic assistant relocated from a standalone project. A WebSocket terminal at
`/devii/ws` (`routers/devii.py`, prefix `/devii`), opened from the **Devii** dropdown item and
auto-opened on `/docs`. Opt-in (`default_enabled=False`).

- **Account scope.** A signed-in user's Devii drives **their own** account: the agent's
  `PlatformClient` uses this instance as base URL and the user's `api_key` as Bearer auth, so
  admins get admin-level tools. The **LLM** calls use the same `api_key` too: the hub threads
  `owner_kind` into `_build_settings` -> `build_settings(cfg, base_url, api_key, owner_kind)`, which
  sets `Settings.ai_key = api_key` for `owner_kind == "user"` (else the configured/internal gateway
  key). So a user's Devii spend is attributed to them at the gateway (`user:<uid>`) and counts toward
  any per-user gateway limit, while guests fall back to the internal key. Guests get an unauthenticated
  sandbox client. Every turn is
  audited in `devii_turns`.
- **Sessions & persistence.** `DeviiHub` keeps one `DeviiSession` per owner-and-channel
  `(owner_kind, owner_id, channel)` - `user`/uid or `guest`/`devii_guest` cookie, channel
  `main` (default) or `docs`. A session holds a `set` of WebSockets and **broadcasts** every
  frame to all of that owner-channel's tabs/devices. After a turn it persists `agent._messages`
  to `devii_conversations` (users only; rehydrated on reconnect → survives restarts), appends a
  `devii_usage_ledger` row, and a `devii_turns` audit row. On `attach` it sends a
  `{"type":"history"}` snapshot so a new tab renders the prior conversation. Tasks persist in
  `devii_tasks` (owner-scoped; guests use an in-memory db via `tasks.store.memory_db()`).
- **Channels (independent conversation thread only).** A `channel` separates the floating
  terminal (`main`) from the in-page docs chat (`docs`) so each has its own conversation thread
  for the same owner. The WS reads `?channel=` (`routers/devii.py`, allowlisted to
  `{"main","docs"}`, falls back to `main`) and threads it `get_or_create(..., channel=)` →
  `DeviiSession(channel=)`. **Only** `devii_conversations` is keyed `(owner_kind, owner_id,
  channel)` (column added + NULL→`main` backfilled + index extended in `init_db`); lessons,
  behavior, virtual tools, tasks, and the `devii_usage_ledger`/`devii_turns` 24h quota stay
  owner-scoped (deliberately shared across channels). `find(...)`/`get_or_create(...)` default
  `channel="main"` so existing callers (e.g. `/devii/adopt`) are unchanged. The `docs` channel
  is driven only by `<dp-docs-chat>` (`static/js/components/AppDocsChat.js`): a light-DOM
  component that calls `/devii/session` (guest cookie), opens `DeviiSocket(handlers, "docs")`,
  renders user/agent bubbles in docs style (`static/css/docs-chat.css`) via the devii markdown
  renderer, stubs any `avatar`/`client` request so the agent never hangs, and seeds its first
  question from the `seed` attribute (the sidebar search box, intercepted). It is mounted on
  `/docs/search.html` (`templates/docs/_chat.html`); the docs pages no longer carry
  `data-devii-autoopen`.
- **Docii (the `docs` channel persona).** The `docs` channel is a documentation-only assistant
  branded **Docii**, built by specialising the same `DeviiSession` on `self.channel == "docs"`
  (no separate service):
  - **Prompt:** `session.py` sets `self._system_prompt = DOCS_SYSTEM_PROMPT` (instead of
    `_system_prompt_for`) - it forces the agent to call `search_docs` first for every question,
    to recursively refine-and-search (reading the returned section `content`, then searching
    again with new keywords) until grounded, to answer ONLY from retrieved docs, and to do no
    platform actions. `_compose_system_prompt()` returns it verbatim for docs (the owner
    `BEHAVIOR_HEADER` is NOT appended, so a user's general Devii behavior rules can't derail it).
  - **Tools:** `_builtin_tools()` filters `CATALOG.tool_schemas_for(...)` to the `DOCS_TOOLS`
    allowlist (`{"search_docs"}`); `_refresh_tools()` for docs sets `self.tools[:] = _builtin_tools()`
    with NO virtual tools. So the docs agent has exactly one tool and cannot wander.
  - **Greeting:** `bootstrap_greeting()` returns `DOCS_GREETING` for docs.
  - `search_docs` (`services/devii/docs/controller.py`) now delegates to the in-process page
    index `docs_search.search_pages(...)` (no HTTP), returning per result `title`, `url`
    (`/docs/{slug}.html`), `score` (BM25), and `content` (page text truncated to `CONTENT_CHARS`),
    admin-filtered by the dispatcher's `is_admin`. So "visiting the search results" is repeated
    `search_docs` calls; the default 40 `max_tool_iterations` leaves ample room to recurse.
  - **References + inline linking:** because results carry real `url`s and `score`s, the prompt
    REQUIRES the answer to (a) link inline to documented pages with markdown links to their `url`
    and (b) end with a `## References` list of the pages used as clickable links with their score.
    The devii markdown renderer makes `/docs/...` links clickable client-side.
  - **Topic gate (follow-up vs new):** `_run_turn` runs `_docs_topic_gate(text)` before the main
    loop (docs channel only). When there is prior history it calls `_classify_topic` (a no-tools
    `LLMClient.complete_text` with `TOPIC_CLASSIFIER_PROMPT`, parsed by `_parse_topic`, defaulting
    to `follow_up` on any failure). On `new` it resets `agent._messages` to `[system]`, clears the
    persisted docs conversation, and emits `{type:"topic", decision, reason}`; the frontend clears
    the log, re-shows the current question, and prints a reasoning note. On `follow_up` it keeps
    context and shows the note.
  - **Isolation from Devii (critical):** the docs channel is built with an EPHEMERAL `owned_db`
    (`hub.get_or_create`: `db if owner_kind=="user" and channel=="main" else memory_db()`), so its
    task/lesson/behavior/virtual-tool stores are private and empty - a Docii reflection never
    pollutes the user's Devii memory and vice-versa. The **Scheduler only starts in the `main`
    channel** (`attach`: `if not self._started and self.channel == "main"`), so Devii's scheduled
    tasks never fire inside a docs session. (Regression fixed: previously the docs session's
    scheduler ran owner tasks over the shared `devii_tasks` table, and the search-only Docii agent
    tried to fulfil a `run_js` task by spamming `search_docs` - tasks are a `main`-only feature.)
  - The `main` channel is untouched (full 90-tool catalog + behavior header + Devii greeting).
- **Docs search mode + BM25 fallback.** The docs search surface is admin-configurable via the
  `docs_search_mode` site setting (`agent` | `bm25`, default `agent`, on `/admin/settings`,
  seeded in `init_db` `operational_defaults`, field in `AdminSettingsForm`). `routers/docs/views.py`
  `docs_page` (slug `search`) resolves the effective mode: `_agent_search_state(request, user,
  is_admin)` returns `disabled` (Devii off, or guests-disabled for a guest), `quota` (viewer
  over their `daily_limit_for`/`spent_24h`), or `ok`. Agent renders **only** when
  `mode=="agent"` and state `ok`; otherwise it runs the classic `docs_search.search(...)` (BM25,
  `docs_search.py`) and renders `templates/docs/_search.html`. `docs_base.html` branches the
  `kind=="search"` include on `search_mode`; on a quota fallback (`quota_fallback`) the BM25 page
  shows a "reached your daily AI assistant limit" hint. So `docs_search.py`/`_search.html` are
  live again (the bm25 path), not orphaned.
- **Per-owner self-learning memory (privacy-critical).** The `LessonStore` (reflect/recall) is
  **owner-scoped, never shared**: `LessonStore(db, owner_kind, owner_id)` filters every read/write
  by owner. The hub builds one per session over the same `owned_db` as the task store - the main
  `db` (table `devii_lessons`) for signed-in users (persistent, isolated, survives restarts) and a
  fresh `memory_db()` for guests (ephemeral, scoped to that web session). `forget_lessons` (agentic
  tool → `LessonStore.clear()`/`delete()`) lets the user purge them; the system prompt forbids
  storing credentials/secrets in lessons. Regression to avoid: do NOT share one `LessonStore`
  across sessions - that leaked one user's reflected lessons (including credentials) into every
  other user's recall.
- **Aggregate analytics (no pagination).** Site analytics is a **documented, admin-secured HTTP
  endpoint** - `GET /admin/analytics` (`routers/admin/ package`, returns JSON, `is_admin` check → 403
  otherwise) backed by `database.get_platform_analytics()` (single UNION query over
  `posts/comments/gists/projects.created_at`, TTL-cached 60s; total members, active users
  24h/7d/30d, signed-in-now, signups, content totals, top authors). Devii calls it as a normal
  `http` catalog action `site_analytics` (GET `/admin/analytics`, `top_n` query param) - the
  platform enforces admin, so it behaves identically for the web and the `devii` CLI and never
  touches the DB directly from the agent. It is documented in the Admin API docs group
  (`docs_api.py`, id `admin-analytics`). The system prompt tells the model to use it for any "how
  many"/"how active" question instead of paging `admin_list_users` (which caused a pagination
  storm), and to `search_docs` before guessing a route rather than probing URLs.
- **Shared browser session.** The terminal and the browser are one session. `/devii/ws` resolves
  its owner from the browser's `session` cookie (`_resolve_ws_owner`), so the terminal is whoever the
  browser is logged in as. Agent-initiated auth propagates: the session watches its own trace stream
  (`_trace`) and, on a successful `login`/`signup`, captures the real session token the
  `PlatformClient` minted against this instance (`session_cookie()`) and broadcasts
  `{"type":"auth","action":"adopt"}`; the browser navigates to single-use `GET /devii/adopt` which
  sets the httpOnly `session` cookie and redirects (reload). On `logout` it broadcasts
  `{"type":"auth","action":"logout"}` and the browser goes to `/auth/logout`. Browser→terminal:
  login/logout are navigations that reconnect the WS; a cross-tab change is caught by a focus check
  against `/devii/session` that reloads. The httpOnly cookie is only ever set/cleared by HTTP
  endpoints, never JS.
- **Financial cap.** The rolling-24h `$` limit is read from `devii_usage_ledger`
  (`UsageLedger.spent_24h`), **not** the in-memory `CostTracker` (which resets on reboot and is
  display-only). Checked before each turn in the router; a turn already over is blocked (the
  over-limit WS error states only "Daily AI quota reached (100%)", never a dollar figure). Pricing
  is admin-configurable - `cost/tracker.py` resolves a `Pricing` object per `CostTracker` rather
  than import-time env constants.
- **Financial data is admin-only.** Any monetary figure (USD cost, pricing, spend, limit) is
  restricted to administrators; members and guests may see only the **percentage** of their 24h
  quota used. The owner's admin status is resolved server-side in `routers/devii.py`
  (`is_admin(user)` from the session/API-key user) and threaded WS → `hub.get_or_create(is_admin=)`
  → `DeviiSession(is_admin=)` → `Dispatcher(is_admin=)`. The `Action` dataclass has a
  `requires_admin` flag; `cost_stats` (full USD breakdown) is `requires_admin=True`, while the new
  `usage_quota` tool returns **only** `{used_pct, turns_today, limit_reached}` with no money and is
  available to everyone. Gating is double: `Catalog.tool_schemas_for(authenticated, is_admin)` never
  hands an admin-only tool's schema to a non-admin, and the dispatcher independently raises
  `AuthRequiredError` for any `requires_admin` action a non-admin attempts. `usage_quota`'s data
  comes from `DeviiSession._quota_snapshot` (ledger `spent_24h`/`turns_24h` over
  `settings.daily_limit_usd`); for non-admin owners the system prompt also appends a hard rule
  forbidding any cost disclosure (defense in depth). `GET /devii/usage` mirrors this: it always
  returns `used_pct`/`turns_today` and adds `spent_24h`/`limit` only for admins. The standalone
  `devii` CLI runs with `is_admin=True` (the local operator owns the process). The catalog's
  cost/analytics HTTP tools `ai_usage` (GET `/admin/ai-usage/data`, USD breakdown) and
  `site_analytics` (GET `/admin/analytics`) are also `requires_admin=True`, so a non-admin session is
  never offered them and the dispatcher blocks them even if named - the platform 403 is no longer the
  only guard. The profile page is correctly split too (`_ai_quota(include_cost=viewer_is_admin)`
  emits dollars only for admins, in both its HTML and JSON forms).
- **Context-window sizing (DeepSeek V4 Flash, 1M tokens).** The upstream model `deepseek-v4-flash`
  (what `deepseek-chat` routes to; default `gateway_model`) has a 1,048,576-token context and
  384,000-token max output. All Devii size limits are **characters**, derived in
  `services/devii/config.py` from one source of truth: `CONTEXT_INPUT_BUDGET_TOKENS =
  CONTEXT_WINDOW_TOKENS - MAX_OUTPUT_TOKENS - SYSTEM_RESERVE_TOKENS` (600,576 tokens), and
  `DEFAULT_CONTEXT_COMPACT_THRESHOLD = CONTEXT_INPUT_BUDGET_TOKENS * CHARS_PER_TOKEN` (3 chars/token
  = ~1.8M chars). At the conservative 3-chars/token estimate the worst-case input at compaction plus
  the full max output plus the system reserve sums to exactly the 1M window, so it can never
  overflow. `DEFAULT_MAX_RESPONSE_CHARS` (200,000) is the master chunk knob: every non-`chunks`
  tool result passes through `wrap_if_large(result, max_response_chars)` in `actions/dispatcher.py`,
  and `read_more` slices are clamped to it, so a file up to ~200KB returns in **one** read instead of
  paging in 12KB slices (the old read_more storm). `OUTPUT_CAP_CHARS` (`agentic/loop.py`, 400,000)
  must stay **above** `max_response_chars` or it would re-truncate a full chunk envelope. `ChunkStore`
  caches up to `STORE_MAX_CHARS` (8MB) per entry, `STORE_MAX_ENTRIES` (16) entries; `SUMMARY_INPUT_CAP`
  (`agentic/compaction.py`, 600,000) bounds what the summarizer ingests when compacting. When changing
  the upstream model, retune from `CONTEXT_WINDOW_TOKENS`/`MAX_OUTPUT_TOKENS` - everything else derives.
- **Multi-worker.** Hubs are per-process, so `/devii/ws` is served **only** by the worker that
  holds the background-service lock (`service_manager.owns_lock()`, set in `main.py` startup);
  a non-owner worker `close(4013)` (an application code in the private 4000-4999 range, reliably
  delivered as the browser `CloseEvent.code`). `DeviiSocket.js` recognises 4013 as "wrong worker,
  not yet settled" and fast-retries in ~200ms **silently** (no "disconnected, reconnecting..."
  notice), so with 2 prod workers the client converges on the owner in a fraction of a second
  instead of bouncing every 1500ms. The visible "connected." notice is emitted on the first server
  frame (`onReady`), never on raw socket open, so an accepted-then-4013-closed handshake on the
  wrong worker is invisible. The disabled/no-service path keeps the standard `1013` (a real,
  user-visible disconnect). The cap stays correct regardless because it reads the DB.
- **Client-side tool channel.** Beyond the avatar, Devii has a `client`/browser channel using the
  same request/response pattern: `services/devii/client/ClientController` is bound on `attach`, and
  `actions/client_actions.py` exposes `get_page_context`, `run_js`, `highlight_element`,
  `clear_highlights`, `show_toast`, `scroll_to_element`, `navigate_to`, `reload_page`
  (`handler="client"`, all `requires_auth=False`). The session's generic
  `_browser_request(channel, action, args)` powers both `avatar` and `client`; the router resolves
  both `avatar_result` and `client_result`. Frontend `static/js/devii/DeviiClient.js` executes the
  actions in the user's own browser and the terminal routes `type:"client"` frames to it, returning
  `client_result`. `navigate_to`/`reload_page` reply *before* unloading so the turn completes, then
  the persistent session reconnects.
  **Target selection (visibility-aware).** Unlike replies/traces, which `_emit` *broadcasts* to
  every tab, a browser request is sent to one **target** chosen by `_pick_target()`. The terminal
  reports each connection's `document` visibility and focus via `{"type":"visibility"}` (on connect,
  `focus`, `blur`, and `visibilitychange`); the session stores it in `_conn_meta` and ranks
  connections `(focused, visible, attach_seq)`, so the command runs on the tab the user is actually
  looking at, falling back to the most recently attached when none reports focus. Regression to
  avoid: do NOT route to a single "last-attached primary" socket - with the `/docs` auto-open tab or
  any second tab, `reload_page`/`navigate_to` ran on the wrong (hidden) tab while still reporting
  success, so "the reload did not happen" from the user's view. `run_js` is gated by the `devii_allow_eval` config field
  (default on), checked in `ClientController` before any round-trip. Overlays (`.devii-hl-box`,
  `.devii-hl-callout`, `.devii-toast`) attach to `document.body`, not the terminal.
- **Web fetch and arbitrary HTTP (`handler="fetch"`).** `services/devii/fetch/FetchController` (a standalone `httpx.AsyncClient`, not `PlatformClient`) backs two tools: `fetch_url` (read a page as readable text, `requires_auth=False`) and `http_request` (`requires_auth=True`) - the general external HTTP client. `http_request` takes `method` (GET/POST/PUT/PATCH/DELETE/HEAD/OPTIONS), `headers` (object), and one body of `json` (object/array, Content-Type set automatically), `form` (url-encoded), or `body` (raw string), and returns `{http_status, headers, content_type, content}`; it does **not** raise on non-2xx so the model can read API errors. Both run through one `_stream()` (`fetch_url` via `_download` with `raise_5xx=True`; `http_request` with `raise_5xx=False`) sharing the SSRF `_guard` (refuses private/loopback unless `DEVII_FETCH_ALLOW_PRIVATE`), `fetch_max_bytes` cap, timeout, and decoding. The `object`-typed params required adding an `object` branch to `Action.tool_schema()` in `spec.py` (emits a valid open-object schema). The `SYSTEM_PROMPT` "WEB REQUESTS AND EXTERNAL APIS" rule tells the agent to call `http_request` DIRECTLY for an API and never wrap it in a self-invoking user-defined tool (a virtual tool only re-runs the agent and cannot do I/O, so wrapping the call recurses into the `MAX_EVAL_DEPTH` guard - the exact failure this fixed). To attach a remote file see `attach_url` / `store_attachment_from_url` under **Attachments and media**.
- **Remote web tools (rsearch, non-platform).** `services/devii/rsearch/RsearchController`
  (`handler="rsearch"`, registered in `registry.py`, specs in `actions/rsearch_actions.py`, all
  `requires_auth=False`) exposes four tools that hit the EXTERNAL public service
  `https://rsearch.app.molodetz.nl` over a standalone `httpx.AsyncClient` (like `FetchController`,
  not the platform-bound `PlatformClient`): `rsearch` (web/image search, optional `content`/`deep`/
  `type=images`), `rsearch_answer` (AI answer grounded in fresh web search, returns answer + sources),
  `rsearch_chat` (direct AI chat, no search), `rsearch_describe_image` (vision describe a public image
  URL). These are NOT platform-specific: the `SYSTEM_PROMPT` "REMOTE WEB TOOLS" section makes the model
  prefer platform tools always and call `rsearch_*` only when the user explicitly asks to search the
  web/an outside source, never to answer questions about this instance. When adding another remote
  service, add a controller + handler literal in `spec.py` + a dispatcher branch rather than routing
  external calls through `PlatformClient` (which is hardwired to the instance base URL).
  rsearch has its **own** read timeout (`rsearch_timeout_seconds`, field `devii_rsearch_timeout` /
  env `DEVII_RSEARCH_TIMEOUT`, default **300s** with a 30s connect cap via `httpx.Timeout`) - it does
  NOT share `fetch_timeout_seconds`, because web-grounded answers legitimately take minutes.
- **All Devii/gateway network timeouts are admin-configurable with a five-minute (300s) floor.** The
  Devii LLM-client and `PlatformClient` read timeout is `timeout_seconds` (field `devii_timeout`),
  the fetch/docs tool timeout is `fetch_timeout_seconds` (field `devii_fetch_timeout`), web search is
  `rsearch_timeout_seconds` (`devii_rsearch_timeout`); each defaults to 300s with `minimum=MIN_TIMEOUT_SECONDS`
  (300). The gateway upstream timeout is `gateway_timeout` (`minimum=TIMEOUT_MIN`=300), which also bounds
  the vision describe-image call (it shares the gateway's httpx client, no per-call override). A stored
  value below the floor fails `ConfigField.coerce()` and `read()` falls back to the default, so old
  sub-300 values upgrade automatically. Devii's LLM timeout was previously a hardcoded 45s while the
  gateway waited up to 180s x retries - large prompts aborted as `[model error] Could not reach the
  model endpoint:` (an httpx timeout, which stringifies to empty). `build_settings()` now reads these
  from config instead of hardcoding them.
- **Config** is all `config_fields` (AI url/model/key, base url, plan/verify toggles, max
  iterations, JS-execution toggle, user/guest 24h caps, guests toggle, pricing). `effective_config()`
  falls back the AI key to `DEVII_AI_KEY` and the base url to the instance origin.
- **Backend confidentiality.** Devii must never disclose the underlying model, provider, or any
  upstream URL - enforced in two layers. The `SYSTEM_PROMPT` (`agent.py`) forbids it even when such
  values appear inside a tool result. Structurally, `text.redact_backend()` runs on every JSON HTTP
  tool response in `format_response()` and scrubs the values of `REDACT_FIELD_KEYS`
  (`gateway_upstream_url`/`gateway_model`/`gateway_vision_url`/`gateway_vision_model`) and any stat
  labelled in `REDACT_STAT_LABELS` (`Model`) to `[hidden]`, so the admin-services tools cannot leak
  the gateway's upstream even though the admin settings UI still shows the real values. Add a config
  key to `REDACT_FIELD_KEYS` if a new field would expose backend infrastructure to the agent.
- **Screen awareness.** The `SYSTEM_PROMPT` instructs Devii, after a mutation, to `get_page_context`
  and `reload_page` when the page the user is viewing displays the data just changed, so settings/list/
  detail views the user is watching update live without a manual refresh.
- **CLI.** `pyproject.toml [project.scripts]` ships `devii = "devplacepy.services.devii.cli:main"`.
  No new dependency (`httpx`/`dataset` already required). The md-clippy avatar is vendored under
  `static/vendor/md-clippy/`; its `index.js` must not set `globalThis.app` (it would clobber the
  DevPlace `app`) and its AI proxy is `/devii/clippy/ai/chat`.

### Database tables

| Table | Purpose |
|-------|---------|
| `news` | All synced articles with `status` (published/draft), `grade`, `slug`, `show_on_landing` |
| `news_images` | Images extracted from article URLs |
| `news_sync` | Sync state per article `guid` - tracks grading history |

### Site settings (seeded on startup)

| Key | Default | Purpose |
|-----|---------|---------|
| `site_name` / `site_description` / `site_tagline` | DevPlace branding | General site metadata |
| `news_grade_threshold` | `"7"` | Minimum AI grade for auto-publish |
| `news_api_url` | `"https://news.app.molodetz.nl/api"` | News source API |
| `news_ai_url` | `"https://openai.app.molodetz.nl/v1/chat/completions"` | AI grading endpoint |
| `news_ai_model` | `"molodetz"` | AI model identifier |
| `max_upload_size_mb` / `allowed_file_types` / `max_attachments_per_resource` | `"10"` / `""` / `"10"` | Upload limits |
| `rate_limit_per_minute` | `"60"` | Mutating requests per IP per window (`main.py` middleware); a `429` carries a `Retry-After: <window>` header |
| `rate_limit_window_seconds` | `"60"` | Rolling window for the rate limit (also the `Retry-After` value) |
| `news_service_interval` | `"3600"` | Seconds between news fetch cycles (`NewsService.run_once` re-reads each cycle) |
| `session_max_age_days` | `"7"` | Standard session cookie + DB session lifetime |
| `session_remember_days` | `"30"` | Remember-me session lifetime |
| `registration_open` | `"1"` | When `"0"`, signup GET shows a closed notice and POST is rejected (`auth.py`) |
| `maintenance_mode` | `"0"` | When `"1"`, non-admins get a 503 (`main.py` maintenance middleware) |
| `maintenance_message` | scheduled-maintenance text | Body shown on the maintenance 503 page |
| `customization_enabled` | `"1"` | When `"0"`, `custom_css_tag`/`custom_js_tag` inject nothing (feature off) |
| `customization_js_enabled` | `"1"` | When `"0"`, custom CSS still serves but custom JS is suppressed |

The seed block in `database.py` is guarded by `if "site_settings" in tables:` - on a brand-new DB the table does not exist yet (dataset creates tables lazily on first insert), so none of these rows are written until the table exists. Correct runtime behavior therefore relies on every consumer passing the production default to `get_setting`/`get_int_setting`, not on the seed.

### Operational settings - read sites and rules

| Setting(s) | Read at | Notes |
|-----------|---------|-------|
| `rate_limit_*` | `rate_limit_middleware` in `main.py` | `max(1, get_int_setting(...))` so `0` can't block all writes |
| `maintenance_mode` / `maintenance_message` | `maintenance_middleware` in `main.py` | Allows `/static`, `/avatar`, `/auth`, `/admin` and admins; everyone else gets `error.html` at 503 |
| `news_service_interval` | `BaseService` reconciling loop via `current_interval()` | `max(60, …)`; edited on the Services tab (not `/admin/settings`); a change applies on the next cycle |
| `service_<name>_enabled` / `service_<name>_command` / `service_<name>_log_size` | `BaseService` reconciling loop | Generic per-service controls written by the Services tab; the loop reconciles within ~1s |
| `session_max_age_days` / `session_remember_days` | `auth.py` signup + login | Multiplied by `SECONDS_PER_DAY`; passed to `create_session(uid, max_age)` so the cookie and the DB session row expire together |
| `registration_open` | `auth.py` `signup_page` (GET) and `signup` (POST) | POST returns before any DB write when closed |

**Booleans are `<select>`, never checkboxes.** The settings save handler (`admin.py`) skips empty form values so empty fields don't clobber existing rows. An unchecked checkbox submits nothing, so it could never be turned off - `registration_open` and `maintenance_mode` use `<option value="1">`/`<option value="0">` so a value is always submitted.

### Adding a new service

1. Create `devplacepy/services/your_service.py` with a class extending `BaseService`. Declare `config_fields` (and set `interval_key`/`min_interval` if reusing an existing key); read them in `run_once` via `self.get_config()`.
2. Override `async def run_once(self) -> None`.
3. Register in `main.py` startup event (outside the lock block, so all workers know the roster):
   ```python
   from devplacepy.services.your_service import YourService
   service_manager.register(YourService())
   ```
4. The service appears automatically on `/admin/services` with start/stop, enable-on-boot, run-now, a generic config form, clear-logs, live status, and the log tail - no template, JS, or router changes needed.

### CLI

```bash
devplace news clear                     # Delete all news from local database
devplace devii reset-quota <username>   # Reset one user's rolling 24h AI quota
devplace devii reset-quota --guests     # Reset every guest quota
devplace devii reset-quota --all        # Reset every quota (users and guests)
```

## Signals Category

The `signals` topic is available as a feed filter sidebar item and post topic. Added CSS variable `--topic-signals: #00bcd4` in `variables.css`, badge class in `base.css`, and dot color in `feed.css`. Allowed in `posts.py` topic validation.

## Date Format

All dates displayed to users use European DD/MM/YYYY format. Implemented via:

- **`format_date(dt_str, include_time=False)`** in `utils.py` - converts ISO datetime → `DD/MM/YYYY` or `DD/MM/YYYY HH:MM`
- Registered as template global in `templating.py`: `{{ format_date(dt) }}`
- **`time_ago()`** returns `DD/MM/YYYY` for items older than 30 days (instead of `"Xmo ago"`)
- Services page has a JS `formatDate()` function for live polling updates

## Admin Pagination

Both `/admin/users` and `/admin/news` use offset-based pagination via a reusable component:

- **`templates/_pagination.html`** - numbered page links with ellipsis, Previous/Next buttons, total count
- Routes accept `?page=N` query param, clamped to valid range
- `per_page = 25`, pagination metadata computed server-side and passed as `pagination` dict
- Only renders when `total_pages > 1`
- CSS in `admin.css` (`.pagination`, `.pagination-btn`, `.pagination-page`, `.pagination-ellipsis`)

## News Detail & Comments

News articles have an internal detail page at `/news/{slug}` with full comment support:

- **Route:** `GET /news/{news_slug}` in `routers/news.py` - resolves by slug first, then UUID
- **Template:** `templates/news_detail.html` - shows image, source, grade, description, content, external link
- **Comments:** Uses `_comment_section.html` with `target_type="news"` - same component as posts/projects
- **`resolve_target_redirect()`** in `comments.py` handles `"news"` → `/news/{slug}`
- Listing links in `news.html` point to internal detail page; "Read on Source" still goes to external URL

## Home page (`GET /`)

The home route (`main.py` `landing()`) never redirects - it renders `templates/landing.html` for everyone, branching on `user`:

- **Guests** get the marketing hero (`Join DevPlace Free` CTA + features grid).
- **Signed-in users** get a personalized hero (`.landing-hero-user`): avatar, "Welcome back, {username}", a `Go to your feed` CTA, a Posts/Stars/Level stat strip (`user_post_count` + the user dict's `stars`/`level`), and quick links. Styles live in `.landing-hero-user`/`.landing-welcome`/`.landing-stats`/`.landing-quicklinks` in `static/css/landing.css`.
- Both states share the Latest Posts + Developer News + "Build With Us" sections. The **Build With Us** section is static HTML/CSS (`.landing-help-*` in `landing.css`): four cards linking to `/docs/index.html` (Documentation), `/swagger` + `/openapi.json` (API Reference), `/issues` (Contribute & Report), and Devii. The Devii card's `Launch Devii` button is a plain `<button data-devii-open>` that opens the globally mounted `DeviiTerminal` (`app.devii`) in place - no extra JS, route, schema, or Devii action; a secondary link points to `/devii/` for the full terminal page.
- Context adds `user`, `is_authenticated`, `user_post_count`; `LandingOut` carries `is_authenticated`/`user_post_count` for the JSON form. `GET /` is documented in `docs_api.py` (id `home`, mapped to `LandingOut`).

### Author diversity (interleave, never drop)

Both the home page Latest Posts section and the main `/feed` *interleave* authors so one prolific account cannot fill a contiguous run, **without dropping any post** (the old per-author cap is gone). Helpers in `database.py`:

- `interleave_by_author(rows, uid_key="user_uid")` - pure reordering of a row list. Greedy and order-preserving: it walks the list and at each step emits the earliest row whose author differs from the last emitted one (deferring a same-author run to the next available author, recursively); when only same-author rows remain it emits them in order. The result is a permutation of the input - never a subset - so each author's own posts keep their relative (date) order and no two consecutive rows share an author unless the whole list is one author. The home route fetches the 6 newest posts and interleaves them.
- `paginate_diverse(table, *clauses, ..., uid_key="user_uid", **filters)` - a drop-in replacement for `paginate` (same `(rows, next_cursor)` contract) that calls `paginate` then `interleave_by_author` on the page. Because the interleave is a pure permutation of the page, the cursor (oldest `created_at` in the page) is identical to `paginate`'s, so infinite scroll stays correct with no gaps/overlaps. `feed.py` `get_feed_posts` uses it for every tab.

When building any new "recent items" list, reuse these instead of raw `find(order_by=...)` / `paginate`.

## Listing search (feed, gists, projects)

The three public listings - `/feed`, `/gists`, `/projects` - share one free-text search box at the top of the left filter panel. Two reuse points keep it DRY and consistent:

- **Data layer:** `database.text_search_clause(table, search, fields=("title", "description"))` returns a single SQLAlchemy `or_(col.ilike("%term%") ...)` clause over the given columns (skipping any not present on the table), or `None` when `search` is blank or the table does not exist yet. Append it as a positional clause to `paginate` / `paginate_diverse` / `table.count(...)` **only when not None** (no search leaves the queries unchanged). Field mapping: projects/gists use `("title", "description")`, posts use `("title", "content")`. `projects.py`, `gists.py`, and `feed.py` all call it; never re-inline an `ilike` search clause.
- **View layer:** `templates/_sidebar_search.html` is the single search-box partial (the `Filter` heading + the GET form). Include it at the very top of `<aside class="sidebar-card">` with three locals: `_action` (the listing URL), `_placeholder`, and `_hidden` (a dict of `name -> value` rendered as hidden inputs so the active category/tab filter survives a search submit - e.g. `{"tab": current_tab, "topic": current_topic}` for the feed). It reads the `search` context var for the current value. Any future listing with a left filter panel reuses this partial plus `text_search_clause`; do not hand-roll another search form.

The `search` value is threaded back into each listing's context and exposed on `FeedOut.search` / `GistsOut.search` / `ProjectsOut.search`, documented as a `search` query param on the `feed-list` / `gists-list` / `projects-list` endpoints in `docs_api.py`, and offered to Devii as the `search` query param on `view_feed` / `list_gists` / `list_projects`.

## Recent comments on listing/feed cards (consistent across post, gist, project, news)

Every public listing card that shows the last few comments must render them with the **same visual hierarchy as the detail page** (depth indentation, vote rail, reply nesting). One pipeline drives all four surfaces - never inline a flat-only render:

- **Data layer:** `database.get_recent_comments_by_target_uids(target_type, target_uids, limit=3, user=None)` is the single batch helper. It runs one `ROW_NUMBER() OVER (PARTITION BY target_uid ORDER BY created_at DESC, id DESC)` window over `comments` filtered by `target_type=:tt` + `deleted_at IS NULL`, builds each item via `_build_comment_items` (identical dict shape to the detail page: `comment`/`author`/`time_ago`/`votes`/`my_vote`/`children`/`attachments`/`reactions`), then **nests the limited set by `parent_uid` into `children`** so replies render indented. Returns `{target_uid: [top-level items]}`. `get_recent_comments_by_post_uids` is now a thin wrapper delegating with `target_type="post"`, so `feed.py` is unchanged and there is no behavior drift.
- **Routers:** `feed.py` (post), `gists.py` (gist), `projects/index.py` (project), `news.py` (news) each batch-fetch with the helper and attach `recent_comments` per item. Gists and news are wrapper dicts (`item["recent_comments"]`); projects are **flat dicts** (`project["recent_comments"]`).
- **Schemas:** `FeedItemOut`, `GistItemOut`, `NewsListItemOut`, and the flat `ProjectListItemOut` all carry `recent_comments: list[CommentItemOut] = []`, so the `respond(model=...)` JSON exposes the same nested tree and never silently drops the key.
- **View:** each card renders the block with the shared `_comment.html` `render_comment(c, 0)` macro inside a `.post-card-comments` wrapper (copied from `_post_card.html`). The macro recurses on `item.children` for indentation. **CSS requirement:** the template must load `post.css` (the `.comment`/`.comment-depth`/`.comment-replies` hierarchy) and `feed.css` (the `.post-card-comments` wrapper). `feed.html`/`gists.html`/`projects.html` already loaded `feed.css`; `news.html` loads neither by default, so it loads both.

## Landing Page News

Articles can be toggled to appear on the landing page via `/admin/news/{uid}/landing`:

- **`show_on_landing`** field on `news` table
- Landing route (`main.py` `GET /`) fetches up to 6 articles with `show_on_landing=1`
- Rendered as a 3-column card grid with image, source, title, date (responsive → 1 column on mobile)
- Toggleable individually from the admin news table

## Public Feed

The feed page (`GET /feed`) is accessible without authentication:

- Uses `get_current_user(request)` instead of `require_user()` - returns `None` for guests
- Guests see posts but not the FAB, create modal, inline comment forms, or following tab
- All POST routes (create, comment, vote) remain guarded by `require_user()`
- Topnav shows Login/Sign Up for unauthenticated visitors; Messages, Admin, notifications for authenticated

### Step 2: Implement backend

- Add/modify the route in `routers/{area}.py`
- Validate form input with a typed `Annotated[Model, Form()]` param (define the model in `models.py`); read raw `await request.form()` only for file uploads
- Use `templates.TemplateResponse(...)` from `devplacepy.templating`
- Redirect with `RedirectResponse(url=..., status_code=302)`
- Log every action: `logger.info(...)`
- New DB fields auto-sync via `dataset` - just add to the insert/update dict
- For ownership checks: `if resource["user_uid"] == user["uid"]`
- For deletion: cascade related data first (comments → votes → post)
- Register new routers in `main.py`: `app.include_router(router, prefix="/{path}")`

### Step 3: Implement frontend

- Template in `templates/`, CSS in `static/css/`, JS in `static/js/Application.js`
- Load CSS via `{% block extra_head %}` with `<link rel="stylesheet">`
- Use `{% extends "base.html" %}` and `{% block content %}`
- Jinja2 globals: `avatar_url()`, `get_unread_count()`, `get_user_projects()`
- For clickable avatars: `{% set _user = ... %}{% include "_avatar_link.html" %}`
- For rendered content: add `class="rendered-content"` and `data-render` attribute
- No NPM, no frameworks - pure ES6 modules

### Step 4: Validate code

Validate each touched language manually and confirm a clean import:

```bash
python -c "from devplacepy.main import app"
```

Zero errors required (Python compiles/imports, JS parses, CSS braces and HTML tags balance).

### Step 5: Run existing tests (only if asked by user)

```bash
make test
```

All tests must pass. Tests stop at first failure (`-x`).

### Step 6: Write new tests

- Add tests in the matching tier/path (`tests/{unit,api,e2e}/<path>.py`)
- Use `alice` for authenticated sessions, `bob` for multi-user
- Use `page`, `app_server` for unauthenticated page checks
- Assert on visible text/content, not internal state
- Use `wait_until="domcontentloaded"` on all `goto()` and `wait_for_url()` calls
- Test both success paths and error/validation paths
- For delete buttons, scope to the specific element type (e.g., `.comment-action-btn`)

### Step 7: Run full suite again (only if asked by user)

```bash
make test
make test-headed   # visual confirmation
```

### Step 8: Document

- Update `AGENTS.md` if new conventions introduced
- Update `README.md` if new routes, config, or dependencies added

## Autonomous Agentic CI Workflow

```python
while feature_not_complete:
    1. Plan: read existing code, design the change
    2. Implement: write code (router → template → CSS → JS)
    3. validate per language + clean import   # must pass
    4. Update AGENTS.md if needed
```

Failures at any step block the workflow. Never skip a failed step.

## CI/CD

Gitea Actions workflow at `.gitea/workflows/test.yaml` runs on every push/PR to `master`. It installs dependencies plus Playwright Chromium, runs the full test suite serially under coverage (`coverage run -m pytest tests/`; the app's uvicorn subprocess writes its own `.coverage.*` file, merged via `coverage combine` thanks to `.coveragerc` `parallel=true`), publishes the coverage HTML as an artifact on every run, and uploads failure screenshots when a test fails. The CI must be green before merging. Changes move through automated DTAP streets: Development (`make dev`), Test (the CI suite + coverage on `master`), Acceptance (the `master` to `production` promotion via `make deploy`), and Production (the Docker Compose stack). Only CI-green `master` commits are promoted to `production`.

## SEO Implementation

All SEO features are implemented across the following locations:

### Core SEO utilities
- `devplacepy/seo.py` - JSON-LD schema generators (WebSite, BreadcrumbList, DiscussionForumPosting, ProfilePage, SoftwareApplication), meta description truncation, schema combiner, sitemap XML generator
- `routers/seo.py` - robots.txt and sitemap.xml routes

### SEO template context
- Every router passes `page_title`, `meta_description`, `meta_robots`, `canonical_url`, `og_title`, `og_description`, `og_image`, `og_type`, `breadcrumbs`, `page_schema` via `base_seo_context()`
- Auth pages: `noindex,nofollow`
- Messages/Notifications: `noindex,nofollow`
- Profiles with < 2 posts: `noindex,follow`
- All other pages: `index,follow`

### Template layer
- `templates/base.html` - dynamic `<title>`, `<meta description>`, `<link canonical>`, `<meta robots>`, Open Graph, Twitter Cards, JSON-LD injection, breadcrumb nav, CDN `dns-prefetch`/`preconnect`
- `static/css/base.css` - `.breadcrumb` (aria-label breadcrumb nav), `.sr-only` (accessible hidden headings)

### Heading hierarchy
- `feed.html` - `<h1 class="sr-only">Feed</h1>`
- `profile.html` - username rendered as `<h1 class="profile-name">`
- `messages.html` - `<h1 class="sr-only">Messages</h1>`
- `projects.html` - `<h1>Projects</h1>`
- `post.html` - post title as `<h1>`, "Related Discussions" as `<h3>`

### Post slugs
- Slug generated on post creation via `slugify()` and stored in `posts.slug` column
- Posts can be looked up by slug or UUID
- Minimum content validation: post body >= 10 chars, comment >= 3 chars

### Related posts
- `templates/post.html` - "Related Discussions" widget at bottom of post page (queried by matching topic)

### Performance
- `loading="lazy"` on all avatar images
- `dns-prefetch` + `preconnect` for CDN resources in `<head>`
- Security headers middleware: `X-Robots-Tag`, `X-Content-Type-Options`

### Default OG image
- `static/og-default.svg` - 1200x630 SVG with DevPlace branding
- Used as fallback `og:image` on all pages

### SEO tests
- SEO tests are split by surface: `tests/api/robotstxt.py`, `tests/api/sitemapxml.py`, and `tests/unit/seo.py` plus the per-page e2e checks - covering robots.txt, sitemap.xml, page titles, noindex, canonical URLs, OG tags, Twitter cards, structured data, security headers

## Engagement: reactions, bookmarks, polls, contribution heatmap

### Emoji reactions
- Curated palette only: `REACTION_EMOJI` in `constants.py` (registered as a template global). `ReactionForm` rejects anything outside it; free-text emoji are not allowed.
- Endpoint `POST /reactions/{target_type}/{target_uid}` (`routers/reactions.py`) toggles one `(user, target, emoji)` row in the `reactions` table. Target types: `post`, `comment`, `gist`, `project`. AJAX (`x-requested-with: fetch`) returns `{counts, mine}`.
- Reactions are **non-ranking** - they never touch `stars` or XP and intentionally send **no notifications** (votes already notify; reactions would be notification spam).
- Batch reads via `get_reactions_by_targets(target_type, uids, user)` in `database.py` (used by feed, profile, comment loader, `load_detail`) - never per-row. The `_reaction_bar.html` partial takes `_type`, `_uid`, `_reactions` ({counts, mine}) and renders the full palette as toggle chips; `ReactionBar.js` uses document-level click delegation. All four engagement controllers (`ReactionBar`, `VoteManager`, `BookmarkManager`, `PollManager`) extend the shared `OptimisticAction` base (the `Http.sendForm -> render -> error` core); each keeps only its own event wiring and `_render`.

### Bookmarks
- `POST /bookmarks/{target_type}/{target_uid}` toggles a `bookmarks` row; `GET /bookmarks/saved` renders the personal list (`saved.html`). Target types: `post`, `gist`, `project`, `news`.
- `_bookmark_button.html` takes `_type`, `_uid`, `_bookmarked`; `BookmarkManager.js` swaps the label/`bookmarked` class from the JSON `{saved}`. Batch state via `get_user_bookmarks(user_uid, target_type, uids)`.

### Polls
- A poll rides on a post (one `polls` row keyed by `post_uid`, options in `poll_options`, one-per-user votes in `poll_votes`). Created in `posts.py:create_poll` when `poll_question` plus ≥2 non-empty `poll_options` are submitted (capped at 6). Both `create_post` and `edit_post` accept the poll fields; `edit_post` only attaches a poll when the post has **none** yet (it never replaces an existing poll). The builders live in the create-post modal (`feed.html`) and the edit-post modal (`post.html`, rendered only when the post has no poll) using `data-poll-toggle` / `data-poll-add-option`.
- `poll_options` accepts either repeated form fields (the web builders, which preserve commas inside an option label) **or** a single newline- or comma-separated string (the API/Devii path). `models.py:normalize_poll_options` splits a lone delimited element - applied as a `mode="before"` validator on `PostForm`/`PostEditForm` - so the documented "one per line or comma separated" agent format actually produces a multi-option poll instead of a single dropped option.
- `POST /polls/{poll_uid}/vote` toggles the voter's choice and returns the full poll dict: voting on a new option switches, voting the **same** option again **retracts** (one vote per user, like reactions). Results (bars + `%`) are **always shown** to everyone; the chosen option gets `.chosen` + a `✓`. Guests see read-only bars with a "Log in to vote" hint (options `disabled`). Batch reads via `get_polls_by_post_uids(post_uids, user)` / `get_poll_for_post(post_uid, user)`.
- Composer poll builder (`feed.html`, `PollManager.js`): poll inputs are `disabled` until "Add poll" is opened (so a never-opened or "Remove poll"-collapsed builder submits **nothing**); a capture-phase `submit` listener blocks an incomplete poll (question + ≥2 options) with an inline error instead of silently dropping it - capture phase + `stopImmediatePropagation()` is required so it pre-empts `FormManager`'s bubble-phase submit handlers.

### Cascade
- `soft_delete_engagement(target_type, uids, deleted_by)` in `database.py` soft-deletes reactions, bookmarks and (for posts) poll rows. It is called from `content.py:delete_content_item` (for the item and its comments) and `comments.py:delete_comment` (which now soft-deletes the comment plus its attachments, votes, and engagement under one `stamp`). Add it to any new delete path. The unfiltered hard `delete_engagement` remains for GC only.

### Contribution heatmap + streaks
- Derived, **no new table**: `get_activity_calendar(user_uid)` aggregates `date(created_at)` across `posts`/`comments`/`gists`/`projects` (cached 120s); `get_activity_heatmap` returns 53 Monday-aligned weeks of `{date, count, level}` (level 0-4); `get_streaks` returns `{current, longest}`. Rendered server-side in `profile.html` (`.heatmap-grid`, styles in `profile.css`). The `On Fire` badge is awarded in `check_milestone_badges` when the current streak ≥ 7.

### Follow graph listings
- `profile.html` has Followers and Following tabs (`?tab=followers` / `?tab=following`, `?page=N`). `profile_page` computes `get_follow_counts(uid)` for the tab labels and, only for those two tabs, `_follow_people(uid, tab, current_user, page)`. The same two listings are exposed as JSON at `GET /profile/{username}/followers` and `GET /profile/{username}/following` (public, 25 per page) returning `{username, mode, count, page, total_pages, <mode>: [{uid, username, bio, is_following}]}`.
- `database.py` helpers: `get_follow_counts(uid)` (`{followers, following}`), `get_follow_list(uid, mode, page)` (paginated people + `build_pagination`, ordered newest-first), and `get_following_among(follower_uid, target_uids)` (single IN-clause set used to set `is_following` per row, avoiding N+1). `mode` is `"followers"` (people who follow `uid`) or `"following"` (people `uid` follows).
- Devii catalog tools `list_followers` / `list_following` (`requires_auth=False`) map to the JSON endpoints; documented in `docs_api.py` under the `profiles` group.

### CSS
- Reactions, bookmarks, polls and the saved page live in `static/css/engagement.css`, loaded globally in `base.html` (the controls appear across feed, detail pages and comments). Heatmap styles are in `profile.css`. Follow-list rows (`.follow-row`, `.follow-user`, `.follow-pagination`) are in `profile.css`.

## Per-user site customization (CSS/JS)

Users (and guests) inject their own CSS and JS, scoped to a **page type** or **globally**, configured conversationally through Devii. It is **per-owner only** - the code runs solely in that owner's own browser sessions (self-XSS, like a userscript), never in anyone else's view. There are **no HTTP routes**: persistence is owner-scoped from the trusted Devii session (the same pattern as `LessonStore`/`TaskStore`), and serving is a template-global injection.

- **Page type** = the matched route template (`request.scope["route"].path`, e.g. `/posts/{slug}`), resolved by `customization.page_type_for(request)`. One value covers all instances of a type (all posts), never a single post. It is also rendered as `data-page-type` on `<main class="page">` and surfaced to Devii via `DeviiClient._context().pageType`.
- **Owner** = `customization.owner_for(request)`: signed-in user -> `("user", uid)`, else the `devii_guest` cookie -> `("guest", cookie)` (session-scoped, prunable), else none. `DEVII_GUEST_COOKIE` lives in `constants.py` (shared by `routers/devii.py` and `customization.py`).
- **Storage** (`database.py`): table `user_customizations` (one row per `owner_kind`, `owner_id`, `scope`, `lang`; `enabled` flag; indexed `idx_user_cust_owner` / `idx_user_cust_scope`). Helpers `get_custom_overrides` (merges **global first, then page-type**, skips disabled; cached under the `"customizations"` cache-version name with `sync_local_cache`/`bump_cache_version`), `get_custom_override`, `list_custom_overrides`, `set_custom_override` (upsert + bump), `delete_custom_override` (+ bump).
- **Per-user suppression toggles** (distinct from the site-wide `customization_enabled` kill-switch): two integer columns on the `users` table - `cust_disable_global` and `cust_disable_pagetype` (0 = showing, 1 = suppressed, ensured in `backfill_api_keys`) - let a user hide their own customizations without deleting the rows. `get_customization_prefs(owner_kind, owner_id)` reads them (always False for non-user owners, so guest rendering is unaffected) and `get_custom_overrides` consults it to skip the `global` piece and/or every non-global (`page_type`) piece for both CSS and JS. `set_customization_pref(owner_id, category, disabled)` (`category` in `CUSTOMIZATION_PREF_COLUMNS` = `{"global","pagetype"}`) writes the column, clears the user cache, and bumps the `"customizations"` cache version. The profile page exposes them as two toggle buttons (`POST /profile/{username}/customization/global` and `/pagetype`, body `CustomizationToggleForm{value}` where `value` = the new ENABLED state) gated to the owner OR an admin via `resolve_customization_target`; the human UI is `templates/profile.html` (`[data-customization]` card, server-rendered state) + `static/js/CustomizationToggle.js` (`Http.send` optimistic flip, registered as `app.customizationToggle`). This is a per-user feature flag stored as explicit `users` columns consulted inside the cached getter - reuse this pattern for future per-user flags.
- **Robustness against hostile CSS**: because users can override `body`, the app's own `position: fixed` chrome is layer-promoted with `will-change: transform` (grouped rules in `base.css` for `.topnav`/overlays/`.modal-overlay`/`.dialog-overlay`/`.context-menu`/`.dp-toast-host`/`.feed-fab`/`.attachment-lightbox`, and in `devii.css` for the devii launcher/window/highlight/toast). This stops a user `background-attachment: fixed` (or any fixed-background) from triggering the Chromium compositing issue that makes other `position: fixed` elements vanish or fail to repaint (it previously wiped the FAB and the create-post modal). It is a paint-only hardening (no layout change, since none of these have viewport-anchored fixed descendants). Any NEW always-on fixed UI element must be added to one of these groups.
- **Injection** (`customization.py` + `templating.py` globals + `base.html`): `custom_css_tag(request)` emits a `<style id="user-custom-css">` placed AFTER `extra_head` so user CSS overrides the design system (closing-tag breakout neutralized by `</` -> `<\/`, valid in CSS). `custom_js_tag(request)` emits the code as a JSON island `<script type="application/json" id="user-custom-js-src">` (with `<`/`>` `<`/`>`-escaped so `</script>` cannot break out) run by a tiny bootstrap via `new Function(JSON.parse(...))`, placed AFTER `Application.js`/`extra_js` so `app` exists. Both are wrapped in try/except that logs and returns empty Markup - **a customization issue must never break page render** (the one intended error suppression). Two `site_settings` kill-switches: `customization_enabled`, `customization_js_enabled`.
- **Devii** (`services/devii/customization/`, `handler="customization"`): `customize_list`, `customize_get`, `customize_set_css`, `customize_set_js`, `customize_reset` (all `requires_auth=False`), plus `customize_set_enabled` (`requires_auth=True`, the suppression toggles - `category` global/pagetype + `enabled`; reversible so not in `CONFIRM_REQUIRED`; `_list` also reports `disabled_global`/`disabled_pagetype`). `CustomizationController(owner_kind, owner_id)` is built in `Dispatcher.__init__` (owner threaded from `DeviiSession` through the `Dispatcher(...)` call). The set/reset tools are in `CONFIRM_REQUIRED`; `confirmation_error` forces Devii to ask **page-type vs global** (set) or confirm deletion (reset) before `confirm=true`. The system-prompt CUSTOMIZATION section tells Devii to preview live with `run_js` (ids `devii-preview-css`/`devii-preview-js`), build on `customize_get` instead of overwriting, and `reload_page` after saving.

## Devii user-defined ("virtual") tools

Users invent new Devii tools in natural language; each is stored per-owner and added to Devii's live LLM tool list, and when called its handler **re-prompts Devii itself** (a self-eval sub-agent) with the stored prompt plus the user's `input`. Full CRUD is Devii-only.

- **Dynamic tool list (the crux).** The session tool list is normally frozen (`session.py` `self.tools = CATALOG.tool_schemas_for(...)`, handed by reference to `Agent`, `AgenticController.bind`, and read live by `react_loop` each `llm.complete`). `DeviiSession._refresh_tools()` runs at the top of every `_run_turn` (inside `self._lock`, before `agent.respond`) and rewrites that list **in place**: `self.tools[:] = CATALOG.tool_schemas_for(self.client.authenticated, self.is_admin) + self._virtual_tool_store.tool_schemas()`. Because every holder shares the same list object, a tool created/edited/deleted (and a mid-session login) takes effect on the **next** turn with no Agent rebuild.
- **Self-eval engine** (`agentic/controller.py`): `_delegate` was refactored onto `_spawn(prompt, tools, system_prompt)`, which runs `react_loop` with a fresh `messages`/`AgentState` under a **depth guard** (`agentic/state.py` `get/set/reset_eval_depth`, `MAX_EVAL_DEPTH=2`) so delegate/eval/virtual tools cannot self-call infinitely. `run_subagent(prompt)` (tools minus `delegate`) is the public engine; the `eval` agentic tool wraps it; `_delegate` keeps its richer JSON report.
- **Store** (`services/devii/virtual_tools/store.py`): `VirtualToolStore(db, owner_kind, owner_id)` over `devii_virtual_tools` (`uid, owner_kind, owner_id, name, description, prompt, input_description, enabled, created_at, updated_at`; indexes `idx_devii_vtools_owner` / `idx_devii_vtools_name`). `tool_schemas()` builds, for each **enabled** row, an LLM schema with a single free-form `input` string. Persistent for users, `memory_db()` for guests (built in `hub.get_or_create` from the shared `owned_db`).
- **Controller** (`virtual_tools/controller.py`): `VirtualToolController(store, evaluator, builtin_names)`, evaluator = `agentic.run_subagent`, `builtin_names = set(CATALOG.by_name())`. CRUD `tool_create/list/get/update/delete` (`handler="virtual_tool"`); `tool_create` validates name `^[a-zA-Z][a-zA-Z0-9_]{1,40}$`, rejects built-in collisions and duplicates, requires `description`+`prompt`. `has(name)` / `run(name, args)` compose `f"{prompt}\n\nUser input: {input}"` and call the evaluator.
- **Dispatch** (`actions/dispatcher.py`): constructed with `virtual_tools=...`; `_run` routes `handler="virtual_tool"` to CRUD; the **unknown-name branch** (`if action is None`) now resolves a virtual tool via `self._virtual_tools.has(name)` -> `run(name, args)` before erroring. So virtual tools live only in the store (not the static catalog) and are resolved on demand; the static `_actions` stays built-in only.
- Registered via `VIRTUAL_TOOL_ACTIONS` (`registry.py`); the `eval` tool via `AGENTIC_ACTIONS`. System-prompt **USER-DEFINED TOOLS (VIBE TOOLS)** section steers creation and warns against deep self-calls.

### Devii self-configured behavior (`services/devii/behavior/`)
Devii can partially configure its OWN system message: every system prompt ends with a `# TRUTH RULES AND BEHAVIOR` section (the `BEHAVIOR_HEADER` constant in `session.py`) whose body is an owner-scoped, persistent set of behavior rules. When the user tells Devii to behave differently, says they expect different behavior, or Devii upsets them, Devii calls the **`update_behavior`** tool (`handler="behavior"`, `requires_auth=False`, single `behavior` arg = the FULL new section content) to persist the change. It is **self-prompted only**: the sole way to change the section is Devii calling that tool - there is no HTTP route or UI, like `LessonStore`/customization. Because Devii already SEES the current section in its own system message, it merges (copy current rules, apply the change, send the whole result) so prior rules are not lost; this is why the tool replaces rather than appends.
- **Store** (`behavior/store.py`): `BehaviorStore(db, owner_kind, owner_id)` over `devii_behavior` (one upserted row per owner keyed on `owner_kind`/`owner_id`; `text()` reads, `set()` upserts; index `idx_devii_behavior_owner`). Persistent for users, `memory_db()` for guests (built in `hub.get_or_create` from the shared `owned_db`, like the other owner stores).
- **Controller** (`behavior/controller.py`): `BehaviorController(store)`, `dispatch("update_behavior", args)` -> `store.set(behavior)`. Built in `DeviiSession` and passed to `Dispatcher(behavior=...)`; the dispatcher routes `handler="behavior"` to it and degrades gracefully ("not available in this context") when unwired (e.g. the standalone CLI, same as `virtual_tools`/`avatar`).
- **Injection + refresh** (`session.py`): `_compose_system_prompt()` = base prompt (`_system_prompt_for(is_admin)`) + `\n\n` + `BEHAVIOR_HEADER` (+ `\n` + body when non-empty; just the header when empty). Used to seed the `Agent` and the scheduler executor worker, and `_refresh_system_prompt()` rewrites `agent._messages[0]["content"]` **at the top of every `_run_turn`** (next to `_refresh_tools`), so a mid-conversation `update_behavior` takes effect on the following turn and the live system message is never overridden by the stale base.
- Registered via `BEHAVIOR_ACTIONS` (`registry.py`); the system-prompt **SELF-CONFIGURED BEHAVIOR (TRUTH RULES)** section in `agent.py` steers when/how to call it.

## Database API service (`devplacepy/services/dbapi/`, `devplacepy/routers/dbapi/`)

**Admin/internal-only** generic data access over the `dataset`/SQLite layer, mounted at `/dbapi`. It exposes structured CRUD per table, a validated raw `query()`, a natural-language-to-SQL designer backed by the internal AI gateway, and async query execution streamed over a websocket. Reuses the existing dataset helpers, the `JobService`/`ProgressHub`/lock-owner websocket pattern, the AI gateway, and the Devii action catalog - it does not re-implement any of them.

- **Auth (single boundary):** `services/dbapi/policy.py` `caller_for(request)` returns a `Caller` for an admin session/admin api_key, or for the `database.internal_gateway_key()` (service-to-service), else `None`. `require_caller` raises `DbApiDenied`; the router's `require_dbapi_caller` turns that into a `403` and an audited `database.access.denied`. Members/guests never reach a handler.
- **Table guard:** `policy.assert_table(name)` enforces a name regex, existence in `db.tables`, and a deny-list (`DEFAULT_DENY = {sessions, password_resets, cache_state}` plus the `dbapi_deny_tables` setting). Used on every table-scoped path and indirectly by the validator's table extraction, so neither a crafted segment nor an NL-designed query can touch a denied table.
- **CRUD (`services/dbapi/crud.py`, `routers/dbapi/crud.py`):** `GET /dbapi/{table}` (keyset pagination newest-first, `?filter.<col>=`, `?gte/lte/gt/lt.<col>=`, `?search=`, `?before=`, `?limit=`, `?include_deleted=`), `GET /dbapi/{table}/{key}/{value}`, `POST /dbapi/{table}` (**born-live**: auto `uid`/`created_at`, `deleted_at:None`/`deleted_by:None` for soft-delete tables; **unknown columns are rejected** so the API never grows or pollutes the schema), `PATCH /dbapi/{table}/{key}/{value}` (never overwrites `uid`/`id`, sets `updated_at`), `DELETE /dbapi/{table}/{key}/{value}` (**soft by default** via `database.soft_delete`, `?hard=true` purges), `POST /dbapi/{table}/{key}/{value}/restore`. Mutations write `database.row.*` audit events.
- **`query()` is hard SELECT-only (`services/dbapi/validate.py`):** `classify()` parses with `sqlglot` (statement type, table list, `has_where/has_join/has_limit`, suspicious notes for missing `WHERE/JOIN/LIMIT`, multiple statements, `ATTACH/PRAGMA/VACUUM`). `validate_select()` rejects anything but a single SELECT, then `dry_run()` does `EXPLAIN` on a **separate read-only** connection (`file:...?mode=ro` + `PRAGMA query_only=ON`) for true validity. `run_select()` executes read-only. `POST /dbapi/query` returns `409` for non-SELECT (pointing at the CRUD routes), `400` for invalid SQL, else rows + a `suspicious` list. Mutations are NEVER possible through `query()` - they go through the structured CRUD.
- **NL-to-SQL (`services/dbapi/nl2sql.py`):** `POST /dbapi/nl` builds a system prompt from the table schema + 5 example rows + a soft-delete rule (auto `deleted_at IS NULL` unless `apply_soft_delete=false`), calls the internal gateway (`INTERNAL_GATEWAY_URL`, model `dbapi_nl_model` or `molodetz`, auth = the admin caller's api_key for attribution else the internal key), and **reprompts with the validator's error until the SQL validates** (max 3 attempts). Returns the validated SQL by default; `execute=true` also runs it read-only and returns rows.
- **Async (`services/dbapi/service.py` `DbApiJobService`, kind `dbquery`):** `POST /dbapi/query/async` validates then `queue.enqueue`s; the service re-validates (defense in depth), streams rows in batches to its own `ProgressHub`, writes the full result to `config.DBAPI_DIR/{uid}/result.json`, and returns stats. `GET /dbapi/query/{uid}` (status), `GET /dbapi/query/{uid}/result` (rows from disk, extends retention), `WS /dbapi/query/{uid}/ws` (lock-owner gated, `4013` retry, snapshot-then-stream - the SEO pattern). Config on `/admin/services`: `dbapi_max_rows`, `dbapi_nl_model`, `dbapi_nl_system_preamble`, `dbapi_deny_tables`, plus the standard Jobs fields.
- **Devii:** admin-only tools `db_list_tables`, `db_table_schema`, `db_list_rows`, `db_get_row`, `db_query` (SELECT-only; surface `suspicious`), `db_design_query` (NL->SQL), and the confirm-gated mutations `db_insert_row`/`db_update_row`/`db_delete_row` (in `CONFIRM_REQUIRED`, each declares a `confirm` boolean param; mutations pass arbitrary columns as a `values_json` JSON string). This is how "a non-SELECT is always confirmed" is honored - raw SQL stays read-only, all writes are the confirm-gated CRUD tools.
- **nginx:** `/dbapi/query/<uid>/ws` has its own upgrade location.

## Pub/Sub service (`devplacepy/services/pubsub/`, `devplacepy/routers/pubsub.py`)

A general-purpose, **database-free** publish/subscribe bus mounted at `/pubsub`, for live broadcast between frontend tabs, background services, and Devii without polling.

- **Hub (`services/pubsub/hub.py`):** `PubSubHub` keyed by topic pattern -> live websockets (no persisted buffer; pub/sub is fire-and-forget). `subscribe/unsubscribe/drop_socket`, `async publish(topic, frame) -> delivered` (drops dead sockets), `topics()`. Wildcards: a subscription to `foo.*` matches `foo` and `foo.bar` (`topic_matches`). Module singleton `pubsub`; `services.pubsub.publish(topic, data)` is the in-process helper for backends.
- **Single-worker model:** the `WS /pubsub/ws` is gated to the service lock-owner worker with the `4013` fast-retry close code (like `/devii/ws`), so every subscriber transparently converges on one worker and the in-process fan-out reaches everyone with no broker and no DB. Deliberate, documented trade-off: ephemeral and best-effort (a crash drops in-flight messages), matching the `background` queue's durability stance.
- **Topic authz (`services/pubsub/policy.py`):** `resolve_actor` reuses the dbapi caller (admin/internal) then falls back to user/guest. `can_subscribe`/`can_publish`: admin/internal anywhere; users publish only to their own `user.{uid}.*` namespace and subscribe to that plus `public.*`; publishing to `public.*` is admin/internal only (prevents cross-user spam); guests get read-only `public.*` when `pubsub_allow_guests` is on. Topic names match `^[A-Za-z0-9_.*-]{1,128}$`; payloads capped at 64 KB.
- **HTTP + introspection:** `POST /pubsub/publish` (admin/internal; returns `409` on a non-owner worker, retry), `GET /pubsub/topics` (admin/internal). Admin/internal publishes record a `pubsub.publish` audit event.
- **Frontend:** `app.pubsub` (`static/js/PubSubClient.js`) is a lazy reconnecting client: `app.pubsub.subscribe(topic, cb)` / `app.pubsub.publish(topic, data)`, `4013`-aware fast retry, re-subscribes on reconnect, client-side wildcard dispatch.
- **Service + nginx:** `PubSubService` (`default_enabled=True`, no-op run loop) surfaces the `pubsub_allow_guests` toggle on `/admin/services` and gates the WS via `is_enabled()`. `/pubsub/ws` has its own nginx upgrade location.
