# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project

DevPlace is a server-rendered social network for developers. FastAPI backend serves Jinja2 templates with pure ES6 module JavaScript on the frontend. SQLite via the `dataset` library (auto-syncs schema). No JS framework, no NPM, no JWT.

## Commands

```bash
make install        # pip install -e .
make ppy            # build the single shared container image (ppy:latest); run once before launching instances
make dev            # uvicorn --reload on port 10500
make prod           # uvicorn 2 workers, port 10500 (backlog 8192)
make test           # Playwright + unit tests, headless, serial (one at a time), -x fail-fast
make test-headed    # same tests in a visible Chromium window (single process)
make locust         # Locust load test, interactive web UI
make locust-headless  # Locust CLI mode for CI
```

The Makefile exports `PYTHONDONTWRITEBYTECODE=1` for every recipe, so no `.pyc` files or `__pycache__` directories are written by any `make` target (tests, dev, coverage, agents, locust). Keep it that way - do not add a target that re-enables bytecode writing. `make clean` removes any stray bytecode left from running Python outside make.

Validate code without running the suite: confirm `python -c "from devplacepy.main import app"` imports clean and check each touched language manually (Python compiles/imports, JS parses, CSS braces and HTML tags balance).

Single test: `python -m pytest tests/e2e/feed.py::test_name -v --tb=line -x`

CLI (installed as `devplace`):
```bash
devplace role get <username>
devplace role set <username> <member|admin>
devplace news clear         # delete all news rows
devplace news sanitize      # strip HTML from news descriptions/content
devplace attachments prune  # remove orphan attachment records/files
devplace devii reset-quota <username>   # reset one user's rolling 24h AI quota
devplace devii reset-quota --guests     # reset every guest quota
devplace devii reset-quota --all        # reset every quota (users and guests)
devplace zips prune         # delete expired zip archives + job rows
devplace zips clear         # delete every zip archive + job row
devplace forks prune        # delete expired completed fork job rows (forked projects persist)
devplace forks clear        # delete every fork job row (forked projects persist)
devplace seo prune          # delete expired SEO audit reports + job rows
devplace seo clear          # delete every SEO audit report + job row
devplace deepsearch prune   # delete expired DeepSearch sessions + job rows + collections
devplace deepsearch clear   # delete every DeepSearch session + job row + collection
devplace backups list       # list recorded backups
devplace backups run <database|uploads|keys|full>  # enqueue a backup (processed by the running server)
devplace backups prune      # remove backup records whose archive file is missing
devplace backups clear      # delete every backup archive + record
devplace containers list    # list container instances
devplace containers reconcile  # run one reconcile pass (desired vs docker ps)
devplace containers prune   # reap orphan containers + dangling images
devplace containers prune-builds  # remove legacy per-project images + clear dockerfiles/builds tables (one-time)
devplace containers gc-workspaces  # remove workspace dirs with no instances
devplace migrate-data       # relocate legacy runtime files into data/ (idempotent; --dry-run to preview)
```

### Runtime data layout (single source of truth)

Every runtime/user-generated artifact lives under one root, `config.DATA_DIR` (`DEVPLACE_DATA_DIR`, default `<repo>/data`). `config.py` derives every runtime path from it and lists them in the `DATA_PATHS` registry; `ensure_data_dirs()` (called at the top of `startup()` and from `database.init_db`, plus a guard at `database.py` import before `dataset.connect`) creates the whole tree before anything is written. **No module computes a runtime path from scratch** - they import the constants (`UPLOADS_DIR`, `ATTACHMENTS_DIR`, `PROJECT_FILES_DIR`, `ZIPS_DIR`, `ZIP_STAGING_DIR`, `FORK_STAGING_DIR`, `KEYS_DIR`, `BOT_DIR`, `LOCKS_DIR`, `CONTAINER_WORKSPACES_DIR`, `DEVII_TASKS_DB`, `DEVII_LESSONS_DB`, `VAPID_*_FILE`, `SERVICE_LOCK_FILE`, `INIT_LOCK_FILE`). Uploads moved physically from `devplacepy/static/uploads/` to `data/uploads/` but are still **served at the unchanged `/static/uploads/` URL** (FastAPI `UploadStaticFiles` mount -> `config.UPLOADS_DIR`; nginx `alias /data/uploads/` from a mounted volume). Stored DB values for attachments/project files are relative (`directory` + `stored_name`), so **no DB rewrite is needed**. `devplace migrate-data` relocates an older scattered install (copy -> verify CRC -> atomic replace -> unlink; idempotent).

**Blob sharding (`attachments._directory_for`, the single shard function reused by `attachments`, `project_files`, and `zip_service`):** blobs are spread into a two-level `xx/yy` directory tree to keep any one directory small. The shard is taken from the **random tail** of the uuid7 (`tail = uid.replace("-", "")`, `f"{tail[-2:]}/{tail[-4:-2]}"`), NOT its leading bytes. This is load-bearing: a uuid7's first 48 bits are a millisecond timestamp, so the leading hex (`uid[:2]`/`uid[2:4]`) advances only every ~35 years / ~50 days respectively - prefix-sharding a time-ordered id funnels every contemporaneous write into ONE bucket (the legacy `data/uploads/attachments/01/9e/` pile is exactly this defect). The last 62 bits are random, so the tail gives a uniform 256x256 fan-out from the first write. Any new shard helper MUST shard on a high-entropy field (uuid tail or a hash), never the head. The change is forward-only: every consumer persists the computed `directory`/`local_path` and reads it back, so pre-existing rows keep resolving from their stored value and only new writes use the new layout (no migration required; `migrate-data` could rebalance the old pile if ever desired).

Tests run on port 10501 with a tempfile SQLite DB and a dedicated `DEVPLACE_DATA_DIR`, and `DEVPLACE_DISABLE_SERVICES=1` so background services don't start. `make test` runs **serially, one test at a time, in a single process**: one isolated DB + data dir + uvicorn subprocess + Chromium, shared across the whole session, so session fixtures (`seeded_db`) and intra-file ordering hold. Serial execution is enforced centrally in `pyproject.toml` (`[tool.pytest.ini_options]` `addopts = "--tb=line -p no:xdist"`); pytest-xdist is no longer a dependency and `-n` is rejected, so the suite can never run concurrently.

## Environment variables

| Var | Default | Purpose |
|-----|---------|---------|
| `DEVPLACE_DATABASE_URL` | `sqlite:///<repo>/data/devplace.db` | Override DB path (tests use this) |
| `SECRET_KEY` | hardcoded fallback | Session signing |
| `DEVPLACE_DISABLE_SERVICES` | unset | When `1`, NewsService and other background services skip start (set by test conftest) |
| `PLAYWRIGHT_HEADLESS` | `1` in tests | Toggle headed mode |
| `DEVPLACE_TEMPLATE_AUTO_RELOAD` | `1` (on) | Jinja template auto-reload. `1` stat-checks every template per render (dev hot-reload); set `0` in production so compiled templates stay cached in memory (the `prod`/`locust` make targets set it). |
| `DEVPLACE_WEB_WORKERS` / `--workers` | `nproc` (prod) | Uvicorn worker count; `prod` derives it from the box core count (`make prod WEB_WORKERS=N` to override). |
| `DEVPLACE_DATA_DIR` | `<repo>/data` | Single root for ALL runtime/user-generated data OUTSIDE the package and OUTSIDE `/static`: the DB (`data/devplace.db`), `data/uploads/` (attachments + project files), `data/devii_*.db`, `data/keys/` (VAPID), `data/locks/`, `data/bot/`, zip/fork staging, and container workspaces. Defined once in `config.py` (`DATA_PATHS` registry + `ensure_data_dirs()`). Point at a volume in prod. Build contexts are ephemeral and live in the OS temp dir. |

## Architecture

### Request pipeline
`main.py` mounts `/static`, registers every router with its prefix, installs middlewares (`X-Robots-Tag`/`X-Content-Type-Options` security headers, a per-IP rate limit held in an in-process `defaultdict`, and a maintenance gate), and registers global 404/500 handlers that render `error.html` via the shared templates instance. The rate limit reads `rate_limit_per_minute` / `rate_limit_window_seconds` from `site_settings` (default 60/60s, floored to `max(1, …)`). **It applies only to mutating methods (`POST`/`PUT`/`DELETE`/`PATCH`); reads (`GET`/`HEAD`) and the whole `/openai` gateway are exempt** - so a burst of `GET` requests is never throttled (verified against production: 400 concurrent `GET /` all `200`, while 150 concurrent `POST /auth/login` allowed exactly 60 then returned `429`). The bucket is keyed on the `X-Real-IP` header (set by the reverse proxy) falling back to `request.client.host`, and an over-limit request gets a `429` carrying `Retry-After: <window>` (HTML body, or the shared JSON error shape for `Accept: application/json` clients). The maintenance middleware short-circuits non-admin requests with a 503 `error.html` when `maintenance_mode="1"`, but always allows `/static`, `/avatar`, `/auth`, `/admin` and admin users so an operator can never lock themselves out.

`@app.on_event("startup")` calls `init_db()` and, unless `DEVPLACE_DISABLE_SERVICES` is set, registers `NewsService` with `service_manager` and kicks off `start_all()` as an asyncio task. `@app.on_event("shutdown")` calls `service_manager.stop_all()`.

`GET /` is the home page and never redirects: guests get the marketing splash, authenticated users get a personalized home (welcome card with their avatar/stats, a feed shortcut, quick links), both sharing the Latest Posts + Developer News sections. It pulls up to 6 articles where `show_on_landing=1` from the `news` table plus 6 recent posts (joined via the batch helpers in `database.py`). The Latest Posts section and the main `/feed` enforce **author diversity** by *interleaving* authors (never dropping posts) via `interleave_by_author` / `paginate_diverse` in `database.py` - the page set is the chronological window, then reordered so no two consecutive posts share an author while each author's own posts stay in date order (a run from one author is broken up by deferring to the next available author, recursively; an all-one-author page stays as-is). `paginate_diverse` mirrors `paginate` (same `(rows, next_cursor)` contract): the interleave is a pure permutation of the page, so the cursor (oldest `created_at` in the page) is unchanged.

### Routing layout
Routers in `devplacepy/routers/` are organised as a **directory tree that mirrors the endpoint (URL) path**, exactly like `tests/`. A domain with a single resource stays one flat file (`feed.py`, `posts.py`, `gists.py`, ...); a domain with several sub-resources is a **package directory** split **one file per sub-resource** (a distinct noun under the domain), never one file per individual endpoint. Each leaf module declares its own `router = APIRouter()` keeping the exact path strings, and the package `__init__.py` aggregates them with `router.include_router(...)`. Because a package exposes `.router` like a module, `main.py` mounts each domain at its prefix unchanged. **Aggregation rules:** a leaf that owns the domain's collection-root (`""`) route must be the package's base router (FastAPI rejects an empty path included under an empty prefix), so the package `__init__` imports that leaf's `router` and includes the rest onto it (see `routers/issues`, `routers/admin`, `routers/projects`); leaves carved out of a former monolith keep their relative path strings and are included with no sub-prefix, while a router folded in from a deeper mount keeps its own paths and is included with a sub-prefix (`admin/__init__` includes `services.router` with `prefix="/services"`). Module-private helpers shared across a package's leaves live in its `_shared.py`. Prefixes are wired in `main.py`:

| Prefix | Router |
|--------|--------|
| `/auth` | auth/ package - one leaf per flow (`signup`, `login`, `logout`, `forgotpassword`, `resetpassword`) |
| `/feed` | feed.py |
| `/posts` | posts.py |
| `/comments` | comments.py |
| `/projects` | projects/ package - `index.py` (listing/detail/create/delete plus owner-only visibility toggles `POST /{slug}/private` and `POST /{slug}/readonly`, and async `POST /{slug}/fork`), `files.py`, and the `containers/` subpackage (below). `main.py` mounts the whole `/projects` tree from this one package |
| `/projects/{slug}/files` | projects/files.py - per-project virtual filesystem (CRUD dirs/files, upload, inline edit, plus line-range ops: `lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append` for large text files) |
| `/profile` | profile/ package - `index.py` (page/search/followers/update/api-key; the **Posts** tab shows only the 10 most recent posts, while the sidebar `posts_count` stat always uses the real `count()` so it is never capped to the rendered 10), `customization.py`, `notifications.py`, `ai_correction.py` (`POST /{username}/ai-correction`, owner-or-admin AI content correction toggle + prompt), `ai_modifier.py` (`POST /{username}/ai-modifier`, owner-or-admin AI modifier toggle + prompt), `usage.py` (`_ai_quota`, re-exported from the package) |
| `/messages` | messages.py - real-time DM chat. `GET ""` (page), `GET /search`, `POST /send` (no-JS fallback), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, read receipts, and in-process presence on top of the existing `messages` table; both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention) and then through `_finalize_and_broadcast`, which awaits any pending SYNC AI correction/modifier futures (DM content runs through both, so typing `@ai <instruction>` executes live) and broadcasts the FINAL corrected/modified content (the WS path passes `request=websocket` and awaits directly since HTTP middleware does not run for websockets; the frontend swaps the sender's optimistic bubble to the authoritative content). Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, deduped against the hub's delivered set). See `AGENTS.md` -> "Real-time messaging" |
| `/notifications` | notifications.py |
| `/votes` | votes.py |
| `/avatar` | avatar.py |
| `/follow` | follow.py |
| `/admin` | admin/ package - one leaf per sub-resource (`index`, `users`, `aiusage`, `aiquota`, `media`, `trash`, `settings`, `notifications`, `news`, `auditlog`, `backups`) plus the folded-in `services.py` and `containers.py` (mounted with `/services` and `/containers` sub-prefixes). `main.py` mounts the whole `/admin` tree from this one package. The `backups` leaf is the admin **Backups** dashboard (`BackupService`, kind `backup`): storage usage, backup archives, and interval/cron backup schedules (CRUD + rotation). See AGENTS.md -> "Backup service" |
| `/admin/services` | admin/services.py |
| `/issues` | issues/ package - issue tracker backed by Gitea (no local issue store): `index.py` (list `?state=`/`?page=`, detail `/{number}` with comments), `create.py` (async AI-enhanced filing `/create` enqueues a `issue_create` job, status at `/jobs/{uid}`), `comment.py` (synchronous, pushes to Gitea + notifies admins), `status.py` (admin open/closed) |
| `/gists` | gists.py |
| `/news` | news.py |
| `/uploads` | uploads.py |
| `/openai` | openai_gateway.py |
| `/devii` | devii.py - WebSocket terminal (`/devii/ws`), page, `/devii/usage`, `/devii/session` |
| `/zips` | zips.py - generic zip job status (`/zips/{uid}`) and archive download (`/zips/{uid}/download`); enqueued from `/projects/{slug}/zip` and `/projects/{slug}/files/zip` |
| `/forks` | forks.py - fork job status (`/forks/{uid}`); enqueued from `/projects/{slug}/fork`. When done its `project_url` points at the new forked project |
| `/tools` | tools/ package - public developer tools surface. `index.py` (`/tools` landing) plus `seo.py` (**SEO Diagnostics**): `GET /tools/seo` page, `POST /tools/seo/run` (enqueue `seo` job, per-owner one-active-job cap), `GET /tools/seo/{uid}` (`SeoJobOut`), `GET /tools/seo/{uid}/report` (HTML+JSON `SeoReportOut`), `WS /tools/seo/{uid}/ws` (live progress, lock-owner gated, close `4013` retry), `GET /tools/seo/{uid}/screenshot/{n}`. Also `deepsearch.py` (**DeepSearch**): `GET /tools/deepsearch` page, `POST /tools/deepsearch/run` (enqueue `deepsearch` job, per-owner one-active-job cap; resolves the user `api_key` into the payload, guests use the internal key), `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (HTML+JSON `DeepsearchSessionOut`), `WS /tools/deepsearch/{uid}/ws` (live progress, lock-owner gated, `4013` retry), `WS /tools/deepsearch/{uid}/chat` (grounded RAG chat over the session collection), `POST /tools/deepsearch/{uid}/{pause|resume|cancel}`, `GET /tools/deepsearch/{uid}/export.{md,json,pdf}`. The shared owner helper is `routers/tools/_shared.py` `owner_for`. Surfaced by a collapsible **Tools** header dropdown (`base.html`, visible to all) |
| `/projects/{slug}/containers` | projects/containers/ subpackage - admin per-project container manager (`instances.py` for creation/lifecycle/exec/logs/metrics/sync plus the exec websocket, `schedules.py` for cron/interval/once schedules, shared helpers in `_shared.py`). Every instance runs the shared `ppy` image. Discoverable from the project detail page (admin-only **Containers** button) and from the admin index |
| `/admin/containers` | admin/containers.py - admin **Containers** manager: `/admin/containers` lists every instance across all projects with inline actions (start/stop/restart/terminal/edit/delete) plus a create modal (project search-select, run-as user search-select, boot language + source editor, restart policy, start-on-boot, env/ports/limits/ingress); `/admin/containers/{uid}` is the per-instance detail page (lifecycle, live logs/metrics, PTY terminal, schedules, ingress, sync, status history) and `/admin/containers/{uid}/edit` edits run-as user, boot language/script/command, restart policy, start-on-boot, and limits. `POST /admin/containers/create`, `/{uid}/edit`, `/{uid}/{start,stop,restart,pause,resume}`, `/{uid}/sync`, `/{uid}/delete` call `api.*` directly under `require_admin` (no docker/exec backend duplicated); `GET /admin/containers/projects/search` and `/users/search` back the create/edit search-selects. The lifecycle and detail views stay layered over the per-project `/projects/{slug}/containers/instances/{uid}/...` endpoints (the instance carries its `project_uid`) |
| `/p/{slug}` | proxy.py - public ingress reverse proxy (HTTP + WebSocket) to a running container instance's published host port, opt-in per instance via `ingress_slug` |
| `/xmlrpc` | xmlrpc.py - reverse-proxies XML-RPC calls to the forking XML-RPC bridge (`services/xmlrpc/`, supervised by `XmlrpcService` on loopback `config.XMLRPC_PORT`), which generates one XML-RPC method per documented REST endpoint from `docs_api.API_GROUPS` (`posts.create`, `feed.list`, ...). One struct of named params per call; auth via in-band `api_key`, `X-API-KEY`/`Bearer` header, or `http://user:pass@host/xmlrpc` Basic. Full introspection + `system.multicall`; REST errors become XML-RPC faults. Exempt from the rate limiter (enforced on the forwarded internal hop). See AGENTS.md -> "XML-RPC bridge" |
| `/api` | devrant/ package - devRant-compatible REST protocol (`auth.py`, `rants.py`, `comments.py`, `notifs.py`). Translates devRant requests onto DevPlace data: rants<->posts, comments/votes onto the native engagement layer, devRant integer ids onto each table's auto-increment `id`. Token auth via `devrant_tokens`. See `### devRant compatibility API` and AGENTS.md -> "devRant compatibility API" |
| `/dbapi` | dbapi/ package - **admin/internal-only** generic database API over `dataset`. `tables.py` (`GET /tables`, `GET /{table}/schema`), `crud.py` (`GET/POST/PATCH/DELETE /{table}[/{key}/{value}]`, soft-delete aware, born-live inserts, `?include_deleted`, `.../restore`), `query.py` (`POST /query` hard SELECT-only validated read-only run; `POST /query/async` + `GET /query/{uid}` + `WS /query/{uid}/ws` via `DbApiJobService` kind `dbquery`), `nl.py` (`POST /nl` natural-language->SQL via the AI gateway, returns validated SQL, `execute=true` runs it read-only). Auth = admin session/api_key OR `internal_gateway_key()` (`services/dbapi/policy.py`); SQL validated by `services/dbapi/validate.py` (sqlglot classify + suspicious-flag + read-only `EXPLAIN` dry-run). Devii tools `db_*` (admin; mutations confirm-gated). See AGENTS.md -> "Database API service" |
| `/pubsub` | pubsub.py - **database-free** publish/subscribe bus. `WS /pubsub/ws` (subscribe/unsubscribe/publish frames, `foo.*` wildcards), `POST /pubsub/publish` + `GET /pubsub/topics` (admin/internal). Lock-owner-gated WS (`4013` retry) so subscribers converge on one worker; topic authz in `services/pubsub/policy.py` (`user.{uid}.*` private, `public.*` shared, admin/internal anywhere, guests opt-in). In-process `services.pubsub.publish(topic, data)` for backends; frontend `app.pubsub`. See AGENTS.md -> "Pub/Sub service" |
| (none) | seo.py - `/robots.txt`, `/sitemap.xml` |

Docs (`routers/docs/pages.py` `DOCS_PAGES`, re-exported from the `routers/docs` package; handlers in `routers/docs/views.py`) entries take optional `admin: True` (hidden + 404 for non-admins, but still indexed and surfaced only to admins by `docs_search`) and `section: "..."` (a nested sidebar group rendered by `docs_base.html`). Sidebar `section`s are grouped under four ordered **audience tiers** (`AUDIENCES` in `pages.py`: `Start here`, `Build with the API`, `Contribute and internals`, `Operate`); `nav_groups(visible_pages)` builds the `[(audience, [(section, [pages])])]` tree and `views.py` passes it as `nav`, so `docs_base.html` renders an audience super-header (`.sidebar-tier`) above each section. The tiering is sidebar-only - `DOCS_PAGES` stays canonical for search/export/routing. The public `getting-started` page (`SECTION_GENERAL`) is the new-contributor on-ramp; gate its deep-internals links with `{% if is_admin(user) %}` so guests get no 404s. Keep one canonical home per concept (the `auth` API group defers auth-method detail to the `authentication` prose page). The member-facing `devii` prose page is functional; admins also get a `Devii internals` section of `devii-*` technical subpages. Prose docs pages are rendered to HTML **server-side** (`docs_prose.py`, mistune GFM): the handler converts the `data-render` markdown block to HTML before sending and drops the `data-render` attribute, so the client never re-renders it (no markdown-to-HTML flicker, faster first paint). Authored example markup inside that block is therefore still HTML-escaped (`&lt;dp-...&gt;`); content outside the block (component live-demo blocks and their `<script type="module">`) passes through untouched. User-generated content elsewhere still uses the client `data-render` pipeline. The public `Components` section (`component-*` prose pages) documents the custom web components with a **live, interactive example** on each page. The public `Styles` section (`styles`, `styles-colors`, `styles-layout`, `styles-responsiveness`, `styles-consistency`) is the design-system reference: the colour tokens and their meaning, the approved page layouts, the responsive breakpoint ladder, and the HARD structural rules every page must follow (taken from the feed/posts page as the canonical implementation). It uses the same live-demo convention (real demo markup OUTSIDE the `data-render` block, example markup inside it entity-escaped). A prose page's `<div class="docs-content" data-render>` is rendered client-side via marked + DOMPurify on `textContent`, so any example markup shown as code inside it MUST be HTML-escaped (`&lt;dp-dialog&gt;`) or the browser parses it as a real element before render; the live demo itself goes in a separate block OUTSIDE the `data-render` div, where a `<script type="module">` (which imports its own component module, since it executes before `Application.js`) wires it up.

### Templates and frontend
- All routers MUST import the shared `templates` from `devplacepy.templating`. Do NOT instantiate `Jinja2Templates` per router - globals (`avatar_url`, `get_unread_count`, `get_user_projects`, `format_date`) are registered on that single instance.
- Templates extend `base.html`, add page CSS via `{% block extra_head %}`, page JS via `{% block extra_js %}`.
- **Static asset URLs are boot-versioned for cache-busting.** Never hardcode a bare `/static/...` href/src: templates wrap it in the `static_url(path)` Jinja global and runtime JS that builds an absolute static URL uses `assetUrl(path)` (`static/js/assetVersion.js`). Both emit a `/static/v<STATIC_VERSION>/...` path segment (`config.STATIC_VERSION` = `DEVPLACE_STATIC_VERSION` env or the boot unix timestamp), so a deploy busts every asset while it stays served `immutable, max-age=31536000`. The version is in the PATH (not a `?v=` query) so relative ES6 imports and relative CSS `url()` inherit it automatically. `/static/uploads/` (user content) and `service-worker.js` are excluded. See `AGENTS.md` -> "Static asset caching and versioning".
- All JS in `static/js/` is ES6 modules, one class per file, instantiated as `app` and reachable everywhere. The browser-side application root is `Application.js`.
- **Custom web components** live in `static/js/components/` (prefix `dp-`): `dp-avatar`, `dp-code`, `dp-content` (wraps the `contentRenderer` markdown/sanitize/media engine behind `data-render`), `dp-toast`, `dp-dialog`, `dp-context-menu`, `dp-upload` (the single file-upload button used everywhere; replaced the old `AttachmentUploader`), `dp-lightbox` (`app.lightbox`: the single image lightbox, attribute-wired via one delegated listener so any `img[data-lightbox]` opens full-size - `data-full` for a distinct full-res URL; attachment thumbnails and all `data-render` markdown images are marked centrally, the latter in `ContentEnhancer`). Each extends `Component` (a thin `HTMLElement` base with `attr`/`boolAttr`/`intAttr` helpers), renders into the **light DOM** (no shadow root, so global CSS applies) and self-registers via `customElements.define` at the bottom of its file. `components/index.js` imports them all and is imported by `Application.js`, so every element is defined site-wide. The singletons are created once in `Application.js` and exposed as `app.dialog` / `app.contextMenu` / `app.toast`. Self-contained presentational widgets become components; partial-bound controllers (votes, reactions, comments) and pure utilities stay plain modules because they enhance server-rendered markup rather than render standalone UI. They are documented in the public docs **Components** section.
- **Shared frontend utilities (do not re-implement these):** `Http` (`static/js/Http.js`) is the single fetch helper (`getJson`, `sendForm`, and `send` which throws `error.message` on non-2xx or a `200 {ok:false}` body); every live-update loop uses `Poller` (`new Poller(fn, intervalMs, {pauseHidden})`); async-job status polls use `JobPoller.run(statusUrl, {onDone, onFailed, onTimeout})` (`ProjectForker`, `ZipDownloader`); click-to-POST engagement controllers (`VoteManager`/`ReactionBar`/`BookmarkManager`/`PollManager`) extend `OptimisticAction` and call `this.submit(url, params, errorTarget, render)`. Floating windows (container terminals and Devii) extend `FloatingWindow`. See `AGENTS.md` -> "Shared frontend utilities (DRY)".
- CDN scripts in `base.html` (marked, highlight.js, emoji-picker-element) MUST use `defer` or `type="module"` - otherwise `wait_until="domcontentloaded"` in Playwright tests will time out.
- For clickable avatars/usernames, reuse `templates/_avatar_link.html` and `templates/_user_link.html` via `{% include %}` with `_user`, `_size`, `_size_class` locals.
- The three public listings (`/feed`, `/gists`, `/projects`) share one search box: the `templates/_sidebar_search.html` partial (included at the top of `.sidebar-card` with `_action`/`_placeholder`/`_hidden` locals) plus the `database.text_search_clause(table, search, fields)` data helper. Any new listing with a left filter panel reuses both - never re-inline an `ilike` search clause or hand-roll another search form. See `AGENTS.md` -> "Listing search".

### Content rendering pipeline
`ContentRenderer.js` runs on every element with `data-render`, in this exact order: emoji shortcode → markdown via `marked` → **`DOMPurify.sanitize`** → `highlight.js` on `<pre><code>` → image URL → YouTube URL → autolink. Uses a `NodeIterator` that skips `CODE`/`PRE`/`SCRIPT`/`STYLE` so URLs inside code blocks are never rewritten.

`DOMPurify.sanitize` runs on the raw `marked` output (untrusted) before `processMedia` injects trusted YouTube iframes, so user content cannot inject script/event-handler/iframe payloads while our own embeds survive. Sanitization is fail-closed: `render()` throws if `DOMPurify` is not loaded rather than emitting unsanitized HTML. Because content is rendered client-side from `element.textContent`, Jinja autoescaping alone does not protect it - this sanitize step is the XSS control. Server-side, JSON-LD emitted via `{{ page_schema | safe }}` is escaped in `seo.py` `_json_ld_dumps` (`<`/`>`/`&` and line separators → `\uXXXX`) to prevent `</script>` breakout.

### Auth
Session cookies named `session`, value is a 64-char hex token from `secrets.token_hex(32)`. Passwords hashed with `pbkdf2_sha256` via passlib. `get_current_user(request)` returns user dict or `None`, with a per-process TTL cache (`_user_cache`, 300s) keyed by token. `require_user(request)` raises a 303 redirect to `/` for guests; `require_admin(request)` redirects to `/feed` for non-admins. Public pages (feed, news detail, project detail, gists) use `get_current_user` so guests can browse - POSTs are all guarded by `require_user`.

### Database (`devplacepy/database.py`)
SQLite via `dataset.connect` with WAL + 30s busy timeout + 256MB mmap. `init_db()` is idempotent - every index is created via `CREATE INDEX IF NOT EXISTS` wrapped in try/except, and seed defaults for `site_settings` only insert when the key doesn't already exist. **`init_db()` also ensures the full column set of any table that code filters on** (`news`, `news_sync`, `attachments` use `get_table(name)` + `create_column_by_example` before their indexes). This is mandatory, not optional: dataset gives a lazily-created table ONLY the columns of its first insert, so a partial insert from a CLI tool/test/maintenance script would otherwise create the table with a reduced schema, and the long-running server caches that stale schema - making later queries on the missing column 500 with `no such column`. When adding a new filtered/indexed column, add it to the matching `init_db()` ensure-block (see `AGENTS.md` -> "Dataset rules -> `init_db()` MUST create every queried column").

**SQLite is synchronous by design and will never be made async (hard rule).** The `dataset`/SQLite layer is synchronous and is called directly inside the async route handlers. This is intentional and more than fast enough for this platform: the database is a local file tuned with WAL, `synchronous=NORMAL`, a 30s busy timeout, an 8MB page cache, and a 256MB mmap, so queries are sub-millisecond and never meaningfully stall the event loop. Do NOT convert the database layer to async, do NOT wrap DB calls in a threadpool/`run_in_executor`/`to_thread`, and do NOT raise the synchronous DB access as a performance, blocking-I/O, or "requests aren't fully async" concern. This is a settled, sophisticated design choice and is not open for revisiting.

Runtime config lives in `site_settings`, read via `get_setting(key, default)` / `get_int_setting(key, default)` (60s TTL cache, invalidated cross-worker via the `cache_state` version table - `get_setting` calls `sync_local_cache("settings", ...)`, writes call `bump_cache_version("settings")`; the `_user_cache` in `utils.py` uses the same primitive under the `auth` name). Besides site/news/upload keys, the **Operational** group is admin-editable at `/admin/settings`: `site_url` (public origin for absolute links incl. container ingress; resolved by `seo.public_base_url()` = setting -> `DEVPLACE_SITE_URL` env -> request origin), `rate_limit_per_minute`, `rate_limit_window_seconds`, `news_service_interval`, `session_max_age_days`, `session_remember_days`, `registration_open`, `maintenance_mode`, `maintenance_message`, `docs_search_mode` (`agent`|`bm25`, default `agent` - picks the `/docs/search.html` surface: the in-page Devii chat or the classic BM25 list; a viewer over their daily AI limit, or a guest when Devii is disabled, auto-falls-back to BM25 via `routers/docs/views.py` `_agent_search_state`). Consumers always pass the production default to `get_setting`, so behavior is correct even before the row exists. Numeric operational values are floored at the call site so an invalid `0` can't lock out writes or stall a service. Booleans are stored as `"0"`/`"1"` and rendered as `<select>` (not checkboxes) because the settings save handler skips empty form values - an unchecked checkbox could never be turned off.

Batch helpers (`get_users_by_uids`, `get_comment_counts_by_post_uids`, `get_vote_counts`, `load_comments`, `get_attachments_by_type`) exist specifically to avoid N+1 queries - use them in feed/listing routes instead of per-row lookups.

`dataset.find()` does NOT accept raw SQL. Use kwargs for equality, dict syntax (`{">=": value}`) for comparisons, or `table.table.columns.<col>.in_([...])` for IN clauses. `db.query("... :p", p=val)` is the raw-SQL escape hatch (named params as kwargs, not a dict). `table.update({"uid": x, ...}, ["uid"])` requires the key columns as a list. Always `if "table" in db.tables:` before raw SQL, since `dataset` only auto-creates tables when something is first inserted.

### Background task queue (`devplacepy/services/background.py`)
The `background` singleton is a generic, lightweight, fire-and-forget offload for non-critical side-effects: `background.submit(fn, *args, **kwargs)` enqueues a synchronous callable onto one in-process `asyncio.Queue` drained by a single consumer, so the work leaves the request path. It is **per-worker** (started in `main.py` `startup()` for every worker outside the service-lock branch, drained in `shutdown()`), unlike `BaseService`/`JobService` which run only in the lock owner - audit records are produced in every worker and a DB-backed queue would only add writes. When the consumer is not running (tests with `DEVPLACE_DISABLE_SERVICES=1`, unit tests, full queue), `submit` runs the callable **inline**, so audit/notification/XP writes stay deterministic for the suite while production defers them. Durability is best-effort in-memory (graceful shutdown drains; a hard crash drops unflushed items), which fits audit/notifications already never raising. Deferral happens at the **canonical choke points** so callers need no change: audit-log persistence (`services/audit/record.py` `_write`), **every XP award** (`utils.award_rewards` -> `_apply_rewards`, covering posts/comments/projects/gists/follow/votes), and **every notification** (`utils.create_notification` -> `_deliver_notification`, the single funnel). Because the reward and notification funnels self-defer, handlers call `award_rewards`/`create_notification` directly - never wrap them in `background.submit` (that double-queues). Handler pattern for a NEW side-effect: do the user-visible write inline, then `background.submit(...)` the rest. See `AGENTS.md` -> "Background task queue". **Feature/first-use badges** ride the same self-deferring model: call `utils.track_action(user_uid, action[, target])` inline at a feature's success point (do NOT wrap it - it `background.submit`s itself), where `action` is a key in `utils.ACHIEVEMENTS` (`action -> [(threshold, badge), ...]`, threshold 1 = first use; `UNIQUE_ACTIONS` use `record_unique_activity` for "N distinct things" like `docs.read`). Source-derivable milestones (counts of existing tables) instead go in `check_milestone_badges`/`_COUNT_MILESTONES`. Badge metadata + `group` live in `BADGE_CATALOG`; the profile shows a grouped earned/locked **Achievements** showcase via `build_achievements`. See `AGENTS.md` -> "Gamification".

### AI content correction (`devplacepy/services/correction.py`)
**Opt-in, default off, server-side.** When a user turns it on (profile **AI content correction** block, owner-only, saved at `POST /profile/{username}/ai-correction`), the prose they author is rewritten by the AI gateway. The per-user `ai_correction_sync` flag (0 = background default, 1 = sync) picks the **apply mode**: background rewrites the stored fields a moment after the write (never slows the request); sync makes the HTTP response wait until the correction is applied. **Sync must never block the event loop:** the correction POSTs to the in-process gateway (`localhost:{PORT}`) and the content helpers are synchronous on the loop thread, so a blocking inline call self-deadlocks the (single) worker against its own gateway request. Instead, sync runs `_run_correction` via `loop.run_in_executor` (loop stays free), stashes the future on `request.scope[PENDING_SCOPE_KEY]`, and the `await_pending_corrections` middleware in `main.py` awaits it after the handler. `CORRECTABLE_FIELDS` is the single source of truth mapping each table to its prose columns (`posts`->title/content, `projects`/`gists`->title/description, `comments`/`messages`->content, `users`->bio); `gists.source_code`, project files and Gitea issues are intentionally excluded (code and external systems). The choke entrypoint `schedule_correction(user, table, uid)` is a no-op unless correction is enabled and the user has an `api_key`, then it runs `_run_correction` off-thread via `loop.run_in_executor` with the future awaited by middleware (sync mode) or via `background.submit` (default), which reads the row back, calls the fail-soft synchronous `correct_text` (POST to `INTERNAL_GATEWAY_URL` via `stealth.stealth_sync_client`, authenticated with the user's OWN api_key for per-user attribution, returning the original on any error or suspiciously large output), and writes only changed fields without touching `updated_at` or the slug. It is hooked at `content.create_content_item`/`edit_content_item`/`create_comment_record`/`edit_comment_record`, `messaging/persist.persist_message`, `profile/index.update_profile`, and the two devRant direct-edit paths, so it applies identically across the web UI, REST/JSON API, Devii and devRant. Settings are three `users` columns (`ai_correction_enabled`/`ai_correction_sync`/`ai_correction_prompt`, ensured in `database.backfill_api_keys()`, seeded born-live in `utils._create_account`); Devii drives it via the owner-scoped `ai_correction_get`/`ai_correction_set` tools. Keep `correction.py` free of `content`/`utils` top-level imports (cycle); `clear_user_cache` is imported lazily. **Per-user usage aggregation:** `correct_text` returns `(text, usage)` parsed from the gateway `X-Gateway-*` headers on success, now including the timing headers `X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`; `_run_correction` accumulates the fields' usage into a `totals` dict and, only when a call succeeded, makes ONE atomic `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` upsert via `database.add_correction_usage(user_uid, totals)` (single dict arg) into the per-user `correction_usage` table (running SUMS `calls`/token/`cost_usd` plus the two latency columns, separate from `users` so it never busts the auth cache). `database.get_correction_usage` also returns computed AVERAGES (avg tokens/call, avg latency, avg tokens/sec, total time). The profile shows the sums plus those performance averages (`_correction_usage`, `correction_usage` on context + `ProfileOut`) to owner-or-admin; the dollar `cost_usd`/`avg_cost_usd` are admin-only in both HTML and JSON (mirrors `_ai_quota`). See `AGENTS.md` -> "AI content correction".

#### AI modifier (`devplacepy/services/ai_modifier.py`)
A sibling of AI content correction that reuses its plumbing (`CORRECTABLE_FIELDS`, `PENDING_SCOPE_KEY`, `gateway_complete`, the sync/background apply-mode mechanics, the per-user usage upsert) and differs only in two ways: it is **enabled by default** and **synchronous by default**, and it runs ONLY where an authored prose field contains an inline `@ai <instruction>` directive (`has_ai_directive`, regex `@ai\s+\S`). When present, the configured prompt (default `config.DEFAULT_MODIFIER_PROMPT` = "Execute what is behind `@ai` (the prompt) and replace that part including `@ai`") tells the model to execute the instruction and replace the marked part including the `@ai` marker; triggerless writes cost nothing. **It is context-aware:** `_run_modification` builds a grounding **context block** once per row via `services/ai_context.py` `build_context(table, uid, row, user_uid)` and passes it to `modify_text`, which appends it to the system message. The block (fail-soft, length-capped) carries the date, the author's stats (username/role/level/stars/posts/rank/followers/member-since/bio), and table-specific location/thread context (a comment's target post + parent reply; a post's topic/project; a project/gist's title/description, and a gist's language + read-only `source_code`; a DM's recipient + recent conversation), so `@ai answer the question above` / `@ai write my bio from my stats` / `@ai reply to this` work. `schedule_modification(user, table, uid, request)` is hooked alongside `schedule_correction` at the same content/comment/messaging/profile entrypoints. Settings are three `users` columns (`ai_modifier_enabled`/`ai_modifier_sync`/`ai_modifier_prompt`, seeded born-live in `utils._create_account`), edited at the owner-or-admin leaf `POST /profile/{username}/ai-modifier` (`routers/profile/ai_modifier.py`, `AiModifierForm`, audit `profile.ai_modifier`), surfaced owner-only on the profile (`ai_modifier_*` + `modifier_usage` on context + `ProfileOut`, dollar cost admin-only), wired by `static/js/AiModifier.js` (`app.aiModifier`), and driven via the Devii `ai_modifier_get`/`ai_modifier_set` tools (`services/devii/ai_modifier/`, `handler="ai_modifier"`). Per-user totals accumulate via `database.add_modifier_usage(user_uid, totals)` into the `modifier_usage` table (same SUMS-plus-latency columns and computed performance averages as correction, shown on the profile with dollar averages admin-only). See `AGENTS.md` -> "AI modifier".

### Background services (`devplacepy/services/`)
`BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation. `ServiceManager` is a singleton that registers, starts, and stops services. `NewsService` fetches articles from `news_api_url`, grades each via `news_ai_url`, and inserts ALL of them into `news` with `status="published"` or `"draft"` based on the configurable `news_grade_threshold` - nothing is silently skipped. The `/admin/services` page polls live status and the log tail.

`GatewayService` (`/openai/v1/*`) is the **single point of truth for AI**. Every other AI consumer (news, bots, Devii guests) defaults its endpoint to `config.INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override via `DEVPLACE_INTERNAL_BASE_URL`), sends the generic model `molodetz`, and authenticates with the auto-generated `gateway_internal_key` (via `database.internal_gateway_key()`) when its own key is unset. **Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default on); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key. **System-message composition (`services/openai_gateway/system_message.py`, `handle_chat` only):** before assembling the upstream chat payload (after vision augmentation, so it reaches every chat consumer) the gateway composes ONE system message in the fixed order **operator preamble -> date line -> client system content**. (1) The admin-editable `gateway_system_preamble` config field (Prompt group, multiline text, default empty) is prepended ahead of the client's system content on every chat call; if the client sent no system message it becomes a new leading one (a blank preamble is a no-op). (2) The **current date in EU `DD/MM/YYYY` (date only, never time** - omitting time keeps the message byte-stable per day so upstream prompt caching stays effective) is injected only when the composed text has NO date in any common format, detected by the `contains_date` regex set (ISO `YYYY-MM-DD`, `D/M/YYYY`, `DD-MM-YYYY`, `DD.MM.YYYY`, `MM/DD/YYYY`, and textual months like `14 June 2026`/`June 14, 2026`/`14 Jun 2026`). Embeddings/passthrough are untouched. Real provider keys/URLs/models live only in the gateway; `database.migrate_ai_gateway_settings()` (run in `init_db()`) generates the internal key, migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into the editable, non-secret `gateway_api_key`/`gateway_vision_key` settings, and rewrites old external AI URLs to the gateway. The gateway is `default_enabled=True`. The gateway also serves OpenAI-compatible text **embeddings** at `POST /openai/v1/embeddings`: clients send the client model `molodetz~embed`, which `handle_embeddings` remaps to the configured embedding model (default `qwen/qwen3-embedding-8b` on OpenRouter, $0.01/1M input tokens), tracked per call as `backend="embed"` in `gateway_usage_ledger`. The gateway records every upstream call to `gateway_usage_ledger` (cost from the upstream native `cost` field when present, else computed from per-1M pricing) and samples `gateway_concurrency_samples`; `services/openai_gateway/analytics.py` rolls these into per-hour/24h token, cost, latency, error, and behavior metrics shown on `/admin/ai-usage` (`AiUsageMonitor.js`, schema `GatewayUsageOut`, Devii action `ai_usage`). It also retries transient upstream failures and trips a circuit breaker (`services/openai_gateway/reliability.py`). TTFT/inter-token latency are not tracked because the upstream is called non-streaming. **Every gateway response also carries per-call `X-Gateway-*` headers** (model, full token breakdown, `Cost-USD`/input/output dollars, native-cost flag, tokens/sec, latency, context window/utilization): `GatewayUsageLedger.record` returns the computed ledger row and each handler's `finalize` maps it through `usage.usage_response_headers` onto the `Response`, so any caller can read its own spend (the AI-correction worker uses this).

To add a service: extend `BaseService`, override `async def run_once(self) -> None`, register in `main.py`'s startup with `service_manager.register(YourService())`. It auto-appears on the services page.

### Async job services (`devplacepy/services/jobs/`)
The **standard** way to run heavy, blocking work off the request path and return a result URL. A shared `jobs` table is the queue (discriminated by a `kind` column; "default" fields plus `payload`/`result` JSON, lifecycle timestamps, `retry_count`, `last_accessed_at`/`expires_at`, and `bytes_in`/`bytes_out`/`item_count` stat columns). `services/jobs/queue.py` (`enqueue`, `get_job`, `touch_job`, `list_jobs`) is pure DB and callable from **any** worker - the row is the queue. `JobService(BaseService)` runs only in the lock owner; each `run_once` reaps finished in-flight tasks, recovers orphaned `running` rows (left by a dead owner) back to `pending` with bounded `retry_count`, refills up to `max_concurrent` oldest pending jobs (uuid7 = FIFO) as `asyncio` tasks, and **sweeps expired artifacts** via the per-kind `cleanup()` hook (retention is built in, default 7 days, admin-configurable; no separate reaper). A new kind subclasses `JobService`, sets `kind`, and implements `async process(self, job) -> dict` and `cleanup(self, job)`. **`ZipService`** (kind `zip`) is the first consumer: `process` materializes a project subtree via `project_files.export_to_dir` (staging under `config.DATA_DIR/zip_staging/{uid}`), compresses it in a **subprocess** (`python -m devplacepy.services.jobs.zip_worker`, stdlib-only, prints stats JSON incl. crc32), names the output `{crc32}.{slug}.zip` under `config.DATA_DIR/zips/{uid[-2:]}/{uid[-4:-2]}/` (sharded on the random tail of the uuid7 via `attachments._directory_for`, not its time-ordered head - see note below), and returns stats. (Runtime artifacts live in `DATA_DIR` = `data/` by default, OUTSIDE the package and NOT under `/static`; the download is served by the `/zips/{uid}/download` route via `FileResponse`, not the static mount.) Enqueue endpoints own authz (`POST /projects/{slug}/zip`, `POST /projects/{slug}/files/zip?path=`); `GET /zips/{uid}` (status `ZipJobOut`) and `GET /zips/{uid}/download` (FileResponse, extends expiry) are **capability URLs** scoped only by the unguessable uuid7, not by owner. Frontend `app.zipDownloader` (`static/js/ZipDownloader.js`) auto-wires any `data-zip-download` element plus the files context menu: POST -> poll `/zips/{uid}` -> trigger download. Devii tools `zip_project`/`zip_status`; CLI `devplace zips prune|clear`. **`ForkService`** (kind `fork`, `services/jobs/fork_service.py`) is the second consumer and forks a project: `process` creates the destination project (via `content.create_content_item`, owned by the forking user), copies the whole virtual FS through `project_files.export_to_dir` -> `config.DATA_DIR/fork_staging/{uid}` -> `import_from_dir(..., skip_names=set())` (exact copy), and records the directional relation via `database.record_fork(source_uid, forked_uid, forked_by_uid)` into the `project_forks` table; on any failure it **rolls back** the half-created project (delete files + relation + row) and re-raises. **Critical difference from zip: the artifact is a permanent project, so `cleanup()` is a no-op on the project** - the retention sweep only removes the job tracking row, never the fork. Enqueue `POST /projects/{slug}/fork` (`require_user` + `can_view_project`, body `ForkForm{title}`); `GET /forks/{uid}` returns `ForkJobOut` whose `project_url` is set once done. Frontend `app.projectForker` (`static/js/ProjectForker.js`) wires `data-fork-project`: prompt for a name -> POST -> poll `/forks/{uid}` -> redirect to `project_url`. The forked project's detail page shows a "Forked from X" link (`database.get_fork_parent`). Devii tools `fork_project`/`fork_status`; CLI `devplace forks prune|clear` (job rows only). **`SeoService`** (kind `seo`, `services/jobs/seo/`) is the third consumer and powers the public **Tools -> SEO Diagnostics** auditor: `process` launches a stdlib+`playwright`+`httpx` subprocess (`python -m devplacepy.services.jobs.seo.worker <payload_json> <output_dir>`) that crawls a single URL or a sitemap (capped at `max_pages`, max 50), builds a `PageContext` per page (rendered DOM via one `page.evaluate`, Core Web Vitals via injected `PerformanceObserver`, screenshot, a raw `httpx` GET for SSR-vs-rendered parity), and runs the **check registry** (`services/jobs/seo/checks/`, `@page_check`/`@site_check` decorators across crawlability, meta, content, links, structured data, social, performance, mobile/accessibility, security, AI-readiness, cross-page) producing a weighted score + grade. The worker emits **NDJSON progress frames on stdout** (stage/target/progress/page/site_checks/report_ready); `service.process` pumps each line into the in-process **`ProgressHub`** (`services/jobs/seo/progress.py`, uid -> set of `asyncio.Queue`) and the `WS /tools/seo/{uid}/ws` (served only by the lock owner, like `/devii/ws`) replays the buffered snapshot then streams live. The full report is written to `config.SEO_REPORTS_DIR/{uid}/report.json` (NOT inlined on a stdout line - avoids the StreamReader limit) and returned as the job result; `cleanup()` removes the report dir. SSRF is guarded by the shared `devplacepy/net_guard.py` `guard_public_url` (extracted from the Devii fetch controller, reused by both). **`playwright` is a core dependency** (chromium installed by `make install` / the Docker image). Devii tools `seo_diagnostics`/`seo_status` (public); CLI `devplace seo prune|clear`. **`DeepsearchService`** (kind `deepsearch`, `services/jobs/deepsearch/`) is the fourth consumer and powers the public **Tools -> DeepSearch** multi-agent web researcher: `process` launches a stdlib+`playwright`+`httpx` subprocess (`python -m devplacepy.services.jobs.deepsearch.worker`) that plans sub-queries (`enhance.py`), searches via rsearch, crawls + de-duplicates sources (`crawl.py`, `guard_public_url` on every URL incl. redirects; PDF URLs are streamed to a temp file and text-extracted via `pdf.py`/`pypdf`, source `pdf`), chunks (`chunking.py`), embeds them through the AI gateway (with a deterministic local fallback in `services/deepsearch/embeddings.py`) into a **per-session ChromaDB collection** (`services/deepsearch/store.py`, `chromadb.PersistentClient` at `config.DEEPSEARCH_CHROMA_DIR`), and runs summarizer/critic/linker agents (`orchestrate.py`) producing a cited report with confidence/source-diversity/gap analysis. NDJSON frames pump into the `ProgressHub` and `WS /tools/deepsearch/{uid}/ws` (lock-owner gated). Report at `config.DEEPSEARCH_DIR/{uid}/report.json`; `cleanup()` drops the collection + dir. Pause/resume/cancel write `control.json` polled by the worker. A grounded RAG chat (`services/deepsearch/chat.py`, hybrid vector+BM25 retrieval, `<dp-deepsearch-chat>`) runs at `WS /tools/deepsearch/{uid}/chat`, answers cited and rendered via `dp-content`, turns in `deepsearch_messages`. Cross-session dedup via the GC-only `deepsearch_url_cache` table. Export md/json/pdf (`services/deepsearch/export.py`, PDF via weasyprint). Viewer flags are `viewer_is_admin`/`viewer_owns`. Devii tools `deepsearch`/`deepsearch_status`/`deepsearch_session` (public); CLI `devplace deepsearch prune|clear`. New unpinned deps `chromadb`, `weasyprint`, `pypdf`.

### Issue tracker (`devplacepy/services/gitea/`)
The `/issues` feature is a **full Gitea integration with no local issue store** - the listing and detail views read issues straight from Gitea (default repo `retoor/devplacepy`) with live status. `services/gitea/` holds `config.py` (the admin-editable `CONFIG_FIELDS` for base URL/owner/repo/token + AI model, surfaced on the `IssueTrackerService` config at `/admin/services`; `gitea_config()` -> `GiteaConfig`), `client.py` (async `GiteaClient` over the REST API, `Authorization: token`, `GiteaError`), `fake.py` (`FakeGiteaClient` test double), `runtime.py` (`get_client`/`set_client` swap), `enhance.py` (`enhance_ticket` rewrites a raw report into a consistent markdown ticket via the **internal AI gateway**, fail-soft to a deterministic template), `store.py` (the only DB state: `issue_tickets` and `issue_comment_authors`, both in `SOFT_DELETE_TABLES` - thin mappings of Gitea number/comment-id -> DevPlace author for attribution + cached `last_status`/`last_comment_count` for update detection), and `service.py` (`IssueTrackerService`, `default_enabled=False`, polls tracked tickets and notifies the **reporter** on developer replies / status changes). Filing is an **async job** (`IssueCreateService`, kind `issue_create`): enhance -> create Gitea issue -> record mapping -> notify reporter; `POST /issues/create` enqueues, frontend `app.issueReporter` polls `/issues/jobs/{uid}`. Comments (`POST /issues/{number}/comment`, notifies **all admins**) and admin status changes (`POST /issues/{number}/status`, open/closed) are synchronous single Gitea calls. All Gitea actions use one shared bot token; per-user identity is shown by storing DevPlace author names locally, never per-user Gitea accounts.

### Audit log (`devplacepy/services/audit/`)
**Admin-only, append-only** record of every state-changing action; the authoritative event catalogue (199 keys across 34 domains) is `events.md`. Package: `store.py` (tables `audit_log` + `audit_log_links`, `ensure_tables`/`insert_event`/`insert_links`/`get_event`/`get_links`/`sweep`), `categories.py` (`category_for(event_key)`), `record.py` (the recorder + link builders), `query.py` (admin list/detail), `service.py` (`AuditService` retention). `ensure_tables()` + 11 indexes are wired into `init_db()` via a local import (audit modules import `database`/`utils`, so the import is deferred to avoid the cycle); `AuditService` is registered in `main.py`. **Two recorder entrypoints (the reuse keystone):** `record(request, event_key, *, user=_UNSET, actor_kind, target_*, old_value, new_value, summary, metadata, result, origin, via_agent, links, category)` for HTTP/WebSocket handlers (`request` may be a `Request` OR a `WebSocket`; auto-derives actor from `get_current_user`, request fields, the `X-Devii-Agent` header -> `origin=devii`/`via_agent=1`, and auto-appends the `actor` link), and `record_system(event_key, *, actor_kind, actor_uid, actor_username, actor_role, origin, via_agent, ...)` for request-less contexts (rewards, notifications, the AI gateway ledger, news/container services, Devii turns/tasks, job services, CLI). **Both are best-effort: wrapped in try/except, they NEVER raise into the caller** - the audited action is never blocked by a logging failure. `summary` is HTML-stripped + 140-char capped; `metadata` is JSON. **Emission is explicit per mutation** (`audit.record(...)`/`record_system(...)` after success, or on the guard branch with `result="denied"`/`"failure"`), with DRY choke points: posts/projects/gists record inside `content.py`; project file ops via the `project_files.py` helpers (read-only guard -> `denied`); container ops in `routers/containers.py` (HTTP path) and in the Devii `actions/dispatcher.py` `_audit_mechanic` hook (agent path; the two paths are disjoint so there is no double-count), which also emits the `devii.*` self-config mechanics. Auth failures, authz denials (in `require_user`/`require_admin`), rate-limit/maintenance blocks, self-role-change/self-disable denials, and quota blocks are recorded with `result` set. **Admin UI:** `GET /admin/audit-log` (paginated, filterable list) + `GET /admin/audit-log/{uid}` (detail + links), both `require_admin`, both negotiate via `respond(..., model=AuditLogOut|AuditEventOut)`; sidebar link sits last (before Settings). `_pagination.html` gained an optional backward-compatible `pagination_query` prefix so filters survive paging. **Devii (read-only):** admins query the same two routes conversationally via the `audit_log`/`audit_event` admin actions (`services/devii/actions/catalog.py`, `handler="http"`, `requires_admin=True`) - no new endpoint, since the `respond(...)` routes already serve JSON to Devii's `Accept: application/json` client (like `admin_list_users`); `audit_log` forwards every filter as a query param and its `AuditLogOut.options` lets the agent discover valid filter values in one call. **Retention:** `AuditService` (daily) prunes rows older than `audit_log_retention_days` (default 90; `0` disables). To add an event: pick/extend a key in `events.md`, extend `category_for` for a new domain, and call the recorder at the mutation point - never gate the action on the recording.

### Container manager (`devplacepy/services/containers/`)
**Admin-only.** Runs supervised container instances, all on one shared prebuilt image. Driven through the `docker` CLI via `asyncio.create_subprocess_exec` behind a pluggable **`Backend` ABC** (`backend/base.py`); `DockerCliBackend` streams run logs line-by-line, `FakeBackend` backs tests; `runtime.get_backend()`/`set_backend()` selects it. **Security: this requires mounting the docker socket, which is root-equivalent on the host - every run/exec/lifecycle/schedule op is `require_admin` (HTTP) and `requires_admin=True` (Devii); never `--privileged`; all calls are arg-list subprocesses (never `shell=True`).** **There is NO in-app image building.** Every instance runs ONE shared prebuilt image, `config.CONTAINER_IMAGE` (default `ppy:latest`, override `DEVPLACE_CONTAINER_IMAGE`), built once from `ppy.Dockerfile` via `make ppy`. There are no per-project Dockerfiles, builds, or `ContainerBuildService`. Data model (`store.py`, indexed in `init_db`): `instances` + `instance_events` + `instance_metrics` (ring-buffered, METRICS_RING=720) + `instance_schedules`. `api.create_instance(project, *, name, ...)` fails fast with a `ContainerError` if `backend.image_exists(CONTAINER_IMAGE)` is false (run `make ppy`); it no longer takes a dockerfile/build. **`ContainerService`** (`service.py`, a reconciler `BaseService`) each tick snapshots `docker ps` (label `devplace.instance=<uid>` is the join key), converges each instance to its `desired_state`, applies restart policies, reaps orphan containers (labeled but no DB row -> no orphans/no lost state), fires due `instance_schedules` (reusing `devii/tasks/schedule.py` `cron_next`/`next_run`), and samples `docker stats`. Only the service lock owner reconciles; HTTP handlers only edit `desired_state`. The `/app` workspace is materialized once per project from the virtual FS (`project_files.export_to_dir`) under `config.CONTAINER_WORKSPACES_DIR` (`DATA_DIR/container_workspaces`, default `data/`), bind-mounted RW; the sync action imports it back (`project_files.import_from_dir`). **Runtime data must live OUTSIDE the `devplacepy/` package**: workspaces in `DATA_DIR` (configurable via `DEVPLACE_DATA_DIR`, NOT under `/static`). `api.py` holds the operations shared by `routers/containers.py` and the Devii `ContainerController` (`services/devii/container/`, `handler="container"`). **UI:** the per-project manager is `templates/containers.html` + `static/js/ContainerManager.js` (instance creation via app modal forms; its instance list links to the shared detail page). The admin **Containers** section (`routers/containers_admin.py`, `templates/containers_admin.html` + `static/js/ContainerList.js`, sidebar link in `admin_base.html`) lists all instances across projects and links each row to the dedicated detail page `templates/containers_instance.html` + `static/js/ContainerInstance.js` (poll-based logs/metrics, lifecycle, schedules, ingress, sync, a one-shot exec box, and an **Open terminal** button). The admin detail page does NOT add its own lifecycle endpoints - it reuses the per-project `/projects/{slug}/containers/instances/{uid}/...` routes (the instance carries its `project_uid`). All container UI styling uses the app design tokens in `static/css/containers.css`. **Floating terminals (interactive shell):** an instance's shell is a floating **xterm.js** window (`static/js/components/ContainerTerminal.js`; xterm + fit addon vendored under `static/vendor/xterm/`, lazy-loaded), opened by the instance-page **Open terminal** button (`app.containerTerminals.open(slug, uid, name)`) or by telling Devii to "attach &lt;container&gt;" (the admin-only `open_terminal` **client** action -> `DeviiClient._openTerminal` -> the same manager). It extends the generic **`FloatingWindow`** base (`static/js/components/FloatingWindow.js` + `static/css/floating-window.css`): draggable/resizable/maximize/fullscreen chrome identical to the Devii window. **The Devii terminal (`devii/devii-terminal.js`) also `extends FloatingWindow`** and reuses the same drag/geometry/preset/resize/persist plus `_window`/`_registerWindow` machinery, overriding only what differs (a `closed` state + launcher FAB, bottom-right default geometry, per-user `--devii-font-size` font, `_contextItems`, and the richer `{state,geometry,focused}` persistence); `devii.css` is unchanged, the only base change being a `get _dragClass()` getter Devii overrides to `devii-dragging`. `app.windows` (`WindowManager`) gives every floating window - Devii included - a shared **last-clicked-on-top** z-order (below the avatar/toasts). The exec WS (`/projects/{slug}/containers/instances/{uid}/exec/ws`, `pty.openpty()` + `docker exec -it`, admin + lock-owner gated) now also accepts a JSON `{"type":"resize","cols","rows"}` control frame: it sets the host pty winsize (`TIOCSWINSZ`) **and** signals the `docker exec` client (`proc.send_signal(SIGWINCH)`) so the new size forwards through to the container's exec tty (vim/top reflow) - the SIGWINCH is required because the host slave is not the client's controlling terminal. Any non-resize frame is raw stdin (backward compatible). The fit addon drives it client-side and the window has a dedicated bottom-right resize grip (`.fw-resize`, since xterm covers the native CSS resize corner). xterm renders ANSI natively, so the old `cleanTerm` stripping is gone for the interactive shell. **Persistence:** a `?session=<name>` query param makes the WS run the shell inside **tmux** (`tmux new-session -A -s <name>`, falling back to bash if tmux is absent) so the session survives WS disconnect/window close (a running server keeps running; closing the WS only detaches - `proc.kill` kills the tmux *client*, not the server). The frontend keeps **one persistent slot** at a time (`ContainerTerminalManager`, the last-opened terminal): it attaches to session `devplace`, **auto-reconnects** with backoff on unexpected close, and is remembered per-user (`ct-persistent:<scope>` in localStorage) so `app.containerTerminals.restore()` (called once in `Application.js`) reopens it on the next page load. Opening another terminal demotes the previous one to non-persistent (its tmux session lingers but the frontend stops auto-managing it). **Right-click** any window opens the shared `dp-context-menu` (`app.contextMenu.attach`): the container terminal menu has Copy (xterm `getSelection`) / Paste (clipboard -> WS stdin) / Sync files / Restart / Terminate (confirm via `dp-dialog`) / Minimize / Normalize / Project (-> `/projects/{slug}`) / Files (-> `/projects/{slug}/files`) / Close; the Devii menu has Copy (selection or input line) / Paste (clipboard -> input at caret) / Minimize / Normalize / Close (it is not bound to a container or project). **Minimize** (smallest usable size) and **Normalize** (comfortable size) are geometry presets (`_presetGeometry`), exposed both as titlebar buttons and menu items on every window. The reconciler defaults disabled (needs docker); enable on `/admin/services`. **The `ppy` image** (`ppy.Dockerfile`, built by `make ppy` with context `devplacepy/services/containers/files`) is a `python:3.13-slim-bookworm` base with Playwright + a broad set of common Python libraries preinstalled, plus CLI tools (`tmux`, `apache2-utils` for `ab`, `procps`/`htop`/`iftop`/`iotop`, `netcat-openbsd` for `nc`, `zip`/`unzip`, `fakeroot`, git/curl/wget/vim/ack), plus the security hotpatch baked in: it `COPY`s the **sudo superclone** (`files/sudo`) over `/usr/local/bin/sudo` (+ symlink `/usr/bin/sudo`; the real `sudo` package is not installed), the **`aptroot` fakeroot wrapper** (`files/aptroot`, symlinked over `apt`/`apt-get`/`dpkg` in `/usr/local/bin` so pravda installs system packages without root), and **`pagent`** (`files/pagent`, the stdlib AI agent) to `/usr/bin/pagent.py` (plus `files/.vimrc` -> `/home/pravda/.vimrc`, whose AI helper targets the same gateway as pagent via `PRAVDA_OPENAI_URL`/`PRAVDA_API_KEY`), evicts any pre-existing uid-1000 user, creates user **`pravda` at `1000:1000`**, hands pravda ownership of the toolchain AND the OS package trees (`chown -R pravda` over `/usr/local/lib`, `/usr/local/bin`, `/usr/lib/python3`, `/opt`, `/app`, `/home/pravda`, plus `/usr/lib`, `/usr/bin`, `/usr/sbin`, `/usr/share`, `/usr/include`, `/etc`, `/var/lib`, `/var/cache`, `/var/log`, `/srv` so `apt`/`dpkg` can write; `~/.local/bin` on `PATH`), and ends on `USER pravda`. **Why uid 1000:** `/app` is bind-mounted from the host and under DooD a container's UID maps 1:1 to the host, so a container running as **root** wrote root-owned files into the workspace, after which the app process (host `retoor`, uid 1000) hit `[Errno 13] Permission denied` in `export_to_dir` on the next instance create - a permanent per-project brick. Pinning the container UID to 1000 makes every write land as `retoor`, and pravda owning the global site-packages means runtime `pip install` succeeds with no elevation. **The sudo superclone** (`files/sudo`, POSIX `sh`) is sudo-CLI-compatible but **never swaps user**: it parses the full flag interface (`-u/-g/-E/-H/-n/-S/-i/-s/--`, leading `VAR=value` assignments, `-V/-l/-v/-k` short-circuits), exports `SUDO_USER`/`SUDO_UID`/`SUDO_GID`/`SUDO_COMMAND`, then `exec`s the command **as pravda (uid 1000)**, so even a blind `sudo ...` cannot create a root-owned file. **Rootless apt (the `aptroot` fakeroot wrapper):** pravda can run `apt`/`apt-get`/`dpkg` directly (no sudo) and install system packages. `files/aptroot` is symlinked over `apt`/`apt-get`/`dpkg` in `/usr/local/bin` (ahead of `/usr/bin` on `PATH`) and execs the real tool under **`fakeroot`** with `APT::Sandbox::User=root`, so dpkg's chown-to-root calls are virtualized while every file actually written lands owned by **pravda (uid 1000)**. Combined with pravda owning the system trees (`/usr`, `/etc`, `/var/lib`, `/var/cache`, `/var/log`, `/srv`, ...), `apt install <pkg>` just works with no escalation and still cannot create a root-owned file on the bind mount (there is no real euid 0). It runs `apt update` first (the image clears `/var/lib/apt/lists`); a package whose maintainer script needs genuine privileged syscalls may still fail - bake those into `ppy.Dockerfile` + `make ppy`. **Trade-off (intentional):** the only genuinely-root op that still does NOT escalate is binding a port < 1024 - use a high port + `/p/<slug>` ingress. The `export_to_dir` unlink-before-write fix remains as belt-and-suspenders. `pagent` reads `PRAVDA_OPENAI_URL`+`PRAVDA_API_KEY` (the gateway), falling back to its public endpoint + `DEEPSEEK_API_KEY`. **`PRAVDA_*` env injection:** `api.run_spec_for` merges `api.pravda_env(instance)` over the instance `env_json` (PRAVDA wins) so every container is launched with `PRAVDA_BASE_URL` (`site_url` via `seo.public_base_url()`), `PRAVDA_OPENAI_URL` (`{base}/openai/v1`), `PRAVDA_API_KEY` (the **creating** user's `api_key`, resolved live), `PRAVDA_USER_UID` (the project **owner** uid), `PRAVDA_CONTAINER_NAME`, `PRAVDA_CONTAINER_UID`, `PRAVDA_INGRESS_URL` (absolute `{base}/p/{ingress_slug}` when the instance is published and `site_url` is set, else relative `/p/{slug}`, else empty) - injected at run via `docker run -e` (never baked into the shared image, never stored as a DB secret). `create_instance` records `created_by` (actor user) and `owner_uid` (`project["user_uid"]`) to back the api-key/uid; pre-existing instances degrade to empty key/uid until recreated, and the base/openai URLs are empty unless `site_url` is set. **Ingress:** an instance can set `ingress_slug` + `ingress_port` (a mapped container port); `routers/proxy.py` (`/p/{slug}`, registered in `main.py`) then **publicly** reverse-proxies HTTP and WebSocket to the address from `api.proxy_target(instance)` (`store.find_instance_by_ingress`; target host/port are derived from the instance, not user input, so no SSRF). **`proxy_target` dials the container's docker bridge gateway + the published host port** (`gateway:host_port`), NOT loopback and NOT the container's own bridge IP. The reconciler records `container_ip`/`container_gateway` from `docker inspect` on every running tick. This is deliberate: `127.0.0.1` fails where docker's `nat OUTPUT` excludes `127.0.0.0/8` from DNAT and no `docker-proxy` binds loopback for a `0.0.0.0`-published port, and the container's own bridge IP can be dropped by docker's bridge-isolation rule (`DOCKER ! -i docker0 -o docker0 -j DROP`); the gateway+published-port path survives both. Setting `DEVPLACE_CONTAINER_PROXY_HOST` overrides the host (e.g. `host.docker.internal` for a containerized app) and still uses the published host port; before a gateway is recorded the proxy falls back to `127.0.0.1`. The Devii container tools return an absolute `ingress_url` built from `seo.public_base_url()` (the admin `site_url` setting), so the agent fetches the public production URL, not localhost (which web tools refuse); unset `site_url` yields a relative `/p/<slug>`. **Production** needs heavy wiring (the container manager drives the host docker daemon), all carried by the opt-in `docker-compose.containers.yml` overlay (docker socket mount, `INSTALL_DOCKER_CLI=true`, `group_add` the docker gid). **The `make docker-build`/`make docker-up` targets always apply this overlay** and self-derive its inputs - `DOCKER_GID` via `stat -c '%g' /var/run/docker.sock` (the socket's owning group) and `DEVPLACE_DATA_DIR` = `$(CURDIR)/data` (the project's own dir at its real host absolute path) - so the manager works out of the box with no `sudo`, no `/srv`, no manual `.env`. A plain `docker compose up -d` drops the overlay (no CLI, no socket) and silently re-breaks it; always update through the make targets. The **DooD bind-mount gotcha**: `docker run -v <path>:/app` resolves `<path>` on the HOST, so `DEVPLACE_DATA_DIR` must be mounted at an identical host+container path (the make targets use `$(CURDIR)/data` on both sides; manual `docker compose` users get the `/srv/devplace-data` default); build contexts go via the docker API tarball so temp is fine. The ingress proxy reaches published ports through each instance's recorded docker bridge gateway (`api.proxy_target` -> `gateway:host_port`); set `DEVPLACE_CONTAINER_PROXY_HOST=host.docker.internal` only when the containerized app cannot route to that gateway. nginx proxies `/p/` with WS upgrade + long timeouts.
- **Bidirectional newer-wins sync (load-bearing direction rule).** Sync is NOT one-directional import. `project_files.sync_dir_bidirectional(project_uid, workspace, user) -> {"exported", "imported"}` is the one helper that reconciles a project's virtual FS against an instance `workspace_dir`: per file the side with the newer timestamp wins (project `updated_at` epoch vs filesystem `st_mtime`, with a 1s skew tolerance favouring export on ties), and a file present on only one side propagates to the other. It NEVER deletes a file - only creates/overwrites the older side. A **read-only** project (`is_readonly`) exports only, never imports (the read-only guard direction). The reconciler calls it (via `api.sync_bidirectional_sync`, a non-blocking `record_event` system actor) before every `_launch` AND on a ~60s wall-clock cadence over running instances (gated by `SYNC_EVERY_SECONDS`, independent of the 5s reconcile tick). Both `api.sync_workspace` (HTTP/Devii) and the reconciler reuse the same helper; the per-instance sync helper files (`.devplace_boot.py`/`.devplace_boot.sh`) are in `SYNC_SKIP_NAMES` so they never round-trip into the project.
- **Run-as user = identity + API key ONLY (load-bearing constraint).** An instance's `run_as_uid` column selects WHICH DevPlace user's identity and `api_key` are injected (`PRAVDA_API_KEY`, `PRAVDA_USER_UID`), resolved in `api.pravda_env` ahead of the `created_by`/`owner_uid` fallback chain. It does NOT change the container OS user, which is ALWAYS `pravda` (uid 1000) - that is required for the bind-mounted `/app` (DooD uid maps 1:1 to host). Validate it against an existing user (`api.validate_run_as`).
- **Boot source precedence (load-bearing).** New columns `boot_language` (`none`|`python`|`bash`) + `boot_script` (multiline source) sit alongside the legacy `boot_command`. Precedence in `api.run_spec_for`: `boot_script` (by language) > `boot_command` > image CMD (`sleep infinity`). When a boot script is set, the reconciler writes it into the workspace (`api.materialize_boot_script`, `.devplace_boot.py`/`.devplace_boot.sh`, excluded from sync) before launch and runs `python|bash /app/.devplace_boot.<ext>`. `api.validate_boot` enforces the language set and a 100k char cap.
- **start_on_boot is per-container.** The `start_on_boot` integer flag (0/1) forces `desired_state=running` ONLY for flagged instances when `ContainerService` first runs (`_boot_pass`, one-time); all others keep their last desired_state.
- **Status-change choke point.** Every reconciler status transition flows through `service._set_status(inst, changes, reason=)`, which diffs old vs new `status`, writes an `instance_events` `status_change` row AND an audit `container.instance.status` `record_system` event on change (formerly-silent `running->stopped`/`->paused`/exit transitions are now logged). The admin detail page renders the events list as a **Status history** card. New code that changes an instance's status in the reconciler must use `_set_status`, never a raw `store.update_instance(... status ...)`.
- **New columns are ensured in `init_db` and defaulted in `store.create_instance`.** `run_as_uid`/`boot_language`/`boot_script`/`start_on_boot` are added to the `instances` ensure-block (filtered/queried columns must exist on the cached schema) and seeded in the `store.create_instance` base dict so pre-existing rows degrade gracefully.

### Devii assistant (`devplacepy/services/devii/`)
`DeviiService` is the in-platform agentic assistant (a relocated standalone agent), exposed as a WebSocket terminal at `/devii/ws` (router `routers/devii.py`, prefix `/devii`) and reachable from the **Devii** user-dropdown item (`data-devii-open`) and auto-opened on `/docs` (`data-devii-autoopen`). Frontend is `static/js/DeviiTerminal.js` (an `Application.js`-instantiated controller, `app.devii`) plus the `devii-terminal`/`devii-avatar` custom elements and ported renderers under `static/js/devii/`, the vendored md-clippy avatar under `static/vendor/md-clippy/`, and `static/css/devii.css`. A signed-in user's Devii operates their **own** account (PlatformClient base_url = this instance, auth = the user's `api_key`); guests get a sandbox. `DeviiHub` keys one `DeviiSession` per owner-and-**channel** `(owner_kind, owner_id, channel)` - `user`/uid or `guest`/cookie, channel `main` (default, the floating terminal) or `docs` (the in-page docs chat, selected via the WS query param `?channel=docs`, allowlisted in `routers/devii.py`) - broadcasting every frame to all of that owner-channel's sockets. A channel is an **independent conversation thread only**: lessons, behavior, virtual tools, tasks, and the 24h quota stay owner-scoped (shared across channels); just `devii_conversations` is keyed `(owner_kind, owner_id, channel)`. The docs page no longer auto-opens the floating terminal; `/docs/search.html` hosts the docs-styled in-page chat component `<dp-docs-chat>` (`static/js/components/AppDocsChat.js`, light DOM, reuses `DeviiSocket` + the devii markdown renderer, styled via `static/css/docs-chat.css`), and the sidebar search box seeds it. The `docs` channel is branded **Docii**, a documentation-only assistant: `session.py` gives it a dedicated `DOCS_SYSTEM_PROMPT` (forces `search_docs`, recursively refine-and-search until grounded, never answer from memory, no platform actions, and ALWAYS emit inline markdown links to doc `url`s plus a `## References` list with BM25 `score`s) with NO behavior-header appended, and `_builtin_tools()`/`_refresh_tools()` restrict the channel to the single `search_docs` tool (filtered by name via `DOCS_TOOLS`, no virtual tools); `bootstrap_greeting()` returns a Docii greeting. `search_docs` delegates to `docs_search.search_pages()` (in-process page index) so each result carries `url`/`score`/`content`, admin-filtered. A **topic gate** runs before each docs turn (`_docs_topic_gate` -> `_classify_topic`, a no-tools `complete_text` call): a new/unrelated question resets the conversation and emits `{type:"topic", decision:"new", reason}` (frontend clears the screen, re-shows the question, prints the reasoning); a follow-up keeps context. **Isolation:** the docs channel uses an ephemeral `owned_db` (private empty task/lesson/behavior/virtual-tool stores) and its **Scheduler only starts in `main`**, so Devii's scheduled tasks never fire inside Docii (and a Docii reflection never pollutes Devii memory). Everything is keyed on `self.channel == "docs"`, so the `main` channel (floating Devii) is unchanged. Conversation history persists to `devii_conversations` (rehydrated on reconnect, surviving restarts); per-turn cost goes to `devii_usage_ledger` (the authoritative 24h-spend source - the in-memory `CostTracker` is display-only) and an audit row to `devii_turns`; tasks live in `devii_tasks` (guests use an in-memory store). The self-learning `LessonStore` (reflect/recall) is **owner-scoped and never shared** - `LessonStore(db, owner_kind, owner_id)` filters every read/write by owner, persistent in `devii_lessons` for users and ephemeral (`memory_db()`) for guests; `forget_lessons` purges them. (Never share one `LessonStore` across sessions - it leaks one user's lessons into others' recall.) The 24h `$` cap is checked before each turn. The effective cap is resolved by `config.effective_daily_limit(cfg, owner_kind, is_admin)` (the single source of truth, used by both `service.daily_limit_for` and `build_settings`): guests use `devii_guest_daily_usd` (default $0.05), users `devii_user_daily_usd` ($1.00), and **administrators are exempt by default** via `devii_admin_daily_usd` (default `0.0`, where **`0` = unlimited**). All three are editable on the Devii service config (`/admin/services`). The WS gate is `if limit > 0 and spent >= limit` so a `0` cap never blocks. Quotas are **resettable** (clearing the owner's `devii_usage_ledger` rows): per-user via `POST /admin/users/{uid}/reset-ai-quota` (button on the admin Users page), globally via `POST /admin/ai-quota/reset-guests` and `/reset-all` (buttons on `/admin/ai-usage`), and from the CLI via `devplace devii reset-quota <username> | --guests | --all`. **Financial data is admin-only:** any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the **percentage** of their 24h quota used. The owner's admin status is resolved server-side (`is_admin(user)`) and threaded WS → `hub.get_or_create(is_admin=)` → `DeviiSession` → `Dispatcher`. The `Action` dataclass has a `requires_admin` flag (`cost_stats` is admin-only USD; the member-safe `usage_quota` returns only `{used_pct, turns_today, limit_reached}`), gated twice: `Catalog.tool_schemas_for(authenticated, is_admin)` withholds the schema from non-admins and the dispatcher raises `AuthRequiredError` for any `requires_admin` action a non-admin attempts. `GET /devii/usage` adds `spent_24h`/`limit` only for admins; the standalone `devii` CLI runs `is_admin=True`. The catalog cost/analytics tools `ai_usage` and `site_analytics` are `requires_admin=True` too. The profile route's `_ai_quota(include_cost=viewer_is_admin)` emits `spent_usd`/`limit_usd` only for admins in **both** the HTML and the `respond(..., model=ProfileOut)` JSON form (hiding dollars in the template alone would still leak them to a member fetching their own profile with `Accept: application/json`). For aggregate questions Devii calls a documented, admin-secured HTTP endpoint `GET /admin/analytics` (`routers/admin.py`, JSON, `is_admin`→403; backed by `database.get_platform_analytics()` - one cached UNION query for active-user windows/signups/totals/top authors) via the normal `http` catalog action `site_analytics`, so it never paginates lists to count and behaves the same on web and the `devii` CLI (the platform enforces admin, not the agent). The system prompt also forbids guessing routes (use `search_docs` first). Devii also has EXTERNAL, non-platform web tools under `services/devii/rsearch/` (`handler="rsearch"`, specs in `actions/rsearch_actions.py`): `rsearch` (web/image search), `rsearch_answer` (web-grounded AI answer), `rsearch_chat` (direct AI chat), `rsearch_describe_image` (vision). They call `https://rsearch.app.molodetz.nl` via a standalone `httpx` client (never `PlatformClient`); the system prompt's "REMOTE WEB TOOLS" rule makes the agent prefer platform tools always and use `rsearch_*` only when the user explicitly asks to search the web/an outside source. Because hubs are per-process, `/devii/ws` is served **only** by the worker holding the service lock (`service_manager.owns_lock()`); a non-owner worker closes with the application code **4013** (private 4000-4999 range, reliably surfaced in the browser `CloseEvent.code`), which `DeviiSocket.js` treats as a silent fast-retry (~200ms, no visible "disconnected" churn) until it lands on the owner. The real "connected." notice fires on the first server frame (`onReady`), not on raw socket open, so the multi-worker bounce is invisible. The disabled-service path still closes with the standard `1013` (a genuine, visible disconnect). A `devii` console script (`pyproject.toml [project.scripts]`) ships the same agent as a standalone terminal.

Devii also drives the **user's own browser** through a `client` tool channel (same WS request/response pattern as the avatar, via `_browser_request`): `services/devii/client/ClientController` + `actions/client_actions.py` expose `get_page_context`, `run_js`, `highlight_element`/`clear_highlights`, `show_toast`, `scroll_to_element`, `navigate_to`, `reload_page` - executed by `static/js/devii/DeviiClient.js` and routed by the terminal (`type:"client"` → `client_result`). This powers live on-screen tutorials, page-context awareness, and navigation/refresh. `run_js` is gated by the `devii_allow_eval` config field (default on). Because one session spans all of an owner's tabs, a client/avatar request is dispatched to a single **target** tab via `_pick_target()` (ranked by reported `(focused, visible, attach_seq)` from the terminal's `{"type":"visibility"}` frames), not broadcast and not the last-attached socket - otherwise `reload_page` reloads a hidden background tab while reporting success.

Devii's project filesystem tools include surgical **line-range editing** (`project_read_lines`, `project_replace_lines`, `project_insert_lines`, `project_delete_lines`, `project_append_file`) so it edits large text files without resending them, and `Dispatcher._run_http` enforces an **overwrite-protection guard**: `project_write_file` is rejected only when the target `(slug, path)` **already exists** (the guard probes `GET /projects/{slug}/files/raw`) and the agent has not read it this session (call `project_read_file` first); creating a new file needs no prior read. Reads are recorded in the per-session `Dispatcher._read_files` set. This steers the agent to update files rather than blindly overwrite them and is agent-scoped only (the human UI and public HTTP API are unaffected). The `WRITING AND EDITING FILES` system-prompt rule mirrors it.

### Per-user customization (`devplacepy/customization.py`)
Users and guests inject their own **CSS** (look) and **JS** (behaviour), scoped to a **page type** or **globally**, configured **only through Devii** (no HTTP routes - persistence is owner-scoped from the trusted Devii session, like `LessonStore`). It is **per-owner only**: the code runs solely in that owner's own browser sessions (self-XSS, like a userscript), never in anyone else's view - never key injection off URL-supplied data. **Page type** = the matched route template (`request.scope["route"].path`, e.g. `/posts/{slug}`) via `page_type_for(request)`; one value covers all instances of a type (all posts), never one post; also rendered as `data-page-type` on `<main>` and surfaced to Devii via `DeviiClient._context().pageType`. **Owner** = `owner_for(request)`: user uid, else the `DEVII_GUEST_COOKIE` (`constants.py`, shared with `routers/devii.py`), else none. **Storage** (`database.py`): table `user_customizations` (per `owner_kind`/`owner_id`/`scope`/`lang`) with helpers `get_custom_overrides` (merges global-then-page-type, cached under the `"customizations"` cache-version name via `sync_local_cache`/`bump_cache_version`), `set_custom_override`, `delete_custom_override`. **Injection**: template globals `custom_css_tag`/`custom_js_tag` (registered in `templating.py`) emit a `<style id="user-custom-css">` after `extra_head` (CSS overrides the design system; `</`->`<\/` neutralizes breakout) and a JSON-island `<script>` after `Application.js` (`<`/`>` escaped, run via `new Function(JSON.parse(...))` so `app` exists). Both fail closed to empty Markup - a customization issue must never break page render. Kill-switches: `customization_enabled`, `customization_js_enabled`. **Devii** (`services/devii/customization/`, `handler="customization"`): `customize_list`/`get`/`set_css`/`set_js`/`reset`, all `requires_auth=False`; `CustomizationController(owner_kind, owner_id)` is built in `Dispatcher.__init__` (owner threaded from `DeviiSession`); set/reset are in `CONFIRM_REQUIRED` so `confirmation_error` forces Devii to ask **page-type vs global** before `confirm=true`. Devii previews live with `run_js` (ids `devii-preview-css`/`devii-preview-js`) and `reload_page` after saving.

### Notification preferences (`database.py`)
Every notification type (`NOTIFICATION_TYPES`) is independently toggleable per user on two channels (in-app, push). Resolution `notification_enabled(user_uid, type, channel)` = live `notification_preferences` override -> admin global default (`notif_default_{type}_{channel}` setting, default `1`) -> on. Enforcement is the single funnel `utils.create_notification` (in-app insert gated by the `in_app` channel, `_schedule_push` by `push`); the `messages.py` DM path was routed through `create_notification` so it is gated too. **`create_notification` defers its work to the background task queue** (`background.submit(_deliver_notification, ...)`), so the preference reads + in-app insert + push schedule + audit all run off the request path (inline in tests). See `### Background task queue`. UI: profile **Notifications** tab (`/profile/{username}?tab=notifications`, owner-or-admin, `POST /profile/{username}/notifications` + `/reset`) and admin global defaults at `/admin/notifications`. Devii tools `notification_list`/`notification_set`/`notification_reset` (`handler="notification"`, owner-scoped, `set`/`reset` in `CONFIRM_REQUIRED`). Storage is the soft-deletable `notification_preferences` table, cached under the `"notif_prefs"` cache-version name. Reuse this canonical-list + resolver + override-table + single-funnel pattern for future per-user per-channel preferences. **Live toasts ride the `in_app` channel:** `services/notification_relay.py` `NotificationRelayService` (lock-owner `BaseService`, like `MessageRelay`) polls `notifications.id > watermark` and publishes each new row to the pub/sub topic `user.{uid}.notifications`; because it runs on the lock owner where every `/pubsub/ws` subscriber converges, the in-process `services.pubsub.publish` reaches the recipient, and because the row exists only when `in_app` is enabled the toast fires exactly for enabled notifications (no new channel). Frontend `static/js/LiveNotifications.js` (`app.liveNotifications`) reads `<body data-user-uid>`, subscribes via `app.pubsub`, and raises a click-through `app.toast`. See AGENTS.md -> "Notification preferences". **The same relay tick also publishes fresh unread counts** `{notifications, messages}` to `user.{uid}.counts` (one publish point covers new notifications AND new DMs, since a DM inserts a `message`-type notification row); `static/js/CounterManager.js` (`app.counters`, constructed with `this.pubsub`) subscribes for instant badge bumps and keeps a 60s `/notifications/counts` poll as a decrement/missed-frame fallback.

### Live view relay (`devplacepy/services/live_view_relay.py`)
`LiveViewRelayService` is a second lock-owner pub/sub bridge (default-enabled, 1s interval, registered in `main.py`) that **replaces per-client HTTP polling on the admin live views with server push, computing a snapshot only for topics that currently have subscribers**. Each tick it reads `pubsub.topics()`, matches concrete subscribed topics against the `VIEWS` registry `(regex, async compute, min_interval)`, throttles per topic, and publishes the **same payload shape the matching HTTP endpoint already returns** (so frontend render code is unchanged). Topics + cadence: `container.list` (4s), `project.{slug}.containers` (3s), `container.{uid}.detail` (4s), `container.{uid}.logs` (3s), `fleet.bots` (2s), `admin.services` (5s), `admin.services.{name}` (5s), `admin.ai-usage.{hours}` (15s) - all admin-only by pub/sub policy. The frontend monitors (`ContainerInstance`/`ContainerList`/`ContainerManager`/`BotMonitor`/`ServiceMonitor`/`AiUsageMonitor`) subscribe via `window.app.pubsub` and keep a lengthened (15-20s) HTTP poll as initial-load + fallback. To add an admin live view: add one `VIEWS` row returning the endpoint payload and subscribe on the frontend. Keep durable/ordered/stateful/raw-I/O channels (messages, Devii, SEO/DeepSearch progress, container PTY) OFF pub/sub. See `pubreport.md` and AGENTS.md -> "Live view relay".

### Devii user-defined tools (`services/devii/virtual_tools/`)
Users invent new Devii tools in natural language ("when I say woeii, do Y"); each is stored per-owner and dynamically added to Devii's LLM tool list, and when called its handler **re-prompts Devii itself** (a self-eval sub-agent) with the stored prompt plus the user's single free-form `input`. CRUD is **Devii-only** (`tool_create`/`tool_list`/`tool_get`/`tool_update`/`tool_delete`, `handler="virtual_tool"`). **Dynamic tool list:** the session tool list is normally frozen, but `DeviiSession._refresh_tools()` runs at the top of every `_run_turn` and rewrites `self.tools` **in place** (`self.tools[:] = CATALOG.tool_schemas_for(...) + virtual_store.tool_schemas()`); since `Agent`/`AgenticController`/`react_loop` share that list object, new/edited/deleted tools take effect next turn with no rebuild - never reassign `self.tools`, always mutate in place. **Self-eval engine:** `agentic/controller.py` `_spawn` (shared by `_delegate`, `run_subagent`, and the new `eval` tool) runs `react_loop` under a contextvar **depth guard** (`agentic/state.py`, `MAX_EVAL_DEPTH=2`) so self-calls can't loop. **Store** `VirtualToolStore(db, owner_kind, owner_id)` over `devii_virtual_tools` (persistent for users, `memory_db()` for guests, built in `hub.get_or_create`). **Dispatch:** the dispatcher's unknown-name branch (`if action is None`) resolves a virtual tool via `self._virtual_tools.has/run` before erroring, so virtual tools live only in the store, not the static catalog. Name validation rejects built-in collisions (`set(CATALOG.by_name())`).

### Devii self-configured behavior (`services/devii/behavior/`)
Devii partially configures its OWN system message: every system prompt ends with a `# TRUTH RULES AND BEHAVIOR` section (`session.py` `BEHAVIOR_HEADER`) whose body is owner-scoped, persistent behavior rules. When the user tells Devii to behave differently, expects different behavior, or Devii upsets them, Devii calls **`update_behavior`** (`handler="behavior"`, `requires_auth=False`, single `behavior` arg = the FULL new section content) to persist it. It is **self-prompted only** - no HTTP route or UI, like the customization/lesson stores. `BehaviorStore(db, owner_kind, owner_id)` upserts one row per owner in `devii_behavior` (persistent for users, `memory_db()` for guests, built in `hub.get_or_create`). `DeviiSession._compose_system_prompt()` appends the header (+ body when set) to the base prompt, and `_refresh_system_prompt()` runs at the top of every `_run_turn` (beside `_refresh_tools`) rewriting `agent._messages[0]` so a mid-conversation update takes effect next turn without the base overriding it. Devii sees the current section in its own prompt, so `update_behavior` replaces (it merges by copying current rules + applying the change). The system-prompt **SELF-CONFIGURED BEHAVIOR (TRUTH RULES)** section in `agent.py` steers it.

### devRant compatibility API (`devplacepy/routers/devrant/`, `devplacepy/services/devrant/`)
A second REST surface mounted at `/api` that mimics the public devRant API on DevPlace data, so legacy devRant clients work unchanged. **The transformation is the whole feature** - no new data model. `routers/devrant/` holds the thin endpoints (`auth.py`, `rants.py`, `comments.py`, `notifs.py`, aggregated in `__init__.py` with a `Depends(ensure_enabled)` gate on the `devrant_api_enabled` setting), and `services/devrant/` holds the pure translation layer (`params.py` merges query + form/JSON body since devRant clients use all three; `tokens.py` issues/validates the `devrant_tokens` triple; `ids.py` maps devRant integer ids onto each table's auto-increment `id`; `serializers.py`/`feed.py`/`profile.py`/`notifications.py` shape devRant JSON from prefetched batch maps; `avatar.py` renders PNGs via `cairosvg`). **ID mapping is load-bearing:** devRant `rant_id`/`comment_id`/`user_id`/`token_id` are exactly `posts.id`/`comments.id`/`users.id`/`devrant_tokens.id` (every `dataset` table already carries an auto-increment `id` PK), so no translation table exists - `post_by_id` is `find_one(id=...)`. **Auth:** `POST /api/users/auth-token` (by username OR email) verifies with passlib and inserts a `devrant_tokens` row (`SOFT_DELETE_TABLES`); every request re-validates `(token_id, token_key, user_id)` via `resolve_user(params)`. **Writes reuse the audited native cores, never duplicate them** - this drove the DRY extraction of `content.apply_vote`, `content.create_comment_record`, `content.delete_comment_record`, `content.set_bookmark`, and `utils.register_account` (the native `routers/votes.py`/`comments.py` and `auth/signup.py` were refactored onto the same functions). So devRant posting/voting/commenting awards XP, fires notifications, writes audit, and soft-deletes identically to the UI. **Mappings:** a rant's `text` is `title\n\n content` (no title for inbound rants, topic forced to `"rant"`); devRant free-form `tags` round-trip verbatim via a new `posts.tags` JSON column (init_db ensures it; falls back to `[topic]`); `profile_skills` is derived from the user `bio`; `favorite`/`unfavorite` map to bookmarks. The envelope helpers are `_shared.dr_ok`/`dr_error` (logical failures stay `200 {success:false}` except bad login -> `400`, matching `devranta/api_test_results.json`). Reference client + captured responses live in `./devranta/`. Host routing (legacy clients hard-code devrant.com) is infra (DNS/reverse-proxy), out of app scope. See README -> "devRant compatibility API" and AGENTS.md -> "devRant compatibility API".

### SEO
`devplacepy/seo.py` generates JSON-LD schemas (WebSite, BreadcrumbList, DiscussionForumPosting, ProfilePage, SoftwareApplication). Every router builds context via `base_seo_context(request, ...)` and merges it into the template response. `base.html` reads `page_title`, `meta_description`, `meta_robots`, `canonical_url`, `og_*`, `breadcrumbs`, `page_schema`. Auth/messages/notifications are `noindex,nofollow`; profiles with fewer than 2 posts are `noindex,follow`. `/robots.txt` and `/sitemap.xml` are served by `routers/seo.py`.

## Conventions (project-specific)

- **No comments, no docstrings in source.** Code is self-documenting.
- **Forbidden variable name patterns:** `_new`, `_old`, `_temp`, `_v2`, `better_`, `my_`, `the_`.
- **Outbound HTTP goes through the stealth client only.** Never instantiate a bare `httpx.AsyncClient(...)` / `httpx.Client(...)` for outbound traffic - build it via `devplacepy.stealth.stealth_async_client(...)` / `stealth_sync_client(...)` (same kwargs as the httpx client it returns; your `headers=` merge so per-call `Authorization`/`Content-Type` win). The async factory routes bytes through **`curl_cffi` (curl-impersonate, BoringSSL)** behind an `httpx` transport adapter (`curl_transport.CurlTransport`), so the TLS JA3/JA4 **and** HTTP/2 Akamai fingerprint are byte-identical to Chrome 146 (not just the headers); it falls back to a pure-httpx Chrome-aligned transport if curl_cffi is missing. Applies to every request (news, gateway upstream, Devii/DeepSearch/SEO fetch, gitea, push, attachments). SSRF-guarded fetches via `net_guard.guarded_async_client(...)` get the fingerprint automatically and still re-guard each redirect hop. The sole exception is `routers/proxy.py` (the `/p/<slug>` reverse proxy relays the user's own headers verbatim); the sync `stealth_sync_client` path stays pure-httpx; Playwright is a real browser, out of scope. See `AGENTS.md` -> "Stealth HTTP client".
- **Form validation is Pydantic-native.** Routers declare a typed body param `data: Annotated[SomeForm, Form()]` (models live in `models.py`); FastAPI validates it and the global `RequestValidationError` handler in `main.py` re-renders auth pages with messages (400) or redirects other routes. Read raw `await request.form()` only when also reading an uploaded file alongside a model - adding a `File()` param would embed the model under its parameter name.
- **`RedirectResponse(url=..., status_code=302)`** - always pass `status_code` as an integer literal, never `status.HTTP_302_FOUND`.
- **Ownership checks:** compare `resource["user_uid"] == user["uid"]` before edit/delete (use `content.is_owner`). **Deletes additionally allow any admin:** every content delete guards `is_owner(...) or is_admin(user)` on the endpoint (posts/gists/projects in `content.delete_content_item`, `comments.delete_comment`, `project_files.project_file_delete`, `media.delete_media`, `uploads.delete_attachment_route`; news is admin-only), so an administrator (including via Devii, which only calls the API) may delete any member's content while members are limited to their own. Edits stay owner-only. Deletes are **soft** (`deleted_at`/`deleted_by`) and cascade comments → votes → engagement → resource under one shared stamp.
- **All dates shown to users are DD/MM/YYYY** (European) and rendered **in the viewer's own timezone, client-side**. Timestamps are stored/emitted as UTC ISO; the display is localized in the browser. Use the `local_dt(iso, mode)` / `dt_ago(iso)` Jinja globals (`templating.py`) for any user-facing **instant** - they emit `<time datetime="UTC-ISO" data-dt data-dt-mode="date|datetime|ago">server-fallback</time>`, and `static/js/LocalTime.js` (`app.localTime`) reformats every `[data-dt]` to the browser's local timezone (`Intl`), with a `MutationObserver` so dynamically-inserted dates (feed, messages, comments) localize automatically and a 60s refresh for relative times. The server fallback (the `<time>` text content, via `format_date`/`time_ago`) is what no-JS clients see. **`format_date()`/`time_ago()` stay as plain-text helpers** for JSON responses, no-JS fallbacks, and the few non-timestamp or attribute-context dates (e.g. project `release_date`/`demo_date`, which are user-entered DD/MM/YYYY strings, and `value=`/`title=` attribute uses) - do NOT wrap those in `local_dt`. Pattern for converting a precomputed `time_ago` string display: `{{ dt_ago(x.created_at) if x.created_at else x.time_ago }}` (falls back to the original string if the raw ISO is absent). New user-facing timestamp displays should use `local_dt`/`dt_ago`.
- **Slug + UUID lookup:** resources with slugs (posts, gists, news, projects) accept either the slug or the bare UUID - see `resolve_by_slug()` in `database.py`. Slugs are generated via `make_combined_slug(title, uid)` which prefixes the last 12 hex chars of the UUID (the uuid7 **random tail**, NOT its leading bytes - the first 48 bits are a millisecond timestamp, so a head prefix is identical for all rows written in the same ~65s window and collides with the same title, making `resolve_by_slug` ambiguous; same reasoning as the blob-shard tail rule).
- **Username:** 3-32 chars, letters/numbers/hyphens/underscores. **Password:** min 6 chars.
- **Roles are stored capitalized:** the `users.role` value is exactly `"Admin"` or `"Member"` (the first registered user becomes `"Admin"`; `auth.py`). Always test admin-ness through the `is_admin(user)` global (`utils.py`, also a Jinja global), which compares `user.get("role") == "Admin"` case-sensitively - never hand-roll a lowercase compare. The **only** lowercase surface is the CLI (`devplace role set <username> <member|admin>` accepts lowercase but writes `role.capitalize()`; `role get` prints `.lower()` for display). A lowercase role in the DB silently fails every admin check.
- **Never pass a `respond()` context key that collides with a Jinja global.** `respond(request, template, ctx, model=XOut)` feeds the **same** `ctx` dict to both the Pydantic model (JSON) and the template render. Jinja context variables shadow `env.globals` across the whole inheritance chain, so a key like `is_admin`/`avatar_url`/`format_date`/`is_self`/`owns`/`guest_disabled` holding a non-callable value turns `base.html`'s `{% if is_admin(user) %}` into `False(user)` -> `TypeError: 'bool' object is not callable`, a 500 that only fires for the branch that calls the global (e.g. logged-in users, not guests). Name viewer/permission flags distinctly (`viewer_is_admin`, not `is_admin`) in both the schema and the context. This was a real issue on `/issues/{number}`; `tests/api/issues/create.py::test_issue_detail_renders_for_{member,admin}` guard it.
- **Avatars:** Multiavatar SVG generated locally from the username seed in <5ms, in-memory cached (cleared on restart). URL `/avatar/multiavatar/{seed}?size={size}`. Falls back to initial-based SVG on error.
- **Project visibility (`is_private`) and read-only (`read_only`)** are owner-controlled integer flags on the `projects` row (see `AGENTS.md` -> "Project visibility and read-only"). Two invariants: (1) private-read access is gated by the single `content.can_view_project(project, user)` predicate (owner or admin) at EVERY read surface - detail, file routes (`_load_viewable_project`), zip, listing, profile, sitemap - never re-implement the check inline; (2) read-only is enforced as a single data-layer guard `project_files._guard_writable(project_uid)` at the top of every mutation entrypoint in `project_files.py`, so it covers the HTTP routes, Devii (via the routes), and container sync at once - add the guard to any NEW file-mutating function, and never enforce read-only only in a route. Devii may flip read-only or visibility (private/public) only after explicit user confirmation, enforced by `dispatcher.confirmation_error` (the `CONFIRM_REQUIRED` set) before the HTTP call; mirror that pattern for any future irreversible agent action. Both owner toggle buttons (`project_detail.html`) likewise carry `data-confirm` so the human UI gates them through `app.dialog` too. **Deletions ALWAYS require confirmation:** `confirmation_error` gates EVERY content delete tool via the `CONFIRM_REQUIRED` set - `delete_post`, `delete_comment`, `delete_gist`, `delete_project`, `project_delete_file`, `delete_media`, `delete_attachment`, `admin_delete_news` - plus `container_instance_action` with `action="delete"` and any `container_exec` whose command matches `dispatcher.DESTRUCTIVE_COMMAND` (rm/rmdir/unlink/shred/truncate/dd/mkfs/wipefs, `find -delete`, redirect over a file, SQL `drop`/`delete from`). The first call is refused with a message; the agent must show the user the exact target and only then pass `confirm=true` (the `DELETING IS ALWAYS CONFIRMED` system-prompt rule mirrors this). Any new name added to `CONFIRM_REQUIRED` is auto-confirmed by a generic fallback in `confirmation_error`. **Load-bearing: every confirmation-gated tool MUST also declare a `confirm` boolean param in its catalog spec** (use the `confirm()` helper in `catalog.py`, or `arg("confirm", ..., kind="boolean")` for container actions). Tool schemas set `additionalProperties: false`, so a gated tool WITHOUT a declared `confirm` param can never receive `confirm=true` from the model - the gate refuses every call and the agent loops forever (this was a real issue on all the delete tools). The param is harmlessly forwarded to the HTTP endpoint (the routes do not declare it; FastAPI ignores undeclared form fields) and ignored by the LOCAL container/customization controllers. The delete tools stay `requires_auth` (not `requires_admin`) - members delete their own, the endpoint's owner-or-admin guard lets admins delete any.

## Modal pattern

`Application.js` `initModals()` toggles a `.visible` CSS class on `.modal-overlay`; the CSS rule `.modal-overlay.visible { display: flex; }` handles visibility. Triggers usually have `href="#"`, so call `e.preventDefault()` in click handlers. `.modal-close` is wired generically - no inline JS needed.

## Polymorphic comments and votes

The `comments` table uses `(target_type, target_uid)` so the same `_comment_section.html` component works for `post`, `project`, `gist`, and `news`. Votes follow the same shape via `/votes/{target_type}/{uid}`. `resolve_target_redirect()` in `comments.py` maps target_type back to the correct detail URL.

## Media (attachments) soft delete

The profile **Media** tab lists every attachment a user uploaded (`get_user_media`); a single upload can be removed via `POST /media/{uid}/delete` (owner or admin) as a **soft delete** - it stamps `deleted_at`/`deleted_by` WITHOUT touching the `target_type`/`target_uid` relation or unlinking the file, so the item disappears from every display surface (the gallery AND its parent post/project) yet stays restorable. The three user-facing read paths (`get_attachments`, `get_attachments_batch`, `get_user_media`) filter `deleted_at IS NULL`; the hard-delete cascades (`delete_attachments_for`, `delete_target_attachments`) stay UNFILTERED. The only hard deletes from the UI are admin trash purge and parent-object deletion. See `AGENTS.md` -> "Profile media gallery and soft-deleted attachments".

## Project-wide soft delete (hard rule)

Attachments are one instance of a platform-wide model: **every removal is a soft delete; only garbage collection is a hard delete.** Removable rows carry `deleted_at` (ISO timestamp) + `deleted_by` (actor uid or `system`); a live row has `deleted_at = NULL` and every list/count read filters `deleted_at IS NULL`. The table set is `database.SOFT_DELETE_TABLES` (content, engagement toggles, sessions, container instances/schedules, Devii task/lesson/virtual-tool stores, customizations, fork relations); `init_db` ensures both columns + an `idx_<t>_deleted` index per table. Core primitives in `database.py`: `soft_delete`, `soft_delete_in`, `restore`, `purge`, `list_deleted`/`count_deleted`, and `restore_event`/`purge_event` (act across all tables sharing one `deleted_at` stamp). `resolve_by_slug(..., include_deleted=)` and `paginate(...)` filter conditionally; `seo._collect` too.

Rules when touching this area:
- **Every INSERT into a soft-deletable table MUST write `deleted_at: None, deleted_by: None`.** `dataset.find(deleted_at=None)` on a table missing the column matches NOTHING (false predicate), silently hiding all rows on a fresh DB - the born-live insert is what creates the column.
- **Any new read** (find/count/query) of a soft-deletable table MUST filter `deleted_at IS NULL`. Use the central helpers/chokepoints instead of inline deletes.
- **Toggles revive, never duplicate:** look up the physical row ignoring `deleted_at`, stamp on toggle-off, clear on re-toggle.
- **Cascades share one `stamp`** (`content.delete_content_item`) so the event restores/purges atomically; pass `deleted_by = actor`.
- **Delete authz is owner-OR-admin on the endpoint** (`is_owner(...) or is_admin(user)`): posts/gists/projects (`content.delete_content_item`), `comments.delete_comment`, `project_files.project_file_delete`, `media.delete_media`, `uploads.delete_attachment_route`; news is admin-only. One endpoint check covers the UI and Devii (Devii only uses the API, as the signed-in user), so admins delete any member's content and members only their own. Any NEW content delete path must reuse this guard, soft-delete, and (for the Devii tool) be added to `dispatcher.CONFIRM_REQUIRED`.
- **GC stays HARD** (job sweep, metrics ring, usage-ledger prune/reset, expired-session cleanup, fork rollback, news-sync image refresh, admin Purge). Logout is soft (auditable); only expiry GC is hard.
- Admin **Trash** at `/admin/trash` (`AdminTrashOut`, `admin_trash.html`) restores/purges by event. Admin-only docs: `docs/soft-delete.html`. See `AGENTS.md` -> "Project-wide soft delete".

## Testing

Playwright (NOT pytest-playwright - uninstall if present, it conflicts). 932 tests in `tests/`, split into **three category directories** by *what they exercise*, one file per endpoint:

- `tests/api/` - HTTP integration tests against the live uvicorn subprocess (`app_server`/`seeded_db`, `requests`/`httpx` vs `BASE_URL`); no browser.
- `tests/e2e/` - Playwright browser tests (`page`/`alice`/`bob`).
- `tests/unit/` - pure in-process tests (the `local_db` fixture or no fixture); call library functions directly, no running server.

**The directory tree mirrors the path** - one segment per directory, the last segment is the file (NOT a flat `_`-joined name):

- **api/e2e are organised by endpoint path.** Each route segment becomes a directory, the final segment becomes the file; `{param}` segments are dropped and each segment is lowercased with non-alphanumerics stripped. `GET /admin/ai-usage` -> `tests/e2e/admin/aiusage.py`, `POST /auth/login` -> `tests/api/auth/login.py`, `GET /projects/{slug}/files/lines` -> `tests/api/projects/files/lines.py`. A collection/bare path that is also a parent of deeper paths uses `index.py` inside its own directory (`/posts` -> `tests/e2e/posts/index.py` alongside `/posts/create` -> `tests/e2e/posts/create.py`); `GET /` -> `tests/<tier>/root.py`. Multiple verbs of one path share one file.
- **unit mirrors the SOURCE module path** of the code under test (strip the `devplacepy.` prefix, dots become directories): tests for `devplacepy/utils.py` -> `tests/unit/utils.py`, for `devplacepy/services/audit/store.py` -> `tests/unit/services/audit/store.py`, for `devplacepy/services/openai_gateway/analytics.py` -> `tests/unit/services/openai_gateway/analytics.py`.

A test's tier is decided by its fixtures: a browser fixture (`page`/`alice`/`bob`) = e2e; `app_server`/`seeded_db` or any HTTP call = api; `local_db`-only or no fixture = unit. Every directory is a package (`__init__.py`). `pyproject.toml` sets `testpaths = ["tests/unit","tests/api","tests/e2e"]` and `python_files = ["*.py"]` (files are NOT `test_`-prefixed; test *functions* still are). `conftest.py` stays at `tests/` root so its fixtures inherit into all subdirs. Helpers shared across files import from the canonical new module path (e.g. `from tests.e2e.post import create_post`). Run a tier with `make test-unit` / `make test-api` / `make test-e2e`; `make test` runs all three. The fixture stack:

- `app_server` (session-scoped): spawns uvicorn as a subprocess on port 10501 (the `PORT` constant) with a tempfile DB.
- `browser_context` (session-scoped): one Playwright context shared by all tests.
- `page` (function-scoped): `clear_cookies()` on the session context, then a fresh page.
- `alice` / `bob`: seeded users `alice_test` / `bob_test` logged in via the login form. `bob` gets its own context for multi-user tests. Returns `(page, user_dict)`.

Required test patterns:
- Every `page.goto(...)` and `page.wait_for_url(...)` MUST pass `wait_until="domcontentloaded"`. CDN libs and avatars cause the default `"load"` to time out.
- Prefer `page.locator(...).wait_for(state="visible")` over `wait_for_selector`.
- Scope ambiguous selectors: comment Delete is `.comment-action-btn:has-text('Delete')`; create-post Post button is `#create-post-modal button.btn-primary:has-text('Post')`; comment textarea is `.comment-form textarea[name='content']`.
- Failure screenshots auto-save to `/tmp/devplace_test_screenshots/` via the `pytest_runtest_makereport` hook in `conftest.py`.
- The server is shared session-wide, so a test that flips a global `site_settings` value (maintenance mode, registration, rate limit, session length) MUST restore it in a `try/finally`. `conftest.py`'s `app_server` seeds `rate_limit_per_minute="1000000"` so the suite is never throttled; rate-limit tests patch `m.get_int_setting` rather than the `RATE_LIMIT` constant (the constant is only a fallback now). Operational tests live in `tests/e2e/operational.py` plus the `tests/api/admin/settings.py` family; rate-limit tests sit with the endpoint they probe (`tests/api/root.py`, `tests/api/auth/login.py`, `tests/api/robotstxt.py`).

**Never run tests unless the user explicitly asks for it.** Not the full suite, not a single file. Validate code with a clean import (`python -c "from devplacepy.main import app"`) and per-language manual checks instead.

## Feature workflow

Every feature in DevPlace is **one data source fanning out into several consumers**. A single route is rendered as HTML, served as JSON, called by the Devii agent, and described in the API docs - all from the same handler. The cardinal failure mode is changing one layer and forgetting a connected one (add a route but no JSON schema; add a JSON endpoint but no Devii tool; ship a feature but never document it). The checklist below is ordered by data flow so nothing is dropped. See `AGENTS.md` -> "Anatomy of a feature" for the deep relationship map and a worked example.

### 1. Understand (read before writing)
Read the router (`routers/{area}.py`), template (`templates/{area}.html`), the matching tests (`tests/{unit,api,e2e}/<endpoint>.py` - per the naming rule in the Testing section), and the matching `AGENTS.md` domain section. Trace the existing data flow: input model -> router -> data helper -> response (HTML and JSON).

### 2. Data layer (`database.py`, `models.py`, `schemas.py`)
- **`database.py`** - add query/batch helpers here, never inline N+1 loops in routers (use `get_users_by_uids`, `build_pagination`, `_in_clause`). Guard raw SQL with `if "table" in db.tables`. Schema auto-syncs via `dataset`; add indexes in `init_db()` with `CREATE INDEX IF NOT EXISTS`.
- **`models.py`** - the Pydantic `Form` model for any new input (`data: Annotated[SomeForm, Form()]`).
- **`schemas.py`** - the Pydantic `*Out` model for the JSON response. If the route serves JSON via `respond(..., model=XOut)`, every new context key the JSON exposes MUST exist on `XOut`, or it is silently dropped.

### 3. Server layer (`routers/{area}.py`, `main.py`)
- Handler with the right auth guard: `get_current_user` (public read), `require_user` (member POST), `require_admin` (admin). POSTs are always guarded.
- **Route order matters**: specific paths (`/{username}/followers`) must be declared before catch-alls (`/{username}`).
- Return HTML+JSON via `respond(request, template, ctx, model=XOut)`; return pure JSON via `JSONResponse`.
- Register any NEW router in `main.py` with its prefix.

### 4. View layer (`templates/`, `static/css/`, `static/js/`)
- Import the shared `templates` from `devplacepy.templating` (never instantiate per router). Extend `base.html`; page CSS in `{% block extra_head %}`, page JS in `{% block extra_js %}`.
- Reuse partials (`_avatar_link.html`, `_user_link.html`) and template globals (`avatar_url`, `format_date`, `guest_disabled`). Dates are DD/MM/YYYY via `format_date`.
- JS is ES6 modules, one class per file, instantiated on `app` (`Application.js` is the root).

### 5. Agent + docs layer (do not skip - these are the most-forgotten)
- **`services/devii/actions/catalog.py`** - if the route is something a user could ask Devii to do, add an `Action` tool (set `requires_auth=False` for public reads). Without this, the feature exists for humans but is invisible to the assistant.
- **`docs_api.py`** - every public/auth endpoint gets an `endpoint()` entry in the correct group, with params and a `sample_response`. **`routers/docs.py`** `DOCS_PAGES` - prose pages.
- **SEO** (`seo.py`) - if a new public page, set `base_seo_context` and any JSON-LD; add to `routers/seo.py` sitemap if indexable.

### 6. Document the change (the three files, every time)
- **`README.md`** - product-facing. Update when routes, config/env vars, dependencies, or user-visible features change. No dev process, no begging, no emoticons.
- **`AGENTS.md`** - the deep companion. Add a domain section for any new mechanic, helper, pitfall, or cross-layer wiring.
- **`CLAUDE.md`** - this file. Update only when a NEW architectural rule, convention, or workflow step is introduced (not per feature).

### 7. Validate (never skip; never run the test suite unless asked)
- Validate each touched language manually (Python compiles/imports, JS parses, CSS braces and HTML tags balance).
- Build: `python -c "from devplacepy.main import app"` must import clean.
- **No em-dash** (neither the long dash character nor its HTML entity) anywhere in touched files - grep and confirm; use a hyphen instead.
- **Never run tests unless the user explicitly asks.**

`AGENTS.md` is the long-form companion to this file with every domain-specific pitfall (notifications, gists CodeMirror setup, public feed rules, post editing, news detail comments, etc.) and the full feature relationship map. Read it when working on an unfamiliar area.
