Update

devplacepy/routers/CLAUDE.md

@@ -0,0 +1,335 @@
+This file documents devplacepy/routers/ - route organization, the full URL prefix map, and small single-file feature behaviors. Claude Code loads it automatically whenever a file under this directory is read or edited.
+
+## Routing layout / prefix table
+
+Routers in `devplacepy/routers/` are organised as a **directory tree that mirrors the endpoint (URL) path**, exactly like `tests/`. A domain with a single resource stays one flat file (`feed.py`, `posts.py`, `gists.py`, ...); a domain with several sub-resources is a **package directory** split **one file per sub-resource** (a distinct noun under the domain), never one file per individual endpoint. Each leaf module declares its own `router = APIRouter()` keeping the exact path strings, and the package `__init__.py` aggregates them with `router.include_router(...)`. Because a package exposes `.router` like a module, `main.py` mounts each domain at its prefix unchanged.
+
+Prefixes are wired in `main.py`:
+
+| Prefix | Router |
+|--------|--------|
+| `/auth` | auth/ package - one leaf per flow (`signup`, `login`, `logout`, `forgotpassword`, `resetpassword`) |
+| `/feed` | feed.py |
+| `/posts` | posts.py |
+| `/comments` | comments.py |
+| `/projects` | projects/ package - `index.py` (listing/detail/create/delete plus owner-only visibility toggles `POST /{slug}/private` and `POST /{slug}/readonly`, and async `POST /{slug}/fork`), `files.py`, and the `containers/` subpackage (below). `main.py` mounts the whole `/projects` tree from this one package. See `routers/projects/CLAUDE.md` for the deep detail on this tree |
+| `/projects/{slug}/files` | projects/files.py - per-project virtual filesystem (CRUD dirs/files, upload, inline edit, plus line-range ops: `lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append` for large text files) |
+| `/profile` | profile/ package - `index.py` (page/search/followers/update/api-key; the **Posts** tab shows only the 10 most recent posts, while the sidebar `posts_count` stat always uses the real `count()` so it is never capped to the rendered 10), `customization.py`, `notifications.py`, `ai_correction.py` (`POST /{username}/ai-correction`, owner-or-admin AI content correction toggle + prompt), `ai_modifier.py` (`POST /{username}/ai-modifier`, owner-or-admin AI modifier toggle + prompt), `telegram.py` (`POST /{username}/telegram`, owner-or-admin Telegram pairing-code request/unpair), `usage.py` (`_ai_quota`, re-exported from the package) |
+| `/messages` | messages.py - real-time DM chat. `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `POST /send` (no-JS fallback), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, read receipts, and in-process presence on top of the existing `messages` table; both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention) and then through `_finalize_and_broadcast`, which awaits any pending SYNC AI correction/modifier futures (DM content runs through both, so typing `@ai <instruction>` executes live) and broadcasts the FINAL corrected/modified content (the WS path passes `request=websocket` and awaits directly since HTTP middleware does not run for websockets; the frontend swaps the sender's optimistic bubble to the authoritative content). Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, deduped against the hub's delivered set). See `devplacepy/services/messaging/CLAUDE.md` |
+| `/notifications` | notifications.py |
+| `/votes` | votes.py |
+| `/reactions` | reactions.py - emoji reaction toggle on any target: `POST /reactions/{target_type}/{target_uid}` (optimistic `ReactionBar`) |
+| `/bookmarks` | bookmarks.py - bookmark/favorite toggle: `GET /bookmarks/saved` (the viewer's saved list) and `POST /bookmarks/{target_type}/{target_uid}` (toggle a bookmark) |
+| `/polls` | polls.py - poll voting: `POST /polls/{poll_uid}/vote` |
+| `/avatar` | avatar.py |
+| `/follow` | follow.py |
+| (none) | relations.py - per-user block/mute relations: `POST /block/{username}`, `/block/unblock/{username}`, `/mute/{username}`, `/mute/unmute/{username}` (soft-deletable `user_relations` rows) |
+| `/leaderboard` | leaderboard.py - `GET /leaderboard` XP/stars leaderboard page |
+| `/admin` | admin/ package - one leaf per sub-resource (`index`, `users`, `aiusage`, `aiquota`, `media`, `trash`, `settings`, `notifications`, `news`, `auditlog`, `backups`) plus the folded-in `services.py` and `containers.py` (mounted with `/services` and `/containers` sub-prefixes). `main.py` mounts the whole `/admin` tree from this one package. The `backups` leaf is the admin **Backups** dashboard (`BackupService`, kind `backup`): storage usage, backup archives, and interval/cron backup schedules (CRUD + rotation). **Archive download is restricted to the primary administrator** (the earliest-created Admin, resolved by `database.get_primary_admin_uid` / `utils.is_primary_admin`): `GET /admin/backups/{uid}/download` 403s every other admin, the `download_url` field is withheld from them at every endpoint (`can_download = is_primary_admin(admin)`, surfaced as `BackupDashboardOut.can_download_backups`), and `BackupMonitor.js` renders their Download control as a disabled button tooltipped `Not available`. See `devplacepy/services/backup/CLAUDE.md` |
+| `/admin/services` | admin/services.py |
+| `/issues` | issues/ package - issue tracker backed by Gitea (no local issue store): `index.py` (list `?state=`/`?page=`, detail `/{number}` with comments), `create.py` (async AI-enhanced filing `/create` enqueues a `issue_create` job, status at `/jobs/{uid}`), `comment.py` (synchronous, pushes to Gitea + notifies admins), `status.py` (admin open/closed), `attachments.py` (file attachments on open issues + comments, mirrored to Gitea native assets; add/list/delete with owner-or-admin + open-state guards) |
+| `/gists` | gists.py |
+| `/news` | news.py |
+| `/uploads` | uploads.py |
+| `/media` | media.py - profile media gallery item soft delete/restore: `POST /media/{uid}/delete` and `POST /media/{uid}/restore` (owner or admin) |
+| `/openai` | openai_gateway.py |
+| `/devii` | devii.py - WebSocket terminal (`/devii/ws`), page, `/devii/usage`, `/devii/session` |
+| `/zips` | zips.py - generic zip job status (`/zips/{uid}`) and archive download (`/zips/{uid}/download`); enqueued from `/projects/{slug}/zip` and `/projects/{slug}/files/zip` |
+| `/forks` | forks.py - fork job status (`/forks/{uid}`); enqueued from `/projects/{slug}/fork`. When done its `project_url` points at the new forked project |
+| `/tools` | tools/ package - public developer tools surface. `index.py` (`/tools` landing) plus `seo.py` (**SEO Diagnostics**): `GET /tools/seo` page, `POST /tools/seo/run` (enqueue `seo` job, per-owner one-active-job cap), `GET /tools/seo/{uid}` (`SeoJobOut`), `GET /tools/seo/{uid}/report` (HTML+JSON `SeoReportOut`), `WS /tools/seo/{uid}/ws` (live progress, lock-owner gated, close `4013` retry), `GET /tools/seo/{uid}/screenshot/{n}`. Also `deepsearch.py` (**DeepSearch**): `GET /tools/deepsearch` page, `POST /tools/deepsearch/run` (enqueue `deepsearch` job, per-owner one-active-job cap; resolves the user `api_key` into the payload, guests use the internal key), `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (HTML+JSON `DeepsearchSessionOut`), `WS /tools/deepsearch/{uid}/ws` (live progress, lock-owner gated, `4013` retry), `WS /tools/deepsearch/{uid}/chat` (grounded RAG chat over the session collection), `POST /tools/deepsearch/{uid}/{pause|resume|cancel}`, `GET /tools/deepsearch/{uid}/export.{md,json,pdf}`. The shared owner helper is `routers/tools/_shared.py` `owner_for`. Also `isslop.py` (**AI Usage Analyzer**): `GET /tools/isslop` page, `POST /tools/isslop/run` (enqueue `isslop` job, per-owner one-active-job cap; owner = user uid or the shared `DEVII_GUEST_COOKIE` guest identity, minted when absent), `GET /tools/isslop/list` (owner history; a signed-in request first claims any guest-cookie analyses via `store.claim_guest_analyses` - a move, never a copy), `GET /tools/isslop/{uid}` (`IsslopAnalysisOut`), `GET /tools/isslop/{uid}/events` (persisted ordered event trail, `?after=SEQ` incremental poll; live frames also publish on pub/sub `public.isslop.{uid}` - the DB trail is the source of truth, pub/sub the fast path), `GET /tools/isslop/{uid}/report` (HTML+JSON `IsslopReportOut`; live `<dp-isslop-run>` while running, server-rendered report via `render_content` when done), `GET /tools/isslop/{uid}/report.md`, `GET /tools/isslop/{uid}/badge.svg` (embeddable SVG authenticity badge). Analyses/reports/badges are permanent public capability URLs (`IsslopService.cleanup` never deletes them; only the job row is swept). Surfaced by a collapsible **Tools** header dropdown (`base.html`, visible to all) |
+| `/projects/{slug}/containers` | projects/containers/ subpackage - admin per-project container manager (`instances.py` for creation/lifecycle/exec/logs/metrics/sync plus the exec websocket, `schedules.py` for cron/interval/once schedules, shared helpers in `_shared.py`). Every instance runs the shared `ppy` image. Discoverable from the project detail page (admin-only **Containers** button) and from the admin index |
+| `/admin/containers` | admin/containers.py - admin **Containers** manager: `/admin/containers` lists every instance across all projects with inline actions (start/stop/restart/terminal/edit/delete) plus a create modal (project search-select, run-as user search-select, boot language + source editor, restart policy, start-on-boot, env/ports/limits/ingress); `/admin/containers/{uid}` is the per-instance detail page (lifecycle, live logs/metrics, PTY terminal, schedules, ingress, sync, status history) and `/admin/containers/{uid}/edit` edits run-as user, boot language/script/command, restart policy, start-on-boot, and limits. `POST /admin/containers/create`, `/{uid}/edit`, `/{uid}/{start,stop,restart,pause,resume}`, `/{uid}/sync`, `/{uid}/delete` call `api.*` directly under `require_admin` (no docker/exec backend duplicated); `GET /admin/containers/projects/search` and `/users/search` back the create/edit search-selects. The lifecycle and detail views stay layered over the per-project `/projects/{slug}/containers/instances/{uid}/...` endpoints (the instance carries its `project_uid`) |
+| `/p/{slug}` | proxy.py - public ingress reverse proxy (HTTP + WebSocket) to a running container instance's published host port, opt-in per instance via `ingress_slug` |
+| `/xmlrpc` | xmlrpc.py - reverse-proxies XML-RPC calls to the forking XML-RPC bridge (`services/xmlrpc/`, supervised by `XmlrpcService` on loopback `config.XMLRPC_PORT`), which generates one XML-RPC method per documented REST endpoint from `docs_api.API_GROUPS` (`posts.create`, `feed.list`, ...). One struct of named params per call; auth via in-band `api_key`, `X-API-KEY`/`Bearer` header, or `http://user:pass@host/xmlrpc` Basic. Full introspection + `system.multicall`; REST errors become XML-RPC faults. Exempt from the rate limiter (enforced on the forwarded internal hop). See `devplacepy/services/xmlrpc/CLAUDE.md` |
+| `/api` | devrant/ package - devRant-compatible REST protocol (`auth.py`, `rants.py`, `comments.py`, `notifs.py`). Translates devRant requests onto DevPlace data: rants<->posts, comments/votes onto the native engagement layer, devRant integer ids onto each table's auto-increment `id`. Token auth via `devrant_tokens`. See `routers/devrant/CLAUDE.md` for the deep detail on this tree |
+| `/dbapi` | dbapi/ package - **primary-administrator-only, strictly READ-ONLY** generic database API over `dataset` (never inserts/updates/deletes/restores; the write surface was removed because it bypassed every per-route admin safeguard). `tables.py` (`GET /tables`, `GET /{table}/schema`), `crud.py` (read only: `GET /{table}` + `GET /{table}/{key}/{value}`, `?include_deleted`), `query.py` (`POST /query` hard SELECT-only validated read-only run; `POST /query/async` + `GET /query/{uid}` + `WS /query/{uid}/ws` via `DbApiJobService` kind `dbquery`), `nl.py` (`POST /nl` natural-language->SQL via the AI gateway, returns validated SELECT, `execute=true` runs it read-only). **Auth = the PRIMARY administrator only** (the earliest-created Admin, `utils.is_primary_admin` / `database.get_primary_admin_uid` - the same identity that gates backup downloads) by session/api_key (`services/dbapi/policy.py`); every other administrator is refused like a member, and there is **no internal-key path** - the gateway `internal_gateway_key()` is NOT accepted (no service-to-service access). SQL validated by `services/dbapi/validate.py` (sqlglot classify + suspicious-flag + read-only `EXPLAIN` dry-run). Devii tools `db_*` are read-only (list/get/query/nl; no write tools) and flagged `requires_primary_admin=True`, so they are added to the LLM tool list ONLY for the primary administrator (every other session never sees them and Devii is unaware they exist). `services/pubsub/policy.py` resolves its own admin/internal actor and does NOT reuse `dbapi.policy.caller_for`, so the primary-admin restriction does not leak onto the pub/sub bus. See `devplacepy/services/dbapi/CLAUDE.md` |
+| `/game` | game/ package - the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard`, `POST /game/{plant,harvest,buy-plot,upgrade,fertilize,daily,perk,prestige,legacy,quests/claim}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}` |
+| (none) | push.py - web push + PWA: `GET`/`POST /push.json` (VAPID public key / subscription register), `GET /service-worker.js`, `GET /manifest.json` |
+| (none) | docs.py (docs/ package) - the documentation site (prose pages + API reference). See `routers/docs/CLAUDE.md` for the deep detail on this tree (`DOCS_PAGES`, audience tiers, prose rendering pipeline) |
+| `/pubsub` | pubsub.py - **database-free** publish/subscribe bus. `WS /pubsub/ws` (subscribe/unsubscribe/publish frames, `foo.*` wildcards), `POST /pubsub/publish` + `GET /pubsub/topics` (admin/internal). Lock-owner-gated WS (`4013` retry) so subscribers converge on one worker; topic authz in `services/pubsub/policy.py` (`user.{uid}.*` private, `public.*` shared, admin/internal anywhere, guests opt-in). In-process `services.pubsub.publish(topic, data)` for backends; frontend `app.pubsub`. See `devplacepy/services/pubsub/CLAUDE.md` |
+| (none) | seo.py - `/robots.txt`, `/sitemap.xml` |
+
+## Route aggregation rules
+
+**Aggregation rules:** a leaf that owns the domain's collection-root (`""`) route must be the package's base router (FastAPI rejects an empty path included under an empty prefix), so the package `__init__` imports that leaf's `router` and includes the rest onto it (see `routers/issues`, `routers/admin`, `routers/projects`); leaves carved out of a former monolith keep their relative path strings and are included with no sub-prefix, while a router folded in from a deeper mount keeps its own paths and is included with a sub-prefix - `admin/__init__` includes `services.router` with `prefix="/services"` and `containers.router` with `prefix="/containers"`. Module-private helpers shared across a package's leaves live in its `_shared.py`. The `/projects` tree (project CRUD + `files.py` + `containers/`) and the `/admin` tree (every admin sub-resource plus the folded-in `services` and `containers`) are each mounted from a single package.
+
+## HTML/JSON content negotiation
+
+Every page/redirect endpoint also returns JSON when the client asks. Core in `devplacepy/responses.py`: `wants_json(request)` (true for `Accept: application/json` or `Content-Type: application/json`; browser `text/html` -> HTML, so existing behaviour is unchanged). **`X-Requested-With: fetch` is deliberately NOT a trigger** - the frontend sends it on form/engagement fetches expecting the old redirect, and the four legacy engagement endpoints (votes/reactions/bookmarks/polls) handle that header themselves. Two helpers replace the direct returns:
+
+- **Page GETs:** `return respond(request, "x.html", context, model=XOut)` - HTML renders the template; JSON does `XOut.model_validate(context).model_dump()`. One context, two renderings.
+- **Action POSTs:** `return action_result(request, url, data=<resource|None>)` - HTML 302 redirects; JSON returns `{ok, redirect, data}`. (Set cookies on the returned response after calling it, as `auth.py` login/signup do.)
+
+Response models live in `devplacepy/schemas.py` (Pydantic v2, `extra="ignore"`, all-Optional so they validate the existing context dicts directly). **Always project users through `UserOut`** (and `AdminUserOut`) - the raw user rows contain `email`/`api_key`/`password_hash`, and the models drop them; never serialize a raw user row. List item shapes vary (feed/gists/news/admin-news are wrapped `{post|gist|article: ...}`; projects are flat rows with `author_name`/`my_vote`) - match the context exactly. Errors negotiate centrally: `main.py` 404/500/validation handlers and the rate-limit/maintenance middleware, plus `utils.require_user`/`require_admin` (401/403 for JSON, 303 redirect for browsers). The four legacy AJAX endpoints (votes/reactions/bookmarks/polls) keep their original flat JSON shapes and are left untouched. Documented in `docs_api.py`'s Conventions group.
+
+## FastAPI patterns
+
+- **All routes are async.** Form data is validated via a typed Pydantic body param: `data: Annotated[SomeForm, Form()]` (models in `models.py`). Read raw `await request.form()` only when also handling an uploaded file (a separate `File()` param would embed the model under its parameter name).
+- **Return `RedirectResponse(url=..., status_code=302)`** for redirects - always pass `status_code` as an integer literal, never `status.HTTP_302_FOUND`.
+- **Return `templates.TemplateResponse("name.html", {...})`** from `devplacepy.templating` to render.
+- **Never create your own `Jinja2Templates` instance.** Import the shared one: `from devplacepy.templating import templates`.
+- **Register new routers in `main.py`:** `app.include_router(router_instance, prefix="/{path}")`
+- **`require_user(request)`** redirects guests (303) to login, but raises **401** when credentials *were* supplied yet invalid (so API clients get a clear error). Only post/comment/vote/etc. routes use it - the feed is public. `require_admin` and `require_user_api` (401-only) build on it. None of these changed signatures, so all auth schemes work through existing call sites.
+- **For a missing detail resource, `raise not_found("X not found")`** (`utils.py`) - it returns an `HTTPException(404)` that the global handler renders as `error.html`. Do not return a bare `HTMLResponse(..., status_code=404)`.
+- **Detail pages reuse `load_detail(table, target_type, slug, user)`** (`content.py`) for item+author+comments+attachments+`star_count`+`my_vote`; list pages reuse `enrich_items(items, key, authors, extra_maps, user=...)`. Prefer these over manual per-row loading (posts/projects/gists detail and feed/gists/profile lists already do).
+- **`get_current_user(request)` resolves ALL auth schemes** (`utils.py`), in order: `session` cookie -> `X-API-KEY` header -> `Authorization: Bearer <api_key>` -> `Authorization: Basic base64(username-or-email:password)`. It memoizes the result on `request.state._auth_user` (resolved once per request - it is called by both the maintenance middleware and the route) and caches users in `_user_cache` (by session token, `"k:"+api_key`, or a hash of the Basic header). Because every router already routes through this one function, API-key/Bearer/Basic auth work on every page/action with no per-route code. Use it for pages viewable by guests too (feed, news detail, projects).
+- **Post deletion must cascade:** delete comments and votes first, then the post. Always check ownership: `post["user_uid"] == user["uid"]`.
+- **Message deduplication needed** when `sender_uid == receiver_uid` (messaging yourself): `seen = set()` of message UIDs before appending to result list.
+
+## Polymorphic comments and votes
+
+The `comments` table uses `(target_type, target_uid)` so the same `_comment_section.html` component works for `post`, `project`, `gist`, and `news`. Votes follow the same shape via `/votes/{target_type}/{uid}`. `resolve_target_redirect()` in `comments.py` maps target_type back to the correct detail URL.
+
+## Small feature notes: comment editing, post editing, inline comments
+
+### Inline comment on feed cards
+
+Every post card on the feed has an inline comment form (`.feed-comment-form`) beneath the post actions. It posts to `/comments/create` like the detail page comment form. Tests must scope Post button clicks to `#create-post-modal button.btn-primary:has-text('Post')` to avoid matching the inline comment's Post button.
+
+### Comment editing
+
+A comment's owner (only the owner, never an admin) sees an inline "Edit" button (`data-action='edit'`) in `_comment.html`. `CommentManager.toggleEditForm` swaps the `.comment-text` for a textarea seeded from its `data-raw` attribute (the raw markdown, since `contentRenderer.applyTo` overwrites `textContent` on first render), posts via `Http.send` to `POST /comments/edit/{comment_uid}`, then re-renders the new body in place with `contentRenderer.applyTo`. The route (`content.edit_comment_record`) is `is_owner`-only, writes `content` + `updated_at`, records the `comment.edit` audit event, and branches on `wants_json`: JSON clients get `CommentEditOut{uid, content, url, updated_at}`, the no-JS form falls back to a redirect to the comment anchor. Edits are NOT soft-delete related (the body is overwritten in place). Devii tool: `edit_comment` (owner-only, no confirm). Scope test Edit clicks to `.comment-action-btn:has-text('Edit')`.
+
+### Post editing
+
+Post owners see an "Edit" button on the post detail page that opens `#edit-post-modal`. The edit form allows changing title, content, and topic. The POST route is `/posts/edit/{post_uid}` with ownership check. The edit modal's textarea has `id="edit-content"` - tests must scope to `.comment-form textarea[name='content']` for comment operations.
+
+## Gists
+
+A dedicated `/gists` page for sharing code snippets. Uses `gists` table (auto-created by `dataset`).
+
+### Database columns
+
+| Column | Type | Notes |
+|--------|------|-------|
+| `uid` | text | UUID |
+| `user_uid` | text | FK -> users.uid |
+| `title` | text | Required, max 200 |
+| `description` | text | Optional, max 5000, markdown (rendered by ContentRenderer) |
+| `source_code` | text | Required, max 50000 |
+| `language` | text | One of 27 supported languages |
+| `slug` | text | `make_combined_slug(title, uid)` |
+| `stars` | int | Net vote count (via `/votes/gist/{uid}`) |
+| `created_at` | text | ISO datetime |
+
+### Routes
+
+| Method | Path | Handler | Auth |
+|--------|------|---------|------|
+| GET | `/gists` | `gists_page` | No |
+| GET | `/gists/{slug}` | `gist_detail` | No |
+| POST | `/gists/create` | `create_gist` | Yes |
+| POST | `/gists/delete/{slug}` | `delete_gist` | Yes (owner) |
+
+### Polymorphic reuse
+
+- **Comments**: Uses `_comment_section.html` with `target_type="gist"` - same component as posts/projects
+- **Voting**: Uses existing `/votes/gist/{uid}` route - updates `gists.stars`
+- **Content rendering**: Description rendered via `ContentRenderer.js` (`.rendered-content[data-render]`)
+- **Profile tab**: "Gists" tab between Projects and Activity on profile pages
+
+### CodeMirror editor
+
+- CodeMirror 5 loaded from CDN in `gists.html` via `{% block extra_js %}`
+- 22 language modes pre-loaded (Python, JS, TS, HTML, CSS, C, C++, Java, Go, Rust, SQL, Bash, YAML, Markdown, Swift, PHP, Ruby, Kotlin, Haskell, Lua, Perl, R, Dart, Scala)
+- `GistEditor.js` initializes CodeMirror on `#gist-source-editor` textarea
+- Language selector dropdown dynamically switches CodeMirror mode
+- `Ctrl+S` shortcut saves and submits the form
+- On form submit, `editor.save()` syncs CodeMirror content back to the hidden textarea
+
+### Display
+
+- Source code rendered in `<pre><code class="language-xxx">` block on detail page
+- Syntax highlighting handled by existing `highlight.js` loaded globally in `base.html`
+- Copy button uses `navigator.clipboard.writeText()`
+- Cards in listing show language badge, title, truncated description, author, star count
+
+### Sitemap
+
+- Latest 500 gists included in sitemap, `changefreq="weekly"`, `priority="0.6"`
+
+## Feed and listing features
+
+### Politics category
+
+The `politics` topic is available as a feed filter sidebar item (icon `&#x1F3DB;`) and post topic. It defines the CSS variable `--topic-politics: #00bcd4` in `variables.css` and the `.badge-politics` class in `base.css`. It is validated like every topic via the canonical `TOPICS` list in `constants.py` (consumed by `models.py` `valid_topic`, `templating.py` `TOPICS` global, `routers/posts.py`, `docs_api`). Unlike the former `signals` topic, `politics` is also part of the bot fleet's rotation - it is in `services/bot/config.py` `CATEGORIES`/`FEED_TOPICS`, carries a modest per-persona weight in `PERSONA_CATEGORY_WEIGHTS`, and has a writing instruction in `services/bot/llm.py` `category_extras`.
+
+### Date format
+
+All dates displayed to users use European DD/MM/YYYY format. Implemented via:
+
+- **`format_date(dt_str, include_time=False)`** in `utils.py` - converts ISO datetime -> `DD/MM/YYYY` or `DD/MM/YYYY HH:MM`
+- Registered as template global in `templating.py`: `{{ format_date(dt) }}`
+- **`time_ago()`** returns `DD/MM/YYYY` for items older than 30 days (instead of `"Xmo ago"`)
+- Services page has a JS `formatDate()` function for live polling updates
+
+### Admin pagination
+
+Both `/admin/users` and `/admin/news` use offset-based pagination via a reusable component:
+
+- **`templates/_pagination.html`** - numbered page links with ellipsis, Previous/Next buttons, total count
+- Routes accept `?page=N` query param, clamped to valid range
+- `per_page = 25`, pagination metadata computed server-side and passed as `pagination` dict
+- Only renders when `total_pages > 1`
+- CSS in `admin.css` (`.pagination`, `.pagination-btn`, `.pagination-page`, `.pagination-ellipsis`)
+
+### News detail and comments
+
+News articles have an internal detail page at `/news/{slug}` with full comment support:
+
+- **Route:** `GET /news/{news_slug}` in `routers/news.py` - resolves by slug first, then UUID
+- **Template:** `templates/news_detail.html` - shows image, source, grade, description, content, external link
+- **Comments:** Uses `_comment_section.html` with `target_type="news"` - same component as posts/projects
+- **`resolve_target_redirect()`** in `comments.py` handles `"news"` -> `/news/{slug}`
+- Listing links in `news.html` point to internal detail page; "Read on Source" still goes to external URL
+
+### Home page (`GET /`)
+
+The home route (`main.py` `landing()`) never redirects - it renders `templates/landing.html` for everyone, branching on `user`:
+
+- **Guests** get the marketing hero (`Join DevPlace Free` CTA + features grid).
+- **Signed-in users** get a personalized hero (`.landing-hero-user`): avatar, "Welcome back, {username}", a `Go to your feed` CTA, a Posts/Stars/Level stat strip (`user_post_count` + the user dict's `stars`/`level`), and quick links. Styles live in `.landing-hero-user`/`.landing-welcome`/`.landing-stats`/`.landing-quicklinks` in `static/css/landing.css`.
+- Both states share the Latest Posts + Developer News + "Build With Us" sections. The **Build With Us** section is static HTML/CSS (`.landing-help-*` in `landing.css`): four cards linking to `/docs/index.html` (Documentation), `/swagger` + `/openapi.json` (API Reference), `/issues` (Contribute & Report), and Devii. The Devii card's `Launch Devii` button is a plain `<button data-devii-open>` that opens the globally mounted `DeviiTerminal` (`app.devii`) in place - no extra JS, route, schema, or Devii action; a secondary link points to `/devii/` for the full terminal page.
+- Context adds `user`, `is_authenticated`, `user_post_count`; `LandingOut` carries `is_authenticated`/`user_post_count` for the JSON form. `GET /` is documented in `docs_api.py` (id `home`, mapped to `LandingOut`).
+
+### Author diversity (interleave, never drop)
+
+Both the home page Latest Posts section and the main `/feed` *interleave* authors so one prolific account cannot fill a contiguous run, **without dropping any post** (the old per-author cap is gone). Helpers in `database.py`:
+
+- `interleave_by_author(rows, uid_key="user_uid")` - pure reordering of a row list. Greedy and order-preserving: it walks the list and at each step emits the earliest row whose author differs from the last emitted one (deferring a same-author run to the next available author, recursively); when only same-author rows remain it emits them in order. The result is a permutation of the input - never a subset - so each author's own posts keep their relative (date) order and no two consecutive rows share an author unless the whole list is one author. The home route fetches the 6 newest posts and interleaves them.
+- `paginate_diverse(table, *clauses, ..., uid_key="user_uid", **filters)` - a drop-in replacement for `paginate` (same `(rows, next_cursor)` contract) that calls `paginate` then `interleave_by_author` on the page. Because the interleave is a pure permutation of the page, the cursor (oldest `created_at` in the page) is identical to `paginate`'s, so infinite scroll stays correct with no gaps/overlaps. `feed.py` `get_feed_posts` uses it for every tab.
+
+When building any new "recent items" list, reuse these instead of raw `find(order_by=...)` / `paginate`.
+
+### Listing search (feed, gists, projects)
+
+The three public listings - `/feed`, `/gists`, `/projects` - share one free-text search box at the top of the left filter panel. Two reuse points keep it DRY and consistent:
+
+- **Data layer:** `database.text_search_clause(table, search, fields=("title", "description"), author_field=None)` returns a single SQLAlchemy `or_(col.ilike("%term%") ...)` clause over the given columns (skipping any not present on the table), or `None` when `search` is blank or the table does not exist yet. Append it as a positional clause to `paginate` / `paginate_diverse` / `table.count(...)` **only when not None** (no search leaves the queries unchanged). Field mapping: projects/gists use `("title", "description")`, posts use `("title", "content")`. **Author-username matching:** posts/gists/projects store `user_uid`, not the author's username text, so a username query never matched a text column - the fix is the `author_field` argument. When set (all three listings pass `author_field="user_uid"`), `text_search_clause` resolves usernames matching the search term to uids via `database.get_uids_by_username_match(search)` (a capped `users.username LIKE` over the existing `idx_users_username` index) and OR-includes `columns[author_field].in_(uids)` in the same clause - no SQL JOIN, staying within the `dataset` query pattern. So each listing now matches its text fields **plus** the author username. `projects.py`, `gists.py`, `feed.py`, and the devRant `services/devrant/feed.py` all call it; never re-inline an `ilike` search clause and never add a JOIN - resolve username->uids and reuse `author_field`.
+- **View layer:** `templates/_sidebar_search.html` is the single search-box partial (the `Filter` heading + the GET form). Include it at the very top of `<aside class="sidebar-card">` with three locals: `_action` (the listing URL), `_placeholder`, and `_hidden` (a dict of `name -> value` rendered as hidden inputs so the active category/tab filter survives a search submit - e.g. `{"tab": current_tab, "topic": current_topic}` for the feed). It reads the `search` context var for the current value. Any future listing with a left filter panel reuses this partial plus `text_search_clause`; do not hand-roll another search form.
+
+The `search` value is threaded back into each listing's context and exposed on `FeedOut.search` / `GistsOut.search` / `ProjectsOut.search`, documented as a `search` query param on the `feed-list` / `gists-list` / `projects-list` endpoints in `docs_api.py`, and offered to Devii as the `search` query param on `view_feed` / `list_gists` / `list_projects`.
+
+### Recent comments on listing/feed cards (consistent across post, gist, project, news)
+
+Every public listing card that shows the last few comments must render them with the **same visual hierarchy as the detail page** (depth indentation, vote rail, reply nesting). One pipeline drives all four surfaces - never inline a flat-only render:
+
+- **Data layer:** `database.get_recent_comments_by_target_uids(target_type, target_uids, limit=3, user=None)` is the single batch helper. It runs one `ROW_NUMBER() OVER (PARTITION BY target_uid ORDER BY created_at DESC, id DESC)` window over `comments` filtered by `target_type=:tt` + `deleted_at IS NULL`, builds each item via `_build_comment_items` (identical dict shape to the detail page: `comment`/`author`/`time_ago`/`votes`/`my_vote`/`children`/`attachments`/`reactions`), then **nests the limited set by `parent_uid` into `children`** so replies render indented. Returns `{target_uid: [top-level items]}`. `get_recent_comments_by_post_uids` is now a thin wrapper delegating with `target_type="post"`, so `feed.py` is unchanged and there is no behavior drift.
+- **Routers:** `feed.py` (post), `gists.py` (gist), `projects/index.py` (project), `news.py` (news) each batch-fetch with the helper and attach `recent_comments` per item. Gists and news are wrapper dicts (`item["recent_comments"]`); projects are **flat dicts** (`project["recent_comments"]`).
+- **Schemas:** `FeedItemOut`, `GistItemOut`, `NewsListItemOut`, and the flat `ProjectListItemOut` all carry `recent_comments: list[CommentItemOut] = []`, so the `respond(model=...)` JSON exposes the same nested tree and never silently drops the key.
+- **View:** each card renders the block with the shared `_comment.html` `render_comment(c, 0)` macro inside a `.post-card-comments` wrapper (copied from `_post_card.html`). The macro recurses on `item.children` for indentation. **CSS requirement:** the template must load `post.css` (the `.comment`/`.comment-depth`/`.comment-replies` hierarchy) and `feed.css` (the `.post-card-comments` wrapper). `feed.html`/`gists.html`/`projects.html` already loaded `feed.css`; `news.html` loads neither by default, so it loads both.
+
+### Landing page news
+
+Articles can be toggled to appear on the landing page via `/admin/news/{uid}/landing`:
+
+- **`show_on_landing`** field on `news` table
+- Landing route (`main.py` `GET /`) fetches up to 6 articles with `show_on_landing=1`
+- Rendered as a 3-column card grid with image, source, title, date (responsive -> 1 column on mobile)
+- Toggleable individually from the admin news table
+
+### Public feed
+
+The feed page (`GET /feed`) is accessible without authentication:
+
+- Uses `get_current_user(request)` instead of `require_user()` - returns `None` for guests
+- Guests see posts but not the FAB, create modal, inline comment forms, or following tab
+- All POST routes (create, comment, vote) remain guarded by `require_user()`
+- Topnav shows Login/Sign Up for unauthenticated visitors; Messages, Admin, notifications for authenticated
+
+## SEO implementation
+
+All SEO features are implemented across the following locations:
+
+### Core SEO utilities
+- `devplacepy/seo.py` - JSON-LD schema generators (WebSite, BreadcrumbList, DiscussionForumPosting, ProfilePage, SoftwareApplication), meta description truncation, schema combiner, sitemap XML generator
+- `routers/seo.py` - robots.txt and sitemap.xml routes
+
+### SEO template context
+- Every router passes `page_title`, `meta_description`, `meta_robots`, `canonical_url`, `og_title`, `og_description`, `og_image`, `og_type`, `breadcrumbs`, `page_schema` via `base_seo_context()`
+- Auth pages: `noindex,nofollow`
+- Messages/Notifications: `noindex,nofollow`
+- Profiles with < 2 posts: `noindex,follow`
+- All other pages: `index,follow`
+
+### Template layer
+- `templates/base.html` - dynamic `<title>`, `<meta description>`, `<link canonical>`, `<meta robots>`, Open Graph, Twitter Cards, JSON-LD injection, breadcrumb nav, CDN `dns-prefetch`/`preconnect`
+- `static/css/base.css` - `.breadcrumb` (aria-label breadcrumb nav), `.sr-only` (accessible hidden headings)
+
+### Heading hierarchy
+- `feed.html` - `<h1 class="sr-only">Feed</h1>`
+- `profile.html` - username rendered as `<h1 class="profile-name">`
+- `messages.html` - `<h1 class="sr-only">Messages</h1>`
+- `projects.html` - `<h1>Projects</h1>`
+- `post.html` - post title as `<h1>`, "Related Discussions" as `<h3>`
+
+### Post slugs
+- Slug generated on post creation via `slugify()` and stored in `posts.slug` column
+- Posts can be looked up by slug or UUID
+- Minimum content validation: post body >= 10 chars, comment >= 3 chars
+
+### Related posts
+- `templates/post.html` - "Related Discussions" widget at bottom of post page (queried by matching topic)
+
+### Performance
+- `loading="lazy"` on all avatar images
+- `dns-prefetch` + `preconnect` for CDN resources in `<head>`
+- Security headers middleware: `X-Robots-Tag`, `X-Content-Type-Options`
+
+### Default OG image
+- `static/og-default.svg` - 1200x630 SVG with DevPlace branding
+- Used as fallback `og:image` on all pages
+
+### SEO tests
+- SEO tests are split by surface: `tests/api/robotstxt.py`, `tests/api/sitemapxml.py`, and `tests/unit/seo.py` plus the per-page e2e checks - covering robots.txt, sitemap.xml, page titles, noindex, canonical URLs, OG tags, Twitter cards, structured data, security headers
+
+## Engagement: reactions, bookmarks, polls, contribution heatmap
+
+### Emoji reactions
+- Curated palette only: `REACTION_EMOJI` in `constants.py` (registered as a template global). `ReactionForm` rejects anything outside it; free-text emoji are not allowed.
+- Endpoint `POST /reactions/{target_type}/{target_uid}` (`routers/reactions.py`) toggles one `(user, target, emoji)` row in the `reactions` table. Target types: `post`, `comment`, `gist`, `project`. AJAX (`x-requested-with: fetch`) returns `{counts, mine}`.
+- Reactions are **non-ranking** - they never touch `stars` or XP and intentionally send **no notifications** (votes already notify; reactions would be notification spam).
+- Batch reads via `get_reactions_by_targets(target_type, uids, user)` in `database.py` (used by feed, profile, comment loader, `load_detail`) - never per-row. The `_reaction_bar.html` partial takes `_type`, `_uid`, `_reactions` ({counts, mine}) and renders the full palette as toggle chips; `ReactionBar.js` uses document-level click delegation. All four engagement controllers (`ReactionBar`, `VoteManager`, `BookmarkManager`, `PollManager`) extend the shared `OptimisticAction` base (the `Http.sendForm -> render -> error` core); each keeps only its own event wiring and `_render`.
+
+### Bookmarks
+- `POST /bookmarks/{target_type}/{target_uid}` toggles a `bookmarks` row; `GET /bookmarks/saved` renders the personal list (`saved.html`). Target types: `post`, `gist`, `project`, `news`.
+- `_bookmark_button.html` takes `_type`, `_uid`, `_bookmarked`; `BookmarkManager.js` swaps the label/`bookmarked` class from the JSON `{saved}`. Batch state via `get_user_bookmarks(user_uid, target_type, uids)`.
+
+### Polls
+- A poll rides on a post (one `polls` row keyed by `post_uid`, options in `poll_options`, one-per-user votes in `poll_votes`). Created in `posts.py:create_poll` when `poll_question` plus >= 2 non-empty `poll_options` are submitted (capped at 6). Both `create_post` and `edit_post` accept the poll fields; `edit_post` only attaches a poll when the post has **none** yet (it never replaces an existing poll). The builders live in the create-post modal (`feed.html`) and the edit-post modal (`post.html`, rendered only when the post has no poll) using `data-poll-toggle` / `data-poll-add-option`.
+- `poll_options` accepts either repeated form fields (the web builders, which preserve commas inside an option label) **or** a single newline- or comma-separated string (the API/Devii path). `models.py:normalize_poll_options` splits a lone delimited element - applied as a `mode="before"` validator on `PostForm`/`PostEditForm` - so the documented "one per line or comma separated" agent format actually produces a multi-option poll instead of a single dropped option.
+- `POST /polls/{poll_uid}/vote` toggles the voter's choice and returns the full poll dict: voting on a new option switches, voting the **same** option again **retracts** (one vote per user, like reactions). Results (bars + `%`) are **always shown** to everyone; the chosen option gets `.chosen` + a checkmark. Guests see read-only bars with a "Log in to vote" hint (options `disabled`). Batch reads via `get_polls_by_post_uids(post_uids, user)` / `get_poll_for_post(post_uid, user)`.
+- Composer poll builder (`feed.html`, `PollManager.js`): poll inputs are `disabled` until "Add poll" is opened (so a never-opened or "Remove poll"-collapsed builder submits **nothing**); a capture-phase `submit` listener blocks an incomplete poll (question + >= 2 options) with an inline error instead of silently dropping it - capture phase + `stopImmediatePropagation()` is required so it pre-empts `FormManager`'s bubble-phase submit handlers.
+
+### Cascade
+- `soft_delete_engagement(target_type, uids, deleted_by)` in `database.py` soft-deletes reactions, bookmarks and (for posts) poll rows. It is called from `content.py:delete_content_item` (for the item and its comments) and `comments.py:delete_comment` (which now soft-deletes the comment plus its attachments, votes, and engagement under one `stamp`). Add it to any new delete path. The unfiltered hard `delete_engagement` remains for GC only.
+
+### Contribution heatmap and streaks
+- Derived, **no new table**: `get_activity_calendar(user_uid)` aggregates `date(created_at)` across `posts`/`comments`/`gists`/`projects` (cached 120s); `get_activity_heatmap` returns 53 Monday-aligned weeks of `{date, count, level}` (level 0-4); `get_streaks` returns `{current, longest}`. Rendered server-side in `profile.html` (`.heatmap-grid`, styles in `profile.css`). The `On Fire` badge is awarded in `check_milestone_badges` when the current streak >= 7.
+
+### Follow graph listings
+- `profile.html` has Followers and Following tabs (`?tab=followers` / `?tab=following`, `?page=N`). `profile_page` computes `get_follow_counts(uid)` for the tab labels and, only for those two tabs, `_follow_people(uid, tab, current_user, page)`. The same two listings are exposed as JSON at `GET /profile/{username}/followers` and `GET /profile/{username}/following` (public, 25 per page) returning `{username, mode, count, page, total_pages, <mode>: [{uid, username, bio, is_following}]}`.
+- `database.py` helpers: `get_follow_counts(uid)` (`{followers, following}`), `get_follow_list(uid, mode, page)` (paginated people + `build_pagination`, ordered newest-first), and `get_following_among(follower_uid, target_uids)` (single IN-clause set used to set `is_following` per row, avoiding N+1). `mode` is `"followers"` (people who follow `uid`) or `"following"` (people `uid` follows).
+- Devii catalog tools `list_followers` / `list_following` (`requires_auth=False`) map to the JSON endpoints; documented in `docs_api.py` under the `profiles` group.
+
+### Block and mute (`routers/relations.py`)
+A logged-in user can **block** or **mute** another user; both are one-directional and reversible. **Block** hides every piece of the blocked user's content from the blocker - posts, comments (any category), feed, listings, issue list, detail pages, and DMs - everywhere EXCEPT the blocked user's own profile page (kept fully visible so the blocker can review and unblock), and it also suppresses any notification that user would generate. **Mute** is the lighter option: it only suppresses the muted user's notifications while their content stays visible. The blocked/muted user is unaffected and is not told.
+
+- **One table** `user_relations(uid, user_uid, target_uid, kind, created_at, deleted_at, deleted_by)` in `SOFT_DELETE_TABLES`; `user_uid` is the actor/owner, `kind` is `"block"` or `"mute"`. Indexes `idx_user_relations_owner_kind` (`user_uid, kind`) and `idx_user_relations_target_kind` (`target_uid, kind`, backs the DM-send reverse check), ensured in `init_db`.
+- **One cached accessor** `database.get_user_relations(viewer_uid) -> {"block": frozenset, "mute": frozenset}` (single query, both kinds), cache-invalidated under the `"relations"` cache-version name; anonymous viewer returns empty frozensets with **zero queries**. Derived helpers `get_blocked_uids`, `get_muted_uids`, `get_silenced_uids` (block | mute) and `invalidate_user_relations(uid)` (local pop + `bump_cache_version("relations")`). This is read on every content listing, so it must stay cache-hot - never query `user_relations` directly in a read path.
+- **Filtering choke points (DRY, do not re-implement inline):** `paginate`/`paginate_diverse` accept a `viewer_uid` kwarg that appends `user_uid NOT IN (blocked)` only when the table has `user_uid` and the block set is non-empty (wired in `feed`/`gists`/`projects` listings; safe no-op for bookmarks/news); the three comment loaders drop blocked authors via `database._drop_blocked(raw, user)` before building trees; `content.load_detail` returns `None` (404) for a blocked author; the landing query and the issues list filter by the block set; messages filter the conversation list/thread and `persist_message` rejects a DM when the recipient blocked the sender (returns `None`, already handled).
+- **Notification suppression is one line** at the single funnel `utils._deliver_notification`: `if related_uid and related_uid in get_silenced_uids(user_uid): return`. This covers follows, mentions, DMs, votes, reactions at once (block silences too, hence `get_silenced_uids`). A non-user `related_uid` is harmless (uuids never collide).
+- **Routes** mirror `follow.py`: `POST /block/{username}`, `POST /block/unblock/{username}`, `POST /mute/{username}`, `POST /mute/unmute/{username}` (born-live insert / soft-delete revive, `invalidate_user_relations`, best-effort audit `relation.block|unblock|mute|unmute`, `action_result`). Mounted with no prefix.
+- **Fan-out:** profile route passes `is_blocked`/`is_muted` (named to avoid the `is_self` Jinja-global collision rule) on the `respond` context + `ProfileOut`; `profile.html` renders Block/Unblock + Mute/Unmute POST forms next to Follow using the existing `data-confirm` (+`data-confirm-danger`) pattern - **no new JS**. Devii actions `block_user`/`unblock_user`/`mute_user`/`unmute_user`; docs in `docs_api.py`; audit prefix `relation` -> category `social`.
+
+### CSS
+- Reactions, bookmarks, polls and the saved page live in `static/css/engagement.css`, loaded globally in `base.html` (the controls appear across feed, detail pages and comments). Heatmap styles are in `profile.css`. Follow-list rows (`.follow-row`, `.follow-user`, `.follow-pagination`) are in `profile.css`.
+
+## Profile activity tab (clickable comments)
+
+The profile **Activity** tab (`/profile/{username}?tab=activity`, public) interleaves the user's 10 most recent posts and 10 most recent comments, newest first. Each activity item is a plain dict built inline in `routers/profile/index.py` (no dedicated DB helper) and carries a `url` so the whole card is clickable, exactly like the feed's post cards:
+
+- **Post items** get `"url": resolve_object_url("post", p["uid"])` -> `/posts/{slug}`.
+- **Comment items** get `"url": f"{resolve_object_url(target_type, target_uid)}#comment-{c['uid']}"`, the SEO-friendly parent detail URL plus the `#comment-{uid}` anchor. This is **byte-identical to a comment notification's `target_url`** (`content.create_comment_record`), so clicking an activity comment reproduces the notification click exactly: it lands on the parent post and `NotificationManager.js` scrolls to / highlights the `id="comment-{uid}"` element (`_comment.html`). Reuse `resolve_object_url` (`database.py`) - never rebuild this URL by hand, and do not call `resolve_object_url("comment", uid)` here (it re-fetches the row the loop already has).
+- **Frontend:** the card is wrapped in the shared full-card overlay link (`card-link-host` + `_card_link.html`), the same pattern `_post_card.html` uses; inner content links (mentions, URLs rendered by `render_content`) stay clickable via the existing `card-link-host a:not(.card-link)` z-index rule (`base.css`). No new CSS.
+- **API:** `ProfileOut.activities` is `list[Any]`, so the added `url` (and, on comments, the existing `target_type` + `uid` = parent target uid) flow into the JSON response with no schema change - this is the parent-post reference exposed on each comment item.