# CLAUDE.md
This file provides guidance to Claude Code when working with code in this repository. It holds only what applies regardless of which part of the codebase is being touched. Deep, subsystem-specific detail lives in nested `CLAUDE.md` files placed inside the relevant directory - Claude Code auto-loads a nested file only when a file under that directory is read or edited, so the always-loaded cost of this repository stays proportional to this file alone. See "Subsystem map" below for the full list.
## Project
DevPlace is a server-rendered social network for developers. FastAPI backend serves Jinja2 templates with pure ES6 module JavaScript on the frontend. SQLite via the `dataset` library (auto-syncs schema). No JS framework, no NPM, no JWT.
- **Database:** `dataset` (auto-syncs schema, uses `uid` for PKs). SQLite.
- **Auth:** `session` cookie, plus `X-API-KEY` / `Authorization: Bearer <api_key>` / HTTP Basic (username-or-email:password) - all resolved in `get_current_user`. PBKDF2-SHA256 via passlib. No JWT. Every user has an `api_key` (uuid7), set at signup and backfilled in `init_db`/`devplace apikey backfill`. Docs site at `/docs` (`routers/docs/` package, `DOCS_PAGES` registry; FastAPI's Swagger is moved to `/swagger` so `/docs` is free).
- **Static:** `devplacepy/static/` mounted at `/static`; URLs are boot-versioned (`/static/v<ts>/...`) via `static_url`/`assetUrl` and served immutable for a year.
- **Templates:** `devplacepy/templates/`. Shared `templates` instance from `devplacepy.templating` - all routers import from there, never instantiate their own.
- **Ports:** 10500 (dev), 10501 (tests; the serial suite uses a single uvicorn subprocess).
- **Username:** letters, numbers, hyphens, underscores only, 3-32 chars. **Password:** minimum 6 chars.
- **Avatars:** Multiavatar SVG generated locally from a seed in <5ms, in-memory cache cleared on restart, fallback to initial-based SVG on error. URL `/avatar/multiavatar/{seed}?size={size}`. The seed is per-user: the nullable `users.avatar_seed` column overrides the username. Always resolve it through the single null-safe choke point `avatar.avatar_seed(user)` (a Jinja global) - `user.get("avatar_seed") or user.get("username")` - never read `avatar_seed` or pass `username` to `avatar_url(...)` directly; every render site (the `_avatar_link.html` partial, `og_image`, the devRant avatar payload, etc.) goes through it. Regenerate is owner-or-admin at `POST /profile/{username}/regenerate-avatar` (writes a fresh `generate_uid()`, invalidates the user cache, audits `profile.avatar.regenerate`); the old seed is never stored, so the old avatar cannot return. Devii tool `regenerate_avatar` (`CONFIRM_REQUIRED`).
## Commands
```bash
make install # pip install -e . + playwright install chromium
make ppy # build the single shared container image (ppy:latest); run once before launching instances
make dev # uvicorn --reload on port 10500, backlog 4096
make prod # uvicorn --workers $(WEB_WORKERS) (defaults to nproc), port 10500 (backlog 8192)
make test # Playwright + unit tests, headless, serial (one at a time), -x fail-fast
make test-headed # same tests in a visible Chromium window (single process)
make locust # Locust load test, interactive web UI
make locust-headless # Locust CLI mode for CI
```
The Makefile exports `PYTHONDONTWRITEBYTECODE=1` for every recipe, so no `.pyc` files or `__pycache__` directories are written by any `make` target. Keep it that way - do not add a target that re-enables bytecode writing. `make clean` removes any stray bytecode left from running Python outside make.
Validate code without running the suite: confirm `python -c "from devplacepy.main import app"` imports clean and check each touched language manually (Python compiles/imports, JS parses, CSS braces and HTML tags balance).
Single test: `python -m pytest tests/e2e/feed.py::test_name -v --tb=line -x`
CLI (installed as `devplace`):
```bash
devplace role get <username>
devplace role set <username> <member|admin>
devplace apikey get <username> # print a user's API key
devplace apikey reset <username> # regenerate a user's API key
devplace apikey backfill # assign API keys to users that lack one
devplace token issue <username> [--label L] # issue a DevPlace access token
devplace token list <username> # list a user's active access tokens
devplace token revoke <token_uid> # revoke a single access token by uid
devplace token revoke-all <username> # revoke all access tokens for a user
devplace token prune # soft-delete all expired access tokens
devplace news clear # delete all news rows
devplace news sanitize # strip HTML from news descriptions/content
devplace attachments prune # remove orphan attachment records/files
devplace devii reset-quota <username> # reset one user's rolling 24h AI quota
devplace devii reset-quota --guests # reset every guest quota
devplace devii reset-quota --all # reset every quota (users and guests)
devplace zips prune # delete expired zip archives + job rows
devplace zips clear # delete every zip archive + job row
devplace forks prune # delete expired completed fork job rows (forked projects persist)
devplace forks clear # delete every fork job row (forked projects persist)
devplace seo prune # delete expired SEO audit reports + job rows
devplace seo clear # delete every SEO audit report + job row
devplace seo-meta prune # delete expired SEO metadata job rows (generated metadata persists)
devplace seo-meta clear # delete every SEO metadata job row (generated metadata persists)
devplace deepsearch prune # delete expired DeepSearch sessions + job rows + collections
devplace deepsearch clear # delete every DeepSearch session + job row + collection
devplace isslop analyze <url> # run a AI usage analysis from the terminal (report persists)
devplace isslop prune # delete expired AI usage analysis job rows (analyses + reports persist)
devplace isslop clear # delete every AI usage analysis, its report and job rows
devplace backups list # list recorded backups
devplace backups run <database|uploads|keys|full> # enqueue a backup (processed by the running server)
devplace backups prune # remove backup records whose archive file is missing
devplace backups clear # delete every backup archive + record
devplace containers list # list container instances
devplace containers reconcile # run one reconcile pass (desired vs docker ps)
devplace containers prune # reap orphan containers + dangling images
devplace containers prune-builds # remove legacy per-project images + clear dockerfiles/builds tables (one-time)
devplace containers gc-workspaces # remove workspace dirs with no instances
devplace emoji-sync # regenerate static/js/emoji-shortcodes.js from the emoji library (run after an emoji dep bump)
devplace migrate-data # relocate legacy runtime files into data/ (idempotent; --dry-run to preview)
```
### Runtime data layout (single source of truth)
Every runtime/user-generated artifact lives under one root, `config.DATA_DIR` (`DEVPLACE_DATA_DIR`, default `<repo>/data`). `config.py` derives every runtime path from it and lists them in the `DATA_PATHS` registry; `ensure_data_dirs()` creates the whole tree before anything is written. **No module computes a runtime path from scratch** - import the constants (`UPLOADS_DIR`, `ATTACHMENTS_DIR`, `PROJECT_FILES_DIR`, `ZIPS_DIR`, `ZIP_STAGING_DIR`, `FORK_STAGING_DIR`, `KEYS_DIR`, `BOT_DIR`, `LOCKS_DIR`, `CONTAINER_WORKSPACES_DIR`, `DEVII_TASKS_DB`, `DEVII_LESSONS_DB`, `VAPID_*_FILE`, `SERVICE_LOCK_FILE`, `INIT_LOCK_FILE`). Uploads are physically under `data/uploads/` but still served at the unchanged `/static/uploads/` URL. Stored DB values for attachments/project files are relative (`directory` + `stored_name`), so no DB rewrite is ever needed for a layout change.
**Blob sharding (`attachments._directory_for`, reused by `attachments`, `project_files`, `zip_service`):** blobs are spread into a two-level `xx/yy` tree keyed on the **random tail** of the uuid7 (`tail[-2:]/tail[-4:-2]`), NEVER the leading bytes - a uuid7's first 48 bits are a millisecond timestamp, so prefix-sharding a time-ordered id funnels every contemporaneous write into one bucket. Any new shard helper MUST shard on a high-entropy field (uuid tail or a hash), never the head. Forward-only: pre-existing rows keep resolving from their stored `directory`/`local_path`.
Tests run on port 10501 with a tempfile SQLite DB and a dedicated `DEVPLACE_DATA_DIR`, `DEVPLACE_DISABLE_SERVICES=1`. `make test` runs **serially, one test at a time, in a single process**, enforced centrally in `pyproject.toml` (pytest-xdist is not a dependency, `-n` is rejected).
## Environment variables
| Var | Default | Purpose |
|-----|---------|---------|
| `DEVPLACE_DATABASE_URL` | `sqlite:///<repo>/data/devplace.db` | Override DB path (tests use this) |
| `SECRET_KEY` | hardcoded fallback | Session signing |
| `DEVPLACE_DISABLE_SERVICES` | unset | When `1`, NewsService and other background services skip start (set by test conftest) |
| `PLAYWRIGHT_HEADLESS` | `1` in tests | Toggle headed mode |
| `DEVPLACE_TEMPLATE_AUTO_RELOAD` | `1` (on) | Jinja template auto-reload. `1` stat-checks every template per render (dev hot-reload); set `0` in production so compiled templates stay cached in memory. |
| `DEVPLACE_WEB_WORKERS` / `--workers` | `nproc` (prod) | Uvicorn worker count; `make prod WEB_WORKERS=N` to override. |
| `DEVPLACE_PRESENCE_TIMEOUT_SECONDS` | `60` | How long after a user's last activity they still count as online. `config.PRESENCE_WRITE_SECONDS` (half of it) throttles `last_seen` writes per worker. |
| `DEVPLACE_PRESENCE_ONLINE_LIMIT` | `30` | Max avatars in the feed's live "Online now" panel. |
| `DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS` | `20` | Grace margin (hysteresis) before an online user is dropped, kills dot/roster flicker at the boundary. |
| `DEVPLACE_DATA_DIR` | `<repo>/data` | Single root for ALL runtime/user-generated data OUTSIDE the package and OUTSIDE `/static`. Point at a volume in prod. |
## Subsystem map
Nested `CLAUDE.md` files (loaded automatically by Claude Code only when a file in that directory is touched):
| Path | Covers |
|------|--------|
| `devplacepy/routers/CLAUDE.md` | Full URL prefix map, route aggregation rules, HTML/JSON negotiation, FastAPI patterns, gists, feed/listing features, engagement (reactions/bookmarks/polls/heatmap/follow/block-mute), SEO implementation, polymorphic comments/votes |
| `devplacepy/routers/projects/CLAUDE.md` | Project detail page, virtual filesystem routes, visibility/read-only UI, deletion confirmation |
| `devplacepy/routers/docs/CLAUDE.md` | The `/docs` documentation site: `DOCS_PAGES`, audience tiers, prose rendering, API tester |
| `devplacepy/routers/devrant/CLAUDE.md` | devRant-compatible REST API (`/api`) |
| `devplacepy/services/containers/CLAUDE.md` | Container manager: backend, security, `ppy` image, ingress, terminals, sync |
| `devplacepy/services/devii/CLAUDE.md` | Devii assistant: sessions/channels, scheduler, virtual tools, self-configured behavior, client browser tools |
| `devplacepy/services/openai_gateway/CLAUDE.md` | AI gateway: `/openai/v1/*`, usage ledger, provider/model routing |
| `devplacepy/services/jobs/CLAUDE.md` | Async job services: zip, fork, SEO diagnostics, SEO metadata, DeepSearch, AI Usage Analyzer |
| `devplacepy/services/audit/CLAUDE.md` | Audit log: recorders, categories, retention |
| `devplacepy/services/backup/CLAUDE.md` | Backup service: targets, worker, schedules, primary-admin-only download |
| `devplacepy/services/telegram/CLAUDE.md` | Telegram bot bridge |
| `devplacepy/services/email/CLAUDE.md` | Devii IMAP/SMTP email tools |
| `devplacepy/services/gitea/CLAUDE.md` | Issue tracker (Gitea-backed, no local issue store) |
| `devplacepy/services/messaging/CLAUDE.md` | Real-time DM chat (WS + relay) |
| `devplacepy/services/xmlrpc/CLAUDE.md` | XML-RPC bridge |
| `devplacepy/services/news/CLAUDE.md` | `NewsService` import pipeline |
| `devplacepy/services/bot/CLAUDE.md` | `BotsService` fleet |
| `devplacepy/services/dbapi/CLAUDE.md` | `/dbapi` primary-admin-only read-only database API |
| `devplacepy/services/pubsub/CLAUDE.md` | Database-free pub/sub bus |
| `devplacepy/services/game/CLAUDE.md` | Code Farm idle game |
| `devplacepy/services/CLAUDE.md` | Background task queue, AI correction/modifier, presence, live view relay, `BaseService`/`ServiceManager` |
| `devplacepy/database/CLAUDE.md` | Dataset rules, indexing conventions, soft delete, tables, site settings |
| `devplacepy/utils/CLAUDE.md` | Notifications and gamification (XP/levels/badges/leaderboard) |
| `devplacepy/static/js/CLAUDE.md` | Custom web components, shared frontend utilities (Http/Poller/FloatingWindow/ScrollMemory) |
| `devplacepy/templates/CLAUDE.md` | Modal system, CDN libraries, shared template partials |
| `tests/CLAUDE.md` | Detailed testing patterns and pitfalls |
`isslop/` at the repo root is a separate, standalone sibling project (own `pyproject.toml`, `Makefile`, port 18732) with its own `isslop/CLAUDE.md` - it is not a nested subsystem of the `devplacepy` package. The integrated engine that DevPlace actually runs (`devplace isslop analyze`, the `/tools/isslop` job service) is a distinct implementation documented in `devplacepy/services/jobs/CLAUDE.md`.
## Architecture
### Request pipeline
`main.py` mounts `/static`, registers every router with its prefix, installs middlewares (security headers, a per-IP rate limit in an in-process `defaultdict`, a maintenance gate), and registers global 404/500 handlers. Rate limiting reads `rate_limit_per_minute`/`rate_limit_window_seconds` from `site_settings` (default 60/60s) and **applies only to mutating methods** (`POST`/`PUT`/`DELETE`/`PATCH`); reads and the whole `/openai` gateway are exempt. Bucket key is `X-Real-IP` falling back to `request.client.host`; an over-limit request gets `429` with `Retry-After`. The maintenance middleware short-circuits non-admin requests with a 503 when `maintenance_mode="1"`, but always allows `/static`, `/avatar`, `/auth`, `/admin`, and admin users. The outermost middleware is `response_timing`: stamps `request.state.request_start` and sets `X-Response-Time` on every response; the `response_time_ms(request)` Jinja global renders it as a fixed bottom-left badge in `base.html`. This is the single generic timing mechanism - never re-time per route.
`@app.on_event("startup")` calls `init_db()` and, unless `DEVPLACE_DISABLE_SERVICES` is set, registers `NewsService` and kicks off `start_all()`. `GET /` never redirects: guests get the marketing splash, authenticated users get a personalized home, both sharing Latest Posts + Developer News. The feed and Latest Posts enforce **author diversity** by interleaving authors (never dropping posts) via `interleave_by_author`/`paginate_diverse` in `database/`.
### Routing layout
Routers in `devplacepy/routers/` are organised as a **directory tree that mirrors the endpoint (URL) path**, exactly like `tests/`. A domain with a single resource stays one flat file (`feed.py`, `posts.py`, ...); a domain with several sub-resources is a **package directory** split one file per sub-resource. Each leaf declares its own `router = APIRouter()`; the package `__init__.py` aggregates with `router.include_router(...)`. A leaf owning the domain's collection-root (`""`) route must be the package's base router (FastAPI rejects an empty path under an empty prefix). Full elaboration, per-domain detail, and the complete deep-dive live in `devplacepy/routers/CLAUDE.md` - read it before touching routing. Compact map:
| Prefix | Router |
|--------|--------|
| `/auth` | auth/ package |
| `/feed`, `/posts`, `/comments` | flat files |
| `/projects`, `/projects/{slug}/files`, `/projects/{slug}/containers` | projects/ package - see `routers/projects/CLAUDE.md` |
| `/profile` | profile/ package (customization, notifications, ai-correction, ai-modifier, telegram, usage) |
| `/messages` | messages.py - see `services/messaging/CLAUDE.md` |
| `/notifications`, `/votes`, `/reactions`, `/bookmarks`, `/polls`, `/avatar`, `/follow`, `/leaderboard` | flat files |
| (none) | relations.py - block/mute |
| `/admin`, `/admin/services`, `/admin/containers` | admin/ package |
| `/issues` | issues/ package - see `services/gitea/CLAUDE.md` |
| `/gists`, `/news`, `/uploads`, `/media` | flat files |
| `/openai` | openai_gateway.py - see `services/openai_gateway/CLAUDE.md` |
| `/devii` | devii.py - see `services/devii/CLAUDE.md` |
| `/zips`, `/forks` | see `services/jobs/CLAUDE.md` |
| `/tools` | tools/ package (SEO diagnostics, DeepSearch, AI Usage Analyzer) - see `services/jobs/CLAUDE.md` |
| `/p/{slug}` | proxy.py - container ingress reverse proxy |
| `/xmlrpc` | xmlrpc.py - see `services/xmlrpc/CLAUDE.md` |
| `/api` | devrant/ package - see `routers/devrant/CLAUDE.md` |
| `/dbapi` | dbapi/ package, **primary-administrator-only, strictly READ-ONLY** - see `services/dbapi/CLAUDE.md` |
| `/game` | game/ package - see `services/game/CLAUDE.md` |
| (none) | push.py (web push/PWA), docs.py (see `routers/docs/CLAUDE.md`) |
| `/pubsub` | pubsub.py - see `services/pubsub/CLAUDE.md` |
| (none) | seo.py - `/robots.txt`, `/sitemap.xml` |
### Templates and frontend
- All routers MUST import the shared `templates` from `devplacepy.templating`. Do NOT instantiate `Jinja2Templates` per router.
- Templates extend `base.html`; page CSS via `{% block extra_head %}`, page JS via `{% block extra_js %}`.
- **Nav active state is the `nav_active` Jinja global**, never an inline path check.
- **Static asset URLs are boot-versioned:** never hardcode a bare `/static/...` href/src - use `static_url(path)` (Jinja) / `assetUrl(path)` (JS).
- All JS is ES6 modules, one class per file, instantiated as `app`. Custom web components (`dp-*` prefix) and shared frontend utilities (`Http`, `Poller`, `JobPoller`, `OptimisticAction`, `FloatingWindow`, `ScrollMemory`) are documented in `devplacepy/static/js/CLAUDE.md` - reuse them, never re-implement.
- CDN scripts in `base.html` MUST use `defer` or `type="module"`.
- Modal system, shared template partials (`_avatar_link.html`, `_user_link.html`, `_sidebar_search.html`) are documented in `devplacepy/templates/CLAUDE.md`.
### Content rendering pipeline
**Server-rendered content and titles are rendered on the BACKEND for SEO** (`devplacepy/rendering.py`, Jinja globals `render_content(text)`/`render_title(text)`). This is the default for all content that exists at request time (post bodies, comments, news, project/gist descriptions, DM history, every content title). It mirrors the client pipeline using **mistune**: em-dash normalization, emoji shortcodes, GFM markdown (`escape=True` - the server XSS control), then a media pass turning bare URLs into embeds and `@mentions` into links. Both are `@lru_cache`d. Server-rendered code blocks get syntax highlighting + Copy button client-side via `ContentEnhancer`.
The CLIENT pipeline (`ContentRenderer.js`, `dp-content`/`dp-title`) is retained ONLY for genuinely live/dynamic content that does not exist at request time (live comments, DM bubbles, Devii/DeepSearch/Docs chat, planning report). Runs `marked` -> **`DOMPurify.sanitize`** (the client XSS control, fail-closed) -> highlight.js -> media/autolink. **Do NOT add `data-render` to server-rendered content - call `render_content`/`render_title` instead.**
**Emoji shortcodes are the full GitHub/Discord `:name:` set** (~4869 names), generated once from the `emoji` library by `rendering.py` `build_emoji_shortcodes()`; the frontend gets the identical map via the generated `static/js/emoji-shortcodes.js` (regenerate with `devplace emoji-sync` after bumping the dependency, never hand-edit).
**Template em-dash normalization:** the shared `templates.env.template_class` runs every rendered template's final HTML through `normalize_em_dash` - the em-dash character and its HTML entity forms all become a plain hyphen, application-wide. One hook; never re-strip em-dash per template.
### Auth
Session cookies named `session`, value a 64-char hex token. Passwords hashed with `pbkdf2_sha256` via passlib. `get_current_user(request)` returns user dict or `None`, per-process TTL cache (300s) keyed by token. `require_user(request)` raises a 303 redirect to `/` for guests; `require_admin(request)` redirects to `/feed` for non-admins. Public pages use `get_current_user` so guests can browse - POSTs are all guarded by `require_user`.
### Database
SQLite via `dataset.connect` with WAL + 30s busy timeout + 256MB mmap. **SQLite is synchronous by design and will never be made async (hard rule).** The `dataset`/SQLite layer is called directly inside async route handlers - the database is a local file tuned with WAL/`synchronous=NORMAL`/8MB page cache/256MB mmap, so queries are sub-millisecond and never meaningfully stall the event loop. Do NOT convert the database layer to async, wrap calls in a threadpool/`run_in_executor`/`to_thread`, or raise synchronous DB access as a blocking-I/O concern - this is a settled design decision, not open for revisiting.
`init_db()` is idempotent and **ensures the full column set of any table code filters on** before creating indexes - `dataset` gives a lazily-created table only the columns of its first insert, so a partial insert elsewhere would leave a reduced schema that later 500s with `no such column`. Full dataset rules, indexing conventions (the soft-delete planner trap), site settings, and the complete table list live in `devplacepy/database/CLAUDE.md`.
Runtime config lives in `site_settings`, read via `get_setting(key, default)`/`get_int_setting(key, default)` (60s TTL cache, cross-worker invalidated via `cache_state`). Batch helpers (`get_users_by_uids`, `get_comment_counts_by_post_uids`, `get_vote_counts`, `load_comments`) exist specifically to avoid N+1 - use them instead of per-row lookups.
### Per-user site customization (CSS/JS)
Users and guests inject their own CSS and JS, scoped to a page type or globally, configured **conversationally through Devii only** (no HTTP routes - persistence is owner-scoped from the trusted Devii session, like `LessonStore`). It runs solely in that owner's own browser sessions (self-XSS, like a userscript) - never in anyone else's view.
- **Page type** = the matched route template (`page_type_for(request)`, e.g. `/posts/{slug}`) - one value covers all instances of a type. **Owner** = `owner_for(request)`: user uid, else the `DEVII_GUEST_COOKIE`, else none.
- **Storage:** table `user_customizations` (per `owner_kind`/`owner_id`/`scope`/`lang`); `get_custom_overrides` merges global-then-page-type, cached under the `"customizations"` cache-version name.
- **Per-user suppression toggles** (distinct from the site-wide `customization_enabled` kill-switch): `cust_disable_global`/`cust_disable_pagetype` columns let a user hide their own customizations without deleting rows, edited at `POST /profile/{username}/customization/{global|pagetype}` (owner or admin).
- **Robustness against hostile CSS:** app chrome (`.topnav`, overlays, modals, context menu, toast host, FAB, lightbox, devii window) is layer-promoted with `will-change: transform` so a user's `position: fixed`/`background-attachment: fixed` CSS cannot trigger a Chromium compositing bug that wipes fixed UI. Any new always-on fixed UI element must join one of these groups.
- **Injection:** `custom_css_tag(request)` emits a `<style>` after `extra_head`; `custom_js_tag(request)` emits a JSON island run via `new Function(JSON.parse(...))` after `Application.js`. Both fail closed to empty Markup - a customization bug must never break page render. Two kill-switches: `customization_enabled`, `customization_js_enabled`.
- **Devii** (`services/devii/customization/`, `handler="customization"`): `customize_list/get/set_css/set_js/reset` (`requires_auth=False`) plus `customize_set_enabled` (the suppression toggles). Set/reset are in `CONFIRM_REQUIRED`, forcing Devii to ask page-type vs global before `confirm=true`. Devii previews live with `run_js`, then `reload_page`.
### Background task queue, AI correction/modifier, background services
`services/background.py` `background.submit(fn, *args)` is a generic fire-and-forget offload onto one per-worker `asyncio.Queue`; when the consumer isn't running (tests, full queue) it runs the callable **inline**, so audit/notification/XP writes stay deterministic for the suite while production defers them. It is the choke point for every audit-log write, every XP award, and every notification - handlers call `award_rewards`/`create_notification` directly (never wrap them in `background.submit`, that double-queues). AI content correction (opt-in, off by default) and the AI modifier (`@ai <instruction>` inline directive, on by default) rewrite user prose via the internal gateway; sync apply mode never blocks the event loop (`run_in_executor` + the `await_pending_corrections` middleware). `BaseService`/`ServiceManager` provide the async run loop and singleton registry for background services (`NewsService`, `GatewayService`, `DeviiService`, `BotsService`, container/audit/telegram reconcilers). Full detail on all of this, plus presence and the live view relay, is in `devplacepy/services/CLAUDE.md`.
### Container manager, Devii assistant, AI gateway, async jobs, audit log
DevPlace ships a Docker-backed **Container manager** (admin-only, one shared `ppy:latest` image, security-hardened rootless workflow via a `sudo` superclone and `aptroot`), the **Devii** in-platform agentic assistant (WebSocket terminal, per-owner-channel sessions, scheduler/reminders, self-learning lessons, user-defined virtual tools, self-configured behavior, browser-automation client tools), the **AI gateway** (`/openai/v1/*`, single point of truth for every AI call, per-user cost attribution, provider/model routing overlay), **async job services** (`JobService` base pattern: zip, fork, SEO diagnostics, SEO metadata generation, DeepSearch, AI Usage Analyzer), and an admin-only append-only **audit log** (`record`/`record_system` entrypoints, `events.md` catalogue of 223 keys, never raises into the caller). These are among the largest subsystems in the codebase - read their dedicated nested `CLAUDE.md` files (see Subsystem map) before working in any of them; do not assume prior knowledge from this summary.
### Telegram bot, email, devRant compatibility API, issue tracker
Devii is also reachable over **Telegram** (one supervised long-poller subprocess, `channel="telegram"` isolated conversation thread), can drive a user's own **external mailbox** over IMAP/SMTP (stdlib only, credentials in a soft-deletable table, SSRF-guarded), and DevPlace exposes a **devRant-compatible REST API** at `/api` (translates devRant requests onto native posts/comments/votes via reused audited cores, ID mapping is `posts.id`/`comments.id` directly). The **issue tracker** at `/issues` has no local store - it reads/writes Gitea live via one shared bot token, filing is an async AI-enhanced job. Full detail in the respective nested `CLAUDE.md` files.
### SEO
`devplacepy/seo.py` generates JSON-LD schemas (WebSite, BreadcrumbList, DiscussionForumPosting, ProfilePage, SoftwareApplication). Every router builds context via `base_seo_context(request, ...)`. Auth/messages/notifications are `noindex,nofollow`; profiles with fewer than 2 posts are `noindex,follow`. `/robots.txt` and `/sitemap.xml` are served by `routers/seo.py`. Full implementation map (template layer, heading hierarchy, slugs, related posts, performance, default OG image, SEO tests) is in `devplacepy/routers/CLAUDE.md`.
## Conventions (project-specific)
- **No comments, no docstrings in source.** Code is self-documenting.
- **Forbidden variable name patterns:** `_new`, `_old`, `_temp`, `_v2`, `better_`, `my_`, `the_`.
- **Outbound HTTP goes through the stealth client only.** Never instantiate a bare `httpx.AsyncClient(...)`/`httpx.Client(...)` for outbound traffic - build it via `devplacepy.stealth.stealth_async_client(...)`/`stealth_sync_client(...)`. The async factory routes bytes through **`curl_cffi` (curl-impersonate, BoringSSL)** behind an `httpx` transport adapter, so TLS JA3/JA4 and the HTTP/2 fingerprint match Chrome 146 byte-for-byte; falls back to a pure-httpx Chrome-aligned transport if `curl_cffi` is missing. Applies to news, gateway upstream, Devii/DeepSearch/SEO fetch, gitea, push, attachments. SSRF-guarded fetches via `net_guard.guarded_async_client(...)` get the fingerprint automatically. Sole exception: `routers/proxy.py` relays the user's own headers verbatim. **Load-bearing gotcha:** cleartext `http://` is forced to HTTP/1.1 (`curl_transport.http_version_for`) - the Chrome-impersonation profile defaults to HTTP/2, but a plaintext origin has no TLS/ALPN to negotiate h2 and uvicorn (HTTP/1.1 only) returns `400 "Invalid HTTP request received."` once a cleartext HTTP/2 body crosses ~128KB. Every internal AI consumer calls `INTERNAL_GATEWAY_URL` = `http://localhost:<port>/openai/v1/...`, so without this override a large Devii turn (big system prompt + tool list) fails on every request. `https://` keeps the HTTP/2 default. `stealth_sync_client` stays pure-httpx Chrome-aligned (no curl_cffi sync adapter).
- **Form validation is Pydantic-native.** Routers declare `data: Annotated[SomeForm, Form()]` (models in `models.py`); the global `RequestValidationError` handler re-renders auth pages with messages (400) or redirects other routes. Read raw `await request.form()` only when also reading an uploaded file alongside a model.
- **`RedirectResponse(url=..., status_code=302)`** - always pass `status_code` as an integer literal, never `status.HTTP_302_FOUND`.
- **Ownership checks:** compare `resource["user_uid"] == user["uid"]` before edit/delete (use `content.is_owner`). **Deletes additionally allow any admin:** `is_owner(...) or is_admin(user)` on every content delete endpoint, so an administrator may delete any member's content while members are limited to their own. Edits stay owner-only. Deletes are **soft** and cascade under one shared stamp - see `devplacepy/database/CLAUDE.md`.
- **All dates shown to users are DD/MM/YYYY** (European), rendered in the viewer's own timezone client-side. Timestamps are stored/emitted as UTC ISO. Use the `local_dt(iso, mode)`/`dt_ago(iso)` Jinja globals for any user-facing instant - they emit `<time data-dt>` and `static/js/LocalTime.js` reformats to local timezone with a `MutationObserver` for dynamic content. `format_date()`/`time_ago()` stay as plain-text helpers for JSON responses, no-JS fallbacks, and non-timestamp date fields (e.g. project `release_date`) - do NOT wrap those in `local_dt`.
- **Slug + UUID lookup:** resources with slugs accept either the slug or the bare UUID via `resolve_by_slug()`. Slugs are `make_combined_slug(title, uid)`, prefixed with the **random tail** of the UUID (never the leading bytes - same timestamp-collision reasoning as blob sharding).
- **Roles are stored capitalized:** `users.role` is exactly `"Admin"` or `"Member"`. Always test admin-ness through the `is_admin(user)` global (case-sensitive `== "Admin"`) - never hand-roll a lowercase compare. **Any write to `users.role` MUST call `database.invalidate_admins_cache()`.**
- **Admin seniority: a junior admin cannot manage a more senior admin.** Every per-user mutation in `routers/admin/users.py` is gated by `_is_senior_admin(actor, target)` - blocks (audits `result="denied"`) when the target is an Admin who registered earlier. Server-side, so it also covers Devii's admin tools.
- **Never pass a `respond()` context key that collides with a Jinja global.** `respond(request, template, ctx, model=XOut)` feeds the same `ctx` to both the Pydantic model (JSON) and the template. A key like `is_admin`/`avatar_url`/`is_self` holding a non-callable value shadows the global across the whole inheritance chain, turning `{% if is_admin(user) %}` into `False(user)` -> `TypeError`, a 500 that fires only for the branch that calls the global. Name viewer/permission flags distinctly (`viewer_is_admin`, not `is_admin`) in both schema and context.
- **Project visibility (`is_private`) and read-only (`read_only`)** are owner-controlled flags on `projects`. Read access is gated by the single `content.can_view_project(project, user)` predicate at EVERY read surface - never re-implement the check inline. Predicate: `not is_private OR is_owner OR (is_admin AND owner is not an admin)` - a project hidden by a member stays visible to any admin, but one hidden by an admin is visible only to that owner admin. **Containers have their own, stricter isolation predicates** (`owns_instance`, `can_view_project_containers`, `can_view_instance`, `can_manage_instance`) - the primary administrator sees/manages every container; any other admin can VIEW others' containers only on public projects and can MANAGE only instances they own. Read-only is enforced as a single data-layer guard `project_files._guard_writable(project_uid)` at the top of every mutation entrypoint - add it to any NEW file-mutating function. Devii may flip read-only/visibility only after explicit confirmation (`CONFIRM_REQUIRED`). Full UI-level detail in `devplacepy/routers/projects/CLAUDE.md`.
- **Deletions ALWAYS require confirmation:** `confirmation_error` gates EVERY content delete tool via `CONFIRM_REQUIRED` (`delete_post`, `delete_comment`, `delete_gist`, `delete_project`, `project_delete_file`, `delete_media`, `delete_attachment`, `admin_delete_news`, container delete, and any `container_exec` matching `dispatcher.DESTRUCTIVE_COMMAND`). The first call is refused; the agent must show the exact target then pass `confirm=true`. **Load-bearing: every confirmation-gated tool MUST also declare a `confirm` boolean param in its catalog spec** - schemas set `additionalProperties: false`, so a gated tool without a declared `confirm` param can never receive it and loops forever.
## Modal pattern
`Application.js` `initModals()` toggles a `.visible` CSS class on `.modal-overlay`; the CSS rule `.modal-overlay.visible { display: flex; }` handles visibility. Triggers usually have `href="#"`, so call `e.preventDefault()`. `.modal-close` is wired generically - no inline JS needed. Full modal/partial/CDN detail in `devplacepy/templates/CLAUDE.md`.
## Polymorphic comments and votes
The `comments` table uses `(target_type, target_uid)` so `_comment_section.html` works for `post`, `project`, `gist`, and `news`. Votes follow the same shape via `/votes/{target_type}/{uid}`. `resolve_target_redirect()` in `comments.py` maps target_type back to the correct detail URL. Full detail (reactions/bookmarks/polls/heatmap/follow/block-mute reuse the same target-type pattern) in `devplacepy/routers/CLAUDE.md`.
## Project-wide soft delete (hard rule)
**Every removal is a soft delete; only garbage collection is a hard delete.** Removable rows carry `deleted_at` (ISO timestamp) + `deleted_by` (actor uid or `system`); a live row has `deleted_at = NULL` and every list/count read filters `deleted_at IS NULL`. The table set is `database.SOFT_DELETE_TABLES`; `init_db` ensures both columns + a partial index per table. Core primitives in `database/`: `soft_delete`, `soft_delete_in`, `restore`, `purge`, `list_deleted`/`count_deleted`, `restore_event`/`purge_event`.
- **Every INSERT into a soft-deletable table MUST write `deleted_at: None, deleted_by: None`** - `dataset.find(deleted_at=None)` on a table missing the column matches NOTHING, silently hiding all rows on a fresh DB.
- **Any new read of a soft-deletable table MUST filter `deleted_at IS NULL`.**
- **Toggles revive, never duplicate:** look up the physical row ignoring `deleted_at`, stamp on toggle-off, clear on re-toggle.
- **Cascades share one `stamp`** so the event restores/purges atomically.
- **Delete authz is owner-OR-admin on the endpoint** - one check covers the UI and Devii.
- **GC stays HARD** (job sweep, metrics ring, usage-ledger prune/reset, expired-session cleanup, fork rollback). Logout is soft (auditable); only expiry GC is hard.
Admin **Trash** at `/admin/trash` restores/purges by event. Full table list, dataset rules, indexing conventions (the soft-delete planner trap), and site settings are in `devplacepy/database/CLAUDE.md`.
## Testing
Playwright (NOT pytest-playwright). Around 1959 tests in `tests/`, split into three category directories by *what they exercise*: `tests/api/` (HTTP integration, no browser), `tests/e2e/` (Playwright browser), `tests/unit/` (pure in-process). **The directory tree mirrors the path** - one segment per directory, last segment is the file. `api`/`e2e` mirror the URL path (`GET /admin/ai-usage` -> `tests/e2e/admin/aiusage.py`); `unit` mirrors the source module path (`devplacepy/utils.py` -> `tests/unit/utils.py`). A test's tier is decided by its fixtures: `page`/`alice`/`bob` = e2e; `app_server`/`seeded_db` or any HTTP call = api; `local_db`-only or no fixture = unit. Run a tier with `make test-unit`/`make test-api`/`make test-e2e`; `make test` runs all three.
Fixture stack: `app_server` (session-scoped uvicorn subprocess on port 10501), `browser_context` (session-scoped Playwright context), `page` (function-scoped, fresh cookies), `alice`/`bob` (seeded logged-in users, `bob` gets its own context for multi-user tests).
Required patterns: every `page.goto(...)`/`page.wait_for_url(...)` MUST pass `wait_until="domcontentloaded"`; prefer `page.locator(...).wait_for(state="visible")` over `wait_for_selector`; scope ambiguous selectors (e.g. comment Delete is `.comment-action-btn:has-text('Delete')`). A test that flips a global `site_settings` value MUST restore it in `try/finally`. Full pitfalls/patterns catalogue is in `tests/CLAUDE.md`.
**Never run tests unless the user explicitly asks for it.** Not the full suite, not a single file. Validate with a clean import (`python -c "from devplacepy.main import app"`) and per-language manual checks instead.
## Feature workflow
Every feature in DevPlace is **one data source fanning out into several consumers**. DevPlace has no isolated change - internalize this before editing. A single handler in `routers/{area}.py` is simultaneously the **four faces of one route**:
1. **HTML** - `respond()` returns a rendered template for browsers.
2. **JSON** - the SAME `respond(..., model=XOut)` returns JSON when `Accept: application/json`. The `*Out` schema is the gate: a context key not declared on `*Out` is silently dropped from JSON even though the template still sees it.
3. **Agent tool** - `services/devii/actions/catalog.py` exposes the route as an `Action` so Devii can call it on the user's behalf. `requires_auth` mirrors the route guard.
4. **Documented endpoint** - `docs_api.py` `endpoint()` describes it (params, auth, `sample_response`) in the right group, rendered at `/docs/{group}.html`.
A new public read almost always needs all four. The cardinal failure mode is changing one face and forgetting a connected one - a real worked example (followers/following listing) touched nine files across data helpers, output schema, route, view, agent tool, API docs, and the docs trio below for one conceptual feature; that count is the norm, not the exception. If a change touches only one file, confirm none of the four faces were missed. Checklist, ordered by data flow:
1. **Understand.** Read the router, template, matching tests (per the tier naming rule above), and the matching nested `CLAUDE.md`. Trace input model -> router -> data helper -> response (HTML and JSON).
2. **Data layer.** `database/` for query/batch helpers (never inline N+1 loops - use `get_users_by_uids`, `build_pagination`, `_in_clause`; guard raw SQL with `if "table" in db.tables`). `models.py` for the Pydantic `Form` input model. `schemas/` for the `*Out` JSON response model - every context key a JSON route exposes via `respond(..., model=XOut)` MUST exist on `XOut` or it is silently dropped. Schema auto-syncs via `dataset`; add indexes in `init_db()` with `CREATE INDEX IF NOT EXISTS`.
3. **Server layer.** Handler with the right auth guard (`get_current_user` public read, `require_user` member POST, `require_admin` admin). Specific paths (`/{username}/followers`) declared before catch-alls (`/{username}`). Return HTML+JSON via `respond(request, template, ctx, model=XOut)` or pure JSON via `JSONResponse`. Register any NEW router in `main.py`.
4. **View layer.** Extend `base.html`; page CSS in `{% block extra_head %}`, page JS in `{% block extra_js %}`. Reuse partials and template globals. JS is ES6 modules, one class per file, instantiated on `app`.
5. **Agent + docs layer (the most-forgotten step - do not skip).** `services/devii/actions/catalog.py` - add an `Action` tool if a user could ask Devii to do this (`requires_auth=False` for public reads). `docs_api.py` - every public/auth endpoint gets an `endpoint()` entry with params and a `sample_response`. `routers/docs/pages.py` `DOCS_PAGES` for prose pages. SEO (`seo.py`) - `base_seo_context` and any JSON-LD for a new public page; add to `routers/seo.py` sitemap if indexable.
6. **Document.** `README.md` (product-facing, routes/config/dependencies/user-visible features), the relevant nested `CLAUDE.md` (any new mechanic, helper, pitfall, or cross-layer wiring), this root `CLAUDE.md` only when a NEW architectural rule or workflow step is introduced (not per feature).
7. **Validate.** Confirm a clean import (`python -c "from devplacepy.main import app"`), check each touched language manually, grep for em-dash in touched files (neither the character nor its HTML entity - use a hyphen). **Never run the test suite unless the user explicitly asks.** Write new tests in the matching tier/path when asked, following the required patterns above.
Failures at any implementation step block the workflow - never skip a failed step.
## CI/CD
Gitea Actions workflow at `.gitea/workflows/test.yaml` runs on every push/PR to `master`: installs dependencies + Playwright Chromium, runs the full suite serially under coverage, publishes coverage HTML as an artifact, uploads failure screenshots. CI must be green before merging. Changes move through DTAP: Development (`make dev`) -> Test (CI suite + coverage on `master`) -> Acceptance (`master` to `production` promotion via `make deploy`) -> Production (Docker Compose stack). Only CI-green `master` commits are promoted to `production`.
## Production deployment
Single-host Docker Compose (`docker-compose.yml`): an **app** container (Uvicorn, 2 workers) behind an **nginx** container. Full operator reference is the admin-only `Production` docs section (`templates/docs/production*.html`); conventions that must not regress:
- **Shared DB and files = same as dev.** The app container bind-mounts the host project root (`.:/app`) and runs as `${DEVPLACE_UID}:${DEVPLACE_GID}` (default `1000`), so it reads/writes the same `data/devplace.db`, `data/uploads/`, `data/devii_*.db`, `data/keys/` (VAPID), and `data/locks/devplace-services.lock` as `make dev`. No `DEVPLACE_DATABASE_URL` override - `config.py` resolves an absolute path under the project's `data/` dir. WAL + the `flock` on `devplace-services.lock` make concurrent dev/prod safe and keep a single background-services owner. SQLite is local-file, so prod and dev must be the **same host**.
- **Code updates need no rebuild** (bind-mounted source); rebuild only on `pyproject.toml` dependency changes. `.env` is git-ignored; `.env.example` is the committed template. The app reads `DEVPLACE_DATABASE_URL`, never `DATABASE_URL`.
- **nginx parity rules** (`nginx/nginx.conf.template`, rendered by `start.sh` via `envsubst` with an allow-list that preserves `$http_upgrade`): `/static/uploads/` must re-apply `nosniff` + a `Content-Disposition` via the `map $uri $upload_disposition` block (`inline` for safe image/video/audio extensions, `attachment` otherwise), mirroring `UploadStaticFiles.INLINE_MEDIA_EXTENSIONS` - an XSS control nginx would otherwise bypass, and the inline branch is what lets video play in production; `/devii/ws` needs the `map $http_upgrade $connection_upgrade` block and `Upgrade`/`Connection` headers or the Devii terminal cannot connect (every new WebSocket route needs its own nginx upgrade location - the catch-all `location /` strips upgrade headers); `client_max_body_size` comes from `NGINX_MAX_BODY_SIZE` (default `50m`) and must be `>= max_upload_size_mb` or uploads 413. nginx serves `devplacepy/static` via a read-only bind mount, so assets stay current without an image rebuild.
Reference in New Issue View Git Blame Copy Permalink