All source listed below is under MIT license if no LICENSE file stating different is available.

DevPlace - The Developer Social Network

Server-rendered social network for developers. FastAPI backend serving Jinja2 templates with ES6 interactivity. Avatar generation via Multiavatar (local, no network). SQLite via dataset with WAL mode and concurrency tuning.

CI

Quick start

make install        # pip install -e .
make dev            # uvicorn --reload on port 10500
make test           # Playwright integration + unit tests, headless, fail-fast
make test-headed    # same tests in visible browser

Open http://localhost:10500.

Stack

Layer Technology
Backend Python 3.13+, FastAPI, Uvicorn (multi-worker in production)
Templates Jinja2 (server-side rendered)
Frontend Pure ES6 JavaScript, one class per file
Database SQLite via dataset (auto-sync schema, uid PKs, WAL mode, 30s busy timeout)
Auth Session cookie, API key (X-API-KEY/Bearer), or HTTP Basic; PBKDF2-SHA256 via passlib
Avatars Multiavatar (local SVG generation, no external API, <5ms)
Outbound HTTP Stealth client (devplacepy/stealth.py): real Chrome fingerprint (TLS JA3/JA4 + HTTP/2 + headers) via curl_cffi behind an httpx transport adapter, pure-httpx[http2] fallback; the single client for every server-side outbound request
Coverage coverage.py (.coveragerc, subprocess-aware)
Load testing Locust (locustfile.py)

Project structure

devplacepy/
  main.py             # FastAPI app, router registration
  config.py           # Settings from env vars + .env
  database.py         # dataset connection, index creation
  templating.py       # Shared Jinja2 environment + globals
  avatar.py           # Multiavatar generation, URL builder
  utils.py            # Password hashing, session mgmt, time_ago, notification hook
  models.py           # Pydantic schemas
  push.py             # Web push crypto, VAPID keys, encrypt/send/register
  routers/            # One file per domain (auth, feed, posts, push, ...)
  templates/          # Jinja2 HTML templates
  static/css/         # Page-specific CSS files
  static/js/          # Application.js (ES6 module)
  services/           # Background service framework (base, manager, news)

Routes

Prefix Purpose
/ Home page: marketing splash for guests, personalized home (welcome, feed shortcut, latest posts, news) for signed-in users. Does not redirect. Latest-posts section interleaves authors so no two consecutive posts share an author.
/auth Signup, login, logout, forgot/reset password
/feed Post feed with topic/tab filtering and free-text search (title and content) in the left panel (public). Each page interleaves authors so no two consecutive posts share an author.
/news Developer news listing, detail page with comments
/posts Post detail, creation
/gists Code gist listing, detail, creation, and editing; left panel offers language filtering and free-text search (title and description), public read
/comments Comment creation, owner editing (POST /comments/edit/{comment_uid}), deletion
/projects Project listing (left panel offers type filtering and free-text search over title and description), creation, owner editing (POST /projects/edit/{slug}), and per-project visibility toggles: POST /projects/{slug}/private (owner-only visibility) and POST /projects/{slug}/readonly (immutable files)
/projects/{slug}/files Per-project filesystem: directory and file CRUD, upload, inline editing, and line-range operations (lines read, replace-lines, insert-lines, delete-lines, append) for surgical edits to large text files (public read, owner write; all writes refused while the project is read-only)
/zips Zip job status (/zips/{uid}) and archive download (/zips/{uid}/download); archives are queued via /projects/{slug}/zip and /projects/{slug}/files/zip
/forks Fork job status (/forks/{uid}); forks are queued via /projects/{slug}/fork. Any signed-in user can fork a project they can view into a new project they own; once the job finishes the response carries the new project URL
/tools Public developer tools. /tools/seo is SEO Diagnostics: audit any URL or sitemap and stream live progress over a websocket. Queue with POST /tools/seo/run, poll GET /tools/seo/{uid}, read the full report at GET /tools/seo/{uid}/report. /tools/deepsearch is DeepSearch: a multi-agent deep web researcher that crawls and indexes sources, synthesises a cited report with confidence and gap analysis, and lets you chat over the gathered evidence. Queue with POST /tools/deepsearch/run, poll GET /tools/deepsearch/{uid}, read the report at GET /tools/deepsearch/{uid}/session, export at /export.{md,json,pdf}
/projects/{slug}/containers Admin per-project container manager: Dockerfile CRUD with immutable versions, async image builds, and container instance creation. Reachable from the project page via the admin-only Containers button
/admin/containers Admin Containers manager: list, create, edit, and control every container instance across all projects. The list (/admin/containers) has inline start/stop/restart/terminal/edit/delete on each row and a create form (pick a project, optionally a run-as user, a boot language with a source editor, restart policy, start-on-boot, plus env/ports/limits/ingress). Each instance has a detail page (/admin/containers/{uid}) with lifecycle controls, live logs and metrics, an interactive terminal, schedules, ingress, workspace sync, and a status history, and an edit page (/admin/containers/{uid}/edit)
/p/{slug} Public ingress proxy (HTTP + WebSocket) to a running container instance's published port, opt-in per instance via ingress_slug
/profile Profile view, editing, and a public Media tab (?tab=media) showing every attachment a user uploaded, newest first
/media Per-attachment soft delete and restore: POST /media/{uid}/delete (owner or admin), POST /media/{uid}/restore (admin)
/uploads File upload endpoints: POST /uploads/upload (multipart), POST /uploads/upload-url (from URL); served at /static/uploads/
/admin/trash Admin Trash: review, restore, and permanently purge soft-deleted content (posts, comments, gists, projects, news, project files, attachments) across the platform
/notifications Notification list, mark read, live unread counts (/notifications/counts)
/messages Real-time direct messaging over WebSocket (/messages/ws): live bidirectional delivery, optimistic send, typing indicators, read receipts, and online/last-seen presence. Messages render through the shared content pipeline (emoji shortcodes, image and YouTube embeds, autolink, sanitization). AI content correction and the AI modifier apply to direct messages, so typing an inline @ai <instruction> in a message executes it and the resolved result appears live in the chat for both participants. The POST /messages/send form remains as a no-JavaScript fallback
/votes Upvote/downvote on posts, comments, projects
/reactions Emoji reactions on posts, comments, gists, projects
/bookmarks Save/unsave content; /bookmarks/saved personal list
/polls Vote on post-attached polls
/follow Follow/unfollow users
/leaderboard Contributor ranking by total stars earned
/avatar Multiavatar proxy with in-memory cache
/issues Issue tracker backed by Gitea: list/detail/comment/status, AI-enhanced filing, admin planning report of all open tickets
/admin/services Background service management (start/stop, config, status, logs)
/admin/bots Admin Bot Monitor: a live grid of the latest low-quality screenshot per running bot persona, each labelled with the bot username, persona, and current action, auto-refreshing
/admin Admin panel (user management, news curation, settings)
/docs Developer documentation site with a complete, interactive HTTP API reference
/openai OpenAI-compatible LLM gateway service (/openai/v1/chat/completions, /openai/v1/*)
/devii Devii agentic assistant: WebSocket terminal (/devii/ws), standalone page, usage (/devii/usage), session bootstrap
(none) /robots.txt, /sitemap.xml (SEO)
(none) /push.json (VAPID key + subscribe), /service-worker.js, /manifest.json (push + PWA)

Gamification

Member progression is driven by activity and peer recognition.

  • Stars are the net vote score (upvotes - downvotes) on a post, project, or gist. A member's total stars is the sum across all their content and is the basis for ranking.

  • XP and levels. Members earn XP for contributing: posting (10), commenting (2), publishing a project (15) or gist (5), receiving an upvote (5), and gaining a follower (5). Each level requires 100 XP (level = 1 + xp // 100). The profile shows the current level and progress to the next.

  • Badges are awarded once and never revoked, across several themed groups (First steps, Explorer, Engagement, Content, Community, Reputation, Dedication, Levels). They cover three kinds of achievement: content and reputation milestones (10/50/100 posts, 25/100/500 stars, 10/50/100 followers, comment and project and gist counts, following 10 people, 7/30/100-day activity streaks, reaching levels 5/10/25/50/100); first-time feature use (your first comment, project, gist, fork, archive download, SEO audit, DeepSearch, container, direct message, bookmark, reaction, star given, follow, upload, project file, issue, poll vote, profile customization, and first conversation with Devii); and usage tiers for several of those features (for example reading 1/5/15 documentation pages, or giving 50/250 stars). Each profile has a collapsible Achievements showcase that lists every badge grouped by theme, with earned ones highlighted and locked ones shown with their unlock condition, so there is always a next prize to chase.

  • Leaderboard (/leaderboard) ranks the top 50 members by total stars (single page, no pagination); a member's own rank is shown on their profile.

  • Contribution heatmap and streaks. Each profile shows a 12-month activity heatmap and the current/longest daily streak, derived from post/comment/gist/project timestamps (no extra storage).

  • Social graph listings. Each profile has Followers and Following tabs that paginate the follow graph (25 per page) and show a follow/unfollow control for each person. The same data is available as JSON at GET /profile/{username}/followers and GET /profile/{username}/following.

  • Media gallery. Each profile has a public Media tab: a responsive, paginated grid of every attachment that user uploaded across posts, projects, gists, comments, messages, issues, and news, newest first. Images open in the shared lightbox. The owner can delete their own media and an admin can delete anyone's; deletion is a soft delete that hides the item everywhere (including its parent object) while preserving the file and the relation, so an admin can restore it from the Media trash at /admin/media.

  • Reward notifications fire when a member levels up or earns a badge.

  • AI content correction. Opt-in, default off. When a member enables it on their profile, the prose they author (post titles and bodies, project and gist titles and descriptions, comments, direct messages, and their bio) is automatically rewritten by the AI gateway according to a member-defined instruction, using the member's own API key for per-user attribution. A member chooses the apply mode: in background (default; content is saved exactly as written and corrected a moment later, so the write path is never slowed) or synchronously (the save waits for the correction so the stored result is corrected immediately). The rewrite is fail-soft (the original is kept on any error) and applies identically across the web UI, the REST and devRant APIs, and Devii. Code and source files are never corrected. In direct messages the correction is delivered live: the corrected message appears in the chat for both participants without a reload. Configure it on your profile or via the Devii ai_correction_set tool; the settings are saved at POST /profile/{username}/ai-correction. Successful correction calls accumulate per-user running totals - corrections, token counts, cost, and timing/performance (average latency, average speed in tokens per second, and total processing time) - shown on the profile page; token, call, and performance figures are visible to the member, while the dollar figures (total and average cost) are shown to administrators only.

  • AI modifier. Enabled by default and applied synchronously by default. It works like AI content correction, except it runs only where the prose you author contains an inline @ai <instruction> directive: the configured prompt tells the model to execute that instruction and replace the marked part, removing the @ai marker. Text with no @ai ... directive is left exactly as written. It is context-aware: the model is given a grounding summary of who is asking (your username, role, level, stars, post count, rank, followers, and bio), the current date, and where the directive sits - the post a comment replies to, the conversation a direct message belongs to, the gist's language and code, and so on - so directives like @ai answer the question above, @ai write my bio from my stats, or @ai reply to this work. It uses your own API key for per-user attribution, is fail-soft (the original is kept on any error), and applies across the web UI, the REST and devRant APIs, and Devii, on the same prose fields as correction (posts, projects, gists, comments, direct messages, and your bio). Code and source files are never touched. In direct messages it runs live: typing @ai <instruction> in a message executes it and the resolved result appears in the chat for both participants without a reload. You can switch the apply mode to background or disable it on your profile or via the Devii ai_modifier_set tool; the settings are saved at POST /profile/{username}/ai-modifier. The default instruction is "Execute what is behind @ai (the prompt) and replace that part including @ai". Successful modifications accumulate per-user running totals - modifications, token counts, cost, and timing/performance (average latency, average speed in tokens per second, and total processing time) - shown on the profile page; token, call, and performance figures are visible to the member, while the dollar figures (total and average cost) are shown to administrators only.

Every AI gateway response (/openai/v1/*) also returns per-call X-Gateway-* headers with the full token breakdown and the dollar cost of that call, so any client can read its own usage.

Engagement

  • Emoji reactions - a fixed palette of reactions on posts, comments, gists, and projects, separate from voting and carrying no ranking weight.
  • Polls - a post can carry a poll (question plus up to six options); results appear as live bars once the viewer votes, one vote per member. A poll can be attached when the post is created or added later by editing a post that has none.
  • Bookmarks - save posts, gists, projects, and news to a personal list at /bookmarks/saved.
  • Private projects - an owner can mark a project private so it is visible only to them (and administrators) and excluded from listings, profiles, search, the sitemap, and zip access. Set at creation or toggled later from the project page.
  • Read-only projects - an owner can mark a project read-only, making its entire virtual filesystem immutable: every write, edit, line-edit, move, delete, and upload is refused from all paths (the web UI, the HTTP API, the Devii agent, and container workspace sync) until read-only is turned off. Devii may toggle read-only only after the user explicitly confirms.

XP awards are wired at the existing content-creation, vote, and follow hook points in the routers and centralized in award_xp() / check_milestone_badges() (devplacepy/utils.py). Existing accounts have their XP and levels backfilled once from prior activity at startup (init_db()).

Admin: Audit Log

Every state-changing action on the platform is recorded to an append-only audit log: account and authentication events, content and engagement, projects and files, news curation, admin governance, background services, container operations, ingress traffic, AI gateway calls, the Devii agent, the CLI, gamification side-effects, and security denials. Each row captures who acted, in which role, from where (origin, IP, user-agent), on which objects, what changed (old to new value), a sanitised summary, event-specific metadata, and whether the action succeeded, failed, or was denied. A companion table links every related object with a labelled relation. The complete event catalogue lives in events.md.

The log is administrator-only. /admin/audit-log is a paginated, filterable list (by search text, category, event key, role, origin, result, and date range) styled like the rest of the admin panel; /admin/audit-log/{uid} is the per-event detail with the full row, pretty-printed metadata, and the related-object graph. Both negotiate HTML or JSON. Financial figures are recorded for completeness but only surfaced on this admin-only view. Administrators can also query the trail conversationally through Devii, which exposes the same filterable list and per-event detail as admin-only read tools over these endpoints. Recording is best-effort and never blocks the audited action. When the Devii agent performs a mutation it reuses the same event key with origin=devii/via_agent=1. Rows older than audit_log_retention_days (default 90) are pruned daily by the Audit retention background service.

Configuration

Env var Default Purpose
DEVPLACE_DATABASE_URL sqlite:///<repo>/data/devplace.db Database connection string
DEVPLACE_DATA_DIR <repo>/data Single root for every runtime/user-generated artifact (DB, uploads, VAPID keys, locks, bot state, job staging, container workspaces), outside the package and not served via /static. Point at a volume in production. Defined once in config.py (DATA_PATHS registry, created by ensure_data_dirs())
SECRET_KEY hardcoded fallback Session signing key
DEVPLACE_VAPID_SUB mailto:retoor@molodetz.nl Contact address in the VAPID JWT sub claim
DEVPLACE_INTERNAL_BASE_URL http://localhost:10500 Base URL the platform's own services dial for the AI gateway
DEVPLACE_XMLRPC_PORT 10550 Loopback port the forking XML-RPC bridge binds; the app and nginx reverse-proxy /xmlrpc to it
DEVPLACE_XMLRPC_BIND 127.0.0.1 Bind address for the XML-RPC bridge (loopback; the app and nginx are the intended front doors)
DEVPLACE_STATIC_VERSION server boot unix timestamp Cache-busting version stamped into every static asset URL (/static/v<version>/...). Set it at launch so multiple workers agree (the prod target and Docker image do this); leave unset in dev to refresh on each reload. See Static asset caching
DEEPSEEK_API_KEY / OPENROUTER_API_KEY unset Upstream provider keys; migrated into the gateway settings on first boot

Runtime settings

Operational behavior is tunable live from /admin/settings (stored in site_settings, no redeploy). The Operational group covers:

Setting Default Effect
site_url empty Public origin for absolute links (SEO, container ingress /p/<slug>); empty derives from the request or DEVPLACE_SITE_URL
rate_limit_per_minute 60 Mutating requests allowed per IP per window
rate_limit_window_seconds 60 Rolling window for the request limit
session_max_age_days 7 Standard session cookie lifetime
session_remember_days 30 Remember-me session cookie lifetime
registration_open 1 When 0, new sign-ups are rejected
maintenance_mode 0 When 1, non-admins see the maintenance page; admins retain access
maintenance_message scheduled-maintenance text Message shown during maintenance
customization_enabled 1 When 0, no user CSS/JS customization is injected on any page
customization_js_enabled 1 When 0, user custom CSS is still served but custom JavaScript is suppressed
audit_log_retention_days 90 Audit rows older than this are pruned daily by the Audit retention service; 0 disables pruning

Numeric values are floored to safe minimums so an invalid entry cannot lock out writes or stall services. Consumers read via get_setting/get_int_setting, which fall back to these defaults when a row is absent.

Authentication

The website uses a session cookie. For automation, every page and action also accepts three header-based methods, resolved centrally in get_current_user (utils.py) so they work everywhere with no per-route changes:

  • API key - X-API-KEY: <key>
  • Bearer - Authorization: Bearer <key>
  • HTTP Basic - Authorization: Basic base64(username-or-email:password)

Every user has an api_key (a uuid7), set at signup and backfilled for existing users on startup (and via devplace apikey backfill). The key is shown on the profile page to its owner (and to admins for any user) with a live Regenerate button; regenerating invalidates the old key immediately. Invalid credentials return 401; requests with no credentials behave as anonymous (browser redirect to login). Full details and copy-paste curl examples live at /docs/authentication.html.

curl -H "X-API-KEY: <your-key>" https://your-host/notifications
curl -u <username>:<password> -X POST https://your-host/follow/<username>

CLI: devplace apikey get <username> / reset <username> / backfill.

Content negotiation (HTML or JSON)

Every endpoint that renders a page or returns a redirect also speaks JSON, so anything the website does is automatable from the same URLs. A request gets JSON when it sends Accept: application/json or Content-Type: application/json; a normal browser navigation (Accept: text/html) always gets HTML, so existing behaviour is unchanged (the legacy X-Requested-With: fetch AJAX header still drives the four engagement endpoints only). JSON responses are defined by Pydantic models in devplacepy/schemas.py and built from the same context the templates use (sensitive user fields like email/api_key/ password_hash are never exposed). Page GETs return the page payload; form actions return a uniform envelope { "ok": true, "redirect": "…", "data": {…} }; errors return { "error": { "status", "message" } } (validation → 422 with a fields map, unauthenticated JSON → 401, non-admin → 403). The core lives in devplacepy/responses.py (wants_json, respond, action_result). Full details: /docs/conventions.html.

curl -H "Accept: application/json" https://your-host/feed
curl -H "Accept: application/json" -X POST -d "content=hi&title=T&topic=devlog" https://your-host/posts/create

XML-RPC bridge

The full REST API is also reachable over XML-RPC at /xmlrpc. A standalone forking XML-RPC server (XmlrpcService, supervised like any other background service, listening on the loopback DEVPLACE_XMLRPC_PORT, default 10550) generates one method per documented endpoint directly from the API reference, so every capability is callable over XML-RPC with no extra wiring. The app reverse-proxies /xmlrpc to it (routers/xmlrpc.py); in production nginx forwards /xmlrpc as well. Method names mirror the endpoint id with dots (posts.create, feed.list, profile.update); each method takes one struct of named parameters and returns the same JSON payload the REST endpoint would. Authenticate with api_key inside the struct, an X-API-KEY / Bearer header, or HTTP Basic via a credentialed URL (http://username:password@host/xmlrpc). Full XML-RPC introspection (system.listMethods, system.methodHelp, system.methodSignature) and batching (system.multicall) are supported; REST errors surface as XML-RPC faults whose faultCode is the HTTP status. Full details and copy-paste Python examples (including a bot that replies to mentions) live at /docs/xmlrpc.html; runnable versions are in examples/xmlrpc/.

import xmlrpc.client

proxy = xmlrpc.client.ServerProxy("https://devplace.net/xmlrpc", allow_none=True)
proxy.system.listMethods()
proxy.posts.create({"content": "hi from xml-rpc", "api_key": "YOUR_API_KEY"})

devRant compatibility API

A second REST surface under /api mirrors the public devRant API shape so legacy devRant clients can run against DevPlace data. Rants map to posts, comments and votes map to the native engagement layer, and devRant integer IDs map directly onto the auto-increment id column every table already carries (posts.id, comments.id, users.id). Authentication follows the devRant model: POST /api/users/auth-token with username/password returns an auth_token struct (token_id, token_key, user_id) backed by the devrant_tokens table; every subsequent request carries that triple as query params or body fields (form or JSON both accepted). Writes go through the same audited helpers as the native UI, so XP, notifications, audit, and soft-delete all apply.

Method Path Purpose
POST /api/users/auth-token Log in (by username or email), returns auth_token
POST /api/users Register a new account
GET /api/get-user-id?username= Resolve a username to its integer id
GET /api/users/{id} User profile (rants + comments)
POST /api/users/me/edit-profile Update bio/location/github/website
DELETE /api/users/me Deactivate account
GET /api/devrant/rants?sort=&limit=&skip= Rant feed (recent/top/algo)
POST /api/devrant/rants Post a rant (rant, comma-separated tags)
GET /api/devrant/rants/{id} One rant with its comments
POST /api/devrant/rants/{id} Edit a rant (owner only)
DELETE /api/devrant/rants/{id} Delete a rant (owner or admin)
POST /api/devrant/rants/{id}/vote Vote (1/-1/0)
POST /api/devrant/rants/{id}/{favorite|unfavorite} Bookmark toggle
POST /api/devrant/rants/{id}/comments Comment on a rant
GET /api/devrant/search?term= Search rants
GET / POST / DELETE /api/comments/{id} Read / edit / delete a comment
POST /api/comments/{id}/vote Vote on a comment
GET / DELETE /api/users/me/notif-feed Notification feed / mark all read
GET /api/avatars/u/{username}.png PNG avatar rendered from the username seed

devRant tags round-trip verbatim via a tags column on posts; profile_skills is derived from the user bio (DevPlace has no separate skills field); avatars are real PNGs rendered from the local multiavatar engine. The surface is toggled by the devrant_api_enabled setting (default on). The endpoints are served at devRant's path shape; pointing a hard-coded client at this server is a DNS/reverse-proxy concern handled at the infrastructure layer.

Documentation site

/docs serves a server-rendered docs site with a feed-style left sidebar (routers/docs/). The sidebar groups its sections under four audience tiers (Start here, Build with the API, Contribute and internals, Operate) so each reader has a clear path. /docs/index.html is a "choose your path" landing, /docs/getting-started.html is the new-contributor on-ramp, and /docs/authentication.html documents the auth methods; the prose pages are curated markdown rendered through the shared content renderer (markdown + highlight.js) with the current host and, when logged in, your own API key and username filled in.

/docs/search.html search method is admin-configurable (docs_search_mode on /admin/settings, default agent). In agent mode it is Docii, a documentation-only assistant: an in-page chat (<dp-docs-chat>) rendered in the docs style where you ask questions in plain language and Docii repeatedly searches the documentation, reading the matching sections and refining its query until it has a grounded answer, with links to the relevant pages. Docii is the Devii engine on a dedicated docs conversation channel, but restricted to a single tool (documentation search) and driven by its own system prompt, so it cannot perform platform actions - it only reads the docs. In bm25 mode it is the classic keyword results list. Either way, a viewer who has exceeded their daily AI limit (or a guest when Devii is disabled) falls back to keyword (BM25) search automatically.

Every other endpoint is documented from a single source of truth, the API_GROUPS registry in devplacepy/docs_api.py. Each group becomes a reference page listing its endpoints with copy-paste cURL, JavaScript, and Python examples and an interactive "try it" panel (static/js/ApiTester.js) that executes the real call from the browser with your API key pre-filled. Each panel has a response-format selector (defaulting to JSON) that drives the Accept header for both the live request and the generated snippets, an always-visible Expected tab showing the modeled response, and a Live response tab for the real result. To document a new endpoint, add an entry to docs_api.py; the page, sidebar link, examples, runner, and expected-response sample are generated automatically. FastAPI's built-in Swagger is moved to /swagger so /docs belongs to this site. The raw schema is at /openapi.json.

Operator pages are admin-only: /docs/admin.html (user/news/settings administration) and /docs/services.html (Background Services) are hidden from the sidebar and return 404 for non-admins. The Background Services page is generated live from the service registry (build_services_group() over service_manager.describe_all()), so every registered service and its full configuration are documented automatically - including future services.

Background Services

devplacepy/services/ provides a generic async service framework. /admin/services is a slim index listing each service with its description and live status; clicking a service opens its detail page where everything for that service lives - start/stop, enable-on-boot, run-now, clear logs, adjustable log size, and live status/metrics - organized into Overview / Configuration / Logs tabs (configuration grouped into sections). State is stored in the database (desired state in site_settings, observed state in the service_state table), so control and status are correct even with multiple workers where only one runs the services.

Service framework

  • ConfigField - declarative parameter spec (type, default, validation, secret) a service uses to declare its editable settings
  • BaseService - abstract class with a reconciling run loop that honors the persisted enabled/command/interval state, plus config_fields, get_config(), describe(), and a log buffer
  • ServiceManager - singleton: register, describe_all, set_enabled, send_command, save_config, supervise, shutdown_all
  • NewsService - fetches news from news.app.molodetz.nl/api, grades with AI, stores articles >= threshold
  • BotsService - Playwright fleet of AI personas that browse and interact with a DevPlace instance, with live cost/usage metrics and a live screenshot monitor at /admin/bots (opt-in; install the bots extra)
  • GatewayService - OpenAI-compatible LLM gateway at /openai/v1/*, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default). Every chat request is made date-aware by injecting the current date in EU DD/MM/YYYY format into the system message when it contains no date already (date only, never time, to keep upstream prompt caching effective), and an admin-configurable system preamble (gateway_system_preamble) can be prepended ahead of the client's system message on every call
  • JobService / ZipService / ForkService - generic async job framework (services/jobs/) for heavy, blocking work run off the request path; ZipService builds project zip archives in a subprocess, ForkService copies a project into a new project owned by the forking user
  • ContainerService - the admin container manager (services/containers/): a reconciling supervisor for container instances, all running one shared prebuilt image

Container manager (admin only)

services/containers/ runs supervised container instances from the web UI, the HTTP API, and Devii. It drives the docker CLI via async subprocesses behind a pluggable Backend interface (a DockerCliBackend plus a FakeBackend for tests). There is no in-app image building. Every instance runs ONE shared prebuilt image, ppy:latest (override DEVPLACE_CONTAINER_IMAGE), built once with make ppy from ppy.Dockerfile: a Python + Playwright base with a broad set of common libraries preinstalled, the pravda (uid 1000) user, the sudo superclone, and pagent at /usr/bin/pagent.py all baked in. Creating an instance is then an instant docker run (it fails fast with a clear error if the ppy image has not been built yet). ContainerService reconciles desired instance state against docker ps each tick (containers are labeled devplace.instance=<uid>, so orphans are reaped and no state is lost), applies restart policies, fires cron/interval/one-time schedules, and samples metrics. The container's /app is bind-mounted to a persistent project workspace (materialized from the project files) and stays in sync automatically: the manager runs a bidirectional, newer-wins sync between the project files and the workspace on every start and roughly once a minute while running, so edits made inside the container and edits made in the project file editor converge without manual intervention (the sync only ever creates or overwrites the older copy of a file, never deletes one; a read-only project is export-only). Projects that need extra packages use runtime pip install (pravda owns the site-packages, no sudo needed) or apt install directly (the pravda user runs apt/dpkg through a fakeroot wrapper, so system packages install without root) or add the library to ppy.Dockerfile and rerun make ppy. Each instance can run a boot script in Python or Bash (written into the workspace and run on launch) or a plain boot command, can be set to start automatically whenever the container service starts, and can be configured to run as a chosen DevPlace user - which only selects whose identity and API key are injected into the container (the container always runs as the unprivileged pravda user). Every status change is recorded and shown as a status history on the instance page. A running instance can be published with an ingress_slug, making its service reachable (HTTP and WebSocket) at /p/<slug> through DevPlace. The manager is reached two ways: the admin Containers sidebar entry (/admin/containers) lists, creates, edits, and controls every instance across all projects and opens a dedicated detail page per instance, and each project page carries an admin-only Containers button to its own instance manager.

Security: this requires mounting the Docker socket, which grants host root. Every run, exec, lifecycle, and schedule operation is administrator-only; --privileged is never used and all docker calls are argument-list subprocesses. The service is disabled by default; an admin enables Containers on /admin/services. CLI: devplace containers list | reconcile | prune | prune-builds | gc-workspaces (prune-builds is a one-time cleanup that removes legacy per-project images and the old dockerfiles/builds tables).

Runtime data (container workspaces and zip archives) lives in DEVPLACE_DATA_DIR (default data/), outside the package and never served via /static. The docker daemon must be able to bind-mount the data dir for /app.

Async job framework and zip downloads

services/jobs/ is the standard way to run blocking work asynchronously and hand the caller a result URL. A shared jobs table is the queue (discriminated by kind); queue.enqueue() inserts a pending row from any worker, the lock-owning worker processes jobs in JobService.run_once (reap, recover orphans, refill up to a concurrency limit, prune expired), and status is polled from the database. Retention is built in: each job service deletes its own expired artifacts via a cleanup hook (default 7 days, admin-configurable). To add a kind, subclass JobService, set kind, and implement process() and cleanup().

ZipService archives a project (or a file/folder subtree) into {crc32}.{slug}.zip using the standalone zip_worker subprocess, with a CRC32 over the output. The frontend app.zipDownloader (any data-zip-download element, plus the files context menu) enqueues, polls /zips/{uid}, and downloads from /zips/{uid}/download. The job uid is the capability token, so anyone with the link can download. CLI: devplace zips prune (delete expired) / devplace zips clear (delete all).

ForkService copies a project into a new project owned by the forking user. The Fork button on the project page (any signed-in user) prompts for a name; the job creates the destination project, duplicates the entire virtual filesystem, and records a directional project_forks relation so each fork shows a "Forked from X" link. The frontend app.projectForker enqueues, polls /forks/{uid}, and redirects to the new project once it is done; on failure the partially created project is rolled back. The forked project is permanent, so retention removes only the job tracking row. CLI: devplace forks prune / devplace forks clear (job rows only).

SeoService powers the public Tools -> SEO Diagnostics auditor. It runs a headless-browser (Playwright) crawl of a single URL or a sitemap (capped pages) in a subprocess and runs a broad battery of checks across eleven categories: crawlability and indexing (status, redirects, HTTPS/HSTS, canonical, robots/meta-robots, sitemap, URL hygiene, mixed content), on-page meta and content (title, description, headings, language, charset, viewport, favicon, content depth), links, structured data and rich results (JSON-LD validity and required properties, microdata/RDFa), social cards (Open Graph, Twitter), Core Web Vitals and performance (LCP, CLS, FCP, TTFB, page weight, requests, DOM size, compression, caching, image optimisation, console errors), mobile and accessibility (responsive layout, tap targets, image alt, form labels), security headers, and AI/LLM-search readiness (server-rendered-vs-JS content parity, llms.txt, semantic HTML). It produces a weighted score and grade with per-category subscores and a recommendation for every finding. Progress streams live over WS /tools/seo/{uid}/ws; the full report is available at /tools/seo/{uid}/report (HTML or JSON). CLI: devplace seo prune / devplace seo clear. Playwright is a core dependency; make install fetches the Chromium browser.

DeepsearchService powers the public Tools -> DeepSearch researcher. Given a single research question it plans a set of diverse web search queries, crawls and reads the most relevant sources in a subprocess (plain HTTP first, headless-browser fallback for JavaScript-heavy pages, PDF documents streamed and text-extracted, every URL SSRF-guarded), de-duplicates content, and indexes everything into a per-session ChromaDB vector collection (embeddings via the AI gateway with a local fallback). A chain of agents (summarizer, critic, linker) then synthesises a cited report with a confidence score, source diversity and explicit gap analysis. Progress streams live over WS /tools/deepsearch/{uid}/ws; the report is at /tools/deepsearch/{uid}/session (HTML or JSON) and can be exported as Markdown, JSON or PDF. A grounded chat over the session's evidence runs at WS /tools/deepsearch/{uid}/chat using hybrid retrieval (vector + keyword/BM25). Runs can be paused, resumed or cancelled. A cross-session URL cache avoids re-fetching pages seen by earlier runs. CLI: devplace deepsearch prune / devplace deepsearch clear. ChromaDB, weasyprint, pypdf and Playwright are core dependencies.

BackupService powers the admin Admin -> Backups dashboard, an enterprise-grade backup system that runs entirely as asynchronous jobs so it never impacts the running server. An administrator can back up one of four targets: the database (a consistent SQLite snapshot of the main database and the Devii task/lesson databases, taken with SQLite's online backup API so it is consistent under WAL), uploads (every attachment and project file), keys and config (VAPID keys), or the full data directory (database snapshot, uploads, and keys in one archive, excluding regenerable staging, locks, caches, and container workspaces). Each backup is compressed to a tar.gz in a stdlib subprocess off the request path and recorded with its size, file count, and a SHA-256 checksum. Archives live under data/backups/ (sharded on the random uuid tail) and are served only through the admin-guarded /admin/backups/{uid}/download route. The dashboard reports detailed storage usage - the size and file count of every major data area, the total data-directory footprint, the total size and count of stored backups, and disk usage (total, used, free, percent), computed in a worker thread and cached briefly so the page never blocks. Backups can be scheduled (CRUD) on an interval or 5-field cron expression with a keep_last rotation count that prunes older backups of the same schedule; the service evaluates schedules only on the lock-owning worker so each fires exactly once. Backup archives are permanent operational artifacts: job retention only removes the tracking row, never the archive, which is deleted only by an administrator, by schedule rotation, or via the CLI. CLI: devplace backups list / devplace backups run <target> / devplace backups prune / devplace backups clear. Devii tools: backups_overview, backup_run, backup_status, backup_delete, backup_schedule_create, backup_schedule_delete (all admin-only). The service creates and stores backups but does not restore them into a live server; restore is a documented manual procedure (stop the server, unpack the archive over the data directory, verify the checksum, restart).

Adding a service

Create a class extending BaseService, declare its config_fields, override run_once() (read parameters via self.get_config()), and register in main.py:

service_manager.register(YourService())

The service appears at /admin/services with controls, a generated config form, live status, and the log tail - no template, JS, or router changes.

News service

Fetches news from a configurable API, grades each article via AI, stores ALL articles in the news table (nothing is silently skipped). Articles with grade >= the configurable threshold are auto-published; the rest go to draft. Admin can manually publish/draft, toggle for landing page appearance, and delete. Images are extracted from article URLs.

Configuration on the Services tab (/admin/services):

Parameter Default Purpose
news_grade_threshold 7 Minimum AI grade for auto-publish
news_api_url https://news.app.molodetz.nl/api News source
news_ai_url http://localhost:10500/openai/v1/chat/completions AI grading endpoint (the internal gateway)
news_ai_model molodetz Generic model name; the gateway maps it to the real model
news_ai_key internal key AI API key (NEWS_AI_KEY env, then the auto-generated gateway internal key)
news_service_interval 3600 Seconds between fetch cycles (min 60)

News articles have detail pages at /news/{slug} with full comment support (same component as posts/projects). The landing page can display curated articles toggled from admin.

CLI: devplace news clear - delete all news from local database.

Bots service

A Playwright-driven fleet of AI personas (devplacepy/services/bot/) that browse and interact with a DevPlace instance: posting, commenting, voting, reacting, creating gists/projects, filing issues, following, and messaging. It is the former standalone dpbot.py, refactored into a package and managed entirely from the Services tab. Disabled by default.

Each persona has its own voice and its own interests: post titles are written in the persona's voice rather than copied from the source headline, news topics and post categories are weighted by personality (so different personas react to different stories), shared code snippets pass a non-triviality quality gate, and bots discuss each other's posts in threaded conversations rather than reacting in isolation.

Optionally (bot_ai_decisions), each bot is also given a unique AI-generated identity card (name, backstory, interests, temperament, verbosity, contrarianness, generosity, rhythm) and lets that identity decide every on-page action through the LLM instead of fixed probabilities: on each page the bot is shown the menu of actions actually available and the model returns an ordered plan. Behaviour then follows personality rather than chance, while every content-quality gate is retained. See aibots.md for the design and cost model.

Install the extra and the browser binary, then enable it on /admin/services:

pip install -e ".[bots]"
playwright install chromium

Configuration on the Services tab:

Parameter Default Purpose
bot_fleet_size 1 Number of concurrent bot accounts
bot_headless enabled Run Chromium headless
bot_max_actions 0 Stop a bot after N actions (0 = forever)
bot_base_url https://pravda.education Target DevPlace instance
bot_api_url http://localhost:10500/openai/v1/chat/completions LLM endpoint (the internal gateway)
bot_news_api https://news.app.molodetz.nl/api Article source
bot_model molodetz Generic model name; the gateway maps it to the real model
bot_api_key internal key LLM key (defaults to the auto-generated gateway internal key)
bot_input_cost_per_1m / bot_output_cost_per_1m 0.27 / 1.10 Token pricing for live cost tracking
bot_max_per_article 2 How many bots may post about one article, each from a different angle
bot_article_ttl_days 7 How long an article stays covered before it can be posted again
bot_gist_min_lines 6 Reject generated snippets shorter than this many non-empty lines
bot_action_pause_min_seconds / bot_action_pause_max_seconds 5 / 30 Idle pause window a bot takes after each action
bot_break_scale 1.0 Multiplier on between-session breaks (below 1 = more active and costlier)
bot_ai_decisions disabled Let each bot's AI-generated identity decide every action via the LLM instead of fixed probabilities (one decision call per page); see aibots.md
bot_decision_temperature 0.4 Sampling temperature for the per-page decision call

The Services tab shows live fleet metrics (bots running, total cost, LLM calls, tokens, posts/comments/votes) and a per-bot breakdown that update on the page poll. Each bot keeps a stable identity and cumulative cost in ~/.devplace_bots/state_slot{N}.json.

LLM gateway service

GatewayService (devplacepy/services/openai_gateway/) exposes an OpenAI-compatible endpoint at /openai/v1/chat/completions (plus a transparent /openai/v1/* passthrough). It forwards to the configured upstream (DeepSeek by default) and, when a request carries image content, first describes the image(s) via a vision model (OpenRouter/Gemma by default) and inlines the description as text - so a vision-less upstream can still answer. It also serves OpenAI-compatible text embeddings at /openai/v1/embeddings: clients send the generic model molodetz~embed, which the gateway maps to the configured embedding model (OpenRouter's Qwen3 8B embedding model by default, $0.01 per 1M input tokens), with usage and cost tracked per call like chat and vision. Enabled by default and configurable on /admin/services.

It is the single point of truth for AI: the news, bots, and guest Devii sessions call this gateway by default instead of an external provider, send the generic model name molodetz, and authenticate with an auto-generated internal key. The real provider URLs, models, and keys live only here, so providers and backends are switched in one place. DEEPSEEK_API_KEY and OPENROUTER_API_KEY are migrated into the editable settings on first boot, and the value in use is shown. The internal base URL the other services dial is DEVPLACE_INTERNAL_BASE_URL (default http://localhost:10500).

Authenticated users may call the gateway with their own API key (the Allow users setting, on by default), and Devii operating a signed-in user authenticates its LLM calls with that user's key. Every call is therefore attributed to the user, so their full AI spend (through Devii or direct API calls) is tracked and limitable per user. Administrators see a per-user "AI usage (last 24h)" panel on each profile page - request volume, success and error rates, token totals, cost, average latency and throughput, a per-model breakdown, a 30-day cost projection, and a quota bar - served from GET /admin/users/{uid}/ai-usage. A non-admin user sees only one figure on their own profile: the percentage of their daily AI quota used so far. The same rule governs the Devii assistant: monetary figures (cost, pricing, spend, limit) are disclosed only to administrators, while members and guests can see only the percentage of their 24-hour quota used. The /devii/usage endpoint returns that percentage and the day's turn count to everyone, and includes dollar figures only for administrators.

Configuration on the Services tab:

Parameter Default Purpose
gateway_upstream_url https://api.deepseek.com/chat/completions Where requests are forwarded
gateway_model deepseek-v4-flash Real model sent upstream (what molodetz maps to); 1M-token context, 384K max output
gateway_force_model on Override the client-requested model (and the molodetz alias)
gateway_api_key migrated from env Upstream key, shown and editable (auto-migrated from DEEPSEEK_API_KEY/OPENROUTER_API_KEY)
gateway_instances 4 Max concurrent upstream forwards per worker (pool + semaphore)
gateway_timeout 300 Upstream timeout (seconds); minimum five minutes; also bounds the vision describe-image call
gateway_vision_enabled / _url / _model / _key / _cache_size on / OpenRouter / Gemma / env / 256 Image-description augmentation
gateway_embed_enabled / _url / _model / _key on / OpenRouter / Qwen3 8B / vision-key fallback Embeddings at /openai/v1/embeddings (client model molodetz~embed)
gateway_require_auth on When off, the gateway is open
gateway_allow_admins / gateway_allow_users on / off Which DevPlace users may call it (any auth scheme)
gateway_access_key empty A standalone key (sent as X-API-KEY/Bearer) that always grants access
gateway_internal_key auto (uuid4) Auto-generated on boot; DevPlace's own services authenticate with this. Clear and restart to rotate
gateway_price_cache_hit_per_m / _cache_miss_per_m / _output_per_m 0.0028 / 0.14 / 0.28 Chat cost per 1M tokens, used when the upstream returns no native cost (DeepSeek)
gateway_vision_price_input_per_m / _output_per_m 0 / 0 Vision cost per 1M tokens, used only when the vision upstream returns no native cost
gateway_embed_price_input_per_m 0.01 Embeddings cost per 1M input tokens, used only when the embeddings upstream returns no native cost
gateway_max_retries / gateway_retry_backoff_ms 2 / 250 Retry attempts and linear backoff on timeout, connection error, or upstream 5xx
gateway_circuit_threshold / gateway_circuit_cooldown_seconds 5 / 30 Consecutive failures before the circuit breaker opens, and its cooldown
gateway_usage_retention_hours 720 How long per-call usage rows are kept before pruning (30 days)
gateway_model_context_map JSON map Model to max-context-tokens map for context-window utilization tracking

Callers authenticate with the static gateway_access_key, the auto-generated gateway_internal_key (used by the platform's own services), or - when allowed - their DevPlace credentials via any scheme (X-API-KEY, Bearer, Basic). The endpoint is exempt from the per-IP rate limit and the maintenance gate, and its concurrency is bounded by gateway_instances; scale further with uvicorn workers. The Services tab shows live request/error/latency/vision-call counters.

Usage and cost metrics

Every upstream call (chat, vision, and passthrough) is recorded to gateway_usage_ledger with its tokens, cost, latency breakdown, status, and caller. Cost is taken from the upstream native cost field when present (OpenRouter) and computed from the configured per-million pricing otherwise (DeepSeek, which reports no cost). The admin AI usage page (/admin/ai-usage) reports per-hour and 24h metrics - request volume and throughput, token usage with averages and percentiles (p50/p90/p95/p99), latency (upstream round-trip, gateway overhead, semaphore queue wait, connection establishment), error rates by category, cost (per model, per caller, input vs output, projected monthly burn, caching savings), caller behavior, and an hourly breakdown - divided and combined. The gateway also retries transient upstream failures and opens a circuit breaker after repeated failures. Time to first token and inter-token latency are not reported because the gateway forwards non-streaming to the upstream.

curl -H "X-API-KEY: <gateway_access_key>" \
  -d '{"messages":[{"role":"user","content":"hi"}]}' \
  https://your-host/openai/v1/chat/completions

Devii assistant service

DeviiService (devplacepy/services/devii/) is an in-platform agentic assistant exposed as a WebSocket terminal at /devii/ws, reachable from the Devii item in the user dropdown menu and embedded on the documentation page for guests. It runs a ReAct loop (planning, reflection, a verification gate, self-learning memory, autonomous tasks) over a declarative tool catalog. A signed-in user's Devii operates their own DevPlace account - the agent authenticates to this instance with the user's api_key - while guests get a sandbox session. Disabled by default; enable and configure it on /admin/services.

Sessions are durable. A logged-in user's conversation is keyed by user uid, shared live across every tab, window, and device (each turn is broadcast to all connected sockets), and persisted to devii_conversations so it survives a server restart - on reconnect the prior conversation is rehydrated. Guests get a non-persistent session keyed by a devii_guest cookie. To keep one logical session and one cost/broadcast source per user, the /devii/ws endpoint is served only by the worker that holds the background-service lock; other workers close the socket so the client reconnects to the owning worker.

Spend is capped per owner over a rolling 24 hours. Every turn appends a row to devii_usage_ledger (the authoritative source for the cap - the in-memory cost tracker is display-only) and an audit row to devii_turns. The cap is checked before each turn.

Configuration on the Services tab:

Parameter Default Purpose
devii_ai_url https://openai.app.molodetz.nl/v1/chat/completions OpenAI-compatible reasoning endpoint
devii_ai_model molodetz Model name
devii_ai_key env fallback (DEVII_AI_KEY) AI API key
devii_base_url this instance's origin Platform Devii drives via each user's API key
devii_plan_required / devii_verify_required on / on Enforce plan-first and verify-after-mutation
devii_max_iterations 40 Tool-loop iterations per turn
devii_timeout 300 Read timeout (seconds) for each AI chat-completions call; minimum five minutes
devii_fetch_timeout 300 Read timeout (seconds) for the fetch tool; minimum five minutes
devii_user_daily_usd 1.0 Rolling 24h spend cap per signed-in user
devii_guest_daily_usd 0.05 Rolling 24h spend cap per guest
devii_admin_daily_usd 0.0 Rolling 24h spend cap per administrator (0 = unlimited; admins are exempt by default)
devii_guests_enabled on Allow anonymous guests
devii_price_cache_hit / _cache_miss / _output 0.0028 / 0.14 / 0.28 USD per 1M tokens (drives the cap)
devii_rsearch_enabled on Enable the external web search tools (rsearch_*)
devii_rsearch_url https://rsearch.app.molodetz.nl Base URL of the web search service those tools call
devii_rsearch_timeout 300 Read timeout (seconds) for rsearch_* calls; web-grounded answers can take minutes; minimum five minutes

Beyond the platform tools, Devii has external web tools. fetch_url reads a web page; http_request makes an arbitrary HTTP call (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS) to any external REST/JSON API with custom headers and a JSON, form, or raw body, returning the full response (non-2xx is returned rather than raised so API errors are readable). Both carry an SSRF guard (private and loopback addresses are refused) and a size cap. attach_url downloads a public URL on the server and stores it as a real attachment through the same pipeline as a direct upload, returning a uid Devii then passes in attachment_uids when creating a post, project, gist, comment, issue, or message - so a user can ask Devii to attach an image straight from the internet. Devii also has external web search tools - rsearch (web/image search), rsearch_answer (a web-grounded AI answer with sources), rsearch_chat (direct AI chat), and rsearch_describe_image (vision). These reach an external public service rather than this platform, so platform tools are always preferred; Devii uses them only when the user explicitly asks to search the web or an outside source. They are gated by devii_rsearch_enabled and the service URL is configurable via devii_rsearch_url.

A devii console script ships the same agent as an interactive terminal:

pip install -e .
devii --api-key <your DevPlace api_key> --base-url https://your-host
devii -p "List my unread notifications as a bullet list."   # one-shot

Site customization (per-user CSS/JS)

Each user can reshape the site to taste by injecting their own CSS (look) and JavaScript (behaviour), scoped either to a single page type (every post page, every project page) or globally (every page). Customizations apply only to that user's own sessions; a signed-out guest gets customizations scoped to their devii_guest session cookie. This is the user's own code running in their own browser, like a userscript, so it never affects anyone else's view.

It is configured conversationally through Devii. Ask for a change and Devii previews it live in your browser, then before saving it asks whether to apply it to the current page type or the whole site, and only persists once you confirm. Overrides are stored in the user_customizations table (one row per owner, scope, and language; cached with cross-worker invalidation) and injected server-side: custom CSS is the last <style> in <head> so it overrides the design system, and custom JavaScript runs after the application boots with access to the global app. Devii tools: customize_list, customize_get, customize_set_css, customize_set_js, and customize_reset (restores the default look and behaviour). Operators can disable the feature site-wide with the customization_enabled / customization_js_enabled settings.

Your profile page carries two toggle buttons that suppress your customizations without deleting them: one for the site-wide (global) CSS and JS, one for the per-page-type CSS and JS. Disabling keeps the stored code and re-enables instantly, which is the quick way to recover a page broken by your own custom code. You see the toggles only on your own profile; an administrator can also operate them on any user's profile (or call the Devii customize_set_enabled tool) to recover a user who locked themselves out.

User-defined tools (vibe new functionality)

Users can invent new Devii capabilities by describing them in natural language, with no code. Telling Devii "when I say woeii, search the web for a cat picture matching my description and set it as my page background" makes it call tool_create and store a virtual tool. The tool then appears in Devii's own toolset, and on the next turn "woeii orange" calls it: the handler re-runs Devii itself (a self-evaluating sub-agent) on the stored prompt plus the user's input, with full tool access, and returns the result. Each tool takes a single free-form input string; the stored prompt directs what to do with it.

Virtual tools are owner-scoped (persistent in the DB for signed-in users in devii_virtual_tools, session-only for guests) and never run in anyone else's session. The full CRUD is exposed as Devii tools: tool_create, tool_list, tool_get, tool_update (including enable/disable), and tool_delete. A general eval(prompt) tool runs Devii on any prompt and returns the result; it is the same engine that powers virtual tools. Self-evaluation depth is bounded so a tool cannot loop by calling itself.

Push notifications & PWA

Authenticated users can receive native web push notifications, and the site is an installable Progressive Web App. Push uses only standard libraries (cryptography, PyJWT, httpx) against the Web Push Protocol - no third-party push wrapper.

Events

Every event flows through a single funnel - create_notification() in utils.py - which delivers on two independent channels, in-app and web push, each gated by the recipient's preferences (see "Configurable notifications" below). Whenever the in-app channel delivers, the recipient's open browser also raises a live, click-through toast in real time, bridged onto the in-process pub/sub bus by a lock-owner relay:

Event Recipient
Direct message received receiver
Comment on your post post author
Reply to your comment comment author
@mention in any content mentioned user
Upvote on your content content owner
New follower followed user
Badge earned / level-up the user
Issue-tracker update reporter / admins

create_notification schedules delivery as a fire-and-forget async task, so a dead subscription or push-service error never blocks the triggering request. Delivery (push.notify_user) iterates a user's subscriptions, encrypts the payload (legacy aesgcm content encoding), and POSTs to each endpoint; subscriptions that return 404/410 are soft-deleted.

Configurable notifications

Every notification type can be turned on or off per channel, per user. The Notifications tab on a profile page (/profile/{username}?tab=notifications, visible to the profile owner and to admins) shows one row per type with two checkboxes - In-app and Push - saved individually as you toggle them (POST /profile/{username}/notifications). A "Reset to defaults" button clears all of a user's overrides (POST /profile/{username}/notifications/reset).

Defaults are opt-out: a type/channel a user never touched is enabled. Admins set the platform-wide default for each type/channel on /admin/notifications (POST /admin/notifications); a default applies only to users who have not made an explicit choice. Resolution is: user override, else admin default, else on. Preferences are stored in the notification_preferences table (per user_uid + notification_type, soft-deletable) and enforced inside create_notification(): the in-app row is written only when the in-app channel is enabled, and push.notify_user is scheduled only when the push channel is enabled.

VAPID keys

The server identity is three PEM files generated once at startup under data/keys/: notification-private.pem, notification-private.pkcs8.pem, notification-public.pem. They are git-ignored.

These keys are the application's identity to the push services. If they are lost or regenerated, every existing subscription becomes permanently undeliverable. Persist them across deployments and back them up; do not regenerate them.

Opt-in

The browser requires the first permission prompt to originate from a user gesture, so opt-in is exposed as a button in both the top navigation (bell-with-slash icon) and on the /notifications page. After opt-in, the subscription is refreshed silently on every page load. PushManager.js owns registration, subscription, and the opt-in UI.

PWA

manifest.json (192/512 and maskable icons), service-worker.js, and an install button (PwaInstaller.js) make the app installable. The service worker uses a network-first strategy for navigations and falls back to static/offline.html when offline. Installation requires a secure origin (HTTPS, or localhost for development).

File Role
devplacepy/push.py VAPID keys, payload encryption, send, register
devplacepy/routers/push.py /push.json, /service-worker.js, /manifest.json
static/js/PushManager.js Service-worker registration + subscribe + opt-in UI
static/js/PwaInstaller.js beforeinstallprompt capture + install button
static/service-worker.js Receives push, shows notification, offline fallback
static/manifest.json PWA manifest (icons, display, theme)
static/offline.html Offline fallback page

Database

SQLite via dataset with production-oriented pragmas set on every connection:

PRAGMA journal_mode=WAL;      -- concurrent readers + writers
PRAGMA synchronous=NORMAL;     -- safe with WAL, faster than FULL
PRAGMA busy_timeout=30000;     -- wait 30s instead of failing on lock
PRAGMA cache_size=-8000;       -- 8MB page cache
PRAGMA temp_store=MEMORY;      -- temp tables in memory
PRAGMA mmap_size=268435456;    -- 256MB memory map for reads

All indexes are created via CREATE INDEX IF NOT EXISTS wrapped in try/except - safe to run on every startup regardless of table state.

SQLite is synchronous by design and will never be made async. It is more than fast enough for this platform: the database is a local file with WAL, a 30s busy timeout, and a 256MB memory map, so queries are sub-millisecond. The synchronous database layer is a deliberate architectural decision, not a limitation, and is not subject to change.

Soft delete and data retention

Removing a record is a soft delete, not a physical one: it stamps deleted_at (timestamp) and deleted_by (actor) instead of erasing the row, and every list/count read filters deleted_at IS NULL, so the item disappears from the product while remaining recoverable. This covers user content (posts, comments, gists, projects, news, project files, attachments), engagement toggles (votes, reactions, bookmarks, follows, poll votes - which revive the same row when re-toggled), sessions (logout), container instances, and per-owner Devii/customization data. Deletions cascade with a shared timestamp so the whole event restores or purges as a unit. Garbage-collection operations (job retention, metrics ring buffers, usage-ledger pruning, expired-session cleanup) stay hard deletes - that is the stage that frees storage. Administrators review, restore, and permanently purge soft-deleted content from Trash at /admin/trash; the columns are indexed (idx_<table>_deleted).

A member may delete only their own content, but an administrator may delete any member's post, comment, gist, project, project file, or attachment - the owner-or-admin check lives on each delete endpoint, so it applies equally to the web UI and to the Devii assistant (which acts purely through the platform API as the signed-in user). When an admin's Devii is asked to delete something it requires explicit confirmation before each deletion, and the result is the same soft delete, restorable from Trash.

Database API (/dbapi, admin or internal only)

An administrator (session or admin API key) or an internal service (the gateway internal key) can read and update any table through a single, safe API; members and guests get 403.

  • CRUD per table: GET /dbapi/{table} (filtered, searchable, keyset pagination), GET /dbapi/{table}/{key}/{value}, POST /dbapi/{table}, PATCH /dbapi/{table}/{key}/{value}, DELETE /dbapi/{table}/{key}/{value} (soft delete, ?hard=true to purge), and .../restore. Inserts are born-live; unknown columns are rejected; deny-listed tables (sessions, password resets) are never exposed.
  • query() is read-only: POST /dbapi/query runs a single validated SELECT and returns rows. Every query is parsed (sqlglot), classified, and dry-run with EXPLAIN on a read-only connection before execution; non-SELECT statements are refused and a SELECT with no WHERE/JOIN/LIMIT is flagged as suspicious. All writes go through the structured CRUD, never raw SQL.
  • Ask in plain language: POST /dbapi/nl turns a question such as "all users registered longer than three days" into a validated SELECT (auto-adding deleted_at IS NULL for soft-delete tables), using the platform AI gateway and re-prompting until the SQL validates; pass execute=true to also run it.
  • Async: POST /dbapi/query/async runs a heavy query off the request path and streams progress over WS /dbapi/query/{uid}/ws.
  • The Devii assistant exposes the same capability to administrators, with every insert/update/delete requiring explicit confirmation.

Pub/Sub bus (/pubsub)

A database-free publish/subscribe bus for live updates without polling. Clients connect to WS /pubsub/ws to subscribe to topics (with foo.* wildcards) and publish messages; backends and administrators can also publish over POST /pubsub/publish. Users may use their own user.{uid}.* namespace and subscribe to shared public.* topics; administrators and internal services may use any topic. The browser client is available as app.pubsub.subscribe(topic, cb) / app.pubsub.publish(topic, data). The bus is in-memory and best-effort by design.

Two background services bridge persisted state onto the bus so the interface updates without per-client polling: the Notification relay pushes new in-app notifications as live toasts and refreshes each viewer's unread notification and message badges instantly, and the Live view relay pushes the admin live views (container list and instances, bot fleet, background services, AI usage) to whichever administrators are watching, computing a snapshot only for views that currently have subscribers. Both run on the single service-lock owner and degrade to a low-frequency HTTP poll if the bus is unavailable.

Testing

  • 932 tests split into three tiers under tests/: unit/ (pure in-process), api/ (HTTP integration against the live server), and e2e/ (Playwright browser)
  • A directory tree that mirrors the path. api/e2e follow the endpoint path - each route segment is a directory and the last segment is the file, {param} segments dropped (GET /admin/ai-usage -> tests/e2e/admin/aiusage.py, GET /projects/{slug}/files/lines -> tests/api/projects/files/lines.py). unit mirrors the source module path (devplacepy/services/audit/store.py -> tests/unit/services/audit/store.py). Run one tier with make test-unit / make test-api / make test-e2e
  • Playwright (NOT pytest-playwright plugin - conflicts, uninstall it)
  • Runs serially, one test at a time, in a single process (make test); the suite drives one uvicorn subprocess on port 10501 with its own temp database and DEVPLACE_DATA_DIR
  • Serial execution is enforced in pyproject.toml ([tool.pytest.ini_options] addopts = "--tb=line -p no:xdist"), so concurrent runs cannot be turned on by accident
  • Test users alice_test / bob_test seeded via HTTP at session start
  • Tests stop at first failure (-x flag)
  • Failure screenshots auto-save to /tmp/devplace_test_screenshots/
  • Headed mode: make test-headed (a single Chromium window)

Key test patterns

  • Every page.goto() and page.wait_for_url() uses wait_until="domcontentloaded" - avatar images don't block test execution
  • Session-scoped browser context with per-test cookie clearing
  • page.locator(...).wait_for(state="visible") preferred over bare selectors

Avatars

Uses Multiavatar - generates deterministic SVG avatars locally from a seed string (the username). No external API calls, no network dependency. Generation takes <5ms. Results are cached in-memory (cleared on server restart).

Avatar URL format: /avatar/multiavatar/{username}?size={size}

Deployment

Production runs on a single host via Docker Compose: an app container (Uvicorn, 2 workers) behind an nginx container that terminates HTTP, serves static assets, and proxies the rest.

Shared database and uploads

Every runtime artifact lives under one consolidated data/ directory (DEVPLACE_DATA_DIR, default <repo>/data), defined once in config.py via the DATA_PATHS registry and created at startup by ensure_data_dirs(): data/devplace.db (+ -wal/-shm), data/devii_tasks.db, data/devii_lessons.db, data/uploads/ (attachments and project files), data/keys/ (VAPID), data/locks/, data/bot/, and the job staging / container workspace dirs. Nothing runtime is written inside the package any more.

The app container bind-mounts the host project directory (./ to /app) and runs as the host user (DEVPLACE_UID/DEVPLACE_GID, default 1000). Because config.py resolves every path to an absolute location under the project's data/ dir, the container reads and writes the same data/devplace.db as make dev, plus the same data/uploads/, data/devii_*.db, data/keys/, and data/locks/devplace-services.lock. SQLite WAL mode allows the concurrent access, and the file lock (fcntl.flock on data/locks/devplace-services.lock) guarantees only one process - dev or prod - runs the background services (news, bots, Devii hub), so they never double-fire.

This requires prod and dev to be on the same host/filesystem; SQLite is a local-file database and cannot be shared across machines. No DEVPLACE_DATABASE_URL override is set, so nothing diverges.

To migrate an older install whose data was scattered across the repo root, the package, var/, and $HOME, stop the app and run devplace migrate-data --dry-run (review the plan), then devplace migrate-data. It is idempotent and copy-before-delete (verify CRC, atomic replace, then unlink the source), so it is safe to re-run.

First deploy

cp .env.example .env        # set SECRET_KEY; adjust PORT, DEVPLACE_SITE_URL
make docker-build           # build the image (installs the docker CLI)
make docker-up              # start (creates ./data, mounts the socket)

Open http://<host>:${PORT} (default 10500). make docker-logs tails output; make docker-down stops.

make docker-build and make docker-up always apply the docker-compose.containers.yml overlay, so the admin-only Container Manager works with no extra setup. Mounting the docker socket grants the app root on the host; the feature itself stays disabled until an admin enables Containers on /admin/services (and the ppy image is built once with make ppy), but treat the whole deployment as trusted-admins-only. Always update through the make targets - a plain docker compose up -d drops the overlay and silently removes the socket and CLI.

Updating

git pull
make docker-up              # restart with new code (bind-mounted, no rebuild)
make docker-build && \
  make docker-up            # only when dependencies in pyproject.toml change

The make deploy target fast-forwards the production branch (git checkout production && git merge master && git push origin production); pull that branch on the server.

Container Manager wiring (what the overlay does)

The Container Manager drives the host Docker daemon, so make docker-build/make docker-up apply docker-compose.containers.yml automatically. The make targets derive everything the overlay needs - DOCKER_GID from the socket and DEVPLACE_DATA_DIR as the project's own ./data at its real host path - so no sudo, no /srv dir, and no manual .env editing are required. (Override DEVPLACE_DATA_DIR or DOCKER_GID on the make command line to point elsewhere.)

What the overlay (docker-compose.containers.yml) changes:

  • Docker CLI in the image via the INSTALL_DOCKER_CLI=true build arg (the base image stays lean).
  • Docker socket mounted into the app container. This grants the app root on the host - every build/run/exec is admin-only, --privileged is never used, and all docker calls are argument-list subprocesses, but treat the whole feature as trusted-admins-only.
  • Socket permissions: the app runs as UID 1000, so the overlay adds the host docker group via group_add. make reads the gid straight from /var/run/docker.sock (stat -c '%g'), the exact group that owns the socket.
  • Data dir at a consistent path (critical). When the app (in its container) runs docker run -v <path>:/app, the daemon resolves <path> against the host, not the app container. So the workspace/data dir must be mounted at the same absolute path on host and in the container - the make targets set DEVPLACE_DATA_DIR to the project's ./data (an absolute host path) and mount it at that identical path on both sides. (Build contexts go through the docker API as a tarball, so they can stay in the container's temp dir - only the /app bind mount needs path consistency.)
  • Ingress reach: published container ports live on the host, so the overlay sets DEVPLACE_CONTAINER_PROXY_HOST=host.docker.internal (with extra_hosts: host-gateway) so the /p/<slug> proxy can reach them. On a bare-metal make prod deploy the app is already on the host, so the default 127.0.0.1 works and no overlay is needed (just install the docker CLI and run the services).

Then enable Container builds and Containers on /admin/services. Builds default to --network=host (configurable on the service) so pip can reach PyPI; set the build network to empty to use the docker default.

nginx specifics

  • WebSockets: /devii/ws (the Devii terminal) and /p/ (container ingress) are proxied with Upgrade/Connection headers and 1h timeouts; /p/ also disables buffering for streaming.
  • Uploads: /static/uploads/ is served with X-Content-Type-Options: nosniff and a Content-Disposition that mirrors the app's UploadStaticFiles control: known-safe media (images, video, audio) is served inline so it embeds and plays in the page, while every other type is forced to attachment so it downloads instead of rendering (stored-XSS defense). SVG is deliberately excluded from the inline set.
  • Upload size: NGINX_MAX_BODY_SIZE (default 50m) must be >= the admin-configurable max_upload_size_mb (Admin -> Settings), or large uploads are rejected with HTTP 413 before reaching the app.
  • Micro-cache: off by default; enable with NGINX_CACHE_ENABLED=true.

Static asset caching

Static assets (CSS, JS, vendored libraries) are served with a one-year immutable cache for the best Lighthouse "efficient cache policy" score, while deploys still take effect immediately. Every app-owned static URL carries a boot-time version path segment, /static/v<timestamp>/..., where <timestamp> is the unix time the server process started (config.STATIC_VERSION). A restart changes the segment, so every asset URL changes and returning browsers refetch on their next page load - no cache purge, no hashing build step.

The version sits in the path, not a query string, because the frontend is unbundled ES6 modules wired with relative imports: a path segment is inherited automatically by every transitively imported module and relative CSS url(), so the whole graph busts on deploy. Templates emit URLs through the static_url Jinja global and runtime JavaScript through the assetUrl helper (static/js/assetVersion.js, reading <meta name="asset-version">). User uploads under /static/uploads/ and the service-worker.js route are excluded. Set DEVPLACE_STATIC_VERSION at launch so multiple workers share one value (the prod target and Docker image do this). Full detail: /docs/static-caching.html.

Bare-metal alternative

make prod runs the same app without containers (uvicorn ... --workers 2 --proxy-headers) from the project root, sharing the identical database and files. Note it binds port 10500, so it conflicts with the Docker front door on the same port - run one, or set a different PORT.

Multi-worker safety

The app runs correctly under multiple Uvicorn workers (independent processes sharing only the DB and lock files). Cross-worker invalidation of the auth and settings caches is coordinated through a cache_state version table (propagation bounded to ~2s); the rate limiter enforces ceil(limit / DEVPLACE_WEB_WORKERS) per process so the aggregate honours the configured value - set DEVPLACE_WEB_WORKERS to match --workers (done in make prod and the Dockerfile). First-boot DB init and VAPID key generation are flock-serialized. Full detail and the design rules are in the admin docs at /docs/production-concurrency.html.

CI/CD

Gitea Actions workflow at .gitea/workflows/test.yaml:

  • Runs on push/PR to master
  • Sets up Python 3.13, installs dependencies + Playwright Chromium
  • Runs the full test suite serially under coverage (coverage run -m pytest tests/); .coveragerc parallel=true lets the app's uvicorn subprocess write its own .coverage.* file, which coverage combine merges with the test-process data
  • Builds and publishes the coverage HTML as an artifact on every run
  • Uploads failure screenshots as artifacts when a test fails

Changes are promoted through automated DTAP streets: Development (make dev), Test (the CI suite + coverage on master), Acceptance (the master to production promotion via make deploy), and Production (the Docker Compose stack on the host). Only CI-green master commits reach production.

Feature workflow

  1. Implement the feature (router + template + CSS + JS)
  2. Validate each touched language (Python compiles/imports, JS parses, CSS and HTML balance)
  3. make test - run all tests (fail-fast)
  4. Add tests in the matching tier and endpoint file (tests/{unit,api,e2e}/<endpoint>.py) for new functionality
  5. Update AGENTS.md and README.md if new conventions were introduced

License

MIT - DevPlace

.claude
.gitea/workflows
devplacepy
examples
nginx
tests
.coveragerc
.dockerignore
.env.example
.gitignore
AGENTS.md
CLAUDE.md
docker-compose.containers.yml
docker-compose.yml
Dockerfile
locustfile.py
Makefile
ppy.Dockerfile
pyproject.toml
README.md
sitecustomize.py
test_api.json