637 lines
31 KiB
Markdown
637 lines
31 KiB
Markdown
|
# DevPlace architecture atlas
|
||
|
|
|
||
|
|
Author: retoor <retoor@molodetz.nl>
|
||
|
|
|
||
|
|
Every diagram below was derived from the source tree, the live SQLite schema and the imported FastAPI app object, not from the documentation. Where the repository documentation and the running code disagree, the disagreement is recorded in the last section.
|
||
|
|
|
||
|
|
| Measure | Value |
|
||
|
|
|---|---|
|
||
|
|
| python files | 644 |
|
||
|
|
| routers | 38 |
|
||
|
|
| http paths | 348 |
|
||
|
|
| websockets | 12 |
|
||
|
|
| services | 27 |
|
||
|
|
| db tables | 102 |
|
||
|
|
| agent tools | 318 |
|
||
|
|
| test functions | 3073 |
|
||
|
|
|
||
|
|
## 01 Deployment topology
|
||
|
|
|
||
|
|
Source: `docker-compose.yml / Dockerfile / nginx/nginx.conf.template`
|
||
|
|
|
||
|
|
Two containers on one bridge network, on a single host. nginx is the only published listener and binds to loopback only. The app container bind-mounts the repository root, so the production process reads the same SQLite file and the same `data/` tree as `make dev`.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
client["Browser / API client / Telegram / devRant client"]
|
||
|
|
subgraph host["Single host"]
|
||
|
|
subgraph appnet["docker network: appnet"]
|
||
|
|
nginx["nginx container<br/>127.0.0.1:PORT to :80<br/>8 websocket upgrade locations"]
|
||
|
|
app["app container<br/>uvicorn devplacepy.main:app<br/>--workers 2 --backlog 8192"]
|
||
|
|
end
|
||
|
|
repo["repository root<br/>bind mount .:/app"]
|
||
|
|
static["devplacepy/static<br/>read only mount"]
|
||
|
|
data["DEVPLACE_DATA_DIR = data/<br/>23 registered paths"]
|
||
|
|
sqlite[("data/devplace.db<br/>SQLite, WAL, 256MB mmap")]
|
||
|
|
lock["data/locks/devplace-services.lock<br/>flock, elects one service owner"]
|
||
|
|
docker["host Docker daemon<br/>ppy:latest instances"]
|
||
|
|
end
|
||
|
|
client --> nginx
|
||
|
|
nginx -->|"proxy_pass, X-Real-IP"| app
|
||
|
|
nginx -->|"/static/, /static/uploads/"| static
|
||
|
|
app --> repo
|
||
|
|
app --> data
|
||
|
|
data --> sqlite
|
||
|
|
app --> lock
|
||
|
|
app -->|"docker CLI backend"| docker
|
||
|
|
```
|
||
|
|
|
||
|
|
nginx depends_on app with condition service_healthy. App healthcheck: interval 30s, timeout 10s, retries 3, start_period 120s, start_interval 2s.
|
||
|
|
|
||
|
|
## 02 Worker boot and service election
|
||
|
|
|
||
|
|
Source: `devplacepy/main.py lifespan`
|
||
|
|
|
||
|
|
Every uvicorn worker runs the full lifespan. Schema work is serialized behind an exclusive lock so workers pay it one after another; background services are elected, so exactly one worker in the host owns them.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
s1["ensure_data_dirs<br/>creates all 23 DATA_PATHS"]
|
||
|
|
s2["init_lock<br/>exclusive flock on INIT_LOCK_FILE"]
|
||
|
|
s3["init_db<br/>idempotent columns and indexes"]
|
||
|
|
s4["ensure_certificates<br/>VAPID keys in data/keys"]
|
||
|
|
s5["service_manager.register x27"]
|
||
|
|
gate{"DEVPLACE_DISABLE_SERVICES set?"}
|
||
|
|
skip["services registered but never started<br/>test mode"]
|
||
|
|
bg["background.start<br/>per worker asyncio queue"]
|
||
|
|
elect{"acquire_service_lock"}
|
||
|
|
owner["set_lock_owner True<br/>supervise all 27 services"]
|
||
|
|
decline["declined, another worker owns them"]
|
||
|
|
vis["start_visit_flusher"]
|
||
|
|
serve(["serving on port 10500"])
|
||
|
|
s1 --> s2 --> s3 --> s4 --> s5 --> gate
|
||
|
|
gate -->|yes| skip --> vis
|
||
|
|
gate -->|no| bg --> elect
|
||
|
|
elect -->|"lock acquired"| owner --> vis
|
||
|
|
elect -->|"lock held elsewhere"| decline --> vis
|
||
|
|
vis --> serve
|
||
|
|
```
|
||
|
|
|
||
|
|
Shutdown reverses it: flush_visits, service_manager.shutdown_all, background.stop.
|
||
|
|
|
||
|
|
## 03 Request pipeline
|
||
|
|
|
||
|
|
Source: `devplacepy/main.py, verified against app.user_middleware`
|
||
|
|
|
||
|
|
Ten middlewares wrap every request. The order below is the real execution order read off the built middleware stack, outermost first. Two of them are plain ASGI classes; the other eight are `BaseHTTPMiddleware` dispatch functions.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
req(["incoming ASGI scope"])
|
||
|
|
m0["0 GZipMiddleware<br/>minimum_size 512, level 5"]
|
||
|
|
m1["1 TunnelDispatchMiddleware<br/>host matches a workspace tunnel?"]
|
||
|
|
tun["routers/tunnel.py<br/>handle_http / handle_ws"]
|
||
|
|
m2["2 response_timing<br/>request.state.request_start, X-Response-Time"]
|
||
|
|
m3["3 visit_statistics"]
|
||
|
|
m4["4 track_presence<br/>throttled last_seen write"]
|
||
|
|
m5["5 maintenance_middleware<br/>503 unless static, avatar, auth, admin"]
|
||
|
|
m6["6 rate_limit_middleware<br/>mutating methods only, per IP bucket"]
|
||
|
|
m7["7 add_security_headers"]
|
||
|
|
m8["8 await_pending_corrections<br/>drains sync AI correction futures"]
|
||
|
|
m9["9 refresh_db_snapshot"]
|
||
|
|
mounts["mounts: /static/uploads, /static/v{ts}, /static"]
|
||
|
|
routes["38 included routers"]
|
||
|
|
handlers["exception handlers<br/>404, 500, RequestValidationError"]
|
||
|
|
req --> m0 --> m1
|
||
|
|
m1 -->|"tunnel host"| tun
|
||
|
|
m1 -->|"normal host"| m2 --> m3 --> m4 --> m5 --> m6 --> m7 --> m8 --> m9
|
||
|
|
m9 --> mounts
|
||
|
|
m9 --> routes
|
||
|
|
routes --> handlers
|
||
|
|
```
|
||
|
|
|
||
|
|
Rate limiting reads rate_limit_per_minute and rate_limit_window_seconds from site_settings and exempts reads and the /openai gateway. Bucket key is X-Real-IP falling back to request.client.host.
|
||
|
|
|
||
|
|
## 04 URL surface
|
||
|
|
|
||
|
|
Source: `devplacepy/main.py include_router calls`
|
||
|
|
|
||
|
|
38 routers, each mounted under one prefix, grouped here by the concern they serve. Directory shape mirrors the URL: a domain with one resource is a flat module, a domain with several sub-resources is a package that aggregates its leaves in `__init__.py`.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart LR
|
||
|
|
app(["FastAPI app"])
|
||
|
|
subgraph identity["Identity and profile"]
|
||
|
|
r1["/auth - auth/"]
|
||
|
|
r2["/profile - profile/"]
|
||
|
|
r3["/avatar - avatar.py"]
|
||
|
|
r4["/follow - follow.py"]
|
||
|
|
r5["(none) - relations.py block, mute"]
|
||
|
|
end
|
||
|
|
subgraph social["Content and engagement"]
|
||
|
|
r6["/feed - feed.py"]
|
||
|
|
r7["/posts - posts.py"]
|
||
|
|
r8["/comments - comments.py"]
|
||
|
|
r9["/gists - gists.py"]
|
||
|
|
r10["/news - news.py"]
|
||
|
|
r11["/votes - votes.py"]
|
||
|
|
r12["/reactions - reactions.py"]
|
||
|
|
r13["/bookmarks - bookmarks.py"]
|
||
|
|
r14["/polls - polls.py"]
|
||
|
|
r15["/awards - awards.py"]
|
||
|
|
r16["/leaderboard - leaderboard.py"]
|
||
|
|
r17["/notifications - notifications.py"]
|
||
|
|
r18["/messages - messages.py"]
|
||
|
|
r19["/uploads - uploads.py"]
|
||
|
|
r20["/media - media.py"]
|
||
|
|
end
|
||
|
|
subgraph work["Projects and compute"]
|
||
|
|
r21["/projects - projects/ incl files, containers"]
|
||
|
|
r22["/p - proxy.py container ingress"]
|
||
|
|
r23["/zips - zips.py"]
|
||
|
|
r24["/forks - forks.py"]
|
||
|
|
r25["/issues - issues/"]
|
||
|
|
end
|
||
|
|
subgraph aiml["AI and tools"]
|
||
|
|
r26["/openai - openai_gateway.py"]
|
||
|
|
r27["/devii - devii.py"]
|
||
|
|
r28["/tools - tools/ seo, deepsearch, isslop"]
|
||
|
|
end
|
||
|
|
subgraph play["Play"]
|
||
|
|
r29["/game - game/"]
|
||
|
|
r30["/quizzes - quizzes/"]
|
||
|
|
end
|
||
|
|
subgraph platform["Platform"]
|
||
|
|
r31["/admin - admin/ 23 modules"]
|
||
|
|
r32["/docs - docs/"]
|
||
|
|
r33["(none) - push.py, seo.py"]
|
||
|
|
end
|
||
|
|
subgraph machine["Machine interfaces"]
|
||
|
|
r34["/api - devrant/"]
|
||
|
|
r35["/dbapi - dbapi/ read only"]
|
||
|
|
r36["/xmlrpc - xmlrpc.py"]
|
||
|
|
r37["/pubsub - pubsub.py"]
|
||
|
|
end
|
||
|
|
app --> identity
|
||
|
|
app --> social
|
||
|
|
app --> work
|
||
|
|
app --> aiml
|
||
|
|
app --> play
|
||
|
|
app --> platform
|
||
|
|
app --> machine
|
||
|
|
```
|
||
|
|
|
||
|
|
| Prefix | Paths | Prefix | Paths | Prefix | Paths |
|
||
|
|
|---|---|---|---|---|---|
|
||
|
|
| /admin | 92 | /profile | 17 | /gists | 5 |
|
||
|
|
| /projects | 46 | /issues | 13 | /messages | 5 |
|
||
|
|
| /tools | 27 | /dbapi | 9 | /notifications | 5 |
|
||
|
|
| /game | 23 | /auth | 6 | /uploads | 5 |
|
||
|
|
| /quizzes | 21 | /devii | 5 | /docs | 4 |
|
||
|
|
| /api | 20 | /posts | 4 | everything else | 36 |
|
||
|
|
|
||
|
|
## 05 Route fan-out
|
||
|
|
|
||
|
|
Source: `devplacepy/responses.py, schemas/, docs_api/, services/devii/actions/`
|
||
|
|
|
||
|
|
One handler serves four consumers. `respond()` feeds the same context dict to a Pydantic model and to a Jinja template, so a key missing from the `*Out` schema is silently absent from JSON while still rendering in HTML. The agent catalog and the docs registry are separate declarations that must be kept in step by hand.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
form["models.py<br/>Pydantic form model via Annotated Form"]
|
||
|
|
guard["guard: get_current_user<br/>require_user / require_admin"]
|
||
|
|
handler["router handler"]
|
||
|
|
helpers["database/ batch helpers<br/>get_users_by_uids, build_pagination"]
|
||
|
|
ctx["context dict"]
|
||
|
|
neg{"wants_json(request)"}
|
||
|
|
json["model.model_validate<br/>JSONResponse"]
|
||
|
|
html["templates.TemplateResponse<br/>extends base.html"]
|
||
|
|
action["services/devii/actions catalog<br/>Action, requires_auth mirrors guard"]
|
||
|
|
docs["docs_api endpoint()<br/>params and sample_response"]
|
||
|
|
seo["seo.py base_seo_context<br/>JSON-LD, sitemap entry"]
|
||
|
|
form --> handler
|
||
|
|
guard --> handler
|
||
|
|
handler --> helpers --> ctx --> neg
|
||
|
|
neg -->|"json accepted"| json
|
||
|
|
neg -->|"otherwise"| html
|
||
|
|
handler -.->|"same capability"| action
|
||
|
|
handler -.->|"same contract"| docs
|
||
|
|
html -.-> seo
|
||
|
|
```
|
||
|
|
|
||
|
|
wants_json is true when the request content-type starts with application/json, or Accept contains application/json and does not contain text/html. Registry sizes: 318 agent actions, 259 documented endpoint entries across 20 API groups, 122 prose docs pages.
|
||
|
|
|
||
|
|
## 06 Identity resolution
|
||
|
|
|
||
|
|
Source: `devplacepy/utils/auth.py`
|
||
|
|
|
||
|
|
A single resolver serves browsers, API clients, the devRant compatibility layer and issued access tokens. The result is memoised on `request.state` for the request and in a process TTL cache keyed by credential.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
start(["get_current_user"])
|
||
|
|
cached{"request.state._auth_user set?"}
|
||
|
|
sess{"session cookie<br/>64 hex chars"}
|
||
|
|
xapi{"X-API-KEY header"}
|
||
|
|
bearer{"Authorization: Bearer"}
|
||
|
|
basic{"Authorization: Basic"}
|
||
|
|
triple["try in order:<br/>users.api_key,<br/>devrant token 40 hex,<br/>access token 64 chars"]
|
||
|
|
user(["user dict"])
|
||
|
|
guest(["None, guest"])
|
||
|
|
gu["require_user: 303 to /"]
|
||
|
|
ga["require_admin: redirect to /feed"]
|
||
|
|
start --> cached
|
||
|
|
cached -->|yes| user
|
||
|
|
cached -->|no| sess
|
||
|
|
sess -->|match| user
|
||
|
|
sess -->|no| xapi
|
||
|
|
xapi -->|present| triple
|
||
|
|
triple -->|match| user
|
||
|
|
triple -->|no match| bearer
|
||
|
|
xapi -->|absent| bearer
|
||
|
|
bearer -->|present| triple
|
||
|
|
bearer -->|absent| basic
|
||
|
|
basic -->|"username-or-email:password, pbkdf2_sha256"| user
|
||
|
|
basic -->|no| guest
|
||
|
|
guest --> gu
|
||
|
|
user --> ga
|
||
|
|
```
|
||
|
|
|
||
|
|
Roles are stored capitalized as Admin or Member and tested through the is_admin global. Admin seniority is enforced per user mutation in routers/admin/users.py.
|
||
|
|
|
||
|
|
## 07 Background service fleet
|
||
|
|
|
||
|
|
Source: `devplacepy/services/manager.py, base.py, jobs/base.py`
|
||
|
|
|
||
|
|
27 singletons registered at boot, supervised only by the worker that won the service lock. Sixteen are long-lived loops on `BaseService`; eleven are queue consumers on `JobService`, which adds retention, concurrency and timeout settings on top and drains rows from the shared `jobs` table.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart LR
|
||
|
|
mgr["ServiceManager<br/>supervise, shutdown_all"]
|
||
|
|
settings[("site_settings<br/>enabled flag and interval per service")]
|
||
|
|
jobs[("jobs table")]
|
||
|
|
subgraph base["BaseService loops - 16"]
|
||
|
|
b1["NewsService"]
|
||
|
|
b2["BotsService"]
|
||
|
|
b3["GatewayService"]
|
||
|
|
b4["DeviiService"]
|
||
|
|
b5["PubSubService"]
|
||
|
|
b6["NotificationRelayService"]
|
||
|
|
b7["LiveViewRelayService"]
|
||
|
|
b8["PresenceRelayService"]
|
||
|
|
b9["IssueTrackerService"]
|
||
|
|
b10["ContainerService"]
|
||
|
|
b11["WorkspaceService"]
|
||
|
|
b12["XmlrpcService"]
|
||
|
|
b13["AuditService"]
|
||
|
|
b14["PushService"]
|
||
|
|
b15["TelegramService"]
|
||
|
|
b16["TelegramOutboxService"]
|
||
|
|
end
|
||
|
|
subgraph job["JobService consumers - 11"]
|
||
|
|
j1["ZipService"]
|
||
|
|
j2["ForkService"]
|
||
|
|
j3["SeoService"]
|
||
|
|
j4["SeoMetaService"]
|
||
|
|
j5["AwardService"]
|
||
|
|
j6["BackupService"]
|
||
|
|
j7["DbApiJobService"]
|
||
|
|
j8["DeepsearchService"]
|
||
|
|
j9["IsslopService"]
|
||
|
|
j10["IssueCreateService"]
|
||
|
|
j11["PlanningReportService"]
|
||
|
|
end
|
||
|
|
mgr --> base
|
||
|
|
mgr --> job
|
||
|
|
settings --> mgr
|
||
|
|
job --> jobs
|
||
|
|
```
|
||
|
|
|
||
|
|
Separate from the fleet, services/background.py offers a fire and forget queue per worker for audit writes, XP awards and notifications; when no consumer runs, as in tests, the callable executes inline so ordering stays deterministic.
|
||
|
|
|
||
|
|
## 08 Data layer
|
||
|
|
|
||
|
|
Source: `devplacepy/database/, devplacepy/config.py`
|
||
|
|
|
||
|
|
One SQLite file reached through `dataset`, called synchronously from async handlers by design. 102 tables exist in the live schema; 52 of them carry the soft delete pair and are restorable as one event from the admin trash.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
db[("data/devplace.db")]
|
||
|
|
subgraph g1["Identity and access - 14"]
|
||
|
|
t1["users, sessions, password_resets<br/>access_tokens, devrant_tokens<br/>user_relations, follows<br/>notification_preferences, user_customizations<br/>push_registration, email_accounts<br/>telegram_links, telegram_pairings, telegram_outbox"]
|
||
|
|
end
|
||
|
|
subgraph g2["Content and engagement - 16"]
|
||
|
|
t2["posts, comments, gists<br/>projects, project_files, project_forks<br/>attachments, news, news_images, news_sync<br/>polls, poll_options, poll_votes<br/>votes, reactions, bookmarks"]
|
||
|
|
end
|
||
|
|
subgraph g3["Reputation - 5"]
|
||
|
|
t3["awards, badges, award_usage<br/>user_activity, user_activity_seen"]
|
||
|
|
end
|
||
|
|
subgraph g4["Code Farm - 9"]
|
||
|
|
t4["game_farms, game_plots, game_quests<br/>game_cosmetics, game_market_ticks<br/>game_steals, game_treasury<br/>game_eras, game_era_results"]
|
||
|
|
end
|
||
|
|
subgraph g5["Quizzes - 5"]
|
||
|
|
t5["quizzes, quiz_questions, quiz_options<br/>quiz_attempts, quiz_answers"]
|
||
|
|
end
|
||
|
|
subgraph g6["Live delivery - 3"]
|
||
|
|
t6["messages, notifications, ws_tickets"]
|
||
|
|
end
|
||
|
|
subgraph g7["Devii - 8"]
|
||
|
|
t7["devii_conversations, devii_turns<br/>devii_tasks, devii_task_runs<br/>devii_lessons, devii_virtual_tools<br/>devii_behavior, devii_usage_ledger"]
|
||
|
|
end
|
||
|
|
subgraph g8["AI gateway - 6"]
|
||
|
|
t8["gateway_providers, gateway_models<br/>gateway_usage_ledger<br/>gateway_quota_rules, gateway_quota_resets<br/>gateway_concurrency_samples"]
|
||
|
|
end
|
||
|
|
subgraph g9["Jobs, tools and usage - 20"]
|
||
|
|
t9["jobs, backups, backup_schedules<br/>deepsearch_sessions, deepsearch_messages, deepsearch_url_cache<br/>isslop_analyses, isslop_reports, isslop_events<br/>isslop_dom_results, isslop_file_results, isslop_image_results<br/>seo_metadata, seo_usage<br/>issue_tickets, issue_comment_authors, issue_usage<br/>correction_usage, modifier_usage, news_usage"]
|
||
|
|
end
|
||
|
|
subgraph g10["Containers - 9"]
|
||
|
|
t10["instances, instance_events, instance_metrics<br/>tunnels, workspace_flags, workspace_quota_rules<br/>builds, dockerfiles, dockerfile_versions"]
|
||
|
|
end
|
||
|
|
subgraph g11["Platform state - 7"]
|
||
|
|
t11["site_settings, cache_state, service_state<br/>audit_log, audit_log_links<br/>visit_stats_hourly, visit_unique_slots"]
|
||
|
|
end
|
||
|
|
db --> g1
|
||
|
|
db --> g2
|
||
|
|
db --> g3
|
||
|
|
db --> g4
|
||
|
|
db --> g5
|
||
|
|
db --> g6
|
||
|
|
db --> g7
|
||
|
|
db --> g8
|
||
|
|
db --> g9
|
||
|
|
db --> g10
|
||
|
|
db --> g11
|
||
|
|
```
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart LR
|
||
|
|
root["DEVPLACE_DATA_DIR<br/>default repo/data"]
|
||
|
|
blobs["uploads, attachments, project_files<br/>sharded xx/yy on the uuid7 random tail"]
|
||
|
|
jobsdir["zips, zip_staging, fork_staging<br/>backups, backup_staging"]
|
||
|
|
reports["seo_reports, planning_reports<br/>dbapi, deepsearch, deepsearch_chroma"]
|
||
|
|
isslop["isslop, isslop_workspaces<br/>isslop_runs, isslop_media"]
|
||
|
|
ws["container_workspaces, workspace_state"]
|
||
|
|
keys["keys VAPID, bot, locks"]
|
||
|
|
dbs["devplace.db, devii_tasks.db, devii_lessons.db"]
|
||
|
|
root --> blobs
|
||
|
|
root --> jobsdir
|
||
|
|
root --> reports
|
||
|
|
root --> isslop
|
||
|
|
root --> ws
|
||
|
|
root --> keys
|
||
|
|
root --> dbs
|
||
|
|
```
|
||
|
|
|
||
|
|
config.DATA_PATHS registers 23 directories and ensure_data_dirs creates the whole tree before any write. Uploads live under data/uploads but are served at the unchanged /static/uploads URL.
|
||
|
|
|
||
|
|
## 09 Content rendering
|
||
|
|
|
||
|
|
Source: `devplacepy/rendering.py, static/js/ContentRenderer.js`
|
||
|
|
|
||
|
|
Two pipelines with a deliberate split: anything that exists at request time is rendered on the server for SEO, and the client pipeline is reserved for content that does not exist yet. Each has its own XSS control at a different point.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
subgraph server["Server, rendering.py, lru_cache"]
|
||
|
|
a1["raw text"]
|
||
|
|
a2["normalize dashes to hyphen"]
|
||
|
|
a3["emoji shortcodes, 4869 names"]
|
||
|
|
a4["mistune GFM, escape=True"]
|
||
|
|
a5["media pass: bare URLs to embeds, mentions to links"]
|
||
|
|
a6["mask emails on rendered text nodes"]
|
||
|
|
a7["render_content / render_title in template"]
|
||
|
|
a8["ContentEnhancer adds highlighting and copy buttons"]
|
||
|
|
a1 --> a2 --> a3 --> a4 --> a5 --> a6 --> a7 --> a8
|
||
|
|
end
|
||
|
|
subgraph client["Client, ContentRenderer.js"]
|
||
|
|
b1["live text: comments, DM bubbles, Devii, DeepSearch"]
|
||
|
|
b2["marked"]
|
||
|
|
b3["DOMPurify.sanitize, fail closed"]
|
||
|
|
b4["highlight.js"]
|
||
|
|
b5["media and autolink pass"]
|
||
|
|
b1 --> b2 --> b3 --> b4 --> b5
|
||
|
|
end
|
||
|
|
```
|
||
|
|
|
||
|
|
The server escapes at the markdown step, the client sanitizes after parsing. Server rendered content must not carry data-render, or both pipelines run over it.
|
||
|
|
|
||
|
|
## 10 AI plane
|
||
|
|
|
||
|
|
Source: `devplacepy/routers/openai_gateway.py, services/openai_gateway/, services/devii/`
|
||
|
|
|
||
|
|
Every model call in the product, internal or external, leaves through one gateway, which is where routing, quota and cost attribution live. Internal consumers call it over loopback HTTP rather than importing a client, so their spend lands in the same ledger.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
ext["external client<br/>/openai/v1/*"]
|
||
|
|
subgraph consumers["Internal consumers via INTERNAL_GATEWAY_URL"]
|
||
|
|
c1["services/devii"]
|
||
|
|
c2["services/correction<br/>AI correction and modifier"]
|
||
|
|
c3["services/news"]
|
||
|
|
c4["services/bot"]
|
||
|
|
c5["services/deepsearch + jobs/deepsearch"]
|
||
|
|
c6["services/dbapi nl2sql"]
|
||
|
|
c7["services/gitea enhance + planning"]
|
||
|
|
c8["jobs/isslop"]
|
||
|
|
end
|
||
|
|
gw["gateway.py<br/>routing.py, quota.py, usage.py<br/>reliability.py, vision.py"]
|
||
|
|
ledger[("gateway_usage_ledger<br/>gateway_quota_rules")]
|
||
|
|
stealth["stealth_async_client<br/>curl_cffi Chrome 146 fingerprint"]
|
||
|
|
up["upstream provider"]
|
||
|
|
subgraph devii["Devii assistant"]
|
||
|
|
d1["registry.py CATALOG<br/>318 tools"]
|
||
|
|
d2["role gating<br/>guest 108, member 246<br/>admin 312, primary 318"]
|
||
|
|
d3["45 tools behind CONFIRM_REQUIRED"]
|
||
|
|
d4["session, tasks, lessons<br/>virtual tools, behavior"]
|
||
|
|
d5["surfaces: /devii ws, Telegram, CLI"]
|
||
|
|
end
|
||
|
|
ext --> gw
|
||
|
|
consumers --> gw
|
||
|
|
gw --> ledger
|
||
|
|
gw --> stealth --> up
|
||
|
|
d1 --> d2 --> d3
|
||
|
|
devii --> c1
|
||
|
|
d4 --> d1
|
||
|
|
d5 --> d4
|
||
|
|
```
|
||
|
|
|
||
|
|
Cleartext loopback calls are forced to HTTP/1.1 in curl_transport.http_version_for, because the Chrome impersonation profile would otherwise negotiate HTTP/2 against an HTTP/1.1 only uvicorn.
|
||
|
|
|
||
|
|
## 11 Containers, workspaces and ingress
|
||
|
|
|
||
|
|
Source: `devplacepy/services/containers/, routers/projects/containers/, routers/proxy.py, routers/tunnel.py`
|
||
|
|
|
||
|
|
One shared image serves every instance. Reconciliation is a loop that compares desired state in the database against what the Docker daemon actually reports, and there are two independent ways in from the outside: a path prefix and a hostname.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
ui["/projects/{slug}/containers<br/>instances, schedules, workspace"]
|
||
|
|
api["services/containers/api.py"]
|
||
|
|
store[("instances, instance_events<br/>instance_metrics, tunnels<br/>workspace_flags, workspace_quota_rules")]
|
||
|
|
svc["ContainerService<br/>reconcile desired vs docker ps"]
|
||
|
|
wsvc["WorkspaceService<br/>provision, certs, quota, tunnels"]
|
||
|
|
backend["backend/docker_cli.py<br/>backend/fake.py for tests"]
|
||
|
|
image["single shared image ppy:latest"]
|
||
|
|
inst["running instance<br/>rootless workflow, aptroot"]
|
||
|
|
wsdir["data/container_workspaces/{uid}"]
|
||
|
|
proxy["/p/{slug}<br/>routers/proxy.py, relays headers verbatim"]
|
||
|
|
tunnel["host based tunnel<br/>TunnelDispatchMiddleware to routers/tunnel.py"]
|
||
|
|
code["workspace editor over websocket<br/>/projects/{slug}/containers/instances/{uid}/code"]
|
||
|
|
exec["terminal over websocket<br/>.../exec/ws"]
|
||
|
|
ui --> api --> store
|
||
|
|
svc --> store
|
||
|
|
wsvc --> store
|
||
|
|
api --> backend
|
||
|
|
svc --> backend
|
||
|
|
wsvc --> wsdir
|
||
|
|
backend --> image --> inst
|
||
|
|
proxy --> inst
|
||
|
|
tunnel --> inst
|
||
|
|
code --> inst
|
||
|
|
exec --> inst
|
||
|
|
```
|
||
|
|
|
||
|
|
Access uses stricter predicates than the rest of the product: owns_instance, can_view_project_containers, can_view_instance, can_manage_instance. The primary administrator sees and manages every container; any other admin can only view others on public projects and manage instances they own.
|
||
|
|
|
||
|
|
## 12 Real time plane
|
||
|
|
|
||
|
|
Source: `routers/*.py websocket handlers, services relays, nginx.conf.template`
|
||
|
|
|
||
|
|
Twelve WebSocket endpoints, each of which needs its own nginx upgrade location because the catch-all location strips upgrade headers. Fan-out to connected sockets goes through in-process hubs driven by relay services on the lock-owning worker.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart LR
|
||
|
|
subgraph sockets["WebSocket endpoints"]
|
||
|
|
s1["/devii/ws"]
|
||
|
|
s2["/messages/ws"]
|
||
|
|
s3["/pubsub/ws"]
|
||
|
|
s4["/tools/seo/{uid}/ws"]
|
||
|
|
s5["/tools/deepsearch/{uid}/ws"]
|
||
|
|
s6["/tools/deepsearch/{uid}/chat"]
|
||
|
|
s7["/dbapi/query/{uid}/ws"]
|
||
|
|
s8[".../containers/instances/{uid}/exec/ws"]
|
||
|
|
s9[".../containers/instances/{uid}/code"]
|
||
|
|
s10[".../code/{path}"]
|
||
|
|
s11["/p/{slug}"]
|
||
|
|
s12["/p/{slug}/{path}"]
|
||
|
|
end
|
||
|
|
subgraph relays["Relay services"]
|
||
|
|
r1["NotificationRelayService"]
|
||
|
|
r2["LiveViewRelayService"]
|
||
|
|
r3["PresenceRelayService<br/>track limit 500, online limit 30"]
|
||
|
|
r4["PubSubService"]
|
||
|
|
end
|
||
|
|
hubs["messaging/hub.py, pubsub/hub.py<br/>devii/hub.py"]
|
||
|
|
tick[("ws_tickets<br/>expiring auth tickets")]
|
||
|
|
s2 --> hubs
|
||
|
|
s3 --> hubs
|
||
|
|
s1 --> hubs
|
||
|
|
relays --> hubs
|
||
|
|
tick --> s2
|
||
|
|
```
|
||
|
|
|
||
|
|
Presence is authoritative from the relay: PRESENCE_TRACK_LIMIT sets the tracked online set, PRESENCE_ONLINE_LIMIT only caps how many the feed panel displays, and PRESENCE_ONLINE_MARGIN_SECONDS provides hysteresis at the boundary.
|
||
|
|
|
||
|
|
## 13 Frontend
|
||
|
|
|
||
|
|
Source: `devplacepy/static/, devplacepy/templates/`
|
||
|
|
|
||
|
|
No framework and no package manager. 131 ES6 modules, one class per file, hung off a single global `app`; 33 of them are custom elements. 217 Jinja templates, of which 107 are the docs site.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart TB
|
||
|
|
base["templates/base.html<br/>extra_head for CSS, extra_js for JS"]
|
||
|
|
partials["shared partials<br/>_avatar_link, _user_link, _comment_section<br/>_post_card, _pagination, _sidebar_search"]
|
||
|
|
appjs["static/js/Application.js<br/>instantiated once as app"]
|
||
|
|
subgraph modules["131 ES6 modules"]
|
||
|
|
u1["utilities<br/>Http, Poller, JobPoller<br/>OptimisticAction, FloatingWindow, ScrollMemory"]
|
||
|
|
u2["33 custom elements<br/>components/ with dp- prefix"]
|
||
|
|
u3["feature modules<br/>chat/, devii/, autoload/"]
|
||
|
|
end
|
||
|
|
css["49 stylesheets<br/>variables.css tokens, per page files"]
|
||
|
|
vers["static_url in Jinja, assetUrl in JS<br/>/static/v{boot ts}/, immutable for a year"]
|
||
|
|
cust["per user CSS and JS injection<br/>custom_css_tag, custom_js_tag"]
|
||
|
|
base --> partials
|
||
|
|
base --> appjs --> modules
|
||
|
|
base --> css
|
||
|
|
base --> cust
|
||
|
|
vers --> css
|
||
|
|
vers --> modules
|
||
|
|
```
|
||
|
|
|
||
|
|
Per user customizations are configured only through Devii, stored in user_customizations, scoped globally or per matched route template, and run solely in that owner's own browser sessions.
|
||
|
|
|
||
|
|
## 14 Test and delivery topology
|
||
|
|
|
||
|
|
Source: `tests/, pyproject.toml, .gitea/workflows/test.yaml`
|
||
|
|
|
||
|
|
Three tiers separated by what they exercise, decided by fixtures, run serially in a single process against one uvicorn subprocess on port 10501 with a temporary database.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
|
||
|
|
flowchart LR
|
||
|
|
unit["tests/unit - 140 modules<br/>mirrors source module path<br/>local_db or no fixture"]
|
||
|
|
api["tests/api - 221 modules<br/>mirrors URL path<br/>app_server, seeded_db"]
|
||
|
|
e2e["tests/e2e - 131 modules<br/>mirrors URL path<br/>page, alice, bob"]
|
||
|
|
srv["uvicorn subprocess :10501<br/>temp SQLite, own DATA_DIR<br/>DEVPLACE_DISABLE_SERVICES=1"]
|
||
|
|
pw["Playwright Chromium<br/>session scoped context"]
|
||
|
|
ci["Gitea Actions on push and PR to master<br/>full suite under coverage"]
|
||
|
|
prod["promotion to production<br/>docker compose"]
|
||
|
|
unit --> ci
|
||
|
|
api --> srv --> ci
|
||
|
|
e2e --> srv
|
||
|
|
e2e --> pw --> ci
|
||
|
|
ci --> prod
|
||
|
|
```
|
||
|
|
|
||
|
|
pytest-xdist is not a dependency and -n is rejected centrally, so no tier can be parallelised by accident.
|
||
|
|
|
||
|
|
## 15 Findings
|
||
|
|
|
||
|
|
Source: `measured against the repository documentation`
|
||
|
|
|
||
|
|
The structure holds up: the router tree mirrors the URL tree, the test tree mirrors both, every runtime path is registered in one place, and every model call has a single exit. The items below are the places where the running code and the written record have drifted apart, or where a structure exists that nothing currently uses. None of them is a functional defect.
|
||
|
|
|
||
|
|
### Middleware order in the documentation is stale
|
||
|
|
|
||
|
|
The root CLAUDE.md states that response_timing is the outermost middleware. Measured from app.user_middleware it is third, behind GZipMiddleware and TunnelDispatchMiddleware, both added after it. The consequence is minor but real: X-Response-Time excludes compression time and excludes tunnel-host dispatch entirely.
|
||
|
|
|
||
|
|
### Two catalogue counts have drifted
|
||
|
|
|
||
|
|
CLAUDE.md cites around 2882 tests and an events.md catalogue of 288 keys. Measured now: 3073 functions named test_ across 492 modules, and 312 distinct dotted event keys in events.md. Both are undercounts in the docs, not missing implementation.
|
||
|
|
|
||
|
|
### instance_schedules is declared soft-deletable but has no table
|
||
|
|
|
||
|
|
It appears in database.SOFT_DELETE_TABLES and in the routers under /projects/{slug}/containers/schedules, but no such table exists in the live schema. dataset creates it lazily on first insert, so this is correct only for as long as every read path tolerates the table being absent.
|
||
|
|
|
||
|
|
### Three legacy container tables still exist
|
||
|
|
|
||
|
|
builds, dockerfiles and dockerfile_versions are present in the live database. The project replaced per-project images with the single shared ppy:latest image and ships devplace containers prune-builds as the one-time cleanup for exactly these rows. The tables are still carried.
|
||
|
|
|
||
|
|
### Five duplicate OpenAPI operation IDs
|
||
|
|
|
||
|
|
Generating the schema warns on editor_proxy twice in routers/projects/containers/workspace.py, passthrough in routers/openai_gateway.py, and proxy_http twice in routers/proxy.py. These are catch-all routes registered for several methods, so a generated client would collide on those names.
|
||
|
|
|
||
|
|
### The one deliberate asymmetry is documented and intentional
|
||
|
|
|
||
|
|
routers/__init__.py holds nothing but the attribution line; unlike every nested router package, the top level does not aggregate. main.py performs all 38 include_router calls directly, which keeps prefix ownership in a single readable block.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
Sources: devplacepy/main.py, the imported FastAPI app object, data/devplace.db sqlite_master, and the working tree at HEAD 192df12b with uncommitted local modifications present. Counts exclude __pycache__ and the virtual environment.
|