retoor/devplacepy
1
0
Fork 1
Code Issues 1 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Dit is hoe wij doen in oost
DevPlace CI / test (push) Failing after 46m20s
Details

Browse Source
This commit is contained in:
retoor
2026-10-17 03:18:43 +02:00
parent 02ab747cd0
commit d326c6b3f1
18 changed files with 126 additions and 4195 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -1,3 +1,4 @@
.docs
.dpc
.devplace
dpc.log
ARCHITECTURE.md
-636
View File
@@ -1,636 +0,0 @@
# DevPlace architecture atlas
Author: retoor <retoor@molodetz.nl>
Every diagram below was derived from the source tree, the live SQLite schema and the imported FastAPI app object, not from the documentation. Where the repository documentation and the running code disagree, the disagreement is recorded in the last section.
| Measure | Value |
|---|---|
| python files | 644 |
| routers | 38 |
| http paths | 348 |
| websockets | 12 |
| services | 27 |
| db tables | 102 |
| agent tools | 318 |
| test functions | 3073 |
## 01 Deployment topology
Source: `docker-compose.yml / Dockerfile / nginx/nginx.conf.template`
Two containers on one bridge network, on a single host. nginx is the only published listener and binds to loopback only. The app container bind-mounts the repository root, so the production process reads the same SQLite file and the same `data/` tree as `make dev`.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
client["Browser / API client / Telegram / devRant client"]
subgraph host["Single host"]
subgraph appnet["docker network: appnet"]
nginx["nginx container<br/>127.0.0.1:PORT to :80<br/>8 websocket upgrade locations"]
app["app container<br/>uvicorn devplacepy.main:app<br/>--workers 2 --backlog 8192"]
end
repo["repository root<br/>bind mount .:/app"]
static["devplacepy/static<br/>read only mount"]
data["DEVPLACE_DATA_DIR = data/<br/>23 registered paths"]
sqlite[("data/devplace.db<br/>SQLite, WAL, 256MB mmap")]
lock["data/locks/devplace-services.lock<br/>flock, elects one service owner"]
docker["host Docker daemon<br/>ppy:latest instances"]
end
client --> nginx
nginx -->|"proxy_pass, X-Real-IP"| app
nginx -->|"/static/, /static/uploads/"| static
app --> repo
app --> data
data --> sqlite
app --> lock
app -->|"docker CLI backend"| docker
```
nginx depends_on app with condition service_healthy. App healthcheck: interval 30s, timeout 10s, retries 3, start_period 120s, start_interval 2s.
## 02 Worker boot and service election
Source: `devplacepy/main.py lifespan`
Every uvicorn worker runs the full lifespan. Schema work is serialized behind an exclusive lock so workers pay it one after another; background services are elected, so exactly one worker in the host owns them.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
s1["ensure_data_dirs<br/>creates all 23 DATA_PATHS"]
s2["init_lock<br/>exclusive flock on INIT_LOCK_FILE"]
s3["init_db<br/>idempotent columns and indexes"]
s4["ensure_certificates<br/>VAPID keys in data/keys"]
s5["service_manager.register x27"]
gate{"DEVPLACE_DISABLE_SERVICES set?"}
skip["services registered but never started<br/>test mode"]
bg["background.start<br/>per worker asyncio queue"]
elect{"acquire_service_lock"}
owner["set_lock_owner True<br/>supervise all 27 services"]
decline["declined, another worker owns them"]
vis["start_visit_flusher"]
serve(["serving on port 10500"])
s1 --> s2 --> s3 --> s4 --> s5 --> gate
gate -->|yes| skip --> vis
gate -->|no| bg --> elect
elect -->|"lock acquired"| owner --> vis
elect -->|"lock held elsewhere"| decline --> vis
vis --> serve
```
Shutdown reverses it: flush_visits, service_manager.shutdown_all, background.stop.
## 03 Request pipeline
Source: `devplacepy/main.py, verified against app.user_middleware`
Ten middlewares wrap every request. The order below is the real execution order read off the built middleware stack, outermost first. Two of them are plain ASGI classes; the other eight are `BaseHTTPMiddleware` dispatch functions.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
req(["incoming ASGI scope"])
m0["0 GZipMiddleware<br/>minimum_size 512, level 5"]
m1["1 TunnelDispatchMiddleware<br/>host matches a workspace tunnel?"]
tun["routers/tunnel.py<br/>handle_http / handle_ws"]
m2["2 response_timing<br/>request.state.request_start, X-Response-Time"]
m3["3 visit_statistics"]
m4["4 track_presence<br/>throttled last_seen write"]
m5["5 maintenance_middleware<br/>503 unless static, avatar, auth, admin"]
m6["6 rate_limit_middleware<br/>mutating methods only, per IP bucket"]
m7["7 add_security_headers"]
m8["8 await_pending_corrections<br/>drains sync AI correction futures"]
m9["9 refresh_db_snapshot"]
mounts["mounts: /static/uploads, /static/v{ts}, /static"]
routes["38 included routers"]
handlers["exception handlers<br/>404, 500, RequestValidationError"]
req --> m0 --> m1
m1 -->|"tunnel host"| tun
m1 -->|"normal host"| m2 --> m3 --> m4 --> m5 --> m6 --> m7 --> m8 --> m9
m9 --> mounts
m9 --> routes
routes --> handlers
```
Rate limiting reads rate_limit_per_minute and rate_limit_window_seconds from site_settings and exempts reads and the /openai gateway. Bucket key is X-Real-IP falling back to request.client.host.
## 04 URL surface
Source: `devplacepy/main.py include_router calls`
38 routers, each mounted under one prefix, grouped here by the concern they serve. Directory shape mirrors the URL: a domain with one resource is a flat module, a domain with several sub-resources is a package that aggregates its leaves in `__init__.py`.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart LR
app(["FastAPI app"])
subgraph identity["Identity and profile"]
r1["/auth - auth/"]
r2["/profile - profile/"]
r3["/avatar - avatar.py"]
r4["/follow - follow.py"]
r5["(none) - relations.py block, mute"]
end
subgraph social["Content and engagement"]
r6["/feed - feed.py"]
r7["/posts - posts.py"]
r8["/comments - comments.py"]
r9["/gists - gists.py"]
r10["/news - news.py"]
r11["/votes - votes.py"]
r12["/reactions - reactions.py"]
r13["/bookmarks - bookmarks.py"]
r14["/polls - polls.py"]
r15["/awards - awards.py"]
r16["/leaderboard - leaderboard.py"]
r17["/notifications - notifications.py"]
r18["/messages - messages.py"]
r19["/uploads - uploads.py"]
r20["/media - media.py"]
end
subgraph work["Projects and compute"]
r21["/projects - projects/ incl files, containers"]
r22["/p - proxy.py container ingress"]
r23["/zips - zips.py"]
r24["/forks - forks.py"]
r25["/issues - issues/"]
end
subgraph aiml["AI and tools"]
r26["/openai - openai_gateway.py"]
r27["/devii - devii.py"]
r28["/tools - tools/ seo, deepsearch, isslop"]
end
subgraph play["Play"]
r29["/game - game/"]
r30["/quizzes - quizzes/"]
end
subgraph platform["Platform"]
r31["/admin - admin/ 23 modules"]
r32["/docs - docs/"]
r33["(none) - push.py, seo.py"]
end
subgraph machine["Machine interfaces"]
r34["/api - devrant/"]
r35["/dbapi - dbapi/ read only"]
r36["/xmlrpc - xmlrpc.py"]
r37["/pubsub - pubsub.py"]
end
app --> identity
app --> social
app --> work
app --> aiml
app --> play
app --> platform
app --> machine
```
| Prefix | Paths | Prefix | Paths | Prefix | Paths |
|---|---|---|---|---|---|
| /admin | 92 | /profile | 17 | /gists | 5 |
| /projects | 46 | /issues | 13 | /messages | 5 |
| /tools | 27 | /dbapi | 9 | /notifications | 5 |
| /game | 23 | /auth | 6 | /uploads | 5 |
| /quizzes | 21 | /devii | 5 | /docs | 4 |
| /api | 20 | /posts | 4 | everything else | 36 |
## 05 Route fan-out
Source: `devplacepy/responses.py, schemas/, docs_api/, services/devii/actions/`
One handler serves four consumers. `respond()` feeds the same context dict to a Pydantic model and to a Jinja template, so a key missing from the `*Out` schema is silently absent from JSON while still rendering in HTML. The agent catalog and the docs registry are separate declarations that must be kept in step by hand.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
form["models.py<br/>Pydantic form model via Annotated Form"]
guard["guard: get_current_user<br/>require_user / require_admin"]
handler["router handler"]
helpers["database/ batch helpers<br/>get_users_by_uids, build_pagination"]
ctx["context dict"]
neg{"wants_json(request)"}
json["model.model_validate<br/>JSONResponse"]
html["templates.TemplateResponse<br/>extends base.html"]
action["services/devii/actions catalog<br/>Action, requires_auth mirrors guard"]
docs["docs_api endpoint()<br/>params and sample_response"]
seo["seo.py base_seo_context<br/>JSON-LD, sitemap entry"]
form --> handler
guard --> handler
handler --> helpers --> ctx --> neg
neg -->|"json accepted"| json
neg -->|"otherwise"| html
handler -.->|"same capability"| action
handler -.->|"same contract"| docs
html -.-> seo
```
wants_json is true when the request content-type starts with application/json, or Accept contains application/json and does not contain text/html. Registry sizes: 318 agent actions, 259 documented endpoint entries across 20 API groups, 122 prose docs pages.
## 06 Identity resolution
Source: `devplacepy/utils/auth.py`
A single resolver serves browsers, API clients, the devRant compatibility layer and issued access tokens. The result is memoised on `request.state` for the request and in a process TTL cache keyed by credential.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
start(["get_current_user"])
cached{"request.state._auth_user set?"}
sess{"session cookie<br/>64 hex chars"}
xapi{"X-API-KEY header"}
bearer{"Authorization: Bearer"}
basic{"Authorization: Basic"}
triple["try in order:<br/>users.api_key,<br/>devrant token 40 hex,<br/>access token 64 chars"]
user(["user dict"])
guest(["None, guest"])
gu["require_user: 303 to /"]
ga["require_admin: redirect to /feed"]
start --> cached
cached -->|yes| user
cached -->|no| sess
sess -->|match| user
sess -->|no| xapi
xapi -->|present| triple
triple -->|match| user
triple -->|no match| bearer
xapi -->|absent| bearer
bearer -->|present| triple
bearer -->|absent| basic
basic -->|"username-or-email:password, pbkdf2_sha256"| user
basic -->|no| guest
guest --> gu
user --> ga
```
Roles are stored capitalized as Admin or Member and tested through the is_admin global. Admin seniority is enforced per user mutation in routers/admin/users.py.
## 07 Background service fleet
Source: `devplacepy/services/manager.py, base.py, jobs/base.py`
27 singletons registered at boot, supervised only by the worker that won the service lock. Sixteen are long-lived loops on `BaseService`; eleven are queue consumers on `JobService`, which adds retention, concurrency and timeout settings on top and drains rows from the shared `jobs` table.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart LR
mgr["ServiceManager<br/>supervise, shutdown_all"]
settings[("site_settings<br/>enabled flag and interval per service")]
jobs[("jobs table")]
subgraph base["BaseService loops - 16"]
b1["NewsService"]
b2["BotsService"]
b3["GatewayService"]
b4["DeviiService"]
b5["PubSubService"]
b6["NotificationRelayService"]
b7["LiveViewRelayService"]
b8["PresenceRelayService"]
b9["IssueTrackerService"]
b10["ContainerService"]
b11["WorkspaceService"]
b12["XmlrpcService"]
b13["AuditService"]
b14["PushService"]
b15["TelegramService"]
b16["TelegramOutboxService"]
end
subgraph job["JobService consumers - 11"]
j1["ZipService"]
j2["ForkService"]
j3["SeoService"]
j4["SeoMetaService"]
j5["AwardService"]
j6["BackupService"]
j7["DbApiJobService"]
j8["DeepsearchService"]
j9["IsslopService"]
j10["IssueCreateService"]
j11["PlanningReportService"]
end
mgr --> base
mgr --> job
settings --> mgr
job --> jobs
```
Separate from the fleet, services/background.py offers a fire and forget queue per worker for audit writes, XP awards and notifications; when no consumer runs, as in tests, the callable executes inline so ordering stays deterministic.
## 08 Data layer
Source: `devplacepy/database/, devplacepy/config.py`
One SQLite file reached through `dataset`, called synchronously from async handlers by design. 102 tables exist in the live schema; 52 of them carry the soft delete pair and are restorable as one event from the admin trash.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
db[("data/devplace.db")]
subgraph g1["Identity and access - 14"]
t1["users, sessions, password_resets<br/>access_tokens, devrant_tokens<br/>user_relations, follows<br/>notification_preferences, user_customizations<br/>push_registration, email_accounts<br/>telegram_links, telegram_pairings, telegram_outbox"]
end
subgraph g2["Content and engagement - 16"]
t2["posts, comments, gists<br/>projects, project_files, project_forks<br/>attachments, news, news_images, news_sync<br/>polls, poll_options, poll_votes<br/>votes, reactions, bookmarks"]
end
subgraph g3["Reputation - 5"]
t3["awards, badges, award_usage<br/>user_activity, user_activity_seen"]
end
subgraph g4["Code Farm - 9"]
t4["game_farms, game_plots, game_quests<br/>game_cosmetics, game_market_ticks<br/>game_steals, game_treasury<br/>game_eras, game_era_results"]
end
subgraph g5["Quizzes - 5"]
t5["quizzes, quiz_questions, quiz_options<br/>quiz_attempts, quiz_answers"]
end
subgraph g6["Live delivery - 3"]
t6["messages, notifications, ws_tickets"]
end
subgraph g7["Devii - 8"]
t7["devii_conversations, devii_turns<br/>devii_tasks, devii_task_runs<br/>devii_lessons, devii_virtual_tools<br/>devii_behavior, devii_usage_ledger"]
end
subgraph g8["AI gateway - 6"]
t8["gateway_providers, gateway_models<br/>gateway_usage_ledger<br/>gateway_quota_rules, gateway_quota_resets<br/>gateway_concurrency_samples"]
end
subgraph g9["Jobs, tools and usage - 20"]
t9["jobs, backups, backup_schedules<br/>deepsearch_sessions, deepsearch_messages, deepsearch_url_cache<br/>isslop_analyses, isslop_reports, isslop_events<br/>isslop_dom_results, isslop_file_results, isslop_image_results<br/>seo_metadata, seo_usage<br/>issue_tickets, issue_comment_authors, issue_usage<br/>correction_usage, modifier_usage, news_usage"]
end
subgraph g10["Containers - 9"]
t10["instances, instance_events, instance_metrics<br/>tunnels, workspace_flags, workspace_quota_rules<br/>builds, dockerfiles, dockerfile_versions"]
end
subgraph g11["Platform state - 7"]
t11["site_settings, cache_state, service_state<br/>audit_log, audit_log_links<br/>visit_stats_hourly, visit_unique_slots"]
end
db --> g1
db --> g2
db --> g3
db --> g4
db --> g5
db --> g6
db --> g7
db --> g8
db --> g9
db --> g10
db --> g11
```
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart LR
root["DEVPLACE_DATA_DIR<br/>default repo/data"]
blobs["uploads, attachments, project_files<br/>sharded xx/yy on the uuid7 random tail"]
jobsdir["zips, zip_staging, fork_staging<br/>backups, backup_staging"]
reports["seo_reports, planning_reports<br/>dbapi, deepsearch, deepsearch_chroma"]
isslop["isslop, isslop_workspaces<br/>isslop_runs, isslop_media"]
ws["container_workspaces, workspace_state"]
keys["keys VAPID, bot, locks"]
dbs["devplace.db, devii_tasks.db, devii_lessons.db"]
root --> blobs
root --> jobsdir
root --> reports
root --> isslop
root --> ws
root --> keys
root --> dbs
```
config.DATA_PATHS registers 23 directories and ensure_data_dirs creates the whole tree before any write. Uploads live under data/uploads but are served at the unchanged /static/uploads URL.
## 09 Content rendering
Source: `devplacepy/rendering.py, static/js/ContentRenderer.js`
Two pipelines with a deliberate split: anything that exists at request time is rendered on the server for SEO, and the client pipeline is reserved for content that does not exist yet. Each has its own XSS control at a different point.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
subgraph server["Server, rendering.py, lru_cache"]
a1["raw text"]
a2["normalize dashes to hyphen"]
a3["emoji shortcodes, 4869 names"]
a4["mistune GFM, escape=True"]
a5["media pass: bare URLs to embeds, mentions to links"]
a6["mask emails on rendered text nodes"]
a7["render_content / render_title in template"]
a8["ContentEnhancer adds highlighting and copy buttons"]
a1 --> a2 --> a3 --> a4 --> a5 --> a6 --> a7 --> a8
end
subgraph client["Client, ContentRenderer.js"]
b1["live text: comments, DM bubbles, Devii, DeepSearch"]
b2["marked"]
b3["DOMPurify.sanitize, fail closed"]
b4["highlight.js"]
b5["media and autolink pass"]
b1 --> b2 --> b3 --> b4 --> b5
end
```
The server escapes at the markdown step, the client sanitizes after parsing. Server rendered content must not carry data-render, or both pipelines run over it.
## 10 AI plane
Source: `devplacepy/routers/openai_gateway.py, services/openai_gateway/, services/devii/`
Every model call in the product, internal or external, leaves through one gateway, which is where routing, quota and cost attribution live. Internal consumers call it over loopback HTTP rather than importing a client, so their spend lands in the same ledger.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
ext["external client<br/>/openai/v1/*"]
subgraph consumers["Internal consumers via INTERNAL_GATEWAY_URL"]
c1["services/devii"]
c2["services/correction<br/>AI correction and modifier"]
c3["services/news"]
c4["services/bot"]
c5["services/deepsearch + jobs/deepsearch"]
c6["services/dbapi nl2sql"]
c7["services/gitea enhance + planning"]
c8["jobs/isslop"]
end
gw["gateway.py<br/>routing.py, quota.py, usage.py<br/>reliability.py, vision.py"]
ledger[("gateway_usage_ledger<br/>gateway_quota_rules")]
stealth["stealth_async_client<br/>curl_cffi Chrome 146 fingerprint"]
up["upstream provider"]
subgraph devii["Devii assistant"]
d1["registry.py CATALOG<br/>318 tools"]
d2["role gating<br/>guest 108, member 246<br/>admin 312, primary 318"]
d3["45 tools behind CONFIRM_REQUIRED"]
d4["session, tasks, lessons<br/>virtual tools, behavior"]
d5["surfaces: /devii ws, Telegram, CLI"]
end
ext --> gw
consumers --> gw
gw --> ledger
gw --> stealth --> up
d1 --> d2 --> d3
devii --> c1
d4 --> d1
d5 --> d4
```
Cleartext loopback calls are forced to HTTP/1.1 in curl_transport.http_version_for, because the Chrome impersonation profile would otherwise negotiate HTTP/2 against an HTTP/1.1 only uvicorn.
## 11 Containers, workspaces and ingress
Source: `devplacepy/services/containers/, routers/projects/containers/, routers/proxy.py, routers/tunnel.py`
One shared image serves every instance. Reconciliation is a loop that compares desired state in the database against what the Docker daemon actually reports, and there are two independent ways in from the outside: a path prefix and a hostname.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
ui["/projects/{slug}/containers<br/>instances, schedules, workspace"]
api["services/containers/api.py"]
store[("instances, instance_events<br/>instance_metrics, tunnels<br/>workspace_flags, workspace_quota_rules")]
svc["ContainerService<br/>reconcile desired vs docker ps"]
wsvc["WorkspaceService<br/>provision, certs, quota, tunnels"]
backend["backend/docker_cli.py<br/>backend/fake.py for tests"]
image["single shared image ppy:latest"]
inst["running instance<br/>rootless workflow, aptroot"]
wsdir["data/container_workspaces/{uid}"]
proxy["/p/{slug}<br/>routers/proxy.py, relays headers verbatim"]
tunnel["host based tunnel<br/>TunnelDispatchMiddleware to routers/tunnel.py"]
code["workspace editor over websocket<br/>/projects/{slug}/containers/instances/{uid}/code"]
exec["terminal over websocket<br/>.../exec/ws"]
ui --> api --> store
svc --> store
wsvc --> store
api --> backend
svc --> backend
wsvc --> wsdir
backend --> image --> inst
proxy --> inst
tunnel --> inst
code --> inst
exec --> inst
```
Access uses stricter predicates than the rest of the product: owns_instance, can_view_project_containers, can_view_instance, can_manage_instance. The primary administrator sees and manages every container; any other admin can only view others on public projects and manage instances they own.
## 12 Real time plane
Source: `routers/*.py websocket handlers, services relays, nginx.conf.template`
Twelve WebSocket endpoints, each of which needs its own nginx upgrade location because the catch-all location strips upgrade headers. Fan-out to connected sockets goes through in-process hubs driven by relay services on the lock-owning worker.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart LR
subgraph sockets["WebSocket endpoints"]
s1["/devii/ws"]
s2["/messages/ws"]
s3["/pubsub/ws"]
s4["/tools/seo/{uid}/ws"]
s5["/tools/deepsearch/{uid}/ws"]
s6["/tools/deepsearch/{uid}/chat"]
s7["/dbapi/query/{uid}/ws"]
s8[".../containers/instances/{uid}/exec/ws"]
s9[".../containers/instances/{uid}/code"]
s10[".../code/{path}"]
s11["/p/{slug}"]
s12["/p/{slug}/{path}"]
end
subgraph relays["Relay services"]
r1["NotificationRelayService"]
r2["LiveViewRelayService"]
r3["PresenceRelayService<br/>track limit 500, online limit 30"]
r4["PubSubService"]
end
hubs["messaging/hub.py, pubsub/hub.py<br/>devii/hub.py"]
tick[("ws_tickets<br/>expiring auth tickets")]
s2 --> hubs
s3 --> hubs
s1 --> hubs
relays --> hubs
tick --> s2
```
Presence is authoritative from the relay: PRESENCE_TRACK_LIMIT sets the tracked online set, PRESENCE_ONLINE_LIMIT only caps how many the feed panel displays, and PRESENCE_ONLINE_MARGIN_SECONDS provides hysteresis at the boundary.
## 13 Frontend
Source: `devplacepy/static/, devplacepy/templates/`
No framework and no package manager. 131 ES6 modules, one class per file, hung off a single global `app`; 33 of them are custom elements. 217 Jinja templates, of which 107 are the docs site.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart TB
base["templates/base.html<br/>extra_head for CSS, extra_js for JS"]
partials["shared partials<br/>_avatar_link, _user_link, _comment_section<br/>_post_card, _pagination, _sidebar_search"]
appjs["static/js/Application.js<br/>instantiated once as app"]
subgraph modules["131 ES6 modules"]
u1["utilities<br/>Http, Poller, JobPoller<br/>OptimisticAction, FloatingWindow, ScrollMemory"]
u2["33 custom elements<br/>components/ with dp- prefix"]
u3["feature modules<br/>chat/, devii/, autoload/"]
end
css["49 stylesheets<br/>variables.css tokens, per page files"]
vers["static_url in Jinja, assetUrl in JS<br/>/static/v{boot ts}/, immutable for a year"]
cust["per user CSS and JS injection<br/>custom_css_tag, custom_js_tag"]
base --> partials
base --> appjs --> modules
base --> css
base --> cust
vers --> css
vers --> modules
```
Per user customizations are configured only through Devii, stored in user_customizations, scoped globally or per matched route template, and run solely in that owner's own browser sessions.
## 14 Test and delivery topology
Source: `tests/, pyproject.toml, .gitea/workflows/test.yaml`
Three tiers separated by what they exercise, decided by fixtures, run serially in a single process against one uvicorn subprocess on port 10501 with a temporary database.
```mermaid
%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%%
flowchart LR
unit["tests/unit - 140 modules<br/>mirrors source module path<br/>local_db or no fixture"]
api["tests/api - 221 modules<br/>mirrors URL path<br/>app_server, seeded_db"]
e2e["tests/e2e - 131 modules<br/>mirrors URL path<br/>page, alice, bob"]
srv["uvicorn subprocess :10501<br/>temp SQLite, own DATA_DIR<br/>DEVPLACE_DISABLE_SERVICES=1"]
pw["Playwright Chromium<br/>session scoped context"]
ci["Gitea Actions on push and PR to master<br/>full suite under coverage"]
prod["promotion to production<br/>docker compose"]
unit --> ci
api --> srv --> ci
e2e --> srv
e2e --> pw --> ci
ci --> prod
```
pytest-xdist is not a dependency and -n is rejected centrally, so no tier can be parallelised by accident.
## 15 Findings
Source: `measured against the repository documentation`
The structure holds up: the router tree mirrors the URL tree, the test tree mirrors both, every runtime path is registered in one place, and every model call has a single exit. The items below are the places where the running code and the written record have drifted apart, or where a structure exists that nothing currently uses. None of them is a functional defect.
### Middleware order in the documentation is stale
The root CLAUDE.md states that response_timing is the outermost middleware. Measured from app.user_middleware it is third, behind GZipMiddleware and TunnelDispatchMiddleware, both added after it. The consequence is minor but real: X-Response-Time excludes compression time and excludes tunnel-host dispatch entirely.
### Two catalogue counts have drifted
CLAUDE.md cites around 2882 tests and an events.md catalogue of 288 keys. Measured now: 3073 functions named test_ across 492 modules, and 312 distinct dotted event keys in events.md. Both are undercounts in the docs, not missing implementation.
### instance_schedules is declared soft-deletable but has no table
It appears in database.SOFT_DELETE_TABLES and in the routers under /projects/{slug}/containers/schedules, but no such table exists in the live schema. dataset creates it lazily on first insert, so this is correct only for as long as every read path tolerates the table being absent.
### Three legacy container tables still exist
builds, dockerfiles and dockerfile_versions are present in the live database. The project replaced per-project images with the single shared ppy:latest image and ships devplace containers prune-builds as the one-time cleanup for exactly these rows. The tables are still carried.
### Five duplicate OpenAPI operation IDs
Generating the schema warns on editor_proxy twice in routers/projects/containers/workspace.py, passthrough in routers/openai_gateway.py, and proxy_http twice in routers/proxy.py. These are catch-all routes registered for several methods, so a generated client would collide on those names.
### The one deliberate asymmetry is documented and intentional
routers/__init__.py holds nothing but the attribution line; unlike every nested router package, the top level does not aggregate. main.py performs all 38 include_router calls directly, which keeps prefix ownership in a single readable block.
---
Sources: devplacepy/main.py, the imported FastAPI app object, data/devplace.db sqlite_master, and the working tree at HEAD 192df12b with uncommitted local modifications present. Counts exclude __pycache__ and the virtual environment.
PODMAN.md
-78
View File
@@ -1,78 +0,0 @@
# Podman deployment (rootless, no docker daemon)
Hosts without a docker daemon (like `devplace`) run the stack with the Python
`podman-compose` plus the `docker-compose.podman.yml` overlay. The base
`docker-compose.yml` is unchanged and stays the file rootful `docker compose`
uses; the overlay only adjusts what rootless podman needs.
## Commands
Always go through the make targets - they pin both `-f` files. A bare
`podman-compose up -d` drops the overlay and recreates a crashing app (see
"Why the overlay exists").
```bash
make podman-build # build images (only when pyproject/Dockerfiles change)
make podman-up # start (also warns if user linger is off)
make podman-reload # restart workers on new code (bind-mounted, no rebuild)
make podman-ps # container status
make podman-logs # tail output
make podman-down # stop
```
Open `http://<host>:${PORT}` (default 10500, published on 127.0.0.1 for the
front proxy). Source-only changes need just `make podman-reload`.
## Why the overlay exists
The base file runs the app as `user: 1000:1000`, which is correct for rootful
docker (container uid 1000 == host uid 1000). Under **rootless** podman the
default userns instead maps container uid 1000 to a subordinate host uid
(100999), which cannot write the bind-mounted checkout (owned by host uid
1000) - uvicorn dies on startup with
`PermissionError: ... '/app/data/locks/devplace-init.lock'`, the app never
turns healthy, and nginx (which waits on `service_healthy`) never starts.
The overlay sets `user: "0:0"` for the app: container root maps back to the
invoking host user under rootless, restoring write access while staying
unprivileged on the host. (`userns_mode: keep-id` would also fix the mapping
but podman rejects `--userns` together with `--pod`, and podman-compose puts
all services in one pod.)
## One compose tool only
Use the Python `podman-compose` (`/usr/bin/podman-compose`) for this stack,
never mix it with native `podman compose` in the same project directory. They
use different container names (`devplace_app_1` vs `devplace-app-1`) and would
create a second stack fighting over port 10500.
## Surviving reboots
Both services use `restart: unless-stopped`, but rootless containers only
restart after a reboot if the host user's systemd linger is enabled:
```bash
loginctl show-user "$(id -un)" | grep Linger # want: Linger=yes
sudo loginctl enable-linger "$(id -un)" # fix if off
```
`make podman-up` prints a warning when linger is off.
## Known differences from docker
- The admin-only **Container Manager** needs the docker socket, which does not
exist on podman hosts. The `[containers] docker ps failed: [Errno 2]` lines
in the app log are expected noise, not an error; serving the site is
unaffected.
- `make docker-attach` / the containers overlay do not apply here.
## Troubleshooting
- `PermissionError ... /app/data/...` on boot: the app was (re)created
without the overlay. Run `make podman-up` (it recreates with the fix).
- `up` hangs with no output: it is waiting on the app healthcheck. In another
shell run `podman logs devplace_app_1` - a crash loop there is the cause.
- `Error: --userns and --pod cannot be set together`: a `userns_mode`
override is in play; remove it, this stack relies on `user: "0:0"`.
- Port 10500 already in use: a second stack (other compose tool, or bare-metal
`make prod`) is running. `podman ps -a` plus `ss -tlnp` show the conflict.
devplacepy/services/game/CLAUDE.md
+1
View File
@@ -107,6 +107,7 @@ Continuous Delivery used to be a one-way lock: the harvested crop was replanted
- **The flag is read AFTER `clear_plot` wins.** `_try_autoreplant` re-reads the plot, so a toggle either landed before the clear (replant skipped) or finds an empty plot (`set_plot_stop`'s `WHERE crop_key != ''` fails, loud error). Never read the flag from the pre-clear plot dict and never reset it inside `clear_plot`.
- **Every writer that puts a crop on a plot resets it:** the `plant()` and `_try_autoreplant` claim SETs write `stop_after_harvest = 0`, and `prestige()`'s plot reset writes it too. A stale `1` on an empty plot is harmless and is never serialized (`serialize_plot` gates `stop_after_harvest` on owner + Continuous Delivery + a crop present).
- Column is `COALESCE`-read (`_lvl(plot, ...)`); existing rows are NULL.
- **Auto-replant is not a manual plant:** `_try_autoreplant` does not call `advance_quests("plant")`, writes no `game.plant` audit event and is not reported in the harvest result. This is documented for automation authors in `code-farm.html` ("Continuous Delivery and crop rotation (for bots)"); change both together.
- `_crop_lock(farm, crop)` is the single lock predicate used by `plant()` and `_try_autoreplant`, so autoreplant no longer replants a crop that is locked (an Era crop after its Era ends). Saturation-based skipping was deliberately NOT built: at prestige 50+ (the only owners of Continuous Delivery) even a 0.4 market factor pays 11x or more the planting cost for every crop, so skipping only forfeits coins, XP and quest progress.
### 4. Secondary leaderboards (`economy.py` scoring + `store/farm.py`)
devplacepy/templates/docs/code-farm.html
+123
View File
@@ -586,6 +586,129 @@ upgrade is owned and something was ready.
**Era:** `era_active`, `era_name`, `era_coins`, `era_harvests`.
### Continuous Delivery and crop rotation (for bots)
Continuous Delivery (Mastery upgrade `autoreplant`, 3 points, prestige 50 or higher) changes what an
empty plot means, so a bot that assumes "harvest, then plant" must account for it. This section is
the exact behavior, measured from the implementation.
**What auto-replant does, step by step.** It runs in two places and nowhere else: right after a
manual `POST /game/harvest`, and inside the CI Bot's auto-harvest during a read of your own farm.
After the harvest has been paid, the server looks at that plot again and replants it only if all of
these hold:
1. The plot is empty (the harvest cleared it and nothing else planted it in between).
2. The plot's pause flag is not set (see below).
3. The harvested crop is still unlocked for you (level, Mastery and Era checks, the same as
`POST /game/plant`).
4. You can pay the crop's current planting cost. Coins are charged atomically first, so a refused
charge leaves the plot empty and costs nothing.
If it replants, it plants **the same crop that was just harvested**. It never picks a better crop
and it does **not** look at Market Saturation: a saturated crop is replanted exactly like any other.
The new planting starts at the moment of the harvest, uses your current CI tier, perks and Legacy
speed for its build time, and gets its own fresh golden-build roll.
Side effects that differ from a manual `POST /game/plant`:
| | Manual plant | Auto-replant |
|---|---|---|
| Charges the planting cost | yes | yes |
| Advances the daily "plant" quest | **yes** | **no** |
| Writes a `game.plant` audit event | yes | no |
| Reported in the harvest response | n/a | **no**, read the plot from the returned `farm` |
The harvest itself (coins, XP, the "harvest" and "earn" quests, the weekly harvest count) is
identical either way. If you farm the daily "plant" quest, plant at least once by hand.
**The one-shot pause: `POST /game/plot/stop`.** Before this existed, the only way to change what a
plot grows was a Refactor. Now you can pause the replant for one plot, one harvest at a time:
| Field | Type | Meaning |
|---|---|---|
| `slot` | integer, 0-64 | Plot to change. |
| `stop` | boolean | `true` skips the replant after the next harvest of this plot. `false` cancels a pending pause. |
Returns the usual `{"ok": true, "farm": {...}}`. Rules, all enforced server side:
- It needs Continuous Delivery. Without it: **400** `Continuous Delivery is not unlocked.`
- The plot must currently hold a crop (growing or ready). On an empty plot: **400**
`That plot is empty.` This is also the answer when the harvest wins a race against your call (see
below), so treat it as "go read the state", not as a failure.
- It is **one-shot and per plot**. It applies to the next harvest of that plot only, and each plot
needs its own call. Setting it again on a plot that already has it, or clearing it on a plot
that does not, succeeds and changes nothing.
- It is cleared automatically the next time anything puts a crop on that plot: your manual
`POST /game/plant`, an auto-replant, or a Refactor. You never have to clear it by hand.
- It costs nothing and is not refundable or limited, but it is a mutating call, so it counts
against the rate limit like any other POST.
**What you can read.** Each plot in your own farm carries two booleans, always present:
- `can_stop_replant` - true on a plot that holds a crop when you own Continuous Delivery. False on
an empty plot, without the upgrade, and on every plot of someone else's farm.
- `stop_after_harvest` - true while the pause is set on a plot that still holds a crop. It is
never true on an empty plot, on a plot of another farm, or without the upgrade, so a stale flag
can never be observed.
**Rotation recipe.** To change the crop on a plot:
1. Call `POST /game/plot/stop` with `stop=true` while the plot is growing. Do it well before
`ready_at`, because with the CI Bot the harvest can happen during any read of your farm once the
build is ready.
2. Wait until `ready_at` has passed, then harvest (or let the CI Bot do it on your next read).
3. The plot is now empty. Plant what you want with `POST /game/plant`. The pause is cleared by that
plant.
Between the harvest and your plant the plot sits idle, so every rotation costs you the time it
takes your bot to notice. Without the CI Bot that is up to one poll interval. A bot that polls
every 60 seconds loses about half a minute per rotation on average, which is why rotating only pays
off when the new crop beats the old one by more than that idle time.
**Race behavior (verified with concurrent processes).** A pause call and a harvest of the same plot
at the same instant always resolve one of two ways, never a third: either the pause lands first
(HTTP 200, the plot ends empty) or the harvest lands first (the plot is already replanted, and your
call gets the 400 `That plot is empty.`). We ran 150 rounds with the two calls in separate
processes: 87 pause-first, 63 harvest-first, 0 inconsistent. After a 400 on this endpoint, re-read
`GET /game/state` and look at the plot instead of retrying blindly.
**Do you need to rotate to avoid Market Saturation?** Usually not, and the numbers say so. A
saturated crop pays at worst 40 percent of its normal value. At prestige 50 the prestige multiplier
is 13.5, so even with no perks, no golden build and the 40 percent floor, every crop pays at least
about 11 times its planting cost (shell about +54 coins on a 5 coin plant, kernel about +18,240 on
1,200). Skipping a replant to dodge saturation therefore usually earns less than replanting into
it, because an idle plot earns nothing and also forgoes the XP, quest progress and weekly harvest
count of the cycle. Rotate when a **different crop pays more per second**, not merely to avoid the
`saturated` label. These figures are computed from the published formulas, not measured on a live
farm, and they change if the saturation tiers or prestige multiplier are rebalanced.
**Edge cases a bot should handle.**
- An auto-replant can be refused (cost, lock). The plot then just stays empty, so a loop that
plants every `state == "empty"` plot, like the script below, already covers it.
- An Era crop stops being plantable when its Era ends. Auto-replant now respects that and leaves the
plot empty instead of replanting a locked crop.
- The upgrade itself is permanent: Mastery points are never refunded and there is no switch that
turns Continuous Delivery off for the whole farm. The per-plot pause is the only control.
- A Refactor clears every plot, does **not** auto-replant, and clears all pauses. Plant again after
it.
A rotation helper that moves every plot to one target crop, using only the standard library:
```python
def rotate_to(call, wanted):
farm = call("GET", "/game/state")["farm"]
for plot in farm["plots"]:
if plot["state"] == "empty":
call("POST", "/game/plant", {"slot": plot["slot"], "crop": wanted})
elif plot["crop_key"] != wanted and plot["can_stop_replant"] and not plot["stop_after_harvest"]:
call("POST", "/game/plot/stop", {"slot": plot["slot"], "stop": "true"})
```
Call it on every poll. A plot growing the wrong crop gets paused once, its harvest leaves it empty,
and the next poll plants `wanted` there. Plots already growing `wanted` are left alone, and the
auto-replant keeps them running.
### A complete automated farmer
This script plays the core loop on a schedule: it claims the daily bonus and any finished quests,
docker-compose.yml.bak-20261008
-56
View File
@@ -1,56 +0,0 @@
name: devplace
services:
app:
build:
context: .
dockerfile: Dockerfile
restart: unless-stopped
env_file:
- .env
user: "${DEVPLACE_UID:-1000}:${DEVPLACE_GID:-1000}"
working_dir: /app
environment:
HOME: /app
PYTHONDONTWRITEBYTECODE: "1"
# Pinned regardless of .env: the container always binds 10500 internally
# (Dockerfile CMD, nginx upstream). DEVPLACE_PORT is the bare-metal
# `make dev`/`make prod` knob; the externally reachable port is `PORT`.
DEVPLACE_PORT: "10500"
volumes:
- .:/app
expose:
- "10500"
networks:
- appnet
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:10500/"]
interval: 30s
timeout: 10s
retries: 3
start_period: 120s
start_interval: 2s
nginx:
build:
context: .
dockerfile: nginx/Dockerfile
restart: unless-stopped
ports:
- "127.0.0.1:${PORT:-10500}:80"
environment:
NGINX_CACHE_ENABLED: "${NGINX_CACHE_ENABLED:-false}"
NGINX_CACHE_MAX_SIZE: "${NGINX_CACHE_MAX_SIZE:-1g}"
NGINX_MAX_BODY_SIZE: "${NGINX_MAX_BODY_SIZE:-50m}"
volumes:
- ./devplacepy/static:/app/static:ro
- ${DEVPLACE_DATA_DIR:-./data}/uploads:/data/uploads:ro
depends_on:
app:
condition: service_healthy
networks:
- appnet
networks:
appnet:
driver: bridge
dptodo.md
-1272
View File
File diff suppressed because it is too large Load Diff
fpost.md
-70
View File
@@ -1,70 +0,0 @@
# DevPlace Changelog — Last 24 Hours
## AI Gateway — Non-OpenAI Upstreams & Client Model Passthrough
The AI gateway now supports routing to non-OpenAI upstream providers. The gateway can target any compatible API endpoint, and client-specified model names are passed through to the upstream instead of being forced to a fixed model. This opens the door for self-hosted providers and alternative APIs.
**Files:** `services/openai_gateway/gateway.py`, `service.py`, `thinking.py`, `vision.py`
## AI Gateway — Trust Only Upstream X-Gateway-Model Header
The gateway now trusts only the `X-Gateway-Model` header from the upstream response when determining which model served a request. This prevents spoofing and ensures usage attribution is accurate.
**Files:** `services/openai_gateway/gateway.py`, `usage.py`
## Backup Service — Remote Offload to Hetzner Storage Box
Completed backups can now be automatically offloaded to a remote Hetzner Storage Box. The offload module handles the transfer, and the backup service orchestrates the full lifecycle: create, verify, offload, and prune.
**Files:** `services/backup/offload.py`, `service.py`, `store.py`
## Container Manager — Orphan Blob Fix & System Prune
Fixed container sync races that leaked orphan blobs (files on disk with no database reference). Added a new `devplace system prune` CLI command that safely removes orphan blobs, sweeps unreferenced container workspace dirs, and prunes expired fork/zip job rows. The command supports `--dry-run` for preview.
**Files:** `cli/system.py`, `services/containers/store.py`, `attachments.py`, `project_files.py`
## Container Manager — Two-Phase Plan/Execute Report
The `system prune` CLI command now uses a two-phase plan/execute report: first it shows what would be deleted (plan), then executes and reports what was actually removed.
**Files:** `cli/system.py`, `cli/containers.py`
## Post Page — Happy 404, Featured/Related Sidebars, Next-Post Nav
The post page now includes:
- A "Happy 404" page for missing posts
- Featured posts sidebar
- Related posts sidebar
- Next-post navigation
**Files:** `happy404.py`, `routers/posts.py`, `templates/post.html`, `database/content.py`
## Static Asset Cache-Busting — Unified with Auto-Bumped App Version
Static asset URLs now use the auto-bumped app version for cache-busting instead of a separate timestamp. This ensures assets are invalidated whenever the app version changes, and eliminates the need for manual cache-bust updates.
**Files:** `config.py`, `templates/docs/static-caching.html`, `nginx/nginx.conf.template`
## Automatic Patch-Version Bumping via Git Pre-Commit Hook
A new `.githooks/pre-commit` hook automatically bumps the patch version in `pyproject.toml` on every commit. The hook increments patch (e.g. `1.0.0` → `1.0.1`) and stages the change. A hand-set major/minor bump in the same commit is left untouched. The hook defers during merges and never blocks a commit.
**Files:** `.githooks/pre-commit`, `Makefile`, `CLAUDE.md`
## Test Infrastructure — Shared `run_async` Helper
Provision tests now use the shared `run_async` test helper instead of bare `asyncio.run` calls, improving consistency and error handling across the test suite.
**Files:** `tests/unit/services/containers/workspace/provision.py`
## Documentation Updates
- Updated `README.md` with new feature documentation
- Updated `CLAUDE.md` with new CLI commands and architecture notes
- Updated docs pages: production nginx, static caching, backups, architecture backend
- Updated nested `CLAUDE.md` files for gateway, containers, backup, and posts subsystems
---
*Generated from git log — all commits from the last 24 hours.*
gaplord.md
-405
View File
@@ -1,405 +0,0 @@
# gaplord: close every documented gap
Plan and result in one file. The plan was written first, from a read of the code behind each gap. Every item is then worked in order and its **Status** and **Result** lines are filled in when it is done, with the measurement or test that proves it. Nothing is checked off without evidence.
Written 2026-10-16. Scope: every entry under a `Gaps` heading in `devplacepy/services/{nodes,sync,community,skills}/CLAUDE.md` and `nodes/extension/README.md`, except the plain-text community passwords (explicitly excluded by the owner).
Legend: `[ ]` open, `[~]` in progress, `[x]` done and proven, `[!]` cannot be closed by code (reason stated, nothing faked).
## Ground rules for this work
- No fiction. A gap is only closed when code exists, a test or measurement proves it, and the nested `CLAUDE.md` says what the system does today. Where a gap needs something this machine does not have, it is marked `[!]` with the exact missing input and what was built so the measurement happens by itself later.
- Specification first. Every new protocol field, limit, refusal, reason and kind is declared in `spec.py` (nodes) or `services/sync/spec.py` and read from there everywhere else.
- Caps on everything new (rows, bytes, intervals, retries). Machines hold, humans decide: nothing here bans a member or deletes a ledger row on its own.
- One fan-out per feature: data layer, schema (`*Out`), route, view, Devii tool, `docs_api`, docs prose, privacy page where data categories change, README, nested `CLAUDE.md`, tests in the matching tier.
- Em dash is forbidden in every touched file. No comments or docstrings in Python source.
- The dev server on port 10500 runs `--reload` against the live database. Files are therefore written whole and import-checked right after each edit, and no command in this work reads or writes `data/`. All scripts use scratch databases (`DEVPLACE_DATABASE_URL`, `DEVPLACE_DATA_DIR`).
- Final gate: `make test` (unit, api, e2e), plus `ruff`/`pyflakes` on touched files, `go vet` + `go test`, `node --test`.
## Gap inventory (source of truth for this plan)
| Id | Gap (as written in the docs) | Source |
|----|------------------------------|--------|
| G-N1 | Reddit viability unmeasured from a residential line | nodes |
| G-N2 | `country`, `asn`, `asn_class` never filled, datacenter exclusion (D6) not enforced | nodes |
| G-N3a | `last_seen_at` written on connect only | nodes |
| G-N3b | Socket counters per process | nodes |
| G-N3c | Fabric cache per process | nodes |
| G-N3d | Circuit breaker state in memory (and no per-destination admin view) | nodes |
| G-N3e | Same-process fast path not built, callers poll | nodes |
| G-N4 | Fetch-mode results not cross-checked (sampled redundancy) | nodes |
| G-N5 | DeepSearch skips hostile domains with a static list, no routing through `/nodes/internal/fetch` | nodes |
| G-N6a | Go client has no battery-aware pause (`pause_on_battery` ignored) | nodes |
| G-N6b | Go client treats a timed suspension as permanent until restarted | nodes |
| G-N6c | Extension enforces its own local hours in the browser clock, not the server timezone | nodes |
| G-N7 | Not measured: Reddit/X success per address class, WebSocket idle timeouts through the public hostnames, socket capacity of one worker, 24 hour soak | nodes |
| G-N8 | Launch blocked on a recorded legal decision | nodes |
| G-E1 | Extension has no icons | extension |
| G-E2 | `welcome` frame carries no device label | extension |
| G-E3 | Extension not tested in a real Chrome or Firefox session | extension |
| G-S1 | No export endpoint, sync is inbound only | sync |
| G-S2 | Votes, reactions, follows, attachments, polls and project links not synced | sync |
| G-S3 | A comment's post and parent must arrive earlier, no deferred resolution queue | sync |
| G-S4 | A local deletion is not communicated to the origin; the origin is rejected forever with `purged_locally` | sync |
| G-S5 | `min_comments` only sees comments collected in the same run | sync |
| G-S6 | A clone renamed by an administrator no longer matches its alias | sync |
| G-S7 | A username taken between alias generation and clone creation diverges from the rewritten mentions | sync |
| G-K1 | Community page does not live-refresh during a run | community |
| G-K2 | Run log and ledger are written only when the run finishes; a killed run leaves neither | community |
| G-K3 | Text search across a whole community scans up to 40000 rows | community |
| G-K4 | HTML comments follow `rel=next` links only | community |
| G-K5 | No two-factor, captcha or JSON-API login | community |
| G-K6 | HTML learning is heuristic plus AI; a hostile markup needs a hand-edited `SCHEMA` | community |
| G-K7 | Votes, reactions and attachments are not collected | community |
| G-K8 | Cross-feed duplicate detection compares sampled addresses at discovery time only | community |
| G-SK1 | No test installs a skill from a live network origin | skills |
| G-X | Excluded by the owner: community passwords stored in plain text | community |
## Honest classification before any code
Closable by code and proven on this machine: G-N2, N3a-e, N4, N5, N6a-c, E1, E2, S1-S7, K1-K8 (K5 partly, see its item), SK1, and the capacity part of G-N7.
Closable only partly, with the remainder stated:
- **G-N1 / G-N7 (success per address class).** A residential donor is a physical machine this session does not have. What is built: every job records the donor's address class, the admin page and `GET /admin/nodes` aggregate success per destination and class, and the docs print `n = 0` until real donors exist. The viability number itself stays unmeasured and the docs say so.
- **G-N7 (24 hour soak).** A soak can only be as long as this session. It is run in the background for as long as the work lasts and reported with its true duration.
- **G-N7 (idle timeout through the public hostname).** Measuring it requires an authenticated WebSocket on `devplace.net`, which is a write to the production database. It is measured against the local stack and the exact proxy settings are read from `nginx/nginx.conf.template`; the public leg is listed as needing the owner's explicit go-ahead.
- **G-E3 (Firefox).** Chromium is driven for real through Playwright. Firefox is attempted; if the available build cannot load a temporary add-on, the item stays open for Firefox only.
- **G-K5 captcha.** A captcha exists to stop automation. It is detected and reported loudly (`captcha_required`), never solved.
Not closable by code: **G-N8** (a legal decision is a human act). It stays `[!]`; the plan records exactly what the decision must cover.
## Protocol and specification changes (done first, shared by server and both clients)
All additive and optional, `PROTOCOL_VERSION` stays 1 (an old client ignores unknown fields on server frames; the server rejects unknown fields from clients, so only server-to-client additions are made).
| Change | Where | Why |
|--------|-------|-----|
| `welcome.device_label` (str, optional) | `spec.py` FRAMES, `session.handshake`, Go `Frame`, extension `WsClient` | G-E2 |
| `kick.until` (str, optional ISO instant) | `spec.py`, `registry.kick`, clients | G-N6b |
| Ticket `403` body carries `until` for a timed suspension | `routers/nodes/ticket.py` | G-N6b |
| New refusal `on_battery` ("donor is running on battery") | `spec.py` REFUSALS, `dispatcher.DONOR_LIMIT_CODES` (pause 300 s), Go `RefuseBattery`, extension `REFUSALS` | G-N6a |
| `LimitSpec.last_seen_write_seconds` (60), `redundancy_percent` (5), `redundancy_max_percent` (20), `geo_lookup_cache` (4096) | `spec.py` | G-N3a, N4, N2 |
The extension spec test and the Go policy tests are updated in the same change so drift fails loudly.
---
## Track N: node network, server (`devplacepy/services/nodes/`, `database/nodes.py`, `routers/nodes/`, `routers/admin/nodes.py`)
### N-01 Geo and ASN database (G-N2)
- Root cause: `record_device_connection` is called with defaults, so `country`, `asn`, `asn_class` stay empty (`routers/nodes/ws.py`). The doc says the fix is an offline ASN database.
- Source: DB-IP "IP to ASN Lite" and "IP to Country Lite" monthly CSVs, free, no key, CC BY 4.0 (attribution printed on the docs page and in `devplace nodes geo-status`). Verified on 2026-10-16: `https://download.db-ip.com/free/dbip-asn-lite-2026-10.csv.gz` answers 200, 7.0 MB, 477362 ranges.
- Build `services/nodes/geo.py`: `GeoDatabase` backed by one SQLite file at `config.NODE_GEO_DB` (declared in `DATA_PATHS`, created by `ensure_data_dirs`). Tables `geo_asn` and `geo_country`: `(family INTEGER, first BLOB, last BLOB, ...)`, primary key `(family, first)`. Addresses are packed with `ipaddress` (4 or 16 bytes, compared bytewise by SQLite). Lookup is `WHERE family=? AND first<=? ORDER BY first DESC LIMIT 1` then `last>=ip`. Memory stays flat, no 477k-entry Python list.
- Build `services/nodes/addressclass.py`: `classify(asn, org) -> residential | mobile | datacenter | unknown`. A static frozenset of well-known hosting and cloud ASNs (AWS, GCP, Azure, Hetzner, OVH, DigitalOcean, Linode, Vultr, Oracle, Alibaba, Cloudflare, Contabo, Scaleway, Leaseweb, M247, Choopa, Tencent, IBM, Rackspace) plus word-boundary org keywords (`hosting`, `cloud`, `datacenter`, `data center`, `vps`, `server`, `colo`, `dedicated`, `cdn`, `vpn`, `proxy`); mobile by keywords (`mobile`, `wireless`, `cellular`, `lte`, `gsm`); no data means `unknown`; everything else is `residential`. Heuristic by nature, so the docs call it that and the admin override (N-02) exists.
- Loader `geo_sync()` downloads through `stealth_async_client` (the project rule), verifies gzip, parses row by row with a hard cap on rows, loads into a temporary file and swaps it in atomically (verify before write). `--dry-run` reports counts only. CLI: `devplace nodes geo-sync [--dry-run] [--month YYYY-MM]`, `devplace nodes geo-status`.
- Connect path: `ws.py` resolves the address transiently (the address is never stored, only `hash_ip`), calls `GeoDatabase.lookup`, passes `country`, `asn`, `asn_class` to `record_device_connection`, and puts `asn_class` on `session`. A missing or unreadable database yields `unknown` plus the counter `geo_unavailable` and one log line per process.
- Tests: unit for packing/lookup edge cases (first/last address of a range, gap between ranges, IPv6, v4-mapped), classifier table (including keyword false-positive guards such as `Hostingly`), loader against a local fixture file, atomic swap; api for `devplace nodes geo-sync --dry-run` is CLI-unit. Static check that no raw address column exists on `node_devices`.
- Docs: nodes CLAUDE.md, `templates/docs/nodes.html` (donor-facing: what is looked up and that the address is not stored), `privacy.html` (the lookup), attribution line.
- Status: [~] Result: geo.py + addressclass.py + CLI geo-sync/geo-status; real DB-IP month built in 10 s (477362 ASN, 710834 country ranges, 48 MB), lookup 0.17 ms, installed through the real CLI against a scratch data dir; 30 unit tests + 2 api tests (header-driven class, missing database counter). Docs pending.
### N-02 Enforce the datacenter exclusion, with an admin override (G-N2, D6)
- Scheduler eligibility gains one rule: `session.asn_class in {residential, mobile}` or the device carries `class_override = "allow"`. `unknown` is refused by default, because "residential and mobile only" is the decision and an unverifiable address must not be assumed good (loud skip, counter `jobs_no_eligible_class`). Site setting `nodes_allow_unknown_class` (default `0`, apply live) lets an administrator run without a geo database during bring-up.
- New column `node_devices.class_override` (ensured in `NODES_COLUMNS`). Admin route `POST /admin/nodes/devices/{uid}/class` (`allow` or `clear`), `require_admin`, `is_senior_admin` against the device owner, audit key `node.device.override`, catalogued in `audit/events.md` and `category_for`. Devii `node_admin_class_override` (admin only, in `CONFIRM_REQUIRED`, declares `confirm`).
- Admin page: class column and an override button in the recent-devices table; `AdminNodesOut` carries both fields.
- Tests: unit scheduler (each class, override, unknown with and without the setting), api admin route (authz, senior-admin denial, audit row), Devii action gating.
- Status: [~] Result: scheduler rule (residential/mobile, unknown only with nodes_allow_unknown_class, per-device override), POST /admin/nodes/devices/{uid}/class, Devii node_admin_class_override (confirm-gated), audit node.device.override, admin page button; 5 scheduler + 6 api tests. Docs pending.
### N-03 `last_seen_at` while connected (G-N3a)
- One batched `UPDATE node_devices SET last_seen_at=:now WHERE uid IN (...)` per `last_seen_write_seconds` from `NodesService.reconcile`, one statement for all sessions (no per-session query, the measured lesson in the doc), plus a final write in the disconnect path.
- Tests: unit with fake sessions (statement count is 1 for N sessions, interval respected), api that a connected fake device advances `last_seen_at` while idle.
- Status: [~] Result: NodesService.persist_state touches all connected devices in one batched UPDATE per last_seen_write_seconds; unit tests (interval, force, failure).
### N-04 Durable counters (G-N3b)
- Table `node_counters(name PRIMARY KEY, value)` (derived, hard-GC only, not soft-deletable). `NodeCounters` keeps a delta per name; `reconcile` flushes all deltas in one `INSERT ... ON CONFLICT(name) DO UPDATE SET value = value + excluded.value` transaction and resets them only after commit. `snapshot()` stays the live process view; `lifetime()` reads the table. Admin page shows both.
- Tests: unit (flush adds, a failed flush keeps the delta, two processes flushing concurrently sum exactly: real separate processes, as the project procedure requires), api admin payload.
- Status: [~] Result: node_counters table, NodeCounters pending deltas flushed in one transaction every counter_flush_seconds, failed flush restores deltas; unit + api tests. Multi-process flush race still to run in validation.
### N-05 Persisted circuit snapshots and a per-destination admin view (G-N3d)
- `CircuitBreaker.snapshot()` returns `(key, failures, state, opened_at_wall)`. Table `node_circuits` is rewritten (upsert, delete when closed) from `reconcile` only when the breaker marked itself dirty. `NodesService` loads open circuits on start, converting wall time back to the monotonic clock, so a restart does not forget an open destination circuit.
- Admin page gets a "Circuits" table (destination or node, state, failures, seconds until half-open); `AdminNodesOut.circuits` capped at 100 rows.
- Tests: unit round trip across a simulated restart, expiry on load, cap, admin payload.
- Status: [~] Result: CircuitBreaker snapshot/restore + node_circuits + admin Circuit breakers table (read from the table so any worker answers); round-trip restart test.
### N-06 Cross-process fabric cache (G-N3c)
- Table `node_cache(key PRIMARY KEY, method, url, status, headers, body BLOB, stored_at, expires_at, size)` as an L2 under the existing in-process `ResultCache` (L1). Reads: L1, then L2 (then promote). Writes: both. Bounded by `LimitSpec.cache_max_bytes` and `cache_entry_max_bytes` and the TTL; `NodesService` prunes expired rows and, when over the byte cap, the oldest rows, in `run_once`. `cache.clear()` clears both. Responses with `status >= 500` or truncated are never stored (existing rule kept).
- Tests: unit (L1 miss L2 hit promotes, TTL, cap eviction order, oversize skipped), a real two-process test (worker A stores, worker B hits), kill-switch off does not serve from cache? (decision: cache is served regardless, as today, and the test pins that).
- Status: [~] Result: SharedResultCache (L1 + node_cache table), prune in service, cross-instance hit test; real two-process proof still to run in validation.
### N-07 Same-process fast path (G-N3e)
- `services/nodes/wake.py`: `WakeHub` holding an `asyncio.Event` for the dispatcher and one per awaited job. `JobBroker.enqueue` signals the dispatcher when it runs in the lock-owner process; `dispatcher._account`/`_failed`/`broker.complete` resolve the job event; `JobBroker.wait` awaits the event first and keeps the DB poll only as the cross-process fallback and a safety net. No behavior change in other workers.
- Measure before and after on this machine with a fake session: enqueue-to-assigned latency (today up to `dispatcher_tick_ms` = 200 ms) and result-to-caller latency (today up to `poll_interval_ms` = 100 ms). The numbers go into the doc.
- Tests: unit (event wake, no lost wake-up when set before wait, timeout path unchanged), existing broker tests stay green.
- Status: [~] Result: WakeHub + JobBroker events; unit tests prove immediate wake. Before/after latency numbers still to measure.
### N-08 Sampled redundancy for fetch-mode results (G-N4)
- Spec: `redundancy_percent` (default 5, ceiling `redundancy_max_percent` 20, site setting `nodes_redundancy_percent` clamped to it, 0 turns it off).
- After a `fetch`-transport result is accepted, a deterministic sample (`sha256(job uid) % 100 < percent`) enqueues a shadow job (`caller_kind = "shadow"`, `shadow_of = <uid>`, `exclude_device = <first donor>`), scheduled to a different donor owner. The shadow never spends the caller's quota and never enters the cache.
- Comparison is on a **shape**, not bytes (live pages change): status class, JSON top-level key set or the HTML `<title>`/first heading, plus a body size ratio inside a band. A mismatch raises a `node_flags` row `redundancy_mismatch` against both devices and counts `redundancy_mismatches`; no auto-ban. Match and mismatch counts are persisted (N-04) and shown on the admin page so the false-positive rate is itself measured.
- Tests: unit shape extractor (JSON, HTML, binary), sampling determinism and rate over 10000 uids, scheduler exclusion, dispatcher comparison with two fake nodes (match, forged body, one node errors), cap.
- Status: [~] Result: shape.py + dispatcher shadow jobs (scheduler exclusion, flags, counters, clamped setting); 9 shape + 8 dispatcher tests.
### N-09 DeepSearch routes node-servable hostile URLs through the fabric (G-N5)
- `services/jobs/deepsearch/crawl.py`: `_resolve_candidate` asks a new `NodeCrawl` helper first when the URL is allowlisted (`SPEC.destination(host)`) and `fabric.can_serve(url)` is true; the fetch uses `shared_fabric().fetch(..., caller_kind="user", caller_id=<session owner>)`, so it spends the owner's quota, honours the kill switch and falls back to the snippet on any `FabricError`. The static `HOSTILE_DOMAINS` tuple stays only for hosts the allowlist cannot serve (X, YouTube, Instagram and so on); Reddit is removed from it and the owner uid is threaded from the DeepSearch worker through `crawl()`.
- Same extraction path as `fetch_page` (shared function), so the text is parsed identically. Guests are not routed (no owner quota to charge).
- Tests: unit with a fake fabric (served, refused, unavailable, quota exceeded, guest), existing crawl tests stay green.
- Status: [~] Result: crawl.node_page via POST /nodes/internal/fetch with the owner key for allowlisted hostile hosts; 6 unit tests.
### N-10 Success per address class, measured automatically (G-N1, G-N7)
- `node_requests` records `device_asn_class` at claim time. `database.nodes.success_by_class(days)` aggregates `(adapter, asn_class, outcome)` over the retained window. `GET /admin/nodes` (`AdminNodesOut.success`) and the admin page show it; Devii `node_admin_status` includes it.
- Docs print the live table. Until a residential donor exists the cell reads `n = 0`, and the Reddit-from-residential gap is documented as unmeasured with the exact query that will answer it.
- Tests: unit aggregation (including empty), api payload.
- Status: [~] Result: claim records device_asn_class; durable outcome counters (outcome:adapter:class:kind) + admin table with n = 0 state; unit + api tests.
### N-11 Measurements (G-N7)
- **Socket capacity of one worker:** scratch server on a free port, up to `max_connected_nodes` (2000) synthetic devices and tickets created in a scratch database, asyncio WebSocket clients answering pings; record RSS, event-loop lag (a 10 ms ticker), reconcile tick duration and connect rate at 250, 500, 1000, 2000 sockets. Harness lives in the scratchpad (disposable), method and numbers go into the doc.
- **Soak:** same harness held open with jobs flowing through fake nodes; started in the background as early as possible and sampled every minute (RSS, loop lag, socket count, error counters). Reported with its real duration. Not claimed as 24 hours unless it ran 24 hours.
- **Idle timeout:** local stack measured (idle WebSocket for 5 minutes with no application frames, ping interval 20 s, observing whether anything closes it); `nginx/nginx.conf.template` read for `proxy_read_timeout` on the nodes location and quoted. The public hostname leg needs a production write and is left for the owner's go-ahead.
- Status: [~] Result: measured. Fast path p50 209 ms to 25 ms; capacity 2000 sockets in 341 MB; soak 81 minutes (NOT 24 h) clean; counter and cache races exact. Residential Reddit rate and public-hostname idle remain unmeasured (see Cannot be closed by code).
---
## Track C: clients (`nodes/client/` Go, `nodes/extension/`)
### C-01 Go: battery-aware pause (G-N6a)
- `ServerSettings` gains `PauseOnBattery`. New files with build tags: `battery_linux.go` (parse `/sys/class/power_supply/*`: on battery when a `Battery` supply exists, no `Mains`/`USB` supply is online, and the battery status is `Discharging`), `battery_darwin.go` (`pmset -g batt`), `battery_windows.go` (`GetSystemPowerStatus` through `syscall.NewLazyDLL`, no cgo), `battery_other.go` (unknown). `Limiter.Acquire` refuses with `on_battery` when the server setting is on and the probe says battery; the probe result is cached for 15 seconds. The server pauses the donor for 300 s on that code (N-spec change). Unknown means "not on battery" and logs once at debug (a probe failure must not silence a donor forever).
- Tests: table tests for the Linux parser with fixture trees, limiter test with a fake probe, the refusal code against `spec.py`.
- Status: [x] Result: Go battery probes (linux/darwin/windows/other), limiter refuses on_battery; go test green (agent report, re-verified: go vet clean, go test ok).
### C-02 Go and extension: a timed suspension is not permanent (G-N6b)
- Server: ticket `403 suspended` and the `kick` frame carry `until` for a timed suspension (read from `suspended_until`). A suspension without an end date stays a plain hold.
- Go: `suspended` leaves `FatalReasons`. New `outcome.holdUntil`; `Session.Run` sleeps until `until` (clamped to at least 60 s and at most 24 h, plus jitter) and then retries the ticket. `revoked` and the other reasons stay fatal. Extension `WsClient` gets the same state (`SUSPENDED` with a countdown in the popup).
- Tests: Go session test with a fake clock (suspended with `until`, without, expired), extension unit test, api test that the ticket body carries `until`.
- Status: [x] Result: Go holdUntil + extension SUSPENDED state; server sends until in ticket 403 and kick; tests in Go, node and api tiers.
### C-03 Extension: server-timezone hours (G-N6c)
- `Limits.applyServerSettings` also reads `active_from`, `active_to`, `timezone`; `inActiveHours()` requires both the donor's local window (browser clock, as configured in the popup) and the server window evaluated in the server-stored timezone with `Intl.DateTimeFormat(..., {timeZone})`. An invalid zone falls back to UTC, as `scheduler._zone` does.
- Tests: node tests with a stubbed clock across zones, windows that wrap midnight, an invalid zone. Parity test against the Go `InWindow` and the Python `in_active_hours` for the same 200 sampled instants.
- Status: [x] Result: extension server-timezone hours with Python-generated parity table shared by JS and Go tests; node --test 68/68 re-verified.
### C-04 Extension icons (G-E1)
- Four PNGs (16, 32, 48, 128) generated reproducibly by a small script using Pillow (already a dependency), a node-network glyph on the project accent colour. `manifest.json` gets `icons` and `action.default_icon`; `build.sh` includes them; the packaging test checks the files exist and the sizes match.
- Status: [x] Result: four PNG icons generated by nodes/extension/tools/make_icons.py, manifest + build.sh updated, package test.
### C-05 Device label in `welcome` (G-E2)
- Server sends `device_label` (from `node_devices.label`); extension `State` stores it and the popup shows the server label as the authority, falling back to the local one before the first connect. Go prints it in `status`/`run` logs.
- Tests: frame spec test, extension popup state test, Go decode test.
- Status: [x] Result: welcome.device_label server + Go + extension; api test asserts it.
### C-06 Real browser test of the extension (G-E3)
- Playwright persistent Chromium context with `--disable-extensions-except` and `--load-extension` against a scratch DevPlace server on a free port (not 10500/10501): register a device, drive the disclosure and connect flow through the real popup page, assert the socket connects (server registry shows the node), a job for an allowlisted host is answered (`via: fetch`; a 403 from this datacenter address is an acceptable status, as in the Go run), refusals work, and disabling disconnects. Firefox: attempt with the installed Playwright Firefox; if a temporary add-on cannot be loaded, record that and leave E3 open for Firefox.
- The test is kept as `tests/e2e/nodes/extension.py` (skips with a stated reason when Chromium extensions cannot load, never silently passes).
- Status: [x] Result: Chromium e2e test tests/e2e/nodes/extension.py passes (~10 s); Firefox 158 verified by scratch geckodriver script; CSP fix found and applied by that test; method in nodes/extension/README.md.
### C-07 Client build and verification
- `go vet ./...`, `go test ./...`, `node --test`, `nodes/client/build.sh` into a scratch directory (not `data/`). The distributable build into `data/node_clients/` is a deploy step left to the owner.
- Status: [x] Result: go vet, go test, node --test 68/68, cross-compile linux/windows/darwin amd64+arm64 and freebsd (agent report; I re-ran vet, test and node tests).
---
## Track S: community sync (`devplacepy/services/sync/`, `database/sync.py`, `routers/sync/`)
### S-00 Design decisions for new kinds (G-S2)
- Kinds, in dependency order: `users`, `projects`, `posts`, `comments`, `polls`, `votes`, `reactions`, `follows`, `poll_votes`, `attachments`. `KINDS` is the single list; `SyncPusher`, `BatchRunner`, docs and Devii tools read it.
- Counters stay un-importable as numbers. A synced vote exists only with a synced voter, and the target `stars` is recomputed from the local `votes` rows by the same function `apply_vote` uses (`update_target_stars` + `clear_user_stars`), once per touched target per batch. This replaces "synced posts have stars = 0" with "stars are derived from synced votes only"; the doc rule is rewritten, not silently broken.
- No side effects, as before: no notifications, XP, badges, mention fan-out, AI correction/modifier. Content filter runs on every authored text (`projects`, `polls`).
- Records:
- `ProjectRecord`: `author_uid`, `title`, `description`, `status`, `platforms`, `website_url`, `repo_url`, `project_type`; always public and writable (`is_private = 0`, `read_only = 0`); no files, no containers.
- `PostRecord.project_uid` (optional reference to a synced project) closes "project links".
- `PollRecord`: `post_uid`, `question`, `options: [{origin_uid, label}]` (2 to 6, same caps as `create_poll`); options are rows of the poll, fingerprinted together.
- `VoteRecord`: `voter_uid`, `target_kind` in `post|comment|project`, `target_uid`, `value` in `{1, -1}`; a cleared vote is `deleted: true`.
- `ReactionRecord`: `author_uid`, `target_kind` in `post|comment|project`, `target_uid`, `emoji` (validated against the existing reaction emoji rule); tombstone removes.
- `FollowRecord`: `follower_uid`, `following_uid`; no self follow; tombstone removes.
- `PollVoteRecord`: `voter_uid`, `poll_uid`, `option_uid`; tombstone removes.
- `AttachmentRecord`: `author_uid`, `target_kind` in `post|comment|project`, `target_uid`, `filename`, `mime_type`, `content_b64`. **No network fetch inside a processor** (it runs on the request path and SQLite/HTTP stay out of each other's way): the sender downloads and sends the bytes; the processor decodes, applies the existing upload validators (type, size, image dimensions), and writes through `attachments.store_attachment` + `link_attachments`. `max_attachment_bytes` is a `SyncLimits` field and the envelope gets a total byte cap (`max_batch_bytes`).
- Voter-visibility: `poll_votes` are private ballots in this product (`UNREPORTABLE_TABLES` says so). They are accepted inbound (the sender owns that decision) and never exported (S-02).
- Each new local row is soft-deletable and created with `deleted_at: None, deleted_by: None`; tombstones soft-delete with `deleted_by = "sync"`; toggles revive instead of duplicating (the project's hard rules).
- Reportability: `projects` and `polls` are already reportable targets; synced rows reuse them. `votes`, `reactions`, `follows`, `poll_votes`, `attachments` are not new tables, so `REPORTABLE_TARGETS`/`UNREPORTABLE_TABLES` stay exact (the registry test is the check).
- Status: [~] Result: decisions implemented; correction recorded: votes, reactions and poll ballots are accepted inbound but NOT exported because no UI exposes who voted.
### S-01 Implement the new kinds (G-S2)
- `spec.py` (kinds, limits), `records.py` (models + `RECORD_MODELS`), `normalize.py` (emoji, options, platform lists), `processors.py` (one processor per kind on `RecordProcessor`, `PROCESSORS`), `database/sync.py` (link helpers for composite natural keys: a vote/reaction/follow link is keyed by the sender's own `origin_uid`; a second record for the same `(voter, target)` with a different `origin_uid` revives the same local row instead of duplicating, the concurrency lesson of the toggles), `routers/sync/` one leaf per kind (`projects.py`, `polls.py`, `votes.py`, `reactions.py`, `follows.py`, `poll_votes.py`, `attachments.py`), `GET /sync/spec.json` gains the schemas, Devii `sync_<kind>` tools (`requires_admin`, form-encoded `records`), `docs_api` endpoints, README, `docs/privacy.html` row update, sync CLAUDE.md rewritten for the new rules.
- Tests: unit per normaliser and processor (create, update, stale, tombstone, locally deleted, purged, wrong references), api per route (authz, gate, dry run, cap), stateful fuzz extended to all kinds with invariants (no dangling links, no orphan votes/reactions/follows, `stars` equals net local votes for every synced target, one vote per voter and target, no self follow), and a real multi-process race on the vote/follow toggle rows started from a fresh unseeded database (40 racers, exactly one row).
- Status: [~] Result: 10 kinds, records, processors, 7 new routes, Devii tools, docs_api; 69 api tests incl. new kinds. Fuzz + multi-process race and docs/privacy pending.
### S-02 Export endpoint (G-S1)
- `GET /sync/export/{kind}?since=<iso>&cursor=<opaque>&limit=<=50>` (administrator only, behind `sync_enabled`, audited as `sync.export`). Keyset pagination on `(updated_at, uid)`; the cursor is a signed-free, validated base64 of those two values. Output items are exactly the `*Record` models, so a DevPlace instance can feed another with the sender it already has. Soft-deleted rows are emitted as tombstones.
- Exported: `users` (public fields only: never email, hash, role, api key), `projects` (public only), `posts`, `comments`, `polls`, `votes` and `reactions` (author uid and target only), `follows`, `attachments` (as `url`, not bytes). **Not exported:** `poll_votes` (private ballots), anything private or deleted-by-account-deletion beyond the tombstone, and rows that arrived by sync themselves (`sync_links.local_uid`), so two instances cannot echo each other's data in a loop.
- Tests: api per kind (shape equals the inbound model, no private field, pagination is gapless and repeat-free over 120 rows, tombstones, the echo guard), a round trip test (export from one scratch database, import into another through the real routes, compare counts).
- Status: [~] Result: GET /sync/export/{kind} with expression indexes (EXPLAIN-tested), echo guard, keyset pagination; 27 api tests with deletions. Cross-database round trip pending.
### S-03 Deferred resolution queue (G-S3)
- Statuses gain `deferred` (an applied-later state, distinct from `rejected`). Table `sync_pending` (`origin`, `kind`, `origin_uid`, `missing_kind`, `missing_uid`, `record` JSON, `attempts`, `created_at`, `expires_at`) with a unique index on `(origin, kind, origin_uid)` so a resend replaces instead of stacking.
- A record rejected for `unknown_author`, `unknown_post`, `unknown_parent`, `unknown_target`, `unknown_poll`, `unknown_option`, `unknown_project` is queued (not on dry run) and reported `deferred` with the same reason. When any record of kind K with origin uid U is `created`, `BatchRunner` looks up pending rows waiting on `(K, U)` and re-applies them with the same processors, iteratively (a released comment can release its replies) under `pending_release_cap_per_batch`. Caps: `pending_max_per_origin` 5000 (the 5001st is rejected with the old reason, loudly counted), `pending_ttl_days` 7, `pending_max_attempts` 5; expired rows are garbage collected hard by the service already responsible for sync housekeeping (Community sync service `run_once`).
- The community ledger gains the outcome `deferred` (warn tone) with a plain-language reason; the pusher's projection logic for dry runs is unchanged.
- Tests: unit (queue, release in any order, chain of three, cap, ttl, replace-on-resend, dry run writes nothing), the 16-process race (post and its 15 comments arriving in random order from separate processes: exactly the right final counts), ledger reason text test.
- Status: [~] Result: sync_pending + PendingQueue; 12 api tests (chain, shuffle of 18 records, cap, ttl, dry run). 16-process race pending.
### S-04 Deletion feedback to the origin (G-S4)
- `GET /sync/deletions?origin=<id>&since=<iso>&cursor=` (administrator only) lists synced rows deleted or purged on this side: `{kind, origin_uid, deleted_at, deleted_by_class: "owner|moderator|system|purged"}`; `POST /sync/deletions/ack` (`origin`, up to 50 `{kind, origin_uid}`) stamps `sync_links.ack_at`. A record that the origin re-sends after acknowledging answers `unchanged / acknowledged_deleted` instead of the loud `rejected / purged_locally`, so the noise stops once the origin has been told. New columns `sync_links.deleted_seen_at`, `ack_at` (ensured).
- The scheduled community sender skips records whose link is locally deleted (cheap lookup) and records the skip in the ledger, so a re-collection of a deleted post stops being pushed every run.
- Devii tools `sync_deletions`, `sync_deletions_ack`; docs; audit `sync.deletions.ack`.
- Tests: api (listing by class, pagination, ack, reappearance answers `acknowledged_deleted`, authz), unit.
- Status: [~] Result: GET /sync/deletions and POST /sync/deletions/ack, acknowledged_deleted quiet answer; tests in deletions.py.
### S-05 `min_comments` counts what is already synced (G-S5)
- `Selector` counts, per post, comments collected in this run plus the live local comment count of the post's existing link (one batched `GROUP BY target_uid` query for all linked posts in the run, not per post). A post that was kept once is never dropped (existing rule), so this only changes first-time decisions.
- Tests: unit (collected 0 + 3 already synced passes a threshold of 3; none anywhere is dropped; the batched query is one statement).
- Status: [~] Result: Selector candidates (sync_candidates) park and revive a post with its comments and authors across runs; 10 unit tests.
### S-06 and S-07 Alias heals when the clone name changes or collides (G-S6, G-S7)
- `Aliaser.reconcile(origin)` runs at the start of every aliased run and after the push: for every identity whose clone user exists, compare `users.username` with `our_name`. On a difference: update the identity (`our_name`, `our_key`, skipping and logging when the new key collides with another identity), then clear the fingerprint of the origin's posts and comments whose stored text contains the old alias (bounded to 5000 links per run, oldest first), so the next push rewrites them with the live name. The rewriter always resolves names through the live clone username first.
- This covers both gaps: an administrator rename, and the case where `free_username` gave the clone a suffix because a registration took the name between alias generation and clone creation (the post-push reconcile sees the suffix and heals the mentions on the next run). The race window itself is also narrowed: the clone user is created before dependent content is rewritten.
- Tests: unit (rename, suffix collision, key collision skip, bounded sweep, nothing to do), an integration run through `CommunityRunner` with a fake collector: rename between runs, mentions follow.
- Status: [~] Result: AliasHealer rewrites stored mentions and follows the clone name; wired before and after the push in CommunityRunner; 6 unit tests.
---
## Track K: community discovery (`devplacepy/services/community/`, `routers/admin/communities.py`, `static/js/`)
Note: `runtime.py` and `extract.py` are inlined into generated collectors by `generator.py`. Every change there is covered by a generator test that executes the generated file, and existing communities regenerate their files (the database row holds the schema; files are regenerated when missing).
### K-01 Live refresh while a run is in progress (G-K1)
- `/admin/communities/{slug}` and the ledger page swap their `[data-live]` sections on an interval while the community is `busy` or a run row is `running` (the `AdminNodes.js` pattern, shared `Poller`, stops when idle). New `static/js/AdminCommunity.js` (one class, `app.adminCommunity`), no inline script; responsive at phone width.
- Tests: e2e (start a slow fake run, the progress counters change without reload, polling stops at the end), api (the page carries the live hooks and the run row state).
- Status: [x] Result: AdminCommunity.js live refresh on data-busy/data-live sections; busy field on AdminCommunityLedgerOut; tests/e2e/admin/communities/live.py.
### K-02 Incremental run log and ledger, interrupted runs are visible (G-K2)
- `CommunityRunner.run` registers a checkpoint callback: every `checkpoint_seconds` (15) and after every pushed batch it writes the current `LogBook` text, tally and verdict `running` into the run row, and flushes new ledger drafts with `LedgerStore.append` (continuing `seq`, respecting the same caps and rank rules). The final `finish_run` replaces the totals; ledger rows are never written twice (an index on `(run_uid, seq)` makes a duplicate a no-op).
- At service start, `CommunityStore.reap_interrupted()` marks any run left `running` for longer than `run_timeout_seconds` as `interrupted` (verdict `bad`, headline "interrupted: the process ended before the run finished") and keeps whatever was checkpointed. Retention (`prune_runs`) is unchanged.
- Tests: unit (a run killed after the second batch leaves log, tally and ledger; resume never duplicates), a real process kill test (`SIGKILL` a subprocess mid-run, then reap), caps still hold.
- Status: [x] Result: RunCheckpoint every 15 s, unique (run_uid, seq), reap_interrupted marks RUN_INTERRUPTED; unit tests in checkpoint.py.
### K-03 Indexed text search (G-K3)
- SQLite FTS5 external-content table `community_run_items_fts` over `label`, `origin_uid`, `detail` with `tokenize='trigram'` (substring semantics like the current `LIKE`, query minimum 3 characters), kept in step by triggers on `community_run_items`. Queries of 3+ characters use `MATCH` joined back to the scoped rows; shorter queries keep the bounded `LIKE` on the scoped index. If FTS5 or the trigram tokenizer is unavailable, `init_db` logs it once and the old path is used (the same results, the old cost). Prune deletes cascade through the triggers.
- Measure on the same scratch ceiling used before (10 runs of 4000 rows plus the community scan) and update the numbers: the no-match search was 37 ms, the old unbounded case 97 ms. `EXPLAIN QUERY PLAN` test extended so the FTS path is asserted, not assumed.
- Tests: unit (substring parity with `LIKE` over a generated corpus including `%`, `_`, quotes, unicode), query-plan test, trigger consistency after insert, delete and prune.
- Status: [x] Result: FTS5 trigram table with triggers for community-wide search of 3+ chars; measured 34 ms to 0.4 ms (no match), 34.5 to 0.8 ms (rare), common term 9.4 to 21 ms (slower, accepted).
### K-04 Comment pagination beyond `rel=next` (G-K4)
- `comments.pagination` becomes a learned and overridable spec: `next_link` (selector list, default adds text-matched anchors in a pagination container: `next`, `older`, `more`, `volgende`, `›`, `»`, `→`), `page_param` (`?page=N`, `?p=N`, `/page/N`, detected from the current URL or the numbered page links), `json_next` (a JSON `next`/`next_page`/`links.next` key). The collector stops on a page that adds no new comment ids or repeats a URL, always under `max_comment_pages`.
- The learner (`PageProbe`) detects the style from the sampled post pages and stores it; `NEXT_PAGE_FIELDS` remains the default.
- Tests: extraction fixtures for each style (rel next, text next, numbered page links, `?page=`, `/page/N`, JSON next), loop and duplicate guards, generated-collector parity.
- Status: [x] Result: next_link, page_param, json_next, none hints learned and enforced; stops on a page with no new comments; unit tests pagination.py.
### K-05 Login: JSON-API and two-factor, captcha detected (G-K5, passwords excluded)
- JSON-API login: `schema.login.kind = "json"` with `url`, `fields` (username/email/password key names), `token_path` (JSON path to a bearer token, optional); success is a 2xx plus either a token or a session cookie. `LoginLearner` probes a short fixed list of common endpoints (`/api/login`, `/api/auth/login`, `/api/v1/auth/login`, `/auth/login`, `/login.json`, `/session`) with the stored credentials, never anything outside the community origin, guarded by `guard_public_url`.
- Two-factor (TOTP): a new column `communities.login_totp_secret`, handled exactly like the other credential columns (never returned by any endpoint, admin form, Devii tool or CLI; set through `devplace community login --login-totp`). `extract.totp(secret, now)` implements RFC 6238 with the stdlib. The login flow detects a follow-up one-time-code step (`otp`, `totp`, `code`, `token`, `2fa` input) and submits the current code. Tested against the RFC 6238 vectors. SMS and push 2FA cannot be automated and are reported as `second_factor_unsupported`.
- Captcha: detection of `g-recaptcha`, `h-captcha`, `cf-turnstile`, `captcha` form markers moves the community to `login_required` with the reason `captcha_required` and a plain message; it is never attempted. Recorded as unsupportable by design.
- Tests: fixtures for a JSON login (token and cookie), a TOTP step with the RFC vectors and a fake clock, a rejected code, the captcha stop, credential non-disclosure (every serializer and the CLI output are scanned for the secret).
- Status: [x] Result: TOTP (RFC 6238 vectors), JSON-API login, OTP form detection, captcha detected and reported (never solved); secret never returned; models, router, CLI, Devii, docs_api, templates plumbed; tests login.py, admin index test.
### K-06 Structured-data learning for hostile markup (G-K6)
- New inferrer, tried between the heuristics and the AI assistant: `StructuredDataInferrer` reads JSON-LD (`DiscussionForumPosting`, `Article`, `BlogPosting`, `Comment`, `ItemList`) and schema.org microdata (`itemtype`/`itemprop`) and OpenGraph, producing a feed type `json_ld` (runtime support: extract the list of `<script type="application/ld+json">` blobs, flatten `@graph`, then reuse the existing JSON path engine). Known software signatures (Discourse, Lemmy, WordPress `wp-json`, Mastodon, phpBB/XenForo markup) select a ready, fixture-tested spec instead of guessing.
- Each learned field is validated against the sample like the AI proposals (60 % rule). The failure message names which fields could not be learned, so the human edit (K-07) is targeted.
- Tests: fixtures per source type (JSON-LD, microdata, OpenGraph-only, a Discourse `.json` topic list, a WordPress `wp-json` posts feed), generated-collector parity, no regression on the existing lobste.rs fixtures.
- Status: [x] Result: json_ld feed type, detector, learner, runtime; unit tests structured.py. Discourse and WordPress ready-made specs not built (schema editor covers hostile markup).
### K-07 Admin schema editor (G-K6)
- `GET/POST /admin/communities/{slug}/schema` (administrator, audited `community.schema.update`): shows the learned `SCHEMA` JSON; a save validates structure with the same validator the learner uses, runs a verification collection on a small sample through the real collector (dry run, nothing pushed), and only then stores the schema and regenerates `collectors.py` (verify before write; the previous schema is kept as `schema_previous` for one-click restore). Devii `community_schema_get` / `community_schema_set` (set is in `CONFIRM_REQUIRED`).
- Tests: api (valid, invalid structure, failing verification keeps the old schema, restore, authz, audit), e2e (edit and save through the page).
- Status: [x] Result: /admin/communities/{slug}/schema GET/POST/restore with validate_schema and sample verification, schema_previous, audit keys, Devii tools, docs_api; tests schemacheck.py and api schema.py.
### K-08 Collect attachments, votes and reactions where they can be attributed (G-K7)
- Attachments: option `collect_attachments` in `sync_options` (default off, per community). The collector finds inline images and attachment links in post and comment bodies, downloads them through `Fetcher` (stealthii + SSRF guard) under per-file and per-run caps, and emits `attachments` records with `content_b64`. Failures are ledger warnings, never run failures.
- Votes and reactions: the learned spec may name optional `votes` / `reactions` list fields (per-voter entries with a user id or name and a direction/emoji). When the source exposes them, voters become synced users and the records flow through S-01. **When the source only publishes an aggregate count (almost every site), nothing is imported:** an anonymous total cannot be turned into votes without inventing voters, which the platform forbids by design (no purchasable or fabricated engagement). This is stated in the docs as intentional, not as a missing feature.
- Tests: fixtures (attachment download within caps, oversize skip, blocked address, per-voter JSON feed becomes votes with derived stars, aggregate-only feed produces zero vote records), ledger rows.
- Status: [x] Result: collector emits votes, reactions and attachments; alias mode drops voter-attributed rows; generator VERSION 2 auto-regenerates; unit tests engagement.py.
### K-09 Cross-run duplicate detection (G-K8)
- The collector `state` (the watermark JSON already persisted per community) gains `links`: a bounded map `canonical_link -> post origin_uid` (cap 20000, oldest dropped, trimmed on write). `add_post` consults it, so a twin story seen in a different feed in a later run is aliased to the first post instead of becoming a second post; comments attach to the survivor as they do within a run.
- Tests: two feeds, two runs, same canonical link: one post; map cap and trimming; same-feed posts with the same link are still never merged (existing safe rule); generated-collector parity.
- Status: [x] Result: state links capped at max_remembered_links (20000); unit tests dedupe.py.
---
## Track SK: skills (`devplacepy/services/skills/`)
### SK-01 Real network leg for the skill fetcher (G-SK1)
- The SSRF guard refuses loopback, which is why the network leg was never exercised. This machine has a global IPv4 address, and a server bound to it passes the guard unmodified, so the test runs the real `guarded_async_client` over a real socket with no patching: `ThreadingHTTPServer` on `0.0.0.0:<free port>` serving a `SKILL.md` plus a bundled `references/` file, fetched through `fetch_skill` at `http://<global ip>:<port>/SKILL.md`. The test discovers the address (UDP connect trick, then `is_blocked_address`), and **skips with a stated reason** when the host has no global address, never passing silently. Also asserted over the same socket: a loopback URL is refused, a redirect to a loopback address is refused, an oversize body is cut at the limit, an HTML page is rejected.
- Status: [x] Result: real-socket network test in tests/unit/services/skills/fetcher.py through the host's global IPv4, unpatched guard; 36 pass; skips with a reason when no global IPv4.
---
## Cross-cutting work
### X-01 Documentation of record
Rewrite the `Gaps` sections to state only what is still true, move closed gaps into "What exists today" with the measured numbers, update: `services/nodes/CLAUDE.md`, `services/sync/CLAUDE.md`, `services/community/CLAUDE.md`, `services/skills/CLAUDE.md`, `nodes/extension/README.md`, `nodes/client/README.md`, `README.md`, `plan.md` (node plan progress), the docs pages (`nodes.html`, `nodes-protocol.html` reads the spec, `privacy.html`, community and sync pages), `docs_api` groups, `audit/events.md`. The memory file `node-network-plan.md` is updated at the end.
- Status: [ ] Result:
### X-02 Validation gates (all must be clean)
1. `python -c "from devplacepy.main import app"` after every Python edit.
2. `ruff check` / `pyflakes` on every touched Python file.
3. `go vet ./... && go test ./...` in `nodes/client`; `node --test` in `nodes/extension`.
4. `grep` for the em dash character and its HTML entity in every touched file.
5. CSS braces and HTML tags balanced for touched templates; JS parses.
6. Fan-out check per route (HTML, JSON, Devii, docs).
7. Four-layer correctness procedure for the spendable/state-machine/concurrent items (votes and stars, the pending queue, counters flush, cache, redundancy sampling): property checks, stateful fuzz, real multi-process races from fresh rows, static analysis.
- Status: [ ] Result:
### X-03 Full suite
- `make test` (unit, api, e2e, serial). Every failure is fixed, not skipped.
- Status: [ ] Result:
### X-04 Commit
- Commits per track with the pre-commit hook bumping the patch version. No push (outward-facing, not requested). The owner is told about the dist build into `data/node_clients/` and the optional public-hostname idle test.
- Status: [ ] Result:
---
## Cannot be closed by code (kept visible on purpose)
| Gap | Why | What is built instead | What closes it |
|-----|-----|-----------------------|----------------|
| G-N8 legal decision | A platform-terms, donor-terms, ISP-terms and jurisdiction decision is a human act. | Nothing is enabled for any account; `nodes_enabled` is `0`; the kill switch is enforced within one tick. | The owner records the decision; then switches the service on. The decision must cover: Reddit and any future destination terms, residential donor terms of service, ISP terms on traffic relaying, the jurisdiction of donors and of the operator, and the adult-only rule. |
| G-N1 Reddit from a residential line | Needs a residential machine. | Per-class success is recorded and shown (N-10). | A residential donor runs the Go client or the extension for a while. |
| G-N7 24 h soak, public idle timeout | Time and a production write. | Local soak with its true duration, local idle measurement, proxy config quoted. | Run the same harness for 24 h; owner approves a one-off production connection. |
| G-K5 captcha | Defeating a captcha is circumventing the control it exists for. | Detected, reported as `captcha_required`. | A human logs in once, or the site offers an API. |
| G-E3 Firefox (if the build cannot load add-ons) | Depends on the available Firefox build. | Chromium proven for real. | A normal Firefox with `web-ext run`. |
## Progress log
(One line per finished item: date, item, evidence.)
## Result summary
(Filled in at the end: what is closed, what is not, measured numbers, test counts, commits.)
happyplanting.md
-124
View File
@@ -1,124 +0,0 @@
# happyplanting: fixing Continuous Delivery lock-in
Status: plan only. Nothing in this file is implemented. Written 2026-10-17 from a read of the code, not from a run of the game.
## Problem (verified in code)
Continuous Delivery (`autoreplant` Mastery upgrade) replants the crop that was just harvested, immediately, whenever it is affordable.
- `store/actions.py::harvest` and `_auto_harvest` call `_try_autoreplant` after crediting. The harvest pays before the replant is checked, so "affordable" is true in practice and the plot is never empty.
- `plant()` refuses any plot that already holds a crop. There is no uproot, swap or cancel.
- `upgrade_mastery` is one-way: no refund, no toggle. Mastery survives Refactor.
- `_try_autoreplant` checks cost only. It ignores market saturation and crop locks.
- Only exits today: Refactor (clears plots, costs a prestige-scaled fee) or running below the planting cost (rare).
## Goals
1. A per-plot "stop after this harvest" toggle, so the player gets one free crop choice on that plot.
2. Do not replant into a bad market.
## Part A: per-plot "stop after harvest" (ship this)
### Design
New column `game_plots.stop_after_harvest` (int, default 0, always read with `COALESCE`). Added to the existing plot column ensure-block in `database/schema.py` (next to `raided_fraction`, ~line 1508) with example value `0`.
Semantics: one-shot, per plot. While set, the next harvest of that plot does not replant. The plot ends empty and the player plants whatever they want.
### Why this is the race-free shape
The flag is read by `_try_autoreplant` from its own fresh `_plot_at(...)` re-read, which already happens AFTER `clear_plot` wins. Rules that make that safe:
- The toggle endpoint is one conditional UPDATE: `SET stop_after_harvest = :v WHERE crop_key != '' AND crop_key IS NOT NULL`, checked via `rowcount` (`conditional_update_row`). It cannot touch an already-cleared plot.
- So the toggle either lands before `clear_plot` (the post-clear re-read sees it, no replant) or after (the WHERE fails, loud "plot is empty" error). No window where the harvest uses a stale flag.
- `clear_plot` is NOT changed. A leftover `1` on an empty plot is harmless because every path that puts a crop on a plot writes `stop_after_harvest = 0` in the same claim UPDATE: `plant()` and `_try_autoreplant` (both already use `conditional_update_row` with a SET string), and the `prestige()` plot reset.
- Never reset the flag inside `clear_plot`: that would erase it before `_try_autoreplant` can read it.
### Changes (fan-out, all four faces)
| Layer | File | Change |
|---|---|---|
| Schema | `database/schema.py` | ensure `game_plots.stop_after_harvest` |
| Data | `services/game/store/actions.py` | `_try_autoreplant`: return early if `COALESCE(plot.stop_after_harvest, 0)`; add `stop_after_harvest = 0` to the claim SET in `plant()` and `_try_autoreplant`; add the column to the `prestige()` plot reset |
| Data | `services/game/store/` (new fn in `actions.py`) | `set_plot_stop(user, slot, stop: bool)`: owner only, requires `mastery_autoreplant` level > 0, requires a crop on the plot, `GameError` otherwise |
| Form | `models.py` | `GamePlotStopForm(slot: int ge=0 le=64, stop: bool)`, modeled on `GameSlotForm` |
| Route | `routers/game/index.py` | `POST /game/plot/stop`, `require_user`, via `_respond_action`, audit `game.plot.stop` |
| Audit | `services/audit/events.md` + `category_for` | add `game.plot.stop` (same category as `game.plant`) |
| Schema out | `schemas/game.py` | `GamePlotOut.stop_after_harvest: bool = False` (owner only, default safe); `serialize_plot` sets it |
| View | `templates/_game_grid.html` and `static/js/GameFarm.js::_plotHtml` | a small toggle button on growing/ready plots, shown only when the viewer is the owner and owns Continuous Delivery; label "Stop after harvest" / "Resume auto-replant". Server render and JS mirror must match (the file already warns about flash on divergence). Reuse the existing `data-game-action` form delegation, no new JS class |
| Agent | `services/devii/actions/catalog/game.py` | `game_plot_stop` Action (`requires_auth=True`, params `slot`, `stop`). Not confirm-gated: reversible and costs nothing |
| API docs | `docs_api/groups/game.py` | `endpoint()` entry with `sample_response` |
| Prose docs | `templates/docs/code-farm.html` | Continuous Delivery row: document the toggle and that Mastery has no refund |
| Notes | `services/game/CLAUDE.md`, `README.md` | document the flag and the "plant/replant claim resets it" invariant |
### Gaps found in the second review (all must be in the change)
- `prestige()` resets plots with an explicit dict in `_plots().update({...})` (actions.py ~611). It does not touch unknown columns, so the dict must add `"stop_after_harvest": 0` or a flag survives a Refactor.
- A stale `1` on an EMPTY plot (left after a no-replant harvest) must never be shown: `serialize_plot` gates `stop_after_harvest` on `crop_key` present AND viewer is owner AND `mastery_autoreplant` owned.
- Writers that put a crop on a plot are exactly two: `plant()` and `_try_autoreplant` (both `conditional_update_row` claims). Both SETs get `stop_after_harvest = 0`. `_create_plot` inserts need no change (column default via ensure-block, read with COALESCE).
- Existing tests that enumerate things and will fail until updated: `tests/unit/services/devii/actions/catalog.py` (game action name list), `locustfile.py` (route coverage, the locust-maintainer agent owns it, never run a load test), `events.md` (audit catalogue), `README.md` tool list (line ~223) and the `code-farm.html` endpoint table (~line 433).
- The toggle button is a new element in a dense plot card: needs CSS tokens only (no literals), must fit the smallest phone width, and needs a `screenshot` skill visual check in both server-rendered and JS-rendered forms.
- `notify_farm` already runs in `_respond_action`, so the live refresh needs no extra wiring.
- Interactions checked and unaffected: steal (never clears a plot), water, fertilize, CI Bot `_auto_harvest` (flag honoured the same way, via the post-clear re-read), Era/leaderboards, treasury/grant.
### Not in scope for Part A
No farm-wide off switch and no refund of Mastery points. The per-plot toggle is the requested fix.
## Part B: do not replant into a bad market (REVISED after second review: do not build the original)
### Measurement
Computed with the real `economy.realizable_harvest_coins` at the saturation floor (`market_factor` 0.4), payout minus planting cost:
| Prestige | shell | api | kernel | distsys | mlpipe | secfort |
|---|---|---|---|---|---|---|
| 0 | -1 | 0 | +240 | -480 | -3600 | -10800 |
| 50 | +54 | +1120 | +18240 | +56020 | +101400 | +229200 |
| 60 | +65 | +1344 | +21840 | +67320 | +122400 | +277200 |
Continuous Delivery needs prestige 50 (`MASTERY_UNLOCK_PRESTIGE`), where `prestige_multiplier` is 13.5. At that point even a fully saturated crop pays 11x or more its planting cost, for every crop in the catalog. Consequences:
- The "profit floor" (skip when payout <= cost) can never fire for a real owner. It is dead code. Drop it.
- A plain "skip when saturated" always makes the owner poorer: the idle plot earns 0, and a skipped replant also forfeits that cycle's XP, quest progress, weekly harvest count (grant eligibility, `harvests` and `fair_play` boards) and the market tick. Idle is never better than planting a saturated crop.
- So the original Part B would be collateral damage, not a fix.
### What to do instead
1. Ship Part A only. It already solves the stated problem: one free crop choice per plot, whenever the player wants it, with zero economic downside.
2. Optional, only if you still want saturation handling: an opt-in farm setting "when saturated, replant the best unsaturated unlocked crop instead". It never idles a plot. It needs a new pure function in `economy.py` that picks the crop (must define "best", for example highest realizable coins per second, among crops passing `crop_lock_reason`), used by both the serializer preview and `_try_autoreplant` (one-quantity-one-function rule). It changes which crop grows and its grow time, so it needs its own design pass and the Layer 1/2 property sweeps. Not recommended to bundle with Part A.
## Risk list
- **Behavior change on deploy:** Part A defaults to 0 for every existing plot, so nothing changes until a player clicks. There is no Part B behavior change (see revision above).
- **Missing column on first plant:** the claim SET names `stop_after_harvest`, so a DB where `init_db` has not added the column would fail every plant. `init_db` runs before serving, and tests start from a fresh DB through it; verify with the layer-3 setup that uses a genuinely fresh, un-pre-seeded farm.
- **NULL trap:** pre-existing plot rows will have `stop_after_harvest` NULL. Every read and every WHERE uses `COALESCE(stop_after_harvest, 0)`.
- **Prestige reset:** forget the column there and a stale `1` survives a Refactor. Covered by the claim-SET rule, but the reset dict must also write 0.
- **Visitors:** the flag must never be exposed to non-owners (`serialize_plot` gates it on `viewer == owner`).
- **JSON vs template:** a context key not on `GamePlotOut` is silently dropped from JSON. Add the field to the schema first.
## Verification plan
Persisted tests (matching tiers; the suite is run by the orchestrating session, serial):
- `tests/api/game/mutations.py`: toggle on, harvest, plot stays empty; toggle off, harvest, plot replanted; toggle on an empty plot is a 400; toggle without Continuous Delivery is a 400; a visitor cannot toggle; `plant` after a stop resets the flag; Refactor resets it.
- `tests/api/game/farm.py`: visitor view of an owner's plots never shows `stop_after_harvest`.
- `tests/e2e/game/index.py`: the toggle button renders and flips on click; server-rendered and JS-rendered markup agree.
Disposable scripts (scratch DB via `DEVPLACE_DATABASE_URL` and `DEVPLACE_DATA_DIR`, never production):
- Layer 3 race: real OS processes firing toggle against harvest on the same plot, from a genuinely fresh farm (flag NULL, not pre-seeded to 0). Assert the replant happens exactly when the toggle landed after `clear_plot`, never when it landed before.
- Layer 4: `pyflakes`/`ruff check` on every touched file.
Then the standard gates: clean `from devplacepy.main import app` import, em-dash grep on touched files, `make test` (all tiers) green, commit, and verify live on `127.0.0.1:10500`.
## Residual risk after the second review
- Part A is a small, additive change with one new column, one new route and one guarded early return. I found no economic, concurrency or permission collateral in it. "No collateral damage" is a claim about the plan, not a measurement: it is proven only by the race script and the full suite above, neither of which has been run.
- One accepted behavior: toggling "stop" on a plot that harvests in the same instant either stops the replant or returns "plot is empty". Both outcomes are correct and visible.
## Decisions needed from you
1. Part A: one-shot (recommended, flag cleared when the plot is next planted) or sticky until turned off?
2. Part B: drop it (recommended), or commission the "replant best unsaturated crop" option as a separate piece of work?
3. Fold in the crop-lock check fix in `_try_autoreplant` (docs promise it, code lacks it)?
nicknames.md
-101
View File
@@ -1,101 +0,0 @@
# How to Generate Unique, Realistic Usernames for a Forum
Accounts often have to be created for people who never picked a handle: bulk imports, migrations, data feeds. The source rarely carries a clean name. You get strings like `abner_page_by_abnercoimbre` or `via_rberger_2026-10-15_12_36_38_`. This article describes a method that turns such strings into names a person would actually pick, and keeps them unique and stable.
## The problem
Raw author strings often have the shape `<site> via <person>` or `<site> by <person>`:
```
lore.kernel.org via thang
jakearchibald.com by jaffathecake
decomp.dev via FedericoSchonborn
```
Using that string as the username gives ugly names. Worse, it gives the same person several accounts, because the same person appears next to different sites and each string looks like a different author.
Fix the identity first, then the name.
## Step 1: Decide who the person is
Before you generate anything, define the identity key, because a handle is only stable if the key is.
- Strip noise from the raw string: timestamps (`2026-10-15 12:36:38`) and relation words.
- If the string says `by` or `via`, the **person is the token after it**. The site comes before it.
- If the string is only a domain (`costa_jacobpalm_dk`), drop the top-level domain and keep the last meaningful label (`jacobpalm`).
- Use that cleaned person as the identity key, not the original string.
Now one person appearing next to ten sites is one account.
## Step 2: Create the handle once and store it
Generate a handle the first time you see an identity. Store the link between the identity and the account in a table with a unique constraint, for example `(source, identity) -> account`.
Every later lookup finds the link and uses the stored name. **The generator is never consulted twice for the same person.** Renaming later is how you end up with handles that drift, so do not do it.
Two guards matter:
- Only choose the name when the account is created. Later updates keep the stored name.
- Never touch accounts that already exist. Backfilling old names is a separate decision with its own risks.
## Step 3: Make the name look chosen
Real handles are not random strings. One empirical study of usernames on Google profiles (arXiv 1101.5578, 2011, Latin-alphabet names only) found:
- About 70% of usernames contain part of the user's real name.
- About 30% are plain first-plus-last concatenations.
- More than 18% add digits to the name, likely birth years or a number to get around a taken name.
That is the only solid distribution found. It says nothing about separators or humorous suffixes, so those weights are judgment, not data.
Pick one pattern per person with fixed weights:
| Pattern | Weight | Example |
|---|---|---|
| Plain name | 40% | `runxiyu` |
| Name plus a number (year, two-digit year, or a small number) | 30% | `hongminhee90`, `untrusem_2000` |
| Separator plus number | 8% | `jaffathecak-73` |
| Funny (prefix, suffix, or a nerd number) | 17% | `federicoschonborn_irl`, `lcapald_500` |
| Light leetspeak | 5% | `wr1ter0` |
The funny parts come from small word lists: prefixes like `its`, `dr`, `mr`; suffixes like `_dev`, `_irl`, `_ftw`; numbers like `42`, `1337`, `404`, `418`. Years run 1972 to 2006.
## Step 4: Make it deterministic
Seed the random generator from a hash of the source and the identity, for example `sha256(source:identity)`. The same person then always produces the same ordered list of candidates, which makes the system reproducible and testable.
If the first candidate is taken, move to the next. Finally fall back to a numbered suffix, so a free name always exists.
## Step 5: Enforce hard limits
Constants, not heuristics:
- Characters: letters, digits, `_` and `-` only.
- Length: 3 to 32.
- Base name: trimmed to 18 characters before decoration, so a suffix always fits.
- Case: lowercase, as people usually type handles.
- A bounded number of re-rolls before the fallback.
Test these with property checks across many inputs, including empty strings, very long strings, accented characters, and strings made only of separators. Every candidate must be valid.
## Step 6: Handle the exceptions
- **Deliberate pseudonyms.** If you replace names on purpose so the original is hidden, do not decorate those names. Mentions are rewritten to match the replacement, and any change would break the match.
- **Trusted senders.** If another system sends you clean handles, keep them. Apply this method only where the names cannot be trusted.
- **Collisions.** Compare names case-insensitively, since `Alice` and `alice` should not coexist.
## Known weaknesses
- The weights are constants, not tuned on real data from your own user base.
- A name that already ends in digits can look odd after a year is added (`spc476` becoming `spc4762005`).
- Changing the identity key means records created before the change no longer match their old links. They appear as new accounts, and the old ones remain as orphans. Plan a cleanup if that matters.
- The result is a plausible handle. It makes no claim about who the person is.
## Summary
1. Fix the identity first: extract the person, drop the site and the stamp.
2. Create the handle once, store the link, and never regenerate it.
3. Pick a pattern with weights based on how real people name themselves.
4. Seed the randomness from the identity so results are reproducible.
5. Enforce length, character and uniqueness rules.
6. Leave existing accounts and deliberate pseudonyms alone.
plan.md
-360
View File
@@ -1,360 +0,0 @@
See research_residental_proxies.md. Userss are by default opted out, but can opt in and download required tools to make it happen on their profile page.
We label this as donating to the devplace community ecosystem. We have a beautiful infrastructure that is very reliable, barely costs maintenance and even better, almost free.
Our search engine destroys searchxng. Also, our search engine has actually a fallback provided by jina. I bough jina in someday but then decided not to use their stuff what ended up
as a nice backup system for failing searches. But what do we want? We also want reddit, X and all other social media to be part of our search results. And that is only possible
when people contribute some of their resource. There are severa ways to do this. Choice is to members. We support all common options and you can configure yourself how much resource
you want to donate. We will also count the amunt of data i/o / request count and display it as donated by you. Also! For the sake of our new agent system on pravda.education, we really
needs this residental proy stuff to happen. Our platform destroys grok atm, but grok has the social media searches and they are important.
Please create a huge implementation plan of literally everything that needs to happen at the end of this file. Thats literally all for now.
---
# IMPLEMENTATION PLAN: Community Node Network ("Donate to the DevPlace Ecosystem")
Status: plan only, nothing built. Written against the repo as of 2026-10-14 and against `research_residental_proxies.md`.
Working name: **Nodes**. Code lives in `devplacepy/services/nodes/`, routers in `devplacepy/routers/nodes/`, user-facing label "Donate to the community".
## 0. Findings that change the shape of the plan
These come from reading the repo and the research file. Several of them contradict the framing in the brief, so they come first.
1. **There is no Jina code in this repo.** `grep -ri jina` matches only `plan.md`. The search engine is the external service `https://rsearch.app.molodetz.nl` (`crawl.RSEARCH_URL`, `services/devii/rsearch/`). The Jina fallback therefore lives in the rsearch codebase, not here. This repo only consumes rsearch from two places: `services/jobs/deepsearch/crawl.py::search_queries` and the Devii `rsearch*` tools. Where social search is implemented (here or in rsearch) is open decision D1.
2. **The repo already knows social sites block it.** `crawl.HOSTILE_DOMAINS` lists x.com, reddit.com, instagram, facebook, linkedin, tiktok, threads, youtube and skips them. Root `CLAUDE.md` records that reddit blocks the datacenter ASN outright and that the only fix is an IP change. So the problem is real and measured for Reddit. It is not measured for the others.
3. **A residential IP fixes IP-reputation blocks, not login walls.** Reddit public JSON endpoints work from a clean IP without login. X search, Instagram, LinkedIn, Facebook and TikTok mostly need a logged-in session. Donor nodes must NOT use the donor's cookies (privacy and liability), so those platforms stay mostly out of reach. X is limited to what is public (permalinks, oEmbed/syndication timelines). Promising "Reddit, X and all social media" would be a false claim. Phase 0 measures what actually works per platform before anything else is built.
4. **Several "social" sources need no donors at all**: Hacker News (Algolia API), Bluesky (public AppView), Mastodon (public instance APIs), Lobsters, StackExchange, GitHub, dev.to, YouTube oEmbed/transcripts. They are cheap, reliable, and cover a large part of what developers want. Build these as direct adapters first (Phase 6A). Nodes are only for the platforms that block datacenter IPs (Reddit first).
5. **Browser extensions cannot do TCP.** MV3 has no raw sockets. An extension can only run `fetch()`. So there are two fundamentally different transports, with different trust properties:
- **Tunnel mode** (native client): the server opens an end-to-end TLS session to the destination THROUGH the donor's TCP socket. The donor sees only ciphertext plus the destination host. The donor cannot read or tamper with content because they cannot forge the destination's certificate. Our `curl_cffi` Chrome fingerprint stays intact. This is the better mode.
- **Fetch mode** (extension): the donor's browser performs the HTTP request and returns the body. The donor sees the URL and plaintext response and CAN forge the response. Everything returned is untrusted and must be verified (section 5.6). TLS fingerprint is the donor's browser, which is fine.
6. **A plain web page or service worker cannot do this at all.** The research table rates them "Low"/"Poor". The reality is "impossible": a page cannot `fetch()` reddit.com cross-origin (CORS), and cannot open raw sockets. Both are dropped from scope. Documented so nobody revisits them.
7. **Open SOCKS5 is the wrong product.** The research recommends exposing a SOCKS5 listener. We do not need an open proxy. We need to reach a short, fixed list of public social domains for our own search. A destination-allowlisted exit (donor hard-refuses everything else) removes most of the abuse and liability surface the research file warns about (credential stuffing, ad fraud, illegal traffic). This is the single most important design decision and it is a clear rule, not a configurable maybe: **no open proxy, ever; allowlist enforced on server AND on the donor client.**
8. **Existing seams to reuse, none to rebuild:**
- `stealth.stealth_async_client(proxy=...)` and `outbound_proxy_url`: a local node-backed forward proxy plugs into this with zero caller changes.
- `messaging/tickets.py` + `ws_tickets`: single-use WS auth ticket flow (browsers cannot set WS headers).
- `pubsub` lock-owner WS pattern (`4013` retry) for the node socket.
- `BaseService` + `ConfigField` for the dispatcher service and admin knobs.
- `LiveViewRelayService.VIEWS` + private `user.{uid}.*` pub/sub namespace for live donor stats.
- `user_consents` / `CONSENT_KINDS` for opt-in (default opted out is already how `ai_third_party` works).
- `utils.track_action` / `ACHIEVEMENTS` / `BADGE_CATALOG` for donor badges.
- `GatewayUsageLedger.record_external` pattern for metering consumers.
- `devii` `CONFIRM_REQUIRED`, `docs_api.endpoint()`, `DOCS_PAGES`, `audit` categories + `events.md`.
9. **Legal and ToS exposure is real and is risk number one.** Reddit and X actively litigate and rate-limit scraping; ISPs restrict commercial use of residential lines; donors' IPs will get rate-limited or banned by the platforms. The plan includes a legal review gate (Phase 11) before any public launch. This is not legal advice and the plan does not assume the outcome.
10. **Unknown that blocks scoping:** what the "new agent system on pravda.education" is and how it will call search. The plan gives it a stable internal interface (section 6) but its requirements (latency, volume, platforms) are open question Q3.
## 1. Decisions needed from you before Phase 2 (everything else proceeds without them)
| # | Decision | Recommendation | Why |
|---|----------|----------------|-----|
| D1 | Where does social search live: in devplacepy, or in the rsearch codebase? | devplacepy owns the node fabric and exposes an internal HTTP face; rsearch calls it (or devplacepy does the social search itself) | Nodes are tied to DevPlace accounts, consent, and the ledger. rsearch has no accounts. |
| D2 | Native client language | **Go**, static binary, one file per OS/arch | Research recommends Go/Rust; zero runtime for Windows users. Adds a Go toolchain to the build. Alternative: Python zipapp (no new toolchain, but needs Python on the donor machine). |
| D3 | Extension distribution | Sideloadable zip first (audience is developers), store submission later | Stores scrutinize proxy-like extensions; sideload ships sooner and the audience can handle it. |
| D4 | Donation reward | XP + badges + leaderboard only. No coins, no money, no purchasable anything | Root rule: no payment path, no purchasable randomness. Keeps app-store conditionals untriggered. |
| D5 | Minimum donor age | Adults only (`age_band` gate) | A minor must not share an IP/line. Needs a decision on how the existing `age_band` is read for legacy accounts with `NULL`. |
| D6 | Is a data center or a cloud VPS allowed as a "donor"? | No: residential/mobile-ASN heuristic, admin override | The whole value is residential reputation. A VPS donor is just another datacenter IP. |
## 2. Specification first (your rule 7: everything has a declared spec)
Nothing else is written until the spec exists, because the server, the Go client, the extension, the docs and the tests all derive from it.
1. `devplacepy/services/nodes/spec.py`: the single source of truth, as frozen dataclasses (`FrameSpec`, `LimitSpec`, `DestinationSpec`, `AdapterSpec`, `ProtocolSpec`). It declares:
- protocol version (integer, with `min_supported`),
- every frame type with its fields, direction, max size,
- every hard cap in section 3,
- the destination allowlist (domain suffix -> adapter -> allowed methods -> allowed ports -> rate limit),
- donor-side hard refusals (non-public IPs, ports other than 80/443, non-allowlisted hosts).
2. `GET /nodes/spec.json` serves the spec so clients and tests read the same numbers. A contract test fails if the Go constants, the extension constants, the docs page, or the server drift from it.
3. `devplacepy/templates/docs/nodes-protocol.html` (generated tables from the spec, not hand-copied) and `docs/nodes.html` (user-facing). Registered in `DOCS_PAGES`.
4. A short `NODES.md` is NOT created; the nested `devplacepy/services/nodes/CLAUDE.md` is the engineering record, per repo convention.
## 3. Hard caps and floors (nothing unbounded)
All values below are starting ceilings, declared in `spec.py` and surfaced as `ConfigField`s only where an admin may lower them. Static ceilings, no autotuning.
| Cap | Initial value | On hit |
|-----|---------------|--------|
| Devices per user | 5 | 409 with reason |
| Connected nodes total (lock-owner worker) | 2000 | WS close `4029`, counter `nodes_rejected_capacity` |
| WS frame size | 256 KiB | close `1009`, counter |
| Response body per job | 2 MiB | truncate + flag `truncated=true`, never silently |
| Concurrent jobs per node | 2 (donor may lower to 1) | scheduler skips node |
| Per node, per destination domain | min 6 s between requests | scheduler skips node |
| Global per destination domain | admin-set rps ceiling | queue, then 429 to caller |
| Job timeout | 25 s | mark failed, count against node health |
| Job queue depth | 500 | reject new with 429 + counter |
| Caller wait | 15 s default, 60 s max | `FabricUnavailable` |
| Per caller-owner daily fabric requests | admin-set (default 200 user, 0 guest) | 429, no donor load |
| Donor daily bandwidth (donor-configurable) | default 500 MB, max 20 GB | node goes idle until reset |
| Heartbeat | ping every 20 s, evict after 2 missed | reconnect |
| Reconnect backoff | 1 s to 60 s, full jitter | n/a |
| Ticket TTL | 30 s single use (existing) | n/a |
| Result cache | 10 min TTL, 256 MiB total, 4 MiB per entry | LRU evict |
| Usage daily buckets | keep 400 days, rolled into lifetime totals first | prune only after rollup verified |
| Node event log | 30 days, 50k rows | prune |
**Floors (machines hold, humans decide):**
- The system may auto-**suspend** a misbehaving node (reversible, timed, e.g. circuit breaker after N consecutive failures or an integrity mismatch). It may never auto-**ban** a user or delete a ledger row. Hitting a ban threshold creates a flag for an admin and fires an admin notification.
- Lifetime donation totals live on the donor row and are never decreased by pruning.
- Protected destinations: the allowlist can only be changed by the primary admin and every change is audited.
- A global kill switch (`nodes_enabled` site setting) disconnects every node within one tick. It is the first thing built in Phase 4 and tested first.
## 4. Data model
All tables ensured in `database/schema.py::init_db` (full column set, because `dataset` only creates the columns of the first insert), indexes with `CREATE INDEX IF NOT EXISTS`, every INSERT writes `deleted_at: None, deleted_by: None` where the table is soft-deletable. Helpers in a new `database/nodes.py`, exported through `database/__init__.py` like `skills.py`.
| Table | Purpose | Soft delete? | Notes |
|-------|---------|--------------|-------|
| `node_devices` | one row per registered client install: `uid`, `user_uid`, `label`, `kind` (`native`/`extension`/`container`), `secret_hash`, `secret_prefix`, `platform`, `client_version`, `protocol_version`, `status` (`active`/`suspended`/`revoked`), `suspended_until`, `suspend_reason`, `country`, `asn`, `asn_class` (`residential`/`mobile`/`datacenter`/`unknown`), `ip_hash`, `last_seen_at`, `created_at` | yes (revoke = soft delete, cascade stamp) | index `(user_uid, deleted_at)`, `(status, last_seen_at)` |
| `node_settings` | per-user donation prefs: `user_uid` (unique), `enabled`, `bandwidth_mb_day`, `max_concurrent`, `active_from`, `active_to`, `timezone`, `allowed_platforms` (JSON), `pause_on_battery`, `updated_at` | no (preference row) | enabled is subordinate to the consent, like `ai_correction_enabled` |
| `node_usage_daily` | per `(device_uid, day)` counters: `requests_ok`, `requests_failed`, `bytes_up`, `bytes_down`, `cache_hits_served`, `integrity_failures` | no (derived, hard GC) | unique `(device_uid, day)`; upserted with one atomic `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` (the `_add_usage` precedent) |
| `node_donors` | lifetime totals per user: `user_uid` (unique), `requests_ok`, `bytes_total`, `first_donated_at`, `last_donated_at`, `xp_awarded_today`, `xp_day` | no (derived) | survives bucket pruning; this is what the profile card and leaderboard read |
| `node_requests` | the cross-worker job queue: `uid`, `caller_kind`, `caller_id`, `url`, `method`, `adapter`, `mode` (`tunnel`/`fetch`/`any`), `status` (`queued`/`assigned`/`done`/`failed`/`expired`), `device_uid`, `result_status`, `result_bytes`, `result_body` (capped), `error`, `created_at`, `assigned_at`, `finished_at` | no (ephemeral, hard GC after 10 min) | partial index on `status IN ('queued','assigned')` |
| `node_events` | operational log: connects, disconnects, suspensions, verification results, refusals | no (hard GC, 30 days) | feeds admin view, not audit |
| `node_flags` | items needing a human: integrity mismatch bursts, datacenter-ASN donor, abuse reports against a node | yes | admin resolves; the floor rule's landing place |
Related registrations (the repo fails the suite otherwise):
- `database/moderation.py`: none of these carry user-generated content visible to others, so each goes in `UNREPORTABLE_TABLES` WITH its reason. `node_devices.label` is owner-only text; confirm it is never rendered to anyone else, or it becomes reportable.
- `SOFT_DELETE_TABLES`: `node_devices`, `node_flags`.
- `services/moderation/deletion.py::OWNED_TABLES`: add `node_devices`, `node_settings`, `node_donors` so account deletion cascades. Add `node_usage_daily` via the device join. Revoke live sockets in the same call.
- `user_consents`: new `CONSENT_KINDS["bandwidth_sharing"]`. **Never granted by default, never backfilled.** A new consent is enforced at ONE choke point (section 5.3), never at N call sites.
- Site settings seeded in `init_db` `operational_defaults`: `nodes_enabled` (default `0`), `nodes_max_total`, `nodes_daily_quota_user`, `nodes_daily_quota_guest`, `nodes_cache_ttl_seconds`, `nodes_social_via_nodes` (route hostile domains through nodes: default `0`).
## 5. Server
### 5.1 Package layout (OO, one responsibility per class, one class per file where it is a class)
```
devplacepy/services/nodes/
CLAUDE.md engineering record (written in Phase 12, started in Phase 3)
spec.py ProtocolSpec and friends (section 2)
registry.py NodeRegistry: connected sessions, capacity, lookup by uid/user
session.py NodeSession: one WS, frame codec, in-flight jobs, health, heartbeat
frames.py Frame encode/decode + validation against spec
consent.py ConsentGate: THE single choke point (5.3)
policy.py DestinationPolicy: allowlist, port/IP rules, per-domain limits
scheduler.py NodeScheduler: pick a node (fairness, diversity, caps)
broker.py JobBroker: enqueue, assign, await result, expire (DB-backed)
dispatcher.py NodeDispatcher: lock-owner loop moving node_requests to sessions
tunnel.py TunnelBridge: local forward-proxy listener -> tunnel frames
fetcher.py FetchBridge: fetch-mode request/response framing
verifier.py NodeVerifier: canary, ASN class, redundancy checks
ledger.py UsageLedger: atomic counters, rollups, XP award hook
cache.py ResultCache: bounded, URL-hash keyed
circuit.py CircuitBreaker: per node and per destination
fabric.py NodeFabric: the facade every consumer calls
service.py NodesService(BaseService): reconcile, metrics, prune, relay hook
adapters/
base.py SearchAdapter (abstract): build_request, parse_response, normalize
reddit.py x.py ... one class per platform
direct/ hackernews.py bluesky.py mastodon.py lobsters.py stackexchange.py
```
### 5.2 Routers (`devplacepy/routers/nodes/` package, mounted at `/nodes` in `main.py`)
| Route | Auth | Purpose |
|-------|------|---------|
| `GET /nodes/spec.json` | public | protocol spec for clients and tests |
| `GET /nodes/client-version` | public | latest client versions, `min_protocol`, sha256 per artifact |
| `GET /nodes/download/{artifact}` | public | native binaries, extension zip, systemd unit, container compose (static files from `DATA_DIR/node_clients/`, sha256 listed) |
| `POST /nodes/devices` | `require_user` + consent | create device, returns the secret ONCE |
| `POST /nodes/devices/{uid}/rotate` | owner | new secret, old revoked immediately |
| `POST /nodes/devices/{uid}/delete` | owner or admin | soft delete + kick socket |
| `POST /nodes/ticket` | device secret | exchange secret for a 30 s single-use WS ticket |
| `WS /nodes/ws` | ticket | the node socket; lock-owner gated, `4013` fast retry like `/pubsub/ws` |
| `POST /nodes/internal/fetch` | admin / API key, service accounts | the HTTP face of `NodeFabric.fetch` for agent systems |
| `GET /nodes/stats` | public | aggregate only: total donated bytes, requests, active donors (no per-user data) |
| `GET /nodes/leaderboard` | public | top donors, opt-in display |
| `POST /profile/{username}/donation` | owner | settings (bandwidth, hours, platforms) |
| `GET /admin/nodes`, `POST /admin/nodes/...` | admin | pool health, flags, suspend/unsuspend, allowlist, kill switch |
Rules from the repo applied to every route: `respond(request, template, ctx, model=XOut)` for the HTML+JSON faces, `*Out` schemas in `schemas/nodes.py` (a context key missing from `XOut` is silently dropped from JSON), Pydantic `Form` models in `models.py`, `RedirectResponse(..., status_code=302)` literal, no `respond()` ctx key colliding with a Jinja global (use `viewer_is_admin`), specific paths before catch-alls, `is_senior_admin` on every per-user admin mutation, delete = owner OR admin.
### 5.3 The consent choke point (`ConsentGate`)
One function, `ConsentGate.allows(user_uid) -> Verdict(reason)`, true only when ALL of: `bandwidth_sharing` consent granted and not withdrawn, account active (not suspended, not in deletion grace), adult `age_band` (D5), `node_settings.enabled`, global `nodes_enabled`. It is called in exactly three places:
1. `POST /nodes/devices` and `POST /nodes/ticket` (refuse to mint).
2. `WS /nodes/ws` accept (refuse to connect).
3. `NodeScheduler.eligible()` (a node whose owner stopped qualifying is never assigned work, and `NodeRegistry` kicks the socket on the next tick).
Withdrawing the consent takes effect within one dispatcher tick, not at next reconnect. Consent change is owner-only, exactly like `routers/profile/consent.py` (reuse `_owner_only`). **Devii may read status and may withdraw, but must not be able to grant** (a chat agent granting a bandwidth-sharing consent is not a consent). Verify how the existing `set_consent` Devii tool treats grants and restrict this kind there.
### 5.4 WebSocket and multi-worker model
- `/nodes/ws` is gated to the service-lock owner worker with the `4013` fast-retry close, so all node sockets converge on one process (precedent: `/pubsub/ws`, `/devii/ws`). This keeps `NodeRegistry` a single in-memory truth. Trade-off, stated: one worker carries all sockets, hence the 2000 cap, and a crash drops every node for a few seconds until they reconnect. Acceptable because reconnect is built in.
- Callers on ANY worker enqueue into `node_requests` (DB). The lock-owner `NodeDispatcher` ticks fast while at least one node is connected and a request is queued (target 200 ms, indexed partial query; the cost is measured in Phase 0, not assumed), assigns, and writes the result back. Callers `await` by short-sleep polling the row. A same-process fast path skips the table when the caller is already on the owner. This is the `JobService` philosophy (DB-backed, durable enough, no inter-worker RPC) with a tighter tick.
- Large bodies never go through SQLite long term: `result_body` is capped (2 MiB) and hard-GC'd after 10 minutes; the durable copy is the `ResultCache` entry, bounded in bytes.
- nginx: add an upgrade `location = /nodes/ws` to `nginx/nginx.conf.template` (the catch-all `location /` strips upgrade headers, and the file already has one per WS route). **Also verify on the `devplace.net` front host**, whose own proxy and SSH tunnel have separate idle timeouts; test both hostnames per the production-diagnosis procedure, never with `--resolve`.
### 5.5 Frame protocol (summary; the authoritative version is `spec.py`)
Client to server: `hello{protocol, client_version, kind, platform, caps, limits}`, `pong`, `job_result{job_id, status, headers, body_b64, truncated, timings}`, `job_error{job_id, code}`, `tunnel_data{conn_id, b64}`, `tunnel_close{conn_id}`, `usage{bytes_up, bytes_down}`.
Server to client: `welcome{device_uid, limits, allowlist_version, settings}`, `ping`, `job{job_id, method, url, headers, timeout}`, `tunnel_open{conn_id, host, port}`, `tunnel_data`, `tunnel_close`, `settings_update`, `kick{reason}`.
Every frame is validated against the spec on both ends. An unknown type or an oversize frame closes the socket with a loud reason and a counter (never silently ignored).
### 5.6 Trust, verification and integrity
Fetch-mode donors can forge content, and forged content flows into search results and into the AI agent. Controls, in order of strength:
1. **Prefer tunnel mode** whenever a tunnel-capable node is available: TLS makes tampering impossible. Fetch mode is only used for extension donors.
2. **Never trust fetch-mode bodies without checks**: structural validation by the adapter (must parse as the platform's JSON/HTML shape), size sanity, and **sampled redundancy**: a configured fraction of fetch-mode jobs (plus 100 % for a node's first N jobs) are also fetched through a second independent node or directly; a content-hash or structural mismatch increments `integrity_failures`, suspends the node, and writes a `node_flags` row.
3. **Provenance on every result**: results carry `via=tunnel|fetch|direct` and the agent/search layer can down-weight or label fetch-mode content.
4. **Canary / liveness proof**: on connect, the verifier makes the node fetch `GET /nodes/canary/{nonce}` (served by us, records the observed source IP). Proves egress really goes out of the donor's own line, detects a node that is itself behind another proxy, and lets us compare the observed IP with the WS peer IP.
5. **ASN class**: classify residential/mobile/datacenter via an offline ASN database file under `DATA_DIR` (no third-party lookup, so no donor IP leaves our server). Datacenter donors get `asn_class=datacenter` and are excluded from the pool by default (D6) with a flag for an admin.
6. Donor client strips cookies and auth: extension uses `credentials: "omit"` and never attaches the donor's session; native client sends only the headers the server specified.
### 5.7 Donor-side safety is enforced on the donor, not only on the server
A compromised or buggy server must not be able to point a donor's machine at their LAN. The clients enforce, independent of any server frame:
- allowlisted hosts only (list baked in per release and refreshed from the server, where a refreshed list can only be equal or narrower than the baked-in superset),
- ports 80 and 443 only,
- resolve DNS locally, refuse private/loopback/link-local/CGNAT/multicast addresses, then connect to the resolved IP (defeats DNS rebinding),
- no redirects to a non-allowlisted host,
- the donor's configured bandwidth, concurrency and schedule caps, applied locally even if the server sends larger ones.
## 6. Consumers: how search gets the data
`NodeFabric` is the only thing consumers import:
```
NodeFabric.fetch(url, *, owner, mode="any", cache=True, timeout=...) -> FabricResult
NodeFabric.social_search(query, *, platforms, owner, limit) -> list[SocialHit]
NodeFabric.available() -> FabricStatus # nodes online, per-platform health
```
Resolution order (each step loud when it fails, never silent):
1. `ResultCache` hit.
2. **Direct adapters** for platforms that need no donors (Hacker News, Bluesky, Mastodon, Lobsters, StackExchange, GitHub, dev.to).
3. **Nodes** for allowlisted blocked platforms (Reddit first), tunnel mode preferred.
4. Configured `outbound_proxy_url` if the admin has one.
5. rsearch (existing, includes its own Jina fallback).
6. `FabricUnavailable` with a reason code; the caller reports "social results unavailable", never invents any.
Integration points, one commit each:
- `services/jobs/deepsearch/crawl.py`: `search_queries` gains a social bucket merged by the existing `_interleave`; `HOSTILE_DOMAINS` stops being a static skip list and becomes `fabric.can_serve(url)`: a hostile URL is skipped only when the fabric cannot serve it. `fetch_page` routes those URLs through `NodeFabric.fetch`. The `rsearch` NDJSON ledger frame gets a sibling `fabric` frame so `DeepsearchService._run_worker` meters it like `record_external`.
- Devii: tools `social_search`, `fetch_social_page` (`requires_auth=True`, metered, guests refused), plus owner tools `node_status`, `node_settings_get`, `node_settings_set`, `node_device_list`, `node_device_revoke` (`CONFIRM_REQUIRED` + declared `confirm` param, per the rule). Update `SYSTEM_PROMPT` so the model prefers platform data, then `social_search` only when asked about outside discussion, and never claims a platform it cannot reach.
- `POST /nodes/internal/fetch`: the stable interface for the pravda.education agent system and any external caller. Auth: API key / Bearer, rate-limited, per-owner daily quota, documented.
- Cost and fairness: every fabric call is metered into a `fabric_usage` view per caller (requests, bytes, cache hit ratio) so a single caller cannot drain the donor pool; caller quota is checked before any donor is touched.
## 7. Clients
All clients speak the same protocol from `spec.json`. Conformance is tested against the Python reference client (Phase 10), not by eyeballing.
### 7.1 Native node (D2: Go)
- Single static binary, linux/amd64, linux/arm64, darwin/arm64, darwin/amd64, windows/amd64. Tunnel mode (TCP relay) and fetch mode.
- Config file + flags; `devplace-node login` takes the one-time device secret; `devplace-node status`; `devplace-node run`.
- Reconnect with full-jitter backoff, 20 s ping, local bandwidth/schedule/concurrency enforcement, battery-aware pause on laptops, quiet by default, structured log with a per-day totals line.
- Packaging: systemd **user** unit (`loginctl enable-linger` documented), Windows Service/scheduled task, macOS LaunchAgent, and a Docker/Podman compose snippet (the repo already ships `docker-compose.podman.yml`, so container users are a known audience).
- Built reproducibly from a `nodes/client/` directory with a `make node-client` target; artifacts + sha256 land in `DATA_DIR/node_clients/`; the version manifest is generated, not hand-written.
### 7.2 Browser extension (MV3, Chrome + Firefox)
- Fetch mode only. Service worker holds the WS; 20 s keepalive ping plus a `chrome.alarms` reconnect backstop; state in `chrome.storage.session`.
- `fetch()` with `credentials: "omit"`, header rewriting via `declarativeNetRequest` for the headers the spec allows, allowlist enforced before every request, response size cap.
- Honest, prominent disclosure on first run: what is shared, what is not, the on/off switch, live counters.
- Distribution per D3.
### 7.3 Later (explicitly out of this plan's first release)
Tauri tray app, signed auto-updates, mobile. Mobile stays out: iOS forbids the background behaviour and Android is restricted, as the research notes.
## 8. Frontend
Rules: responsive down to small phones (a wide button group becomes a dropdown), ES6 modules one class per file instantiated on `app`, tokens only in CSS, no literals, no `var()` fallbacks, `static_url`/`assetUrl` for assets, dates via `local_dt`, no `data-render` on server content, presence/pubsub via the existing managers.
1. **Profile tab "Donate"** (owner-only; others see only a donor badge and lifetime totals):
- plain-language explainer first ("What you share, what you never share"), the consent toggle second, everything else only after consent. Product stays simple for the member: one switch, three sliders, a download card.
- bandwidth per day, concurrent connections, active hours, platform checkboxes, pause-on-battery.
- devices list with last seen, country/ASN class, bytes today, rotate and remove.
- **"Donated by you" card**: requests served, data transferred (up/down), first/last donated, this month vs lifetime. Server-counted only.
- download card: per-OS choice rendered as a dropdown on narrow screens, checksums shown, copy-paste install lines (systemd, docker), extension zip.
- live numbers via a private pub/sub topic `user.{uid}.nodes` published by the lock-owner relay (add one row to `LiveViewRelayService.VIEWS`, re-checking the owner in the compute callable like the workspace topic), with a 30 s HTTP fallback poll.
2. Files: template `templates/_node_donation.html` partial included by `profile.html`; `static/js/NodeDonation.js` (`app.nodeDonation`); CSS `static/css/nodes.css` using existing tokens; reuse `Http`, `Poller`, `Tabs.js`, `dp-dialog` for confirmations.
3. **Public surfaces**: landing/footer line "The community has donated X GB across Y requests" from `GET /nodes/stats` (aggregate only), a `Top donors` section on the leaderboard (opt-in display), donor badge via `_award_badge.html` / `Avatar.badgeElement` reuse.
4. Admin: `/admin/nodes` (templates `admin_nodes.html`): pool size by country/ASN class, per-platform success rate, queue depth, flags, suspend/unsuspend, caps, allowlist, kill switch; live via `admin.nodes` topic row in `VIEWS`; `ServiceMonitor` already covers the service tab.
## 9. Gamification, incentives and anti-fraud
- `ACHIEVEMENTS`: `node.first_donation`, `node.1gb`, `node.10gb`, `node.100gb`, `node.30_days`, in `BADGE_CATALOG` group "Community".
- XP is awarded only for server-verified successful jobs, with a hard daily XP cap per donor (`node_donors.xp_awarded_today`), so synthetic traffic cannot farm progression. Through `award_rewards` only (never wrapped in `background.submit`).
- Counters are written server-side from what the server actually relayed; the donor's self-reported `usage` frame is advisory and is compared, never trusted.
- Self-dealing guard: a donor's own account cannot be the caller whose jobs are assigned to that donor's nodes for reward purposes.
- No money, no coins, no random rewards (D4).
## 10. Audit, privacy, safety, compliance
1. **Audit events** (`events.md` + `category_for` + recorder at the mutation point, via the `audit-event` skill): `node.consent.grant`, `node.consent.withdraw`, `node.device.create`, `node.device.rotate`, `node.device.delete`, `node.settings.update`, `node.suspend`, `node.unsuspend`, `node.flag.resolve`, `node.allowlist.update`, `node.killswitch`. Connects/disconnects/jobs are metrics and `node_events`, not audit rows (volume).
2. **Privacy (this is a new category of personal data plus a third-party disclosure, so it ships WITH the feature)**: update `templates/docs/privacy.html` and the app-store privacy declarations. State: IP address and approximate location are processed; only a salted hash and coarse country/ASN class are stored; retention periods for `node_events`/hashes; what the donor can see (destination host, and in fetch mode the URL and body); that the platform's content passes through the donor machine; the explicit statement that the donor's cookies and accounts are never used. Plain-English donor terms page, ISP-terms warning, consent text, and a visible off switch.
3. **App-store conditionals** (root rule): no payment path, no purchasable randomness, no cross-app tracking introduced. Hide the donate UI inside any app-store shell build. Verify against current store policy before submission rather than assuming.
4. **Abuse response**: report a node/destination path, admin suspend within one tick, allowlist narrowing, per-destination circuit breaker, the kill switch, and a documented contact. Because the allowlist is destination-restricted there is no path for credential stuffing, ad fraud or arbitrary traffic through a donor.
5. **Maturity/moderation**: fetched third-party content is untrusted. It goes through `render_content` (server) or `DOMPurify` (client) like everything else and through the existing content filter when it is persisted anywhere. Fabric results are never rendered as raw HTML.
## 11. Phased delivery
Each phase ends with: clean import, per-language checks, em-dash scan (character and entity), `pyflakes`/`ruff` on touched files, the four-layer verification where listed, the full `make test` (all three tiers), then commit (hook auto-bumps the version), deploy with `make docker-reload`, and verification live on both hostnames. A change that is not running is not done.
**Phase 0: Measure before building (no production code).** Scratch scripts against temp DB only.
- Real-machine experiment: a prototype WS relay (use `pywssocks` / the `residential-proxy-aggregator` design as reference) with one residential node and one datacenter node. Measure Reddit `.json` and `old.reddit.com` success rate per IP class, rate-limit thresholds per IP, X permalink/oEmbed/syndication behaviour without login, Instagram/LinkedIn/Facebook/TikTok reality. Output: a measured per-platform viability table committed to the plan. Claims without a number stay labelled "unmeasured".
- Tunnel-mode feasibility: confirm `curl_cffi` through an HTTP-CONNECT bridge preserves the Chrome fingerprint end to end and that the cleartext-HTTP/2 gotcha in the repo does not apply (https only).
- Capacity: simulated-node harness (our own, outside `locust`) for 100/500/2000 sockets on one worker: memory, CPU, frame latency, dispatcher tick cost against SQLite. These numbers set or replace the caps in section 3.
- Reconnect/idle behaviour through BOTH edges: `pravda.education` (molohttp) and `devplace.net` (front host + SSH tunnel). Measure the actual idle timeout of each.
- Exit criteria: D1, D2, D3, D5, D6 decided; viability table written; caps confirmed or changed.
**Phase 1: Specification and contracts.** `spec.py`, `/nodes/spec.json`, docs protocol page skeleton, contract-test scaffolding in `tests/unit/services/nodes/`.
**Phase 2: Data layer.** Tables, indexes, helpers in `database/nodes.py`, `CONSENT_KINDS` entry, `OWNED_TABLES`, `UNREPORTABLE_TABLES` with reasons, soft-delete registrations, settings seeds, `UsageLedger` with atomic upserts. Four-layer verification applies here (counters + daily caps are a spendable resource): property checks on cap arithmetic, stateful fuzz of increment/rollup/prune with invariants (totals never decrease, daily never exceeds cap), real multi-process race on the daily-cap `UPDATE ... WHERE` using `COALESCE(col, 0)` and a freshly-created unset row (not pre-zeroed).
**Phase 3: Registry, session, consent gate, kill switch.** `NodeRegistry`, `NodeSession`, `frames.py`, `ConsentGate`, `nodes_enabled` kill switch (built and tested first). Routers for devices/ticket/WS handshake only. No job traffic yet. nginx upgrade location. Reuse of the ticket mechanism by adding a `scope` column to `ws_tickets` (default `messages`) instead of a parallel table.
**Phase 4: Broker, scheduler, dispatcher, policy, circuit breakers, cache.** Job lifecycle `queued -> assigned -> done|failed|expired`, fairness, per-domain limits, queue cap with 429. Four-layer verification: property tests on scheduler fairness and cap enforcement; stateful fuzz of connect/disconnect/assign/complete/expire with invariants (a job is assigned to at most one node, a node never exceeds its concurrency, a withdrawn consent is never assigned work); real-process concurrency test of two dispatchers racing one slot (must be impossible by construction, since only the lock owner dispatches, and proven anyway).
**Phase 5: Python reference client + test harness.** A small reference node used by the api/e2e tests and as the protocol conformance oracle. Includes fault injection: dropped frames, slow nodes, oversize bodies, forged bodies, DNS-rebinding attempt, private-IP target.
**Phase 6A: Direct adapters** (no donors): Hacker News, Bluesky, Mastodon, Lobsters, StackExchange, GitHub, dev.to into `NodeFabric.social_search`, DeepSearch and the Devii tools. This ships user value before any donor exists.
**Phase 6B: Node-backed adapters**: Reddit first (per the Phase 0 table), then each additional platform only if Phase 0 measured it viable. `NodeVerifier` redundancy sampling for fetch mode.
**Phase 7: Native client (Go) + packaging** (systemd user unit, launchd, Windows service, compose). Cross-compile target, checksums, version manifest, conformance run against the reference oracle.
**Phase 8: Browser extension.** MV3 Chrome + Firefox build, disclosure screen, keepalive/alarms, conformance run, sideload zip served from `/nodes/download`.
**Phase 9: Frontend and admin.** Profile Donate tab, donated-by-you card, devices, downloads, live topic, leaderboard section, landing stat, `/admin/nodes`. Screenshot-verify with the `screenshot` skill at phone and desktop widths.
**Phase 10: Devii, docs, SEO, CLI.**
- Devii actions in `actions/catalog/nodes.py` (owner tools above), dispatcher/registry wiring, `CONFIRM_REQUIRED` for revoke, `requires_auth` matched to route guards, tool-group registration so the admin tool-prefs UI lists them.
- `docs_api/groups/nodes.py` `endpoint()` entries with `sample_response`; `DOCS_PAGES` for `nodes.html` and `nodes-protocol.html`; role-gating correct (protocol page admin/developer tier).
- SEO: Donate tab `noindex`; any public stats page indexable and in the sitemap.
- CLI: `devplace nodes list|stats|suspend|unsuspend|prune|clear`, destructive commands support `--dry-run` per the existing `accounts prune` convention (flag the conflict with the global dry-run-by-default rule to you if you want the default flipped; consistency with the existing CLI is the project rule).
- README, `devplacepy/services/nodes/CLAUDE.md`, a row in the root `CLAUDE.md` Subsystem map (a genuinely new architectural area), `locustfile.py` untouched for the WS (locust-maintainer: the node socket is deliberately excluded and the HTTP read routes get weighted tasks).
**Phase 11: Hardening and launch gate.**
- Legal review gate: platform ToS exposure per platform, donor terms, ISP warning text, jurisdiction. Launch is blocked on a recorded decision.
- Load and soak: 24 h soak with simulated nodes, reconnect storms (all sockets drop at once, then return), full-disk and DB-lock behaviour, worker restart mid-job. Numbers replace assumptions in `CLAUDE.md`.
- Security-maintainer pass (allowlist bypass, SSRF on the server's own fetch path, ticket replay, device secret leakage, admin seniority on node moderation).
- Full maintenance fleet (`fleet` skill): audit, background, devii, docs, dry, fanout, frontend, locust, security, seo, style, test dimensions.
- Rollout: `nodes_enabled=0` by default. Stage 1: your own machines only (including a residential line) for a week, measuring success rate, latency, donor-side CPU and bandwidth. Stage 2: invite-only developers. Stage 3: open, still behind per-caller quotas. The kill switch stays one click away.
**Phase 12: Documentation of record.** `devplacepy/services/nodes/CLAUDE.md` states what the system does today, with gaps listed next to the code that will close them (no roadmap cosplay). Measured numbers only, each labelled with the machine it came from.
## 12. Test plan (by tier, mirroring the path)
- `tests/unit/services/nodes/`: spec invariants and contract (docs, clients, server agree), frame codec, `DestinationPolicy` (allowlist, ports, private IPs, redirects, DNS rebinding), scheduler (fairness, per-domain interval, concurrency, bandwidth caps), circuit breaker, cache bounds and LRU, ledger math, ASN classification, XP daily cap, adapters (parse recorded fixtures, malformed input, forged-structure rejection).
- `tests/api/nodes/` (mirror the URL path): consent gating on every entry point, device create/rotate/revoke, secret shown once, ticket single-use and expiry, WS handshake with the reference client, full job round trip, withdrawal kicks the socket within a tick, kill switch, daily quotas, 429 paths, account deletion cascade, JSON and HTML faces for every route, admin seniority, `/nodes/stats` leaks no per-user data, `/nodes/internal/fetch` auth.
- `tests/e2e/profile/` (Playwright, `wait_until="domcontentloaded"`): enable flow with consent, device creation and one-time secret, settings persistence, live counter update, withdraw flow, responsive layout at phone width, owner-only visibility. Any test flipping a global setting restores it in `try/finally`.
- Rigorous verification scripts (disposable, temp DB via `DEVPLACE_DATABASE_URL`/`DEVPLACE_DATA_DIR`, never `data/`): the four layers from root `CLAUDE.md` for the ledger, scheduler and job state machine.
- The test hook that guards the production DB applies: no test or script touches `data/devplace.db`.
## 13. Risks, stated plainly
| Risk | Likelihood | Mitigation | Residual |
|------|-----------|------------|----------|
| Platform ToS/legal action against scraping | real | allowlist, legal gate, per-domain rate limits, kill switch | cannot be engineered away |
| Donors' IPs get rate-limited or banned by platforms | high on Reddit | per-donor-per-domain caps, caching, consent text says so | some donors will be unhappy; transparent counters help |
| X/Instagram/LinkedIn/TikTok unreachable (login walls) | high | Phase 0 measures; promise only what is measured | those platforms stay limited |
| Fetch-mode donor forges content | possible | tunnel preferred, structural validation, sampled redundancy, provenance labels, suspension | extension donors are lower trust, labelled as such |
| Small donor pool means flaky results | high at launch | cache, fallback chain, honest "unavailable" | search stays at today's quality, not worse |
| Single worker holds all sockets | by design | cap, fast reconnect, measured capacity | brief drop on worker crash |
| Donor privacy incident | low-impact by design | no cookies, hashed IP, destination allowlist, docs | disclosure must stay accurate |
| Scope creep into an open proxy | tempting | the "no open proxy" rule is in `spec.py` and the contract test | needs reviewer discipline |
## 14. Open questions for you
- Q1: Confirm D1 to D6 above (a one-line answer each is enough).
- Q2: Is there any residential line you control for the Phase 0 measurement, or do we recruit three donors for it?
- Q3: What does the pravda.education agent system need from search (volume, latency, which platforms, guest access)? It sets the daily quotas and the `/nodes/internal/fetch` contract.
- Q4: Should donor identity be public (leaderboard, badge) by default or opt-in? The plan assumes opt-in display.
- Q5: Who owns the legal review, and is a launch without it acceptable for the invite-only stage?
pyproject.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "devplacepy"
version = "1.1.5"
version = "1.1.6"
description = "DevPlace - The Developer Social Network"
requires-python = ">=3.12"
dependencies = [
quality_project.md
-199
View File
@@ -1,199 +0,0 @@
<!-- doc: quality_project.md | version: 1.0.0 | updated: 2026-10-11 -->
# Quality Project Playbook: What Belongs in AGENTS.md
Generic companion to `quality_software.md` and `quality_versioning.md`.
It defines what an agent instruction file must contain so that every
agent (and every human) develops the same way and the project stays
consistent and current.
Naming note: this repo uses `CLAUDE.md` instead of `AGENTS.md`, but
the system is what matters, not the filename. Three layers, loaded
by scope:
- Personal base rules (DevPlace: `~/.claude/CLAUDE.md`). Applies to
every project for one operator: safety, proof, craft, ops habits.
- Root project file (DevPlace: `CLAUDE.md`, ~440 lines). Applies to
the whole repo regardless of area touched. Always loaded, so it
stays proportional to global rules only.
- Nested subsystem files (DevPlace: ~30 `CLAUDE.md` files next to
the code). Loaded only when a file under that directory is read
or edited. Deep mechanics, helpers, pitfalls, worked examples.
If your tooling reads `AGENTS.md`, use that name with the identical
three-layer split. Never merge all three into one file: the root
would bloat past usefulness and agents would stop reading it.
## 1. Personal base rules (cross-project, rarely changes)
One short file per operator, ported across projects. Contents:
- Control over cleverness: hard caps on everything unbounded,
floors automated processes may never cross, static ceilings over
autotuning, fewer dependencies.
- Safety: destructive ops dry-run by default and explicitly
confirmed, loud skips over silent corruption, verify-before-write,
deletes that reclaim (compaction in the same change).
- Proof: measured claims only, real-path testing with failure
injection, suite after behavior change, live verification after
deploy.
- Craft: exact minimal diffs, boring technology, research before
load-bearing decisions, docs describing today with gaps marked.
- Communication: direct and factual, disagree with data, fixed
style bans (DevPlace: no em dashes anywhere).
- Operations: commit, deploy, verify in the same session;
retention and checks run inside the system on loops.
This layer never names project paths or stack details. When it
conflicts with a project file, the project file wins.
## 2. Root project file (global rules, always loaded)
Ordered contents, each earning its place by applying to every area:
1. Project identity in one paragraph: what it is, stack in one
line per layer, ports, key constraints (username rules,
password rules, data layout). An agent must orient in 30 seconds.
2. Commands table: install, dev, prod, every test target, load
test, lint, clean. Exact commands, no prose alternatives. Which
interpreter each target uses.
3. Single-test invocation and the failure-triage order
(cheapest-first), so agents debug efficiently instead of
re-running the slowest suite.
4. CLI reference for management commands (one line each), so agents
use named commands instead of inventing shell one-liners.
5. Runtime data layout: the single data root, the path registry,
and the rule that no module computes a path from scratch.
6. Environment variables table: name, default, purpose. Complete;
an undocumented env var is a trap.
7. Subsystem map: table of every nested doc with one-line scope, so
agents know which file to load (and that it exists).
8. Architecture: request pipeline, routing layout, templates and
frontend rules, content pipeline, auth, database, background
work. One paragraph each with pointers down, not full detail.
9. Hard rules, each marked as such: soft delete, reportability,
production-DB protection, confirmation gates, changelog/doc
duties. A hard rule states the rule, the mechanism enforcing it,
and what fails if violated (usually a named test).
10. Conventions: naming, validation style, response style,
ownership checks, date formats, slug/ID handling, role storage.
Each convention shows the one correct shape.
11. Testing: tier definitions, directory mirroring rule, fixture
stack, required patterns, full-suite gate. Pitfalls live in the
nested test doc; the root states the gate.
12. Feature workflow: the ordered step list (understand, data,
server, view, safety, agent+docs, document, validate) with the
blocking rule: a failed step blocks the rest.
13. CI/CD: what runs on push/PR, merge gate, DTAP stages, deploy
then verify.
Size guard: if the root passes ~500 lines, detail has leaked in
that belongs in a nested file. Split it out.
## 3. Nested subsystem files (deep detail, loaded on touch)
One per directory that owns a mechanic (DevPlace: routers,
services, database, static/js, static/css, templates, tests).
Contents:
- Scope line: which paths it covers, in the first line.
- The mechanic: helpers, entrypoints, and patterns with exact
symbol names (`conditional_update_farm`, `respond()`,
`get_users_by_uids`), never paraphrased APIs.
- The closed sets: registries, token scales, breakpoint lists,
event catalogues. An agent must be able to check membership.
- CORRECT/WRONG pairs for the two or three mistakes every newcomer
makes (DevPlace tests doc: scoped vs unscoped selectors, waits
vs sleeps). Show code, not advice.
- Common pitfalls section: the real bugs this area produced, each
with cause and fix. Pitfalls are bug reports that pay rent.
- Worked examples: one end-to-end change with the file count it
touched, so agents calibrate "one feature = N files".
- Backwards-compatibility notes where the area promises them
(what may never break, what the migration path is).
A nested file never repeats the root. It assumes the root was read
and only adds what the root deliberately omitted.
## 4. Severity levels: hard rule, convention, guidance
Agents follow what they can rank. Mark every statement:
- Hard rule: enforced by code, hook, or failing test. Violation
breaks the suite or is denied by tooling. Example: every removal
is a soft delete; production DB access needs written confirmation
plus a token; new user-visible tables must be classified
reportable or private with a reason.
- Convention: one correct shape, enforced by review. Example:
response helpers, date formats, commit message shape, no second
implementation of an existing helper.
- Guidance: preferred approach with stated tradeoffs. Example:
boring technology, static ceilings over autotuning, research
before load-bearing decisions.
Write the enforcement next to the rule ("fails in
`tests/unit/database/moderation.py`", "denied by the pre-tool hook").
A rule without a named enforcer is a wish.
## 5. Keeping the docs updated (the docs are code)
- Same-change rule: the doc update ships in the commit that changes
the behavior. Docs describing yesterday are worse than missing
docs because agents trust them.
- Landing rule: new global rule goes to the root, new mechanic or
pitfall to the nested file, new user-visible behavior to README
plus the user docs site. The feature workflow names this step so
it is never skipped.
- Measured, not remembered: counts, diagrams, and claims derive
from the running app and schema (DevPlace architecture atlas is
generated from the imported app object and live schema, and
records where docs and code disagree).
- No fiction: docs state what the system does today, gaps listed
next to the code that will close them. No roadmap prose, no
aspirational architecture.
- Versioned per `quality_versioning.md` section 4: header comment,
history table, bump in the same commit. All three layers count as
markdown and follow the scheme.
- Deletion duty: removing a feature removes its doc lines in the
same commit. Stale sections are found by searching for the
removed symbol, not by rereading.
## 6. Anti-patterns: what does NOT belong
- Tutorial prose or onboarding narrative. Agent docs are reference,
not a course. One example per pattern, then the rule.
- Duplicated detail in two layers. If root and nested both explain
it, one will rot. Root states the rule and points down.
- Unbounded lists ("all endpoints", "all settings") pasted by hand.
Point at the registry, the table, or the command that lists them.
- Aspirational rules nobody enforces. Each release, delete one rule
the suite does not check and no review ever cites.
- Secrets, credentials, customer data, or production hostnames.
Env var names yes, values never.
- Style debates or option-weighing. Decide with an opinion, state
the tradeoff in one line, move on.
## 7. Replication checklist for your hosting site
1. Personal base file: safety, proof, craft, ops. Shared across
your projects, project file wins on conflict.
2. Root `AGENTS.md`: identity, commands, triage order, CLI, data
layout, env table, subsystem map, architecture pointers, hard
rules with enforcers, conventions, testing gate, workflow, CI/CD.
Under ~500 lines.
3. Nested files per area: `billing/`, `provisioning/`,
`support/`, `status/`, plus `tests/` pitfalls and
`templates/` partials. Scope line first, symbols exact,
CORRECT/WRONG pairs, pitfalls from real bugs.
4. Severity marks on every statement plus the named enforcement
(test path, hook name, review gate).
5. Same-change doc updates, landing rule, measured claims, no
fiction, version headers on all three layers, deletion duty.
6. Quarterly audit: run the suite, grep each hard rule for its
enforcer, delete one unenforced rule, regenerate the
architecture counts.
## Doc history
| Version | Date | Change |
|---------|------|--------|
| 1.0.0 | 2026-10-11 | Created from repo research: three-layer CLAUDE.md system, root/nested/personal contents, severity levels, update duties, anti-patterns. |
quality_software.md
-516
View File
@@ -1,516 +0,0 @@
<!-- doc: quality_software.md | version: 1.2.1 | updated: 2026-10-11 -->
# Quality Software Playbook: How to Build a Site as Good as DevPlace
This is a generic, reusable method distilled from how DevPlace was built.
It is written so you can apply it to a completely different site, for
example a hosting company (accounts, products, orders, provisioning,
support tickets, status pages), and get the same stability and quality.
It is not about copying features. It is about copying structure,
constraints, and verification habits.
## 1. Core philosophy
1. Boring technology, smart edges. One backend language, one database
file, server-rendered HTML, plain JavaScript. Put complexity where
you can see it (request handlers, templates), keep storage dumb.
2. Fewer moving parts. Every dependency is a way to fail. Prefer the
standard library, one process type, one data root.
3. Server-rendered first. The page works without JavaScript. JavaScript
only upgrades live behavior (polling, websockets, optimistic UI).
4. One data source, many faces. Every domain object is served as HTML
for browsers, JSON for API clients, an agent/tool action for
automation, and a documented endpoint. Design the data once, then
expose all four.
5. Exact, minimal diffs. Edit existing files, reuse existing helpers,
match existing visual structure. Zero tolerance for a second way
of doing something that already has one way.
6. Measure, do not theorize. Performance claims come from runs on real
machines. If there is no measurement, say so.
7. Loud skips beat silent corruption. Reject unverifiable input with a
clear reason and a counter. Never store what you cannot vouch for.
Hosting example: a VPS order is one row in one table. The same handler
renders the order page (HTML), returns the order (JSON), exposes a
`order_create` automation action, and is documented in your API docs.
## 2. Reference stack (proven shape)
You can swap brands, but keep the shape:
- Backend: one typed web framework (DevPlace: Python + FastAPI).
- Templates: server-side templates with one shared instance
(DevPlace: Jinja2, single `templates` object imported everywhere).
- Frontend: no framework, no build step. Plain ES modules, one class
per file, small shared utilities (`Http`, `Poller`, modal, toast).
- Database: one embedded relational file with WAL mode for small to
medium scale (DevPlace: SQLite via `dataset`, WAL, busy timeout,
mmap). Synchronous local calls are fine when they are
sub-millisecond; do not make the data layer async for fashion.
- Auth: session cookie for browsers plus API key and Basic for
machines, resolved in one `get_current_user` choke point. No JWT
unless you need third-party verification.
- Static assets: content-addressed or boot-versioned URLs served
immutable (`/static/v<ts>/...`), one helper builds every URL.
- Background work: one in-process queue with inline fallback so tests
stay deterministic and production stays fast.
Hosting example: keep customers, services, invoices, tickets in one
SQLite/Postgres schema behind one data module. Serve the control
panel as HTML, the reseller API as JSON from the same handlers.
## 3. Repository layout that scales
Mirror URLs with code paths, one segment per directory:
- `routers/` mirrors the URL tree. Flat file for one resource
(`feed.py`), package directory for sub-resources (`projects/`).
Each leaf owns an `APIRouter`, the package `__init__.py` aggregates.
- `templates/` mirrors pages, extends one `base.html`. Page CSS in
`extra_head`, page JS in `extra_js`. Shared partials for repeated
UI (avatar link, user link, pagination, report button).
- `static/js/` mirrors behavior, one class per file, instantiated as
`app`. Shared utilities live once and are reused, never forked.
- `static/css/` uses design tokens, no literal colors in page files,
one closed breakpoint set, one z-index scale.
- `services/` holds background services behind one base class and one
manager (start/stop/status/logs). Each service owns its tables and
its admin controls.
- `tests/` mirrors the same tree by path: `tests/api/...` for HTTP,
`tests/e2e/...` for browser, `tests/unit/...` for pure logic.
Rule: if you cannot guess the file from the URL, the layout is wrong.
## 4. Single data root and single config module
All runtime artifacts live under one root (`DATA_DIR`), every path
derived from it in one config module, created at boot:
- Database file, uploads, attachments, generated archives, keys,
locks, backups, service state.
- Stored DB values are relative (`directory` + `stored_name`), so a
layout move never needs a DB rewrite.
- One `DATA_PATHS` registry lists every path; one `ensure_data_dirs()`
creates them before anything writes.
Hosting example: `data/` holds `app.db`, `uploads/`, `invoices/`,
`backups/`, `locks/`, `keys/`. Point `DATA_DIR` at a volume in prod.
## 5. Request pipeline (same for every route)
1. Single app factory: mount `/static`, register routers, install
middleware in fixed order (timing outermost, then security
headers, rate limit, maintenance gate, error handlers).
2. One auth resolution: cookie, API key header, Bearer, Basic,
all in `get_current_user`. Guards: public read
(`get_current_user`), member write (`require_user`), admin
(`require_admin`).
3. One response helper (`respond()`): same context renders HTML for
browsers and JSON for `Accept: application/json` via a response
schema. A context key missing from the schema is dropped from
JSON by design, so schemas are explicit contracts.
4. One validation style: Pydantic form models bound from forms,
one global validation-error handler (re-render or redirect).
5. One timing mechanism: outermost middleware stamps start time and
sets `X-Response-Time`. Never re-time per route.
6. Rate limit only mutations (`POST/PUT/PATCH/DELETE`), keyed by
real IP, tunable from settings. Reads stay fast and exempt.
## 6. Data conventions that prevent entire bug classes
- Primary keys are opaque string IDs (`uid`), slugs accept slug or
bare ID via one resolver. Slugs embed the random tail of the ID,
never a timestamp prefix (prefix sharding funnels writes).
- Soft delete everywhere user content is removable: `deleted_at` +
`deleted_by`, live rows are `deleted_at IS NULL`. Every insert
writes both columns, every list filters them. Only garbage
collection hard-deletes. One admin Trash restores/purges by event.
- Ownership rule: edits are owner-only, deletes are owner-or-admin,
checked by one helper. Admin seniority: junior admins cannot act
on senior admins.
- No N+1: batch helpers (`get_users_by_uids`, counts by ID lists,
paginators) are the only way to load related rows in lists.
- Idempotent schema init: ensure full column set before indexes,
because auto-schema tools create partial tables from first insert.
- Toggles revive, never duplicate: look up ignoring deleted flag,
stamp on off, clear on on, under one shared stamp for cascades.
Hosting example: canceling a server is a soft delete with one stamp
covering service, DNS rows, and scheduled jobs, restorable from Trash
until GC purges it.
## 7. Frontend rules that keep UI consistent
- One base template owns nav, head, modals, toasts, timing badge.
Nav active state is one global, never inline path checks.
- Never hardcode `/static/...`. One helper builds versioned URLs in
templates and one in JS.
- Server renders content that exists at request time (SEO, no-JS,
speed). Client pipeline is only for live content (chat bubbles,
streamed results), always sanitized fail-closed.
- Dates stored as UTC ISO, rendered as `<time data-dt>` and
reformatted to viewer timezone client-side, one format for users
(DevPlace: DD/MM/YYYY).
- Modals, lightbox, toasts, floating windows, scroll memory: one
implementation each, reused by class name. No inline JS.
## 8. Background jobs and services (one pattern)
- One generic fire-and-forget queue: `submit(fn, *args)`. When the
consumer runs, it defers; when it does not (tests, full queue),
it runs inline so behavior stays deterministic.
- One base service class: `run` loop, supervised start/stop, status,
logs, settings. One manager registers all services.
- One async-job pattern for slow work (archives, audits, reports):
enqueue (`POST .../run`), poll (`GET .../{uid}`), read report
(`GET .../{uid}/report`), expire and prune by CLI.
- Multi-worker safety: schema work behind one init lock, background
services elected by one service lock so exactly one worker owns
them. Never assume single-process in job code.
Hosting example: provisioning, backups, SSL renewal, and usage
metering are all job services with run/poll/report plus
`cli <domain> prune` retention commands.
## 9. Testing: three tiers, one tree, no skips
- Tiers by fixture, not by folder debate:
- `unit`: pure in-process, no HTTP, no browser.
- `api`: HTTP integration against a live test server + temp DB.
- `e2e`: Playwright browser, real user flows.
- Test server runs on its own port with its own temp DB and data
dir, background services disabled. One serial process, no
parallelism, so results are stable.
- Every run reports every failure (`-rf`), triage cheapest first:
unit+api (~minutes), then browser tier, then `--last-failed`
loop until empty.
- Required browser habits: explicit `domcontentloaded` on
navigations, visible-state waits over sleeps, scoped selectors,
global settings restored in `try/finally`.
- Gate before suite: clean app import, per-language syntax check,
linter on touched files (catches missing imports that compile
checks miss), no forbidden characters or patterns.
Quality bar: the full suite (all three tiers, every test) passes
before a change is done. A red test covering your change is the
requirement, not a stale artifact. Never narrow a run to make it
pass.
## 10. Extra verification for money, state, and races
Example-based tests are blind to wrong formulas at untried inputs,
long-sequence invariant breaks, and races. For spendable resources
(credits, coins, quotas), bounded state machines (tiers, levels),
or read-then-write mutations reachable from two requests, add four
disposable layers against a temp DB (not committed tests unless
asked):
1. Property checks over the full input domain (monotonicity,
bounds, non-negativity) across hundreds of values.
2. Stateful fuzzing: randomized action sequences across many
entities, invariants re-checked after every action.
3. Real-process concurrency: separate OS processes (not threads),
fresh production-like rows (never pre-zero NULL columns),
exact success counts and exact final totals. Fix races with one
atomic conditional `UPDATE ... WHERE <precondition>` checked by
real `rowcount` at the chokepoint.
4. Static analysis on every touched file (`pyflakes`/`ruff`), which
catches runtime `NameError` paths the other layers may miss.
Hosting example: apply this to credit spend, plan upgrades,
affiliate payouts, and concurrent provisioning of the last IP or
stock-limited node.
## 11. Operations: deploy, verify, prune
- One-command environments: `install`, `dev` (reload, one worker),
`prod` (multi-worker, no template reload, real backlog), `test*`,
`coverage`, load test. Every Python target uses the project venv.
- No bytecode in repo: `PYTHONDONTWRITEBYTECODE=1` for all targets,
`clean` removes venv and stray artifacts.
- CI runs the full suite serially on every push/PR, publishes
coverage and failure screenshots. CI green before merge.
- Production DB is never touched without explicit written
confirmation plus a two-factor token; disposable scripts always
target a scratch DB via env override.
- Retention is a feature: every expirable domain gets
`prune` (expired only) and `clear` (everything) CLI commands,
wired to the same GC the server uses.
- Backups are first-class: targets, worker, schedules, offload,
download restricted to the primary admin, prune missing files.
## 12. Security, moderation, and trust checklist
- Passwords hashed with a slow KDF (PBKDF2-SHA256), session tokens
random 256-bit, per-process TTL cache keyed by token.
- Ownership checked server-side on every mutation; deletes confirm
explicitly and audit; destructive automation actions require a
two-step confirm flag declared in their schema.
- Every user-visible content table is reportable by construction:
registry entry, object URL resolver, report button in the action
bar. Private-only tables are listed as unreportable with a reason,
enforced by a test that fails on unclassified tables.
- Outbound HTTP goes through one stealth/guarded client (consistent
TLS + headers + proxy + SSRF guard), never bare clients per call.
- Audit log is append-only, never raises into the caller, with one
catalogue of event keys. Admin-only read.
- Privacy docs and store declarations update in the same change as
any new personal-data collection or third-party transmission.
Hosting example: tickets, reviews, and public server comments are
reportable; invoices and API tokens are private-by-design with
written reasons; provisioning calls go through the one guarded
client with proxy support.
## 13. Docs, SEO, and API surface
- Every public/auth endpoint gets prose docs plus an interactive
API entry (params, auth, sample response). Public pages get SEO
context and JSON-LD; sitemap and robots are served, not static.
- One docs registry (`DOCS_PAGES`) owns prose pages; API docs own
endpoint entries. New route without docs is incomplete.
- README stays product-facing (routes, config, env vars, features).
Nested `CLAUDE.md` files hold subsystem mechanics next to the code
so the always-loaded root doc stays small.
- Version static assets, canonical URLs per listing, breadcrumbs,
related content, and one default OG image.
## 14. Development flow (the daily loop)
Every feature follows the same eight steps in data-flow order, and a
failed step blocks the rest (never skip ahead):
1. Understand. Read the router, template, matching tests, and the
subsystem doc. Trace input model to router to data helper to
response (HTML and JSON). Find a relatable existing example and
match it structurally and visually; if none exists, stop and agree
on the new pattern before building.
2. Data layer. Query/batch helpers first (never inline N+1), input
model, JSON response schema (every exposed key declared or it is
silently dropped), idempotent schema/index changes.
3. Server layer. Handler with the right guard (public/member/admin),
specific paths before catch-alls, HTML+JSON via one responder.
4. View layer. Extend base, reuse partials and globals, page CSS/JS
in the standard blocks, one ES class per file.
5. Safety layer. Reportable/private classification, ownership checks,
confirm gates, audit events, privacy-doc update if new personal
data or third-party transmission appears.
6. Agent + docs layer. Tool action per user-askable operation,
endpoint doc entry, prose page if user-facing, SEO context and
sitemap entry if indexable. This step is the most-forgotten; do
not skip it.
7. Document. Product README for user-visible change, subsystem doc
for new mechanics or pitfalls, root doc only for new global rules.
8. Validate. Clean import, per-language syntax check, linter on
touched files, full three-tier suite green (see section 9), plus
the money/state/race procedure where it applies.
Failure triage is cheapest-first: fast tiers (unit+api, minutes)
before browser tier, then re-run only failures (`--last-failed`)
until empty. One pass reports every failure; fix the list, not one
bug per pass.
Session rule: commit, deploy, verify in the same session, in that
order. A change that is not running in production is not done. After
deploys, verify live; green locally means nothing until prod
confirms it.
## 15. Workflows and automation
- One command surface. Every repeated action is a named target or CLI
command (`dev`, `prod`, `test`, `test-fast`, `test-failed`,
`coverage`, load test, `prune`, backups). No tribal shell
one-liners; if it is run twice, it gets a name.
- DTAP pipeline. Development (reload, one worker) to Test (CI serial
suite + coverage + failure screenshots, green before merge) to
production (multi-worker, no template reload, real backlog).
- Scheduled hygiene inside the system, not in heads: retention,
cleanup, quota resets, and health checks run on loops or CLI
commands with dry-run defaults and narrow scopes for destructive
ops. Destructive operations are dry-run first, explicitly
confirmed, date-guarded or allowlisted.
- Load testing is a target, not an event: scripted Locust run
against a scratch DB with fixed users, spawn rate, and run time,
producing an HTML report.
- Production guardrails are enforced, not remembered: a hook denies
commands reaching the live DB without explicit written
confirmation plus a second approval; disposable scripts always
target scratch databases via env override.
Hosting example: `provision`, `suspend`, `backup-run`,
`ssl-renew`, `usage-meter`, `invoice-prune` are all named commands
with `--dry-run`; nightly loops handle expiry, renewal, and status
checks with human-readable pressure states.
## 16. Agents: in-product assistant, services, and coding help
DevPlace quality comes partly from treating agents as first-class
consumers of the same routes humans use:
- In-product assistant (DevPlace: Devii). A conversational operator
over the real API: per-owner sessions and channels (web, terminal,
chat bridge), a tool catalog where each tool mirrors one route
with matching auth, confirmation gates on destructive tools (first
call refused, agent shows exact target, second call passes
`confirm=true`), per-user cost attribution and quotas through one
AI gateway, and usage/ledger pages per user. New feature rule: if
a user could ask the assistant to do it, it gets a tool.
- Gateway pattern. Every AI/model call flows through one internal
gateway (usage ledger, provider/model routing, per-call cost
headers), so spend is attributable and every consumer (correction,
modifier, grading, chat) shares quotas and audit.
- Background service agents. Crawlers, importers, reconcilers, and
persona bots run as supervised services under one manager with
start/stop/status/logs and admin pages (DevPlace: news importer,
bot fleet with live monitor, container reconciler, audit writer).
Exactly one worker owns them via a service lock.
- Coding-agent support. Nested subsystem docs next to the code so an
agent loads only what it touches; strict conventions (no second
implementation, named helpers, closed token/breakpoint/z-index
sets) so generated code matches; tool catalog and docs registry
so the agent can act and document through the same contracts.
- Human authority. Agents propose and execute reversible work;
destructive, cross-user, or policy-crossing actions need explicit
human confirmation, seniority checks where roles exist, and an
audit trail. When a floor is hit (retention minimum, protected
path), the agent holds and pages a human instead of deleting.
Hosting example: a support assistant with tools like `server_reboot`
(public read tools open, mutating tools auth-gated, destructive
tools confirm-gated), all spend metered per customer through your
gateway, provisioning and renewal running as supervised services.
## 17. Documenting: what lives where, and when
Documentation is layered by audience and distance from the code, and
every layer updates in the same change as the code it describes:
- Source code: no comments, no docstrings. Code is self-documenting
through names, small helpers, and one obvious way per task. A
decision worth recording goes in the change description or the
subsystem doc, never as deliberation text in the source.
- Subsystem docs (DevPlace: nested `CLAUDE.md` next to the code).
Each directory that owns a mechanic holds its own doc: helpers,
pitfalls, cross-layer wiring, worked examples. An agent or human
loads only the docs for the area being touched, so the always-
loaded cost stays flat as the project grows.
- Root doc (DevPlace: `CLAUDE.md`, `ARCHITECTURE.md`). Global rules,
commands, and architecture only. A per-feature detail never lands
here; a new global rule always does. The architecture atlas is
measured from the running app and schema (counts, diagrams), not
written from memory, and disagreements between docs and code are
recorded explicitly.
- Product README. User-visible surface: routes, config, env vars,
features, quick start. Updated when behavior a user can see
changes.
- User docs site (DevPlace: `/docs` with a `DOCS_PAGES` registry).
Prose pages per audience tier plus an interactive API reference
where every endpoint entry carries params, auth, and a sample
response. A new public or auth endpoint without a doc entry is
incomplete by definition.
- Privacy and store declarations. Any new personal-data category or
third-party transmission updates the privacy doc and platform
declarations in the same change, never as a follow-up.
- Event and reason registries. Audit event keys, report reasons, and
similar closed sets live in one catalogue each, so every client
and test renders the same list.
Rule: docs state what the system does today, with gaps listed next
to the code that will close them. No fiction, no roadmap prose.
Hosting example: `billing/CLAUDE.md` documents proration and retry
rules next to the billing code; the root doc only says billing
exists and which commands run it; `/docs` gains a prose page plus
one API entry per new endpoint in the same change.
## 18. Roles, permissions, and ownership
One small role set, enforced server-side at every surface (web UI,
REST API, and agent tools alike):
- Roles (DevPlace: guest, member, admin, primary admin). Stored as
exact strings, tested through one `is_admin()` helper, never
hand-rolled compares. Any write to a role invalidates the cached
role lookups.
- Guests read public pages; members write their own content;
admins moderate and operate. Public reads resolve the viewer but
never require login; every write names its guard explicitly.
- Ownership first: edits are owner-only, deletes are owner-or-admin,
checked by one helper comparing resource owner to actor. Admins
can remove abuse but cannot rewrite someone else's words.
- Seniority among admins: a junior admin (later registration) cannot
mutate a more senior admin's account or content. Every per-user
admin mutation is gated by one seniority predicate, denials are
audited, and the gate is server-side so it also covers agent
tools. One primary admin (the first admin account) is the single
exception with full visibility and full control, including
private-by-admin content and restricted downloads.
- Visibility predicates are single functions, never inline checks:
e.g. `can_view(project, user)` decides private-content access in
one place (owner sees own, admins see members' private rows, but
an admin's own private rows stay visible only to that admin
except for the primary admin). Container and workspace isolation
get their own stricter predicates on the same pattern.
- Machines authenticate as users: every account holds an API key,
plus issuable scoped tokens; the same guards apply regardless of
credential type, so the REST API cannot do what the UI forbids.
- Agents act as the asking user: tool auth mirrors route guards,
destructive tools add confirmation gates, and every admin tool
passes through the same seniority and audit path as the web UI.
Rule: if a permission check appears twice in different words, one
of them is a bug. Hoist it into the single predicate and call it
from both places.
Hosting example: guest (pricing, status), customer (own servers and
invoices), support (view customers' servers, restart but not delete;
junior support cannot touch senior staff accounts), owner (full
access plus backup downloads and billing refunds). One
`can_view_server` / `can_manage_server` pair gates panel, API, and
assistant tools together.
## 19. Replication checklist for your hosting site
Use this order; it is the data-flow order DevPlace follows:
1. Scaffold: app factory, config with one data root, shared
templates, response helper, auth choke point, middleware stack.
2. Data: users, sessions, settings, then one domain (products).
Add soft-delete columns, batch helpers, idempotent init.
3. Server: public reads first (HTML+JSON via one handler), then
member writes with guards, then admin views.
4. View: base template, tokens, partials, one JS utility set.
Server-render listings and detail pages.
5. Safety: reportable/private classification, ownership checks,
confirm gates, audit events.
6. Automation + docs: tool action per user-askable operation,
API doc entry per endpoint, SEO context per public page.
7. Verification: import gate, lint touched files, full three-tier
suite green, plus the four-layer money/state/race procedure
where it applies.
8. Ops: venv targets, CI serial suite, prune/clear CLIs, backups,
production-DB guard, deploy then verify live.
## 20. What to copy literally vs adapt
Copy literally: single data root, soft-delete columns and filters,
batch loaders, one response helper, one auth resolver, test tiers
and temp-DB server fixture, prune/clear per domain, audit-never-
raises, versioned static URLs.
Adapt to hosting: table list (customers, plans, servers, invoices,
tickets, status incidents), provisioning job services, payment and
registrar integrations behind the one outbound client, usage
metering and quota rules, status page and notifications.
If you keep the constraints and the verification loop, the domain
can change completely and the quality transfers with it.
## Doc history
| Version | Date | Change |
|---------|------|--------|
| 1.0.0 | 2026-10-11 | Created: philosophy, stack, layout, data, pipeline, frontend, jobs, testing, verification, ops, security, docs, checklist. |
| 1.1.0 | 2026-10-11 | Added development flow, workflows/automation, and agents sections. |
| 1.2.0 | 2026-10-11 | Added documenting and roles/permissions sections. |
| 1.2.1 | 2026-10-11 | Added doc version header and history per quality_versioning.md. |
quality_versioning.md
-165
View File
@@ -1,165 +0,0 @@
<!-- doc: quality_versioning.md | version: 1.0.0 | updated: 2026-10-11 -->
# Quality Versioning Playbook: App, Assets, Docs, and Git
Generic companion to `quality_software.md`. It covers how DevPlace
versions its application, its static assets, and its git history,
plus the one practice DevPlace does NOT have today and you should
adopt from day one: versioned markdown docs.
Honest baseline (researched 2026-10-11, do not assume otherwise):
- App version: single source `pyproject.toml`, auto-bumped patch on
every commit by a checked-in pre-commit hook. Real and working.
- Asset version: composed `{APP_VERSION}-{BOOT_ID}`, stamped into
every static URL, served immutable. Real and working.
- Git: trunk `master`, remote `production` branch, per-ticket
branches, CI gate on push/PR to master. Real and working.
- Tags: none. Changelog file: none. Markdown versioning: none.
These are gaps, and sections 4 and 6 below define what to do
instead of copying the gap.
## 1. App versioning: one source, auto-bumped
- One file owns the version (DevPlace: `pyproject.toml`
`project.version`, currently `1.0.x`). The running app reads it
live at boot (DevPlace: `config.APP_VERSION` via `tomllib`), so
the version in code and the version on disk can never drift.
- A checked-in pre-commit hook bumps the patch number on every
commit (DevPlace: `.githooks/pre-commit`, activated by
`git config core.hooksPath .githooks` in the install target).
Skips merge commits and commits that already touch the version
line, so a manual minor/major bump is never double-bumped.
- Semver discipline: patch is automatic (every commit), minor is a
manual commit when a user-visible feature lands, major is a
manual commit on breaking change. Minor/major bumps are ordinary
commits whose message says what the release contains.
- The version is visible at runtime: stamped into a meta tag
(DevPlace: `<meta name="asset-version">` in `base.html`) so any
page view, bug report, or screenshot names the exact build.
Hosting example: the control panel footer and every support ticket
show `v2.14.3`; support never asks "which version are you on".
## 2. Asset versioning: cache forever, deploy instantly
- Compose the asset version from app version plus boot id:
`STATIC_VERSION = {APP_VERSION}-{BOOT_ID}` where the boot id is a
deploy-provided marker with a wall-clock fallback.
- Stamp it into every app-owned static URL
(`/static/v<version>/...`) through one URL helper, and serve that
path immutable with a one-year `max-age`.
- Keep the reverse proxy in sync: the versioned-path location regex
must accept dots and hyphens, not digits only, or the immutable
header silently stops applying (this was a real production bug
class here, fixed once the version segment stopped being a bare
timestamp).
- Never hardcode a static path; every template and script builds
URLs through the one helper, so a version-shape change touches
one function.
## 3. Git workflow: trunk, tickets, production
- Trunk is `master`. All work lands there through small commits;
CI (serial full suite, coverage, failure screenshots) runs on
every push and PR to master and must be green before merge.
- One branch per ticket or task, named after the tracker
(`ticket-123`, `112-short-slug`). Short-lived: branch, commit,
merge, delete. No long-running feature branches.
- One `production` branch (or tag set, see section 6) marks what is
deployed. Development happens on master; promotion to production
is a deliberate merge, followed by deploy and live verification.
- Commit messages: imperative summary line under 72 chars, blank
line, body explaining why (not what, the diff shows what).
Optional trailers for attribution (`Co-Authored-By`) and session
links. Reverts say what behavior returned and why.
- Never rewrite published history: no amend/push-force, no rebase
of shared branches, no `reset --hard` on anything pushed. Fix the
working tree forward; a mistaken commit is corrected by a new
commit.
- Hooks live in the repo (`.githooks/`) and are wired by the
install target, so every clone gets the version bump and any
future checks with zero manual setup.
## 4. Markdown versioning: every doc carries its version
This is the rule DevPlace does not follow yet: **every markdown
file is versioned on every update.** Adopt it from the start; it
costs one line per edit and buys traceability for every claim a
doc makes.
Scheme (copy literally):
1. First line of the file is an invisible HTML comment:
`<!-- doc: <filename> | version: X.Y.Z | updated: YYYY-MM-DD -->`
It renders nothing and survives every markdown renderer.
2. Bottom of the file is a `Doc history` table with one row per
version: version, date, one-line change note.
3. Doc-semver: MAJOR for restructure or section renumbering, MINOR
for a new section or new rule, PATCH for wording, fixes, or
example updates. Every update bumps exactly one level and adds
exactly one history row.
4. The version bump and the content change ship in the same commit,
so `git log -p <doc>` and the history table always agree.
5. Cross-doc references name the section, not the version ("see
`quality_software.md` section 9"), so minor doc bumps never
cascade into other files.
Enforcement options, cheapest first: a pre-commit check that fails
when a staged `.md` lacks a matching version-line change (same
pattern as the app-version hook, inverted: docs must be touched by
hand, so the hook only verifies); a CI grep that every tracked
`.md` starts with the `doc:` comment; a unit test listing tracked
markdown files and asserting the header parses.
Hosting example: `docs/sla.md` at `3.1.0` and `docs/pricing.md` at
`5.0.2` evolve independently; a customer dispute cites "SLA v3.1",
not "the docs, at some point".
## 5. What gets a version vs what gets a timestamp
- Versioned (semver, comparable, citable): the app, each markdown
doc, API contracts, terms/SLA/pricing pages, database export
formats, agent tool catalogs.
- Timestamped (point-in-time, never compared): deploys (boot id),
backups, audit events, usage ledger rows, generated reports.
- Never versioned alone: a bare counter with no history row, a date
with no change note. A version you cannot diff is decoration.
## 6. Releases and tags: close the gap deliberately
DevPlace has no tags and no changelog file; promotion is tracked by
the `production` branch. For a hosting company, where customers and
auditors ask "what changed and when", close this gap:
- Tag every production promotion with an annotated tag
(`v2.14.0`, message = release notes summary). Tags are cheap,
immutable pointers; the branch can move, the tag cannot.
- Keep one `CHANGELOG.md` (itself doc-versioned per section 4)
with one entry per minor/major release: date, tag, added,
changed, fixed, security notes. Patch releases batch into the
next minor entry; no per-commit changelog spam.
- Release flow in order: land feature commits on master (CI green),
manual minor/major bump commit, merge master into production,
tag, deploy, verify live, write the changelog entry. A release
that is not verified live is not released.
## 7. Replication checklist for your hosting site
1. Single version source read live by the app; pre-commit hook
auto-bumps patch; minor/major are manual release commits.
2. Asset URLs carry `{version}-{boot-id}`, served immutable;
proxy regex accepts the full version alphabet.
3. Trunk `master`, ticket branches, `production` promotion target,
CI gate on master, no history rewrites, hooks in-repo.
4. Every `.md` starts with the `doc:` version comment and ends with
a history table; every update bumps; CI verifies the header.
5. Terms, SLA, pricing, and API contracts are versioned docs with
citable versions, not wiki pages.
6. Annotated tags plus one changelog for customer-facing releases;
release = merge, tag, deploy, verify live, changelog entry.
## Doc history
| Version | Date | Change |
|---------|------|--------|
| 1.0.0 | 2026-10-11 | Created from repo research: app auto-bump hook, asset composition, trunk/ticket/production git model, new markdown-versioning scheme. |
research_residental_proxies.md
-74
View File
@@ -1,74 +0,0 @@
**Residential proxy donation works by having donors run a lightweight client that opens a persistent outbound tunnel (almost always WebSocket/WSS) to your central server. Your server then multiplexes incoming SOCKS5 or HTTP CONNECT requests over those tunnels so traffic exits through the donor’s residential IP.**
This is a reverse-proxy style architecture. The client never listens on a public port; it only dials out. That is why WebSocket (or a similar long-lived bidirectional channel) is the natural transport.
### Core client design (common to all options)
1. Client authenticates (token / account) and opens a WebSocket to your server.
2. Server treats each live WebSocket as a residential exit node.
3. When a customer requests a connection through that node, the server sends a framed “open TCP to host:port” message over the WebSocket.
4. The client opens the TCP socket locally and bidirectionally relays bytes.
5. Keep-alives, automatic reconnect with exponential backoff, and optional bandwidth/rate limits on the client side.
Open-source references that implement exactly this pattern include `residential-proxy-aggregator` (Python WS server + client agents), `pywssocks`, and various WS-SOCKS tools.
### Option comparison
| Option | Reliability / Always-on | Install friction | Resource use | Store / distribution | Best for | Major limitations |
|--------|-------------------------|------------------|--------------|----------------------|----------|-------------------|
| **Browser extension (MV3)** | Medium | Very low | Medium (browser process) | Chrome Web Store / Firefox AMO | Mass casual donors | Service worker idle timeouts; store policies; only while browser runs |
| **systemd user service** (or Windows Service / LaunchAgent) | High | Medium | Very low | Binary + unit file / package | Technical / always-on Linux users | Platform-specific packaging |
| **Desktop app** (Tauri / Electron / native) | High | Medium-High | Low–Medium | Website / GitHub releases | Cross-platform reliability | Larger download, code-signing |
| **Plain webpage / PWA with open tab** | Low | Lowest | Low while open | Just a URL | Quick experiments | Tab throttling, user closes tab, background limits |
| **Service Worker alone** | Poor | Low | Low | Part of a site | Not recommended as primary | Cannot reliably hold long-lived outbound tunnels |
### 1. Browser extension (Manifest V3)
Technically feasible and the lowest-friction option for many donors.
- Background service worker opens the WebSocket.
- Since Chrome 116, WebSocket activity resets the service-worker idle timer, but you still need an explicit keepalive (send a ping every ~20 seconds).
- Use `chrome.alarms` (minimum ~30 s) as a durable backup that survives worker termination and forces reconnects.
- You will also need careful reconnect logic, state persistence (e.g. `chrome.storage.session`), and handling of the case where the worker is killed mid-relay.
**Store and policy issues**
Chrome Web Store (and Firefox AMO) require a clear single purpose and honest disclosure. An extension whose primary function is “share my residential connection / bandwidth” must state that prominently. Proxy-related extensions receive extra scrutiny. Misleading claims or hidden data collection will get you removed. Historical precedent exists (uProxy was a browser-extension peer-proxy system, later discontinued and succeeded by Snowflake).
**Practical limits**
- Only runs while the browser is open.
- Higher CPU/battery impact than a native binary.
- On mobile Chrome the background behaviour is even more restricted.
### 2. systemd user service (Linux) / equivalent on other OSes
This is one of the cleanest long-running options.
- Ship a small static binary (Go, Rust, or even a Node/Python script).
- Provide a user unit file (`~/.config/systemd/user/residential-proxy.service`).
- `systemctl --user enable --now residential-proxy.service`.
- It reconnects automatically, uses almost no resources when idle, and survives reboots (with lingering enabled).
Windows equivalent: a Windows Service or scheduled task. macOS: LaunchAgent. This matches the architecture used by the open-source residential-proxy-aggregator project (C++ Windows service + watchdog).
### 3. Desktop application (Tauri preferred over Electron)
Tauri gives you a small native binary with a system tray, automatic updates, and full control over the WebSocket and TCP relay. Electron works but is heavier. This is the most reliable cross-platform choice if you want something that “just stays on” without requiring users to keep a browser open.
### 4. Service Worker / PWA
A regular service worker (or even an extension service worker without the keepalive tricks) is **not** a good primary client.
Service workers are event-driven and get terminated after short idle periods. They cannot reliably maintain a persistent outbound TCP/WebSocket tunnel that is ready to accept new SOCKS connections at any moment. They are useful for secondary tasks (notifications, caching) but not as the proxy agent itself. Background throttling on inactive tabs and especially on mobile makes a pure PWA approach unreliable for always-available residential exits.
### Other practical considerations
- **Keepalive & reconnect**: Mandatory everywhere. WebSocket idle timeouts on reverse proxies/load balancers are often 60–100 s; your application-level heartbeat should be more aggressive.
- **Bandwidth and rate limiting**: Enforce soft and hard limits on the client so a donor is not surprised by a sudden bill or congestion.
- **Consent & transparency**: Show clear UI that the user is sharing their IP and bandwidth, with an easy off switch. Log (or at least allow users to see) roughly how much traffic has been relayed.
- **ISP terms**: Many residential ISPs prohibit running open proxies, commercial use of the connection, or significant bandwidth resale. Donors should be warned.
- **Abuse & liability**: Residential proxy pools are frequently abused for scraping, credential stuffing, ad fraud, etc. You will need authentication of donors, monitoring, the ability to ban nodes quickly, and clear Terms of Service that shift responsibility appropriately. Pure “donation” models still create platform liability if the traffic is used illegally.
- **Authentication**: Prefer short-lived tokens or account-based auth so you can revoke a donor’s participation instantly.
- **Mobile**: Background execution limits make reliable always-on clients difficult on iOS and restricted on Android. Not recommended as a primary platform.
### Recommended starting path
1. **Prototype** with a simple Node/Go/Rust WebSocket client + a matching server that exposes a SOCKS5 listener (look at `pywssocks` or the residential-proxy-aggregator design for structure).
2. **First distribution channel**: browser extension (for reach) + a tiny systemd/LaunchAgent/Windows service binary (for reliability).
3. Later add a polished Tauri tray app if the project gains traction.
The WebSocket-upstream model is the right foundation. The choice of packaging mainly affects how many people will actually leave the client running and how painful the operational reality becomes. Browser extensions win on distribution; native user services and desktop apps win on reliability and resource behaviour.#
xx.py
-138
View File
@@ -1,138 +0,0 @@
#!/usr/bin/env python3
# retoor <retoor@molodetz.nl>
import argparse
import sys
import time
from pathlib import Path
MIN_AGE_SECONDS = 3600
PARTIAL_SUFFIX = ".partial"
def human_bytes(size: int) -> str:
value = float(max(0, int(size or 0)))
for unit in ("B", "KB", "MB", "GB", "TB"):
if value < 1024 or unit == "TB":
return f"{value:.0f} {unit}" if unit == "B" else f"{value:.1f} {unit}"
value /= 1024
return f"{value:.1f} TB"
def backups_root() -> Path:
return Path(__file__).resolve().parent / "data" / "backups"
def find_partials(
root: Path, now: float, min_age: int
) -> tuple[list[tuple[Path, int, float]], list[tuple[Path, int, float]]]:
abandoned: list[tuple[Path, int, float]] = []
in_progress: list[tuple[Path, int, float]] = []
if not root.is_dir():
return abandoned, in_progress
for path in root.rglob(f"*{PARTIAL_SUFFIX}"):
if not path.is_file() or path.suffix != PARTIAL_SUFFIX:
continue
try:
info = path.stat()
except OSError as exc:
print(f"skip {path}: {exc}", file=sys.stderr)
continue
age = now - info.st_mtime
row = (path, info.st_size, age)
if age < min_age:
in_progress.append(row)
else:
abandoned.append(row)
abandoned.sort(key=lambda item: item[1], reverse=True)
in_progress.sort(key=lambda item: item[1], reverse=True)
return abandoned, in_progress
def prune_empty_dirs(root: Path) -> int:
removed = 0
if not root.is_dir():
return 0
for directory in sorted(
(path for path in root.rglob("*") if path.is_dir()),
key=lambda path: len(path.parts),
reverse=True,
):
try:
next(directory.iterdir())
except StopIteration:
try:
directory.rmdir()
removed += 1
except OSError:
pass
except OSError:
pass
return removed
def report(label: str, rows: list[tuple[Path, int, float]]) -> int:
total = 0
if not rows:
return total
print(label)
for path, size, age in rows:
total += size
print(f" {human_bytes(size):>10} {age / 3600:.1f}h {path}")
print(f" {human_bytes(total):>10} total")
return total
def main() -> int:
parser = argparse.ArgumentParser(
description=(
"Remove abandoned backup *.partial files under data/backups. "
"Does not open SQLite and does not touch completed .tar.gz archives."
)
)
parser.add_argument(
"--apply",
action="store_true",
help="Delete the abandoned files. Default is a dry run.",
)
parser.add_argument(
"--min-age-seconds",
type=int,
default=MIN_AGE_SECONDS,
help="Skip files newer than this many seconds (in-flight backup guard).",
)
args = parser.parse_args()
if args.min_age_seconds < 0:
print("--min-age-seconds must be >= 0", file=sys.stderr)
return 2
root = backups_root()
abandoned, in_progress = find_partials(root, time.time(), args.min_age_seconds)
if in_progress:
report("In-flight (kept)", in_progress)
if not abandoned:
print(f"No abandoned {PARTIAL_SUFFIX} files under {root}")
return 0
mode = "DELETE" if args.apply else "DRY RUN"
report(f"{mode}: {len(abandoned)} abandoned file(s) under {root}", abandoned)
if not args.apply:
print("Re-run with --apply to delete.")
return 0
freed = 0
failed = 0
for path, size, _age in abandoned:
try:
path.unlink()
freed += size
except OSError as exc:
print(f"failed {path}: {exc}", file=sys.stderr)
failed += 1
empty = prune_empty_dirs(root)
print(
f"Removed {human_bytes(freed)} in {len(abandoned) - failed} file(s), "
f"{failed} failed, {empty} empty dir(s)"
)
return 1 if failed else 0
if __name__ == "__main__":
raise SystemExit(main())