diff --git a/.gitignore b/.gitignore index 047695ac..b7947f70 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ +.docs .dpc .devplace dpc.log diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md deleted file mode 100644 index 69a7019b..00000000 --- a/ARCHITECTURE.md +++ /dev/null @@ -1,636 +0,0 @@ -# DevPlace architecture atlas - -Author: retoor - -Every diagram below was derived from the source tree, the live SQLite schema and the imported FastAPI app object, not from the documentation. Where the repository documentation and the running code disagree, the disagreement is recorded in the last section. - -| Measure | Value | -|---|---| -| python files | 644 | -| routers | 38 | -| http paths | 348 | -| websockets | 12 | -| services | 27 | -| db tables | 102 | -| agent tools | 318 | -| test functions | 3073 | - -## 01 Deployment topology - -Source: `docker-compose.yml / Dockerfile / nginx/nginx.conf.template` - -Two containers on one bridge network, on a single host. nginx is the only published listener and binds to loopback only. The app container bind-mounts the repository root, so the production process reads the same SQLite file and the same `data/` tree as `make dev`. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - client["Browser / API client / Telegram / devRant client"] - subgraph host["Single host"] - subgraph appnet["docker network: appnet"] - nginx["nginx container
127.0.0.1:PORT to :80
8 websocket upgrade locations"] - app["app container
uvicorn devplacepy.main:app
--workers 2 --backlog 8192"] - end - repo["repository root
bind mount .:/app"] - static["devplacepy/static
read only mount"] - data["DEVPLACE_DATA_DIR = data/
23 registered paths"] - sqlite[("data/devplace.db
SQLite, WAL, 256MB mmap")] - lock["data/locks/devplace-services.lock
flock, elects one service owner"] - docker["host Docker daemon
ppy:latest instances"] - end - client --> nginx - nginx -->|"proxy_pass, X-Real-IP"| app - nginx -->|"/static/, /static/uploads/"| static - app --> repo - app --> data - data --> sqlite - app --> lock - app -->|"docker CLI backend"| docker -``` - -nginx depends_on app with condition service_healthy. App healthcheck: interval 30s, timeout 10s, retries 3, start_period 120s, start_interval 2s. - -## 02 Worker boot and service election - -Source: `devplacepy/main.py lifespan` - -Every uvicorn worker runs the full lifespan. Schema work is serialized behind an exclusive lock so workers pay it one after another; background services are elected, so exactly one worker in the host owns them. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - s1["ensure_data_dirs
creates all 23 DATA_PATHS"] - s2["init_lock
exclusive flock on INIT_LOCK_FILE"] - s3["init_db
idempotent columns and indexes"] - s4["ensure_certificates
VAPID keys in data/keys"] - s5["service_manager.register x27"] - gate{"DEVPLACE_DISABLE_SERVICES set?"} - skip["services registered but never started
test mode"] - bg["background.start
per worker asyncio queue"] - elect{"acquire_service_lock"} - owner["set_lock_owner True
supervise all 27 services"] - decline["declined, another worker owns them"] - vis["start_visit_flusher"] - serve(["serving on port 10500"]) - s1 --> s2 --> s3 --> s4 --> s5 --> gate - gate -->|yes| skip --> vis - gate -->|no| bg --> elect - elect -->|"lock acquired"| owner --> vis - elect -->|"lock held elsewhere"| decline --> vis - vis --> serve -``` - -Shutdown reverses it: flush_visits, service_manager.shutdown_all, background.stop. - -## 03 Request pipeline - -Source: `devplacepy/main.py, verified against app.user_middleware` - -Ten middlewares wrap every request. The order below is the real execution order read off the built middleware stack, outermost first. Two of them are plain ASGI classes; the other eight are `BaseHTTPMiddleware` dispatch functions. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - req(["incoming ASGI scope"]) - m0["0 GZipMiddleware
minimum_size 512, level 5"] - m1["1 TunnelDispatchMiddleware
host matches a workspace tunnel?"] - tun["routers/tunnel.py
handle_http / handle_ws"] - m2["2 response_timing
request.state.request_start, X-Response-Time"] - m3["3 visit_statistics"] - m4["4 track_presence
throttled last_seen write"] - m5["5 maintenance_middleware
503 unless static, avatar, auth, admin"] - m6["6 rate_limit_middleware
mutating methods only, per IP bucket"] - m7["7 add_security_headers"] - m8["8 await_pending_corrections
drains sync AI correction futures"] - m9["9 refresh_db_snapshot"] - mounts["mounts: /static/uploads, /static/v{ts}, /static"] - routes["38 included routers"] - handlers["exception handlers
404, 500, RequestValidationError"] - req --> m0 --> m1 - m1 -->|"tunnel host"| tun - m1 -->|"normal host"| m2 --> m3 --> m4 --> m5 --> m6 --> m7 --> m8 --> m9 - m9 --> mounts - m9 --> routes - routes --> handlers -``` - -Rate limiting reads rate_limit_per_minute and rate_limit_window_seconds from site_settings and exempts reads and the /openai gateway. Bucket key is X-Real-IP falling back to request.client.host. - -## 04 URL surface - -Source: `devplacepy/main.py include_router calls` - -38 routers, each mounted under one prefix, grouped here by the concern they serve. Directory shape mirrors the URL: a domain with one resource is a flat module, a domain with several sub-resources is a package that aggregates its leaves in `__init__.py`. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart LR - app(["FastAPI app"]) - subgraph identity["Identity and profile"] - r1["/auth - auth/"] - r2["/profile - profile/"] - r3["/avatar - avatar.py"] - r4["/follow - follow.py"] - r5["(none) - relations.py block, mute"] - end - subgraph social["Content and engagement"] - r6["/feed - feed.py"] - r7["/posts - posts.py"] - r8["/comments - comments.py"] - r9["/gists - gists.py"] - r10["/news - news.py"] - r11["/votes - votes.py"] - r12["/reactions - reactions.py"] - r13["/bookmarks - bookmarks.py"] - r14["/polls - polls.py"] - r15["/awards - awards.py"] - r16["/leaderboard - leaderboard.py"] - r17["/notifications - notifications.py"] - r18["/messages - messages.py"] - r19["/uploads - uploads.py"] - r20["/media - media.py"] - end - subgraph work["Projects and compute"] - r21["/projects - projects/ incl files, containers"] - r22["/p - proxy.py container ingress"] - r23["/zips - zips.py"] - r24["/forks - forks.py"] - r25["/issues - issues/"] - end - subgraph aiml["AI and tools"] - r26["/openai - openai_gateway.py"] - r27["/devii - devii.py"] - r28["/tools - tools/ seo, deepsearch, isslop"] - end - subgraph play["Play"] - r29["/game - game/"] - r30["/quizzes - quizzes/"] - end - subgraph platform["Platform"] - r31["/admin - admin/ 23 modules"] - r32["/docs - docs/"] - r33["(none) - push.py, seo.py"] - end - subgraph machine["Machine interfaces"] - r34["/api - devrant/"] - r35["/dbapi - dbapi/ read only"] - r36["/xmlrpc - xmlrpc.py"] - r37["/pubsub - pubsub.py"] - end - app --> identity - app --> social - app --> work - app --> aiml - app --> play - app --> platform - app --> machine -``` - -| Prefix | Paths | Prefix | Paths | Prefix | Paths | -|---|---|---|---|---|---| -| /admin | 92 | /profile | 17 | /gists | 5 | -| /projects | 46 | /issues | 13 | /messages | 5 | -| /tools | 27 | /dbapi | 9 | /notifications | 5 | -| /game | 23 | /auth | 6 | /uploads | 5 | -| /quizzes | 21 | /devii | 5 | /docs | 4 | -| /api | 20 | /posts | 4 | everything else | 36 | - -## 05 Route fan-out - -Source: `devplacepy/responses.py, schemas/, docs_api/, services/devii/actions/` - -One handler serves four consumers. `respond()` feeds the same context dict to a Pydantic model and to a Jinja template, so a key missing from the `*Out` schema is silently absent from JSON while still rendering in HTML. The agent catalog and the docs registry are separate declarations that must be kept in step by hand. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - form["models.py
Pydantic form model via Annotated Form"] - guard["guard: get_current_user
require_user / require_admin"] - handler["router handler"] - helpers["database/ batch helpers
get_users_by_uids, build_pagination"] - ctx["context dict"] - neg{"wants_json(request)"} - json["model.model_validate
JSONResponse"] - html["templates.TemplateResponse
extends base.html"] - action["services/devii/actions catalog
Action, requires_auth mirrors guard"] - docs["docs_api endpoint()
params and sample_response"] - seo["seo.py base_seo_context
JSON-LD, sitemap entry"] - form --> handler - guard --> handler - handler --> helpers --> ctx --> neg - neg -->|"json accepted"| json - neg -->|"otherwise"| html - handler -.->|"same capability"| action - handler -.->|"same contract"| docs - html -.-> seo -``` - -wants_json is true when the request content-type starts with application/json, or Accept contains application/json and does not contain text/html. Registry sizes: 318 agent actions, 259 documented endpoint entries across 20 API groups, 122 prose docs pages. - -## 06 Identity resolution - -Source: `devplacepy/utils/auth.py` - -A single resolver serves browsers, API clients, the devRant compatibility layer and issued access tokens. The result is memoised on `request.state` for the request and in a process TTL cache keyed by credential. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - start(["get_current_user"]) - cached{"request.state._auth_user set?"} - sess{"session cookie
64 hex chars"} - xapi{"X-API-KEY header"} - bearer{"Authorization: Bearer"} - basic{"Authorization: Basic"} - triple["try in order:
users.api_key,
devrant token 40 hex,
access token 64 chars"] - user(["user dict"]) - guest(["None, guest"]) - gu["require_user: 303 to /"] - ga["require_admin: redirect to /feed"] - start --> cached - cached -->|yes| user - cached -->|no| sess - sess -->|match| user - sess -->|no| xapi - xapi -->|present| triple - triple -->|match| user - triple -->|no match| bearer - xapi -->|absent| bearer - bearer -->|present| triple - bearer -->|absent| basic - basic -->|"username-or-email:password, pbkdf2_sha256"| user - basic -->|no| guest - guest --> gu - user --> ga -``` - -Roles are stored capitalized as Admin or Member and tested through the is_admin global. Admin seniority is enforced per user mutation in routers/admin/users.py. - -## 07 Background service fleet - -Source: `devplacepy/services/manager.py, base.py, jobs/base.py` - -27 singletons registered at boot, supervised only by the worker that won the service lock. Sixteen are long-lived loops on `BaseService`; eleven are queue consumers on `JobService`, which adds retention, concurrency and timeout settings on top and drains rows from the shared `jobs` table. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart LR - mgr["ServiceManager
supervise, shutdown_all"] - settings[("site_settings
enabled flag and interval per service")] - jobs[("jobs table")] - subgraph base["BaseService loops - 16"] - b1["NewsService"] - b2["BotsService"] - b3["GatewayService"] - b4["DeviiService"] - b5["PubSubService"] - b6["NotificationRelayService"] - b7["LiveViewRelayService"] - b8["PresenceRelayService"] - b9["IssueTrackerService"] - b10["ContainerService"] - b11["WorkspaceService"] - b12["XmlrpcService"] - b13["AuditService"] - b14["PushService"] - b15["TelegramService"] - b16["TelegramOutboxService"] - end - subgraph job["JobService consumers - 11"] - j1["ZipService"] - j2["ForkService"] - j3["SeoService"] - j4["SeoMetaService"] - j5["AwardService"] - j6["BackupService"] - j7["DbApiJobService"] - j8["DeepsearchService"] - j9["IsslopService"] - j10["IssueCreateService"] - j11["PlanningReportService"] - end - mgr --> base - mgr --> job - settings --> mgr - job --> jobs -``` - -Separate from the fleet, services/background.py offers a fire and forget queue per worker for audit writes, XP awards and notifications; when no consumer runs, as in tests, the callable executes inline so ordering stays deterministic. - -## 08 Data layer - -Source: `devplacepy/database/, devplacepy/config.py` - -One SQLite file reached through `dataset`, called synchronously from async handlers by design. 102 tables exist in the live schema; 52 of them carry the soft delete pair and are restorable as one event from the admin trash. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - db[("data/devplace.db")] - subgraph g1["Identity and access - 14"] - t1["users, sessions, password_resets
access_tokens, devrant_tokens
user_relations, follows
notification_preferences, user_customizations
push_registration, email_accounts
telegram_links, telegram_pairings, telegram_outbox"] - end - subgraph g2["Content and engagement - 16"] - t2["posts, comments, gists
projects, project_files, project_forks
attachments, news, news_images, news_sync
polls, poll_options, poll_votes
votes, reactions, bookmarks"] - end - subgraph g3["Reputation - 5"] - t3["awards, badges, award_usage
user_activity, user_activity_seen"] - end - subgraph g4["Code Farm - 9"] - t4["game_farms, game_plots, game_quests
game_cosmetics, game_market_ticks
game_steals, game_treasury
game_eras, game_era_results"] - end - subgraph g5["Quizzes - 5"] - t5["quizzes, quiz_questions, quiz_options
quiz_attempts, quiz_answers"] - end - subgraph g6["Live delivery - 3"] - t6["messages, notifications, ws_tickets"] - end - subgraph g7["Devii - 8"] - t7["devii_conversations, devii_turns
devii_tasks, devii_task_runs
devii_lessons, devii_virtual_tools
devii_behavior, devii_usage_ledger"] - end - subgraph g8["AI gateway - 6"] - t8["gateway_providers, gateway_models
gateway_usage_ledger
gateway_quota_rules, gateway_quota_resets
gateway_concurrency_samples"] - end - subgraph g9["Jobs, tools and usage - 20"] - t9["jobs, backups, backup_schedules
deepsearch_sessions, deepsearch_messages, deepsearch_url_cache
isslop_analyses, isslop_reports, isslop_events
isslop_dom_results, isslop_file_results, isslop_image_results
seo_metadata, seo_usage
issue_tickets, issue_comment_authors, issue_usage
correction_usage, modifier_usage, news_usage"] - end - subgraph g10["Containers - 9"] - t10["instances, instance_events, instance_metrics
tunnels, workspace_flags, workspace_quota_rules
builds, dockerfiles, dockerfile_versions"] - end - subgraph g11["Platform state - 7"] - t11["site_settings, cache_state, service_state
audit_log, audit_log_links
visit_stats_hourly, visit_unique_slots"] - end - db --> g1 - db --> g2 - db --> g3 - db --> g4 - db --> g5 - db --> g6 - db --> g7 - db --> g8 - db --> g9 - db --> g10 - db --> g11 -``` - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart LR - root["DEVPLACE_DATA_DIR
default repo/data"] - blobs["uploads, attachments, project_files
sharded xx/yy on the uuid7 random tail"] - jobsdir["zips, zip_staging, fork_staging
backups, backup_staging"] - reports["seo_reports, planning_reports
dbapi, deepsearch, deepsearch_chroma"] - isslop["isslop, isslop_workspaces
isslop_runs, isslop_media"] - ws["container_workspaces, workspace_state"] - keys["keys VAPID, bot, locks"] - dbs["devplace.db, devii_tasks.db, devii_lessons.db"] - root --> blobs - root --> jobsdir - root --> reports - root --> isslop - root --> ws - root --> keys - root --> dbs -``` - -config.DATA_PATHS registers 23 directories and ensure_data_dirs creates the whole tree before any write. Uploads live under data/uploads but are served at the unchanged /static/uploads URL. - -## 09 Content rendering - -Source: `devplacepy/rendering.py, static/js/ContentRenderer.js` - -Two pipelines with a deliberate split: anything that exists at request time is rendered on the server for SEO, and the client pipeline is reserved for content that does not exist yet. Each has its own XSS control at a different point. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"13px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - subgraph server["Server, rendering.py, lru_cache"] - a1["raw text"] - a2["normalize dashes to hyphen"] - a3["emoji shortcodes, 4869 names"] - a4["mistune GFM, escape=True"] - a5["media pass: bare URLs to embeds, mentions to links"] - a6["mask emails on rendered text nodes"] - a7["render_content / render_title in template"] - a8["ContentEnhancer adds highlighting and copy buttons"] - a1 --> a2 --> a3 --> a4 --> a5 --> a6 --> a7 --> a8 - end - subgraph client["Client, ContentRenderer.js"] - b1["live text: comments, DM bubbles, Devii, DeepSearch"] - b2["marked"] - b3["DOMPurify.sanitize, fail closed"] - b4["highlight.js"] - b5["media and autolink pass"] - b1 --> b2 --> b3 --> b4 --> b5 - end -``` - -The server escapes at the markdown step, the client sanitizes after parsing. Server rendered content must not carry data-render, or both pipelines run over it. - -## 10 AI plane - -Source: `devplacepy/routers/openai_gateway.py, services/openai_gateway/, services/devii/` - -Every model call in the product, internal or external, leaves through one gateway, which is where routing, quota and cost attribution live. Internal consumers call it over loopback HTTP rather than importing a client, so their spend lands in the same ledger. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - ext["external client
/openai/v1/*"] - subgraph consumers["Internal consumers via INTERNAL_GATEWAY_URL"] - c1["services/devii"] - c2["services/correction
AI correction and modifier"] - c3["services/news"] - c4["services/bot"] - c5["services/deepsearch + jobs/deepsearch"] - c6["services/dbapi nl2sql"] - c7["services/gitea enhance + planning"] - c8["jobs/isslop"] - end - gw["gateway.py
routing.py, quota.py, usage.py
reliability.py, vision.py"] - ledger[("gateway_usage_ledger
gateway_quota_rules")] - stealth["stealth_async_client
curl_cffi Chrome 146 fingerprint"] - up["upstream provider"] - subgraph devii["Devii assistant"] - d1["registry.py CATALOG
318 tools"] - d2["role gating
guest 108, member 246
admin 312, primary 318"] - d3["45 tools behind CONFIRM_REQUIRED"] - d4["session, tasks, lessons
virtual tools, behavior"] - d5["surfaces: /devii ws, Telegram, CLI"] - end - ext --> gw - consumers --> gw - gw --> ledger - gw --> stealth --> up - d1 --> d2 --> d3 - devii --> c1 - d4 --> d1 - d5 --> d4 -``` - -Cleartext loopback calls are forced to HTTP/1.1 in curl_transport.http_version_for, because the Chrome impersonation profile would otherwise negotiate HTTP/2 against an HTTP/1.1 only uvicorn. - -## 11 Containers, workspaces and ingress - -Source: `devplacepy/services/containers/, routers/projects/containers/, routers/proxy.py, routers/tunnel.py` - -One shared image serves every instance. Reconciliation is a loop that compares desired state in the database against what the Docker daemon actually reports, and there are two independent ways in from the outside: a path prefix and a hostname. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - ui["/projects/{slug}/containers
instances, schedules, workspace"] - api["services/containers/api.py"] - store[("instances, instance_events
instance_metrics, tunnels
workspace_flags, workspace_quota_rules")] - svc["ContainerService
reconcile desired vs docker ps"] - wsvc["WorkspaceService
provision, certs, quota, tunnels"] - backend["backend/docker_cli.py
backend/fake.py for tests"] - image["single shared image ppy:latest"] - inst["running instance
rootless workflow, aptroot"] - wsdir["data/container_workspaces/{uid}"] - proxy["/p/{slug}
routers/proxy.py, relays headers verbatim"] - tunnel["host based tunnel
TunnelDispatchMiddleware to routers/tunnel.py"] - code["workspace editor over websocket
/projects/{slug}/containers/instances/{uid}/code"] - exec["terminal over websocket
.../exec/ws"] - ui --> api --> store - svc --> store - wsvc --> store - api --> backend - svc --> backend - wsvc --> wsdir - backend --> image --> inst - proxy --> inst - tunnel --> inst - code --> inst - exec --> inst -``` - -Access uses stricter predicates than the rest of the product: owns_instance, can_view_project_containers, can_view_instance, can_manage_instance. The primary administrator sees and manages every container; any other admin can only view others on public projects and manage instances they own. - -## 12 Real time plane - -Source: `routers/*.py websocket handlers, services relays, nginx.conf.template` - -Twelve WebSocket endpoints, each of which needs its own nginx upgrade location because the catch-all location strips upgrade headers. Fan-out to connected sockets goes through in-process hubs driven by relay services on the lock-owning worker. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart LR - subgraph sockets["WebSocket endpoints"] - s1["/devii/ws"] - s2["/messages/ws"] - s3["/pubsub/ws"] - s4["/tools/seo/{uid}/ws"] - s5["/tools/deepsearch/{uid}/ws"] - s6["/tools/deepsearch/{uid}/chat"] - s7["/dbapi/query/{uid}/ws"] - s8[".../containers/instances/{uid}/exec/ws"] - s9[".../containers/instances/{uid}/code"] - s10[".../code/{path}"] - s11["/p/{slug}"] - s12["/p/{slug}/{path}"] - end - subgraph relays["Relay services"] - r1["NotificationRelayService"] - r2["LiveViewRelayService"] - r3["PresenceRelayService
track limit 500, online limit 30"] - r4["PubSubService"] - end - hubs["messaging/hub.py, pubsub/hub.py
devii/hub.py"] - tick[("ws_tickets
expiring auth tickets")] - s2 --> hubs - s3 --> hubs - s1 --> hubs - relays --> hubs - tick --> s2 -``` - -Presence is authoritative from the relay: PRESENCE_TRACK_LIMIT sets the tracked online set, PRESENCE_ONLINE_LIMIT only caps how many the feed panel displays, and PRESENCE_ONLINE_MARGIN_SECONDS provides hysteresis at the boundary. - -## 13 Frontend - -Source: `devplacepy/static/, devplacepy/templates/` - -No framework and no package manager. 131 ES6 modules, one class per file, hung off a single global `app`; 33 of them are custom elements. 217 Jinja templates, of which 107 are the docs site. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart TB - base["templates/base.html
extra_head for CSS, extra_js for JS"] - partials["shared partials
_avatar_link, _user_link, _comment_section
_post_card, _pagination, _sidebar_search"] - appjs["static/js/Application.js
instantiated once as app"] - subgraph modules["131 ES6 modules"] - u1["utilities
Http, Poller, JobPoller
OptimisticAction, FloatingWindow, ScrollMemory"] - u2["33 custom elements
components/ with dp- prefix"] - u3["feature modules
chat/, devii/, autoload/"] - end - css["49 stylesheets
variables.css tokens, per page files"] - vers["static_url in Jinja, assetUrl in JS
/static/v{boot ts}/, immutable for a year"] - cust["per user CSS and JS injection
custom_css_tag, custom_js_tag"] - base --> partials - base --> appjs --> modules - base --> css - base --> cust - vers --> css - vers --> modules -``` - -Per user customizations are configured only through Devii, stored in user_customizations, scoped globally or per matched route template, and run solely in that owner's own browser sessions. - -## 14 Test and delivery topology - -Source: `tests/, pyproject.toml, .gitea/workflows/test.yaml` - -Three tiers separated by what they exercise, decided by fixtures, run serially in a single process against one uvicorn subprocess on port 10501 with a temporary database. - -```mermaid -%%{init:{"theme":"base","themeVariables":{"background":"#F4F8F7","primaryColor":"#E4EEED","primaryTextColor":"#0F1A1B","primaryBorderColor":"#0A6D6A","secondaryColor":"#EDE6D9","tertiaryColor":"#F4F8F7","lineColor":"#6B7C7C","textColor":"#0F1A1B","fontFamily":"ui-monospace,SFMono-Regular,Menlo,monospace","fontSize":"12px","clusterBkg":"#FAFCFC","clusterBorder":"#C6D4D3"}}}%% -flowchart LR - unit["tests/unit - 140 modules
mirrors source module path
local_db or no fixture"] - api["tests/api - 221 modules
mirrors URL path
app_server, seeded_db"] - e2e["tests/e2e - 131 modules
mirrors URL path
page, alice, bob"] - srv["uvicorn subprocess :10501
temp SQLite, own DATA_DIR
DEVPLACE_DISABLE_SERVICES=1"] - pw["Playwright Chromium
session scoped context"] - ci["Gitea Actions on push and PR to master
full suite under coverage"] - prod["promotion to production
docker compose"] - unit --> ci - api --> srv --> ci - e2e --> srv - e2e --> pw --> ci - ci --> prod -``` - -pytest-xdist is not a dependency and -n is rejected centrally, so no tier can be parallelised by accident. - -## 15 Findings - -Source: `measured against the repository documentation` - -The structure holds up: the router tree mirrors the URL tree, the test tree mirrors both, every runtime path is registered in one place, and every model call has a single exit. The items below are the places where the running code and the written record have drifted apart, or where a structure exists that nothing currently uses. None of them is a functional defect. - -### Middleware order in the documentation is stale - -The root CLAUDE.md states that response_timing is the outermost middleware. Measured from app.user_middleware it is third, behind GZipMiddleware and TunnelDispatchMiddleware, both added after it. The consequence is minor but real: X-Response-Time excludes compression time and excludes tunnel-host dispatch entirely. - -### Two catalogue counts have drifted - -CLAUDE.md cites around 2882 tests and an events.md catalogue of 288 keys. Measured now: 3073 functions named test_ across 492 modules, and 312 distinct dotted event keys in events.md. Both are undercounts in the docs, not missing implementation. - -### instance_schedules is declared soft-deletable but has no table - -It appears in database.SOFT_DELETE_TABLES and in the routers under /projects/{slug}/containers/schedules, but no such table exists in the live schema. dataset creates it lazily on first insert, so this is correct only for as long as every read path tolerates the table being absent. - -### Three legacy container tables still exist - -builds, dockerfiles and dockerfile_versions are present in the live database. The project replaced per-project images with the single shared ppy:latest image and ships devplace containers prune-builds as the one-time cleanup for exactly these rows. The tables are still carried. - -### Five duplicate OpenAPI operation IDs - -Generating the schema warns on editor_proxy twice in routers/projects/containers/workspace.py, passthrough in routers/openai_gateway.py, and proxy_http twice in routers/proxy.py. These are catch-all routes registered for several methods, so a generated client would collide on those names. - -### The one deliberate asymmetry is documented and intentional - -routers/__init__.py holds nothing but the attribution line; unlike every nested router package, the top level does not aggregate. main.py performs all 38 include_router calls directly, which keeps prefix ownership in a single readable block. - ---- - -Sources: devplacepy/main.py, the imported FastAPI app object, data/devplace.db sqlite_master, and the working tree at HEAD 192df12b with uncommitted local modifications present. Counts exclude __pycache__ and the virtual environment. diff --git a/PODMAN.md b/PODMAN.md deleted file mode 100644 index d500e450..00000000 --- a/PODMAN.md +++ /dev/null @@ -1,78 +0,0 @@ -# Podman deployment (rootless, no docker daemon) - -Hosts without a docker daemon (like `devplace`) run the stack with the Python -`podman-compose` plus the `docker-compose.podman.yml` overlay. The base -`docker-compose.yml` is unchanged and stays the file rootful `docker compose` -uses; the overlay only adjusts what rootless podman needs. - -## Commands - -Always go through the make targets - they pin both `-f` files. A bare -`podman-compose up -d` drops the overlay and recreates a crashing app (see -"Why the overlay exists"). - -```bash -make podman-build # build images (only when pyproject/Dockerfiles change) -make podman-up # start (also warns if user linger is off) -make podman-reload # restart workers on new code (bind-mounted, no rebuild) -make podman-ps # container status -make podman-logs # tail output -make podman-down # stop -``` - -Open `http://:${PORT}` (default 10500, published on 127.0.0.1 for the -front proxy). Source-only changes need just `make podman-reload`. - -## Why the overlay exists - -The base file runs the app as `user: 1000:1000`, which is correct for rootful -docker (container uid 1000 == host uid 1000). Under **rootless** podman the -default userns instead maps container uid 1000 to a subordinate host uid -(100999), which cannot write the bind-mounted checkout (owned by host uid -1000) - uvicorn dies on startup with -`PermissionError: ... '/app/data/locks/devplace-init.lock'`, the app never -turns healthy, and nginx (which waits on `service_healthy`) never starts. - -The overlay sets `user: "0:0"` for the app: container root maps back to the -invoking host user under rootless, restoring write access while staying -unprivileged on the host. (`userns_mode: keep-id` would also fix the mapping -but podman rejects `--userns` together with `--pod`, and podman-compose puts -all services in one pod.) - -## One compose tool only - -Use the Python `podman-compose` (`/usr/bin/podman-compose`) for this stack, -never mix it with native `podman compose` in the same project directory. They -use different container names (`devplace_app_1` vs `devplace-app-1`) and would -create a second stack fighting over port 10500. - -## Surviving reboots - -Both services use `restart: unless-stopped`, but rootless containers only -restart after a reboot if the host user's systemd linger is enabled: - -```bash -loginctl show-user "$(id -un)" | grep Linger # want: Linger=yes -sudo loginctl enable-linger "$(id -un)" # fix if off -``` - -`make podman-up` prints a warning when linger is off. - -## Known differences from docker - -- The admin-only **Container Manager** needs the docker socket, which does not - exist on podman hosts. The `[containers] docker ps failed: [Errno 2]` lines - in the app log are expected noise, not an error; serving the site is - unaffected. -- `make docker-attach` / the containers overlay do not apply here. - -## Troubleshooting - -- `PermissionError ... /app/data/...` on boot: the app was (re)created - without the overlay. Run `make podman-up` (it recreates with the fix). -- `up` hangs with no output: it is waiting on the app healthcheck. In another - shell run `podman logs devplace_app_1` - a crash loop there is the cause. -- `Error: --userns and --pod cannot be set together`: a `userns_mode` - override is in play; remove it, this stack relies on `user: "0:0"`. -- Port 10500 already in use: a second stack (other compose tool, or bare-metal - `make prod`) is running. `podman ps -a` plus `ss -tlnp` show the conflict. diff --git a/devplacepy/services/game/CLAUDE.md b/devplacepy/services/game/CLAUDE.md index 17700d49..0c0cf0e0 100644 --- a/devplacepy/services/game/CLAUDE.md +++ b/devplacepy/services/game/CLAUDE.md @@ -107,6 +107,7 @@ Continuous Delivery used to be a one-way lock: the harvested crop was replanted - **The flag is read AFTER `clear_plot` wins.** `_try_autoreplant` re-reads the plot, so a toggle either landed before the clear (replant skipped) or finds an empty plot (`set_plot_stop`'s `WHERE crop_key != ''` fails, loud error). Never read the flag from the pre-clear plot dict and never reset it inside `clear_plot`. - **Every writer that puts a crop on a plot resets it:** the `plant()` and `_try_autoreplant` claim SETs write `stop_after_harvest = 0`, and `prestige()`'s plot reset writes it too. A stale `1` on an empty plot is harmless and is never serialized (`serialize_plot` gates `stop_after_harvest` on owner + Continuous Delivery + a crop present). - Column is `COALESCE`-read (`_lvl(plot, ...)`); existing rows are NULL. +- **Auto-replant is not a manual plant:** `_try_autoreplant` does not call `advance_quests("plant")`, writes no `game.plant` audit event and is not reported in the harvest result. This is documented for automation authors in `code-farm.html` ("Continuous Delivery and crop rotation (for bots)"); change both together. - `_crop_lock(farm, crop)` is the single lock predicate used by `plant()` and `_try_autoreplant`, so autoreplant no longer replants a crop that is locked (an Era crop after its Era ends). Saturation-based skipping was deliberately NOT built: at prestige 50+ (the only owners of Continuous Delivery) even a 0.4 market factor pays 11x or more the planting cost for every crop, so skipping only forfeits coins, XP and quest progress. ### 4. Secondary leaderboards (`economy.py` scoring + `store/farm.py`) diff --git a/devplacepy/templates/docs/code-farm.html b/devplacepy/templates/docs/code-farm.html index 45bfe9b2..eb63e23d 100644 --- a/devplacepy/templates/docs/code-farm.html +++ b/devplacepy/templates/docs/code-farm.html @@ -586,6 +586,129 @@ upgrade is owned and something was ready. **Era:** `era_active`, `era_name`, `era_coins`, `era_harvests`. +### Continuous Delivery and crop rotation (for bots) + +Continuous Delivery (Mastery upgrade `autoreplant`, 3 points, prestige 50 or higher) changes what an +empty plot means, so a bot that assumes "harvest, then plant" must account for it. This section is +the exact behavior, measured from the implementation. + +**What auto-replant does, step by step.** It runs in two places and nowhere else: right after a +manual `POST /game/harvest`, and inside the CI Bot's auto-harvest during a read of your own farm. +After the harvest has been paid, the server looks at that plot again and replants it only if all of +these hold: + +1. The plot is empty (the harvest cleared it and nothing else planted it in between). +2. The plot's pause flag is not set (see below). +3. The harvested crop is still unlocked for you (level, Mastery and Era checks, the same as + `POST /game/plant`). +4. You can pay the crop's current planting cost. Coins are charged atomically first, so a refused + charge leaves the plot empty and costs nothing. + +If it replants, it plants **the same crop that was just harvested**. It never picks a better crop +and it does **not** look at Market Saturation: a saturated crop is replanted exactly like any other. +The new planting starts at the moment of the harvest, uses your current CI tier, perks and Legacy +speed for its build time, and gets its own fresh golden-build roll. + +Side effects that differ from a manual `POST /game/plant`: + +| | Manual plant | Auto-replant | +|---|---|---| +| Charges the planting cost | yes | yes | +| Advances the daily "plant" quest | **yes** | **no** | +| Writes a `game.plant` audit event | yes | no | +| Reported in the harvest response | n/a | **no**, read the plot from the returned `farm` | + +The harvest itself (coins, XP, the "harvest" and "earn" quests, the weekly harvest count) is +identical either way. If you farm the daily "plant" quest, plant at least once by hand. + +**The one-shot pause: `POST /game/plot/stop`.** Before this existed, the only way to change what a +plot grows was a Refactor. Now you can pause the replant for one plot, one harvest at a time: + +| Field | Type | Meaning | +|---|---|---| +| `slot` | integer, 0-64 | Plot to change. | +| `stop` | boolean | `true` skips the replant after the next harvest of this plot. `false` cancels a pending pause. | + +Returns the usual `{"ok": true, "farm": {...}}`. Rules, all enforced server side: + +- It needs Continuous Delivery. Without it: **400** `Continuous Delivery is not unlocked.` +- The plot must currently hold a crop (growing or ready). On an empty plot: **400** + `That plot is empty.` This is also the answer when the harvest wins a race against your call (see + below), so treat it as "go read the state", not as a failure. +- It is **one-shot and per plot**. It applies to the next harvest of that plot only, and each plot + needs its own call. Setting it again on a plot that already has it, or clearing it on a plot + that does not, succeeds and changes nothing. +- It is cleared automatically the next time anything puts a crop on that plot: your manual + `POST /game/plant`, an auto-replant, or a Refactor. You never have to clear it by hand. +- It costs nothing and is not refundable or limited, but it is a mutating call, so it counts + against the rate limit like any other POST. + +**What you can read.** Each plot in your own farm carries two booleans, always present: + +- `can_stop_replant` - true on a plot that holds a crop when you own Continuous Delivery. False on + an empty plot, without the upgrade, and on every plot of someone else's farm. +- `stop_after_harvest` - true while the pause is set on a plot that still holds a crop. It is + never true on an empty plot, on a plot of another farm, or without the upgrade, so a stale flag + can never be observed. + +**Rotation recipe.** To change the crop on a plot: + +1. Call `POST /game/plot/stop` with `stop=true` while the plot is growing. Do it well before + `ready_at`, because with the CI Bot the harvest can happen during any read of your farm once the + build is ready. +2. Wait until `ready_at` has passed, then harvest (or let the CI Bot do it on your next read). +3. The plot is now empty. Plant what you want with `POST /game/plant`. The pause is cleared by that + plant. + +Between the harvest and your plant the plot sits idle, so every rotation costs you the time it +takes your bot to notice. Without the CI Bot that is up to one poll interval. A bot that polls +every 60 seconds loses about half a minute per rotation on average, which is why rotating only pays +off when the new crop beats the old one by more than that idle time. + +**Race behavior (verified with concurrent processes).** A pause call and a harvest of the same plot +at the same instant always resolve one of two ways, never a third: either the pause lands first +(HTTP 200, the plot ends empty) or the harvest lands first (the plot is already replanted, and your +call gets the 400 `That plot is empty.`). We ran 150 rounds with the two calls in separate +processes: 87 pause-first, 63 harvest-first, 0 inconsistent. After a 400 on this endpoint, re-read +`GET /game/state` and look at the plot instead of retrying blindly. + +**Do you need to rotate to avoid Market Saturation?** Usually not, and the numbers say so. A +saturated crop pays at worst 40 percent of its normal value. At prestige 50 the prestige multiplier +is 13.5, so even with no perks, no golden build and the 40 percent floor, every crop pays at least +about 11 times its planting cost (shell about +54 coins on a 5 coin plant, kernel about +18,240 on +1,200). Skipping a replant to dodge saturation therefore usually earns less than replanting into +it, because an idle plot earns nothing and also forgoes the XP, quest progress and weekly harvest +count of the cycle. Rotate when a **different crop pays more per second**, not merely to avoid the +`saturated` label. These figures are computed from the published formulas, not measured on a live +farm, and they change if the saturation tiers or prestige multiplier are rebalanced. + +**Edge cases a bot should handle.** + +- An auto-replant can be refused (cost, lock). The plot then just stays empty, so a loop that + plants every `state == "empty"` plot, like the script below, already covers it. +- An Era crop stops being plantable when its Era ends. Auto-replant now respects that and leaves the + plot empty instead of replanting a locked crop. +- The upgrade itself is permanent: Mastery points are never refunded and there is no switch that + turns Continuous Delivery off for the whole farm. The per-plot pause is the only control. +- A Refactor clears every plot, does **not** auto-replant, and clears all pauses. Plant again after + it. + +A rotation helper that moves every plot to one target crop, using only the standard library: + +```python +def rotate_to(call, wanted): + farm = call("GET", "/game/state")["farm"] + for plot in farm["plots"]: + if plot["state"] == "empty": + call("POST", "/game/plant", {"slot": plot["slot"], "crop": wanted}) + elif plot["crop_key"] != wanted and plot["can_stop_replant"] and not plot["stop_after_harvest"]: + call("POST", "/game/plot/stop", {"slot": plot["slot"], "stop": "true"}) +``` + +Call it on every poll. A plot growing the wrong crop gets paused once, its harvest leaves it empty, +and the next poll plants `wanted` there. Plots already growing `wanted` are left alone, and the +auto-replant keeps them running. + ### A complete automated farmer This script plays the core loop on a schedule: it claims the daily bonus and any finished quests, diff --git a/docker-compose.yml.bak-20261008 b/docker-compose.yml.bak-20261008 deleted file mode 100644 index 4ea9a368..00000000 --- a/docker-compose.yml.bak-20261008 +++ /dev/null @@ -1,56 +0,0 @@ -name: devplace - -services: - app: - build: - context: . - dockerfile: Dockerfile - restart: unless-stopped - env_file: - - .env - user: "${DEVPLACE_UID:-1000}:${DEVPLACE_GID:-1000}" - working_dir: /app - environment: - HOME: /app - PYTHONDONTWRITEBYTECODE: "1" - # Pinned regardless of .env: the container always binds 10500 internally - # (Dockerfile CMD, nginx upstream). DEVPLACE_PORT is the bare-metal - # `make dev`/`make prod` knob; the externally reachable port is `PORT`. - DEVPLACE_PORT: "10500" - volumes: - - .:/app - expose: - - "10500" - networks: - - appnet - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:10500/"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 120s - start_interval: 2s - - nginx: - build: - context: . - dockerfile: nginx/Dockerfile - restart: unless-stopped - ports: - - "127.0.0.1:${PORT:-10500}:80" - environment: - NGINX_CACHE_ENABLED: "${NGINX_CACHE_ENABLED:-false}" - NGINX_CACHE_MAX_SIZE: "${NGINX_CACHE_MAX_SIZE:-1g}" - NGINX_MAX_BODY_SIZE: "${NGINX_MAX_BODY_SIZE:-50m}" - volumes: - - ./devplacepy/static:/app/static:ro - - ${DEVPLACE_DATA_DIR:-./data}/uploads:/data/uploads:ro - depends_on: - app: - condition: service_healthy - networks: - - appnet - -networks: - appnet: - driver: bridge diff --git a/dptodo.md b/dptodo.md deleted file mode 100644 index 3c9c3716..00000000 --- a/dptodo.md +++ /dev/null @@ -1,1272 +0,0 @@ -# DevPlace.net — Comprehensive Issue Tracker & blindxfish Critique Report - -**Generated:** 2026-09-10 13:01:58 UTC -**Source Data:** https://devplace.net/issues (native issue tracker), blindxfish critique research -**Compiled by:** molodetz (Worker/Developer Agent) - ---- - -## TABLE OF CONTENTS - -1. [Executive Summary](#1-executive-summary) -2. [blindxfish — Profile & Complete Critique Log](#2-blindxfish--profile--complete-critique-log) -3. [All Open Actionable Issues — Detailed Breakdown](#3-all-open-actionable-issues--detailed-breakdown) - - [3.1 Critical Priority](#31-critical-priority) - - [3.2 High Priority](#32-high-priority) - - [3.3 Medium Priority](#33-medium-priority) - - [3.4 Low Priority](#34-low-priority) -4. [Issues Filed BY blindxfish (Actionable)](#4-issues-filed-by-blindxfish-actionable) -5. [Issues AFFECTING blindxfish](#5-issues-affecting-blindxfish) -6. [Excluded / Non-Actionable Issues](#6-excluded--non-actionable-issues) -7. [Recommended Actions & Prioritization Matrix](#7-recommended-actions--prioritization-matrix) -8. [Appendix: Closed Issues Reference](#8-appendix-closed-issues-reference) - ---- - -## 1. EXECUTIVE SUMMARY - -### Counts at a Glance - -| Metric | Value | -|--------|-------| -| Total Issues in Tracker | 133 | -| Open Issues | 44 | -| Closed Issues | 89 | -| **Actionable Open Issues** | **41** | -| Excluded (spam/test/vague) | 3 | -| Issues filed by blindxfish (open) | 5 | -| Issues affecting blindxfish | 1 | -| Issues already fixed (need closing) | 3 | -| Issues with PRs opened but still open | 4+ | - -### Priority Overview - -| Priority | Count | Key Themes | -|----------|-------|------------| -| **Critical** | 1 | Session cookie auth bug (#83) | -| **High** | 11 | Security, stability, data corruption, core UX broken | -| **Medium** | 19 | Feature requests, UX improvements, API completeness | -| **Low** | 10 | Polish, minor bugs, fun requests | - -### Top Concerns - -1. **#132** — Missing notification on comment reply (131 comments, 457+ hours of failed attempts) — **Most critical UX gap** -2. **#83** — Session cookie not set after login (auth broken for users) — **Critical security/auth issue** -3. **#155** — Push unsubscribe endpoint missing (privacy/security) — 46 comments, 40+ failed attempts -4. **#161** — avatar_seed missing from search results (mention avatars wrong) — 36 comments, 30+ failed attempts -5. **#70** — Unauthorized external probes on AI gateway — **Active security vulnerability** -6. **#84** — AI markdownifyer corrupts valid Markdown — **Data integrity issue** -7. **#102** — XML-RPC bridge crash loop — **Stability issue** - -### blindxfish Context - -blindxfish is a DevPlace Admin (level 323, 32,223 XP, Netherlands) who has contributed visual/design work to the platform while also filing several feature requests and critiquing UX issues. They were also affected by AI quota exhaustion (#74). Their issues (#57, #61) have been resolved; #148 (Code Farm raiding) has an open PR; #58 was duplicative/nonsensical. - ---- - -## 2. BLINDXFISH — PROFILE & COMPLETE CRITIQUE LOG - -### User Profile - -| Field | Details | -|-------|---------| -| **DevPlace Username** | blindxfish | -| **Role** | Admin | -| **Level** | 323 | -| **XP** | 32,223 | -| **Location** | Netherlands | -| **GitHub** | https://github.com/blindxfish (NodeOpenSmartDriver, Loading) | -| **itch.io** | https://blindxfish.itch.io/ (Nuclear Airship Operator, Tower Together, etc.) | -| **devRant** | @blindXfish2627 | - -### All Known Critiques & Posts by blindxfish About DevPlace - -#### 2.1 devRant Rants (External Platform Critiques) - -**Rant #19386928** — UI Design Dissatisfaction -> "I have completely dismantled and readembled my.devplace, basically revamped for game development, refactored so I don't need an anti vomit pill to work on it." - -- **Date:** Not specified in source data -- **Topic:** UI/UX design quality -- **Assessment:** blindxfish found the default profile page design so poor that they personally rebuilt it entirely. This indicates a significant gap between the platform's default visual quality and what power users expect. The hyperbolic language ("anti vomit pill") suggests genuine frustration. - -**Rant #19423487** — Visual Contribution Acknowledgment -> "@blindxfish couldn't live with my design 'skills' so he helped a bit" - -- **Date:** Not specified in source data -- **Topic:** Collaborative contribution -- **Assessment:** Platform admin acknowledged blindxfish stepped in to help with design work because they "couldn't live with" their own design skills. This confirms blindxfish has legitimate design ability and has actively improved the platform visually. - -**Rant #19442359** — Public Recognition -> "we received nice contributions from @blindxfish visually" - -- **Date:** Not specified in source data -- **Topic:** Visual contributions -- **Assessment:** Official acknowledgment from the team that blindxfish made positive visual/design contributions to the platform. - -#### 2.2 Issues Filed by blindxfish on DevPlace - -**Issue #57** — Add ability to paste images into issue description -- **Status:** ALREADY FIXED (paste handling implemented in IssueReporter.js) -- **Created:** 2026-07-01 -- **Comments:** 27 -- **Assessment:** Valid UX request. Should be closed. - -**Issue #61** — Make card columns scroll independently of the feed -- **Status:** ALREADY FIXED (fix applied in production) -- **Created:** 2026-07-01 -- **Comments:** 10 -- **Assessment:** Valid UX request. Should be closed. - -**Issue #58** — Integrate MicroKingdoms plots as farms in DevPlace and the farm game -- **Status:** DUPLICATE/NONSENSE -- **Created:** 2026-07-01 -- **Comments:** 9 -- **Assessment:** Based on misunderstanding of existing Code Farm mechanics. Code Farm already has farms/plots/stealing. MicroKingdoms concept does not exist in codebase. Should be closed. - -**Issue #148** — Add risk, defense systems, and penalties to Code Farm raiding -- **Status:** IN PROGRESS (PR #149 opened by typosaurus) -- **Created:** 2026-07-27 -- **Comments:** 12 -- **Assessment:** Legitimate game design feature request. Proposes purchasable defenses (mines/traps, security cameras, guard dogs, insurance) and penalties for raiders (fines, escalating consequences, cooldowns). Implementation drafted and PR opened. Track to merge. - -**Issue #135** — Add devlog timeline of related posts to project page -- **Status:** CLOSED -- **Created:** 2026-07-25 -- **Comments:** 17 -- **Assessment:** Closed issue — likely implemented or superseded. - -**Issue #60** — Add online status card for followed users on feed sidebar -- **Status:** CLOSED -- **Created:** 2026-07-01 -- **Comments:** 2 -- **Assessment:** Closed issue. - -**Issue #59** — Integrate Snek Chat natively into DevPlace for real-time communication -- **Status:** CLOSED -- **Created:** 2026-07-01 -- **Comments:** 1 -- **Assessment:** Closed issue. - -#### 2.3 blindxfish as Affected User - -**Issue #74** — AI quota exhausted for m30n and blindxfish; quotas not reset, users blocked for days -- **Author:** retoor -- **Affected Users:** m30n, **blindxfish** -- **Created:** 2026-07-13 -- **Comments:** 8 -- **Assessment:** blindxfish was a *victim* of this issue, not the reporter. Investigation confirmed quotas DO reset automatically; the issue is operational/configurational rather than a code defect. The $1.00/24h cap may be too low. Consider raising the cap. - -### blindxfish Critique Synthesis - -blindxfish's critiques reveal three main themes: - -1. **Visual Quality Gap:** The default UI/profile design is substandard enough that a power user/admin felt compelled to rebuild it entirely. This is both a criticism and a testament to blindxfish's design capabilities. - -2. **Feature Gaps:** blindxfish identified legitimate UX needs (image pasting in issues, independent column scrolling) which were subsequently implemented. - -3. **Game Design Vision:** blindxfish proposed meaningful game mechanics additions (Code Farm raiding defenses) that align with the platform's gamification direction. - ---- - -## 3. ALL OPEN ACTIONABLE ISSUES — DETAILED BREAKDOWN - -### 3.1 CRITICAL PRIORITY - -#### ISSUE #83 — Session cookie not set in browser after successful login form submission - -| Field | Value | -|-------|-------| -| **Issue Number** | #83 | -| **Title** | Session cookie not set in browser after successful login form submission | -| **Status** | open | -| **Author** | m30n | -| **Author UID** | 019eab6f-e7cc-7a53-8e11-76ca73521e3b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 10 | -| **Created** | 2026-07-16T03:47:21+02:00 | -| **Updated** | 2026-07-23T01:55:54+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/83 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** This is a **critical authentication bug**. If the session cookie is not set after login, users cannot maintain authenticated sessions. This effectively breaks the login flow for some users, preventing access to private features (messages, notifications, game, profile settings). This could affect all users depending on browser/cookie configuration. - -**Recommended Action:** Immediate investigation. Check Set-Cookie header in login response, verify cookie domain/path attributes, check SameSite attribute compatibility. - ---- - -### 3.2 HIGH PRIORITY - -#### ISSUE #132 — Missing notification when another user comments on a post the user commented on - -| Field | Value | -|-------|-------| -| **Issue Number** | #132 | -| **Title** | Missing notification when another user comments on a post the user commented on | -| **Status** | open | -| **Author** | D-04got10-01 | -| **Author UID** | 912f0198-16fb-46de-91c3-5c582b359447 | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 131 | -| **Created** | 2026-07-24T15:29:09+02:00 | -| **Updated** | 2026-08-15T02:40:19+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/132 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** This is the **most-commented open issue** (131 comments) and represents a major UX gap. On devRant (the predecessor platform), receiving notifications for replies to your comments was standard behavior. Without this, users miss conversations and engagement drops significantly. Core functionality delivered via PRs (#133, #142, #163) but admin defaults verification and test writing consistently failed. Duration exceeded 457 hours across multiple runs. - -**Recommended Action:** Verify admin notification defaults are correctly configured. Write and run tests. Close once verified. - ---- - -#### ISSUE #155 — Add server-side push unsubscribe endpoint (APNs device tokens stay registered after logout) - -| Field | Value | -|-------|-------| -| **Issue Number** | #155 | -| **Title** | Add server-side push unsubscribe endpoint (APNs device tokens stay registered after logout) | -| **Status** | open | -| **Author** | Lensflare | -| **Author UID** | 019eca8b-15ed-7131-8e16-804f36f185fd | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 46 | -| **Created** | 2026-08-04T17:17:05+02:00 | -| **Updated** | 2026-08-08T19:26:18+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/155 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Privacy and security concern. When users log out of native apps, APNs device tokens remain registered on Apple's servers, meaning they continue receiving push notifications. This is a privacy violation — logged-out users should not receive pushes. Needs a `DELETE /push.json` endpoint. 46 comments, 40+ failed implementation attempts over 87+ hours. - -**Recommended Action:** Implement DELETE endpoint for push token unregistration. Ensure it works for both APNs and any other push providers. - ---- - -#### ISSUE #84 — AI markdownifyer plugin corrupts valid Markdown structures - -| Field | Value | -|-------|-------| -| **Issue Number** | #84 | -| **Title** | AI markdownifyer plugin corrupts valid Markdown structures | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 48 | -| **Created** | 2026-07-18T14:01:26+02:00 | -| **Updated** | 2026-07-23T04:20:53+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/84 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Content corruption bug. The AI markdown processing plugin is breaking valid Markdown structures, which affects content rendering across the platform. 48 comments indicate significant user impact. - -**Recommended Action:** Audit the markdownifyer plugin's transformation logic. Add regression tests for common Markdown patterns (lists, code blocks, headers, links). - ---- - -#### ISSUE #85 — Duplicate chat message submissions produce inconsistent AI-corrected content - -| Field | Value | -|-------|-------| -| **Issue Number** | #85 | -| **Title** | Duplicate chat message submissions produce inconsistent AI-corrected content | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 44 | -| **Created** | 2026-07-18T14:01:26+02:00 | -| **Updated** | 2026-07-23T03:33:33+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/85 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Data integrity bug. Duplicate message submissions cause inconsistent AI corrections, leading to corrupted chat content. 44 comments suggest frequent occurrence. - -**Recommended Action:** Implement deduplication at the chat message submission layer. Add idempotency keys for message creation. - ---- - -#### ISSUE #86 — Investigate chat system rendering bug observed while away from dev environment - -| Field | Value | -|-------|-------| -| **Issue Number** | #86 | -| **Title** | Investigate chat system rendering bug observed while away from dev environment | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 35 | -| **Created** | 2026-07-18T14:01:32+02:00 | -| **Updated** | 2026-07-23T03:32:15+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/86 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Chat rendering bug observed during absence from dev environment. 35 comments indicate ongoing investigation. Chat is a core feature — rendering issues directly impact user experience. - -**Recommended Action:** Reproduce the rendering bug. Check for race conditions, state management issues, or component lifecycle problems. - ---- - -#### ISSUE #70 — Restrict OpenAI gateway endpoint to internal network after detecting unauthorized external probes - -| Field | Value | -|-------|-------| -| **Issue Number** | #70 | -| **Title** | Restrict OpenAI gateway endpoint to internal network after detecting unauthorized external probes | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 21 | -| **Created** | 2026-07-13T00:37:05+02:00 | -| **Updated** | 2026-07-23T02:52:41+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/70 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** **Active security vulnerability.** Unauthorized external probes detected against the OpenAI gateway endpoint. This could lead to API key abuse, unexpected costs, or data exfiltration. Must restrict to internal network immediately. - -**Recommended Action:** Implement IP-based access control (firewall rules, nginx config, or middleware). Add monitoring/alerting for unauthorized access attempts. - ---- - -#### ISSUE #72 — Fix empty gateway_embed_key causing embeddings API fallback to invalid key - -| Field | Value | -|-------|-------| -| **Issue Number** | #72 | -| **Title** | Fix empty gateway_embed_key causing embeddings API fallback to invalid key | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 27 | -| **Created** | 2026-07-13T00:37:09+02:00 | -| **Updated** | 2026-07-23T02:45:40+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/72 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** API bug where an empty `gateway_embed_key` causes the embeddings API to fall back to an invalid key, breaking embedding functionality. 27 comments indicate persistent impact. - -**Recommended Action:** Add validation for `gateway_embed_key`. Implement proper fallback chain with clear error messages when no valid key is available. - ---- - -#### ISSUE #75 — Google Read Aloud bot repeatedly crawls auth-gated endpoints (/messages, /notifications, /game) - -| Field | Value | -|-------|-------| -| **Issue Number** | #75 | -| **Title** | Google Read Aloud bot repeatedly crawls auth-gated endpoints (/messages, /notifications, /game) | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 8 | -| **Created** | 2026-07-13T00:37:23+02:00 | -| **Updated** | 2026-07-23T03:02:49+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/75 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Security issue — a Google bot is crawling private/auth-gated endpoints. While these should be protected by auth, the bot activity wastes server resources and could expose sensitive information if auth checks fail. - -**Recommended Action:** Block the Google Read Aloud bot via robots.txt, user-agent filtering, or WAF rules. Monitor for continued access attempts. - ---- - -#### ISSUE #94 — Vote/star button unresponsive on certain posts/comments (intermittent) - -| Field | Value | -|-------|-------| -| **Issue Number** | #94 | -| **Title** | Vote/star button unresponsive on certain posts/comments (intermittent) | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 56 | -| **Created** | 2026-07-18T14:03:49+02:00 | -| **Updated** | 2026-07-23T05:28:49+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/94 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Core interaction bug — voting and starring are fundamental platform actions. Intermittent unresponsiveness means users sometimes cannot interact with content. 56 comments indicate widespread impact. Second-highest comment count among open issues. - -**Recommended Action:** Investigate event handler binding, race conditions, and error states. Add error handling and user feedback for failed vote operations. - ---- - -#### ISSUE #93 — Fix incorrect Content-Length headers causing truncated responses and timeouts - -| Field | Value | -|-------|-------| -| **Issue Number** | #93 | -| **Title** | Fix incorrect Content-Length headers causing truncated responses and timeouts | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 26 | -| **Created** | 2026-07-18T14:03:49+02:00 | -| **Updated** | 2026-07-23T04:06:50+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/93 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** HTTP protocol bug. Incorrect Content-Length headers cause clients to receive truncated responses and timeout waiting for data that will never arrive. This affects all API consumers and web clients. - -**Recommended Action:** Audit all response handlers for correct Content-Length calculation. Consider switching to chunked transfer encoding where appropriate. - ---- - -#### ISSUE #102 — XML-RPC bridge crashes in 15-second fork-and-exit loop - -| Field | Value | -|-------|-------| -| **Issue Number** | #102 | -| **Title** | XML-RPC bridge crashes in 15-second fork-and-exit loop | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 42 | -| **Created** | 2026-07-18T14:11:30+02:00 | -| **Updated** | 2026-07-23T04:25:25+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/102 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Stability bug. The XML-RPC bridge enters a crash loop, spawning and exiting processes every 15 seconds. This wastes CPU/memory resources and renders the XML-RPC service unavailable. 42 comments indicate persistent disruption. - -**Recommended Action:** Fix the root cause of the fork-and-exit loop. Add process supervision (systemd, supervisord) to prevent resource exhaustion. - ---- - -### 3.3 MEDIUM PRIORITY - -#### ISSUE #167 — Suppress notifications for active one-to-one conversations - -| Field | Value | -|-------|-------| -| **Issue Number** | #167 | -| **Title** | Suppress notifications for active one-to-one conversations | -| **Status** | open | -| **Author** | Lensflare | -| **Author UID** | 019eca8b-15ed-7131-8e16-804f36f185fd | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 3 | -| **Created** | 2026-08-14T22:43:21+02:00 | -| **Updated** | 2026-08-15T03:51:24+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/167 | -| **Description** | New messages in active 1:1 conversations produce notification spam. Desired: suppress all notifications when user is already in that conversation. | -| **Key Comments** | Claimed by typosaurus team; attempted fix failed; no PR delivered. | - -**Priority Justification:** Valid UX improvement. Users actively chatting in DMs get spammed with notifications for messages they're already seeing. Reduces notification noise. - -**Recommended Action:** Implement notification suppression for active conversation windows. Track active tab/session state. - ---- - -#### ISSUE #162 — Add non-color indicator to vote button state - -| Field | Value | -|-------|-------| -| **Issue Number** | #162 | -| **Title** | Add non-color indicator to vote button state | -| **Status** | open | -| **Author** | D-04got10-01 | -| **Author UID** | 912f0198-16fb-46de-91c3-5c582b359447 | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 9 | -| **Created** | 2026-08-05T14:28:48+02:00 | -| **Updated** | 2026-08-16T05:02:16+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/162 | -| **Description** | Vote buttons (+/-) indicate voted state only by color change — insufficient for color vision deficiencies. Needs non-color cue (glyph, shape, outline, fill). Reporter suggested Unicode equivalents or bracket notation like `[+]`. Lensflare suggested iOS-style filled circles. | -| **Key Comments** | Claimed by typosaurus; 3 commits made; PR opened (#168); still marked open. | - -**Priority Justification:** Accessibility requirement. Color-only indicators fail WCAG guidelines and exclude colorblind users. Rare success case — typosaurus delivered 3 commits and opened PR #168. - -**Recommended Action:** Review PR #168, merge if acceptable, close issue. - ---- - -#### ISSUE #161 — Include avatar_seed in user search results for mention autocomplete - -| Field | Value | -|-------|-------| -| **Issue Number** | #161 | -| **Title** | Include avatar_seed in user search results for mention autocomplete | -| **Status** | open | -| **Author** | Lensflare | -| **Author UID** | 019eca8b-15ed-7131-8e16-804f36f185fd | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 36 | -| **Created** | 2026-08-05T13:04:02+02:00 | -| **Updated** | 2026-08-16T04:06:28+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/161 | -| **Description** | `/profile/search` and `/messages/search` return only `{uid, username}` per result, omitting `avatar_seed`. This causes mention dropdowns to show wrong avatars for users with custom seeds. Native iOS client needs this fix. | -| **Key Comments** | 30+ failed typosaurus attempts over many hours; retoor commented asking about delivery. | - -**Priority Justification:** Technical improvement affecting mention UX. Users with custom avatars see wrong avatars in mentions. Particularly impactful for the native iOS client. 36 comments, 30+ failed attempts. - -**Recommended Action:** Add `avatar_seed` field to search result DTOs. Update frontend mention dropdown to use the seed. - ---- - -#### ISSUE #154 — Fix notifications Load More feedback and URL-encode before cursor - -| Field | Value | -|-------|-------| -| **Issue Number** | #154 | -| **Title** | Fix notifications Load More feedback and URL-encode before cursor | -| **Status** | open | -| **Author** | Lensflare | -| **Author UID** | 019eca8b-15ed-7131-8e16-804f36f185fd | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 0 | -| **Created** | 2026-08-04T14:43:38+02:00 | -| **Updated** | 2026-08-04T14:43:38+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/154 | -| **Description** | The "Load More" button on `/notifications` appears broken: clicking it causes full page navigation with no loading state; scroll position resets to top and visible content looks identical to previous page. Secondary issue: `before` cursor ships raw unencoded `+` in query string. Expected: AJAX append keeping scroll position, or at minimum loading state + time range display. Cursor should be percent-encoded. | -| **Key Comments** | None | - -**Priority Justification:** Bug fix for notification pagination UX. Two issues: (1) Load More behaves like full page nav instead of AJAX append, (2) cursor parameter not URL-encoded. Both degrade the notification browsing experience. - -**Recommended Action:** Implement AJAX-based Load More with loading spinner. Percent-encode cursor values in query strings. - ---- - -#### ISSUE #152 — Fix pull request links in bot answers including trailing punctuation - -| Field | Value | -|-------|-------| -| **Issue Number** | #152 | -| **Title** | Fix pull request links in bot answers including trailing punctuation | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 16 | -| **Created** | 2026-07-28T21:43:07+02:00 | -| **Updated** | 2026-08-02T01:00:15+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/152 | -| **Description** | Bot posts PR URLs at end of sentences with trailing punctuation included inside hyperlink, causing 404 errors. | -| **Key Comments** | typosaurus implemented fix on first attempt, passed all checks, pushed branch `typosaurus/ticket-152`, opened PR #153. retoor thanked bots but noted the formatting bot stripped half the description. Alice noted trailing period could be validly part of URL. | - -**Priority Justification:** Minor bug in bot-generated content. Trailing punctuation in PR URLs causes 404s. Fix implemented in PR #153 but issue remains open. - -**Recommended Action:** Review and merge PR #153. Close issue. - ---- - -#### ISSUE #114 — Add API support for vote value 0 to allow vote retraction - -| Field | Value | -|-------|-------| -| **Issue Number** | #114 | -| **Title** | Add API support for vote value 0 to allow vote retraction | -| **Status** | open | -| **Author** | Lensflare | -| **Author UID** | 019eca8b-15ed-7131-8e16-804f36f185fd | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 43 | -| **Created** | 2026-07-22T16:14:15+02:00 | -| **Updated** | 2026-07-27T02:21:57+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/114 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** API completeness issue. Users cannot retract votes through the API — there's no way to send a "vote value 0" to undo a previous vote. This limits client-side UX options for vote correction. - -**Recommended Action:** Add support for `vote_value=0` in the voting API endpoint to allow vote retraction. - ---- - -#### ISSUE #111 — Add optionality info to API definition for nullable properties - -| Field | Value | -|-------|-------| -| **Issue Number** | #111 | -| **Title** | Add optionality info to API definition for nullable properties | -| **Status** | open | -| **Author** | Lensflare | -| **Author UID** | 019eca8b-15ed-7131-8e16-804f36f185fd | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 25 | -| **Created** | 2026-07-21T01:04:08+02:00 | -| **Updated** | 2026-07-23T05:08:29+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/111 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** API documentation improvement. Client developers need to know which properties are nullable vs required. Without this, clients may crash on unexpected null values or fail to handle optional fields correctly. - -**Recommended Action:** Update OpenAPI/Swagger spec to mark nullable properties with `nullable: true` and appropriate descriptions. - ---- - -#### ISSUE #110 — Private Message Chat Scrolling Issue - -| Field | Value | -|-------|-------| -| **Issue Number** | #110 | -| **Title** | Private Message Chat Scrolling Issue | -| **Status** | open | -| **Author** | Lensflare | -| **Author UID** | 019eca8b-15ed-7131-8e16-804f36f185fd | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 21 | -| **Created** | 2026-07-20T16:19:16+02:00 | -| **Updated** | 2026-07-23T05:46:35+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/110 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Bug in DM chat scrolling behavior. Affects the private messaging experience. 21 comments indicate persistent user frustration. - -**Recommended Action:** Investigate scroll position management in the DM chat component. Ensure auto-scroll-to-bottom works correctly on new messages. - ---- - -#### ISSUE #92 — Tagging a non-existent username causes noticeable processing delay - -| Field | Value | -|-------|-------| -| **Issue Number** | #92 | -| **Title** | Tagging a non-existent username causes noticeable processing delay | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 7 | -| **Created** | 2026-07-18T14:03:42+02:00 | -| **Updated** | 2026-07-23T03:20:00+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/92 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Performance bug. When users type a mention that doesn't match any existing user, the system takes noticeably long to respond with "user not found." This creates a poor typing experience. - -**Recommended Action:** Implement client-side debounce and timeout for username lookups. Return early if character count is below minimum threshold for valid usernames. - ---- - -#### ISSUE #90 — Add shareable permalinks to comments for consistent cross-referencing - -| Field | Value | -|-------|-------| -| **Issue Number** | #90 | -| **Title** | Add shareable permalinks to comments for consistent cross-referencing | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 19 | -| **Created** | 2026-07-18T14:03:35+02:00 | -| **Updated** | 2026-07-23T03:07:54+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/90 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Useful feature for referencing specific comments. Currently no stable URL format for individual comments, making it hard to link to specific points in discussions. - -**Recommended Action:** Generate unique comment IDs and create permalink routes like `/post/{id}/comment/{comment_id}`. - ---- - -#### ISSUE #89 — Render attachments and embeds on post front page listing - -| Field | Value | -|-------|-------| -| **Issue Number** | #89 | -| **Title** | Render attachments and embeds on post front page listing | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 45 | -| **Created** | 2026-07-18T14:03:34+02:00 | -| **Updated** | 2026-07-23T04:06:45+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/89 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** UX improvement for content discovery. Attachments and embeds only render on detail pages, so users must click into every post to see images/videos. Previewing them on the listing page would significantly improve feed engagement. - -**Recommended Action:** Include attachment/embed metadata in list endpoints. Render thumbnails/previews inline on post cards. - ---- - -#### ISSUE #87 — Fix notification dismissal marking DMs as read without user viewing them - -| Field | Value | -|-------|-------| -| **Issue Number** | #87 | -| **Title** | Fix notification dismissal marking DMs as read without user viewing them | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 8 | -| **Created** | 2026-07-18T14:01:33+02:00 | -| **Updated** | 2026-07-23T03:06:42+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/87 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Bug where dismissing a notification incorrectly marks the associated DM conversation as read. Users lose the unread indicator even though they haven't actually viewed the message content. - -**Recommended Action:** Decouple notification dismissal from message read status. Only mark DMs as read when the user actually opens the conversation. - ---- - -#### ISSUE #82 — Add new-line support to chat messages - -| Field | Value | -|-------|-------| -| **Issue Number** | #82 | -| **Title** | Add new-line support to chat messages | -| **Status** | open | -| **Author** | Wojtek322 | -| **Author UID** | 019eaca6-11c9-77b2-8c53-fc1ded7802a8 | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 9 | -| **Created** | 2026-07-14T13:00:20+02:00 | -| **Updated** | 2026-07-23T04:06:03+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/82 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Chat UX limitation. Users cannot send multiline messages in chat, forcing them to use separate messages for related content. Common expectation for modern chat platforms. - -**Recommended Action:** Allow Shift+Enter for newlines in chat input. Send on Enter (single newline). Store and render multiline content properly. - ---- - -#### ISSUE #73 — Configure automated backup schedule and address critical disk usage - -| Field | Value | -|-------|-------| -| **Issue Number** | #73 | -| **Title** | Configure automated backup schedule and address critical disk usage | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 38 | -| **Created** | 2026-07-13T00:37:13+02:00 | -| **Updated** | 2026-07-23T04:06:15+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/73 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Infrastructure concern. Critical disk usage needs addressing and automated backups should be configured. 38 comments suggest ongoing operational concern. - -**Recommended Action:** Set up cron-based database dumps and file backups. Implement log rotation and old file cleanup. Monitor disk usage with alerts. - ---- - -#### ISSUE #71 — Remove void-world system preamble from AI gateway that conflicts with production grading tasks - -| Field | Value | -|-------|-------| -| **Issue Number** | #71 | -| **Title** | Remove void-world system preamble from AI gateway that conflicts with production grading tasks | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 22 | -| **Created** | 2026-07-13T00:37:09+02:00 | -| **Updated** | 2026-07-23T04:03:06+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/71 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** AI gateway configuration issue. The "void-world" system preamble sent to the AI conflicts with production grading tasks, potentially causing incorrect responses. - -**Recommended Action:** Audit AI gateway system prompts. Remove or conditionally apply the void-world preamble based on task context. - ---- - -#### ISSUE #64 — Back navigation should hide modal if open - -| Field | Value | -|-------|-------| -| **Issue Number** | #64 | -| **Title** | Back navigation should hide modal if open | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 38 | -| **Created** | 2026-07-09T12:59:56+02:00 | -| **Updated** | 2026-07-23T04:27:41+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/64 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** UX bug. Pressing browser back button doesn't close open modals, leaving them overlaid on the previous page. Confusing and visually broken. - -**Recommended Action:** Listen for popstate events and close any open modals. Update URL hash/state when opening modals for proper back-navigation handling. - ---- - -### 3.4 LOW PRIORITY - -#### ISSUE #97 — Implement @note tag to create personal note overview similar to bookmarks - -| Field | Value | -|-------|-------| -| **Issue Number** | #97 | -| **Title** | Implement @note tag to create personal note overview similar to bookmarks | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 10 | -| **Created** | 2026-07-18T14:04:02+02:00 | -| **Updated** | 2026-07-23T01:40:27+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/97 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Feature request for personal notes functionality. Similar to bookmarks but more lightweight. Low urgency but could improve user engagement. - -**Recommended Action:** Plan as low-priority feature. Design @note tag syntax and personal notes listing page. - ---- - -#### ISSUE #95 — Add loading/in-progress animations to post action buttons - -| Field | Value | -|-------|-------| -| **Issue Number** | #95 | -| **Title** | Add loading/in-progress animations to post action buttons | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 54 | -| **Created** | 2026-07-18T14:03:57+02:00 | -| **Updated** | 2026-07-23T03:12:17+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/95 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** UX polish. Post action buttons (vote, reply, etc.) provide no visual feedback during async operations. Users don't know if their action was registered. 54 comments indicate strong desire for this polish. - -**Recommended Action:** Add spinner/loading state to all interactive buttons during API calls. Disable buttons while loading to prevent double-submission. - ---- - -#### ISSUE #66 — Add markdown rendering toggle for gists with markdown source type - -| Field | Value | -|-------|-------| -| **Issue Number** | #66 | -| **Title** | Add markdown rendering toggle for gists with markdown source type | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 9 | -| **Created** | 2026-07-12T10:13:56+02:00 | -| **Updated** | 2026-07-23T01:41:51+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/66 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Feature request for gist viewing. Users want to toggle between raw markdown and rendered HTML for markdown-type gists. Useful for comparing source vs output. - -**Recommended Action:** Add a toggle button on gist view pages for markdown source type gists. - ---- - -#### ISSUE #65 — Please jack up comment length - -| Field | Value | -|-------|-------| -| **Issue Number** | #65 | -| **Title** | Please jack up comment length | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 20 | -| **Created** | 2026-07-09T12:59:56+02:00 | -| **Updated** | 2026-07-23T04:27:41+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/65 | -| **Description** | Vague/joke title "Please jack up comment length" — not actionable as stated. However, the underlying request for longer comments may be valid. Note: Issue #98 (closed) already addressed increasing comment max length to match post max length. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Title is vague/joke-like. However, Issue #98 (closed) already increased comment max length to match post max length. This issue may be superseded. Verify if additional length increase is still needed beyond what #98 accomplished. - -**Recommended Action:** Cross-reference with closed Issue #98. If #98 already addresses this, close as duplicate. Otherwise, clarify requirements and reopen. - ---- - -#### ISSUE #48 — Make DevPlace logo clickable to navigate to splash screen or personal home page - -| Field | Value | -|-------|-------| -| **Issue Number** | #48 | -| **Title** | Make DevPlace logo clickable to navigate to splash screen or personal home page | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 13 | -| **Created** | 2026-06-21T17:50:58+02:00 | -| **Updated** | 2026-07-23T01:38:22+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/48 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Minor UX enhancement. Standard web convention is for logos to be clickable and navigate to the homepage/splash screen. Quick win for polish. - -**Recommended Action:** Wrap logo in anchor tag pointing to `/` or splash page. - ---- - -#### ISSUE #44 — Devii window shrinks when double-clicking outside of it - -| Field | Value | -|-------|-------| -| **Issue Number** | #44 | -| **Title** | Devii window shrinks when double-clicking outside of it | -| **Status** | open | -| **Author** | D-04got10-01 | -| **Author UID** | 912f0198-16fb-46de-91c3-5c582b359447 | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 30 | -| **Created** | 2026-06-17T22:34:18+02:00 | -| **Updated** | 2026-07-23T15:23:45+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/44 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Bug in the Devii AI assistant window. Double-clicking outside the window causes it to shrink unexpectedly. 30 comments indicate recurring user reports. - -**Recommended Action:** Prevent resize on double-click outside window bounds. Add event listener to block unintended resize triggers. - ---- - -#### ISSUE #10 — Add deep research history with ability to reopen and continue chats - -| Field | Value | -|-------|-------| -| **Issue Number** | #10 | -| **Title** | Add deep research history with ability to reopen and continue chats | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 8 | -| **Created** | 2026-06-14T04:29:30+02:00 | -| **Updated** | 2026-07-23T01:30:01+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/10 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** Feature request for deep research/chat history persistence. Users want to revisit and continue previous research sessions. Important for a research-focused AI tool. - -**Recommended Action:** Design chat history storage schema. Implement history listing and resume functionality. - ---- - -#### ISSUE #4 — Add paste-to-upload attachment support in comment textbox - -| Field | Value | -|-------|-------| -| **Issue Number** | #4 | -| **Title** | Add paste-to-upload attachment support in comment textbox | -| **Status** | open | -| **Author** | retoor | -| **Author UID** | 130de4ec-dbdf-4399-b157-299f6ec0049b | -| **Assignee** | None (unassigned) | -| **Labels** | None | -| **Comments Count** | 63 | -| **Created** | 2026-06-12T22:33:17+02:00 | -| **Updated** | 2026-07-23T02:18:35+02:00 | -| **URL** | http://retoor.molodetz.nl/retoor/devplacepy/issues/4 | -| **Description** | No body text provided. | -| **Key Comments** | None available in extracted data | - -**Priority Justification:** UX feature — users want to paste images/files directly into comment textboxes (Ctrl+V) instead of using a file picker. 63 comments indicate strong demand. Note: Issue #57 (by blindxfish) addressed image pasting specifically for issue descriptions, which is now fixed. This broader request covers all comment textboxes. - -**Recommended Action:** Implement paste event handler in comment textboxes to detect clipboard images and upload them. Show upload progress and thumbnail preview. - ---- - -## 4. ISSUES FILED BY BLINDXFISH (ACTIONABLE) - -### Summary Table - -| # | Title | Status | Comments | Assessment | -|---|-------|--------|----------|------------| -| #57 | Add ability to paste images into issue description | ALREADY FIXED | 27 | Implemented in IssueReporter.js. **CLOSE.** | -| #61 | Make card columns scroll independently of the feed | ALREADY FIXED | 10 | Fix applied in production. **CLOSE.** | -| #58 | Integrate MicroKingdoms plots as farms in DevPlace | DUPLICATE/NONSENSE | 9 | Concept doesn't exist in codebase. **CLOSE.** | -| #148 | Add risk, defense systems, and penalties to Code Farm raiding | IN PROGRESS | 12 | PR #149 opened. **TRACK TO MERGE.** | -| #135 | Add devlog timeline of related posts to project page | CLOSED | 17 | Already closed. | -| #60 | Add online status card for followed users on feed sidebar | CLOSED | 2 | Already closed. | -| #59 | Integrate Snek Chat natively into DevPlace | CLOSED | 1 | Already closed. | - -### Detailed Analysis - -#### #57 — Paste Images in Issue Descriptions (CLOSE AS FIXED) -blindxfish requested the ability to paste images directly into issue descriptions. This was a valid UX improvement — users expected the same paste-to-upload capability that exists in comment textboxes (Issue #4). The feature has been implemented in `IssueReporter.js`. - -**Action:** Close issue #57. - -#### #61 — Independent Card Column Scrolling (CLOSE AS FIXED) -blindxfish requested that card columns scroll independently of the main feed. This was a valid UX improvement for multi-column layouts where users wanted to browse different sections simultaneously. The fix has been applied in production. - -**Action:** Close issue #61. - -#### #58 — MicroKingdoms Farm Integration (CLOSE AS DUPLICATE/NONSENSE) -blindxfish requested integration of "MicroKingdoms plots" as farms in DevPlace and the farm game. Investigation reveals: -- The Code Farm already has farms, plots, and stealing mechanics -- "MicroKingdoms" is not a known concept in the codebase -- This appears to be based on a misunderstanding of existing features - -**Action:** Close issue #58 as duplicate/nonsensical. - -#### #148 — Code Farm Raiding Defenses (TRACK TO MERGE) -blindxfish proposed a comprehensive game design feature for Code Farm raiding: -- **Defenses for farm owners:** mines/traps, security cameras/alarms, guard dogs/patrols, insurance -- **Penalties for raiders:** fines, escalating consequences, cooldowns -- **Risk/reward mechanic:** Raiders can check target defense level before attempting; higher defense = bigger penalty if caught but bigger reward if successful - -typosaurus successfully implemented this feature and opened PR #149. - -**Action:** Review PR #149, merge if acceptable. This is a legitimate and well-thought-out game design addition. - ---- - -## 5. ISSUES AFFECTING BLINDXFISH - -### Issue #74 — AI Quota Exhaustion (OPERATIONAL — NOT CODE DEFECT) - -| Field | Value | -|-------|-------| -| **Issue Number** | #74 | -| **Title** | AI quota exhausted for m30n and blindxfish; quotas not reset, users blocked for days | -| **Status** | open | -| **Author** | retoor | -| **Affected Users** | m30n, **blindxfish** | -| **Created** | 2026-07-13T00:37:19+02:00 | -| **Updated** | 2026-07-23T02:57:23+02:00 | -| **Comments** | 8 | - -**Assessment:** blindxfish was a *victim* of this issue, not the reporter. Investigation confirmed that AI quotas DO reset automatically — the issue is operational/configurational rather than a code defect. The $1.00/24h spending cap may be too low for active users. - -**Recommended Action:** -- Consider raising the daily spending cap above $1.00 -- Add proactive alerting when users approach quota limits -- Communicate quota reset schedule to users - ---- - -## 6. EXCLUDED / NON-ACTIONABLE ISSUES - -The following 3 open issues were excluded from the actionable list: - -### Issue #174 — Unable to manually plant distributed systems, ml pipelines, security fortress in code farm -- **Author:** Wojtek322 -- **Reason for exclusion:** Absurd title with no real technical issue described. Likely a joke or troll post. 0 comments. No reproducible bug or feature request identifiable. -- **Recommendation:** Close as spam/troll. - -### Issue #109 — Request to create a meme about current software development status -- **Author:** retoor -- **Reason for exclusion:** Fun request, not a real bug or feature. Does not require engineering effort. -- **Recommendation:** Close as low-value/non-technical. - -### Issue #65 — Please jack up comment length -- **Author:** retoor -- **Reason for exclusion:** Vague/joke title "Please jack up comment length" — not actionable as stated. Superseded by closed Issue #98 which already increased comment max length to match post max length. -- **Recommendation:** Close as duplicate of #98. - ---- - -## 7. RECOMMENDED ACTIONS & PRIORITIZATION MATRIX - -### Phase 1: Immediate (This Week) - -| Priority | Issue | Action | Owner | -|----------|-------|--------|-------| -| **Critical** | #83 | Fix session cookie not set after login | Engineering | -| **High** | #70 | Restrict OpenAI gateway to internal network | Security/Infra | -| **High** | #102 | Fix XML-RPC bridge crash loop | Engineering | -| **High** | #93 | Fix Content-Length headers | Engineering | -| **High** | #75 | Block Google Read Aloud bot | Infra | -| **Close** | #57 | Close as already fixed | Admin | -| **Close** | #61 | Close as already fixed | Admin | -| **Close** | #58 | Close as duplicate/nonsense | Admin | - -### Phase 2: Short Term (Next 2 Weeks) - -| Priority | Issue | Action | Owner | -|----------|-------|--------|-------| -| **High** | #84 | Fix AI markdownifyer corruption | Engineering | -| **High** | #85 | Fix duplicate chat message deduplication | Engineering | -| **High** | #86 | Investigate chat rendering bug | Engineering | -| **High** | #94 | Fix intermittent vote button failures | Engineering | -| **High** | #72 | Fix empty gateway_embed_key fallback | Engineering | -| **High** | #155 | Implement push unsubscribe endpoint | Engineering | -| **High** | #132 | Verify admin notification defaults, write tests | Engineering | -| **Merge** | #148 | Review and merge PR #149 (Code Farm raiding) | Admin | -| **Merge** | #162 | Review and merge PR #168 (non-color vote indicator) | Admin | -| **Merge** | #152 | Review and merge PR #153 (PR link punctuation) | Admin | - -### Phase 3: Medium Term (Next Month) - -| Priority | Issue | Action | Owner | -|----------|-------|--------|-------| -| **Medium** | #167 | Suppress notifications for active DMs | Engineering | -| **Medium** | #161 | Add avatar_seed to search results | Engineering | -| **Medium** | #154 | Fix Load More pagination + URL encoding | Engineering | -| **Medium** | #114 | Add vote value 0 for retraction | Engineering | -| **Medium** | #111 | Add optionality info to API spec | Engineering | -| **Medium** | #110 | Fix DM chat scrolling | Engineering | -| **Medium** | #92 | Fix tagging delay for non-existent users | Engineering | -| **Medium** | #90 | Add comment permalinks | Engineering | -| **Medium** | #89 | Render attachments on post listing | Engineering | -| **Medium** | #87 | Fix notification dismissal marking DMs read | Engineering | -| **Medium** | #82 | Add newline support to chat | Engineering | -| **Medium** | #73 | Configure automated backups | Infra | -| **Medium** | #71 | Remove void-world preamble | Engineering | -| **Medium** | #64 | Fix back navigation hiding modals | Engineering | -| **Medium** | #44 | Fix Devii window shrink bug | Engineering | -| **Medium** | #10 | Add deep research history | Engineering | -| **Medium** | #4 | Add paste-to-upload in comments | Engineering | - -### Phase 4: Low Priority / Polish - -| Priority | Issue | Action | Owner | -|----------|-------|--------|-------| -| **Low** | #97 | Implement @note tag | Engineering | -| **Low** | #95 | Add loading animations to buttons | Engineering | -| **Low** | #66 | Add markdown toggle for gists | Engineering | -| **Low** | #48 | Make logo clickable | Engineering | -| **Low** | #10 | Add deep research history | Engineering | -| **Low** | #4 | Add paste-to-upload in comments | Engineering | - -### Operational Items - -| Item | Action | -|------|--------| -| Issue #74 | Raise $1.00/24h AI quota cap; add proactive alerting | -| Issue #65 | Close as duplicate of #98 | -| Issue #174 | Close as spam/troll | -| Issue #109 | Close as low-value/non-technical | - ---- - -## 8. APPENDIX: CLOSED ISSUES REFERENCE - -For completeness, here are notable closed issues relevant to the investigation: - -### Closed Issues by blindxfish -| # | Title | Status | Comments | Notes | -|---|-------|--------|----------|-------| -| #135 | Add devlog timeline of related posts to project page | closed | 17 | Likely implemented | -| #60 | Add online status card for followed users on feed sidebar | closed | 2 | Likely implemented | -| #59 | Integrate Snek Chat natively into DevPlace | closed | 1 | Likely implemented | - -### Notable Closed Issues (Security/Stability) -| # | Title | Author | Comments | Notes | -|---|-------|--------|----------|-------| -| #107 | Investigate repeated failed login attempts on Lensflare account | retoor | 0 | Security incident investigated | -| #105 | Add rate limiting to login endpoint after credential stuffing attack | retoor | 7 | Attack from 88.198.21.243 | -| #106 | Add URL format validation to SEO diagnostics job queue | retoor | 42 | SEO pipeline fix | -| #104 | DeepSearch fails with Playwright async context manager error | retoor | 43 | Playwright bug fix | -| #68 | Zip service: 3/3 jobs failed (100% failure rate) | retoor | 58 | Project downloads broken | -| #69 | SEO Diagnostics: 55% job failure rate | retoor | 29 | SEO pipeline reliability | -| #76 | Remove 10 stale stopped containers wasting disk space | retoor | 5 | Disk cleanup | -| #77 | Add pruning policy for devii_lessons table | retoor | 0 | DB maintenance | - -### Notable Closed Issues (Features) -| # | Title | Author | Comments | Notes | -|---|-------|--------|----------|-------| -| #150 | Show linked project on post details page | Lensflare | 21 | Feature implemented | -| #146 | Add missing icon field to badges API response | Lensflare | 15 | API completeness | -| #157 | Add badge earned info to user profile badges API | Lensflare | 16 | API completeness | -| #156 | Enable APNs push provider configuration | Lensflare | 2 | Push setup | -| #169 | Implement links to official iOS app in web app | bytebitch | 4 | App promotion | -| #138 | Fix navigation bar link icons and text layout | retoor | 22 | CSS fix | -| #115 | Feed REST returns my_vote as 0 for all posts | Lensflare | 41 | Bug fix | -| #113 | Fix badge names returning null in profile | Lensflare | 38 | Bug fix | -| #112 | Expose level progress percentage in profile API | Lensflare | 31 | API completeness | - ---- - -## DATA SOURCES & METHODOLOGY - -### Source Files Used -1. **`/tmp/issues_open.txt`** — Raw export of all 44 open issues from the DevPlace issue tracker API (https://devplace.net/issues) -2. **`/home/dev/projects/r/todo.md`** — Previous investigation report with validated issue assessments and blindxfish critique summary -3. **`/home/dev/projects/r/devplace_issues_complete.md`** — Complete issue tracker export (133 total issues: 44 open, 89 closed) with detailed descriptions and comment summaries - -### Filtering Methodology -- **Included:** All open issues with identifiable technical substance (bugs, feature requests, security concerns, infrastructure needs) -- **Excluded:** - - #174 — Spam/troll (absurd title, no real issue) - - #109 — Non-technical (meme creation request) - - #65 — Vague/joke title, superseded by closed #98 - -### Limitations -- Most issue bodies contain "No body text provided" — detailed descriptions come from comment threads in the complete export -- Comment content was summarized rather than fully transcribed due to volume (some issues have 100+ comments) -- blindxfish_external_critique file (`/tmp/blindxfish_critical.txt`) did not exist; data sourced from todo.md and devplace_issues_complete.md - ---- - -**Document compiled:** 2026-09-10 13:01:58 UTC -**Total actionable issues:** 41 -**blindxfish-related items:** 8 (5 open issues filed, 1 issue affecting, 2 already-fixed needing closure) diff --git a/fpost.md b/fpost.md deleted file mode 100644 index aa93e37c..00000000 --- a/fpost.md +++ /dev/null @@ -1,70 +0,0 @@ -# DevPlace Changelog — Last 24 Hours - -## AI Gateway — Non-OpenAI Upstreams & Client Model Passthrough - -The AI gateway now supports routing to non-OpenAI upstream providers. The gateway can target any compatible API endpoint, and client-specified model names are passed through to the upstream instead of being forced to a fixed model. This opens the door for self-hosted providers and alternative APIs. - -**Files:** `services/openai_gateway/gateway.py`, `service.py`, `thinking.py`, `vision.py` - -## AI Gateway — Trust Only Upstream X-Gateway-Model Header - -The gateway now trusts only the `X-Gateway-Model` header from the upstream response when determining which model served a request. This prevents spoofing and ensures usage attribution is accurate. - -**Files:** `services/openai_gateway/gateway.py`, `usage.py` - -## Backup Service — Remote Offload to Hetzner Storage Box - -Completed backups can now be automatically offloaded to a remote Hetzner Storage Box. The offload module handles the transfer, and the backup service orchestrates the full lifecycle: create, verify, offload, and prune. - -**Files:** `services/backup/offload.py`, `service.py`, `store.py` - -## Container Manager — Orphan Blob Fix & System Prune - -Fixed container sync races that leaked orphan blobs (files on disk with no database reference). Added a new `devplace system prune` CLI command that safely removes orphan blobs, sweeps unreferenced container workspace dirs, and prunes expired fork/zip job rows. The command supports `--dry-run` for preview. - -**Files:** `cli/system.py`, `services/containers/store.py`, `attachments.py`, `project_files.py` - -## Container Manager — Two-Phase Plan/Execute Report - -The `system prune` CLI command now uses a two-phase plan/execute report: first it shows what would be deleted (plan), then executes and reports what was actually removed. - -**Files:** `cli/system.py`, `cli/containers.py` - -## Post Page — Happy 404, Featured/Related Sidebars, Next-Post Nav - -The post page now includes: -- A "Happy 404" page for missing posts -- Featured posts sidebar -- Related posts sidebar -- Next-post navigation - -**Files:** `happy404.py`, `routers/posts.py`, `templates/post.html`, `database/content.py` - -## Static Asset Cache-Busting — Unified with Auto-Bumped App Version - -Static asset URLs now use the auto-bumped app version for cache-busting instead of a separate timestamp. This ensures assets are invalidated whenever the app version changes, and eliminates the need for manual cache-bust updates. - -**Files:** `config.py`, `templates/docs/static-caching.html`, `nginx/nginx.conf.template` - -## Automatic Patch-Version Bumping via Git Pre-Commit Hook - -A new `.githooks/pre-commit` hook automatically bumps the patch version in `pyproject.toml` on every commit. The hook increments patch (e.g. `1.0.0` → `1.0.1`) and stages the change. A hand-set major/minor bump in the same commit is left untouched. The hook defers during merges and never blocks a commit. - -**Files:** `.githooks/pre-commit`, `Makefile`, `CLAUDE.md` - -## Test Infrastructure — Shared `run_async` Helper - -Provision tests now use the shared `run_async` test helper instead of bare `asyncio.run` calls, improving consistency and error handling across the test suite. - -**Files:** `tests/unit/services/containers/workspace/provision.py` - -## Documentation Updates - -- Updated `README.md` with new feature documentation -- Updated `CLAUDE.md` with new CLI commands and architecture notes -- Updated docs pages: production nginx, static caching, backups, architecture backend -- Updated nested `CLAUDE.md` files for gateway, containers, backup, and posts subsystems - ---- - -*Generated from git log — all commits from the last 24 hours.* diff --git a/gaplord.md b/gaplord.md deleted file mode 100644 index 49f52db3..00000000 --- a/gaplord.md +++ /dev/null @@ -1,405 +0,0 @@ -# gaplord: close every documented gap - -Plan and result in one file. The plan was written first, from a read of the code behind each gap. Every item is then worked in order and its **Status** and **Result** lines are filled in when it is done, with the measurement or test that proves it. Nothing is checked off without evidence. - -Written 2026-10-16. Scope: every entry under a `Gaps` heading in `devplacepy/services/{nodes,sync,community,skills}/CLAUDE.md` and `nodes/extension/README.md`, except the plain-text community passwords (explicitly excluded by the owner). - -Legend: `[ ]` open, `[~]` in progress, `[x]` done and proven, `[!]` cannot be closed by code (reason stated, nothing faked). - -## Ground rules for this work - -- No fiction. A gap is only closed when code exists, a test or measurement proves it, and the nested `CLAUDE.md` says what the system does today. Where a gap needs something this machine does not have, it is marked `[!]` with the exact missing input and what was built so the measurement happens by itself later. -- Specification first. Every new protocol field, limit, refusal, reason and kind is declared in `spec.py` (nodes) or `services/sync/spec.py` and read from there everywhere else. -- Caps on everything new (rows, bytes, intervals, retries). Machines hold, humans decide: nothing here bans a member or deletes a ledger row on its own. -- One fan-out per feature: data layer, schema (`*Out`), route, view, Devii tool, `docs_api`, docs prose, privacy page where data categories change, README, nested `CLAUDE.md`, tests in the matching tier. -- Em dash is forbidden in every touched file. No comments or docstrings in Python source. -- The dev server on port 10500 runs `--reload` against the live database. Files are therefore written whole and import-checked right after each edit, and no command in this work reads or writes `data/`. All scripts use scratch databases (`DEVPLACE_DATABASE_URL`, `DEVPLACE_DATA_DIR`). -- Final gate: `make test` (unit, api, e2e), plus `ruff`/`pyflakes` on touched files, `go vet` + `go test`, `node --test`. - -## Gap inventory (source of truth for this plan) - -| Id | Gap (as written in the docs) | Source | -|----|------------------------------|--------| -| G-N1 | Reddit viability unmeasured from a residential line | nodes | -| G-N2 | `country`, `asn`, `asn_class` never filled, datacenter exclusion (D6) not enforced | nodes | -| G-N3a | `last_seen_at` written on connect only | nodes | -| G-N3b | Socket counters per process | nodes | -| G-N3c | Fabric cache per process | nodes | -| G-N3d | Circuit breaker state in memory (and no per-destination admin view) | nodes | -| G-N3e | Same-process fast path not built, callers poll | nodes | -| G-N4 | Fetch-mode results not cross-checked (sampled redundancy) | nodes | -| G-N5 | DeepSearch skips hostile domains with a static list, no routing through `/nodes/internal/fetch` | nodes | -| G-N6a | Go client has no battery-aware pause (`pause_on_battery` ignored) | nodes | -| G-N6b | Go client treats a timed suspension as permanent until restarted | nodes | -| G-N6c | Extension enforces its own local hours in the browser clock, not the server timezone | nodes | -| G-N7 | Not measured: Reddit/X success per address class, WebSocket idle timeouts through the public hostnames, socket capacity of one worker, 24 hour soak | nodes | -| G-N8 | Launch blocked on a recorded legal decision | nodes | -| G-E1 | Extension has no icons | extension | -| G-E2 | `welcome` frame carries no device label | extension | -| G-E3 | Extension not tested in a real Chrome or Firefox session | extension | -| G-S1 | No export endpoint, sync is inbound only | sync | -| G-S2 | Votes, reactions, follows, attachments, polls and project links not synced | sync | -| G-S3 | A comment's post and parent must arrive earlier, no deferred resolution queue | sync | -| G-S4 | A local deletion is not communicated to the origin; the origin is rejected forever with `purged_locally` | sync | -| G-S5 | `min_comments` only sees comments collected in the same run | sync | -| G-S6 | A clone renamed by an administrator no longer matches its alias | sync | -| G-S7 | A username taken between alias generation and clone creation diverges from the rewritten mentions | sync | -| G-K1 | Community page does not live-refresh during a run | community | -| G-K2 | Run log and ledger are written only when the run finishes; a killed run leaves neither | community | -| G-K3 | Text search across a whole community scans up to 40000 rows | community | -| G-K4 | HTML comments follow `rel=next` links only | community | -| G-K5 | No two-factor, captcha or JSON-API login | community | -| G-K6 | HTML learning is heuristic plus AI; a hostile markup needs a hand-edited `SCHEMA` | community | -| G-K7 | Votes, reactions and attachments are not collected | community | -| G-K8 | Cross-feed duplicate detection compares sampled addresses at discovery time only | community | -| G-SK1 | No test installs a skill from a live network origin | skills | -| G-X | Excluded by the owner: community passwords stored in plain text | community | - -## Honest classification before any code - -Closable by code and proven on this machine: G-N2, N3a-e, N4, N5, N6a-c, E1, E2, S1-S7, K1-K8 (K5 partly, see its item), SK1, and the capacity part of G-N7. - -Closable only partly, with the remainder stated: - -- **G-N1 / G-N7 (success per address class).** A residential donor is a physical machine this session does not have. What is built: every job records the donor's address class, the admin page and `GET /admin/nodes` aggregate success per destination and class, and the docs print `n = 0` until real donors exist. The viability number itself stays unmeasured and the docs say so. -- **G-N7 (24 hour soak).** A soak can only be as long as this session. It is run in the background for as long as the work lasts and reported with its true duration. -- **G-N7 (idle timeout through the public hostname).** Measuring it requires an authenticated WebSocket on `devplace.net`, which is a write to the production database. It is measured against the local stack and the exact proxy settings are read from `nginx/nginx.conf.template`; the public leg is listed as needing the owner's explicit go-ahead. -- **G-E3 (Firefox).** Chromium is driven for real through Playwright. Firefox is attempted; if the available build cannot load a temporary add-on, the item stays open for Firefox only. -- **G-K5 captcha.** A captcha exists to stop automation. It is detected and reported loudly (`captcha_required`), never solved. - -Not closable by code: **G-N8** (a legal decision is a human act). It stays `[!]`; the plan records exactly what the decision must cover. - -## Protocol and specification changes (done first, shared by server and both clients) - -All additive and optional, `PROTOCOL_VERSION` stays 1 (an old client ignores unknown fields on server frames; the server rejects unknown fields from clients, so only server-to-client additions are made). - -| Change | Where | Why | -|--------|-------|-----| -| `welcome.device_label` (str, optional) | `spec.py` FRAMES, `session.handshake`, Go `Frame`, extension `WsClient` | G-E2 | -| `kick.until` (str, optional ISO instant) | `spec.py`, `registry.kick`, clients | G-N6b | -| Ticket `403` body carries `until` for a timed suspension | `routers/nodes/ticket.py` | G-N6b | -| New refusal `on_battery` ("donor is running on battery") | `spec.py` REFUSALS, `dispatcher.DONOR_LIMIT_CODES` (pause 300 s), Go `RefuseBattery`, extension `REFUSALS` | G-N6a | -| `LimitSpec.last_seen_write_seconds` (60), `redundancy_percent` (5), `redundancy_max_percent` (20), `geo_lookup_cache` (4096) | `spec.py` | G-N3a, N4, N2 | - -The extension spec test and the Go policy tests are updated in the same change so drift fails loudly. - ---- - -## Track N: node network, server (`devplacepy/services/nodes/`, `database/nodes.py`, `routers/nodes/`, `routers/admin/nodes.py`) - -### N-01 Geo and ASN database (G-N2) - -- Root cause: `record_device_connection` is called with defaults, so `country`, `asn`, `asn_class` stay empty (`routers/nodes/ws.py`). The doc says the fix is an offline ASN database. -- Source: DB-IP "IP to ASN Lite" and "IP to Country Lite" monthly CSVs, free, no key, CC BY 4.0 (attribution printed on the docs page and in `devplace nodes geo-status`). Verified on 2026-10-16: `https://download.db-ip.com/free/dbip-asn-lite-2026-10.csv.gz` answers 200, 7.0 MB, 477362 ranges. -- Build `services/nodes/geo.py`: `GeoDatabase` backed by one SQLite file at `config.NODE_GEO_DB` (declared in `DATA_PATHS`, created by `ensure_data_dirs`). Tables `geo_asn` and `geo_country`: `(family INTEGER, first BLOB, last BLOB, ...)`, primary key `(family, first)`. Addresses are packed with `ipaddress` (4 or 16 bytes, compared bytewise by SQLite). Lookup is `WHERE family=? AND first<=? ORDER BY first DESC LIMIT 1` then `last>=ip`. Memory stays flat, no 477k-entry Python list. -- Build `services/nodes/addressclass.py`: `classify(asn, org) -> residential | mobile | datacenter | unknown`. A static frozenset of well-known hosting and cloud ASNs (AWS, GCP, Azure, Hetzner, OVH, DigitalOcean, Linode, Vultr, Oracle, Alibaba, Cloudflare, Contabo, Scaleway, Leaseweb, M247, Choopa, Tencent, IBM, Rackspace) plus word-boundary org keywords (`hosting`, `cloud`, `datacenter`, `data center`, `vps`, `server`, `colo`, `dedicated`, `cdn`, `vpn`, `proxy`); mobile by keywords (`mobile`, `wireless`, `cellular`, `lte`, `gsm`); no data means `unknown`; everything else is `residential`. Heuristic by nature, so the docs call it that and the admin override (N-02) exists. -- Loader `geo_sync()` downloads through `stealth_async_client` (the project rule), verifies gzip, parses row by row with a hard cap on rows, loads into a temporary file and swaps it in atomically (verify before write). `--dry-run` reports counts only. CLI: `devplace nodes geo-sync [--dry-run] [--month YYYY-MM]`, `devplace nodes geo-status`. -- Connect path: `ws.py` resolves the address transiently (the address is never stored, only `hash_ip`), calls `GeoDatabase.lookup`, passes `country`, `asn`, `asn_class` to `record_device_connection`, and puts `asn_class` on `session`. A missing or unreadable database yields `unknown` plus the counter `geo_unavailable` and one log line per process. -- Tests: unit for packing/lookup edge cases (first/last address of a range, gap between ranges, IPv6, v4-mapped), classifier table (including keyword false-positive guards such as `Hostingly`), loader against a local fixture file, atomic swap; api for `devplace nodes geo-sync --dry-run` is CLI-unit. Static check that no raw address column exists on `node_devices`. -- Docs: nodes CLAUDE.md, `templates/docs/nodes.html` (donor-facing: what is looked up and that the address is not stored), `privacy.html` (the lookup), attribution line. -- Status: [~] Result: geo.py + addressclass.py + CLI geo-sync/geo-status; real DB-IP month built in 10 s (477362 ASN, 710834 country ranges, 48 MB), lookup 0.17 ms, installed through the real CLI against a scratch data dir; 30 unit tests + 2 api tests (header-driven class, missing database counter). Docs pending. - -### N-02 Enforce the datacenter exclusion, with an admin override (G-N2, D6) - -- Scheduler eligibility gains one rule: `session.asn_class in {residential, mobile}` or the device carries `class_override = "allow"`. `unknown` is refused by default, because "residential and mobile only" is the decision and an unverifiable address must not be assumed good (loud skip, counter `jobs_no_eligible_class`). Site setting `nodes_allow_unknown_class` (default `0`, apply live) lets an administrator run without a geo database during bring-up. -- New column `node_devices.class_override` (ensured in `NODES_COLUMNS`). Admin route `POST /admin/nodes/devices/{uid}/class` (`allow` or `clear`), `require_admin`, `is_senior_admin` against the device owner, audit key `node.device.override`, catalogued in `audit/events.md` and `category_for`. Devii `node_admin_class_override` (admin only, in `CONFIRM_REQUIRED`, declares `confirm`). -- Admin page: class column and an override button in the recent-devices table; `AdminNodesOut` carries both fields. -- Tests: unit scheduler (each class, override, unknown with and without the setting), api admin route (authz, senior-admin denial, audit row), Devii action gating. -- Status: [~] Result: scheduler rule (residential/mobile, unknown only with nodes_allow_unknown_class, per-device override), POST /admin/nodes/devices/{uid}/class, Devii node_admin_class_override (confirm-gated), audit node.device.override, admin page button; 5 scheduler + 6 api tests. Docs pending. - -### N-03 `last_seen_at` while connected (G-N3a) - -- One batched `UPDATE node_devices SET last_seen_at=:now WHERE uid IN (...)` per `last_seen_write_seconds` from `NodesService.reconcile`, one statement for all sessions (no per-session query, the measured lesson in the doc), plus a final write in the disconnect path. -- Tests: unit with fake sessions (statement count is 1 for N sessions, interval respected), api that a connected fake device advances `last_seen_at` while idle. -- Status: [~] Result: NodesService.persist_state touches all connected devices in one batched UPDATE per last_seen_write_seconds; unit tests (interval, force, failure). - -### N-04 Durable counters (G-N3b) - -- Table `node_counters(name PRIMARY KEY, value)` (derived, hard-GC only, not soft-deletable). `NodeCounters` keeps a delta per name; `reconcile` flushes all deltas in one `INSERT ... ON CONFLICT(name) DO UPDATE SET value = value + excluded.value` transaction and resets them only after commit. `snapshot()` stays the live process view; `lifetime()` reads the table. Admin page shows both. -- Tests: unit (flush adds, a failed flush keeps the delta, two processes flushing concurrently sum exactly: real separate processes, as the project procedure requires), api admin payload. -- Status: [~] Result: node_counters table, NodeCounters pending deltas flushed in one transaction every counter_flush_seconds, failed flush restores deltas; unit + api tests. Multi-process flush race still to run in validation. - -### N-05 Persisted circuit snapshots and a per-destination admin view (G-N3d) - -- `CircuitBreaker.snapshot()` returns `(key, failures, state, opened_at_wall)`. Table `node_circuits` is rewritten (upsert, delete when closed) from `reconcile` only when the breaker marked itself dirty. `NodesService` loads open circuits on start, converting wall time back to the monotonic clock, so a restart does not forget an open destination circuit. -- Admin page gets a "Circuits" table (destination or node, state, failures, seconds until half-open); `AdminNodesOut.circuits` capped at 100 rows. -- Tests: unit round trip across a simulated restart, expiry on load, cap, admin payload. -- Status: [~] Result: CircuitBreaker snapshot/restore + node_circuits + admin Circuit breakers table (read from the table so any worker answers); round-trip restart test. - -### N-06 Cross-process fabric cache (G-N3c) - -- Table `node_cache(key PRIMARY KEY, method, url, status, headers, body BLOB, stored_at, expires_at, size)` as an L2 under the existing in-process `ResultCache` (L1). Reads: L1, then L2 (then promote). Writes: both. Bounded by `LimitSpec.cache_max_bytes` and `cache_entry_max_bytes` and the TTL; `NodesService` prunes expired rows and, when over the byte cap, the oldest rows, in `run_once`. `cache.clear()` clears both. Responses with `status >= 500` or truncated are never stored (existing rule kept). -- Tests: unit (L1 miss L2 hit promotes, TTL, cap eviction order, oversize skipped), a real two-process test (worker A stores, worker B hits), kill-switch off does not serve from cache? (decision: cache is served regardless, as today, and the test pins that). -- Status: [~] Result: SharedResultCache (L1 + node_cache table), prune in service, cross-instance hit test; real two-process proof still to run in validation. - -### N-07 Same-process fast path (G-N3e) - -- `services/nodes/wake.py`: `WakeHub` holding an `asyncio.Event` for the dispatcher and one per awaited job. `JobBroker.enqueue` signals the dispatcher when it runs in the lock-owner process; `dispatcher._account`/`_failed`/`broker.complete` resolve the job event; `JobBroker.wait` awaits the event first and keeps the DB poll only as the cross-process fallback and a safety net. No behavior change in other workers. -- Measure before and after on this machine with a fake session: enqueue-to-assigned latency (today up to `dispatcher_tick_ms` = 200 ms) and result-to-caller latency (today up to `poll_interval_ms` = 100 ms). The numbers go into the doc. -- Tests: unit (event wake, no lost wake-up when set before wait, timeout path unchanged), existing broker tests stay green. -- Status: [~] Result: WakeHub + JobBroker events; unit tests prove immediate wake. Before/after latency numbers still to measure. - -### N-08 Sampled redundancy for fetch-mode results (G-N4) - -- Spec: `redundancy_percent` (default 5, ceiling `redundancy_max_percent` 20, site setting `nodes_redundancy_percent` clamped to it, 0 turns it off). -- After a `fetch`-transport result is accepted, a deterministic sample (`sha256(job uid) % 100 < percent`) enqueues a shadow job (`caller_kind = "shadow"`, `shadow_of = `, `exclude_device = `), scheduled to a different donor owner. The shadow never spends the caller's quota and never enters the cache. -- Comparison is on a **shape**, not bytes (live pages change): status class, JSON top-level key set or the HTML ``/first heading, plus a body size ratio inside a band. A mismatch raises a `node_flags` row `redundancy_mismatch` against both devices and counts `redundancy_mismatches`; no auto-ban. Match and mismatch counts are persisted (N-04) and shown on the admin page so the false-positive rate is itself measured. -- Tests: unit shape extractor (JSON, HTML, binary), sampling determinism and rate over 10000 uids, scheduler exclusion, dispatcher comparison with two fake nodes (match, forged body, one node errors), cap. -- Status: [~] Result: shape.py + dispatcher shadow jobs (scheduler exclusion, flags, counters, clamped setting); 9 shape + 8 dispatcher tests. - -### N-09 DeepSearch routes node-servable hostile URLs through the fabric (G-N5) - -- `services/jobs/deepsearch/crawl.py`: `_resolve_candidate` asks a new `NodeCrawl` helper first when the URL is allowlisted (`SPEC.destination(host)`) and `fabric.can_serve(url)` is true; the fetch uses `shared_fabric().fetch(..., caller_kind="user", caller_id=<session owner>)`, so it spends the owner's quota, honours the kill switch and falls back to the snippet on any `FabricError`. The static `HOSTILE_DOMAINS` tuple stays only for hosts the allowlist cannot serve (X, YouTube, Instagram and so on); Reddit is removed from it and the owner uid is threaded from the DeepSearch worker through `crawl()`. -- Same extraction path as `fetch_page` (shared function), so the text is parsed identically. Guests are not routed (no owner quota to charge). -- Tests: unit with a fake fabric (served, refused, unavailable, quota exceeded, guest), existing crawl tests stay green. -- Status: [~] Result: crawl.node_page via POST /nodes/internal/fetch with the owner key for allowlisted hostile hosts; 6 unit tests. - -### N-10 Success per address class, measured automatically (G-N1, G-N7) - -- `node_requests` records `device_asn_class` at claim time. `database.nodes.success_by_class(days)` aggregates `(adapter, asn_class, outcome)` over the retained window. `GET /admin/nodes` (`AdminNodesOut.success`) and the admin page show it; Devii `node_admin_status` includes it. -- Docs print the live table. Until a residential donor exists the cell reads `n = 0`, and the Reddit-from-residential gap is documented as unmeasured with the exact query that will answer it. -- Tests: unit aggregation (including empty), api payload. -- Status: [~] Result: claim records device_asn_class; durable outcome counters (outcome:adapter:class:kind) + admin table with n = 0 state; unit + api tests. - -### N-11 Measurements (G-N7) - -- **Socket capacity of one worker:** scratch server on a free port, up to `max_connected_nodes` (2000) synthetic devices and tickets created in a scratch database, asyncio WebSocket clients answering pings; record RSS, event-loop lag (a 10 ms ticker), reconcile tick duration and connect rate at 250, 500, 1000, 2000 sockets. Harness lives in the scratchpad (disposable), method and numbers go into the doc. -- **Soak:** same harness held open with jobs flowing through fake nodes; started in the background as early as possible and sampled every minute (RSS, loop lag, socket count, error counters). Reported with its real duration. Not claimed as 24 hours unless it ran 24 hours. -- **Idle timeout:** local stack measured (idle WebSocket for 5 minutes with no application frames, ping interval 20 s, observing whether anything closes it); `nginx/nginx.conf.template` read for `proxy_read_timeout` on the nodes location and quoted. The public hostname leg needs a production write and is left for the owner's go-ahead. -- Status: [~] Result: measured. Fast path p50 209 ms to 25 ms; capacity 2000 sockets in 341 MB; soak 81 minutes (NOT 24 h) clean; counter and cache races exact. Residential Reddit rate and public-hostname idle remain unmeasured (see Cannot be closed by code). - ---- - -## Track C: clients (`nodes/client/` Go, `nodes/extension/`) - -### C-01 Go: battery-aware pause (G-N6a) - -- `ServerSettings` gains `PauseOnBattery`. New files with build tags: `battery_linux.go` (parse `/sys/class/power_supply/*`: on battery when a `Battery` supply exists, no `Mains`/`USB` supply is online, and the battery status is `Discharging`), `battery_darwin.go` (`pmset -g batt`), `battery_windows.go` (`GetSystemPowerStatus` through `syscall.NewLazyDLL`, no cgo), `battery_other.go` (unknown). `Limiter.Acquire` refuses with `on_battery` when the server setting is on and the probe says battery; the probe result is cached for 15 seconds. The server pauses the donor for 300 s on that code (N-spec change). Unknown means "not on battery" and logs once at debug (a probe failure must not silence a donor forever). -- Tests: table tests for the Linux parser with fixture trees, limiter test with a fake probe, the refusal code against `spec.py`. -- Status: [x] Result: Go battery probes (linux/darwin/windows/other), limiter refuses on_battery; go test green (agent report, re-verified: go vet clean, go test ok). - -### C-02 Go and extension: a timed suspension is not permanent (G-N6b) - -- Server: ticket `403 suspended` and the `kick` frame carry `until` for a timed suspension (read from `suspended_until`). A suspension without an end date stays a plain hold. -- Go: `suspended` leaves `FatalReasons`. New `outcome.holdUntil`; `Session.Run` sleeps until `until` (clamped to at least 60 s and at most 24 h, plus jitter) and then retries the ticket. `revoked` and the other reasons stay fatal. Extension `WsClient` gets the same state (`SUSPENDED` with a countdown in the popup). -- Tests: Go session test with a fake clock (suspended with `until`, without, expired), extension unit test, api test that the ticket body carries `until`. -- Status: [x] Result: Go holdUntil + extension SUSPENDED state; server sends until in ticket 403 and kick; tests in Go, node and api tiers. - -### C-03 Extension: server-timezone hours (G-N6c) - -- `Limits.applyServerSettings` also reads `active_from`, `active_to`, `timezone`; `inActiveHours()` requires both the donor's local window (browser clock, as configured in the popup) and the server window evaluated in the server-stored timezone with `Intl.DateTimeFormat(..., {timeZone})`. An invalid zone falls back to UTC, as `scheduler._zone` does. -- Tests: node tests with a stubbed clock across zones, windows that wrap midnight, an invalid zone. Parity test against the Go `InWindow` and the Python `in_active_hours` for the same 200 sampled instants. -- Status: [x] Result: extension server-timezone hours with Python-generated parity table shared by JS and Go tests; node --test 68/68 re-verified. - -### C-04 Extension icons (G-E1) - -- Four PNGs (16, 32, 48, 128) generated reproducibly by a small script using Pillow (already a dependency), a node-network glyph on the project accent colour. `manifest.json` gets `icons` and `action.default_icon`; `build.sh` includes them; the packaging test checks the files exist and the sizes match. -- Status: [x] Result: four PNG icons generated by nodes/extension/tools/make_icons.py, manifest + build.sh updated, package test. - -### C-05 Device label in `welcome` (G-E2) - -- Server sends `device_label` (from `node_devices.label`); extension `State` stores it and the popup shows the server label as the authority, falling back to the local one before the first connect. Go prints it in `status`/`run` logs. -- Tests: frame spec test, extension popup state test, Go decode test. -- Status: [x] Result: welcome.device_label server + Go + extension; api test asserts it. - -### C-06 Real browser test of the extension (G-E3) - -- Playwright persistent Chromium context with `--disable-extensions-except` and `--load-extension` against a scratch DevPlace server on a free port (not 10500/10501): register a device, drive the disclosure and connect flow through the real popup page, assert the socket connects (server registry shows the node), a job for an allowlisted host is answered (`via: fetch`; a 403 from this datacenter address is an acceptable status, as in the Go run), refusals work, and disabling disconnects. Firefox: attempt with the installed Playwright Firefox; if a temporary add-on cannot be loaded, record that and leave E3 open for Firefox. -- The test is kept as `tests/e2e/nodes/extension.py` (skips with a stated reason when Chromium extensions cannot load, never silently passes). -- Status: [x] Result: Chromium e2e test tests/e2e/nodes/extension.py passes (~10 s); Firefox 158 verified by scratch geckodriver script; CSP fix found and applied by that test; method in nodes/extension/README.md. - -### C-07 Client build and verification - -- `go vet ./...`, `go test ./...`, `node --test`, `nodes/client/build.sh` into a scratch directory (not `data/`). The distributable build into `data/node_clients/` is a deploy step left to the owner. -- Status: [x] Result: go vet, go test, node --test 68/68, cross-compile linux/windows/darwin amd64+arm64 and freebsd (agent report; I re-ran vet, test and node tests). - ---- - -## Track S: community sync (`devplacepy/services/sync/`, `database/sync.py`, `routers/sync/`) - -### S-00 Design decisions for new kinds (G-S2) - -- Kinds, in dependency order: `users`, `projects`, `posts`, `comments`, `polls`, `votes`, `reactions`, `follows`, `poll_votes`, `attachments`. `KINDS` is the single list; `SyncPusher`, `BatchRunner`, docs and Devii tools read it. -- Counters stay un-importable as numbers. A synced vote exists only with a synced voter, and the target `stars` is recomputed from the local `votes` rows by the same function `apply_vote` uses (`update_target_stars` + `clear_user_stars`), once per touched target per batch. This replaces "synced posts have stars = 0" with "stars are derived from synced votes only"; the doc rule is rewritten, not silently broken. -- No side effects, as before: no notifications, XP, badges, mention fan-out, AI correction/modifier. Content filter runs on every authored text (`projects`, `polls`). -- Records: - - `ProjectRecord`: `author_uid`, `title`, `description`, `status`, `platforms`, `website_url`, `repo_url`, `project_type`; always public and writable (`is_private = 0`, `read_only = 0`); no files, no containers. - - `PostRecord.project_uid` (optional reference to a synced project) closes "project links". - - `PollRecord`: `post_uid`, `question`, `options: [{origin_uid, label}]` (2 to 6, same caps as `create_poll`); options are rows of the poll, fingerprinted together. - - `VoteRecord`: `voter_uid`, `target_kind` in `post|comment|project`, `target_uid`, `value` in `{1, -1}`; a cleared vote is `deleted: true`. - - `ReactionRecord`: `author_uid`, `target_kind` in `post|comment|project`, `target_uid`, `emoji` (validated against the existing reaction emoji rule); tombstone removes. - - `FollowRecord`: `follower_uid`, `following_uid`; no self follow; tombstone removes. - - `PollVoteRecord`: `voter_uid`, `poll_uid`, `option_uid`; tombstone removes. - - `AttachmentRecord`: `author_uid`, `target_kind` in `post|comment|project`, `target_uid`, `filename`, `mime_type`, `content_b64`. **No network fetch inside a processor** (it runs on the request path and SQLite/HTTP stay out of each other's way): the sender downloads and sends the bytes; the processor decodes, applies the existing upload validators (type, size, image dimensions), and writes through `attachments.store_attachment` + `link_attachments`. `max_attachment_bytes` is a `SyncLimits` field and the envelope gets a total byte cap (`max_batch_bytes`). -- Voter-visibility: `poll_votes` are private ballots in this product (`UNREPORTABLE_TABLES` says so). They are accepted inbound (the sender owns that decision) and never exported (S-02). -- Each new local row is soft-deletable and created with `deleted_at: None, deleted_by: None`; tombstones soft-delete with `deleted_by = "sync"`; toggles revive instead of duplicating (the project's hard rules). -- Reportability: `projects` and `polls` are already reportable targets; synced rows reuse them. `votes`, `reactions`, `follows`, `poll_votes`, `attachments` are not new tables, so `REPORTABLE_TARGETS`/`UNREPORTABLE_TABLES` stay exact (the registry test is the check). -- Status: [~] Result: decisions implemented; correction recorded: votes, reactions and poll ballots are accepted inbound but NOT exported because no UI exposes who voted. - -### S-01 Implement the new kinds (G-S2) - -- `spec.py` (kinds, limits), `records.py` (models + `RECORD_MODELS`), `normalize.py` (emoji, options, platform lists), `processors.py` (one processor per kind on `RecordProcessor`, `PROCESSORS`), `database/sync.py` (link helpers for composite natural keys: a vote/reaction/follow link is keyed by the sender's own `origin_uid`; a second record for the same `(voter, target)` with a different `origin_uid` revives the same local row instead of duplicating, the concurrency lesson of the toggles), `routers/sync/` one leaf per kind (`projects.py`, `polls.py`, `votes.py`, `reactions.py`, `follows.py`, `poll_votes.py`, `attachments.py`), `GET /sync/spec.json` gains the schemas, Devii `sync_<kind>` tools (`requires_admin`, form-encoded `records`), `docs_api` endpoints, README, `docs/privacy.html` row update, sync CLAUDE.md rewritten for the new rules. -- Tests: unit per normaliser and processor (create, update, stale, tombstone, locally deleted, purged, wrong references), api per route (authz, gate, dry run, cap), stateful fuzz extended to all kinds with invariants (no dangling links, no orphan votes/reactions/follows, `stars` equals net local votes for every synced target, one vote per voter and target, no self follow), and a real multi-process race on the vote/follow toggle rows started from a fresh unseeded database (40 racers, exactly one row). -- Status: [~] Result: 10 kinds, records, processors, 7 new routes, Devii tools, docs_api; 69 api tests incl. new kinds. Fuzz + multi-process race and docs/privacy pending. - -### S-02 Export endpoint (G-S1) - -- `GET /sync/export/{kind}?since=<iso>&cursor=<opaque>&limit=<=50>` (administrator only, behind `sync_enabled`, audited as `sync.export`). Keyset pagination on `(updated_at, uid)`; the cursor is a signed-free, validated base64 of those two values. Output items are exactly the `*Record` models, so a DevPlace instance can feed another with the sender it already has. Soft-deleted rows are emitted as tombstones. -- Exported: `users` (public fields only: never email, hash, role, api key), `projects` (public only), `posts`, `comments`, `polls`, `votes` and `reactions` (author uid and target only), `follows`, `attachments` (as `url`, not bytes). **Not exported:** `poll_votes` (private ballots), anything private or deleted-by-account-deletion beyond the tombstone, and rows that arrived by sync themselves (`sync_links.local_uid`), so two instances cannot echo each other's data in a loop. -- Tests: api per kind (shape equals the inbound model, no private field, pagination is gapless and repeat-free over 120 rows, tombstones, the echo guard), a round trip test (export from one scratch database, import into another through the real routes, compare counts). -- Status: [~] Result: GET /sync/export/{kind} with expression indexes (EXPLAIN-tested), echo guard, keyset pagination; 27 api tests with deletions. Cross-database round trip pending. - -### S-03 Deferred resolution queue (G-S3) - -- Statuses gain `deferred` (an applied-later state, distinct from `rejected`). Table `sync_pending` (`origin`, `kind`, `origin_uid`, `missing_kind`, `missing_uid`, `record` JSON, `attempts`, `created_at`, `expires_at`) with a unique index on `(origin, kind, origin_uid)` so a resend replaces instead of stacking. -- A record rejected for `unknown_author`, `unknown_post`, `unknown_parent`, `unknown_target`, `unknown_poll`, `unknown_option`, `unknown_project` is queued (not on dry run) and reported `deferred` with the same reason. When any record of kind K with origin uid U is `created`, `BatchRunner` looks up pending rows waiting on `(K, U)` and re-applies them with the same processors, iteratively (a released comment can release its replies) under `pending_release_cap_per_batch`. Caps: `pending_max_per_origin` 5000 (the 5001st is rejected with the old reason, loudly counted), `pending_ttl_days` 7, `pending_max_attempts` 5; expired rows are garbage collected hard by the service already responsible for sync housekeeping (Community sync service `run_once`). -- The community ledger gains the outcome `deferred` (warn tone) with a plain-language reason; the pusher's projection logic for dry runs is unchanged. -- Tests: unit (queue, release in any order, chain of three, cap, ttl, replace-on-resend, dry run writes nothing), the 16-process race (post and its 15 comments arriving in random order from separate processes: exactly the right final counts), ledger reason text test. -- Status: [~] Result: sync_pending + PendingQueue; 12 api tests (chain, shuffle of 18 records, cap, ttl, dry run). 16-process race pending. - -### S-04 Deletion feedback to the origin (G-S4) - -- `GET /sync/deletions?origin=<id>&since=<iso>&cursor=` (administrator only) lists synced rows deleted or purged on this side: `{kind, origin_uid, deleted_at, deleted_by_class: "owner|moderator|system|purged"}`; `POST /sync/deletions/ack` (`origin`, up to 50 `{kind, origin_uid}`) stamps `sync_links.ack_at`. A record that the origin re-sends after acknowledging answers `unchanged / acknowledged_deleted` instead of the loud `rejected / purged_locally`, so the noise stops once the origin has been told. New columns `sync_links.deleted_seen_at`, `ack_at` (ensured). -- The scheduled community sender skips records whose link is locally deleted (cheap lookup) and records the skip in the ledger, so a re-collection of a deleted post stops being pushed every run. -- Devii tools `sync_deletions`, `sync_deletions_ack`; docs; audit `sync.deletions.ack`. -- Tests: api (listing by class, pagination, ack, reappearance answers `acknowledged_deleted`, authz), unit. -- Status: [~] Result: GET /sync/deletions and POST /sync/deletions/ack, acknowledged_deleted quiet answer; tests in deletions.py. - -### S-05 `min_comments` counts what is already synced (G-S5) - -- `Selector` counts, per post, comments collected in this run plus the live local comment count of the post's existing link (one batched `GROUP BY target_uid` query for all linked posts in the run, not per post). A post that was kept once is never dropped (existing rule), so this only changes first-time decisions. -- Tests: unit (collected 0 + 3 already synced passes a threshold of 3; none anywhere is dropped; the batched query is one statement). -- Status: [~] Result: Selector candidates (sync_candidates) park and revive a post with its comments and authors across runs; 10 unit tests. - -### S-06 and S-07 Alias heals when the clone name changes or collides (G-S6, G-S7) - -- `Aliaser.reconcile(origin)` runs at the start of every aliased run and after the push: for every identity whose clone user exists, compare `users.username` with `our_name`. On a difference: update the identity (`our_name`, `our_key`, skipping and logging when the new key collides with another identity), then clear the fingerprint of the origin's posts and comments whose stored text contains the old alias (bounded to 5000 links per run, oldest first), so the next push rewrites them with the live name. The rewriter always resolves names through the live clone username first. -- This covers both gaps: an administrator rename, and the case where `free_username` gave the clone a suffix because a registration took the name between alias generation and clone creation (the post-push reconcile sees the suffix and heals the mentions on the next run). The race window itself is also narrowed: the clone user is created before dependent content is rewritten. -- Tests: unit (rename, suffix collision, key collision skip, bounded sweep, nothing to do), an integration run through `CommunityRunner` with a fake collector: rename between runs, mentions follow. -- Status: [~] Result: AliasHealer rewrites stored mentions and follows the clone name; wired before and after the push in CommunityRunner; 6 unit tests. - ---- - -## Track K: community discovery (`devplacepy/services/community/`, `routers/admin/communities.py`, `static/js/`) - -Note: `runtime.py` and `extract.py` are inlined into generated collectors by `generator.py`. Every change there is covered by a generator test that executes the generated file, and existing communities regenerate their files (the database row holds the schema; files are regenerated when missing). - -### K-01 Live refresh while a run is in progress (G-K1) - -- `/admin/communities/{slug}` and the ledger page swap their `[data-live]` sections on an interval while the community is `busy` or a run row is `running` (the `AdminNodes.js` pattern, shared `Poller`, stops when idle). New `static/js/AdminCommunity.js` (one class, `app.adminCommunity`), no inline script; responsive at phone width. -- Tests: e2e (start a slow fake run, the progress counters change without reload, polling stops at the end), api (the page carries the live hooks and the run row state). -- Status: [x] Result: AdminCommunity.js live refresh on data-busy/data-live sections; busy field on AdminCommunityLedgerOut; tests/e2e/admin/communities/live.py. - -### K-02 Incremental run log and ledger, interrupted runs are visible (G-K2) - -- `CommunityRunner.run` registers a checkpoint callback: every `checkpoint_seconds` (15) and after every pushed batch it writes the current `LogBook` text, tally and verdict `running` into the run row, and flushes new ledger drafts with `LedgerStore.append` (continuing `seq`, respecting the same caps and rank rules). The final `finish_run` replaces the totals; ledger rows are never written twice (an index on `(run_uid, seq)` makes a duplicate a no-op). -- At service start, `CommunityStore.reap_interrupted()` marks any run left `running` for longer than `run_timeout_seconds` as `interrupted` (verdict `bad`, headline "interrupted: the process ended before the run finished") and keeps whatever was checkpointed. Retention (`prune_runs`) is unchanged. -- Tests: unit (a run killed after the second batch leaves log, tally and ledger; resume never duplicates), a real process kill test (`SIGKILL` a subprocess mid-run, then reap), caps still hold. -- Status: [x] Result: RunCheckpoint every 15 s, unique (run_uid, seq), reap_interrupted marks RUN_INTERRUPTED; unit tests in checkpoint.py. - -### K-03 Indexed text search (G-K3) - -- SQLite FTS5 external-content table `community_run_items_fts` over `label`, `origin_uid`, `detail` with `tokenize='trigram'` (substring semantics like the current `LIKE`, query minimum 3 characters), kept in step by triggers on `community_run_items`. Queries of 3+ characters use `MATCH` joined back to the scoped rows; shorter queries keep the bounded `LIKE` on the scoped index. If FTS5 or the trigram tokenizer is unavailable, `init_db` logs it once and the old path is used (the same results, the old cost). Prune deletes cascade through the triggers. -- Measure on the same scratch ceiling used before (10 runs of 4000 rows plus the community scan) and update the numbers: the no-match search was 37 ms, the old unbounded case 97 ms. `EXPLAIN QUERY PLAN` test extended so the FTS path is asserted, not assumed. -- Tests: unit (substring parity with `LIKE` over a generated corpus including `%`, `_`, quotes, unicode), query-plan test, trigger consistency after insert, delete and prune. -- Status: [x] Result: FTS5 trigram table with triggers for community-wide search of 3+ chars; measured 34 ms to 0.4 ms (no match), 34.5 to 0.8 ms (rare), common term 9.4 to 21 ms (slower, accepted). - -### K-04 Comment pagination beyond `rel=next` (G-K4) - -- `comments.pagination` becomes a learned and overridable spec: `next_link` (selector list, default adds text-matched anchors in a pagination container: `next`, `older`, `more`, `volgende`, `›`, `»`, `→`), `page_param` (`?page=N`, `?p=N`, `/page/N`, detected from the current URL or the numbered page links), `json_next` (a JSON `next`/`next_page`/`links.next` key). The collector stops on a page that adds no new comment ids or repeats a URL, always under `max_comment_pages`. -- The learner (`PageProbe`) detects the style from the sampled post pages and stores it; `NEXT_PAGE_FIELDS` remains the default. -- Tests: extraction fixtures for each style (rel next, text next, numbered page links, `?page=`, `/page/N`, JSON next), loop and duplicate guards, generated-collector parity. -- Status: [x] Result: next_link, page_param, json_next, none hints learned and enforced; stops on a page with no new comments; unit tests pagination.py. - -### K-05 Login: JSON-API and two-factor, captcha detected (G-K5, passwords excluded) - -- JSON-API login: `schema.login.kind = "json"` with `url`, `fields` (username/email/password key names), `token_path` (JSON path to a bearer token, optional); success is a 2xx plus either a token or a session cookie. `LoginLearner` probes a short fixed list of common endpoints (`/api/login`, `/api/auth/login`, `/api/v1/auth/login`, `/auth/login`, `/login.json`, `/session`) with the stored credentials, never anything outside the community origin, guarded by `guard_public_url`. -- Two-factor (TOTP): a new column `communities.login_totp_secret`, handled exactly like the other credential columns (never returned by any endpoint, admin form, Devii tool or CLI; set through `devplace community login --login-totp`). `extract.totp(secret, now)` implements RFC 6238 with the stdlib. The login flow detects a follow-up one-time-code step (`otp`, `totp`, `code`, `token`, `2fa` input) and submits the current code. Tested against the RFC 6238 vectors. SMS and push 2FA cannot be automated and are reported as `second_factor_unsupported`. -- Captcha: detection of `g-recaptcha`, `h-captcha`, `cf-turnstile`, `captcha` form markers moves the community to `login_required` with the reason `captcha_required` and a plain message; it is never attempted. Recorded as unsupportable by design. -- Tests: fixtures for a JSON login (token and cookie), a TOTP step with the RFC vectors and a fake clock, a rejected code, the captcha stop, credential non-disclosure (every serializer and the CLI output are scanned for the secret). -- Status: [x] Result: TOTP (RFC 6238 vectors), JSON-API login, OTP form detection, captcha detected and reported (never solved); secret never returned; models, router, CLI, Devii, docs_api, templates plumbed; tests login.py, admin index test. - -### K-06 Structured-data learning for hostile markup (G-K6) - -- New inferrer, tried between the heuristics and the AI assistant: `StructuredDataInferrer` reads JSON-LD (`DiscussionForumPosting`, `Article`, `BlogPosting`, `Comment`, `ItemList`) and schema.org microdata (`itemtype`/`itemprop`) and OpenGraph, producing a feed type `json_ld` (runtime support: extract the list of `<script type="application/ld+json">` blobs, flatten `@graph`, then reuse the existing JSON path engine). Known software signatures (Discourse, Lemmy, WordPress `wp-json`, Mastodon, phpBB/XenForo markup) select a ready, fixture-tested spec instead of guessing. -- Each learned field is validated against the sample like the AI proposals (60 % rule). The failure message names which fields could not be learned, so the human edit (K-07) is targeted. -- Tests: fixtures per source type (JSON-LD, microdata, OpenGraph-only, a Discourse `.json` topic list, a WordPress `wp-json` posts feed), generated-collector parity, no regression on the existing lobste.rs fixtures. -- Status: [x] Result: json_ld feed type, detector, learner, runtime; unit tests structured.py. Discourse and WordPress ready-made specs not built (schema editor covers hostile markup). - -### K-07 Admin schema editor (G-K6) - -- `GET/POST /admin/communities/{slug}/schema` (administrator, audited `community.schema.update`): shows the learned `SCHEMA` JSON; a save validates structure with the same validator the learner uses, runs a verification collection on a small sample through the real collector (dry run, nothing pushed), and only then stores the schema and regenerates `collectors.py` (verify before write; the previous schema is kept as `schema_previous` for one-click restore). Devii `community_schema_get` / `community_schema_set` (set is in `CONFIRM_REQUIRED`). -- Tests: api (valid, invalid structure, failing verification keeps the old schema, restore, authz, audit), e2e (edit and save through the page). -- Status: [x] Result: /admin/communities/{slug}/schema GET/POST/restore with validate_schema and sample verification, schema_previous, audit keys, Devii tools, docs_api; tests schemacheck.py and api schema.py. - -### K-08 Collect attachments, votes and reactions where they can be attributed (G-K7) - -- Attachments: option `collect_attachments` in `sync_options` (default off, per community). The collector finds inline images and attachment links in post and comment bodies, downloads them through `Fetcher` (stealthii + SSRF guard) under per-file and per-run caps, and emits `attachments` records with `content_b64`. Failures are ledger warnings, never run failures. -- Votes and reactions: the learned spec may name optional `votes` / `reactions` list fields (per-voter entries with a user id or name and a direction/emoji). When the source exposes them, voters become synced users and the records flow through S-01. **When the source only publishes an aggregate count (almost every site), nothing is imported:** an anonymous total cannot be turned into votes without inventing voters, which the platform forbids by design (no purchasable or fabricated engagement). This is stated in the docs as intentional, not as a missing feature. -- Tests: fixtures (attachment download within caps, oversize skip, blocked address, per-voter JSON feed becomes votes with derived stars, aggregate-only feed produces zero vote records), ledger rows. -- Status: [x] Result: collector emits votes, reactions and attachments; alias mode drops voter-attributed rows; generator VERSION 2 auto-regenerates; unit tests engagement.py. - -### K-09 Cross-run duplicate detection (G-K8) - -- The collector `state` (the watermark JSON already persisted per community) gains `links`: a bounded map `canonical_link -> post origin_uid` (cap 20000, oldest dropped, trimmed on write). `add_post` consults it, so a twin story seen in a different feed in a later run is aliased to the first post instead of becoming a second post; comments attach to the survivor as they do within a run. -- Tests: two feeds, two runs, same canonical link: one post; map cap and trimming; same-feed posts with the same link are still never merged (existing safe rule); generated-collector parity. -- Status: [x] Result: state links capped at max_remembered_links (20000); unit tests dedupe.py. - ---- - -## Track SK: skills (`devplacepy/services/skills/`) - -### SK-01 Real network leg for the skill fetcher (G-SK1) - -- The SSRF guard refuses loopback, which is why the network leg was never exercised. This machine has a global IPv4 address, and a server bound to it passes the guard unmodified, so the test runs the real `guarded_async_client` over a real socket with no patching: `ThreadingHTTPServer` on `0.0.0.0:<free port>` serving a `SKILL.md` plus a bundled `references/` file, fetched through `fetch_skill` at `http://<global ip>:<port>/SKILL.md`. The test discovers the address (UDP connect trick, then `is_blocked_address`), and **skips with a stated reason** when the host has no global address, never passing silently. Also asserted over the same socket: a loopback URL is refused, a redirect to a loopback address is refused, an oversize body is cut at the limit, an HTML page is rejected. -- Status: [x] Result: real-socket network test in tests/unit/services/skills/fetcher.py through the host's global IPv4, unpatched guard; 36 pass; skips with a reason when no global IPv4. - ---- - -## Cross-cutting work - -### X-01 Documentation of record - -Rewrite the `Gaps` sections to state only what is still true, move closed gaps into "What exists today" with the measured numbers, update: `services/nodes/CLAUDE.md`, `services/sync/CLAUDE.md`, `services/community/CLAUDE.md`, `services/skills/CLAUDE.md`, `nodes/extension/README.md`, `nodes/client/README.md`, `README.md`, `plan.md` (node plan progress), the docs pages (`nodes.html`, `nodes-protocol.html` reads the spec, `privacy.html`, community and sync pages), `docs_api` groups, `audit/events.md`. The memory file `node-network-plan.md` is updated at the end. -- Status: [ ] Result: - -### X-02 Validation gates (all must be clean) - -1. `python -c "from devplacepy.main import app"` after every Python edit. -2. `ruff check` / `pyflakes` on every touched Python file. -3. `go vet ./... && go test ./...` in `nodes/client`; `node --test` in `nodes/extension`. -4. `grep` for the em dash character and its HTML entity in every touched file. -5. CSS braces and HTML tags balanced for touched templates; JS parses. -6. Fan-out check per route (HTML, JSON, Devii, docs). -7. Four-layer correctness procedure for the spendable/state-machine/concurrent items (votes and stars, the pending queue, counters flush, cache, redundancy sampling): property checks, stateful fuzz, real multi-process races from fresh rows, static analysis. -- Status: [ ] Result: - -### X-03 Full suite - -- `make test` (unit, api, e2e, serial). Every failure is fixed, not skipped. -- Status: [ ] Result: - -### X-04 Commit - -- Commits per track with the pre-commit hook bumping the patch version. No push (outward-facing, not requested). The owner is told about the dist build into `data/node_clients/` and the optional public-hostname idle test. -- Status: [ ] Result: - ---- - -## Cannot be closed by code (kept visible on purpose) - -| Gap | Why | What is built instead | What closes it | -|-----|-----|-----------------------|----------------| -| G-N8 legal decision | A platform-terms, donor-terms, ISP-terms and jurisdiction decision is a human act. | Nothing is enabled for any account; `nodes_enabled` is `0`; the kill switch is enforced within one tick. | The owner records the decision; then switches the service on. The decision must cover: Reddit and any future destination terms, residential donor terms of service, ISP terms on traffic relaying, the jurisdiction of donors and of the operator, and the adult-only rule. | -| G-N1 Reddit from a residential line | Needs a residential machine. | Per-class success is recorded and shown (N-10). | A residential donor runs the Go client or the extension for a while. | -| G-N7 24 h soak, public idle timeout | Time and a production write. | Local soak with its true duration, local idle measurement, proxy config quoted. | Run the same harness for 24 h; owner approves a one-off production connection. | -| G-K5 captcha | Defeating a captcha is circumventing the control it exists for. | Detected, reported as `captcha_required`. | A human logs in once, or the site offers an API. | -| G-E3 Firefox (if the build cannot load add-ons) | Depends on the available Firefox build. | Chromium proven for real. | A normal Firefox with `web-ext run`. | - -## Progress log - -(One line per finished item: date, item, evidence.) - -## Result summary - -(Filled in at the end: what is closed, what is not, measured numbers, test counts, commits.) diff --git a/happyplanting.md b/happyplanting.md deleted file mode 100644 index 51f060f6..00000000 --- a/happyplanting.md +++ /dev/null @@ -1,124 +0,0 @@ -# happyplanting: fixing Continuous Delivery lock-in - -Status: plan only. Nothing in this file is implemented. Written 2026-10-17 from a read of the code, not from a run of the game. - -## Problem (verified in code) - -Continuous Delivery (`autoreplant` Mastery upgrade) replants the crop that was just harvested, immediately, whenever it is affordable. - -- `store/actions.py::harvest` and `_auto_harvest` call `_try_autoreplant` after crediting. The harvest pays before the replant is checked, so "affordable" is true in practice and the plot is never empty. -- `plant()` refuses any plot that already holds a crop. There is no uproot, swap or cancel. -- `upgrade_mastery` is one-way: no refund, no toggle. Mastery survives Refactor. -- `_try_autoreplant` checks cost only. It ignores market saturation and crop locks. -- Only exits today: Refactor (clears plots, costs a prestige-scaled fee) or running below the planting cost (rare). - -## Goals - -1. A per-plot "stop after this harvest" toggle, so the player gets one free crop choice on that plot. -2. Do not replant into a bad market. - -## Part A: per-plot "stop after harvest" (ship this) - -### Design - -New column `game_plots.stop_after_harvest` (int, default 0, always read with `COALESCE`). Added to the existing plot column ensure-block in `database/schema.py` (next to `raided_fraction`, ~line 1508) with example value `0`. - -Semantics: one-shot, per plot. While set, the next harvest of that plot does not replant. The plot ends empty and the player plants whatever they want. - -### Why this is the race-free shape - -The flag is read by `_try_autoreplant` from its own fresh `_plot_at(...)` re-read, which already happens AFTER `clear_plot` wins. Rules that make that safe: - -- The toggle endpoint is one conditional UPDATE: `SET stop_after_harvest = :v WHERE crop_key != '' AND crop_key IS NOT NULL`, checked via `rowcount` (`conditional_update_row`). It cannot touch an already-cleared plot. -- So the toggle either lands before `clear_plot` (the post-clear re-read sees it, no replant) or after (the WHERE fails, loud "plot is empty" error). No window where the harvest uses a stale flag. -- `clear_plot` is NOT changed. A leftover `1` on an empty plot is harmless because every path that puts a crop on a plot writes `stop_after_harvest = 0` in the same claim UPDATE: `plant()` and `_try_autoreplant` (both already use `conditional_update_row` with a SET string), and the `prestige()` plot reset. -- Never reset the flag inside `clear_plot`: that would erase it before `_try_autoreplant` can read it. - -### Changes (fan-out, all four faces) - -| Layer | File | Change | -|---|---|---| -| Schema | `database/schema.py` | ensure `game_plots.stop_after_harvest` | -| Data | `services/game/store/actions.py` | `_try_autoreplant`: return early if `COALESCE(plot.stop_after_harvest, 0)`; add `stop_after_harvest = 0` to the claim SET in `plant()` and `_try_autoreplant`; add the column to the `prestige()` plot reset | -| Data | `services/game/store/` (new fn in `actions.py`) | `set_plot_stop(user, slot, stop: bool)`: owner only, requires `mastery_autoreplant` level > 0, requires a crop on the plot, `GameError` otherwise | -| Form | `models.py` | `GamePlotStopForm(slot: int ge=0 le=64, stop: bool)`, modeled on `GameSlotForm` | -| Route | `routers/game/index.py` | `POST /game/plot/stop`, `require_user`, via `_respond_action`, audit `game.plot.stop` | -| Audit | `services/audit/events.md` + `category_for` | add `game.plot.stop` (same category as `game.plant`) | -| Schema out | `schemas/game.py` | `GamePlotOut.stop_after_harvest: bool = False` (owner only, default safe); `serialize_plot` sets it | -| View | `templates/_game_grid.html` and `static/js/GameFarm.js::_plotHtml` | a small toggle button on growing/ready plots, shown only when the viewer is the owner and owns Continuous Delivery; label "Stop after harvest" / "Resume auto-replant". Server render and JS mirror must match (the file already warns about flash on divergence). Reuse the existing `data-game-action` form delegation, no new JS class | -| Agent | `services/devii/actions/catalog/game.py` | `game_plot_stop` Action (`requires_auth=True`, params `slot`, `stop`). Not confirm-gated: reversible and costs nothing | -| API docs | `docs_api/groups/game.py` | `endpoint()` entry with `sample_response` | -| Prose docs | `templates/docs/code-farm.html` | Continuous Delivery row: document the toggle and that Mastery has no refund | -| Notes | `services/game/CLAUDE.md`, `README.md` | document the flag and the "plant/replant claim resets it" invariant | - -### Gaps found in the second review (all must be in the change) - -- `prestige()` resets plots with an explicit dict in `_plots().update({...})` (actions.py ~611). It does not touch unknown columns, so the dict must add `"stop_after_harvest": 0` or a flag survives a Refactor. -- A stale `1` on an EMPTY plot (left after a no-replant harvest) must never be shown: `serialize_plot` gates `stop_after_harvest` on `crop_key` present AND viewer is owner AND `mastery_autoreplant` owned. -- Writers that put a crop on a plot are exactly two: `plant()` and `_try_autoreplant` (both `conditional_update_row` claims). Both SETs get `stop_after_harvest = 0`. `_create_plot` inserts need no change (column default via ensure-block, read with COALESCE). -- Existing tests that enumerate things and will fail until updated: `tests/unit/services/devii/actions/catalog.py` (game action name list), `locustfile.py` (route coverage, the locust-maintainer agent owns it, never run a load test), `events.md` (audit catalogue), `README.md` tool list (line ~223) and the `code-farm.html` endpoint table (~line 433). -- The toggle button is a new element in a dense plot card: needs CSS tokens only (no literals), must fit the smallest phone width, and needs a `screenshot` skill visual check in both server-rendered and JS-rendered forms. -- `notify_farm` already runs in `_respond_action`, so the live refresh needs no extra wiring. -- Interactions checked and unaffected: steal (never clears a plot), water, fertilize, CI Bot `_auto_harvest` (flag honoured the same way, via the post-clear re-read), Era/leaderboards, treasury/grant. - -### Not in scope for Part A - -No farm-wide off switch and no refund of Mastery points. The per-plot toggle is the requested fix. - -## Part B: do not replant into a bad market (REVISED after second review: do not build the original) - -### Measurement - -Computed with the real `economy.realizable_harvest_coins` at the saturation floor (`market_factor` 0.4), payout minus planting cost: - -| Prestige | shell | api | kernel | distsys | mlpipe | secfort | -|---|---|---|---|---|---|---| -| 0 | -1 | 0 | +240 | -480 | -3600 | -10800 | -| 50 | +54 | +1120 | +18240 | +56020 | +101400 | +229200 | -| 60 | +65 | +1344 | +21840 | +67320 | +122400 | +277200 | - -Continuous Delivery needs prestige 50 (`MASTERY_UNLOCK_PRESTIGE`), where `prestige_multiplier` is 13.5. At that point even a fully saturated crop pays 11x or more its planting cost, for every crop in the catalog. Consequences: - -- The "profit floor" (skip when payout <= cost) can never fire for a real owner. It is dead code. Drop it. -- A plain "skip when saturated" always makes the owner poorer: the idle plot earns 0, and a skipped replant also forfeits that cycle's XP, quest progress, weekly harvest count (grant eligibility, `harvests` and `fair_play` boards) and the market tick. Idle is never better than planting a saturated crop. -- So the original Part B would be collateral damage, not a fix. - -### What to do instead - -1. Ship Part A only. It already solves the stated problem: one free crop choice per plot, whenever the player wants it, with zero economic downside. -2. Optional, only if you still want saturation handling: an opt-in farm setting "when saturated, replant the best unsaturated unlocked crop instead". It never idles a plot. It needs a new pure function in `economy.py` that picks the crop (must define "best", for example highest realizable coins per second, among crops passing `crop_lock_reason`), used by both the serializer preview and `_try_autoreplant` (one-quantity-one-function rule). It changes which crop grows and its grow time, so it needs its own design pass and the Layer 1/2 property sweeps. Not recommended to bundle with Part A. - -## Risk list - -- **Behavior change on deploy:** Part A defaults to 0 for every existing plot, so nothing changes until a player clicks. There is no Part B behavior change (see revision above). -- **Missing column on first plant:** the claim SET names `stop_after_harvest`, so a DB where `init_db` has not added the column would fail every plant. `init_db` runs before serving, and tests start from a fresh DB through it; verify with the layer-3 setup that uses a genuinely fresh, un-pre-seeded farm. -- **NULL trap:** pre-existing plot rows will have `stop_after_harvest` NULL. Every read and every WHERE uses `COALESCE(stop_after_harvest, 0)`. -- **Prestige reset:** forget the column there and a stale `1` survives a Refactor. Covered by the claim-SET rule, but the reset dict must also write 0. -- **Visitors:** the flag must never be exposed to non-owners (`serialize_plot` gates it on `viewer == owner`). -- **JSON vs template:** a context key not on `GamePlotOut` is silently dropped from JSON. Add the field to the schema first. - -## Verification plan - -Persisted tests (matching tiers; the suite is run by the orchestrating session, serial): - -- `tests/api/game/mutations.py`: toggle on, harvest, plot stays empty; toggle off, harvest, plot replanted; toggle on an empty plot is a 400; toggle without Continuous Delivery is a 400; a visitor cannot toggle; `plant` after a stop resets the flag; Refactor resets it. -- `tests/api/game/farm.py`: visitor view of an owner's plots never shows `stop_after_harvest`. -- `tests/e2e/game/index.py`: the toggle button renders and flips on click; server-rendered and JS-rendered markup agree. - -Disposable scripts (scratch DB via `DEVPLACE_DATABASE_URL` and `DEVPLACE_DATA_DIR`, never production): - -- Layer 3 race: real OS processes firing toggle against harvest on the same plot, from a genuinely fresh farm (flag NULL, not pre-seeded to 0). Assert the replant happens exactly when the toggle landed after `clear_plot`, never when it landed before. -- Layer 4: `pyflakes`/`ruff check` on every touched file. - -Then the standard gates: clean `from devplacepy.main import app` import, em-dash grep on touched files, `make test` (all tiers) green, commit, and verify live on `127.0.0.1:10500`. - -## Residual risk after the second review - -- Part A is a small, additive change with one new column, one new route and one guarded early return. I found no economic, concurrency or permission collateral in it. "No collateral damage" is a claim about the plan, not a measurement: it is proven only by the race script and the full suite above, neither of which has been run. -- One accepted behavior: toggling "stop" on a plot that harvests in the same instant either stops the replant or returns "plot is empty". Both outcomes are correct and visible. - -## Decisions needed from you - -1. Part A: one-shot (recommended, flag cleared when the plot is next planted) or sticky until turned off? -2. Part B: drop it (recommended), or commission the "replant best unsaturated crop" option as a separate piece of work? -3. Fold in the crop-lock check fix in `_try_autoreplant` (docs promise it, code lacks it)? diff --git a/nicknames.md b/nicknames.md deleted file mode 100644 index 786cc7f7..00000000 --- a/nicknames.md +++ /dev/null @@ -1,101 +0,0 @@ -# How to Generate Unique, Realistic Usernames for a Forum - -Accounts often have to be created for people who never picked a handle: bulk imports, migrations, data feeds. The source rarely carries a clean name. You get strings like `abner_page_by_abnercoimbre` or `via_rberger_2026-10-15_12_36_38_`. This article describes a method that turns such strings into names a person would actually pick, and keeps them unique and stable. - -## The problem - -Raw author strings often have the shape `<site> via <person>` or `<site> by <person>`: - -``` -lore.kernel.org via thang -jakearchibald.com by jaffathecake -decomp.dev via FedericoSchonborn -``` - -Using that string as the username gives ugly names. Worse, it gives the same person several accounts, because the same person appears next to different sites and each string looks like a different author. - -Fix the identity first, then the name. - -## Step 1: Decide who the person is - -Before you generate anything, define the identity key, because a handle is only stable if the key is. - -- Strip noise from the raw string: timestamps (`2026-10-15 12:36:38`) and relation words. -- If the string says `by` or `via`, the **person is the token after it**. The site comes before it. -- If the string is only a domain (`costa_jacobpalm_dk`), drop the top-level domain and keep the last meaningful label (`jacobpalm`). -- Use that cleaned person as the identity key, not the original string. - -Now one person appearing next to ten sites is one account. - -## Step 2: Create the handle once and store it - -Generate a handle the first time you see an identity. Store the link between the identity and the account in a table with a unique constraint, for example `(source, identity) -> account`. - -Every later lookup finds the link and uses the stored name. **The generator is never consulted twice for the same person.** Renaming later is how you end up with handles that drift, so do not do it. - -Two guards matter: - -- Only choose the name when the account is created. Later updates keep the stored name. -- Never touch accounts that already exist. Backfilling old names is a separate decision with its own risks. - -## Step 3: Make the name look chosen - -Real handles are not random strings. One empirical study of usernames on Google profiles (arXiv 1101.5578, 2011, Latin-alphabet names only) found: - -- About 70% of usernames contain part of the user's real name. -- About 30% are plain first-plus-last concatenations. -- More than 18% add digits to the name, likely birth years or a number to get around a taken name. - -That is the only solid distribution found. It says nothing about separators or humorous suffixes, so those weights are judgment, not data. - -Pick one pattern per person with fixed weights: - -| Pattern | Weight | Example | -|---|---|---| -| Plain name | 40% | `runxiyu` | -| Name plus a number (year, two-digit year, or a small number) | 30% | `hongminhee90`, `untrusem_2000` | -| Separator plus number | 8% | `jaffathecak-73` | -| Funny (prefix, suffix, or a nerd number) | 17% | `federicoschonborn_irl`, `lcapald_500` | -| Light leetspeak | 5% | `wr1ter0` | - -The funny parts come from small word lists: prefixes like `its`, `dr`, `mr`; suffixes like `_dev`, `_irl`, `_ftw`; numbers like `42`, `1337`, `404`, `418`. Years run 1972 to 2006. - -## Step 4: Make it deterministic - -Seed the random generator from a hash of the source and the identity, for example `sha256(source:identity)`. The same person then always produces the same ordered list of candidates, which makes the system reproducible and testable. - -If the first candidate is taken, move to the next. Finally fall back to a numbered suffix, so a free name always exists. - -## Step 5: Enforce hard limits - -Constants, not heuristics: - -- Characters: letters, digits, `_` and `-` only. -- Length: 3 to 32. -- Base name: trimmed to 18 characters before decoration, so a suffix always fits. -- Case: lowercase, as people usually type handles. -- A bounded number of re-rolls before the fallback. - -Test these with property checks across many inputs, including empty strings, very long strings, accented characters, and strings made only of separators. Every candidate must be valid. - -## Step 6: Handle the exceptions - -- **Deliberate pseudonyms.** If you replace names on purpose so the original is hidden, do not decorate those names. Mentions are rewritten to match the replacement, and any change would break the match. -- **Trusted senders.** If another system sends you clean handles, keep them. Apply this method only where the names cannot be trusted. -- **Collisions.** Compare names case-insensitively, since `Alice` and `alice` should not coexist. - -## Known weaknesses - -- The weights are constants, not tuned on real data from your own user base. -- A name that already ends in digits can look odd after a year is added (`spc476` becoming `spc4762005`). -- Changing the identity key means records created before the change no longer match their old links. They appear as new accounts, and the old ones remain as orphans. Plan a cleanup if that matters. -- The result is a plausible handle. It makes no claim about who the person is. - -## Summary - -1. Fix the identity first: extract the person, drop the site and the stamp. -2. Create the handle once, store the link, and never regenerate it. -3. Pick a pattern with weights based on how real people name themselves. -4. Seed the randomness from the identity so results are reproducible. -5. Enforce length, character and uniqueness rules. -6. Leave existing accounts and deliberate pseudonyms alone. diff --git a/plan.md b/plan.md deleted file mode 100644 index 7196e07a..00000000 --- a/plan.md +++ /dev/null @@ -1,360 +0,0 @@ -See research_residental_proxies.md. Userss are by default opted out, but can opt in and download required tools to make it happen on their profile page. -We label this as donating to the devplace community ecosystem. We have a beautiful infrastructure that is very reliable, barely costs maintenance and even better, almost free. -Our search engine destroys searchxng. Also, our search engine has actually a fallback provided by jina. I bough jina in someday but then decided not to use their stuff what ended up -as a nice backup system for failing searches. But what do we want? We also want reddit, X and all other social media to be part of our search results. And that is only possible -when people contribute some of their resource. There are severa ways to do this. Choice is to members. We support all common options and you can configure yourself how much resource -you want to donate. We will also count the amunt of data i/o / request count and display it as donated by you. Also! For the sake of our new agent system on pravda.education, we really -needs this residental proy stuff to happen. Our platform destroys grok atm, but grok has the social media searches and they are important. - -Please create a huge implementation plan of literally everything that needs to happen at the end of this file. Thats literally all for now. - ---- - -# IMPLEMENTATION PLAN: Community Node Network ("Donate to the DevPlace Ecosystem") - -Status: plan only, nothing built. Written against the repo as of 2026-10-14 and against `research_residental_proxies.md`. -Working name: **Nodes**. Code lives in `devplacepy/services/nodes/`, routers in `devplacepy/routers/nodes/`, user-facing label "Donate to the community". - -## 0. Findings that change the shape of the plan - -These come from reading the repo and the research file. Several of them contradict the framing in the brief, so they come first. - -1. **There is no Jina code in this repo.** `grep -ri jina` matches only `plan.md`. The search engine is the external service `https://rsearch.app.molodetz.nl` (`crawl.RSEARCH_URL`, `services/devii/rsearch/`). The Jina fallback therefore lives in the rsearch codebase, not here. This repo only consumes rsearch from two places: `services/jobs/deepsearch/crawl.py::search_queries` and the Devii `rsearch*` tools. Where social search is implemented (here or in rsearch) is open decision D1. -2. **The repo already knows social sites block it.** `crawl.HOSTILE_DOMAINS` lists x.com, reddit.com, instagram, facebook, linkedin, tiktok, threads, youtube and skips them. Root `CLAUDE.md` records that reddit blocks the datacenter ASN outright and that the only fix is an IP change. So the problem is real and measured for Reddit. It is not measured for the others. -3. **A residential IP fixes IP-reputation blocks, not login walls.** Reddit public JSON endpoints work from a clean IP without login. X search, Instagram, LinkedIn, Facebook and TikTok mostly need a logged-in session. Donor nodes must NOT use the donor's cookies (privacy and liability), so those platforms stay mostly out of reach. X is limited to what is public (permalinks, oEmbed/syndication timelines). Promising "Reddit, X and all social media" would be a false claim. Phase 0 measures what actually works per platform before anything else is built. -4. **Several "social" sources need no donors at all**: Hacker News (Algolia API), Bluesky (public AppView), Mastodon (public instance APIs), Lobsters, StackExchange, GitHub, dev.to, YouTube oEmbed/transcripts. They are cheap, reliable, and cover a large part of what developers want. Build these as direct adapters first (Phase 6A). Nodes are only for the platforms that block datacenter IPs (Reddit first). -5. **Browser extensions cannot do TCP.** MV3 has no raw sockets. An extension can only run `fetch()`. So there are two fundamentally different transports, with different trust properties: - - **Tunnel mode** (native client): the server opens an end-to-end TLS session to the destination THROUGH the donor's TCP socket. The donor sees only ciphertext plus the destination host. The donor cannot read or tamper with content because they cannot forge the destination's certificate. Our `curl_cffi` Chrome fingerprint stays intact. This is the better mode. - - **Fetch mode** (extension): the donor's browser performs the HTTP request and returns the body. The donor sees the URL and plaintext response and CAN forge the response. Everything returned is untrusted and must be verified (section 5.6). TLS fingerprint is the donor's browser, which is fine. -6. **A plain web page or service worker cannot do this at all.** The research table rates them "Low"/"Poor". The reality is "impossible": a page cannot `fetch()` reddit.com cross-origin (CORS), and cannot open raw sockets. Both are dropped from scope. Documented so nobody revisits them. -7. **Open SOCKS5 is the wrong product.** The research recommends exposing a SOCKS5 listener. We do not need an open proxy. We need to reach a short, fixed list of public social domains for our own search. A destination-allowlisted exit (donor hard-refuses everything else) removes most of the abuse and liability surface the research file warns about (credential stuffing, ad fraud, illegal traffic). This is the single most important design decision and it is a clear rule, not a configurable maybe: **no open proxy, ever; allowlist enforced on server AND on the donor client.** -8. **Existing seams to reuse, none to rebuild:** - - `stealth.stealth_async_client(proxy=...)` and `outbound_proxy_url`: a local node-backed forward proxy plugs into this with zero caller changes. - - `messaging/tickets.py` + `ws_tickets`: single-use WS auth ticket flow (browsers cannot set WS headers). - - `pubsub` lock-owner WS pattern (`4013` retry) for the node socket. - - `BaseService` + `ConfigField` for the dispatcher service and admin knobs. - - `LiveViewRelayService.VIEWS` + private `user.{uid}.*` pub/sub namespace for live donor stats. - - `user_consents` / `CONSENT_KINDS` for opt-in (default opted out is already how `ai_third_party` works). - - `utils.track_action` / `ACHIEVEMENTS` / `BADGE_CATALOG` for donor badges. - - `GatewayUsageLedger.record_external` pattern for metering consumers. - - `devii` `CONFIRM_REQUIRED`, `docs_api.endpoint()`, `DOCS_PAGES`, `audit` categories + `events.md`. -9. **Legal and ToS exposure is real and is risk number one.** Reddit and X actively litigate and rate-limit scraping; ISPs restrict commercial use of residential lines; donors' IPs will get rate-limited or banned by the platforms. The plan includes a legal review gate (Phase 11) before any public launch. This is not legal advice and the plan does not assume the outcome. -10. **Unknown that blocks scoping:** what the "new agent system on pravda.education" is and how it will call search. The plan gives it a stable internal interface (section 6) but its requirements (latency, volume, platforms) are open question Q3. - -## 1. Decisions needed from you before Phase 2 (everything else proceeds without them) - -| # | Decision | Recommendation | Why | -|---|----------|----------------|-----| -| D1 | Where does social search live: in devplacepy, or in the rsearch codebase? | devplacepy owns the node fabric and exposes an internal HTTP face; rsearch calls it (or devplacepy does the social search itself) | Nodes are tied to DevPlace accounts, consent, and the ledger. rsearch has no accounts. | -| D2 | Native client language | **Go**, static binary, one file per OS/arch | Research recommends Go/Rust; zero runtime for Windows users. Adds a Go toolchain to the build. Alternative: Python zipapp (no new toolchain, but needs Python on the donor machine). | -| D3 | Extension distribution | Sideloadable zip first (audience is developers), store submission later | Stores scrutinize proxy-like extensions; sideload ships sooner and the audience can handle it. | -| D4 | Donation reward | XP + badges + leaderboard only. No coins, no money, no purchasable anything | Root rule: no payment path, no purchasable randomness. Keeps app-store conditionals untriggered. | -| D5 | Minimum donor age | Adults only (`age_band` gate) | A minor must not share an IP/line. Needs a decision on how the existing `age_band` is read for legacy accounts with `NULL`. | -| D6 | Is a data center or a cloud VPS allowed as a "donor"? | No: residential/mobile-ASN heuristic, admin override | The whole value is residential reputation. A VPS donor is just another datacenter IP. | - -## 2. Specification first (your rule 7: everything has a declared spec) - -Nothing else is written until the spec exists, because the server, the Go client, the extension, the docs and the tests all derive from it. - -1. `devplacepy/services/nodes/spec.py`: the single source of truth, as frozen dataclasses (`FrameSpec`, `LimitSpec`, `DestinationSpec`, `AdapterSpec`, `ProtocolSpec`). It declares: - - protocol version (integer, with `min_supported`), - - every frame type with its fields, direction, max size, - - every hard cap in section 3, - - the destination allowlist (domain suffix -> adapter -> allowed methods -> allowed ports -> rate limit), - - donor-side hard refusals (non-public IPs, ports other than 80/443, non-allowlisted hosts). -2. `GET /nodes/spec.json` serves the spec so clients and tests read the same numbers. A contract test fails if the Go constants, the extension constants, the docs page, or the server drift from it. -3. `devplacepy/templates/docs/nodes-protocol.html` (generated tables from the spec, not hand-copied) and `docs/nodes.html` (user-facing). Registered in `DOCS_PAGES`. -4. A short `NODES.md` is NOT created; the nested `devplacepy/services/nodes/CLAUDE.md` is the engineering record, per repo convention. - -## 3. Hard caps and floors (nothing unbounded) - -All values below are starting ceilings, declared in `spec.py` and surfaced as `ConfigField`s only where an admin may lower them. Static ceilings, no autotuning. - -| Cap | Initial value | On hit | -|-----|---------------|--------| -| Devices per user | 5 | 409 with reason | -| Connected nodes total (lock-owner worker) | 2000 | WS close `4029`, counter `nodes_rejected_capacity` | -| WS frame size | 256 KiB | close `1009`, counter | -| Response body per job | 2 MiB | truncate + flag `truncated=true`, never silently | -| Concurrent jobs per node | 2 (donor may lower to 1) | scheduler skips node | -| Per node, per destination domain | min 6 s between requests | scheduler skips node | -| Global per destination domain | admin-set rps ceiling | queue, then 429 to caller | -| Job timeout | 25 s | mark failed, count against node health | -| Job queue depth | 500 | reject new with 429 + counter | -| Caller wait | 15 s default, 60 s max | `FabricUnavailable` | -| Per caller-owner daily fabric requests | admin-set (default 200 user, 0 guest) | 429, no donor load | -| Donor daily bandwidth (donor-configurable) | default 500 MB, max 20 GB | node goes idle until reset | -| Heartbeat | ping every 20 s, evict after 2 missed | reconnect | -| Reconnect backoff | 1 s to 60 s, full jitter | n/a | -| Ticket TTL | 30 s single use (existing) | n/a | -| Result cache | 10 min TTL, 256 MiB total, 4 MiB per entry | LRU evict | -| Usage daily buckets | keep 400 days, rolled into lifetime totals first | prune only after rollup verified | -| Node event log | 30 days, 50k rows | prune | - -**Floors (machines hold, humans decide):** -- The system may auto-**suspend** a misbehaving node (reversible, timed, e.g. circuit breaker after N consecutive failures or an integrity mismatch). It may never auto-**ban** a user or delete a ledger row. Hitting a ban threshold creates a flag for an admin and fires an admin notification. -- Lifetime donation totals live on the donor row and are never decreased by pruning. -- Protected destinations: the allowlist can only be changed by the primary admin and every change is audited. -- A global kill switch (`nodes_enabled` site setting) disconnects every node within one tick. It is the first thing built in Phase 4 and tested first. - -## 4. Data model - -All tables ensured in `database/schema.py::init_db` (full column set, because `dataset` only creates the columns of the first insert), indexes with `CREATE INDEX IF NOT EXISTS`, every INSERT writes `deleted_at: None, deleted_by: None` where the table is soft-deletable. Helpers in a new `database/nodes.py`, exported through `database/__init__.py` like `skills.py`. - -| Table | Purpose | Soft delete? | Notes | -|-------|---------|--------------|-------| -| `node_devices` | one row per registered client install: `uid`, `user_uid`, `label`, `kind` (`native`/`extension`/`container`), `secret_hash`, `secret_prefix`, `platform`, `client_version`, `protocol_version`, `status` (`active`/`suspended`/`revoked`), `suspended_until`, `suspend_reason`, `country`, `asn`, `asn_class` (`residential`/`mobile`/`datacenter`/`unknown`), `ip_hash`, `last_seen_at`, `created_at` | yes (revoke = soft delete, cascade stamp) | index `(user_uid, deleted_at)`, `(status, last_seen_at)` | -| `node_settings` | per-user donation prefs: `user_uid` (unique), `enabled`, `bandwidth_mb_day`, `max_concurrent`, `active_from`, `active_to`, `timezone`, `allowed_platforms` (JSON), `pause_on_battery`, `updated_at` | no (preference row) | enabled is subordinate to the consent, like `ai_correction_enabled` | -| `node_usage_daily` | per `(device_uid, day)` counters: `requests_ok`, `requests_failed`, `bytes_up`, `bytes_down`, `cache_hits_served`, `integrity_failures` | no (derived, hard GC) | unique `(device_uid, day)`; upserted with one atomic `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` (the `_add_usage` precedent) | -| `node_donors` | lifetime totals per user: `user_uid` (unique), `requests_ok`, `bytes_total`, `first_donated_at`, `last_donated_at`, `xp_awarded_today`, `xp_day` | no (derived) | survives bucket pruning; this is what the profile card and leaderboard read | -| `node_requests` | the cross-worker job queue: `uid`, `caller_kind`, `caller_id`, `url`, `method`, `adapter`, `mode` (`tunnel`/`fetch`/`any`), `status` (`queued`/`assigned`/`done`/`failed`/`expired`), `device_uid`, `result_status`, `result_bytes`, `result_body` (capped), `error`, `created_at`, `assigned_at`, `finished_at` | no (ephemeral, hard GC after 10 min) | partial index on `status IN ('queued','assigned')` | -| `node_events` | operational log: connects, disconnects, suspensions, verification results, refusals | no (hard GC, 30 days) | feeds admin view, not audit | -| `node_flags` | items needing a human: integrity mismatch bursts, datacenter-ASN donor, abuse reports against a node | yes | admin resolves; the floor rule's landing place | - -Related registrations (the repo fails the suite otherwise): -- `database/moderation.py`: none of these carry user-generated content visible to others, so each goes in `UNREPORTABLE_TABLES` WITH its reason. `node_devices.label` is owner-only text; confirm it is never rendered to anyone else, or it becomes reportable. -- `SOFT_DELETE_TABLES`: `node_devices`, `node_flags`. -- `services/moderation/deletion.py::OWNED_TABLES`: add `node_devices`, `node_settings`, `node_donors` so account deletion cascades. Add `node_usage_daily` via the device join. Revoke live sockets in the same call. -- `user_consents`: new `CONSENT_KINDS["bandwidth_sharing"]`. **Never granted by default, never backfilled.** A new consent is enforced at ONE choke point (section 5.3), never at N call sites. -- Site settings seeded in `init_db` `operational_defaults`: `nodes_enabled` (default `0`), `nodes_max_total`, `nodes_daily_quota_user`, `nodes_daily_quota_guest`, `nodes_cache_ttl_seconds`, `nodes_social_via_nodes` (route hostile domains through nodes: default `0`). - -## 5. Server - -### 5.1 Package layout (OO, one responsibility per class, one class per file where it is a class) - -``` -devplacepy/services/nodes/ - CLAUDE.md engineering record (written in Phase 12, started in Phase 3) - spec.py ProtocolSpec and friends (section 2) - registry.py NodeRegistry: connected sessions, capacity, lookup by uid/user - session.py NodeSession: one WS, frame codec, in-flight jobs, health, heartbeat - frames.py Frame encode/decode + validation against spec - consent.py ConsentGate: THE single choke point (5.3) - policy.py DestinationPolicy: allowlist, port/IP rules, per-domain limits - scheduler.py NodeScheduler: pick a node (fairness, diversity, caps) - broker.py JobBroker: enqueue, assign, await result, expire (DB-backed) - dispatcher.py NodeDispatcher: lock-owner loop moving node_requests to sessions - tunnel.py TunnelBridge: local forward-proxy listener -> tunnel frames - fetcher.py FetchBridge: fetch-mode request/response framing - verifier.py NodeVerifier: canary, ASN class, redundancy checks - ledger.py UsageLedger: atomic counters, rollups, XP award hook - cache.py ResultCache: bounded, URL-hash keyed - circuit.py CircuitBreaker: per node and per destination - fabric.py NodeFabric: the facade every consumer calls - service.py NodesService(BaseService): reconcile, metrics, prune, relay hook - adapters/ - base.py SearchAdapter (abstract): build_request, parse_response, normalize - reddit.py x.py ... one class per platform - direct/ hackernews.py bluesky.py mastodon.py lobsters.py stackexchange.py -``` - -### 5.2 Routers (`devplacepy/routers/nodes/` package, mounted at `/nodes` in `main.py`) - -| Route | Auth | Purpose | -|-------|------|---------| -| `GET /nodes/spec.json` | public | protocol spec for clients and tests | -| `GET /nodes/client-version` | public | latest client versions, `min_protocol`, sha256 per artifact | -| `GET /nodes/download/{artifact}` | public | native binaries, extension zip, systemd unit, container compose (static files from `DATA_DIR/node_clients/`, sha256 listed) | -| `POST /nodes/devices` | `require_user` + consent | create device, returns the secret ONCE | -| `POST /nodes/devices/{uid}/rotate` | owner | new secret, old revoked immediately | -| `POST /nodes/devices/{uid}/delete` | owner or admin | soft delete + kick socket | -| `POST /nodes/ticket` | device secret | exchange secret for a 30 s single-use WS ticket | -| `WS /nodes/ws` | ticket | the node socket; lock-owner gated, `4013` fast retry like `/pubsub/ws` | -| `POST /nodes/internal/fetch` | admin / API key, service accounts | the HTTP face of `NodeFabric.fetch` for agent systems | -| `GET /nodes/stats` | public | aggregate only: total donated bytes, requests, active donors (no per-user data) | -| `GET /nodes/leaderboard` | public | top donors, opt-in display | -| `POST /profile/{username}/donation` | owner | settings (bandwidth, hours, platforms) | -| `GET /admin/nodes`, `POST /admin/nodes/...` | admin | pool health, flags, suspend/unsuspend, allowlist, kill switch | - -Rules from the repo applied to every route: `respond(request, template, ctx, model=XOut)` for the HTML+JSON faces, `*Out` schemas in `schemas/nodes.py` (a context key missing from `XOut` is silently dropped from JSON), Pydantic `Form` models in `models.py`, `RedirectResponse(..., status_code=302)` literal, no `respond()` ctx key colliding with a Jinja global (use `viewer_is_admin`), specific paths before catch-alls, `is_senior_admin` on every per-user admin mutation, delete = owner OR admin. - -### 5.3 The consent choke point (`ConsentGate`) - -One function, `ConsentGate.allows(user_uid) -> Verdict(reason)`, true only when ALL of: `bandwidth_sharing` consent granted and not withdrawn, account active (not suspended, not in deletion grace), adult `age_band` (D5), `node_settings.enabled`, global `nodes_enabled`. It is called in exactly three places: -1. `POST /nodes/devices` and `POST /nodes/ticket` (refuse to mint). -2. `WS /nodes/ws` accept (refuse to connect). -3. `NodeScheduler.eligible()` (a node whose owner stopped qualifying is never assigned work, and `NodeRegistry` kicks the socket on the next tick). - -Withdrawing the consent takes effect within one dispatcher tick, not at next reconnect. Consent change is owner-only, exactly like `routers/profile/consent.py` (reuse `_owner_only`). **Devii may read status and may withdraw, but must not be able to grant** (a chat agent granting a bandwidth-sharing consent is not a consent). Verify how the existing `set_consent` Devii tool treats grants and restrict this kind there. - -### 5.4 WebSocket and multi-worker model - -- `/nodes/ws` is gated to the service-lock owner worker with the `4013` fast-retry close, so all node sockets converge on one process (precedent: `/pubsub/ws`, `/devii/ws`). This keeps `NodeRegistry` a single in-memory truth. Trade-off, stated: one worker carries all sockets, hence the 2000 cap, and a crash drops every node for a few seconds until they reconnect. Acceptable because reconnect is built in. -- Callers on ANY worker enqueue into `node_requests` (DB). The lock-owner `NodeDispatcher` ticks fast while at least one node is connected and a request is queued (target 200 ms, indexed partial query; the cost is measured in Phase 0, not assumed), assigns, and writes the result back. Callers `await` by short-sleep polling the row. A same-process fast path skips the table when the caller is already on the owner. This is the `JobService` philosophy (DB-backed, durable enough, no inter-worker RPC) with a tighter tick. -- Large bodies never go through SQLite long term: `result_body` is capped (2 MiB) and hard-GC'd after 10 minutes; the durable copy is the `ResultCache` entry, bounded in bytes. -- nginx: add an upgrade `location = /nodes/ws` to `nginx/nginx.conf.template` (the catch-all `location /` strips upgrade headers, and the file already has one per WS route). **Also verify on the `devplace.net` front host**, whose own proxy and SSH tunnel have separate idle timeouts; test both hostnames per the production-diagnosis procedure, never with `--resolve`. - -### 5.5 Frame protocol (summary; the authoritative version is `spec.py`) - -Client to server: `hello{protocol, client_version, kind, platform, caps, limits}`, `pong`, `job_result{job_id, status, headers, body_b64, truncated, timings}`, `job_error{job_id, code}`, `tunnel_data{conn_id, b64}`, `tunnel_close{conn_id}`, `usage{bytes_up, bytes_down}`. -Server to client: `welcome{device_uid, limits, allowlist_version, settings}`, `ping`, `job{job_id, method, url, headers, timeout}`, `tunnel_open{conn_id, host, port}`, `tunnel_data`, `tunnel_close`, `settings_update`, `kick{reason}`. -Every frame is validated against the spec on both ends. An unknown type or an oversize frame closes the socket with a loud reason and a counter (never silently ignored). - -### 5.6 Trust, verification and integrity - -Fetch-mode donors can forge content, and forged content flows into search results and into the AI agent. Controls, in order of strength: -1. **Prefer tunnel mode** whenever a tunnel-capable node is available: TLS makes tampering impossible. Fetch mode is only used for extension donors. -2. **Never trust fetch-mode bodies without checks**: structural validation by the adapter (must parse as the platform's JSON/HTML shape), size sanity, and **sampled redundancy**: a configured fraction of fetch-mode jobs (plus 100 % for a node's first N jobs) are also fetched through a second independent node or directly; a content-hash or structural mismatch increments `integrity_failures`, suspends the node, and writes a `node_flags` row. -3. **Provenance on every result**: results carry `via=tunnel|fetch|direct` and the agent/search layer can down-weight or label fetch-mode content. -4. **Canary / liveness proof**: on connect, the verifier makes the node fetch `GET /nodes/canary/{nonce}` (served by us, records the observed source IP). Proves egress really goes out of the donor's own line, detects a node that is itself behind another proxy, and lets us compare the observed IP with the WS peer IP. -5. **ASN class**: classify residential/mobile/datacenter via an offline ASN database file under `DATA_DIR` (no third-party lookup, so no donor IP leaves our server). Datacenter donors get `asn_class=datacenter` and are excluded from the pool by default (D6) with a flag for an admin. -6. Donor client strips cookies and auth: extension uses `credentials: "omit"` and never attaches the donor's session; native client sends only the headers the server specified. - -### 5.7 Donor-side safety is enforced on the donor, not only on the server - -A compromised or buggy server must not be able to point a donor's machine at their LAN. The clients enforce, independent of any server frame: -- allowlisted hosts only (list baked in per release and refreshed from the server, where a refreshed list can only be equal or narrower than the baked-in superset), -- ports 80 and 443 only, -- resolve DNS locally, refuse private/loopback/link-local/CGNAT/multicast addresses, then connect to the resolved IP (defeats DNS rebinding), -- no redirects to a non-allowlisted host, -- the donor's configured bandwidth, concurrency and schedule caps, applied locally even if the server sends larger ones. - -## 6. Consumers: how search gets the data - -`NodeFabric` is the only thing consumers import: - -``` -NodeFabric.fetch(url, *, owner, mode="any", cache=True, timeout=...) -> FabricResult -NodeFabric.social_search(query, *, platforms, owner, limit) -> list[SocialHit] -NodeFabric.available() -> FabricStatus # nodes online, per-platform health -``` - -Resolution order (each step loud when it fails, never silent): -1. `ResultCache` hit. -2. **Direct adapters** for platforms that need no donors (Hacker News, Bluesky, Mastodon, Lobsters, StackExchange, GitHub, dev.to). -3. **Nodes** for allowlisted blocked platforms (Reddit first), tunnel mode preferred. -4. Configured `outbound_proxy_url` if the admin has one. -5. rsearch (existing, includes its own Jina fallback). -6. `FabricUnavailable` with a reason code; the caller reports "social results unavailable", never invents any. - -Integration points, one commit each: -- `services/jobs/deepsearch/crawl.py`: `search_queries` gains a social bucket merged by the existing `_interleave`; `HOSTILE_DOMAINS` stops being a static skip list and becomes `fabric.can_serve(url)`: a hostile URL is skipped only when the fabric cannot serve it. `fetch_page` routes those URLs through `NodeFabric.fetch`. The `rsearch` NDJSON ledger frame gets a sibling `fabric` frame so `DeepsearchService._run_worker` meters it like `record_external`. -- Devii: tools `social_search`, `fetch_social_page` (`requires_auth=True`, metered, guests refused), plus owner tools `node_status`, `node_settings_get`, `node_settings_set`, `node_device_list`, `node_device_revoke` (`CONFIRM_REQUIRED` + declared `confirm` param, per the rule). Update `SYSTEM_PROMPT` so the model prefers platform data, then `social_search` only when asked about outside discussion, and never claims a platform it cannot reach. -- `POST /nodes/internal/fetch`: the stable interface for the pravda.education agent system and any external caller. Auth: API key / Bearer, rate-limited, per-owner daily quota, documented. -- Cost and fairness: every fabric call is metered into a `fabric_usage` view per caller (requests, bytes, cache hit ratio) so a single caller cannot drain the donor pool; caller quota is checked before any donor is touched. - -## 7. Clients - -All clients speak the same protocol from `spec.json`. Conformance is tested against the Python reference client (Phase 10), not by eyeballing. - -### 7.1 Native node (D2: Go) -- Single static binary, linux/amd64, linux/arm64, darwin/arm64, darwin/amd64, windows/amd64. Tunnel mode (TCP relay) and fetch mode. -- Config file + flags; `devplace-node login` takes the one-time device secret; `devplace-node status`; `devplace-node run`. -- Reconnect with full-jitter backoff, 20 s ping, local bandwidth/schedule/concurrency enforcement, battery-aware pause on laptops, quiet by default, structured log with a per-day totals line. -- Packaging: systemd **user** unit (`loginctl enable-linger` documented), Windows Service/scheduled task, macOS LaunchAgent, and a Docker/Podman compose snippet (the repo already ships `docker-compose.podman.yml`, so container users are a known audience). -- Built reproducibly from a `nodes/client/` directory with a `make node-client` target; artifacts + sha256 land in `DATA_DIR/node_clients/`; the version manifest is generated, not hand-written. - -### 7.2 Browser extension (MV3, Chrome + Firefox) -- Fetch mode only. Service worker holds the WS; 20 s keepalive ping plus a `chrome.alarms` reconnect backstop; state in `chrome.storage.session`. -- `fetch()` with `credentials: "omit"`, header rewriting via `declarativeNetRequest` for the headers the spec allows, allowlist enforced before every request, response size cap. -- Honest, prominent disclosure on first run: what is shared, what is not, the on/off switch, live counters. -- Distribution per D3. - -### 7.3 Later (explicitly out of this plan's first release) -Tauri tray app, signed auto-updates, mobile. Mobile stays out: iOS forbids the background behaviour and Android is restricted, as the research notes. - -## 8. Frontend - -Rules: responsive down to small phones (a wide button group becomes a dropdown), ES6 modules one class per file instantiated on `app`, tokens only in CSS, no literals, no `var()` fallbacks, `static_url`/`assetUrl` for assets, dates via `local_dt`, no `data-render` on server content, presence/pubsub via the existing managers. - -1. **Profile tab "Donate"** (owner-only; others see only a donor badge and lifetime totals): - - plain-language explainer first ("What you share, what you never share"), the consent toggle second, everything else only after consent. Product stays simple for the member: one switch, three sliders, a download card. - - bandwidth per day, concurrent connections, active hours, platform checkboxes, pause-on-battery. - - devices list with last seen, country/ASN class, bytes today, rotate and remove. - - **"Donated by you" card**: requests served, data transferred (up/down), first/last donated, this month vs lifetime. Server-counted only. - - download card: per-OS choice rendered as a dropdown on narrow screens, checksums shown, copy-paste install lines (systemd, docker), extension zip. - - live numbers via a private pub/sub topic `user.{uid}.nodes` published by the lock-owner relay (add one row to `LiveViewRelayService.VIEWS`, re-checking the owner in the compute callable like the workspace topic), with a 30 s HTTP fallback poll. -2. Files: template `templates/_node_donation.html` partial included by `profile.html`; `static/js/NodeDonation.js` (`app.nodeDonation`); CSS `static/css/nodes.css` using existing tokens; reuse `Http`, `Poller`, `Tabs.js`, `dp-dialog` for confirmations. -3. **Public surfaces**: landing/footer line "The community has donated X GB across Y requests" from `GET /nodes/stats` (aggregate only), a `Top donors` section on the leaderboard (opt-in display), donor badge via `_award_badge.html` / `Avatar.badgeElement` reuse. -4. Admin: `/admin/nodes` (templates `admin_nodes.html`): pool size by country/ASN class, per-platform success rate, queue depth, flags, suspend/unsuspend, caps, allowlist, kill switch; live via `admin.nodes` topic row in `VIEWS`; `ServiceMonitor` already covers the service tab. - -## 9. Gamification, incentives and anti-fraud - -- `ACHIEVEMENTS`: `node.first_donation`, `node.1gb`, `node.10gb`, `node.100gb`, `node.30_days`, in `BADGE_CATALOG` group "Community". -- XP is awarded only for server-verified successful jobs, with a hard daily XP cap per donor (`node_donors.xp_awarded_today`), so synthetic traffic cannot farm progression. Through `award_rewards` only (never wrapped in `background.submit`). -- Counters are written server-side from what the server actually relayed; the donor's self-reported `usage` frame is advisory and is compared, never trusted. -- Self-dealing guard: a donor's own account cannot be the caller whose jobs are assigned to that donor's nodes for reward purposes. -- No money, no coins, no random rewards (D4). - -## 10. Audit, privacy, safety, compliance - -1. **Audit events** (`events.md` + `category_for` + recorder at the mutation point, via the `audit-event` skill): `node.consent.grant`, `node.consent.withdraw`, `node.device.create`, `node.device.rotate`, `node.device.delete`, `node.settings.update`, `node.suspend`, `node.unsuspend`, `node.flag.resolve`, `node.allowlist.update`, `node.killswitch`. Connects/disconnects/jobs are metrics and `node_events`, not audit rows (volume). -2. **Privacy (this is a new category of personal data plus a third-party disclosure, so it ships WITH the feature)**: update `templates/docs/privacy.html` and the app-store privacy declarations. State: IP address and approximate location are processed; only a salted hash and coarse country/ASN class are stored; retention periods for `node_events`/hashes; what the donor can see (destination host, and in fetch mode the URL and body); that the platform's content passes through the donor machine; the explicit statement that the donor's cookies and accounts are never used. Plain-English donor terms page, ISP-terms warning, consent text, and a visible off switch. -3. **App-store conditionals** (root rule): no payment path, no purchasable randomness, no cross-app tracking introduced. Hide the donate UI inside any app-store shell build. Verify against current store policy before submission rather than assuming. -4. **Abuse response**: report a node/destination path, admin suspend within one tick, allowlist narrowing, per-destination circuit breaker, the kill switch, and a documented contact. Because the allowlist is destination-restricted there is no path for credential stuffing, ad fraud or arbitrary traffic through a donor. -5. **Maturity/moderation**: fetched third-party content is untrusted. It goes through `render_content` (server) or `DOMPurify` (client) like everything else and through the existing content filter when it is persisted anywhere. Fabric results are never rendered as raw HTML. - -## 11. Phased delivery - -Each phase ends with: clean import, per-language checks, em-dash scan (character and entity), `pyflakes`/`ruff` on touched files, the four-layer verification where listed, the full `make test` (all three tiers), then commit (hook auto-bumps the version), deploy with `make docker-reload`, and verification live on both hostnames. A change that is not running is not done. - -**Phase 0: Measure before building (no production code).** Scratch scripts against temp DB only. -- Real-machine experiment: a prototype WS relay (use `pywssocks` / the `residential-proxy-aggregator` design as reference) with one residential node and one datacenter node. Measure Reddit `.json` and `old.reddit.com` success rate per IP class, rate-limit thresholds per IP, X permalink/oEmbed/syndication behaviour without login, Instagram/LinkedIn/Facebook/TikTok reality. Output: a measured per-platform viability table committed to the plan. Claims without a number stay labelled "unmeasured". -- Tunnel-mode feasibility: confirm `curl_cffi` through an HTTP-CONNECT bridge preserves the Chrome fingerprint end to end and that the cleartext-HTTP/2 gotcha in the repo does not apply (https only). -- Capacity: simulated-node harness (our own, outside `locust`) for 100/500/2000 sockets on one worker: memory, CPU, frame latency, dispatcher tick cost against SQLite. These numbers set or replace the caps in section 3. -- Reconnect/idle behaviour through BOTH edges: `pravda.education` (molohttp) and `devplace.net` (front host + SSH tunnel). Measure the actual idle timeout of each. -- Exit criteria: D1, D2, D3, D5, D6 decided; viability table written; caps confirmed or changed. - -**Phase 1: Specification and contracts.** `spec.py`, `/nodes/spec.json`, docs protocol page skeleton, contract-test scaffolding in `tests/unit/services/nodes/`. - -**Phase 2: Data layer.** Tables, indexes, helpers in `database/nodes.py`, `CONSENT_KINDS` entry, `OWNED_TABLES`, `UNREPORTABLE_TABLES` with reasons, soft-delete registrations, settings seeds, `UsageLedger` with atomic upserts. Four-layer verification applies here (counters + daily caps are a spendable resource): property checks on cap arithmetic, stateful fuzz of increment/rollup/prune with invariants (totals never decrease, daily never exceeds cap), real multi-process race on the daily-cap `UPDATE ... WHERE` using `COALESCE(col, 0)` and a freshly-created unset row (not pre-zeroed). - -**Phase 3: Registry, session, consent gate, kill switch.** `NodeRegistry`, `NodeSession`, `frames.py`, `ConsentGate`, `nodes_enabled` kill switch (built and tested first). Routers for devices/ticket/WS handshake only. No job traffic yet. nginx upgrade location. Reuse of the ticket mechanism by adding a `scope` column to `ws_tickets` (default `messages`) instead of a parallel table. - -**Phase 4: Broker, scheduler, dispatcher, policy, circuit breakers, cache.** Job lifecycle `queued -> assigned -> done|failed|expired`, fairness, per-domain limits, queue cap with 429. Four-layer verification: property tests on scheduler fairness and cap enforcement; stateful fuzz of connect/disconnect/assign/complete/expire with invariants (a job is assigned to at most one node, a node never exceeds its concurrency, a withdrawn consent is never assigned work); real-process concurrency test of two dispatchers racing one slot (must be impossible by construction, since only the lock owner dispatches, and proven anyway). - -**Phase 5: Python reference client + test harness.** A small reference node used by the api/e2e tests and as the protocol conformance oracle. Includes fault injection: dropped frames, slow nodes, oversize bodies, forged bodies, DNS-rebinding attempt, private-IP target. - -**Phase 6A: Direct adapters** (no donors): Hacker News, Bluesky, Mastodon, Lobsters, StackExchange, GitHub, dev.to into `NodeFabric.social_search`, DeepSearch and the Devii tools. This ships user value before any donor exists. -**Phase 6B: Node-backed adapters**: Reddit first (per the Phase 0 table), then each additional platform only if Phase 0 measured it viable. `NodeVerifier` redundancy sampling for fetch mode. - -**Phase 7: Native client (Go) + packaging** (systemd user unit, launchd, Windows service, compose). Cross-compile target, checksums, version manifest, conformance run against the reference oracle. - -**Phase 8: Browser extension.** MV3 Chrome + Firefox build, disclosure screen, keepalive/alarms, conformance run, sideload zip served from `/nodes/download`. - -**Phase 9: Frontend and admin.** Profile Donate tab, donated-by-you card, devices, downloads, live topic, leaderboard section, landing stat, `/admin/nodes`. Screenshot-verify with the `screenshot` skill at phone and desktop widths. - -**Phase 10: Devii, docs, SEO, CLI.** -- Devii actions in `actions/catalog/nodes.py` (owner tools above), dispatcher/registry wiring, `CONFIRM_REQUIRED` for revoke, `requires_auth` matched to route guards, tool-group registration so the admin tool-prefs UI lists them. -- `docs_api/groups/nodes.py` `endpoint()` entries with `sample_response`; `DOCS_PAGES` for `nodes.html` and `nodes-protocol.html`; role-gating correct (protocol page admin/developer tier). -- SEO: Donate tab `noindex`; any public stats page indexable and in the sitemap. -- CLI: `devplace nodes list|stats|suspend|unsuspend|prune|clear`, destructive commands support `--dry-run` per the existing `accounts prune` convention (flag the conflict with the global dry-run-by-default rule to you if you want the default flipped; consistency with the existing CLI is the project rule). -- README, `devplacepy/services/nodes/CLAUDE.md`, a row in the root `CLAUDE.md` Subsystem map (a genuinely new architectural area), `locustfile.py` untouched for the WS (locust-maintainer: the node socket is deliberately excluded and the HTTP read routes get weighted tasks). - -**Phase 11: Hardening and launch gate.** -- Legal review gate: platform ToS exposure per platform, donor terms, ISP warning text, jurisdiction. Launch is blocked on a recorded decision. -- Load and soak: 24 h soak with simulated nodes, reconnect storms (all sockets drop at once, then return), full-disk and DB-lock behaviour, worker restart mid-job. Numbers replace assumptions in `CLAUDE.md`. -- Security-maintainer pass (allowlist bypass, SSRF on the server's own fetch path, ticket replay, device secret leakage, admin seniority on node moderation). -- Full maintenance fleet (`fleet` skill): audit, background, devii, docs, dry, fanout, frontend, locust, security, seo, style, test dimensions. -- Rollout: `nodes_enabled=0` by default. Stage 1: your own machines only (including a residential line) for a week, measuring success rate, latency, donor-side CPU and bandwidth. Stage 2: invite-only developers. Stage 3: open, still behind per-caller quotas. The kill switch stays one click away. - -**Phase 12: Documentation of record.** `devplacepy/services/nodes/CLAUDE.md` states what the system does today, with gaps listed next to the code that will close them (no roadmap cosplay). Measured numbers only, each labelled with the machine it came from. - -## 12. Test plan (by tier, mirroring the path) - -- `tests/unit/services/nodes/`: spec invariants and contract (docs, clients, server agree), frame codec, `DestinationPolicy` (allowlist, ports, private IPs, redirects, DNS rebinding), scheduler (fairness, per-domain interval, concurrency, bandwidth caps), circuit breaker, cache bounds and LRU, ledger math, ASN classification, XP daily cap, adapters (parse recorded fixtures, malformed input, forged-structure rejection). -- `tests/api/nodes/` (mirror the URL path): consent gating on every entry point, device create/rotate/revoke, secret shown once, ticket single-use and expiry, WS handshake with the reference client, full job round trip, withdrawal kicks the socket within a tick, kill switch, daily quotas, 429 paths, account deletion cascade, JSON and HTML faces for every route, admin seniority, `/nodes/stats` leaks no per-user data, `/nodes/internal/fetch` auth. -- `tests/e2e/profile/` (Playwright, `wait_until="domcontentloaded"`): enable flow with consent, device creation and one-time secret, settings persistence, live counter update, withdraw flow, responsive layout at phone width, owner-only visibility. Any test flipping a global setting restores it in `try/finally`. -- Rigorous verification scripts (disposable, temp DB via `DEVPLACE_DATABASE_URL`/`DEVPLACE_DATA_DIR`, never `data/`): the four layers from root `CLAUDE.md` for the ledger, scheduler and job state machine. -- The test hook that guards the production DB applies: no test or script touches `data/devplace.db`. - -## 13. Risks, stated plainly - -| Risk | Likelihood | Mitigation | Residual | -|------|-----------|------------|----------| -| Platform ToS/legal action against scraping | real | allowlist, legal gate, per-domain rate limits, kill switch | cannot be engineered away | -| Donors' IPs get rate-limited or banned by platforms | high on Reddit | per-donor-per-domain caps, caching, consent text says so | some donors will be unhappy; transparent counters help | -| X/Instagram/LinkedIn/TikTok unreachable (login walls) | high | Phase 0 measures; promise only what is measured | those platforms stay limited | -| Fetch-mode donor forges content | possible | tunnel preferred, structural validation, sampled redundancy, provenance labels, suspension | extension donors are lower trust, labelled as such | -| Small donor pool means flaky results | high at launch | cache, fallback chain, honest "unavailable" | search stays at today's quality, not worse | -| Single worker holds all sockets | by design | cap, fast reconnect, measured capacity | brief drop on worker crash | -| Donor privacy incident | low-impact by design | no cookies, hashed IP, destination allowlist, docs | disclosure must stay accurate | -| Scope creep into an open proxy | tempting | the "no open proxy" rule is in `spec.py` and the contract test | needs reviewer discipline | - -## 14. Open questions for you - -- Q1: Confirm D1 to D6 above (a one-line answer each is enough). -- Q2: Is there any residential line you control for the Phase 0 measurement, or do we recruit three donors for it? -- Q3: What does the pravda.education agent system need from search (volume, latency, which platforms, guest access)? It sets the daily quotas and the `/nodes/internal/fetch` contract. -- Q4: Should donor identity be public (leaderboard, badge) by default or opt-in? The plan assumes opt-in display. -- Q5: Who owns the legal review, and is a launch without it acceptable for the invite-only stage? diff --git a/pyproject.toml b/pyproject.toml index eeb2c175..692b9b7a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "devplacepy" -version = "1.1.5" +version = "1.1.6" description = "DevPlace - The Developer Social Network" requires-python = ">=3.12" dependencies = [ diff --git a/quality_project.md b/quality_project.md deleted file mode 100644 index 4da7e569..00000000 --- a/quality_project.md +++ /dev/null @@ -1,199 +0,0 @@ -<!-- doc: quality_project.md | version: 1.0.0 | updated: 2026-10-11 --> -# Quality Project Playbook: What Belongs in AGENTS.md - -Generic companion to `quality_software.md` and `quality_versioning.md`. -It defines what an agent instruction file must contain so that every -agent (and every human) develops the same way and the project stays -consistent and current. - -Naming note: this repo uses `CLAUDE.md` instead of `AGENTS.md`, but -the system is what matters, not the filename. Three layers, loaded -by scope: - -- Personal base rules (DevPlace: `~/.claude/CLAUDE.md`). Applies to - every project for one operator: safety, proof, craft, ops habits. -- Root project file (DevPlace: `CLAUDE.md`, ~440 lines). Applies to - the whole repo regardless of area touched. Always loaded, so it - stays proportional to global rules only. -- Nested subsystem files (DevPlace: ~30 `CLAUDE.md` files next to - the code). Loaded only when a file under that directory is read - or edited. Deep mechanics, helpers, pitfalls, worked examples. - -If your tooling reads `AGENTS.md`, use that name with the identical -three-layer split. Never merge all three into one file: the root -would bloat past usefulness and agents would stop reading it. - -## 1. Personal base rules (cross-project, rarely changes) - -One short file per operator, ported across projects. Contents: - -- Control over cleverness: hard caps on everything unbounded, - floors automated processes may never cross, static ceilings over - autotuning, fewer dependencies. -- Safety: destructive ops dry-run by default and explicitly - confirmed, loud skips over silent corruption, verify-before-write, - deletes that reclaim (compaction in the same change). -- Proof: measured claims only, real-path testing with failure - injection, suite after behavior change, live verification after - deploy. -- Craft: exact minimal diffs, boring technology, research before - load-bearing decisions, docs describing today with gaps marked. -- Communication: direct and factual, disagree with data, fixed - style bans (DevPlace: no em dashes anywhere). -- Operations: commit, deploy, verify in the same session; - retention and checks run inside the system on loops. - -This layer never names project paths or stack details. When it -conflicts with a project file, the project file wins. - -## 2. Root project file (global rules, always loaded) - -Ordered contents, each earning its place by applying to every area: - -1. Project identity in one paragraph: what it is, stack in one - line per layer, ports, key constraints (username rules, - password rules, data layout). An agent must orient in 30 seconds. -2. Commands table: install, dev, prod, every test target, load - test, lint, clean. Exact commands, no prose alternatives. Which - interpreter each target uses. -3. Single-test invocation and the failure-triage order - (cheapest-first), so agents debug efficiently instead of - re-running the slowest suite. -4. CLI reference for management commands (one line each), so agents - use named commands instead of inventing shell one-liners. -5. Runtime data layout: the single data root, the path registry, - and the rule that no module computes a path from scratch. -6. Environment variables table: name, default, purpose. Complete; - an undocumented env var is a trap. -7. Subsystem map: table of every nested doc with one-line scope, so - agents know which file to load (and that it exists). -8. Architecture: request pipeline, routing layout, templates and - frontend rules, content pipeline, auth, database, background - work. One paragraph each with pointers down, not full detail. -9. Hard rules, each marked as such: soft delete, reportability, - production-DB protection, confirmation gates, changelog/doc - duties. A hard rule states the rule, the mechanism enforcing it, - and what fails if violated (usually a named test). -10. Conventions: naming, validation style, response style, - ownership checks, date formats, slug/ID handling, role storage. - Each convention shows the one correct shape. -11. Testing: tier definitions, directory mirroring rule, fixture - stack, required patterns, full-suite gate. Pitfalls live in the - nested test doc; the root states the gate. -12. Feature workflow: the ordered step list (understand, data, - server, view, safety, agent+docs, document, validate) with the - blocking rule: a failed step blocks the rest. -13. CI/CD: what runs on push/PR, merge gate, DTAP stages, deploy - then verify. - -Size guard: if the root passes ~500 lines, detail has leaked in -that belongs in a nested file. Split it out. - -## 3. Nested subsystem files (deep detail, loaded on touch) - -One per directory that owns a mechanic (DevPlace: routers, -services, database, static/js, static/css, templates, tests). -Contents: - -- Scope line: which paths it covers, in the first line. -- The mechanic: helpers, entrypoints, and patterns with exact - symbol names (`conditional_update_farm`, `respond()`, - `get_users_by_uids`), never paraphrased APIs. -- The closed sets: registries, token scales, breakpoint lists, - event catalogues. An agent must be able to check membership. -- CORRECT/WRONG pairs for the two or three mistakes every newcomer - makes (DevPlace tests doc: scoped vs unscoped selectors, waits - vs sleeps). Show code, not advice. -- Common pitfalls section: the real bugs this area produced, each - with cause and fix. Pitfalls are bug reports that pay rent. -- Worked examples: one end-to-end change with the file count it - touched, so agents calibrate "one feature = N files". -- Backwards-compatibility notes where the area promises them - (what may never break, what the migration path is). - -A nested file never repeats the root. It assumes the root was read -and only adds what the root deliberately omitted. - -## 4. Severity levels: hard rule, convention, guidance - -Agents follow what they can rank. Mark every statement: - -- Hard rule: enforced by code, hook, or failing test. Violation - breaks the suite or is denied by tooling. Example: every removal - is a soft delete; production DB access needs written confirmation - plus a token; new user-visible tables must be classified - reportable or private with a reason. -- Convention: one correct shape, enforced by review. Example: - response helpers, date formats, commit message shape, no second - implementation of an existing helper. -- Guidance: preferred approach with stated tradeoffs. Example: - boring technology, static ceilings over autotuning, research - before load-bearing decisions. - -Write the enforcement next to the rule ("fails in -`tests/unit/database/moderation.py`", "denied by the pre-tool hook"). -A rule without a named enforcer is a wish. - -## 5. Keeping the docs updated (the docs are code) - -- Same-change rule: the doc update ships in the commit that changes - the behavior. Docs describing yesterday are worse than missing - docs because agents trust them. -- Landing rule: new global rule goes to the root, new mechanic or - pitfall to the nested file, new user-visible behavior to README - plus the user docs site. The feature workflow names this step so - it is never skipped. -- Measured, not remembered: counts, diagrams, and claims derive - from the running app and schema (DevPlace architecture atlas is - generated from the imported app object and live schema, and - records where docs and code disagree). -- No fiction: docs state what the system does today, gaps listed - next to the code that will close them. No roadmap prose, no - aspirational architecture. -- Versioned per `quality_versioning.md` section 4: header comment, - history table, bump in the same commit. All three layers count as - markdown and follow the scheme. -- Deletion duty: removing a feature removes its doc lines in the - same commit. Stale sections are found by searching for the - removed symbol, not by rereading. - -## 6. Anti-patterns: what does NOT belong - -- Tutorial prose or onboarding narrative. Agent docs are reference, - not a course. One example per pattern, then the rule. -- Duplicated detail in two layers. If root and nested both explain - it, one will rot. Root states the rule and points down. -- Unbounded lists ("all endpoints", "all settings") pasted by hand. - Point at the registry, the table, or the command that lists them. -- Aspirational rules nobody enforces. Each release, delete one rule - the suite does not check and no review ever cites. -- Secrets, credentials, customer data, or production hostnames. - Env var names yes, values never. -- Style debates or option-weighing. Decide with an opinion, state - the tradeoff in one line, move on. - -## 7. Replication checklist for your hosting site - -1. Personal base file: safety, proof, craft, ops. Shared across - your projects, project file wins on conflict. -2. Root `AGENTS.md`: identity, commands, triage order, CLI, data - layout, env table, subsystem map, architecture pointers, hard - rules with enforcers, conventions, testing gate, workflow, CI/CD. - Under ~500 lines. -3. Nested files per area: `billing/`, `provisioning/`, - `support/`, `status/`, plus `tests/` pitfalls and - `templates/` partials. Scope line first, symbols exact, - CORRECT/WRONG pairs, pitfalls from real bugs. -4. Severity marks on every statement plus the named enforcement - (test path, hook name, review gate). -5. Same-change doc updates, landing rule, measured claims, no - fiction, version headers on all three layers, deletion duty. -6. Quarterly audit: run the suite, grep each hard rule for its - enforcer, delete one unenforced rule, regenerate the - architecture counts. - -## Doc history - -| Version | Date | Change | -|---------|------|--------| -| 1.0.0 | 2026-10-11 | Created from repo research: three-layer CLAUDE.md system, root/nested/personal contents, severity levels, update duties, anti-patterns. | diff --git a/quality_software.md b/quality_software.md deleted file mode 100644 index b597537a..00000000 --- a/quality_software.md +++ /dev/null @@ -1,516 +0,0 @@ -<!-- doc: quality_software.md | version: 1.2.1 | updated: 2026-10-11 --> -# Quality Software Playbook: How to Build a Site as Good as DevPlace - -This is a generic, reusable method distilled from how DevPlace was built. -It is written so you can apply it to a completely different site, for -example a hosting company (accounts, products, orders, provisioning, -support tickets, status pages), and get the same stability and quality. - -It is not about copying features. It is about copying structure, -constraints, and verification habits. - -## 1. Core philosophy - -1. Boring technology, smart edges. One backend language, one database - file, server-rendered HTML, plain JavaScript. Put complexity where - you can see it (request handlers, templates), keep storage dumb. -2. Fewer moving parts. Every dependency is a way to fail. Prefer the - standard library, one process type, one data root. -3. Server-rendered first. The page works without JavaScript. JavaScript - only upgrades live behavior (polling, websockets, optimistic UI). -4. One data source, many faces. Every domain object is served as HTML - for browsers, JSON for API clients, an agent/tool action for - automation, and a documented endpoint. Design the data once, then - expose all four. -5. Exact, minimal diffs. Edit existing files, reuse existing helpers, - match existing visual structure. Zero tolerance for a second way - of doing something that already has one way. -6. Measure, do not theorize. Performance claims come from runs on real - machines. If there is no measurement, say so. -7. Loud skips beat silent corruption. Reject unverifiable input with a - clear reason and a counter. Never store what you cannot vouch for. - -Hosting example: a VPS order is one row in one table. The same handler -renders the order page (HTML), returns the order (JSON), exposes a -`order_create` automation action, and is documented in your API docs. - -## 2. Reference stack (proven shape) - -You can swap brands, but keep the shape: - -- Backend: one typed web framework (DevPlace: Python + FastAPI). -- Templates: server-side templates with one shared instance - (DevPlace: Jinja2, single `templates` object imported everywhere). -- Frontend: no framework, no build step. Plain ES modules, one class - per file, small shared utilities (`Http`, `Poller`, modal, toast). -- Database: one embedded relational file with WAL mode for small to - medium scale (DevPlace: SQLite via `dataset`, WAL, busy timeout, - mmap). Synchronous local calls are fine when they are - sub-millisecond; do not make the data layer async for fashion. -- Auth: session cookie for browsers plus API key and Basic for - machines, resolved in one `get_current_user` choke point. No JWT - unless you need third-party verification. -- Static assets: content-addressed or boot-versioned URLs served - immutable (`/static/v<ts>/...`), one helper builds every URL. -- Background work: one in-process queue with inline fallback so tests - stay deterministic and production stays fast. - -Hosting example: keep customers, services, invoices, tickets in one -SQLite/Postgres schema behind one data module. Serve the control -panel as HTML, the reseller API as JSON from the same handlers. - -## 3. Repository layout that scales - -Mirror URLs with code paths, one segment per directory: - -- `routers/` mirrors the URL tree. Flat file for one resource - (`feed.py`), package directory for sub-resources (`projects/`). - Each leaf owns an `APIRouter`, the package `__init__.py` aggregates. -- `templates/` mirrors pages, extends one `base.html`. Page CSS in - `extra_head`, page JS in `extra_js`. Shared partials for repeated - UI (avatar link, user link, pagination, report button). -- `static/js/` mirrors behavior, one class per file, instantiated as - `app`. Shared utilities live once and are reused, never forked. -- `static/css/` uses design tokens, no literal colors in page files, - one closed breakpoint set, one z-index scale. -- `services/` holds background services behind one base class and one - manager (start/stop/status/logs). Each service owns its tables and - its admin controls. -- `tests/` mirrors the same tree by path: `tests/api/...` for HTTP, - `tests/e2e/...` for browser, `tests/unit/...` for pure logic. - -Rule: if you cannot guess the file from the URL, the layout is wrong. - -## 4. Single data root and single config module - -All runtime artifacts live under one root (`DATA_DIR`), every path -derived from it in one config module, created at boot: - -- Database file, uploads, attachments, generated archives, keys, - locks, backups, service state. -- Stored DB values are relative (`directory` + `stored_name`), so a - layout move never needs a DB rewrite. -- One `DATA_PATHS` registry lists every path; one `ensure_data_dirs()` - creates them before anything writes. - -Hosting example: `data/` holds `app.db`, `uploads/`, `invoices/`, -`backups/`, `locks/`, `keys/`. Point `DATA_DIR` at a volume in prod. - -## 5. Request pipeline (same for every route) - -1. Single app factory: mount `/static`, register routers, install - middleware in fixed order (timing outermost, then security - headers, rate limit, maintenance gate, error handlers). -2. One auth resolution: cookie, API key header, Bearer, Basic, - all in `get_current_user`. Guards: public read - (`get_current_user`), member write (`require_user`), admin - (`require_admin`). -3. One response helper (`respond()`): same context renders HTML for - browsers and JSON for `Accept: application/json` via a response - schema. A context key missing from the schema is dropped from - JSON by design, so schemas are explicit contracts. -4. One validation style: Pydantic form models bound from forms, - one global validation-error handler (re-render or redirect). -5. One timing mechanism: outermost middleware stamps start time and - sets `X-Response-Time`. Never re-time per route. -6. Rate limit only mutations (`POST/PUT/PATCH/DELETE`), keyed by - real IP, tunable from settings. Reads stay fast and exempt. - -## 6. Data conventions that prevent entire bug classes - -- Primary keys are opaque string IDs (`uid`), slugs accept slug or - bare ID via one resolver. Slugs embed the random tail of the ID, - never a timestamp prefix (prefix sharding funnels writes). -- Soft delete everywhere user content is removable: `deleted_at` + - `deleted_by`, live rows are `deleted_at IS NULL`. Every insert - writes both columns, every list filters them. Only garbage - collection hard-deletes. One admin Trash restores/purges by event. -- Ownership rule: edits are owner-only, deletes are owner-or-admin, - checked by one helper. Admin seniority: junior admins cannot act - on senior admins. -- No N+1: batch helpers (`get_users_by_uids`, counts by ID lists, - paginators) are the only way to load related rows in lists. -- Idempotent schema init: ensure full column set before indexes, - because auto-schema tools create partial tables from first insert. -- Toggles revive, never duplicate: look up ignoring deleted flag, - stamp on off, clear on on, under one shared stamp for cascades. - -Hosting example: canceling a server is a soft delete with one stamp -covering service, DNS rows, and scheduled jobs, restorable from Trash -until GC purges it. - -## 7. Frontend rules that keep UI consistent - -- One base template owns nav, head, modals, toasts, timing badge. - Nav active state is one global, never inline path checks. -- Never hardcode `/static/...`. One helper builds versioned URLs in - templates and one in JS. -- Server renders content that exists at request time (SEO, no-JS, - speed). Client pipeline is only for live content (chat bubbles, - streamed results), always sanitized fail-closed. -- Dates stored as UTC ISO, rendered as `<time data-dt>` and - reformatted to viewer timezone client-side, one format for users - (DevPlace: DD/MM/YYYY). -- Modals, lightbox, toasts, floating windows, scroll memory: one - implementation each, reused by class name. No inline JS. - -## 8. Background jobs and services (one pattern) - -- One generic fire-and-forget queue: `submit(fn, *args)`. When the - consumer runs, it defers; when it does not (tests, full queue), - it runs inline so behavior stays deterministic. -- One base service class: `run` loop, supervised start/stop, status, - logs, settings. One manager registers all services. -- One async-job pattern for slow work (archives, audits, reports): - enqueue (`POST .../run`), poll (`GET .../{uid}`), read report - (`GET .../{uid}/report`), expire and prune by CLI. -- Multi-worker safety: schema work behind one init lock, background - services elected by one service lock so exactly one worker owns - them. Never assume single-process in job code. - -Hosting example: provisioning, backups, SSL renewal, and usage -metering are all job services with run/poll/report plus -`cli <domain> prune` retention commands. - -## 9. Testing: three tiers, one tree, no skips - -- Tiers by fixture, not by folder debate: - - `unit`: pure in-process, no HTTP, no browser. - - `api`: HTTP integration against a live test server + temp DB. - - `e2e`: Playwright browser, real user flows. -- Test server runs on its own port with its own temp DB and data - dir, background services disabled. One serial process, no - parallelism, so results are stable. -- Every run reports every failure (`-rf`), triage cheapest first: - unit+api (~minutes), then browser tier, then `--last-failed` - loop until empty. -- Required browser habits: explicit `domcontentloaded` on - navigations, visible-state waits over sleeps, scoped selectors, - global settings restored in `try/finally`. -- Gate before suite: clean app import, per-language syntax check, - linter on touched files (catches missing imports that compile - checks miss), no forbidden characters or patterns. - -Quality bar: the full suite (all three tiers, every test) passes -before a change is done. A red test covering your change is the -requirement, not a stale artifact. Never narrow a run to make it -pass. - -## 10. Extra verification for money, state, and races - -Example-based tests are blind to wrong formulas at untried inputs, -long-sequence invariant breaks, and races. For spendable resources -(credits, coins, quotas), bounded state machines (tiers, levels), -or read-then-write mutations reachable from two requests, add four -disposable layers against a temp DB (not committed tests unless -asked): - -1. Property checks over the full input domain (monotonicity, - bounds, non-negativity) across hundreds of values. -2. Stateful fuzzing: randomized action sequences across many - entities, invariants re-checked after every action. -3. Real-process concurrency: separate OS processes (not threads), - fresh production-like rows (never pre-zero NULL columns), - exact success counts and exact final totals. Fix races with one - atomic conditional `UPDATE ... WHERE <precondition>` checked by - real `rowcount` at the chokepoint. -4. Static analysis on every touched file (`pyflakes`/`ruff`), which - catches runtime `NameError` paths the other layers may miss. - -Hosting example: apply this to credit spend, plan upgrades, -affiliate payouts, and concurrent provisioning of the last IP or -stock-limited node. - -## 11. Operations: deploy, verify, prune - -- One-command environments: `install`, `dev` (reload, one worker), - `prod` (multi-worker, no template reload, real backlog), `test*`, - `coverage`, load test. Every Python target uses the project venv. -- No bytecode in repo: `PYTHONDONTWRITEBYTECODE=1` for all targets, - `clean` removes venv and stray artifacts. -- CI runs the full suite serially on every push/PR, publishes - coverage and failure screenshots. CI green before merge. -- Production DB is never touched without explicit written - confirmation plus a two-factor token; disposable scripts always - target a scratch DB via env override. -- Retention is a feature: every expirable domain gets - `prune` (expired only) and `clear` (everything) CLI commands, - wired to the same GC the server uses. -- Backups are first-class: targets, worker, schedules, offload, - download restricted to the primary admin, prune missing files. - -## 12. Security, moderation, and trust checklist - -- Passwords hashed with a slow KDF (PBKDF2-SHA256), session tokens - random 256-bit, per-process TTL cache keyed by token. -- Ownership checked server-side on every mutation; deletes confirm - explicitly and audit; destructive automation actions require a - two-step confirm flag declared in their schema. -- Every user-visible content table is reportable by construction: - registry entry, object URL resolver, report button in the action - bar. Private-only tables are listed as unreportable with a reason, - enforced by a test that fails on unclassified tables. -- Outbound HTTP goes through one stealth/guarded client (consistent - TLS + headers + proxy + SSRF guard), never bare clients per call. -- Audit log is append-only, never raises into the caller, with one - catalogue of event keys. Admin-only read. -- Privacy docs and store declarations update in the same change as - any new personal-data collection or third-party transmission. - -Hosting example: tickets, reviews, and public server comments are -reportable; invoices and API tokens are private-by-design with -written reasons; provisioning calls go through the one guarded -client with proxy support. - -## 13. Docs, SEO, and API surface - -- Every public/auth endpoint gets prose docs plus an interactive - API entry (params, auth, sample response). Public pages get SEO - context and JSON-LD; sitemap and robots are served, not static. -- One docs registry (`DOCS_PAGES`) owns prose pages; API docs own - endpoint entries. New route without docs is incomplete. -- README stays product-facing (routes, config, env vars, features). - Nested `CLAUDE.md` files hold subsystem mechanics next to the code - so the always-loaded root doc stays small. -- Version static assets, canonical URLs per listing, breadcrumbs, - related content, and one default OG image. - -## 14. Development flow (the daily loop) - -Every feature follows the same eight steps in data-flow order, and a -failed step blocks the rest (never skip ahead): - -1. Understand. Read the router, template, matching tests, and the - subsystem doc. Trace input model to router to data helper to - response (HTML and JSON). Find a relatable existing example and - match it structurally and visually; if none exists, stop and agree - on the new pattern before building. -2. Data layer. Query/batch helpers first (never inline N+1), input - model, JSON response schema (every exposed key declared or it is - silently dropped), idempotent schema/index changes. -3. Server layer. Handler with the right guard (public/member/admin), - specific paths before catch-alls, HTML+JSON via one responder. -4. View layer. Extend base, reuse partials and globals, page CSS/JS - in the standard blocks, one ES class per file. -5. Safety layer. Reportable/private classification, ownership checks, - confirm gates, audit events, privacy-doc update if new personal - data or third-party transmission appears. -6. Agent + docs layer. Tool action per user-askable operation, - endpoint doc entry, prose page if user-facing, SEO context and - sitemap entry if indexable. This step is the most-forgotten; do - not skip it. -7. Document. Product README for user-visible change, subsystem doc - for new mechanics or pitfalls, root doc only for new global rules. -8. Validate. Clean import, per-language syntax check, linter on - touched files, full three-tier suite green (see section 9), plus - the money/state/race procedure where it applies. - -Failure triage is cheapest-first: fast tiers (unit+api, minutes) -before browser tier, then re-run only failures (`--last-failed`) -until empty. One pass reports every failure; fix the list, not one -bug per pass. - -Session rule: commit, deploy, verify in the same session, in that -order. A change that is not running in production is not done. After -deploys, verify live; green locally means nothing until prod -confirms it. - -## 15. Workflows and automation - -- One command surface. Every repeated action is a named target or CLI - command (`dev`, `prod`, `test`, `test-fast`, `test-failed`, - `coverage`, load test, `prune`, backups). No tribal shell - one-liners; if it is run twice, it gets a name. -- DTAP pipeline. Development (reload, one worker) to Test (CI serial - suite + coverage + failure screenshots, green before merge) to - production (multi-worker, no template reload, real backlog). -- Scheduled hygiene inside the system, not in heads: retention, - cleanup, quota resets, and health checks run on loops or CLI - commands with dry-run defaults and narrow scopes for destructive - ops. Destructive operations are dry-run first, explicitly - confirmed, date-guarded or allowlisted. -- Load testing is a target, not an event: scripted Locust run - against a scratch DB with fixed users, spawn rate, and run time, - producing an HTML report. -- Production guardrails are enforced, not remembered: a hook denies - commands reaching the live DB without explicit written - confirmation plus a second approval; disposable scripts always - target scratch databases via env override. - -Hosting example: `provision`, `suspend`, `backup-run`, -`ssl-renew`, `usage-meter`, `invoice-prune` are all named commands -with `--dry-run`; nightly loops handle expiry, renewal, and status -checks with human-readable pressure states. - -## 16. Agents: in-product assistant, services, and coding help - -DevPlace quality comes partly from treating agents as first-class -consumers of the same routes humans use: - -- In-product assistant (DevPlace: Devii). A conversational operator - over the real API: per-owner sessions and channels (web, terminal, - chat bridge), a tool catalog where each tool mirrors one route - with matching auth, confirmation gates on destructive tools (first - call refused, agent shows exact target, second call passes - `confirm=true`), per-user cost attribution and quotas through one - AI gateway, and usage/ledger pages per user. New feature rule: if - a user could ask the assistant to do it, it gets a tool. -- Gateway pattern. Every AI/model call flows through one internal - gateway (usage ledger, provider/model routing, per-call cost - headers), so spend is attributable and every consumer (correction, - modifier, grading, chat) shares quotas and audit. -- Background service agents. Crawlers, importers, reconcilers, and - persona bots run as supervised services under one manager with - start/stop/status/logs and admin pages (DevPlace: news importer, - bot fleet with live monitor, container reconciler, audit writer). - Exactly one worker owns them via a service lock. -- Coding-agent support. Nested subsystem docs next to the code so an - agent loads only what it touches; strict conventions (no second - implementation, named helpers, closed token/breakpoint/z-index - sets) so generated code matches; tool catalog and docs registry - so the agent can act and document through the same contracts. -- Human authority. Agents propose and execute reversible work; - destructive, cross-user, or policy-crossing actions need explicit - human confirmation, seniority checks where roles exist, and an - audit trail. When a floor is hit (retention minimum, protected - path), the agent holds and pages a human instead of deleting. - -Hosting example: a support assistant with tools like `server_reboot` -(public read tools open, mutating tools auth-gated, destructive -tools confirm-gated), all spend metered per customer through your -gateway, provisioning and renewal running as supervised services. - -## 17. Documenting: what lives where, and when - -Documentation is layered by audience and distance from the code, and -every layer updates in the same change as the code it describes: - -- Source code: no comments, no docstrings. Code is self-documenting - through names, small helpers, and one obvious way per task. A - decision worth recording goes in the change description or the - subsystem doc, never as deliberation text in the source. -- Subsystem docs (DevPlace: nested `CLAUDE.md` next to the code). - Each directory that owns a mechanic holds its own doc: helpers, - pitfalls, cross-layer wiring, worked examples. An agent or human - loads only the docs for the area being touched, so the always- - loaded cost stays flat as the project grows. -- Root doc (DevPlace: `CLAUDE.md`, `ARCHITECTURE.md`). Global rules, - commands, and architecture only. A per-feature detail never lands - here; a new global rule always does. The architecture atlas is - measured from the running app and schema (counts, diagrams), not - written from memory, and disagreements between docs and code are - recorded explicitly. -- Product README. User-visible surface: routes, config, env vars, - features, quick start. Updated when behavior a user can see - changes. -- User docs site (DevPlace: `/docs` with a `DOCS_PAGES` registry). - Prose pages per audience tier plus an interactive API reference - where every endpoint entry carries params, auth, and a sample - response. A new public or auth endpoint without a doc entry is - incomplete by definition. -- Privacy and store declarations. Any new personal-data category or - third-party transmission updates the privacy doc and platform - declarations in the same change, never as a follow-up. -- Event and reason registries. Audit event keys, report reasons, and - similar closed sets live in one catalogue each, so every client - and test renders the same list. - -Rule: docs state what the system does today, with gaps listed next -to the code that will close them. No fiction, no roadmap prose. - -Hosting example: `billing/CLAUDE.md` documents proration and retry -rules next to the billing code; the root doc only says billing -exists and which commands run it; `/docs` gains a prose page plus -one API entry per new endpoint in the same change. - -## 18. Roles, permissions, and ownership - -One small role set, enforced server-side at every surface (web UI, -REST API, and agent tools alike): - -- Roles (DevPlace: guest, member, admin, primary admin). Stored as - exact strings, tested through one `is_admin()` helper, never - hand-rolled compares. Any write to a role invalidates the cached - role lookups. -- Guests read public pages; members write their own content; - admins moderate and operate. Public reads resolve the viewer but - never require login; every write names its guard explicitly. -- Ownership first: edits are owner-only, deletes are owner-or-admin, - checked by one helper comparing resource owner to actor. Admins - can remove abuse but cannot rewrite someone else's words. -- Seniority among admins: a junior admin (later registration) cannot - mutate a more senior admin's account or content. Every per-user - admin mutation is gated by one seniority predicate, denials are - audited, and the gate is server-side so it also covers agent - tools. One primary admin (the first admin account) is the single - exception with full visibility and full control, including - private-by-admin content and restricted downloads. -- Visibility predicates are single functions, never inline checks: - e.g. `can_view(project, user)` decides private-content access in - one place (owner sees own, admins see members' private rows, but - an admin's own private rows stay visible only to that admin - except for the primary admin). Container and workspace isolation - get their own stricter predicates on the same pattern. -- Machines authenticate as users: every account holds an API key, - plus issuable scoped tokens; the same guards apply regardless of - credential type, so the REST API cannot do what the UI forbids. -- Agents act as the asking user: tool auth mirrors route guards, - destructive tools add confirmation gates, and every admin tool - passes through the same seniority and audit path as the web UI. - -Rule: if a permission check appears twice in different words, one -of them is a bug. Hoist it into the single predicate and call it -from both places. - -Hosting example: guest (pricing, status), customer (own servers and -invoices), support (view customers' servers, restart but not delete; -junior support cannot touch senior staff accounts), owner (full -access plus backup downloads and billing refunds). One -`can_view_server` / `can_manage_server` pair gates panel, API, and -assistant tools together. - -## 19. Replication checklist for your hosting site - -Use this order; it is the data-flow order DevPlace follows: - -1. Scaffold: app factory, config with one data root, shared - templates, response helper, auth choke point, middleware stack. -2. Data: users, sessions, settings, then one domain (products). - Add soft-delete columns, batch helpers, idempotent init. -3. Server: public reads first (HTML+JSON via one handler), then - member writes with guards, then admin views. -4. View: base template, tokens, partials, one JS utility set. - Server-render listings and detail pages. -5. Safety: reportable/private classification, ownership checks, - confirm gates, audit events. -6. Automation + docs: tool action per user-askable operation, - API doc entry per endpoint, SEO context per public page. -7. Verification: import gate, lint touched files, full three-tier - suite green, plus the four-layer money/state/race procedure - where it applies. -8. Ops: venv targets, CI serial suite, prune/clear CLIs, backups, - production-DB guard, deploy then verify live. - -## 20. What to copy literally vs adapt - -Copy literally: single data root, soft-delete columns and filters, -batch loaders, one response helper, one auth resolver, test tiers -and temp-DB server fixture, prune/clear per domain, audit-never- -raises, versioned static URLs. - -Adapt to hosting: table list (customers, plans, servers, invoices, -tickets, status incidents), provisioning job services, payment and -registrar integrations behind the one outbound client, usage -metering and quota rules, status page and notifications. - -If you keep the constraints and the verification loop, the domain -can change completely and the quality transfers with it. - -## Doc history - -| Version | Date | Change | -|---------|------|--------| -| 1.0.0 | 2026-10-11 | Created: philosophy, stack, layout, data, pipeline, frontend, jobs, testing, verification, ops, security, docs, checklist. | -| 1.1.0 | 2026-10-11 | Added development flow, workflows/automation, and agents sections. | -| 1.2.0 | 2026-10-11 | Added documenting and roles/permissions sections. | -| 1.2.1 | 2026-10-11 | Added doc version header and history per quality_versioning.md. | diff --git a/quality_versioning.md b/quality_versioning.md deleted file mode 100644 index 0f28ce85..00000000 --- a/quality_versioning.md +++ /dev/null @@ -1,165 +0,0 @@ -<!-- doc: quality_versioning.md | version: 1.0.0 | updated: 2026-10-11 --> -# Quality Versioning Playbook: App, Assets, Docs, and Git - -Generic companion to `quality_software.md`. It covers how DevPlace -versions its application, its static assets, and its git history, -plus the one practice DevPlace does NOT have today and you should -adopt from day one: versioned markdown docs. - -Honest baseline (researched 2026-10-11, do not assume otherwise): - -- App version: single source `pyproject.toml`, auto-bumped patch on - every commit by a checked-in pre-commit hook. Real and working. -- Asset version: composed `{APP_VERSION}-{BOOT_ID}`, stamped into - every static URL, served immutable. Real and working. -- Git: trunk `master`, remote `production` branch, per-ticket - branches, CI gate on push/PR to master. Real and working. -- Tags: none. Changelog file: none. Markdown versioning: none. - These are gaps, and sections 4 and 6 below define what to do - instead of copying the gap. - -## 1. App versioning: one source, auto-bumped - -- One file owns the version (DevPlace: `pyproject.toml` - `project.version`, currently `1.0.x`). The running app reads it - live at boot (DevPlace: `config.APP_VERSION` via `tomllib`), so - the version in code and the version on disk can never drift. -- A checked-in pre-commit hook bumps the patch number on every - commit (DevPlace: `.githooks/pre-commit`, activated by - `git config core.hooksPath .githooks` in the install target). - Skips merge commits and commits that already touch the version - line, so a manual minor/major bump is never double-bumped. -- Semver discipline: patch is automatic (every commit), minor is a - manual commit when a user-visible feature lands, major is a - manual commit on breaking change. Minor/major bumps are ordinary - commits whose message says what the release contains. -- The version is visible at runtime: stamped into a meta tag - (DevPlace: `<meta name="asset-version">` in `base.html`) so any - page view, bug report, or screenshot names the exact build. - -Hosting example: the control panel footer and every support ticket -show `v2.14.3`; support never asks "which version are you on". - -## 2. Asset versioning: cache forever, deploy instantly - -- Compose the asset version from app version plus boot id: - `STATIC_VERSION = {APP_VERSION}-{BOOT_ID}` where the boot id is a - deploy-provided marker with a wall-clock fallback. -- Stamp it into every app-owned static URL - (`/static/v<version>/...`) through one URL helper, and serve that - path immutable with a one-year `max-age`. -- Keep the reverse proxy in sync: the versioned-path location regex - must accept dots and hyphens, not digits only, or the immutable - header silently stops applying (this was a real production bug - class here, fixed once the version segment stopped being a bare - timestamp). -- Never hardcode a static path; every template and script builds - URLs through the one helper, so a version-shape change touches - one function. - -## 3. Git workflow: trunk, tickets, production - -- Trunk is `master`. All work lands there through small commits; - CI (serial full suite, coverage, failure screenshots) runs on - every push and PR to master and must be green before merge. -- One branch per ticket or task, named after the tracker - (`ticket-123`, `112-short-slug`). Short-lived: branch, commit, - merge, delete. No long-running feature branches. -- One `production` branch (or tag set, see section 6) marks what is - deployed. Development happens on master; promotion to production - is a deliberate merge, followed by deploy and live verification. -- Commit messages: imperative summary line under 72 chars, blank - line, body explaining why (not what, the diff shows what). - Optional trailers for attribution (`Co-Authored-By`) and session - links. Reverts say what behavior returned and why. -- Never rewrite published history: no amend/push-force, no rebase - of shared branches, no `reset --hard` on anything pushed. Fix the - working tree forward; a mistaken commit is corrected by a new - commit. -- Hooks live in the repo (`.githooks/`) and are wired by the - install target, so every clone gets the version bump and any - future checks with zero manual setup. - -## 4. Markdown versioning: every doc carries its version - -This is the rule DevPlace does not follow yet: **every markdown -file is versioned on every update.** Adopt it from the start; it -costs one line per edit and buys traceability for every claim a -doc makes. - -Scheme (copy literally): - -1. First line of the file is an invisible HTML comment: - `<!-- doc: <filename> | version: X.Y.Z | updated: YYYY-MM-DD -->` - It renders nothing and survives every markdown renderer. -2. Bottom of the file is a `Doc history` table with one row per - version: version, date, one-line change note. -3. Doc-semver: MAJOR for restructure or section renumbering, MINOR - for a new section or new rule, PATCH for wording, fixes, or - example updates. Every update bumps exactly one level and adds - exactly one history row. -4. The version bump and the content change ship in the same commit, - so `git log -p <doc>` and the history table always agree. -5. Cross-doc references name the section, not the version ("see - `quality_software.md` section 9"), so minor doc bumps never - cascade into other files. - -Enforcement options, cheapest first: a pre-commit check that fails -when a staged `.md` lacks a matching version-line change (same -pattern as the app-version hook, inverted: docs must be touched by -hand, so the hook only verifies); a CI grep that every tracked -`.md` starts with the `doc:` comment; a unit test listing tracked -markdown files and asserting the header parses. - -Hosting example: `docs/sla.md` at `3.1.0` and `docs/pricing.md` at -`5.0.2` evolve independently; a customer dispute cites "SLA v3.1", -not "the docs, at some point". - -## 5. What gets a version vs what gets a timestamp - -- Versioned (semver, comparable, citable): the app, each markdown - doc, API contracts, terms/SLA/pricing pages, database export - formats, agent tool catalogs. -- Timestamped (point-in-time, never compared): deploys (boot id), - backups, audit events, usage ledger rows, generated reports. -- Never versioned alone: a bare counter with no history row, a date - with no change note. A version you cannot diff is decoration. - -## 6. Releases and tags: close the gap deliberately - -DevPlace has no tags and no changelog file; promotion is tracked by -the `production` branch. For a hosting company, where customers and -auditors ask "what changed and when", close this gap: - -- Tag every production promotion with an annotated tag - (`v2.14.0`, message = release notes summary). Tags are cheap, - immutable pointers; the branch can move, the tag cannot. -- Keep one `CHANGELOG.md` (itself doc-versioned per section 4) - with one entry per minor/major release: date, tag, added, - changed, fixed, security notes. Patch releases batch into the - next minor entry; no per-commit changelog spam. -- Release flow in order: land feature commits on master (CI green), - manual minor/major bump commit, merge master into production, - tag, deploy, verify live, write the changelog entry. A release - that is not verified live is not released. - -## 7. Replication checklist for your hosting site - -1. Single version source read live by the app; pre-commit hook - auto-bumps patch; minor/major are manual release commits. -2. Asset URLs carry `{version}-{boot-id}`, served immutable; - proxy regex accepts the full version alphabet. -3. Trunk `master`, ticket branches, `production` promotion target, - CI gate on master, no history rewrites, hooks in-repo. -4. Every `.md` starts with the `doc:` version comment and ends with - a history table; every update bumps; CI verifies the header. -5. Terms, SLA, pricing, and API contracts are versioned docs with - citable versions, not wiki pages. -6. Annotated tags plus one changelog for customer-facing releases; - release = merge, tag, deploy, verify live, changelog entry. - -## Doc history - -| Version | Date | Change | -|---------|------|--------| -| 1.0.0 | 2026-10-11 | Created from repo research: app auto-bump hook, asset composition, trunk/ticket/production git model, new markdown-versioning scheme. | diff --git a/research_residental_proxies.md b/research_residental_proxies.md deleted file mode 100644 index 8dc102c0..00000000 --- a/research_residental_proxies.md +++ /dev/null @@ -1,74 +0,0 @@ -**Residential proxy donation works by having donors run a lightweight client that opens a persistent outbound tunnel (almost always WebSocket/WSS) to your central server. Your server then multiplexes incoming SOCKS5 or HTTP CONNECT requests over those tunnels so traffic exits through the donor’s residential IP.** - -This is a reverse-proxy style architecture. The client never listens on a public port; it only dials out. That is why WebSocket (or a similar long-lived bidirectional channel) is the natural transport. - -### Core client design (common to all options) -1. Client authenticates (token / account) and opens a WebSocket to your server. -2. Server treats each live WebSocket as a residential exit node. -3. When a customer requests a connection through that node, the server sends a framed “open TCP to host:port” message over the WebSocket. -4. The client opens the TCP socket locally and bidirectionally relays bytes. -5. Keep-alives, automatic reconnect with exponential backoff, and optional bandwidth/rate limits on the client side. - -Open-source references that implement exactly this pattern include `residential-proxy-aggregator` (Python WS server + client agents), `pywssocks`, and various WS-SOCKS tools. - -### Option comparison - -| Option | Reliability / Always-on | Install friction | Resource use | Store / distribution | Best for | Major limitations | -|--------|-------------------------|------------------|--------------|----------------------|----------|-------------------| -| **Browser extension (MV3)** | Medium | Very low | Medium (browser process) | Chrome Web Store / Firefox AMO | Mass casual donors | Service worker idle timeouts; store policies; only while browser runs | -| **systemd user service** (or Windows Service / LaunchAgent) | High | Medium | Very low | Binary + unit file / package | Technical / always-on Linux users | Platform-specific packaging | -| **Desktop app** (Tauri / Electron / native) | High | Medium-High | Low–Medium | Website / GitHub releases | Cross-platform reliability | Larger download, code-signing | -| **Plain webpage / PWA with open tab** | Low | Lowest | Low while open | Just a URL | Quick experiments | Tab throttling, user closes tab, background limits | -| **Service Worker alone** | Poor | Low | Low | Part of a site | Not recommended as primary | Cannot reliably hold long-lived outbound tunnels | - -### 1. Browser extension (Manifest V3) -Technically feasible and the lowest-friction option for many donors. - -- Background service worker opens the WebSocket. -- Since Chrome 116, WebSocket activity resets the service-worker idle timer, but you still need an explicit keepalive (send a ping every ~20 seconds). -- Use `chrome.alarms` (minimum ~30 s) as a durable backup that survives worker termination and forces reconnects. -- You will also need careful reconnect logic, state persistence (e.g. `chrome.storage.session`), and handling of the case where the worker is killed mid-relay. - -**Store and policy issues** -Chrome Web Store (and Firefox AMO) require a clear single purpose and honest disclosure. An extension whose primary function is “share my residential connection / bandwidth” must state that prominently. Proxy-related extensions receive extra scrutiny. Misleading claims or hidden data collection will get you removed. Historical precedent exists (uProxy was a browser-extension peer-proxy system, later discontinued and succeeded by Snowflake). - -**Practical limits** -- Only runs while the browser is open. -- Higher CPU/battery impact than a native binary. -- On mobile Chrome the background behaviour is even more restricted. - -### 2. systemd user service (Linux) / equivalent on other OSes -This is one of the cleanest long-running options. - -- Ship a small static binary (Go, Rust, or even a Node/Python script). -- Provide a user unit file (`~/.config/systemd/user/residential-proxy.service`). -- `systemctl --user enable --now residential-proxy.service`. -- It reconnects automatically, uses almost no resources when idle, and survives reboots (with lingering enabled). - -Windows equivalent: a Windows Service or scheduled task. macOS: LaunchAgent. This matches the architecture used by the open-source residential-proxy-aggregator project (C++ Windows service + watchdog). - -### 3. Desktop application (Tauri preferred over Electron) -Tauri gives you a small native binary with a system tray, automatic updates, and full control over the WebSocket and TCP relay. Electron works but is heavier. This is the most reliable cross-platform choice if you want something that “just stays on” without requiring users to keep a browser open. - -### 4. Service Worker / PWA -A regular service worker (or even an extension service worker without the keepalive tricks) is **not** a good primary client. - -Service workers are event-driven and get terminated after short idle periods. They cannot reliably maintain a persistent outbound TCP/WebSocket tunnel that is ready to accept new SOCKS connections at any moment. They are useful for secondary tasks (notifications, caching) but not as the proxy agent itself. Background throttling on inactive tabs and especially on mobile makes a pure PWA approach unreliable for always-available residential exits. - -### Other practical considerations - -- **Keepalive & reconnect**: Mandatory everywhere. WebSocket idle timeouts on reverse proxies/load balancers are often 60–100 s; your application-level heartbeat should be more aggressive. -- **Bandwidth and rate limiting**: Enforce soft and hard limits on the client so a donor is not surprised by a sudden bill or congestion. -- **Consent & transparency**: Show clear UI that the user is sharing their IP and bandwidth, with an easy off switch. Log (or at least allow users to see) roughly how much traffic has been relayed. -- **ISP terms**: Many residential ISPs prohibit running open proxies, commercial use of the connection, or significant bandwidth resale. Donors should be warned. -- **Abuse & liability**: Residential proxy pools are frequently abused for scraping, credential stuffing, ad fraud, etc. You will need authentication of donors, monitoring, the ability to ban nodes quickly, and clear Terms of Service that shift responsibility appropriately. Pure “donation” models still create platform liability if the traffic is used illegally. -- **Authentication**: Prefer short-lived tokens or account-based auth so you can revoke a donor’s participation instantly. -- **Mobile**: Background execution limits make reliable always-on clients difficult on iOS and restricted on Android. Not recommended as a primary platform. - -### Recommended starting path - -1. **Prototype** with a simple Node/Go/Rust WebSocket client + a matching server that exposes a SOCKS5 listener (look at `pywssocks` or the residential-proxy-aggregator design for structure). -2. **First distribution channel**: browser extension (for reach) + a tiny systemd/LaunchAgent/Windows service binary (for reliability). -3. Later add a polished Tauri tray app if the project gains traction. - -The WebSocket-upstream model is the right foundation. The choice of packaging mainly affects how many people will actually leave the client running and how painful the operational reality becomes. Browser extensions win on distribution; native user services and desktop apps win on reliability and resource behaviour.# diff --git a/xx.py b/xx.py deleted file mode 100755 index 6ad5a8d1..00000000 --- a/xx.py +++ /dev/null @@ -1,138 +0,0 @@ -#!/usr/bin/env python3 -# retoor <retoor@molodetz.nl> - -import argparse -import sys -import time -from pathlib import Path - -MIN_AGE_SECONDS = 3600 -PARTIAL_SUFFIX = ".partial" - - -def human_bytes(size: int) -> str: - value = float(max(0, int(size or 0))) - for unit in ("B", "KB", "MB", "GB", "TB"): - if value < 1024 or unit == "TB": - return f"{value:.0f} {unit}" if unit == "B" else f"{value:.1f} {unit}" - value /= 1024 - return f"{value:.1f} TB" - - -def backups_root() -> Path: - return Path(__file__).resolve().parent / "data" / "backups" - - -def find_partials( - root: Path, now: float, min_age: int -) -> tuple[list[tuple[Path, int, float]], list[tuple[Path, int, float]]]: - abandoned: list[tuple[Path, int, float]] = [] - in_progress: list[tuple[Path, int, float]] = [] - if not root.is_dir(): - return abandoned, in_progress - for path in root.rglob(f"*{PARTIAL_SUFFIX}"): - if not path.is_file() or path.suffix != PARTIAL_SUFFIX: - continue - try: - info = path.stat() - except OSError as exc: - print(f"skip {path}: {exc}", file=sys.stderr) - continue - age = now - info.st_mtime - row = (path, info.st_size, age) - if age < min_age: - in_progress.append(row) - else: - abandoned.append(row) - abandoned.sort(key=lambda item: item[1], reverse=True) - in_progress.sort(key=lambda item: item[1], reverse=True) - return abandoned, in_progress - - -def prune_empty_dirs(root: Path) -> int: - removed = 0 - if not root.is_dir(): - return 0 - for directory in sorted( - (path for path in root.rglob("*") if path.is_dir()), - key=lambda path: len(path.parts), - reverse=True, - ): - try: - next(directory.iterdir()) - except StopIteration: - try: - directory.rmdir() - removed += 1 - except OSError: - pass - except OSError: - pass - return removed - - -def report(label: str, rows: list[tuple[Path, int, float]]) -> int: - total = 0 - if not rows: - return total - print(label) - for path, size, age in rows: - total += size - print(f" {human_bytes(size):>10} {age / 3600:.1f}h {path}") - print(f" {human_bytes(total):>10} total") - return total - - -def main() -> int: - parser = argparse.ArgumentParser( - description=( - "Remove abandoned backup *.partial files under data/backups. " - "Does not open SQLite and does not touch completed .tar.gz archives." - ) - ) - parser.add_argument( - "--apply", - action="store_true", - help="Delete the abandoned files. Default is a dry run.", - ) - parser.add_argument( - "--min-age-seconds", - type=int, - default=MIN_AGE_SECONDS, - help="Skip files newer than this many seconds (in-flight backup guard).", - ) - args = parser.parse_args() - if args.min_age_seconds < 0: - print("--min-age-seconds must be >= 0", file=sys.stderr) - return 2 - root = backups_root() - abandoned, in_progress = find_partials(root, time.time(), args.min_age_seconds) - if in_progress: - report("In-flight (kept)", in_progress) - if not abandoned: - print(f"No abandoned {PARTIAL_SUFFIX} files under {root}") - return 0 - mode = "DELETE" if args.apply else "DRY RUN" - report(f"{mode}: {len(abandoned)} abandoned file(s) under {root}", abandoned) - if not args.apply: - print("Re-run with --apply to delete.") - return 0 - freed = 0 - failed = 0 - for path, size, _age in abandoned: - try: - path.unlink() - freed += size - except OSError as exc: - print(f"failed {path}: {exc}", file=sys.stderr) - failed += 1 - empty = prune_empty_dirs(root) - print( - f"Removed {human_bytes(freed)} in {len(abandoned) - failed} file(s), " - f"{failed} failed, {empty} empty dir(s)" - ) - return 1 if failed else 0 - - -if __name__ == "__main__": - raise SystemExit(main())