devplacepy/devplacepy/services/nodes/CLAUDE.md

This file documents the community node network (devplacepy/services/nodes/, devplacepy/routers/nodes/): members opt in to donate a little of their connection so search can reach sites that block datacenter addresses. Claude Code auto-loads it whenever a file under devplacepy/services/nodes/ is read or edited. The design record, with the measured and unmeasured claims separated, is plan.md at the repository root.

What exists today

The specification, data layer, registry, session, consent gate, kill switch, destination policy, queue, scheduler, dispatcher, circuit breakers, cache, the fabric facade, tunnel mode, social search adapters, the Go client, the browser extension, the Donate tab, the admin page, the Devii tools, the CLI and the docs are built and tested. The gaps below list what is unmeasured or missing. Nothing here is enabled for any account.

Piece	File	State
Protocol specification	spec.py	built, tested
GET /nodes/spec.json	routers/nodes/spec.py	built, tested, public
Generated protocol docs page	templates/docs/nodes-protocol.html	built, reads the nodes_spec Jinja global
API docs group	docs_api/groups/nodes.py	built, one endpoint
Tables and helpers	database/nodes.py, NODES_COLUMNS in database/schema.py	built, tested
Usage ledger	ledger.py (UsageLedger, DonationSummary)	built, tested
bandwidth_sharing consent kind	CONSENT_KINDS in database/moderation.py	registered, never granted by default, enforced by ConsentGate
Frame codec	frames.py (decode, encode, FrameError)	built, tested, validates every frame against spec.py
Consent choke point	consent.py (ConsentGate, Verdict)	built, tested, single and batch
Socket session	session.py (NodeSession)	built, tested with a fake socket and a real one
Registry and counters	registry.py (NodeRegistry, NodeCounters)	built, tested
Supervisor	service.py (NodesService)	built, registered in main.py, default enabled
Device, ticket and socket routes	routers/nodes/devices.py, ticket.py, ws.py	built, tested
nginx upgrade block for /nodes/ws	nginx/nginx.conf.template	added
Destination policy	policy.py (DestinationPolicy, Target, is_public_address)	built, tested
Result cache	cache.py (ResultCache)	built, tested, per process
Circuit breaker	circuit.py (CircuitBreaker)	built, tested
Scheduler	scheduler.py (NodeScheduler, in_active_hours)	built, tested
Queue and broker	broker.py (JobBroker), database/nodes.py request helpers	built, tested
Dispatcher	dispatcher.py (NodeDispatcher)	built, tested, runs in the lock owner
Fabric facade	fabric.py (NodeFabric, FabricResult, FabricError)	built, tested, used by social.py and POST /nodes/internal/fetch
Admin page and actions	routers/admin/nodes.py, templates/admin_nodes.html, static/js/AdminNodes.js	built

Administration (/admin/nodes)

GET /admin/nodes (HTML and JSON AdminNodesOut) is the administrator view: pool counters read from the nodes row in service_state (so any worker answers, at most one persist interval stale; pool.online is false when the heartbeat is stale), queue depth and counts by status, device counts by status, kind, address class and country, the current UTC day's totals, open flags and the 100 most recently seen devices (both capped), the kill switch and the six settings with defaults and ceilings. Per-destination circuit state is not shown: it lives in memory on the lock-owner worker and only the number of open circuits is persisted, so a per-destination view would need a new persisted snapshot. Device counts use the stored status, so a suspension that has expired still counts as suspended until the next write; the recent-devices list uses effective_status.

POST routes, all require_admin, all audited: /killswitch (node.killswitch, flips or sets nodes_enabled), /settings (node.admin.settings, only submitted fields change, values above the ceiling are clamped and reported under clamped), /devices/{uid}/suspend (node.suspend, 1 to 10080 minutes) and /unsuspend (node.unsuspend), /flags/{uid}/resolve (node.flag.resolve). The per-device and per-flag routes are gated by is_senior_admin against the device or flag owner and answer through deny_senior (audited denied). Suspension and the kill switch take effect within one reconcile tick on whichever worker holds the sockets. Ceilings: nodes_max_total and nodes_cache_ttl_seconds use LimitSpec; the quota ceilings are constants in the router because LimitSpec only carries defaults for them.

The page refreshes every 10 seconds by fetching its own HTML and swapping the [data-live] sections (AdminNodes.js, app.adminNodes); the settings form and the suspend modal are never swapped, so typing is not interrupted. Devii: node_admin_status, node_admin_suspend and node_admin_killswitch (the last two in CONFIRM_REQUIRED), node_admin_unsuspend, all requires_admin.

The specification is the contract

spec.py is the single source of truth. The server, the Go client, the browser extension, the docs page and the tests all read it. Nothing may restate a number, a frame field, an allowlisted host or a refusal code anywhere else: import it, or read it from /nodes/spec.json.

• ProtocolSpec holds version, min_supported, LimitSpec, FrameSpec tuple, DestinationSpec tuple and RefusalSpec tuple. Every one is a frozen dataclass. SPEC.describe() is the JSON form and the only thing the router returns.
• open_proxy is field(default=False, init=False): it cannot be set through the constructor. There is no open proxy mode and there will not be one. Changing that is a protocol change that needs a new PROTOCOL_VERSION, not a flag.
• ALLOWED_PORTS is (80, 443). A DestinationSpec may narrow it, never widen it (a unit test fails otherwise).
• DestinationSpec.matches is suffix-safe: a host matches when it equals the suffix or ends with . plus the suffix, so evilreddit.com and reddit.com.evil.net never match reddit.com.
• LimitSpec is the list of hard caps from the plan. Nothing is unbounded and nothing autotunes: a constant you can read beats magic you cannot debug. A number an admin may lower becomes a ConfigField later; the ceiling stays here.
• A frame type is unique per direction. An unknown type, a missing required field, an unexpected field or an oversize frame is a protocol error that closes the socket loudly, never an ignore.

The docs page nodes-protocol.html renders its tables from the nodes_spec Jinja global (templating.py, NODES_SPEC.describe()), so it cannot drift from the code. tests/api/nodes/spec.py asserts that every frame, destination, refusal code and limit name appears on the rendered page.

Decisions taken (recorded so they are not re-litigated)

Id	Decision
D1	devplacepy owns the node fabric; rsearch is a consumer, not the home
D2	the native client is Go, one static binary per OS and architecture
D3	the extension ships as a sideloadable zip first, store submission later
D4	rewards are XP, badges and a leaderboard only: no coins, no money, nothing purchasable
D5	donation has no age gate: any active member with consent may donate
D6	a datacenter address is not a donor: residential and mobile only, admin override

Data layer (Phase 2)

Seven tables, all ensured with their full column set in init_db (_ensure_nodes_tables): node_devices, node_settings, node_usage_daily, node_donors, node_flags (soft-deletable, in SOFT_DELETE_TABLES and UNREPORTABLE_TABLES with a reason), and node_requests, node_events (ephemeral, hard garbage collection only). Account deletion cascades through OWNED_TABLES for devices, settings, donors and daily usage.

• Atomic by one statement. The device cap is an INSERT ... SELECT ... WHERE (SELECT COUNT(*)...) < max, the daily bandwidth cap is an UPDATE ... WHERE COALESCE(bytes_up,0)+COALESCE(bytes_down,0)+:n <= :cap, and the daily XP cap is a conditional UPDATE that rolls the day over inside the same statement. Verified with real separate processes against fresh, unseeded rows (40 racers on a 10-job cap gave exactly 10, 24 device creations gave exactly 5, 30 XP claims gave exactly the cap).
• record_job is one transaction: the daily bucket (INSERT OR IGNORE then the conditional UPDATE) and the lifetime node_donors upsert commit together or not at all. A job refused by the cap counts nothing. A failed job counts requests_failed and never moves bytes or lifetime totals.
• Lifetime totals are maintained at write time, so pruning the daily buckets (usage_daily_retention_days) can never lower them. Prune functions take dry_run and use only the constants in LimitSpec; there is no argument that lowers a retention floor.
• Secrets: dpn_ plus 32 random bytes, shown once at create or rotate, stored as sha256 plus a 12 character lookup prefix, compared with hmac.compare_digest. IP addresses are stored only as a keyed hash (hash_ip).
• Settings are clamped on write and on read (normalise_settings): bandwidth to 1..max_donor_mb_per_day, concurrency to 1..max_concurrent_per_node, hours must be HH:MM, platforms are intersected with the spec's adapters. A member with no row is opted out.
• Site settings seeded in init_db: nodes_enabled (0), nodes_max_total, nodes_daily_quota_user, nodes_daily_quota_guest, nodes_cache_ttl_seconds, nodes_social_via_nodes (0). Like every seed, consumers pass the default to get_setting because the seed does not run on an empty database.

Connecting a node (Phase 3)

1. A member consents to bandwidth_sharing and turns donation on for themselves (the profile UI for this is Phase 9; today it is POST /profile/{username}/consent and save_node_settings).
2. POST /nodes/devices registers a device and returns its dpn_ secret once. GET /nodes/devices, POST /nodes/devices/{uid}/rotate and /delete manage it. All are require_user_api (401, never a redirect) and answer Cache-Control: no-store.
3. The node calls POST /nodes/ticket with Authorization: Bearer dpn_.... A session cookie or a member API key is not a device secret and gets 401. The ticket is a ws_tickets row with scope="nodes" and the device uid, valid ticket_ttl_seconds, redeemed by one conditional UPDATE (16 processes racing one ticket gave exactly one winner; a ticket of the wrong scope never redeems).
4. The node opens WS /nodes/ws?ticket=.... The route only runs on the service-lock owner (4013 elsewhere, the client retries fast, the /pubsub/ws precedent). Order of refusal: service disabled 1013, not the owner 4013, bad ticket or device or status 1008, consent gate 1008 with the reason, capacity 4029, then the handshake.
5. hello must arrive within hello_timeout_seconds, with a supported protocol, a kind equal to the registered device kind and a non-empty list of known modes. The server answers welcome with the full limits and allowlist from the spec. A second connection of the same device kicks the first with replaced.
6. The session pings every ping_seconds; any frame from the client resets the missed counter and missed_pings_before_evict unanswered pings close it with heartbeat. usage frames are advisory and only summed on the session. Every other client frame is a protocol error until Phase 4 assigns jobs.

Jobs (Phase 4)

A job is a row in node_requests, so any worker can ask and only the lock owner serves. Life cycle: queued -> assigned -> done, or failed / expired; every transition is one conditional UPDATE.

1. NodeFabric.fetch(url, caller_kind=, caller_id=) (any worker): policy check, result cache, kill switch and "any node online" (read from the nodes row in service_state, so it is at most one persist interval stale), then the caller's daily quota, then enqueue, then poll the row every poll_interval_ms until it is terminal or the caller's wait ends. One retry on node_disconnected, timeout, send_failed, concurrency_exceeded, outside_hours, bandwidth_exhausted; the retry does not spend a second quota unit. Bounded: queue depth (max_queue_depth, a conditional INSERT), max_wait_seconds, queue_max_wait_seconds.
2. NodeDispatcher.tick (lock owner, every dispatcher_tick_ms, started by ensure_dispatching() from the socket route or on_enable): expire, sweep timed-out in-flight jobs, then for each queued row re-check the policy, ask the scheduler for a node, claim the row atomically, send the job frame.
3. The node answers with job_result or job_error. Session._handle only accepts an answer for a job it has in flight; anything else is a protocol error and closes the socket.

Scheduler eligibility (every one must hold): socket open and not paused, speaks a transport the job allows (tunnel is preferred over fetch for any; a node that speaks neither gets nothing), is not the caller's own account, below min(server, donor setting, hello limits) concurrency, the destination's adapter is in the donor's allowed_platforms, inside the donor's active hours in the donor's timezone (windows may wrap midnight), below the donor's daily byte cap, at least min_seconds_per_node_domain since this node last started a job for the destination suffix, node circuit closed. Globally the destination needs 1/max_rps seconds since its last job and a closed destination circuit. Choice: fewest in-flight, then least recently assigned, then random. Per-destination state is keyed by the allowlist suffix, so a.reddit.com ... z.reddit.com cannot grow it.

Trust. via is always fetch for node results. A result is checked before it is stored: status 100-599, strict base64, body capped at max_body_bytes (truncated and flagged, never silent), headers reduced to at most 64 short strings with set-cookie, set-cookie2, authorization and proxy-authenticate dropped. A malformed result is an integrity failure: the request fails, integrity_failures is counted on the device and the process, a node_flags row is raised for a human, and the node is NOT auto-banned. A donor refusing a destination the server allowed raises a refusal_mismatch flag. Donor limit codes (bandwidth_exhausted, outside_hours, concurrency_exceeded) pause the node instead of blaming it. HTTP 429 and 5xx count against the destination circuit (they are the signal that stops hammering a rate-limiting site); the result is still returned to the caller. Sampled redundancy between two donors is not built (Phase 6B).

Accounting. A successful job calls record_job (daily bucket and lifetime totals in one transaction). If the donor's cap would be crossed the result is still delivered but the node is stopped for the day. Failures call record_failure. Bytes up are estimated (len(url) + 256) because the server never sees the donor's real request.

Measured (scratch SQLite on this machine, 2000 synthetic sessions, 500 queued requests, batch 50): an idle tick costs 0.1 ms; a tick where no node is eligible because every node is outside its hours cost 225 ms before two fixes (the active-hours window is now cached per minute and a destination found saturated is scanned once per tick, not once per queued request) and 8.4 ms after; a destination held back by its rate ceiling costs 4.7 ms. Races with 16 to 60 real processes on fresh rows: the queue cap admitted exactly the remaining slots, one request was claimed by exactly one assignment, a caller quota of 5 admitted exactly 5. These are one machine and synthetic sessions, not a soak test.

The consent choke point

ConsentGate is the only place that decides whether a member's node may exist. allows_many(user_uids) decides a whole set with a fixed number of queries and allows(uid) is the one-element case of it, so the single and batch answers cannot diverge (a test compares them). Order of reasons: kill_switch, no_account, deletion_pending, account_inactive, suspended, no_consent, not_enabled. It is called at device registration, ticket issue, socket accept and every supervisor tick.

NodesService.reconcile runs every reconcile_seconds, asks the gate once for all connected owners and loads all devices in one query, then kicks with a reason: kill_switch, revoked, suspended, or the gate's reason. This is why withdrawing the consent, switching donation off, deleting the device or flipping nodes_enabled takes effect within one tick even when the request landed on a different worker than the one holding the sockets. Measured (scratch SQLite, this machine, 2000 synthetic sessions): one tick costs about 110 ms; the per-session version of the same check cost about 1 ms per session, so 488 ms at 500 sessions and an extrapolated 2 s at 2000, which would have blocked the event loop for the whole tick. Do not reintroduce a per-session query in reconcile.

Withdrawal is member-only: MEMBER_ONLY_CONSENTS in database/moderation.py lists consents that Devii (header X-Devii-Agent: 1, resolved in routers/profile/consent.py) may withdraw but never grant, and the acceptance service lists bandwidth_sharing in EXCLUDED_CONSENTS.

Two transports, two trust levels

tunnel (native client) relays raw TCP so TLS runs end to end between DevPlace and the destination: the donor cannot read or forge content. fetch (browser extension, which has no raw sockets) lets the donor read and forge the response, so every fetch result is verified and labelled. A plain web page or a service worker cannot do either (CORS, no sockets) and is out of scope for good.

Donate tab (member UI)

The owner-only Donate tab on the profile (?tab=donate, templates/_node_donation.html, static/js/NodeDonation.js as app.nodeDonation, static/css/nodes.css) is a thin client of the routes above: it renders /nodes/me server-side (context key node_me, only for the owner on that tab, noindex,follow), then polls GET /nodes/me every 30 seconds with the shared Poller. Consent goes through POST /profile/{username}/consent, never a second consent route. The device secret exists only in the browser memory and DOM until "I have saved it" is pressed; the install lines are built client-side from GET /nodes/client-version. Prose for members is templates/docs/nodes.html; the data categories are in templates/docs/privacy.html.

Tunnel mode (tunnel.py)

For a tunnel or any job the scheduler prefers a node whose hello.modes include tunnel. The dispatcher marks the transport on the row (node_requests.mode becomes the transport actually used, which is how FabricResult.via says tunnel or fetch), then runs the request itself in the lock owner: TunnelBridge.fetch opens a one-shot listener on 127.0.0.1:<ephemeral>, points a stealth (curl_cffi, Chrome fingerprint) client at it as an HTTP proxy, and the Relay answers exactly one CONNECT host:port for the job's own host and port (anything else is 403, a second CONNECT is 405, a donor that refuses is 502). Bytes are relayed as tunnel_data frames (48 KiB chunks, base64, both directions, RELAY_CAP_BYTES total, queue_full and bad_base64 end the tunnel with a code). TLS runs between DevPlace and the destination, so the donor sees the host and ciphertext and cannot read or forge the page; the ledger counts the real bytes relayed. Late tunnel_* frames for a closed relay are ignored (the donor legitimately sends tunnel_close after the server did). Redirects are not followed; the caller gets the 3xx. In fetch mode the job frame now carries JOB_HEADERS (a descriptive bot user agent), because a Go or browser client would otherwise send its own.

Social search (social.py, adapters/)

SocialSearch.search fans one query out to adapters in parallel (asyncio.gather, 20 s each) and merges hits round-robin. Direct adapters (adapters/direct.py: Hacker News, Stack Overflow, GitHub issues, dev.to by tag, Mastodon by hashtag) need no donors. adapters/reddit.py is node-backed and only used when nodes_social_via_nodes is 1; otherwise it is reported as nodes_not_used. Every adapter parses a response structurally and raises AdapterError on a forged or wrong shape; platforms that fail are listed under unavailable with a code and never filled with invented results. Measured on this machine (2026-10-15, datacenter address): Hacker News Algolia, Stack Exchange, GitHub search, dev.to tag API and Mastodon hashtag timelines answer; Bluesky searchPosts returns 403 without auth, Mastodon /api/v2/search returns nothing without login, Lobsters search sits behind a bot wall, dev.to has no usable text search, and Reddit returns 403 to this address. Those are why Bluesky and Lobsters are not adapters. Surfaces: GET /nodes/social/search (daily per-member quota nodes_daily_social_user, default 300), Devii social_search, the DeepSearch deepsearch_social option (off by default, signed-in owners only, goes through the HTTP route with the owner's key).

Server mechanisms added after the first build

• Address class (D6 enforced). geo.py keeps DB-IP lite ASN and country data in a SQLite range file under config.NODE_GEO_DIR (devplace nodes geo-sync [--dry-run] [--month], geo-status; CC BY 4.0, attribution on /docs/privacy.html). Measured: build about 10 s, 477362 ASN and 710834 country rows, 48 MB, lookup 0.17 ms. addressclass.classify maps an ASN to residential, mobile, datacenter, business or unknown. The scheduler only assigns jobs to ELIGIBLE_CLASSES (residential, mobile) unless a device carries class_override == "allow" (admin, POST /admin/nodes/devices/{uid}/class, audit node.device.override, Devii node_admin_class_override confirm gated) or nodes_allow_unknown_class=1 admits unknown. Refusals are counted in NodeScheduler.class_refusals. A missing geo file is counters.geo_unavailable plus one warning, never a silent pass.
• Durable state. last_seen_at writes are batched (touch_devices, last_seen_write_seconds). Counters flush as deltas into node_counters every counter_flush_seconds (lifetime totals survive restart). Circuit breaker state is saved to node_circuits (snapshot/restore, capped at circuit_snapshot_max). The fabric cache is SharedResultCache: an in-process L1 over the node_cache table, so a result fetched by one worker serves the others.
• Fast path. wake.py WakeHub wakes the dispatch loop and a waiting caller in the same process the moment a job is queued or done; the DB poll remains the cross-process fallback. Measured end to end (one donor, 300 jobs): polling p50 209 ms, p95 214 ms; fast path p50 25 ms, p95 about 30 ms, max about 38 ms. A single 6.2 s cold-start outlier appeared once and did not reproduce.
• Sampled redundancy. redundancy_percent (5, ceiling redundancy_max_percent 20) of fetch-mode jobs are shadowed to a second donor (shadow_of, exclude_device); shape.py compares structure, and a mismatch raises the flag redundancy_mismatch for a human. Counters: redundancy_checked, redundancy_matched, redundancy_mismatches, redundancy_inconclusive.
• Outcome counters per address class. outcome:<adapter>:<class>:<ok|blocked|failed> in node_counters, shown on /admin/nodes. This is the instrument for the residential success rate; it reads n = 0 until real donors exist.
• DeepSearch routes hostile URLs through POST /nodes/internal/fetch with the owner's api key (crawl.node_servable, node_page) when nodes_social_via_nodes is on.
• Battery. on_battery is a donor limit code (pause 300 s, retryable). welcome.device_label and kick.until are optional protocol fields; a suspended ticket 403 carries until.

Measured capacity (one worker, scratch server, 2000 simulated donors on this machine): 2000 sockets connected in 8 s; RSS 156 MB at 250, 190 MB at 500, 241 MB at 1000, 341 MB at 2000; idle dispatch tick p50 67 ms (max 160 ms); event loop lag p99 78 ms idle; 14.2 jobs/s completed at 15 offered, 0 client failures. A soak at 2000 sockets, 10 jobs/s and 3 percent a minute churn ran 4844 s (about 81 minutes, not 24 hours): RSS 346 to 361 MB, 42909 of 42913 jobs done, 1 failed, 0 integrity failures, 2026 of 2026 redundancy checks matched, loop lag p99 73 to 106 ms. Races, with separate OS processes: 16 workers x 20 counter flushes summed exactly (1600 of 1600); 12 racing cache puts left 1 row and all 12 readers hit.

Clients

nodes/client/ is the Go donor client (standard library only, build.sh cross-compiles five targets and writes manifest.json). It reads battery state on Linux, macOS and Windows (RefuseBattery, PauseOnBattery) and holds a timed suspension until until (clamped 60 s to 24 h). nodes/extension/ is the MV3 extension (Chrome and Firefox, fetch mode only; icons 16/32/48/128; a suspended state with suspendedUntil; active hours in the server timezone; the manifest sets content_security_policy.extension_pages explicitly because Firefox's default MV3 policy adds upgrade-insecure-requests, which rewrote ws://127.0.0.1 to wss:// and broke loopback connections). make node-client builds both into data/node_clients/, served by GET /nodes/client-version and GET /nodes/download/{artifact} (strict names, no symlinks, SHA-256 from the file). Verified in real browsers: Chromium via tests/e2e/nodes/extension.py (disclosure, connect, online, a fetch-mode reddit job answered with 403 via fetch, suspend, disable) and Firefox 158 with geckodriver (connected, job done with 403 and 190240 bytes, suspended with until, disable). The Firefox script is not in the repository; the method is in nodes/extension/README.md. Go: go vet and go test pass, cross-compiles clean.

Gaps (what is not built or not measured)

• Reddit success from a residential line is unmeasured. The counters exist and read n = 0; every real run so far used a datacenter donor and got 403. X, Instagram, LinkedIn, Facebook and TikTok stay unreachable (login walls); nothing here claims them.
• No 24 hour soak. The longest run was about 81 minutes. The caps in LimitSpec are ceilings backed by that run, not by a day.
• WebSocket idle timeout through the public hostname is unmeasured. It needs a production connection, which needs the owner's go-ahead.
• The extension has no battery pause (browsers expose no battery state to extensions); the Go client has one.
• The geo file needs a monthly geo-sync; between syncs a new ASN is unknown.
• Windows and macOS client builds compile but were not run.
• Launch is blocked on a recorded legal decision (platform terms, donor terms, ISP terms, jurisdiction). Nothing here is enabled for any account: nodes_enabled is 0 until an administrator switches it on. This is a human decision and no code closes it.

Rules for work in this directory

• Every user-facing or protocol-facing claim in the docs must describe what runs today. A page for an unbuilt feature states its status at the top, as nodes-protocol.html does.
• A new consent is enforced at one choke point, never at call sites (see services/moderation/CLAUDE.md).
• A new table here is either in SOFT_DELETE_TABLES or documented as derived and hard-garbage-collected; it is either in REPORTABLE_TARGETS or in UNREPORTABLE_TABLES with a reason.
• The machine may suspend a node (reversible, timed). It may never ban a user or delete a ledger row on its own: it raises a flag for a human.