@@ -2,7 +2,7 @@ This file documents the community node network (`devplacepy/services/nodes/`, `d
|
||||
|
||||
## What exists today
|
||||
|
||||
Phase 1 (the specification), Phase 2 (the data layer and usage ledger) and Phase 3 (registry, session, consent gate, kill switch, device and ticket routes, the lock-owner socket) are built. No job traffic flows yet. Everything else is planned and listed under "Gaps" next to the phase that closes it. Nothing here is enabled for any account.
|
||||
The specification, data layer, registry, session, consent gate, kill switch, destination policy, queue, scheduler, dispatcher, circuit breakers, cache, the fabric facade, tunnel mode, social search adapters, the Go client, the browser extension, the Donate tab, the admin page, the Devii tools, the CLI and the docs are built and tested. The gaps below list what is unmeasured or missing. Nothing here is enabled for any account.
|
||||
|
||||
| Piece | File | State |
|
||||
|-------|------|-------|
|
||||
@@ -10,7 +10,7 @@ Phase 1 (the specification), Phase 2 (the data layer and usage ledger) and Phase
|
||||
| `GET /nodes/spec.json` | `routers/nodes/spec.py` | built, tested, public |
|
||||
| Generated protocol docs page | `templates/docs/nodes-protocol.html` | built, reads the `nodes_spec` Jinja global |
|
||||
| API docs group | `docs_api/groups/nodes.py` | built, one endpoint |
|
||||
| Tables and helpers | `database/nodes.py`, `NODES_COLUMNS` in `database/schema.py` | built, tested, no route uses them yet |
|
||||
| Tables and helpers | `database/nodes.py`, `NODES_COLUMNS` in `database/schema.py` | built, tested |
|
||||
| Usage ledger | `ledger.py` (`UsageLedger`, `DonationSummary`) | built, tested |
|
||||
| `bandwidth_sharing` consent kind | `CONSENT_KINDS` in `database/moderation.py` | registered, never granted by default, enforced by `ConsentGate` |
|
||||
| Frame codec | `frames.py` (`decode`, `encode`, `FrameError`) | built, tested, validates every frame against `spec.py` |
|
||||
@@ -20,6 +20,22 @@ Phase 1 (the specification), Phase 2 (the data layer and usage ledger) and Phase
|
||||
| Supervisor | `service.py` (`NodesService`) | built, registered in `main.py`, default enabled |
|
||||
| Device, ticket and socket routes | `routers/nodes/devices.py`, `ticket.py`, `ws.py` | built, tested |
|
||||
| nginx upgrade block for `/nodes/ws` | `nginx/nginx.conf.template` | added |
|
||||
| Destination policy | `policy.py` (`DestinationPolicy`, `Target`, `is_public_address`) | built, tested |
|
||||
| Result cache | `cache.py` (`ResultCache`) | built, tested, per process |
|
||||
| Circuit breaker | `circuit.py` (`CircuitBreaker`) | built, tested |
|
||||
| Scheduler | `scheduler.py` (`NodeScheduler`, `in_active_hours`) | built, tested |
|
||||
| Queue and broker | `broker.py` (`JobBroker`), `database/nodes.py` request helpers | built, tested |
|
||||
| Dispatcher | `dispatcher.py` (`NodeDispatcher`) | built, tested, runs in the lock owner |
|
||||
| Fabric facade | `fabric.py` (`NodeFabric`, `FabricResult`, `FabricError`) | built, tested, used by `social.py` and `POST /nodes/internal/fetch` |
|
||||
| Admin page and actions | `routers/admin/nodes.py`, `templates/admin_nodes.html`, `static/js/AdminNodes.js` | built |
|
||||
|
||||
## Administration (`/admin/nodes`)
|
||||
|
||||
`GET /admin/nodes` (HTML and JSON `AdminNodesOut`) is the administrator view: pool counters read from the `nodes` row in `service_state` (so any worker answers, at most one persist interval stale; `pool.online` is false when the heartbeat is stale), queue depth and counts by status, device counts by status, kind, address class and country, the current UTC day's totals, open flags and the 100 most recently seen devices (both capped), the kill switch and the six settings with defaults and ceilings. **Per-destination circuit state is not shown:** it lives in memory on the lock-owner worker and only the number of open circuits is persisted, so a per-destination view would need a new persisted snapshot. Device counts use the stored status, so a suspension that has expired still counts as `suspended` until the next write; the recent-devices list uses `effective_status`.
|
||||
|
||||
POST routes, all `require_admin`, all audited: `/killswitch` (`node.killswitch`, flips or sets `nodes_enabled`), `/settings` (`node.admin.settings`, only submitted fields change, values above the ceiling are clamped and reported under `clamped`), `/devices/{uid}/suspend` (`node.suspend`, 1 to 10080 minutes) and `/unsuspend` (`node.unsuspend`), `/flags/{uid}/resolve` (`node.flag.resolve`). The per-device and per-flag routes are gated by `is_senior_admin` against the device or flag owner and answer through `deny_senior` (audited `denied`). Suspension and the kill switch take effect within one `reconcile` tick on whichever worker holds the sockets. Ceilings: `nodes_max_total` and `nodes_cache_ttl_seconds` use `LimitSpec`; the quota ceilings are constants in the router because `LimitSpec` only carries defaults for them.
|
||||
|
||||
The page refreshes every 10 seconds by fetching its own HTML and swapping the `[data-live]` sections (`AdminNodes.js`, `app.adminNodes`); the settings form and the suspend modal are never swapped, so typing is not interrupted. Devii: `node_admin_status`, `node_admin_suspend` and `node_admin_killswitch` (the last two in `CONFIRM_REQUIRED`), `node_admin_unsuspend`, all `requires_admin`.
|
||||
|
||||
## The specification is the contract
|
||||
|
||||
@@ -65,6 +81,22 @@ Seven tables, all ensured with their full column set in `init_db` (`_ensure_node
|
||||
5. `hello` must arrive within `hello_timeout_seconds`, with a supported `protocol`, a `kind` equal to the registered device kind and a non-empty list of known `modes`. The server answers `welcome` with the full limits and allowlist from the spec. A second connection of the same device kicks the first with `replaced`.
|
||||
6. The session pings every `ping_seconds`; any frame from the client resets the missed counter and `missed_pings_before_evict` unanswered pings close it with `heartbeat`. `usage` frames are advisory and only summed on the session. Every other client frame is a protocol error until Phase 4 assigns jobs.
|
||||
|
||||
## Jobs (Phase 4)
|
||||
|
||||
A job is a row in `node_requests`, so any worker can ask and only the lock owner serves. Life cycle: `queued` -> `assigned` -> `done`, or `failed` / `expired`; every transition is one conditional `UPDATE`.
|
||||
|
||||
1. `NodeFabric.fetch(url, caller_kind=, caller_id=)` (any worker): policy check, result cache, kill switch and "any node online" (read from the `nodes` row in `service_state`, so it is at most one persist interval stale), then the **caller's daily quota**, then enqueue, then poll the row every `poll_interval_ms` until it is terminal or the caller's wait ends. One retry on `node_disconnected`, `timeout`, `send_failed`, `concurrency_exceeded`, `outside_hours`, `bandwidth_exhausted`; the retry does not spend a second quota unit. Bounded: queue depth (`max_queue_depth`, a conditional `INSERT`), `max_wait_seconds`, `queue_max_wait_seconds`.
|
||||
2. `NodeDispatcher.tick` (lock owner, every `dispatcher_tick_ms`, started by `ensure_dispatching()` from the socket route or `on_enable`): expire, sweep timed-out in-flight jobs, then for each queued row re-check the policy, ask the scheduler for a node, claim the row atomically, send the `job` frame.
|
||||
3. The node answers with `job_result` or `job_error`. `Session._handle` only accepts an answer for a job it has in flight; anything else is a protocol error and closes the socket.
|
||||
|
||||
**Scheduler eligibility** (every one must hold): socket open and not paused, speaks a transport the job allows (`tunnel` is preferred over `fetch` for `any`; a node that speaks neither gets nothing), is not the caller's own account, below `min(server, donor setting, hello limits)` concurrency, the destination's adapter is in the donor's `allowed_platforms`, inside the donor's active hours in the donor's timezone (windows may wrap midnight), below the donor's daily byte cap, at least `min_seconds_per_node_domain` since this node last started a job for the destination suffix, node circuit closed. Globally the destination needs `1/max_rps` seconds since its last job and a closed destination circuit. Choice: fewest in-flight, then least recently assigned, then random. Per-destination state is keyed by the allowlist suffix, so `a.reddit.com` ... `z.reddit.com` cannot grow it.
|
||||
|
||||
**Trust.** `via` is always `fetch` for node results. A result is checked before it is stored: status 100-599, strict base64, body capped at `max_body_bytes` (truncated and flagged, never silent), headers reduced to at most 64 short strings with `set-cookie`, `set-cookie2`, `authorization` and `proxy-authenticate` dropped. A malformed result is an **integrity failure**: the request fails, `integrity_failures` is counted on the device and the process, a `node_flags` row is raised for a human, and the node is NOT auto-banned. A donor refusing a destination the server allowed raises a `refusal_mismatch` flag. Donor limit codes (`bandwidth_exhausted`, `outside_hours`, `concurrency_exceeded`) pause the node instead of blaming it. HTTP 429 and 5xx count against the destination circuit (they are the signal that stops hammering a rate-limiting site); the result is still returned to the caller. Sampled redundancy between two donors is not built (Phase 6B).
|
||||
|
||||
**Accounting.** A successful job calls `record_job` (daily bucket and lifetime totals in one transaction). If the donor's cap would be crossed the result is still delivered but the node is stopped for the day. Failures call `record_failure`. Bytes up are estimated (`len(url) + 256`) because the server never sees the donor's real request.
|
||||
|
||||
**Measured** (scratch SQLite on this machine, 2000 synthetic sessions, 500 queued requests, batch 50): an idle tick costs 0.1 ms; a tick where no node is eligible because every node is outside its hours cost 225 ms before two fixes (the active-hours window is now cached per minute and a destination found saturated is scanned once per tick, not once per queued request) and 8.4 ms after; a destination held back by its rate ceiling costs 4.7 ms. Races with 16 to 60 real processes on fresh rows: the queue cap admitted exactly the remaining slots, one request was claimed by exactly one assignment, a caller quota of 5 admitted exactly 5. These are one machine and synthetic sessions, not a soak test.
|
||||
|
||||
## The consent choke point
|
||||
|
||||
`ConsentGate` is the only place that decides whether a member's node may exist. `allows_many(user_uids)` decides a whole set with a fixed number of queries and `allows(uid)` is the one-element case of it, so the single and batch answers cannot diverge (a test compares them). Order of reasons: `kill_switch`, `no_account`, `deletion_pending`, `account_inactive`, `suspended`, `not_adult`, `no_consent`, `not_enabled`. It is called at device registration, ticket issue, socket accept and every supervisor tick. An account whose `age_band` is empty (created before age bands existed) is refused as not adult, loudly, rather than assumed adult.
|
||||
@@ -77,13 +109,32 @@ Withdrawal is member-only: `MEMBER_ONLY_CONSENTS` in `database/moderation.py` li
|
||||
|
||||
`tunnel` (native client) relays raw TCP so TLS runs end to end between DevPlace and the destination: the donor cannot read or forge content. `fetch` (browser extension, which has no raw sockets) lets the donor read and forge the response, so every `fetch` result is verified and labelled. A plain web page or a service worker cannot do either (CORS, no sockets) and is out of scope for good.
|
||||
|
||||
## Gaps (what the next phases close)
|
||||
## Donate tab (member UI)
|
||||
|
||||
- Phase 2 leftovers: the `bandwidth_sharing` consent can currently be granted through `POST /profile/{username}/consent` and by Devii's `set_consent` (Devii calls that route as the user). Nothing reads it yet, so it is inert, but Phase 3 must make Devii unable to grant it (Devii may read and withdraw) at the same time as `ConsentGate` lands. No XP or badge is awarded yet (`UsageLedger.claim_xp` only reserves the daily budget); the privacy page and app-store declarations are updated with the first phase that lets a device connect, because until then no personal data is collected.
|
||||
- Phase 3 leftovers: `country`, `asn` and `asn_class` on a device are never filled (they stay `unknown`), so the datacenter exclusion (D6) is not enforced yet; it needs the offline ASN database from Phase 4. `last_seen_at` is written on connect only, not per heartbeat, so a device list cannot yet show who is online right now. The socket counters are per process and are shown in the service metrics only. The `devplace.net` front proxy idle timeout for a long-lived socket is unmeasured.
|
||||
- Phase 4: broker, scheduler, dispatcher, destination policy, circuit breakers, result cache.
|
||||
- Phase 5 onward: the reference client, adapters, the Go client, the extension, the profile UI, Devii tools, the CLI, the privacy and donor-terms pages, the legal gate.
|
||||
- Not measured yet: Reddit and X success rates per address class, WebSocket idle timeouts through `pravda.education` and `devplace.net`, dispatcher tick cost against SQLite, socket capacity of one worker. The caps in `LimitSpec` are starting ceilings, not measurements.
|
||||
The owner-only `Donate` tab on the profile (`?tab=donate`, `templates/_node_donation.html`, `static/js/NodeDonation.js` as `app.nodeDonation`, `static/css/nodes.css`) is a thin client of the routes above: it renders `/nodes/me` server-side (context key `node_me`, only for the owner on that tab, `noindex,follow`), then polls `GET /nodes/me` every 30 seconds with the shared `Poller`. Consent goes through `POST /profile/{username}/consent`, never a second consent route. The device secret exists only in the browser memory and DOM until "I have saved it" is pressed; the install lines are built client-side from `GET /nodes/client-version`. Prose for members is `templates/docs/nodes.html`; the data categories are in `templates/docs/privacy.html`.
|
||||
|
||||
## Tunnel mode (`tunnel.py`)
|
||||
|
||||
For a `tunnel` or `any` job the scheduler prefers a node whose `hello.modes` include `tunnel`. The dispatcher marks the transport on the row (`node_requests.mode` becomes the transport actually used, which is how `FabricResult.via` says `tunnel` or `fetch`), then runs the request itself in the lock owner: `TunnelBridge.fetch` opens a one-shot listener on `127.0.0.1:<ephemeral>`, points a stealth (curl_cffi, Chrome fingerprint) client at it as an HTTP proxy, and the `Relay` answers exactly one `CONNECT host:port` for the job's own host and port (anything else is `403`, a second `CONNECT` is `405`, a donor that refuses is `502`). Bytes are relayed as `tunnel_data` frames (48 KiB chunks, base64, both directions, `RELAY_CAP_BYTES` total, `queue_full` and `bad_base64` end the tunnel with a code). TLS runs between DevPlace and the destination, so the donor sees the host and ciphertext and cannot read or forge the page; the ledger counts the real bytes relayed. Late `tunnel_*` frames for a closed relay are ignored (the donor legitimately sends `tunnel_close` after the server did). Redirects are not followed; the caller gets the 3xx. In `fetch` mode the `job` frame now carries `JOB_HEADERS` (a descriptive bot user agent), because a Go or browser client would otherwise send its own.
|
||||
|
||||
## Social search (`social.py`, `adapters/`)
|
||||
|
||||
`SocialSearch.search` fans one query out to adapters in parallel (`asyncio.gather`, 20 s each) and merges hits round-robin. Direct adapters (`adapters/direct.py`: Hacker News, Stack Overflow, GitHub issues, dev.to by tag, Mastodon by hashtag) need no donors. `adapters/reddit.py` is node-backed and only used when `nodes_social_via_nodes` is `1`; otherwise it is reported as `nodes_not_used`. Every adapter parses a response structurally and raises `AdapterError` on a forged or wrong shape; platforms that fail are listed under `unavailable` with a code and never filled with invented results. **Measured on this machine (2026-10-15, datacenter address):** Hacker News Algolia, Stack Exchange, GitHub search, dev.to tag API and Mastodon hashtag timelines answer; Bluesky `searchPosts` returns 403 without auth, Mastodon `/api/v2/search` returns nothing without login, Lobsters search sits behind a bot wall, dev.to has no usable text search, and Reddit returns 403 to this address. Those are why Bluesky and Lobsters are not adapters. Surfaces: `GET /nodes/social/search` (daily per-member quota `nodes_daily_social_user`, default 300), Devii `social_search`, the DeepSearch `deepsearch_social` option (off by default, signed-in owners only, goes through the HTTP route with the owner's key).
|
||||
|
||||
## Clients
|
||||
|
||||
`nodes/client/` is the Go donor client (standard library only, `build.sh` cross-compiles five targets and writes `manifest.json`). `nodes/extension/` is the MV3 extension (Chrome and Firefox, fetch mode only, 49 `node --test` tests, one of which reads `spec.py` and fails on drift). `make node-client` builds both into `data/node_clients/`, which `GET /nodes/client-version` and `GET /nodes/download/{artifact}` serve (names must match a strict pattern, symlinks and the manifest are never served, the SHA-256 is computed from the file). **Verified end to end on 2026-10-15** against a scratch server on this machine: the Go client logged in, connected, answered a `tunnel` job for `reddit.com` (`via: tunnel`, status 403 because the donor was this same datacenter address) and stayed connected across repeated jobs. Not verified: a residential donor, the extension in a real browser, Windows and macOS builds running.
|
||||
|
||||
## Gaps (what is not built or not measured)
|
||||
|
||||
- **Reddit viability is unmeasured from a residential line.** The only real run used a datacenter donor and got 403, which proves the path, not the success rate. X, Instagram, LinkedIn, Facebook and TikTok stay unreachable (login walls); nothing here claims them.
|
||||
- `country`, `asn` and `asn_class` on a device are never filled, so the datacenter exclusion (D6) is not enforced: the scheduler treats every node as eligible. It needs an offline ASN database file.
|
||||
- `last_seen_at` is written on connect only. Socket counters are per process. The fabric cache is per process. Circuit breaker state is in memory. The same-process fast path is not built, callers poll at `poll_interval_ms`.
|
||||
- Fetch-mode results are not cross-checked (sampled redundancy between two donors is not built). They are structurally checked and labelled `via: fetch`, and `tunnel` is preferred whenever a node can do it.
|
||||
- `DeepSearch` still skips hostile domains with a static list; Reddit URLs from social search are used through their snippet. Routing a hostile URL fetch through `/nodes/internal/fetch` is not built.
|
||||
- The Go client has no battery-aware pause (`pause_on_battery` is stored and sent but ignored by it), and treats a timed suspension as permanent until restarted. The extension enforces its own local hours in the browser clock, not the server timezone.
|
||||
- Not measured: Reddit and X success rates per address class, WebSocket idle timeouts through `pravda.education` and `devplace.net`, socket capacity of one worker, a 24 hour soak. The caps in `LimitSpec` are starting ceilings, not measurements.
|
||||
- **Launch is blocked on a recorded legal decision** (platform terms, donor terms, ISP terms, jurisdiction). Nothing here is enabled for any account: `nodes_enabled` is `0` until an administrator switches it on.
|
||||
|
||||
## Rules for work in this directory
|
||||
|
||||
|
||||
Reference in New Issue
Block a user