See research_residental_proxies.md. Userss are by default opted out, but can opt in and download required tools to make it happen on their profile page.
We label this as donating to the devplace community ecosystem. We have a beautiful infrastructure that is very reliable, barely costs maintenance and even better, almost free.
Our search engine destroys searchxng. Also, our search engine has actually a fallback provided by jina. I bough jina in someday but then decided not to use their stuff what ended up
as a nice backup system for failing searches. But what do we want? We also want reddit, X and all other social media to be part of our search results. And that is only possible 
when people contribute some of their resource. There are severa ways to do this. Choice is to members. We support all common options and you can configure yourself how much resource 
you want to donate. We will also count the amunt of data i/o / request count and display it as donated by you. Also! For the sake of our new agent system on pravda.education, we really 
needs this residental proy stuff to happen. Our platform destroys grok atm, but grok has the social media searches and they are important.

Please create a huge implementation plan of literally everything that needs to happen at the end of this file. Thats literally all for now.

---

# IMPLEMENTATION PLAN: Community Node Network ("Donate to the DevPlace Ecosystem")

Status: plan only, nothing built. Written against the repo as of 2026-10-14 and against `research_residental_proxies.md`.
Working name: **Nodes**. Code lives in `devplacepy/services/nodes/`, routers in `devplacepy/routers/nodes/`, user-facing label "Donate to the community".

## 0. Findings that change the shape of the plan

These come from reading the repo and the research file. Several of them contradict the framing in the brief, so they come first.

1. **There is no Jina code in this repo.** `grep -ri jina` matches only `plan.md`. The search engine is the external service `https://rsearch.app.molodetz.nl` (`crawl.RSEARCH_URL`, `services/devii/rsearch/`). The Jina fallback therefore lives in the rsearch codebase, not here. This repo only consumes rsearch from two places: `services/jobs/deepsearch/crawl.py::search_queries` and the Devii `rsearch*` tools. Where social search is implemented (here or in rsearch) is open decision D1.
2. **The repo already knows social sites block it.** `crawl.HOSTILE_DOMAINS` lists x.com, reddit.com, instagram, facebook, linkedin, tiktok, threads, youtube and skips them. Root `CLAUDE.md` records that reddit blocks the datacenter ASN outright and that the only fix is an IP change. So the problem is real and measured for Reddit. It is not measured for the others.
3. **A residential IP fixes IP-reputation blocks, not login walls.** Reddit public JSON endpoints work from a clean IP without login. X search, Instagram, LinkedIn, Facebook and TikTok mostly need a logged-in session. Donor nodes must NOT use the donor's cookies (privacy and liability), so those platforms stay mostly out of reach. X is limited to what is public (permalinks, oEmbed/syndication timelines). Promising "Reddit, X and all social media" would be a false claim. Phase 0 measures what actually works per platform before anything else is built.
4. **Several "social" sources need no donors at all**: Hacker News (Algolia API), Bluesky (public AppView), Mastodon (public instance APIs), Lobsters, StackExchange, GitHub, dev.to, YouTube oEmbed/transcripts. They are cheap, reliable, and cover a large part of what developers want. Build these as direct adapters first (Phase 6A). Nodes are only for the platforms that block datacenter IPs (Reddit first).
5. **Browser extensions cannot do TCP.** MV3 has no raw sockets. An extension can only run `fetch()`. So there are two fundamentally different transports, with different trust properties:
   - **Tunnel mode** (native client): the server opens an end-to-end TLS session to the destination THROUGH the donor's TCP socket. The donor sees only ciphertext plus the destination host. The donor cannot read or tamper with content because they cannot forge the destination's certificate. Our `curl_cffi` Chrome fingerprint stays intact. This is the better mode.
   - **Fetch mode** (extension): the donor's browser performs the HTTP request and returns the body. The donor sees the URL and plaintext response and CAN forge the response. Everything returned is untrusted and must be verified (section 5.6). TLS fingerprint is the donor's browser, which is fine.
6. **A plain web page or service worker cannot do this at all.** The research table rates them "Low"/"Poor". The reality is "impossible": a page cannot `fetch()` reddit.com cross-origin (CORS), and cannot open raw sockets. Both are dropped from scope. Documented so nobody revisits them.
7. **Open SOCKS5 is the wrong product.** The research recommends exposing a SOCKS5 listener. We do not need an open proxy. We need to reach a short, fixed list of public social domains for our own search. A destination-allowlisted exit (donor hard-refuses everything else) removes most of the abuse and liability surface the research file warns about (credential stuffing, ad fraud, illegal traffic). This is the single most important design decision and it is a clear rule, not a configurable maybe: **no open proxy, ever; allowlist enforced on server AND on the donor client.**
8. **Existing seams to reuse, none to rebuild:**
   - `stealth.stealth_async_client(proxy=...)` and `outbound_proxy_url`: a local node-backed forward proxy plugs into this with zero caller changes.
   - `messaging/tickets.py` + `ws_tickets`: single-use WS auth ticket flow (browsers cannot set WS headers).
   - `pubsub` lock-owner WS pattern (`4013` retry) for the node socket.
   - `BaseService` + `ConfigField` for the dispatcher service and admin knobs.
   - `LiveViewRelayService.VIEWS` + private `user.{uid}.*` pub/sub namespace for live donor stats.
   - `user_consents` / `CONSENT_KINDS` for opt-in (default opted out is already how `ai_third_party` works).
   - `utils.track_action` / `ACHIEVEMENTS` / `BADGE_CATALOG` for donor badges.
   - `GatewayUsageLedger.record_external` pattern for metering consumers.
   - `devii` `CONFIRM_REQUIRED`, `docs_api.endpoint()`, `DOCS_PAGES`, `audit` categories + `events.md`.
9. **Legal and ToS exposure is real and is risk number one.** Reddit and X actively litigate and rate-limit scraping; ISPs restrict commercial use of residential lines; donors' IPs will get rate-limited or banned by the platforms. The plan includes a legal review gate (Phase 11) before any public launch. This is not legal advice and the plan does not assume the outcome.
10. **Unknown that blocks scoping:** what the "new agent system on pravda.education" is and how it will call search. The plan gives it a stable internal interface (section 6) but its requirements (latency, volume, platforms) are open question Q3.

## 1. Decisions needed from you before Phase 2 (everything else proceeds without them)

| # | Decision | Recommendation | Why |
|---|----------|----------------|-----|
| D1 | Where does social search live: in devplacepy, or in the rsearch codebase? | devplacepy owns the node fabric and exposes an internal HTTP face; rsearch calls it (or devplacepy does the social search itself) | Nodes are tied to DevPlace accounts, consent, and the ledger. rsearch has no accounts. |
| D2 | Native client language | **Go**, static binary, one file per OS/arch | Research recommends Go/Rust; zero runtime for Windows users. Adds a Go toolchain to the build. Alternative: Python zipapp (no new toolchain, but needs Python on the donor machine). |
| D3 | Extension distribution | Sideloadable zip first (audience is developers), store submission later | Stores scrutinize proxy-like extensions; sideload ships sooner and the audience can handle it. |
| D4 | Donation reward | XP + badges + leaderboard only. No coins, no money, no purchasable anything | Root rule: no payment path, no purchasable randomness. Keeps app-store conditionals untriggered. |
| D5 | Minimum donor age | Adults only (`age_band` gate) | A minor must not share an IP/line. Needs a decision on how the existing `age_band` is read for legacy accounts with `NULL`. |
| D6 | Is a data center or a cloud VPS allowed as a "donor"? | No: residential/mobile-ASN heuristic, admin override | The whole value is residential reputation. A VPS donor is just another datacenter IP. |

## 2. Specification first (your rule 7: everything has a declared spec)

Nothing else is written until the spec exists, because the server, the Go client, the extension, the docs and the tests all derive from it.

1. `devplacepy/services/nodes/spec.py`: the single source of truth, as frozen dataclasses (`FrameSpec`, `LimitSpec`, `DestinationSpec`, `AdapterSpec`, `ProtocolSpec`). It declares:
   - protocol version (integer, with `min_supported`),
   - every frame type with its fields, direction, max size,
   - every hard cap in section 3,
   - the destination allowlist (domain suffix -> adapter -> allowed methods -> allowed ports -> rate limit),
   - donor-side hard refusals (non-public IPs, ports other than 80/443, non-allowlisted hosts).
2. `GET /nodes/spec.json` serves the spec so clients and tests read the same numbers. A contract test fails if the Go constants, the extension constants, the docs page, or the server drift from it.
3. `devplacepy/templates/docs/nodes-protocol.html` (generated tables from the spec, not hand-copied) and `docs/nodes.html` (user-facing). Registered in `DOCS_PAGES`.
4. A short `NODES.md` is NOT created; the nested `devplacepy/services/nodes/CLAUDE.md` is the engineering record, per repo convention.

## 3. Hard caps and floors (nothing unbounded)

All values below are starting ceilings, declared in `spec.py` and surfaced as `ConfigField`s only where an admin may lower them. Static ceilings, no autotuning.

| Cap | Initial value | On hit |
|-----|---------------|--------|
| Devices per user | 5 | 409 with reason |
| Connected nodes total (lock-owner worker) | 2000 | WS close `4029`, counter `nodes_rejected_capacity` |
| WS frame size | 256 KiB | close `1009`, counter |
| Response body per job | 2 MiB | truncate + flag `truncated=true`, never silently |
| Concurrent jobs per node | 2 (donor may lower to 1) | scheduler skips node |
| Per node, per destination domain | min 6 s between requests | scheduler skips node |
| Global per destination domain | admin-set rps ceiling | queue, then 429 to caller |
| Job timeout | 25 s | mark failed, count against node health |
| Job queue depth | 500 | reject new with 429 + counter |
| Caller wait | 15 s default, 60 s max | `FabricUnavailable` |
| Per caller-owner daily fabric requests | admin-set (default 200 user, 0 guest) | 429, no donor load |
| Donor daily bandwidth (donor-configurable) | default 500 MB, max 20 GB | node goes idle until reset |
| Heartbeat | ping every 20 s, evict after 2 missed | reconnect |
| Reconnect backoff | 1 s to 60 s, full jitter | n/a |
| Ticket TTL | 30 s single use (existing) | n/a |
| Result cache | 10 min TTL, 256 MiB total, 4 MiB per entry | LRU evict |
| Usage daily buckets | keep 400 days, rolled into lifetime totals first | prune only after rollup verified |
| Node event log | 30 days, 50k rows | prune |

**Floors (machines hold, humans decide):**
- The system may auto-**suspend** a misbehaving node (reversible, timed, e.g. circuit breaker after N consecutive failures or an integrity mismatch). It may never auto-**ban** a user or delete a ledger row. Hitting a ban threshold creates a flag for an admin and fires an admin notification.
- Lifetime donation totals live on the donor row and are never decreased by pruning.
- Protected destinations: the allowlist can only be changed by the primary admin and every change is audited.
- A global kill switch (`nodes_enabled` site setting) disconnects every node within one tick. It is the first thing built in Phase 4 and tested first.

## 4. Data model

All tables ensured in `database/schema.py::init_db` (full column set, because `dataset` only creates the columns of the first insert), indexes with `CREATE INDEX IF NOT EXISTS`, every INSERT writes `deleted_at: None, deleted_by: None` where the table is soft-deletable. Helpers in a new `database/nodes.py`, exported through `database/__init__.py` like `skills.py`.

| Table | Purpose | Soft delete? | Notes |
|-------|---------|--------------|-------|
| `node_devices` | one row per registered client install: `uid`, `user_uid`, `label`, `kind` (`native`/`extension`/`container`), `secret_hash`, `secret_prefix`, `platform`, `client_version`, `protocol_version`, `status` (`active`/`suspended`/`revoked`), `suspended_until`, `suspend_reason`, `country`, `asn`, `asn_class` (`residential`/`mobile`/`datacenter`/`unknown`), `ip_hash`, `last_seen_at`, `created_at` | yes (revoke = soft delete, cascade stamp) | index `(user_uid, deleted_at)`, `(status, last_seen_at)` |
| `node_settings` | per-user donation prefs: `user_uid` (unique), `enabled`, `bandwidth_mb_day`, `max_concurrent`, `active_from`, `active_to`, `timezone`, `allowed_platforms` (JSON), `pause_on_battery`, `updated_at` | no (preference row) | enabled is subordinate to the consent, like `ai_correction_enabled` |
| `node_usage_daily` | per `(device_uid, day)` counters: `requests_ok`, `requests_failed`, `bytes_up`, `bytes_down`, `cache_hits_served`, `integrity_failures` | no (derived, hard GC) | unique `(device_uid, day)`; upserted with one atomic `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` (the `_add_usage` precedent) |
| `node_donors` | lifetime totals per user: `user_uid` (unique), `requests_ok`, `bytes_total`, `first_donated_at`, `last_donated_at`, `xp_awarded_today`, `xp_day` | no (derived) | survives bucket pruning; this is what the profile card and leaderboard read |
| `node_requests` | the cross-worker job queue: `uid`, `caller_kind`, `caller_id`, `url`, `method`, `adapter`, `mode` (`tunnel`/`fetch`/`any`), `status` (`queued`/`assigned`/`done`/`failed`/`expired`), `device_uid`, `result_status`, `result_bytes`, `result_body` (capped), `error`, `created_at`, `assigned_at`, `finished_at` | no (ephemeral, hard GC after 10 min) | partial index on `status IN ('queued','assigned')` |
| `node_events` | operational log: connects, disconnects, suspensions, verification results, refusals | no (hard GC, 30 days) | feeds admin view, not audit |
| `node_flags` | items needing a human: integrity mismatch bursts, datacenter-ASN donor, abuse reports against a node | yes | admin resolves; the floor rule's landing place |

Related registrations (the repo fails the suite otherwise):
- `database/moderation.py`: none of these carry user-generated content visible to others, so each goes in `UNREPORTABLE_TABLES` WITH its reason. `node_devices.label` is owner-only text; confirm it is never rendered to anyone else, or it becomes reportable.
- `SOFT_DELETE_TABLES`: `node_devices`, `node_flags`.
- `services/moderation/deletion.py::OWNED_TABLES`: add `node_devices`, `node_settings`, `node_donors` so account deletion cascades. Add `node_usage_daily` via the device join. Revoke live sockets in the same call.
- `user_consents`: new `CONSENT_KINDS["bandwidth_sharing"]`. **Never granted by default, never backfilled.** A new consent is enforced at ONE choke point (section 5.3), never at N call sites.
- Site settings seeded in `init_db` `operational_defaults`: `nodes_enabled` (default `0`), `nodes_max_total`, `nodes_daily_quota_user`, `nodes_daily_quota_guest`, `nodes_cache_ttl_seconds`, `nodes_social_via_nodes` (route hostile domains through nodes: default `0`).

## 5. Server

### 5.1 Package layout (OO, one responsibility per class, one class per file where it is a class)

```
devplacepy/services/nodes/
  CLAUDE.md                 engineering record (written in Phase 12, started in Phase 3)
  spec.py                   ProtocolSpec and friends (section 2)
  registry.py               NodeRegistry: connected sessions, capacity, lookup by uid/user
  session.py                NodeSession: one WS, frame codec, in-flight jobs, health, heartbeat
  frames.py                 Frame encode/decode + validation against spec
  consent.py                ConsentGate: THE single choke point (5.3)
  policy.py                 DestinationPolicy: allowlist, port/IP rules, per-domain limits
  scheduler.py              NodeScheduler: pick a node (fairness, diversity, caps)
  broker.py                 JobBroker: enqueue, assign, await result, expire (DB-backed)
  dispatcher.py             NodeDispatcher: lock-owner loop moving node_requests to sessions
  tunnel.py                 TunnelBridge: local forward-proxy listener -> tunnel frames
  fetcher.py                FetchBridge: fetch-mode request/response framing
  verifier.py               NodeVerifier: canary, ASN class, redundancy checks
  ledger.py                 UsageLedger: atomic counters, rollups, XP award hook
  cache.py                  ResultCache: bounded, URL-hash keyed
  circuit.py                CircuitBreaker: per node and per destination
  fabric.py                 NodeFabric: the facade every consumer calls
  service.py                NodesService(BaseService): reconcile, metrics, prune, relay hook
  adapters/
    base.py                 SearchAdapter (abstract): build_request, parse_response, normalize
    reddit.py  x.py  ...    one class per platform
    direct/                 hackernews.py bluesky.py mastodon.py lobsters.py stackexchange.py
```

### 5.2 Routers (`devplacepy/routers/nodes/` package, mounted at `/nodes` in `main.py`)

| Route | Auth | Purpose |
|-------|------|---------|
| `GET /nodes/spec.json` | public | protocol spec for clients and tests |
| `GET /nodes/client-version` | public | latest client versions, `min_protocol`, sha256 per artifact |
| `GET /nodes/download/{artifact}` | public | native binaries, extension zip, systemd unit, container compose (static files from `DATA_DIR/node_clients/`, sha256 listed) |
| `POST /nodes/devices` | `require_user` + consent | create device, returns the secret ONCE |
| `POST /nodes/devices/{uid}/rotate` | owner | new secret, old revoked immediately |
| `POST /nodes/devices/{uid}/delete` | owner or admin | soft delete + kick socket |
| `POST /nodes/ticket` | device secret | exchange secret for a 30 s single-use WS ticket |
| `WS /nodes/ws` | ticket | the node socket; lock-owner gated, `4013` fast retry like `/pubsub/ws` |
| `POST /nodes/internal/fetch` | admin / API key, service accounts | the HTTP face of `NodeFabric.fetch` for agent systems |
| `GET /nodes/stats` | public | aggregate only: total donated bytes, requests, active donors (no per-user data) |
| `GET /nodes/leaderboard` | public | top donors, opt-in display |
| `POST /profile/{username}/donation` | owner | settings (bandwidth, hours, platforms) |
| `GET /admin/nodes`, `POST /admin/nodes/...` | admin | pool health, flags, suspend/unsuspend, allowlist, kill switch |

Rules from the repo applied to every route: `respond(request, template, ctx, model=XOut)` for the HTML+JSON faces, `*Out` schemas in `schemas/nodes.py` (a context key missing from `XOut` is silently dropped from JSON), Pydantic `Form` models in `models.py`, `RedirectResponse(..., status_code=302)` literal, no `respond()` ctx key colliding with a Jinja global (use `viewer_is_admin`), specific paths before catch-alls, `is_senior_admin` on every per-user admin mutation, delete = owner OR admin.

### 5.3 The consent choke point (`ConsentGate`)

One function, `ConsentGate.allows(user_uid) -> Verdict(reason)`, true only when ALL of: `bandwidth_sharing` consent granted and not withdrawn, account active (not suspended, not in deletion grace), adult `age_band` (D5), `node_settings.enabled`, global `nodes_enabled`. It is called in exactly three places:
1. `POST /nodes/devices` and `POST /nodes/ticket` (refuse to mint).
2. `WS /nodes/ws` accept (refuse to connect).
3. `NodeScheduler.eligible()` (a node whose owner stopped qualifying is never assigned work, and `NodeRegistry` kicks the socket on the next tick).

Withdrawing the consent takes effect within one dispatcher tick, not at next reconnect. Consent change is owner-only, exactly like `routers/profile/consent.py` (reuse `_owner_only`). **Devii may read status and may withdraw, but must not be able to grant** (a chat agent granting a bandwidth-sharing consent is not a consent). Verify how the existing `set_consent` Devii tool treats grants and restrict this kind there.

### 5.4 WebSocket and multi-worker model

- `/nodes/ws` is gated to the service-lock owner worker with the `4013` fast-retry close, so all node sockets converge on one process (precedent: `/pubsub/ws`, `/devii/ws`). This keeps `NodeRegistry` a single in-memory truth. Trade-off, stated: one worker carries all sockets, hence the 2000 cap, and a crash drops every node for a few seconds until they reconnect. Acceptable because reconnect is built in.
- Callers on ANY worker enqueue into `node_requests` (DB). The lock-owner `NodeDispatcher` ticks fast while at least one node is connected and a request is queued (target 200 ms, indexed partial query; the cost is measured in Phase 0, not assumed), assigns, and writes the result back. Callers `await` by short-sleep polling the row. A same-process fast path skips the table when the caller is already on the owner. This is the `JobService` philosophy (DB-backed, durable enough, no inter-worker RPC) with a tighter tick.
- Large bodies never go through SQLite long term: `result_body` is capped (2 MiB) and hard-GC'd after 10 minutes; the durable copy is the `ResultCache` entry, bounded in bytes.
- nginx: add an upgrade `location = /nodes/ws` to `nginx/nginx.conf.template` (the catch-all `location /` strips upgrade headers, and the file already has one per WS route). **Also verify on the `devplace.net` front host**, whose own proxy and SSH tunnel have separate idle timeouts; test both hostnames per the production-diagnosis procedure, never with `--resolve`.

### 5.5 Frame protocol (summary; the authoritative version is `spec.py`)

Client to server: `hello{protocol, client_version, kind, platform, caps, limits}`, `pong`, `job_result{job_id, status, headers, body_b64, truncated, timings}`, `job_error{job_id, code}`, `tunnel_data{conn_id, b64}`, `tunnel_close{conn_id}`, `usage{bytes_up, bytes_down}`.
Server to client: `welcome{device_uid, limits, allowlist_version, settings}`, `ping`, `job{job_id, method, url, headers, timeout}`, `tunnel_open{conn_id, host, port}`, `tunnel_data`, `tunnel_close`, `settings_update`, `kick{reason}`.
Every frame is validated against the spec on both ends. An unknown type or an oversize frame closes the socket with a loud reason and a counter (never silently ignored).

### 5.6 Trust, verification and integrity

Fetch-mode donors can forge content, and forged content flows into search results and into the AI agent. Controls, in order of strength:
1. **Prefer tunnel mode** whenever a tunnel-capable node is available: TLS makes tampering impossible. Fetch mode is only used for extension donors.
2. **Never trust fetch-mode bodies without checks**: structural validation by the adapter (must parse as the platform's JSON/HTML shape), size sanity, and **sampled redundancy**: a configured fraction of fetch-mode jobs (plus 100 % for a node's first N jobs) are also fetched through a second independent node or directly; a content-hash or structural mismatch increments `integrity_failures`, suspends the node, and writes a `node_flags` row.
3. **Provenance on every result**: results carry `via=tunnel|fetch|direct` and the agent/search layer can down-weight or label fetch-mode content.
4. **Canary / liveness proof**: on connect, the verifier makes the node fetch `GET /nodes/canary/{nonce}` (served by us, records the observed source IP). Proves egress really goes out of the donor's own line, detects a node that is itself behind another proxy, and lets us compare the observed IP with the WS peer IP.
5. **ASN class**: classify residential/mobile/datacenter via an offline ASN database file under `DATA_DIR` (no third-party lookup, so no donor IP leaves our server). Datacenter donors get `asn_class=datacenter` and are excluded from the pool by default (D6) with a flag for an admin.
6. Donor client strips cookies and auth: extension uses `credentials: "omit"` and never attaches the donor's session; native client sends only the headers the server specified.

### 5.7 Donor-side safety is enforced on the donor, not only on the server

A compromised or buggy server must not be able to point a donor's machine at their LAN. The clients enforce, independent of any server frame:
- allowlisted hosts only (list baked in per release and refreshed from the server, where a refreshed list can only be equal or narrower than the baked-in superset),
- ports 80 and 443 only,
- resolve DNS locally, refuse private/loopback/link-local/CGNAT/multicast addresses, then connect to the resolved IP (defeats DNS rebinding),
- no redirects to a non-allowlisted host,
- the donor's configured bandwidth, concurrency and schedule caps, applied locally even if the server sends larger ones.

## 6. Consumers: how search gets the data

`NodeFabric` is the only thing consumers import:

```
NodeFabric.fetch(url, *, owner, mode="any", cache=True, timeout=...) -> FabricResult
NodeFabric.social_search(query, *, platforms, owner, limit) -> list[SocialHit]
NodeFabric.available() -> FabricStatus   # nodes online, per-platform health
```

Resolution order (each step loud when it fails, never silent):
1. `ResultCache` hit.
2. **Direct adapters** for platforms that need no donors (Hacker News, Bluesky, Mastodon, Lobsters, StackExchange, GitHub, dev.to).
3. **Nodes** for allowlisted blocked platforms (Reddit first), tunnel mode preferred.
4. Configured `outbound_proxy_url` if the admin has one.
5. rsearch (existing, includes its own Jina fallback).
6. `FabricUnavailable` with a reason code; the caller reports "social results unavailable", never invents any.

Integration points, one commit each:
- `services/jobs/deepsearch/crawl.py`: `search_queries` gains a social bucket merged by the existing `_interleave`; `HOSTILE_DOMAINS` stops being a static skip list and becomes `fabric.can_serve(url)`: a hostile URL is skipped only when the fabric cannot serve it. `fetch_page` routes those URLs through `NodeFabric.fetch`. The `rsearch` NDJSON ledger frame gets a sibling `fabric` frame so `DeepsearchService._run_worker` meters it like `record_external`.
- Devii: tools `social_search`, `fetch_social_page` (`requires_auth=True`, metered, guests refused), plus owner tools `node_status`, `node_settings_get`, `node_settings_set`, `node_device_list`, `node_device_revoke` (`CONFIRM_REQUIRED` + declared `confirm` param, per the rule). Update `SYSTEM_PROMPT` so the model prefers platform data, then `social_search` only when asked about outside discussion, and never claims a platform it cannot reach.
- `POST /nodes/internal/fetch`: the stable interface for the pravda.education agent system and any external caller. Auth: API key / Bearer, rate-limited, per-owner daily quota, documented.
- Cost and fairness: every fabric call is metered into a `fabric_usage` view per caller (requests, bytes, cache hit ratio) so a single caller cannot drain the donor pool; caller quota is checked before any donor is touched.

## 7. Clients

All clients speak the same protocol from `spec.json`. Conformance is tested against the Python reference client (Phase 10), not by eyeballing.

### 7.1 Native node (D2: Go)
- Single static binary, linux/amd64, linux/arm64, darwin/arm64, darwin/amd64, windows/amd64. Tunnel mode (TCP relay) and fetch mode.
- Config file + flags; `devplace-node login` takes the one-time device secret; `devplace-node status`; `devplace-node run`.
- Reconnect with full-jitter backoff, 20 s ping, local bandwidth/schedule/concurrency enforcement, battery-aware pause on laptops, quiet by default, structured log with a per-day totals line.
- Packaging: systemd **user** unit (`loginctl enable-linger` documented), Windows Service/scheduled task, macOS LaunchAgent, and a Docker/Podman compose snippet (the repo already ships `docker-compose.podman.yml`, so container users are a known audience).
- Built reproducibly from a `nodes/client/` directory with a `make node-client` target; artifacts + sha256 land in `DATA_DIR/node_clients/`; the version manifest is generated, not hand-written.

### 7.2 Browser extension (MV3, Chrome + Firefox)
- Fetch mode only. Service worker holds the WS; 20 s keepalive ping plus a `chrome.alarms` reconnect backstop; state in `chrome.storage.session`.
- `fetch()` with `credentials: "omit"`, header rewriting via `declarativeNetRequest` for the headers the spec allows, allowlist enforced before every request, response size cap.
- Honest, prominent disclosure on first run: what is shared, what is not, the on/off switch, live counters.
- Distribution per D3.

### 7.3 Later (explicitly out of this plan's first release)
Tauri tray app, signed auto-updates, mobile. Mobile stays out: iOS forbids the background behaviour and Android is restricted, as the research notes.

## 8. Frontend

Rules: responsive down to small phones (a wide button group becomes a dropdown), ES6 modules one class per file instantiated on `app`, tokens only in CSS, no literals, no `var()` fallbacks, `static_url`/`assetUrl` for assets, dates via `local_dt`, no `data-render` on server content, presence/pubsub via the existing managers.

1. **Profile tab "Donate"** (owner-only; others see only a donor badge and lifetime totals):
   - plain-language explainer first ("What you share, what you never share"), the consent toggle second, everything else only after consent. Product stays simple for the member: one switch, three sliders, a download card.
   - bandwidth per day, concurrent connections, active hours, platform checkboxes, pause-on-battery.
   - devices list with last seen, country/ASN class, bytes today, rotate and remove.
   - **"Donated by you" card**: requests served, data transferred (up/down), first/last donated, this month vs lifetime. Server-counted only.
   - download card: per-OS choice rendered as a dropdown on narrow screens, checksums shown, copy-paste install lines (systemd, docker), extension zip.
   - live numbers via a private pub/sub topic `user.{uid}.nodes` published by the lock-owner relay (add one row to `LiveViewRelayService.VIEWS`, re-checking the owner in the compute callable like the workspace topic), with a 30 s HTTP fallback poll.
2. Files: template `templates/_node_donation.html` partial included by `profile.html`; `static/js/NodeDonation.js` (`app.nodeDonation`); CSS `static/css/nodes.css` using existing tokens; reuse `Http`, `Poller`, `Tabs.js`, `dp-dialog` for confirmations.
3. **Public surfaces**: landing/footer line "The community has donated X GB across Y requests" from `GET /nodes/stats` (aggregate only), a `Top donors` section on the leaderboard (opt-in display), donor badge via `_award_badge.html` / `Avatar.badgeElement` reuse.
4. Admin: `/admin/nodes` (templates `admin_nodes.html`): pool size by country/ASN class, per-platform success rate, queue depth, flags, suspend/unsuspend, caps, allowlist, kill switch; live via `admin.nodes` topic row in `VIEWS`; `ServiceMonitor` already covers the service tab.

## 9. Gamification, incentives and anti-fraud

- `ACHIEVEMENTS`: `node.first_donation`, `node.1gb`, `node.10gb`, `node.100gb`, `node.30_days`, in `BADGE_CATALOG` group "Community".
- XP is awarded only for server-verified successful jobs, with a hard daily XP cap per donor (`node_donors.xp_awarded_today`), so synthetic traffic cannot farm progression. Through `award_rewards` only (never wrapped in `background.submit`).
- Counters are written server-side from what the server actually relayed; the donor's self-reported `usage` frame is advisory and is compared, never trusted.
- Self-dealing guard: a donor's own account cannot be the caller whose jobs are assigned to that donor's nodes for reward purposes.
- No money, no coins, no random rewards (D4).

## 10. Audit, privacy, safety, compliance

1. **Audit events** (`events.md` + `category_for` + recorder at the mutation point, via the `audit-event` skill): `node.consent.grant`, `node.consent.withdraw`, `node.device.create`, `node.device.rotate`, `node.device.delete`, `node.settings.update`, `node.suspend`, `node.unsuspend`, `node.flag.resolve`, `node.allowlist.update`, `node.killswitch`. Connects/disconnects/jobs are metrics and `node_events`, not audit rows (volume).
2. **Privacy (this is a new category of personal data plus a third-party disclosure, so it ships WITH the feature)**: update `templates/docs/privacy.html` and the app-store privacy declarations. State: IP address and approximate location are processed; only a salted hash and coarse country/ASN class are stored; retention periods for `node_events`/hashes; what the donor can see (destination host, and in fetch mode the URL and body); that the platform's content passes through the donor machine; the explicit statement that the donor's cookies and accounts are never used. Plain-English donor terms page, ISP-terms warning, consent text, and a visible off switch.
3. **App-store conditionals** (root rule): no payment path, no purchasable randomness, no cross-app tracking introduced. Hide the donate UI inside any app-store shell build. Verify against current store policy before submission rather than assuming.
4. **Abuse response**: report a node/destination path, admin suspend within one tick, allowlist narrowing, per-destination circuit breaker, the kill switch, and a documented contact. Because the allowlist is destination-restricted there is no path for credential stuffing, ad fraud or arbitrary traffic through a donor.
5. **Maturity/moderation**: fetched third-party content is untrusted. It goes through `render_content` (server) or `DOMPurify` (client) like everything else and through the existing content filter when it is persisted anywhere. Fabric results are never rendered as raw HTML.

## 11. Phased delivery

Each phase ends with: clean import, per-language checks, em-dash scan (character and entity), `pyflakes`/`ruff` on touched files, the four-layer verification where listed, the full `make test` (all three tiers), then commit (hook auto-bumps the version), deploy with `make docker-reload`, and verification live on both hostnames. A change that is not running is not done.

**Phase 0: Measure before building (no production code).** Scratch scripts against temp DB only.
- Real-machine experiment: a prototype WS relay (use `pywssocks` / the `residential-proxy-aggregator` design as reference) with one residential node and one datacenter node. Measure Reddit `.json` and `old.reddit.com` success rate per IP class, rate-limit thresholds per IP, X permalink/oEmbed/syndication behaviour without login, Instagram/LinkedIn/Facebook/TikTok reality. Output: a measured per-platform viability table committed to the plan. Claims without a number stay labelled "unmeasured".
- Tunnel-mode feasibility: confirm `curl_cffi` through an HTTP-CONNECT bridge preserves the Chrome fingerprint end to end and that the cleartext-HTTP/2 gotcha in the repo does not apply (https only).
- Capacity: simulated-node harness (our own, outside `locust`) for 100/500/2000 sockets on one worker: memory, CPU, frame latency, dispatcher tick cost against SQLite. These numbers set or replace the caps in section 3.
- Reconnect/idle behaviour through BOTH edges: `pravda.education` (molohttp) and `devplace.net` (front host + SSH tunnel). Measure the actual idle timeout of each.
- Exit criteria: D1, D2, D3, D5, D6 decided; viability table written; caps confirmed or changed.

**Phase 1: Specification and contracts.** `spec.py`, `/nodes/spec.json`, docs protocol page skeleton, contract-test scaffolding in `tests/unit/services/nodes/`.

**Phase 2: Data layer.** Tables, indexes, helpers in `database/nodes.py`, `CONSENT_KINDS` entry, `OWNED_TABLES`, `UNREPORTABLE_TABLES` with reasons, soft-delete registrations, settings seeds, `UsageLedger` with atomic upserts. Four-layer verification applies here (counters + daily caps are a spendable resource): property checks on cap arithmetic, stateful fuzz of increment/rollup/prune with invariants (totals never decrease, daily never exceeds cap), real multi-process race on the daily-cap `UPDATE ... WHERE` using `COALESCE(col, 0)` and a freshly-created unset row (not pre-zeroed).

**Phase 3: Registry, session, consent gate, kill switch.** `NodeRegistry`, `NodeSession`, `frames.py`, `ConsentGate`, `nodes_enabled` kill switch (built and tested first). Routers for devices/ticket/WS handshake only. No job traffic yet. nginx upgrade location. Reuse of the ticket mechanism by adding a `scope` column to `ws_tickets` (default `messages`) instead of a parallel table.

**Phase 4: Broker, scheduler, dispatcher, policy, circuit breakers, cache.** Job lifecycle `queued -> assigned -> done|failed|expired`, fairness, per-domain limits, queue cap with 429. Four-layer verification: property tests on scheduler fairness and cap enforcement; stateful fuzz of connect/disconnect/assign/complete/expire with invariants (a job is assigned to at most one node, a node never exceeds its concurrency, a withdrawn consent is never assigned work); real-process concurrency test of two dispatchers racing one slot (must be impossible by construction, since only the lock owner dispatches, and proven anyway).

**Phase 5: Python reference client + test harness.** A small reference node used by the api/e2e tests and as the protocol conformance oracle. Includes fault injection: dropped frames, slow nodes, oversize bodies, forged bodies, DNS-rebinding attempt, private-IP target.

**Phase 6A: Direct adapters** (no donors): Hacker News, Bluesky, Mastodon, Lobsters, StackExchange, GitHub, dev.to into `NodeFabric.social_search`, DeepSearch and the Devii tools. This ships user value before any donor exists.
**Phase 6B: Node-backed adapters**: Reddit first (per the Phase 0 table), then each additional platform only if Phase 0 measured it viable. `NodeVerifier` redundancy sampling for fetch mode.

**Phase 7: Native client (Go) + packaging** (systemd user unit, launchd, Windows service, compose). Cross-compile target, checksums, version manifest, conformance run against the reference oracle.

**Phase 8: Browser extension.** MV3 Chrome + Firefox build, disclosure screen, keepalive/alarms, conformance run, sideload zip served from `/nodes/download`.

**Phase 9: Frontend and admin.** Profile Donate tab, donated-by-you card, devices, downloads, live topic, leaderboard section, landing stat, `/admin/nodes`. Screenshot-verify with the `screenshot` skill at phone and desktop widths.

**Phase 10: Devii, docs, SEO, CLI.**
- Devii actions in `actions/catalog/nodes.py` (owner tools above), dispatcher/registry wiring, `CONFIRM_REQUIRED` for revoke, `requires_auth` matched to route guards, tool-group registration so the admin tool-prefs UI lists them.
- `docs_api/groups/nodes.py` `endpoint()` entries with `sample_response`; `DOCS_PAGES` for `nodes.html` and `nodes-protocol.html`; role-gating correct (protocol page admin/developer tier).
- SEO: Donate tab `noindex`; any public stats page indexable and in the sitemap.
- CLI: `devplace nodes list|stats|suspend|unsuspend|prune|clear`, destructive commands support `--dry-run` per the existing `accounts prune` convention (flag the conflict with the global dry-run-by-default rule to you if you want the default flipped; consistency with the existing CLI is the project rule).
- README, `devplacepy/services/nodes/CLAUDE.md`, a row in the root `CLAUDE.md` Subsystem map (a genuinely new architectural area), `locustfile.py` untouched for the WS (locust-maintainer: the node socket is deliberately excluded and the HTTP read routes get weighted tasks).

**Phase 11: Hardening and launch gate.**
- Legal review gate: platform ToS exposure per platform, donor terms, ISP warning text, jurisdiction. Launch is blocked on a recorded decision.
- Load and soak: 24 h soak with simulated nodes, reconnect storms (all sockets drop at once, then return), full-disk and DB-lock behaviour, worker restart mid-job. Numbers replace assumptions in `CLAUDE.md`.
- Security-maintainer pass (allowlist bypass, SSRF on the server's own fetch path, ticket replay, device secret leakage, admin seniority on node moderation).
- Full maintenance fleet (`fleet` skill): audit, background, devii, docs, dry, fanout, frontend, locust, security, seo, style, test dimensions.
- Rollout: `nodes_enabled=0` by default. Stage 1: your own machines only (including a residential line) for a week, measuring success rate, latency, donor-side CPU and bandwidth. Stage 2: invite-only developers. Stage 3: open, still behind per-caller quotas. The kill switch stays one click away.

**Phase 12: Documentation of record.** `devplacepy/services/nodes/CLAUDE.md` states what the system does today, with gaps listed next to the code that will close them (no roadmap cosplay). Measured numbers only, each labelled with the machine it came from.

## 12. Test plan (by tier, mirroring the path)

- `tests/unit/services/nodes/`: spec invariants and contract (docs, clients, server agree), frame codec, `DestinationPolicy` (allowlist, ports, private IPs, redirects, DNS rebinding), scheduler (fairness, per-domain interval, concurrency, bandwidth caps), circuit breaker, cache bounds and LRU, ledger math, ASN classification, XP daily cap, adapters (parse recorded fixtures, malformed input, forged-structure rejection).
- `tests/api/nodes/` (mirror the URL path): consent gating on every entry point, device create/rotate/revoke, secret shown once, ticket single-use and expiry, WS handshake with the reference client, full job round trip, withdrawal kicks the socket within a tick, kill switch, daily quotas, 429 paths, account deletion cascade, JSON and HTML faces for every route, admin seniority, `/nodes/stats` leaks no per-user data, `/nodes/internal/fetch` auth.
- `tests/e2e/profile/` (Playwright, `wait_until="domcontentloaded"`): enable flow with consent, device creation and one-time secret, settings persistence, live counter update, withdraw flow, responsive layout at phone width, owner-only visibility. Any test flipping a global setting restores it in `try/finally`.
- Rigorous verification scripts (disposable, temp DB via `DEVPLACE_DATABASE_URL`/`DEVPLACE_DATA_DIR`, never `data/`): the four layers from root `CLAUDE.md` for the ledger, scheduler and job state machine.
- The test hook that guards the production DB applies: no test or script touches `data/devplace.db`.

## 13. Risks, stated plainly

| Risk | Likelihood | Mitigation | Residual |
|------|-----------|------------|----------|
| Platform ToS/legal action against scraping | real | allowlist, legal gate, per-domain rate limits, kill switch | cannot be engineered away |
| Donors' IPs get rate-limited or banned by platforms | high on Reddit | per-donor-per-domain caps, caching, consent text says so | some donors will be unhappy; transparent counters help |
| X/Instagram/LinkedIn/TikTok unreachable (login walls) | high | Phase 0 measures; promise only what is measured | those platforms stay limited |
| Fetch-mode donor forges content | possible | tunnel preferred, structural validation, sampled redundancy, provenance labels, suspension | extension donors are lower trust, labelled as such |
| Small donor pool means flaky results | high at launch | cache, fallback chain, honest "unavailable" | search stays at today's quality, not worse |
| Single worker holds all sockets | by design | cap, fast reconnect, measured capacity | brief drop on worker crash |
| Donor privacy incident | low-impact by design | no cookies, hashed IP, destination allowlist, docs | disclosure must stay accurate |
| Scope creep into an open proxy | tempting | the "no open proxy" rule is in `spec.py` and the contract test | needs reviewer discipline |

## 14. Open questions for you

- Q1: Confirm D1 to D6 above (a one-line answer each is enough).
- Q2: Is there any residential line you control for the Phase 0 measurement, or do we recruit three donors for it?
- Q3: What does the pravda.education agent system need from search (volume, latency, which platforms, guest access)? It sets the daily quotas and the `/nodes/internal/fetch` contract.
- Q4: Should donor identity be public (leaderboard, badge) by default or opt-in? The plan assumes opt-in display.
- Q5: Who owns the legal review, and is a launch without it acceptable for the invite-only stage?