Code Farm: per-plot auto-replant pause and crop-lock check; community sync and node work

Continuous Delivery could never be turned off once a crop was planted. Add a
one-shot per-plot "stop after harvest" flag (POST /game/plot/stop, Devii tool
game_plot_stop, game_plots.stop_after_harvest), cleared by the next plant,
autoreplant claim or Refactor. Autoreplant now also respects crop locks through
the shared _crop_lock predicate. Plan and measurements in happyplanting.md.

Also includes the pending community sync, node network and related work in the
tree; the full suite passed (4999) with all of it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

devplacepy/services/nodes/CLAUDE.md

@@ -121,20 +121,31 @@ For a `tunnel` or `any` job the scheduler prefers a node whose `hello.modes` inc
 
 `SocialSearch.search` fans one query out to adapters in parallel (`asyncio.gather`, 20 s each) and merges hits round-robin. Direct adapters (`adapters/direct.py`: Hacker News, Stack Overflow, GitHub issues, dev.to by tag, Mastodon by hashtag) need no donors. `adapters/reddit.py` is node-backed and only used when `nodes_social_via_nodes` is `1`; otherwise it is reported as `nodes_not_used`. Every adapter parses a response structurally and raises `AdapterError` on a forged or wrong shape; platforms that fail are listed under `unavailable` with a code and never filled with invented results. **Measured on this machine (2026-10-15, datacenter address):** Hacker News Algolia, Stack Exchange, GitHub search, dev.to tag API and Mastodon hashtag timelines answer; Bluesky `searchPosts` returns 403 without auth, Mastodon `/api/v2/search` returns nothing without login, Lobsters search sits behind a bot wall, dev.to has no usable text search, and Reddit returns 403 to this address. Those are why Bluesky and Lobsters are not adapters. Surfaces: `GET /nodes/social/search` (daily per-member quota `nodes_daily_social_user`, default 300), Devii `social_search`, the DeepSearch `deepsearch_social` option (off by default, signed-in owners only, goes through the HTTP route with the owner's key).
 
+## Server mechanisms added after the first build
+
+- **Address class (D6 enforced).** `geo.py` keeps DB-IP lite ASN and country data in a SQLite range file under `config.NODE_GEO_DIR` (`devplace nodes geo-sync [--dry-run] [--month]`, `geo-status`; CC BY 4.0, attribution on `/docs/privacy.html`). Measured: build about 10 s, 477362 ASN and 710834 country rows, 48 MB, lookup 0.17 ms. `addressclass.classify` maps an ASN to `residential`, `mobile`, `datacenter`, `business` or `unknown`. The scheduler only assigns jobs to `ELIGIBLE_CLASSES` (residential, mobile) unless a device carries `class_override == "allow"` (admin, `POST /admin/nodes/devices/{uid}/class`, audit `node.device.override`, Devii `node_admin_class_override` confirm gated) or `nodes_allow_unknown_class=1` admits unknown. Refusals are counted in `NodeScheduler.class_refusals`. A missing geo file is `counters.geo_unavailable` plus one warning, never a silent pass.
+- **Durable state.** `last_seen_at` writes are batched (`touch_devices`, `last_seen_write_seconds`). Counters flush as deltas into `node_counters` every `counter_flush_seconds` (lifetime totals survive restart). Circuit breaker state is saved to `node_circuits` (`snapshot`/`restore`, capped at `circuit_snapshot_max`). The fabric cache is `SharedResultCache`: an in-process L1 over the `node_cache` table, so a result fetched by one worker serves the others.
+- **Fast path.** `wake.py` `WakeHub` wakes the dispatch loop and a waiting caller in the same process the moment a job is queued or done; the DB poll remains the cross-process fallback. Measured end to end (one donor, 300 jobs): polling p50 209 ms, p95 214 ms; fast path p50 25 ms, p95 about 30 ms, max about 38 ms. A single 6.2 s cold-start outlier appeared once and did not reproduce.
+- **Sampled redundancy.** `redundancy_percent` (5, ceiling `redundancy_max_percent` 20) of fetch-mode jobs are shadowed to a second donor (`shadow_of`, `exclude_device`); `shape.py` compares structure, and a mismatch raises the flag `redundancy_mismatch` for a human. Counters: `redundancy_checked`, `redundancy_matched`, `redundancy_mismatches`, `redundancy_inconclusive`.
+- **Outcome counters per address class.** `outcome:<adapter>:<class>:<ok|blocked|failed>` in `node_counters`, shown on `/admin/nodes`. This is the instrument for the residential success rate; it reads `n = 0` until real donors exist.
+- **DeepSearch** routes hostile URLs through `POST /nodes/internal/fetch` with the owner's api key (`crawl.node_servable`, `node_page`) when `nodes_social_via_nodes` is on.
+- **Battery.** `on_battery` is a donor limit code (pause 300 s, retryable). `welcome.device_label` and `kick.until` are optional protocol fields; a suspended ticket 403 carries `until`.
+
+**Measured capacity (one worker, scratch server, 2000 simulated donors on this machine):** 2000 sockets connected in 8 s; RSS 156 MB at 250, 190 MB at 500, 241 MB at 1000, 341 MB at 2000; idle dispatch tick p50 67 ms (max 160 ms); event loop lag p99 78 ms idle; 14.2 jobs/s completed at 15 offered, 0 client failures. A soak at 2000 sockets, 10 jobs/s and 3 percent a minute churn ran 4844 s (about 81 minutes, not 24 hours): RSS 346 to 361 MB, 42909 of 42913 jobs done, 1 failed, 0 integrity failures, 2026 of 2026 redundancy checks matched, loop lag p99 73 to 106 ms. Races, with separate OS processes: 16 workers x 20 counter flushes summed exactly (1600 of 1600); 12 racing cache puts left 1 row and all 12 readers hit.
+
 ## Clients
 
-`nodes/client/` is the Go donor client (standard library only, `build.sh` cross-compiles five targets and writes `manifest.json`). `nodes/extension/` is the MV3 extension (Chrome and Firefox, fetch mode only, 49 `node --test` tests, one of which reads `spec.py` and fails on drift). `make node-client` builds both into `data/node_clients/`, which `GET /nodes/client-version` and `GET /nodes/download/{artifact}` serve (names must match a strict pattern, symlinks and the manifest are never served, the SHA-256 is computed from the file). **Verified end to end on 2026-10-15** against a scratch server on this machine: the Go client logged in, connected, answered a `tunnel` job for `reddit.com` (`via: tunnel`, status 403 because the donor was this same datacenter address) and stayed connected across repeated jobs. Not verified: a residential donor, the extension in a real browser, Windows and macOS builds running.
+`nodes/client/` is the Go donor client (standard library only, `build.sh` cross-compiles five targets and writes `manifest.json`). It reads battery state on Linux, macOS and Windows (`RefuseBattery`, `PauseOnBattery`) and holds a timed suspension until `until` (clamped 60 s to 24 h). `nodes/extension/` is the MV3 extension (Chrome and Firefox, fetch mode only; icons 16/32/48/128; a suspended state with `suspendedUntil`; active hours in the server timezone; the manifest sets `content_security_policy.extension_pages` explicitly because Firefox's default MV3 policy adds `upgrade-insecure-requests`, which rewrote `ws://127.0.0.1` to `wss://` and broke loopback connections). `make node-client` builds both into `data/node_clients/`, served by `GET /nodes/client-version` and `GET /nodes/download/{artifact}` (strict names, no symlinks, SHA-256 from the file). **Verified in real browsers:** Chromium via `tests/e2e/nodes/extension.py` (disclosure, connect, online, a fetch-mode reddit job answered with 403 via `fetch`, suspend, disable) and Firefox 158 with geckodriver (connected, job done with 403 and 190240 bytes, suspended with `until`, disable). The Firefox script is not in the repository; the method is in `nodes/extension/README.md`. Go: `go vet` and `go test` pass, cross-compiles clean.
 
 ## Gaps (what is not built or not measured)
 
-- **Reddit viability is unmeasured from a residential line.** The only real run used a datacenter donor and got 403, which proves the path, not the success rate. X, Instagram, LinkedIn, Facebook and TikTok stay unreachable (login walls); nothing here claims them.
-- `country`, `asn` and `asn_class` on a device are never filled, so the datacenter exclusion (D6) is not enforced: the scheduler treats every node as eligible. It needs an offline ASN database file.
-- `last_seen_at` is written on connect only. Socket counters are per process. The fabric cache is per process. Circuit breaker state is in memory. The same-process fast path is not built, callers poll at `poll_interval_ms`.
-- Fetch-mode results are not cross-checked (sampled redundancy between two donors is not built). They are structurally checked and labelled `via: fetch`, and `tunnel` is preferred whenever a node can do it.
-- `DeepSearch` still skips hostile domains with a static list; Reddit URLs from social search are used through their snippet. Routing a hostile URL fetch through `/nodes/internal/fetch` is not built.
-- The Go client has no battery-aware pause (`pause_on_battery` is stored and sent but ignored by it), and treats a timed suspension as permanent until restarted. The extension enforces its own local hours in the browser clock, not the server timezone.
-- Not measured: Reddit and X success rates per address class, WebSocket idle timeouts through `pravda.education` and `devplace.net`, socket capacity of one worker, a 24 hour soak. The caps in `LimitSpec` are starting ceilings, not measurements.
-- **Launch is blocked on a recorded legal decision** (platform terms, donor terms, ISP terms, jurisdiction). Nothing here is enabled for any account: `nodes_enabled` is `0` until an administrator switches it on.
+- **Reddit success from a residential line is unmeasured.** The counters exist and read `n = 0`; every real run so far used a datacenter donor and got 403. X, Instagram, LinkedIn, Facebook and TikTok stay unreachable (login walls); nothing here claims them.
+- **No 24 hour soak.** The longest run was about 81 minutes. The caps in `LimitSpec` are ceilings backed by that run, not by a day.
+- **WebSocket idle timeout through the public hostname is unmeasured.** It needs a production connection, which needs the owner's go-ahead.
+- The extension has no battery pause (browsers expose no battery state to extensions); the Go client has one.
+- The geo file needs a monthly `geo-sync`; between syncs a new ASN is `unknown`.
+- Windows and macOS client builds compile but were not run.
+- **Launch is blocked on a recorded legal decision** (platform terms, donor terms, ISP terms, jurisdiction). Nothing here is enabled for any account: `nodes_enabled` is `0` until an administrator switches it on. This is a human decision and no code closes it.
 
 ## Rules for work in this directory
 

devplacepy/services/nodes/addressclass.py

@@ -0,0 +1,66 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import re
+
+CLASS_RESIDENTIAL, CLASS_MOBILE, CLASS_DATACENTER, CLASS_UNKNOWN = (
+    "residential",
+    "mobile",
+    "datacenter",
+    "unknown",
+)
+ELIGIBLE_CLASSES = (CLASS_RESIDENTIAL, CLASS_MOBILE)
+
+DATACENTER_ASNS = frozenset(
+    {
+        16509, 14618, 8987, 7224,
+        15169, 396982, 19527,
+        8075, 8068, 8069,
+        24940, 213230,
+        16276,
+        14061, 393406,
+        63949, 20940, 16625,
+        20473,
+        31898, 792,
+        45102, 37963,
+        45090, 132203,
+        13335,
+        51167,
+        12876,
+        60781, 16265, 30633,
+        9009,
+        36351,
+        27357, 19994,
+        54113,
+        47583,
+        197540,
+        8560,
+        26496,
+        22612,
+    }
+)
+
+DATACENTER_WORDS = re.compile(
+    r"\b(hosting|cloud|datacenter|data[ -]cent(?:er|re)|vps|servers?|colo(?:cation)?|dedicated|cdn|vpn|proxy|hoster|webhost\w*)\b",
+    re.IGNORECASE,
+)
+MOBILE_WORDS = re.compile(
+    r"\b(mobile|mobil|mobilfunk|wireless|cellular|lte|gsm|3g|4g|5g)\b", re.IGNORECASE
+)
+
+
+def classify(asn: int | str | None, org: str | None) -> str:
+    try:
+        number = int(asn) if asn not in (None, "") else 0
+    except (TypeError, ValueError):
+        number = 0
+    name = (org or "").strip()
+    if number <= 0 and not name:
+        return CLASS_UNKNOWN
+    if number in DATACENTER_ASNS or DATACENTER_WORDS.search(name):
+        return CLASS_DATACENTER
+    if MOBILE_WORDS.search(name):
+        return CLASS_MOBILE
+    return CLASS_RESIDENTIAL
+

devplacepy/services/nodes/broker.py

@@ -12,6 +12,7 @@ from dataclasses import dataclass
 from devplacepy.database import nodes as store
 from devplacepy.services.nodes.policy import Target
 from devplacepy.services.nodes.spec import SPEC
+from devplacepy.services.nodes.wake import HUB
 
 LIMITS = SPEC.limits
 
@@ -72,19 +73,37 @@ def decode_body(body_b64: str) -> tuple[bytes, bool]:
 
 
 class JobBroker:
-    def enqueue(self, caller_kind: str, caller_id: str, target: Target, mode: str) -> str:
-        return store.enqueue_request(
-            caller_kind, caller_id, target.url, target.method, target.adapter, mode
+    def enqueue(
+        self,
+        caller_kind: str,
+        caller_id: str,
+        target: Target,
+        mode: str,
+        *,
+        shadow_of: str = "",
+        exclude_device: str = "",
+    ) -> str:
+        uid = store.enqueue_request(
+            caller_kind,
+            caller_id,
+            target.url,
+            target.method,
+            target.adapter,
+            mode,
+            shadow_of=shadow_of,
+            exclude_device=exclude_device,
         )
+        HUB.notify_queued()
+        return uid
 
     def queued(self, limit: int = LIMITS.dispatch_batch) -> list[dict]:
         return store.list_queued(limit)
 
-    def claim(self, uid: str, device_uid: str) -> bool:
-        return store.claim_request(uid, device_uid)
+    def claim(self, uid: str, device_uid: str, asn_class: str = "") -> bool:
+        return store.claim_request(uid, device_uid, asn_class)
 
     def complete(self, uid: str, http_status: int, headers: dict, body: bytes, truncated: bool) -> bool:
-        return store.finish_request(
+        done = store.finish_request(
             uid,
             result_status=http_status,
             result_headers=json.dumps(headers),
@@ -92,9 +111,15 @@ class JobBroker:
             result_bytes=len(body),
             truncated=truncated,
         )
+        if done:
+            HUB.job_done(uid)
+        return done
 
     def fail(self, uid: str, error: str, *, expired: bool = False) -> bool:
-        return store.fail_request(uid, error, expired=expired)
+        failed = store.fail_request(uid, error, expired=expired)
+        if failed:
+            HUB.job_done(uid)
+        return failed
 
     def expire(self) -> dict[str, int]:
         return store.expire_requests()
@@ -117,11 +142,22 @@ class JobBroker:
     async def wait(self, uid: str, timeout: float) -> BrokerResult:
         deadline = time.monotonic() + timeout
         pause = LIMITS.poll_interval_ms / 1000
-        while True:
-            row = store.get_request(uid)
-            if row is not None and row["status"] in TERMINAL:
-                return self.read(uid)
-            if time.monotonic() >= deadline:
-                self.fail(uid, "caller_timeout")
-                return self.read(uid)
-            await asyncio.sleep(pause)
+        event = HUB.job_event(uid)
+        try:
+            while True:
+                row = store.get_request(uid)
+                if row is not None and row["status"] in TERMINAL:
+                    return self.read(uid)
+                if time.monotonic() >= deadline:
+                    self.fail(uid, "caller_timeout")
+                    return self.read(uid)
+                if event is None:
+                    await asyncio.sleep(pause)
+                    continue
+                event.clear()
+                try:
+                    await asyncio.wait_for(event.wait(), pause)
+                except asyncio.TimeoutError:
+                    pass
+        finally:
+            HUB.forget(uid)

devplacepy/services/nodes/cache.py

@@ -2,12 +2,16 @@
 
 from __future__ import annotations
 
+import base64
 import hashlib
+import json
 import time
+from datetime import datetime, timezone
 from collections import OrderedDict
 from dataclasses import dataclass
 from typing import Callable
 
+from devplacepy.database import nodes as store
 from devplacepy.services.nodes.spec import SPEC
 
 LIMITS = SPEC.limits
@@ -70,7 +74,14 @@ class ResultCache:
         return entry
 
     def put(
-        self, method: str, url: str, status: int, headers: dict, body: bytes, truncated: bool
+        self,
+        method: str,
+        url: str,
+        status: int,
+        headers: dict,
+        body: bytes,
+        truncated: bool,
+        age_seconds: float = 0.0,
     ) -> bool:
         if self.ttl_seconds <= 0 or status >= 500 or truncated:
             return False
@@ -78,7 +89,9 @@ class ResultCache:
             return False
         key = cache_key(method, url)
         self._drop(key)
-        self._entries[key] = CachedResult(status, dict(headers), body, truncated, self._clock())
+        self._entries[key] = CachedResult(
+            status, dict(headers), body, truncated, self._clock() - age_seconds
+        )
         self._bytes += len(body)
         while self._bytes > self.max_bytes and self._entries:
             oldest = next(iter(self._entries))
@@ -88,3 +101,56 @@ class ResultCache:
     def clear(self) -> None:
         self._entries.clear()
         self._bytes = 0
+
+
+class SharedResultCache(ResultCache):
+    def get(self, method: str, url: str) -> CachedResult | None:
+        local = super().get(method, url)
+        if local is not None:
+            return local
+        row = store.cache_get(cache_key(method, url))
+        if row is None:
+            return None
+        try:
+            headers = json.loads(row["headers"] or "{}")
+            body = base64.b64decode(row["body"] or "")
+            stored = datetime.fromisoformat(row["stored_at"])
+        except (ValueError, TypeError):
+            return None
+        age = max(0.0, (datetime.now(timezone.utc) - stored).total_seconds())
+        if age >= self.ttl_seconds:
+            return None
+        super().put(method, url, int(row["status"]), headers, body, False, age_seconds=age)
+        entry = self._entries.get(cache_key(method, url))
+        if entry is not None:
+            self.misses -= 1
+            self.hits += 1
+        return entry
+
+    def put(
+        self,
+        method: str,
+        url: str,
+        status: int,
+        headers: dict,
+        body: bytes,
+        truncated: bool,
+        age_seconds: float = 0.0,
+    ) -> bool:
+        kept = super().put(method, url, status, headers, body, truncated, age_seconds)
+        if kept:
+            store.cache_put(
+                cache_key(method, url),
+                method.upper(),
+                url,
+                status,
+                json.dumps(dict(headers)),
+                base64.b64encode(body).decode(),
+                len(body),
+                self.ttl_seconds,
+            )
+        return kept
+
+    def clear(self) -> None:
+        super().clear()
+        store.cache_clear()

devplacepy/services/nodes/circuit.py

@@ -28,11 +28,14 @@ class CircuitBreaker:
         failures: int = LIMITS.breaker_failures,
         open_seconds: float = LIMITS.breaker_open_seconds,
         clock: Callable[[], float] = time.monotonic,
+        wall: Callable[[], float] = time.time,
     ) -> None:
         self.failures = max(1, failures)
         self.open_seconds = open_seconds
         self._clock = clock
+        self._wall = wall
         self._circuits: dict[Hashable, _Circuit] = {}
+        self.dirty = False
 
     def state(self, key: Hashable) -> str:
         circuit = self._circuits.get(key)
@@ -55,7 +58,8 @@ class CircuitBreaker:
             self._circuits[key].probing = True
 
     def success(self, key: Hashable) -> None:
-        self._circuits.pop(key, None)
+        if self._circuits.pop(key, None) is not None:
+            self.dirty = True
 
     def failure(self, key: Hashable) -> None:
         circuit = self._circuits.setdefault(key, _Circuit())
@@ -63,9 +67,49 @@ class CircuitBreaker:
         circuit.failures += 1
         if circuit.failures >= self.failures:
             circuit.opened_at = self._clock()
+        self.dirty = True
 
     def forget(self, key: Hashable) -> None:
-        self._circuits.pop(key, None)
+        if self._circuits.pop(key, None) is not None:
+            self.dirty = True
 
     def open_count(self) -> int:
         return sum(1 for key in list(self._circuits) if self.state(key) != CLOSED)
+
+    def snapshot(self) -> list[dict]:
+        now_clock, now_wall = self._clock(), self._wall()
+        rows = []
+        for key, circuit in list(self._circuits.items()):
+            state = self.state(key)
+            rows.append(
+                {
+                    "key": ":".join(str(part) for part in key) if isinstance(key, tuple) else str(key),
+                    "failures": circuit.failures,
+                    "state": state,
+                    "opened_at": now_wall - (now_clock - circuit.opened_at) if circuit.failures >= self.failures else 0.0,
+                    "seconds_left": max(0.0, self.open_seconds - (now_clock - circuit.opened_at))
+                    if state == OPEN
+                    else 0.0,
+                }
+            )
+        rows.sort(key=lambda item: (-item["failures"], item["key"]))
+        return rows
+
+    def restore(self, rows: list[dict]) -> int:
+        now_clock, now_wall = self._clock(), self._wall()
+        restored = 0
+        for row in rows:
+            key = tuple(str(row["key"]).split(":", 1))
+            if len(key) != 2:
+                continue
+            failures = int(row.get("failures") or 0)
+            opened_wall = float(row.get("opened_at") or 0.0)
+            if failures < self.failures or opened_wall <= 0:
+                continue
+            age = max(0.0, now_wall - opened_wall)
+            if age >= self.open_seconds * 2:
+                continue
+            circuit = _Circuit(failures=failures, opened_at=now_clock - age)
+            self._circuits[key] = circuit
+            restored += 1
+        return restored

devplacepy/services/nodes/dispatcher.py

@@ -3,16 +3,21 @@
 from __future__ import annotations
 
 import asyncio
+import base64
+import hashlib
+import json
 import logging
 import time
 from typing import Callable
 
+from devplacepy.database import get_int_setting
 from devplacepy.database import nodes as store
 from devplacepy.services.nodes.broker import BodyError, JobBroker, clean_headers, decode_body
 from devplacepy.services.nodes.circuit import CircuitBreaker
 from devplacepy.services.nodes.ledger import UsageLedger
 from devplacepy.services.nodes.policy import DestinationPolicy, PolicyError, Target
 from devplacepy.services.nodes.registry import NodeCounters, NodeRegistry
+from devplacepy.services.nodes.shape import INCONCLUSIVE, MATCH, compare, shape_of
 from devplacepy.services.nodes.scheduler import MEGABYTE, NodeScheduler
 from devplacepy.services.nodes.session import Inflight, NodeSession
 from devplacepy.services.nodes.spec import MODE_FETCH, MODE_TUNNEL, SPEC
@@ -31,9 +36,13 @@ REQUEST_OVERHEAD_BYTES = 256
 DONOR_LIMIT_PAUSE_SECONDS = 300
 CONCURRENCY_PAUSE_SECONDS = 5
 THROTTLED_STATUSES = (429,)
+BLOCKED_STATUSES = (401, 403, 429, 451)
+SHADOW_KIND = "shadow"
+REDUNDANCY_SETTING = "nodes_redundancy_percent"
 DONOR_LIMIT_CODES = {
     "bandwidth_exhausted": None,
     "outside_hours": DONOR_LIMIT_PAUSE_SECONDS,
+    "on_battery": DONOR_LIMIT_PAUSE_SECONDS,
     "concurrency_exceeded": CONCURRENCY_PAUSE_SECONDS,
 }
 REFUSAL_CODES = frozenset(item.code for item in SPEC.refusals) - set(DONOR_LIMIT_CODES) - {
@@ -91,16 +100,24 @@ class NodeDispatcher:
             self.broker.fail(uid, exc.code)
             self.counters.inc("jobs_refused")
             return 0
-        key = (target.destination.suffix, row["mode"])
+        excluded = row.get("exclude_device") or ""
+        key = (target.destination.suffix, row["mode"], excluded)
         if key in saturated:
             return 0
         owner = row["caller_id"] if row["caller_kind"] == "user" else ""
-        found = self.scheduler.candidates(target, row["mode"])
+        if excluded:
+            prior = self.registry.get(excluded)
+            owner = prior.user_uid if prior is not None else owner
+        found = [
+            item
+            for item in self.scheduler.candidates(target, row["mode"])
+            if item.device_uid != excluded
+        ]
         if not found:
             saturated.add(key)
             return 0
         session = self.scheduler.choose(found, owner)
-        if session is None or not self.broker.claim(uid, session.device_uid):
+        if session is None or not self.broker.claim(uid, session.device_uid, session.asn_class):
             return 0
         job = Inflight(
             job_id=uid,
@@ -110,10 +127,13 @@ class NodeDispatcher:
             url=target.url,
             started=self._clock(),
             bytes_up=len(target.url.encode()) + REQUEST_OVERHEAD_BYTES,
+            adapter=target.adapter,
+            shadow_of=row.get("shadow_of") or "",
         )
         session.inflight[uid] = job
         self.scheduler.mark_started(session, target)
         transport = self.scheduler.transport_of(session, target, row["mode"]) or MODE_FETCH
+        job.transport = transport
         store.set_request_transport(uid, transport)
         if transport == MODE_TUNNEL:
             self._tunnel_tasks[uid] = asyncio.create_task(self._run_tunnel(session, job, target))
@@ -140,6 +160,7 @@ class NodeDispatcher:
         self.counters.inc("jobs_failed")
         if node_fault:
             self.breaker.failure(("node", session.device_uid))
+            self._observe(session, job, "failed")
         store.add_event("job_failed", device_uid=session.device_uid, user_uid=session.user_uid, detail=code)
         self.ledger.record_failure(session.device_uid, session.user_uid)
 
@@ -196,6 +217,7 @@ class NodeDispatcher:
         if not self.broker.complete(job.job_id, status, headers, body, truncated):
             return
         self._account(session, job, status, job.bytes_up, len(body))
+        self._after_result(session, job, status, headers, body)
 
     async def _run_tunnel(self, session: NodeSession, job: Inflight, target: Target) -> None:
         try:
@@ -215,9 +237,8 @@ class NodeDispatcher:
             return
         if result.truncated:
             self.counters.inc("results_truncated")
-        if not self.broker.complete(
-            job.job_id, result.status, clean_headers(result.headers), result.body, result.truncated
-        ):
+        headers = clean_headers(result.headers)
+        if not self.broker.complete(job.job_id, result.status, headers, result.body, result.truncated):
             return
         self._account(session, job, result.status, result.bytes_up, result.bytes_down)
 
@@ -237,12 +258,84 @@ class NodeDispatcher:
         if not accepted:
             session.used_today = int(session.settings.get("bandwidth_mb_day") or 0) * MEGABYTE
         self.breaker.success(("node", session.device_uid))
+        self._observe(session, job, "blocked" if status >= 500 or status in BLOCKED_STATUSES else "ok")
         destination = ("destination", job.destination)
         if status >= 500 or status in THROTTLED_STATUSES:
             self.breaker.failure(destination)
         else:
             self.breaker.success(destination)
 
+    def _observe(self, session: NodeSession, job: Inflight, kind: str) -> None:
+        if job.shadow_of or not job.adapter:
+            return
+        try:
+            self.counters.observe(job.adapter, session.asn_class, kind)
+        except KeyError:
+            logger.warning("outcome counter name rejected for %s", job.adapter)
+
+    @staticmethod
+    def redundancy_percent() -> int:
+        configured = get_int_setting(REDUNDANCY_SETTING, LIMITS.redundancy_percent)
+        return max(0, min(configured, LIMITS.redundancy_max_percent))
+
+    @staticmethod
+    def sampled(uid: str, percent: int) -> bool:
+        if percent <= 0:
+            return False
+        return int(hashlib.sha256(uid.encode()).hexdigest()[:8], 16) % 100 < percent
+
+    def _after_result(self, session: NodeSession, job: Inflight, status: int, headers: dict, body: bytes) -> None:
+        if job.shadow_of:
+            self._compare_shadow(session, job, status, headers, body)
+            return
+        if job.transport != MODE_FETCH or not self.sampled(job.job_id, self.redundancy_percent()):
+            return
+        row = store.get_request(job.job_id) or {}
+        try:
+            target = self.policy.check(job.url, job.method, MODE_FETCH)
+        except PolicyError:
+            return
+        try:
+            self.broker.enqueue(
+                SHADOW_KIND,
+                job.job_id,
+                target,
+                MODE_FETCH,
+                shadow_of=job.job_id,
+                exclude_device=row.get("device_uid") or session.device_uid,
+            )
+        except store.QueueFull:
+            return
+
+    def _compare_shadow(self, session: NodeSession, job: Inflight, status: int, headers: dict, body: bytes) -> None:
+        original = store.get_request(job.shadow_of)
+        if original is None or original.get("status") != "done":
+            self.counters.inc("redundancy_inconclusive")
+            return
+        try:
+            first_body = base64.b64decode(original.get("result_body") or "")
+            first_headers = json.loads(original.get("result_headers") or "{}")
+        except (ValueError, TypeError):
+            self.counters.inc("redundancy_inconclusive")
+            return
+        verdict = compare(
+            shape_of(int(original.get("result_status") or 0), first_headers, first_body),
+            shape_of(status, headers, body),
+        )
+        self.counters.inc("redundancy_checked")
+        if verdict == MATCH:
+            self.counters.inc("redundancy_matched")
+        elif verdict == INCONCLUSIVE:
+            self.counters.inc("redundancy_inconclusive")
+        else:
+            self.counters.inc("redundancy_mismatches")
+            store.raise_flag(
+                "redundancy_mismatch",
+                device_uid=session.device_uid,
+                user_uid=session.user_uid,
+                detail=f"{job.destination} first={original.get('device_uid')} second={session.device_uid}",
+            )
+
     async def sweep_inflight(self) -> int:
         limit = LIMITS.job_timeout_seconds + LIMITS.job_grace_seconds
         now = self._clock()

devplacepy/services/nodes/fabric.py

@@ -11,7 +11,7 @@ from devplacepy.database import get_int_setting, get_table
 from devplacepy.database import nodes as store
 from devplacepy.services.base import BaseService
 from devplacepy.services.nodes.broker import FAILED, JobBroker
-from devplacepy.services.nodes.cache import ResultCache
+from devplacepy.services.nodes.cache import ResultCache, SharedResultCache
 from devplacepy.services.nodes.consent import ConsentGate
 from devplacepy.services.nodes.policy import DestinationPolicy, PolicyError, Target
 from devplacepy.services.nodes.spec import SPEC
@@ -27,6 +27,7 @@ RETRYABLE = frozenset(
         "send_failed",
         "concurrency_exceeded",
         "outside_hours",
+        "on_battery",
         "bandwidth_exhausted",
     }
 )
@@ -81,7 +82,7 @@ class NodeFabric:
     ) -> None:
         self.policy = policy or DestinationPolicy()
         self.broker = broker or JobBroker()
-        self.cache = cache or ResultCache()
+        self.cache = cache or SharedResultCache()
         self.gate = gate or ConsentGate()
 
     def available(self) -> FabricStatus:

devplacepy/services/nodes/geo.py

@@ -0,0 +1,293 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import csv
+import gzip
+import ipaddress
+import logging
+import os
+import sqlite3
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+
+from devplacepy import config
+from devplacepy.services.nodes.addressclass import classify
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass(frozen=True)
+class GeoSpec:
+    asn_url: str = "https://download.db-ip.com/free/dbip-asn-lite-{month}.csv.gz"
+    country_url: str = "https://download.db-ip.com/free/dbip-country-lite-{month}.csv.gz"
+    attribution: str = "IP geolocation by DB-IP.com (CC BY 4.0)"
+    max_rows: int = 2_000_000
+    min_asn_rows: int = 100_000
+    min_country_rows: int = 100_000
+    insert_chunk: int = 5000
+    download_timeout_seconds: int = 180
+    max_download_bytes: int = 64 * 1024 * 1024
+    months_back: int = 2
+    checks: tuple[tuple[str, int], ...] = (("1.1.1.1", 13335), ("8.8.8.8", 15169))
+
+
+SPEC = GeoSpec()
+
+SCHEMA = (
+    "CREATE TABLE geo_asn (family INTEGER NOT NULL, first BLOB NOT NULL, last BLOB NOT NULL, "
+    "asn INTEGER NOT NULL, org TEXT NOT NULL, PRIMARY KEY (family, first)) WITHOUT ROWID",
+    "CREATE TABLE geo_country (family INTEGER NOT NULL, first BLOB NOT NULL, last BLOB NOT NULL, "
+    "country TEXT NOT NULL, PRIMARY KEY (family, first)) WITHOUT ROWID",
+    "CREATE TABLE geo_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)",
+)
+
+
+UNKNOWN_COUNTRY = "ZZ"
+
+
+class GeoError(RuntimeError):
+    pass
+
+
+@dataclass(frozen=True)
+class GeoResult:
+    country: str = ""
+    asn: int = 0
+    org: str = ""
+    asn_class: str = "unknown"
+
+
+def pack(address: str) -> tuple[int, bytes]:
+    ip = ipaddress.ip_address(address.strip().strip("[]"))
+    if ip.version == 6 and ip.ipv4_mapped is not None:
+        ip = ip.ipv4_mapped
+    return ip.version, ip.packed
+
+
+class GeoDatabase:
+    def __init__(self, path: Path | None = None) -> None:
+        self.path = Path(path) if path else config.NODE_GEO_DB
+        self.warned = False
+
+    def available(self) -> bool:
+        return self.path.is_file() and self.path.stat().st_size > 0
+
+    def _connect(self) -> sqlite3.Connection:
+        return sqlite3.connect(f"file:{self.path}?mode=ro", uri=True, timeout=5)
+
+    def _range(self, conn: sqlite3.Connection, table: str, columns: str, family: int, packed: bytes):
+        row = conn.execute(
+            f"SELECT {columns}, last FROM {table} WHERE family = ? AND first <= ? "
+            "ORDER BY first DESC LIMIT 1",
+            (family, packed),
+        ).fetchone()
+        if row is None or row[-1] < packed:
+            return None
+        return row[:-1]
+
+    def lookup(self, address: str) -> GeoResult:
+        if not address or not self.available():
+            return GeoResult()
+        try:
+            family, packed = pack(address)
+        except ValueError:
+            return GeoResult()
+        try:
+            conn = self._connect()
+        except sqlite3.Error:
+            return GeoResult()
+        try:
+            asn_row = self._range(conn, "geo_asn", "asn, org", family, packed)
+            country_row = self._range(conn, "geo_country", "country", family, packed)
+        except sqlite3.Error:
+            return GeoResult()
+        finally:
+            conn.close()
+        asn, org = (int(asn_row[0]), str(asn_row[1])) if asn_row else (0, "")
+        country = str(country_row[0]) if country_row else ""
+        country = "" if country.upper() == UNKNOWN_COUNTRY else country
+        return GeoResult(country=country[:2].upper(), asn=asn, org=org, asn_class=classify(asn, org))
+
+    def status(self) -> dict:
+        if not self.available():
+            return {"available": False, "path": str(self.path)}
+        conn = self._connect()
+        try:
+            meta = dict(conn.execute("SELECT key, value FROM geo_meta").fetchall())
+        except sqlite3.Error:
+            meta = {}
+        finally:
+            conn.close()
+        return {"available": True, "path": str(self.path), **meta, "attribution": SPEC.attribution}
+
+
+def months(now: datetime | None = None) -> list[str]:
+    moment = now or datetime.now(timezone.utc)
+    found = []
+    for back in range(SPEC.months_back + 1):
+        year, month = moment.year, moment.month - back
+        while month < 1:
+            month += 12
+            year -= 1
+        found.append(f"{year:04d}-{month:02d}")
+    return found
+
+
+def parse_asn_rows(lines):
+    for row in csv.reader(lines):
+        if len(row) < 4:
+            continue
+        first, last, asn, org = row[0], row[1], row[2], row[3]
+        try:
+            family, low = pack(first)
+            _, high = pack(last)
+            number = int(asn)
+        except ValueError:
+            continue
+        yield family, low, high, number, org[:200]
+
+
+def parse_country_rows(lines):
+    for row in csv.reader(lines):
+        if len(row) < 3:
+            continue
+        try:
+            family, low = pack(row[0])
+            _, high = pack(row[1])
+        except ValueError:
+            continue
+        yield family, low, high, row[2][:2].upper()
+
+
+def _insert(conn: sqlite3.Connection, sql: str, rows) -> int:
+    count = 0
+    batch = []
+    for row in rows:
+        batch.append(row)
+        count += 1
+        if count > SPEC.max_rows:
+            raise GeoError(f"more than {SPEC.max_rows} rows, refusing to load")
+        if len(batch) >= SPEC.insert_chunk:
+            conn.executemany(sql, batch)
+            batch = []
+    if batch:
+        conn.executemany(sql, batch)
+    return count
+
+
+def build(asn_lines, country_lines, target: Path, month: str) -> dict:
+    temporary = target.with_suffix(".tmp")
+    temporary.unlink(missing_ok=True)
+    conn = sqlite3.connect(temporary)
+    try:
+        for statement in SCHEMA:
+            conn.execute(statement)
+        asn_rows = _insert(conn, "INSERT OR REPLACE INTO geo_asn VALUES (?,?,?,?,?)", parse_asn_rows(asn_lines))
+        country_rows = _insert(conn, "INSERT OR REPLACE INTO geo_country VALUES (?,?,?,?)", parse_country_rows(country_lines))
+        if asn_rows < SPEC.min_asn_rows or country_rows < SPEC.min_country_rows:
+            raise GeoError(
+                f"dataset too small (asn {asn_rows}, country {country_rows}), refusing to replace the current one"
+            )
+        conn.executemany(
+            "INSERT INTO geo_meta VALUES (?,?)",
+            [
+                ("month", month),
+                ("loaded_at", datetime.now(timezone.utc).isoformat()),
+                ("asn_rows", str(asn_rows)),
+                ("country_rows", str(country_rows)),
+            ],
+        )
+        conn.commit()
+    except Exception:
+        conn.close()
+        temporary.unlink(missing_ok=True)
+        raise
+    conn.close()
+    verify(temporary)
+    return {"month": month, "asn_rows": asn_rows, "country_rows": country_rows, "temporary": temporary}
+
+
+def verify(path: Path) -> None:
+    probe = GeoDatabase(path)
+    for address, expected in SPEC.checks:
+        found = probe.lookup(address).asn
+        if found != expected:
+            path.unlink(missing_ok=True)
+            raise GeoError(f"sanity check failed: {address} resolved to AS{found}, expected AS{expected}")
+
+
+def install(built: dict, target: Path) -> None:
+    os.replace(built["temporary"], target)
+
+
+async def _download(url: str, destination: Path) -> bool:
+    from devplacepy.stealth import stealth_async_client
+
+    async with stealth_async_client(timeout=SPEC.download_timeout_seconds, follow_redirects=True) as client:
+        async with client.stream("GET", url) as response:
+            if response.status_code == 404:
+                return False
+            if response.status_code != 200:
+                raise GeoError(f"{url} answered {response.status_code}")
+            total = 0
+            with open(destination, "wb") as handle:
+                async for chunk in response.aiter_bytes():
+                    total += len(chunk)
+                    if total > SPEC.max_download_bytes:
+                        raise GeoError(f"{url} exceeds {SPEC.max_download_bytes} bytes")
+                    handle.write(chunk)
+    return True
+
+
+async def fetch_month(month: str, workdir: Path) -> tuple[Path, Path] | None:
+    asn_file = workdir / f"asn-{month}.csv.gz"
+    country_file = workdir / f"country-{month}.csv.gz"
+    if not await _download(SPEC.asn_url.format(month=month), asn_file):
+        return None
+    if not await _download(SPEC.country_url.format(month=month), country_file):
+        return None
+    return asn_file, country_file
+
+
+def build_from_files(asn_file: Path, country_file: Path, target: Path, month: str) -> dict:
+    with gzip.open(asn_file, "rt", encoding="utf-8", newline="") as asn_handle, gzip.open(
+        country_file, "rt", encoding="utf-8", newline=""
+    ) as country_handle:
+        return build(asn_handle, country_handle, target, month)
+
+
+async def geo_sync(*, dry_run: bool = False, month: str = "", target: Path | None = None, workdir: Path | None = None) -> dict:
+    destination = Path(target) if target else config.NODE_GEO_DB
+    destination.parent.mkdir(parents=True, exist_ok=True)
+    scratch = Path(workdir) if workdir else destination.parent
+    scratch.mkdir(parents=True, exist_ok=True)
+    candidates = [month] if month else months()
+    for candidate in candidates:
+        files = await fetch_month(candidate, scratch)
+        if files is None:
+            continue
+        try:
+            built = build_from_files(files[0], files[1], destination, candidate)
+        finally:
+            for item in files:
+                item.unlink(missing_ok=True)
+        summary = {key: value for key, value in built.items() if key != "temporary"}
+        if dry_run:
+            built["temporary"].unlink(missing_ok=True)
+            return {**summary, "installed": False}
+        install(built, destination)
+        return {**summary, "installed": True}
+    raise GeoError(f"no dataset found for {', '.join(candidates)}")
+
+
+_shared: GeoDatabase | None = None
+
+
+def shared_geo() -> GeoDatabase:
+    global _shared
+    if _shared is None or _shared.path != config.NODE_GEO_DB:
+        _shared = GeoDatabase()
+    return _shared
+

devplacepy/services/nodes/registry.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import re
 from collections import Counter
 
 from devplacepy.database import get_int_setting
@@ -26,21 +27,57 @@ COUNTER_NAMES: tuple[str, ...] = (
     "jobs_refused",
     "integrity_failures",
     "results_truncated",
+    "geo_unavailable",
+    "redundancy_checked",
+    "redundancy_matched",
+    "redundancy_mismatches",
+    "redundancy_inconclusive",
 )
 
+OUTCOME_PREFIX = "outcome:"
+OUTCOME_KINDS = ("ok", "blocked", "failed")
+OUTCOME_PATTERN = re.compile(r"^outcome:[a-z0-9_]{1,24}:[a-z]{1,12}:(ok|blocked|failed)$")
+
+
+def outcome_name(adapter: str, asn_class: str, kind: str) -> str:
+    name = f"{OUTCOME_PREFIX}{adapter}:{asn_class or 'unknown'}:{kind}"
+    if not OUTCOME_PATTERN.match(name):
+        raise KeyError(name)
+    return name
+
 
 class NodeCounters:
     def __init__(self) -> None:
         self._counts: Counter[str] = Counter({name: 0 for name in COUNTER_NAMES})
+        self._outcomes: Counter[str] = Counter()
+        self._pending: Counter[str] = Counter()
 
     def inc(self, name: str) -> None:
         if name not in COUNTER_NAMES:
             raise KeyError(name)
         self._counts[name] += 1
+        self._pending[name] += 1
+
+    def observe(self, adapter: str, asn_class: str, kind: str) -> None:
+        name = outcome_name(adapter, asn_class, kind)
+        self._outcomes[name] += 1
+        self._pending[name] += 1
 
     def snapshot(self) -> dict[str, int]:
         return {name: self._counts[name] for name in COUNTER_NAMES}
 
+    def outcomes(self) -> dict[str, int]:
+        return dict(self._outcomes)
+
+    def take_pending(self) -> dict[str, int]:
+        taken = {name: value for name, value in self._pending.items() if value > 0}
+        self._pending.clear()
+        return taken
+
+    def restore_pending(self, taken: dict[str, int]) -> None:
+        for name, value in taken.items():
+            self._pending[name] += value
+
 
 class NodeRegistry:
     def __init__(self, counters: NodeCounters) -> None:
@@ -78,10 +115,10 @@ class NodeRegistry:
         if self._sessions.get(session.device_uid) is session:
             del self._sessions[session.device_uid]
 
-    async def kick(self, session: NodeSession, reason: str) -> None:
+    async def kick(self, session: NodeSession, reason: str, until: str = "") -> None:
         self.release(session)
         self.counters.inc("kicked")
-        await session.kick(reason)
+        await session.kick(reason, until)
 
     async def kick_all(self, reason: str) -> int:
         sessions = self.sessions()

devplacepy/services/nodes/scheduler.py

@@ -9,6 +9,8 @@ from datetime import datetime, timezone
 from typing import Callable
 from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
 
+from devplacepy.database import get_int_setting
+from devplacepy.services.nodes.addressclass import CLASS_UNKNOWN, ELIGIBLE_CLASSES
 from devplacepy.services.nodes.circuit import CircuitBreaker
 from devplacepy.services.nodes.policy import Target
 from devplacepy.services.nodes.registry import NodeRegistry
@@ -20,6 +22,8 @@ MEGABYTE = 1024 * 1024
 MINUTES_PER_DAY = 24 * 60
 
 SUPPORTED_MODES = (MODE_TUNNEL, MODE_FETCH)
+OVERRIDE_ALLOW = "allow"
+ALLOW_UNKNOWN_SETTING = "nodes_allow_unknown_class"
 
 
 def _minutes(value: str) -> int | None:
@@ -73,6 +77,7 @@ class NodeScheduler:
         self._wall = wall
         self._rng = rng or random.Random()
         self._destination_started: dict[str, float] = {}
+        self.class_refusals = 0
 
     @staticmethod
     def node_key(session: NodeSession) -> tuple[str, str]:
@@ -102,9 +107,30 @@ class NodeScheduler:
         last = self._destination_started.get(target.destination.suffix)
         return last is None or self._clock() - last >= interval
 
-    def node_ready(self, session: NodeSession, target: Target, transport: str) -> bool:
+    @staticmethod
+    def allow_unknown_class() -> bool:
+        return get_int_setting(ALLOW_UNKNOWN_SETTING, 0) == 1
+
+    @staticmethod
+    def class_allowed(session: NodeSession, allow_unknown: bool) -> bool:
+        if session.class_override == OVERRIDE_ALLOW or session.asn_class in ELIGIBLE_CLASSES:
+            return True
+        return allow_unknown and session.asn_class == CLASS_UNKNOWN
+
+    def node_ready(
+        self,
+        session: NodeSession,
+        target: Target,
+        transport: str,
+        allow_unknown: bool | None = None,
+    ) -> bool:
         if session.closed or self._clock() < session.paused_until:
             return False
+        if allow_unknown is None:
+            allow_unknown = self.allow_unknown_class()
+        if not self.class_allowed(session, allow_unknown):
+            self.class_refusals += 1
+            return False
         if transport not in session.modes:
             return False
         if len(session.inflight) >= session.max_concurrent:
@@ -125,11 +151,12 @@ class NodeScheduler:
     def candidates(self, target: Target, mode: str = MODE_ANY) -> list[NodeSession]:
         if not self.destination_ready(target) or not self.breaker.allows(self.destination_key(target)):
             return []
+        allow_unknown = self.allow_unknown_class()
         for transport in self.transports_for(target, mode):
             found = [
                 session
                 for session in self.registry.sessions()
-                if self.node_ready(session, target, transport)
+                if self.node_ready(session, target, transport, allow_unknown)
             ]
             if found:
                 return found

devplacepy/services/nodes/service.py

@@ -9,9 +9,14 @@ import time
 from devplacepy.database.nodes import (
     STATUS_ACTIVE,
     add_event,
+    cache_prune,
     effective_status,
+    flush_counters,
     get_devices_by_uids,
     get_settings_by_users,
+    load_circuits,
+    save_circuits,
+    touch_devices,
 )
 from devplacepy.services.base import BaseService, ConfigField
 from devplacepy.services.nodes.broker import JobBroker
@@ -23,6 +28,7 @@ from devplacepy.services.nodes.ledger import UsageLedger
 from devplacepy.services.nodes.registry import NodeCounters, NodeRegistry
 from devplacepy.services.nodes.scheduler import NodeScheduler
 from devplacepy.services.nodes.spec import SPEC
+from devplacepy.services.nodes.wake import HUB
 
 logger = logging.getLogger(__name__)
 
@@ -84,6 +90,8 @@ class NodesService(BaseService):
         )
         self.fabric = NodeFabric(broker=self.broker, gate=self.gate)
         self._last_prune = 0.0
+        self._last_touch = 0.0
+        self._last_flush = 0.0
         self._dispatch_task: asyncio.Task | None = None
 
     def ensure_dispatching(self) -> None:
@@ -91,6 +99,7 @@ class NodesService(BaseService):
         task = self._dispatch_task
         if task is not None and not task.done() and task.get_loop() is loop:
             return
+        HUB.attach()
         self._dispatch_task = loop.create_task(self._dispatch_loop())
 
     async def _dispatch_loop(self) -> None:
@@ -102,9 +111,12 @@ class NodesService(BaseService):
                 raise
             except Exception:
                 logger.exception("node dispatch tick failed")
-            await asyncio.sleep(pause)
+            await HUB.wait_queued(pause)
 
     async def on_enable(self) -> None:
+        restored = self.breaker.restore(load_circuits())
+        if restored:
+            self.log(f"restored {restored} open circuit(s)")
         self.ensure_dispatching()
 
     async def _refresh_settings(self, sessions) -> None:
@@ -127,11 +139,13 @@ class NodesService(BaseService):
         verdicts = self.gate.allows_many(item.user_uid for item in sessions)
         kicked = 0
         for session in sessions:
-            reason = self._reason_to_kick(
-                switched_on, devices.get(session.device_uid), verdicts[session.user_uid]
-            )
+            device = devices.get(session.device_uid)
+            if device is not None:
+                session.class_override = device.get("class_override") or ""
+            reason = self._reason_to_kick(switched_on, device, verdicts[session.user_uid])
             if reason:
-                await self.registry.kick(session, reason)
+                until = (device.get("suspended_until") or "").strip() if reason == "suspended" and device else ""
+                await self.registry.kick(session, reason, until)
                 add_event(
                     "kick",
                     device_uid=session.device_uid,
@@ -155,13 +169,37 @@ class NodesService(BaseService):
     def _prune_due(self) -> bool:
         return time.monotonic() - self._last_prune >= LIMITS.prune_interval_seconds
 
+    def persist_state(self, *, force: bool = False) -> None:
+        now = time.monotonic()
+        if force or now - self._last_touch >= LIMITS.last_seen_write_seconds:
+            self._last_touch = now
+            touch_devices(item.device_uid for item in self.registry.sessions())
+        if force or now - self._last_flush >= LIMITS.counter_flush_seconds:
+            self._last_flush = now
+            pending = self.counters.take_pending()
+            try:
+                flush_counters(pending)
+            except Exception:
+                self.counters.restore_pending(pending)
+                raise
+        if self.breaker.dirty:
+            self.breaker.dirty = False
+            try:
+                save_circuits(self.breaker.snapshot())
+            except Exception:
+                self.breaker.dirty = True
+                raise
+
     async def run_once(self) -> None:
         kicked = await self.reconcile()
         if kicked:
             self.log(f"disconnected {kicked} node(s)")
+        self.persist_state()
         if self._prune_due():
             self._last_prune = time.monotonic()
             pruned = self.ledger.prune()
+            removed = cache_prune(LIMITS.cache_max_bytes)
+            pruned["cache"] = removed["expired"] + removed["evicted"]
             if any(pruned.values()):
                 self.log(f"pruned {pruned}")
 
@@ -170,6 +208,7 @@ class NodesService(BaseService):
             self._dispatch_task.cancel()
             self._dispatch_task = None
         await self.registry.kick_all("service_stopped")
+        self.persist_state(force=True)
 
     def collect_metrics(self) -> dict:
         counts = self.counters.snapshot()

devplacepy/services/nodes/session.py

@@ -35,6 +35,9 @@ class Inflight:
     url: str
     started: float
     bytes_up: int
+    adapter: str = ""
+    shadow_of: str = ""
+    transport: str = ""
 
 
 JobSink = Callable[["NodeSession", Inflight, dict], Awaitable[None]]
@@ -46,6 +49,8 @@ class NodeSession:
         self.device = device
         self.device_uid: str = device["uid"]
         self.user_uid: str = device["user_uid"]
+        self.asn_class: str = device.get("asn_class") or "unknown"
+        self.class_override: str = device.get("class_override") or ""
         self.connected_at = time.monotonic()
         self.missed_pings = 0
         self.hello: dict = {}
@@ -96,11 +101,14 @@ class NodeSession:
         except Exception:
             pass
 
-    async def kick(self, reason: str) -> None:
+    async def kick(self, reason: str, until: str = "") -> None:
         if self.closed:
             return
         try:
-            await self.send("kick", reason=reason)
+            if until:
+                await self.send("kick", reason=reason, until=until)
+            else:
+                await self.send("kick", reason=reason)
         except Exception:
             pass
         await self.close(CLOSE_KICKED, reason)
@@ -135,6 +143,7 @@ class NodeSession:
             limits=LIMITS.describe(),
             destinations=[item.describe() for item in SPEC.destinations],
             settings=self.settings,
+            device_label=str(self.device.get("label") or ""),
         )
         return frame
 

devplacepy/services/nodes/shape.py

@@ -0,0 +1,70 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass
+
+MATCH, MISMATCH, INCONCLUSIVE = "match", "mismatch", "inconclusive"
+SIZE_BAND = 4.0
+SMALL_BODY_BYTES = 512
+NESTED_KEY_LIMIT = 64
+TITLE = re.compile(rb"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
+HEADING = re.compile(rb"<h1[^>]*>(.*?)</h1>", re.IGNORECASE | re.DOTALL)
+TAGS = re.compile(rb"<[^>]+>")
+SPACE = re.compile(rb"\s+")
+
+
+@dataclass(frozen=True)
+class Shape:
+    status: int
+    kind: str
+    signature: frozenset | str
+    size: int
+
+    @property
+    def clean(self) -> bool:
+        return 200 <= self.status < 400
+
+
+def _keys(value, depth: int = 0) -> frozenset:
+    if not isinstance(value, dict):
+        return frozenset()
+    found = set(value)
+    if depth < 2:
+        for key, item in list(value.items())[:NESTED_KEY_LIMIT]:
+            found.update(f"{key}.{inner}" for inner in _keys(item, depth + 1))
+    return frozenset(found)
+
+
+def _text(raw: bytes) -> str:
+    return SPACE.sub(b" ", TAGS.sub(b"", raw)).strip().decode("utf-8", "replace")[:200]
+
+
+def shape_of(status: int, headers: dict, body: bytes) -> Shape:
+    kind = str(headers.get("content-type", "")).split(";")[0].strip().lower()
+    size = len(body)
+    if "json" in kind:
+        try:
+            return Shape(status, "json", _keys(json.loads(body)), size)
+        except ValueError:
+            return Shape(status, "json-invalid", "", size)
+    if "html" in kind:
+        match = TITLE.search(body) or HEADING.search(body)
+        return Shape(status, "html", _text(match.group(1)) if match else "", size)
+    return Shape(status, kind or "unknown", "", size)
+
+
+def compare(first: Shape, second: Shape) -> str:
+    if not (first.clean and second.clean):
+        return INCONCLUSIVE
+    if first.kind != second.kind:
+        return MISMATCH
+    if first.signature != second.signature:
+        return MISMATCH
+    if min(first.size, second.size) >= SMALL_BODY_BYTES:
+        ratio = max(first.size, second.size) / max(1, min(first.size, second.size))
+        if ratio > SIZE_BAND:
+            return MISMATCH
+    return MATCH

devplacepy/services/nodes/spec.py

@@ -104,6 +104,11 @@ class LimitSpec:
     dispatch_batch: int = 50
     url_max_chars: int = 2048
     poll_interval_ms: int = 100
+    last_seen_write_seconds: int = 60
+    redundancy_percent: int = 5
+    redundancy_max_percent: int = 20
+    circuit_snapshot_max: int = 100
+    counter_flush_seconds: int = 10
 
     def describe(self) -> dict[str, int]:
         return dict(self.__dict__)
@@ -232,6 +237,7 @@ FRAMES: tuple[FrameSpec, ...] = (
             _f("limits", "object"),
             _f("destinations", "list"),
             _f("settings", "object"),
+            _f("device_label", "str", False),
         ),
     ),
     FrameSpec("ping", SERVER_TO_CLIENT),
@@ -258,7 +264,7 @@ FRAMES: tuple[FrameSpec, ...] = (
     ),
     FrameSpec("tunnel_close", SERVER_TO_CLIENT, (_f("conn_id", "str"),)),
     FrameSpec("settings_update", SERVER_TO_CLIENT, (_f("settings", "object"),)),
-    FrameSpec("kick", SERVER_TO_CLIENT, (_f("reason", "str"),)),
+    FrameSpec("kick", SERVER_TO_CLIENT, (_f("reason", "str"), _f("until", "str", False))),
 )
 
 DESTINATIONS: tuple[DestinationSpec, ...] = (
@@ -275,6 +281,7 @@ REFUSALS: tuple[RefusalSpec, ...] = (
     RefusalSpec("bandwidth_exhausted", "donor daily bandwidth cap reached"),
     RefusalSpec("concurrency_exceeded", "donor concurrency cap reached"),
     RefusalSpec("outside_hours", "outside the donor's active hours"),
+    RefusalSpec("on_battery", "donor is running on battery and asked to pause"),
     RefusalSpec("body_too_large", "response exceeds the body cap"),
     RefusalSpec("timeout", "job exceeded its timeout"),
 )

devplacepy/services/nodes/wake.py

@@ -0,0 +1,70 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import asyncio
+
+
+class WakeHub:
+    def __init__(self) -> None:
+        self.enabled = True
+        self._loop: asyncio.AbstractEventLoop | None = None
+        self._queued: asyncio.Event | None = None
+        self._jobs: dict[str, asyncio.Event] = {}
+
+    def attach(self) -> None:
+        self._loop = asyncio.get_running_loop()
+        self._queued = asyncio.Event()
+
+    def _set(self, event: asyncio.Event) -> None:
+        loop = self._loop
+        if loop is None or loop.is_closed():
+            return
+        try:
+            running = asyncio.get_running_loop()
+        except RuntimeError:
+            running = None
+        if running is loop:
+            event.set()
+        else:
+            loop.call_soon_threadsafe(event.set)
+
+    def notify_queued(self) -> None:
+        if self.enabled and self._queued is not None:
+            self._set(self._queued)
+
+    async def wait_queued(self, timeout: float) -> bool:
+        event = self._queued
+        if not self.enabled or event is None:
+            await asyncio.sleep(timeout)
+            return False
+        try:
+            await asyncio.wait_for(event.wait(), timeout)
+        except asyncio.TimeoutError:
+            return False
+        event.clear()
+        return True
+
+    def job_event(self, uid: str) -> asyncio.Event | None:
+        if not self.enabled or self._loop is None:
+            return None
+        try:
+            if asyncio.get_running_loop() is not self._loop:
+                return None
+        except RuntimeError:
+            return None
+        event = self._jobs.get(uid)
+        if event is None:
+            event = self._jobs[uid] = asyncio.Event()
+        return event
+
+    def job_done(self, uid: str) -> None:
+        event = self._jobs.get(uid)
+        if event is not None:
+            self._set(event)
+
+    def forget(self, uid: str) -> None:
+        self._jobs.pop(uid, None)
+
+
+HUB = WakeHub()