Add community node data layer: tables, usage ledger, bandwidth_sharing consent

Phase 2 of plan.md. Seven node_* tables with atomic device, bandwidth and XP caps, a transactional job ledger whose lifetime totals survive pruning, clamped donor settings, and the bandwidth_sharing consent kind. The acceptance service now has an explicit exclusion list so it can never grant that consent. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-14 08:32:17 +02:00
parent f55c7ee5f4
commit b616bce0ea
12 changed files with 1154 additions and 6 deletions
@@ -2,7 +2,7 @@ This file documents the community node network (`devplacepy/services/nodes/`, `d

 ## What exists today

-Only Phase 1 (the specification) is built. Everything else is planned and listed under "Gaps" next to the phase that closes it. Nothing here is enabled for any account.
+Phase 1 (the specification) and Phase 2 (the data layer and usage ledger) are built. Everything else is planned and listed under "Gaps" next to the phase that closes it. Nothing here is enabled for any account.

 | Piece | File | State |
 |-------|------|-------|
@@ -10,6 +10,9 @@ Only Phase 1 (the specification) is built. Everything else is planned and listed
 | `GET /nodes/spec.json` | `routers/nodes/spec.py` | built, tested, public |
 | Generated protocol docs page | `templates/docs/nodes-protocol.html` | built, reads the `nodes_spec` Jinja global |
 | API docs group | `docs_api/groups/nodes.py` | built, one endpoint |
+| Tables and helpers | `database/nodes.py`, `NODES_COLUMNS` in `database/schema.py` | built, tested, no route uses them yet |
+| Usage ledger | `ledger.py` (`UsageLedger`, `DonationSummary`) | built, tested |
+| `bandwidth_sharing` consent kind | `CONSENT_KINDS` in `database/moderation.py` | registered, never granted by default, enforced nowhere yet |

 ## The specification is the contract

@@ -35,13 +38,24 @@ The docs page `nodes-protocol.html` renders its tables from the `nodes_spec` Jin
 | D5 | donors must be adults (`age_band`) |
 | D6 | a datacenter address is not a donor: residential and mobile only, admin override |

+## Data layer (Phase 2)
+
+Seven tables, all ensured with their full column set in `init_db` (`_ensure_nodes_tables`): `node_devices`, `node_settings`, `node_usage_daily`, `node_donors`, `node_flags` (soft-deletable, in `SOFT_DELETE_TABLES` and `UNREPORTABLE_TABLES` with a reason), and `node_requests`, `node_events` (ephemeral, hard garbage collection only). Account deletion cascades through `OWNED_TABLES` for devices, settings, donors and daily usage.
+
+- **Atomic by one statement.** The device cap is an `INSERT ... SELECT ... WHERE (SELECT COUNT(*)...) < max`, the daily bandwidth cap is an `UPDATE ... WHERE COALESCE(bytes_up,0)+COALESCE(bytes_down,0)+:n <= :cap`, and the daily XP cap is a conditional `UPDATE` that rolls the day over inside the same statement. Verified with real separate processes against fresh, unseeded rows (40 racers on a 10-job cap gave exactly 10, 24 device creations gave exactly 5, 30 XP claims gave exactly the cap).
+- **`record_job` is one transaction**: the daily bucket (`INSERT OR IGNORE` then the conditional `UPDATE`) and the lifetime `node_donors` upsert commit together or not at all. A job refused by the cap counts nothing. A failed job counts `requests_failed` and never moves bytes or lifetime totals.
+- **Lifetime totals are maintained at write time**, so pruning the daily buckets (`usage_daily_retention_days`) can never lower them. Prune functions take `dry_run` and use only the constants in `LimitSpec`; there is no argument that lowers a retention floor.
+- **Secrets**: `dpn_` plus 32 random bytes, shown once at create or rotate, stored as sha256 plus a 12 character lookup prefix, compared with `hmac.compare_digest`. IP addresses are stored only as a keyed hash (`hash_ip`).
+- **Settings are clamped on write and on read** (`normalise_settings`): bandwidth to `1..max_donor_mb_per_day`, concurrency to `1..max_concurrent_per_node`, hours must be `HH:MM`, platforms are intersected with the spec's adapters. A member with no row is opted out.
+- Site settings seeded in `init_db`: `nodes_enabled` (`0`), `nodes_max_total`, `nodes_daily_quota_user`, `nodes_daily_quota_guest`, `nodes_cache_ttl_seconds`, `nodes_social_via_nodes` (`0`). Like every seed, consumers pass the default to `get_setting` because the seed does not run on an empty database.
+
 ## Two transports, two trust levels

 `tunnel` (native client) relays raw TCP so TLS runs end to end between DevPlace and the destination: the donor cannot read or forge content. `fetch` (browser extension, which has no raw sockets) lets the donor read and forge the response, so every `fetch` result is verified and labelled. A plain web page or a service worker cannot do either (CORS, no sockets) and is out of scope for good.

 ## Gaps (what the next phases close)

- Phase 2: tables, indexes, the `bandwidth_sharing` consent kind, `OWNED_TABLES` and `UNREPORTABLE_TABLES` entries, the usage ledger.
+- Phase 2 leftovers: the `bandwidth_sharing` consent can currently be granted through `POST /profile/{username}/consent` and by Devii's `set_consent` (Devii calls that route as the user). Nothing reads it yet, so it is inert, but Phase 3 must make Devii unable to grant it (Devii may read and withdraw) at the same time as `ConsentGate` lands. No XP or badge is awarded yet (`UsageLedger.claim_xp` only reserves the daily budget); the privacy page and app-store declarations are updated with the first phase that lets a device connect, because until then no personal data is collected.
 - Phase 3: registry, session, the consent choke point, the kill switch, device and ticket routes, the lock-owner WebSocket, the nginx upgrade location.
 - Phase 4: broker, scheduler, dispatcher, destination policy, circuit breakers, result cache.
 - Phase 5 onward: the reference client, adapters, the Go client, the extension, the profile UI, Devii tools, the CLI, the privacy and donor-terms pages, the legal gate.
@@ -0,0 +1,77 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime, timezone
+
+from devplacepy.database import nodes as store
+
+
+@dataclass(frozen=True)
+class DonationSummary:
+    lifetime_requests: int
+    lifetime_bytes: int
+    first_donated_at: str
+    last_donated_at: str
+    month_requests: int
+    month_bytes_up: int
+    month_bytes_down: int
+    today_requests: int
+    today_bytes: int
+
+    def describe(self) -> dict:
+        return dict(self.__dict__)
+
+
+class UsageLedger:
+    def record_success(
+        self,
+        device_uid: str,
+        user_uid: str,
+        *,
+        bytes_up: int,
+        bytes_down: int,
+        cache_hit: bool = False,
+    ) -> bool:
+        return store.record_job(
+            device_uid,
+            user_uid,
+            ok=True,
+            bytes_up=bytes_up,
+            bytes_down=bytes_down,
+            cache_hit=cache_hit,
+        )
+
+    def record_failure(self, device_uid: str, user_uid: str) -> None:
+        store.record_job(device_uid, user_uid, ok=False)
+
+    def record_integrity_failure(self, device_uid: str, user_uid: str) -> None:
+        store.record_integrity_failure(device_uid, user_uid)
+
+    def claim_xp(self, user_uid: str, amount: int) -> bool:
+        return store.claim_xp(user_uid, amount)
+
+    def summary(self, user_uid: str) -> DonationSummary:
+        now = datetime.now(timezone.utc)
+        donor = store.get_donor(user_uid)
+        month = store.usage_since(user_uid, now.replace(day=1).date().isoformat())
+        today = store.usage_since(user_uid, now.date().isoformat())
+        return DonationSummary(
+            lifetime_requests=donor["requests_ok"],
+            lifetime_bytes=donor["bytes_total"],
+            first_donated_at=donor["first_donated_at"],
+            last_donated_at=donor["last_donated_at"],
+            month_requests=month["requests_ok"],
+            month_bytes_up=month["bytes_up"],
+            month_bytes_down=month["bytes_down"],
+            today_requests=today["requests_ok"],
+            today_bytes=today["bytes_up"] + today["bytes_down"],
+        )
+
+    def prune(self, *, dry_run: bool = False) -> dict:
+        return {
+            "usage_daily": store.prune_usage_daily(dry_run=dry_run),
+            "events": store.prune_events(dry_run=dry_run),
+            "requests": store.prune_requests(dry_run=dry_run),
+        }