Add community node socket: devices, tickets, consent gate, kill switch
Phase 3 of plan.md. Device registration with one-time secrets, scoped single-use socket tickets, the lock-owner WS /nodes/ws with handshake and heartbeat eviction, a single batch ConsentGate, and NodesService which disconnects nodes within one tick when consent, device or the nodes_enabled kill switch changes. A reconcile of 2000 sessions measured about 110 ms after replacing per-session queries with batches. Devii may withdraw but never grant bandwidth_sharing. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 5.5
@@ -2,7 +2,7 @@ This file documents the community node network (`devplacepy/services/nodes/`, `d
|
||||
|
||||
## What exists today
|
||||
|
||||
Phase 1 (the specification) and Phase 2 (the data layer and usage ledger) are built. Everything else is planned and listed under "Gaps" next to the phase that closes it. Nothing here is enabled for any account.
|
||||
Phase 1 (the specification), Phase 2 (the data layer and usage ledger) and Phase 3 (registry, session, consent gate, kill switch, device and ticket routes, the lock-owner socket) are built. No job traffic flows yet. Everything else is planned and listed under "Gaps" next to the phase that closes it. Nothing here is enabled for any account.
|
||||
|
||||
| Piece | File | State |
|
||||
|-------|------|-------|
|
||||
@@ -12,7 +12,14 @@ Phase 1 (the specification) and Phase 2 (the data layer and usage ledger) are bu
|
||||
| API docs group | `docs_api/groups/nodes.py` | built, one endpoint |
|
||||
| Tables and helpers | `database/nodes.py`, `NODES_COLUMNS` in `database/schema.py` | built, tested, no route uses them yet |
|
||||
| Usage ledger | `ledger.py` (`UsageLedger`, `DonationSummary`) | built, tested |
|
||||
| `bandwidth_sharing` consent kind | `CONSENT_KINDS` in `database/moderation.py` | registered, never granted by default, enforced nowhere yet |
|
||||
| `bandwidth_sharing` consent kind | `CONSENT_KINDS` in `database/moderation.py` | registered, never granted by default, enforced by `ConsentGate` |
|
||||
| Frame codec | `frames.py` (`decode`, `encode`, `FrameError`) | built, tested, validates every frame against `spec.py` |
|
||||
| Consent choke point | `consent.py` (`ConsentGate`, `Verdict`) | built, tested, single and batch |
|
||||
| Socket session | `session.py` (`NodeSession`) | built, tested with a fake socket and a real one |
|
||||
| Registry and counters | `registry.py` (`NodeRegistry`, `NodeCounters`) | built, tested |
|
||||
| Supervisor | `service.py` (`NodesService`) | built, registered in `main.py`, default enabled |
|
||||
| Device, ticket and socket routes | `routers/nodes/devices.py`, `ticket.py`, `ws.py` | built, tested |
|
||||
| nginx upgrade block for `/nodes/ws` | `nginx/nginx.conf.template` | added |
|
||||
|
||||
## The specification is the contract
|
||||
|
||||
@@ -49,6 +56,23 @@ Seven tables, all ensured with their full column set in `init_db` (`_ensure_node
|
||||
- **Settings are clamped on write and on read** (`normalise_settings`): bandwidth to `1..max_donor_mb_per_day`, concurrency to `1..max_concurrent_per_node`, hours must be `HH:MM`, platforms are intersected with the spec's adapters. A member with no row is opted out.
|
||||
- Site settings seeded in `init_db`: `nodes_enabled` (`0`), `nodes_max_total`, `nodes_daily_quota_user`, `nodes_daily_quota_guest`, `nodes_cache_ttl_seconds`, `nodes_social_via_nodes` (`0`). Like every seed, consumers pass the default to `get_setting` because the seed does not run on an empty database.
|
||||
|
||||
## Connecting a node (Phase 3)
|
||||
|
||||
1. A member consents to `bandwidth_sharing` and turns donation on for themselves (the profile UI for this is Phase 9; today it is `POST /profile/{username}/consent` and `save_node_settings`).
|
||||
2. `POST /nodes/devices` registers a device and returns its `dpn_` secret once. `GET /nodes/devices`, `POST /nodes/devices/{uid}/rotate` and `/delete` manage it. All are `require_user_api` (401, never a redirect) and answer `Cache-Control: no-store`.
|
||||
3. The node calls `POST /nodes/ticket` with `Authorization: Bearer dpn_...`. A session cookie or a member API key is not a device secret and gets 401. The ticket is a `ws_tickets` row with `scope="nodes"` and the device uid, valid `ticket_ttl_seconds`, redeemed by one conditional `UPDATE` (16 processes racing one ticket gave exactly one winner; a ticket of the wrong scope never redeems).
|
||||
4. The node opens `WS /nodes/ws?ticket=...`. The route only runs on the service-lock owner (`4013` elsewhere, the client retries fast, the `/pubsub/ws` precedent). Order of refusal: service disabled `1013`, not the owner `4013`, bad ticket or device or status `1008`, consent gate `1008` with the reason, capacity `4029`, then the handshake.
|
||||
5. `hello` must arrive within `hello_timeout_seconds`, with a supported `protocol`, a `kind` equal to the registered device kind and a non-empty list of known `modes`. The server answers `welcome` with the full limits and allowlist from the spec. A second connection of the same device kicks the first with `replaced`.
|
||||
6. The session pings every `ping_seconds`; any frame from the client resets the missed counter and `missed_pings_before_evict` unanswered pings close it with `heartbeat`. `usage` frames are advisory and only summed on the session. Every other client frame is a protocol error until Phase 4 assigns jobs.
|
||||
|
||||
## The consent choke point
|
||||
|
||||
`ConsentGate` is the only place that decides whether a member's node may exist. `allows_many(user_uids)` decides a whole set with a fixed number of queries and `allows(uid)` is the one-element case of it, so the single and batch answers cannot diverge (a test compares them). Order of reasons: `kill_switch`, `no_account`, `deletion_pending`, `account_inactive`, `suspended`, `not_adult`, `no_consent`, `not_enabled`. It is called at device registration, ticket issue, socket accept and every supervisor tick. An account whose `age_band` is empty (created before age bands existed) is refused as not adult, loudly, rather than assumed adult.
|
||||
|
||||
`NodesService.reconcile` runs every `reconcile_seconds`, asks the gate once for all connected owners and loads all devices in one query, then kicks with a reason: `kill_switch`, `revoked`, `suspended`, or the gate's reason. This is why withdrawing the consent, switching donation off, deleting the device or flipping `nodes_enabled` takes effect within one tick even when the request landed on a different worker than the one holding the sockets. **Measured** (scratch SQLite, this machine, 2000 synthetic sessions): one tick costs about 110 ms; the per-session version of the same check cost about 1 ms per session, so 488 ms at 500 sessions and an extrapolated 2 s at 2000, which would have blocked the event loop for the whole tick. Do not reintroduce a per-session query in `reconcile`.
|
||||
|
||||
Withdrawal is member-only: `MEMBER_ONLY_CONSENTS` in `database/moderation.py` lists consents that Devii (header `X-Devii-Agent: 1`, resolved in `routers/profile/consent.py`) may withdraw but never grant, and the acceptance service lists `bandwidth_sharing` in `EXCLUDED_CONSENTS`.
|
||||
|
||||
## Two transports, two trust levels
|
||||
|
||||
`tunnel` (native client) relays raw TCP so TLS runs end to end between DevPlace and the destination: the donor cannot read or forge content. `fetch` (browser extension, which has no raw sockets) lets the donor read and forge the response, so every `fetch` result is verified and labelled. A plain web page or a service worker cannot do either (CORS, no sockets) and is out of scope for good.
|
||||
@@ -56,7 +80,7 @@ Seven tables, all ensured with their full column set in `init_db` (`_ensure_node
|
||||
## Gaps (what the next phases close)
|
||||
|
||||
- Phase 2 leftovers: the `bandwidth_sharing` consent can currently be granted through `POST /profile/{username}/consent` and by Devii's `set_consent` (Devii calls that route as the user). Nothing reads it yet, so it is inert, but Phase 3 must make Devii unable to grant it (Devii may read and withdraw) at the same time as `ConsentGate` lands. No XP or badge is awarded yet (`UsageLedger.claim_xp` only reserves the daily budget); the privacy page and app-store declarations are updated with the first phase that lets a device connect, because until then no personal data is collected.
|
||||
- Phase 3: registry, session, the consent choke point, the kill switch, device and ticket routes, the lock-owner WebSocket, the nginx upgrade location.
|
||||
- Phase 3 leftovers: `country`, `asn` and `asn_class` on a device are never filled (they stay `unknown`), so the datacenter exclusion (D6) is not enforced yet; it needs the offline ASN database from Phase 4. `last_seen_at` is written on connect only, not per heartbeat, so a device list cannot yet show who is online right now. The socket counters are per process and are shown in the service metrics only. The `devplace.net` front proxy idle timeout for a long-lived socket is unmeasured.
|
||||
- Phase 4: broker, scheduler, dispatcher, destination policy, circuit breakers, result cache.
|
||||
- Phase 5 onward: the reference client, adapters, the Go client, the extension, the profile UI, Devii tools, the CLI, the privacy and donor-terms pages, the legal gate.
|
||||
- Not measured yet: Reddit and X success rates per address class, WebSocket idle timeouts through `pravda.education` and `devplace.net`, dispatcher tick cost against SQLite, socket capacity of one worker. The caps in `LimitSpec` are starting ceilings, not measurements.
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from devplacepy.database import (
|
||||
consent_granted_many,
|
||||
get_setting,
|
||||
get_users_by_uids,
|
||||
is_account_active,
|
||||
suspension_active,
|
||||
)
|
||||
from devplacepy.database.nodes import BATCH_SIZE, get_enabled_by_users
|
||||
|
||||
CONSENT_KIND = "bandwidth_sharing"
|
||||
ADULT_BAND = "adult"
|
||||
|
||||
REASONS: dict[str, str] = {
|
||||
"kill_switch": "Community donation is switched off",
|
||||
"no_account": "Account not found",
|
||||
"account_inactive": "Account is not active",
|
||||
"deletion_pending": "Account is being deleted",
|
||||
"suspended": "Account is suspended",
|
||||
"not_adult": "Only adults can donate bandwidth",
|
||||
"no_consent": "Bandwidth sharing consent has not been given",
|
||||
"not_enabled": "Donation is turned off in the member's settings",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Verdict:
|
||||
allowed: bool
|
||||
reason: str = ""
|
||||
|
||||
@property
|
||||
def message(self) -> str:
|
||||
return REASONS.get(self.reason, "") if not self.allowed else ""
|
||||
|
||||
|
||||
ALLOWED = Verdict(True)
|
||||
|
||||
|
||||
class ConsentGate:
|
||||
def switched_on(self) -> bool:
|
||||
return get_setting("nodes_enabled", "0") == "1"
|
||||
|
||||
def decide(self, user: dict | None, consented: bool, enabled: bool) -> Verdict:
|
||||
if not user:
|
||||
return Verdict(False, "no_account")
|
||||
if (user.get("deletion_requested_at") or "").strip():
|
||||
return Verdict(False, "deletion_pending")
|
||||
if not is_account_active(user):
|
||||
return Verdict(False, "account_inactive")
|
||||
if suspension_active(user):
|
||||
return Verdict(False, "suspended")
|
||||
if (user.get("age_band") or "") != ADULT_BAND:
|
||||
return Verdict(False, "not_adult")
|
||||
if not consented:
|
||||
return Verdict(False, "no_consent")
|
||||
if not enabled:
|
||||
return Verdict(False, "not_enabled")
|
||||
return ALLOWED
|
||||
|
||||
def allows_many(self, user_uids) -> dict[str, Verdict]:
|
||||
unique = list(dict.fromkeys(uid for uid in user_uids))
|
||||
if not self.switched_on():
|
||||
return {uid: Verdict(False, "kill_switch") for uid in unique}
|
||||
verdicts: dict[str, Verdict] = {}
|
||||
for start in range(0, len(unique), BATCH_SIZE):
|
||||
batch = unique[start : start + BATCH_SIZE]
|
||||
users = get_users_by_uids(batch)
|
||||
consented = consent_granted_many("user", batch, CONSENT_KIND)
|
||||
enabled = get_enabled_by_users(batch)
|
||||
for uid in batch:
|
||||
verdicts[uid] = self.decide(
|
||||
users.get(uid), uid in consented, enabled.get(uid, False)
|
||||
)
|
||||
return verdicts
|
||||
|
||||
def allows(self, user_uid: str) -> Verdict:
|
||||
if not user_uid:
|
||||
return Verdict(False, "kill_switch" if not self.switched_on() else "no_account")
|
||||
return self.allows_many([user_uid])[user_uid]
|
||||
@@ -0,0 +1,90 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from devplacepy.services.nodes.spec import (
|
||||
CLIENT_TO_SERVER,
|
||||
CLOSE_FRAME_TOO_LARGE,
|
||||
CLOSE_PROTOCOL,
|
||||
SPEC,
|
||||
)
|
||||
|
||||
FIELD_TYPES: dict[str, type] = {
|
||||
"str": str,
|
||||
"int": int,
|
||||
"bool": bool,
|
||||
"object": dict,
|
||||
"list": list,
|
||||
}
|
||||
|
||||
|
||||
class FrameError(ValueError):
|
||||
def __init__(self, code: str, reason: str, close_code: int = CLOSE_PROTOCOL):
|
||||
super().__init__(reason)
|
||||
self.code = code
|
||||
self.reason = reason
|
||||
self.close_code = close_code
|
||||
|
||||
|
||||
def ceiling(direction: str) -> int:
|
||||
return max(item.max_bytes for item in SPEC.frames if item.direction == direction)
|
||||
|
||||
|
||||
def _matches(value: Any, type_name: str) -> bool:
|
||||
expected = FIELD_TYPES[type_name]
|
||||
if expected is int:
|
||||
return isinstance(value, int) and not isinstance(value, bool)
|
||||
return isinstance(value, expected)
|
||||
|
||||
|
||||
def validate(frame: dict, direction: str) -> dict:
|
||||
kind = frame.get("type")
|
||||
spec = SPEC.frame(kind, direction) if isinstance(kind, str) else None
|
||||
if spec is None:
|
||||
raise FrameError("unknown_frame", f"unknown frame type {kind!r}")
|
||||
unexpected = set(frame) - spec.allowed()
|
||||
if unexpected:
|
||||
raise FrameError("unexpected_field", f"unexpected field {sorted(unexpected)[0]}")
|
||||
for item in spec.fields:
|
||||
if item.name not in frame:
|
||||
if item.required:
|
||||
raise FrameError("missing_field", f"missing field {item.name}")
|
||||
continue
|
||||
if not _matches(frame[item.name], item.type):
|
||||
raise FrameError("bad_field_type", f"field {item.name} must be {item.type}")
|
||||
return frame
|
||||
|
||||
|
||||
def decode(raw: str | bytes, direction: str = CLIENT_TO_SERVER) -> dict:
|
||||
size = len(raw.encode()) if isinstance(raw, str) else len(raw)
|
||||
if size > ceiling(direction):
|
||||
raise FrameError(
|
||||
"frame_too_large", "frame exceeds the size cap", CLOSE_FRAME_TOO_LARGE
|
||||
)
|
||||
try:
|
||||
frame = json.loads(raw)
|
||||
except ValueError:
|
||||
raise FrameError("bad_json", "frame is not valid JSON")
|
||||
if not isinstance(frame, dict):
|
||||
raise FrameError("bad_json", "frame must be a JSON object")
|
||||
validate(frame, direction)
|
||||
limit = SPEC.frame(frame["type"], direction).max_bytes
|
||||
if size > limit:
|
||||
raise FrameError(
|
||||
"frame_too_large", "frame exceeds its size cap", CLOSE_FRAME_TOO_LARGE
|
||||
)
|
||||
return frame
|
||||
|
||||
|
||||
def encode(frame_type: str, direction: str, **fields: Any) -> str:
|
||||
frame = {"type": frame_type, **fields}
|
||||
validate(frame, direction)
|
||||
text = json.dumps(frame, separators=(",", ":"))
|
||||
if len(text.encode()) > SPEC.frame(frame_type, direction).max_bytes:
|
||||
raise FrameError(
|
||||
"frame_too_large", "frame exceeds its size cap", CLOSE_FRAME_TOO_LARGE
|
||||
)
|
||||
return text
|
||||
@@ -0,0 +1,84 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
|
||||
from devplacepy.database import get_int_setting
|
||||
from devplacepy.services.nodes.session import NodeSession
|
||||
from devplacepy.services.nodes.spec import SPEC
|
||||
|
||||
LIMITS = SPEC.limits
|
||||
|
||||
COUNTER_NAMES: tuple[str, ...] = (
|
||||
"connected_total",
|
||||
"rejected_capacity",
|
||||
"rejected_ticket",
|
||||
"rejected_consent",
|
||||
"rejected_device",
|
||||
"protocol_errors",
|
||||
"evicted_heartbeat",
|
||||
"kicked",
|
||||
"replaced",
|
||||
)
|
||||
|
||||
|
||||
class NodeCounters:
|
||||
def __init__(self) -> None:
|
||||
self._counts: Counter[str] = Counter({name: 0 for name in COUNTER_NAMES})
|
||||
|
||||
def inc(self, name: str) -> None:
|
||||
if name not in COUNTER_NAMES:
|
||||
raise KeyError(name)
|
||||
self._counts[name] += 1
|
||||
|
||||
def snapshot(self) -> dict[str, int]:
|
||||
return {name: self._counts[name] for name in COUNTER_NAMES}
|
||||
|
||||
|
||||
class NodeRegistry:
|
||||
def __init__(self, counters: NodeCounters) -> None:
|
||||
self._sessions: dict[str, NodeSession] = {}
|
||||
self.counters = counters
|
||||
|
||||
def capacity(self) -> int:
|
||||
configured = get_int_setting("nodes_max_total", LIMITS.max_connected_nodes)
|
||||
return max(0, min(configured, LIMITS.max_connected_nodes))
|
||||
|
||||
def count(self) -> int:
|
||||
return len(self._sessions)
|
||||
|
||||
def sessions(self) -> list[NodeSession]:
|
||||
return list(self._sessions.values())
|
||||
|
||||
def get(self, device_uid: str) -> NodeSession | None:
|
||||
return self._sessions.get(device_uid)
|
||||
|
||||
def by_user(self, user_uid: str) -> list[NodeSession]:
|
||||
return [item for item in self._sessions.values() if item.user_uid == user_uid]
|
||||
|
||||
def admit(self, session: NodeSession) -> tuple[bool, NodeSession | None]:
|
||||
previous = self._sessions.get(session.device_uid)
|
||||
if previous is None and self.count() >= self.capacity():
|
||||
self.counters.inc("rejected_capacity")
|
||||
return False, None
|
||||
self._sessions[session.device_uid] = session
|
||||
self.counters.inc("connected_total")
|
||||
if previous is not None:
|
||||
self.counters.inc("replaced")
|
||||
return True, previous
|
||||
|
||||
def release(self, session: NodeSession) -> None:
|
||||
if self._sessions.get(session.device_uid) is session:
|
||||
del self._sessions[session.device_uid]
|
||||
|
||||
async def kick(self, session: NodeSession, reason: str) -> None:
|
||||
self.release(session)
|
||||
self.counters.inc("kicked")
|
||||
await session.kick(reason)
|
||||
|
||||
async def kick_all(self, reason: str) -> int:
|
||||
sessions = self.sessions()
|
||||
for session in sessions:
|
||||
await self.kick(session, reason)
|
||||
return len(sessions)
|
||||
@@ -0,0 +1,131 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
|
||||
from devplacepy.database.nodes import (
|
||||
STATUS_ACTIVE,
|
||||
add_event,
|
||||
effective_status,
|
||||
get_devices_by_uids,
|
||||
)
|
||||
from devplacepy.services.base import BaseService, ConfigField
|
||||
from devplacepy.services.nodes.consent import ConsentGate
|
||||
from devplacepy.services.nodes.ledger import UsageLedger
|
||||
from devplacepy.services.nodes.registry import NodeCounters, NodeRegistry
|
||||
from devplacepy.services.nodes.spec import SPEC
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
LIMITS = SPEC.limits
|
||||
SERVICE_NAME = "nodes"
|
||||
|
||||
|
||||
class NodesService(BaseService):
|
||||
title = "Community nodes"
|
||||
description = (
|
||||
"Holds the connected donor nodes, enforces the consent gate and the kill "
|
||||
"switch within one tick, and prunes the donation counters."
|
||||
)
|
||||
details = (
|
||||
"Donation is off until an administrator turns it on here and a member has "
|
||||
"consented, enabled it for themselves and registered a device. Switching it "
|
||||
"off disconnects every node on the next tick. The service never bans a "
|
||||
"member and never deletes a ledger row on its own."
|
||||
)
|
||||
min_interval = 1
|
||||
METRICS_SECONDS = 5
|
||||
config_fields = [
|
||||
ConfigField(
|
||||
"nodes_enabled",
|
||||
"Community donation",
|
||||
type="bool",
|
||||
default=False,
|
||||
help="Master kill switch. Off disconnects every node on the next tick.",
|
||||
group="Safety",
|
||||
),
|
||||
ConfigField(
|
||||
"nodes_max_total",
|
||||
"Maximum connected nodes",
|
||||
type="int",
|
||||
default=LIMITS.max_connected_nodes,
|
||||
minimum=0,
|
||||
maximum=LIMITS.max_connected_nodes,
|
||||
help="Hard ceiling on simultaneous sockets in this process.",
|
||||
group="Safety",
|
||||
),
|
||||
]
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__(SERVICE_NAME, interval_seconds=LIMITS.reconcile_seconds)
|
||||
self.counters = NodeCounters()
|
||||
self.registry = NodeRegistry(self.counters)
|
||||
self.gate = ConsentGate()
|
||||
self.ledger = UsageLedger()
|
||||
self._last_prune = 0.0
|
||||
|
||||
async def reconcile(self) -> int:
|
||||
sessions = self.registry.sessions()
|
||||
if not sessions:
|
||||
return 0
|
||||
switched_on = self.gate.switched_on()
|
||||
devices = get_devices_by_uids(item.device_uid for item in sessions)
|
||||
verdicts = self.gate.allows_many(item.user_uid for item in sessions)
|
||||
kicked = 0
|
||||
for session in sessions:
|
||||
reason = self._reason_to_kick(
|
||||
switched_on, devices.get(session.device_uid), verdicts[session.user_uid]
|
||||
)
|
||||
if reason:
|
||||
await self.registry.kick(session, reason)
|
||||
add_event(
|
||||
"kick",
|
||||
device_uid=session.device_uid,
|
||||
user_uid=session.user_uid,
|
||||
detail=reason,
|
||||
)
|
||||
kicked += 1
|
||||
return kicked
|
||||
|
||||
@staticmethod
|
||||
def _reason_to_kick(switched_on: bool, device: dict | None, verdict) -> str:
|
||||
if not switched_on:
|
||||
return "kill_switch"
|
||||
if device is None:
|
||||
return "revoked"
|
||||
if effective_status(device) != STATUS_ACTIVE:
|
||||
return "suspended"
|
||||
return "" if verdict.allowed else verdict.reason
|
||||
|
||||
def _prune_due(self) -> bool:
|
||||
return time.monotonic() - self._last_prune >= LIMITS.prune_interval_seconds
|
||||
|
||||
async def run_once(self) -> None:
|
||||
kicked = await self.reconcile()
|
||||
if kicked:
|
||||
self.log(f"disconnected {kicked} node(s)")
|
||||
if self._prune_due():
|
||||
self._last_prune = time.monotonic()
|
||||
pruned = self.ledger.prune()
|
||||
if any(pruned.values()):
|
||||
self.log(f"pruned {pruned}")
|
||||
|
||||
async def on_disable(self) -> None:
|
||||
await self.registry.kick_all("service_stopped")
|
||||
|
||||
def collect_metrics(self) -> dict:
|
||||
counts = self.counters.snapshot()
|
||||
return {
|
||||
"stats": [
|
||||
{"label": "connected", "value": self.registry.count()},
|
||||
{"label": "capacity", "value": self.registry.capacity()},
|
||||
{"label": "accepted", "value": counts["connected_total"]},
|
||||
{"label": "kicked", "value": counts["kicked"]},
|
||||
],
|
||||
"table": {
|
||||
"columns": ["counter", "value"],
|
||||
"rows": [[name, value] for name, value in counts.items()],
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
|
||||
from devplacepy.database.nodes import get_node_settings
|
||||
from devplacepy.services.nodes.frames import FrameError, decode, encode
|
||||
from devplacepy.services.nodes.spec import (
|
||||
CLIENT_TO_SERVER,
|
||||
CLOSE_KICKED,
|
||||
CLOSE_POLICY,
|
||||
MODES,
|
||||
SERVER_TO_CLIENT,
|
||||
SPEC,
|
||||
)
|
||||
|
||||
LIMITS = SPEC.limits
|
||||
|
||||
END_HEARTBEAT = "heartbeat"
|
||||
END_UNEXPECTED_FRAME = "unexpected_frame"
|
||||
|
||||
|
||||
class NodeSession:
|
||||
def __init__(self, websocket, device: dict) -> None:
|
||||
self.websocket = websocket
|
||||
self.device = device
|
||||
self.device_uid: str = device["uid"]
|
||||
self.user_uid: str = device["user_uid"]
|
||||
self.connected_at = time.monotonic()
|
||||
self.missed_pings = 0
|
||||
self.hello: dict = {}
|
||||
self.reported_up = 0
|
||||
self.reported_down = 0
|
||||
self.closed = False
|
||||
|
||||
async def send(self, frame_type: str, **fields) -> None:
|
||||
await self.websocket.send_text(encode(frame_type, SERVER_TO_CLIENT, **fields))
|
||||
|
||||
async def close(self, code: int, reason: str = "") -> None:
|
||||
if self.closed:
|
||||
return
|
||||
self.closed = True
|
||||
try:
|
||||
await self.websocket.close(code=code, reason=reason[:120])
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def kick(self, reason: str) -> None:
|
||||
if self.closed:
|
||||
return
|
||||
try:
|
||||
await self.send("kick", reason=reason)
|
||||
except Exception:
|
||||
pass
|
||||
await self.close(CLOSE_KICKED, reason)
|
||||
|
||||
def _check_hello(self, frame: dict) -> None:
|
||||
if not SPEC.min_supported <= frame["protocol"] <= SPEC.version:
|
||||
raise FrameError("protocol_unsupported", "protocol version is not supported", CLOSE_POLICY)
|
||||
if frame["kind"] != self.device.get("kind"):
|
||||
raise FrameError("kind_mismatch", "client kind does not match the registered device", CLOSE_POLICY)
|
||||
if not frame["modes"] or any(mode not in MODES for mode in frame["modes"]):
|
||||
raise FrameError("bad_modes", "modes must be a non-empty list of known modes")
|
||||
|
||||
async def handshake(self) -> dict:
|
||||
try:
|
||||
raw = await asyncio.wait_for(
|
||||
self.websocket.receive_text(), timeout=LIMITS.hello_timeout_seconds
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
raise FrameError("hello_timeout", "no hello frame arrived in time", CLOSE_POLICY)
|
||||
frame = decode(raw, CLIENT_TO_SERVER)
|
||||
if frame["type"] != "hello":
|
||||
raise FrameError("expected_hello", "the first frame must be hello")
|
||||
self._check_hello(frame)
|
||||
self.hello = frame
|
||||
await self.send(
|
||||
"welcome",
|
||||
device_uid=self.device_uid,
|
||||
limits=LIMITS.describe(),
|
||||
destinations=[item.describe() for item in SPEC.destinations],
|
||||
settings=get_node_settings(self.user_uid),
|
||||
)
|
||||
return frame
|
||||
|
||||
def _handle(self, frame: dict) -> None:
|
||||
kind = frame["type"]
|
||||
if kind == "pong":
|
||||
return
|
||||
if kind == "usage":
|
||||
self.reported_up += max(0, frame["bytes_up"])
|
||||
self.reported_down += max(0, frame["bytes_down"])
|
||||
return
|
||||
raise FrameError(END_UNEXPECTED_FRAME, f"unexpected {kind} frame, no job is assigned")
|
||||
|
||||
async def run(self) -> str:
|
||||
while not self.closed:
|
||||
try:
|
||||
raw = await asyncio.wait_for(
|
||||
self.websocket.receive_text(), timeout=LIMITS.ping_seconds
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
if self.missed_pings >= LIMITS.missed_pings_before_evict:
|
||||
await self.close(CLOSE_POLICY, END_HEARTBEAT)
|
||||
return END_HEARTBEAT
|
||||
self.missed_pings += 1
|
||||
await self.send("ping")
|
||||
continue
|
||||
self.missed_pings = 0
|
||||
self._handle(decode(raw, CLIENT_TO_SERVER))
|
||||
return "closed"
|
||||
@@ -94,6 +94,9 @@ class LimitSpec:
|
||||
request_retention_seconds: int = 600
|
||||
xp_daily_cap: int = 50
|
||||
dispatcher_tick_ms: int = 200
|
||||
hello_timeout_seconds: int = 10
|
||||
reconcile_seconds: int = 2
|
||||
prune_interval_seconds: int = 3600
|
||||
|
||||
def describe(self) -> dict[str, int]:
|
||||
return dict(self.__dict__)
|
||||
|
||||
Reference in New Issue
Block a user