retoor/devplacepy
1
0
Fork 1
Code Issues 1 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Zo doen wijdat in oost
DevPlace CI / test (push) Failing after 3m13s
Details

Browse Source
This commit is contained in:
retoor
2026-10-16 07:20:44 +02:00
parent 25d9cb1c7d
commit 6f80a820da
229 changed files with 23894 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devplacepy/services/nodes/CLAUDE.md
+59 -8
View File
@@ -2,7 +2,7 @@ This file documents the community node network (`devplacepy/services/nodes/`, `d
## What exists today
Phase 1 (the specification), Phase 2 (the data layer and usage ledger) and Phase 3 (registry, session, consent gate, kill switch, device and ticket routes, the lock-owner socket) are built. No job traffic flows yet. Everything else is planned and listed under "Gaps" next to the phase that closes it. Nothing here is enabled for any account.
The specification, data layer, registry, session, consent gate, kill switch, destination policy, queue, scheduler, dispatcher, circuit breakers, cache, the fabric facade, tunnel mode, social search adapters, the Go client, the browser extension, the Donate tab, the admin page, the Devii tools, the CLI and the docs are built and tested. The gaps below list what is unmeasured or missing. Nothing here is enabled for any account.
| Piece | File | State |
|-------|------|-------|
@@ -10,7 +10,7 @@ Phase 1 (the specification), Phase 2 (the data layer and usage ledger) and Phase
| `GET /nodes/spec.json` | `routers/nodes/spec.py` | built, tested, public |
| Generated protocol docs page | `templates/docs/nodes-protocol.html` | built, reads the `nodes_spec` Jinja global |
| API docs group | `docs_api/groups/nodes.py` | built, one endpoint |
| Tables and helpers | `database/nodes.py`, `NODES_COLUMNS` in `database/schema.py` | built, tested, no route uses them yet |
| Tables and helpers | `database/nodes.py`, `NODES_COLUMNS` in `database/schema.py` | built, tested |
| Usage ledger | `ledger.py` (`UsageLedger`, `DonationSummary`) | built, tested |
| `bandwidth_sharing` consent kind | `CONSENT_KINDS` in `database/moderation.py` | registered, never granted by default, enforced by `ConsentGate` |
| Frame codec | `frames.py` (`decode`, `encode`, `FrameError`) | built, tested, validates every frame against `spec.py` |
@@ -20,6 +20,22 @@ Phase 1 (the specification), Phase 2 (the data layer and usage ledger) and Phase
| Supervisor | `service.py` (`NodesService`) | built, registered in `main.py`, default enabled |
| Device, ticket and socket routes | `routers/nodes/devices.py`, `ticket.py`, `ws.py` | built, tested |
| nginx upgrade block for `/nodes/ws` | `nginx/nginx.conf.template` | added |
| Destination policy | `policy.py` (`DestinationPolicy`, `Target`, `is_public_address`) | built, tested |
| Result cache | `cache.py` (`ResultCache`) | built, tested, per process |
| Circuit breaker | `circuit.py` (`CircuitBreaker`) | built, tested |
| Scheduler | `scheduler.py` (`NodeScheduler`, `in_active_hours`) | built, tested |
| Queue and broker | `broker.py` (`JobBroker`), `database/nodes.py` request helpers | built, tested |
| Dispatcher | `dispatcher.py` (`NodeDispatcher`) | built, tested, runs in the lock owner |
| Fabric facade | `fabric.py` (`NodeFabric`, `FabricResult`, `FabricError`) | built, tested, used by `social.py` and `POST /nodes/internal/fetch` |
| Admin page and actions | `routers/admin/nodes.py`, `templates/admin_nodes.html`, `static/js/AdminNodes.js` | built |
## Administration (`/admin/nodes`)
`GET /admin/nodes` (HTML and JSON `AdminNodesOut`) is the administrator view: pool counters read from the `nodes` row in `service_state` (so any worker answers, at most one persist interval stale; `pool.online` is false when the heartbeat is stale), queue depth and counts by status, device counts by status, kind, address class and country, the current UTC day's totals, open flags and the 100 most recently seen devices (both capped), the kill switch and the six settings with defaults and ceilings. **Per-destination circuit state is not shown:** it lives in memory on the lock-owner worker and only the number of open circuits is persisted, so a per-destination view would need a new persisted snapshot. Device counts use the stored status, so a suspension that has expired still counts as `suspended` until the next write; the recent-devices list uses `effective_status`.
POST routes, all `require_admin`, all audited: `/killswitch` (`node.killswitch`, flips or sets `nodes_enabled`), `/settings` (`node.admin.settings`, only submitted fields change, values above the ceiling are clamped and reported under `clamped`), `/devices/{uid}/suspend` (`node.suspend`, 1 to 10080 minutes) and `/unsuspend` (`node.unsuspend`), `/flags/{uid}/resolve` (`node.flag.resolve`). The per-device and per-flag routes are gated by `is_senior_admin` against the device or flag owner and answer through `deny_senior` (audited `denied`). Suspension and the kill switch take effect within one `reconcile` tick on whichever worker holds the sockets. Ceilings: `nodes_max_total` and `nodes_cache_ttl_seconds` use `LimitSpec`; the quota ceilings are constants in the router because `LimitSpec` only carries defaults for them.
The page refreshes every 10 seconds by fetching its own HTML and swapping the `[data-live]` sections (`AdminNodes.js`, `app.adminNodes`); the settings form and the suspend modal are never swapped, so typing is not interrupted. Devii: `node_admin_status`, `node_admin_suspend` and `node_admin_killswitch` (the last two in `CONFIRM_REQUIRED`), `node_admin_unsuspend`, all `requires_admin`.
## The specification is the contract
@@ -65,6 +81,22 @@ Seven tables, all ensured with their full column set in `init_db` (`_ensure_node
5. `hello` must arrive within `hello_timeout_seconds`, with a supported `protocol`, a `kind` equal to the registered device kind and a non-empty list of known `modes`. The server answers `welcome` with the full limits and allowlist from the spec. A second connection of the same device kicks the first with `replaced`.
6. The session pings every `ping_seconds`; any frame from the client resets the missed counter and `missed_pings_before_evict` unanswered pings close it with `heartbeat`. `usage` frames are advisory and only summed on the session. Every other client frame is a protocol error until Phase 4 assigns jobs.
## Jobs (Phase 4)
A job is a row in `node_requests`, so any worker can ask and only the lock owner serves. Life cycle: `queued` -> `assigned` -> `done`, or `failed` / `expired`; every transition is one conditional `UPDATE`.
1. `NodeFabric.fetch(url, caller_kind=, caller_id=)` (any worker): policy check, result cache, kill switch and "any node online" (read from the `nodes` row in `service_state`, so it is at most one persist interval stale), then the **caller's daily quota**, then enqueue, then poll the row every `poll_interval_ms` until it is terminal or the caller's wait ends. One retry on `node_disconnected`, `timeout`, `send_failed`, `concurrency_exceeded`, `outside_hours`, `bandwidth_exhausted`; the retry does not spend a second quota unit. Bounded: queue depth (`max_queue_depth`, a conditional `INSERT`), `max_wait_seconds`, `queue_max_wait_seconds`.
2. `NodeDispatcher.tick` (lock owner, every `dispatcher_tick_ms`, started by `ensure_dispatching()` from the socket route or `on_enable`): expire, sweep timed-out in-flight jobs, then for each queued row re-check the policy, ask the scheduler for a node, claim the row atomically, send the `job` frame.
3. The node answers with `job_result` or `job_error`. `Session._handle` only accepts an answer for a job it has in flight; anything else is a protocol error and closes the socket.
**Scheduler eligibility** (every one must hold): socket open and not paused, speaks a transport the job allows (`tunnel` is preferred over `fetch` for `any`; a node that speaks neither gets nothing), is not the caller's own account, below `min(server, donor setting, hello limits)` concurrency, the destination's adapter is in the donor's `allowed_platforms`, inside the donor's active hours in the donor's timezone (windows may wrap midnight), below the donor's daily byte cap, at least `min_seconds_per_node_domain` since this node last started a job for the destination suffix, node circuit closed. Globally the destination needs `1/max_rps` seconds since its last job and a closed destination circuit. Choice: fewest in-flight, then least recently assigned, then random. Per-destination state is keyed by the allowlist suffix, so `a.reddit.com` ... `z.reddit.com` cannot grow it.
**Trust.** `via` is always `fetch` for node results. A result is checked before it is stored: status 100-599, strict base64, body capped at `max_body_bytes` (truncated and flagged, never silent), headers reduced to at most 64 short strings with `set-cookie`, `set-cookie2`, `authorization` and `proxy-authenticate` dropped. A malformed result is an **integrity failure**: the request fails, `integrity_failures` is counted on the device and the process, a `node_flags` row is raised for a human, and the node is NOT auto-banned. A donor refusing a destination the server allowed raises a `refusal_mismatch` flag. Donor limit codes (`bandwidth_exhausted`, `outside_hours`, `concurrency_exceeded`) pause the node instead of blaming it. HTTP 429 and 5xx count against the destination circuit (they are the signal that stops hammering a rate-limiting site); the result is still returned to the caller. Sampled redundancy between two donors is not built (Phase 6B).
**Accounting.** A successful job calls `record_job` (daily bucket and lifetime totals in one transaction). If the donor's cap would be crossed the result is still delivered but the node is stopped for the day. Failures call `record_failure`. Bytes up are estimated (`len(url) + 256`) because the server never sees the donor's real request.
**Measured** (scratch SQLite on this machine, 2000 synthetic sessions, 500 queued requests, batch 50): an idle tick costs 0.1 ms; a tick where no node is eligible because every node is outside its hours cost 225 ms before two fixes (the active-hours window is now cached per minute and a destination found saturated is scanned once per tick, not once per queued request) and 8.4 ms after; a destination held back by its rate ceiling costs 4.7 ms. Races with 16 to 60 real processes on fresh rows: the queue cap admitted exactly the remaining slots, one request was claimed by exactly one assignment, a caller quota of 5 admitted exactly 5. These are one machine and synthetic sessions, not a soak test.
## The consent choke point
`ConsentGate` is the only place that decides whether a member's node may exist. `allows_many(user_uids)` decides a whole set with a fixed number of queries and `allows(uid)` is the one-element case of it, so the single and batch answers cannot diverge (a test compares them). Order of reasons: `kill_switch`, `no_account`, `deletion_pending`, `account_inactive`, `suspended`, `not_adult`, `no_consent`, `not_enabled`. It is called at device registration, ticket issue, socket accept and every supervisor tick. An account whose `age_band` is empty (created before age bands existed) is refused as not adult, loudly, rather than assumed adult.
@@ -77,13 +109,32 @@ Withdrawal is member-only: `MEMBER_ONLY_CONSENTS` in `database/moderation.py` li
`tunnel` (native client) relays raw TCP so TLS runs end to end between DevPlace and the destination: the donor cannot read or forge content. `fetch` (browser extension, which has no raw sockets) lets the donor read and forge the response, so every `fetch` result is verified and labelled. A plain web page or a service worker cannot do either (CORS, no sockets) and is out of scope for good.
## Gaps (what the next phases close)
## Donate tab (member UI)
- Phase 2 leftovers: the `bandwidth_sharing` consent can currently be granted through `POST /profile/{username}/consent` and by Devii's `set_consent` (Devii calls that route as the user). Nothing reads it yet, so it is inert, but Phase 3 must make Devii unable to grant it (Devii may read and withdraw) at the same time as `ConsentGate` lands. No XP or badge is awarded yet (`UsageLedger.claim_xp` only reserves the daily budget); the privacy page and app-store declarations are updated with the first phase that lets a device connect, because until then no personal data is collected.
- Phase 3 leftovers: `country`, `asn` and `asn_class` on a device are never filled (they stay `unknown`), so the datacenter exclusion (D6) is not enforced yet; it needs the offline ASN database from Phase 4. `last_seen_at` is written on connect only, not per heartbeat, so a device list cannot yet show who is online right now. The socket counters are per process and are shown in the service metrics only. The `devplace.net` front proxy idle timeout for a long-lived socket is unmeasured.
- Phase 4: broker, scheduler, dispatcher, destination policy, circuit breakers, result cache.
- Phase 5 onward: the reference client, adapters, the Go client, the extension, the profile UI, Devii tools, the CLI, the privacy and donor-terms pages, the legal gate.
- Not measured yet: Reddit and X success rates per address class, WebSocket idle timeouts through `pravda.education` and `devplace.net`, dispatcher tick cost against SQLite, socket capacity of one worker. The caps in `LimitSpec` are starting ceilings, not measurements.
The owner-only `Donate` tab on the profile (`?tab=donate`, `templates/_node_donation.html`, `static/js/NodeDonation.js` as `app.nodeDonation`, `static/css/nodes.css`) is a thin client of the routes above: it renders `/nodes/me` server-side (context key `node_me`, only for the owner on that tab, `noindex,follow`), then polls `GET /nodes/me` every 30 seconds with the shared `Poller`. Consent goes through `POST /profile/{username}/consent`, never a second consent route. The device secret exists only in the browser memory and DOM until "I have saved it" is pressed; the install lines are built client-side from `GET /nodes/client-version`. Prose for members is `templates/docs/nodes.html`; the data categories are in `templates/docs/privacy.html`.
## Tunnel mode (`tunnel.py`)
For a `tunnel` or `any` job the scheduler prefers a node whose `hello.modes` include `tunnel`. The dispatcher marks the transport on the row (`node_requests.mode` becomes the transport actually used, which is how `FabricResult.via` says `tunnel` or `fetch`), then runs the request itself in the lock owner: `TunnelBridge.fetch` opens a one-shot listener on `127.0.0.1:<ephemeral>`, points a stealth (curl_cffi, Chrome fingerprint) client at it as an HTTP proxy, and the `Relay` answers exactly one `CONNECT host:port` for the job's own host and port (anything else is `403`, a second `CONNECT` is `405`, a donor that refuses is `502`). Bytes are relayed as `tunnel_data` frames (48 KiB chunks, base64, both directions, `RELAY_CAP_BYTES` total, `queue_full` and `bad_base64` end the tunnel with a code). TLS runs between DevPlace and the destination, so the donor sees the host and ciphertext and cannot read or forge the page; the ledger counts the real bytes relayed. Late `tunnel_*` frames for a closed relay are ignored (the donor legitimately sends `tunnel_close` after the server did). Redirects are not followed; the caller gets the 3xx. In `fetch` mode the `job` frame now carries `JOB_HEADERS` (a descriptive bot user agent), because a Go or browser client would otherwise send its own.
## Social search (`social.py`, `adapters/`)
`SocialSearch.search` fans one query out to adapters in parallel (`asyncio.gather`, 20 s each) and merges hits round-robin. Direct adapters (`adapters/direct.py`: Hacker News, Stack Overflow, GitHub issues, dev.to by tag, Mastodon by hashtag) need no donors. `adapters/reddit.py` is node-backed and only used when `nodes_social_via_nodes` is `1`; otherwise it is reported as `nodes_not_used`. Every adapter parses a response structurally and raises `AdapterError` on a forged or wrong shape; platforms that fail are listed under `unavailable` with a code and never filled with invented results. **Measured on this machine (2026-10-15, datacenter address):** Hacker News Algolia, Stack Exchange, GitHub search, dev.to tag API and Mastodon hashtag timelines answer; Bluesky `searchPosts` returns 403 without auth, Mastodon `/api/v2/search` returns nothing without login, Lobsters search sits behind a bot wall, dev.to has no usable text search, and Reddit returns 403 to this address. Those are why Bluesky and Lobsters are not adapters. Surfaces: `GET /nodes/social/search` (daily per-member quota `nodes_daily_social_user`, default 300), Devii `social_search`, the DeepSearch `deepsearch_social` option (off by default, signed-in owners only, goes through the HTTP route with the owner's key).
## Clients
`nodes/client/` is the Go donor client (standard library only, `build.sh` cross-compiles five targets and writes `manifest.json`). `nodes/extension/` is the MV3 extension (Chrome and Firefox, fetch mode only, 49 `node --test` tests, one of which reads `spec.py` and fails on drift). `make node-client` builds both into `data/node_clients/`, which `GET /nodes/client-version` and `GET /nodes/download/{artifact}` serve (names must match a strict pattern, symlinks and the manifest are never served, the SHA-256 is computed from the file). **Verified end to end on 2026-10-15** against a scratch server on this machine: the Go client logged in, connected, answered a `tunnel` job for `reddit.com` (`via: tunnel`, status 403 because the donor was this same datacenter address) and stayed connected across repeated jobs. Not verified: a residential donor, the extension in a real browser, Windows and macOS builds running.
## Gaps (what is not built or not measured)
- **Reddit viability is unmeasured from a residential line.** The only real run used a datacenter donor and got 403, which proves the path, not the success rate. X, Instagram, LinkedIn, Facebook and TikTok stay unreachable (login walls); nothing here claims them.
- `country`, `asn` and `asn_class` on a device are never filled, so the datacenter exclusion (D6) is not enforced: the scheduler treats every node as eligible. It needs an offline ASN database file.
- `last_seen_at` is written on connect only. Socket counters are per process. The fabric cache is per process. Circuit breaker state is in memory. The same-process fast path is not built, callers poll at `poll_interval_ms`.
- Fetch-mode results are not cross-checked (sampled redundancy between two donors is not built). They are structurally checked and labelled `via: fetch`, and `tunnel` is preferred whenever a node can do it.
- `DeepSearch` still skips hostile domains with a static list; Reddit URLs from social search are used through their snippet. Routing a hostile URL fetch through `/nodes/internal/fetch` is not built.
- The Go client has no battery-aware pause (`pause_on_battery` is stored and sent but ignored by it), and treats a timed suspension as permanent until restarted. The extension enforces its own local hours in the browser clock, not the server timezone.
- Not measured: Reddit and X success rates per address class, WebSocket idle timeouts through `pravda.education` and `devplace.net`, socket capacity of one worker, a 24 hour soak. The caps in `LimitSpec` are starting ceilings, not measurements.
- **Launch is blocked on a recorded legal decision** (platform terms, donor terms, ISP terms, jurisdiction). Nothing here is enabled for any account: `nodes_enabled` is `0` until an administrator switches it on.
## Rules for work in this directory
devplacepy/services/nodes/adapters/__init__.py
+1
View File
@@ -0,0 +1 @@
# retoor <retoor@molodetz.nl>
devplacepy/services/nodes/adapters/base.py
+101
View File
@@ -0,0 +1,101 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import html
import json
import re
from dataclasses import dataclass
from typing import Any
TITLE_MAX = 300
SNIPPET_MAX = 500
NAME_MAX = 80
MAX_BODY_BYTES = 2 * 1024 * 1024
TAG = re.compile(r"<[^>]+>")
WS = re.compile(r"\s+")
TAG_WORD = re.compile(r"[a-z0-9]{2,30}")
class AdapterError(Exception):
def __init__(self, code: str, message: str = ""):
super().__init__(message or code)
self.code = code
self.message = message or code
@dataclass(frozen=True)
class SocialHit:
platform: str
title: str
url: str
snippet: str = ""
author: str = ""
score: int = 0
comments: int = 0
created_at: str = ""
via: str = "direct"
def as_dict(self) -> dict[str, Any]:
return dict(self.__dict__)
def clean(text: Any, limit: int) -> str:
if not isinstance(text, str):
return ""
plain = html.unescape(TAG.sub(" ", text))
return WS.sub(" ", plain).strip()[:limit]
def number(value: Any) -> int:
if isinstance(value, bool) or not isinstance(value, (int, float)):
return 0
return max(0, int(value))
def tag_for(query: str) -> str:
words = TAG_WORD.findall(query.lower())
return max(words, key=len) if words else ""
def load_json(body: bytes) -> Any:
if len(body) > MAX_BODY_BYTES:
raise AdapterError("body_too_large")
try:
return json.loads(body)
except ValueError:
raise AdapterError("not_json", "the response is not JSON")
def expect(value: Any, kind: type, what: str) -> Any:
if not isinstance(value, kind):
raise AdapterError("bad_structure", f"{what} has an unexpected shape")
return value
class SearchAdapter:
name = ""
direct = True
def request(self, query: str, limit: int) -> str:
raise NotImplementedError
def parse(self, body: bytes, limit: int) -> list[SocialHit]:
raise NotImplementedError
def hit(self, **fields: Any) -> SocialHit | None:
url = fields.get("url") or ""
title = clean(fields.get("title"), TITLE_MAX)
if not title or not isinstance(url, str) or not url.startswith(("http://", "https://")):
return None
return SocialHit(
platform=self.name,
title=title,
url=url[:2048],
snippet=clean(fields.get("snippet"), SNIPPET_MAX),
author=clean(fields.get("author"), NAME_MAX),
score=number(fields.get("score")),
comments=number(fields.get("comments")),
created_at=str(fields.get("created_at") or "")[:40],
via=fields.get("via") or "direct",
)
devplacepy/services/nodes/adapters/direct.py
+197
View File
@@ -0,0 +1,197 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
from urllib.parse import quote, urlencode
import httpx
from devplacepy import stealth
from devplacepy.services.nodes.adapters.base import (
MAX_BODY_BYTES,
AdapterError,
SearchAdapter,
SocialHit,
clean,
expect,
load_json,
number,
tag_for,
)
TIMEOUT_SECONDS = 12.0
ACCEPT = {"Accept": "application/json"}
async def get(url: str) -> bytes:
timeout = httpx.Timeout(TIMEOUT_SECONDS, connect=6.0)
try:
async with stealth.stealth_async_client(
headers=ACCEPT, follow_redirects=True, timeout=timeout
) as client:
response = await client.get(url)
except httpx.HTTPError as exc:
raise AdapterError("network", type(exc).__name__)
if response.status_code == 429:
raise AdapterError("rate_limited")
if response.status_code >= 400:
raise AdapterError("upstream", f"HTTP {response.status_code}")
body = response.content
if len(body) > MAX_BODY_BYTES:
raise AdapterError("body_too_large")
return body
class HackerNews(SearchAdapter):
name = "hackernews"
def request(self, query: str, limit: int) -> str:
params = urlencode({"query": query, "tags": "story", "hitsPerPage": limit})
return f"https://hn.algolia.com/api/v1/search?{params}"
def parse(self, body: bytes, limit: int) -> list[SocialHit]:
data = expect(load_json(body), dict, "response")
hits = []
for item in expect(data.get("hits"), list, "hits")[:limit]:
if not isinstance(item, dict):
continue
thread = f"https://news.ycombinator.com/item?id={quote(str(item.get('objectID') or ''))}"
found = self.hit(
title=item.get("title") or item.get("story_title"),
url=thread,
snippet=item.get("story_text") or item.get("url") or "",
author=item.get("author"),
score=item.get("points"),
comments=item.get("num_comments"),
created_at=item.get("created_at"),
)
if found and item.get("objectID"):
hits.append(found)
return hits
class StackExchange(SearchAdapter):
name = "stackexchange"
def request(self, query: str, limit: int) -> str:
params = urlencode(
{
"q": query,
"site": "stackoverflow",
"pagesize": limit,
"order": "desc",
"sort": "relevance",
}
)
return f"https://api.stackexchange.com/2.3/search/advanced?{params}"
def parse(self, body: bytes, limit: int) -> list[SocialHit]:
data = expect(load_json(body), dict, "response")
hits = []
for item in expect(data.get("items"), list, "items")[:limit]:
if not isinstance(item, dict):
continue
owner = item.get("owner") if isinstance(item.get("owner"), dict) else {}
tags = item.get("tags") if isinstance(item.get("tags"), list) else []
found = self.hit(
title=item.get("title"),
url=item.get("link"),
snippet=", ".join(str(tag) for tag in tags[:8]),
author=owner.get("display_name"),
score=item.get("score"),
comments=item.get("answer_count"),
created_at=item.get("creation_date"),
)
if found:
hits.append(found)
return hits
class GitHub(SearchAdapter):
name = "github"
def request(self, query: str, limit: int) -> str:
params = urlencode({"q": query, "per_page": limit})
return f"https://api.github.com/search/issues?{params}"
def parse(self, body: bytes, limit: int) -> list[SocialHit]:
data = expect(load_json(body), dict, "response")
hits = []
for item in expect(data.get("items"), list, "items")[:limit]:
if not isinstance(item, dict):
continue
user = item.get("user") if isinstance(item.get("user"), dict) else {}
reactions = item.get("reactions") if isinstance(item.get("reactions"), dict) else {}
found = self.hit(
title=item.get("title"),
url=item.get("html_url"),
snippet=item.get("body"),
author=user.get("login"),
score=reactions.get("total_count"),
comments=item.get("comments"),
created_at=item.get("created_at"),
)
if found:
hits.append(found)
return hits
class DevTo(SearchAdapter):
name = "devto"
def request(self, query: str, limit: int) -> str:
tag = tag_for(query)
if not tag:
raise AdapterError("no_tag", "dev.to is searched by tag and the query has none")
return f"https://dev.to/api/articles?{urlencode({'tag': tag, 'per_page': limit})}"
def parse(self, body: bytes, limit: int) -> list[SocialHit]:
data = expect(load_json(body), list, "response")
hits = []
for item in data[:limit]:
if not isinstance(item, dict):
continue
user = item.get("user") if isinstance(item.get("user"), dict) else {}
found = self.hit(
title=item.get("title"),
url=item.get("url"),
snippet=item.get("description"),
author=user.get("username"),
score=item.get("public_reactions_count"),
comments=item.get("comments_count"),
created_at=item.get("published_at"),
)
if found:
hits.append(found)
return hits
class Mastodon(SearchAdapter):
name = "mastodon"
def request(self, query: str, limit: int) -> str:
tag = tag_for(query)
if not tag:
raise AdapterError("no_tag", "Mastodon is searched by hashtag and the query has none")
return f"https://mastodon.social/api/v1/timelines/tag/{quote(tag)}?{urlencode({'limit': min(limit, 40)})}"
def parse(self, body: bytes, limit: int) -> list[SocialHit]:
data = expect(load_json(body), list, "response")
hits = []
for item in data[:limit]:
if not isinstance(item, dict):
continue
account = item.get("account") if isinstance(item.get("account"), dict) else {}
content = clean(item.get("content"), 500)
found = self.hit(
title=content[:120],
url=item.get("url"),
snippet=content,
author=account.get("acct"),
score=number(item.get("favourites_count")) + number(item.get("reblogs_count")),
comments=item.get("replies_count"),
created_at=item.get("created_at"),
)
if found:
hits.append(found)
return hits
devplacepy/services/nodes/adapters/reddit.py
+43
View File
@@ -0,0 +1,43 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
from urllib.parse import urlencode
from devplacepy.services.nodes.adapters.base import SearchAdapter, SocialHit, expect, load_json
BASE = "https://www.reddit.com"
class Reddit(SearchAdapter):
name = "reddit"
direct = False
def request(self, query: str, limit: int) -> str:
params = urlencode({"q": query, "limit": limit, "raw_json": 1, "sort": "relevance"})
return f"{BASE}/search.json?{params}"
def parse(self, body: bytes, limit: int) -> list[SocialHit]:
data = expect(load_json(body), dict, "response")
listing = expect(data.get("data"), dict, "listing")
hits = []
for child in expect(listing.get("children"), list, "children")[:limit]:
item = child.get("data") if isinstance(child, dict) else None
if not isinstance(item, dict):
continue
permalink = item.get("permalink")
if not isinstance(permalink, str) or not permalink.startswith("/r/"):
continue
found = self.hit(
title=item.get("title"),
url=BASE + permalink,
snippet=item.get("selftext"),
author=item.get("author"),
score=item.get("score"),
comments=item.get("num_comments"),
created_at=item.get("created_utc"),
via="fetch",
)
if found:
hits.append(found)
return hits
devplacepy/services/nodes/adapters/registry.py
+12
View File
@@ -0,0 +1,12 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
from devplacepy.services.nodes.adapters.base import SearchAdapter
from devplacepy.services.nodes.adapters.direct import DevTo, GitHub, HackerNews, Mastodon, StackExchange
from devplacepy.services.nodes.adapters.reddit import Reddit
ADAPTERS: dict[str, SearchAdapter] = {
item.name: item for item in (HackerNews(), StackExchange(), GitHub(), DevTo(), Mastodon(), Reddit())
}
DEFAULT_PLATFORMS = ("hackernews", "stackexchange", "github", "devto", "mastodon", "reddit")
devplacepy/services/nodes/broker.py
+127
View File
@@ -0,0 +1,127 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import asyncio
import base64
import binascii
import json
import time
from dataclasses import dataclass
from devplacepy.database import nodes as store
from devplacepy.services.nodes.policy import Target
from devplacepy.services.nodes.spec import SPEC
LIMITS = SPEC.limits
DONE = "done"
FAILED = "failed"
EXPIRED = "expired"
TERMINAL = (DONE, FAILED, EXPIRED)
MAX_HEADERS = 64
MAX_HEADER_CHARS = 1024
DROPPED_HEADERS = frozenset({"set-cookie", "set-cookie2", "authorization", "proxy-authenticate"})
@dataclass(frozen=True)
class BrokerResult:
uid: str
status: str
http_status: int
headers: dict
body: bytes
truncated: bool
error: str
via: str = "fetch"
@property
def ok(self) -> bool:
return self.status == DONE
class BodyError(ValueError):
pass
def clean_headers(raw) -> dict[str, str]:
if not isinstance(raw, dict):
return {}
kept: dict[str, str] = {}
for key, value in raw.items():
if len(kept) >= MAX_HEADERS:
break
if not isinstance(key, str) or not isinstance(value, str):
continue
name = key.strip().lower()
if not name or name in DROPPED_HEADERS:
continue
kept[name[:MAX_HEADER_CHARS]] = value[:MAX_HEADER_CHARS]
return kept
def decode_body(body_b64: str) -> tuple[bytes, bool]:
try:
body = base64.b64decode(body_b64.encode(), validate=True)
except (binascii.Error, ValueError):
raise BodyError("bad_base64")
if len(body) > LIMITS.max_body_bytes:
return body[: LIMITS.max_body_bytes], True
return body, False
class JobBroker:
def enqueue(self, caller_kind: str, caller_id: str, target: Target, mode: str) -> str:
return store.enqueue_request(
caller_kind, caller_id, target.url, target.method, target.adapter, mode
)
def queued(self, limit: int = LIMITS.dispatch_batch) -> list[dict]:
return store.list_queued(limit)
def claim(self, uid: str, device_uid: str) -> bool:
return store.claim_request(uid, device_uid)
def complete(self, uid: str, http_status: int, headers: dict, body: bytes, truncated: bool) -> bool:
return store.finish_request(
uid,
result_status=http_status,
result_headers=json.dumps(headers),
result_body=base64.b64encode(body).decode(),
result_bytes=len(body),
truncated=truncated,
)
def fail(self, uid: str, error: str, *, expired: bool = False) -> bool:
return store.fail_request(uid, error, expired=expired)
def expire(self) -> dict[str, int]:
return store.expire_requests()
def read(self, uid: str) -> BrokerResult | None:
row = store.get_request(uid)
if row is None:
return None
return BrokerResult(
uid=row["uid"],
status=row["status"],
http_status=int(row.get("result_status") or 0),
headers=json.loads(row.get("result_headers") or "{}") if row.get("result_headers") else {},
body=base64.b64decode(row["result_body"]) if row.get("result_body") else b"",
truncated=bool(row.get("truncated")),
error=row.get("error") or "",
via=row.get("mode") if row.get("mode") in ("tunnel", "fetch") else "fetch",
)
async def wait(self, uid: str, timeout: float) -> BrokerResult:
deadline = time.monotonic() + timeout
pause = LIMITS.poll_interval_ms / 1000
while True:
row = store.get_request(uid)
if row is not None and row["status"] in TERMINAL:
return self.read(uid)
if time.monotonic() >= deadline:
self.fail(uid, "caller_timeout")
return self.read(uid)
await asyncio.sleep(pause)
devplacepy/services/nodes/cache.py
+90
View File
@@ -0,0 +1,90 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import hashlib
import time
from collections import OrderedDict
from dataclasses import dataclass
from typing import Callable
from devplacepy.services.nodes.spec import SPEC
LIMITS = SPEC.limits
@dataclass(frozen=True)
class CachedResult:
status: int
headers: dict
body: bytes
truncated: bool
stored_at: float
def cache_key(method: str, url: str) -> str:
return hashlib.sha256(f"{method.upper()} {url}".encode()).hexdigest()
class ResultCache:
def __init__(
self,
ttl_seconds: int = LIMITS.cache_ttl_seconds,
max_bytes: int = LIMITS.cache_max_bytes,
entry_max_bytes: int = LIMITS.cache_entry_max_bytes,
clock: Callable[[], float] = time.monotonic,
) -> None:
self.ttl_seconds = max(0, min(ttl_seconds, LIMITS.cache_ttl_seconds))
self.max_bytes = max_bytes
self.entry_max_bytes = min(entry_max_bytes, LIMITS.cache_entry_max_bytes)
self._clock = clock
self._entries: OrderedDict[str, CachedResult] = OrderedDict()
self._bytes = 0
self.hits = 0
self.misses = 0
@property
def size_bytes(self) -> int:
return self._bytes
def __len__(self) -> int:
return len(self._entries)
def _drop(self, key: str) -> None:
entry = self._entries.pop(key, None)
if entry is not None:
self._bytes -= len(entry.body)
def get(self, method: str, url: str) -> CachedResult | None:
key = cache_key(method, url)
entry = self._entries.get(key)
if entry is None:
self.misses += 1
return None
if self._clock() - entry.stored_at >= self.ttl_seconds:
self._drop(key)
self.misses += 1
return None
self._entries.move_to_end(key)
self.hits += 1
return entry
def put(
self, method: str, url: str, status: int, headers: dict, body: bytes, truncated: bool
) -> bool:
if self.ttl_seconds <= 0 or status >= 500 or truncated:
return False
if len(body) > self.entry_max_bytes or len(body) > self.max_bytes:
return False
key = cache_key(method, url)
self._drop(key)
self._entries[key] = CachedResult(status, dict(headers), body, truncated, self._clock())
self._bytes += len(body)
while self._bytes > self.max_bytes and self._entries:
oldest = next(iter(self._entries))
self._drop(oldest)
return True
def clear(self) -> None:
self._entries.clear()
self._bytes = 0
devplacepy/services/nodes/circuit.py
+71
View File
@@ -0,0 +1,71 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import time
from dataclasses import dataclass
from typing import Callable, Hashable
from devplacepy.services.nodes.spec import SPEC
LIMITS = SPEC.limits
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
@dataclass
class _Circuit:
failures: int = 0
opened_at: float = 0.0
probing: bool = False
class CircuitBreaker:
def __init__(
self,
failures: int = LIMITS.breaker_failures,
open_seconds: float = LIMITS.breaker_open_seconds,
clock: Callable[[], float] = time.monotonic,
) -> None:
self.failures = max(1, failures)
self.open_seconds = open_seconds
self._clock = clock
self._circuits: dict[Hashable, _Circuit] = {}
def state(self, key: Hashable) -> str:
circuit = self._circuits.get(key)
if circuit is None or circuit.failures < self.failures:
return CLOSED
if self._clock() - circuit.opened_at < self.open_seconds:
return OPEN
return HALF_OPEN
def allows(self, key: Hashable) -> bool:
state = self.state(key)
if state == CLOSED:
return True
if state == OPEN:
return False
return not self._circuits[key].probing
def begin(self, key: Hashable) -> None:
if self.state(key) == HALF_OPEN:
self._circuits[key].probing = True
def success(self, key: Hashable) -> None:
self._circuits.pop(key, None)
def failure(self, key: Hashable) -> None:
circuit = self._circuits.setdefault(key, _Circuit())
circuit.probing = False
circuit.failures += 1
if circuit.failures >= self.failures:
circuit.opened_at = self._clock()
def forget(self, key: Hashable) -> None:
self._circuits.pop(key, None)
def open_count(self) -> int:
return sum(1 for key in list(self._circuits) if self.state(key) != CLOSED)
devplacepy/services/nodes/dispatcher.py
+273
View File
@@ -0,0 +1,273 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import asyncio
import logging
import time
from typing import Callable
from devplacepy.database import nodes as store
from devplacepy.services.nodes.broker import BodyError, JobBroker, clean_headers, decode_body
from devplacepy.services.nodes.circuit import CircuitBreaker
from devplacepy.services.nodes.ledger import UsageLedger
from devplacepy.services.nodes.policy import DestinationPolicy, PolicyError, Target
from devplacepy.services.nodes.registry import NodeCounters, NodeRegistry
from devplacepy.services.nodes.scheduler import MEGABYTE, NodeScheduler
from devplacepy.services.nodes.session import Inflight, NodeSession
from devplacepy.services.nodes.spec import MODE_FETCH, MODE_TUNNEL, SPEC
from devplacepy.services.nodes.tunnel import TunnelBridge, TunnelError
logger = logging.getLogger(__name__)
LIMITS = SPEC.limits
JOB_HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; DevPlaceBot/1.0; +https://devplace.net)",
"Accept": "application/json, text/html;q=0.9, */*;q=0.5",
}
REQUEST_OVERHEAD_BYTES = 256
DONOR_LIMIT_PAUSE_SECONDS = 300
CONCURRENCY_PAUSE_SECONDS = 5
THROTTLED_STATUSES = (429,)
DONOR_LIMIT_CODES = {
"bandwidth_exhausted": None,
"outside_hours": DONOR_LIMIT_PAUSE_SECONDS,
"concurrency_exceeded": CONCURRENCY_PAUSE_SECONDS,
}
REFUSAL_CODES = frozenset(item.code for item in SPEC.refusals) - set(DONOR_LIMIT_CODES) - {
"body_too_large",
"timeout",
}
class NodeDispatcher:
def __init__(
self,
registry: NodeRegistry,
scheduler: NodeScheduler,
broker: JobBroker,
breaker: CircuitBreaker,
ledger: UsageLedger,
counters: NodeCounters,
policy: DestinationPolicy | None = None,
clock: Callable[[], float] = time.monotonic,
tunnels: TunnelBridge | None = None,
) -> None:
self.tunnels = tunnels or TunnelBridge()
self._tunnel_tasks: dict[str, asyncio.Task] = {}
self.registry = registry
self.scheduler = scheduler
self.broker = broker
self.breaker = breaker
self.ledger = ledger
self.counters = counters
self.policy = policy or DestinationPolicy()
self._clock = clock
def attach(self, session: NodeSession) -> None:
session.job_sink = self.on_job_frame
session.tunnel_sink = self.tunnels.on_frame
async def tick(self) -> int:
if store.count_open_requests() == 0:
return 0
self.broker.expire()
await self.sweep_inflight()
if self.registry.count() == 0:
return 0
assigned = 0
saturated: set[tuple[str, str]] = set()
for row in self.broker.queued():
assigned += await self._assign(row, saturated)
return assigned
async def _assign(self, row: dict, saturated: set) -> int:
uid = row["uid"]
try:
target = self.policy.check(row["url"], row["method"], row["mode"])
except PolicyError as exc:
self.broker.fail(uid, exc.code)
self.counters.inc("jobs_refused")
return 0
key = (target.destination.suffix, row["mode"])
if key in saturated:
return 0
owner = row["caller_id"] if row["caller_kind"] == "user" else ""
found = self.scheduler.candidates(target, row["mode"])
if not found:
saturated.add(key)
return 0
session = self.scheduler.choose(found, owner)
if session is None or not self.broker.claim(uid, session.device_uid):
return 0
job = Inflight(
job_id=uid,
host=target.host,
destination=target.destination.suffix,
method=target.method,
url=target.url,
started=self._clock(),
bytes_up=len(target.url.encode()) + REQUEST_OVERHEAD_BYTES,
)
session.inflight[uid] = job
self.scheduler.mark_started(session, target)
transport = self.scheduler.transport_of(session, target, row["mode"]) or MODE_FETCH
store.set_request_transport(uid, transport)
if transport == MODE_TUNNEL:
self._tunnel_tasks[uid] = asyncio.create_task(self._run_tunnel(session, job, target))
self.counters.inc("jobs_assigned")
return 1
try:
await session.send(
"job",
job_id=uid,
method=target.method,
url=target.url,
headers=JOB_HEADERS,
timeout=LIMITS.job_timeout_seconds,
)
except Exception:
session.inflight.pop(uid, None)
self._failed(session, job, "send_failed")
return 0
self.counters.inc("jobs_assigned")
return 1
def _failed(self, session: NodeSession, job: Inflight, code: str, node_fault: bool = True) -> None:
self.broker.fail(job.job_id, code)
self.counters.inc("jobs_failed")
if node_fault:
self.breaker.failure(("node", session.device_uid))
store.add_event("job_failed", device_uid=session.device_uid, user_uid=session.user_uid, detail=code)
self.ledger.record_failure(session.device_uid, session.user_uid)
async def on_job_frame(self, session: NodeSession, job: Inflight, frame: dict) -> None:
session.inflight.pop(job.job_id, None)
if frame["type"] == "job_error":
self._on_error(session, job, frame["code"])
return
self._on_result(session, job, frame)
def _integrity_failure(self, session: NodeSession, job: Inflight, code: str) -> None:
self.counters.inc("integrity_failures")
self._failed(session, job, code)
self.ledger.record_integrity_failure(session.device_uid, session.user_uid)
store.raise_flag(
"integrity", device_uid=session.device_uid, user_uid=session.user_uid, detail=code
)
def _on_error(self, session: NodeSession, job: Inflight, code: str) -> None:
if code in DONOR_LIMIT_CODES:
pause = DONOR_LIMIT_CODES[code]
if pause is None:
session.used_today = int(session.settings.get("bandwidth_mb_day") or 0) * MEGABYTE
else:
session.paused_until = self._clock() + pause
self._failed(session, job, code, node_fault=False)
return
if code in REFUSAL_CODES:
self.counters.inc("jobs_refused")
store.raise_flag(
"refusal_mismatch",
device_uid=session.device_uid,
user_uid=session.user_uid,
detail=f"{code} {job.destination}",
)
self._failed(session, job, code, node_fault=False)
return
self._failed(session, job, code)
def _on_result(self, session: NodeSession, job: Inflight, frame: dict) -> None:
status = frame["status"]
if not 100 <= status <= 599:
self._integrity_failure(session, job, "bad_status")
return
try:
body, over = decode_body(frame["body_b64"])
except BodyError as exc:
self._integrity_failure(session, job, str(exc))
return
truncated = bool(frame.get("truncated")) or over
if truncated:
self.counters.inc("results_truncated")
headers = clean_headers(frame.get("headers"))
if not self.broker.complete(job.job_id, status, headers, body, truncated):
return
self._account(session, job, status, job.bytes_up, len(body))
async def _run_tunnel(self, session: NodeSession, job: Inflight, target: Target) -> None:
try:
result = await self.tunnels.fetch(session, target, LIMITS.job_timeout_seconds)
except asyncio.CancelledError:
raise
except TunnelError as exc:
self._tunnel_ended(session, job, exc.code)
return
except Exception:
logger.exception("tunnel job %s failed", job.job_id)
self._tunnel_ended(session, job, "tunnel_failed")
return
finally:
self._tunnel_tasks.pop(job.job_id, None)
if session.inflight.pop(job.job_id, None) is None:
return
if result.truncated:
self.counters.inc("results_truncated")
if not self.broker.complete(
job.job_id, result.status, clean_headers(result.headers), result.body, result.truncated
):
return
self._account(session, job, result.status, result.bytes_up, result.bytes_down)
def _tunnel_ended(self, session: NodeSession, job: Inflight, code: str) -> None:
if session.inflight.pop(job.job_id, None) is not None:
self._failed(session, job, code)
def _account(self, session: NodeSession, job: Inflight, status: int, up: int, down: int) -> None:
self.counters.inc("jobs_done")
spent = up + down
accepted = self.ledger.record_success(
session.device_uid, session.user_uid, bytes_up=up, bytes_down=down
)
session.used_today += spent
if accepted:
self.ledger.reward(session.user_uid)
if not accepted:
session.used_today = int(session.settings.get("bandwidth_mb_day") or 0) * MEGABYTE
self.breaker.success(("node", session.device_uid))
destination = ("destination", job.destination)
if status >= 500 or status in THROTTLED_STATUSES:
self.breaker.failure(destination)
else:
self.breaker.success(destination)
async def sweep_inflight(self) -> int:
limit = LIMITS.job_timeout_seconds + LIMITS.job_grace_seconds
now = self._clock()
swept = 0
for session in self.registry.sessions():
for job in list(session.inflight.values()):
if now - job.started > limit:
task = self._tunnel_tasks.pop(job.job_id, None)
if task is not None:
task.cancel()
session.inflight.pop(job.job_id, None)
self._failed(session, job, "timeout")
swept += 1
return swept
def release_session(self, session: NodeSession) -> int:
jobs = list(session.inflight.values())
session.inflight.clear()
self.tunnels.release_session(session)
for job in jobs:
task = self._tunnel_tasks.pop(job.job_id, None)
if task is not None:
task.cancel()
self._failed(session, job, "node_disconnected")
return len(jobs)
def target_for(self, url: str, method: str = "GET", mode: str = "any") -> Target:
return self.policy.check(url, method, mode)
devplacepy/services/nodes/fabric.py
+204
View File
@@ -0,0 +1,204 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import json
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from devplacepy.database import get_int_setting, get_table
from devplacepy.database import nodes as store
from devplacepy.services.base import BaseService
from devplacepy.services.nodes.broker import FAILED, JobBroker
from devplacepy.services.nodes.cache import ResultCache
from devplacepy.services.nodes.consent import ConsentGate
from devplacepy.services.nodes.policy import DestinationPolicy, PolicyError, Target
from devplacepy.services.nodes.spec import SPEC
LIMITS = SPEC.limits
VIA_FETCH = "fetch"
VIA_CACHE = "cache"
RETRYABLE = frozenset(
{
"node_disconnected",
"timeout",
"send_failed",
"concurrency_exceeded",
"outside_hours",
"bandwidth_exhausted",
}
)
MAX_ATTEMPTS = 2
GUEST_KIND = "guest"
USER_KIND = "user"
class FabricError(Exception):
def __init__(self, code: str, message: str = ""):
super().__init__(message or code)
self.code = code
self.message = message or code
class FabricRefused(FabricError):
pass
class FabricUnavailable(FabricError):
pass
class FabricQuotaExceeded(FabricError):
pass
@dataclass(frozen=True)
class FabricResult:
url: str
status: int
headers: dict
body: bytes
via: str
truncated: bool
elapsed_ms: int
@dataclass(frozen=True)
class FabricStatus:
enabled: bool
nodes_online: int
class NodeFabric:
def __init__(
self,
policy: DestinationPolicy | None = None,
broker: JobBroker | None = None,
cache: ResultCache | None = None,
gate: ConsentGate | None = None,
) -> None:
self.policy = policy or DestinationPolicy()
self.broker = broker or JobBroker()
self.cache = cache or ResultCache()
self.gate = gate or ConsentGate()
def available(self) -> FabricStatus:
enabled = self.gate.switched_on()
return FabricStatus(enabled, self._online() if enabled else 0)
@staticmethod
def _online() -> int:
row = get_table("service_state").find_one(name="nodes")
if not row or not row.get("heartbeat"):
return 0
try:
beat = datetime.fromisoformat(row["heartbeat"])
except ValueError:
return 0
if beat.tzinfo is None:
beat = beat.replace(tzinfo=timezone.utc)
if (datetime.now(timezone.utc) - beat).total_seconds() > BaseService.STALE_SECONDS * 2:
return 0
try:
stats = json.loads(row.get("metrics") or "{}").get("stats") or []
except ValueError:
return 0
for item in stats:
if item.get("label") == "connected":
return int(item.get("value") or 0)
return 0
def can_serve(self, url: str, method: str = "GET") -> bool:
try:
self.policy.check(url, method)
except PolicyError:
return False
status = self.available()
return status.enabled and status.nodes_online > 0
def _quota(self, caller_kind: str) -> int:
if caller_kind == USER_KIND:
return get_int_setting("nodes_daily_quota_user", LIMITS.default_daily_requests_user)
return get_int_setting("nodes_daily_quota_guest", LIMITS.default_daily_requests_guest)
def _sync_cache_ttl(self) -> None:
configured = get_int_setting("nodes_cache_ttl_seconds", LIMITS.cache_ttl_seconds)
self.cache.ttl_seconds = max(0, min(configured, LIMITS.cache_ttl_seconds))
async def fetch(
self,
url: str,
*,
caller_kind: str,
caller_id: str,
method: str = "GET",
mode: str = "any",
use_cache: bool = True,
timeout: float | None = None,
) -> FabricResult:
try:
target = self.policy.check(url, method, mode)
except PolicyError as exc:
raise FabricRefused(exc.code, exc.reason)
started = time.monotonic()
self._sync_cache_ttl()
if use_cache:
cached = self.cache.get(target.method, target.url)
if cached is not None:
return FabricResult(
target.url, cached.status, cached.headers, cached.body, VIA_CACHE, cached.truncated, 0
)
status = self.available()
if not status.enabled:
raise FabricUnavailable("kill_switch", "Community donation is switched off")
if status.nodes_online <= 0:
raise FabricUnavailable("no_nodes", "No donor node is connected")
if not store.reserve_caller_request(caller_kind, caller_id, self._quota(caller_kind)):
raise FabricQuotaExceeded("quota", "Daily node request quota reached")
wait = min(float(timeout or LIMITS.default_wait_seconds), float(LIMITS.max_wait_seconds))
result = await self._run(target, caller_kind, caller_id, mode, wait)
if use_cache:
self.cache.put(
target.method, target.url, result.http_status, result.headers, result.body, result.truncated
)
return FabricResult(
target.url,
result.http_status,
result.headers,
result.body,
result.via,
result.truncated,
int((time.monotonic() - started) * 1000),
)
async def _run(self, target: Target, caller_kind: str, caller_id: str, mode: str, wait: float):
deadline = time.monotonic() + wait
error = ""
for _ in range(MAX_ATTEMPTS):
remaining = deadline - time.monotonic()
if remaining <= 0:
break
try:
uid = self.broker.enqueue(caller_kind, caller_id, target, mode)
except store.QueueFull:
raise FabricUnavailable("queue_full", "The node queue is full")
result = await self.broker.wait(uid, remaining)
if result.ok:
return result
error = result.error or result.status
if result.status == FAILED and result.error in RETRYABLE:
continue
break
raise FabricUnavailable(str(error) or "failed", f"Node fetch failed: {error}")
_shared: NodeFabric | None = None
def shared_fabric() -> NodeFabric:
global _shared
if _shared is None:
_shared = NodeFabric()
return _shared
devplacepy/services/nodes/ledger.py
+25
View File
@@ -7,6 +7,17 @@ from datetime import datetime, timezone
from devplacepy.database import nodes as store
XP_PER_JOB = 1
GIGABYTE = 1024 * 1024 * 1024
BYTE_BADGES = (
(GIGABYTE, "Gigabyte Donor"),
(10 * GIGABYTE, "10 GB Donor"),
(100 * GIGABYTE, "100 GB Donor"),
)
FIRST_BADGE = "Donor"
STEADY_DAYS = 30
STEADY_BADGE = "Steady Donor"
@dataclass(frozen=True)
class DonationSummary:
@@ -49,6 +60,20 @@ class UsageLedger:
def record_integrity_failure(self, device_uid: str, user_uid: str) -> None:
store.record_integrity_failure(device_uid, user_uid)
def reward(self, user_uid: str) -> None:
from devplacepy.utils import award_badge, award_rewards, notify_badge
if self.claim_xp(user_uid, XP_PER_JOB):
award_rewards(user_uid, XP_PER_JOB)
donor = store.get_donor(user_uid)
earned = [FIRST_BADGE] if donor["requests_ok"] >= 1 else []
earned += [name for size, name in BYTE_BADGES if donor["bytes_total"] >= size]
if donor["requests_ok"] >= STEADY_DAYS and store.donation_days(user_uid) >= STEADY_DAYS:
earned.append(STEADY_BADGE)
for name in earned:
if award_badge(user_uid, name):
notify_badge(user_uid, name)
def claim_xp(self, user_uid: str, amount: int) -> bool:
return store.claim_xp(user_uid, amount)
devplacepy/services/nodes/policy.py
+91
View File
@@ -0,0 +1,91 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import ipaddress
from dataclasses import dataclass
from urllib.parse import urlsplit
from devplacepy.services.nodes.spec import ALLOWED_PORTS, MODES, SPEC, DestinationSpec
LIMITS = SPEC.limits
DEFAULT_PORTS = {"http": 80, "https": 443}
SCHEMES = tuple(DEFAULT_PORTS)
SHARED_ADDRESS_SPACE = ipaddress.ip_network("100.64.0.0/10")
class PolicyError(ValueError):
def __init__(self, code: str, reason: str = ""):
super().__init__(reason or code)
self.code = code
self.reason = reason or code
@dataclass(frozen=True)
class Target:
url: str
method: str
host: str
port: int
destination: DestinationSpec
@property
def adapter(self) -> str:
return self.destination.adapter
def is_public_address(address: str) -> bool:
try:
ip = ipaddress.ip_address(address)
except ValueError:
return False
if ip.version == 4 and ip in SHARED_ADDRESS_SPACE:
return False
return ip.is_global and not ip.is_multicast
class DestinationPolicy:
def check(self, url: str, method: str = "GET", mode: str = "any") -> Target:
if mode not in MODES:
raise PolicyError("bad_mode", f"unknown mode {mode!r}")
if not isinstance(url, str) or not url or len(url) > LIMITS.url_max_chars:
raise PolicyError("bad_url", "url is missing or too long")
try:
parts = urlsplit(url.strip())
port = parts.port
except ValueError:
raise PolicyError("bad_url", "url cannot be parsed")
if parts.scheme not in SCHEMES:
raise PolicyError("bad_url", "only http and https are allowed")
if parts.username is not None or parts.password is not None:
raise PolicyError("bad_url", "credentials in a url are not allowed")
host = (parts.hostname or "").lower().rstrip(".")
if not host:
raise PolicyError("bad_url", "url has no host")
if _is_address_literal(host):
raise PolicyError("host_not_allowed", "address literals are not allowed")
destination = SPEC.destination(host)
if destination is None:
raise PolicyError("host_not_allowed", "destination is not on the allowlist")
port = port or DEFAULT_PORTS[parts.scheme]
if port not in ALLOWED_PORTS or port not in destination.ports:
raise PolicyError("port_not_allowed", "port is not allowed")
verb = (method or "").upper()
if verb not in destination.methods:
raise PolicyError("method_not_allowed", "method is not allowed here")
return Target(parts.geturl(), verb, host, port, destination)
def check_redirect(self, location: str, method: str = "GET") -> Target:
try:
return self.check(location, method)
except PolicyError as exc:
raise PolicyError("redirect_not_allowed", exc.reason)
def _is_address_literal(host: str) -> bool:
try:
ipaddress.ip_address(host.strip("[]"))
return True
except ValueError:
return False
devplacepy/services/nodes/registry.py
+6
View File
@@ -20,6 +20,12 @@ COUNTER_NAMES: tuple[str, ...] = (
"evicted_heartbeat",
"kicked",
"replaced",
"jobs_assigned",
"jobs_done",
"jobs_failed",
"jobs_refused",
"integrity_failures",
"results_truncated",
)
devplacepy/services/nodes/scheduler.py
+163
View File
@@ -0,0 +1,163 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import random
import time
from functools import lru_cache
from datetime import datetime, timezone
from typing import Callable
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
from devplacepy.services.nodes.circuit import CircuitBreaker
from devplacepy.services.nodes.policy import Target
from devplacepy.services.nodes.registry import NodeRegistry
from devplacepy.services.nodes.session import NodeSession
from devplacepy.services.nodes.spec import MODE_ANY, MODE_FETCH, MODE_TUNNEL, SPEC
LIMITS = SPEC.limits
MEGABYTE = 1024 * 1024
MINUTES_PER_DAY = 24 * 60
SUPPORTED_MODES = (MODE_TUNNEL, MODE_FETCH)
def _minutes(value: str) -> int | None:
try:
hours, minutes = value.split(":")
return int(hours) * 60 + int(minutes)
except (ValueError, AttributeError):
return None
@lru_cache(maxsize=256)
def _zone(name: str):
try:
return ZoneInfo(name or "UTC")
except (ZoneInfoNotFoundError, ValueError):
return timezone.utc
@lru_cache(maxsize=8192)
def _window_open(start: str, end: str, zone_name: str, minute_epoch: int) -> bool:
first, last = _minutes(start), _minutes(end)
if first is None or last is None or first == last:
return True
moment = datetime.fromtimestamp(minute_epoch * 60, timezone.utc)
local = moment.astimezone(_zone(zone_name))
now = local.hour * 60 + local.minute
if first < last:
return first <= now < last
return now >= first or now < last
def in_active_hours(settings: dict, moment: datetime) -> bool:
start, end = settings.get("active_from") or "", settings.get("active_to") or ""
if not start and not end:
return True
return _window_open(start, end, settings.get("timezone") or "UTC", int(moment.timestamp() // 60))
class NodeScheduler:
def __init__(
self,
registry: NodeRegistry,
breaker: CircuitBreaker,
clock: Callable[[], float] = time.monotonic,
wall: Callable[[], datetime] = lambda: datetime.now(timezone.utc),
rng: random.Random | None = None,
) -> None:
self.registry = registry
self.breaker = breaker
self._clock = clock
self._wall = wall
self._rng = rng or random.Random()
self._destination_started: dict[str, float] = {}
@staticmethod
def node_key(session: NodeSession) -> tuple[str, str]:
return ("node", session.device_uid)
@staticmethod
def destination_key(target: Target) -> tuple[str, str]:
return ("destination", target.destination.suffix)
def transports_for(self, target: Target, mode: str) -> tuple[str, ...]:
if mode not in (MODE_ANY, *SUPPORTED_MODES):
return ()
return tuple(
item
for item in SUPPORTED_MODES
if item in target.destination.modes and mode in (MODE_ANY, item)
)
def transport_of(self, session: NodeSession, target: Target, mode: str) -> str | None:
for item in self.transports_for(target, mode):
if item in session.modes:
return item
return None
def destination_ready(self, target: Target) -> bool:
interval = 1.0 / max(target.destination.max_rps, 0.001)
last = self._destination_started.get(target.destination.suffix)
return last is None or self._clock() - last >= interval
def node_ready(self, session: NodeSession, target: Target, transport: str) -> bool:
if session.closed or self._clock() < session.paused_until:
return False
if transport not in session.modes:
return False
if len(session.inflight) >= session.max_concurrent:
return False
if target.adapter not in (session.settings.get("allowed_platforms") or []):
return False
if not in_active_hours(session.settings, self._wall()):
return False
session.roll_day()
cap = int(session.settings.get("bandwidth_mb_day") or 0) * MEGABYTE
if session.used_today >= cap:
return False
last = session.host_started.get(target.destination.suffix)
if last is not None and self._clock() - last < LIMITS.min_seconds_per_node_domain:
return False
return self.breaker.allows(self.node_key(session))
def candidates(self, target: Target, mode: str = MODE_ANY) -> list[NodeSession]:
if not self.destination_ready(target) or not self.breaker.allows(self.destination_key(target)):
return []
for transport in self.transports_for(target, mode):
found = [
session
for session in self.registry.sessions()
if self.node_ready(session, target, transport)
]
if found:
return found
return []
def eligible(self, target: Target, mode: str = MODE_ANY, caller_owner: str = "") -> list[NodeSession]:
found = self.candidates(target, mode)
if not caller_owner:
return found
return [session for session in found if session.user_uid != caller_owner]
def choose(self, found: list[NodeSession], caller_owner: str = "") -> NodeSession | None:
if caller_owner:
found = [session for session in found if session.user_uid != caller_owner]
if not found:
return None
return min(
found,
key=lambda item: (len(item.inflight), item.last_assigned, self._rng.random()),
)
def pick(self, target: Target, mode: str = MODE_ANY, caller_owner: str = "") -> NodeSession | None:
return self.choose(self.candidates(target, mode), caller_owner)
def mark_started(self, session: NodeSession, target: Target) -> None:
now = self._clock()
session.last_assigned = now
session.host_started[target.destination.suffix] = now
self._destination_started[target.destination.suffix] = now
self.breaker.begin(self.node_key(session))
self.breaker.begin(self.destination_key(target))
devplacepy/services/nodes/service.py
+59
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import asyncio
import logging
import time
@@ -10,11 +11,17 @@ from devplacepy.database.nodes import (
add_event,
effective_status,
get_devices_by_uids,
get_settings_by_users,
)
from devplacepy.services.base import BaseService, ConfigField
from devplacepy.services.nodes.broker import JobBroker
from devplacepy.services.nodes.circuit import CircuitBreaker
from devplacepy.services.nodes.consent import ConsentGate
from devplacepy.services.nodes.dispatcher import NodeDispatcher
from devplacepy.services.nodes.fabric import NodeFabric
from devplacepy.services.nodes.ledger import UsageLedger
from devplacepy.services.nodes.registry import NodeCounters, NodeRegistry
from devplacepy.services.nodes.scheduler import NodeScheduler
from devplacepy.services.nodes.spec import SPEC
logger = logging.getLogger(__name__)
@@ -64,7 +71,52 @@ class NodesService(BaseService):
self.registry = NodeRegistry(self.counters)
self.gate = ConsentGate()
self.ledger = UsageLedger()
self.breaker = CircuitBreaker()
self.broker = JobBroker()
self.scheduler = NodeScheduler(self.registry, self.breaker)
self.dispatcher = NodeDispatcher(
self.registry,
self.scheduler,
self.broker,
self.breaker,
self.ledger,
self.counters,
)
self.fabric = NodeFabric(broker=self.broker, gate=self.gate)
self._last_prune = 0.0
self._dispatch_task: asyncio.Task | None = None
def ensure_dispatching(self) -> None:
loop = asyncio.get_running_loop()
task = self._dispatch_task
if task is not None and not task.done() and task.get_loop() is loop:
return
self._dispatch_task = loop.create_task(self._dispatch_loop())
async def _dispatch_loop(self) -> None:
pause = LIMITS.dispatcher_tick_ms / 1000
while True:
try:
await self.dispatcher.tick()
except asyncio.CancelledError:
raise
except Exception:
logger.exception("node dispatch tick failed")
await asyncio.sleep(pause)
async def on_enable(self) -> None:
self.ensure_dispatching()
async def _refresh_settings(self, sessions) -> None:
fresh = get_settings_by_users(item.user_uid for item in sessions)
for session in sessions:
current = fresh.get(session.user_uid)
if current is not None and current != session.settings:
session.settings = current
try:
await session.send("settings_update", settings=current)
except Exception:
pass
async def reconcile(self) -> int:
sessions = self.registry.sessions()
@@ -87,6 +139,7 @@ class NodesService(BaseService):
detail=reason,
)
kicked += 1
await self._refresh_settings(self.registry.sessions())
return kicked
@staticmethod
@@ -113,6 +166,9 @@ class NodesService(BaseService):
self.log(f"pruned {pruned}")
async def on_disable(self) -> None:
if self._dispatch_task is not None:
self._dispatch_task.cancel()
self._dispatch_task = None
await self.registry.kick_all("service_stopped")
def collect_metrics(self) -> dict:
@@ -123,6 +179,9 @@ class NodesService(BaseService):
{"label": "capacity", "value": self.registry.capacity()},
{"label": "accepted", "value": counts["connected_total"]},
{"label": "kicked", "value": counts["kicked"]},
{"label": "jobs done", "value": counts["jobs_done"]},
{"label": "jobs failed", "value": counts["jobs_failed"]},
{"label": "open circuits", "value": self.breaker.open_count()},
],
"table": {
"columns": ["counter", "value"],
devplacepy/services/nodes/session.py
+69 -5
View File
@@ -4,8 +4,11 @@ from __future__ import annotations
import asyncio
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Awaitable, Callable
from devplacepy.database.nodes import get_node_settings
from devplacepy.database.nodes import get_node_settings, usage_today
from devplacepy.services.nodes.frames import FrameError, decode, encode
from devplacepy.services.nodes.spec import (
CLIENT_TO_SERVER,
@@ -20,6 +23,21 @@ LIMITS = SPEC.limits
END_HEARTBEAT = "heartbeat"
END_UNEXPECTED_FRAME = "unexpected_frame"
TUNNEL_FRAMES = ("tunnel_opened", "tunnel_data", "tunnel_close")
@dataclass
class Inflight:
job_id: str
host: str
destination: str
method: str
url: str
started: float
bytes_up: int
JobSink = Callable[["NodeSession", Inflight, dict], Awaitable[None]]
class NodeSession:
@@ -34,6 +52,37 @@ class NodeSession:
self.reported_up = 0
self.reported_down = 0
self.closed = False
self.settings: dict = {}
self.inflight: dict[str, Inflight] = {}
self.host_started: dict[str, float] = {}
self.used_today = 0
self.used_day = ""
self.paused_until = 0.0
self.last_assigned = 0.0
self.job_sink: JobSink | None = None
self.tunnel_sink: Callable[["NodeSession", dict], bool] | None = None
self.tunnels: dict = {}
@property
def modes(self) -> list[str]:
return list(self.hello.get("modes") or [])
@property
def max_concurrent(self) -> int:
limit = min(
LIMITS.max_concurrent_per_node,
int(self.settings.get("max_concurrent") or LIMITS.max_concurrent_per_node),
)
declared = (self.hello.get("limits") or {}).get("max_concurrent")
if isinstance(declared, int) and not isinstance(declared, bool) and declared >= 1:
limit = min(limit, declared)
return max(1, limit)
def roll_day(self) -> None:
today = datetime.now(timezone.utc).date().isoformat()
if self.used_day != today:
self.used_day = today
self.used_today = 0
async def send(self, frame_type: str, **fields) -> None:
await self.websocket.send_text(encode(frame_type, SERVER_TO_CLIENT, **fields))
@@ -76,16 +125,20 @@ class NodeSession:
raise FrameError("expected_hello", "the first frame must be hello")
self._check_hello(frame)
self.hello = frame
self.settings = get_node_settings(self.user_uid)
self.roll_day()
today = usage_today(self.device_uid)
self.used_today = today["bytes_up"] + today["bytes_down"]
await self.send(
"welcome",
device_uid=self.device_uid,
limits=LIMITS.describe(),
destinations=[item.describe() for item in SPEC.destinations],
settings=get_node_settings(self.user_uid),
settings=self.settings,
)
return frame
def _handle(self, frame: dict) -> None:
async def _handle(self, frame: dict) -> None:
kind = frame["type"]
if kind == "pong":
return
@@ -93,7 +146,18 @@ class NodeSession:
self.reported_up += max(0, frame["bytes_up"])
self.reported_down += max(0, frame["bytes_down"])
return
raise FrameError(END_UNEXPECTED_FRAME, f"unexpected {kind} frame, no job is assigned")
if kind in TUNNEL_FRAMES:
if self.tunnel_sink is not None:
self.tunnel_sink(self, frame)
return
if kind in ("job_result", "job_error"):
job = self.inflight.get(frame["job_id"])
if job is not None and self.job_sink is not None:
await self.job_sink(self, job, frame)
return
raise FrameError(
END_UNEXPECTED_FRAME, f"unexpected {kind} frame, no such job is assigned"
)
async def run(self) -> str:
while not self.closed:
@@ -109,5 +173,5 @@ class NodeSession:
await self.send("ping")
continue
self.missed_pings = 0
self._handle(decode(raw, CLIENT_TO_SERVER))
await self._handle(decode(raw, CLIENT_TO_SERVER))
return "closed"
devplacepy/services/nodes/social.py
+96
View File
@@ -0,0 +1,96 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import asyncio
from dataclasses import dataclass, field
from itertools import zip_longest
from devplacepy.database import get_setting
from devplacepy.services.nodes.adapters.base import AdapterError, SearchAdapter, SocialHit
from devplacepy.services.nodes.adapters.direct import get as direct_get
from devplacepy.services.nodes.adapters.registry import ADAPTERS, DEFAULT_PLATFORMS
from devplacepy.services.nodes.fabric import FabricError, NodeFabric, shared_fabric
MAX_LIMIT = 20
DEFAULT_LIMIT = 8
MAX_QUERY_CHARS = 200
MAX_TOTAL_HITS = 60
PLATFORM_TIMEOUT_SECONDS = 20.0
@dataclass
class SocialOutcome:
hits: list[SocialHit] = field(default_factory=list)
unavailable: dict[str, str] = field(default_factory=dict)
def as_dict(self) -> dict:
return {
"hits": [item.as_dict() for item in self.hits],
"unavailable": dict(self.unavailable),
}
class SocialSearch:
def __init__(self, fabric: NodeFabric | None = None) -> None:
self.fabric = fabric or shared_fabric()
@staticmethod
def platforms_for(requested: list[str] | None) -> list[str]:
if not requested:
return list(DEFAULT_PLATFORMS)
return [name for name in dict.fromkeys(requested) if name in ADAPTERS]
async def search(
self,
query: str,
*,
caller_kind: str,
caller_id: str,
platforms: list[str] | None = None,
limit: int = DEFAULT_LIMIT,
) -> SocialOutcome:
query = " ".join(str(query or "").split())[:MAX_QUERY_CHARS]
outcome = SocialOutcome()
if not query:
return outcome
limit = max(1, min(int(limit or DEFAULT_LIMIT), MAX_LIMIT))
names = self.platforms_for(platforms)
results = await asyncio.gather(
*(self._one(ADAPTERS[name], query, limit, caller_kind, caller_id) for name in names)
)
buckets = []
for name, (hits, code) in zip(names, results):
if code:
outcome.unavailable[name] = code
buckets.append(hits)
for tier in zip_longest(*buckets):
outcome.hits.extend(item for item in tier if item)
outcome.hits = outcome.hits[:MAX_TOTAL_HITS]
return outcome
async def _one(
self, adapter: SearchAdapter, query: str, limit: int, caller_kind: str, caller_id: str
) -> tuple[list[SocialHit], str]:
try:
return await asyncio.wait_for(
self._fetch(adapter, query, limit, caller_kind, caller_id),
PLATFORM_TIMEOUT_SECONDS,
), ""
except asyncio.TimeoutError:
return [], "timeout"
except (AdapterError, FabricError) as exc:
return [], exc.code
async def _fetch(
self, adapter: SearchAdapter, query: str, limit: int, caller_kind: str, caller_id: str
) -> list[SocialHit]:
url = adapter.request(query, limit)
if adapter.direct:
return adapter.parse(await direct_get(url), limit)
if get_setting("nodes_social_via_nodes", "0") != "1":
raise AdapterError("nodes_not_used", "Routing through donor nodes is switched off")
result = await self.fabric.fetch(url, caller_kind=caller_kind, caller_id=caller_id)
if result.status >= 400:
raise AdapterError("upstream", f"HTTP {result.status}")
return adapter.parse(result.body, limit)
devplacepy/services/nodes/spec.py
+7
View File
@@ -97,6 +97,13 @@ class LimitSpec:
hello_timeout_seconds: int = 10
reconcile_seconds: int = 2
prune_interval_seconds: int = 3600
breaker_failures: int = 5
breaker_open_seconds: int = 60
queue_max_wait_seconds: int = 30
job_grace_seconds: int = 5
dispatch_batch: int = 50
url_max_chars: int = 2048
poll_interval_ms: int = 100
def describe(self) -> dict[str, int]:
return dict(self.__dict__)
devplacepy/services/nodes/tunnel.py
+220
View File
@@ -0,0 +1,220 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import asyncio
import base64
import binascii
import logging
from dataclasses import dataclass
import httpx
from devplacepy import stealth
from devplacepy.services.nodes.policy import Target
from devplacepy.services.nodes.session import NodeSession
from devplacepy.services.nodes.spec import SPEC
from devplacepy.utils import generate_uid
logger = logging.getLogger(__name__)
LIMITS = SPEC.limits
LOOPBACK = "127.0.0.1"
CHUNK_BYTES = 48 * 1024
REQUEST_HEAD_MAX = 8 * 1024
OPEN_TIMEOUT_SECONDS = 10.0
POLL_SECONDS = 1.0
TLS_OVERHEAD_BYTES = 512 * 1024
RELAY_CAP_BYTES = LIMITS.max_body_bytes * 2 + TLS_OVERHEAD_BYTES
CLOSED = object()
class TunnelError(Exception):
def __init__(self, code: str):
super().__init__(code)
self.code = code
@dataclass(frozen=True)
class TunnelResult:
status: int
headers: dict
body: bytes
truncated: bool
bytes_up: int
bytes_down: int
class Relay:
def __init__(self, session: NodeSession, target: Target) -> None:
self.session = session
self.target = target
self.conn_id = generate_uid()
self.frames: asyncio.Queue = asyncio.Queue(maxsize=256)
self.up = 0
self.down = 0
self.error = ""
self.used = False
def on_frame(self, frame: dict) -> None:
try:
self.frames.put_nowait(frame)
except asyncio.QueueFull:
self.error = "queue_full"
def drop(self) -> None:
try:
self.frames.put_nowait(CLOSED)
except asyncio.QueueFull:
pass
async def handle(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
try:
await self._serve(reader, writer)
except (ConnectionError, asyncio.IncompleteReadError, asyncio.LimitOverrunError):
pass
finally:
writer.close()
async def _serve(self, reader, writer) -> None:
head = await reader.readuntil(b"\r\n\r\n")
if len(head) > REQUEST_HEAD_MAX:
return
parts = head.split(b"\r\n", 1)[0].decode("latin-1", "replace").split()
if len(parts) < 2 or parts[0].upper() != "CONNECT" or self.used:
await self._refuse(writer, 405)
return
host, _, port = parts[1].rpartition(":")
if host.lower() != self.target.host.lower() or port != str(self.target.port):
self.error = "host_not_allowed"
await self._refuse(writer, 403)
return
self.used = True
self.session.tunnels[self.conn_id] = self
await self.session.send("tunnel_open", conn_id=self.conn_id, host=self.target.host, port=self.target.port)
if not await self._opened():
await self._refuse(writer, 502)
return
writer.write(b"HTTP/1.1 200 Connection established\r\n\r\n")
await writer.drain()
uplink = asyncio.create_task(self._uplink(reader))
try:
await self._downlink(writer)
finally:
uplink.cancel()
await self._close_donor()
@staticmethod
async def _refuse(writer, status: int) -> None:
writer.write(f"HTTP/1.1 {status} Refused\r\nContent-Length: 0\r\n\r\n".encode())
await writer.drain()
async def _opened(self) -> bool:
try:
frame = await asyncio.wait_for(self.frames.get(), OPEN_TIMEOUT_SECONDS)
except asyncio.TimeoutError:
self.error = "tunnel_open_timeout"
return False
if frame is CLOSED or frame["type"] != "tunnel_opened" or not frame["ok"]:
self.error = (frame.get("code") if isinstance(frame, dict) else "") or "tunnel_refused"
return False
return True
async def _uplink(self, reader) -> None:
while True:
chunk = await reader.read(CHUNK_BYTES)
if not chunk:
return
self.up += len(chunk)
if self.up + self.down > RELAY_CAP_BYTES:
self.error = "body_too_large"
self.drop()
return
await self.session.send(
"tunnel_data", conn_id=self.conn_id, b64=base64.b64encode(chunk).decode()
)
async def _downlink(self, writer) -> None:
while not self.error:
try:
frame = await asyncio.wait_for(self.frames.get(), POLL_SECONDS)
except asyncio.TimeoutError:
continue
if frame is CLOSED or frame["type"] == "tunnel_close":
return
if frame["type"] != "tunnel_data":
self.error = "bad_frame"
return
try:
data = base64.b64decode(frame["b64"].encode(), validate=True)
except (binascii.Error, ValueError):
self.error = "bad_base64"
return
self.down += len(data)
if self.up + self.down > RELAY_CAP_BYTES:
self.error = "body_too_large"
return
writer.write(data)
await writer.drain()
async def _close_donor(self) -> None:
self.session.tunnels.pop(self.conn_id, None)
try:
await self.session.send("tunnel_close", conn_id=self.conn_id)
except Exception:
pass
class TunnelBridge:
async def fetch(self, session: NodeSession, target: Target, timeout: float) -> TunnelResult:
relay = Relay(session, target)
server = await asyncio.start_server(relay.handle, LOOPBACK, 0)
port = server.sockets[0].getsockname()[1]
try:
return await asyncio.wait_for(
self._request(relay, target, f"http://{LOOPBACK}:{port}", timeout), timeout
)
except asyncio.TimeoutError:
raise TunnelError("timeout")
finally:
server.close()
session.tunnels.pop(relay.conn_id, None)
async def _request(self, relay: Relay, target: Target, proxy: str, timeout: float) -> TunnelResult:
limit = LIMITS.max_body_bytes
try:
async with stealth.stealth_async_client(
proxy=proxy, follow_redirects=False, timeout=httpx.Timeout(timeout, connect=OPEN_TIMEOUT_SECONDS + 2)
) as client:
async with client.stream(target.method, target.url) as response:
chunks = bytearray()
truncated = False
async for chunk in response.aiter_bytes():
chunks.extend(chunk)
if len(chunks) > limit:
truncated = True
del chunks[limit:]
break
return TunnelResult(
response.status_code,
dict(response.headers),
bytes(chunks),
truncated,
relay.up,
relay.down,
)
except httpx.HTTPError as exc:
raise TunnelError(relay.error or f"tunnel_failed:{type(exc).__name__}")
def on_frame(self, session: NodeSession, frame: dict) -> bool:
relay = session.tunnels.get(frame["conn_id"])
if relay is None:
return False
relay.on_frame(frame)
return True
def release_session(self, session: NodeSession) -> None:
for relay in list(session.tunnels.values()):
relay.drop()
session.tunnels.clear()