devplacepy/gaplord.md

gaplord: close every documented gap

Plan and result in one file. The plan was written first, from a read of the code behind each gap. Every item is then worked in order and its Status and Result lines are filled in when it is done, with the measurement or test that proves it. Nothing is checked off without evidence.

Written 2026-10-16. Scope: every entry under a Gaps heading in devplacepy/services/{nodes,sync,community,skills}/CLAUDE.md and nodes/extension/README.md, except the plain-text community passwords (explicitly excluded by the owner).

Legend: [ ] open, [~] in progress, [x] done and proven, [!] cannot be closed by code (reason stated, nothing faked).

Ground rules for this work

• No fiction. A gap is only closed when code exists, a test or measurement proves it, and the nested CLAUDE.md says what the system does today. Where a gap needs something this machine does not have, it is marked [!] with the exact missing input and what was built so the measurement happens by itself later.
• Specification first. Every new protocol field, limit, refusal, reason and kind is declared in spec.py (nodes) or services/sync/spec.py and read from there everywhere else.
• Caps on everything new (rows, bytes, intervals, retries). Machines hold, humans decide: nothing here bans a member or deletes a ledger row on its own.
• One fan-out per feature: data layer, schema (*Out), route, view, Devii tool, docs_api, docs prose, privacy page where data categories change, README, nested CLAUDE.md, tests in the matching tier.
• Em dash is forbidden in every touched file. No comments or docstrings in Python source.
• The dev server on port 10500 runs --reload against the live database. Files are therefore written whole and import-checked right after each edit, and no command in this work reads or writes data/. All scripts use scratch databases (DEVPLACE_DATABASE_URL, DEVPLACE_DATA_DIR).
• Final gate: make test (unit, api, e2e), plus ruff/pyflakes on touched files, go vet + go test, node --test.

Gap inventory (source of truth for this plan)

Id	Gap (as written in the docs)	Source
G-N1	Reddit viability unmeasured from a residential line	nodes
G-N2	country, asn, asn_class never filled, datacenter exclusion (D6) not enforced	nodes
G-N3a	last_seen_at written on connect only	nodes
G-N3b	Socket counters per process	nodes
G-N3c	Fabric cache per process	nodes
G-N3d	Circuit breaker state in memory (and no per-destination admin view)	nodes
G-N3e	Same-process fast path not built, callers poll	nodes
G-N4	Fetch-mode results not cross-checked (sampled redundancy)	nodes
G-N5	DeepSearch skips hostile domains with a static list, no routing through /nodes/internal/fetch	nodes
G-N6a	Go client has no battery-aware pause (pause_on_battery ignored)	nodes
G-N6b	Go client treats a timed suspension as permanent until restarted	nodes
G-N6c	Extension enforces its own local hours in the browser clock, not the server timezone	nodes
G-N7	Not measured: Reddit/X success per address class, WebSocket idle timeouts through the public hostnames, socket capacity of one worker, 24 hour soak	nodes
G-N8	Launch blocked on a recorded legal decision	nodes
G-E1	Extension has no icons	extension
G-E2	welcome frame carries no device label	extension
G-E3	Extension not tested in a real Chrome or Firefox session	extension
G-S1	No export endpoint, sync is inbound only	sync
G-S2	Votes, reactions, follows, attachments, polls and project links not synced	sync
G-S3	A comment's post and parent must arrive earlier, no deferred resolution queue	sync
G-S4	A local deletion is not communicated to the origin; the origin is rejected forever with purged_locally	sync
G-S5	min_comments only sees comments collected in the same run	sync
G-S6	A clone renamed by an administrator no longer matches its alias	sync
G-S7	A username taken between alias generation and clone creation diverges from the rewritten mentions	sync
G-K1	Community page does not live-refresh during a run	community
G-K2	Run log and ledger are written only when the run finishes; a killed run leaves neither	community
G-K3	Text search across a whole community scans up to 40000 rows	community
G-K4	HTML comments follow rel=next links only	community
G-K5	No two-factor, captcha or JSON-API login	community
G-K6	HTML learning is heuristic plus AI; a hostile markup needs a hand-edited SCHEMA	community
G-K7	Votes, reactions and attachments are not collected	community
G-K8	Cross-feed duplicate detection compares sampled addresses at discovery time only	community
G-SK1	No test installs a skill from a live network origin	skills
G-X	Excluded by the owner: community passwords stored in plain text	community

Honest classification before any code

Closable by code and proven on this machine: G-N2, N3a-e, N4, N5, N6a-c, E1, E2, S1-S7, K1-K8 (K5 partly, see its item), SK1, and the capacity part of G-N7.

Closable only partly, with the remainder stated:

• G-N1 / G-N7 (success per address class). A residential donor is a physical machine this session does not have. What is built: every job records the donor's address class, the admin page and GET /admin/nodes aggregate success per destination and class, and the docs print n = 0 until real donors exist. The viability number itself stays unmeasured and the docs say so.
• G-N7 (24 hour soak). A soak can only be as long as this session. It is run in the background for as long as the work lasts and reported with its true duration.
• G-N7 (idle timeout through the public hostname). Measuring it requires an authenticated WebSocket on devplace.net, which is a write to the production database. It is measured against the local stack and the exact proxy settings are read from nginx/nginx.conf.template; the public leg is listed as needing the owner's explicit go-ahead.
• G-E3 (Firefox). Chromium is driven for real through Playwright. Firefox is attempted; if the available build cannot load a temporary add-on, the item stays open for Firefox only.
• G-K5 captcha. A captcha exists to stop automation. It is detected and reported loudly (captcha_required), never solved.

Not closable by code: G-N8 (a legal decision is a human act). It stays [!]; the plan records exactly what the decision must cover.

Protocol and specification changes (done first, shared by server and both clients)

All additive and optional, PROTOCOL_VERSION stays 1 (an old client ignores unknown fields on server frames; the server rejects unknown fields from clients, so only server-to-client additions are made).

Change	Where	Why
welcome.device_label (str, optional)	spec.py FRAMES, session.handshake, Go Frame, extension WsClient	G-E2
kick.until (str, optional ISO instant)	spec.py, registry.kick, clients	G-N6b
Ticket 403 body carries until for a timed suspension	routers/nodes/ticket.py	G-N6b
New refusal on_battery ("donor is running on battery")	spec.py REFUSALS, dispatcher.DONOR_LIMIT_CODES (pause 300 s), Go RefuseBattery, extension REFUSALS	G-N6a
LimitSpec.last_seen_write_seconds (60), redundancy_percent (5), redundancy_max_percent (20), geo_lookup_cache (4096)	spec.py	G-N3a, N4, N2

The extension spec test and the Go policy tests are updated in the same change so drift fails loudly.

---

Track N: node network, server (devplacepy/services/nodes/, database/nodes.py, routers/nodes/, routers/admin/nodes.py)

N-01 Geo and ASN database (G-N2)

• Root cause: record_device_connection is called with defaults, so country, asn, asn_class stay empty (routers/nodes/ws.py). The doc says the fix is an offline ASN database.
• Source: DB-IP "IP to ASN Lite" and "IP to Country Lite" monthly CSVs, free, no key, CC BY 4.0 (attribution printed on the docs page and in devplace nodes geo-status). Verified on 2026-10-16: https://download.db-ip.com/free/dbip-asn-lite-2026-10.csv.gz answers 200, 7.0 MB, 477362 ranges.
• Build services/nodes/geo.py: GeoDatabase backed by one SQLite file at config.NODE_GEO_DB (declared in DATA_PATHS, created by ensure_data_dirs). Tables geo_asn and geo_country: (family INTEGER, first BLOB, last BLOB, ...), primary key (family, first). Addresses are packed with ipaddress (4 or 16 bytes, compared bytewise by SQLite). Lookup is WHERE family=? AND first<=? ORDER BY first DESC LIMIT 1 then last>=ip. Memory stays flat, no 477k-entry Python list.
• Build services/nodes/addressclass.py: classify(asn, org) -> residential | mobile | datacenter | unknown. A static frozenset of well-known hosting and cloud ASNs (AWS, GCP, Azure, Hetzner, OVH, DigitalOcean, Linode, Vultr, Oracle, Alibaba, Cloudflare, Contabo, Scaleway, Leaseweb, M247, Choopa, Tencent, IBM, Rackspace) plus word-boundary org keywords (hosting, cloud, datacenter, data center, vps, server, colo, dedicated, cdn, vpn, proxy); mobile by keywords (mobile, wireless, cellular, lte, gsm); no data means unknown; everything else is residential. Heuristic by nature, so the docs call it that and the admin override (N-02) exists.
• Loader geo_sync() downloads through stealth_async_client (the project rule), verifies gzip, parses row by row with a hard cap on rows, loads into a temporary file and swaps it in atomically (verify before write). --dry-run reports counts only. CLI: devplace nodes geo-sync [--dry-run] [--month YYYY-MM], devplace nodes geo-status.
• Connect path: ws.py resolves the address transiently (the address is never stored, only hash_ip), calls GeoDatabase.lookup, passes country, asn, asn_class to record_device_connection, and puts asn_class on session. A missing or unreadable database yields unknown plus the counter geo_unavailable and one log line per process.
• Tests: unit for packing/lookup edge cases (first/last address of a range, gap between ranges, IPv6, v4-mapped), classifier table (including keyword false-positive guards such as Hostingly), loader against a local fixture file, atomic swap; api for devplace nodes geo-sync --dry-run is CLI-unit. Static check that no raw address column exists on node_devices.
• Docs: nodes CLAUDE.md, templates/docs/nodes.html (donor-facing: what is looked up and that the address is not stored), privacy.html (the lookup), attribution line.
• Status: [~] Result: geo.py + addressclass.py + CLI geo-sync/geo-status; real DB-IP month built in 10 s (477362 ASN, 710834 country ranges, 48 MB), lookup 0.17 ms, installed through the real CLI against a scratch data dir; 30 unit tests + 2 api tests (header-driven class, missing database counter). Docs pending.

N-02 Enforce the datacenter exclusion, with an admin override (G-N2, D6)

• Scheduler eligibility gains one rule: session.asn_class in {residential, mobile} or the device carries class_override = "allow". unknown is refused by default, because "residential and mobile only" is the decision and an unverifiable address must not be assumed good (loud skip, counter jobs_no_eligible_class). Site setting nodes_allow_unknown_class (default 0, apply live) lets an administrator run without a geo database during bring-up.
• New column node_devices.class_override (ensured in NODES_COLUMNS). Admin route POST /admin/nodes/devices/{uid}/class (allow or clear), require_admin, is_senior_admin against the device owner, audit key node.device.override, catalogued in audit/events.md and category_for. Devii node_admin_class_override (admin only, in CONFIRM_REQUIRED, declares confirm).
• Admin page: class column and an override button in the recent-devices table; AdminNodesOut carries both fields.
• Tests: unit scheduler (each class, override, unknown with and without the setting), api admin route (authz, senior-admin denial, audit row), Devii action gating.
• Status: [~] Result: scheduler rule (residential/mobile, unknown only with nodes_allow_unknown_class, per-device override), POST /admin/nodes/devices/{uid}/class, Devii node_admin_class_override (confirm-gated), audit node.device.override, admin page button; 5 scheduler + 6 api tests. Docs pending.

N-03 last_seen_at while connected (G-N3a)

• One batched UPDATE node_devices SET last_seen_at=:now WHERE uid IN (...) per last_seen_write_seconds from NodesService.reconcile, one statement for all sessions (no per-session query, the measured lesson in the doc), plus a final write in the disconnect path.
• Tests: unit with fake sessions (statement count is 1 for N sessions, interval respected), api that a connected fake device advances last_seen_at while idle.
• Status: [~] Result: NodesService.persist_state touches all connected devices in one batched UPDATE per last_seen_write_seconds; unit tests (interval, force, failure).

N-04 Durable counters (G-N3b)

• Table node_counters(name PRIMARY KEY, value) (derived, hard-GC only, not soft-deletable). NodeCounters keeps a delta per name; reconcile flushes all deltas in one INSERT ... ON CONFLICT(name) DO UPDATE SET value = value + excluded.value transaction and resets them only after commit. snapshot() stays the live process view; lifetime() reads the table. Admin page shows both.
• Tests: unit (flush adds, a failed flush keeps the delta, two processes flushing concurrently sum exactly: real separate processes, as the project procedure requires), api admin payload.
• Status: [~] Result: node_counters table, NodeCounters pending deltas flushed in one transaction every counter_flush_seconds, failed flush restores deltas; unit + api tests. Multi-process flush race still to run in validation.

N-05 Persisted circuit snapshots and a per-destination admin view (G-N3d)

• CircuitBreaker.snapshot() returns (key, failures, state, opened_at_wall). Table node_circuits is rewritten (upsert, delete when closed) from reconcile only when the breaker marked itself dirty. NodesService loads open circuits on start, converting wall time back to the monotonic clock, so a restart does not forget an open destination circuit.
• Admin page gets a "Circuits" table (destination or node, state, failures, seconds until half-open); AdminNodesOut.circuits capped at 100 rows.
• Tests: unit round trip across a simulated restart, expiry on load, cap, admin payload.
• Status: [~] Result: CircuitBreaker snapshot/restore + node_circuits + admin Circuit breakers table (read from the table so any worker answers); round-trip restart test.

N-06 Cross-process fabric cache (G-N3c)

• Table node_cache(key PRIMARY KEY, method, url, status, headers, body BLOB, stored_at, expires_at, size) as an L2 under the existing in-process ResultCache (L1). Reads: L1, then L2 (then promote). Writes: both. Bounded by LimitSpec.cache_max_bytes and cache_entry_max_bytes and the TTL; NodesService prunes expired rows and, when over the byte cap, the oldest rows, in run_once. cache.clear() clears both. Responses with status >= 500 or truncated are never stored (existing rule kept).
• Tests: unit (L1 miss L2 hit promotes, TTL, cap eviction order, oversize skipped), a real two-process test (worker A stores, worker B hits), kill-switch off does not serve from cache? (decision: cache is served regardless, as today, and the test pins that).
• Status: [~] Result: SharedResultCache (L1 + node_cache table), prune in service, cross-instance hit test; real two-process proof still to run in validation.

N-07 Same-process fast path (G-N3e)

• services/nodes/wake.py: WakeHub holding an asyncio.Event for the dispatcher and one per awaited job. JobBroker.enqueue signals the dispatcher when it runs in the lock-owner process; dispatcher._account/_failed/broker.complete resolve the job event; JobBroker.wait awaits the event first and keeps the DB poll only as the cross-process fallback and a safety net. No behavior change in other workers.
• Measure before and after on this machine with a fake session: enqueue-to-assigned latency (today up to dispatcher_tick_ms = 200 ms) and result-to-caller latency (today up to poll_interval_ms = 100 ms). The numbers go into the doc.
• Tests: unit (event wake, no lost wake-up when set before wait, timeout path unchanged), existing broker tests stay green.
• Status: [~] Result: WakeHub + JobBroker events; unit tests prove immediate wake. Before/after latency numbers still to measure.

N-08 Sampled redundancy for fetch-mode results (G-N4)

• Spec: redundancy_percent (default 5, ceiling redundancy_max_percent 20, site setting nodes_redundancy_percent clamped to it, 0 turns it off).
• After a fetch-transport result is accepted, a deterministic sample (sha256(job uid) % 100 < percent) enqueues a shadow job (caller_kind = "shadow", shadow_of = <uid>, exclude_device = <first donor>), scheduled to a different donor owner. The shadow never spends the caller's quota and never enters the cache.
• Comparison is on a shape, not bytes (live pages change): status class, JSON top-level key set or the HTML <title>/first heading, plus a body size ratio inside a band. A mismatch raises a node_flags row redundancy_mismatch against both devices and counts redundancy_mismatches; no auto-ban. Match and mismatch counts are persisted (N-04) and shown on the admin page so the false-positive rate is itself measured.
• Tests: unit shape extractor (JSON, HTML, binary), sampling determinism and rate over 10000 uids, scheduler exclusion, dispatcher comparison with two fake nodes (match, forged body, one node errors), cap.
• Status: [~] Result: shape.py + dispatcher shadow jobs (scheduler exclusion, flags, counters, clamped setting); 9 shape + 8 dispatcher tests.

N-09 DeepSearch routes node-servable hostile URLs through the fabric (G-N5)

• services/jobs/deepsearch/crawl.py: _resolve_candidate asks a new NodeCrawl helper first when the URL is allowlisted (SPEC.destination(host)) and fabric.can_serve(url) is true; the fetch uses shared_fabric().fetch(..., caller_kind="user", caller_id=<session owner>), so it spends the owner's quota, honours the kill switch and falls back to the snippet on any FabricError. The static HOSTILE_DOMAINS tuple stays only for hosts the allowlist cannot serve (X, YouTube, Instagram and so on); Reddit is removed from it and the owner uid is threaded from the DeepSearch worker through crawl().
• Same extraction path as fetch_page (shared function), so the text is parsed identically. Guests are not routed (no owner quota to charge).
• Tests: unit with a fake fabric (served, refused, unavailable, quota exceeded, guest), existing crawl tests stay green.
• Status: [~] Result: crawl.node_page via POST /nodes/internal/fetch with the owner key for allowlisted hostile hosts; 6 unit tests.

N-10 Success per address class, measured automatically (G-N1, G-N7)

• node_requests records device_asn_class at claim time. database.nodes.success_by_class(days) aggregates (adapter, asn_class, outcome) over the retained window. GET /admin/nodes (AdminNodesOut.success) and the admin page show it; Devii node_admin_status includes it.
• Docs print the live table. Until a residential donor exists the cell reads n = 0, and the Reddit-from-residential gap is documented as unmeasured with the exact query that will answer it.
• Tests: unit aggregation (including empty), api payload.
• Status: [~] Result: claim records device_asn_class; durable outcome counters (outcome:adapter:class:kind) + admin table with n = 0 state; unit + api tests.

N-11 Measurements (G-N7)

• Socket capacity of one worker: scratch server on a free port, up to max_connected_nodes (2000) synthetic devices and tickets created in a scratch database, asyncio WebSocket clients answering pings; record RSS, event-loop lag (a 10 ms ticker), reconcile tick duration and connect rate at 250, 500, 1000, 2000 sockets. Harness lives in the scratchpad (disposable), method and numbers go into the doc.
• Soak: same harness held open with jobs flowing through fake nodes; started in the background as early as possible and sampled every minute (RSS, loop lag, socket count, error counters). Reported with its real duration. Not claimed as 24 hours unless it ran 24 hours.
• Idle timeout: local stack measured (idle WebSocket for 5 minutes with no application frames, ping interval 20 s, observing whether anything closes it); nginx/nginx.conf.template read for proxy_read_timeout on the nodes location and quoted. The public hostname leg needs a production write and is left for the owner's go-ahead.
• Status: [~] Result: measured. Fast path p50 209 ms to 25 ms; capacity 2000 sockets in 341 MB; soak 81 minutes (NOT 24 h) clean; counter and cache races exact. Residential Reddit rate and public-hostname idle remain unmeasured (see Cannot be closed by code).

---

Track C: clients (nodes/client/ Go, nodes/extension/)

C-01 Go: battery-aware pause (G-N6a)

• ServerSettings gains PauseOnBattery. New files with build tags: battery_linux.go (parse /sys/class/power_supply/*: on battery when a Battery supply exists, no Mains/USB supply is online, and the battery status is Discharging), battery_darwin.go (pmset -g batt), battery_windows.go (GetSystemPowerStatus through syscall.NewLazyDLL, no cgo), battery_other.go (unknown). Limiter.Acquire refuses with on_battery when the server setting is on and the probe says battery; the probe result is cached for 15 seconds. The server pauses the donor for 300 s on that code (N-spec change). Unknown means "not on battery" and logs once at debug (a probe failure must not silence a donor forever).
• Tests: table tests for the Linux parser with fixture trees, limiter test with a fake probe, the refusal code against spec.py.
• Status: [x] Result: Go battery probes (linux/darwin/windows/other), limiter refuses on_battery; go test green (agent report, re-verified: go vet clean, go test ok).

C-02 Go and extension: a timed suspension is not permanent (G-N6b)

• Server: ticket 403 suspended and the kick frame carry until for a timed suspension (read from suspended_until). A suspension without an end date stays a plain hold.
• Go: suspended leaves FatalReasons. New outcome.holdUntil; Session.Run sleeps until until (clamped to at least 60 s and at most 24 h, plus jitter) and then retries the ticket. revoked and the other reasons stay fatal. Extension WsClient gets the same state (SUSPENDED with a countdown in the popup).
• Tests: Go session test with a fake clock (suspended with until, without, expired), extension unit test, api test that the ticket body carries until.
• Status: [x] Result: Go holdUntil + extension SUSPENDED state; server sends until in ticket 403 and kick; tests in Go, node and api tiers.

C-03 Extension: server-timezone hours (G-N6c)

• Limits.applyServerSettings also reads active_from, active_to, timezone; inActiveHours() requires both the donor's local window (browser clock, as configured in the popup) and the server window evaluated in the server-stored timezone with Intl.DateTimeFormat(..., {timeZone}). An invalid zone falls back to UTC, as scheduler._zone does.
• Tests: node tests with a stubbed clock across zones, windows that wrap midnight, an invalid zone. Parity test against the Go InWindow and the Python in_active_hours for the same 200 sampled instants.
• Status: [x] Result: extension server-timezone hours with Python-generated parity table shared by JS and Go tests; node --test 68/68 re-verified.

C-04 Extension icons (G-E1)

• Four PNGs (16, 32, 48, 128) generated reproducibly by a small script using Pillow (already a dependency), a node-network glyph on the project accent colour. manifest.json gets icons and action.default_icon; build.sh includes them; the packaging test checks the files exist and the sizes match.
• Status: [x] Result: four PNG icons generated by nodes/extension/tools/make_icons.py, manifest + build.sh updated, package test.

C-05 Device label in welcome (G-E2)

• Server sends device_label (from node_devices.label); extension State stores it and the popup shows the server label as the authority, falling back to the local one before the first connect. Go prints it in status/run logs.
• Tests: frame spec test, extension popup state test, Go decode test.
• Status: [x] Result: welcome.device_label server + Go + extension; api test asserts it.

C-06 Real browser test of the extension (G-E3)

• Playwright persistent Chromium context with --disable-extensions-except and --load-extension against a scratch DevPlace server on a free port (not 10500/10501): register a device, drive the disclosure and connect flow through the real popup page, assert the socket connects (server registry shows the node), a job for an allowlisted host is answered (via: fetch; a 403 from this datacenter address is an acceptable status, as in the Go run), refusals work, and disabling disconnects. Firefox: attempt with the installed Playwright Firefox; if a temporary add-on cannot be loaded, record that and leave E3 open for Firefox.
• The test is kept as tests/e2e/nodes/extension.py (skips with a stated reason when Chromium extensions cannot load, never silently passes).
• Status: [x] Result: Chromium e2e test tests/e2e/nodes/extension.py passes (~10 s); Firefox 158 verified by scratch geckodriver script; CSP fix found and applied by that test; method in nodes/extension/README.md.

C-07 Client build and verification

• go vet ./..., go test ./..., node --test, nodes/client/build.sh into a scratch directory (not data/). The distributable build into data/node_clients/ is a deploy step left to the owner.
• Status: [x] Result: go vet, go test, node --test 68/68, cross-compile linux/windows/darwin amd64+arm64 and freebsd (agent report; I re-ran vet, test and node tests).

---

Track S: community sync (devplacepy/services/sync/, database/sync.py, routers/sync/)

S-00 Design decisions for new kinds (G-S2)

• Kinds, in dependency order: users, projects, posts, comments, polls, votes, reactions, follows, poll_votes, attachments. KINDS is the single list; SyncPusher, BatchRunner, docs and Devii tools read it.
• Counters stay un-importable as numbers. A synced vote exists only with a synced voter, and the target stars is recomputed from the local votes rows by the same function apply_vote uses (update_target_stars + clear_user_stars), once per touched target per batch. This replaces "synced posts have stars = 0" with "stars are derived from synced votes only"; the doc rule is rewritten, not silently broken.
• No side effects, as before: no notifications, XP, badges, mention fan-out, AI correction/modifier. Content filter runs on every authored text (projects, polls).
• Records:
  • ProjectRecord: author_uid, title, description, status, platforms, website_url, repo_url, project_type; always public and writable (is_private = 0, read_only = 0); no files, no containers.
  • PostRecord.project_uid (optional reference to a synced project) closes "project links".
  • PollRecord: post_uid, question, options: [{origin_uid, label}] (2 to 6, same caps as create_poll); options are rows of the poll, fingerprinted together.
  • VoteRecord: voter_uid, target_kind in post|comment|project, target_uid, value in {1, -1}; a cleared vote is deleted: true.
  • ReactionRecord: author_uid, target_kind in post|comment|project, target_uid, emoji (validated against the existing reaction emoji rule); tombstone removes.
  • FollowRecord: follower_uid, following_uid; no self follow; tombstone removes.
  • PollVoteRecord: voter_uid, poll_uid, option_uid; tombstone removes.
  • AttachmentRecord: author_uid, target_kind in post|comment|project, target_uid, filename, mime_type, content_b64. No network fetch inside a processor (it runs on the request path and SQLite/HTTP stay out of each other's way): the sender downloads and sends the bytes; the processor decodes, applies the existing upload validators (type, size, image dimensions), and writes through attachments.store_attachment + link_attachments. max_attachment_bytes is a SyncLimits field and the envelope gets a total byte cap (max_batch_bytes).
• Voter-visibility: poll_votes are private ballots in this product (UNREPORTABLE_TABLES says so). They are accepted inbound (the sender owns that decision) and never exported (S-02).
• Each new local row is soft-deletable and created with deleted_at: None, deleted_by: None; tombstones soft-delete with deleted_by = "sync"; toggles revive instead of duplicating (the project's hard rules).
• Reportability: projects and polls are already reportable targets; synced rows reuse them. votes, reactions, follows, poll_votes, attachments are not new tables, so REPORTABLE_TARGETS/UNREPORTABLE_TABLES stay exact (the registry test is the check).
• Status: [~] Result: decisions implemented; correction recorded: votes, reactions and poll ballots are accepted inbound but NOT exported because no UI exposes who voted.

S-01 Implement the new kinds (G-S2)

• spec.py (kinds, limits), records.py (models + RECORD_MODELS), normalize.py (emoji, options, platform lists), processors.py (one processor per kind on RecordProcessor, PROCESSORS), database/sync.py (link helpers for composite natural keys: a vote/reaction/follow link is keyed by the sender's own origin_uid; a second record for the same (voter, target) with a different origin_uid revives the same local row instead of duplicating, the concurrency lesson of the toggles), routers/sync/ one leaf per kind (projects.py, polls.py, votes.py, reactions.py, follows.py, poll_votes.py, attachments.py), GET /sync/spec.json gains the schemas, Devii sync_<kind> tools (requires_admin, form-encoded records), docs_api endpoints, README, docs/privacy.html row update, sync CLAUDE.md rewritten for the new rules.
• Tests: unit per normaliser and processor (create, update, stale, tombstone, locally deleted, purged, wrong references), api per route (authz, gate, dry run, cap), stateful fuzz extended to all kinds with invariants (no dangling links, no orphan votes/reactions/follows, stars equals net local votes for every synced target, one vote per voter and target, no self follow), and a real multi-process race on the vote/follow toggle rows started from a fresh unseeded database (40 racers, exactly one row).
• Status: [~] Result: 10 kinds, records, processors, 7 new routes, Devii tools, docs_api; 69 api tests incl. new kinds. Fuzz + multi-process race and docs/privacy pending.

S-02 Export endpoint (G-S1)

• GET /sync/export/{kind}?since=<iso>&cursor=<opaque>&limit=<=50> (administrator only, behind sync_enabled, audited as sync.export). Keyset pagination on (updated_at, uid); the cursor is a signed-free, validated base64 of those two values. Output items are exactly the *Record models, so a DevPlace instance can feed another with the sender it already has. Soft-deleted rows are emitted as tombstones.
• Exported: users (public fields only: never email, hash, role, api key), projects (public only), posts, comments, polls, votes and reactions (author uid and target only), follows, attachments (as url, not bytes). Not exported: poll_votes (private ballots), anything private or deleted-by-account-deletion beyond the tombstone, and rows that arrived by sync themselves (sync_links.local_uid), so two instances cannot echo each other's data in a loop.
• Tests: api per kind (shape equals the inbound model, no private field, pagination is gapless and repeat-free over 120 rows, tombstones, the echo guard), a round trip test (export from one scratch database, import into another through the real routes, compare counts).
• Status: [~] Result: GET /sync/export/{kind} with expression indexes (EXPLAIN-tested), echo guard, keyset pagination; 27 api tests with deletions. Cross-database round trip pending.

S-03 Deferred resolution queue (G-S3)

• Statuses gain deferred (an applied-later state, distinct from rejected). Table sync_pending (origin, kind, origin_uid, missing_kind, missing_uid, record JSON, attempts, created_at, expires_at) with a unique index on (origin, kind, origin_uid) so a resend replaces instead of stacking.
• A record rejected for unknown_author, unknown_post, unknown_parent, unknown_target, unknown_poll, unknown_option, unknown_project is queued (not on dry run) and reported deferred with the same reason. When any record of kind K with origin uid U is created, BatchRunner looks up pending rows waiting on (K, U) and re-applies them with the same processors, iteratively (a released comment can release its replies) under pending_release_cap_per_batch. Caps: pending_max_per_origin 5000 (the 5001st is rejected with the old reason, loudly counted), pending_ttl_days 7, pending_max_attempts 5; expired rows are garbage collected hard by the service already responsible for sync housekeeping (Community sync service run_once).
• The community ledger gains the outcome deferred (warn tone) with a plain-language reason; the pusher's projection logic for dry runs is unchanged.
• Tests: unit (queue, release in any order, chain of three, cap, ttl, replace-on-resend, dry run writes nothing), the 16-process race (post and its 15 comments arriving in random order from separate processes: exactly the right final counts), ledger reason text test.
• Status: [~] Result: sync_pending + PendingQueue; 12 api tests (chain, shuffle of 18 records, cap, ttl, dry run). 16-process race pending.

S-04 Deletion feedback to the origin (G-S4)

• GET /sync/deletions?origin=<id>&since=<iso>&cursor= (administrator only) lists synced rows deleted or purged on this side: {kind, origin_uid, deleted_at, deleted_by_class: "owner|moderator|system|purged"}; POST /sync/deletions/ack (origin, up to 50 {kind, origin_uid}) stamps sync_links.ack_at. A record that the origin re-sends after acknowledging answers unchanged / acknowledged_deleted instead of the loud rejected / purged_locally, so the noise stops once the origin has been told. New columns sync_links.deleted_seen_at, ack_at (ensured).
• The scheduled community sender skips records whose link is locally deleted (cheap lookup) and records the skip in the ledger, so a re-collection of a deleted post stops being pushed every run.
• Devii tools sync_deletions, sync_deletions_ack; docs; audit sync.deletions.ack.
• Tests: api (listing by class, pagination, ack, reappearance answers acknowledged_deleted, authz), unit.
• Status: [~] Result: GET /sync/deletions and POST /sync/deletions/ack, acknowledged_deleted quiet answer; tests in deletions.py.

S-05 min_comments counts what is already synced (G-S5)

• Selector counts, per post, comments collected in this run plus the live local comment count of the post's existing link (one batched GROUP BY target_uid query for all linked posts in the run, not per post). A post that was kept once is never dropped (existing rule), so this only changes first-time decisions.
• Tests: unit (collected 0 + 3 already synced passes a threshold of 3; none anywhere is dropped; the batched query is one statement).
• Status: [~] Result: Selector candidates (sync_candidates) park and revive a post with its comments and authors across runs; 10 unit tests.

S-06 and S-07 Alias heals when the clone name changes or collides (G-S6, G-S7)

• Aliaser.reconcile(origin) runs at the start of every aliased run and after the push: for every identity whose clone user exists, compare users.username with our_name. On a difference: update the identity (our_name, our_key, skipping and logging when the new key collides with another identity), then clear the fingerprint of the origin's posts and comments whose stored text contains the old alias (bounded to 5000 links per run, oldest first), so the next push rewrites them with the live name. The rewriter always resolves names through the live clone username first.
• This covers both gaps: an administrator rename, and the case where free_username gave the clone a suffix because a registration took the name between alias generation and clone creation (the post-push reconcile sees the suffix and heals the mentions on the next run). The race window itself is also narrowed: the clone user is created before dependent content is rewritten.
• Tests: unit (rename, suffix collision, key collision skip, bounded sweep, nothing to do), an integration run through CommunityRunner with a fake collector: rename between runs, mentions follow.
• Status: [~] Result: AliasHealer rewrites stored mentions and follows the clone name; wired before and after the push in CommunityRunner; 6 unit tests.

---

Track K: community discovery (devplacepy/services/community/, routers/admin/communities.py, static/js/)

Note: runtime.py and extract.py are inlined into generated collectors by generator.py. Every change there is covered by a generator test that executes the generated file, and existing communities regenerate their files (the database row holds the schema; files are regenerated when missing).

K-01 Live refresh while a run is in progress (G-K1)

• /admin/communities/{slug} and the ledger page swap their [data-live] sections on an interval while the community is busy or a run row is running (the AdminNodes.js pattern, shared Poller, stops when idle). New static/js/AdminCommunity.js (one class, app.adminCommunity), no inline script; responsive at phone width.
• Tests: e2e (start a slow fake run, the progress counters change without reload, polling stops at the end), api (the page carries the live hooks and the run row state).
• Status: [x] Result: AdminCommunity.js live refresh on data-busy/data-live sections; busy field on AdminCommunityLedgerOut; tests/e2e/admin/communities/live.py.

K-02 Incremental run log and ledger, interrupted runs are visible (G-K2)

• CommunityRunner.run registers a checkpoint callback: every checkpoint_seconds (15) and after every pushed batch it writes the current LogBook text, tally and verdict running into the run row, and flushes new ledger drafts with LedgerStore.append (continuing seq, respecting the same caps and rank rules). The final finish_run replaces the totals; ledger rows are never written twice (an index on (run_uid, seq) makes a duplicate a no-op).
• At service start, CommunityStore.reap_interrupted() marks any run left running for longer than run_timeout_seconds as interrupted (verdict bad, headline "interrupted: the process ended before the run finished") and keeps whatever was checkpointed. Retention (prune_runs) is unchanged.
• Tests: unit (a run killed after the second batch leaves log, tally and ledger; resume never duplicates), a real process kill test (SIGKILL a subprocess mid-run, then reap), caps still hold.
• Status: [x] Result: RunCheckpoint every 15 s, unique (run_uid, seq), reap_interrupted marks RUN_INTERRUPTED; unit tests in checkpoint.py.

K-03 Indexed text search (G-K3)

• SQLite FTS5 external-content table community_run_items_fts over label, origin_uid, detail with tokenize='trigram' (substring semantics like the current LIKE, query minimum 3 characters), kept in step by triggers on community_run_items. Queries of 3+ characters use MATCH joined back to the scoped rows; shorter queries keep the bounded LIKE on the scoped index. If FTS5 or the trigram tokenizer is unavailable, init_db logs it once and the old path is used (the same results, the old cost). Prune deletes cascade through the triggers.
• Measure on the same scratch ceiling used before (10 runs of 4000 rows plus the community scan) and update the numbers: the no-match search was 37 ms, the old unbounded case 97 ms. EXPLAIN QUERY PLAN test extended so the FTS path is asserted, not assumed.
• Tests: unit (substring parity with LIKE over a generated corpus including %, _, quotes, unicode), query-plan test, trigger consistency after insert, delete and prune.
• Status: [x] Result: FTS5 trigram table with triggers for community-wide search of 3+ chars; measured 34 ms to 0.4 ms (no match), 34.5 to 0.8 ms (rare), common term 9.4 to 21 ms (slower, accepted).

K-04 Comment pagination beyond rel=next (G-K4)

• comments.pagination becomes a learned and overridable spec: next_link (selector list, default adds text-matched anchors in a pagination container: next, older, more, volgende, ›, », →), page_param (?page=N, ?p=N, /page/N, detected from the current URL or the numbered page links), json_next (a JSON next/next_page/links.next key). The collector stops on a page that adds no new comment ids or repeats a URL, always under max_comment_pages.
• The learner (PageProbe) detects the style from the sampled post pages and stores it; NEXT_PAGE_FIELDS remains the default.
• Tests: extraction fixtures for each style (rel next, text next, numbered page links, ?page=, /page/N, JSON next), loop and duplicate guards, generated-collector parity.
• Status: [x] Result: next_link, page_param, json_next, none hints learned and enforced; stops on a page with no new comments; unit tests pagination.py.

K-05 Login: JSON-API and two-factor, captcha detected (G-K5, passwords excluded)

• JSON-API login: schema.login.kind = "json" with url, fields (username/email/password key names), token_path (JSON path to a bearer token, optional); success is a 2xx plus either a token or a session cookie. LoginLearner probes a short fixed list of common endpoints (/api/login, /api/auth/login, /api/v1/auth/login, /auth/login, /login.json, /session) with the stored credentials, never anything outside the community origin, guarded by guard_public_url.
• Two-factor (TOTP): a new column communities.login_totp_secret, handled exactly like the other credential columns (never returned by any endpoint, admin form, Devii tool or CLI; set through devplace community login --login-totp). extract.totp(secret, now) implements RFC 6238 with the stdlib. The login flow detects a follow-up one-time-code step (otp, totp, code, token, 2fa input) and submits the current code. Tested against the RFC 6238 vectors. SMS and push 2FA cannot be automated and are reported as second_factor_unsupported.
• Captcha: detection of g-recaptcha, h-captcha, cf-turnstile, captcha form markers moves the community to login_required with the reason captcha_required and a plain message; it is never attempted. Recorded as unsupportable by design.
• Tests: fixtures for a JSON login (token and cookie), a TOTP step with the RFC vectors and a fake clock, a rejected code, the captcha stop, credential non-disclosure (every serializer and the CLI output are scanned for the secret).
• Status: [x] Result: TOTP (RFC 6238 vectors), JSON-API login, OTP form detection, captcha detected and reported (never solved); secret never returned; models, router, CLI, Devii, docs_api, templates plumbed; tests login.py, admin index test.

K-06 Structured-data learning for hostile markup (G-K6)

• New inferrer, tried between the heuristics and the AI assistant: StructuredDataInferrer reads JSON-LD (DiscussionForumPosting, Article, BlogPosting, Comment, ItemList) and schema.org microdata (itemtype/itemprop) and OpenGraph, producing a feed type json_ld (runtime support: extract the list of <script type="application/ld+json"> blobs, flatten @graph, then reuse the existing JSON path engine). Known software signatures (Discourse, Lemmy, WordPress wp-json, Mastodon, phpBB/XenForo markup) select a ready, fixture-tested spec instead of guessing.
• Each learned field is validated against the sample like the AI proposals (60 % rule). The failure message names which fields could not be learned, so the human edit (K-07) is targeted.
• Tests: fixtures per source type (JSON-LD, microdata, OpenGraph-only, a Discourse .json topic list, a WordPress wp-json posts feed), generated-collector parity, no regression on the existing lobste.rs fixtures.
• Status: [x] Result: json_ld feed type, detector, learner, runtime; unit tests structured.py. Discourse and WordPress ready-made specs not built (schema editor covers hostile markup).

K-07 Admin schema editor (G-K6)

• GET/POST /admin/communities/{slug}/schema (administrator, audited community.schema.update): shows the learned SCHEMA JSON; a save validates structure with the same validator the learner uses, runs a verification collection on a small sample through the real collector (dry run, nothing pushed), and only then stores the schema and regenerates collectors.py (verify before write; the previous schema is kept as schema_previous for one-click restore). Devii community_schema_get / community_schema_set (set is in CONFIRM_REQUIRED).
• Tests: api (valid, invalid structure, failing verification keeps the old schema, restore, authz, audit), e2e (edit and save through the page).
• Status: [x] Result: /admin/communities/{slug}/schema GET/POST/restore with validate_schema and sample verification, schema_previous, audit keys, Devii tools, docs_api; tests schemacheck.py and api schema.py.

K-08 Collect attachments, votes and reactions where they can be attributed (G-K7)

• Attachments: option collect_attachments in sync_options (default off, per community). The collector finds inline images and attachment links in post and comment bodies, downloads them through Fetcher (stealthii + SSRF guard) under per-file and per-run caps, and emits attachments records with content_b64. Failures are ledger warnings, never run failures.
• Votes and reactions: the learned spec may name optional votes / reactions list fields (per-voter entries with a user id or name and a direction/emoji). When the source exposes them, voters become synced users and the records flow through S-01. When the source only publishes an aggregate count (almost every site), nothing is imported: an anonymous total cannot be turned into votes without inventing voters, which the platform forbids by design (no purchasable or fabricated engagement). This is stated in the docs as intentional, not as a missing feature.
• Tests: fixtures (attachment download within caps, oversize skip, blocked address, per-voter JSON feed becomes votes with derived stars, aggregate-only feed produces zero vote records), ledger rows.
• Status: [x] Result: collector emits votes, reactions and attachments; alias mode drops voter-attributed rows; generator VERSION 2 auto-regenerates; unit tests engagement.py.

K-09 Cross-run duplicate detection (G-K8)

• The collector state (the watermark JSON already persisted per community) gains links: a bounded map canonical_link -> post origin_uid (cap 20000, oldest dropped, trimmed on write). add_post consults it, so a twin story seen in a different feed in a later run is aliased to the first post instead of becoming a second post; comments attach to the survivor as they do within a run.
• Tests: two feeds, two runs, same canonical link: one post; map cap and trimming; same-feed posts with the same link are still never merged (existing safe rule); generated-collector parity.
• Status: [x] Result: state links capped at max_remembered_links (20000); unit tests dedupe.py.

---

Track SK: skills (devplacepy/services/skills/)

SK-01 Real network leg for the skill fetcher (G-SK1)

• The SSRF guard refuses loopback, which is why the network leg was never exercised. This machine has a global IPv4 address, and a server bound to it passes the guard unmodified, so the test runs the real guarded_async_client over a real socket with no patching: ThreadingHTTPServer on 0.0.0.0:<free port> serving a SKILL.md plus a bundled references/ file, fetched through fetch_skill at http://<global ip>:<port>/SKILL.md. The test discovers the address (UDP connect trick, then is_blocked_address), and skips with a stated reason when the host has no global address, never passing silently. Also asserted over the same socket: a loopback URL is refused, a redirect to a loopback address is refused, an oversize body is cut at the limit, an HTML page is rejected.
• Status: [x] Result: real-socket network test in tests/unit/services/skills/fetcher.py through the host's global IPv4, unpatched guard; 36 pass; skips with a reason when no global IPv4.

---

Cross-cutting work

X-01 Documentation of record

Rewrite the Gaps sections to state only what is still true, move closed gaps into "What exists today" with the measured numbers, update: services/nodes/CLAUDE.md, services/sync/CLAUDE.md, services/community/CLAUDE.md, services/skills/CLAUDE.md, nodes/extension/README.md, nodes/client/README.md, README.md, plan.md (node plan progress), the docs pages (nodes.html, nodes-protocol.html reads the spec, privacy.html, community and sync pages), docs_api groups, audit/events.md. The memory file node-network-plan.md is updated at the end.

• Status: [ ] Result:

X-02 Validation gates (all must be clean)

1. python -c "from devplacepy.main import app" after every Python edit.
2. ruff check / pyflakes on every touched Python file.
3. go vet ./... && go test ./... in nodes/client; node --test in nodes/extension.
4. grep for the em dash character and its HTML entity in every touched file.
5. CSS braces and HTML tags balanced for touched templates; JS parses.
6. Fan-out check per route (HTML, JSON, Devii, docs).
7. Four-layer correctness procedure for the spendable/state-machine/concurrent items (votes and stars, the pending queue, counters flush, cache, redundancy sampling): property checks, stateful fuzz, real multi-process races from fresh rows, static analysis.

• Status: [ ] Result:

X-03 Full suite

• make test (unit, api, e2e, serial). Every failure is fixed, not skipped.
• Status: [ ] Result:

X-04 Commit

• Commits per track with the pre-commit hook bumping the patch version. No push (outward-facing, not requested). The owner is told about the dist build into data/node_clients/ and the optional public-hostname idle test.
• Status: [ ] Result:

---

Cannot be closed by code (kept visible on purpose)

Gap	Why	What is built instead	What closes it
G-N8 legal decision	A platform-terms, donor-terms, ISP-terms and jurisdiction decision is a human act.	Nothing is enabled for any account; nodes_enabled is 0; the kill switch is enforced within one tick.	The owner records the decision; then switches the service on. The decision must cover: Reddit and any future destination terms, residential donor terms of service, ISP terms on traffic relaying, the jurisdiction of donors and of the operator, and the adult-only rule.
G-N1 Reddit from a residential line	Needs a residential machine.	Per-class success is recorded and shown (N-10).	A residential donor runs the Go client or the extension for a while.
G-N7 24 h soak, public idle timeout	Time and a production write.	Local soak with its true duration, local idle measurement, proxy config quoted.	Run the same harness for 24 h; owner approves a one-off production connection.
G-K5 captcha	Defeating a captcha is circumventing the control it exists for.	Detected, reported as captcha_required.	A human logs in once, or the site offers an API.
G-E3 Firefox (if the build cannot load add-ons)	Depends on the available Firefox build.	Chromium proven for real.	A normal Firefox with web-ext run.

Progress log

(One line per finished item: date, item, evidence.)

Result summary

(Filled in at the end: what is closed, what is not, measured numbers, test counts, commits.)