Code Farm: per-plot auto-replant pause and crop-lock check; community sync and node work
DevPlace CI / test (push) Failing after 44m45s

Continuous Delivery could never be turned off once a crop was planted. Add a
one-shot per-plot "stop after harvest" flag (POST /game/plot/stop, Devii tool
game_plot_stop, game_plots.stop_after_harvest), cleared by the next plant,
autoreplant claim or Refactor. Autoreplant now also respects crop locks through
the shared _crop_lock predicate. Plan and measurements in happyplanting.md.

Also includes the pending community sync, node network and related work in the
tree; the full suite passed (4999) with all of it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-17 02:23:43 +02:00
co-authored by Claude Sonnet 5.5
parent ce899fc5e1
commit 02ab747cd0
203 changed files with 11624 additions and 301 deletions
+20 -9
View File
@@ -121,20 +121,31 @@ For a `tunnel` or `any` job the scheduler prefers a node whose `hello.modes` inc
`SocialSearch.search` fans one query out to adapters in parallel (`asyncio.gather`, 20 s each) and merges hits round-robin. Direct adapters (`adapters/direct.py`: Hacker News, Stack Overflow, GitHub issues, dev.to by tag, Mastodon by hashtag) need no donors. `adapters/reddit.py` is node-backed and only used when `nodes_social_via_nodes` is `1`; otherwise it is reported as `nodes_not_used`. Every adapter parses a response structurally and raises `AdapterError` on a forged or wrong shape; platforms that fail are listed under `unavailable` with a code and never filled with invented results. **Measured on this machine (2026-10-15, datacenter address):** Hacker News Algolia, Stack Exchange, GitHub search, dev.to tag API and Mastodon hashtag timelines answer; Bluesky `searchPosts` returns 403 without auth, Mastodon `/api/v2/search` returns nothing without login, Lobsters search sits behind a bot wall, dev.to has no usable text search, and Reddit returns 403 to this address. Those are why Bluesky and Lobsters are not adapters. Surfaces: `GET /nodes/social/search` (daily per-member quota `nodes_daily_social_user`, default 300), Devii `social_search`, the DeepSearch `deepsearch_social` option (off by default, signed-in owners only, goes through the HTTP route with the owner's key).
## Server mechanisms added after the first build
- **Address class (D6 enforced).** `geo.py` keeps DB-IP lite ASN and country data in a SQLite range file under `config.NODE_GEO_DIR` (`devplace nodes geo-sync [--dry-run] [--month]`, `geo-status`; CC BY 4.0, attribution on `/docs/privacy.html`). Measured: build about 10 s, 477362 ASN and 710834 country rows, 48 MB, lookup 0.17 ms. `addressclass.classify` maps an ASN to `residential`, `mobile`, `datacenter`, `business` or `unknown`. The scheduler only assigns jobs to `ELIGIBLE_CLASSES` (residential, mobile) unless a device carries `class_override == "allow"` (admin, `POST /admin/nodes/devices/{uid}/class`, audit `node.device.override`, Devii `node_admin_class_override` confirm gated) or `nodes_allow_unknown_class=1` admits unknown. Refusals are counted in `NodeScheduler.class_refusals`. A missing geo file is `counters.geo_unavailable` plus one warning, never a silent pass.
- **Durable state.** `last_seen_at` writes are batched (`touch_devices`, `last_seen_write_seconds`). Counters flush as deltas into `node_counters` every `counter_flush_seconds` (lifetime totals survive restart). Circuit breaker state is saved to `node_circuits` (`snapshot`/`restore`, capped at `circuit_snapshot_max`). The fabric cache is `SharedResultCache`: an in-process L1 over the `node_cache` table, so a result fetched by one worker serves the others.
- **Fast path.** `wake.py` `WakeHub` wakes the dispatch loop and a waiting caller in the same process the moment a job is queued or done; the DB poll remains the cross-process fallback. Measured end to end (one donor, 300 jobs): polling p50 209 ms, p95 214 ms; fast path p50 25 ms, p95 about 30 ms, max about 38 ms. A single 6.2 s cold-start outlier appeared once and did not reproduce.
- **Sampled redundancy.** `redundancy_percent` (5, ceiling `redundancy_max_percent` 20) of fetch-mode jobs are shadowed to a second donor (`shadow_of`, `exclude_device`); `shape.py` compares structure, and a mismatch raises the flag `redundancy_mismatch` for a human. Counters: `redundancy_checked`, `redundancy_matched`, `redundancy_mismatches`, `redundancy_inconclusive`.
- **Outcome counters per address class.** `outcome:<adapter>:<class>:<ok|blocked|failed>` in `node_counters`, shown on `/admin/nodes`. This is the instrument for the residential success rate; it reads `n = 0` until real donors exist.
- **DeepSearch** routes hostile URLs through `POST /nodes/internal/fetch` with the owner's api key (`crawl.node_servable`, `node_page`) when `nodes_social_via_nodes` is on.
- **Battery.** `on_battery` is a donor limit code (pause 300 s, retryable). `welcome.device_label` and `kick.until` are optional protocol fields; a suspended ticket 403 carries `until`.
**Measured capacity (one worker, scratch server, 2000 simulated donors on this machine):** 2000 sockets connected in 8 s; RSS 156 MB at 250, 190 MB at 500, 241 MB at 1000, 341 MB at 2000; idle dispatch tick p50 67 ms (max 160 ms); event loop lag p99 78 ms idle; 14.2 jobs/s completed at 15 offered, 0 client failures. A soak at 2000 sockets, 10 jobs/s and 3 percent a minute churn ran 4844 s (about 81 minutes, not 24 hours): RSS 346 to 361 MB, 42909 of 42913 jobs done, 1 failed, 0 integrity failures, 2026 of 2026 redundancy checks matched, loop lag p99 73 to 106 ms. Races, with separate OS processes: 16 workers x 20 counter flushes summed exactly (1600 of 1600); 12 racing cache puts left 1 row and all 12 readers hit.
## Clients
`nodes/client/` is the Go donor client (standard library only, `build.sh` cross-compiles five targets and writes `manifest.json`). `nodes/extension/` is the MV3 extension (Chrome and Firefox, fetch mode only, 49 `node --test` tests, one of which reads `spec.py` and fails on drift). `make node-client` builds both into `data/node_clients/`, which `GET /nodes/client-version` and `GET /nodes/download/{artifact}` serve (names must match a strict pattern, symlinks and the manifest are never served, the SHA-256 is computed from the file). **Verified end to end on 2026-10-15** against a scratch server on this machine: the Go client logged in, connected, answered a `tunnel` job for `reddit.com` (`via: tunnel`, status 403 because the donor was this same datacenter address) and stayed connected across repeated jobs. Not verified: a residential donor, the extension in a real browser, Windows and macOS builds running.
`nodes/client/` is the Go donor client (standard library only, `build.sh` cross-compiles five targets and writes `manifest.json`). It reads battery state on Linux, macOS and Windows (`RefuseBattery`, `PauseOnBattery`) and holds a timed suspension until `until` (clamped 60 s to 24 h). `nodes/extension/` is the MV3 extension (Chrome and Firefox, fetch mode only; icons 16/32/48/128; a suspended state with `suspendedUntil`; active hours in the server timezone; the manifest sets `content_security_policy.extension_pages` explicitly because Firefox's default MV3 policy adds `upgrade-insecure-requests`, which rewrote `ws://127.0.0.1` to `wss://` and broke loopback connections). `make node-client` builds both into `data/node_clients/`, served by `GET /nodes/client-version` and `GET /nodes/download/{artifact}` (strict names, no symlinks, SHA-256 from the file). **Verified in real browsers:** Chromium via `tests/e2e/nodes/extension.py` (disclosure, connect, online, a fetch-mode reddit job answered with 403 via `fetch`, suspend, disable) and Firefox 158 with geckodriver (connected, job done with 403 and 190240 bytes, suspended with `until`, disable). The Firefox script is not in the repository; the method is in `nodes/extension/README.md`. Go: `go vet` and `go test` pass, cross-compiles clean.
## Gaps (what is not built or not measured)
- **Reddit viability is unmeasured from a residential line.** The only real run used a datacenter donor and got 403, which proves the path, not the success rate. X, Instagram, LinkedIn, Facebook and TikTok stay unreachable (login walls); nothing here claims them.
- `country`, `asn` and `asn_class` on a device are never filled, so the datacenter exclusion (D6) is not enforced: the scheduler treats every node as eligible. It needs an offline ASN database file.
- `last_seen_at` is written on connect only. Socket counters are per process. The fabric cache is per process. Circuit breaker state is in memory. The same-process fast path is not built, callers poll at `poll_interval_ms`.
- Fetch-mode results are not cross-checked (sampled redundancy between two donors is not built). They are structurally checked and labelled `via: fetch`, and `tunnel` is preferred whenever a node can do it.
- `DeepSearch` still skips hostile domains with a static list; Reddit URLs from social search are used through their snippet. Routing a hostile URL fetch through `/nodes/internal/fetch` is not built.
- The Go client has no battery-aware pause (`pause_on_battery` is stored and sent but ignored by it), and treats a timed suspension as permanent until restarted. The extension enforces its own local hours in the browser clock, not the server timezone.
- Not measured: Reddit and X success rates per address class, WebSocket idle timeouts through `pravda.education` and `devplace.net`, socket capacity of one worker, a 24 hour soak. The caps in `LimitSpec` are starting ceilings, not measurements.
- **Launch is blocked on a recorded legal decision** (platform terms, donor terms, ISP terms, jurisdiction). Nothing here is enabled for any account: `nodes_enabled` is `0` until an administrator switches it on.
- **Reddit success from a residential line is unmeasured.** The counters exist and read `n = 0`; every real run so far used a datacenter donor and got 403. X, Instagram, LinkedIn, Facebook and TikTok stay unreachable (login walls); nothing here claims them.
- **No 24 hour soak.** The longest run was about 81 minutes. The caps in `LimitSpec` are ceilings backed by that run, not by a day.
- **WebSocket idle timeout through the public hostname is unmeasured.** It needs a production connection, which needs the owner's go-ahead.
- The extension has no battery pause (browsers expose no battery state to extensions); the Go client has one.
- The geo file needs a monthly `geo-sync`; between syncs a new ASN is `unknown`.
- Windows and macOS client builds compile but were not run.
- **Launch is blocked on a recorded legal decision** (platform terms, donor terms, ISP terms, jurisdiction). Nothing here is enabled for any account: `nodes_enabled` is `0` until an administrator switches it on. This is a human decision and no code closes it.
## Rules for work in this directory