Code Farm: per-plot auto-replant pause and crop-lock check; community sync and node work

Continuous Delivery could never be turned off once a crop was planted. Add a
one-shot per-plot "stop after harvest" flag (POST /game/plot/stop, Devii tool
game_plot_stop, game_plots.stop_after_harvest), cleared by the next plant,
autoreplant claim or Refactor. Autoreplant now also respects crop locks through
the shared _crop_lock predicate. Plan and measurements in happyplanting.md.

Also includes the pending community sync, node network and related work in the
tree; the full suite passed (4999) with all of it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

README.md

@@ -66,7 +66,7 @@ devplacepy/
 | `/topics` | Crawlable per-topic category pages (public): `/topics` hub links every topic with a live post count, `/topics/{topic}` lists that topic's posts with its own canonical URL, title, and breadcrumbs |
 | `/skills` | Per-user Agent Skills: list, detail, install from a URL, enable/disable, delete. Private to the owner, `noindex` |
 | `/nodes` | Community node network: `GET /nodes/spec.json`, `GET/POST /nodes/devices` (+ `/rotate`, `/delete`), `POST /nodes/ticket`, `WS /nodes/ws`. Off by default; also `GET /nodes/me`, `POST /nodes/settings`, `GET /nodes/social/search`, `POST /nodes/internal/fetch`, `GET /nodes/stats`, `GET /nodes/leaderboard`, `GET /nodes/client-version`, `GET /nodes/download/{artifact}` (see `plan.md`) |
-| `/sync` | Community sync: `GET /sync/spec.json`, `POST /sync/{users,posts,comments}`. Administrator only, off by default (`sync_enabled`), idempotent batch upserts keyed on the origin community's ids |
+| `/sync` | Community sync: `GET /sync/spec.json`, `POST /sync/{users,projects,posts,comments,polls,votes,reactions,follows,poll-votes,attachments}`, `GET /sync/export/{kind}`, `GET /sync/deletions`, `POST /sync/deletions/ack`. Administrator only, off by default (`sync_enabled`), idempotent batch upserts keyed on the origin community's ids |
 | `/news` | Developer news listing, detail page with comments |
 | `/posts` | Post detail, creation |
 | `/gists` | Code gist listing, detail, creation, and editing; left panel offers language filtering and free-text `search` (title, description, and author username), public read. A plain Markdown gist has a **View rendered / View raw** toggle beside its Copy button, switching between the raw source and the same rendered view a `Markdown Rendered` gist always shows |
@@ -220,7 +220,7 @@ The **Code Farm** (`/game`) is a cooperative idle game in the spirit of Farmvill
 - **Leaderboards.** Several boards, selectable from the game page: **Overall score** (a composite weighing refactor/prestige count, XP, lifetime harvests, a capped coin contribution, CI tier, plots, perks, and streak - the cap keeps it a measure of what you built rather than what you hoard), **Prestige**, **Harvests this week**, **Raid efficiency** (average coins per successful raid), **Fastest to Kernel** (time since your last refactor), **Fair play** (rewards recent activity over hoarding), and (when running) the current **Era** board.
 - **Eras (admin-managed seasons).** Administrators can start an Era at `/admin/game`: every farm's *visible* Era coins/harvests counters reset to zero, but real coin balances, prestige, Stars, Legacy, and Mastery are never touched. Ending an Era ranks farms by Era score (which gives prestige only partial weight, so veterans keep an edge without it being insurmountable), awards Stars to the top 10, and permanently records the results.
 
-The farm refreshes live over the pub/sub bus (a watered build appears on the owner's screen at once) and every plot countdown ticks client-side. Every endpoint also answers JSON, and Devii can play the game on the member's behalf via the `game_*` tools (`game_state`, `game_plant`, `game_harvest`, `game_buy_plot`, `game_upgrade_ci`, `game_fertilize`, `game_daily`, `game_claim_quest`, `game_water`, `game_steal`, `game_view_farm`, `game_leaderboard`, `game_upgrade_perk`, `game_upgrade_legacy`, `game_prestige`, `game_claim_grant`, `game_upgrade_mastery`, `game_buy_infrastructure`, `game_upgrade_defense`, `game_buy_cosmetic`, `game_equip_cosmetic`, `game_downgrade_defense`). See the API reference group **Code Farm** and the full player guide at `/docs/code-farm.html`.
+The farm refreshes live over the pub/sub bus (a watered build appears on the owner's screen at once) and every plot countdown ticks client-side. Every endpoint also answers JSON, and Devii can play the game on the member's behalf via the `game_*` tools (`game_state`, `game_plant`, `game_harvest`, `game_plot_stop`, `game_buy_plot`, `game_upgrade_ci`, `game_fertilize`, `game_daily`, `game_claim_quest`, `game_water`, `game_steal`, `game_view_farm`, `game_leaderboard`, `game_upgrade_perk`, `game_upgrade_legacy`, `game_prestige`, `game_claim_grant`, `game_upgrade_mastery`, `game_buy_infrastructure`, `game_upgrade_defense`, `game_buy_cosmetic`, `game_equip_cosmetic`, `game_downgrade_defense`). See the API reference group **Code Farm** and the full player guide at `/docs/code-farm.html`.
 
 ## Engagement
 
@@ -255,11 +255,11 @@ Members can install [Agent Skills](https://agentskills.io) from a URL at `/skill
 
 ## Community sync
 
-Users, posts and comments can be copied in from another community. The sending side posts batches of up to 50 records to `POST /sync/users`, `/sync/posts` and `/sync/comments` (administrator API key, `sync_enabled` set to `1`). Each record carries the id it has in the origin community (`origin_uid`); the server keeps a `sync_links` row from `(origin, kind, origin_uid)` to the local row, so a repeated record is an update or a no-op and never a duplicate. Records are validated and normalized with Pydantic (`devplacepy/services/sync/records.py`) and every record gets its own result, so one bad record never fails the batch. Votes, stars, XP and levels cannot be verified across communities and are not imported; they are listed under `ignored` in the result. Synced content sends no notifications, awards no XP, runs no AI correction and passes the content filter. `sync_origins` optionally restricts which origins may send. Limits, statuses and the record schemas are published at `GET /sync/spec.json`. The opt-in Community sync service (`CommunityService`) is the scheduled sender: it learns an external community, runs its generated collectors and feeds this API. A run refused by the kill switch, the allowlist or the aliasing switch is recorded as `skipped` and is never counted as a failure. Every run also writes a per-record sync log (`/admin/communities/{slug}/ledger?run=<uid>`): a verdict, exact counts per outcome (created, updated, unchanged, stale, skipped, warning, rejected, failed, error), and a filterable, paginated list with the reason each record was not synced in plain words. It is index-bound and capped, so it stays fast however large a run is. There is no export endpoint. Usage is written up in `prompt.md`.
+Users, posts and comments can be copied in from another community. The sending side posts batches of up to 50 records to `POST /sync/users`, `/sync/posts` and `/sync/comments` (administrator API key, `sync_enabled` set to `1`). Each record carries the id it has in the origin community (`origin_uid`); the server keeps a `sync_links` row from `(origin, kind, origin_uid)` to the local row, so a repeated record is an update or a no-op and never a duplicate. Records are validated and normalized with Pydantic (`devplacepy/services/sync/records.py`) and every record gets its own result, so one bad record never fails the batch. Stars, XP and levels cannot be verified across communities and are not imported; they are listed under `ignored` in the result. A record whose post, parent, poll or project has not arrived yet is parked in a bounded pending queue and released when it does. Synced content sends no notifications, awards no XP, runs no AI correction and passes the content filter. `sync_origins` optionally restricts which origins may send. Limits, statuses and the record schemas are published at `GET /sync/spec.json`. The opt-in Community sync service (`CommunityService`) is the scheduled sender: it learns an external community, runs its generated collectors and feeds this API. A run refused by the kill switch, the allowlist or the aliasing switch is recorded as `skipped` and is never counted as a failure. Every run also writes a per-record sync log (`/admin/communities/{slug}/ledger?run=<uid>`): a verdict, exact counts per outcome (created, updated, unchanged, stale, skipped, warning, rejected, failed, error), and a filterable, paginated list with the reason each record was not synced in plain words. It is index-bound and capped, so it stays fast however large a run is. The origin can read what we hold through `GET /sync/export/{kind}` (public data only, rows that arrived by sync are never echoed back) and which of its records we removed through `GET /sync/deletions`. Usage is written up in `prompt.md`.
 
 ### Community discovery
 
-Administrators can add an external community by name or address at `/admin/communities` (or `devplace community discover`). The system searches (rsearch), detects RSS, Atom, JSON and HTML feeds, learns the structure once, records the community's description, rules and size for the admin detail page, generates standalone collector code under `data/communities/<slug>/`, and then runs it on a schedule through the sync API above. Members never see these communities. Login-protected communities take a stored account; JavaScript pages are rendered through stealthii; the internal AI model helps with HTML structure it cannot read otherwise. Administrators can manage everything through the admin pages, `devplace community ...` and Devii. Dry run (`--dry-run`, the admin button, and the default of the `Community sync` service) validates and logs everything without writing. Design and limits: `add_community.md` and `devplacepy/services/community/CLAUDE.md`. Per community, administrators can optionally (all off by default) sample which posts are accepted, filter by topic and minimum comments, jitter the schedule, translate content, and store authors under permanent pseudonyms with every mention rewritten (`community_sync_additional.md`); this is administrator-only and invisible to members.
+Administrators can add an external community by name or address at `/admin/communities` (or `devplace community discover`). The system searches (rsearch), detects RSS, Atom, JSON and HTML feeds, learns the structure once, records the community's description, rules and size for the admin detail page, generates standalone collector code under `data/communities/<slug>/`, and then runs it on a schedule through the sync API above. Members never see these communities. Login-protected communities take a stored account (form, JSON-API and TOTP logins; a captcha is detected and reported, never solved); a run keeps checkpoints so a killed run stays visible; administrators can edit and restore a community's learned schema at `/admin/communities/{slug}/schema`; new accounts created by a community run get a realistic handle derived from the author (the person behind a `by`/`via` is extracted from feed strings, then a seeded, weighted pattern adds a year, number or light humor; existing accounts are never renamed; `devplacepy/services/sync/CLAUDE.md`, "Realistic handles"); JavaScript pages are rendered through stealthii; the internal AI model helps with HTML structure it cannot read otherwise. Administrators can manage everything through the admin pages, `devplace community ...` and Devii. Dry run (`--dry-run`, the admin button, and the default of the `Community sync` service) validates and logs everything without writing. Design and limits: `add_community.md` and `devplacepy/services/community/CLAUDE.md`. Per community, administrators can optionally (all off by default) sample which posts are accepted, filter by topic and minimum comments, jitter the schedule, translate content, and store authors under permanent pseudonyms with every mention rewritten (`community_sync_additional.md`); this is administrator-only and invisible to members.
 
 ## Community nodes (built, off by default)
 

devplacepy/attachments.py

@@ -274,14 +274,14 @@ def delete_inline_image(filename):
         logger.warning(f"Failed to delete inline image {filename}: {e}")
 
 
-def store_attachment(file_bytes, original_filename, user_uid):
+def store_attachment(file_bytes, original_filename, user_uid, uid=None):
     if len(file_bytes) > _get_max_upload_bytes():
         return None
     ext = Path(original_filename).suffix.lower()
     if not is_extension_allowed(ext):
         return None
 
-    uid = generate_uid()
+    uid = uid or generate_uid()
     stored_name = f"{uid}{ext}"
     directory = _directory_for(uid)
     file_dir = ATTACHMENTS_DIR / directory

devplacepy/cli/community.py

@@ -18,6 +18,7 @@ def _credentials(args) -> dict:
         "username": args.login_user or "",
         "email": args.login_email or "",
         "password": args.login_password or os.environ.get("COMMUNITY_PASSWORD", ""),
+        "totp": args.login_totp or os.environ.get("COMMUNITY_TOTP_SECRET", ""),
     }
 
 
@@ -102,14 +103,14 @@ def cmd_community_login(args):
         print(f"Unknown community '{args.slug}'")
         sys.exit(1)
     if args.clear:
-        store.update(row["uid"], login_username="", login_email="", login_password="")
+        store.update(row["uid"], login_username="", login_email="", login_password="", login_totp_secret="")
         print(f"Removed the stored account of {row['slug']}")
     else:
         secrets = _credentials(args)
         if not secrets["password"] or not (secrets["username"] or secrets["email"]):
             print("Give --login-user or --login-email, and --login-password (or COMMUNITY_PASSWORD)")
             sys.exit(1)
-        store.update(row["uid"], login_username=secrets["username"], login_email=secrets["email"], login_password=secrets["password"])
+        store.update(row["uid"], login_username=secrets["username"], login_email=secrets["email"], login_password=secrets["password"], login_totp_secret=secrets["totp"])
         print(f"Stored the account for {row['slug']}; run 'community discover \"{row['query']}\" --relearn' to learn behind the login")
     _audit_cli("cli.community.login", f"CLI changed the stored account of community {row['slug']}", target_label=row["slug"])
 
@@ -133,6 +134,7 @@ def add_login_arguments(parser):
     parser.add_argument("--login-user", help="username of an account on the community")
     parser.add_argument("--login-email", help="email of an account on the community")
     parser.add_argument("--login-password", help="password (or set COMMUNITY_PASSWORD)")
+    parser.add_argument("--login-totp", help="authenticator secret of the account, for a login with two-factor codes (or set COMMUNITY_TOTP_SECRET)")
 
 
 def register_community(subparsers):

devplacepy/cli/nodes.py

@@ -110,6 +110,33 @@ def cmd_nodes_clear(args):
         _audit_cli("cli.nodes.clear","CLI cleared the node job queue and event log", metadata=counts)
 
 
+def cmd_nodes_geo_sync(args):
+    import asyncio
+
+    from devplacepy.services.nodes.geo import GeoError, geo_sync
+
+    try:
+        summary = asyncio.run(geo_sync(dry_run=args.dry_run, month=args.month))
+    except GeoError as exc:
+        print(f"geo-sync refused: {exc}")
+        sys.exit(1)
+    verb = "installed" if summary["installed"] else "verified, not installed (dry run)"
+    print(f"{verb}: month {summary['month']}, {summary['asn_rows']} ASN ranges, {summary['country_rows']} country ranges")
+    if summary["installed"]:
+        _audit_cli("cli.nodes.geo_sync", f"CLI installed the node address database for {summary['month']}", metadata={"month": summary["month"]})
+
+
+def cmd_nodes_geo_status(args):
+    from devplacepy.services.nodes.geo import shared_geo
+
+    status = shared_geo().status()
+    if not status["available"]:
+        print(f"not installed at {status['path']}; every donor is classed unknown")
+        return
+    print(f"month {status.get('month')}, {status.get('asn_rows')} ASN ranges, {status.get('country_rows')} country ranges, loaded {status.get('loaded_at')}")
+    print(status["attribution"])
+
+
 def register_nodes(subparsers):
     nodes = subparsers.add_parser("nodes", help="Community donor node network")
     sub = nodes.add_subparsers(title="action", dest="action")
@@ -130,3 +157,8 @@ def register_nodes(subparsers):
     clear = sub.add_parser("clear", help="Empty the job queue and the event log (never usage or donor totals)")
     clear.add_argument("--force", action="store_true", help="actually delete; without it only counts are shown")
     clear.set_defaults(func=cmd_nodes_clear)
+    geo_sync = sub.add_parser("geo-sync", help="Download the DB-IP address database used to class donor addresses")
+    geo_sync.add_argument("--dry-run", action="store_true", help="download and verify, do not install")
+    geo_sync.add_argument("--month", default="", help="YYYY-MM, default the newest published")
+    geo_sync.set_defaults(func=cmd_nodes_geo_sync)
+    sub.add_parser("geo-status", help="Show the installed address database").set_defaults(func=cmd_nodes_geo_status)

devplacepy/config.py

@@ -26,6 +26,8 @@ FORK_STAGING_DIR = DATA_DIR / "fork_staging"
 BACKUPS_DIR = DATA_DIR / "backups"
 BACKUP_STAGING_DIR = DATA_DIR / "backup_staging"
 NODE_CLIENTS_DIR = DATA_DIR / "node_clients"
+NODE_GEO_DIR = DATA_DIR / "node_geo"
+NODE_GEO_DB = NODE_GEO_DIR / "geo.db"
 
 RCLONE_BIN = environ.get("DEVPLACE_RCLONE_BIN", "rclone")
 RCLONE_CONFIG_FILE = environ.get(
@@ -155,6 +157,7 @@ DATA_PATHS: dict[str, Path] = {
     "backups": BACKUPS_DIR,
     "backup_staging": BACKUP_STAGING_DIR,
     "node_clients": NODE_CLIENTS_DIR,
+    "node_geo": NODE_GEO_DIR,
     "seo_reports": SEO_REPORTS_DIR,
     "planning_reports": PLANNING_REPORTS_DIR,
     "dbapi": DBAPI_DIR,

devplacepy/database/core.py

@@ -2,6 +2,7 @@
 
 import dataset
 import logging
+import warnings
 from pathlib import Path
 from sqlalchemy import or_
 from sqlalchemy.pool import NullPool
@@ -20,6 +21,8 @@ from devplacepy.config import (
 
 logger = logging.getLogger(__name__)
 
+warnings.filterwarnings("ignore", message="Skipped unsupported reflection of expression-based index")
+
 ensure_data_dirs()
 if DATABASE_URL.startswith("sqlite:///"):
     _db_file = DATABASE_URL[len("sqlite:///") :]

devplacepy/database/nodes.py

@@ -24,6 +24,7 @@ LABEL_MAX = 60
 EVENT_DETAIL_MAX = 500
 MEGABYTE = 1024 * 1024
 BATCH_SIZE = 500
+COUNTER_ROW_LIMIT = 500
 HOURS_PATTERN = re.compile(r"^([01]\d|2[0-3]):[0-5]\d$")
 PLATFORMS = tuple(sorted({item.adapter for item in SPEC.destinations}))
 FLAG_OPEN = "open"
@@ -714,14 +715,18 @@ def enqueue_request(
     method: str,
     adapter: str,
     mode: str,
+    *,
+    shadow_of: str = "",
+    exclude_device: str = "",
 ) -> str:
     uid = _uid()
     created = _execute(
         "INSERT INTO node_requests (uid, caller_kind, caller_id, url, method, adapter, "
         "mode, status, device_uid, result_status, result_bytes, result_body, "
-        "result_headers, truncated, error, created_at, assigned_at, finished_at) "
+        "result_headers, truncated, error, device_asn_class, shadow_of, exclude_device, "
+        "created_at, assigned_at, finished_at) "
         "SELECT :uid, :kind, :caller, :url, :method, :adapter, :mode, 'queued', '', 0, 0, "
-        "'', '', 0, '', :now, '', '' "
+        "'', '', 0, '', '', :shadow_of, :exclude_device, :now, '', '' "
         "WHERE (SELECT COUNT(*) FROM node_requests WHERE status IN ('queued','assigned')) "
         "< :max_depth",
         {
@@ -732,6 +737,8 @@ def enqueue_request(
             "method": method,
             "adapter": adapter,
             "mode": mode,
+            "shadow_of": shadow_of,
+            "exclude_device": exclude_device,
             "now": _now_iso(),
             "max_depth": LIMITS.max_queue_depth,
         },
@@ -766,12 +773,18 @@ def count_open_requests() -> int:
     )["c"]
 
 
-def claim_request(request_uid: str, device_uid: str) -> bool:
+def claim_request(request_uid: str, device_uid: str, asn_class: str = "") -> bool:
     return (
         _execute(
             "UPDATE node_requests SET status = 'assigned', device_uid = :device, "
-            "assigned_at = :now WHERE uid = :uid AND status = 'queued'",
-            {"device": device_uid, "now": _now_iso(), "uid": request_uid},
+            "device_asn_class = :asn_class, assigned_at = :now "
+            "WHERE uid = :uid AND status = 'queued'",
+            {
+                "device": device_uid,
+                "asn_class": asn_class,
+                "now": _now_iso(),
+                "uid": request_uid,
+            },
         )
         == 1
     )
@@ -989,3 +1002,165 @@ def recent_devices(limit: int = ADMIN_LIST_LIMIT) -> list[dict]:
             n=limit,
         )
     ]
+
+
+def set_class_override(device_uid: str, value: str) -> bool:
+    return (
+        _execute(
+            "UPDATE node_devices SET class_override = :value, updated_at = :now "
+            "WHERE uid = :uid AND deleted_at IS NULL",
+            {"value": value, "now": _now_iso(), "uid": device_uid},
+        )
+        == 1
+    )
+
+
+def touch_devices(device_uids, moment: datetime | None = None) -> int:
+    stamp = (moment or datetime.now(timezone.utc)).isoformat()
+    touched = 0
+    for chunk in _batches(device_uids):
+        placeholders, params = _in_clause(chunk)
+        touched += _execute(
+            f"UPDATE node_devices SET last_seen_at = :now WHERE uid IN ({placeholders})",
+            {"now": stamp, **params},
+        )
+    return touched
+
+
+def flush_counters(deltas: dict[str, int]) -> int:
+    rows = {name: int(value) for name, value in deltas.items() if int(value) > 0}
+    if not rows:
+        return 0
+    now = _now_iso()
+    with db:
+        for name, value in rows.items():
+            db.executable.execute(
+                text(
+                    "INSERT INTO node_counters (name, value, updated_at) VALUES (:name, :value, :now) "
+                    "ON CONFLICT(name) DO UPDATE SET value = COALESCE(value, 0) + excluded.value, "
+                    "updated_at = excluded.updated_at"
+                ),
+                {"name": name, "value": value, "now": now},
+            )
+    return len(rows)
+
+
+def lifetime_counters(prefix: str = "") -> dict[str, int]:
+    rows = db.query(
+        "SELECT name, COALESCE(value, 0) AS value FROM node_counters "
+        "WHERE name LIKE :prefix ESCAPE '\\' ORDER BY name LIMIT :n",
+        prefix=prefix.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") + "%",
+        n=COUNTER_ROW_LIMIT,
+    )
+    return {row["name"]: int(row["value"]) for row in rows}
+
+
+def save_circuits(rows: list[dict]) -> int:
+    now = _now_iso()
+    with db:
+        db.executable.execute(text("DELETE FROM node_circuits"))
+        for row in rows[: LIMITS.circuit_snapshot_max]:
+            db.executable.execute(
+                text(
+                    "INSERT INTO node_circuits (key, failures, state, opened_at, updated_at) "
+                    "VALUES (:key, :failures, :state, :opened_at, :now)"
+                ),
+                {
+                    "key": row["key"],
+                    "failures": int(row["failures"]),
+                    "state": row["state"],
+                    "opened_at": float(row["opened_at"]),
+                    "now": now,
+                },
+            )
+    return min(len(rows), LIMITS.circuit_snapshot_max)
+
+
+def load_circuits() -> list[dict]:
+    return [
+        dict(row)
+        for row in db.query(
+            "SELECT key, failures, state, opened_at FROM node_circuits LIMIT :n",
+            n=LIMITS.circuit_snapshot_max,
+        )
+    ]
+
+
+def cache_get(key: str, moment: datetime | None = None) -> dict | None:
+    now = (moment or datetime.now(timezone.utc)).isoformat()
+    rows = list(
+        db.query(
+            "SELECT status, headers, body, stored_at FROM node_cache "
+            "WHERE key = :key AND expires_at > :now LIMIT 1",
+            key=key,
+            now=now,
+        )
+    )
+    return dict(rows[0]) if rows else None
+
+
+def cache_put(
+    key: str,
+    method: str,
+    url: str,
+    status: int,
+    headers: str,
+    body: str,
+    size: int,
+    ttl_seconds: int,
+    moment: datetime | None = None,
+) -> bool:
+    if ttl_seconds <= 0:
+        return False
+    now = moment or datetime.now(timezone.utc)
+    return (
+        _execute(
+            "INSERT INTO node_cache (key, method, url, status, headers, body, size, stored_at, expires_at) "
+            "VALUES (:key, :method, :url, :status, :headers, :body, :size, :stored, :expires) "
+            "ON CONFLICT(key) DO UPDATE SET status = excluded.status, headers = excluded.headers, "
+            "body = excluded.body, size = excluded.size, stored_at = excluded.stored_at, "
+            "expires_at = excluded.expires_at",
+            {
+                "key": key,
+                "method": method,
+                "url": url[: LIMITS.url_max_chars],
+                "status": int(status),
+                "headers": headers,
+                "body": body,
+                "size": int(size),
+                "stored": now.isoformat(),
+                "expires": (now + timedelta(seconds=ttl_seconds)).isoformat(),
+            },
+        )
+        == 1
+    )
+
+
+def cache_prune(max_bytes: int, moment: datetime | None = None) -> dict[str, int]:
+    now = (moment or datetime.now(timezone.utc)).isoformat()
+    expired = _execute("DELETE FROM node_cache WHERE expires_at <= :now", {"now": now})
+    total = next(iter(db.query("SELECT COALESCE(SUM(size), 0) AS total FROM node_cache")))["total"]
+    evicted = 0
+    while total > max_bytes:
+        rows = list(
+            db.query("SELECT key, size FROM node_cache ORDER BY stored_at, id LIMIT :n", n=BATCH_SIZE)
+        )
+        if not rows:
+            break
+        for row in rows:
+            if total <= max_bytes:
+                break
+            evicted += _execute("DELETE FROM node_cache WHERE key = :key", {"key": row["key"]})
+            total -= int(row["size"] or 0)
+    return {"expired": expired, "evicted": evicted, "bytes": max(0, int(total))}
+
+
+def cache_clear() -> int:
+    return _execute("DELETE FROM node_cache", {})
+
+
+def cache_stats() -> dict[str, int]:
+    row = next(
+        iter(db.query("SELECT COUNT(*) AS entries, COALESCE(SUM(size), 0) AS bytes FROM node_cache"))
+    )
+    return {"entries": int(row["entries"]), "bytes": int(row["bytes"])}

devplacepy/database/schema.py

@@ -1506,6 +1506,7 @@ def init_db():
         ("ready_at", ""),
         ("watered_by", "[]"),
         ("raided_fraction", 0.0),
+        ("stop_after_harvest", 0),
         ("created_at", ""),
         ("updated_at", ""),
     ):
@@ -1863,6 +1864,10 @@ def init_db():
     for table in SOFT_DELETE_TABLES:
         ensure_soft_delete_columns(table)
 
+    from .sync import ensure_export_indexes
+
+    ensure_export_indexes()
+
     _index(
         db,
         "comments",
@@ -2222,6 +2227,7 @@ NODES_COLUMNS: dict[str, tuple[tuple[str, object], ...]] = {
         ("country", ""),
         ("asn", ""),
         ("asn_class", ""),
+        ("class_override", ""),
         ("ip_hash", ""),
         ("last_seen_at", ""),
         ("created_at", ""),
@@ -2292,10 +2298,36 @@ NODES_COLUMNS: dict[str, tuple[tuple[str, object], ...]] = {
         ("result_headers", ""),
         ("truncated", 0),
         ("error", ""),
+        ("device_asn_class", ""),
+        ("shadow_of", ""),
+        ("exclude_device", ""),
         ("created_at", ""),
         ("assigned_at", ""),
         ("finished_at", ""),
     ),
+    "node_counters": (
+        ("name", ""),
+        ("value", 0),
+        ("updated_at", ""),
+    ),
+    "node_circuits": (
+        ("key", ""),
+        ("failures", 0),
+        ("state", ""),
+        ("opened_at", 0.0),
+        ("updated_at", ""),
+    ),
+    "node_cache": (
+        ("key", ""),
+        ("method", ""),
+        ("url", ""),
+        ("status", 0),
+        ("headers", ""),
+        ("body", ""),
+        ("size", 0),
+        ("stored_at", ""),
+        ("expires_at", ""),
+    ),
     "node_caller_daily": (
         ("uid", ""),
         ("caller_kind", ""),
@@ -2360,6 +2392,11 @@ def _ensure_nodes_tables() -> None:
         where="status IN ('queued','assigned')",
     )
     _index(db, "node_requests", "idx_node_requests_created", ["created_at"])
+    _index(db, "node_requests", "idx_node_requests_shadow", ["shadow_of"], where="shadow_of != ''")
+    _index(db, "node_counters", "idx_node_counters_name", ["name"], unique=True)
+    _index(db, "node_circuits", "idx_node_circuits_key", ["key"], unique=True)
+    _index(db, "node_cache", "idx_node_cache_key", ["key"], unique=True)
+    _index(db, "node_cache", "idx_node_cache_expires", ["expires_at"])
     _index(
         db,
         "node_caller_daily",
@@ -2390,6 +2427,28 @@ SYNC_COLUMNS: dict[str, tuple[tuple[str, object], ...]] = {
         ("origin_updated_at", ""),
         ("first_synced_at", ""),
         ("synced_at", ""),
+        ("ack_at", ""),
+    ),
+    "sync_pending": (
+        ("uid", ""),
+        ("origin", ""),
+        ("kind", ""),
+        ("origin_uid", ""),
+        ("missing_kind", ""),
+        ("missing_uid", ""),
+        ("record", ""),
+        ("attempts", 0),
+        ("created_at", ""),
+        ("expires_at", ""),
+    ),
+    "sync_candidates": (
+        ("uid", ""),
+        ("origin", ""),
+        ("post_uid", ""),
+        ("record", ""),
+        ("comment_count", 0),
+        ("created_at", ""),
+        ("expires_at", ""),
     ),
     "sync_identities": (
         ("uid", ""),
@@ -2420,6 +2479,11 @@ def _ensure_sync_tables() -> None:
         unique=True,
     )
     _index(db, "sync_links", "idx_sync_links_local", ["kind", "local_uid"])
+    _index(db, "sync_pending", "idx_sync_pending_record", ["origin", "kind", "origin_uid"], unique=True)
+    _index(db, "sync_pending", "idx_sync_pending_missing", ["origin", "missing_kind", "missing_uid"])
+    _index(db, "sync_pending", "idx_sync_pending_expires", ["expires_at"])
+    _index(db, "sync_candidates", "idx_sync_candidates_post", ["origin", "post_uid"], unique=True)
+    _index(db, "sync_candidates", "idx_sync_candidates_expires", ["expires_at"])
     _index(
         db,
         "sync_identities",
@@ -2461,6 +2525,8 @@ COMMUNITY_COLUMNS: dict[str, tuple[tuple[str, object], ...]] = {
         ("login_username", ""),
         ("login_email", ""),
         ("login_password", ""),
+        ("login_totp_secret", ""),
+        ("schema_previous", ""),
         ("sync_options", ""),
         ("claimed_at", ""),
         ("deleted_at", ""),
@@ -2516,6 +2582,37 @@ def _ensure_community_tables() -> None:
     _index(db, "community_run_items", "idx_community_items_run_outcome", ["run_uid", "outcome", "kind", "seq"])
     _index(db, "community_run_items", "idx_community_items_community_rank", ["community_uid", "rank", "created_at DESC", "seq"])
     _index(db, "community_run_items", "idx_community_items_community_outcome", ["community_uid", "outcome", "created_at DESC", "seq"])
+    _index(db, "community_run_items", "idx_community_items_run_seq", ["run_uid", "seq"], unique=True)
+    _ensure_ledger_search()
+
+
+LEDGER_SEARCH_TABLE = "community_run_items_fts"
+LEDGER_SEARCH_COLUMNS = ("label", "origin_uid", "detail")
+
+
+def _ensure_ledger_search() -> None:
+    try:
+        exists = bool(list(db.query("SELECT name FROM sqlite_master WHERE name = :name", name=LEDGER_SEARCH_TABLE)))
+        columns = ", ".join(LEDGER_SEARCH_COLUMNS)
+        new_columns = ", ".join(f"new.{column}" for column in LEDGER_SEARCH_COLUMNS)
+        old_columns = ", ".join(f"old.{column}" for column in LEDGER_SEARCH_COLUMNS)
+        with db:
+            db.query(
+                f"CREATE VIRTUAL TABLE IF NOT EXISTS {LEDGER_SEARCH_TABLE} USING fts5({columns}, "
+                "content='community_run_items', content_rowid='id', tokenize='trigram')"
+            )
+            db.query(
+                "CREATE TRIGGER IF NOT EXISTS community_run_items_fts_insert AFTER INSERT ON community_run_items BEGIN "
+                f"INSERT INTO {LEDGER_SEARCH_TABLE}(rowid, {columns}) VALUES (new.id, {new_columns}); END"
+            )
+            db.query(
+                "CREATE TRIGGER IF NOT EXISTS community_run_items_fts_delete AFTER DELETE ON community_run_items BEGIN "
+                f"INSERT INTO {LEDGER_SEARCH_TABLE}({LEDGER_SEARCH_TABLE}, rowid, {columns}) VALUES ('delete', old.id, {old_columns}); END"
+            )
+            if not exists:
+                db.query(f"INSERT INTO {LEDGER_SEARCH_TABLE}({LEDGER_SEARCH_TABLE}) VALUES ('rebuild')")
+    except Exception as exc:
+        logger.warning("Ledger full-text search is unavailable, the text filter will scan: %s", exc)
 
 
 def _ensure_moderation_tables() -> None:

devplacepy/database/sync.py

@@ -2,6 +2,8 @@
 
 from __future__ import annotations
 
+from datetime import datetime, timedelta, timezone
+
 from sqlalchemy import text
 
 from devplacepy.utils.text import generate_uid
@@ -129,3 +131,230 @@ def claim_identity(
 
 def save_style_samples(identity_uid: str, samples_json: str) -> None:
     get_table("sync_identities").update({"uid": identity_uid, "style_samples": samples_json}, ["uid"])
+
+
+PENDING_TABLE = "sync_pending"
+
+
+def park_pending(
+    origin: str,
+    kind: str,
+    origin_uid: str,
+    missing_kind: str,
+    missing_uid: str,
+    record_json: str,
+    *,
+    attempts: int,
+    ttl_days: int,
+    cap: int,
+) -> bool:
+    now = datetime.now(timezone.utc)
+    with db:
+        result = db.executable.execute(
+            text(
+                "INSERT INTO sync_pending (uid, origin, kind, origin_uid, missing_kind, missing_uid, "
+                "record, attempts, created_at, expires_at) "
+                "SELECT :uid, :origin, :kind, :origin_uid, :missing_kind, :missing_uid, :record, "
+                ":attempts, :now, :expires "
+                "WHERE (SELECT COUNT(*) FROM sync_pending WHERE origin = :origin) < :cap "
+                "ON CONFLICT(origin, kind, origin_uid) DO UPDATE SET "
+                "missing_kind = excluded.missing_kind, missing_uid = excluded.missing_uid, "
+                "record = excluded.record, attempts = excluded.attempts, expires_at = excluded.expires_at"
+            ),
+            {
+                "uid": generate_uid(),
+                "origin": origin,
+                "kind": kind,
+                "origin_uid": origin_uid,
+                "missing_kind": missing_kind,
+                "missing_uid": missing_uid,
+                "record": record_json,
+                "attempts": int(attempts),
+                "now": now.isoformat(),
+                "expires": (now + timedelta(days=ttl_days)).isoformat(),
+                "cap": int(cap),
+            },
+        )
+        return result.rowcount >= 1
+
+
+def pending_exists(origin: str, kind: str, origin_uid: str) -> bool:
+    return get_table(PENDING_TABLE).find_one(origin=origin, kind=kind, origin_uid=origin_uid) is not None
+
+
+def ready_pending(limit: int, origin: str = "") -> list[dict]:
+    clause = "AND p.origin = :origin " if origin else ""
+    rows = db.query(
+        "SELECT p.* FROM sync_pending p JOIN sync_links l ON l.origin = p.origin "
+        "AND l.kind = p.missing_kind AND l.origin_uid = p.missing_uid "
+        f"WHERE 1 = 1 {clause}ORDER BY p.created_at, p.id LIMIT :n",
+        n=int(limit),
+        **({"origin": origin} if origin else {}),
+    )
+    return [dict(row) for row in rows]
+
+
+def delete_pending(uid: str) -> int:
+    with db:
+        return db.executable.execute(text("DELETE FROM sync_pending WHERE uid = :uid"), {"uid": uid}).rowcount
+
+
+def expire_pending(max_attempts: int) -> int:
+    with db:
+        return db.executable.execute(
+            text("DELETE FROM sync_pending WHERE expires_at <= :now OR attempts >= :attempts"),
+            {"now": _now_iso(), "attempts": int(max_attempts)},
+        ).rowcount
+
+
+def pending_counts() -> dict[str, int]:
+    rows = db.query("SELECT origin, COUNT(*) AS n FROM sync_pending GROUP BY origin ORDER BY n DESC LIMIT 100")
+    return {row["origin"]: int(row["n"]) for row in rows}
+
+
+STAR_TABLES = {"post": "posts", "project": "projects"}
+
+
+def recompute_stars(target_type: str, target_uid: str) -> int | None:
+    table = STAR_TABLES.get(target_type)
+    if table is None:
+        return None
+    with db:
+        db.executable.execute(
+            text(
+                f"UPDATE {table} SET stars = (SELECT COALESCE(SUM(value), 0) FROM votes "
+                "WHERE target_uid = :uid AND target_type = :type AND deleted_at IS NULL) "
+                "WHERE uid = :uid"
+            ),
+            {"uid": target_uid, "type": target_type},
+        )
+    row = get_table(table).find_one(uid=target_uid)
+    return int(row["stars"] or 0) if row else None
+
+
+EXPORT_CHANGED: dict[str, str] = {
+    "users": "created_at",
+    "projects": "COALESCE(deleted_at, updated_at, created_at)",
+    "posts": "COALESCE(deleted_at, updated_at, created_at)",
+    "comments": "COALESCE(deleted_at, updated_at, created_at)",
+    "polls": "COALESCE(deleted_at, created_at)",
+    "follows": "COALESCE(deleted_at, created_at)",
+    "attachments": "COALESCE(deleted_at, created_at)",
+}
+
+
+def ensure_export_indexes() -> None:
+    from .core import _index
+
+    for table, expression in EXPORT_CHANGED.items():
+        if table in db.tables:
+            _index(db, table, f"idx_{table}_changed", [expression, "uid"])
+
+
+def deletion_rows(origin: str, kind: str, table: str, after_id: int, limit: int) -> list[dict]:
+    if table not in db.tables:
+        return []
+    soft = db[table].has_column("deleted_at")
+    columns = "t.deleted_at AS deleted_at, t.deleted_by AS deleted_by" if soft else "NULL AS deleted_at, NULL AS deleted_by"
+    deleted = "OR (t.deleted_at IS NOT NULL AND COALESCE(t.deleted_by, '') != :actor)" if soft else ""
+    rows = db.query(
+        f"SELECT l.id AS link_id, l.kind AS kind, l.origin_uid AS origin_uid, t.uid AS local_uid, {columns} "
+        f"FROM sync_links l LEFT JOIN {table} t ON t.uid = l.local_uid "
+        "WHERE l.origin = :origin AND l.kind = :kind AND l.id > :after "
+        "AND COALESCE(l.ack_at, '') = '' "
+        f"AND (t.uid IS NULL {deleted}) "
+        "ORDER BY l.id LIMIT :n",
+        origin=origin,
+        kind=kind,
+        after=int(after_id),
+        n=int(limit),
+        **({"actor": "sync"} if soft else {}),
+    )
+    return [dict(row) for row in rows]
+
+
+def acknowledge_links(origin: str, items: list[tuple[str, str]]) -> int:
+    stamp = _now_iso()
+    done = 0
+    with db:
+        for kind, origin_uid in items:
+            done += db.executable.execute(
+                text(
+                    "UPDATE sync_links SET ack_at = :now WHERE origin = :origin AND kind = :kind "
+                    "AND origin_uid = :origin_uid AND COALESCE(ack_at, '') = ''"
+                ),
+                {"now": stamp, "origin": origin, "kind": kind, "origin_uid": origin_uid},
+            ).rowcount
+    return done
+
+
+def rename_identity(identity_uid: str, our_name: str) -> bool:
+    try:
+        with db:
+            result = db.executable.execute(
+                text("UPDATE sync_identities SET our_name = :name, our_key = :key WHERE uid = :uid"),
+                {"name": our_name, "key": our_name.lower(), "uid": identity_uid},
+            )
+        return result.rowcount == 1
+    except Exception:
+        return False
+
+
+def linked_rows_containing(origin: str, kind: str, table: str, columns: tuple[str, ...], needle: str, limit: int) -> list[dict]:
+    pattern = "%" + needle.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") + "%"
+    matches = " OR ".join(f"t.{column} LIKE :pattern ESCAPE '\\'" for column in columns)
+    selected = ", ".join(f"t.{column} AS {column}" for column in columns)
+    rows = db.query(
+        f"SELECT t.uid AS uid, {selected} FROM sync_links l JOIN {table} t ON t.uid = l.local_uid "
+        f"WHERE l.origin = :origin AND l.kind = :kind AND ({matches}) ORDER BY l.id LIMIT :limit",
+        origin=origin,
+        kind=kind,
+        pattern=pattern,
+        limit=int(limit),
+    )
+    return [dict(row) for row in rows]
+
+
+def load_candidates(origin: str) -> list[dict]:
+    expire_candidates()
+    return [dict(row) for row in get_table("sync_candidates").find(origin=origin, order_by=["id"])]
+
+
+def save_candidate(origin: str, post_uid: str, record_json: str, comment_count: int, *, ttl_days: int, cap: int) -> bool:
+    now = datetime.now(timezone.utc)
+    with db:
+        result = db.executable.execute(
+            text(
+                "INSERT INTO sync_candidates (uid, origin, post_uid, record, comment_count, created_at, expires_at) "
+                "SELECT :uid, :origin, :post_uid, :record, :count, :now, :expires "
+                "WHERE (SELECT COUNT(*) FROM sync_candidates WHERE origin = :origin) < :cap "
+                "ON CONFLICT(origin, post_uid) DO UPDATE SET record = excluded.record, "
+                "comment_count = excluded.comment_count"
+            ),
+            {
+                "uid": generate_uid(),
+                "origin": origin,
+                "post_uid": post_uid,
+                "record": record_json,
+                "count": int(comment_count),
+                "now": now.isoformat(),
+                "expires": (now + timedelta(days=ttl_days)).isoformat(),
+                "cap": int(cap),
+            },
+        )
+        return result.rowcount >= 1
+
+
+def delete_candidate(origin: str, post_uid: str) -> int:
+    with db:
+        return db.executable.execute(
+            text("DELETE FROM sync_candidates WHERE origin = :origin AND post_uid = :post_uid"),
+            {"origin": origin, "post_uid": post_uid},
+        ).rowcount
+
+
+def expire_candidates() -> int:
+    with db:
+        return db.executable.execute(
+            text("DELETE FROM sync_candidates WHERE expires_at <= :now"), {"now": _now_iso()}
+        ).rowcount

devplacepy/docs_api/groups/community.py

@@ -7,6 +7,7 @@ ACCOUNT = [
     field("login_username", "form", "string", False, "bot", "Username of an account on the community, for a login-protected community."),
     field("login_email", "form", "string", False, "bot@example.com", "Email of that account, when the community logs in by email."),
     field("login_password", "form", "string", False, "", "Password of that account. Stored for the collectors and never returned by any endpoint."),
+    field("login_totp_secret", "form", "string", False, "", "Authenticator (TOTP) secret of that account when it uses two-factor codes. Stored for the collectors and never returned by any endpoint."),
 ]
 
 ITEM_SAMPLE = {
@@ -93,6 +94,41 @@ Every endpoint follows the shared [Conventions & Errors](/docs/conventions.html)
                 "busy": False,
             },
         ),
+        endpoint(
+            id="community-schema",
+            method="GET",
+            path="/admin/communities/{slug}/schema",
+            title="Learned schema",
+            summary="The complete learned schema of a community as JSON, editable by hand.",
+            auth="admin",
+            params=[SLUG],
+            sample_response={"community": ITEM_SAMPLE, "schema_text": "{...}", "has_previous": False, "errors": [], "notice": ""},
+        ),
+        endpoint(
+            id="community-schema-save",
+            method="POST",
+            path="/admin/communities/{slug}/schema",
+            title="Replace the learned schema",
+            summary="Validate a schema, check it against a small live sample, and only then store it. The previous schema is kept.",
+            auth="admin",
+            encoding="form",
+            destructive=True,
+            params=[SLUG, field("schema", "form", "string", True, "{...}", "The complete schema as a JSON object string.")],
+            sample_response={"ok": True, "redirect": "/admin/communities/SLUG/schema", "data": {"posts": 5, "comments": 12}},
+            notes=["`400` lists every structural problem; `422` means the live sample collected no post. Nothing is stored in either case. Audited as `admin.community.schema`."],
+        ),
+        endpoint(
+            id="community-schema-restore",
+            method="POST",
+            path="/admin/communities/{slug}/schema/restore",
+            title="Restore the previous schema",
+            summary="Swap back to the schema that was replaced by the last save.",
+            auth="admin",
+            encoding="form",
+            params=[SLUG],
+            sample_response={"ok": True, "redirect": "/admin/communities/SLUG/schema"},
+            notes=["`409` when there is no previous schema. Audited as `admin.community.schema_restore`."],
+        ),
         endpoint(
             id="community-ledger",
             method="GET",
@@ -177,6 +213,7 @@ Every endpoint follows the shared [Conventions & Errors](/docs/conventions.html)
                 field("min_comments", "form", "integer", False, "0", "Minimum comments collected in the run for a new post to be accepted."),
                 field("topics", "form", "string", False, "devlog, rant", "Comma separated topic allow-list; empty accepts all."),
                 field("jitter_percent", "form", "integer", False, "0", "Randomise the next run time by up to this percentage of the interval, 0 to 50."),
+                field("collect_attachments", "form", "boolean", False, "false", "Download the images and PDFs linked from posts and comments (at most 4 per post, 40 and 25 MB per run) and store them as attachments."),
             ],
             sample_response={"ok": True, "redirect": "/admin/communities/other-example-org"},
         ),

devplacepy/docs_api/groups/game.py

@@ -163,6 +163,19 @@ client are on the [Code Farm guide](/docs/code-farm.html).
                 params=[field("slot", "form", "integer", True, "0", "Plot slot index, 0-based.")],
                 sample_response={"ok": True, "farm": {"coins": 86}},
             ),
+            endpoint(
+                id="game-plot-stop",
+                method="POST",
+                path="/game/plot/stop",
+                title="Pause auto-replant on a plot",
+                summary="Needs the Continuous Delivery Mastery upgrade. With stop true, the next harvest of this plot does not replant, so you can choose a different crop. The pause clears itself when the plot is planted again. With stop false it cancels a pending pause. Rejected on an empty plot.",
+                auth="user",
+                params=[
+                    field("slot", "form", "integer", True, "0", "Plot slot index, 0-based."),
+                    field("stop", "form", "boolean", True, "true", "`true` to skip the replant after the next harvest, `false` to cancel."),
+                ],
+                sample_response={"ok": True, "farm": {"plots": [{"slot": 0, "can_stop_replant": True, "stop_after_harvest": True}]}},
+            ),
             endpoint(
                 id="game-buy-plot",
                 method="POST",

devplacepy/docs_api/groups/nodes.py

@@ -386,6 +386,22 @@ page. Every endpoint follows the shared [Conventions & Errors](/docs/conventions
             sample_response={"ok": True, "redirect": "/admin/nodes", "data": {"uid": "DEVICE_UID", "minutes": 60}},
             notes=["Refused (audited as `denied`) when the device owner is a more senior administrator. Audited as `node.suspend`."],
         ),
+        endpoint(
+            id="admin-nodes-class",
+            method="POST",
+            path="/admin/nodes/devices/{device_uid}/class",
+            title="Override a device's address class",
+            summary="Let one donor device serve jobs although its address is not residential or mobile.",
+            auth="admin",
+            encoding="form",
+            destructive=True,
+            params=[
+                field("device_uid", "path", "string", True, "DEVICE_UID", "Device uid."),
+                field("value", "form", "string", True, "allow", "`allow` to override, `clear` to remove the override."),
+            ],
+            sample_response={"ok": True, "redirect": "/admin/nodes", "data": {"uid": "DEVICE_UID", "class_override": "allow"}},
+            notes=["Refused (audited as `denied`) when the device owner is a more senior administrator. Audited as `node.device.override`."],
+        ),
         endpoint(
             id="admin-nodes-unsuspend",
             method="POST",

devplacepy/docs_api/groups/sync.py

@@ -13,11 +13,12 @@ RESULT_SAMPLE = {
     "kind": "posts",
     "dry_run": False,
     "total": 2,
-    "counts": {"created": 1, "updated": 0, "unchanged": 0, "stale": 0, "rejected": 1, "failed": 0},
+    "counts": {"created": 1, "updated": 0, "unchanged": 0, "stale": 0, "rejected": 0, "failed": 0, "deferred": 1},
     "results": [
-        {"index": 0, "origin_uid": "p-1", "status": "created", "reason": "", "local_uid": "LOCAL_UID", "errors": [], "ignored": ["stars"]},
-        {"index": 1, "origin_uid": "p-2", "status": "rejected", "reason": "unknown_author", "local_uid": "", "errors": ["author u-9 is not synced yet, send users first"], "ignored": []},
+        {"index": 0, "origin_uid": "p-1", "status": "created", "reason": "", "local_uid": "LOCAL_UID", "errors": [], "ignored": ["stars"], "needs_kind": "", "needs_uid": ""},
+        {"index": 1, "origin_uid": "p-2", "status": "deferred", "reason": "unknown_author", "local_uid": "", "errors": ["author u-9 is not synced yet, send users first"], "ignored": [], "needs_kind": "users", "needs_uid": "u-9"},
     ],
+    "released": {"created": 3},
 }
 
 GROUP = {
@@ -27,7 +28,7 @@ GROUP = {
     "intro": """
 # Community sync
 
-Copy users, posts and comments from another community into this one. The API is idempotent
+Copy users, projects, posts, comments, polls, votes, reactions, follows, poll ballots and attachments from another community into this one, and pull this community's own public data out. The API is idempotent
 and safe to repeat: every record carries the id it has in the **origin** community
 (`origin_uid`), the server remembers which local row that id maps to, and a repeated record is
 an update or a no-op, never a duplicate.
@@ -35,16 +36,26 @@ an update or a no-op, never a duplicate.
 - **Validated and normalized here.** The sender posts raw data; the server fixes what can be
   fixed (usernames, whitespace, control characters, unknown topics, timestamps) and rejects what
   cannot, record by record. One bad record never fails the batch.
-- **Counters are not imported.** Votes, stars, XP and levels cannot be verified across
-  communities, so they are ignored and listed under `ignored` in the result.
+- **Counters are never sent as numbers.** Stars, XP and levels cannot be verified across
+  communities, so they are ignored and listed under `ignored` in the result. A post or project
+  gets stars only from synced votes, and a vote needs a synced voter and a synced target.
 - **No side effects.** Synced content sends no notifications, awards no XP and runs no AI
   correction. Content that matches the content filter is rejected.
-- **Order matters.** Send users first, then posts, then comments (parents before replies).
+- **Order is forgiven, not required.** Send users, projects, posts, comments, polls, then the
+  engagement kinds. A record whose author, post, parent, target, poll or project is not synced
+  yet is answered `deferred` with `needs_kind` and `needs_uid`, parked for seven days (at most
+  5000 per origin) and applied by itself when that record arrives; the response then lists
+  `released` counts.
 - **Off by default.** An administrator enables it at `/admin/settings` (`sync_enabled`) and may
   restrict senders with `sync_origins` (comma separated; empty allows any origin).
 - **JSON or form.** The body is JSON. A form-encoded body is accepted too, with `records` as a
-  JSON array string; Devii uses that form through the `sync_users`, `sync_posts` and
-  `sync_comments` tools.
+  JSON array string; Devii uses that form through the `sync_<kind>` tools.
+- **Pull feed.** `GET /sync/export/{kind}` pages out this community's own users, projects,
+  posts, comments, polls, follows and attachments. Votes, reactions and poll ballots are never
+  exported because who voted is not public, and rows that were themselves synced in are never
+  echoed back.
+- **Deletions are reported.** `GET /sync/deletions` lists synced records deleted or purged here
+  and `POST /sync/deletions/ack` stops them being reported.
 
 The accepted fields and every limit are published at `/sync/spec.json`. Every endpoint follows
 the shared [Conventions & Errors](/docs/conventions.html).
@@ -60,8 +71,8 @@ the shared [Conventions & Errors](/docs/conventions.html).
             interactive=True,
             sample_response={
                 "limits": {"max_batch_records": 50},
-                "kinds": ["users", "posts", "comments"],
-                "statuses": ["created", "updated", "unchanged", "stale", "rejected", "failed"],
+                "kinds": ["users", "projects", "posts", "comments", "polls", "votes", "reactions", "follows", "poll_votes", "attachments"],
+                "statuses": ["created", "updated", "unchanged", "stale", "rejected", "failed", "deferred"],
                 "reasons": ["invalid", "unknown_author"],
                 "records": {"users": {"title": "UserRecord", "type": "object"}},
             },
@@ -115,5 +126,163 @@ the shared [Conventions & Errors](/docs/conventions.html).
                 "A parent that belongs to a different post is rejected with `parent_mismatch`.",
             ],
         ),
+        endpoint(
+            id="sync-projects",
+            method="POST",
+            path="/sync/projects",
+            title="Upsert projects",
+            summary="Create or update public, writable projects. The author must already be synced.",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=ENVELOPE_PARAMS,
+            sample_response=RESULT_SAMPLE,
+            notes=[
+                "Record fields: `origin_uid`, `author_uid`, `title`, `description`, `created_at`, optional `project_type` (game, game_asset, software, mobile_app, website), `status`, `platforms`, `website_url`, `repo_url`, `updated_at`, `deleted`.",
+                "Posts link to a project with `project_uid`.",
+            ],
+        ),
+        endpoint(
+            id="sync-polls",
+            method="POST",
+            path="/sync/polls",
+            title="Upsert polls",
+            summary="Create or update the poll of a synced post (one poll per post, 2 to 6 options).",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=ENVELOPE_PARAMS,
+            sample_response=RESULT_SAMPLE,
+            notes=[
+                "Record fields: `origin_uid`, `author_uid`, `post_uid`, `question`, `options` (a list of `{origin_uid, label}`), `created_at`, optional `deleted`.",
+                "Editing the options adds, renames and removes them; ballots for a removed option stay.",
+            ],
+        ),
+        endpoint(
+            id="sync-votes",
+            method="POST",
+            path="/sync/votes",
+            title="Upsert votes",
+            summary="Create, change or clear a vote of a synced user on a synced post, comment or project.",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=ENVELOPE_PARAMS,
+            sample_response=RESULT_SAMPLE,
+            notes=[
+                "Record fields: `origin_uid`, `voter_uid`, `target_kind` (post, comment, project), `target_uid`, `value` (1 or -1), `created_at`, optional `deleted` to clear.",
+                "A second origin id for the same voter and target revives the same row. Stars of posts and projects are recomputed from the live synced votes.",
+            ],
+        ),
+        endpoint(
+            id="sync-reactions",
+            method="POST",
+            path="/sync/reactions",
+            title="Upsert reactions",
+            summary="Add or remove an emoji reaction of a synced user.",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=ENVELOPE_PARAMS,
+            sample_response=RESULT_SAMPLE,
+            notes=["Record fields: `origin_uid`, `author_uid`, `target_kind`, `target_uid`, `emoji` (one emoji), `created_at`, optional `deleted`."],
+        ),
+        endpoint(
+            id="sync-follows",
+            method="POST",
+            path="/sync/follows",
+            title="Upsert follows",
+            summary="Create or remove a follow between two synced users. No notification, XP or badge is produced.",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=ENVELOPE_PARAMS,
+            sample_response=RESULT_SAMPLE,
+            notes=["Record fields: `origin_uid`, `follower_uid`, `following_uid`, `created_at`, optional `deleted`. Following yourself is rejected with `self_follow`."],
+        ),
+        endpoint(
+            id="sync-poll-votes",
+            method="POST",
+            path="/sync/poll_votes",
+            title="Upsert poll ballots",
+            summary="Record a synced user's ballot in a synced poll (one per voter; changing the option updates it).",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=ENVELOPE_PARAMS,
+            sample_response=RESULT_SAMPLE,
+            notes=["Record fields: `origin_uid`, `voter_uid`, `poll_uid`, `option_uid`, `created_at`, optional `deleted`. An option that does not exist in that poll is rejected with `unknown_option`."],
+        ),
+        endpoint(
+            id="sync-attachments",
+            method="POST",
+            path="/sync/attachments",
+            title="Upsert attachments",
+            summary="Store a file for a synced post, comment or project. The sender downloads the file and sends its bytes.",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=ENVELOPE_PARAMS,
+            sample_response=RESULT_SAMPLE,
+            notes=[
+                "Record fields: `origin_uid`, `author_uid`, `target_kind`, `target_uid`, `filename`, `content_b64` (strict base64, at most 5 MB), `created_at`, optional `deleted`.",
+                "The normal upload rules apply (allowed types, size). An attachment cannot change: a known `origin_uid` with different bytes is rejected with `immutable`; only a tombstone or a restore is accepted.",
+            ],
+        ),
+        endpoint(
+            id="sync-export",
+            method="GET",
+            path="/sync/export/{kind}",
+            title="Export own records",
+            summary="Page out this community's users, projects, posts, comments, polls, follows or attachments in sync record form.",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=[
+                field("kind", "path", "string", True, "posts", "users, projects, posts, comments, polls, follows or attachments."),
+                field("since", "query", "string", False, "", "Only records changed after this ISO 8601 instant. Users have no change time, so for them it filters by sign-up time."),
+                field("cursor", "query", "string", False, "", "The `next_cursor` of the previous page."),
+                field("limit", "query", "integer", False, "50", "Records per page, 1 to 50."),
+            ],
+            sample_response={"kind": "posts", "count": 1, "has_more": False, "next_cursor": "", "records": [{"origin_uid": "POST_UID", "author_uid": "USER_UID", "content": "...", "deleted": False}]},
+            notes=[
+                "Only public data: no email, password hash, role or API key; private projects are left out; attachments carry a `url` instead of bytes.",
+                "Votes, reactions and poll ballots are never exported. Rows that arrived by sync are never exported, so two communities cannot echo each other's data.",
+                "Soft-deleted rows appear as records with `deleted: true`. Audited as `sync.export`.",
+            ],
+        ),
+        endpoint(
+            id="sync-deletions",
+            method="GET",
+            path="/sync/deletions",
+            title="List local deletions",
+            summary="Synced records of one origin that were deleted or purged on this side and not yet acknowledged.",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=[
+                field("origin", "query", "string", True, "other.example.org", "The sending community's origin id."),
+                field("cursor", "query", "string", False, "", "The `next_cursor` of the previous page."),
+                field("limit", "query", "integer", False, "50", "Entries per page, 1 to 50."),
+            ],
+            sample_response={"origin": "other.example.org", "count": 1, "has_more": False, "next_cursor": "", "items": [{"kind": "posts", "origin_uid": "p-1", "by": "member", "deleted_at": "2026-10-16T10:00:00+00:00"}]},
+            notes=["`by` is `purged` (the row is gone), `system` or `member`. A deletion made by the sync itself is never listed."],
+        ),
+        endpoint(
+            id="sync-deletions-ack",
+            method="POST",
+            path="/sync/deletions/ack",
+            title="Acknowledge local deletions",
+            summary="Tell this community that the origin has seen the deletions, so they are no longer listed or reported.",
+            auth="admin",
+            encoding="json",
+            interactive=False,
+            params=[
+                field("origin", "body", "string", True, "other.example.org", "The sending community's origin id."),
+                field("items", "body", "array", True, "[]", "1-50 objects `{kind, origin_uid}`."),
+            ],
+            sample_response={"origin": "other.example.org", "acknowledged": 1},
+            notes=["After acknowledging, a re-sent record answers `unchanged` with reason `acknowledged_deleted`. Audited as `sync.deletions.ack`."],
+        ),
     ],
 }

devplacepy/models.py

@@ -333,6 +333,7 @@ class CommunityAddForm(BaseModel):
     login_username: str = Field(default="", max_length=200)
     login_email: str = Field(default="", max_length=254)
     login_password: str = Field(default="", max_length=500)
+    login_totp_secret: str = Field(default="", max_length=200)
 
     @field_validator("query")
     @classmethod
@@ -340,12 +341,19 @@ class CommunityAddForm(BaseModel):
         return value.strip()
 
 
+class CommunitySchemaForm(BaseModel):
+    schema_text: str = Field(min_length=2, max_length=300_000, alias="schema")
+
+    model_config = {"populate_by_name": True}
+
+
 class CommunityUpdateForm(BaseModel):
     interval_seconds: int = Field(ge=900, le=2592000)
     enabled: bool = False
     login_username: str = Field(default="", max_length=200)
     login_email: str = Field(default="", max_length=254)
     login_password: str = Field(default="", max_length=500)
+    login_totp_secret: str = Field(default="", max_length=200)
     clear_login: bool = False
     alias: Optional[bool] = None
     translate_to: Optional[str] = Field(default=None, max_length=12)
@@ -353,6 +361,7 @@ class CommunityUpdateForm(BaseModel):
     min_comments: Optional[int] = Field(default=None, ge=0, le=1000)
     topics: Optional[str] = Field(default=None, max_length=500)
     jitter_percent: Optional[int] = Field(default=None, ge=0, le=50)
+    collect_attachments: Optional[bool] = None
 
 
 class CommunityRunForm(BaseModel):
@@ -793,6 +802,11 @@ class GameSlotForm(BaseModel):
     slot: int = Field(ge=0, le=64)
 
 
+class GamePlotStopForm(BaseModel):
+    slot: int = Field(ge=0, le=64)
+    stop: bool
+
+
 class GamePerkForm(BaseModel):
     perk: str = Field(min_length=1, max_length=40)
 
@@ -1206,6 +1220,12 @@ class AdminNodeSettingsForm(BaseModel):
     nodes_cache_ttl_seconds: Optional[int] = Field(default=None, ge=0, le=NODE_SETTING_FORM_MAX)
     nodes_social_via_nodes: Optional[Literal["0", "1"]] = None
     nodes_daily_social_user: Optional[int] = Field(default=None, ge=0, le=NODE_SETTING_FORM_MAX)
+    nodes_allow_unknown_class: Optional[Literal["0", "1"]] = None
+    nodes_redundancy_percent: Optional[int] = Field(default=None, ge=0, le=NODE_SETTING_FORM_MAX)
+
+
+class AdminNodeClassForm(BaseModel):
+    value: Literal["allow", "clear"]
 
 
 class AdminNodeSuspendForm(BaseModel):

devplacepy/routers/CLAUDE.md

@@ -49,7 +49,7 @@ Prefixes are wired in `main.py`:
 | `/dbapi` | dbapi/ package - **primary-administrator-only, strictly READ-ONLY** generic database API over `dataset` (never inserts/updates/deletes/restores; the write surface was removed because it bypassed every per-route admin safeguard). `tables.py` (`GET /tables`, `GET /{table}/schema`), `crud.py` (read only: `GET /{table}` + `GET /{table}/{key}/{value}`, `?include_deleted`), `query.py` (`POST /query` hard SELECT-only validated read-only run; `POST /query/async` + `GET /query/{uid}` + `WS /query/{uid}/ws` via `DbApiJobService` kind `dbquery`), `nl.py` (`POST /nl` natural-language->SQL via the AI gateway, returns validated SELECT, `execute=true` runs it read-only). **Auth = the PRIMARY administrator only** (the earliest-created Admin, `utils.is_primary_admin` / `database.get_primary_admin_uid` - the same identity that gates backup downloads) by session/api_key (`services/dbapi/policy.py`); every other administrator is refused like a member, and there is **no internal-key path** - the gateway `internal_gateway_key()` is NOT accepted (no service-to-service access). SQL validated by `services/dbapi/validate.py` (sqlglot classify + suspicious-flag + read-only `EXPLAIN` dry-run). Devii tools `db_*` are read-only (list/get/query/nl; no write tools) and flagged `requires_primary_admin=True`, so they are added to the LLM tool list ONLY for the primary administrator (every other session never sees them and Devii is unaware they exist). `services/pubsub/policy.py` resolves its own admin/internal actor and does NOT reuse `dbapi.policy.caller_for`, so the primary-admin restriction does not leak onto the pub/sub bus. See `devplacepy/services/dbapi/CLAUDE.md` |
 | `/quizzes` | quizzes/ package - **Quizzes**: `index.py` (hub, create, import, detail, export, edit, delete, publish, per-quiz leaderboard, cross-quiz scoreboard), `questions.py` (builder page + question CRUD + reorder), `attempts.py` (start/resume, play, answer, finish, results). The hub `GET /quizzes` is the approved three-column `.feed-layout` reused verbatim from `/feed`: filters + search left (`?filter=all|todo|done|mine|drafts`, `?search=`, `?page=`), the quiz list centre with the **New quiz** and **Create quiz with Devii** actions, and the cross-quiz **scoreboard** rail right (score per user, best attempt per quiz, 15s display cache). Publishing is terminal - every write on a published quiz is a 400 and there is no unpublish route. See `devplacepy/services/quiz/CLAUDE.md` |
 | `/battles` | battles.py - **Opinion Wars**, week-long two-faction battles attached to posts: `GET ""` (listing, quizzes-style filters active/ended/mine + search), `GET /{uid}` (JSON-only full state, runs lazy resolution), `GET /{uid}/events?after=` (durable event trail replay), `POST /{uid}/join` (join or switch faction), `POST /{uid}/fight` (spend 25 Code Farm coins, deal level-weighted damage, 24h cooldown). The battle card renders on the parent post (no battle detail HTML page). See `devplacepy/services/opinionwar/CLAUDE.md` |
-| `/game` | game/ package - the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard?board=`, `POST /game/{plant,harvest,buy-plot,upgrade,fertilize,daily,grant,perk,prestige,legacy,mastery,quests/claim}`, `POST /game/{defense/upgrade,defense/downgrade,infrastructure/buy,cosmetics/buy,cosmetics/equip}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}`. See `devplacepy/services/game/CLAUDE.md` |
+| `/game` | game/ package - the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard?board=`, `POST /game/{plant,harvest,plot/stop,buy-plot,upgrade,fertilize,daily,grant,perk,prestige,legacy,mastery,quests/claim}`, `POST /game/{defense/upgrade,defense/downgrade,infrastructure/buy,cosmetics/buy,cosmetics/equip}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}`. See `devplacepy/services/game/CLAUDE.md` |
 | (none) | push.py - push + PWA: `GET /push.json` (VAPID public key + the providers accepting registrations; `apns` includes `environment` when active), `POST /push.json` (register with any active provider; a body without `provider` is a `webpush` body; an APNs body is `token` plus optional `client_id` for device-stable upsert; created/revived rows are probed and the JSON may include `delivered`/`error`; an unknown, disabled or unconfigured provider is a 400), `DELETE /push.json` (unregister exactly the one registration named by `endpoint`/`token`/`client_id`, same identity priority as registration; idempotent, always 200 `{unregistered}`; wired into `PushManager.js`'s logout-link interceptor so a webpush subscription is dropped before the browser navigates to `/auth/logout`), `GET /service-worker.js`, `GET /manifest.json`. Provider protocol and delivery live in `devplacepy/push/` - see `devplacepy/push/CLAUDE.md` |
 | (none) | docs.py (docs/ package) - the documentation site (prose pages + API reference). See `routers/docs/CLAUDE.md` for the deep detail on this tree (`DOCS_PAGES`, audience tiers, prose rendering pipeline) |
 | `/skills` | skills.py - per-user Agent Skills: `GET ""` (list), `GET /{name}` (detail), `POST /install` (fetch a `SKILL.md` from a URL, update in place), `POST /{name}/enabled`, `POST /{name}/delete` (owner, or an administrator naming `owner`). Private to the owner, `noindex`. See `devplacepy/services/skills/CLAUDE.md` |

devplacepy/routers/admin/communities.py

@@ -1,6 +1,7 @@
 # retoor <retoor@molodetz.nl>
 
 import asyncio
+import json
 import logging
 from datetime import datetime
 from typing import Annotated
@@ -12,8 +13,8 @@ from fastapi.responses import HTMLResponse
 from devplacepy.database import get_setting, get_table, get_users_by_uids, resolve_object_url
 from devplacepy.database.sync import count_links, identity_map
 from devplacepy.responses import action_result, json_error, respond
-from devplacepy.schemas import AdminCommunitiesOut, AdminCommunityLedgerOut, AdminCommunityOut
-from devplacepy.models import CommunityAddForm, CommunityRunForm, CommunityUpdateForm
+from devplacepy.schemas import AdminCommunitiesOut, AdminCommunityLedgerOut, AdminCommunityOut, AdminCommunitySchemaOut
+from devplacepy.models import CommunityAddForm, CommunityRunForm, CommunitySchemaForm, CommunityUpdateForm
 from devplacepy.seo import base_seo_context, site_url, website_schema
 from devplacepy.services.audit import record as audit
 from devplacepy.services.community.discoverer import slug_for_query
@@ -31,7 +32,8 @@ from devplacepy.services.community.ledger import (
 from devplacepy.services.community.ledger_store import ledger_store
 from devplacepy.services.community.runner import TRIGGER_ADMIN, CommunityRunner
 from devplacepy.services.community.service import DRY_RUN_KEY
-from devplacepy.services.community.spec import STATUS_LEARNING
+from devplacepy.services.community.schemacheck import validate_schema
+from devplacepy.services.community.spec import STATUS_LEARNING, STATUS_READY
 from devplacepy.services.community.store import iso, now, store
 from devplacepy.services.sync.gate import sync_enabled
 from devplacepy.services.sync.options import SyncOptions
@@ -44,10 +46,11 @@ background_jobs: set[asyncio.Task] = set()
 LIST_URL = "/admin/communities"
 
 
-def spawn(coroutine) -> None:
+async def spawn(coroutine) -> None:
     task = asyncio.create_task(coroutine)
     background_jobs.add(task)
     task.add_done_callback(background_jobs.discard)
+    await asyncio.sleep(0)
 
 
 def service_flags() -> dict:
@@ -58,8 +61,9 @@ def service_flags() -> dict:
 
 
 def public(row: dict) -> dict:
-    view = {key: value for key, value in row.items() if key != "login_password"}
+    view = {key: value for key, value in row.items() if key not in ("login_password", "login_totp_secret")}
     view["has_login"] = bool(row.get("login_password"))
+    view["has_totp"] = bool(row.get("login_totp_secret"))
     return view
 
 
@@ -71,7 +75,7 @@ def require_community(slug: str) -> dict:
 
 
 def credentials_of(data) -> dict:
-    return {"username": data.login_username.strip(), "email": data.login_email.strip(), "password": data.login_password}
+    return {"username": data.login_username.strip(), "email": data.login_email.strip(), "password": data.login_password, "totp": data.login_totp_secret.strip()}
 
 
 def seo(request: Request, title: str, path: str, crumb: str):
@@ -207,8 +211,9 @@ async def admin_community_add(request: Request, data: Annotated[CommunityAddForm
             login_username=secrets["username"],
             login_email=secrets["email"],
             login_password=secrets["password"],
+            login_totp_secret=secrets["totp"],
         )
-        spawn(CommunityRunner().discover(data.query, admin["uid"], relearn_slug=slug))
+        await spawn(CommunityRunner().discover(data.query, admin["uid"], relearn_slug=slug))
         audit.record(
             request,
             "admin.community.add",
@@ -226,6 +231,7 @@ async def admin_community_add(request: Request, data: Annotated[CommunityAddForm
 async def admin_community(request: Request, slug: str):
     admin = require_admin(request)
     row = require_community(slug)
+    store.reap_interrupted()
     source = files.read(row["slug"])
     return respond(
         request,
@@ -289,6 +295,7 @@ async def admin_community_ledger(
             "run": run_row,
             "run_errors": (run_row or {}).get("errors") or [],
             "run_log": (run_row or {}).get("log") or "",
+            "busy": (run_row["status"] == "running") if run_row else store.busy(row["uid"]),
             "verdict": {"level": run_row["verdict"], "title": LEVEL_TITLES.get(run_row["verdict"], ""), "headline": run_row["headline"]} if run_row else {},
             "tiles": [
                 {"key": o.key, "label": o.label, "tone": o.tone, "help": o.help, "count": int(exact.get(o.key, 0)), "listed": int(listed.get(o.key, 0))}
@@ -311,7 +318,7 @@ async def admin_community_run(request: Request, slug: str, data: Annotated[Commu
     row = require_community(slug)
     if store.busy(row["uid"]):
         return json_error(409, "A run is already in progress for this community")
-    spawn(CommunityRunner().run(row, dry_run=data.dry_run, trigger=TRIGGER_ADMIN))
+    await spawn(CommunityRunner().run(row, dry_run=data.dry_run, trigger=TRIGGER_ADMIN))
     audit.record(
         request,
         "admin.community.run",
@@ -331,7 +338,7 @@ async def admin_community_relearn(request: Request, slug: str):
     row = require_community(slug)
     if store.busy(row["uid"]):
         return json_error(409, "A run is already in progress for this community")
-    spawn(CommunityRunner().discover(row["query"] or row["url"], admin["uid"], relearn_slug=slug))
+    await spawn(CommunityRunner().discover(row["query"] or row["url"], admin["uid"], relearn_slug=slug))
     audit.record(
         request,
         "admin.community.relearn",
@@ -350,9 +357,9 @@ async def admin_community_update(request: Request, slug: str, data: Annotated[Co
     row = require_community(slug)
     changes = {"interval_seconds": data.interval_seconds, "enabled": int(data.enabled)}
     if data.clear_login:
-        changes.update(login_username="", login_email="", login_password="")
+        changes.update(login_username="", login_email="", login_password="", login_totp_secret="")
     else:
-        for column, value in (("login_username", data.login_username.strip()), ("login_email", data.login_email.strip()), ("login_password", data.login_password)):
+        for column, value in (("login_username", data.login_username.strip()), ("login_email", data.login_email.strip()), ("login_password", data.login_password), ("login_totp_secret", data.login_totp_secret.strip())):
             if value:
                 changes[column] = value
     submitted = {name: getattr(data, name) for name in SyncOptions.__dataclass_fields__ if getattr(data, name) is not None}
@@ -372,12 +379,104 @@ async def admin_community_update(request: Request, slug: str, data: Annotated[Co
         target_type="community",
         target_uid=row["uid"],
         target_label=slug,
-        metadata={"interval_seconds": data.interval_seconds, "enabled": data.enabled, "sync_options": changes.get("sync_options"), "login_changed": data.clear_login or bool(data.login_password or data.login_username or data.login_email)},
+        metadata={"interval_seconds": data.interval_seconds, "enabled": data.enabled, "sync_options": changes.get("sync_options"), "login_changed": data.clear_login or bool(data.login_password or data.login_username or data.login_email or data.login_totp_secret)},
         summary=f"{admin['username']} updated community {slug}",
     )
     return action_result(request, f"{LIST_URL}/{slug}")
 
 
+def schema_text(schema) -> str:
+    return json.dumps(schema, indent=2, ensure_ascii=False)
+
+
+def schema_page(request: Request, admin: dict, row: dict, text: str, errors: list[str], notice: str, status_code: int):
+    context = {
+        **seo(request, f"{row['name'] or row['slug']} schema", f"{LIST_URL}/{row['slug']}/schema", "Communities"),
+        "request": request,
+        "user": admin,
+        "community": public(row),
+        "schema_text": text,
+        "has_previous": bool(row.get("schema_previous")),
+        "errors": errors,
+        "notice": notice,
+        "admin_section": "communities",
+    }
+    response = respond(request, "admin_community_schema.html", context, model=AdminCommunitySchemaOut)
+    response.status_code = status_code
+    return response
+
+
+@router.get("/communities/{slug}/schema", response_class=HTMLResponse)
+async def admin_community_schema(request: Request, slug: str):
+    admin = require_admin(request)
+    row = require_community(slug)
+    return schema_page(request, admin, row, schema_text(row.get("schema") or {}), [], "", 200)
+
+
+@router.post("/communities/{slug}/schema")
+async def admin_community_schema_save(request: Request, slug: str, data: Annotated[CommunitySchemaForm, Form()]):
+    admin = require_admin(request)
+    row = require_community(slug)
+    if store.busy(row["uid"]):
+        return json_error(409, "A run is already in progress for this community")
+    try:
+        schema = json.loads(data.schema_text)
+    except ValueError as exc:
+        return schema_page(request, admin, row, data.schema_text, [f"The text is not valid JSON: {exc}"], "", 400)
+    errors = validate_schema(schema)
+    if errors:
+        return schema_page(request, admin, row, data.schema_text, errors, "", 400)
+    verdict = await CommunityRunner().verify_schema(row, schema)
+    if not verdict.ok:
+        failed = ["The schema was checked against the live source and was not saved."] + verdict.problems[:10]
+        return schema_page(request, admin, row, data.schema_text, failed, "", 422)
+    store.update(
+        row["uid"],
+        schema=schema,
+        schema_previous=row.get("schema") or {},
+        status=STATUS_READY,
+        last_error="",
+        failure_count=0,
+        enabled=1,
+    )
+    files.write(row["slug"], schema)
+    audit.record(
+        request,
+        "admin.community.schema",
+        user=admin,
+        target_type="community",
+        target_uid=row["uid"],
+        target_label=slug,
+        metadata={"feeds": len(schema["feeds"]), "sample_posts": verdict.posts, "sample_comments": verdict.comments},
+        summary=f"{admin['username']} replaced the learned schema of community {slug}",
+    )
+    return action_result(request, f"{LIST_URL}/{slug}/schema", data={"posts": verdict.posts, "comments": verdict.comments})
+
+
+@router.post("/communities/{slug}/schema/restore")
+async def admin_community_schema_restore(request: Request, slug: str):
+    admin = require_admin(request)
+    row = require_community(slug)
+    previous = row.get("schema_previous") or {}
+    if not previous:
+        return json_error(409, "There is no previous schema to restore")
+    errors = validate_schema(previous)
+    if errors:
+        return schema_page(request, admin, row, schema_text(previous), errors, "", 400)
+    store.update(row["uid"], schema=previous, schema_previous=row.get("schema") or {})
+    files.write(row["slug"], previous)
+    audit.record(
+        request,
+        "admin.community.schema_restore",
+        user=admin,
+        target_type="community",
+        target_uid=row["uid"],
+        target_label=slug,
+        summary=f"{admin['username']} restored the previous schema of community {slug}",
+    )
+    return action_result(request, f"{LIST_URL}/{slug}/schema")
+
+
 @router.post("/communities/{slug}/delete")
 async def admin_community_delete(request: Request, slug: str):
     admin = require_admin(request)

devplacepy/routers/admin/nodes.py

@@ -17,24 +17,35 @@ from devplacepy.database.nodes import (
     add_event,
     count_open_requests,
     device_breakdown,
+    cache_stats,
     effective_status,
     get_device,
+    lifetime_counters,
     list_flags,
+    load_circuits,
     recent_devices,
     request_status_counts,
     resolve_flag,
+    set_class_override,
     suspend_device,
     unsuspend_device,
     usage_day_totals,
 )
 from devplacepy.dependencies import json_or_form
-from devplacepy.models import AdminNodeKillswitchForm, AdminNodeSettingsForm, AdminNodeSuspendForm
+from devplacepy.models import (
+    AdminNodeClassForm,
+    AdminNodeKillswitchForm,
+    AdminNodeSettingsForm,
+    AdminNodeSuspendForm,
+)
 from devplacepy.responses import action_result, json_error, respond
 from devplacepy.routers.admin._shared import deny_senior, is_senior_admin
 from devplacepy.routers.nodes.social import SOCIAL_DAILY_DEFAULT
 from devplacepy.schemas import (
+    AdminNodeCircuitOut,
     AdminNodeDeviceOut,
     AdminNodeFlagOut,
+    AdminNodeOutcomeOut,
     AdminNodePoolOut,
     AdminNodeQueueOut,
     AdminNodeSettingOut,
@@ -43,6 +54,9 @@ from devplacepy.schemas import (
 from devplacepy.seo import base_seo_context, site_url, website_schema
 from devplacepy.services.audit import record as audit
 from devplacepy.services.base import BaseService
+from devplacepy.services.nodes.circuit import CircuitBreaker
+from devplacepy.services.nodes.geo import shared_geo
+from devplacepy.services.nodes.registry import OUTCOME_PREFIX
 from devplacepy.utils import not_found, require_admin
 
 logger = logging.getLogger(__name__)
@@ -81,6 +95,8 @@ SETTING_FIELDS: tuple[NodeSettingField, ...] = (
     NodeSettingField("nodes_cache_ttl_seconds", "Result cache lifetime (seconds)", LIMITS.cache_ttl_seconds, LIMITS.cache_ttl_seconds),
     NodeSettingField("nodes_daily_social_user", "Daily social searches per member", SOCIAL_DAILY_DEFAULT, QUOTA_SOCIAL_CEILING),
     NodeSettingField("nodes_social_via_nodes", "Route social search through nodes", 0, None, "flag"),
+    NodeSettingField("nodes_allow_unknown_class", "Allow donors whose address class is unknown", 0, None, "flag"),
+    NodeSettingField("nodes_redundancy_percent", "Redundancy sample of fetch jobs (percent)", LIMITS.redundancy_percent, LIMITS.redundancy_max_percent),
 )
 SETTING_FIELDS_BY_KEY = {field.key: field for field in SETTING_FIELDS}
 
@@ -151,6 +167,8 @@ def device_views(rows: list[dict], owners: dict) -> list[AdminNodeDeviceOut]:
             suspend_reason=row.get("suspend_reason") or "",
             country=row.get("country") or "",
             asn_class=row.get("asn_class") or "unknown",
+            asn=row.get("asn") or "",
+            class_override=row.get("class_override") or "",
             last_seen_at=row.get("last_seen_at") or None,
             created_at=row.get("created_at"),
             owner_uid=row.get("user_uid") or "",
@@ -177,7 +195,44 @@ def flag_views(rows: list[dict], owners: dict, devices: dict) -> list[AdminNodeF
     ]
 
 
+def circuit_views() -> list[AdminNodeCircuitOut]:
+    now = datetime.now(timezone.utc).timestamp()
+    breaker = CircuitBreaker()
+    views = []
+    for row in load_circuits():
+        age = max(0.0, now - float(row.get("opened_at") or 0.0))
+        failures = int(row.get("failures") or 0)
+        if failures < breaker.failures:
+            state, left = "closed", 0
+        elif age < breaker.open_seconds:
+            state, left = "open", int(breaker.open_seconds - age)
+        else:
+            state, left = "half_open", 0
+        views.append(AdminNodeCircuitOut(key=row["key"], state=state, failures=failures, seconds_left=left))
+    return views
+
+
+def outcome_views(counters: dict[str, int]) -> list[AdminNodeOutcomeOut]:
+    grouped: dict[tuple[str, str], dict[str, int]] = {}
+    for name, value in counters.items():
+        _, adapter, asn_class, kind = name.split(":")
+        grouped.setdefault((adapter, asn_class), {})[kind] = value
+    return [
+        AdminNodeOutcomeOut(
+            adapter=adapter,
+            asn_class=asn_class,
+            ok=kinds.get("ok", 0),
+            blocked=kinds.get("blocked", 0),
+            failed=kinds.get("failed", 0),
+            total=sum(kinds.values()),
+        )
+        for (adapter, asn_class), kinds in sorted(grouped.items())
+    ]
+
+
 def overview() -> dict:
+    lifetime = lifetime_counters()
+    outcomes = {name: value for name, value in lifetime.items() if name.startswith(OUTCOME_PREFIX)}
     devices = recent_devices(ADMIN_LIST_LIMIT)
     flags = list_flags()[:ADMIN_LIST_LIMIT]
     device_by_uid = {row["uid"]: row for row in devices}
@@ -199,6 +254,11 @@ def overview() -> dict:
         "devices": device_views(devices, owners),
         "settings": settings_view(),
         "limits": LIMITS.describe(),
+        "circuits": circuit_views(),
+        "outcomes": outcome_views(outcomes),
+        "lifetime": {name: value for name, value in lifetime.items() if name not in outcomes},
+        "geo": shared_geo().status(),
+        "cache": cache_stats(),
         "admin_section": "nodes",
     }
 
@@ -328,6 +388,39 @@ async def admin_nodes_suspend(
     return action_result(request, LIST_URL, data={"uid": device_uid, "minutes": data.minutes})
 
 
+@router.post("/nodes/devices/{device_uid}/class")
+async def admin_nodes_class(
+    request: Request,
+    device_uid: str,
+    data: Annotated[AdminNodeClassForm, Depends(json_or_form(AdminNodeClassForm))],
+):
+    admin = require_admin(request)
+    device = require_device(device_uid)
+    owner = owner_of(device)
+    if is_senior_admin(admin, owner):
+        return deny_senior(request, admin, owner["uid"], owner, "node.device.override", LIST_URL)
+    value = "allow" if data.value == "allow" else ""
+    before = device.get("class_override") or ""
+    set_class_override(device_uid, value)
+    audit.record(
+        request,
+        "node.device.override",
+        user=admin,
+        target_type="node_device",
+        target_uid=device_uid,
+        target_label=device.get("label") or device_uid,
+        old_value=before,
+        new_value=value,
+        metadata={"owner_uid": device.get("user_uid"), "asn_class": device.get("asn_class") or "unknown"},
+        summary=(
+            f"admin {admin['username']} {'allowed' if value else 'cleared the override for'} "
+            f"node device {device.get('label') or device_uid} regardless of its address class"
+        ),
+        links=[audit.target("user", device.get("user_uid"), (owner or {}).get("username"))],
+    )
+    return action_result(request, LIST_URL, data={"uid": device_uid, "class_override": value})
+
+
 @router.post("/nodes/devices/{device_uid}/unsuspend")
 async def admin_nodes_unsuspend(request: Request, device_uid: str):
     admin = require_admin(request)

devplacepy/routers/game/index.py

@@ -12,6 +12,7 @@ from devplacepy.models import (
     GameMasteryForm,
     GamePerkForm,
     GamePlantForm,
+    GamePlotStopForm,
     GameQuestForm,
     GameSlotForm,
 )
@@ -98,6 +99,27 @@ async def game_plant(request: Request, data: Annotated[GamePlantForm, Form()]):
     )
 
 
+@router.post("/plot/stop")
+async def game_plot_stop(request: Request, data: Annotated[GamePlotStopForm, Form()]):
+    user = require_user(request)
+
+    def recorded(result):
+        audit.record(
+            request,
+            "game.plot.stop",
+            user=user,
+            metadata=result,
+            summary=(
+                f"{user['username']} {'paused' if result['stop'] else 'resumed'} "
+                f"auto-replant on plot {result['slot']}"
+            ),
+        )
+
+    return await _respond_action(
+        request, user, lambda: store.set_plot_stop(user, data.slot, data.stop), recorded
+    )
+
+
 @router.post("/harvest")
 async def game_harvest(request: Request, data: Annotated[GameSlotForm, Form()]):
     user = require_user(request)

devplacepy/routers/nodes/ticket.py

@@ -36,7 +36,12 @@ async def node_ticket(request: Request):
     if device is None:
         return json_error(401, "Unknown device secret")
     if effective_status(device) != STATUS_ACTIVE:
-        return json_error(403, "This device is suspended", reason="suspended")
+        return json_error(
+            403,
+            "This device is suspended",
+            reason="suspended",
+            until=(device.get("suspended_until") or "").strip(),
+        )
     verdict = GATE.allows(device["user_uid"])
     if not verdict.allowed:
         return json_error(403, verdict.message, reason=verdict.reason)

devplacepy/routers/nodes/ws.py

@@ -4,6 +4,7 @@ import logging
 
 from fastapi import APIRouter, WebSocket, WebSocketDisconnect
 
+from devplacepy.database import refresh_snapshot
 from devplacepy.database.nodes import (
     STATUS_ACTIVE,
     add_event,
@@ -16,6 +17,7 @@ from devplacepy.routers.nodes._shared import GATE, NODES_SCOPE
 from devplacepy.services.manager import service_manager
 from devplacepy.services.messaging.tickets import redeem_ticket_row
 from devplacepy.services.nodes.frames import FrameError
+from devplacepy.services.nodes.geo import shared_geo
 from devplacepy.services.nodes.session import NodeSession
 from devplacepy.services.nodes.spec import (
     CLOSE_CAPACITY,
@@ -39,6 +41,7 @@ def _address(websocket: WebSocket) -> str:
 @router.websocket("/ws")
 async def node_ws(websocket: WebSocket, ticket: str = ""):
     await websocket.accept()
+    refresh_snapshot()
     service = service_manager.get_service("nodes")
     if service is None or not service.is_enabled():
         await websocket.close(code=TRY_AGAIN_LATER)
@@ -78,12 +81,24 @@ async def node_ws(websocket: WebSocket, ticket: str = ""):
     ended = "disconnect"
     try:
         hello = await session.handshake()
+        address = _address(websocket)
+        geo = shared_geo()
+        if not geo.available():
+            counters.inc("geo_unavailable")
+            if not geo.warned:
+                geo.warned = True
+                logger.warning("node geo database is missing at %s, every donor is class unknown", geo.path)
+        found = geo.lookup(address)
+        session.asn_class = found.asn_class
         record_device_connection(
             device["uid"],
             platform=hello["platform"],
             client_version=hello["client_version"],
             protocol_version=hello["protocol"],
-            ip_hash=hash_ip(_address(websocket)),
+            country=found.country,
+            asn=str(found.asn) if found.asn else "",
+            asn_class=found.asn_class,
+            ip_hash=hash_ip(address),
         )
         add_event("connect", device_uid=device["uid"], user_uid=device["user_uid"])
         ended = await session.run()

devplacepy/routers/sync/__init__.py

@@ -2,10 +2,33 @@
 
 from fastapi import APIRouter
 
-from . import comments, posts, spec, users
+from . import (
+    attachments,
+    comments,
+    deletions,
+    export,
+    follows,
+    poll_votes,
+    polls,
+    posts,
+    projects,
+    reactions,
+    spec,
+    users,
+    votes,
+)
 
 router = APIRouter()
 router.include_router(spec.router)
 router.include_router(users.router)
+router.include_router(projects.router)
 router.include_router(posts.router)
 router.include_router(comments.router)
+router.include_router(polls.router)
+router.include_router(votes.router)
+router.include_router(reactions.router)
+router.include_router(follows.router)
+router.include_router(poll_votes.router)
+router.include_router(attachments.router)
+router.include_router(export.router)
+router.include_router(deletions.router)

devplacepy/routers/sync/attachments.py

@@ -0,0 +1,21 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, Request
+
+from devplacepy.dependencies import json_or_form
+from devplacepy.routers.sync._shared import require_sync_caller, run_batch
+from devplacepy.services.sync.records import SyncBatchForm
+from devplacepy.services.sync.spec import KIND_ATTACHMENTS
+
+router = APIRouter()
+
+
+@router.post("/attachments")
+async def sync_attachments(
+    request: Request,
+    user: Annotated[dict, Depends(require_sync_caller)],
+    data: Annotated[SyncBatchForm, Depends(json_or_form(SyncBatchForm))],
+):
+    return run_batch(request, user, KIND_ATTACHMENTS, data)

devplacepy/routers/sync/deletions.py

@@ -0,0 +1,65 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, Request
+from fastapi.responses import JSONResponse
+
+from devplacepy.dependencies import json_or_form
+from devplacepy.responses import json_error
+from devplacepy.routers.sync._shared import NO_STORE, require_sync_caller
+from devplacepy.schemas.sync import SyncAckOut, SyncDeletionsOut
+from devplacepy.services.audit import record as audit
+from devplacepy.services.sync.deletions import DeletionCursorError, acknowledge, deletions_page
+from devplacepy.services.sync.gate import sync_enabled
+from devplacepy.services.sync.normalize import normalize_origin
+from devplacepy.services.sync.records import SyncAckForm
+from devplacepy.services.sync.spec import LIMITS
+
+router = APIRouter()
+
+
+@router.get("/deletions")
+async def sync_deletions(
+    request: Request,
+    origin: str,
+    user: dict = Depends(require_sync_caller),
+    cursor: str = "",
+    limit: int = LIMITS.deletions_page_size,
+):
+    if not sync_enabled():
+        return json_error(503, "Community sync is disabled")
+    try:
+        name = normalize_origin(origin)
+        page = deletions_page(name, cursor, limit)
+    except (DeletionCursorError, ValueError) as exc:
+        return json_error(400, str(exc))
+    body = SyncDeletionsOut(
+        origin=name,
+        count=len(page.items),
+        has_more=page.has_more,
+        next_cursor=page.next_cursor,
+        items=page.items,
+    )
+    return JSONResponse(body.model_dump(mode="json"), headers=NO_STORE)
+
+
+@router.post("/deletions/ack")
+async def sync_deletions_ack(
+    request: Request,
+    user: Annotated[dict, Depends(require_sync_caller)],
+    data: Annotated[SyncAckForm, Depends(json_or_form(SyncAckForm))],
+):
+    if not sync_enabled():
+        return json_error(503, "Community sync is disabled")
+    done = acknowledge(data.origin, data.items)
+    audit.record(
+        request,
+        "sync.deletions.ack",
+        user=user,
+        target_type="sync",
+        target_label=data.origin,
+        metadata={"origin": data.origin, "items": len(data.items), "acknowledged": done},
+        summary=f"{user['username']} acknowledged {done} local deletions for {data.origin}",
+    )
+    return JSONResponse(SyncAckOut(origin=data.origin, acknowledged=done).model_dump(mode="json"), headers=NO_STORE)

devplacepy/routers/sync/export.py

@@ -0,0 +1,52 @@
+# retoor <retoor@molodetz.nl>
+
+from fastapi import APIRouter, Depends, Request
+from fastapi.responses import JSONResponse
+
+from devplacepy.responses import json_error
+from devplacepy.routers.sync._shared import NO_STORE, require_sync_caller
+from devplacepy.schemas.sync import SyncExportOut
+from devplacepy.services.audit import record as audit
+from devplacepy.services.sync.exporter import ExportError, export_page
+from devplacepy.services.sync.gate import sync_enabled
+from devplacepy.services.sync.normalize import instant_iso
+from devplacepy.services.sync.spec import EXPORT_KINDS, LIMITS
+
+router = APIRouter()
+
+
+@router.get("/export/{kind}")
+async def sync_export(
+    request: Request,
+    kind: str,
+    user: dict = Depends(require_sync_caller),
+    since: str = "",
+    cursor: str = "",
+    limit: int = LIMITS.export_page_size,
+):
+    if not sync_enabled():
+        return json_error(503, "Community sync is disabled")
+    if kind not in EXPORT_KINDS:
+        return json_error(404, f"{kind} cannot be exported", reason="not_exportable", exportable=list(EXPORT_KINDS))
+    try:
+        floor = instant_iso(since) if since else ""
+        page = export_page(kind, floor, cursor, limit)
+    except (ExportError, ValueError) as exc:
+        return json_error(400, str(exc))
+    audit.record(
+        request,
+        "sync.export",
+        user=user,
+        target_type="sync",
+        target_label=kind,
+        metadata={"kind": kind, "count": len(page.records), "since": floor, "has_more": page.has_more},
+        summary=f"{user['username']} exported {len(page.records)} {kind}",
+    )
+    body = SyncExportOut(
+        kind=kind,
+        count=len(page.records),
+        has_more=page.has_more,
+        next_cursor=page.next_cursor,
+        records=page.records,
+    )
+    return JSONResponse(body.model_dump(mode="json"), headers=NO_STORE)

devplacepy/routers/sync/follows.py

@@ -0,0 +1,21 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, Request
+
+from devplacepy.dependencies import json_or_form
+from devplacepy.routers.sync._shared import require_sync_caller, run_batch
+from devplacepy.services.sync.records import SyncBatchForm
+from devplacepy.services.sync.spec import KIND_FOLLOWS
+
+router = APIRouter()
+
+
+@router.post("/follows")
+async def sync_follows(
+    request: Request,
+    user: Annotated[dict, Depends(require_sync_caller)],
+    data: Annotated[SyncBatchForm, Depends(json_or_form(SyncBatchForm))],
+):
+    return run_batch(request, user, KIND_FOLLOWS, data)

devplacepy/routers/sync/poll_votes.py

@@ -0,0 +1,21 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, Request
+
+from devplacepy.dependencies import json_or_form
+from devplacepy.routers.sync._shared import require_sync_caller, run_batch
+from devplacepy.services.sync.records import SyncBatchForm
+from devplacepy.services.sync.spec import KIND_POLL_VOTES
+
+router = APIRouter()
+
+
+@router.post("/poll_votes")
+async def sync_poll_votes(
+    request: Request,
+    user: Annotated[dict, Depends(require_sync_caller)],
+    data: Annotated[SyncBatchForm, Depends(json_or_form(SyncBatchForm))],
+):
+    return run_batch(request, user, KIND_POLL_VOTES, data)

devplacepy/routers/sync/polls.py

@@ -0,0 +1,21 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, Request
+
+from devplacepy.dependencies import json_or_form
+from devplacepy.routers.sync._shared import require_sync_caller, run_batch
+from devplacepy.services.sync.records import SyncBatchForm
+from devplacepy.services.sync.spec import KIND_POLLS
+
+router = APIRouter()
+
+
+@router.post("/polls")
+async def sync_polls(
+    request: Request,
+    user: Annotated[dict, Depends(require_sync_caller)],
+    data: Annotated[SyncBatchForm, Depends(json_or_form(SyncBatchForm))],
+):
+    return run_batch(request, user, KIND_POLLS, data)

devplacepy/routers/sync/projects.py

@@ -0,0 +1,21 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, Request
+
+from devplacepy.dependencies import json_or_form
+from devplacepy.routers.sync._shared import require_sync_caller, run_batch
+from devplacepy.services.sync.records import SyncBatchForm
+from devplacepy.services.sync.spec import KIND_PROJECTS
+
+router = APIRouter()
+
+
+@router.post("/projects")
+async def sync_projects(
+    request: Request,
+    user: Annotated[dict, Depends(require_sync_caller)],
+    data: Annotated[SyncBatchForm, Depends(json_or_form(SyncBatchForm))],
+):
+    return run_batch(request, user, KIND_PROJECTS, data)

devplacepy/routers/sync/reactions.py

@@ -0,0 +1,21 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, Request
+
+from devplacepy.dependencies import json_or_form
+from devplacepy.routers.sync._shared import require_sync_caller, run_batch
+from devplacepy.services.sync.records import SyncBatchForm
+from devplacepy.services.sync.spec import KIND_REACTIONS
+
+router = APIRouter()
+
+
+@router.post("/reactions")
+async def sync_reactions(
+    request: Request,
+    user: Annotated[dict, Depends(require_sync_caller)],
+    data: Annotated[SyncBatchForm, Depends(json_or_form(SyncBatchForm))],
+):
+    return run_batch(request, user, KIND_REACTIONS, data)

devplacepy/routers/sync/votes.py

@@ -0,0 +1,21 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, Request
+
+from devplacepy.dependencies import json_or_form
+from devplacepy.routers.sync._shared import require_sync_caller, run_batch
+from devplacepy.services.sync.records import SyncBatchForm
+from devplacepy.services.sync.spec import KIND_VOTES
+
+router = APIRouter()
+
+
+@router.post("/votes")
+async def sync_votes(
+    request: Request,
+    user: Annotated[dict, Depends(require_sync_caller)],
+    data: Annotated[SyncBatchForm, Depends(json_or_form(SyncBatchForm))],
+):
+    return run_batch(request, user, KIND_VOTES, data)

devplacepy/schemas/__init__.py

@@ -116,12 +116,15 @@ from devplacepy.schemas.admin import (
     AdminCommunitiesOut,
     AdminCommunityItemOut,
     AdminCommunityLedgerOut,
+    AdminCommunitySchemaOut,
     AdminCommunityOut,
     AdminDeviiTaskItemOut,
     AdminDeviiTasksOut,
     AdminGameOut,
     AdminMediaItemOut,
+    AdminNodeCircuitOut,
     AdminNodeDeviceOut,
+    AdminNodeOutcomeOut,
     AdminNodeFlagOut,
     AdminNodePoolOut,
     AdminNodeQueueOut,

devplacepy/schemas/admin.py

@@ -134,6 +134,7 @@ class AdminCommunityItemOut(_Out):
     login_username: str = ""
     login_email: str = ""
     has_login: bool = False
+    has_totp: bool = False
 
 
 class AdminCommunitiesOut(_Out):
@@ -202,12 +203,22 @@ class AdminLedgerTileOut(_Out):
     listed: int = 0
 
 
+class AdminCommunitySchemaOut(_Out):
+    community: AdminCommunityItemOut
+    schema_text: str = ""
+    has_previous: bool = False
+    errors: list[str] = []
+    notice: str = ""
+    admin_section: Optional[str] = None
+
+
 class AdminCommunityLedgerOut(_Out):
     community: AdminCommunityItemOut
     scope: str = "community"
     run: Optional[AdminRunSummaryOut] = None
     run_errors: list[str] = []
     run_log: str = ""
+    busy: bool = False
     verdict: dict = {}
     tiles: list[AdminLedgerTileOut] = []
     items: list[AdminLedgerItemOut] = []
@@ -228,6 +239,8 @@ class AdminNodeDeviceOut(_Out):
     suspend_reason: str = ""
     country: str = ""
     asn_class: str = "unknown"
+    asn: str = ""
+    class_override: str = ""
     last_seen_at: Optional[str] = None
     created_at: Optional[str] = None
     owner_uid: str = ""
@@ -277,6 +290,22 @@ class AdminNodeQueueOut(_Out):
     expired: int = 0
 
 
+class AdminNodeCircuitOut(_Out):
+    key: str = ""
+    state: str = "closed"
+    failures: int = 0
+    seconds_left: int = 0
+
+
+class AdminNodeOutcomeOut(_Out):
+    adapter: str = ""
+    asn_class: str = "unknown"
+    ok: int = 0
+    blocked: int = 0
+    failed: int = 0
+    total: int = 0
+
+
 class AdminNodesOut(_Out):
     enabled: bool = False
     pool: AdminNodePoolOut = AdminNodePoolOut()
@@ -290,4 +319,9 @@ class AdminNodesOut(_Out):
     devices: list[AdminNodeDeviceOut] = []
     settings: list[AdminNodeSettingOut] = []
     limits: dict[str, int] = {}
+    circuits: list[AdminNodeCircuitOut] = []
+    outcomes: list[AdminNodeOutcomeOut] = []
+    lifetime: dict[str, int] = {}
+    geo: dict[str, Any] = {}
+    cache: dict[str, int] = {}
     admin_section: Optional[str] = None

devplacepy/schemas/game.py

@@ -44,6 +44,8 @@ class GamePlotOut(_Out):
     fertilize_cost: int = 0
     raided_fraction: float = 0.0
     raided_pct: int = 0
+    can_stop_replant: bool = False
+    stop_after_harvest: bool = False
 
 
 class GamePerkOut(_Out):

devplacepy/schemas/sync.py

@@ -13,6 +13,8 @@ class SyncRecordResultOut(_Out):
     local_uid: str = ""
     errors: list[str] = []
     ignored: list[str] = []
+    needs_kind: str = ""
+    needs_uid: str = ""
 
 
 class SyncBatchOut(_Out):
@@ -22,3 +24,32 @@ class SyncBatchOut(_Out):
     total: int = 0
     counts: dict[str, int] = {}
     results: list[SyncRecordResultOut] = []
+    released: dict[str, int] = {}
+
+
+class SyncExportOut(_Out):
+    kind: str
+    count: int = 0
+    has_more: bool = False
+    next_cursor: str = ""
+    records: list[dict] = []
+
+
+class SyncDeletionOut(_Out):
+    kind: str
+    origin_uid: str
+    by: str = "member"
+    deleted_at: str = ""
+
+
+class SyncDeletionsOut(_Out):
+    origin: str
+    count: int = 0
+    has_more: bool = False
+    next_cursor: str = ""
+    items: list[SyncDeletionOut] = []
+
+
+class SyncAckOut(_Out):
+    origin: str
+    acknowledged: int = 0

devplacepy/services/community/CLAUDE.md

@@ -19,6 +19,7 @@ This file documents community discovery (`devplacepy/services/community/`, `rout
 | Devii tools (`community_*`), docs group, privacy row | `services/devii/actions/catalog/community.py`, `docs_api/groups/community.py` | built, all administrator-only |
 | AI structure assistant | `ai.py` | built, optional, every proposal validated against the sample |
 | Login and credentials | `extract.find_login_form`, `runtime.Collector.login`, `discoverer.LoginLearner` | built |
+| Checkpoints, schema validation | `checkpoint.py`, `schemacheck.py` | built, tested |
 
 ## Flow
 
@@ -31,7 +32,7 @@ Feed types: `rss`, `atom`, `json_api`, `html_list`. Learned field maps use the *
 - **stealthii is a project dependency** (`pyproject.toml`, git URL; the Dockerfile installs `git`). `fetcher.Fetcher` shares one stealth Chromium per event loop (`SharedBrowser`), applies the SSRF guard (`guard_public_url`) before every request and every `page.goto`, and waits at least `delay_seconds` between requests to one host. Without the browser it falls back to the guarded plain client and logs that JavaScript rendering is unavailable. `CommunityService.on_disable` closes the browser.
 - **JavaScript pages.** If a page yields nothing statically, detection renders it (`Fetcher.render_html`, `page.content()`); the feed is then marked `"render": true` and the collector renders it on every run. The same fallback applies to post pages when comments only exist after rendering.
 - **Comments and post bodies from HTML pages.** For any feed (RSS, Atom, JSON, HTML list) whose posts expose an address, `PageProbe` opens up to three post pages, finds the repeated comment element (class hint `comment|reply|response|answer`, nesting from DOM containment, id from `::attr(id)` else a stable hash), and learns a `mode: "page"` comments spec. It also learns a `post.detail.content` selector when the post page carries more text than the listing. JSON feeds still prefer `inline`/`url` comment modes.
-- **Login.** A community can store a username, an email and a password (columns `login_*`, never returned by any endpoint, admin form, Devii tool or CLI). Discovery finds the login form (page, login links, `/login`-style paths, rendered page), logs in with the session `community:<origin>`, then learns behind the wall. `schema.login` holds the form (action, field names, hidden defaults re-read at login for CSRF tokens, or selectors for a JavaScript page); `login_required` makes a missing account a clear `LoginRequired`. A rejected password raises `LoginFailed` and the community goes to `login_required`. Not supported: JSON-API logins, two-factor, captchas.
+- **Login.** A community can store a username, an email and a password (columns `login_*`, never returned by any endpoint, admin form, Devii tool or CLI). Discovery finds the login form (page, login links, `/login`-style paths, rendered page), logs in with the session `community:<origin>`, then learns behind the wall. `schema.login` holds the form (action, field names, hidden defaults re-read at login for CSRF tokens, or selectors for a JavaScript page); `login_required` makes a missing account a clear `LoginRequired`. A rejected password raises `LoginFailed` and the community goes to `login_required`. Two-factor (TOTP) and JSON-API logins are supported; captchas are not (see Additions).
 - **AI.** When HTML heuristics are weak (no author, no title, content only raw text, no date) or find no item container, comment structure or post body, `AiAssistant` asks the internal gateway model (site setting `community_ai_model`, default `molodetz`; `community_ai=0` turns it off) for selectors. A proposal is accepted per field only if it matches at least 60% of the sampled elements. If the gateway is unreachable the learner continues with heuristics and logs it.
 - **The discussion page is not the article.** A feed can link to an external article while the discussion lives elsewhere (lobste.rs: RSS `<link>` is the article, `<comments>` the discussion). The learner reads a `comments_url` field (a JSON/RSS key such as `comments`/`discussion`, or an HTML anchor whose text says "comments"/"replies"/"discuss") and learns the comment structure from those pages, while the full-text `detail` is still learned from the post `url`. `Collector.discussion_link` falls back to the post link. Measured on lobste.rs: 0 comments before, 325 comments for 100 posts after.
 - **Identifiers must be distinct.** For HTML lists `HtmlInferrer.first(distinct=True)` rejects an id/url selector that yields the same value for most items (the first `a[href]` of a lobste.rs story is the constant `/login` vote link, which collapsed every story into one identity and gave the title "1"). Anchor classes are tried by text length, and the title reuses the chosen anchor. If no distinct selector exists the feed is rejected, never learned with a constant id.
@@ -49,7 +50,7 @@ Every run writes one row per record that needs an explanation, so an administrat
 - **Links.** Each row links to the synced record here (batched, at most three queries a page) and to the source (`Collector.sources`, the post link; a comment links to its post's source). A removed record has no local link.
 - **A refusal is not a failure.** See "Refusals are not failures"; a refused run is `skipped` with an `attention` verdict, not an error.
 
-Gaps: the page does not live-refresh while a run is in progress (reload it); the run log and the ledger are written when the run finishes, so a run that is killed mid-way leaves neither; a text search across the whole community scans up to 40000 rows.
+The ledger page live-refreshes while a run is busy (`AdminCommunity.js`, `data-busy`/`data-live` sections). A run writes a checkpoint every `checkpoint_seconds` (15, `checkpoint.py` `RunCheckpoint`): tally, log and ledger rows so far, so a killed run keeps its work. `CommunityStore.reap_interrupted` (called by the service and the admin page) marks a run that stopped heartbeating as `interrupted`. Unique index `(run_uid, seq)` keeps the checkpoint append idempotent. Community-wide text search of 3+ characters uses the FTS5 trigram table `community_run_items_fts` (insert/delete triggers, rebuilt when first created); run scope keeps LIKE. Measured on 40000 rows: no-match 34 ms to 0.4 ms, rare term 34.5 ms to 0.8 ms, a common term (10000+ hits) 9.4 ms to 21 ms (slower, accepted).
 
 ## Output: one engine, concrete file
 
@@ -72,14 +73,23 @@ Gaps: the page does not live-refresh while a run is in progress (reload it); the
 
 Real run against `https://lobste.rs` (dry run, scratch database): RSS, JSON and HTML feeds detected, description/favicon/language/rules text learned, 25 posts and 24 authors collected from one page. Fixture tests cover RSS, Atom, JSON with inline comments and page pagination, and HTML lists. With stealthii and Chromium the same run learned lobste.rs from its `/rss` feed plus post-page comments (`li.comments_subtree`). Not measured: a real login-protected site, cursor pagination against a live site, the AI path against a live gateway (the local gateway rejected the scratch key, so heuristics ran). Fixtures cover JavaScript rendering, form login, a rejected password, RSS and HTML comments from post pages, feed deduplication and the AI validation.
 
+## Additions since the first build
+
+- **Comment pagination** (`extract.find_next_page`, hints `next_link`, `page_param`, `json_next`, `none`, learned into `comments.pagination`). The runtime stops on a page with no new comments and honors `max_comment_pages`.
+- **Logins.** TOTP second factor (`totp()` checked against the RFC 6238 vectors; secret in `communities.login_totp_secret`, never returned, `has_totp` only), JSON-API logins (`login.mode = json`, token header), OTP form detection. A captcha is detected (`detect_captcha`) and reported as `CaptchaRequired`; it is never solved.
+- **Structured data.** Feed type `json_ld` (schema.org `Article`/`DiscussionForumPosting` blocks), detected by `FeedDetector.ld_candidate`.
+- **Schema editor.** `/admin/communities/{slug}/schema` (GET and POST, plus `/schema/restore`) validates with `schemacheck.validate_schema` (ceilings, floors, selector syntax), verifies on a sample (`sample_limits`) before saving, keeps `schema_previous`, and regenerates `collectors.py`. Audit keys `admin.community.schema` and `admin.community.schema_restore`; Devii `community_schema_get` and `community_schema_set` (confirm gated).
+- **Engagement.** The collector now emits votes, reactions and attachments where the source exposes them (`options.attachments`, `SyncOptions.collect_attachments`). While aliasing is on, votes and reactions are dropped (no identity for the voter) and attachments take the target's author. The generator is `VERSION = 2`; `files.load` regenerates an outdated collectors file and keeps its embedded schema.
+- **Usernames.** `SyncPusher` runs batches with `humanize=True`, so a new account gets a realistic handle derived from the author (see "Realistic handles" in `devplacepy/services/sync/CLAUDE.md`); existing accounts keep their names. The collector also splits a feed author of the form `<site> by|via <person>` (Lobsters sends this for every story) with `extract.split_submitter`, so the person alone is the username and the identity key: one person sharing from many sites is one account with one link, and the stored handle always comes from the database after the first sight. Generator `VERSION = 3` regenerates existing collector files; because the key changed, `by`/`via` authors collected before this change do not match their old links and appear as new accounts on the next run (the old accounts are left as they are).
+- **Cross-run dedupe.** `state["links"]`, capped at `max_remembered_links` (20000), skips a link already seen in an earlier run.
+
 ## Gaps
 
-- HTML post-page comments follow `rel=next` links only (`a[rel=next]` or `link[rel=next]`); a site that paginates comments any other way yields the first `max_comment_pages` it can reach, or the first page only.
-- No two-factor, captcha or JSON-API login.
-- HTML learning is heuristic plus AI; a site whose markup defeats both needs a hand-edited `SCHEMA`.
-- Votes, reactions and attachments are not collected (the sync API does not take them).
+- A site with no JSON-LD, feed, API or repeating structure needs a hand-written schema through the editor; ready-made Discourse and WordPress specs were not built.
+- Captchas are detected, never solved; the run is reported and stops.
 - Cross-feed duplicate detection compares the sampled post addresses at discovery time only.
-- Passwords are stored in plain text in the database, as chosen for now; they are masked everywhere they could be displayed.
+- Not measured against a live login-protected site, live TOTP, or live cursor pagination; fixtures cover them.
+- Passwords are stored in plain text in the database, as chosen for now (owner decision, excluded from the gap work); they are masked everywhere they could be displayed.
 
 ## Selection, aliasing, translation
 

devplacepy/services/community/checkpoint.py

@@ -0,0 +1,51 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import asyncio
+import logging
+
+from devplacepy.services.community.ledger import LEVEL_RUNNING, tally_of
+from devplacepy.services.community.logbook import LogBook
+from devplacepy.services.community.spec import LIMITS
+
+logger = logging.getLogger("community.run")
+
+
+class RunCheckpoint:
+    def __init__(self, store, ledgers, run_uid: str, community_uid: str, report, live, book: LogBook):
+        self.store = store
+        self.ledgers = ledgers
+        self.run_uid = run_uid
+        self.community_uid = community_uid
+        self.report = report
+        self.live = live
+        self.book = book
+        self.saved = 0
+        self.writes = 0
+
+    def flush(self) -> None:
+        ledger = self.report.ledger
+        pushed = dict(self.live.counts)
+        self.saved += self.ledgers.append(self.run_uid, self.community_uid, ledger, self.saved)
+        self.store.update_run(
+            self.run_uid,
+            log=self.book.text(),
+            collected=self.report.collected,
+            pushed=pushed,
+            errors=list(self.report.errors),
+            tally=tally_of(pushed, ledger),
+            verdict=LEVEL_RUNNING,
+            headline="Running now.",
+            items_stored=self.saved,
+            items_dropped=ledger.dropped,
+        )
+        self.writes += 1
+
+    async def loop(self) -> None:
+        while True:
+            await asyncio.sleep(LIMITS.checkpoint_seconds)
+            try:
+                self.flush()
+            except Exception:
+                logger.exception("run checkpoint failed, the run continues")

devplacepy/services/community/discoverer.py

@@ -9,10 +9,10 @@ from dataclasses import dataclass, field
 from urllib.parse import urlsplit
 
 from devplacepy.services.community.ai import AiAssistant
-from devplacepy.services.community.extract import canonical_link, field_text, find_login_form, parse_html, select_nodes
+from devplacepy.services.community.extract import canonical_link, detect_captcha, field_text, find_login_form, json_or_none, leaf_paths, parse_html, select_nodes
 from devplacepy.services.community.feeds import FeedDetector
 from devplacepy.services.community.learner import MetadataLearner, StructureLearner
-from devplacepy.services.community.runtime import Collector, LoginFailed, LoginRequired
+from devplacepy.services.community.runtime import CaptchaRequired, Collector, LoginFailed, LoginRequired, SecondFactorRequired
 from devplacepy.services.community.search import SearchPhase
 from devplacepy.services.community.spec import (
     LIMITS,
@@ -57,6 +57,10 @@ def slug_for_query(query: str) -> str:
     return slug_of(origin_of(address)) if address else re.sub(r"[^a-z0-9]+", "-", query.lower()).strip("-")[:63]
 
 
+JSON_LOGIN_PATHS = ("/api/login", "/api/auth/login", "/api/v1/auth/login", "/api/v1/login", "/auth/login", "/login.json", "/api/session", "/session", "/api/users/login")
+TOKEN_KEYS = re.compile(r"^(token|access_token|accesstoken|jwt|id_token|auth_token|authtoken|bearer)$", re.I)
+SUCCESS_KEYS = ("user", "data", "success", "ok", "session")
+MAX_JSON_LOGIN_ATTEMPTS = 12
 LOGIN_PATHS = ("/login", "/signin", "/sign-in", "/accounts/login", "/users/sign_in", "/auth/login", "/user/login")
 LOGIN_LABEL = re.compile(r"\b(log ?in|sign ?in)\b", re.I)
 MAX_LOGIN_PAGES = 6
@@ -78,9 +82,49 @@ class LoginLearner:
             return ""
         return text if status < 400 else ""
 
+    async def find_json(self, base_url: str, credentials: dict) -> dict | None:
+        origin = "{0.scheme}://{0.netloc}".format(urlsplit(base_url))
+        by_email = bool(credentials.get("email")) and not credentials.get("username")
+        identity = credentials.get("email") if by_email else credentials.get("username") or credentials.get("email")
+        keys = ("email", "username", "login") if by_email else ("username", "login", "email")
+        attempts = 0
+        for path in JSON_LOGIN_PATHS:
+            for key in keys:
+                if attempts >= MAX_JSON_LOGIN_ATTEMPTS:
+                    return None
+                attempts += 1
+                url = origin + path
+                try:
+                    response = await self.fetcher.post(url, json={key: identity, "password": credentials["password"]})
+                    status, text = response.status, await response.text()
+                except Exception as exc:
+                    logger.info("login: json probe %s failed: %s: %s", url, type(exc).__name__, exc)
+                    break
+                if status in (404, 405) or status >= 500:
+                    break
+                spec = self.json_spec(url, key, "email" if key == "email" else "username", status, text)
+                if spec:
+                    logger.info("login: found a JSON login at %s", url)
+                    return spec
+        return None
+
+    @staticmethod
+    def json_spec(url: str, key: str, identity: str, status: int, text: str) -> dict | None:
+        document = json_or_none(text)
+        if status >= 400 or not isinstance(document, dict) or any(document.get(name) for name in ("error", "errors")):
+            return None
+        token = next((path for path, value in leaf_paths(document).items() if TOKEN_KEYS.match(path.rsplit(".", 1)[-1]) and isinstance(value, str) and value), "")
+        if not token and not any(document.get(name) for name in SUCCESS_KEYS):
+            return None
+        spec = {"mode": "json", "url": url, "username_field": key, "password_field": "password", "identity": identity}
+        if token:
+            spec.update(token_path=token, token_scheme="Bearer")
+        return spec
+
     async def find(self, base_url: str, html: str) -> dict | None:
         spec = find_login_form(parse_html(html), base_url) if html else None
         if spec:
+            spec["captcha"] = detect_captcha(html)
             return spec
         urls = []
         if html:
@@ -92,9 +136,11 @@ class LoginLearner:
         urls = list(dict.fromkeys(urls))[:MAX_LOGIN_PAGES]
         for render in (False, True):
             for url in urls[: MAX_LOGIN_PAGES if not render else 2]:
-                spec = find_login_form(parse_html(await self.page(url, render)), url)
+                text = await self.page(url, render)
+                spec = find_login_form(parse_html(text), url)
                 if spec:
                     spec["render"] = render
+                    spec["captcha"] = detect_captcha(text)
                     logger.info("login: found a %s login form at %s%s", spec["mode"], url, " (needs JavaScript)" if render else "")
                     return spec
         return None
@@ -146,11 +192,20 @@ class CommunityDiscoverer:
         login = None
         if walled or self.has_credentials():
             login = await self.logins.find(url, context.get("html", ""))
+            if login is None and self.has_credentials():
+                login = await self.logins.find_json(url, self.credentials)
         if login and self.has_credentials():
             probe = {"community": origin, "origin": origin, "base_url": url, "login": login, "limits": LIMITS.runtime()}
             try:
                 await Collector(self.fetcher, probe, credentials=self.credentials).login()
                 logger.info("discover: logged in to %s", origin)
+            except (CaptchaRequired, SecondFactorRequired) as exc:
+                result.error, result.status = str(exc), STATUS_LOGIN
+                result.url, result.origin, result.slug = url, origin, slug_of(origin)
+                result.name = result.name or origin
+                result.schema = self.schema(result, [], login_required=True, login=login)
+                logger.error("discover: %s", exc)
+                return False
             except LoginFailed as exc:
                 result.error, result.status = str(exc), STATUS_FAILED
                 logger.error("discover: %s", exc)
@@ -175,7 +230,12 @@ class CommunityDiscoverer:
         if not feeds:
             if walled and login:
                 result.status = STATUS_LOGIN
-                result.error = "the community requires a login; add the account's username or email and password and relearn" if not self.has_credentials() else "no feed could be learned even after logging in"
+                if login.get("captcha"):
+                    result.error = "the login page of this community uses a captcha, which cannot be automated; sign in by hand or use a site that offers an API"
+                elif not self.has_credentials():
+                    result.error = "the community requires a login; add the account's username or email and password and relearn"
+                else:
+                    result.error = "no feed could be learned even after logging in"
                 result.schema = self.schema(result, [], login_required=True, login=login)
                 logger.warning("discover: %s", result.error)
             else:

devplacepy/services/community/extract.py

@@ -2,8 +2,13 @@
 
 from __future__ import annotations
 
+import base64
+import hashlib
+import hmac
 import json
 import re
+import struct
+import time
 import xml.etree.ElementTree as ElementTree
 from datetime import datetime, timezone
 from email.utils import parsedate_to_datetime
@@ -381,6 +386,16 @@ def block_text(node: Node) -> str:
     return "".join(parts)
 
 
+SUBMITTER = re.compile(r"^\S+\s+(?:by|via)\s+(\S+?)(?:\s+\d{4}-\d{2}-\d{2}.*)?$", re.I)
+BARE_SUBMITTER = re.compile(r"^(?:by|via)\s+(\S+?)(?:\s+\d{4}-\d{2}-\d{2}.*)?$", re.I)
+
+
+def split_submitter(name: str) -> str:
+    text = " ".join(str(name or "").split())
+    found = SUBMITTER.match(text) or BARE_SUBMITTER.match(text)
+    return found.group(1) if found else text
+
+
 def strip_tags(text: str) -> str:
     if "<" not in text:
         return text.strip()
@@ -415,6 +430,84 @@ def leaf_paths(data, prefix: str = "$", depth: int = 0) -> dict[str, object]:
     return found
 
 
+NEXT_PAGE_FIELDS = (
+    "a[rel=next]::attr(href)",
+    "link[rel=next]::attr(href)",
+    "a.next::attr(href)",
+    "li.next a::attr(href)",
+    ".next a::attr(href)",
+    "a[class*=next]::attr(href)",
+    "a[aria-label^=Next]::attr(href)",
+)
+PAGINATION_CONTAINERS = ("nav", ".pagination", ".pager", ".paginator", "[class*=pagination]", "[class*=pager]")
+NEXT_LABEL = re.compile(
+    r"^(?:next(?: page)?|older(?: comments| posts| replies)?|more(?: comments| replies)?|load more|volgende(?: pagina)?|weiter|suivant|siguiente|[\u203a\u00bb\u2192>]+|next\s*[\u203a\u00bb\u2192>]+|[\u203a\u00bb\u2192>]+\s*next)$",
+    re.I,
+)
+PAGE_NUMBER = re.compile(r"(?:[?&](?:page|p|pg|paged|pagenum|pageno|pn|cp)=(\d+))|(?:/page/(\d+))|(?:/p(\d+)/?$)", re.I)
+PAGE_PARAM_NAMES = ("page", "p", "pg", "paged", "pagenum", "pageno", "pn", "cp")
+UNUSABLE_HREF = ("#", "javascript:", "mailto:", "tel:")
+
+
+def page_number(url: str) -> int | None:
+    match = PAGE_NUMBER.search(url)
+    if not match:
+        return None
+    return int(next(group for group in match.groups() if group))
+
+
+def with_page_number(url: str, number: int, param: str = "page") -> str:
+    if re.search(r"/page/\d+", url):
+        return re.sub(r"/page/\d+", f"/page/{number}", url, count=1)
+    parts = urlsplit(url)
+    query = [item for item in parts.query.split("&") if item and item.split("=")[0].lower() not in PAGE_PARAM_NAMES]
+    query.append(f"{param}={number}")
+    return parts._replace(query="&".join(query)).geturl()
+
+
+def contained_in(node: Node, containers: list[Node]) -> bool:
+    wanted = {id(item) for item in containers}
+    cursor = node.parent
+    while cursor is not None:
+        if id(cursor) in wanted:
+            return True
+        cursor = cursor.parent
+    return False
+
+
+def usable_href(href: str) -> bool:
+    return bool(href) and not href.strip().lower().startswith(UNUSABLE_HREF)
+
+
+def find_next_page(root: Node, current: str, hint: dict | None = None) -> str:
+    hint = hint or {}
+    style = hint.get("style", "auto")
+    if style == "none":
+        return ""
+    if style == "next_link" and hint.get("next"):
+        value = html_field(root, hint["next"])
+        return urljoin(current, value) if usable_href(value) else ""
+    if style == "page_param":
+        return with_page_number(current, (page_number(current) or 1) + 1, hint.get("param") or "page")
+    for spec in NEXT_PAGE_FIELDS:
+        value = html_field(root, spec)
+        if usable_href(value):
+            return urljoin(current, value)
+    anchors = [node for node in root.descendants() if node.tag == "a" and usable_href(node.attrs.get("href", ""))]
+    containers = [node for spec in PAGINATION_CONTAINERS for node in select_nodes(root, spec)]
+    for anchor in anchors:
+        label = anchor.text() or anchor.attrs.get("aria-label", "") or anchor.attrs.get("title", "")
+        if NEXT_LABEL.match(label.strip()) and (contained_in(anchor, containers) or not label.strip().isalpha()):
+            return urljoin(current, anchor.attrs["href"])
+    here = urlsplit(current)
+    wanted = (page_number(current) or 1) + 1
+    for anchor in anchors:
+        target = urljoin(current, anchor.attrs["href"])
+        if urlsplit(target).path.rstrip("/").rsplit("/page/", 1)[0] == here.path.rstrip("/").rsplit("/page/", 1)[0] and page_number(target) == wanted:
+            return target
+    return ""
+
+
 IDENTITY_NAME = re.compile(r"(user|login|email|mail|account|identifier|handle|name)", re.I)
 SKIP_INPUT_TYPES = frozenset({"submit", "button", "image", "reset", "checkbox", "radio", "file"})
 
@@ -472,3 +565,139 @@ def find_login_form(root: Node, base_url: str) -> dict | None:
             "submit_selector": "button[type=submit], input[type=submit]",
         }
     return None
+
+
+OTP_NAME = re.compile(r"(otp|totp|2fa|two[-_ ]?factor|one[-_ ]?time|verification[-_ ]?code|auth(?:enticator)?[-_ ]?code|security[-_ ]?code|mfa|token|^code$)", re.I)
+CAPTCHA_MARKERS = re.compile(r"(g-recaptcha|grecaptcha|h-captcha|hcaptcha|cf-turnstile|turnstile\.js|captcha|funcaptcha|arkoselabs)", re.I)
+TOTP_DIGITS = 6
+TOTP_STEP = 30
+
+
+def totp(secret: str, at: float | None = None, digits: int = TOTP_DIGITS, step: int = TOTP_STEP) -> str:
+    cleaned = re.sub(r"[\s-]", "", secret or "").upper()
+    cleaned += "=" * (-len(cleaned) % 8)
+    key = base64.b32decode(cleaned, casefold=True)
+    counter = int((time.time() if at is None else at) // step)
+    digest = hmac.new(key, struct.pack(">Q", counter), hashlib.sha1).digest()
+    offset = digest[-1] & 0x0F
+    value = struct.unpack(">I", digest[offset : offset + 4])[0] & 0x7FFFFFFF
+    return str(value % (10**digits)).zfill(digits)
+
+
+def is_otp_input(node: Node) -> bool:
+    name = node.attrs.get("name", "")
+    if not name:
+        return False
+    if node.attrs.get("autocomplete", "").lower() == "one-time-code":
+        return True
+    kind = node.attrs.get("type", "text").lower()
+    return kind in ("text", "number", "tel") and bool(OTP_NAME.search(name) or OTP_NAME.search(node.attrs.get("id", "")))
+
+
+def find_otp_form(root: Node, base_url: str) -> dict | None:
+    for form in select_nodes(root, "form"):
+        inputs = [node for node in form.descendants() if node.tag == "input"]
+        field = next((node for node in inputs if is_otp_input(node)), None)
+        if field is None:
+            continue
+        return {
+            "action": urljoin(base_url, form.attrs.get("action") or base_url),
+            "method": (form.attrs.get("method") or "post").upper(),
+            "field": field.attrs["name"],
+            "fields": {
+                node.attrs["name"]: node.attrs.get("value", "")
+                for node in inputs
+                if node.attrs.get("name")
+                and node is not field
+                and node.attrs.get("type", "text").lower() not in SKIP_INPUT_TYPES
+            },
+        }
+    return None
+
+
+LD_POST_TYPES = frozenset(
+    {"article", "newsarticle", "blogposting", "discussionforumposting", "socialmediaposting", "techarticle", "report", "question", "answer", "qapage", "liveblogposting"}
+)
+LD_COMMENT_TYPES = frozenset({"comment", "answer"})
+LD_MAX_ITEMS = 200
+LD_MAX_DEPTH = 6
+
+
+def ld_types(node: dict) -> set[str]:
+    declared = node.get("@type")
+    values = declared if isinstance(declared, list) else [declared]
+    return {str(item).lower().rsplit("/", 1)[-1] for item in values if item}
+
+
+def ld_text(value) -> str:
+    if isinstance(value, dict):
+        return ld_text(value.get("name") or value.get("@value") or value.get("text") or "")
+    if isinstance(value, list):
+        return ld_text(value[0]) if value else ""
+    return str(value or "").strip()
+
+
+def ld_author(value) -> dict:
+    node = value[0] if isinstance(value, list) and value else value
+    if isinstance(node, dict):
+        name = ld_text(node.get("name") or node.get("alternateName") or node.get("identifier"))
+        identity = ld_text(node.get("@id") or node.get("url") or node.get("identifier"))
+    else:
+        name, identity = ld_text(node), ""
+    result = {}
+    if name:
+        result["username"] = name
+    if identity:
+        result["id"] = identity
+    return result
+
+
+def ld_normalize(node: dict, depth: int = 0) -> dict:
+    content = ld_text(node.get("articleBody") or node.get("text") or node.get("description") or "")
+    normalized = {
+        "id": ld_text(node.get("@id") or node.get("identifier") or node.get("url")),
+        "url": ld_text(node.get("url") or node.get("mainEntityOfPage")),
+        "title": ld_text(node.get("headline") or node.get("name")),
+        "content": content,
+        "created_at": ld_text(node.get("datePublished") or node.get("dateCreated")),
+        "updated_at": ld_text(node.get("dateModified")),
+        "author": ld_author(node.get("author") or node.get("creator")),
+    }
+    raw_comments = node.get("comment") or node.get("suggestedAnswer") or node.get("acceptedAnswer") or []
+    if isinstance(raw_comments, dict):
+        raw_comments = [raw_comments]
+    if depth < LD_MAX_DEPTH and isinstance(raw_comments, list):
+        normalized["comments"] = [ld_normalize(item, depth + 1) for item in raw_comments if isinstance(item, dict)][:LD_MAX_ITEMS]
+    return {key: value for key, value in normalized.items() if value not in ("", {}, [])}
+
+
+def ld_walk(value, found: list, depth: int = 0) -> None:
+    if depth > LD_MAX_DEPTH or len(found) >= LD_MAX_ITEMS:
+        return
+    if isinstance(value, list):
+        for item in value:
+            ld_walk(item, found, depth + 1)
+    elif isinstance(value, dict):
+        if ld_types(value) & LD_POST_TYPES and not ld_types(value) <= LD_COMMENT_TYPES:
+            found.append(ld_normalize(value))
+            return
+        for key in ("@graph", "itemListElement", "item", "mainEntity", "hasPart"):
+            if key in value:
+                ld_walk(value[key], found, depth + 1)
+
+
+def extract_json_ld_posts(root: Node) -> list[dict]:
+    found: list[dict] = []
+    for script in select_nodes(root, "script"):
+        if "ld+json" not in script.attrs.get("type", "").lower():
+            continue
+        raw = "".join(chunk for chunk in script.chunks if isinstance(chunk, str)).strip()
+        try:
+            ld_walk(json.loads(raw), found)
+        except ValueError:
+            continue
+    return [item for item in found if item.get("id") and (item.get("title") or item.get("content"))]
+
+
+def detect_captcha(text: str) -> bool:
+    return bool(CAPTCHA_MARKERS.search(text or ""))

devplacepy/services/community/feeds.py

@@ -10,6 +10,7 @@ from urllib.parse import urljoin, urlsplit
 
 from devplacepy.services.community.extract import (
     Node,
+    extract_json_ld_posts,
     json_or_none,
     parse_feed,
     parse_html,
@@ -19,6 +20,7 @@ from devplacepy.services.community.spec import (
     FEED_ATOM,
     FEED_HTML,
     FEED_JSON,
+    FEED_JSONLD,
     FEED_PROBE_PATHS,
     FEED_RSS,
     LIMITS,
@@ -180,7 +182,7 @@ class FeedDetector:
             for path in FEED_PROBE_PATHS[: LIMITS.max_probe_paths]:
                 await self.consider(urljoin(origin, path), found, context)
         if not found and root is not None:
-            candidate = await self.html_with_ai(base_url, text, root)
+            candidate = await self.html_with_ai(base_url, text, root) or self.ld_candidate(base_url, root)
             if candidate:
                 found[base_url] = candidate
         if not found and not direct:
@@ -202,7 +204,7 @@ class FeedDetector:
             return
         candidate = classify_text(url, body)
         if candidate is None and status < 300:
-            candidate = self.html_candidate(url, body)
+            candidate = self.html_candidate(url, body) or self.ld_candidate(url, parse_html(body) if "<" in body[:2000] else None)
         if candidate:
             logger.info("detect: %s feed at %s (%d sample items)", candidate.type, url, len(candidate.items))
             found[url] = candidate
@@ -225,7 +227,7 @@ class FeedDetector:
         return FeedCandidate(url, FEED_HTML, items=select_nodes(root, selector), document=root, item_selector=selector)
 
     def preference(self, candidate: FeedCandidate) -> int:
-        order = {FEED_JSON: 0, FEED_ATOM: 1, FEED_RSS: 2, FEED_HTML: 3}
+        order = {FEED_JSON: 0, FEED_ATOM: 1, FEED_RSS: 2, FEED_JSONLD: 3, FEED_HTML: 4}
         return order[candidate.type]
 
     def advertised(self, root: Node, base_url: str) -> list[str]:
@@ -256,6 +258,14 @@ class FeedDetector:
         selector, nodes = found
         return FeedCandidate(url, FEED_HTML, items=nodes, document=root, item_selector=selector)
 
+    def ld_candidate(self, url: str, root: Node | None) -> FeedCandidate | None:
+        if root is None:
+            return None
+        items = extract_json_ld_posts(root)
+        if len(items) < 2:
+            return None
+        return FeedCandidate(url, FEED_JSONLD, items=items, document=items)
+
     def looks_like_login(self, root: Node) -> bool:
         if select_nodes(root, "input[type=password]") and not find_item_selector(root):
             return True

devplacepy/services/community/fetcher.py

@@ -115,27 +115,35 @@ class Fetcher:
         if self.browser is None and self.client is None:
             await self.start()
 
-    async def get(self, url: str, params: dict | None = None, timeout: float | None = None, session: str | None = None):
+    async def get(self, url: str, params: dict | None = None, timeout: float | None = None, session: str | None = None, headers: dict | None = None):
         await self.ready()
         await guard_public_url(url)
         await self.polite(url)
         logger.info("GET %s params=%s", url, params or {})
         if self.browser is not None:
-            response = await self.browser.get(url, params=params, timeout=timeout or LIMITS.timeout_seconds, session=session or self.session)
+            response = await self.browser.get(url, params=params, headers=headers, timeout=timeout or LIMITS.timeout_seconds, session=session or self.session)
             logger.info("GET %s -> %s", url, response.status)
             return response
-        raw = await self.client.get(url, params=params, timeout=timeout or LIMITS.timeout_seconds)
+        raw = await self.client.get(url, params=params, headers=headers, timeout=timeout or LIMITS.timeout_seconds)
         logger.info("GET %s -> %s", url, raw.status_code)
         return FetchResponse(raw)
 
-    async def post(self, url: str, data: dict | None = None, timeout: float | None = None, session: str | None = None):
+    async def post(
+        self,
+        url: str,
+        data: dict | None = None,
+        timeout: float | None = None,
+        session: str | None = None,
+        json: dict | None = None,
+        headers: dict | None = None,
+    ):
         await self.ready()
         await guard_public_url(url)
         await self.polite(url)
-        logger.info("POST %s fields=%s", url, sorted((data or {}).keys()))
+        logger.info("POST %s fields=%s", url, sorted((json or data or {}).keys()))
         if self.browser is not None:
-            return await self.browser.post(url, data=data, timeout=timeout or LIMITS.timeout_seconds, session=session or self.session)
-        return FetchResponse(await self.client.post(url, data=data, timeout=timeout or LIMITS.timeout_seconds))
+            return await self.browser.post(url, data=data, json=json, headers=headers, timeout=timeout or LIMITS.timeout_seconds, session=session or self.session)
+        return FetchResponse(await self.client.post(url, data=data, json=json, headers=headers, timeout=timeout or LIMITS.timeout_seconds))
 
     @asynccontextmanager
     async def page(self, session: str | None = None):

devplacepy/services/community/files.py

@@ -27,6 +27,10 @@ class CommunityFiles:
         path = self.collectors_path(slug)
         if not path.exists():
             self.write(slug, schema)
+        else:
+            source = path.read_text()
+            if not CodeGenerator.is_current(source):
+                self.write(slug, CodeGenerator.embedded_schema(source) or schema)
         spec = importlib.util.spec_from_file_location(f"community_collectors_{slug.replace('-', '_')}", path)
         module = importlib.util.module_from_spec(spec)
         spec.loader.exec_module(module)

devplacepy/services/community/generator.py

@@ -14,8 +14,8 @@ HEADER_LINE = re.compile(r"^# retoor <[^>]*>\n", re.M)
 
 ENTRYPOINT = '''
 
-async def collect_all(stealth, state=None, only=None, sleep=None, credentials=None):
-    collector = Collector(stealth, SCHEMA, state, sleep, credentials)
+async def collect_all(stealth, state=None, only=None, sleep=None, credentials=None, options=None):
+    collector = Collector(stealth, SCHEMA, state, sleep, credentials, options)
     return await collector.run(only)
 {wrappers}
 
@@ -53,19 +53,40 @@ if __name__ == "__main__":
 
 WRAPPER = '''
 
-async def collect_{name}(stealth, state=None, sleep=None, credentials=None):
+async def collect_{name}(stealth, state=None, sleep=None, credentials=None, options=None):
     # {kind} feed {url}
     # post fields: {fields}
     # comments: {comments}
     # pagination: {pagination}
-    return await collect_all(stealth, state, only="{feed_id}", sleep=sleep, credentials=credentials)
+    return await collect_all(stealth, state, only="{feed_id}", sleep=sleep, credentials=credentials, options=options)
 '''
 
 
+EMBEDDED_SCHEMA = re.compile(r"^SCHEMA = json\.loads\(r'''(.*?)'''\)$", re.M | re.S)
+VERSION_MARK = re.compile(r"^# generator-version: (\d+)$", re.M)
+
+
 class CodeGenerator:
+    VERSION = 3
+
     def __init__(self, schema: dict):
         self.schema = schema
 
+    @classmethod
+    def is_current(cls, source: str) -> bool:
+        found = VERSION_MARK.search(source)
+        return bool(found) and int(found.group(1)) == cls.VERSION
+
+    @staticmethod
+    def embedded_schema(source: str) -> dict | None:
+        found = EMBEDDED_SCHEMA.search(source)
+        if not found:
+            return None
+        try:
+            return json.loads(found.group(1).replace("\\u0027", "'"))
+        except ValueError:
+            return None
+
     @staticmethod
     def module_source(module) -> str:
         text = Path(module.__file__).read_text()
@@ -94,6 +115,7 @@ class CodeGenerator:
         header = (
             "# Generated by DevPlace community discovery. Community: "
             f"{self.schema.get('community')} (origin {self.schema.get('origin')}).\n"
+            f"# generator-version: {self.VERSION}\n"
             "# Self-contained: standard library plus stealthii when run directly.\n"
             "# A login-protected community reads COMMUNITY_USERNAME / COMMUNITY_EMAIL / COMMUNITY_PASSWORD from the environment.\n"
             "# Adjust SCHEMA below to change a selector or path; collect_all() reads it at runtime.\n"

devplacepy/services/community/learner.py

@@ -8,6 +8,7 @@ import statistics
 from urllib.parse import urljoin, urlsplit
 
 from devplacepy.services.community.extract import (
+    NEXT_PAGE_FIELDS,
     NOW_TOKEN,
     Node,
     field_text,
@@ -26,6 +27,7 @@ from devplacepy.services.community.feeds import FeedCandidate, find_comment_sele
 from devplacepy.services.community.spec import (
     FEED_HTML,
     FEED_JSON,
+    FEED_JSONLD,
     LIMITS,
 )
 
@@ -56,6 +58,11 @@ COMMENT_LINK_TEXT = re.compile(r"\bcomments?\b|\breplies\b|\bdiscuss", re.I)
 MIN_TITLE_LENGTH = 5
 STATIC_LINK_SPECS = ("h1 a::attr(href)", "h2 a::attr(href)", "h3 a::attr(href)", "a.title::attr(href)", "a.u-url::attr(href)")
 STATIC_TITLE_SPECS = ("h1", "h2", "h3", "a.title", "a.u-url")
+VOTE_LIST_KEYS = re.compile(r"^(votes|upvotes|upvoters|likes|likers|voters)$", re.I)
+REACTION_LIST_KEYS = re.compile(r"^(reactions|emoji_reactions|emojis)$", re.I)
+VOTE_VALUE_KEYS = re.compile(r"^(value|direction|vote|dir|type|score)$", re.I)
+EMOJI_KEYS = re.compile(r"^(emoji|reaction|reaction_type|reaction_emoji)$", re.I)
+ENGAGEMENT_MIN_ENTRIES = 2
 COMMENT_PROBES = ("{feed}/{id}/comments", "{feed}/{id}", "/api/posts/{id}/comments", "/api/comments?post_id={id}")
 
 
@@ -255,6 +262,45 @@ class HtmlInferrer:
         return {"fields": {"username": [name]} if name else {}}
 
 
+def learn_engagement(samples: list) -> dict:
+    inferrer = FieldInferrer([sample for sample in samples if isinstance(sample, dict)])
+    found: dict = {}
+    for kind, pattern in (("votes", VOTE_LIST_KEYS), ("reactions", REACTION_LIST_KEYS)):
+        for path in inferrer.list_paths(pattern):
+            entries = [entry for sample in inferrer.samples for match in select(sample, path) if isinstance(match, list) for entry in match if isinstance(entry, dict)]
+            if len(entries) < ENGAGEMENT_MIN_ENTRIES:
+                continue
+            sub = FieldInferrer(entries)
+            author = sub.author_spec()
+            if not author["fields"].get("username"):
+                continue
+            fields: dict = {}
+            if kind == "reactions":
+                emoji = [candidate for candidate in sub.candidates(EMOJI_KEYS.pattern) if sub.coverage_of(candidate) >= sub.coverage]
+                if not emoji:
+                    continue
+                fields["emoji"] = emoji[:1]
+            else:
+                value = [candidate for candidate in sub.candidates(VOTE_VALUE_KEYS.pattern) if sub.coverage_of(candidate) >= sub.coverage]
+                if value:
+                    fields["value"] = value[:1]
+            moment = sub.pick_all("created_at", lambda text: normalize_instant(text) is not None)[:1]
+            if moment:
+                fields["created_at"] = moment
+            found[kind] = {"list_path": path, "fields": fields, "author": author}
+            break
+    return found
+
+
+def json_next_path(document) -> str | None:
+    if not isinstance(document, dict):
+        return None
+    for path, value in leaf_paths(document).items():
+        if NEXT_KEYS.match(last_segment(path)) and isinstance(value, str) and value.startswith(("http", "/", "?")):
+            return path
+    return None
+
+
 class PaginationProbe:
     def __init__(self, fetcher):
         self.fetcher = fetcher
@@ -327,13 +373,14 @@ class CommentProbe:
             if any(isinstance(match, list) and match for sample in entries for match in select(sample, path)):
                 spec["replies_path"] = path
                 break
+        spec.update(learn_engagement(entries))
         return spec
 
     def usable(self, spec: dict) -> bool:
         return "id" in spec["fields"] and "content" in spec["fields"]
 
     async def learn(self, candidate: FeedCandidate, post_fields: dict, inferrer: FieldInferrer | None) -> dict:
-        if candidate.type != FEED_JSON or inferrer is None or not candidate.items:
+        if candidate.type not in (FEED_JSON, FEED_JSONLD) or inferrer is None or not candidate.items:
             return {"mode": "none", "reason": f"comments are not learned for {candidate.type} feeds"}
         for path in inferrer.comment_lists():
             entries = [entry for sample in candidate.items for match in select(sample, path) if isinstance(match, list) for entry in match if isinstance(entry, dict)]
@@ -380,7 +427,11 @@ class CommentProbe:
         spec = self.comment_fields([entry for entry in entries if isinstance(entry, dict)])
         if not self.usable(spec):
             return None
-        return {"mode": "url", "url_template": template, "list_path": path, **spec}
+        learned = {"mode": "url", "url_template": template, "list_path": path, **spec}
+        next_path = json_next_path(document)
+        if next_path:
+            learned["pagination"] = {"style": "json_next", "path": next_path}
+        return learned
 
 
 class PageProbe:
@@ -486,9 +537,19 @@ class PageProbe:
                 comments = self.ai_comment_spec(proposal, render)
         detail = await self.post_detail(candidate, post, post_links, links, roots, render)
         if comments["mode"] == "page":
+            declared = self.comment_pagination(roots)
+            if declared:
+                comments["pagination"] = declared
             logger.info("pages: comments via %s (%s)", comments["item_selector"], sorted(comments["fields"]))
         return comments, detail
 
+    @staticmethod
+    def comment_pagination(roots: list[Node]) -> dict | None:
+        for spec in NEXT_PAGE_FIELDS:
+            if any(html_field(root, spec) for root in roots):
+                return {"style": "next_link", "next": spec}
+        return None
+
     async def post_detail(self, candidate: FeedCandidate, post: dict, post_links: list[str], links: list[str], roots: list[Node], render: bool) -> dict | None:
         if post_links == links:
             return await self.detail(candidate, post, roots, render)
@@ -630,6 +691,8 @@ class StructureLearner:
         feed = self.as_feed(candidate, post, author, pagination)
         if detail:
             feed["post"]["detail"] = detail
+        if inferrer is not None:
+            feed["post"].update(learn_engagement(samples))
         feed["id"] = f"feed{index + 1}"
         feed["comments"] = comments
         feed["post"]["source_footer"] = True

devplacepy/services/community/ledger.py

@@ -10,7 +10,16 @@ from devplacepy.services.sync.spec import (
     KIND_COMMENTS,
     KIND_POSTS,
     KIND_USERS,
+    KINDS,
+    REASON_ACK_DELETED,
     REASON_CONTENT_REFUSED,
+    REASON_IMMUTABLE,
+    REASON_QUEUE_FULL,
+    REASON_SELF_FOLLOW,
+    REASON_UNKNOWN_OPTION,
+    REASON_UNKNOWN_POLL,
+    REASON_UNKNOWN_PROJECT,
+    REASON_UNKNOWN_TARGET,
     REASON_INTERNAL,
     REASON_INVALID,
     REASON_LOCALLY_DELETED,
@@ -22,6 +31,7 @@ from devplacepy.services.sync.spec import (
     REASON_UNKNOWN_PARENT,
     REASON_UNKNOWN_POST,
     SKIP_IDENTITY,
+    SKIP_ALIAS_ENGAGEMENT,
     SKIP_IDENTITY_CAP,
     SKIP_MIN_COMMENTS,
     SKIP_NO_AUTHOR,
@@ -34,7 +44,7 @@ from devplacepy.services.sync.spec import (
 )
 
 KIND_FEED = "feed"
-LEDGER_KINDS = (KIND_USERS, KIND_POSTS, KIND_COMMENTS, KIND_FEED)
+LEDGER_KINDS = (*KINDS, KIND_FEED)
 
 OUTCOME_CREATED = "created"
 OUTCOME_UPDATED = "updated"
@@ -43,6 +53,7 @@ OUTCOME_STALE = "stale"
 OUTCOME_SKIPPED = "skipped"
 OUTCOME_REJECTED = "rejected"
 OUTCOME_FAILED = "failed"
+OUTCOME_DEFERRED = "deferred"
 OUTCOME_WARNING = "warning"
 OUTCOME_ERROR = "error"
 
@@ -69,6 +80,8 @@ LEVEL_TITLES = {
 REASON_ARTICLE_UNREACHABLE = "article_unreachable"
 REASON_TRUNCATED = "truncated"
 REASON_COLLECT_ERROR = "collect_error"
+REASON_ATTACHMENT_UNREACHABLE = "attachment_unreachable"
+REASON_ATTACHMENT_TOO_LARGE = "attachment_too_large"
 
 
 @dataclass(frozen=True)
@@ -101,6 +114,7 @@ OUTCOMES: tuple[Outcome, ...] = (
     Outcome(OUTCOME_UPDATED, "Updated", TONE_GOOD, True, "Changed at the origin and updated here."),
     Outcome(OUTCOME_UNCHANGED, "Unchanged", TONE_MUTED, True, "Already synced and identical (only counted), or held for a reason such as deleted here (listed with the reason)."),
     Outcome(OUTCOME_STALE, "Stale", TONE_MUTED, True, "The origin sent an older version than the one already stored."),
+    Outcome(OUTCOME_DEFERRED, "Waiting", TONE_WARN, True, "Parked until a record it depends on arrives, then applied by itself."),
     Outcome(OUTCOME_SKIPPED, "Skipped", TONE_WARN, True, "Held back before sending by the selection, aliasing or translation rules."),
     Outcome(OUTCOME_WARNING, "Warning", TONE_WARN, True, "Synced, but something degraded on the way."),
     Outcome(OUTCOME_REJECTED, "Rejected", TONE_BAD, True, "Refused by validation, the content filter or an earlier moderation decision."),
@@ -123,6 +137,14 @@ REASON_TEXT: dict[str, tuple[str, str]] = {
     REASON_LOCALLY_DELETED: ("Deleted here", "A moderator or the owner deleted it on this side. Moderation wins over the origin."),
     REASON_OLDER_THAN_SYNCED: ("Older than stored", "The origin sent a version older than the one already stored, so it was ignored."),
     REASON_TOMBSTONE_UNKNOWN: ("Deleted at the origin, never synced", "The origin reports a deletion for a record that was never synced here. Nothing to remove."),
+    REASON_UNKNOWN_TARGET: ("Target not synced", "The post, comment or project this vote, reaction or attachment belongs to is not synced here yet."),
+    REASON_UNKNOWN_POLL: ("Poll not synced", "The poll this ballot belongs to is not synced here yet."),
+    REASON_UNKNOWN_OPTION: ("Option unknown", "The poll has no such option, so the ballot cannot be counted."),
+    REASON_UNKNOWN_PROJECT: ("Project not synced", "The project this post links to is not synced here yet."),
+    REASON_SELF_FOLLOW: ("Follows itself", "An account cannot follow itself."),
+    REASON_IMMUTABLE: ("Cannot change", "An attachment cannot change. Send a new id, or a deletion."),
+    REASON_ACK_DELETED: ("Deleted here, acknowledged", "This record is gone on this side and the origin has acknowledged it, so it is not reported again."),
+    REASON_QUEUE_FULL: ("Waiting list full", "The waiting list for this origin is full, so the record was not parked."),
     REASON_INTERNAL: ("Internal error", "Something unexpected broke while processing this record. The run log has the traceback."),
     SKIP_SAMPLED_OUT: ("Sampled out", "Outside the percentage of new posts this community accepts. The same post always gets the same answer."),
     SKIP_MIN_COMMENTS: ("Too few comments", "Fewer comments were collected this run than the minimum you set."),
@@ -132,9 +154,12 @@ REASON_TEXT: dict[str, tuple[str, str]] = {
     SKIP_IDENTITY_CAP: ("Identity cap reached", "The per-run limit of new pseudonyms was reached. Retried on the next run."),
     SKIP_TRANSLATION: ("Translation failed", "The translation did not come back or the per-run translation limit was reached. Retried on the next run."),
     SKIP_REWRITE: ("Rewrite failed", "Replacing original names failed. The item is dropped rather than stored under a real name. Retried on the next run."),
+    SKIP_ALIAS_ENGAGEMENT: ("Not sent under pseudonyms", "Votes and reactions name the people who gave them, so they are not sent while identity aliasing is on."),
     SKIP_PARENT: ("Parent was skipped", "The post or comment this depends on was skipped, so this one was too."),
     REASON_ARTICLE_UNREACHABLE: ("Article unreachable", "The original article could not be fetched. The post was imported with the text from the listing."),
     REASON_TRUNCATED: ("Cut to the limit", "The text was longer than the platform allows and was shortened. The source link is kept."),
+    REASON_ATTACHMENT_UNREACHABLE: ("Attachment unreachable", "A file linked from the post could not be downloaded, so it is not attached here."),
+    REASON_ATTACHMENT_TOO_LARGE: ("Attachment too large", "A file linked from the post is over the size limit, so it is not attached here."),
     REASON_COLLECT_ERROR: ("Source error", "A page or feed could not be read. The detail has the address and the answer."),
 }
 RETRIED = frozenset(TRANSIENT_SKIPS)
@@ -156,7 +181,10 @@ def label_for(kind: str, record: dict) -> str:
         return clip(record.get("username"), LEDGER.label_chars)
     if kind == KIND_POSTS:
         return clip(record.get("title") or record.get("content"), LEDGER.label_chars)
-    return clip(record.get("content") or record.get("title") or record.get("username"), LEDGER.label_chars)
+    return clip(
+        record.get("content") or record.get("title") or record.get("username") or record.get("question") or record.get("filename") or record.get("emoji") or record.get("origin_uid"),
+        LEDGER.label_chars,
+    )
 
 
 @dataclass
@@ -250,8 +278,10 @@ def verdict(status: str, dry_run: bool, tally: dict) -> dict:
         return {"level": LEVEL_ATTENTION, "headline": "Not attempted: community sync is switched off or the origin is not allowed." + tail}
     if status == "failed":
         return {"level": LEVEL_BAD, "headline": "The run failed before anything could be synced." + tail}
+    if status == "interrupted":
+        return {"level": LEVEL_BAD, "headline": "Interrupted: the process ended before this run finished. What was synced up to then is listed." + tail}
     bad = [f"{tally[key]} {key}" for key in (OUTCOME_FAILED, OUTCOME_ERROR, OUTCOME_REJECTED) if tally.get(key)]
-    warn = [f"{tally[key]} {key}" for key in (OUTCOME_SKIPPED, OUTCOME_WARNING) if tally.get(key)]
+    warn = [f"{tally[key]} {key}" for key in (OUTCOME_DEFERRED, OUTCOME_SKIPPED, OUTCOME_WARNING) if tally.get(key)]
     if tally.get(OUTCOME_FAILED) or tally.get(OUTCOME_ERROR):
         return {"level": LEVEL_BAD, "headline": ", ".join(bad + warn).capitalize() + "." + tail}
     if bad or warn:

devplacepy/services/community/ledger_store.py

@@ -10,6 +10,7 @@ from devplacepy.services.community.ledger import (
     REASON_TEXT,
     Ledger,
 )
+from devplacepy.services.community.spec import LIMITS
 from devplacepy.services.community.store import ITEMS_TABLE, iso, now
 from devplacepy.utils.text import generate_uid
 
@@ -21,9 +22,26 @@ def escape_like(text: str) -> str:
     return text.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
 
 
+SEARCH_TABLE = "community_run_items_fts"
+
+
 class LedgerStore:
+    def __init__(self) -> None:
+        self._search: bool | None = None
+
+    def search_ready(self) -> bool:
+        if not self._search:
+            self._search = TABLE in db.tables and bool(
+                list(db.query("SELECT name FROM sqlite_master WHERE name = :name", name=SEARCH_TABLE))
+            )
+        return self._search
+
     def save(self, run_uid: str, community_uid: str, ledger: Ledger) -> int:
-        if not ledger.items:
+        return self.append(run_uid, community_uid, ledger, 0)
+
+    def append(self, run_uid: str, community_uid: str, ledger: Ledger, offset: int) -> int:
+        drafts = ledger.items[offset:]
+        if not drafts:
             return 0
         stamp = iso(now())
         rows = [
@@ -31,7 +49,7 @@ class LedgerStore:
                 "uid": generate_uid(),
                 "run_uid": run_uid,
                 "community_uid": community_uid,
-                "seq": seq,
+                "seq": offset + seq,
                 "rank": draft.rank,
                 "kind": draft.kind,
                 "outcome": draft.outcome,
@@ -43,7 +61,7 @@ class LedgerStore:
                 "detail": draft.detail,
                 "created_at": stamp,
             }
-            for seq, draft in enumerate(ledger.items)
+            for seq, draft in enumerate(drafts)
         ]
         get_table(TABLE).insert_many(rows, chunk_size=INSERT_CHUNK)
         return len(rows)
@@ -93,7 +111,10 @@ class LedgerStore:
             clauses.append("reason = :reason")
             params["reason"] = str(filters["reason"])[:80]
         text = str(filters.get("q") or "").strip()[:80]
-        if text:
+        if text and not run_uid and len(text) >= LIMITS.search_min_chars and self.search_ready():
+            clauses.append(f"id IN (SELECT rowid FROM {SEARCH_TABLE} WHERE {SEARCH_TABLE} MATCH :q)")
+            params["q"] = '"' + text.replace('"', '""') + '"'
+        elif text:
             clauses.append("(label LIKE :q ESCAPE '\\' OR origin_uid LIKE :q ESCAPE '\\' OR detail LIKE :q ESCAPE '\\')")
             params["q"] = f"%{escape_like(text)}%"
         narrowed = "outcome" in params

devplacepy/services/community/pusher.py

@@ -10,7 +10,7 @@ from devplacepy.services.community.ledger import OUTCOME_CREATED, OUTCOME_UNCHAN
 from devplacepy.services.community.spec import LIMITS
 from devplacepy.services.sync.batch import BatchRunner
 from devplacepy.services.sync.records import SyncBatchForm
-from devplacepy.services.sync.spec import KINDS, REASON_TOMBSTONE_UNKNOWN, STATUSES
+from devplacepy.services.sync.spec import DEFERRABLE_REASONS, KINDS, REASON_TOMBSTONE_UNKNOWN, STATUSES
 
 logger = logging.getLogger("community.push")
 
@@ -25,18 +25,12 @@ class PushReport:
         return sum(per.get(status, 0) for per in self.counts.values())
 
 
-DEPENDENCIES = {
-    "unknown_author": ("users", "author_uid"),
-    "unknown_post": ("posts", "post_uid"),
-    "unknown_parent": ("comments", "parent_uid"),
-}
-
-
 class SyncPusher:
-    def __init__(self, origin: str, dry_run: bool = False, ledger: Ledger | None = None):
+    def __init__(self, origin: str, dry_run: bool = False, ledger: Ledger | None = None, report: PushReport | None = None):
         self.origin = origin
         self.dry_run = dry_run
         self.ledger = ledger
+        self.live = report
 
     def note(self, kind: str, result, raw: dict, outcome: str, reason: str | None = None) -> None:
         if self.ledger is None:
@@ -56,23 +50,27 @@ class SyncPusher:
 
     @staticmethod
     def satisfied_by_projection(result, raw: dict, projected: dict) -> bool:
-        if result.status != "rejected" or result.reason not in DEPENDENCIES:
+        if result.status != "rejected" or result.reason not in DEFERRABLE_REASONS:
             return False
-        kind, field_name = DEPENDENCIES[result.reason]
-        return str(raw.get(field_name)) in projected[kind]
+        return result.needs_uid in projected.get(result.needs_kind, ())
 
     def push(self, records: dict[str, list[dict]]) -> PushReport:
-        report = PushReport(dry_run=self.dry_run)
+        report = self.live or PushReport(dry_run=self.dry_run)
+        report.dry_run = self.dry_run
         mode = "DRY RUN" if self.dry_run else "LIVE"
         projected: dict[str, set[str]] = {kind: set() for kind in KINDS}
         for kind in KINDS:
             rows = records.get(kind, [])
+            if not rows:
+                report.counts[kind] = {}
+                continue
             tally: Counter = Counter()
             logger.info("push[%s]: %s %d %s record(s) in batches of %d", mode, self.origin, len(rows), kind, LIMITS.batch_records)
             for start in range(0, len(rows), LIMITS.batch_records):
                 chunk = rows[start : start + LIMITS.batch_records]
-                outcome = BatchRunner(kind).run(SyncBatchForm(origin=self.origin, records=chunk, dry_run=self.dry_run))
+                outcome = BatchRunner(kind, humanize=True).run(SyncBatchForm(origin=self.origin, records=chunk, dry_run=self.dry_run))
                 tally.update(outcome.counts)
+                report.counts[kind] = {status: tally.get(status, 0) for status in STATUSES if tally.get(status, 0)}
                 for result, raw in zip(outcome.results, chunk):
                     if result.status == "created":
                         projected[kind].add(result.origin_uid)

devplacepy/services/community/runner.py

@@ -7,16 +7,19 @@ import logging
 from dataclasses import dataclass, field
 
 from devplacepy.services.sync.aliaser import AliasRefused, Aliaser
+from devplacepy.services.sync.healing import AliasHealer
 from devplacepy.services.sync.options import SyncOptions, alias_allowed
 from devplacepy.services.sync.selection import Selector
+from devplacepy.services.community.checkpoint import RunCheckpoint
 from devplacepy.services.community.discoverer import CommunityDiscoverer, Discovery
 from devplacepy.services.community.fetcher import Fetcher
 from devplacepy.services.community.files import CommunityFiles, files as default_files
 from devplacepy.services.community.ledger import Ledger, tally_of, verdict
 from devplacepy.services.community.ledger_store import LedgerStore, ledger_store as default_ledger_store
 from devplacepy.services.community.logbook import LogBook
-from devplacepy.services.community.pusher import SyncPusher
-from devplacepy.services.community.runtime import LoginRequired
+from devplacepy.services.community.pusher import PushReport, SyncPusher
+from devplacepy.services.community.runtime import Collector, LoginRequired
+from devplacepy.services.community.schemacheck import sample_limits
 from devplacepy.services.community.spec import (
     LIMITS,
     RUN_DRY,
@@ -29,6 +32,7 @@ from devplacepy.services.community.spec import (
 )
 from devplacepy.services.community.store import CommunityStore, iso, now, store as default_store
 from devplacepy.services.sync.gate import refusal
+from devplacepy.services.sync.spec import SKIP_ALIAS_ENGAGEMENT, SKIP_PARENT
 
 logger = logging.getLogger("community.run")
 
@@ -36,6 +40,20 @@ TRIGGER_CLI = "cli"
 TRIGGER_ADMIN = "admin"
 TRIGGER_SERVICE = "service"
 KINDS = ("users", "posts", "comments")
+COLLECTED_KINDS = (*KINDS, "votes", "reactions", "attachments")
+ENGAGEMENT_KINDS = ("votes", "reactions", "attachments")
+
+
+def options_alias_active(row: dict) -> bool:
+    return SyncOptions.from_row(row).alias
+
+
+@dataclass
+class SchemaVerdict:
+    ok: bool
+    posts: int
+    comments: int
+    problems: list
 
 
 @dataclass
@@ -49,6 +67,7 @@ class RunReport:
     run_uid: str = ""
     watermark: dict = field(default_factory=dict)
     ledger: Ledger = field(default_factory=Ledger)
+    live: PushReport = field(default_factory=lambda: PushReport(dry_run=False))
 
 
 class CommunityRunner:
@@ -120,6 +139,7 @@ class CommunityRunner:
                 login_username=credentials.get("username", ""),
                 login_email=credentials.get("email", ""),
                 login_password=credentials.get("password", ""),
+                login_totp_secret=credentials.get("totp", ""),
             )
         if existing:
             row = self.store.update(existing["uid"], **fields, enabled=1 if result.status == STATUS_READY else 0)
@@ -130,9 +150,13 @@ class CommunityRunner:
         return row
 
     async def run(self, row: dict, *, dry_run: bool, trigger: str) -> RunReport:
+        self.store.reap_interrupted()
         run_uid = self.store.start_run(row["uid"], trigger, dry_run)
         report = RunReport(status=RUN_FAILED, dry_run=dry_run, run_uid=run_uid)
+        report.live.dry_run = dry_run
         with LogBook() as book:
+            checkpoint = RunCheckpoint(self.store, self.ledgers, run_uid, row["uid"], report, report.live, book)
+            ticker = asyncio.create_task(checkpoint.loop())
             mode = "DRY RUN" if dry_run else "LIVE"
             logger.info("run[%s]: %s (%s) trigger=%s", mode, row["name"], row["origin"], trigger)
             try:
@@ -152,12 +176,14 @@ class CommunityRunner:
             for error in report.errors:
                 logger.error("run[%s]: %s", mode, error)
             logger.info("run[%s]: finished with status %s", mode, report.status)
+            ticker.cancel()
+            await asyncio.gather(ticker, return_exceptions=True)
             report.log = book.text()
         if not dry_run:
             self.settle(row, report)
         tally = tally_of(report.pushed, report.ledger)
         outcome = verdict(report.status, report.dry_run, tally)
-        stored = self.ledgers.save(run_uid, row["uid"], report.ledger)
+        stored = checkpoint.saved + self.ledgers.append(run_uid, row["uid"], report.ledger, checkpoint.saved)
         self.store.finish_run(
             run_uid,
             report.status,
@@ -174,6 +200,27 @@ class CommunityRunner:
         self.store.prune_runs(row["uid"])
         return report
 
+    async def verify_schema(self, row: dict, schema: dict) -> "SchemaVerdict":
+        sample = {**schema, "limits": sample_limits(schema)}
+        try:
+            async with self.fetcher_factory() as fetcher:
+                result = await asyncio.wait_for(
+                    Collector(fetcher, sample, None, sleep=self.sleep, credentials=self.store.credentials(row)).run(),
+                    timeout=LIMITS.run_timeout_seconds,
+                )
+        except asyncio.TimeoutError:
+            return SchemaVerdict(False, 0, 0, [f"the verification run exceeded {LIMITS.run_timeout_seconds}s"])
+        except LoginRequired as exc:
+            return SchemaVerdict(False, 0, 0, [str(exc)])
+        except Exception as exc:
+            logger.exception("schema verification crashed")
+            return SchemaVerdict(False, 0, 0, [f"{type(exc).__name__}: {exc}"])
+        posts, comments = len(result["records"]["posts"]), len(result["records"]["comments"])
+        problems = list(result.get("errors", []))
+        if not posts:
+            problems.append("the schema collected no posts from the live source")
+        return SchemaVerdict(bool(result["ok"]) and posts > 0, posts, comments, problems)
+
     async def execute(self, row: dict, report: RunReport) -> None:
         refused = refusal(row["origin"])
         if refused and not report.dry_run:
@@ -183,10 +230,17 @@ class CommunityRunner:
         if refused:
             logger.warning("run[DRY RUN]: %s; continuing because nothing is written", refused)
         module = self.files.load(row["slug"], row["schema"])
+        options = SyncOptions.from_row(row)
         async with self.fetcher_factory() as fetcher:
-            result = await module.collect_all(fetcher, row.get("watermark") or None, sleep=self.sleep, credentials=self.store.credentials(row))
+            result = await module.collect_all(
+                fetcher,
+                row.get("watermark") or None,
+                sleep=self.sleep,
+                credentials=self.store.credentials(row),
+                options={"attachments": options.collect_attachments},
+            )
         records = result["records"]
-        report.collected = {kind: len(records[kind]) for kind in KINDS}
+        report.collected = {kind: len(records.get(kind, [])) for kind in COLLECTED_KINDS if records.get(kind) or kind in KINDS}
         logger.info("run: collected %s", report.collected)
         report.errors.extend(result.get("errors", []))
         for warning in result.get("warnings", []):
@@ -199,13 +253,14 @@ class CommunityRunner:
         if not result["ok"]:
             report.errors.append("no feed could be collected")
             return
-        options = SyncOptions.from_row(row)
         prepared = await self.prepare(row, options, records, report)
         if prepared is None:
             report.status = RUN_SKIPPED
             return
         records, skipped, held = prepared
-        push = await asyncio.to_thread(SyncPusher(row["origin"], dry_run=report.dry_run, ledger=report.ledger).push, records)
+        push = await asyncio.to_thread(SyncPusher(row["origin"], dry_run=report.dry_run, ledger=report.ledger, report=report.live).push, records)
+        if options_alias_active(row) and not report.dry_run:
+            self.heal(row["origin"])
         report.pushed = push.counts | ({"skipped": dict(skipped)} if skipped else {})
         report.errors.extend(push.problems[:20])
         report.watermark = {} if held else result["state"]
@@ -215,8 +270,20 @@ class CommunityRunner:
         failed = push.total("failed")
         report.status = RUN_DRY if report.dry_run else (RUN_PARTIAL if report.errors or failed else RUN_OK)
 
+    @staticmethod
+    def heal(origin: str) -> None:
+        healed = AliasHealer(origin).heal()
+        if healed.renamed:
+            logger.info("alias: %d alias(es) followed their clone, %d stored text(s) rewritten", len(healed.renamed), healed.rewritten)
+        if healed.conflicts:
+            logger.warning("alias: %d alias(es) could not follow their clone: %s", len(healed.conflicts), ", ".join(healed.conflicts))
+
     async def prepare(self, row: dict, options: SyncOptions, records: dict, report: RunReport):
-        selector = Selector(row["origin"], options)
+        selector = Selector(row["origin"], options, persist=not report.dry_run)
+        if options.alias and not report.dry_run:
+            self.heal(row["origin"])
+        extras = {kind: records.get(kind, []) for kind in ENGAGEMENT_KINDS}
+        collected_users = list(records.get("users", []))
         records, skipped = selector.select(records)
         for kind, record, reason in selector.dropped:
             report.ledger.add_dropped_record(kind, record, reason)
@@ -235,10 +302,35 @@ class CommunityRunner:
                 report.ledger.add_dropped_record(kind, record, reason)
             skipped.update(outcome.skipped)
             logger.info("alias: %d substitution(s), %d translated, %d new identities", outcome.substitutions, outcome.translated, outcome.identities_created)
+        records = self.attach_engagement(records, extras, collected_users, options, skipped, report)
         if skipped:
             logger.info("select: skipped %s", dict(skipped))
         return records, skipped, held
 
+    @staticmethod
+    def attach_engagement(records: dict, extras: dict, collected_users: list, options: SyncOptions, skipped, report: RunReport) -> dict:
+        authors = {("post", str(post.get("origin_uid"))): post.get("author_uid") for post in records.get("posts", [])}
+        authors |= {("comment", str(comment.get("origin_uid"))): comment.get("author_uid") for comment in records.get("comments", [])}
+        merged = dict(records)
+        for kind in ENGAGEMENT_KINDS:
+            kept = []
+            for record in extras.get(kind, []):
+                target = (record.get("target_kind"), str(record.get("target_uid")))
+                if target not in authors:
+                    skipped[SKIP_PARENT] += 1
+                    report.ledger.add_dropped_record(kind, record, SKIP_PARENT)
+                elif options.alias and kind != "attachments":
+                    skipped[SKIP_ALIAS_ENGAGEMENT] += 1
+                    report.ledger.add_dropped_record(kind, record, SKIP_ALIAS_ENGAGEMENT)
+                else:
+                    kept.append({**record, "author_uid": authors[target]} if kind == "attachments" and options.alias else record)
+            merged[kind] = kept
+        needed = {str(item.get(field)) for kind in ("votes", "reactions") for item in merged.get(kind, []) for field in ("voter_uid", "author_uid")}
+        if needed:
+            known = {str(user.get("origin_uid")) for user in merged.get("users", [])}
+            merged["users"] = [*merged.get("users", []), *[user for user in collected_users if str(user.get("origin_uid")) in needed - known]]
+        return merged
+
     def settle(self, row: dict, report: RunReport) -> None:
         current = self.store.get(row["uid"])
         if current is None:

devplacepy/services/community/runtime.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import asyncio
+import base64
 import hashlib
 import json
 import re
@@ -12,8 +13,13 @@ from urllib.parse import urljoin
 from devplacepy.services.community.extract import (
     Node,
     canonical_link,
+    detect_captcha,
+    extract_json_ld_posts,
+    split_submitter,
     field_text,
     find_login_form,
+    find_next_page,
+    find_otp_form,
     html_field,
     json_or_none,
     normalize_instant,
@@ -23,6 +29,7 @@ from devplacepy.services.community.extract import (
     select_nodes,
     strip_tags,
     text_at,
+    totp,
 )
 
 UID_PATTERN = re.compile(r"^[A-Za-z0-9._:@-]+$")
@@ -35,7 +42,11 @@ MAX_COMMENT_LENGTH = 125000
 MAX_REPLY_DEPTH = 5
 RETRY_STATUSES = frozenset({408, 425, 429, 500, 502, 503, 504})
 RENDER_SETTLE_MS = 800
-NEXT_PAGE_FIELDS = ("a[rel=next]::attr(href)", "link[rel=next]::attr(href)")
+ATTACHMENT_EXTENSIONS = (".png", ".jpg", ".jpeg", ".gif", ".webp", ".pdf")
+ATTACHMENT_TYPES = ("image/", "application/pdf")
+HTML_FILE_URL = re.compile(r"""(?:src|href|data-src|data-original)\s*=\s*["']([^"'\s>]+)["']""", re.I)
+DOWN_VOTES = re.compile(r"^(-1|-|down|downvote|dislike|negative|minus)$", re.I)
+DEFAULT_OPTIONS = {"attachments": False, "max_attachments": 40, "max_attachments_per_post": 4, "max_attachment_bytes": 2_000_000, "max_attachment_total": 25_000_000}
 DEFAULT_LIMITS = {
     "max_pages": 3,
     "max_posts": 150,
@@ -47,6 +58,7 @@ DEFAULT_LIMITS = {
     "backoff_seconds": 2.0,
     "timeout_seconds": 30,
     "max_bytes": 5_000_000,
+    "max_remembered_links": 20_000,
 }
 
 
@@ -62,10 +74,34 @@ class LoginFailed(LoginRequired):
     pass
 
 
+class CaptchaRequired(LoginRequired):
+    pass
+
+
+class SecondFactorRequired(LoginFailed):
+    pass
+
+
 class PageMissing(CollectError):
     pass
 
 
+CODE_KEYS = ("otp_required", "totp_required", "two_factor", "two_factor_required", "requires_2fa", "mfa_required", "second_factor", "needs_otp")
+CODE_PHRASES = re.compile(r"(one[- ]time (?:code|password)|verification code|authenticator|two[- ]factor|2fa code|otp)", re.I)
+
+
+def code_challenge(document, text: str, status: int) -> bool:
+    if isinstance(document, dict):
+        lowered = {str(key).lower(): value for key, value in document.items()}
+        if any(lowered.get(key) for key in CODE_KEYS):
+            return True
+    return status in (400, 401, 403, 422) and bool(CODE_PHRASES.search(text or ""))
+
+
+def url_hint(feed: dict) -> str:
+    return feed.get("url") or ""
+
+
 def uid_for(value) -> str:
     text = str(value if value is not None else "").strip()
     if text and len(text) <= UID_MAX_LENGTH and UID_PATTERN.match(text):
@@ -88,10 +124,16 @@ def flatten_matches(matches: list) -> list:
 
 
 class Collector:
-    def __init__(self, stealth, schema: dict, state: dict | None = None, sleep=None, credentials: dict | None = None):
+    def __init__(self, stealth, schema: dict, state: dict | None = None, sleep=None, credentials: dict | None = None, options: dict | None = None):
         self.stealth = stealth
+        self.options = {**DEFAULT_OPTIONS, **(options or {})}
+        self.votes: dict[str, dict] = {}
+        self.reactions: dict[str, dict] = {}
+        self.attachments: list[dict] = []
+        self.file_candidates: list[dict] = []
         self.schema = schema
-        self.state = {"feeds": dict((state or {}).get("feeds", {}))}
+        remembered = (state or {}).get("links") or {}
+        self.state = {"feeds": dict((state or {}).get("feeds", {})), "links": dict(remembered)}
         self.sleep = sleep or asyncio.sleep
         self.credentials = credentials or {}
         self.limits = {**DEFAULT_LIMITS, **schema.get("limits", {})}
@@ -105,8 +147,13 @@ class Collector:
         self.warnings: list[str] = []
         self.notices: list[dict] = []
         self.sources: dict[str, str] = {}
-        self.link_owner: dict[str, tuple[str, str]] = {}
+        self.link_owner: dict[str, tuple[str, str]] = {
+            key: (value[0], value[1])
+            for key, value in remembered.items()
+            if isinstance(value, (list, tuple)) and len(value) == 2
+        }
         self.aliases: dict[str, str] = {}
+        self.headers: dict[str, str] = {}
 
     async def attempt(self, label: str, action) -> str:
         last = "no attempt"
@@ -138,18 +185,22 @@ class Collector:
             return await self.attempt(url, lambda: self.render_html(url))
 
         async def action():
-            response = await self.stealth.get(url, params=params, timeout=self.limits["timeout_seconds"], session=self.session)
+            response = await self.stealth.get(url, params=params, timeout=self.limits["timeout_seconds"], session=self.session, headers=self.headers or None)
             return int(response.status), await response.text()
 
         return await self.attempt(url, action)
 
     async def post(self, url: str, data: dict) -> str:
         async def action():
-            response = await self.stealth.post(url, data=data, timeout=self.limits["timeout_seconds"], session=self.session)
+            response = await self.stealth.post(url, data=data, timeout=self.limits["timeout_seconds"], session=self.session, headers=self.headers or None)
             return int(response.status), await response.text()
 
         return await self.attempt(url, action)
 
+    async def post_json(self, url: str, body: dict) -> tuple[int, str]:
+        response = await self.stealth.post(url, json=body, timeout=self.limits["timeout_seconds"], session=self.session, headers=self.headers or None)
+        return int(response.status), await response.text()
+
     async def render_html(self, url: str):
         async with self.stealth.page(session=self.session) as page:
             response = await page.goto(url, wait_until="domcontentloaded")
@@ -165,6 +216,9 @@ class Collector:
     def has_credentials(self) -> bool:
         return bool(self.credentials.get("password") and (self.credentials.get("username") or self.credentials.get("email")))
 
+    def community_name(self) -> str:
+        return self.schema.get("community", "community")
+
     async def login(self) -> bool:
         spec = self.schema.get("login")
         if not spec or not self.has_credentials():
@@ -172,26 +226,77 @@ class Collector:
         identity = self.credentials.get("email") if spec.get("identity") == "email" else self.credentials.get("username")
         identity = identity or self.credentials.get("username") or self.credentials.get("email")
         render = bool(spec.get("render"))
+        if spec.get("captcha"):
+            raise CaptchaRequired(f"{self.community_name()} protects its login with a captcha, which cannot be automated; sign in by hand or use a site that offers an API")
+        if spec.get("mode") == "json":
+            await self.login_json(spec, identity)
+            return True
         if spec.get("mode") == "form":
-            root = parse_html(await self.fetch(spec["url"], render=render))
-            form = find_login_form(root, spec["url"]) or spec
-            data = {**form.get("fields", {}), form["username_field"]: identity, form["password_field"]: self.credentials["password"]}
-            if str(form.get("method", "POST")).upper() == "GET":
-                await self.fetch(form["action"], params=data)
-            else:
-                await self.post(form["action"], data)
+            await self.login_form(spec, identity, render)
         else:
             async with self.stealth.page(session=self.session) as page:
                 await page.goto(spec["url"], wait_until="domcontentloaded")
+                if detect_captcha(await page.content()):
+                    raise CaptchaRequired(f"{self.community_name()} protects its login with a captcha, which cannot be automated")
                 await page.fill(spec["username_selector"], identity)
                 await page.fill(spec["password_selector"], self.credentials["password"])
                 await page.click(spec["submit_selector"])
                 await page.wait_for_timeout(RENDER_SETTLE_MS * 2)
-        after = parse_html(await self.fetch(spec.get("check_url") or self.schema.get("base_url") or spec["url"], render=render))
-        if select_nodes(after, "input[type=password]"):
-            raise LoginFailed(f"{self.schema.get('community', 'community')} rejected the stored credentials")
+        after_text = await self.fetch(spec.get("check_url") or self.schema.get("base_url") or spec["url"], render=render)
+        if select_nodes(parse_html(after_text), "input[type=password]"):
+            if detect_captcha(after_text):
+                raise CaptchaRequired(f"{self.community_name()} asked for a captcha while signing in, which cannot be automated")
+            raise LoginFailed(f"{self.community_name()} rejected the stored credentials")
         return True
 
+    async def login_form(self, spec: dict, identity: str, render: bool) -> None:
+        page_text = await self.fetch(spec["url"], render=render)
+        if detect_captcha(page_text):
+            raise CaptchaRequired(f"{self.community_name()} protects its login with a captcha, which cannot be automated; sign in by hand or use a site that offers an API")
+        form = find_login_form(parse_html(page_text), spec["url"]) or spec
+        data = {**form.get("fields", {}), form["username_field"]: identity, form["password_field"]: self.credentials["password"]}
+        if str(form.get("method", "POST")).upper() == "GET":
+            answer = await self.fetch(form["action"], params=data)
+        else:
+            answer = await self.post(form["action"], data)
+        await self.second_factor(answer, form["action"])
+
+    async def second_factor(self, answer: str, base_url: str) -> None:
+        form = find_otp_form(parse_html(answer), base_url)
+        if form is None:
+            return
+        secret = self.credentials.get("totp")
+        if not secret:
+            raise SecondFactorRequired(
+                f"{self.community_name()} asks for a one-time code; add the account's authenticator secret (TOTP) to the community. A code sent by SMS or push cannot be automated"
+            )
+        data = {**form["fields"], form["field"]: totp(secret)}
+        reply = await (self.fetch(form["action"], params=data) if form["method"] == "GET" else self.post(form["action"], data))
+        if find_otp_form(parse_html(reply), form["action"]):
+            raise SecondFactorRequired(f"{self.community_name()} rejected the one-time code; check the authenticator secret and the clock of this server")
+
+    async def login_json(self, spec: dict, identity: str) -> None:
+        body = {**spec.get("fields", {}), spec["username_field"]: identity, spec["password_field"]: self.credentials["password"]}
+        status, text = await self.post_json(spec["url"], body)
+        document = json_or_none(text)
+        if code_challenge(document, text, status):
+            secret = self.credentials.get("totp")
+            if not secret:
+                raise SecondFactorRequired(
+                    f"{self.community_name()} asks for a one-time code; add the account's authenticator secret (TOTP) to the community. A code sent by SMS or push cannot be automated"
+                )
+            status, text = await self.post_json(spec["url"], {**body, spec.get("otp_field") or "otp": totp(secret)})
+            document = json_or_none(text)
+            if code_challenge(document, text, status):
+                raise SecondFactorRequired(f"{self.community_name()} rejected the one-time code; check the authenticator secret and the clock of this server")
+        if status >= 400:
+            raise LoginFailed(f"{self.community_name()} rejected the stored credentials (status {status})")
+        token = text_at(document, spec["token_path"]) if spec.get("token_path") and document is not None else ""
+        if spec.get("token_path") and not token:
+            raise LoginFailed(f"{self.community_name()} answered the login without the expected token")
+        if token:
+            self.headers["Authorization"] = f"{spec.get('token_scheme') or 'Bearer'} {token}"
+
     def parse(self, feed: dict, text: str):
         kind = feed["type"]
         if kind == "json_api":
@@ -199,6 +304,9 @@ class Collector:
             if document is None:
                 raise CollectError(f"{feed['url']} did not return JSON")
             return document, flatten_matches(select(document, feed.get("list_path") or "$"))
+        if kind == "json_ld":
+            items = extract_json_ld_posts(parse_html(text))
+            return items, items
         if kind in ("rss", "atom"):
             _, meta, items = parse_feed(text)
             return {"meta": meta}, items
@@ -347,6 +455,7 @@ class Collector:
             return owner[0]
         if key:
             self.link_owner.setdefault(key, (post_uid, feed["id"]))
+            self.remember_link(key, self.link_owner[key])
         record = {
             "origin_uid": post_uid,
             "author_uid": self.add_author(feed.get("author") or {}, item, created),
@@ -357,15 +466,24 @@ class Collector:
         if updated:
             record["updated_at"] = updated
         self.posts[post_uid] = record
+        self.add_engagement(feed["post"], item, "post", post_uid, created)
+        self.note_files(item, detail, "post", post_uid, record["author_uid"], created, link or url_hint(feed))
         if link:
             self.sources[post_uid] = link
         if cut:
             self.notice("posts", post_uid, "truncated", title, f"{cut} characters over the {MAX_POST_LENGTH} limit were cut")
         return post_uid
 
+    def remember_link(self, key: str, owner: tuple[str, str]) -> None:
+        links = self.state["links"]
+        links.pop(key, None)
+        links[key] = list(owner)
+        while len(links) > int(self.limits["max_remembered_links"]):
+            links.pop(next(iter(links)))
+
     def add_author(self, spec: dict, item, moment: str) -> str:
         fields = spec.get("fields", {})
-        username = strip_tags(field_text(item, fields.get("username")))
+        username = split_submitter(strip_tags(field_text(item, fields.get("username"))))
         identifier = field_text(item, fields.get("id"))
         if not username and not identifier:
             username, identifier = "anonymous", "anonymous"
@@ -395,9 +513,7 @@ class Collector:
                     self.add_comments(spec, flatten_matches(select(item, spec["list_path"])), post_uid, None, 0)
                 elif mode == "url":
                     target = spec["url_template"].replace("{id}", identifier).replace("{uid}", post_uid)
-                    document = json_or_none(await self.fetch(urljoin(base, target), render=bool(spec.get("render"))))
-                    entries = flatten_matches(select(document, spec.get("list_path") or "$")) if document is not None else []
-                    self.add_comments(spec, entries, post_uid, None, 0)
+                    await self.collect_comment_documents(spec, urljoin(base, target), post_uid)
                 elif mode == "page":
                     link = self.discussion_link(feed, item, base)
                     if link:
@@ -409,6 +525,22 @@ class Collector:
             except CollectError as exc:
                 self.errors.append(f"comments {identifier}: {exc}")
 
+    async def collect_comment_documents(self, spec: dict, link: str, post_uid: str) -> None:
+        visited: set[str] = set()
+        pagination = spec.get("pagination") or {}
+        for _ in range(int(self.limits["max_comment_pages"])):
+            if link in visited:
+                break
+            visited.add(link)
+            document = json_or_none(await self.fetch(link, render=bool(spec.get("render"))))
+            entries = flatten_matches(select(document, spec.get("list_path") or "$")) if document is not None else []
+            before = len(self.comments)
+            self.add_comments(spec, entries, post_uid, None, 0)
+            target = text_at(document, pagination["path"]) if pagination.get("style") == "json_next" and document is not None else ""
+            if len(self.comments) == before or not target:
+                break
+            link = urljoin(link, target)
+
     async def collect_comment_pages(self, feed: dict, spec: dict, link: str, post_uid: str) -> None:
         render = bool(feed.get("render") or spec.get("render"))
         visited: set[str] = set()
@@ -417,11 +549,13 @@ class Collector:
                 break
             visited.add(link)
             root = await self.page_root(link, render)
+            before = len(self.comments)
             self.add_page_comments(spec, root, post_uid)
-            target = next((value for value in (html_field(root, field) for field in NEXT_PAGE_FIELDS) if value), "")
-            if not target:
+            if len(self.comments) == before:
+                break
+            link = find_next_page(root, link, spec.get("pagination"))
+            if not link:
                 break
-            link = urljoin(link, target)
 
     def add_comments(self, spec: dict, entries: list, post_uid: str, parent_uid: str | None, depth: int) -> None:
         fields = spec["fields"]
@@ -482,6 +616,90 @@ class Collector:
         if updated:
             record["updated_at"] = updated
         self.comments[comment_uid] = record
+        self.add_engagement(spec, entry, "comment", comment_uid, created)
+        self.note_files(entry, None, "comment", comment_uid, record["author_uid"], created, "")
+
+    def add_engagement(self, spec: dict, item, target_kind: str, target_uid: str, created: str) -> None:
+        if isinstance(item, Node):
+            return
+        for kind, bucket in (("votes", self.votes), ("reactions", self.reactions)):
+            engagement = spec.get(kind)
+            if not engagement:
+                continue
+            for entry in flatten_matches(select(item, engagement["list_path"])):
+                if not isinstance(entry, dict):
+                    continue
+                fields = engagement.get("fields", {})
+                voter = self.add_author(engagement.get("author") or {}, entry, created)
+                moment = normalize_instant(field_text(entry, fields.get("created_at"))) or created
+                if kind == "votes":
+                    value = -1 if DOWN_VOTES.match(strip_tags(field_text(entry, fields.get("value"))).strip()) else 1
+                    key = f"vote:{target_kind}:{target_uid}:{voter}"
+                    record = {"origin_uid": uid_for(key), "voter_uid": voter, "target_kind": target_kind, "target_uid": target_uid, "value": value, "created_at": moment}
+                else:
+                    emoji = strip_tags(field_text(entry, fields.get("emoji"))).strip()
+                    if not emoji or len(emoji) > 16 or emoji.isalnum():
+                        continue
+                    key = f"reaction:{target_kind}:{target_uid}:{voter}:{emoji}"
+                    record = {"origin_uid": uid_for(key), "author_uid": voter, "target_kind": target_kind, "target_uid": target_uid, "emoji": emoji, "created_at": moment}
+                bucket[record["origin_uid"]] = record
+
+    def note_files(self, item, detail, target_kind: str, target_uid: str, author_uid: str, created: str, base: str) -> None:
+        if not self.options.get("attachments"):
+            return
+        found: list[str] = []
+        for source in (item, detail):
+            if isinstance(source, Node):
+                for node in source.descendants():
+                    for attribute in ("src", "href", "data-src"):
+                        if node.tag in ("img", "a", "source") and node.attrs.get(attribute):
+                            found.append(node.attrs[attribute])
+            elif isinstance(source, (dict, str)):
+                text = json.dumps(source, default=str) if isinstance(source, dict) else source
+                found.extend(HTML_FILE_URL.findall(text.replace("\\/", "/").replace('\\"', '"')))
+        taken = 0
+        seen: set[str] = set()
+        for raw in found:
+            absolute = urljoin(base, raw.strip()) if base else raw.strip()
+            path = absolute.split("?", 1)[0].lower()
+            if not absolute.startswith(("http://", "https://")) or not path.endswith(ATTACHMENT_EXTENSIONS) or absolute in seen:
+                continue
+            seen.add(absolute)
+            self.file_candidates.append({"url": absolute, "target_kind": target_kind, "target_uid": target_uid, "author_uid": author_uid, "created_at": created})
+            taken += 1
+            if taken >= int(self.options["max_attachments_per_post"]):
+                break
+
+    async def download_files(self) -> None:
+        total = 0
+        for candidate in self.file_candidates[: int(self.options["max_attachments"])]:
+            url = candidate["url"]
+            try:
+                response = await self.stealth.get(url, timeout=self.limits["timeout_seconds"], session=self.session, headers=self.headers or None)
+                kind = str(getattr(response, "headers", {}).get("content-type", "")).split(";")[0].strip().lower() if getattr(response, "headers", None) else ""
+                if int(response.status) >= 400 or (kind and not kind.startswith(ATTACHMENT_TYPES)):
+                    self.notice(candidate["target_kind"] + "s", candidate["target_uid"], "attachment_unreachable", url, f"{url}: status {response.status}")
+                    continue
+                data = await response.read()
+            except Exception as exc:
+                self.notice(candidate["target_kind"] + "s", candidate["target_uid"], "attachment_unreachable", url, f"{url}: {type(exc).__name__}: {exc}")
+                continue
+            if not data or len(data) > int(self.options["max_attachment_bytes"]) or total + len(data) > int(self.options["max_attachment_total"]):
+                self.notice(candidate["target_kind"] + "s", candidate["target_uid"], "attachment_too_large", url, f"{url}: {len(data)} bytes is over the limit")
+                continue
+            total += len(data)
+            filename = url.split("?", 1)[0].rstrip("/").rsplit("/", 1)[-1] or "file"
+            self.attachments.append(
+                {
+                    "origin_uid": uid_for(f"file:{url}"),
+                    "author_uid": candidate["author_uid"],
+                    "target_kind": candidate["target_kind"],
+                    "target_uid": candidate["target_uid"],
+                    "filename": filename[:200],
+                    "content_b64": base64.b64encode(data).decode(),
+                    "created_at": candidate["created_at"],
+                }
+            )
 
     def ordered_comments(self) -> list[dict]:
         ordered: list[dict] = []
@@ -518,6 +736,8 @@ class Collector:
             except CollectError as exc:
                 self.errors.append(f"{feed['id']}: {exc}")
                 reports[feed["id"]] = {"error": str(exc)}
+        if self.options.get("attachments"):
+            await self.download_files()
         failed = [name for name, report in reports.items() if "error" in report]
         return {
             "community": community,
@@ -527,6 +747,9 @@ class Collector:
                 "users": list(self.users.values()),
                 "posts": list(self.posts.values()),
                 "comments": self.ordered_comments(),
+                "votes": list(self.votes.values()),
+                "reactions": list(self.reactions.values()),
+                "attachments": self.attachments,
             },
             "state": self.state,
             "feeds": reports,

devplacepy/services/community/schemacheck.py

@@ -0,0 +1,254 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import re
+from urllib.parse import urlsplit
+
+from devplacepy.services.community.extract import parse_path, parse_selector
+from devplacepy.services.community.runtime import DEFAULT_LIMITS
+from devplacepy.services.community.spec import FEED_HTML, FEED_TYPES, LIMITS
+
+FEED_ID = re.compile(r"^[a-z0-9_-]{1,40}$")
+ORIGIN = re.compile(r"^[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?$")
+COMMENT_MODES = ("none", "inline", "url", "page")
+PAGINATION_STYLES = ("none", "page", "offset", "cursor", "next_link")
+COMMENT_PAGINATION_STYLES = ("none", "next_link", "page_param", "json_next")
+LOGIN_MODES = ("form", "page", "json")
+MAX_FEEDS = 8
+MAX_SPEC_CHARS = 500
+CEILINGS = {
+    "max_pages": 20,
+    "max_posts": 1000,
+    "max_comment_posts": 100,
+    "max_comment_pages": 10,
+    "comment_recheck": 50,
+    "retries": 5,
+    "timeout_seconds": 120,
+    "max_bytes": 20_000_000,
+    "max_remembered_links": 50_000,
+    "backoff_seconds": 30,
+}
+FLOORS = {"delay_seconds": 0.5}
+MAX_DELAY = 60.0
+REQUIRED_POST_FIELDS = ("id",)
+TEXT_POST_FIELDS = ("content", "title")
+
+
+class SchemaError(ValueError):
+    def __init__(self, errors: list[str]):
+        super().__init__("; ".join(errors[:3]))
+        self.errors = errors
+
+
+def check_spec(spec, html: bool, where: str, errors: list[str]) -> None:
+    if not isinstance(spec, str) or not spec.strip() or len(spec) > MAX_SPEC_CHARS:
+        errors.append(f"{where}: a non-empty path or selector of at most {MAX_SPEC_CHARS} characters is required")
+        return
+    if spec == "$now":
+        return
+    try:
+        if html:
+            selector, _, accessor = spec.partition("::")
+            if selector.strip():
+                parse_selector(selector)
+            if accessor and accessor != "text" and not re.fullmatch(r"attr\([\w:-]+\)", accessor):
+                errors.append(f"{where}: unsupported accessor {accessor!r}, use text or attr(name)")
+        else:
+            parse_path(spec)
+    except ValueError as exc:
+        errors.append(f"{where}: {exc}")
+
+
+def check_fields(fields, html: bool, where: str, errors: list[str]) -> None:
+    if not isinstance(fields, dict):
+        errors.append(f"{where}: must be an object of field names to lists of paths")
+        return
+    for name, specs in fields.items():
+        if isinstance(specs, str):
+            specs = [specs]
+        if not isinstance(specs, list) or not specs:
+            errors.append(f"{where}.{name}: must be a non-empty list of paths")
+            continue
+        for index, spec in enumerate(specs):
+            check_spec(spec, html, f"{where}.{name}[{index}]", errors)
+
+
+def check_author(author, html: bool, where: str, errors: list[str]) -> None:
+    if author in (None, {}):
+        return
+    if not isinstance(author, dict):
+        errors.append(f"{where}: must be an object")
+        return
+    check_fields(author.get("fields", {}), html, f"{where}.fields", errors)
+
+
+def check_comments(comments, feed_html: bool, where: str, errors: list[str]) -> None:
+    if comments in (None, {}):
+        return
+    if not isinstance(comments, dict) or comments.get("mode") not in COMMENT_MODES:
+        errors.append(f"{where}.mode: must be one of {', '.join(COMMENT_MODES)}")
+        return
+    mode = comments["mode"]
+    if mode == "none":
+        return
+    page = mode == "page"
+    check_fields(comments.get("fields", {}), page, f"{where}.fields", errors)
+    fields = comments.get("fields") or {}
+    for required in ("id", "content") if mode != "page" else ("content",):
+        if required not in fields:
+            errors.append(f"{where}.fields.{required}: is required for comments in {mode} mode")
+    check_author(comments.get("author"), page, f"{where}.author", errors)
+    if mode in ("inline", "url") and comments.get("list_path") is not None:
+        check_spec(comments["list_path"], False, f"{where}.list_path", errors)
+    if mode == "inline" and not comments.get("list_path"):
+        errors.append(f"{where}.list_path: is required for inline comments")
+    if mode == "url":
+        template = comments.get("url_template")
+        if not isinstance(template, str) or not ("{id}" in template or "{uid}" in template):
+            errors.append(f"{where}.url_template: must contain {{id}} or {{uid}}")
+    if page:
+        if not comments.get("item_selector"):
+            errors.append(f"{where}.item_selector: is required for page comments")
+        else:
+            check_spec(comments["item_selector"], True, f"{where}.item_selector", errors)
+    hint = comments.get("pagination")
+    if hint:
+        if not isinstance(hint, dict) or hint.get("style") not in COMMENT_PAGINATION_STYLES:
+            errors.append(f"{where}.pagination.style: must be one of {', '.join(COMMENT_PAGINATION_STYLES)}")
+        elif hint["style"] == "next_link":
+            check_spec(hint.get("next"), True, f"{where}.pagination.next", errors)
+        elif hint["style"] == "json_next":
+            check_spec(hint.get("path"), False, f"{where}.pagination.path", errors)
+
+
+def check_pagination(pagination, html: bool, where: str, errors: list[str]) -> None:
+    if pagination in (None, {}):
+        return
+    style = pagination.get("style") if isinstance(pagination, dict) else None
+    if style not in PAGINATION_STYLES:
+        errors.append(f"{where}.style: must be one of {', '.join(PAGINATION_STYLES)}")
+        return
+    if style in ("page", "offset", "cursor") and not pagination.get("param"):
+        errors.append(f"{where}.param: is required for {style} pagination")
+    if style == "offset" and not pagination.get("per_page"):
+        errors.append(f"{where}.per_page: is required for offset pagination")
+    if style == "next_link":
+        check_spec(pagination.get("next"), html, f"{where}.next", errors)
+    if style == "cursor":
+        check_spec(pagination.get("cursor_path"), False, f"{where}.cursor_path", errors)
+
+
+def check_feed(feed, index: int, seen: set[str], errors: list[str]) -> None:
+    where = f"feeds[{index}]"
+    if not isinstance(feed, dict):
+        errors.append(f"{where}: must be an object")
+        return
+    identifier = feed.get("id")
+    if not isinstance(identifier, str) or not FEED_ID.match(identifier):
+        errors.append(f"{where}.id: 1-40 characters of a-z, 0-9, _ and -")
+    elif identifier in seen:
+        errors.append(f"{where}.id: {identifier!r} is used twice")
+    else:
+        seen.add(identifier)
+    parts = urlsplit(str(feed.get("url") or ""))
+    if parts.scheme not in ("http", "https") or not parts.hostname:
+        errors.append(f"{where}.url: must be an http or https address")
+    kind = feed.get("type")
+    if kind not in FEED_TYPES:
+        errors.append(f"{where}.type: must be one of {', '.join(FEED_TYPES)}")
+        return
+    html = kind == FEED_HTML
+    post = feed.get("post")
+    if not isinstance(post, dict):
+        errors.append(f"{where}.post: is required")
+        return
+    check_fields(post.get("fields"), html, f"{where}.post.fields", errors)
+    fields = post.get("fields") or {}
+    for required in REQUIRED_POST_FIELDS:
+        if required not in fields:
+            errors.append(f"{where}.post.fields.{required}: is required")
+    if not any(name in fields for name in TEXT_POST_FIELDS):
+        errors.append(f"{where}.post.fields: needs content or title")
+    detail = post.get("detail")
+    if detail:
+        check_fields({"content": detail.get("content")}, True, f"{where}.post.detail", errors)
+    if html:
+        if not feed.get("item_selector"):
+            errors.append(f"{where}.item_selector: is required for html_list feeds")
+        else:
+            check_spec(feed["item_selector"], True, f"{where}.item_selector", errors)
+    elif kind in ("json_api", "json_ld") and feed.get("list_path") is not None:
+        check_spec(feed["list_path"], False, f"{where}.list_path", errors)
+    check_author(feed.get("author"), html, f"{where}.author", errors)
+    check_pagination(feed.get("pagination"), html, f"{where}.pagination", errors)
+    check_comments(feed.get("comments"), html, f"{where}.comments", errors)
+
+
+def check_limits(limits, errors: list[str]) -> None:
+    if limits in (None, {}):
+        return
+    if not isinstance(limits, dict):
+        errors.append("limits: must be an object")
+        return
+    for name, value in limits.items():
+        if name not in DEFAULT_LIMITS:
+            errors.append(f"limits.{name}: unknown limit")
+        elif isinstance(value, bool) or not isinstance(value, (int, float)) or value <= 0:
+            errors.append(f"limits.{name}: must be a positive number")
+        elif name in CEILINGS and value > CEILINGS[name]:
+            errors.append(f"limits.{name}: at most {CEILINGS[name]}")
+        elif name in FLOORS and value < FLOORS[name]:
+            errors.append(f"limits.{name}: at least {FLOORS[name]} seconds, requests must stay polite")
+        elif name == "delay_seconds" and value > MAX_DELAY:
+            errors.append(f"limits.delay_seconds: at most {MAX_DELAY} seconds")
+
+
+def check_login(login, errors: list[str]) -> None:
+    if login in (None, {}):
+        return
+    if not isinstance(login, dict) or login.get("mode") not in LOGIN_MODES:
+        errors.append(f"login.mode: must be one of {', '.join(LOGIN_MODES)}")
+        return
+    if login["mode"] == "json":
+        for required in ("url", "username_field", "password_field"):
+            if not login.get(required):
+                errors.append(f"login.{required}: is required for a JSON login")
+        if login.get("token_path"):
+            check_spec(login["token_path"], False, "login.token_path", errors)
+    elif not login.get("url"):
+        errors.append("login.url: is required")
+
+
+def validate_schema(schema) -> list[str]:
+    errors: list[str] = []
+    if not isinstance(schema, dict):
+        return ["the schema must be a JSON object"]
+    origin = schema.get("origin")
+    if not isinstance(origin, str) or not ORIGIN.match(origin):
+        errors.append("origin: a host name of a-z, 0-9, dots and hyphens")
+    if not isinstance(schema.get("community"), str) or not schema["community"].strip():
+        errors.append("community: a name is required")
+    feeds = schema.get("feeds")
+    if not isinstance(feeds, list) or not feeds:
+        errors.append("feeds: at least one feed is required")
+    elif len(feeds) > MAX_FEEDS:
+        errors.append(f"feeds: at most {MAX_FEEDS}")
+    else:
+        seen: set[str] = set()
+        for index, feed in enumerate(feeds):
+            check_feed(feed, index, seen, errors)
+    check_limits(schema.get("limits"), errors)
+    check_login(schema.get("login"), errors)
+    return errors
+
+
+def sample_limits(schema: dict) -> dict:
+    return {
+        **schema.get("limits", {}),
+        "max_pages": 1,
+        "max_posts": LIMITS.sample_posts,
+        "max_comment_posts": LIMITS.sample_comment_posts,
+        "max_comment_pages": 1,
+        "delay_seconds": max(float(schema.get("limits", {}).get("delay_seconds", LIMITS.delay_seconds)), LIMITS.delay_seconds),
+    }

devplacepy/services/community/service.py

@@ -73,6 +73,9 @@ class CommunityService(BaseService):
         dry = config[DRY_RUN_KEY]
         budget = config[MAX_PER_RUN_KEY]
         mode = "DRY RUN" if dry else "LIVE"
+        reaped = store.reap_interrupted()
+        if reaped:
+            self.log(f"marked {reaped} interrupted run(s)")
         for row in store.pending(budget):
             budget -= 1
             self.log(f"[{mode}] discovering pending community {row['query']!r}")

devplacepy/services/community/spec.py

@@ -8,7 +8,8 @@ FEED_RSS = "rss"
 FEED_ATOM = "atom"
 FEED_JSON = "json_api"
 FEED_HTML = "html_list"
-FEED_TYPES = (FEED_RSS, FEED_ATOM, FEED_JSON, FEED_HTML)
+FEED_JSONLD = "json_ld"
+FEED_TYPES = (FEED_RSS, FEED_ATOM, FEED_JSON, FEED_HTML, FEED_JSONLD)
 
 STATUS_PENDING = "pending"
 STATUS_LEARNING = "learning"
@@ -30,6 +31,7 @@ RUN_PARTIAL = "partial"
 RUN_FAILED = "failed"
 RUN_DRY = "dry_run"
 RUN_SKIPPED = "skipped"
+RUN_INTERRUPTED = "interrupted"
 
 SCHEMA_VERSION = 1
 RSEARCH_URL = "https://rsearch.app.molodetz.nl/search"
@@ -60,8 +62,12 @@ class CommunityLimits:
     max_communities_per_run: int = 3
     runs_kept: int = 30
     ledger_runs_kept: int = 10
+    checkpoint_seconds: int = 15
+    interrupted_grace_seconds: int = 60
+    search_min_chars: int = 3
     batch_records: int = 50
     min_field_coverage: float = 0.6
+    max_remembered_links: int = 20_000
 
     def runtime(self) -> dict:
         return {
@@ -75,6 +81,7 @@ class CommunityLimits:
             "backoff_seconds": self.backoff_seconds,
             "timeout_seconds": self.timeout_seconds,
             "max_bytes": self.max_bytes,
+            "max_remembered_links": self.max_remembered_links,
         }
 
 

devplacepy/services/community/store.py

@@ -9,6 +9,7 @@ from devplacepy.database import db, get_table
 from devplacepy.database.soft_delete import soft_delete
 from devplacepy.services.community.spec import (
     LIMITS,
+    RUN_INTERRUPTED,
     STATUS_FAILING,
     STATUS_LEARNING,
     STATUS_PENDING,
@@ -17,7 +18,7 @@ from devplacepy.services.community.spec import (
 from devplacepy.services.sync.selection import jittered
 from devplacepy.utils.text import generate_uid
 
-JSON_COLUMNS = ("schema", "watermark", "totals", "sync_options")
+JSON_COLUMNS = ("schema", "schema_previous", "watermark", "totals", "sync_options")
 RUN_JSON_COLUMNS = ("collected", "pushed", "errors", "tally")
 RUN_SUMMARY_COLUMNS = "uid, community_uid, started_at, finished_at, status, dry_run, trigger, pushed, tally, verdict, headline, items_stored, items_dropped"
 ITEMS_TABLE = "community_run_items"
@@ -30,6 +31,7 @@ EDITABLE = (
     "user_count",
     "status",
     "schema",
+    "schema_previous",
     "watermark",
     "enabled",
     "interval_seconds",
@@ -46,6 +48,7 @@ EDITABLE = (
     "login_username",
     "login_email",
     "login_password",
+    "login_totp_secret",
     "sync_options",
     "claimed_at",
 )
@@ -105,6 +108,7 @@ class CommunityStore:
             "username": row.get("login_username") or "",
             "email": row.get("login_email") or "",
             "password": row.get("login_password") or "",
+            "totp": row.get("login_totp_secret") or "",
         }
 
     def due(self, limit: int) -> list[dict]:
@@ -167,6 +171,23 @@ class CommunityStore:
         )
         return uid
 
+    def update_run(self, uid: str, **fields) -> None:
+        get_table("community_runs").update({"uid": uid, **encode(fields, RUN_JSON_COLUMNS)}, ["uid"])
+
+    def reap_interrupted(self) -> int:
+        if "community_runs" not in db.tables:
+            return 0
+        horizon = iso(now() - timedelta(seconds=LIMITS.run_timeout_seconds + LIMITS.interrupted_grace_seconds))
+        stale = [row["uid"] for row in get_table("community_runs").find(status="running") if row["started_at"] < horizon]
+        for uid in stale:
+            self.finish_run(
+                uid,
+                RUN_INTERRUPTED,
+                verdict="bad",
+                headline="Interrupted: the process ended before this run finished. What was synced up to then is listed.",
+            )
+        return len(stale)
+
     def finish_run(self, uid: str, status: str, **fields) -> None:
         get_table("community_runs").update(
             {"uid": uid, "status": status, "finished_at": iso(now()), **encode(fields, RUN_JSON_COLUMNS)}, ["uid"]

devplacepy/services/devii/actions/catalog/community.py

@@ -10,6 +10,7 @@ ACCOUNT = (
     body("login_username", "Username of an account on that community, for a login-protected community."),
     body("login_email", "Email of that account, when the community logs in by email."),
     body("login_password", "Password of that account. Stored for the scheduled collectors, never shown again."),
+    body("login_totp_secret", "Authenticator (TOTP) secret of that account when it uses two-factor codes. Stored for the collectors, never shown again."),
 )
 
 COMMUNITY_ACTIONS: tuple[Action, ...] = (
@@ -61,6 +62,34 @@ COMMUNITY_ACTIONS: tuple[Action, ...] = (
         ),
         requires_admin=True,
     ),
+    Action(
+        name="community_schema_get",
+        method="GET",
+        path="/admin/communities/{slug}/schema",
+        summary="Show the full learned schema of a community as JSON (admin only)",
+        description="The schema lists the feeds and the paths or selectors that hold each field. Read it before proposing a change with community_schema_set.",
+        params=(SLUG,),
+        requires_admin=True,
+        read_only=True,
+    ),
+    Action(
+        name="community_schema_set",
+        method="POST",
+        path="/admin/communities/{slug}/schema",
+        summary="Replace the learned schema of a community after it passed a live check (admin only)",
+        description=(
+            "The structure is validated, then a small live sample must collect at least one post; otherwise nothing is "
+            "stored and the reasons are returned. The old schema is kept for a one-click restore. Show the administrator "
+            "the exact change, get explicit confirmation, then call again with confirm=true."
+        ),
+        params=(
+            SLUG,
+            body("schema", "The complete schema as one JSON object string.", required=True),
+            confirm(),
+        ),
+        requires_admin=True,
+        ajax=True,
+    ),
     Action(
         name="community_add",
         method="POST",

devplacepy/services/devii/actions/catalog/game.py

@@ -75,6 +75,23 @@ GAME_ACTIONS: tuple[Action, ...] = (
             Param(name="slot", location="body", description="Plot slot index.", required=True, type="integer"),
         ),
     ),
+    Action(
+        name="game_plot_stop",
+        method="POST",
+        path="/game/plot/stop",
+        summary="Pause or resume Continuous Delivery auto-replant on one Code Farm plot",
+        description=(
+            "Needs the Continuous Delivery Mastery upgrade. stop=true makes the next harvest of "
+            "that plot leave it empty so a different crop can be planted; the pause clears "
+            "itself once the plot is planted again. stop=false cancels a pending pause."
+        ),
+        handler="http",
+        requires_auth=True,
+        params=(
+            Param(name="slot", location="body", description="Plot slot index.", required=True, type="integer"),
+            body("stop", "true to skip the replant after the next harvest, false to cancel.", required=True, type="boolean"),
+        ),
+    ),
     Action(
         name="game_buy_plot",
         method="POST",

devplacepy/services/devii/actions/catalog/nodes.py

@@ -130,6 +130,24 @@ NODE_ACTIONS: tuple[Action, ...] = (
         requires_admin=True,
         params=(ADMIN_DEVICE,),
     ),
+    Action(
+        name="node_admin_class_override",
+        method="POST",
+        path="/admin/nodes/devices/{device_uid}/class",
+        summary="Allow or stop allowing one donor device regardless of its address class",
+        description=(
+            "Donors on datacenter or unknown addresses are not scheduled unless an administrator overrides "
+            "the device. value=allow lets this device serve jobs; value=clear removes the override. Show the "
+            "administrator the device, its owner and its address class, get explicit confirmation, then call "
+            "again with confirm=true."
+        ),
+        requires_admin=True,
+        params=(
+            ADMIN_DEVICE,
+            body("value", "allow to override the address class rule, clear to remove the override.", required=True),
+            confirm(),
+        ),
+    ),
     Action(
         name="node_admin_killswitch",
         method="POST",

devplacepy/services/devii/actions/catalog/sync.py

@@ -3,7 +3,7 @@
 from __future__ import annotations
 
 from ..spec import Action
-from ._shared import body
+from ._shared import body, path, query
 
 ENVELOPE = (
     body("origin", "Stable id of the sending community, for example other.example.org.", required=True),
@@ -41,4 +41,85 @@ SYNC_ACTIONS: tuple[Action, ...] = (
         "Upsert comments copied from another community",
         "Creates or updates comments; author_uid, post_uid and any parent_uid are origin ids that must already be synced.",
     ),
+    sync_action(
+        "projects",
+        "Upsert public projects copied from another community",
+        "Creates or updates projects as public and writable; author_uid is the author's origin id. Posts reference a project through project_uid.",
+    ),
+    sync_action(
+        "polls",
+        "Upsert polls copied from another community",
+        "One poll per post with 2 to 6 options, each option carrying its own origin_uid and label.",
+    ),
+    sync_action(
+        "votes",
+        "Upsert votes copied from another community",
+        "A vote needs a synced voter and a synced target (target_kind post, comment or project, value 1 or -1). The target's stars are derived from synced votes only.",
+    ),
+    sync_action(
+        "reactions",
+        "Upsert emoji reactions copied from another community",
+        "A reaction needs a synced author and a synced target; emoji must be a single emoji.",
+    ),
+    sync_action(
+        "follows",
+        "Upsert follow relations copied from another community",
+        "follower_uid and following_uid are origin ids of synced users. A user cannot follow themselves.",
+    ),
+    sync_action(
+        "poll_votes",
+        "Upsert poll ballots copied from another community",
+        "One ballot per voter and poll; option_uid is the option's origin id within poll_uid.",
+    ),
+    sync_action(
+        "attachments",
+        "Upsert attachments copied from another community",
+        "The sender downloads the file and sends it as content_b64 (at most 5 MB). Attachments cannot change; only a tombstone or a restore is accepted for a known origin_uid.",
+    ),
+    Action(
+        name="sync_export",
+        method="GET",
+        path="/sync/export/{kind}",
+        summary="Export this community's own users, projects, posts, comments, polls, follows or attachments in sync record form",
+        description=(
+            "Administrator pull feed for another community. Pages of at most 50 records, keyset paginated: pass the "
+            "returned next_cursor back until has_more is false. Rows that were themselves synced in are never exported. "
+            "Votes, reactions and ballots are never exported because who voted is not public."
+        ),
+        params=(
+            path("kind", "One of users, projects, posts, comments, polls, follows, attachments."),
+            query("since", "Only records changed after this ISO 8601 instant."),
+            query("cursor", "The next_cursor of the previous page."),
+            query("limit", "Records per page, 1 to 50."),
+        ),
+        requires_admin=True,
+        read_only=True,
+    ),
+    Action(
+        name="sync_deletions",
+        method="GET",
+        path="/sync/deletions",
+        summary="List synced records that were deleted or purged on this side and not yet acknowledged by the origin",
+        description="Lets a sending community learn which of its records are gone here. Page with next_cursor, then acknowledge with sync_deletions_ack.",
+        params=(
+            query("origin", "The sending community's origin id.", required=True),
+            query("cursor", "The next_cursor of the previous page."),
+            query("limit", "Entries per page, 1 to 50."),
+        ),
+        requires_admin=True,
+        read_only=True,
+    ),
+    Action(
+        name="sync_deletions_ack",
+        method="POST",
+        path="/sync/deletions/ack",
+        summary="Acknowledge deletions so the origin is no longer told about them",
+        description="After acknowledging, a re-sent record answers unchanged with reason acknowledged_deleted instead of a rejection.",
+        ajax=True,
+        params=(
+            body("origin", "The sending community's origin id.", required=True),
+            body("items", "JSON array string of at most 50 objects like {\"kind\": \"posts\", \"origin_uid\": \"p-1\"}.", required=True),
+        ),
+        requires_admin=True,
+    ),
 )

devplacepy/services/devii/actions/dispatcher.py

@@ -34,8 +34,10 @@ MUTATING_METHODS = ("POST", "DELETE", "PUT", "PATCH")
 
 CONFIRM_REQUIRED = {
     "community_delete",
+    "community_schema_set",
     "node_device_revoke",
     "node_admin_suspend",
+    "node_admin_class_override",
     "node_admin_killswitch",
     "join_battle",
     "fight_battle",
@@ -251,6 +253,12 @@ def confirmation_error(name: str, arguments: dict[str, Any]) -> ToolInputError |
             f"Deleting community '{slug}' stops its synchronisation and moves it to the trash; content already "
             "synchronised stays. Show the user the community, get explicit confirmation, then call again with confirm=true."
         )
+    if name == "community_schema_set":
+        slug = str(arguments.get("slug", "")).strip() or "(unspecified)"
+        return ToolInputError(
+            f"Replacing the learned schema of community '{slug}' changes what every future run collects. "
+            "Show the administrator the exact change, get explicit confirmation, then call again with confirm=true."
+        )
     if name == "node_device_revoke":
         uid = str(arguments.get("device_uid", "")).strip() or "(unspecified)"
         return ToolInputError(
@@ -264,6 +272,13 @@ def confirmation_error(name: str, arguments: dict[str, Any]) -> ToolInputError |
             "the whole period. Show the administrator the exact device and owner, get explicit confirmation, "
             "then call again with confirm=true."
         )
+    if name == "node_admin_class_override":
+        uid = str(arguments.get("device_uid", "")).strip() or "(unspecified)"
+        return ToolInputError(
+            f"Overriding the address class rule for donor device '{uid}' lets it serve jobs although its "
+            "address is not a residential or mobile one. Show the administrator the device, its owner and its "
+            "address class, get explicit confirmation, then call again with confirm=true."
+        )
     if name == "node_admin_killswitch":
         return ToolInputError(
             "The community donation switch is global: turning it off disconnects every donor node within "

devplacepy/services/game/CLAUDE.md

@@ -4,7 +4,7 @@ This file documents the Code Farm idle game. Claude Code auto-loads it when a fi
 
 ## Overview
 
-`game/` package (routers, mounted at `/game`) is the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard?board=`, `POST /game/{plant,harvest,buy-plot,upgrade,fertilize,daily,grant,perk,prestige,legacy,mastery,quests/claim}`, `POST /game/{defense/upgrade,defense/downgrade,infrastructure/buy,cosmetics/buy,cosmetics/equip}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}`. Customer-facing documentation is the `docs_api.py` **Code Farm** group (`docs_api/groups/game.py`, enum `options` on every key param) and the prose guide `templates/docs/code-farm.html` (`/docs/code-farm.html` - the complete rules, every exact formula and catalog table, the full farm/plot field reference, the JSON error shape, and a stdlib automation client); keep BOTH in lockstep with `economy.py` whenever a constant, formula, catalog entry, field, or endpoint changes.
+`game/` package (routers, mounted at `/game`) is the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard?board=`, `POST /game/{plant,harvest,plot/stop,buy-plot,upgrade,fertilize,daily,grant,perk,prestige,legacy,mastery,quests/claim}`, `POST /game/{defense/upgrade,defense/downgrade,infrastructure/buy,cosmetics/buy,cosmetics/equip}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}`. Customer-facing documentation is the `docs_api.py` **Code Farm** group (`docs_api/groups/game.py`, enum `options` on every key param) and the prose guide `templates/docs/code-farm.html` (`/docs/code-farm.html` - the complete rules, every exact formula and catalog table, the full farm/plot field reference, the JSON error shape, and a stdlib automation client); keep BOTH in lockstep with `economy.py` whenever a constant, formula, catalog entry, field, or endpoint changes.
 
 A cooperative-and-competitive idle game (Farmville-style) mounted at `/game`, member-only to play, public to view another farm. Cooperative loop = watering a neighbour's growing build; competitive loop = stealing a neighbour's ready build.
 
@@ -100,6 +100,15 @@ Orthogonal to prestige, modeled directly on the existing Legacy shape (leveled,
 
 Three new high-tier crops (`distsys`/`mlpipe`/`secfort` in `economy.CROPS`) extend the `Crop` dataclass with three new **trailing, defaulted** fields (`min_mastery: int = 0`, `steal_immune: bool = False`, `era_key: str | None = None`) - safe because every existing `Crop(...)` construction is positional over the original 8 non-defaulted fields. They set `min_level=MAX_LEVEL` (so, like every other crop, they re-lock after each refactor until the player re-levels to 20 - consistent with how Kernel etc. already behave post-prestige) and `min_mastery=1` (gated on `mastery_points_earned_total`, checked both in `store/actions.py::plant()` - real enforcement - and in `economy.crop_payload`'s `locked` flag - display). `secfort` (Security Fortress) is `steal_immune=True`: `serialize_plot` forces `can_steal=False`/`steal_reason="immune"` and `store/actions.py::steal()` raises immediately, before any grace/cooldown check, when the target crop is immune.
 
+### Per-plot auto-replant pause (`game_plots.stop_after_harvest`)
+
+Continuous Delivery used to be a one-way lock: the harvested crop was replanted at once, `plant()` refuses an occupied plot and nothing uproots, so a crop could only change at a Refactor. `POST /game/plot/stop` (`store.set_plot_stop`, `GamePlotStopForm{slot, stop}`, Devii `game_plot_stop`, audit `game.plot.stop`) sets a **one-shot** flag on one plot. Invariants:
+
+- **The flag is read AFTER `clear_plot` wins.** `_try_autoreplant` re-reads the plot, so a toggle either landed before the clear (replant skipped) or finds an empty plot (`set_plot_stop`'s `WHERE crop_key != ''` fails, loud error). Never read the flag from the pre-clear plot dict and never reset it inside `clear_plot`.
+- **Every writer that puts a crop on a plot resets it:** the `plant()` and `_try_autoreplant` claim SETs write `stop_after_harvest = 0`, and `prestige()`'s plot reset writes it too. A stale `1` on an empty plot is harmless and is never serialized (`serialize_plot` gates `stop_after_harvest` on owner + Continuous Delivery + a crop present).
+- Column is `COALESCE`-read (`_lvl(plot, ...)`); existing rows are NULL.
+- `_crop_lock(farm, crop)` is the single lock predicate used by `plant()` and `_try_autoreplant`, so autoreplant no longer replants a crop that is locked (an Era crop after its Era ends). Saturation-based skipping was deliberately NOT built: at prestige 50+ (the only owners of Continuous Delivery) even a 0.4 market factor pays 11x or more the planting cost for every crop, so skipping only forfeits coins, XP and quest progress.
+
 ### 4. Secondary leaderboards (`economy.py` scoring + `store/farm.py`)
 
 `GET /game/leaderboard` gained an optional `board` query param (default `"score"`, so every existing caller/Devii/docs entry that omits it sees byte-identical output to before this shipped). `store/farm.py::leaderboard_for(board, limit)` dispatches through `LEADERBOARD_BOARDS` (`score` = the original `farm_score` board, `prestige`, `harvests` = this week's `harvests_week` counter, `raids` = average coins per **successful** raid with a minimum of `MIN_RAIDS_FOR_EFFICIENCY_BOARD` (3) qualifying raids - deliberately not "attempts," since failed steals are never persisted today and adding that write would be a spam vector for no real value, `time_to_kernel` = seconds between a farm's last `prestiged_at` and its next Kernel harvest, `fair_play` = `economy.fair_play_score(harvests_week, coins)` which rewards recent activity and penalizes hoarding) plus `era` (dispatches to `store/era.py::leaderboard_era`, always empty when no Era is running). `GameFarm.js._loadLeaderboard` renders a board-appropriate value per row instead of always showing the raw composite score - `raid_avg` (formatted `"Nc/raid"`) for the `raids` board and a formatted duration for `time_to_kernel`, falling back to `score` for every other board. New `game_farms` columns backing these: `harvests_week`/`harvests_week_start` (**incremented inside `credit_farm`'s atomic UPDATE** whenever `harvests > 0` - a CASE that rolls the counter over to the new ISO week when `harvests_week_start` differs, else adds; also reset lazily in `serialize_farm` via `_reset_harvests_week` for idle weeks, same lazy-no-tick idiom as everything else. The increment is load-bearing: it feeds grant eligibility, the `harvests` board, and `fair_play` - it was originally missing entirely, leaving the community grant permanently unclaimable and the weekly board all zeros, a liveness bug invisible to pure-safety fuzzing and caught only by the layer-3 grant race setup), `prestiged_at` (stamped every `prestige()`), `last_kernel_harvest_prestige`/`time_to_kernel_seconds` (updated in `harvest`/`_auto_harvest` the first time a Kernel is harvested since the last refactor).

devplacepy/services/game/store/__init__.py

@@ -18,6 +18,7 @@ from .actions import (
     harvest,
     plant,
     prestige,
+    set_plot_stop,
     steal,
     upgrade_legacy,
     upgrade_perk,

devplacepy/services/game/store/actions.py

@@ -38,16 +38,20 @@ from .quests import advance_quests, ensure_quests
 from .serialize import _daily_available, _plot_state, _watered_by
 
 
+def _crop_lock(farm: dict, crop) -> tuple[str, str] | None:
+    progress = economy.level_progress(int(farm.get("xp", 0)))
+    era_locked = bool(crop.era_key and crop.era_key != active_era_name())
+    return economy.crop_lock_reason(
+        crop, progress["level"], _lvl(farm, "mastery_points_earned_total"), era_locked
+    )
+
+
 def plant(user: dict, slot: int, crop_key: str) -> dict:
     farm = ensure_farm(user["uid"])
     crop = economy.crop_for(crop_key)
     if not crop:
         raise GameError("Unknown crop type.")
-    progress = economy.level_progress(int(farm.get("xp", 0)))
-    era_locked = bool(crop.era_key and crop.era_key != active_era_name())
-    lock = economy.crop_lock_reason(
-        crop, progress["level"], _lvl(farm, "mastery_points_earned_total"), era_locked
-    )
+    lock = _crop_lock(farm, crop)
     if lock:
         raise GameError(f"{crop.name} {lock[1]}.")
     plot = _plot_at(farm["uid"], slot)
@@ -77,7 +81,7 @@ def plant(user: dict, slot: int, crop_key: str) -> dict:
     claimed = conditional_update_row(
         "game_plots",
         plot["uid"],
-        "crop_key = :crop, planted_at = :planted, ready_at = :ready, watered_by = '[]', raided_fraction = 0",
+        "crop_key = :crop, planted_at = :planted, ready_at = :ready, watered_by = '[]', raided_fraction = 0, stop_after_harvest = 0",
         "crop_key = '' OR crop_key IS NULL",
         {"crop": crop.key, "planted": _iso(now), "ready": _iso(ready_at)},
     )
@@ -168,6 +172,8 @@ def _try_autoreplant(farm: dict, slot: int, crop, now) -> None:
     plot = _plot_at(farm["uid"], slot)
     if not plot or plot.get("crop_key"):
         return
+    if _lvl(plot, "stop_after_harvest") or _crop_lock(farm, crop):
+        return
     cost = economy.effective_plant_cost(crop, _lvl(farm, "perk_discount"))
     if int(farm.get("coins", 0)) < cost:
         return
@@ -187,7 +193,7 @@ def _try_autoreplant(farm: dict, slot: int, crop, now) -> None:
     claimed = conditional_update_row(
         "game_plots",
         plot["uid"],
-        "crop_key = :crop, planted_at = :planted, ready_at = :ready, watered_by = '[]', raided_fraction = 0",
+        "crop_key = :crop, planted_at = :planted, ready_at = :ready, watered_by = '[]', raided_fraction = 0, stop_after_harvest = 0",
         "crop_key = '' OR crop_key IS NULL",
         {"crop": crop.key, "planted": _iso(now), "ready": _iso(ready_at)},
     )
@@ -195,6 +201,25 @@ def _try_autoreplant(farm: dict, slot: int, crop, now) -> None:
         refund_farm(farm["uid"], cost)
 
 
+def set_plot_stop(user: dict, slot: int, stop: bool) -> dict:
+    farm = ensure_farm(user["uid"])
+    if not _lvl(farm, MASTERY_COLUMN["autoreplant"]):
+        raise GameError("Continuous Delivery is not unlocked.")
+    plot = _plot_at(farm["uid"], slot)
+    if not plot or not plot.get("crop_key"):
+        raise GameError("That plot is empty.")
+    updated = conditional_update_row(
+        "game_plots",
+        plot["uid"],
+        "stop_after_harvest = :stop",
+        "crop_key != '' AND crop_key IS NOT NULL",
+        {"stop": 1 if stop else 0},
+    )
+    if updated == 0:
+        raise GameError("That plot is empty.")
+    return {"slot": slot, "stop": bool(stop), "crop": plot.get("crop_key", "")}
+
+
 def water(visitor: dict, owner: dict, slot: int) -> dict:
     if visitor["uid"] == owner["uid"]:
         raise GameError("You cannot water your own build.")
@@ -615,6 +640,7 @@ def prestige(user: dict) -> dict:
                     "planted_at": "",
                     "ready_at": "",
                     "watered_by": "[]",
+                    "stop_after_harvest": 0,
                     "updated_at": now,
                 },
                 ["uid"],

devplacepy/services/game/store/serialize.py

@@ -65,6 +65,7 @@ def serialize_plot(
     canary: bool = False,
     underdog: bool = False,
     contract_boost: bool = False,
+    autoreplant: bool = False,
 ) -> dict:
     state = _plot_state(plot, now)
     crop = economy.crop_for(plot.get("crop_key", ""))
@@ -133,6 +134,7 @@ def serialize_plot(
         if can_steal
         else 0
     )
+    can_stop_replant = is_owner and autoreplant and bool(crop)
     fertilize_cost = 0
     if state == "growing" and is_owner and crop:
         full_grow = economy.grow_seconds_for(
@@ -172,6 +174,8 @@ def serialize_plot(
         "fertilize_cost": fertilize_cost,
         "raided_fraction": round(raided_fraction, 4),
         "raided_pct": round(raided_fraction * 100),
+        "can_stop_replant": can_stop_replant,
+        "stop_after_harvest": can_stop_replant and bool(plot.get("stop_after_harvest")),
     }
 
 
@@ -238,6 +242,7 @@ def serialize_farm(
             canary=owns_infrastructure(farm, "canary"),
             underdog=underdog_active,
             contract_boost=contract_active,
+            autoreplant=bool(_lvl(farm, "mastery_autoreplant")),
         )
         for plot in plots
     ]

devplacepy/services/jobs/deepsearch/crawl.py

@@ -18,6 +18,8 @@ from devplacepy import stealth
 from devplacepy.config import INTERNAL_BASE_URL
 from devplacepy.net_guard import BlockedAddressError, guard_public_url, guarded_async_client
 
+from devplacepy.services.nodes.policy import DestinationPolicy, PolicyError
+
 from .extract import extract_html, relevant_links
 from .pdf import MAX_PDF_BYTES, extract_pdf_text, is_pdf
 
@@ -319,11 +321,62 @@ async def fetch_page(url: str, depth: int, browser=None) -> CrawledPage | None:
     )
 
 
-async def _resolve_candidate(candidate: dict, depth: int, browser=None) -> CrawledPage | None:
+NODE_FETCH_TIMEOUT_SECONDS = 40.0
+NODE_FETCH_WAIT_SECONDS = 25
+
+
+def node_servable(url: str) -> bool:
+    try:
+        DestinationPolicy().check(url)
+    except PolicyError:
+        return False
+    return True
+
+
+async def node_page(url: str, depth: int, api_key: str) -> CrawledPage | None:
+    if not api_key or not node_servable(url):
+        return None
+    try:
+        async with stealth.stealth_async_client(
+            base_url=INTERNAL_BASE_URL,
+            headers={"Authorization": f"Bearer {api_key}", "Accept": "application/json"},
+            timeout=httpx.Timeout(NODE_FETCH_TIMEOUT_SECONDS, connect=10.0),
+        ) as client:
+            response = await client.post(
+                "/nodes/internal/fetch", data={"url": url, "timeout": NODE_FETCH_WAIT_SECONDS}
+            )
+        if response.status_code >= 400:
+            return None
+        data = response.json()
+    except (httpx.HTTPError, ValueError) as exc:
+        logger.info("deepsearch node fetch failed for %s: %s", url, exc)
+        return None
+    if int(data.get("status") or 0) >= 400:
+        return None
+    extracted = extract_html(data.get("text") or "", base_url=url)
+    if len(extracted.text) < MIN_PAGE_CHARS:
+        return None
+    return CrawledPage(
+        url=url,
+        title=extracted.title or url,
+        text=extracted.text,
+        source=f"node-{data.get('via') or 'fetch'}",
+        status=int(data.get("status") or 200),
+        depth=depth,
+        links=extracted.links,
+    )
+
+
+async def _resolve_candidate(
+    candidate: dict, depth: int, browser=None, api_key: str = ""
+) -> CrawledPage | None:
     url = candidate["url"]
     snippet_page = _snippet_page(candidate, depth)
     if _is_hostile(url):
-        return snippet_page
+        page = await node_page(url, depth, api_key)
+        if page and snippet_page:
+            return page if len(page.text) >= len(snippet_page.text) else snippet_page
+        return page or snippet_page
     page = await fetch_page(url, depth, browser)
     if page and snippet_page:
         return page if len(page.text) >= len(snippet_page.text) else snippet_page
@@ -339,6 +392,7 @@ async def crawl(
     query: str = "",
     depth: int = 1,
     seen_hashes: set[str] | None = None,
+    api_key: str = "",
 ) -> CrawlOutcome:
     from playwright.async_api import async_playwright
 
@@ -387,7 +441,7 @@ async def crawl(
                         emit({"type": "page_cached", "url": candidate["url"], "reason": "seen in a prior run"})
                 fetch_start = time.perf_counter()
                 results = await asyncio.gather(
-                    *(_resolve_candidate(candidate, level, browser) for candidate in batch),
+                    *(_resolve_candidate(candidate, level, browser, api_key) for candidate in batch),
                     return_exceptions=True,
                 )
                 elapsed_ms = int((time.perf_counter() - fetch_start) * 1000)

devplacepy/services/jobs/deepsearch/worker.py

@@ -189,6 +189,7 @@ async def _run(payload: dict, output_dir: Path) -> dict:
         should_stop,
         query=query,
         depth=depth,
+        api_key=api_key,
     )
 
     round_no = 0
@@ -227,6 +228,7 @@ async def _run(payload: dict, output_dir: Path) -> dict:
             query=query,
             depth=REFINE_CRAWL_DEPTH,
             seen_hashes=outcome.seen_hashes,
+            api_key=api_key,
         )
         if not extra.pages:
             break

devplacepy/services/nodes/CLAUDE.md

@@ -121,20 +121,31 @@ For a `tunnel` or `any` job the scheduler prefers a node whose `hello.modes` inc
 
 `SocialSearch.search` fans one query out to adapters in parallel (`asyncio.gather`, 20 s each) and merges hits round-robin. Direct adapters (`adapters/direct.py`: Hacker News, Stack Overflow, GitHub issues, dev.to by tag, Mastodon by hashtag) need no donors. `adapters/reddit.py` is node-backed and only used when `nodes_social_via_nodes` is `1`; otherwise it is reported as `nodes_not_used`. Every adapter parses a response structurally and raises `AdapterError` on a forged or wrong shape; platforms that fail are listed under `unavailable` with a code and never filled with invented results. **Measured on this machine (2026-10-15, datacenter address):** Hacker News Algolia, Stack Exchange, GitHub search, dev.to tag API and Mastodon hashtag timelines answer; Bluesky `searchPosts` returns 403 without auth, Mastodon `/api/v2/search` returns nothing without login, Lobsters search sits behind a bot wall, dev.to has no usable text search, and Reddit returns 403 to this address. Those are why Bluesky and Lobsters are not adapters. Surfaces: `GET /nodes/social/search` (daily per-member quota `nodes_daily_social_user`, default 300), Devii `social_search`, the DeepSearch `deepsearch_social` option (off by default, signed-in owners only, goes through the HTTP route with the owner's key).
 
+## Server mechanisms added after the first build
+
+- **Address class (D6 enforced).** `geo.py` keeps DB-IP lite ASN and country data in a SQLite range file under `config.NODE_GEO_DIR` (`devplace nodes geo-sync [--dry-run] [--month]`, `geo-status`; CC BY 4.0, attribution on `/docs/privacy.html`). Measured: build about 10 s, 477362 ASN and 710834 country rows, 48 MB, lookup 0.17 ms. `addressclass.classify` maps an ASN to `residential`, `mobile`, `datacenter`, `business` or `unknown`. The scheduler only assigns jobs to `ELIGIBLE_CLASSES` (residential, mobile) unless a device carries `class_override == "allow"` (admin, `POST /admin/nodes/devices/{uid}/class`, audit `node.device.override`, Devii `node_admin_class_override` confirm gated) or `nodes_allow_unknown_class=1` admits unknown. Refusals are counted in `NodeScheduler.class_refusals`. A missing geo file is `counters.geo_unavailable` plus one warning, never a silent pass.
+- **Durable state.** `last_seen_at` writes are batched (`touch_devices`, `last_seen_write_seconds`). Counters flush as deltas into `node_counters` every `counter_flush_seconds` (lifetime totals survive restart). Circuit breaker state is saved to `node_circuits` (`snapshot`/`restore`, capped at `circuit_snapshot_max`). The fabric cache is `SharedResultCache`: an in-process L1 over the `node_cache` table, so a result fetched by one worker serves the others.
+- **Fast path.** `wake.py` `WakeHub` wakes the dispatch loop and a waiting caller in the same process the moment a job is queued or done; the DB poll remains the cross-process fallback. Measured end to end (one donor, 300 jobs): polling p50 209 ms, p95 214 ms; fast path p50 25 ms, p95 about 30 ms, max about 38 ms. A single 6.2 s cold-start outlier appeared once and did not reproduce.
+- **Sampled redundancy.** `redundancy_percent` (5, ceiling `redundancy_max_percent` 20) of fetch-mode jobs are shadowed to a second donor (`shadow_of`, `exclude_device`); `shape.py` compares structure, and a mismatch raises the flag `redundancy_mismatch` for a human. Counters: `redundancy_checked`, `redundancy_matched`, `redundancy_mismatches`, `redundancy_inconclusive`.
+- **Outcome counters per address class.** `outcome:<adapter>:<class>:<ok|blocked|failed>` in `node_counters`, shown on `/admin/nodes`. This is the instrument for the residential success rate; it reads `n = 0` until real donors exist.
+- **DeepSearch** routes hostile URLs through `POST /nodes/internal/fetch` with the owner's api key (`crawl.node_servable`, `node_page`) when `nodes_social_via_nodes` is on.
+- **Battery.** `on_battery` is a donor limit code (pause 300 s, retryable). `welcome.device_label` and `kick.until` are optional protocol fields; a suspended ticket 403 carries `until`.
+
+**Measured capacity (one worker, scratch server, 2000 simulated donors on this machine):** 2000 sockets connected in 8 s; RSS 156 MB at 250, 190 MB at 500, 241 MB at 1000, 341 MB at 2000; idle dispatch tick p50 67 ms (max 160 ms); event loop lag p99 78 ms idle; 14.2 jobs/s completed at 15 offered, 0 client failures. A soak at 2000 sockets, 10 jobs/s and 3 percent a minute churn ran 4844 s (about 81 minutes, not 24 hours): RSS 346 to 361 MB, 42909 of 42913 jobs done, 1 failed, 0 integrity failures, 2026 of 2026 redundancy checks matched, loop lag p99 73 to 106 ms. Races, with separate OS processes: 16 workers x 20 counter flushes summed exactly (1600 of 1600); 12 racing cache puts left 1 row and all 12 readers hit.
+
 ## Clients
 
-`nodes/client/` is the Go donor client (standard library only, `build.sh` cross-compiles five targets and writes `manifest.json`). `nodes/extension/` is the MV3 extension (Chrome and Firefox, fetch mode only, 49 `node --test` tests, one of which reads `spec.py` and fails on drift). `make node-client` builds both into `data/node_clients/`, which `GET /nodes/client-version` and `GET /nodes/download/{artifact}` serve (names must match a strict pattern, symlinks and the manifest are never served, the SHA-256 is computed from the file). **Verified end to end on 2026-10-15** against a scratch server on this machine: the Go client logged in, connected, answered a `tunnel` job for `reddit.com` (`via: tunnel`, status 403 because the donor was this same datacenter address) and stayed connected across repeated jobs. Not verified: a residential donor, the extension in a real browser, Windows and macOS builds running.
+`nodes/client/` is the Go donor client (standard library only, `build.sh` cross-compiles five targets and writes `manifest.json`). It reads battery state on Linux, macOS and Windows (`RefuseBattery`, `PauseOnBattery`) and holds a timed suspension until `until` (clamped 60 s to 24 h). `nodes/extension/` is the MV3 extension (Chrome and Firefox, fetch mode only; icons 16/32/48/128; a suspended state with `suspendedUntil`; active hours in the server timezone; the manifest sets `content_security_policy.extension_pages` explicitly because Firefox's default MV3 policy adds `upgrade-insecure-requests`, which rewrote `ws://127.0.0.1` to `wss://` and broke loopback connections). `make node-client` builds both into `data/node_clients/`, served by `GET /nodes/client-version` and `GET /nodes/download/{artifact}` (strict names, no symlinks, SHA-256 from the file). **Verified in real browsers:** Chromium via `tests/e2e/nodes/extension.py` (disclosure, connect, online, a fetch-mode reddit job answered with 403 via `fetch`, suspend, disable) and Firefox 158 with geckodriver (connected, job done with 403 and 190240 bytes, suspended with `until`, disable). The Firefox script is not in the repository; the method is in `nodes/extension/README.md`. Go: `go vet` and `go test` pass, cross-compiles clean.
 
 ## Gaps (what is not built or not measured)
 
-- **Reddit viability is unmeasured from a residential line.** The only real run used a datacenter donor and got 403, which proves the path, not the success rate. X, Instagram, LinkedIn, Facebook and TikTok stay unreachable (login walls); nothing here claims them.
-- `country`, `asn` and `asn_class` on a device are never filled, so the datacenter exclusion (D6) is not enforced: the scheduler treats every node as eligible. It needs an offline ASN database file.
-- `last_seen_at` is written on connect only. Socket counters are per process. The fabric cache is per process. Circuit breaker state is in memory. The same-process fast path is not built, callers poll at `poll_interval_ms`.
-- Fetch-mode results are not cross-checked (sampled redundancy between two donors is not built). They are structurally checked and labelled `via: fetch`, and `tunnel` is preferred whenever a node can do it.
-- `DeepSearch` still skips hostile domains with a static list; Reddit URLs from social search are used through their snippet. Routing a hostile URL fetch through `/nodes/internal/fetch` is not built.
-- The Go client has no battery-aware pause (`pause_on_battery` is stored and sent but ignored by it), and treats a timed suspension as permanent until restarted. The extension enforces its own local hours in the browser clock, not the server timezone.
-- Not measured: Reddit and X success rates per address class, WebSocket idle timeouts through `pravda.education` and `devplace.net`, socket capacity of one worker, a 24 hour soak. The caps in `LimitSpec` are starting ceilings, not measurements.
-- **Launch is blocked on a recorded legal decision** (platform terms, donor terms, ISP terms, jurisdiction). Nothing here is enabled for any account: `nodes_enabled` is `0` until an administrator switches it on.
+- **Reddit success from a residential line is unmeasured.** The counters exist and read `n = 0`; every real run so far used a datacenter donor and got 403. X, Instagram, LinkedIn, Facebook and TikTok stay unreachable (login walls); nothing here claims them.
+- **No 24 hour soak.** The longest run was about 81 minutes. The caps in `LimitSpec` are ceilings backed by that run, not by a day.
+- **WebSocket idle timeout through the public hostname is unmeasured.** It needs a production connection, which needs the owner's go-ahead.
+- The extension has no battery pause (browsers expose no battery state to extensions); the Go client has one.
+- The geo file needs a monthly `geo-sync`; between syncs a new ASN is `unknown`.
+- Windows and macOS client builds compile but were not run.
+- **Launch is blocked on a recorded legal decision** (platform terms, donor terms, ISP terms, jurisdiction). Nothing here is enabled for any account: `nodes_enabled` is `0` until an administrator switches it on. This is a human decision and no code closes it.
 
 ## Rules for work in this directory
 

devplacepy/services/nodes/addressclass.py

@@ -0,0 +1,66 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import re
+
+CLASS_RESIDENTIAL, CLASS_MOBILE, CLASS_DATACENTER, CLASS_UNKNOWN = (
+    "residential",
+    "mobile",
+    "datacenter",
+    "unknown",
+)
+ELIGIBLE_CLASSES = (CLASS_RESIDENTIAL, CLASS_MOBILE)
+
+DATACENTER_ASNS = frozenset(
+    {
+        16509, 14618, 8987, 7224,
+        15169, 396982, 19527,
+        8075, 8068, 8069,
+        24940, 213230,
+        16276,
+        14061, 393406,
+        63949, 20940, 16625,
+        20473,
+        31898, 792,
+        45102, 37963,
+        45090, 132203,
+        13335,
+        51167,
+        12876,
+        60781, 16265, 30633,
+        9009,
+        36351,
+        27357, 19994,
+        54113,
+        47583,
+        197540,
+        8560,
+        26496,
+        22612,
+    }
+)
+
+DATACENTER_WORDS = re.compile(
+    r"\b(hosting|cloud|datacenter|data[ -]cent(?:er|re)|vps|servers?|colo(?:cation)?|dedicated|cdn|vpn|proxy|hoster|webhost\w*)\b",
+    re.IGNORECASE,
+)
+MOBILE_WORDS = re.compile(
+    r"\b(mobile|mobil|mobilfunk|wireless|cellular|lte|gsm|3g|4g|5g)\b", re.IGNORECASE
+)
+
+
+def classify(asn: int | str | None, org: str | None) -> str:
+    try:
+        number = int(asn) if asn not in (None, "") else 0
+    except (TypeError, ValueError):
+        number = 0
+    name = (org or "").strip()
+    if number <= 0 and not name:
+        return CLASS_UNKNOWN
+    if number in DATACENTER_ASNS or DATACENTER_WORDS.search(name):
+        return CLASS_DATACENTER
+    if MOBILE_WORDS.search(name):
+        return CLASS_MOBILE
+    return CLASS_RESIDENTIAL
+

devplacepy/services/nodes/broker.py

@@ -12,6 +12,7 @@ from dataclasses import dataclass
 from devplacepy.database import nodes as store
 from devplacepy.services.nodes.policy import Target
 from devplacepy.services.nodes.spec import SPEC
+from devplacepy.services.nodes.wake import HUB
 
 LIMITS = SPEC.limits
 
@@ -72,19 +73,37 @@ def decode_body(body_b64: str) -> tuple[bytes, bool]:
 
 
 class JobBroker:
-    def enqueue(self, caller_kind: str, caller_id: str, target: Target, mode: str) -> str:
-        return store.enqueue_request(
-            caller_kind, caller_id, target.url, target.method, target.adapter, mode
+    def enqueue(
+        self,
+        caller_kind: str,
+        caller_id: str,
+        target: Target,
+        mode: str,
+        *,
+        shadow_of: str = "",
+        exclude_device: str = "",
+    ) -> str:
+        uid = store.enqueue_request(
+            caller_kind,
+            caller_id,
+            target.url,
+            target.method,
+            target.adapter,
+            mode,
+            shadow_of=shadow_of,
+            exclude_device=exclude_device,
         )
+        HUB.notify_queued()
+        return uid
 
     def queued(self, limit: int = LIMITS.dispatch_batch) -> list[dict]:
         return store.list_queued(limit)
 
-    def claim(self, uid: str, device_uid: str) -> bool:
-        return store.claim_request(uid, device_uid)
+    def claim(self, uid: str, device_uid: str, asn_class: str = "") -> bool:
+        return store.claim_request(uid, device_uid, asn_class)
 
     def complete(self, uid: str, http_status: int, headers: dict, body: bytes, truncated: bool) -> bool:
-        return store.finish_request(
+        done = store.finish_request(
             uid,
             result_status=http_status,
             result_headers=json.dumps(headers),
@@ -92,9 +111,15 @@ class JobBroker:
             result_bytes=len(body),
             truncated=truncated,
         )
+        if done:
+            HUB.job_done(uid)
+        return done
 
     def fail(self, uid: str, error: str, *, expired: bool = False) -> bool:
-        return store.fail_request(uid, error, expired=expired)
+        failed = store.fail_request(uid, error, expired=expired)
+        if failed:
+            HUB.job_done(uid)
+        return failed
 
     def expire(self) -> dict[str, int]:
         return store.expire_requests()
@@ -117,11 +142,22 @@ class JobBroker:
     async def wait(self, uid: str, timeout: float) -> BrokerResult:
         deadline = time.monotonic() + timeout
         pause = LIMITS.poll_interval_ms / 1000
-        while True:
-            row = store.get_request(uid)
-            if row is not None and row["status"] in TERMINAL:
-                return self.read(uid)
-            if time.monotonic() >= deadline:
-                self.fail(uid, "caller_timeout")
-                return self.read(uid)
-            await asyncio.sleep(pause)
+        event = HUB.job_event(uid)
+        try:
+            while True:
+                row = store.get_request(uid)
+                if row is not None and row["status"] in TERMINAL:
+                    return self.read(uid)
+                if time.monotonic() >= deadline:
+                    self.fail(uid, "caller_timeout")
+                    return self.read(uid)
+                if event is None:
+                    await asyncio.sleep(pause)
+                    continue
+                event.clear()
+                try:
+                    await asyncio.wait_for(event.wait(), pause)
+                except asyncio.TimeoutError:
+                    pass
+        finally:
+            HUB.forget(uid)

devplacepy/services/nodes/cache.py

@@ -2,12 +2,16 @@
 
 from __future__ import annotations
 
+import base64
 import hashlib
+import json
 import time
+from datetime import datetime, timezone
 from collections import OrderedDict
 from dataclasses import dataclass
 from typing import Callable
 
+from devplacepy.database import nodes as store
 from devplacepy.services.nodes.spec import SPEC
 
 LIMITS = SPEC.limits
@@ -70,7 +74,14 @@ class ResultCache:
         return entry
 
     def put(
-        self, method: str, url: str, status: int, headers: dict, body: bytes, truncated: bool
+        self,
+        method: str,
+        url: str,
+        status: int,
+        headers: dict,
+        body: bytes,
+        truncated: bool,
+        age_seconds: float = 0.0,
     ) -> bool:
         if self.ttl_seconds <= 0 or status >= 500 or truncated:
             return False
@@ -78,7 +89,9 @@ class ResultCache:
             return False
         key = cache_key(method, url)
         self._drop(key)
-        self._entries[key] = CachedResult(status, dict(headers), body, truncated, self._clock())
+        self._entries[key] = CachedResult(
+            status, dict(headers), body, truncated, self._clock() - age_seconds
+        )
         self._bytes += len(body)
         while self._bytes > self.max_bytes and self._entries:
             oldest = next(iter(self._entries))
@@ -88,3 +101,56 @@ class ResultCache:
     def clear(self) -> None:
         self._entries.clear()
         self._bytes = 0
+
+
+class SharedResultCache(ResultCache):
+    def get(self, method: str, url: str) -> CachedResult | None:
+        local = super().get(method, url)
+        if local is not None:
+            return local
+        row = store.cache_get(cache_key(method, url))
+        if row is None:
+            return None
+        try:
+            headers = json.loads(row["headers"] or "{}")
+            body = base64.b64decode(row["body"] or "")
+            stored = datetime.fromisoformat(row["stored_at"])
+        except (ValueError, TypeError):
+            return None
+        age = max(0.0, (datetime.now(timezone.utc) - stored).total_seconds())
+        if age >= self.ttl_seconds:
+            return None
+        super().put(method, url, int(row["status"]), headers, body, False, age_seconds=age)
+        entry = self._entries.get(cache_key(method, url))
+        if entry is not None:
+            self.misses -= 1
+            self.hits += 1
+        return entry
+
+    def put(
+        self,
+        method: str,
+        url: str,
+        status: int,
+        headers: dict,
+        body: bytes,
+        truncated: bool,
+        age_seconds: float = 0.0,
+    ) -> bool:
+        kept = super().put(method, url, status, headers, body, truncated, age_seconds)
+        if kept:
+            store.cache_put(
+                cache_key(method, url),
+                method.upper(),
+                url,
+                status,
+                json.dumps(dict(headers)),
+                base64.b64encode(body).decode(),
+                len(body),
+                self.ttl_seconds,
+            )
+        return kept
+
+    def clear(self) -> None:
+        super().clear()
+        store.cache_clear()

devplacepy/services/nodes/circuit.py

@@ -28,11 +28,14 @@ class CircuitBreaker:
         failures: int = LIMITS.breaker_failures,
         open_seconds: float = LIMITS.breaker_open_seconds,
         clock: Callable[[], float] = time.monotonic,
+        wall: Callable[[], float] = time.time,
     ) -> None:
         self.failures = max(1, failures)
         self.open_seconds = open_seconds
         self._clock = clock
+        self._wall = wall
         self._circuits: dict[Hashable, _Circuit] = {}
+        self.dirty = False
 
     def state(self, key: Hashable) -> str:
         circuit = self._circuits.get(key)
@@ -55,7 +58,8 @@ class CircuitBreaker:
             self._circuits[key].probing = True
 
     def success(self, key: Hashable) -> None:
-        self._circuits.pop(key, None)
+        if self._circuits.pop(key, None) is not None:
+            self.dirty = True
 
     def failure(self, key: Hashable) -> None:
         circuit = self._circuits.setdefault(key, _Circuit())
@@ -63,9 +67,49 @@ class CircuitBreaker:
         circuit.failures += 1
         if circuit.failures >= self.failures:
             circuit.opened_at = self._clock()
+        self.dirty = True
 
     def forget(self, key: Hashable) -> None:
-        self._circuits.pop(key, None)
+        if self._circuits.pop(key, None) is not None:
+            self.dirty = True
 
     def open_count(self) -> int:
         return sum(1 for key in list(self._circuits) if self.state(key) != CLOSED)
+
+    def snapshot(self) -> list[dict]:
+        now_clock, now_wall = self._clock(), self._wall()
+        rows = []
+        for key, circuit in list(self._circuits.items()):
+            state = self.state(key)
+            rows.append(
+                {
+                    "key": ":".join(str(part) for part in key) if isinstance(key, tuple) else str(key),
+                    "failures": circuit.failures,
+                    "state": state,
+                    "opened_at": now_wall - (now_clock - circuit.opened_at) if circuit.failures >= self.failures else 0.0,
+                    "seconds_left": max(0.0, self.open_seconds - (now_clock - circuit.opened_at))
+                    if state == OPEN
+                    else 0.0,
+                }
+            )
+        rows.sort(key=lambda item: (-item["failures"], item["key"]))
+        return rows
+
+    def restore(self, rows: list[dict]) -> int:
+        now_clock, now_wall = self._clock(), self._wall()
+        restored = 0
+        for row in rows:
+            key = tuple(str(row["key"]).split(":", 1))
+            if len(key) != 2:
+                continue
+            failures = int(row.get("failures") or 0)
+            opened_wall = float(row.get("opened_at") or 0.0)
+            if failures < self.failures or opened_wall <= 0:
+                continue
+            age = max(0.0, now_wall - opened_wall)
+            if age >= self.open_seconds * 2:
+                continue
+            circuit = _Circuit(failures=failures, opened_at=now_clock - age)
+            self._circuits[key] = circuit
+            restored += 1
+        return restored

devplacepy/services/nodes/dispatcher.py

@@ -3,16 +3,21 @@
 from __future__ import annotations
 
 import asyncio
+import base64
+import hashlib
+import json
 import logging
 import time
 from typing import Callable
 
+from devplacepy.database import get_int_setting
 from devplacepy.database import nodes as store
 from devplacepy.services.nodes.broker import BodyError, JobBroker, clean_headers, decode_body
 from devplacepy.services.nodes.circuit import CircuitBreaker
 from devplacepy.services.nodes.ledger import UsageLedger
 from devplacepy.services.nodes.policy import DestinationPolicy, PolicyError, Target
 from devplacepy.services.nodes.registry import NodeCounters, NodeRegistry
+from devplacepy.services.nodes.shape import INCONCLUSIVE, MATCH, compare, shape_of
 from devplacepy.services.nodes.scheduler import MEGABYTE, NodeScheduler
 from devplacepy.services.nodes.session import Inflight, NodeSession
 from devplacepy.services.nodes.spec import MODE_FETCH, MODE_TUNNEL, SPEC
@@ -31,9 +36,13 @@ REQUEST_OVERHEAD_BYTES = 256
 DONOR_LIMIT_PAUSE_SECONDS = 300
 CONCURRENCY_PAUSE_SECONDS = 5
 THROTTLED_STATUSES = (429,)
+BLOCKED_STATUSES = (401, 403, 429, 451)
+SHADOW_KIND = "shadow"
+REDUNDANCY_SETTING = "nodes_redundancy_percent"
 DONOR_LIMIT_CODES = {
     "bandwidth_exhausted": None,
     "outside_hours": DONOR_LIMIT_PAUSE_SECONDS,
+    "on_battery": DONOR_LIMIT_PAUSE_SECONDS,
     "concurrency_exceeded": CONCURRENCY_PAUSE_SECONDS,
 }
 REFUSAL_CODES = frozenset(item.code for item in SPEC.refusals) - set(DONOR_LIMIT_CODES) - {
@@ -91,16 +100,24 @@ class NodeDispatcher:
             self.broker.fail(uid, exc.code)
             self.counters.inc("jobs_refused")
             return 0
-        key = (target.destination.suffix, row["mode"])
+        excluded = row.get("exclude_device") or ""
+        key = (target.destination.suffix, row["mode"], excluded)
         if key in saturated:
             return 0
         owner = row["caller_id"] if row["caller_kind"] == "user" else ""
-        found = self.scheduler.candidates(target, row["mode"])
+        if excluded:
+            prior = self.registry.get(excluded)
+            owner = prior.user_uid if prior is not None else owner
+        found = [
+            item
+            for item in self.scheduler.candidates(target, row["mode"])
+            if item.device_uid != excluded
+        ]
         if not found:
             saturated.add(key)
             return 0
         session = self.scheduler.choose(found, owner)
-        if session is None or not self.broker.claim(uid, session.device_uid):
+        if session is None or not self.broker.claim(uid, session.device_uid, session.asn_class):
             return 0
         job = Inflight(
             job_id=uid,
@@ -110,10 +127,13 @@ class NodeDispatcher:
             url=target.url,
             started=self._clock(),
             bytes_up=len(target.url.encode()) + REQUEST_OVERHEAD_BYTES,
+            adapter=target.adapter,
+            shadow_of=row.get("shadow_of") or "",
         )
         session.inflight[uid] = job
         self.scheduler.mark_started(session, target)
         transport = self.scheduler.transport_of(session, target, row["mode"]) or MODE_FETCH
+        job.transport = transport
         store.set_request_transport(uid, transport)
         if transport == MODE_TUNNEL:
             self._tunnel_tasks[uid] = asyncio.create_task(self._run_tunnel(session, job, target))
@@ -140,6 +160,7 @@ class NodeDispatcher:
         self.counters.inc("jobs_failed")
         if node_fault:
             self.breaker.failure(("node", session.device_uid))
+            self._observe(session, job, "failed")
         store.add_event("job_failed", device_uid=session.device_uid, user_uid=session.user_uid, detail=code)
         self.ledger.record_failure(session.device_uid, session.user_uid)
 
@@ -196,6 +217,7 @@ class NodeDispatcher:
         if not self.broker.complete(job.job_id, status, headers, body, truncated):
             return
         self._account(session, job, status, job.bytes_up, len(body))
+        self._after_result(session, job, status, headers, body)
 
     async def _run_tunnel(self, session: NodeSession, job: Inflight, target: Target) -> None:
         try:
@@ -215,9 +237,8 @@ class NodeDispatcher:
             return
         if result.truncated:
             self.counters.inc("results_truncated")
-        if not self.broker.complete(
-            job.job_id, result.status, clean_headers(result.headers), result.body, result.truncated
-        ):
+        headers = clean_headers(result.headers)
+        if not self.broker.complete(job.job_id, result.status, headers, result.body, result.truncated):
             return
         self._account(session, job, result.status, result.bytes_up, result.bytes_down)
 
@@ -237,12 +258,84 @@ class NodeDispatcher:
         if not accepted:
             session.used_today = int(session.settings.get("bandwidth_mb_day") or 0) * MEGABYTE
         self.breaker.success(("node", session.device_uid))
+        self._observe(session, job, "blocked" if status >= 500 or status in BLOCKED_STATUSES else "ok")
         destination = ("destination", job.destination)
         if status >= 500 or status in THROTTLED_STATUSES:
             self.breaker.failure(destination)
         else:
             self.breaker.success(destination)
 
+    def _observe(self, session: NodeSession, job: Inflight, kind: str) -> None:
+        if job.shadow_of or not job.adapter:
+            return
+        try:
+            self.counters.observe(job.adapter, session.asn_class, kind)
+        except KeyError:
+            logger.warning("outcome counter name rejected for %s", job.adapter)
+
+    @staticmethod
+    def redundancy_percent() -> int:
+        configured = get_int_setting(REDUNDANCY_SETTING, LIMITS.redundancy_percent)
+        return max(0, min(configured, LIMITS.redundancy_max_percent))
+
+    @staticmethod
+    def sampled(uid: str, percent: int) -> bool:
+        if percent <= 0:
+            return False
+        return int(hashlib.sha256(uid.encode()).hexdigest()[:8], 16) % 100 < percent
+
+    def _after_result(self, session: NodeSession, job: Inflight, status: int, headers: dict, body: bytes) -> None:
+        if job.shadow_of:
+            self._compare_shadow(session, job, status, headers, body)
+            return
+        if job.transport != MODE_FETCH or not self.sampled(job.job_id, self.redundancy_percent()):
+            return
+        row = store.get_request(job.job_id) or {}
+        try:
+            target = self.policy.check(job.url, job.method, MODE_FETCH)
+        except PolicyError:
+            return
+        try:
+            self.broker.enqueue(
+                SHADOW_KIND,
+                job.job_id,
+                target,
+                MODE_FETCH,
+                shadow_of=job.job_id,
+                exclude_device=row.get("device_uid") or session.device_uid,
+            )
+        except store.QueueFull:
+            return
+
+    def _compare_shadow(self, session: NodeSession, job: Inflight, status: int, headers: dict, body: bytes) -> None:
+        original = store.get_request(job.shadow_of)
+        if original is None or original.get("status") != "done":
+            self.counters.inc("redundancy_inconclusive")
+            return
+        try:
+            first_body = base64.b64decode(original.get("result_body") or "")
+            first_headers = json.loads(original.get("result_headers") or "{}")
+        except (ValueError, TypeError):
+            self.counters.inc("redundancy_inconclusive")
+            return
+        verdict = compare(
+            shape_of(int(original.get("result_status") or 0), first_headers, first_body),
+            shape_of(status, headers, body),
+        )
+        self.counters.inc("redundancy_checked")
+        if verdict == MATCH:
+            self.counters.inc("redundancy_matched")
+        elif verdict == INCONCLUSIVE:
+            self.counters.inc("redundancy_inconclusive")
+        else:
+            self.counters.inc("redundancy_mismatches")
+            store.raise_flag(
+                "redundancy_mismatch",
+                device_uid=session.device_uid,
+                user_uid=session.user_uid,
+                detail=f"{job.destination} first={original.get('device_uid')} second={session.device_uid}",
+            )
+
     async def sweep_inflight(self) -> int:
         limit = LIMITS.job_timeout_seconds + LIMITS.job_grace_seconds
         now = self._clock()

devplacepy/services/nodes/fabric.py

@@ -11,7 +11,7 @@ from devplacepy.database import get_int_setting, get_table
 from devplacepy.database import nodes as store
 from devplacepy.services.base import BaseService
 from devplacepy.services.nodes.broker import FAILED, JobBroker
-from devplacepy.services.nodes.cache import ResultCache
+from devplacepy.services.nodes.cache import ResultCache, SharedResultCache
 from devplacepy.services.nodes.consent import ConsentGate
 from devplacepy.services.nodes.policy import DestinationPolicy, PolicyError, Target
 from devplacepy.services.nodes.spec import SPEC
@@ -27,6 +27,7 @@ RETRYABLE = frozenset(
         "send_failed",
         "concurrency_exceeded",
         "outside_hours",
+        "on_battery",
         "bandwidth_exhausted",
     }
 )
@@ -81,7 +82,7 @@ class NodeFabric:
     ) -> None:
         self.policy = policy or DestinationPolicy()
         self.broker = broker or JobBroker()
-        self.cache = cache or ResultCache()
+        self.cache = cache or SharedResultCache()
         self.gate = gate or ConsentGate()
 
     def available(self) -> FabricStatus:

devplacepy/services/nodes/geo.py

@@ -0,0 +1,293 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import csv
+import gzip
+import ipaddress
+import logging
+import os
+import sqlite3
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+
+from devplacepy import config
+from devplacepy.services.nodes.addressclass import classify
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass(frozen=True)
+class GeoSpec:
+    asn_url: str = "https://download.db-ip.com/free/dbip-asn-lite-{month}.csv.gz"
+    country_url: str = "https://download.db-ip.com/free/dbip-country-lite-{month}.csv.gz"
+    attribution: str = "IP geolocation by DB-IP.com (CC BY 4.0)"
+    max_rows: int = 2_000_000
+    min_asn_rows: int = 100_000
+    min_country_rows: int = 100_000
+    insert_chunk: int = 5000
+    download_timeout_seconds: int = 180
+    max_download_bytes: int = 64 * 1024 * 1024
+    months_back: int = 2
+    checks: tuple[tuple[str, int], ...] = (("1.1.1.1", 13335), ("8.8.8.8", 15169))
+
+
+SPEC = GeoSpec()
+
+SCHEMA = (
+    "CREATE TABLE geo_asn (family INTEGER NOT NULL, first BLOB NOT NULL, last BLOB NOT NULL, "
+    "asn INTEGER NOT NULL, org TEXT NOT NULL, PRIMARY KEY (family, first)) WITHOUT ROWID",
+    "CREATE TABLE geo_country (family INTEGER NOT NULL, first BLOB NOT NULL, last BLOB NOT NULL, "
+    "country TEXT NOT NULL, PRIMARY KEY (family, first)) WITHOUT ROWID",
+    "CREATE TABLE geo_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)",
+)
+
+
+UNKNOWN_COUNTRY = "ZZ"
+
+
+class GeoError(RuntimeError):
+    pass
+
+
+@dataclass(frozen=True)
+class GeoResult:
+    country: str = ""
+    asn: int = 0
+    org: str = ""
+    asn_class: str = "unknown"
+
+
+def pack(address: str) -> tuple[int, bytes]:
+    ip = ipaddress.ip_address(address.strip().strip("[]"))
+    if ip.version == 6 and ip.ipv4_mapped is not None:
+        ip = ip.ipv4_mapped
+    return ip.version, ip.packed
+
+
+class GeoDatabase:
+    def __init__(self, path: Path | None = None) -> None:
+        self.path = Path(path) if path else config.NODE_GEO_DB
+        self.warned = False
+
+    def available(self) -> bool:
+        return self.path.is_file() and self.path.stat().st_size > 0
+
+    def _connect(self) -> sqlite3.Connection:
+        return sqlite3.connect(f"file:{self.path}?mode=ro", uri=True, timeout=5)
+
+    def _range(self, conn: sqlite3.Connection, table: str, columns: str, family: int, packed: bytes):
+        row = conn.execute(
+            f"SELECT {columns}, last FROM {table} WHERE family = ? AND first <= ? "
+            "ORDER BY first DESC LIMIT 1",
+            (family, packed),
+        ).fetchone()
+        if row is None or row[-1] < packed:
+            return None
+        return row[:-1]
+
+    def lookup(self, address: str) -> GeoResult:
+        if not address or not self.available():
+            return GeoResult()
+        try:
+            family, packed = pack(address)
+        except ValueError:
+            return GeoResult()
+        try:
+            conn = self._connect()
+        except sqlite3.Error:
+            return GeoResult()
+        try:
+            asn_row = self._range(conn, "geo_asn", "asn, org", family, packed)
+            country_row = self._range(conn, "geo_country", "country", family, packed)
+        except sqlite3.Error:
+            return GeoResult()
+        finally:
+            conn.close()
+        asn, org = (int(asn_row[0]), str(asn_row[1])) if asn_row else (0, "")
+        country = str(country_row[0]) if country_row else ""
+        country = "" if country.upper() == UNKNOWN_COUNTRY else country
+        return GeoResult(country=country[:2].upper(), asn=asn, org=org, asn_class=classify(asn, org))
+
+    def status(self) -> dict:
+        if not self.available():
+            return {"available": False, "path": str(self.path)}
+        conn = self._connect()
+        try:
+            meta = dict(conn.execute("SELECT key, value FROM geo_meta").fetchall())
+        except sqlite3.Error:
+            meta = {}
+        finally:
+            conn.close()
+        return {"available": True, "path": str(self.path), **meta, "attribution": SPEC.attribution}
+
+
+def months(now: datetime | None = None) -> list[str]:
+    moment = now or datetime.now(timezone.utc)
+    found = []
+    for back in range(SPEC.months_back + 1):
+        year, month = moment.year, moment.month - back
+        while month < 1:
+            month += 12
+            year -= 1
+        found.append(f"{year:04d}-{month:02d}")
+    return found
+
+
+def parse_asn_rows(lines):
+    for row in csv.reader(lines):
+        if len(row) < 4:
+            continue
+        first, last, asn, org = row[0], row[1], row[2], row[3]
+        try:
+            family, low = pack(first)
+            _, high = pack(last)
+            number = int(asn)
+        except ValueError:
+            continue
+        yield family, low, high, number, org[:200]
+
+
+def parse_country_rows(lines):
+    for row in csv.reader(lines):
+        if len(row) < 3:
+            continue
+        try:
+            family, low = pack(row[0])
+            _, high = pack(row[1])
+        except ValueError:
+            continue
+        yield family, low, high, row[2][:2].upper()
+
+
+def _insert(conn: sqlite3.Connection, sql: str, rows) -> int:
+    count = 0
+    batch = []
+    for row in rows:
+        batch.append(row)
+        count += 1
+        if count > SPEC.max_rows:
+            raise GeoError(f"more than {SPEC.max_rows} rows, refusing to load")
+        if len(batch) >= SPEC.insert_chunk:
+            conn.executemany(sql, batch)
+            batch = []
+    if batch:
+        conn.executemany(sql, batch)
+    return count
+
+
+def build(asn_lines, country_lines, target: Path, month: str) -> dict:
+    temporary = target.with_suffix(".tmp")
+    temporary.unlink(missing_ok=True)
+    conn = sqlite3.connect(temporary)
+    try:
+        for statement in SCHEMA:
+            conn.execute(statement)
+        asn_rows = _insert(conn, "INSERT OR REPLACE INTO geo_asn VALUES (?,?,?,?,?)", parse_asn_rows(asn_lines))
+        country_rows = _insert(conn, "INSERT OR REPLACE INTO geo_country VALUES (?,?,?,?)", parse_country_rows(country_lines))
+        if asn_rows < SPEC.min_asn_rows or country_rows < SPEC.min_country_rows:
+            raise GeoError(
+                f"dataset too small (asn {asn_rows}, country {country_rows}), refusing to replace the current one"
+            )
+        conn.executemany(
+            "INSERT INTO geo_meta VALUES (?,?)",
+            [
+                ("month", month),
+                ("loaded_at", datetime.now(timezone.utc).isoformat()),
+                ("asn_rows", str(asn_rows)),
+                ("country_rows", str(country_rows)),
+            ],
+        )
+        conn.commit()
+    except Exception:
+        conn.close()
+        temporary.unlink(missing_ok=True)
+        raise
+    conn.close()
+    verify(temporary)
+    return {"month": month, "asn_rows": asn_rows, "country_rows": country_rows, "temporary": temporary}
+
+
+def verify(path: Path) -> None:
+    probe = GeoDatabase(path)
+    for address, expected in SPEC.checks:
+        found = probe.lookup(address).asn
+        if found != expected:
+            path.unlink(missing_ok=True)
+            raise GeoError(f"sanity check failed: {address} resolved to AS{found}, expected AS{expected}")
+
+
+def install(built: dict, target: Path) -> None:
+    os.replace(built["temporary"], target)
+
+
+async def _download(url: str, destination: Path) -> bool:
+    from devplacepy.stealth import stealth_async_client
+
+    async with stealth_async_client(timeout=SPEC.download_timeout_seconds, follow_redirects=True) as client:
+        async with client.stream("GET", url) as response:
+            if response.status_code == 404:
+                return False
+            if response.status_code != 200:
+                raise GeoError(f"{url} answered {response.status_code}")
+            total = 0
+            with open(destination, "wb") as handle:
+                async for chunk in response.aiter_bytes():
+                    total += len(chunk)
+                    if total > SPEC.max_download_bytes:
+                        raise GeoError(f"{url} exceeds {SPEC.max_download_bytes} bytes")
+                    handle.write(chunk)
+    return True
+
+
+async def fetch_month(month: str, workdir: Path) -> tuple[Path, Path] | None:
+    asn_file = workdir / f"asn-{month}.csv.gz"
+    country_file = workdir / f"country-{month}.csv.gz"
+    if not await _download(SPEC.asn_url.format(month=month), asn_file):
+        return None
+    if not await _download(SPEC.country_url.format(month=month), country_file):
+        return None
+    return asn_file, country_file
+
+
+def build_from_files(asn_file: Path, country_file: Path, target: Path, month: str) -> dict:
+    with gzip.open(asn_file, "rt", encoding="utf-8", newline="") as asn_handle, gzip.open(
+        country_file, "rt", encoding="utf-8", newline=""
+    ) as country_handle:
+        return build(asn_handle, country_handle, target, month)
+
+
+async def geo_sync(*, dry_run: bool = False, month: str = "", target: Path | None = None, workdir: Path | None = None) -> dict:
+    destination = Path(target) if target else config.NODE_GEO_DB
+    destination.parent.mkdir(parents=True, exist_ok=True)
+    scratch = Path(workdir) if workdir else destination.parent
+    scratch.mkdir(parents=True, exist_ok=True)
+    candidates = [month] if month else months()
+    for candidate in candidates:
+        files = await fetch_month(candidate, scratch)
+        if files is None:
+            continue
+        try:
+            built = build_from_files(files[0], files[1], destination, candidate)
+        finally:
+            for item in files:
+                item.unlink(missing_ok=True)
+        summary = {key: value for key, value in built.items() if key != "temporary"}
+        if dry_run:
+            built["temporary"].unlink(missing_ok=True)
+            return {**summary, "installed": False}
+        install(built, destination)
+        return {**summary, "installed": True}
+    raise GeoError(f"no dataset found for {', '.join(candidates)}")
+
+
+_shared: GeoDatabase | None = None
+
+
+def shared_geo() -> GeoDatabase:
+    global _shared
+    if _shared is None or _shared.path != config.NODE_GEO_DB:
+        _shared = GeoDatabase()
+    return _shared
+

devplacepy/services/nodes/registry.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import re
 from collections import Counter
 
 from devplacepy.database import get_int_setting
@@ -26,21 +27,57 @@ COUNTER_NAMES: tuple[str, ...] = (
     "jobs_refused",
     "integrity_failures",
     "results_truncated",
+    "geo_unavailable",
+    "redundancy_checked",
+    "redundancy_matched",
+    "redundancy_mismatches",
+    "redundancy_inconclusive",
 )
 
+OUTCOME_PREFIX = "outcome:"
+OUTCOME_KINDS = ("ok", "blocked", "failed")
+OUTCOME_PATTERN = re.compile(r"^outcome:[a-z0-9_]{1,24}:[a-z]{1,12}:(ok|blocked|failed)$")
+
+
+def outcome_name(adapter: str, asn_class: str, kind: str) -> str:
+    name = f"{OUTCOME_PREFIX}{adapter}:{asn_class or 'unknown'}:{kind}"
+    if not OUTCOME_PATTERN.match(name):
+        raise KeyError(name)
+    return name
+
 
 class NodeCounters:
     def __init__(self) -> None:
         self._counts: Counter[str] = Counter({name: 0 for name in COUNTER_NAMES})
+        self._outcomes: Counter[str] = Counter()
+        self._pending: Counter[str] = Counter()
 
     def inc(self, name: str) -> None:
         if name not in COUNTER_NAMES:
             raise KeyError(name)
         self._counts[name] += 1
+        self._pending[name] += 1
+
+    def observe(self, adapter: str, asn_class: str, kind: str) -> None:
+        name = outcome_name(adapter, asn_class, kind)
+        self._outcomes[name] += 1
+        self._pending[name] += 1
 
     def snapshot(self) -> dict[str, int]:
         return {name: self._counts[name] for name in COUNTER_NAMES}
 
+    def outcomes(self) -> dict[str, int]:
+        return dict(self._outcomes)
+
+    def take_pending(self) -> dict[str, int]:
+        taken = {name: value for name, value in self._pending.items() if value > 0}
+        self._pending.clear()
+        return taken
+
+    def restore_pending(self, taken: dict[str, int]) -> None:
+        for name, value in taken.items():
+            self._pending[name] += value
+
 
 class NodeRegistry:
     def __init__(self, counters: NodeCounters) -> None:
@@ -78,10 +115,10 @@ class NodeRegistry:
         if self._sessions.get(session.device_uid) is session:
             del self._sessions[session.device_uid]
 
-    async def kick(self, session: NodeSession, reason: str) -> None:
+    async def kick(self, session: NodeSession, reason: str, until: str = "") -> None:
         self.release(session)
         self.counters.inc("kicked")
-        await session.kick(reason)
+        await session.kick(reason, until)
 
     async def kick_all(self, reason: str) -> int:
         sessions = self.sessions()

devplacepy/services/nodes/scheduler.py

@@ -9,6 +9,8 @@ from datetime import datetime, timezone
 from typing import Callable
 from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
 
+from devplacepy.database import get_int_setting
+from devplacepy.services.nodes.addressclass import CLASS_UNKNOWN, ELIGIBLE_CLASSES
 from devplacepy.services.nodes.circuit import CircuitBreaker
 from devplacepy.services.nodes.policy import Target
 from devplacepy.services.nodes.registry import NodeRegistry
@@ -20,6 +22,8 @@ MEGABYTE = 1024 * 1024
 MINUTES_PER_DAY = 24 * 60
 
 SUPPORTED_MODES = (MODE_TUNNEL, MODE_FETCH)
+OVERRIDE_ALLOW = "allow"
+ALLOW_UNKNOWN_SETTING = "nodes_allow_unknown_class"
 
 
 def _minutes(value: str) -> int | None:
@@ -73,6 +77,7 @@ class NodeScheduler:
         self._wall = wall
         self._rng = rng or random.Random()
         self._destination_started: dict[str, float] = {}
+        self.class_refusals = 0
 
     @staticmethod
     def node_key(session: NodeSession) -> tuple[str, str]:
@@ -102,9 +107,30 @@ class NodeScheduler:
         last = self._destination_started.get(target.destination.suffix)
         return last is None or self._clock() - last >= interval
 
-    def node_ready(self, session: NodeSession, target: Target, transport: str) -> bool:
+    @staticmethod
+    def allow_unknown_class() -> bool:
+        return get_int_setting(ALLOW_UNKNOWN_SETTING, 0) == 1
+
+    @staticmethod
+    def class_allowed(session: NodeSession, allow_unknown: bool) -> bool:
+        if session.class_override == OVERRIDE_ALLOW or session.asn_class in ELIGIBLE_CLASSES:
+            return True
+        return allow_unknown and session.asn_class == CLASS_UNKNOWN
+
+    def node_ready(
+        self,
+        session: NodeSession,
+        target: Target,
+        transport: str,
+        allow_unknown: bool | None = None,
+    ) -> bool:
         if session.closed or self._clock() < session.paused_until:
             return False
+        if allow_unknown is None:
+            allow_unknown = self.allow_unknown_class()
+        if not self.class_allowed(session, allow_unknown):
+            self.class_refusals += 1
+            return False
         if transport not in session.modes:
             return False
         if len(session.inflight) >= session.max_concurrent:
@@ -125,11 +151,12 @@ class NodeScheduler:
     def candidates(self, target: Target, mode: str = MODE_ANY) -> list[NodeSession]:
         if not self.destination_ready(target) or not self.breaker.allows(self.destination_key(target)):
             return []
+        allow_unknown = self.allow_unknown_class()
         for transport in self.transports_for(target, mode):
             found = [
                 session
                 for session in self.registry.sessions()
-                if self.node_ready(session, target, transport)
+                if self.node_ready(session, target, transport, allow_unknown)
             ]
             if found:
                 return found

devplacepy/services/nodes/service.py

@@ -9,9 +9,14 @@ import time
 from devplacepy.database.nodes import (
     STATUS_ACTIVE,
     add_event,
+    cache_prune,
     effective_status,
+    flush_counters,
     get_devices_by_uids,
     get_settings_by_users,
+    load_circuits,
+    save_circuits,
+    touch_devices,
 )
 from devplacepy.services.base import BaseService, ConfigField
 from devplacepy.services.nodes.broker import JobBroker
@@ -23,6 +28,7 @@ from devplacepy.services.nodes.ledger import UsageLedger
 from devplacepy.services.nodes.registry import NodeCounters, NodeRegistry
 from devplacepy.services.nodes.scheduler import NodeScheduler
 from devplacepy.services.nodes.spec import SPEC
+from devplacepy.services.nodes.wake import HUB
 
 logger = logging.getLogger(__name__)
 
@@ -84,6 +90,8 @@ class NodesService(BaseService):
         )
         self.fabric = NodeFabric(broker=self.broker, gate=self.gate)
         self._last_prune = 0.0
+        self._last_touch = 0.0
+        self._last_flush = 0.0
         self._dispatch_task: asyncio.Task | None = None
 
     def ensure_dispatching(self) -> None:
@@ -91,6 +99,7 @@ class NodesService(BaseService):
         task = self._dispatch_task
         if task is not None and not task.done() and task.get_loop() is loop:
             return
+        HUB.attach()
         self._dispatch_task = loop.create_task(self._dispatch_loop())
 
     async def _dispatch_loop(self) -> None:
@@ -102,9 +111,12 @@ class NodesService(BaseService):
                 raise
             except Exception:
                 logger.exception("node dispatch tick failed")
-            await asyncio.sleep(pause)
+            await HUB.wait_queued(pause)
 
     async def on_enable(self) -> None:
+        restored = self.breaker.restore(load_circuits())
+        if restored:
+            self.log(f"restored {restored} open circuit(s)")
         self.ensure_dispatching()
 
     async def _refresh_settings(self, sessions) -> None:
@@ -127,11 +139,13 @@ class NodesService(BaseService):
         verdicts = self.gate.allows_many(item.user_uid for item in sessions)
         kicked = 0
         for session in sessions:
-            reason = self._reason_to_kick(
-                switched_on, devices.get(session.device_uid), verdicts[session.user_uid]
-            )
+            device = devices.get(session.device_uid)
+            if device is not None:
+                session.class_override = device.get("class_override") or ""
+            reason = self._reason_to_kick(switched_on, device, verdicts[session.user_uid])
             if reason:
-                await self.registry.kick(session, reason)
+                until = (device.get("suspended_until") or "").strip() if reason == "suspended" and device else ""
+                await self.registry.kick(session, reason, until)
                 add_event(
                     "kick",
                     device_uid=session.device_uid,
@@ -155,13 +169,37 @@ class NodesService(BaseService):
     def _prune_due(self) -> bool:
         return time.monotonic() - self._last_prune >= LIMITS.prune_interval_seconds
 
+    def persist_state(self, *, force: bool = False) -> None:
+        now = time.monotonic()
+        if force or now - self._last_touch >= LIMITS.last_seen_write_seconds:
+            self._last_touch = now
+            touch_devices(item.device_uid for item in self.registry.sessions())
+        if force or now - self._last_flush >= LIMITS.counter_flush_seconds:
+            self._last_flush = now
+            pending = self.counters.take_pending()
+            try:
+                flush_counters(pending)
+            except Exception:
+                self.counters.restore_pending(pending)
+                raise
+        if self.breaker.dirty:
+            self.breaker.dirty = False
+            try:
+                save_circuits(self.breaker.snapshot())
+            except Exception:
+                self.breaker.dirty = True
+                raise
+
     async def run_once(self) -> None:
         kicked = await self.reconcile()
         if kicked:
             self.log(f"disconnected {kicked} node(s)")
+        self.persist_state()
         if self._prune_due():
             self._last_prune = time.monotonic()
             pruned = self.ledger.prune()
+            removed = cache_prune(LIMITS.cache_max_bytes)
+            pruned["cache"] = removed["expired"] + removed["evicted"]
             if any(pruned.values()):
                 self.log(f"pruned {pruned}")
 
@@ -170,6 +208,7 @@ class NodesService(BaseService):
             self._dispatch_task.cancel()
             self._dispatch_task = None
         await self.registry.kick_all("service_stopped")
+        self.persist_state(force=True)
 
     def collect_metrics(self) -> dict:
         counts = self.counters.snapshot()

devplacepy/services/nodes/session.py

@@ -35,6 +35,9 @@ class Inflight:
     url: str
     started: float
     bytes_up: int
+    adapter: str = ""
+    shadow_of: str = ""
+    transport: str = ""
 
 
 JobSink = Callable[["NodeSession", Inflight, dict], Awaitable[None]]
@@ -46,6 +49,8 @@ class NodeSession:
         self.device = device
         self.device_uid: str = device["uid"]
         self.user_uid: str = device["user_uid"]
+        self.asn_class: str = device.get("asn_class") or "unknown"
+        self.class_override: str = device.get("class_override") or ""
         self.connected_at = time.monotonic()
         self.missed_pings = 0
         self.hello: dict = {}
@@ -96,11 +101,14 @@ class NodeSession:
         except Exception:
             pass
 
-    async def kick(self, reason: str) -> None:
+    async def kick(self, reason: str, until: str = "") -> None:
         if self.closed:
             return
         try:
-            await self.send("kick", reason=reason)
+            if until:
+                await self.send("kick", reason=reason, until=until)
+            else:
+                await self.send("kick", reason=reason)
         except Exception:
             pass
         await self.close(CLOSE_KICKED, reason)
@@ -135,6 +143,7 @@ class NodeSession:
             limits=LIMITS.describe(),
             destinations=[item.describe() for item in SPEC.destinations],
             settings=self.settings,
+            device_label=str(self.device.get("label") or ""),
         )
         return frame
 

devplacepy/services/nodes/shape.py

@@ -0,0 +1,70 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass
+
+MATCH, MISMATCH, INCONCLUSIVE = "match", "mismatch", "inconclusive"
+SIZE_BAND = 4.0
+SMALL_BODY_BYTES = 512
+NESTED_KEY_LIMIT = 64
+TITLE = re.compile(rb"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
+HEADING = re.compile(rb"<h1[^>]*>(.*?)</h1>", re.IGNORECASE | re.DOTALL)
+TAGS = re.compile(rb"<[^>]+>")
+SPACE = re.compile(rb"\s+")
+
+
+@dataclass(frozen=True)
+class Shape:
+    status: int
+    kind: str
+    signature: frozenset | str
+    size: int
+
+    @property
+    def clean(self) -> bool:
+        return 200 <= self.status < 400
+
+
+def _keys(value, depth: int = 0) -> frozenset:
+    if not isinstance(value, dict):
+        return frozenset()
+    found = set(value)
+    if depth < 2:
+        for key, item in list(value.items())[:NESTED_KEY_LIMIT]:
+            found.update(f"{key}.{inner}" for inner in _keys(item, depth + 1))
+    return frozenset(found)
+
+
+def _text(raw: bytes) -> str:
+    return SPACE.sub(b" ", TAGS.sub(b"", raw)).strip().decode("utf-8", "replace")[:200]
+
+
+def shape_of(status: int, headers: dict, body: bytes) -> Shape:
+    kind = str(headers.get("content-type", "")).split(";")[0].strip().lower()
+    size = len(body)
+    if "json" in kind:
+        try:
+            return Shape(status, "json", _keys(json.loads(body)), size)
+        except ValueError:
+            return Shape(status, "json-invalid", "", size)
+    if "html" in kind:
+        match = TITLE.search(body) or HEADING.search(body)
+        return Shape(status, "html", _text(match.group(1)) if match else "", size)
+    return Shape(status, kind or "unknown", "", size)
+
+
+def compare(first: Shape, second: Shape) -> str:
+    if not (first.clean and second.clean):
+        return INCONCLUSIVE
+    if first.kind != second.kind:
+        return MISMATCH
+    if first.signature != second.signature:
+        return MISMATCH
+    if min(first.size, second.size) >= SMALL_BODY_BYTES:
+        ratio = max(first.size, second.size) / max(1, min(first.size, second.size))
+        if ratio > SIZE_BAND:
+            return MISMATCH
+    return MATCH

devplacepy/services/nodes/spec.py

@@ -104,6 +104,11 @@ class LimitSpec:
     dispatch_batch: int = 50
     url_max_chars: int = 2048
     poll_interval_ms: int = 100
+    last_seen_write_seconds: int = 60
+    redundancy_percent: int = 5
+    redundancy_max_percent: int = 20
+    circuit_snapshot_max: int = 100
+    counter_flush_seconds: int = 10
 
     def describe(self) -> dict[str, int]:
         return dict(self.__dict__)
@@ -232,6 +237,7 @@ FRAMES: tuple[FrameSpec, ...] = (
             _f("limits", "object"),
             _f("destinations", "list"),
             _f("settings", "object"),
+            _f("device_label", "str", False),
         ),
     ),
     FrameSpec("ping", SERVER_TO_CLIENT),
@@ -258,7 +264,7 @@ FRAMES: tuple[FrameSpec, ...] = (
     ),
     FrameSpec("tunnel_close", SERVER_TO_CLIENT, (_f("conn_id", "str"),)),
     FrameSpec("settings_update", SERVER_TO_CLIENT, (_f("settings", "object"),)),
-    FrameSpec("kick", SERVER_TO_CLIENT, (_f("reason", "str"),)),
+    FrameSpec("kick", SERVER_TO_CLIENT, (_f("reason", "str"), _f("until", "str", False))),
 )
 
 DESTINATIONS: tuple[DestinationSpec, ...] = (
@@ -275,6 +281,7 @@ REFUSALS: tuple[RefusalSpec, ...] = (
     RefusalSpec("bandwidth_exhausted", "donor daily bandwidth cap reached"),
     RefusalSpec("concurrency_exceeded", "donor concurrency cap reached"),
     RefusalSpec("outside_hours", "outside the donor's active hours"),
+    RefusalSpec("on_battery", "donor is running on battery and asked to pause"),
     RefusalSpec("body_too_large", "response exceeds the body cap"),
     RefusalSpec("timeout", "job exceeded its timeout"),
 )

devplacepy/services/nodes/wake.py

@@ -0,0 +1,70 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import asyncio
+
+
+class WakeHub:
+    def __init__(self) -> None:
+        self.enabled = True
+        self._loop: asyncio.AbstractEventLoop | None = None
+        self._queued: asyncio.Event | None = None
+        self._jobs: dict[str, asyncio.Event] = {}
+
+    def attach(self) -> None:
+        self._loop = asyncio.get_running_loop()
+        self._queued = asyncio.Event()
+
+    def _set(self, event: asyncio.Event) -> None:
+        loop = self._loop
+        if loop is None or loop.is_closed():
+            return
+        try:
+            running = asyncio.get_running_loop()
+        except RuntimeError:
+            running = None
+        if running is loop:
+            event.set()
+        else:
+            loop.call_soon_threadsafe(event.set)
+
+    def notify_queued(self) -> None:
+        if self.enabled and self._queued is not None:
+            self._set(self._queued)
+
+    async def wait_queued(self, timeout: float) -> bool:
+        event = self._queued
+        if not self.enabled or event is None:
+            await asyncio.sleep(timeout)
+            return False
+        try:
+            await asyncio.wait_for(event.wait(), timeout)
+        except asyncio.TimeoutError:
+            return False
+        event.clear()
+        return True
+
+    def job_event(self, uid: str) -> asyncio.Event | None:
+        if not self.enabled or self._loop is None:
+            return None
+        try:
+            if asyncio.get_running_loop() is not self._loop:
+                return None
+        except RuntimeError:
+            return None
+        event = self._jobs.get(uid)
+        if event is None:
+            event = self._jobs[uid] = asyncio.Event()
+        return event
+
+    def job_done(self, uid: str) -> None:
+        event = self._jobs.get(uid)
+        if event is not None:
+            self._set(event)
+
+    def forget(self, uid: str) -> None:
+        self._jobs.pop(uid, None)
+
+
+HUB = WakeHub()

devplacepy/services/skills/CLAUDE.md

@@ -44,4 +44,4 @@ A `SKILL.md` file (YAML frontmatter with a required `name` and `description`, th
 
 ## Gaps
 
-- No test installs from a live network origin. The SSRF guard refuses the loopback address a local test server would use, so the network leg (`guarded_async_client` against a real socket) is exercised only in production.
+- The network leg is covered by `tests/unit/services/skills/fetcher.py`, which serves a real local socket and reaches it through the host's own global IPv4 so the SSRF guard runs unpatched (bundled files, directory URL, loopback and redirect-to-loopback refused, redirect loop, oversize, HTML, 404; 36 tests). It skips with a stated reason when the host has no global IPv4 (a CI runner behind NAT).

devplacepy/services/sync/CLAUDE.md

@@ -14,11 +14,12 @@ This file documents community sync (`devplacepy/services/sync/`, `devplacepy/rou
 | Devii tools `sync_users`, `sync_posts`, `sync_comments` | `services/devii/actions/catalog/sync.py` | built, `requires_admin`; Devii sends forms, so `records` is a JSON array string (`SyncBatchForm.decode_records`) |
 | Admin settings fields `sync_enabled`, `sync_origins` | `AdminSettingsForm`, `admin_settings.html`, `admin_save_settings` | built, origins normalized on save |
 | Scheduled sender | `services/community/` (`CommunityService`, see its CLAUDE.md) | built, opt-in, dry run by default |
-| Export endpoint, vote and reaction sync | none | not built, see Gaps |
+| Projects, polls, votes, reactions, follows, poll votes, attachments | `processors.py`, one leaf per kind in `routers/sync/` | built, tested |
+| Pending queue, export, deletions, alias healing | `pending.py`, `exporter.py`, `deletions.py`, `healing.py` | built, tested |
 
 ## Model
 
-Every synced row has a `sync_links` row `(origin, kind, origin_uid) -> local_uid`, unique on the triple. `origin` is the sender's stable id (a hostname, lower-cased, 2-63 characters of `a-z0-9.-`). `kind` is `users`, `posts` or `comments`. The link also stores a `fingerprint` of the normalized record (sha256 prefix, excludes `updated_at`) and `origin_updated_at`.
+Every synced row has a `sync_links` row `(origin, kind, origin_uid) -> local_uid`, unique on the triple. `origin` is the sender's stable id (a hostname, lower-cased, 2-63 characters of `a-z0-9.-`). `kind` is one of the ten in `spec.py` (`users`, `projects`, `posts`, `comments`, `polls`, `votes`, `reactions`, `follows`, `poll_votes`, `attachments`). The link also stores a `fingerprint` of the normalized record (sha256 prefix, excludes `updated_at`) and `origin_updated_at`.
 
 Per record, `RecordProcessor.apply` does: validate (Pydantic), resolve references (author, post, parent) through the links, content filter, then:
 
@@ -52,13 +53,31 @@ The link is claimed with one `INSERT OR IGNORE` against the unique index and `ro
 
 A stateful fuzz (3000 random upserts, tombstones and edits over two origins and all three kinds, invariants checked every 100 steps: no dangling links, no duplicate usernames, no orphan comments, parents on the same post, `updated_at >= created_at`) produced no `failed` result and no violation.
 
+## Added since the first build
+
+- **Ten kinds.** Projects (a post's `project_uid` is omitted from the fingerprint when None), polls with options, votes and reactions (`recompute_stars` is one atomic UPDATE, so stars match the vote rows), follows (natural-key upsert, `self_follow` refused), poll votes, attachments (base64 content with a sha256 fingerprint, capped by `attachment_bytes`).
+- **Pending queue (`pending.py`, table `sync_pending`).** A record whose post, parent, poll or project is not here yet is `deferred` and parked, bounded by a per-origin cap (`pending_queue_full` when exceeded) and expiry; `PendingQueue.release` retries it when the missing record arrives, and the batch result carries `released`. `unknown_option` is never deferred. Measured: 16 processes sending one post, one user and a comment in shuffled order left 16 of 16 comment links and 0 parked records.
+- **Export (`GET /sync/export/{kind}`).** Keyset pagination over `(changed, uid)` with expression indexes (`EXPORT_CHANGED`, `ensure_export_indexes`), an echo guard that never re-exports a row that arrived by sync, public data only (private projects are not exported, a post's private project link is redacted), audited as `sync.export`. Votes, reactions and poll votes are not exported because no surface exposes who cast them.
+- **Deletions (`GET /sync/deletions`, `POST /sync/deletions/ack`).** The origin learns which of its records we removed; an acknowledged record answers `acknowledged_deleted` afterwards instead of resurrecting. Audited as `sync.deletions.ack`.
+- **Alias healing (`healing.py`, `AliasHealer`).** Follows an administrator rename of a clone and suffixed clones, rewrites stored mentions, at most `heal_rewrites_per_run` rewrites a run, runs before and after the push.
+- **Candidates.** `Selector(persist=)` parks posts held back by `min_comments` in `sync_candidates` and revives them when later runs see enough comments.
+
+Measured: 40 processes racing one new vote against one post gave 1 `created`, 39 `unchanged`, 1 vote row, 1 link and `stars = 1`.
+
+## Realistic handles for new community users (`handles.py`)
+
+Names that reach us from a feed are often not handles at all (`abner_page_by_abnercoimbre`, `libreboot_org_via_runxiyu`, `via_rberger_2026-10-15_12_36_38_`). `HandleBuilder` (spec: `HandleLimits` / `HANDLES` in `spec.py`) turns the sent name into a handle a person would choose, and applies **only** when a batch is run with `humanize=True`, which only `SyncPusher` (community runs) sets. The direct `/sync/*` API keeps the sender's names exactly as before.
+
+- **Extract the person.** The stamp is removed, the token after the last `by` or `via` wins over the site before it, and a domain-shaped name (`costa_jacobpalm_dk`) reduces to its second-level label (`jacobpalm`, a fixed top-level-label list in the spec). Lower case, at most 18 characters of base.
+- **Then a pattern, weighted from the one empirical study found** (arXiv 1101.5578, 2011, Latin names only: about 70% of real usernames contain part of the real name, about 30% are plain concatenations, over 18% add digits such as a birth year): plain 40, name plus number 30 (a 1972-2006 year, a two-digit year or a number up to 99), separator plus number 8, funny 17 (a prefix such as `its`/`dr`, a suffix such as `_dev`/`_ftw`, or `42`/`1337`/`404`), light leetspeak 5. The weights are constants in the spec, not tuned; the study gave no figure for separators or the funny forms, so those weights are ours.
+- **Deterministic.** The RNG is seeded with `sha256(origin:origin_uid)`, so a record always gets the same candidates; a taken name falls through to the next candidate and finally to the old `-<origin>` and numeric suffixes.
+- **Existing users are never renamed.** The name is chosen on insert only; under `humanize` an update keeps the stored username. Users created before this change keep their names. Alias clones (a name that is an identity in `sync_identities`) are never decorated, because the rewritten mentions must match them.
+
 ## Gaps
 
-- **No export endpoint.** Sync is inbound only. The scheduled sender is `CommunityService` (`services/community/`), which keeps a per-community watermark and pushes users, posts, comments in that order.
-- **Votes, reactions, follows, attachments, polls and project links are not synced.** Counters will not match the origin.
-- **A comment's post and parent must be sent in an earlier batch or earlier in the same batch.** There is no deferred resolution queue; the result reason tells the sender what to send first.
-- **Privacy disclosure.** Synced profile data is disclosed in `/docs/privacy.html` (data table row "Imported community content").
-- A user's own deletion on this side is final, but nothing tells the origin; the next sync of that user is `rejected` / `purged_locally` forever.
+- Votes, reactions and poll votes are import only: there is no export of who voted, so a two-way mirror of engagement is not possible.
+- A user's own deletion on this side is final and the origin learns of it only through `/sync/deletions`; a sender that ignores that endpoint sees `rejected` / `purged_locally` forever.
+- Privacy disclosure: synced profile data, engagement and attachments are disclosed in `/docs/privacy.html` (data table row "Imported community content").
 
 ## Selection, identity aliasing and translation (community runs only)
 

devplacepy/services/sync/batch.py

@@ -5,25 +5,28 @@ from __future__ import annotations
 from collections import Counter
 
 from devplacepy.schemas.sync import SyncBatchOut, SyncRecordResultOut
+from devplacepy.services.sync.pending import PendingQueue
 from devplacepy.services.sync.processors import PROCESSORS
 from devplacepy.services.sync.records import SyncBatchForm
 from devplacepy.services.sync.spec import STATUSES
 
 
 class BatchRunner:
-    def __init__(self, kind: str, *, request=None):
+    def __init__(self, kind: str, *, request=None, humanize: bool = False):
         self.kind = kind
         self.request = request
+        self.humanize = humanize
 
     def run(self, data: SyncBatchForm) -> SyncBatchOut:
         processor = PROCESSORS[self.kind](
-            data.origin, dry_run=data.dry_run, request=self.request
+            data.origin, dry_run=data.dry_run, request=self.request, humanize=self.humanize
         )
         results = [
             SyncRecordResultOut(**vars(processor.process(index, raw)))
             for index, raw in enumerate(data.records)
         ]
         tally = Counter(result.status for result in results)
+        released = {} if data.dry_run else dict(PendingQueue().release(data.origin))
         return SyncBatchOut(
             origin=data.origin,
             kind=self.kind,
@@ -31,4 +34,5 @@ class BatchRunner:
             total=len(results),
             counts={status: tally.get(status, 0) for status in STATUSES},
             results=results,
+            released=released,
         )

devplacepy/services/sync/deletions.py

@@ -0,0 +1,73 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from devplacepy.database.sync import acknowledge_links, deletion_rows
+from devplacepy.services.sync.spec import KINDS, LIMITS, SYNC_ACTOR
+
+CURSOR_SEPARATOR = ":"
+BY_PURGED, BY_SYSTEM, BY_MEMBER = "purged", "system", "member"
+
+
+class DeletionCursorError(ValueError):
+    pass
+
+
+@dataclass
+class DeletionPage:
+    items: list[dict] = field(default_factory=list)
+    next_cursor: str = ""
+    has_more: bool = False
+
+
+def parse_cursor(raw: str) -> tuple[int, int]:
+    if not raw:
+        return 0, 0
+    try:
+        kind_index, after = (int(part) for part in raw.split(CURSOR_SEPARATOR))
+    except ValueError:
+        raise DeletionCursorError("the cursor is not valid")
+    if not 0 <= kind_index < len(KINDS) or after < 0:
+        raise DeletionCursorError("the cursor is not valid")
+    return kind_index, after
+
+
+def classify(row: dict) -> str:
+    if row["local_uid"] is None:
+        return BY_PURGED
+    return BY_SYSTEM if (row.get("deleted_by") or "") in ("system", SYNC_ACTOR) else BY_MEMBER
+
+
+def deletions_page(origin: str, cursor: str = "", limit: int | None = None) -> DeletionPage:
+    size = max(1, min(int(limit or LIMITS.deletions_page_size), LIMITS.deletions_page_size))
+    kind_index, after = parse_cursor(cursor)
+    page = DeletionPage()
+    for index in range(kind_index, len(KINDS)):
+        kind = KINDS[index]
+        room = size + 1 - len(page.items)
+        rows = deletion_rows(origin, kind, kind, after if index == kind_index else 0, room)
+        for row in rows:
+            page.items.append(
+                {
+                    "kind": kind,
+                    "origin_uid": row["origin_uid"],
+                    "by": classify(row),
+                    "deleted_at": row.get("deleted_at") or "",
+                    "_cursor": f"{index}{CURSOR_SEPARATOR}{row['link_id']}",
+                }
+            )
+        if len(page.items) > size:
+            break
+    page.has_more = len(page.items) > size
+    page.items = page.items[:size]
+    if page.has_more and page.items:
+        page.next_cursor = page.items[-1]["_cursor"]
+    for item in page.items:
+        item.pop("_cursor", None)
+    return page
+
+
+def acknowledge(origin: str, items: list[dict]) -> int:
+    return acknowledge_links(origin, [(item["kind"], item["origin_uid"]) for item in items])

devplacepy/services/sync/exporter.py

@@ -0,0 +1,287 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import base64
+import json
+from dataclasses import dataclass, field
+
+from devplacepy.database import db
+from devplacepy.database.core import _in_clause
+from devplacepy.database.sync import EXPORT_CHANGED
+from devplacepy.services.sync.normalize import instant_iso
+from devplacepy.services.sync.spec import (
+    EXPORT_KINDS,
+    KIND_ATTACHMENTS,
+    KIND_COMMENTS,
+    KIND_FOLLOWS,
+    KIND_POLLS,
+    KIND_POSTS,
+    KIND_PROJECTS,
+    KIND_USERS,
+    LIMITS,
+)
+
+ATTACHMENT_URL = "/static/uploads/attachments/{directory}/{stored_name}"
+
+
+class ExportError(ValueError):
+    pass
+
+
+@dataclass
+class ExportPage:
+    kind: str
+    records: list[dict] = field(default_factory=list)
+    next_cursor: str = ""
+    has_more: bool = False
+
+
+def encode_cursor(changed: str, uid: str) -> str:
+    return base64.urlsafe_b64encode(json.dumps([changed, uid]).encode()).decode()
+
+
+def decode_cursor(raw: str) -> tuple[str, str]:
+    if not raw:
+        return "", ""
+    try:
+        changed, uid = json.loads(base64.urlsafe_b64decode(raw.encode()))
+    except (ValueError, TypeError):
+        raise ExportError("the cursor is not valid")
+    if not isinstance(changed, str) or not isinstance(uid, str):
+        raise ExportError("the cursor is not valid")
+    return changed, uid
+
+
+def iso_or_empty(value) -> str:
+    return instant_iso(value) if value else ""
+
+
+class Exporter:
+    kind = ""
+    table = ""
+    extra_where = ""
+
+    def changed(self) -> str:
+        return EXPORT_CHANGED[self.table]
+
+    def prepare(self, rows: list[dict]) -> dict:
+        return {}
+
+    def record(self, row: dict, context: dict) -> dict:
+        raise NotImplementedError
+
+    def deleted(self, row: dict) -> bool:
+        return bool(row.get("deleted_at"))
+
+    def page(self, since: str, cursor: str, limit: int) -> ExportPage:
+        if self.table not in db.tables:
+            return ExportPage(self.kind)
+        changed = self.changed()
+        after_changed, after_uid = decode_cursor(cursor)
+        clauses = [
+            "NOT EXISTS (SELECT 1 FROM sync_links l WHERE l.kind = :kind AND l.local_uid = t.uid)",
+        ]
+        params: dict = {"kind": self.kind, "n": limit + 1}
+        if self.extra_where:
+            clauses.append(self.extra_where)
+        if since:
+            clauses.append(f"{changed} > :since")
+            params["since"] = since
+        if cursor:
+            clauses.append(f"({changed} > :after_changed OR ({changed} = :after_changed AND uid > :after_uid))")
+            params["after_changed"] = after_changed
+            params["after_uid"] = after_uid
+        rows = [
+            dict(row)
+            for row in db.query(
+                f"SELECT *, {changed} AS _changed FROM {self.table} t WHERE {' AND '.join(clauses)} "
+                f"ORDER BY {changed}, uid LIMIT :n",
+                **params,
+            )
+        ]
+        has_more = len(rows) > limit
+        rows = rows[:limit]
+        context = self.prepare(rows)
+        records = [self.record(row, context) for row in rows]
+        next_cursor = encode_cursor(str(rows[-1]["_changed"]), rows[-1]["uid"]) if rows and has_more else ""
+        return ExportPage(self.kind, records, next_cursor, has_more)
+
+
+class UserExporter(Exporter):
+    kind = KIND_USERS
+    table = "users"
+
+    def deleted(self, row: dict) -> bool:
+        return not (row.get("is_active") is None or row.get("is_active")) or bool(row.get("deletion_requested_at"))
+
+    def record(self, row: dict, context: dict) -> dict:
+        return {
+            "origin_uid": row["uid"],
+            "username": row["username"],
+            "bio": row.get("bio") or "",
+            "location": row.get("location") or "",
+            "git_link": row.get("git_link") or "",
+            "website": row.get("website") or "",
+            "avatar_seed": row.get("avatar_seed"),
+            "created_at": iso_or_empty(row.get("created_at")),
+            "deleted": self.deleted(row),
+        }
+
+
+class ProjectExporter(Exporter):
+    kind = KIND_PROJECTS
+    table = "projects"
+    extra_where = "COALESCE(is_private, 0) = 0"
+
+    def record(self, row: dict, context: dict) -> dict:
+        return {
+            "origin_uid": row["uid"],
+            "author_uid": row["user_uid"],
+            "title": row.get("title") or "",
+            "description": row.get("description") or "",
+            "project_type": row.get("project_type") or "software",
+            "status": row.get("status") or "",
+            "platforms": row.get("platforms") or "",
+            "website_url": row.get("website_url") or "",
+            "repo_url": row.get("repo_url") or "",
+            "created_at": iso_or_empty(row.get("created_at")),
+            "updated_at": iso_or_empty(row.get("updated_at")),
+            "deleted": self.deleted(row),
+        }
+
+
+class PostExporter(Exporter):
+    kind = KIND_POSTS
+    table = "posts"
+
+    def prepare(self, rows: list[dict]) -> dict:
+        project_uids = [row["project_uid"] for row in rows if row.get("project_uid")]
+        public: set[str] = set()
+        if project_uids and "projects" in db.tables:
+            placeholders, params = _in_clause(project_uids)
+            found = db.query(
+                f"SELECT uid FROM projects WHERE uid IN ({placeholders}) AND COALESCE(is_private, 0) = 0",
+                **params,
+            )
+            public = {row["uid"] for row in found}
+        return {"public_projects": public}
+
+    def record(self, row: dict, context: dict) -> dict:
+        project = row.get("project_uid")
+        return {
+            "origin_uid": row["uid"],
+            "author_uid": row["user_uid"],
+            "title": row.get("title") or "",
+            "content": row.get("content") or "",
+            "topic": row.get("topic") or "random",
+            "project_uid": project if project in context["public_projects"] else None,
+            "created_at": iso_or_empty(row.get("created_at")),
+            "updated_at": iso_or_empty(row.get("updated_at")),
+            "deleted": self.deleted(row),
+        }
+
+
+class CommentExporter(Exporter):
+    kind = KIND_COMMENTS
+    table = "comments"
+    extra_where = "target_type = 'post'"
+
+    def record(self, row: dict, context: dict) -> dict:
+        return {
+            "origin_uid": row["uid"],
+            "author_uid": row["user_uid"],
+            "post_uid": row["target_uid"],
+            "parent_uid": row.get("parent_uid"),
+            "content": row.get("content") or "",
+            "created_at": iso_or_empty(row.get("created_at")),
+            "updated_at": iso_or_empty(row.get("updated_at")),
+            "deleted": self.deleted(row),
+        }
+
+
+class PollExporter(Exporter):
+    kind = KIND_POLLS
+    table = "polls"
+
+    def prepare(self, rows: list[dict]) -> dict:
+        if not rows or "poll_options" not in db.tables:
+            return {"options": {}}
+        placeholders, params = _in_clause([row["uid"] for row in rows])
+        options: dict[str, list[dict]] = {}
+        for option in db.query(
+            f"SELECT uid, poll_uid, label, position FROM poll_options WHERE poll_uid IN ({placeholders}) "
+            "AND deleted_at IS NULL ORDER BY position, id",
+            **params,
+        ):
+            options.setdefault(option["poll_uid"], []).append(
+                {"origin_uid": option["uid"], "label": option["label"]}
+            )
+        return {"options": options}
+
+    def record(self, row: dict, context: dict) -> dict:
+        return {
+            "origin_uid": row["uid"],
+            "author_uid": row["user_uid"],
+            "post_uid": row["post_uid"],
+            "question": row.get("question") or "",
+            "options": context["options"].get(row["uid"], []),
+            "created_at": iso_or_empty(row.get("created_at")),
+            "deleted": self.deleted(row),
+        }
+
+
+class FollowExporter(Exporter):
+    kind = KIND_FOLLOWS
+    table = "follows"
+
+    def record(self, row: dict, context: dict) -> dict:
+        return {
+            "origin_uid": row["uid"],
+            "follower_uid": row["follower_uid"],
+            "following_uid": row["following_uid"],
+            "created_at": iso_or_empty(row.get("created_at")),
+            "deleted": self.deleted(row),
+        }
+
+
+class AttachmentExporter(Exporter):
+    kind = KIND_ATTACHMENTS
+    table = "attachments"
+    extra_where = "target_type IN ('post', 'comment', 'project') AND COALESCE(target_uid, '') != ''"
+
+    def record(self, row: dict, context: dict) -> dict:
+        return {
+            "origin_uid": row["uid"],
+            "author_uid": row["user_uid"],
+            "target_kind": row["target_type"],
+            "target_uid": row["target_uid"],
+            "filename": row.get("original_filename") or "",
+            "mime_type": row.get("mime_type") or "",
+            "file_size": int(row.get("file_size") or 0),
+            "url": ATTACHMENT_URL.format(directory=row.get("directory") or "", stored_name=row.get("stored_name") or ""),
+            "created_at": iso_or_empty(row.get("created_at")),
+            "deleted": self.deleted(row),
+        }
+
+
+EXPORTERS: dict[str, Exporter] = {
+    item.kind: item
+    for item in (
+        UserExporter(),
+        ProjectExporter(),
+        PostExporter(),
+        CommentExporter(),
+        PollExporter(),
+        FollowExporter(),
+        AttachmentExporter(),
+    )
+}
+
+
+def export_page(kind: str, since: str = "", cursor: str = "", limit: int | None = None) -> ExportPage:
+    if kind not in EXPORT_KINDS or kind not in EXPORTERS:
+        raise ExportError(f"kind {kind!r} cannot be exported")
+    size = max(1, min(int(limit or LIMITS.export_page_size), LIMITS.export_page_size))
+    return EXPORTERS[kind].page(since, cursor, size)
+

devplacepy/services/sync/handles.py

@@ -0,0 +1,109 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import hashlib
+import random
+import re
+from collections.abc import Iterator
+
+from devplacepy.services.sync.normalize import username_candidates
+from devplacepy.services.sync.spec import HANDLES
+
+TIMESTAMP = re.compile(r"\d{4}-\d{2}-\d{2}[0-9_:\-]*")
+TOKEN_SPLIT = re.compile(r"[_\s]+")
+HANDLE_STRIP = re.compile(r"[^a-z0-9_-]+")
+HANDLE_PATTERN = re.compile(r"^[a-z0-9_-]+$")
+PATTERNS = ("plain", "digits", "separated", "funny", "leet")
+PATTERN_WEIGHTS = (
+    HANDLES.plain_weight,
+    HANDLES.digits_weight,
+    HANDLES.separated_weight,
+    HANDLES.funny_weight,
+    HANDLES.leet_weight,
+)
+NUMBER_KINDS = ("year", "short", "two_digit")
+NUMBER_WEIGHTS = (HANDLES.year_weight, HANDLES.short_number_weight, HANDLES.two_digit_weight)
+
+
+class HandleBuilder:
+    def __init__(self, origin: str, origin_uid: str):
+        self.origin = origin
+        seed = hashlib.sha256(f"{origin}:{origin_uid}".encode()).digest()
+        self.rng = random.Random(seed)
+
+    @staticmethod
+    def person(raw: str) -> str:
+        tokens = [token for token in TOKEN_SPLIT.split(TIMESTAMP.sub(" ", str(raw or ""))) if token]
+        marks = [index for index, token in enumerate(tokens) if token.lower() in HANDLES.relation_words]
+        for index in reversed(marks):
+            if index + 1 < len(tokens):
+                tokens = tokens[index + 1 : index + 2]
+                break
+        else:
+            tokens = [token for index, token in enumerate(tokens) if index not in marks] or tokens
+        if len(tokens) > 1 and tokens[-1].lower() in HANDLES.top_level_labels:
+            labels = [token for token in tokens[:-1] if token.lower() not in HANDLES.ignored_labels]
+            tokens = labels[-1:] or tokens[:-1]
+        base = HANDLE_STRIP.sub("", "_".join(tokens).lower()).strip("_-")
+        return re.sub(r"[_-]{2,}", "_", base)
+
+    def base(self, raw: str) -> str:
+        base = self.person(raw)[: HANDLES.max_base_length].rstrip("_-")
+        if len(base) < HANDLES.min_length:
+            base = f"{base}{self.number()}"
+        return base
+
+    def number(self) -> str:
+        kind = self.rng.choices(NUMBER_KINDS, NUMBER_WEIGHTS)[0]
+        if kind == "year":
+            return str(self.rng.randint(HANDLES.year_first, HANDLES.year_last))
+        if kind == "two_digit":
+            return f"{self.rng.randint(HANDLES.year_first, HANDLES.year_last) % 100:02d}"
+        return str(self.rng.randint(1, HANDLES.short_number_max))
+
+    def leet(self, base: str) -> str | None:
+        options = [(plain, swap) for plain, swap in HANDLES.leet_map if plain in base]
+        if not options:
+            return None
+        plain, swap = self.rng.choice(options)
+        return base.replace(plain, swap, self.rng.choice((1, 2)))
+
+    def fit(self, head: str, stem: str, tail: str) -> str:
+        room = HANDLES.max_length - len(head) - len(tail)
+        return f"{head}{stem[: max(room, 1)].rstrip('_-')}{tail}"
+
+    def render(self, base: str, pattern: str) -> str:
+        if pattern == "digits":
+            return self.fit("", base, self.number())
+        if pattern == "separated":
+            return self.fit("", base, f"{self.rng.choice('_-')}{self.number()}")
+        if pattern == "funny":
+            style = self.rng.choice(("prefix", "suffix", "number", "separated"))
+            if style == "prefix":
+                return self.fit(self.rng.choice(HANDLES.prefixes), base, "")
+            if style == "suffix":
+                return self.fit("", base, self.rng.choice(HANDLES.suffixes))
+            number = self.rng.choice(HANDLES.funny_numbers)
+            return self.fit("", base, number if style == "number" else f"_{number}")
+        if pattern == "leet":
+            swapped = self.leet(base)
+            return self.fit("", swapped, "") if swapped else self.render(base, "digits")
+        return base
+
+    def candidates(self, raw: str) -> Iterator[str]:
+        base = self.base(raw)
+        seen: set[str] = set()
+        picks = [self.rng.choices(PATTERNS, PATTERN_WEIGHTS)[0]]
+        picks += [self.rng.choice(PATTERNS[1:]) for _ in range(HANDLES.rerolls)]
+        for pattern in picks:
+            candidate = self.render(base, pattern)
+            if candidate in seen or not self.valid(candidate):
+                continue
+            seen.add(candidate)
+            yield candidate
+        yield from username_candidates(base if self.valid(base) else f"{base}{self.number()}", self.origin)
+
+    @staticmethod
+    def valid(candidate: str) -> bool:
+        return HANDLES.min_length <= len(candidate) <= HANDLES.max_length and bool(HANDLE_PATTERN.match(candidate))

devplacepy/services/sync/healing.py

@@ -0,0 +1,68 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass, field
+
+from devplacepy.database import get_table
+from devplacepy.database.sync import (
+    get_link,
+    identity_map,
+    identity_name_taken,
+    linked_rows_containing,
+    rename_identity,
+)
+from devplacepy.services.sync.rewriter import MentionRewriter
+from devplacepy.services.sync.spec import ALIAS, KIND_COMMENTS, KIND_POSTS, KIND_USERS
+
+logger = logging.getLogger("sync.alias")
+
+REWRITE_TARGETS = ((KIND_POSTS, "posts", ("title", "content")), (KIND_COMMENTS, "comments", ("content",)))
+
+
+@dataclass
+class HealReport:
+    renamed: dict[str, str] = field(default_factory=dict)
+    conflicts: list[str] = field(default_factory=list)
+    rewritten: int = 0
+
+
+class AliasHealer:
+    def __init__(self, origin: str):
+        self.origin = origin
+
+    def heal(self) -> HealReport:
+        report = HealReport()
+        for source_key, row in identity_map(self.origin).items():
+            link = get_link(self.origin, KIND_USERS, source_key)
+            if link is None:
+                continue
+            clone = get_table("users").find_one(uid=link["local_uid"])
+            if clone is None or not clone.get("username") or clone["username"] == row["our_name"]:
+                continue
+            current = clone["username"]
+            if current.lower() != row["our_key"] and identity_name_taken(current.lower()):
+                report.conflicts.append(source_key)
+                logger.warning("alias %s of %s cannot follow the clone to %s: the name is another identity", row["our_name"], self.origin, current)
+                continue
+            if not rename_identity(row["uid"], current):
+                report.conflicts.append(source_key)
+                continue
+            report.renamed[row["our_name"]] = current
+            logger.info("alias of %s followed its clone from %s to %s", self.origin, row["our_name"], current)
+        for old, new in report.renamed.items():
+            report.rewritten += self.rewrite_mentions(old, new)
+        return report
+
+    def rewrite_mentions(self, old: str, new: str) -> int:
+        rewriter = MentionRewriter({old: new})
+        changed = 0
+        for kind, table, columns in REWRITE_TARGETS:
+            for row in linked_rows_containing(self.origin, kind, table, columns, old, ALIAS.heal_rewrites_per_run):
+                values = {column: rewriter.rewrite(row[column] or "").text for column in columns if row.get(column)}
+                values = {column: text for column, text in values.items() if text != row[column]}
+                if values:
+                    get_table(table).update({"uid": row["uid"], **values}, ["uid"])
+                    changed += 1
+        return changed

devplacepy/services/sync/options.py

@@ -25,6 +25,7 @@ class SyncOptions:
     min_comments: int = 0
     topics: tuple[str, ...] = ()
     jitter_percent: int = 0
+    collect_attachments: bool = False
 
     @classmethod
     def from_row(cls, row: dict | None) -> "SyncOptions":
@@ -53,7 +54,10 @@ class SyncOptions:
         jitter = int(values.get("jitter_percent") or 0)
         if not 0 <= jitter <= ALIAS.max_jitter_percent:
             raise ValueError(f"jitter_percent must be between 0 and {ALIAS.max_jitter_percent}")
-        return cls(bool(values.get("alias")), translate_to, sample, minimum, tuple(dict.fromkeys(topics)), jitter)
+        return cls(
+            bool(values.get("alias")), translate_to, sample, minimum, tuple(dict.fromkeys(topics)), jitter,
+            bool(values.get("collect_attachments")),
+        )
 
     def merged(self, **changes) -> "SyncOptions":
         values = self.to_dict() | {key: value for key, value in changes.items() if value is not None}

devplacepy/services/sync/pending.py

@@ -0,0 +1,51 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import json
+import logging
+from collections import Counter
+
+from devplacepy.database.sync import (
+    delete_pending,
+    expire_pending,
+    pending_counts,
+    ready_pending,
+)
+from devplacepy.services.sync.processors import PROCESSORS
+from devplacepy.services.sync.spec import LIMITS, STATUS_DEFERRED
+
+logger = logging.getLogger(__name__)
+
+PAGE = 50
+
+
+class PendingQueue:
+    def release(self, origin: str = "", cap: int | None = None) -> Counter:
+        budget = cap if cap is not None else LIMITS.pending_release_cap
+        released: Counter = Counter()
+        spent = 0
+        while spent < budget:
+            expire_pending(LIMITS.pending_max_attempts)
+            rows = ready_pending(min(PAGE, budget - spent), origin)
+            if not rows:
+                break
+            for row in rows:
+                delete_pending(row["uid"])
+                spent += 1
+                try:
+                    raw = json.loads(row["record"])
+                except ValueError:
+                    logger.warning("pending record %s is not valid JSON, dropped", row["uid"])
+                    continue
+                processor = PROCESSORS[row["kind"]](row["origin"], attempts=int(row.get("attempts") or 0) + 1)
+                status = processor.process(0, raw).status
+                if status != STATUS_DEFERRED:
+                    released[status] += 1
+        return released
+
+    def counts(self) -> dict[str, int]:
+        return pending_counts()
+
+    def sweep(self) -> int:
+        return expire_pending(LIMITS.pending_max_attempts)

devplacepy/services/sync/processors.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import json
 import logging
 import secrets
 import time
@@ -14,49 +15,80 @@ from devplacepy.database import get_table, refresh_snapshot
 from devplacepy.database.content import invalidate_user_recent_cache
 from devplacepy.database.pagination import clear_user_post_count
 from devplacepy.database.soft_delete import restore, soft_delete
+from devplacepy.database.ranking import clear_user_stars, get_target_owner_uid
 from devplacepy.database.sync import (
     advance_link,
     claim_link,
     get_link,
+    park_pending,
+    recompute_stars,
     release_link,
     username_taken,
 )
+from devplacepy.database.sync import identity_name_taken
 from devplacepy.services.moderation.screening import (
     ContentRefused,
     record as record_screening,
     refuse_if_blocked,
     screen_fields,
 )
+from devplacepy.services.sync.handles import HandleBuilder
 from devplacepy.services.sync.normalize import fingerprint, username_candidates
 from devplacepy.services.sync.records import (
+    AttachmentRecord,
     CommentRecord,
+    FollowRecord,
+    PollRecord,
+    PollVoteRecord,
     PostRecord,
+    ProjectRecord,
+    ReactionRecord,
     SyncRecord,
     UserRecord,
+    VoteRecord,
 )
 from devplacepy.services.sync.spec import (
+    DEFERRABLE_REASONS,
+    KIND_ATTACHMENTS,
     KIND_COMMENTS,
+    KIND_FOLLOWS,
+    KIND_POLL_OPTIONS,
+    KIND_POLL_VOTES,
+    KIND_POLLS,
     KIND_POSTS,
+    KIND_PROJECTS,
+    KIND_REACTIONS,
     KIND_USERS,
+    KIND_VOTES,
     LIMITS,
+    REASON_ACK_DELETED,
     REASON_CONTENT_REFUSED,
+    REASON_IMMUTABLE,
     REASON_INTERNAL,
     REASON_INVALID,
     REASON_LOCALLY_DELETED,
     REASON_OLDER_THAN_SYNCED,
     REASON_PARENT_MISMATCH,
     REASON_PURGED_LOCALLY,
+    REASON_QUEUE_FULL,
+    REASON_SELF_FOLLOW,
     REASON_TOMBSTONE_UNKNOWN,
     REASON_UNKNOWN_AUTHOR,
+    REASON_UNKNOWN_OPTION,
     REASON_UNKNOWN_PARENT,
+    REASON_UNKNOWN_POLL,
     REASON_UNKNOWN_POST,
+    REASON_UNKNOWN_PROJECT,
+    REASON_UNKNOWN_TARGET,
     STATUS_CREATED,
+    STATUS_DEFERRED,
     STATUS_FAILED,
     STATUS_REJECTED,
     STATUS_STALE,
     STATUS_UNCHANGED,
     STATUS_UPDATED,
     SYNC_ACTOR,
+    TARGET_KINDS,
 )
 from devplacepy.utils.accounts import account_row
 from devplacepy.utils.authcache import clear_user_cache
@@ -81,9 +113,10 @@ def unusable_password_hash() -> str:
 
 
 class Refusal(Exception):
-    def __init__(self, reason: str, message: str):
+    def __init__(self, reason: str, message: str, needs: tuple[str, str] | None = None):
         self.reason = reason
         self.message = message
+        self.needs = needs
         super().__init__(message)
 
 
@@ -96,6 +129,8 @@ class SyncResult:
     local_uid: str = ""
     errors: list[str] = field(default_factory=list)
     ignored: list[str] = field(default_factory=list)
+    needs_kind: str = ""
+    needs_uid: str = ""
 
 
 class RecordProcessor:
@@ -103,10 +138,12 @@ class RecordProcessor:
     table: str = ""
     model: type[SyncRecord] = SyncRecord
 
-    def __init__(self, origin: str, *, dry_run: bool = False, request=None):
+    def __init__(self, origin: str, *, dry_run: bool = False, request=None, attempts: int = 0, humanize: bool = False):
         self.origin = origin
+        self.humanize = humanize
         self.dry_run = dry_run
         self.request = request
+        self.attempts = attempts
 
     def process(self, index: int, raw: dict) -> SyncResult:
         origin_uid = str(raw.get("origin_uid", "")) if isinstance(raw, dict) else ""
@@ -125,13 +162,7 @@ class RecordProcessor:
         try:
             result = self.apply(index, record)
         except Refusal as refusal:
-            result = SyncResult(
-                index,
-                record.origin_uid,
-                STATUS_REJECTED,
-                refusal.reason,
-                errors=[refusal.message],
-            )
+            result = self.refused(index, record, raw, refusal)
         except ContentRefused as refused:
             result = SyncResult(
                 index,
@@ -150,6 +181,44 @@ class RecordProcessor:
         result.ignored = ignored
         return result
 
+    def refused(self, index: int, record: SyncRecord, raw: dict, refusal: Refusal) -> SyncResult:
+        needs_kind, needs_uid = refusal.needs or ("", "")
+        result = SyncResult(
+            index,
+            record.origin_uid,
+            STATUS_REJECTED,
+            refusal.reason,
+            errors=[refusal.message],
+            needs_kind=needs_kind,
+            needs_uid=needs_uid,
+        )
+        if self.dry_run or refusal.reason not in DEFERRABLE_REASONS or not refusal.needs:
+            return result
+        return self.defer(result, raw, refusal)
+
+    def defer(self, result: SyncResult, raw: dict, refusal: Refusal) -> SyncResult:
+        encoded = json.dumps(raw, default=str)
+        if len(encoded) > LIMITS.pending_max_record_bytes:
+            result.errors.append("record is too large to wait for its dependency")
+            return result
+        parked = park_pending(
+            self.origin,
+            self.kind,
+            result.origin_uid,
+            result.needs_kind,
+            result.needs_uid,
+            encoded,
+            attempts=self.attempts,
+            ttl_days=LIMITS.pending_ttl_days,
+            cap=LIMITS.pending_max_per_origin,
+        )
+        if not parked:
+            result.reason = refusal.reason
+            result.errors.append(f"{REASON_QUEUE_FULL}: the waiting list for this origin is full")
+            return result
+        result.status = STATUS_DEFERRED
+        return result
+
     def ignored_fields(self, raw) -> list[str]:
         if not isinstance(raw, dict):
             return []
@@ -164,16 +233,16 @@ class RecordProcessor:
         return messages
 
     def apply(self, index: int, record: SyncRecord) -> SyncResult:
+        link = get_link(self.origin, self.kind, record.origin_uid)
+        if link is None and record.deleted:
+            return SyncResult(
+                index, record.origin_uid, STATUS_UNCHANGED, REASON_TOMBSTONE_UNKNOWN
+            )
         refs = self.resolve(record)
         if not record.deleted:
             self.screen(record)
         digest = fingerprint(record.content_fields())
-        link = get_link(self.origin, self.kind, record.origin_uid)
         if link is None:
-            if record.deleted:
-                return SyncResult(
-                    index, record.origin_uid, STATUS_UNCHANGED, REASON_TOMBSTONE_UNKNOWN
-                )
             return self.create(index, record, refs, digest)
         return self.update(index, record, refs, digest, link)
 
@@ -231,6 +300,10 @@ class RecordProcessor:
     ) -> SyncResult:
         local_uid = link["local_uid"]
         row = self.find_local(link)
+        if row is None and link.get("ack_at"):
+            return SyncResult(
+                index, record.origin_uid, STATUS_UNCHANGED, REASON_ACK_DELETED, local_uid=local_uid
+            )
         if row is None:
             return SyncResult(
                 index,
@@ -253,7 +326,7 @@ class RecordProcessor:
                 index,
                 record.origin_uid,
                 STATUS_UNCHANGED,
-                REASON_LOCALLY_DELETED,
+                REASON_ACK_DELETED if link.get("ack_at") else REASON_LOCALLY_DELETED,
                 local_uid=local_uid,
             )
         if digest == link.get("fingerprint"):
@@ -287,12 +360,27 @@ class RecordProcessor:
     def after_write(self, local_uid: str, record: SyncRecord, refs: dict) -> None:
         return None
 
-    def author_uid(self, record: SyncRecord) -> str:
-        link = get_link(self.origin, KIND_USERS, record.author_uid)
+    def user_ref(self, origin_uid: str, role: str = "author") -> str:
+        link = get_link(self.origin, KIND_USERS, origin_uid)
         if link is None:
             raise Refusal(
                 REASON_UNKNOWN_AUTHOR,
-                f"author {record.author_uid} is not synced yet, send users first",
+                f"{role} {origin_uid} is not synced yet, send users first",
+                (KIND_USERS, origin_uid),
+            )
+        return link["local_uid"]
+
+    def author_uid(self, record: SyncRecord) -> str:
+        return self.user_ref(record.author_uid)
+
+    def target_ref(self, target_kind: str, target_uid: str) -> str:
+        kind = TARGET_KINDS[target_kind]
+        link = get_link(self.origin, kind, target_uid)
+        if link is None:
+            raise Refusal(
+                REASON_UNKNOWN_TARGET,
+                f"{target_kind} {target_uid} is not synced yet, send {kind} first",
+                (kind, target_uid),
             )
         return link["local_uid"]
 
@@ -334,11 +422,19 @@ class UserProcessor(RecordProcessor):
         )
 
     def free_username(self, record: UserRecord, current: str = "") -> str:
-        for candidate in username_candidates(record.username, self.origin):
+        for candidate in self.username_options(record, current):
             if candidate == current or not username_taken(candidate):
                 return candidate
         return current or record.username
 
+    def username_options(self, record: UserRecord, current: str):
+        if self.humanize and current:
+            yield current
+        elif self.humanize and not identity_name_taken(record.username.lower()):
+            yield from HandleBuilder(self.origin, record.origin_uid).candidates(record.username)
+        else:
+            yield from username_candidates(record.username, self.origin)
+
     def insert(self, local_uid: str, record: UserRecord, refs: dict) -> None:
         now = datetime.now(timezone.utc).isoformat()
         row = account_row(
@@ -386,7 +482,17 @@ class PostProcessor(RecordProcessor):
     model = PostRecord
 
     def resolve(self, record: PostRecord) -> dict:
-        return {"user_uid": self.author_uid(record)}
+        refs = {"user_uid": self.author_uid(record), "project_uid": None}
+        if record.project_uid:
+            link = get_link(self.origin, KIND_PROJECTS, record.project_uid)
+            if link is None:
+                raise Refusal(
+                    REASON_UNKNOWN_PROJECT,
+                    f"project {record.project_uid} is not synced yet, send projects first",
+                    (KIND_PROJECTS, record.project_uid),
+                )
+            refs["project_uid"] = link["local_uid"]
+        return refs
 
     def screen(self, record: PostRecord) -> None:
         refuse_if_blocked(
@@ -404,7 +510,7 @@ class PostProcessor(RecordProcessor):
                 "title": record.title or None,
                 "content": record.content,
                 "topic": record.topic,
-                "project_uid": None,
+                "project_uid": refs["project_uid"],
                 "image": None,
                 "stars": 0,
                 "created_at": record.created_at,
@@ -421,6 +527,7 @@ class PostProcessor(RecordProcessor):
                 "title": record.title or None,
                 "content": record.content,
                 "topic": record.topic,
+                "project_uid": refs["project_uid"],
                 "updated_at": record.updated_at,
             },
             ["uid"],
@@ -451,6 +558,7 @@ class CommentProcessor(RecordProcessor):
             raise Refusal(
                 REASON_UNKNOWN_POST,
                 f"post {record.post_uid} is not synced yet, send posts first",
+                (KIND_POSTS, record.post_uid),
             )
         post_uid = post_link["local_uid"]
         parent_uid = None
@@ -461,6 +569,7 @@ class CommentProcessor(RecordProcessor):
                     REASON_UNKNOWN_PARENT,
                     f"parent comment {record.parent_uid} is not synced yet, "
                     "send parents before replies",
+                    (KIND_COMMENTS, record.parent_uid),
                 )
             parent = get_table("comments").find_one(uid=parent_link["local_uid"])
             if parent is None or parent.get("target_uid") != post_uid:
@@ -509,8 +618,400 @@ class CommentProcessor(RecordProcessor):
             )
 
 
+
+
+class ProjectProcessor(RecordProcessor):
+    kind = KIND_PROJECTS
+    table = "projects"
+    model = ProjectRecord
+
+    def resolve(self, record: ProjectRecord) -> dict:
+        return {"user_uid": self.author_uid(record)}
+
+    def screen(self, record: ProjectRecord) -> None:
+        refuse_if_blocked(
+            screen_fields("projects", {"title": record.title, "description": record.description})
+        )
+
+    def insert(self, local_uid: str, record: ProjectRecord, refs: dict) -> None:
+        get_table("projects").insert(
+            {
+                "uid": local_uid,
+                "user_uid": refs["user_uid"],
+                "slug": make_combined_slug(record.title, local_uid),
+                "title": record.title,
+                "description": record.description,
+                "project_type": record.project_type,
+                "status": record.status,
+                "platforms": record.platforms,
+                "website_url": record.website_url,
+                "repo_url": record.repo_url,
+                "release_date": "",
+                "demo_date": "",
+                "cover_attachment_uid": "",
+                "logo_attachment_uid": "",
+                "is_private": 0,
+                "read_only": 0,
+                "stars": 0,
+                "created_at": record.created_at,
+                "updated_at": record.updated_at,
+                "deleted_at": None,
+                "deleted_by": None,
+            }
+        )
+
+    def write(self, row: dict, record: ProjectRecord, refs: dict) -> None:
+        get_table("projects").update(
+            {
+                "uid": row["uid"],
+                "title": record.title,
+                "description": record.description,
+                "project_type": record.project_type,
+                "status": record.status,
+                "platforms": record.platforms,
+                "website_url": record.website_url,
+                "repo_url": record.repo_url,
+                "updated_at": record.updated_at,
+            },
+            ["uid"],
+        )
+        self.apply_tombstone(row, record)
+
+    def after_write(self, local_uid: str, record: ProjectRecord, refs: dict) -> None:
+        from devplacepy.templating import clear_user_projects_cache
+
+        clear_user_projects_cache(refs["user_uid"])
+        invalidate_user_recent_cache("projects", refs["user_uid"])
+        if not record.deleted:
+            self.screen_and_report(
+                local_uid,
+                "project",
+                {"title": record.title, "description": record.description},
+                refs["user_uid"],
+            )
+
+
+class NaturalKeyProcessor(RecordProcessor):
+    def natural_key(self, record: SyncRecord, refs: dict) -> dict:
+        raise NotImplementedError
+
+    def create(self, index: int, record: SyncRecord, refs: dict, digest: str) -> SyncResult:
+        existing = get_table(self.table).find_one(**self.natural_key(record, refs))
+        if existing is not None and existing.get("deleted_at") and existing.get("deleted_by") != SYNC_ACTOR:
+            return SyncResult(
+                index, record.origin_uid, STATUS_UNCHANGED, REASON_LOCALLY_DELETED, local_uid=existing["uid"]
+            )
+        local_uid = existing["uid"] if existing is not None else generate_uid()
+        if self.dry_run:
+            return SyncResult(index, record.origin_uid, STATUS_CREATED, local_uid="")
+        self.prepare()
+        claimed = claim_link(
+            self.origin, self.kind, record.origin_uid, local_uid, digest, record.updated_at
+        )
+        if not claimed:
+            link = get_link(self.origin, self.kind, record.origin_uid)
+            return self.update(index, record, refs, digest, link)
+        try:
+            if existing is not None:
+                self.write(existing, record, refs)
+            else:
+                self.insert(local_uid, record, refs)
+        except Exception:
+            release_link(self.origin, self.kind, record.origin_uid, local_uid)
+            raise
+        self.after_write(local_uid, record, refs)
+        return SyncResult(index, record.origin_uid, STATUS_CREATED, local_uid=local_uid)
+
+
+class VoteProcessor(NaturalKeyProcessor):
+    kind = KIND_VOTES
+    table = "votes"
+    model = VoteRecord
+
+    def resolve(self, record: VoteRecord) -> dict:
+        return {
+            "user_uid": self.user_ref(record.voter_uid, "voter"),
+            "target_uid": self.target_ref(record.target_kind, record.target_uid),
+        }
+
+    def natural_key(self, record: VoteRecord, refs: dict) -> dict:
+        return {"user_uid": refs["user_uid"], "target_uid": refs["target_uid"], "target_type": record.target_kind}
+
+    def insert(self, local_uid: str, record: VoteRecord, refs: dict) -> None:
+        get_table("votes").insert(
+            {
+                "uid": local_uid,
+                "user_uid": refs["user_uid"],
+                "target_uid": refs["target_uid"],
+                "target_type": record.target_kind,
+                "value": record.value,
+                "created_at": record.created_at,
+                "deleted_at": None,
+                "deleted_by": None,
+            }
+        )
+
+    def write(self, row: dict, record: VoteRecord, refs: dict) -> None:
+        get_table("votes").update({"uid": row["uid"], "value": record.value}, ["uid"])
+        self.apply_tombstone(row, record)
+
+    def after_write(self, local_uid: str, record: VoteRecord, refs: dict) -> None:
+        recompute_stars(record.target_kind, refs["target_uid"])
+        owner = get_target_owner_uid(record.target_kind, refs["target_uid"])
+        if owner:
+            clear_user_stars(owner)
+
+
+class ReactionProcessor(NaturalKeyProcessor):
+    kind = KIND_REACTIONS
+    table = "reactions"
+    model = ReactionRecord
+
+    def resolve(self, record: ReactionRecord) -> dict:
+        return {
+            "user_uid": self.author_uid(record),
+            "target_uid": self.target_ref(record.target_kind, record.target_uid),
+        }
+
+    def natural_key(self, record: ReactionRecord, refs: dict) -> dict:
+        return {
+            "user_uid": refs["user_uid"],
+            "target_uid": refs["target_uid"],
+            "target_type": record.target_kind,
+            "emoji": record.emoji,
+        }
+
+    def insert(self, local_uid: str, record: ReactionRecord, refs: dict) -> None:
+        get_table("reactions").insert(
+            {
+                "uid": local_uid,
+                "user_uid": refs["user_uid"],
+                "target_uid": refs["target_uid"],
+                "target_type": record.target_kind,
+                "emoji": record.emoji,
+                "created_at": record.created_at,
+                "deleted_at": None,
+                "deleted_by": None,
+            }
+        )
+
+    def write(self, row: dict, record: ReactionRecord, refs: dict) -> None:
+        self.apply_tombstone(row, record)
+
+
+class FollowProcessor(NaturalKeyProcessor):
+    kind = KIND_FOLLOWS
+    table = "follows"
+    model = FollowRecord
+
+    def resolve(self, record: FollowRecord) -> dict:
+        follower = self.user_ref(record.follower_uid, "follower")
+        following = self.user_ref(record.following_uid, "followed user")
+        if follower == following:
+            raise Refusal(REASON_SELF_FOLLOW, "an account cannot follow itself")
+        return {"follower_uid": follower, "following_uid": following}
+
+    def natural_key(self, record: FollowRecord, refs: dict) -> dict:
+        return dict(refs)
+
+    def insert(self, local_uid: str, record: FollowRecord, refs: dict) -> None:
+        get_table("follows").insert(
+            {
+                "uid": local_uid,
+                "follower_uid": refs["follower_uid"],
+                "following_uid": refs["following_uid"],
+                "created_at": record.created_at,
+                "deleted_at": None,
+                "deleted_by": None,
+            }
+        )
+
+    def write(self, row: dict, record: FollowRecord, refs: dict) -> None:
+        self.apply_tombstone(row, record)
+
+
+class PollProcessor(RecordProcessor):
+    kind = KIND_POLLS
+    table = "polls"
+    model = PollRecord
+
+    def resolve(self, record: PollRecord) -> dict:
+        link = get_link(self.origin, KIND_POSTS, record.post_uid)
+        if link is None:
+            raise Refusal(
+                REASON_UNKNOWN_POST,
+                f"post {record.post_uid} is not synced yet, send posts first",
+                (KIND_POSTS, record.post_uid),
+            )
+        return {"user_uid": self.author_uid(record), "post_uid": link["local_uid"]}
+
+    def screen(self, record: PollRecord) -> None:
+        text = "\n".join([record.question, *(item.label for item in record.options)])
+        refuse_if_blocked(screen_fields("posts", {"content": text}))
+
+    @staticmethod
+    def option_key(record: PollRecord, option) -> str:
+        return f"{record.origin_uid}:{option.origin_uid}"
+
+    def insert(self, local_uid: str, record: PollRecord, refs: dict) -> None:
+        taken = get_table("polls").find_one(post_uid=refs["post_uid"], deleted_at=None)
+        if taken is not None:
+            raise Refusal(REASON_INVALID, "the post already has a poll")
+        get_table("polls").insert(
+            {
+                "uid": local_uid,
+                "post_uid": refs["post_uid"],
+                "user_uid": refs["user_uid"],
+                "question": record.question,
+                "created_at": record.created_at,
+                "deleted_at": None,
+                "deleted_by": None,
+            }
+        )
+        for position, option in enumerate(record.options):
+            self.add_option(local_uid, record, option, position)
+
+    def add_option(self, poll_uid: str, record: PollRecord, option, position: int) -> None:
+        option_uid = generate_uid()
+        if not claim_link(
+            self.origin, KIND_POLL_OPTIONS, self.option_key(record, option), option_uid,
+            fingerprint({"label": option.label}), record.updated_at,
+        ):
+            return
+        get_table("poll_options").insert(
+            {
+                "uid": option_uid,
+                "poll_uid": poll_uid,
+                "label": option.label,
+                "position": position,
+                "deleted_at": None,
+                "deleted_by": None,
+            }
+        )
+
+    def write(self, row: dict, record: PollRecord, refs: dict) -> None:
+        get_table("polls").update({"uid": row["uid"], "question": record.question}, ["uid"])
+        wanted = set()
+        options = get_table("poll_options")
+        for position, option in enumerate(record.options):
+            key = self.option_key(record, option)
+            wanted.add(key)
+            link = get_link(self.origin, KIND_POLL_OPTIONS, key)
+            if link is None:
+                self.add_option(row["uid"], record, option, position)
+                continue
+            options.update(
+                {"uid": link["local_uid"], "label": option.label, "position": position, "deleted_at": None, "deleted_by": None},
+                ["uid"],
+            )
+        prefix = f"{record.origin_uid}:"
+        for link in get_table("sync_links").find(origin=self.origin, kind=KIND_POLL_OPTIONS):
+            if link["origin_uid"].startswith(prefix) and link["origin_uid"] not in wanted:
+                soft_delete("poll_options", SYNC_ACTOR, uid=link["local_uid"])
+        self.apply_tombstone(row, record)
+
+    def apply_tombstone(self, row: dict, record: PollRecord) -> None:
+        if record.deleted and not row.get("deleted_at"):
+            stamp = datetime.now(timezone.utc).isoformat()
+            soft_delete("polls", SYNC_ACTOR, stamp=stamp, uid=row["uid"])
+            soft_delete("poll_options", SYNC_ACTOR, stamp=stamp, poll_uid=row["uid"])
+            soft_delete("poll_votes", SYNC_ACTOR, stamp=stamp, poll_uid=row["uid"])
+        elif not record.deleted and row.get("deleted_at"):
+            restore("polls", uid=row["uid"])
+            restore("poll_options", poll_uid=row["uid"])
+            restore("poll_votes", poll_uid=row["uid"])
+
+
+class PollVoteProcessor(NaturalKeyProcessor):
+    kind = KIND_POLL_VOTES
+    table = "poll_votes"
+    model = PollVoteRecord
+
+    def resolve(self, record: PollVoteRecord) -> dict:
+        voter = self.user_ref(record.voter_uid, "voter")
+        poll = get_link(self.origin, KIND_POLLS, record.poll_uid)
+        if poll is None:
+            raise Refusal(
+                REASON_UNKNOWN_POLL,
+                f"poll {record.poll_uid} is not synced yet, send polls first",
+                (KIND_POLLS, record.poll_uid),
+            )
+        key = f"{record.poll_uid}:{record.option_uid}"
+        option = get_link(self.origin, KIND_POLL_OPTIONS, key)
+        if option is None:
+            raise Refusal(
+                REASON_UNKNOWN_OPTION,
+                f"option {record.option_uid} of poll {record.poll_uid} is not synced",
+                (KIND_POLLS, record.poll_uid),
+            )
+        found = get_table("poll_options").find_one(uid=option["local_uid"])
+        if found is None or found["poll_uid"] != poll["local_uid"]:
+            raise Refusal(REASON_PARENT_MISMATCH, "the option does not belong to the poll")
+        return {"user_uid": voter, "poll_uid": poll["local_uid"], "option_uid": option["local_uid"]}
+
+    def natural_key(self, record: PollVoteRecord, refs: dict) -> dict:
+        return {"poll_uid": refs["poll_uid"], "user_uid": refs["user_uid"]}
+
+    def insert(self, local_uid: str, record: PollVoteRecord, refs: dict) -> None:
+        get_table("poll_votes").insert(
+            {
+                "uid": local_uid,
+                "poll_uid": refs["poll_uid"],
+                "option_uid": refs["option_uid"],
+                "user_uid": refs["user_uid"],
+                "created_at": record.created_at,
+                "deleted_at": None,
+                "deleted_by": None,
+            }
+        )
+
+    def write(self, row: dict, record: PollVoteRecord, refs: dict) -> None:
+        get_table("poll_votes").update({"uid": row["uid"], "option_uid": refs["option_uid"]}, ["uid"])
+        self.apply_tombstone(row, record)
+
+
+class AttachmentProcessor(RecordProcessor):
+    kind = KIND_ATTACHMENTS
+    table = "attachments"
+    model = AttachmentRecord
+
+    def resolve(self, record: AttachmentRecord) -> dict:
+        return {
+            "user_uid": self.author_uid(record),
+            "target_uid": self.target_ref(record.target_kind, record.target_uid),
+        }
+
+    def screen(self, record: AttachmentRecord) -> None:
+        from devplacepy.attachments import is_extension_allowed
+
+        extension = "." + record.filename.rsplit(".", 1)[-1].lower() if "." in record.filename else ""
+        if not is_extension_allowed(extension):
+            raise Refusal(REASON_INVALID, f"files of type {extension or 'without extension'} are not accepted")
+
+    def insert(self, local_uid: str, record: AttachmentRecord, refs: dict) -> None:
+        from devplacepy.attachments import link_attachments, store_attachment
+
+        stored = store_attachment(record.content_bytes(), record.filename, refs["user_uid"], uid=local_uid)
+        if stored is None:
+            raise Refusal(REASON_INVALID, "the file is not an allowed type or exceeds the upload limit")
+        link_attachments([local_uid], record.target_kind, refs["target_uid"])
+        get_table("attachments").update({"uid": local_uid, "created_at": record.created_at}, ["uid"])
+
+    def write(self, row: dict, record: AttachmentRecord, refs: dict) -> None:
+        if record.deleted == bool(row.get("deleted_at")):
+            raise Refusal(REASON_IMMUTABLE, "an attachment cannot change, send a new origin_uid")
+        self.apply_tombstone(row, record)
+
+
 PROCESSORS: dict[str, type[RecordProcessor]] = {
     KIND_USERS: UserProcessor,
+    KIND_PROJECTS: ProjectProcessor,
     KIND_POSTS: PostProcessor,
     KIND_COMMENTS: CommentProcessor,
+    KIND_POLLS: PollProcessor,
+    KIND_VOTES: VoteProcessor,
+    KIND_REACTIONS: ReactionProcessor,
+    KIND_FOLLOWS: FollowProcessor,
+    KIND_POLL_VOTES: PollVoteProcessor,
+    KIND_ATTACHMENTS: AttachmentProcessor,
 }

devplacepy/services/sync/records.py

@@ -2,13 +2,17 @@
 
 from __future__ import annotations
 
+import base64
+import binascii
+import hashlib
 import json
-from typing import ClassVar, Optional
+from typing import ClassVar, Literal, Optional
 
 from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
 
 from devplacepy.constants import TOPICS
 from devplacepy.models import normalize_website_url
+from devplacepy.rendering import is_single_emoji
 from devplacepy.services.sync.normalize import (
     clean_text,
     instant_iso,
@@ -17,12 +21,34 @@ from devplacepy.services.sync.normalize import (
     normalize_username,
 )
 from devplacepy.services.sync.spec import (
+    KIND_ATTACHMENTS,
     KIND_COMMENTS,
+    KIND_FOLLOWS,
+    KIND_POLL_VOTES,
+    KIND_POLLS,
     KIND_POSTS,
+    KIND_PROJECTS,
+    KIND_REACTIONS,
     KIND_USERS,
+    KIND_VOTES,
+    KINDS,
     LIMITS,
 )
 
+REFERENCE_FIELDS = (
+    "origin_uid",
+    "author_uid",
+    "post_uid",
+    "voter_uid",
+    "follower_uid",
+    "following_uid",
+    "target_uid",
+    "poll_uid",
+    "option_uid",
+)
+OPTIONAL_REFERENCE_FIELDS = ("parent_uid", "project_uid")
+TargetKind = Literal["post", "comment", "project"]
+
 
 class SyncBatchForm(BaseModel):
     origin: str
@@ -51,9 +77,46 @@ class SyncBatchForm(BaseModel):
     def not_empty(cls, value):
         if not value:
             raise ValueError("records must contain at least one record")
+        if len(json.dumps(value, default=str)) > LIMITS.max_batch_bytes:
+            raise ValueError(f"records exceed {LIMITS.max_batch_bytes} bytes")
         return value
 
 
+class SyncAckForm(BaseModel):
+    origin: str
+    items: list[dict] = Field(max_length=LIMITS.deletions_page_size)
+
+    @field_validator("origin")
+    @classmethod
+    def valid_origin(cls, value):
+        return normalize_origin(value)
+
+    @field_validator("items", mode="before")
+    @classmethod
+    def decode_items(cls, value):
+        if isinstance(value, list) and value and all(isinstance(item, str) for item in value):
+            value = value[0] if len(value) == 1 else "[" + ",".join(value) + "]"
+        if isinstance(value, str):
+            try:
+                return json.loads(value)
+            except ValueError:
+                raise ValueError("items must be a JSON array")
+        return value
+
+    @field_validator("items")
+    @classmethod
+    def valid_items(cls, value):
+        if not value:
+            raise ValueError("items must contain at least one entry")
+        cleaned = []
+        for item in value:
+            kind = str(item.get("kind", ""))
+            if kind not in KINDS:
+                raise ValueError(f"unknown kind {kind!r}")
+            cleaned.append({"kind": kind, "origin_uid": normalize_origin_uid(item.get("origin_uid"))})
+        return cleaned
+
+
 class SyncRecord(BaseModel):
     model_config = ConfigDict(extra="ignore")
 
@@ -65,14 +128,12 @@ class SyncRecord(BaseModel):
     updated_at: Optional[str] = None
     deleted: bool = False
 
-    @field_validator(
-        "origin_uid", "author_uid", "post_uid", mode="before", check_fields=False
-    )
+    @field_validator(*REFERENCE_FIELDS, mode="before", check_fields=False)
     @classmethod
     def valid_reference(cls, value):
         return normalize_origin_uid(value)
 
-    @field_validator("parent_uid", mode="before", check_fields=False)
+    @field_validator(*OPTIONAL_REFERENCE_FIELDS, mode="before", check_fields=False)
     @classmethod
     def valid_optional_reference(cls, value):
         if value is None or value == "":
@@ -154,14 +215,82 @@ class UserRecord(SyncRecord):
         return text
 
 
+class ProjectRecord(SyncRecord):
+    kind: ClassVar[str] = KIND_PROJECTS
+    reference_fields: ClassVar[tuple[str, ...]] = ("author_uid",)
+
+    author_uid: str
+    title: str
+    description: str
+    project_type: Literal["game", "game_asset", "software", "mobile_app", "website"] = "software"
+    status: str = "In Development"
+    platforms: str = ""
+    website_url: str = ""
+    repo_url: str = ""
+
+    @field_validator("title", mode="before")
+    @classmethod
+    def valid_title(cls, value):
+        text = clean_text(value, multiline=False)
+        if not text or len(text) > LIMITS.max_project_title_length:
+            raise ValueError(f"title must be 1-{LIMITS.max_project_title_length} characters")
+        return text
+
+    @field_validator("description", mode="before")
+    @classmethod
+    def valid_description(cls, value):
+        text = clean_text(value, multiline=True)
+        if not text or len(text) > LIMITS.max_project_description_length:
+            raise ValueError(
+                f"description must be 1-{LIMITS.max_project_description_length} characters"
+            )
+        return text
+
+    @field_validator("status", "platforms", mode="before")
+    @classmethod
+    def valid_short_text(cls, value):
+        return clean_text(value, multiline=False)[: LIMITS.max_project_field_length]
+
+    @field_validator("website_url", "repo_url", mode="before")
+    @classmethod
+    def valid_link(cls, value):
+        text = clean_text(value, multiline=False)
+        if not text or len(text) > LIMITS.max_url_length:
+            return ""
+        try:
+            return normalize_website_url(text)
+        except ValueError:
+            return ""
+
+    @field_validator("project_type", mode="before")
+    @classmethod
+    def known_type(cls, value):
+        text = clean_text(value, multiline=False).lower()
+        return text if text in ("game", "game_asset", "software", "mobile_app", "website") else "software"
+
+
 class PostRecord(SyncRecord):
     kind: ClassVar[str] = KIND_POSTS
-    reference_fields: ClassVar[tuple[str, ...]] = ("author_uid",)
+    reference_fields: ClassVar[tuple[str, ...]] = ("author_uid", "project_uid")
 
     author_uid: str
     title: str = ""
     content: str
     topic: str = "random"
+    project_uid: Optional[str] = None
+
+    @field_validator("project_uid", mode="before")
+    @classmethod
+    def valid_project(cls, value):
+        if value is None or value == "":
+            return None
+        return normalize_origin_uid(value)
+
+    def content_fields(self) -> dict:
+        fields = super().content_fields()
+        if fields.get("project_uid") is None:
+            fields.pop("project_uid", None)
+        return fields
 
     @field_validator("title", mode="before")
     @classmethod
@@ -208,8 +337,151 @@ class CommentRecord(SyncRecord):
         return text
 
 
+class PollOptionRecord(BaseModel):
+    model_config = ConfigDict(extra="ignore")
+
+    origin_uid: str
+    label: str
+
+    @field_validator("origin_uid", mode="before")
+    @classmethod
+    def valid_reference(cls, value):
+        return normalize_origin_uid(value)
+
+    @field_validator("label", mode="before")
+    @classmethod
+    def valid_label(cls, value):
+        text = clean_text(value, multiline=False)
+        if not text or len(text) > LIMITS.max_poll_option_length:
+            raise ValueError(f"option label must be 1-{LIMITS.max_poll_option_length} characters")
+        return text
+
+
+class PollRecord(SyncRecord):
+    kind: ClassVar[str] = KIND_POLLS
+    reference_fields: ClassVar[tuple[str, ...]] = ("author_uid", "post_uid")
+
+    author_uid: str
+    post_uid: str
+    question: str
+    options: list[PollOptionRecord]
+
+    @field_validator("question", mode="before")
+    @classmethod
+    def valid_question(cls, value):
+        text = clean_text(value, multiline=False)
+        if not text or len(text) > LIMITS.max_poll_question_length:
+            raise ValueError(f"question must be 1-{LIMITS.max_poll_question_length} characters")
+        return text
+
+    @model_validator(mode="after")
+    def valid_options(self):
+        if not LIMITS.min_poll_options <= len(self.options) <= LIMITS.max_poll_options:
+            raise ValueError(
+                f"a poll needs {LIMITS.min_poll_options}-{LIMITS.max_poll_options} options"
+            )
+        if len({item.origin_uid for item in self.options}) != len(self.options):
+            raise ValueError("option ids must be unique within a poll")
+        return self
+
+
+class VoteRecord(SyncRecord):
+    kind: ClassVar[str] = KIND_VOTES
+    reference_fields: ClassVar[tuple[str, ...]] = ("voter_uid", "target_uid")
+
+    voter_uid: str
+    target_kind: TargetKind
+    target_uid: str
+    value: Literal[1, -1]
+
+
+class ReactionRecord(SyncRecord):
+    kind: ClassVar[str] = KIND_REACTIONS
+    reference_fields: ClassVar[tuple[str, ...]] = ("author_uid", "target_uid")
+
+    author_uid: str
+    target_kind: TargetKind
+    target_uid: str
+    emoji: str
+
+    @field_validator("emoji", mode="before")
+    @classmethod
+    def valid_emoji(cls, value):
+        text = clean_text(value, multiline=False)
+        if not text or len(text) > LIMITS.max_emoji_length or not is_single_emoji(text):
+            raise ValueError("emoji must be a single emoji")
+        return text
+
+
+class FollowRecord(SyncRecord):
+    kind: ClassVar[str] = KIND_FOLLOWS
+    reference_fields: ClassVar[tuple[str, ...]] = ("follower_uid", "following_uid")
+
+    follower_uid: str
+    following_uid: str
+
+
+class PollVoteRecord(SyncRecord):
+    kind: ClassVar[str] = KIND_POLL_VOTES
+    reference_fields: ClassVar[tuple[str, ...]] = ("voter_uid", "poll_uid", "option_uid")
+
+    voter_uid: str
+    poll_uid: str
+    option_uid: str
+
+
+class AttachmentRecord(SyncRecord):
+    kind: ClassVar[str] = KIND_ATTACHMENTS
+    reference_fields: ClassVar[tuple[str, ...]] = ("author_uid", "target_uid")
+
+    author_uid: str
+    target_kind: TargetKind
+    target_uid: str
+    filename: str
+    content_b64: str = ""
+
+    @field_validator("filename", mode="before")
+    @classmethod
+    def valid_filename(cls, value):
+        text = clean_text(value, multiline=False).replace("\\", "/").rsplit("/", 1)[-1]
+        if not text or len(text) > LIMITS.max_filename_length:
+            raise ValueError(f"filename must be 1-{LIMITS.max_filename_length} characters")
+        return text
+
+    @model_validator(mode="after")
+    def valid_content(self):
+        if self.deleted and not self.content_b64:
+            return self
+        try:
+            raw = base64.b64decode(self.content_b64.encode(), validate=True)
+        except (binascii.Error, ValueError):
+            raise ValueError("content_b64 must be strict base64")
+        if not raw:
+            raise ValueError("content_b64 is empty")
+        if len(raw) > LIMITS.max_attachment_bytes:
+            raise ValueError(f"attachment exceeds {LIMITS.max_attachment_bytes} bytes")
+        self._digest = hashlib.sha256(raw).hexdigest()
+        return self
+
+    def content_bytes(self) -> bytes:
+        return base64.b64decode(self.content_b64.encode())
+
+    def content_fields(self) -> dict:
+        fields = super().content_fields()
+        fields.pop("content_b64", None)
+        fields["sha256"] = getattr(self, "_digest", "")
+        return fields
+
+
 RECORD_MODELS = {
     KIND_USERS: UserRecord,
+    KIND_PROJECTS: ProjectRecord,
     KIND_POSTS: PostRecord,
     KIND_COMMENTS: CommentRecord,
+    KIND_POLLS: PollRecord,
+    KIND_VOTES: VoteRecord,
+    KIND_REACTIONS: ReactionRecord,
+    KIND_FOLLOWS: FollowRecord,
+    KIND_POLL_VOTES: PollVoteRecord,
+    KIND_ATTACHMENTS: AttachmentRecord,
 }

devplacepy/services/sync/selection.py

@@ -3,13 +3,20 @@
 from __future__ import annotations
 
 import hashlib
+import json
 import random
 from collections import Counter
 
-from devplacepy.database.sync import get_links_by_origin_uids
+from devplacepy.database.sync import (
+    delete_candidate,
+    get_links_by_origin_uids,
+    load_candidates,
+    save_candidate,
+)
 from devplacepy.services.sync.options import SyncOptions
 from devplacepy.services.sync.spec import (
     KIND_POSTS,
+    LIMITS,
     SKIP_MIN_COMMENTS,
     SKIP_PARENT,
     SKIP_SAMPLED_OUT,
@@ -32,28 +39,81 @@ def jittered(seconds: int, percent: int, rng: random.Random | None = None) -> in
 
 
 class Selector:
-    def __init__(self, origin: str, options: SyncOptions):
+    def __init__(self, origin: str, options: SyncOptions, *, persist: bool = False):
         self.origin = origin
         self.options = options
+        self.persist = persist
         self.dropped: list[tuple[str, dict, str]] = []
+        self.revived = 0
+        self.parked = 0
+
+    def revive(self, records: dict[str, list[dict]]) -> dict[str, list[dict]]:
+        if not self.persist or not self.options.min_comments:
+            return records
+        posts = {str(item.get("origin_uid")): item for item in records.get("posts", [])}
+        comments: dict[str, dict] = {}
+        users: dict[str, dict] = {}
+        for candidate in load_candidates(self.origin):
+            try:
+                data = json.loads(candidate["record"])
+            except ValueError:
+                delete_candidate(self.origin, candidate["post_uid"])
+                continue
+            posts.setdefault(str(data["post"].get("origin_uid")), data["post"])
+            comments.update({str(item.get("origin_uid")): item for item in data.get("comments", [])})
+            users.update({str(item.get("origin_uid")): item for item in data.get("users", [])})
+            self.revived += 1
+        if not self.revived:
+            return records
+        comments.update({str(item.get("origin_uid")): item for item in records.get("comments", [])})
+        users.update({str(item.get("origin_uid")): item for item in records.get("users", [])})
+        return {**records, "users": list(users.values()), "posts": list(posts.values()), "comments": list(comments.values())}
+
+    def park(self, post: dict, comments: list[dict], users: dict[str, dict]) -> None:
+        if not self.persist:
+            return
+        wanted = {str(post.get("author_uid"))} | {str(item.get("author_uid")) for item in comments}
+        data = {"post": post, "comments": comments, "users": [users[uid] for uid in wanted if uid in users]}
+        encoded = json.dumps(data, default=str)
+        if len(encoded) > LIMITS.candidate_max_bytes:
+            return
+        if save_candidate(
+            self.origin,
+            str(post.get("origin_uid")),
+            encoded,
+            len(comments),
+            ttl_days=LIMITS.candidate_ttl_days,
+            cap=LIMITS.candidate_max_per_origin,
+        ):
+            self.parked += 1
 
     def select(self, records: dict[str, list[dict]]) -> tuple[dict[str, list[dict]], Counter]:
         skipped: Counter = Counter()
         if not self.options.selective():
             return records, skipped
+        records = self.revive(records)
         posts = records.get("posts", [])
         known = get_links_by_origin_uids(self.origin, KIND_POSTS, [str(post.get("origin_uid")) for post in posts])
-        counts = Counter(str(comment.get("post_uid")) for comment in records.get("comments", []))
+        users = {str(item.get("origin_uid")): item for item in records.get("users", [])}
+        by_post: dict[str, list[dict]] = {}
+        for comment in records.get("comments", []):
+            by_post.setdefault(str(comment.get("post_uid")), []).append(comment)
         kept, dropped = [], set()
         for post in posts:
             uid = str(post.get("origin_uid"))
-            reason = None if uid in known else self.refusal(post, counts[uid])
+            reason = None if uid in known else self.refusal(post, len(by_post.get(uid, [])))
             if reason:
                 skipped[reason] += 1
                 dropped.add(uid)
                 self.dropped.append(("posts", post, reason))
+                if reason == SKIP_MIN_COMMENTS:
+                    self.park(post, by_post.get(uid, []), users)
+                elif self.persist:
+                    delete_candidate(self.origin, uid)
             else:
                 kept.append(post)
+                if self.persist:
+                    delete_candidate(self.origin, uid)
         comments = []
         for comment in records.get("comments", []):
             if str(comment.get("post_uid")) in dropped:

devplacepy/services/sync/spec.py

@@ -5,9 +5,38 @@ from __future__ import annotations
 from dataclasses import dataclass
 
 KIND_USERS = "users"
+KIND_PROJECTS = "projects"
 KIND_POSTS = "posts"
 KIND_COMMENTS = "comments"
-KINDS = (KIND_USERS, KIND_POSTS, KIND_COMMENTS)
+KIND_POLLS = "polls"
+KIND_VOTES = "votes"
+KIND_REACTIONS = "reactions"
+KIND_FOLLOWS = "follows"
+KIND_POLL_VOTES = "poll_votes"
+KIND_ATTACHMENTS = "attachments"
+KIND_POLL_OPTIONS = "poll_options"
+KINDS = (
+    KIND_USERS,
+    KIND_PROJECTS,
+    KIND_POSTS,
+    KIND_COMMENTS,
+    KIND_POLLS,
+    KIND_VOTES,
+    KIND_REACTIONS,
+    KIND_FOLLOWS,
+    KIND_POLL_VOTES,
+    KIND_ATTACHMENTS,
+)
+TARGET_KINDS = {"post": KIND_POSTS, "comment": KIND_COMMENTS, "project": KIND_PROJECTS}
+EXPORT_KINDS = (
+    KIND_USERS,
+    KIND_PROJECTS,
+    KIND_POSTS,
+    KIND_COMMENTS,
+    KIND_POLLS,
+    KIND_FOLLOWS,
+    KIND_ATTACHMENTS,
+)
 
 STATUS_CREATED = "created"
 STATUS_UPDATED = "updated"
@@ -15,6 +44,7 @@ STATUS_UNCHANGED = "unchanged"
 STATUS_STALE = "stale"
 STATUS_REJECTED = "rejected"
 STATUS_FAILED = "failed"
+STATUS_DEFERRED = "deferred"
 STATUSES = (
     STATUS_CREATED,
     STATUS_UPDATED,
@@ -22,6 +52,7 @@ STATUSES = (
     STATUS_STALE,
     STATUS_REJECTED,
     STATUS_FAILED,
+    STATUS_DEFERRED,
 )
 
 REASON_INVALID = "invalid"
@@ -35,6 +66,14 @@ REASON_LOCALLY_DELETED = "locally_deleted"
 REASON_OLDER_THAN_SYNCED = "older_than_synced"
 REASON_TOMBSTONE_UNKNOWN = "tombstone_for_unknown"
 REASON_INTERNAL = "internal_error"
+REASON_UNKNOWN_TARGET = "unknown_target"
+REASON_UNKNOWN_POLL = "unknown_poll"
+REASON_UNKNOWN_OPTION = "unknown_option"
+REASON_UNKNOWN_PROJECT = "unknown_project"
+REASON_SELF_FOLLOW = "self_follow"
+REASON_IMMUTABLE = "immutable"
+REASON_ACK_DELETED = "acknowledged_deleted"
+REASON_QUEUE_FULL = "pending_queue_full"
 REASONS = (
     REASON_INVALID,
     REASON_UNKNOWN_AUTHOR,
@@ -47,6 +86,22 @@ REASONS = (
     REASON_OLDER_THAN_SYNCED,
     REASON_TOMBSTONE_UNKNOWN,
     REASON_INTERNAL,
+    REASON_UNKNOWN_TARGET,
+    REASON_UNKNOWN_POLL,
+    REASON_UNKNOWN_OPTION,
+    REASON_UNKNOWN_PROJECT,
+    REASON_SELF_FOLLOW,
+    REASON_IMMUTABLE,
+    REASON_ACK_DELETED,
+    REASON_QUEUE_FULL,
+)
+DEFERRABLE_REASONS = (
+    REASON_UNKNOWN_AUTHOR,
+    REASON_UNKNOWN_POST,
+    REASON_UNKNOWN_PARENT,
+    REASON_UNKNOWN_TARGET,
+    REASON_UNKNOWN_POLL,
+    REASON_UNKNOWN_PROJECT,
 )
 
 SYNC_ACTOR = "sync"
@@ -60,6 +115,7 @@ SKIP_TRANSLATION = "translation_failed"
 SKIP_REWRITE = "rewrite_failed"
 SKIP_PARENT = "parent_skipped"
 SKIP_IDENTITY_CAP = "identity_cap"
+SKIP_ALIAS_ENGAGEMENT = "alias_engagement"
 TRANSIENT_SKIPS = (SKIP_IDENTITY, SKIP_TRANSLATION, SKIP_REWRITE, SKIP_IDENTITY_CAP)
 SKIP_REASONS = (
     SKIP_SAMPLED_OUT,
@@ -71,6 +127,7 @@ SKIP_REASONS = (
     SKIP_REWRITE,
     SKIP_PARENT,
     SKIP_IDENTITY_CAP,
+    SKIP_ALIAS_ENGAGEMENT,
 )
 
 
@@ -94,6 +151,27 @@ class SyncLimits:
     future_skew_seconds: int = 300
     earliest_year: int = 2000
     max_errors_reported: int = 5
+    max_batch_bytes: int = 32 * 1024 * 1024
+    max_attachment_bytes: int = 5 * 1024 * 1024
+    max_filename_length: int = 200
+    max_project_title_length: int = 200
+    max_project_description_length: int = 5000
+    max_project_field_length: int = 500
+    max_poll_question_length: int = 500
+    max_poll_option_length: int = 200
+    min_poll_options: int = 2
+    max_poll_options: int = 6
+    max_emoji_length: int = 16
+    pending_max_per_origin: int = 5000
+    pending_ttl_days: int = 7
+    pending_max_attempts: int = 5
+    pending_release_cap: int = 200
+    pending_max_record_bytes: int = 256 * 1024
+    export_page_size: int = 50
+    deletions_page_size: int = 50
+    candidate_ttl_days: int = 7
+    candidate_max_per_origin: int = 1000
+    candidate_max_bytes: int = 512 * 1024
 
 
 @dataclass(frozen=True)
@@ -109,6 +187,7 @@ class AliasLimits:
     standalone_min_length: int = 4
     mention_min_length: int = 3
     max_new_identities_per_run: int = 100
+    heal_rewrites_per_run: int = 5000
     max_translations_per_run: int = 40
     translate_timeout_seconds: int = 90
     name_timeout_seconds: int = 60
@@ -118,6 +197,40 @@ class AliasLimits:
     banned_fragments: tuple[str, ...] = ("user", "bot", "gpt", "claude", "assistant", "anon")
 
 
+@dataclass(frozen=True)
+class HandleLimits:
+    min_length: int = 3
+    max_length: int = 32
+    max_base_length: int = 18
+    plain_weight: int = 40
+    digits_weight: int = 30
+    separated_weight: int = 8
+    funny_weight: int = 17
+    leet_weight: int = 5
+    year_weight: int = 35
+    short_number_weight: int = 40
+    two_digit_weight: int = 25
+    year_first: int = 1972
+    year_last: int = 2006
+    short_number_max: int = 99
+    rerolls: int = 24
+    top_level_labels: tuple[str, ...] = (
+        "com", "net", "org", "io", "dev", "me", "sh", "co", "app", "ai", "be", "nl", "de", "fr", "uk", "us",
+        "ca", "au", "dk", "se", "no", "fi", "ru", "jp", "cn", "info", "blog", "page", "xyz", "tech", "site",
+        "online", "cc", "tv", "fm", "ly", "gg", "so", "im", "is", "it", "es", "pl", "cz", "ch", "at", "eu",
+        "pt", "br", "in", "kr", "tw", "hk", "sg", "nz", "za", "mx", "ar", "cl", "gl", "pub", "wiki", "social",
+    )
+    ignored_labels: tuple[str, ...] = ("www", "blog", "mail", "m", "en")
+    relation_words: tuple[str, ...] = ("by", "via")
+    prefixes: tuple[str, ...] = ("its", "real", "mr", "just", "not", "dr", "captain", "sir", "the")
+    suffixes: tuple[str, ...] = (
+        "_dev", "codes", "hq", "_ftw", "_sh", "_irl", "_the_great", "_wasnt_here", "_has_bugs",
+        "_on_fire", "_not_found", "_sudo",
+    )
+    funny_numbers: tuple[str, ...] = ("42", "1337", "404", "007", "9000", "8080", "31337", "101", "64", "256", "500", "418")
+    leet_map: tuple[tuple[str, str], ...] = (("a", "4"), ("e", "3"), ("o", "0"), ("i", "1"), ("s", "5"))
+
+
 @dataclass(frozen=True)
 class SyncSpec:
     limits: SyncLimits = SyncLimits()
@@ -137,3 +250,4 @@ class SyncSpec:
 SPEC = SyncSpec()
 LIMITS = SPEC.limits
 ALIAS = AliasLimits()
+HANDLES = HandleLimits()

devplacepy/static/css/admin.css

@@ -621,11 +621,13 @@
 }
 
 .nodes-state.off,
-.nodes-status.revoked {
+.nodes-status.revoked,
+.nodes-status.open {
     color: var(--danger);
 }
 
-.nodes-status.suspended {
+.nodes-status.suspended,
+.nodes-status.half_open {
     color: var(--warning);
 }
 

devplacepy/static/css/communities.css

@@ -442,3 +442,23 @@
 .ledger-runs td[data-label="Mode"] {
     white-space: nowrap;
 }
+
+.community-schema-form {
+    display: grid;
+    gap: var(--space-md);
+    margin-bottom: var(--space-lg);
+}
+
+.community-schema-text {
+    width: 100%;
+    min-height: 24rem;
+    padding: var(--space-md);
+    border: 1px solid var(--border);
+    border-radius: var(--radius);
+    background: var(--bg-card);
+    color: var(--text-primary);
+    font-family: var(--font-mono);
+    font-size: 0.8125rem;
+    line-height: 1.5;
+    resize: vertical;
+}

devplacepy/static/js/AdminCommunity.js

@@ -0,0 +1,39 @@
+// retoor <retoor@molodetz.nl>
+
+import { Poller } from "./Poller.js";
+
+const REFRESH_MS = 3000;
+const BUSY = "true";
+
+class AdminCommunity {
+    constructor() {
+        this.root = document.querySelector("[data-community-root]");
+        if (!this.root || this.root.dataset.busy !== BUSY) return;
+        this.poller = new Poller(() => this.refresh(), REFRESH_MS, { pauseHidden: true, immediate: false });
+    }
+
+    async refresh() {
+        const response = await fetch(window.location.href, { headers: { "Accept": "text/html" }, credentials: "same-origin" });
+        if (!response.ok) return;
+        const fresh = new DOMParser().parseFromString(await response.text(), "text/html");
+        const next = fresh.querySelector("[data-community-root]");
+        if (!next) return;
+        this.root.querySelectorAll("[data-live]").forEach((section) => {
+            const replacement = next.querySelector(`[data-live="${section.dataset.live}"]`);
+            if (replacement) this.swap(section, replacement);
+        });
+        this.root.dataset.busy = next.dataset.busy;
+        if (next.dataset.busy !== BUSY) this.poller.stop();
+    }
+
+    swap(section, replacement) {
+        const open = [...section.querySelectorAll("details[data-keep-open]")].map((item) => item.open);
+        section.innerHTML = replacement.innerHTML;
+        section.querySelectorAll("details[data-keep-open]").forEach((item, index) => {
+            item.open = Boolean(open[index]);
+        });
+    }
+}
+
+window.AdminCommunity = AdminCommunity;
+export { AdminCommunity };

devplacepy/static/js/AdminNodes.js

@@ -44,6 +44,10 @@ class AdminNodes {
                 this.openSuspend(uid, button.dataset.label);
                 return;
             }
+            if (action === "override") {
+                await this.override(uid, button.dataset.value, button.dataset.label);
+                return;
+            }
             const urls = {
                 "unsuspend": `/admin/nodes/devices/${uid}/unsuspend`,
                 "resolve-flag": `/admin/nodes/flags/${uid}/resolve`,
@@ -52,6 +56,16 @@ class AdminNodes {
         });
     }
 
+    async override(uid, value, label) {
+        const allowing = value === "allow";
+        const message = allowing
+            ? `Let ${label} serve jobs although its address is not residential or mobile?`
+            : `Remove the address class override of ${label}?`;
+        const ok = await window.app.dialog.confirm({ message, danger: allowing });
+        if (!ok) return;
+        await this.post(`/admin/nodes/devices/${uid}/class`, { value }, allowing ? "Override set" : "Override cleared");
+    }
+
     openSuspend(uid, label) {
         this.suspendForm.action = `/admin/nodes/devices/${uid}/suspend`;
         this.suspendTitle.textContent = `Suspend device: ${label}`;

devplacepy/static/js/Application.js

@@ -28,6 +28,7 @@ import { TelegramPairing } from "./TelegramPairing.js";
 import { NodeDonation } from "./NodeDonation.js";
 import { AdminNotificationDefaults } from "./AdminNotificationDefaults.js";
 import { AdminNodes } from "./AdminNodes.js";
+import { AdminCommunity } from "./AdminCommunity.js";
 import { UserAiUsage } from "./UserAiUsage.js";
 import { Tabs } from "./Tabs.js";
 import { DeviiTerminal } from "./DeviiTerminal.js";
@@ -105,6 +106,7 @@ class Application {
         this.nodeDonation = new NodeDonation();
         this.adminNotificationDefaults = new AdminNotificationDefaults();
         this.adminNodes = new AdminNodes();
+        this.adminCommunity = new AdminCommunity();
         this.userAiUsage = new UserAiUsage();
         this.tabs = new Tabs();
         this.windows = new WindowManager();

devplacepy/static/js/GameFarm.js

@@ -278,6 +278,13 @@ export class GameFarm {
         return `<div class="game-grid" data-game-grid>${cells}${buy}</div>`;
     }
 
+    _plotStopHtml(plot) {
+        if (!plot.can_stop_replant) return "";
+        const next = plot.stop_after_harvest ? "false" : "true";
+        const label = plot.stop_after_harvest ? "&#9654; Resume auto-replant" : "&#9209; Stop after harvest";
+        return `<form method="post" action="/game/plot/stop" data-game-action="plot-stop"><input type="hidden" name="slot" value="${plot.slot}"><input type="hidden" name="stop" value="${next}"><button type="submit" class="btn btn-sm" aria-pressed="${plot.stop_after_harvest ? "true" : "false"}">${label}</button></form>`;
+    }
+
     _plotHtml(plot, farm) {
         const golden = plot.is_golden ? " game-plot-golden" : "";
         const head = `<div class="game-plot game-plot-${plot.state}${golden}" data-slot="${plot.slot}" data-state="${plot.state}" data-ready-at="${plot.ready_at}">`;
@@ -316,6 +323,7 @@ export class GameFarm {
             if (farm.is_owner && plot.fertilize_cost) {
                 body += `<form method="post" action="/game/fertilize" data-game-action="fertilize"><input type="hidden" name="slot" value="${plot.slot}"><button type="submit" class="btn btn-sm">&#9889; Fertilize (${Format.coins(plot.fertilize_cost)})</button></form>`;
             }
+            body += this._plotStopHtml(plot);
         } else {
             const goldenMark = plot.is_golden ? " &#10024;" : "";
             const raided = plot.raided_pct ? `<span class="plot-raided">Raided ${plot.raided_pct}%</span>` : "";
@@ -323,6 +331,7 @@ export class GameFarm {
             if (farm.is_owner) {
                 const label = plot.is_golden ? "Harvest golden" : "Harvest";
                 body += `<form method="post" action="/game/harvest" data-game-action="harvest"><input type="hidden" name="slot" value="${plot.slot}"><button type="submit" class="btn btn-sm btn-primary">${label}</button></form>`;
+                body += this._plotStopHtml(plot);
             } else if (plot.can_steal) {
                 body += `<form method="post" action="/game/farm/${farm.owner_username}/steal" data-game-action="steal"><input type="hidden" name="slot" value="${plot.slot}"><button type="submit" class="btn btn-sm btn-danger" data-confirm="Steal this ready build?">Steal +${Format.coins(plot.steal_coins)}</button></form>`;
             } else if (plot.steal_reason === "stripped") {

devplacepy/templates/_game_grid.html

@@ -32,6 +32,9 @@
             <button type="submit" class="btn btn-sm">&#9889; Fertilize ({{ format_coins(plot.fertilize_cost) }})</button>
         </form>
         {% endif %}
+        {% if plot.can_stop_replant %}
+        {% include "_game_plot_stop.html" %}
+        {% endif %}
         {% else %}
         <span class="plot-icon plot-ready-icon" aria-hidden="true">{{ plot.crop_icon }}</span>
         <span class="plot-crop-name">{{ plot.crop_name }}{% if plot.is_golden %} &#10024;{% endif %}</span>
@@ -41,6 +44,9 @@
             <input type="hidden" name="slot" value="{{ plot.slot }}">
             <button type="submit" class="btn btn-sm btn-primary">Harvest{% if plot.is_golden %} golden{% endif %}</button>
         </form>
+        {% if plot.can_stop_replant %}
+        {% include "_game_plot_stop.html" %}
+        {% endif %}
         {% elif plot.can_steal %}
         <form method="post" action="/game/farm/{{ farm.owner_username }}/steal" data-game-action="steal">
             <input type="hidden" name="slot" value="{{ plot.slot }}">

devplacepy/templates/_game_plot_stop.html

@@ -0,0 +1,5 @@
+<form method="post" action="/game/plot/stop" data-game-action="plot-stop">
+    <input type="hidden" name="slot" value="{{ plot.slot }}">
+    <input type="hidden" name="stop" value="{{ 'false' if plot.stop_after_harvest else 'true' }}">
+    <button type="submit" class="btn btn-sm" aria-pressed="{{ 'true' if plot.stop_after_harvest else 'false' }}">{% if plot.stop_after_harvest %}&#9654; Resume auto-replant{% else %}&#9209; Stop after harvest{% endif %}</button>
+</form>

devplacepy/templates/admin_communities.html

@@ -33,6 +33,9 @@
         <label for="community-login-password">Password
             <input type="password" id="community-login-password" name="login_password" maxlength="500" autocomplete="new-password">
         </label>
+        <label for="community-login-totp">Authenticator secret (only for two-factor accounts)
+            <input type="password" id="community-login-totp" name="login_totp_secret" maxlength="200" autocomplete="off">
+        </label>
     </details>
     <button type="submit" class="admin-btn admin-btn-primary">Discover</button>
 </form>