Community sync: fix comment, identity and refusal bugs; add per-record sync ledger

Fixes found by investigating the live lobste.rs run:
- Comments came back 0: the learner probed the external article, not the
  discussion page. A comments_url field (RSS <comments>, JSON key, or an HTML
  "comments" anchor) now drives comment learning; the article is still used
  for the post body.
- HTML id selectors collapsed every story into one identity (constant /login
  vote link). Id/url selectors must now be distinct; no distinct selector
  rejects the feed.
- Over-long posts were rejected; title, post and comment are truncated to the
  sync limits and reported as a warning.
- A failed article fetch is a warning, not a run error.
- A gate refusal (sync disabled, origin not allowed, aliasing off) ends the run
  as skipped and no longer counts toward the failure floor.
- The same story from two feeds becomes one post, comments attached to it.
- HTML comments follow rel=next up to max_comment_pages.
- A dry run now predicts an unknown-record tombstone as unchanged, like a live
  run.
- /nodes is exempt from the happy-404 decoy so a missing download is a real 404.

Sync log (admin): every run stores a verdict, exact per-outcome counts and one
row per record needing an explanation, with the reason in plain words, at
/admin/communities/{slug}/ledger (HTML, JSON, Devii tool, docs entry).
Bounded: 2000 routine + 2000 problem rows per run, 50 per page, detail kept for
the 10 newest runs, four indexes matching every ORDER BY (asserted by an
EXPLAIN QUERY PLAN test). Measured worst case 37 ms at the 40000-row ceiling.

Also: privacy disclosure for imported community content, status colour channel
tokens, and tests/unit/cli/nodes.py renamed to fix a duplicate-basename
collection error.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-16 09:26:08 +02:00
co-authored by Claude Sonnet 5.5
parent 6f80a820da
commit 8d18994c2e
35 changed files with 1987 additions and 69 deletions
+1 -1
View File
@@ -255,7 +255,7 @@ Members can install [Agent Skills](https://agentskills.io) from a URL at `/skill
## Community sync
Users, posts and comments can be copied in from another community. The sending side posts batches of up to 50 records to `POST /sync/users`, `/sync/posts` and `/sync/comments` (administrator API key, `sync_enabled` set to `1`). Each record carries the id it has in the origin community (`origin_uid`); the server keeps a `sync_links` row from `(origin, kind, origin_uid)` to the local row, so a repeated record is an update or a no-op and never a duplicate. Records are validated and normalized with Pydantic (`devplacepy/services/sync/records.py`) and every record gets its own result, so one bad record never fails the batch. Votes, stars, XP and levels cannot be verified across communities and are not imported; they are listed under `ignored` in the result. Synced content sends no notifications, awards no XP, runs no AI correction and passes the content filter. `sync_origins` optionally restricts which origins may send. Limits, statuses and the record schemas are published at `GET /sync/spec.json`. There is no export endpoint and no scheduled sync service yet; the batch API is the building block for it. Usage is written up in `prompt.md`.
Users, posts and comments can be copied in from another community. The sending side posts batches of up to 50 records to `POST /sync/users`, `/sync/posts` and `/sync/comments` (administrator API key, `sync_enabled` set to `1`). Each record carries the id it has in the origin community (`origin_uid`); the server keeps a `sync_links` row from `(origin, kind, origin_uid)` to the local row, so a repeated record is an update or a no-op and never a duplicate. Records are validated and normalized with Pydantic (`devplacepy/services/sync/records.py`) and every record gets its own result, so one bad record never fails the batch. Votes, stars, XP and levels cannot be verified across communities and are not imported; they are listed under `ignored` in the result. Synced content sends no notifications, awards no XP, runs no AI correction and passes the content filter. `sync_origins` optionally restricts which origins may send. Limits, statuses and the record schemas are published at `GET /sync/spec.json`. The opt-in Community sync service (`CommunityService`) is the scheduled sender: it learns an external community, runs its generated collectors and feeds this API. A run refused by the kill switch, the allowlist or the aliasing switch is recorded as `skipped` and is never counted as a failure. Every run also writes a per-record sync log (`/admin/communities/{slug}/ledger?run=<uid>`): a verdict, exact counts per outcome (created, updated, unchanged, stale, skipped, warning, rejected, failed, error), and a filterable, paginated list with the reason each record was not synced in plain words. It is index-bound and capped, so it stays fast however large a run is. There is no export endpoint. Usage is written up in `prompt.md`.
### Community discovery