devplacepy/devplacepy/services/community/CLAUDE.md

This file documents community discovery (devplacepy/services/community/, routers/admin/communities.py, cli/community.py): finding an external community, learning its structure once, generating collectors, and feeding what they collect through the sync API. Claude Code auto-loads it whenever a file under devplacepy/services/community/ is read or edited. The design record is add_community.md.

What exists today

Piece	File	State
Constants, caps, statuses	spec.py (CommunityLimits)	built
Path, feed and CSS extraction (stdlib only)	extract.py	built, tested
Collector engine (retry, backoff, pagination, incremental, comments)	runtime.py	built, tested
Search (rsearch), feed detection, structure learning, metadata	search.py, feeds.py, learner.py, discoverer.py	built, tested
Code generation	generator.py	built, tested
Persistence	store.py (communities, community_runs), files.py	built
Push into sync, with dry run	pusher.py	built
Run orchestration	runner.py (CommunityRunner)	built
Scheduled service	service.py (CommunityService, opt-in, dry run by default)	built
Admin pages	/admin/communities, /admin/communities/{slug}, /admin/communities/{slug}/ledger (the sync log)	built
Sync ledger (per-record result and reason, per run)	ledger.py, ledger_store.py, table community_run_items	built
CLI	devplace community discover|run|login|list	built
Devii tools (community_*), docs group, privacy row	services/devii/actions/catalog/community.py, docs_api/groups/community.py	built, all administrator-only
AI structure assistant	ai.py	built, optional, every proposal validated against the sample
Login and credentials	extract.find_login_form, runtime.Collector.login, discoverer.LoginLearner	built

Flow

discover -> SearchPhase (a direct address skips rsearch; otherwise rsearch.app.molodetz.nl/search, ranked by host/title match, social sites dropped, at most 5 candidates) -> FeedDetector (the page itself, <link rel=alternate>, same-host "latest/posts/..." links one level deep, then a fixed list of probe paths; JS-only pages escalate to Fetcher.render) -> StructureLearner per feed -> MetadataLearner (name, description, language, favicon, member count, rules text) -> Collector verification run on a small sample. A community is only ready when the verification run collected at least one post. No feed plus a login wall is login_required; anything else is failed with the reason.

Feed types: rss, atom, json_api, html_list. Learned field maps use the sync record field names (id, title, content, created_at, updated_at, author username/id, comment parent_id), each an ordered list of alternates; the collector emits records ready for UserRecord/PostRecord/CommentRecord. JSON paths are $.a.b[0], [*]. XML feeds are converted to dicts so the same paths work ($.link.@href). HTML fields are selector::text or selector::attr(name).

Browser, JavaScript, login, AI

• stealthii is a project dependency (pyproject.toml, git URL; the Dockerfile installs git). fetcher.Fetcher shares one stealth Chromium per event loop (SharedBrowser), applies the SSRF guard (guard_public_url) before every request and every page.goto, and waits at least delay_seconds between requests to one host. Without the browser it falls back to the guarded plain client and logs that JavaScript rendering is unavailable. CommunityService.on_disable closes the browser.
• JavaScript pages. If a page yields nothing statically, detection renders it (Fetcher.render_html, page.content()); the feed is then marked "render": true and the collector renders it on every run. The same fallback applies to post pages when comments only exist after rendering.
• Comments and post bodies from HTML pages. For any feed (RSS, Atom, JSON, HTML list) whose posts expose an address, PageProbe opens up to three post pages, finds the repeated comment element (class hint comment|reply|response|answer, nesting from DOM containment, id from ::attr(id) else a stable hash), and learns a mode: "page" comments spec. It also learns a post.detail.content selector when the post page carries more text than the listing. JSON feeds still prefer inline/url comment modes.
• Login. A community can store a username, an email and a password (columns login_*, never returned by any endpoint, admin form, Devii tool or CLI). Discovery finds the login form (page, login links, /login-style paths, rendered page), logs in with the session community:<origin>, then learns behind the wall. schema.login holds the form (action, field names, hidden defaults re-read at login for CSRF tokens, or selectors for a JavaScript page); login_required makes a missing account a clear LoginRequired. A rejected password raises LoginFailed and the community goes to login_required. Not supported: JSON-API logins, two-factor, captchas.
• AI. When HTML heuristics are weak (no author, no title, content only raw text, no date) or find no item container, comment structure or post body, AiAssistant asks the internal gateway model (site setting community_ai_model, default molodetz; community_ai=0 turns it off) for selectors. A proposal is accepted per field only if it matches at least 60% of the sampled elements. If the gateway is unreachable the learner continues with heuristics and logs it.
• The discussion page is not the article. A feed can link to an external article while the discussion lives elsewhere (lobste.rs: RSS <link> is the article, <comments> the discussion). The learner reads a comments_url field (a JSON/RSS key such as comments/discussion, or an HTML anchor whose text says "comments"/"replies"/"discuss") and learns the comment structure from those pages, while the full-text detail is still learned from the post url. Collector.discussion_link falls back to the post link. Measured on lobste.rs: 0 comments before, 325 comments for 100 posts after.
• Identifiers must be distinct. For HTML lists HtmlInferrer.first(distinct=True) rejects an id/url selector that yields the same value for most items (the first a[href] of a lobste.rs story is the constant /login vote link, which collapsed every story into one identity and gave the title "1"). Anchor classes are tried by text length, and the title reuses the chosen anchor. If no distinct selector exists the feed is rejected, never learned with a constant id.
• Duplicate feeds. Feeds whose sample posts overlap by half or more are learned once (the one with comments wins); comment feeds (/comments.rss) are never treated as post feeds; generic probe paths are only tried when no feed is advertised.

The sync ledger (admin log: what was synced, what was not, and why)

Every run writes one row per record that needs an explanation, so an administrator can answer "is everything alright" from /admin/communities/{slug}/ledger?run=<uid> without reading a log. The page, the JSON (AdminCommunityLedgerOut), the Devii tool community_ledger and the docs endpoint community-ledger are four faces of one route. Never hand-write an outcome or a reason string elsewhere: ledger.py is the specification (OUTCOMES, REASON_TEXT, LEVEL_TITLES, LedgerLimits), and a test fails when a reason the system can emit has no plain-language explanation.

• Outcomes. created, updated, unchanged, stale, skipped, warning, rejected, failed, error, each with a tone (good, warn, bad, muted). Plain repeats (unchanged with no reason) are counted exactly but never stored; unchanged WITH a reason (deleted here, tombstone for a record never synced) is stored because that is a "not synced, and why".
• Where rows come from (one capture path, no second ledger). SyncPusher records every non-plain result, Selector.dropped and AliasOutcome.dropped record each record they hold back (skipped plus the reason), the collector's notices record an unreachable article or a truncation (warning), and collector errors become error. The counts on the page come from run.tally (exact, from the push result), never from counting stored rows, so a capped list still shows true numbers.
• Bounded by construction. LedgerLimits: 2000 routine rows and 2000 problem rows per run (problems can never be displaced by routine rows; the overflow is counted in items_dropped and shown as a banner), 50 rows per page, 200 pages, counts capped at 10000, record detail kept for the 10 newest runs only (CommunityLimits.ledger_runs_kept; older runs keep their tally, verdict and log). Retention lives in CommunityStore.prune_runs, the same change as the run delete. No query on the page can touch more than the hard ceiling of 40000 rows.
• Fast by index, enforced by a test. Four indexes (idx_community_items_run_rank, ..._run_outcome, ..._community_rank and ..._community_outcome, the last two declared created_at DESC) match the exact ORDER BY of every query, and tests/unit/services/community/ledger.py asserts with EXPLAIN QUERY PLAN that no query scans the table or sorts a community-wide read. Reason options for the community-wide view come from the static spec, not a GROUP BY. Measured on a scratch database at the ceiling (10 runs of 4000 rows): run view 1-4 ms, community view 1-2 ms, community kind+reason filter 11 ms, community text search with no match (full scan) 37 ms. Before bounding the detail window to 10 runs the same search took 97 ms and the reason facets 108 ms at 120000 rows.
• Verdict. ledger.verdict(status, dry_run, tally) returns ok, idle, attention, bad or running plus a one-line headline; it is stored on the run (verdict, headline) so the runs table never loads logs. Runs from before the ledger have no tally: the page derives one from their pushed counts.
• Links. Each row links to the synced record here (batched, at most three queries a page) and to the source (Collector.sources, the post link; a comment links to its post's source). A removed record has no local link.
• A refusal is not a failure. See "Refusals are not failures"; a refused run is skipped with an attention verdict, not an error.

Gaps: the page does not live-refresh while a run is in progress (reload it); the run log and the ledger are written when the run finishes, so a run that is killed mid-way leaves neither; a text search across the whole community scans up to 40000 rows.

Output: one engine, concrete file

runtime.py is the only collector implementation. CodeGenerator writes data/communities/<slug>/collectors.py by inlining extract.py and runtime.py (stdlib only), the learned SCHEMA as a literal, and one collect_<feed> wrapper per feed with the selectors in comments; plus schema.json. The service imports that file, so a human edit to SCHEMA takes effect. python collectors.py runs it standalone with stealthii; a login-protected community reads COMMUNITY_USERNAME/COMMUNITY_EMAIL/COMMUNITY_PASSWORD from the environment. The database row holds the schema; the files are regenerated when missing.

Rules that are deliberate

• Dry run is first class. devplace community run --dry-run, the admin "Dry run" button and the service setting community_dry_run (default on) collect and validate through the real sync processors with dry_run=True, log every step, write nothing and keep the watermark. Dependent records (a post whose author this dry run would create) are projected as created instead of unknown_author. A live run needs sync_enabled=1 and honors sync_origins (services/sync/gate.py, shared with the /sync routes).
• Nothing unbounded. Pages, posts, comment fetches, comment pages per post (max_comment_pages), request delay, retries, body size, run time, runs kept (30) and communities per tick are static constants in CommunityLimits; no autotuning.
• Refusals are not failures. A run refused by the gate (sync_enabled=0, origin not in sync_origins) or by the aliasing switch ends skipped: the failure counter is untouched, the community stays ready, and the next run is scheduled normally. Only a run that tried and failed counts toward the floor.
• Floor. After max_failures (5) consecutive failed runs a community is disabled with status failing and the log says to relearn. The machine never deletes; a human relearns or deletes (delete is soft, communities is in SOFT_DELETE_TABLES and UNREPORTABLE_TABLES).
• Sync limits are enforced at collection. Post body (125000), title (500) and comment (125000) are truncated in the runtime to the SyncLimits values, keeping the Source: footer, so an over-long article is imported shortened instead of rejected. A failed article fetch (detail) is a warning in the result and the log, not an error: the post is imported with its listing text and the run is not partial because of it. A failed discussion fetch stays an error, because comments are lost.
• One story, one post across feeds. Posts from different feeds with the same canonical link (extract.canonical_link) collapse to the first one collected, and comments are attached to the surviving post (Collector.aliases). Same-feed posts are never merged, so a site that links one root address from many posts is safe. Across runs the identity still comes from each feed's own id field, so two feeds that list the same story with different ids can still create two posts if the first run saw only one of them; the discovery step drops twin feeds by link overlap to avoid this.
• Incremental. The watermark is the newest updated_at/created_at per feed; a page that is entirely older ends pagination. The newest 10 posts have their comments re-fetched every run; duplicates are absorbed by the sync fingerprint (unchanged).
• Outbound HTTP goes through Fetcher (stealthii plus the SSRF guard), never a bare client.
• Order. users, posts, comments, in batches of 50. A comment's post and parent are always earlier in the batch.
• Admin add and relearn run as per-worker asyncio tasks; the page shows a busy banner and a run row while they work.

Measured

Real run against https://lobste.rs (dry run, scratch database): RSS, JSON and HTML feeds detected, description/favicon/language/rules text learned, 25 posts and 24 authors collected from one page. Fixture tests cover RSS, Atom, JSON with inline comments and page pagination, and HTML lists. With stealthii and Chromium the same run learned lobste.rs from its /rss feed plus post-page comments (li.comments_subtree). Not measured: a real login-protected site, cursor pagination against a live site, the AI path against a live gateway (the local gateway rejected the scratch key, so heuristics ran). Fixtures cover JavaScript rendering, form login, a rejected password, RSS and HTML comments from post pages, feed deduplication and the AI validation.

Gaps

• HTML post-page comments follow rel=next links only (a[rel=next] or link[rel=next]); a site that paginates comments any other way yields the first max_comment_pages it can reach, or the first page only.
• No two-factor, captcha or JSON-API login.
• HTML learning is heuristic plus AI; a site whose markup defeats both needs a hand-edited SCHEMA.
• Votes, reactions and attachments are not collected (the sync API does not take them).
• Cross-feed duplicate detection compares the sampled post addresses at discovery time only.
• Passwords are stored in plain text in the database, as chosen for now; they are masked everywhere they could be displayed.

Selection, aliasing, translation

Per-community options (sync_options) and the community_alias_enabled kill switch are documented in devplacepy/services/sync/CLAUDE.md ("Selection, identity aliasing and translation"). The runner calls prepare between collection and SyncPusher; skip counts appear in the run's pushed["skipped"] and in the totals as skipped_<reason>.