Community sync: fix comment, identity and refusal bugs; add per-record sync ledger
Fixes found by investigating the live lobste.rs run:
- Comments came back 0: the learner probed the external article, not the
discussion page. A comments_url field (RSS <comments>, JSON key, or an HTML
"comments" anchor) now drives comment learning; the article is still used
for the post body.
- HTML id selectors collapsed every story into one identity (constant /login
vote link). Id/url selectors must now be distinct; no distinct selector
rejects the feed.
- Over-long posts were rejected; title, post and comment are truncated to the
sync limits and reported as a warning.
- A failed article fetch is a warning, not a run error.
- A gate refusal (sync disabled, origin not allowed, aliasing off) ends the run
as skipped and no longer counts toward the failure floor.
- The same story from two feeds becomes one post, comments attached to it.
- HTML comments follow rel=next up to max_comment_pages.
- A dry run now predicts an unknown-record tombstone as unchanged, like a live
run.
- /nodes is exempt from the happy-404 decoy so a missing download is a real 404.
Sync log (admin): every run stores a verdict, exact per-outcome counts and one
row per record needing an explanation, with the reason in plain words, at
/admin/communities/{slug}/ledger (HTML, JSON, Devii tool, docs entry).
Bounded: 2000 routine + 2000 problem rows per run, 50 per page, detail kept for
the 10 newest runs, four indexes matching every ORDER BY (asserted by an
EXPLAIN QUERY PLAN test). Measured worst case 37 ms at the 40000-row ceiling.
Also: privacy disclosure for imported community content, status colour channel
tokens, and tests/unit/cli/nodes.py renamed to fix a duplicate-basename
collection error.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 5.5
@@ -255,7 +255,7 @@ Members can install [Agent Skills](https://agentskills.io) from a URL at `/skill
|
||||
|
||||
## Community sync
|
||||
|
||||
Users, posts and comments can be copied in from another community. The sending side posts batches of up to 50 records to `POST /sync/users`, `/sync/posts` and `/sync/comments` (administrator API key, `sync_enabled` set to `1`). Each record carries the id it has in the origin community (`origin_uid`); the server keeps a `sync_links` row from `(origin, kind, origin_uid)` to the local row, so a repeated record is an update or a no-op and never a duplicate. Records are validated and normalized with Pydantic (`devplacepy/services/sync/records.py`) and every record gets its own result, so one bad record never fails the batch. Votes, stars, XP and levels cannot be verified across communities and are not imported; they are listed under `ignored` in the result. Synced content sends no notifications, awards no XP, runs no AI correction and passes the content filter. `sync_origins` optionally restricts which origins may send. Limits, statuses and the record schemas are published at `GET /sync/spec.json`. There is no export endpoint and no scheduled sync service yet; the batch API is the building block for it. Usage is written up in `prompt.md`.
|
||||
Users, posts and comments can be copied in from another community. The sending side posts batches of up to 50 records to `POST /sync/users`, `/sync/posts` and `/sync/comments` (administrator API key, `sync_enabled` set to `1`). Each record carries the id it has in the origin community (`origin_uid`); the server keeps a `sync_links` row from `(origin, kind, origin_uid)` to the local row, so a repeated record is an update or a no-op and never a duplicate. Records are validated and normalized with Pydantic (`devplacepy/services/sync/records.py`) and every record gets its own result, so one bad record never fails the batch. Votes, stars, XP and levels cannot be verified across communities and are not imported; they are listed under `ignored` in the result. Synced content sends no notifications, awards no XP, runs no AI correction and passes the content filter. `sync_origins` optionally restricts which origins may send. Limits, statuses and the record schemas are published at `GET /sync/spec.json`. The opt-in Community sync service (`CommunityService`) is the scheduled sender: it learns an external community, runs its generated collectors and feeds this API. A run refused by the kill switch, the allowlist or the aliasing switch is recorded as `skipped` and is never counted as a failure. Every run also writes a per-record sync log (`/admin/communities/{slug}/ledger?run=<uid>`): a verdict, exact counts per outcome (created, updated, unchanged, stale, skipped, warning, rejected, failed, error), and a filterable, paginated list with the reason each record was not synced in plain words. It is index-bound and capped, so it stays fast however large a run is. There is no export endpoint. Usage is written up in `prompt.md`.
|
||||
|
||||
### Community discovery
|
||||
|
||||
|
||||
@@ -2478,6 +2478,27 @@ COMMUNITY_COLUMNS: dict[str, tuple[tuple[str, object], ...]] = {
|
||||
("pushed", ""),
|
||||
("errors", ""),
|
||||
("log", ""),
|
||||
("tally", ""),
|
||||
("verdict", ""),
|
||||
("headline", ""),
|
||||
("items_stored", 0),
|
||||
("items_dropped", 0),
|
||||
),
|
||||
"community_run_items": (
|
||||
("uid", ""),
|
||||
("run_uid", ""),
|
||||
("community_uid", ""),
|
||||
("seq", 0),
|
||||
("rank", 0),
|
||||
("kind", ""),
|
||||
("outcome", ""),
|
||||
("reason", ""),
|
||||
("origin_uid", ""),
|
||||
("local_uid", ""),
|
||||
("label", ""),
|
||||
("source_url", ""),
|
||||
("detail", ""),
|
||||
("created_at", ""),
|
||||
),
|
||||
}
|
||||
|
||||
@@ -2491,6 +2512,10 @@ def _ensure_community_tables() -> None:
|
||||
_index(db, "communities", "idx_communities_slug", ["slug"], unique=True, where="deleted_at IS NULL")
|
||||
_index(db, "communities", "idx_communities_due", ["status", "enabled", "next_run_at"])
|
||||
_index(db, "community_runs", "idx_community_runs_community", ["community_uid", "started_at"])
|
||||
_index(db, "community_run_items", "idx_community_items_run_rank", ["run_uid", "rank", "seq"])
|
||||
_index(db, "community_run_items", "idx_community_items_run_outcome", ["run_uid", "outcome", "kind", "seq"])
|
||||
_index(db, "community_run_items", "idx_community_items_community_rank", ["community_uid", "rank", "created_at DESC", "seq"])
|
||||
_index(db, "community_run_items", "idx_community_items_community_outcome", ["community_uid", "outcome", "created_at DESC", "seq"])
|
||||
|
||||
|
||||
def _ensure_moderation_tables() -> None:
|
||||
|
||||
@@ -44,6 +44,7 @@ then runs them on a schedule, pushing users, posts and comments through the
|
||||
system finds the login form (or, for a page that needs JavaScript, drives it in the browser), logs
|
||||
in before discovery and before every run. The password is never returned.
|
||||
- **Dry run.** `dry_run=true` on a run validates and logs everything and writes nothing.
|
||||
- **Sync log.** Every run records a verdict, exact counts per outcome and one row per record that needs an explanation (created, updated, skipped, rejected, failed, warnings, errors) with the reason in plain words. Read it with the sync log endpoint, per run or across the most recent runs.
|
||||
- **Asynchronous.** Adding, relearning and running return at once; read the result from the detail
|
||||
endpoint.
|
||||
- **AI assistance.** For HTML pages whose structure the heuristics cannot read, the internal AI model
|
||||
@@ -92,6 +93,45 @@ Every endpoint follows the shared [Conventions & Errors](/docs/conventions.html)
|
||||
"busy": False,
|
||||
},
|
||||
),
|
||||
endpoint(
|
||||
id="community-ledger",
|
||||
method="GET",
|
||||
path="/admin/communities/{slug}/ledger",
|
||||
title="Sync log",
|
||||
summary="Every record a run handled, with the result and the reason, problems first. Without run it spans the most recent runs.",
|
||||
auth="admin",
|
||||
interactive=True,
|
||||
params=[
|
||||
SLUG,
|
||||
field("run", "query", "string", False, "RUN_UID", "A run uid from the community detail. Empty lists the most recent runs."),
|
||||
field("kind", "query", "string", False, "posts", "users, posts, comments or feed."),
|
||||
field("outcome", "query", "string", False, "rejected", "created, updated, unchanged, stale, skipped, warning, rejected, failed or error."),
|
||||
field("reason", "query", "string", False, "content_refused", "A reason key from options.reason."),
|
||||
field("q", "query", "string", False, "", "Search the title, name, origin id and detail."),
|
||||
field("page", "query", "integer", False, "1", "Page number, 50 records per page, at most 200 pages."),
|
||||
],
|
||||
sample_response={
|
||||
"community": ITEM_SAMPLE,
|
||||
"scope": "run",
|
||||
"run": {"uid": "RUN_UID", "status": "partial", "dry_run": False, "verdict": "attention", "headline": "Synced, with 1 rejected."},
|
||||
"verdict": {"level": "attention", "title": "Look at this", "headline": "Synced, with 1 rejected."},
|
||||
"tiles": [{"key": "created", "label": "Created", "tone": "good", "count": 99, "listed": 99}, {"key": "rejected", "label": "Rejected", "tone": "bad", "count": 1, "listed": 1}],
|
||||
"items": [
|
||||
{
|
||||
"kind": "posts",
|
||||
"outcome": "rejected",
|
||||
"reason": "content_refused",
|
||||
"reason_title": "Content filter",
|
||||
"label": "A post the filter refused",
|
||||
"origin_uid": "h:4215bd02760d2fa7415c3a7952b0966f",
|
||||
"detail": "",
|
||||
}
|
||||
],
|
||||
"pagination": {"page": 1, "per_page": 50, "total": 1, "total_pages": 1, "capped": False},
|
||||
"filters": {"outcome": "rejected"},
|
||||
"options": {"kind": ["posts"], "reason": [{"value": "content_refused", "title": "Content filter"}]},
|
||||
},
|
||||
),
|
||||
endpoint(
|
||||
id="community-run",
|
||||
method="POST",
|
||||
|
||||
@@ -21,7 +21,7 @@ logger = logging.getLogger("happy404")
|
||||
POOL_TTL_SECONDS = int(os.environ.get("DEVPLACE_HAPPY_404_POOL_TTL", "300"))
|
||||
POOL_SIZE = 100
|
||||
|
||||
API_PATH_PREFIXES = ("/api", "/dbapi", "/openai", "/xmlrpc", "/swagger", "/openapi.json")
|
||||
API_PATH_PREFIXES = ("/api", "/dbapi", "/openai", "/xmlrpc", "/swagger", "/openapi.json", "/nodes")
|
||||
|
||||
_pool_cache = TTLCache(ttl=POOL_TTL_SECONDS, max_size=1)
|
||||
|
||||
|
||||
@@ -2,20 +2,33 @@
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Annotated
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from fastapi import APIRouter, Form, Request
|
||||
from fastapi.responses import HTMLResponse
|
||||
|
||||
from devplacepy.database import get_setting
|
||||
from devplacepy.database import get_setting, get_table, get_users_by_uids, resolve_object_url
|
||||
from devplacepy.database.sync import count_links, identity_map
|
||||
from devplacepy.responses import action_result, json_error, respond
|
||||
from devplacepy.schemas import AdminCommunitiesOut, AdminCommunityOut
|
||||
from devplacepy.schemas import AdminCommunitiesOut, AdminCommunityLedgerOut, AdminCommunityOut
|
||||
from devplacepy.models import CommunityAddForm, CommunityRunForm, CommunityUpdateForm
|
||||
from devplacepy.seo import base_seo_context, site_url, website_schema
|
||||
from devplacepy.services.audit import record as audit
|
||||
from devplacepy.services.community.discoverer import slug_for_query
|
||||
from devplacepy.services.community.files import files
|
||||
from devplacepy.services.community.ledger import (
|
||||
LEDGER_KINDS,
|
||||
LEVEL_TITLES,
|
||||
OUTCOME_BY_KEY,
|
||||
OUTCOMES,
|
||||
Ledger,
|
||||
explain,
|
||||
tally_of,
|
||||
verdict as run_verdict,
|
||||
)
|
||||
from devplacepy.services.community.ledger_store import ledger_store
|
||||
from devplacepy.services.community.runner import TRIGGER_ADMIN, CommunityRunner
|
||||
from devplacepy.services.community.service import DRY_RUN_KEY
|
||||
from devplacepy.services.community.spec import STATUS_LEARNING
|
||||
@@ -77,6 +90,70 @@ def seo(request: Request, title: str, path: str, crumb: str):
|
||||
)
|
||||
|
||||
|
||||
FILTER_KEYS = ("kind", "outcome", "reason", "q")
|
||||
|
||||
|
||||
def seconds_between(start: str | None, end: str | None) -> int | None:
|
||||
try:
|
||||
return max(0, round((datetime.fromisoformat(end) - datetime.fromisoformat(start)).total_seconds()))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def run_view(run: dict) -> dict:
|
||||
view = dict(run)
|
||||
view["dry_run"] = bool(run.get("dry_run"))
|
||||
for column in ("items_stored", "items_dropped"):
|
||||
view[column] = int(run.get(column) or 0)
|
||||
for column in ("verdict", "headline"):
|
||||
view[column] = run.get(column) or ""
|
||||
if not view.get("tally") and view.get("pushed"):
|
||||
view["tally"] = tally_of(view["pushed"], Ledger())
|
||||
view["duration_seconds"] = seconds_between(run.get("started_at"), run.get("finished_at"))
|
||||
if not view.get("verdict"):
|
||||
outcome = run_verdict(run.get("status", ""), view["dry_run"], view.get("tally") or {})
|
||||
view["verdict"], view["headline"] = outcome["level"], outcome["headline"]
|
||||
return view
|
||||
|
||||
|
||||
def local_links(rows: list[dict]) -> dict[str, str]:
|
||||
wanted: dict[str, list[str]] = {"posts": [], "users": [], "comments": []}
|
||||
for row in rows:
|
||||
if row.get("local_uid") and row.get("kind") in wanted:
|
||||
wanted[row["kind"]].append(row["local_uid"])
|
||||
links: dict[str, str] = {}
|
||||
if wanted["posts"]:
|
||||
table = get_table("posts")
|
||||
for found in table.find(table.table.columns.uid.in_(wanted["posts"]), deleted_at=None):
|
||||
links[found["uid"]] = f"/posts/{found.get('slug') or found['uid']}"
|
||||
for uid, found in get_users_by_uids(wanted["users"]).items():
|
||||
links[uid] = f"/profile/{found['username']}"
|
||||
if wanted["comments"]:
|
||||
table = get_table("comments")
|
||||
for found in table.find(table.table.columns.uid.in_(wanted["comments"]), deleted_at=None):
|
||||
target = resolve_object_url(found["target_type"], found["target_uid"])
|
||||
links[found["uid"]] = f"{target}#comment-{found['uid']}" if target else ""
|
||||
return links
|
||||
|
||||
|
||||
def item_view(row: dict, links: dict[str, str]) -> dict:
|
||||
view = dict(row)
|
||||
view["tone"] = OUTCOME_BY_KEY[row["outcome"]].tone if row.get("outcome") in OUTCOME_BY_KEY else "muted"
|
||||
why = explain(row.get("reason", ""))
|
||||
view.update(reason_title=why["title"], reason_text=why["text"], retried=why["retried"])
|
||||
view["local_url"] = links.get(row.get("local_uid") or "", "")
|
||||
return view
|
||||
|
||||
|
||||
def ledger_options(scope: dict) -> dict:
|
||||
facets = ledger_store.facets(**scope)
|
||||
return {
|
||||
"kind": [kind for kind in LEDGER_KINDS if kind in facets["kind"]],
|
||||
"outcome": [{"value": outcome.key, "label": outcome.label} for outcome in OUTCOMES],
|
||||
"reason": [{"value": reason, "title": explain(reason)["title"]} for reason in facets["reason"]],
|
||||
}
|
||||
|
||||
|
||||
def feed_summary(feed: dict) -> dict:
|
||||
return {
|
||||
"id": feed["id"],
|
||||
@@ -162,7 +239,7 @@ async def admin_community(request: Request, slug: str):
|
||||
"favicon": row.get("favicon") or "",
|
||||
"totals": row.get("totals") or {},
|
||||
"feeds": [feed_summary(feed) for feed in (row.get("schema") or {}).get("feeds", [])],
|
||||
"runs": store.runs(row["uid"]),
|
||||
"runs": [run_view(run) for run in store.run_summaries(row["uid"])],
|
||||
"collectors_path": str(files.collectors_path(row["slug"])) if source else "",
|
||||
"collectors_lines": len(source.splitlines()),
|
||||
"busy": store.busy(row["uid"]),
|
||||
@@ -173,6 +250,61 @@ async def admin_community(request: Request, slug: str):
|
||||
)
|
||||
|
||||
|
||||
@router.get("/communities/{slug}/ledger", response_class=HTMLResponse)
|
||||
async def admin_community_ledger(
|
||||
request: Request,
|
||||
slug: str,
|
||||
run: str = "",
|
||||
kind: str = "",
|
||||
outcome: str = "",
|
||||
reason: str = "",
|
||||
q: str = "",
|
||||
page: int = 1,
|
||||
):
|
||||
admin = require_admin(request)
|
||||
row = require_community(slug)
|
||||
run_row = None
|
||||
if run:
|
||||
found = store.run(run)
|
||||
if found is None or found["community_uid"] != row["uid"]:
|
||||
raise not_found("Unknown run")
|
||||
run_row = run_view(found)
|
||||
scope = {"run_uid": run_row["uid"]} if run_row else {"community_uid": row["uid"]}
|
||||
filters = {key: value for key, value in zip(FILTER_KEYS, (kind, outcome, reason, q)) if value}
|
||||
rows, pagination = ledger_store.page(filters, page, **scope)
|
||||
links = local_links(rows)
|
||||
listed = ledger_store.counts(**scope)
|
||||
exact = (run_row or {}).get("tally") or listed
|
||||
path = f"{LIST_URL}/{slug}/ledger"
|
||||
query = urlencode({**({"run": run_row["uid"]} if run_row else {}), **filters})
|
||||
return respond(
|
||||
request,
|
||||
"admin_community_ledger.html",
|
||||
{
|
||||
**seo(request, f"{row['name'] or slug} sync log", path, "Communities"),
|
||||
"request": request,
|
||||
"user": admin,
|
||||
"community": public(row),
|
||||
"scope": "run" if run_row else "community",
|
||||
"run": run_row,
|
||||
"run_errors": (run_row or {}).get("errors") or [],
|
||||
"run_log": (run_row or {}).get("log") or "",
|
||||
"verdict": {"level": run_row["verdict"], "title": LEVEL_TITLES.get(run_row["verdict"], ""), "headline": run_row["headline"]} if run_row else {},
|
||||
"tiles": [
|
||||
{"key": o.key, "label": o.label, "tone": o.tone, "help": o.help, "count": int(exact.get(o.key, 0)), "listed": int(listed.get(o.key, 0))}
|
||||
for o in OUTCOMES
|
||||
],
|
||||
"items": [item_view(item, links) for item in rows],
|
||||
"pagination": pagination,
|
||||
"pagination_query": f"{query}&" if query else "",
|
||||
"filters": filters,
|
||||
"options": ledger_options(scope),
|
||||
"admin_section": "communities",
|
||||
},
|
||||
model=AdminCommunityLedgerOut,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/communities/{slug}/run")
|
||||
async def admin_community_run(request: Request, slug: str, data: Annotated[CommunityRunForm, Form()]):
|
||||
admin = require_admin(request)
|
||||
|
||||
@@ -115,6 +115,7 @@ from devplacepy.schemas.backups import (
|
||||
from devplacepy.schemas.admin import (
|
||||
AdminCommunitiesOut,
|
||||
AdminCommunityItemOut,
|
||||
AdminCommunityLedgerOut,
|
||||
AdminCommunityOut,
|
||||
AdminDeviiTaskItemOut,
|
||||
AdminDeviiTasksOut,
|
||||
|
||||
@@ -158,6 +158,65 @@ class AdminCommunityOut(_Out):
|
||||
admin_section: Optional[str] = None
|
||||
|
||||
|
||||
class AdminRunSummaryOut(_Out):
|
||||
uid: str
|
||||
community_uid: str = ""
|
||||
started_at: Optional[str] = None
|
||||
finished_at: Optional[str] = None
|
||||
status: str = ""
|
||||
dry_run: bool = False
|
||||
trigger: str = ""
|
||||
tally: dict = {}
|
||||
verdict: str = ""
|
||||
headline: str = ""
|
||||
items_stored: int = 0
|
||||
items_dropped: int = 0
|
||||
|
||||
|
||||
class AdminLedgerItemOut(_Out):
|
||||
uid: str
|
||||
run_uid: str = ""
|
||||
seq: int = 0
|
||||
kind: str = ""
|
||||
outcome: str = ""
|
||||
tone: str = ""
|
||||
reason: str = ""
|
||||
reason_title: str = ""
|
||||
reason_text: str = ""
|
||||
retried: bool = False
|
||||
origin_uid: str = ""
|
||||
local_uid: str = ""
|
||||
local_url: str = ""
|
||||
label: str = ""
|
||||
source_url: str = ""
|
||||
detail: str = ""
|
||||
created_at: Optional[str] = None
|
||||
|
||||
|
||||
class AdminLedgerTileOut(_Out):
|
||||
key: str
|
||||
label: str
|
||||
tone: str
|
||||
help: str = ""
|
||||
count: int = 0
|
||||
listed: int = 0
|
||||
|
||||
|
||||
class AdminCommunityLedgerOut(_Out):
|
||||
community: AdminCommunityItemOut
|
||||
scope: str = "community"
|
||||
run: Optional[AdminRunSummaryOut] = None
|
||||
run_errors: list[str] = []
|
||||
run_log: str = ""
|
||||
verdict: dict = {}
|
||||
tiles: list[AdminLedgerTileOut] = []
|
||||
items: list[AdminLedgerItemOut] = []
|
||||
pagination: dict = {}
|
||||
filters: dict = {}
|
||||
options: dict = {}
|
||||
admin_section: Optional[str] = None
|
||||
|
||||
|
||||
class AdminNodeDeviceOut(_Out):
|
||||
uid: str = ""
|
||||
label: str = ""
|
||||
|
||||
@@ -13,7 +13,8 @@ This file documents community discovery (`devplacepy/services/community/`, `rout
|
||||
| Push into sync, with dry run | `pusher.py` | built |
|
||||
| Run orchestration | `runner.py` (`CommunityRunner`) | built |
|
||||
| Scheduled service | `service.py` (`CommunityService`, opt-in, dry run by default) | built |
|
||||
| Admin pages | `/admin/communities`, `/admin/communities/{slug}` | built |
|
||||
| Admin pages | `/admin/communities`, `/admin/communities/{slug}`, `/admin/communities/{slug}/ledger` (the sync log) | built |
|
||||
| Sync ledger (per-record result and reason, per run) | `ledger.py`, `ledger_store.py`, table `community_run_items` | built |
|
||||
| CLI | `devplace community discover\|run\|login\|list` | built |
|
||||
| Devii tools (`community_*`), docs group, privacy row | `services/devii/actions/catalog/community.py`, `docs_api/groups/community.py` | built, all administrator-only |
|
||||
| AI structure assistant | `ai.py` | built, optional, every proposal validated against the sample |
|
||||
@@ -32,8 +33,24 @@ Feed types: `rss`, `atom`, `json_api`, `html_list`. Learned field maps use the *
|
||||
- **Comments and post bodies from HTML pages.** For any feed (RSS, Atom, JSON, HTML list) whose posts expose an address, `PageProbe` opens up to three post pages, finds the repeated comment element (class hint `comment|reply|response|answer`, nesting from DOM containment, id from `::attr(id)` else a stable hash), and learns a `mode: "page"` comments spec. It also learns a `post.detail.content` selector when the post page carries more text than the listing. JSON feeds still prefer `inline`/`url` comment modes.
|
||||
- **Login.** A community can store a username, an email and a password (columns `login_*`, never returned by any endpoint, admin form, Devii tool or CLI). Discovery finds the login form (page, login links, `/login`-style paths, rendered page), logs in with the session `community:<origin>`, then learns behind the wall. `schema.login` holds the form (action, field names, hidden defaults re-read at login for CSRF tokens, or selectors for a JavaScript page); `login_required` makes a missing account a clear `LoginRequired`. A rejected password raises `LoginFailed` and the community goes to `login_required`. Not supported: JSON-API logins, two-factor, captchas.
|
||||
- **AI.** When HTML heuristics are weak (no author, no title, content only raw text, no date) or find no item container, comment structure or post body, `AiAssistant` asks the internal gateway model (site setting `community_ai_model`, default `molodetz`; `community_ai=0` turns it off) for selectors. A proposal is accepted per field only if it matches at least 60% of the sampled elements. If the gateway is unreachable the learner continues with heuristics and logs it.
|
||||
- **The discussion page is not the article.** A feed can link to an external article while the discussion lives elsewhere (lobste.rs: RSS `<link>` is the article, `<comments>` the discussion). The learner reads a `comments_url` field (a JSON/RSS key such as `comments`/`discussion`, or an HTML anchor whose text says "comments"/"replies"/"discuss") and learns the comment structure from those pages, while the full-text `detail` is still learned from the post `url`. `Collector.discussion_link` falls back to the post link. Measured on lobste.rs: 0 comments before, 325 comments for 100 posts after.
|
||||
- **Identifiers must be distinct.** For HTML lists `HtmlInferrer.first(distinct=True)` rejects an id/url selector that yields the same value for most items (the first `a[href]` of a lobste.rs story is the constant `/login` vote link, which collapsed every story into one identity and gave the title "1"). Anchor classes are tried by text length, and the title reuses the chosen anchor. If no distinct selector exists the feed is rejected, never learned with a constant id.
|
||||
- **Duplicate feeds.** Feeds whose sample posts overlap by half or more are learned once (the one with comments wins); comment feeds (`/comments.rss`) are never treated as post feeds; generic probe paths are only tried when no feed is advertised.
|
||||
|
||||
## The sync ledger (admin log: what was synced, what was not, and why)
|
||||
|
||||
Every run writes one row per record that needs an explanation, so an administrator can answer "is everything alright" from `/admin/communities/{slug}/ledger?run=<uid>` without reading a log. The page, the JSON (`AdminCommunityLedgerOut`), the Devii tool `community_ledger` and the docs endpoint `community-ledger` are four faces of one route. Never hand-write an outcome or a reason string elsewhere: `ledger.py` is the specification (`OUTCOMES`, `REASON_TEXT`, `LEVEL_TITLES`, `LedgerLimits`), and a test fails when a reason the system can emit has no plain-language explanation.
|
||||
|
||||
- **Outcomes.** `created`, `updated`, `unchanged`, `stale`, `skipped`, `warning`, `rejected`, `failed`, `error`, each with a tone (good, warn, bad, muted). Plain repeats (`unchanged` with no reason) are counted exactly but never stored; `unchanged` WITH a reason (deleted here, tombstone for a record never synced) is stored because that is a "not synced, and why".
|
||||
- **Where rows come from (one capture path, no second ledger).** `SyncPusher` records every non-plain result, `Selector.dropped` and `AliasOutcome.dropped` record each record they hold back (`skipped` plus the reason), the collector's `notices` record an unreachable article or a truncation (`warning`), and collector errors become `error`. The counts on the page come from `run.tally` (exact, from the push result), never from counting stored rows, so a capped list still shows true numbers.
|
||||
- **Bounded by construction.** `LedgerLimits`: 2000 routine rows and 2000 problem rows per run (problems can never be displaced by routine rows; the overflow is counted in `items_dropped` and shown as a banner), 50 rows per page, 200 pages, counts capped at 10000, record detail kept for the 10 newest runs only (`CommunityLimits.ledger_runs_kept`; older runs keep their tally, verdict and log). Retention lives in `CommunityStore.prune_runs`, the same change as the run delete. No query on the page can touch more than the hard ceiling of 40000 rows.
|
||||
- **Fast by index, enforced by a test.** Four indexes (`idx_community_items_run_rank`, `..._run_outcome`, `..._community_rank` and `..._community_outcome`, the last two declared `created_at DESC`) match the exact `ORDER BY` of every query, and `tests/unit/services/community/ledger.py` asserts with `EXPLAIN QUERY PLAN` that no query scans the table or sorts a community-wide read. Reason options for the community-wide view come from the static spec, not a `GROUP BY`. Measured on a scratch database at the ceiling (10 runs of 4000 rows): run view 1-4 ms, community view 1-2 ms, community kind+reason filter 11 ms, community text search with no match (full scan) 37 ms. Before bounding the detail window to 10 runs the same search took 97 ms and the reason facets 108 ms at 120000 rows.
|
||||
- **Verdict.** `ledger.verdict(status, dry_run, tally)` returns `ok`, `idle`, `attention`, `bad` or `running` plus a one-line headline; it is stored on the run (`verdict`, `headline`) so the runs table never loads logs. Runs from before the ledger have no tally: the page derives one from their `pushed` counts.
|
||||
- **Links.** Each row links to the synced record here (batched, at most three queries a page) and to the source (`Collector.sources`, the post link; a comment links to its post's source). A removed record has no local link.
|
||||
- **A refusal is not a failure.** See "Refusals are not failures"; a refused run is `skipped` with an `attention` verdict, not an error.
|
||||
|
||||
Gaps: the page does not live-refresh while a run is in progress (reload it); the run log and the ledger are written when the run finishes, so a run that is killed mid-way leaves neither; a text search across the whole community scans up to 40000 rows.
|
||||
|
||||
## Output: one engine, concrete file
|
||||
|
||||
`runtime.py` is the only collector implementation. `CodeGenerator` writes `data/communities/<slug>/collectors.py` by inlining `extract.py` and `runtime.py` (stdlib only), the learned `SCHEMA` as a literal, and one `collect_<feed>` wrapper per feed with the selectors in comments; plus `schema.json`. The service imports that file, so a human edit to `SCHEMA` takes effect. `python collectors.py` runs it standalone with `stealthii`; a login-protected community reads `COMMUNITY_USERNAME`/`COMMUNITY_EMAIL`/`COMMUNITY_PASSWORD` from the environment. The database row holds the schema; the files are regenerated when missing.
|
||||
@@ -41,8 +58,11 @@ Feed types: `rss`, `atom`, `json_api`, `html_list`. Learned field maps use the *
|
||||
## Rules that are deliberate
|
||||
|
||||
- **Dry run is first class.** `devplace community run --dry-run`, the admin "Dry run" button and the service setting `community_dry_run` (default on) collect and validate through the real sync processors with `dry_run=True`, log every step, write nothing and keep the watermark. Dependent records (a post whose author this dry run would create) are projected as `created` instead of `unknown_author`. A live run needs `sync_enabled=1` and honors `sync_origins` (`services/sync/gate.py`, shared with the `/sync` routes).
|
||||
- **Nothing unbounded.** Pages, posts, comment fetches, request delay, retries, body size, run time, runs kept (30) and communities per tick are static constants in `CommunityLimits`; no autotuning.
|
||||
- **Nothing unbounded.** Pages, posts, comment fetches, comment pages per post (`max_comment_pages`), request delay, retries, body size, run time, runs kept (30) and communities per tick are static constants in `CommunityLimits`; no autotuning.
|
||||
- **Refusals are not failures.** A run refused by the gate (`sync_enabled=0`, origin not in `sync_origins`) or by the aliasing switch ends `skipped`: the failure counter is untouched, the community stays `ready`, and the next run is scheduled normally. Only a run that tried and failed counts toward the floor.
|
||||
- **Floor.** After `max_failures` (5) consecutive failed runs a community is disabled with status `failing` and the log says to relearn. The machine never deletes; a human relearns or deletes (`delete` is soft, `communities` is in `SOFT_DELETE_TABLES` and `UNREPORTABLE_TABLES`).
|
||||
- **Sync limits are enforced at collection.** Post body (125000), title (500) and comment (125000) are truncated in the runtime to the `SyncLimits` values, keeping the `Source:` footer, so an over-long article is imported shortened instead of rejected. A failed article fetch (`detail`) is a `warning` in the result and the log, not an error: the post is imported with its listing text and the run is not `partial` because of it. A failed discussion fetch stays an error, because comments are lost.
|
||||
- **One story, one post across feeds.** Posts from different feeds with the same canonical link (`extract.canonical_link`) collapse to the first one collected, and comments are attached to the surviving post (`Collector.aliases`). Same-feed posts are never merged, so a site that links one root address from many posts is safe. Across runs the identity still comes from each feed's own id field, so two feeds that list the same story with different ids can still create two posts if the first run saw only one of them; the discovery step drops twin feeds by link overlap to avoid this.
|
||||
- **Incremental.** The watermark is the newest `updated_at`/`created_at` per feed; a page that is entirely older ends pagination. The newest 10 posts have their comments re-fetched every run; duplicates are absorbed by the sync fingerprint (`unchanged`).
|
||||
- **Outbound HTTP** goes through `Fetcher` (stealthii plus the SSRF guard), never a bare client.
|
||||
- **Order.** users, posts, comments, in batches of 50. A comment's post and parent are always earlier in the batch.
|
||||
@@ -54,7 +74,7 @@ Real run against `https://lobste.rs` (dry run, scratch database): RSS, JSON and
|
||||
|
||||
## Gaps
|
||||
|
||||
- Comment pagination on HTML post pages is not followed (first page only); `max_comment_posts` bounds the pages opened per run.
|
||||
- HTML post-page comments follow `rel=next` links only (`a[rel=next]` or `link[rel=next]`); a site that paginates comments any other way yields the first `max_comment_pages` it can reach, or the first page only.
|
||||
- No two-factor, captcha or JSON-API login.
|
||||
- HTML learning is heuristic plus AI; a site whose markup defeats both needs a hand-edited `SCHEMA`.
|
||||
- Votes, reactions and attachments are not collected (the sync API does not take them).
|
||||
|
||||
@@ -9,7 +9,7 @@ from dataclasses import dataclass, field
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from devplacepy.services.community.ai import AiAssistant
|
||||
from devplacepy.services.community.extract import field_text, find_login_form, parse_html, select_nodes
|
||||
from devplacepy.services.community.extract import canonical_link, field_text, find_login_form, parse_html, select_nodes
|
||||
from devplacepy.services.community.feeds import FeedDetector
|
||||
from devplacepy.services.community.learner import MetadataLearner, StructureLearner
|
||||
from devplacepy.services.community.runtime import Collector, LoginFailed, LoginRequired
|
||||
@@ -195,15 +195,10 @@ class CommunityDiscoverer:
|
||||
)
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def canonical(link: str) -> str:
|
||||
parts = urlsplit(link)
|
||||
return f"{(parts.hostname or '').removeprefix('www.')}{parts.path.rstrip('/')}".lower()
|
||||
|
||||
def links_of(self, candidate, feed: dict) -> set[str]:
|
||||
spec = feed["post"]["fields"].get("url") or feed["post"]["fields"].get("id")
|
||||
return {
|
||||
self.canonical(urljoin(candidate.url, field_text(item, spec)))
|
||||
canonical_link(urljoin(candidate.url, field_text(item, spec)))
|
||||
for item in candidate.items[: LIMITS.sample_posts]
|
||||
if field_text(item, spec)
|
||||
}
|
||||
|
||||
@@ -8,7 +8,7 @@ import xml.etree.ElementTree as ElementTree
|
||||
from datetime import datetime, timezone
|
||||
from email.utils import parsedate_to_datetime
|
||||
from html.parser import HTMLParser
|
||||
from urllib.parse import urljoin
|
||||
from urllib.parse import urljoin, urlsplit
|
||||
|
||||
PATH_TOKEN = re.compile(r"""\.\s*([^.\[\]]+)|\[\s*(\*|-?\d+|"[^"]*"|'[^']*')\s*\]""")
|
||||
SELECTOR_PART = re.compile(r"^([a-zA-Z][a-zA-Z0-9-]*|\*)?((?:[#.][\w-]+|\[[^\]]+\])*)$")
|
||||
@@ -304,6 +304,11 @@ def select_nodes(root: Node, selector: str) -> list[Node]:
|
||||
return [node for node in root.descendants() if path_matches(node, steps, len(steps) - 1)]
|
||||
|
||||
|
||||
def canonical_link(link: str) -> str:
|
||||
parts = urlsplit(link)
|
||||
return f"{(parts.hostname or '').removeprefix('www.')}{parts.path.rstrip('/')}".lower()
|
||||
|
||||
|
||||
def html_field(node: Node, spec: str) -> str:
|
||||
selector, _, accessor = spec.partition("::")
|
||||
target = node
|
||||
|
||||
@@ -36,6 +36,7 @@ PATTERNS = {
|
||||
"url": r"^(url|link|permalink|href|@href|html_url|canonical_url|web_url|share_url)$",
|
||||
"title": r"^(title|headline|subject|topic_title|name)$",
|
||||
"content": r"^(content_html|content|body_html|body|text|selftext|message|encoded|raw|cooked|description|summary|excerpt)$",
|
||||
"comments_url": r"^(comments|comments_url|comment_url|comments_link|discussion|discussion_url|replies_url)$",
|
||||
"created_at": r"^(created_at|created|createdat|published|published_at|pubdate|date|timestamp|created_utc|posted_at|time|dateadded)$",
|
||||
"updated_at": r"^(updated_at|updated|modified|edited_at|last_edited|bumped_at|last_activity)$",
|
||||
"parent_id": r"^(parent_id|parent|in_reply_to|reply_to|parent_comment_id|parentid)$",
|
||||
@@ -51,9 +52,17 @@ CURSOR_KEYS = re.compile(r"^(next_cursor|cursor|after|next_token|nextcursor|cont
|
||||
PAGE_PARAMS = ("page", "p", "paged")
|
||||
USER_COUNT = re.compile(r"([\d][\d,.\s]{0,12}\d|\d)\s*(?:\+\s*)?(members|users|developers|registered|people)\b", re.I)
|
||||
RULE_PATHS = ("/rules", "/about", "/faq", "/guidelines", "/terms")
|
||||
COMMENT_LINK_TEXT = re.compile(r"\bcomments?\b|\breplies\b|\bdiscuss", re.I)
|
||||
MIN_TITLE_LENGTH = 5
|
||||
STATIC_LINK_SPECS = ("h1 a::attr(href)", "h2 a::attr(href)", "h3 a::attr(href)", "a.title::attr(href)", "a.u-url::attr(href)")
|
||||
STATIC_TITLE_SPECS = ("h1", "h2", "h3", "a.title", "a.u-url")
|
||||
COMMENT_PROBES = ("{feed}/{id}/comments", "{feed}/{id}", "/api/posts/{id}/comments", "/api/comments?post_id={id}")
|
||||
|
||||
|
||||
def is_link(value: str) -> bool:
|
||||
return value.startswith(("http://", "https://", "/"))
|
||||
|
||||
|
||||
def last_segment(path: str) -> str:
|
||||
return re.split(r"[.\[]", path.rstrip("]"))[-1] if path else ""
|
||||
|
||||
@@ -93,6 +102,7 @@ class FieldInferrer:
|
||||
fields = {
|
||||
"id": self.pick_all("id") or self.pick_all("url"),
|
||||
"url": self.pick_all("url"),
|
||||
"comments_url": self.pick_all("comments_url", is_link)[:1],
|
||||
"title": self.pick_all("title")[:1],
|
||||
"content": self.pick_all("content", longest=True)[:3],
|
||||
"created_at": self.pick_all("created_at", valid_date)[:1],
|
||||
@@ -100,7 +110,7 @@ class FieldInferrer:
|
||||
}
|
||||
if not fields["created_at"]:
|
||||
fields["created_at"] = [NOW_TOKEN]
|
||||
return {key: value[:1] if key in ("id", "url") else value for key, value in fields.items() if value}
|
||||
return {key: value[:1] if key in ("id", "url", "comments_url") else value for key, value in fields.items() if value}
|
||||
|
||||
def author_spec(self) -> dict:
|
||||
regex_keys = [path for path in self.paths if AUTHOR_KEYS.match(last_segment(path))]
|
||||
@@ -152,21 +162,63 @@ class HtmlInferrer:
|
||||
hits = sum(1 for node in self.nodes if field_text(node, spec))
|
||||
return hits / len(self.nodes) if self.nodes else 0.0
|
||||
|
||||
def first(self, specs: list[str]) -> str | None:
|
||||
def distinct_share(self, spec: str) -> float:
|
||||
values = {field_text(node, spec) for node in self.nodes}
|
||||
values.discard("")
|
||||
return len(values) / len(self.nodes) if self.nodes else 0.0
|
||||
|
||||
def median_length(self, spec: str) -> float:
|
||||
lengths = [len(field_text(node, spec)) for node in self.nodes]
|
||||
return statistics.median(lengths) if lengths else 0.0
|
||||
|
||||
def first(self, specs: list[str], distinct: bool = False, minimum: int = 0) -> str | None:
|
||||
for spec in specs:
|
||||
if self.share(spec) >= LIMITS.min_field_coverage:
|
||||
return spec
|
||||
if self.share(spec) < LIMITS.min_field_coverage:
|
||||
continue
|
||||
if distinct and self.distinct_share(spec) < LIMITS.min_field_coverage:
|
||||
continue
|
||||
if minimum and self.median_length(spec) < minimum:
|
||||
continue
|
||||
return spec
|
||||
return None
|
||||
|
||||
def anchor_classes(self, text_filter=None) -> list[str]:
|
||||
found: dict[str, list[int]] = {}
|
||||
for node in self.nodes:
|
||||
for anchor in node.descendants():
|
||||
if anchor.tag != "a" or not anchor.attrs.get("href"):
|
||||
continue
|
||||
if text_filter is not None and not text_filter(anchor.text()):
|
||||
continue
|
||||
for name in anchor.classes():
|
||||
found.setdefault(name, []).append(len(anchor.text()))
|
||||
ranked = sorted(found, key=lambda name: (-statistics.median(found[name]), name))
|
||||
return [f"a.{name}" for name in ranked]
|
||||
|
||||
def link_specs(self) -> list[str]:
|
||||
generated = [f"{base}::attr(href)" for base in self.anchor_classes()]
|
||||
return [*STATIC_LINK_SPECS, *generated, "a[href]::attr(href)"]
|
||||
|
||||
def discussion_spec(self, link: str | None) -> str | None:
|
||||
candidates = [f"{base}::attr(href)" for base in self.anchor_classes(lambda text: bool(COMMENT_LINK_TEXT.search(text)))]
|
||||
spec = self.first(candidates, distinct=True)
|
||||
if spec is None or spec == link:
|
||||
return None
|
||||
return spec
|
||||
|
||||
def post_fields(self) -> dict:
|
||||
link = self.first(["h1 a::attr(href)", "h2 a::attr(href)", "h3 a::attr(href)", "a.title::attr(href)", "a[href]::attr(href)"])
|
||||
title = self.first(["h1", "h2", "h3", "a.title", "a[href]"])
|
||||
link = self.first(self.link_specs(), distinct=True)
|
||||
base = link[: -len("::attr(href)")] if link else None
|
||||
title = self.first([*STATIC_TITLE_SPECS, *([base] if base else [])], minimum=MIN_TITLE_LENGTH)
|
||||
content = self.first(["p", ".content", ".body", ".excerpt", ".summary", "::text"])
|
||||
created = self.first(["time::attr(datetime)", "time", "[datetime]::attr(datetime)"])
|
||||
discussion = self.discussion_spec(link)
|
||||
fields: dict = {}
|
||||
if link:
|
||||
fields["id"] = [link]
|
||||
fields["url"] = [link]
|
||||
if discussion:
|
||||
fields["comments_url"] = [discussion]
|
||||
if title:
|
||||
fields["title"] = [title]
|
||||
if content:
|
||||
@@ -353,10 +405,10 @@ class PageProbe:
|
||||
return None
|
||||
return parse_html(text) if status < 400 and text else None
|
||||
|
||||
def links(self, candidate: FeedCandidate, post: dict) -> list[str]:
|
||||
def links(self, candidate: FeedCandidate, post: dict, field: str = "url") -> list[str]:
|
||||
found = []
|
||||
for item in candidate.items:
|
||||
target = field_text(item, post.get("url"))
|
||||
target = field_text(item, post.get(field))
|
||||
if target:
|
||||
absolute = urljoin(candidate.url, target)
|
||||
if absolute not in found:
|
||||
@@ -410,9 +462,10 @@ class PageProbe:
|
||||
|
||||
async def learn(self, candidate: FeedCandidate, post: dict) -> tuple[dict, dict | None]:
|
||||
none = {"mode": "none", "reason": "no comments found on the post pages"}
|
||||
if not post.get("url"):
|
||||
if not post.get("url") and not post.get("comments_url"):
|
||||
return {"mode": "none", "reason": "posts expose no address to open"}, None
|
||||
links = self.links(candidate, post)
|
||||
post_links = self.links(candidate, post)
|
||||
links = self.links(candidate, post, "comments_url") or post_links
|
||||
render = candidate.render
|
||||
roots = [root for root in [await self.load(link, render) for link in links[: LIMITS.sample_comment_posts]] if root is not None]
|
||||
if not roots:
|
||||
@@ -431,11 +484,17 @@ class PageProbe:
|
||||
proposal = await self.ai.comment_structure(roots[0])
|
||||
if proposal:
|
||||
comments = self.ai_comment_spec(proposal, render)
|
||||
detail = await self.detail(candidate, post, roots, render)
|
||||
detail = await self.post_detail(candidate, post, post_links, links, roots, render)
|
||||
if comments["mode"] == "page":
|
||||
logger.info("pages: comments via %s (%s)", comments["item_selector"], sorted(comments["fields"]))
|
||||
return comments, detail
|
||||
|
||||
async def post_detail(self, candidate: FeedCandidate, post: dict, post_links: list[str], links: list[str], roots: list[Node], render: bool) -> dict | None:
|
||||
if post_links == links:
|
||||
return await self.detail(candidate, post, roots, render)
|
||||
own = [root for root in [await self.load(link, candidate.render) for link in post_links[: LIMITS.sample_comment_posts]] if root is not None]
|
||||
return await self.detail(candidate, post, own, candidate.render) if own else None
|
||||
|
||||
async def detail(self, candidate: FeedCandidate, post: dict, roots: list[Node], render: bool) -> dict | None:
|
||||
listed = statistics.median(len(field_text(item, post.get("content"))) for item in candidate.items[: LIMITS.sample_posts]) if post.get("content") else 0
|
||||
chosen = None
|
||||
|
||||
@@ -0,0 +1,261 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from devplacepy.services.sync.spec import (
|
||||
KIND_COMMENTS,
|
||||
KIND_POSTS,
|
||||
KIND_USERS,
|
||||
REASON_CONTENT_REFUSED,
|
||||
REASON_INTERNAL,
|
||||
REASON_INVALID,
|
||||
REASON_LOCALLY_DELETED,
|
||||
REASON_OLDER_THAN_SYNCED,
|
||||
REASON_PARENT_MISMATCH,
|
||||
REASON_PURGED_LOCALLY,
|
||||
REASON_TOMBSTONE_UNKNOWN,
|
||||
REASON_UNKNOWN_AUTHOR,
|
||||
REASON_UNKNOWN_PARENT,
|
||||
REASON_UNKNOWN_POST,
|
||||
SKIP_IDENTITY,
|
||||
SKIP_IDENTITY_CAP,
|
||||
SKIP_MIN_COMMENTS,
|
||||
SKIP_NO_AUTHOR,
|
||||
SKIP_PARENT,
|
||||
SKIP_REWRITE,
|
||||
SKIP_SAMPLED_OUT,
|
||||
SKIP_TOPIC,
|
||||
SKIP_TRANSLATION,
|
||||
TRANSIENT_SKIPS,
|
||||
)
|
||||
|
||||
KIND_FEED = "feed"
|
||||
LEDGER_KINDS = (KIND_USERS, KIND_POSTS, KIND_COMMENTS, KIND_FEED)
|
||||
|
||||
OUTCOME_CREATED = "created"
|
||||
OUTCOME_UPDATED = "updated"
|
||||
OUTCOME_UNCHANGED = "unchanged"
|
||||
OUTCOME_STALE = "stale"
|
||||
OUTCOME_SKIPPED = "skipped"
|
||||
OUTCOME_REJECTED = "rejected"
|
||||
OUTCOME_FAILED = "failed"
|
||||
OUTCOME_WARNING = "warning"
|
||||
OUTCOME_ERROR = "error"
|
||||
|
||||
TONE_BAD = "bad"
|
||||
TONE_WARN = "warn"
|
||||
TONE_GOOD = "good"
|
||||
TONE_MUTED = "muted"
|
||||
TONE_RANK = {TONE_BAD: 0, TONE_WARN: 1, TONE_GOOD: 2, TONE_MUTED: 3}
|
||||
|
||||
LEVEL_OK = "ok"
|
||||
LEVEL_ATTENTION = "attention"
|
||||
LEVEL_BAD = "bad"
|
||||
LEVEL_IDLE = "idle"
|
||||
LEVEL_RUNNING = "running"
|
||||
|
||||
LEVEL_TITLES = {
|
||||
LEVEL_OK: "All good",
|
||||
LEVEL_ATTENTION: "Look at this",
|
||||
LEVEL_BAD: "Needs attention",
|
||||
LEVEL_IDLE: "Nothing new",
|
||||
LEVEL_RUNNING: "Running",
|
||||
}
|
||||
|
||||
REASON_ARTICLE_UNREACHABLE = "article_unreachable"
|
||||
REASON_TRUNCATED = "truncated"
|
||||
REASON_COLLECT_ERROR = "collect_error"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LedgerLimits:
|
||||
routine_per_run: int = 2000
|
||||
problems_per_run: int = 2000
|
||||
page_size: int = 50
|
||||
max_pages: int = 200
|
||||
count_cap: int = 10000
|
||||
label_chars: int = 140
|
||||
detail_chars: int = 300
|
||||
url_chars: int = 500
|
||||
reason_options: int = 40
|
||||
|
||||
|
||||
LEDGER = LedgerLimits()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Outcome:
|
||||
key: str
|
||||
label: str
|
||||
tone: str
|
||||
stored: bool
|
||||
help: str
|
||||
|
||||
|
||||
OUTCOMES: tuple[Outcome, ...] = (
|
||||
Outcome(OUTCOME_CREATED, "Created", TONE_GOOD, True, "New in this community."),
|
||||
Outcome(OUTCOME_UPDATED, "Updated", TONE_GOOD, True, "Changed at the origin and updated here."),
|
||||
Outcome(OUTCOME_UNCHANGED, "Unchanged", TONE_MUTED, True, "Already synced and identical (only counted), or held for a reason such as deleted here (listed with the reason)."),
|
||||
Outcome(OUTCOME_STALE, "Stale", TONE_MUTED, True, "The origin sent an older version than the one already stored."),
|
||||
Outcome(OUTCOME_SKIPPED, "Skipped", TONE_WARN, True, "Held back before sending by the selection, aliasing or translation rules."),
|
||||
Outcome(OUTCOME_WARNING, "Warning", TONE_WARN, True, "Synced, but something degraded on the way."),
|
||||
Outcome(OUTCOME_REJECTED, "Rejected", TONE_BAD, True, "Refused by validation, the content filter or an earlier moderation decision."),
|
||||
Outcome(OUTCOME_FAILED, "Failed", TONE_BAD, True, "An unexpected error inside the sync. The traceback is in the run log."),
|
||||
Outcome(OUTCOME_ERROR, "Error", TONE_BAD, True, "The source could not be read."),
|
||||
)
|
||||
OUTCOME_BY_KEY = {outcome.key: outcome for outcome in OUTCOMES}
|
||||
OUTCOME_KEYS = tuple(outcome.key for outcome in OUTCOMES)
|
||||
STORED_OUTCOMES = tuple(outcome.key for outcome in OUTCOMES if outcome.stored)
|
||||
ROUTINE_OUTCOMES = (OUTCOME_CREATED, OUTCOME_UPDATED, OUTCOME_STALE, OUTCOME_UNCHANGED)
|
||||
|
||||
REASON_TEXT: dict[str, tuple[str, str]] = {
|
||||
REASON_INVALID: ("Invalid record", "A field failed validation: a value was missing, too long, too short or malformed. The detail names the field."),
|
||||
REASON_UNKNOWN_AUTHOR: ("Author not synced", "The author was not in the batch and has no link here yet."),
|
||||
REASON_UNKNOWN_POST: ("Post not synced", "The post this comment belongs to is not synced here, so the comment has nowhere to go."),
|
||||
REASON_UNKNOWN_PARENT: ("Parent comment not synced", "The comment this one replies to is not synced here."),
|
||||
REASON_PARENT_MISMATCH: ("Parent on another post", "The comment replies to a comment that belongs to a different post."),
|
||||
REASON_CONTENT_REFUSED: ("Content filter", "The platform content filter refused this text. A report was filed the same way a native post would be."),
|
||||
REASON_PURGED_LOCALLY: ("Purged here", "This record was permanently removed on this side. A purged record is never recreated."),
|
||||
REASON_LOCALLY_DELETED: ("Deleted here", "A moderator or the owner deleted it on this side. Moderation wins over the origin."),
|
||||
REASON_OLDER_THAN_SYNCED: ("Older than stored", "The origin sent a version older than the one already stored, so it was ignored."),
|
||||
REASON_TOMBSTONE_UNKNOWN: ("Deleted at the origin, never synced", "The origin reports a deletion for a record that was never synced here. Nothing to remove."),
|
||||
REASON_INTERNAL: ("Internal error", "Something unexpected broke while processing this record. The run log has the traceback."),
|
||||
SKIP_SAMPLED_OUT: ("Sampled out", "Outside the percentage of new posts this community accepts. The same post always gets the same answer."),
|
||||
SKIP_MIN_COMMENTS: ("Too few comments", "Fewer comments were collected this run than the minimum you set."),
|
||||
SKIP_TOPIC: ("Topic not allowed", "The post's topic is not on the allow-list."),
|
||||
SKIP_NO_AUTHOR: ("No author", "The author could not be determined, so the item cannot be attributed."),
|
||||
SKIP_IDENTITY: ("Identity unavailable", "A pseudonym could not be created for this author or a mentioned name. Retried on the next run."),
|
||||
SKIP_IDENTITY_CAP: ("Identity cap reached", "The per-run limit of new pseudonyms was reached. Retried on the next run."),
|
||||
SKIP_TRANSLATION: ("Translation failed", "The translation did not come back or the per-run translation limit was reached. Retried on the next run."),
|
||||
SKIP_REWRITE: ("Rewrite failed", "Replacing original names failed. The item is dropped rather than stored under a real name. Retried on the next run."),
|
||||
SKIP_PARENT: ("Parent was skipped", "The post or comment this depends on was skipped, so this one was too."),
|
||||
REASON_ARTICLE_UNREACHABLE: ("Article unreachable", "The original article could not be fetched. The post was imported with the text from the listing."),
|
||||
REASON_TRUNCATED: ("Cut to the limit", "The text was longer than the platform allows and was shortened. The source link is kept."),
|
||||
REASON_COLLECT_ERROR: ("Source error", "A page or feed could not be read. The detail has the address and the answer."),
|
||||
}
|
||||
RETRIED = frozenset(TRANSIENT_SKIPS)
|
||||
|
||||
|
||||
def explain(reason: str) -> dict:
|
||||
if not reason:
|
||||
return {"title": "", "text": "", "retried": False}
|
||||
title, text = REASON_TEXT.get(reason, (reason.replace("_", " ").capitalize(), "No description is recorded for this reason."))
|
||||
return {"title": title, "text": text, "retried": reason in RETRIED}
|
||||
|
||||
|
||||
def clip(value, limit: int) -> str:
|
||||
return re.sub(r"\s+", " ", str(value or "")).strip()[:limit]
|
||||
|
||||
|
||||
def label_for(kind: str, record: dict) -> str:
|
||||
if kind == KIND_USERS:
|
||||
return clip(record.get("username"), LEDGER.label_chars)
|
||||
if kind == KIND_POSTS:
|
||||
return clip(record.get("title") or record.get("content"), LEDGER.label_chars)
|
||||
return clip(record.get("content") or record.get("title") or record.get("username"), LEDGER.label_chars)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Draft:
|
||||
kind: str
|
||||
outcome: str
|
||||
reason: str
|
||||
origin_uid: str
|
||||
local_uid: str
|
||||
label: str
|
||||
source_url: str
|
||||
detail: str
|
||||
|
||||
@property
|
||||
def rank(self) -> int:
|
||||
return TONE_RANK[OUTCOME_BY_KEY[self.outcome].tone]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Ledger:
|
||||
routine_limit: int = LEDGER.routine_per_run
|
||||
problem_limit: int = LEDGER.problems_per_run
|
||||
sources: dict = field(default_factory=dict)
|
||||
items: list = field(default_factory=list)
|
||||
counts: Counter = field(default_factory=Counter)
|
||||
dropped: int = 0
|
||||
_routine: int = 0
|
||||
_problems: int = 0
|
||||
|
||||
def add(self, kind: str, outcome: str, *, reason: str = "", origin_uid: str = "", local_uid: str = "", label: str = "", detail: str = "", source_key: str = "") -> bool:
|
||||
self.counts[outcome] += 1
|
||||
routine = outcome in ROUTINE_OUTCOMES
|
||||
used, limit = (self._routine, self.routine_limit) if routine else (self._problems, self.problem_limit)
|
||||
if used >= limit:
|
||||
self.dropped += 1
|
||||
return False
|
||||
if routine:
|
||||
self._routine += 1
|
||||
else:
|
||||
self._problems += 1
|
||||
self.items.append(
|
||||
Draft(
|
||||
kind=kind,
|
||||
outcome=outcome,
|
||||
reason=reason,
|
||||
origin_uid=clip(origin_uid, 80),
|
||||
local_uid=clip(local_uid, 80),
|
||||
label=clip(label, LEDGER.label_chars),
|
||||
source_url=clip(self.sources.get(source_key or origin_uid, ""), LEDGER.url_chars),
|
||||
detail=clip(detail, LEDGER.detail_chars),
|
||||
)
|
||||
)
|
||||
return True
|
||||
|
||||
def add_notice(self, notice: dict) -> None:
|
||||
self.add(
|
||||
notice.get("kind", KIND_FEED),
|
||||
OUTCOME_WARNING,
|
||||
reason=notice.get("reason", ""),
|
||||
origin_uid=notice.get("origin_uid", ""),
|
||||
label=notice.get("label", ""),
|
||||
detail=notice.get("detail", ""),
|
||||
)
|
||||
|
||||
def add_error(self, message: str) -> None:
|
||||
self.add(KIND_FEED, OUTCOME_ERROR, reason=REASON_COLLECT_ERROR, detail=message)
|
||||
|
||||
def add_dropped_record(self, kind: str, record: dict, reason: str) -> None:
|
||||
source_key = str(record.get("post_uid") or "") if kind == KIND_COMMENTS else ""
|
||||
self.add(kind, OUTCOME_SKIPPED, reason=reason, origin_uid=str(record.get("origin_uid") or ""), label=label_for(kind, record), source_key=source_key)
|
||||
|
||||
|
||||
def tally_of(pushed: dict, ledger: Ledger) -> dict:
|
||||
tally: Counter = Counter()
|
||||
for kind, counts in (pushed or {}).items():
|
||||
if kind == "skipped":
|
||||
tally[OUTCOME_SKIPPED] += sum(counts.values())
|
||||
continue
|
||||
for status, number in counts.items():
|
||||
tally[status] += number
|
||||
tally[OUTCOME_WARNING] += ledger.counts[OUTCOME_WARNING]
|
||||
tally[OUTCOME_ERROR] += ledger.counts[OUTCOME_ERROR]
|
||||
return {key: tally[key] for key in OUTCOME_KEYS if tally[key]}
|
||||
|
||||
|
||||
def verdict(status: str, dry_run: bool, tally: dict) -> dict:
|
||||
tail = " Nothing was written (dry run)." if dry_run else ""
|
||||
if status == "running":
|
||||
return {"level": LEVEL_RUNNING, "headline": "Running now."}
|
||||
if status == "skipped":
|
||||
return {"level": LEVEL_ATTENTION, "headline": "Not attempted: community sync is switched off or the origin is not allowed." + tail}
|
||||
if status == "failed":
|
||||
return {"level": LEVEL_BAD, "headline": "The run failed before anything could be synced." + tail}
|
||||
bad = [f"{tally[key]} {key}" for key in (OUTCOME_FAILED, OUTCOME_ERROR, OUTCOME_REJECTED) if tally.get(key)]
|
||||
warn = [f"{tally[key]} {key}" for key in (OUTCOME_SKIPPED, OUTCOME_WARNING) if tally.get(key)]
|
||||
if tally.get(OUTCOME_FAILED) or tally.get(OUTCOME_ERROR):
|
||||
return {"level": LEVEL_BAD, "headline": ", ".join(bad + warn).capitalize() + "." + tail}
|
||||
if bad or warn:
|
||||
return {"level": LEVEL_ATTENTION, "headline": "Synced, with " + ", ".join(bad + warn) + "." + tail}
|
||||
if not tally.get(OUTCOME_CREATED) and not tally.get(OUTCOME_UPDATED):
|
||||
return {"level": LEVEL_IDLE, "headline": "Nothing new since the last run." + tail}
|
||||
return {"level": LEVEL_OK, "headline": "Everything synced." + tail}
|
||||
@@ -0,0 +1,124 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from devplacepy.database import build_pagination, db, get_table
|
||||
from devplacepy.services.community.ledger import (
|
||||
LEDGER,
|
||||
LEDGER_KINDS,
|
||||
OUTCOME_KEYS,
|
||||
REASON_TEXT,
|
||||
Ledger,
|
||||
)
|
||||
from devplacepy.services.community.store import ITEMS_TABLE, iso, now
|
||||
from devplacepy.utils.text import generate_uid
|
||||
|
||||
TABLE = ITEMS_TABLE
|
||||
INSERT_CHUNK = 500
|
||||
|
||||
|
||||
def escape_like(text: str) -> str:
|
||||
return text.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
|
||||
|
||||
class LedgerStore:
|
||||
def save(self, run_uid: str, community_uid: str, ledger: Ledger) -> int:
|
||||
if not ledger.items:
|
||||
return 0
|
||||
stamp = iso(now())
|
||||
rows = [
|
||||
{
|
||||
"uid": generate_uid(),
|
||||
"run_uid": run_uid,
|
||||
"community_uid": community_uid,
|
||||
"seq": seq,
|
||||
"rank": draft.rank,
|
||||
"kind": draft.kind,
|
||||
"outcome": draft.outcome,
|
||||
"reason": draft.reason,
|
||||
"origin_uid": draft.origin_uid,
|
||||
"local_uid": draft.local_uid,
|
||||
"label": draft.label,
|
||||
"source_url": draft.source_url,
|
||||
"detail": draft.detail,
|
||||
"created_at": stamp,
|
||||
}
|
||||
for seq, draft in enumerate(ledger.items)
|
||||
]
|
||||
get_table(TABLE).insert_many(rows, chunk_size=INSERT_CHUNK)
|
||||
return len(rows)
|
||||
|
||||
def purge_run(self, run_uid: str) -> int:
|
||||
return get_table(TABLE).delete(run_uid=run_uid)
|
||||
|
||||
@staticmethod
|
||||
def scope(run_uid: str | None, community_uid: str | None) -> tuple[list[str], dict]:
|
||||
if run_uid:
|
||||
return ["run_uid = :run_uid"], {"run_uid": run_uid}
|
||||
if community_uid:
|
||||
return ["community_uid = :community_uid"], {"community_uid": community_uid}
|
||||
raise ValueError("a ledger query needs a run or a community")
|
||||
|
||||
def counts(self, run_uid: str | None = None, community_uid: str | None = None) -> dict[str, int]:
|
||||
if TABLE not in db.tables:
|
||||
return {}
|
||||
clauses, params = self.scope(run_uid, community_uid)
|
||||
rows = db.query(f"SELECT outcome, COUNT(*) AS n FROM {TABLE} WHERE {' AND '.join(clauses)} GROUP BY outcome", **params)
|
||||
return {row["outcome"]: int(row["n"]) for row in rows}
|
||||
|
||||
def facets(self, run_uid: str | None = None, community_uid: str | None = None) -> dict[str, list[str]]:
|
||||
if not run_uid:
|
||||
return {"kind": list(LEDGER_KINDS), "reason": list(REASON_TEXT)}
|
||||
if TABLE not in db.tables:
|
||||
return {"kind": [], "reason": []}
|
||||
clauses, params = self.scope(run_uid, community_uid)
|
||||
where = " AND ".join(clauses)
|
||||
kinds = db.query(f"SELECT DISTINCT kind AS value FROM {TABLE} WHERE {where} ORDER BY kind", **params)
|
||||
reasons = db.query(
|
||||
f"SELECT reason AS value, COUNT(*) AS n FROM {TABLE} WHERE {where} AND reason != '' GROUP BY reason ORDER BY n DESC LIMIT :cap",
|
||||
cap=LEDGER.reason_options,
|
||||
**params,
|
||||
)
|
||||
return {"kind": [row["value"] for row in kinds], "reason": [row["value"] for row in reasons]}
|
||||
|
||||
def compile(self, filters: dict, *, run_uid: str | None = None, community_uid: str | None = None) -> tuple[str, dict, str]:
|
||||
clauses, params = self.scope(run_uid, community_uid)
|
||||
if filters.get("kind") in LEDGER_KINDS:
|
||||
clauses.append("kind = :kind")
|
||||
params["kind"] = filters["kind"]
|
||||
if filters.get("outcome") in OUTCOME_KEYS:
|
||||
clauses.append("outcome = :outcome")
|
||||
params["outcome"] = filters["outcome"]
|
||||
if filters.get("reason"):
|
||||
clauses.append("reason = :reason")
|
||||
params["reason"] = str(filters["reason"])[:80]
|
||||
text = str(filters.get("q") or "").strip()[:80]
|
||||
if text:
|
||||
clauses.append("(label LIKE :q ESCAPE '\\' OR origin_uid LIKE :q ESCAPE '\\' OR detail LIKE :q ESCAPE '\\')")
|
||||
params["q"] = f"%{escape_like(text)}%"
|
||||
narrowed = "outcome" in params
|
||||
order = ("seq" if narrowed else "rank, seq") if run_uid else ("created_at DESC, seq" if narrowed else "rank, created_at DESC, seq")
|
||||
return " AND ".join(clauses), params, order
|
||||
|
||||
def page(self, filters: dict, page: int, *, run_uid: str | None = None, community_uid: str | None = None) -> tuple[list[dict], dict]:
|
||||
empty = build_pagination(1, 0, LEDGER.page_size) | {"capped": False}
|
||||
if TABLE not in db.tables:
|
||||
return [], empty
|
||||
where, params, order = self.compile(filters, run_uid=run_uid, community_uid=community_uid)
|
||||
counted = list(db.query(f"SELECT COUNT(*) AS n FROM (SELECT 1 FROM {TABLE} WHERE {where} LIMIT :cap)", cap=LEDGER.count_cap + 1, **params))
|
||||
total = int(counted[0]["n"]) if counted else 0
|
||||
capped = total > LEDGER.count_cap
|
||||
total = min(total, LEDGER.count_cap)
|
||||
pagination = build_pagination(min(max(page, 1), LEDGER.max_pages), total, LEDGER.page_size)
|
||||
rows = list(
|
||||
db.query(
|
||||
f"SELECT * FROM {TABLE} WHERE {where} ORDER BY {order} LIMIT :limit OFFSET :offset",
|
||||
limit=LEDGER.page_size,
|
||||
offset=(pagination["page"] - 1) * LEDGER.page_size,
|
||||
**params,
|
||||
)
|
||||
)
|
||||
return rows, pagination | {"capped": capped}
|
||||
|
||||
|
||||
ledger_store = LedgerStore()
|
||||
@@ -6,10 +6,11 @@ import logging
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from devplacepy.services.community.ledger import OUTCOME_CREATED, OUTCOME_UNCHANGED, Ledger, label_for
|
||||
from devplacepy.services.community.spec import LIMITS
|
||||
from devplacepy.services.sync.batch import BatchRunner
|
||||
from devplacepy.services.sync.records import SyncBatchForm
|
||||
from devplacepy.services.sync.spec import KINDS, STATUSES
|
||||
from devplacepy.services.sync.spec import KINDS, REASON_TOMBSTONE_UNKNOWN, STATUSES
|
||||
|
||||
logger = logging.getLogger("community.push")
|
||||
|
||||
@@ -32,9 +33,26 @@ DEPENDENCIES = {
|
||||
|
||||
|
||||
class SyncPusher:
|
||||
def __init__(self, origin: str, dry_run: bool = False):
|
||||
def __init__(self, origin: str, dry_run: bool = False, ledger: Ledger | None = None):
|
||||
self.origin = origin
|
||||
self.dry_run = dry_run
|
||||
self.ledger = ledger
|
||||
|
||||
def note(self, kind: str, result, raw: dict, outcome: str, reason: str | None = None) -> None:
|
||||
if self.ledger is None:
|
||||
return
|
||||
if outcome == OUTCOME_UNCHANGED and not (reason or result.reason):
|
||||
return
|
||||
self.ledger.add(
|
||||
kind,
|
||||
outcome,
|
||||
reason=reason if reason is not None else ("" if outcome == OUTCOME_CREATED else result.reason),
|
||||
origin_uid=result.origin_uid,
|
||||
local_uid=result.local_uid,
|
||||
label=label_for(kind, raw),
|
||||
detail="; ".join(result.errors[:2]),
|
||||
source_key=str(raw.get("post_uid") or "") if kind == "comments" else "",
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def satisfied_by_projection(result, raw: dict, projected: dict) -> bool:
|
||||
@@ -58,12 +76,19 @@ class SyncPusher:
|
||||
for result, raw in zip(outcome.results, chunk):
|
||||
if result.status == "created":
|
||||
projected[kind].add(result.origin_uid)
|
||||
if self.dry_run and self.satisfied_by_projection(result, raw, projected) and raw.get("deleted"):
|
||||
tally["rejected"] -= 1
|
||||
tally["unchanged"] += 1
|
||||
self.note(kind, result, raw, OUTCOME_UNCHANGED, REASON_TOMBSTONE_UNKNOWN)
|
||||
continue
|
||||
if self.dry_run and self.satisfied_by_projection(result, raw, projected):
|
||||
tally["rejected"] -= 1
|
||||
tally["created"] += 1
|
||||
projected[kind].add(result.origin_uid)
|
||||
self.note(kind, result, raw, OUTCOME_CREATED)
|
||||
logger.info("push[%s]: %s %s depends on a record this dry run would create; counted as created", mode, kind, result.origin_uid)
|
||||
continue
|
||||
self.note(kind, result, raw, result.status)
|
||||
if result.status in ("rejected", "failed"):
|
||||
line = f"{kind} {result.origin_uid}: {result.status} {result.reason} {'; '.join(result.errors[:2])}".strip()
|
||||
report.problems.append(line)
|
||||
|
||||
@@ -12,6 +12,8 @@ from devplacepy.services.sync.selection import Selector
|
||||
from devplacepy.services.community.discoverer import CommunityDiscoverer, Discovery
|
||||
from devplacepy.services.community.fetcher import Fetcher
|
||||
from devplacepy.services.community.files import CommunityFiles, files as default_files
|
||||
from devplacepy.services.community.ledger import Ledger, tally_of, verdict
|
||||
from devplacepy.services.community.ledger_store import LedgerStore, ledger_store as default_ledger_store
|
||||
from devplacepy.services.community.logbook import LogBook
|
||||
from devplacepy.services.community.pusher import SyncPusher
|
||||
from devplacepy.services.community.runtime import LoginRequired
|
||||
@@ -21,6 +23,7 @@ from devplacepy.services.community.spec import (
|
||||
RUN_FAILED,
|
||||
RUN_OK,
|
||||
RUN_PARTIAL,
|
||||
RUN_SKIPPED,
|
||||
STATUS_LOGIN,
|
||||
STATUS_READY,
|
||||
)
|
||||
@@ -45,14 +48,16 @@ class RunReport:
|
||||
log: str = ""
|
||||
run_uid: str = ""
|
||||
watermark: dict = field(default_factory=dict)
|
||||
ledger: Ledger = field(default_factory=Ledger)
|
||||
|
||||
|
||||
class CommunityRunner:
|
||||
def __init__(self, fetcher_factory=Fetcher, store: CommunityStore | None = None, files: CommunityFiles | None = None, sleep=None):
|
||||
def __init__(self, fetcher_factory=Fetcher, store: CommunityStore | None = None, files: CommunityFiles | None = None, sleep=None, ledgers: LedgerStore | None = None):
|
||||
self.fetcher_factory = fetcher_factory
|
||||
self.sleep = sleep
|
||||
self.store = store or default_store
|
||||
self.files = files or default_files
|
||||
self.ledgers = ledgers or default_ledger_store
|
||||
|
||||
async def discover(
|
||||
self,
|
||||
@@ -134,20 +139,38 @@ class CommunityRunner:
|
||||
await asyncio.wait_for(self.execute(row, report), timeout=LIMITS.run_timeout_seconds)
|
||||
except asyncio.TimeoutError:
|
||||
report.errors.append(f"run exceeded {LIMITS.run_timeout_seconds}s")
|
||||
report.ledger.add_error(report.errors[-1])
|
||||
except LoginRequired as exc:
|
||||
report.errors.append(str(exc))
|
||||
report.ledger.add_error(str(exc))
|
||||
if not dry_run:
|
||||
self.store.update(row["uid"], status=STATUS_LOGIN, enabled=0, last_error=str(exc))
|
||||
except Exception as exc:
|
||||
logger.exception("run[%s]: unexpected failure", mode)
|
||||
report.errors.append(f"{type(exc).__name__}: {exc}")
|
||||
report.ledger.add_error(report.errors[-1])
|
||||
for error in report.errors:
|
||||
logger.error("run[%s]: %s", mode, error)
|
||||
logger.info("run[%s]: finished with status %s", mode, report.status)
|
||||
report.log = book.text()
|
||||
if not dry_run:
|
||||
self.settle(row, report)
|
||||
self.store.finish_run(run_uid, report.status, collected=report.collected, pushed=report.pushed, errors=report.errors, log=report.log)
|
||||
tally = tally_of(report.pushed, report.ledger)
|
||||
outcome = verdict(report.status, report.dry_run, tally)
|
||||
stored = self.ledgers.save(run_uid, row["uid"], report.ledger)
|
||||
self.store.finish_run(
|
||||
run_uid,
|
||||
report.status,
|
||||
collected=report.collected,
|
||||
pushed=report.pushed,
|
||||
errors=report.errors,
|
||||
log=report.log,
|
||||
tally=tally,
|
||||
verdict=outcome["level"],
|
||||
headline=outcome["headline"],
|
||||
items_stored=stored,
|
||||
items_dropped=report.ledger.dropped,
|
||||
)
|
||||
self.store.prune_runs(row["uid"])
|
||||
return report
|
||||
|
||||
@@ -155,6 +178,7 @@ class CommunityRunner:
|
||||
refused = refusal(row["origin"])
|
||||
if refused and not report.dry_run:
|
||||
report.errors.append(refused)
|
||||
report.status = RUN_SKIPPED
|
||||
return
|
||||
if refused:
|
||||
logger.warning("run[DRY RUN]: %s; continuing because nothing is written", refused)
|
||||
@@ -165,15 +189,23 @@ class CommunityRunner:
|
||||
report.collected = {kind: len(records[kind]) for kind in KINDS}
|
||||
logger.info("run: collected %s", report.collected)
|
||||
report.errors.extend(result.get("errors", []))
|
||||
for warning in result.get("warnings", []):
|
||||
logger.warning("run: %s", warning)
|
||||
report.ledger.sources.update(result.get("sources", {}))
|
||||
for notice in result.get("notices", []):
|
||||
report.ledger.add_notice(notice)
|
||||
for error in result.get("errors", []):
|
||||
report.ledger.add_error(error)
|
||||
if not result["ok"]:
|
||||
report.errors.append("no feed could be collected")
|
||||
return
|
||||
options = SyncOptions.from_row(row)
|
||||
prepared = await self.prepare(row, options, records, report)
|
||||
if prepared is None:
|
||||
report.status = RUN_SKIPPED
|
||||
return
|
||||
records, skipped, held = prepared
|
||||
push = await asyncio.to_thread(SyncPusher(row["origin"], dry_run=report.dry_run).push, records)
|
||||
push = await asyncio.to_thread(SyncPusher(row["origin"], dry_run=report.dry_run, ledger=report.ledger).push, records)
|
||||
report.pushed = push.counts | ({"skipped": dict(skipped)} if skipped else {})
|
||||
report.errors.extend(push.problems[:20])
|
||||
report.watermark = {} if held else result["state"]
|
||||
@@ -184,7 +216,10 @@ class CommunityRunner:
|
||||
report.status = RUN_DRY if report.dry_run else (RUN_PARTIAL if report.errors or failed else RUN_OK)
|
||||
|
||||
async def prepare(self, row: dict, options: SyncOptions, records: dict, report: RunReport):
|
||||
records, skipped = Selector(row["origin"], options).select(records)
|
||||
selector = Selector(row["origin"], options)
|
||||
records, skipped = selector.select(records)
|
||||
for kind, record, reason in selector.dropped:
|
||||
report.ledger.add_dropped_record(kind, record, reason)
|
||||
held = 0
|
||||
if options.alias or options.translate_to:
|
||||
if options.alias and not alias_allowed():
|
||||
@@ -196,6 +231,8 @@ class CommunityRunner:
|
||||
report.errors.append(str(exc))
|
||||
return None
|
||||
records, held = outcome.records, outcome.transient()
|
||||
for kind, record, reason in outcome.dropped:
|
||||
report.ledger.add_dropped_record(kind, record, reason)
|
||||
skipped.update(outcome.skipped)
|
||||
logger.info("alias: %d substitution(s), %d translated, %d new identities", outcome.substitutions, outcome.translated, outcome.identities_created)
|
||||
if skipped:
|
||||
@@ -223,6 +260,10 @@ class CommunityRunner:
|
||||
last_run_at=iso(now()),
|
||||
)
|
||||
self.store.schedule_next(row["uid"], int(current.get("interval_seconds") or LIMITS.default_interval_seconds), SyncOptions.from_row(current).jitter_percent)
|
||||
elif report.status == RUN_SKIPPED:
|
||||
self.store.update(row["uid"], last_status=report.status, last_error="; ".join(report.errors)[:1000], last_run_at=iso(now()))
|
||||
self.store.schedule_next(row["uid"], int(current.get("interval_seconds") or LIMITS.default_interval_seconds), SyncOptions.from_row(current).jitter_percent)
|
||||
logger.warning("run: skipped for %s by configuration, not counted as a failure: %s", row["slug"], "; ".join(report.errors))
|
||||
elif current["status"] != STATUS_LOGIN:
|
||||
failed = self.store.fail(current, "; ".join(report.errors) or "run failed")
|
||||
self.store.schedule_next(row["uid"], int(failed.get("interval_seconds") or LIMITS.default_interval_seconds), SyncOptions.from_row(failed).jitter_percent)
|
||||
|
||||
@@ -11,6 +11,7 @@ from urllib.parse import urljoin
|
||||
|
||||
from devplacepy.services.community.extract import (
|
||||
Node,
|
||||
canonical_link,
|
||||
field_text,
|
||||
find_login_form,
|
||||
html_field,
|
||||
@@ -28,13 +29,18 @@ UID_PATTERN = re.compile(r"^[A-Za-z0-9._:@-]+$")
|
||||
UID_MAX_LENGTH = 64
|
||||
MIN_POST_LENGTH = 10
|
||||
MIN_COMMENT_LENGTH = 3
|
||||
MAX_TITLE_LENGTH = 500
|
||||
MAX_POST_LENGTH = 125000
|
||||
MAX_COMMENT_LENGTH = 125000
|
||||
MAX_REPLY_DEPTH = 5
|
||||
RETRY_STATUSES = frozenset({408, 425, 429, 500, 502, 503, 504})
|
||||
RENDER_SETTLE_MS = 800
|
||||
NEXT_PAGE_FIELDS = ("a[rel=next]::attr(href)", "link[rel=next]::attr(href)")
|
||||
DEFAULT_LIMITS = {
|
||||
"max_pages": 3,
|
||||
"max_posts": 150,
|
||||
"max_comment_posts": 25,
|
||||
"max_comment_pages": 3,
|
||||
"comment_recheck": 10,
|
||||
"delay_seconds": 1.0,
|
||||
"retries": 3,
|
||||
@@ -96,6 +102,11 @@ class Collector:
|
||||
self.comments: dict[str, dict] = {}
|
||||
self.pages: dict[str, Node] = {}
|
||||
self.errors: list[str] = []
|
||||
self.warnings: list[str] = []
|
||||
self.notices: list[dict] = []
|
||||
self.sources: dict[str, str] = {}
|
||||
self.link_owner: dict[str, tuple[str, str]] = {}
|
||||
self.aliases: dict[str, str] = {}
|
||||
|
||||
async def attempt(self, label: str, action) -> str:
|
||||
last = "no attempt"
|
||||
@@ -213,10 +224,17 @@ class Collector:
|
||||
fields = feed["post"]["fields"]
|
||||
return normalize_instant(field_text(item, fields.get("updated_at")) or field_text(item, fields.get("created_at")))
|
||||
|
||||
def notice(self, kind: str, origin_uid: str, reason: str, label: str, detail: str) -> None:
|
||||
self.notices.append({"kind": kind, "origin_uid": origin_uid, "reason": reason, "label": label, "detail": detail})
|
||||
|
||||
def post_link(self, feed: dict, item, base: str) -> str:
|
||||
link = field_text(item, feed["post"]["fields"].get("url"))
|
||||
return urljoin(base, link) if link else ""
|
||||
|
||||
def discussion_link(self, feed: dict, item, base: str) -> str:
|
||||
link = field_text(item, feed["post"]["fields"].get("comments_url"))
|
||||
return urljoin(base, link) if link else self.post_link(feed, item, base)
|
||||
|
||||
def post_identifier(self, feed: dict, item, base: str) -> str:
|
||||
raw = field_text(item, feed["post"]["fields"].get("id"))
|
||||
return absolute_id(raw, base) if raw else self.post_link(feed, item, base)
|
||||
@@ -293,7 +311,9 @@ class Collector:
|
||||
except LoginRequired:
|
||||
raise
|
||||
except CollectError as exc:
|
||||
self.errors.append(f"detail {link}: {exc}")
|
||||
self.warnings.append(f"detail {link}: {exc}; the post is imported with its listing text")
|
||||
identifier = self.post_identifier(feed, item, base)
|
||||
self.notice("posts", uid_for(identifier) if identifier else "", "article_unreachable", strip_tags(field_text(item, feed["post"]["fields"].get("title"))), f"{link}: {exc}")
|
||||
return found
|
||||
|
||||
def add_post(self, feed: dict, item, base: str, detail: Node | None = None) -> str | None:
|
||||
@@ -308,8 +328,11 @@ class Collector:
|
||||
full = strip_tags(field_text(detail, feed["post"]["detail"]["content"]))
|
||||
if len(full) > len(content):
|
||||
content = full
|
||||
if feed["post"].get("source_footer", True) and link:
|
||||
content = f"{content}\n\nSource: {link}".strip()
|
||||
footer = f"\n\nSource: {link}" if feed["post"].get("source_footer", True) and link else ""
|
||||
room = max(MAX_POST_LENGTH - len(footer), 0)
|
||||
cut = (len(content) - room) if len(content) > room else 0
|
||||
title = title[:MAX_TITLE_LENGTH]
|
||||
content = f"{content[:room]}{footer}".strip()
|
||||
if len(content) < MIN_POST_LENGTH:
|
||||
content = f"{title}\n\n{content}".strip()
|
||||
if len(content) < MIN_POST_LENGTH:
|
||||
@@ -317,6 +340,13 @@ class Collector:
|
||||
created = normalize_instant(field_text(item, fields.get("created_at"))) or iso_now()
|
||||
updated = normalize_instant(field_text(item, fields.get("updated_at")))
|
||||
post_uid = uid_for(identifier)
|
||||
key = canonical_link(link) if link else ""
|
||||
owner = self.link_owner.get(key) if key else None
|
||||
if owner and owner[1] != feed["id"] and owner[0] != post_uid:
|
||||
self.aliases[post_uid] = owner[0]
|
||||
return owner[0]
|
||||
if key:
|
||||
self.link_owner.setdefault(key, (post_uid, feed["id"]))
|
||||
record = {
|
||||
"origin_uid": post_uid,
|
||||
"author_uid": self.add_author(feed.get("author") or {}, item, created),
|
||||
@@ -327,6 +357,10 @@ class Collector:
|
||||
if updated:
|
||||
record["updated_at"] = updated
|
||||
self.posts[post_uid] = record
|
||||
if link:
|
||||
self.sources[post_uid] = link
|
||||
if cut:
|
||||
self.notice("posts", post_uid, "truncated", title, f"{cut} characters over the {MAX_POST_LENGTH} limit were cut")
|
||||
return post_uid
|
||||
|
||||
def add_author(self, spec: dict, item, moment: str) -> str:
|
||||
@@ -353,7 +387,8 @@ class Collector:
|
||||
for item, base in [*fresh, *recent]:
|
||||
identifier = self.post_identifier(feed, item, base)
|
||||
if identifier:
|
||||
targets.setdefault(uid_for(identifier), (item, base, identifier))
|
||||
own = uid_for(identifier)
|
||||
targets.setdefault(self.aliases.get(own, own), (item, base, identifier))
|
||||
for post_uid, (item, base, identifier) in list(targets.items())[: int(self.limits["max_comment_posts"])]:
|
||||
try:
|
||||
if mode == "inline":
|
||||
@@ -364,10 +399,9 @@ class Collector:
|
||||
entries = flatten_matches(select(document, spec.get("list_path") or "$")) if document is not None else []
|
||||
self.add_comments(spec, entries, post_uid, None, 0)
|
||||
elif mode == "page":
|
||||
link = self.post_link(feed, item, base)
|
||||
link = self.discussion_link(feed, item, base)
|
||||
if link:
|
||||
root = await self.page_root(link, bool(feed.get("render") or spec.get("render")))
|
||||
self.add_page_comments(spec, root, post_uid)
|
||||
await self.collect_comment_pages(feed, spec, link, post_uid)
|
||||
except PageMissing:
|
||||
continue
|
||||
except LoginRequired:
|
||||
@@ -375,6 +409,20 @@ class Collector:
|
||||
except CollectError as exc:
|
||||
self.errors.append(f"comments {identifier}: {exc}")
|
||||
|
||||
async def collect_comment_pages(self, feed: dict, spec: dict, link: str, post_uid: str) -> None:
|
||||
render = bool(feed.get("render") or spec.get("render"))
|
||||
visited: set[str] = set()
|
||||
for _ in range(int(self.limits["max_comment_pages"])):
|
||||
if link in visited:
|
||||
break
|
||||
visited.add(link)
|
||||
root = await self.page_root(link, render)
|
||||
self.add_page_comments(spec, root, post_uid)
|
||||
target = next((value for value in (html_field(root, field) for field in NEXT_PAGE_FIELDS) if value), "")
|
||||
if not target:
|
||||
break
|
||||
link = urljoin(link, target)
|
||||
|
||||
def add_comments(self, spec: dict, entries: list, post_uid: str, parent_uid: str | None, depth: int) -> None:
|
||||
fields = spec["fields"]
|
||||
for entry in entries:
|
||||
@@ -425,7 +473,7 @@ class Collector:
|
||||
"origin_uid": comment_uid,
|
||||
"author_uid": self.add_author(spec.get("author", {}), entry, created),
|
||||
"post_uid": post_uid,
|
||||
"content": content,
|
||||
"content": content[:MAX_COMMENT_LENGTH],
|
||||
"created_at": created,
|
||||
}
|
||||
if parent and parent != comment_uid:
|
||||
@@ -483,5 +531,8 @@ class Collector:
|
||||
"state": self.state,
|
||||
"feeds": reports,
|
||||
"errors": self.errors,
|
||||
"warnings": self.warnings,
|
||||
"notices": self.notices,
|
||||
"sources": self.sources,
|
||||
"ok": bool(reports) and len(failed) < len(reports),
|
||||
}
|
||||
|
||||
@@ -29,6 +29,7 @@ RUN_OK = "ok"
|
||||
RUN_PARTIAL = "partial"
|
||||
RUN_FAILED = "failed"
|
||||
RUN_DRY = "dry_run"
|
||||
RUN_SKIPPED = "skipped"
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
RSEARCH_URL = "https://rsearch.app.molodetz.nl/search"
|
||||
@@ -45,6 +46,7 @@ class CommunityLimits:
|
||||
max_pages: int = 3
|
||||
max_posts: int = 150
|
||||
max_comment_posts: int = 25
|
||||
max_comment_pages: int = 3
|
||||
comment_recheck: int = 10
|
||||
delay_seconds: float = 1.0
|
||||
retries: int = 3
|
||||
@@ -57,6 +59,7 @@ class CommunityLimits:
|
||||
max_failures: int = 5
|
||||
max_communities_per_run: int = 3
|
||||
runs_kept: int = 30
|
||||
ledger_runs_kept: int = 10
|
||||
batch_records: int = 50
|
||||
min_field_coverage: float = 0.6
|
||||
|
||||
@@ -65,6 +68,7 @@ class CommunityLimits:
|
||||
"max_pages": self.max_pages,
|
||||
"max_posts": self.max_posts,
|
||||
"max_comment_posts": self.max_comment_posts,
|
||||
"max_comment_pages": self.max_comment_pages,
|
||||
"comment_recheck": self.comment_recheck,
|
||||
"delay_seconds": self.delay_seconds,
|
||||
"retries": self.retries,
|
||||
|
||||
@@ -5,7 +5,7 @@ from __future__ import annotations
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from devplacepy.database import get_table
|
||||
from devplacepy.database import db, get_table
|
||||
from devplacepy.database.soft_delete import soft_delete
|
||||
from devplacepy.services.community.spec import (
|
||||
LIMITS,
|
||||
@@ -18,7 +18,9 @@ from devplacepy.services.sync.selection import jittered
|
||||
from devplacepy.utils.text import generate_uid
|
||||
|
||||
JSON_COLUMNS = ("schema", "watermark", "totals", "sync_options")
|
||||
RUN_JSON_COLUMNS = ("collected", "pushed", "errors")
|
||||
RUN_JSON_COLUMNS = ("collected", "pushed", "errors", "tally")
|
||||
RUN_SUMMARY_COLUMNS = "uid, community_uid, started_at, finished_at, status, dry_run, trigger, pushed, tally, verdict, headline, items_stored, items_dropped"
|
||||
ITEMS_TABLE = "community_run_items"
|
||||
EDITABLE = (
|
||||
"name",
|
||||
"description",
|
||||
@@ -174,6 +176,19 @@ class CommunityStore:
|
||||
rows = get_table("community_runs").find(community_uid=community_uid, order_by=["-started_at"], _limit=limit)
|
||||
return [decode(row, RUN_JSON_COLUMNS) for row in rows]
|
||||
|
||||
def run_summaries(self, community_uid: str, limit: int = LIMITS.runs_kept) -> list[dict]:
|
||||
if "community_runs" not in db.tables:
|
||||
return []
|
||||
rows = db.query(
|
||||
f"SELECT {RUN_SUMMARY_COLUMNS} FROM community_runs WHERE community_uid = :uid ORDER BY started_at DESC LIMIT :limit",
|
||||
uid=community_uid,
|
||||
limit=limit,
|
||||
)
|
||||
return [decode(row, ("tally", "pushed")) for row in rows]
|
||||
|
||||
def run(self, run_uid: str) -> dict | None:
|
||||
return decode(get_table("community_runs").find_one(uid=run_uid), RUN_JSON_COLUMNS)
|
||||
|
||||
def busy(self, community_uid: str) -> bool:
|
||||
horizon = iso(now() - timedelta(seconds=LIMITS.run_timeout_seconds))
|
||||
rows = get_table("community_runs").find(community_uid=community_uid, status="running")
|
||||
@@ -184,7 +199,10 @@ class CommunityStore:
|
||||
|
||||
def prune_runs(self, community_uid: str) -> int:
|
||||
table = get_table("community_runs")
|
||||
stale = list(table.find(community_uid=community_uid, order_by=["-started_at"]))[LIMITS.runs_kept :]
|
||||
ranked = list(table.find(community_uid=community_uid, order_by=["-started_at"]))
|
||||
for row in ranked[LIMITS.ledger_runs_kept :]:
|
||||
get_table(ITEMS_TABLE).delete(run_uid=row["uid"])
|
||||
stale = ranked[LIMITS.runs_kept :]
|
||||
for row in stale:
|
||||
table.delete(uid=row["uid"])
|
||||
return len(stale)
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from ..spec import Action
|
||||
from ._shared import body, confirm, path
|
||||
from ._shared import body, confirm, path, query
|
||||
|
||||
SLUG = path("slug", "Community slug, as returned by community_list.")
|
||||
ACCOUNT = (
|
||||
@@ -38,6 +38,29 @@ COMMUNITY_ACTIONS: tuple[Action, ...] = (
|
||||
params=(SLUG,),
|
||||
requires_admin=True,
|
||||
),
|
||||
Action(
|
||||
name="community_ledger",
|
||||
method="GET",
|
||||
path="/admin/communities/{slug}/ledger",
|
||||
summary="Show exactly what a community's sync did, record by record, and why (admin only)",
|
||||
description=(
|
||||
"Without run: the most recent records across every run, problems first. With run (a run uid from "
|
||||
"community_get): that run's verdict, exact counts per outcome (created, updated, unchanged, stale, "
|
||||
"skipped, warning, rejected, failed, error), the errors, and a paginated list of records with the "
|
||||
"reason each was not synced in plain words. Filter by kind (users, posts, comments, feed), outcome, "
|
||||
"reason and a search text. Use it to answer whether a sync is alright and why a record is missing."
|
||||
),
|
||||
params=(
|
||||
SLUG,
|
||||
query("run", "Run uid from community_get. Leave empty for all recent runs."),
|
||||
query("kind", "users, posts, comments or feed."),
|
||||
query("outcome", "created, updated, unchanged, stale, skipped, warning, rejected, failed or error."),
|
||||
query("reason", "A reason key as listed in options.reason, for example content_refused or sampled_out."),
|
||||
query("q", "Search the title, name, origin id and detail."),
|
||||
query("page", "Page number, 50 records per page."),
|
||||
),
|
||||
requires_admin=True,
|
||||
),
|
||||
Action(
|
||||
name="community_add",
|
||||
method="POST",
|
||||
|
||||
@@ -13,7 +13,8 @@ This file documents community sync (`devplacepy/services/sync/`, `devplacepy/rou
|
||||
| Routes | `routers/sync/` | built, tested |
|
||||
| Devii tools `sync_users`, `sync_posts`, `sync_comments` | `services/devii/actions/catalog/sync.py` | built, `requires_admin`; Devii sends forms, so `records` is a JSON array string (`SyncBatchForm.decode_records`) |
|
||||
| Admin settings fields `sync_enabled`, `sync_origins` | `AdminSettingsForm`, `admin_settings.html`, `admin_save_settings` | built, origins normalized on save |
|
||||
| Scheduled sync service, export endpoint, vote and reaction sync | none | not built, see Gaps |
|
||||
| Scheduled sender | `services/community/` (`CommunityService`, see its CLAUDE.md) | built, opt-in, dry run by default |
|
||||
| Export endpoint, vote and reaction sync | none | not built, see Gaps |
|
||||
|
||||
## Model
|
||||
|
||||
@@ -53,10 +54,10 @@ A stateful fuzz (3000 random upserts, tombstones and edits over two origins and
|
||||
|
||||
## Gaps
|
||||
|
||||
- **No export endpoint and no scheduled sync service.** The sender today is a script reading the source community's own API. The service with one job per community is the next step; it should keep a per-origin watermark on `updated_at` and call these endpoints in the order users, posts, comments.
|
||||
- **No export endpoint.** Sync is inbound only. The scheduled sender is `CommunityService` (`services/community/`), which keeps a per-community watermark and pushes users, posts, comments in that order.
|
||||
- **Votes, reactions, follows, attachments, polls and project links are not synced.** Counters will not match the origin.
|
||||
- **A comment's post and parent must be sent in an earlier batch or earlier in the same batch.** There is no deferred resolution queue; the result reason tells the sender what to send first.
|
||||
- **Privacy disclosure.** Synced profile data (username, bio, location, links) comes from a third party. `/docs/privacy.html` has not been updated for it.
|
||||
- **Privacy disclosure.** Synced profile data is disclosed in `/docs/privacy.html` (data table row "Imported community content").
|
||||
- A user's own deletion on this side is final, but nothing tells the origin; the next sync of that user is `rejected` / `purged_locally` forever.
|
||||
|
||||
## Selection, identity aliasing and translation (community runs only)
|
||||
|
||||
@@ -40,6 +40,7 @@ class ItemSkipped(Exception):
|
||||
class AliasOutcome:
|
||||
records: dict[str, list[dict]]
|
||||
skipped: Counter = field(default_factory=Counter)
|
||||
dropped: list = field(default_factory=list)
|
||||
substitutions: int = 0
|
||||
translated: int = 0
|
||||
identities_created: int = 0
|
||||
@@ -76,7 +77,7 @@ class Aliaser:
|
||||
users = {str(user.get("origin_uid")): user for user in records.get("users", [])}
|
||||
kept_posts = set()
|
||||
for post in records.get("posts", []):
|
||||
kept = await self.process(post, users, ("title", "content"))
|
||||
kept = await self.process(post, users, ("title", "content"), "posts")
|
||||
if kept:
|
||||
self.outcome.records["posts"].append(kept)
|
||||
kept_posts.add(str(kept.get("origin_uid")))
|
||||
@@ -85,8 +86,9 @@ class Aliaser:
|
||||
parent = comment.get("parent_uid")
|
||||
if str(comment.get("post_uid")) not in kept_posts or (parent and str(parent) not in placed):
|
||||
self.outcome.skipped[SKIP_PARENT] += 1
|
||||
self.outcome.dropped.append(("comments", comment, SKIP_PARENT))
|
||||
continue
|
||||
kept = await self.process(comment, users, ("content",))
|
||||
kept = await self.process(comment, users, ("content",), "comments")
|
||||
if kept:
|
||||
self.outcome.records["comments"].append(kept)
|
||||
placed.add(str(comment.get("origin_uid")))
|
||||
@@ -113,14 +115,16 @@ class Aliaser:
|
||||
for name in names.values():
|
||||
await self.resolver.ensure(name)
|
||||
|
||||
async def process(self, record: dict, users: dict[str, dict], fields: tuple[str, ...]) -> dict | None:
|
||||
async def process(self, record: dict, users: dict[str, dict], fields: tuple[str, ...], kind: str = "posts") -> dict | None:
|
||||
try:
|
||||
return await self.transform(record, users, fields)
|
||||
except ItemSkipped as skipped:
|
||||
self.outcome.skipped[skipped.reason] += 1
|
||||
self.outcome.dropped.append((kind, record, skipped.reason))
|
||||
logger.info("skipped %s %s: %s", self.origin, record.get("origin_uid"), skipped.reason)
|
||||
except Exception:
|
||||
self.outcome.skipped[SKIP_REWRITE] += 1
|
||||
self.outcome.dropped.append((kind, record, SKIP_REWRITE))
|
||||
logger.exception("fail closed: %s %s skipped", self.origin, record.get("origin_uid"))
|
||||
return None
|
||||
|
||||
|
||||
@@ -35,6 +35,7 @@ class Selector:
|
||||
def __init__(self, origin: str, options: SyncOptions):
|
||||
self.origin = origin
|
||||
self.options = options
|
||||
self.dropped: list[tuple[str, dict, str]] = []
|
||||
|
||||
def select(self, records: dict[str, list[dict]]) -> tuple[dict[str, list[dict]], Counter]:
|
||||
skipped: Counter = Counter()
|
||||
@@ -50,12 +51,14 @@ class Selector:
|
||||
if reason:
|
||||
skipped[reason] += 1
|
||||
dropped.add(uid)
|
||||
self.dropped.append(("posts", post, reason))
|
||||
else:
|
||||
kept.append(post)
|
||||
comments = []
|
||||
for comment in records.get("comments", []):
|
||||
if str(comment.get("post_uid")) in dropped:
|
||||
skipped[SKIP_PARENT] += 1
|
||||
self.dropped.append(("comments", comment, SKIP_PARENT))
|
||||
else:
|
||||
comments.append(comment)
|
||||
return {**records, "posts": kept, "comments": comments}, skipped
|
||||
|
||||
@@ -10,11 +10,15 @@
|
||||
|
||||
.community-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(0, 1fr));
|
||||
grid-template-columns: repeat(auto-fit, minmax(min(100%, 18rem), 1fr));
|
||||
gap: var(--space-md);
|
||||
margin-bottom: var(--space-md);
|
||||
}
|
||||
|
||||
.community-grid code {
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.community-panel {
|
||||
min-width: 0;
|
||||
padding: var(--space-md);
|
||||
@@ -98,3 +102,343 @@
|
||||
display: block;
|
||||
margin-top: var(--space-sm);
|
||||
}
|
||||
|
||||
.ledger-intro {
|
||||
margin-bottom: var(--space-md);
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.ledger-verdict {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
align-items: baseline;
|
||||
gap: var(--space-xs) var(--space-md);
|
||||
margin-bottom: var(--space-md);
|
||||
padding: var(--space-md) var(--space-lg);
|
||||
border: 1px solid var(--border-light);
|
||||
border-left-width: 4px;
|
||||
border-radius: var(--radius-lg);
|
||||
background: var(--bg-card);
|
||||
}
|
||||
|
||||
.ledger-verdict-title {
|
||||
font-size: 1.125rem;
|
||||
}
|
||||
|
||||
.ledger-verdict-text {
|
||||
flex: 1 1 16rem;
|
||||
min-width: 0;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.ledger-verdict-meta {
|
||||
flex: 1 1 100%;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.ledger-verdict-ok {
|
||||
border-left-color: var(--success);
|
||||
background: rgba(var(--success-rgb), 0.08);
|
||||
}
|
||||
|
||||
.ledger-verdict-ok .ledger-verdict-title {
|
||||
color: var(--success);
|
||||
}
|
||||
|
||||
.ledger-verdict-attention,
|
||||
.ledger-verdict-idle {
|
||||
border-left-color: var(--warning);
|
||||
}
|
||||
|
||||
.ledger-verdict-attention {
|
||||
background: rgba(var(--warning-rgb), 0.08);
|
||||
}
|
||||
|
||||
.ledger-verdict-attention .ledger-verdict-title {
|
||||
color: var(--warning);
|
||||
}
|
||||
|
||||
.ledger-verdict-bad {
|
||||
border-left-color: var(--danger);
|
||||
background: rgba(var(--danger-rgb), 0.08);
|
||||
}
|
||||
|
||||
.ledger-verdict-bad .ledger-verdict-title {
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
.ledger-verdict-running {
|
||||
border-left-color: var(--info);
|
||||
background: rgba(var(--info-rgb), 0.08);
|
||||
}
|
||||
|
||||
.ledger-verdict-running .ledger-verdict-title {
|
||||
color: var(--info);
|
||||
}
|
||||
|
||||
.ledger-tiles {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fill, minmax(6.5rem, 1fr));
|
||||
gap: var(--space-sm);
|
||||
margin-bottom: var(--space-md);
|
||||
}
|
||||
|
||||
.ledger-tile {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: var(--space-xs);
|
||||
min-width: 0;
|
||||
padding: var(--space-sm) var(--space-md);
|
||||
border: 1px solid var(--border-light);
|
||||
border-radius: var(--radius);
|
||||
background: var(--bg-card);
|
||||
color: var(--text-primary);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.ledger-tile:hover,
|
||||
.ledger-tile:focus-visible {
|
||||
background: var(--bg-card-hover);
|
||||
}
|
||||
|
||||
.ledger-tile.is-active {
|
||||
border-color: var(--accent);
|
||||
}
|
||||
|
||||
.ledger-tile.is-zero {
|
||||
opacity: 0.55;
|
||||
}
|
||||
|
||||
.ledger-tile-count {
|
||||
font-size: 1.5rem;
|
||||
font-weight: 700;
|
||||
line-height: 1;
|
||||
}
|
||||
|
||||
.ledger-tile-label {
|
||||
color: var(--text-muted);
|
||||
font-size: 0.75rem;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.04em;
|
||||
}
|
||||
|
||||
.ledger-tile.ledger-tone-good .ledger-tile-count {
|
||||
color: var(--success);
|
||||
}
|
||||
|
||||
.ledger-tile.ledger-tone-warn .ledger-tile-count {
|
||||
color: var(--warning);
|
||||
}
|
||||
|
||||
.ledger-tile.ledger-tone-bad .ledger-tile-count {
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
.ledger-tile.ledger-tone-muted .ledger-tile-count {
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.ledger-filters {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: var(--space-sm);
|
||||
align-items: flex-end;
|
||||
margin-bottom: var(--space-md);
|
||||
}
|
||||
|
||||
.ledger-filters label {
|
||||
flex: 1 1 10rem;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.ledger-filters select,
|
||||
.ledger-filters input[type="text"] {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.ledger-filter-actions {
|
||||
display: flex;
|
||||
gap: var(--space-sm);
|
||||
}
|
||||
|
||||
.ledger-table td {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.ledger-table small {
|
||||
display: block;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.ledger-record {
|
||||
max-width: 24rem;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.ledger-label {
|
||||
display: block;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.ledger-id {
|
||||
font-family: var(--font-mono);
|
||||
}
|
||||
|
||||
.ledger-links {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: var(--space-sm);
|
||||
}
|
||||
|
||||
.ledger-retry {
|
||||
color: var(--info);
|
||||
}
|
||||
|
||||
.ledger-badge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
padding: 0.125rem 0.5rem;
|
||||
border-radius: 999px;
|
||||
font-size: 0.6875rem;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.03em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.ledger-badge.ledger-tone-good {
|
||||
background: rgba(var(--success-rgb), 0.15);
|
||||
color: var(--success);
|
||||
}
|
||||
|
||||
.ledger-badge.ledger-tone-warn {
|
||||
background: rgba(var(--warning-rgb), 0.15);
|
||||
color: var(--warning);
|
||||
}
|
||||
|
||||
.ledger-badge.ledger-tone-bad {
|
||||
background: rgba(var(--danger-rgb), 0.15);
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
.ledger-badge.ledger-tone-muted {
|
||||
background: var(--bg-card-hover);
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.ledger-row.ledger-tone-bad {
|
||||
box-shadow: inset 3px 0 0 var(--danger);
|
||||
}
|
||||
|
||||
.ledger-row.ledger-tone-warn {
|
||||
box-shadow: inset 3px 0 0 var(--warning);
|
||||
}
|
||||
|
||||
.ledger-log {
|
||||
margin-top: var(--space-md);
|
||||
}
|
||||
|
||||
.ledger-runs td {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.ledger-chips {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: var(--space-xs);
|
||||
}
|
||||
|
||||
.ledger-verdict-chip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
padding: 0.125rem 0.5rem;
|
||||
border-radius: 999px;
|
||||
font-size: 0.6875rem;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.03em;
|
||||
background: var(--bg-card-hover);
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.ledger-verdict-chip.ledger-verdict-ok {
|
||||
background: rgba(var(--success-rgb), 0.15);
|
||||
color: var(--success);
|
||||
}
|
||||
|
||||
.ledger-verdict-chip.ledger-verdict-attention {
|
||||
background: rgba(var(--warning-rgb), 0.15);
|
||||
color: var(--warning);
|
||||
}
|
||||
|
||||
.ledger-verdict-chip.ledger-verdict-bad {
|
||||
background: rgba(var(--danger-rgb), 0.15);
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
.ledger-verdict-chip.ledger-verdict-running {
|
||||
background: rgba(var(--info-rgb), 0.15);
|
||||
color: var(--info);
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.ledger-table thead {
|
||||
position: absolute;
|
||||
width: 1px;
|
||||
height: 1px;
|
||||
overflow: hidden;
|
||||
clip: rect(0 0 0 0);
|
||||
}
|
||||
|
||||
.ledger-table,
|
||||
.ledger-table tbody,
|
||||
.ledger-table tr,
|
||||
.ledger-table td {
|
||||
display: block;
|
||||
width: 100%;
|
||||
max-width: none;
|
||||
}
|
||||
|
||||
.ledger-table tr {
|
||||
margin-bottom: var(--space-sm);
|
||||
padding: var(--space-sm);
|
||||
border: 1px solid var(--border-light);
|
||||
border-radius: var(--radius);
|
||||
}
|
||||
|
||||
.ledger-table td {
|
||||
display: grid;
|
||||
grid-template-columns: 5rem minmax(0, 1fr);
|
||||
gap: var(--space-sm);
|
||||
border: 0;
|
||||
padding: var(--space-xs) 0;
|
||||
}
|
||||
|
||||
.ledger-table td::before {
|
||||
content: attr(data-label);
|
||||
color: var(--text-muted);
|
||||
font-size: 0.75rem;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.ledger-table td.admin-empty {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.ledger-table td.admin-empty::before {
|
||||
content: none;
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 480px) {
|
||||
.ledger-tiles {
|
||||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||||
}
|
||||
|
||||
.ledger-filter-actions {
|
||||
flex: 1 1 100%;
|
||||
}
|
||||
}
|
||||
|
||||
.ledger-runs td[data-label="Mode"] {
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
@@ -30,6 +30,10 @@
|
||||
--warning: #fbbf24;
|
||||
--danger: #f87171;
|
||||
--info: #60a5fa;
|
||||
--success-rgb: 52, 211, 153;
|
||||
--warning-rgb: 251, 191, 36;
|
||||
--danger-rgb: 248, 113, 113;
|
||||
--info-rgb: 96, 165, 250;
|
||||
--on-warning: #1c1200;
|
||||
|
||||
--radius: 8px;
|
||||
|
||||
@@ -136,18 +136,42 @@
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h3>Recent runs</h3>
|
||||
{% for run in runs %}
|
||||
<details class="community-panel">
|
||||
<summary>
|
||||
{{ local_dt(run['started_at']) }} - {{ run['status'] }}{% if run['dry_run'] %} (dry run){% endif %}
|
||||
<small class="admin-text-muted">{{ run['trigger'] }}</small>
|
||||
</summary>
|
||||
<p class="admin-text-muted">collected {{ run['collected'] }} / pushed {{ run['pushed'] }}</p>
|
||||
{% for error in run['errors'] %}<p class="community-banner">{{ error }}</p>{% endfor %}
|
||||
<pre class="community-log">{{ run['log'] }}</pre>
|
||||
</details>
|
||||
{% else %}
|
||||
<p class="admin-empty">No runs yet.</p>
|
||||
{% endfor %}
|
||||
<div class="admin-toolbar">
|
||||
<h3>Recent runs</h3>
|
||||
<a href="/admin/communities/{{ community['slug'] }}/ledger" class="admin-btn admin-btn-sm">Everything the sync did</a>
|
||||
</div>
|
||||
<div class="admin-table-wrap">
|
||||
<table class="admin-table ledger-table ledger-runs">
|
||||
<caption class="sr-only">The most recent runs of this community, newest first</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">Result</th>
|
||||
<th scope="col">Started</th>
|
||||
<th scope="col">What happened</th>
|
||||
<th scope="col">Mode</th>
|
||||
<th scope="col"><span class="sr-only">Open</span></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for run in runs %}
|
||||
<tr>
|
||||
<td data-label="Result">
|
||||
<span class="ledger-verdict-chip ledger-verdict-{{ run['verdict'] }}">{{ run['status'] }}</span>
|
||||
<small class="admin-text-muted">{{ run['headline'] }}</small>
|
||||
</td>
|
||||
<td data-label="Started">{{ local_dt(run['started_at']) }}{% if run['duration_seconds'] is not none %}<small class="admin-text-muted">{{ run['duration_seconds'] }}s</small>{% endif %}</td>
|
||||
<td data-label="What happened">
|
||||
<span class="ledger-chips">
|
||||
{% for key, count in run['tally'].items() %}<span class="ledger-badge ledger-tone-{{ {'created': 'good', 'updated': 'good', 'unchanged': 'muted', 'stale': 'muted', 'skipped': 'warn', 'warning': 'warn'}.get(key, 'bad') }}">{{ count }} {{ key }}</span>{% else %}<span class="admin-text-muted">nothing</span>{% endfor %}
|
||||
</span>
|
||||
</td>
|
||||
<td data-label="Mode">{% if run['dry_run'] %}dry run{% else %}live{% endif %}<small class="admin-text-muted">{{ run['trigger'] }}</small></td>
|
||||
<td data-label="Open"><a href="/admin/communities/{{ community['slug'] }}/ledger?run={{ run['uid'] }}" class="admin-btn admin-btn-sm" aria-label="Open the log of the run started {{ run['started_at'] }}">Open</a></td>
|
||||
</tr>
|
||||
{% else %}
|
||||
<tr><td colspan="5" class="admin-empty">No runs yet.</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% endblock %}
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
{% extends "admin_base.html" %}
|
||||
{% block extra_head %}
|
||||
{{ super() }}
|
||||
<link rel="stylesheet" href="{{ static_url('/static/css/communities.css') }}">
|
||||
{% endblock %}
|
||||
{% block admin_content %}
|
||||
{% set slug = community['slug'] %}
|
||||
{% set base_path = '/admin/communities/' ~ slug ~ '/ledger' %}
|
||||
{% set run_param = ('run=' ~ run['uid'] ~ '&') if run else '' %}
|
||||
<div class="admin-toolbar">
|
||||
<h2>{{ community['name'] or slug }} <small class="admin-text-muted">{% if run %}run log{% else %}everything the sync did{% endif %}</small></h2>
|
||||
<a href="/admin/communities/{{ slug }}" class="admin-btn admin-btn-sm">Back to community</a>
|
||||
</div>
|
||||
|
||||
{% if run %}
|
||||
<section class="ledger-verdict ledger-verdict-{{ verdict['level'] }}" role="status" aria-label="Run result">
|
||||
<strong class="ledger-verdict-title">{{ verdict['title'] }}</strong>
|
||||
<span class="ledger-verdict-text">{{ verdict['headline'] }}</span>
|
||||
<small class="ledger-verdict-meta">
|
||||
{{ local_dt(run['started_at']) }}
|
||||
· {{ run['trigger'] }}
|
||||
· {% if run['dry_run'] %}dry run{% else %}live{% endif %}
|
||||
{% if run['duration_seconds'] is not none %}· {{ run['duration_seconds'] }}s{% endif %}
|
||||
· {{ run['status'] }}
|
||||
</small>
|
||||
</section>
|
||||
{% else %}
|
||||
<p class="ledger-intro">The most recent records across every run, problems first. Open a run from the community page to see exactly what that run did.</p>
|
||||
{% endif %}
|
||||
|
||||
<div class="ledger-tiles" role="group" aria-label="Outcomes">
|
||||
<a class="ledger-tile ledger-tile-all {% if not filters.get('outcome') %}is-active{% endif %}" href="{{ base_path }}{% if run %}?run={{ run['uid'] }}{% endif %}">
|
||||
<span class="ledger-tile-count">{{ tiles|sum(attribute='listed') }}</span>
|
||||
<span class="ledger-tile-label">All listed</span>
|
||||
</a>
|
||||
{% for tile in tiles %}
|
||||
<a class="ledger-tile ledger-tone-{{ tile['tone'] }} {% if tile['count'] == 0 %}is-zero{% endif %} {% if filters.get('outcome') == tile['key'] %}is-active{% endif %}"
|
||||
href="{{ base_path }}?{{ run_param }}outcome={{ tile['key'] }}" title="{{ tile['help'] }}">
|
||||
<span class="ledger-tile-count">{{ tile['count'] }}</span>
|
||||
<span class="ledger-tile-label">{{ tile['label'] }}</span>
|
||||
</a>
|
||||
{% endfor %}
|
||||
</div>
|
||||
|
||||
{% if run and run['items_dropped'] %}
|
||||
<p class="community-banner" role="status">{{ run['items_dropped'] }} more records are not listed below: the list keeps at most a fixed number per run. The counts above are exact.</p>
|
||||
{% endif %}
|
||||
{% for error in run_errors %}
|
||||
<p class="community-banner" role="alert">{{ error }}</p>
|
||||
{% endfor %}
|
||||
|
||||
<form method="get" action="{{ base_path }}" class="ledger-filters" role="search" aria-label="Filter the sync log">
|
||||
{% if run %}<input type="hidden" name="run" value="{{ run['uid'] }}">{% endif %}
|
||||
<label for="ledger-kind">Kind
|
||||
<select id="ledger-kind" name="kind" class="admin-settings-select">
|
||||
<option value="">All</option>
|
||||
{% for value in options['kind'] %}<option value="{{ value }}" {% if filters.get('kind') == value %}selected{% endif %}>{{ value }}</option>{% endfor %}
|
||||
</select>
|
||||
</label>
|
||||
<label for="ledger-outcome">Result
|
||||
<select id="ledger-outcome" name="outcome" class="admin-settings-select">
|
||||
<option value="">All</option>
|
||||
{% for option in options['outcome'] %}<option value="{{ option['value'] }}" {% if filters.get('outcome') == option['value'] %}selected{% endif %}>{{ option['label'] }}</option>{% endfor %}
|
||||
</select>
|
||||
</label>
|
||||
<label for="ledger-reason">Why
|
||||
<select id="ledger-reason" name="reason" class="admin-settings-select">
|
||||
<option value="">Any reason</option>
|
||||
{% for option in options['reason'] %}<option value="{{ option['value'] }}" {% if filters.get('reason') == option['value'] %}selected{% endif %}>{{ option['title'] }}</option>{% endfor %}
|
||||
</select>
|
||||
</label>
|
||||
<label for="ledger-q">Search
|
||||
<input type="text" id="ledger-q" name="q" maxlength="80" value="{{ filters.get('q', '') }}" placeholder="title, name, id, detail">
|
||||
</label>
|
||||
<div class="ledger-filter-actions">
|
||||
<button type="submit" class="admin-btn admin-btn-sm admin-btn-primary">Filter</button>
|
||||
<a href="{{ base_path }}{% if run %}?run={{ run['uid'] }}{% endif %}" class="admin-btn admin-btn-sm">Clear</a>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
<div class="admin-table-wrap">
|
||||
<table class="admin-table ledger-table">
|
||||
<caption class="sr-only">Records the sync handled, problems first</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">Result</th>
|
||||
<th scope="col">Kind</th>
|
||||
<th scope="col">Record</th>
|
||||
<th scope="col">Why</th>
|
||||
<th scope="col">Detail</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for item in items %}
|
||||
<tr class="ledger-row ledger-tone-{{ item['tone'] }}">
|
||||
<td data-label="Result"><span class="ledger-badge ledger-tone-{{ item['tone'] }}">{{ item['outcome'] }}</span></td>
|
||||
<td data-label="Kind">{{ item['kind'] }}</td>
|
||||
<td data-label="Record" class="ledger-record">
|
||||
<span class="ledger-label">{{ item['label'] or '(no title)' }}</span>
|
||||
<small class="admin-text-muted ledger-id">{{ item['origin_uid'] }}</small>
|
||||
<span class="ledger-links">
|
||||
{% if item['local_url'] %}<a href="{{ item['local_url'] }}">Open here</a>{% endif %}
|
||||
{% if item['source_url'] %}<a href="{{ item['source_url'] }}" rel="noopener noreferrer" target="_blank">Open at source</a>{% endif %}
|
||||
</span>
|
||||
</td>
|
||||
<td data-label="Why">
|
||||
{% if item['reason'] %}
|
||||
<strong>{{ item['reason_title'] }}</strong>
|
||||
<small class="admin-text-muted">{{ item['reason_text'] }}</small>
|
||||
{% if item['retried'] %}<small class="ledger-retry">Retried on the next run.</small>{% endif %}
|
||||
{% else %}<span class="admin-text-muted">-</span>{% endif %}
|
||||
</td>
|
||||
<td data-label="Detail"><small class="ledger-detail">{{ item['detail'] or '-' }}</small></td>
|
||||
</tr>
|
||||
{% else %}
|
||||
<tr><td colspan="5" class="admin-empty">{% if filters %}Nothing matches these filters.{% elif run and run['items_stored'] %}The record list of this run was removed: only the most recent runs keep it. The counts above and the log below are kept.{% elif run %}Nothing to list for this run: every record was unchanged, or nothing was collected.{% else %}Nothing has been synced yet.{% endif %}</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% if pagination['capped'] %}<p class="admin-text-muted">Showing the first {{ pagination['total'] }} matches. Narrow the filters to see the rest.</p>{% endif %}
|
||||
|
||||
{% include "_pagination.html" %}
|
||||
|
||||
{% if run_log %}
|
||||
<details class="community-panel ledger-log">
|
||||
<summary>Full run log</summary>
|
||||
<pre class="community-log">{{ run_log }}</pre>
|
||||
</details>
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
@@ -17,6 +17,7 @@ used, and how it is retained, deleted and shared.
|
||||
| Assistant | Prompts and results of assistant, search and analysis features you invoke | Deliver the feature and attribute its cost to you |
|
||||
| Moderation | Reports you file, reports filed about you, and decisions taken | Operate the safety controls the platform is required to have |
|
||||
| Donated bandwidth | Only if you donate: your device label and secret hash, a keyed hash of your IP address (never the address itself), an approximate country and network class, request and byte counters, and node events | Run the community node network and show you what you donated. See section 10. |
|
||||
| Imported community content | Where an administrator synchronises an external community: public usernames, bios, locations, links, posts and comments published there, as the origin made them public. No email address, password, vote count, XP or role is imported, and imported accounts cannot sign in. Imported rows may be reported and removed like any other content | Mirror a public community inside DevPlace. If you are the author of imported content, ask for its removal through a report or by contacting an administrator. |
|
||||
|
||||
DevPlace runs **no third-party analytics, no advertising and no cross-site or cross-app tracking**.
|
||||
There is no tracking identifier shared with any other company.
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "devplacepy"
|
||||
version = "1.1.2"
|
||||
version = "1.1.3"
|
||||
description = "DevPlace - The Developer Social Network"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
|
||||
@@ -143,8 +143,9 @@ def test_live_run_refused_when_sync_disabled(client, admin, fake_world):
|
||||
try:
|
||||
client.post("/admin/communities/forum-test/run", data={}, headers=admin["headers"])
|
||||
run = wait_for(lambda: next((item for item in store.runs(row["uid"]) if item["status"] != "running"), None))
|
||||
assert run["status"] == "failed"
|
||||
assert run["status"] == "skipped" and run["verdict"] == "attention"
|
||||
assert any("disabled" in error for error in run["errors"])
|
||||
assert store.get("forum-test")["failure_count"] == 0 and store.get("forum-test")["status"] == "ready"
|
||||
finally:
|
||||
set_setting("sync_enabled", "1")
|
||||
|
||||
@@ -168,6 +169,7 @@ GATED = [
|
||||
("get", "/admin/communities", {}),
|
||||
("post", "/admin/communities", {"query": "https://forum.test/api/posts"}),
|
||||
("get", "/admin/communities/forum-test", {}),
|
||||
("get", "/admin/communities/forum-test/ledger", {}),
|
||||
("post", "/admin/communities/forum-test/run", {}),
|
||||
("post", "/admin/communities/forum-test/relearn", {}),
|
||||
("post", "/admin/communities/forum-test/update", {"interval_seconds": "7200"}),
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from devplacepy.database import get_table, set_setting
|
||||
from devplacepy.services.community.ledger_store import ledger_store
|
||||
from devplacepy.services.community.store import store
|
||||
from tests.api.admin.communities.index import ( # noqa: F401
|
||||
add_ready,
|
||||
admin,
|
||||
client,
|
||||
fake_world,
|
||||
wait_for,
|
||||
)
|
||||
|
||||
LEDGER = "/admin/communities/forum-test/ledger"
|
||||
|
||||
|
||||
def finished_runs(row):
|
||||
return [item for item in store.runs(row["uid"]) if item["status"] != "running"]
|
||||
|
||||
|
||||
def live_run(client, admin, row, count=1):
|
||||
client.post("/admin/communities/forum-test/run", data={}, headers=admin["headers"])
|
||||
return wait_for(lambda: len(finished_runs(row)) >= count and store.runs(row["uid"])[0])
|
||||
|
||||
|
||||
def test_run_page_shows_verdict_exact_tiles_and_every_created_record(client, admin, fake_world):
|
||||
row = add_ready(client, admin)
|
||||
run = live_run(client, admin, row)
|
||||
assert run["verdict"] == "ok" and run["headline"] == "Everything synced."
|
||||
body = client.get(f"{LEDGER}?run={run['uid']}", headers=admin["headers"]).json()
|
||||
assert body["scope"] == "run" and body["verdict"]["title"] == "All good"
|
||||
tiles = {tile["key"]: tile for tile in body["tiles"]}
|
||||
assert tiles["created"]["count"] == tiles["created"]["listed"] == run["tally"]["created"] > 10
|
||||
assert tiles["rejected"]["count"] == 0 and tiles["failed"]["count"] == 0
|
||||
assert body["pagination"]["total"] == tiles["created"]["listed"]
|
||||
posts = [item for item in body["items"] if item["kind"] == "posts"]
|
||||
assert posts and all(item["outcome"] == "created" and item["label"] for item in posts)
|
||||
assert any(item["local_url"].startswith("/posts/") for item in posts)
|
||||
users = [item for item in body["items"] if item["kind"] == "users"]
|
||||
assert users and all(item["local_url"].startswith("/profile/") for item in users)
|
||||
|
||||
|
||||
def test_second_run_lists_nothing_and_says_nothing_is_new(client, admin, fake_world):
|
||||
row = add_ready(client, admin)
|
||||
live_run(client, admin, row)
|
||||
second = live_run(client, admin, row, count=2)
|
||||
assert second["verdict"] == "idle" and second["items_stored"] == 0
|
||||
body = client.get(f"{LEDGER}?run={second['uid']}", headers=admin["headers"]).json()
|
||||
assert body["items"] == [] and {tile["key"]: tile["count"] for tile in body["tiles"]}["unchanged"] > 0
|
||||
|
||||
|
||||
def test_kind_and_outcome_filters_narrow_the_list_and_are_echoed(client, admin, fake_world):
|
||||
row = add_ready(client, admin)
|
||||
run = live_run(client, admin, row)
|
||||
uid = run["uid"]
|
||||
assert ledger_store.counts(run_uid=uid)["created"] == run["tally"]["created"]
|
||||
body = client.get(f"{LEDGER}?run={uid}&outcome=created&kind=posts", headers=admin["headers"]).json()
|
||||
assert body["filters"] == {"kind": "posts", "outcome": "created"}
|
||||
assert all(item["kind"] == "posts" for item in body["items"])
|
||||
|
||||
|
||||
def test_filters_search_and_pagination_never_break(client, admin, fake_world):
|
||||
row = add_ready(client, admin)
|
||||
run = live_run(client, admin, row)
|
||||
uid = run["uid"]
|
||||
for query in (
|
||||
f"?run={uid}&q=%25",
|
||||
f"?run={uid}&q=_",
|
||||
f"?run={uid}&q=" + "x" * 500,
|
||||
f"?run={uid}&page=999999",
|
||||
f"?run={uid}&page=-4",
|
||||
f"?run={uid}&kind=nope&outcome=nope&reason=%27%3B--",
|
||||
"",
|
||||
"?page=999999&q=%25",
|
||||
):
|
||||
response = client.get(LEDGER + query, headers=admin["headers"])
|
||||
assert response.status_code == 200, (query, response.text[:200])
|
||||
assert response.json()["pagination"]["per_page"] == 50
|
||||
html = client.get(f"{LEDGER}?run={uid}", headers={"X-API-KEY": admin["headers"]["X-API-KEY"]})
|
||||
assert html.status_code == 200 and "All good" in html.text and "ledger-tile" in html.text
|
||||
|
||||
|
||||
def test_unknown_run_and_another_communitys_run_are_404(client, admin, fake_world):
|
||||
row = add_ready(client, admin)
|
||||
assert client.get(f"{LEDGER}?run=nope", headers=admin["headers"]).status_code == 404
|
||||
other = store.create("other-test", "other", "tester", name="other")
|
||||
foreign = store.start_run(other["uid"], "test", False)
|
||||
assert client.get(f"{LEDGER}?run={foreign}", headers=admin["headers"]).status_code == 404
|
||||
assert client.get("/admin/communities/missing/ledger", headers=admin["headers"]).status_code == 404
|
||||
assert row
|
||||
|
||||
|
||||
def test_community_page_lists_runs_without_loading_their_logs(client, admin, fake_world):
|
||||
row = add_ready(client, admin)
|
||||
live_run(client, admin, row)
|
||||
body = client.get("/admin/communities/forum-test", headers=admin["headers"]).json()
|
||||
assert body["runs"] and all("log" not in run for run in body["runs"])
|
||||
assert body["runs"][0]["verdict"] == "ok" and body["runs"][0]["tally"]["created"] > 0
|
||||
html = client.get("/admin/communities/forum-test", headers={"X-API-KEY": admin["headers"]["X-API-KEY"]})
|
||||
assert "Everything the sync did" in html.text and f"ledger?run={body['runs'][0]['uid']}" in html.text
|
||||
|
||||
|
||||
def test_a_run_from_before_the_ledger_still_shows_a_tally(client, admin, fake_world):
|
||||
row = add_ready(client, admin)
|
||||
legacy = store.start_run(row["uid"], "admin", False)
|
||||
store.finish_run(legacy, "ok", collected={"posts": 3}, pushed={"posts": {"created": 3}}, errors=[], log="old")
|
||||
runs = client.get("/admin/communities/forum-test", headers=admin["headers"]).json()["runs"]
|
||||
old = next(run for run in runs if run["uid"] == legacy)
|
||||
assert old["tally"] == {"created": 3} and old["verdict"] == "ok"
|
||||
page = client.get(f"{LEDGER}?run={legacy}", headers=admin["headers"])
|
||||
assert page.status_code == 200 and page.json()["run_log"] == "old"
|
||||
|
||||
|
||||
def test_the_ledger_is_gone_with_its_run(client, admin, fake_world):
|
||||
row = add_ready(client, admin)
|
||||
run = live_run(client, admin, row)
|
||||
assert get_table("community_run_items").count(run_uid=run["uid"]) > 0
|
||||
for _ in range(32):
|
||||
store.finish_run(store.start_run(row["uid"], "test", True), "dry_run")
|
||||
store.prune_runs(row["uid"])
|
||||
assert get_table("community_run_items").count(run_uid=run["uid"]) == 0
|
||||
set_setting("sync_enabled", "1")
|
||||
@@ -0,0 +1,121 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import copy
|
||||
|
||||
from tests.conftest import run_async
|
||||
|
||||
from devplacepy.services.community.discoverer import CommunityDiscoverer
|
||||
from devplacepy.services.community.learner import HtmlInferrer
|
||||
from devplacepy.services.community.runtime import MAX_POST_LENGTH, Collector
|
||||
from devplacepy.services.community.extract import parse_html, select_nodes
|
||||
from devplacepy.services.sync.records import RECORD_MODELS
|
||||
from tests.unit.services.community.fakes import (
|
||||
FakeFetcher,
|
||||
RSS,
|
||||
detail_page,
|
||||
json_community,
|
||||
no_sleep,
|
||||
)
|
||||
|
||||
LINKED_ITEM = """<item><title>Linked story {n} on testing</title><link>https://elsewhere.test/a/{n}</link>
|
||||
<guid>https://blog.test/s/{n}</guid><pubDate>Tue, {n:02d} Oct 2026 10:00:00 GMT</pubDate>
|
||||
<comments>https://blog.test/s/{n}</comments>
|
||||
<dc:creator>Alice</dc:creator><description><p>Body of story {n} with enough text.</p></description></item>"""
|
||||
|
||||
|
||||
def discussion_community() -> dict:
|
||||
items = "".join(LINKED_ITEM.format(n=n) for n in range(1, 7))
|
||||
routes = {
|
||||
"https://blog.test/": '<html><head><link rel="alternate" type="application/rss+xml" href="/feed.xml">'
|
||||
"<title>Blog Test</title></head><body>hi</body></html>",
|
||||
"https://blog.test/feed.xml": RSS.format(items=items),
|
||||
}
|
||||
for n in range(1, 7):
|
||||
routes[f"https://elsewhere.test/a/{n}"] = f"<html><body><article>{'External article text. ' * 20}</article></body></html>"
|
||||
routes[f"https://blog.test/s/{n}"] = detail_page("story", n)
|
||||
return routes
|
||||
|
||||
|
||||
def learn(routes: dict, query: str) -> dict:
|
||||
result = run_async(CommunityDiscoverer(FakeFetcher(routes)).discover(query))
|
||||
assert result.schema, result.error
|
||||
return result.schema
|
||||
|
||||
|
||||
def test_rss_comments_are_learned_from_the_discussion_link_not_the_article():
|
||||
routes = discussion_community()
|
||||
schema = learn(routes, "https://blog.test/")
|
||||
feed = schema["feeds"][0]
|
||||
assert feed["post"]["fields"]["comments_url"] == ["$.comments"]
|
||||
assert feed["comments"]["mode"] == "page"
|
||||
result = run_async(Collector(FakeFetcher(routes), schema, sleep=no_sleep).run())
|
||||
assert result["records"]["comments"]
|
||||
posts = {post["origin_uid"] for post in result["records"]["posts"]}
|
||||
assert {comment["post_uid"] for comment in result["records"]["comments"]} <= posts
|
||||
|
||||
|
||||
def test_post_body_still_comes_from_the_article_page():
|
||||
routes = discussion_community()
|
||||
schema = learn(routes, "https://blog.test/")
|
||||
result = run_async(Collector(FakeFetcher(routes), schema, sleep=no_sleep).run())
|
||||
assert all("External article text." in post["content"] for post in result["records"]["posts"])
|
||||
|
||||
|
||||
LISTING = """<div class="story"><a class="vote" href="/login">1</a>
|
||||
<a class="u-url" href="https://far.test/{n}">Story {n} about shells</a>
|
||||
<a class="discuss" href="/s/{n}">{n} comments</a><span class="byline">via user{n}</span>
|
||||
<time datetime="2026-10-{n:02d}T09:00:00Z">d</time><p>Excerpt of story {n} that is long enough to count.</p></div>"""
|
||||
|
||||
|
||||
def listing_nodes() -> list:
|
||||
root = parse_html("<html><body>" + "".join(LISTING.format(n=n) for n in range(1, 9)) + "</body></html>")
|
||||
return select_nodes(root, "div.story")
|
||||
|
||||
|
||||
def test_constant_first_anchor_is_never_chosen_as_identifier():
|
||||
fields = HtmlInferrer(listing_nodes(), "https://board.test/").post_fields()
|
||||
assert fields["id"] == ["a.u-url::attr(href)"]
|
||||
assert fields["title"] == ["a.u-url"]
|
||||
assert fields["comments_url"] == ["a.discuss::attr(href)"]
|
||||
|
||||
|
||||
def test_no_distinct_anchor_means_no_identifier():
|
||||
page = "".join('<div class="story"><a href="/login">x</a><p>Some text that is long enough here.</p></div>' for _ in range(6))
|
||||
nodes = select_nodes(parse_html(f"<html><body>{page}</body></html>"), "div.story")
|
||||
assert "id" not in HtmlInferrer(nodes, "https://board.test/").post_fields()
|
||||
|
||||
|
||||
def test_oversized_posts_and_titles_are_truncated_to_valid_sync_records():
|
||||
routes = json_community()
|
||||
schema = learn(routes, "https://forum.test/api/posts")
|
||||
routes["https://forum.test/api/posts"]["items"][0]["content_html"] = "<p>" + "x" * (MAX_POST_LENGTH * 2) + "</p>"
|
||||
routes["https://forum.test/api/posts"]["items"][0]["title"] = "t" * 900
|
||||
result = run_async(Collector(FakeFetcher(routes), schema, sleep=no_sleep).run())
|
||||
for post in result["records"]["posts"]:
|
||||
assert len(post["content"]) <= MAX_POST_LENGTH
|
||||
RECORD_MODELS["posts"].model_validate(post)
|
||||
assert any(len(post["content"]) == MAX_POST_LENGTH for post in result["records"]["posts"])
|
||||
|
||||
|
||||
def test_failed_article_fetch_is_a_warning_and_the_post_is_still_imported():
|
||||
routes = discussion_community()
|
||||
schema = learn(routes, "https://blog.test/")
|
||||
assert schema["feeds"][0]["post"].get("detail")
|
||||
routes["https://elsewhere.test/a/1"] = (403, "forbidden")
|
||||
result = run_async(Collector(FakeFetcher(routes), schema, sleep=no_sleep).run())
|
||||
assert result["errors"] == []
|
||||
assert any("elsewhere.test/a/1" in line for line in result["warnings"])
|
||||
assert len(result["records"]["posts"]) == 6
|
||||
|
||||
|
||||
def test_same_story_from_two_feeds_is_one_post_and_comments_follow_it():
|
||||
routes = discussion_community()
|
||||
schema = learn(routes, "https://blog.test/")
|
||||
second = copy.deepcopy(schema["feeds"][0])
|
||||
second["id"] = "feed2"
|
||||
second["post"]["fields"]["id"] = ["$.link"]
|
||||
schema["feeds"].append(second)
|
||||
result = run_async(Collector(FakeFetcher(routes), schema, sleep=no_sleep).run())
|
||||
assert len(result["records"]["posts"]) == 6
|
||||
posts = {post["origin_uid"] for post in result["records"]["posts"]}
|
||||
assert {comment["post_uid"] for comment in result["records"]["comments"]} <= posts
|
||||
@@ -166,3 +166,41 @@ def test_javascript_feed_is_collected_through_the_browser(tmp_path):
|
||||
module = load(CodeGenerator(schema).write(tmp_path))
|
||||
collected = run_async(module.collect_all(FakeFetcher(routes), sleep=no_sleep))
|
||||
assert len(collected["records"]["posts"]) == 6
|
||||
|
||||
|
||||
def post_comment_count(result: dict, post_origin_uid: str) -> int:
|
||||
return sum(1 for comment in result["records"]["comments"] if comment["post_uid"] == post_origin_uid)
|
||||
|
||||
|
||||
def paginated_board(next_href: str) -> dict:
|
||||
from tests.unit.services.community.fakes import detail_page
|
||||
|
||||
routes = html_community()
|
||||
first = detail_page("thread", 1).replace("</body>", f'<a rel="next" href="{next_href}">more</a></body>')
|
||||
routes["https://board.test/t/1"] = first
|
||||
routes["https://board.test/t/1?cp=2"] = detail_page("thread", 1, comments=4)
|
||||
return routes
|
||||
|
||||
|
||||
def test_html_comments_follow_the_next_page_link():
|
||||
routes = paginated_board("/t/1?cp=2")
|
||||
schema = learn(html_community(), "https://board.test/")
|
||||
result = run_async(Collector(FakeFetcher(routes), schema, sleep=no_sleep).run())
|
||||
first_post = next(post["origin_uid"] for post in result["records"]["posts"] if post["title"].startswith("Thread 1 "))
|
||||
assert post_comment_count(result, first_post) == 5
|
||||
|
||||
|
||||
def test_html_comment_pagination_stops_on_a_cycle_and_at_the_page_cap():
|
||||
routes = paginated_board("/t/1")
|
||||
schema = learn(html_community(), "https://board.test/")
|
||||
fetcher = FakeFetcher(routes)
|
||||
run_async(Collector(fetcher, schema, sleep=no_sleep).run())
|
||||
assert fetcher.requests.count("https://board.test/t/1") == 1
|
||||
routes = paginated_board("/t/1?cp=2")
|
||||
routes["https://board.test/t/1?cp=2"] = routes["https://board.test/t/1?cp=2"].replace("</body>", '<a rel="next" href="/t/1?cp=3">more</a></body>')
|
||||
routes["https://board.test/t/1?cp=3"] = routes["https://board.test/t/1?cp=2"].replace("cp=3", "cp=4")
|
||||
routes["https://board.test/t/1?cp=4"] = routes["https://board.test/t/1?cp=2"]
|
||||
fetcher = FakeFetcher(routes)
|
||||
schema["limits"] = {**schema.get("limits", {}), "max_comment_pages": 3}
|
||||
run_async(Collector(fetcher, schema, sleep=no_sleep).run())
|
||||
assert "https://board.test/t/1?cp=4" not in fetcher.requests
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.conftest import run_async
|
||||
|
||||
from devplacepy.database import db, get_setting, get_table, init_db, set_setting
|
||||
from devplacepy.services.community.ledger import (
|
||||
LEDGER,
|
||||
Ledger,
|
||||
OUTCOME_KEYS,
|
||||
REASON_TEXT,
|
||||
explain,
|
||||
label_for,
|
||||
tally_of,
|
||||
verdict,
|
||||
)
|
||||
from devplacepy.services.community.ledger_store import TABLE, ledger_store
|
||||
from devplacepy.services.community.pusher import SyncPusher
|
||||
from devplacepy.services.community.runner import CommunityRunner
|
||||
from devplacepy.services.community.store import store
|
||||
from devplacepy.services.sync.options import SyncOptions
|
||||
from devplacepy.services.sync.selection import Selector
|
||||
from devplacepy.services.sync.spec import REASONS, SKIP_REASONS
|
||||
from tests.unit.services.community.fakes import FakeFetcher, json_community, no_sleep
|
||||
|
||||
ORIGIN = "ledger-unit.test"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def world():
|
||||
init_db()
|
||||
previous = {key: get_setting(key, "") for key in ("sync_enabled", "sync_origins")}
|
||||
set_setting("sync_enabled", "1")
|
||||
set_setting("sync_origins", "")
|
||||
get_table("communities").delete()
|
||||
get_table("community_runs").delete()
|
||||
get_table(TABLE).delete()
|
||||
get_table("sync_links").delete(origin=ORIGIN)
|
||||
get_table("sync_links").delete(origin="forum.test")
|
||||
yield
|
||||
for key, value in previous.items():
|
||||
set_setting(key, value)
|
||||
|
||||
|
||||
def records() -> dict:
|
||||
return {
|
||||
"users": [{"origin_uid": "u1", "username": "ledgerone", "created_at": "2026-10-01T10:00:00Z"}],
|
||||
"posts": [
|
||||
{"origin_uid": "p1", "author_uid": "u1", "title": "A good post", "content": "Body of a good post, long enough.", "created_at": "2026-10-01T10:00:00Z"},
|
||||
{"origin_uid": "p2", "author_uid": "ghost", "title": "Orphan post", "content": "Written by someone who was never sent.", "created_at": "2026-10-01T10:00:00Z"},
|
||||
{"origin_uid": "p3", "title": "Gone", "deleted": True, "author_uid": "u1", "content": "Deleted at the origin before it ever arrived.", "created_at": "2026-10-01T10:00:00Z"},
|
||||
],
|
||||
"comments": [],
|
||||
}
|
||||
|
||||
|
||||
def test_pusher_records_created_rejected_and_held_records_with_reasons(world):
|
||||
ledger = Ledger()
|
||||
report = SyncPusher(ORIGIN, ledger=ledger).push(records())
|
||||
by_origin = {item.origin_uid: item for item in ledger.items}
|
||||
assert by_origin["u1"].outcome == "created" and by_origin["u1"].label == "ledgerone" and by_origin["u1"].local_uid
|
||||
assert by_origin["p1"].outcome == "created" and by_origin["p1"].label == "A good post"
|
||||
assert by_origin["p2"].outcome == "rejected" and by_origin["p2"].reason == "unknown_author" and by_origin["p2"].label == "Orphan post"
|
||||
assert by_origin["p3"].outcome == "unchanged" and by_origin["p3"].reason == "tombstone_for_unknown"
|
||||
assert tally_of(report.counts, ledger) == {"created": 2, "unchanged": 1, "rejected": 1}
|
||||
|
||||
|
||||
def test_repeating_the_same_batch_lists_nothing_that_is_merely_unchanged(world):
|
||||
SyncPusher(ORIGIN).push(records())
|
||||
ledger = Ledger()
|
||||
report = SyncPusher(ORIGIN, ledger=ledger).push(records())
|
||||
assert {item.origin_uid for item in ledger.items} == {"p2", "p3"}
|
||||
assert report.counts["users"] == {"unchanged": 1}
|
||||
|
||||
|
||||
def test_a_dry_run_lists_what_it_would_create(world):
|
||||
ledger = Ledger()
|
||||
SyncPusher(ORIGIN, dry_run=True, ledger=ledger).push(records())
|
||||
created = [item for item in ledger.items if item.outcome == "created"]
|
||||
assert {item.origin_uid for item in created} == {"u1", "p1"}
|
||||
assert get_table("sync_links").count(origin=ORIGIN) == 0
|
||||
|
||||
|
||||
def test_caps_never_let_routine_rows_push_out_problems_and_counts_stay_exact():
|
||||
ledger = Ledger(routine_limit=3, problem_limit=2)
|
||||
for number in range(10):
|
||||
ledger.add("posts", "created", origin_uid=f"c{number}")
|
||||
for number in range(5):
|
||||
ledger.add("posts", "rejected", reason="invalid", origin_uid=f"r{number}")
|
||||
kept = [item.outcome for item in ledger.items]
|
||||
assert kept.count("created") == 3 and kept.count("rejected") == 2
|
||||
assert ledger.dropped == 7 + 3
|
||||
assert ledger.counts["created"] == 10 and ledger.counts["rejected"] == 5
|
||||
|
||||
|
||||
def test_every_reason_the_system_can_emit_has_a_plain_explanation():
|
||||
for reason in (*REASONS, *SKIP_REASONS, "article_unreachable", "truncated", "collect_error"):
|
||||
assert reason in REASON_TEXT, reason
|
||||
assert explain(reason)["title"] and explain(reason)["text"]
|
||||
assert explain("never_heard_of_it")["title"] == "Never heard of it"
|
||||
assert explain("identity_failed")["retried"] and not explain("sampled_out")["retried"]
|
||||
assert explain("") == {"title": "", "text": "", "retried": False}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"status,dry,tally,level",
|
||||
[
|
||||
("ok", False, {"created": 5}, "ok"),
|
||||
("ok", False, {"unchanged": 40}, "idle"),
|
||||
("ok", False, {}, "idle"),
|
||||
("partial", False, {"created": 5, "rejected": 1}, "attention"),
|
||||
("partial", False, {"created": 5, "skipped": 3}, "attention"),
|
||||
("partial", False, {"created": 5, "warning": 1}, "attention"),
|
||||
("partial", False, {"created": 5, "failed": 1}, "bad"),
|
||||
("partial", False, {"error": 2}, "bad"),
|
||||
("failed", False, {}, "bad"),
|
||||
("skipped", False, {}, "attention"),
|
||||
("running", False, {}, "running"),
|
||||
("dry_run", True, {"created": 5}, "ok"),
|
||||
],
|
||||
)
|
||||
def test_verdict_levels(status, dry, tally, level):
|
||||
outcome = verdict(status, dry, tally)
|
||||
assert outcome["level"] == level and outcome["headline"]
|
||||
assert outcome["headline"].endswith("(dry run).") == (dry and status != "running")
|
||||
|
||||
|
||||
def test_tally_merges_sync_statuses_skips_and_collector_findings():
|
||||
ledger = Ledger()
|
||||
ledger.add("posts", "warning", reason="truncated")
|
||||
ledger.add("feed", "error", reason="collect_error")
|
||||
pushed = {"users": {"created": 2}, "posts": {"created": 1, "updated": 1, "rejected": 1}, "skipped": {"sampled_out": 4, "topic_not_allowed": 1}}
|
||||
assert tally_of(pushed, ledger) == {"created": 3, "updated": 1, "rejected": 1, "skipped": 5, "warning": 1, "error": 1}
|
||||
assert set(tally_of(pushed, ledger)) <= set(OUTCOME_KEYS)
|
||||
|
||||
|
||||
def test_labels_are_clipped_single_lines():
|
||||
assert label_for("users", {"username": "bob"}) == "bob"
|
||||
assert label_for("posts", {"title": "", "content": "line one\n\nline two"}) == "line one line two"
|
||||
assert len(label_for("comments", {"content": "x" * 900})) == LEDGER.label_chars
|
||||
|
||||
|
||||
def test_selector_reports_every_record_it_drops():
|
||||
selector = Selector(ORIGIN, SyncOptions.build(sample_percent=1))
|
||||
batch = {
|
||||
"posts": [{"origin_uid": f"s{number}", "title": "t", "content": "c" * 20} for number in range(60)],
|
||||
"comments": [{"origin_uid": f"c{number}", "post_uid": f"s{number}", "content": "hey"} for number in range(60)],
|
||||
}
|
||||
kept, skipped = selector.select(batch)
|
||||
assert len(selector.dropped) == sum(skipped.values()) > 0
|
||||
assert {reason for _, _, reason in selector.dropped} <= {"sampled_out", "parent_skipped"}
|
||||
ledger = Ledger()
|
||||
for kind, record, reason in selector.dropped:
|
||||
ledger.add_dropped_record(kind, record, reason)
|
||||
assert ledger.counts["skipped"] == len(selector.dropped) and all(item.reason for item in ledger.items)
|
||||
|
||||
|
||||
def test_a_run_stores_its_ledger_and_a_repeat_stores_nothing(world):
|
||||
routes = json_community()
|
||||
runner = CommunityRunner(fetcher_factory=lambda: FakeFetcher(routes), sleep=no_sleep)
|
||||
store.create("forum-test", "https://forum.test/api/posts", "tester", name="forum")
|
||||
run_async(runner.discover("https://forum.test/api/posts", "tester", relearn_slug="forum-test"))
|
||||
first = run_async(runner.run(store.get("forum-test"), dry_run=False, trigger="test"))
|
||||
saved = store.run(first.run_uid)
|
||||
assert saved["items_stored"] == get_table(TABLE).count(run_uid=first.run_uid) > 0
|
||||
assert saved["tally"]["created"] == ledger_store.counts(run_uid=first.run_uid)["created"]
|
||||
assert saved["verdict"] == "ok" and saved["items_dropped"] == 0
|
||||
second = run_async(runner.run(store.get("forum-test"), dry_run=False, trigger="test"))
|
||||
again = store.run(second.run_uid)
|
||||
assert again["items_stored"] == 0 and again["verdict"] == "idle" and again["tally"]["unchanged"] > 0
|
||||
|
||||
|
||||
def test_query_plans_use_an_index_and_never_sort_or_scan(world):
|
||||
ledger = Ledger(routine_limit=4000, problem_limit=4000)
|
||||
for number in range(3000):
|
||||
ledger.add("posts", ("created", "rejected", "skipped", "updated")[number % 4], reason="invalid", origin_uid=f"o{number}", label=f"title {number}")
|
||||
ledger_store.save("plan-run", "plan-community", ledger)
|
||||
db.query("ANALYZE")
|
||||
cases = [
|
||||
({}, {"run_uid": "plan-run"}),
|
||||
({"outcome": "rejected"}, {"run_uid": "plan-run"}),
|
||||
({"outcome": "rejected", "kind": "posts"}, {"run_uid": "plan-run"}),
|
||||
({}, {"community_uid": "plan-community"}),
|
||||
({"outcome": "skipped"}, {"community_uid": "plan-community"}),
|
||||
]
|
||||
for filters, scope in cases:
|
||||
where, params, order = ledger_store.compile(filters, **scope)
|
||||
for sql in (
|
||||
f"SELECT * FROM {TABLE} WHERE {where} ORDER BY {order} LIMIT 50 OFFSET 0",
|
||||
f"SELECT COUNT(*) FROM (SELECT 1 FROM {TABLE} WHERE {where} LIMIT 10001)",
|
||||
):
|
||||
plan = " | ".join(row["detail"] for row in db.query("EXPLAIN QUERY PLAN " + sql, **params))
|
||||
assert "USING INDEX" in plan or "USING COVERING INDEX" in plan, (filters, scope, plan)
|
||||
assert "SCAN community_run_items" not in plan, plan
|
||||
if not filters.get("outcome") or not scope.get("run_uid"):
|
||||
assert "TEMP B-TREE" not in plan, (filters, scope, plan)
|
||||
assert ledger_store.counts(run_uid="plan-run") == {"created": 750, "rejected": 750, "skipped": 750, "updated": 750}
|
||||
rows, pagination = ledger_store.page({"q": "title 7%"}, 1, run_uid="plan-run")
|
||||
assert rows == [] and pagination["total"] == 0
|
||||
rows, pagination = ledger_store.page({"q": "title 77"}, 1, run_uid="plan-run")
|
||||
assert pagination["total"] == 11 and len(rows) == 11
|
||||
rows, pagination = ledger_store.page({}, 10**6, run_uid="plan-run")
|
||||
assert pagination["page"] == pagination["total_pages"] and len(rows) == 50 and pagination["capped"] is False
|
||||
json.dumps(rows)
|
||||
|
||||
|
||||
def test_record_detail_is_kept_for_the_newest_runs_only_but_their_counts_stay(world):
|
||||
from devplacepy.services.community.spec import LIMITS
|
||||
|
||||
community = store.create("keep-test", "q", "tester", name="keep")
|
||||
uids = []
|
||||
for number in range(LIMITS.ledger_runs_kept + 3):
|
||||
run_uid = store.start_run(community["uid"], "test", False)
|
||||
ledger = Ledger()
|
||||
ledger.add("posts", "created", origin_uid=f"k{number}")
|
||||
ledger_store.save(run_uid, community["uid"], ledger)
|
||||
store.finish_run(run_uid, "ok", tally={"created": 1}, items_stored=1)
|
||||
uids.append(run_uid)
|
||||
store.prune_runs(community["uid"])
|
||||
kept = [get_table(TABLE).count(run_uid=uid) for uid in uids]
|
||||
assert sum(kept) == LIMITS.ledger_runs_kept and kept[0] == 0 and kept[-1] == 1
|
||||
assert len(store.run_summaries(community["uid"])) == LIMITS.ledger_runs_kept + 3
|
||||
assert store.run(uids[0])["tally"] == {"created": 1}
|
||||
@@ -7,7 +7,7 @@ import pytest
|
||||
from devplacepy.database import get_setting, get_table, init_db, set_setting
|
||||
from devplacepy.services.community.runner import CommunityRunner
|
||||
from devplacepy.services.community.service import DRY_RUN_KEY, CommunityService
|
||||
from devplacepy.services.community.spec import STATUS_FAILING, STATUS_READY
|
||||
from devplacepy.services.community.spec import LIMITS, RUN_SKIPPED, STATUS_FAILING, STATUS_READY
|
||||
from devplacepy.services.community.store import store
|
||||
from tests.unit.services.community.fakes import FakeFetcher, json_community, no_sleep
|
||||
|
||||
@@ -79,3 +79,17 @@ def test_run_history_is_bounded(world):
|
||||
for _ in range(35):
|
||||
run_async(runner.run(store.get("forum-test"), dry_run=True, trigger="test"))
|
||||
assert len(store.runs(row["uid"], limit=100)) == 30
|
||||
|
||||
|
||||
def test_gate_refusal_skips_the_run_without_counting_a_failure(world):
|
||||
_, runner = world
|
||||
row = seed(runner)
|
||||
set_setting("sync_enabled", "0")
|
||||
for _ in range(LIMITS.max_failures + 1):
|
||||
report = run_async(runner.run(store.get("forum-test"), dry_run=False, trigger="test"))
|
||||
assert report.status == RUN_SKIPPED
|
||||
final = store.get("forum-test")
|
||||
assert final["status"] == STATUS_READY and final["enabled"]
|
||||
assert final["failure_count"] == 0 and final["last_status"] == RUN_SKIPPED
|
||||
assert final["next_run_at"] and get_table("sync_links").count(origin="forum.test") == 0
|
||||
assert store.runs(row["uid"])[0]["status"] == RUN_SKIPPED
|
||||
|
||||
Reference in New Issue
Block a user