devplacepy/devplacepy/services/news/CLAUDE.md

News service (devplacepy/services/news/)

This file documents the automated developer-news import pipeline. Claude Code auto-loads it when a file under devplacepy/services/news/ is read or edited.

Overview

BaseService provides the async run loop, a deque(maxlen=20) log buffer, and graceful cancellation. ServiceManager is a singleton that registers, starts, and stops services. NewsService (services/news/service.py, default_enabled=True, registered unconditionally in main.py) is a fully automatic, zero-maintenance import pipeline: it fetches articles from news_api_url, cleans each (HTML strip + clean_news_text/JUNK_PATTERNS Reddit-boilerplate removal), fetches and perceptually compares the images (SSRF-guarded fetch + Pillow decode + imagehash phash off-thread; placeholder = too small / undecodable / a phash shared across 2+ different articles), grades deterministically (AI grade on cleaned text via news_ai_url + a reliability gate + a unique-image bonus and thin-content penalty -> effective grade, raw in ai_grade), reformats every article that clears the publish threshold into clean Markdown (_format_article + the editable news_format_prompt; paragraphs, ## headings, lists and code, preserving every fact; fail-soft to the cleaned original, toggle news_format_enabled; draft-bound articles - valid but below news_grade_threshold - are never sent for formatting, saving the call entirely), and inserts ALL of them into news with status="published"/"draft" on news_grade_threshold - nothing is silently skipped. After the loop it auto-rotates Featured + Landing (_apply_landing_selection, top scored unique-image articles in the recent window) while honouring per-row featured_locked/landing_locked set when an admin manually toggles. AI usage is metered like the correction/modifier consumers: each gateway call's X-Gateway-* response headers are parsed (parse_usage_headers) and the run's totals accumulated into the durable single-row news_usage table (database.add_news_usage/get_news_usage, same SUMs-plus-computed-averages shape as correction_usage); NewsService.collect_metrics() surfaces calls/tokens/cost and the per-call averages as stats on the admin-only /admin/services page. The full ruleset is on NewsService.description and surfaces on /admin/services, which also polls live status and the log tail.

Per-run flow

The per-run flow per article is: clean text -> fetch and perceptually compare images -> AI-grade the cleaned text -> reliability gate -> compute an effective score -> publish/Feature decision -> AI-reformat the body into Markdown (published articles only, i.e. effective_score >= news_grade_threshold) -> store -> then a single post-loop Landing rotation.

• Fetches GET {news_api_url} -> {"articles": [...]}; already-synced external_ids (from news_sync) are skipped.
• Every article is graded via AI on the CLEANED text: POST {news_ai_url} with model {news_ai_model}, temperature 0. Both default to the free, local aquality quality model (AQUALITY_NEWS_GRADING_URL = https://aquality.cloud.pravda.education/v1/chat/completions / AQUALITY_NEWS_GRADING_MODEL = "aquality", devplacepy/config.py) - a deterministic scikit-learn regressor trained on this site's own editorial history (grade/status columns), served behind an OpenAI-chat-completions-compatible shim so no code here changes to call it. No API key is required (the key still falls back to internal_gateway_key() when news_ai_key/NEWS_AI_KEY is unset, but aquality ignores it unless its own NEWS_QUALITY_API_KEY is configured). migrate_ai_gateway_settings() (database/schema.py) one-time-migrates any existing news_ai_url/news_ai_model still on the old internal-gateway defaults over to aquality, without touching a value an admin has customized. Point news_ai_url at a real generative chat model instead to go back to LLM-based grading (higher quality, billed).
• ALL articles are inserted into news regardless of grade (never silently skipped). Articles re-synced each run (upsert by external_id); grade, status, image, and images updated each cycle. Slugs via make_combined_slug(title, uid).
• All parameters (news_api_url, news_ai_url, news_ai_model, news_grade_threshold, news_ai_key, news_format_enabled, news_format_prompt, interval) are declared as config_fields and edited on the Services tab. The full grading ruleset is carried on NewsService.description (the GRADING_RULES_DESCRIPTION module string) and renders on /admin/services.

AI reformatting (_format_article, FORMAT_PROMPT_SPEC)

After grading, every PUBLISHED article (one that passed the reliability gate, got a grade, and whose effective_score reaches news_grade_threshold - published in run_once, not merely result.valid) has its body reformatted by the AI into clean Markdown - short paragraphs, ## section headings, bullet/numbered lists, and inline/fenced code - turning the source wall of text into a readable article. It reuses the grading endpoint/model/key (news_ai_url/news_ai_model/_get_ai_key) at temperature 0.3, FORMAT_MAX_TOKENS=6000, with the cleaned description+content (capped at FORMAT_INPUT_MAX_CHARS=14000) appended to the editable news_format_prompt. The prompt forbids inventing/removing facts and only restructures. The result is fence-stripped (_strip_md_fence), validated to be at least MIN_BODY_CHARS, capped at FORMAT_OUTPUT_MAX_CHARS=30000, and stored as the news.content (rendered server-side by render_content, the markdown engine, on news_detail.html). It is fail-soft: on any error, an empty/too-short result, or news_format_enabled off, the cleaned original content is stored unchanged. Toggle with the news_format_enabled bool config field. Defaults to False since aquality, the default news_ai_url grading model, is a scoring-only classifier/regressor with no text-generation capability - it answers this call's prompt (no Description:/Content: labels, so the shared GRADE_PROMPT_PATTERN regex on the aquality side does not match) with an empty reply, which this fail-soft path already treats as "keep the cleaned original." Enable it only after pointing news_ai_url at a real generative chat model.

AI usage metering and stats (the shared usage.py helpers, news_usage, collect_metrics)

Service-level AI spend is metered exactly like the per-user correction_usage/modifier_usage consumers, and the accumulation/formatting plumbing is shared, not copied per service, in services/openai_gateway/usage.py beside parse_usage_headers: USAGE_FIELDS (the seven summed columns), new_usage_totals() (a zeroed totals dict), accumulate_usage(totals, response) (parses the X-Gateway-* headers via parse_usage_headers - defensive getattr(response, "headers", None), so fake responses without headers are a no-op - and adds each field into the totals; totals=None is a no-op so the metering is opt-in per call site), and usage_metric_cards(usage) (the one stat-card list builder). For NewsService the grading and formatting gateway calls are its only AI spend: on every successful gateway response it calls accumulate_usage(usage_totals, resp) into a per-run dict from new_usage_totals(), and after the run database.add_news_usage(usage_totals) does ONE INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col upsert into the durable single-row news_usage table (the same _add_usage helper as the correction/modifier tables, keyed on the constant NEWS_USAGE_KEY="news" in the user_uid column, ensured + unique-indexed in init_db). database.get_news_usage() (the same _get_usage) returns the running SUMs plus computed AVERAGES (avg tokens/call, avg cost/call, avg upstream latency, avg tokens/sec). NewsService.collect_metrics() is {"stats": usage_metric_cards(get_news_usage())}, which BaseService._persist_state serializes (every ~3s tick, regardless of the service's enabled state) and ServiceMonitor.renderMetrics shows on the admin-only /admin/services page - consistent with BotsService (live fleet cost) and the financial-data-is-admin-only rule (the services page is require_admin).

The issue tracker is the second service consumer and reuses all of this 1:1: its two AI call sites (services/gitea/enhance.py enhance_ticket for ticket filing and services/gitea/planning.py generate_plan for the planning document) take an optional totals dict and call accumulate_usage(totals, response) after raise_for_status; the owning job services (IssueCreateService, PlanningReportService) build a new_usage_totals(), pass it in, and database.add_issue_usage(totals) (constant ISSUE_USAGE_KEY="issues", table issue_usage, ensured beside news_usage) when totals["calls"]. IssueTrackerService.collect_metrics() is {"stats": usage_metric_cards(get_issue_usage())}, so the aggregate ticket-AI spend shows on the Issue Tracker service page even though the poller defaults to disabled (the supervisor still ticks and persists metrics). To meter a NEW service AI call: thread a new_usage_totals() dict to the call site, accumulate_usage the response, persist with a matching add_<x>_usage/single-row <x>_usage table, and surface it via collect_metrics -> usage_metric_cards.

Content cleaning (clean_news_text, JUNK_PATTERNS)

After strip_html, removes Reddit boilerplate (submitted by /u/<user>, submitted by ... to /r/<sub>, standalone [link], [comments], [N comments], trailing submitted by ... [link] [comments]) and collapses whitespace. Applied to title (light), description, and content BEFORE grading and BEFORE storage.

Image fetch + perceptual placeholder/uniqueness detection

Per article up to IMG_PER_ARTICLE=5 candidate image URLs (from _get_article_images) are fetched through net_guard.guarded_async_client (SSRF-safe, re-guards redirects), capped at IMG_FETCH_MAX_BYTES=5_000_000 and a short timeout; each is decoded with Pillow and phashed (imagehash) via loop.run_in_executor so the CPU work never blocks the loop, recording width/height. Reject as placeholder on decode/fetch failure or width/height < MIN_IMAGE_DIMENSION=100. Across the WHOLE batch, phashes within Hamming distance PHASH_DISTANCE=5 (compared via imagehash.hex_to_hash) that span 2+ DIFFERENT articles are a shared logo/placeholder and are flagged placeholder for all of them. has_unique_image = at least one non-placeholder candidate; its URL becomes the article's image_url. Each news_images row stores phash/width/height/is_placeholder; the news row stores image_url and has_unique_image.

Deterministic grading (constants at module top)

MIN_TITLE_CHARS=12, MIN_BODY_CHARS=200, MAX_TITLE_CAPS_RATIO=0.6, MIN_IMAGE_DIMENSION=100, PHASH_DISTANCE=5, IMG_PER_ARTICLE=5, IMG_FETCH_MAX_BYTES=5_000_000, UNIQUE_IMAGE_BONUS=2, THIN_CONTENT_PENALTY=2, FEATURE_MIN_SCORE=8, LANDING_MIN_SCORE=9, LANDING_MAX=6, LANDING_RECENCY_DAYS=4. Per article: (1) the raw AI grade (1-10) is stored in ai_grade; None is invalid. (2) The reliability gate (reliability_reason) forces draft (never Featured/Landing) when the url is empty/invalid, the cleaned title < MIN_TITLE_CHARS, the combined cleaned body < MIN_BODY_CHARS, or the title uppercase-letter ratio > MAX_TITLE_CAPS_RATIO; the reason is recorded in news_sync.status as rejected_quality:<reason> and in the audit metadata. (3) effective_score = clamp(ai_grade + UNIQUE_IMAGE_BONUS if has_unique_image - THIN_CONTENT_PENALTY if body marginal-but-not-gated, 1..10) (marginal = body < MIN_BODY_CHARS * 2); stored in grade (so /news ordering reflects final quality). (4) status = "published" when valid and effective_score >= news_grade_threshold else draft. (5) featured = 1 when published AND has_unique_image AND effective_score >= FEATURE_MIN_SCORE (respecting the lock).

Auto-rotating Featured + Landing with admin lock

After the per-article loop, _apply_landing_selection queries recent (<= LANDING_RECENCY_DAYS) rows that are published + featured + has_unique_image + NOT landing_locked, sorts by (-grade, -synced_at), sets show_on_landing=1 for the top LANDING_MAX with grade >= LANDING_MIN_SCORE and show_on_landing=0 for the rest of THAT service-managed set (self-rotating). When an admin manually toggles Featured or Landing (routers/admin/news.py), the matching featured_locked/landing_locked is set to 1 so the service stops auto-managing that row; updates of a locked Featured row never overwrite featured. New news columns: ai_grade, featured, has_unique_image, image_url, featured_locked, landing_locked, author, article_published (all added to the init_db ensure-block - the stale-schema rule). Audit events news.service.ingest/news.service.publish/news.service.draft/news.service.reject/news.service.landing (record_system, category news). New dep: imagehash (Pillow already present).

Editorial labels are admin-only on the public surfaces

The ★ Featured badge and the Grade {{ grade }} chip are internal editorial signals, so on every PUBLIC news surface - the /news listing (news.html), the article page (news_detail.html), and the landing/home Developer News cards (landing.html) - both are wrapped in {% if is_admin(user) %}. Members and guests never see them; administrators still see them everywhere (including the public pages) and the admin Manage News area (admin_news.html) is unchanged, where the grade and featured state are the editorial controls. The article's source name and timestamp stay visible to everyone. Gating only the rendered labels does not touch the stored grade/featured columns, the grading/landing logic, or ordering (/news is still ordered by effective grade).