# News service (`devplacepy/services/news/`)

This file documents the automated developer-news import pipeline. Claude Code auto-loads it when a file under `devplacepy/services/news/` is read or edited.

## Overview

`BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation. `ServiceManager` is a singleton that registers, starts, and stops services. `NewsService` (`services/news/service.py`, `default_enabled=True`, registered unconditionally in `main.py`) is a fully automatic, zero-maintenance import pipeline: it fetches articles from `news_api_url`, **cleans** each (HTML strip + `clean_news_text`/`JUNK_PATTERNS` Reddit-boilerplate removal), **fetches and perceptually compares the images** (SSRF-guarded fetch + Pillow decode + `imagehash` phash off-thread; placeholder = too small / undecodable / a phash shared across 2+ different articles), **grades deterministically** (AI grade on cleaned text via `news_ai_url` + a reliability gate + a unique-image bonus and thin-content penalty -> effective `grade`, raw in `ai_grade`), **reformats every article that clears the publish threshold into clean Markdown** (`_format_article` + the editable `news_format_prompt`, its OWN `news_format_url`/`news_format_model`/`news_format_key` - independent of grading, see "AI reformatting" below; paragraphs, `## ` headings, lists and code, preserving every fact; fail-soft to the cleaned original, toggle `news_format_enabled`; draft-bound articles - valid but below `news_grade_threshold` - are never sent for formatting, saving the call entirely), and inserts ALL of them into `news` with `status="published"`/`"draft"` on `news_grade_threshold` - nothing is silently skipped. After the loop it **auto-rotates Featured + Landing** (`_apply_landing_selection`, top scored unique-image articles in the recent window) while honouring per-row `featured_locked`/`landing_locked` set when an admin manually toggles. **AI usage is metered like the correction/modifier consumers**: each gateway call's `X-Gateway-*` response headers are parsed (`parse_usage_headers`) and the run's totals accumulated into the durable single-row `news_usage` table (`database.add_news_usage`/`get_news_usage`, same SUMs-plus-computed-averages shape as `correction_usage`); `NewsService.collect_metrics()` surfaces calls/tokens/cost and the per-call averages as `stats` on the admin-only `/admin/services` page. The full ruleset is on `NewsService.description` and surfaces on `/admin/services`, which also polls live status and the log tail.

## Per-run flow

The per-run flow per article is: clean text -> fetch and perceptually compare images -> AI-grade the cleaned text -> reliability gate -> compute an effective score -> publish/Feature decision -> AI-reformat the body into Markdown (published articles only, i.e. `effective_score >= news_grade_threshold`) -> store -> then a single post-loop Landing rotation.

- Fetches `GET {news_api_url}` -> `{"articles": [...]}`; already-synced `external_id`s (from `news_sync`) are skipped.
- Every article is graded via AI on the CLEANED text: `POST {news_ai_url}` with model `{news_ai_model}`, temperature 0. Both default to the free, local **aquality** quality model (`AQUALITY_NEWS_GRADING_URL` = `https://aquality.cloud.pravda.education/v1/chat/completions` / `AQUALITY_NEWS_GRADING_MODEL` = `"aquality"`, `devplacepy/config.py`) - a deterministic scikit-learn regressor trained on this site's own editorial history (`grade`/`status` columns), served behind an OpenAI-chat-completions-compatible shim so no code here changes to call it. No API key is required (the key still falls back to `internal_gateway_key()` when `news_ai_key`/`NEWS_AI_KEY` is unset, but aquality ignores it unless its own `NEWS_QUALITY_API_KEY` is configured). `migrate_ai_gateway_settings()` (`database/schema.py`) one-time-migrates any existing `news_ai_url`/`news_ai_model` still on the old internal-gateway defaults over to aquality, without touching a value an admin has customized. Point `news_ai_url` at a real generative chat model instead to go back to LLM-based grading (higher quality, billed).
- ALL articles are inserted into `news` regardless of grade (never silently skipped). Articles re-synced each run (upsert by `external_id`); grade, status, image, and images updated each cycle. Slugs via `make_combined_slug(title, uid)`.
- All parameters (`news_api_url`, `news_ai_url`, `news_ai_model`, `news_grade_threshold`, `news_ai_key`, `news_format_enabled`, `news_format_url`, `news_format_model`, `news_format_key`, `news_format_prompt`, interval) are declared as `config_fields` and edited on the Services tab. The full grading ruleset is carried on `NewsService.description` (the `GRADING_RULES_DESCRIPTION` module string) and renders on `/admin/services`.

## AI reformatting (`_format_article`, `FORMAT_PROMPT_SPEC`)

After grading, every PUBLISHED article (one that passed the reliability gate, got a grade, and whose `effective_score` reaches `news_grade_threshold` - `published` in `run_once`, not merely `result.valid`) has its body reformatted by the AI into clean Markdown - short paragraphs, `## ` section headings, bullet/numbered lists, and inline/fenced code - turning the source wall of text into a readable article.

**Formatting has its OWN endpoint/model/key, entirely independent of grading (load-bearing - do not merge them back together).** `news_format_url`/`news_format_model` (group "AI formatting") default to `INTERNAL_GATEWAY_URL`/`INTERNAL_MODEL` (`FORMAT_URL_DEFAULT`/`FORMAT_MODEL_DEFAULT`, `constants.py`) - a real generative model out of the box - which is why `news_format_enabled` can default to `True`-capable behavior the moment it is turned on, with zero extra configuration. This is deliberate: the default GRADING endpoint (`news_ai_url`) is aquality, a scoring-only classifier/regressor with no text-generation capability, so reusing it for formatting would silently no-op (aquality answers the formatting prompt - no `Description:`/`Content:` labels, so its `GRADE_PROMPT_PATTERN` regex does not match - with an empty reply, which the fail-soft path already treats as "keep the cleaned original", masking the misconfiguration as a no-op rather than a visible error). Splitting the two AI calls into independent config groups means an admin never has to choose between free grading and generative formatting. `news_format_key` (secret) resolves via `_get_ai_key("news_format_key", fallback_setting="news_ai_key")` - blank falls back to `news_ai_key`, then the gateway's internal key - so a shared custom key still works with zero extra setup, but a different provider per call is fully supported.

The call runs at `temperature 0.3`, `FORMAT_MAX_TOKENS=6000`, with the cleaned `description`+`content` (capped at `FORMAT_INPUT_MAX_CHARS=14000`) appended to the editable `news_format_prompt`. The prompt forbids inventing/removing facts and only restructures. The result is fence-stripped (`_strip_md_fence`), validated to be at least `MIN_BODY_CHARS`, capped at `FORMAT_OUTPUT_MAX_CHARS=30000`, and stored as the `news.content` (rendered server-side by `render_content`, the markdown engine, on `news_detail.html`). It is fail-soft: on any error, an empty/too-short result, or `news_format_enabled` off, the cleaned original content is stored unchanged. Toggle with the `news_format_enabled` bool config field (still defaults to `False` - grading and formatting are billed independently, and reformatting every article is an admin opt-in, not because it depends on `news_ai_url` any more).

## AI usage metering and stats (the shared `usage.py` helpers, `news_usage`, `collect_metrics`)

Service-level AI spend is metered exactly like the per-user `correction_usage`/`modifier_usage` consumers, and the accumulation/formatting plumbing is **shared, not copied per service**, in `services/openai_gateway/usage.py` beside `parse_usage_headers`: `USAGE_FIELDS` (the seven summed columns), `new_usage_totals()` (a zeroed totals dict), `accumulate_usage(totals, response)` (parses the `X-Gateway-*` headers via `parse_usage_headers` - defensive `getattr(response, "headers", None)`, so fake responses without headers are a no-op - and adds each field into the totals; `totals=None` is a no-op so the metering is opt-in per call site), and `usage_metric_cards(usage)` (the one stat-card list builder). For NewsService the grading and formatting gateway calls are its only AI spend: on every successful gateway response it calls `accumulate_usage(usage_totals, resp)` into a per-run dict from `new_usage_totals()`, and after the run `database.add_news_usage(usage_totals)` does ONE `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` upsert into the durable single-row `news_usage` table (the same `_add_usage` helper as the correction/modifier tables, keyed on the constant `NEWS_USAGE_KEY="news"` in the `user_uid` column, ensured + unique-indexed in `init_db`). `database.get_news_usage()` (the same `_get_usage`) returns the running SUMs plus computed AVERAGES (avg tokens/call, avg cost/call, avg upstream latency, avg tokens/sec). `NewsService.collect_metrics()` is `{"stats": usage_metric_cards(get_news_usage())}`, which `BaseService._persist_state` serializes (every ~3s tick, regardless of the service's enabled state) and `ServiceMonitor.renderMetrics` shows on the admin-only `/admin/services` page - consistent with `BotsService` (live fleet cost) and the financial-data-is-admin-only rule (the services page is `require_admin`).

**The issue tracker is the second service consumer and reuses all of this 1:1:** its two AI call sites (`services/gitea/enhance.py` `enhance_ticket` for ticket filing and `services/gitea/planning.py` `generate_plan` for the planning document) take an optional `totals` dict and call `accumulate_usage(totals, response)` after `raise_for_status`; the owning job services (`IssueCreateService`, `PlanningReportService`) build a `new_usage_totals()`, pass it in, and `database.add_issue_usage(totals)` (constant `ISSUE_USAGE_KEY="issues"`, table `issue_usage`, ensured beside `news_usage`) when `totals["calls"]`. `IssueTrackerService.collect_metrics()` is `{"stats": usage_metric_cards(get_issue_usage())}`, so the aggregate ticket-AI spend shows on the Issue Tracker service page even though the poller defaults to disabled (the supervisor still ticks and persists metrics). To meter a NEW service AI call: thread a `new_usage_totals()` dict to the call site, `accumulate_usage` the response, persist with a matching `add_<x>_usage`/single-row `<x>_usage` table, and surface it via `collect_metrics` -> `usage_metric_cards`.

## Content cleaning (`clean_news_text`, `JUNK_PATTERNS`)

After `strip_html`, removes Reddit boilerplate (`submitted by /u/<user>`, `submitted by ... to /r/<sub>`, standalone `[link]`, `[comments]`, `[N comments]`, trailing `submitted by ... [link] [comments]`) and collapses whitespace. Applied to title (light), description, and content BEFORE grading and BEFORE storage.

## Image fetch + perceptual placeholder/uniqueness detection

Per article up to `IMG_PER_ARTICLE=5` candidate image URLs (from `_get_article_images`) are fetched through `net_guard.guarded_async_client` (SSRF-safe, re-guards redirects), capped at `IMG_FETCH_MAX_BYTES=5_000_000` and a short timeout; each is decoded with Pillow and phashed (`imagehash`) via `loop.run_in_executor` so the CPU work never blocks the loop, recording width/height. Reject as placeholder on decode/fetch failure or width/height `< MIN_IMAGE_DIMENSION=100`. Across the WHOLE batch, phashes within Hamming distance `PHASH_DISTANCE=5` (compared via `imagehash.hex_to_hash`) that span 2+ DIFFERENT articles are a shared logo/placeholder and are flagged placeholder for all of them. `has_unique_image` = at least one non-placeholder candidate; its URL becomes the article's `image_url`. Each `news_images` row stores `phash`/`width`/`height`/`is_placeholder`; the `news` row stores `image_url` and `has_unique_image`.

## Deterministic grading (constants at module top)

`MIN_TITLE_CHARS=12`, `MIN_BODY_CHARS=200`, `MAX_TITLE_CAPS_RATIO=0.6`, `MIN_IMAGE_DIMENSION=100`, `PHASH_DISTANCE=5`, `IMG_PER_ARTICLE=5`, `IMG_FETCH_MAX_BYTES=5_000_000`, `UNIQUE_IMAGE_BONUS=2`, `THIN_CONTENT_PENALTY=2`, `FEATURE_MIN_SCORE=8`, `LANDING_MIN_SCORE=9`, `LANDING_MAX=6`, `LANDING_RECENCY_DAYS=4`. Per article: (1) the raw AI grade (1-10) is stored in `ai_grade`; `None` is invalid. (2) The reliability gate (`reliability_reason`) forces draft (never Featured/Landing) when the url is empty/invalid, the cleaned title `< MIN_TITLE_CHARS`, the combined cleaned body `< MIN_BODY_CHARS`, or the title uppercase-letter ratio `> MAX_TITLE_CAPS_RATIO`; the reason is recorded in `news_sync.status` as `rejected_quality:<reason>` and in the audit metadata. (3) `effective_score = clamp(ai_grade + UNIQUE_IMAGE_BONUS if has_unique_image - THIN_CONTENT_PENALTY if body marginal-but-not-gated, 1..10)` (marginal = body `< MIN_BODY_CHARS * 2`); stored in `grade` (so `/news` ordering reflects final quality). (4) `status = "published"` when valid and `effective_score >= news_grade_threshold` else `draft`. (5) `featured = 1` when published AND `has_unique_image` AND `effective_score >= FEATURE_MIN_SCORE` (respecting the lock).

## Auto-rotating Featured + Landing with admin lock

After the per-article loop, `_apply_landing_selection` queries recent (`<= LANDING_RECENCY_DAYS`) rows that are published + featured + `has_unique_image` + NOT `landing_locked`, sorts by `(-grade, -synced_at)`, sets `show_on_landing=1` for the top `LANDING_MAX` with `grade >= LANDING_MIN_SCORE` and `show_on_landing=0` for the rest of THAT service-managed set (self-rotating). When an admin manually toggles Featured or Landing (`routers/admin/news.py`), the matching `featured_locked`/`landing_locked` is set to 1 so the service stops auto-managing that row; updates of a locked Featured row never overwrite `featured`. New `news` columns: `ai_grade`, `featured`, `has_unique_image`, `image_url`, `featured_locked`, `landing_locked`, `author`, `article_published` (all added to the `init_db` ensure-block - the stale-schema rule). Audit events `news.service.ingest`/`news.service.publish`/`news.service.draft`/`news.service.reject`/`news.service.landing` (`record_system`, category `news`). New dep: `imagehash` (`Pillow` already present).

## Editorial labels are admin-only on the public surfaces

The `&#x2605; Featured` badge and the `Grade {{ grade }}` chip are internal editorial signals, so on every PUBLIC news surface - the `/news` listing (`news.html`), the article page (`news_detail.html`), and the landing/home Developer News cards (`landing.html`) - both are wrapped in `{% if is_admin(user) %}`. Members and guests never see them; administrators still see them everywhere (including the public pages) and the admin **Manage News** area (`admin_news.html`) is unchanged, where the grade and featured state are the editorial controls. The article's source name and timestamp stay visible to everyone. Gating only the rendered labels does not touch the stored `grade`/`featured` columns, the grading/landing logic, or ordering (`/news` is still ordered by effective `grade`).
