|
<div class="docs-content" data-render>
|
|
# NewsService
|
|
|
|
`name` `news` - enabled by default - 3600s interval (minimum 60s). It is a fully automatic, zero-maintenance import pipeline: it fetches developer news from a configured feed, cleans each article, fetches and perceptually compares the images to reject placeholders and detect uniqueness, grades each article deterministically, reformats every valid one into clean Markdown, and auto-rotates the best ones to Featured and Landing. Source: `services/news/service.py`. See also [Reading service data](/docs/services-data.html).
|
|
|
|
> Audience: administrators and maintainers.
|
|
|
|
## What run_once does
|
|
|
|
1. Reads the source URL, grading URL, model, and threshold from configuration.
|
|
2. Fetches the feed over HTTP (30s timeout) and reads its `articles` list. On error it logs and returns.
|
|
3. Loads the set of already-synced external ids from `news_sync` so seen articles are skipped.
|
|
4. For each new article it cleans the text, fetches up to 5 candidate images, perceptually hashes them, grades the article on the cleaned text, computes a final score, reformats a valid article into clean Markdown (when `news_format_enabled` is on, fail-soft to the cleaned original), decides published versus draft and Featured, upserts the `news` row, records the sync state, and stores the images with their hashes.
|
|
5. After the per-article loop it rotates Landing: the top Featured unique-image articles from the recent window are shown on the landing page and the rest of the service-managed set is cleared.
|
|
6. Logs a summary: new, updated, draft, rejected, grading-failed, and skipped counts.
|
|
|
|
**Content cleaning:** after HTML stripping the title (light), description, and content have Reddit boilerplate removed (`submitted by /u/<user>`, `submitted by ... to /r/<sub>`, standalone `[link]`, `[comments]`, `[N comments]`, and trailing `submitted by ... [link] [comments]`) and whitespace collapsed, both before grading and before storage.
|
|
|
|
**Image uniqueness:** each candidate image is fetched through the SSRF-guarded client (capped at 5 MB, short timeout), decoded with Pillow off the event loop, and perceptually hashed (`imagehash` phash) with its width and height recorded. An image under 100px on either side, or one that fails to fetch or decode, is a placeholder. Hashes within Hamming distance 5 that span two or more different articles are a shared logo or placeholder and are rejected for all of them. An article with at least one genuinely unique image is marked `has_unique_image` and its first such image becomes `image_url`.
|
|
|
|
**Grading:** the cleaned title, description, and content (each length-capped) are sent to the grading endpoint at temperature 0 asking for a single integer 1 to 10 (the raw AI grade, stored in `ai_grade`). A reliability gate forces draft (never Featured or Landing) when the url is empty or invalid, the cleaned title is shorter than 12 chars, the combined cleaned body is shorter than 200 chars, or the title uppercase-letter ratio exceeds 0.6 (the gate reason is recorded in `news_sync.status` as `rejected_quality:<reason>`). The effective score is `clamp(ai_grade + 2 if unique image - 2 if the body is marginal but not gated, 1..10)`, stored in `grade`. An article is published when valid and its effective score reaches `news_grade_threshold`; it is Featured when published, has a unique image, and scores at least 8. A grading failure stores it as a draft with grade 0 and sync status `grading_failed`. Nothing is silently discarded.
|
|
|
|
**Auto Featured and Landing with admin lock:** after the run the service selects the top 6 published, Featured, unique-image articles from the last 4 days scoring at least 9 for the landing page and clears the rest of its managed set (self-rotating). When an admin manually toggles Featured or Landing in the admin UI, that row's `featured_locked` or `landing_locked` is set to 1 and the service never auto-manages it again.
|
|
|
|
## Configuration fields
|
|
|
|
- `news_api_url` (url, default `https://news.app.molodetz.nl/api`) - the source feed, group Source.
|
|
- `news_ai_url` (url, default the internal gateway) - the chat-completions endpoint used to grade, group AI grading.
|
|
- `news_ai_model` (str, default the internal model) - the model sent to the grading endpoint.
|
|
- `news_grade_prompt` (text) - the exact grading rubric sent to the model; defaults to the specification below and is editable on the service page. The cleaned Title, Description and Content are appended automatically.
|
|
- `news_grade_threshold` (int, default 7, range 1 to 10) - articles whose effective score (AI grade plus the unique-image bonus minus the thin-content penalty) reaches this publish, below drafts.
|
|
- `news_ai_key` (secret) - defaults to the `NEWS_AI_KEY` env var, then the gateway internal key.
|
|
- `news_format_enabled` (bool, default on, group AI formatting) - when on, every valid article is reformatted into clean Markdown after grading.
|
|
- `news_format_prompt` (text, group AI formatting) - the instruction sent to the AI to reformat each cleaned article into Markdown; the Title and body are appended automatically. It must preserve every fact and output only the reformatted Markdown body.
|
|
|
|
Plus the inherited Enabled, Run interval, and Log buffer size fields.
|
|
|
|
## Grading prompt specification
|
|
|
|
This is the default value of the editable `news_grade_prompt` field. It is sent verbatim to the grading model at temperature 0, with the cleaned `Title`, `Description` and `Content` of the article appended on their own lines. The model must return only a single integer from 1 to 10, which becomes the raw `ai_grade`.
|
|
|
|
```
|
|
You are an expert content strategist and editor for DevPlace, a server-rendered social network for software developers. Decide how strongly a given article should be published, expressed as a single grade.
|
|
|
|
Site context:
|
|
- Audience: software developers, hobbyists and professionals, interested in software development, AI agents, open-source tools, systems programming, and self-hosting.
|
|
- Mission: deliver deep technical insight, critical analysis, and practical value; build authority and reader trust; avoid low-effort, rehashed, or clickbait content.
|
|
- Tone: technical, practical, nuanced, skeptical of hype; no sensationalism or unverified claims.
|
|
|
|
Evaluate the article against these weighted criteria and weigh them internally:
|
|
1. Relevance and alignment to the developer niche and audience needs (20%).
|
|
2. Quality and accuracy: depth, originality, factual correctness, sourcing, technical correctness, clear writing (25%).
|
|
3. Engagement and readability: strong title/intro/flow, useful examples, appropriate length (15%).
|
|
4. Originality and uniqueness: new perspective, data, or analysis; not duplicated generic web content (15%).
|
|
5. SEO and discoverability: keyword opportunity without stuffing, shareable (10%).
|
|
6. Ethics, legality and risk: no misinformation, plagiarism, or brand-safety issues; controversial topics handled with nuance and evidence (10%).
|
|
7. Strategic fit for the site (5%).
|
|
|
|
Convert your overall judgment to a single integer grade from 1 to 10:
|
|
- 9-10: excellent, publish as-is.
|
|
- 7-8: good, worth publishing.
|
|
- 5-6: marginal, weak fit or quality.
|
|
- 1-4: reject, low quality or off-topic.
|
|
|
|
Return ONLY a single integer from 1 to 10, nothing else.
|
|
|
|
Title: ...
|
|
Description: ...
|
|
Content: ...
|
|
```
|
|
|
|
## Tables it writes
|
|
|
|
| Table | Purpose |
|
|
|-------|---------|
|
|
| `news` | The article rows: `uid`, `slug`, `external_id`, `title`, `description` (cleaned, capped 5000), `url`, `image_url` (the chosen unique image), `has_unique_image`, `source_name`, `grade` (effective score), `ai_grade` (raw AI grade), `status` (`published` or `draft`), `featured`, `featured_locked`, `landing_locked`, `show_on_landing`, `content` (cleaned, capped 10000), `author`, `article_published`, `synced_at`. Existing rows are updated in place by `external_id` (locked Featured stays untouched). |
|
|
| `news_sync` | One row per external id tracking `status` (`graded`, `grading_failed`, or `rejected_quality:<reason>`) and `synced_at`, so an article is graded only once. |
|
|
| `news_images` | Images fetched per article, keyed by `news_uid`, each with `url`, `alt_text`, `phash`, `width`, `height`, and `is_placeholder`; replaced wholesale when the article is updated. |
|
|
| `news_usage` | Single-row rolling AI usage totals (grading and formatting calls, tokens, cost, and latency) accumulated per run via `add_news_usage`; surfaced as the service `stats` metric cards. |
|
|
|
|
## Data it offers
|
|
|
|
NewsService overrides `collect_metrics()` to return a `stats` block of AI usage cards rolled up from the `news_usage` table (grading and formatting calls, tokens, cost, and the per-call averages, the same shape as the correction and modifier usage tables). Framework health is surfaced alongside it at `GET /admin/services/data`: `status` (`running` / `stopped` / `stalled`), `last_run`, `next_run`, `heartbeat`, and the `log_buffer`, where each run records its new/updated/draft/failed/skipped summary. The graded articles are the product, visible under `/news` and in the `news` table.
|
|
</div>
|