This file documents the async job services subsystem (devplacepy/services/jobs/ and its subdirectories deepsearch/, isslop/, seo/) - the standard pattern for running heavy blocking work off the request path. Claude Code loads it automatically whenever a file under this directory is read or edited.

## Overview: the JobService pattern

The standard way to run heavy, blocking work off the request path and hand back a result URL, used by every job kind in this subsystem (`zip`, `planning`, `fork`, `seo`, `seo_meta`, `deepsearch`, `isslop`). Reference: admin docs `Architecture -> Async job services` and `Services -> ZipService`.

- **The DB row is the queue.** `services/jobs/queue.py` (`enqueue`, `get_job`, `touch_job`, `list_jobs`) is pure DB and callable from any worker's handler - enqueue is a fast, non-blocking insert, status reads work from any worker, and only the lock owner processes. All kinds share ONE `jobs` table discriminated by a `kind` column: common lifecycle timestamps, `retry_count`, `last_accessed_at`/`expires_at`, `bytes_in`/`bytes_out`/`item_count` stat columns, plus `payload`/`result` JSON columns where kind-specific fields live.
- **`JobService(BaseService)`** runs only in the lock owner. Each `run_once`: reap finished in-flight tasks -> recover orphaned `running` rows (left by a dead owner, since there is only one processor) back to `pending` with bounded `retry_count` -> refill up to `max_concurrent` oldest `pending` jobs (uuid7 sorts FIFO) as `asyncio` tasks -> sweep rows past `expires_at` via the per-kind `cleanup()` hook. In-flight tasks span ticks (kept in an instance map); the loop returns immediately each tick, so keep the interval short (default 2s).
- **No atomic claim is needed** (single processor by construction) and **no separate reaper exists** - retention is built into every job service, default 7 days, admin-configurable; downloading/reading a result extends `expires_at` via `touch_job`.
- **Add a kind:** subclass `JobService`, set `kind`, implement `async process(self, job) -> dict` (return the `result` dict incl. stat keys) and `cleanup(self, job)`; register the service instance in `main.py`; add enqueue endpoints that own authz, a status route, a download/report route, a Devii tool, and docs. **A new `JobService` needs a server restart to go live.**
- **Permanent vs expiring artifacts (load-bearing distinction - decide this per kind).** Most kinds produce a DISPOSABLE artifact that expires with the retention sweep: `ZipService`, `SeoService`, and `DeepsearchService` all delete their real output (archive / report+screenshots / vector collection+report) inside `cleanup()`. Two kinds are different: `ForkService`'s artifact is a **permanent project**, and the AI Usage Analyzer's (`isslop`) artifacts (analysis, events, file/image results, report, badge) are **permanent public capability URLs** - both make `cleanup()` a no-op on the real artifact and let the retention sweep remove only the `jobs` tracking row. Get this decision right for any new kind: if the output should outlive the job the way a fork or an isslop report does, do not wire `cleanup()` to delete it.

## ZipService (kind `zip`)

- `process` materializes a project subtree via `project_files.export_to_dir` (staging under `config.DATA_DIR/zip_staging/{uid}`), compresses it in a **subprocess** (`python -m devplacepy.services.jobs.zip_worker`, stdlib-only, prints stats JSON incl. crc32), names the output `{crc32}.{slug}.zip` under `config.DATA_DIR/zips/{uid[-2:]}/{uid[-4:-2]}/` (sharded on the random tail of the uuid7 via `attachments._directory_for`, NOT its time-ordered head), and removes staging.
- Runtime artifacts live in `DATA_DIR` (`data/` by default), OUTSIDE the package and NOT under `/static`; the download is served by the `/zips/{uid}/download` route via `FileResponse`, not the static mount.
- Enqueue endpoints own authz: `POST /projects/{slug}/zip`, `POST /projects/{slug}/files/zip?path=`.
- `GET /zips/{uid}` (status `ZipJobOut`) and `GET /zips/{uid}/download` (FileResponse, extends expiry via `touch_job`) are **capability URLs** scoped only by the unguessable uuid7, not by owner - the owner is stored for attribution, not access control, matching publicly-viewable projects.
- Frontend `app.zipDownloader` (`static/js/ZipDownloader.js`) auto-wires any `data-zip-download` element plus the files context menu: POST -> poll `/zips/{uid}` via the shared `JobPoller` -> trigger download.
- Devii tools `zip_project`/`zip_status`; CLI `devplace zips prune|clear`.
- Disposable: `cleanup()` deletes the archive and the retention sweep prunes both the file and the `jobs` row.

## Planning report generator (kind `planning`, `services/jobs/planning_service.py`)

`PlanningReportService` is **admin-only** and builds a complete, phased markdown implementation document from a selectable set of open Gitea tickets, intended to be handed straight to a coding agent for one-shot execution, off the request path via the same async-job pattern as zip.

- `process` collects open tickets via `services/gitea/planning.py` `collect_open_issues(client, limit=MAX_ISSUES)` (the paginated `list_issues(state="open", limit=50)` loop, cap 50; reused by the admin page too), narrows them to the **selected ticket numbers** carried in the job payload (`{"numbers": [int, ...]}`, preserving selection order - an empty/absent list keeps all open, so the no-arg Devii action and any legacy enqueue stay backward compatible), and builds the markdown via `generate_plan(issues, config)` (an AI-or-fallback helper mirroring `enhance.py`).
- **AI path:** passes each ticket's FULL, verbatim description (`_body`, capped only at the `BODY_MAX=40000` per-ticket safety limit, not the old 600-char excerpt) to the internal gateway via `stealth.stealth_async_client` with a high output budget (`MAX_TOKENS=32000`, `PLAN_MAX=600000` final cap, `PLANNING_TIMEOUT_SECONDS=600` client timeout) and a `SYSTEM_PROMPT` that demands a self-contained document: a `## Execution Order` list, then `## Phase K: <name>` headings, and under each phase a `### #N <title>` subsection per ticket carrying labelled **Original ticket** (verbatim blockquote), **Goal**, **Dependencies**, **Affected areas / files**, **Implementation steps**, **Acceptance criteria**, and **Risks / open questions** blocks - preserving every detail with nothing summarised away.
- **Because an LLM can still summarise or truncate, the AI document is never trusted to be complete on its own:** `generate_plan` always appends a deterministic `# Appendix: Source Tickets (verbatim)` section built straight from the issue dicts by `planning.py` `verbatim_tickets(issues)` (per ticket `## #N <title>`, labels, the `html_url` source link, and the full body as a blockquote), so every covered ticket's full text is guaranteed present inline AND in the appendix - the document is genuinely self-contained.
- Fail-soft `_fallback` (when `ai_enhance` is off or any `httpx.HTTPError`/`ValueError`/`KeyError`/`IndexError` occurs) groups by primary label then ticket number, emitting the same `## Execution Order` + `## Phase K` shape with each ticket's full description reproduced verbatim (already self-contained, so no extra appendix).
- Writes the markdown to `config.PLANNING_REPORTS_DIR/_directory_for(uid)/{crc32}.{slug}.md` (sharded on the uuid7 random tail, crc32 over the markdown bytes), and returns `{download_url, local_path, final_name, markdown, ai_used, item_count, bytes_out}`. The `markdown` string is carried in the result so the status route can hand it to the renderer without re-reading disk. `cleanup` unlinks the file (retention prunes only the artifact + job row).
- Audit `issue.planning.request` (enqueue) and `issue.planning.generate` (success/failure) via `record`/`record_system` with an `audit.job(uid)` link.
- Routes (all `require_admin`): `POST /issues/planning` (enqueue, `503` when Gitea unconfigured, `PlanningForm` with a comma-separated `numbers` field parsed to `list[int]` and stored in the payload, returns `{uid, status_url}`), `GET /issues/planning/{uid}` (`PlanningJobOut`: status, `markdown`, `download_url`, `issue_count`, `ai_used`), `GET /issues/planning/{uid}/download` (`FileResponse`, `text/markdown`, traversal-guarded against `PLANNING_REPORTS_DIR`, extends retention via `touch_job`). The `/issues/{number}` detail route uses the `{number:int}` path converter so `/issues/planning` is never captured by it.
- Admin entry point: `GET /admin/issues/planning` (`routers/admin/issues.py`, `admin_issues_planning.html`) fetches the open tickets server-side via `collect_open_issues` (wrapped so a Gitea/network failure renders an empty state, never a 500) and renders a **selectable checkbox list** (all checked by default, a select-all/none master, and a live selected count) as the step in between, plus the **Generate planning** button, the `JobPoller` status panel, the `<dp-content>` render target, and a **Download** anchor; the issues listing shows an admin-only **Generate planning** link (`viewer_is_admin` in `issues_page`).
- Frontend driver `static/js/PlanningGenerator.js` (`app.planningGenerator`): tracks the checkbox selection (master toggle, count, disables Generate at zero selected), sends the chosen numbers as the `numbers` form field, POST -> `JobPoller.run` (widened to `intervalMs:2000, maxAttempts:400` = ~800s so it outlives the longer detailed generation) -> replace the `<dp-content>` with a fresh one holding `status.markdown` so it re-renders, set the download href.
- Devii tool `planning_report_generate` (`requires_admin=True`). The renderer's built-in copy button (the `dp-content` **Copy** button styled in `markdown.css`, identical to the per-code-block copy button) satisfies the copy-source requirement and is opt-out via the `no-copy` boolean attribute on `<dp-content>`.

## Project fork - ForkService (kind `fork`, `services/jobs/fork_service.py`)

Forking copies a source project into a brand-new project owned by the forking user, off the request path via the same async-job pattern as the zip flow.

- **Enqueue:** `POST /projects/{slug}/fork` (`routers/projects/index.py`) - `require_user` then `can_view_project(source, user)` (any logged-in user may fork any project they can view: public projects and their own private ones). Body is `ForkForm{title}` (the destination project name). It enqueues a `fork` job (`{source_project_uid, title, forked_by_uid}`, owner `("user", uid)`) and returns `{uid, status_url}`. No work happens on the request path.
- **`process`** looks up the source project row and the forking user row (raises -> job `failed` if either is missing), creates the destination project with `content.create_content_item("projects", "project", user, fields, ...)` (so slug/XP/logging match a normal create; metadata - `description`, `project_type`, `platforms`, `status`, dates, `is_private` - is copied from the source, `read_only` reset to 0), copies the whole virtual FS off-thread (`export_to_dir(source, "", staging)` -> `import_from_dir(new_uid, staging, user, skip_names=set())`, an **exact** copy including binaries, staging under `config.DATA_DIR/fork_staging/{uid}`), then records the relation with `database.record_fork(source_uid, new_uid, forked_by_uid)`. Result: `{project_uid, project_url, source_project_uid, item_count}`.
- **Rollback:** any failure after the project is created triggers `_rollback(new_uid)` (`project_files.delete_all_project_files` + `delete_fork_relations` + delete the `projects` row) before re-raising, so a failed fork leaves no orphan project.
- **Cleanup is a no-op on the project.** Unlike zip's disposable archive, a fork's artifact is a **permanent project** that must outlive the job. `cleanup()` only removes any leftover staging dir; the retention sweep deletes the job tracking row, never the forked project. `devplace forks prune|clear` likewise deletes job rows only.
- **Relation model:** the `project_forks` table (`uid`, `source_project_uid`, `forked_project_uid`, `forked_by_uid`, `created_at`) records direction explicitly (source -> forked), indexed on both project columns. Helpers in `database.py`: `record_fork`, `get_fork_parent(forked_uid)` (the source project row, for the "Forked from X" link on the project detail page), `count_forks(source_uid)`, and `delete_fork_relations(project_uid)` (called on project delete in `content.delete_content_item`, and in fork rollback).
- **Frontend:** `app.projectForker` (`static/js/ProjectForker.js`) wires `data-fork-project` (a Fork button on the project detail page, visible to any logged-in user): prompt for a name via `app.dialog.prompt` -> `Http.sendForm` POST -> `JobPoller.run("/forks/{uid}", ...)` -> on `done` redirect to `project_url`. The forked project's detail page shows a "Forked from X" link (`database.get_fork_parent`).
- **Status route** `GET /forks/{uid}` (`routers/forks.py`, `ForkJobOut`) exposes `project_uid`/`project_url`/`source_project_uid` only once `status == done`. Devii tools `fork_project`/`fork_status`; docs `projects-fork`/`forks-status`.

## SEO Diagnostics tool - SeoService (kind `seo`, `services/jobs/seo/`, `routers/tools/`)

The public **Tools -> SEO Diagnostics** auditor crawls a URL or sitemap with a headless browser and runs a broad battery of SEO checks, on the **same async-job pattern as zip/fork** plus a live websocket. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap, and the shared SSRF guard.

- **Surface:** a collapsible **Tools** dropdown in `base.html` (desktop center nav + a mobile section, visible to everyone) toggled by `MobileNav.initToolsDropdown`. `GET /tools` lists tools; `GET /tools/seo` is the auditor page (`static/js/SeoDiagnostics.js` -> `app.seoDiagnostics`, instantiated page-side in the template, not in `Application.js`).
- **Enqueue:** `POST /tools/seo/run` (`routers/tools/seo.py`, body `SeoRunForm{url, mode: url|sitemap, max_pages 1-50}`). Owner is `("user", uid)` or `("guest", X-Real-IP)`. It rejects with `429` if the owner already has a pending/running `seo` job, then enqueues `{url, mode, max_pages, allow_private:False}` and returns `{uid, status_url, ws_url}`.
- **`process`** writes the payload to `config.SEO_REPORTS_DIR/{uid}/payload.json`, launches `python -m devplacepy.services.jobs.seo.worker <payload_json> <output_dir>` via `create_subprocess_exec` (high `limit=` so big lines never overflow the StreamReader), reads **NDJSON frames from stdout** line by line (stage/target/progress/page/site_checks/report_ready), forwards each into the in-process **`ProgressHub`** (`services/jobs/seo/progress.py`, uid -> set of `asyncio.Queue`), and on completion loads `output_dir/report.json` as the job result. `cleanup()` clears the hub buffer and removes the report dir.
- **Worker** (`worker.py`, subprocess): `crawler.crawl_target` resolves the target (single URL, or sitemap `<loc>` URLs capped at `max_pages`) and fetches `robots.txt`/`sitemap.xml`/`llms.txt` with `httpx`. The audited target host is guarded once with `net_guard.guard_public_url`; candidate URLs **sharing that host are pre-approved** (no redundant per-URL `getaddrinfo` - a transient DNS failure or a self-hosted server resolving its own domain must not blank the whole crawl), and only cross-host sitemap entries are re-guarded. **In sitemap mode the crawler never falls back to auditing the sitemap document itself**: if no page URLs survive it raises a clear error (a stray `or [target]` fallback previously rendered the sitemap XML as one 56k-node "page" with no title/H1). For each page it launches one Playwright navigation: a single `page.evaluate(EXTRACT_SCRIPT)` returns the whole DOM contract (title/metas/canonical/headings/images/links/jsonld/og/twitter/semantic/mixed-content/word-count), an injected `PerformanceObserver` (`add_init_script(INIT_SCRIPT)`) captures LCP/CLS, navigation timing gives TTFB/FCP/transfer/protocol, a mobile-viewport pass measures overflow/tap-targets, a screenshot is saved, and a raw `httpx` GET supplies the SSR HTML for the rendered-vs-server parity check.
- **Check registry** (`checks/`): `base.py` defines the `Check`/`PageContext`/`SiteContext` dataclasses and the `@page_check`/`@site_check` decorators (collected into `PAGE_CHECKS`/`SITE_CHECKS`); one module per category (`crawl`, `meta`, `headings`, `links`, `structured_data`, `social`, `performance`, `mobile_a11y`, `security`, `ai_readiness`, `crosspage`). `registry.run_page_checks`/`run_site_checks` run them defensively (one failing check never aborts a page), and `compute_score` produces a severity-weighted overall score + grade and per-category subscores. **To add a check:** write a function decorated `@page_check`/`@site_check` in the right category module and import that module in `registry.py`.
- **Live progress WS:** `WS /tools/seo/{uid}/ws` is served **only by the service-lock owner** (closes `4013` for a fast retry on a non-owner worker, like `/devii/ws`); it replays `hub.snapshot(uid)` then streams `hub.subscribe(uid)`. The frontend `SeoProgressSocket` mirrors the `DeviiSocket` 4013/reconnect pattern. **The `done` frame carries the full report inline** (and the router's late-join terminal branch reads it from `job.result`); the client renders from `frame.report` directly. This is load-bearing: `service.process` publishes `done` from inside `process()`, BEFORE the JobService base persists the result on the next reap tick (~2s later), so a client that fetched `/tools/seo/{uid}/report` on the `done` signal would race the DB write and get an empty (all-zero) report. **Do not "simplify" this back into a fetch-on-done.** After publishing `done`, `process()` calls `hub.clear(uid)` to bound the in-memory buffer (late reconnects fall back to the DB-backed terminal branch).
- **Status/report routes:** `GET /tools/seo/{uid}` (`SeoJobOut`), `GET /tools/seo/{uid}/report` (`respond(..., SeoReportOut)`, HTML or JSON), `GET /tools/seo/{uid}/screenshot/{n}` (FileResponse from `SEO_REPORTS_DIR`, path-guarded). All are **capability URLs** scoped by the unguessable uuid7. The full report is written to `config.SEO_REPORTS_DIR/{uid}/report.json` (NOT inlined on a stdout line - avoids the StreamReader limit). Devii tools `seo_diagnostics`/`seo_status`/`seo_report` (public); docs `tools-seo-*`; CLI `devplace seo prune|clear`; audit `seo.run.request|complete|failed` (category `tools`).
- **SSRF guard is shared:** `devplacepy/net_guard.py` (`guard_public_url`, `is_blocked_address`, `effective_address`) was extracted from the Devii fetch controller, which now imports it; the crawler reuses it. **`playwright` is a core dependency** (Chromium installed by `make install` / the Docker image).
- Disposable: `cleanup()` removes the report dir; retention prunes both the artifacts and the `jobs` row.
- **Production nginx needs a dedicated WS location.** In `nginx/nginx.conf.template` the catch-all `location /` sets `Connection ""` (no upgrade) and a 60s timeout, so any websocket route that falls through to it fails the handshake (browser: `WebSocket connection failed`, no close code). The progress socket has its own `location ~ ^/tools/seo/[^/]+/ws$` block forwarding `Upgrade`/`Connection` with a 1h timeout, mirroring `/devii/ws`. **Any new websocket path must add its own upgrade `location` above `location /`** (see docs `Production -> nginx`).

## SEO metadata service - SeoMetaService (kind `seo_meta`, `services/jobs/seo_meta_service.py`, `services/seo_meta.py`, `seo_meta_text.py`)

`SeoMetaService` generates a clean, SEO-optimized title/description/keywords for every published content item (types `post`, `project`, `gist`, `news`, `issue`) off the request path and **meters its own AI spend**. It is a distinct concern from the public **Tools -> SEO Diagnostics** auditor (kind `seo`) - do NOT conflate the two kinds; they share only the `jobs` queue table. It is the **constructive counterpart** to the diagnostics tool: diagnostics audits a URL, this one populates the on-page metadata. It is a `JobService` like `ForkService`: the artifact (the `seo_metadata` row) is **permanent**, so `cleanup()` is a no-op and the retention sweep removes only the job tracking row.

- **Tables.** `seo_metadata` is polymorphic and soft-deletable (in `SOFT_DELETE_TABLES`, born-live `deleted_at`/`deleted_by`): `uid, target_type, target_uid, seo_title, seo_description, seo_keywords, status (ready|pending|failed), source (ai|plain), generated_at, created_at, updated_at`, keyed UNIQUE on `(target_type, target_uid)`. `init_db()` ensures every column, the UNIQUE `idx_seo_metadata_target` and the live `idx_seo_metadata_status (status, deleted_at)` index. `seo_usage` is a single-row config-like usage table (NOT soft-delete) mirroring `news_usage`, keyed `SEO_USAGE_KEY="seo_meta"`. Helpers in `database.py`: `get_seo_metadata`/`get_seo_metadata_batch`/`has_fresh_seo_metadata`/`upsert_seo_metadata`/`mark_seo_metadata_stale` (every read filters `deleted_at IS NULL`; `get_seo_metadata` returns only `status="ready"` live rows) and `add_seo_usage`/`get_seo_usage`.
- **Choke helper.** `services/seo_meta.py` `schedule_seo_meta(target_type, uid, regenerate=False)` and `schedule_seo_meta_for_table(table, uid, ...)` are import-cycle-free (only `database` + `queue`). They no-op for unknown types, missing uid, or (without `regenerate`) when a fresh `ready` row exists (`database.has_fresh_seo_metadata`); `regenerate=True` marks the row stale first (`database.mark_seo_metadata_stale`); both skip a target with an existing pending/running `seo_meta` job; otherwise `queue.enqueue("seo_meta", {target_type, target_uid}, "system", "seo_meta")`. Hooked at `content.create_content_item` (create, no-op guard) and `content.edit_content_item` (regenerate), the news publish sites in `services/news.py` (`status=="published"` only; existing-row update path uses `regenerate=True`), and `IssueCreateService` after the Gitea ticket is recorded. Because the work is async via the queue (NOT `run_in_executor`), the helper only enqueues.
- **`process`** loads the target row (posts/projects/gists/news via `get_table`, issues via `gitea.store.get_ticket`), builds grounding via `services/ai_context.build_context` (fail-soft for news/empty `user_uid`), and calls the gateway **off-thread** (`asyncio.to_thread(correction.gateway_complete, internal_gateway_key(), system, source_text, timeout)` - the synchronous gateway call posts to the in-process gateway on localhost, so it MUST run via `to_thread` or it self-deadlocks the single worker - the same lesson as `correction.py`'s sync mode). It demands strict JSON `{seo_title, seo_description, seo_keywords}`, parses fail-soft, re-clamps every field server-side via `seo_meta_text.clamp_generated`, and falls back to `seo_meta_text.plain_seo_defaults` (status `failed`, source `plain`) on any failure - **the fields are never empty**. Usage accumulates via `correction.new_usage_totals` and flushes once with `database.add_seo_usage(totals)` when `calls>0` (the single-row `seo_usage` table, mirroring `news_usage`). It emits an audit `seo.meta.generate`/`seo.meta.failed` (`record_system`, category `tools`) and `upsert_seo_metadata(...)`.
- **Backfill.** `run_once` calls `super().run_once()` then a bounded backfill sweep (gated by `seo_meta_backfill_enabled`, `seo_meta_backfill_batch` per type per tick) over published content lacking a fresh `ready` row, so pre-existing items get metadata with no one-shot migration.
- **Admin surface.** `collect_metrics()` merges the `JobService` job-pipeline stats with `usage_metric_cards(get_seo_usage())`, so the **SEO Metadata** card on `/admin/services` shows both the live task pipeline and the AI cost/averages; the existing `admin.services.{name}` pub/sub topic + `live_view_relay` row pushes it live with NO new VIEWS row. A standard-paginated task list reads `queue.list_jobs(kind="seo_meta")` with `database.build_pagination` + `_pagination.html` when a dedicated page is desired.
- **Clamps (single source of truth, `seo_meta_text.py`):** `seo_title` hard cap 60 (word-boundary, single hyphen, keyword front-loaded); `seo_description` hard cap 160 (word-boundary via `seo.truncate`, key message in the first 120 chars); `seo_keywords` 5-8 distinct lowercase comma-joined terms (the `<meta keywords>` tag is dead for Google but the feature mandates it - emit a SHORT honest list, never stuffed). `plain_text_from_markdown` reuses `rendering._render_content` + `utils.strip_html` so markdown (and em-dash) never leaks.
- **SEO consumption fix (`seo.py` `base_seo_context`).** The meta description is now markdown-stripped (`plain_markdown`/`plain_text_from_markdown`, fixing the prior raw-markdown leak), `meta_keywords` is emitted (a safe plain string, NOT a Jinja-global name), and a `seo_target=(target_type, target_uid)` param makes it consume the ready `seo_metadata` row when present, else `plain_seo_defaults`. `base_seo_context`'s new `keywords`/`seo_target` params are OPTIONAL with safe defaults so existing callers are unaffected; the five detail routers (posts/projects/gists/news/issues) pass `seo_target`. `og_title`/`twitter:title` use the bare `seo_title` (drops the redundant " - DevPlace" suffix in social cards); `base.html` adds `<meta name="keywords">`, `og:image:width/height/alt` and `twitter:image:alt`. The per-type JSON-LD (`discussion_forum_posting`, `software_application_schema`, `news_article_schema`, `software_source_code_schema`) route their text/description through `plain_markdown`.
- **Fan-out.** Schema `SeoMetaOut`; read route `GET /tools/seo-meta/{target_type}/{target_uid}` (`routers/tools/index.py`, public, JSON) returning the ready row or a plain default with status `pending`; Devii action `seo_meta_status` (public, read-only) + docs `tools-seo-meta-status`; CLI `devplace seo-meta prune|clear` (job rows only; the metadata persists); events `seo.meta.generate|failed`. Registered in `main.py` alongside `SeoService`. **A new `JobService` needs a server restart to go live.**

## DeepSearch tool - DeepsearchService (kind `deepsearch`, `services/jobs/deepsearch/`, `services/deepsearch/`, `routers/tools/deepsearch.py`)

The public **Tools -> DeepSearch** researcher is a multi-agent deep web researcher built on the **same async-job + ProgressHub + 4013-WS pattern as the SEO tool**, plus a per-session vector store and a grounded RAG chat. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap (1-30), depth cap (1-4), and the shared SSRF guard. Reuse the SEO tool as the template for any new Tools async job.

- **Owner helper is shared:** `routers/tools/_shared.py` `owner_for(request)` returns `("user", uid)` or `("guest", X-Real-IP)`; both `seo.py` and `deepsearch.py` import it (do not re-inline the owner derivation).
- **Enqueue:** `POST /tools/deepsearch/run` (body `DeepsearchRunForm{query, depth 1-4, max_pages 1-30}`). It rejects with `429` if the owner already has a pending/running `deepsearch` job. It resolves the **logged-in user's `users.api_key`** (guests use `database.internal_gateway_key()`) into the job payload for per-user embedding/LLM spend attribution, generates the uid up front, writes a `deepsearch_sessions` row (`create_deepsearch_session`), enqueues the job carrying `{query, depth, max_pages, api_key, collection}`, and returns `{uid, status_url, ws_url}`. The enqueue uses a local `_enqueue` (not `queue.enqueue`) so the session uid and the job uid match.
- **`process`** writes `control.json` (state `running`) + `payload.json` (augmented with the cross-session `cached_hashes`) under `config.DEEPSEARCH_DIR/{uid}`, launches `python -m devplacepy.services.jobs.deepsearch.worker <payload_json> <output_dir>` via `create_subprocess_exec` (high `limit=`), pumps **NDJSON stdout frames** into the in-process `ProgressHub` (`progress.py`), and on completion loads `output_dir/report.json`, persists the URL cache (`upsert_deepsearch_url_cache`), and updates the session row. `cleanup()` **drops the ChromaDB collection** (`VectorStore.drop`) and removes the session dir. Disposable: the collection + report dir are both deleted, unlike Fork/isslop.
- **Worker pipeline** (`worker.py`, stdlib + httpx + playwright, importable subprocess): `enhance.plan_queries` (gateway -> JSON sub-queries, deterministic fallback) -> `crawl.search_queries` (rsearch via standalone httpx, never `PlatformClient`; per-query result buckets are **round-robin interleaved** so every planned angle contributes pages, never just the first query) -> `crawl.crawl` (batches of `CRAWL_CONCURRENCY` concurrent httpx fetches then playwright render fallback, `guard_public_url` on the URL and every redirect, content-hash + URL-hash dedup; **`depth` follows in-page links**: after each level the links of every crawled page are scored by query-token overlap via `extract.relevant_links` and the top `LINKS_PER_PAGE` unseen ones form the next level, `depth=1` disables following) -> `chunking.chunk_text` -> `embeddings.embed_texts` (gateway, **local hashing fallback** when unavailable) -> `store.VectorStore.add` (Chroma) -> `orchestrate.orchestrate` (retrieval-grounded agents, see below). The worker writes `report.json` and `url_cache.json` and emits a `report_ready` frame carrying `synthesis`.
- **Search-provided content is a first-class source (`crawl.py`, the second junk-report fix).** `search_queries` calls rsearch with `content=true`, so each candidate carries the search engine's own readable `content`/`description` extract. This matters because the top sources for many questions are **bot-hostile** (X/Twitter, YouTube, Reddit, Facebook, Instagram, LinkedIn, TikTok - `HOSTILE_DOMAINS`): a headless fetch of those hits a login/consent wall ("Before you continue to YouTube", "Sign in to X") and yields near-zero text, which is why a 12-source run used to collapse to ~14 chunks. Now `crawl._resolve_candidate` **skips the fetch entirely for a hostile domain and uses the rsearch snippet** (`_snippet_page`, `source="search"`, `SNIPPET_MIN_CHARS` floor), and for every other domain it fetches normally but keeps the rsearch snippet as a **floor** (uses whichever of crawl-text vs snippet is longer), so a walled or thin page still contributes its real content instead of being dropped. This alone took a query from "cannot be answered" to a correct cited answer (14 -> 61 chunks, diversity 0.333 -> 0.75). **Never revert `content=true` and never send a headless render at a `HOSTILE_DOMAINS` host.**
- **Content extraction (`extract.py`, stdlib only):** `extract_html(raw, base_url)` is a readability-grade `HTMLParser` extractor used by both the httpx and playwright fetch paths (the old naive regex tag-stripper produced nav/cookie-banner boilerplate as "content" - the historic root cause of junk reports). It skips `script/style/nav/header/footer/aside/form` and ARIA `role=navigation|banner|contentinfo|...` regions, prefers `<article>`/`<main>` when they carry at least `MIN_CONTENT_TOTAL` chars, drops link-dense blocks (`MAX_LINK_DENSITY`, menus) and sub-`MIN_BLOCK_CHARS` fragments, unescapes entities, and emits real paragraphs joined by blank lines - which also makes `chunking.chunk_text`'s paragraph split actually fire (the flattened text used to be sliced mid-sentence). It also returns the page's `(url, anchor_text)` links (absolute, deduped, nav links excluded) for depth crawling; `relevant_links(links, query, limit)` scores them by query-token overlap and filters non-document extensions.
- **No "gaps"/critic agent (removed - do not reintroduce).** DeepSearch used to run a fourth "critic" agent that produced an "Open gaps" list. It was removed end-to-end (orchestrate, worker report, `DeepsearchSessionOut`, router context, session template, markdown/HTML export, `DeepsearchTool.js` agent labels, CSS) because it routinely emitted misleading, self-contradictory gaps on correctly-cited reports (the historic cause was that the critic was fed a truncated, retrieval-ordered source slice that dropped cited source numbers out of its window, so it fabricated "citation [n] is not in the report / no evidence provided"). **Do NOT reintroduce a `gaps` field or a critic agent.** The `_numbered_source_digest(pages)` helper survives and is used by the **linker** - a compact per-source `[n] title (url)\nexcerpt` block for EVERY page where the header line is always emitted even when the excerpt is trimmed, so all source numbers `1..N` are guaranteed present (never pass a raw `context[:N]` slice to an agent that reasons about source numbers). Regression: `tests/unit/services/jobs/deepsearch/orchestrate.py::test_linker_receives_full_source_list` / `::test_numbered_source_digest_keeps_every_source_number_under_cap`.
- **Orchestration (`orchestrate.py`) is retrieval-grounded and markdown-first.** The worker indexes BEFORE analysis and passes the `VectorStore` + planned queries to `orchestrate`, which embeds the question and each sub-query and pulls `hybrid_search` top chunks (round-robin merged, up to `CONTEXT_CHUNKS_MAX`), building the context from the RETRIEVED passages grouped per source - the source numbers `[n]` align with the report's `sources` list (page order), so inline citations, finding citations, and the rendered numbered source list agree. Page-head excerpts are only the fallback when retrieval is empty. Synthesis is **two-step to avoid the markdown-inside-JSON trap**: the summarizer writes a plain markdown report (`REPORT_MAX_TOKENS`, retried once if empty), then a separate `extractor` agent returns the findings JSON (retried once, tolerant `_parse_json` handles code fences and trailing garbage); the linker (confidence) failure is caught and never discards the report. The three agents in the pipeline today are summarizer, extractor, and linker - there is no fourth "critic" stage. Only a failed/empty report falls back to `_heuristic`, which stamps `synthesis="heuristic"` and emits a `status:"failed"` agent frame - the degradation is VISIBLE: `report.synthesis` flows through `DeepsearchSessionOut.synthesis`, the session template renders a "Degraded report" banner (`.ds-degraded`), and the markdown export carries the same note. A successful run stamps `synthesis="agents"`. **Never re-inline synthesis into a single JSON blob and never let a synthesis failure ship silently.**
- **PDF ingestion (`crawl.py` `fetch_page` + `pdf.py`):** a crawled candidate is treated as a PDF when its `content-type` is `application/pdf`/`application/x-pdf`, its URL path ends in `.pdf`, or its first bytes match the `%PDF-` magic (`pdf.is_pdf`). `fetch_page` streams the body and caps it at `MAX_PDF_BYTES` (15 MB); for a PDF it calls `pdf.extract_pdf_text`, which writes the raw binary to a `tempfile` temp location (cleaned up via `Path.unlink(missing_ok=True)` in `finally`), parses it with `pypdf` (`PdfReader`, capped at `MAX_PDF_PAGES` = 50, title pulled from metadata), and normalizes whitespace. The resulting `CrawledPage` carries `source="pdf"`; PDFs skip the Playwright fallback. Everything downstream is source-agnostic (chunking/embedding/orchestration read `page.text`/`page.source` unchanged), so no other module changes. New unpinned dep `pypdf` (pure-python, no system deps).
- **Pause/resume/cancel:** `POST /tools/deepsearch/{uid}/{pause|resume|cancel}` (owner-gated) rewrite `control.json`; the worker's `should_stop` callback polls it between source fetches (paused = sleep-loop, cancelled = stop). State lives in a file, not the job row, so the running-in-a-subprocess worker can read it without a DB round-trip.
- **Vector store (`services/deepsearch/store.py`):** `VectorStore` wraps `chromadb.PersistentClient(path=config.DEEPSEARCH_CHROMA_DIR)`, one collection per session (`ds_<uid>`). `Chunk` is the dataclass. `hybrid_search` blends cosine vector similarity with a BM25 keyword score (weights `HYBRID_VECTOR_WEIGHT`/`HYBRID_KEYWORD_WEIGHT`) over the candidate set, with optional metadata `where` filters. `embeddings.py` `embed_texts` calls the gateway embeddings endpoint and **falls back to a deterministic local hashing vector** on any failure (so the tool degrades, never breaks).
- **RAG chat (`services/deepsearch/chat.py` + `WS /tools/deepsearch/{uid}/chat`):** a dedicated lightweight loop (NOT the Devii hub), served **only by the service-lock owner** (closes `4013` for fast retry). Answers are grounded ONLY in the session collection via `hybrid_search`, cited inline, rendered client-side via `dp-content`. Turns persist to `deepsearch_messages` and audit `deepsearch.chat`. Frontend component `<dp-deepsearch-chat>` (`static/js/components/AppDeepsearchChat.js`) clones `AppDocsChat`'s framing but uses its own WebSocket to the chat path.
- **Status/report/export routes:** `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (`respond(..., DeepsearchSessionOut)`, HTML or JSON), `GET /tools/deepsearch/{uid}/export.{md,json,pdf}` (`services/deepsearch/export.py`; PDF via weasyprint). All are **capability URLs** scoped by the unguessable uuid7. **Viewer-flag discipline:** the session schema/context use `viewer_is_admin`/`viewer_owns` (never `is_admin`/`owns`) so a `respond()` context key never shadows a Jinja global (the same class of issue as the issues `/{number}` route). `tests/api/tools/deepsearch/session.py` guards the HTML render.
- **Completion race (load-bearing read-path fix).** The worker writes `report.json` to disk and `service.process` publishes the `done` frame **from inside `process()`**, but the `JobService` framework only commits `jobs.result`/`status=DONE` afterwards, in `_reap()` -> `_finish_done()` on a later tick. The frontend navigates to the session page the instant it receives `done`, so a read that keyed only off `jobs.status == DONE` returned an EMPTY report (`None` score, 0 sources) until a manual refresh. Fix: `_report_for(uid, job)` returns `job.result.report` when the job is `DONE` and non-empty, else falls back to the on-disk `report.json` (`_report_from_disk`, `DEEPSEARCH_DIR/{uid}/report.json`) - which exists before the `done` frame is ever sent - and returns `{}` only for a `FAILED` job or a genuinely still-running job with no report on disk. `_session_context` derives `done`/`status` from `bool(report)` (not raw job status), and the chat WS gate accepts `session.status == "done"` (set inside `process()` before the publish) as ready. `_export_report` reuses the same fallback. Regression: `tests/api/tools/deepsearch/session.py::test_session_reads_disk_report_before_result_commit`. **Any new read of a job result that a client reaches immediately after a `done`/`session_url` frame must use this same on-disk fallback, never bare `jobs.status`.**
- **Clickable inline citations (`services/deepsearch/citations.py`).** The report/findings carry `[n]` markers (and the model sometimes emits `[3][9][1-2]`); the `link_citations(html, source_count)` template global (registered in `templating.py`) rewrites each `[n]` and each `[a-b]` range into `<a class="ds-cite" href="#ds-source-n">[n]</a>` anchors that jump to the numbered `<li id="ds-source-n">` in the Sources list (source numbering is page order, matching the `[n]` the summarizer was given). It splits out `<a>`/`<code>`/`<pre>` regions first so markers inside links/code are left alone, expands ranges to individual links, and drops out-of-range numbers (no broken anchors). The session template nests it over the server render: `{{ link_citations(render_content(summary), sources|length) }}` and `{{ link_citations(finding.detail|e, sources|length) }}`, plus a per-finding `.ds-finding-cites` chip row from `finding.citations`. `.ds-cite`/`.ds-sources li:target` styling lives in `deepsearch.css`. The report prompt asks for one number per bracket (never a range) so output is consistent, but the linkifier handles ranges regardless. Regression: `tests/unit/services/deepsearch/citations.py`.
- **Tables (`deepsearch_sessions`, `deepsearch_messages` soft-deletable + in `SOFT_DELETE_TABLES`; `deepsearch_url_cache` GC-only):** columns are ensured in `init_db()` (every queried column) with indexes. Every insert writes `deleted_at:None/deleted_by:None`; every read filters `deleted_at IS NULL`.
- **Frontend** (do not hand-roll): `DeepsearchTool.js` (`app.deepsearchTool`) drives the form via `Http.send`, watches `DeepsearchProgressSocket` (cloned from `SeoProgressSocket`, 4013 retry), and wires pause/resume/cancel. `static/css/deepsearch.css` uses the design tokens and is mobile-responsive.
- **Progress frame protocol (append-only, the worker<->JS contract):** `phases.py` is the single source of truth for phase identity, shared by `worker.py` (emit) and `DeepsearchTool.js` (render). `PHASE_ORDER = [planning, searching, crawling, indexing, analysis, synthesis]`; `worker._stage(stage, message, phase)` emits BOTH the legacy `stage` frame (byte-identical to before) AND a parallel `phase` frame `{phase, index, total, label}` so the timeline strip advances. The first emitted frame carries `version:1`. Every other frame type and its keys: `substep` (planning angles, `phase`+`message`; also emitted by analysis grounding), `queries`, `candidates`, `rsearch`, `progress` (`done`/`total`/`url`/`depth`), `page_loaded` (now `source`/`render`/`depth`/`elapsed_ms`/`done`/`total`), `page_cached`/`page_skipped`/`page_duplicate` (now `reason`/`elapsed_ms`), `embed_batch` (`batch`/`total_batches`/`backend`/`done`/`total`, emitted before AND after each batch), `embed_done` (`backend`/`chunk_count`), `agent` (`agent` one of summarizer|extractor|linker, `stage`/`status` start|done|failed, with `elapsed_ms`/`tokens_in`/`tokens_out` on done), `report_ready` (now also `synthesis`), `done` (`session_url`), `failed` (`message`). **The contract is append-only: never rename or drop a frame type**; `service._run_worker` pumps every stdout line into the `ProgressHub` untouched, so new frame types reach the WS with no handler change. `tests/api/tools/deepsearch/index.py` is the append-only regression guard.
- **RAG-audit hardening (engine correctness, no route/schema change):** (1) **Embedding-dimension consistency** - gateway embeddings carry provider-native dims while `local_embed` is fixed 256-dim; the worker `_index_chunks` now decides the backend ONCE per job (the first gateway failure or non-gateway result forces local for ALL remaining batches), and `VectorStore.add` drops any vector whose length differs from the collection's established dim, so one collection never mixes dims (cosine search across mixed dims is corrupt). `embeddings.EmbedResult.dims` and `VectorStore.dims` (lazily probed from the collection) expose the dimension; `chat.retrieve` re-embeds the query locally and skips retrieval if it still cannot match the stored dim. (2) **Citation grounding** - `orchestrate` drops any finding with no citations, and the summarizer prompt forbids uncited claims and treats the QUESTION as data not an instruction (prompt-injection reduction via `_sanitize_question`); `chat._strip_unmatched_markers` removes any inline `[n]` marker that does not map to an emitted citation. (3) **Confidence calibration** - when only a single domain was crawled, `orchestrate` caps confidence by the source-diversity-derived ceiling (overconfidence guard). (4) `EmbeddingCache` is bounded at `EMBED_CACHE_MAX` to cap in-memory growth.
- Devii tools `deepsearch`/`deepsearch_status`/`deepsearch_session` (public); docs `tools-deepsearch`; CLI `devplace deepsearch prune|clear`; audit `deepsearch.run.request|complete|failed` + `deepsearch.chat` (category `tools`). **New dependencies:** `chromadb`, `weasyprint`, `pypdf` (all unpinned). New runtime dirs `config.DEEPSEARCH_DIR`/`DEEPSEARCH_CHROMA_DIR` are registered in `DATA_PATHS`. **Add a dedicated nginx WS `location` for `/tools/deepsearch/{uid}/ws` and `/chat`** above `location /` for production, like the SEO and Devii sockets.

## AI Usage Analyzer tool - IsslopService (kind `isslop`, `services/jobs/isslop/`, `routers/tools/isslop.py`)

The public **Tools -> AI Usage Analyzer** classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. It is built on the standard async-job pattern (a `JobService` running a subprocess worker), but its live channel is **pub/sub, not a dedicated WS route**: every worker event is published to `public.isslop.{uid}` AND persisted to `isslop_events`, and the frontend pairs the pub/sub subscription with an incremental `GET /tools/isslop/{uid}/events?after=SEQ` poll, so guests (who cannot subscribe to `public.*` unless `pubsub_allow_guests` is on) and reconnecting tabs replay from the durable trail. **Never rely on pub/sub alone for this tool: the DB event trail is the source of truth, pub/sub is the fast path.**

- **Engine layout:** `services/jobs/isslop/` holds `acquisition/` (git probe via `git ls-remote`, depth-1 clone with size preflight + live 3 GB kill guard, stealth Playwright website crawler with HTTP fallback, path-traversal-safe workspace helpers), `analysis/` (exclusion rules, stylometric metrics, language detection, per-repo baselines, `signals/` with one detector family per file, two-axis scoring), `agent/` (gateway LLM client, per-file classifier, vision reviewer, report writer with deterministic fallback), plus `pipeline.py` (the event-yielding run), `worker.py` (subprocess entry), `events.py` (frame protocol), `persistence.py` (`EventPersister` writes events/file results/image results/report and stamps the analysis row), `store.py` (all DB access), `badge.py` (SVG), `service.py` (`IsslopService`), `config.py` (all constants + `WorkerSettings`).
- **Worker contract:** `IsslopService.process` writes the worker payload (url + admin toggles + gateway endpoint/model/key) to `config.ISSLOP_RUNS_DIR/{uid}/payload.json`, resolves the workspace under `config.ISSLOP_WORKSPACES_DIR` (`workspace_for` rejects any path escaping the root), launches `python -m devplacepy.services.jobs.isslop.worker <payload> <workspace>`, and relays each NDJSON stdout line through `EventPersister.apply` (SQLite) then `pubsub.publish`. The workspace and run dir are removed in a `finally`; the pipeline also deletes the workspace itself as its final act, so **no acquired source survives an analysis** - only the report and its evidence rows.
- **AI through the gateway only:** `agent/llm.py` talks solely to `config.INTERNAL_GATEWAY_URL` with model `molodetz` and `database.internal_gateway_key()` (vision uses the same model - the gateway handles image parts). `review_available`/`vision_available` gate the AI and image stages; on any gateway failure the static engine remains authoritative and the report falls back to the deterministic composer. All HTTP (gateway, git size preflight, website fallback crawl) goes through `stealth_async_client`.
- **Artifacts are permanent, the job row is not.** Like `ForkService`, `cleanup()` never touches `isslop_analyses`/`isslop_events`/`isslop_file_results`/`isslop_image_results`/`isslop_reports` - the report and badge are public capability URLs meant to outlive the run; the retention sweep removes only the `jobs` tracking row. `devplace isslop clear` is the only bulk hard-delete (plus per-analysis `store.purge_analysis`).
- **Ownership and guest history sync:** the owner is `("user", uid)` or `("guest", DEVII_GUEST_COOKIE)` - NOT the tools `_shared.owner_for` IP fallback, because history must survive IP changes and be claimable. The page/list/run handlers mint the guest cookie when absent (same cookie as Devii/customization, one guest identity platform-wide). `_sync_guest_history` runs on page and list requests: when a signed-in user still carries a guest cookie, `store.claim_guest_analyses` re-owns those rows via UPDATE (a move, never a copy - no duplicate data). One active analysis per owner (`429` otherwise, audited `denied`).
- **Tables:** `isslop_analyses` is soft-deletable (in `SOFT_DELETE_TABLES`, born-live inserts, reads filter `deleted_at IS NULL`, indexed on `(owner_kind, owner_id, created_at)`/`status`/`content_hash`); the evidence tables (`isslop_events` keyed `(analysis_uid, seq)`, `isslop_file_results`, `isslop_image_results`, `isslop_reports` UNIQUE on `analysis_uid`) are GC-only evidence purged with their analysis. All ensured in `init_db()`.
- **Routes** (all under `/tools/isslop`, capability URLs): `GET ""` page, `POST /run` (`IsslopRunForm`, http/git/ssh URL pattern), `GET /list` (owner history), `GET /{uid}` (`IsslopAnalysisOut`), `GET /{uid}/events` (ordered replay), `GET /{uid}/report` (`respond(..., IsslopReportOut)` - HTML shows the live `<dp-isslop-run>` while running and the server-rendered report when completed; the markdown body goes through `render_content`), `GET /{uid}/report.md`, `GET /{uid}/badge.svg` (self-contained SVG, hardcoded colors by design - it must render on external sites). Badge/report URLs are absolute via `seo.site_url`.
- **Frontend:** two site-wide web components (`static/js/components/AppIsslop.js` `<dp-isslop>` = submit form + history list; `AppIsslopRun.js` `<dp-isslop-run>` = live progress feed), registered in `components/index.js`, light DOM, reusing `Http`/`Poller` and `app.pubsub`. Page CSS `static/css/isslop.css` (design tokens). On `done` the run component reloads the page so the report is the server-rendered (SEO/`render_content`) version, never a client re-render.
- **Verdict blending is per-file, never a global mean (load-bearing).** The AI review pass runs its per-file gateway calls concurrently (`pipeline.AI_REVIEW_CONCURRENCY`, semaphore-bounded like the image pass) and adjusts ONLY the files it actually reviewed: `pipeline.apply_ai_verdicts` blends each sampled file's static origin/quality with its own verdict (0.6/0.4), then the WHOLE repo is re-aggregated with the normal SLOC/criticality weights, and `scoring.ai_fraction` maps per-file origin scores through a smooth 35-65 ramp (never the old hard 45/55 buckets). **Never reintroduce a repo-level mean of the 12 sampled verdicts** - it hands a tiny sample a fixed 40% of the verdict, so the LLM's clustered hedging values (30/40/50) drown thousands of files of static evidence and unrelated projects converge on identical percentages (the real-world twin-84%-human defect). **Single source of truth for the final verdict:** the SCORE event, the DONE payload and `generate_report` all consume the SAME final `RepoScores` (image influence applied via `scoring.adjust_for_images`, which recomputes slop/grade/human together) - the summary grade and the report body grade can therefore never disagree; `tests/unit/services/jobs/isslop/pipeline.py` guards both invariants.
- **Image evidence thumbnails + retry-safe evidence.** Workspaces die with the run, so the vision stage persists an aspect-preserving WebP thumbnail per reviewed image (`vision.make_thumbnail`, sha1-of-relative-path name) into `config.ISSLOP_MEDIA_DIR/{uid}` (the dir comes to the worker via the payload `media_dir`); the `thumb` name rides the `image` event, is stored on `isslop_image_results`, and is served by `GET /tools/isslop/{uid}/media/{name}` (strict `^[a-f0-9]{16}\.webp$` name pattern + `is_relative_to` root check - never loosen either). The report page renders the thumbnails with `data-lightbox` (the shared `app.lightbox` opens them full-size) and the live feed shows a tiny inline preview per image event. **`store.reset_evidence(uid)` runs at the top of every `IsslopService.process`** - a retried job (orphan recovery) previously re-inserted its events/file/image rows, duplicating every image and file in the report; any new evidence table MUST be added to `reset_evidence` AND `purge_analysis`.
- **Template provenance (the "ships defaults" detector).** `analysis/templates.py` `detect_template(workspace)` scores starter-template evidence repo-wide (it reads files the inventory excludes, like `package.json`): known template slugs/authors in the manifest (+ `ct3aMetadata`), README template marketing (weights capped so a wordy README cannot alone confirm), and the kitchen-sink scaffold constellation (count of standard scaffold artifacts past 3 freebies). The saturating score feeds `scoring.adjust_for_template` - a no-op below 35, a 0.7x floor on ai_percent/origin when confident, 0.85x when >= 70 - applied LAST in the pipeline scoring stage, after the AI and image blends. Near-certain evidence (>= 70) also FORCES category `ai-slop` (defaults shipped as-is are slop by the canonical definition - clean scaffold code never earns an untouched template `sophisticated-ai`, whose meaning is 'the presenter decided'); the confident band caps a `human-*` category at `uncertain`. Calibration truth set (guarded by tests): the six stock boilerplates (ixartz x2, create-t3-app, vercel ai-chatbot x2, fullstack-nextjs-app-template) grade C-D with markers listed, while devplacepy itself scores 0.0 with zero markers - tune weights against BOTH sides, never only the slop set. The evidence rides the `signal` inventory event, the SCORE payload (`template_score`/`template_markers`) and the report's Template Provenance section; admin toggle `isslop_template_detection`.
- **Rendered-DOM signal family (`analysis/domsignals/`).** A homepage-only, live-browser companion to the text-based `signals/webtells.py` checks: `acquisition/browser.py`'s `StealthBrowser.capture()` loads the page and returns the actual rendered DOM (computed styles, a class-name census, meta tags, headings/landmarks, console warnings, network response hosts/headers, a screenshot) via `DOM_EXTRACT_SCRIPT` in `acquisition/domcapture.py`. `domsignals/base.py` defines its own `@dom_check`/`@dom_site_check` decorator registry, mirroring the SEO job's `checks/` `@page_check`/`@site_check` mechanics, but emitting isslop's own `Signal` type (not a new one). Eight category modules (`builders`, `color`, `typography`, `layout`, `copy`, `metaseo`, `accessibility`, `buildsignals`) contribute 49 distinct signal codes; `domsignals/aggregate.py` `aggregate_dom_evidence` runs every registered check over the captured page(s) and saturates the weighted total into a `DomEvidence` (score/bucket/signals/detected_builder/builder_confidence). This brings the engine's total to twenty-one detector families (13 text-based in `signals/`, 8 rendered-DOM in `domsignals/`) and 126 signal codes (77 text-based, 49 rendered-DOM) - update these counts again the next time a family is added or removed, never leave them stale.
- **Homepage-only by design (`config.DOM_ANALYSIS_MAX_PAGES = 1`).** Capturing a full DOM, console, network trail and screenshot on every crawled page would multiply the browser cost of a run, so the pass runs ONLY against the first page at crawl depth 0 (`acquisition/website.py` gates `depth == 0 and index < DOM_ANALYSIS_MAX_PAGES`). Bump the constant later if the tool needs multi-page DOM coverage - `DomSiteContext`/`dom_site_check` (e.g. `detect_duplicate_meta_description`) already support more than one page. Git-repository sources never populate `dom_snapshots` (`dom_sink` is threaded only through `crawl_website`, never `clone_repository`), and a run where the browser could not be launched simply yields an empty page list, so `aggregate_dom_evidence([])` is a clean no-op (`DomEvidence(score=0.0, bucket="none")`).
- **Scoring order is load-bearing: images -> DOM -> template, never reorder.** `pipeline.py` applies `scoring.adjust_for_images`, then `adjust_for_dom_signals`, then `adjust_for_template`, in that exact sequence, and `adjust_for_template` MUST stay last. `adjust_for_dom_signals` blends `DomEvidence.score` into `ai_percent` at a small, deliberately cautious weight (`config.DOM_AI_WEIGHT = 0.12`, since this whole signal family is new and uncalibrated) UNLESS a confident builder match (`builder_confidence >= DOM_BUILDER_CONFIDENT_THRESHOLD`) forces the category to `ai-slop` via the same `_force_slop_category` helper the template detector uses. Because `adjust_for_template` runs after it, a template match can still floor/force the category further; nothing may run after `adjust_for_template`, since a later step would silently undo a forced `ai-slop` category.
- **DOM evidence never attaches to a per-file `FileScore`/`FileContext` (do not bolt it on).** `DomEvidence` is repo/page-level evidence blended once into the final `RepoScores`, exactly like template provenance and the image mean - never distributed across individual files. A rendered page has no SLOC and no line numbers to weight against, and its evidence (computed styles, a screenshot, console/network output) is not textual, so it cannot be scored, sampled or displayed through the SLOC-weighted per-file model the rest of the engine uses. Any new DOM check emits page-level `Signal`s into `DomEvidence.signals`, never into a `FileScore.signals` list.
- **`isslop_dom_results` follows the same evidence-table obligation as every other isslop table.** It mirrors `isslop_image_results` (`store.py` `TABLE_DOM_RESULTS`) and is already wired into both `store.reset_evidence(uid)` and `store.purge_analysis(uid)`; any future evidence table added under `domsignals/` must be added to both the same way.
- **DEP_UNRESOLVED is alias-aware (do not regress).** The JS/TS unresolved-import detector (`signals/hallucination.py`) flags imports that match no package.json dependency (all four sections), Node builtin or local path - i.e. phantom/hallucinated dependencies. It MUST skip everything that is not an npm specifier: relative/absolute/URL imports, `node:` and any `scheme:` specifier, the non-package prefixes `@/`, `~`, `#`, `$` (tsconfig/subpath/Svelte aliases - none are valid npm names), and every prefix parsed from `tsconfig.json`/`jsconfig.json` `compilerOptions.paths` (`engine._javascript_alias_prefixes`, carried on `RepoContext.javascript_alias_prefixes`). tsconfig is JSONC and alias keys contain `/*`, so comments are stripped with the string-aware scanner `_strip_jsonc_comments` - NEVER a comment regex (a regex eats the `"@/*"` alias strings themselves; this was a real defect). Before the alias awareness the detector flagged nearly every file of a standard Next.js app; after, only genuine phantom deps remain. `tests/unit/services/jobs/isslop/hallucination.py` guards it.
- **Clickable source references (annotated source viewer).** The static stage persists the FULL source of every signal-bearing file (cap `SOURCE_CAP_FILES`=60 files / 200KB each) into the same per-analysis media dir (`_persist_source`, `s<sha1>.txt`); the name rides the `file` event and the `isslop_file_results.source` column. `GET /tools/isslop/{uid}/source?path=...&line=N` renders `isslop_source.html`: server-rendered line table (Jinja autoescape covers XSS) with line-number anchors `#LN`, signal lines highlighted with inline annotation chips, a focused line, and a findings nav in the sidebar; the strict `^s[a-f0-9]{16}\.txt$` + `is_relative_to` checks mirror the media route. Everything referencing a file links there: the file-results table path, each signal chip (`&line=N#LN`), and the report prose - `_linkify_sources` rewrites backticked paths in the report markdown into links BEFORE `render_content`, and the reporter system prompt requires the model to backtick every path it mentions. `reset_evidence`/`purge_analysis` already sweep the media dir, so sources share the thumbnail lifecycle.
- **No absolute paths ever leave the engine:** every user-facing path is workspace-relative (`relative_to(workspace)`); keep it that way in new detectors/events.
- **Docs heading anchors + `.docs-toc` (platform-wide):** `docs_prose._render_markdown` post-processes every prose page, stamping a slugified `id` on each `h2`/`h3` (`heading_slug`, GFM-style, deduplicated with `-N` suffixes) and appending a hover-visible `.docs-heading-anchor` permalink; `scroll-margin-top` keeps targets below the topnav. The `isslop-checks` page uses this for its clickable Contents grid: a `.docs-toc` nav placed OUTSIDE the `data-render` block (raw HTML passes through untouched) whose `href="#slug"` values are computed with the SAME `heading_slug` function at generation time - reuse `.docs-toc`/`.docs-toc-item`/`.docs-toc-count` (styled in `docs.css`) for any other long docs page, and never hand-write a slug that `heading_slug` would not produce. `tests/unit/docs_prose.py` guards the slugging and injection.
- Devii tools `isslop`/`isslop_status`/`isslop_report`/`isslop_list` (member-only, `requires_auth=True` per policy - the HTTP surface stays public); docs `tools-isslop` + `isslop-checks` (the full plain-language check catalog); CLI `devplace isslop analyze|prune|clear`; audit `isslop.run.request|complete|failed` (category `tools`); achievement key `isslop` ("Slop Hunter"). New unpinned dep `playwright-stealth`; runtime dirs `config.ISSLOP_DIR`/`ISSLOP_WORKSPACES_DIR`/`ISSLOP_RUNS_DIR` in `DATA_PATHS`.
