devplacepy/devplacepy/services/jobs/CLAUDE.md

This file documents the async job services subsystem (devplacepy/services/jobs/ and its subdirectories deepsearch/, isslop/, seo/) - the standard pattern for running heavy blocking work off the request path. Claude Code loads it automatically whenever a file under this directory is read or edited.

Overview: the JobService pattern

The standard way to run heavy, blocking work off the request path and hand back a result URL, used by every job kind in this subsystem (zip, planning, fork, seo, seo_meta, deepsearch, isslop). Reference: admin docs Architecture -> Async job services and Services -> ZipService.

• The DB row is the queue. services/jobs/queue.py (enqueue, get_job, touch_job, list_jobs) is pure DB and callable from any worker's handler - enqueue is a fast, non-blocking insert, status reads work from any worker, and only the lock owner processes. All kinds share ONE jobs table discriminated by a kind column: common lifecycle timestamps, retry_count, last_accessed_at/expires_at, bytes_in/bytes_out/item_count stat columns, plus payload/result JSON columns where kind-specific fields live.
• JobService(BaseService) runs only in the lock owner. Each run_once: reap finished in-flight tasks -> recover orphaned running rows (left by a dead owner, since there is only one processor) back to pending with bounded retry_count -> refill up to max_concurrent oldest pending jobs (uuid7 sorts FIFO) as asyncio tasks -> sweep rows past expires_at via the per-kind cleanup() hook. In-flight tasks span ticks (kept in an instance map); the loop returns immediately each tick, so keep the interval short (default 2s).
• No atomic claim is needed (single processor by construction) and no separate reaper exists - retention is built into every job service, default 7 days, admin-configurable; downloading/reading a result extends expires_at via touch_job.
• Add a kind: subclass JobService, set kind, implement async process(self, job) -> dict (return the result dict incl. stat keys) and cleanup(self, job); register the service instance in main.py; add enqueue endpoints that own authz, a status route, a download/report route, a Devii tool, and docs. A new JobService needs a server restart to go live.
• Permanent vs expiring artifacts (load-bearing distinction - decide this per kind). Most kinds produce a DISPOSABLE artifact that expires with the retention sweep: ZipService, SeoService, and DeepsearchService all delete their real output (archive / report+screenshots / vector collection+report) inside cleanup(). Two kinds are different: ForkService's artifact is a permanent project, and the AI Usage Analyzer's (isslop) artifacts (analysis, events, file/image results, report, badge) are permanent public capability URLs - both make cleanup() a no-op on the real artifact and let the retention sweep remove only the jobs tracking row. Get this decision right for any new kind: if the output should outlive the job the way a fork or an isslop report does, do not wire cleanup() to delete it.

ZipService (kind zip)

• process materializes a project subtree via project_files.export_to_dir (staging under config.DATA_DIR/zip_staging/{uid}), compresses it in a subprocess (python -m devplacepy.services.jobs.zip_worker, stdlib-only, prints stats JSON incl. crc32), names the output {crc32}.{slug}.zip under config.DATA_DIR/zips/{uid[-2:]}/{uid[-4:-2]}/ (sharded on the random tail of the uuid7 via attachments._directory_for, NOT its time-ordered head), and removes staging.
• Runtime artifacts live in DATA_DIR (data/ by default), OUTSIDE the package and NOT under /static; the download is served by the /zips/{uid}/download route via FileResponse, not the static mount.
• Enqueue endpoints own authz: POST /projects/{slug}/zip, POST /projects/{slug}/files/zip?path=.
• GET /zips/{uid} (status ZipJobOut) and GET /zips/{uid}/download (FileResponse, extends expiry via touch_job) are capability URLs scoped only by the unguessable uuid7, not by owner - the owner is stored for attribution, not access control, matching publicly-viewable projects.
• Frontend app.zipDownloader (static/js/ZipDownloader.js) auto-wires any data-zip-download element plus the files context menu: POST -> poll /zips/{uid} via the shared JobPoller -> trigger download.
• Devii tools zip_project/zip_status; CLI devplace zips prune|clear.
• Disposable: cleanup() deletes the archive and the retention sweep prunes both the file and the jobs row.

Planning report generator (kind planning, services/jobs/planning_service.py)

PlanningReportService is admin-only and builds a complete, phased markdown implementation document from a selectable set of open Gitea tickets, intended to be handed straight to a coding agent for one-shot execution, off the request path via the same async-job pattern as zip.

• process collects open tickets via services/gitea/planning.py collect_open_issues(client, limit=MAX_ISSUES) (the paginated list_issues(state="open", limit=50) loop, cap 50; reused by the admin page too), narrows them to the selected ticket numbers carried in the job payload ({"numbers": [int, ...]}, preserving selection order - an empty/absent list keeps all open, so the no-arg Devii action and any legacy enqueue stay backward compatible), and builds the markdown via generate_plan(issues, config) (an AI-or-fallback helper mirroring enhance.py).
• AI path: passes each ticket's FULL, verbatim description (_body, capped only at the BODY_MAX=40000 per-ticket safety limit, not the old 600-char excerpt) to the internal gateway via stealth.stealth_async_client with a high output budget (MAX_TOKENS=32000, PLAN_MAX=600000 final cap, PLANNING_TIMEOUT_SECONDS=600 client timeout) and a SYSTEM_PROMPT that demands a self-contained document: a ## Execution Order list, then ## Phase K: <name> headings, and under each phase a ### #N <title> subsection per ticket carrying labelled Original ticket (verbatim blockquote), Goal, Dependencies, Affected areas / files, Implementation steps, Acceptance criteria, and Risks / open questions blocks - preserving every detail with nothing summarised away.
• Because an LLM can still summarise or truncate, the AI document is never trusted to be complete on its own: generate_plan always appends a deterministic # Appendix: Source Tickets (verbatim) section built straight from the issue dicts by planning.py verbatim_tickets(issues) (per ticket ## #N <title>, labels, the html_url source link, and the full body as a blockquote), so every covered ticket's full text is guaranteed present inline AND in the appendix - the document is genuinely self-contained.
• Fail-soft _fallback (when ai_enhance is off or any httpx.HTTPError/ValueError/KeyError/IndexError occurs) groups by primary label then ticket number, emitting the same ## Execution Order + ## Phase K shape with each ticket's full description reproduced verbatim (already self-contained, so no extra appendix).
• Writes the markdown to config.PLANNING_REPORTS_DIR/_directory_for(uid)/{crc32}.{slug}.md (sharded on the uuid7 random tail, crc32 over the markdown bytes), and returns {download_url, local_path, final_name, markdown, ai_used, item_count, bytes_out}. The markdown string is carried in the result so the status route can hand it to the renderer without re-reading disk. cleanup unlinks the file (retention prunes only the artifact + job row).
• Audit issue.planning.request (enqueue) and issue.planning.generate (success/failure) via record/record_system with an audit.job(uid) link.
• Routes (all require_admin): POST /issues/planning (enqueue, 503 when Gitea unconfigured, PlanningForm with a comma-separated numbers field parsed to list[int] and stored in the payload, returns {uid, status_url}), GET /issues/planning/{uid} (PlanningJobOut: status, markdown, download_url, issue_count, ai_used), GET /issues/planning/{uid}/download (FileResponse, text/markdown, traversal-guarded against PLANNING_REPORTS_DIR, extends retention via touch_job). The /issues/{number} detail route uses the {number:int} path converter so /issues/planning is never captured by it.
• Admin entry point: GET /admin/issues/planning (routers/admin/issues.py, admin_issues_planning.html) fetches the open tickets server-side via collect_open_issues (wrapped so a Gitea/network failure renders an empty state, never a 500) and renders a selectable checkbox list (all checked by default, a select-all/none master, and a live selected count) as the step in between, plus the Generate planning button, the JobPoller status panel, the <dp-content> render target, and a Download anchor; the issues listing shows an admin-only Generate planning link (viewer_is_admin in issues_page).
• Frontend driver static/js/PlanningGenerator.js (app.planningGenerator): tracks the checkbox selection (master toggle, count, disables Generate at zero selected), sends the chosen numbers as the numbers form field, POST -> JobPoller.run (widened to intervalMs:2000, maxAttempts:400 = ~800s so it outlives the longer detailed generation) -> replace the <dp-content> with a fresh one holding status.markdown so it re-renders, set the download href.
• Devii tool planning_report_generate (requires_admin=True). The renderer's built-in copy button (the dp-content Copy button styled in markdown.css, identical to the per-code-block copy button) satisfies the copy-source requirement and is opt-out via the no-copy boolean attribute on <dp-content>.

Project fork - ForkService (kind fork, services/jobs/fork_service.py)

Forking copies a source project into a brand-new project owned by the forking user, off the request path via the same async-job pattern as the zip flow.

• Enqueue: POST /projects/{slug}/fork (routers/projects/index.py) - require_user then can_view_project(source, user) (any logged-in user may fork any project they can view: public projects and their own private ones). Body is ForkForm{title} (the destination project name). It enqueues a fork job ({source_project_uid, title, forked_by_uid}, owner ("user", uid)) and returns {uid, status_url}. No work happens on the request path.
• process looks up the source project row and the forking user row (raises -> job failed if either is missing), creates the destination project with content.create_content_item("projects", "project", user, fields, ...) (so slug/XP/logging match a normal create; metadata - description, project_type, platforms, status, dates, is_private - is copied from the source, read_only reset to 0), copies the whole virtual FS off-thread (export_to_dir(source, "", staging) -> import_from_dir(new_uid, staging, user, skip_names=set()), an exact copy including binaries, staging under config.DATA_DIR/fork_staging/{uid}), then records the relation with database.record_fork(source_uid, new_uid, forked_by_uid). Result: {project_uid, project_url, source_project_uid, item_count}.
• Rollback: any failure after the project is created triggers _rollback(new_uid) (project_files.delete_all_project_files + delete_fork_relations + delete the projects row) before re-raising, so a failed fork leaves no orphan project.
• Cleanup is a no-op on the project. Unlike zip's disposable archive, a fork's artifact is a permanent project that must outlive the job. cleanup() only removes any leftover staging dir; the retention sweep deletes the job tracking row, never the forked project. devplace forks prune|clear likewise deletes job rows only.
• Relation model: the project_forks table (uid, source_project_uid, forked_project_uid, forked_by_uid, created_at) records direction explicitly (source -> forked), indexed on both project columns. Helpers in database.py: record_fork, get_fork_parent(forked_uid) (the source project row, for the "Forked from X" link on the project detail page), count_forks(source_uid), and delete_fork_relations(project_uid) (called on project delete in content.delete_content_item, and in fork rollback).
• Frontend: app.projectForker (static/js/ProjectForker.js) wires data-fork-project (a Fork button on the project detail page, visible to any logged-in user): prompt for a name via app.dialog.prompt -> Http.sendForm POST -> JobPoller.run("/forks/{uid}", ...) -> on done redirect to project_url. The forked project's detail page shows a "Forked from X" link (database.get_fork_parent).
• Status route GET /forks/{uid} (routers/forks.py, ForkJobOut) exposes project_uid/project_url/source_project_uid only once status == done. Devii tools fork_project/fork_status; docs projects-fork/forks-status.

SEO Diagnostics tool - SeoService (kind seo, services/jobs/seo/, routers/tools/)

The public Tools -> SEO Diagnostics auditor crawls a URL or sitemap with a headless browser and runs a broad battery of SEO checks, on the same async-job pattern as zip/fork plus a live websocket. It is public (guests included); abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap, and the shared SSRF guard.

• Surface: a collapsible Tools dropdown in base.html (desktop center nav + a mobile section, visible to everyone) toggled by MobileNav.initToolsDropdown. GET /tools lists tools; GET /tools/seo is the auditor page (static/js/SeoDiagnostics.js -> app.seoDiagnostics, instantiated page-side in the template, not in Application.js).
• Enqueue: POST /tools/seo/run (routers/tools/seo.py, body SeoRunForm{url, mode: url|sitemap, max_pages 1-50}). Owner is ("user", uid) or ("guest", X-Real-IP). It rejects with 429 if the owner already has a pending/running seo job, then enqueues {url, mode, max_pages, allow_private:False} and returns {uid, status_url, ws_url}.
• process writes the payload to config.SEO_REPORTS_DIR/{uid}/payload.json, launches python -m devplacepy.services.jobs.seo.worker <payload_json> <output_dir> via create_subprocess_exec (high limit= so big lines never overflow the StreamReader), reads NDJSON frames from stdout line by line (stage/target/progress/page/site_checks/report_ready), forwards each into the in-process ProgressHub (services/jobs/seo/progress.py, uid -> set of asyncio.Queue), and on completion loads output_dir/report.json as the job result. cleanup() clears the hub buffer and removes the report dir.
• Worker (worker.py, subprocess): crawler.crawl_target resolves the target (single URL, or sitemap <loc> URLs capped at max_pages) and fetches robots.txt/sitemap.xml/llms.txt with httpx. The audited target host is guarded once with net_guard.guard_public_url; candidate URLs sharing that host are pre-approved (no redundant per-URL getaddrinfo - a transient DNS failure or a self-hosted server resolving its own domain must not blank the whole crawl), and only cross-host sitemap entries are re-guarded. In sitemap mode the crawler never falls back to auditing the sitemap document itself: if no page URLs survive it raises a clear error (a stray or [target] fallback previously rendered the sitemap XML as one 56k-node "page" with no title/H1). For each page it launches one Playwright navigation: a single page.evaluate(EXTRACT_SCRIPT) returns the whole DOM contract (title/metas/canonical/headings/images/links/jsonld/og/twitter/semantic/mixed-content/word-count), an injected PerformanceObserver (add_init_script(INIT_SCRIPT)) captures LCP/CLS, navigation timing gives TTFB/FCP/transfer/protocol, a mobile-viewport pass measures overflow/tap-targets, a screenshot is saved, and a raw httpx GET supplies the SSR HTML for the rendered-vs-server parity check.
• Check registry (checks/): base.py defines the Check/PageContext/SiteContext dataclasses and the @page_check/@site_check decorators (collected into PAGE_CHECKS/SITE_CHECKS); one module per category (crawl, meta, headings, links, structured_data, social, performance, mobile_a11y, security, ai_readiness, crosspage). registry.run_page_checks/run_site_checks run them defensively (one failing check never aborts a page), and compute_score produces a severity-weighted overall score + grade and per-category subscores. To add a check: write a function decorated @page_check/@site_check in the right category module and import that module in registry.py.
• Live progress WS: WS /tools/seo/{uid}/ws is served only by the service-lock owner (closes 4013 for a fast retry on a non-owner worker, like /devii/ws); it replays hub.snapshot(uid) then streams hub.subscribe(uid). The frontend SeoProgressSocket mirrors the DeviiSocket 4013/reconnect pattern. The done frame carries the full report inline (and the router's late-join terminal branch reads it from job.result); the client renders from frame.report directly. This is load-bearing: service.process publishes done from inside process(), BEFORE the JobService base persists the result on the next reap tick (~2s later), so a client that fetched /tools/seo/{uid}/report on the done signal would race the DB write and get an empty (all-zero) report. Do not "simplify" this back into a fetch-on-done. After publishing done, process() calls hub.clear(uid) to bound the in-memory buffer (late reconnects fall back to the DB-backed terminal branch).
• Status/report routes: GET /tools/seo/{uid} (SeoJobOut), GET /tools/seo/{uid}/report (respond(..., SeoReportOut), HTML or JSON), GET /tools/seo/{uid}/screenshot/{n} (FileResponse from SEO_REPORTS_DIR, path-guarded). All are capability URLs scoped by the unguessable uuid7. The full report is written to config.SEO_REPORTS_DIR/{uid}/report.json (NOT inlined on a stdout line - avoids the StreamReader limit). Devii tools seo_diagnostics/seo_status/seo_report (public); docs tools-seo-*; CLI devplace seo prune|clear; audit seo.run.request|complete|failed (category tools).
• SSRF guard is shared: devplacepy/net_guard.py (guard_public_url, is_blocked_address, effective_address) was extracted from the Devii fetch controller, which now imports it; the crawler reuses it. playwright is a core dependency (Chromium installed by make install / the Docker image).
• Disposable: cleanup() removes the report dir; retention prunes both the artifacts and the jobs row.
• Production nginx needs a dedicated WS location. In nginx/nginx.conf.template the catch-all location / sets Connection "" (no upgrade) and a 60s timeout, so any websocket route that falls through to it fails the handshake (browser: WebSocket connection failed, no close code). The progress socket has its own location ~ ^/tools/seo/[^/]+/ws$ block forwarding Upgrade/Connection with a 1h timeout, mirroring /devii/ws. Any new websocket path must add its own upgrade location above location / (see docs Production -> nginx).

SEO metadata service - SeoMetaService (kind seo_meta, services/jobs/seo_meta_service.py, services/seo_meta.py, seo_meta_text.py)

SeoMetaService generates a clean, SEO-optimized title/description/keywords for every published content item (types post, project, gist, news, issue) off the request path and meters its own AI spend. It is a distinct concern from the public Tools -> SEO Diagnostics auditor (kind seo) - do NOT conflate the two kinds; they share only the jobs queue table. It is the constructive counterpart to the diagnostics tool: diagnostics audits a URL, this one populates the on-page metadata. It is a JobService like ForkService: the artifact (the seo_metadata row) is permanent, so cleanup() is a no-op and the retention sweep removes only the job tracking row.

• Tables. seo_metadata is polymorphic and soft-deletable (in SOFT_DELETE_TABLES, born-live deleted_at/deleted_by): uid, target_type, target_uid, seo_title, seo_description, seo_keywords, status (ready|pending|failed), source (ai|plain), generated_at, created_at, updated_at, keyed UNIQUE on (target_type, target_uid). init_db() ensures every column, the UNIQUE idx_seo_metadata_target and the live idx_seo_metadata_status (status, deleted_at) index. seo_usage is a single-row config-like usage table (NOT soft-delete) mirroring news_usage, keyed SEO_USAGE_KEY="seo_meta". Helpers in database.py: get_seo_metadata/get_seo_metadata_batch/has_fresh_seo_metadata/upsert_seo_metadata/mark_seo_metadata_stale (every read filters deleted_at IS NULL; get_seo_metadata returns only status="ready" live rows) and add_seo_usage/get_seo_usage.
• Choke helper. services/seo_meta.py schedule_seo_meta(target_type, uid, regenerate=False) and schedule_seo_meta_for_table(table, uid, ...) are import-cycle-free (only database + queue). They no-op for unknown types, missing uid, or (without regenerate) when a fresh ready row exists (database.has_fresh_seo_metadata); regenerate=True marks the row stale first (database.mark_seo_metadata_stale); both skip a target with an existing pending/running seo_meta job; otherwise queue.enqueue("seo_meta", {target_type, target_uid}, "system", "seo_meta"). Hooked at content.create_content_item (create, no-op guard) and content.edit_content_item (regenerate), the news publish sites in services/news.py (status=="published" only; existing-row update path uses regenerate=True), and IssueCreateService after the Gitea ticket is recorded. Because the work is async via the queue (NOT run_in_executor), the helper only enqueues.
• process loads the target row (posts/projects/gists/news via get_table, issues via gitea.store.get_ticket), builds grounding via services/ai_context.build_context (fail-soft for news/empty user_uid), and calls the gateway off-thread (asyncio.to_thread(correction.gateway_complete, internal_gateway_key(), system, source_text, timeout) - the synchronous gateway call posts to the in-process gateway on localhost, so it MUST run via to_thread or it self-deadlocks the single worker - the same lesson as correction.py's sync mode). It demands strict JSON {seo_title, seo_description, seo_keywords}, parses fail-soft, re-clamps every field server-side via seo_meta_text.clamp_generated, and falls back to seo_meta_text.plain_seo_defaults (status failed, source plain) on any failure - the fields are never empty. Usage accumulates via correction.new_usage_totals and flushes once with database.add_seo_usage(totals) when calls>0 (the single-row seo_usage table, mirroring news_usage). It emits an audit seo.meta.generate/seo.meta.failed (record_system, category tools) and upsert_seo_metadata(...).
• Backfill. run_once calls super().run_once() then a bounded backfill sweep (gated by seo_meta_backfill_enabled, seo_meta_backfill_batch per type per tick) over published content lacking a fresh ready row, so pre-existing items get metadata with no one-shot migration.
• Admin surface. collect_metrics() merges the JobService job-pipeline stats with usage_metric_cards(get_seo_usage()), so the SEO Metadata card on /admin/services shows both the live task pipeline and the AI cost/averages; the existing admin.services.{name} pub/sub topic + live_view_relay row pushes it live with NO new VIEWS row. A standard-paginated task list reads queue.list_jobs(kind="seo_meta") with database.build_pagination + _pagination.html when a dedicated page is desired.
• Clamps (single source of truth, seo_meta_text.py): seo_title hard cap 60 (word-boundary, single hyphen, keyword front-loaded); seo_description hard cap 160 (word-boundary via seo.truncate, key message in the first 120 chars); seo_keywords 5-8 distinct lowercase comma-joined terms (the <meta keywords> tag is dead for Google but the feature mandates it - emit a SHORT honest list, never stuffed). plain_text_from_markdown reuses rendering._render_content + utils.strip_html so markdown (and em-dash) never leaks.
• SEO consumption fix (seo.py base_seo_context). The meta description is now markdown-stripped (plain_markdown/plain_text_from_markdown, fixing the prior raw-markdown leak), meta_keywords is emitted (a safe plain string, NOT a Jinja-global name), and a seo_target=(target_type, target_uid) param makes it consume the ready seo_metadata row when present, else plain_seo_defaults. base_seo_context's new keywords/seo_target params are OPTIONAL with safe defaults so existing callers are unaffected; the five detail routers (posts/projects/gists/news/issues) pass seo_target. og_title/twitter:title use the bare seo_title (drops the redundant " - DevPlace" suffix in social cards); base.html adds <meta name="keywords">, og:image:width/height/alt and twitter:image:alt. The per-type JSON-LD (discussion_forum_posting, software_application_schema, news_article_schema, software_source_code_schema) route their text/description through plain_markdown.
• Fan-out. Schema SeoMetaOut; read route GET /tools/seo-meta/{target_type}/{target_uid} (routers/tools/index.py, public, JSON) returning the ready row or a plain default with status pending; Devii action seo_meta_status (public, read-only) + docs tools-seo-meta-status; CLI devplace seo-meta prune|clear (job rows only; the metadata persists); events seo.meta.generate|failed. Registered in main.py alongside SeoService. A new JobService needs a server restart to go live.

DeepSearch tool - DeepsearchService (kind deepsearch, services/jobs/deepsearch/, services/deepsearch/, routers/tools/deepsearch.py)

The public Tools -> DeepSearch researcher is a multi-agent deep web researcher built on the same async-job + ProgressHub + 4013-WS pattern as the SEO tool, plus a per-session vector store and a grounded RAG chat. It is public (guests included); abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap (1-30), depth cap (1-4), and the shared SSRF guard. Reuse the SEO tool as the template for any new Tools async job.

• Owner helper is shared: routers/tools/_shared.py owner_for(request) returns ("user", uid) or ("guest", X-Real-IP); both seo.py and deepsearch.py import it (do not re-inline the owner derivation).
• Enqueue: POST /tools/deepsearch/run (body DeepsearchRunForm{query, depth 1-4, max_pages 1-30}). It rejects with 429 if the owner already has a pending/running deepsearch job. It resolves the logged-in user's users.api_key (guests use database.internal_gateway_key()) into the job payload for per-user embedding/LLM spend attribution, generates the uid up front, writes a deepsearch_sessions row (create_deepsearch_session), enqueues the job carrying {query, depth, max_pages, api_key, collection}, and returns {uid, status_url, ws_url}. The enqueue uses a local _enqueue (not queue.enqueue) so the session uid and the job uid match.
• process writes control.json (state running) + payload.json (augmented with the cross-session cached_hashes) under config.DEEPSEARCH_DIR/{uid}, launches python -m devplacepy.services.jobs.deepsearch.worker <payload_json> <output_dir> via create_subprocess_exec (high limit=), pumps NDJSON stdout frames into the in-process ProgressHub (progress.py), and on completion loads output_dir/report.json, persists the URL cache (upsert_deepsearch_url_cache), and updates the session row. cleanup() drops the ChromaDB collection (VectorStore.drop) and removes the session dir. Disposable: the collection + report dir are both deleted, unlike Fork/isslop.
• Worker pipeline (worker.py, stdlib + httpx + playwright, importable subprocess): enhance.plan_queries (gateway -> JSON sub-queries, deterministic fallback) -> crawl.search_queries (rsearch via standalone httpx, never PlatformClient; per-query result buckets are round-robin interleaved so every planned angle contributes pages, never just the first query) -> crawl.crawl (batches of CRAWL_CONCURRENCY concurrent httpx fetches then playwright render fallback, guard_public_url on the URL and every redirect, content-hash + URL-hash dedup; depth follows in-page links: after each level the links of every crawled page are scored by query-token overlap via extract.relevant_links and the top LINKS_PER_PAGE unseen ones form the next level, depth=1 disables following) -> chunking.chunk_text -> embeddings.embed_texts (gateway, local hashing fallback when unavailable) -> store.VectorStore.add (Chroma) -> orchestrate.orchestrate (retrieval-grounded agents, see below). The worker writes report.json and url_cache.json and emits a report_ready frame carrying synthesis.
• Search-provided content is a first-class source (crawl.py, the second junk-report fix). search_queries calls rsearch with content=true, so each candidate carries the search engine's own readable content/description extract. This matters because the top sources for many questions are bot-hostile (X/Twitter, YouTube, Reddit, Facebook, Instagram, LinkedIn, TikTok - HOSTILE_DOMAINS): a headless fetch of those hits a login/consent wall ("Before you continue to YouTube", "Sign in to X") and yields near-zero text, which is why a 12-source run used to collapse to ~14 chunks. Now crawl._resolve_candidate skips the fetch entirely for a hostile domain and uses the rsearch snippet (_snippet_page, source="search", SNIPPET_MIN_CHARS floor), and for every other domain it fetches normally but keeps the rsearch snippet as a floor (uses whichever of crawl-text vs snippet is longer), so a walled or thin page still contributes its real content instead of being dropped. This alone took a query from "cannot be answered" to a correct cited answer (14 -> 61 chunks, diversity 0.333 -> 0.75). Never revert content=true and never send a headless render at a HOSTILE_DOMAINS host.
• Content extraction (extract.py, stdlib only): extract_html(raw, base_url) is a readability-grade HTMLParser extractor used by both the httpx and playwright fetch paths (the old naive regex tag-stripper produced nav/cookie-banner boilerplate as "content" - the historic root cause of junk reports). It skips script/style/nav/header/footer/aside/form and ARIA role=navigation|banner|contentinfo|... regions, prefers <article>/<main> when they carry at least MIN_CONTENT_TOTAL chars, drops link-dense blocks (MAX_LINK_DENSITY, menus) and sub-MIN_BLOCK_CHARS fragments, unescapes entities, and emits real paragraphs joined by blank lines - which also makes chunking.chunk_text's paragraph split actually fire (the flattened text used to be sliced mid-sentence). It also returns the page's (url, anchor_text) links (absolute, deduped, nav links excluded) for depth crawling; relevant_links(links, query, limit) scores them by query-token overlap and filters non-document extensions.
• No "gaps"/critic agent (removed - do not reintroduce). DeepSearch used to run a fourth "critic" agent that produced an "Open gaps" list. It was removed end-to-end (orchestrate, worker report, DeepsearchSessionOut, router context, session template, markdown/HTML export, DeepsearchTool.js agent labels, CSS) because it routinely emitted misleading, self-contradictory gaps on correctly-cited reports (the historic cause was that the critic was fed a truncated, retrieval-ordered source slice that dropped cited source numbers out of its window, so it fabricated "citation [n] is not in the report / no evidence provided"). Do NOT reintroduce a gaps field or a critic agent. The _numbered_source_digest(pages) helper survives and is used by the linker - a compact per-source [n] title (url)\nexcerpt block for EVERY page where the header line is always emitted even when the excerpt is trimmed, so all source numbers 1..N are guaranteed present (never pass a raw context[:N] slice to an agent that reasons about source numbers). Regression: tests/unit/services/jobs/deepsearch/orchestrate.py::test_linker_receives_full_source_list / ::test_numbered_source_digest_keeps_every_source_number_under_cap.
• Orchestration (orchestrate.py) is retrieval-grounded and markdown-first. The worker indexes BEFORE analysis and passes the VectorStore + planned queries to orchestrate, which embeds the question and each sub-query and pulls hybrid_search top chunks (round-robin merged, up to CONTEXT_CHUNKS_MAX), building the context from the RETRIEVED passages grouped per source - the source numbers [n] align with the report's sources list (page order), so inline citations, finding citations, and the rendered numbered source list agree. Page-head excerpts are only the fallback when retrieval is empty. Synthesis is two-step to avoid the markdown-inside-JSON trap: the summarizer writes a plain markdown report (REPORT_MAX_TOKENS, retried once if empty), then a separate extractor agent returns the findings JSON (retried once, tolerant _parse_json handles code fences and trailing garbage); the linker (confidence) failure is caught and never discards the report. The three agents in the pipeline today are summarizer, extractor, and linker - there is no fourth "critic" stage. Only a failed/empty report falls back to _heuristic, which stamps synthesis="heuristic" and emits a status:"failed" agent frame - the degradation is VISIBLE: report.synthesis flows through DeepsearchSessionOut.synthesis, the session template renders a "Degraded report" banner (.ds-degraded), and the markdown export carries the same note. A successful run stamps synthesis="agents". Never re-inline synthesis into a single JSON blob and never let a synthesis failure ship silently.
• PDF ingestion (crawl.py fetch_page + pdf.py): a crawled candidate is treated as a PDF when its content-type is application/pdf/application/x-pdf, its URL path ends in .pdf, or its first bytes match the %PDF- magic (pdf.is_pdf). fetch_page streams the body and caps it at MAX_PDF_BYTES (15 MB); for a PDF it calls pdf.extract_pdf_text, which writes the raw binary to a tempfile temp location (cleaned up via Path.unlink(missing_ok=True) in finally), parses it with pypdf (PdfReader, capped at MAX_PDF_PAGES = 50, title pulled from metadata), and normalizes whitespace. The resulting CrawledPage carries source="pdf"; PDFs skip the Playwright fallback. Everything downstream is source-agnostic (chunking/embedding/orchestration read page.text/page.source unchanged), so no other module changes. New unpinned dep pypdf (pure-python, no system deps).
• Pause/resume/cancel: POST /tools/deepsearch/{uid}/{pause|resume|cancel} (owner-gated) rewrite control.json; the worker's should_stop callback polls it between source fetches (paused = sleep-loop, cancelled = stop). State lives in a file, not the job row, so the running-in-a-subprocess worker can read it without a DB round-trip.
• Vector store (services/deepsearch/store.py): VectorStore wraps chromadb.PersistentClient(path=config.DEEPSEARCH_CHROMA_DIR), one collection per session (ds_<uid>). Chunk is the dataclass. hybrid_search blends cosine vector similarity with a BM25 keyword score (weights HYBRID_VECTOR_WEIGHT/HYBRID_KEYWORD_WEIGHT) over the candidate set, with optional metadata where filters. embeddings.py embed_texts calls the gateway embeddings endpoint and falls back to a deterministic local hashing vector on any failure (so the tool degrades, never breaks).
• RAG chat (services/deepsearch/chat.py + WS /tools/deepsearch/{uid}/chat): a dedicated lightweight loop (NOT the Devii hub), served only by the service-lock owner (closes 4013 for fast retry). Answers are grounded ONLY in the session collection via hybrid_search, cited inline, rendered client-side via dp-content. Turns persist to deepsearch_messages and audit deepsearch.chat. Frontend component <dp-deepsearch-chat> (static/js/components/AppDeepsearchChat.js) clones AppDocsChat's framing but uses its own WebSocket to the chat path.
• Status/report/export routes: GET /tools/deepsearch/{uid} (DeepsearchJobOut), GET /tools/deepsearch/{uid}/session (respond(..., DeepsearchSessionOut), HTML or JSON), GET /tools/deepsearch/{uid}/export.{md,json,pdf} (services/deepsearch/export.py; PDF via weasyprint). All are capability URLs scoped by the unguessable uuid7. Viewer-flag discipline: the session schema/context use viewer_is_admin/viewer_owns (never is_admin/owns) so a respond() context key never shadows a Jinja global (the same class of issue as the issues /{number} route). tests/api/tools/deepsearch/session.py guards the HTML render.
• Completion race (load-bearing read-path fix). The worker writes report.json to disk and service.process publishes the done frame from inside process(), but the JobService framework only commits jobs.result/status=DONE afterwards, in _reap() -> _finish_done() on a later tick. The frontend navigates to the session page the instant it receives done, so a read that keyed only off jobs.status == DONE returned an EMPTY report (None score, 0 sources) until a manual refresh. Fix: _report_for(uid, job) returns job.result.report when the job is DONE and non-empty, else falls back to the on-disk report.json (_report_from_disk, DEEPSEARCH_DIR/{uid}/report.json) - which exists before the done frame is ever sent - and returns {} only for a FAILED job or a genuinely still-running job with no report on disk. _session_context derives done/status from bool(report) (not raw job status), and the chat WS gate accepts session.status == "done" (set inside process() before the publish) as ready. _export_report reuses the same fallback. Regression: tests/api/tools/deepsearch/session.py::test_session_reads_disk_report_before_result_commit. Any new read of a job result that a client reaches immediately after a done/session_url frame must use this same on-disk fallback, never bare jobs.status.
• Clickable inline citations (services/deepsearch/citations.py). The report/findings carry [n] markers (and the model sometimes emits [3][9][1-2]); the link_citations(html, source_count) template global (registered in templating.py) rewrites each [n] and each [a-b] range into <a class="ds-cite" href="#ds-source-n">[n]</a> anchors that jump to the numbered <li id="ds-source-n"> in the Sources list (source numbering is page order, matching the [n] the summarizer was given). It splits out <a>/<code>/<pre> regions first so markers inside links/code are left alone, expands ranges to individual links, and drops out-of-range numbers (no broken anchors). The session template nests it over the server render: {{ link_citations(render_content(summary), sources|length) }} and {{ link_citations(finding.detail|e, sources|length) }}, plus a per-finding .ds-finding-cites chip row from finding.citations. .ds-cite/.ds-sources li:target styling lives in deepsearch.css. The report prompt asks for one number per bracket (never a range) so output is consistent, but the linkifier handles ranges regardless. Regression: tests/unit/services/deepsearch/citations.py.
• Tables (deepsearch_sessions, deepsearch_messages soft-deletable + in SOFT_DELETE_TABLES; deepsearch_url_cache GC-only): columns are ensured in init_db() (every queried column) with indexes. Every insert writes deleted_at:None/deleted_by:None; every read filters deleted_at IS NULL.
• Frontend (do not hand-roll): DeepsearchTool.js (app.deepsearchTool) drives the form via Http.send, watches DeepsearchProgressSocket (cloned from SeoProgressSocket, 4013 retry), and wires pause/resume/cancel. static/css/deepsearch.css uses the design tokens and is mobile-responsive.
• Progress frame protocol (append-only, the worker<->JS contract): phases.py is the single source of truth for phase identity, shared by worker.py (emit) and DeepsearchTool.js (render). PHASE_ORDER = [planning, searching, crawling, indexing, analysis, synthesis]; worker._stage(stage, message, phase) emits BOTH the legacy stage frame (byte-identical to before) AND a parallel phase frame {phase, index, total, label} so the timeline strip advances. The first emitted frame carries version:1. Every other frame type and its keys: substep (planning angles, phase+message; also emitted by analysis grounding), queries, candidates, rsearch, progress (done/total/url/depth), page_loaded (now source/render/depth/elapsed_ms/done/total), page_cached/page_skipped/page_duplicate (now reason/elapsed_ms), embed_batch (batch/total_batches/backend/done/total, emitted before AND after each batch), embed_done (backend/chunk_count), agent (agent one of summarizer|extractor|linker, stage/status start|done|failed, with elapsed_ms/tokens_in/tokens_out on done), report_ready (now also synthesis), done (session_url), failed (message). The contract is append-only: never rename or drop a frame type; service._run_worker pumps every stdout line into the ProgressHub untouched, so new frame types reach the WS with no handler change. tests/api/tools/deepsearch/index.py is the append-only regression guard.
• RAG-audit hardening (engine correctness, no route/schema change): (1) Embedding-dimension consistency - gateway embeddings carry provider-native dims while local_embed is fixed 256-dim; the worker _index_chunks now decides the backend ONCE per job (the first gateway failure or non-gateway result forces local for ALL remaining batches), and VectorStore.add drops any vector whose length differs from the collection's established dim, so one collection never mixes dims (cosine search across mixed dims is corrupt). embeddings.EmbedResult.dims and VectorStore.dims (lazily probed from the collection) expose the dimension; chat.retrieve re-embeds the query locally and skips retrieval if it still cannot match the stored dim. (2) Citation grounding - orchestrate drops any finding with no citations, and the summarizer prompt forbids uncited claims and treats the QUESTION as data not an instruction (prompt-injection reduction via _sanitize_question); chat._strip_unmatched_markers removes any inline [n] marker that does not map to an emitted citation. (3) Confidence calibration - when only a single domain was crawled, orchestrate caps confidence by the source-diversity-derived ceiling (overconfidence guard). (4) EmbeddingCache is bounded at EMBED_CACHE_MAX to cap in-memory growth.
• Devii tools deepsearch/deepsearch_status/deepsearch_session (public); docs tools-deepsearch; CLI devplace deepsearch prune|clear; audit deepsearch.run.request|complete|failed + deepsearch.chat (category tools). New dependencies: chromadb, weasyprint, pypdf (all unpinned). New runtime dirs config.DEEPSEARCH_DIR/DEEPSEARCH_CHROMA_DIR are registered in DATA_PATHS. Add a dedicated nginx WS location for /tools/deepsearch/{uid}/ws and /chat above location / for production, like the SEO and Devii sockets.

AI Usage Analyzer tool - IsslopService (kind isslop, services/jobs/isslop/, routers/tools/isslop.py)

The public Tools -> AI Usage Analyzer classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. It is built on the standard async-job pattern (a JobService running a subprocess worker), but its live channel is pub/sub, not a dedicated WS route: every worker event is published to public.isslop.{uid} AND persisted to isslop_events, and the frontend pairs the pub/sub subscription with an incremental GET /tools/isslop/{uid}/events?after=SEQ poll, so guests (who cannot subscribe to public.* unless pubsub_allow_guests is on) and reconnecting tabs replay from the durable trail. Never rely on pub/sub alone for this tool: the DB event trail is the source of truth, pub/sub is the fast path.

• Engine layout: services/jobs/isslop/ holds acquisition/ (git probe via git ls-remote, depth-1 clone with size preflight + live 3 GB kill guard, stealth Playwright website crawler with HTTP fallback, path-traversal-safe workspace helpers), analysis/ (exclusion rules, stylometric metrics, language detection, per-repo baselines, signals/ with one detector family per file, two-axis scoring), agent/ (gateway LLM client, per-file classifier, vision reviewer, report writer with deterministic fallback), plus pipeline.py (the event-yielding run), worker.py (subprocess entry), events.py (frame protocol), persistence.py (EventPersister writes events/file results/image results/report and stamps the analysis row), store.py (all DB access), badge.py (SVG), service.py (IsslopService), config.py (all constants + WorkerSettings).
• Worker contract: IsslopService.process writes the worker payload (url + admin toggles + gateway endpoint/model/key) to config.ISSLOP_RUNS_DIR/{uid}/payload.json, resolves the workspace under config.ISSLOP_WORKSPACES_DIR (workspace_for rejects any path escaping the root), launches python -m devplacepy.services.jobs.isslop.worker <payload> <workspace>, and relays each NDJSON stdout line through EventPersister.apply (SQLite) then pubsub.publish. The workspace and run dir are removed in a finally; the pipeline also deletes the workspace itself as its final act, so no acquired source survives an analysis - only the report and its evidence rows.
• AI through the gateway only: agent/llm.py talks solely to config.INTERNAL_GATEWAY_URL with model molodetz and database.internal_gateway_key() (vision uses the same model - the gateway handles image parts). review_available/vision_available gate the AI and image stages; on any gateway failure the static engine remains authoritative and the report falls back to the deterministic composer. All HTTP (gateway, git size preflight, website fallback crawl) goes through stealth_async_client.
• Artifacts are permanent, the job row is not. Like ForkService, cleanup() never touches isslop_analyses/isslop_events/isslop_file_results/isslop_image_results/isslop_reports - the report and badge are public capability URLs meant to outlive the run; the retention sweep removes only the jobs tracking row. devplace isslop clear is the only bulk hard-delete (plus per-analysis store.purge_analysis).
• Ownership and guest history sync: the owner is ("user", uid) or ("guest", DEVII_GUEST_COOKIE) - NOT the tools _shared.owner_for IP fallback, because history must survive IP changes and be claimable. The page/list/run handlers mint the guest cookie when absent (same cookie as Devii/customization, one guest identity platform-wide). _sync_guest_history runs on page and list requests: when a signed-in user still carries a guest cookie, store.claim_guest_analyses re-owns those rows via UPDATE (a move, never a copy - no duplicate data). One active analysis per owner (429 otherwise, audited denied).
• Tables: isslop_analyses is soft-deletable (in SOFT_DELETE_TABLES, born-live inserts, reads filter deleted_at IS NULL, indexed on (owner_kind, owner_id, created_at)/status/content_hash); the evidence tables (isslop_events keyed (analysis_uid, seq), isslop_file_results, isslop_image_results, isslop_reports UNIQUE on analysis_uid) are GC-only evidence purged with their analysis. All ensured in init_db().
• Routes (all under /tools/isslop, capability URLs): GET "" page, POST /run (IsslopRunForm, http/git/ssh URL pattern), GET /list (owner history), GET /{uid} (IsslopAnalysisOut), GET /{uid}/events (ordered replay), GET /{uid}/report (respond(..., IsslopReportOut) - HTML shows the live <dp-isslop-run> while running and the server-rendered report when completed; the markdown body goes through render_content), GET /{uid}/report.md, GET /{uid}/badge.svg (self-contained SVG, hardcoded colors by design - it must render on external sites). Badge/report URLs are absolute via seo.site_url.
• Frontend: two site-wide web components (static/js/components/AppIsslop.js <dp-isslop> = submit form + history list; AppIsslopRun.js <dp-isslop-run> = live progress feed), registered in components/index.js, light DOM, reusing Http/Poller and app.pubsub. Page CSS static/css/isslop.css (design tokens). On done the run component reloads the page so the report is the server-rendered (SEO/render_content) version, never a client re-render.
• Verdict blending is per-file, never a global mean (load-bearing). The AI review pass runs its per-file gateway calls concurrently (pipeline.AI_REVIEW_CONCURRENCY, semaphore-bounded like the image pass) and adjusts ONLY the files it actually reviewed: pipeline.apply_ai_verdicts blends each sampled file's static origin/quality with its own verdict (0.6/0.4), then the WHOLE repo is re-aggregated with the normal SLOC/criticality weights, and scoring.ai_fraction maps per-file origin scores through a smooth 35-65 ramp (never the old hard 45/55 buckets). Never reintroduce a repo-level mean of the 12 sampled verdicts - it hands a tiny sample a fixed 40% of the verdict, so the LLM's clustered hedging values (30/40/50) drown thousands of files of static evidence and unrelated projects converge on identical percentages (the real-world twin-84%-human defect). Single source of truth for the final verdict: the SCORE event, the DONE payload and generate_report all consume the SAME final RepoScores (image influence applied via scoring.adjust_for_images, which recomputes slop/grade/human together) - the summary grade and the report body grade can therefore never disagree; tests/unit/services/jobs/isslop/pipeline.py guards both invariants.
• Image evidence thumbnails + retry-safe evidence. Workspaces die with the run, so the vision stage persists an aspect-preserving WebP thumbnail per reviewed image (vision.make_thumbnail, sha1-of-relative-path name) into config.ISSLOP_MEDIA_DIR/{uid} (the dir comes to the worker via the payload media_dir); the thumb name rides the image event, is stored on isslop_image_results, and is served by GET /tools/isslop/{uid}/media/{name} (strict ^[a-f0-9]{16}\.webp$ name pattern + is_relative_to root check - never loosen either). The report page renders the thumbnails with data-lightbox (the shared app.lightbox opens them full-size) and the live feed shows a tiny inline preview per image event. store.reset_evidence(uid) runs at the top of every IsslopService.process - a retried job (orphan recovery) previously re-inserted its events/file/image rows, duplicating every image and file in the report; any new evidence table MUST be added to reset_evidence AND purge_analysis.
• Template provenance (the "ships defaults" detector). analysis/templates.py detect_template(workspace) scores starter-template evidence repo-wide (it reads files the inventory excludes, like package.json): known template slugs/authors in the manifest (+ ct3aMetadata), README template marketing (weights capped so a wordy README cannot alone confirm), and the kitchen-sink scaffold constellation (count of standard scaffold artifacts past 3 freebies). The saturating score feeds scoring.adjust_for_template - a no-op below 35, a 0.7x floor on ai_percent/origin when confident, 0.85x when >= 70 - applied LAST in the pipeline scoring stage, after the AI and image blends. Near-certain evidence (>= 70) also FORCES category ai-slop (defaults shipped as-is are slop by the canonical definition - clean scaffold code never earns an untouched template sophisticated-ai, whose meaning is 'the presenter decided'); the confident band caps a human-* category at uncertain. Calibration truth set (guarded by tests): the six stock boilerplates (ixartz x2, create-t3-app, vercel ai-chatbot x2, fullstack-nextjs-app-template) grade C-D with markers listed, while devplacepy itself scores 0.0 with zero markers - tune weights against BOTH sides, never only the slop set. The evidence rides the signal inventory event, the SCORE payload (template_score/template_markers) and the report's Template Provenance section; admin toggle isslop_template_detection.
• Rendered-DOM signal family (analysis/domsignals/). A homepage-only, live-browser companion to the text-based signals/webtells.py checks: acquisition/browser.py's StealthBrowser.capture() loads the page and returns the actual rendered DOM (computed styles, a class-name census, meta tags, headings/landmarks, console warnings, network response hosts/headers, a screenshot) via DOM_EXTRACT_SCRIPT in acquisition/domcapture.py. domsignals/base.py defines its own @dom_check/@dom_site_check decorator registry, mirroring the SEO job's checks/ @page_check/@site_check mechanics, but emitting isslop's own Signal type (not a new one). Eight category modules (builders, color, typography, layout, copy, metaseo, accessibility, buildsignals) contribute 49 distinct signal codes; domsignals/aggregate.py aggregate_dom_evidence runs every registered check over the captured page(s) and saturates the weighted total into a DomEvidence (score/bucket/signals/detected_builder/builder_confidence). This brings the engine's total to twenty-one detector families (13 text-based in signals/, 8 rendered-DOM in domsignals/) and 126 signal codes (77 text-based, 49 rendered-DOM) - update these counts again the next time a family is added or removed, never leave them stale.
• Homepage-only by design (config.DOM_ANALYSIS_MAX_PAGES = 1). Capturing a full DOM, console, network trail and screenshot on every crawled page would multiply the browser cost of a run, so the pass runs ONLY against the first page at crawl depth 0 (acquisition/website.py gates depth == 0 and index < DOM_ANALYSIS_MAX_PAGES). Bump the constant later if the tool needs multi-page DOM coverage - DomSiteContext/dom_site_check (e.g. detect_duplicate_meta_description) already support more than one page. Git-repository sources never populate dom_snapshots (dom_sink is threaded only through crawl_website, never clone_repository), and a run where the browser could not be launched simply yields an empty page list, so aggregate_dom_evidence([]) is a clean no-op (DomEvidence(score=0.0, bucket="none")).
• Scoring order is load-bearing: images -> DOM -> template, never reorder. pipeline.py applies scoring.adjust_for_images, then adjust_for_dom_signals, then adjust_for_template, in that exact sequence, and adjust_for_template MUST stay last. adjust_for_dom_signals blends DomEvidence.score into ai_percent at a small, deliberately cautious weight (config.DOM_AI_WEIGHT = 0.12, since this whole signal family is new and uncalibrated) UNLESS a confident builder match (builder_confidence >= DOM_BUILDER_CONFIDENT_THRESHOLD) forces the category to ai-slop via the same _force_slop_category helper the template detector uses. Because adjust_for_template runs after it, a template match can still floor/force the category further; nothing may run after adjust_for_template, since a later step would silently undo a forced ai-slop category.
• DOM evidence never attaches to a per-file FileScore/FileContext (do not bolt it on). DomEvidence is repo/page-level evidence blended once into the final RepoScores, exactly like template provenance and the image mean - never distributed across individual files. A rendered page has no SLOC and no line numbers to weight against, and its evidence (computed styles, a screenshot, console/network output) is not textual, so it cannot be scored, sampled or displayed through the SLOC-weighted per-file model the rest of the engine uses. Any new DOM check emits page-level Signals into DomEvidence.signals, never into a FileScore.signals list.
• isslop_dom_results follows the same evidence-table obligation as every other isslop table. It mirrors isslop_image_results (store.py TABLE_DOM_RESULTS) and is already wired into both store.reset_evidence(uid) and store.purge_analysis(uid); any future evidence table added under domsignals/ must be added to both the same way.
• DEP_UNRESOLVED is alias-aware (do not regress). The JS/TS unresolved-import detector (signals/hallucination.py) flags imports that match no package.json dependency (all four sections), Node builtin or local path - i.e. phantom/hallucinated dependencies. It MUST skip everything that is not an npm specifier: relative/absolute/URL imports, node: and any scheme: specifier, the non-package prefixes @/, ~, #, $ (tsconfig/subpath/Svelte aliases - none are valid npm names), and every prefix parsed from tsconfig.json/jsconfig.json compilerOptions.paths (engine._javascript_alias_prefixes, carried on RepoContext.javascript_alias_prefixes). tsconfig is JSONC and alias keys contain /*, so comments are stripped with the string-aware scanner _strip_jsonc_comments - NEVER a comment regex (a regex eats the "@/*" alias strings themselves; this was a real defect). Before the alias awareness the detector flagged nearly every file of a standard Next.js app; after, only genuine phantom deps remain. tests/unit/services/jobs/isslop/hallucination.py guards it.
• Clickable source references (annotated source viewer). The static stage persists the FULL source of every signal-bearing file (cap SOURCE_CAP_FILES=60 files / 200KB each) into the same per-analysis media dir (_persist_source, s<sha1>.txt); the name rides the file event and the isslop_file_results.source column. GET /tools/isslop/{uid}/source?path=...&line=N renders isslop_source.html: server-rendered line table (Jinja autoescape covers XSS) with line-number anchors #LN, signal lines highlighted with inline annotation chips, a focused line, and a findings nav in the sidebar; the strict ^s[a-f0-9]{16}\.txt$ + is_relative_to checks mirror the media route. Everything referencing a file links there: the file-results table path, each signal chip (&line=N#LN), and the report prose - _linkify_sources rewrites backticked paths in the report markdown into links BEFORE render_content, and the reporter system prompt requires the model to backtick every path it mentions. reset_evidence/purge_analysis already sweep the media dir, so sources share the thumbnail lifecycle.
• No absolute paths ever leave the engine: every user-facing path is workspace-relative (relative_to(workspace)); keep it that way in new detectors/events.
• Docs heading anchors + .docs-toc (platform-wide): docs_prose._render_markdown post-processes every prose page, stamping a slugified id on each h2/h3 (heading_slug, GFM-style, deduplicated with -N suffixes) and appending a hover-visible .docs-heading-anchor permalink; scroll-margin-top keeps targets below the topnav. The isslop-checks page uses this for its clickable Contents grid: a .docs-toc nav placed OUTSIDE the data-render block (raw HTML passes through untouched) whose href="#slug" values are computed with the SAME heading_slug function at generation time - reuse .docs-toc/.docs-toc-item/.docs-toc-count (styled in docs.css) for any other long docs page, and never hand-write a slug that heading_slug would not produce. tests/unit/docs_prose.py guards the slugging and injection.
• Devii tools isslop/isslop_status/isslop_report/isslop_list (member-only, requires_auth=True per policy - the HTTP surface stays public); docs tools-isslop + isslop-checks (the full plain-language check catalog); CLI devplace isslop analyze|prune|clear; audit isslop.run.request|complete|failed (category tools); achievement key isslop ("Slop Hunter"). New unpinned dep playwright-stealth; runtime dirs config.ISSLOP_DIR/ISSLOP_WORKSPACES_DIR/ISSLOP_RUNS_DIR in DATA_PATHS.