Add thread notifications, SEO topic pages, and fix quiz auto-advance

Notifications: a new "thread" type notifies every other commenter on a
post whenever anyone comments on it, disregarding reply hierarchy -
excluding the actor and whoever already got a comment/reply
notification for that same event, so no one is double-notified.
Implemented via a background-deferred fan-out mirroring the existing
mention-notification pattern.

SEO: discussion_forum_posting() now embeds up to 20 of a post's
comments as nested schema.org Comment entities (not just an aggregate
count), and a new /topics hub plus /topics/{topic} pages give the
feed's topic filter real, independently crawlable/indexable URLs -
/feed?topic=X was never indexable since its canonical strips the
query string back to bare /feed. Both are wired end to end (schemas,
Devii actions, docs API, sitemap, locustfile load-test coverage).

Quiz player: the auto-advance to the next question used to hide the
just-answered slide in the same tick as rendering the grade, so on
any multi-question quiz the Correct/Not correct feedback was never
actually visible before the view moved on. Delayed via setTimeout,
with the pending timer cleared on manual navigation and on
disconnect so it can't race or fire on a removed component.

Also includes other local changes already in progress in this
working tree before this session (messaging, push delivery,
deepsearch jobs, game economy, quiz builder) - verified by the full
suite passing (3467 tests) but not authored or individually reviewed
in this session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VL9Xn57W5UR3HZbbuuzxdK

CLAUDE.md

@@ -19,7 +19,7 @@ DevPlace is a server-rendered social network for developers. FastAPI backend ser
 ## Commands
 
 ```bash
-make install        # pip install -e . + playwright install chromium
+make install        # create .venv if needed, pip install -e ".[dev]", playwright chromium
 make ppy            # build the single shared container image (ppy:latest); run once before launching instances
 make dev            # uvicorn --reload on port 10500, backlog 4096
 make prod           # uvicorn --workers $(WEB_WORKERS) (defaults to nproc), port 10500 (backlog 8192)
@@ -33,6 +33,8 @@ make locust         # Locust load test, interactive web UI
 make locust-headless  # Locust CLI mode for CI
 ```
 
+Every Python make target (`install`, `dev`, `prod`, `test*`, `coverage*`, `locust*`) uses `.venv/bin/python`. If `.venv` is missing, make creates it from `python3`, installs `-e ".[dev]"`, and installs Playwright Chromium before running the target. `make install` refreshes that environment. `make clean` removes `.venv` as well as stray bytecode.
+
 The Makefile exports `PYTHONDONTWRITEBYTECODE=1` for every recipe, so no `.pyc` files or `__pycache__` directories are written by any `make` target. Keep it that way - do not add a target that re-enables bytecode writing. `make clean` removes any stray bytecode left from running Python outside make.
 
 Preliminary validation: confirm `python -c "from devplacepy.main import app"` imports clean and check each touched language manually (Python compiles/imports, JS parses, CSS braces and HTML tags balance). These checks are gates on the way to the real validation, never a substitute for it: **every change ends with the full test suite (`make test` - all three tiers, every test) and it must pass.**
@@ -183,6 +185,7 @@ Routers in `devplacepy/routers/` are organised as a **directory tree that mirror
 |--------|--------|
 | `/auth` | auth/ package |
 | `/feed`, `/posts`, `/comments` | flat files |
+| `/topics` | topics.py - crawlable per-topic category index pages (`/topics` hub, `/topics/{topic}` listing) |
 | `/projects`, `/projects/{slug}/files`, `/projects/{slug}/containers` | projects/ package - see `routers/projects/CLAUDE.md` |
 | `/profile` | profile/ package (customization, notifications, ai-correction, ai-modifier, interactions, telegram, usage) |
 | `/messages` | messages.py - see `services/messaging/CLAUDE.md` |
@@ -282,7 +285,7 @@ The escape hatch is deliberately two-factor and must never be self-served: after
 
 - **No comments, no docstrings in source.** Code is self-documenting.
 - **Forbidden variable name patterns:** `_new`, `_old`, `_temp`, `_v2`, `better_`, `my_`, `the_`.
-- **Outbound HTTP goes through the stealth client only.** Never instantiate a bare `httpx.AsyncClient(...)`/`httpx.Client(...)` for outbound traffic - build it via `devplacepy.stealth.stealth_async_client(...)`/`stealth_sync_client(...)`. The async factory routes bytes through **`curl_cffi` (curl-impersonate, BoringSSL)** behind an `httpx` transport adapter, so TLS JA3/JA4 and the HTTP/2 fingerprint match Chrome 146 byte-for-byte; falls back to a pure-httpx Chrome-aligned transport if `curl_cffi` is missing. Applies to news, gateway upstream, Devii/DeepSearch/SEO fetch, gitea, push, attachments. SSRF-guarded fetches via `net_guard.guarded_async_client(...)` get the fingerprint automatically. Sole exception: the reverse-proxy forwarding core `services/containers/forward.py` (and the three routers that call it - `routers/proxy.py`, `routers/tunnel.py`, the workspace editor route), which relays the user's own headers verbatim and therefore keeps a plain `httpx.AsyncClient`; bolting the Chrome identity onto it would overwrite the very headers it exists to forward. **Load-bearing gotcha:** cleartext `http://` is forced to HTTP/1.1 (`curl_transport.http_version_for`) - the Chrome-impersonation profile defaults to HTTP/2, but a plaintext origin has no TLS/ALPN to negotiate h2 and uvicorn (HTTP/1.1 only) returns `400 "Invalid HTTP request received."` once a cleartext HTTP/2 body crosses ~128KB. Every internal AI consumer calls `INTERNAL_GATEWAY_URL` = `http://localhost:<port>/openai/v1/...`, so without this override a large Devii turn (big system prompt + tool list) fails on every request. `https://` keeps the HTTP/2 default. `stealth_sync_client` stays pure-httpx Chrome-aligned (no curl_cffi sync adapter). **Never bolt a hand-written browser-header dict (UA/`sec-ch-ua`/`Sec-Fetch-*`) onto a `stealth_async_client`/`stealth_sync_client` call** - curl_cffi's impersonation already emits an internally-consistent Chrome 146 identity across TLS, UA, and Client Hints; a caller-supplied header set claiming a different Chrome version or platform creates a UA-vs-TLS-vs-Client-Hints mismatch that is a textbook anti-bot tripwire (this was a real bug in Devii's `fetch_url`/`http_request` tool, fixed by deleting its hardcoded Chrome-131 header block). `stealth.detect_consent_gate(html)` recognizes the common JS-redirect cookie-consent wall pattern (DPG Media's `nu.nl` and sibling sites, and generically any page gated behind a `decodeURIComponent(...)` callback referencing a "privacy gate"/`consent.js`) - reuse it wherever a fetched page's real content sits behind that gate rather than special-casing a domain. `stealth_transport`/`stealth_async_client`/`stealth_sync_client` all accept a `proxy=` override, defaulting to `stealth.configured_proxy_url()` (the admin-editable `outbound_proxy_url` site setting, falling back to `DEVPLACE_OUTBOUND_PROXY_URL` when the setting is unset) - this is the only real fix for a destination that blocks by IP/ASN reputation (e.g. reddit.com returns an identical "blocked by network security" page across every TLS/header fingerprint tried, confirmed by testing curl_cffi's full impersonation list - it is blocking the datacenter ASN outright, not fingerprinting the client).
+- **Outbound HTTP goes through the stealth client only.** Never instantiate a bare `httpx.AsyncClient(...)`/`httpx.Client(...)` for outbound traffic - build it via `devplacepy.stealth.stealth_async_client(...)`/`stealth_sync_client(...)`. The async factory routes bytes through **`curl_cffi` (curl-impersonate, BoringSSL)** behind an `httpx` transport adapter, so TLS JA3/JA4 and the HTTP/2 fingerprint match Chrome 146 byte-for-byte; falls back to a pure-httpx Chrome-aligned transport if `curl_cffi` is missing. Applies to news, gateway upstream, Devii/DeepSearch/SEO fetch, gitea, web push, attachments. SSRF-guarded fetches via `net_guard.guarded_async_client(...)` get the fingerprint automatically. Two exceptions keep a plain `httpx.AsyncClient`: (1) the reverse-proxy forwarding core `services/containers/forward.py` (and the three routers that call it - `routers/proxy.py`, `routers/tunnel.py`, the workspace editor route), which relays the user's own headers verbatim - bolting the Chrome identity onto it would overwrite the very headers it exists to forward; (2) the APNs provider (`push/providers/apns.py` `gateway_client` / `delivery_client`) which talks to Apple's HTTP/2 provider API with `httpx.AsyncClient(http2=True, trust_env=False)` - Chrome impersonation, HTTP/2 PRIORITY frames, HPACK indexing of `:path`/`authorization`, extra browser headers, and the outbound proxy all break or starve that API. Web Push stays on stealth. **Load-bearing gotcha:** cleartext `http://` is forced to HTTP/1.1 (`curl_transport.http_version_for`) - the Chrome-impersonation profile defaults to HTTP/2, but a plaintext origin has no TLS/ALPN to negotiate h2 and uvicorn (HTTP/1.1 only) returns `400 "Invalid HTTP request received."` once a cleartext HTTP/2 body crosses ~128KB. Every internal AI consumer calls `INTERNAL_GATEWAY_URL` = `http://localhost:<port>/openai/v1/...`, so without this override a large Devii turn (big system prompt + tool list) fails on every request. `https://` keeps the HTTP/2 default. `stealth_sync_client` stays pure-httpx Chrome-aligned (no curl_cffi sync adapter). **Never bolt a hand-written browser-header dict (UA/`sec-ch-ua`/`Sec-Fetch-*`) onto a `stealth_async_client`/`stealth_sync_client` call** - curl_cffi's impersonation already emits an internally-consistent Chrome 146 identity across TLS, UA, and Client Hints; a caller-supplied header set claiming a different Chrome version or platform creates a UA-vs-TLS-vs-Client-Hints mismatch that is a textbook anti-bot tripwire (this was a real bug in Devii's `fetch_url`/`http_request` tool, fixed by deleting its hardcoded Chrome-131 header block). `stealth.detect_consent_gate(html)` recognizes the common JS-redirect cookie-consent wall pattern (DPG Media's `nu.nl` and sibling sites, and generically any page gated behind a `decodeURIComponent(...)` callback referencing a "privacy gate"/`consent.js`) - reuse it wherever a fetched page's real content sits behind that gate rather than special-casing a domain. `stealth_transport`/`stealth_async_client`/`stealth_sync_client` all accept a `proxy=` override, defaulting to `stealth.configured_proxy_url()` (the admin-editable `outbound_proxy_url` site setting, falling back to `DEVPLACE_OUTBOUND_PROXY_URL` when the setting is unset) - this is the only real fix for a destination that blocks by IP/ASN reputation (e.g. reddit.com returns an identical "blocked by network security" page across every TLS/header fingerprint tried, confirmed by testing curl_cffi's full impersonation list - it is blocking the datacenter ASN outright, not fingerprinting the client).
 - **Form validation is Pydantic-native.** Routers declare `data: Annotated[SomeForm, Form()]` (models in `models.py`); the global `RequestValidationError` handler re-renders auth pages with messages (400) or redirects other routes. Read raw `await request.form()` only when also reading an uploaded file alongside a model.
 - **`RedirectResponse(url=..., status_code=302)`** - always pass `status_code` as an integer literal, never `status.HTTP_302_FOUND`.
 - **Ownership checks:** compare `resource["user_uid"] == user["uid"]` before edit/delete (use `content.is_owner`). **Deletes additionally allow any admin:** `is_owner(...) or is_admin(user)` on every content delete endpoint, so an administrator may delete any member's content while members are limited to their own. Edits stay owner-only. Deletes are **soft** and cascade under one shared stamp - see `devplacepy/database/CLAUDE.md`.

Makefile

@@ -9,20 +9,39 @@ LOCUST_WEB_WORKERS ?= 4
 WEB_WORKERS ?= $(shell nproc 2>/dev/null || echo 2)
 DEVPLACE_RATE_LIMIT ?= 1000000
 
+VENV ?= $(CURDIR)/.venv
+PYTHON := $(VENV)/bin/python
+VENV_STAMP := $(VENV)/.installed
+BOOTSTRAP_PYTHON := $(shell command -v python3 2>/dev/null || command -v python 2>/dev/null)
+
 PYTHONDONTWRITEBYTECODE := 1
 export PYTHONDONTWRITEBYTECODE
 
-.PHONY: install dev clean tree tree-loc zip test test-headed test-unit test-api test-e2e test-fast test-failed test-first-failure test-slowest test-cache-clean coverage coverage-headed coverage-html locust locust-headless
+.PHONY: venv install dev prod clean tree tree-loc zip test test-headed test-unit test-api test-e2e test-fast test-failed test-first-failure test-slowest test-cache-clean coverage coverage-headed coverage-html locust locust-headless
 
-install:
-	pip install -e .
-	python -m playwright install chromium
+$(PYTHON):
+	@test -n "$(BOOTSTRAP_PYTHON)" || { echo "python3 is required to create $(VENV)"; exit 1; }
+	$(BOOTSTRAP_PYTHON) -m venv $(VENV)
 
-dev:
-	uvicorn devplacepy.main:app --reload --reload-dir devplacepy --host 0.0.0.0 --port 10500 --backlog 4096
+$(VENV_STAMP): $(PYTHON) pyproject.toml
+	$(PYTHON) -m pip install -U pip
+	$(PYTHON) -m pip install -e ".[dev]"
+	$(PYTHON) -m playwright install chromium
+	touch $(VENV_STAMP)
 
-prod:
-	DEVPLACE_STATIC_VERSION=$$(date +%s) DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(WEB_WORKERS) uvicorn devplacepy.main:app --host 0.0.0.0 --port 10500 --workers $(WEB_WORKERS) --backlog 8192 --proxy-headers --forwarded-allow-ips '*'
+venv: $(VENV_STAMP)
+
+install: $(PYTHON)
+	$(PYTHON) -m pip install -U pip
+	$(PYTHON) -m pip install -e ".[dev]"
+	$(PYTHON) -m playwright install chromium
+	touch $(VENV_STAMP)
+
+dev: $(VENV_STAMP)
+	$(PYTHON) -m uvicorn devplacepy.main:app --reload --reload-dir devplacepy --host 0.0.0.0 --port 10500 --backlog 4096
+
+prod: $(VENV_STAMP)
+	DEVPLACE_STATIC_VERSION=$$(date +%s) DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(WEB_WORKERS) $(PYTHON) -m uvicorn devplacepy.main:app --host 0.0.0.0 --port 10500 --workers $(WEB_WORKERS) --backlog 8192 --proxy-headers --forwarded-allow-ips '*'
 
 delete-pyc:
 	find . -name "__pycache__" -type d -prune -exec rm -rf {} + 2>/dev/null || true
@@ -42,78 +61,78 @@ zip:
 	@git ls-files -z | xargs -0 zip -q $(notdir $(CURDIR)).zip
 	@printf 'Wrote %s (%s files)\n' "$(notdir $(CURDIR)).zip" "$$(git ls-files | wc -l)"
 
-test:
-	PLAYWRIGHT_HEADLESS=1 python -m pytest tests/
+test: $(VENV_STAMP)
+	PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/
 
-test-headed:
-	PLAYWRIGHT_HEADLESS=0 python -m pytest tests/
+test-headed: $(VENV_STAMP)
+	PLAYWRIGHT_HEADLESS=0 $(PYTHON) -m pytest tests/
 
-test-unit:
-	python -m pytest tests/unit
+test-unit: $(VENV_STAMP)
+	$(PYTHON) -m pytest tests/unit
 
-test-api:
-	python -m pytest tests/api
+test-api: $(VENV_STAMP)
+	$(PYTHON) -m pytest tests/api
 
-test-e2e:
-	PLAYWRIGHT_HEADLESS=1 python -m pytest tests/e2e
+test-e2e: $(VENV_STAMP)
+	PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/e2e
 
-test-fast:
-	python -m pytest tests/unit tests/api
+test-fast: $(VENV_STAMP)
+	$(PYTHON) -m pytest tests/unit tests/api
 
-test-failed:
-	PLAYWRIGHT_HEADLESS=1 python -m pytest tests/ --last-failed --last-failed-no-failures none
+test-failed: $(VENV_STAMP)
+	PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/ --last-failed --last-failed-no-failures none
 
-test-first-failure:
-	PLAYWRIGHT_HEADLESS=1 python -m pytest tests/ -x
+test-first-failure: $(VENV_STAMP)
+	PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/ -x
 
-test-slowest:
-	PLAYWRIGHT_HEADLESS=1 python -m pytest tests/ --durations=40
+test-slowest: $(VENV_STAMP)
+	PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/ --durations=40
 
-coverage:
+coverage: $(VENV_STAMP)
 	rm -f .coverage .coverage.*
 	COVERAGE_PROCESS_START=$(CURDIR)/.coveragerc PLAYWRIGHT_HEADLESS=1 \
-		python -m coverage run -m pytest tests/
-	python -m coverage combine
-	python -m coverage report
+		$(PYTHON) -m coverage run -m pytest tests/
+	$(PYTHON) -m coverage combine
+	$(PYTHON) -m coverage report
 
-coverage-headed:
+coverage-headed: $(VENV_STAMP)
 	rm -f .coverage .coverage.*
 	COVERAGE_PROCESS_START=$(CURDIR)/.coveragerc PLAYWRIGHT_HEADLESS=0 \
-		python -m coverage run -m pytest tests/
-	python -m coverage combine
-	python -m coverage report
+		$(PYTHON) -m coverage run -m pytest tests/
+	$(PYTHON) -m coverage combine
+	$(PYTHON) -m coverage report
 
 coverage-html: coverage
-	python -m coverage html
+	$(PYTHON) -m coverage html
 	@echo "Report written to htmlcov/index.html"
 
-locust:
+locust: $(VENV_STAMP)
 	export DEVPLACE_DATABASE_URL="sqlite:///$(LOCUST_DB)"; \
 	export DEVPLACE_RATE_LIMIT=$(DEVPLACE_RATE_LIMIT); \
 	fuser -k $(LOCUST_PORT)/tcp 2>/dev/null || true; \
 	sleep 1; \
 	mkdir -p $(LOCUST_DB_DIR); \
 	rm -f $(LOCUST_DB); \
-	DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(LOCUST_WEB_WORKERS) uvicorn devplacepy.main:app --host 127.0.0.1 --port $(LOCUST_PORT) --workers $(LOCUST_WEB_WORKERS) --backlog 8192 > /tmp/devplace_locust_server.log 2>&1 & \
+	DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(LOCUST_WEB_WORKERS) $(PYTHON) -m uvicorn devplacepy.main:app --host 127.0.0.1 --port $(LOCUST_PORT) --workers $(LOCUST_WEB_WORKERS) --backlog 8192 > /tmp/devplace_locust_server.log 2>&1 & \
 	PID=$$!; \
 	while kill -0 $$PID 2>/dev/null && ! curl -s http://127.0.0.1:$(LOCUST_PORT)/ > /dev/null 2>&1; do sleep 0.5; done; \
 	if ! kill -0 $$PID 2>/dev/null; then echo "Server failed to start (port $(LOCUST_PORT) busy?). See /tmp/devplace_locust_server.log"; exit 1; fi; \
-	locust --host http://127.0.0.1:$(LOCUST_PORT) --web-port $(LOCUST_WEB_PORT); \
+	$(PYTHON) -m locust --host http://127.0.0.1:$(LOCUST_PORT) --web-port $(LOCUST_WEB_PORT); \
 	kill $$PID 2>/dev/null || true; \
 	rm -rf $(LOCUST_DB_DIR)
 
-locust-headless:
+locust-headless: $(VENV_STAMP)
 	export DEVPLACE_DATABASE_URL="sqlite:///$(LOCUST_DB)"; \
 	export DEVPLACE_RATE_LIMIT=$(DEVPLACE_RATE_LIMIT); \
 	fuser -k $(LOCUST_PORT)/tcp 2>/dev/null || true; \
 	sleep 1; \
 	mkdir -p $(LOCUST_DB_DIR); \
 	rm -f $(LOCUST_DB); \
-	DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(LOCUST_WEB_WORKERS) uvicorn devplacepy.main:app --host 127.0.0.1 --port $(LOCUST_PORT) --workers $(LOCUST_WEB_WORKERS) --backlog 8192 > /tmp/devplace_locust_server.log 2>&1 & \
+	DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(LOCUST_WEB_WORKERS) $(PYTHON) -m uvicorn devplacepy.main:app --host 127.0.0.1 --port $(LOCUST_PORT) --workers $(LOCUST_WEB_WORKERS) --backlog 8192 > /tmp/devplace_locust_server.log 2>&1 & \
 	PID=$$!; \
 	while kill -0 $$PID 2>/dev/null && ! curl -s http://127.0.0.1:$(LOCUST_PORT)/ > /dev/null 2>&1; do sleep 0.5; done; \
 	if ! kill -0 $$PID 2>/dev/null; then echo "Server failed to start (port $(LOCUST_PORT) busy?). See /tmp/devplace_locust_server.log"; exit 1; fi; \
-	locust --host http://127.0.0.1:$(LOCUST_PORT) --headless -u $(LOCUST_USERS) -r $(LOCUST_SPAWN_RATE) --run-time $(LOCUST_RUN_TIME) --html=$(LOCUST_DB_DIR)/report.html; \
+	$(PYTHON) -m locust --host http://127.0.0.1:$(LOCUST_PORT) --headless -u $(LOCUST_USERS) -r $(LOCUST_SPAWN_RATE) --run-time $(LOCUST_RUN_TIME) --html=$(LOCUST_DB_DIR)/report.html; \
 	kill $$PID 2>/dev/null || true; \
 	rm -rf $(LOCUST_DB_DIR)
 

README.md

@@ -7,7 +7,7 @@ Server-rendered social network for developers. FastAPI backend serving Jinja2 te
 ## Quick start
 
 ```bash
-make install        # pip install -e .
+make install        # create .venv if needed, pip install -e ".[dev]", playwright chromium
 make dev            # uvicorn --reload on port 10500
 make test           # Playwright integration + unit tests, headless, fail-fast
 make test-headed    # same tests in visible browser
@@ -82,7 +82,7 @@ devplacepy/
 | `/admin/trash` | Admin **Trash**: review, restore, and permanently purge soft-deleted content (posts, comments, gists, projects, news, project files, attachments) across the platform |
 | `/admin/devii-tasks` | Admin **Devii tasks**: every scheduled task across all owners with its schedule, run count, expiry and failure streak, plus per-task disable and delete |
 | `/notifications` | Notification list, mark read, live unread counts (`/notifications/counts`) |
-| `/messages` | Real-time direct messaging over WebSocket (`/messages/ws`): live bidirectional delivery, optimistic send, typing indicators, read receipts, and online/last-seen presence. Messages render through the shared content pipeline (emoji shortcodes, image, video and audio embeds, autolink, sanitization). AI content correction and the AI modifier apply to direct messages, so typing an inline `@ai <instruction>` in a message executes it and the resolved result appears live in the chat for both participants (the `message` WS frame carries an additive `ai_processed` flag when a pending correction/modification was applied). An opened conversation loads its 500 most recent messages; older history is retained in the database. `GET /messages/conversations` returns the conversation list as JSON for a live client refresh; the `POST /messages/send` form remains as a no-JavaScript fallback and now accepts an attachment-only, empty-content message. `POST /messages/ws-ticket` exchanges a session/API-key auth to a short-lived (30s), single-use ticket a browser WebSocket can carry as `?ticket=...` on the handshake, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header - this is what makes off-session embedding of the chat possible. CLI: `devplace messaging prune-tickets` removes expired tickets |
+| `/messages` | Real-time direct messaging over WebSocket (`/messages/ws`): live bidirectional delivery, optimistic send, typing indicators, read receipts, reconnect catch-up, and online/last-seen presence. Messages render through the shared content pipeline (emoji shortcodes, image, video and audio embeds, autolink, sanitization). AI content correction and the AI modifier apply to direct messages: the original is delivered immediately, then a second `message` frame with `ai_processed` replaces the bubble for both participants when the rewrite lands (including across workers). An opened conversation loads its 500 most recent messages and can page older rows with `?before=`; switching threads does not drop the socket. `GET /messages/conversations` returns the conversation list as JSON for a live client refresh; the `POST /messages/send` form remains as a no-JavaScript fallback and now accepts an attachment-only, empty-content message. `POST /messages/ws-ticket` exchanges a session/API-key auth to a short-lived (30s), single-use ticket a browser WebSocket can carry as `?ticket=...` on the handshake, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header - this is what makes off-session embedding of the chat possible. CLI: `devplace messaging prune-tickets` removes expired tickets |
 | `/votes` | Upvote/downvote on posts, comments, projects |
 | `/reactions` | Emoji reactions on posts, comments, gists, projects |
 | `/bookmarks` | Save/unsave content; `/bookmarks/saved` personal list |
@@ -452,7 +452,7 @@ and its full configuration are documented automatically - including future servi
 - **`ConfigField`** - declarative parameter spec (type, default, validation, secret) a service uses to declare its editable settings
 - **`BaseService`** - abstract class with a reconciling run loop that honors the persisted `enabled`/command/interval state, plus `config_fields`, `get_config()`, `describe()`, and a log buffer
 - **`ServiceManager`** - singleton: `register`, `describe_all`, `set_enabled`, `send_command`, `save_config`, `supervise`, `shutdown_all`
-- **`NewsService`** - a fully automatic, zero-maintenance news pipeline: fetches news from `news.app.molodetz.nl/api`, cleans each article, fetches and perceptually compares the images to reject placeholders and detect uniqueness, grades each one deterministically, reformats every valid article into clean Markdown (paragraphs, headings, lists) with the AI so the source wall of text reads as a proper article, and auto-rotates the best articles to Featured and the landing page. Its AI spend is metered from the gateway response headers and reported on the admin Services page (calls, tokens, total cost, and per-call averages)
+- **`NewsService`** - a fully automatic, zero-maintenance news pipeline: fetches news from `news.app.molodetz.nl/api`, cleans each article, fetches and perceptually compares the images to reject placeholders and detect uniqueness, grades each one against a free, local, deterministic quality model (aquality - no billed LLM call, no API key), optionally reformats articles that clear the publish grade threshold into clean Markdown with a generative AI model (off by default, since the default grading model cannot generate text), and auto-rotates the best articles to Featured and the landing page. Any AI spend it does incur is metered from the gateway response headers and reported on the admin Services page (calls, tokens, total cost, and per-call averages)
 - **`BotsService`** - Playwright fleet of AI personas that browse and interact with a DevPlace instance, with live cost/usage metrics and a live screenshot monitor at `/admin/bots` (opt-in; install the `bots` extra)
 - **`GatewayService`** - OpenAI-compatible LLM gateway at `/openai/v1/*`, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default). Every chat request is made date-aware by injecting the current date in EU `DD/MM/YYYY` format into the system message when it contains no date already (date only, never time, to keep upstream prompt caching effective), and an admin-configurable system preamble (`gateway_system_preamble`) can be prepended ahead of the client's system message on every call. **Provider and model routing** (admin **Gateway** page, `/admin/gateway`) maps any number of requested model names onto named upstream providers and target models, each with its own pricing economy (input, output and cache-hit/cache-miss rates per million tokens) and an optional vision model that describes image content before forwarding, so text and vision models are merged transparently. Unmapped requests fall through to the default upstream unchanged, so existing clients are unaffected
 - **`JobService` / `ZipService` / `ForkService`** - generic async job framework (`services/jobs/`) for heavy, blocking work run off the request path; `ZipService` builds project zip archives in a subprocess, `ForkService` copies a project into a new project owned by the forking user
@@ -524,7 +524,7 @@ restart.
 
 `SeoMetaService` is a separate AI subservice that generates a clean, search-optimised title, description and short keyword list for every published post, project, gist, news article and issue, entirely off the request path so it never slows the web server. The work is queued whenever content is created, edited or published; until the AI value is ready a plain-content default (built from the markdown-stripped text and clamped to safe lengths) fills the fields, so a page's metadata is **always populated, never empty**. The service uses the built-in internal AI gateway and meters its own AI cost and statistics in a dedicated usage table, surfaced together with its live task pipeline on the **Admin -> Services** page. This release also fixes the on-page metadata: the `<meta name="description">` is now stripped of markdown markup (it previously leaked `#`, `**` and `[](...)` from the raw body), a `<meta name="keywords">` tag is emitted (a short honest list, not stuffed), and social-card image dimensions and alt text are added. CLI: `devplace seo-meta prune` / `devplace seo-meta clear` (job rows only; the generated metadata persists).
 
-`DeepsearchService` powers the public **Tools -> DeepSearch** researcher. Given a single research question it plans a set of diverse web search queries, interleaves their results so every angle contributes sources, and crawls the most relevant pages in a subprocess (concurrent fetches, plain HTTP first with a headless-browser fallback for JavaScript-heavy pages, PDF documents streamed and text-extracted, every URL SSRF-guarded). Social sites that block bots (X, YouTube, Reddit and similar) contribute the readable text supplied by the search engine, so their content is not lost to a login wall. A readability-grade extractor isolates the main article content of each page (navigation, cookie banners and footers are discarded), and the configured depth follows the most relevant in-page links for deeper coverage. Content is de-duplicated and indexed into a per-session ChromaDB vector collection (embeddings via the AI gateway with a local fallback). A chain of agents (report writer, findings extractor, linker) then synthesises a thorough cited markdown report grounded on the passages retrieved from that index, with key findings, a confidence score and source diversity; if synthesis fails the report is clearly marked as degraded instead of silently shipping raw source material. Progress streams live over `WS /tools/deepsearch/{uid}/ws`; the report is at `/tools/deepsearch/{uid}/session` (HTML or JSON) and can be exported as Markdown, JSON or PDF. A grounded chat over the session's evidence runs at `WS /tools/deepsearch/{uid}/chat` using hybrid retrieval (vector + keyword/BM25). Runs can be paused, resumed or cancelled. A cross-session URL cache avoids re-fetching pages seen by earlier runs. CLI: `devplace deepsearch prune` / `devplace deepsearch clear`. ChromaDB, weasyprint, pypdf and Playwright are core dependencies.
+`DeepsearchService` powers the public **Tools -> DeepSearch** researcher, an iterative agent modelled on the way Perplexity and OpenAI's own deep-research agents work: search, read, decide what is still missing, search again. Given a single research question it plans a set of diverse web search queries, interleaves their results so every angle contributes sources, and crawls the most relevant pages in a subprocess (concurrent fetches, plain HTTP first with a headless-browser fallback for JavaScript-heavy pages, PDF documents streamed and text-extracted, every URL SSRF-guarded). After the first pass it runs up to two automatic gap-filling refinement rounds - a lightweight planner looks at what has been gathered so far and proposes a few more targeted queries only if real coverage gaps remain, bounded by the page budget, and stops the moment nothing more is needed. Social sites that block bots (X, YouTube, Reddit and similar) contribute the readable text supplied by the search engine, so their content is not lost to a login wall. A readability-grade extractor isolates the main article content of each page (navigation, cookie banners and footers are discarded), and the configured depth follows the most relevant in-page links for deeper coverage. Content is de-duplicated and indexed into a per-session ChromaDB vector collection (embeddings via the AI gateway with a local fallback); retrieval reranks candidate passages with Maximal Marginal Relevance so the report is grounded on genuinely diverse evidence rather than several near-duplicate hits on the same page. A chain of agents (report writer, findings extractor, linker) then synthesises a thorough cited markdown report grounded on the passages retrieved from that index, with key findings, a confidence score, source diversity and a short list of suggested follow-up questions you can ask straight into the chat; if synthesis fails the report is clearly marked as degraded instead of silently shipping raw source material. Progress streams live over `WS /tools/deepsearch/{uid}/ws`; the report is at `/tools/deepsearch/{uid}/session` (HTML or JSON) and can be exported as Markdown, JSON or PDF. A grounded chat over the session's evidence runs at `WS /tools/deepsearch/{uid}/chat` using hybrid retrieval (vector + keyword/BM25). Runs can be paused, resumed or cancelled. A cross-session URL cache avoids re-fetching pages seen by earlier runs. CLI: `devplace deepsearch prune` / `devplace deepsearch clear`. ChromaDB, weasyprint, pypdf and Playwright are core dependencies.
 
 `IsslopService` powers the public **Tools -> AI Usage Analyzer**, which classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. Sources are acquired in an isolated subprocess (git URLs are probed with `git ls-remote` and shallow-cloned with a 3 GB guard; websites render in a stealth headless browser with an HTTP fallback, bounded by depth, file and byte caps), inventoried with exclusion rules, and scored by a multi-signal static engine (twenty-one detector families, 126 signal types across an origin axis and a quality-deficit axis). For a live website, the analyzer also opens the home page in a headless browser and inspects what actually renders: it fingerprints AI website-builder platforms directly (Lovable, Bolt.new, Framer and others) and checks the page's real computed styles, layout and build artifacts, not just its file contents. Representative files receive an AI review pass and images an AI-generation review through the internal gateway (model `molodetz`, internal key); the static engine stays authoritative when the gateway is unreachable. Every pipeline step is persisted as an ordered event trail and streamed live over the pub/sub topic `public.isslop.{uid}`. The verdict is an A-F authenticity grade with a human/AI split and one of five categories (`ai-slop`, `sophisticated-ai`, `human-clean`, `human-messy`, `uncertain`), published as a persistent report with an embeddable SVG badge. Members keep their analysis history on their account; guest history is session-bound and claimed by the account on the first signed-in visit. Admin settings (private-host allowance, AI/image review toggles, image cap, retention, concurrency) live on `/admin/services`. CLI: `devplace isslop analyze <url>` / `devplace isslop prune` / `devplace isslop clear`. Playwright plus playwright-stealth back the website crawler.
 
@@ -557,9 +557,10 @@ Configuration on the Services tab (`/admin/services`):
 |-----------|---------|---------|
 | `news_grade_threshold` | `7` | Minimum AI grade for auto-publish |
 | `news_api_url` | `https://news.app.molodetz.nl/api` | News source |
-| `news_ai_url` | `http://localhost:10500/openai/v1/chat/completions` | AI grading endpoint (the internal gateway) |
-| `news_ai_model` | `molodetz` | Generic model name; the gateway maps it to the real model |
-| `news_ai_key` | internal key | AI API key (`NEWS_AI_KEY` env, then the auto-generated gateway internal key) |
+| `news_ai_url` | `https://aquality.cloud.pravda.education/v1/chat/completions` | AI grading endpoint - a free, local, deterministic quality model by default |
+| `news_ai_model` | `aquality` | Generic model name; echoed back, not used for routing |
+| `news_ai_key` | internal key | AI API key (`NEWS_AI_KEY` env, then the auto-generated gateway internal key); unused by the default aquality endpoint |
+| `news_format_enabled` | `false` | Reformat article bodies into Markdown with AI after grading; off by default since aquality only scores and cannot generate text - enable when `news_ai_url` points at a generative chat model |
 | `news_service_interval` | `3600` | Seconds between fetch cycles (min 60) |
 
 News articles have detail pages at `/news/{slug}` with full comment support (same component as posts/projects). The landing page can display curated articles toggled from admin.
@@ -919,13 +920,17 @@ receives a notification through every provider they hold a live subscription for
 | Provider | Registration | Transport |
 |----------|--------------|-----------|
 | `webpush` | `PushSubscription` from the browser `PushManager` (endpoint + `p256dh`/`auth` keys) | Web Push Protocol, VAPID signed, `aesgcm` encrypted payload |
-| `apns` | Hexadecimal device token | `POST https://api.push.apple.com/3/device/{token}` over HTTP/2, ES256 provider token |
+| `apns` | Hexadecimal device token, optional stable `client_id` | Dedicated HTTP/2 client to `api.push.apple.com` or `api.sandbox.push.apple.com` (`POST /3/device/{token}`), ES256 provider token. Not the stealth/Chrome client. |
 
 `POST /push.json` accepts a registration for any active provider; a body without a
-`provider` field is a `webpush` body, so browsers need no change. `GET /push.json` returns
-the VAPID public key plus the providers currently accepting registrations. A provider that
-is disabled or not fully configured accepts no registrations and is skipped during
-delivery, so an unconfigured provider is inert rather than an error.
+`provider` field is a `webpush` body, so browsers need no change. An APNs body is
+`{"provider": "apns", "token": "...", "client_id": "..."}` - `client_id` is optional and
+identifies the device across Apple token rotations, so a new token updates that row
+instead of inserting another. A token-only body still works and revives a previously
+dead token. `GET /push.json` returns the VAPID public key plus the providers currently
+accepting registrations; when `apns` is active it includes `environment` (`production` or
+`sandbox`). A provider that is disabled or not fully configured accepts no registrations
+and is skipped during delivery, so an unconfigured provider is inert rather than an error.
 
 Every provider setting is edited at **`/admin/services/push`**: per provider an `Enabled`
 toggle, the VAPID subject for `webpush`, and team id, key id, `.p8` auth key (stored as a
@@ -951,6 +956,7 @@ in real time, bridged onto the in-process pub/sub bus by a lock-owner relay:
 | Direct message received | receiver |
 | Comment on your post | post author |
 | Reply to your comment | comment author |
+| Any comment on a post you've also commented on | every other commenter on that post |
 | `@mention` in any content | mentioned user |
 | Upvote on your content | content owner |
 | New follower | followed user |
@@ -960,10 +966,12 @@ in real time, bridged onto the in-process pub/sub bus by a lock-owner relay:
 `create_notification` schedules delivery as a fire-and-forget async task, so a dead
 subscription or push-service error never blocks the triggering request. Delivery
 (`push.notify_user`) reads a user's subscriptions once, groups them by provider, builds
-each provider's payload once, and sends over a single shared HTTP client. A subscription
-the push service reports as gone (`404`/`410` for Web Push, `410` or an `Unregistered`
-class reason for APNs) is soft-deleted; any other failure is logged and the subscription is
-kept.
+each provider's payload once, and sends over that provider's own HTTP client (stealth for
+Web Push, a dedicated HTTP/2 client for APNs). A subscription the push service reports as
+gone (`404`/`410` for Web Push, `410` or an `Unregistered` class reason for APNs) is
+soft-deleted and the provider reason is logged; any other failure is logged and the
+subscription is kept. APNs environment is stored per registration so a sandbox debug
+token and a production token can coexist.
 
 A notification is also **marked read automatically when you open the page that shows its
 content** - viewing a post clears its comment, reply, upvote and mention notifications;
@@ -1022,7 +1030,7 @@ offline. Installation requires a secure origin (HTTPS, or `localhost` for develo
 |------|------|
 | `devplacepy/push/providers/` | Provider protocol, Web Push (VAPID keys, payload encryption), APNs |
 | `devplacepy/push/store.py` | `push_registration` reads and writes |
-| `devplacepy/push/delivery.py` | `notify_user` - group by provider, deliver, reap dead subscriptions |
+| `devplacepy/push/delivery.py` | `notify_user` / `notify_registration` - group by provider, per-provider client, reap dead subscriptions |
 | `devplacepy/services/push/service.py` | Provider configuration at `/admin/services/push`, retention sweep, metrics |
 | `devplacepy/routers/push.py` | `/push.json`, `/service-worker.js`, `/manifest.json` |
 | `static/js/PushManager.js` | Service-worker registration + subscribe + opt-in UI |

devplacepy/config.py

@@ -72,6 +72,9 @@ INTERNAL_MODEL = "molodetz"
 INTERNAL_EMBED_MODEL = "molodetz~embed"
 INTERNAL_IMAGE_MODEL = "molodetz-img-small"
 
+AQUALITY_NEWS_GRADING_URL = "https://aquality.cloud.pravda.education/v1/chat/completions"
+AQUALITY_NEWS_GRADING_MODEL = "aquality"
+
 AWARD_GIVE_COOLDOWN_HOURS_DEFAULT = 24
 AWARD_RECEIVE_COOLDOWN_HOURS_DEFAULT = 24
 AWARD_DISPLAY_HOURS_DEFAULT = 24

devplacepy/constants.py

@@ -2,6 +2,16 @@
 
 TOPICS = ["devlog", "showcase", "question", "rant", "fun", "random", "politics"]
 
+TOPIC_LABELS = {
+    "devlog": "Devlog",
+    "showcase": "Showcase",
+    "question": "Question",
+    "rant": "Rant",
+    "fun": "Fun",
+    "random": "Random",
+    "politics": "Politics",
+}
+
 REACTION_EMOJI = [
     "\U0001f44d",
     "❤️",

devplacepy/content.py

@@ -46,6 +46,7 @@ from devplacepy.utils import (
     award_rewards,
     track_action,
     create_notification,
+    create_thread_notifications,
     create_mention_notifications,
     is_admin,
     is_primary_admin,
@@ -438,6 +439,7 @@ def create_comment_record(
     comment_url = f"{redirect_url}#comment-{comment_uid}"
 
     if target_type == "post":
+        already_notified = {user["uid"]}
         if parent_uid:
             parent = get_table("comments").find_one(uid=parent_uid, deleted_at=None)
             if parent and parent["user_uid"] != user["uid"]:
@@ -448,6 +450,7 @@ def create_comment_record(
                     user["uid"],
                     comment_url,
                 )
+                already_notified.add(parent["user_uid"])
         else:
             posts = get_table("posts")
             post = posts.find_one(uid=target_uid)
@@ -461,6 +464,9 @@ def create_comment_record(
                     user["uid"],
                     comment_url,
                 )
+                already_notified.add(post["user_uid"])
+
+        create_thread_notifications(target_uid, user["uid"], comment_url, already_notified)
 
     create_mention_notifications(content, user["uid"], comment_url)
     record_screening(

devplacepy/database/CLAUDE.md

@@ -198,8 +198,8 @@ Site settings are seeded on startup (`site_settings` table):
 | `site_name` / `site_description` / `site_tagline` | DevPlace branding | General site metadata |
 | `news_grade_threshold` | `"7"` | Minimum AI grade for auto-publish |
 | `news_api_url` | `"https://news.app.molodetz.nl/api"` | News source API |
-| `news_ai_url` | `"https://openai.app.molodetz.nl/v1/chat/completions"` | AI grading endpoint |
-| `news_ai_model` | `"molodetz"` | AI model identifier |
+| `news_ai_url` | `"https://aquality.cloud.pravda.education/v1/chat/completions"` | AI grading endpoint - the free, local aquality quality model by default (see `devplacepy/services/news/CLAUDE.md`) |
+| `news_ai_model` | `"aquality"` | AI model identifier |
 | `max_upload_size_mb` / `allowed_file_types` / `max_attachments_per_resource` | `"10"` / `""` / `"10"` | Upload limits |
 | `rate_limit_per_minute` | `"60"` | Mutating requests per IP per window (`main.py` middleware); a `429` carries a `Retry-After: <window>` header |
 | `rate_limit_window_seconds` | `"60"` | Rolling window for the rate limit (also the `Retry-After` value) |

devplacepy/database/core.py

@@ -9,6 +9,8 @@ from collections import defaultdict
 from datetime import datetime, timedelta, timezone
 from devplacepy.cache import TTLCache
 from devplacepy.config import (
+    AQUALITY_NEWS_GRADING_MODEL,
+    AQUALITY_NEWS_GRADING_URL,
     DATABASE_URL,
     DEFAULT_CORRECTION_PROMPT,
     DEFAULT_MODIFIER_PROMPT,

devplacepy/database/notifications.py

@@ -8,6 +8,7 @@ from .soft_delete import soft_delete
 NOTIFICATION_TYPES = [
     {"key": "comment", "label": "Comments", "description": "Someone comments on your post"},
     {"key": "reply", "label": "Replies", "description": "Someone replies to your comment"},
+    {"key": "thread", "label": "Thread activity", "description": "Someone else comments on a post you've commented on"},
     {"key": "mention", "label": "Mentions", "description": "Someone mentions you with @username"},
     {"key": "vote", "label": "Upvotes", "description": "Someone ++'d your content"},
     {"key": "follow", "label": "Followers", "description": "Someone starts following you"},

devplacepy/database/schema.py

@@ -1,6 +1,6 @@
 # retoor <retoor@molodetz.nl>
 
-from .core import DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT, INTERNAL_GATEWAY_URL, _drop_index, _index, _uid_index, db, defaultdict, get_table, logger
+from .core import AQUALITY_NEWS_GRADING_MODEL, AQUALITY_NEWS_GRADING_URL, DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT, INTERNAL_GATEWAY_URL, _drop_index, _index, _uid_index, db, defaultdict, get_table, logger
 from .settings import get_setting, set_setting
 from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns
 from .ranking import _authors_cache
@@ -130,6 +130,7 @@ def init_db():
         ("content", ""),
         ("read", False),
         ("created_at", ""),
+        ("updated_at", ""),
     ):
         if not messages.has_column(column):
             messages.create_column_by_example(column, example)
@@ -144,6 +145,7 @@ def init_db():
         "idx_messages_conversation_rev",
         ["receiver_uid", "sender_uid"],
     )
+    _index(db, "messages", "idx_messages_updated_at", ["updated_at"])
     _index(db, "notifications", "idx_notifications_user", ["user_uid"])
     _index(db, "notifications", "idx_notifications_user_read", ["user_uid", "read"])
     push_registration = get_table("push_registration")
@@ -155,19 +157,38 @@ def init_db():
         ("key_auth", ""),
         ("key_p256dh", ""),
         ("token", ""),
+        ("client_id", ""),
+        ("environment", ""),
         ("created_at", ""),
+        ("registered_at", ""),
         ("deleted_at", ""),
     ):
         if not push_registration.has_column(column):
             push_registration.create_column_by_example(column, example)
     _index(db, "push_registration", "idx_push_registration_user", ["user_uid"])
     _index(db, "push_registration", "idx_push_registration_provider", ["provider"])
+    _index(
+        db,
+        "push_registration",
+        "idx_push_registration_client",
+        ["user_uid", "provider", "client_id"],
+    )
+    _index(
+        db,
+        "push_registration",
+        "idx_push_registration_token",
+        ["user_uid", "provider", "token"],
+    )
     if "push_registration" in db.tables:
         with db:
             db.query(
                 "UPDATE push_registration SET provider = 'webpush' "
                 "WHERE provider IS NULL OR provider = ''"
             )
+            db.query(
+                "UPDATE push_registration SET registered_at = created_at "
+                "WHERE registered_at IS NULL OR registered_at = ''"
+            )
     _index(db, "sessions", "idx_sessions_token", ["session_token"])
     projects = get_table("projects")
     for column, example in (
@@ -1799,8 +1820,8 @@ def init_db():
         news_defaults = {
             "news_grade_threshold": "7",
             "news_api_url": "https://news.app.molodetz.nl/api",
-            "news_ai_url": INTERNAL_GATEWAY_URL,
-            "news_ai_model": "molodetz",
+            "news_ai_url": AQUALITY_NEWS_GRADING_URL,
+            "news_ai_model": AQUALITY_NEWS_GRADING_MODEL,
         }
         for key, value in news_defaults.items():
             existing = db["site_settings"].find_one(key=key)
@@ -2046,6 +2067,12 @@ def migrate_ai_gateway_settings() -> None:
         if get_setting(key, "") == OLD_GATEWAY_URL:
             set_setting(key, INTERNAL_GATEWAY_URL)
             logger.info(f"Migrated {key} to the internal gateway")
+    if get_setting("news_ai_url", "") == INTERNAL_GATEWAY_URL:
+        set_setting("news_ai_url", AQUALITY_NEWS_GRADING_URL)
+        logger.info("Migrated news_ai_url to the free aquality grading model")
+    if get_setting("news_ai_model", "") in ("molodetz", ""):
+        set_setting("news_ai_model", AQUALITY_NEWS_GRADING_MODEL)
+        logger.info("Migrated news_ai_model to aquality")
     if get_setting("bot_model", "") == "deepseek-chat":
         set_setting("bot_model", "molodetz")
         logger.info("Migrated bot_model to molodetz")

devplacepy/docs_api/groups/content.py

@@ -64,6 +64,33 @@ four ways to sign requests.
                     field("before", "query", "string", False, "", "Pagination cursor."),
                 ],
             ),
+            endpoint(
+                id="topics-hub",
+                method="GET",
+                path="/topics",
+                title="Browse topics",
+                summary="The topics hub - every post topic with its live post count, linking to its own crawlable listing page.",
+                auth="public",
+                interactive=True,
+            ),
+            endpoint(
+                id="topics-list",
+                method="GET",
+                path="/topics/{topic}",
+                title="Browse one topic",
+                summary=(
+                    "A single topic's post listing, on its own permanent, crawlable URL (unlike /feed?topic=, "
+                    "whose canonical collapses back to /feed). Same author-interleaved pagination as the feed."
+                ),
+                auth="public",
+                interactive=True,
+                params=[
+                    field(
+                        "topic", "path", "enum", True, "devlog", "Topic key.", TOPICS
+                    ),
+                    field("before", "query", "string", False, "", "Pagination cursor."),
+                ],
+            ),
             endpoint(
                 id="posts-create",
                 method="POST",

devplacepy/docs_api/groups/messaging.py

@@ -41,6 +41,14 @@ four ways to sign requests.
                         "",
                         "Jump to a conversation by username.",
                     ),
+                    field(
+                        "before",
+                        "query",
+                        "string",
+                        False,
+                        "",
+                        "ISO timestamp. When set, return the page of messages strictly older than this instant (for loading earlier history).",
+                    ),
                 ],
             ),
             endpoint(

devplacepy/docs_api/groups/push.py

@@ -14,9 +14,11 @@ implements the Web Push protocol: fetch the public VAPID key, then register a
 Push Notification service device token and is only offered when an administrator has
 configured it.
 
-`GET /push.json` lists the providers that currently accept registrations. A registration
-body without a `provider` field is a `webpush` registration, so existing clients need no
-change.
+`GET /push.json` lists the providers that currently accept registrations. When `apns` is
+active it includes `environment` (`production` or `sandbox`) so a native client can match
+its build. A registration body without a `provider` field is a `webpush` registration, so
+existing clients need no change. An APNs body may include a stable `client_id` so a later
+token rotation updates the same device instead of inserting another row.
 
 There is no server-side unsubscribe endpoint: unsubscription is handled entirely in the
 browser by calling `PushManager.unsubscribe()` on the subscription. The server stops delivering
@@ -37,7 +39,10 @@ four ways to sign requests.
                 auth="public",
                 sample_response={
                     "publicKey": "BASE64_VAPID_KEY",
-                    "providers": {"webpush": {"publicKey": "BASE64_VAPID_KEY"}},
+                    "providers": {
+                        "webpush": {"publicKey": "BASE64_VAPID_KEY"},
+                        "apns": {"environment": "production"},
+                    },
                 },
             ),
             endpoint(
@@ -80,15 +85,24 @@ four ways to sign requests.
                         "string",
                         False,
                         "a1b2c3...",
-                        "Hexadecimal device token. Required for apns.",
+                        "Hexadecimal device token. Required for apns. Spaces and angle brackets are stripped.",
+                    ),
+                    field(
+                        "client_id",
+                        "json",
+                        "string",
+                        False,
+                        "vendor-uuid",
+                        "Stable per-device id for apns. When present, a new token updates this device instead of inserting a row.",
                     ),
                 ],
                 notes=[
                     'A webpush body is JSON: `{"endpoint": "...", "keys": {"p256dh": "...", "auth": "..."}}`.',
-                    'An APNs body is JSON: `{"provider": "apns", "token": "..."}`.',
+                    'An APNs body is JSON: `{"provider": "apns", "token": "...", "client_id": "..."}`. `client_id` is optional; token-only bodies keep working and revive a previously dead token.',
                     "A provider that is unknown, disabled or unconfigured returns 400.",
+                    "A newly created or revived registration is probed immediately. The response then includes `delivered` and, on failure, `error` with the provider reason. `registered` stays true so existing clients keep working.",
                 ],
-                sample_response={"registered": True},
+                sample_response={"registered": True, "delivered": True},
             ),
         ],
 }

devplacepy/docs_api/groups/tools.py

@@ -226,6 +226,7 @@ status and report.
                         {"title": "Invention", "detail": "...", "confidence": 0.8, "citations": [1]}
                     ],
                     "sources": [{"url": "https://example.com", "title": "Example", "source": "httpx"}],
+                    "follow_up_questions": ["Who else worked on the transistor?", "How did it replace vacuum tubes?"],
                     "chat_ws_url": "/tools/deepsearch/DEEPSEARCH_JOB_UID/chat",
                     "export_md_url": "/tools/deepsearch/DEEPSEARCH_JOB_UID/export.md",
                     "export_json_url": "/tools/deepsearch/DEEPSEARCH_JOB_UID/export.json",

devplacepy/main.py

@@ -52,6 +52,7 @@ from devplacepy.routers import (
     battles,
     feed,
     posts,
+    topics,
     comments,
     projects,
     profile,
@@ -315,6 +316,9 @@ async def lifespan(app: FastAPI):
     from devplacepy.services.containers import forward
 
     await forward.close_client()
+    from devplacepy import push
+
+    await push.shutdown_providers()
 
 
 app = FastAPI(
@@ -471,6 +475,7 @@ async def on_validation_error(request: Request, exc: RequestValidationError):
 app.include_router(auth.router, prefix="/auth")
 app.include_router(feed.router, prefix="/feed")
 app.include_router(posts.router, prefix="/posts")
+app.include_router(topics.router, prefix="/topics")
 app.include_router(comments.router, prefix="/comments")
 app.include_router(projects.router, prefix="/projects")
 app.include_router(profile.router, prefix="/profile")

devplacepy/push/CLAUDE.md

@@ -2,33 +2,35 @@ This file documents `devplacepy/push/` - push notification delivery and its prov
 
 ## What this package is
 
-One delivery library behind one public surface. `devplacepy.push` exports `notify_user`, `register`, `ensure_certificates`, `public_key_standard_b64` and the Web Push crypto helpers; every caller in the codebase (`main.py`, `utils/notifications.py`, `routers/push.py`) imports only those names. Everything else is internal to the package.
+One delivery library behind one public surface. `devplacepy.push` exports `notify_user`, `notify_registration`, `register`, `ensure_certificates`, `public_key_standard_b64` and the Web Push crypto helpers; every caller in the codebase (`main.py`, `utils/notifications.py`, `routers/push.py`) imports only those names. Everything else is internal to the package.
 
 | Module | Role |
 |---|---|
 | `providers/base.py` | `PushProvider` protocol, the `Delivery` outcome and the three outcome constants |
 | `providers/webpush.py` | VAPID key material, `aesgcm` payload encryption, the Web Push provider |
-| `providers/apns.py` | Apple Push Notification service provider (token based, HTTP/2) |
+| `providers/apns.py` | Apple Push Notification service provider (token based, dedicated HTTP/2 client) |
 | `providers/__init__.py` | `PROVIDERS` registry, `get`, `active`, `is_active`, `admin_fields`, `client_config` |
-| `store.py` | Every `push_registration` read and write |
-| `delivery.py` | `notify_user`: group by provider, one shared client, one prepared body per provider |
+| `store.py` | Every `push_registration` read and write, including identity upsert and revive |
+| `delivery.py` | `notify_user` / `notify_registration`: group by provider, one client per provider, one prepared body per provider |
 
 The admin configuration surface lives in `devplacepy/services/push/service.py` (`PushService`), not here.
 
 ## Adding a provider
 
-1. Write `providers/<name>.py` with a `PushProvider` subclass: `name`, `label`, `config_fields`, `is_configured`, `parse_registration`, `prepare`, `deliver`, optionally `client_config`.
+1. Write `providers/<name>.py` with a `PushProvider` subclass: `name`, `label`, `config_fields`, `is_configured`, `parse_registration`, `prepare`, `deliver`, optionally `client_config`, `stamp_registration`, `delivery_client`.
 2. Add one entry to `PROVIDERS` in `providers/__init__.py`.
 
 That is the whole change. The registration route, the delivery loop, the admin page, the audit record, the metrics and the docs are written against the protocol and need no edit. The `Enabled` toggle (`push_<name>_enabled`) comes from the base class, so a provider never declares its own.
 
+The default `delivery_client` is `stealth_async_client`. Override it only when the destination is a first-party API that must not see Chrome impersonation, PRIORITY frames, extra browser headers, or the outbound proxy. APNs is that case.
+
 ## Invariants
 
-- **Zero cost for the request.** Delivery is reached only through `utils/notifications.py` `_schedule_push`, a fire-and-forget task. Never make a route await `notify_user`, and never add a queue or a table to this path.
+- **Zero cost for the request, except the welcome probe.** Delivery of real notifications is reached only through `utils/notifications.py` `_schedule_push`, a fire-and-forget task. `POST /push.json` awaits `notify_registration` only for a newly created or revived row, so the client can see `delivered` / `error`. Never add a queue or a table to this path.
 - **`deliver` never raises.** Return `Delivery(REJECTED, detail)` instead. `delivery.py` guards anyway, but a raising provider costs a log line per subscription.
 - **A provider that is not configured is inert, never an error.** `is_configured()` is false, `is_active()` is false, the delivery loop skips it, and `POST /push.json` refuses a registration for it with 400. Nothing else in the platform notices.
-- **`DEAD` is the only outcome that touches the database.** It soft-deletes the registration (`deleted_at`), exactly like a `404`/`410` Web Push endpoint always did. `REJECTED` keeps the row.
-- **Every insert writes `deleted_at: None`,** and every read filters `deleted_at IS NULL`. `push_registration` deliberately stays out of `SOFT_DELETE_TABLES` (no `deleted_by`, not restorable from Trash) - a dead device token has no owner action to undo.
+- **`DEAD` is the only outcome that touches the database.** It soft-deletes the registration (`deleted_at`), exactly like a `404`/`410` Web Push endpoint always did - unless `Delivery.dead_before` names an instant the row was proven live again after (APNs 410 `timestamp` vs. `registered_at`, see "APNs specifics"), in which case the delete is skipped. `REJECTED` keeps the row. The Apple/Web Push reason is logged at WARNING (`Push dead via ...: <detail>`); do not drop `outcome.detail`.
+- **Every insert writes `deleted_at: None`,** and every read of the live set filters `deleted_at IS NULL`. Identity lookups for upsert/revive intentionally ignore `deleted_at` so a client that re-registers a previously dead token comes back live. `push_registration` deliberately stays out of `SOFT_DELETE_TABLES` (no `deleted_by`, not restorable from Trash) - a dead device token has no owner action to undo.
 - **A row without a provider is a Web Push row.** `store.provider_of` resolves `None`/`""` to `DEFAULT_PROVIDER`, so a row written by an old worker during a deploy still delivers. `init_db` backfills the column once with a single converging `UPDATE`.
 
 ## Storage
@@ -40,13 +42,31 @@ That is the whole change. The registration route, the delivery loop, the admin p
 | `provider` | `webpush` | `apns` |
 | `endpoint`, `key_auth`, `key_p256dh` | set | `NULL` |
 | `token` | `NULL` | device token |
+| `client_id` | unused | optional stable per-device id |
+| `environment` | unused | `production` or `sandbox`, stamped server-side at register |
+| `registered_at` | stamped on insert/merge | stamped on insert/merge |
 
-Deduplication is generic: `store.register` looks up `user_uid` + `provider` + exactly the fields the provider's `parse_registration` returned, so a provider never writes its own identity rule.
+`registered_at` (ISO, both providers) is stamped on insert and on every `_merge` that actually changes a row (including revival). It exists solely to arbitrate the dead-token race described below - it is not a general "last seen" field.
+
+`store.register` returns `RegistrationWrite(record, created, revived)`. Identity, in order:
+
+1. `user_uid` + `provider` + `client_id` (live or dead), if `client_id` is present.
+2. `user_uid` + `provider` + `token` (or `endpoint` for Web Push), live or dead.
+3. Exact live match on the remaining fields.
+4. Insert.
+
+A match **updates** the row (token, client_id, environment) and clears `deleted_at` if it was dead. Token rotation with the same `client_id` therefore replaces the token on one row. Sibling live rows that share the new token, the previous token, or the same `client_id` are marked dead so one device cannot accumulate duplicates. A body without `client_id` still works: the same token revives, a new token inserts. `push.update` is a real update, not a no-op of an identical POST.
 
 ## APNs specifics
 
-- `POST https://{host}/3/device/{token}` over HTTP/2, host from `push_apns_environment` (`api.push.apple.com` or `api.sandbox.push.apple.com`; an unrecognised value falls back to production). HTTP/2 comes from `stealth_async_client` because the origin is `https` - the cleartext downgrade in `curl_transport` does not apply.
-- Provider token: `ES256`, header `kid` = key id, claims `iss` = team id and `iat`. Cached per credential fingerprint for 45 minutes, so a worker signs at most one token per 45 minutes; Apple refuses tokens regenerated faster than every 20 minutes. Editing any credential changes the fingerprint and takes effect on the next delivery, with no restart.
+- **Persistent HTTP/2 connection, not one per notification.** `apns.cached_client(timeout)` lazily creates ONE module-level `httpx.AsyncClient(http2=True, ...)` and reuses it across every `notify_user`/`notify_registration` call for the lifetime of the process, following Apple's explicit guidance to keep the connection open rather than repeatedly opening/closing (`sending-notification-requests-to-apns`). `ApnsProvider.closes_delivery_client()` returns `False` so `delivery.py` never closes it after a batch (Web Push still opens/closes per call via `stealth_async_client`, `closes_delivery_client()` defaulting `True` on the base class). Closed once, gracefully, in `main.py`'s shutdown via `push.shutdown_providers()` -> `ApnsProvider.aclose()` -> `apns.close_client()`, mirroring the identical `services/containers/forward.py` `client()`/`close_client()` pattern. Not stealth, not curl_cffi Chrome impersonation, not the outbound proxy - Apple's provider API is a first-party HTTP/2 service; browser PRIORITY frames, HPACK indexing of `:path`/`authorization`, extra `sec-ch-ua` headers, and a scraping proxy all violate that contract. This is the second documented exception to the stealth-only outbound rule (the other is container reverse-proxy forwarding). If the admin-configured delivery timeout changes, the cached client is rebuilt with the new timeout on next use and the old one is left for GC (not explicitly closed) - a deliberate, rare-path simplification.
+- `POST https://{host}/3/device/{token}` over HTTP/2. Host comes from the **row's** `environment` if set, else `push_apns_environment` (`api.push.apple.com` or `api.sandbox.push.apple.com`; an unrecognised value falls back to production). Stamping environment per row lets a sandbox debug token and a production TestFlight token coexist.
+- `GET /push.json` advertises `providers.apns.environment` when APNs is active so a native client can refuse to register a sandbox token against production.
+- `parse_registration` requires a hexadecimal `token` (64-200 digits after stripping spaces and `<>`) and optionally `client_id` (string, max 128). A `client_id` that is not a string is a 400, not a silent drop. Server stamps `environment`; the client cannot pick the host.
+- **Provider token (JWT):** `ES256`, header `kid` = key id, claims `iss` = team id and `iat`. Cached per credential fingerprint for 45 minutes (Apple refuses tokens regenerated faster than every 20 minutes, and requires a fresh one at least once an hour). **Shared cross-worker via `site_settings` key `push_apns_shared_token`** (JSON `{fingerprint, token, issued_at}`, read/written through the existing `get_setting`/`set_setting` cache-version machinery, propagating to sibling workers within ~1s like every other settings key) - a worker that finds no valid in-process cache checks the shared copy before signing a new one, so `make prod`'s multiple uvicorn workers converge on presenting Apple the SAME token instead of each independently re-signing on its own clock (which could otherwise interleave closer together than Apple's per-credential minimum spacing, worst case when workers boot near-simultaneously). Editing any credential changes the fingerprint and takes effect on the next delivery, with no restart.
+- **A rejected provider token is evicted immediately, not left to expire from cache.** A 403 status, or any reason in `AUTH_REASONS` (`InvalidProviderToken`, `ExpiredProviderToken`, `BadCertificate`, `BadCertificateEnvironment`, `Forbidden`, `MissingProviderToken` - all provider-credential-shaped per Apple's reason table, never about a specific device), calls `invalidate_provider_token()`: clears the in-process cache AND the shared DB copy, so the very next delivery attempt (any worker) re-signs instead of retrying the same rejected token for up to 45 minutes.
 - A `.p8` that does not parse is cached as a failure for the same window, so a misconfiguration costs one error log per window rather than one parse per notification.
-- `410`, or any status carrying reason `BadDeviceToken`, `Unregistered`, `ExpiredToken`, `DeviceTokenNotForTopic` or `TopicDisallowed`, is `DEAD`. Everything else is `REJECTED`.
+- **`DEAD_REASONS` is deliberately narrow: only `BadDeviceToken` (400), `Unregistered` (410) and `ExpiredToken` (410).** These are the only reasons in Apple's documented table that describe the TOKEN itself as permanently invalid. `DeviceTokenNotForTopic` and `TopicDisallowed` are 400 errors about the **topic/provisioning matching the connection**, not the token - they fire identically for every token when `push_apns_topic` is misconfigured or the certificate/entitlements don't match, so treating them as dead would soft-delete the entire APNs subscriber base (a table deliberately kept OUT of `SOFT_DELETE_TABLES`, hence unrestorable) on the first delivery after a one-field admin typo. Never add a topic/provisioning-shaped reason to `DEAD_REASONS`.
+- **A 410's `timestamp` is honored before deleting.** Apple's 410 body carries `{"reason": "Unregistered", "timestamp": <ms epoch>}` - the instant Apple last confirmed the token dead, which can trail real device state by "several days" per Apple engineering guidance (410 delivery is intentionally non-deterministic; do not use it to infer app-uninstall timing). `apns._dead_before` converts it to ISO and it rides on `Delivery.dead_before`; `store.mark_dead(id, dead_before)` compares it against the row's `registered_at` and **skips the delete** (logs at INFO instead) if the registration was re-registered/revived after that instant - closing the race where an in-flight delivery against a stale row state would otherwise undo a concurrent revival. `dead_before` is only ever set for a 410; the `BadDeviceToken`/`ExpiredToken` paths pass `None` (unconditional delete, as before), since those reasons carry no `timestamp`.
 - The shared payload dict (`title`, `message`, `icon`, `url`) is translated once per batch into `aps.alert` plus the custom `url`/`icon` keys, mirroring what `service-worker.js` does for Web Push. `thread-id` mirrors the service worker's notification `tag`.
+- `POST /push.json` probes a created or revived row immediately via `notify_registration`. The JSON is `{registered: true, delivered: bool, error?: string}`. `registered` stays true so existing clients keep working; `delivered`/`error` surface Apple's reason instead of killing the row silently from the client's point of view. The row is still marked dead on a `DEAD` outcome (subject to the `dead_before` guard above) so later `notify_user` calls do not keep hitting a known-bad token.

devplacepy/push/__init__.py

@@ -1,6 +1,7 @@
 # retoor <retoor@molodetz.nl>
 
-from devplacepy.push.delivery import notify_user
+from devplacepy.push.delivery import notify_registration, notify_user
+from devplacepy.push.providers import shutdown as shutdown_providers
 from devplacepy.push.providers.webpush import (
     browser_base64,
     create_notification_authorization,
@@ -23,7 +24,9 @@ __all__ = [
     "generate_private_key",
     "generate_public_key",
     "hkdf",
+    "notify_registration",
     "notify_user",
     "public_key_standard_b64",
     "register",
+    "shutdown_providers",
 ]

devplacepy/push/delivery.py

@@ -3,9 +3,9 @@
 import logging
 from typing import Any
 
-from devplacepy import stealth
 from devplacepy.database import get_int_setting
 from devplacepy.push import providers, store
+from devplacepy.push.providers.base import Delivery
 
 logger = logging.getLogger(__name__)
 
@@ -29,6 +29,10 @@ def group_by_provider(
     return grouped
 
 
+def _open_client(provider, timeout: float):
+    return provider.delivery_client(timeout)
+
+
 async def notify_user(user_uid: str, payload: dict[str, Any]) -> None:
     registrations = store.active_for_user(user_uid)
     if not registrations:
@@ -36,48 +40,86 @@ async def notify_user(user_uid: str, payload: dict[str, Any]) -> None:
         return
 
     grouped = group_by_provider(registrations)
-    async with stealth.stealth_async_client(timeout=timeout_seconds()) as client:
-        for name, rows in grouped.items():
-            provider = providers.PROVIDERS.get(name)
-            if provider is None:
-                logger.warning(
-                    "Unknown push provider %s on %s subscriptions of user %s",
-                    name,
-                    len(rows),
-                    user_uid,
-                )
-                continue
-            if not providers.is_active(provider):
-                logger.debug(
-                    "Push provider %s is not active; skipping %s subscriptions",
-                    name,
-                    len(rows),
-                )
-                continue
-            try:
-                prepared = provider.prepare(payload)
-            except Exception as exc:
-                logger.error("Push provider %s could not build a payload: %s", name, exc)
-                continue
+    timeout = timeout_seconds()
+    for name, rows in grouped.items():
+        provider = providers.PROVIDERS.get(name)
+        if provider is None:
+            logger.warning(
+                "Unknown push provider %s on %s subscriptions of user %s",
+                name,
+                len(rows),
+                user_uid,
+            )
+            continue
+        if not providers.is_active(provider):
+            logger.debug(
+                "Push provider %s is not active; skipping %s subscriptions",
+                name,
+                len(rows),
+            )
+            continue
+        try:
+            prepared = provider.prepare(payload)
+        except Exception as exc:
+            logger.error("Push provider %s could not build a payload: %s", name, exc)
+            continue
+        try:
+            client = _open_client(provider, timeout)
+        except Exception as exc:
+            logger.error("Push provider %s could not open a client: %s", name, exc)
+            continue
+        try:
             for registration in rows:
                 await _deliver_one(provider, client, registration, prepared, user_uid)
+        finally:
+            if provider.closes_delivery_client():
+                await client.aclose()
 
 
-async def _deliver_one(provider, client, registration, prepared, user_uid) -> None:
+async def notify_registration(
+    registration: dict[str, Any], payload: dict[str, Any]
+) -> Delivery:
+    name = store.provider_of(registration)
+    provider = providers.PROVIDERS.get(name)
+    if provider is None:
+        return Delivery(providers.REJECTED, f"unknown provider {name}")
+    if not providers.is_active(provider):
+        return Delivery(providers.REJECTED, f"provider {name} is not active")
+    try:
+        prepared = provider.prepare(payload)
+    except Exception as exc:
+        return Delivery(providers.REJECTED, str(exc))
+    user_uid = registration.get("user_uid") or ""
+    try:
+        client = _open_client(provider, timeout_seconds())
+    except Exception as exc:
+        return Delivery(providers.REJECTED, str(exc))
+    try:
+        return await _deliver_one(provider, client, registration, prepared, user_uid)
+    finally:
+        if provider.closes_delivery_client():
+            await client.aclose()
+
+
+async def _deliver_one(provider, client, registration, prepared, user_uid) -> Delivery:
     try:
         outcome = await provider.deliver(client, registration, prepared)
     except Exception as exc:
         logger.error("Push provider %s raised for %s: %s", provider.name, user_uid, exc)
-        return
+        return Delivery(providers.REJECTED, str(exc))
     if outcome.status == providers.ACCEPTED:
         logger.debug("Push delivered to %s via %s", user_uid, provider.name)
-        return
+        return outcome
     if outcome.status == providers.DEAD:
+        logger.warning(
+            "Push dead via %s for %s: %s", provider.name, user_uid, outcome.detail
+        )
         try:
-            store.mark_dead(registration["id"])
+            store.mark_dead(registration["id"], outcome.dead_before)
         except Exception as exc:
             logger.error("Could not soft-delete push subscription: %s", exc)
-        return
+        return outcome
     logger.warning(
         "Push rejected by %s for %s: %s", provider.name, user_uid, outcome.detail
     )
+    return outcome

devplacepy/push/providers/__init__.py

@@ -35,6 +35,7 @@ __all__ = [
     "get",
     "is_active",
     "names",
+    "shutdown",
 ]
 
 
@@ -74,3 +75,11 @@ def _client_config(provider: PushProvider) -> dict[str, Any]:
     except Exception as exc:
         logger.error("Push provider %s failed to describe itself: %s", provider.name, exc)
         return {}
+
+
+async def shutdown() -> None:
+    for provider in PROVIDERS.values():
+        try:
+            await provider.aclose()
+        except Exception as exc:
+            logger.error("Push provider %s failed to close: %s", provider.name, exc)

devplacepy/push/providers/apns.py

@@ -5,13 +5,14 @@ import json
 import logging
 import string
 import time
+from datetime import datetime, timezone
 from typing import Any
 
 import httpx
 import jwt
 
 from devplacepy.config import SECONDS_PER_DAY
-from devplacepy.database import get_setting
+from devplacepy.database import get_setting, set_setting
 from devplacepy.push.providers.base import (
     ACCEPTED,
     DEAD,
@@ -44,20 +45,35 @@ ENVIRONMENT_OPTIONS = [
 TOKEN_REFRESH_SECONDS = 45 * 60
 TOKEN_MIN_LENGTH = 64
 TOKEN_MAX_LENGTH = 200
+CLIENT_ID_MAX_LENGTH = 128
 THREAD_ID = "devplace-notification"
 PUSH_TYPE = "alert"
 PRIORITY = "10"
-DEAD_REASONS = frozenset(
+SHARED_TOKEN_KEY = "push_apns_shared_token"
+
+# Per Apple's documented reason table, only these mean the TOKEN itself is
+# permanently dead (410 Unregistered/ExpiredToken, 400 BadDeviceToken).
+# DeviceTokenNotForTopic and TopicDisallowed are topic/provisioning
+# misconfigurations at 400 - they affect every token uniformly and must
+# never be treated as a reason to delete a registration.
+DEAD_REASONS = frozenset({"BadDeviceToken", "ExpiredToken", "Unregistered"})
+
+# 403 reasons meaning the provider (JWT) token/credential itself was
+# rejected, not any device token. The cached token must be dropped so the
+# next attempt re-signs instead of retrying the same rejected token.
+AUTH_REASONS = frozenset(
     {
-        "BadDeviceToken",
-        "DeviceTokenNotForTopic",
-        "ExpiredToken",
-        "Unregistered",
-        "TopicDisallowed",
+        "BadCertificate",
+        "BadCertificateEnvironment",
+        "ExpiredProviderToken",
+        "Forbidden",
+        "InvalidProviderToken",
+        "MissingProviderToken",
     }
 )
 
 _token_state: dict[str, Any] = {}
+_client: httpx.AsyncClient | None = None
 
 
 def _setting(key: str) -> str:
@@ -70,13 +86,78 @@ def _environment() -> str:
 
 
 def host() -> str:
-    return HOSTS[_environment()]
+    return host_for(_environment())
+
+
+def host_for(environment: str | None) -> str:
+    value = (environment or "").strip() or _environment()
+    return HOSTS[value] if value in HOSTS else HOSTS[DEFAULT_ENVIRONMENT]
+
+
+def gateway_client(timeout: float) -> httpx.AsyncClient:
+    return httpx.AsyncClient(
+        http2=True,
+        timeout=timeout,
+        trust_env=False,
+        verify=True,
+        follow_redirects=False,
+    )
+
+
+def cached_client(timeout: float) -> httpx.AsyncClient:
+    global _client
+    if _client is None or _client.is_closed:
+        _client = gateway_client(timeout)
+    return _client
+
+
+async def close_client() -> None:
+    global _client
+    if _client is not None and not _client.is_closed:
+        await _client.aclose()
+    _client = None
+
+
+def _normalize_token(token: str) -> str:
+    return "".join(
+        character.lower() for character in token if character in string.hexdigits
+    )
 
 
 def _fingerprint(team_id: str, key_id: str, auth_key: str) -> str:
     return hashlib.sha256(f"{team_id}:{key_id}:{auth_key}".encode("utf-8")).hexdigest()
 
 
+def _read_shared_token() -> dict[str, Any] | None:
+    raw = get_setting(SHARED_TOKEN_KEY, "")
+    if not raw:
+        return None
+    try:
+        data = json.loads(raw)
+    except ValueError:
+        return None
+    if (
+        not isinstance(data, dict)
+        or not isinstance(data.get("fingerprint"), str)
+        or not isinstance(data.get("token"), str)
+        or not isinstance(data.get("issued_at"), int)
+    ):
+        return None
+    return data
+
+
+def _write_shared_token(fingerprint: str, token: str, issued_at: int) -> None:
+    set_setting(
+        SHARED_TOKEN_KEY,
+        json.dumps({"fingerprint": fingerprint, "token": token, "issued_at": issued_at}),
+    )
+
+
+def invalidate_provider_token() -> None:
+    _token_state.pop("current", None)
+    set_setting(SHARED_TOKEN_KEY, "")
+
+
 def provider_token(team_id: str, key_id: str, auth_key: str) -> str:
     fingerprint = _fingerprint(team_id, key_id, auth_key)
     issued_at = int(time.time())
@@ -89,6 +170,21 @@ def provider_token(team_id: str, key_id: str, auth_key: str) -> str:
         if state["token"] is None:
             raise ValueError(state["error"])
         return state["token"]
+
+    shared = _read_shared_token()
+    if (
+        shared
+        and shared["fingerprint"] == fingerprint
+        and issued_at - shared["issued_at"] < TOKEN_REFRESH_SECONDS
+    ):
+        _token_state["current"] = {
+            "token": shared["token"],
+            "error": "",
+            "issued_at": shared["issued_at"],
+            "fingerprint": fingerprint,
+        }
+        return shared["token"]
+
     try:
         token = jwt.encode(
             {"iss": team_id, "iat": issued_at},
@@ -112,6 +208,7 @@ def provider_token(team_id: str, key_id: str, auth_key: str) -> str:
         "issued_at": issued_at,
         "fingerprint": fingerprint,
     }
+    _write_shared_token(fingerprint, token, issued_at)
     return token
 
 
@@ -125,6 +222,22 @@ def _reason(response: httpx.Response) -> str:
     return ""
 
 
+def _dead_before(response: httpx.Response) -> str | None:
+    try:
+        body = response.json()
+    except ValueError:
+        return None
+    if not isinstance(body, dict):
+        return None
+    raw = body.get("timestamp")
+    if not isinstance(raw, (int, float)) or isinstance(raw, bool):
+        return None
+    try:
+        return datetime.fromtimestamp(raw / 1000, tz=timezone.utc).isoformat()
+    except (OverflowError, OSError, ValueError):
+        return None
+
+
 class ApnsProvider(PushProvider):
     name = "apns"
     label = PROVIDER_LABEL
@@ -181,16 +294,41 @@ class ApnsProvider(PushProvider):
             and _setting(TOPIC_KEY)
         )
 
+    def client_config(self) -> dict[str, Any]:
+        return {"environment": _environment()}
+
+    def stamp_registration(self, fields: dict[str, Any]) -> dict[str, Any]:
+        return {**fields, "environment": _environment()}
+
+    def delivery_client(self, timeout: float) -> httpx.AsyncClient:
+        return cached_client(timeout)
+
+    def closes_delivery_client(self) -> bool:
+        return False
+
+    async def aclose(self) -> None:
+        await close_client()
+
     def parse_registration(self, body: dict[str, Any]) -> dict[str, Any] | None:
         token = body.get("token")
         if not isinstance(token, str):
             return None
-        token = token.strip()
+        token = _normalize_token(token)
         if not TOKEN_MIN_LENGTH <= len(token) <= TOKEN_MAX_LENGTH:
             return None
-        if any(character not in string.hexdigits for character in token):
+        fields: dict[str, Any] = {"token": token}
+        client_id = body.get("client_id")
+        if client_id is None:
+            return fields
+        if not isinstance(client_id, str):
             return None
-        return {"token": token}
+        client_id = client_id.strip()
+        if not client_id:
+            return fields
+        if len(client_id) > CLIENT_ID_MAX_LENGTH:
+            return None
+        fields["client_id"] = client_id
+        return fields
 
     def prepare(self, payload: dict[str, Any]) -> str:
         return json.dumps(
@@ -228,7 +366,7 @@ class ApnsProvider(PushProvider):
         try:
             headers = self.headers()
             response = await client.post(
-                f"https://{host()}/3/device/{token}",
+                f"https://{host_for(registration.get('environment'))}/3/device/{token}",
                 headers=headers,
                 content=prepared.encode("utf-8"),
             )
@@ -238,6 +376,9 @@ class ApnsProvider(PushProvider):
             return Delivery(ACCEPTED)
         reason = _reason(response)
         detail = f"{response.status_code} {reason}".strip()
+        if response.status_code == 403 or reason in AUTH_REASONS:
+            invalidate_provider_token()
         if response.status_code == 410 or reason in DEAD_REASONS:
-            return Delivery(DEAD, detail)
+            dead_before = _dead_before(response) if response.status_code == 410 else None
+            return Delivery(DEAD, detail, dead_before)
         return Delivery(REJECTED, detail)

devplacepy/push/providers/base.py

@@ -18,6 +18,7 @@ REJECTED = "rejected"
 class Delivery:
     status: str
     detail: str = ""
+    dead_before: str | None = None
 
 
 class PushProvider(ABC):
@@ -51,6 +52,20 @@ class PushProvider(ABC):
     def client_config(self) -> dict[str, Any]:
         return {}
 
+    def stamp_registration(self, fields: dict[str, Any]) -> dict[str, Any]:
+        return fields
+
+    def delivery_client(self, timeout: float) -> httpx.AsyncClient:
+        from devplacepy import stealth
+
+        return stealth.stealth_async_client(timeout=timeout)
+
+    def closes_delivery_client(self) -> bool:
+        return True
+
+    async def aclose(self) -> None:
+        return None
+
     @abstractmethod
     def is_configured(self) -> bool: ...
 

devplacepy/push/store.py

@@ -1,6 +1,7 @@
 # retoor <retoor@molodetz.nl>
 
 import logging
+from dataclasses import dataclass
 from datetime import datetime, timezone
 from typing import Any
 
@@ -13,6 +14,17 @@ logger = logging.getLogger(__name__)
 TABLE = "push_registration"
 
 
+@dataclass(frozen=True)
+class RegistrationWrite:
+    record: dict[str, Any]
+    created: bool
+    revived: bool
+
+    @property
+    def probe(self) -> bool:
+        return self.created or self.revived
+
+
 def table():
     return get_table(TABLE)
 
@@ -25,31 +37,161 @@ def active_for_user(user_uid: str) -> list[dict[str, Any]]:
     return list(table().find(user_uid=user_uid, deleted_at=None))
 
 
+def _filled(fields: dict[str, Any]) -> dict[str, Any]:
+    filled: dict[str, Any] = {}
+    for key, value in fields.items():
+        if value is None:
+            continue
+        if isinstance(value, str):
+            value = value.strip()
+            if not value:
+                continue
+        filled[key] = value
+    return filled
+
+
+def _prefer_live(rows: list[dict[str, Any]]) -> dict[str, Any] | None:
+    if not rows:
+        return None
+    live = [row for row in rows if not row.get("deleted_at")]
+    return (live or rows)[-1]
+
+
+def _lookup(user_uid: str, provider: str, **identity: Any) -> dict[str, Any] | None:
+    return _prefer_live(
+        list(table().find(user_uid=user_uid, provider=provider, **identity))
+    )
+
+
+def _with_id(record: dict[str, Any]) -> dict[str, Any]:
+    if record.get("id"):
+        return record
+    found = table().find_one(uid=record.get("uid"))
+    return found or record
+
+
+def _retire_duplicates(
+    user_uid: str,
+    provider: str,
+    keep_id: int,
+    fields: dict[str, Any],
+    previous_token: str | None = None,
+) -> None:
+    token = fields.get("token")
+    client_id = fields.get("client_id")
+    for row in table().find(user_uid=user_uid, provider=provider, deleted_at=None):
+        if row["id"] == keep_id:
+            continue
+        if client_id and row.get("client_id") == client_id:
+            mark_dead(row["id"])
+            continue
+        if token and row.get("token") == token:
+            mark_dead(row["id"])
+            continue
+        if previous_token and row.get("token") == previous_token:
+            mark_dead(row["id"])
+
+
+def _merge(
+    existing: dict[str, Any], fields: dict[str, Any]
+) -> RegistrationWrite:
+    revived = bool(existing.get("deleted_at"))
+    patch: dict[str, Any] = {"id": existing["id"]}
+    if revived:
+        patch["deleted_at"] = None
+    changed = revived
+    for key, value in fields.items():
+        if existing.get(key) != value:
+            patch[key] = value
+            changed = True
+    if not changed:
+        return RegistrationWrite(_with_id(existing), False, False)
+    patch["registered_at"] = datetime.now(timezone.utc).isoformat()
+    previous_token = existing.get("token")
+    table().update(patch, ["id"])
+    merged = {**existing, **patch}
+    if revived:
+        merged["deleted_at"] = None
+    merged = _with_id(merged)
+    _retire_duplicates(
+        existing["user_uid"],
+        existing["provider"],
+        merged["id"],
+        fields,
+        previous_token=previous_token if previous_token != fields.get("token") else None,
+    )
+    logger.info(
+        "Updated %s push subscription for user %s%s",
+        existing.get("provider"),
+        existing.get("user_uid"),
+        " (revived)" if revived else "",
+    )
+    return RegistrationWrite(merged, False, revived)
+
+
 def register(
     user_uid: str, provider: str, fields: dict[str, Any]
-) -> tuple[dict[str, Any], bool]:
+) -> RegistrationWrite:
+    fields = _filled(fields)
     registrations = table()
-    existing = registrations.find_one(
+    client_id = fields.get("client_id")
+    token = fields.get("token")
+    endpoint = fields.get("endpoint")
+
+    if client_id:
+        existing = _lookup(user_uid, provider, client_id=client_id)
+        if existing:
+            return _merge(existing, fields)
+
+    if token:
+        existing = _lookup(user_uid, provider, token=token)
+        if existing:
+            return _merge(existing, fields)
+
+    if endpoint:
+        existing = _lookup(user_uid, provider, endpoint=endpoint)
+        if existing:
+            return _merge(existing, fields)
+
+    live = registrations.find_one(
         user_uid=user_uid, provider=provider, deleted_at=None, **fields
     )
-    if existing:
-        logger.debug("Push subscription already registered for user %s", user_uid)
-        return existing, False
+    if live:
+        return RegistrationWrite(_with_id(live), False, False)
 
+    now = datetime.now(timezone.utc).isoformat()
     record = {
         "uid": generate_uid(),
         "user_uid": user_uid,
         "provider": provider,
-        "created_at": datetime.now(timezone.utc).isoformat(),
+        "created_at": now,
+        "registered_at": now,
         "deleted_at": None,
         **fields,
     }
-    registrations.insert(record)
+    inserted = registrations.insert(record)
+    if isinstance(inserted, int):
+        record["id"] = inserted
+    record = _with_id(record)
+    if record.get("id"):
+        _retire_duplicates(user_uid, provider, record["id"], fields)
     logger.info("Registered %s push subscription for user %s", provider, user_uid)
-    return record, True
+    return RegistrationWrite(record, True, False)
 
 
-def mark_dead(registration_id: int) -> None:
+def mark_dead(registration_id: int, dead_before: str | None = None) -> None:
+    if dead_before is not None:
+        row = table().find_one(id=registration_id)
+        registered_at = row.get("registered_at") if row else None
+        if registered_at and registered_at > dead_before:
+            logger.info(
+                "Skipped marking push subscription id=%s dead: registered again at %s "
+                "after the provider confirmed it dead at %s",
+                registration_id,
+                registered_at,
+                dead_before,
+            )
+            return
     table().update(
         {"id": registration_id, "deleted_at": datetime.now(timezone.utc).isoformat()},
         ["id"],

devplacepy/routers/CLAUDE.md

@@ -11,11 +11,12 @@ Prefixes are wired in `main.py`:
 | `/auth` | auth/ package - one leaf per flow (`signup`, `login`, `logout`, `forgotpassword`, `resetpassword`) |
 | `/feed` | feed.py |
 | `/posts` | posts.py |
+| `/topics` | topics.py - crawlable per-topic category index pages (`GET /topics` hub, `GET /topics/{topic}` per-topic post listing over the same `TOPICS` set as the feed sidebar filter). Reuses `feed.py`'s `get_feed_posts`/`enrich_post_cards` so a topic page is a fully-enriched `_post_card.html` listing, not a stripped-down duplicate. Unlike `/feed?topic=X` (whose canonical strips the query string back to bare `/feed`, so it is never indexed as a distinct page), each `/topics/{topic}` page has its own canonical URL, unique title/description, breadcrumbs, and a sitemap entry - see "SEO implementation" below |
 | `/comments` | comments.py |
 | `/projects` | projects/ package - `index.py` (listing/detail/create/delete plus owner-only visibility toggles `POST /{slug}/private` and `POST /{slug}/readonly`, and async `POST /{slug}/fork`), `files.py`, and the `containers/` subpackage (below). `main.py` mounts the whole `/projects` tree from this one package. See `routers/projects/CLAUDE.md` for the deep detail on this tree |
 | `/projects/{slug}/files` | projects/files.py - per-project virtual filesystem (CRUD dirs/files, upload, inline edit, plus line-range ops: `lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append` for large text files) |
 | `/profile` | profile/ package - `index.py` (page/search/followers/update/api-key; the **Posts** tab shows only the 10 most recent posts, while the sidebar `posts_count` stat always uses the real `count()` so it is never capped to the rendered 10), `customization.py`, `notifications.py`, `ai_correction.py` (`POST /{username}/ai-correction`, owner-or-admin AI content correction toggle + prompt), `ai_modifier.py` (`POST /{username}/ai-modifier`, owner-or-admin AI modifier toggle + prompt), `interactions.py` (`POST /{username}/interactions`, owner-or-admin Devii interactive widgets preference; admin default via `devii_interactions_default`), `telegram.py` (`POST /{username}/telegram`, owner-or-admin Telegram pairing-code request/unpair), `usage.py` (`_ai_quota`, re-exported from the package) |
-| `/messages` | messages.py - real-time DM chat. `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `GET /conversations` (flat JSON conversation list, same `get_conversations` helper as the page), `POST /send` (no-JS fallback, also accepts an attachment-only empty-content message), `POST /ws-ticket` (issues a 30s single-use WS auth ticket so a browser WebSocket can authenticate cross-context, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, and read receipts on top of the existing `messages` table - presence is NOT part of this WS (it is the generic `PresenceManager`/pub/sub mechanism, see `devplacepy/services/CLAUDE.md`). Both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention) and then through `_finalize_and_broadcast`, which awaits any pending SYNC AI correction/modifier futures (DM content runs through both, so typing `@ai <instruction>` executes live), broadcasts the FINAL corrected/modified content with an additive `ai_processed` frame flag (the WS path passes `request=websocket` and awaits directly since HTTP middleware does not run for websockets; the frontend swaps the sender's optimistic bubble to the authoritative content). Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, deduped against the hub's delivered set). See `devplacepy/services/messaging/CLAUDE.md` |
+| `/messages` | messages.py - real-time DM chat. `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `GET /conversations` (flat JSON conversation list, same `get_conversations` helper as the page), `POST /send` (no-JS fallback, also accepts an attachment-only empty-content message), `POST /ws-ticket` (issues a 30s single-use WS auth ticket so a browser WebSocket can authenticate cross-context, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, and read receipts on top of the existing `messages` table - presence is NOT part of this WS (it is the generic `PresenceManager`/pub/sub mechanism, see `devplacepy/services/CLAUDE.md`). Both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention) and then through `_finalize_and_broadcast`, which broadcasts the persisted row immediately and then applies SYNC AI correction/modifier (HTTP awaits so the JSON body is final; WS schedules it so the receive loop never blocks). An in-place rewrite stamps `messages.updated_at` and emits a second `ai_processed` frame; other workers pick it up from `message_relay._tick_updates`. Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, plus `updated_at` for revisions, new rows deduped against the hub's delivered set). See `devplacepy/services/messaging/CLAUDE.md` |
 | `/notifications` | notifications.py |
 | `/votes` | votes.py |
 | `/reactions` | reactions.py - emoji reaction toggle on any target: `POST /reactions/{target_type}/{target_uid}` (optimistic `ReactionBar`) |
@@ -48,7 +49,7 @@ Prefixes are wired in `main.py`:
 | `/quizzes` | quizzes/ package - **Quizzes**: `index.py` (hub, create, import, detail, export, edit, delete, publish, per-quiz leaderboard, cross-quiz scoreboard), `questions.py` (builder page + question CRUD + reorder), `attempts.py` (start/resume, play, answer, finish, results). The hub `GET /quizzes` is the approved three-column `.feed-layout` reused verbatim from `/feed`: filters + search left (`?filter=all|todo|done|mine|drafts`, `?search=`, `?page=`), the quiz list centre with the **New quiz** and **Create quiz with Devii** actions, and the cross-quiz **scoreboard** rail right (score per user, best attempt per quiz, 15s display cache). Publishing is terminal - every write on a published quiz is a 400 and there is no unpublish route. See `devplacepy/services/quiz/CLAUDE.md` |
 | `/battles` | battles.py - **Opinion Wars**, week-long two-faction battles attached to posts: `GET ""` (listing, quizzes-style filters active/ended/mine + search), `GET /{uid}` (JSON-only full state, runs lazy resolution), `GET /{uid}/events?after=` (durable event trail replay), `POST /{uid}/join` (join or switch faction), `POST /{uid}/fight` (spend 25 Code Farm coins, deal level-weighted damage, 24h cooldown). The battle card renders on the parent post (no battle detail HTML page). See `devplacepy/services/opinionwar/CLAUDE.md` |
 | `/game` | game/ package - the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard?board=`, `POST /game/{plant,harvest,buy-plot,upgrade,fertilize,daily,grant,perk,prestige,legacy,mastery,quests/claim}`, `POST /game/{defense/upgrade,defense/downgrade,infrastructure/buy,cosmetics/buy,cosmetics/equip}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}`. See `devplacepy/services/game/CLAUDE.md` |
-| (none) | push.py - push + PWA: `GET /push.json` (VAPID public key + the providers accepting registrations), `POST /push.json` (register with any active provider; a body without `provider` is a `webpush` body, so existing clients are unchanged; an unknown, disabled or unconfigured provider is a 400), `GET /service-worker.js`, `GET /manifest.json`. Provider protocol and delivery live in `devplacepy/push/` - see `devplacepy/push/CLAUDE.md` |
+| (none) | push.py - push + PWA: `GET /push.json` (VAPID public key + the providers accepting registrations; `apns` includes `environment` when active), `POST /push.json` (register with any active provider; a body without `provider` is a `webpush` body; an APNs body is `token` plus optional `client_id` for device-stable upsert; created/revived rows are probed and the JSON may include `delivered`/`error`; an unknown, disabled or unconfigured provider is a 400), `GET /service-worker.js`, `GET /manifest.json`. Provider protocol and delivery live in `devplacepy/push/` - see `devplacepy/push/CLAUDE.md` |
 | (none) | docs.py (docs/ package) - the documentation site (prose pages + API reference). See `routers/docs/CLAUDE.md` for the deep detail on this tree (`DOCS_PAGES`, audience tiers, prose rendering pipeline) |
 | `/pubsub` | pubsub.py - **database-free** publish/subscribe bus. `WS /pubsub/ws` (subscribe/unsubscribe/publish frames, `foo.*` wildcards), `POST /pubsub/publish` + `GET /pubsub/topics` (admin/internal). Lock-owner-gated WS (`4013` retry) so subscribers converge on one worker; topic authz in `services/pubsub/policy.py` (`user.{uid}.*` private, `public.*` shared, admin/internal anywhere, guests opt-in). In-process `services.pubsub.publish(topic, data)` for backends; frontend `app.pubsub`. See `devplacepy/services/pubsub/CLAUDE.md` |
 | (none) | seo.py - `/robots.txt`, `/sitemap.xml` |
@@ -250,6 +251,14 @@ All SEO features are implemented across the following locations:
 - `devplacepy/seo.py` - JSON-LD schema generators (WebSite, BreadcrumbList, DiscussionForumPosting, ProfilePage, SoftwareApplication), meta description truncation, schema combiner, sitemap XML generator
 - `routers/seo.py` - robots.txt and sitemap.xml routes
 
+### DiscussionForumPosting nested comments
+
+`discussion_forum_posting(post, author, comment_count, star_count, base_url, comments=None)` embeds up to `MAX_SCHEMA_COMMENTS` (20) of the post's comments as nested `comment: [{"@type": "Comment", "text", "author", "datePublished"}, ...]` entities - not just the aggregate `CommentAction` `InteractionCounter`, which stays for the total count. `seo.comment_schema_list(comment_tree, base_url)` flattens the already-loaded comment tree (`content.load_detail`'s `detail["comments"]`, the same nested `{comment, author, children}` shape `_comment.html` renders) depth-first up to the cap - it does not re-query the database. `posts.py::view_post` is the only call site; a future post-like discussion surface (project/gist/news comments) can reuse `comment_schema_list` the same way once/if it gets a `DiscussionForumPosting` schema of its own.
+
+### Topic category pages (`/topics`)
+
+`routers/topics.py` gives the feed's `TOPICS` filter (`constants.py`) real, independently-crawlable pages instead of only a `?topic=` query param (whose canonical collapses back to bare `/feed` - see `base_seo_context`, `canonical = f"{base}{request.url.path}"`, which drops the query string on purpose). `GET /topics` is a hub linking every topic (with a live post count); `GET /topics/{topic}` is a full post listing for that topic, built from the exact same `get_feed_posts`/`enrich_post_cards` pair `feed.py` uses (`enrich_post_cards` was extracted out of `feed_page` specifically so this page is not a second, drifting copy of the attachments/reactions/bookmarks/poll/war enrichment loop). Each topic page gets its own canonical URL, unique title/description, breadcrumbs (Home > Topics > {label}), a `rel=next` link when paginated (`list_page_seo`/`next_page_url`, same mechanism as `/feed`/`/news`), and a real crawlable `_load_more.html` link (not JS-only infinite scroll) for reaching older posts. Both `/topics` and every `/topics/{topic}` are in `sitemap.xml`.
+
 ### SEO template context
 - Every router passes `page_title`, `meta_description`, `meta_robots`, `canonical_url`, `og_title`, `og_description`, `og_image`, `og_type`, `breadcrumbs`, `page_schema` via `base_seo_context()`
 - Auth pages: `noindex,nofollow`

devplacepy/routers/feed.py

@@ -81,21 +81,7 @@ def get_feed_posts(
     return result, next_cursor
 
 
-@router.get("", response_class=HTMLResponse)
-async def feed_page(
-    request: Request,
-    tab: str = "all",
-    topic: str = None,
-    search: str = "",
-    before: str = None,
-):
-    user = get_current_user(request)
-    posts, next_cursor = get_feed_posts(user, tab, topic, search, before)
-    stats = get_site_stats()
-    top_authors = get_top_authors(5)
-    daily_topic = get_daily_topic()
-    online_users = presence.online_users()
-
+def enrich_post_cards(posts, user):
     post_uids_list = [item["post"]["uid"] for item in posts]
     attachments_map = get_attachments_batch("post", post_uids_list)
     recent_comments = get_recent_comments_by_post_uids(post_uids_list, 3, user)
@@ -113,6 +99,25 @@ async def feed_page(
         item["bookmarked"] = uid in bookmark_set
         item["poll"] = polls_map.get(uid)
         item["war"] = wars_map.get(uid)
+    return posts
+
+
+@router.get("", response_class=HTMLResponse)
+async def feed_page(
+    request: Request,
+    tab: str = "all",
+    topic: str = None,
+    search: str = "",
+    before: str = None,
+):
+    user = get_current_user(request)
+    posts, next_cursor = get_feed_posts(user, tab, topic, search, before)
+    stats = get_site_stats()
+    top_authors = get_top_authors(5)
+    daily_topic = get_daily_topic()
+    online_users = presence.online_users()
+
+    posts = enrich_post_cards(posts, user)
 
     seo_ctx = list_page_seo(
         request,

devplacepy/routers/messages.py

@@ -20,7 +20,6 @@ from devplacepy.templating import clear_messages_cache
 from devplacepy.utils import (
     require_user,
     time_ago,
-    is_admin,
     _user_from_session,
     _user_from_api_key,
 )
@@ -31,6 +30,7 @@ from devplacepy.services import presence
 from devplacepy.services.audit import record as audit
 from devplacepy.services.correction import PENDING_SCOPE_KEY
 from devplacepy.dependencies import json_or_form
+from devplacepy.services.moderation.screening import ContentRefused
 from devplacepy.services.messaging import (
     issue_ticket,
     message_frame,
@@ -38,6 +38,7 @@ from devplacepy.services.messaging import (
     message_relay,
     persist_message,
     redeem_ticket,
+    stamp_content_revision,
 )
 
 logger = logging.getLogger(__name__)
@@ -113,22 +114,38 @@ def get_conversations(user_uid: str):
         conv.pop("other_uid", None)
     return conversations
 
-def get_conversation_messages(user_uid: str, other_uid: str):
+def get_conversation_messages(user_uid: str, other_uid: str, before: str = ""):
     if other_uid in get_blocked_uids(user_uid):
         return [], None
     if "messages" not in db.tables:
         return [], get_users_by_uids([other_uid]).get(other_uid)
-    msgs = list(
-        db.query(
-            "SELECT * FROM messages"
-            " WHERE (sender_uid = :me AND receiver_uid = :other)"
-            "    OR (sender_uid = :other AND receiver_uid = :me)"
-            " ORDER BY created_at DESC, id DESC LIMIT :lim",
-            me=user_uid,
-            other=other_uid,
-            lim=CONVERSATION_MESSAGE_LIMIT,
+    before = (before or "").strip()
+    if before:
+        msgs = list(
+            db.query(
+                "SELECT * FROM messages"
+                " WHERE ((sender_uid = :me AND receiver_uid = :other)"
+                "    OR (sender_uid = :other AND receiver_uid = :me))"
+                "   AND created_at < :before"
+                " ORDER BY created_at DESC, id DESC LIMIT :lim",
+                me=user_uid,
+                other=other_uid,
+                before=before,
+                lim=CONVERSATION_MESSAGE_LIMIT,
+            )
+        )
+    else:
+        msgs = list(
+            db.query(
+                "SELECT * FROM messages"
+                " WHERE (sender_uid = :me AND receiver_uid = :other)"
+                "    OR (sender_uid = :other AND receiver_uid = :me)"
+                " ORDER BY created_at DESC, id DESC LIMIT :lim",
+                me=user_uid,
+                other=other_uid,
+                lim=CONVERSATION_MESSAGE_LIMIT,
+            )
         )
-    )
     msgs.reverse()
 
     user_ids = list({m["sender_uid"] for m in msgs} | {other_uid})
@@ -158,7 +175,9 @@ def get_conversation_messages(user_uid: str, other_uid: str):
     return result, other_user
 
 @router.get("", response_class=HTMLResponse)
-async def messages_page(request: Request, with_uid: str = None, search: str = ""):
+async def messages_page(
+    request: Request, with_uid: str = None, search: str = "", before: str = ""
+):
     user = require_user(request)
     conversations = get_conversations(user["uid"])
 
@@ -174,25 +193,28 @@ async def messages_page(request: Request, with_uid: str = None, search: str = ""
     other_online = False
     other_last_seen = None
     if with_uid:
-        messages, other_user = get_conversation_messages(user["uid"], with_uid)
-        mark_conversation_read(user["uid"], with_uid)
-        mark_notifications_read_by_target(
-            user["uid"], f"/messages?with_uid={with_uid}"
+        messages, other_user = get_conversation_messages(
+            user["uid"], with_uid, before=before
         )
         current_conversation = with_uid
         other_online = presence.is_online(other_user)
         other_last_seen = other_user.get("last_seen") if other_user else None
-        audit.record(
-            request,
-            "message.read_on_view",
-            user=user,
-            target_type="user",
-            target_uid=with_uid,
-            target_label=other_user.get("username") if other_user else None,
-            metadata={"message_count": len(messages)},
-            summary=f"{user['username']} read messages from {other_user.get('username') if other_user else with_uid}",
-            links=[audit.target("user", with_uid, other_user.get("username") if other_user else None)],
-        )
+        if not before:
+            mark_conversation_read(user["uid"], with_uid)
+            mark_notifications_read_by_target(
+                user["uid"], f"/messages?with_uid={with_uid}"
+            )
+            audit.record(
+                request,
+                "message.read_on_view",
+                user=user,
+                target_type="user",
+                target_uid=with_uid,
+                target_label=other_user.get("username") if other_user else None,
+                metadata={"message_count": len(messages)},
+                summary=f"{user['username']} read messages from {other_user.get('username') if other_user else with_uid}",
+                links=[audit.target("user", with_uid, other_user.get("username") if other_user else None)],
+            )
 
     seo_ctx = base_seo_context(
         request,
@@ -259,7 +281,7 @@ async def send_message(request: Request, data: Annotated[MessageForm, Depends(js
         return action_result(request, "/messages")
 
     ai_processed = await _finalize_and_broadcast(
-        user, message, request, client_id=data.client_id
+        user, message, request, client_id=data.client_id, wait_ai=True
     )
     frame = message_frame(
         message, user.get("username", ""), data.client_id,
@@ -271,31 +293,107 @@ async def send_message(request: Request, data: Annotated[MessageForm, Depends(js
 
 async def broadcast_message(
     sender: dict, message: dict, client_id: Optional[str] = None,
-    ai_processed: bool = False,
+    ai_processed: bool = False, ai_pending: bool = False,
 ) -> None:
     frame = message_frame(
         message, sender.get("username", ""), client_id,
         sender_role=sender.get("role"), ai_processed=ai_processed,
     )
-    message_hub.mark_delivered(message["uid"])
+    frame["ai_pending"] = ai_pending
+    if not ai_processed:
+        message_hub.mark_delivered(message["uid"])
     targets = [message["sender_uid"], message["receiver_uid"]]
     await message_hub.send_to_users(targets, frame)
 
+async def _await_ai_and_push(
+    sender: dict, message: dict, pending: list, client_id: Optional[str]
+) -> bool:
+    await asyncio.gather(*pending, return_exceptions=True)
+    pending.clear()
+    row = get_table("messages").find_one(uid=message["uid"])
+    if not row:
+        return False
+    changed = row.get("content") != message.get("content")
+    if changed:
+        message["content"] = row["content"]
+        stamp_content_revision(message["uid"])
+    await broadcast_message(sender, message, client_id, ai_processed=True)
+    return changed
+
 async def _finalize_and_broadcast(
-    sender: dict, message: dict, request: object, client_id: Optional[str] = None
+    sender: dict,
+    message: dict,
+    request: object,
+    client_id: Optional[str] = None,
+    wait_ai: bool = True,
 ) -> bool:
-    message_hub.mark_delivered(message["uid"])
     scope = getattr(request, "scope", None)
     pending = scope.get(PENDING_SCOPE_KEY) if scope is not None else None
-    ai_processed = bool(pending)
-    if pending:
-        await asyncio.gather(*pending, return_exceptions=True)
-        pending.clear()
-        row = get_table("messages").find_one(uid=message["uid"])
-        if row:
-            message["content"] = row["content"]
-    await broadcast_message(sender, message, client_id, ai_processed=ai_processed)
-    return ai_processed
+    has_pending = bool(pending)
+    await broadcast_message(
+        sender, message, client_id, ai_processed=False, ai_pending=has_pending
+    )
+    if not pending:
+        return False
+    if wait_ai:
+        return await _await_ai_and_push(sender, message, pending, client_id)
+    asyncio.create_task(_await_ai_and_push(sender, message, pending, client_id))
+    return False
+
+SYNC_LIMIT = 200
+
+
+async def _sync_missed(user_uid: str, data: dict, websocket: WebSocket) -> None:
+    since = str(data.get("since") or "").strip()
+    with_uid = str(data.get("with_uid") or "").strip()
+    if not since or "messages" not in db.tables:
+        return
+    if with_uid:
+        rows = list(
+            db.query(
+                "SELECT * FROM messages"
+                " WHERE ((sender_uid = :me AND receiver_uid = :other)"
+                "    OR (sender_uid = :other AND receiver_uid = :me))"
+                "   AND (created_at > :since"
+                "        OR (updated_at IS NOT NULL AND updated_at > :since))"
+                " ORDER BY created_at ASC, id ASC LIMIT :lim",
+                me=user_uid,
+                other=with_uid,
+                since=since,
+                lim=SYNC_LIMIT,
+            )
+        )
+    else:
+        rows = list(
+            db.query(
+                "SELECT * FROM messages"
+                " WHERE (sender_uid = :me OR receiver_uid = :me)"
+                "   AND (created_at > :since"
+                "        OR (updated_at IS NOT NULL AND updated_at > :since))"
+                " ORDER BY created_at ASC, id ASC LIMIT :lim",
+                me=user_uid,
+                since=since,
+                lim=SYNC_LIMIT,
+            )
+        )
+    if not rows:
+        return
+    sender_uids = {row["sender_uid"] for row in rows}
+    senders = get_users_by_uids(list(sender_uids)) if sender_uids else {}
+    for row in rows:
+        sender = senders.get(row["sender_uid"]) or {}
+        frame = message_frame(
+            dict(row),
+            sender.get("username", ""),
+            sender_role=sender.get("role"),
+            ai_processed=bool(row.get("updated_at")),
+        )
+        try:
+            await websocket.send_json(frame)
+        except Exception:  # noqa: BLE001
+            logger.debug("sync frame dropped for %s", user_uid)
+            return
+
 
 def _resolve_ws_user(websocket: WebSocket):
     user = _user_from_session(websocket)
@@ -351,20 +449,34 @@ async def messages_ws(websocket: WebSocket):
                 attachment_uids = [str(a) for a in raw_attachments][:MAX_WS_ATTACHMENTS]
                 if not receiver_uid:
                     continue
-                message = persist_message(
-                    user,
-                    receiver_uid,
-                    content,
-                    attachment_uids,
-                    request=websocket,
-                    origin="websocket",
-                )
+                try:
+                    message = persist_message(
+                        user,
+                        receiver_uid,
+                        content,
+                        attachment_uids,
+                        request=websocket,
+                        origin="websocket",
+                    )
+                except ContentRefused as exc:
+                    await websocket.send_json(
+                        {
+                            "type": "error",
+                            "client_id": client_id,
+                            "text": exc.message,
+                        }
+                    )
+                    continue
                 if message is None:
                     await websocket.send_json(
                         {"type": "error", "client_id": client_id, "text": "Message not sent."}
                     )
                     continue
-                await _finalize_and_broadcast(user, message, websocket, client_id)
+                await _finalize_and_broadcast(
+                    user, message, websocket, client_id, wait_ai=False
+                )
+            elif kind == "sync":
+                await _sync_missed(user_uid, data, websocket)
             elif kind == "typing":
                 receiver_uid = str(data.get("receiver_uid", "")).strip()
                 if receiver_uid:

devplacepy/routers/posts.py

@@ -37,6 +37,7 @@ from devplacepy.seo import (
     site_url,
     website_schema,
     discussion_forum_posting,
+    comment_schema_list,
 )
 from devplacepy.attachments import save_inline_image
 from devplacepy.models import PostForm, PostEditForm
@@ -179,7 +180,12 @@ async def view_post(request: Request, post_slug: str):
         schemas=[
             website_schema(base),
             discussion_forum_posting(
-                post, author, comment_count, detail["star_count"], base
+                post,
+                author,
+                comment_count,
+                detail["star_count"],
+                base,
+                comments=comment_schema_list(top_level, base),
             ),
         ],
     )

devplacepy/routers/push.py

@@ -49,17 +49,24 @@ async def push_register(request: Request) -> JSONResponse:
     if fields is None:
         return JSONResponse({"error": "Invalid request"}, status_code=400)
 
-    _, created = push.register(user["uid"], provider.name, fields)
+    fields = provider.stamp_registration(fields)
+    write = push.register(user["uid"], provider.name, fields)
 
-    if created:
+    delivered = None
+    detail = ""
+    if write.probe:
         try:
-            await push.notify_user(user["uid"], WELCOME_PAYLOAD)
+            outcome = await push.notify_registration(write.record, WELCOME_PAYLOAD)
+            delivered = outcome.status == providers.ACCEPTED
+            detail = outcome.detail
         except Exception as exc:
             logger.warning("Welcome push failed for %s: %s", user["uid"], exc)
+            delivered = False
+            detail = str(exc)
 
     audit.record(
         request,
-        "push.subscribe" if created else "push.update",
+        "push.subscribe" if write.created else "push.update",
         user=user,
         target_type="user",
         target_uid=user["uid"],
@@ -69,12 +76,19 @@ async def push_register(request: Request) -> JSONResponse:
             "endpoint_host": urlparse(fields["endpoint"]).hostname
             if fields.get("endpoint")
             else None,
-            "created": created,
+            "created": write.created,
+            "revived": write.revived,
+            "has_client_id": bool(fields.get("client_id")),
         },
-        summary=f"{user.get('username')} {'registered' if created else 'updated'} a push subscription",
+        summary=f"{user.get('username')} {'registered' if write.created else 'updated'} a push subscription",
         links=[audit.target("user", user["uid"], user.get("username"))],
     )
-    return JSONResponse({"registered": True})
+    payload: dict = {"registered": True}
+    if delivered is not None:
+        payload["delivered"] = delivered
+        if detail and not delivered:
+            payload["error"] = detail
+    return JSONResponse(payload)
 
 
 @router.get("/service-worker.js")

devplacepy/routers/tools/deepsearch.py

@@ -243,6 +243,7 @@ def _session_context(request: Request, uid: str, job: dict, session: dict) -> di
         "sources": report.get("sources", []),
         "findings": report.get("findings", []),
         "timeline": report.get("timeline", []),
+        "follow_up_questions": report.get("follow_up_questions", []),
         "chat_ws_url": f"/tools/deepsearch/{uid}/chat" if done else None,
         "export_md_url": f"/tools/deepsearch/{uid}/export.md" if done else None,
         "export_json_url": f"/tools/deepsearch/{uid}/export.json" if done else None,

devplacepy/routers/topics.py

@@ -0,0 +1,84 @@
+# retoor <retoor@molodetz.nl>
+
+from fastapi import APIRouter, Request
+from fastapi.responses import HTMLResponse
+from devplacepy.constants import TOPICS, TOPIC_LABELS
+from devplacepy.database import get_table
+from devplacepy.routers.feed import get_feed_posts, enrich_post_cards
+from devplacepy.utils import get_current_user, not_found
+from devplacepy.seo import list_page_seo, next_page_url
+from devplacepy.responses import respond
+from devplacepy.schemas import TopicOut, TopicsHubOut
+
+router = APIRouter()
+
+
+@router.get("", response_class=HTMLResponse)
+async def topics_hub(request: Request):
+    user = get_current_user(request)
+    posts_table = get_table("posts")
+    topics = [
+        {
+            "key": topic,
+            "label": TOPIC_LABELS.get(topic, topic.title()),
+            "post_count": posts_table.count(topic=topic, deleted_at=None),
+        }
+        for topic in TOPICS
+    ]
+
+    seo_ctx = list_page_seo(
+        request,
+        title="Topics",
+        description="Browse DevPlace posts by topic: devlog, showcase, questions, rants, fun, and more.",
+        breadcrumbs=[
+            {"name": "Home", "url": "/feed"},
+            {"name": "Topics", "url": "/topics"},
+        ],
+    )
+    return respond(
+        request,
+        "topics.html",
+        {
+            **seo_ctx,
+            "request": request,
+            "user": user,
+            "topics": topics,
+        },
+        model=TopicsHubOut,
+    )
+
+
+@router.get("/{topic}", response_class=HTMLResponse)
+async def topic_page(request: Request, topic: str, before: str = None):
+    if topic not in TOPICS:
+        raise not_found("Topic not found")
+    user = get_current_user(request)
+    posts, next_cursor = get_feed_posts(user, "all", topic, "", before)
+    posts = enrich_post_cards(posts, user)
+    label = TOPIC_LABELS.get(topic, topic.title())
+
+    seo_ctx = list_page_seo(
+        request,
+        title=f"{label} posts",
+        description=f"Browse {label.lower()} posts from developers on DevPlace.",
+        breadcrumbs=[
+            {"name": "Home", "url": "/feed"},
+            {"name": "Topics", "url": "/topics"},
+            {"name": label, "url": f"/topics/{topic}"},
+        ],
+        next_url=next_page_url(request, next_cursor),
+    )
+    return respond(
+        request,
+        "topic.html",
+        {
+            **seo_ctx,
+            "request": request,
+            "user": user,
+            "posts": posts,
+            "topic": topic,
+            "topic_label": label,
+            "next_cursor": next_cursor,
+        },
+        model=TopicOut,
+    )

devplacepy/schemas/__init__.py

@@ -52,6 +52,9 @@ from devplacepy.schemas.listings import (
     ProjectsOut,
     SavedItemOut,
     SavedOut,
+    TopicOut,
+    TopicSummaryOut,
+    TopicsHubOut,
 )
 from devplacepy.schemas.profile import (
     MediaItemOut,

devplacepy/schemas/game.py

@@ -13,8 +13,11 @@ class GameCropOut(_Out):
     reward_coins: int = 0
     reward_xp: int = 0
     min_level: int = 1
+    min_mastery: int = 0
     grow_seconds: int = 0
     locked: bool = False
+    locked_reason: str = ""
+    locked_text: str = ""
     market_state: str = "normal"
 
 

devplacepy/schemas/jobs.py

@@ -132,6 +132,7 @@ class DeepsearchSessionOut(_Out):
     sources: list = []
     findings: list = []
     timeline: list = []
+    follow_up_questions: list = []
     chat_ws_url: Optional[str] = None
     export_md_url: Optional[str] = None
     export_json_url: Optional[str] = None

devplacepy/schemas/listings.py

@@ -108,6 +108,23 @@ class SavedItemOut(_Out):
     time_ago: Optional[str] = None
 
 
+class TopicOut(_Out):
+    posts: list[FeedItemOut] = []
+    topic: str = ""
+    topic_label: str = ""
+    next_cursor: Optional[str] = None
+
+
+class TopicSummaryOut(_Out):
+    key: str = ""
+    label: str = ""
+    post_count: int = 0
+
+
+class TopicsHubOut(_Out):
+    topics: list[TopicSummaryOut] = []
+
+
 class FeedOut(_Out):
     posts: list[FeedItemOut] = []
     current_tab: Optional[str] = None

devplacepy/seo.py

@@ -91,7 +91,41 @@ def breadcrumb_schema(items, base_url):
     }
 
 
-def discussion_forum_posting(post, author, comment_count, star_count, base_url):
+MAX_SCHEMA_COMMENTS = 20
+
+
+def comment_schema(comment_item, base_url):
+    comment = comment_item["comment"]
+    author = comment_item.get("author")
+    return {
+        "@type": "Comment",
+        "text": truncate(plain_markdown(comment.get("content", "")), 300),
+        "author": {
+            "@type": "Person",
+            "name": author["username"] if author else "Unknown",
+            "url": f"{base_url}/profile/{author['username']}" if author else "",
+        },
+        "datePublished": comment.get("created_at", ""),
+    }
+
+
+def comment_schema_list(comment_tree, base_url, limit=MAX_SCHEMA_COMMENTS):
+    flat = []
+
+    def walk(items):
+        for item in items:
+            if len(flat) >= limit:
+                return
+            flat.append(comment_schema(item, base_url))
+            walk(item.get("children", []))
+
+    walk(comment_tree)
+    return flat
+
+
+def discussion_forum_posting(
+    post, author, comment_count, star_count, base_url, comments=None
+):
     schema = {
         "@type": "DiscussionForumPosting",
         "headline": post.get("title") or "Untitled",
@@ -117,6 +151,8 @@ def discussion_forum_posting(post, author, comment_count, star_count, base_url):
             },
         ],
     }
+    if comments:
+        schema["comment"] = comments
     return schema
 
 
@@ -421,6 +457,13 @@ def _build_sitemap(base_url):
     urlset.append(url_element(f"{base_url}/", changefreq="daily", priority="1.0"))
     urlset.append(url_element(f"{base_url}/feed", changefreq="hourly", priority="0.9"))
     urlset.append(url_element(f"{base_url}/news", changefreq="hourly", priority="0.9"))
+    urlset.append(url_element(f"{base_url}/topics", changefreq="daily", priority="0.7"))
+    from devplacepy.constants import TOPICS
+
+    for topic in TOPICS:
+        urlset.append(
+            url_element(f"{base_url}/topics/{topic}", changefreq="daily", priority="0.7")
+        )
     urlset.append(
         url_element(f"{base_url}/projects", changefreq="daily", priority="0.8")
     )

devplacepy/services/ai_modifier.py

@@ -95,6 +95,10 @@ def _run_modification(
     if updates:
         updates["uid"] = uid
         get_table(table).update(updates, ["uid"])
+        if table == "messages":
+            from devplacepy.services.messaging.persist import push_content_revision
+
+            push_content_revision(uid, ai_processed=True)
         if table == "users" and user_uid:
             from devplacepy.utils import clear_user_cache
 

devplacepy/services/correction.py

@@ -202,6 +202,10 @@ def _run_correction(
     if updates:
         updates["uid"] = uid
         get_table(table).update(updates, ["uid"])
+        if table == "messages":
+            from devplacepy.services.messaging.persist import push_content_revision
+
+            push_content_revision(uid, ai_processed=True)
         if table == "users" and user_uid:
             from devplacepy.utils import clear_user_cache
 

devplacepy/services/deepsearch/export.py

@@ -70,6 +70,13 @@ def to_markdown(report: dict) -> str:
             url = source.get("url", "")
             lines.append(f"{index}. [{title}]({url})")
         lines.append("")
+    follow_ups = report.get("follow_up_questions") or []
+    if follow_ups:
+        lines.append("## Ask next")
+        lines.append("")
+        for question in follow_ups:
+            lines.append(f"- {question}")
+        lines.append("")
     return "\n".join(lines)
 
 
@@ -113,6 +120,12 @@ def _html_document(report: dict) -> str:
             title = html.escape(source.get("title") or source.get("url", ""))
             parts.append(f"<li><a href='{url}'>{title}</a></li>")
         parts.append("</ol>")
+    follow_ups = report.get("follow_up_questions") or []
+    if follow_ups:
+        parts.append("<h2>Ask next</h2><ul>")
+        for question in follow_ups:
+            parts.append(f"<li>{html.escape(question)}</li>")
+        parts.append("</ul>")
     parts.append("</body></html>")
     return "".join(parts)
 

devplacepy/services/deepsearch/store.py

@@ -34,6 +34,7 @@ class Chunk:
     position: int = 0
     score: float = 0.0
     metadata: dict = field(default_factory=dict)
+    embedding: list[float] | None = None
 
 
 def _tokenize(text: str) -> list[str]:
@@ -176,16 +177,18 @@ class VectorStore:
             query_embeddings=[query_vector],
             n_results=top_k,
             where=where or None,
-            include=["documents", "metadatas", "distances"],
+            include=["documents", "metadatas", "distances", "embeddings"],
         )
         ids = (result.get("ids") or [[]])[0]
         documents = (result.get("documents") or [[]])[0]
         metadatas = (result.get("metadatas") or [[]])[0]
         distances = (result.get("distances") or [[]])[0]
+        embeddings = (result.get("embeddings") or [[]])[0]
         chunks: list[Chunk] = []
         for index, uid in enumerate(ids):
             meta = metadatas[index] if index < len(metadatas) else {}
             distance = distances[index] if index < len(distances) else 1.0
+            vector = embeddings[index] if index < len(embeddings) else None
             chunks.append(
                 Chunk(
                     uid=uid,
@@ -197,6 +200,7 @@ class VectorStore:
                     position=int(meta.get("position", 0) or 0),
                     score=1.0 - float(distance),
                     metadata=dict(meta),
+                    embedding=[float(v) for v in vector] if vector is not None else None,
                 )
             )
         return chunks

devplacepy/services/devii/actions/catalog/posts.py

@@ -20,6 +20,25 @@ POSTS_ACTIONS: tuple[Action, ...] = (
             query("before", "Pagination cursor."),
         ),
     ),
+    Action(
+        name="view_topics",
+        method="GET",
+        path="/topics",
+        summary="View the topics hub - every post topic with its live post count",
+        requires_auth=False,
+        params=(),
+    ),
+    Action(
+        name="view_topic",
+        method="GET",
+        path="/topics/{topic}",
+        summary="View one topic's post listing (e.g. devlog, showcase, question, rant, fun, random, politics)",
+        requires_auth=False,
+        params=(
+            path("topic", "Topic key, one of the values TOPICS lists."),
+            query("before", "Pagination cursor."),
+        ),
+    ),
     Action(
         name="create_post",
         method="POST",

devplacepy/services/devrant/notifications.py

@@ -8,6 +8,7 @@ from devplacepy.services.devrant.ids import to_unix, now_unix
 DEVRANT_TYPE_MAP = {
     "comment": "comment_discuss",
     "reply": "comment_discuss",
+    "thread": "comment_discuss",
     "mention": "comment_mention",
     "vote": "rant_vote",
     "follow": "rant_sub",

devplacepy/services/game/economy.py

@@ -213,6 +213,18 @@ def unlocked_crops(
     ]
 
 
+def crop_lock_reason(
+    crop: Crop, level: int, mastery_earned: int, era_locked: bool = False
+) -> tuple[str, str] | None:
+    if crop.min_level > level:
+        return "level", f"unlocks at level {crop.min_level}"
+    if crop.min_mastery > mastery_earned:
+        return "mastery", "unlocks after reaching Mastery (Refactor to prestige 50)"
+    if era_locked:
+        return "era", "is not available right now"
+    return None
+
+
 def crop_payload(
     crop: Crop,
     ci_tier: int,
@@ -234,6 +246,7 @@ def crop_payload(
         market_state = "saturated"
     elif market_factor > 1.0:
         market_state = "boosted"
+    lock = crop_lock_reason(crop, level, mastery_earned, era_locked)
     return {
         "key": crop.key,
         "name": crop.name,
@@ -244,10 +257,13 @@ def crop_payload(
         ),
         "reward_xp": crop.reward_xp,
         "min_level": crop.min_level,
+        "min_mastery": crop.min_mastery,
         "grow_seconds": grow_seconds_for(
             crop, ci_tier, growth_level, legacy_speed_level, registry_boost
         ),
-        "locked": crop.min_level > level or crop.min_mastery > mastery_earned or era_locked,
+        "locked": lock is not None,
+        "locked_reason": lock[0] if lock else "",
+        "locked_text": f"{crop.name} {lock[1]}." if lock else "",
         "market_state": market_state,
     }
 

devplacepy/services/game/store/actions.py

@@ -44,12 +44,12 @@ def plant(user: dict, slot: int, crop_key: str) -> dict:
     if not crop:
         raise GameError("Unknown crop type.")
     progress = economy.level_progress(int(farm.get("xp", 0)))
-    if crop.min_level > progress["level"]:
-        raise GameError(f"{crop.name} unlocks at level {crop.min_level}.")
-    if crop.min_mastery > _lvl(farm, "mastery_points_earned_total"):
-        raise GameError(f"{crop.name} unlocks after reaching Mastery.")
-    if crop.era_key and crop.era_key != active_era_name():
-        raise GameError(f"{crop.name} is not available right now.")
+    era_locked = bool(crop.era_key and crop.era_key != active_era_name())
+    lock = economy.crop_lock_reason(
+        crop, progress["level"], _lvl(farm, "mastery_points_earned_total"), era_locked
+    )
+    if lock:
+        raise GameError(f"{crop.name} {lock[1]}.")
     plot = _plot_at(farm["uid"], slot)
     if not plot:
         raise GameError("That plot does not exist.")

devplacepy/services/jobs/CLAUDE.md

@@ -98,6 +98,9 @@ The public **Tools -> DeepSearch** researcher is a multi-agent deep web research
 - **Frontend** (do not hand-roll): `DeepsearchTool.js` (`app.deepsearchTool`) drives the form via `Http.send`, watches `DeepsearchProgressSocket` (cloned from `SeoProgressSocket`, 4013 retry), and wires pause/resume/cancel. `static/css/deepsearch.css` uses the design tokens and is mobile-responsive.
 - **Progress frame protocol (append-only, the worker<->JS contract):** `phases.py` is the single source of truth for phase identity, shared by `worker.py` (emit) and `DeepsearchTool.js` (render). `PHASE_ORDER = [planning, searching, crawling, indexing, analysis, synthesis]`; `worker._stage(stage, message, phase)` emits BOTH the legacy `stage` frame (byte-identical to before) AND a parallel `phase` frame `{phase, index, total, label}` so the timeline strip advances. The first emitted frame carries `version:1`. Every other frame type and its keys: `substep` (planning angles, `phase`+`message`; also emitted by analysis grounding), `queries`, `candidates`, `rsearch`, `progress` (`done`/`total`/`url`/`depth`), `page_loaded` (now `source`/`render`/`depth`/`elapsed_ms`/`done`/`total`), `page_cached`/`page_skipped`/`page_duplicate` (now `reason`/`elapsed_ms`), `embed_batch` (`batch`/`total_batches`/`backend`/`done`/`total`, emitted before AND after each batch), `embed_done` (`backend`/`chunk_count`), `agent` (`agent` one of summarizer|extractor|linker, `stage`/`status` start|done|failed, with `elapsed_ms`/`tokens_in`/`tokens_out` on done), `report_ready` (now also `synthesis`), `done` (`session_url`), `failed` (`message`). **The contract is append-only: never rename or drop a frame type**; `service._run_worker` pumps every stdout line into the `ProgressHub` untouched, so new frame types reach the WS with no handler change. `tests/api/tools/deepsearch/index.py` is the append-only regression guard.
 - **RAG-audit hardening (engine correctness, no route/schema change):** (1) **Embedding-dimension consistency** - gateway embeddings carry provider-native dims while `local_embed` is fixed 256-dim; the worker `_index_chunks` now decides the backend ONCE per job (the first gateway failure or non-gateway result forces local for ALL remaining batches), and `VectorStore.add` drops any vector whose length differs from the collection's established dim, so one collection never mixes dims (cosine search across mixed dims is corrupt). `embeddings.EmbedResult.dims` and `VectorStore.dims` (lazily probed from the collection) expose the dimension; `chat.retrieve` re-embeds the query locally and skips retrieval if it still cannot match the stored dim. (2) **Citation grounding** - `orchestrate` drops any finding with no citations, and the summarizer prompt forbids uncited claims and treats the QUESTION as data not an instruction (prompt-injection reduction via `_sanitize_question`); `chat._strip_unmatched_markers` removes any inline `[n]` marker that does not map to an emitted citation. (3) **Confidence calibration** - when only a single domain was crawled, `orchestrate` caps confidence by the source-diversity-derived ceiling (overconfidence guard). (4) `EmbeddingCache` is bounded at `EMBED_CACHE_MAX` to cap in-memory growth.
+- **Iterative gap-filling refinement rounds (`worker.py`, `enhance.plan_followup_queries`).** A single plan-search-crawl pass under-covers a broad question the way a real deep-research agent never would - real systems iterate (search, read, decide what is still missing, search again). After the initial crawl, `worker._run` loops up to `MAX_REFINE_ROUNDS = 2` times, each iteration asking `plan_followup_queries(query, covered_titles, api_key, emit)` (a small planner call mirroring `enhance.plan_queries`, given only the TITLES already gathered) for 0-3 extra queries that would surface uncovered information; an empty reply (planner unavailable, or the sources already look sufficient) stops the loop immediately - it never forces an extra round. Each round is bounded by the remaining `max_pages` budget, follows-links-off (`depth=1`, cheaper than the initial crawl's link-following), and shares `outcome.seen_hashes` with the initial crawl via `crawl()`'s new `seen_hashes` param so content dedup spans rounds; new pages are appended to the same `outcome.pages` list before indexing, so indexing/synthesis are unchanged. **This is an internal control-flow signal only - it produces MORE QUERIES, never a user-visible "gaps" report or critic verdict, so it does not reintroduce the removed critic agent** (see below). Frames reuse the existing `substep`/`queries`/`candidates` types (a `round` key is additive).
+- **Diversity-aware retrieval via MMR (`orchestrate._mmr_select`, `store.py`).** `VectorStore._vector_search_sync` now includes `"embeddings"` in the Chroma query and carries each result's vector on `Chunk.embedding`. `orchestrate._retrieve_chunks` pools every per-sub-query hit (deduped by uid, capped `MMR_POOL_MAX = 60`) instead of round-robin-truncating it directly, then `_mmr_select` (Maximal Marginal Relevance, `MMR_LAMBDA = 0.7`) greedily picks the final `CONTEXT_CHUNKS_MAX` chunks trading relevance to the question against redundancy with what is already selected - this is the standard fix for the case where many sub-queries all surface the same page's top passage, which used to crowd out real topic diversity in the LLM's context window. Falls back to the original order when any candidate lacks an embedding (old data, embedding failure).
+- **Suggested follow-up questions ("Ask next"), Perplexity-style.** After a successful agents-path synthesis, `orchestrate._suggest_followups(question, summary, api_key)` asks one small, separate completion for 3-4 natural next questions a reader would ask, stored on `Orchestration.follow_up_questions` / `report["follow_up_questions"]` / `DeepsearchSessionOut.follow_up_questions`. Fails soft to `[]` (no heuristic fallback needed - it is a nice-to-have, not part of answering the question) and is never generated on the heuristic path. Rendered as clickable chips in `deepsearch_session.html` (`.ds-followup-chip`, shown only when the chat is available) that call the new `AppDeepsearchChat.ask(text)` method, feeding the question straight into the grounded RAG chat. Included in the markdown/HTML/PDF exports under "Ask next". **This is a different feature from the removed critic/gaps agent**: it never critiques or lists what the report is missing, it only suggests what to ask next, exactly like Perplexity's related-questions row.
 - Devii tools `deepsearch`/`deepsearch_status`/`deepsearch_session` (public); docs `tools-deepsearch`; CLI `devplace deepsearch prune|clear`; audit `deepsearch.run.request|complete|failed` + `deepsearch.chat` (category `tools`). **New dependencies:** `chromadb`, `weasyprint`, `pypdf` (all unpinned). New runtime dirs `config.DEEPSEARCH_DIR`/`DEEPSEARCH_CHROMA_DIR` are registered in `DATA_PATHS`. **Add a dedicated nginx WS `location` for `/tools/deepsearch/{uid}/ws` and `/chat`** above `location /` for production, like the SEO and Devii sockets.
 
 ## AI Usage Analyzer tool - IsslopService (kind `isslop`, `services/jobs/isslop/`, `routers/tools/isslop.py`)

devplacepy/services/jobs/deepsearch/crawl.py

@@ -292,10 +292,11 @@ async def crawl(
     should_stop: Callable[[], Awaitable[bool]],
     query: str = "",
     depth: int = 1,
+    seen_hashes: set[str] | None = None,
 ) -> CrawlOutcome:
     from playwright.async_api import async_playwright
 
-    outcome = CrawlOutcome()
+    outcome = CrawlOutcome(seen_hashes=seen_hashes if seen_hashes is not None else set())
     fetched = 0
     seen_urls = {candidate["url"] for candidate in candidates}
     level_candidates = list(candidates)

devplacepy/services/jobs/deepsearch/enhance.py

@@ -10,19 +10,29 @@ from typing import Callable
 from devplacepy import stealth
 from devplacepy.config import INTERNAL_GATEWAY_URL, INTERNAL_MODEL
 
-from .phases import PHASE_PLANNING
+from .phases import PHASE_PLANNING, PHASE_SEARCHING
 
 logger = logging.getLogger(__name__)
 
 ENHANCE_TIMEOUT_SECONDS = 90.0
 ENHANCE_MAX_TOKENS = 400
 MAX_SUBQUERIES = 6
+MAX_FOLLOWUP_QUERIES = 3
+MAX_COVERED_TITLES = 40
 
 PLANNER_PROMPT = (
     "You plan web research. Given a research question, produce a JSON object with one "
     "key 'queries': an array of 3 to 6 concise, diverse web search queries that together "
     "cover the question from multiple angles. Return ONLY the JSON object, no prose."
 )
+FOLLOWUP_PROMPT = (
+    "You plan additional web research to fill coverage gaps. Given a research QUESTION "
+    "and the TITLES of sources already gathered, decide whether more searching would "
+    "surface information the current sources do not yet cover. Produce a JSON object "
+    "with one key 'queries': an array of 0 to 3 concise web search queries targeting "
+    "what is missing. Return an empty array when the sources already cover the question "
+    "well. Return ONLY the JSON object, no prose."
+)
 
 
 def _fallback(query: str) -> list[str]:
@@ -115,3 +125,55 @@ async def plan_queries(
         }
     )
     return fallback
+
+
+async def plan_followup_queries(
+    query: str,
+    covered_titles: list[str],
+    api_key: str,
+    emit: Callable[[dict], None] = _noop,
+) -> list[str]:
+    if not covered_titles:
+        return []
+    titles_block = "\n".join(f"- {title}" for title in covered_titles[:MAX_COVERED_TITLES])
+    payload = {
+        "model": INTERNAL_MODEL,
+        "messages": [
+            {"role": "system", "content": FOLLOWUP_PROMPT},
+            {
+                "role": "user",
+                "content": f"QUESTION: {query}\n\nALREADY COVERED (source titles):\n{titles_block}",
+            },
+        ],
+        "max_tokens": ENHANCE_MAX_TOKENS,
+        "temperature": 0.3,
+    }
+    headers = {
+        "Authorization": f"Bearer {api_key}",
+        "Content-Type": "application/json",
+        "X-App-Reference": "devplace-deepsearch-v-1-0-0",
+    }
+    try:
+        async with stealth.stealth_async_client(timeout=ENHANCE_TIMEOUT_SECONDS) as client:
+            response = await client.post(
+                INTERNAL_GATEWAY_URL, json=payload, headers=headers
+            )
+        if response.status_code >= 400:
+            raise RuntimeError(f"followup planner gateway returned {response.status_code}")
+        data = response.json()
+        content = (
+            (data.get("choices") or [{}])[0].get("message", {}).get("content") or ""
+        )
+        parsed = _parse(content)[:MAX_FOLLOWUP_QUERIES]
+        if parsed:
+            emit(
+                {
+                    "type": "substep",
+                    "phase": PHASE_SEARCHING,
+                    "message": f"Planned {len(parsed)} follow-up queries to fill gaps",
+                }
+            )
+        return parsed
+    except Exception as exc:
+        logger.info("deepsearch followup planner unavailable, skipping refinement: %s", exc)
+        return []

devplacepy/services/jobs/deepsearch/orchestrate.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 import asyncio
 import json
 import logging
+import math
 import re
 from collections.abc import Callable
 from dataclasses import dataclass, field
@@ -31,6 +32,15 @@ PAGE_EXCERPT_CHARS = 3000
 MAX_CONTEXT_CHARS = 36000
 SCORE_MAX = 100
 CONFIDENCE_BASELINE = 0.35
+MMR_LAMBDA = 0.7
+MMR_POOL_MAX = 60
+FOLLOWUP_QUESTIONS_PROMPT = (
+    "Given the QUESTION and the REPORT below, suggest natural follow-up questions a "
+    "curious reader would ask next. Return ONLY a JSON object with key 'questions': "
+    "an array of 3 to 4 short, self-contained question strings. No prose."
+)
+FOLLOWUP_QUESTIONS_MAX_TOKENS = 200
+MAX_FOLLOWUP_QUESTIONS = 4
 
 
 @dataclass
@@ -41,6 +51,7 @@ class Orchestration:
     source_diversity: float = 0.0
     score: int = 0
     synthesis: str = "agents"
+    follow_up_questions: list[str] = field(default_factory=list)
 
 
 def _domain(url: str) -> str:
@@ -66,6 +77,42 @@ def _sanitize_question(question: str) -> str:
     return cleaned[:QUESTION_MAX_CHARS]
 
 
+def _cosine(a: list[float], b: list[float]) -> float:
+    if not a or not b or len(a) != len(b):
+        return 0.0
+    dot = sum(x * y for x, y in zip(a, b))
+    norm_a = math.sqrt(sum(x * x for x in a))
+    norm_b = math.sqrt(sum(y * y for y in b))
+    if not norm_a or not norm_b:
+        return 0.0
+    return dot / (norm_a * norm_b)
+
+
+def _mmr_select(
+    chunks: list, query_vector: list[float], limit: int, lambda_mult: float = MMR_LAMBDA
+) -> list:
+    if not chunks:
+        return []
+    if not query_vector or any(getattr(chunk, "embedding", None) is None for chunk in chunks):
+        return chunks[:limit]
+    selected: list = []
+    remaining = list(chunks)
+    while remaining and len(selected) < limit:
+        best_index = 0
+        best_score = float("-inf")
+        for index, candidate in enumerate(remaining):
+            relevance = _cosine(candidate.embedding, query_vector)
+            redundancy = max(
+                (_cosine(candidate.embedding, chosen.embedding) for chosen in selected),
+                default=0.0,
+            )
+            mmr_score = lambda_mult * relevance - (1 - lambda_mult) * redundancy
+            if mmr_score > best_score:
+                best_score, best_index = mmr_score, index
+        selected.append(remaining.pop(best_index))
+    return selected
+
+
 async def _retrieve_chunks(question: str, queries: list[str], store, api_key: str) -> list:
     texts = [question]
     for query in queries or []:
@@ -88,17 +135,19 @@ async def _retrieve_chunks(question: str, queries: list[str], store, api_key: st
             )
         )
     )
-    merged: list = []
+    pool: list = []
     seen: set[str] = set()
     for tier in zip_longest(*per_query):
         for chunk in tier:
             if chunk is None or chunk.uid in seen:
                 continue
             seen.add(chunk.uid)
-            merged.append(chunk)
-            if len(merged) >= CONTEXT_CHUNKS_MAX:
-                return merged
-    return merged
+            pool.append(chunk)
+            if len(pool) >= MMR_POOL_MAX:
+                break
+        if len(pool) >= MMR_POOL_MAX:
+            break
+    return _mmr_select(pool, vectors[0] if vectors else [], CONTEXT_CHUNKS_MAX)
 
 
 def _chunk_context(chunks: list, pages: list) -> str:
@@ -344,6 +393,29 @@ async def _extract_findings(
     return [], usage
 
 
+async def _suggest_followups(question: str, summary: str, api_key: str) -> list[str]:
+    try:
+        text, _usage = await _complete(
+            [
+                {"role": "system", "content": FOLLOWUP_QUESTIONS_PROMPT},
+                {
+                    "role": "user",
+                    "content": f"QUESTION: {question}\n\nREPORT:\n{summary[:4000]}",
+                },
+            ],
+            api_key,
+            FOLLOWUP_QUESTIONS_MAX_TOKENS,
+        )
+        questions = _parse_json(text).get("questions")
+        if not isinstance(questions, list):
+            return []
+        cleaned = [str(item).strip() for item in questions if str(item).strip()]
+        return cleaned[:MAX_FOLLOWUP_QUESTIONS]
+    except Exception as exc:
+        logger.info("deepsearch follow-up questions unavailable: %s", exc)
+        return []
+
+
 async def orchestrate(
     question: str,
     pages: list,
@@ -423,6 +495,7 @@ async def orchestrate(
         score = int(
             min(SCORE_MAX, (confidence * 0.5 + diversity * 0.3 + coverage * 0.2) * SCORE_MAX)
         )
+        follow_up_questions = await _suggest_followups(question, summary, api_key)
         return Orchestration(
             summary=summary,
             findings=findings,
@@ -430,6 +503,7 @@ async def orchestrate(
             source_diversity=diversity,
             score=score,
             synthesis="agents",
+            follow_up_questions=follow_up_questions,
         )
     except Exception as exc:
         logger.warning("deepsearch orchestration failed, using heuristic: %s", exc)

devplacepy/services/jobs/deepsearch/worker.py

@@ -14,7 +14,7 @@ from devplacepy.utils import generate_uid
 
 from .chunking import chunk_text
 from .crawl import content_hash, crawl, search_queries, url_hash
-from .enhance import plan_queries
+from .enhance import plan_followup_queries, plan_queries
 from .orchestrate import orchestrate
 from .phases import (
     PHASE_ANALYSIS,
@@ -32,6 +32,8 @@ CONTROL_FILE = "control.json"
 PAUSE_POLL_SECONDS = 1.0
 EMBED_BATCH = 64
 FRAME_VERSION = 1
+MAX_REFINE_ROUNDS = 2
+REFINE_CRAWL_DEPTH = 1
 
 _first_frame_sent = False
 
@@ -183,6 +185,45 @@ async def _run(payload: dict, output_dir: Path) -> dict:
         depth=depth,
     )
 
+    round_no = 0
+    while (
+        len(outcome.pages) < max_pages
+        and round_no < MAX_REFINE_ROUNDS
+        and not await should_stop()
+    ):
+        covered_titles = [page.title for page in outcome.pages if page.title]
+        followups = await plan_followup_queries(query, covered_titles, api_key, _emit)
+        if not followups:
+            break
+        round_no += 1
+        _emit(
+            {
+                "type": "substep",
+                "phase": PHASE_SEARCHING,
+                "message": f"Refinement round {round_no}: filling gaps with {len(followups)} more queries",
+            }
+        )
+        _emit({"type": "queries", "queries": followups, "round": round_no})
+        more_candidates = await search_queries(followups, _emit)
+        existing_urls = {page.url for page in outcome.pages}
+        more_candidates = [c for c in more_candidates if c["url"] not in existing_urls]
+        if not more_candidates:
+            break
+        _emit({"type": "candidates", "count": len(more_candidates), "round": round_no})
+        extra = await crawl(
+            more_candidates,
+            max_pages - len(outcome.pages),
+            _emit,
+            lambda url: url_hash(url) in cached_hashes,
+            should_stop,
+            query=query,
+            depth=REFINE_CRAWL_DEPTH,
+            seen_hashes=outcome.seen_hashes,
+        )
+        if not extra.pages:
+            break
+        outcome.pages.extend(extra.pages)
+
     new_cache = [
         {
             "url_hash": url_hash(page.url),
@@ -224,6 +265,7 @@ async def _run(payload: dict, output_dir: Path) -> dict:
         "confidence": result.confidence,
         "source_diversity": result.source_diversity,
         "synthesis": result.synthesis,
+        "follow_up_questions": result.follow_up_questions,
         "page_count": len(outcome.pages),
         "chunk_count": chunk_count,
         "embed_backend": embed_backend,

devplacepy/services/messaging/CLAUDE.md

@@ -49,11 +49,11 @@ Because the WS accepts on every worker, a message persisted on worker A must sti
 
 ## AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content
 
-`"messages": ("content",)` is in the `CORRECTABLE_FIELDS` registry and `persist_message` invokes `schedule_correction`/`schedule_modification` (sender = the user), so typing `@ai <instruction>` in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through `routers/messages._finalize_and_broadcast(sender, message, request, client_id=None)`: it reads any pending SYNC correction/modification futures stashed on `request.scope[PENDING_SCOPE_KEY]` (the same `PENDING_SCOPE_KEY`/`loop.run_in_executor` mechanism the HTTP `await_pending_corrections` middleware uses) into `ai_processed = bool(pending)` **before** awaiting/clearing them, awaits them, RE-READS the message row, and broadcasts the FINAL (corrected/modified) content via `broadcast_message(sender, message, client_id, ai_processed=ai_processed)`. The HTTP `POST /send` and the WS `/ws` send path both call it; the WS path passes `request=websocket` to `persist_message` so the modifier stashes its executor future on the websocket scope and the handler awaits it directly (HTTP middleware does not run for websockets). A message with no `@ai` directive and correction off has nothing pending, so `ai_processed` is `False` and the broadcast is immediate with zero added latency. `broadcast_message` forwards `ai_processed` straight into `message_frame(..., ai_processed=ai_processed)` - see "WS protocol frames" below for the frame shape. This flag is informational only: the server never retains the pre-AI text anywhere (`table.update` overwrites in place), so it exists purely to drive a client-side "Adjusted by AI" affordance, not any kind of diff/revert capability.
+`"messages": ("content",)` is in the `CORRECTABLE_FIELDS` registry and `persist_message` invokes `schedule_correction`/`schedule_modification` (sender = the user), so typing `@ai <instruction>` in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through `routers/messages._finalize_and_broadcast`: it broadcasts the persisted row immediately (`ai_pending=true` when sync futures were stashed on `request.scope[PENDING_SCOPE_KEY]`), then applies any pending SYNC correction/modifier. The HTTP `POST /send` path awaits that apply so the JSON body is the final text; the WS send path schedules it with `asyncio.create_task` so the receive loop never blocks on the gateway. After an in-place rewrite, `_run_correction`/`_run_modification` call `push_content_revision` which stamps `messages.updated_at` and pushes a second `message` frame (`ai_processed=true`) to local sockets. Cross-worker peers pick the revision up from `message_relay._tick_updates` (`SELECT ... WHERE updated_at > watermark`, independent of the new-row `id` watermark and of `was_delivered`). The client matches the second frame on `data-msg-uid` and replaces the bubble body. The WS loop also accepts `{type:"sync", since, with_uid?}` to replay rows created or revised after `since` (reconnect catch-up) and `{type:"ping"}` (ignored). `ContentRefused` on a WS send returns `{type:"error"}` and keeps the socket open.
 
 ## WS protocol frames
 
-Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`. Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; `ai_processed` is additive - `true` only when the broadcast content is the result of an awaited pending correction/modifier future, so an unmodified send is `false` with zero extra computation), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender). There is no `presence` frame on this socket - see below.
+Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below.
 
 ## Presence is NOT part of the messaging WS
 
@@ -65,8 +65,8 @@ Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubbl
 
 ## Frontend
 
-The messaging frontend is the single self-booting custom element `<dp-chat>` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `templates/messages.html` renders `<dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true">` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, modeled on `PubSubClient.js`'s real exponential backoff instead of a flat-delay retry, same `4013` fast-path), sends over WS with an optimistic pending bubble keyed by `client_id` (the FIRST, unconditional branch of the incoming-frame handler matches `sender_uid === selfUid && client_id` against a pending bubble before any other branch - the fix for the old, permanently-disabled `appendOptimistic`), reconciles on the echoed `message` frame (falling to a `.failed`/tap-to-retry state after an 8s timeout with no echo), appends incoming messages live, auto-scrolls, throttles + shows the typing indicator, flips the read-receipt double-check, live-updates the conversation-list preview + unread dot + ordering, and groups consecutive same-sender messages within a 300s gap (`static/js/chat/MessageGrouping.js`, shared between the initial adopted history and the live-append path). If the socket is not open the send `<form method=POST>` submits normally (no-JS fallback). Presence in `mode="page"` needs no component code - the adopted markup keeps its `data-presence-uid` attributes, so the page-global `PresenceManager` already drives the dot/last-seen label; `<dp-chat mode="embed">` (no page-global `PresenceManager`) opens its own scoped `PubSubClient` subscription instead. An opt-in `ai-indicator="true"` attribute shows a brief "Adjusted by AI" caption when a reconciled echo's `ai_processed` flag is true and the content differs from what was locally typed - never a diff/revert UI. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px, with a `theme`/`accent` attribute override mechanism (scoped inline custom properties on the component's own root) for future off-site embedding.
+The messaging frontend is the single self-booting custom element `<dp-chat>` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `<dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true">` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. The send button is disabled only while `dp-upload` is busy, never while a send is in flight. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px.
 
 ## Attachments stream live
 
-The single `message_frame` builder (`services/messaging/persist.py`, used by BOTH `broadcast_message` and the relay) fetches `get_attachments("message", uid)` and includes a slimmed list (`uid`/`url`/`thumbnail_url`/`is_image`/`is_video`/`is_audio`/`original_filename`/`file_size`/`mime_type`) on every frame via `_slim_attachment`. `is_audio` is derived identically to every other attachment surface in the app (`mime_type.startswith("audio/")`, already computed by `attachments._row_to_attachment` and simply forwarded here) - `AppChat._renderAttachments` mirrors `_attachment_display.html`'s type branches (image -> `img.gallery-thumb` marked `data-lightbox`+`data-full`, video -> `<video>`, audio -> `<audio controls>`, else a download link) - no `innerHTML`, so it stays XSS-safe, and a live-delivered audio attachment now renders identically to a page-refreshed one instead of falling back to a generic download link until reload. The sender's optimistic bubble shows an "Uploading attachment(s)..." placeholder until its own echo arrives and swaps in the real gallery. **Attachment-only messages (empty caption) are allowed on both send paths**: the WS `send` handler always allowed it (it reads `content` as a raw string with no Pydantic model), and the HTTP `POST /messages/send` form now does too (`MessageForm.content` is `min_length=0`) - `persist_message` accepts empty content when `attachment_uids` is non-empty (`if not content and not attachment_uids: return None`) and is the single place that rule lives. `dp-upload` (mode `attachment`, default `show-chips` off so only the `(N)` count badge shows) uploads each file to `/uploads/upload` and writes the uids into its hidden `attachment_uids` input; `AppChat._collectAttachments()` reads that before the send, then `dp-upload.clear()` resets it. The send button shows a spinner and is disabled while busy: `AppChat._refreshSendButton()` ORs `_uploading` (driven by the `dp-upload:busy` event) with `_pendingSends` (a map of in-flight send `client_id`s, cleared on the echo or an 8s safety timeout that flips the bubble to `.failed`/tap-to-retry), toggling `.is-sending` + `disabled` on `.messages-send-btn`.
+The single `message_frame` builder (`services/messaging/persist.py`, used by BOTH `broadcast_message` and the relay) fetches `get_attachments("message", uid)` and includes a slimmed list (`uid`/`url`/`thumbnail_url`/`is_image`/`is_video`/`is_audio`/`original_filename`/`file_size`/`mime_type`) on every frame via `_slim_attachment`. `is_audio` is derived identically to every other attachment surface in the app (`mime_type.startswith("audio/")`, already computed by `attachments._row_to_attachment` and simply forwarded here) - `AppChat._renderAttachments` mirrors `_attachment_display.html`'s type branches (image -> `img.gallery-thumb` marked `data-lightbox`+`data-full`, video -> `<video>`, audio -> `<audio controls>`, else a download link) - no `innerHTML`, so it stays XSS-safe, and a live-delivered audio attachment now renders identically to a page-refreshed one instead of falling back to a generic download link until reload. The sender's optimistic bubble shows an "Uploading attachment(s)..." placeholder until its own echo arrives and swaps in the real gallery. **Attachment-only messages (empty caption) are allowed on both send paths**: the WS `send` handler always allowed it (it reads `content` as a raw string with no Pydantic model), and the HTTP `POST /messages/send` form now does too (`MessageForm.content` is `min_length=0`) - `persist_message` accepts empty content when `attachment_uids` is non-empty (`if not content and not attachment_uids: return None`) and is the single place that rule lives. `dp-upload` (mode `attachment`, default `show-chips` off so only the `(N)` count badge shows) uploads each file to `/uploads/upload` and writes the uids into its hidden `attachment_uids` input; `AppChat._collectAttachments()` reads that before the send, then `dp-upload.clear()` resets it. The send button shows a spinner and is disabled only while `dp-upload` is busy (`dp-upload:busy`); in-flight sends stay in `_pendingSends` and flip the bubble to `.failed`/tap-to-retry if no echo arrives (20s), without locking the composer.

devplacepy/services/messaging/__init__.py

@@ -1,7 +1,12 @@
 # retoor <retoor@molodetz.nl>
 
 from devplacepy.services.messaging.hub import message_hub
-from devplacepy.services.messaging.persist import message_frame, persist_message
+from devplacepy.services.messaging.persist import (
+    message_frame,
+    persist_message,
+    push_content_revision,
+    stamp_content_revision,
+)
 from devplacepy.services.messaging.relay import message_relay
 from devplacepy.services.messaging.tickets import issue_ticket, redeem_ticket
 
@@ -11,5 +16,7 @@ __all__ = [
     "message_hub",
     "message_relay",
     "persist_message",
+    "push_content_revision",
     "redeem_ticket",
+    "stamp_content_revision",
 ]

devplacepy/services/messaging/persist.py

@@ -1,11 +1,12 @@
 # retoor <retoor@molodetz.nl>
 
+import asyncio
 import logging
 from datetime import datetime, timezone
 from typing import Any, Optional
 
 from devplacepy.attachments import get_attachments, link_attachments
-from devplacepy.database import get_table, get_blocked_uids
+from devplacepy.database import get_table, get_blocked_uids, get_users_by_uids
 from devplacepy.templating import clear_messages_cache
 from devplacepy.utils import (
     create_mention_notifications,
@@ -100,6 +101,7 @@ def persist_message(
             "content": content,
             "read": False,
             "created_at": created_at,
+            "updated_at": None,
         }
     )
 
@@ -179,4 +181,50 @@ def persist_message(
         "content": content,
         "read": False,
         "created_at": created_at,
+        "updated_at": None,
     }
+
+
+def stamp_content_revision(message_uid: str) -> Optional[dict[str, Any]]:
+    if not message_uid:
+        return None
+    table = get_table("messages")
+    row = table.find_one(uid=message_uid)
+    if not row:
+        return None
+    updated_at = datetime.now(timezone.utc).isoformat()
+    table.update({"uid": message_uid, "updated_at": updated_at}, ["uid"])
+    row["updated_at"] = updated_at
+    return dict(row)
+
+
+def push_content_revision(message_uid: str, *, ai_processed: bool = True) -> None:
+    row = stamp_content_revision(message_uid)
+    if not row:
+        return
+    sender = get_users_by_uids([row["sender_uid"]]).get(row["sender_uid"]) or {}
+    frame = message_frame(
+        row,
+        sender.get("username", ""),
+        sender_role=sender.get("role"),
+        ai_processed=ai_processed,
+    )
+    targets = [row["sender_uid"], row["receiver_uid"]]
+    try:
+        loop = asyncio.get_running_loop()
+    except RuntimeError:
+        loop = None
+    if loop is not None:
+        loop.create_task(_hub_send(targets, frame))
+        return
+    from devplacepy.services.background import background
+
+    bg_loop = background.loop
+    if bg_loop is not None:
+        asyncio.run_coroutine_threadsafe(_hub_send(targets, frame), bg_loop)
+
+
+async def _hub_send(user_uids: list[str], frame: dict[str, Any]) -> None:
+    from devplacepy.services.messaging.hub import message_hub
+
+    await message_hub.send_to_users(user_uids, frame)

devplacepy/services/messaging/relay.py

@@ -2,6 +2,7 @@
 
 import asyncio
 import logging
+from datetime import datetime, timezone
 from typing import Optional
 
 from devplacepy.database import db, get_users_by_uids
@@ -18,6 +19,7 @@ class MessageRelay:
     def __init__(self) -> None:
         self._task: Optional[asyncio.Task] = None
         self._watermark: int = 0
+        self._update_watermark: str = ""
         self._primed: bool = False
 
     def start(self) -> None:
@@ -37,11 +39,13 @@ class MessageRelay:
         try:
             if not self._primed:
                 self._watermark = self._max_id()
+                self._update_watermark = datetime.now(timezone.utc).isoformat()
                 self._primed = True
                 logger.debug("message relay primed at watermark %d", self._watermark)
             while message_hub.has_connections():
                 try:
                     await self._tick()
+                    await self._tick_updates()
                 except Exception:  # noqa: BLE001
                     logger.exception("message relay tick failed")
                 await asyncio.sleep(POLL_INTERVAL_SECONDS)
@@ -85,5 +89,47 @@ class MessageRelay:
             self._watermark,
         )
 
+    async def _tick_updates(self) -> None:
+        if "messages" not in db.tables or not self._update_watermark:
+            return
+        rows = list(
+            db.query(
+                "SELECT * FROM messages"
+                " WHERE updated_at IS NOT NULL AND updated_at > :uwm"
+                " ORDER BY updated_at ASC LIMIT :lim",
+                uwm=self._update_watermark,
+                lim=BATCH_LIMIT,
+            )
+        )
+        if not rows:
+            return
+        connected = message_hub.connected_user_uids()
+        pending = [
+            row
+            for row in rows
+            if row["sender_uid"] in connected or row["receiver_uid"] in connected
+        ]
+        sender_uids = {row["sender_uid"] for row in pending}
+        senders = get_users_by_uids(list(sender_uids)) if sender_uids else {}
+        for row in pending:
+            sender = senders.get(row["sender_uid"]) or {}
+            frame = message_frame(
+                dict(row),
+                sender.get("username", ""),
+                sender_role=sender.get("role"),
+                ai_processed=True,
+            )
+            await message_hub.send_to_users(
+                [row["sender_uid"], row["receiver_uid"]], frame
+            )
+        latest = max((row.get("updated_at") or "") for row in rows)
+        if latest:
+            self._update_watermark = latest
+        logger.debug(
+            "message relay pushed %d of %d content revisions",
+            len(pending),
+            len(rows),
+        )
+
 
 message_relay = MessageRelay()

devplacepy/services/news/CLAUDE.md

@@ -4,20 +4,20 @@ This file documents the automated developer-news import pipeline. Claude Code au
 
 ## Overview
 
-`BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation. `ServiceManager` is a singleton that registers, starts, and stops services. `NewsService` (`services/news/service.py`, `default_enabled=True`, registered unconditionally in `main.py`) is a fully automatic, zero-maintenance import pipeline: it fetches articles from `news_api_url`, **cleans** each (HTML strip + `clean_news_text`/`JUNK_PATTERNS` Reddit-boilerplate removal), **fetches and perceptually compares the images** (SSRF-guarded fetch + Pillow decode + `imagehash` phash off-thread; placeholder = too small / undecodable / a phash shared across 2+ different articles), **grades deterministically** (AI grade on cleaned text via `news_ai_url` + a reliability gate + a unique-image bonus and thin-content penalty -> effective `grade`, raw in `ai_grade`), **reformats every valid article into clean Markdown** (`_format_article` + the editable `news_format_prompt`; paragraphs, `## ` headings, lists and code, preserving every fact; fail-soft to the cleaned original, toggle `news_format_enabled`), and inserts ALL of them into `news` with `status="published"`/`"draft"` on `news_grade_threshold` - nothing is silently skipped. After the loop it **auto-rotates Featured + Landing** (`_apply_landing_selection`, top scored unique-image articles in the recent window) while honouring per-row `featured_locked`/`landing_locked` set when an admin manually toggles. **AI usage is metered like the correction/modifier consumers**: each gateway call's `X-Gateway-*` response headers are parsed (`parse_usage_headers`) and the run's totals accumulated into the durable single-row `news_usage` table (`database.add_news_usage`/`get_news_usage`, same SUMs-plus-computed-averages shape as `correction_usage`); `NewsService.collect_metrics()` surfaces calls/tokens/cost and the per-call averages as `stats` on the admin-only `/admin/services` page. The full ruleset is on `NewsService.description` and surfaces on `/admin/services`, which also polls live status and the log tail.
+`BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation. `ServiceManager` is a singleton that registers, starts, and stops services. `NewsService` (`services/news/service.py`, `default_enabled=True`, registered unconditionally in `main.py`) is a fully automatic, zero-maintenance import pipeline: it fetches articles from `news_api_url`, **cleans** each (HTML strip + `clean_news_text`/`JUNK_PATTERNS` Reddit-boilerplate removal), **fetches and perceptually compares the images** (SSRF-guarded fetch + Pillow decode + `imagehash` phash off-thread; placeholder = too small / undecodable / a phash shared across 2+ different articles), **grades deterministically** (AI grade on cleaned text via `news_ai_url` + a reliability gate + a unique-image bonus and thin-content penalty -> effective `grade`, raw in `ai_grade`), **reformats every article that clears the publish threshold into clean Markdown** (`_format_article` + the editable `news_format_prompt`; paragraphs, `## ` headings, lists and code, preserving every fact; fail-soft to the cleaned original, toggle `news_format_enabled`; draft-bound articles - valid but below `news_grade_threshold` - are never sent for formatting, saving the call entirely), and inserts ALL of them into `news` with `status="published"`/`"draft"` on `news_grade_threshold` - nothing is silently skipped. After the loop it **auto-rotates Featured + Landing** (`_apply_landing_selection`, top scored unique-image articles in the recent window) while honouring per-row `featured_locked`/`landing_locked` set when an admin manually toggles. **AI usage is metered like the correction/modifier consumers**: each gateway call's `X-Gateway-*` response headers are parsed (`parse_usage_headers`) and the run's totals accumulated into the durable single-row `news_usage` table (`database.add_news_usage`/`get_news_usage`, same SUMs-plus-computed-averages shape as `correction_usage`); `NewsService.collect_metrics()` surfaces calls/tokens/cost and the per-call averages as `stats` on the admin-only `/admin/services` page. The full ruleset is on `NewsService.description` and surfaces on `/admin/services`, which also polls live status and the log tail.
 
 ## Per-run flow
 
-The per-run flow per article is: clean text -> fetch and perceptually compare images -> AI-grade the cleaned text -> reliability gate -> compute an effective score -> AI-reformat the body into Markdown (valid articles only) -> publish/Feature decision -> store -> then a single post-loop Landing rotation.
+The per-run flow per article is: clean text -> fetch and perceptually compare images -> AI-grade the cleaned text -> reliability gate -> compute an effective score -> publish/Feature decision -> AI-reformat the body into Markdown (published articles only, i.e. `effective_score >= news_grade_threshold`) -> store -> then a single post-loop Landing rotation.
 
 - Fetches `GET {news_api_url}` -> `{"articles": [...]}`; already-synced `external_id`s (from `news_sync`) are skipped.
-- Every article is graded via AI on the CLEANED text: `POST {news_ai_url}` with model `{news_ai_model}`, temperature 0. Both default to the internal gateway (`INTERNAL_GATEWAY_URL` / `molodetz`); the key falls back to `internal_gateway_key()` when `news_ai_key`/`NEWS_AI_KEY` is unset.
+- Every article is graded via AI on the CLEANED text: `POST {news_ai_url}` with model `{news_ai_model}`, temperature 0. Both default to the free, local **aquality** quality model (`AQUALITY_NEWS_GRADING_URL` = `https://aquality.cloud.pravda.education/v1/chat/completions` / `AQUALITY_NEWS_GRADING_MODEL` = `"aquality"`, `devplacepy/config.py`) - a deterministic scikit-learn regressor trained on this site's own editorial history (`grade`/`status` columns), served behind an OpenAI-chat-completions-compatible shim so no code here changes to call it. No API key is required (the key still falls back to `internal_gateway_key()` when `news_ai_key`/`NEWS_AI_KEY` is unset, but aquality ignores it unless its own `NEWS_QUALITY_API_KEY` is configured). `migrate_ai_gateway_settings()` (`database/schema.py`) one-time-migrates any existing `news_ai_url`/`news_ai_model` still on the old internal-gateway defaults over to aquality, without touching a value an admin has customized. Point `news_ai_url` at a real generative chat model instead to go back to LLM-based grading (higher quality, billed).
 - ALL articles are inserted into `news` regardless of grade (never silently skipped). Articles re-synced each run (upsert by `external_id`); grade, status, image, and images updated each cycle. Slugs via `make_combined_slug(title, uid)`.
 - All parameters (`news_api_url`, `news_ai_url`, `news_ai_model`, `news_grade_threshold`, `news_ai_key`, `news_format_enabled`, `news_format_prompt`, interval) are declared as `config_fields` and edited on the Services tab. The full grading ruleset is carried on `NewsService.description` (the `GRADING_RULES_DESCRIPTION` module string) and renders on `/admin/services`.
 
 ## AI reformatting (`_format_article`, `FORMAT_PROMPT_SPEC`)
 
-After grading, every VALID article (one that passed the reliability gate and got a grade) has its body reformatted by the AI into clean Markdown - short paragraphs, `## ` section headings, bullet/numbered lists, and inline/fenced code - turning the source wall of text into a readable article. It reuses the grading endpoint/model/key (`news_ai_url`/`news_ai_model`/`_get_ai_key`) at `temperature 0.3`, `FORMAT_MAX_TOKENS=6000`, with the cleaned `description`+`content` (capped at `FORMAT_INPUT_MAX_CHARS=14000`) appended to the editable `news_format_prompt`. The prompt forbids inventing/removing facts and only restructures. The result is fence-stripped (`_strip_md_fence`), validated to be at least `MIN_BODY_CHARS`, capped at `FORMAT_OUTPUT_MAX_CHARS=30000`, and stored as the `news.content` (rendered server-side by `render_content`, the markdown engine, on `news_detail.html`). It is fail-soft: on any error, an empty/too-short result, or `news_format_enabled` off, the cleaned original content is stored unchanged. Toggle with the `news_format_enabled` bool config field.
+After grading, every PUBLISHED article (one that passed the reliability gate, got a grade, and whose `effective_score` reaches `news_grade_threshold` - `published` in `run_once`, not merely `result.valid`) has its body reformatted by the AI into clean Markdown - short paragraphs, `## ` section headings, bullet/numbered lists, and inline/fenced code - turning the source wall of text into a readable article. It reuses the grading endpoint/model/key (`news_ai_url`/`news_ai_model`/`_get_ai_key`) at `temperature 0.3`, `FORMAT_MAX_TOKENS=6000`, with the cleaned `description`+`content` (capped at `FORMAT_INPUT_MAX_CHARS=14000`) appended to the editable `news_format_prompt`. The prompt forbids inventing/removing facts and only restructures. The result is fence-stripped (`_strip_md_fence`), validated to be at least `MIN_BODY_CHARS`, capped at `FORMAT_OUTPUT_MAX_CHARS=30000`, and stored as the `news.content` (rendered server-side by `render_content`, the markdown engine, on `news_detail.html`). It is fail-soft: on any error, an empty/too-short result, or `news_format_enabled` off, the cleaned original content is stored unchanged. Toggle with the `news_format_enabled` bool config field. **Defaults to `False`** since aquality, the default `news_ai_url` grading model, is a scoring-only classifier/regressor with no text-generation capability - it answers this call's prompt (no `Description:`/`Content:` labels, so the shared `GRADE_PROMPT_PATTERN` regex on the aquality side does not match) with an empty reply, which this fail-soft path already treats as "keep the cleaned original." Enable it only after pointing `news_ai_url` at a real generative chat model.
 
 ## AI usage metering and stats (the shared `usage.py` helpers, `news_usage`, `collect_metrics`)
 

devplacepy/services/news/constants.py

@@ -2,11 +2,18 @@
 
 import re
 
-from devplacepy.config import INTERNAL_GATEWAY_URL, INTERNAL_MODEL
+from devplacepy.config import (
+    AQUALITY_NEWS_GRADING_MODEL,
+    AQUALITY_NEWS_GRADING_URL,
+    INTERNAL_GATEWAY_URL,
+    INTERNAL_MODEL,
+)
 
 NEWS_API_URL_DEFAULT = "https://news.app.molodetz.nl/api"
-AI_URL_DEFAULT = INTERNAL_GATEWAY_URL
-AI_MODEL_DEFAULT = INTERNAL_MODEL
+AQUALITY_URL_DEFAULT = AQUALITY_NEWS_GRADING_URL
+AQUALITY_MODEL_DEFAULT = AQUALITY_NEWS_GRADING_MODEL
+AI_URL_DEFAULT = AQUALITY_URL_DEFAULT
+AI_MODEL_DEFAULT = AQUALITY_MODEL_DEFAULT
 GRADE_THRESHOLD_DEFAULT = 7
 GRADE_MAX_TOKENS = 2000
 FORMAT_MAX_TOKENS = 6000
@@ -130,12 +137,15 @@ GRADING_RULES_DESCRIPTION = (
     "(Configuration tab), sent to the model at temperature 0 with the cleaned "
     "Title, Description and Content appended; it must return a single integer "
     "from 1 to 10.\n\n"
-    "Formatting: after grading, every valid article is reformatted by the AI "
-    "into clean Markdown (paragraphs, section headings, lists and code spans) "
-    "using the editable news_format_prompt field, preserving every fact while "
-    "turning the source wall of text into a readable article. The formatted "
-    "Markdown replaces the stored content; on any failure the cleaned original "
-    "is kept. Disable with the news_format_enabled toggle.\n\n"
+    "Formatting: after grading, every article that clears the publish "
+    "threshold (i.e. will be stored as published, not draft) is reformatted "
+    "by the AI into clean Markdown (paragraphs, section headings, lists and "
+    "code spans) using the editable news_format_prompt field, preserving "
+    "every fact while turning the source wall of text into a readable "
+    "article. Draft-bound articles are never sent for formatting. The "
+    "formatted Markdown replaces the stored content; on any failure the "
+    "cleaned original is kept. Disable with the news_format_enabled "
+    "toggle.\n\n"
     "Scoring: the AI grade (1-10, temperature 0) is the base. effective_score = "
     f"clamp(ai_grade + {UNIQUE_IMAGE_BONUS} if unique image - "
     f"{THIN_CONTENT_PENALTY} if the body is marginal but not gated, 1..10). The "

devplacepy/services/news/service.py

@@ -80,7 +80,11 @@ class NewsService(BaseService):
             "AI grading URL",
             type="url",
             default=AI_URL_DEFAULT,
-            help="Chat-completions endpoint used to grade each cleaned article.",
+            help=(
+                "Chat-completions endpoint used to grade each cleaned "
+                "article. Defaults to the free, local aquality quality "
+                "model - no billed LLM call, no API key required."
+            ),
             group="AI grading",
         ),
         ConfigField(
@@ -131,10 +135,13 @@ class NewsService(BaseService):
             "news_format_enabled",
             "Reformat content with AI",
             type="bool",
-            default=True,
+            default=False,
             help=(
                 "When enabled, every valid article is reformatted into clean "
-                "Markdown (paragraphs, headings, lists) after grading."
+                "Markdown (paragraphs, headings, lists) after grading. The "
+                "default grading endpoint (aquality) is a scoring-only model "
+                "and cannot reformat text, so this defaults off; enable it "
+                "only when news_ai_url points at a generative chat model."
             ),
             group="AI formatting",
         ),
@@ -252,7 +259,7 @@ class NewsService(BaseService):
                     )
 
                     formatted_content = ""
-                    if result.valid and format_enabled:
+                    if published and format_enabled:
                         formatted_content = await self._format_article(
                             article, ai_url, ai_model, client, usage_totals
                         )

devplacepy/services/quiz/CLAUDE.md

@@ -226,6 +226,8 @@ for `target_type == "quiz"`, soft-deleting questions, options, attempts and answ
 | CLI | `devplace quiz prune` - hard-deletes abandoned and expired attempts older than `QUIZ_ATTEMPT_RETENTION_DAYS`. Completed attempts are never pruned; they are the player's record |
 | Frontend | `dp-quiz-player`/`dp-quiz-builder` (light DOM, adopt the server-rendered markup), `static/css/quiz.css` (adds only what is new - the layout and card chrome come from `feed.css`/`sidebar.css`) |
 
+**The player delays auto-advance so the grade is actually seen.** `AppQuizPlayer._advance` used to call `_showSlide(index + 1)` synchronously in the same tick as rendering the grade, so on any quiz with more than one question the just-answered slide was hidden before a single frame ever painted its `.quiz-grade` - the grade existed in the DOM but the user (and Playwright) never saw it. It now schedules the advance via `setTimeout(..., ADVANCE_DELAY_MS)` (1200ms) so the "Correct"/"Not correct" verdict is visible for a beat before the next question slides in. A single-question quiz never hits this path at all (`index >= this._slides.length - 1` short-circuits), which is why the bug was invisible to any test built around a one-question quiz. `_showSlide` (called by the pending timeout, by manual Previous/Next, and by `disconnectedCallback`) always clears any outstanding `this._advanceTimeout` first, and `_advance` clears one before scheduling a new one - without both, a stale timer from an earlier answer could fire after the user manually navigated away, silently snapping the view back, or two timers scheduled in quick succession (answering out of order via manual nav) could race and land on the wrong slide.
+
 **Generic Devii prompt seeding.** The hub's *Create quiz with Devii* button uses the platform-wide
 `data-devii-prompt` attribute: `DeviiTerminal.bindTriggers` reads it and passes it to
 `open(prompt)`, which calls `devii-terminal.prefill(text)`. It never auto-sends - the member reads

devplacepy/static/css/deepsearch.css

@@ -488,6 +488,30 @@
     outline-offset: 2px;
 }
 
+.ds-followup-list {
+    display: flex;
+    flex-wrap: wrap;
+    gap: var(--space-sm);
+}
+
+.ds-followup-chip {
+    background: var(--bg-input);
+    border: 1px solid var(--border);
+    border-radius: var(--radius-input);
+    color: var(--text-primary);
+    cursor: pointer;
+    font-size: 0.85rem;
+    padding: var(--space-xs) var(--space-md);
+    text-align: left;
+    transition: border-color 0.15s ease, color 0.15s ease;
+}
+
+.ds-followup-chip:hover,
+.ds-followup-chip:focus-visible {
+    border-color: var(--accent);
+    color: var(--accent);
+}
+
 .ds-chat-pane {
     background: var(--bg-card);
     border: 1px solid var(--border);

devplacepy/static/css/feed.css

@@ -640,3 +640,31 @@
     max-height: 14rem;
     overflow: hidden;
 }
+
+.topics-grid {
+    display: grid;
+    grid-template-columns: repeat(auto-fill, minmax(160px, 1fr));
+    gap: 1rem;
+    margin-top: 1.5rem;
+}
+
+.topic-card {
+    display: flex;
+    flex-direction: column;
+    align-items: flex-start;
+    gap: 0.5rem;
+    background: var(--bg-card);
+    border: 1px solid var(--border);
+    border-radius: var(--radius-lg);
+    padding: 1.25rem;
+}
+
+.topic-card:hover {
+    border-color: var(--border-light);
+    box-shadow: var(--shadow-sm);
+}
+
+.topic-card-count {
+    font-size: 0.875rem;
+    color: var(--text-muted);
+}

devplacepy/static/js/ContentEnhancer.js

@@ -48,14 +48,18 @@ export class ContentEnhancer {
             const picker = new EmojiPicker(textarea);
             btn.addEventListener("click", () => picker.toggle());
 
-            const parent = textarea.parentElement;
-            if (parent) {
-                const actions = parent.querySelector(".comment-form-actions");
-                if (actions) {
-                    actions.style.position = "relative";
-                    actions.appendChild(picker.wrapper);
-                    actions.insertBefore(btn, actions.firstChild);
-                }
+            const form = textarea.closest(".comment-form, .messages-input-area");
+            const actions = form && form.querySelector(".comment-form-actions");
+            if (actions) {
+                actions.style.position = "relative";
+                actions.appendChild(picker.wrapper);
+                actions.insertBefore(btn, actions.firstChild);
+                return;
+            }
+            if (form && form.classList.contains("messages-input-area")) {
+                const sendBtn = form.querySelector(".messages-send-btn");
+                if (sendBtn) form.insertBefore(btn, sendBtn);
+                else form.appendChild(btn);
             }
         });
     }

devplacepy/static/js/GameFarm.js

@@ -287,14 +287,21 @@ export class GameFarm {
                 const options = farm.crops
                     .map((crop) => {
                         const disabled = crop.locked || crop.cost > farm.coins ? " disabled" : "";
-                        const lock = crop.locked ? ` (Lv ${crop.min_level})` : "";
+                        const title = crop.locked ? ` title="${crop.locked_text}"` : "";
+                        const lock = crop.locked_reason === "mastery"
+                            ? " (Mastery)"
+                            : crop.locked_reason === "era"
+                              ? " (Era)"
+                              : crop.locked
+                                ? ` (Lv ${crop.min_level})`
+                                : "";
                         const market =
                             crop.market_state === "saturated"
                                 ? " (saturated)"
                                 : crop.market_state === "boosted"
                                   ? " (boosted)"
                                   : "";
-                        return `<option value="${crop.key}"${disabled}>${crop.icon} ${crop.name} - ${Format.coins(crop.cost)}${lock}${market}</option>`;
+                        return `<option value="${crop.key}"${disabled}${title}>${crop.icon} ${crop.name} - ${Format.coins(crop.cost)}${lock}${market}</option>`;
                     })
                     .join("");
                 body = `<form class="plot-plant-form" method="post" action="/game/plant" data-game-action="plant"><input type="hidden" name="slot" value="${plot.slot}"><select name="crop" class="plot-crop-select" aria-label="Choose a project to build">${options}</select><button type="submit" class="btn btn-sm btn-primary">Plant</button></form>`;

devplacepy/static/js/MobileNav.js

@@ -3,8 +3,6 @@
 export class MobileNav {
     constructor() {
         this.initMobileNav();
-        this.initMessagesResponsive();
-        this.initMessageThread();
         this.initProfileDropdown();
         this.initToolsDropdown();
     }
@@ -55,55 +53,6 @@ export class MobileNav {
         });
     }
 
-    initMessagesResponsive() {
-        const list = document.querySelector(".messages-list");
-        const main = document.querySelector(".messages-main");
-        const backBtn = document.getElementById("messages-back-btn");
-        if (!list || !main) return;
-
-        const isMobile = () => window.innerWidth <= 768;
-
-        if (isMobile()) {
-            if (window.location.search.includes("with_uid=")) {
-                list.classList.add("hide");
-                main.classList.remove("hide");
-            } else {
-                list.classList.remove("hide");
-                main.classList.add("hide");
-            }
-        }
-
-        if (backBtn) {
-            backBtn.addEventListener("click", () => {
-                if (!isMobile()) return;
-                list.classList.remove("hide");
-                main.classList.add("hide");
-                window.history.replaceState(null, "", "/messages");
-            });
-        }
-
-        list.querySelectorAll(".conversation-item").forEach((item) => {
-            item.addEventListener("click", (e) => {
-                if (!isMobile()) return;
-                list.classList.add("hide");
-                main.classList.remove("hide");
-            });
-        });
-
-        const mq = window.matchMedia("(max-width: 768px)");
-        mq.addEventListener("change", () => {
-            if (!isMobile()) {
-                list.classList.remove("hide");
-                main.classList.remove("hide");
-            }
-        });
-    }
-
-    initMessageThread() {
-        const thread = document.querySelector(".messages-thread");
-        if (thread) thread.scrollTop = thread.scrollHeight;
-    }
-
     initProfileDropdown() {
         const dropdown = document.querySelector(".topnav-user-dropdown");
         if (!dropdown) return;

devplacepy/static/js/chat/ChatSocket.js

@@ -4,6 +4,7 @@ const WRONG_WORKER_CODE = 4013;
 const WRONG_WORKER_RETRY_MS = 200;
 const INITIAL_BACKOFF_MS = 200;
 const MAX_BACKOFF_MS = 5000;
+const PING_MS = 25000;
 
 export class ChatSocket {
     constructor(url, handlers) {
@@ -13,6 +14,7 @@ export class ChatSocket {
         this._shouldRun = false;
         this._settled = false;
         this._backoff = INITIAL_BACKOFF_MS;
+        this._pingId = null;
     }
 
     connect() {
@@ -29,6 +31,8 @@ export class ChatSocket {
         const socket = new WebSocket(this.url);
         this.socket = socket;
 
+        socket.addEventListener("open", () => this._startPing());
+
         socket.addEventListener("message", (event) => {
             let payload;
             try {
@@ -39,12 +43,14 @@ export class ChatSocket {
             if (!this._settled) {
                 this._settled = true;
                 this._backoff = INITIAL_BACKOFF_MS;
-                this._emit("onReady", payload);
+                this._emit("onReady", payload && payload.type === "ready" ? payload : { type: "ready" });
             }
+            if (payload && payload.type === "ready") return;
             this._emit("onMessage", payload);
         });
 
         socket.addEventListener("close", (event) => {
+            this._stopPing();
             const wrongWorker = !this._settled && event.code === WRONG_WORKER_CODE;
             this._settled = false;
             if (wrongWorker) {
@@ -74,9 +80,24 @@ export class ChatSocket {
 
     close() {
         this._shouldRun = false;
+        this._stopPing();
         if (this.socket) this.socket.close();
     }
 
+    _startPing() {
+        this._stopPing();
+        this._pingId = window.setInterval(() => {
+            if (this.isOpen()) this.send({ type: "ping" });
+        }, PING_MS);
+    }
+
+    _stopPing() {
+        if (this._pingId) {
+            window.clearInterval(this._pingId);
+            this._pingId = null;
+        }
+    }
+
     _emit(name, payload) {
         const handler = this.handlers[name];
         if (typeof handler === "function") handler(payload);

devplacepy/static/js/components/AppContent.js

@@ -17,7 +17,8 @@ export class AppContent extends Component {
             return;
         }
         this._rendered = true;
-        this._source = this.textContent;
+        this._source = this.dataset.source || this.textContent;
+        this.dataset.source = this._source;
         this.classList.add("rendered-content");
         contentRenderer.applyTo(this);
         if (typeof hljs !== "undefined") {

devplacepy/static/js/components/AppDeepsearchChat.js

@@ -124,6 +124,10 @@ export class AppDeepsearchChat extends Component {
         }
     }
 
+    ask(text) {
+        this._submit(text);
+    }
+
     _submit(raw) {
         const text = (raw || "").trim();
         if (!text || this._busy) return;

devplacepy/static/js/components/AppQuizPlayer.js

@@ -5,6 +5,7 @@ import { Http } from "../Http.js";
 import { Format } from "../Format.js";
 
 const TICK_MS = 1000;
+const ADVANCE_DELAY_MS = 1200;
 const CHECKING_LABEL = "Checking…";
 
 export class AppQuizPlayer extends Component {
@@ -24,6 +25,7 @@ export class AppQuizPlayer extends Component {
 
     disconnectedCallback() {
         if (this._interval) clearInterval(this._interval);
+        if (this._advanceTimeout) clearTimeout(this._advanceTimeout);
     }
 
     _initSlides() {
@@ -57,6 +59,10 @@ export class AppQuizPlayer extends Component {
 
     _showSlide(index) {
         if (!this._slides.length) return;
+        if (this._advanceTimeout) {
+            clearTimeout(this._advanceTimeout);
+            this._advanceTimeout = null;
+        }
         this._current = Math.max(0, Math.min(this._slides.length - 1, index));
         this._slides.forEach((slide, i) => {
             slide.hidden = i !== this._current;
@@ -177,7 +183,9 @@ export class AppQuizPlayer extends Component {
         const question = form.closest("[data-quiz-question]");
         if (!question || !this._slides.length) return;
         const index = this._slides.indexOf(question);
-        if (index !== -1 && index < this._slides.length - 1) this._showSlide(index + 1);
+        if (index === -1 || index >= this._slides.length - 1) return;
+        if (this._advanceTimeout) clearTimeout(this._advanceTimeout);
+        this._advanceTimeout = setTimeout(() => this._showSlide(index + 1), ADVANCE_DELAY_MS);
     }
 }
 

devplacepy/templates/_game_grid.html

@@ -7,7 +7,7 @@
             <input type="hidden" name="slot" value="{{ plot.slot }}">
             <select name="crop" class="plot-crop-select" aria-label="Choose a project to build">
                 {% for crop in farm.crops %}
-                <option value="{{ crop.key }}"{% if crop.locked or crop.cost > farm.coins %} disabled{% endif %}>{{ crop.icon }} {{ crop.name }} - {{ format_coins(crop.cost) }}{% if crop.locked %} (Lv {{ crop.min_level }}){% endif %}{% if crop.market_state == 'saturated' %} (saturated){% elif crop.market_state == 'boosted' %} (boosted){% endif %}</option>
+                <option value="{{ crop.key }}"{% if crop.locked or crop.cost > farm.coins %} disabled{% endif %}{% if crop.locked %} title="{{ crop.locked_text }}"{% endif %}>{{ crop.icon }} {{ crop.name }} - {{ format_coins(crop.cost) }}{% if crop.locked_reason == 'mastery' %} (Mastery){% elif crop.locked_reason == 'era' %} (Era){% elif crop.locked %} (Lv {{ crop.min_level }}){% endif %}{% if crop.market_state == 'saturated' %} (saturated){% elif crop.market_state == 'boosted' %} (boosted){% endif %}</option>
                 {% endfor %}
             </select>
             <button type="submit" class="btn btn-sm btn-primary">Plant</button>

devplacepy/templates/docs/claude-manual.html

@@ -40,7 +40,7 @@ on it.
 ## Before you start
 
 ```
-make install      # editable install (first time)
+make install      # create .venv if needed, editable install + playwright (first time)
 /serve            # start the dev server on port 10500 and confirm health
 ```
 

devplacepy/templates/docs/getting-started.html

@@ -11,7 +11,7 @@ itself.
 DevPlace is a Python package. Install it editable, then start the reloading dev server:
 
 ```bash
-make install      # pip install -e .
+make install      # create .venv if needed, pip install -e ".[dev]", playwright chromium
 make dev          # uvicorn --reload on port 10500
 ```
 

devplacepy/templates/docs/testing-make.html

@@ -8,8 +8,7 @@ Every test workflow has a `make` target, so commands stay short and identical ac
 Install the package with its development extras (`pytest`, Playwright, coverage) and the browser binary once:
 
 ```bash
-make install                          # pip install -e . AND playwright install chromium
-pip install -e ".[dev]"               # test dependencies (pytest, coverage, requests)
+make install                          # create .venv if needed, pip install -e ".[dev]", playwright chromium
 ```
 
 ## Correctness suite

devplacepy/templates/messages.html

@@ -69,7 +69,7 @@
 
         <form class="messages-input-area" method="POST" action="/messages/send" data-live-form>
             <input type="hidden" name="receiver_uid" value="{{ other_user['uid'] }}">
-            <textarea name="content" placeholder="Type a message..." maxlength="2000" autocomplete="off" data-mention aria-label="Type a message" rows="1"></textarea>
+            <textarea name="content" placeholder="Type a message..." maxlength="2000" autocomplete="off" data-mention class="emoji-picker-target" aria-label="Type a message" rows="1"></textarea>
             <dp-upload multiple paste
                 max-size="{{ max_upload_size_mb() }}"
                 max-files="5"

devplacepy/templates/tools/deepsearch_session.html

@@ -78,6 +78,17 @@
             </ol>
         </section>
         {% endif %}
+
+        {% if follow_up_questions and chat_ws_url %}
+        <section class="ds-card ds-followups">
+            <h2>Ask next</h2>
+            <div class="ds-followup-list">
+                {% for question in follow_up_questions %}
+                <button type="button" class="ds-followup-chip" data-followup>{{ question }}</button>
+                {% endfor %}
+            </div>
+        </section>
+        {% endif %}
     </div>
 
     <aside class="ds-chat-pane" aria-label="Ask the research">
@@ -94,5 +105,11 @@
 {% block extra_js %}
 <script type="module">
     import "{{ static_url('/static/js/components/AppDeepsearchChat.js') }}";
+    document.querySelectorAll("[data-followup]").forEach((button) => {
+        button.addEventListener("click", () => {
+            const chat = document.querySelector("dp-deepsearch-chat");
+            if (chat) chat.ask(button.textContent);
+        });
+    });
 </script>
 {% endblock %}

devplacepy/templates/topic.html

@@ -0,0 +1,24 @@
+{% extends "base.html" %}
+{% block extra_head %}
+<link rel="stylesheet" href="{{ static_url('/static/css/feed.css') }}">
+<link rel="stylesheet" href="{{ static_url('/static/css/post.css') }}">
+{% endblock %}
+{% block content %}
+<div class="saved-page">
+    <a href="/topics" class="back-link">&larr; All topics</a>
+    <h1 class="saved-title">
+        <span class="badge badge-{{ topic }}">{{ topic_label }}</span>
+        posts
+    </h1>
+
+    <div class="feed-posts" role="feed" aria-label="{{ topic_label }} posts">
+        {% for item in posts %}
+        {% set _author = item.author %}{% set _time = item.time_ago %}{% set _show_share = true %}{% set _show_comment_form = true %}{% include "_post_card.html" %}
+        {% else %}
+        <div class="empty-state">No {{ topic_label.lower() }} posts yet.</div>
+        {% endfor %}
+    </div>
+
+    {% include "_load_more.html" %}
+</div>
+{% endblock %}

devplacepy/templates/topics.html

@@ -0,0 +1,22 @@
+{% extends "base.html" %}
+{% block extra_head %}
+<link rel="stylesheet" href="{{ static_url('/static/css/feed.css') }}">
+{% endblock %}
+{% block content %}
+<div class="saved-page">
+    <h1 class="saved-title">Topics</h1>
+    <p class="subtitle">Browse posts by topic.</p>
+
+    <div class="topics-grid">
+        {% for t in topics %}
+        <div class="topic-card card-link-host">
+            {% set _href = "/topics/" ~ t.key %}
+            {% set _label = t.label %}
+            {% include "_card_link.html" %}
+            <span class="badge badge-{{ t.key }}">{{ t.label }}</span>
+            <span class="topic-card-count">{{ t.post_count }} post{{ 's' if t.post_count != 1 else '' }}</span>
+        </div>
+        {% endfor %}
+    </div>
+</div>
+{% endblock %}

devplacepy/utils/CLAUDE.md

@@ -14,6 +14,7 @@ The top-nav bell and Messages link carry `data-counter="notifications"` / `data-
 |------|---------|----------|-----------|
 | `comment` | Top-level comment on post | `comments.py` | `post["user_uid"] != user["uid"]` |
 | `reply` | Reply to a comment | `comments.py` | `parent["user_uid"] != user["uid"]` |
+| `thread` | Any comment on a post, for every other user who has also commented on that post | `content.py` `create_comment_record` -> `utils.py` `create_thread_notifications` | Disregards comment hierarchy (parent/child) - fires for every distinct commenter on the post except the actor and whoever already got `comment`/`reply` for this same event |
 | `vote` | Upvote on post or comment | `votes.py` | `value == 1` AND voter != owner |
 | `follow` | Follow another user | `follow.py` | Always (self-follow blocked upstream) |
 | `message` | Send a message | `messages.py` | Always (different user) |
@@ -27,6 +28,8 @@ The `moderation` type carries both faces of the safety layer: the acknowledgemen
 
 `level`/`badge` notifications have `related_uid == user_uid` (self), so the notifications template renders them with the recipient's own avatar; the template is type-agnostic (driven by `message`/`actor`), so no per-type handling is needed.
 
+`thread` is the flat, hierarchy-agnostic counterpart to `comment`/`reply`: after those two fire (or don't - e.g. the actor is the post owner), `create_comment_record` queries the post's distinct non-deleted commenters and fans out one `thread` notification to each, excluding the actor and anyone already notified as the post owner (`comment`) or the reply parent (`comment`/`reply`) for this exact event - so a given comment never double-notifies the same recipient under two types. Only `target_type == "post"` fires it, matching the existing `comment`/`reply` restriction (project/gist/news comments do not notify at all today). Like `create_mention_notifications`, the fan-out (`utils.py` `create_thread_notifications` / `_deliver_thread_notifications`) is deferred through `background.submit`, never inlined on the request path.
+
 ### Auto-mark-read on content view
 
 Beyond the explicit clears (`/notifications/open/{uid}`, `POST /notifications/mark-read/{uid}`, `POST /notifications/mark-all-read`), a notification is **automatically marked read when the user opens the page where its referenced content is visible**. The single primitive is `database.mark_notifications_read_by_target(user_uid, target_url)`: it stamps `read = 1` on every unread notification of that user whose stored `target_url` equals `target_url` OR begins with `target_url + "#"` (so a post-page key clears its `#comment-{uid}` variants), then `clear_unread_cache(user_uid)`. It returns the count and short-circuits (no write, no cache bump) when nothing matches, so a detail view by a user with no relevant unread notifications costs only one indexed read. There is **no** `target_type`/`target_uid` column - the `target_url` string (produced by `resolve_object_url`) is the only content key, so callers build the page key with that same generator (or the identical literal the create-site used) to guarantee a match with zero drift.
@@ -35,7 +38,7 @@ Call sites (each only when a real user is present and the resource resolved, pla
 
 | GET handler | page key | clears |
 |-------------|----------|--------|
-| `posts.py::view_post` | `resolve_object_url("post", uid)` | comment, reply, vote, mention (post + its comments) |
+| `posts.py::view_post` | `resolve_object_url("post", uid)` | comment, reply, thread, vote, mention (post + its comments) |
 | `projects/index.py::project_detail` | `resolve_object_url("project", uid)` | vote, mention |
 | `gists.py::gist_detail` | `resolve_object_url("gist", uid)` | vote, mention |
 | `news.py::news_detail_page` | `resolve_object_url("news", uid)` | comment, vote, mention |

devplacepy/utils/__init__.py

@@ -94,6 +94,8 @@ from devplacepy.utils.notifications import (
     _schedule_telegram,
     create_notification,
     _deliver_notification,
+    create_thread_notifications,
+    _deliver_thread_notifications,
     create_mention_notifications,
     _deliver_mention_notifications,
 )

devplacepy/utils/notifications.py

@@ -129,6 +129,37 @@ def _deliver_notification(
     )
 
 
+def create_thread_notifications(
+    target_uid: str, actor_uid: str, comment_url: str, exclude_uids: set[str]
+) -> None:
+    background.submit(
+        _deliver_thread_notifications, target_uid, actor_uid, comment_url, exclude_uids
+    )
+
+
+def _deliver_thread_notifications(
+    target_uid: str, actor_uid: str, comment_url: str, exclude_uids: set[str]
+) -> None:
+    users = get_table("users")
+    actor = users.find_one(uid=actor_uid)
+    if not actor:
+        return
+    exclude = set(exclude_uids) | {actor_uid}
+    comments = get_table("comments")
+    participant_uids = {
+        row["user_uid"]
+        for row in comments.find(target_type="post", target_uid=target_uid, deleted_at=None)
+    } - exclude
+    for participant_uid in participant_uids:
+        create_notification(
+            participant_uid,
+            "thread",
+            f"{actor['username']} also commented on a post you commented on",
+            actor_uid,
+            comment_url,
+        )
+
+
 def create_mention_notifications(content: str, actor_uid: str, target_url: str) -> None:
     background.submit(_deliver_mention_notifications, content, actor_uid, target_url)
 

events.md

@@ -464,7 +464,7 @@ Every state-changing action in DevPlace records one append-only row through `dev
 | `push.subscribe` | `routers/push.py` |
 | `push.update` | `routers/push.py` |
 
-Both carry `metadata.provider` (`webpush`, `apns`, ...) plus `created`; `metadata.endpoint_host` is set for endpoint-based providers and `null` for token-based ones.
+Both carry `metadata.provider` (`webpush`, `apns`, ...) plus `created`, `revived`, and `has_client_id`; `metadata.endpoint_host` is set for endpoint-based providers and `null` for token-based ones. `push.update` fires when an existing row is re-posted or a dead token is revived, including APNs token rotation on the same `client_id`.
 
 ## Gamification (`reward`)
 

locustfile.py

@@ -30,7 +30,7 @@ SEO_JOB_UIDS = []
 POLLS = []
 ADMIN_USER = {}
 ADMIN_TARGETS = {}
-TOPICS = ["devlog", "showcase", "question", "rant", "fun", "random"]
+TOPICS = ["devlog", "showcase", "question", "rant", "fun", "random", "politics"]
 PROJECT_TYPES = ["game", "game_asset", "software", "mobile_app", "website"]
 GIST_LANGUAGES = ["python", "javascript", "go", "rust", "bash", "sql", "json"]
 REACTION_EMOJIS = [
@@ -571,6 +571,19 @@ class DevPlaceUser(HttpUser):
             params["before"] = cursor.group(1)
             self.client.get("/feed", params=params, name="feed?before")
 
+    @task(2)
+    def view_topics(self):
+        self.client.get("/topics", name="topics")
+        topic = random.choice(TOPICS)
+        resp = self.client.get(f"/topics/{topic}", name="topics/[topic]")
+        cursor = re.search(r'/topics/[^"?]+\?before=([^"&]+)', resp.text)
+        if cursor:
+            self.client.get(
+                f"/topics/{topic}",
+                params={"before": cursor.group(1)},
+                name="topics/[topic]?before",
+            )
+
     @task(4)
     def view_post(self):
         slugs = POST_SLUGS if POST_SLUGS else POST_UIDS
@@ -1819,10 +1832,15 @@ class AnonymousUser(HttpUser):
                 "/leaderboard",
                 "/reports/reasons",
                 "/workspaces/index",
+                "/topics",
             ]
         )
         self.client.get(path, name="public/[page]")
 
+    @task(2)
+    def browse_topic(self):
+        self.client.get(f"/topics/{random.choice(TOPICS)}", name="topics/[topic]")
+
     @task(1)
     def browse_docs(self):
         self.client.get(

tests/api/comments/create.py

@@ -351,3 +351,147 @@ def test_comment_links_multiple_attachments(app_server):
     assert u1 in r.text and u2 in r.text, (
         "both attachments must be linked and displayed on the comment"
     )
+
+
+def test_thread_notifies_other_commenters_disregarding_hierarchy(app_server):
+    owner, _ = _session_comments()
+    title = f"thread-test-{int(time.time() * 1000)}"
+    post_uid = _create_post_comments(owner, title)
+
+    first, first_name = _session_comments()
+    first.post(
+        f"{BASE_URL}/comments/create",
+        data={
+            "content": "First commenter",
+            "target_type": "post",
+            "post_uid": post_uid,
+            "target_uid": post_uid,
+        },
+        allow_redirects=False,
+    )
+    refresh_snapshot()
+    first_uid = _db_user(first_name)["uid"]
+
+    second, _ = _session_comments()
+    second.post(
+        f"{BASE_URL}/comments/create",
+        data={
+            "content": "Second commenter, unrelated to the first's comment",
+            "target_type": "post",
+            "post_uid": post_uid,
+            "target_uid": post_uid,
+        },
+        allow_redirects=False,
+    )
+    refresh_snapshot()
+    assert get_table("notifications").count(user_uid=first_uid, type="thread") == 1
+
+
+def test_thread_does_not_notify_self_when_commenting_again(app_server):
+    owner, _ = _session_comments()
+    title = f"thread-self-{int(time.time() * 1000)}"
+    post_uid = _create_post_comments(owner, title)
+
+    commenter, commenter_name = _session_comments()
+    commenter.post(
+        f"{BASE_URL}/comments/create",
+        data={
+            "content": "First comment from this user",
+            "target_type": "post",
+            "post_uid": post_uid,
+            "target_uid": post_uid,
+        },
+        allow_redirects=False,
+    )
+    refresh_snapshot()
+    commenter_uid = _db_user(commenter_name)["uid"]
+
+    commenter.post(
+        f"{BASE_URL}/comments/create",
+        data={
+            "content": "Same user comments again on the same post",
+            "target_type": "post",
+            "post_uid": post_uid,
+            "target_uid": post_uid,
+        },
+        allow_redirects=False,
+    )
+    refresh_snapshot()
+    assert get_table("notifications").count(user_uid=commenter_uid, type="thread") == 0
+
+
+def test_thread_notification_excludes_actor_and_already_notified_participants(app_server):
+    owner, owner_name = _session_comments()
+    title = f"thread-excl-{int(time.time() * 1000)}"
+    post_uid = _create_post_comments(owner, title)
+    refresh_snapshot()
+    owner_uid = _db_user(owner_name)["uid"]
+
+    owner.post(
+        f"{BASE_URL}/comments/create",
+        data={
+            "content": "Owner's own top-level comment",
+            "target_type": "post",
+            "post_uid": post_uid,
+            "target_uid": post_uid,
+        },
+        allow_redirects=False,
+    )
+    refresh_snapshot()
+    assert get_table("notifications").count(user_uid=owner_uid, type="thread") == 0
+    parent = get_table("comments").find_one(target_uid=post_uid, user_uid=owner_uid)
+
+    replier, replier_name = _session_comments()
+    replier.post(
+        f"{BASE_URL}/comments/create",
+        data={
+            "content": "Reply to the post owner's comment",
+            "target_type": "post",
+            "post_uid": post_uid,
+            "target_uid": post_uid,
+            "parent_uid": parent["uid"],
+        },
+        allow_redirects=False,
+    )
+    refresh_snapshot()
+    assert get_table("notifications").count(user_uid=owner_uid, type="reply") == 1
+    assert get_table("notifications").count(user_uid=owner_uid, type="thread") == 0
+
+    third, _ = _session_comments()
+    third.post(
+        f"{BASE_URL}/comments/create",
+        data={
+            "content": "Another top-level comment",
+            "target_type": "post",
+            "post_uid": post_uid,
+            "target_uid": post_uid,
+        },
+        allow_redirects=False,
+    )
+    refresh_snapshot()
+    assert get_table("notifications").count(user_uid=owner_uid, type="comment") == 1
+    assert get_table("notifications").count(user_uid=owner_uid, type="thread") == 0
+    replier_uid = _db_user(replier_name)["uid"]
+    assert get_table("notifications").count(user_uid=replier_uid, type="thread") == 1
+
+
+def test_post_page_embeds_comment_as_nested_schema(app_server):
+    s, _ = _session_comments()
+    title = f"schema-comment-{int(time.time() * 1000)}"
+    post_uid = _create_post_comments(s, title)
+    slug = get_table("posts").find_one(uid=post_uid)["slug"]
+    unique_comment_text = f"a schema-visible comment {int(time.time() * 1000)}"
+    s.post(
+        f"{BASE_URL}/comments/create",
+        data={
+            "content": unique_comment_text,
+            "target_type": "post",
+            "post_uid": post_uid,
+            "target_uid": post_uid,
+        },
+        allow_redirects=False,
+    )
+    r = requests.get(f"{BASE_URL}/posts/{slug}", allow_redirects=False)
+    assert r.status_code == 200
+    assert '"@type": "Comment"' in r.text
+    assert unique_comment_text in r.text

tests/api/messages/ws.py

@@ -179,3 +179,68 @@ def test_ws_read_marks_and_notifies(app_server):
         )
     )
     assert rows, "messages should be marked read after the read frame"
+
+
+def test_ws_sync_replays_messages_since(app_server):
+    sender_session, _ = _member()
+    receiver_session, receiver_name = _member()
+    receiver_uid = _db_user(receiver_name)["uid"]
+    since = "1970-01-01T00:00:00+00:00"
+    sender_session.post(
+        f"{BASE_URL}/messages/send",
+        headers={"Accept": "application/json"},
+        data={"content": "catch me up", "receiver_uid": receiver_uid},
+    )
+
+    async def run():
+        async with websockets.connect(
+            WS_URL, additional_headers=_cookie_header(receiver_session)
+        ) as receiver_ws:
+            await _recv_until(receiver_ws, "ready")
+            await receiver_ws.send(
+                json.dumps({"type": "sync", "since": since, "with_uid": ""})
+            )
+            frame = await _recv_until(receiver_ws, "message")
+            assert frame["content"] == "catch me up"
+            assert frame["receiver_uid"] == receiver_uid
+
+    asyncio.run(run())
+
+
+def test_ws_ping_and_second_send_keep_the_socket_open(app_server):
+    sender_session, _ = _member()
+    receiver_session, receiver_name = _member()
+    receiver_uid = _db_user(receiver_name)["uid"]
+
+    async def run():
+        async with websockets.connect(
+            WS_URL, additional_headers=_cookie_header(sender_session)
+        ) as sender_ws:
+            await _recv_until(sender_ws, "ready")
+            await sender_ws.send(
+                json.dumps(
+                    {
+                        "type": "send",
+                        "receiver_uid": receiver_uid,
+                        "content": "hello over the wire",
+                        "client_id": "after-error",
+                    }
+                )
+            )
+            echo = await _recv_until(sender_ws, "message")
+            assert echo["client_id"] == "after-error"
+            await sender_ws.send(json.dumps({"type": "ping"}))
+            await sender_ws.send(
+                json.dumps(
+                    {
+                        "type": "send",
+                        "receiver_uid": receiver_uid,
+                        "content": "still open",
+                        "client_id": "still-open",
+                    }
+                )
+            )
+            second = await _recv_until(sender_ws, "message")
+            assert second["client_id"] == "still-open"
+
+    asyncio.run(run())

tests/api/pushjson.py

@@ -119,3 +119,17 @@ def test_register_non_object_body_rejected(app_server):
     s = _session_push()
     r = s.post(f"{BASE_URL}/push.json", json=["nope"])
     assert r.status_code == 400
+
+
+def test_register_ignores_client_id_on_webpush(app_server):
+    s = _session_push()
+    r = s.post(
+        f"{BASE_URL}/push.json",
+        json={
+            "endpoint": "https://push.example.com/with-client",
+            "keys": {"p256dh": "p256dh_fake", "auth": "auth_fake"},
+            "client_id": "browser-tab",
+        },
+    )
+    assert r.status_code == 200, r.text
+    assert r.json().get("registered") is True

tests/api/sitemapxml.py

@@ -267,6 +267,13 @@ def test_sitemap_headers_and_static_entries(app_server):
     assert f"{BASE_URL}/gists" in r.text
 
 
+def test_sitemap_lists_the_topics_hub_and_every_topic(app_server):
+    r = requests.get(f"{BASE_URL}/sitemap.xml")
+    assert f"{BASE_URL}/topics</loc>" in r.text
+    for topic in ("devlog", "showcase", "question", "rant", "fun", "random", "politics"):
+        assert f"{BASE_URL}/topics/{topic}</loc>" in r.text
+
+
 def test_sitemap_lists_the_public_moderation_surfaces(app_server):
     xml = requests.get(f"{BASE_URL}/sitemap.xml").text
     assert f"<loc>{BASE_URL}/reports/reasons</loc>" in xml

tests/api/tools/deepsearch/session.py

@@ -34,6 +34,7 @@ def _seed_done_session(owner_id="ds-session-owner"):
             {"title": "Finding one", "detail": "Detail.", "confidence": 0.8, "citations": [1]}
         ],
         "sources": [{"url": "https://example.com", "title": "Example", "source": "httpx"}],
+        "follow_up_questions": ["What is a follow-up question?"],
         "score": 70,
         "confidence": 0.7,
         "source_diversity": 0.5,
@@ -95,6 +96,7 @@ def test_session_json_shape(app_server):
         assert body["export_md_url"] == f"/tools/deepsearch/{uid}/export.md"
         assert body["findings"]
         assert body["sources"]
+        assert body["follow_up_questions"] == ["What is a follow-up question?"]
         assert "viewer_is_admin" in body
         assert "viewer_owns" in body
     finally:
@@ -108,6 +110,8 @@ def test_session_html_renders_without_jinja_global_collision(app_server):
         assert r.status_code == 200, r.text
         assert "the question" in r.text
         assert "dp-deepsearch-chat" in r.text
+        assert "What is a follow-up question?" in r.text
+        assert "data-followup" in r.text
     finally:
         _clear()
 

tests/api/topics.py

@@ -0,0 +1,152 @@
+# retoor <retoor@molodetz.nl>
+
+import time
+from datetime import datetime, timedelta, timezone
+import requests
+from tests.conftest import BASE_URL
+from devplacepy.database import get_table, refresh_snapshot
+from devplacepy.database.pagination import PAGE_SIZE
+from devplacepy.utils import generate_uid
+
+JSON_topics = {"Accept": "application/json"}
+_counter_topics = [0]
+
+
+def _session_topics():
+    _counter_topics[0] += 1
+    name = f"tpc{int(time.time() * 1000)}{_counter_topics[0]}"
+    s = requests.Session()
+    s.post(
+        f"{BASE_URL}/auth/signup",
+        data={
+            "username": name,
+            "email": f"{name}@t.dev",
+            "password": "secret123",
+            "confirm_password": "secret123",
+            "birth_date": "1990-01-01",
+            "accept_terms": "1",
+        },
+        allow_redirects=True,
+    )
+    return s, name
+
+
+def _create_post_topics(session, title, topic):
+    r = session.post(
+        f"{BASE_URL}/posts/create",
+        data={"content": f"Post body for {title}", "title": title, "topic": topic},
+        allow_redirects=False,
+    )
+    return r.headers["location"].split("/posts/")[-1]
+
+
+def _create_post_direct_topics(user_uid, topic, order, marker=None):
+    uid = generate_uid()
+    marker = marker or f"tpc-{uid[:8]}"
+    get_table("posts").insert(
+        {
+            "deleted_at": None,
+            "deleted_by": None,
+            "uid": uid,
+            "user_uid": user_uid,
+            "slug": f"{uid[:8]}-{topic}-post",
+            "title": marker,
+            "content": f"Direct topic post content {order}",
+            "topic": topic,
+            "project_uid": None,
+            "image": None,
+            "stars": 0,
+            "created_at": (
+                datetime.now(timezone.utc) + timedelta(seconds=order)
+            ).isoformat(),
+        }
+    )
+    return uid, marker
+
+
+def _topic_hub_entry(topic):
+    r = requests.get(f"{BASE_URL}/topics", headers=JSON_topics)
+    data = r.json()
+    return next(t for t in data["topics"] if t["key"] == topic)
+
+
+def test_topics_hub_lists_every_topic(app_server):
+    r = requests.get(f"{BASE_URL}/topics", headers=JSON_topics)
+    assert r.status_code == 200
+    data = r.json()
+    keys = {t["key"] for t in data["topics"]}
+    assert keys == {"devlog", "showcase", "question", "rant", "fun", "random", "politics"}
+
+
+def test_topics_hub_post_count_reflects_new_posts(app_server):
+    s, name = _session_topics()
+    refresh_snapshot()
+    user_uid = get_table("users").find_one(username=name)["uid"]
+    before = _topic_hub_entry("rant")["post_count"]
+    for i in range(3):
+        _create_post_direct_topics(user_uid, "rant", i)
+    refresh_snapshot()
+    after = _topic_hub_entry("rant")["post_count"]
+    assert after - before == 3
+
+
+def test_topic_page_lists_only_that_topics_posts(app_server):
+    s, _ = _session_topics()
+    unique = int(time.time() * 1000)
+    devlog_title = f"devlog-only-{unique}"
+    showcase_title = f"showcase-only-{unique}"
+    _create_post_topics(s, devlog_title, "devlog")
+    _create_post_topics(s, showcase_title, "showcase")
+
+    r = requests.get(f"{BASE_URL}/topics/devlog", headers=JSON_topics)
+    assert r.status_code == 200
+    data = r.json()
+    assert data["topic"] == "devlog"
+    titles = [item["post"]["title"] for item in data["posts"]]
+    assert devlog_title in titles
+    assert showcase_title not in titles
+
+
+def test_topic_page_rejects_an_unknown_topic(app_server):
+    r = requests.get(f"{BASE_URL}/topics/not-a-real-topic", allow_redirects=False)
+    assert r.status_code == 404
+
+
+def test_topic_page_canonical_and_breadcrumbs_are_topic_specific(app_server):
+    r = requests.get(f"{BASE_URL}/topics/showcase", allow_redirects=False)
+    assert r.status_code == 200
+    assert 'href="' in r.text
+    assert "/topics/showcase" in r.text
+    assert "Topics" in r.text
+
+
+def test_topic_page_pagination_crosses_a_page_boundary(app_server):
+    s, name = _session_topics()
+    refresh_snapshot()
+    user_uid = get_table("users").find_one(username=name)["uid"]
+    count = PAGE_SIZE + 1
+    markers = []
+    for i in range(count):
+        _, marker = _create_post_direct_topics(user_uid, "fun", i, marker=f"tpcpag-{i}")
+        markers.append(marker)
+    refresh_snapshot()
+
+    r = requests.get(f"{BASE_URL}/topics/fun", headers=JSON_topics)
+    assert r.status_code == 200
+    body = r.json()
+    assert len(body["posts"]) == PAGE_SIZE
+    assert body["next_cursor"] is not None
+
+    r2 = requests.get(
+        f"{BASE_URL}/topics/fun",
+        headers=JSON_topics,
+        params={"before": body["next_cursor"]},
+    )
+    assert r2.status_code == 200
+    body2 = r2.json()
+    assert len(body2["posts"]) >= 1
+
+    seen_titles = {item["post"]["title"] for item in body["posts"]} | {
+        item["post"]["title"] for item in body2["posts"]
+    }
+    assert set(markers).issubset(seen_titles)

tests/e2e/messages.py

@@ -382,3 +382,27 @@ def test_mobile_single_pane_back_button(mobile_page):
     expect(conv_list).to_be_visible()
     expect(thread).to_be_hidden()
     assert page.url == f"{BASE_URL}/messages"
+
+
+def test_live_incoming_message_shows_report_button(alice, bob):
+    page_a, user_a = alice
+    page_b, user_b = bob
+    uid_a = get_table("users").find_one(username=user_a["username"])["uid"]
+    uid_b = get_table("users").find_one(username=user_b["username"])["uid"]
+
+    page_a.goto(
+        f"{BASE_URL}/messages?with_uid={uid_b}", wait_until="domcontentloaded"
+    )
+    page_b.goto(
+        f"{BASE_URL}/messages?with_uid={uid_a}", wait_until="domcontentloaded"
+    )
+
+    msg = f"Flag me {int(time.time() * 1000)}"
+    textarea = page_a.locator(".messages-input-area textarea[name='content']")
+    textarea.wait_for(state="visible")
+    textarea.fill(msg)
+    page_a.locator(".messages-send-btn").click()
+
+    bubble = page_b.locator(f".message-bubble.theirs:has-text('{msg}')")
+    bubble.wait_for(state="visible", timeout=10000)
+    expect(bubble.locator(".message-report-btn")).to_be_visible()

tests/unit/database/schema.py

@@ -3,7 +3,7 @@
 from devplacepy.database import get_table, init_db
 
 FILTERED_COLUMNS = {
-    "messages": ("uid", "sender_uid", "receiver_uid", "content", "read", "created_at"),
+    "messages": ("uid", "sender_uid", "receiver_uid", "content", "read", "created_at", "updated_at"),
     "workspace_quota_rules": ("uid", "owner_kind", "owner_id", "deleted_at"),
     "workspace_editor_prefs": ("uid", "owner_kind", "owner_id", "deleted_at"),
 }

tests/unit/push.py

@@ -302,6 +302,91 @@ def test_apns_delivery_maps_statuses(monkeypatch):
     assert throttled.status == providers.REJECTED
 
 
+def test_apns_dead_reasons_excludes_topic_misconfiguration(monkeypatch):
+    from devplacepy.push import providers
+
+    not_for_topic, _ = _apns_response_status(
+        monkeypatch, 400, {"reason": "DeviceTokenNotForTopic"}
+    )
+    assert not_for_topic.status == providers.REJECTED
+
+    disallowed, _ = _apns_response_status(monkeypatch, 400, {"reason": "TopicDisallowed"})
+    assert disallowed.status == providers.REJECTED
+
+    expired, _ = _apns_response_status(monkeypatch, 410, {"reason": "ExpiredToken"})
+    assert expired.status == providers.DEAD
+
+
+def test_apns_dead_carries_apns_confirmation_timestamp(monkeypatch):
+    from datetime import datetime, timezone
+    from devplacepy.push import providers
+
+    gone, _ = _apns_response_status(
+        monkeypatch, 410, {"reason": "Unregistered", "timestamp": 1700000000000}
+    )
+    assert gone.status == providers.DEAD
+    assert gone.dead_before == datetime.fromtimestamp(
+        1700000000000 / 1000, tz=timezone.utc
+    ).isoformat()
+
+    bad_token, _ = _apns_response_status(monkeypatch, 400, {"reason": "BadDeviceToken"})
+    assert bad_token.status == providers.DEAD
+    assert bad_token.dead_before is None
+
+
+def test_apns_auth_failure_invalidates_cached_token(monkeypatch, local_db):
+    import httpx
+    from tests.conftest import run_async
+    from devplacepy.push import providers
+    from devplacepy.push.providers import apns
+
+    _apns_settings(
+        monkeypatch,
+        **{
+            apns.TEAM_ID_KEY: "TEAMID1234",
+            apns.KEY_ID_KEY: "KEYID12345",
+            apns.AUTH_KEY_KEY: _ec_private_key_pem(),
+            apns.TOPIC_KEY: "nl.molodetz.devplace",
+        },
+    )
+    provider = providers.PROVIDERS["apns"]
+    pem = _ec_private_key_pem()
+    first = apns.provider_token("TEAMID1234", "KEYID12345", pem)
+    assert apns._token_state["current"]["token"] == first
+    assert apns._read_shared_token()["token"] == first
+
+    def handler(request):
+        return httpx.Response(403, json={"reason": "InvalidProviderToken"})
+
+    async def run():
+        transport = httpx.MockTransport(handler)
+        async with httpx.AsyncClient(transport=transport) as client:
+            return await provider.deliver(
+                client, {"token": "a" * 64}, provider.prepare({"message": "hi"})
+            )
+
+    outcome = run_async(run())
+    assert outcome.status == providers.REJECTED
+    assert "current" not in apns._token_state
+    assert apns._read_shared_token() is None
+
+    second = apns.provider_token("TEAMID1234", "KEYID12345", pem)
+    assert second != first
+
+
+def test_apns_provider_token_is_shared_across_workers(monkeypatch, local_db):
+    from devplacepy.push.providers import apns
+
+    _apns_settings(monkeypatch)
+    pem = _ec_private_key_pem()
+    token = apns.provider_token("TEAMID1234", "KEYID12345", pem)
+
+    apns._token_state.clear()
+    reused = apns.provider_token("TEAMID1234", "KEYID12345", pem)
+    assert reused == token
+    assert apns._token_state["current"]["token"] == token
+
+
 def test_apns_delivery_without_configuration_never_raises(monkeypatch):
     import httpx
     from tests.conftest import run_async
@@ -345,3 +430,271 @@ def test_delivery_timeout_is_clamped(monkeypatch):
 
     monkeypatch.setattr(delivery, "get_int_setting", lambda key, default: 100000)
     assert delivery.timeout_seconds() == float(delivery.MAX_TIMEOUT_SECONDS)
+
+
+def test_apns_parse_registration_accepts_client_id_and_normalizes_token():
+    from devplacepy.push import providers
+
+    apns = providers.PROVIDERS["apns"]
+    token = "A1B2C3D4" * 8
+    assert apns.parse_registration(
+        {"token": f"<{token[:8]} {token[8:]}>", "client_id": " device-1 "}
+    ) == {"token": token.lower(), "client_id": "device-1"}
+    assert apns.parse_registration({"token": token, "client_id": ""}) == {
+        "token": token.lower()
+    }
+    assert apns.parse_registration({"token": token, "client_id": {"uid": "x"}}) is None
+    assert apns.parse_registration({"token": token, "client_id": "x" * 200}) is None
+    assert apns.parse_registration({"client_id": "device-1"}) is None
+
+
+def test_apns_client_config_advertises_environment(monkeypatch):
+    from devplacepy.push import providers
+    from devplacepy.push.providers import apns
+
+    _apns_settings(monkeypatch, **{apns.ENVIRONMENT_KEY: "sandbox"})
+    assert providers.PROVIDERS["apns"].client_config() == {"environment": "sandbox"}
+    _apns_settings(monkeypatch, **{apns.ENVIRONMENT_KEY: "nonsense"})
+    assert providers.PROVIDERS["apns"].client_config() == {"environment": "production"}
+
+
+def test_apns_stamp_registration_sets_environment(monkeypatch):
+    from devplacepy.push import providers
+    from devplacepy.push.providers import apns
+
+    _apns_settings(monkeypatch, **{apns.ENVIRONMENT_KEY: "sandbox"})
+    stamped = providers.PROVIDERS["apns"].stamp_registration({"token": "a" * 64})
+    assert stamped["environment"] == "sandbox"
+
+
+def test_apns_delivery_uses_registration_environment(monkeypatch):
+    from devplacepy.push import providers
+    from devplacepy.push.providers import apns
+
+    _apns_settings(
+        monkeypatch,
+        **{
+            apns.TEAM_ID_KEY: "TEAMID1234",
+            apns.KEY_ID_KEY: "KEYID12345",
+            apns.AUTH_KEY_KEY: _ec_private_key_pem(),
+            apns.TOPIC_KEY: "nl.molodetz.devplace",
+            apns.ENVIRONMENT_KEY: "production",
+        },
+    )
+    seen = {}
+
+    def handler(request):
+        seen["url"] = str(request.url)
+        return __import__("httpx").Response(200, json={})
+
+    async def run():
+        import httpx
+
+        transport = httpx.MockTransport(handler)
+        async with httpx.AsyncClient(transport=transport) as client:
+            return await providers.PROVIDERS["apns"].deliver(
+                client,
+                {"token": "a" * 64, "environment": "sandbox"},
+                "{}",
+            )
+
+    from tests.conftest import run_async
+
+    outcome = run_async(run())
+    assert outcome.status == providers.ACCEPTED
+    assert seen["url"].startswith("https://api.sandbox.push.apple.com/3/device/")
+
+
+def test_apns_delivery_client_is_plain_http2():
+    import httpx
+    from devplacepy.curl_transport import CurlTransport
+    from devplacepy.push.providers import apns
+    from tests.conftest import run_async
+
+    client = apns.gateway_client(5.0)
+    assert isinstance(client, httpx.AsyncClient)
+    assert not isinstance(client._transport, CurlTransport)
+    run_async(client.aclose())
+
+
+def test_apns_delivery_client_persists_across_calls():
+    from devplacepy.push import providers
+    from devplacepy.push.providers import apns
+    from tests.conftest import run_async
+
+    provider = providers.PROVIDERS["apns"]
+    assert provider.closes_delivery_client() is False
+    run_async(apns.close_client())
+    try:
+        first = provider.delivery_client(5.0)
+        second = provider.delivery_client(5.0)
+        assert first is second
+        assert not first.is_closed
+        run_async(apns.close_client())
+        assert first.is_closed
+        third = provider.delivery_client(5.0)
+        assert third is not first
+    finally:
+        run_async(apns.close_client())
+
+
+def test_webpush_closes_delivery_client_by_default():
+    from devplacepy.push import providers
+
+    provider = providers.PROVIDERS["webpush"]
+    assert provider.closes_delivery_client() is True
+
+
+def test_store_upserts_by_client_id_and_revives_dead_tokens(local_db):
+    from devplacepy.push import store
+
+    user_uid = "user-apns-1"
+    token = "a" * 64
+    write = store.register(user_uid, "apns", {"token": token, "client_id": "phone"})
+    assert write.created is True
+    assert write.revived is False
+    assert write.probe is True
+
+    same = store.register(user_uid, "apns", {"token": token, "client_id": "phone"})
+    assert same.created is False
+    assert same.revived is False
+    assert same.record["uid"] == write.record["uid"]
+
+    store.mark_dead(write.record["id"])
+    revived = store.register(user_uid, "apns", {"token": token, "client_id": "phone"})
+    assert revived.created is False
+    assert revived.revived is True
+    assert revived.record["deleted_at"] is None
+    assert len(store.active_for_user(user_uid)) == 1
+
+    rotated = store.register(
+        user_uid, "apns", {"token": "b" * 64, "client_id": "phone"}
+    )
+    assert rotated.created is False
+    assert rotated.record["token"] == "b" * 64
+    assert rotated.record["uid"] == write.record["uid"]
+    assert len(store.active_for_user(user_uid)) == 1
+
+
+def test_store_revives_token_only_registration(local_db):
+    from devplacepy.push import store
+
+    user_uid = "user-apns-2"
+    token = "c" * 64
+    write = store.register(user_uid, "apns", {"token": token})
+    store.mark_dead(write.record["id"])
+    revived = store.register(user_uid, "apns", {"token": token, "client_id": "phone-2"})
+    assert revived.revived is True
+    assert revived.record["client_id"] == "phone-2"
+    assert len(store.active_for_user(user_uid)) == 1
+
+
+def test_store_attaches_client_id_to_existing_token_row(local_db):
+    from devplacepy.push import store
+
+    user_uid = "user-apns-3"
+    token = "d" * 64
+    first = store.register(user_uid, "apns", {"token": token})
+    second = store.register(user_uid, "apns", {"token": token, "client_id": "phone-3"})
+    assert second.created is False
+    assert second.record["uid"] == first.record["uid"]
+    assert second.record["client_id"] == "phone-3"
+
+
+def test_store_mark_dead_skips_a_registration_revived_after_confirmation(local_db):
+    from datetime import datetime, timedelta, timezone
+    from devplacepy.push import store
+
+    user_uid = "user-apns-4"
+    token = "f" * 64
+    write = store.register(user_uid, "apns", {"token": token})
+    registered_at = write.record["registered_at"]
+    assert registered_at
+
+    stale_confirmation = (
+        datetime.fromisoformat(registered_at) - timedelta(minutes=5)
+    ).isoformat()
+    store.mark_dead(write.record["id"], stale_confirmation)
+    assert len(store.active_for_user(user_uid)) == 1
+
+    fresh_confirmation = (
+        datetime.now(timezone.utc) + timedelta(minutes=5)
+    ).isoformat()
+    store.mark_dead(write.record["id"], fresh_confirmation)
+    assert store.active_for_user(user_uid) == []
+
+
+def test_dead_delivery_logs_the_reason(monkeypatch, caplog):
+    import logging
+    import httpx
+    from tests.conftest import run_async
+    from devplacepy.push import providers
+    from devplacepy.push.delivery import _deliver_one
+
+    class _FakeStore:
+        def __init__(self):
+            self.dead = []
+
+        def mark_dead(self, registration_id, dead_before=None):
+            self.dead.append(registration_id)
+
+    fake = _FakeStore()
+    monkeypatch.setattr("devplacepy.push.delivery.store", fake)
+
+    async def dead_deliver(client, registration, prepared):
+        return providers.Delivery(providers.DEAD, "400 BadDeviceToken")
+
+    provider = providers.PROVIDERS["apns"]
+    monkeypatch.setattr(provider, "deliver", dead_deliver)
+
+    async def run():
+        async with httpx.AsyncClient() as client:
+            return await _deliver_one(
+                provider, client, {"id": 9, "token": "a" * 64}, "{}", "user-x"
+            )
+
+    with caplog.at_level(logging.WARNING):
+        outcome = run_async(run())
+    assert outcome.status == providers.DEAD
+    assert fake.dead == [9]
+    assert "400 BadDeviceToken" in caplog.text
+
+
+def test_notify_user_opens_apns_gateway_not_stealth(monkeypatch, local_db):
+    import httpx
+    from tests.conftest import run_async
+    from devplacepy.push import store
+    from devplacepy.push.delivery import notify_user
+    from devplacepy.push.providers import apns
+
+    _apns_settings(
+        monkeypatch,
+        **{
+            apns.TEAM_ID_KEY: "TEAMID1234",
+            apns.KEY_ID_KEY: "KEYID12345",
+            apns.AUTH_KEY_KEY: _ec_private_key_pem(),
+            apns.TOPIC_KEY: "nl.molodetz.devplace",
+        },
+    )
+    store.register("user-gw", "apns", {"token": "e" * 64, "environment": "production"})
+    opened = []
+
+    def fake_gateway(timeout):
+        opened.append(timeout)
+        return httpx.AsyncClient(
+            transport=httpx.MockTransport(lambda request: httpx.Response(200, json={}))
+        )
+
+    def fail_stealth(**kwargs):
+        raise AssertionError("APNs must not use stealth")
+
+    monkeypatch.setattr(apns, "gateway_client", fake_gateway)
+    monkeypatch.setattr(
+        "devplacepy.stealth.stealth_async_client", fail_stealth
+    )
+    run_async(apns.close_client())
+    try:
+        run_async(notify_user("user-gw", {"title": "t", "message": "m"}))
+    finally:
+        run_async(apns.close_client())
+    assert opened

tests/unit/seo.py

@@ -150,3 +150,69 @@ def test_base_seo_context_consumes_ready_metadata(monkeypatch):
     assert "Generated Title" in ctx["page_title"]
     assert ctx["meta_description"] == "Generated clean description"
     assert ctx["meta_keywords"] == "generated, metadata"
+
+
+def _comment_item(uid, content, children=None, author="alice"):
+    return {
+        "comment": {"uid": uid, "content": content, "created_at": "2026-01-01T00:00:00+00:00"},
+        "author": {"username": author},
+        "children": children or [],
+    }
+
+
+def test_comment_schema_shapes_a_comment_type():
+    item = _comment_item("c1", "Nice post, thanks!")
+    schema = seo.comment_schema(item, "https://x.test")
+    assert schema["@type"] == "Comment"
+    assert schema["text"] == "Nice post, thanks!"
+    assert schema["author"] == {
+        "@type": "Person",
+        "name": "alice",
+        "url": "https://x.test/profile/alice",
+    }
+    assert schema["datePublished"] == "2026-01-01T00:00:00+00:00"
+
+
+def test_comment_schema_handles_a_missing_author():
+    item = _comment_item("c1", "text", author=None)
+    item["author"] = None
+    schema = seo.comment_schema(item, "https://x.test")
+    assert schema["author"] == {"@type": "Person", "name": "Unknown", "url": ""}
+
+
+def test_comment_schema_list_flattens_replies_depth_first():
+    reply = _comment_item("c2", "a reply")
+    top = _comment_item("c1", "a top-level comment", children=[reply])
+    flat = seo.comment_schema_list([top], "https://x.test")
+    assert [c["text"] for c in flat] == ["a top-level comment", "a reply"]
+
+
+def test_comment_schema_list_respects_the_limit():
+    tree = [_comment_item(f"c{i}", f"comment {i}") for i in range(30)]
+    flat = seo.comment_schema_list(tree, "https://x.test", limit=5)
+    assert len(flat) == 5
+
+
+def test_discussion_forum_posting_omits_comment_key_without_comments():
+    schema = seo.discussion_forum_posting(
+        {"uid": "p1", "title": "T", "content": "body", "created_at": "2026-01-01"},
+        {"username": "alice"},
+        0,
+        0,
+        "https://x.test",
+    )
+    assert "comment" not in schema
+
+
+def test_discussion_forum_posting_embeds_nested_comments():
+    top = _comment_item("c1", "a top-level comment")
+    schema = seo.discussion_forum_posting(
+        {"uid": "p1", "title": "T", "content": "body", "created_at": "2026-01-01"},
+        {"username": "alice"},
+        1,
+        0,
+        "https://x.test",
+        comments=seo.comment_schema_list([top], "https://x.test"),
+    )
+    assert schema["comment"][0]["@type"] == "Comment"
+    assert schema["comment"][0]["text"] == "a top-level comment"

tests/unit/services/deepsearch/store.py

@@ -64,6 +64,22 @@ def test_dims_reflects_embedding_width():
         store.drop()
 
 
+def test_hybrid_search_results_carry_embedding_vectors():
+    store = VectorStore("ds_store_test_embeddings")
+    try:
+        chunks = _chunks()
+        vectors = local_embed([c.text for c in chunks]).vectors
+        run_async(store.add(chunks, vectors))
+        query = "where was the transistor invented"
+        query_vector = local_embed([query]).vectors[0]
+        results = run_async(store.hybrid_search(query, query_vector, top_k=3))
+        assert results
+        assert all(result.embedding for result in results)
+        assert all(len(result.embedding) == len(vectors[0]) for result in results)
+    finally:
+        store.drop()
+
+
 def test_coverage_analytics_empty_collection():
     store = VectorStore("ds_store_test_cov_empty")
     try:

tests/unit/services/game/economy.py

@@ -393,8 +393,27 @@ def test_unlocked_crops_gates_on_mastery_and_level():
 
 def test_crop_payload_locked_by_mastery():
     distsys = economy.crop_for("distsys")
-    assert economy.crop_payload(distsys, 1, economy.MAX_LEVEL, mastery_earned=0)["locked"] is True
-    assert economy.crop_payload(distsys, 1, economy.MAX_LEVEL, mastery_earned=1)["locked"] is False
+    locked = economy.crop_payload(distsys, 1, economy.MAX_LEVEL, mastery_earned=0)
+    assert locked["locked"] is True
+    assert locked["locked_reason"] == "mastery"
+    assert "Mastery" in locked["locked_text"]
+    unlocked = economy.crop_payload(distsys, 1, economy.MAX_LEVEL, mastery_earned=1)
+    assert unlocked["locked"] is False
+    assert unlocked["locked_reason"] == ""
+
+
+def test_crop_payload_locked_by_level_reason():
+    rust = economy.crop_for("rust")
+    locked = economy.crop_payload(rust, 1, 1)
+    assert locked["locked_reason"] == "level"
+    assert str(rust.min_level) in locked["locked_text"]
+
+
+def test_crop_lock_reason_matches_plant_enforcement():
+    distsys = economy.crop_for("distsys")
+    assert economy.crop_lock_reason(distsys, economy.MAX_LEVEL, 0) == ("mastery", "unlocks after reaching Mastery (Refactor to prestige 50)")
+    assert economy.crop_lock_reason(distsys, economy.MAX_LEVEL, 1) is None
+    assert economy.crop_lock_reason(distsys, 1, 1)[0] == "level"
 
 
 def test_secfort_crop_is_steal_immune():

tests/unit/services/jobs/deepsearch/enhance.py

@@ -0,0 +1,90 @@
+# retoor <retoor@molodetz.nl>
+
+import json
+
+from tests.conftest import run_async
+
+from devplacepy.services.jobs.deepsearch import enhance as enhance_module
+from devplacepy.services.jobs.deepsearch.enhance import plan_followup_queries
+
+
+class _FakeResponse:
+    def __init__(self, status_code, payload):
+        self.status_code = status_code
+        self._payload = payload
+
+    def json(self):
+        return self._payload
+
+
+class _FakeClient:
+    def __init__(self, response):
+        self._response = response
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, *exc):
+        return False
+
+    async def post(self, *args, **kwargs):
+        return self._response
+
+
+def _client_returning(response):
+    def factory(**kwargs):
+        return _FakeClient(response)
+
+    return factory
+
+
+def test_plan_followup_queries_empty_without_covered_titles():
+    result = run_async(plan_followup_queries("q", [], "k"))
+    assert result == []
+
+
+def test_plan_followup_queries_parses_gateway_response(monkeypatch):
+    payload = {"choices": [{"message": {"content": json.dumps({"queries": ["a", "b"]})}}]}
+    monkeypatch.setattr(
+        enhance_module.stealth,
+        "stealth_async_client",
+        _client_returning(_FakeResponse(200, payload)),
+    )
+    result = run_async(plan_followup_queries("q", ["Existing source"], "k"))
+    assert result == ["a", "b"]
+
+
+def test_plan_followup_queries_empty_when_sources_already_cover_question(monkeypatch):
+    payload = {"choices": [{"message": {"content": json.dumps({"queries": []})}}]}
+    monkeypatch.setattr(
+        enhance_module.stealth,
+        "stealth_async_client",
+        _client_returning(_FakeResponse(200, payload)),
+    )
+    result = run_async(plan_followup_queries("q", ["Existing source"], "k"))
+    assert result == []
+
+
+def test_plan_followup_queries_fails_soft_on_gateway_error(monkeypatch):
+    monkeypatch.setattr(
+        enhance_module.stealth,
+        "stealth_async_client",
+        _client_returning(_FakeResponse(500, {})),
+    )
+    result = run_async(plan_followup_queries("q", ["Existing source"], "k"))
+    assert result == []
+
+
+def test_plan_followup_queries_caps_at_max(monkeypatch):
+    payload = {
+        "choices": [
+            {"message": {"content": json.dumps({"queries": ["a", "b", "c", "d", "e"]})}}
+        ]
+    }
+    monkeypatch.setattr(
+        enhance_module.stealth,
+        "stealth_async_client",
+        _client_returning(_FakeResponse(200, payload)),
+    )
+    result = run_async(plan_followup_queries("q", ["Existing source"], "k"))
+    assert len(result) == enhance_module.MAX_FOLLOWUP_QUERIES

tests/unit/services/jobs/deepsearch/orchestrate.py

@@ -4,9 +4,16 @@ import json
 
 from tests.conftest import run_async
 
+from devplacepy.services.deepsearch.store import Chunk
 from devplacepy.services.jobs.deepsearch import orchestrate as orchestrate_module
 from devplacepy.services.jobs.deepsearch.crawl import CrawledPage
-from devplacepy.services.jobs.deepsearch.orchestrate import Orchestration, orchestrate, source_diversity
+from devplacepy.services.jobs.deepsearch.orchestrate import (
+    Orchestration,
+    _cosine,
+    _mmr_select,
+    orchestrate,
+    source_diversity,
+)
 
 
 def _page(url, text="content " * 30):
@@ -149,6 +156,67 @@ def test_linker_receives_full_source_list(monkeypatch):
         assert f"[{n}]" in seen["linker"]
 
 
+def test_cosine_identical_vectors_is_one():
+    assert round(_cosine([1.0, 0.0], [1.0, 0.0]), 6) == 1.0
+
+
+def test_cosine_orthogonal_vectors_is_zero():
+    assert _cosine([1.0, 0.0], [0.0, 1.0]) == 0.0
+
+
+def test_cosine_mismatched_or_empty_is_zero():
+    assert _cosine([], [1.0]) == 0.0
+    assert _cosine([1.0], [1.0, 0.0]) == 0.0
+
+
+def test_mmr_select_prefers_diverse_over_redundant():
+    query_vector = [1.0, 0.0, 0.0]
+    most_relevant = Chunk(uid="a", text="a", url="https://a", title="A", embedding=[0.9, 0.436, 0.0])
+    near_duplicate = Chunk(uid="b", text="b", url="https://a2", title="B", embedding=[0.85, 0.527, 0.0])
+    diverse = Chunk(uid="c", text="c", url="https://c", title="C", embedding=[0.85, 0.0, 0.527])
+    selected = _mmr_select([most_relevant, near_duplicate, diverse], query_vector, limit=2)
+    assert {chunk.uid for chunk in selected} == {"a", "c"}
+
+
+def test_mmr_select_falls_back_when_embeddings_missing():
+    chunks = [Chunk(uid="a", text="a", url="https://a", title="A")]
+    assert _mmr_select(chunks, [1.0, 0.0], limit=1) == chunks
+
+
+def test_mmr_select_empty_input():
+    assert _mmr_select([], [1.0, 0.0], limit=3) == []
+
+
+def test_orchestrate_grounded_run_includes_follow_up_questions(monkeypatch):
+    replies = iter(
+        [
+            "## Answer\nA grounded answer [1].",
+            json.dumps(
+                {
+                    "findings": [
+                        {"title": "Finding", "detail": "Detail", "confidence": 0.7, "citations": [1]}
+                    ]
+                }
+            ),
+            json.dumps({"confidence": 0.8}),
+            json.dumps({"questions": ["What about X?", "How does Y compare?"]}),
+        ]
+    )
+
+    async def fake_request_completion(messages, api_key, **kwargs):
+        return ({"choices": [{"message": {"content": next(replies)}}]}, {}, 5)
+
+    monkeypatch.setattr(orchestrate_module, "request_completion", fake_request_completion)
+    pages = [_page("https://a.example"), _page("https://b.example")]
+    result = run_async(orchestrate("question", pages, "k", lambda frame: None))
+    assert result.follow_up_questions == ["What about X?", "How does Y compare?"]
+
+
+def test_orchestrate_heuristic_path_has_no_follow_up_questions():
+    result = run_async(orchestrate("q", [], "k", lambda frame: None))
+    assert result.follow_up_questions == []
+
+
 def test_parse_json_tolerates_fences_and_trailing_garbage():
     from devplacepy.services.jobs.deepsearch.orchestrate import _parse_json
 

tests/unit/services/jobs/deepsearch/worker.py

@@ -20,13 +20,18 @@ def _patch_pipeline(monkeypatch, pages):
     async def fake_search(queries, emit=lambda frame: None):
         return [{"url": page.url, "title": page.title, "description": ""} for page in pages]
 
-    async def fake_crawl(candidates, max_pages, emit, is_cached, should_stop, query="", depth=1):
-        outcome = CrawlOutcome()
+    async def fake_crawl(
+        candidates, max_pages, emit, is_cached, should_stop, query="", depth=1, seen_hashes=None
+    ):
+        outcome = CrawlOutcome(seen_hashes=seen_hashes if seen_hashes is not None else set())
         for page in pages[:max_pages]:
             emit({"type": "page_loaded", "url": page.url, "done": 1, "total": len(pages)})
             outcome.pages.append(page)
         return outcome
 
+    async def fake_plan_followups(query, covered_titles, api_key, emit=lambda frame: None):
+        return []
+
     def fake_embed(texts, api_key):
         return local_embed(texts)
 
@@ -47,6 +52,7 @@ def _patch_pipeline(monkeypatch, pages):
     monkeypatch.setattr(worker_module, "plan_queries", fake_plan)
     monkeypatch.setattr(worker_module, "search_queries", fake_search)
     monkeypatch.setattr(worker_module, "crawl", fake_crawl)
+    monkeypatch.setattr(worker_module, "plan_followup_queries", fake_plan_followups)
     monkeypatch.setattr(worker_module, "embed_texts", fake_embed_async)
     monkeypatch.setattr(worker_module, "orchestrate", fake_orchestrate)
 
@@ -141,6 +147,78 @@ def test_index_chunks_empty_pages_emits_done(monkeypatch):
         store.drop()
 
 
+def test_worker_run_performs_refinement_round_when_budget_remains(monkeypatch):
+    initial_page = CrawledPage(
+        url="https://example.com/a",
+        title="Page A",
+        text="The transistor was invented at Bell Labs. " * 20,
+        source="httpx",
+        status=200,
+    )
+    refined_page = CrawledPage(
+        url="https://other.example/b",
+        title="Page B",
+        text="Semiconductors are made from silicon. " * 20,
+        source="httpx",
+        status=200,
+    )
+    _patch_pipeline(monkeypatch, [initial_page])
+
+    followup_calls = []
+
+    async def fake_plan_followups_once(query, covered_titles, api_key, emit=lambda frame: None):
+        if followup_calls:
+            return []
+        followup_calls.append(covered_titles)
+        return ["a more specific angle"]
+
+    async def fake_search_followup(queries, emit=lambda frame: None):
+        return [{"url": refined_page.url, "title": refined_page.title, "description": ""}]
+
+    async def fake_crawl_refinement(
+        candidates, max_pages, emit, is_cached, should_stop, query="", depth=1, seen_hashes=None
+    ):
+        outcome = CrawlOutcome(seen_hashes=seen_hashes if seen_hashes is not None else set())
+        outcome.pages.append(refined_page)
+        return outcome
+
+    monkeypatch.setattr(worker_module, "plan_followup_queries", fake_plan_followups_once)
+
+    real_search_queries = worker_module.search_queries
+    real_crawl = worker_module.crawl
+    call_count = {"search": 0, "crawl": 0}
+
+    async def routed_search(queries, emit=lambda frame: None):
+        call_count["search"] += 1
+        if call_count["search"] == 1:
+            return await real_search_queries(queries, emit)
+        return await fake_search_followup(queries, emit)
+
+    async def routed_crawl(*args, **kwargs):
+        call_count["crawl"] += 1
+        if call_count["crawl"] == 1:
+            return await real_crawl(*args, **kwargs)
+        return await fake_crawl_refinement(*args, **kwargs)
+
+    monkeypatch.setattr(worker_module, "search_queries", routed_search)
+    monkeypatch.setattr(worker_module, "crawl", routed_crawl)
+
+    with tempfile.TemporaryDirectory() as tmp:
+        output_dir = Path(tmp)
+        payload = {
+            "query": "history of the transistor",
+            "max_pages": 5,
+            "depth": 2,
+            "api_key": "k",
+            "collection": "ds_worker_refine_test",
+            "cached_hashes": [],
+        }
+        report = run_async(worker_module._run(payload, output_dir))
+        assert report["page_count"] == 2
+        assert followup_calls and followup_calls[0] == ["Page A"]
+        VectorStore("ds_worker_refine_test").drop()
+
+
 def test_worker_control_cancel_stops(monkeypatch):
     pages = [
         CrawledPage(url="https://x.example", title="X", text="content " * 40, source="httpx", status=200)

tests/unit/services/messaging/persist.py

@@ -0,0 +1,28 @@
+# retoor <retoor@molodetz.nl>
+
+from datetime import datetime, timezone
+
+from devplacepy.database import get_table
+from devplacepy.utils import generate_uid
+from devplacepy.services.messaging.persist import stamp_content_revision
+
+
+def test_stamp_content_revision_sets_updated_at(local_db):
+    uid = generate_uid()
+    get_table("messages").insert(
+        {
+            "uid": uid,
+            "sender_uid": "s1",
+            "receiver_uid": "r1",
+            "content": "hello",
+            "read": False,
+            "created_at": datetime.now(timezone.utc).isoformat(),
+            "updated_at": None,
+        }
+    )
+    row = stamp_content_revision(uid)
+    assert row is not None
+    assert row["updated_at"]
+    stored = get_table("messages").find_one(uid=uid)
+    assert stored["updated_at"] == row["updated_at"]
+    assert stored["content"] == "hello"