diff --git a/CLAUDE.md b/CLAUDE.md index 203a7e6..a743da9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,7 +19,7 @@ DevPlace is a server-rendered social network for developers. FastAPI backend ser ## Commands ```bash -make install # pip install -e . + playwright install chromium +make install # create .venv if needed, pip install -e ".[dev]", playwright chromium make ppy # build the single shared container image (ppy:latest); run once before launching instances make dev # uvicorn --reload on port 10500, backlog 4096 make prod # uvicorn --workers $(WEB_WORKERS) (defaults to nproc), port 10500 (backlog 8192) @@ -33,6 +33,8 @@ make locust # Locust load test, interactive web UI make locust-headless # Locust CLI mode for CI ``` +Every Python make target (`install`, `dev`, `prod`, `test*`, `coverage*`, `locust*`) uses `.venv/bin/python`. If `.venv` is missing, make creates it from `python3`, installs `-e ".[dev]"`, and installs Playwright Chromium before running the target. `make install` refreshes that environment. `make clean` removes `.venv` as well as stray bytecode. + The Makefile exports `PYTHONDONTWRITEBYTECODE=1` for every recipe, so no `.pyc` files or `__pycache__` directories are written by any `make` target. Keep it that way - do not add a target that re-enables bytecode writing. `make clean` removes any stray bytecode left from running Python outside make. Preliminary validation: confirm `python -c "from devplacepy.main import app"` imports clean and check each touched language manually (Python compiles/imports, JS parses, CSS braces and HTML tags balance). These checks are gates on the way to the real validation, never a substitute for it: **every change ends with the full test suite (`make test` - all three tiers, every test) and it must pass.** @@ -183,6 +185,7 @@ Routers in `devplacepy/routers/` are organised as a **directory tree that mirror |--------|--------| | `/auth` | auth/ package | | `/feed`, `/posts`, `/comments` | flat files | +| `/topics` | topics.py - crawlable per-topic category index pages (`/topics` hub, `/topics/{topic}` listing) | | `/projects`, `/projects/{slug}/files`, `/projects/{slug}/containers` | projects/ package - see `routers/projects/CLAUDE.md` | | `/profile` | profile/ package (customization, notifications, ai-correction, ai-modifier, interactions, telegram, usage) | | `/messages` | messages.py - see `services/messaging/CLAUDE.md` | @@ -282,7 +285,7 @@ The escape hatch is deliberately two-factor and must never be self-served: after - **No comments, no docstrings in source.** Code is self-documenting. - **Forbidden variable name patterns:** `_new`, `_old`, `_temp`, `_v2`, `better_`, `my_`, `the_`. -- **Outbound HTTP goes through the stealth client only.** Never instantiate a bare `httpx.AsyncClient(...)`/`httpx.Client(...)` for outbound traffic - build it via `devplacepy.stealth.stealth_async_client(...)`/`stealth_sync_client(...)`. The async factory routes bytes through **`curl_cffi` (curl-impersonate, BoringSSL)** behind an `httpx` transport adapter, so TLS JA3/JA4 and the HTTP/2 fingerprint match Chrome 146 byte-for-byte; falls back to a pure-httpx Chrome-aligned transport if `curl_cffi` is missing. Applies to news, gateway upstream, Devii/DeepSearch/SEO fetch, gitea, push, attachments. SSRF-guarded fetches via `net_guard.guarded_async_client(...)` get the fingerprint automatically. Sole exception: the reverse-proxy forwarding core `services/containers/forward.py` (and the three routers that call it - `routers/proxy.py`, `routers/tunnel.py`, the workspace editor route), which relays the user's own headers verbatim and therefore keeps a plain `httpx.AsyncClient`; bolting the Chrome identity onto it would overwrite the very headers it exists to forward. **Load-bearing gotcha:** cleartext `http://` is forced to HTTP/1.1 (`curl_transport.http_version_for`) - the Chrome-impersonation profile defaults to HTTP/2, but a plaintext origin has no TLS/ALPN to negotiate h2 and uvicorn (HTTP/1.1 only) returns `400 "Invalid HTTP request received."` once a cleartext HTTP/2 body crosses ~128KB. Every internal AI consumer calls `INTERNAL_GATEWAY_URL` = `http://localhost:/openai/v1/...`, so without this override a large Devii turn (big system prompt + tool list) fails on every request. `https://` keeps the HTTP/2 default. `stealth_sync_client` stays pure-httpx Chrome-aligned (no curl_cffi sync adapter). **Never bolt a hand-written browser-header dict (UA/`sec-ch-ua`/`Sec-Fetch-*`) onto a `stealth_async_client`/`stealth_sync_client` call** - curl_cffi's impersonation already emits an internally-consistent Chrome 146 identity across TLS, UA, and Client Hints; a caller-supplied header set claiming a different Chrome version or platform creates a UA-vs-TLS-vs-Client-Hints mismatch that is a textbook anti-bot tripwire (this was a real bug in Devii's `fetch_url`/`http_request` tool, fixed by deleting its hardcoded Chrome-131 header block). `stealth.detect_consent_gate(html)` recognizes the common JS-redirect cookie-consent wall pattern (DPG Media's `nu.nl` and sibling sites, and generically any page gated behind a `decodeURIComponent(...)` callback referencing a "privacy gate"/`consent.js`) - reuse it wherever a fetched page's real content sits behind that gate rather than special-casing a domain. `stealth_transport`/`stealth_async_client`/`stealth_sync_client` all accept a `proxy=` override, defaulting to `stealth.configured_proxy_url()` (the admin-editable `outbound_proxy_url` site setting, falling back to `DEVPLACE_OUTBOUND_PROXY_URL` when the setting is unset) - this is the only real fix for a destination that blocks by IP/ASN reputation (e.g. reddit.com returns an identical "blocked by network security" page across every TLS/header fingerprint tried, confirmed by testing curl_cffi's full impersonation list - it is blocking the datacenter ASN outright, not fingerprinting the client). +- **Outbound HTTP goes through the stealth client only.** Never instantiate a bare `httpx.AsyncClient(...)`/`httpx.Client(...)` for outbound traffic - build it via `devplacepy.stealth.stealth_async_client(...)`/`stealth_sync_client(...)`. The async factory routes bytes through **`curl_cffi` (curl-impersonate, BoringSSL)** behind an `httpx` transport adapter, so TLS JA3/JA4 and the HTTP/2 fingerprint match Chrome 146 byte-for-byte; falls back to a pure-httpx Chrome-aligned transport if `curl_cffi` is missing. Applies to news, gateway upstream, Devii/DeepSearch/SEO fetch, gitea, web push, attachments. SSRF-guarded fetches via `net_guard.guarded_async_client(...)` get the fingerprint automatically. Two exceptions keep a plain `httpx.AsyncClient`: (1) the reverse-proxy forwarding core `services/containers/forward.py` (and the three routers that call it - `routers/proxy.py`, `routers/tunnel.py`, the workspace editor route), which relays the user's own headers verbatim - bolting the Chrome identity onto it would overwrite the very headers it exists to forward; (2) the APNs provider (`push/providers/apns.py` `gateway_client` / `delivery_client`) which talks to Apple's HTTP/2 provider API with `httpx.AsyncClient(http2=True, trust_env=False)` - Chrome impersonation, HTTP/2 PRIORITY frames, HPACK indexing of `:path`/`authorization`, extra browser headers, and the outbound proxy all break or starve that API. Web Push stays on stealth. **Load-bearing gotcha:** cleartext `http://` is forced to HTTP/1.1 (`curl_transport.http_version_for`) - the Chrome-impersonation profile defaults to HTTP/2, but a plaintext origin has no TLS/ALPN to negotiate h2 and uvicorn (HTTP/1.1 only) returns `400 "Invalid HTTP request received."` once a cleartext HTTP/2 body crosses ~128KB. Every internal AI consumer calls `INTERNAL_GATEWAY_URL` = `http://localhost:/openai/v1/...`, so without this override a large Devii turn (big system prompt + tool list) fails on every request. `https://` keeps the HTTP/2 default. `stealth_sync_client` stays pure-httpx Chrome-aligned (no curl_cffi sync adapter). **Never bolt a hand-written browser-header dict (UA/`sec-ch-ua`/`Sec-Fetch-*`) onto a `stealth_async_client`/`stealth_sync_client` call** - curl_cffi's impersonation already emits an internally-consistent Chrome 146 identity across TLS, UA, and Client Hints; a caller-supplied header set claiming a different Chrome version or platform creates a UA-vs-TLS-vs-Client-Hints mismatch that is a textbook anti-bot tripwire (this was a real bug in Devii's `fetch_url`/`http_request` tool, fixed by deleting its hardcoded Chrome-131 header block). `stealth.detect_consent_gate(html)` recognizes the common JS-redirect cookie-consent wall pattern (DPG Media's `nu.nl` and sibling sites, and generically any page gated behind a `decodeURIComponent(...)` callback referencing a "privacy gate"/`consent.js`) - reuse it wherever a fetched page's real content sits behind that gate rather than special-casing a domain. `stealth_transport`/`stealth_async_client`/`stealth_sync_client` all accept a `proxy=` override, defaulting to `stealth.configured_proxy_url()` (the admin-editable `outbound_proxy_url` site setting, falling back to `DEVPLACE_OUTBOUND_PROXY_URL` when the setting is unset) - this is the only real fix for a destination that blocks by IP/ASN reputation (e.g. reddit.com returns an identical "blocked by network security" page across every TLS/header fingerprint tried, confirmed by testing curl_cffi's full impersonation list - it is blocking the datacenter ASN outright, not fingerprinting the client). - **Form validation is Pydantic-native.** Routers declare `data: Annotated[SomeForm, Form()]` (models in `models.py`); the global `RequestValidationError` handler re-renders auth pages with messages (400) or redirects other routes. Read raw `await request.form()` only when also reading an uploaded file alongside a model. - **`RedirectResponse(url=..., status_code=302)`** - always pass `status_code` as an integer literal, never `status.HTTP_302_FOUND`. - **Ownership checks:** compare `resource["user_uid"] == user["uid"]` before edit/delete (use `content.is_owner`). **Deletes additionally allow any admin:** `is_owner(...) or is_admin(user)` on every content delete endpoint, so an administrator may delete any member's content while members are limited to their own. Edits stay owner-only. Deletes are **soft** and cascade under one shared stamp - see `devplacepy/database/CLAUDE.md`. diff --git a/Makefile b/Makefile index 5838ac4..5c56c86 100644 --- a/Makefile +++ b/Makefile @@ -9,20 +9,39 @@ LOCUST_WEB_WORKERS ?= 4 WEB_WORKERS ?= $(shell nproc 2>/dev/null || echo 2) DEVPLACE_RATE_LIMIT ?= 1000000 +VENV ?= $(CURDIR)/.venv +PYTHON := $(VENV)/bin/python +VENV_STAMP := $(VENV)/.installed +BOOTSTRAP_PYTHON := $(shell command -v python3 2>/dev/null || command -v python 2>/dev/null) + PYTHONDONTWRITEBYTECODE := 1 export PYTHONDONTWRITEBYTECODE -.PHONY: install dev clean tree tree-loc zip test test-headed test-unit test-api test-e2e test-fast test-failed test-first-failure test-slowest test-cache-clean coverage coverage-headed coverage-html locust locust-headless +.PHONY: venv install dev prod clean tree tree-loc zip test test-headed test-unit test-api test-e2e test-fast test-failed test-first-failure test-slowest test-cache-clean coverage coverage-headed coverage-html locust locust-headless -install: - pip install -e . - python -m playwright install chromium +$(PYTHON): + @test -n "$(BOOTSTRAP_PYTHON)" || { echo "python3 is required to create $(VENV)"; exit 1; } + $(BOOTSTRAP_PYTHON) -m venv $(VENV) -dev: - uvicorn devplacepy.main:app --reload --reload-dir devplacepy --host 0.0.0.0 --port 10500 --backlog 4096 +$(VENV_STAMP): $(PYTHON) pyproject.toml + $(PYTHON) -m pip install -U pip + $(PYTHON) -m pip install -e ".[dev]" + $(PYTHON) -m playwright install chromium + touch $(VENV_STAMP) -prod: - DEVPLACE_STATIC_VERSION=$$(date +%s) DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(WEB_WORKERS) uvicorn devplacepy.main:app --host 0.0.0.0 --port 10500 --workers $(WEB_WORKERS) --backlog 8192 --proxy-headers --forwarded-allow-ips '*' +venv: $(VENV_STAMP) + +install: $(PYTHON) + $(PYTHON) -m pip install -U pip + $(PYTHON) -m pip install -e ".[dev]" + $(PYTHON) -m playwright install chromium + touch $(VENV_STAMP) + +dev: $(VENV_STAMP) + $(PYTHON) -m uvicorn devplacepy.main:app --reload --reload-dir devplacepy --host 0.0.0.0 --port 10500 --backlog 4096 + +prod: $(VENV_STAMP) + DEVPLACE_STATIC_VERSION=$$(date +%s) DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(WEB_WORKERS) $(PYTHON) -m uvicorn devplacepy.main:app --host 0.0.0.0 --port 10500 --workers $(WEB_WORKERS) --backlog 8192 --proxy-headers --forwarded-allow-ips '*' delete-pyc: find . -name "__pycache__" -type d -prune -exec rm -rf {} + 2>/dev/null || true @@ -42,78 +61,78 @@ zip: @git ls-files -z | xargs -0 zip -q $(notdir $(CURDIR)).zip @printf 'Wrote %s (%s files)\n' "$(notdir $(CURDIR)).zip" "$$(git ls-files | wc -l)" -test: - PLAYWRIGHT_HEADLESS=1 python -m pytest tests/ +test: $(VENV_STAMP) + PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/ -test-headed: - PLAYWRIGHT_HEADLESS=0 python -m pytest tests/ +test-headed: $(VENV_STAMP) + PLAYWRIGHT_HEADLESS=0 $(PYTHON) -m pytest tests/ -test-unit: - python -m pytest tests/unit +test-unit: $(VENV_STAMP) + $(PYTHON) -m pytest tests/unit -test-api: - python -m pytest tests/api +test-api: $(VENV_STAMP) + $(PYTHON) -m pytest tests/api -test-e2e: - PLAYWRIGHT_HEADLESS=1 python -m pytest tests/e2e +test-e2e: $(VENV_STAMP) + PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/e2e -test-fast: - python -m pytest tests/unit tests/api +test-fast: $(VENV_STAMP) + $(PYTHON) -m pytest tests/unit tests/api -test-failed: - PLAYWRIGHT_HEADLESS=1 python -m pytest tests/ --last-failed --last-failed-no-failures none +test-failed: $(VENV_STAMP) + PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/ --last-failed --last-failed-no-failures none -test-first-failure: - PLAYWRIGHT_HEADLESS=1 python -m pytest tests/ -x +test-first-failure: $(VENV_STAMP) + PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/ -x -test-slowest: - PLAYWRIGHT_HEADLESS=1 python -m pytest tests/ --durations=40 +test-slowest: $(VENV_STAMP) + PLAYWRIGHT_HEADLESS=1 $(PYTHON) -m pytest tests/ --durations=40 -coverage: +coverage: $(VENV_STAMP) rm -f .coverage .coverage.* COVERAGE_PROCESS_START=$(CURDIR)/.coveragerc PLAYWRIGHT_HEADLESS=1 \ - python -m coverage run -m pytest tests/ - python -m coverage combine - python -m coverage report + $(PYTHON) -m coverage run -m pytest tests/ + $(PYTHON) -m coverage combine + $(PYTHON) -m coverage report -coverage-headed: +coverage-headed: $(VENV_STAMP) rm -f .coverage .coverage.* COVERAGE_PROCESS_START=$(CURDIR)/.coveragerc PLAYWRIGHT_HEADLESS=0 \ - python -m coverage run -m pytest tests/ - python -m coverage combine - python -m coverage report + $(PYTHON) -m coverage run -m pytest tests/ + $(PYTHON) -m coverage combine + $(PYTHON) -m coverage report coverage-html: coverage - python -m coverage html + $(PYTHON) -m coverage html @echo "Report written to htmlcov/index.html" -locust: +locust: $(VENV_STAMP) export DEVPLACE_DATABASE_URL="sqlite:///$(LOCUST_DB)"; \ export DEVPLACE_RATE_LIMIT=$(DEVPLACE_RATE_LIMIT); \ fuser -k $(LOCUST_PORT)/tcp 2>/dev/null || true; \ sleep 1; \ mkdir -p $(LOCUST_DB_DIR); \ rm -f $(LOCUST_DB); \ - DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(LOCUST_WEB_WORKERS) uvicorn devplacepy.main:app --host 127.0.0.1 --port $(LOCUST_PORT) --workers $(LOCUST_WEB_WORKERS) --backlog 8192 > /tmp/devplace_locust_server.log 2>&1 & \ + DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(LOCUST_WEB_WORKERS) $(PYTHON) -m uvicorn devplacepy.main:app --host 127.0.0.1 --port $(LOCUST_PORT) --workers $(LOCUST_WEB_WORKERS) --backlog 8192 > /tmp/devplace_locust_server.log 2>&1 & \ PID=$$!; \ while kill -0 $$PID 2>/dev/null && ! curl -s http://127.0.0.1:$(LOCUST_PORT)/ > /dev/null 2>&1; do sleep 0.5; done; \ if ! kill -0 $$PID 2>/dev/null; then echo "Server failed to start (port $(LOCUST_PORT) busy?). See /tmp/devplace_locust_server.log"; exit 1; fi; \ - locust --host http://127.0.0.1:$(LOCUST_PORT) --web-port $(LOCUST_WEB_PORT); \ + $(PYTHON) -m locust --host http://127.0.0.1:$(LOCUST_PORT) --web-port $(LOCUST_WEB_PORT); \ kill $$PID 2>/dev/null || true; \ rm -rf $(LOCUST_DB_DIR) -locust-headless: +locust-headless: $(VENV_STAMP) export DEVPLACE_DATABASE_URL="sqlite:///$(LOCUST_DB)"; \ export DEVPLACE_RATE_LIMIT=$(DEVPLACE_RATE_LIMIT); \ fuser -k $(LOCUST_PORT)/tcp 2>/dev/null || true; \ sleep 1; \ mkdir -p $(LOCUST_DB_DIR); \ rm -f $(LOCUST_DB); \ - DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(LOCUST_WEB_WORKERS) uvicorn devplacepy.main:app --host 127.0.0.1 --port $(LOCUST_PORT) --workers $(LOCUST_WEB_WORKERS) --backlog 8192 > /tmp/devplace_locust_server.log 2>&1 & \ + DEVPLACE_TEMPLATE_AUTO_RELOAD=0 DEVPLACE_WEB_WORKERS=$(LOCUST_WEB_WORKERS) $(PYTHON) -m uvicorn devplacepy.main:app --host 127.0.0.1 --port $(LOCUST_PORT) --workers $(LOCUST_WEB_WORKERS) --backlog 8192 > /tmp/devplace_locust_server.log 2>&1 & \ PID=$$!; \ while kill -0 $$PID 2>/dev/null && ! curl -s http://127.0.0.1:$(LOCUST_PORT)/ > /dev/null 2>&1; do sleep 0.5; done; \ if ! kill -0 $$PID 2>/dev/null; then echo "Server failed to start (port $(LOCUST_PORT) busy?). See /tmp/devplace_locust_server.log"; exit 1; fi; \ - locust --host http://127.0.0.1:$(LOCUST_PORT) --headless -u $(LOCUST_USERS) -r $(LOCUST_SPAWN_RATE) --run-time $(LOCUST_RUN_TIME) --html=$(LOCUST_DB_DIR)/report.html; \ + $(PYTHON) -m locust --host http://127.0.0.1:$(LOCUST_PORT) --headless -u $(LOCUST_USERS) -r $(LOCUST_SPAWN_RATE) --run-time $(LOCUST_RUN_TIME) --html=$(LOCUST_DB_DIR)/report.html; \ kill $$PID 2>/dev/null || true; \ rm -rf $(LOCUST_DB_DIR) diff --git a/README.md b/README.md index 9c59830..4dfbf2f 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Server-rendered social network for developers. FastAPI backend serving Jinja2 te ## Quick start ```bash -make install # pip install -e . +make install # create .venv if needed, pip install -e ".[dev]", playwright chromium make dev # uvicorn --reload on port 10500 make test # Playwright integration + unit tests, headless, fail-fast make test-headed # same tests in visible browser @@ -82,7 +82,7 @@ devplacepy/ | `/admin/trash` | Admin **Trash**: review, restore, and permanently purge soft-deleted content (posts, comments, gists, projects, news, project files, attachments) across the platform | | `/admin/devii-tasks` | Admin **Devii tasks**: every scheduled task across all owners with its schedule, run count, expiry and failure streak, plus per-task disable and delete | | `/notifications` | Notification list, mark read, live unread counts (`/notifications/counts`) | -| `/messages` | Real-time direct messaging over WebSocket (`/messages/ws`): live bidirectional delivery, optimistic send, typing indicators, read receipts, and online/last-seen presence. Messages render through the shared content pipeline (emoji shortcodes, image, video and audio embeds, autolink, sanitization). AI content correction and the AI modifier apply to direct messages, so typing an inline `@ai ` in a message executes it and the resolved result appears live in the chat for both participants (the `message` WS frame carries an additive `ai_processed` flag when a pending correction/modification was applied). An opened conversation loads its 500 most recent messages; older history is retained in the database. `GET /messages/conversations` returns the conversation list as JSON for a live client refresh; the `POST /messages/send` form remains as a no-JavaScript fallback and now accepts an attachment-only, empty-content message. `POST /messages/ws-ticket` exchanges a session/API-key auth to a short-lived (30s), single-use ticket a browser WebSocket can carry as `?ticket=...` on the handshake, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header - this is what makes off-session embedding of the chat possible. CLI: `devplace messaging prune-tickets` removes expired tickets | +| `/messages` | Real-time direct messaging over WebSocket (`/messages/ws`): live bidirectional delivery, optimistic send, typing indicators, read receipts, reconnect catch-up, and online/last-seen presence. Messages render through the shared content pipeline (emoji shortcodes, image, video and audio embeds, autolink, sanitization). AI content correction and the AI modifier apply to direct messages: the original is delivered immediately, then a second `message` frame with `ai_processed` replaces the bubble for both participants when the rewrite lands (including across workers). An opened conversation loads its 500 most recent messages and can page older rows with `?before=`; switching threads does not drop the socket. `GET /messages/conversations` returns the conversation list as JSON for a live client refresh; the `POST /messages/send` form remains as a no-JavaScript fallback and now accepts an attachment-only, empty-content message. `POST /messages/ws-ticket` exchanges a session/API-key auth to a short-lived (30s), single-use ticket a browser WebSocket can carry as `?ticket=...` on the handshake, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header - this is what makes off-session embedding of the chat possible. CLI: `devplace messaging prune-tickets` removes expired tickets | | `/votes` | Upvote/downvote on posts, comments, projects | | `/reactions` | Emoji reactions on posts, comments, gists, projects | | `/bookmarks` | Save/unsave content; `/bookmarks/saved` personal list | @@ -452,7 +452,7 @@ and its full configuration are documented automatically - including future servi - **`ConfigField`** - declarative parameter spec (type, default, validation, secret) a service uses to declare its editable settings - **`BaseService`** - abstract class with a reconciling run loop that honors the persisted `enabled`/command/interval state, plus `config_fields`, `get_config()`, `describe()`, and a log buffer - **`ServiceManager`** - singleton: `register`, `describe_all`, `set_enabled`, `send_command`, `save_config`, `supervise`, `shutdown_all` -- **`NewsService`** - a fully automatic, zero-maintenance news pipeline: fetches news from `news.app.molodetz.nl/api`, cleans each article, fetches and perceptually compares the images to reject placeholders and detect uniqueness, grades each one deterministically, reformats every valid article into clean Markdown (paragraphs, headings, lists) with the AI so the source wall of text reads as a proper article, and auto-rotates the best articles to Featured and the landing page. Its AI spend is metered from the gateway response headers and reported on the admin Services page (calls, tokens, total cost, and per-call averages) +- **`NewsService`** - a fully automatic, zero-maintenance news pipeline: fetches news from `news.app.molodetz.nl/api`, cleans each article, fetches and perceptually compares the images to reject placeholders and detect uniqueness, grades each one against a free, local, deterministic quality model (aquality - no billed LLM call, no API key), optionally reformats articles that clear the publish grade threshold into clean Markdown with a generative AI model (off by default, since the default grading model cannot generate text), and auto-rotates the best articles to Featured and the landing page. Any AI spend it does incur is metered from the gateway response headers and reported on the admin Services page (calls, tokens, total cost, and per-call averages) - **`BotsService`** - Playwright fleet of AI personas that browse and interact with a DevPlace instance, with live cost/usage metrics and a live screenshot monitor at `/admin/bots` (opt-in; install the `bots` extra) - **`GatewayService`** - OpenAI-compatible LLM gateway at `/openai/v1/*`, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default). Every chat request is made date-aware by injecting the current date in EU `DD/MM/YYYY` format into the system message when it contains no date already (date only, never time, to keep upstream prompt caching effective), and an admin-configurable system preamble (`gateway_system_preamble`) can be prepended ahead of the client's system message on every call. **Provider and model routing** (admin **Gateway** page, `/admin/gateway`) maps any number of requested model names onto named upstream providers and target models, each with its own pricing economy (input, output and cache-hit/cache-miss rates per million tokens) and an optional vision model that describes image content before forwarding, so text and vision models are merged transparently. Unmapped requests fall through to the default upstream unchanged, so existing clients are unaffected - **`JobService` / `ZipService` / `ForkService`** - generic async job framework (`services/jobs/`) for heavy, blocking work run off the request path; `ZipService` builds project zip archives in a subprocess, `ForkService` copies a project into a new project owned by the forking user @@ -524,7 +524,7 @@ restart. `SeoMetaService` is a separate AI subservice that generates a clean, search-optimised title, description and short keyword list for every published post, project, gist, news article and issue, entirely off the request path so it never slows the web server. The work is queued whenever content is created, edited or published; until the AI value is ready a plain-content default (built from the markdown-stripped text and clamped to safe lengths) fills the fields, so a page's metadata is **always populated, never empty**. The service uses the built-in internal AI gateway and meters its own AI cost and statistics in a dedicated usage table, surfaced together with its live task pipeline on the **Admin -> Services** page. This release also fixes the on-page metadata: the `` is now stripped of markdown markup (it previously leaked `#`, `**` and `[](...)` from the raw body), a `` tag is emitted (a short honest list, not stuffed), and social-card image dimensions and alt text are added. CLI: `devplace seo-meta prune` / `devplace seo-meta clear` (job rows only; the generated metadata persists). -`DeepsearchService` powers the public **Tools -> DeepSearch** researcher. Given a single research question it plans a set of diverse web search queries, interleaves their results so every angle contributes sources, and crawls the most relevant pages in a subprocess (concurrent fetches, plain HTTP first with a headless-browser fallback for JavaScript-heavy pages, PDF documents streamed and text-extracted, every URL SSRF-guarded). Social sites that block bots (X, YouTube, Reddit and similar) contribute the readable text supplied by the search engine, so their content is not lost to a login wall. A readability-grade extractor isolates the main article content of each page (navigation, cookie banners and footers are discarded), and the configured depth follows the most relevant in-page links for deeper coverage. Content is de-duplicated and indexed into a per-session ChromaDB vector collection (embeddings via the AI gateway with a local fallback). A chain of agents (report writer, findings extractor, linker) then synthesises a thorough cited markdown report grounded on the passages retrieved from that index, with key findings, a confidence score and source diversity; if synthesis fails the report is clearly marked as degraded instead of silently shipping raw source material. Progress streams live over `WS /tools/deepsearch/{uid}/ws`; the report is at `/tools/deepsearch/{uid}/session` (HTML or JSON) and can be exported as Markdown, JSON or PDF. A grounded chat over the session's evidence runs at `WS /tools/deepsearch/{uid}/chat` using hybrid retrieval (vector + keyword/BM25). Runs can be paused, resumed or cancelled. A cross-session URL cache avoids re-fetching pages seen by earlier runs. CLI: `devplace deepsearch prune` / `devplace deepsearch clear`. ChromaDB, weasyprint, pypdf and Playwright are core dependencies. +`DeepsearchService` powers the public **Tools -> DeepSearch** researcher, an iterative agent modelled on the way Perplexity and OpenAI's own deep-research agents work: search, read, decide what is still missing, search again. Given a single research question it plans a set of diverse web search queries, interleaves their results so every angle contributes sources, and crawls the most relevant pages in a subprocess (concurrent fetches, plain HTTP first with a headless-browser fallback for JavaScript-heavy pages, PDF documents streamed and text-extracted, every URL SSRF-guarded). After the first pass it runs up to two automatic gap-filling refinement rounds - a lightweight planner looks at what has been gathered so far and proposes a few more targeted queries only if real coverage gaps remain, bounded by the page budget, and stops the moment nothing more is needed. Social sites that block bots (X, YouTube, Reddit and similar) contribute the readable text supplied by the search engine, so their content is not lost to a login wall. A readability-grade extractor isolates the main article content of each page (navigation, cookie banners and footers are discarded), and the configured depth follows the most relevant in-page links for deeper coverage. Content is de-duplicated and indexed into a per-session ChromaDB vector collection (embeddings via the AI gateway with a local fallback); retrieval reranks candidate passages with Maximal Marginal Relevance so the report is grounded on genuinely diverse evidence rather than several near-duplicate hits on the same page. A chain of agents (report writer, findings extractor, linker) then synthesises a thorough cited markdown report grounded on the passages retrieved from that index, with key findings, a confidence score, source diversity and a short list of suggested follow-up questions you can ask straight into the chat; if synthesis fails the report is clearly marked as degraded instead of silently shipping raw source material. Progress streams live over `WS /tools/deepsearch/{uid}/ws`; the report is at `/tools/deepsearch/{uid}/session` (HTML or JSON) and can be exported as Markdown, JSON or PDF. A grounded chat over the session's evidence runs at `WS /tools/deepsearch/{uid}/chat` using hybrid retrieval (vector + keyword/BM25). Runs can be paused, resumed or cancelled. A cross-session URL cache avoids re-fetching pages seen by earlier runs. CLI: `devplace deepsearch prune` / `devplace deepsearch clear`. ChromaDB, weasyprint, pypdf and Playwright are core dependencies. `IsslopService` powers the public **Tools -> AI Usage Analyzer**, which classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. Sources are acquired in an isolated subprocess (git URLs are probed with `git ls-remote` and shallow-cloned with a 3 GB guard; websites render in a stealth headless browser with an HTTP fallback, bounded by depth, file and byte caps), inventoried with exclusion rules, and scored by a multi-signal static engine (twenty-one detector families, 126 signal types across an origin axis and a quality-deficit axis). For a live website, the analyzer also opens the home page in a headless browser and inspects what actually renders: it fingerprints AI website-builder platforms directly (Lovable, Bolt.new, Framer and others) and checks the page's real computed styles, layout and build artifacts, not just its file contents. Representative files receive an AI review pass and images an AI-generation review through the internal gateway (model `molodetz`, internal key); the static engine stays authoritative when the gateway is unreachable. Every pipeline step is persisted as an ordered event trail and streamed live over the pub/sub topic `public.isslop.{uid}`. The verdict is an A-F authenticity grade with a human/AI split and one of five categories (`ai-slop`, `sophisticated-ai`, `human-clean`, `human-messy`, `uncertain`), published as a persistent report with an embeddable SVG badge. Members keep their analysis history on their account; guest history is session-bound and claimed by the account on the first signed-in visit. Admin settings (private-host allowance, AI/image review toggles, image cap, retention, concurrency) live on `/admin/services`. CLI: `devplace isslop analyze ` / `devplace isslop prune` / `devplace isslop clear`. Playwright plus playwright-stealth back the website crawler. @@ -557,9 +557,10 @@ Configuration on the Services tab (`/admin/services`): |-----------|---------|---------| | `news_grade_threshold` | `7` | Minimum AI grade for auto-publish | | `news_api_url` | `https://news.app.molodetz.nl/api` | News source | -| `news_ai_url` | `http://localhost:10500/openai/v1/chat/completions` | AI grading endpoint (the internal gateway) | -| `news_ai_model` | `molodetz` | Generic model name; the gateway maps it to the real model | -| `news_ai_key` | internal key | AI API key (`NEWS_AI_KEY` env, then the auto-generated gateway internal key) | +| `news_ai_url` | `https://aquality.cloud.pravda.education/v1/chat/completions` | AI grading endpoint - a free, local, deterministic quality model by default | +| `news_ai_model` | `aquality` | Generic model name; echoed back, not used for routing | +| `news_ai_key` | internal key | AI API key (`NEWS_AI_KEY` env, then the auto-generated gateway internal key); unused by the default aquality endpoint | +| `news_format_enabled` | `false` | Reformat article bodies into Markdown with AI after grading; off by default since aquality only scores and cannot generate text - enable when `news_ai_url` points at a generative chat model | | `news_service_interval` | `3600` | Seconds between fetch cycles (min 60) | News articles have detail pages at `/news/{slug}` with full comment support (same component as posts/projects). The landing page can display curated articles toggled from admin. @@ -919,13 +920,17 @@ receives a notification through every provider they hold a live subscription for | Provider | Registration | Transport | |----------|--------------|-----------| | `webpush` | `PushSubscription` from the browser `PushManager` (endpoint + `p256dh`/`auth` keys) | Web Push Protocol, VAPID signed, `aesgcm` encrypted payload | -| `apns` | Hexadecimal device token | `POST https://api.push.apple.com/3/device/{token}` over HTTP/2, ES256 provider token | +| `apns` | Hexadecimal device token, optional stable `client_id` | Dedicated HTTP/2 client to `api.push.apple.com` or `api.sandbox.push.apple.com` (`POST /3/device/{token}`), ES256 provider token. Not the stealth/Chrome client. | `POST /push.json` accepts a registration for any active provider; a body without a -`provider` field is a `webpush` body, so browsers need no change. `GET /push.json` returns -the VAPID public key plus the providers currently accepting registrations. A provider that -is disabled or not fully configured accepts no registrations and is skipped during -delivery, so an unconfigured provider is inert rather than an error. +`provider` field is a `webpush` body, so browsers need no change. An APNs body is +`{"provider": "apns", "token": "...", "client_id": "..."}` - `client_id` is optional and +identifies the device across Apple token rotations, so a new token updates that row +instead of inserting another. A token-only body still works and revives a previously +dead token. `GET /push.json` returns the VAPID public key plus the providers currently +accepting registrations; when `apns` is active it includes `environment` (`production` or +`sandbox`). A provider that is disabled or not fully configured accepts no registrations +and is skipped during delivery, so an unconfigured provider is inert rather than an error. Every provider setting is edited at **`/admin/services/push`**: per provider an `Enabled` toggle, the VAPID subject for `webpush`, and team id, key id, `.p8` auth key (stored as a @@ -951,6 +956,7 @@ in real time, bridged onto the in-process pub/sub bus by a lock-owner relay: | Direct message received | receiver | | Comment on your post | post author | | Reply to your comment | comment author | +| Any comment on a post you've also commented on | every other commenter on that post | | `@mention` in any content | mentioned user | | Upvote on your content | content owner | | New follower | followed user | @@ -960,10 +966,12 @@ in real time, bridged onto the in-process pub/sub bus by a lock-owner relay: `create_notification` schedules delivery as a fire-and-forget async task, so a dead subscription or push-service error never blocks the triggering request. Delivery (`push.notify_user`) reads a user's subscriptions once, groups them by provider, builds -each provider's payload once, and sends over a single shared HTTP client. A subscription -the push service reports as gone (`404`/`410` for Web Push, `410` or an `Unregistered` -class reason for APNs) is soft-deleted; any other failure is logged and the subscription is -kept. +each provider's payload once, and sends over that provider's own HTTP client (stealth for +Web Push, a dedicated HTTP/2 client for APNs). A subscription the push service reports as +gone (`404`/`410` for Web Push, `410` or an `Unregistered` class reason for APNs) is +soft-deleted and the provider reason is logged; any other failure is logged and the +subscription is kept. APNs environment is stored per registration so a sandbox debug +token and a production token can coexist. A notification is also **marked read automatically when you open the page that shows its content** - viewing a post clears its comment, reply, upvote and mention notifications; @@ -1022,7 +1030,7 @@ offline. Installation requires a secure origin (HTTPS, or `localhost` for develo |------|------| | `devplacepy/push/providers/` | Provider protocol, Web Push (VAPID keys, payload encryption), APNs | | `devplacepy/push/store.py` | `push_registration` reads and writes | -| `devplacepy/push/delivery.py` | `notify_user` - group by provider, deliver, reap dead subscriptions | +| `devplacepy/push/delivery.py` | `notify_user` / `notify_registration` - group by provider, per-provider client, reap dead subscriptions | | `devplacepy/services/push/service.py` | Provider configuration at `/admin/services/push`, retention sweep, metrics | | `devplacepy/routers/push.py` | `/push.json`, `/service-worker.js`, `/manifest.json` | | `static/js/PushManager.js` | Service-worker registration + subscribe + opt-in UI | diff --git a/devplacepy/config.py b/devplacepy/config.py index e335fe4..dfd2719 100644 --- a/devplacepy/config.py +++ b/devplacepy/config.py @@ -72,6 +72,9 @@ INTERNAL_MODEL = "molodetz" INTERNAL_EMBED_MODEL = "molodetz~embed" INTERNAL_IMAGE_MODEL = "molodetz-img-small" +AQUALITY_NEWS_GRADING_URL = "https://aquality.cloud.pravda.education/v1/chat/completions" +AQUALITY_NEWS_GRADING_MODEL = "aquality" + AWARD_GIVE_COOLDOWN_HOURS_DEFAULT = 24 AWARD_RECEIVE_COOLDOWN_HOURS_DEFAULT = 24 AWARD_DISPLAY_HOURS_DEFAULT = 24 diff --git a/devplacepy/constants.py b/devplacepy/constants.py index 0401861..a7df351 100644 --- a/devplacepy/constants.py +++ b/devplacepy/constants.py @@ -2,6 +2,16 @@ TOPICS = ["devlog", "showcase", "question", "rant", "fun", "random", "politics"] +TOPIC_LABELS = { + "devlog": "Devlog", + "showcase": "Showcase", + "question": "Question", + "rant": "Rant", + "fun": "Fun", + "random": "Random", + "politics": "Politics", +} + REACTION_EMOJI = [ "\U0001f44d", "❤️", diff --git a/devplacepy/content.py b/devplacepy/content.py index 81edad8..244289b 100644 --- a/devplacepy/content.py +++ b/devplacepy/content.py @@ -46,6 +46,7 @@ from devplacepy.utils import ( award_rewards, track_action, create_notification, + create_thread_notifications, create_mention_notifications, is_admin, is_primary_admin, @@ -438,6 +439,7 @@ def create_comment_record( comment_url = f"{redirect_url}#comment-{comment_uid}" if target_type == "post": + already_notified = {user["uid"]} if parent_uid: parent = get_table("comments").find_one(uid=parent_uid, deleted_at=None) if parent and parent["user_uid"] != user["uid"]: @@ -448,6 +450,7 @@ def create_comment_record( user["uid"], comment_url, ) + already_notified.add(parent["user_uid"]) else: posts = get_table("posts") post = posts.find_one(uid=target_uid) @@ -461,6 +464,9 @@ def create_comment_record( user["uid"], comment_url, ) + already_notified.add(post["user_uid"]) + + create_thread_notifications(target_uid, user["uid"], comment_url, already_notified) create_mention_notifications(content, user["uid"], comment_url) record_screening( diff --git a/devplacepy/database/CLAUDE.md b/devplacepy/database/CLAUDE.md index 5efa45b..1826658 100644 --- a/devplacepy/database/CLAUDE.md +++ b/devplacepy/database/CLAUDE.md @@ -198,8 +198,8 @@ Site settings are seeded on startup (`site_settings` table): | `site_name` / `site_description` / `site_tagline` | DevPlace branding | General site metadata | | `news_grade_threshold` | `"7"` | Minimum AI grade for auto-publish | | `news_api_url` | `"https://news.app.molodetz.nl/api"` | News source API | -| `news_ai_url` | `"https://openai.app.molodetz.nl/v1/chat/completions"` | AI grading endpoint | -| `news_ai_model` | `"molodetz"` | AI model identifier | +| `news_ai_url` | `"https://aquality.cloud.pravda.education/v1/chat/completions"` | AI grading endpoint - the free, local aquality quality model by default (see `devplacepy/services/news/CLAUDE.md`) | +| `news_ai_model` | `"aquality"` | AI model identifier | | `max_upload_size_mb` / `allowed_file_types` / `max_attachments_per_resource` | `"10"` / `""` / `"10"` | Upload limits | | `rate_limit_per_minute` | `"60"` | Mutating requests per IP per window (`main.py` middleware); a `429` carries a `Retry-After: ` header | | `rate_limit_window_seconds` | `"60"` | Rolling window for the rate limit (also the `Retry-After` value) | diff --git a/devplacepy/database/core.py b/devplacepy/database/core.py index febd70b..bbbaca9 100644 --- a/devplacepy/database/core.py +++ b/devplacepy/database/core.py @@ -9,6 +9,8 @@ from collections import defaultdict from datetime import datetime, timedelta, timezone from devplacepy.cache import TTLCache from devplacepy.config import ( + AQUALITY_NEWS_GRADING_MODEL, + AQUALITY_NEWS_GRADING_URL, DATABASE_URL, DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT, diff --git a/devplacepy/database/notifications.py b/devplacepy/database/notifications.py index 3496005..7e705ad 100644 --- a/devplacepy/database/notifications.py +++ b/devplacepy/database/notifications.py @@ -8,6 +8,7 @@ from .soft_delete import soft_delete NOTIFICATION_TYPES = [ {"key": "comment", "label": "Comments", "description": "Someone comments on your post"}, {"key": "reply", "label": "Replies", "description": "Someone replies to your comment"}, + {"key": "thread", "label": "Thread activity", "description": "Someone else comments on a post you've commented on"}, {"key": "mention", "label": "Mentions", "description": "Someone mentions you with @username"}, {"key": "vote", "label": "Upvotes", "description": "Someone ++'d your content"}, {"key": "follow", "label": "Followers", "description": "Someone starts following you"}, diff --git a/devplacepy/database/schema.py b/devplacepy/database/schema.py index 21e3637..c40dc9e 100644 --- a/devplacepy/database/schema.py +++ b/devplacepy/database/schema.py @@ -1,6 +1,6 @@ # retoor -from .core import DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT, INTERNAL_GATEWAY_URL, _drop_index, _index, _uid_index, db, defaultdict, get_table, logger +from .core import AQUALITY_NEWS_GRADING_MODEL, AQUALITY_NEWS_GRADING_URL, DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT, INTERNAL_GATEWAY_URL, _drop_index, _index, _uid_index, db, defaultdict, get_table, logger from .settings import get_setting, set_setting from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns from .ranking import _authors_cache @@ -130,6 +130,7 @@ def init_db(): ("content", ""), ("read", False), ("created_at", ""), + ("updated_at", ""), ): if not messages.has_column(column): messages.create_column_by_example(column, example) @@ -144,6 +145,7 @@ def init_db(): "idx_messages_conversation_rev", ["receiver_uid", "sender_uid"], ) + _index(db, "messages", "idx_messages_updated_at", ["updated_at"]) _index(db, "notifications", "idx_notifications_user", ["user_uid"]) _index(db, "notifications", "idx_notifications_user_read", ["user_uid", "read"]) push_registration = get_table("push_registration") @@ -155,19 +157,38 @@ def init_db(): ("key_auth", ""), ("key_p256dh", ""), ("token", ""), + ("client_id", ""), + ("environment", ""), ("created_at", ""), + ("registered_at", ""), ("deleted_at", ""), ): if not push_registration.has_column(column): push_registration.create_column_by_example(column, example) _index(db, "push_registration", "idx_push_registration_user", ["user_uid"]) _index(db, "push_registration", "idx_push_registration_provider", ["provider"]) + _index( + db, + "push_registration", + "idx_push_registration_client", + ["user_uid", "provider", "client_id"], + ) + _index( + db, + "push_registration", + "idx_push_registration_token", + ["user_uid", "provider", "token"], + ) if "push_registration" in db.tables: with db: db.query( "UPDATE push_registration SET provider = 'webpush' " "WHERE provider IS NULL OR provider = ''" ) + db.query( + "UPDATE push_registration SET registered_at = created_at " + "WHERE registered_at IS NULL OR registered_at = ''" + ) _index(db, "sessions", "idx_sessions_token", ["session_token"]) projects = get_table("projects") for column, example in ( @@ -1799,8 +1820,8 @@ def init_db(): news_defaults = { "news_grade_threshold": "7", "news_api_url": "https://news.app.molodetz.nl/api", - "news_ai_url": INTERNAL_GATEWAY_URL, - "news_ai_model": "molodetz", + "news_ai_url": AQUALITY_NEWS_GRADING_URL, + "news_ai_model": AQUALITY_NEWS_GRADING_MODEL, } for key, value in news_defaults.items(): existing = db["site_settings"].find_one(key=key) @@ -2046,6 +2067,12 @@ def migrate_ai_gateway_settings() -> None: if get_setting(key, "") == OLD_GATEWAY_URL: set_setting(key, INTERNAL_GATEWAY_URL) logger.info(f"Migrated {key} to the internal gateway") + if get_setting("news_ai_url", "") == INTERNAL_GATEWAY_URL: + set_setting("news_ai_url", AQUALITY_NEWS_GRADING_URL) + logger.info("Migrated news_ai_url to the free aquality grading model") + if get_setting("news_ai_model", "") in ("molodetz", ""): + set_setting("news_ai_model", AQUALITY_NEWS_GRADING_MODEL) + logger.info("Migrated news_ai_model to aquality") if get_setting("bot_model", "") == "deepseek-chat": set_setting("bot_model", "molodetz") logger.info("Migrated bot_model to molodetz") diff --git a/devplacepy/docs_api/groups/content.py b/devplacepy/docs_api/groups/content.py index 80cb84c..e80b2ea 100644 --- a/devplacepy/docs_api/groups/content.py +++ b/devplacepy/docs_api/groups/content.py @@ -64,6 +64,33 @@ four ways to sign requests. field("before", "query", "string", False, "", "Pagination cursor."), ], ), + endpoint( + id="topics-hub", + method="GET", + path="/topics", + title="Browse topics", + summary="The topics hub - every post topic with its live post count, linking to its own crawlable listing page.", + auth="public", + interactive=True, + ), + endpoint( + id="topics-list", + method="GET", + path="/topics/{topic}", + title="Browse one topic", + summary=( + "A single topic's post listing, on its own permanent, crawlable URL (unlike /feed?topic=, " + "whose canonical collapses back to /feed). Same author-interleaved pagination as the feed." + ), + auth="public", + interactive=True, + params=[ + field( + "topic", "path", "enum", True, "devlog", "Topic key.", TOPICS + ), + field("before", "query", "string", False, "", "Pagination cursor."), + ], + ), endpoint( id="posts-create", method="POST", diff --git a/devplacepy/docs_api/groups/messaging.py b/devplacepy/docs_api/groups/messaging.py index 37d7fe2..2447f57 100644 --- a/devplacepy/docs_api/groups/messaging.py +++ b/devplacepy/docs_api/groups/messaging.py @@ -41,6 +41,14 @@ four ways to sign requests. "", "Jump to a conversation by username.", ), + field( + "before", + "query", + "string", + False, + "", + "ISO timestamp. When set, return the page of messages strictly older than this instant (for loading earlier history).", + ), ], ), endpoint( diff --git a/devplacepy/docs_api/groups/push.py b/devplacepy/docs_api/groups/push.py index 882a634..2cd6398 100644 --- a/devplacepy/docs_api/groups/push.py +++ b/devplacepy/docs_api/groups/push.py @@ -14,9 +14,11 @@ implements the Web Push protocol: fetch the public VAPID key, then register a Push Notification service device token and is only offered when an administrator has configured it. -`GET /push.json` lists the providers that currently accept registrations. A registration -body without a `provider` field is a `webpush` registration, so existing clients need no -change. +`GET /push.json` lists the providers that currently accept registrations. When `apns` is +active it includes `environment` (`production` or `sandbox`) so a native client can match +its build. A registration body without a `provider` field is a `webpush` registration, so +existing clients need no change. An APNs body may include a stable `client_id` so a later +token rotation updates the same device instead of inserting another row. There is no server-side unsubscribe endpoint: unsubscription is handled entirely in the browser by calling `PushManager.unsubscribe()` on the subscription. The server stops delivering @@ -37,7 +39,10 @@ four ways to sign requests. auth="public", sample_response={ "publicKey": "BASE64_VAPID_KEY", - "providers": {"webpush": {"publicKey": "BASE64_VAPID_KEY"}}, + "providers": { + "webpush": {"publicKey": "BASE64_VAPID_KEY"}, + "apns": {"environment": "production"}, + }, }, ), endpoint( @@ -80,15 +85,24 @@ four ways to sign requests. "string", False, "a1b2c3...", - "Hexadecimal device token. Required for apns.", + "Hexadecimal device token. Required for apns. Spaces and angle brackets are stripped.", + ), + field( + "client_id", + "json", + "string", + False, + "vendor-uuid", + "Stable per-device id for apns. When present, a new token updates this device instead of inserting a row.", ), ], notes=[ 'A webpush body is JSON: `{"endpoint": "...", "keys": {"p256dh": "...", "auth": "..."}}`.', - 'An APNs body is JSON: `{"provider": "apns", "token": "..."}`.', + 'An APNs body is JSON: `{"provider": "apns", "token": "...", "client_id": "..."}`. `client_id` is optional; token-only bodies keep working and revive a previously dead token.', "A provider that is unknown, disabled or unconfigured returns 400.", + "A newly created or revived registration is probed immediately. The response then includes `delivered` and, on failure, `error` with the provider reason. `registered` stays true so existing clients keep working.", ], - sample_response={"registered": True}, + sample_response={"registered": True, "delivered": True}, ), ], } diff --git a/devplacepy/docs_api/groups/tools.py b/devplacepy/docs_api/groups/tools.py index adda8ae..84e70a8 100644 --- a/devplacepy/docs_api/groups/tools.py +++ b/devplacepy/docs_api/groups/tools.py @@ -226,6 +226,7 @@ status and report. {"title": "Invention", "detail": "...", "confidence": 0.8, "citations": [1]} ], "sources": [{"url": "https://example.com", "title": "Example", "source": "httpx"}], + "follow_up_questions": ["Who else worked on the transistor?", "How did it replace vacuum tubes?"], "chat_ws_url": "/tools/deepsearch/DEEPSEARCH_JOB_UID/chat", "export_md_url": "/tools/deepsearch/DEEPSEARCH_JOB_UID/export.md", "export_json_url": "/tools/deepsearch/DEEPSEARCH_JOB_UID/export.json", diff --git a/devplacepy/main.py b/devplacepy/main.py index 56c82c6..1816b37 100644 --- a/devplacepy/main.py +++ b/devplacepy/main.py @@ -52,6 +52,7 @@ from devplacepy.routers import ( battles, feed, posts, + topics, comments, projects, profile, @@ -315,6 +316,9 @@ async def lifespan(app: FastAPI): from devplacepy.services.containers import forward await forward.close_client() + from devplacepy import push + + await push.shutdown_providers() app = FastAPI( @@ -471,6 +475,7 @@ async def on_validation_error(request: Request, exc: RequestValidationError): app.include_router(auth.router, prefix="/auth") app.include_router(feed.router, prefix="/feed") app.include_router(posts.router, prefix="/posts") +app.include_router(topics.router, prefix="/topics") app.include_router(comments.router, prefix="/comments") app.include_router(projects.router, prefix="/projects") app.include_router(profile.router, prefix="/profile") diff --git a/devplacepy/push/CLAUDE.md b/devplacepy/push/CLAUDE.md index e1149b7..da8a8ff 100644 --- a/devplacepy/push/CLAUDE.md +++ b/devplacepy/push/CLAUDE.md @@ -2,33 +2,35 @@ This file documents `devplacepy/push/` - push notification delivery and its prov ## What this package is -One delivery library behind one public surface. `devplacepy.push` exports `notify_user`, `register`, `ensure_certificates`, `public_key_standard_b64` and the Web Push crypto helpers; every caller in the codebase (`main.py`, `utils/notifications.py`, `routers/push.py`) imports only those names. Everything else is internal to the package. +One delivery library behind one public surface. `devplacepy.push` exports `notify_user`, `notify_registration`, `register`, `ensure_certificates`, `public_key_standard_b64` and the Web Push crypto helpers; every caller in the codebase (`main.py`, `utils/notifications.py`, `routers/push.py`) imports only those names. Everything else is internal to the package. | Module | Role | |---|---| | `providers/base.py` | `PushProvider` protocol, the `Delivery` outcome and the three outcome constants | | `providers/webpush.py` | VAPID key material, `aesgcm` payload encryption, the Web Push provider | -| `providers/apns.py` | Apple Push Notification service provider (token based, HTTP/2) | +| `providers/apns.py` | Apple Push Notification service provider (token based, dedicated HTTP/2 client) | | `providers/__init__.py` | `PROVIDERS` registry, `get`, `active`, `is_active`, `admin_fields`, `client_config` | -| `store.py` | Every `push_registration` read and write | -| `delivery.py` | `notify_user`: group by provider, one shared client, one prepared body per provider | +| `store.py` | Every `push_registration` read and write, including identity upsert and revive | +| `delivery.py` | `notify_user` / `notify_registration`: group by provider, one client per provider, one prepared body per provider | The admin configuration surface lives in `devplacepy/services/push/service.py` (`PushService`), not here. ## Adding a provider -1. Write `providers/.py` with a `PushProvider` subclass: `name`, `label`, `config_fields`, `is_configured`, `parse_registration`, `prepare`, `deliver`, optionally `client_config`. +1. Write `providers/.py` with a `PushProvider` subclass: `name`, `label`, `config_fields`, `is_configured`, `parse_registration`, `prepare`, `deliver`, optionally `client_config`, `stamp_registration`, `delivery_client`. 2. Add one entry to `PROVIDERS` in `providers/__init__.py`. That is the whole change. The registration route, the delivery loop, the admin page, the audit record, the metrics and the docs are written against the protocol and need no edit. The `Enabled` toggle (`push__enabled`) comes from the base class, so a provider never declares its own. +The default `delivery_client` is `stealth_async_client`. Override it only when the destination is a first-party API that must not see Chrome impersonation, PRIORITY frames, extra browser headers, or the outbound proxy. APNs is that case. + ## Invariants -- **Zero cost for the request.** Delivery is reached only through `utils/notifications.py` `_schedule_push`, a fire-and-forget task. Never make a route await `notify_user`, and never add a queue or a table to this path. +- **Zero cost for the request, except the welcome probe.** Delivery of real notifications is reached only through `utils/notifications.py` `_schedule_push`, a fire-and-forget task. `POST /push.json` awaits `notify_registration` only for a newly created or revived row, so the client can see `delivered` / `error`. Never add a queue or a table to this path. - **`deliver` never raises.** Return `Delivery(REJECTED, detail)` instead. `delivery.py` guards anyway, but a raising provider costs a log line per subscription. - **A provider that is not configured is inert, never an error.** `is_configured()` is false, `is_active()` is false, the delivery loop skips it, and `POST /push.json` refuses a registration for it with 400. Nothing else in the platform notices. -- **`DEAD` is the only outcome that touches the database.** It soft-deletes the registration (`deleted_at`), exactly like a `404`/`410` Web Push endpoint always did. `REJECTED` keeps the row. -- **Every insert writes `deleted_at: None`,** and every read filters `deleted_at IS NULL`. `push_registration` deliberately stays out of `SOFT_DELETE_TABLES` (no `deleted_by`, not restorable from Trash) - a dead device token has no owner action to undo. +- **`DEAD` is the only outcome that touches the database.** It soft-deletes the registration (`deleted_at`), exactly like a `404`/`410` Web Push endpoint always did - unless `Delivery.dead_before` names an instant the row was proven live again after (APNs 410 `timestamp` vs. `registered_at`, see "APNs specifics"), in which case the delete is skipped. `REJECTED` keeps the row. The Apple/Web Push reason is logged at WARNING (`Push dead via ...: `); do not drop `outcome.detail`. +- **Every insert writes `deleted_at: None`,** and every read of the live set filters `deleted_at IS NULL`. Identity lookups for upsert/revive intentionally ignore `deleted_at` so a client that re-registers a previously dead token comes back live. `push_registration` deliberately stays out of `SOFT_DELETE_TABLES` (no `deleted_by`, not restorable from Trash) - a dead device token has no owner action to undo. - **A row without a provider is a Web Push row.** `store.provider_of` resolves `None`/`""` to `DEFAULT_PROVIDER`, so a row written by an old worker during a deploy still delivers. `init_db` backfills the column once with a single converging `UPDATE`. ## Storage @@ -40,13 +42,31 @@ That is the whole change. The registration route, the delivery loop, the admin p | `provider` | `webpush` | `apns` | | `endpoint`, `key_auth`, `key_p256dh` | set | `NULL` | | `token` | `NULL` | device token | +| `client_id` | unused | optional stable per-device id | +| `environment` | unused | `production` or `sandbox`, stamped server-side at register | +| `registered_at` | stamped on insert/merge | stamped on insert/merge | -Deduplication is generic: `store.register` looks up `user_uid` + `provider` + exactly the fields the provider's `parse_registration` returned, so a provider never writes its own identity rule. +`registered_at` (ISO, both providers) is stamped on insert and on every `_merge` that actually changes a row (including revival). It exists solely to arbitrate the dead-token race described below - it is not a general "last seen" field. + +`store.register` returns `RegistrationWrite(record, created, revived)`. Identity, in order: + +1. `user_uid` + `provider` + `client_id` (live or dead), if `client_id` is present. +2. `user_uid` + `provider` + `token` (or `endpoint` for Web Push), live or dead. +3. Exact live match on the remaining fields. +4. Insert. + +A match **updates** the row (token, client_id, environment) and clears `deleted_at` if it was dead. Token rotation with the same `client_id` therefore replaces the token on one row. Sibling live rows that share the new token, the previous token, or the same `client_id` are marked dead so one device cannot accumulate duplicates. A body without `client_id` still works: the same token revives, a new token inserts. `push.update` is a real update, not a no-op of an identical POST. ## APNs specifics -- `POST https://{host}/3/device/{token}` over HTTP/2, host from `push_apns_environment` (`api.push.apple.com` or `api.sandbox.push.apple.com`; an unrecognised value falls back to production). HTTP/2 comes from `stealth_async_client` because the origin is `https` - the cleartext downgrade in `curl_transport` does not apply. -- Provider token: `ES256`, header `kid` = key id, claims `iss` = team id and `iat`. Cached per credential fingerprint for 45 minutes, so a worker signs at most one token per 45 minutes; Apple refuses tokens regenerated faster than every 20 minutes. Editing any credential changes the fingerprint and takes effect on the next delivery, with no restart. +- **Persistent HTTP/2 connection, not one per notification.** `apns.cached_client(timeout)` lazily creates ONE module-level `httpx.AsyncClient(http2=True, ...)` and reuses it across every `notify_user`/`notify_registration` call for the lifetime of the process, following Apple's explicit guidance to keep the connection open rather than repeatedly opening/closing (`sending-notification-requests-to-apns`). `ApnsProvider.closes_delivery_client()` returns `False` so `delivery.py` never closes it after a batch (Web Push still opens/closes per call via `stealth_async_client`, `closes_delivery_client()` defaulting `True` on the base class). Closed once, gracefully, in `main.py`'s shutdown via `push.shutdown_providers()` -> `ApnsProvider.aclose()` -> `apns.close_client()`, mirroring the identical `services/containers/forward.py` `client()`/`close_client()` pattern. Not stealth, not curl_cffi Chrome impersonation, not the outbound proxy - Apple's provider API is a first-party HTTP/2 service; browser PRIORITY frames, HPACK indexing of `:path`/`authorization`, extra `sec-ch-ua` headers, and a scraping proxy all violate that contract. This is the second documented exception to the stealth-only outbound rule (the other is container reverse-proxy forwarding). If the admin-configured delivery timeout changes, the cached client is rebuilt with the new timeout on next use and the old one is left for GC (not explicitly closed) - a deliberate, rare-path simplification. +- `POST https://{host}/3/device/{token}` over HTTP/2. Host comes from the **row's** `environment` if set, else `push_apns_environment` (`api.push.apple.com` or `api.sandbox.push.apple.com`; an unrecognised value falls back to production). Stamping environment per row lets a sandbox debug token and a production TestFlight token coexist. +- `GET /push.json` advertises `providers.apns.environment` when APNs is active so a native client can refuse to register a sandbox token against production. +- `parse_registration` requires a hexadecimal `token` (64-200 digits after stripping spaces and `<>`) and optionally `client_id` (string, max 128). A `client_id` that is not a string is a 400, not a silent drop. Server stamps `environment`; the client cannot pick the host. +- **Provider token (JWT):** `ES256`, header `kid` = key id, claims `iss` = team id and `iat`. Cached per credential fingerprint for 45 minutes (Apple refuses tokens regenerated faster than every 20 minutes, and requires a fresh one at least once an hour). **Shared cross-worker via `site_settings` key `push_apns_shared_token`** (JSON `{fingerprint, token, issued_at}`, read/written through the existing `get_setting`/`set_setting` cache-version machinery, propagating to sibling workers within ~1s like every other settings key) - a worker that finds no valid in-process cache checks the shared copy before signing a new one, so `make prod`'s multiple uvicorn workers converge on presenting Apple the SAME token instead of each independently re-signing on its own clock (which could otherwise interleave closer together than Apple's per-credential minimum spacing, worst case when workers boot near-simultaneously). Editing any credential changes the fingerprint and takes effect on the next delivery, with no restart. +- **A rejected provider token is evicted immediately, not left to expire from cache.** A 403 status, or any reason in `AUTH_REASONS` (`InvalidProviderToken`, `ExpiredProviderToken`, `BadCertificate`, `BadCertificateEnvironment`, `Forbidden`, `MissingProviderToken` - all provider-credential-shaped per Apple's reason table, never about a specific device), calls `invalidate_provider_token()`: clears the in-process cache AND the shared DB copy, so the very next delivery attempt (any worker) re-signs instead of retrying the same rejected token for up to 45 minutes. - A `.p8` that does not parse is cached as a failure for the same window, so a misconfiguration costs one error log per window rather than one parse per notification. -- `410`, or any status carrying reason `BadDeviceToken`, `Unregistered`, `ExpiredToken`, `DeviceTokenNotForTopic` or `TopicDisallowed`, is `DEAD`. Everything else is `REJECTED`. +- **`DEAD_REASONS` is deliberately narrow: only `BadDeviceToken` (400), `Unregistered` (410) and `ExpiredToken` (410).** These are the only reasons in Apple's documented table that describe the TOKEN itself as permanently invalid. `DeviceTokenNotForTopic` and `TopicDisallowed` are 400 errors about the **topic/provisioning matching the connection**, not the token - they fire identically for every token when `push_apns_topic` is misconfigured or the certificate/entitlements don't match, so treating them as dead would soft-delete the entire APNs subscriber base (a table deliberately kept OUT of `SOFT_DELETE_TABLES`, hence unrestorable) on the first delivery after a one-field admin typo. Never add a topic/provisioning-shaped reason to `DEAD_REASONS`. +- **A 410's `timestamp` is honored before deleting.** Apple's 410 body carries `{"reason": "Unregistered", "timestamp": }` - the instant Apple last confirmed the token dead, which can trail real device state by "several days" per Apple engineering guidance (410 delivery is intentionally non-deterministic; do not use it to infer app-uninstall timing). `apns._dead_before` converts it to ISO and it rides on `Delivery.dead_before`; `store.mark_dead(id, dead_before)` compares it against the row's `registered_at` and **skips the delete** (logs at INFO instead) if the registration was re-registered/revived after that instant - closing the race where an in-flight delivery against a stale row state would otherwise undo a concurrent revival. `dead_before` is only ever set for a 410; the `BadDeviceToken`/`ExpiredToken` paths pass `None` (unconditional delete, as before), since those reasons carry no `timestamp`. - The shared payload dict (`title`, `message`, `icon`, `url`) is translated once per batch into `aps.alert` plus the custom `url`/`icon` keys, mirroring what `service-worker.js` does for Web Push. `thread-id` mirrors the service worker's notification `tag`. +- `POST /push.json` probes a created or revived row immediately via `notify_registration`. The JSON is `{registered: true, delivered: bool, error?: string}`. `registered` stays true so existing clients keep working; `delivered`/`error` surface Apple's reason instead of killing the row silently from the client's point of view. The row is still marked dead on a `DEAD` outcome (subject to the `dead_before` guard above) so later `notify_user` calls do not keep hitting a known-bad token. diff --git a/devplacepy/push/__init__.py b/devplacepy/push/__init__.py index 81e1227..1aecdb2 100644 --- a/devplacepy/push/__init__.py +++ b/devplacepy/push/__init__.py @@ -1,6 +1,7 @@ # retoor -from devplacepy.push.delivery import notify_user +from devplacepy.push.delivery import notify_registration, notify_user +from devplacepy.push.providers import shutdown as shutdown_providers from devplacepy.push.providers.webpush import ( browser_base64, create_notification_authorization, @@ -23,7 +24,9 @@ __all__ = [ "generate_private_key", "generate_public_key", "hkdf", + "notify_registration", "notify_user", "public_key_standard_b64", "register", + "shutdown_providers", ] diff --git a/devplacepy/push/delivery.py b/devplacepy/push/delivery.py index 994ea67..2e3d3b2 100644 --- a/devplacepy/push/delivery.py +++ b/devplacepy/push/delivery.py @@ -3,9 +3,9 @@ import logging from typing import Any -from devplacepy import stealth from devplacepy.database import get_int_setting from devplacepy.push import providers, store +from devplacepy.push.providers.base import Delivery logger = logging.getLogger(__name__) @@ -29,6 +29,10 @@ def group_by_provider( return grouped +def _open_client(provider, timeout: float): + return provider.delivery_client(timeout) + + async def notify_user(user_uid: str, payload: dict[str, Any]) -> None: registrations = store.active_for_user(user_uid) if not registrations: @@ -36,48 +40,86 @@ async def notify_user(user_uid: str, payload: dict[str, Any]) -> None: return grouped = group_by_provider(registrations) - async with stealth.stealth_async_client(timeout=timeout_seconds()) as client: - for name, rows in grouped.items(): - provider = providers.PROVIDERS.get(name) - if provider is None: - logger.warning( - "Unknown push provider %s on %s subscriptions of user %s", - name, - len(rows), - user_uid, - ) - continue - if not providers.is_active(provider): - logger.debug( - "Push provider %s is not active; skipping %s subscriptions", - name, - len(rows), - ) - continue - try: - prepared = provider.prepare(payload) - except Exception as exc: - logger.error("Push provider %s could not build a payload: %s", name, exc) - continue + timeout = timeout_seconds() + for name, rows in grouped.items(): + provider = providers.PROVIDERS.get(name) + if provider is None: + logger.warning( + "Unknown push provider %s on %s subscriptions of user %s", + name, + len(rows), + user_uid, + ) + continue + if not providers.is_active(provider): + logger.debug( + "Push provider %s is not active; skipping %s subscriptions", + name, + len(rows), + ) + continue + try: + prepared = provider.prepare(payload) + except Exception as exc: + logger.error("Push provider %s could not build a payload: %s", name, exc) + continue + try: + client = _open_client(provider, timeout) + except Exception as exc: + logger.error("Push provider %s could not open a client: %s", name, exc) + continue + try: for registration in rows: await _deliver_one(provider, client, registration, prepared, user_uid) + finally: + if provider.closes_delivery_client(): + await client.aclose() -async def _deliver_one(provider, client, registration, prepared, user_uid) -> None: +async def notify_registration( + registration: dict[str, Any], payload: dict[str, Any] +) -> Delivery: + name = store.provider_of(registration) + provider = providers.PROVIDERS.get(name) + if provider is None: + return Delivery(providers.REJECTED, f"unknown provider {name}") + if not providers.is_active(provider): + return Delivery(providers.REJECTED, f"provider {name} is not active") + try: + prepared = provider.prepare(payload) + except Exception as exc: + return Delivery(providers.REJECTED, str(exc)) + user_uid = registration.get("user_uid") or "" + try: + client = _open_client(provider, timeout_seconds()) + except Exception as exc: + return Delivery(providers.REJECTED, str(exc)) + try: + return await _deliver_one(provider, client, registration, prepared, user_uid) + finally: + if provider.closes_delivery_client(): + await client.aclose() + + +async def _deliver_one(provider, client, registration, prepared, user_uid) -> Delivery: try: outcome = await provider.deliver(client, registration, prepared) except Exception as exc: logger.error("Push provider %s raised for %s: %s", provider.name, user_uid, exc) - return + return Delivery(providers.REJECTED, str(exc)) if outcome.status == providers.ACCEPTED: logger.debug("Push delivered to %s via %s", user_uid, provider.name) - return + return outcome if outcome.status == providers.DEAD: + logger.warning( + "Push dead via %s for %s: %s", provider.name, user_uid, outcome.detail + ) try: - store.mark_dead(registration["id"]) + store.mark_dead(registration["id"], outcome.dead_before) except Exception as exc: logger.error("Could not soft-delete push subscription: %s", exc) - return + return outcome logger.warning( "Push rejected by %s for %s: %s", provider.name, user_uid, outcome.detail ) + return outcome diff --git a/devplacepy/push/providers/__init__.py b/devplacepy/push/providers/__init__.py index e8619e4..f556cbb 100644 --- a/devplacepy/push/providers/__init__.py +++ b/devplacepy/push/providers/__init__.py @@ -35,6 +35,7 @@ __all__ = [ "get", "is_active", "names", + "shutdown", ] @@ -74,3 +75,11 @@ def _client_config(provider: PushProvider) -> dict[str, Any]: except Exception as exc: logger.error("Push provider %s failed to describe itself: %s", provider.name, exc) return {} + + +async def shutdown() -> None: + for provider in PROVIDERS.values(): + try: + await provider.aclose() + except Exception as exc: + logger.error("Push provider %s failed to close: %s", provider.name, exc) diff --git a/devplacepy/push/providers/apns.py b/devplacepy/push/providers/apns.py index 1445ad8..6124301 100644 --- a/devplacepy/push/providers/apns.py +++ b/devplacepy/push/providers/apns.py @@ -5,13 +5,14 @@ import json import logging import string import time +from datetime import datetime, timezone from typing import Any import httpx import jwt from devplacepy.config import SECONDS_PER_DAY -from devplacepy.database import get_setting +from devplacepy.database import get_setting, set_setting from devplacepy.push.providers.base import ( ACCEPTED, DEAD, @@ -44,20 +45,35 @@ ENVIRONMENT_OPTIONS = [ TOKEN_REFRESH_SECONDS = 45 * 60 TOKEN_MIN_LENGTH = 64 TOKEN_MAX_LENGTH = 200 +CLIENT_ID_MAX_LENGTH = 128 THREAD_ID = "devplace-notification" PUSH_TYPE = "alert" PRIORITY = "10" -DEAD_REASONS = frozenset( +SHARED_TOKEN_KEY = "push_apns_shared_token" + +# Per Apple's documented reason table, only these mean the TOKEN itself is +# permanently dead (410 Unregistered/ExpiredToken, 400 BadDeviceToken). +# DeviceTokenNotForTopic and TopicDisallowed are topic/provisioning +# misconfigurations at 400 - they affect every token uniformly and must +# never be treated as a reason to delete a registration. +DEAD_REASONS = frozenset({"BadDeviceToken", "ExpiredToken", "Unregistered"}) + +# 403 reasons meaning the provider (JWT) token/credential itself was +# rejected, not any device token. The cached token must be dropped so the +# next attempt re-signs instead of retrying the same rejected token. +AUTH_REASONS = frozenset( { - "BadDeviceToken", - "DeviceTokenNotForTopic", - "ExpiredToken", - "Unregistered", - "TopicDisallowed", + "BadCertificate", + "BadCertificateEnvironment", + "ExpiredProviderToken", + "Forbidden", + "InvalidProviderToken", + "MissingProviderToken", } ) _token_state: dict[str, Any] = {} +_client: httpx.AsyncClient | None = None def _setting(key: str) -> str: @@ -70,13 +86,78 @@ def _environment() -> str: def host() -> str: - return HOSTS[_environment()] + return host_for(_environment()) + + +def host_for(environment: str | None) -> str: + value = (environment or "").strip() or _environment() + return HOSTS[value] if value in HOSTS else HOSTS[DEFAULT_ENVIRONMENT] + + +def gateway_client(timeout: float) -> httpx.AsyncClient: + return httpx.AsyncClient( + http2=True, + timeout=timeout, + trust_env=False, + verify=True, + follow_redirects=False, + ) + + +def cached_client(timeout: float) -> httpx.AsyncClient: + global _client + if _client is None or _client.is_closed: + _client = gateway_client(timeout) + return _client + + +async def close_client() -> None: + global _client + if _client is not None and not _client.is_closed: + await _client.aclose() + _client = None + + +def _normalize_token(token: str) -> str: + return "".join( + character.lower() for character in token if character in string.hexdigits + ) def _fingerprint(team_id: str, key_id: str, auth_key: str) -> str: return hashlib.sha256(f"{team_id}:{key_id}:{auth_key}".encode("utf-8")).hexdigest() +def _read_shared_token() -> dict[str, Any] | None: + raw = get_setting(SHARED_TOKEN_KEY, "") + if not raw: + return None + try: + data = json.loads(raw) + except ValueError: + return None + if ( + not isinstance(data, dict) + or not isinstance(data.get("fingerprint"), str) + or not isinstance(data.get("token"), str) + or not isinstance(data.get("issued_at"), int) + ): + return None + return data + + +def _write_shared_token(fingerprint: str, token: str, issued_at: int) -> None: + set_setting( + SHARED_TOKEN_KEY, + json.dumps({"fingerprint": fingerprint, "token": token, "issued_at": issued_at}), + ) + + +def invalidate_provider_token() -> None: + _token_state.pop("current", None) + set_setting(SHARED_TOKEN_KEY, "") + + def provider_token(team_id: str, key_id: str, auth_key: str) -> str: fingerprint = _fingerprint(team_id, key_id, auth_key) issued_at = int(time.time()) @@ -89,6 +170,21 @@ def provider_token(team_id: str, key_id: str, auth_key: str) -> str: if state["token"] is None: raise ValueError(state["error"]) return state["token"] + + shared = _read_shared_token() + if ( + shared + and shared["fingerprint"] == fingerprint + and issued_at - shared["issued_at"] < TOKEN_REFRESH_SECONDS + ): + _token_state["current"] = { + "token": shared["token"], + "error": "", + "issued_at": shared["issued_at"], + "fingerprint": fingerprint, + } + return shared["token"] + try: token = jwt.encode( {"iss": team_id, "iat": issued_at}, @@ -112,6 +208,7 @@ def provider_token(team_id: str, key_id: str, auth_key: str) -> str: "issued_at": issued_at, "fingerprint": fingerprint, } + _write_shared_token(fingerprint, token, issued_at) return token @@ -125,6 +222,22 @@ def _reason(response: httpx.Response) -> str: return "" +def _dead_before(response: httpx.Response) -> str | None: + try: + body = response.json() + except ValueError: + return None + if not isinstance(body, dict): + return None + raw = body.get("timestamp") + if not isinstance(raw, (int, float)) or isinstance(raw, bool): + return None + try: + return datetime.fromtimestamp(raw / 1000, tz=timezone.utc).isoformat() + except (OverflowError, OSError, ValueError): + return None + + class ApnsProvider(PushProvider): name = "apns" label = PROVIDER_LABEL @@ -181,16 +294,41 @@ class ApnsProvider(PushProvider): and _setting(TOPIC_KEY) ) + def client_config(self) -> dict[str, Any]: + return {"environment": _environment()} + + def stamp_registration(self, fields: dict[str, Any]) -> dict[str, Any]: + return {**fields, "environment": _environment()} + + def delivery_client(self, timeout: float) -> httpx.AsyncClient: + return cached_client(timeout) + + def closes_delivery_client(self) -> bool: + return False + + async def aclose(self) -> None: + await close_client() + def parse_registration(self, body: dict[str, Any]) -> dict[str, Any] | None: token = body.get("token") if not isinstance(token, str): return None - token = token.strip() + token = _normalize_token(token) if not TOKEN_MIN_LENGTH <= len(token) <= TOKEN_MAX_LENGTH: return None - if any(character not in string.hexdigits for character in token): + fields: dict[str, Any] = {"token": token} + client_id = body.get("client_id") + if client_id is None: + return fields + if not isinstance(client_id, str): return None - return {"token": token} + client_id = client_id.strip() + if not client_id: + return fields + if len(client_id) > CLIENT_ID_MAX_LENGTH: + return None + fields["client_id"] = client_id + return fields def prepare(self, payload: dict[str, Any]) -> str: return json.dumps( @@ -228,7 +366,7 @@ class ApnsProvider(PushProvider): try: headers = self.headers() response = await client.post( - f"https://{host()}/3/device/{token}", + f"https://{host_for(registration.get('environment'))}/3/device/{token}", headers=headers, content=prepared.encode("utf-8"), ) @@ -238,6 +376,9 @@ class ApnsProvider(PushProvider): return Delivery(ACCEPTED) reason = _reason(response) detail = f"{response.status_code} {reason}".strip() + if response.status_code == 403 or reason in AUTH_REASONS: + invalidate_provider_token() if response.status_code == 410 or reason in DEAD_REASONS: - return Delivery(DEAD, detail) + dead_before = _dead_before(response) if response.status_code == 410 else None + return Delivery(DEAD, detail, dead_before) return Delivery(REJECTED, detail) diff --git a/devplacepy/push/providers/base.py b/devplacepy/push/providers/base.py index 4e8d71f..f2e5891 100644 --- a/devplacepy/push/providers/base.py +++ b/devplacepy/push/providers/base.py @@ -18,6 +18,7 @@ REJECTED = "rejected" class Delivery: status: str detail: str = "" + dead_before: str | None = None class PushProvider(ABC): @@ -51,6 +52,20 @@ class PushProvider(ABC): def client_config(self) -> dict[str, Any]: return {} + def stamp_registration(self, fields: dict[str, Any]) -> dict[str, Any]: + return fields + + def delivery_client(self, timeout: float) -> httpx.AsyncClient: + from devplacepy import stealth + + return stealth.stealth_async_client(timeout=timeout) + + def closes_delivery_client(self) -> bool: + return True + + async def aclose(self) -> None: + return None + @abstractmethod def is_configured(self) -> bool: ... diff --git a/devplacepy/push/store.py b/devplacepy/push/store.py index 24fa3eb..cfa175c 100644 --- a/devplacepy/push/store.py +++ b/devplacepy/push/store.py @@ -1,6 +1,7 @@ # retoor import logging +from dataclasses import dataclass from datetime import datetime, timezone from typing import Any @@ -13,6 +14,17 @@ logger = logging.getLogger(__name__) TABLE = "push_registration" +@dataclass(frozen=True) +class RegistrationWrite: + record: dict[str, Any] + created: bool + revived: bool + + @property + def probe(self) -> bool: + return self.created or self.revived + + def table(): return get_table(TABLE) @@ -25,31 +37,161 @@ def active_for_user(user_uid: str) -> list[dict[str, Any]]: return list(table().find(user_uid=user_uid, deleted_at=None)) +def _filled(fields: dict[str, Any]) -> dict[str, Any]: + filled: dict[str, Any] = {} + for key, value in fields.items(): + if value is None: + continue + if isinstance(value, str): + value = value.strip() + if not value: + continue + filled[key] = value + return filled + + +def _prefer_live(rows: list[dict[str, Any]]) -> dict[str, Any] | None: + if not rows: + return None + live = [row for row in rows if not row.get("deleted_at")] + return (live or rows)[-1] + + +def _lookup(user_uid: str, provider: str, **identity: Any) -> dict[str, Any] | None: + return _prefer_live( + list(table().find(user_uid=user_uid, provider=provider, **identity)) + ) + + +def _with_id(record: dict[str, Any]) -> dict[str, Any]: + if record.get("id"): + return record + found = table().find_one(uid=record.get("uid")) + return found or record + + +def _retire_duplicates( + user_uid: str, + provider: str, + keep_id: int, + fields: dict[str, Any], + previous_token: str | None = None, +) -> None: + token = fields.get("token") + client_id = fields.get("client_id") + for row in table().find(user_uid=user_uid, provider=provider, deleted_at=None): + if row["id"] == keep_id: + continue + if client_id and row.get("client_id") == client_id: + mark_dead(row["id"]) + continue + if token and row.get("token") == token: + mark_dead(row["id"]) + continue + if previous_token and row.get("token") == previous_token: + mark_dead(row["id"]) + + +def _merge( + existing: dict[str, Any], fields: dict[str, Any] +) -> RegistrationWrite: + revived = bool(existing.get("deleted_at")) + patch: dict[str, Any] = {"id": existing["id"]} + if revived: + patch["deleted_at"] = None + changed = revived + for key, value in fields.items(): + if existing.get(key) != value: + patch[key] = value + changed = True + if not changed: + return RegistrationWrite(_with_id(existing), False, False) + patch["registered_at"] = datetime.now(timezone.utc).isoformat() + previous_token = existing.get("token") + table().update(patch, ["id"]) + merged = {**existing, **patch} + if revived: + merged["deleted_at"] = None + merged = _with_id(merged) + _retire_duplicates( + existing["user_uid"], + existing["provider"], + merged["id"], + fields, + previous_token=previous_token if previous_token != fields.get("token") else None, + ) + logger.info( + "Updated %s push subscription for user %s%s", + existing.get("provider"), + existing.get("user_uid"), + " (revived)" if revived else "", + ) + return RegistrationWrite(merged, False, revived) + + def register( user_uid: str, provider: str, fields: dict[str, Any] -) -> tuple[dict[str, Any], bool]: +) -> RegistrationWrite: + fields = _filled(fields) registrations = table() - existing = registrations.find_one( + client_id = fields.get("client_id") + token = fields.get("token") + endpoint = fields.get("endpoint") + + if client_id: + existing = _lookup(user_uid, provider, client_id=client_id) + if existing: + return _merge(existing, fields) + + if token: + existing = _lookup(user_uid, provider, token=token) + if existing: + return _merge(existing, fields) + + if endpoint: + existing = _lookup(user_uid, provider, endpoint=endpoint) + if existing: + return _merge(existing, fields) + + live = registrations.find_one( user_uid=user_uid, provider=provider, deleted_at=None, **fields ) - if existing: - logger.debug("Push subscription already registered for user %s", user_uid) - return existing, False + if live: + return RegistrationWrite(_with_id(live), False, False) + now = datetime.now(timezone.utc).isoformat() record = { "uid": generate_uid(), "user_uid": user_uid, "provider": provider, - "created_at": datetime.now(timezone.utc).isoformat(), + "created_at": now, + "registered_at": now, "deleted_at": None, **fields, } - registrations.insert(record) + inserted = registrations.insert(record) + if isinstance(inserted, int): + record["id"] = inserted + record = _with_id(record) + if record.get("id"): + _retire_duplicates(user_uid, provider, record["id"], fields) logger.info("Registered %s push subscription for user %s", provider, user_uid) - return record, True + return RegistrationWrite(record, True, False) -def mark_dead(registration_id: int) -> None: +def mark_dead(registration_id: int, dead_before: str | None = None) -> None: + if dead_before is not None: + row = table().find_one(id=registration_id) + registered_at = row.get("registered_at") if row else None + if registered_at and registered_at > dead_before: + logger.info( + "Skipped marking push subscription id=%s dead: registered again at %s " + "after the provider confirmed it dead at %s", + registration_id, + registered_at, + dead_before, + ) + return table().update( {"id": registration_id, "deleted_at": datetime.now(timezone.utc).isoformat()}, ["id"], diff --git a/devplacepy/routers/CLAUDE.md b/devplacepy/routers/CLAUDE.md index a9afc17..5ee82ad 100644 --- a/devplacepy/routers/CLAUDE.md +++ b/devplacepy/routers/CLAUDE.md @@ -11,11 +11,12 @@ Prefixes are wired in `main.py`: | `/auth` | auth/ package - one leaf per flow (`signup`, `login`, `logout`, `forgotpassword`, `resetpassword`) | | `/feed` | feed.py | | `/posts` | posts.py | +| `/topics` | topics.py - crawlable per-topic category index pages (`GET /topics` hub, `GET /topics/{topic}` per-topic post listing over the same `TOPICS` set as the feed sidebar filter). Reuses `feed.py`'s `get_feed_posts`/`enrich_post_cards` so a topic page is a fully-enriched `_post_card.html` listing, not a stripped-down duplicate. Unlike `/feed?topic=X` (whose canonical strips the query string back to bare `/feed`, so it is never indexed as a distinct page), each `/topics/{topic}` page has its own canonical URL, unique title/description, breadcrumbs, and a sitemap entry - see "SEO implementation" below | | `/comments` | comments.py | | `/projects` | projects/ package - `index.py` (listing/detail/create/delete plus owner-only visibility toggles `POST /{slug}/private` and `POST /{slug}/readonly`, and async `POST /{slug}/fork`), `files.py`, and the `containers/` subpackage (below). `main.py` mounts the whole `/projects` tree from this one package. See `routers/projects/CLAUDE.md` for the deep detail on this tree | | `/projects/{slug}/files` | projects/files.py - per-project virtual filesystem (CRUD dirs/files, upload, inline edit, plus line-range ops: `lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append` for large text files) | | `/profile` | profile/ package - `index.py` (page/search/followers/update/api-key; the **Posts** tab shows only the 10 most recent posts, while the sidebar `posts_count` stat always uses the real `count()` so it is never capped to the rendered 10), `customization.py`, `notifications.py`, `ai_correction.py` (`POST /{username}/ai-correction`, owner-or-admin AI content correction toggle + prompt), `ai_modifier.py` (`POST /{username}/ai-modifier`, owner-or-admin AI modifier toggle + prompt), `interactions.py` (`POST /{username}/interactions`, owner-or-admin Devii interactive widgets preference; admin default via `devii_interactions_default`), `telegram.py` (`POST /{username}/telegram`, owner-or-admin Telegram pairing-code request/unpair), `usage.py` (`_ai_quota`, re-exported from the package) | -| `/messages` | messages.py - real-time DM chat. `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `GET /conversations` (flat JSON conversation list, same `get_conversations` helper as the page), `POST /send` (no-JS fallback, also accepts an attachment-only empty-content message), `POST /ws-ticket` (issues a 30s single-use WS auth ticket so a browser WebSocket can authenticate cross-context, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, and read receipts on top of the existing `messages` table - presence is NOT part of this WS (it is the generic `PresenceManager`/pub/sub mechanism, see `devplacepy/services/CLAUDE.md`). Both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention) and then through `_finalize_and_broadcast`, which awaits any pending SYNC AI correction/modifier futures (DM content runs through both, so typing `@ai ` executes live), broadcasts the FINAL corrected/modified content with an additive `ai_processed` frame flag (the WS path passes `request=websocket` and awaits directly since HTTP middleware does not run for websockets; the frontend swaps the sender's optimistic bubble to the authoritative content). Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, deduped against the hub's delivered set). See `devplacepy/services/messaging/CLAUDE.md` | +| `/messages` | messages.py - real-time DM chat. `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `GET /conversations` (flat JSON conversation list, same `get_conversations` helper as the page), `POST /send` (no-JS fallback, also accepts an attachment-only empty-content message), `POST /ws-ticket` (issues a 30s single-use WS auth ticket so a browser WebSocket can authenticate cross-context, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, and read receipts on top of the existing `messages` table - presence is NOT part of this WS (it is the generic `PresenceManager`/pub/sub mechanism, see `devplacepy/services/CLAUDE.md`). Both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention) and then through `_finalize_and_broadcast`, which broadcasts the persisted row immediately and then applies SYNC AI correction/modifier (HTTP awaits so the JSON body is final; WS schedules it so the receive loop never blocks). An in-place rewrite stamps `messages.updated_at` and emits a second `ai_processed` frame; other workers pick it up from `message_relay._tick_updates`. Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, plus `updated_at` for revisions, new rows deduped against the hub's delivered set). See `devplacepy/services/messaging/CLAUDE.md` | | `/notifications` | notifications.py | | `/votes` | votes.py | | `/reactions` | reactions.py - emoji reaction toggle on any target: `POST /reactions/{target_type}/{target_uid}` (optimistic `ReactionBar`) | @@ -48,7 +49,7 @@ Prefixes are wired in `main.py`: | `/quizzes` | quizzes/ package - **Quizzes**: `index.py` (hub, create, import, detail, export, edit, delete, publish, per-quiz leaderboard, cross-quiz scoreboard), `questions.py` (builder page + question CRUD + reorder), `attempts.py` (start/resume, play, answer, finish, results). The hub `GET /quizzes` is the approved three-column `.feed-layout` reused verbatim from `/feed`: filters + search left (`?filter=all|todo|done|mine|drafts`, `?search=`, `?page=`), the quiz list centre with the **New quiz** and **Create quiz with Devii** actions, and the cross-quiz **scoreboard** rail right (score per user, best attempt per quiz, 15s display cache). Publishing is terminal - every write on a published quiz is a 400 and there is no unpublish route. See `devplacepy/services/quiz/CLAUDE.md` | | `/battles` | battles.py - **Opinion Wars**, week-long two-faction battles attached to posts: `GET ""` (listing, quizzes-style filters active/ended/mine + search), `GET /{uid}` (JSON-only full state, runs lazy resolution), `GET /{uid}/events?after=` (durable event trail replay), `POST /{uid}/join` (join or switch faction), `POST /{uid}/fight` (spend 25 Code Farm coins, deal level-weighted damage, 24h cooldown). The battle card renders on the parent post (no battle detail HTML page). See `devplacepy/services/opinionwar/CLAUDE.md` | | `/game` | game/ package - the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard?board=`, `POST /game/{plant,harvest,buy-plot,upgrade,fertilize,daily,grant,perk,prestige,legacy,mastery,quests/claim}`, `POST /game/{defense/upgrade,defense/downgrade,infrastructure/buy,cosmetics/buy,cosmetics/equip}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}`. See `devplacepy/services/game/CLAUDE.md` | -| (none) | push.py - push + PWA: `GET /push.json` (VAPID public key + the providers accepting registrations), `POST /push.json` (register with any active provider; a body without `provider` is a `webpush` body, so existing clients are unchanged; an unknown, disabled or unconfigured provider is a 400), `GET /service-worker.js`, `GET /manifest.json`. Provider protocol and delivery live in `devplacepy/push/` - see `devplacepy/push/CLAUDE.md` | +| (none) | push.py - push + PWA: `GET /push.json` (VAPID public key + the providers accepting registrations; `apns` includes `environment` when active), `POST /push.json` (register with any active provider; a body without `provider` is a `webpush` body; an APNs body is `token` plus optional `client_id` for device-stable upsert; created/revived rows are probed and the JSON may include `delivered`/`error`; an unknown, disabled or unconfigured provider is a 400), `GET /service-worker.js`, `GET /manifest.json`. Provider protocol and delivery live in `devplacepy/push/` - see `devplacepy/push/CLAUDE.md` | | (none) | docs.py (docs/ package) - the documentation site (prose pages + API reference). See `routers/docs/CLAUDE.md` for the deep detail on this tree (`DOCS_PAGES`, audience tiers, prose rendering pipeline) | | `/pubsub` | pubsub.py - **database-free** publish/subscribe bus. `WS /pubsub/ws` (subscribe/unsubscribe/publish frames, `foo.*` wildcards), `POST /pubsub/publish` + `GET /pubsub/topics` (admin/internal). Lock-owner-gated WS (`4013` retry) so subscribers converge on one worker; topic authz in `services/pubsub/policy.py` (`user.{uid}.*` private, `public.*` shared, admin/internal anywhere, guests opt-in). In-process `services.pubsub.publish(topic, data)` for backends; frontend `app.pubsub`. See `devplacepy/services/pubsub/CLAUDE.md` | | (none) | seo.py - `/robots.txt`, `/sitemap.xml` | @@ -250,6 +251,14 @@ All SEO features are implemented across the following locations: - `devplacepy/seo.py` - JSON-LD schema generators (WebSite, BreadcrumbList, DiscussionForumPosting, ProfilePage, SoftwareApplication), meta description truncation, schema combiner, sitemap XML generator - `routers/seo.py` - robots.txt and sitemap.xml routes +### DiscussionForumPosting nested comments + +`discussion_forum_posting(post, author, comment_count, star_count, base_url, comments=None)` embeds up to `MAX_SCHEMA_COMMENTS` (20) of the post's comments as nested `comment: [{"@type": "Comment", "text", "author", "datePublished"}, ...]` entities - not just the aggregate `CommentAction` `InteractionCounter`, which stays for the total count. `seo.comment_schema_list(comment_tree, base_url)` flattens the already-loaded comment tree (`content.load_detail`'s `detail["comments"]`, the same nested `{comment, author, children}` shape `_comment.html` renders) depth-first up to the cap - it does not re-query the database. `posts.py::view_post` is the only call site; a future post-like discussion surface (project/gist/news comments) can reuse `comment_schema_list` the same way once/if it gets a `DiscussionForumPosting` schema of its own. + +### Topic category pages (`/topics`) + +`routers/topics.py` gives the feed's `TOPICS` filter (`constants.py`) real, independently-crawlable pages instead of only a `?topic=` query param (whose canonical collapses back to bare `/feed` - see `base_seo_context`, `canonical = f"{base}{request.url.path}"`, which drops the query string on purpose). `GET /topics` is a hub linking every topic (with a live post count); `GET /topics/{topic}` is a full post listing for that topic, built from the exact same `get_feed_posts`/`enrich_post_cards` pair `feed.py` uses (`enrich_post_cards` was extracted out of `feed_page` specifically so this page is not a second, drifting copy of the attachments/reactions/bookmarks/poll/war enrichment loop). Each topic page gets its own canonical URL, unique title/description, breadcrumbs (Home > Topics > {label}), a `rel=next` link when paginated (`list_page_seo`/`next_page_url`, same mechanism as `/feed`/`/news`), and a real crawlable `_load_more.html` link (not JS-only infinite scroll) for reaching older posts. Both `/topics` and every `/topics/{topic}` are in `sitemap.xml`. + ### SEO template context - Every router passes `page_title`, `meta_description`, `meta_robots`, `canonical_url`, `og_title`, `og_description`, `og_image`, `og_type`, `breadcrumbs`, `page_schema` via `base_seo_context()` - Auth pages: `noindex,nofollow` diff --git a/devplacepy/routers/feed.py b/devplacepy/routers/feed.py index 7a068cf..af8a167 100644 --- a/devplacepy/routers/feed.py +++ b/devplacepy/routers/feed.py @@ -81,21 +81,7 @@ def get_feed_posts( return result, next_cursor -@router.get("", response_class=HTMLResponse) -async def feed_page( - request: Request, - tab: str = "all", - topic: str = None, - search: str = "", - before: str = None, -): - user = get_current_user(request) - posts, next_cursor = get_feed_posts(user, tab, topic, search, before) - stats = get_site_stats() - top_authors = get_top_authors(5) - daily_topic = get_daily_topic() - online_users = presence.online_users() - +def enrich_post_cards(posts, user): post_uids_list = [item["post"]["uid"] for item in posts] attachments_map = get_attachments_batch("post", post_uids_list) recent_comments = get_recent_comments_by_post_uids(post_uids_list, 3, user) @@ -113,6 +99,25 @@ async def feed_page( item["bookmarked"] = uid in bookmark_set item["poll"] = polls_map.get(uid) item["war"] = wars_map.get(uid) + return posts + + +@router.get("", response_class=HTMLResponse) +async def feed_page( + request: Request, + tab: str = "all", + topic: str = None, + search: str = "", + before: str = None, +): + user = get_current_user(request) + posts, next_cursor = get_feed_posts(user, tab, topic, search, before) + stats = get_site_stats() + top_authors = get_top_authors(5) + daily_topic = get_daily_topic() + online_users = presence.online_users() + + posts = enrich_post_cards(posts, user) seo_ctx = list_page_seo( request, diff --git a/devplacepy/routers/messages.py b/devplacepy/routers/messages.py index a309692..452ec42 100644 --- a/devplacepy/routers/messages.py +++ b/devplacepy/routers/messages.py @@ -20,7 +20,6 @@ from devplacepy.templating import clear_messages_cache from devplacepy.utils import ( require_user, time_ago, - is_admin, _user_from_session, _user_from_api_key, ) @@ -31,6 +30,7 @@ from devplacepy.services import presence from devplacepy.services.audit import record as audit from devplacepy.services.correction import PENDING_SCOPE_KEY from devplacepy.dependencies import json_or_form +from devplacepy.services.moderation.screening import ContentRefused from devplacepy.services.messaging import ( issue_ticket, message_frame, @@ -38,6 +38,7 @@ from devplacepy.services.messaging import ( message_relay, persist_message, redeem_ticket, + stamp_content_revision, ) logger = logging.getLogger(__name__) @@ -113,22 +114,38 @@ def get_conversations(user_uid: str): conv.pop("other_uid", None) return conversations -def get_conversation_messages(user_uid: str, other_uid: str): +def get_conversation_messages(user_uid: str, other_uid: str, before: str = ""): if other_uid in get_blocked_uids(user_uid): return [], None if "messages" not in db.tables: return [], get_users_by_uids([other_uid]).get(other_uid) - msgs = list( - db.query( - "SELECT * FROM messages" - " WHERE (sender_uid = :me AND receiver_uid = :other)" - " OR (sender_uid = :other AND receiver_uid = :me)" - " ORDER BY created_at DESC, id DESC LIMIT :lim", - me=user_uid, - other=other_uid, - lim=CONVERSATION_MESSAGE_LIMIT, + before = (before or "").strip() + if before: + msgs = list( + db.query( + "SELECT * FROM messages" + " WHERE ((sender_uid = :me AND receiver_uid = :other)" + " OR (sender_uid = :other AND receiver_uid = :me))" + " AND created_at < :before" + " ORDER BY created_at DESC, id DESC LIMIT :lim", + me=user_uid, + other=other_uid, + before=before, + lim=CONVERSATION_MESSAGE_LIMIT, + ) + ) + else: + msgs = list( + db.query( + "SELECT * FROM messages" + " WHERE (sender_uid = :me AND receiver_uid = :other)" + " OR (sender_uid = :other AND receiver_uid = :me)" + " ORDER BY created_at DESC, id DESC LIMIT :lim", + me=user_uid, + other=other_uid, + lim=CONVERSATION_MESSAGE_LIMIT, + ) ) - ) msgs.reverse() user_ids = list({m["sender_uid"] for m in msgs} | {other_uid}) @@ -158,7 +175,9 @@ def get_conversation_messages(user_uid: str, other_uid: str): return result, other_user @router.get("", response_class=HTMLResponse) -async def messages_page(request: Request, with_uid: str = None, search: str = ""): +async def messages_page( + request: Request, with_uid: str = None, search: str = "", before: str = "" +): user = require_user(request) conversations = get_conversations(user["uid"]) @@ -174,25 +193,28 @@ async def messages_page(request: Request, with_uid: str = None, search: str = "" other_online = False other_last_seen = None if with_uid: - messages, other_user = get_conversation_messages(user["uid"], with_uid) - mark_conversation_read(user["uid"], with_uid) - mark_notifications_read_by_target( - user["uid"], f"/messages?with_uid={with_uid}" + messages, other_user = get_conversation_messages( + user["uid"], with_uid, before=before ) current_conversation = with_uid other_online = presence.is_online(other_user) other_last_seen = other_user.get("last_seen") if other_user else None - audit.record( - request, - "message.read_on_view", - user=user, - target_type="user", - target_uid=with_uid, - target_label=other_user.get("username") if other_user else None, - metadata={"message_count": len(messages)}, - summary=f"{user['username']} read messages from {other_user.get('username') if other_user else with_uid}", - links=[audit.target("user", with_uid, other_user.get("username") if other_user else None)], - ) + if not before: + mark_conversation_read(user["uid"], with_uid) + mark_notifications_read_by_target( + user["uid"], f"/messages?with_uid={with_uid}" + ) + audit.record( + request, + "message.read_on_view", + user=user, + target_type="user", + target_uid=with_uid, + target_label=other_user.get("username") if other_user else None, + metadata={"message_count": len(messages)}, + summary=f"{user['username']} read messages from {other_user.get('username') if other_user else with_uid}", + links=[audit.target("user", with_uid, other_user.get("username") if other_user else None)], + ) seo_ctx = base_seo_context( request, @@ -259,7 +281,7 @@ async def send_message(request: Request, data: Annotated[MessageForm, Depends(js return action_result(request, "/messages") ai_processed = await _finalize_and_broadcast( - user, message, request, client_id=data.client_id + user, message, request, client_id=data.client_id, wait_ai=True ) frame = message_frame( message, user.get("username", ""), data.client_id, @@ -271,31 +293,107 @@ async def send_message(request: Request, data: Annotated[MessageForm, Depends(js async def broadcast_message( sender: dict, message: dict, client_id: Optional[str] = None, - ai_processed: bool = False, + ai_processed: bool = False, ai_pending: bool = False, ) -> None: frame = message_frame( message, sender.get("username", ""), client_id, sender_role=sender.get("role"), ai_processed=ai_processed, ) - message_hub.mark_delivered(message["uid"]) + frame["ai_pending"] = ai_pending + if not ai_processed: + message_hub.mark_delivered(message["uid"]) targets = [message["sender_uid"], message["receiver_uid"]] await message_hub.send_to_users(targets, frame) +async def _await_ai_and_push( + sender: dict, message: dict, pending: list, client_id: Optional[str] +) -> bool: + await asyncio.gather(*pending, return_exceptions=True) + pending.clear() + row = get_table("messages").find_one(uid=message["uid"]) + if not row: + return False + changed = row.get("content") != message.get("content") + if changed: + message["content"] = row["content"] + stamp_content_revision(message["uid"]) + await broadcast_message(sender, message, client_id, ai_processed=True) + return changed + async def _finalize_and_broadcast( - sender: dict, message: dict, request: object, client_id: Optional[str] = None + sender: dict, + message: dict, + request: object, + client_id: Optional[str] = None, + wait_ai: bool = True, ) -> bool: - message_hub.mark_delivered(message["uid"]) scope = getattr(request, "scope", None) pending = scope.get(PENDING_SCOPE_KEY) if scope is not None else None - ai_processed = bool(pending) - if pending: - await asyncio.gather(*pending, return_exceptions=True) - pending.clear() - row = get_table("messages").find_one(uid=message["uid"]) - if row: - message["content"] = row["content"] - await broadcast_message(sender, message, client_id, ai_processed=ai_processed) - return ai_processed + has_pending = bool(pending) + await broadcast_message( + sender, message, client_id, ai_processed=False, ai_pending=has_pending + ) + if not pending: + return False + if wait_ai: + return await _await_ai_and_push(sender, message, pending, client_id) + asyncio.create_task(_await_ai_and_push(sender, message, pending, client_id)) + return False + +SYNC_LIMIT = 200 + + +async def _sync_missed(user_uid: str, data: dict, websocket: WebSocket) -> None: + since = str(data.get("since") or "").strip() + with_uid = str(data.get("with_uid") or "").strip() + if not since or "messages" not in db.tables: + return + if with_uid: + rows = list( + db.query( + "SELECT * FROM messages" + " WHERE ((sender_uid = :me AND receiver_uid = :other)" + " OR (sender_uid = :other AND receiver_uid = :me))" + " AND (created_at > :since" + " OR (updated_at IS NOT NULL AND updated_at > :since))" + " ORDER BY created_at ASC, id ASC LIMIT :lim", + me=user_uid, + other=with_uid, + since=since, + lim=SYNC_LIMIT, + ) + ) + else: + rows = list( + db.query( + "SELECT * FROM messages" + " WHERE (sender_uid = :me OR receiver_uid = :me)" + " AND (created_at > :since" + " OR (updated_at IS NOT NULL AND updated_at > :since))" + " ORDER BY created_at ASC, id ASC LIMIT :lim", + me=user_uid, + since=since, + lim=SYNC_LIMIT, + ) + ) + if not rows: + return + sender_uids = {row["sender_uid"] for row in rows} + senders = get_users_by_uids(list(sender_uids)) if sender_uids else {} + for row in rows: + sender = senders.get(row["sender_uid"]) or {} + frame = message_frame( + dict(row), + sender.get("username", ""), + sender_role=sender.get("role"), + ai_processed=bool(row.get("updated_at")), + ) + try: + await websocket.send_json(frame) + except Exception: # noqa: BLE001 + logger.debug("sync frame dropped for %s", user_uid) + return + def _resolve_ws_user(websocket: WebSocket): user = _user_from_session(websocket) @@ -351,20 +449,34 @@ async def messages_ws(websocket: WebSocket): attachment_uids = [str(a) for a in raw_attachments][:MAX_WS_ATTACHMENTS] if not receiver_uid: continue - message = persist_message( - user, - receiver_uid, - content, - attachment_uids, - request=websocket, - origin="websocket", - ) + try: + message = persist_message( + user, + receiver_uid, + content, + attachment_uids, + request=websocket, + origin="websocket", + ) + except ContentRefused as exc: + await websocket.send_json( + { + "type": "error", + "client_id": client_id, + "text": exc.message, + } + ) + continue if message is None: await websocket.send_json( {"type": "error", "client_id": client_id, "text": "Message not sent."} ) continue - await _finalize_and_broadcast(user, message, websocket, client_id) + await _finalize_and_broadcast( + user, message, websocket, client_id, wait_ai=False + ) + elif kind == "sync": + await _sync_missed(user_uid, data, websocket) elif kind == "typing": receiver_uid = str(data.get("receiver_uid", "")).strip() if receiver_uid: diff --git a/devplacepy/routers/posts.py b/devplacepy/routers/posts.py index 90c1909..898a76e 100644 --- a/devplacepy/routers/posts.py +++ b/devplacepy/routers/posts.py @@ -37,6 +37,7 @@ from devplacepy.seo import ( site_url, website_schema, discussion_forum_posting, + comment_schema_list, ) from devplacepy.attachments import save_inline_image from devplacepy.models import PostForm, PostEditForm @@ -179,7 +180,12 @@ async def view_post(request: Request, post_slug: str): schemas=[ website_schema(base), discussion_forum_posting( - post, author, comment_count, detail["star_count"], base + post, + author, + comment_count, + detail["star_count"], + base, + comments=comment_schema_list(top_level, base), ), ], ) diff --git a/devplacepy/routers/push.py b/devplacepy/routers/push.py index ad347e3..e8b1b51 100644 --- a/devplacepy/routers/push.py +++ b/devplacepy/routers/push.py @@ -49,17 +49,24 @@ async def push_register(request: Request) -> JSONResponse: if fields is None: return JSONResponse({"error": "Invalid request"}, status_code=400) - _, created = push.register(user["uid"], provider.name, fields) + fields = provider.stamp_registration(fields) + write = push.register(user["uid"], provider.name, fields) - if created: + delivered = None + detail = "" + if write.probe: try: - await push.notify_user(user["uid"], WELCOME_PAYLOAD) + outcome = await push.notify_registration(write.record, WELCOME_PAYLOAD) + delivered = outcome.status == providers.ACCEPTED + detail = outcome.detail except Exception as exc: logger.warning("Welcome push failed for %s: %s", user["uid"], exc) + delivered = False + detail = str(exc) audit.record( request, - "push.subscribe" if created else "push.update", + "push.subscribe" if write.created else "push.update", user=user, target_type="user", target_uid=user["uid"], @@ -69,12 +76,19 @@ async def push_register(request: Request) -> JSONResponse: "endpoint_host": urlparse(fields["endpoint"]).hostname if fields.get("endpoint") else None, - "created": created, + "created": write.created, + "revived": write.revived, + "has_client_id": bool(fields.get("client_id")), }, - summary=f"{user.get('username')} {'registered' if created else 'updated'} a push subscription", + summary=f"{user.get('username')} {'registered' if write.created else 'updated'} a push subscription", links=[audit.target("user", user["uid"], user.get("username"))], ) - return JSONResponse({"registered": True}) + payload: dict = {"registered": True} + if delivered is not None: + payload["delivered"] = delivered + if detail and not delivered: + payload["error"] = detail + return JSONResponse(payload) @router.get("/service-worker.js") diff --git a/devplacepy/routers/tools/deepsearch.py b/devplacepy/routers/tools/deepsearch.py index 1d118d1..08523e7 100644 --- a/devplacepy/routers/tools/deepsearch.py +++ b/devplacepy/routers/tools/deepsearch.py @@ -243,6 +243,7 @@ def _session_context(request: Request, uid: str, job: dict, session: dict) -> di "sources": report.get("sources", []), "findings": report.get("findings", []), "timeline": report.get("timeline", []), + "follow_up_questions": report.get("follow_up_questions", []), "chat_ws_url": f"/tools/deepsearch/{uid}/chat" if done else None, "export_md_url": f"/tools/deepsearch/{uid}/export.md" if done else None, "export_json_url": f"/tools/deepsearch/{uid}/export.json" if done else None, diff --git a/devplacepy/routers/topics.py b/devplacepy/routers/topics.py new file mode 100644 index 0000000..960edf5 --- /dev/null +++ b/devplacepy/routers/topics.py @@ -0,0 +1,84 @@ +# retoor + +from fastapi import APIRouter, Request +from fastapi.responses import HTMLResponse +from devplacepy.constants import TOPICS, TOPIC_LABELS +from devplacepy.database import get_table +from devplacepy.routers.feed import get_feed_posts, enrich_post_cards +from devplacepy.utils import get_current_user, not_found +from devplacepy.seo import list_page_seo, next_page_url +from devplacepy.responses import respond +from devplacepy.schemas import TopicOut, TopicsHubOut + +router = APIRouter() + + +@router.get("", response_class=HTMLResponse) +async def topics_hub(request: Request): + user = get_current_user(request) + posts_table = get_table("posts") + topics = [ + { + "key": topic, + "label": TOPIC_LABELS.get(topic, topic.title()), + "post_count": posts_table.count(topic=topic, deleted_at=None), + } + for topic in TOPICS + ] + + seo_ctx = list_page_seo( + request, + title="Topics", + description="Browse DevPlace posts by topic: devlog, showcase, questions, rants, fun, and more.", + breadcrumbs=[ + {"name": "Home", "url": "/feed"}, + {"name": "Topics", "url": "/topics"}, + ], + ) + return respond( + request, + "topics.html", + { + **seo_ctx, + "request": request, + "user": user, + "topics": topics, + }, + model=TopicsHubOut, + ) + + +@router.get("/{topic}", response_class=HTMLResponse) +async def topic_page(request: Request, topic: str, before: str = None): + if topic not in TOPICS: + raise not_found("Topic not found") + user = get_current_user(request) + posts, next_cursor = get_feed_posts(user, "all", topic, "", before) + posts = enrich_post_cards(posts, user) + label = TOPIC_LABELS.get(topic, topic.title()) + + seo_ctx = list_page_seo( + request, + title=f"{label} posts", + description=f"Browse {label.lower()} posts from developers on DevPlace.", + breadcrumbs=[ + {"name": "Home", "url": "/feed"}, + {"name": "Topics", "url": "/topics"}, + {"name": label, "url": f"/topics/{topic}"}, + ], + next_url=next_page_url(request, next_cursor), + ) + return respond( + request, + "topic.html", + { + **seo_ctx, + "request": request, + "user": user, + "posts": posts, + "topic": topic, + "topic_label": label, + "next_cursor": next_cursor, + }, + model=TopicOut, + ) diff --git a/devplacepy/schemas/__init__.py b/devplacepy/schemas/__init__.py index 0e36118..b63d03a 100644 --- a/devplacepy/schemas/__init__.py +++ b/devplacepy/schemas/__init__.py @@ -52,6 +52,9 @@ from devplacepy.schemas.listings import ( ProjectsOut, SavedItemOut, SavedOut, + TopicOut, + TopicSummaryOut, + TopicsHubOut, ) from devplacepy.schemas.profile import ( MediaItemOut, diff --git a/devplacepy/schemas/game.py b/devplacepy/schemas/game.py index 75aae19..d4cff9e 100644 --- a/devplacepy/schemas/game.py +++ b/devplacepy/schemas/game.py @@ -13,8 +13,11 @@ class GameCropOut(_Out): reward_coins: int = 0 reward_xp: int = 0 min_level: int = 1 + min_mastery: int = 0 grow_seconds: int = 0 locked: bool = False + locked_reason: str = "" + locked_text: str = "" market_state: str = "normal" diff --git a/devplacepy/schemas/jobs.py b/devplacepy/schemas/jobs.py index a4c7ce7..d2aa9b9 100644 --- a/devplacepy/schemas/jobs.py +++ b/devplacepy/schemas/jobs.py @@ -132,6 +132,7 @@ class DeepsearchSessionOut(_Out): sources: list = [] findings: list = [] timeline: list = [] + follow_up_questions: list = [] chat_ws_url: Optional[str] = None export_md_url: Optional[str] = None export_json_url: Optional[str] = None diff --git a/devplacepy/schemas/listings.py b/devplacepy/schemas/listings.py index b1ed184..b39e2ee 100644 --- a/devplacepy/schemas/listings.py +++ b/devplacepy/schemas/listings.py @@ -108,6 +108,23 @@ class SavedItemOut(_Out): time_ago: Optional[str] = None +class TopicOut(_Out): + posts: list[FeedItemOut] = [] + topic: str = "" + topic_label: str = "" + next_cursor: Optional[str] = None + + +class TopicSummaryOut(_Out): + key: str = "" + label: str = "" + post_count: int = 0 + + +class TopicsHubOut(_Out): + topics: list[TopicSummaryOut] = [] + + class FeedOut(_Out): posts: list[FeedItemOut] = [] current_tab: Optional[str] = None diff --git a/devplacepy/seo.py b/devplacepy/seo.py index 2f1c4f1..063fd7e 100644 --- a/devplacepy/seo.py +++ b/devplacepy/seo.py @@ -91,7 +91,41 @@ def breadcrumb_schema(items, base_url): } -def discussion_forum_posting(post, author, comment_count, star_count, base_url): +MAX_SCHEMA_COMMENTS = 20 + + +def comment_schema(comment_item, base_url): + comment = comment_item["comment"] + author = comment_item.get("author") + return { + "@type": "Comment", + "text": truncate(plain_markdown(comment.get("content", "")), 300), + "author": { + "@type": "Person", + "name": author["username"] if author else "Unknown", + "url": f"{base_url}/profile/{author['username']}" if author else "", + }, + "datePublished": comment.get("created_at", ""), + } + + +def comment_schema_list(comment_tree, base_url, limit=MAX_SCHEMA_COMMENTS): + flat = [] + + def walk(items): + for item in items: + if len(flat) >= limit: + return + flat.append(comment_schema(item, base_url)) + walk(item.get("children", [])) + + walk(comment_tree) + return flat + + +def discussion_forum_posting( + post, author, comment_count, star_count, base_url, comments=None +): schema = { "@type": "DiscussionForumPosting", "headline": post.get("title") or "Untitled", @@ -117,6 +151,8 @@ def discussion_forum_posting(post, author, comment_count, star_count, base_url): }, ], } + if comments: + schema["comment"] = comments return schema @@ -421,6 +457,13 @@ def _build_sitemap(base_url): urlset.append(url_element(f"{base_url}/", changefreq="daily", priority="1.0")) urlset.append(url_element(f"{base_url}/feed", changefreq="hourly", priority="0.9")) urlset.append(url_element(f"{base_url}/news", changefreq="hourly", priority="0.9")) + urlset.append(url_element(f"{base_url}/topics", changefreq="daily", priority="0.7")) + from devplacepy.constants import TOPICS + + for topic in TOPICS: + urlset.append( + url_element(f"{base_url}/topics/{topic}", changefreq="daily", priority="0.7") + ) urlset.append( url_element(f"{base_url}/projects", changefreq="daily", priority="0.8") ) diff --git a/devplacepy/services/ai_modifier.py b/devplacepy/services/ai_modifier.py index 1edf639..4bbef68 100644 --- a/devplacepy/services/ai_modifier.py +++ b/devplacepy/services/ai_modifier.py @@ -95,6 +95,10 @@ def _run_modification( if updates: updates["uid"] = uid get_table(table).update(updates, ["uid"]) + if table == "messages": + from devplacepy.services.messaging.persist import push_content_revision + + push_content_revision(uid, ai_processed=True) if table == "users" and user_uid: from devplacepy.utils import clear_user_cache diff --git a/devplacepy/services/correction.py b/devplacepy/services/correction.py index 7e42314..b66c2c1 100644 --- a/devplacepy/services/correction.py +++ b/devplacepy/services/correction.py @@ -202,6 +202,10 @@ def _run_correction( if updates: updates["uid"] = uid get_table(table).update(updates, ["uid"]) + if table == "messages": + from devplacepy.services.messaging.persist import push_content_revision + + push_content_revision(uid, ai_processed=True) if table == "users" and user_uid: from devplacepy.utils import clear_user_cache diff --git a/devplacepy/services/deepsearch/export.py b/devplacepy/services/deepsearch/export.py index 71771db..f4a5004 100644 --- a/devplacepy/services/deepsearch/export.py +++ b/devplacepy/services/deepsearch/export.py @@ -70,6 +70,13 @@ def to_markdown(report: dict) -> str: url = source.get("url", "") lines.append(f"{index}. [{title}]({url})") lines.append("") + follow_ups = report.get("follow_up_questions") or [] + if follow_ups: + lines.append("## Ask next") + lines.append("") + for question in follow_ups: + lines.append(f"- {question}") + lines.append("") return "\n".join(lines) @@ -113,6 +120,12 @@ def _html_document(report: dict) -> str: title = html.escape(source.get("title") or source.get("url", "")) parts.append(f"
  • {title}
    • ") parts.append("") + follow_ups = report.get("follow_up_questions") or [] + if follow_ups: + parts.append("

    Ask next

      ") + for question in follow_ups: + parts.append(f"
    • {html.escape(question)}
      • ") + parts.append("
    ") parts.append("") return "".join(parts) diff --git a/devplacepy/services/deepsearch/store.py b/devplacepy/services/deepsearch/store.py index 129a253..9389c51 100644 --- a/devplacepy/services/deepsearch/store.py +++ b/devplacepy/services/deepsearch/store.py @@ -34,6 +34,7 @@ class Chunk: position: int = 0 score: float = 0.0 metadata: dict = field(default_factory=dict) + embedding: list[float] | None = None def _tokenize(text: str) -> list[str]: @@ -176,16 +177,18 @@ class VectorStore: query_embeddings=[query_vector], n_results=top_k, where=where or None, - include=["documents", "metadatas", "distances"], + include=["documents", "metadatas", "distances", "embeddings"], ) ids = (result.get("ids") or [[]])[0] documents = (result.get("documents") or [[]])[0] metadatas = (result.get("metadatas") or [[]])[0] distances = (result.get("distances") or [[]])[0] + embeddings = (result.get("embeddings") or [[]])[0] chunks: list[Chunk] = [] for index, uid in enumerate(ids): meta = metadatas[index] if index < len(metadatas) else {} distance = distances[index] if index < len(distances) else 1.0 + vector = embeddings[index] if index < len(embeddings) else None chunks.append( Chunk( uid=uid, @@ -197,6 +200,7 @@ class VectorStore: position=int(meta.get("position", 0) or 0), score=1.0 - float(distance), metadata=dict(meta), + embedding=[float(v) for v in vector] if vector is not None else None, ) ) return chunks diff --git a/devplacepy/services/devii/actions/catalog/posts.py b/devplacepy/services/devii/actions/catalog/posts.py index d2fd87e..5bb69c5 100644 --- a/devplacepy/services/devii/actions/catalog/posts.py +++ b/devplacepy/services/devii/actions/catalog/posts.py @@ -20,6 +20,25 @@ POSTS_ACTIONS: tuple[Action, ...] = ( query("before", "Pagination cursor."), ), ), + Action( + name="view_topics", + method="GET", + path="/topics", + summary="View the topics hub - every post topic with its live post count", + requires_auth=False, + params=(), + ), + Action( + name="view_topic", + method="GET", + path="/topics/{topic}", + summary="View one topic's post listing (e.g. devlog, showcase, question, rant, fun, random, politics)", + requires_auth=False, + params=( + path("topic", "Topic key, one of the values TOPICS lists."), + query("before", "Pagination cursor."), + ), + ), Action( name="create_post", method="POST", diff --git a/devplacepy/services/devrant/notifications.py b/devplacepy/services/devrant/notifications.py index 4dd2e03..c241430 100644 --- a/devplacepy/services/devrant/notifications.py +++ b/devplacepy/services/devrant/notifications.py @@ -8,6 +8,7 @@ from devplacepy.services.devrant.ids import to_unix, now_unix DEVRANT_TYPE_MAP = { "comment": "comment_discuss", "reply": "comment_discuss", + "thread": "comment_discuss", "mention": "comment_mention", "vote": "rant_vote", "follow": "rant_sub", diff --git a/devplacepy/services/game/economy.py b/devplacepy/services/game/economy.py index 55cf484..15a6800 100644 --- a/devplacepy/services/game/economy.py +++ b/devplacepy/services/game/economy.py @@ -213,6 +213,18 @@ def unlocked_crops( ] +def crop_lock_reason( + crop: Crop, level: int, mastery_earned: int, era_locked: bool = False +) -> tuple[str, str] | None: + if crop.min_level > level: + return "level", f"unlocks at level {crop.min_level}" + if crop.min_mastery > mastery_earned: + return "mastery", "unlocks after reaching Mastery (Refactor to prestige 50)" + if era_locked: + return "era", "is not available right now" + return None + + def crop_payload( crop: Crop, ci_tier: int, @@ -234,6 +246,7 @@ def crop_payload( market_state = "saturated" elif market_factor > 1.0: market_state = "boosted" + lock = crop_lock_reason(crop, level, mastery_earned, era_locked) return { "key": crop.key, "name": crop.name, @@ -244,10 +257,13 @@ def crop_payload( ), "reward_xp": crop.reward_xp, "min_level": crop.min_level, + "min_mastery": crop.min_mastery, "grow_seconds": grow_seconds_for( crop, ci_tier, growth_level, legacy_speed_level, registry_boost ), - "locked": crop.min_level > level or crop.min_mastery > mastery_earned or era_locked, + "locked": lock is not None, + "locked_reason": lock[0] if lock else "", + "locked_text": f"{crop.name} {lock[1]}." if lock else "", "market_state": market_state, } diff --git a/devplacepy/services/game/store/actions.py b/devplacepy/services/game/store/actions.py index 29bed23..c86ef31 100644 --- a/devplacepy/services/game/store/actions.py +++ b/devplacepy/services/game/store/actions.py @@ -44,12 +44,12 @@ def plant(user: dict, slot: int, crop_key: str) -> dict: if not crop: raise GameError("Unknown crop type.") progress = economy.level_progress(int(farm.get("xp", 0))) - if crop.min_level > progress["level"]: - raise GameError(f"{crop.name} unlocks at level {crop.min_level}.") - if crop.min_mastery > _lvl(farm, "mastery_points_earned_total"): - raise GameError(f"{crop.name} unlocks after reaching Mastery.") - if crop.era_key and crop.era_key != active_era_name(): - raise GameError(f"{crop.name} is not available right now.") + era_locked = bool(crop.era_key and crop.era_key != active_era_name()) + lock = economy.crop_lock_reason( + crop, progress["level"], _lvl(farm, "mastery_points_earned_total"), era_locked + ) + if lock: + raise GameError(f"{crop.name} {lock[1]}.") plot = _plot_at(farm["uid"], slot) if not plot: raise GameError("That plot does not exist.") diff --git a/devplacepy/services/jobs/CLAUDE.md b/devplacepy/services/jobs/CLAUDE.md index de2b001..97a0ef2 100644 --- a/devplacepy/services/jobs/CLAUDE.md +++ b/devplacepy/services/jobs/CLAUDE.md @@ -98,6 +98,9 @@ The public **Tools -> DeepSearch** researcher is a multi-agent deep web research - **Frontend** (do not hand-roll): `DeepsearchTool.js` (`app.deepsearchTool`) drives the form via `Http.send`, watches `DeepsearchProgressSocket` (cloned from `SeoProgressSocket`, 4013 retry), and wires pause/resume/cancel. `static/css/deepsearch.css` uses the design tokens and is mobile-responsive. - **Progress frame protocol (append-only, the worker<->JS contract):** `phases.py` is the single source of truth for phase identity, shared by `worker.py` (emit) and `DeepsearchTool.js` (render). `PHASE_ORDER = [planning, searching, crawling, indexing, analysis, synthesis]`; `worker._stage(stage, message, phase)` emits BOTH the legacy `stage` frame (byte-identical to before) AND a parallel `phase` frame `{phase, index, total, label}` so the timeline strip advances. The first emitted frame carries `version:1`. Every other frame type and its keys: `substep` (planning angles, `phase`+`message`; also emitted by analysis grounding), `queries`, `candidates`, `rsearch`, `progress` (`done`/`total`/`url`/`depth`), `page_loaded` (now `source`/`render`/`depth`/`elapsed_ms`/`done`/`total`), `page_cached`/`page_skipped`/`page_duplicate` (now `reason`/`elapsed_ms`), `embed_batch` (`batch`/`total_batches`/`backend`/`done`/`total`, emitted before AND after each batch), `embed_done` (`backend`/`chunk_count`), `agent` (`agent` one of summarizer|extractor|linker, `stage`/`status` start|done|failed, with `elapsed_ms`/`tokens_in`/`tokens_out` on done), `report_ready` (now also `synthesis`), `done` (`session_url`), `failed` (`message`). **The contract is append-only: never rename or drop a frame type**; `service._run_worker` pumps every stdout line into the `ProgressHub` untouched, so new frame types reach the WS with no handler change. `tests/api/tools/deepsearch/index.py` is the append-only regression guard. - **RAG-audit hardening (engine correctness, no route/schema change):** (1) **Embedding-dimension consistency** - gateway embeddings carry provider-native dims while `local_embed` is fixed 256-dim; the worker `_index_chunks` now decides the backend ONCE per job (the first gateway failure or non-gateway result forces local for ALL remaining batches), and `VectorStore.add` drops any vector whose length differs from the collection's established dim, so one collection never mixes dims (cosine search across mixed dims is corrupt). `embeddings.EmbedResult.dims` and `VectorStore.dims` (lazily probed from the collection) expose the dimension; `chat.retrieve` re-embeds the query locally and skips retrieval if it still cannot match the stored dim. (2) **Citation grounding** - `orchestrate` drops any finding with no citations, and the summarizer prompt forbids uncited claims and treats the QUESTION as data not an instruction (prompt-injection reduction via `_sanitize_question`); `chat._strip_unmatched_markers` removes any inline `[n]` marker that does not map to an emitted citation. (3) **Confidence calibration** - when only a single domain was crawled, `orchestrate` caps confidence by the source-diversity-derived ceiling (overconfidence guard). (4) `EmbeddingCache` is bounded at `EMBED_CACHE_MAX` to cap in-memory growth. +- **Iterative gap-filling refinement rounds (`worker.py`, `enhance.plan_followup_queries`).** A single plan-search-crawl pass under-covers a broad question the way a real deep-research agent never would - real systems iterate (search, read, decide what is still missing, search again). After the initial crawl, `worker._run` loops up to `MAX_REFINE_ROUNDS = 2` times, each iteration asking `plan_followup_queries(query, covered_titles, api_key, emit)` (a small planner call mirroring `enhance.plan_queries`, given only the TITLES already gathered) for 0-3 extra queries that would surface uncovered information; an empty reply (planner unavailable, or the sources already look sufficient) stops the loop immediately - it never forces an extra round. Each round is bounded by the remaining `max_pages` budget, follows-links-off (`depth=1`, cheaper than the initial crawl's link-following), and shares `outcome.seen_hashes` with the initial crawl via `crawl()`'s new `seen_hashes` param so content dedup spans rounds; new pages are appended to the same `outcome.pages` list before indexing, so indexing/synthesis are unchanged. **This is an internal control-flow signal only - it produces MORE QUERIES, never a user-visible "gaps" report or critic verdict, so it does not reintroduce the removed critic agent** (see below). Frames reuse the existing `substep`/`queries`/`candidates` types (a `round` key is additive). +- **Diversity-aware retrieval via MMR (`orchestrate._mmr_select`, `store.py`).** `VectorStore._vector_search_sync` now includes `"embeddings"` in the Chroma query and carries each result's vector on `Chunk.embedding`. `orchestrate._retrieve_chunks` pools every per-sub-query hit (deduped by uid, capped `MMR_POOL_MAX = 60`) instead of round-robin-truncating it directly, then `_mmr_select` (Maximal Marginal Relevance, `MMR_LAMBDA = 0.7`) greedily picks the final `CONTEXT_CHUNKS_MAX` chunks trading relevance to the question against redundancy with what is already selected - this is the standard fix for the case where many sub-queries all surface the same page's top passage, which used to crowd out real topic diversity in the LLM's context window. Falls back to the original order when any candidate lacks an embedding (old data, embedding failure). +- **Suggested follow-up questions ("Ask next"), Perplexity-style.** After a successful agents-path synthesis, `orchestrate._suggest_followups(question, summary, api_key)` asks one small, separate completion for 3-4 natural next questions a reader would ask, stored on `Orchestration.follow_up_questions` / `report["follow_up_questions"]` / `DeepsearchSessionOut.follow_up_questions`. Fails soft to `[]` (no heuristic fallback needed - it is a nice-to-have, not part of answering the question) and is never generated on the heuristic path. Rendered as clickable chips in `deepsearch_session.html` (`.ds-followup-chip`, shown only when the chat is available) that call the new `AppDeepsearchChat.ask(text)` method, feeding the question straight into the grounded RAG chat. Included in the markdown/HTML/PDF exports under "Ask next". **This is a different feature from the removed critic/gaps agent**: it never critiques or lists what the report is missing, it only suggests what to ask next, exactly like Perplexity's related-questions row. - Devii tools `deepsearch`/`deepsearch_status`/`deepsearch_session` (public); docs `tools-deepsearch`; CLI `devplace deepsearch prune|clear`; audit `deepsearch.run.request|complete|failed` + `deepsearch.chat` (category `tools`). **New dependencies:** `chromadb`, `weasyprint`, `pypdf` (all unpinned). New runtime dirs `config.DEEPSEARCH_DIR`/`DEEPSEARCH_CHROMA_DIR` are registered in `DATA_PATHS`. **Add a dedicated nginx WS `location` for `/tools/deepsearch/{uid}/ws` and `/chat`** above `location /` for production, like the SEO and Devii sockets. ## AI Usage Analyzer tool - IsslopService (kind `isslop`, `services/jobs/isslop/`, `routers/tools/isslop.py`) diff --git a/devplacepy/services/jobs/deepsearch/crawl.py b/devplacepy/services/jobs/deepsearch/crawl.py index 74d917f..65d32cb 100644 --- a/devplacepy/services/jobs/deepsearch/crawl.py +++ b/devplacepy/services/jobs/deepsearch/crawl.py @@ -292,10 +292,11 @@ async def crawl( should_stop: Callable[[], Awaitable[bool]], query: str = "", depth: int = 1, + seen_hashes: set[str] | None = None, ) -> CrawlOutcome: from playwright.async_api import async_playwright - outcome = CrawlOutcome() + outcome = CrawlOutcome(seen_hashes=seen_hashes if seen_hashes is not None else set()) fetched = 0 seen_urls = {candidate["url"] for candidate in candidates} level_candidates = list(candidates) diff --git a/devplacepy/services/jobs/deepsearch/enhance.py b/devplacepy/services/jobs/deepsearch/enhance.py index 19a3a53..d1cea99 100644 --- a/devplacepy/services/jobs/deepsearch/enhance.py +++ b/devplacepy/services/jobs/deepsearch/enhance.py @@ -10,19 +10,29 @@ from typing import Callable from devplacepy import stealth from devplacepy.config import INTERNAL_GATEWAY_URL, INTERNAL_MODEL -from .phases import PHASE_PLANNING +from .phases import PHASE_PLANNING, PHASE_SEARCHING logger = logging.getLogger(__name__) ENHANCE_TIMEOUT_SECONDS = 90.0 ENHANCE_MAX_TOKENS = 400 MAX_SUBQUERIES = 6 +MAX_FOLLOWUP_QUERIES = 3 +MAX_COVERED_TITLES = 40 PLANNER_PROMPT = ( "You plan web research. Given a research question, produce a JSON object with one " "key 'queries': an array of 3 to 6 concise, diverse web search queries that together " "cover the question from multiple angles. Return ONLY the JSON object, no prose." ) +FOLLOWUP_PROMPT = ( + "You plan additional web research to fill coverage gaps. Given a research QUESTION " + "and the TITLES of sources already gathered, decide whether more searching would " + "surface information the current sources do not yet cover. Produce a JSON object " + "with one key 'queries': an array of 0 to 3 concise web search queries targeting " + "what is missing. Return an empty array when the sources already cover the question " + "well. Return ONLY the JSON object, no prose." +) def _fallback(query: str) -> list[str]: @@ -115,3 +125,55 @@ async def plan_queries( } ) return fallback + + +async def plan_followup_queries( + query: str, + covered_titles: list[str], + api_key: str, + emit: Callable[[dict], None] = _noop, +) -> list[str]: + if not covered_titles: + return [] + titles_block = "\n".join(f"- {title}" for title in covered_titles[:MAX_COVERED_TITLES]) + payload = { + "model": INTERNAL_MODEL, + "messages": [ + {"role": "system", "content": FOLLOWUP_PROMPT}, + { + "role": "user", + "content": f"QUESTION: {query}\n\nALREADY COVERED (source titles):\n{titles_block}", + }, + ], + "max_tokens": ENHANCE_MAX_TOKENS, + "temperature": 0.3, + } + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + "X-App-Reference": "devplace-deepsearch-v-1-0-0", + } + try: + async with stealth.stealth_async_client(timeout=ENHANCE_TIMEOUT_SECONDS) as client: + response = await client.post( + INTERNAL_GATEWAY_URL, json=payload, headers=headers + ) + if response.status_code >= 400: + raise RuntimeError(f"followup planner gateway returned {response.status_code}") + data = response.json() + content = ( + (data.get("choices") or [{}])[0].get("message", {}).get("content") or "" + ) + parsed = _parse(content)[:MAX_FOLLOWUP_QUERIES] + if parsed: + emit( + { + "type": "substep", + "phase": PHASE_SEARCHING, + "message": f"Planned {len(parsed)} follow-up queries to fill gaps", + } + ) + return parsed + except Exception as exc: + logger.info("deepsearch followup planner unavailable, skipping refinement: %s", exc) + return [] diff --git a/devplacepy/services/jobs/deepsearch/orchestrate.py b/devplacepy/services/jobs/deepsearch/orchestrate.py index 73778d5..1c2d609 100644 --- a/devplacepy/services/jobs/deepsearch/orchestrate.py +++ b/devplacepy/services/jobs/deepsearch/orchestrate.py @@ -5,6 +5,7 @@ from __future__ import annotations import asyncio import json import logging +import math import re from collections.abc import Callable from dataclasses import dataclass, field @@ -31,6 +32,15 @@ PAGE_EXCERPT_CHARS = 3000 MAX_CONTEXT_CHARS = 36000 SCORE_MAX = 100 CONFIDENCE_BASELINE = 0.35 +MMR_LAMBDA = 0.7 +MMR_POOL_MAX = 60 +FOLLOWUP_QUESTIONS_PROMPT = ( + "Given the QUESTION and the REPORT below, suggest natural follow-up questions a " + "curious reader would ask next. Return ONLY a JSON object with key 'questions': " + "an array of 3 to 4 short, self-contained question strings. No prose." +) +FOLLOWUP_QUESTIONS_MAX_TOKENS = 200 +MAX_FOLLOWUP_QUESTIONS = 4 @dataclass @@ -41,6 +51,7 @@ class Orchestration: source_diversity: float = 0.0 score: int = 0 synthesis: str = "agents" + follow_up_questions: list[str] = field(default_factory=list) def _domain(url: str) -> str: @@ -66,6 +77,42 @@ def _sanitize_question(question: str) -> str: return cleaned[:QUESTION_MAX_CHARS] +def _cosine(a: list[float], b: list[float]) -> float: + if not a or not b or len(a) != len(b): + return 0.0 + dot = sum(x * y for x, y in zip(a, b)) + norm_a = math.sqrt(sum(x * x for x in a)) + norm_b = math.sqrt(sum(y * y for y in b)) + if not norm_a or not norm_b: + return 0.0 + return dot / (norm_a * norm_b) + + +def _mmr_select( + chunks: list, query_vector: list[float], limit: int, lambda_mult: float = MMR_LAMBDA +) -> list: + if not chunks: + return [] + if not query_vector or any(getattr(chunk, "embedding", None) is None for chunk in chunks): + return chunks[:limit] + selected: list = [] + remaining = list(chunks) + while remaining and len(selected) < limit: + best_index = 0 + best_score = float("-inf") + for index, candidate in enumerate(remaining): + relevance = _cosine(candidate.embedding, query_vector) + redundancy = max( + (_cosine(candidate.embedding, chosen.embedding) for chosen in selected), + default=0.0, + ) + mmr_score = lambda_mult * relevance - (1 - lambda_mult) * redundancy + if mmr_score > best_score: + best_score, best_index = mmr_score, index + selected.append(remaining.pop(best_index)) + return selected + + async def _retrieve_chunks(question: str, queries: list[str], store, api_key: str) -> list: texts = [question] for query in queries or []: @@ -88,17 +135,19 @@ async def _retrieve_chunks(question: str, queries: list[str], store, api_key: st ) ) ) - merged: list = [] + pool: list = [] seen: set[str] = set() for tier in zip_longest(*per_query): for chunk in tier: if chunk is None or chunk.uid in seen: continue seen.add(chunk.uid) - merged.append(chunk) - if len(merged) >= CONTEXT_CHUNKS_MAX: - return merged - return merged + pool.append(chunk) + if len(pool) >= MMR_POOL_MAX: + break + if len(pool) >= MMR_POOL_MAX: + break + return _mmr_select(pool, vectors[0] if vectors else [], CONTEXT_CHUNKS_MAX) def _chunk_context(chunks: list, pages: list) -> str: @@ -344,6 +393,29 @@ async def _extract_findings( return [], usage +async def _suggest_followups(question: str, summary: str, api_key: str) -> list[str]: + try: + text, _usage = await _complete( + [ + {"role": "system", "content": FOLLOWUP_QUESTIONS_PROMPT}, + { + "role": "user", + "content": f"QUESTION: {question}\n\nREPORT:\n{summary[:4000]}", + }, + ], + api_key, + FOLLOWUP_QUESTIONS_MAX_TOKENS, + ) + questions = _parse_json(text).get("questions") + if not isinstance(questions, list): + return [] + cleaned = [str(item).strip() for item in questions if str(item).strip()] + return cleaned[:MAX_FOLLOWUP_QUESTIONS] + except Exception as exc: + logger.info("deepsearch follow-up questions unavailable: %s", exc) + return [] + + async def orchestrate( question: str, pages: list, @@ -423,6 +495,7 @@ async def orchestrate( score = int( min(SCORE_MAX, (confidence * 0.5 + diversity * 0.3 + coverage * 0.2) * SCORE_MAX) ) + follow_up_questions = await _suggest_followups(question, summary, api_key) return Orchestration( summary=summary, findings=findings, @@ -430,6 +503,7 @@ async def orchestrate( source_diversity=diversity, score=score, synthesis="agents", + follow_up_questions=follow_up_questions, ) except Exception as exc: logger.warning("deepsearch orchestration failed, using heuristic: %s", exc) diff --git a/devplacepy/services/jobs/deepsearch/worker.py b/devplacepy/services/jobs/deepsearch/worker.py index d5958e4..0edd65c 100644 --- a/devplacepy/services/jobs/deepsearch/worker.py +++ b/devplacepy/services/jobs/deepsearch/worker.py @@ -14,7 +14,7 @@ from devplacepy.utils import generate_uid from .chunking import chunk_text from .crawl import content_hash, crawl, search_queries, url_hash -from .enhance import plan_queries +from .enhance import plan_followup_queries, plan_queries from .orchestrate import orchestrate from .phases import ( PHASE_ANALYSIS, @@ -32,6 +32,8 @@ CONTROL_FILE = "control.json" PAUSE_POLL_SECONDS = 1.0 EMBED_BATCH = 64 FRAME_VERSION = 1 +MAX_REFINE_ROUNDS = 2 +REFINE_CRAWL_DEPTH = 1 _first_frame_sent = False @@ -183,6 +185,45 @@ async def _run(payload: dict, output_dir: Path) -> dict: depth=depth, ) + round_no = 0 + while ( + len(outcome.pages) < max_pages + and round_no < MAX_REFINE_ROUNDS + and not await should_stop() + ): + covered_titles = [page.title for page in outcome.pages if page.title] + followups = await plan_followup_queries(query, covered_titles, api_key, _emit) + if not followups: + break + round_no += 1 + _emit( + { + "type": "substep", + "phase": PHASE_SEARCHING, + "message": f"Refinement round {round_no}: filling gaps with {len(followups)} more queries", + } + ) + _emit({"type": "queries", "queries": followups, "round": round_no}) + more_candidates = await search_queries(followups, _emit) + existing_urls = {page.url for page in outcome.pages} + more_candidates = [c for c in more_candidates if c["url"] not in existing_urls] + if not more_candidates: + break + _emit({"type": "candidates", "count": len(more_candidates), "round": round_no}) + extra = await crawl( + more_candidates, + max_pages - len(outcome.pages), + _emit, + lambda url: url_hash(url) in cached_hashes, + should_stop, + query=query, + depth=REFINE_CRAWL_DEPTH, + seen_hashes=outcome.seen_hashes, + ) + if not extra.pages: + break + outcome.pages.extend(extra.pages) + new_cache = [ { "url_hash": url_hash(page.url), @@ -224,6 +265,7 @@ async def _run(payload: dict, output_dir: Path) -> dict: "confidence": result.confidence, "source_diversity": result.source_diversity, "synthesis": result.synthesis, + "follow_up_questions": result.follow_up_questions, "page_count": len(outcome.pages), "chunk_count": chunk_count, "embed_backend": embed_backend, diff --git a/devplacepy/services/messaging/CLAUDE.md b/devplacepy/services/messaging/CLAUDE.md index 20140b8..c165f7a 100644 --- a/devplacepy/services/messaging/CLAUDE.md +++ b/devplacepy/services/messaging/CLAUDE.md @@ -49,11 +49,11 @@ Because the WS accepts on every worker, a message persisted on worker A must sti ## AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content -`"messages": ("content",)` is in the `CORRECTABLE_FIELDS` registry and `persist_message` invokes `schedule_correction`/`schedule_modification` (sender = the user), so typing `@ai ` in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through `routers/messages._finalize_and_broadcast(sender, message, request, client_id=None)`: it reads any pending SYNC correction/modification futures stashed on `request.scope[PENDING_SCOPE_KEY]` (the same `PENDING_SCOPE_KEY`/`loop.run_in_executor` mechanism the HTTP `await_pending_corrections` middleware uses) into `ai_processed = bool(pending)` **before** awaiting/clearing them, awaits them, RE-READS the message row, and broadcasts the FINAL (corrected/modified) content via `broadcast_message(sender, message, client_id, ai_processed=ai_processed)`. The HTTP `POST /send` and the WS `/ws` send path both call it; the WS path passes `request=websocket` to `persist_message` so the modifier stashes its executor future on the websocket scope and the handler awaits it directly (HTTP middleware does not run for websockets). A message with no `@ai` directive and correction off has nothing pending, so `ai_processed` is `False` and the broadcast is immediate with zero added latency. `broadcast_message` forwards `ai_processed` straight into `message_frame(..., ai_processed=ai_processed)` - see "WS protocol frames" below for the frame shape. This flag is informational only: the server never retains the pre-AI text anywhere (`table.update` overwrites in place), so it exists purely to drive a client-side "Adjusted by AI" affordance, not any kind of diff/revert capability. +`"messages": ("content",)` is in the `CORRECTABLE_FIELDS` registry and `persist_message` invokes `schedule_correction`/`schedule_modification` (sender = the user), so typing `@ai ` in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through `routers/messages._finalize_and_broadcast`: it broadcasts the persisted row immediately (`ai_pending=true` when sync futures were stashed on `request.scope[PENDING_SCOPE_KEY]`), then applies any pending SYNC correction/modifier. The HTTP `POST /send` path awaits that apply so the JSON body is the final text; the WS send path schedules it with `asyncio.create_task` so the receive loop never blocks on the gateway. After an in-place rewrite, `_run_correction`/`_run_modification` call `push_content_revision` which stamps `messages.updated_at` and pushes a second `message` frame (`ai_processed=true`) to local sockets. Cross-worker peers pick the revision up from `message_relay._tick_updates` (`SELECT ... WHERE updated_at > watermark`, independent of the new-row `id` watermark and of `was_delivered`). The client matches the second frame on `data-msg-uid` and replaces the bubble body. The WS loop also accepts `{type:"sync", since, with_uid?}` to replay rows created or revised after `since` (reconnect catch-up) and `{type:"ping"}` (ignored). `ContentRefused` on a WS send returns `{type:"error"}` and keeps the socket open. ## WS protocol frames -Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`. Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; `ai_processed` is additive - `true` only when the broadcast content is the result of an awaited pending correction/modifier future, so an unmodified send is `false` with zero extra computation), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender). There is no `presence` frame on this socket - see below. +Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below. ## Presence is NOT part of the messaging WS @@ -65,8 +65,8 @@ Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubbl ## Frontend -The messaging frontend is the single self-booting custom element `` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `templates/messages.html` renders `` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, modeled on `PubSubClient.js`'s real exponential backoff instead of a flat-delay retry, same `4013` fast-path), sends over WS with an optimistic pending bubble keyed by `client_id` (the FIRST, unconditional branch of the incoming-frame handler matches `sender_uid === selfUid && client_id` against a pending bubble before any other branch - the fix for the old, permanently-disabled `appendOptimistic`), reconciles on the echoed `message` frame (falling to a `.failed`/tap-to-retry state after an 8s timeout with no echo), appends incoming messages live, auto-scrolls, throttles + shows the typing indicator, flips the read-receipt double-check, live-updates the conversation-list preview + unread dot + ordering, and groups consecutive same-sender messages within a 300s gap (`static/js/chat/MessageGrouping.js`, shared between the initial adopted history and the live-append path). If the socket is not open the send `
    ` submits normally (no-JS fallback). Presence in `mode="page"` needs no component code - the adopted markup keeps its `data-presence-uid` attributes, so the page-global `PresenceManager` already drives the dot/last-seen label; `` (no page-global `PresenceManager`) opens its own scoped `PubSubClient` subscription instead. An opt-in `ai-indicator="true"` attribute shows a brief "Adjusted by AI" caption when a reconciled echo's `ai_processed` flag is true and the content differs from what was locally typed - never a diff/revert UI. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px, with a `theme`/`accent` attribute override mechanism (scoped inline custom properties on the component's own root) for future off-site embedding. +The messaging frontend is the single self-booting custom element `` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. The send button is disabled only while `dp-upload` is busy, never while a send is in flight. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px. ## Attachments stream live -The single `message_frame` builder (`services/messaging/persist.py`, used by BOTH `broadcast_message` and the relay) fetches `get_attachments("message", uid)` and includes a slimmed list (`uid`/`url`/`thumbnail_url`/`is_image`/`is_video`/`is_audio`/`original_filename`/`file_size`/`mime_type`) on every frame via `_slim_attachment`. `is_audio` is derived identically to every other attachment surface in the app (`mime_type.startswith("audio/")`, already computed by `attachments._row_to_attachment` and simply forwarded here) - `AppChat._renderAttachments` mirrors `_attachment_display.html`'s type branches (image -> `img.gallery-thumb` marked `data-lightbox`+`data-full`, video -> `