retoor/devplacepy
1
0
Fork 1
Code Issues 44 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Add thread notifications, SEO topic pages, and fix quiz auto-advance

Browse Source 
Notifications: a new "thread" type notifies every other commenter on a
post whenever anyone comments on it, disregarding reply hierarchy -
excluding the actor and whoever already got a comment/reply
notification for that same event, so no one is double-notified.
Implemented via a background-deferred fan-out mirroring the existing
mention-notification pattern.

SEO: discussion_forum_posting() now embeds up to 20 of a post's
comments as nested schema.org Comment entities (not just an aggregate
count), and a new /topics hub plus /topics/{topic} pages give the
feed's topic filter real, independently crawlable/indexable URLs -
/feed?topic=X was never indexable since its canonical strips the
query string back to bare /feed. Both are wired end to end (schemas,
Devii actions, docs API, sitemap, locustfile load-test coverage).

Quiz player: the auto-advance to the next question used to hide the
just-answered slide in the same tick as rendering the grade, so on
any multi-question quiz the Correct/Not correct feedback was never
actually visible before the view moved on. Delayed via setTimeout,
with the pending timer cleared on manual navigation and on
disconnect so it can't race or fire on a removed component.

Also includes other local changes already in progress in this
working tree before this session (messaging, push delivery,
deepsearch jobs, game economy, quiz builder) - verified by the full
suite passing (3467 tests) but not authored or individually reviewed
in this session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VL9Xn57W5UR3HZbbuuzxdK
This commit is contained in:
retoor
2026-09-03 08:47:57 +02:00
co-authored by Claude Sonnet 5
parent afb4799869
commit 572e022584
93 changed files with 2788 additions and 331 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+25 -17
View File
@@ -7,7 +7,7 @@ Server-rendered social network for developers. FastAPI backend serving Jinja2 te
## Quick start
```bash
make install # pip install -e .
make install # create .venv if needed, pip install -e ".[dev]", playwright chromium
make dev # uvicorn --reload on port 10500
make test # Playwright integration + unit tests, headless, fail-fast
make test-headed # same tests in visible browser
@@ -82,7 +82,7 @@ devplacepy/
| `/admin/trash` | Admin **Trash**: review, restore, and permanently purge soft-deleted content (posts, comments, gists, projects, news, project files, attachments) across the platform |
| `/admin/devii-tasks` | Admin **Devii tasks**: every scheduled task across all owners with its schedule, run count, expiry and failure streak, plus per-task disable and delete |
| `/notifications` | Notification list, mark read, live unread counts (`/notifications/counts`) |
| `/messages` | Real-time direct messaging over WebSocket (`/messages/ws`): live bidirectional delivery, optimistic send, typing indicators, read receipts, and online/last-seen presence. Messages render through the shared content pipeline (emoji shortcodes, image, video and audio embeds, autolink, sanitization). AI content correction and the AI modifier apply to direct messages, so typing an inline `@ai <instruction>` in a message executes it and the resolved result appears live in the chat for both participants (the `message` WS frame carries an additive `ai_processed` flag when a pending correction/modification was applied). An opened conversation loads its 500 most recent messages; older history is retained in the database. `GET /messages/conversations` returns the conversation list as JSON for a live client refresh; the `POST /messages/send` form remains as a no-JavaScript fallback and now accepts an attachment-only, empty-content message. `POST /messages/ws-ticket` exchanges a session/API-key auth to a short-lived (30s), single-use ticket a browser WebSocket can carry as `?ticket=...` on the handshake, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header - this is what makes off-session embedding of the chat possible. CLI: `devplace messaging prune-tickets` removes expired tickets |
| `/messages` | Real-time direct messaging over WebSocket (`/messages/ws`): live bidirectional delivery, optimistic send, typing indicators, read receipts, reconnect catch-up, and online/last-seen presence. Messages render through the shared content pipeline (emoji shortcodes, image, video and audio embeds, autolink, sanitization). AI content correction and the AI modifier apply to direct messages: the original is delivered immediately, then a second `message` frame with `ai_processed` replaces the bubble for both participants when the rewrite lands (including across workers). An opened conversation loads its 500 most recent messages and can page older rows with `?before=`; switching threads does not drop the socket. `GET /messages/conversations` returns the conversation list as JSON for a live client refresh; the `POST /messages/send` form remains as a no-JavaScript fallback and now accepts an attachment-only, empty-content message. `POST /messages/ws-ticket` exchanges a session/API-key auth to a short-lived (30s), single-use ticket a browser WebSocket can carry as `?ticket=...` on the handshake, since a native `WebSocket` cannot set an `Authorization`/`X-API-KEY` header - this is what makes off-session embedding of the chat possible. CLI: `devplace messaging prune-tickets` removes expired tickets |
| `/votes` | Upvote/downvote on posts, comments, projects |
| `/reactions` | Emoji reactions on posts, comments, gists, projects |
| `/bookmarks` | Save/unsave content; `/bookmarks/saved` personal list |
@@ -452,7 +452,7 @@ and its full configuration are documented automatically - including future servi
- **`ConfigField`** - declarative parameter spec (type, default, validation, secret) a service uses to declare its editable settings
- **`BaseService`** - abstract class with a reconciling run loop that honors the persisted `enabled`/command/interval state, plus `config_fields`, `get_config()`, `describe()`, and a log buffer
- **`ServiceManager`** - singleton: `register`, `describe_all`, `set_enabled`, `send_command`, `save_config`, `supervise`, `shutdown_all`
- **`NewsService`** - a fully automatic, zero-maintenance news pipeline: fetches news from `news.app.molodetz.nl/api`, cleans each article, fetches and perceptually compares the images to reject placeholders and detect uniqueness, grades each one deterministically, reformats every valid article into clean Markdown (paragraphs, headings, lists) with the AI so the source wall of text reads as a proper article, and auto-rotates the best articles to Featured and the landing page. Its AI spend is metered from the gateway response headers and reported on the admin Services page (calls, tokens, total cost, and per-call averages)
- **`NewsService`** - a fully automatic, zero-maintenance news pipeline: fetches news from `news.app.molodetz.nl/api`, cleans each article, fetches and perceptually compares the images to reject placeholders and detect uniqueness, grades each one against a free, local, deterministic quality model (aquality - no billed LLM call, no API key), optionally reformats articles that clear the publish grade threshold into clean Markdown with a generative AI model (off by default, since the default grading model cannot generate text), and auto-rotates the best articles to Featured and the landing page. Any AI spend it does incur is metered from the gateway response headers and reported on the admin Services page (calls, tokens, total cost, and per-call averages)
- **`BotsService`** - Playwright fleet of AI personas that browse and interact with a DevPlace instance, with live cost/usage metrics and a live screenshot monitor at `/admin/bots` (opt-in; install the `bots` extra)
- **`GatewayService`** - OpenAI-compatible LLM gateway at `/openai/v1/*`, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default). Every chat request is made date-aware by injecting the current date in EU `DD/MM/YYYY` format into the system message when it contains no date already (date only, never time, to keep upstream prompt caching effective), and an admin-configurable system preamble (`gateway_system_preamble`) can be prepended ahead of the client's system message on every call. **Provider and model routing** (admin **Gateway** page, `/admin/gateway`) maps any number of requested model names onto named upstream providers and target models, each with its own pricing economy (input, output and cache-hit/cache-miss rates per million tokens) and an optional vision model that describes image content before forwarding, so text and vision models are merged transparently. Unmapped requests fall through to the default upstream unchanged, so existing clients are unaffected
- **`JobService` / `ZipService` / `ForkService`** - generic async job framework (`services/jobs/`) for heavy, blocking work run off the request path; `ZipService` builds project zip archives in a subprocess, `ForkService` copies a project into a new project owned by the forking user
@@ -524,7 +524,7 @@ restart.
`SeoMetaService` is a separate AI subservice that generates a clean, search-optimised title, description and short keyword list for every published post, project, gist, news article and issue, entirely off the request path so it never slows the web server. The work is queued whenever content is created, edited or published; until the AI value is ready a plain-content default (built from the markdown-stripped text and clamped to safe lengths) fills the fields, so a page's metadata is **always populated, never empty**. The service uses the built-in internal AI gateway and meters its own AI cost and statistics in a dedicated usage table, surfaced together with its live task pipeline on the **Admin -> Services** page. This release also fixes the on-page metadata: the `<meta name="description">` is now stripped of markdown markup (it previously leaked `#`, `**` and `[](...)` from the raw body), a `<meta name="keywords">` tag is emitted (a short honest list, not stuffed), and social-card image dimensions and alt text are added. CLI: `devplace seo-meta prune` / `devplace seo-meta clear` (job rows only; the generated metadata persists).
`DeepsearchService` powers the public **Tools -> DeepSearch** researcher. Given a single research question it plans a set of diverse web search queries, interleaves their results so every angle contributes sources, and crawls the most relevant pages in a subprocess (concurrent fetches, plain HTTP first with a headless-browser fallback for JavaScript-heavy pages, PDF documents streamed and text-extracted, every URL SSRF-guarded). Social sites that block bots (X, YouTube, Reddit and similar) contribute the readable text supplied by the search engine, so their content is not lost to a login wall. A readability-grade extractor isolates the main article content of each page (navigation, cookie banners and footers are discarded), and the configured depth follows the most relevant in-page links for deeper coverage. Content is de-duplicated and indexed into a per-session ChromaDB vector collection (embeddings via the AI gateway with a local fallback). A chain of agents (report writer, findings extractor, linker) then synthesises a thorough cited markdown report grounded on the passages retrieved from that index, with key findings, a confidence score and source diversity; if synthesis fails the report is clearly marked as degraded instead of silently shipping raw source material. Progress streams live over `WS /tools/deepsearch/{uid}/ws`; the report is at `/tools/deepsearch/{uid}/session` (HTML or JSON) and can be exported as Markdown, JSON or PDF. A grounded chat over the session's evidence runs at `WS /tools/deepsearch/{uid}/chat` using hybrid retrieval (vector + keyword/BM25). Runs can be paused, resumed or cancelled. A cross-session URL cache avoids re-fetching pages seen by earlier runs. CLI: `devplace deepsearch prune` / `devplace deepsearch clear`. ChromaDB, weasyprint, pypdf and Playwright are core dependencies.
`DeepsearchService` powers the public **Tools -> DeepSearch** researcher, an iterative agent modelled on the way Perplexity and OpenAI's own deep-research agents work: search, read, decide what is still missing, search again. Given a single research question it plans a set of diverse web search queries, interleaves their results so every angle contributes sources, and crawls the most relevant pages in a subprocess (concurrent fetches, plain HTTP first with a headless-browser fallback for JavaScript-heavy pages, PDF documents streamed and text-extracted, every URL SSRF-guarded). After the first pass it runs up to two automatic gap-filling refinement rounds - a lightweight planner looks at what has been gathered so far and proposes a few more targeted queries only if real coverage gaps remain, bounded by the page budget, and stops the moment nothing more is needed. Social sites that block bots (X, YouTube, Reddit and similar) contribute the readable text supplied by the search engine, so their content is not lost to a login wall. A readability-grade extractor isolates the main article content of each page (navigation, cookie banners and footers are discarded), and the configured depth follows the most relevant in-page links for deeper coverage. Content is de-duplicated and indexed into a per-session ChromaDB vector collection (embeddings via the AI gateway with a local fallback); retrieval reranks candidate passages with Maximal Marginal Relevance so the report is grounded on genuinely diverse evidence rather than several near-duplicate hits on the same page. A chain of agents (report writer, findings extractor, linker) then synthesises a thorough cited markdown report grounded on the passages retrieved from that index, with key findings, a confidence score, source diversity and a short list of suggested follow-up questions you can ask straight into the chat; if synthesis fails the report is clearly marked as degraded instead of silently shipping raw source material. Progress streams live over `WS /tools/deepsearch/{uid}/ws`; the report is at `/tools/deepsearch/{uid}/session` (HTML or JSON) and can be exported as Markdown, JSON or PDF. A grounded chat over the session's evidence runs at `WS /tools/deepsearch/{uid}/chat` using hybrid retrieval (vector + keyword/BM25). Runs can be paused, resumed or cancelled. A cross-session URL cache avoids re-fetching pages seen by earlier runs. CLI: `devplace deepsearch prune` / `devplace deepsearch clear`. ChromaDB, weasyprint, pypdf and Playwright are core dependencies.
`IsslopService` powers the public **Tools -> AI Usage Analyzer**, which classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. Sources are acquired in an isolated subprocess (git URLs are probed with `git ls-remote` and shallow-cloned with a 3 GB guard; websites render in a stealth headless browser with an HTTP fallback, bounded by depth, file and byte caps), inventoried with exclusion rules, and scored by a multi-signal static engine (twenty-one detector families, 126 signal types across an origin axis and a quality-deficit axis). For a live website, the analyzer also opens the home page in a headless browser and inspects what actually renders: it fingerprints AI website-builder platforms directly (Lovable, Bolt.new, Framer and others) and checks the page's real computed styles, layout and build artifacts, not just its file contents. Representative files receive an AI review pass and images an AI-generation review through the internal gateway (model `molodetz`, internal key); the static engine stays authoritative when the gateway is unreachable. Every pipeline step is persisted as an ordered event trail and streamed live over the pub/sub topic `public.isslop.{uid}`. The verdict is an A-F authenticity grade with a human/AI split and one of five categories (`ai-slop`, `sophisticated-ai`, `human-clean`, `human-messy`, `uncertain`), published as a persistent report with an embeddable SVG badge. Members keep their analysis history on their account; guest history is session-bound and claimed by the account on the first signed-in visit. Admin settings (private-host allowance, AI/image review toggles, image cap, retention, concurrency) live on `/admin/services`. CLI: `devplace isslop analyze <url>` / `devplace isslop prune` / `devplace isslop clear`. Playwright plus playwright-stealth back the website crawler.
@@ -557,9 +557,10 @@ Configuration on the Services tab (`/admin/services`):
|-----------|---------|---------|
| `news_grade_threshold` | `7` | Minimum AI grade for auto-publish |
| `news_api_url` | `https://news.app.molodetz.nl/api` | News source |
| `news_ai_url` | `http://localhost:10500/openai/v1/chat/completions` | AI grading endpoint (the internal gateway) |
| `news_ai_model` | `molodetz` | Generic model name; the gateway maps it to the real model |
| `news_ai_key` | internal key | AI API key (`NEWS_AI_KEY` env, then the auto-generated gateway internal key) |
| `news_ai_url` | `https://aquality.cloud.pravda.education/v1/chat/completions` | AI grading endpoint - a free, local, deterministic quality model by default |
| `news_ai_model` | `aquality` | Generic model name; echoed back, not used for routing |
| `news_ai_key` | internal key | AI API key (`NEWS_AI_KEY` env, then the auto-generated gateway internal key); unused by the default aquality endpoint |
| `news_format_enabled` | `false` | Reformat article bodies into Markdown with AI after grading; off by default since aquality only scores and cannot generate text - enable when `news_ai_url` points at a generative chat model |
| `news_service_interval` | `3600` | Seconds between fetch cycles (min 60) |
News articles have detail pages at `/news/{slug}` with full comment support (same component as posts/projects). The landing page can display curated articles toggled from admin.
@@ -919,13 +920,17 @@ receives a notification through every provider they hold a live subscription for
| Provider | Registration | Transport |
|----------|--------------|-----------|
| `webpush` | `PushSubscription` from the browser `PushManager` (endpoint + `p256dh`/`auth` keys) | Web Push Protocol, VAPID signed, `aesgcm` encrypted payload |
| `apns` | Hexadecimal device token | `POST https://api.push.apple.com/3/device/{token}` over HTTP/2, ES256 provider token |
| `apns` | Hexadecimal device token, optional stable `client_id` | Dedicated HTTP/2 client to `api.push.apple.com` or `api.sandbox.push.apple.com` (`POST /3/device/{token}`), ES256 provider token. Not the stealth/Chrome client. |
`POST /push.json` accepts a registration for any active provider; a body without a
`provider` field is a `webpush` body, so browsers need no change. `GET /push.json` returns
the VAPID public key plus the providers currently accepting registrations. A provider that
is disabled or not fully configured accepts no registrations and is skipped during
delivery, so an unconfigured provider is inert rather than an error.
`provider` field is a `webpush` body, so browsers need no change. An APNs body is
`{"provider": "apns", "token": "...", "client_id": "..."}` - `client_id` is optional and
identifies the device across Apple token rotations, so a new token updates that row
instead of inserting another. A token-only body still works and revives a previously
dead token. `GET /push.json` returns the VAPID public key plus the providers currently
accepting registrations; when `apns` is active it includes `environment` (`production` or
`sandbox`). A provider that is disabled or not fully configured accepts no registrations
and is skipped during delivery, so an unconfigured provider is inert rather than an error.
Every provider setting is edited at **`/admin/services/push`**: per provider an `Enabled`
toggle, the VAPID subject for `webpush`, and team id, key id, `.p8` auth key (stored as a
@@ -951,6 +956,7 @@ in real time, bridged onto the in-process pub/sub bus by a lock-owner relay:
| Direct message received | receiver |
| Comment on your post | post author |
| Reply to your comment | comment author |
| Any comment on a post you've also commented on | every other commenter on that post |
| `@mention` in any content | mentioned user |
| Upvote on your content | content owner |
| New follower | followed user |
@@ -960,10 +966,12 @@ in real time, bridged onto the in-process pub/sub bus by a lock-owner relay:
`create_notification` schedules delivery as a fire-and-forget async task, so a dead
subscription or push-service error never blocks the triggering request. Delivery
(`push.notify_user`) reads a user's subscriptions once, groups them by provider, builds
each provider's payload once, and sends over a single shared HTTP client. A subscription
the push service reports as gone (`404`/`410` for Web Push, `410` or an `Unregistered`
class reason for APNs) is soft-deleted; any other failure is logged and the subscription is
kept.
each provider's payload once, and sends over that provider's own HTTP client (stealth for
Web Push, a dedicated HTTP/2 client for APNs). A subscription the push service reports as
gone (`404`/`410` for Web Push, `410` or an `Unregistered` class reason for APNs) is
soft-deleted and the provider reason is logged; any other failure is logged and the
subscription is kept. APNs environment is stored per registration so a sandbox debug
token and a production token can coexist.
A notification is also **marked read automatically when you open the page that shows its
content** - viewing a post clears its comment, reply, upvote and mention notifications;
@@ -1022,7 +1030,7 @@ offline. Installation requires a secure origin (HTTPS, or `localhost` for develo
|------|------|
| `devplacepy/push/providers/` | Provider protocol, Web Push (VAPID keys, payload encryption), APNs |
| `devplacepy/push/store.py` | `push_registration` reads and writes |
| `devplacepy/push/delivery.py` | `notify_user` - group by provider, deliver, reap dead subscriptions |
| `devplacepy/push/delivery.py` | `notify_user` / `notify_registration` - group by provider, per-provider client, reap dead subscriptions |
| `devplacepy/services/push/service.py` | Provider configuration at `/admin/services/push`, retention sweep, metrics |
| `devplacepy/routers/push.py` | `/push.json`, `/service-worker.js`, `/manifest.json` |
| `static/js/PushManager.js` | Service-worker registration + subscribe + opt-in UI |