Files
devplacepy/devplacepy/services/messaging/CLAUDE.md
T
retoorandClaude Sonnet 5.5 f705099928 Inline YouTube Music embeds with one shared embed spec
YouTube Music has no player of its own and music.youtube.com sends
X-Frame-Options SAMEORIGIN, so music links render YouTube's own player
with an Open in YouTube Music caption. Tracks, playlists and OLAK5uy
albums embed; browse, channel, search and library stay plain links.

- embeds.py is the single URL spec (exact host allowlists, query-order
  independent, validated ids only); embed-spec.js is generated from it by
  the embeds-sync CLI command and drift is tested
- rendering.py and ContentRenderer.js both follow it; the old duplicate
  YouTube regexes are removed
- max 3 music embeds per content item, overflow stays a plain link
- fixes evilyoutube.com matching and watch?feature=share&v=ID links
- fixes the invalid unicode CSS escape that printed u2197 after every
  external link in rendered content
- docs, privacy third-party row, parser table tests, server/client parity

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 11:53:55 +01:00

26 KiB

Messaging (devplacepy/services/messaging/)

This file documents the real-time direct-message chat subsystem. Claude Code auto-loads it when a file under devplacepy/services/messaging/ is read or edited.

Overview

The /messages feature is a live WebSocket chat layered over the SAME messages table and audit/notification cores as before; there is no parallel chat store. The WS only adds live delivery, typing, and read receipts on top of the existing persistence - presence is NOT part of this WS (see "Presence is not part of the messaging WS" below).

messages.py (the router, mounted at /messages) exposes: GET "" (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest CONVERSATION_MESSAGE_LIMIT = 500 messages - keep these queries bounded), GET /search, GET /conversations (flat JSON conversation list, for a client-side refresh without a full page reload), POST /send (no-JS fallback, now also accepting an attachment-only empty-content message), POST /ws-ticket (issues a short-lived WS auth ticket), and WS /messages/ws (NOT lock-owner gated - accepts on every worker; closes 1008 for guests).

Bounded page queries (load-bearing performance rule)

get_conversations is ONE window-function query (ROW_NUMBER() OVER (PARTITION BY other_uid ORDER BY created_at DESC, id DESC) over sender_uid = :me OR receiver_uid = :me, the get_recent_comments_by_target_uids precedent) that returns only each partner's latest row - never load the user's full message history into Python to build the sidebar. It backs both the server-rendered GET /messages page and the JSON GET /messages/conversations route (same helper, no duplicated query). get_conversation_messages loads at most CONVERSATION_MESSAGE_LIMIT (500) most-recent rows (ORDER BY created_at DESC, id DESC LIMIT, reversed to ascending in Python); older history stays in the DB and is simply not rendered. Both are covered by idx_messages_conversation/_rev (verified EXPLAIN QUERY PLAN, MULTI-INDEX OR, no table scan). messages is NOT in SOFT_DELETE_TABLES, so neither query filters deleted_at. Any new message listing must stay bounded and indexed the same way.

GET /messages/conversations

Additive JSON endpoint (require_user), added so a frontend component can refresh the conversation list without a full page reload. It calls the exact same get_conversations(user_uid) helper GET /messages already uses, then projects each entry through schemas.ConversationOut (which nests other_user as UserOut, stripping email/api_key/password_hash) before returning {"conversations": [...]}. Like GET /messages/search, this is flat JSON, not content-negotiated through respond().

WS endpoint /messages/ws

Lives on the existing messages router (no new router/prefix; already mounted at /messages). It is NOT lock-owner gated (deliberately - unlike /devii/ws): await websocket.accept(), resolve the owner via _resolve_ws_user, and accept on EVERY worker. Gating on service_manager.owns_lock() was the original design and was wrong here: when no worker holds the service lock (services disabled, or the lock not yet acquired) every socket got closed 4013 and the client fast-retried every 200ms forever (a connection storm with no stream, and the send form fell back to a full-page POST reload). Chat must not depend on the background-service lock. Messaging requires an authenticated user - guests are closed with 1008. The receive loop is wrapped in try/except WebSocketDisconnect + a broad except (never crash the worker) and ALWAYS unregisters the socket in finally.

_resolve_ws_user auth resolution order

  1. Session cookie (_user_from_session) - the same-origin page mode path; browsers send cookies on a same-origin WS upgrade, so this needs no special handling.
  2. WS ticket (ticket query param, tried AFTER the session-cookie check and BEFORE the header checks - additive, inserted here specifically so it never changes same-origin page-mode behavior): if present, redeem_ticket(ticket) looks it up; on success it resolves to the ticket's owner user row. This is the ONLY way a real browser can authenticate a cross-context (embed mode / third-party) WebSocket, because the native WebSocket constructor cannot set custom request headers - there is no way to attach X-API-KEY/Authorization to new WebSocket(url) in any browser. See "WS auth tickets" below for the full mechanism. If the ticket is missing/expired/already-used, resolution simply falls through to the header checks below (never a hard reject at this point - only the final if not user in the WS handler closes the socket).
  3. X-API-KEY header, else Authorization: Bearer <key> - the existing non-browser API-client path (unreachable from a browser's native WebSocket, but real and unchanged for programmatic clients that can set headers, e.g. a server-side bot).

WS auth tickets (services/messaging/tickets.py, table ws_tickets)

Short-lived, single-use tokens that let a browser-based WebSocket authenticate without ever needing to set a request header on the handshake. POST /messages/ws-ticket (require_user, so it accepts the same HTTP-reachable auth as any other guarded route - session cookie, X-API-KEY, or Authorization: Bearer - over a normal fetch/XHR, where headers work fine) calls issue_ticket(user_uid) and returns {"ticket": "<opaque hex token>", "expires_in": 30}. The client then opens wss://.../messages/ws?ticket=<ticket> - a query parameter, which a WS handshake CAN carry.

  • issue_ticket(user_uid) -> str inserts a ws_tickets row (uid uuid7, token = secrets.token_hex(24), user_uid, created_at, expires_at = now + 30s, used_at=None) and returns the token.
  • ws_tickets also carries scope (empty or messages for chat, nodes for the node socket) and device_uid. redeem_ticket_row(token, scope) is the one conditional UPDATE (single use, unexpired, COALESCE(scope, 'messages') = :scope), so a node ticket can never open a chat socket and the reverse. issue_ticket(user_uid, scope=, device_uid=, ttl_seconds=).
  • redeem_ticket(token) -> Optional[str] looks the row up by token; returns None if missing, already used (used_at set), or past expires_at; otherwise stamps used_at and returns the user_uid. Redemption is single-use by construction - a ticket leaked via referrer or browser history cannot be replayed once redeemed, and it self-expires after 30 seconds even if never used.
  • ws_tickets is NOT in SOFT_DELETE_TABLES - it is an ephemeral, single-use artifact, garbage-collected hard via devplace messaging prune-tickets (deletes rows where expires_at < now), the same shape as the zip/fork/backup prune CLI commands. database/schema.py init_db() ensures its columns (uid, token, user_uid, created_at, expires_at, used_at) and a unique index on token (the hot lookup path) plus an index on expires_at (the prune sweep).
  • This is strictly additive to _resolve_ws_user - the existing session-cookie and header paths are completely unchanged, so mode="page" behavior and every existing non-browser API consumer are unaffected.

Connection registry

The ConnectionManager singleton message_hub (services/messaging/hub.py), one per worker: user_uid -> set[WebSocket], plus a bounded (4000-entry) delivered LRU dedupe set (mark_delivered/was_delivered) used to dedupe direct vs relayed delivery. register/unregister track connections for delivery only - no presence data lives here (see below).

Cross-worker delivery (the relay)

Because the WS accepts on every worker, a message persisted on worker A must still reach a recipient whose socket lives on worker B. services/messaging/relay.py message_relay (a per-worker singleton asyncio loop, started on the first socket connect, self-stops when message_hub.has_connections() is false) polls SELECT * FROM messages WHERE id > :watermark (uuid7-backed autoincrement id) every ~1s and pushes any row whose sender/receiver has a LOCAL socket and that was not already was_delivered to those local sockets, advancing the watermark to the max row seen. Same-worker sends are delivered INSTANTLY by broadcast_message (which calls message_hub.mark_delivered so the relay skips them); the relay only fills the cross-worker gap, so same-worker latency is zero and cross-worker latency is bounded by the poll interval. The relay primes its watermark to the current MAX(id) on first start so it never replays history. Trade-off: typing/read frames are in-process per worker only - across the two make prod workers those ephemeral signals reach only same-worker peers (message delivery is always correct). On a single dev worker everything is instant and complete.

DRY persist choke point

services/messaging/persist.py persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin, client_id=None) is the ONE function that inserts the row, links attachments, fires create_notification + clear_messages_cache + create_mention_notifications, logs, and writes the message.send audit event. Both the WS send handler and the HTTP POST /messages/send handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii send_message action is handler="http" -> POST /messages/send, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; content itself is allowed to be empty (MessageForm.content is min_length=0) as long as at least one attachment is present - persist_message is the single source of truth for that rule (if not content and not attachment_uids: return None), so the HTTP form model deliberately does not duplicate it. Whenever request is not None it audits via audit.record(request, ...) - this covers BOTH the HTTP Request and the WS path, since the WS handler passes request=websocket and a WebSocket object is just as non-None as a Request (audit.record never reads HTTP-specific attributes off it beyond what's already supplied explicitly via user=sender). audit.record_system(..., actor_kind="user", origin=origin) is the fallback used ONLY when persist_message is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same message.send key/category either way, no new event invented; typing/read are ephemeral and NOT audited.

Send dedupe (double-submit safety)

persist_message dedupes on (sender_uid, client_id) before it ever inserts a row: _find_recent_duplicate(sender_uid, client_id) looks up the most recent messages row with the same sender_uid and the same client-supplied client_id whose created_at falls inside DEDUPE_WINDOW_SECONDS (30s). A hit short-circuits persist_message and returns that ALREADY-persisted row verbatim (no error, no second insert, no second AI correction/notification/audit) - both the WS send handler and POST /messages/send now pass their client_id straight through, so a network-level retry that resends the identical payload (same client_id) collapses onto the original row instead of producing a second, independently-AI-corrected message. client_id is attacker-controlled but the dedupe key is always scoped by the server-resolved sender_uid, so a client can only dedupe its own sends. The column is nullable (messages.client_id, ensured in init_db(), indexed by idx_messages_dedupe (sender_uid, client_id)); a request with no client_id (any future non-dp-chat caller) skips the check entirely and always inserts. This is defense-in-depth alongside the client-side send-lock below - the lock stops a human double-click from ever firing two requests, the server dedupe stops a retried identical request (same client_id) from ever becoming two rows.

Client-side send-lock (AppChat.js). _initComposer's submit handler and the Enter-to-send path both funnel through one form submit listener, which now bails out while this._sendLocked is true. _sendViaSocket sets the lock (and disables the send button via the existing _refreshSendButton(), reusing the .is-sending spinner state) for every genuine new send (never for a .failed-bubble retry, tracked separately so a retry never blocks a fresh message) and only releases it once that specific client_id clears out of _pendingSends - on the echoed reconcile (_reconcilePending -> _clearPendingSend) or the 20s send timeout (_failPendingSend). Multiple sends never race the lock: _lockingSends is a Set of in-flight new-send client_ids, and the composer re-enables only when it is empty.

AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content

"messages": ("content",) is in the CORRECTABLE_FIELDS registry and persist_message invokes schedule_correction/schedule_modification (sender = the user), so typing @ai <instruction> in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through routers/messages._finalize_and_broadcast: it broadcasts the persisted row immediately (ai_pending=true when sync futures were stashed on request.scope[PENDING_SCOPE_KEY]), then applies any pending SYNC correction/modifier. The HTTP POST /send path awaits that apply so the JSON body is the final text; the WS send path schedules it with asyncio.create_task so the receive loop never blocks on the gateway. After an in-place rewrite, _run_correction/_run_modification call push_content_revision which stamps messages.updated_at and pushes a second message frame (ai_processed=true) to local sockets. Cross-worker peers pick the revision up from message_relay._tick_updates (SELECT ... WHERE updated_at > watermark, independent of the new-row id watermark and of was_delivered). The client matches the second frame on data-msg-uid and replaces the bubble body. The WS loop also accepts {type:"sync", since, with_uid?} to replay rows created or revised after since (reconnect catch-up) and {type:"ping"} (ignored). ContentRefused on a WS send returns {type:"error"} and keeps the socket open.

WS protocol frames

Client -> server: {type:"send", receiver_uid, content, attachment_uids?, client_id}, {type:"typing", receiver_uid} (throttled client-side), {type:"read", with_uid}, {type:"active", with_uid} (marks the sender as currently viewing that conversation - see "Active-conversation notification suppression" below), {type:"sync", since, with_uid?} (replay created-or-revised rows after since), {type:"ping"} (keepalive, ignored). Server -> client: {type:"ready", user_uid} (sent first on connect), {type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending} (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed client_id; a first frame may set ai_pending while a sync job runs, and a later frame for the same uid sets ai_processed with the rewritten body), {type:"typing", from_uid} (to the receiver only), {type:"read", by_uid} (read-receipt to the other user), {type:"error", client_id, text} (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no presence frame on this socket - see below. active has no server -> client echo; it only writes the DB-backed marker consumed server-side by persist_message.

Active-conversation notification suppression

A message notification (and its live toast) is suppressed when the receiver is demonstrably looking at that exact conversation right now, so an open DM thread never also produces a redundant bell/toast for the same message. The message itself still delivers live over the WS/relay path exactly as before - only the create_notification("message", ...) call in persist_message is skipped.

  • Marker: two columns on users, active_conversation_uid (the uid of the conversation partner currently being viewed) and active_conversation_at (UTC ISO timestamp of the last time it was refreshed), ensured in database.backfill_api_keys() like every other non-signup users column. This follows the services/presence.py last_seen pattern deliberately - a WS connection is per-worker (message_hub has no cross-worker visibility), but the sender and receiver of a DM can be on different uvicorn workers, so the "is the receiver looking at this" check has to be readable from ANY worker. A DB column is the only cross-worker-correct medium here, exactly as presence is.
  • Write path: services/messaging/active_conversation.py touch_active_conversation(viewer_uid, with_uid), called from the WS handler's new {type:"active", with_uid} frame. Throttled per-worker exactly like presence.touch (WRITE_THROTTLE_SECONDS = 5, an in-memory dict[f"{viewer_uid}:{with_uid}" -> monotonic] so a burst of focus/visibility events costs at most one write per 5s per conversation) before the UPDATE users SET active_conversation_uid=..., active_conversation_at=... write.
  • Client trigger (AppChat.js): _sendActiveMarker() sends the frame whenever _threadIsActive() is true (desktop: any open thread; mobile: only when the thread pane, not the list pane, is showing). It fires wherever the existing _markRead() already fires (initial WS connect, opening/switching a conversation) plus two new listeners bound in _bindActiveConversation(): window focus and document visibilitychange (visibilityState === "visible") - so returning to a backgrounded/blurred tab that still has the conversation open re-asserts the marker.
  • Read path / freshness window: is_actively_viewing(viewer_uid, other_uid) (same module) reads the receiver's user row, checks active_conversation_uid == other_uid, and requires active_conversation_at to be less than FRESH_SECONDS = 20 old. persist_message calls it as is_actively_viewing(receiver_uid, sender_uid) right before the create_notification call and skips only that call on a hit; clear_messages_cache(receiver_uid) still runs unconditionally, and the WS/relay delivery of the message frame is completely unaffected (it is broadcast from routers/messages.py, not from persist_message). The freshness window (20s) is intentionally wider than the write throttle (5s) so it never lapses between two consecutive throttled writes, and the marker is a soft "recently confirmed as active" signal, not a hard connection state - if it goes stale (tab closed, thread switched away without a fresh trigger) the very next message just resumes notifying normally, which is the safe failure direction.

Presence is NOT part of the messaging WS

Presence is handled entirely by the generic, cross-worker-correct mechanism documented in devplacepy/services/CLAUDE.md ("Online presence"): a single users.last_seen column, written by the track_presence HTTP middleware, read server-side at page load via presence.is_online(user), and kept live client-side by PresenceManager (static/js/PresenceManager.js) subscribing any [data-presence-uid] element to the single pub/sub topic public.presence.roster. The messages page's #messages-presence span carries data-presence-uid/data-presence-last-seen like every other avatar-presence site in the app - it is not messaging-specific code. An earlier design had WS-connect-based presence living in message_hub (is_online/last_seen, an _announce_presence helper, and a WS presence frame); that design was per-worker only and broke with more than one uvicorn worker, so it was removed outright. message_hub today tracks ONLY socket connections for message delivery - it has no presence state, and there is no presence frame anywhere in the current WS protocol (client or server). Never re-implement WS-connect presence on this socket - reuse presence.is_online, the public.presence.roster topic, and PresenceManager, exactly as every other presence surface in the app does. dp-chat also no longer owns a private presence renderer or PubSubClient - that duplicate disagreed with the feed roster and was removed.

Renderer integration (XSS control)

Live/echoed message bubbles are NEVER injected as raw HTML. AppChat._buildBubble (static/js/components/AppChat.js) creates a <dp-content> element and sets its textContent to the message body - never innerHTML - so the element's own client-side pipeline (emoji shortcode -> marked -> DOMPurify.sanitize -> highlight.js -> image/YouTube and YouTube Music/autolink) renders it safely, exactly like every other live-content surface that uses <dp-content>. This gives Discord-style emoji, image, YouTube and YouTube Music embeds (static/js/EmbedParser.js, spec in devplacepy/embeds.py), autolinking, and sanitization for free.

Frontend

The messaging frontend is the single self-booting custom element <dp-chat> (static/js/components/AppChat.js), which replaced the old page-controller trio MessagesLayout.js/MessagesSocket.js/MessageSearch.js (all deleted) and messages.css (superseded by static/css/chat.css) - see devplacepy/static/js/CLAUDE.md for the component roster entry. MobileNav.js does not own the mobile pane (that duplicate was removed; dp-chat is the only pane controller). templates/messages.html renders <dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true"> wrapping the SAME server-rendered .messages-list/.messages-main/.messages-thread/.message-bubble markup as before (no-JS/crawler fallback), which the component adopts on connectedCallback rather than discarding. It owns a ChatSocket (static/js/chat/ChatSocket.js, exponential backoff, 4013 fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by client_id, reconciles on the echoed message frame, applies later ai_processed frames in place (compare dp-content[data-source], never rendered textContent), injects the Report control on every live incoming bubble, catches up with {type:"sync"} on reconnect, and switches conversations without dropping the socket (GET /messages?with_uid= JSON + history.pushState). Older history loads via GET /messages?with_uid=&before= when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from with-uid / .show-list before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses visualViewport.offsetTop + height; Enter-to-send is desktop-only. The send button is disabled while dp-upload is busy AND while a genuine new send is in flight (the send-lock, see "Send dedupe" above) - a .failed-bubble retry never engages it, so retrying an old message never blocks typing/sending a new one. Presence in mode="page" stays on the page-global PresenceManager. An opt-in ai-indicator="true" attribute shows "Adjusting..." while ai_pending and "Adjusted by AI" when the revision lands. Styling is in static/css/chat.css using variables.css tokens only, responsive down to 360px.

Auto-scroll never fights a manual scroll-up. _startScrollWatcher/_stabilizeScroll force scrollTop = scrollHeight on every thread mutation (new bubble, attachment/image finishing loading) ONLY while _userAtBottom is true, re-checked every requestAnimationFrame up to STABILIZE_MAX_FRAMES. The one thing that used to break this: while a multi-frame stabilize loop was in flight (_stabilizePending), the real scroll event listener (_bindAutoScroll) ignored EVERY scroll event outright, including a genuine user wheel/touch scroll-up that happened to land inside that window - so a rapid burst of incoming messages (each restarting the stabilize loop while attachments were still loading) could silently overwrite a user's attempt to scroll up to read history. The fix: every programmatic scroll (_scrollThreadToEnd, _stabilizeScroll) records the exact scrollTop it just set in _lastForcedScrollTop; the scroll listener only skips its own echo (_stabilizePending && scrollTop === _lastForcedScrollTop) and always processes a scroll that lands anywhere else, so a real user scroll-up is detected and _userAtBottom flips to false immediately, even mid-stabilization. Never gate scroll-intent detection on _stabilizePending alone again.

Attachments stream live

The single message_frame builder (services/messaging/persist.py, used by BOTH broadcast_message and the relay) fetches get_attachments("message", uid) and includes a slimmed list (uid/url/thumbnail_url/is_image/is_video/is_audio/original_filename/file_size/mime_type) on every frame via _slim_attachment. is_audio is derived identically to every other attachment surface in the app (mime_type.startswith("audio/"), already computed by attachments._row_to_attachment and simply forwarded here) - AppChat._renderAttachments mirrors _attachment_display.html's type branches (image -> img.gallery-thumb marked data-lightbox+data-full, video -> <video>, audio -> <audio controls>, else a download link) - no innerHTML, so it stays XSS-safe, and a live-delivered audio attachment now renders identically to a page-refreshed one instead of falling back to a generic download link until reload. The sender's optimistic bubble shows an "Uploading attachment(s)..." placeholder until its own echo arrives and swaps in the real gallery. Attachment-only messages (empty caption) are allowed on both send paths: the WS send handler always allowed it (it reads content as a raw string with no Pydantic model), and the HTTP POST /messages/send form now does too (MessageForm.content is min_length=0) - persist_message accepts empty content when attachment_uids is non-empty (if not content and not attachment_uids: return None) and is the single place that rule lives. dp-upload (mode attachment, default show-chips off so only the (N) count badge shows) uploads each file to /uploads/upload and writes the uids into its hidden attachment_uids input; AppChat._collectAttachments() reads that before the send, then dp-upload.clear() resets it. The send button shows a spinner and is disabled only while dp-upload is busy (dp-upload:busy); in-flight sends stay in _pendingSends and flip the bubble to .failed/tap-to-retry if no echo arrives (20s), without locking the composer.