devplacepy/devplacepy/services/messaging/CLAUDE.md

Messaging (devplacepy/services/messaging/)

This file documents the real-time direct-message chat subsystem. Claude Code auto-loads it when a file under devplacepy/services/messaging/ is read or edited.

Overview

The /messages feature is a live WebSocket chat layered over the SAME messages table and audit/notification cores as before; there is no parallel chat store. The WS only adds live delivery, typing, and read receipts on top of the existing persistence - presence is NOT part of this WS (see "Presence is not part of the messaging WS" below).

messages.py (the router, mounted at /messages) exposes: GET "" (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest CONVERSATION_MESSAGE_LIMIT = 500 messages - keep these queries bounded), GET /search, GET /conversations (flat JSON conversation list, for a client-side refresh without a full page reload), POST /send (no-JS fallback, now also accepting an attachment-only empty-content message), POST /ws-ticket (issues a short-lived WS auth ticket), and WS /messages/ws (NOT lock-owner gated - accepts on every worker; closes 1008 for guests).

Bounded page queries (load-bearing performance rule)

get_conversations is ONE window-function query (ROW_NUMBER() OVER (PARTITION BY other_uid ORDER BY created_at DESC, id DESC) over sender_uid = :me OR receiver_uid = :me, the get_recent_comments_by_target_uids precedent) that returns only each partner's latest row - never load the user's full message history into Python to build the sidebar. It backs both the server-rendered GET /messages page and the JSON GET /messages/conversations route (same helper, no duplicated query). get_conversation_messages loads at most CONVERSATION_MESSAGE_LIMIT (500) most-recent rows (ORDER BY created_at DESC, id DESC LIMIT, reversed to ascending in Python); older history stays in the DB and is simply not rendered. Both are covered by idx_messages_conversation/_rev (verified EXPLAIN QUERY PLAN, MULTI-INDEX OR, no table scan). messages is NOT in SOFT_DELETE_TABLES, so neither query filters deleted_at. Any new message listing must stay bounded and indexed the same way.

GET /messages/conversations

Additive JSON endpoint (require_user), added so a frontend component can refresh the conversation list without a full page reload. It calls the exact same get_conversations(user_uid) helper GET /messages already uses, then projects each entry through schemas.ConversationOut (which nests other_user as UserOut, stripping email/api_key/password_hash) before returning {"conversations": [...]}. Like GET /messages/search, this is flat JSON, not content-negotiated through respond().

WS endpoint /messages/ws

Lives on the existing messages router (no new router/prefix; already mounted at /messages). It is NOT lock-owner gated (deliberately - unlike /devii/ws): await websocket.accept(), resolve the owner via _resolve_ws_user, and accept on EVERY worker. Gating on service_manager.owns_lock() was the original design and was wrong here: when no worker holds the service lock (services disabled, or the lock not yet acquired) every socket got closed 4013 and the client fast-retried every 200ms forever (a connection storm with no stream, and the send form fell back to a full-page POST reload). Chat must not depend on the background-service lock. Messaging requires an authenticated user - guests are closed with 1008. The receive loop is wrapped in try/except WebSocketDisconnect + a broad except (never crash the worker) and ALWAYS unregisters the socket in finally.

_resolve_ws_user auth resolution order

1. Session cookie (_user_from_session) - the same-origin page mode path; browsers send cookies on a same-origin WS upgrade, so this needs no special handling.
2. WS ticket (ticket query param, tried AFTER the session-cookie check and BEFORE the header checks - additive, inserted here specifically so it never changes same-origin page-mode behavior): if present, redeem_ticket(ticket) looks it up; on success it resolves to the ticket's owner user row. This is the ONLY way a real browser can authenticate a cross-context (embed mode / third-party) WebSocket, because the native WebSocket constructor cannot set custom request headers - there is no way to attach X-API-KEY/Authorization to new WebSocket(url) in any browser. See "WS auth tickets" below for the full mechanism. If the ticket is missing/expired/already-used, resolution simply falls through to the header checks below (never a hard reject at this point - only the final if not user in the WS handler closes the socket).
3. X-API-KEY header, else Authorization: Bearer <key> - the existing non-browser API-client path (unreachable from a browser's native WebSocket, but real and unchanged for programmatic clients that can set headers, e.g. a server-side bot).

WS auth tickets (services/messaging/tickets.py, table ws_tickets)

Short-lived, single-use tokens that let a browser-based WebSocket authenticate without ever needing to set a request header on the handshake. POST /messages/ws-ticket (require_user, so it accepts the same HTTP-reachable auth as any other guarded route - session cookie, X-API-KEY, or Authorization: Bearer - over a normal fetch/XHR, where headers work fine) calls issue_ticket(user_uid) and returns {"ticket": "<opaque hex token>", "expires_in": 30}. The client then opens wss://.../messages/ws?ticket=<ticket> - a query parameter, which a WS handshake CAN carry.

• issue_ticket(user_uid) -> str inserts a ws_tickets row (uid uuid7, token = secrets.token_hex(24), user_uid, created_at, expires_at = now + 30s, used_at=None) and returns the token.
• redeem_ticket(token) -> Optional[str] looks the row up by token; returns None if missing, already used (used_at set), or past expires_at; otherwise stamps used_at and returns the user_uid. Redemption is single-use by construction - a ticket leaked via referrer or browser history cannot be replayed once redeemed, and it self-expires after 30 seconds even if never used.
• ws_tickets is NOT in SOFT_DELETE_TABLES - it is an ephemeral, single-use artifact, garbage-collected hard via devplace messaging prune-tickets (deletes rows where expires_at < now), the same shape as the zip/fork/backup prune CLI commands. database/schema.py init_db() ensures its columns (uid, token, user_uid, created_at, expires_at, used_at) and a unique index on token (the hot lookup path) plus an index on expires_at (the prune sweep).
• This is strictly additive to _resolve_ws_user - the existing session-cookie and header paths are completely unchanged, so mode="page" behavior and every existing non-browser API consumer are unaffected.

Connection registry

The ConnectionManager singleton message_hub (services/messaging/hub.py), one per worker: user_uid -> set[WebSocket], plus a bounded (4000-entry) delivered LRU dedupe set (mark_delivered/was_delivered) used to dedupe direct vs relayed delivery. register/unregister track connections for delivery only - no presence data lives here (see below).

Cross-worker delivery (the relay)

Because the WS accepts on every worker, a message persisted on worker A must still reach a recipient whose socket lives on worker B. services/messaging/relay.py message_relay (a per-worker singleton asyncio loop, started on the first socket connect, self-stops when message_hub.has_connections() is false) polls SELECT * FROM messages WHERE id > :watermark (uuid7-backed autoincrement id) every ~1s and pushes any row whose sender/receiver has a LOCAL socket and that was not already was_delivered to those local sockets, advancing the watermark to the max row seen. Same-worker sends are delivered INSTANTLY by broadcast_message (which calls message_hub.mark_delivered so the relay skips them); the relay only fills the cross-worker gap, so same-worker latency is zero and cross-worker latency is bounded by the poll interval. The relay primes its watermark to the current MAX(id) on first start so it never replays history. Trade-off: typing/read frames are in-process per worker only - across the two make prod workers those ephemeral signals reach only same-worker peers (message delivery is always correct). On a single dev worker everything is instant and complete.

DRY persist choke point

services/messaging/persist.py persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin) is the ONE function that inserts the row, links attachments, fires create_notification + clear_messages_cache + create_mention_notifications, logs, and writes the message.send audit event. Both the WS send handler and the HTTP POST /messages/send handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii send_message action is handler="http" -> POST /messages/send, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; content itself is allowed to be empty (MessageForm.content is min_length=0) as long as at least one attachment is present - persist_message is the single source of truth for that rule (if not content and not attachment_uids: return None), so the HTTP form model deliberately does not duplicate it. Whenever request is not None it audits via audit.record(request, ...) - this covers BOTH the HTTP Request and the WS path, since the WS handler passes request=websocket and a WebSocket object is just as non-None as a Request (audit.record never reads HTTP-specific attributes off it beyond what's already supplied explicitly via user=sender). audit.record_system(..., actor_kind="user", origin=origin) is the fallback used ONLY when persist_message is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same message.send key/category either way, no new event invented; typing/read are ephemeral and NOT audited.

AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content

"messages": ("content",) is in the CORRECTABLE_FIELDS registry and persist_message invokes schedule_correction/schedule_modification (sender = the user), so typing @ai <instruction> in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through routers/messages._finalize_and_broadcast: it broadcasts the persisted row immediately (ai_pending=true when sync futures were stashed on request.scope[PENDING_SCOPE_KEY]), then applies any pending SYNC correction/modifier. The HTTP POST /send path awaits that apply so the JSON body is the final text; the WS send path schedules it with asyncio.create_task so the receive loop never blocks on the gateway. After an in-place rewrite, _run_correction/_run_modification call push_content_revision which stamps messages.updated_at and pushes a second message frame (ai_processed=true) to local sockets. Cross-worker peers pick the revision up from message_relay._tick_updates (SELECT ... WHERE updated_at > watermark, independent of the new-row id watermark and of was_delivered). The client matches the second frame on data-msg-uid and replaces the bubble body. The WS loop also accepts {type:"sync", since, with_uid?} to replay rows created or revised after since (reconnect catch-up) and {type:"ping"} (ignored). ContentRefused on a WS send returns {type:"error"} and keeps the socket open.

WS protocol frames

Client -> server: {type:"send", receiver_uid, content, attachment_uids?, client_id}, {type:"typing", receiver_uid} (throttled client-side), {type:"read", with_uid}, {type:"sync", since, with_uid?} (replay created-or-revised rows after since), {type:"ping"} (keepalive, ignored). Server -> client: {type:"ready", user_uid} (sent first on connect), {type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending} (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed client_id; a first frame may set ai_pending while a sync job runs, and a later frame for the same uid sets ai_processed with the rewritten body), {type:"typing", from_uid} (to the receiver only), {type:"read", by_uid} (read-receipt to the other user), {type:"error", client_id, text} (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no presence frame on this socket - see below.

Presence is NOT part of the messaging WS

Presence is handled entirely by the generic, cross-worker-correct mechanism documented in devplacepy/services/CLAUDE.md ("Online presence"): a single users.last_seen column, written by the track_presence HTTP middleware, read server-side at page load via presence.is_online(user), and kept live client-side by PresenceManager (static/js/PresenceManager.js) subscribing any [data-presence-uid] element to the single pub/sub topic public.presence.roster. The messages page's #messages-presence span carries data-presence-uid/data-presence-last-seen like every other avatar-presence site in the app - it is not messaging-specific code. An earlier design had WS-connect-based presence living in message_hub (is_online/last_seen, an _announce_presence helper, and a WS presence frame); that design was per-worker only and broke with more than one uvicorn worker, so it was removed outright. message_hub today tracks ONLY socket connections for message delivery - it has no presence state, and there is no presence frame anywhere in the current WS protocol (client or server). Never re-implement WS-connect presence on this socket - reuse presence.is_online, the public.presence.roster topic, and PresenceManager, exactly as every other presence surface in the app does. dp-chat also no longer owns a private presence renderer or PubSubClient - that duplicate disagreed with the feed roster and was removed.

Renderer integration (XSS control)

Live/echoed message bubbles are NEVER injected as raw HTML. AppChat._buildBubble (static/js/components/AppChat.js) creates a <dp-content> element and sets its textContent to the message body - never innerHTML - so the element's own client-side pipeline (emoji shortcode -> marked -> DOMPurify.sanitize -> highlight.js -> image/YouTube/autolink) renders it safely, exactly like every other live-content surface that uses <dp-content>. This gives Discord-style emoji, image and YouTube embeds, autolinking, and sanitization for free.

Frontend

The messaging frontend is the single self-booting custom element <dp-chat> (static/js/components/AppChat.js), which replaced the old page-controller trio MessagesLayout.js/MessagesSocket.js/MessageSearch.js (all deleted) and messages.css (superseded by static/css/chat.css) - see devplacepy/static/js/CLAUDE.md for the component roster entry. MobileNav.js does not own the mobile pane (that duplicate was removed; dp-chat is the only pane controller). templates/messages.html renders <dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true"> wrapping the SAME server-rendered .messages-list/.messages-main/.messages-thread/.message-bubble markup as before (no-JS/crawler fallback), which the component adopts on connectedCallback rather than discarding. It owns a ChatSocket (static/js/chat/ChatSocket.js, exponential backoff, 4013 fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by client_id, reconciles on the echoed message frame, applies later ai_processed frames in place (compare dp-content[data-source], never rendered textContent), injects the Report control on every live incoming bubble, catches up with {type:"sync"} on reconnect, and switches conversations without dropping the socket (GET /messages?with_uid= JSON + history.pushState). Older history loads via GET /messages?with_uid=&before= when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from with-uid / .show-list before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses visualViewport.offsetTop + height; Enter-to-send is desktop-only. The send button is disabled only while dp-upload is busy, never while a send is in flight. Presence in mode="page" stays on the page-global PresenceManager. An opt-in ai-indicator="true" attribute shows "Adjusting..." while ai_pending and "Adjusted by AI" when the revision lands. Styling is in static/css/chat.css using variables.css tokens only, responsive down to 360px.

Attachments stream live

The single message_frame builder (services/messaging/persist.py, used by BOTH broadcast_message and the relay) fetches get_attachments("message", uid) and includes a slimmed list (uid/url/thumbnail_url/is_image/is_video/is_audio/original_filename/file_size/mime_type) on every frame via _slim_attachment. is_audio is derived identically to every other attachment surface in the app (mime_type.startswith("audio/"), already computed by attachments._row_to_attachment and simply forwarded here) - AppChat._renderAttachments mirrors _attachment_display.html's type branches (image -> img.gallery-thumb marked data-lightbox+data-full, video -> <video>, audio -> <audio controls>, else a download link) - no innerHTML, so it stays XSS-safe, and a live-delivered audio attachment now renders identically to a page-refreshed one instead of falling back to a generic download link until reload. The sender's optimistic bubble shows an "Uploading attachment(s)..." placeholder until its own echo arrives and swaps in the real gallery. Attachment-only messages (empty caption) are allowed on both send paths: the WS send handler always allowed it (it reads content as a raw string with no Pydantic model), and the HTTP POST /messages/send form now does too (MessageForm.content is min_length=0) - persist_message accepts empty content when attachment_uids is non-empty (if not content and not attachment_uids: return None) and is the single place that rule lives. dp-upload (mode attachment, default show-chips off so only the (N) count badge shows) uploads each file to /uploads/upload and writes the uids into its hidden attachment_uids input; AppChat._collectAttachments() reads that before the send, then dp-upload.clear() resets it. The send button shows a spinner and is disabled only while dp-upload is busy (dp-upload:busy); in-flight sends stay in _pendingSends and flip the bubble to .failed/tap-to-retry if no echo arrives (20s), without locking the composer.