Messaging (`devplacepy/services/messaging/`)

This file documents the real-time direct-message chat subsystem. Claude Code auto-loads it when a file under devplacepy/services/messaging/ is read or edited.

Overview

The /messages feature is a live WebSocket chat layered over the SAME messages table and audit/notification cores as before; there is no parallel chat store. The WS only adds live delivery, typing, and read receipts on top of the existing persistence - presence is NOT part of this WS (see "Presence is not part of the messaging WS" below).

messages.py (the router, mounted at /messages) exposes: GET "" (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest CONVERSATION_MESSAGE_LIMIT = 500 messages - keep these queries bounded), GET /search, GET /conversations (flat JSON conversation list, for a client-side refresh without a full page reload), POST /send (no-JS fallback, now also accepting an attachment-only empty-content message), POST /ws-ticket (issues a short-lived WS auth ticket), and WS /messages/ws (NOT lock-owner gated - accepts on every worker; closes 1008 for guests).

Bounded page queries (load-bearing performance rule)

get_conversations is ONE window-function query (ROW_NUMBER() OVER (PARTITION BY other_uid ORDER BY created_at DESC, id DESC) over sender_uid = :me OR receiver_uid = :me, the get_recent_comments_by_target_uids precedent) that returns only each partner's latest row - never load the user's full message history into Python to build the sidebar. It backs both the server-rendered GET /messages page and the JSON GET /messages/conversations route (same helper, no duplicated query). get_conversation_messages loads at most CONVERSATION_MESSAGE_LIMIT (500) most-recent rows (ORDER BY created_at DESC, id DESC LIMIT, reversed to ascending in Python); older history stays in the DB and is simply not rendered. Both are covered by idx_messages_conversation/_rev (verified EXPLAIN QUERY PLAN, MULTI-INDEX OR, no table scan). messages is NOT in SOFT_DELETE_TABLES, so neither query filters deleted_at. Any new message listing must stay bounded and indexed the same way.

`GET /messages/conversations`

Additive JSON endpoint (require_user), added so a frontend component can refresh the conversation list without a full page reload. It calls the exact same get_conversations(user_uid) helper GET /messages already uses, then projects each entry through schemas.ConversationOut (which nests other_user as UserOut, stripping email/api_key/password_hash) before returning {"conversations": [...]}. Like GET /messages/search, this is flat JSON, not content-negotiated through respond().

WS endpoint `/messages/ws`

Lives on the existing messages router (no new router/prefix; already mounted at /messages). It is NOT lock-owner gated (deliberately - unlike /devii/ws): await websocket.accept(), resolve the owner via _resolve_ws_user, and accept on EVERY worker. Gating on service_manager.owns_lock() was the original design and was wrong here: when no worker holds the service lock (services disabled, or the lock not yet acquired) every socket got closed 4013 and the client fast-retried every 200ms forever (a connection storm with no stream, and the send form fell back to a full-page POST reload). Chat must not depend on the background-service lock. Messaging requires an authenticated user - guests are closed with 1008. The receive loop is wrapped in try/except WebSocketDisconnect + a broad except (never crash the worker) and ALWAYS unregisters the socket in finally.

`_resolve_ws_user` auth resolution order

Session cookie (_user_from_session) - the same-origin page mode path; browsers send cookies on a same-origin WS upgrade, so this needs no special handling.
WS ticket (ticket query param, tried AFTER the session-cookie check and BEFORE the header checks - additive, inserted here specifically so it never changes same-origin page-mode behavior): if present, redeem_ticket(ticket) looks it up; on success it resolves to the ticket's owner user row. This is the ONLY way a real browser can authenticate a cross-context (embed mode / third-party) WebSocket, because the native WebSocket constructor cannot set custom request headers - there is no way to attach X-API-KEY/Authorization to new WebSocket(url) in any browser. See "WS auth tickets" below for the full mechanism. If the ticket is missing/expired/already-used, resolution simply falls through to the header checks below (never a hard reject at this point - only the final if not user in the WS handler closes the socket).
X-API-KEY header, else Authorization: Bearer <key> - the existing non-browser API-client path (unreachable from a browser's native WebSocket, but real and unchanged for programmatic clients that can set headers, e.g. a server-side bot).

WS auth tickets (`services/messaging/tickets.py`, table `ws_tickets`)

Short-lived, single-use tokens that let a browser-based WebSocket authenticate without ever needing to set a request header on the handshake. POST /messages/ws-ticket (require_user, so it accepts the same HTTP-reachable auth as any other guarded route - session cookie, X-API-KEY, or Authorization: Bearer - over a normal fetch/XHR, where headers work fine) calls issue_ticket(user_uid) and returns {"ticket": "<opaque hex token>", "expires_in": 30}. The client then opens wss://.../messages/ws?ticket=<ticket> - a query parameter, which a WS handshake CAN carry.

issue_ticket(user_uid) -> str inserts a ws_tickets row (uid uuid7, token = secrets.token_hex(24), user_uid, created_at, expires_at = now + 30s, used_at=None) and returns the token.
redeem_ticket(token) -> Optional[str] looks the row up by token; returns None if missing, already used (used_at set), or past expires_at; otherwise stamps used_at and returns the user_uid. Redemption is single-use by construction - a ticket leaked via referrer or browser history cannot be replayed once redeemed, and it self-expires after 30 seconds even if never used.
ws_tickets is NOT in SOFT_DELETE_TABLES - it is an ephemeral, single-use artifact, garbage-collected hard via devplace messaging prune-tickets (deletes rows where expires_at < now), the same shape as the zip/fork/backup prune CLI commands. database/schema.py init_db() ensures its columns (uid, token, user_uid, created_at, expires_at, used_at) and a unique index on token (the hot lookup path) plus an index on expires_at (the prune sweep).
This is strictly additive to _resolve_ws_user - the existing session-cookie and header paths are completely unchanged, so mode="page" behavior and every existing non-browser API consumer are unaffected.

Connection registry

The ConnectionManager singleton message_hub (services/messaging/hub.py), one per worker: user_uid -> set[WebSocket], plus a bounded (4000-entry) delivered LRU dedupe set (mark_delivered/was_delivered) used to dedupe direct vs relayed delivery. register/unregister track connections for delivery only - no presence data lives here (see below).

Cross-worker delivery (the relay)

Because the WS accepts on every worker, a message persisted on worker A must still reach a recipient whose socket lives on worker B. services/messaging/relay.py message_relay (a per-worker singleton asyncio loop, started on the first socket connect, self-stops when message_hub.has_connections() is false) polls SELECT * FROM messages WHERE id > :watermark (uuid7-backed autoincrement id) every ~1s and pushes any row whose sender/receiver has a LOCAL socket and that was not already was_delivered to those local sockets, advancing the watermark to the max row seen. Same-worker sends are delivered INSTANTLY by broadcast_message (which calls message_hub.mark_delivered so the relay skips them); the relay only fills the cross-worker gap, so same-worker latency is zero and cross-worker latency is bounded by the poll interval. The relay primes its watermark to the current MAX(id) on first start so it never replays history. Trade-off: typing/read frames are in-process per worker only - across the two make prod workers those ephemeral signals reach only same-worker peers (message delivery is always correct). On a single dev worker everything is instant and complete.

DRY persist choke point

services/messaging/persist.py persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin) is the ONE function that inserts the row, links attachments, fires create_notification + clear_messages_cache + create_mention_notifications, logs, and writes the message.send audit event. Both the WS send handler and the HTTP POST /messages/send handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii send_message action is handler="http" -> POST /messages/send, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; content itself is allowed to be empty (MessageForm.content is min_length=0) as long as at least one attachment is present - persist_message is the single source of truth for that rule (if not content and not attachment_uids: return None), so the HTTP form model deliberately does not duplicate it. Whenever request is not None it audits via audit.record(request, ...) - this covers BOTH the HTTP Request and the WS path, since the WS handler passes request=websocket and a WebSocket object is just as non-None as a Request (audit.record never reads HTTP-specific attributes off it beyond what's already supplied explicitly via user=sender). audit.record_system(..., actor_kind="user", origin=origin) is the fallback used ONLY when persist_message is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same message.send key/category either way, no new event invented; typing/read are ephemeral and NOT audited.

AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content

"messages": ("content",) is in the CORRECTABLE_FIELDS registry and persist_message invokes schedule_correction/schedule_modification (sender = the user), so typing @ai <instruction> in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through routers/messages._finalize_and_broadcast(sender, message, request, client_id=None): it reads any pending SYNC correction/modification futures stashed on request.scope[PENDING_SCOPE_KEY] (the same PENDING_SCOPE_KEY/loop.run_in_executor mechanism the HTTP await_pending_corrections middleware uses) into ai_processed = bool(pending) before awaiting/clearing them, awaits them, RE-READS the message row, and broadcasts the FINAL (corrected/modified) content via broadcast_message(sender, message, client_id, ai_processed=ai_processed). The HTTP POST /send and the WS /ws send path both call it; the WS path passes request=websocket to persist_message so the modifier stashes its executor future on the websocket scope and the handler awaits it directly (HTTP middleware does not run for websockets). A message with no @ai directive and correction off has nothing pending, so ai_processed is False and the broadcast is immediate with zero added latency. broadcast_message forwards ai_processed straight into message_frame(..., ai_processed=ai_processed) - see "WS protocol frames" below for the frame shape. This flag is informational only: the server never retains the pre-AI text anywhere (table.update overwrites in place), so it exists purely to drive a client-side "Adjusted by AI" affordance, not any kind of diff/revert capability.

WS protocol frames

Client -> server: {type:"send", receiver_uid, content, attachment_uids?, client_id}, {type:"typing", receiver_uid} (throttled client-side), {type:"read", with_uid}. Server -> client: {type:"ready", user_uid} (sent first on connect), {type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed} (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed client_id; ai_processed is additive - true only when the broadcast content is the result of an awaited pending correction/modifier future, so an unmodified send is false with zero extra computation), {type:"typing", from_uid} (to the receiver only), {type:"read", by_uid} (read-receipt to the other user), {type:"error", client_id, text} (sender only, on a dropped send, e.g. the recipient blocked the sender). There is no presence frame on this socket - see below.

Presence is NOT part of the messaging WS

Presence is handled entirely by the generic, cross-worker-correct mechanism documented in devplacepy/services/CLAUDE.md ("Online presence"): a single users.last_seen column, written by the track_presence HTTP middleware, read server-side at page load via presence.is_online(user), and kept live client-side by PresenceManager (static/js/PresenceManager.js) subscribing any [data-presence-uid] element to the pub/sub topic public.presence.{uid}. The messages page's #messages-presence span carries data-presence-uid/data-presence-last-seen like every other avatar-presence site in the app - it is not messaging-specific code. An earlier design had WS-connect-based presence living in message_hub (is_online/last_seen, an _announce_presence helper, and a WS presence frame); that design was per-worker only and broke with more than one uvicorn worker, so it was removed outright. message_hub today tracks ONLY socket connections for message delivery - it has no presence state, and there is no presence frame anywhere in the current WS protocol (client or server). Never re-implement WS-connect presence on this socket - reuse presence.is_online, the public.presence.{uid} topic, and PresenceManager, exactly as every other presence surface in the app does.

Renderer integration (XSS control)

Live/echoed message bubbles are NEVER injected as raw HTML. AppChat._buildBubble (static/js/components/AppChat.js) creates a <dp-content> element and sets its textContent to the message body - never innerHTML - so the element's own client-side pipeline (emoji shortcode -> marked -> DOMPurify.sanitize -> highlight.js -> image/YouTube/autolink) renders it safely, exactly like every other live-content surface that uses <dp-content>. This gives Discord-style emoji, image and YouTube embeds, autolinking, and sanitization for free.

Frontend

The messaging frontend is the single self-booting custom element <dp-chat> (static/js/components/AppChat.js), which replaced the old page-controller trio MessagesLayout.js/MessagesSocket.js/MessageSearch.js (all deleted) and messages.css (superseded by static/css/chat.css) - see devplacepy/static/js/CLAUDE.md for the component roster entry. templates/messages.html renders <dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true"> wrapping the SAME server-rendered .messages-list/.messages-main/.messages-thread/.message-bubble markup as before (no-JS/crawler fallback), which the component adopts on connectedCallback rather than discarding. It owns a ChatSocket (static/js/chat/ChatSocket.js, modeled on PubSubClient.js's real exponential backoff instead of a flat-delay retry, same 4013 fast-path), sends over WS with an optimistic pending bubble keyed by client_id (the FIRST, unconditional branch of the incoming-frame handler matches sender_uid === selfUid && client_id against a pending bubble before any other branch - the fix for the old, permanently-disabled appendOptimistic), reconciles on the echoed message frame (falling to a .failed/tap-to-retry state after an 8s timeout with no echo), appends incoming messages live, auto-scrolls, throttles + shows the typing indicator, flips the read-receipt double-check, live-updates the conversation-list preview + unread dot + ordering, and groups consecutive same-sender messages within a 300s gap (static/js/chat/MessageGrouping.js, shared between the initial adopted history and the live-append path). If the socket is not open the send <form method=POST> submits normally (no-JS fallback). Presence in mode="page" needs no component code - the adopted markup keeps its data-presence-uid attributes, so the page-global PresenceManager already drives the dot/last-seen label; <dp-chat mode="embed"> (no page-global PresenceManager) opens its own scoped PubSubClient subscription instead. An opt-in ai-indicator="true" attribute shows a brief "Adjusted by AI" caption when a reconciled echo's ai_processed flag is true and the content differs from what was locally typed - never a diff/revert UI. Styling is in static/css/chat.css using variables.css tokens only, responsive down to 360px, with a theme/accent attribute override mechanism (scoped inline custom properties on the component's own root) for future off-site embedding.

Attachments stream live

The single message_frame builder (services/messaging/persist.py, used by BOTH broadcast_message and the relay) fetches get_attachments("message", uid) and includes a slimmed list (uid/url/thumbnail_url/is_image/is_video/is_audio/original_filename/file_size/mime_type) on every frame via _slim_attachment. is_audio is derived identically to every other attachment surface in the app (mime_type.startswith("audio/"), already computed by attachments._row_to_attachment and simply forwarded here) - AppChat._renderAttachments mirrors _attachment_display.html's type branches (image -> img.gallery-thumb marked data-lightbox+data-full, video -> <video>, audio -> <audio controls>, else a download link) - no innerHTML, so it stays XSS-safe, and a live-delivered audio attachment now renders identically to a page-refreshed one instead of falling back to a generic download link until reload. The sender's optimistic bubble shows an "Uploading attachment(s)..." placeholder until its own echo arrives and swaps in the real gallery. Attachment-only messages (empty caption) are allowed on both send paths: the WS send handler always allowed it (it reads content as a raw string with no Pydantic model), and the HTTP POST /messages/send form now does too (MessageForm.content is min_length=0) - persist_message accepts empty content when attachment_uids is non-empty (if not content and not attachment_uids: return None) and is the single place that rule lives. dp-upload (mode attachment, default show-chips off so only the (N) count badge shows) uploads each file to /uploads/upload and writes the uids into its hidden attachment_uids input; AppChat._collectAttachments() reads that before the send, then dp-upload.clear() resets it. The send button shows a spinner and is disabled while busy: AppChat._refreshSendButton() ORs _uploading (driven by the dp-upload:busy event) with _pendingSends (a map of in-flight send client_ids, cleared on the echo or an 8s safety timeout that flips the bubble to .failed/tap-to-retry), toggling .is-sending + disabled on .messages-send-btn.

Messaging (devplacepy/services/messaging/)