This file documents the real-time direct-message chat subsystem. Claude Code auto-loads it when a file under `devplacepy/services/messaging/` is read or edited.
The `/messages` feature is a live WebSocket chat layered over the SAME `messages` table and audit/notification cores as before; there is no parallel chat store. The WS only adds live delivery, typing, and read receipts on top of the existing persistence - presence is NOT part of this WS (see "Presence is not part of the messaging WS" below).
`messages.py` (the router, mounted at `/messages`) exposes: `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `GET /conversations` (flat JSON conversation list, for a client-side refresh without a full page reload), `POST /send` (no-JS fallback, now also accepting an attachment-only empty-content message), `POST /ws-ticket` (issues a short-lived WS auth ticket), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests).
`get_conversations` is ONE window-function query (`ROW_NUMBER() OVER (PARTITION BY other_uid ORDER BY created_at DESC, id DESC)` over `sender_uid = :me OR receiver_uid = :me`, the `get_recent_comments_by_target_uids` precedent) that returns only each partner's latest row - never load the user's full message history into Python to build the sidebar. It backs both the server-rendered `GET /messages` page and the JSON `GET /messages/conversations` route (same helper, no duplicated query). `get_conversation_messages` loads at most `CONVERSATION_MESSAGE_LIMIT` (500) most-recent rows (`ORDER BY created_at DESC, id DESC LIMIT`, reversed to ascending in Python); older history stays in the DB and is simply not rendered. Both are covered by `idx_messages_conversation`/`_rev` (verified `EXPLAIN QUERY PLAN`, MULTI-INDEX OR, no table scan). `messages` is NOT in `SOFT_DELETE_TABLES`, so neither query filters `deleted_at`. Any new message listing must stay bounded and indexed the same way.
## `GET /messages/conversations`
Additive JSON endpoint (`require_user`), added so a frontend component can refresh the conversation list without a full page reload. It calls the exact same `get_conversations(user_uid)` helper `GET /messages` already uses, then projects each entry through `schemas.ConversationOut` (which nests `other_user` as `UserOut`, stripping `email`/`api_key`/`password_hash`) before returning `{"conversations": [...]}`. Like `GET /messages/search`, this is flat JSON, not content-negotiated through `respond()`.
Lives on the existing messages router (no new router/prefix; already mounted at `/messages`). It is **NOT lock-owner gated** (deliberately - unlike `/devii/ws`): `await websocket.accept()`, resolve the owner via `_resolve_ws_user`, and accept on EVERY worker. Gating on `service_manager.owns_lock()` was the original design and was wrong here: when no worker holds the service lock (services disabled, or the lock not yet acquired) every socket got closed `4013` and the client fast-retried every 200ms forever (a connection storm with no stream, and the send form fell back to a full-page POST reload). Chat must not depend on the background-service lock. Messaging requires an authenticated user - guests are closed with `1008`. The receive loop is wrapped in `try/except WebSocketDisconnect` + a broad `except` (never crash the worker) and ALWAYS unregisters the socket in `finally`.
### `_resolve_ws_user` auth resolution order
1.**Session cookie** (`_user_from_session`) - the same-origin `page` mode path; browsers send cookies on a same-origin WS upgrade, so this needs no special handling.
2.**WS ticket** (`ticket` query param, tried AFTER the session-cookie check and BEFORE the header checks - additive, inserted here specifically so it never changes same-origin `page`-mode behavior): if present, `redeem_ticket(ticket)` looks it up; on success it resolves to the ticket's owner user row. This is the ONLY way a real browser can authenticate a cross-context (`embed` mode / third-party) WebSocket, because the native `WebSocket` constructor cannot set custom request headers - there is no way to attach `X-API-KEY`/`Authorization` to `new WebSocket(url)` in any browser. See "WS auth tickets" below for the full mechanism. If the ticket is missing/expired/already-used, resolution simply falls through to the header checks below (never a hard reject at this point - only the final `if not user` in the WS handler closes the socket).
3.**`X-API-KEY` header**, else **`Authorization: Bearer <key>`** - the existing non-browser API-client path (unreachable from a browser's native `WebSocket`, but real and unchanged for programmatic clients that can set headers, e.g. a server-side bot).
Short-lived, single-use tokens that let a browser-based WebSocket authenticate without ever needing to set a request header on the handshake. `POST /messages/ws-ticket` (`require_user`, so it accepts the same HTTP-reachable auth as any other guarded route - session cookie, `X-API-KEY`, or `Authorization: Bearer` - over a normal fetch/XHR, where headers work fine) calls `issue_ticket(user_uid)` and returns `{"ticket": "<opaque hex token>", "expires_in": 30}`. The client then opens `wss://.../messages/ws?ticket=<ticket>` - a query parameter, which a WS handshake CAN carry.
-`issue_ticket(user_uid) -> str` inserts a `ws_tickets` row (`uid` uuid7, `token` = `secrets.token_hex(24)`, `user_uid`, `created_at`, `expires_at` = now + 30s, `used_at=None`) and returns the token.
-`redeem_ticket(token) -> Optional[str]` looks the row up by `token`; returns `None` if missing, already used (`used_at` set), or past `expires_at`; otherwise stamps `used_at` and returns the `user_uid`. Redemption is single-use by construction - a ticket leaked via referrer or browser history cannot be replayed once redeemed, and it self-expires after 30 seconds even if never used.
-`ws_tickets` is NOT in `SOFT_DELETE_TABLES` - it is an ephemeral, single-use artifact, garbage-collected hard via `devplace messaging prune-tickets` (deletes rows where `expires_at < now`), the same shape as the zip/fork/backup `prune` CLI commands. `database/schema.py``init_db()` ensures its columns (`uid`, `token`, `user_uid`, `created_at`, `expires_at`, `used_at`) and a unique index on `token` (the hot lookup path) plus an index on `expires_at` (the prune sweep).
- This is strictly additive to `_resolve_ws_user` - the existing session-cookie and header paths are completely unchanged, so `mode="page"` behavior and every existing non-browser API consumer are unaffected.
The `ConnectionManager` singleton `message_hub` (`services/messaging/hub.py`), **one per worker**: `user_uid -> set[WebSocket]`, plus a bounded (4000-entry) `delivered` LRU dedupe set (`mark_delivered`/`was_delivered`) used to dedupe direct vs relayed delivery. `register`/`unregister` track connections for delivery only - **no presence data lives here** (see below).
Because the WS accepts on every worker, a message persisted on worker A must still reach a recipient whose socket lives on worker B. `services/messaging/relay.py``message_relay` (a per-worker singleton asyncio loop, started on the first socket connect, self-stops when `message_hub.has_connections()` is false) polls `SELECT * FROM messages WHERE id > :watermark` (uuid7-backed autoincrement `id`) every ~1s and pushes any row whose sender/receiver has a LOCAL socket and that was not already `was_delivered` to those local sockets, advancing the watermark to the max row seen. Same-worker sends are delivered INSTANTLY by `broadcast_message` (which calls `message_hub.mark_delivered` so the relay skips them); the relay only fills the cross-worker gap, so same-worker latency is zero and cross-worker latency is bounded by the poll interval. The relay primes its watermark to the current `MAX(id)` on first start so it never replays history. **Trade-off:** typing/read frames are in-process per worker only - across the two `make prod` workers those ephemeral signals reach only same-worker peers (message delivery is always correct). On a single dev worker everything is instant and complete.
`services/messaging/persist.py``persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; `content` itself is allowed to be empty (`MessageForm.content` is `min_length=0`) as long as at least one attachment is present - `persist_message` is the single source of truth for that rule (`if not content and not attachment_uids: return None`), so the HTTP form model deliberately does not duplicate it. Whenever `request` is not `None` it audits via `audit.record(request, ...)` - this covers BOTH the HTTP `Request` and the WS path, since the WS handler passes `request=websocket` and a `WebSocket` object is just as non-`None` as a `Request` (`audit.record` never reads HTTP-specific attributes off it beyond what's already supplied explicitly via `user=sender`). `audit.record_system(..., actor_kind="user", origin=origin)` is the fallback used ONLY when `persist_message` is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same `message.send` key/category either way, no new event invented; typing/read are ephemeral and NOT audited.
`"messages": ("content",)` is in the `CORRECTABLE_FIELDS` registry and `persist_message` invokes `schedule_correction`/`schedule_modification` (sender = the user), so typing `@ai <instruction>` in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through `routers/messages._finalize_and_broadcast(sender, message, request, client_id=None)`: it reads any pending SYNC correction/modification futures stashed on `request.scope[PENDING_SCOPE_KEY]` (the same `PENDING_SCOPE_KEY`/`loop.run_in_executor` mechanism the HTTP `await_pending_corrections` middleware uses) into `ai_processed = bool(pending)`**before** awaiting/clearing them, awaits them, RE-READS the message row, and broadcasts the FINAL (corrected/modified) content via `broadcast_message(sender, message, client_id, ai_processed=ai_processed)`. The HTTP `POST /send` and the WS `/ws` send path both call it; the WS path passes `request=websocket` to `persist_message` so the modifier stashes its executor future on the websocket scope and the handler awaits it directly (HTTP middleware does not run for websockets). A message with no `@ai` directive and correction off has nothing pending, so `ai_processed` is `False` and the broadcast is immediate with zero added latency. `broadcast_message` forwards `ai_processed` straight into `message_frame(..., ai_processed=ai_processed)` - see "WS protocol frames" below for the frame shape. This flag is informational only: the server never retains the pre-AI text anywhere (`table.update` overwrites in place), so it exists purely to drive a client-side "Adjusted by AI" affordance, not any kind of diff/revert capability.
Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`. Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; `ai_processed` is additive - `true` only when the broadcast content is the result of an awaited pending correction/modifier future, so an unmodified send is `false` with zero extra computation), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender). There is no `presence` frame on this socket - see below.
## Presence is NOT part of the messaging WS
Presence is handled entirely by the generic, cross-worker-correct mechanism documented in `devplacepy/services/CLAUDE.md` ("Online presence"): a single `users.last_seen` column, written by the `track_presence` HTTP middleware, read server-side at page load via `presence.is_online(user)`, and kept live client-side by `PresenceManager` (`static/js/PresenceManager.js`) subscribing any `[data-presence-uid]` element to the pub/sub topic `public.presence.{uid}`. The messages page's `#messages-presence` span carries `data-presence-uid`/`data-presence-last-seen` like every other avatar-presence site in the app - it is not messaging-specific code. An earlier design had WS-connect-based presence living in `message_hub` (`is_online`/`last_seen`, an `_announce_presence` helper, and a WS `presence` frame); that design was per-worker only and broke with more than one uvicorn worker, so it was removed outright. `message_hub` today tracks ONLY socket connections for message delivery - it has no presence state, and there is no `presence` frame anywhere in the current WS protocol (client or server). **Never re-implement WS-connect presence on this socket** - reuse `presence.is_online`, the `public.presence.{uid}` topic, and `PresenceManager`, exactly as every other presence surface in the app does.
Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubble` (`static/js/components/AppChat.js`) creates a `<dp-content>` element and sets its `textContent` to the message body - never `innerHTML` - so the element's own client-side pipeline (emoji shortcode -> marked -> `DOMPurify.sanitize` -> highlight.js -> image/YouTube/autolink) renders it safely, exactly like every other live-content surface that uses `<dp-content>`. This gives Discord-style emoji, image and YouTube embeds, autolinking, and sanitization for free.
The messaging frontend is the single self-booting custom element `<dp-chat>` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `templates/messages.html` renders `<dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true">` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, modeled on `PubSubClient.js`'s real exponential backoff instead of a flat-delay retry, same `4013` fast-path), sends over WS with an optimistic pending bubble keyed by `client_id` (the FIRST, unconditional branch of the incoming-frame handler matches `sender_uid === selfUid && client_id` against a pending bubble before any other branch - the fix for the old, permanently-disabled `appendOptimistic`), reconciles on the echoed `message` frame (falling to a `.failed`/tap-to-retry state after an 8s timeout with no echo), appends incoming messages live, auto-scrolls, throttles + shows the typing indicator, flips the read-receipt double-check, live-updates the conversation-list preview + unread dot + ordering, and groups consecutive same-sender messages within a 300s gap (`static/js/chat/MessageGrouping.js`, shared between the initial adopted history and the live-append path). If the socket is not open the send `<form method=POST>` submits normally (no-JS fallback). Presence in `mode="page"` needs no component code - the adopted markup keeps its `data-presence-uid` attributes, so the page-global `PresenceManager` already drives the dot/last-seen label; `<dp-chat mode="embed">` (no page-global `PresenceManager`) opens its own scoped `PubSubClient` subscription instead. An opt-in `ai-indicator="true"` attribute shows a brief "Adjusted by AI" caption when a reconciled echo's `ai_processed` flag is true and the content differs from what was locally typed - never a diff/revert UI. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px, with a `theme`/`accent` attribute override mechanism (scoped inline custom properties on the component's own root) for future off-site embedding.
The single `message_frame` builder (`services/messaging/persist.py`, used by BOTH `broadcast_message` and the relay) fetches `get_attachments("message", uid)` and includes a slimmed list (`uid`/`url`/`thumbnail_url`/`is_image`/`is_video`/`is_audio`/`original_filename`/`file_size`/`mime_type`) on every frame via `_slim_attachment`. `is_audio` is derived identically to every other attachment surface in the app (`mime_type.startswith("audio/")`, already computed by `attachments._row_to_attachment` and simply forwarded here) - `AppChat._renderAttachments` mirrors `_attachment_display.html`'s type branches (image -> `img.gallery-thumb` marked `data-lightbox`+`data-full`, video -> `<video>`, audio -> `<audio controls>`, else a download link) - no `innerHTML`, so it stays XSS-safe, and a live-delivered audio attachment now renders identically to a page-refreshed one instead of falling back to a generic download link until reload. The sender's optimistic bubble shows an "Uploading attachment(s)..." placeholder until its own echo arrives and swaps in the real gallery. **Attachment-only messages (empty caption) are allowed on both send paths**: the WS `send` handler always allowed it (it reads `content` as a raw string with no Pydantic model), and the HTTP `POST /messages/send` form now does too (`MessageForm.content` is `min_length=0`) - `persist_message` accepts empty content when `attachment_uids` is non-empty (`if not content and not attachment_uids: return None`) and is the single place that rule lives. `dp-upload` (mode `attachment`, default `show-chips` off so only the `(N)` count badge shows) uploads each file to `/uploads/upload` and writes the uids into its hidden `attachment_uids` input; `AppChat._collectAttachments()` reads that before the send, then `dp-upload.clear()` resets it. The send button shows a spinner and is disabled while busy: `AppChat._refreshSendButton()` ORs `_uploading` (driven by the `dp-upload:busy` event) with `_pendingSends` (a map of in-flight send `client_id`s, cleared on the echo or an 8s safety timeout that flips the bubble to `.failed`/tap-to-retry), toggling `.is-sending` + `disabled` on `.messages-send-btn`.
