This file documents the real-time direct-message chat subsystem. Claude Code auto-loads it when a file under `devplacepy/services/messaging/` is read or edited.
The `/messages` feature is a live WebSocket chat layered over the SAME `messages` table and audit/notification cores as before; there is no parallel chat store. The WS only adds live delivery, typing, and read receipts on top of the existing persistence - presence is NOT part of this WS (see "Presence is not part of the messaging WS" below).
`messages.py` (the router, mounted at `/messages`) exposes: `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `GET /conversations` (flat JSON conversation list, for a client-side refresh without a full page reload), `POST /send` (no-JS fallback, now also accepting an attachment-only empty-content message), `POST /ws-ticket` (issues a short-lived WS auth ticket), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests).
`get_conversations` is ONE window-function query (`ROW_NUMBER() OVER (PARTITION BY other_uid ORDER BY created_at DESC, id DESC)` over `sender_uid = :me OR receiver_uid = :me`, the `get_recent_comments_by_target_uids` precedent) that returns only each partner's latest row - never load the user's full message history into Python to build the sidebar. It backs both the server-rendered `GET /messages` page and the JSON `GET /messages/conversations` route (same helper, no duplicated query). `get_conversation_messages` loads at most `CONVERSATION_MESSAGE_LIMIT` (500) most-recent rows (`ORDER BY created_at DESC, id DESC LIMIT`, reversed to ascending in Python); older history stays in the DB and is simply not rendered. Both are covered by `idx_messages_conversation`/`_rev` (verified `EXPLAIN QUERY PLAN`, MULTI-INDEX OR, no table scan). `messages` is NOT in `SOFT_DELETE_TABLES`, so neither query filters `deleted_at`. Any new message listing must stay bounded and indexed the same way.
## `GET /messages/conversations`
Additive JSON endpoint (`require_user`), added so a frontend component can refresh the conversation list without a full page reload. It calls the exact same `get_conversations(user_uid)` helper `GET /messages` already uses, then projects each entry through `schemas.ConversationOut` (which nests `other_user` as `UserOut`, stripping `email`/`api_key`/`password_hash`) before returning `{"conversations": [...]}`. Like `GET /messages/search`, this is flat JSON, not content-negotiated through `respond()`.
Lives on the existing messages router (no new router/prefix; already mounted at `/messages`). It is **NOT lock-owner gated** (deliberately - unlike `/devii/ws`): `await websocket.accept()`, resolve the owner via `_resolve_ws_user`, and accept on EVERY worker. Gating on `service_manager.owns_lock()` was the original design and was wrong here: when no worker holds the service lock (services disabled, or the lock not yet acquired) every socket got closed `4013` and the client fast-retried every 200ms forever (a connection storm with no stream, and the send form fell back to a full-page POST reload). Chat must not depend on the background-service lock. Messaging requires an authenticated user - guests are closed with `1008`. The receive loop is wrapped in `try/except WebSocketDisconnect` + a broad `except` (never crash the worker) and ALWAYS unregisters the socket in `finally`.
### `_resolve_ws_user` auth resolution order
1.**Session cookie** (`_user_from_session`) - the same-origin `page` mode path; browsers send cookies on a same-origin WS upgrade, so this needs no special handling.
2.**WS ticket** (`ticket` query param, tried AFTER the session-cookie check and BEFORE the header checks - additive, inserted here specifically so it never changes same-origin `page`-mode behavior): if present, `redeem_ticket(ticket)` looks it up; on success it resolves to the ticket's owner user row. This is the ONLY way a real browser can authenticate a cross-context (`embed` mode / third-party) WebSocket, because the native `WebSocket` constructor cannot set custom request headers - there is no way to attach `X-API-KEY`/`Authorization` to `new WebSocket(url)` in any browser. See "WS auth tickets" below for the full mechanism. If the ticket is missing/expired/already-used, resolution simply falls through to the header checks below (never a hard reject at this point - only the final `if not user` in the WS handler closes the socket).
3.**`X-API-KEY` header**, else **`Authorization: Bearer <key>`** - the existing non-browser API-client path (unreachable from a browser's native `WebSocket`, but real and unchanged for programmatic clients that can set headers, e.g. a server-side bot).
Short-lived, single-use tokens that let a browser-based WebSocket authenticate without ever needing to set a request header on the handshake. `POST /messages/ws-ticket` (`require_user`, so it accepts the same HTTP-reachable auth as any other guarded route - session cookie, `X-API-KEY`, or `Authorization: Bearer` - over a normal fetch/XHR, where headers work fine) calls `issue_ticket(user_uid)` and returns `{"ticket": "<opaque hex token>", "expires_in": 30}`. The client then opens `wss://.../messages/ws?ticket=<ticket>` - a query parameter, which a WS handshake CAN carry.
-`issue_ticket(user_uid) -> str` inserts a `ws_tickets` row (`uid` uuid7, `token` = `secrets.token_hex(24)`, `user_uid`, `created_at`, `expires_at` = now + 30s, `used_at=None`) and returns the token.
-`ws_tickets` also carries `scope` (empty or `messages` for chat, `nodes` for the node socket) and `device_uid`. `redeem_ticket_row(token, scope)` is the one conditional `UPDATE` (single use, unexpired, `COALESCE(scope, 'messages') = :scope`), so a node ticket can never open a chat socket and the reverse. `issue_ticket(user_uid, scope=, device_uid=, ttl_seconds=)`.
-`redeem_ticket(token) -> Optional[str]` looks the row up by `token`; returns `None` if missing, already used (`used_at` set), or past `expires_at`; otherwise stamps `used_at` and returns the `user_uid`. Redemption is single-use by construction - a ticket leaked via referrer or browser history cannot be replayed once redeemed, and it self-expires after 30 seconds even if never used.
-`ws_tickets` is NOT in `SOFT_DELETE_TABLES` - it is an ephemeral, single-use artifact, garbage-collected hard via `devplace messaging prune-tickets` (deletes rows where `expires_at < now`), the same shape as the zip/fork/backup `prune` CLI commands. `database/schema.py``init_db()` ensures its columns (`uid`, `token`, `user_uid`, `created_at`, `expires_at`, `used_at`) and a unique index on `token` (the hot lookup path) plus an index on `expires_at` (the prune sweep).
- This is strictly additive to `_resolve_ws_user` - the existing session-cookie and header paths are completely unchanged, so `mode="page"` behavior and every existing non-browser API consumer are unaffected.
The `ConnectionManager` singleton `message_hub` (`services/messaging/hub.py`), **one per worker**: `user_uid -> set[WebSocket]`, plus a bounded (4000-entry) `delivered` LRU dedupe set (`mark_delivered`/`was_delivered`) used to dedupe direct vs relayed delivery. `register`/`unregister` track connections for delivery only - **no presence data lives here** (see below).
Because the WS accepts on every worker, a message persisted on worker A must still reach a recipient whose socket lives on worker B. `services/messaging/relay.py``message_relay` (a per-worker singleton asyncio loop, started on the first socket connect, self-stops when `message_hub.has_connections()` is false) polls `SELECT * FROM messages WHERE id > :watermark` (uuid7-backed autoincrement `id`) every ~1s and pushes any row whose sender/receiver has a LOCAL socket and that was not already `was_delivered` to those local sockets, advancing the watermark to the max row seen. Same-worker sends are delivered INSTANTLY by `broadcast_message` (which calls `message_hub.mark_delivered` so the relay skips them); the relay only fills the cross-worker gap, so same-worker latency is zero and cross-worker latency is bounded by the poll interval. The relay primes its watermark to the current `MAX(id)` on first start so it never replays history. **Trade-off:** typing/read frames are in-process per worker only - across the two `make prod` workers those ephemeral signals reach only same-worker peers (message delivery is always correct). On a single dev worker everything is instant and complete.
`services/messaging/persist.py``persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin, client_id=None)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; `content` itself is allowed to be empty (`MessageForm.content` is `min_length=0`) as long as at least one attachment is present - `persist_message` is the single source of truth for that rule (`if not content and not attachment_uids: return None`), so the HTTP form model deliberately does not duplicate it. Whenever `request` is not `None` it audits via `audit.record(request, ...)` - this covers BOTH the HTTP `Request` and the WS path, since the WS handler passes `request=websocket` and a `WebSocket` object is just as non-`None` as a `Request` (`audit.record` never reads HTTP-specific attributes off it beyond what's already supplied explicitly via `user=sender`). `audit.record_system(..., actor_kind="user", origin=origin)` is the fallback used ONLY when `persist_message` is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same `message.send` key/category either way, no new event invented; typing/read are ephemeral and NOT audited.
### Send dedupe (double-submit safety)
`persist_message` dedupes on `(sender_uid, client_id)`**before** it ever inserts a row: `_find_recent_duplicate(sender_uid, client_id)` looks up the most recent `messages` row with the same `sender_uid` and the same client-supplied `client_id` whose `created_at` falls inside `DEDUPE_WINDOW_SECONDS` (30s). A hit short-circuits `persist_message` and returns that ALREADY-persisted row verbatim (no error, no second insert, no second AI correction/notification/audit) - both the WS `send` handler and `POST /messages/send` now pass their `client_id` straight through, so a network-level retry that resends the identical payload (same `client_id`) collapses onto the original row instead of producing a second, independently-AI-corrected message. `client_id` is attacker-controlled but the dedupe key is always scoped by the server-resolved `sender_uid`, so a client can only dedupe its own sends. The column is nullable (`messages.client_id`, ensured in `init_db()`, indexed by `idx_messages_dedupe (sender_uid, client_id)`); a request with no `client_id` (any future non-`dp-chat` caller) skips the check entirely and always inserts. This is defense-in-depth alongside the client-side send-lock below - the lock stops a human double-click from ever firing two requests, the server dedupe stops a retried *identical* request (same `client_id`) from ever becoming two rows.
**Client-side send-lock (`AppChat.js`).**`_initComposer`'s submit handler and the Enter-to-send path both funnel through one `form submit` listener, which now bails out while `this._sendLocked` is true. `_sendViaSocket` sets the lock (and disables the send button via the existing `_refreshSendButton()`, reusing the `.is-sending` spinner state) for every genuine new send (never for a `.failed`-bubble retry, tracked separately so a retry never blocks a fresh message) and only releases it once that specific `client_id` clears out of `_pendingSends` - on the echoed reconcile (`_reconcilePending` -> `_clearPendingSend`) or the 20s send timeout (`_failPendingSend`). Multiple sends never race the lock: `_lockingSends` is a `Set` of in-flight new-send `client_id`s, and the composer re-enables only when it is empty.
`"messages": ("content",)` is in the `CORRECTABLE_FIELDS` registry and `persist_message` invokes `schedule_correction`/`schedule_modification` (sender = the user), so typing `@ai <instruction>` in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through `routers/messages._finalize_and_broadcast`: it broadcasts the persisted row immediately (`ai_pending=true` when sync futures were stashed on `request.scope[PENDING_SCOPE_KEY]`), then applies any pending SYNC correction/modifier. The HTTP `POST /send` path awaits that apply so the JSON body is the final text; the WS send path schedules it with `asyncio.create_task` so the receive loop never blocks on the gateway. After an in-place rewrite, `_run_correction`/`_run_modification` call `push_content_revision` which stamps `messages.updated_at` and pushes a second `message` frame (`ai_processed=true`) to local sockets. Cross-worker peers pick the revision up from `message_relay._tick_updates` (`SELECT ... WHERE updated_at > watermark`, independent of the new-row `id` watermark and of `was_delivered`). The client matches the second frame on `data-msg-uid` and replaces the bubble body. The WS loop also accepts `{type:"sync", since, with_uid?}` to replay rows created or revised after `since` (reconnect catch-up) and `{type:"ping"}` (ignored). `ContentRefused` on a WS send returns `{type:"error"}` and keeps the socket open.
Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"active", with_uid}` (marks the sender as currently viewing that conversation - see "Active-conversation notification suppression" below), `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below. `active` has no server -> client echo; it only writes the DB-backed marker consumed server-side by `persist_message`.
## Active-conversation notification suppression
A `message` notification (and its live toast) is suppressed when the receiver is demonstrably looking at that exact conversation right now, so an open DM thread never also produces a redundant bell/toast for the same message. The message itself still delivers live over the WS/relay path exactly as before - only the `create_notification("message", ...)` call in `persist_message` is skipped.
- **Marker: two columns on `users`**, `active_conversation_uid` (the uid of the conversation partner currently being viewed) and `active_conversation_at` (UTC ISO timestamp of the last time it was refreshed), ensured in `database.backfill_api_keys()` like every other non-signup `users` column. This follows the `services/presence.py``last_seen` pattern deliberately - a WS connection is per-worker (`message_hub` has no cross-worker visibility), but the sender and receiver of a DM can be on different uvicorn workers, so the "is the receiver looking at this" check has to be readable from ANY worker. A DB column is the only cross-worker-correct medium here, exactly as presence is.
- **Write path**: `services/messaging/active_conversation.py``touch_active_conversation(viewer_uid, with_uid)`, called from the WS handler's new `{type:"active", with_uid}` frame. Throttled per-worker exactly like `presence.touch` (`WRITE_THROTTLE_SECONDS = 5`, an in-memory `dict[f"{viewer_uid}:{with_uid}" -> monotonic]` so a burst of focus/visibility events costs at most one write per 5s per conversation) before the `UPDATE users SET active_conversation_uid=..., active_conversation_at=...` write.
- **Client trigger (`AppChat.js`)**: `_sendActiveMarker()` sends the frame whenever `_threadIsActive()` is true (desktop: any open thread; mobile: only when the thread pane, not the list pane, is showing). It fires wherever the existing `_markRead()` already fires (initial WS connect, opening/switching a conversation) plus two new listeners bound in `_bindActiveConversation()`: `window``focus` and `document``visibilitychange` (`visibilityState === "visible"`) - so returning to a backgrounded/blurred tab that still has the conversation open re-asserts the marker.
- **Read path / freshness window**: `is_actively_viewing(viewer_uid, other_uid)` (same module) reads the receiver's user row, checks `active_conversation_uid == other_uid`, and requires `active_conversation_at` to be less than `FRESH_SECONDS = 20` old. `persist_message` calls it as `is_actively_viewing(receiver_uid, sender_uid)` right before the `create_notification` call and skips only that call on a hit; `clear_messages_cache(receiver_uid)` still runs unconditionally, and the WS/relay delivery of the message frame is completely unaffected (it is broadcast from `routers/messages.py`, not from `persist_message`). The freshness window (20s) is intentionally wider than the write throttle (5s) so it never lapses between two consecutive throttled writes, and the marker is a soft "recently confirmed as active" signal, not a hard connection state - if it goes stale (tab closed, thread switched away without a fresh trigger) the very next message just resumes notifying normally, which is the safe failure direction.
Presence is handled entirely by the generic, cross-worker-correct mechanism documented in `devplacepy/services/CLAUDE.md` ("Online presence"): a single `users.last_seen` column, written by the `track_presence` HTTP middleware, read server-side at page load via `presence.is_online(user)`, and kept live client-side by `PresenceManager` (`static/js/PresenceManager.js`) subscribing any `[data-presence-uid]` element to the single pub/sub topic `public.presence.roster`. The messages page's `#messages-presence` span carries `data-presence-uid`/`data-presence-last-seen` like every other avatar-presence site in the app - it is not messaging-specific code. An earlier design had WS-connect-based presence living in `message_hub` (`is_online`/`last_seen`, an `_announce_presence` helper, and a WS `presence` frame); that design was per-worker only and broke with more than one uvicorn worker, so it was removed outright. `message_hub` today tracks ONLY socket connections for message delivery - it has no presence state, and there is no `presence` frame anywhere in the current WS protocol (client or server). **Never re-implement WS-connect presence on this socket** - reuse `presence.is_online`, the `public.presence.roster` topic, and `PresenceManager`, exactly as every other presence surface in the app does. `dp-chat` also no longer owns a private presence renderer or `PubSubClient` - that duplicate disagreed with the feed roster and was removed.
Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubble` (`static/js/components/AppChat.js`) creates a `<dp-content>` element and sets its `textContent` to the message body - never `innerHTML` - so the element's own client-side pipeline (emoji shortcode -> marked -> `DOMPurify.sanitize` -> highlight.js -> image/YouTube and YouTube Music/autolink) renders it safely, exactly like every other live-content surface that uses `<dp-content>`. This gives Discord-style emoji, image, YouTube and YouTube Music embeds (`static/js/EmbedParser.js`, spec in `devplacepy/embeds.py`), autolinking, and sanitization for free.
The messaging frontend is the single self-booting custom element `<dp-chat>` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `<dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true">` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. **The send button is disabled while `dp-upload` is busy AND while a genuine new send is in flight** (the send-lock, see "Send dedupe" above) - a `.failed`-bubble retry never engages it, so retrying an old message never blocks typing/sending a new one. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px.
**Auto-scroll never fights a manual scroll-up.**`_startScrollWatcher`/`_stabilizeScroll` force `scrollTop = scrollHeight` on every thread mutation (new bubble, attachment/image finishing loading) ONLY while `_userAtBottom` is true, re-checked every `requestAnimationFrame` up to `STABILIZE_MAX_FRAMES`. The one thing that used to break this: while a multi-frame stabilize loop was in flight (`_stabilizePending`), the real `scroll` event listener (`_bindAutoScroll`) ignored EVERY scroll event outright, including a genuine user wheel/touch scroll-up that happened to land inside that window - so a rapid burst of incoming messages (each restarting the stabilize loop while attachments were still loading) could silently overwrite a user's attempt to scroll up to read history. The fix: every programmatic scroll (`_scrollThreadToEnd`, `_stabilizeScroll`) records the exact `scrollTop` it just set in `_lastForcedScrollTop`; the `scroll` listener only skips its own echo (`_stabilizePending && scrollTop === _lastForcedScrollTop`) and always processes a scroll that lands anywhere else, so a real user scroll-up is detected and `_userAtBottom` flips to `false` immediately, even mid-stabilization. Never gate scroll-intent detection on `_stabilizePending` alone again.
The single `message_frame` builder (`services/messaging/persist.py`, used by BOTH `broadcast_message` and the relay) fetches `get_attachments("message", uid)` and includes a slimmed list (`uid`/`url`/`thumbnail_url`/`is_image`/`is_video`/`is_audio`/`original_filename`/`file_size`/`mime_type`) on every frame via `_slim_attachment`. `is_audio` is derived identically to every other attachment surface in the app (`mime_type.startswith("audio/")`, already computed by `attachments._row_to_attachment` and simply forwarded here) - `AppChat._renderAttachments` mirrors `_attachment_display.html`'s type branches (image -> `img.gallery-thumb` marked `data-lightbox`+`data-full`, video -> `<video>`, audio -> `<audio controls>`, else a download link) - no `innerHTML`, so it stays XSS-safe, and a live-delivered audio attachment now renders identically to a page-refreshed one instead of falling back to a generic download link until reload. The sender's optimistic bubble shows an "Uploading attachment(s)..." placeholder until its own echo arrives and swaps in the real gallery. **Attachment-only messages (empty caption) are allowed on both send paths**: the WS `send` handler always allowed it (it reads `content` as a raw string with no Pydantic model), and the HTTP `POST /messages/send` form now does too (`MessageForm.content` is `min_length=0`) - `persist_message` accepts empty content when `attachment_uids` is non-empty (`if not content and not attachment_uids: return None`) and is the single place that rule lives. `dp-upload` (mode `attachment`, default `show-chips` off so only the `(N)` count badge shows) uploads each file to `/uploads/upload` and writes the uids into its hidden `attachment_uids` input; `AppChat._collectAttachments()` reads that before the send, then `dp-upload.clear()` resets it. The send button shows a spinner and is disabled only while `dp-upload` is busy (`dp-upload:busy`); in-flight sends stay in `_pendingSends` and flip the bubble to `.failed`/tap-to-retry if no echo arrives (20s), without locking the composer.