# Messaging (`devplacepy/services/messaging/`)

This file documents the real-time direct-message chat subsystem. Claude Code auto-loads it when a file under `devplacepy/services/messaging/` is read or edited.

## Overview

The `/messages` feature is a live WebSocket chat layered over the SAME `messages` table and audit/notification cores as before; there is no parallel chat store. The WS only adds live delivery, typing, and read receipts on top of the existing persistence - presence is NOT part of this WS (see "Presence is not part of the messaging WS" below).

`messages.py` (the router, mounted at `/messages`) exposes: `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `GET /conversations` (flat JSON conversation list, for a client-side refresh without a full page reload), `POST /send` (no-JS fallback, now also accepting an attachment-only empty-content message), `POST /ws-ticket` (issues a short-lived WS auth ticket), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests).

## Bounded page queries (load-bearing performance rule)

`get_conversations` is ONE window-function query (`ROW_NUMBER() OVER (PARTITION BY other_uid ORDER BY created_at DESC, id DESC)` over `sender_uid = :me OR receiver_uid = :me`, the `get_recent_comments_by_target_uids` precedent) that returns only each partner's latest row - never load the user's full message history into Python to build the sidebar. It backs both the server-rendered `GET /messages` page and the JSON `GET /messages/conversations` route (same helper, no duplicated query). `get_conversation_messages` loads at most `CONVERSATION_MESSAGE_LIMIT` (500) most-recent rows (`ORDER BY created_at DESC, id DESC LIMIT`, reversed to ascending in Python); older history stays in the DB and is simply not rendered. Both are covered by `idx_messages_conversation`/`_rev` (verified `EXPLAIN QUERY PLAN`, MULTI-INDEX OR, no table scan). `messages` is NOT in `SOFT_DELETE_TABLES`, so neither query filters `deleted_at`. Any new message listing must stay bounded and indexed the same way.

## `GET /messages/conversations`

Additive JSON endpoint (`require_user`), added so a frontend component can refresh the conversation list without a full page reload. It calls the exact same `get_conversations(user_uid)` helper `GET /messages` already uses, then projects each entry through `schemas.ConversationOut` (which nests `other_user` as `UserOut`, stripping `email`/`api_key`/`password_hash`) before returning `{"conversations": [...]}`. Like `GET /messages/search`, this is flat JSON, not content-negotiated through `respond()`.

## WS endpoint `/messages/ws`

Lives on the existing messages router (no new router/prefix; already mounted at `/messages`). It is **NOT lock-owner gated** (deliberately - unlike `/devii/ws`): `await websocket.accept()`, resolve the owner via `_resolve_ws_user`, and accept on EVERY worker. Gating on `service_manager.owns_lock()` was the original design and was wrong here: when no worker holds the service lock (services disabled, or the lock not yet acquired) every socket got closed `4013` and the client fast-retried every 200ms forever (a connection storm with no stream, and the send form fell back to a full-page POST reload). Chat must not depend on the background-service lock. Messaging requires an authenticated user - guests are closed with `1008`. The receive loop is wrapped in `try/except WebSocketDisconnect` + a broad `except` (never crash the worker) and ALWAYS unregisters the socket in `finally`.

### `_resolve_ws_user` auth resolution order

1. **Session cookie** (`_user_from_session`) - the same-origin `page` mode path; browsers send cookies on a same-origin WS upgrade, so this needs no special handling.
2. **WS ticket** (`ticket` query param, tried AFTER the session-cookie check and BEFORE the header checks - additive, inserted here specifically so it never changes same-origin `page`-mode behavior): if present, `redeem_ticket(ticket)` looks it up; on success it resolves to the ticket's owner user row. This is the ONLY way a real browser can authenticate a cross-context (`embed` mode / third-party) WebSocket, because the native `WebSocket` constructor cannot set custom request headers - there is no way to attach `X-API-KEY`/`Authorization` to `new WebSocket(url)` in any browser. See "WS auth tickets" below for the full mechanism. If the ticket is missing/expired/already-used, resolution simply falls through to the header checks below (never a hard reject at this point - only the final `if not user` in the WS handler closes the socket).
3. **`X-API-KEY` header**, else **`Authorization: Bearer <key>`** - the existing non-browser API-client path (unreachable from a browser's native `WebSocket`, but real and unchanged for programmatic clients that can set headers, e.g. a server-side bot).

## WS auth tickets (`services/messaging/tickets.py`, table `ws_tickets`)

Short-lived, single-use tokens that let a browser-based WebSocket authenticate without ever needing to set a request header on the handshake. `POST /messages/ws-ticket` (`require_user`, so it accepts the same HTTP-reachable auth as any other guarded route - session cookie, `X-API-KEY`, or `Authorization: Bearer` - over a normal fetch/XHR, where headers work fine) calls `issue_ticket(user_uid)` and returns `{"ticket": "<opaque hex token>", "expires_in": 30}`. The client then opens `wss://.../messages/ws?ticket=<ticket>` - a query parameter, which a WS handshake CAN carry.

- `issue_ticket(user_uid) -> str` inserts a `ws_tickets` row (`uid` uuid7, `token` = `secrets.token_hex(24)`, `user_uid`, `created_at`, `expires_at` = now + 30s, `used_at=None`) and returns the token.
- `redeem_ticket(token) -> Optional[str]` looks the row up by `token`; returns `None` if missing, already used (`used_at` set), or past `expires_at`; otherwise stamps `used_at` and returns the `user_uid`. Redemption is single-use by construction - a ticket leaked via referrer or browser history cannot be replayed once redeemed, and it self-expires after 30 seconds even if never used.
- `ws_tickets` is NOT in `SOFT_DELETE_TABLES` - it is an ephemeral, single-use artifact, garbage-collected hard via `devplace messaging prune-tickets` (deletes rows where `expires_at < now`), the same shape as the zip/fork/backup `prune` CLI commands. `database/schema.py` `init_db()` ensures its columns (`uid`, `token`, `user_uid`, `created_at`, `expires_at`, `used_at`) and a unique index on `token` (the hot lookup path) plus an index on `expires_at` (the prune sweep).
- This is strictly additive to `_resolve_ws_user` - the existing session-cookie and header paths are completely unchanged, so `mode="page"` behavior and every existing non-browser API consumer are unaffected.

## Connection registry

The `ConnectionManager` singleton `message_hub` (`services/messaging/hub.py`), **one per worker**: `user_uid -> set[WebSocket]`, plus a bounded (4000-entry) `delivered` LRU dedupe set (`mark_delivered`/`was_delivered`) used to dedupe direct vs relayed delivery. `register`/`unregister` track connections for delivery only - **no presence data lives here** (see below).

## Cross-worker delivery (the relay)

Because the WS accepts on every worker, a message persisted on worker A must still reach a recipient whose socket lives on worker B. `services/messaging/relay.py` `message_relay` (a per-worker singleton asyncio loop, started on the first socket connect, self-stops when `message_hub.has_connections()` is false) polls `SELECT * FROM messages WHERE id > :watermark` (uuid7-backed autoincrement `id`) every ~1s and pushes any row whose sender/receiver has a LOCAL socket and that was not already `was_delivered` to those local sockets, advancing the watermark to the max row seen. Same-worker sends are delivered INSTANTLY by `broadcast_message` (which calls `message_hub.mark_delivered` so the relay skips them); the relay only fills the cross-worker gap, so same-worker latency is zero and cross-worker latency is bounded by the poll interval. The relay primes its watermark to the current `MAX(id)` on first start so it never replays history. **Trade-off:** typing/read frames are in-process per worker only - across the two `make prod` workers those ephemeral signals reach only same-worker peers (message delivery is always correct). On a single dev worker everything is instant and complete.

## DRY persist choke point

`services/messaging/persist.py` `persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; `content` itself is allowed to be empty (`MessageForm.content` is `min_length=0`) as long as at least one attachment is present - `persist_message` is the single source of truth for that rule (`if not content and not attachment_uids: return None`), so the HTTP form model deliberately does not duplicate it. Whenever `request` is not `None` it audits via `audit.record(request, ...)` - this covers BOTH the HTTP `Request` and the WS path, since the WS handler passes `request=websocket` and a `WebSocket` object is just as non-`None` as a `Request` (`audit.record` never reads HTTP-specific attributes off it beyond what's already supplied explicitly via `user=sender`). `audit.record_system(..., actor_kind="user", origin=origin)` is the fallback used ONLY when `persist_message` is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same `message.send` key/category either way, no new event invented; typing/read are ephemeral and NOT audited.

## AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content

`"messages": ("content",)` is in the `CORRECTABLE_FIELDS` registry and `persist_message` invokes `schedule_correction`/`schedule_modification` (sender = the user), so typing `@ai <instruction>` in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through `routers/messages._finalize_and_broadcast`: it broadcasts the persisted row immediately (`ai_pending=true` when sync futures were stashed on `request.scope[PENDING_SCOPE_KEY]`), then applies any pending SYNC correction/modifier. The HTTP `POST /send` path awaits that apply so the JSON body is the final text; the WS send path schedules it with `asyncio.create_task` so the receive loop never blocks on the gateway. After an in-place rewrite, `_run_correction`/`_run_modification` call `push_content_revision` which stamps `messages.updated_at` and pushes a second `message` frame (`ai_processed=true`) to local sockets. Cross-worker peers pick the revision up from `message_relay._tick_updates` (`SELECT ... WHERE updated_at > watermark`, independent of the new-row `id` watermark and of `was_delivered`). The client matches the second frame on `data-msg-uid` and replaces the bubble body. The WS loop also accepts `{type:"sync", since, with_uid?}` to replay rows created or revised after `since` (reconnect catch-up) and `{type:"ping"}` (ignored). `ContentRefused` on a WS send returns `{type:"error"}` and keeps the socket open.

## WS protocol frames

Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below.

## Presence is NOT part of the messaging WS

Presence is handled entirely by the generic, cross-worker-correct mechanism documented in `devplacepy/services/CLAUDE.md` ("Online presence"): a single `users.last_seen` column, written by the `track_presence` HTTP middleware, read server-side at page load via `presence.is_online(user)`, and kept live client-side by `PresenceManager` (`static/js/PresenceManager.js`) subscribing any `[data-presence-uid]` element to the single pub/sub topic `public.presence.roster`. The messages page's `#messages-presence` span carries `data-presence-uid`/`data-presence-last-seen` like every other avatar-presence site in the app - it is not messaging-specific code. An earlier design had WS-connect-based presence living in `message_hub` (`is_online`/`last_seen`, an `_announce_presence` helper, and a WS `presence` frame); that design was per-worker only and broke with more than one uvicorn worker, so it was removed outright. `message_hub` today tracks ONLY socket connections for message delivery - it has no presence state, and there is no `presence` frame anywhere in the current WS protocol (client or server). **Never re-implement WS-connect presence on this socket** - reuse `presence.is_online`, the `public.presence.roster` topic, and `PresenceManager`, exactly as every other presence surface in the app does. `dp-chat` also no longer owns a private presence renderer or `PubSubClient` - that duplicate disagreed with the feed roster and was removed.

## Renderer integration (XSS control)

Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubble` (`static/js/components/AppChat.js`) creates a `<dp-content>` element and sets its `textContent` to the message body - never `innerHTML` - so the element's own client-side pipeline (emoji shortcode -> marked -> `DOMPurify.sanitize` -> highlight.js -> image/YouTube/autolink) renders it safely, exactly like every other live-content surface that uses `<dp-content>`. This gives Discord-style emoji, image and YouTube embeds, autolinking, and sanitization for free.

## Frontend

The messaging frontend is the single self-booting custom element `<dp-chat>` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `<dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true">` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. The send button is disabled only while `dp-upload` is busy, never while a send is in flight. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px.

## Attachments stream live

The single `message_frame` builder (`services/messaging/persist.py`, used by BOTH `broadcast_message` and the relay) fetches `get_attachments("message", uid)` and includes a slimmed list (`uid`/`url`/`thumbnail_url`/`is_image`/`is_video`/`is_audio`/`original_filename`/`file_size`/`mime_type`) on every frame via `_slim_attachment`. `is_audio` is derived identically to every other attachment surface in the app (`mime_type.startswith("audio/")`, already computed by `attachments._row_to_attachment` and simply forwarded here) - `AppChat._renderAttachments` mirrors `_attachment_display.html`'s type branches (image -> `img.gallery-thumb` marked `data-lightbox`+`data-full`, video -> `<video>`, audio -> `<audio controls>`, else a download link) - no `innerHTML`, so it stays XSS-safe, and a live-delivered audio attachment now renders identically to a page-refreshed one instead of falling back to a generic download link until reload. The sender's optimistic bubble shows an "Uploading attachment(s)..." placeholder until its own echo arrives and swaps in the real gallery. **Attachment-only messages (empty caption) are allowed on both send paths**: the WS `send` handler always allowed it (it reads `content` as a raw string with no Pydantic model), and the HTTP `POST /messages/send` form now does too (`MessageForm.content` is `min_length=0`) - `persist_message` accepts empty content when `attachment_uids` is non-empty (`if not content and not attachment_uids: return None`) and is the single place that rule lives. `dp-upload` (mode `attachment`, default `show-chips` off so only the `(N)` count badge shows) uploads each file to `/uploads/upload` and writes the uids into its hidden `attachment_uids` input; `AppChat._collectAttachments()` reads that before the send, then `dp-upload.clear()` resets it. The send button shows a spinner and is disabled only while `dp-upload` is busy (`dp-upload:busy`); in-flight sends stay in `_pendingSends` and flip the bubble to `.failed`/tap-to-retry if no echo arrives (20s), without locking the composer.
