Add personal notes, DeepSearch history, backup offload, and gateway auth throttling
DevPlace CI / test (push) Failing after 28m20s

Also streamline the top navigation: drop the Tools dropdown, make Quizzes
and Battles icon-only entries, and remove the Workspace, Containers and
Editor entry points from the project detail page.
This commit is contained in:
2026-09-12 20:32:31 +02:00
parent c4f7d01b2d
commit cae139ead9
158 changed files with 6156 additions and 377 deletions
+20 -3
View File
@@ -45,7 +45,13 @@ Because the WS accepts on every worker, a message persisted on worker A must sti
## DRY persist choke point
`services/messaging/persist.py` `persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; `content` itself is allowed to be empty (`MessageForm.content` is `min_length=0`) as long as at least one attachment is present - `persist_message` is the single source of truth for that rule (`if not content and not attachment_uids: return None`), so the HTTP form model deliberately does not duplicate it. Whenever `request` is not `None` it audits via `audit.record(request, ...)` - this covers BOTH the HTTP `Request` and the WS path, since the WS handler passes `request=websocket` and a `WebSocket` object is just as non-`None` as a `Request` (`audit.record` never reads HTTP-specific attributes off it beyond what's already supplied explicitly via `user=sender`). `audit.record_system(..., actor_kind="user", origin=origin)` is the fallback used ONLY when `persist_message` is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same `message.send` key/category either way, no new event invented; typing/read are ephemeral and NOT audited.
`services/messaging/persist.py` `persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin, client_id=None)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; `content` itself is allowed to be empty (`MessageForm.content` is `min_length=0`) as long as at least one attachment is present - `persist_message` is the single source of truth for that rule (`if not content and not attachment_uids: return None`), so the HTTP form model deliberately does not duplicate it. Whenever `request` is not `None` it audits via `audit.record(request, ...)` - this covers BOTH the HTTP `Request` and the WS path, since the WS handler passes `request=websocket` and a `WebSocket` object is just as non-`None` as a `Request` (`audit.record` never reads HTTP-specific attributes off it beyond what's already supplied explicitly via `user=sender`). `audit.record_system(..., actor_kind="user", origin=origin)` is the fallback used ONLY when `persist_message` is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same `message.send` key/category either way, no new event invented; typing/read are ephemeral and NOT audited.
### Send dedupe (double-submit safety)
`persist_message` dedupes on `(sender_uid, client_id)` **before** it ever inserts a row: `_find_recent_duplicate(sender_uid, client_id)` looks up the most recent `messages` row with the same `sender_uid` and the same client-supplied `client_id` whose `created_at` falls inside `DEDUPE_WINDOW_SECONDS` (30s). A hit short-circuits `persist_message` and returns that ALREADY-persisted row verbatim (no error, no second insert, no second AI correction/notification/audit) - both the WS `send` handler and `POST /messages/send` now pass their `client_id` straight through, so a network-level retry that resends the identical payload (same `client_id`) collapses onto the original row instead of producing a second, independently-AI-corrected message. `client_id` is attacker-controlled but the dedupe key is always scoped by the server-resolved `sender_uid`, so a client can only dedupe its own sends. The column is nullable (`messages.client_id`, ensured in `init_db()`, indexed by `idx_messages_dedupe (sender_uid, client_id)`); a request with no `client_id` (any future non-`dp-chat` caller) skips the check entirely and always inserts. This is defense-in-depth alongside the client-side send-lock below - the lock stops a human double-click from ever firing two requests, the server dedupe stops a retried *identical* request (same `client_id`) from ever becoming two rows.
**Client-side send-lock (`AppChat.js`).** `_initComposer`'s submit handler and the Enter-to-send path both funnel through one `form submit` listener, which now bails out while `this._sendLocked` is true. `_sendViaSocket` sets the lock (and disables the send button via the existing `_refreshSendButton()`, reusing the `.is-sending` spinner state) for every genuine new send (never for a `.failed`-bubble retry, tracked separately so a retry never blocks a fresh message) and only releases it once that specific `client_id` clears out of `_pendingSends` - on the echoed reconcile (`_reconcilePending` -> `_clearPendingSend`) or the 20s send timeout (`_failPendingSend`). Multiple sends never race the lock: `_lockingSends` is a `Set` of in-flight new-send `client_id`s, and the composer re-enables only when it is empty.
## AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content
@@ -53,7 +59,16 @@ Because the WS accepts on every worker, a message persisted on worker A must sti
## WS protocol frames
Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below.
Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"active", with_uid}` (marks the sender as currently viewing that conversation - see "Active-conversation notification suppression" below), `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below. `active` has no server -> client echo; it only writes the DB-backed marker consumed server-side by `persist_message`.
## Active-conversation notification suppression
A `message` notification (and its live toast) is suppressed when the receiver is demonstrably looking at that exact conversation right now, so an open DM thread never also produces a redundant bell/toast for the same message. The message itself still delivers live over the WS/relay path exactly as before - only the `create_notification("message", ...)` call in `persist_message` is skipped.
- **Marker: two columns on `users`**, `active_conversation_uid` (the uid of the conversation partner currently being viewed) and `active_conversation_at` (UTC ISO timestamp of the last time it was refreshed), ensured in `database.backfill_api_keys()` like every other non-signup `users` column. This follows the `services/presence.py` `last_seen` pattern deliberately - a WS connection is per-worker (`message_hub` has no cross-worker visibility), but the sender and receiver of a DM can be on different uvicorn workers, so the "is the receiver looking at this" check has to be readable from ANY worker. A DB column is the only cross-worker-correct medium here, exactly as presence is.
- **Write path**: `services/messaging/active_conversation.py` `touch_active_conversation(viewer_uid, with_uid)`, called from the WS handler's new `{type:"active", with_uid}` frame. Throttled per-worker exactly like `presence.touch` (`WRITE_THROTTLE_SECONDS = 5`, an in-memory `dict[f"{viewer_uid}:{with_uid}" -> monotonic]` so a burst of focus/visibility events costs at most one write per 5s per conversation) before the `UPDATE users SET active_conversation_uid=..., active_conversation_at=...` write.
- **Client trigger (`AppChat.js`)**: `_sendActiveMarker()` sends the frame whenever `_threadIsActive()` is true (desktop: any open thread; mobile: only when the thread pane, not the list pane, is showing). It fires wherever the existing `_markRead()` already fires (initial WS connect, opening/switching a conversation) plus two new listeners bound in `_bindActiveConversation()`: `window` `focus` and `document` `visibilitychange` (`visibilityState === "visible"`) - so returning to a backgrounded/blurred tab that still has the conversation open re-asserts the marker.
- **Read path / freshness window**: `is_actively_viewing(viewer_uid, other_uid)` (same module) reads the receiver's user row, checks `active_conversation_uid == other_uid`, and requires `active_conversation_at` to be less than `FRESH_SECONDS = 20` old. `persist_message` calls it as `is_actively_viewing(receiver_uid, sender_uid)` right before the `create_notification` call and skips only that call on a hit; `clear_messages_cache(receiver_uid)` still runs unconditionally, and the WS/relay delivery of the message frame is completely unaffected (it is broadcast from `routers/messages.py`, not from `persist_message`). The freshness window (20s) is intentionally wider than the write throttle (5s) so it never lapses between two consecutive throttled writes, and the marker is a soft "recently confirmed as active" signal, not a hard connection state - if it goes stale (tab closed, thread switched away without a fresh trigger) the very next message just resumes notifying normally, which is the safe failure direction.
## Presence is NOT part of the messaging WS
@@ -65,7 +80,9 @@ Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubbl
## Frontend
The messaging frontend is the single self-booting custom element `<dp-chat>` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `<dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true">` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. The send button is disabled only while `dp-upload` is busy, never while a send is in flight. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px.
The messaging frontend is the single self-booting custom element `<dp-chat>` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `<dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true">` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. **The send button is disabled while `dp-upload` is busy AND while a genuine new send is in flight** (the send-lock, see "Send dedupe" above) - a `.failed`-bubble retry never engages it, so retrying an old message never blocks typing/sending a new one. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px.
**Auto-scroll never fights a manual scroll-up.** `_startScrollWatcher`/`_stabilizeScroll` force `scrollTop = scrollHeight` on every thread mutation (new bubble, attachment/image finishing loading) ONLY while `_userAtBottom` is true, re-checked every `requestAnimationFrame` up to `STABILIZE_MAX_FRAMES`. The one thing that used to break this: while a multi-frame stabilize loop was in flight (`_stabilizePending`), the real `scroll` event listener (`_bindAutoScroll`) ignored EVERY scroll event outright, including a genuine user wheel/touch scroll-up that happened to land inside that window - so a rapid burst of incoming messages (each restarting the stabilize loop while attachments were still loading) could silently overwrite a user's attempt to scroll up to read history. The fix: every programmatic scroll (`_scrollThreadToEnd`, `_stabilizeScroll`) records the exact `scrollTop` it just set in `_lastForcedScrollTop`; the `scroll` listener only skips its own echo (`_stabilizePending && scrollTop === _lastForcedScrollTop`) and always processes a scroll that lands anywhere else, so a real user scroll-up is detected and `_userAtBottom` flips to `false` immediately, even mid-stabilization. Never gate scroll-intent detection on `_stabilizePending` alone again.
## Attachments stream live