moloapp/plans/06-realtime.md

# 06 - Realtime

Title: Realtime (WebSocket, SSE, Socket.IO, Streaming, WebDAV, Client Parity)
Scope: plan.md sections 2, 6-7 partial, 10 step 4 partial.

## Purpose and philosophy

Realtime turns MoloApp from a request/response framework into a live
system: bidirectional WebSocket channels, one-way server push over SSE,
multiplexed Socket.IO namespaces with rooms, unbounded streaming
responses, and WebDAV resource management, every one of them usable
through the same middleware, auth, session, logging and metrics path as
plain HTTP handlers.

Philosophy:

- One connection model. A realtime connection is a long-lived request
  context, not a parallel universe. It enters through routing,
  middleware, auth and sessions exactly like any handler, then upgrades.
- Protocol correctness first. Wire formats follow the governing
  standard byte for byte (RFC 6455 framing, WHATWG event-stream
  grammar, Socket.IO v5 packet encoding, RFC 4918 method semantics),
  verified against third-party clients, not only our own.
- Client parity is a release gate. Per plan.md section 2, everything
  served ships a matching client (WebSocket, SSE, Socket.IO, WebDAV),
  sharing config, auth, reconnect and logging behavior with the server
  side. A protocol without its client is not done.
- Backpressure everywhere. Slow consumers must slow producers or be
  shed by rule, never buffer unboundedly until the process dies.
- Layered, not duplicated. Socket.IO sits on Engine.IO, Engine.IO sits
  on the WebSocket/long-polling transports, streaming sits on the
  response hierarchy from 05-request-routing.md. Shared behavior lives
  once in a base class.

## Best practices (from research)

1. WebSocket handshake: validate `Upgrade: websocket` and
   `Connection: Upgrade`, require a `Sec-WebSocket-Key`, answer
   `101 Switching Protocols` with `Sec-WebSocket-Accept` computed as
   base64(SHA-1(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11")).
   (RFC 6455)
2. WebSocket framing: opcodes 0x0 continuation, 0x1 text (must be valid
   UTF-8), 0x2 binary, 0x8 close, 0x9 ping, 0xA pong; control frames
   max 125 bytes payload and never fragmented; server MUST fail
   connections whose client frames are unmasked, and MUST NOT mask
   server-to-client frames. (RFC 6455)
3. Liveness: answer every ping with a pong carrying identical payload,
   run an application heartbeat (ping interval plus pong timeout) and
   close dead peers with a close frame and status code. (RFC 6455)
4. SSE wire format: `Content-Type: text/event-stream`, UTF-8 only,
   fields `event:`, `data:`, `id:`, `retry:`, blank line dispatches;
   default event type is `message`; multiline data joins with LF.
   (WHATWG HTML 9.2)
5. SSE resumability: server assigns monotonically increasing `id:` per
   stream; on reconnect the client sends `Last-Event-ID` and the
   server replays missed events; HTTP 204 tells a client to stop
   reconnecting; initial `retry:` sets reconnect delay. (WHATWG HTML)
6. Engine.IO v4.1: transports polling then WebSocket upgrade
   (WebTransport out of scope for v1); packet types open 0, close 1,
   ping 2, pong 3, message 4, upgrade 5, noop 6; handshake returns
   JSON `{sid, upgrades, pingInterval, pingTimeout, maxPayload}`;
   default path `/engine.io/` with `EIO=4` query; at most one active
   polling GET and one WebSocket per session. (Engine.IO protocol)
7. Socket.IO v5: packet types CONNECT 0, DISCONNECT 1, EVENT 2,
   ACK 3, CONNECT_ERROR 4, BINARY_EVENT 5, BINARY_ACK 6; text
   encoding `<type>[<attachments>-][<nsp>,][<ackId>]<json>`, each
   packet wrapped as one Engine.IO message (wire prefix `4`);
   namespace omitted when `/`; CONNECT handshake carries session id
   or refusal reason; binary payloads extracted as attachments with
   placeholder objects. (Socket.IO protocol)
8. WebDAV: implement PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK,
   UNLOCK over the RFC 4918 data model; multistatus 207 with
   `DAV:` XML bodies; honor `Depth: 0,1,infinity` (infinity capped,
   see Specifications); advertise compliance via the `DAV:` header;
   locks are advisory with tokens, timeouts and refresh. (RFC 4918)
9. Streaming: chunked transfer with explicit flush points, SSE and
   file/upstream passthrough stream with bounded buffers and pause
   signals; compression never buffers a whole stream. Corroborated by
   HTTP semantics practice and the WHATWG line-by-line processing
   model.

## Structure

Directories (one class or concern per file, deep structure):

- `molo/realtime/websocket/`: `server/`
  (`WebSocketView`, `WebSocketResponse`, `WebSocketHandler`,
  `FrameParser`, `FrameSerializer`, `HandshakeValidator`,
  `HeartbeatManager`), `client/` (`WebSocketClient`,
  `ClientHandshake`, `ReconnectPolicy` shared with all clients).
- `molo/realtime/sse/`: `SseResponse`, `EventEncoder`,
  `EventStore` (replay buffer interface), `SseClient`
  (EventSource-compatible: readyState CONNECTING/OPEN/CLOSED,
  Last-Event-ID tracking).
- `molo/realtime/engineio/`: `EngineSession`, `PollingTransport`,
  `WebSocketTransport`, `UpgradeManager`, `HeartbeatScheduler`,
  `PacketCodec` (types 0-6, polling payload framing).
- `molo/realtime/socketio/`: `SocketIoServer`, `Namespace`,
  `RoomManager`, `EventDispatcher`, `AckTracker`, `PacketCodec`
  (types 0-6, binary placeholder extraction), `client/`
  (`SocketIoClient` with identical namespace/room/event API).
- `molo/realtime/streaming/`: `ChunkedBody`, `PassthroughBody`,
  `BackpressureValve`, `FlushPolicy` (used by SSE, streaming
  responses, WebDAV passthrough).
- `molo/realtime/webdav/`: `server/` (`DavView`, `PropfindHandler`,
  `ProppatchHandler`, `MkcolHandler`, `CopyMoveHandler`,
  `LockManager`, `MultistatusBuilder`, `QuotaGuard`), `client/`
  (`WebDavClient` with PROPFIND/GET/PUT/DELETE passthrough,
  response cache, quota accounting).
- `molo/realtime/shared/`: `ConnectionContext` (auth/session/logging
  handle shared by every realtime connection), `SendQueue`
  (bounded per-connection queue), `RateLimiter` hook.

Namespaces: `molo::realtime::websocket`, `::sse`, `::engineio`,
`::socketio`, `::streaming`, `::webdav`. All connection types derive
from `ConnectionContext`; all clients derive from `BaseRealtimeClient`
(auth, reconnect, backoff, logging in one place).

## Rules (hard, numbered)

1. Every realtime endpoint mounts on a router and passes through the
   full middleware, auth and session chain before upgrade. No
   side-channel listeners.
2. Server MUST reject client WebSocket frames without the mask bit by
   failing the connection; server MUST NOT mask its own frames.
3. Text frames MUST contain valid UTF-8; invalid UTF-8 fails the
   connection with close code 1007.
4. Control frames MUST have payload length at most 125 bytes and FIN
   set; fragmented control frames are a protocol error (1002).
5. Max message size, max frame count per message, max connections per
   IP/user/route, send-queue depth and idle timeout are static caps
   from config; exceeding them closes or sheds by rule, never grows.
6. SSE responses MUST send `Content-Type: text/event-stream`,
   `Cache-Control: no-store`, and MUST flush headers immediately so
   intermediaries do not buffer the stream.
7. Socket.IO and Engine.IO packet codecs MUST round-trip the official
   test vectors (packet type tables in Specifications) before any
   feature work builds on them.
8. Client parity: no realtime server feature merges without its
   matching client supporting the same auth, reconnect/backoff and
   logging behavior in the same change.
9. Slow consumers MUST be shed: when a send queue is full, apply the
   route policy (drop oldest, drop newest, or close with code) and
   count it in metrics. Unbounded queues are forbidden.
10. WebDAV `Depth: infinity` MUST be rewritten to the configured max
    depth; recursive operations MUST enforce quotas and per-request
    resource ceilings.
11. Locks MUST have finite timeouts and MUST auto-expire; lock tokens
    MUST be unguessable random values.
12. Streaming and passthrough paths MUST propagate backpressure to the
    producer (pause reads) and MUST support client disconnect cleanup
    within one heartbeat interval.

## Development guidelines

- Build bottom-up: handshake plus framing codecs first with byte-level
  tests against captured third-party traffic (browsers, `websocat`
  equivalents, official Socket.IO client), then views, then rooms and
  namespaces, then clients, then WebDAV last.
- Every codec gets a fuzzer-fed unit harness plus golden vectors from
  the official protocol test suites; every view gets integration tests
  through the real stack per 10-workflow-example.md.
- Reuse, never fork: heartbeat, reconnect with jittered exponential
  backoff, send queues and logging context are implemented once in
  `shared/` and composed into each protocol.
- Coroutine handlers (`co_await`) for message loops; blocking work
  (XML parsing for large PROPFIND, disk-backed event replay) goes to
  the off-loop executor defined in 01-runtime.md.
- Log connection lifecycle (open, upgrade, heartbeat loss, close code
  and reason) at fixed levels with correlation IDs per
  02-observability.md; per-connection debug logging is gated and
  capped.
- Document every wire-visible behavior in the component spec before
  coding it; docs state what the system does today, gaps listed next
  to the code that closes them.

## Specifications

- WebSocket handshake: `GET` with `Upgrade: websocket`,
  `Connection: Upgrade`, `Sec-WebSocket-Key` (16 random bytes,
  base64), optional `Sec-WebSocket-Protocol` subprotocol
  negotiation (server selects at most one, echoes it, else fails);
  response `101` with `Sec-WebSocket-Accept`. Origin checked
  against CORS policy from 03-security.md.
- WebSocket opcodes: 0x0 cont, 0x1 text, 0x2 binary, 0x8 close
  (0-125 byte body: 2-byte code plus UTF-8 reason), 0x9 ping,
  0xA pong. Close codes used: 1000 normal, 1001 going away,
  1002 protocol error, 1003 unsupported data, 1007 bad UTF-8,
  1008 policy violation, 1009 message too big, 1011 internal
  error. Fragmented messages reassembled up to max message size.
- Caps (defaults, all overridable): max message 1 MiB, max frame
  payload 256 KiB, max frames per message 64, send queue 256
  messages, connections per IP 128, per route 100k, idle timeout
  60 s, ping interval 25 s, pong timeout 20 s (Engine.IO defaults
  aligned), close handshake timeout 5 s.
- SSE event grammar: lines `field: value`, fields `event`, `data`,
  `id`, `retry`, `:` comment (keep-alive); blank line dispatches;
  `retry:` is milliseconds, clamped to 1 s..300 s; server sends
  comment keep-alive every 15 s; `Last-Event-ID` replay window
  holds last 1000 events or 5 minutes per stream, whichever is
  smaller.
- `SseClient` states CONNECTING 0, OPEN 1, CLOSED 2; fails the
  connection on non-200 status or wrong content type; honors
  301/307 redirect and 204 stop; sends stored last event id on
  reconnect.
- Engine.IO: path `/engine.io/`, query `EIO=4`,
  `transport=polling|websocket`, `sid` after handshake; open
  packet JSON keys exactly `sid, upgrades, pingInterval,
  pingTimeout, maxPayload`; polling GET long-holds up to 25 s;
  POST bodies carry length-prefixed packet payloads per the
  polling encoding; upgrade probe `ping/probe` then `upgrade`.
- Socket.IO: namespaces start with `/`, no commas; rooms are
  server-side sets with join/leave, broadcast excludes sender by
  default with explicit include flag; EVENT payload is a JSON
  array whose first element is the event name; ack id is a
  monotonically increasing integer per connection with 30 s
  default ack timeout; binary attachments use
  `{"_placeholder":true,"num":N}` markers, max 16 attachments
  and 8 MiB total per packet by default.
- Rooms and broadcast fan-out MUST complete in O(subscribers)
  with per-recipient queue caps; broadcast to 0 subscribers is a
  no-op counted in metrics, not an error.
- Streaming: chunked bodies flush at least every 64 KiB or 100 ms
  during activity; passthrough (proxy/file/WebDAV GET) uses a
  256 KiB ring buffer with producer pause; range and conditional
  requests supported on file passthrough.
- WebDAV server: resource tree rooted at a configured collection;
  methods GET, PUT, DELETE plus PROPFIND (allprop, propname,
  prop selections), PROPPATCH (set/remove with 207 per-property
  status), MKCOL, COPY/MOVE (with `Overwrite`, `Destination`,
  `Depth` headers), LOCK (exclusive and shared, `Timeout`,
  `Depth: 0`), UNLOCK (matching `Lock-Token`); `DAV: 1` header
  on OPTIONS and responses; XML bodies use `DAV:` namespace,
  max XML body 1 MiB, max depth 32.
- WebDAV client: streams GET directly into a caller response body
  (no full buffering), conditional GET with ETag cache (max 128
  entries, 64 MiB), quota accounting hooks per transfer.
- Replication hooks: connection open/close, room join/leave and
  broadcast envelopes are emitted as typed events for
  07-supervisor-replication.md; v1 guarantees at-most-once
  delivery across instances, at-least-once only inside one
  process via the send queue.

## Execution steps (in build order)

1. `shared/`: ConnectionContext, SendQueue, BaseRealtimeClient
   (auth, jittered backoff, logging), BackpressureValve.
2. WebSocket: HandshakeValidator, FrameParser/Serializer,
   WebSocketView/Handler/Response, HeartbeatManager.
3. `WebSocketClient` plus conformance tests against a third-party
   peer.
4. SSE: EventEncoder, SseResponse, EventStore replay,
   `SseClient`.
5. Streaming: ChunkedBody, PassthroughBody, FlushPolicy; wire
   through SSE and file download.
6. Engine.IO: PacketCodec, EngineSession, PollingTransport,
   WebSocketTransport, UpgradeManager, HeartbeatScheduler.
7. Socket.IO: PacketCodec (text plus binary), Namespace,
   RoomManager, EventDispatcher, AckTracker, SocketIoServer.
8. `SocketIoClient` with namespace/room/event parity and
   long-polling fallback.
9. WebDAV server: MultistatusBuilder, PROPFIND/PROPPATCH/MKCOL,
   COPY/MOVE, LockManager, QuotaGuard, DavView.
10. `WebDavClient` with streaming GET passthrough, cache, quota.
11. Cross-cutting: metrics per connection/room/route, trace spans
    for upgrade and broadcast, living-example realtime pages.

## Validation (how to verify each step)

1. Shared: unit tests prove queue shed policies and backoff
   timings; no unbounded growth under a blocked consumer test.
2. WebSocket: golden handshake vectors (key/accept pairs);
   byte-level frame round-trips including fragmentation,
   masking rejection, UTF-8 rejection, control-frame limits;
   interop test against an independent client implementation.
3. Client: connects to a third-party echo server, survives
   restart with backoff, exchanges text plus binary.
4. SSE: byte-exact event encoding tests; browser/EventSource
   interop including reconnect with Last-Event-ID replay and
   204 stop; wrong content-type fails cleanly.
5. Streaming: 1 GiB passthrough with flat memory profile;
   disconnect mid-stream releases all resources within one
   heartbeat interval; flush latency within policy.
6. Engine.IO: open/handshake JSON shape, ping/pong heartbeat
   timing, polling-to-WebSocket upgrade sequence, duplicate
   transport rejection, unknown sid returns 400.
7. Socket.IO: official packet vectors for all 7 types including
   binary placeholders; namespace connect/refuse; rooms
   join/leave/broadcast; ack round-trip and timeout.
8. Client parity: scripted session where our client drives the
   official server behaviors (namespaces, rooms, acks, binary)
   and vice versa; long-polling-only mode passes with
   WebSocket disabled.
9. WebDAV: PROPFIND/MKCOL/COPY/MOVE/LOCK/UNLOCK sequences
   against litmus-style assertions; Depth infinity capped;
   quota exceeded returns 507; lock expiry releases.
10. Client: PROPFIND listings, streaming GET into a response
    with bounded memory, ETag cache hits, quota accounting.
11. Observability: connection counts, message rates, shed
    counts, room sizes and broadcast fan-out exported per
    02-observability.md; full integration suite green through
    the real stack.

## References

- Depends on 01-runtime.md: event loop, lifecycle, timeouts,
  HTTP/1.1 framing and upgrade path, compression interplay.
- Depends on 02-observability.md: log format, correlation IDs,
  metrics and tracing primitives, zero-cost gating.
- Depends on 03-security.md: origin/CORS checks, rate limits,
  body caps, auth/session resolution before upgrade.
- Depends on 05-request-routing.md: routers, view base classes,
  response hierarchy, middleware ordering, DI access to shared
  services.
- Feeds 04-data.md: pubsub bridge for cross-instance broadcast
  and event replay storage interfaces.
- Feeds 07-supervisor-replication.md: connection/room events,
  broadcast envelopes, delivery guarantees, multi-instance
  session sharing.
- Feeds 08-clients-openapi.md: shared client transport behavior
  (timeouts, retries, TLS, test doubles) reused by realtime
  clients.
- Feeds 10-workflow-example.md: integration test templates for
  realtime routes and living-example chat, feed and DAV pages.

## Open questions

1. WebTransport (Engine.IO transport) needs HTTP/3, which
   01-runtime.md lists as a future path. Stay out of v1: yes,
   unless runtime lands H3 early.
2. Should Socket.IO v2/v3 protocol compat be offered? Proposal:
   no, v5 plus Engine.IO v4 only; document the refusal clearly.
3. Cross-instance room broadcast transport: reuse pubsub from
   04-data.md or a dedicated replication channel in
   07-supervisor-replication.md? Needs a joint decision.
4. SSE replay durability: in-memory ring per process for v1, or
   persistent store from the start? Proposal: in-memory with a
   documented interface for a persistent EventStore later.
5. WebDAV ACL (RFC 3744) and versioning (DeltaV): explicitly out
   of v1; confirm no v1 consumer needs them.

## Sources

1. IETF RFC 6455, The WebSocket Protocol (Fette, Melnikov,
   Dec 2011, Standards Track). Handshake, framing, opcodes,
   masking, close codes. URL:
   https://www.rfc-editor.org/rfc/rfc6455 (retrieved
   2026-10-23).
2. WHATWG HTML Living Standard, section 9.2 Server-sent events
   (EventSource interface, processing model, event-stream
   parsing and interpretation). Wire grammar, reconnect,
   Last-Event-ID, 204 stop. URL:
   https://html.spec.whatwg.org/multipage/server-sent-events.html
   (retrieved 2026-10-23).
3. Socket.IO Protocol v5, official docs (packet types 0-6,
   encoding format, binary placeholders, sample session).
   URL: https://socket.io/docs/v4/socket-io-protocol/
   (retrieved 2026-10-23).
4. Engine.IO Protocol v4.1, official docs (transports,
   handshake, heartbeat, upgrade, packet encoding). URL:
   https://socket.io/docs/v4/engine-io-protocol/ (retrieved
   2026-10-23).
5. IETF RFC 4918, HTTP Extensions for WebDAV (Dusseault, ed.,
   Jun 2007, Standards Track, obsoletes RFC 2518). Methods
   PROPFIND/PROPPATCH/MKCOL/COPY/MOVE/LOCK/UNLOCK, 207
   Multistatus, Depth/DAV headers. URL:
   https://www.rfc-editor.org/rfc/rfc4918 (retrieved
   2026-10-23).