feat(clients): live: Gemini Live API over WebSocket, RFC 6455 client, session, relay, discovery check
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,165 @@
|
||||
# Google Live API - Specification
|
||||
|
||||
Owned directory: `src/google/live/`. Namespace `molo::google::live`. Depends
|
||||
on `molo::google` (errors, `CloudContext`, `valid_segment`), `molo::clients`
|
||||
(JSON DOM, `ITlsSession`) and the Base64 codec. Does not use `src/realtime`:
|
||||
its WebSocket client is a byte-pump over a caller supplied transport with no
|
||||
socket, TLS or timeout handling, and `libgoogle` does not link the realtime
|
||||
part. An RFC 6455 client lives here instead.
|
||||
|
||||
Scope: the Gemini Live API (bidirectional text, audio, video) over a
|
||||
WebSocket, on Vertex AI and on the Gemini API (API key form). Typed setup,
|
||||
typed client messages, typed server events, bounded queues, clean close, and
|
||||
an HTTP relay for hosts without WebSocket.
|
||||
|
||||
## 1. Module map (one class per concern)
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings (`LiveCaps`) |
|
||||
| `live_spec.h` | The single table of every wire field name used (`kLiveSpec`), checked by `tools/check_discovery.py` |
|
||||
| `ws_error.h` | `WsError` (kind, HTTP status of a refused upgrade) |
|
||||
| `ws_url.h/.cpp` | `WsUrl`: ws/wss parsing, `display()` without the query |
|
||||
| `ws_frame.h/.cpp` | `encode_frame`, `FrameDecoder` (strict RFC 6455), `valid_utf8` |
|
||||
| `ws_handshake.h/.cpp` | Key, SHA-1 accept, request building, 101 validation |
|
||||
| `ws_stream.h/.cpp` | `IByteStream`, `TcpStream`, `TlsStream` (over `ITlsSession`), `open_stream` |
|
||||
| `ws_client.h/.cpp` | `WsClient`: masking, fragmentation, ping/pong, close, caps |
|
||||
| `live_endpoint.h/.cpp` | `LiveEndpoint`: URL, auth headers, model naming per mode |
|
||||
| `live_config.h/.cpp` | `LiveConfig`: typed setup message |
|
||||
| `live_requests.h/.cpp` | `LiveRequests`: text, turn, audio, video, audioStreamEnd, toolResponse |
|
||||
| `live_events.h/.cpp` | `LiveEvent`, `LiveParser`: typed server events |
|
||||
| `live_session.h/.cpp` | `LiveSession`: connect, send, bounded event queue, close |
|
||||
| `live_relay.h/.cpp` | `LiveRelay`: sessions by id, events by cursor, `event_to_json` |
|
||||
| `tools/check_discovery.py` | Discovery and endpoint check (python3 stdlib) |
|
||||
|
||||
## 2. Static caps (see caps.h)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Frame payload (inbound) | 4 MiB, checked from the header before buffering |
|
||||
| Message (inbound, reassembled) | 8 MiB |
|
||||
| Message (outbound) | 2 MiB |
|
||||
| Handshake response | 16 KiB |
|
||||
| Control frame payload | 125 (RFC), close reason 123 |
|
||||
| Connect / handshake / setup timeout | 10 s / 10 s / 15 s |
|
||||
| Socket I/O timeout, reader poll | 5 s, 200 ms |
|
||||
| Close grace | 1.5 s |
|
||||
| Session lifetime | 1800 s, then closed with a `Closed` event |
|
||||
| Queued events / bytes | 1024 / 8 MiB, oldest dropped and counted (`stats().dropped`) |
|
||||
| Text / audio chunk / video chunk | 32000 B / 256 KiB / 1 MiB |
|
||||
| System instruction / function declarations / tool responses | 8000 chars / 16 / 16 |
|
||||
| Relay: sessions / log / page | 8 / 512 events / 100 events |
|
||||
|
||||
## 3. Contracts
|
||||
|
||||
- Auth forms (`LiveEndpoint`):
|
||||
- `vertex(ctx, token)`: `wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent` (`global`: `aiplatform.googleapis.com`), header `Authorization: Bearer <token>`, model `projects/{p}/locations/{l}/publishers/google/models/{m}`.
|
||||
- `vertex_express(ctx, key)`: same URL, header `x-goog-api-key`. The key is never in the URL.
|
||||
- `api_key(key)`: `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=<key>`, model `models/{m}`. The documented form puts the key in the query, the one place the foundation rule "key never in a URL" cannot hold; `WsUrl::display()` and every error omit the query.
|
||||
- `url_override` replaces the URL (loopback tests, proxies).
|
||||
- No credential: `NotConfiguredError` (503 NOT_CONFIGURED) before any socket. Bad input (model id, voice, modality, caps, mime, header characters): `GoogleError` 400 INVALID_ARGUMENT before any socket or send.
|
||||
- Upstream failures map to `GoogleError` with service `live`: refused upgrade 400/401/403/404/429 keep their status, other upgrade failures and I/O are 502 UNAVAILABLE, timeouts 504 DEADLINE_EXCEEDED, a close frame before `setupComplete` is 409 FAILED_PRECONDITION carrying Google's close code and reason (e.g. 1008 "Publisher model ... was not found").
|
||||
- Framing: client frames always masked; server frames must be unmasked; RSV bits, unknown opcodes, fragmented or oversize control frames, non-minimal lengths, continuation without start, new message inside a fragmented one, invalid UTF-8 text all close with 1002 / 1007 and throw. Over-cap frame or message closes with 1009. Pings are answered with the same payload. A close frame is echoed; a dropped connection without close frame reports code 1006.
|
||||
- Server JSON arrives in text and in binary frames; both are parsed.
|
||||
- One server message can yield several events. Order: setupComplete; model text and inline data in part order; input/output transcription; interrupted; generationComplete; turnComplete last. Inline `audio/*` is `Audio` (decoded bytes), other inline data `Media`. Undecodable messages become one `Error` event, unknown ones `Unknown`, never silence.
|
||||
- Sending is thread safe; one reader thread receives. `close()` sends a close frame, waits up to the grace, joins.
|
||||
- Relay: the session resumption handle is not sent to browsers. Ids are `s` plus 12 hex chars.
|
||||
- TLS: `TlsStream` uses `ITlsSession` (wolfSSL) on a blocking fd with socket read and write timeouts. The backend reports timeout and dropped connection alike (zero read or a read error), so the stream tells them apart by elapsed time against the read timeout. It ignores SIGPIPE process wide (the backend writes without `MSG_NOSIGNAL`).
|
||||
|
||||
## 4. Example routes (`example/app/agent/LiveDemo.cpp`)
|
||||
|
||||
| Route | Behaviour |
|
||||
|---|---|
|
||||
| `GET /agent/live` | mobile-first page, script `example/assets/agent_live.js`; matrix row `agent-live` |
|
||||
| `GET /agent/live/status` | configured, auth mode, default model, caps |
|
||||
| `POST /agent/live/sessions` | create, 201 `{id}`; 503 `google_not_configured` without credential; 422 bad input; 429 at the session cap |
|
||||
| `POST /agent/live/sessions/{id}/send` | `{kind: text\|audio\|video\|audio_end, ...}`; 422, 404 unknown id |
|
||||
| `GET /agent/live/sessions/{id}/events?cursor=N&max=M` | events after the cursor, `next_cursor`, `open`, `dropped` |
|
||||
| `POST /agent/live/sessions/{id}/close` | 200, then 404 |
|
||||
|
||||
Credential resolved from the environment: `GOOGLE_ACCESS_TOKEN` plus
|
||||
`GOOGLE_CLOUD_PROJECT` (Vertex OAuth), else `GOOGLE_API_KEY` starting `AQ.`
|
||||
plus project (Vertex express), else any `GOOGLE_API_KEY` (Gemini API form).
|
||||
`GOOGLE_LIVE_URL` overrides the WebSocket URL.
|
||||
|
||||
## 5. Verified against discovery
|
||||
|
||||
`python3 -I src/google/live/tools/check_discovery.py`, run 2026-10-24: 69
|
||||
spec rows checked (2 Discovery documents, 1 documentation page), 0
|
||||
failures; 3 unauthenticated upgrade probes answered `101 Switching
|
||||
Protocols` with the correct `Sec-WebSocket-Accept`. A bogus service name
|
||||
answers 400, so the probe discriminates.
|
||||
|
||||
- `generativelanguage/v1beta` Discovery: the setup message
|
||||
`BidiGenerateContentSetup` and its nested types (generation config, speech
|
||||
config, content, tool, function declaration, session resumption, context
|
||||
window compression, blob, function call/response, usage metadata).
|
||||
- `aiplatform/v1` Discovery: only value types (`Transcription {text,
|
||||
finished}`, `Blob`); it has no Bidi setup or message schemas.
|
||||
- Documentation only (no Discovery document describes them): the message
|
||||
envelopes (`setup`, `clientContent`, `realtimeInput`, `toolResponse`;
|
||||
`setupComplete`, `serverContent`, `toolCall`, `toolCallCancellation`,
|
||||
`goAway`, `sessionResumptionUpdate`, `usageMetadata`) and their fields.
|
||||
Checked by name against https://ai.google.dev/api/live.
|
||||
- Endpoints: v1 and v1beta1 `LlmBidiService/BidiGenerateContent` on
|
||||
`us-central1-aiplatform.googleapis.com` and the Gemini API
|
||||
`GenerativeService.BidiGenerateContent` exist (101 on upgrade).
|
||||
|
||||
## 6. Verified live
|
||||
|
||||
All 2026-10-24, project from `GOOGLE_CLOUD_PROJECT`, model
|
||||
`gemini-live-2.5-flash-native-audio`, region `us-central1`, over wss with the
|
||||
wolfSSL session. Each run: upgrade, setup, `setupComplete`, one text turn
|
||||
("Say hello in one short sentence."), receive until `turnComplete`, close.
|
||||
|
||||
| Form | Result |
|
||||
|---|---|
|
||||
| Vertex, express key, header `x-goog-api-key` | works: `setupComplete`, output transcription "Hello there!", 4 audio chunks (`audio/pcm`, 11-15 KB each), `generationComplete`, `turnComplete`, clean close |
|
||||
| Vertex, OAuth bearer (service account token) | works, same event sequence |
|
||||
| Vertex, express key as `Authorization: Bearer` | refused: close 1008 "Request had invalid authentication credentials" |
|
||||
| Gemini API form with the express key (`?key=`) | upgrade ok, close 1008 "Gemini API ... has not been used in project ... or it is disabled" (the key's project has the API disabled; the form itself is accepted) |
|
||||
| `responseModalities: ["TEXT"]` on the native audio model | close 1007 "Text output is not supported for native audio output model." |
|
||||
| region `global`, or model `gemini-2.0-flash-live-001` | close 1008 "Publisher model ... was not found" |
|
||||
|
||||
Observed: server audio mime type is plain `audio/pcm` (no `rate=`); the
|
||||
docs state 24 kHz output and 16 kHz input (`audio/pcm;rate=16000`). The
|
||||
server closes TCP after its close frame without TLS close_notify.
|
||||
Opt-in tests: `MOLO_LIVE_NET=1` with `GOOGLE_CLOUD_PROJECT` and
|
||||
`GOOGLE_API_KEY` or `GOOGLE_ACCESS_TOKEN` (`live_net.*`, loud SKIP otherwise).
|
||||
|
||||
## 7. Verified locally (deterministic, no Google)
|
||||
|
||||
Framing, handshake failures, fragmentation with interleaved ping, protocol
|
||||
violation closes, caps, abrupt disconnect, the TLS seam (a passthrough
|
||||
`ITlsSession` with wolfSSL's zero-read contract) run against a loopback
|
||||
RFC 6455 server written in `tests/src/google/live/loopback.h`. Session and
|
||||
relay tests use the same server to carry hand-written JSON in the documented
|
||||
shapes; that proves the module's handling of those shapes, not Google's
|
||||
behavior. Request bodies are checked to use only names in `kLiveSpec`.
|
||||
|
||||
## Sources
|
||||
|
||||
- https://ai.google.dev/api/live (WebSocket API reference: messages, fields)
|
||||
- https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket (URL with `?key=`, setup, `audio/pcm;rate=16000`)
|
||||
- https://generativelanguage.googleapis.com/$discovery/rest?version=v1beta
|
||||
- https://aiplatform.googleapis.com/$discovery/rest?version=v1 and `version=v1beta1`
|
||||
- https://gemini-ex.hexdocs.pm/live_api.md (community client: Vertex path `wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent`)
|
||||
- https://docs.litellm.ai/docs/providers/vertex_realtime (Vertex hosts per region, OAuth bearer)
|
||||
- RFC 6455
|
||||
|
||||
## Unverified
|
||||
|
||||
- The Vertex Live reference page (docs.cloud.google.com ... reference/models/live-api) could not be read in full (very large page, navigation only in the fetched part), so Vertex-specific message differences from the Gemini API reference are unread. The Vertex endpoint path and both auth forms are verified live instead.
|
||||
- `?key=` on the Vertex endpoint: only seen to be accepted at upgrade, no turn run.
|
||||
- Not exercised live: tool calls and tool responses, `toolCallCancellation`, `goAway`, session resumption (handle exchange, resume), context window compression, video input, `audioStreamEnd`, `clientContent` turns, input transcription, activity/VAD settings, interruption. They follow the documented shapes and are covered by local tests only.
|
||||
- Express key requirement of a project in the Vertex model name: the run used the project; not tried without.
|
||||
- `timeLeft` is parsed as a protobuf duration string ("30s"); the format is from the Duration type, not from a captured message.
|
||||
- Session limits (audio-only 15 min, audio+video 2 min) are from a community guide; the module enforces its own 1800 s cap.
|
||||
|
||||
## Gaps
|
||||
|
||||
- Outbound messages are single frames (no outbound fragmentation).
|
||||
- Proactivity, `historyConfig`, labels, VAD tuning, media resolution, affective dialog are not in `LiveConfig` (setup can be extended in `live_config.cpp`).
|
||||
- No automatic reconnect or resume on `goAway`; the handle is available from `LiveSession::resumption_handle()`.
|
||||
- Token refresh is the caller's job (foundation gap).
|
||||
- The relay polls; no push to the browser (the example server has no WebSocket).
|
||||
Reference in New Issue
Block a user