18 KiB
Google Live API - Specification
Owned directory: src/google/live/. Namespace molo::google::live. Depends
on molo::google (errors, CloudContext, valid_segment), molo::clients
(JSON DOM, ITlsSession) and the Base64 codec. Does not use src/realtime:
its WebSocket client is a byte-pump over a caller supplied transport with no
socket, TLS or timeout handling, and libgoogle does not link the realtime
part. An RFC 6455 client lives here instead.
Scope: the Gemini Live API (bidirectional text, audio, video) over a WebSocket, on Vertex AI and on the Gemini API (API key form). Typed setup, typed client messages, typed server events, bounded queues, clean close, and an HTTP relay for hosts without WebSocket.
1. Module map (one class per concern)
| File | Concern |
|---|---|
caps.h |
Static ceilings (LiveCaps) |
live_spec.h |
The single table of every wire field name used (kLiveSpec), checked by tools/check_discovery.py |
ws_error.h |
WsError (kind, HTTP status of a refused upgrade) |
ws_url.h/.cpp |
WsUrl: ws/wss parsing, display() without the query |
ws_frame.h/.cpp |
encode_frame, FrameDecoder (strict RFC 6455), valid_utf8 |
ws_handshake.h/.cpp |
Key, SHA-1 accept, request building, 101 validation |
ws_stream.h/.cpp |
IByteStream, TcpStream, TlsStream (over ITlsSession), open_stream |
ws_client.h/.cpp |
WsClient: masking, fragmentation, ping/pong, close, caps |
live_endpoint.h/.cpp |
LiveEndpoint: URL, auth headers, model naming per mode |
live_config.h/.cpp |
LiveConfig: typed setup message (modalities, voice, tools, transcription, resumption, compression, VAD / activity handling, affective dialog, media resolution) |
live_requests.h/.cpp |
LiveRequests: text, turn, audio, video, audioStreamEnd, toolResponse |
live_events.h/.cpp |
LiveEvent, LiveParser: typed server events |
live_session.h/.cpp |
LiveSession: connect, resume (new connection with the latest handle), send, bounded event queue, close |
live_relay.h/.cpp |
LiveRelay: sessions by id, events by cursor, event_to_json |
tools/check_discovery.py |
Discovery and endpoint check (python3 stdlib) |
2. Static caps (see caps.h)
| Knob | Value |
|---|---|
| Frame payload (inbound) | 4 MiB, checked from the header before buffering |
| Message (inbound, reassembled) | 8 MiB |
| Message (outbound) | 2 MiB |
| Handshake response | 16 KiB |
| Control frame payload | 125 (RFC), close reason 123 |
| Connect / handshake / setup timeout | 10 s / 10 s / 15 s |
| Socket I/O timeout, reader poll | 5 s, 200 ms |
| Close grace | 1.5 s |
| Session lifetime | 1800 s, then closed with a Closed event |
resume() calls per session |
5 |
| Queued events / bytes | 1024 / 8 MiB, oldest dropped and counted (stats().dropped) |
| Text / audio chunk / video chunk | 32000 B / 256 KiB / 1 MiB |
| System instruction / function declarations / tool responses | 8000 chars / 16 / 16 |
| Relay: sessions / log / page | 8 / 512 events / 100 events |
3. Contracts
- Auth forms (
LiveEndpoint):vertex(ctx, token):wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent(global:aiplatform.googleapis.com), headerAuthorization: Bearer <token>, modelprojects/{p}/locations/{l}/publishers/google/models/{m}.vertex_express(ctx, key): same URL, headerx-goog-api-key. The key is never in the URL.api_key(key):wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=<key>, modelmodels/{m}. The documented form puts the key in the query, the one place the foundation rule "key never in a URL" cannot hold;WsUrl::display()and every error omit the query.url_overridereplaces the URL (loopback tests, proxies).
- No credential:
NotConfiguredError(503 NOT_CONFIGURED) before any socket. Bad input (model id, voice, modality, caps, mime, header characters):GoogleError400 INVALID_ARGUMENT before any socket or send. - Upstream failures map to
GoogleErrorwith servicelive: refused upgrade 400/401/403/404/429 keep their status, other upgrade failures and I/O are 502 UNAVAILABLE, timeouts 504 DEADLINE_EXCEEDED, a close frame beforesetupCompleteis 409 FAILED_PRECONDITION carrying Google's close code and reason (e.g. 1008 "Publisher model ... was not found"). - Framing: client frames always masked; server frames must be unmasked; RSV bits, unknown opcodes, fragmented or oversize control frames, non-minimal lengths, continuation without start, new message inside a fragmented one, invalid UTF-8 text all close with 1002 / 1007 and throw. Over-cap frame or message closes with 1009. Pings are answered with the same payload. A close frame is echoed; a dropped connection without close frame reports code 1006.
- Server JSON arrives in text and in binary frames; both are parsed.
- One server message can yield several events. Order: setupComplete; model text and inline data in part order; input/output transcription; interrupted; generationComplete; turnComplete last. Inline
audio/*isAudio(decoded bytes), other inline dataMedia. Undecodable messages become oneErrorevent, unknown onesUnknown, never silence. - Sending is thread safe; one reader thread receives.
close()sends a close frame, waits up to the grace, joins. - Resumption: with
config.session_resumptionthe server sendssessionResumptionUpdate(newHandle,resumable); the session keeps the latest resumable handle.LiveSession::resume()stops the current connection, opens a new one with that handle insetup.sessionResumption.handle, waits forsetupCompleteand restarts the reader; queued events and sequence numbers carry on, and the old connection'sClosedevent stays in the queue. It is explicit (call it on aGoAwayevent or after aClosedevent), never automatic, and capped at 5 per session. Without the option or without a handle it throws before any socket. - Setup options beyond text and audio:
activity_detection(automaticActivityDetection: disabled, start / end sensitivity, prefix padding 0..5000 ms, silence duration 0..10000 ms),activity_handling,turn_coverage,affective_dialog,media_resolution,compression_trigger_tokens/compression_target_tokens(int64, sent as JSON strings). Enum values are checked before any socket. Not inLiveConfig:proactivity(an emptyProactivityConfigin Discovery, no fields to declare),historyConfig,labels. - Relay: the session resumption handle is not sent to browsers. Ids are
splus 12 hex chars. - TLS:
TlsStreamusesITlsSession(wolfSSL) on a blocking fd with socket read and write timeouts. The wolfSSL session (src/clients/transports/tls_wolfssl.cpp) now tells the cases apart: a socket read or write timeout throwsTlsTimeoutError(aTlsError), a closed connection (TLS close_notify, or a TCP drop without one) returns 0 withlast_read_status() == TlsReadStatus::kClosed. Measured against a local TLS 1.3 server with a 500 msSO_RCVTIMEO: silence iswolfSSL_read-1 withWOLFSSL_ERROR_WANT_READ, a drop without close_notify is 0 thenWOLFSSL_ERROR_SYSCALL, a close_notify is 0 thenWOLFSSL_ERROR_ZERO_RETURN(the old code compared these against theWOLFSSL_CBIO_ERR_*constants, which never matched). A session that does not say (kUnknown, e.g. a test passthrough) falls back to elapsed time against the read timeout. Writes go through a send callback withMSG_NOSIGNAL, so a vanished peer cannot raise SIGPIPE;TlsStreamno longer changes the process signal disposition. AnITlsSessionsupplied throughLiveOptions::tlsmust not raise SIGPIPE either.
4. Example routes (example/app/agent/LiveDemo.cpp)
| Route | Behaviour |
|---|---|
GET /agent/live |
mobile-first page, script example/assets/agent_live.js; matrix row agent-live |
GET /agent/live/status |
configured, auth mode, default model, caps |
POST /agent/live/sessions |
create, 201 {id}; 503 google_not_configured without credential; 422 bad input; 429 at the session cap |
POST /agent/live/sessions/{id}/send |
{kind: text|audio|video|audio_end, ...}; 422, 404 unknown id |
GET /agent/live/sessions/{id}/events?cursor=N&max=M |
events after the cursor, next_cursor, open, dropped |
POST /agent/live/sessions/{id}/close |
200, then 404 |
Credential resolved from the environment: GOOGLE_ACCESS_TOKEN plus
GOOGLE_CLOUD_PROJECT (Vertex OAuth), else GOOGLE_API_KEY starting AQ.
plus project (Vertex express), else any GOOGLE_API_KEY (Gemini API form).
GOOGLE_LIVE_URL overrides the WebSocket URL.
5. Verified against discovery
python3 -I src/google/live/tools/check_discovery.py, run 2026-10-24: 81
spec rows checked (2 Discovery documents, 1 documentation page), 0
failures; 3 unauthenticated upgrade probes answered 101 Switching Protocols with the correct Sec-WebSocket-Accept. A bogus service name
answers 400, so the probe discriminates.
generativelanguage/v1betaDiscovery: the setup messageBidiGenerateContentSetupand its nested types (generation config, speech config, content, tool, function declaration, session resumption, context window compression, blob, function call/response, usage metadata).aiplatform/v1Discovery: only value types (Transcription {text, finished},Blob); it has no Bidi setup or message schemas.- Documentation only (no Discovery document describes them): the message
envelopes (
setup,clientContent,realtimeInput,toolResponse;setupComplete,serverContent,toolCall,toolCallCancellation,goAway,sessionResumptionUpdate,usageMetadata) and their fields. Checked by name against https://ai.google.dev/api/live. - Endpoints: v1 and v1beta1
LlmBidiService/BidiGenerateContentonus-central1-aiplatform.googleapis.comand the Gemini APIGenerativeService.BidiGenerateContentexist (101 on upgrade).
6. Verified live
All 2026-10-24, project from GOOGLE_CLOUD_PROJECT, model
gemini-live-2.5-flash-native-audio, region us-central1, over wss with the
wolfSSL session. Each run: upgrade, setup, setupComplete, one text turn
("Say hello in one short sentence."), receive until turnComplete, close.
| Form | Result |
|---|---|
Vertex, express key, header x-goog-api-key |
works: setupComplete, output transcription "Hello there!", 4 audio chunks (audio/pcm, 11-15 KB each), generationComplete, turnComplete, clean close |
| Vertex, OAuth bearer (service account token) | works, same event sequence |
Vertex, express key as Authorization: Bearer |
refused: close 1008 "Request had invalid authentication credentials" |
Gemini API form with the express key (?key=) |
upgrade ok, close 1008 "Gemini API ... has not been used in project ... or it is disabled" (the key's project has the API disabled; the form itself is accepted) |
responseModalities: ["TEXT"] on the native audio model |
close 1007 "Text output is not supported for native audio output model." |
region global, or model gemini-2.0-flash-live-001 |
close 1008 "Publisher model ... was not found" |
Observed: server audio mime type is plain audio/pcm (no rate=); the
docs state 24 kHz output and 16 kHz input (audio/pcm;rate=16000). The
server closes TCP after its close frame without TLS close_notify.
Opt-in tests: MOLO_LIVE_NET=1 with GOOGLE_CLOUD_PROJECT and
GOOGLE_API_KEY or GOOGLE_ACCESS_TOKEN (live_net.*, loud SKIP otherwise).
Second round, 2026-10-24, same model and region, every test once with the
express key (header) and once with the OAuth bearer, a few tiny messages each
(tests/src/google/live/test_live_net.cpp):
| Feature | Result |
|---|---|
| Tool call round trip | setup.tools.functionDeclarations with get_temperature(city); the model sent toolCall.functionCalls[] with id, name and args {"city":"Utrecht"}; the turn that carried the call ends with generationComplete / turnComplete (empty transcription), so a client must not stop at that first turnComplete; after toolResponse.functionResponses[{id,name,response}] the model answered in a second turn ("The temperature in Utrecht is currently 17 degrees Celsius."), both auth forms |
clientContent turn |
turns[{role:user,parts[{text}]}] + turnComplete:true starts a model turn, both auth forms (it carried the tool question above and the resumption turns) |
| Output transcription | outputTranscription events with text, both auth forms |
| Input transcription | audio in (the model's own speech, resampled 24 kHz to 16 kHz, audio/pcm;rate=16000 chunks of 250 ms): inputTranscription events arrived, both auth forms. Needed about 1 s of trailing silence: audioStreamEnd alone, without silence, produced no event at all within 30 s (it is accepted without error); silence followed by audioStreamEnd worked |
| Session resumption | sessionResumption: {} in setup: sessionResumptionUpdate {newHandle, resumable:true} (36 characters) arrived around the turn end. Reconnecting with sessionResumption.handle (new object and LiveSession::resume() on the same object) kept the context: asked for "the secret word" (pineapple, mango) the model answered it, both auth forms |
| Interruption | new realtimeInput.text while the model spoke: an interrupted event, both auth forms |
| Video input | realtimeInput.video {data, mimeType:"image/jpeg"} frames (4 frames, 700 ms apart) of a solid red 256x256 JPEG, then a text question: "Red." on both auth forms. A 64x64 frame was accepted without error but the model answered that it could not see an image; the first turn after frames is sometimes empty, so the test asks once more |
goAway |
idle session with resumption on, express key: goAway after 514 s with timeLeft 57 s (parsed from the protobuf duration string), Closed code 1000 at 571 s. The server's connection lifetime is about 9.5 minutes here |
| Setup options | automaticActivityDetection (sensitivities LOW, padding 100 ms, silence 700 ms), activityHandling, mediaResolution, contextWindowCompression (triggerTokens, slidingWindow.targetTokens as strings): accepted, a turn completed, both auth forms |
?key= on the Vertex endpoint |
observed with a one-off websocket-client probe (not in the tree): upgrade, setup with the project model name and setupComplete, no turn |
Express key, model name without a project (publishers/google/models/{m}) |
observed with the same probe: setupComplete |
7. Verified locally (deterministic, no Google)
Framing, handshake failures, fragmentation with interleaved ping, protocol
violation closes, caps, abrupt disconnect, the TLS seam (a passthrough
ITlsSession with wolfSSL's zero-read contract) run against a loopback
RFC 6455 server written in tests/src/google/live/loopback.h. Session and
relay tests use the same server to carry hand-written JSON in the documented
shapes; that proves the module's handling of those shapes, not Google's
behavior. Request bodies are checked to use only names in kLiveSpec. The
loopback server can serve several sequential connections (used for
resume()). The TLS seam has two fakes: a passthrough with wolfSSL's old
zero-read contract (elapsed-time fallback) and one that reports
TlsTimeoutError / TlsReadStatus::kClosed (a close arriving after more than
half the read timeout is still a close). The wolfSSL session itself was
measured against a local TLS 1.3 server (see Contracts, TLS); there is no
automated test of the real session against a self-signed peer, because it
loads only the system CA store.
Sources
- https://ai.google.dev/api/live (WebSocket API reference: messages, fields)
- https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket (URL with
?key=, setup,audio/pcm;rate=16000) - https://generativelanguage.googleapis.com/$discovery/rest?version=v1beta
- https://aiplatform.googleapis.com/$discovery/rest?version=v1 and
version=v1beta1 - https://gemini-ex.hexdocs.pm/live_api.md (community client: Vertex path
wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent) - https://docs.litellm.ai/docs/providers/vertex_realtime (Vertex hosts per region, OAuth bearer)
- RFC 6455
Unverified
- The Vertex Live reference page (docs.cloud.google.com ... reference/models/live-api) could not be read in full, and a second attempt in the gap-closing round was refused by the fetch tool (session limit), so Vertex-specific message differences from the Gemini API reference are still unread. The Vertex endpoint path, both auth forms and every message shape used above were verified live instead.
toolCallCancellationwas never seen: a pendingtoolCallfollowed by new input did not produce it (probe, express key); its shape (ids) is from the Gemini API reference and local tests only.- Not exercised live:
activityStart/activityEndmessages (manual VAD),generationConfig.thinkingConfig, the Gemini API form (api_key) past upgrade (the express key's project has the API disabled), and the effect ofaffective_dialog(setup option accepted by the locally built message only). - Session limits (audio-only 15 min, audio+video 2 min) are from a community guide; the one measured lifetime is about 9.5 minutes (goAway at 514 s, close at 571 s, express key, audio only, resumption on), so treat the guide's numbers as unconfirmed. The module enforces its own 1800 s cap.
- Whether
audioStreamEndflushes an utterance when automatic VAD is disabled: only measured with VAD on.
Gaps
- Outbound messages are single frames (no outbound fragmentation); the 2 MiB cap bounds them.
proactivityhas no field in the Discovery schema (ProactivityConfigis empty),historyConfigandlabelsare not inLiveConfig.- Resuming is explicit (
resume()); nothing reconnects on its own after agoAway. - Token refresh is the caller's job (foundation gap): a Live session started with an OAuth token cannot outlive it.
- The relay polls; no push to the browser (the example server has no WebSocket).