feat(clients): adk+live: close gaps (YAML config import, VertexModel only, MCP tools/call and A2A card live, AP2 v0.2, live features verified, wolfSSL timeout/close status and MSG_NOSIGNAL)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
+43
-15
@@ -25,10 +25,10 @@ an HTTP relay for hosts without WebSocket.
|
||||
| `ws_stream.h/.cpp` | `IByteStream`, `TcpStream`, `TlsStream` (over `ITlsSession`), `open_stream` |
|
||||
| `ws_client.h/.cpp` | `WsClient`: masking, fragmentation, ping/pong, close, caps |
|
||||
| `live_endpoint.h/.cpp` | `LiveEndpoint`: URL, auth headers, model naming per mode |
|
||||
| `live_config.h/.cpp` | `LiveConfig`: typed setup message |
|
||||
| `live_config.h/.cpp` | `LiveConfig`: typed setup message (modalities, voice, tools, transcription, resumption, compression, VAD / activity handling, affective dialog, media resolution) |
|
||||
| `live_requests.h/.cpp` | `LiveRequests`: text, turn, audio, video, audioStreamEnd, toolResponse |
|
||||
| `live_events.h/.cpp` | `LiveEvent`, `LiveParser`: typed server events |
|
||||
| `live_session.h/.cpp` | `LiveSession`: connect, send, bounded event queue, close |
|
||||
| `live_session.h/.cpp` | `LiveSession`: connect, resume (new connection with the latest handle), send, bounded event queue, close |
|
||||
| `live_relay.h/.cpp` | `LiveRelay`: sessions by id, events by cursor, `event_to_json` |
|
||||
| `tools/check_discovery.py` | Discovery and endpoint check (python3 stdlib) |
|
||||
|
||||
@@ -45,6 +45,7 @@ an HTTP relay for hosts without WebSocket.
|
||||
| Socket I/O timeout, reader poll | 5 s, 200 ms |
|
||||
| Close grace | 1.5 s |
|
||||
| Session lifetime | 1800 s, then closed with a `Closed` event |
|
||||
| `resume()` calls per session | 5 |
|
||||
| Queued events / bytes | 1024 / 8 MiB, oldest dropped and counted (`stats().dropped`) |
|
||||
| Text / audio chunk / video chunk | 32000 B / 256 KiB / 1 MiB |
|
||||
| System instruction / function declarations / tool responses | 8000 chars / 16 / 16 |
|
||||
@@ -63,8 +64,10 @@ an HTTP relay for hosts without WebSocket.
|
||||
- Server JSON arrives in text and in binary frames; both are parsed.
|
||||
- One server message can yield several events. Order: setupComplete; model text and inline data in part order; input/output transcription; interrupted; generationComplete; turnComplete last. Inline `audio/*` is `Audio` (decoded bytes), other inline data `Media`. Undecodable messages become one `Error` event, unknown ones `Unknown`, never silence.
|
||||
- Sending is thread safe; one reader thread receives. `close()` sends a close frame, waits up to the grace, joins.
|
||||
- Resumption: with `config.session_resumption` the server sends `sessionResumptionUpdate` (`newHandle`, `resumable`); the session keeps the latest resumable handle. `LiveSession::resume()` stops the current connection, opens a new one with that handle in `setup.sessionResumption.handle`, waits for `setupComplete` and restarts the reader; queued events and sequence numbers carry on, and the old connection's `Closed` event stays in the queue. It is explicit (call it on a `GoAway` event or after a `Closed` event), never automatic, and capped at 5 per session. Without the option or without a handle it throws before any socket.
|
||||
- Setup options beyond text and audio: `activity_detection` (automaticActivityDetection: disabled, start / end sensitivity, prefix padding 0..5000 ms, silence duration 0..10000 ms), `activity_handling`, `turn_coverage`, `affective_dialog`, `media_resolution`, `compression_trigger_tokens` / `compression_target_tokens` (int64, sent as JSON strings). Enum values are checked before any socket. Not in `LiveConfig`: `proactivity` (an empty `ProactivityConfig` in Discovery, no fields to declare), `historyConfig`, `labels`.
|
||||
- Relay: the session resumption handle is not sent to browsers. Ids are `s` plus 12 hex chars.
|
||||
- TLS: `TlsStream` uses `ITlsSession` (wolfSSL) on a blocking fd with socket read and write timeouts. The backend reports timeout and dropped connection alike (zero read or a read error), so the stream tells them apart by elapsed time against the read timeout. It ignores SIGPIPE process wide (the backend writes without `MSG_NOSIGNAL`).
|
||||
- TLS: `TlsStream` uses `ITlsSession` (wolfSSL) on a blocking fd with socket read and write timeouts. The wolfSSL session (`src/clients/transports/tls_wolfssl.cpp`) now tells the cases apart: a socket read or write timeout throws `TlsTimeoutError` (a `TlsError`), a closed connection (TLS close_notify, or a TCP drop without one) returns 0 with `last_read_status() == TlsReadStatus::kClosed`. Measured against a local TLS 1.3 server with a 500 ms `SO_RCVTIMEO`: silence is `wolfSSL_read` -1 with `WOLFSSL_ERROR_WANT_READ`, a drop without close_notify is 0 then `WOLFSSL_ERROR_SYSCALL`, a close_notify is 0 then `WOLFSSL_ERROR_ZERO_RETURN` (the old code compared these against the `WOLFSSL_CBIO_ERR_*` constants, which never matched). A session that does not say (`kUnknown`, e.g. a test passthrough) falls back to elapsed time against the read timeout. Writes go through a send callback with `MSG_NOSIGNAL`, so a vanished peer cannot raise SIGPIPE; `TlsStream` no longer changes the process signal disposition. An `ITlsSession` supplied through `LiveOptions::tls` must not raise SIGPIPE either.
|
||||
|
||||
## 4. Example routes (`example/app/agent/LiveDemo.cpp`)
|
||||
|
||||
@@ -84,7 +87,7 @@ plus project (Vertex express), else any `GOOGLE_API_KEY` (Gemini API form).
|
||||
|
||||
## 5. Verified against discovery
|
||||
|
||||
`python3 -I src/google/live/tools/check_discovery.py`, run 2026-10-24: 69
|
||||
`python3 -I src/google/live/tools/check_discovery.py`, run 2026-10-24: 81
|
||||
spec rows checked (2 Discovery documents, 1 documentation page), 0
|
||||
failures; 3 unauthenticated upgrade probes answered `101 Switching
|
||||
Protocols` with the correct `Sec-WebSocket-Accept`. A bogus service name
|
||||
@@ -127,6 +130,24 @@ server closes TCP after its close frame without TLS close_notify.
|
||||
Opt-in tests: `MOLO_LIVE_NET=1` with `GOOGLE_CLOUD_PROJECT` and
|
||||
`GOOGLE_API_KEY` or `GOOGLE_ACCESS_TOKEN` (`live_net.*`, loud SKIP otherwise).
|
||||
|
||||
Second round, 2026-10-24, same model and region, every test once with the
|
||||
express key (header) and once with the OAuth bearer, a few tiny messages each
|
||||
(`tests/src/google/live/test_live_net.cpp`):
|
||||
|
||||
| Feature | Result |
|
||||
|---|---|
|
||||
| Tool call round trip | `setup.tools.functionDeclarations` with `get_temperature(city)`; the model sent `toolCall.functionCalls[]` with `id`, `name` and `args {"city":"Utrecht"}`; the turn that carried the call ends with `generationComplete` / `turnComplete` (empty transcription), so a client must not stop at that first `turnComplete`; after `toolResponse.functionResponses[{id,name,response}]` the model answered in a second turn ("The temperature in Utrecht is currently 17 degrees Celsius."), both auth forms |
|
||||
| `clientContent` turn | `turns[{role:user,parts[{text}]}]` + `turnComplete:true` starts a model turn, both auth forms (it carried the tool question above and the resumption turns) |
|
||||
| Output transcription | `outputTranscription` events with text, both auth forms |
|
||||
| Input transcription | audio in (the model's own speech, resampled 24 kHz to 16 kHz, `audio/pcm;rate=16000` chunks of 250 ms): `inputTranscription` events arrived, both auth forms. Needed about 1 s of trailing silence: `audioStreamEnd` alone, without silence, produced no event at all within 30 s (it is accepted without error); silence followed by `audioStreamEnd` worked |
|
||||
| Session resumption | `sessionResumption: {}` in setup: `sessionResumptionUpdate {newHandle, resumable:true}` (36 characters) arrived around the turn end. Reconnecting with `sessionResumption.handle` (new object and `LiveSession::resume()` on the same object) kept the context: asked for "the secret word" (pineapple, mango) the model answered it, both auth forms |
|
||||
| Interruption | new `realtimeInput.text` while the model spoke: an `interrupted` event, both auth forms |
|
||||
| Video input | `realtimeInput.video {data, mimeType:"image/jpeg"}` frames (4 frames, 700 ms apart) of a solid red 256x256 JPEG, then a text question: "Red." on both auth forms. A 64x64 frame was accepted without error but the model answered that it could not see an image; the first turn after frames is sometimes empty, so the test asks once more |
|
||||
| `goAway` | idle session with resumption on, express key: `goAway` after 514 s with `timeLeft` 57 s (parsed from the protobuf duration string), `Closed` code 1000 at 571 s. The server's connection lifetime is about 9.5 minutes here |
|
||||
| Setup options | `automaticActivityDetection` (sensitivities LOW, padding 100 ms, silence 700 ms), `activityHandling`, `mediaResolution`, `contextWindowCompression` (`triggerTokens`, `slidingWindow.targetTokens` as strings): accepted, a turn completed, both auth forms |
|
||||
| `?key=` on the Vertex endpoint | observed with a one-off websocket-client probe (not in the tree): upgrade, setup with the project model name and `setupComplete`, no turn |
|
||||
| Express key, model name without a project (`publishers/google/models/{m}`) | observed with the same probe: `setupComplete` |
|
||||
|
||||
## 7. Verified locally (deterministic, no Google)
|
||||
|
||||
Framing, handshake failures, fragmentation with interleaved ping, protocol
|
||||
@@ -135,7 +156,15 @@ violation closes, caps, abrupt disconnect, the TLS seam (a passthrough
|
||||
RFC 6455 server written in `tests/src/google/live/loopback.h`. Session and
|
||||
relay tests use the same server to carry hand-written JSON in the documented
|
||||
shapes; that proves the module's handling of those shapes, not Google's
|
||||
behavior. Request bodies are checked to use only names in `kLiveSpec`.
|
||||
behavior. Request bodies are checked to use only names in `kLiveSpec`. The
|
||||
loopback server can serve several sequential connections (used for
|
||||
`resume()`). The TLS seam has two fakes: a passthrough with wolfSSL's old
|
||||
zero-read contract (elapsed-time fallback) and one that reports
|
||||
`TlsTimeoutError` / `TlsReadStatus::kClosed` (a close arriving after more than
|
||||
half the read timeout is still a close). The wolfSSL session itself was
|
||||
measured against a local TLS 1.3 server (see Contracts, TLS); there is no
|
||||
automated test of the real session against a self-signed peer, because it
|
||||
loads only the system CA store.
|
||||
|
||||
## Sources
|
||||
|
||||
@@ -149,17 +178,16 @@ behavior. Request bodies are checked to use only names in `kLiveSpec`.
|
||||
|
||||
## Unverified
|
||||
|
||||
- The Vertex Live reference page (docs.cloud.google.com ... reference/models/live-api) could not be read in full (very large page, navigation only in the fetched part), so Vertex-specific message differences from the Gemini API reference are unread. The Vertex endpoint path and both auth forms are verified live instead.
|
||||
- `?key=` on the Vertex endpoint: only seen to be accepted at upgrade, no turn run.
|
||||
- Not exercised live: tool calls and tool responses, `toolCallCancellation`, `goAway`, session resumption (handle exchange, resume), context window compression, video input, `audioStreamEnd`, `clientContent` turns, input transcription, activity/VAD settings, interruption. They follow the documented shapes and are covered by local tests only.
|
||||
- Express key requirement of a project in the Vertex model name: the run used the project; not tried without.
|
||||
- `timeLeft` is parsed as a protobuf duration string ("30s"); the format is from the Duration type, not from a captured message.
|
||||
- Session limits (audio-only 15 min, audio+video 2 min) are from a community guide; the module enforces its own 1800 s cap.
|
||||
- The Vertex Live reference page (docs.cloud.google.com ... reference/models/live-api) could not be read in full, and a second attempt in the gap-closing round was refused by the fetch tool (session limit), so Vertex-specific message differences from the Gemini API reference are still unread. The Vertex endpoint path, both auth forms and every message shape used above were verified live instead.
|
||||
- `toolCallCancellation` was never seen: a pending `toolCall` followed by new input did not produce it (probe, express key); its shape (`ids`) is from the Gemini API reference and local tests only.
|
||||
- Not exercised live: `activityStart` / `activityEnd` messages (manual VAD), `generationConfig.thinkingConfig`, the Gemini API form (`api_key`) past upgrade (the express key's project has the API disabled), and the effect of `affective_dialog` (setup option accepted by the locally built message only).
|
||||
- Session limits (audio-only 15 min, audio+video 2 min) are from a community guide; the one measured lifetime is about 9.5 minutes (goAway at 514 s, close at 571 s, express key, audio only, resumption on), so treat the guide's numbers as unconfirmed. The module enforces its own 1800 s cap.
|
||||
- Whether `audioStreamEnd` flushes an utterance when automatic VAD is disabled: only measured with VAD on.
|
||||
|
||||
## Gaps
|
||||
|
||||
- Outbound messages are single frames (no outbound fragmentation).
|
||||
- Proactivity, `historyConfig`, labels, VAD tuning, media resolution, affective dialog are not in `LiveConfig` (setup can be extended in `live_config.cpp`).
|
||||
- No automatic reconnect or resume on `goAway`; the handle is available from `LiveSession::resumption_handle()`.
|
||||
- Token refresh is the caller's job (foundation gap).
|
||||
- Outbound messages are single frames (no outbound fragmentation); the 2 MiB cap bounds them.
|
||||
- `proactivity` has no field in the Discovery schema (`ProactivityConfig` is empty), `historyConfig` and `labels` are not in `LiveConfig`.
|
||||
- Resuming is explicit (`resume()`); nothing reconnects on its own after a `goAway`.
|
||||
- Token refresh is the caller's job (foundation gap): a Live session started with an OAuth token cannot outlive it.
|
||||
- The relay polls; no push to the browser (the example server has no WebSocket).
|
||||
|
||||
@@ -25,6 +25,7 @@ struct LiveCaps {
|
||||
// Session.
|
||||
static constexpr int kSetupTimeoutMs = 15000;
|
||||
static constexpr int kMaxSessionSeconds = 1800;
|
||||
static constexpr int kMaxResumes = 5; // LiveSession::resume() calls per session
|
||||
static constexpr std::size_t kMaxQueuedEvents = 1024;
|
||||
static constexpr std::size_t kMaxQueuedBytes = 8u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxTextChars = 32000;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#include "google/live/live_config.h"
|
||||
|
||||
#include <cctype>
|
||||
#include <initializer_list>
|
||||
|
||||
#include "google/errors.h"
|
||||
#include "google/live/caps.h"
|
||||
@@ -23,6 +24,11 @@ bool valid_handle(const std::string& s) {
|
||||
if (c <= 0x20 || c == 0x7f || c == '"') return false;
|
||||
return true;
|
||||
}
|
||||
bool one_of(const std::string& v, std::initializer_list<const char*> allowed) {
|
||||
for (const char* a : allowed)
|
||||
if (v == a) return true;
|
||||
return false;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Json LiveConfig::to_setup(const LiveEndpoint& endpoint) const {
|
||||
@@ -72,8 +78,56 @@ Json LiveConfig::to_setup(const LiveEndpoint& endpoint) const {
|
||||
}
|
||||
gen.set("speechConfig", speech);
|
||||
}
|
||||
if (affective_dialog) gen.set("enableAffectiveDialog", true);
|
||||
if (!media_resolution.empty()) {
|
||||
if (!one_of(media_resolution, {"MEDIA_RESOLUTION_LOW", "MEDIA_RESOLUTION_MEDIUM", "MEDIA_RESOLUTION_HIGH"}))
|
||||
throw GoogleError::invalid("media_resolution must be MEDIA_RESOLUTION_LOW, _MEDIUM or _HIGH");
|
||||
gen.set("mediaResolution", media_resolution);
|
||||
}
|
||||
setup.set("generationConfig", gen);
|
||||
|
||||
if (activity_detection || !activity_handling.empty() || !turn_coverage.empty()) {
|
||||
Json rt = Json::object();
|
||||
if (activity_detection) {
|
||||
const LiveActivityDetection& v = *activity_detection;
|
||||
Json d = Json::object();
|
||||
if (v.disabled) d.set("disabled", true);
|
||||
if (!v.start_sensitivity.empty()) {
|
||||
if (!one_of(v.start_sensitivity, {"START_SENSITIVITY_HIGH", "START_SENSITIVITY_LOW"}))
|
||||
throw GoogleError::invalid("start_sensitivity must be START_SENSITIVITY_HIGH or _LOW");
|
||||
d.set("startOfSpeechSensitivity", v.start_sensitivity);
|
||||
}
|
||||
if (!v.end_sensitivity.empty()) {
|
||||
if (!one_of(v.end_sensitivity, {"END_SENSITIVITY_HIGH", "END_SENSITIVITY_LOW"}))
|
||||
throw GoogleError::invalid("end_sensitivity must be END_SENSITIVITY_HIGH or _LOW");
|
||||
d.set("endOfSpeechSensitivity", v.end_sensitivity);
|
||||
}
|
||||
if (v.prefix_padding_ms != -1) {
|
||||
if (v.prefix_padding_ms < 0 || v.prefix_padding_ms > 5000)
|
||||
throw GoogleError::invalid("prefix_padding_ms must be within 0..5000");
|
||||
d.set("prefixPaddingMs", v.prefix_padding_ms);
|
||||
}
|
||||
if (v.silence_duration_ms != -1) {
|
||||
if (v.silence_duration_ms < 0 || v.silence_duration_ms > 10000)
|
||||
throw GoogleError::invalid("silence_duration_ms must be within 0..10000");
|
||||
d.set("silenceDurationMs", v.silence_duration_ms);
|
||||
}
|
||||
rt.set("automaticActivityDetection", d);
|
||||
}
|
||||
if (!activity_handling.empty()) {
|
||||
if (!one_of(activity_handling, {"START_OF_ACTIVITY_INTERRUPTS", "NO_INTERRUPTION"}))
|
||||
throw GoogleError::invalid("activity_handling must be START_OF_ACTIVITY_INTERRUPTS or NO_INTERRUPTION");
|
||||
rt.set("activityHandling", activity_handling);
|
||||
}
|
||||
if (!turn_coverage.empty()) {
|
||||
if (!one_of(turn_coverage, {"TURN_INCLUDES_ONLY_ACTIVITY", "TURN_INCLUDES_ALL_INPUT",
|
||||
"TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO"}))
|
||||
throw GoogleError::invalid("turn_coverage is not a TURN_INCLUDES_* value");
|
||||
rt.set("turnCoverage", turn_coverage);
|
||||
}
|
||||
setup.set("realtimeInputConfig", rt);
|
||||
}
|
||||
|
||||
if (!system_instruction.empty()) {
|
||||
if (system_instruction.size() > LiveCaps::kMaxSystemChars)
|
||||
throw GoogleError::invalid("system instruction over cap");
|
||||
@@ -125,8 +179,15 @@ Json LiveConfig::to_setup(const LiveEndpoint& endpoint) const {
|
||||
setup.set("sessionResumption", r);
|
||||
}
|
||||
if (context_compression) {
|
||||
if (compression_trigger_tokens < 0 || compression_target_tokens < 0 ||
|
||||
compression_trigger_tokens > 10000000 || compression_target_tokens > 10000000)
|
||||
throw GoogleError::invalid("compression token counts must be within 0..10000000");
|
||||
Json c = Json::object();
|
||||
c.set("slidingWindow", Json::object());
|
||||
Json window = Json::object();
|
||||
// int64 fields travel as JSON strings (proto3 JSON mapping).
|
||||
if (compression_target_tokens) window.set("targetTokens", std::to_string(compression_target_tokens));
|
||||
c.set("slidingWindow", window);
|
||||
if (compression_trigger_tokens) c.set("triggerTokens", std::to_string(compression_trigger_tokens));
|
||||
setup.set("contextWindowCompression", c);
|
||||
}
|
||||
Json root = Json::object();
|
||||
|
||||
@@ -18,6 +18,15 @@ struct LiveFunctionDeclaration {
|
||||
clients::body::Json parameters;
|
||||
};
|
||||
|
||||
// realtimeInputConfig.automaticActivityDetection (server side VAD).
|
||||
struct LiveActivityDetection {
|
||||
bool disabled = false;
|
||||
std::string start_sensitivity; // START_SENSITIVITY_HIGH | _LOW, empty: server default
|
||||
std::string end_sensitivity; // END_SENSITIVITY_HIGH | _LOW
|
||||
int prefix_padding_ms = -1; // -1: server default, else 0..5000
|
||||
int silence_duration_ms = -1; // -1: server default, else 0..10000
|
||||
};
|
||||
|
||||
struct LiveConfig {
|
||||
std::string model = "gemini-live-2.5-flash-native-audio";
|
||||
std::vector<std::string> response_modalities = {"AUDIO"}; // AUDIO or TEXT
|
||||
@@ -30,7 +39,15 @@ struct LiveConfig {
|
||||
bool output_transcription = false;
|
||||
bool session_resumption = false;
|
||||
std::string resumption_handle; // resume a previous session
|
||||
bool context_compression = false; // slidingWindow with server defaults
|
||||
bool context_compression = false; // slidingWindow, server defaults unless set:
|
||||
long compression_trigger_tokens = 0; // contextWindowCompression.triggerTokens, 0: default
|
||||
long compression_target_tokens = 0; // slidingWindow.targetTokens, 0: default
|
||||
std::optional<LiveActivityDetection> activity_detection;
|
||||
std::string activity_handling; // START_OF_ACTIVITY_INTERRUPTS | NO_INTERRUPTION
|
||||
std::string turn_coverage; // TURN_INCLUDES_ONLY_ACTIVITY | TURN_INCLUDES_ALL_INPUT |
|
||||
// TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO
|
||||
bool affective_dialog = false; // generationConfig.enableAffectiveDialog
|
||||
std::string media_resolution; // MEDIA_RESOLUTION_LOW | _MEDIUM | _HIGH
|
||||
std::vector<LiveFunctionDeclaration> functions;
|
||||
|
||||
// Validates and renders {"setup": {...}}. Throws GoogleError (invalid).
|
||||
|
||||
@@ -55,6 +55,10 @@ LiveSession::~LiveSession() {
|
||||
void LiveSession::connect() {
|
||||
if (connected_.exchange(true)) throw GoogleError::invalid("session already connected");
|
||||
endpoint_.validate();
|
||||
open_transport();
|
||||
}
|
||||
|
||||
void LiveSession::open_transport() {
|
||||
std::string setup = config_.to_setup(endpoint_).dump();
|
||||
try {
|
||||
WsUrl url = WsUrl::parse(endpoint_.url());
|
||||
@@ -83,10 +87,44 @@ void LiveSession::connect() {
|
||||
ws_.reset();
|
||||
throw map_error(e);
|
||||
}
|
||||
stop_ = false;
|
||||
close_deadline_ms_ = 0;
|
||||
open_ = true;
|
||||
reader_ = std::thread([this] { reader_loop(); });
|
||||
}
|
||||
|
||||
void LiveSession::stop_transport() {
|
||||
if (!ws_) return;
|
||||
if (open_.exchange(false) || reader_.joinable()) {
|
||||
close_deadline_ms_ = now_ms() + LiveCaps::kCloseGraceMs;
|
||||
ws_->close(1000, "client closing");
|
||||
}
|
||||
if (reader_.joinable()) reader_.join();
|
||||
ws_->shutdown();
|
||||
}
|
||||
|
||||
void LiveSession::resume() {
|
||||
if (!connected_) throw GoogleError(409, "FAILED_PRECONDITION", "session is not connected", "live");
|
||||
if (!config_.session_resumption)
|
||||
throw GoogleError::invalid("resume() needs config.session_resumption");
|
||||
if (resumes_ >= LiveCaps::kMaxResumes)
|
||||
throw GoogleError(429, "RESOURCE_EXHAUSTED", "resume cap reached", "live");
|
||||
std::string handle = resumption_handle();
|
||||
if (handle.empty())
|
||||
throw GoogleError(409, "FAILED_PRECONDITION", "no resumption handle received yet", "live");
|
||||
++resumes_;
|
||||
stop_transport();
|
||||
ws_.reset();
|
||||
config_.resumption_handle = handle;
|
||||
try {
|
||||
open_transport();
|
||||
} catch (...) {
|
||||
open_ = false;
|
||||
cv_.notify_all();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
void LiveSession::send_raw(const std::string& json) {
|
||||
if (!open_ || !ws_) throw GoogleError(409, "FAILED_PRECONDITION", "session is not open", "live");
|
||||
try {
|
||||
@@ -202,12 +240,7 @@ std::size_t LiveSession::pending() const {
|
||||
|
||||
void LiveSession::close() {
|
||||
if (!ws_) return;
|
||||
if (open_.exchange(false) || reader_.joinable()) {
|
||||
close_deadline_ms_ = now_ms() + LiveCaps::kCloseGraceMs;
|
||||
ws_->close(1000, "client closing");
|
||||
}
|
||||
if (reader_.joinable()) reader_.join();
|
||||
ws_->shutdown();
|
||||
stop_transport();
|
||||
cv_.notify_all();
|
||||
}
|
||||
|
||||
|
||||
@@ -45,6 +45,16 @@ class LiveSession {
|
||||
// for upstream failures). Call once.
|
||||
void connect();
|
||||
|
||||
// Reconnects with the latest session resumption handle (needs
|
||||
// config.session_resumption and a received handle; at most kMaxResumes per
|
||||
// session). Typical use: on a GoAway event, or after a Closed event with
|
||||
// a handle in hand. Stops the old connection (its Closed event is queued),
|
||||
// opens a new one with the handle in setup, waits for setupComplete (a
|
||||
// second SetupComplete event is queued) and restarts the reader. Queued
|
||||
// events and sequence numbers carry on. Throws like connect(); on failure
|
||||
// the session is closed.
|
||||
void resume();
|
||||
|
||||
void send_text(const std::string& text);
|
||||
void send_turn(const std::string& text, bool turn_complete = true);
|
||||
void send_audio(const std::string& pcm,
|
||||
@@ -69,6 +79,8 @@ class LiveSession {
|
||||
std::string resumption_handle() const;
|
||||
|
||||
private:
|
||||
void open_transport(); // connect, upgrade, setup, wait setupComplete, start reader
|
||||
void stop_transport(); // close frame, join reader, drop the socket
|
||||
void send_raw(const std::string& json);
|
||||
void push(LiveEvent e);
|
||||
void reader_loop();
|
||||
@@ -91,6 +103,7 @@ class LiveSession {
|
||||
std::uint64_t seq_ = 0;
|
||||
Stats stats_;
|
||||
std::string handle_;
|
||||
int resumes_ = 0;
|
||||
};
|
||||
|
||||
} // namespace molo::google::live
|
||||
|
||||
@@ -57,6 +57,18 @@ inline constexpr LiveSpecRow kLiveSpec[] = {
|
||||
{"FunctionDeclaration", "parameters", LIVE_GL},
|
||||
{"SessionResumptionConfig", "handle", LIVE_GL},
|
||||
{"ContextWindowCompressionConfig", "slidingWindow", LIVE_GL},
|
||||
{"ContextWindowCompressionConfig", "triggerTokens", LIVE_GL},
|
||||
{"SlidingWindow", "targetTokens", LIVE_GL},
|
||||
{"RealtimeInputConfig", "automaticActivityDetection", LIVE_GL},
|
||||
{"RealtimeInputConfig", "activityHandling", LIVE_GL},
|
||||
{"RealtimeInputConfig", "turnCoverage", LIVE_GL},
|
||||
{"AutomaticActivityDetection", "disabled", LIVE_GL},
|
||||
{"AutomaticActivityDetection", "startOfSpeechSensitivity", LIVE_GL},
|
||||
{"AutomaticActivityDetection", "endOfSpeechSensitivity", LIVE_GL},
|
||||
{"AutomaticActivityDetection", "prefixPaddingMs", LIVE_GL},
|
||||
{"AutomaticActivityDetection", "silenceDurationMs", LIVE_GL},
|
||||
{"GenerationConfig", "enableAffectiveDialog", LIVE_GL},
|
||||
{"GenerationConfig", "mediaResolution", LIVE_GL},
|
||||
{"FunctionCall", "id", LIVE_GL},
|
||||
{"FunctionCall", "name", LIVE_GL},
|
||||
{"FunctionCall", "args", LIVE_GL},
|
||||
|
||||
@@ -9,9 +9,7 @@
|
||||
|
||||
#include <cerrno>
|
||||
#include <chrono>
|
||||
#include <csignal>
|
||||
#include <cstring>
|
||||
#include <mutex>
|
||||
|
||||
#include "clients/transports/tls_wolfssl.h"
|
||||
#include "google/live/ws_error.h"
|
||||
@@ -81,10 +79,6 @@ TlsStream::TlsStream(int fd,
|
||||
: fd_(fd), read_timeout_ms_(io_timeout_ms), session_(std::move(session)) {
|
||||
set_timeout(fd_, SO_RCVTIMEO, io_timeout_ms);
|
||||
set_timeout(fd_, SO_SNDTIMEO, io_timeout_ms);
|
||||
// The TLS backend writes without MSG_NOSIGNAL; a peer that vanished would
|
||||
// otherwise kill the process. Process-wide, once.
|
||||
static std::once_flag sigpipe_once;
|
||||
std::call_once(sigpipe_once, [] { std::signal(SIGPIPE, SIG_IGN); });
|
||||
try {
|
||||
session_->handshake(fd_, hostname, false);
|
||||
} catch (const clients::transports::TlsError& e) {
|
||||
@@ -107,21 +101,25 @@ void TlsStream::set_read_timeout_ms(int ms) {
|
||||
set_timeout(fd_, SO_RCVTIMEO, ms);
|
||||
}
|
||||
|
||||
// ITlsSession::recv_some returns 0 for both "timed out" and "peer closed".
|
||||
// A return well before the socket timeout is a close; one that waited out
|
||||
// at least half the timeout is a timeout.
|
||||
// The wolfSSL session throws TlsTimeoutError on a socket timeout and reports
|
||||
// last_read_status() == kClosed for a close_notify or a TCP drop (Google
|
||||
// closes TCP after its close frame, without close_notify). A session that does
|
||||
// not say (kUnknown, e.g. a test passthrough returning 0 for both) falls back
|
||||
// to elapsed time: a return well before the socket timeout is a close, one
|
||||
// that waited out at least half of it is a timeout.
|
||||
ReadResult TlsStream::read(char* out, std::size_t max) {
|
||||
auto t0 = std::chrono::steady_clock::now();
|
||||
std::size_t n = 0;
|
||||
// The backend reports a socket timeout and a dropped connection (Google
|
||||
// closes TCP after its close frame, without close_notify) both as a read
|
||||
// error or a zero read. Elapsed time tells them apart.
|
||||
try {
|
||||
n = session_->recv_some(out, max);
|
||||
} catch (const clients::transports::TlsTimeoutError&) {
|
||||
return {ReadStatus::Timeout, 0};
|
||||
} catch (const clients::transports::TlsError&) {
|
||||
n = 0;
|
||||
return {ReadStatus::Eof, 0}; // a broken connection, not a timeout
|
||||
}
|
||||
if (n > 0) return {ReadStatus::Data, n};
|
||||
if (session_->last_read_status() == clients::transports::TlsReadStatus::kClosed)
|
||||
return {ReadStatus::Eof, 0};
|
||||
auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
std::chrono::steady_clock::now() - t0)
|
||||
.count();
|
||||
|
||||
Reference in New Issue
Block a user