feat(clients): google cloud voice: generic client, TTS, STT, Gemini audio, live voice page
src/google: GoogleClient over molo::clients (key in x-goog-api-key, never in
URLs), Service base, TextToSpeech, SpeechToText, Generative (replies, audio
in, audio out), Recognizer/Synthesizer seams with cloud, gemini and vertex
backends, VoicePipeline with bounded history. Example: /voice live
conversation page, /voice/{say,hear,turn,reset}, /google/status. Keys load
from env, .env.json.local, .env.json. Paid routes are excluded from locust.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,90 @@
|
||||
# Google Cloud APIs - Specification
|
||||
|
||||
Owned directory: `src/google/`. Namespace `molo::google`. Depends on
|
||||
`molo::clients` (HTTP client, JSON DOM, transports) and the Base64 codec in
|
||||
`src/security/crypto`. No other part is included.
|
||||
|
||||
Purpose: one generic way to call any Google Cloud REST API, plus concrete
|
||||
services built on it. Voice (text-to-speech, speech-to-text, Gemini reply and audio,
|
||||
conversation pipeline) is the first set. Adding a service means one
|
||||
`ServiceEndpoint` constant and one `Service` subclass; no change to the
|
||||
client, credentials, errors, or caps.
|
||||
|
||||
## 1. Module map (one class or concern per file)
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings, never autotuned |
|
||||
| `errors.h/.cpp` | `GoogleError` (http status, Google status, message), `NotConfiguredError`, parser for Google's `{"error":{...}}` body |
|
||||
| `credentials.h/.cpp` | `Credentials`: API key and OAuth bearer providers, env loader |
|
||||
| `endpoints.h/.cpp` | `ServiceEndpoint` plus the catalog of known services |
|
||||
| `client.h/.cpp` | `GoogleClient`: auth, JSON in/out, error mapping, over any `clients::Transport` |
|
||||
| `service.h/.cpp` | `Service` base class: holds client plus endpoint, one `call()` |
|
||||
| `text_to_speech.h/.cpp` | `TextToSpeech::synthesize`, `voices` |
|
||||
| `speech_to_text.h/.cpp` | `SpeechToText::recognize` |
|
||||
| `generative.h/.cpp` | `Generative::generate` (Gemini `generateContent`) |
|
||||
| `voice_io.h/.cpp` | `Recognizer`, `Synthesizer` seams plus request and audio types |
|
||||
| `gemini_voice.h/.cpp` | Gemini-based `Recognizer` and `Synthesizer` |
|
||||
| `voice_backend.h/.cpp` | `cloud` / `gemini` / `vertex` selection and wiring |
|
||||
| `wav.h/.cpp` | PCM16 to WAV |
|
||||
| `voice.h/.cpp` | `Responder`, `GenerativeResponder`, `VoiceSessions`, `VoicePipeline` |
|
||||
|
||||
## 2. Static caps (see caps.h)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| TTS text characters per request | 5000 |
|
||||
| STT audio bytes per request | 10 MiB (Google sync limit) |
|
||||
| Generation output tokens | 300 default, 1024 max |
|
||||
| Conversation history | 20 turns per conversation |
|
||||
| Conversations held in memory | 1000, oldest evicted |
|
||||
| Conversation id | `[A-Za-z0-9_-]`, 1 to 64 chars |
|
||||
| Message text per history entry | 4000 characters |
|
||||
| Request timeout (total) | 60 s (client default) |
|
||||
|
||||
## 3. Contracts
|
||||
|
||||
- Credentials: `GOOGLE_API_KEY` is read once by `GoogleClient::from_env()`
|
||||
from the process environment, else `.env.json.local` (git-ignored, put
|
||||
secrets here), else `.env.json`, in the working directory. All env knobs
|
||||
below follow the same order.
|
||||
The key travels in the `x-goog-api-key` header, never in the URL, so it
|
||||
never reaches a span, log line, or proxy URL. Bearer tokens (OAuth, for
|
||||
APIs that refuse keys) use the same seam: `Credentials::bearer`.
|
||||
- No key: `configured()` is false and every call throws `NotConfiguredError`
|
||||
(HTTP 503, status `NOT_CONFIGURED`) before any network traffic. Loud, not
|
||||
silent.
|
||||
- Upstream failure: any non-2xx becomes `GoogleError` carrying the HTTP
|
||||
status, Google's `status` string (`PERMISSION_DENIED`, `RESOURCE_EXHAUSTED`,
|
||||
...) and message. Transport failures become `GoogleError` with status
|
||||
`UNAVAILABLE`. The key is never part of an error message.
|
||||
- Invalid input (empty text, text over cap, empty or oversized audio, bad
|
||||
conversation id) throws `GoogleError` 400 `INVALID_ARGUMENT` before any
|
||||
network traffic.
|
||||
- Transports are swappable: production uses `HttpTransport` (HTTPS via
|
||||
wolfSSL), tests use `MockTransport`. Calling code does not change.
|
||||
- Voice replies disable model thinking (`thinkingBudget` 0): reasoning tokens
|
||||
count against the output cap and truncated spoken replies in practice.
|
||||
- Backends (`voice_backend.h`): `cloud` (classic TTS/STT), `gemini`
|
||||
(generativelanguage audio in/out), `vertex` (aiplatform, express keys
|
||||
starting `AQ.`). `GOOGLE_API_BACKEND` overrides the key-prefix guess.
|
||||
Gemini TTS returns raw PCM; the server wraps it as WAV.
|
||||
- Env knobs: `GOOGLE_API_KEY`, `GOOGLE_VOICE_LANG` (default `nl-NL`),
|
||||
`GOOGLE_VOICE_NAME` (default empty: Google picks the language default),
|
||||
`GOOGLE_GENAI_MODEL` (default `gemini-3.5-flash`).
|
||||
- `VoicePipeline::turn` = recognize, respond, synthesize. Silence (empty
|
||||
transcript) returns an empty result with no reply and no synthesis, so a
|
||||
quiet room costs one STT call and nothing else.
|
||||
- History is bounded per conversation and conversations are bounded in
|
||||
count; both evict oldest first.
|
||||
|
||||
## 4. Gaps
|
||||
|
||||
- Streaming recognition (gRPC/WebSocket) and the Gemini Live API are not
|
||||
implemented; turns are request/response over REST.
|
||||
- Service-account JWT exchange for OAuth tokens: callers pass a ready token
|
||||
to `Credentials::bearer`. Owner: this part, `credentials.cpp`.
|
||||
- Free-tier quota accounting is Google's; `RESOURCE_EXHAUSTED` is surfaced,
|
||||
not tracked.
|
||||
- Live behavior against Google needs a real `GOOGLE_API_KEY`; the suite
|
||||
proves request shape and error mapping with scripted responses only.
|
||||
@@ -0,0 +1,20 @@
|
||||
#pragma once
|
||||
|
||||
// Static ceilings for the Google Cloud part. Never autotuned.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
struct Caps {
|
||||
static constexpr std::size_t kMaxTtsChars = 5000;
|
||||
static constexpr std::size_t kMaxSttBytes = 10u * 1024u * 1024u;
|
||||
static constexpr int kDefaultOutputTokens = 300;
|
||||
static constexpr int kMaxOutputTokens = 1024;
|
||||
static constexpr std::size_t kMaxHistoryTurns = 20;
|
||||
static constexpr std::size_t kMaxConversations = 1000;
|
||||
static constexpr std::size_t kMaxConversationId = 64;
|
||||
static constexpr std::size_t kMaxMessageChars = 4000;
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,64 @@
|
||||
#include "google/client.h"
|
||||
|
||||
#include "clients/transports/http_transport.h"
|
||||
#include "google/credentials.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
namespace {
|
||||
clients::ClientConfig make_config(std::shared_ptr<clients::Auth> auth) {
|
||||
clients::ClientConfig config;
|
||||
config.auth = std::move(auth);
|
||||
config.persist_cookies = false;
|
||||
return config;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
GoogleClient::GoogleClient(std::unique_ptr<clients::Transport> transport,
|
||||
std::shared_ptr<clients::Auth> auth)
|
||||
: client_(std::move(transport), make_config(auth)),
|
||||
configured_(auth != nullptr) {}
|
||||
|
||||
std::shared_ptr<GoogleClient> GoogleClient::from_env() {
|
||||
return std::make_shared<GoogleClient>(
|
||||
std::make_unique<clients::transports::HttpTransport>(),
|
||||
Credentials::from_env());
|
||||
}
|
||||
|
||||
clients::body::Json GoogleClient::post_json(const ServiceEndpoint& service,
|
||||
const std::string& path,
|
||||
const clients::body::Json& body) {
|
||||
if (!configured_) throw NotConfiguredError(service.name);
|
||||
try {
|
||||
return finish(service, client_.post_json_blocking(
|
||||
service.base_url + path, body));
|
||||
} catch (const GoogleError&) {
|
||||
throw;
|
||||
} catch (const std::exception& e) {
|
||||
throw GoogleError(502, "UNAVAILABLE", e.what(), service.name);
|
||||
}
|
||||
}
|
||||
|
||||
clients::body::Json GoogleClient::get_json(const ServiceEndpoint& service,
|
||||
const std::string& path) {
|
||||
if (!configured_) throw NotConfiguredError(service.name);
|
||||
try {
|
||||
return finish(service, client_.get_blocking(service.base_url + path));
|
||||
} catch (const GoogleError&) {
|
||||
throw;
|
||||
} catch (const std::exception& e) {
|
||||
throw GoogleError(502, "UNAVAILABLE", e.what(), service.name);
|
||||
}
|
||||
}
|
||||
|
||||
clients::body::Json GoogleClient::finish(
|
||||
const ServiceEndpoint& service, const clients::ClientResponse& response) {
|
||||
if (!response.is_success()) throw error_from_response(service.name, response);
|
||||
try {
|
||||
return response.json();
|
||||
} catch (const std::exception& e) {
|
||||
throw GoogleError(502, "BAD_RESPONSE", e.what(), service.name);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,45 @@
|
||||
#pragma once
|
||||
|
||||
// GoogleClient: the one place that talks to Google. Applies the credential,
|
||||
// sends JSON, maps failures to GoogleError. Runs over any clients::Transport
|
||||
// (HttpTransport in production, MockTransport in tests).
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "clients/client.h"
|
||||
#include "google/endpoints.h"
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
class GoogleClient {
|
||||
public:
|
||||
// auth may be null: the client is then "not configured" and every call
|
||||
// throws NotConfiguredError before touching the network.
|
||||
GoogleClient(std::unique_ptr<clients::Transport> transport,
|
||||
std::shared_ptr<clients::Auth> auth);
|
||||
|
||||
// Real HTTPS transport plus GOOGLE_API_KEY from the environment.
|
||||
static std::shared_ptr<GoogleClient> from_env();
|
||||
|
||||
bool configured() const { return configured_; }
|
||||
|
||||
clients::body::Json post_json(const ServiceEndpoint& service,
|
||||
const std::string& path,
|
||||
const clients::body::Json& body);
|
||||
clients::body::Json get_json(const ServiceEndpoint& service,
|
||||
const std::string& path);
|
||||
|
||||
clients::ClientStatsSnapshot stats() const { return client_.stats(); }
|
||||
|
||||
private:
|
||||
clients::body::Json finish(const ServiceEndpoint& service,
|
||||
const clients::ClientResponse& response);
|
||||
|
||||
clients::Client client_;
|
||||
bool configured_;
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,55 @@
|
||||
#include "google/credentials.h"
|
||||
|
||||
#include <cstdlib>
|
||||
#include <fstream>
|
||||
#include <sstream>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
std::shared_ptr<clients::Auth> Credentials::api_key(std::string key) {
|
||||
return std::make_shared<clients::ApiKeyAuth>(std::move(key), "x-goog-api-key");
|
||||
}
|
||||
|
||||
std::shared_ptr<clients::Auth> Credentials::bearer(std::string token) {
|
||||
return std::make_shared<clients::BearerAuth>(std::move(token));
|
||||
}
|
||||
|
||||
namespace {
|
||||
constexpr std::size_t kMaxConfigBytes = 64 * 1024;
|
||||
|
||||
// String value of `name` in a flat JSON object file, or empty (missing,
|
||||
// oversized, malformed, or not a string: all treated as "not set").
|
||||
std::string from_json_file(const char* path, const char* name) {
|
||||
std::ifstream in(path, std::ios::binary);
|
||||
if (!in) return "";
|
||||
std::ostringstream text;
|
||||
text << in.rdbuf();
|
||||
if (text.str().size() > kMaxConfigBytes) return "";
|
||||
try {
|
||||
clients::body::Json doc = clients::body::Json::parse(text.str());
|
||||
const clients::body::Json* v = doc.is_object() ? doc.find(name) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : "";
|
||||
} catch (const std::exception&) {
|
||||
return "";
|
||||
}
|
||||
}
|
||||
} // namespace
|
||||
|
||||
// Order: process environment, then .env.json.local (git-ignored, for
|
||||
// secrets), then .env.json, both in the working directory.
|
||||
std::string Credentials::env(const char* name) {
|
||||
if (const char* v = std::getenv(name); v && *v) return v;
|
||||
std::string local = from_json_file(".env.json.local", name);
|
||||
if (!local.empty()) return local;
|
||||
return from_json_file(".env.json", name);
|
||||
}
|
||||
|
||||
std::shared_ptr<clients::Auth> Credentials::from_env(const char* name) {
|
||||
std::string key = env(name);
|
||||
if (key.empty()) return nullptr;
|
||||
return api_key(std::move(key));
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,27 @@
|
||||
#pragma once
|
||||
|
||||
// Credential providers for Google APIs. A credential is a clients::Auth, so
|
||||
// it is applied to every request by the client. API keys go in the
|
||||
// x-goog-api-key header, never in the URL.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "clients/auth.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
class Credentials {
|
||||
public:
|
||||
static std::shared_ptr<clients::Auth> api_key(std::string key);
|
||||
// OAuth access token for APIs that do not accept keys.
|
||||
static std::shared_ptr<clients::Auth> bearer(std::string token);
|
||||
// Value of `name` from the process environment, else .env.json.local,
|
||||
// else .env.json (working directory); empty when unset everywhere.
|
||||
static std::string env(const char* name = "GOOGLE_API_KEY");
|
||||
// api_key(env()) or nullptr when unset or empty.
|
||||
static std::shared_ptr<clients::Auth> from_env(
|
||||
const char* name = "GOOGLE_API_KEY");
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,24 @@
|
||||
#include "google/endpoints.h"
|
||||
|
||||
namespace molo::google::endpoints {
|
||||
|
||||
ServiceEndpoint text_to_speech() {
|
||||
return {"texttospeech", "https://texttospeech.googleapis.com/v1"};
|
||||
}
|
||||
ServiceEndpoint speech_to_text() {
|
||||
return {"speech", "https://speech.googleapis.com/v1"};
|
||||
}
|
||||
ServiceEndpoint generative() {
|
||||
return {"generativelanguage",
|
||||
"https://generativelanguage.googleapis.com/v1beta"};
|
||||
}
|
||||
|
||||
ServiceEndpoint vertex_generative() {
|
||||
return {"vertex", "https://aiplatform.googleapis.com/v1/publishers/google"};
|
||||
}
|
||||
|
||||
std::vector<ServiceEndpoint> catalog() {
|
||||
return {text_to_speech(), speech_to_text(), generative(), vertex_generative()};
|
||||
}
|
||||
|
||||
} // namespace molo::google::endpoints
|
||||
@@ -0,0 +1,26 @@
|
||||
#pragma once
|
||||
|
||||
// A Google service is a name plus a base URL (host and version prefix).
|
||||
// New Google APIs are one constant here and one Service subclass.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
struct ServiceEndpoint {
|
||||
std::string name;
|
||||
std::string base_url; // No trailing slash.
|
||||
};
|
||||
|
||||
namespace endpoints {
|
||||
ServiceEndpoint text_to_speech();
|
||||
ServiceEndpoint speech_to_text();
|
||||
ServiceEndpoint generative();
|
||||
// Gemini through Vertex AI (express-mode API keys start with "AQ.").
|
||||
ServiceEndpoint vertex_generative();
|
||||
// Every endpoint this build knows, for status pages.
|
||||
std::vector<ServiceEndpoint> catalog();
|
||||
} // namespace endpoints
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,43 @@
|
||||
#include "google/errors.h"
|
||||
|
||||
#include "clients/body/json.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
GoogleError::GoogleError(int http_status, std::string status,
|
||||
const std::string& message, std::string service)
|
||||
: std::runtime_error("molo::google: " + status + ": " + message),
|
||||
http_status_(http_status),
|
||||
status_(std::move(status)),
|
||||
service_(std::move(service)),
|
||||
message_(message) {}
|
||||
|
||||
GoogleError GoogleError::invalid(const std::string& message) {
|
||||
return GoogleError(400, "INVALID_ARGUMENT", message);
|
||||
}
|
||||
|
||||
NotConfiguredError::NotConfiguredError(const std::string& service)
|
||||
: GoogleError(503, "NOT_CONFIGURED",
|
||||
"no Google credential configured (set GOOGLE_API_KEY)",
|
||||
service) {}
|
||||
|
||||
GoogleError error_from_response(const std::string& service,
|
||||
const clients::ClientResponse& response) {
|
||||
std::string status = "UNKNOWN";
|
||||
std::string message = "HTTP " + std::to_string(response.status());
|
||||
try {
|
||||
clients::body::Json doc = response.json();
|
||||
const clients::body::Json* err = doc.find("error");
|
||||
if (err && err->is_object()) {
|
||||
if (const auto* s = err->find("status"); s && s->is_string())
|
||||
status = s->as_string();
|
||||
if (const auto* m = err->find("message"); m && m->is_string())
|
||||
message = m->as_string();
|
||||
}
|
||||
} catch (const std::exception&) {
|
||||
// Non-JSON error body: keep the generic status and message.
|
||||
}
|
||||
return GoogleError(response.status(), status, message, service);
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,45 @@
|
||||
#pragma once
|
||||
|
||||
// One error type for every Google call. Carries the HTTP status, Google's
|
||||
// own status string and a message; never the credential.
|
||||
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
|
||||
#include "clients/response.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
class GoogleError : public std::runtime_error {
|
||||
public:
|
||||
GoogleError(int http_status, std::string status, const std::string& message,
|
||||
std::string service = "");
|
||||
|
||||
int http_status() const noexcept { return http_status_; }
|
||||
const std::string& status() const noexcept { return status_; }
|
||||
const std::string& service() const noexcept { return service_; }
|
||||
// The message alone, without the "molo::google: STATUS:" prefix of what().
|
||||
const std::string& message() const noexcept { return message_; }
|
||||
|
||||
// Validation failure raised before any network traffic.
|
||||
static GoogleError invalid(const std::string& message);
|
||||
|
||||
private:
|
||||
int http_status_;
|
||||
std::string status_;
|
||||
std::string service_;
|
||||
std::string message_;
|
||||
};
|
||||
|
||||
// Raised when no credential is configured. HTTP 503, status NOT_CONFIGURED.
|
||||
class NotConfiguredError : public GoogleError {
|
||||
public:
|
||||
explicit NotConfiguredError(const std::string& service);
|
||||
};
|
||||
|
||||
// Maps a non-2xx response (Google's {"error":{code,message,status}} body, or
|
||||
// anything else) to a GoogleError.
|
||||
GoogleError error_from_response(const std::string& service,
|
||||
const clients::ClientResponse& response);
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,27 @@
|
||||
#include "google/gemini_voice.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
GeminiRecognizer::GeminiRecognizer(std::shared_ptr<Generative> generative,
|
||||
std::string model)
|
||||
: generative_(std::move(generative)), model_(std::move(model)) {}
|
||||
|
||||
Transcript GeminiRecognizer::recognize(const RecognitionRequest& request) const {
|
||||
Transcript out;
|
||||
out.text = generative_->transcribe(
|
||||
model_.empty() ? Generative::default_stt_model() : model_, request.audio,
|
||||
speech_encoding_mime(request.encoding), request.language);
|
||||
return out;
|
||||
}
|
||||
|
||||
GeminiSynthesizer::GeminiSynthesizer(std::shared_ptr<Generative> generative,
|
||||
std::string model)
|
||||
: generative_(std::move(generative)), model_(std::move(model)) {}
|
||||
|
||||
Audio GeminiSynthesizer::synthesize(const SynthesisRequest& request) const {
|
||||
return generative_->speak(
|
||||
model_.empty() ? Generative::default_tts_model() : model_, request.text,
|
||||
request.voice);
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,37 @@
|
||||
#pragma once
|
||||
|
||||
// Recognizer and Synthesizer over Gemini: audio understanding for speech to
|
||||
// text, Gemini TTS models for text to speech. Works with any credential the
|
||||
// Gemini endpoint accepts (AI Studio key, Vertex express key, OAuth).
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "google/generative.h"
|
||||
#include "google/voice_io.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
class GeminiRecognizer : public Recognizer {
|
||||
public:
|
||||
explicit GeminiRecognizer(std::shared_ptr<Generative> generative,
|
||||
std::string model = "");
|
||||
Transcript recognize(const RecognitionRequest& request) const override;
|
||||
|
||||
private:
|
||||
std::shared_ptr<Generative> generative_;
|
||||
std::string model_;
|
||||
};
|
||||
|
||||
class GeminiSynthesizer : public Synthesizer {
|
||||
public:
|
||||
explicit GeminiSynthesizer(std::shared_ptr<Generative> generative,
|
||||
std::string model = "");
|
||||
Audio synthesize(const SynthesisRequest& request) const override;
|
||||
|
||||
private:
|
||||
std::shared_ptr<Generative> generative_;
|
||||
std::string model_;
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,209 @@
|
||||
#include "google/generative.h"
|
||||
|
||||
#include "google/caps.h"
|
||||
#include "google/credentials.h"
|
||||
#include "google/endpoints.h"
|
||||
#include "google/wav.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
Generative::Generative(std::shared_ptr<GoogleClient> client,
|
||||
ServiceEndpoint endpoint)
|
||||
: Service(std::move(client), std::move(endpoint)) {}
|
||||
|
||||
std::string Generative::default_model() {
|
||||
std::string m = Credentials::env("GOOGLE_GENAI_MODEL");
|
||||
return m.empty() ? "gemini-3.5-flash" : m;
|
||||
}
|
||||
|
||||
std::string Generative::default_stt_model() {
|
||||
std::string m = Credentials::env("GOOGLE_STT_MODEL");
|
||||
return m.empty() ? default_model() : m;
|
||||
}
|
||||
|
||||
std::string Generative::default_tts_model() {
|
||||
std::string m = Credentials::env("GOOGLE_TTS_MODEL");
|
||||
return m.empty() ? "gemini-2.5-flash-tts" : m;
|
||||
}
|
||||
|
||||
Json Generative::generate_content(const std::string& model,
|
||||
const Json& body) const {
|
||||
return call("/models/" + model + ":generateContent", body);
|
||||
}
|
||||
|
||||
std::string Generative::generate(const GenerationRequest& request) const {
|
||||
if (request.messages.empty()) throw GoogleError::invalid("no messages");
|
||||
if (request.max_output_tokens < 1 ||
|
||||
request.max_output_tokens > Caps::kMaxOutputTokens)
|
||||
throw GoogleError::invalid("max_output_tokens outside 1.." +
|
||||
std::to_string(Caps::kMaxOutputTokens));
|
||||
|
||||
Json contents = Json::array();
|
||||
for (const ChatMessage& m : request.messages) {
|
||||
if (m.role != "user" && m.role != "model")
|
||||
throw GoogleError::invalid("role must be user or model");
|
||||
if (m.text.size() > Caps::kMaxMessageChars)
|
||||
throw GoogleError::invalid("message over " +
|
||||
std::to_string(Caps::kMaxMessageChars) +
|
||||
" characters");
|
||||
Json part = Json::object();
|
||||
part.set("text", Json(m.text));
|
||||
Json parts = Json::array();
|
||||
parts.push(part);
|
||||
Json content = Json::object();
|
||||
content.set("role", Json(m.role));
|
||||
content.set("parts", parts);
|
||||
contents.push(content);
|
||||
}
|
||||
Json gen = Json::object();
|
||||
gen.set("maxOutputTokens", Json(request.max_output_tokens));
|
||||
gen.set("temperature", Json(request.temperature));
|
||||
if (request.thinking_budget >= 0) {
|
||||
Json thinking = Json::object();
|
||||
thinking.set("thinkingBudget", Json(request.thinking_budget));
|
||||
gen.set("thinkingConfig", thinking);
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("contents", contents);
|
||||
body.set("generationConfig", gen);
|
||||
if (!request.system.empty()) {
|
||||
Json part = Json::object();
|
||||
part.set("text", Json(request.system));
|
||||
Json parts = Json::array();
|
||||
parts.push(part);
|
||||
Json sys = Json::object();
|
||||
sys.set("parts", parts);
|
||||
body.set("systemInstruction", sys);
|
||||
}
|
||||
|
||||
std::string model = request.model.empty() ? default_model() : request.model;
|
||||
Json reply = generate_content(model, body);
|
||||
|
||||
std::string text;
|
||||
const Json* candidates = reply.find("candidates");
|
||||
if (candidates && candidates->is_array() && !candidates->as_array().empty()) {
|
||||
const Json* content = candidates->as_array().front().find("content");
|
||||
const Json* parts = content ? content->find("parts") : nullptr;
|
||||
if (parts && parts->is_array())
|
||||
for (const Json& p : parts->as_array()) text += string_at(p, "text");
|
||||
}
|
||||
if (text.empty())
|
||||
throw GoogleError(502, "EMPTY_RESPONSE", "model returned no text",
|
||||
endpoint_.name);
|
||||
return text;
|
||||
}
|
||||
|
||||
std::string Generative::transcribe(const std::string& model,
|
||||
const std::string& audio,
|
||||
const std::string& mime,
|
||||
const std::string& language) const {
|
||||
if (audio.empty()) throw GoogleError::invalid("audio is empty");
|
||||
if (audio.size() > Caps::kMaxSttBytes)
|
||||
throw GoogleError::invalid("audio over " +
|
||||
std::to_string(Caps::kMaxSttBytes) + " bytes");
|
||||
Json inline_data = Json::object();
|
||||
inline_data.set("mimeType", Json(mime));
|
||||
inline_data.set("data", Json(security::crypto::Base64::encode(audio)));
|
||||
Json audio_part = Json::object();
|
||||
audio_part.set("inlineData", inline_data);
|
||||
Json text_part = Json::object();
|
||||
text_part.set("text",
|
||||
Json("Transcribe the speech in this audio verbatim. The "
|
||||
"language is probably " + language + ". Output only the "
|
||||
"transcript. If there is no intelligible speech, output "
|
||||
"nothing at all."));
|
||||
Json parts = Json::array();
|
||||
parts.push(audio_part);
|
||||
parts.push(text_part);
|
||||
Json content = Json::object();
|
||||
content.set("role", Json("user"));
|
||||
content.set("parts", parts);
|
||||
Json contents = Json::array();
|
||||
contents.push(content);
|
||||
Json gen = Json::object();
|
||||
gen.set("temperature", Json(0.0));
|
||||
Json no_thinking = Json::object();
|
||||
no_thinking.set("thinkingBudget", Json(0));
|
||||
gen.set("thinkingConfig", no_thinking);
|
||||
Json body = Json::object();
|
||||
body.set("contents", contents);
|
||||
body.set("generationConfig", gen);
|
||||
|
||||
Json reply = generate_content(model, body);
|
||||
std::string text;
|
||||
const Json* candidates = reply.find("candidates");
|
||||
if (candidates && candidates->is_array() && !candidates->as_array().empty()) {
|
||||
const Json* c = candidates->as_array().front().find("content");
|
||||
const Json* p = c ? c->find("parts") : nullptr;
|
||||
if (p && p->is_array())
|
||||
for (const Json& part : p->as_array()) text += string_at(part, "text");
|
||||
}
|
||||
std::size_t a = text.find_first_not_of(" \t\r\n");
|
||||
std::size_t b = text.find_last_not_of(" \t\r\n");
|
||||
return a == std::string::npos ? "" : text.substr(a, b - a + 1);
|
||||
}
|
||||
|
||||
Audio Generative::speak(const std::string& model, const std::string& text,
|
||||
const std::string& voice) const {
|
||||
if (text.empty()) throw GoogleError::invalid("text is empty");
|
||||
if (text.size() > Caps::kMaxTtsChars)
|
||||
throw GoogleError::invalid("text over " +
|
||||
std::to_string(Caps::kMaxTtsChars) + " characters");
|
||||
Json part = Json::object();
|
||||
part.set("text", Json(text));
|
||||
Json parts = Json::array();
|
||||
parts.push(part);
|
||||
Json content = Json::object();
|
||||
content.set("role", Json("user"));
|
||||
content.set("parts", parts);
|
||||
Json contents = Json::array();
|
||||
contents.push(content);
|
||||
Json prebuilt = Json::object();
|
||||
prebuilt.set("voiceName", Json(voice.empty() ? kDefaultTtsVoice : voice));
|
||||
Json voice_config = Json::object();
|
||||
voice_config.set("prebuiltVoiceConfig", prebuilt);
|
||||
Json speech = Json::object();
|
||||
speech.set("voiceConfig", voice_config);
|
||||
Json modalities = Json::array();
|
||||
modalities.push(Json("AUDIO"));
|
||||
Json gen = Json::object();
|
||||
gen.set("responseModalities", modalities);
|
||||
gen.set("speechConfig", speech);
|
||||
Json body = Json::object();
|
||||
body.set("contents", contents);
|
||||
body.set("generationConfig", gen);
|
||||
|
||||
Json reply = generate_content(model, body);
|
||||
const Json* candidates = reply.find("candidates");
|
||||
if (candidates && candidates->is_array() && !candidates->as_array().empty()) {
|
||||
const Json* c = candidates->as_array().front().find("content");
|
||||
const Json* p = c ? c->find("parts") : nullptr;
|
||||
if (p && p->is_array())
|
||||
for (const Json& pt : p->as_array()) {
|
||||
const Json* data = pt.find("inlineData");
|
||||
if (!data) continue;
|
||||
std::string pcm;
|
||||
if (!security::crypto::Base64::decode(string_at(*data, "data"), pcm) ||
|
||||
pcm.empty())
|
||||
continue;
|
||||
std::string mime = string_at(*data, "mimeType");
|
||||
Audio out;
|
||||
if (mime.find("L16") != std::string::npos ||
|
||||
mime.find("pcm") != std::string::npos) {
|
||||
out.bytes = pcm16_to_wav(pcm, sample_rate_from_mime(mime));
|
||||
out.mime = "audio/wav";
|
||||
} else {
|
||||
out.bytes = std::move(pcm);
|
||||
out.mime = mime.empty() ? "audio/wav" : mime;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
}
|
||||
throw GoogleError(502, "EMPTY_RESPONSE", "model returned no audio",
|
||||
endpoint_.name);
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,54 @@
|
||||
#pragma once
|
||||
|
||||
// Generative Language API (Gemini): models/{model}:generateContent.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/endpoints.h"
|
||||
#include "google/service.h"
|
||||
#include "google/voice_io.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
struct ChatMessage {
|
||||
std::string role; // "user" or "model"
|
||||
std::string text;
|
||||
};
|
||||
|
||||
struct GenerationRequest {
|
||||
std::string model; // Empty: GOOGLE_GENAI_MODEL or the built-in default.
|
||||
std::string system;
|
||||
std::vector<ChatMessage> messages;
|
||||
int max_output_tokens = 300;
|
||||
double temperature = 0.7;
|
||||
// Reasoning tokens count against max_output_tokens on thinking models and
|
||||
// can truncate a spoken reply. -1: model default; 0: no thinking.
|
||||
int thinking_budget = -1;
|
||||
};
|
||||
|
||||
class Generative : public Service {
|
||||
public:
|
||||
explicit Generative(std::shared_ptr<GoogleClient> client,
|
||||
ServiceEndpoint endpoint = endpoints::generative());
|
||||
|
||||
static std::string default_model();
|
||||
static std::string default_stt_model(); // GOOGLE_STT_MODEL, else default_model().
|
||||
static std::string default_tts_model(); // GOOGLE_TTS_MODEL, else gemini-2.5-flash-tts.
|
||||
static constexpr const char* kDefaultTtsVoice = "Kore";
|
||||
|
||||
std::string generate(const GenerationRequest& request) const;
|
||||
// Audio in, text out (Gemini audio understanding). Empty string: no speech.
|
||||
std::string transcribe(const std::string& model, const std::string& audio,
|
||||
const std::string& mime,
|
||||
const std::string& language) const;
|
||||
// Text in, spoken WAV out (Gemini TTS models return raw 24 kHz PCM).
|
||||
Audio speak(const std::string& model, const std::string& text,
|
||||
const std::string& voice) const;
|
||||
|
||||
private:
|
||||
clients::body::Json generate_content(const std::string& model,
|
||||
const clients::body::Json& body) const;
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,50 @@
|
||||
# Per-part fragment for the Google Cloud APIs (owned dir: src/google).
|
||||
# Included by the root Makefile, or run standalone:
|
||||
# make -f src/google/google.mk [google|google-test|google-clean]
|
||||
# Root wiring: `all` depends on `google`, `test` on `google-test`,
|
||||
# `clean` on `google-clean`.
|
||||
|
||||
ROOT ?= .
|
||||
CXX ?= g++
|
||||
GOOGLE_CXXFLAGS ?= -std=c++23 -Wall -Wextra -Werror -DMOLO_HAS_WOLFSSL
|
||||
CLIENTS_LIB ?= $(ROOT)/build/clients/libclients.a
|
||||
OPERABILITY_LIB ?= build/liboperability.a
|
||||
GOOGLE_INCS := -I$(ROOT)/src -I$(ROOT)/tests/src/clients
|
||||
|
||||
GOOGLE_SRCS := $(sort $(wildcard $(ROOT)/src/google/*.cpp)) \
|
||||
$(ROOT)/src/security/crypto/Base64.cpp
|
||||
GOOGLE_TEST_SRCS := $(sort $(wildcard $(ROOT)/tests/src/google/*.cpp))
|
||||
|
||||
GOOGLE_BUILD := $(ROOT)/build/google
|
||||
GOOGLE_OBJDIR := $(GOOGLE_BUILD)/objs
|
||||
GOOGLE_LIB := $(GOOGLE_BUILD)/libgoogle.a
|
||||
GOOGLE_TEST_BIN := $(ROOT)/build/google_test
|
||||
|
||||
GOOGLE_OBJS := $(patsubst $(ROOT)/%.cpp,$(GOOGLE_OBJDIR)/%.o,$(GOOGLE_SRCS))
|
||||
|
||||
# Header changes must rebuild dependents (a stale object with an old class
|
||||
# layout corrupts the heap).
|
||||
-include $(GOOGLE_OBJS:.o=.d)
|
||||
|
||||
.PHONY: google google-test google-clean
|
||||
|
||||
google: $(GOOGLE_LIB)
|
||||
|
||||
$(GOOGLE_OBJDIR)/%.o: $(ROOT)/%.cpp
|
||||
@mkdir -p $(dir $@)
|
||||
$(CXX) $(GOOGLE_CXXFLAGS) -MMD -MP -I$(ROOT)/src -c $< -o $@
|
||||
|
||||
$(GOOGLE_LIB): $(GOOGLE_OBJS)
|
||||
@mkdir -p $(dir $@)
|
||||
$(AR) rcs $@ $^
|
||||
|
||||
# The clients test harness (framework.h plus main.cpp) is reused, not copied.
|
||||
google-test: $(GOOGLE_TEST_BIN)
|
||||
$(GOOGLE_TEST_BIN)
|
||||
|
||||
$(GOOGLE_TEST_BIN): $(GOOGLE_OBJS) $(GOOGLE_TEST_SRCS) $(CLIENTS_LIB) $(OPERABILITY_LIB) $(ROOT)/tests/src/clients/main.cpp
|
||||
@mkdir -p $(dir $@)
|
||||
$(CXX) $(GOOGLE_CXXFLAGS) $(GOOGLE_INCS) $(GOOGLE_OBJS) $(GOOGLE_TEST_SRCS) $(ROOT)/tests/src/clients/main.cpp $(CLIENTS_LIB) $(OPERABILITY_LIB) -lwolfssl -o $@
|
||||
|
||||
google-clean:
|
||||
rm -rf $(GOOGLE_BUILD) $(GOOGLE_TEST_BIN)
|
||||
@@ -0,0 +1,30 @@
|
||||
#include "google/service.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
Service::Service(std::shared_ptr<GoogleClient> client, ServiceEndpoint endpoint)
|
||||
: client_(std::move(client)), endpoint_(std::move(endpoint)) {}
|
||||
|
||||
clients::body::Json Service::call(const std::string& path,
|
||||
const clients::body::Json& body) const {
|
||||
return client_->post_json(endpoint_, path, body);
|
||||
}
|
||||
|
||||
clients::body::Json Service::fetch(const std::string& path) const {
|
||||
return client_->get_json(endpoint_, path);
|
||||
}
|
||||
|
||||
std::string Service::string_at(const clients::body::Json& doc,
|
||||
const std::string& key,
|
||||
const std::string& fallback) {
|
||||
const auto* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : fallback;
|
||||
}
|
||||
|
||||
double Service::number_at(const clients::body::Json& doc,
|
||||
const std::string& key, double fallback) {
|
||||
const auto* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
return v && v->is_number() ? v->as_number() : fallback;
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,38 @@
|
||||
#pragma once
|
||||
|
||||
// Base class for every concrete Google service. A subclass owns its request
|
||||
// and response shapes; transport, auth and errors live in GoogleClient.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "google/client.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
class Service {
|
||||
public:
|
||||
Service(std::shared_ptr<GoogleClient> client, ServiceEndpoint endpoint);
|
||||
virtual ~Service() = default;
|
||||
|
||||
const ServiceEndpoint& endpoint() const { return endpoint_; }
|
||||
bool configured() const { return client_->configured(); }
|
||||
|
||||
protected:
|
||||
clients::body::Json call(const std::string& path,
|
||||
const clients::body::Json& body) const;
|
||||
clients::body::Json fetch(const std::string& path) const;
|
||||
|
||||
// Tolerant readers for response documents.
|
||||
static std::string string_at(const clients::body::Json& doc,
|
||||
const std::string& key,
|
||||
const std::string& fallback = "");
|
||||
static double number_at(const clients::body::Json& doc,
|
||||
const std::string& key, double fallback = 0.0);
|
||||
|
||||
std::shared_ptr<GoogleClient> client_;
|
||||
ServiceEndpoint endpoint_;
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,51 @@
|
||||
#include "google/speech_to_text.h"
|
||||
|
||||
#include "google/caps.h"
|
||||
#include "google/endpoints.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
SpeechToText::SpeechToText(std::shared_ptr<GoogleClient> client)
|
||||
: Service(std::move(client), endpoints::speech_to_text()) {}
|
||||
|
||||
Transcript SpeechToText::recognize(const RecognitionRequest& request) const {
|
||||
if (request.audio.empty()) throw GoogleError::invalid("audio is empty");
|
||||
if (request.audio.size() > Caps::kMaxSttBytes)
|
||||
throw GoogleError::invalid("audio over " +
|
||||
std::to_string(Caps::kMaxSttBytes) + " bytes");
|
||||
|
||||
Json config = Json::object();
|
||||
config.set("languageCode", Json(request.language));
|
||||
config.set("encoding", Json(speech_encoding_name(request.encoding)));
|
||||
config.set("enableAutomaticPunctuation", Json(request.punctuation));
|
||||
if (request.sample_rate_hz > 0)
|
||||
config.set("sampleRateHertz", Json(request.sample_rate_hz));
|
||||
Json audio = Json::object();
|
||||
audio.set("content", Json(security::crypto::Base64::encode(request.audio)));
|
||||
Json body = Json::object();
|
||||
body.set("config", config);
|
||||
body.set("audio", audio);
|
||||
|
||||
Json reply = call("/speech:recognize", body);
|
||||
Transcript out;
|
||||
const Json* results = reply.find("results");
|
||||
if (!results || !results->is_array()) return out;
|
||||
bool first = true;
|
||||
for (const Json& r : results->as_array()) {
|
||||
const Json* alts = r.find("alternatives");
|
||||
if (!alts || !alts->is_array() || alts->as_array().empty()) continue;
|
||||
const Json& best = alts->as_array().front();
|
||||
std::string piece = string_at(best, "transcript");
|
||||
if (piece.empty()) continue;
|
||||
if (!out.text.empty()) out.text += " ";
|
||||
out.text += piece;
|
||||
if (first) out.confidence = number_at(best, "confidence");
|
||||
first = false;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// Cloud Speech-to-Text: speech:recognize (synchronous, up to ~1 minute).
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "google/service.h"
|
||||
#include "google/voice_io.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
class SpeechToText : public Service, public Recognizer {
|
||||
public:
|
||||
explicit SpeechToText(std::shared_ptr<GoogleClient> client);
|
||||
|
||||
Transcript recognize(const RecognitionRequest& request) const override;
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,65 @@
|
||||
#include "google/text_to_speech.h"
|
||||
|
||||
#include "google/caps.h"
|
||||
#include "google/endpoints.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
TextToSpeech::TextToSpeech(std::shared_ptr<GoogleClient> client)
|
||||
: Service(std::move(client), endpoints::text_to_speech()) {}
|
||||
|
||||
Audio TextToSpeech::synthesize(const SynthesisRequest& request) const {
|
||||
if (request.text.empty()) throw GoogleError::invalid("text is empty");
|
||||
if (request.text.size() > Caps::kMaxTtsChars)
|
||||
throw GoogleError::invalid("text over " +
|
||||
std::to_string(Caps::kMaxTtsChars) + " characters");
|
||||
if (request.speaking_rate < 0.25 || request.speaking_rate > 4.0)
|
||||
throw GoogleError::invalid("speaking_rate outside 0.25..4.0");
|
||||
|
||||
Json input = Json::object();
|
||||
input.set(request.ssml ? "ssml" : "text", Json(request.text));
|
||||
Json voice = Json::object();
|
||||
voice.set("languageCode", Json(request.language));
|
||||
if (!request.voice.empty()) voice.set("name", Json(request.voice));
|
||||
Json audio_config = Json::object();
|
||||
audio_config.set("audioEncoding", Json(audio_encoding_name(request.encoding)));
|
||||
audio_config.set("speakingRate", Json(request.speaking_rate));
|
||||
Json body = Json::object();
|
||||
body.set("input", input);
|
||||
body.set("voice", voice);
|
||||
body.set("audioConfig", audio_config);
|
||||
|
||||
Json reply = call("/text:synthesize", body);
|
||||
Audio out;
|
||||
out.mime = audio_encoding_mime(request.encoding);
|
||||
if (!security::crypto::Base64::decode(string_at(reply, "audioContent"),
|
||||
out.bytes) ||
|
||||
out.bytes.empty())
|
||||
throw GoogleError(502, "BAD_RESPONSE", "missing or invalid audioContent",
|
||||
endpoint_.name);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<Voice> TextToSpeech::voices(const std::string& language) const {
|
||||
Json reply = fetch(language.empty() ? "/voices"
|
||||
: "/voices?languageCode=" + language);
|
||||
std::vector<Voice> out;
|
||||
const Json* list = reply.find("voices");
|
||||
if (!list || !list->is_array()) return out;
|
||||
for (const Json& v : list->as_array()) {
|
||||
Voice voice;
|
||||
voice.name = string_at(v, "name");
|
||||
voice.gender = string_at(v, "ssmlGender");
|
||||
if (const Json* codes = v.find("languageCodes");
|
||||
codes && codes->is_array())
|
||||
for (const Json& c : codes->as_array())
|
||||
if (c.is_string()) voice.languages.push_back(c.as_string());
|
||||
out.push_back(std::move(voice));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,27 @@
|
||||
#pragma once
|
||||
|
||||
// Cloud Text-to-Speech: text:synthesize and voices.list.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/service.h"
|
||||
#include "google/voice_io.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
struct Voice {
|
||||
std::string name;
|
||||
std::string gender;
|
||||
std::vector<std::string> languages;
|
||||
};
|
||||
|
||||
class TextToSpeech : public Service, public Synthesizer {
|
||||
public:
|
||||
explicit TextToSpeech(std::shared_ptr<GoogleClient> client);
|
||||
|
||||
Audio synthesize(const SynthesisRequest& request) const override;
|
||||
std::vector<Voice> voices(const std::string& language = "") const;
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,144 @@
|
||||
#include "google/voice.h"
|
||||
|
||||
#include "google/caps.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
GenerativeResponder::GenerativeResponder(std::shared_ptr<Generative> generative,
|
||||
std::string system)
|
||||
: generative_(std::move(generative)), system_(std::move(system)) {
|
||||
if (system_.empty())
|
||||
system_ =
|
||||
"You are a friendly voice assistant in a live spoken conversation. "
|
||||
"Answer in the language of the user, in one to three short sentences, "
|
||||
"plain text only: no markdown, lists, or emoji.";
|
||||
}
|
||||
|
||||
std::string GenerativeResponder::reply(const std::vector<ChatMessage>& history,
|
||||
const std::string& language) {
|
||||
GenerationRequest request;
|
||||
request.system = system_ + " Conversation language code: " + language + ".";
|
||||
request.messages = history;
|
||||
request.thinking_budget = 0; // Thinking would eat the spoken reply's budget.
|
||||
return generative_->generate(request);
|
||||
}
|
||||
|
||||
bool VoiceSessions::valid_id(const std::string& id) {
|
||||
if (id.empty() || id.size() > Caps::kMaxConversationId) return false;
|
||||
for (char c : id) {
|
||||
bool ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') ||
|
||||
(c >= '0' && c <= '9') || c == '-' || c == '_';
|
||||
if (!ok) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
std::vector<ChatMessage> VoiceSessions::append(const std::string& id,
|
||||
ChatMessage message) {
|
||||
if (!valid_id(id)) throw GoogleError::invalid("bad conversation id");
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(id);
|
||||
if (it == entries_.end()) {
|
||||
if (entries_.size() >= Caps::kMaxConversations) {
|
||||
entries_.erase(order_.front());
|
||||
order_.pop_front();
|
||||
}
|
||||
order_.push_back(id);
|
||||
Entry entry;
|
||||
entry.order = std::prev(order_.end());
|
||||
it = entries_.emplace(id, std::move(entry)).first;
|
||||
} else {
|
||||
order_.splice(order_.end(), order_, it->second.order);
|
||||
}
|
||||
auto& history = it->second.history;
|
||||
history.push_back(std::move(message));
|
||||
while (history.size() > Caps::kMaxHistoryTurns * 2) history.pop_front();
|
||||
// Gemini requires the first message to be from the user.
|
||||
while (!history.empty() && history.front().role != "user")
|
||||
history.pop_front();
|
||||
return std::vector<ChatMessage>(history.begin(), history.end());
|
||||
}
|
||||
|
||||
void VoiceSessions::drop_last(const std::string& id) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(id);
|
||||
if (it != entries_.end() && !it->second.history.empty())
|
||||
it->second.history.pop_back();
|
||||
}
|
||||
|
||||
void VoiceSessions::reset(const std::string& id) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(id);
|
||||
if (it == entries_.end()) return;
|
||||
order_.erase(it->second.order);
|
||||
entries_.erase(it);
|
||||
}
|
||||
|
||||
std::size_t VoiceSessions::size() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return entries_.size();
|
||||
}
|
||||
|
||||
std::size_t VoiceSessions::history_size(const std::string& id) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(id);
|
||||
return it == entries_.end() ? 0 : it->second.history.size();
|
||||
}
|
||||
|
||||
VoicePipeline::VoicePipeline(std::shared_ptr<Recognizer> stt,
|
||||
std::shared_ptr<Responder> responder,
|
||||
std::shared_ptr<Synthesizer> tts,
|
||||
std::shared_ptr<VoiceSessions> sessions)
|
||||
: stt_(std::move(stt)),
|
||||
responder_(std::move(responder)),
|
||||
tts_(std::move(tts)),
|
||||
sessions_(std::move(sessions)) {}
|
||||
|
||||
VoiceTurn VoicePipeline::turn(const std::string& id,
|
||||
const RecognitionRequest& heard,
|
||||
const std::string& voice) {
|
||||
if (!VoiceSessions::valid_id(id))
|
||||
throw GoogleError::invalid("bad conversation id");
|
||||
Transcript transcript = stt_->recognize(heard);
|
||||
if (transcript.text.empty()) {
|
||||
VoiceTurn silent;
|
||||
silent.language = heard.language;
|
||||
return silent;
|
||||
}
|
||||
return respond(id, transcript.text, heard.language, voice);
|
||||
}
|
||||
|
||||
VoiceTurn VoicePipeline::turn_text(const std::string& id,
|
||||
const std::string& text,
|
||||
const std::string& language,
|
||||
const std::string& voice) {
|
||||
if (!VoiceSessions::valid_id(id))
|
||||
throw GoogleError::invalid("bad conversation id");
|
||||
if (text.empty()) throw GoogleError::invalid("text is empty");
|
||||
return respond(id, text, language, voice);
|
||||
}
|
||||
|
||||
VoiceTurn VoicePipeline::respond(const std::string& id, const std::string& text,
|
||||
const std::string& language,
|
||||
const std::string& voice) {
|
||||
VoiceTurn out;
|
||||
out.transcript = text;
|
||||
out.language = language;
|
||||
std::string clipped = text.substr(0, Caps::kMaxMessageChars);
|
||||
auto history = sessions_->append(id, {"user", clipped});
|
||||
try {
|
||||
out.reply = responder_->reply(history, language);
|
||||
SynthesisRequest speak;
|
||||
speak.text = out.reply.substr(0, Caps::kMaxTtsChars);
|
||||
speak.language = language;
|
||||
speak.voice = voice;
|
||||
out.audio = tts_->synthesize(speak);
|
||||
} catch (...) {
|
||||
sessions_->drop_last(id); // A failed turn leaves no half conversation.
|
||||
throw;
|
||||
}
|
||||
sessions_->append(id, {"model", out.reply.substr(0, Caps::kMaxMessageChars)});
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,100 @@
|
||||
#pragma once
|
||||
|
||||
// Voice conversation: recognize -> respond -> synthesize, with bounded
|
||||
// per-conversation history. Responder is the seam: Gemini today, anything
|
||||
// else tomorrow.
|
||||
|
||||
#include <deque>
|
||||
#include <list>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/generative.h"
|
||||
#include "google/speech_to_text.h"
|
||||
#include "google/text_to_speech.h"
|
||||
#include "google/voice_io.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
class Responder {
|
||||
public:
|
||||
virtual ~Responder() = default;
|
||||
// history ends with the new user message.
|
||||
virtual std::string reply(const std::vector<ChatMessage>& history,
|
||||
const std::string& language) = 0;
|
||||
};
|
||||
|
||||
class GenerativeResponder : public Responder {
|
||||
public:
|
||||
explicit GenerativeResponder(std::shared_ptr<Generative> generative,
|
||||
std::string system = "");
|
||||
std::string reply(const std::vector<ChatMessage>& history,
|
||||
const std::string& language) override;
|
||||
|
||||
private:
|
||||
std::shared_ptr<Generative> generative_;
|
||||
std::string system_;
|
||||
};
|
||||
|
||||
// Bounded conversation memory. Both history length and conversation count
|
||||
// are capped; oldest entries are evicted first.
|
||||
class VoiceSessions {
|
||||
public:
|
||||
static bool valid_id(const std::string& id);
|
||||
|
||||
// Appends and returns the full history snapshot (including this message).
|
||||
std::vector<ChatMessage> append(const std::string& id, ChatMessage message);
|
||||
// Removes the last message (used to roll back a failed turn).
|
||||
void drop_last(const std::string& id);
|
||||
void reset(const std::string& id);
|
||||
std::size_t size() const;
|
||||
std::size_t history_size(const std::string& id) const;
|
||||
|
||||
private:
|
||||
struct Entry {
|
||||
std::deque<ChatMessage> history;
|
||||
std::list<std::string>::iterator order;
|
||||
};
|
||||
mutable std::mutex mutex_;
|
||||
std::map<std::string, Entry> entries_;
|
||||
std::list<std::string> order_; // Front is oldest.
|
||||
};
|
||||
|
||||
struct VoiceTurn {
|
||||
std::string transcript; // Empty: nothing was heard.
|
||||
std::string reply;
|
||||
Audio audio;
|
||||
std::string language;
|
||||
};
|
||||
|
||||
class VoicePipeline {
|
||||
public:
|
||||
VoicePipeline(std::shared_ptr<Recognizer> stt,
|
||||
std::shared_ptr<Responder> responder,
|
||||
std::shared_ptr<Synthesizer> tts,
|
||||
std::shared_ptr<VoiceSessions> sessions);
|
||||
|
||||
// Full turn from audio.
|
||||
VoiceTurn turn(const std::string& conversation_id,
|
||||
const RecognitionRequest& heard, const std::string& voice = "");
|
||||
// Text turn (typed instead of spoken): respond and synthesize.
|
||||
VoiceTurn turn_text(const std::string& conversation_id,
|
||||
const std::string& text, const std::string& language,
|
||||
const std::string& voice = "");
|
||||
|
||||
VoiceSessions& sessions() { return *sessions_; }
|
||||
|
||||
private:
|
||||
VoiceTurn respond(const std::string& id, const std::string& text,
|
||||
const std::string& language, const std::string& voice);
|
||||
|
||||
std::shared_ptr<Recognizer> stt_;
|
||||
std::shared_ptr<Responder> responder_;
|
||||
std::shared_ptr<Synthesizer> tts_;
|
||||
std::shared_ptr<VoiceSessions> sessions_;
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,46 @@
|
||||
#include "google/voice_backend.h"
|
||||
|
||||
#include "google/credentials.h"
|
||||
#include "google/gemini_voice.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
const char* voice_backend_name(VoiceBackend backend) {
|
||||
switch (backend) {
|
||||
case VoiceBackend::kCloud: return "cloud";
|
||||
case VoiceBackend::kGemini: return "gemini";
|
||||
case VoiceBackend::kVertex: return "vertex";
|
||||
}
|
||||
return "cloud";
|
||||
}
|
||||
|
||||
VoiceBackend voice_backend_from_env() {
|
||||
std::string pick = Credentials::env("GOOGLE_API_BACKEND");
|
||||
if (pick == "cloud") return VoiceBackend::kCloud;
|
||||
if (pick == "gemini") return VoiceBackend::kGemini;
|
||||
if (pick == "vertex") return VoiceBackend::kVertex;
|
||||
return Credentials::env("GOOGLE_API_KEY").rfind("AQ.", 0) == 0
|
||||
? VoiceBackend::kVertex
|
||||
: VoiceBackend::kCloud;
|
||||
}
|
||||
|
||||
VoiceParts make_voice_parts(std::shared_ptr<GoogleClient> client,
|
||||
VoiceBackend backend) {
|
||||
VoiceParts parts;
|
||||
if (backend == VoiceBackend::kCloud) {
|
||||
auto generative = std::make_shared<Generative>(client);
|
||||
parts.recognizer = std::make_shared<SpeechToText>(client);
|
||||
parts.synthesizer = std::make_shared<TextToSpeech>(client);
|
||||
parts.responder = std::make_shared<GenerativeResponder>(generative);
|
||||
return parts;
|
||||
}
|
||||
auto generative = std::make_shared<Generative>(
|
||||
client, backend == VoiceBackend::kVertex ? endpoints::vertex_generative()
|
||||
: endpoints::generative());
|
||||
parts.recognizer = std::make_shared<GeminiRecognizer>(generative);
|
||||
parts.synthesizer = std::make_shared<GeminiSynthesizer>(generative);
|
||||
parts.responder = std::make_shared<GenerativeResponder>(generative);
|
||||
return parts;
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,33 @@
|
||||
#pragma once
|
||||
|
||||
// Picks the three voice parts for a credential. One switch, three backends:
|
||||
// kCloud classic Cloud Text-to-Speech + Speech-to-Text (need OAuth or an
|
||||
// old-style key) with Gemini on generativelanguage for replies
|
||||
// kGemini Gemini audio in/out on generativelanguage (AI Studio key)
|
||||
// kVertex Gemini audio in/out on Vertex AI (express key, "AQ." prefix)
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "google/client.h"
|
||||
#include "google/voice.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
enum class VoiceBackend { kCloud, kGemini, kVertex };
|
||||
|
||||
const char* voice_backend_name(VoiceBackend backend);
|
||||
// GOOGLE_API_BACKEND (cloud|gemini|vertex) wins; else keys starting with
|
||||
// "AQ." are Vertex express keys; everything else is classic cloud.
|
||||
VoiceBackend voice_backend_from_env();
|
||||
|
||||
struct VoiceParts {
|
||||
std::shared_ptr<Recognizer> recognizer;
|
||||
std::shared_ptr<Responder> responder;
|
||||
std::shared_ptr<Synthesizer> synthesizer;
|
||||
};
|
||||
|
||||
VoiceParts make_voice_parts(std::shared_ptr<GoogleClient> client,
|
||||
VoiceBackend backend);
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,56 @@
|
||||
#include "google/voice_io.h"
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
const char* audio_encoding_name(AudioEncoding encoding) {
|
||||
switch (encoding) {
|
||||
case AudioEncoding::kMp3: return "MP3";
|
||||
case AudioEncoding::kLinear16: return "LINEAR16";
|
||||
case AudioEncoding::kOggOpus: return "OGG_OPUS";
|
||||
}
|
||||
return "MP3";
|
||||
}
|
||||
|
||||
const char* audio_encoding_mime(AudioEncoding encoding) {
|
||||
switch (encoding) {
|
||||
case AudioEncoding::kMp3: return "audio/mpeg";
|
||||
case AudioEncoding::kLinear16: return "audio/wav";
|
||||
case AudioEncoding::kOggOpus: return "audio/ogg";
|
||||
}
|
||||
return "audio/mpeg";
|
||||
}
|
||||
|
||||
const char* speech_encoding_name(SpeechEncoding encoding) {
|
||||
switch (encoding) {
|
||||
case SpeechEncoding::kWebmOpus: return "WEBM_OPUS";
|
||||
case SpeechEncoding::kOggOpus: return "OGG_OPUS";
|
||||
case SpeechEncoding::kFlac: return "FLAC";
|
||||
case SpeechEncoding::kLinear16: return "LINEAR16";
|
||||
case SpeechEncoding::kMp3: return "MP3";
|
||||
}
|
||||
return "WEBM_OPUS";
|
||||
}
|
||||
|
||||
const char* speech_encoding_mime(SpeechEncoding encoding) {
|
||||
switch (encoding) {
|
||||
case SpeechEncoding::kWebmOpus: return "audio/webm";
|
||||
case SpeechEncoding::kOggOpus: return "audio/ogg";
|
||||
case SpeechEncoding::kFlac: return "audio/flac";
|
||||
case SpeechEncoding::kLinear16: return "audio/wav";
|
||||
case SpeechEncoding::kMp3: return "audio/mpeg";
|
||||
}
|
||||
return "audio/webm";
|
||||
}
|
||||
|
||||
bool speech_encoding_from_mime(const std::string& mime, SpeechEncoding& out) {
|
||||
auto has = [&](const char* s) { return mime.find(s) != std::string::npos; };
|
||||
if (has("webm")) out = SpeechEncoding::kWebmOpus;
|
||||
else if (has("ogg")) out = SpeechEncoding::kOggOpus;
|
||||
else if (has("flac")) out = SpeechEncoding::kFlac;
|
||||
else if (has("wav") || has("l16") || has("pcm")) out = SpeechEncoding::kLinear16;
|
||||
else if (has("mpeg") || has("mp3")) out = SpeechEncoding::kMp3;
|
||||
else return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,63 @@
|
||||
#pragma once
|
||||
|
||||
// The two speech seams of a voice pipeline. Backends implement them:
|
||||
// classic Cloud TTS/STT (TextToSpeech, SpeechToText) or Gemini audio
|
||||
// (GeminiSynthesizer, GeminiRecognizer). Callers never know which.
|
||||
|
||||
#include <string>
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
enum class AudioEncoding { kMp3, kLinear16, kOggOpus };
|
||||
|
||||
const char* audio_encoding_name(AudioEncoding encoding); // Google enum name.
|
||||
const char* audio_encoding_mime(AudioEncoding encoding);
|
||||
|
||||
struct SynthesisRequest {
|
||||
std::string text;
|
||||
bool ssml = false; // Classic Cloud TTS only.
|
||||
std::string language = "nl-NL";
|
||||
std::string voice; // Empty: backend default.
|
||||
AudioEncoding encoding = AudioEncoding::kMp3; // Classic Cloud TTS only.
|
||||
double speaking_rate = 1.0; // 0.25 .. 4.0, classic Cloud TTS only.
|
||||
};
|
||||
|
||||
struct Audio {
|
||||
std::string bytes;
|
||||
std::string mime;
|
||||
};
|
||||
|
||||
enum class SpeechEncoding { kWebmOpus, kOggOpus, kFlac, kLinear16, kMp3 };
|
||||
|
||||
const char* speech_encoding_name(SpeechEncoding encoding); // Google enum name.
|
||||
const char* speech_encoding_mime(SpeechEncoding encoding); // Content type.
|
||||
// Maps a browser/content type ("audio/webm;codecs=opus", "audio/ogg", ...)
|
||||
// to an encoding. Returns false when unknown.
|
||||
bool speech_encoding_from_mime(const std::string& mime, SpeechEncoding& out);
|
||||
|
||||
struct RecognitionRequest {
|
||||
std::string audio; // Raw bytes.
|
||||
SpeechEncoding encoding = SpeechEncoding::kWebmOpus;
|
||||
std::string language = "nl-NL";
|
||||
int sample_rate_hz = 0; // 0: taken from the container header.
|
||||
bool punctuation = true;
|
||||
};
|
||||
|
||||
struct Transcript {
|
||||
std::string text;
|
||||
double confidence = 0.0; // 0 when the backend does not report one.
|
||||
};
|
||||
|
||||
class Synthesizer {
|
||||
public:
|
||||
virtual ~Synthesizer() = default;
|
||||
virtual Audio synthesize(const SynthesisRequest& request) const = 0;
|
||||
};
|
||||
|
||||
class Recognizer {
|
||||
public:
|
||||
virtual ~Recognizer() = default;
|
||||
virtual Transcript recognize(const RecognitionRequest& request) const = 0;
|
||||
};
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,46 @@
|
||||
#include "google/wav.h"
|
||||
|
||||
#include <cstdint>
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
namespace {
|
||||
void put16(std::string& out, std::uint16_t v) {
|
||||
out.push_back(static_cast<char>(v & 0xff));
|
||||
out.push_back(static_cast<char>((v >> 8) & 0xff));
|
||||
}
|
||||
void put32(std::string& out, std::uint32_t v) {
|
||||
for (int i = 0; i < 4; ++i) out.push_back(static_cast<char>((v >> (8 * i)) & 0xff));
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::string pcm16_to_wav(const std::string& pcm, int sample_rate_hz,
|
||||
int channels) {
|
||||
std::string out;
|
||||
out.reserve(44 + pcm.size());
|
||||
out += "RIFF";
|
||||
put32(out, static_cast<std::uint32_t>(36 + pcm.size()));
|
||||
out += "WAVEfmt ";
|
||||
put32(out, 16);
|
||||
put16(out, 1); // PCM
|
||||
put16(out, static_cast<std::uint16_t>(channels));
|
||||
put32(out, static_cast<std::uint32_t>(sample_rate_hz));
|
||||
put32(out, static_cast<std::uint32_t>(sample_rate_hz * channels * 2));
|
||||
put16(out, static_cast<std::uint16_t>(channels * 2));
|
||||
put16(out, 16);
|
||||
out += "data";
|
||||
put32(out, static_cast<std::uint32_t>(pcm.size()));
|
||||
out += pcm;
|
||||
return out;
|
||||
}
|
||||
|
||||
int sample_rate_from_mime(const std::string& mime, int fallback) {
|
||||
std::size_t pos = mime.find("rate=");
|
||||
if (pos == std::string::npos) return fallback;
|
||||
int rate = 0;
|
||||
for (std::size_t i = pos + 5; i < mime.size() && mime[i] >= '0' && mime[i] <= '9'; ++i)
|
||||
rate = rate * 10 + (mime[i] - '0');
|
||||
return rate >= 8000 && rate <= 96000 ? rate : fallback;
|
||||
}
|
||||
|
||||
} // namespace molo::google
|
||||
@@ -0,0 +1,17 @@
|
||||
#pragma once
|
||||
|
||||
// Gemini TTS returns raw 16-bit PCM; browsers need a container.
|
||||
|
||||
#include <string>
|
||||
|
||||
namespace molo::google {
|
||||
|
||||
// Wraps little-endian mono/stereo 16-bit PCM in a RIFF/WAVE header.
|
||||
std::string pcm16_to_wav(const std::string& pcm, int sample_rate_hz,
|
||||
int channels = 1);
|
||||
|
||||
// Sample rate from a mime like "audio/L16;codec=pcm;rate=24000"; fallback
|
||||
// when absent.
|
||||
int sample_rate_from_mime(const std::string& mime, int fallback = 24000);
|
||||
|
||||
} // namespace molo::google
|
||||
Reference in New Issue
Block a user