feat(clients): google cloud voice: generic client, TTS, STT, Gemini audio, live voice page
src/google: GoogleClient over molo::clients (key in x-goog-api-key, never in
URLs), Service base, TextToSpeech, SpeechToText, Generative (replies, audio
in, audio out), Recognizer/Synthesizer seams with cloud, gemini and vertex
backends, VoicePipeline with bounded history. Example: /voice live
conversation page, /voice/{say,hear,turn,reset}, /google/status. Keys load
from env, .env.json.local, .env.json. Paid routes are excluded from locust.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
+8
-2
@@ -14,7 +14,7 @@ Open `http://127.0.0.1:8080/` and click through the matrix; `/stats` has the
|
||||
charts (Chart.js is embedded in the binary, no CDN). Narrow the
|
||||
window to 360px: the top nav collapses into a dropdown menu.
|
||||
|
||||
Run the suite (105 route-level integration tests, real stack, no mocks):
|
||||
Run the suite (111 route-level integration tests, real stack, no mocks):
|
||||
|
||||
```sh
|
||||
make -f example/example.mk example-test
|
||||
@@ -114,6 +114,12 @@ on a missing row.
|
||||
|
||||
| statistics | `/stats` | `tests/example/stats/page_test.cpp` | `plans/02-observability.md` |
|
||||
| statistics data | `/stats/data` | `tests/example/stats/data_test.cpp` | `plans/02-observability.md` |
|
||||
| google status | `/google/status` | `tests/example/google/status_test.cpp` | `src/google/SPEC.md` |
|
||||
| voice conversation page | `/voice` | `tests/example/voice/page_test.cpp` | `src/google/SPEC.md` |
|
||||
| voice text to speech | `/voice/say` | `tests/example/voice/say_test.cpp` | `src/google/SPEC.md` |
|
||||
| voice speech to text | `/voice/hear` | `tests/example/voice/hear_test.cpp` | `src/google/SPEC.md` |
|
||||
| voice turn | `/voice/turn` | `tests/example/voice/turn_test.cpp` | `src/google/SPEC.md` |
|
||||
| voice reset | `/voice/reset` | `tests/example/voice/reset_test.cpp` | `src/google/SPEC.md` |
|
||||
|
||||
### Extended features
|
||||
|
||||
@@ -203,7 +209,7 @@ example/
|
||||
fw/ # skeletal framework public API (namespace molo::example::fw)
|
||||
app/ # demo routes (namespace molo::example::app)
|
||||
static/ # disk static assets (responsive CSS, tiny JS)
|
||||
assets/ # embedded at build time (Chart.js, stats.js) via src/cli/embed
|
||||
assets/ # embedded at build time (Chart.js, stats.js, voice.js) via src/cli/embed
|
||||
tools/ # build helper wrapping the embedder
|
||||
tests/example/ # route-level integration tests mirroring URLs
|
||||
```
|
||||
|
||||
+26
-5
@@ -45,6 +45,20 @@ owning sibling plan and the file that will close it.
|
||||
builder, singleflight, ttl store). Each is a skeleton of the real `src/`
|
||||
module; per-module gaps are in the header of its `fw/` file.
|
||||
|
||||
16. Voice (`src/google`, the real framework part, not a skeleton):
|
||||
`GET /voice` is a hands-free conversation page. The browser detects
|
||||
speech from the microphone level, records one utterance, posts it to
|
||||
`POST /voice/turn`, plays the spoken answer, and listens again;
|
||||
speaking over the answer interrupts it. The server runs Google
|
||||
speech-to-text, a Gemini reply and Google text-to-speech per turn, with
|
||||
bounded per-conversation history. `POST /voice/say` and `/voice/hear`
|
||||
expose the two speech services alone; `GET /google/status` shows
|
||||
credential state and known services (never the key). No
|
||||
`GOOGLE_API_KEY`: 503 `google_not_configured`. Upstream failure: 502
|
||||
`google_error` with Google's status. Bad input: 422 or 415 before any
|
||||
network call. Routes and tests use a scripted transport; the key lives
|
||||
only in the `x-goog-api-key` header.
|
||||
|
||||
## Caps
|
||||
|
||||
All ceilings live as literals in `example/fw/Caps.hpp`: 1 MiB request body,
|
||||
@@ -76,11 +90,13 @@ auth, 403 bad credentials, 404 unknown, 405 wrong method, 406 no format,
|
||||
|
||||
## Isolation
|
||||
|
||||
May depend on: C++23 stdlib, POSIX sockets, `example/fw/` only. Build-time
|
||||
only: the embedder sources in `src/cli/embed` (asset embedding). May NOT
|
||||
depend on: any `src/` tree (does not exist yet in this worktree), wolfSSL
|
||||
(stub behind `ITlsContext`), real Postgres/SQLite (in-memory doubles),
|
||||
real network except loopback. Namespaces mirror directories:
|
||||
May depend on: C++23 stdlib, POSIX sockets, `example/fw/`, and the real
|
||||
framework parts it links on purpose: `src/google` over `src/clients` (HTTPS
|
||||
via wolfSSL) and `src/operability`. Build-time only: the embedder sources in
|
||||
`src/cli/embed` (asset embedding). May NOT
|
||||
depend on: other `src/` trees, real Postgres/SQLite (in-memory doubles),
|
||||
real network except Google through `src/google` (tests script that
|
||||
transport; only the opt-in `MOLO_LIVE_NET=1` tests touch the network). Namespaces mirror directories:
|
||||
`molo::example::fw` and `molo::example::app`.
|
||||
|
||||
## Gaps
|
||||
@@ -98,6 +114,11 @@ real network except loopback. Namespaces mirror directories:
|
||||
- Full validation coercions, template sandbox, DI container, flash
|
||||
messages: owner 05-request-routing.md, files `example/fw/Validate.*`,
|
||||
`example/fw/Templates.*`, `example/app/App.*`.
|
||||
- Voice turns are request/response over REST (about 2 to 4 s per turn).
|
||||
The loopback server accepts sequentially, so one in-flight turn blocks
|
||||
other requests meanwhile. Streaming recognition and the Gemini Live API:
|
||||
owner `src/google/SPEC.md`, gaps section. Owner of concurrent accept:
|
||||
01-runtime.md, `example/fw/Server.*`.
|
||||
- Server accepts sequentially and stops after 512 connections (cap above);
|
||||
the stats page shows connections left. Owner: 01-runtime.md,
|
||||
`example/fw/Server.*`.
|
||||
|
||||
@@ -67,6 +67,10 @@ std::shared_ptr<fw::Application> build_application(Services& services) {
|
||||
register_resilience_demo(resilience, services);
|
||||
app->include_router("", resilience);
|
||||
|
||||
fw::Router voice;
|
||||
register_voice_demo(voice, services);
|
||||
app->include_router("", voice);
|
||||
|
||||
register_chain_demo(app, services);
|
||||
return app;
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
// Gap: real molo::di::Container and plugin registry in 05-request-routing.md.
|
||||
|
||||
#include <map>
|
||||
#include <optional>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
@@ -18,6 +19,9 @@
|
||||
#include "../fw/Observ.hpp"
|
||||
#include "../fw/Stats.hpp"
|
||||
#include "../fw/Templates.hpp"
|
||||
#include "google/client.h"
|
||||
#include "google/voice.h"
|
||||
#include "google/voice_backend.h"
|
||||
|
||||
namespace molo::example::app {
|
||||
|
||||
@@ -49,6 +53,12 @@ struct Services {
|
||||
std::shared_ptr<fw::OpenApi> openapi = std::make_shared<fw::OpenApi>();
|
||||
std::shared_ptr<fw::Environment> templates =
|
||||
std::make_shared<fw::Environment>();
|
||||
// Google Cloud client (null: built from GOOGLE_API_KEY on first use) and
|
||||
// the voice pipeline wired on it. Tests inject a scripted client.
|
||||
std::shared_ptr<google::GoogleClient> google;
|
||||
std::shared_ptr<google::VoicePipeline> voice;
|
||||
// Unset: chosen from GOOGLE_API_BACKEND / the key prefix (see src/google).
|
||||
std::optional<google::VoiceBackend> voice_backend;
|
||||
// Feature matrix rows (feature, route) shown on the index page. Each
|
||||
// register_*_demo adds its own rows with add_feature().
|
||||
std::vector<std::pair<std::string, std::string>> features;
|
||||
@@ -76,6 +86,7 @@ void register_security_extra_demo(fw::Router& router, Services& services);
|
||||
void register_platform_demo(fw::Router& router, Services& services);
|
||||
void register_enterprise_demo(fw::Router& router, Services& services);
|
||||
void register_resilience_demo(fw::Router& router, Services& services);
|
||||
void register_voice_demo(fw::Router& router, Services& services);
|
||||
|
||||
void load_templates(Services& services);
|
||||
void load_embedded(Services& services);
|
||||
|
||||
@@ -22,6 +22,7 @@ const char kBase[] = R"BASE(<!DOCTYPE html>
|
||||
<a href="/templates/page">Templates</a>
|
||||
<a href="/forms/contact">Forms</a>
|
||||
<a href="/stats">Stats</a>
|
||||
<a href="/voice">Voice</a>
|
||||
<a href="/docs">Docs</a>
|
||||
<a href="/health">Health</a>
|
||||
</nav>
|
||||
@@ -32,6 +33,7 @@ const char kBase[] = R"BASE(<!DOCTYPE html>
|
||||
<a href="/templates/page">Templates</a>
|
||||
<a href="/forms/contact">Forms</a>
|
||||
<a href="/stats">Stats</a>
|
||||
<a href="/voice">Voice</a>
|
||||
<a href="/docs">Docs</a>
|
||||
<a href="/health">Health</a>
|
||||
</nav>
|
||||
@@ -168,6 +170,37 @@ after its connection cap, so auto refresh is off by default.</p>
|
||||
{% endblock %}
|
||||
)STATS";
|
||||
|
||||
const char kVoice[] = R"VOICE({% extends "base.html" %}
|
||||
{% block content %}
|
||||
<h1>Voice conversation</h1>
|
||||
<p class="hint">Talk, pause, and the assistant answers out loud. Google
|
||||
speech-to-text, Gemini, and Google text-to-speech run on the server.
|
||||
Speak over the answer to interrupt it.</p>
|
||||
<p id="v-config" class="hint">Checking Google credentials...</p>
|
||||
<section class="voice">
|
||||
<div class="voice-controls">
|
||||
<button id="v-toggle" type="button" class="primary" disabled>Start conversation</button>
|
||||
<label class="inline">Language
|
||||
<select id="v-lang">
|
||||
<option value="nl-NL">Nederlands</option>
|
||||
<option value="en-US">English</option>
|
||||
<option value="de-DE">Deutsch</option>
|
||||
</select></label>
|
||||
</div>
|
||||
<div class="voice-state"><span id="v-state" class="pill">idle</span>
|
||||
<div class="meter"><div id="v-level"></div></div></div>
|
||||
<p id="v-error" class="error" role="alert"></p>
|
||||
<ol id="v-log" class="voice-log"></ol>
|
||||
<form id="v-form" class="voice-form">
|
||||
<input id="v-text" type="text" maxlength="500" placeholder="Or type a message" autocomplete="off">
|
||||
<button type="submit">Send</button>
|
||||
<button id="v-reset" type="button">Reset</button>
|
||||
</form>
|
||||
</section>
|
||||
<script src="/embedded/voice.js"></script>
|
||||
{% endblock %}
|
||||
)VOICE";
|
||||
|
||||
const char kDocs[] = R"DOCS({% extends "base.html" %}
|
||||
{% block content %}
|
||||
<h1>API docs</h1>
|
||||
@@ -198,6 +231,7 @@ void load_templates(Services& services) {
|
||||
services.templates->add_template("page.html", kPage);
|
||||
services.templates->add_template("form_page.html", kFormPage);
|
||||
services.templates->add_template("docs.html", kDocs);
|
||||
services.templates->add_template("voice.html", kVoice);
|
||||
services.templates->add_template("stats.html", kStats);
|
||||
services.templates->add_template("error.html", kError);
|
||||
services.templates->add_template("nav_hint.html", kNavHint);
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
#include "App.hpp"
|
||||
|
||||
#include <cctype>
|
||||
#include <cstdlib>
|
||||
|
||||
#include "../fw/Router.hpp"
|
||||
#include "../fw/Templates.hpp"
|
||||
#include "../fw/Validate.hpp"
|
||||
#include "google/credentials.h"
|
||||
#include "google/endpoints.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::example::app {
|
||||
|
||||
// Google Cloud voice over the real src/google part: text-to-speech,
|
||||
// speech-to-text, a Gemini reply, and the live conversation page.
|
||||
// Gap: turns are request/response over REST; streaming recognition and the
|
||||
// Gemini Live API are listed in src/google/SPEC.md.
|
||||
|
||||
namespace {
|
||||
|
||||
std::string env_or(const char* name, const char* fallback) {
|
||||
std::string v = google::Credentials::env(name);
|
||||
return v.empty() ? std::string(fallback) : v;
|
||||
}
|
||||
|
||||
// "nl-NL", "en-US": two or three lowercase letters, dash, two uppercase.
|
||||
bool valid_language(const std::string& s) {
|
||||
std::size_t dash = s.find('-');
|
||||
if (dash < 2 || dash > 3 || s.size() != dash + 3) return false;
|
||||
for (std::size_t i = 0; i < dash; ++i)
|
||||
if (!std::islower((unsigned char)s[i])) return false;
|
||||
return std::isupper((unsigned char)s[dash + 1]) &&
|
||||
std::isupper((unsigned char)s[dash + 2]);
|
||||
}
|
||||
|
||||
std::string language_of(const std::string& value) {
|
||||
std::string lang = value.empty() ? env_or("GOOGLE_VOICE_LANG", "nl-NL") : value;
|
||||
if (!valid_language(lang))
|
||||
throw fw::HttpError(422, "validation_error", "lang must look like nl-NL");
|
||||
return lang;
|
||||
}
|
||||
|
||||
bool valid_voice(const std::string& s) {
|
||||
if (s.size() > 64) return false;
|
||||
for (char c : s)
|
||||
if (!std::isalnum((unsigned char)c) && c != '-' && c != '_') return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string voice_of(const std::string& value) {
|
||||
std::string voice = value.empty() ? google::Credentials::env("GOOGLE_VOICE_NAME") : value;
|
||||
if (!valid_voice(voice))
|
||||
throw fw::HttpError(422, "validation_error", "voice has invalid characters");
|
||||
return voice;
|
||||
}
|
||||
|
||||
// Maps Google part errors onto the example's JSON error format.
|
||||
[[noreturn]] void rethrow(const google::GoogleError& e) {
|
||||
if (e.http_status() == 503 && e.status() == "NOT_CONFIGURED")
|
||||
throw fw::HttpError(503, "google_not_configured", e.message());
|
||||
// Local input validation carries no service name; upstream errors do.
|
||||
if (e.service().empty() && e.status() == "INVALID_ARGUMENT")
|
||||
throw fw::HttpError(422, "validation_error", e.message());
|
||||
throw fw::HttpError(502, "google_error",
|
||||
e.status() + ": " + e.message());
|
||||
}
|
||||
|
||||
template <typename F>
|
||||
auto guarded(F&& f) -> decltype(f()) {
|
||||
try {
|
||||
return f();
|
||||
} catch (const google::GoogleError& e) {
|
||||
rethrow(e);
|
||||
}
|
||||
}
|
||||
|
||||
std::string quote(const std::string& s) {
|
||||
return "\"" + fw::json_escape(s) + "\"";
|
||||
}
|
||||
|
||||
fw::Response no_store(fw::Response res) {
|
||||
res.headers.set("Cache-Control", "no-store");
|
||||
return res;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void register_voice_demo(fw::Router& router, Services& services) {
|
||||
if (!services.google) services.google = google::GoogleClient::from_env();
|
||||
auto gclient = services.google;
|
||||
google::VoiceBackend backend =
|
||||
services.voice_backend.value_or(google::voice_backend_from_env());
|
||||
google::VoiceParts parts = google::make_voice_parts(gclient, backend);
|
||||
auto tts = parts.synthesizer;
|
||||
auto stt = parts.recognizer;
|
||||
services.voice = std::make_shared<google::VoicePipeline>(
|
||||
stt, parts.responder, tts, std::make_shared<google::VoiceSessions>());
|
||||
auto pipeline = services.voice;
|
||||
fw::Environment* env = services.templates.get();
|
||||
|
||||
services.add_feature("google", "/google/status");
|
||||
services.add_feature("voice", "/voice");
|
||||
|
||||
router.get("/google/status", "google-status", [gclient, backend](fw::Request req) {
|
||||
(void)req;
|
||||
std::string out = std::string("{\"configured\":") +
|
||||
(gclient->configured() ? "true" : "false") +
|
||||
",\"credential\":\"x-goog-api-key\",\"backend\":" +
|
||||
quote(google::voice_backend_name(backend)) + ",\"model\":" +
|
||||
quote(google::Generative::default_model()) +
|
||||
",\"language\":" + quote(env_or("GOOGLE_VOICE_LANG", "nl-NL")) +
|
||||
",\"services\":[";
|
||||
bool first = true;
|
||||
for (const auto& e : google::endpoints::catalog()) {
|
||||
out += std::string(first ? "" : ",") + "{\"name\":" + quote(e.name) +
|
||||
",\"base_url\":" + quote(e.base_url) + "}";
|
||||
first = false;
|
||||
}
|
||||
return no_store(fw::JsonResponse(out + "]}", 200));
|
||||
});
|
||||
|
||||
router.get("/voice", "voice-page", [env](fw::Request req) {
|
||||
(void)req;
|
||||
fw::TemplateCtx ctx;
|
||||
ctx["title"] = std::string("Voice conversation");
|
||||
return fw::Response(200, env->render("voice.html", ctx),
|
||||
"text/html; charset=utf-8");
|
||||
});
|
||||
|
||||
// Text in, spoken audio (mp3) out.
|
||||
router.post("/voice/say", "voice-say", [tts](fw::Request req) {
|
||||
auto in = fw::parse_json_object_flat(req.body);
|
||||
google::SynthesisRequest speak;
|
||||
speak.text = in.count("text") ? in["text"] : "";
|
||||
speak.language = language_of(in.count("lang") ? in["lang"] : "");
|
||||
speak.voice = voice_of(in.count("voice") ? in["voice"] : "");
|
||||
return guarded([&] {
|
||||
google::Audio audio = tts->synthesize(speak);
|
||||
return no_store(fw::Response(200, audio.bytes, audio.mime));
|
||||
});
|
||||
});
|
||||
|
||||
// Audio bytes in (Content-Type names the encoding), transcript out.
|
||||
router.post("/voice/hear", "voice-hear", [stt](fw::Request req) {
|
||||
google::RecognitionRequest heard;
|
||||
heard.audio = req.body;
|
||||
if (!google::speech_encoding_from_mime(req.headers.get("Content-Type"),
|
||||
heard.encoding))
|
||||
throw fw::HttpError(415, "unsupported_media_type",
|
||||
"send audio/webm, audio/ogg, audio/flac, audio/wav or audio/mpeg");
|
||||
heard.language = language_of(req.query_or("lang", ""));
|
||||
return guarded([&] {
|
||||
google::Transcript t = stt->recognize(heard);
|
||||
return no_store(fw::JsonResponse(
|
||||
"{\"text\":" + quote(t.text) +
|
||||
",\"confidence\":" + std::to_string(t.confidence) + "}",
|
||||
200));
|
||||
});
|
||||
});
|
||||
|
||||
// One conversation turn. Audio body = spoken turn; JSON body {"text"} =
|
||||
// typed turn. Both answer {transcript, reply, audio_b64, mime, language};
|
||||
// silence answers an empty transcript and no audio.
|
||||
router.post("/voice/turn", "voice-turn", [pipeline](fw::Request req) {
|
||||
std::string cid = req.query_or("cid", "");
|
||||
if (!google::VoiceSessions::valid_id(cid))
|
||||
throw fw::HttpError(422, "validation_error",
|
||||
"cid must be 1-64 chars of A-Z a-z 0-9 - _");
|
||||
std::string lang = language_of(req.query_or("lang", ""));
|
||||
std::string voice = voice_of(req.query_or("voice", ""));
|
||||
std::string ctype = req.headers.get("Content-Type");
|
||||
return guarded([&] {
|
||||
google::VoiceTurn turn;
|
||||
if (ctype.find("json") != std::string::npos) {
|
||||
auto in = fw::parse_json_object_flat(req.body);
|
||||
turn = pipeline->turn_text(cid, in.count("text") ? in["text"] : "",
|
||||
lang, voice);
|
||||
} else {
|
||||
google::RecognitionRequest heard;
|
||||
heard.audio = req.body;
|
||||
heard.language = lang;
|
||||
if (!google::speech_encoding_from_mime(ctype, heard.encoding))
|
||||
throw fw::HttpError(415, "unsupported_media_type",
|
||||
"send audio/webm, audio/ogg, audio/flac, audio/wav or audio/mpeg");
|
||||
turn = pipeline->turn(cid, heard, voice);
|
||||
}
|
||||
return no_store(fw::JsonResponse(
|
||||
"{\"transcript\":" + quote(turn.transcript) +
|
||||
",\"reply\":" + quote(turn.reply) + ",\"language\":" +
|
||||
quote(turn.language) + ",\"mime\":" + quote(turn.audio.mime) +
|
||||
",\"audio_b64\":\"" +
|
||||
security::crypto::Base64::encode(turn.audio.bytes) + "\"}",
|
||||
200));
|
||||
});
|
||||
});
|
||||
|
||||
router.post("/voice/reset", "voice-reset", [pipeline](fw::Request req) {
|
||||
std::string cid = req.query_or("cid", "");
|
||||
if (!google::VoiceSessions::valid_id(cid))
|
||||
throw fw::HttpError(422, "validation_error",
|
||||
"cid must be 1-64 chars of A-Z a-z 0-9 - _");
|
||||
pipeline->sessions().reset(cid);
|
||||
return fw::JsonResponse("{\"reset\":true}", 200);
|
||||
});
|
||||
}
|
||||
|
||||
} // namespace molo::example::app
|
||||
@@ -0,0 +1,244 @@
|
||||
// Voice conversation page: hands-free loop in the browser.
|
||||
// listen -> detect speech (mic level) -> record one utterance -> POST
|
||||
// /voice/turn -> play the spoken answer -> listen again. Speaking over the
|
||||
// answer interrupts it. All limits are constants below.
|
||||
(function () {
|
||||
"use strict";
|
||||
var LEVEL_START = 0.04; // RMS that counts as speech
|
||||
var LEVEL_BARGE = 0.09; // RMS that interrupts the assistant's voice
|
||||
var ONSET_MS = 80; // speech must last this long to start recording
|
||||
var BARGE_MS = 300; // ... and this long to interrupt playback
|
||||
var SILENCE_MS = 900; // quiet this long ends the utterance
|
||||
var MIN_SPEECH_MS = 300; // shorter utterances are noise, discarded
|
||||
var MAX_UTTERANCE_MS = 30000;
|
||||
var TICK_MS = 40;
|
||||
|
||||
var $ = function (id) { return document.getElementById(id); };
|
||||
var toggle = $("v-toggle"), langSel = $("v-lang"), stateEl = $("v-state");
|
||||
var levelEl = $("v-level"), errorEl = $("v-error"), logEl = $("v-log");
|
||||
var form = $("v-form"), textEl = $("v-text"), resetBtn = $("v-reset");
|
||||
|
||||
var cid = newId();
|
||||
var stream = null, ctx = null, analyser = null, buf = null, timer = null;
|
||||
var recorder = null, chunks = [], recStart = 0, voicedMs = 0, quietMs = 0;
|
||||
var onsetMs = 0, bargeMs = 0;
|
||||
var player = null;
|
||||
var state = "idle"; // idle | listening | hearing | thinking | speaking
|
||||
|
||||
function newId() {
|
||||
var a = new Uint8Array(12);
|
||||
crypto.getRandomValues(a);
|
||||
return Array.prototype.map.call(a, function (b) {
|
||||
return ("0" + b.toString(16)).slice(-2);
|
||||
}).join("");
|
||||
}
|
||||
function setState(s) {
|
||||
state = s;
|
||||
stateEl.textContent = s;
|
||||
stateEl.className = "pill " + s;
|
||||
}
|
||||
function showError(msg) { errorEl.textContent = msg || ""; }
|
||||
function addLog(who, text) {
|
||||
var li = document.createElement("li");
|
||||
li.className = who;
|
||||
var b = document.createElement("b");
|
||||
b.textContent = who === "you" ? "You" : "Assistant";
|
||||
li.appendChild(b);
|
||||
li.appendChild(document.createTextNode(": " + text));
|
||||
logEl.appendChild(li);
|
||||
li.scrollIntoView({ block: "nearest" });
|
||||
}
|
||||
function errorText(res, body) {
|
||||
try {
|
||||
var j = JSON.parse(body);
|
||||
return j.message || j.error || ("HTTP " + res.status);
|
||||
} catch (e) { return "HTTP " + res.status; }
|
||||
}
|
||||
|
||||
function rms() {
|
||||
analyser.getByteTimeDomainData(buf);
|
||||
var sum = 0;
|
||||
for (var i = 0; i < buf.length; i++) {
|
||||
var v = (buf[i] - 128) / 128;
|
||||
sum += v * v;
|
||||
}
|
||||
return Math.sqrt(sum / buf.length);
|
||||
}
|
||||
|
||||
function pickMime() {
|
||||
var list = ["audio/webm;codecs=opus", "audio/ogg;codecs=opus", "audio/webm"];
|
||||
for (var i = 0; i < list.length; i++)
|
||||
if (window.MediaRecorder && MediaRecorder.isTypeSupported(list[i])) return list[i];
|
||||
return "";
|
||||
}
|
||||
|
||||
function startRecording() {
|
||||
chunks = [];
|
||||
var mime = pickMime();
|
||||
recorder = mime ? new MediaRecorder(stream, { mimeType: mime }) : new MediaRecorder(stream);
|
||||
recorder.ondataavailable = function (e) { if (e.data.size) chunks.push(e.data); };
|
||||
recorder.start(250);
|
||||
recStart = Date.now();
|
||||
voicedMs = 0; quietMs = 0;
|
||||
setState("hearing");
|
||||
}
|
||||
|
||||
function stopRecording(send) {
|
||||
var r = recorder, voiced = voicedMs;
|
||||
recorder = null;
|
||||
r.onstop = function () {
|
||||
var blob = new Blob(chunks, { type: r.mimeType || "audio/webm" });
|
||||
chunks = [];
|
||||
if (send && voiced >= MIN_SPEECH_MS && state !== "idle") sendAudio(blob);
|
||||
else if (state !== "idle") setState("listening");
|
||||
};
|
||||
r.stop();
|
||||
}
|
||||
|
||||
function turnUrl() {
|
||||
return "/voice/turn?cid=" + encodeURIComponent(cid) +
|
||||
"&lang=" + encodeURIComponent(langSel.value);
|
||||
}
|
||||
|
||||
function sendAudio(blob) {
|
||||
setState("thinking");
|
||||
post(turnUrl(), blob, blob.type || "audio/webm");
|
||||
}
|
||||
|
||||
function post(url, body, type) {
|
||||
showError("");
|
||||
fetch(url, { method: "POST", headers: { "Content-Type": type }, body: body })
|
||||
.then(function (res) { return res.text().then(function (t) { return [res, t]; }); })
|
||||
.then(function (pair) {
|
||||
var res = pair[0], text = pair[1];
|
||||
if (!res.ok) {
|
||||
showError(errorText(res, text));
|
||||
if (res.status === 503) { stop(); return; }
|
||||
if (state !== "idle") setState(stream ? "listening" : "idle");
|
||||
return;
|
||||
}
|
||||
handleTurn(JSON.parse(text));
|
||||
})
|
||||
.catch(function (err) {
|
||||
showError("Network error: " + err.message);
|
||||
if (state !== "idle") setState(stream ? "listening" : "idle");
|
||||
});
|
||||
}
|
||||
|
||||
function handleTurn(turn) {
|
||||
if (!turn.transcript) { // nothing intelligible was heard
|
||||
if (state !== "idle") setState(stream ? "listening" : "idle");
|
||||
return;
|
||||
}
|
||||
addLog("you", turn.transcript);
|
||||
addLog("bot", turn.reply);
|
||||
if (state === "idle" && !stream) return;
|
||||
play(turn);
|
||||
}
|
||||
|
||||
function play(turn) {
|
||||
stopPlayback();
|
||||
player = new Audio("data:" + turn.mime + ";base64," + turn.audio_b64);
|
||||
var mine = player;
|
||||
mine.onended = function () {
|
||||
if (player === mine) { player = null; if (stream) setState("listening"); else setState("idle"); }
|
||||
};
|
||||
mine.onerror = function () {
|
||||
if (player === mine) { player = null; showError("Could not play audio"); if (stream) setState("listening"); }
|
||||
};
|
||||
setState(stream ? "speaking" : "idle");
|
||||
mine.play().catch(function (e) { showError("Playback blocked: " + e.message); if (stream) setState("listening"); });
|
||||
}
|
||||
|
||||
function stopPlayback() {
|
||||
if (player) { var p = player; player = null; p.onended = null; p.pause(); }
|
||||
}
|
||||
|
||||
function tick() {
|
||||
var level = rms();
|
||||
levelEl.style.width = Math.min(100, Math.round(level * 400)) + "%";
|
||||
if (state === "listening") {
|
||||
onsetMs = level > LEVEL_START ? onsetMs + TICK_MS : 0;
|
||||
if (onsetMs >= ONSET_MS) { onsetMs = 0; startRecording(); }
|
||||
} else if (state === "speaking") {
|
||||
bargeMs = level > LEVEL_BARGE ? bargeMs + TICK_MS : 0;
|
||||
if (bargeMs >= BARGE_MS) { bargeMs = 0; stopPlayback(); startRecording(); }
|
||||
} else if (state === "hearing") {
|
||||
if (level > LEVEL_START) { voicedMs += TICK_MS; quietMs = 0; }
|
||||
else quietMs += TICK_MS;
|
||||
if (quietMs >= SILENCE_MS || Date.now() - recStart >= MAX_UTTERANCE_MS)
|
||||
stopRecording(true);
|
||||
}
|
||||
}
|
||||
|
||||
function start() {
|
||||
showError("");
|
||||
if (!navigator.mediaDevices || !window.MediaRecorder) {
|
||||
showError("This browser has no microphone recording support.");
|
||||
return;
|
||||
}
|
||||
navigator.mediaDevices.getUserMedia({
|
||||
audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true }
|
||||
}).then(function (s) {
|
||||
stream = s;
|
||||
ctx = new (window.AudioContext || window.webkitAudioContext)();
|
||||
var src = ctx.createMediaStreamSource(stream);
|
||||
analyser = ctx.createAnalyser();
|
||||
analyser.fftSize = 1024;
|
||||
buf = new Uint8Array(analyser.fftSize);
|
||||
src.connect(analyser);
|
||||
onsetMs = 0; bargeMs = 0;
|
||||
timer = setInterval(tick, TICK_MS);
|
||||
toggle.textContent = "Stop conversation";
|
||||
toggle.classList.add("live");
|
||||
setState("listening");
|
||||
}).catch(function (err) {
|
||||
showError("Microphone unavailable: " + err.message);
|
||||
});
|
||||
}
|
||||
|
||||
function stop() {
|
||||
if (timer) { clearInterval(timer); timer = null; }
|
||||
if (recorder) { var r = recorder; recorder = null; r.onstop = null; try { r.stop(); } catch (e) {} }
|
||||
stopPlayback();
|
||||
if (stream) { stream.getTracks().forEach(function (t) { t.stop(); }); stream = null; }
|
||||
if (ctx) { ctx.close(); ctx = null; }
|
||||
levelEl.style.width = "0";
|
||||
toggle.textContent = "Start conversation";
|
||||
toggle.classList.remove("live");
|
||||
setState("idle");
|
||||
}
|
||||
|
||||
toggle.addEventListener("click", function () { if (stream) stop(); else start(); });
|
||||
|
||||
form.addEventListener("submit", function (ev) {
|
||||
ev.preventDefault();
|
||||
var text = textEl.value.trim();
|
||||
if (!text || state === "thinking") return;
|
||||
textEl.value = "";
|
||||
stopPlayback();
|
||||
setState("thinking");
|
||||
post(turnUrl(), JSON.stringify({ text: text }), "application/json");
|
||||
});
|
||||
|
||||
resetBtn.addEventListener("click", function () {
|
||||
fetch("/voice/reset?cid=" + encodeURIComponent(cid), { method: "POST" });
|
||||
cid = newId();
|
||||
logEl.innerHTML = "";
|
||||
showError("");
|
||||
});
|
||||
|
||||
fetch("/google/status").then(function (r) { return r.json(); }).then(function (j) {
|
||||
var cfg = $("v-config");
|
||||
if (j.configured) {
|
||||
cfg.textContent = "Google credentials found. Model: " + j.model + ".";
|
||||
cfg.className = "ok";
|
||||
toggle.disabled = false;
|
||||
} else {
|
||||
cfg.textContent = "No Google API key. Start the server with GOOGLE_API_KEY set.";
|
||||
cfg.className = "error";
|
||||
}
|
||||
}).catch(function () { $("v-config").textContent = "Status unavailable."; });
|
||||
|
||||
setState("idle");
|
||||
})();
|
||||
@@ -12,6 +12,10 @@ HERE = Path(__file__).resolve().parent
|
||||
APP = HERE / "app"
|
||||
LOCUST = HERE / "locustfile.py"
|
||||
|
||||
# Routes that call paid Google APIs: never load tested. The check fails if
|
||||
# locust references one, and does not require them to be covered.
|
||||
EXCLUDED = {"/voice/say", "/voice/hear", "/voice/turn", "/voice/reset"}
|
||||
|
||||
FILLS = {"name": "locust", "id": "1", "status": "404", "tenant": "acme"}
|
||||
|
||||
|
||||
@@ -55,7 +59,7 @@ def main() -> int:
|
||||
routes = collect_routes()
|
||||
urls, methods = collect_locust_urls()
|
||||
missing: list[str] = []
|
||||
for r in sorted(routes):
|
||||
for r in sorted(routes - EXCLUDED):
|
||||
c = fill(r)
|
||||
ok = (
|
||||
c in urls
|
||||
@@ -68,6 +72,9 @@ def main() -> int:
|
||||
)
|
||||
if not ok:
|
||||
missing.append(f"{r} -> {c}")
|
||||
leaked = sorted(EXCLUDED & urls)
|
||||
for r in leaked:
|
||||
print("FORBIDDEN (costs money):", r)
|
||||
required_methods = {"get", "post", "put", "options"}
|
||||
missing_methods = required_methods - methods
|
||||
print(f"routes={len(routes)} urls={len(urls)} methods={sorted(methods)}")
|
||||
@@ -75,10 +82,13 @@ def main() -> int:
|
||||
print("MISSING METHODS:", sorted(missing_methods))
|
||||
for m in missing:
|
||||
print("MISS", m)
|
||||
if leaked:
|
||||
print(f"FAIL: {len(leaked)} paid Google routes in locust")
|
||||
return 1
|
||||
if missing or missing_methods:
|
||||
print(f"FAIL: {len(missing)} missing routes, {len(missing_methods)} missing methods")
|
||||
return 1
|
||||
print(f"OK: {len(routes)}/{len(routes)} routes covered, methods {sorted(methods)}")
|
||||
print(f"OK: {len(routes) - len(EXCLUDED)} routes covered, {len(EXCLUDED)} paid Google routes excluded, methods {sorted(methods)}")
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
+19
-8
@@ -52,7 +52,8 @@ EXAMPLE_SRCS = \
|
||||
$(EXAMPLE_DIR)/app/SecurityExtraDemo.cpp \
|
||||
$(EXAMPLE_DIR)/app/PlatformDemo.cpp \
|
||||
$(EXAMPLE_DIR)/app/EnterpriseDemo.cpp \
|
||||
$(EXAMPLE_DIR)/app/ResilienceDemo.cpp
|
||||
$(EXAMPLE_DIR)/app/ResilienceDemo.cpp \
|
||||
$(EXAMPLE_DIR)/app/VoiceDemo.cpp
|
||||
|
||||
# Tests are discovered: every tests/example/**/*_test.cpp is one route test.
|
||||
EXAMPLE_TEST_JOBS ?= 8
|
||||
@@ -68,10 +69,15 @@ EXAMPLE_ASSETS = \
|
||||
$(EXAMPLE_DIR)/assets/app.css \
|
||||
$(EXAMPLE_DIR)/assets/chart.umd.min.js \
|
||||
$(EXAMPLE_DIR)/assets/hello.txt \
|
||||
$(EXAMPLE_DIR)/assets/stats.js
|
||||
EXAMPLE_INC = -I$(EXAMPLE_DIR) -I$(EXAMPLE_BUILD)
|
||||
$(EXAMPLE_DIR)/assets/stats.js \
|
||||
$(EXAMPLE_DIR)/assets/voice.js
|
||||
EXAMPLE_INC = -I$(EXAMPLE_DIR) -I$(EXAMPLE_BUILD) -Isrc
|
||||
# Real framework parts the example links: Google Cloud (src/google) over the
|
||||
# HTTP client (src/clients, wolfSSL TLS) and the operability breaker.
|
||||
EXAMPLE_REAL_LIBS = build/google/libgoogle.a build/clients/libclients.a \
|
||||
build/liboperability.a
|
||||
|
||||
.PHONY: example example-assets example-test example-clean example-locust
|
||||
.PHONY: example example-real-libs example-assets example-test example-clean example-locust
|
||||
|
||||
# Asset embedding is part of the build: example/assets (Chart.js included)
|
||||
# becomes generated C++ under build/example/gen via the MoloApp embedder.
|
||||
@@ -86,7 +92,12 @@ $(EXAMPLE_GEN)/assets.cpp: $(EXAMPLE_ASSETS) $(EXAMPLE_DIR)/tools/embed_assets.c
|
||||
$(EXAMPLE_BUILD)/embed_assets $(EXAMPLE_DIR)/assets \
|
||||
$(EXAMPLE_GEN)/assets.cpp $(EXAMPLE_GEN)/assets.hpp molo::example::gen
|
||||
|
||||
example: example-assets
|
||||
example-real-libs:
|
||||
$(MAKE) -f src/operability.mk build/liboperability.a
|
||||
$(MAKE) -f src/clients/clients.mk clients
|
||||
$(MAKE) -f src/google/google.mk google
|
||||
|
||||
example: example-assets example-real-libs
|
||||
mkdir -p $(EXAMPLE_BUILD)/obj
|
||||
for s in $(EXAMPLE_SRCS); do \
|
||||
o=$(EXAMPLE_BUILD)/obj/`echo $$s | tr '/' '_'`.o; \
|
||||
@@ -98,7 +109,7 @@ example: example-assets
|
||||
$(MAKE) -f src/supervisor/supervisor.mk build/supervisor/libsupervisor.a
|
||||
$(CXX) $(EXAMPLE_CXXFLAGS) -I$(EXAMPLE_DIR) -Isrc $(EXAMPLE_DIR)/main.cpp \
|
||||
$(EXAMPLE_BUILD)/libexample.a build/supervisor/libsupervisor.a \
|
||||
-o $(EXAMPLE_BUILD)/living-server
|
||||
$(EXAMPLE_REAL_LIBS) -lwolfssl -o $(EXAMPLE_BUILD)/living-server
|
||||
|
||||
example-test: example
|
||||
rm -rf $(EXAMPLE_BUILD)/tests
|
||||
@@ -107,8 +118,8 @@ example-test: example
|
||||
xargs -P $(EXAMPLE_TEST_JOBS) -I{} sh -c ' \
|
||||
t={}; n=`echo $$t | sed "s|$(EXAMPLE_TESTDIR)/||; s|/|_|g; s|\\.cpp$$||"`; \
|
||||
b=$(EXAMPLE_BUILD)/tests/$$n; \
|
||||
if $(CXX) $(EXAMPLE_CXXFLAGS) -I$(EXAMPLE_DIR) -I$(EXAMPLE_TESTDIR) \
|
||||
$$t $(EXAMPLE_BUILD)/libexample.a -o $$b > $$b.log 2>&1; \
|
||||
if $(CXX) $(EXAMPLE_CXXFLAGS) -I$(EXAMPLE_DIR) -I$(EXAMPLE_TESTDIR) -Isrc \
|
||||
$$t $(EXAMPLE_BUILD)/libexample.a $(EXAMPLE_REAL_LIBS) -lwolfssl -o $$b > $$b.log 2>&1; \
|
||||
then $$b >> $$b.log 2>&1; else echo "COMPILE FAILED: $$t" >> $$b.log; false; fi; \
|
||||
echo $$? > $$b.rc' < $(EXAMPLE_BUILD)/tests/list.txt
|
||||
pass=0; fail=0; \
|
||||
|
||||
@@ -39,6 +39,7 @@ GETS = [
|
||||
"/csp/page", "/csp/report",
|
||||
"/di/services", "/di/resolve/locust",
|
||||
"/embedded/chart.umd.min.js", "/embedded/hello.txt", "/embedded/stats.js",
|
||||
"/embedded/voice.js", "/google/status", "/voice",
|
||||
"/events/fire", "/expose/record", "/expose/stats",
|
||||
"/files/", "/files/welcome.txt",
|
||||
"/flash", "/flash/set",
|
||||
@@ -87,6 +88,8 @@ GETS = [
|
||||
"/webdav/files/test.txt",
|
||||
]
|
||||
|
||||
# NOT load tested on purpose: /voice/say, /voice/hear, /voice/turn and
|
||||
# /voice/reset call paid Google APIs (check_locust_coverage.py enforces this).
|
||||
POSTS = [
|
||||
("/audit/log", {"event": "locust"}),
|
||||
("/audit/retention/hold", {"hold": True}),
|
||||
|
||||
@@ -76,3 +76,29 @@ label.inline input { padding: 0; }
|
||||
.nav-wide { display: flex; }
|
||||
.nav-narrow { display: none; }
|
||||
}
|
||||
|
||||
/* Voice conversation page */
|
||||
.voice { display: flex; flex-direction: column; gap: 0.75rem; }
|
||||
.voice-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 0.75rem; }
|
||||
.voice-controls button.primary { flex: 1 1 12rem; padding: 0.8rem 1rem; font-weight: 600;
|
||||
border: 0; border-radius: 6px; background: var(--series-1); color: #fff; }
|
||||
.voice-controls button.primary.live { background: var(--series-8); }
|
||||
.voice-controls button:disabled { opacity: 0.5; }
|
||||
.voice-controls select { font-size: 1rem; padding: 0.4rem; }
|
||||
.voice-state { display: flex; align-items: center; gap: 0.75rem; }
|
||||
.pill { min-width: 6rem; text-align: center; padding: 0.2rem 0.6rem; border-radius: 1rem;
|
||||
border: 1px solid var(--border); color: var(--text-secondary); }
|
||||
.pill.listening { border-color: var(--series-3); color: var(--series-3); }
|
||||
.pill.hearing { border-color: var(--series-2); color: var(--series-2); }
|
||||
.pill.thinking { border-color: var(--series-4); color: var(--series-4); }
|
||||
.pill.speaking { border-color: var(--series-1); color: var(--series-1); }
|
||||
.meter { flex: 1; height: 0.6rem; border: 1px solid var(--border); border-radius: 0.3rem; overflow: hidden; }
|
||||
.meter > div { height: 100%; width: 0; background: var(--series-3); }
|
||||
.voice-log { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column;
|
||||
gap: 0.4rem; max-height: 22rem; overflow-y: auto; }
|
||||
.voice-log li { padding: 0.4rem 0.6rem; border: 1px solid var(--border); border-radius: 6px;
|
||||
overflow-wrap: anywhere; }
|
||||
.voice-log li.you { border-left: 4px solid var(--series-2); }
|
||||
.voice-log li.bot { border-left: 4px solid var(--series-1); }
|
||||
.voice-form { flex-direction: row; flex-wrap: wrap; max-width: none; }
|
||||
.voice-form input[type="text"] { flex: 1 1 12rem; min-width: 0; }
|
||||
|
||||
Reference in New Issue
Block a user