feat(clients): audio: ElevenLabs, AssemblyAI, Deepgram, Cartesia, Speechmatics, Hume
Table driven REST (509 rows verified against vendor specs by tools/check_spec.py), WebSockets over the src/google/live client, neutral SpeechSynthesizer / SpeechRecognizer seam, /ai/audio demo with dry runs, 110 package tests and 5 route tests. PlayHT has no public API; LeMUR is gone from AssemblyAI's spec. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Vendored
+273
@@ -0,0 +1,273 @@
|
||||
# Audio vendors - Specification
|
||||
|
||||
Owned directory: `src/vendors/audio/`. Namespace `molo::audio` (one sub
|
||||
namespace per provider: `elevenlabs`, `assemblyai`, `deepgram`, `cartesia`,
|
||||
`speechmatics`, `hume`). Plan: `plans/18-breadth.md` section 2. Method: plans
|
||||
16 and 17. Depends on `src/llm` (`ApiClient`, `Service`, `ApiError`,
|
||||
`SseDecoder`, `Config`) and `src/google/live` (RFC 6455 `WsClient`). It does
|
||||
not edit either.
|
||||
|
||||
Status 2026-10-25: implemented; local suite green; no vendor key was available,
|
||||
so no success response from any vendor has been seen (see Verified live and
|
||||
Unverified).
|
||||
|
||||
## 1. Scope
|
||||
|
||||
| Provider | REST rows | WebSockets | Neutral seam |
|
||||
|---|---|---|---|
|
||||
| ElevenLabs | 325 | TTS stream-input, realtime STT (Scribe), conversational AI agent | synthesizer, recognizer |
|
||||
| AssemblyAI | 14 (api 10, streaming token 1, LLM Gateway 3) | streaming v3 | recognizer |
|
||||
| Deepgram | 51 | listen v1, listen v2 (Flux), speak v1, speak v2, voice agent | synthesizer, recognizer |
|
||||
| Cartesia | 53 | TTS (contexts), STT turns (auto finalize), STT manual finalize | synthesizer, recognizer |
|
||||
| Speechmatics | 17 (batch, TTS preview, temporary keys, management) | realtime v2 | synthesizer, recognizer |
|
||||
| Hume | 49 | TTS stream-input, EVI chat | synthesizer |
|
||||
|
||||
ElevenLabs rows cover text to speech (buffered, streamed, timestamps), speech
|
||||
to text, voices (list, get, settings, shared library, voice design, PVC and
|
||||
cloning reads and writes), dubbing, sound effects, music, conversational AI
|
||||
agents (agents, conversations, knowledge base, tools, phone numbers, tests,
|
||||
MCP servers, batch calling, ...), history, models, user, subscription,
|
||||
pronunciation dictionaries, voice changer, audio isolation, forced alignment,
|
||||
text to dialogue. Excluded on purpose (explicit list in `tools/check_spec.py`
|
||||
`EL_EXCLUDE`, 96 operations): workspace administration, Studio projects,
|
||||
Audio Native, productions, flows, speech engine, assets, service accounts.
|
||||
|
||||
PlayHT: no public API exists to wrap. `api.play.ht` and `api.play.ai` do not
|
||||
resolve (checked 2026-10-25). Nothing is invented; `Providers::absent()` says so.
|
||||
|
||||
AssemblyAI LeMUR: no longer in the OpenAPI document or the docs index, and
|
||||
`POST /lemur/v3/generate/task` answers 404 unauthenticated. The LLM Gateway
|
||||
(chat completions, speech understanding, models) is wrapped instead.
|
||||
AssemblyAI's Sync and Dictation APIs have docs pages but no machine readable
|
||||
specification: not wrapped (Gaps).
|
||||
|
||||
Hume has no speech to text product. EVI is a voice agent; Expression
|
||||
Measurement analyses prosody (REST batch runs wrapped; its streaming
|
||||
WebSocket is not).
|
||||
|
||||
## 2. Module map (one concern per file)
|
||||
|
||||
Shared layer (`src/vendors/audio/`):
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | static ceilings |
|
||||
| `call_spec.h/.cpp` | `CallSpec` row, `CallTable`, path rendering with id validation, destructive heuristic |
|
||||
| `query.h/.cpp` | percent-encoded query builder with a size cap |
|
||||
| `hosts.h/.cpp` | `Hosts` (named hosts), `ByteStreamer` (incremental bodies, JSON error sniff), `HostFactory`, allowlisted base URL overrides |
|
||||
| `audio_service.h/.cpp` | `AudioService`: table driven `call`, `call_raw`, `call_bytes`, `call_form`, `call_stream`, `call_destructive`, `preview`; every guard before the network |
|
||||
| `socket.h/.cpp` | `AudioSocket` over `WsClient`, `Vocabulary` (client / server message names) |
|
||||
| `speech.h/.cpp` | the neutral seam: `SpeechSynthesizer`, `SpeechRecognizer`, `validate`, `split_text`, `ChunkedSynthesizer`, `PollPolicy` / `poll` |
|
||||
| `providers.h/.cpp` | one descriptor per vendor (env var, key page, products, table) |
|
||||
| `registry.cpp` | `SpeechRegistry`: name to seam object |
|
||||
|
||||
Per provider (`<provider>/`): `calls.cpp` (the table between `CALLS-BEGIN` /
|
||||
`CALLS-END`), `calls.h` (ids), `client.h/.cpp` (endpoints, auth, hosts),
|
||||
service classes (ElevenLabs: `tts`, `stt`, `voices`, `creative`, `dubbing`,
|
||||
`agents`, `account`, plus `tts_stream`, `scribe_stream`, `convai_session`;
|
||||
AssemblyAI: `transcripts`, `gateway`, `stream`; Deepgram: `listen`, `speak`,
|
||||
`read`, `manage`, `streams`; Cartesia: `tts`, `voices`, `stt`, `tts_socket`;
|
||||
Speechmatics: `batch`, `tts`, `manage`, `realtime`; Hume: `tts`, `evi`,
|
||||
`sockets`), and `speech.h/.cpp` (the seam implementations).
|
||||
|
||||
Demo: `example/app/ai/AudioDemo.cpp`, asset `example/assets/ai_audio.js`.
|
||||
Tests: `tests/src/vendors/audio/`, `tests/example/ai/audio/`.
|
||||
|
||||
## 3. The neutral seam
|
||||
|
||||
`SpeechSynthesizer::synthesize(SpeechRequest)` returns audio bytes;
|
||||
`SpeechRecognizer::transcribe(TranscribeRequest)` returns a `Transcript` (text,
|
||||
language, duration, words with times and speakers, the provider's own document).
|
||||
Formats are `mp3`, `wav`, `pcm` and map to each provider's own names (ElevenLabs
|
||||
`mp3_44100_128`, Deepgram `encoding=linear16&container=wav`, Cartesia
|
||||
`{container, encoding, sample_rate}`, ...). `ChunkedSynthesizer` splits text over
|
||||
a provider's per call limit at sentence boundaries and joins mp3 / pcm audio
|
||||
(wav is refused past one chunk). Asynchronous recognizers (AssemblyAI,
|
||||
Speechmatics) upload, submit and poll with `poll()`: the clock is the sum of
|
||||
the sleeps, the interval has a floor of 500 ms, the wait a ceiling of 1800 s.
|
||||
The seam lives in this package; `src/llm` is untouched.
|
||||
|
||||
## 4. Static caps (see `caps.h` and each `*Caps` struct)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Synthesis input | 100000 chars (ElevenLabs 40000, Deepgram 2000, Cartesia 20000, Hume 5000 per utterance) |
|
||||
| Splitter chunks | 64 per call, 5000 chars default |
|
||||
| JSON body | 8 MiB; demo request body 4 MiB |
|
||||
| Audio in / out kept | 128 MiB / 64 MiB; streamed audio 64 MiB |
|
||||
| Poll | interval 500..15000 ms, wait <= 1800 s |
|
||||
| Page walks | 20 pages, 1000 items |
|
||||
| WebSocket | session 1800 s, wait per read <= 120 s, 4096 events per collect, frame <= 4 MiB |
|
||||
| Keyword / keyterm lists | 100 |
|
||||
|
||||
## 5. Contracts
|
||||
|
||||
- Key present or `NotConfiguredError` (HTTP 503) before any traffic; local
|
||||
validation comes first, so bad input is a 400 (`INVALID_ARGUMENT`), not a 503.
|
||||
- The key travels in a header only: `xi-api-key` (ElevenLabs), `Authorization:
|
||||
<key>` (AssemblyAI), `Authorization: Token <key>` (Deepgram), `Authorization:
|
||||
Bearer <key>` + `Cartesia-Version: 2026-08-14` (Cartesia), `Authorization:
|
||||
Bearer <key>` (Speechmatics), `X-Hume-Api-Key` (Hume). Never in a URL, an
|
||||
error or `url()` of a socket. Two documented exceptions are handled: an
|
||||
AssemblyAI temporary token and an ElevenLabs signed agent URL carry a
|
||||
credential in the URL by the vendor's design; both classes hide it from `url()`.
|
||||
- Every REST call is a table row. Path segments pass `Service::valid_id` (an
|
||||
`{email}` segment is validated and percent-encoded). Body fields and query
|
||||
parameters are checked against the row; required fields must be present.
|
||||
- `call` refuses destructive rows (DELETE, or ids with cancel / disable /
|
||||
remove / revoke / purge / leave); `call_destructive` needs `confirm=true`.
|
||||
Typed writes (create agent, clone voice, start dubbing, create API key, ...)
|
||||
take `confirm` too.
|
||||
- Base URL overrides (`ELEVENLABS_BASE_URL`, `DEEPGRAM_BASE_URL`,
|
||||
`ASSEMBLYAI_BASE_URL`, `SPEECHMATICS_BATCH_URL`, `SPEECHMATICS_RT_URL`) are
|
||||
accepted only when they equal a documented regional host.
|
||||
- Credentials: `ELEVENLABS_API_KEY`, `ASSEMBLYAI_API_KEY`, `DEEPGRAM_API_KEY`,
|
||||
`CARTESIA_API_KEY`, `SPEECHMATICS_API_KEY`, `HUME_API_KEY`, and
|
||||
`SPEECHMATICS_MANAGEMENT_TOKEN` for the Speechmatics management host.
|
||||
|
||||
## 6. Required credentials and consoles
|
||||
|
||||
| Provider | Env var | Page |
|
||||
|---|---|---|
|
||||
| ElevenLabs | `ELEVENLABS_API_KEY` | https://elevenlabs.io/app/settings/api-keys (unverified) |
|
||||
| AssemblyAI | `ASSEMBLYAI_API_KEY` | https://www.assemblyai.com/dashboard/api-keys (unverified) |
|
||||
| Deepgram | `DEEPGRAM_API_KEY` | https://console.deepgram.com (unverified) |
|
||||
| Cartesia | `CARTESIA_API_KEY` | https://play.cartesia.ai/keys (unverified) |
|
||||
| Speechmatics | `SPEECHMATICS_API_KEY` | https://portal.speechmatics.com/settings/api-keys/ (docs) |
|
||||
| Speechmatics | `SPEECHMATICS_MANAGEMENT_TOKEN` | docs.speechmatics.com/administration/management-tokens |
|
||||
| Hume | `HUME_API_KEY` | https://app.hume.ai/keys (unverified) |
|
||||
|
||||
## Sources
|
||||
|
||||
| Provider | Machine readable sources used | Second source |
|
||||
|---|---|---|
|
||||
| ElevenLabs | `https://api.elevenlabs.io/openapi.json` (421 operations); AsyncAPI 2.6 `https://elevenlabs.io/docs/asyncapi/{elevenagents,api-reference,api-reference-2}.json` | `llms.txt` page index |
|
||||
| AssemblyAI | `https://www.assemblyai.com/docs/openapi.yaml` (v1.3.4), `docs/api-reference/specs/{streaming-token,llm-gateway,streaming}.yaml` (AsyncAPI 3) | `assemblyai-node-sdk` generated types, `llms.txt` |
|
||||
| Deepgram | `deepgram-api-specs` `openapi.yml` and `asyncapi.yml` (Fern generated) | `llms.txt` |
|
||||
| Cartesia | `cartesia-python` (Stainless, generated from Cartesia's OpenAPI): `api.md`, `.stats.yml` (53 endpoints), `*_params.py`, event types; `docs.cartesia.ai/api-reference/tts/websocket.md` (AsyncAPI page) | docs host answers bots with a challenge page, so no direct OpenAPI fetch |
|
||||
| Speechmatics | `speechmatics-python-sdk` (`batch`, `rt`, `tts`), `docs.speechmatics.com/api-ref/*.md` and `get-started/authentication.md` | no OpenAPI file published outside the rendered docs |
|
||||
| Hume | `dev.hume.ai/openapi/{text-to-speech-tts,speech-to-speech-evi,expression-measurement-api}.json`; `hume-python-sdk` (`tts/raw_client.py` for the synthesis calls the OpenAPI omits, event types for the sockets) | `llms.txt` |
|
||||
|
||||
## Verified against spec
|
||||
|
||||
`python3 -I src/vendors/audio/tools/check_spec.py` (stdlib only, read-only
|
||||
network), run 2026-10-25, exit 0:
|
||||
|
||||
```
|
||||
elevenlabs ok enum values checked 28, rows checked 325, spec operations covered 325, spec operations excluded 96, ws names checked 95
|
||||
assemblyai ok rows checked 14, spec operations covered 14, ws names checked 42
|
||||
deepgram ok rows checked 52, spec operations covered 50, ws names checked 133
|
||||
cartesia ok rows checked 53, sdk endpoints (stats) 53, ws names checked 40
|
||||
speechmatics ok enum values checked 6, rows checked 17, ws names checked 29
|
||||
hume ok rows checked 49, spec operations covered 43, ws names checked 37
|
||||
```
|
||||
|
||||
It fails on: a row absent from its source; a body field, required field, or
|
||||
query parameter not in the source; an operationId mismatch; a spec operation
|
||||
in a covered area that is neither in the table nor excluded; a WebSocket query
|
||||
parameter or message name in the C++ that the AsyncAPI document or SDK types do
|
||||
not contain; the ElevenLabs `output_format` enum differing from the code; and
|
||||
AssemblyAI's spec mentioning LeMUR again. Mutation check: renaming a Deepgram
|
||||
query parameter in `calls.cpp` produced two failures (then reverted).
|
||||
YAML documents are scanned by indentation (no YAML parser in the stdlib).
|
||||
|
||||
## Verified live
|
||||
|
||||
Unauthenticated probes, 2026-10-25 (curl, and `MOLO_LIVE_NET=1 make ai-test`
|
||||
runs `audio_live.unauthenticated_probes` against the same hosts):
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| `GET api.elevenlabs.io/v1/user`, `/v1/history`, `/v1/convai/agents`, `POST /v1/text-to-speech/{id}` | 401 `authentication_error`, body `{"detail":{"type","code","message","status","request_id"}}` |
|
||||
| `GET api.elevenlabs.io/v1/voices` | 200 without a key (public premade voices) |
|
||||
| `GET api.elevenlabs.io/v1/models` | 404 `workspace_not_found` ("Workspace 1anonymous1") |
|
||||
| `GET/POST api.assemblyai.com/v2/transcript` | 401 `{"error": "Authentication error, API token missing/invalid"}` |
|
||||
| `GET streaming.assemblyai.com/v3/token` | 422: header `authorization` and query `expires_in_seconds` required |
|
||||
| `GET llm-gateway.assemblyai.com/v1/models` | 200 without a key |
|
||||
| `POST api.assemblyai.com/lemur/v3/generate/task` | 404 `Not found` |
|
||||
| `GET api.deepgram.com/v1/projects`, `POST /v1/listen`, `POST /v1/speak` | 401 `INVALID_AUTH` / `UNAUTHORIZED` |
|
||||
| `GET api.deepgram.com/v1/models`, `agent.deepgram.com/v1/agent/settings/think/models` | 200 without a key |
|
||||
| `GET api.cartesia.ai/` | 200 `{"ok":true,"version":"2026-08-14"}` |
|
||||
| `GET api.cartesia.ai/voices`, `/agents` | 401 `Unauthorized: You must be logged in...` |
|
||||
| `POST api.cartesia.ai/tts/bytes` without `Cartesia-Version` | 400 `Cartesia-Version header is required in YYYY-MM-DD form, latest version is 2026-08-14` |
|
||||
| `GET asr.api.speechmatics.com/v2/jobs`, `eu1.asr...`, `POST preview.tts.speechmatics.com/generate/sarah` | 401 (nginx page) |
|
||||
| `GET mp.speechmatics.com/v1/projects` | 401 `{"code":401,"error":"Unauthorized"}` |
|
||||
| `GET api.hume.ai/v0/tts/voices`, `/v0/evi/chats`, `/v0/evi/configs`, `POST /v0/tts` | 401 `Credentials are required to access this resource.` |
|
||||
| `GET api.cloud.hume.ai/v1/expression/run` | 401 `{"message":"Unauthorized"}` |
|
||||
| `GET api.hume.ai/v0/batch/jobs` | 401 (the older batch path still exists; not wrapped, not in the current spec) |
|
||||
| `api.play.ht`, `api.play.ai` | do not resolve |
|
||||
|
||||
Opt-in authenticated read-only tests (`MOLO_LIVE_NET=1` plus the key; one
|
||||
smallest ElevenLabs generation only with `MOLO_LIVE_PAID=1`) print a loud
|
||||
`SKIP` without the key.
|
||||
|
||||
Local suite: `make ai-test` runs 110 audio tests (`audio_core`, `audio_elevenlabs`,
|
||||
`audio_deepgram`, `audio_assemblyai`, `audio_cartesia`, `audio_speechmatics`,
|
||||
`audio_hume`, `audio_live`), all WebSockets against a test-owned loopback
|
||||
server; `make -f example/example.mk example-test` runs 5 audio route tests
|
||||
(98 checks).
|
||||
|
||||
## Unverified
|
||||
|
||||
Everything below was inferred, not read from a published specification or seen
|
||||
in a response:
|
||||
|
||||
- Any success response body of any vendor. Folding code (`fold`, `collect`) is
|
||||
checked against shapes copied from the OpenAPI examples or SDK types, labelled
|
||||
as fixtures in the tests.
|
||||
- Authentication on the REST hosts with a real key (header names come from
|
||||
specs and SDKs; the 401s above prove the paths, not the credential style).
|
||||
- ElevenLabs `speed` range of voice settings (the OpenAPI states none), the TTS
|
||||
length ceiling of 40000 characters (largest model limit in the docs).
|
||||
- Deepgram TTS REST 2000 character limit (TTS guide), `POST /v1/listen` with a
|
||||
raw audio body and `Content-Type` (the Fern OpenAPI lists only the JSON `url`
|
||||
form; row source `guide:`), the `Token` scheme for the Agent host.
|
||||
- AssemblyAI `speakers_expected` range 1..100, LLM Gateway `role` set and
|
||||
`temperature` range 0..2, and whether `speech_models` is now required on
|
||||
`POST /v2/transcript`.
|
||||
- Cartesia: `/tts/websocket` auth by header (the docs page lists only the
|
||||
`cartesia_version` parameter; the SDK sends the headers), infill and voice
|
||||
changer multipart part names (`output_format[...]`, `voice[id]`, taken from
|
||||
the SDK aliases).
|
||||
- Speechmatics: management API base `https://mp.speechmatics.com/v1` (the
|
||||
temporary key page shows it; the 401 above shows the host answers), the
|
||||
management paths use hyphens (`/api-keys`) while the temporary key endpoint
|
||||
uses an underscore (`/api_keys`), both as documented; job status `expired`;
|
||||
TTS voices `sarah theo megan jack` and the two formats (SDK enum; no list
|
||||
endpoint exists).
|
||||
- Hume: the Expression Measurement host `api.cloud.hume.ai` comes from its
|
||||
OpenAPI `servers`; `format.type` values mp3 / wav / pcm and
|
||||
`include_timestamp_types` values from the SDK; WebSocket `Sec-WebSocket`
|
||||
upgrade acceptance with the header key (the SDK sends the same header).
|
||||
- ElevenLabs agent WebSocket: a private agent needs a signed URL; the client
|
||||
messages used (`conversation_initiation_client_data`, `user_audio_chunk`,
|
||||
`user_message`, `pong`, `client_tool_result`) are in the AsyncAPI document,
|
||||
their runtime ordering is not exercised against the real service.
|
||||
|
||||
## Gaps
|
||||
|
||||
- `ApiClient::stream` is POST-only SSE with no status or headers, so incremental
|
||||
audio uses this package's own `ByteStreamer`. It cannot see the HTTP status of a
|
||||
stream; it treats a small JSON object with `detail` / `error` / `message` at
|
||||
the start of the body as the vendor's error. Owner `src/llm`: expose status and
|
||||
headers on streams and a raw-bytes stream.
|
||||
- `ApiClient::post_multipart` is POST-only; PUT / PATCH multipart goes through
|
||||
`request_raw` with the encoded body. Owner `src/llm`.
|
||||
- The shared HTTP client opens a circuit breaker after ten consecutive failures
|
||||
per client; tests split long call sequences across rigs because of it.
|
||||
- Not wrapped: AssemblyAI Sync and Dictation APIs (no specification), AssemblyAI
|
||||
legacy streaming v2, Hume Expression Measurement streaming socket, Deepgram
|
||||
`/v1/agent` REST configuration beyond the rows in the OpenAPI document (all
|
||||
are in the table), ElevenLabs multi-context and dialogue WebSockets and the
|
||||
Realtime Translation socket (documented in the AsyncAPI, no typed client),
|
||||
Speechmatics Agent STT socket, Speechmatics Flow.
|
||||
- ElevenLabs excluded areas (section 1) can be added by moving a prefix from
|
||||
`EL_EXCLUDE` to `EL_INCLUDE` and regenerating rows.
|
||||
- No retries and no rate-limit handling beyond what the shared client does; no
|
||||
usage / cost accounting.
|
||||
- The Deepgram `Token` prefix is applied at `HostFactory::from_env`; tests
|
||||
inject the final header value. A temporary JWT (`Bearer`) is not supported.
|
||||
|
||||
## Required APIs
|
||||
|
||||
Not a Google package: no Google services are used.
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
#include "vendors/audio/assemblyai/calls.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
// One row per REST call. tools/check_spec.py parses the block between the
|
||||
// markers: {id, method, path, host, body, fields, required, query, response, source}.
|
||||
// CALLS-BEGIN
|
||||
static const CallSpec kRows[] = {
|
||||
{"listTranscripts", "GET", "/v2/transcript", "api", "-", "", "", "limit,status,created_on,before_id,after_id,throttled_only", "json", "openapi:listTranscripts"},
|
||||
{"createTranscript", "POST", "/v2/transcript", "api", "json", "audio_url,audio_end_at,audio_start_from,auto_chapters,auto_highlights,content_safety,content_safety_confidence,custom_spelling,disfluencies,domain,entity_detection,filter_profanity,format_text,iab_categories,keyterms_prompt,language_code,language_codes,language_confidence_threshold,language_detection,language_detection_options,multichannel,prompt,punctuate,redact_pii,redact_pii_audio,redact_pii_audio_options,redact_pii_audio_quality,redact_pii_policies,redact_pii_sub,redact_pii_return_unredacted,redact_static_entities,sentiment_analysis,speaker_labels,speaker_options,speakers_expected,speech_models,speech_threshold,speech_understanding,summarization,summary_model,summary_type,remove_audio_tags,temperature,webhook_auth_header_name,webhook_auth_header_value,webhook_url,custom_topics,speech_model,topics", "audio_url", "", "json", "openapi:createTranscript"},
|
||||
{"getTranscript", "GET", "/v2/transcript/{transcript_id}", "api", "-", "", "", "", "json", "openapi:getTranscript"},
|
||||
{"deleteTranscript", "DELETE", "/v2/transcript/{transcript_id}", "api", "-", "", "", "", "json", "openapi:deleteTranscript"},
|
||||
{"getTranscriptParagraphs", "GET", "/v2/transcript/{transcript_id}/paragraphs", "api", "-", "", "", "", "json", "openapi:getTranscriptParagraphs"},
|
||||
{"getRedactedAudio", "GET", "/v2/transcript/{transcript_id}/redacted-audio", "api", "-", "", "", "", "json", "openapi:getRedactedAudio"},
|
||||
{"getTranscriptSentences", "GET", "/v2/transcript/{transcript_id}/sentences", "api", "-", "", "", "", "json", "openapi:getTranscriptSentences"},
|
||||
{"wordSearch", "GET", "/v2/transcript/{transcript_id}/word-search", "api", "-", "", "", "words", "json", "openapi:wordSearch"},
|
||||
{"getSubtitles", "GET", "/v2/transcript/{transcript_id}/{subtitle_format}", "api", "-", "", "", "chars_per_caption", "text", "openapi:getSubtitles"},
|
||||
{"uploadFile", "POST", "/v2/upload", "api", "bytes", "", "", "", "json", "openapi:uploadFile"},
|
||||
{"generateStreamingToken", "GET", "/v3/token", "streaming", "-", "", "", "expires_in_seconds,max_session_duration_seconds", "json", "openapi-streaming-token:generateStreamingToken"},
|
||||
{"createChatCompletion", "POST", "/chat/completions", "gateway", "json", "messages,prompt,transcript_id,model,model_region,max_tokens,temperature,stream,tools,tool_choice,response_format,fallbacks,fallback_config,post_processing_steps", "model", "", "json", "openapi-llm-gateway:createChatCompletion"},
|
||||
{"listAvailableModels", "GET", "/models", "gateway", "-", "", "", "", "json", "openapi-llm-gateway:listAvailableModels"},
|
||||
{"createSpeechUnderstanding", "POST", "/understanding", "gateway", "json", "transcript_id,speech_understanding", "transcript_id,speech_understanding", "", "json", "openapi-llm-gateway:createSpeechUnderstanding"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
static_assert(sizeof(kRows) / sizeof(kRows[0]) == 14, "assemblyai call table size");
|
||||
|
||||
const CallTable& calls() {
|
||||
static const CallTable table("assemblyai", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
Vendored
+30
@@ -0,0 +1,30 @@
|
||||
#pragma once
|
||||
|
||||
// The AssemblyAI call table: the pre-recorded API (https://www.assemblyai.com/docs/openapi.yaml),
|
||||
// the streaming token endpoint (docs/api-reference/specs/streaming-token.yaml) and the
|
||||
// LLM Gateway (docs/api-reference/specs/llm-gateway.yaml). Hosts: api, streaming, gateway.
|
||||
|
||||
#include "vendors/audio/call_spec.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
const CallTable& calls();
|
||||
|
||||
struct AssemblyCalls {
|
||||
static constexpr const char* kUpload = "uploadFile";
|
||||
static constexpr const char* kSubmit = "createTranscript";
|
||||
static constexpr const char* kList = "listTranscripts";
|
||||
static constexpr const char* kGet = "getTranscript";
|
||||
static constexpr const char* kDelete = "deleteTranscript";
|
||||
static constexpr const char* kSentences = "getTranscriptSentences";
|
||||
static constexpr const char* kParagraphs = "getTranscriptParagraphs";
|
||||
static constexpr const char* kWordSearch = "wordSearch";
|
||||
static constexpr const char* kSubtitles = "getSubtitles";
|
||||
static constexpr const char* kRedactedAudio = "getRedactedAudio";
|
||||
static constexpr const char* kStreamingToken = "generateStreamingToken";
|
||||
static constexpr const char* kChat = "createChatCompletion";
|
||||
static constexpr const char* kGatewayModels = "listAvailableModels";
|
||||
static constexpr const char* kUnderstanding = "createSpeechUnderstanding";
|
||||
};
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
#include "vendors/audio/assemblyai/client.h"
|
||||
|
||||
#include "llm/config.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
std::string AssemblyEndpoints::base_url() {
|
||||
return allowlisted_base(kBaseEnv, kDefaultBase, {kDefaultBase, kEuBase});
|
||||
}
|
||||
|
||||
bool AssemblyEndpoints::eu() { return base_url() == kEuBase; }
|
||||
|
||||
std::string AssemblyEndpoints::streaming_host_url() {
|
||||
return eu() ? "https://streaming.eu.assemblyai.com" : "https://streaming.assemblyai.com";
|
||||
}
|
||||
|
||||
llm::ApiClientConfig AssemblyEndpoints::config(const std::string& base) {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = kVendor;
|
||||
c.base_url = base;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = "Authorization";
|
||||
return c;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> AssemblyEndpoints::from_env() {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", HostFactory::from_env(config(base_url())));
|
||||
hosts->add("streaming", HostFactory::from_env(config(streaming_host_url())));
|
||||
hosts->add("gateway", HostFactory::from_env(config(kGatewayBase)));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> AssemblyEndpoints::with(AudioHost api, AudioHost streaming, AudioHost gateway) {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", std::move(api));
|
||||
hosts->add("streaming", std::move(streaming));
|
||||
hosts->add("gateway", std::move(gateway));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::string AssemblyEndpoints::key() { return llm::Config::get(kKeyEnv); }
|
||||
|
||||
AudioSocket::Headers AssemblyEndpoints::ws_headers(const std::string& key) {
|
||||
return {{"Authorization", key}};
|
||||
}
|
||||
|
||||
std::string AssemblyEndpoints::ws_url() { return to_ws_url(streaming_host_url()) + "/v3/ws"; }
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
#pragma once
|
||||
|
||||
// AssemblyAI endpoints and credentials. The key is the whole Authorization
|
||||
// header value (no Bearer prefix) on every host.
|
||||
// api https://api.assemblyai.com (EU: https://api.eu.assemblyai.com)
|
||||
// streaming https://streaming.assemblyai.com token endpoint; WebSocket wss://streaming.assemblyai.com/v3/ws
|
||||
// gateway https://llm-gateway.assemblyai.com/v1
|
||||
// ASSEMBLYAI_BASE_URL selects the EU API host; the streaming host follows it.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/hosts.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
struct AssemblyEndpoints {
|
||||
static constexpr const char* kVendor = "assemblyai";
|
||||
static constexpr const char* kKeyEnv = "ASSEMBLYAI_API_KEY";
|
||||
static constexpr const char* kBaseEnv = "ASSEMBLYAI_BASE_URL";
|
||||
static constexpr const char* kDefaultBase = "https://api.assemblyai.com";
|
||||
static constexpr const char* kEuBase = "https://api.eu.assemblyai.com";
|
||||
static constexpr const char* kGatewayBase = "https://llm-gateway.assemblyai.com/v1";
|
||||
|
||||
static std::string base_url();
|
||||
static bool eu();
|
||||
static std::string streaming_host_url(); // https://streaming[.eu].assemblyai.com
|
||||
static llm::ApiClientConfig config(const std::string& base);
|
||||
static std::shared_ptr<const Hosts> from_env();
|
||||
// Injected hosts (tests): "api", "streaming", "gateway".
|
||||
static std::shared_ptr<const Hosts> with(AudioHost api, AudioHost streaming, AudioHost gateway);
|
||||
static std::string key();
|
||||
static AudioSocket::Headers ws_headers(const std::string& key);
|
||||
static std::string ws_url(); // wss://streaming[.eu].assemblyai.com/v3/ws
|
||||
};
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
#include "vendors/audio/assemblyai/gateway.h"
|
||||
|
||||
#include "llm/caps.h"
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
using C = AssemblyCalls;
|
||||
|
||||
Json Gateway::build(const GatewayChat& c) const {
|
||||
require_text("model", c.model, 128);
|
||||
bool has_messages = !c.messages.empty();
|
||||
bool has_prompt = !c.prompt.empty();
|
||||
if (has_messages == has_prompt) bad("give exactly one of messages and prompt");
|
||||
if (c.messages.size() > llm::Caps::kMaxMessages) bad("too many messages");
|
||||
Json o = Json::object();
|
||||
o.set("model", Json(c.model));
|
||||
if (has_messages) {
|
||||
Json arr = Json::array();
|
||||
for (const auto& m : c.messages) {
|
||||
if (m.role != "system" && m.role != "user" && m.role != "assistant" && m.role != "tool")
|
||||
bad("message role must be system, user, assistant or tool");
|
||||
require_text("message content", m.content, AudioCaps::kMaxTextChars, false);
|
||||
Json j = Json::object();
|
||||
j.set("role", Json(m.role));
|
||||
j.set("content", Json(m.content));
|
||||
arr.push(j);
|
||||
}
|
||||
o.set("messages", arr);
|
||||
} else {
|
||||
require_text("prompt", c.prompt, AudioCaps::kMaxTextChars);
|
||||
o.set("prompt", Json(c.prompt));
|
||||
}
|
||||
if (!c.transcript_id.empty()) {
|
||||
check_id("transcript", c.transcript_id);
|
||||
o.set("transcript_id", Json(c.transcript_id));
|
||||
}
|
||||
if (c.max_tokens < 0 || c.max_tokens > 1000000) bad("max_tokens is out of range");
|
||||
if (c.max_tokens > 0) o.set("max_tokens", Json(c.max_tokens));
|
||||
if (c.temperature >= 0) {
|
||||
require_range("temperature", c.temperature, 0, 2);
|
||||
o.set("temperature", Json(c.temperature));
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
Json Gateway::chat(const GatewayChat& c) {
|
||||
Json body = build(c);
|
||||
return call(C::kChat, {}, &body);
|
||||
}
|
||||
|
||||
Json Gateway::models() { return call(C::kGatewayModels); }
|
||||
|
||||
Json Gateway::understanding(const std::string& transcript_id, const Json& su) {
|
||||
check_id("transcript", transcript_id);
|
||||
if (!su.is_object()) bad("speech_understanding must be an object");
|
||||
Json body = Json::object();
|
||||
body.set("transcript_id", Json(transcript_id));
|
||||
body.set("speech_understanding", su);
|
||||
return call(C::kUnderstanding, {}, &body);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
#pragma once
|
||||
|
||||
// AssemblyAI LLM Gateway (https://llm-gateway.assemblyai.com/v1): chat
|
||||
// completions over text or a transcript id, speech understanding, model list.
|
||||
// LeMUR, the earlier transcript LLM API, no longer appears in the published
|
||||
// OpenAPI document or the docs index and answers 404 (see SPEC); the gateway
|
||||
// is its replacement and the only LLM surface wrapped here.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/assemblyai/calls.h"
|
||||
#include "vendors/audio/assemblyai/client.h"
|
||||
#include "vendors/audio/audio_service.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
struct GatewayMessage {
|
||||
std::string role; // system | user | assistant | tool
|
||||
std::string content;
|
||||
};
|
||||
|
||||
struct GatewayChat {
|
||||
std::string model; // required, for example a model id from models()
|
||||
std::vector<GatewayMessage> messages; // or prompt + transcript_id
|
||||
std::string prompt;
|
||||
std::string transcript_id;
|
||||
int max_tokens = 0; // 0 unset
|
||||
double temperature = -1; // 0..2, -1 unset
|
||||
};
|
||||
|
||||
class Gateway : public AudioService {
|
||||
public:
|
||||
explicit Gateway(std::shared_ptr<const Hosts> hosts) : AudioService(calls(), std::move(hosts)) {}
|
||||
Gateway() : Gateway(AssemblyEndpoints::from_env()) {}
|
||||
|
||||
Json build(const GatewayChat& chat) const;
|
||||
Json chat(const GatewayChat& chat);
|
||||
Json models();
|
||||
// Applies speech understanding (translation, speaker identification, custom formatting) to a finished transcript.
|
||||
Json understanding(const std::string& transcript_id, const Json& speech_understanding);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
#include "vendors/audio/assemblyai/speech.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
TranscriptRequest AssemblyRecognizer::map(const TranscribeRequest& r, const std::string& audio_url) const {
|
||||
TranscriptRequest t;
|
||||
t.audio_url = audio_url;
|
||||
if (!r.model.empty()) t.speech_models.push_back(r.model);
|
||||
if (r.language.empty()) t.language_detection = 1;
|
||||
else t.language_code = r.language;
|
||||
t.punctuate = 1;
|
||||
t.format_text = 1;
|
||||
if (r.diarize) t.speaker_labels = 1;
|
||||
t.keyterms_prompt = r.keywords;
|
||||
return t;
|
||||
}
|
||||
|
||||
Transcript AssemblyRecognizer::fold(const Json& doc) {
|
||||
Transcript t;
|
||||
t.provider = "assemblyai";
|
||||
t.raw = doc;
|
||||
if (!doc.is_object()) return t;
|
||||
t.text = llm::Service::string_at(doc, "text");
|
||||
t.language = llm::Service::string_at(doc, "language_code");
|
||||
if (const Json* d = doc.find("audio_duration"); d && d->is_number()) t.duration = d->as_number();
|
||||
if (const Json* words = doc.find("words"); words && words->is_array()) {
|
||||
for (const Json& w : words->as_array()) {
|
||||
if (!w.is_object()) continue;
|
||||
TranscriptWord word;
|
||||
word.text = llm::Service::string_at(w, "text");
|
||||
// AssemblyAI times are milliseconds.
|
||||
if (const Json* x = w.find("start"); x && x->is_number()) word.start = x->as_number() / 1000.0;
|
||||
if (const Json* x = w.find("end"); x && x->is_number()) word.end = x->as_number() / 1000.0;
|
||||
if (const Json* x = w.find("confidence"); x && x->is_number()) word.confidence = x->as_number();
|
||||
word.speaker = llm::Service::string_at(w, "speaker");
|
||||
t.words.push_back(word);
|
||||
}
|
||||
}
|
||||
return t;
|
||||
}
|
||||
|
||||
Transcript AssemblyRecognizer::transcribe(const TranscribeRequest& r) {
|
||||
validate(r);
|
||||
std::string url = r.url;
|
||||
if (url.empty()) url = transcripts_.upload(r.audio);
|
||||
Json submitted = transcripts_.submit(map(r, url));
|
||||
std::string id = llm::Service::string_at(submitted, "id");
|
||||
if (id.empty()) throw llm::ApiError("assemblyai", 502, "BAD_RESPONSE", "submit returned no id");
|
||||
PollPolicy policy = PollPolicy::clamped(3000, r.max_wait_seconds);
|
||||
return fold(transcripts_.wait(id, policy, sleep_ms_));
|
||||
}
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
#pragma once
|
||||
|
||||
// AssemblyAI behind the neutral recognizer seam. AssemblyAI has no speech
|
||||
// synthesis product, so there is no synthesizer.
|
||||
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
|
||||
#include "vendors/audio/assemblyai/transcripts.h"
|
||||
#include "vendors/audio/speech.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
class AssemblyRecognizer : public SpeechRecognizer {
|
||||
public:
|
||||
explicit AssemblyRecognizer(std::shared_ptr<const Hosts> hosts = AssemblyEndpoints::from_env(),
|
||||
std::function<void(int)> sleep_ms = nullptr)
|
||||
: transcripts_(std::move(hosts)), sleep_ms_(std::move(sleep_ms)) {}
|
||||
std::string name() const override { return "assemblyai"; }
|
||||
bool configured() const override { return transcripts_.configured(); }
|
||||
// Uploads local audio, submits, polls to completion within max_wait_seconds.
|
||||
Transcript transcribe(const TranscribeRequest& request) override;
|
||||
TranscriptRequest map(const TranscribeRequest& request, const std::string& audio_url) const;
|
||||
static Transcript fold(const Json& doc);
|
||||
|
||||
private:
|
||||
Transcripts transcripts_;
|
||||
std::function<void(int)> sleep_ms_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
#include "vendors/audio/assemblyai/stream.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("assemblyai", why); }
|
||||
|
||||
std::string str_at(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
|
||||
Json typed(const char* type) {
|
||||
Json m = Json::object();
|
||||
m.set("type", Json(type));
|
||||
return m;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const std::vector<std::string>& TranscriptStream::query_params() {
|
||||
// streaming.yaml, channel parameters.
|
||||
static const std::vector<std::string> v = {
|
||||
"token", "speech_model", "language_codes", "language_detection", "domain", "mode",
|
||||
"session_heartbeat", "encoding", "sample_rate", "min_turn_silence", "max_turn_silence",
|
||||
"end_of_turn_confidence_threshold", "vad_threshold", "interruption_delay", "include_partial_turns",
|
||||
"format_turns", "prompt", "keyterms_prompt", "agent_context", "previous_context_n_turns",
|
||||
"speaker_labels", "max_speakers", "speaker_labels_revision_interval_ms", "voice_focus",
|
||||
"voice_focus_threshold", "redact_pii", "redact_pii_policies", "redact_pii_sub", "filter_profanity",
|
||||
"llm_gateway", "inactivity_timeout"};
|
||||
return v;
|
||||
}
|
||||
|
||||
Vocabulary TranscriptStream::vocabulary() {
|
||||
return Vocabulary("type", {"UpdateConfiguration", "ForceEndpoint", "Terminate", "KeepAlive"},
|
||||
{"Begin", "SpeechStarted", "Turn", "SpeakerRevision", "Termination", "Heartbeat",
|
||||
"LLMGatewayResponse"});
|
||||
}
|
||||
|
||||
TranscriptStream::TranscriptStream(std::string api_key, StreamParams params, std::string url, SocketOptions socket) {
|
||||
Query q;
|
||||
for (const auto& [k, v] : params.pairs) {
|
||||
bool ok = false;
|
||||
for (const auto& a : query_params()) ok = ok || a == k;
|
||||
if (!ok || k == "token") invalid("unknown stream parameter '" + k + "'");
|
||||
if (v.size() > 4096) invalid("stream parameter value is too long");
|
||||
q.add(k, v);
|
||||
}
|
||||
AudioSocket::Headers headers;
|
||||
std::string base = url.empty() ? AssemblyEndpoints::ws_url() : url;
|
||||
public_url_ = base + q.str();
|
||||
if (!params.token.empty()) {
|
||||
q.add("token", params.token); // temporary token travels in the URL by the provider's design
|
||||
} else {
|
||||
headers = AssemblyEndpoints::ws_headers(api_key);
|
||||
}
|
||||
socket_ = std::make_unique<AudioSocket>("assemblyai", base + q.str(), headers, vocabulary(), std::move(socket));
|
||||
}
|
||||
|
||||
void TranscriptStream::update_configuration(const Json& fields) {
|
||||
if (!fields.is_object() || fields.as_object().empty()) invalid("fields must be a non-empty object");
|
||||
Json m = fields;
|
||||
m.set("type", Json("UpdateConfiguration"));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
void TranscriptStream::force_endpoint() { socket_->send_json(typed("ForceEndpoint")); }
|
||||
void TranscriptStream::keep_alive() { socket_->send_json(typed("KeepAlive")); }
|
||||
void TranscriptStream::terminate() { socket_->send_json(typed("Terminate")); }
|
||||
|
||||
std::optional<StreamTurn> TranscriptStream::as_turn(const SocketEvent& ev) {
|
||||
if (ev.kind != SocketEvent::Kind::kText || !ev.json_ok || ev.type != "Turn") return std::nullopt;
|
||||
StreamTurn t;
|
||||
t.raw = ev.json;
|
||||
t.transcript = str_at(ev.json, "transcript");
|
||||
if (const Json* x = ev.json.find("end_of_turn"); x && x->is_bool()) t.end_of_turn = x->as_bool();
|
||||
if (const Json* x = ev.json.find("turn_is_formatted"); x && x->is_bool()) t.formatted = x->as_bool();
|
||||
if (const Json* x = ev.json.find("turn_order"); x && x->is_number()) t.turn_order = x->as_int();
|
||||
if (const Json* x = ev.json.find("end_of_turn_confidence"); x && x->is_number()) t.end_of_turn_confidence = x->as_number();
|
||||
t.speaker_label = str_at(ev.json, "speaker_label");
|
||||
return t;
|
||||
}
|
||||
|
||||
StreamResult TranscriptStream::collect(int max_wait_ms) {
|
||||
StreamResult out;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto ev = socket_->next(max_wait_ms);
|
||||
if (!ev) return out;
|
||||
++out.events;
|
||||
if (ev->kind == SocketEvent::Kind::kClose) {
|
||||
out.closed = true;
|
||||
return out;
|
||||
}
|
||||
if (!ev->json_ok) continue;
|
||||
if (ev->type == "Begin") {
|
||||
out.session_id = str_at(ev->json, "id");
|
||||
} else if (ev->type == "SpeechStarted") {
|
||||
++out.speech_started;
|
||||
} else if (ev->type == "Turn") {
|
||||
auto t = as_turn(*ev);
|
||||
if (t && t->end_of_turn) {
|
||||
// With format_turns the server repeats the final turn once formatted: keep the formatted one.
|
||||
if (!out.final_turns.empty() && t->formatted) {
|
||||
std::string& last = out.final_turns.back();
|
||||
std::size_t at = out.transcript.rfind(last);
|
||||
last = t->transcript;
|
||||
if (at != std::string::npos) out.transcript = out.transcript.substr(0, at) + last;
|
||||
} else if (!t->transcript.empty()) {
|
||||
out.final_turns.push_back(t->transcript);
|
||||
out.transcript += (out.transcript.empty() ? "" : " ") + t->transcript;
|
||||
}
|
||||
out.partial.clear();
|
||||
} else if (t) {
|
||||
out.partial = t->transcript;
|
||||
}
|
||||
} else if (ev->type == "Termination") {
|
||||
out.terminated = true;
|
||||
if (const Json* d = ev->json.find("audio_duration_seconds"); d && d->is_number()) out.audio_duration_seconds = d->as_number();
|
||||
return out;
|
||||
} else if (ev->json.find("error")) {
|
||||
out.error = str_at(ev->json, "error");
|
||||
if (out.error.empty()) out.error = "stream error";
|
||||
return out;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+81
@@ -0,0 +1,81 @@
|
||||
#pragma once
|
||||
|
||||
// AssemblyAI streaming speech to text (v3): wss://streaming.assemblyai.com/v3/ws.
|
||||
// Messages follow https://www.assemblyai.com/docs/api-reference/specs/streaming.yaml:
|
||||
// client sends binary audio plus UpdateConfiguration / ForceEndpoint / Terminate /
|
||||
// KeepAlive, server sends Begin / SpeechStarted / Turn / SpeakerRevision /
|
||||
// Termination / Heartbeat / LLMGatewayResponse.
|
||||
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/assemblyai/client.h"
|
||||
#include "vendors/audio/audio_service.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
struct StreamParams {
|
||||
// name -> value; names are checked against the channel's query parameters.
|
||||
std::vector<std::pair<std::string, std::string>> pairs;
|
||||
std::string token; // temporary token instead of the Authorization header (browser style)
|
||||
StreamParams& add(const std::string& name, const std::string& value) {
|
||||
pairs.emplace_back(name, value);
|
||||
return *this;
|
||||
}
|
||||
};
|
||||
|
||||
struct StreamTurn {
|
||||
std::string transcript;
|
||||
bool end_of_turn = false;
|
||||
bool formatted = false;
|
||||
long turn_order = -1;
|
||||
double end_of_turn_confidence = 0;
|
||||
std::string speaker_label;
|
||||
Json raw;
|
||||
};
|
||||
|
||||
struct StreamResult {
|
||||
std::string session_id;
|
||||
std::vector<std::string> final_turns; // end_of_turn transcripts (formatted ones when format_turns is on)
|
||||
std::string transcript; // joined by a space
|
||||
std::string partial;
|
||||
double audio_duration_seconds = -1; // from Termination
|
||||
std::size_t speech_started = 0;
|
||||
std::string error;
|
||||
bool terminated = false;
|
||||
bool closed = false;
|
||||
std::size_t events = 0;
|
||||
};
|
||||
|
||||
class TranscriptStream {
|
||||
public:
|
||||
TranscriptStream(std::string api_key, StreamParams params, std::string url = "", SocketOptions socket = {});
|
||||
|
||||
// The URL without a temporary token (safe to log).
|
||||
std::string url() const { return public_url_; }
|
||||
void connect() { socket_->connect(); }
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
void send_audio(const std::string& audio) { socket_->send_binary(audio); }
|
||||
void update_configuration(const Json& fields); // {"type":"UpdateConfiguration",...}
|
||||
void force_endpoint(); // {"type":"ForceEndpoint"}
|
||||
void keep_alive(); // {"type":"KeepAlive"}
|
||||
void terminate(); // {"type":"Terminate"}
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
static std::optional<StreamTurn> as_turn(const SocketEvent& ev);
|
||||
// Reads until Termination, error, close, or silence of `max_wait_ms`.
|
||||
StreamResult collect(int max_wait_ms);
|
||||
|
||||
static const std::vector<std::string>& query_params();
|
||||
static Vocabulary vocabulary();
|
||||
|
||||
private:
|
||||
std::string public_url_;
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+183
@@ -0,0 +1,183 @@
|
||||
#include "vendors/audio/assemblyai/transcripts.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
using C = AssemblyCalls;
|
||||
|
||||
namespace {
|
||||
void tri(Json& o, const char* name, int v) {
|
||||
if (v >= 0) o.set(name, Json(v == 1));
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::string Transcripts::upload(const std::string& audio) {
|
||||
if (audio.size() > AssemblyCaps::kMaxUploadBytes) bad("audio is over the byte cap");
|
||||
llm::RawResponse raw = call_bytes(C::kUpload, {}, audio, "application/octet-stream");
|
||||
Json doc;
|
||||
try {
|
||||
doc = Json::parse(raw.body);
|
||||
} catch (const std::exception& e) {
|
||||
throw llm::ApiError("assemblyai", 502, "BAD_RESPONSE", e.what());
|
||||
}
|
||||
const Json* url = doc.is_object() ? doc.find("upload_url") : nullptr;
|
||||
if (!url || !url->is_string()) throw llm::ApiError("assemblyai", 502, "BAD_RESPONSE", "upload_url missing");
|
||||
return url->as_string();
|
||||
}
|
||||
|
||||
Json Transcripts::build(const TranscriptRequest& r) const {
|
||||
if (r.audio_url.rfind("https://", 0) != 0 && r.audio_url.rfind("http://", 0) != 0)
|
||||
bad("audio_url must be http or https (use upload() for local audio)");
|
||||
require_text("audio_url", r.audio_url, AudioCaps::kMaxFieldChars);
|
||||
Json o = Json::object();
|
||||
o.set("audio_url", Json(r.audio_url));
|
||||
if (!r.speech_models.empty()) {
|
||||
if (r.speech_models.size() > 8) bad("at most 8 speech_models");
|
||||
Json arr = Json::array();
|
||||
for (const auto& m : r.speech_models) {
|
||||
require_text("speech model", m, 64);
|
||||
arr.push(Json(m));
|
||||
}
|
||||
o.set("speech_models", arr);
|
||||
}
|
||||
if (!r.language_code.empty()) {
|
||||
require_text("language_code", r.language_code, 16);
|
||||
o.set("language_code", Json(r.language_code));
|
||||
}
|
||||
tri(o, "language_detection", r.language_detection);
|
||||
tri(o, "punctuate", r.punctuate);
|
||||
tri(o, "format_text", r.format_text);
|
||||
tri(o, "disfluencies", r.disfluencies);
|
||||
tri(o, "multichannel", r.multichannel);
|
||||
tri(o, "speaker_labels", r.speaker_labels);
|
||||
if (r.speakers_expected < 0 || r.speakers_expected > 100) bad("speakers_expected must be 0 or 1..100");
|
||||
if (r.speakers_expected > 0) o.set("speakers_expected", Json(r.speakers_expected));
|
||||
if (!r.keyterms_prompt.empty()) {
|
||||
if (r.keyterms_prompt.size() > AssemblyCaps::kMaxKeyterms) bad("too many keyterms_prompt entries");
|
||||
Json arr = Json::array();
|
||||
for (const auto& k : r.keyterms_prompt) {
|
||||
require_text("keyterm", k, 100);
|
||||
arr.push(Json(k));
|
||||
}
|
||||
o.set("keyterms_prompt", arr);
|
||||
}
|
||||
if (!r.prompt.empty()) {
|
||||
require_text("prompt", r.prompt, 10000);
|
||||
o.set("prompt", Json(r.prompt));
|
||||
}
|
||||
tri(o, "auto_chapters", r.auto_chapters);
|
||||
tri(o, "sentiment_analysis", r.sentiment_analysis);
|
||||
tri(o, "entity_detection", r.entity_detection);
|
||||
tri(o, "summarization", r.summarization);
|
||||
if (!r.webhook_url.empty()) {
|
||||
if (r.webhook_url.rfind("https://", 0) != 0) bad("webhook_url must be https");
|
||||
require_text("webhook_url", r.webhook_url, AudioCaps::kMaxFieldChars);
|
||||
o.set("webhook_url", Json(r.webhook_url));
|
||||
}
|
||||
if (r.extra.is_object())
|
||||
for (const auto& [k, v] : r.extra.as_object()) {
|
||||
if (o.find(k)) bad("extra repeats the field '" + k + "'");
|
||||
o.set(k, v);
|
||||
}
|
||||
check_body(table_.get(C::kSubmit), &o);
|
||||
return o;
|
||||
}
|
||||
|
||||
Json Transcripts::submit(const TranscriptRequest& r) {
|
||||
Json body = build(r);
|
||||
return call(C::kSubmit, {}, &body);
|
||||
}
|
||||
|
||||
Json Transcripts::get(const std::string& id) {
|
||||
check_id("transcript", id);
|
||||
return call(C::kGet, {id});
|
||||
}
|
||||
|
||||
Json Transcripts::list(const Json& q) {
|
||||
if (q.is_object()) {
|
||||
const Json* l = q.find("limit");
|
||||
if (l && l->is_number() && (l->as_int() < 1 || l->as_int() > AssemblyCaps::kMaxListLimit))
|
||||
bad("limit must be 1..200");
|
||||
}
|
||||
return call(C::kList, {}, nullptr, query_from(C::kList, q));
|
||||
}
|
||||
|
||||
Json Transcripts::remove(const std::string& id, bool confirm) {
|
||||
check_id("transcript", id);
|
||||
return call_destructive(C::kDelete, {id}, nullptr, Query(), confirm);
|
||||
}
|
||||
|
||||
Json Transcripts::sentences(const std::string& id) {
|
||||
check_id("transcript", id);
|
||||
return call(C::kSentences, {id});
|
||||
}
|
||||
|
||||
Json Transcripts::paragraphs(const std::string& id) {
|
||||
check_id("transcript", id);
|
||||
return call(C::kParagraphs, {id});
|
||||
}
|
||||
|
||||
Json Transcripts::word_search(const std::string& id, const std::vector<std::string>& words) {
|
||||
check_id("transcript", id);
|
||||
if (words.empty() || words.size() > AssemblyCaps::kMaxSearchWords) bad("words must hold 1..50 entries");
|
||||
std::string joined;
|
||||
for (const auto& w : words) {
|
||||
require_text("word", w, 100);
|
||||
if (w.find(',') != std::string::npos) bad("words must not contain commas");
|
||||
joined += (joined.empty() ? "" : ",") + w;
|
||||
}
|
||||
Query q;
|
||||
q.add("words", joined);
|
||||
return call(C::kWordSearch, {id}, nullptr, q);
|
||||
}
|
||||
|
||||
std::string Transcripts::subtitles(const std::string& id, const std::string& format, int chars_per_caption) {
|
||||
check_id("transcript", id);
|
||||
if (format != "srt" && format != "vtt") bad("format must be srt or vtt");
|
||||
Query q;
|
||||
if (chars_per_caption != 0) {
|
||||
if (chars_per_caption < 1 || chars_per_caption > 1000) bad("chars_per_caption must be 0 or 1..1000");
|
||||
q.add_int("chars_per_caption", chars_per_caption);
|
||||
}
|
||||
return call_raw(C::kSubtitles, {id, format}, nullptr, q).body;
|
||||
}
|
||||
|
||||
Json Transcripts::redacted_audio(const std::string& id) {
|
||||
check_id("transcript", id);
|
||||
return call(C::kRedactedAudio, {id});
|
||||
}
|
||||
|
||||
Json Transcripts::wait(const std::string& id, const PollPolicy& policy, const std::function<void(int)>& sleep_ms) {
|
||||
check_id("transcript", id);
|
||||
Json last;
|
||||
std::string status;
|
||||
PollOutcome out = poll(
|
||||
policy,
|
||||
[&](int) {
|
||||
last = get(id);
|
||||
status = Service::string_at(last, "status");
|
||||
return status == "completed" || status == "error";
|
||||
},
|
||||
sleep_ms);
|
||||
if (!out.done)
|
||||
throw llm::ApiError("assemblyai", 408, "poll_timeout",
|
||||
"transcript " + id + " is still " + (status.empty() ? "pending" : status) + " after " +
|
||||
std::to_string(out.waited_ms / 1000) + " s");
|
||||
if (status == "error")
|
||||
throw llm::ApiError("assemblyai", 422, "transcript_error", Service::string_at(last, "error", "transcription failed"));
|
||||
return last;
|
||||
}
|
||||
|
||||
Json Transcripts::streaming_token(int expires, int max_session) {
|
||||
if (expires < 1 || expires > 600) bad("expires_in_seconds must be 1..600");
|
||||
Query q;
|
||||
q.add_int("expires_in_seconds", expires);
|
||||
if (max_session != 0) {
|
||||
if (max_session < 60 || max_session > 10800) bad("max_session_duration_seconds must be 60..10800");
|
||||
q.add_int("max_session_duration_seconds", max_session);
|
||||
}
|
||||
return call(C::kStreamingToken, {}, nullptr, q);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
#pragma once
|
||||
|
||||
// Pre-recorded transcription: upload, submit, poll, read, list, delete, and
|
||||
// the derived reads (sentences, paragraphs, word search, subtitles, redacted
|
||||
// audio).
|
||||
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/assemblyai/calls.h"
|
||||
#include "vendors/audio/assemblyai/client.h"
|
||||
#include "vendors/audio/audio_service.h"
|
||||
#include "vendors/audio/speech.h"
|
||||
|
||||
namespace molo::audio::assemblyai {
|
||||
|
||||
struct AssemblyCaps {
|
||||
static constexpr std::size_t kMaxUploadBytes = 128u * 1024u * 1024u; // our ceiling (provider allows 5 GB)
|
||||
static constexpr std::size_t kMaxWordBoost = 1000;
|
||||
static constexpr std::size_t kMaxKeyterms = 1000; // keyterms_prompt limit in the docs is 1000 for pre-recorded
|
||||
static constexpr int kMaxListLimit = 200;
|
||||
static constexpr std::size_t kMaxSearchWords = 50;
|
||||
};
|
||||
|
||||
struct TranscriptRequest {
|
||||
std::string audio_url; // required; upload() returns one
|
||||
std::vector<std::string> speech_models; // e.g. universal-3-pro, universal-2 (priority order)
|
||||
std::string language_code; // "" with language_detection
|
||||
int language_detection = -1;
|
||||
int punctuate = -1, format_text = -1, disfluencies = -1, multichannel = -1;
|
||||
int speaker_labels = -1;
|
||||
int speakers_expected = 0; // 0 unset, else 1..100
|
||||
std::vector<std::string> keyterms_prompt; // <= kMaxKeyterms
|
||||
std::string prompt;
|
||||
int auto_chapters = -1, sentiment_analysis = -1, entity_detection = -1, summarization = -1;
|
||||
std::string webhook_url;
|
||||
Json extra; // any other createTranscript field: object, validated against the table
|
||||
};
|
||||
|
||||
class Transcripts : public AudioService {
|
||||
public:
|
||||
explicit Transcripts(std::shared_ptr<const Hosts> hosts)
|
||||
: AudioService(calls(), std::move(hosts)) {}
|
||||
Transcripts() : Transcripts(AssemblyEndpoints::from_env()) {}
|
||||
|
||||
// Uploads bytes; returns the upload_url to use as audio_url.
|
||||
std::string upload(const std::string& audio);
|
||||
Json build(const TranscriptRequest& request) const;
|
||||
Json submit(const TranscriptRequest& request);
|
||||
Json get(const std::string& transcript_id);
|
||||
Json list(const Json& query_object = Json());
|
||||
Json remove(const std::string& transcript_id, bool confirm);
|
||||
Json sentences(const std::string& transcript_id);
|
||||
Json paragraphs(const std::string& transcript_id);
|
||||
Json word_search(const std::string& transcript_id, const std::vector<std::string>& words);
|
||||
std::string subtitles(const std::string& transcript_id, const std::string& format, int chars_per_caption = 0);
|
||||
Json redacted_audio(const std::string& transcript_id);
|
||||
|
||||
// Polls get() until status is completed or error, within the policy.
|
||||
// `sleep_ms` null = real sleep. Returns the final document; throws
|
||||
// llm::ApiError (408, "poll_timeout") when the cap is reached first and the
|
||||
// transcript is still queued or processing, and (422) when status is error.
|
||||
Json wait(const std::string& transcript_id, const PollPolicy& policy,
|
||||
const std::function<void(int)>& sleep_ms = nullptr);
|
||||
// Streaming session token (temporary, for browsers).
|
||||
Json streaming_token(int expires_in_seconds, int max_session_duration_seconds = 0);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::assemblyai
|
||||
Vendored
+211
@@ -0,0 +1,211 @@
|
||||
#include "vendors/audio/audio_service.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
AudioService::AudioService(const CallTable& table, std::shared_ptr<const Hosts> hosts)
|
||||
: llm::Service(hosts->first().api),
|
||||
table_(table),
|
||||
hosts_(std::move(hosts)),
|
||||
vendor_(table.provider()) {}
|
||||
|
||||
void AudioService::bad(const std::string& why) const { throw llm::ApiError::invalid(vendor_, why); }
|
||||
|
||||
const AudioHost& AudioService::host_of(const CallSpec& spec) const {
|
||||
return hosts_->get(spec.host, vendor_);
|
||||
}
|
||||
|
||||
bool AudioService::configured_for(const std::string& id) const {
|
||||
return host_of(table_.get(id)).configured();
|
||||
}
|
||||
|
||||
void AudioService::require_text(const std::string& what, const std::string& s, std::size_t max_chars,
|
||||
bool required) const {
|
||||
if (required && s.empty()) bad(what + " is required");
|
||||
if (s.size() > max_chars) bad(what + " is over " + std::to_string(max_chars) + " characters");
|
||||
for (unsigned char c : s)
|
||||
if (c == 0) bad(what + " has a NUL byte");
|
||||
}
|
||||
|
||||
void AudioService::require_range(const std::string& what, double v, double lo, double hi) const {
|
||||
if (!(v >= lo && v <= hi)) bad(what + " must be in " + std::to_string(lo) + ".." + std::to_string(hi));
|
||||
}
|
||||
|
||||
void AudioService::require_confirm(const std::string& what, bool confirm) const {
|
||||
if (!confirm) bad(what + " is destructive or costs money: pass confirm=true");
|
||||
}
|
||||
|
||||
void AudioService::check_body(const CallSpec& spec, const Json* body) const {
|
||||
std::string kind = spec.body;
|
||||
if (kind != "json") {
|
||||
if (body) bad(std::string(spec.id) + " takes no JSON body");
|
||||
return;
|
||||
}
|
||||
auto required = csv_list(spec.required);
|
||||
if (!body) {
|
||||
if (!required.empty()) bad(std::string(spec.id) + " needs a JSON body");
|
||||
return;
|
||||
}
|
||||
if (!body->is_object()) bad("body must be a JSON object");
|
||||
if (body->dump().size() > AudioCaps::kMaxJsonBodyBytes) bad("body is over the byte cap");
|
||||
std::string fields = spec.fields;
|
||||
if (fields != "*") {
|
||||
for (const auto& [k, v] : body->as_object()) {
|
||||
(void)v;
|
||||
if (!csv_has(spec.fields, k)) bad("unknown body field '" + k + "' for " + spec.id);
|
||||
}
|
||||
}
|
||||
for (const auto& r : required) {
|
||||
const Json* v = body->find(r);
|
||||
if (!v || v->is_null()) bad("missing required field '" + r + "' for " + spec.id);
|
||||
}
|
||||
}
|
||||
|
||||
void AudioService::check_form(const CallSpec& spec, const clients::body::MultipartBody& form) const {
|
||||
if (std::string(spec.body) != "form") bad(std::string(spec.id) + " is not a multipart call");
|
||||
if (form.encoded_size() > AudioCaps::kMaxAudioInBytes) bad("upload is over the byte cap");
|
||||
std::string fields = spec.fields;
|
||||
for (const auto& part : form.parts()) {
|
||||
if (fields != "*" && !csv_has(spec.fields, part.name))
|
||||
bad("unknown form field '" + part.name + "' for " + spec.id);
|
||||
}
|
||||
for (const auto& r : csv_list(spec.required)) {
|
||||
bool found = false;
|
||||
for (const auto& part : form.parts()) found = found || part.name == r;
|
||||
if (!found) bad("missing required form field '" + r + "' for " + spec.id);
|
||||
}
|
||||
}
|
||||
|
||||
void AudioService::check_query(const CallSpec& spec, const Query& query) const {
|
||||
for (const auto& [k, v] : query.pairs()) {
|
||||
(void)v;
|
||||
if (!csv_has(spec.query, k)) bad("unknown query parameter '" + k + "' for " + spec.id);
|
||||
}
|
||||
}
|
||||
|
||||
Query AudioService::query_from(const std::string& id, const Json& object) const {
|
||||
const CallSpec& spec = table_.get(id);
|
||||
Query q;
|
||||
if (object.is_null()) return q;
|
||||
if (!object.is_object()) bad("query must be a JSON object");
|
||||
for (const auto& [k, v] : object.as_object()) {
|
||||
if (!csv_has(spec.query, k)) bad("unknown query parameter '" + k + "' for " + id);
|
||||
if (!q.add_json(k, v)) bad("query parameter '" + k + "' must be a scalar or a list of scalars");
|
||||
}
|
||||
return q;
|
||||
}
|
||||
|
||||
std::string AudioService::path_of(const CallSpec& spec, std::initializer_list<std::string> segments,
|
||||
const Query& query) const {
|
||||
check_query(spec, query);
|
||||
std::string path = render_path(vendor_, spec, segments);
|
||||
try {
|
||||
return path + query.str();
|
||||
} catch (const std::length_error&) {
|
||||
bad("query string is over the cap");
|
||||
}
|
||||
}
|
||||
|
||||
std::string AudioService::preview(const std::string& id, std::initializer_list<std::string> segments,
|
||||
const Json* body, const Query& query) const {
|
||||
const CallSpec& spec = table_.get(id);
|
||||
check_body(spec, body);
|
||||
return std::string(spec.method) + " " + path_of(spec, segments, query);
|
||||
}
|
||||
|
||||
Json AudioService::call(const std::string& id, std::initializer_list<std::string> segments,
|
||||
const Json* body, const Query& query) {
|
||||
const CallSpec& spec = table_.get(id);
|
||||
if (is_destructive(spec)) bad(std::string(spec.id) + " is destructive: use call_destructive");
|
||||
std::string kind = spec.response;
|
||||
if (kind == "bytes" || kind == "text") bad(std::string(spec.id) + " returns " + kind + ": use call_raw");
|
||||
if (std::string(spec.body) == "form" || std::string(spec.body) == "bytes")
|
||||
bad(std::string(spec.id) + " needs a " + spec.body + " body: use call_form / call_bytes");
|
||||
check_body(spec, body);
|
||||
std::string path = path_of(spec, segments, query);
|
||||
return host_of(spec).api->request(method_of(spec), path, body);
|
||||
}
|
||||
|
||||
llm::RawResponse AudioService::call_raw(const std::string& id,
|
||||
std::initializer_list<std::string> segments,
|
||||
const Json* body, const Query& query) {
|
||||
const CallSpec& spec = table_.get(id);
|
||||
if (is_destructive(spec)) bad(std::string(spec.id) + " is destructive: use call_destructive");
|
||||
if (std::string(spec.body) == "form" || std::string(spec.body) == "bytes")
|
||||
bad(std::string(spec.id) + " needs a " + spec.body + " body");
|
||||
check_body(spec, body);
|
||||
std::string path = path_of(spec, segments, query);
|
||||
std::string payload = body ? body->dump() : std::string();
|
||||
return host_of(spec).api->request_raw(method_of(spec), path, body ? &payload : nullptr,
|
||||
"application/json");
|
||||
}
|
||||
|
||||
llm::RawResponse AudioService::call_bytes(const std::string& id,
|
||||
std::initializer_list<std::string> segments,
|
||||
const std::string& data, const std::string& content_type,
|
||||
const Query& query) {
|
||||
const CallSpec& spec = table_.get(id);
|
||||
if (std::string(spec.body) != "bytes") bad(std::string(spec.id) + " does not take raw bytes");
|
||||
if (data.empty()) bad("audio data is required");
|
||||
if (data.size() > AudioCaps::kMaxAudioInBytes) bad("audio is over the byte cap");
|
||||
if (content_type.empty() || content_type.size() > 128) bad("content type is required");
|
||||
std::string path = path_of(spec, segments, query);
|
||||
return host_of(spec).api->request_raw(method_of(spec), path, &data, content_type);
|
||||
}
|
||||
|
||||
Json AudioService::call_form(const std::string& id, std::initializer_list<std::string> segments,
|
||||
const clients::body::MultipartBody& form, const Query& query) {
|
||||
const CallSpec& spec = table_.get(id);
|
||||
if (is_destructive(spec)) bad(std::string(spec.id) + " is destructive: use call_destructive");
|
||||
check_form(spec, form);
|
||||
std::string path = path_of(spec, segments, query);
|
||||
if (method_of(spec) == clients::Method::kPost) return host_of(spec).api->post_multipart(path, form);
|
||||
// PUT / PATCH multipart: the shared client only has POST, so send the encoded body.
|
||||
std::string payload = form.encode();
|
||||
llm::RawResponse raw = host_of(spec).api->request_raw(method_of(spec), path, &payload, form.content_type());
|
||||
if (raw.body.find_first_not_of(" \t\r\n") == std::string::npos) return Json::object();
|
||||
try {
|
||||
return Json::parse(raw.body);
|
||||
} catch (const std::exception& e) {
|
||||
throw llm::ApiError(vendor_, 502, "BAD_RESPONSE", e.what());
|
||||
}
|
||||
}
|
||||
|
||||
llm::RawResponse AudioService::call_form_raw(const std::string& id,
|
||||
std::initializer_list<std::string> segments,
|
||||
const clients::body::MultipartBody& form,
|
||||
const Query& query) {
|
||||
const CallSpec& spec = table_.get(id);
|
||||
if (is_destructive(spec)) bad(std::string(spec.id) + " is destructive: use call_destructive");
|
||||
check_form(spec, form);
|
||||
std::string path = path_of(spec, segments, query);
|
||||
std::string payload = form.encode();
|
||||
return host_of(spec).api->request_raw(method_of(spec), path, &payload, form.content_type());
|
||||
}
|
||||
|
||||
ByteStreamer::Result AudioService::call_stream(const std::string& id,
|
||||
std::initializer_list<std::string> segments,
|
||||
const Json* body, const Query& query,
|
||||
const ChunkSink& sink) {
|
||||
const CallSpec& spec = table_.get(id);
|
||||
if (std::string(spec.response) != "bytes") bad(std::string(spec.id) + " does not stream bytes");
|
||||
if (is_destructive(spec)) bad(std::string(spec.id) + " is destructive: use call_destructive");
|
||||
check_body(spec, body);
|
||||
std::string path = path_of(spec, segments, query);
|
||||
const AudioHost& host = host_of(spec);
|
||||
return host.streamer->stream(method_of(spec), path, body, sink);
|
||||
}
|
||||
|
||||
Json AudioService::call_destructive(const std::string& id,
|
||||
std::initializer_list<std::string> segments, const Json* body,
|
||||
const Query& query, bool confirm) {
|
||||
const CallSpec& spec = table_.get(id);
|
||||
if (!is_destructive(spec)) bad(std::string(spec.id) + " is not a destructive call");
|
||||
check_body(spec, body);
|
||||
std::string path = path_of(spec, segments, query);
|
||||
require_confirm(spec.id, confirm);
|
||||
return host_of(spec).api->request(method_of(spec), path, body);
|
||||
}
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+86
@@ -0,0 +1,86 @@
|
||||
#pragma once
|
||||
|
||||
// Base of every provider service in this package. Table driven: a call is
|
||||
// named by its id in the provider's CallTable, never by a literal path. The
|
||||
// base validates path segments, body fields, required fields, query
|
||||
// parameters and the destructive guard before anything reaches the network.
|
||||
|
||||
#include <initializer_list>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/form.h"
|
||||
#include "llm/api_client.h"
|
||||
#include "vendors/audio/call_spec.h"
|
||||
#include "vendors/audio/hosts.h"
|
||||
#include "vendors/audio/query.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
class AudioService : public llm::Service {
|
||||
public:
|
||||
AudioService(const CallTable& table, std::shared_ptr<const Hosts> hosts);
|
||||
|
||||
const CallTable& table() const { return table_; }
|
||||
const std::string& vendor() const { return vendor_; }
|
||||
const Hosts& hosts() const { return *hosts_; }
|
||||
// True when the host the call lives on has a key.
|
||||
bool configured_for(const std::string& id) const;
|
||||
|
||||
// JSON in, JSON out. Rows whose response is bytes or text, and destructive
|
||||
// rows, are refused here: use call_raw / call_destructive.
|
||||
Json call(const std::string& id, std::initializer_list<std::string> segments = {},
|
||||
const Json* body = nullptr, const Query& query = Query());
|
||||
// Any row. Returns the buffered response (audio, text, files).
|
||||
llm::RawResponse call_raw(const std::string& id, std::initializer_list<std::string> segments = {},
|
||||
const Json* body = nullptr, const Query& query = Query());
|
||||
// Raw octets as request body (AssemblyAI upload, Deepgram listen).
|
||||
llm::RawResponse call_bytes(const std::string& id, std::initializer_list<std::string> segments,
|
||||
const std::string& data, const std::string& content_type,
|
||||
const Query& query = Query());
|
||||
// multipart/form-data request; response parsed as JSON.
|
||||
Json call_form(const std::string& id, std::initializer_list<std::string> segments,
|
||||
const clients::body::MultipartBody& form, const Query& query = Query());
|
||||
// multipart/form-data request; response returned buffered (voice changer, isolation).
|
||||
llm::RawResponse call_form_raw(const std::string& id, std::initializer_list<std::string> segments,
|
||||
const clients::body::MultipartBody& form,
|
||||
const Query& query = Query());
|
||||
// Incremental response body (streamed audio). `sink` returns false to stop.
|
||||
ByteStreamer::Result call_stream(const std::string& id,
|
||||
std::initializer_list<std::string> segments, const Json* body,
|
||||
const Query& query, const ChunkSink& sink);
|
||||
// Destructive rows (DELETE, cancel, disable, remove, revoke): explicit, no default.
|
||||
Json call_destructive(const std::string& id, std::initializer_list<std::string> segments,
|
||||
const Json* body, const Query& query, bool confirm);
|
||||
|
||||
// Everything the call would send, without sending it: the final path with
|
||||
// query string. Throws llm::ApiError::invalid exactly like the real call.
|
||||
std::string preview(const std::string& id, std::initializer_list<std::string> segments,
|
||||
const Json* body = nullptr, const Query& query = Query()) const;
|
||||
|
||||
// Checks (throw llm::ApiError::invalid).
|
||||
void check_body(const CallSpec& spec, const Json* body) const;
|
||||
void check_form(const CallSpec& spec, const clients::body::MultipartBody& form) const;
|
||||
void check_query(const CallSpec& spec, const Query& query) const;
|
||||
void require_text(const std::string& what, const std::string& s, std::size_t max_chars,
|
||||
bool required = true) const;
|
||||
void require_range(const std::string& what, double v, double lo, double hi) const;
|
||||
void require_confirm(const std::string& what, bool confirm) const;
|
||||
[[noreturn]] void bad(const std::string& why) const;
|
||||
|
||||
// Converts a JSON object of scalars / arrays to a Query restricted to the
|
||||
// call's declared query parameters (unknown names are refused).
|
||||
Query query_from(const std::string& id, const Json& object) const;
|
||||
|
||||
protected:
|
||||
const AudioHost& host_of(const CallSpec& spec) const;
|
||||
std::string path_of(const CallSpec& spec, std::initializer_list<std::string> segments,
|
||||
const Query& query) const;
|
||||
void guard_method_kind(const CallSpec& spec, const char* want_body) const;
|
||||
|
||||
const CallTable& table_;
|
||||
std::shared_ptr<const Hosts> hosts_;
|
||||
std::string vendor_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+132
@@ -0,0 +1,132 @@
|
||||
#include "vendors/audio/call_spec.h"
|
||||
|
||||
#include <cctype>
|
||||
|
||||
#include "llm/api_client.h"
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
const CallSpec* CallTable::find(const std::string& id) const {
|
||||
for (std::size_t i = 0; i < count_; ++i)
|
||||
if (id == rows_[i].id) return &rows_[i];
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
const CallSpec& CallTable::get(const std::string& id) const {
|
||||
const CallSpec* s = find(id);
|
||||
if (!s) throw llm::ApiError::invalid(provider_, "unknown call id: " + id);
|
||||
return *s;
|
||||
}
|
||||
|
||||
std::vector<std::string> csv_list(const char* csv) {
|
||||
std::vector<std::string> out;
|
||||
std::string cur;
|
||||
for (const char* p = csv; p && *p; ++p) {
|
||||
if (*p == ',') {
|
||||
if (!cur.empty()) out.push_back(cur);
|
||||
cur.clear();
|
||||
} else {
|
||||
cur.push_back(*p);
|
||||
}
|
||||
}
|
||||
if (!cur.empty()) out.push_back(cur);
|
||||
return out;
|
||||
}
|
||||
|
||||
bool csv_has(const char* csv, const std::string& name) {
|
||||
std::string cur;
|
||||
for (const char* p = csv; ; ++p) {
|
||||
if (*p == ',' || *p == '\0') {
|
||||
if (cur == name) return true;
|
||||
cur.clear();
|
||||
if (*p == '\0') return false;
|
||||
} else {
|
||||
cur.push_back(*p);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
clients::Method method_of(const CallSpec& spec) {
|
||||
std::string m = spec.method;
|
||||
if (m == "GET") return clients::Method::kGet;
|
||||
if (m == "POST") return clients::Method::kPost;
|
||||
if (m == "PUT") return clients::Method::kPut;
|
||||
if (m == "PATCH") return clients::Method::kPatch;
|
||||
return clients::Method::kDelete;
|
||||
}
|
||||
|
||||
std::size_t placeholder_count(const CallSpec& spec) {
|
||||
std::size_t n = 0;
|
||||
for (const char* p = spec.path; *p; ++p) n += *p == '{';
|
||||
return n;
|
||||
}
|
||||
|
||||
bool is_destructive(const CallSpec& spec) {
|
||||
if (std::string(spec.method) == "DELETE") return true;
|
||||
std::string id = spec.id;
|
||||
for (char& c : id) c = static_cast<char>(std::tolower(static_cast<unsigned char>(c)));
|
||||
for (const char* word : {"delete", "cancel", "disable", "remove", "revoke", "purge", "leave"})
|
||||
if (id.find(word) != std::string::npos) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
std::string percent(const std::string& s) {
|
||||
static const char* hex = "0123456789ABCDEF";
|
||||
std::string out;
|
||||
for (unsigned char c : s) {
|
||||
if (std::isalnum(c) || c == '-' || c == '_' || c == '.' || c == '~') {
|
||||
out.push_back(static_cast<char>(c));
|
||||
} else {
|
||||
out.push_back('%');
|
||||
out.push_back(hex[c >> 4]);
|
||||
out.push_back(hex[c & 15]);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool plausible_email(const std::string& s) {
|
||||
if (s.size() < 3 || s.size() > 254) return false;
|
||||
std::size_t at = s.find('@');
|
||||
if (at == std::string::npos || at == 0 || at + 1 >= s.size() || s.find('@', at + 1) != std::string::npos)
|
||||
return false;
|
||||
for (unsigned char c : s)
|
||||
if (c <= 32 || c == '/' || c == '\\' || c == '?' || c == '#') return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string render_path(const std::string& vendor, const CallSpec& spec,
|
||||
std::initializer_list<std::string> segments) {
|
||||
std::string path = spec.path;
|
||||
std::string out;
|
||||
auto it = segments.begin();
|
||||
for (std::size_t i = 0; i < path.size(); ++i) {
|
||||
if (path[i] != '{') {
|
||||
out.push_back(path[i]);
|
||||
continue;
|
||||
}
|
||||
std::size_t close = path.find('}', i);
|
||||
if (close == std::string::npos || it == segments.end())
|
||||
throw llm::ApiError::invalid(vendor, "wrong number of path segments for " + std::string(spec.id));
|
||||
std::string name = path.substr(i + 1, close - i - 1);
|
||||
if (name.find("email") != std::string::npos) {
|
||||
if (!plausible_email(*it)) throw llm::ApiError::invalid(vendor, "bad email path segment");
|
||||
out += percent(*it);
|
||||
} else {
|
||||
if (!llm::Service::valid_id(*it)) throw llm::ApiError::invalid(vendor, "bad " + name + " path segment");
|
||||
out += *it;
|
||||
}
|
||||
++it;
|
||||
i = close;
|
||||
}
|
||||
if (it != segments.end())
|
||||
throw llm::ApiError::invalid(vendor, "wrong number of path segments for " + std::string(spec.id));
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+65
@@ -0,0 +1,65 @@
|
||||
#pragma once
|
||||
|
||||
// The call table type shared by every provider. A provider owns exactly one
|
||||
// table (a static array of CallSpec rows in its calls.cpp). Every REST call
|
||||
// the package can make is one row: HTTP method, path template, the host key,
|
||||
// request body kind and field list, query parameter list, response kind and
|
||||
// the spec operation it was taken from. tools/check_spec.py verifies the rows
|
||||
// against the vendor's published specification.
|
||||
|
||||
#include <cstddef>
|
||||
#include <initializer_list>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/request.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
struct CallSpec {
|
||||
const char* id; // operationId (or a stable name where the spec has none)
|
||||
const char* method; // GET POST PUT PATCH DELETE
|
||||
const char* path; // path template exactly as the spec prints it: /v1/x/{id}
|
||||
const char* host; // host key: "api", "gateway", "streaming", ...
|
||||
const char* body; // "-" none | "json" | "form" (multipart) | "bytes" (raw octets)
|
||||
const char* fields; // comma list of top level body fields; "*" = open schema
|
||||
const char* required; // comma list of required body fields
|
||||
const char* query; // comma list of query parameters
|
||||
const char* response; // "json" | "bytes" | "text" | "none"
|
||||
const char* source; // "openapi:<operationId>" | "sdk:<file>" | "docs:<page>"
|
||||
};
|
||||
|
||||
class CallTable {
|
||||
public:
|
||||
CallTable(const char* provider, const CallSpec* rows, std::size_t count)
|
||||
: provider_(provider), rows_(rows), count_(count) {}
|
||||
|
||||
const char* provider() const { return provider_; }
|
||||
std::size_t size() const { return count_; }
|
||||
const CallSpec& at(std::size_t i) const { return rows_[i]; }
|
||||
const CallSpec* find(const std::string& id) const;
|
||||
// Throws llm::ApiError::invalid for an unknown id.
|
||||
const CallSpec& get(const std::string& id) const;
|
||||
|
||||
private:
|
||||
const char* provider_;
|
||||
const CallSpec* rows_;
|
||||
std::size_t count_;
|
||||
};
|
||||
|
||||
// Helpers over the comma lists.
|
||||
std::vector<std::string> csv_list(const char* csv);
|
||||
bool csv_has(const char* csv, const std::string& name);
|
||||
|
||||
clients::Method method_of(const CallSpec& spec);
|
||||
std::size_t placeholder_count(const CallSpec& spec);
|
||||
// True for DELETE and for ids that cancel, disable, remove, revoke or purge.
|
||||
bool is_destructive(const CallSpec& spec);
|
||||
|
||||
// Fills `{placeholders}` in order. Every segment must be a plain id
|
||||
// (llm::Service::valid_id); a placeholder named *email* takes an address and
|
||||
// is percent-encoded. Wrong count or bad segment: llm::ApiError::invalid.
|
||||
std::string render_path(const std::string& vendor, const CallSpec& spec,
|
||||
std::initializer_list<std::string> segments);
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+44
@@ -0,0 +1,44 @@
|
||||
#pragma once
|
||||
|
||||
// Static ceilings for the audio vendor package. Never autotuned. Provider
|
||||
// limits that are documented are cited in SPEC.md; everything here is our own
|
||||
// ceiling and sits at or below the provider's.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
struct AudioCaps {
|
||||
// Text and ids.
|
||||
static constexpr std::size_t kMaxTextChars = 100000; // one synthesis input
|
||||
static constexpr std::size_t kMaxChunkChars = 5000; // one chunk of the splitter
|
||||
static constexpr std::size_t kMaxChunks = 64; // chunks per splitter call
|
||||
static constexpr std::size_t kMaxFieldChars = 8192; // names, prompts, urls
|
||||
static constexpr std::size_t kMaxListItems = 1000; // items in a request list
|
||||
static constexpr std::size_t kMaxQueryChars = 8000; // rendered query string
|
||||
// Payloads.
|
||||
static constexpr std::size_t kMaxJsonBodyBytes = 8u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxAudioInBytes = 128u * 1024u * 1024u; // upload / inline audio
|
||||
static constexpr std::size_t kMaxAudioOutBytes = 64u * 1024u * 1024u; // synthesized audio kept
|
||||
static constexpr std::size_t kErrorProbeBytes = 64u * 1024u; // json error sniff on streams
|
||||
// Paging and polling.
|
||||
static constexpr std::size_t kMaxPages = 20;
|
||||
static constexpr int kMaxPollSeconds = 1800;
|
||||
static constexpr int kMinPollIntervalMs = 500; // floor: never poll faster
|
||||
static constexpr int kMaxPollIntervalMs = 15000;
|
||||
// WebSocket sessions.
|
||||
static constexpr int kConnectTimeoutMs = 10000;
|
||||
static constexpr int kHandshakeTimeoutMs = 10000;
|
||||
static constexpr int kIoTimeoutMs = 5000;
|
||||
static constexpr int kPollSliceMs = 200;
|
||||
static constexpr int kMaxSessionSeconds = 1800;
|
||||
static constexpr int kMaxWaitMs = 120000; // one next()/collect() wait
|
||||
static constexpr std::size_t kMaxEventsPerCollect = 4096;
|
||||
static constexpr std::size_t kMaxSocketFrameBytes = 4u * 1024u * 1024u; // one frame we send
|
||||
static constexpr std::size_t kMaxStreamAudioBytes = 64u * 1024u * 1024u; // audio kept per session
|
||||
// Provider specific, all documented in the module SPEC.
|
||||
static constexpr std::size_t kMaxKeywords = 100;
|
||||
static constexpr std::size_t kMaxSpeakers = 64;
|
||||
};
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+72
@@ -0,0 +1,72 @@
|
||||
#include "vendors/audio/cartesia/calls.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
// One row per REST call. tools/check_spec.py parses the block between the
|
||||
// markers: {id, method, path, host, body, fields, required, query, response, source}.
|
||||
// CALLS-BEGIN
|
||||
static const CallSpec kRows[] = {
|
||||
{"get_status", "GET", "/", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /"},
|
||||
{"agents_retrieve", "GET", "/agents/{agent_id}", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /agents/{agent_id}"},
|
||||
{"agents_update", "PATCH", "/agents/{agent_id}", "api", "json", "description,name,tts_language,tts_voice", "", "", "json", "sdk:cartesia-python api.md patch /agents/{agent_id}"},
|
||||
{"agents_list", "GET", "/agents", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /agents"},
|
||||
{"agents_delete", "DELETE", "/agents/{agent_id}", "api", "-", "", "", "", "none", "sdk:cartesia-python api.md delete /agents/{agent_id}"},
|
||||
{"agents_list_phone_numbers", "GET", "/agents/{agent_id}/phone-numbers", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /agents/{agent_id}/phone-numbers"},
|
||||
{"agents_list_templates", "GET", "/agents/templates", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /agents/templates"},
|
||||
{"calls_retrieve", "GET", "/agents/calls/{call_id}", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /agents/calls/{call_id}"},
|
||||
{"calls_list", "GET", "/agents/calls", "api", "-", "", "", "agent_id,ending_before,expand,limit,starting_after", "json", "sdk:cartesia-python api.md get /agents/calls"},
|
||||
{"calls_download_audio", "GET", "/agents/calls/{call_id}/audio", "api", "-", "", "", "", "bytes", "sdk:cartesia-python api.md get /agents/calls/{call_id}/audio"},
|
||||
{"metrics_create", "POST", "/agents/metrics", "api", "json", "display_name,name,prompt", "name,prompt", "", "json", "sdk:cartesia-python api.md post /agents/metrics"},
|
||||
{"metrics_retrieve", "GET", "/agents/metrics/{metric_id}", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /agents/metrics/{metric_id}"},
|
||||
{"metrics_list", "GET", "/agents/metrics", "api", "-", "", "", "limit,starting_after", "json", "sdk:cartesia-python api.md get /agents/metrics"},
|
||||
{"metrics_add_to_agent", "POST", "/agents/{agent_id}/metrics/{metric_id}", "api", "-", "", "", "", "none", "sdk:cartesia-python api.md post /agents/{agent_id}/metrics/{metric_id}"},
|
||||
{"metrics_remove_from_agent", "DELETE", "/agents/{agent_id}/metrics/{metric_id}", "api", "-", "", "", "", "none", "sdk:cartesia-python api.md delete /agents/{agent_id}/metrics/{metric_id}"},
|
||||
{"metrics_results_list", "GET", "/agents/metrics/results", "api", "-", "", "", "agent_id,call_id,deployment_id,end_date,ending_before,limit,metric_id,start_date,starting_after", "json", "sdk:cartesia-python api.md get /agents/metrics/results"},
|
||||
{"metrics_results_export", "GET", "/agents/metrics/results/export", "api", "-", "", "", "agent_id,call_id,deployment_id,end_date,metric_id,start_date", "text", "sdk:cartesia-python api.md get /agents/metrics/results/export"},
|
||||
{"deployments_retrieve", "GET", "/agents/deployments/{deployment_id}", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /agents/deployments/{deployment_id}"},
|
||||
{"deployments_list", "GET", "/agents/{agent_id}/deployments", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /agents/{agent_id}/deployments"},
|
||||
{"access_token_create", "POST", "/access-token", "api", "json", "expires_in,grants", "", "", "json", "sdk:cartesia-python api.md post /access-token"},
|
||||
{"datasets_create", "POST", "/datasets/", "api", "json", "description,name", "description,name", "", "json", "sdk:cartesia-python api.md post /datasets/"},
|
||||
{"datasets_retrieve", "GET", "/datasets/{id}", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /datasets/{id}"},
|
||||
{"datasets_update", "PATCH", "/datasets/{id}", "api", "json", "description,name", "description,name", "", "none", "sdk:cartesia-python api.md patch /datasets/{id}"},
|
||||
{"datasets_list", "GET", "/datasets/", "api", "-", "", "", "ending_before,limit,starting_after", "json", "sdk:cartesia-python api.md get /datasets/"},
|
||||
{"datasets_delete", "DELETE", "/datasets/{id}", "api", "-", "", "", "", "none", "sdk:cartesia-python api.md delete /datasets/{id}"},
|
||||
{"datasets_files_list", "GET", "/datasets/{id}/files", "api", "-", "", "", "ending_before,limit,starting_after", "json", "sdk:cartesia-python api.md get /datasets/{id}/files"},
|
||||
{"datasets_files_delete", "DELETE", "/datasets/{id}/files/{file_id}", "api", "-", "", "", "", "none", "sdk:cartesia-python api.md delete /datasets/{id}/files/{fileID}"},
|
||||
{"datasets_files_upload", "POST", "/datasets/{id}/files", "api", "form", "file,purpose", "file", "", "none", "sdk:cartesia-python api.md post /datasets/{id}/files"},
|
||||
{"fine_tunes_create", "POST", "/fine-tunes/", "api", "json", "accent,dataset,description,language,model_id,name", "dataset,description,language,model_id,name", "", "json", "sdk:cartesia-python api.md post /fine-tunes/"},
|
||||
{"fine_tunes_retrieve", "GET", "/fine-tunes/{id}", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /fine-tunes/{id}"},
|
||||
{"fine_tunes_list", "GET", "/fine-tunes/", "api", "-", "", "", "ending_before,limit,starting_after", "json", "sdk:cartesia-python api.md get /fine-tunes/"},
|
||||
{"fine_tunes_delete", "DELETE", "/fine-tunes/{id}", "api", "-", "", "", "", "none", "sdk:cartesia-python api.md delete /fine-tunes/{id}"},
|
||||
{"fine_tunes_list_voices", "GET", "/fine-tunes/{id}/voices", "api", "-", "", "", "ending_before,limit,starting_after", "json", "sdk:cartesia-python api.md get /fine-tunes/{id}/voices"},
|
||||
{"pronunciation_dicts_create", "POST", "/pronunciation-dicts/", "api", "json", "items,name", "name", "", "json", "sdk:cartesia-python api.md post /pronunciation-dicts/"},
|
||||
{"pronunciation_dicts_retrieve", "GET", "/pronunciation-dicts/{id}", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /pronunciation-dicts/{id}"},
|
||||
{"pronunciation_dicts_update", "PATCH", "/pronunciation-dicts/{id}", "api", "json", "items,name", "", "", "json", "sdk:cartesia-python api.md patch /pronunciation-dicts/{id}"},
|
||||
{"pronunciation_dicts_list", "GET", "/pronunciation-dicts/", "api", "-", "", "", "ending_before,limit,starting_after", "json", "sdk:cartesia-python api.md get /pronunciation-dicts/"},
|
||||
{"pronunciation_dicts_delete", "DELETE", "/pronunciation-dicts/{id}", "api", "-", "", "", "", "none", "sdk:cartesia-python api.md delete /pronunciation-dicts/{id}"},
|
||||
{"stt_transcribe", "POST", "/stt", "api", "form", "file,model,encoding,sample_rate,language,timestamp_granularities[]", "file,model", "", "json", "sdk:cartesia-python api.md post /stt"},
|
||||
{"tts_generate", "POST", "/tts/bytes", "api", "json", "model_id,output_format,transcript,voice,generation_config,language,locale,normalization,pronunciation_dict_id,save,speed", "model_id,output_format,transcript,voice", "", "bytes", "sdk:cartesia-python api.md post /tts/bytes"},
|
||||
{"tts_generate_sse", "POST", "/tts/sse", "api", "json", "model_id,output_format,transcript,voice,add_phoneme_timestamps,add_timestamps,context_id,generation_config,language,locale,normalization,pronunciation_dict_id,speed,use_normalized_timestamps", "model_id,output_format,transcript,voice", "", "bytes", "sdk:cartesia-python api.md post /tts/sse"},
|
||||
{"tts_infill", "POST", "/infill/bytes", "api", "form", "language,left_audio,model_id,output_format[container],output_format[encoding],output_format[sample_rate],output_format[bit_rate],right_audio,transcript,voice_id", "left_audio,right_audio,transcript,voice_id", "", "bytes", "sdk:cartesia-python api.md post /infill/bytes"},
|
||||
{"voice_changer_generate", "POST", "/voice-changer/bytes", "api", "form", "clip,output_format[bit_rate],output_format[container],output_format[encoding],output_format[sample_rate],voice[id]", "clip,voice[id]", "", "bytes", "sdk:cartesia-python api.md post /voice-changer/bytes"},
|
||||
{"voice_changer_generate_sse", "POST", "/voice-changer/sse", "api", "form", "clip,output_format[bit_rate],output_format[container],output_format[encoding],output_format[sample_rate],voice[id]", "clip,voice[id]", "", "bytes", "sdk:cartesia-python api.md post /voice-changer/sse"},
|
||||
{"voices_update", "PATCH", "/voices/{id}", "api", "json", "accent,description,gender,name", "", "", "json", "sdk:cartesia-python api.md patch /voices/{id}"},
|
||||
{"voices_list", "GET", "/voices", "api", "-", "", "", "ending_before,expand,gender,is_owner,limit,q,starting_after", "json", "sdk:cartesia-python api.md get /voices"},
|
||||
{"voices_delete", "DELETE", "/voices/{id}", "api", "-", "", "", "", "none", "sdk:cartesia-python api.md delete /voices/{id}"},
|
||||
{"voices_add_accents", "PATCH", "/voices/{id}/accents", "api", "json", "accents", "accents", "", "json", "sdk:cartesia-python api.md patch /voices/{id}/accents"},
|
||||
{"voices_clone", "POST", "/voices/clone", "api", "form", "accent,base_voice_id,clip,description,language,name", "clip,language,name", "", "json", "sdk:cartesia-python api.md post /voices/clone"},
|
||||
{"voices_delete_accent", "DELETE", "/voices/{id}/accents/{accent_id}", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md delete /voices/{id}/accents/{accent_id}"},
|
||||
{"voices_get", "GET", "/voices/{id}", "api", "-", "", "", "expand", "json", "sdk:cartesia-python api.md get /voices/{id}"},
|
||||
{"voices_list_accents", "GET", "/accents", "api", "-", "", "", "", "json", "sdk:cartesia-python api.md get /accents"},
|
||||
{"voices_localize", "POST", "/voices/localize", "api", "json", "accent,description,dialect,language,name,original_speaker_gender,tagline,voice_id", "accent,name,voice_id", "", "json", "sdk:cartesia-python api.md post /voices/localize"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
static_assert(sizeof(kRows) / sizeof(kRows[0]) == 53, "cartesia call table size");
|
||||
|
||||
const CallTable& calls() {
|
||||
static const CallTable table("cartesia", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
Vendored
+39
@@ -0,0 +1,39 @@
|
||||
#pragma once
|
||||
|
||||
// The Cartesia call table. Cartesia's docs host answers machine fetches with a
|
||||
// bot-check page, so the rows come from the official Python SDK (generated
|
||||
// from Cartesia's OpenAPI document): api.md for method and path, the
|
||||
// *_params.py files for fields. Version 2026-08-14.
|
||||
|
||||
#include "vendors/audio/call_spec.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
const CallTable& calls();
|
||||
|
||||
struct CartesiaCalls {
|
||||
static constexpr const char* kStatus = "get_status";
|
||||
static constexpr const char* kTts = "tts_generate";
|
||||
static constexpr const char* kTtsSse = "tts_generate_sse";
|
||||
static constexpr const char* kInfill = "tts_infill";
|
||||
static constexpr const char* kStt = "stt_transcribe";
|
||||
static constexpr const char* kVoices = "voices_list";
|
||||
static constexpr const char* kVoice = "voices_get";
|
||||
static constexpr const char* kVoiceUpdate = "voices_update";
|
||||
static constexpr const char* kVoiceDelete = "voices_delete";
|
||||
static constexpr const char* kVoiceClone = "voices_clone";
|
||||
static constexpr const char* kVoiceLocalize = "voices_localize";
|
||||
static constexpr const char* kAccents = "voices_list_accents";
|
||||
static constexpr const char* kVoiceChanger = "voice_changer_generate";
|
||||
static constexpr const char* kAccessToken = "access_token_create";
|
||||
static constexpr const char* kAgents = "agents_list";
|
||||
static constexpr const char* kAgent = "agents_retrieve";
|
||||
static constexpr const char* kCalls = "calls_list";
|
||||
static constexpr const char* kCall = "calls_retrieve";
|
||||
static constexpr const char* kCallAudio = "calls_download_audio";
|
||||
static constexpr const char* kPronunciationList = "pronunciation_dicts_list";
|
||||
static constexpr const char* kFineTunes = "fine_tunes_list";
|
||||
static constexpr const char* kDatasets = "datasets_list";
|
||||
};
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
#include "vendors/audio/cartesia/client.h"
|
||||
|
||||
#include "llm/config.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
llm::ApiClientConfig CartesiaEndpoints::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = kVendor;
|
||||
c.base_url = kBase;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kBearer;
|
||||
c.default_headers["Cartesia-Version"] = kVersion;
|
||||
return c;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> CartesiaEndpoints::from_env() {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", HostFactory::from_env(config()));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> CartesiaEndpoints::with(AudioHost host) {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", std::move(host));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::string CartesiaEndpoints::key() { return llm::Config::get(kKeyEnv); }
|
||||
|
||||
AudioSocket::Headers CartesiaEndpoints::ws_headers(const std::string& key) {
|
||||
return {{"Authorization", "Bearer " + key}, {"Cartesia-Version", kVersion}};
|
||||
}
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
Vendored
+31
@@ -0,0 +1,31 @@
|
||||
#pragma once
|
||||
|
||||
// Cartesia endpoints and credentials.
|
||||
// REST https://api.cartesia.ai Authorization: Bearer $CARTESIA_API_KEY, Cartesia-Version: 2026-08-14
|
||||
// WebSocket wss://api.cartesia.ai same headers, plus ?cartesia_version=2026-08-14
|
||||
// The API version is a required header (an unversioned POST answers 400,
|
||||
// observed 2026-10-25) and is pinned here, not taken from configuration.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/hosts.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
struct CartesiaEndpoints {
|
||||
static constexpr const char* kVendor = "cartesia";
|
||||
static constexpr const char* kKeyEnv = "CARTESIA_API_KEY";
|
||||
static constexpr const char* kBase = "https://api.cartesia.ai";
|
||||
static constexpr const char* kWsBase = "wss://api.cartesia.ai";
|
||||
static constexpr const char* kVersion = "2026-08-14";
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
static std::shared_ptr<const Hosts> from_env();
|
||||
static std::shared_ptr<const Hosts> with(AudioHost host); // one host called "api"
|
||||
static std::string key();
|
||||
static AudioSocket::Headers ws_headers(const std::string& key);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
#include "vendors/audio/cartesia/speech.h"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("cartesia", why); }
|
||||
} // namespace
|
||||
|
||||
CartesiaSynthesizer::CartesiaSynthesizer(std::shared_ptr<const Hosts> hosts) : tts_(hosts), voices_(hosts) {}
|
||||
|
||||
CartesiaTtsRequest CartesiaSynthesizer::map(const SpeechRequest& r) const {
|
||||
validate(r);
|
||||
if (r.voice.empty()) invalid("a voice id is required (list voices first)");
|
||||
CartesiaTtsRequest t;
|
||||
if (!r.model.empty()) t.model_id = r.model;
|
||||
t.transcript = r.text;
|
||||
t.voice_id = r.voice;
|
||||
t.language = r.language.substr(0, 2);
|
||||
t.output_format.container = r.format == "pcm" ? "raw" : r.format;
|
||||
t.output_format.encoding = "pcm_s16le";
|
||||
if (r.sample_rate != 0) t.output_format.sample_rate = r.sample_rate;
|
||||
if (r.format == "mp3" && r.sample_rate == 0) t.output_format.sample_rate = 44100;
|
||||
if (r.speed > 0) t.gen_speed = std::clamp(r.speed, CartesiaCaps::kMinGenSpeed, CartesiaCaps::kMaxGenSpeed);
|
||||
return t;
|
||||
}
|
||||
|
||||
SpeechResult CartesiaSynthesizer::synthesize(const SpeechRequest& r) {
|
||||
CartesiaAudio a = tts_.synthesize(map(r));
|
||||
SpeechResult out;
|
||||
out.audio = std::move(a.bytes);
|
||||
out.content_type = a.content_type.empty() ? content_type_for(r.format) : a.content_type;
|
||||
out.format = r.format;
|
||||
out.provider = name();
|
||||
return out;
|
||||
}
|
||||
|
||||
Json CartesiaSynthesizer::voices() { return voices_.list(); }
|
||||
|
||||
CartesiaRecognizer::CartesiaRecognizer(std::shared_ptr<const Hosts> hosts) : stt_(std::move(hosts)) {}
|
||||
|
||||
CartesiaSttRequest CartesiaRecognizer::map(const TranscribeRequest& r) const {
|
||||
validate(r);
|
||||
if (r.audio.empty()) invalid("Cartesia batch transcription takes audio bytes, not a URL");
|
||||
CartesiaSttRequest s;
|
||||
s.audio = r.audio;
|
||||
s.filename = r.filename;
|
||||
s.content_type = r.content_type;
|
||||
if (!r.model.empty()) s.model = r.model;
|
||||
s.language = r.language.substr(0, 2);
|
||||
s.word_timestamps = true;
|
||||
return s;
|
||||
}
|
||||
|
||||
Transcript CartesiaRecognizer::fold(const Json& doc) {
|
||||
Transcript t;
|
||||
t.provider = "cartesia";
|
||||
t.raw = doc;
|
||||
if (!doc.is_object()) return t;
|
||||
t.text = llm::Service::string_at(doc, "text");
|
||||
t.language = llm::Service::string_at(doc, "language");
|
||||
if (const Json* d = doc.find("duration"); d && d->is_number()) t.duration = d->as_number();
|
||||
if (const Json* words = doc.find("words"); words && words->is_array()) {
|
||||
for (const Json& w : words->as_array()) {
|
||||
if (!w.is_object()) continue;
|
||||
TranscriptWord word;
|
||||
word.text = llm::Service::string_at(w, "word");
|
||||
if (const Json* x = w.find("start"); x && x->is_number()) word.start = x->as_number();
|
||||
if (const Json* x = w.find("end"); x && x->is_number()) word.end = x->as_number();
|
||||
t.words.push_back(word);
|
||||
}
|
||||
}
|
||||
return t;
|
||||
}
|
||||
|
||||
Transcript CartesiaRecognizer::transcribe(const TranscribeRequest& r) { return fold(stt_.transcribe(map(r))); }
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
Vendored
+43
@@ -0,0 +1,43 @@
|
||||
#pragma once
|
||||
|
||||
// Cartesia behind the neutral speech seam.
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "vendors/audio/cartesia/stt.h"
|
||||
#include "vendors/audio/cartesia/tts.h"
|
||||
#include "vendors/audio/cartesia/voices.h"
|
||||
#include "vendors/audio/speech.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
class CartesiaSynthesizer : public SpeechSynthesizer {
|
||||
public:
|
||||
explicit CartesiaSynthesizer(std::shared_ptr<const Hosts> hosts = CartesiaEndpoints::from_env());
|
||||
std::string name() const override { return "cartesia"; }
|
||||
bool configured() const override { return tts_.configured(); }
|
||||
std::size_t max_chars() const override { return CartesiaCaps::kMaxTranscriptChars; }
|
||||
std::vector<std::string> formats() const override { return {"mp3", "wav", "pcm"}; }
|
||||
SpeechResult synthesize(const SpeechRequest& request) override;
|
||||
Json voices() override;
|
||||
CartesiaTtsRequest map(const SpeechRequest& request) const;
|
||||
|
||||
private:
|
||||
CartesiaTts tts_;
|
||||
CartesiaVoices voices_;
|
||||
};
|
||||
|
||||
class CartesiaRecognizer : public SpeechRecognizer {
|
||||
public:
|
||||
explicit CartesiaRecognizer(std::shared_ptr<const Hosts> hosts = CartesiaEndpoints::from_env());
|
||||
std::string name() const override { return "cartesia"; }
|
||||
bool configured() const override { return stt_.configured(); }
|
||||
Transcript transcribe(const TranscribeRequest& request) override; // audio bytes only (no URL input)
|
||||
CartesiaSttRequest map(const TranscribeRequest& request) const;
|
||||
static Transcript fold(const Json& doc);
|
||||
|
||||
private:
|
||||
CartesiaStt stt_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
Vendored
+177
@@ -0,0 +1,177 @@
|
||||
#include "vendors/audio/cartesia/stt.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
using C = CartesiaCalls;
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("cartesia", why); }
|
||||
|
||||
bool valid_encoding(const std::string& e) {
|
||||
for (const char* ok : {"pcm_s16le", "pcm_s32le", "pcm_f16le", "pcm_f32le", "pcm_mulaw", "pcm_alaw"})
|
||||
if (e == ok) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
std::string str_at(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
} // namespace
|
||||
|
||||
clients::body::MultipartBody CartesiaStt::build(const CartesiaSttRequest& r) const {
|
||||
if (r.audio.empty() || r.audio.size() > CartesiaCaps::kMaxFileBytes) bad("audio must be 1 byte..100 MiB");
|
||||
require_text("model", r.model, 128);
|
||||
require_text("language", r.language, 16, false);
|
||||
if (!r.encoding.empty() && !valid_encoding(r.encoding)) bad("unknown encoding");
|
||||
if (r.sample_rate < 0 || r.sample_rate > 96000) bad("sample_rate must be 0 or up to 96000");
|
||||
clients::body::MultipartBody form;
|
||||
form.add_file("file", r.filename, r.content_type, r.audio);
|
||||
form.add_text("model", r.model);
|
||||
if (!r.language.empty()) form.add_text("language", r.language);
|
||||
if (!r.encoding.empty()) form.add_text("encoding", r.encoding);
|
||||
if (r.sample_rate > 0) form.add_text("sample_rate", std::to_string(r.sample_rate));
|
||||
if (r.word_timestamps) form.add_text("timestamp_granularities[]", "word");
|
||||
return form;
|
||||
}
|
||||
|
||||
Json CartesiaStt::transcribe(const CartesiaSttRequest& r) { return call_form(C::kStt, {}, build(r)); }
|
||||
|
||||
Vocabulary CartesiaSttSocket::auto_vocabulary() {
|
||||
// stt_auto_finalize_websocket_request / _response in the Python SDK types.
|
||||
return Vocabulary("type", {"close", "config"},
|
||||
{"connected", "turn.start", "turn.update", "turn.eager_end", "turn.resume", "turn.end", "error"});
|
||||
}
|
||||
Vocabulary CartesiaSttSocket::manual_vocabulary() {
|
||||
return Vocabulary("type", {}, {"transcript", "flush_done", "done", "error"});
|
||||
}
|
||||
|
||||
CartesiaSttSocket::CartesiaSttSocket(std::string api_key, SttSocketParams p, bool auto_finalize, std::string url,
|
||||
SocketOptions socket)
|
||||
: auto_(auto_finalize) {
|
||||
if (!valid_encoding(p.encoding)) invalid("unknown encoding");
|
||||
if (p.sample_rate < 8000 || p.sample_rate > 96000) invalid("sample_rate must be 8000..96000");
|
||||
if (p.model.empty() || p.model.size() > 128) invalid("model is required");
|
||||
if (p.keyterms.size() > AudioCaps::kMaxKeywords) invalid("too many keyterms");
|
||||
Query q;
|
||||
q.add("encoding", p.encoding);
|
||||
q.add("model", p.model);
|
||||
q.add_int("sample_rate", p.sample_rate);
|
||||
for (const auto& k : p.keyterms) {
|
||||
if (k.empty() || k.size() > 100) invalid("keyterms are 1..100 characters");
|
||||
q.add("keyterm", k);
|
||||
}
|
||||
if (auto_finalize) {
|
||||
if (p.turn_eager_end_threshold >= 0) q.add("turn_eager_end_threshold", std::to_string(p.turn_eager_end_threshold));
|
||||
if (p.turn_end_threshold >= 0) q.add("turn_end_threshold", std::to_string(p.turn_end_threshold));
|
||||
if (p.turn_start_threshold >= 0) q.add("turn_start_threshold", std::to_string(p.turn_start_threshold));
|
||||
if (p.turn_end_timeout_ms >= 0) q.add("turn_end_timeout_ms", std::to_string(static_cast<long>(p.turn_end_timeout_ms)));
|
||||
} else {
|
||||
if (!p.language.empty()) q.add("language", p.language);
|
||||
if (p.max_silence_duration_secs >= 0) q.add("max_silence_duration_secs", std::to_string(p.max_silence_duration_secs));
|
||||
if (p.min_volume >= 0) q.add("min_volume", std::to_string(p.min_volume));
|
||||
}
|
||||
std::string base = url.empty() ? std::string(CartesiaEndpoints::kWsBase) + (auto_finalize ? "/stt/turns/websocket" : "/stt/websocket") : url;
|
||||
socket_ = std::make_unique<AudioSocket>("cartesia", base + q.str(), CartesiaEndpoints::ws_headers(api_key),
|
||||
auto_finalize ? auto_vocabulary() : manual_vocabulary(), std::move(socket));
|
||||
}
|
||||
|
||||
void CartesiaSttSocket::finalize() {
|
||||
if (auto_) invalid("finalize belongs to the manual-finalize socket");
|
||||
socket_->send_text("finalize");
|
||||
}
|
||||
|
||||
void CartesiaSttSocket::end() {
|
||||
if (auto_) {
|
||||
Json m = Json::object();
|
||||
m.set("type", Json("close"));
|
||||
socket_->send_json(m);
|
||||
} else {
|
||||
socket_->send_text("close");
|
||||
}
|
||||
}
|
||||
|
||||
void CartesiaSttSocket::configure_turn(double eager, double end_t, double start_t, double timeout_ms) {
|
||||
if (!auto_) invalid("turn configuration belongs to the auto-finalize socket");
|
||||
Json turn = Json::object();
|
||||
if (eager >= 0) {
|
||||
if (eager < 0.3 || eager > 0.6) invalid("eager_end_threshold must be 0.3..0.6");
|
||||
turn.set("eager_end_threshold", Json(eager));
|
||||
}
|
||||
if (end_t >= 0) {
|
||||
if (end_t < 0.05 || end_t > 0.5) invalid("end_threshold must be 0.05..0.5");
|
||||
turn.set("end_threshold", Json(end_t));
|
||||
}
|
||||
if (start_t >= 0) {
|
||||
if (start_t < 0.5 || start_t > 0.9) invalid("start_threshold must be 0.5..0.9");
|
||||
turn.set("start_threshold", Json(start_t));
|
||||
}
|
||||
if (timeout_ms >= 0) {
|
||||
if (timeout_ms < 640 || timeout_ms > 11200) invalid("end_timeout_ms must be 640..11200");
|
||||
turn.set("end_timeout_ms", Json(timeout_ms));
|
||||
}
|
||||
Json m = Json::object();
|
||||
m.set("type", Json("config"));
|
||||
m.set("turn", turn);
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
SttEvent CartesiaSttSocket::decode(const SocketEvent& ev) {
|
||||
SttEvent out;
|
||||
if (ev.kind != SocketEvent::Kind::kText || !ev.json_ok) return out;
|
||||
out.raw = ev.json;
|
||||
out.type = ev.type;
|
||||
out.request_id = str_at(ev.json, "request_id");
|
||||
if (ev.type.rfind("turn.", 0) == 0) {
|
||||
out.transcript = str_at(ev.json, "transcript");
|
||||
out.is_final = ev.type == "turn.end";
|
||||
} else if (ev.type == "transcript") {
|
||||
out.transcript = str_at(ev.json, "text");
|
||||
if (const Json* f = ev.json.find("is_final"); f && f->is_bool()) out.is_final = f->as_bool();
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
SttSocketResult CartesiaSttSocket::collect(int max_wait_ms) {
|
||||
SttSocketResult out;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto raw = socket_->next(max_wait_ms);
|
||||
if (!raw) return out;
|
||||
++out.events;
|
||||
if (raw->kind == SocketEvent::Kind::kClose) {
|
||||
out.closed = true;
|
||||
return out;
|
||||
}
|
||||
SttEvent ev = decode(*raw);
|
||||
if (!ev.request_id.empty()) out.request_id = ev.request_id;
|
||||
if (ev.type == "error") {
|
||||
out.error = str_at(ev.raw, "message");
|
||||
if (out.error.empty()) out.error = str_at(ev.raw, "title");
|
||||
if (out.error.empty()) out.error = "stream error";
|
||||
return out;
|
||||
}
|
||||
if (ev.type == "turn.end") {
|
||||
out.finals.push_back(ev.transcript);
|
||||
out.transcript += (out.transcript.empty() ? "" : " ") + ev.transcript;
|
||||
out.partial.clear();
|
||||
} else if (ev.type == "turn.update" || ev.type == "turn.eager_end") {
|
||||
out.partial = ev.transcript;
|
||||
} else if (ev.type == "transcript") {
|
||||
if (ev.is_final) {
|
||||
// Manual mode deltas concatenate exactly as received (the API says: do not trim or add whitespace).
|
||||
out.finals.push_back(ev.transcript);
|
||||
out.transcript += ev.transcript;
|
||||
} else {
|
||||
out.partial = ev.transcript;
|
||||
}
|
||||
} else if (ev.type == "done") {
|
||||
out.done = true;
|
||||
return out;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
Vendored
+93
@@ -0,0 +1,93 @@
|
||||
#pragma once
|
||||
|
||||
// Cartesia speech to text: batch POST /stt (multipart) and the two realtime
|
||||
// WebSockets: /stt/turns/websocket (auto finalize, turn events) and
|
||||
// /stt/websocket (manual finalize).
|
||||
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/cartesia/tts.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
struct CartesiaSttRequest {
|
||||
std::string audio; // required bytes
|
||||
std::string filename = "audio";
|
||||
std::string content_type = "application/octet-stream";
|
||||
std::string model = "ink-whisper"; // ink-whisper or a dated snapshot
|
||||
std::string language; // ISO 639-1 code, "" = detect
|
||||
std::string encoding; // for raw audio: pcm_s16le pcm_s32le pcm_f16le pcm_f32le pcm_mulaw pcm_alaw
|
||||
int sample_rate = 0;
|
||||
bool word_timestamps = false; // timestamp_granularities[]=word
|
||||
};
|
||||
|
||||
class CartesiaStt : public CartesiaService {
|
||||
public:
|
||||
using CartesiaService::CartesiaService;
|
||||
clients::body::MultipartBody build(const CartesiaSttRequest& request) const;
|
||||
Json transcribe(const CartesiaSttRequest& request);
|
||||
};
|
||||
|
||||
struct SttSocketParams {
|
||||
std::string encoding = "pcm_s16le"; // required by the SDK signature
|
||||
std::string model = "ink-whisper";
|
||||
int sample_rate = 16000;
|
||||
std::vector<std::string> keyterms;
|
||||
// auto-finalize only (/stt/turns/websocket)
|
||||
double turn_eager_end_threshold = -1, turn_end_threshold = -1, turn_start_threshold = -1;
|
||||
double turn_end_timeout_ms = -1;
|
||||
// manual-finalize only (/stt/websocket)
|
||||
std::string language;
|
||||
double max_silence_duration_secs = -1, min_volume = -1;
|
||||
};
|
||||
|
||||
struct SttEvent {
|
||||
std::string type; // connected turn.start turn.update turn.eager_end turn.resume turn.end transcript flush_done done error
|
||||
std::string transcript; // turn.* events: transcript; manual: text delta
|
||||
bool is_final = false;
|
||||
std::string request_id;
|
||||
Json raw;
|
||||
};
|
||||
|
||||
struct SttSocketResult {
|
||||
std::vector<std::string> finals; // turn.end transcripts (auto) or is_final text deltas (manual)
|
||||
std::string transcript; // auto: joined with a space; manual: concatenated exactly (no added whitespace)
|
||||
std::string partial;
|
||||
std::string request_id;
|
||||
std::string error;
|
||||
bool done = false;
|
||||
bool closed = false;
|
||||
std::size_t events = 0;
|
||||
};
|
||||
|
||||
class CartesiaSttSocket {
|
||||
public:
|
||||
// auto_finalize true = /stt/turns/websocket, false = /stt/websocket.
|
||||
CartesiaSttSocket(std::string api_key, SttSocketParams params, bool auto_finalize, std::string url = "",
|
||||
SocketOptions socket = {});
|
||||
std::string url() const { return socket_->url(); }
|
||||
void connect() { socket_->connect(); }
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
void send_audio(const std::string& audio) { socket_->send_binary(audio); }
|
||||
void finalize(); // manual: the text frame "finalize"
|
||||
void end(); // auto: {"type":"close"}; manual: the text frame "close"
|
||||
void configure_turn(double eager_end, double end_threshold, double start_threshold, double end_timeout_ms); // auto only
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
static SttEvent decode(const SocketEvent& ev);
|
||||
SttSocketResult collect(int max_wait_ms);
|
||||
|
||||
static Vocabulary auto_vocabulary();
|
||||
static Vocabulary manual_vocabulary();
|
||||
|
||||
private:
|
||||
bool auto_;
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
Vendored
+117
@@ -0,0 +1,117 @@
|
||||
#include "vendors/audio/cartesia/tts.h"
|
||||
|
||||
#include "llm/sse.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
using C = CartesiaCalls;
|
||||
|
||||
namespace {
|
||||
bool in(int v, std::initializer_list<int> ok) {
|
||||
for (int x : ok)
|
||||
if (x == v) return true;
|
||||
return false;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Json OutputFormat::to_json(const CartesiaService& svc) const {
|
||||
if (!in(sample_rate, {8000, 16000, 22050, 24000, 44100, 48000}))
|
||||
svc.bad("sample_rate must be 8000, 16000, 22050, 24000, 44100 or 48000");
|
||||
Json o = Json::object();
|
||||
o.set("container", Json(container));
|
||||
o.set("sample_rate", Json(sample_rate));
|
||||
if (container == "mp3") {
|
||||
if (!in(bit_rate, {32000, 64000, 96000, 128000, 192000}))
|
||||
svc.bad("mp3 bit_rate must be 32000, 64000, 96000, 128000 or 192000");
|
||||
o.set("bit_rate", Json(bit_rate));
|
||||
} else if (container == "raw" || container == "wav") {
|
||||
if (encoding != "pcm_f32le" && encoding != "pcm_s16le" && encoding != "pcm_mulaw" && encoding != "pcm_alaw")
|
||||
svc.bad("encoding must be pcm_f32le, pcm_s16le, pcm_mulaw or pcm_alaw");
|
||||
o.set("encoding", Json(encoding));
|
||||
} else {
|
||||
svc.bad("container must be raw, wav or mp3");
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
Json CartesiaTts::build(const CartesiaTtsRequest& r, bool sse) const {
|
||||
require_text("model_id", r.model_id, 128);
|
||||
require_text("transcript", r.transcript, CartesiaCaps::kMaxTranscriptChars);
|
||||
check_id("voice", r.voice_id);
|
||||
Json o = Json::object();
|
||||
o.set("model_id", Json(r.model_id));
|
||||
o.set("transcript", Json(r.transcript));
|
||||
Json voice = Json::object();
|
||||
voice.set("id", Json(r.voice_id));
|
||||
o.set("voice", voice);
|
||||
o.set("output_format", r.output_format.to_json(*this));
|
||||
if (!r.language.empty()) {
|
||||
require_text("language", r.language, 16);
|
||||
o.set("language", Json(r.language));
|
||||
}
|
||||
if (!r.speed.empty()) {
|
||||
if (r.speed != "slow" && r.speed != "normal" && r.speed != "fast") bad("speed must be slow, normal or fast");
|
||||
o.set("speed", Json(r.speed));
|
||||
}
|
||||
Json gen = Json::object();
|
||||
if (!r.emotion.empty()) {
|
||||
require_text("emotion", r.emotion, 64);
|
||||
gen.set("emotion", Json(r.emotion));
|
||||
}
|
||||
if (r.gen_speed >= 0) {
|
||||
require_range("generation speed", r.gen_speed, CartesiaCaps::kMinGenSpeed, CartesiaCaps::kMaxGenSpeed);
|
||||
gen.set("speed", Json(r.gen_speed));
|
||||
}
|
||||
if (r.volume >= 0) {
|
||||
require_range("volume", r.volume, CartesiaCaps::kMinVolume, CartesiaCaps::kMaxVolume);
|
||||
gen.set("volume", Json(r.volume));
|
||||
}
|
||||
if (!gen.as_object().empty()) o.set("generation_config", gen);
|
||||
if (!r.pronunciation_dict_id.empty()) {
|
||||
check_id("pronunciation dictionary", r.pronunciation_dict_id);
|
||||
o.set("pronunciation_dict_id", Json(r.pronunciation_dict_id));
|
||||
}
|
||||
if (!r.normalization.empty()) {
|
||||
require_text("normalization", r.normalization, 16);
|
||||
o.set("normalization", Json(r.normalization));
|
||||
}
|
||||
check_body(table_.get(sse ? C::kTtsSse : C::kTts), &o);
|
||||
return o;
|
||||
}
|
||||
|
||||
CartesiaAudio CartesiaTts::synthesize(const CartesiaTtsRequest& r) {
|
||||
Json body = build(r);
|
||||
llm::RawResponse raw = call_raw(C::kTts, {}, &body);
|
||||
return CartesiaAudio{raw.body, raw.content_type};
|
||||
}
|
||||
|
||||
ByteStreamer::Result CartesiaTts::stream(const CartesiaTtsRequest& r,
|
||||
const std::function<bool(const std::string&)>& sink) {
|
||||
Json body = build(r, true);
|
||||
llm::SseDecoder decoder("cartesia", [&](const llm::SseEvent& ev) -> bool {
|
||||
const Json& d = ev.data;
|
||||
if (!d.is_object()) return true;
|
||||
const Json* type = d.find("type");
|
||||
std::string t = type && type->is_string() ? type->as_string() : ev.event;
|
||||
if (t == "error") {
|
||||
const Json* m = d.find("message");
|
||||
throw llm::ApiError("cartesia", 502, "stream_error", m && m->is_string() ? m->as_string() : "stream error");
|
||||
}
|
||||
if (t == "done") return false;
|
||||
if (t != "chunk") return true;
|
||||
const Json* data = d.find("data");
|
||||
if (!data || !data->is_string()) return true;
|
||||
std::string raw;
|
||||
if (!security::crypto::Base64::decode(data->as_string(), raw))
|
||||
throw llm::ApiError("cartesia", 502, "BAD_RESPONSE", "audio chunk is not valid base64");
|
||||
return raw.empty() ? true : sink(raw);
|
||||
});
|
||||
ByteStreamer::Result res = call_stream(C::kTtsSse, {}, &body, Query(), [&](const std::string& chunk) {
|
||||
return decoder.feed(chunk);
|
||||
});
|
||||
decoder.finish();
|
||||
return res;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
Vendored
+68
@@ -0,0 +1,68 @@
|
||||
#pragma once
|
||||
|
||||
// Cartesia text to speech over HTTP: /tts/bytes (buffered audio) and
|
||||
// /tts/sse (base64 chunks in server-sent events). WebSocket contexts are in
|
||||
// TtsSocket.
|
||||
|
||||
#include <functional>
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/audio_service.h"
|
||||
#include "vendors/audio/cartesia/calls.h"
|
||||
#include "vendors/audio/cartesia/client.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
struct CartesiaCaps {
|
||||
static constexpr std::size_t kMaxTranscriptChars = 20000; // our ceiling for one request
|
||||
static constexpr double kMinGenSpeed = 0.6, kMaxGenSpeed = 1.5; // generation_config.speed (SDK docstring)
|
||||
static constexpr double kMinVolume = 0.5, kMaxVolume = 2.0; // generation_config.volume
|
||||
static constexpr std::size_t kMaxFileBytes = 100u * 1024u * 1024u;
|
||||
static constexpr int kMaxListLimit = 100;
|
||||
};
|
||||
|
||||
class CartesiaService : public AudioService {
|
||||
public:
|
||||
explicit CartesiaService(std::shared_ptr<const Hosts> hosts) : AudioService(calls(), std::move(hosts)) {}
|
||||
CartesiaService() : CartesiaService(CartesiaEndpoints::from_env()) {}
|
||||
};
|
||||
|
||||
// Audio container and encoding of a synthesis result.
|
||||
struct OutputFormat {
|
||||
std::string container = "wav"; // raw | wav | mp3
|
||||
std::string encoding = "pcm_s16le"; // raw/wav: pcm_f32le pcm_s16le pcm_mulaw pcm_alaw
|
||||
int sample_rate = 44100; // 8000 16000 22050 24000 44100 48000
|
||||
int bit_rate = 128000; // mp3: 32000 64000 96000 128000 192000
|
||||
Json to_json(const CartesiaService& svc) const;
|
||||
};
|
||||
|
||||
struct CartesiaTtsRequest {
|
||||
std::string model_id = "sonic-3"; // sonic-3.5 sonic-3 sonic-latest or a dated snapshot
|
||||
std::string transcript; // required
|
||||
std::string voice_id; // required
|
||||
std::string language; // "" = model default
|
||||
OutputFormat output_format;
|
||||
std::string speed; // "", slow, normal, fast (model speed)
|
||||
// generation_config (sonic-3 only); negative = unset
|
||||
std::string emotion;
|
||||
double gen_speed = -1; // 0.6..1.5
|
||||
double volume = -1; // 0.5..2.0
|
||||
std::string pronunciation_dict_id;
|
||||
std::string normalization; // "", auto, off, or a locale code
|
||||
};
|
||||
|
||||
struct CartesiaAudio {
|
||||
std::string bytes;
|
||||
std::string content_type;
|
||||
};
|
||||
|
||||
class CartesiaTts : public CartesiaService {
|
||||
public:
|
||||
using CartesiaService::CartesiaService;
|
||||
Json build(const CartesiaTtsRequest& request, bool sse = false) const;
|
||||
CartesiaAudio synthesize(const CartesiaTtsRequest& request);
|
||||
// Streams decoded audio chunks from /tts/sse; `sink` returns false to stop.
|
||||
ByteStreamer::Result stream(const CartesiaTtsRequest& request, const std::function<bool(const std::string&)>& sink);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
#include "vendors/audio/cartesia/tts_socket.h"
|
||||
|
||||
#include "security/crypto/Base64.h"
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("cartesia", why); }
|
||||
|
||||
std::string str_at(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Vocabulary TtsSocket::vocabulary() {
|
||||
// Client messages (generation request, cancel) have no `type` field; server types from websocket_response.py.
|
||||
return Vocabulary("type", {}, {"chunk", "flush_done", "done", "timestamps", "phoneme_timestamps", "error"});
|
||||
}
|
||||
|
||||
TtsSocket::TtsSocket(std::string api_key, std::string url, SocketOptions socket) {
|
||||
std::string base = url.empty() ? std::string(CartesiaEndpoints::kWsBase) + "/tts/websocket" : url;
|
||||
base = AudioSocket::with_query(base, std::string("cartesia_version=") + CartesiaEndpoints::kVersion);
|
||||
socket_ = std::make_unique<AudioSocket>("cartesia", base, CartesiaEndpoints::ws_headers(api_key), vocabulary(),
|
||||
std::move(socket));
|
||||
}
|
||||
|
||||
Json TtsSocket::build(const TtsSocketRequest& r) {
|
||||
if (!llm::Service::valid_id(r.context_id)) invalid("bad context id");
|
||||
if (r.model_id.empty() || r.model_id.size() > 128) invalid("model_id is required");
|
||||
if (r.transcript.empty() || r.transcript.size() > CartesiaCaps::kMaxTranscriptChars)
|
||||
invalid("transcript must be 1.." + std::to_string(CartesiaCaps::kMaxTranscriptChars) + " characters");
|
||||
if (r.transcript.find('\0') != std::string::npos) invalid("transcript has a NUL byte");
|
||||
if (!llm::Service::valid_id(r.voice_id)) invalid("bad voice id");
|
||||
bool rate_ok = false;
|
||||
for (int v : {8000, 16000, 22050, 24000, 44100, 48000}) rate_ok = rate_ok || v == r.sample_rate;
|
||||
if (!rate_ok) invalid("sample_rate must be 8000, 16000, 22050, 24000, 44100 or 48000");
|
||||
if (r.encoding != "pcm_f32le" && r.encoding != "pcm_s16le" && r.encoding != "pcm_mulaw" && r.encoding != "pcm_alaw")
|
||||
invalid("encoding must be pcm_f32le, pcm_s16le, pcm_mulaw or pcm_alaw");
|
||||
Json o = Json::object();
|
||||
o.set("context_id", Json(r.context_id));
|
||||
o.set("model_id", Json(r.model_id));
|
||||
o.set("transcript", Json(r.transcript));
|
||||
Json voice = Json::object();
|
||||
voice.set("id", Json(r.voice_id));
|
||||
o.set("voice", voice);
|
||||
Json fmt = Json::object();
|
||||
fmt.set("container", Json("raw"));
|
||||
fmt.set("encoding", Json(r.encoding));
|
||||
fmt.set("sample_rate", Json(r.sample_rate));
|
||||
o.set("output_format", fmt);
|
||||
if (!r.language.empty()) {
|
||||
if (r.language.size() > 16) invalid("language is too long");
|
||||
o.set("language", Json(r.language));
|
||||
}
|
||||
if (r.continue_ >= 0) o.set("continue", Json(r.continue_ == 1));
|
||||
if (r.flush >= 0) o.set("flush", Json(r.flush == 1));
|
||||
if (r.add_timestamps >= 0) o.set("add_timestamps", Json(r.add_timestamps == 1));
|
||||
if (r.add_phoneme_timestamps >= 0) o.set("add_phoneme_timestamps", Json(r.add_phoneme_timestamps == 1));
|
||||
if (r.max_buffer_delay_ms >= 0) {
|
||||
if (r.max_buffer_delay_ms > 5000) invalid("max_buffer_delay_ms must be 0..5000");
|
||||
o.set("max_buffer_delay_ms", Json(r.max_buffer_delay_ms));
|
||||
}
|
||||
if (!r.speed.empty()) {
|
||||
if (r.speed != "slow" && r.speed != "normal" && r.speed != "fast") invalid("speed must be slow, normal or fast");
|
||||
o.set("speed", Json(r.speed));
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
void TtsSocket::generate(const TtsSocketRequest& r) { socket_->send_json(build(r), true); }
|
||||
|
||||
void TtsSocket::cancel(const std::string& context_id) {
|
||||
if (!llm::Service::valid_id(context_id)) invalid("bad context id");
|
||||
Json m = Json::object();
|
||||
m.set("context_id", Json(context_id));
|
||||
m.set("cancel", Json(true));
|
||||
socket_->send_json(m, true);
|
||||
}
|
||||
|
||||
TtsSocketResult TtsSocket::collect(const std::string& context_id, int max_wait_ms, bool stop_at_flush) {
|
||||
TtsSocketResult out;
|
||||
out.context_id = context_id;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto ev = socket_->next(max_wait_ms);
|
||||
if (!ev) return out;
|
||||
if (ev->kind == SocketEvent::Kind::kClose) {
|
||||
out.closed = true;
|
||||
return out;
|
||||
}
|
||||
if (!ev->json_ok || !ev->json.is_object()) continue;
|
||||
std::string ctx = str_at(ev->json, "context_id");
|
||||
if (!ctx.empty() && ctx != context_id) continue; // another context on this connection
|
||||
if (ev->type == "chunk") {
|
||||
std::string raw;
|
||||
if (!security::crypto::Base64::decode(str_at(ev->json, "data"), raw))
|
||||
throw llm::ApiError("cartesia", 502, "BAD_RESPONSE", "audio chunk is not valid base64");
|
||||
if (out.audio.size() + raw.size() > AudioCaps::kMaxStreamAudioBytes) {
|
||||
out.truncated = true;
|
||||
return out;
|
||||
}
|
||||
out.audio += raw;
|
||||
++out.chunks;
|
||||
if (const Json* f = ev->json.find("flush_id"); f && f->is_number()) out.flush_id = f->as_int();
|
||||
} else if (ev->type == "timestamps") {
|
||||
if (const Json* w = ev->json.find("word_timestamps"); w) out.timestamps.push_back(*w);
|
||||
} else if (ev->type == "phoneme_timestamps") {
|
||||
if (const Json* w = ev->json.find("phoneme_timestamps"); w) out.phoneme_timestamps.push_back(*w);
|
||||
} else if (ev->type == "flush_done") {
|
||||
out.flush_done = true;
|
||||
if (const Json* f = ev->json.find("flush_id"); f && f->is_number()) out.flush_id = f->as_int();
|
||||
if (stop_at_flush) return out;
|
||||
} else if (ev->type == "done") {
|
||||
out.done = true;
|
||||
return out;
|
||||
} else if (ev->type == "error") {
|
||||
out.error = str_at(ev->json, "message");
|
||||
if (out.error.empty()) out.error = str_at(ev->json, "title");
|
||||
if (out.error.empty()) out.error = "stream error";
|
||||
return out;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
#pragma once
|
||||
|
||||
// Cartesia text to speech over WebSocket (wss://api.cartesia.ai/tts/websocket):
|
||||
// several contexts on one connection, continuations (`continue`), flushes,
|
||||
// cancel, word and phoneme timestamps. Message shapes: Python SDK
|
||||
// types/generation_request.py, websocket_client_event.py, websocket_response.py.
|
||||
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/cartesia/tts.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
struct TtsSocketRequest {
|
||||
std::string context_id; // required, you choose it: [A-Za-z0-9._:-]{1,256}
|
||||
std::string model_id = "sonic-3";
|
||||
std::string transcript; // required, may be a partial transcript when continue_ is true
|
||||
std::string voice_id; // required, same for every request of one context
|
||||
std::string language;
|
||||
// The websocket output is raw audio only.
|
||||
std::string encoding = "pcm_s16le";
|
||||
int sample_rate = 44100;
|
||||
int continue_ = -1; // -1 unset: whether more input follows on this context
|
||||
int flush = -1;
|
||||
int add_timestamps = -1;
|
||||
int add_phoneme_timestamps = -1;
|
||||
int max_buffer_delay_ms = -1; // 0..5000
|
||||
std::string speed; // "", slow, normal, fast
|
||||
};
|
||||
|
||||
struct TtsSocketResult {
|
||||
std::string context_id;
|
||||
std::string audio; // decoded chunks of that context
|
||||
std::size_t chunks = 0;
|
||||
std::vector<Json> timestamps; // word_timestamps objects
|
||||
std::vector<Json> phoneme_timestamps;
|
||||
long flush_id = 0; // last flush_id seen
|
||||
bool flush_done = false;
|
||||
bool done = false;
|
||||
bool truncated = false;
|
||||
std::string error;
|
||||
bool closed = false;
|
||||
};
|
||||
|
||||
class TtsSocket {
|
||||
public:
|
||||
TtsSocket(std::string api_key, std::string url = "", SocketOptions socket = {});
|
||||
std::string url() const { return socket_->url(); }
|
||||
void connect() { socket_->connect(); }
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
static Json build(const TtsSocketRequest& request); // validates
|
||||
void generate(const TtsSocketRequest& request);
|
||||
void cancel(const std::string& context_id);
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
// Reads events of `context_id` until done / flush_done / error / close or silence.
|
||||
TtsSocketResult collect(const std::string& context_id, int max_wait_ms, bool stop_at_flush = true);
|
||||
|
||||
static Vocabulary vocabulary();
|
||||
|
||||
private:
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
#include "vendors/audio/cartesia/voices.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
using C = CartesiaCalls;
|
||||
|
||||
Json CartesiaVoices::list(const Json& q) {
|
||||
if (q.is_object()) {
|
||||
const Json* l = q.find("limit");
|
||||
if (l && l->is_number() && (l->as_int() < 1 || l->as_int() > CartesiaCaps::kMaxListLimit))
|
||||
bad("limit must be 1..100");
|
||||
}
|
||||
return call(C::kVoices, {}, nullptr, query_from(C::kVoices, q));
|
||||
}
|
||||
|
||||
VoiceListing CartesiaVoices::list_all(std::size_t max_pages, int limit, const std::string& q) {
|
||||
if (max_pages == 0 || max_pages > AudioCaps::kMaxPages) bad("max_pages must be 1.." + std::to_string(AudioCaps::kMaxPages));
|
||||
VoiceListing out;
|
||||
Json query = Json::object();
|
||||
query.set("limit", Json(limit));
|
||||
if (!q.empty()) query.set("q", Json(q));
|
||||
for (std::size_t page = 0; page < max_pages; ++page) {
|
||||
Json doc = list(query);
|
||||
++out.pages;
|
||||
const Json* data = doc.find("data");
|
||||
std::string last_id;
|
||||
if (data && data->is_array()) {
|
||||
for (const Json& v : data->as_array()) {
|
||||
if (out.voices.as_array().size() >= AudioCaps::kMaxListItems) {
|
||||
out.truncated = true;
|
||||
return out;
|
||||
}
|
||||
out.voices.push(v);
|
||||
last_id = string_at(v, "id");
|
||||
}
|
||||
}
|
||||
const Json* more = doc.find("has_more");
|
||||
if (!more || !more->is_bool() || !more->as_bool() || last_id.empty()) return out;
|
||||
query.set("starting_after", Json(last_id));
|
||||
}
|
||||
out.truncated = true;
|
||||
return out;
|
||||
}
|
||||
|
||||
Json CartesiaVoices::get(const std::string& id, bool expand) {
|
||||
check_id("voice", id);
|
||||
Query q;
|
||||
if (expand) q.add("expand", "preview_file_url");
|
||||
return call(C::kVoice, {id}, nullptr, q);
|
||||
}
|
||||
|
||||
Json CartesiaVoices::accents() { return call(C::kAccents); }
|
||||
|
||||
Json CartesiaVoices::clone(const std::string& clip, const std::string& filename, const std::string& content_type,
|
||||
const std::string& name, const std::string& language, const std::string& description,
|
||||
bool confirm) {
|
||||
if (clip.empty() || clip.size() > CartesiaCaps::kMaxFileBytes) bad("clip must be 1 byte..100 MiB");
|
||||
require_text("name", name, 200);
|
||||
require_text("language", language, 16);
|
||||
require_text("description", description, 2000, false);
|
||||
require_confirm("cloning a voice", confirm);
|
||||
clients::body::MultipartBody form;
|
||||
form.add_file("clip", filename.empty() ? "clip" : filename, content_type, clip);
|
||||
form.add_text("name", name);
|
||||
form.add_text("language", language);
|
||||
if (!description.empty()) form.add_text("description", description);
|
||||
return call_form(C::kVoiceClone, {}, form);
|
||||
}
|
||||
|
||||
Json CartesiaVoices::localize(const Json& body, bool confirm) {
|
||||
check_body(table_.get(C::kVoiceLocalize), &body);
|
||||
require_confirm("localizing a voice", confirm);
|
||||
return call(C::kVoiceLocalize, {}, &body);
|
||||
}
|
||||
|
||||
Json CartesiaVoices::update(const std::string& id, const Json& body, bool confirm) {
|
||||
check_id("voice", id);
|
||||
check_body(table_.get(C::kVoiceUpdate), &body);
|
||||
require_confirm("updating a voice", confirm);
|
||||
return call(C::kVoiceUpdate, {id}, &body);
|
||||
}
|
||||
|
||||
Json CartesiaVoices::remove(const std::string& id, bool confirm) {
|
||||
check_id("voice", id);
|
||||
return call_destructive(C::kVoiceDelete, {id}, nullptr, Query(), confirm);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
Vendored
+35
@@ -0,0 +1,35 @@
|
||||
#pragma once
|
||||
|
||||
// Voices: list, get, accents (read side) and clone, localize, update, delete
|
||||
// (writes: confirm required).
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/cartesia/tts.h"
|
||||
|
||||
namespace molo::audio::cartesia {
|
||||
|
||||
struct VoiceListing {
|
||||
Json voices = Json::array();
|
||||
bool truncated = false; // the page or item cap stopped the walk while more existed
|
||||
std::size_t pages = 0;
|
||||
};
|
||||
|
||||
class CartesiaVoices : public CartesiaService {
|
||||
public:
|
||||
using CartesiaService::CartesiaService;
|
||||
Json list(const Json& query_object = Json());
|
||||
// Walks starting_after cursors (id of the last voice) up to max_pages.
|
||||
VoiceListing list_all(std::size_t max_pages, int limit = 100, const std::string& q = "");
|
||||
Json get(const std::string& voice_id, bool expand_preview = false);
|
||||
Json accents();
|
||||
// Cloning needs a clip (bytes), name and language; costs credits.
|
||||
Json clone(const std::string& clip, const std::string& filename, const std::string& content_type,
|
||||
const std::string& name, const std::string& language, const std::string& description,
|
||||
bool confirm);
|
||||
Json localize(const Json& body, bool confirm);
|
||||
Json update(const std::string& voice_id, const Json& body, bool confirm);
|
||||
Json remove(const std::string& voice_id, bool confirm);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::cartesia
|
||||
Vendored
+70
@@ -0,0 +1,70 @@
|
||||
#include "vendors/audio/deepgram/calls.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
// One row per REST call. tools/check_spec.py parses the block between the
|
||||
// markers: {id, method, path, host, body, fields, required, query, response, source}.
|
||||
// CALLS-BEGIN
|
||||
static const CallSpec kRows[] = {
|
||||
{"tokens_grant", "POST", "/v1/auth/grant", "api", "json", "ttl_seconds", "", "", "json", "openapi:tokens_grant"},
|
||||
{"media_transcribe", "POST", "/v1/listen", "api", "json", "url", "url", "callback,callback_method,extra,sentiment,summarize,tag,topics,custom_topic,custom_topic_mode,intents,custom_intent,custom_intent_mode,detect_entities,detect_language,diarize,diarize_model,dictation,encoding,filler_words,keyterm,keywords,language,language_hint,measurements,model,multichannel,numerals,paragraphs,profanity_filter,punctuate,redact,replace,search,smart_format,utterances,utt_split,version,mip_opt_out", "json", "openapi:media_transcribe"},
|
||||
{"models_list_3", "GET", "/v1/models", "api", "-", "", "", "include_outdated", "json", "openapi:models_list_3"},
|
||||
{"models_get_2", "GET", "/v1/models/{model_id}", "api", "-", "", "", "", "json", "openapi:models_get_2"},
|
||||
{"projects_list", "GET", "/v1/projects", "api", "-", "", "", "", "json", "openapi:projects_list"},
|
||||
{"projects_get", "GET", "/v1/projects/{project_id}", "api", "-", "", "", "limit,page", "json", "openapi:projects_get"},
|
||||
{"projects_update", "PATCH", "/v1/projects/{project_id}", "api", "json", "name", "", "", "json", "openapi:projects_update"},
|
||||
{"projects_delete", "DELETE", "/v1/projects/{project_id}", "api", "-", "", "", "", "json", "openapi:projects_delete"},
|
||||
{"variables_list", "GET", "/v1/projects/{project_id}/agent-variables", "api", "-", "", "", "", "json", "openapi:variables_list"},
|
||||
{"variables_create", "POST", "/v1/projects/{project_id}/agent-variables", "api", "json", "key,value,api_version", "key,value", "", "json", "openapi:variables_create"},
|
||||
{"variables_get", "GET", "/v1/projects/{project_id}/agent-variables/{variable_id}", "api", "-", "", "", "", "json", "openapi:variables_get"},
|
||||
{"variables_update", "PATCH", "/v1/projects/{project_id}/agent-variables/{variable_id}", "api", "json", "value", "value", "", "json", "openapi:variables_update"},
|
||||
{"variables_delete", "DELETE", "/v1/projects/{project_id}/agent-variables/{variable_id}", "api", "-", "", "", "", "json", "openapi:variables_delete"},
|
||||
{"configurations_list", "GET", "/v1/projects/{project_id}/agents", "api", "-", "", "", "", "json", "openapi:configurations_list"},
|
||||
{"configurations_create", "POST", "/v1/projects/{project_id}/agents", "api", "json", "config,metadata,api_version", "config", "", "json", "openapi:configurations_create"},
|
||||
{"configurations_get", "GET", "/v1/projects/{project_id}/agents/{agent_id}", "api", "-", "", "", "", "json", "openapi:configurations_get"},
|
||||
{"configurations_update", "PUT", "/v1/projects/{project_id}/agents/{agent_id}", "api", "json", "metadata", "metadata", "", "json", "openapi:configurations_update"},
|
||||
{"configurations_delete", "DELETE", "/v1/projects/{project_id}/agents/{agent_id}", "api", "-", "", "", "", "json", "openapi:configurations_delete"},
|
||||
{"balances_list", "GET", "/v1/projects/{project_id}/balances", "api", "-", "", "", "", "json", "openapi:balances_list"},
|
||||
{"balances_get", "GET", "/v1/projects/{project_id}/balances/{balance_id}", "api", "-", "", "", "", "json", "openapi:balances_get"},
|
||||
{"breakdown_list", "GET", "/v1/projects/{project_id}/billing/breakdown", "api", "-", "", "", "start,end,accessor,deployment,tag,line_item,grouping", "json", "openapi:breakdown_list"},
|
||||
{"fields_list_2", "GET", "/v1/projects/{project_id}/billing/fields", "api", "-", "", "", "start,end", "json", "openapi:fields_list_2"},
|
||||
{"invites_list", "GET", "/v1/projects/{project_id}/invites", "api", "-", "", "", "", "json", "openapi:invites_list"},
|
||||
{"invites_create", "POST", "/v1/projects/{project_id}/invites", "api", "json", "email,scope", "email,scope", "", "json", "openapi:invites_create"},
|
||||
{"invites_delete", "DELETE", "/v1/projects/{project_id}/invites/{email}", "api", "-", "", "", "", "json", "openapi:invites_delete"},
|
||||
{"keys_list", "GET", "/v1/projects/{project_id}/keys", "api", "-", "", "", "status", "json", "openapi:keys_list"},
|
||||
{"keys_create", "POST", "/v1/projects/{project_id}/keys", "api", "json", "*", "", "", "json", "openapi:keys_create"},
|
||||
{"keys_get", "GET", "/v1/projects/{project_id}/keys/{key_id}", "api", "-", "", "", "", "json", "openapi:keys_get"},
|
||||
{"keys_delete", "DELETE", "/v1/projects/{project_id}/keys/{key_id}", "api", "-", "", "", "", "json", "openapi:keys_delete"},
|
||||
{"projects_leave", "DELETE", "/v1/projects/{project_id}/leave", "api", "-", "", "", "", "json", "openapi:projects_leave"},
|
||||
{"members_list", "GET", "/v1/projects/{project_id}/members", "api", "-", "", "", "", "json", "openapi:members_list"},
|
||||
{"members_delete", "DELETE", "/v1/projects/{project_id}/members/{member_id}", "api", "-", "", "", "", "json", "openapi:members_delete"},
|
||||
{"scopes_list", "GET", "/v1/projects/{project_id}/members/{member_id}/scopes", "api", "-", "", "", "", "json", "openapi:scopes_list"},
|
||||
{"scopes_update", "PUT", "/v1/projects/{project_id}/members/{member_id}/scopes", "api", "json", "scope", "scope", "", "json", "openapi:scopes_update"},
|
||||
{"models_list_2", "GET", "/v1/projects/{project_id}/models", "api", "-", "", "", "include_outdated", "json", "openapi:models_list_2"},
|
||||
{"models_get", "GET", "/v1/projects/{project_id}/models/{model_id}", "api", "-", "", "", "", "json", "openapi:models_get"},
|
||||
{"purchases_list", "GET", "/v1/projects/{project_id}/purchases", "api", "-", "", "", "limit", "json", "openapi:purchases_list"},
|
||||
{"requests_list", "GET", "/v1/projects/{project_id}/requests", "api", "-", "", "", "start,end,limit,page,accessor,request_id,deployment,endpoint,method,status", "json", "openapi:requests_list"},
|
||||
{"requests_get", "GET", "/v1/projects/{project_id}/requests/{request_id}", "api", "-", "", "", "", "json", "openapi:requests_get"},
|
||||
{"distributionCredentials_list", "GET", "/v1/projects/{project_id}/self-hosted/distribution/credentials", "api", "-", "", "", "", "json", "openapi:distributionCredentials_list"},
|
||||
{"distributionCredentials_create", "POST", "/v1/projects/{project_id}/self-hosted/distribution/credentials", "api", "json", "comment", "", "scopes,provider", "json", "openapi:distributionCredentials_create"},
|
||||
{"distributionCredentials_get", "GET", "/v1/projects/{project_id}/self-hosted/distribution/credentials/{distribution_credentials_id}", "api", "-", "", "", "", "json", "openapi:distributionCredentials_get"},
|
||||
{"distributionCredentials_delete", "DELETE", "/v1/projects/{project_id}/self-hosted/distribution/credentials/{distribution_credentials_id}", "api", "-", "", "", "", "json", "openapi:distributionCredentials_delete"},
|
||||
{"usage_get", "GET", "/v1/projects/{project_id}/usage", "api", "-", "", "", "start,end,accessor,alternatives,callback_method,callback,channels,custom_intent_mode,custom_intent,custom_topic_mode,custom_topic,deployment,detect_entities,detect_language,diarize,dictation,encoding,endpoint,extra,filler_words,intents,keyterm,keywords,language,measurements,method,model,multichannel,numerals,paragraphs,profanity_filter,punctuate,redact,replace,sample_rate,search,sentiment,smart_format,summarize,tag,topics,utt_split,utterances,version", "json", "openapi:usage_get"},
|
||||
{"breakdown_get", "GET", "/v1/projects/{project_id}/usage/breakdown", "api", "-", "", "", "start,end,grouping,accessor,alternatives,callback_method,callback,channels,custom_intent_mode,custom_intent,custom_topic_mode,custom_topic,deployment,detect_entities,detect_language,diarize,dictation,encoding,endpoint,extra,filler_words,intents,keyterm,keywords,language,measurements,method,model,multichannel,numerals,paragraphs,profanity_filter,punctuate,redact,replace,sample_rate,search,sentiment,smart_format,summarize,tag,topics,utt_split,utterances,version", "json", "openapi:breakdown_get"},
|
||||
{"fields_list", "GET", "/v1/projects/{project_id}/usage/fields", "api", "-", "", "", "start,end", "json", "openapi:fields_list"},
|
||||
{"text_analyze", "POST", "/v1/read", "api", "json", "url,text", "", "callback,callback_method,sentiment,summarize,tag,topics,custom_topic,custom_topic_mode,intents,custom_intent,custom_intent_mode,language", "json", "openapi:text_analyze"},
|
||||
{"audio_generate", "POST", "/v1/speak", "api", "json", "text", "text", "callback,callback_method,mip_opt_out,tag,bit_rate,container,encoding,model,sample_rate,speed", "bytes", "openapi:audio_generate"},
|
||||
{"audio_generate_2", "POST", "/v2/speak", "api", "json", "text", "text", "callback,callback_method,mip_opt_out,tag,bit_rate,container,encoding,expressivity,model,sample_rate,speed,priority", "bytes", "openapi:audio_generate_2"},
|
||||
{"agent_think_models", "GET", "/v1/agent/settings/think/models", "agent", "-", "", "", "", "json", "openapi:models_list"},
|
||||
{"media_transcribe_bytes", "POST", "/v1/listen", "api", "bytes", "", "", "callback,callback_method,extra,sentiment,summarize,tag,topics,custom_topic,custom_topic_mode,intents,custom_intent,custom_intent_mode,detect_entities,detect_language,diarize,diarize_model,dictation,encoding,filler_words,keyterm,keywords,language,language_hint,measurements,model,multichannel,numerals,paragraphs,profanity_filter,punctuate,redact,replace,search,smart_format,utterances,utt_split,version,mip_opt_out", "json", "guide:listen-pre-recorded (raw audio body; the Fern OpenAPI lists only the JSON url form)"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
static_assert(sizeof(kRows) / sizeof(kRows[0]) == 51, "deepgram call table size");
|
||||
|
||||
const CallTable& calls() {
|
||||
static const CallTable table("deepgram", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+38
@@ -0,0 +1,38 @@
|
||||
#pragma once
|
||||
|
||||
// The Deepgram call table (https://github.com/deepgram/deepgram-api-specs
|
||||
// openapi.yml, Fern generated). Services refer to rows by the ids below.
|
||||
|
||||
#include "vendors/audio/call_spec.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
const CallTable& calls();
|
||||
|
||||
struct DeepgramCalls {
|
||||
static constexpr const char* kListenUrl = "media_transcribe";
|
||||
static constexpr const char* kListenBytes = "media_transcribe_bytes";
|
||||
static constexpr const char* kSpeak = "audio_generate";
|
||||
static constexpr const char* kSpeakV2 = "audio_generate_2";
|
||||
static constexpr const char* kRead = "text_analyze";
|
||||
static constexpr const char* kModels = "models_list_3";
|
||||
static constexpr const char* kModel = "models_get_2";
|
||||
static constexpr const char* kAgentModels = "agent_think_models";
|
||||
static constexpr const char* kGrant = "tokens_grant";
|
||||
static constexpr const char* kProjects = "projects_list";
|
||||
static constexpr const char* kProject = "projects_get";
|
||||
static constexpr const char* kKeys = "keys_list";
|
||||
static constexpr const char* kKeyGet = "keys_get";
|
||||
static constexpr const char* kKeyCreate = "keys_create";
|
||||
static constexpr const char* kKeyDelete = "keys_delete";
|
||||
static constexpr const char* kMembers = "members_list";
|
||||
static constexpr const char* kBalances = "balances_list";
|
||||
static constexpr const char* kRequests = "requests_list";
|
||||
static constexpr const char* kRequest = "requests_get";
|
||||
static constexpr const char* kUsageBreakdown = "breakdown_get";
|
||||
static constexpr const char* kUsageFields = "fields_list";
|
||||
static constexpr const char* kBillingBreakdown = "breakdown_list";
|
||||
static constexpr const char* kPurchases = "purchases_list";
|
||||
};
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
#include "vendors/audio/deepgram/client.h"
|
||||
|
||||
#include "llm/config.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
std::string DeepgramEndpoints::base_url() {
|
||||
return allowlisted_base(kBaseEnv, kDefaultBase, {"https://api.deepgram.com", "https://api.eu.deepgram.com"});
|
||||
}
|
||||
|
||||
llm::ApiClientConfig DeepgramEndpoints::config(const std::string& vendor_base) {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = kVendor;
|
||||
c.base_url = vendor_base;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = "Authorization";
|
||||
return c;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> DeepgramEndpoints::from_env() {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", HostFactory::from_env(config(base_url()), kKeyPrefix));
|
||||
hosts->add("agent", HostFactory::from_env(config(kAgentBase), kKeyPrefix));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> DeepgramEndpoints::with(AudioHost api, AudioHost agent) {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", std::move(api));
|
||||
hosts->add("agent", std::move(agent));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::string DeepgramEndpoints::key() { return llm::Config::get(kKeyEnv); }
|
||||
|
||||
AudioSocket::Headers DeepgramEndpoints::ws_headers(const std::string& key) {
|
||||
return {{"Authorization", std::string(kKeyPrefix) + key}};
|
||||
}
|
||||
|
||||
std::string DeepgramEndpoints::ws_base() { return to_ws_url(base_url()); }
|
||||
std::string DeepgramEndpoints::agent_ws_base() { return to_ws_url(kAgentBase); }
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+38
@@ -0,0 +1,38 @@
|
||||
#pragma once
|
||||
|
||||
// Deepgram endpoints and credentials.
|
||||
// REST https://api.deepgram.com Authorization: Token $DEEPGRAM_API_KEY
|
||||
// Agent https://agent.deepgram.com same key (agent settings catalogue)
|
||||
// WS wss://api.deepgram.com same header; the agent socket is wss://agent.deepgram.com
|
||||
// DEEPGRAM_BASE_URL may select the EU API host (https://api.eu.deepgram.com).
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/hosts.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
struct DeepgramEndpoints {
|
||||
static constexpr const char* kVendor = "deepgram";
|
||||
static constexpr const char* kKeyEnv = "DEEPGRAM_API_KEY";
|
||||
static constexpr const char* kBaseEnv = "DEEPGRAM_BASE_URL";
|
||||
static constexpr const char* kDefaultBase = "https://api.deepgram.com";
|
||||
static constexpr const char* kAgentBase = "https://agent.deepgram.com";
|
||||
// The API key authenticates as "Token <key>"; a temporary token from
|
||||
// /v1/auth/grant authenticates as "Bearer <jwt>" and is out of scope here.
|
||||
static constexpr const char* kKeyPrefix = "Token ";
|
||||
|
||||
static std::string base_url();
|
||||
static llm::ApiClientConfig config(const std::string& vendor_base);
|
||||
static std::shared_ptr<const Hosts> from_env();
|
||||
// Injected hosts (tests): "api" and "agent".
|
||||
static std::shared_ptr<const Hosts> with(AudioHost api, AudioHost agent);
|
||||
static std::string key();
|
||||
static AudioSocket::Headers ws_headers(const std::string& key); // Authorization: Token <key>
|
||||
static std::string ws_base(); // wss://api...
|
||||
static std::string agent_ws_base(); // wss://agent...
|
||||
};
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
#include "vendors/audio/deepgram/listen.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
using C = DeepgramCalls;
|
||||
|
||||
namespace {
|
||||
void tri(Query& q, const char* name, int v) {
|
||||
if (v >= 0) q.add_bool(name, v == 1);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Query Listen::query(const ListenOptions& o, const char* call_id) const {
|
||||
Query q;
|
||||
if (!o.model.empty()) {
|
||||
if (!valid_token(o.model)) bad("model has characters Deepgram does not accept");
|
||||
q.add("model", o.model);
|
||||
}
|
||||
if (!o.language.empty()) {
|
||||
if (!valid_token(o.language)) bad("bad language");
|
||||
q.add("language", o.language);
|
||||
}
|
||||
tri(q, "detect_language", o.detect_language);
|
||||
tri(q, "punctuate", o.punctuate);
|
||||
tri(q, "smart_format", o.smart_format);
|
||||
tri(q, "diarize", o.diarize);
|
||||
tri(q, "utterances", o.utterances);
|
||||
tri(q, "paragraphs", o.paragraphs);
|
||||
tri(q, "numerals", o.numerals);
|
||||
tri(q, "profanity_filter", o.profanity_filter);
|
||||
tri(q, "filler_words", o.filler_words);
|
||||
tri(q, "multichannel", o.multichannel);
|
||||
tri(q, "dictation", o.dictation);
|
||||
tri(q, "sentiment", o.sentiment);
|
||||
tri(q, "topics", o.topics);
|
||||
tri(q, "intents", o.intents);
|
||||
tri(q, "detect_entities", o.detect_entities);
|
||||
tri(q, "mip_opt_out", o.mip_opt_out);
|
||||
if (!o.summarize.empty()) {
|
||||
if (o.summarize != "v2") bad("summarize must be v2");
|
||||
q.add("summarize", o.summarize);
|
||||
}
|
||||
if (!o.encoding.empty()) {
|
||||
if (!valid_token(o.encoding)) bad("bad encoding");
|
||||
q.add("encoding", o.encoding);
|
||||
}
|
||||
struct L { const char* name; const std::vector<std::string>* values; };
|
||||
for (const L& l : {L{"keywords", &o.keywords}, L{"keyterm", &o.keyterms}, L{"search", &o.search},
|
||||
L{"replace", &o.replace}, L{"redact", &o.redact}, L{"tag", &o.tags}}) {
|
||||
if (l.values->size() > DeepgramCaps::kMaxTerms) bad(std::string("too many ") + l.name);
|
||||
for (const auto& v : *l.values) {
|
||||
require_text(l.name, v, DeepgramCaps::kMaxTermChars);
|
||||
q.add(l.name, v);
|
||||
}
|
||||
}
|
||||
q.add_if("version", o.version);
|
||||
q.add_if("callback", o.callback);
|
||||
q.add_if("callback_method", o.callback_method);
|
||||
if (o.extra.is_object())
|
||||
for (const auto& [k, v] : o.extra.as_object())
|
||||
if (!q.add_json(k, v)) bad("extra '" + k + "' must be a scalar or a list of scalars");
|
||||
check_query(table_.get(call_id), q);
|
||||
return q;
|
||||
}
|
||||
|
||||
Json Listen::transcribe_url(const std::string& url, const ListenOptions& o) {
|
||||
if (url.rfind("https://", 0) != 0 && url.rfind("http://", 0) != 0) bad("url must be http or https");
|
||||
require_text("url", url, AudioCaps::kMaxFieldChars);
|
||||
Json body = Json::object();
|
||||
body.set("url", Json(url));
|
||||
return call(C::kListenUrl, {}, &body, query(o, C::kListenUrl));
|
||||
}
|
||||
|
||||
Json Listen::transcribe_bytes(const std::string& audio, const std::string& content_type,
|
||||
const ListenOptions& o) {
|
||||
if (audio.size() > DeepgramCaps::kMaxFileBytes) bad("audio is over the byte cap");
|
||||
llm::RawResponse raw = call_bytes(C::kListenBytes, {}, audio, content_type, query(o, C::kListenBytes));
|
||||
try {
|
||||
return Json::parse(raw.body);
|
||||
} catch (const std::exception& e) {
|
||||
throw llm::ApiError("deepgram", 502, "BAD_RESPONSE", e.what());
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+39
@@ -0,0 +1,39 @@
|
||||
#pragma once
|
||||
|
||||
// Pre-recorded speech to text: POST /v1/listen with a URL (JSON body) or the
|
||||
// audio bytes as the body. Options become query parameters; every name is
|
||||
// checked against the call table.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/deepgram/service.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
struct ListenOptions {
|
||||
std::string model; // "" = server default; token charset
|
||||
std::string language; // BCP-47
|
||||
int detect_language = -1; // tri state: -1 unset, 0 false, 1 true
|
||||
int punctuate = -1, smart_format = -1, diarize = -1, utterances = -1, paragraphs = -1;
|
||||
int numerals = -1, profanity_filter = -1, filler_words = -1, multichannel = -1, dictation = -1;
|
||||
int sentiment = -1, topics = -1, intents = -1, detect_entities = -1, mip_opt_out = -1;
|
||||
std::string summarize; // "", "v2"
|
||||
std::string encoding; // headerless raw audio only
|
||||
std::vector<std::string> keywords, keyterms, search, replace, redact, tags;
|
||||
std::string version, callback, callback_method;
|
||||
Json extra; // any other query parameter in the table: object of scalars / lists
|
||||
};
|
||||
|
||||
class Listen : public DeepgramService {
|
||||
public:
|
||||
using DeepgramService::DeepgramService;
|
||||
Query query(const ListenOptions& options, const char* call_id) const;
|
||||
// Remote media by URL.
|
||||
Json transcribe_url(const std::string& url, const ListenOptions& options = {});
|
||||
// Audio bytes as the body; content_type like audio/wav, audio/mpeg, audio/*.
|
||||
Json transcribe_bytes(const std::string& audio, const std::string& content_type,
|
||||
const ListenOptions& options = {});
|
||||
};
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
+100
@@ -0,0 +1,100 @@
|
||||
#include "vendors/audio/deepgram/manage.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
using C = DeepgramCalls;
|
||||
|
||||
Json Manage::projects() { return call(C::kProjects); }
|
||||
|
||||
Json Manage::project(const std::string& id, const Json& q) {
|
||||
check_id("project", id);
|
||||
return call(C::kProject, {id}, nullptr, query_from(C::kProject, q));
|
||||
}
|
||||
|
||||
Json Manage::models(bool outdated) {
|
||||
Query q;
|
||||
if (outdated) q.add_bool("include_outdated", true);
|
||||
return call(C::kModels, {}, nullptr, q);
|
||||
}
|
||||
|
||||
Json Manage::model(const std::string& id) {
|
||||
check_id("model", id);
|
||||
return call(C::kModel, {id});
|
||||
}
|
||||
|
||||
Json Manage::project_models(const std::string& id, bool outdated) {
|
||||
check_id("project", id);
|
||||
Query q;
|
||||
if (outdated) q.add_bool("include_outdated", true);
|
||||
return call("models_list_2", {id}, nullptr, q);
|
||||
}
|
||||
|
||||
Json Manage::agent_think_models() { return call(C::kAgentModels); }
|
||||
|
||||
Json Manage::keys(const std::string& id, const std::string& status) {
|
||||
check_id("project", id);
|
||||
Query q;
|
||||
if (!status.empty()) {
|
||||
if (status != "active" && status != "expired") bad("status must be active or expired");
|
||||
q.add("status", status);
|
||||
}
|
||||
return call(C::kKeys, {id}, nullptr, q);
|
||||
}
|
||||
|
||||
Json Manage::key(const std::string& id, const std::string& key_id) {
|
||||
check_id("project", id);
|
||||
check_id("key", key_id);
|
||||
return call(C::kKeyGet, {id, key_id});
|
||||
}
|
||||
|
||||
Json Manage::create_key(const std::string& id, const Json& body, bool confirm) {
|
||||
check_id("project", id);
|
||||
check_body(table_.get(C::kKeyCreate), &body);
|
||||
require_confirm("creating an API key", confirm);
|
||||
return call(C::kKeyCreate, {id}, &body);
|
||||
}
|
||||
|
||||
Json Manage::delete_key(const std::string& id, const std::string& key_id, bool confirm) {
|
||||
check_id("project", id);
|
||||
check_id("key", key_id);
|
||||
return call_destructive(C::kKeyDelete, {id, key_id}, nullptr, Query(), confirm);
|
||||
}
|
||||
|
||||
Json Manage::members(const std::string& id) {
|
||||
check_id("project", id);
|
||||
return call(C::kMembers, {id});
|
||||
}
|
||||
|
||||
Json Manage::balances(const std::string& id) {
|
||||
check_id("project", id);
|
||||
return call(C::kBalances, {id});
|
||||
}
|
||||
|
||||
Json Manage::requests(const std::string& id, const Json& q) {
|
||||
check_id("project", id);
|
||||
return call(C::kRequests, {id}, nullptr, query_from(C::kRequests, q));
|
||||
}
|
||||
|
||||
Json Manage::usage_breakdown(const std::string& id, const Json& q) {
|
||||
check_id("project", id);
|
||||
return call(C::kUsageBreakdown, {id}, nullptr, query_from(C::kUsageBreakdown, q));
|
||||
}
|
||||
|
||||
Json Manage::usage_fields(const std::string& id, const Json& q) {
|
||||
check_id("project", id);
|
||||
return call(C::kUsageFields, {id}, nullptr, query_from(C::kUsageFields, q));
|
||||
}
|
||||
|
||||
Json Manage::billing_breakdown(const std::string& id, const Json& q) {
|
||||
check_id("project", id);
|
||||
return call(C::kBillingBreakdown, {id}, nullptr, query_from(C::kBillingBreakdown, q));
|
||||
}
|
||||
|
||||
Json Manage::grant_token(int ttl) {
|
||||
if (ttl < 1 || ttl > 3600) bad("ttl_seconds must be 1..3600");
|
||||
Json body = Json::object();
|
||||
body.set("ttl_seconds", Json(ttl));
|
||||
return call(C::kGrant, {}, &body);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+40
@@ -0,0 +1,40 @@
|
||||
#pragma once
|
||||
|
||||
// Management API: projects, keys, members, balances, requests, usage,
|
||||
// billing, model catalogue, temporary tokens. Reads are plain calls; key
|
||||
// creation and every delete need confirm. Further rows (invites, scopes,
|
||||
// self-hosted credentials, agent variables and configurations) are in the
|
||||
// call table and reached with call() / call_destructive().
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/deepgram/service.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
class Manage : public DeepgramService {
|
||||
public:
|
||||
using DeepgramService::DeepgramService;
|
||||
|
||||
Json projects();
|
||||
Json project(const std::string& project_id, const Json& query_object = Json());
|
||||
Json models(bool include_outdated = false);
|
||||
Json model(const std::string& model_id);
|
||||
Json project_models(const std::string& project_id, bool include_outdated = false);
|
||||
Json agent_think_models();
|
||||
Json keys(const std::string& project_id, const std::string& status = "");
|
||||
Json key(const std::string& project_id, const std::string& key_id);
|
||||
// Creates a key (the secret is returned once): a write, confirm required.
|
||||
Json create_key(const std::string& project_id, const Json& body, bool confirm);
|
||||
Json delete_key(const std::string& project_id, const std::string& key_id, bool confirm);
|
||||
Json members(const std::string& project_id);
|
||||
Json balances(const std::string& project_id);
|
||||
Json requests(const std::string& project_id, const Json& query_object = Json());
|
||||
Json usage_breakdown(const std::string& project_id, const Json& query_object = Json());
|
||||
Json usage_fields(const std::string& project_id, const Json& query_object = Json());
|
||||
Json billing_breakdown(const std::string& project_id, const Json& query_object = Json());
|
||||
// Short lived access token (JWT) for browser use; ttl 1..3600 seconds.
|
||||
Json grant_token(int ttl_seconds);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+53
@@ -0,0 +1,53 @@
|
||||
#include "vendors/audio/deepgram/read.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
using C = DeepgramCalls;
|
||||
|
||||
Json Read::build(const ReadRequest& r) const {
|
||||
if (r.text.empty() == r.url.empty()) bad("give exactly one of text and url");
|
||||
Json body = Json::object();
|
||||
if (!r.text.empty()) {
|
||||
require_text("text", r.text, DeepgramCaps::kMaxReadChars);
|
||||
body.set("text", Json(r.text));
|
||||
} else {
|
||||
if (r.url.rfind("https://", 0) != 0 && r.url.rfind("http://", 0) != 0) bad("url must be http or https");
|
||||
require_text("url", r.url, 8192);
|
||||
body.set("url", Json(r.url));
|
||||
}
|
||||
return body;
|
||||
}
|
||||
|
||||
Query Read::query(const ReadRequest& r) const {
|
||||
Query q;
|
||||
if (!r.language.empty()) {
|
||||
if (!valid_token(r.language)) bad("bad language");
|
||||
q.add("language", r.language);
|
||||
}
|
||||
if (!r.summarize.empty()) {
|
||||
if (r.summarize != "v2") bad("summarize must be v2");
|
||||
q.add("summarize", r.summarize);
|
||||
}
|
||||
if (r.topics >= 0) q.add_bool("topics", r.topics == 1);
|
||||
if (r.intents >= 0) q.add_bool("intents", r.intents == 1);
|
||||
if (r.sentiment >= 0) q.add_bool("sentiment", r.sentiment == 1);
|
||||
if (r.custom_topics.size() > DeepgramCaps::kMaxTerms || r.custom_intents.size() > DeepgramCaps::kMaxTerms)
|
||||
bad("too many custom topics or intents");
|
||||
for (const auto& t : r.custom_topics) { require_text("custom_topic", t, DeepgramCaps::kMaxTermChars); q.add("custom_topic", t); }
|
||||
for (const auto& t : r.custom_intents) { require_text("custom_intent", t, DeepgramCaps::kMaxTermChars); q.add("custom_intent", t); }
|
||||
for (const auto& [name, mode] : {std::pair<const char*, const std::string*>{"custom_topic_mode", &r.custom_topic_mode},
|
||||
{"custom_intent_mode", &r.custom_intent_mode}}) {
|
||||
if (mode->empty()) continue;
|
||||
if (*mode != "extended" && *mode != "strict") bad(std::string(name) + " must be extended or strict");
|
||||
q.add(name, *mode);
|
||||
}
|
||||
check_query(table_.get(C::kRead), q);
|
||||
return q;
|
||||
}
|
||||
|
||||
Json Read::analyze(const ReadRequest& r) {
|
||||
Json body = build(r);
|
||||
return call(C::kRead, {}, &body, query(r));
|
||||
}
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+30
@@ -0,0 +1,30 @@
|
||||
#pragma once
|
||||
|
||||
// Text intelligence: POST /v1/read (summaries, topics, intents, sentiment).
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/deepgram/service.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
struct ReadRequest {
|
||||
std::string text; // exactly one of text and url
|
||||
std::string url;
|
||||
std::string language = "en"; // the API supports English only
|
||||
std::string summarize; // "", v2
|
||||
int topics = -1, intents = -1, sentiment = -1;
|
||||
std::vector<std::string> custom_topics, custom_intents;
|
||||
std::string custom_topic_mode, custom_intent_mode; // "", extended, strict
|
||||
};
|
||||
|
||||
class Read : public DeepgramService {
|
||||
public:
|
||||
using DeepgramService::DeepgramService;
|
||||
Json build(const ReadRequest& request) const;
|
||||
Query query(const ReadRequest& request) const;
|
||||
Json analyze(const ReadRequest& request);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
#include "vendors/audio/deepgram/service.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
bool valid_token(const std::string& s) {
|
||||
if (s.empty() || s.size() > 128) return false;
|
||||
for (char c : s)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' ||
|
||||
c == '_' || c == '.' || c == ':'))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+29
@@ -0,0 +1,29 @@
|
||||
#pragma once
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "vendors/audio/audio_service.h"
|
||||
#include "vendors/audio/deepgram/calls.h"
|
||||
#include "vendors/audio/deepgram/client.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
struct DeepgramCaps {
|
||||
static constexpr std::size_t kMaxSpeakChars = 2000; // Deepgram TTS REST input, per the TTS guide
|
||||
static constexpr std::size_t kMaxReadChars = 100000; // our ceiling for /v1/read text
|
||||
static constexpr std::size_t kMaxFileBytes = 128u * 1024u * 1024u; // our ceiling (provider allows 2 GB)
|
||||
static constexpr std::size_t kMaxTerms = 100; // keywords / keyterm / search / replace / redact entries
|
||||
static constexpr std::size_t kMaxTermChars = 100;
|
||||
};
|
||||
|
||||
class DeepgramService : public AudioService {
|
||||
public:
|
||||
explicit DeepgramService(std::shared_ptr<const Hosts> hosts)
|
||||
: AudioService(calls(), std::move(hosts)) {}
|
||||
DeepgramService() : DeepgramService(DeepgramEndpoints::from_env()) {}
|
||||
};
|
||||
|
||||
// A model name or other token Deepgram puts in a query: [A-Za-z0-9._:-]{1,128}.
|
||||
bool valid_token(const std::string& s);
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+59
@@ -0,0 +1,59 @@
|
||||
#include "vendors/audio/deepgram/speak.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
using C = DeepgramCalls;
|
||||
|
||||
Json Speak::build(const SpeakOptions& o) const {
|
||||
require_text("text", o.text, DeepgramCaps::kMaxSpeakChars);
|
||||
Json body = Json::object();
|
||||
body.set("text", Json(o.text));
|
||||
return body;
|
||||
}
|
||||
|
||||
Query Speak::query(const SpeakOptions& o, bool v2) const {
|
||||
Query q;
|
||||
if (!o.model.empty()) {
|
||||
if (!valid_token(o.model)) bad("model has characters Deepgram does not accept");
|
||||
q.add("model", o.model);
|
||||
}
|
||||
if (!o.encoding.empty()) {
|
||||
if (!valid_token(o.encoding)) bad("bad encoding");
|
||||
q.add("encoding", o.encoding);
|
||||
}
|
||||
if (!o.container.empty()) {
|
||||
if (!valid_token(o.container)) bad("bad container");
|
||||
q.add("container", o.container);
|
||||
}
|
||||
if (o.sample_rate < 0 || o.sample_rate > 96000) bad("sample_rate must be 0 or up to 96000");
|
||||
if (o.sample_rate > 0) q.add_int("sample_rate", o.sample_rate);
|
||||
if (o.bit_rate < 0 || o.bit_rate > 1000000) bad("bit_rate is out of range");
|
||||
if (o.bit_rate > 0) q.add_int("bit_rate", o.bit_rate);
|
||||
if (o.speed < 0) bad("speed must not be negative");
|
||||
if (o.speed > 0) q.add("speed", std::to_string(o.speed));
|
||||
if (v2) {
|
||||
q.add_if("expressivity", o.expressivity);
|
||||
q.add_if("priority", o.priority);
|
||||
} else if (!o.expressivity.empty() || !o.priority.empty()) {
|
||||
bad("expressivity and priority belong to /v2/speak");
|
||||
}
|
||||
q.add_if("tag", o.tag);
|
||||
q.add_if("callback", o.callback);
|
||||
q.add_if("callback_method", o.callback_method);
|
||||
if (o.mip_opt_out >= 0) q.add_bool("mip_opt_out", o.mip_opt_out == 1);
|
||||
check_query(table_.get(v2 ? C::kSpeakV2 : C::kSpeak), q);
|
||||
return q;
|
||||
}
|
||||
|
||||
SpeakResult Speak::synthesize(const SpeakOptions& o, bool v2) {
|
||||
Json body = build(o);
|
||||
llm::RawResponse raw = call_raw(v2 ? C::kSpeakV2 : C::kSpeak, {}, &body, query(o, v2));
|
||||
return SpeakResult{raw.body, raw.content_type};
|
||||
}
|
||||
|
||||
ByteStreamer::Result Speak::stream(const SpeakOptions& o, const ChunkSink& sink, bool v2) {
|
||||
Json body = build(o);
|
||||
return call_stream(v2 ? C::kSpeakV2 : C::kSpeak, {}, &body, query(o, v2), sink);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+40
@@ -0,0 +1,40 @@
|
||||
#pragma once
|
||||
|
||||
// Text to speech: POST /v1/speak (Aura) and POST /v2/speak (Flux TTS).
|
||||
// The response body is the audio.
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/deepgram/service.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
struct SpeakOptions {
|
||||
std::string text; // 1..2000 characters
|
||||
std::string model; // "" = server default, e.g. aura-2-thalia-en
|
||||
std::string encoding; // linear16 mulaw alaw mp3 opus flac aac
|
||||
std::string container; // wav ogg none
|
||||
int sample_rate = 0; // 0 unset
|
||||
int bit_rate = 0; // 0 unset
|
||||
double speed = 0; // 0 unset (v1/v2 speed query parameter)
|
||||
std::string expressivity; // v2 only
|
||||
std::string priority; // v2 only
|
||||
std::string tag, callback, callback_method;
|
||||
int mip_opt_out = -1;
|
||||
};
|
||||
|
||||
struct SpeakResult {
|
||||
std::string audio;
|
||||
std::string content_type;
|
||||
};
|
||||
|
||||
class Speak : public DeepgramService {
|
||||
public:
|
||||
using DeepgramService::DeepgramService;
|
||||
Json build(const SpeakOptions& options) const;
|
||||
Query query(const SpeakOptions& options, bool v2) const;
|
||||
SpeakResult synthesize(const SpeakOptions& options, bool v2 = false);
|
||||
ByteStreamer::Result stream(const SpeakOptions& options, const ChunkSink& sink, bool v2 = false);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
+103
@@ -0,0 +1,103 @@
|
||||
#include "vendors/audio/deepgram/speech.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("deepgram", why); }
|
||||
} // namespace
|
||||
|
||||
DeepgramSynthesizer::DeepgramSynthesizer(std::shared_ptr<const Hosts> hosts) : speak_(hosts), manage_(hosts) {}
|
||||
|
||||
SpeakOptions DeepgramSynthesizer::map(const SpeechRequest& r) const {
|
||||
validate(r);
|
||||
SpeakOptions o;
|
||||
o.text = r.text;
|
||||
o.model = !r.voice.empty() ? r.voice : r.model; // Aura voices are models: aura-2-thalia-en
|
||||
if (r.format == "mp3") {
|
||||
o.encoding = "mp3";
|
||||
if (r.sample_rate != 0) invalid("mp3 has a fixed sample rate");
|
||||
} else {
|
||||
o.encoding = "linear16";
|
||||
o.container = r.format == "wav" ? "wav" : "none";
|
||||
if (r.sample_rate != 0) {
|
||||
bool ok = false;
|
||||
for (int v : {8000, 16000, 24000, 32000, 48000}) ok = ok || v == r.sample_rate;
|
||||
if (!ok) invalid("linear16 sample_rate must be 8000, 16000, 24000, 32000 or 48000");
|
||||
o.sample_rate = r.sample_rate;
|
||||
}
|
||||
}
|
||||
if (r.speed > 0) o.speed = r.speed;
|
||||
return o;
|
||||
}
|
||||
|
||||
SpeechResult DeepgramSynthesizer::synthesize(const SpeechRequest& r) {
|
||||
SpeakResult res = speak_.synthesize(map(r));
|
||||
SpeechResult out;
|
||||
out.audio = std::move(res.audio);
|
||||
out.content_type = res.content_type.empty() ? content_type_for(r.format) : res.content_type;
|
||||
out.format = r.format;
|
||||
out.provider = name();
|
||||
return out;
|
||||
}
|
||||
|
||||
Json DeepgramSynthesizer::voices() { return manage_.models(); }
|
||||
|
||||
DeepgramRecognizer::DeepgramRecognizer(std::shared_ptr<const Hosts> hosts) : listen_(std::move(hosts)) {}
|
||||
|
||||
ListenOptions DeepgramRecognizer::map(const TranscribeRequest& r) const {
|
||||
validate(r);
|
||||
ListenOptions o;
|
||||
o.model = r.model;
|
||||
o.language = r.language;
|
||||
if (r.language.empty()) o.detect_language = 1;
|
||||
o.punctuate = 1;
|
||||
o.smart_format = 1;
|
||||
o.utterances = 1;
|
||||
if (r.diarize) o.diarize = 1;
|
||||
o.keyterms = r.keywords;
|
||||
return o;
|
||||
}
|
||||
|
||||
Transcript DeepgramRecognizer::fold(const Json& doc) {
|
||||
Transcript t;
|
||||
t.provider = "deepgram";
|
||||
t.raw = doc;
|
||||
if (!doc.is_object()) return t;
|
||||
if (const Json* meta = doc.find("metadata"); meta && meta->is_object()) {
|
||||
if (const Json* d = meta->find("duration"); d && d->is_number()) t.duration = d->as_number();
|
||||
}
|
||||
const Json* results = doc.find("results");
|
||||
const Json* channels = results && results->is_object() ? results->find("channels") : nullptr;
|
||||
if (!channels || !channels->is_array() || channels->as_array().empty()) return t;
|
||||
const Json& ch = channels->as_array()[0];
|
||||
if (const Json* l = ch.find("detected_language"); l && l->is_string()) t.language = l->as_string();
|
||||
const Json* alts = ch.find("alternatives");
|
||||
if (!alts || !alts->is_array() || alts->as_array().empty()) return t;
|
||||
const Json& alt = alts->as_array()[0];
|
||||
if (const Json* x = alt.find("transcript"); x && x->is_string()) t.text = x->as_string();
|
||||
if (const Json* words = alt.find("words"); words && words->is_array()) {
|
||||
for (const Json& w : words->as_array()) {
|
||||
if (!w.is_object()) continue;
|
||||
TranscriptWord word;
|
||||
const Json* pw = w.find("punctuated_word");
|
||||
const Json* raw = w.find("word");
|
||||
if (pw && pw->is_string()) word.text = pw->as_string();
|
||||
else if (raw && raw->is_string()) word.text = raw->as_string();
|
||||
if (const Json* x = w.find("start"); x && x->is_number()) word.start = x->as_number();
|
||||
if (const Json* x = w.find("end"); x && x->is_number()) word.end = x->as_number();
|
||||
if (const Json* x = w.find("confidence"); x && x->is_number()) word.confidence = x->as_number();
|
||||
if (const Json* x = w.find("speaker"); x && x->is_number()) word.speaker = std::to_string(x->as_int());
|
||||
t.words.push_back(word);
|
||||
}
|
||||
}
|
||||
return t;
|
||||
}
|
||||
|
||||
Transcript DeepgramRecognizer::transcribe(const TranscribeRequest& r) {
|
||||
ListenOptions o = map(r);
|
||||
Json doc = !r.url.empty() ? listen_.transcribe_url(r.url, o)
|
||||
: listen_.transcribe_bytes(r.audio, r.content_type, o);
|
||||
return fold(doc);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+43
@@ -0,0 +1,43 @@
|
||||
#pragma once
|
||||
|
||||
// Deepgram behind the neutral speech seam.
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "vendors/audio/deepgram/listen.h"
|
||||
#include "vendors/audio/deepgram/manage.h"
|
||||
#include "vendors/audio/deepgram/speak.h"
|
||||
#include "vendors/audio/speech.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
class DeepgramSynthesizer : public SpeechSynthesizer {
|
||||
public:
|
||||
explicit DeepgramSynthesizer(std::shared_ptr<const Hosts> hosts = DeepgramEndpoints::from_env());
|
||||
std::string name() const override { return "deepgram"; }
|
||||
bool configured() const override { return speak_.configured(); }
|
||||
std::size_t max_chars() const override { return DeepgramCaps::kMaxSpeakChars; }
|
||||
std::vector<std::string> formats() const override { return {"mp3", "wav", "pcm"}; }
|
||||
SpeechResult synthesize(const SpeechRequest& request) override;
|
||||
Json voices() override; // the model catalogue (tts section)
|
||||
SpeakOptions map(const SpeechRequest& request) const;
|
||||
|
||||
private:
|
||||
Speak speak_;
|
||||
Manage manage_;
|
||||
};
|
||||
|
||||
class DeepgramRecognizer : public SpeechRecognizer {
|
||||
public:
|
||||
explicit DeepgramRecognizer(std::shared_ptr<const Hosts> hosts = DeepgramEndpoints::from_env());
|
||||
std::string name() const override { return "deepgram"; }
|
||||
bool configured() const override { return listen_.configured(); }
|
||||
Transcript transcribe(const TranscribeRequest& request) override;
|
||||
ListenOptions map(const TranscribeRequest& request) const;
|
||||
static Transcript fold(const Json& doc);
|
||||
|
||||
private:
|
||||
Listen listen_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
+325
@@ -0,0 +1,325 @@
|
||||
#include "vendors/audio/deepgram/streams.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("deepgram", why); }
|
||||
|
||||
std::string with_params(const std::string& base, const StreamParams& p, const std::vector<std::string>& allowed) {
|
||||
Query q;
|
||||
for (const auto& [k, v] : p.pairs) {
|
||||
bool ok = false;
|
||||
for (const auto& a : allowed) ok = ok || a == k;
|
||||
if (!ok) invalid("unknown stream parameter '" + k + "'");
|
||||
if (v.size() > 256) invalid("stream parameter value is too long");
|
||||
q.add(k, v);
|
||||
}
|
||||
std::string s = q.str();
|
||||
return s.empty() ? base : base + s;
|
||||
}
|
||||
|
||||
std::string str_at(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
|
||||
double num_at(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_number() ? v->as_number() : 0.0;
|
||||
}
|
||||
|
||||
bool bool_at(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_bool() && v->as_bool();
|
||||
}
|
||||
|
||||
Json typed(const char* type) {
|
||||
Json m = Json::object();
|
||||
m.set("type", Json(type));
|
||||
return m;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const std::vector<std::string>& StreamChannels::listen_v1_params() {
|
||||
static const std::vector<std::string> v = {
|
||||
"callback", "callback_method", "channels", "detect_entities", "diarize", "diarize_model", "dictation",
|
||||
"encoding", "endpointing", "extra", "interim_results", "keyterm", "keywords", "language",
|
||||
"language_hint", "mip_opt_out", "model", "multichannel", "numerals", "profanity_filter", "punctuate",
|
||||
"redact", "replace", "sample_rate", "search", "smart_format", "tag", "utterance_end_ms", "vad_events",
|
||||
"version"};
|
||||
return v;
|
||||
}
|
||||
const std::vector<std::string>& StreamChannels::listen_v2_params() {
|
||||
static const std::vector<std::string> v = {
|
||||
"model", "encoding", "sample_rate", "eager_eot_threshold", "eot_threshold", "eot_timeout_ms",
|
||||
"keyterm", "language_hint", "profanity_filter", "numerals", "redact", "mip_opt_out", "tag"};
|
||||
return v;
|
||||
}
|
||||
const std::vector<std::string>& StreamChannels::speak_v1_params() {
|
||||
static const std::vector<std::string> v = {"encoding", "mip_opt_out", "model", "sample_rate", "speed"};
|
||||
return v;
|
||||
}
|
||||
const std::vector<std::string>& StreamChannels::speak_v2_params() {
|
||||
static const std::vector<std::string> v = {"model", "encoding", "sample_rate", "speed", "expressivity",
|
||||
"mip_opt_out", "tag"};
|
||||
return v;
|
||||
}
|
||||
|
||||
Vocabulary StreamChannels::listen_v1() {
|
||||
return Vocabulary("type", {"KeepAlive", "Finalize", "CloseStream", "Configure"},
|
||||
{"Results", "Metadata", "UtteranceEnd", "SpeechStarted", "ConfigureSuccess",
|
||||
"ConfigureFailure", "Error"});
|
||||
}
|
||||
Vocabulary StreamChannels::listen_v2() {
|
||||
return Vocabulary("type", {"CloseStream", "ForceEndTurn", "Configure"},
|
||||
{"Connected", "TurnInfo", "ConfigureSuccess", "ConfigureFailure", "Warning", "Error"});
|
||||
}
|
||||
Vocabulary StreamChannels::speak_v1() {
|
||||
return Vocabulary("type", {"Speak", "Flush", "Clear", "Close"},
|
||||
{"Metadata", "Flushed", "Cleared", "Warning"});
|
||||
}
|
||||
Vocabulary StreamChannels::speak_v2() {
|
||||
return Vocabulary("type", {"Speak", "Flush", "Interrupt", "Configure", "Close"},
|
||||
{"Connected", "SpeechStarted", "SpeechMetadata", "SpeechInterrupted", "Flushed",
|
||||
"SessionMetadata", "ConfigureSuccess", "ConfigureFailure", "Warning", "Error"});
|
||||
}
|
||||
Vocabulary StreamChannels::agent() {
|
||||
return Vocabulary(
|
||||
"type",
|
||||
{"Settings", "UpdateListen", "UpdateThink", "UpdateSpeak", "InjectUserMessage", "InjectAgentMessage",
|
||||
"FunctionCallResponse", "KeepAlive", "UpdatePrompt", "ForceEndTurn"},
|
||||
{"Welcome", "SettingsApplied", "ConversationText", "UserStartedSpeaking", "AgentThinking",
|
||||
"FunctionCallRequest", "FunctionCallResponse", "FunctionCallCancelled", "AgentStartedSpeaking",
|
||||
"AgentAudioDone", "Error", "Warning", "History", "PromptUpdated", "SpeakUpdated", "ThinkUpdated",
|
||||
"ListenUpdated", "InjectionRefused", "LatencyReport"});
|
||||
}
|
||||
|
||||
// ---- listen ---------------------------------------------------------------
|
||||
|
||||
ListenStream::ListenStream(std::string api_key, StreamParams params, bool flux, std::string url,
|
||||
SocketOptions socket)
|
||||
: flux_(flux) {
|
||||
std::string base = url.empty() ? DeepgramEndpoints::ws_base() + (flux ? "/v2/listen" : "/v1/listen") : url;
|
||||
std::string full = with_params(base, params, flux ? StreamChannels::listen_v2_params() : StreamChannels::listen_v1_params());
|
||||
socket_ = std::make_unique<AudioSocket>("deepgram", full, DeepgramEndpoints::ws_headers(api_key),
|
||||
flux ? StreamChannels::listen_v2() : StreamChannels::listen_v1(),
|
||||
std::move(socket));
|
||||
}
|
||||
|
||||
void ListenStream::keep_alive() { socket_->send_json(typed("KeepAlive")); }
|
||||
void ListenStream::finalize() {
|
||||
if (flux_) invalid("Finalize belongs to /v1/listen; use force_end_turn on /v2/listen");
|
||||
socket_->send_json(typed("Finalize"));
|
||||
}
|
||||
void ListenStream::force_end_turn() {
|
||||
if (!flux_) invalid("ForceEndTurn belongs to /v2/listen; use finalize on /v1/listen");
|
||||
socket_->send_json(typed("ForceEndTurn"));
|
||||
}
|
||||
void ListenStream::close_stream() { socket_->send_json(typed("CloseStream")); }
|
||||
|
||||
ListenEvent ListenStream::decode(const SocketEvent& ev) {
|
||||
ListenEvent out;
|
||||
if (ev.kind == SocketEvent::Kind::kClose) {
|
||||
out.kind = ListenEvent::Kind::kClose;
|
||||
return out;
|
||||
}
|
||||
if (ev.kind != SocketEvent::Kind::kText || !ev.json_ok) return out;
|
||||
out.raw = ev.json;
|
||||
const Json& j = ev.json;
|
||||
if (ev.type == "Results") {
|
||||
out.kind = ListenEvent::Kind::kResults;
|
||||
out.is_final = bool_at(j, "is_final");
|
||||
out.speech_final = bool_at(j, "speech_final");
|
||||
if (const Json* ch = j.find("channel"); ch && ch->is_object()) {
|
||||
if (const Json* alts = ch->find("alternatives"); alts && alts->is_array() && !alts->as_array().empty()) {
|
||||
out.transcript = str_at(alts->as_array()[0], "transcript");
|
||||
out.confidence = num_at(alts->as_array()[0], "confidence");
|
||||
}
|
||||
}
|
||||
} else if (ev.type == "Metadata") {
|
||||
out.kind = ListenEvent::Kind::kMetadata;
|
||||
out.request_id = str_at(j, "request_id");
|
||||
} else if (ev.type == "UtteranceEnd") {
|
||||
out.kind = ListenEvent::Kind::kUtteranceEnd;
|
||||
} else if (ev.type == "SpeechStarted") {
|
||||
out.kind = ListenEvent::Kind::kSpeechStarted;
|
||||
} else if (ev.type == "TurnInfo") {
|
||||
out.kind = ListenEvent::Kind::kTurn;
|
||||
out.turn_event = str_at(j, "event");
|
||||
out.transcript = str_at(j, "transcript");
|
||||
out.confidence = num_at(j, "end_of_turn_confidence");
|
||||
out.is_final = out.turn_event == "EndOfTurn";
|
||||
} else if (ev.type == "Error" || ev.type == "ConfigureFailure") {
|
||||
out.kind = ListenEvent::Kind::kError;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
ListenResult ListenStream::collect(int max_wait_ms) {
|
||||
ListenResult out;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto ev = socket_->next(max_wait_ms);
|
||||
if (!ev) return out;
|
||||
++out.events;
|
||||
ListenEvent d = decode(*ev);
|
||||
switch (d.kind) {
|
||||
case ListenEvent::Kind::kResults:
|
||||
case ListenEvent::Kind::kTurn:
|
||||
if (d.is_final) {
|
||||
if (!d.transcript.empty()) {
|
||||
out.finals.push_back(d.transcript);
|
||||
out.transcript += (out.transcript.empty() ? "" : " ") + d.transcript;
|
||||
}
|
||||
out.interim.clear();
|
||||
} else {
|
||||
out.interim = d.transcript;
|
||||
}
|
||||
break;
|
||||
case ListenEvent::Kind::kUtteranceEnd: ++out.utterance_ends; break;
|
||||
case ListenEvent::Kind::kMetadata:
|
||||
out.request_id = d.request_id;
|
||||
out.duration = num_at(d.raw, "duration");
|
||||
return out;
|
||||
case ListenEvent::Kind::kError:
|
||||
out.error = str_at(d.raw, "description");
|
||||
if (out.error.empty()) out.error = str_at(d.raw, "message");
|
||||
if (out.error.empty()) out.error = "stream error";
|
||||
return out;
|
||||
case ListenEvent::Kind::kClose: out.closed = true; return out;
|
||||
default: break;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---- speak ----------------------------------------------------------------
|
||||
|
||||
SpeakStream::SpeakStream(std::string api_key, StreamParams params, bool v2, std::string url,
|
||||
SocketOptions socket)
|
||||
: v2_(v2) {
|
||||
std::string base = url.empty() ? DeepgramEndpoints::ws_base() + (v2 ? "/v2/speak" : "/v1/speak") : url;
|
||||
std::string full = with_params(base, params, v2 ? StreamChannels::speak_v2_params() : StreamChannels::speak_v1_params());
|
||||
socket_ = std::make_unique<AudioSocket>("deepgram", full, DeepgramEndpoints::ws_headers(api_key),
|
||||
v2 ? StreamChannels::speak_v2() : StreamChannels::speak_v1(),
|
||||
std::move(socket));
|
||||
}
|
||||
|
||||
void SpeakStream::speak(const std::string& text) {
|
||||
if (text.empty() || text.size() > DeepgramCaps::kMaxSpeakChars) invalid("text must be 1..2000 characters");
|
||||
if (text.find('\0') != std::string::npos) invalid("text has a NUL byte");
|
||||
Json m = typed("Speak");
|
||||
m.set("text", Json(text));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
void SpeakStream::flush() { socket_->send_json(typed("Flush")); }
|
||||
void SpeakStream::clear() {
|
||||
if (v2_) invalid("Clear belongs to /v1/speak; use interrupt on /v2/speak");
|
||||
socket_->send_json(typed("Clear"));
|
||||
}
|
||||
void SpeakStream::interrupt(double playback_offset) {
|
||||
if (!v2_) invalid("Interrupt belongs to /v2/speak; use clear on /v1/speak");
|
||||
if (playback_offset < 0 || playback_offset > 36000) invalid("playback_offset must be 0..36000");
|
||||
Json m = typed("Interrupt");
|
||||
m.set("playback_offset", Json(playback_offset));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
void SpeakStream::end() { socket_->send_json(typed("Close")); }
|
||||
|
||||
SpeakStreamResult SpeakStream::collect(int max_wait_ms) {
|
||||
SpeakStreamResult out;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto ev = socket_->next(max_wait_ms);
|
||||
if (!ev) return out;
|
||||
if (ev->kind == SocketEvent::Kind::kClose) {
|
||||
out.closed = true;
|
||||
return out;
|
||||
}
|
||||
if (ev->kind == SocketEvent::Kind::kBinary) {
|
||||
if (out.audio.size() + ev->bytes.size() > AudioCaps::kMaxStreamAudioBytes) {
|
||||
out.truncated = true;
|
||||
return out;
|
||||
}
|
||||
out.audio += ev->bytes;
|
||||
++out.chunks;
|
||||
continue;
|
||||
}
|
||||
if (!ev->json_ok) continue;
|
||||
out.metadata.push_back(ev->json);
|
||||
if (ev->type == "Flushed") {
|
||||
out.flushed = true;
|
||||
return out;
|
||||
}
|
||||
if (ev->type == "Cleared") {
|
||||
out.cleared = true;
|
||||
return out;
|
||||
}
|
||||
if (ev->type == "Warning") out.warning = str_at(ev->json, "description");
|
||||
if (ev->type == "Error") {
|
||||
out.error = str_at(ev->json, "description");
|
||||
if (out.error.empty()) out.error = "stream error";
|
||||
return out;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---- agent ----------------------------------------------------------------
|
||||
|
||||
AgentSession::AgentSession(std::string api_key, std::string url, SocketOptions socket) {
|
||||
std::string full = url.empty() ? DeepgramEndpoints::agent_ws_base() + "/v1/agent/converse" : url;
|
||||
socket_ = std::make_unique<AudioSocket>("deepgram", full, DeepgramEndpoints::ws_headers(api_key),
|
||||
StreamChannels::agent(), std::move(socket));
|
||||
}
|
||||
|
||||
void AgentSession::send_settings(const Json& settings) {
|
||||
if (!settings.is_object()) invalid("settings must be a JSON object");
|
||||
Json m = settings;
|
||||
m.set("type", Json("Settings"));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void AgentSession::inject_user_message(const std::string& content) {
|
||||
if (content.empty() || content.size() > 20000) invalid("content must be 1..20000 characters");
|
||||
Json m = typed("InjectUserMessage");
|
||||
m.set("content", Json(content));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void AgentSession::inject_agent_message(const std::string& message) {
|
||||
if (message.empty() || message.size() > 20000) invalid("message must be 1..20000 characters");
|
||||
Json m = typed("InjectAgentMessage");
|
||||
m.set("message", Json(message));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void AgentSession::keep_alive() { socket_->send_json(typed("KeepAlive")); }
|
||||
|
||||
void AgentSession::send_function_result(const std::string& id, const std::string& name, const std::string& content) {
|
||||
if (!llm::Service::valid_id(id)) invalid("bad function call id");
|
||||
if (name.empty() || name.size() > 256) invalid("name must be 1..256 characters");
|
||||
if (content.size() > 200000) invalid("content is over 200000 characters");
|
||||
Json m = typed("FunctionCallResponse");
|
||||
m.set("id", Json(id));
|
||||
m.set("name", Json(name));
|
||||
m.set("content", Json(content));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
std::optional<AgentEvent> AgentSession::next(int wait_ms) {
|
||||
auto ev = socket_->next(wait_ms);
|
||||
if (!ev) return std::nullopt;
|
||||
AgentEvent out;
|
||||
if (ev->kind == SocketEvent::Kind::kBinary) {
|
||||
out.binary = true;
|
||||
out.audio = ev->bytes;
|
||||
out.type = "audio";
|
||||
} else {
|
||||
out.type = ev->type;
|
||||
out.json = ev->json;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
Vendored
+153
@@ -0,0 +1,153 @@
|
||||
#pragma once
|
||||
|
||||
// Deepgram WebSockets (AsyncAPI https://github.com/deepgram/deepgram-api-specs
|
||||
// asyncapi.yml): live speech to text (/v1/listen and the Flux turn model
|
||||
// /v2/listen), live text to speech (/v1/speak and /v2/speak) and the Voice
|
||||
// Agent (wss://agent.deepgram.com/v1/agent/converse).
|
||||
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/deepgram/service.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::deepgram {
|
||||
|
||||
// Query parameters a channel accepts (copied from the AsyncAPI channel
|
||||
// bindings; check_spec.py compares them). Used to refuse typos before connecting.
|
||||
struct StreamChannels {
|
||||
static const std::vector<std::string>& listen_v1_params();
|
||||
static const std::vector<std::string>& listen_v2_params();
|
||||
static const std::vector<std::string>& speak_v1_params();
|
||||
static const std::vector<std::string>& speak_v2_params();
|
||||
static Vocabulary listen_v1();
|
||||
static Vocabulary listen_v2();
|
||||
static Vocabulary speak_v1();
|
||||
static Vocabulary speak_v2();
|
||||
static Vocabulary agent();
|
||||
};
|
||||
|
||||
struct StreamParams {
|
||||
// name -> value; each name must be one the channel accepts. Lists repeat the key.
|
||||
std::vector<std::pair<std::string, std::string>> pairs;
|
||||
StreamParams& add(const std::string& name, const std::string& value) {
|
||||
pairs.emplace_back(name, value);
|
||||
return *this;
|
||||
}
|
||||
};
|
||||
|
||||
struct ListenEvent {
|
||||
enum class Kind { kOther, kResults, kMetadata, kUtteranceEnd, kSpeechStarted, kTurn, kError, kClose };
|
||||
Kind kind = Kind::kOther;
|
||||
std::string transcript; // alternatives[0].transcript (v1) or transcript (v2 TurnInfo)
|
||||
bool is_final = false; // v1 is_final, v2 EndOfTurn
|
||||
bool speech_final = false;
|
||||
double confidence = 0;
|
||||
std::string turn_event; // v2: StartOfTurn Update EagerEndOfTurn TurnResumed EndOfTurn
|
||||
std::string request_id;
|
||||
Json raw;
|
||||
};
|
||||
|
||||
struct ListenResult {
|
||||
std::vector<std::string> finals; // final transcripts in order
|
||||
std::string transcript; // finals joined by a space
|
||||
std::string interim; // last interim
|
||||
std::size_t utterance_ends = 0;
|
||||
std::string request_id;
|
||||
double duration = 0;
|
||||
std::string error;
|
||||
bool closed = false;
|
||||
std::size_t events = 0;
|
||||
};
|
||||
|
||||
class ListenStream {
|
||||
public:
|
||||
ListenStream(std::string api_key, StreamParams params, bool flux = false, std::string url = "",
|
||||
SocketOptions socket = {});
|
||||
std::string url() const { return socket_->url(); }
|
||||
void connect() { socket_->connect(); }
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
void send_audio(const std::string& audio) { socket_->send_binary(audio); }
|
||||
void keep_alive(); // {"type":"KeepAlive"}
|
||||
void finalize(); // {"type":"Finalize"} (v1)
|
||||
void force_end_turn(); // {"type":"ForceEndTurn"} (v2)
|
||||
void close_stream(); // {"type":"CloseStream"}
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
static ListenEvent decode(const SocketEvent& ev);
|
||||
// Reads until Metadata (end), error, close, or silence of `max_wait_ms`.
|
||||
ListenResult collect(int max_wait_ms);
|
||||
|
||||
private:
|
||||
bool flux_;
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
struct SpeakStreamResult {
|
||||
std::string audio;
|
||||
std::size_t chunks = 0;
|
||||
bool flushed = false;
|
||||
bool cleared = false;
|
||||
bool truncated = false;
|
||||
std::vector<Json> metadata; // JSON control messages in order
|
||||
std::string warning;
|
||||
std::string error;
|
||||
bool closed = false;
|
||||
};
|
||||
|
||||
class SpeakStream {
|
||||
public:
|
||||
SpeakStream(std::string api_key, StreamParams params, bool v2 = false, std::string url = "",
|
||||
SocketOptions socket = {});
|
||||
std::string url() const { return socket_->url(); }
|
||||
void connect() { socket_->connect(); }
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
void speak(const std::string& text); // {"type":"Speak","text":...}, 1..2000 chars
|
||||
void flush(); // {"type":"Flush"}
|
||||
void clear(); // {"type":"Clear"} (v1)
|
||||
void interrupt(double playback_offset); // {"type":"Interrupt"} (v2)
|
||||
void end(); // {"type":"Close"}
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
// Reads until Flushed/Cleared, error, close, or silence.
|
||||
SpeakStreamResult collect(int max_wait_ms);
|
||||
|
||||
private:
|
||||
bool v2_;
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
// Voice Agent: settings are a free-form object (the schema is large and
|
||||
// provider owned); message names are enforced by the vocabulary.
|
||||
struct AgentEvent {
|
||||
std::string type;
|
||||
Json json;
|
||||
std::string audio; // binary frames carry agent audio
|
||||
bool binary = false;
|
||||
};
|
||||
|
||||
class AgentSession {
|
||||
public:
|
||||
AgentSession(std::string api_key, std::string url = "", SocketOptions socket = {});
|
||||
std::string url() const { return socket_->url(); }
|
||||
void connect() { socket_->connect(); }
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
void send_settings(const Json& settings); // {"type":"Settings",...}; Settings fields as the spec has them
|
||||
void send_audio(const std::string& audio) { socket_->send_binary(audio); }
|
||||
void inject_user_message(const std::string& content);
|
||||
void inject_agent_message(const std::string& message);
|
||||
void keep_alive();
|
||||
void send_function_result(const std::string& id, const std::string& name, const std::string& content);
|
||||
std::optional<AgentEvent> next(int wait_ms);
|
||||
|
||||
private:
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::deepgram
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
#include "vendors/audio/elevenlabs/account.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
using C = ElevenCalls;
|
||||
|
||||
Json Account::models() { return call(C::kModels); }
|
||||
Json Account::user() { return call(C::kUser); }
|
||||
Json Account::subscription() { return call(C::kSubscription); }
|
||||
|
||||
Json Account::history(const Json& q) {
|
||||
if (q.is_object()) {
|
||||
const Json* size = q.find("page_size");
|
||||
if (size && size->is_number() && (size->as_int() < 1 || size->as_int() > 1000))
|
||||
bad("page_size must be 1..1000");
|
||||
}
|
||||
return call(C::kHistory, {}, nullptr, query_from(C::kHistory, q));
|
||||
}
|
||||
|
||||
Json Account::history_item(const std::string& id) {
|
||||
check_id("history item", id);
|
||||
return call(C::kHistoryItem, {id});
|
||||
}
|
||||
|
||||
AudioBlob Account::history_audio(const std::string& id) {
|
||||
check_id("history item", id);
|
||||
llm::RawResponse raw = call_raw(C::kHistoryAudio, {id});
|
||||
return AudioBlob{raw.body, raw.content_type};
|
||||
}
|
||||
|
||||
Json Account::remove_history_item(const std::string& id, bool confirm) {
|
||||
check_id("history item", id);
|
||||
return call_destructive(C::kHistoryDelete, {id}, nullptr, Query(), confirm);
|
||||
}
|
||||
|
||||
Json Account::single_use_token(const std::string& type) {
|
||||
check_id("token type", type);
|
||||
return call(C::kSingleUseToken, {type});
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
#pragma once
|
||||
|
||||
// Account reads: generation history, models, user, subscription; plus
|
||||
// short-lived single-use tokens for browser WebSockets.
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
#include "vendors/audio/elevenlabs/tts.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
class Account : public ElevenService {
|
||||
public:
|
||||
using ElevenService::ElevenService;
|
||||
|
||||
Json models();
|
||||
Json user();
|
||||
Json subscription();
|
||||
// History of generated speech, one page (page_size <= 1000 per the OpenAPI text).
|
||||
Json history(const Json& query_object = Json());
|
||||
Json history_item(const std::string& history_item_id);
|
||||
AudioBlob history_audio(const std::string& history_item_id);
|
||||
Json remove_history_item(const std::string& history_item_id, bool confirm);
|
||||
// token_type is a path segment (for example realtime_scribe or tts_websocket);
|
||||
// the token is a credential, never log it.
|
||||
Json single_use_token(const std::string& token_type);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
#include "vendors/audio/elevenlabs/agents.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
using C = ElevenCalls;
|
||||
|
||||
Json Agents::list(const Json& q) { return call(C::kAgents, {}, nullptr, query_from(C::kAgents, q)); }
|
||||
|
||||
Json Agents::list_all(std::size_t max_pages, int page_size) {
|
||||
if (max_pages == 0 || max_pages > AudioCaps::kMaxPages) bad("max_pages must be 1.." + std::to_string(AudioCaps::kMaxPages));
|
||||
if (page_size < 1 || page_size > ElevenCaps::kMaxPageSize) bad("page_size must be 1..100");
|
||||
Json agents = Json::array();
|
||||
Json q = Json::object();
|
||||
q.set("page_size", Json(page_size));
|
||||
bool truncated = true;
|
||||
std::size_t pages = 0;
|
||||
for (; pages < max_pages;) {
|
||||
Json doc = list(q);
|
||||
++pages;
|
||||
if (const Json* a = doc.find("agents"); a && a->is_array())
|
||||
for (const Json& e : a->as_array()) {
|
||||
if (agents.as_array().size() >= AudioCaps::kMaxListItems) {
|
||||
Json out = Json::object();
|
||||
out.set("agents", agents);
|
||||
out.set("truncated", Json(true));
|
||||
out.set("pages", Json(static_cast<int>(pages)));
|
||||
return out;
|
||||
}
|
||||
agents.push(e);
|
||||
}
|
||||
const Json* more = doc.find("has_more");
|
||||
const Json* cursor = doc.find("next_cursor");
|
||||
if (!more || !more->is_bool() || !more->as_bool() || !cursor || !cursor->is_string() ||
|
||||
cursor->as_string().empty()) {
|
||||
truncated = false;
|
||||
break;
|
||||
}
|
||||
q.set("cursor", Json(cursor->as_string()));
|
||||
}
|
||||
Json out = Json::object();
|
||||
out.set("agents", agents);
|
||||
out.set("truncated", Json(truncated));
|
||||
out.set("pages", Json(static_cast<int>(pages)));
|
||||
return out;
|
||||
}
|
||||
|
||||
Json Agents::get(const std::string& id, const Json& q) {
|
||||
check_id("agent", id);
|
||||
return call(C::kAgentGet, {id}, nullptr, query_from(C::kAgentGet, q));
|
||||
}
|
||||
|
||||
Json Agents::create(const Json& body, bool confirm, bool enable_versioning) {
|
||||
check_body(table_.get(C::kAgentCreate), &body);
|
||||
require_confirm("creating an agent", confirm);
|
||||
Query q;
|
||||
if (enable_versioning) q.add_bool("enable_versioning", true);
|
||||
return call(C::kAgentCreate, {}, &body, q);
|
||||
}
|
||||
|
||||
Json Agents::patch(const std::string& id, const Json& body, bool confirm) {
|
||||
check_id("agent", id);
|
||||
check_body(table_.get(C::kAgentPatch), &body);
|
||||
require_confirm("changing an agent", confirm);
|
||||
return call(C::kAgentPatch, {id}, &body);
|
||||
}
|
||||
|
||||
Json Agents::remove(const std::string& id, bool confirm, bool force) {
|
||||
check_id("agent", id);
|
||||
Query q;
|
||||
if (force) q.add_bool("force", true);
|
||||
return call_destructive(C::kAgentDelete, {id}, nullptr, q, confirm);
|
||||
}
|
||||
|
||||
Json Agents::link(const std::string& id) {
|
||||
check_id("agent", id);
|
||||
return call(C::kAgentLink, {id});
|
||||
}
|
||||
|
||||
Json Agents::signed_url(const std::string& id, const Json& q) {
|
||||
Query query = query_from(C::kSignedUrl, q);
|
||||
check_id("agent", id);
|
||||
Query with_agent;
|
||||
with_agent.add("agent_id", id);
|
||||
for (const auto& [k, v] : query.pairs())
|
||||
if (k != "agent_id") with_agent.add(k, v);
|
||||
return call(C::kSignedUrl, {}, nullptr, with_agent);
|
||||
}
|
||||
|
||||
Json Agents::conversations(const Json& q) {
|
||||
return call(C::kConversations, {}, nullptr, query_from(C::kConversations, q));
|
||||
}
|
||||
|
||||
Json Agents::conversation(const std::string& id, const Json& q) {
|
||||
check_id("conversation", id);
|
||||
return call(C::kConversationGet, {id}, nullptr, query_from(C::kConversationGet, q));
|
||||
}
|
||||
|
||||
AudioBlob Agents::conversation_audio(const std::string& id) {
|
||||
check_id("conversation", id);
|
||||
llm::RawResponse raw = call_raw(C::kConversationAudio, {id});
|
||||
return AudioBlob{raw.body, raw.content_type};
|
||||
}
|
||||
|
||||
Json Agents::remove_conversation(const std::string& id, bool confirm) {
|
||||
check_id("conversation", id);
|
||||
return call_destructive(C::kConversationDelete, {id}, nullptr, Query(), confirm);
|
||||
}
|
||||
|
||||
Json Agents::llms() { return call(C::kLlmList); }
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
#pragma once
|
||||
|
||||
// Conversational AI agents (ElevenAgents): agent CRUD, signed URLs for the
|
||||
// conversation WebSocket, conversation history and audio, the LLM catalogue.
|
||||
// Writes (create, patch, delete) need confirm. Everything else under
|
||||
// /v1/convai (tools, knowledge base, phone numbers, tests, MCP servers, ...)
|
||||
// is in the call table and reached with call() / call_destructive().
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
#include "vendors/audio/elevenlabs/tts.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
class Agents : public ElevenService {
|
||||
public:
|
||||
using ElevenService::ElevenService;
|
||||
|
||||
Json list(const Json& query_object = Json());
|
||||
// Walks cursors up to max_pages; `truncated` is set when the cap stopped it.
|
||||
Json list_all(std::size_t max_pages, int page_size = 100);
|
||||
Json get(const std::string& agent_id, const Json& query_object = Json());
|
||||
Json create(const Json& body, bool confirm, bool enable_versioning = false);
|
||||
Json patch(const std::string& agent_id, const Json& body, bool confirm);
|
||||
Json remove(const std::string& agent_id, bool confirm, bool force = false);
|
||||
Json link(const std::string& agent_id);
|
||||
// Signed wss URL for a conversation with a private agent (valid 15 minutes
|
||||
// per the docs): the URL itself is a credential, never log it.
|
||||
Json signed_url(const std::string& agent_id, const Json& query_object = Json());
|
||||
|
||||
Json conversations(const Json& query_object = Json());
|
||||
Json conversation(const std::string& conversation_id, const Json& query_object = Json());
|
||||
AudioBlob conversation_audio(const std::string& conversation_id);
|
||||
Json remove_conversation(const std::string& conversation_id, bool confirm);
|
||||
Json llms();
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+344
@@ -0,0 +1,344 @@
|
||||
#include "vendors/audio/elevenlabs/calls.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
// One row per REST call. tools/check_spec.py parses the block between the
|
||||
// markers: {id, method, path, host, body, fields, required, query, response, source}.
|
||||
// CALLS-BEGIN
|
||||
static const CallSpec kRows[] = {
|
||||
{"audio_isolation", "POST", "/v1/audio-isolation", "api", "form", "audio,file_format,preview_b64", "audio", "", "bytes", "openapi:audio_isolation"},
|
||||
{"get_audio_isolation_history", "GET", "/v1/audio-isolation/history", "api", "-", "", "", "page_size,page,search", "json", "openapi:get_audio_isolation_history"},
|
||||
{"delete_audio_isolation_history_item", "DELETE", "/v1/audio-isolation/history/{history_item_id}", "api", "-", "", "", "", "none", "openapi:delete_audio_isolation_history_item"},
|
||||
{"audio_isolation_stream", "POST", "/v1/audio-isolation/stream", "api", "form", "audio,file_format", "audio", "", "bytes", "openapi:audio_isolation_stream"},
|
||||
{"list_chat_response_tests_route", "GET", "/v1/convai/agent-testing", "api", "-", "", "", "cursor,page_size,search,parent_folder_id,types,include_folders,sort_mode,sharing_mode,created_by_user_id", "json", "openapi:list_chat_response_tests_route"},
|
||||
{"agent_testing_bulk_move_route", "POST", "/v1/convai/agent-testing/bulk-move", "api", "json", "entity_ids,move_to", "entity_ids", "", "json", "openapi:agent_testing_bulk_move_route"},
|
||||
{"create_agent_response_test_route", "POST", "/v1/convai/agent-testing/create", "api", "json", "from_conversation_metadata,dynamic_variables,chat_history,conversation_initiation_source,environment,type,success_condition,success_examples,failure_examples,name,parent_folder_id,tool_call_parameters,check_any_tool_matches,success_conditions,simulation_scenario,simulation_max_turns,simulation_environment,tool_mock_config,tool_mock_overrides,evaluation_model,simulated_user_model", "", "", "json", "openapi:create_agent_response_test_route"},
|
||||
{"create_agent_test_folder_route", "POST", "/v1/convai/agent-testing/folders", "api", "json", "name,parent_folder_id", "name", "", "json", "openapi:create_agent_test_folder_route"},
|
||||
{"get_agent_test_folder_route", "GET", "/v1/convai/agent-testing/folders/{folder_id}", "api", "-", "", "", "", "json", "openapi:get_agent_test_folder_route"},
|
||||
{"update_agent_test_folder_route", "PATCH", "/v1/convai/agent-testing/folders/{folder_id}", "api", "json", "name", "name", "", "json", "openapi:update_agent_test_folder_route"},
|
||||
{"delete_agent_test_folder_route", "DELETE", "/v1/convai/agent-testing/folders/{folder_id}", "api", "-", "", "", "force", "none", "openapi:delete_agent_test_folder_route"},
|
||||
{"get_agent_response_tests_summaries_route", "POST", "/v1/convai/agent-testing/summaries", "api", "json", "test_ids", "test_ids", "", "json", "openapi:get_agent_response_tests_summaries_route"},
|
||||
{"get_agent_response_test_route", "GET", "/v1/convai/agent-testing/{test_id}", "api", "-", "", "", "", "json", "openapi:get_agent_response_test_route"},
|
||||
{"update_agent_response_test_route", "PUT", "/v1/convai/agent-testing/{test_id}", "api", "json", "from_conversation_metadata,dynamic_variables,chat_history,conversation_initiation_source,environment,type,success_condition,success_examples,failure_examples,name,parent_folder_id,tool_call_parameters,check_any_tool_matches,success_conditions,simulation_scenario,simulation_max_turns,simulation_environment,tool_mock_config,tool_mock_overrides,evaluation_model,simulated_user_model", "", "", "json", "openapi:update_agent_response_test_route"},
|
||||
{"delete_chat_response_test_route", "DELETE", "/v1/convai/agent-testing/{test_id}", "api", "-", "", "", "", "json", "openapi:delete_chat_response_test_route"},
|
||||
{"get_agent_knowledge_base_size", "GET", "/v1/convai/agent/{agent_id}/knowledge-base/size", "api", "-", "", "", "", "json", "openapi:get_agent_knowledge_base_size"},
|
||||
{"get_agent_llm_expected_cost_calculation", "POST", "/v1/convai/agent/{agent_id}/llm-usage/calculate", "api", "json", "prompt_length,number_of_pages,rag_enabled,service_tier", "", "", "json", "openapi:get_agent_llm_expected_cost_calculation"},
|
||||
{"get_agents_route", "GET", "/v1/convai/agents", "api", "-", "", "", "page_size,search,archived,show_only_owned_agents,created_by_user_id,tags,sort_direction,sort_by,cursor", "json", "openapi:get_agents_route"},
|
||||
{"create_agent_route", "POST", "/v1/convai/agents/create", "api", "json", "conversation_config,platform_settings,workflow,name,tags", "conversation_config", "enable_versioning", "json", "openapi:create_agent_route"},
|
||||
{"get_agent_summaries_route", "GET", "/v1/convai/agents/summaries", "api", "-", "", "", "agent_ids", "json", "openapi:get_agent_summaries_route"},
|
||||
{"get_agent_route", "GET", "/v1/convai/agents/{agent_id}", "api", "-", "", "", "version_id,branch_id,include_procedure_generated_workflow_elements", "json", "openapi:get_agent_route"},
|
||||
{"patch_agent_settings_route", "PATCH", "/v1/convai/agents/{agent_id}", "api", "json", "conversation_config,platform_settings,workflow,name,tags,version_description,procedures", "", "enable_versioning_if_not_enabled,branch_id,include_procedure_generated_workflow_elements", "json", "openapi:patch_agent_settings_route"},
|
||||
{"delete_agent_route", "DELETE", "/v1/convai/agents/{agent_id}", "api", "-", "", "", "force", "none", "openapi:delete_agent_route"},
|
||||
{"post_agent_avatar_route", "POST", "/v1/convai/agents/{agent_id}/avatar", "api", "form", "avatar_file", "avatar_file", "", "json", "openapi:post_agent_avatar_route"},
|
||||
{"get_branches_route", "GET", "/v1/convai/agents/{agent_id}/branches", "api", "-", "", "", "include_archived,limit,include_commit_status", "json", "openapi:get_branches_route"},
|
||||
{"create_branch_route", "POST", "/v1/convai/agents/{agent_id}/branches", "api", "json", "parent_version_id,name,description,conversation_config,platform_settings,workflow,include_draft", "parent_version_id,name,description", "", "json", "openapi:create_branch_route"},
|
||||
{"get_branch_route", "GET", "/v1/convai/agents/{agent_id}/branches/{branch_id}", "api", "-", "", "", "", "json", "openapi:get_branch_route"},
|
||||
{"update_branch_route", "PATCH", "/v1/convai/agents/{agent_id}/branches/{branch_id}", "api", "json", "name,description,is_archived,protection_status,auto_rebase", "", "", "json", "openapi:update_branch_route"},
|
||||
{"list_procedures_route", "GET", "/v1/convai/agents/{agent_id}/branches/{branch_id}/procedures", "api", "-", "", "", "agent_version_id", "json", "openapi:list_procedures_route"},
|
||||
{"create_procedure_route", "POST", "/v1/convai/agents/{agent_id}/branches/{branch_id}/procedures", "api", "json", "name,content,type,trigger,folder_parent_id", "", "", "json", "openapi:create_procedure_route"},
|
||||
{"compile_procedures_route", "POST", "/v1/convai/agents/{agent_id}/branches/{branch_id}/procedures/compile", "api", "-", "", "", "", "json", "openapi:compile_procedures_route"},
|
||||
{"get_procedure_route", "GET", "/v1/convai/agents/{agent_id}/branches/{branch_id}/procedures/{procedure_id}", "api", "-", "", "", "version_id,agent_version_id", "json", "openapi:get_procedure_route"},
|
||||
{"remove_procedure_route", "DELETE", "/v1/convai/agents/{agent_id}/branches/{branch_id}/procedures/{procedure_id}", "api", "-", "", "", "", "json", "openapi:remove_procedure_route"},
|
||||
{"get_procedure_draft_route", "GET", "/v1/convai/agents/{agent_id}/branches/{branch_id}/procedures/{procedure_id}/draft", "api", "-", "", "", "", "json", "openapi:get_procedure_draft_route"},
|
||||
{"update_procedure_draft_route", "PATCH", "/v1/convai/agents/{agent_id}/branches/{branch_id}/procedures/{procedure_id}/draft", "api", "json", "name,content,type,trigger", "name,content,type", "", "json", "openapi:update_procedure_draft_route"},
|
||||
{"delete_procedure_draft_route", "DELETE", "/v1/convai/agents/{agent_id}/branches/{branch_id}/procedures/{procedure_id}/draft", "api", "-", "", "", "", "json", "openapi:delete_procedure_draft_route"},
|
||||
{"rebase_branch_onto_main", "POST", "/v1/convai/agents/{agent_id}/branches/{branch_id}/rebase", "api", "json", "*", "", "source_branch_id", "json", "openapi:rebase_branch_onto_main"},
|
||||
{"rebase_preview_route", "GET", "/v1/convai/agents/{agent_id}/branches/{branch_id}/rebase-preview", "api", "-", "", "", "source_branch_id,include_procedure_generated_workflow_elements", "json", "openapi:rebase_preview_route"},
|
||||
{"merge_branch_into_target", "POST", "/v1/convai/agents/{agent_id}/branches/{source_branch_id}/merge", "api", "json", "archive_source_branch,force", "", "target_branch_id", "json", "openapi:merge_branch_into_target"},
|
||||
{"merge_preview_route", "GET", "/v1/convai/agents/{agent_id}/branches/{source_branch_id}/merge-preview", "api", "-", "", "", "target_branch_id,force,include_procedure_generated_workflow_elements", "json", "openapi:merge_preview_route"},
|
||||
{"list_agent_deployments_route", "GET", "/v1/convai/agents/{agent_id}/deployments", "api", "-", "", "", "page,page_size", "json", "openapi:list_agent_deployments_route"},
|
||||
{"create_agent_deployment_route", "POST", "/v1/convai/agents/{agent_id}/deployments", "api", "json", "deployment_request", "deployment_request", "", "json", "openapi:create_agent_deployment_route"},
|
||||
{"create_agent_draft_route", "POST", "/v1/convai/agents/{agent_id}/drafts", "api", "json", "conversation_config,platform_settings,workflow,name,tags", "conversation_config,platform_settings,workflow,name", "branch_id", "json", "openapi:create_agent_draft_route"},
|
||||
{"delete_agent_draft_route", "DELETE", "/v1/convai/agents/{agent_id}/drafts", "api", "-", "", "", "branch_id", "json", "openapi:delete_agent_draft_route"},
|
||||
{"duplicate_agent_route", "POST", "/v1/convai/agents/{agent_id}/duplicate", "api", "json", "name", "", "", "json", "openapi:duplicate_agent_route"},
|
||||
{"post_agent_hold_audio_route", "POST", "/v1/convai/agents/{agent_id}/hold-audio", "api", "form", "hold_audio_file", "hold_audio_file", "", "json", "openapi:post_agent_hold_audio_route"},
|
||||
{"delete_agent_hold_audio_route", "DELETE", "/v1/convai/agents/{agent_id}/hold-audio", "api", "-", "", "", "", "json", "openapi:delete_agent_hold_audio_route"},
|
||||
{"query_agent_knowledge_base_rag_route", "POST", "/v1/convai/agents/{agent_id}/knowledge-base/rag-query", "api", "json", "query,use_agent_defaults,max_documents_length,max_retrieved_rag_chunks_count", "query", "branch_id", "json", "openapi:query_agent_knowledge_base_rag_route"},
|
||||
{"get_agent_link_route", "GET", "/v1/convai/agents/{agent_id}/link", "api", "-", "", "", "", "json", "openapi:get_agent_link_route"},
|
||||
{"list_merge_proposals_route", "GET", "/v1/convai/agents/{agent_id}/merge-proposals", "api", "-", "", "", "status,source_branch_id,target_branch_id,search,page_size,cursor", "json", "openapi:list_merge_proposals_route"},
|
||||
{"create_merge_proposal_route", "POST", "/v1/convai/agents/{agent_id}/merge-proposals", "api", "json", "source_branch_id,target_branch_id,title,description,requested_reviewer_user_ids,triage_ticket_id", "source_branch_id,target_branch_id,title", "", "json", "openapi:create_merge_proposal_route"},
|
||||
{"get_merge_proposal_route", "GET", "/v1/convai/agents/{agent_id}/merge-proposals/{merge_proposal_id}", "api", "-", "", "", "", "json", "openapi:get_merge_proposal_route"},
|
||||
{"update_merge_proposal_route", "PATCH", "/v1/convai/agents/{agent_id}/merge-proposals/{merge_proposal_id}", "api", "json", "title,description,requested_reviewer_user_ids,close", "", "", "json", "openapi:update_merge_proposal_route"},
|
||||
{"add_merge_proposal_comment_route", "POST", "/v1/convai/agents/{agent_id}/merge-proposals/{merge_proposal_id}/comments", "api", "json", "body", "body", "", "json", "openapi:add_merge_proposal_comment_route"},
|
||||
{"accept_merge_proposal_route", "POST", "/v1/convai/agents/{agent_id}/merge-proposals/{merge_proposal_id}/merge", "api", "json", "archive_source_branch,force", "", "", "json", "openapi:accept_merge_proposal_route"},
|
||||
{"submit_merge_proposal_review_route", "POST", "/v1/convai/agents/{agent_id}/merge-proposals/{merge_proposal_id}/reviews", "api", "json", "state,comment", "state", "", "json", "openapi:submit_merge_proposal_review_route"},
|
||||
{"run_agent_test_suite_route", "POST", "/v1/convai/agents/{agent_id}/run-tests", "api", "json", "name,tests,agent_config_override,branch_id,repeat_count", "tests", "", "json", "openapi:run_agent_test_suite_route"},
|
||||
{"run_conversation_simulation_route", "POST", "/v1/convai/agents/{agent_id}/simulate-conversation", "api", "json", "simulation_specification,extra_evaluation_criteria,new_turns_limit", "simulation_specification", "", "json", "openapi:run_conversation_simulation_route"},
|
||||
{"run_conversation_simulation_route_stream", "POST", "/v1/convai/agents/{agent_id}/simulate-conversation/stream", "api", "json", "simulation_specification,extra_evaluation_criteria,new_turns_limit", "simulation_specification", "", "none", "openapi:run_conversation_simulation_route_stream"},
|
||||
{"get_agent_topics_route", "GET", "/v1/convai/agents/{agent_id}/topics", "api", "-", "", "", "page_size,sort_by,sort_direction,from_unix_secs,to_unix_secs,include_evaluation_criteria,cursor", "json", "openapi:get_agent_topics_route"},
|
||||
{"list_agent_conversation_tickets_route", "GET", "/v1/convai/agents/{agent_id}/triage-tickets", "api", "-", "", "", "page_size,conversation_id,status,sources,priorities,sort_by,owner_user_id,assignee_user_id,issue_type,label,merged_into_ticket_id,search,cursor", "json", "openapi:list_agent_conversation_tickets_route"},
|
||||
{"create_manual_agent_ticket_route", "POST", "/v1/convai/agents/{agent_id}/triage-tickets", "api", "json", "title,qa_comment,priority", "qa_comment", "", "json", "openapi:create_manual_agent_ticket_route"},
|
||||
{"get_assignable_users_route", "GET", "/v1/convai/agents/{agent_id}/triage-tickets/assignable-users", "api", "-", "", "", "", "json", "openapi:get_assignable_users_route"},
|
||||
{"get_version_metadata_route", "GET", "/v1/convai/agents/{agent_id}/versions/{version_id}", "api", "-", "", "", "", "json", "openapi:get_version_metadata_route"},
|
||||
{"get_agent_widget_route", "GET", "/v1/convai/agents/{agent_id}/widget", "api", "-", "", "", "conversation_signature", "json", "openapi:get_agent_widget_route"},
|
||||
{"get_live_count", "GET", "/v1/convai/analytics/live-count", "api", "-", "", "", "agent_id,agent_ids", "json", "openapi:get_live_count"},
|
||||
{"create_batch_call", "POST", "/v1/convai/batch-calling/submit", "api", "json", "call_name,agent_id,recipients,scheduled_time_unix,agent_phone_number_id,whatsapp_params,timezone,branch_id,environment,telephony_call_config,target_concurrency_limit", "call_name,agent_id,recipients", "", "json", "openapi:create_batch_call"},
|
||||
{"get_workspace_batch_calls", "GET", "/v1/convai/batch-calling/workspace", "api", "-", "", "", "limit,last_doc,agent_id", "json", "openapi:get_workspace_batch_calls"},
|
||||
{"get_batch_call", "GET", "/v1/convai/batch-calling/{batch_id}", "api", "-", "", "", "", "json", "openapi:get_batch_call"},
|
||||
{"delete_batch_call", "DELETE", "/v1/convai/batch-calling/{batch_id}", "api", "-", "", "", "", "none", "openapi:delete_batch_call"},
|
||||
{"cancel_batch_call", "POST", "/v1/convai/batch-calling/{batch_id}/cancel", "api", "-", "", "", "", "json", "openapi:cancel_batch_call"},
|
||||
{"export_batch_call", "GET", "/v1/convai/batch-calling/{batch_id}/export", "api", "-", "", "", "limit", "text", "openapi:export_batch_call"},
|
||||
{"retry_batch_call", "POST", "/v1/convai/batch-calling/{batch_id}/retry", "api", "-", "", "", "", "json", "openapi:retry_batch_call"},
|
||||
{"get_conversation_signed_link", "GET", "/v1/convai/conversation/get-signed-url", "api", "-", "", "", "agent_id,include_conversation_id,branch_id,version_id,environment,debug_events_request", "json", "openapi:get_conversation_signed_link"},
|
||||
{"get_signed_url_deprecated", "GET", "/v1/convai/conversation/get_signed_url", "api", "-", "", "", "agent_id,include_conversation_id,branch_id,version_id,environment,debug_events_request", "json", "openapi:get_signed_url_deprecated"},
|
||||
{"get_livekit_token", "GET", "/v1/convai/conversation/token", "api", "-", "", "", "agent_id,participant_name,branch_id,version_id,environment,debug_events_request", "json", "openapi:get_livekit_token"},
|
||||
{"get_conversation_histories_route", "GET", "/v1/convai/conversations", "api", "-", "", "", "cursor,agent_id,visited_agent_ids,visited_agent_branch_ids,triggered_procedure_ids,call_successful,call_start_before_unix,call_start_after_unix,call_duration_min_secs,call_duration_max_secs,rating_max,rating_min,has_feedback_comment,user_id,evaluation_params,data_collection_params,dynamic_variable_params,data_collection_ids,evaluation_criteria_ids,tool_names,tool_names_successful,tool_names_errored,include_invalid_tool_calls,main_languages,page_size,summary_mode,search,conversation_initiation_source,text_only,conversation_product_type,branch_id,version_id,parent_conversation_id,topic_ids,exclude_statuses,tag_ids,workflow_node_entered_id,termination_reasons,guardrail_types,custom_guardrail_names,sort_direction", "json", "openapi:get_conversation_histories_route"},
|
||||
{"smart_search_conversation_messages_route", "GET", "/v1/convai/conversations/messages/smart-search", "api", "-", "", "", "text_query,agent_id,branch_id,page_size,cursor", "json", "openapi:smart_search_conversation_messages_route"},
|
||||
{"text_search_conversation_messages_route", "GET", "/v1/convai/conversations/messages/text-search", "api", "-", "", "", "text_query,agent_id,visited_agent_ids,visited_agent_branch_ids,triggered_procedure_ids,call_successful,call_start_before_unix,call_start_after_unix,call_duration_min_secs,call_duration_max_secs,rating_max,rating_min,has_feedback_comment,user_id,evaluation_params,data_collection_params,dynamic_variable_params,tool_names,tool_names_successful,tool_names_errored,include_invalid_tool_calls,main_languages,exclude_statuses,termination_reasons,page_size,summary_mode,conversation_initiation_source,text_only,conversation_product_type,branch_id,version_id,topic_ids,sort_by,cursor", "json", "openapi:text_search_conversation_messages_route"},
|
||||
{"resolve_conversation_reference_route", "GET", "/v1/convai/conversations/resolve", "api", "-", "", "", "agent_id,reference", "json", "openapi:resolve_conversation_reference_route"},
|
||||
{"get_conversation_history_route", "GET", "/v1/convai/conversations/{conversation_id}", "api", "-", "", "", "format", "json", "openapi:get_conversation_history_route"},
|
||||
{"delete_conversation_route", "DELETE", "/v1/convai/conversations/{conversation_id}", "api", "-", "", "", "", "json", "openapi:delete_conversation_route"},
|
||||
{"run_conversation_evaluations", "POST", "/v1/convai/conversations/{conversation_id}/analysis/evaluations/run", "api", "json", "evaluation_id,scope", "evaluation_id", "", "json", "openapi:run_conversation_evaluations"},
|
||||
{"run_conversation_analysis", "POST", "/v1/convai/conversations/{conversation_id}/analysis/run", "api", "-", "", "", "", "json", "openapi:run_conversation_analysis"},
|
||||
{"get_conversation_audio_route", "GET", "/v1/convai/conversations/{conversation_id}/audio", "api", "-", "", "", "", "bytes", "openapi:get_conversation_audio_route"},
|
||||
{"post_conversation_feedback_route", "POST", "/v1/convai/conversations/{conversation_id}/feedback", "api", "json", "feedback", "", "", "json", "openapi:post_conversation_feedback_route"},
|
||||
{"upload_file_route", "POST", "/v1/convai/conversations/{conversation_id}/files", "api", "form", "file", "file", "", "json", "openapi:upload_file_route"},
|
||||
{"cancel_file_upload_route", "DELETE", "/v1/convai/conversations/{conversation_id}/files/{file_id}", "api", "-", "", "", "", "json", "openapi:cancel_file_upload_route"},
|
||||
{"get_conversation_sip_messages", "GET", "/v1/convai/conversations/{conversation_id}/sip-messages", "api", "-", "", "", "page_size,cursor", "json", "openapi:get_conversation_sip_messages"},
|
||||
{"get_conversation_summary_route", "GET", "/v1/convai/conversations/{conversation_id}/summary", "api", "-", "", "", "max_messages", "json", "openapi:get_conversation_summary_route"},
|
||||
{"assign_conversation_tags_route", "POST", "/v1/convai/conversations/{conversation_id}/tags", "api", "json", "tag_ids", "tag_ids", "", "none", "openapi:assign_conversation_tags_route"},
|
||||
{"unassign_conversation_tag_route", "DELETE", "/v1/convai/conversations/{conversation_id}/tags/{tag_id}", "api", "-", "", "", "", "none", "openapi:unassign_conversation_tag_route"},
|
||||
{"list_environment_variables", "GET", "/v1/convai/environment-variables", "api", "-", "", "", "cursor,page_size,label,environment,type", "json", "openapi:list_environment_variables"},
|
||||
{"create_environment_variable", "POST", "/v1/convai/environment-variables", "api", "json", "type,label,values", "", "", "json", "openapi:create_environment_variable"},
|
||||
{"get_environment_variable", "GET", "/v1/convai/environment-variables/{env_var_id}", "api", "-", "", "", "", "json", "openapi:get_environment_variable"},
|
||||
{"update_environment_variable", "PATCH", "/v1/convai/environment-variables/{env_var_id}", "api", "json", "values", "values", "", "json", "openapi:update_environment_variable"},
|
||||
{"list_environments", "GET", "/v1/convai/environments", "api", "-", "", "", "", "json", "openapi:list_environments"},
|
||||
{"create_environment", "POST", "/v1/convai/environments", "api", "json", "name", "name", "", "json", "openapi:create_environment"},
|
||||
{"delete_environment", "DELETE", "/v1/convai/environments/{environment}", "api", "-", "", "", "", "none", "openapi:delete_environment"},
|
||||
{"handle_exotel_outbound_call", "POST", "/v1/convai/exotel/outbound-call", "api", "json", "agent_id,agent_phone_number_id,to_number,conversation_initiation_client_data,telephony_call_config", "agent_id,agent_phone_number_id,to_number", "", "json", "openapi:handle_exotel_outbound_call"},
|
||||
{"get_knowledge_base_list_route", "GET", "/v1/convai/knowledge-base", "api", "-", "", "", "page_size,search,show_only_owned_documents,created_by_user_id,types,parent_folder_id,ancestor_folder_id,folders_first,sort_direction,sort_by,cursor", "json", "openapi:get_knowledge_base_list_route"},
|
||||
{"add_documentation_to_knowledge_base", "POST", "/v1/convai/knowledge-base", "api", "form", "name,url,file", "", "agent_id", "json", "openapi:add_documentation_to_knowledge_base"},
|
||||
{"post_knowledge_base_bulk_delete_route", "POST", "/v1/convai/knowledge-base/bulk-delete", "api", "json", "document_ids,force", "document_ids", "", "json", "openapi:post_knowledge_base_bulk_delete_route"},
|
||||
{"post_knowledge_base_bulk_move_route", "POST", "/v1/convai/knowledge-base/bulk-move", "api", "json", "document_ids,move_to", "document_ids", "", "none", "openapi:post_knowledge_base_bulk_move_route"},
|
||||
{"list_crawl_jobs_route", "GET", "/v1/convai/knowledge-base/crawl", "api", "-", "", "", "include_job_ids,page_size,cursor", "json", "openapi:list_crawl_jobs_route"},
|
||||
{"create_crawl_job_route", "POST", "/v1/convai/knowledge-base/crawl", "api", "json", "url,max_depth,max_pages,pattern,exclude_patterns,sitemap_urls,parent_folder_id,enable_auto_sync,auto_remove,auto_discover,minimum_frequency_days", "url", "", "json", "openapi:create_crawl_job_route"},
|
||||
{"get_crawl_job_route", "GET", "/v1/convai/knowledge-base/crawl/{crawl_job_id}", "api", "-", "", "", "", "json", "openapi:get_crawl_job_route"},
|
||||
{"cancel_crawl_job_route", "POST", "/v1/convai/knowledge-base/crawl/{crawl_job_id}/cancel", "api", "-", "", "", "", "json", "openapi:cancel_crawl_job_route"},
|
||||
{"get_knowledge_base_bulk_dependent_agents_route", "POST", "/v1/convai/knowledge-base/dependent-agents", "api", "json", "document_ids", "document_ids", "dependent_type,page_size,cursor", "json", "openapi:get_knowledge_base_bulk_dependent_agents_route"},
|
||||
{"create_file_document_route", "POST", "/v1/convai/knowledge-base/file", "api", "form", "file,name,parent_folder_id", "file", "", "json", "openapi:create_file_document_route"},
|
||||
{"create_folder_route", "POST", "/v1/convai/knowledge-base/folder", "api", "json", "name,parent_folder_id,enable_auto_sync,auto_remove,minimum_frequency_days", "name", "", "json", "openapi:create_folder_route"},
|
||||
{"get_rag_index_overview", "GET", "/v1/convai/knowledge-base/rag-index", "api", "-", "", "", "", "json", "openapi:get_rag_index_overview"},
|
||||
{"get_or_create_rag_indexes", "POST", "/v1/convai/knowledge-base/rag-index", "api", "json", "items", "items", "", "json", "openapi:get_or_create_rag_indexes"},
|
||||
{"search_knowledge_base_content_route", "GET", "/v1/convai/knowledge-base/search", "api", "-", "", "", "query,page_size,types,cursor", "json", "openapi:search_knowledge_base_content_route"},
|
||||
{"get_agent_knowledge_base_summaries_route", "GET", "/v1/convai/knowledge-base/summaries", "api", "-", "", "", "document_ids", "json", "openapi:get_agent_knowledge_base_summaries_route"},
|
||||
{"create_text_document_route", "POST", "/v1/convai/knowledge-base/text", "api", "json", "text,name,parent_folder_id", "text", "", "json", "openapi:create_text_document_route"},
|
||||
{"create_url_document_route", "POST", "/v1/convai/knowledge-base/url", "api", "json", "url,name,parent_folder_id,enable_auto_sync,auto_remove,minimum_frequency_days", "url", "", "json", "openapi:create_url_document_route"},
|
||||
{"post_knowledge_base_move_route", "POST", "/v1/convai/knowledge-base/{document_id}/move", "api", "json", "move_to", "", "", "none", "openapi:post_knowledge_base_move_route"},
|
||||
{"get_documentation_from_knowledge_base", "GET", "/v1/convai/knowledge-base/{documentation_id}", "api", "-", "", "", "agent_id", "json", "openapi:get_documentation_from_knowledge_base"},
|
||||
{"update_document_route", "PATCH", "/v1/convai/knowledge-base/{documentation_id}", "api", "json", "name,content", "", "", "json", "openapi:update_document_route"},
|
||||
{"delete_knowledge_base_document", "DELETE", "/v1/convai/knowledge-base/{documentation_id}", "api", "-", "", "", "force", "json", "openapi:delete_knowledge_base_document"},
|
||||
{"get_documentation_chunk_from_knowledge_base", "GET", "/v1/convai/knowledge-base/{documentation_id}/chunk/{chunk_id}", "api", "-", "", "", "embedding_model", "json", "openapi:get_documentation_chunk_from_knowledge_base"},
|
||||
{"get_documentation_chunks_from_knowledge_base", "GET", "/v1/convai/knowledge-base/{documentation_id}/chunks", "api", "-", "", "", "embedding_model,page_size,cursor", "json", "openapi:get_documentation_chunks_from_knowledge_base"},
|
||||
{"get_knowledge_base_content", "GET", "/v1/convai/knowledge-base/{documentation_id}/content", "api", "-", "", "", "", "text", "openapi:get_knowledge_base_content"},
|
||||
{"get_knowledge_base_dependent_agents", "GET", "/v1/convai/knowledge-base/{documentation_id}/dependent-agents", "api", "-", "", "", "dependent_type,page_size,cursor", "json", "openapi:get_knowledge_base_dependent_agents"},
|
||||
{"get_rag_indexes", "GET", "/v1/convai/knowledge-base/{documentation_id}/rag-index", "api", "-", "", "", "", "json", "openapi:get_rag_indexes"},
|
||||
{"rag_index_status", "POST", "/v1/convai/knowledge-base/{documentation_id}/rag-index", "api", "json", "model", "model", "", "json", "openapi:rag_index_status"},
|
||||
{"delete_rag_index", "DELETE", "/v1/convai/knowledge-base/{documentation_id}/rag-index/{rag_index_id}", "api", "-", "", "", "", "json", "openapi:delete_rag_index"},
|
||||
{"refresh_url_document_route", "POST", "/v1/convai/knowledge-base/{documentation_id}/refresh", "api", "-", "", "", "", "json", "openapi:refresh_url_document_route"},
|
||||
{"get_knowledge_base_source_file_url", "GET", "/v1/convai/knowledge-base/{documentation_id}/source-file-url", "api", "-", "", "", "", "json", "openapi:get_knowledge_base_source_file_url"},
|
||||
{"update_file_document_route", "PATCH", "/v1/convai/knowledge-base/{documentation_id}/update-file", "api", "form", "file", "file", "", "json", "openapi:update_file_document_route"},
|
||||
{"get_public_llm_expected_cost_calculation", "POST", "/v1/convai/llm-usage/calculate", "api", "json", "prompt_length,number_of_pages,rag_enabled,service_tier", "prompt_length,number_of_pages,rag_enabled", "", "json", "openapi:get_public_llm_expected_cost_calculation"},
|
||||
{"list_available_llms", "GET", "/v1/convai/llm/list", "api", "-", "", "", "", "json", "openapi:list_available_llms"},
|
||||
{"list_mcp_servers_route", "GET", "/v1/convai/mcp-servers", "api", "-", "", "", "", "json", "openapi:list_mcp_servers_route"},
|
||||
{"create_mcp_server_route", "POST", "/v1/convai/mcp-servers", "api", "json", "config", "config", "", "json", "openapi:create_mcp_server_route"},
|
||||
{"get_mcp_route", "GET", "/v1/convai/mcp-servers/{mcp_server_id}", "api", "-", "", "", "", "json", "openapi:get_mcp_route"},
|
||||
{"update_mcp_server_config_route", "PATCH", "/v1/convai/mcp-servers/{mcp_server_id}", "api", "json", "approval_policy,force_pre_tool_speech,pre_tool_speech,disable_interruptions,interruption_mode,tool_call_sound,tool_call_sound_behavior,execution_mode,response_timeout_secs,request_headers,request_meta,disable_compression,secret_token,auth_connection", "", "", "json", "openapi:update_mcp_server_config_route"},
|
||||
{"delete_mcp_server_route", "DELETE", "/v1/convai/mcp-servers/{mcp_server_id}", "api", "-", "", "", "", "json", "openapi:delete_mcp_server_route"},
|
||||
{"update_mcp_server_approval_policy_route", "PATCH", "/v1/convai/mcp-servers/{mcp_server_id}/approval-policy", "api", "json", "approval_policy", "approval_policy", "", "json", "openapi:update_mcp_server_approval_policy_route"},
|
||||
{"add_mcp_server_tool_approval_route", "POST", "/v1/convai/mcp-servers/{mcp_server_id}/tool-approvals", "api", "json", "tool_name,tool_description,input_schema,approval_policy", "tool_name,tool_description", "", "json", "openapi:add_mcp_server_tool_approval_route"},
|
||||
{"remove_mcp_server_tool_approval_route", "DELETE", "/v1/convai/mcp-servers/{mcp_server_id}/tool-approvals/{tool_name}", "api", "-", "", "", "", "json", "openapi:remove_mcp_server_tool_approval_route"},
|
||||
{"add_mcp_tool_config_override_route", "POST", "/v1/convai/mcp-servers/{mcp_server_id}/tool-configs", "api", "json", "force_pre_tool_speech,pre_tool_speech,disable_interruptions,interruption_mode,tool_call_sound,tool_call_sound_behavior,execution_mode,response_timeout_secs,assignments,input_overrides,response_mocks,tool_name", "tool_name", "environment", "json", "openapi:add_mcp_tool_config_override_route"},
|
||||
{"get_mcp_tool_config_override_route", "GET", "/v1/convai/mcp-servers/{mcp_server_id}/tool-configs/{tool_name}", "api", "-", "", "", "", "json", "openapi:get_mcp_tool_config_override_route"},
|
||||
{"update_mcp_tool_config_override_route", "PATCH", "/v1/convai/mcp-servers/{mcp_server_id}/tool-configs/{tool_name}", "api", "json", "force_pre_tool_speech,pre_tool_speech,disable_interruptions,interruption_mode,tool_call_sound,tool_call_sound_behavior,execution_mode,response_timeout_secs,assignments,input_overrides,response_mocks", "", "environment", "json", "openapi:update_mcp_tool_config_override_route"},
|
||||
{"remove_mcp_tool_config_override_route", "DELETE", "/v1/convai/mcp-servers/{mcp_server_id}/tool-configs/{tool_name}", "api", "-", "", "", "", "json", "openapi:remove_mcp_tool_config_override_route"},
|
||||
{"list_mcp_server_tools_route", "GET", "/v1/convai/mcp-servers/{mcp_server_id}/tools", "api", "-", "", "", "environment", "json", "openapi:list_mcp_server_tools_route"},
|
||||
{"list_phone_numbers_route", "GET", "/v1/convai/phone-numbers", "api", "-", "", "", "provider,agent_id,branch_id", "json", "openapi:list_phone_numbers_route"},
|
||||
{"create_phone_number_route", "POST", "/v1/convai/phone-numbers", "api", "json", "phone_number,label,supports_inbound,supports_outbound,provider,agent_id,sid,token,account_auth_token,region_config,enable_sms,account_sid,api_key,api_token,api_subdomain,app_id,applet_url,inbound_trunk_config,outbound_trunk_config", "", "", "json", "openapi:create_phone_number_route"},
|
||||
{"get_phone_number_route", "GET", "/v1/convai/phone-numbers/{phone_number_id}", "api", "-", "", "", "", "json", "openapi:get_phone_number_route"},
|
||||
{"update_phone_number_route", "PATCH", "/v1/convai/phone-numbers/{phone_number_id}", "api", "json", "agent_id,label,inbound_trunk_config,outbound_trunk_config,livekit_stack,store_sip_messages,environment,branch_id", "", "", "json", "openapi:update_phone_number_route"},
|
||||
{"delete_phone_number_route", "DELETE", "/v1/convai/phone-numbers/{phone_number_id}", "api", "-", "", "", "", "json", "openapi:delete_phone_number_route"},
|
||||
{"list_sip_messages", "GET", "/v1/convai/phone-numbers/{phone_number_id}/sip-messages", "api", "-", "", "", "page_size,cursor", "json", "openapi:list_sip_messages"},
|
||||
{"create_realtime_session", "POST", "/v1/convai/realtime/client_secrets", "api", "json", "session,expires_after", "", "", "json", "openapi:create_realtime_session"},
|
||||
{"get_secrets_route", "GET", "/v1/convai/secrets", "api", "-", "", "", "page_size,dependency_limit,search,cursor", "json", "openapi:get_secrets_route"},
|
||||
{"create_secret_route", "POST", "/v1/convai/secrets", "api", "json", "type,name,value", "type,name,value", "", "json", "openapi:create_secret_route"},
|
||||
{"get_secret_route", "GET", "/v1/convai/secrets/{secret_id}", "api", "-", "", "", "", "json", "openapi:get_secret_route"},
|
||||
{"update_secret_route", "PATCH", "/v1/convai/secrets/{secret_id}", "api", "json", "type,name,value", "type,name,value", "", "json", "openapi:update_secret_route"},
|
||||
{"delete_secret_route", "DELETE", "/v1/convai/secrets/{secret_id}", "api", "-", "", "", "", "none", "openapi:delete_secret_route"},
|
||||
{"get_secret_dependencies_route", "GET", "/v1/convai/secrets/{secret_id}/dependencies/{resource_type}", "api", "-", "", "", "page_size,cursor", "json", "openapi:get_secret_dependencies_route"},
|
||||
{"get_settings_route", "GET", "/v1/convai/settings", "api", "-", "", "", "", "json", "openapi:get_settings_route"},
|
||||
{"update_settings_route", "PATCH", "/v1/convai/settings", "api", "json", "conversation_initiation_client_data_webhook,webhooks,can_use_mcp_servers,rag_retention_period_days,conversation_embedding_retention_days,default_livekit_stack", "", "", "json", "openapi:update_settings_route"},
|
||||
{"get_dashboard_settings_route", "GET", "/v1/convai/settings/dashboard", "api", "-", "", "", "", "json", "openapi:get_dashboard_settings_route"},
|
||||
{"update_dashboard_settings_route", "PATCH", "/v1/convai/settings/dashboard", "api", "json", "charts", "", "", "json", "openapi:update_dashboard_settings_route"},
|
||||
{"handle_sip_trunk_outbound_call", "POST", "/v1/convai/sip-trunk/outbound-call", "api", "json", "agent_id,agent_phone_number_id,to_number,conversation_initiation_client_data,telephony_call_config", "agent_id,agent_phone_number_id,to_number", "", "json", "openapi:handle_sip_trunk_outbound_call"},
|
||||
{"list_conversation_tags_route", "GET", "/v1/convai/tags", "api", "-", "", "", "page_size,cursor", "json", "openapi:list_conversation_tags_route"},
|
||||
{"create_conversation_tag_route", "POST", "/v1/convai/tags", "api", "json", "title,description", "title", "", "json", "openapi:create_conversation_tag_route"},
|
||||
{"get_conversation_tag_route", "GET", "/v1/convai/tags/{tag_id}", "api", "-", "", "", "", "json", "openapi:get_conversation_tag_route"},
|
||||
{"update_conversation_tag_route", "PATCH", "/v1/convai/tags/{tag_id}", "api", "json", "title,description", "", "", "json", "openapi:update_conversation_tag_route"},
|
||||
{"delete_conversation_tag_route", "DELETE", "/v1/convai/tags/{tag_id}", "api", "-", "", "", "", "none", "openapi:delete_conversation_tag_route"},
|
||||
{"list_test_invocations_route", "GET", "/v1/convai/test-invocations", "api", "-", "", "", "agent_id,branch_id,test_id,start_time,end_time,version_id,page_size,search,cursor", "json", "openapi:list_test_invocations_route"},
|
||||
{"get_test_invocation_route", "GET", "/v1/convai/test-invocations/{test_invocation_id}", "api", "-", "", "", "", "json", "openapi:get_test_invocation_route"},
|
||||
{"update_test_invocation_route", "PATCH", "/v1/convai/test-invocations/{test_invocation_id}", "api", "json", "name", "name", "", "json", "openapi:update_test_invocation_route"},
|
||||
{"cancel_test_invocation_route", "POST", "/v1/convai/test-invocations/{test_invocation_id}/cancel", "api", "-", "", "", "", "none", "openapi:cancel_test_invocation_route"},
|
||||
{"resubmit_tests_route", "POST", "/v1/convai/test-invocations/{test_invocation_id}/resubmit", "api", "json", "test_run_ids,agent_config_override,agent_id,branch_id", "test_run_ids,agent_id", "", "json", "openapi:resubmit_tests_route"},
|
||||
{"get_tools_route", "GET", "/v1/convai/tools", "api", "-", "", "", "search,page_size,show_only_owned_documents,created_by_user_id,types,sort_direction,sort_by,used_by_agent_id,cursor", "json", "openapi:get_tools_route"},
|
||||
{"add_tool_route", "POST", "/v1/convai/tools", "api", "json", "tool_config,response_mocks", "tool_config", "", "json", "openapi:add_tool_route"},
|
||||
{"get_tool_route", "GET", "/v1/convai/tools/{tool_id}", "api", "-", "", "", "environment", "json", "openapi:get_tool_route"},
|
||||
{"update_tool_route", "PATCH", "/v1/convai/tools/{tool_id}", "api", "json", "tool_config,response_mocks", "tool_config", "", "json", "openapi:update_tool_route"},
|
||||
{"delete_tool_route", "DELETE", "/v1/convai/tools/{tool_id}", "api", "-", "", "", "force", "json", "openapi:delete_tool_route"},
|
||||
{"get_tool_dependent_agents_route", "GET", "/v1/convai/tools/{tool_id}/dependent-agents", "api", "-", "", "", "cursor,page_size", "json", "openapi:get_tool_dependent_agents_route"},
|
||||
{"get_tool_executions_route", "GET", "/v1/convai/tools/{tool_id}/executions", "api", "-", "", "", "cursor,page_size,is_error,agent_id,branch_id,start_time,end_time", "json", "openapi:get_tool_executions_route"},
|
||||
{"list_workspace_conversation_tickets_route", "GET", "/v1/convai/triage-tickets", "api", "-", "", "", "page_size,agent_id,status,sources,priorities,sort_by,owner_user_id,assignee_user_id,issue_type,label,merged_into_ticket_id,search,cursor", "json", "openapi:list_workspace_conversation_tickets_route"},
|
||||
{"create_agent_conversation_ticket_route", "POST", "/v1/convai/triage-tickets", "api", "json", "conversation_id,title,qa_comment,turn_comments,priority", "conversation_id", "", "json", "openapi:create_agent_conversation_ticket_route"},
|
||||
{"get_agent_conversation_ticket_route", "GET", "/v1/convai/triage-tickets/{agentqa_ticket_id}", "api", "-", "", "", "", "json", "openapi:get_agent_conversation_ticket_route"},
|
||||
{"update_agent_conversation_ticket_route", "PATCH", "/v1/convai/triage-tickets/{agentqa_ticket_id}", "api", "json", "title,status,assignee_user_id,priority", "", "", "json", "openapi:update_agent_conversation_ticket_route"},
|
||||
{"delete_agent_conversation_ticket_route", "DELETE", "/v1/convai/triage-tickets/{agentqa_ticket_id}", "api", "-", "", "", "", "none", "openapi:delete_agent_conversation_ticket_route"},
|
||||
{"add_ticket_comment_route", "POST", "/v1/convai/triage-tickets/{agentqa_ticket_id}/comments", "api", "json", "comment", "comment", "", "json", "openapi:add_ticket_comment_route"},
|
||||
{"merge_agent_conversation_tickets_route", "POST", "/v1/convai/triage-tickets/{agentqa_ticket_id}/merge", "api", "json", "source_ticket_ids", "source_ticket_ids", "", "json", "openapi:merge_agent_conversation_tickets_route"},
|
||||
{"add_turn_comment_route", "POST", "/v1/convai/triage-tickets/{agentqa_ticket_id}/turn-comments", "api", "json", "turn_index,comment", "turn_index,comment", "", "json", "openapi:add_turn_comment_route"},
|
||||
{"handle_twilio_outbound_call", "POST", "/v1/convai/twilio/outbound-call", "api", "json", "agent_id,agent_phone_number_id,to_number,conversation_initiation_client_data,call_recording_enabled,telephony_call_config", "agent_id,agent_phone_number_id,to_number", "", "json", "openapi:handle_twilio_outbound_call"},
|
||||
{"register_twilio_call", "POST", "/v1/convai/twilio/register-call", "api", "json", "agent_id,from_number,to_number,direction,conversation_initiation_client_data", "agent_id,from_number,to_number", "", "text", "openapi:register_twilio_call"},
|
||||
{"get_conversation_users_route", "GET", "/v1/convai/users", "api", "-", "", "", "agent_id,branch_id,call_start_before_unix,call_start_after_unix,search,page_size,sort_by,sort_direction,cursor", "json", "openapi:get_conversation_users_route"},
|
||||
{"list_phone_numbers_page_route", "GET", "/v1/convai/v2/phone-numbers", "api", "-", "", "", "page_size,search,label,phone_number,provider,supports_outbound,agent_id,branch_id,sort_by,sort_direction,cursor", "json", "openapi:list_phone_numbers_page_route"},
|
||||
{"list_whatsapp_accounts", "GET", "/v1/convai/whatsapp-accounts", "api", "-", "", "", "agent_id", "json", "openapi:list_whatsapp_accounts"},
|
||||
{"get_whatsapp_account", "GET", "/v1/convai/whatsapp-accounts/{phone_number_id}", "api", "-", "", "", "", "json", "openapi:get_whatsapp_account"},
|
||||
{"update_whatsapp_account", "PATCH", "/v1/convai/whatsapp-accounts/{phone_number_id}", "api", "json", "assigned_agent_id,enable_messaging,enable_audio_message_response,enable_typing_indicator", "", "", "json", "openapi:update_whatsapp_account"},
|
||||
{"delete_whatsapp_account", "DELETE", "/v1/convai/whatsapp-accounts/{phone_number_id}", "api", "-", "", "", "", "json", "openapi:delete_whatsapp_account"},
|
||||
{"whatsapp_outbound_call", "POST", "/v1/convai/whatsapp/outbound-call", "api", "json", "whatsapp_phone_number_id,whatsapp_user_id,whatsapp_call_permission_request_template_name,whatsapp_call_permission_request_template_language_code,agent_id,conversation_initiation_client_data", "whatsapp_phone_number_id,whatsapp_user_id,whatsapp_call_permission_request_template_name,whatsapp_call_permission_request_template_language_code,agent_id", "", "json", "openapi:whatsapp_outbound_call"},
|
||||
{"whatsapp_outbound_message", "POST", "/v1/convai/whatsapp/outbound-message", "api", "json", "whatsapp_phone_number_id,whatsapp_user_id,template_name,template_language_code,template_params,agent_id,conversation_initiation_client_data", "whatsapp_phone_number_id,whatsapp_user_id,template_name,template_language_code,template_params,agent_id", "", "json", "openapi:whatsapp_outbound_message"},
|
||||
{"list_dubs", "GET", "/v1/dubbing", "api", "-", "", "", "cursor,page_size,dubbing_status,dubbing_statuses,dubbing_models,target_language_codes,creation_sources,filter_by_creator,order_by,order_direction", "json", "openapi:list_dubs"},
|
||||
{"create_dubbing", "POST", "/v1/dubbing", "api", "form", "file,csv_file,foreground_audio_file,background_audio_file,name,source_url,source_lang,target_lang,target_accent,num_speakers,watermark,start_time,end_time,highest_resolution,drop_background_audio,use_profanity_filter,dubbing_studio,disable_voice_cloning,mode,csv_fps", "", "", "json", "openapi:create_dubbing"},
|
||||
{"dubbing_project_list", "GET", "/v1/dubbing/project", "api", "-", "", "", "cursor,page_size,status,sort_direction", "json", "openapi:dubbing_project_list"},
|
||||
{"dubbing_project_create", "POST", "/v1/dubbing/project", "api", "form", "file,source_url,reference,source_language,model_id,keyterms,webhook_ids,target_language,transcript", "", "", "json", "openapi:dubbing_project_create"},
|
||||
{"dubbing_project_get", "GET", "/v1/dubbing/project/{project_id}", "api", "-", "", "", "", "json", "openapi:dubbing_project_get"},
|
||||
{"dubbing_project_delete", "DELETE", "/v1/dubbing/project/{project_id}", "api", "-", "", "", "", "none", "openapi:dubbing_project_delete"},
|
||||
{"dubbing_language_list", "GET", "/v1/dubbing/project/{project_id}/language", "api", "-", "", "", "cursor,page_size,status", "json", "openapi:dubbing_language_list"},
|
||||
{"dubbing_language_create", "POST", "/v1/dubbing/project/{project_id}/language", "api", "json", "target_language,voice_settings,translations", "target_language", "", "json", "openapi:dubbing_language_create"},
|
||||
{"dubbing_language_get", "GET", "/v1/dubbing/project/{project_id}/language/{language_id}", "api", "-", "", "", "", "json", "openapi:dubbing_language_get"},
|
||||
{"dubbing_language_delete", "DELETE", "/v1/dubbing/project/{project_id}/language/{language_id}", "api", "-", "", "", "", "none", "openapi:dubbing_language_delete"},
|
||||
{"dubbing_target_transcript_get", "GET", "/v1/dubbing/project/{project_id}/language/{language_id}/transcript", "api", "-", "", "", "", "json", "openapi:dubbing_target_transcript_get"},
|
||||
{"dubbing_target_transcript_regenerate", "POST", "/v1/dubbing/project/{project_id}/language/{language_id}/transcript/regenerate", "api", "-", "", "", "", "json", "openapi:dubbing_target_transcript_regenerate"},
|
||||
{"dubbing_target_transcript_segment_update", "PATCH", "/v1/dubbing/project/{project_id}/language/{language_id}/transcript/segment/{segment_id}", "api", "json", "translation", "", "", "json", "openapi:dubbing_target_transcript_segment_update"},
|
||||
{"dubbing_target_transcript_segments_update", "PATCH", "/v1/dubbing/project/{project_id}/language/{language_id}/transcript/segments", "api", "json", "segments", "segments", "", "json", "openapi:dubbing_target_transcript_segments_update"},
|
||||
{"dubbing_transcript_get", "GET", "/v1/dubbing/project/{project_id}/transcript", "api", "-", "", "", "", "json", "openapi:dubbing_transcript_get"},
|
||||
{"dubbing_transcript_segment_add", "POST", "/v1/dubbing/project/{project_id}/transcript/segment", "api", "json", "text,speaker_id,start_s,end_s", "text,speaker_id,start_s,end_s", "", "json", "openapi:dubbing_transcript_segment_add"},
|
||||
{"dubbing_transcript_segment_update", "PATCH", "/v1/dubbing/project/{project_id}/transcript/segment/{segment_id}", "api", "json", "text,speaker_id,start_s,end_s", "", "", "json", "openapi:dubbing_transcript_segment_update"},
|
||||
{"dubbing_transcript_segment_delete", "DELETE", "/v1/dubbing/project/{project_id}/transcript/segment/{segment_id}", "api", "-", "", "", "", "json", "openapi:dubbing_transcript_segment_delete"},
|
||||
{"dubbing_transcript_segments_update", "PATCH", "/v1/dubbing/project/{project_id}/transcript/segments", "api", "json", "segments", "segments", "", "json", "openapi:dubbing_transcript_segments_update"},
|
||||
{"get_dubbing_resource", "GET", "/v1/dubbing/resource/{dubbing_id}", "api", "-", "", "", "", "json", "openapi:get_dubbing_resource"},
|
||||
{"dub", "POST", "/v1/dubbing/resource/{dubbing_id}/dub", "api", "json", "segments,languages", "segments,languages", "", "json", "openapi:dub"},
|
||||
{"add_language", "POST", "/v1/dubbing/resource/{dubbing_id}/language", "api", "json", "language", "language", "", "json", "openapi:add_language"},
|
||||
{"migrate_segments", "POST", "/v1/dubbing/resource/{dubbing_id}/migrate-segments", "api", "json", "segment_ids,speaker_id", "segment_ids,speaker_id", "", "json", "openapi:migrate_segments"},
|
||||
{"render", "POST", "/v1/dubbing/resource/{dubbing_id}/render/{language}", "api", "json", "render_type,normalize_volume", "render_type", "", "json", "openapi:render"},
|
||||
{"delete_segment", "DELETE", "/v1/dubbing/resource/{dubbing_id}/segment/{segment_id}", "api", "-", "", "", "", "json", "openapi:delete_segment"},
|
||||
{"update_segment_language", "PATCH", "/v1/dubbing/resource/{dubbing_id}/segment/{segment_id}/{language}", "api", "json", "start_time,end_time,text", "", "", "json", "openapi:update_segment_language"},
|
||||
{"create_speaker", "POST", "/v1/dubbing/resource/{dubbing_id}/speaker", "api", "json", "speaker_name,voice_id,voice_stability,voice_similarity,voice_style", "", "", "json", "openapi:create_speaker"},
|
||||
{"update_speaker", "PATCH", "/v1/dubbing/resource/{dubbing_id}/speaker/{speaker_id}", "api", "json", "speaker_name,voice_id,voice_stability,voice_similarity,voice_style,languages", "", "", "json", "openapi:update_speaker"},
|
||||
{"create_clip", "POST", "/v1/dubbing/resource/{dubbing_id}/speaker/{speaker_id}/segment", "api", "json", "start_time,end_time,text,translations", "start_time,end_time", "", "json", "openapi:create_clip"},
|
||||
{"get_similar_voices_for_speaker", "GET", "/v1/dubbing/resource/{dubbing_id}/speaker/{speaker_id}/similar-voices", "api", "-", "", "", "", "json", "openapi:get_similar_voices_for_speaker"},
|
||||
{"transcribe", "POST", "/v1/dubbing/resource/{dubbing_id}/transcribe", "api", "json", "segments", "segments", "", "json", "openapi:transcribe"},
|
||||
{"translate", "POST", "/v1/dubbing/resource/{dubbing_id}/translate", "api", "json", "segments,languages", "segments,languages", "", "json", "openapi:translate"},
|
||||
{"get_dubbed_metadata", "GET", "/v1/dubbing/{dubbing_id}", "api", "-", "", "", "", "json", "openapi:get_dubbed_metadata"},
|
||||
{"delete_dubbing", "DELETE", "/v1/dubbing/{dubbing_id}", "api", "-", "", "", "", "json", "openapi:delete_dubbing"},
|
||||
{"get_dubbed_file", "GET", "/v1/dubbing/{dubbing_id}/audio/{language_code}", "api", "-", "", "", "", "bytes", "openapi:get_dubbed_file"},
|
||||
{"get_dubbed_transcript_file", "GET", "/v1/dubbing/{dubbing_id}/transcript/{language_code}", "api", "-", "", "", "format_type", "json", "openapi:get_dubbed_transcript_file"},
|
||||
{"get_dubbing_transcripts", "GET", "/v1/dubbing/{dubbing_id}/transcripts/{language_code}/format/{format_type}", "api", "-", "", "", "", "json", "openapi:get_dubbing_transcripts"},
|
||||
{"forced_alignment", "POST", "/v1/forced-alignment", "api", "form", "file,text", "file,text", "", "json", "openapi:forced_alignment"},
|
||||
{"get_speech_history", "GET", "/v1/history", "api", "-", "", "", "page_size,start_after_history_item_id,voice_id,model_id,date_before_unix,date_after_unix,sort_direction,search,source", "json", "openapi:get_speech_history"},
|
||||
{"download_speech_history_items", "POST", "/v1/history/download", "api", "json", "history_item_ids,output_format", "history_item_ids", "", "bytes", "openapi:download_speech_history_items"},
|
||||
{"get_speech_history_item_by_id", "GET", "/v1/history/{history_item_id}", "api", "-", "", "", "", "json", "openapi:get_speech_history_item_by_id"},
|
||||
{"delete_speech_history_item", "DELETE", "/v1/history/{history_item_id}", "api", "-", "", "", "", "json", "openapi:delete_speech_history_item"},
|
||||
{"get_audio_full_from_speech_history_item", "GET", "/v1/history/{history_item_id}/audio", "api", "-", "", "", "", "bytes", "openapi:get_audio_full_from_speech_history_item"},
|
||||
{"get_models", "GET", "/v1/models", "api", "-", "", "", "", "json", "openapi:get_models"},
|
||||
{"music_generate", "POST", "/v1/music", "api", "json", "prompt,generation_mode,music_prompt,lyrics_text,composition_plan,music_length_ms,model_id,seed,force_instrumental,finetune_id,finetune_strength,use_phonetic_names,respect_sections_durations,store_for_inpainting,sign_with_c2pa", "", "output_format,enable_logging", "bytes", "openapi:generate"},
|
||||
{"compose_detailed", "POST", "/v1/music/detailed", "api", "json", "prompt,generation_mode,music_prompt,lyrics_text,composition_plan,music_length_ms,model_id,seed,force_instrumental,finetune_id,finetune_strength,use_phonetic_names,respect_sections_durations,store_for_inpainting,with_timestamps,with_waveform_visual,sign_with_c2pa,model_style_prefix", "", "output_format,enable_logging", "json", "openapi:compose_detailed"},
|
||||
{"compose_detailed_stream", "POST", "/v1/music/detailed/stream", "api", "json", "prompt,generation_mode,music_prompt,lyrics_text,composition_plan,music_length_ms,model_id,seed,force_instrumental,finetune_id,finetune_strength,use_phonetic_names,store_for_inpainting,with_timestamps,with_waveform_visual", "", "output_format,enable_logging", "bytes", "openapi:compose_detailed_stream"},
|
||||
{"get_finetunes", "GET", "/v1/music/finetunes", "api", "-", "", "", "cursor,page_size,visibility,created_by,sort,sort_direction", "json", "openapi:get_finetunes"},
|
||||
{"create_finetune", "POST", "/v1/music/finetunes", "api", "form", "name,primary_genre,files,tags,visibility,model_id", "name,primary_genre", "", "json", "openapi:create_finetune"},
|
||||
{"get_finetune", "GET", "/v1/music/finetunes/{finetune_id}", "api", "-", "", "", "", "json", "openapi:get_finetune"},
|
||||
{"update_finetune", "PATCH", "/v1/music/finetunes/{finetune_id}", "api", "json", "name,tags,primary_genre,visibility", "", "", "json", "openapi:update_finetune"},
|
||||
{"delete_finetune", "DELETE", "/v1/music/finetunes/{finetune_id}", "api", "-", "", "", "", "json", "openapi:delete_finetune"},
|
||||
{"compose_plan", "POST", "/v1/music/plan", "api", "json", "prompt,music_length_ms,source_composition_plan,model_id", "prompt", "enable_logging", "json", "openapi:compose_plan"},
|
||||
{"separate_song_stems", "POST", "/v1/music/stem-separation", "api", "form", "file,stem_variation_id,sign_with_c2pa", "file", "output_format,enable_logging", "bytes", "openapi:separate_song_stems"},
|
||||
{"stream_compose", "POST", "/v1/music/stream", "api", "json", "prompt,generation_mode,music_prompt,lyrics_text,composition_plan,music_length_ms,model_id,seed,force_instrumental,finetune_id,finetune_strength,use_phonetic_names,store_for_inpainting", "", "output_format,enable_logging", "bytes", "openapi:stream_compose"},
|
||||
{"upload_song", "POST", "/v1/music/upload", "api", "form", "file,extract_composition_plan,with_timestamps,with_waveform_visual", "file", "", "json", "openapi:upload_song"},
|
||||
{"video_to_music", "POST", "/v1/music/video-to-music", "api", "form", "videos,description,tags,model_id,add_sfx,separate_tracks,use_audio_as_reference,sign_with_c2pa", "videos", "output_format,enable_logging", "bytes", "openapi:video_to_music"},
|
||||
{"get_pronunciation_dictionaries_metadata", "GET", "/v1/pronunciation-dictionaries", "api", "-", "", "", "cursor,page_size,sort,sort_direction,include_archived", "json", "openapi:get_pronunciation_dictionaries_metadata"},
|
||||
{"add_from_file", "POST", "/v1/pronunciation-dictionaries/add-from-file", "api", "form", "name,file,description,workspace_access", "name", "", "json", "openapi:add_from_file"},
|
||||
{"add_from_rules", "POST", "/v1/pronunciation-dictionaries/add-from-rules", "api", "json", "rules,name,description,workspace_access", "rules,name", "", "json", "openapi:add_from_rules"},
|
||||
{"get_pronunciation_dictionary_version_pls", "GET", "/v1/pronunciation-dictionaries/{dictionary_id}/{version_id}/download", "api", "-", "", "", "", "text", "openapi:get_pronunciation_dictionary_version_pls"},
|
||||
{"get_pronunciation_dictionary_metadata", "GET", "/v1/pronunciation-dictionaries/{pronunciation_dictionary_id}", "api", "-", "", "", "", "json", "openapi:get_pronunciation_dictionary_metadata"},
|
||||
{"patch_pronunciation_dictionary", "PATCH", "/v1/pronunciation-dictionaries/{pronunciation_dictionary_id}", "api", "json", "archived,name", "", "", "json", "openapi:patch_pronunciation_dictionary"},
|
||||
{"add_rules", "POST", "/v1/pronunciation-dictionaries/{pronunciation_dictionary_id}/add-rules", "api", "json", "rules", "rules", "", "json", "openapi:add_rules"},
|
||||
{"remove_rules", "POST", "/v1/pronunciation-dictionaries/{pronunciation_dictionary_id}/remove-rules", "api", "json", "rule_strings", "rule_strings", "", "json", "openapi:remove_rules"},
|
||||
{"set_rules", "POST", "/v1/pronunciation-dictionaries/{pronunciation_dictionary_id}/set-rules", "api", "json", "rules", "rules", "", "json", "openapi:set_rules"},
|
||||
{"get_library_voices", "GET", "/v1/shared-voices", "api", "-", "", "", "page_size,category,gender,age,accent,language,locale,search,use_cases,descriptives,featured,min_notice_period_days,include_custom_rates,include_live_moderated,reader_app_enabled,owner_id,sort,page", "json", "openapi:get_library_voices"},
|
||||
{"get_similar_library_voices", "POST", "/v1/similar-voices", "api", "form", "audio_file,similarity_threshold,top_k", "", "", "json", "openapi:get_similar_library_voices"},
|
||||
{"get_single_use_token", "POST", "/v1/single-use-token/{token_type}", "api", "-", "", "", "", "json", "openapi:get_single_use_token"},
|
||||
{"sound_generation", "POST", "/v1/sound-generation", "api", "json", "text,loop,duration_seconds,prompt_influence,model_id", "text", "output_format,enable_logging", "bytes", "openapi:sound_generation"},
|
||||
{"speech_to_text", "POST", "/v1/speech-to-text", "api", "form", "model_id,file,language_code,transcript_edit,tag_audio_events,num_speakers,timestamps_granularity,diarize,diarization_threshold,additional_formats,file_format,cloud_storage_url,source_url,webhook,webhook_id,temperature,seed,use_multi_channel,multichannel_output_style,webhook_metadata,entity_detection,no_verbatim,use_speaker_library,detect_speaker_roles,entity_redaction,entity_redaction_mode,keyterms", "model_id", "token,enable_logging", "json", "openapi:speech_to_text"},
|
||||
{"get_transcript_by_id", "GET", "/v1/speech-to-text/transcripts/{transcription_id}", "api", "-", "", "", "", "json", "openapi:get_transcript_by_id"},
|
||||
{"delete_transcript_by_id", "DELETE", "/v1/speech-to-text/transcripts/{transcription_id}", "api", "-", "", "", "", "json", "openapi:delete_transcript_by_id"},
|
||||
{"text_to_dialogue", "POST", "/v1/text-to-dialogue", "api", "json", "inputs,model_id,language_code,settings,previous_text,future_text,pronunciation_dictionary_locators,seed,apply_text_normalization,use_pvc_as_ivc,previous_request_ids,next_request_ids,audio_effects", "inputs", "output_format,enable_logging", "bytes", "openapi:text_to_dialogue"},
|
||||
{"text_to_dialogue_stream", "POST", "/v1/text-to-dialogue/stream", "api", "json", "inputs,model_id,language_code,settings,previous_text,future_text,pronunciation_dictionary_locators,seed,apply_text_normalization,use_pvc_as_ivc,previous_request_ids,next_request_ids,audio_effects", "inputs", "output_format,enable_logging", "bytes", "openapi:text_to_dialogue_stream"},
|
||||
{"text_to_dialogue_stream_with_timestamps", "POST", "/v1/text-to-dialogue/stream/with-timestamps", "api", "json", "inputs,model_id,language_code,settings,pronunciation_dictionary_locators,seed,apply_text_normalization,use_pvc_as_ivc,previous_request_ids,next_request_ids,previous_text,future_text,audio_effects", "inputs", "output_format,enable_logging", "bytes", "openapi:text_to_dialogue_stream_with_timestamps"},
|
||||
{"text_to_dialogue_full_with_timestamps", "POST", "/v1/text-to-dialogue/with-timestamps", "api", "json", "inputs,model_id,language_code,settings,pronunciation_dictionary_locators,seed,apply_text_normalization,use_pvc_as_ivc,previous_request_ids,next_request_ids,previous_text,future_text,audio_effects", "inputs", "output_format,enable_logging", "json", "openapi:text_to_dialogue_full_with_timestamps"},
|
||||
{"text_to_speech_full", "POST", "/v1/text-to-speech/{voice_id}", "api", "json", "text,model_id,language_code,voice_settings,pronunciation_dictionary_locators,seed,previous_text,next_text,previous_request_ids,next_request_ids,use_pvc_as_ivc,apply_text_normalization,apply_language_text_normalization,audio_effects", "text", "enable_logging,optimize_streaming_latency,output_format", "bytes", "openapi:text_to_speech_full"},
|
||||
{"text_to_speech_stream", "POST", "/v1/text-to-speech/{voice_id}/stream", "api", "json", "text,model_id,language_code,voice_settings,pronunciation_dictionary_locators,seed,previous_text,next_text,previous_request_ids,next_request_ids,use_pvc_as_ivc,apply_text_normalization,apply_language_text_normalization,audio_effects", "text", "enable_logging,optimize_streaming_latency,output_format", "bytes", "openapi:text_to_speech_stream"},
|
||||
{"text_to_speech_stream_with_timestamps", "POST", "/v1/text-to-speech/{voice_id}/stream/with-timestamps", "api", "json", "text,model_id,language_code,voice_settings,pronunciation_dictionary_locators,seed,previous_text,next_text,previous_request_ids,next_request_ids,use_pvc_as_ivc,apply_text_normalization,apply_language_text_normalization,audio_effects", "text", "enable_logging,optimize_streaming_latency,output_format", "bytes", "openapi:text_to_speech_stream_with_timestamps"},
|
||||
{"text_to_speech_full_with_timestamps", "POST", "/v1/text-to-speech/{voice_id}/with-timestamps", "api", "json", "text,model_id,language_code,voice_settings,pronunciation_dictionary_locators,seed,previous_text,next_text,previous_request_ids,next_request_ids,use_pvc_as_ivc,apply_text_normalization,apply_language_text_normalization,audio_effects", "text", "enable_logging,optimize_streaming_latency,output_format", "json", "openapi:text_to_speech_full_with_timestamps"},
|
||||
{"create_voice", "POST", "/v1/text-to-voice", "api", "json", "voice_name,voice_description,generated_voice_id,labels,played_not_selected_voice_ids", "voice_name,voice_description,generated_voice_id", "", "json", "openapi:create_voice"},
|
||||
{"text_to_voice", "POST", "/v1/text-to-voice/create-previews", "api", "json", "voice_description,text,auto_generate_text,loudness,quality,seed,guidance_scale,should_enhance", "voice_description", "output_format", "json", "openapi:text_to_voice"},
|
||||
{"text_to_voice_design", "POST", "/v1/text-to-voice/design", "api", "json", "voice_description,model_id,text,auto_generate_text,loudness,seed,guidance_scale,stream_previews,should_enhance,remixing_session_id,remixing_session_iteration_id,quality,reference_audio_base64,prompt_strength", "voice_description", "output_format", "json", "openapi:text_to_voice_design"},
|
||||
{"text_to_voice_preview_stream", "GET", "/v1/text-to-voice/{generated_voice_id}/stream", "api", "-", "", "", "", "bytes", "openapi:text_to_voice_preview_stream"},
|
||||
{"text_to_voice_remix", "POST", "/v1/text-to-voice/{voice_id}/remix", "api", "json", "voice_description,text,auto_generate_text,loudness,seed,guidance_scale,stream_previews,remixing_session_id,remixing_session_iteration_id,prompt_strength", "voice_description", "output_format", "json", "openapi:text_to_voice_remix"},
|
||||
{"usage_characters", "GET", "/v1/usage/character-stats", "api", "-", "", "", "start_unix,end_unix,include_workspace_metrics,breakdown_type,aggregation_interval,aggregation_bucket_size,metric", "json", "openapi:usage_characters"},
|
||||
{"get_user_info", "GET", "/v1/user", "api", "-", "", "", "", "json", "openapi:get_user_info"},
|
||||
{"get_user_subscription_info", "GET", "/v1/user/subscription", "api", "-", "", "", "", "json", "openapi:get_user_subscription_info"},
|
||||
{"speech_to_speech_full", "POST", "/v1/voice-changer/{voice_id}", "api", "form", "audio,model_id,voice_settings,seed,remove_background_noise,file_format", "audio", "enable_logging,optimize_streaming_latency,output_format", "bytes", "openapi:speech_to_speech_full"},
|
||||
{"speech_to_speech_stream", "POST", "/v1/voice-changer/{voice_id}/stream", "api", "form", "audio,model_id,voice_settings,seed,remove_background_noise,file_format", "audio", "enable_logging,optimize_streaming_latency,output_format", "bytes", "openapi:speech_to_speech_stream"},
|
||||
{"get_voices", "GET", "/v1/voices", "api", "-", "", "", "show_legacy", "json", "openapi:get_voices"},
|
||||
{"get_voice_accents", "GET", "/v1/voices/accents", "api", "-", "", "", "language,model_id", "json", "openapi:get_voice_accents"},
|
||||
{"add_voice", "POST", "/v1/voices/add", "api", "form", "name,files,remove_background_noise,description,labels", "name,files", "", "json", "openapi:add_voice"},
|
||||
{"add_sharing_voice", "POST", "/v1/voices/add/{public_user_id}/{voice_id}", "api", "json", "new_name,bookmarked", "new_name", "", "json", "openapi:add_sharing_voice"},
|
||||
{"get_user_voice_collections", "GET", "/v1/voices/collections", "api", "-", "", "", "", "json", "openapi:get_user_voice_collections"},
|
||||
{"create_or_update_user_voice_collection", "POST", "/v1/voices/collections", "api", "json", "title,icon", "title,icon", "", "json", "openapi:create_or_update_user_voice_collection"},
|
||||
{"get_user_voice_collection", "GET", "/v1/voices/collections/{collection_id}", "api", "-", "", "", "", "json", "openapi:get_user_voice_collection"},
|
||||
{"update_user_voice_collection", "PATCH", "/v1/voices/collections/{collection_id}", "api", "json", "title,icon", "", "", "json", "openapi:update_user_voice_collection"},
|
||||
{"delete_user_voice_collection", "DELETE", "/v1/voices/collections/{collection_id}", "api", "-", "", "", "", "json", "openapi:delete_user_voice_collection"},
|
||||
{"add_voice_to_user_voice_collection", "POST", "/v1/voices/collections/{collection_id}/voice/{voice_id}", "api", "-", "", "", "", "json", "openapi:add_voice_to_user_voice_collection"},
|
||||
{"remove_voice_from_user_voice_collection", "DELETE", "/v1/voices/collections/{collection_id}/voice/{voice_id}", "api", "-", "", "", "", "json", "openapi:remove_voice_from_user_voice_collection"},
|
||||
{"create_pvc_voice", "POST", "/v1/voices/pvc", "api", "json", "name,language,description,labels", "name,language", "", "json", "openapi:create_pvc_voice"},
|
||||
{"edit_pvc_voice", "POST", "/v1/voices/pvc/{voice_id}", "api", "json", "name,language,description,labels", "", "", "json", "openapi:edit_pvc_voice"},
|
||||
{"get_pvc_voice_captcha", "GET", "/v1/voices/pvc/{voice_id}/captcha", "api", "-", "", "", "", "none", "openapi:get_pvc_voice_captcha"},
|
||||
{"verify_pvc_voice_captcha", "POST", "/v1/voices/pvc/{voice_id}/captcha", "api", "form", "recording", "recording", "", "json", "openapi:verify_pvc_voice_captcha"},
|
||||
{"add_pvc_voice_samples", "POST", "/v1/voices/pvc/{voice_id}/samples", "api", "form", "files,remove_background_noise", "files", "", "json", "openapi:add_pvc_voice_samples"},
|
||||
{"edit_pvc_voice_sample", "POST", "/v1/voices/pvc/{voice_id}/samples/{sample_id}", "api", "json", "remove_background_noise,selected_speaker_ids,trim_start_time,trim_end_time,file_name", "", "", "json", "openapi:edit_pvc_voice_sample"},
|
||||
{"delete_pvc_voice_sample", "DELETE", "/v1/voices/pvc/{voice_id}/samples/{sample_id}", "api", "-", "", "", "", "json", "openapi:delete_pvc_voice_sample"},
|
||||
{"get_pvc_sample_audio", "GET", "/v1/voices/pvc/{voice_id}/samples/{sample_id}/audio", "api", "-", "", "", "remove_background_noise", "json", "openapi:get_pvc_sample_audio"},
|
||||
{"start_speaker_separation", "POST", "/v1/voices/pvc/{voice_id}/samples/{sample_id}/separate-speakers", "api", "-", "", "", "", "json", "openapi:start_speaker_separation"},
|
||||
{"get_pvc_sample_speakers", "GET", "/v1/voices/pvc/{voice_id}/samples/{sample_id}/speakers", "api", "-", "", "", "", "json", "openapi:get_pvc_sample_speakers"},
|
||||
{"get_speaker_audio", "GET", "/v1/voices/pvc/{voice_id}/samples/{sample_id}/speakers/{speaker_id}/audio", "api", "-", "", "", "", "json", "openapi:get_speaker_audio"},
|
||||
{"get_pvc_sample_visual_waveform", "GET", "/v1/voices/pvc/{voice_id}/samples/{sample_id}/waveform", "api", "-", "", "", "", "json", "openapi:get_pvc_sample_visual_waveform"},
|
||||
{"run_pvc_voice_training", "POST", "/v1/voices/pvc/{voice_id}/train", "api", "json", "model_id", "", "", "json", "openapi:run_pvc_voice_training"},
|
||||
{"request_pvc_manual_verification", "POST", "/v1/voices/pvc/{voice_id}/verification", "api", "form", "files,extra_text", "files", "", "json", "openapi:request_pvc_manual_verification"},
|
||||
{"get_voice_settings_default", "GET", "/v1/voices/settings/default", "api", "-", "", "", "", "json", "openapi:get_voice_settings_default"},
|
||||
{"get_voice_by_id", "GET", "/v1/voices/{voice_id}", "api", "-", "", "", "with_settings", "json", "openapi:get_voice_by_id"},
|
||||
{"delete_voice", "DELETE", "/v1/voices/{voice_id}", "api", "-", "", "", "", "json", "openapi:delete_voice"},
|
||||
{"edit_voice", "POST", "/v1/voices/{voice_id}/edit", "api", "form", "name,files,remove_background_noise,description,labels,moderate_metadata", "name", "", "json", "openapi:edit_voice"},
|
||||
{"replicate_voice_to_isolated_environment", "POST", "/v1/voices/{voice_id}/replicate-to-isolated-environment", "api", "json", "target_workspace_id,preserve_voice_id", "target_workspace_id", "", "json", "openapi:replicate_voice_to_isolated_environment"},
|
||||
{"delete_sample", "DELETE", "/v1/voices/{voice_id}/samples/{sample_id}", "api", "-", "", "", "", "json", "openapi:delete_sample"},
|
||||
{"get_audio_from_sample", "GET", "/v1/voices/{voice_id}/samples/{sample_id}/audio", "api", "-", "", "", "", "bytes", "openapi:get_audio_from_sample"},
|
||||
{"get_voice_settings", "GET", "/v1/voices/{voice_id}/settings", "api", "-", "", "", "", "json", "openapi:get_voice_settings"},
|
||||
{"edit_voice_settings", "POST", "/v1/voices/{voice_id}/settings/edit", "api", "json", "stability,use_speaker_boost,similarity_boost,style,speed,clone_strength", "", "", "json", "openapi:edit_voice_settings"},
|
||||
{"get_user_voices_v2", "GET", "/v2/voices", "api", "-", "", "", "next_page_token,page_size,search,sort,sort_direction,voice_type,category,fine_tuning_state,collection_id,gender,age,language,accent,use_cases,min_notice_period_days,include_custom_rates,include_live_moderated,high_quality,include_total_count,voice_ids", "json", "openapi:get_user_voices_v2"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
static_assert(sizeof(kRows) / sizeof(kRows[0]) == 325, "elevenlabs call table size");
|
||||
|
||||
const CallTable& calls() {
|
||||
static const CallTable table("elevenlabs", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
Vendored
+72
@@ -0,0 +1,72 @@
|
||||
#pragma once
|
||||
|
||||
// The ElevenLabs call table (https://api.elevenlabs.io/openapi.json, one row
|
||||
// per operation in the product areas this package covers). Services refer to
|
||||
// rows by the ids in `ElevenCalls`; nothing builds a path from a literal.
|
||||
|
||||
#include "vendors/audio/call_spec.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
const CallTable& calls();
|
||||
|
||||
struct ElevenCalls {
|
||||
// Speech
|
||||
static constexpr const char* kTts = "text_to_speech_full";
|
||||
static constexpr const char* kTtsStream = "text_to_speech_stream";
|
||||
static constexpr const char* kTtsTimestamps = "text_to_speech_full_with_timestamps";
|
||||
static constexpr const char* kTtsStreamTimestamps = "text_to_speech_stream_with_timestamps";
|
||||
static constexpr const char* kStt = "speech_to_text";
|
||||
static constexpr const char* kSttGet = "get_transcript_by_id";
|
||||
static constexpr const char* kSttDelete = "delete_transcript_by_id";
|
||||
static constexpr const char* kDialogue = "text_to_dialogue";
|
||||
static constexpr const char* kVoiceChanger = "speech_to_speech_full";
|
||||
static constexpr const char* kIsolation = "audio_isolation";
|
||||
static constexpr const char* kAlignment = "forced_alignment";
|
||||
// Voices
|
||||
static constexpr const char* kVoices = "get_voices";
|
||||
static constexpr const char* kVoicesV2 = "get_user_voices_v2";
|
||||
static constexpr const char* kVoiceGet = "get_voice_by_id";
|
||||
static constexpr const char* kVoiceSettings = "get_voice_settings";
|
||||
static constexpr const char* kVoiceSettingsDefault = "get_voice_settings_default";
|
||||
static constexpr const char* kVoiceDelete = "delete_voice";
|
||||
static constexpr const char* kVoiceAdd = "add_voice";
|
||||
static constexpr const char* kSharedVoices = "get_library_voices";
|
||||
static constexpr const char* kVoiceDesign = "text_to_voice_design";
|
||||
static constexpr const char* kVoiceDesignSave = "create_voice";
|
||||
static constexpr const char* kVoiceDesignPreview = "text_to_voice_preview_stream";
|
||||
// Creative
|
||||
static constexpr const char* kSfx = "sound_generation";
|
||||
static constexpr const char* kMusic = "music_generate";
|
||||
static constexpr const char* kMusicPlan = "compose_plan";
|
||||
static constexpr const char* kMusicStream = "stream_compose";
|
||||
// Dubbing
|
||||
static constexpr const char* kDubCreate = "dubbing_project_create";
|
||||
static constexpr const char* kDubList = "dubbing_project_list";
|
||||
static constexpr const char* kDubGet = "dubbing_project_get";
|
||||
static constexpr const char* kDubDelete = "dubbing_project_delete";
|
||||
// Agents (conversational AI)
|
||||
static constexpr const char* kAgents = "get_agents_route";
|
||||
static constexpr const char* kAgentGet = "get_agent_route";
|
||||
static constexpr const char* kAgentCreate = "create_agent_route";
|
||||
static constexpr const char* kAgentPatch = "patch_agent_settings_route";
|
||||
static constexpr const char* kAgentDelete = "delete_agent_route";
|
||||
static constexpr const char* kAgentLink = "get_agent_link_route";
|
||||
static constexpr const char* kSignedUrl = "get_conversation_signed_link";
|
||||
static constexpr const char* kConversations = "get_conversation_histories_route";
|
||||
static constexpr const char* kConversationGet = "get_conversation_history_route";
|
||||
static constexpr const char* kConversationDelete = "delete_conversation_route";
|
||||
static constexpr const char* kConversationAudio = "get_conversation_audio_route";
|
||||
static constexpr const char* kLlmList = "list_available_llms";
|
||||
// Account
|
||||
static constexpr const char* kHistory = "get_speech_history";
|
||||
static constexpr const char* kHistoryItem = "get_speech_history_item_by_id";
|
||||
static constexpr const char* kHistoryAudio = "get_audio_full_from_speech_history_item";
|
||||
static constexpr const char* kHistoryDelete = "delete_speech_history_item";
|
||||
static constexpr const char* kModels = "get_models";
|
||||
static constexpr const char* kUser = "get_user_info";
|
||||
static constexpr const char* kSubscription = "get_user_subscription_info";
|
||||
static constexpr const char* kSingleUseToken = "get_single_use_token";
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
#include "vendors/audio/elevenlabs/client.h"
|
||||
|
||||
#include "llm/config.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
std::string ElevenEndpoints::base_url() {
|
||||
return allowlisted_base(kBaseEnv, kDefaultBase,
|
||||
{"https://api.elevenlabs.io", "https://api.us.elevenlabs.io",
|
||||
"https://api.eu.residency.elevenlabs.io",
|
||||
"https://api.in.residency.elevenlabs.io",
|
||||
"https://api.sg.residency.elevenlabs.io"});
|
||||
}
|
||||
|
||||
llm::ApiClientConfig ElevenEndpoints::config(const std::string& base) {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = kVendor;
|
||||
c.base_url = base;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = kKeyHeader;
|
||||
return c;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> ElevenEndpoints::from_env() {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", HostFactory::from_env(config(base_url())));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> ElevenEndpoints::with(AudioHost host) {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", std::move(host));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::string ElevenEndpoints::key() { return llm::Config::get(kKeyEnv); }
|
||||
|
||||
AudioSocket::Headers ElevenEndpoints::ws_headers(const std::string& key) {
|
||||
return {{kKeyHeader, key}};
|
||||
}
|
||||
|
||||
std::string ElevenEndpoints::ws_base() { return to_ws_url(base_url()); }
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
#pragma once
|
||||
|
||||
// ElevenLabs endpoints and credentials.
|
||||
// REST https://api.elevenlabs.io header xi-api-key: $ELEVENLABS_API_KEY
|
||||
// WebSocket wss://api.elevenlabs.io header xi-api-key
|
||||
// Regional hosts (data residency) are chosen with ELEVENLABS_BASE_URL and only
|
||||
// the hosts in the OpenAPI/AsyncAPI server lists are accepted.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/hosts.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct ElevenEndpoints {
|
||||
static constexpr const char* kVendor = "elevenlabs";
|
||||
static constexpr const char* kKeyEnv = "ELEVENLABS_API_KEY";
|
||||
static constexpr const char* kBaseEnv = "ELEVENLABS_BASE_URL";
|
||||
static constexpr const char* kDefaultBase = "https://api.elevenlabs.io";
|
||||
static constexpr const char* kKeyHeader = "xi-api-key";
|
||||
|
||||
static std::string base_url(); // allowlisted override or the default
|
||||
static llm::ApiClientConfig config(const std::string& base = kDefaultBase);
|
||||
// Real HTTPS transports, key from llm::Config.
|
||||
static std::shared_ptr<const Hosts> from_env();
|
||||
// Injected host (tests): one host called "api".
|
||||
static std::shared_ptr<const Hosts> with(AudioHost host);
|
||||
static std::string key(); // "" when unset
|
||||
static AudioSocket::Headers ws_headers(const std::string& key);
|
||||
static std::string ws_base(); // wss://... of base_url()
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+171
@@ -0,0 +1,171 @@
|
||||
#include "vendors/audio/elevenlabs/convai_session.h"
|
||||
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("elevenlabs", why); }
|
||||
|
||||
std::string str_at(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
|
||||
const Json* obj_at(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_object() ? v : nullptr;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Vocabulary ConvaiSession::vocabulary() {
|
||||
// AsyncAPI channel /v1/convai/conversation. Audio uploads have no type field.
|
||||
return Vocabulary(
|
||||
"type",
|
||||
{"conversation_initiation_client_data", "pong", "user_message", "user_activity",
|
||||
"contextual_update", "client_tool_result", "mcp_tool_approval_result", "feedback",
|
||||
"multimodal_message", "file_input"},
|
||||
{"conversation_initiation_metadata", "user_transcript", "agent_response",
|
||||
"agent_response_correction", "agent_response_metadata", "agent_response_complete",
|
||||
"agent_chat_response_part", "audio", "interruption", "ping", "client_tool_call",
|
||||
"agent_tool_request", "agent_tool_response", "agent_tool_response_full_payload", "vad_score",
|
||||
"context_usage", "mcp_tool_call", "mcp_connection_status", "queue_status",
|
||||
"guardrail_triggered", "client_error"});
|
||||
}
|
||||
|
||||
ConvaiSession::ConvaiSession(std::string api_key, ConvaiOptions options) : options_(std::move(options)) {
|
||||
std::string url;
|
||||
AudioSocket::Headers headers;
|
||||
if (!options_.signed_url.empty()) {
|
||||
if (options_.signed_url.rfind("wss://", 0) != 0 && options_.signed_url.rfind("ws://", 0) != 0)
|
||||
invalid("signed_url must be a ws or wss URL");
|
||||
url = options_.signed_url; // signature stays inside the socket, never reported
|
||||
} else {
|
||||
if (!llm::Service::valid_id(options_.agent_id)) invalid("bad agent id");
|
||||
std::string base = options_.url.empty() ? ElevenEndpoints::ws_base() + "/v1/convai/conversation" : options_.url;
|
||||
url = AudioSocket::with_query(base, "agent_id=" + Query::encode(options_.agent_id));
|
||||
if (!api_key.empty()) headers = ElevenEndpoints::ws_headers(api_key);
|
||||
}
|
||||
socket_ = std::make_unique<AudioSocket>("elevenlabs", url, headers, vocabulary(), options_.socket);
|
||||
}
|
||||
|
||||
std::string ConvaiSession::url() const {
|
||||
if (!options_.signed_url.empty()) {
|
||||
std::string u = options_.signed_url;
|
||||
std::size_t q = u.find('?');
|
||||
return q == std::string::npos ? u : u.substr(0, q) + "?<signed>";
|
||||
}
|
||||
return socket_->url();
|
||||
}
|
||||
|
||||
void ConvaiSession::connect() {
|
||||
socket_->connect();
|
||||
Json init = options_.initiation.is_object() ? options_.initiation : Json::object();
|
||||
init.set("type", Json("conversation_initiation_client_data"));
|
||||
socket_->send_json(init);
|
||||
}
|
||||
|
||||
void ConvaiSession::send_audio(const std::string& pcm) {
|
||||
if (pcm.empty() || pcm.size() > 1024u * 1024u) invalid("audio chunk must be 1 byte..1 MiB");
|
||||
Json m = Json::object();
|
||||
m.set("user_audio_chunk", Json(security::crypto::Base64::encode(pcm)));
|
||||
socket_->send_json(m, true);
|
||||
}
|
||||
|
||||
void ConvaiSession::send_user_message(const std::string& text) {
|
||||
if (text.empty() || text.size() > 20000) invalid("text must be 1..20000 characters");
|
||||
Json m = Json::object();
|
||||
m.set("type", Json("user_message"));
|
||||
m.set("text", Json(text));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void ConvaiSession::send_user_activity() {
|
||||
Json m = Json::object();
|
||||
m.set("type", Json("user_activity"));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void ConvaiSession::send_contextual_update(const std::string& text) {
|
||||
if (text.empty() || text.size() > 20000) invalid("text must be 1..20000 characters");
|
||||
Json m = Json::object();
|
||||
m.set("type", Json("contextual_update"));
|
||||
m.set("text", Json(text));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void ConvaiSession::send_tool_result(const std::string& id, const std::string& result, bool is_error) {
|
||||
if (!llm::Service::valid_id(id)) invalid("bad tool_call_id");
|
||||
if (result.size() > 200000) invalid("result is over 200000 characters");
|
||||
Json m = Json::object();
|
||||
m.set("type", Json("client_tool_result"));
|
||||
m.set("tool_call_id", Json(id));
|
||||
m.set("result", Json(result));
|
||||
m.set("is_error", Json(is_error));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void ConvaiSession::fold(const SocketEvent& ev, ConvaiTurn& turn) {
|
||||
if (ev.kind == SocketEvent::Kind::kClose) {
|
||||
turn.closed = true;
|
||||
return;
|
||||
}
|
||||
if (ev.kind != SocketEvent::Kind::kText || !ev.json_ok) return;
|
||||
const Json& j = ev.json;
|
||||
const std::string& t = ev.type;
|
||||
if (t == "conversation_initiation_metadata") {
|
||||
if (const Json* e = obj_at(j, "conversation_initiation_metadata_event")) {
|
||||
turn.conversation_id = str_at(*e, "conversation_id");
|
||||
turn.agent_audio_format = str_at(*e, "agent_output_audio_format");
|
||||
turn.user_audio_format = str_at(*e, "user_input_audio_format");
|
||||
}
|
||||
} else if (t == "user_transcript") {
|
||||
if (const Json* e = obj_at(j, "user_transcription_event")) turn.user_transcripts.push_back(str_at(*e, "user_transcript"));
|
||||
} else if (t == "agent_response") {
|
||||
if (const Json* e = obj_at(j, "agent_response_event")) turn.agent_responses.push_back(str_at(*e, "agent_response"));
|
||||
} else if (t == "audio") {
|
||||
if (const Json* e = obj_at(j, "audio_event")) {
|
||||
std::string raw;
|
||||
if (!security::crypto::Base64::decode(str_at(*e, "audio_base_64"), raw))
|
||||
throw llm::ApiError("elevenlabs", 502, "BAD_RESPONSE", "audio event is not valid base64");
|
||||
if (turn.audio.size() + raw.size() > AudioCaps::kMaxStreamAudioBytes) {
|
||||
turn.truncated = true;
|
||||
} else {
|
||||
turn.audio += raw;
|
||||
++turn.audio_chunks;
|
||||
}
|
||||
}
|
||||
} else if (t == "interruption") {
|
||||
turn.interrupted = true;
|
||||
} else if (t == "ping") {
|
||||
++turn.pings;
|
||||
Json pong = Json::object();
|
||||
pong.set("type", Json("pong"));
|
||||
if (const Json* e = obj_at(j, "ping_event")) {
|
||||
if (const Json* id = e->find("event_id")) pong.set("event_id", *id);
|
||||
}
|
||||
socket_->send_json(pong);
|
||||
} else if (t == "client_tool_call") {
|
||||
if (const Json* e = obj_at(j, "client_tool_call")) turn.tool_calls.push_back(*e);
|
||||
} else if (t == "agent_response_complete") {
|
||||
turn.complete = true;
|
||||
} else if (t == "client_error" || t == "error") {
|
||||
turn.error = str_at(j, "message");
|
||||
if (turn.error.empty()) turn.error = t;
|
||||
}
|
||||
}
|
||||
|
||||
ConvaiTurn ConvaiSession::collect_turn(int max_wait_ms) {
|
||||
ConvaiTurn turn;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto ev = socket_->next(max_wait_ms);
|
||||
if (!ev) return turn;
|
||||
fold(*ev, turn);
|
||||
if (turn.complete || turn.closed || !turn.error.empty() || turn.truncated) return turn;
|
||||
}
|
||||
return turn;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
#pragma once
|
||||
|
||||
// A conversation with an ElevenLabs conversational AI agent over WebSocket
|
||||
// (wss://api.elevenlabs.io/v1/convai/conversation). Public agents connect by
|
||||
// agent id, private agents with a signed URL (see Agents::signed_url); the
|
||||
// signed URL is a credential and is never exposed by this class.
|
||||
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct ConvaiOptions {
|
||||
std::string agent_id; // public or key-authenticated agent
|
||||
std::string signed_url; // wss URL from Agents::signed_url (takes the place of agent_id and key)
|
||||
std::string url; // "" = wss://api.elevenlabs.io/v1/convai/conversation (tests)
|
||||
Json initiation; // conversation_initiation_client_data fields: conversation_config_override, dynamic_variables, ...
|
||||
SocketOptions socket;
|
||||
};
|
||||
|
||||
struct ConvaiTurn {
|
||||
std::string conversation_id;
|
||||
std::string user_audio_format, agent_audio_format;
|
||||
std::vector<std::string> user_transcripts;
|
||||
std::vector<std::string> agent_responses;
|
||||
std::string audio; // concatenated agent audio (decoded)
|
||||
std::size_t audio_chunks = 0;
|
||||
std::vector<Json> tool_calls; // client_tool_call objects: answer with send_tool_result
|
||||
bool interrupted = false;
|
||||
bool complete = false; // agent_response_complete seen
|
||||
bool closed = false;
|
||||
bool truncated = false;
|
||||
std::string error;
|
||||
std::size_t pings = 0; // answered automatically
|
||||
};
|
||||
|
||||
class ConvaiSession {
|
||||
public:
|
||||
// `api_key` may be empty for public agents and signed URLs.
|
||||
ConvaiSession(std::string api_key, ConvaiOptions options);
|
||||
|
||||
std::string url() const; // without the signature/credential
|
||||
void connect(); // sends conversation_initiation_client_data
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
void send_audio(const std::string& pcm16_16k_mono); // {"user_audio_chunk": base64}
|
||||
void send_user_message(const std::string& text);
|
||||
void send_user_activity();
|
||||
void send_contextual_update(const std::string& text);
|
||||
void send_tool_result(const std::string& tool_call_id, const std::string& result, bool is_error);
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
// Reads events (answering pings) until agent_response_complete, an error,
|
||||
// close, or `max_wait_ms` without an event.
|
||||
ConvaiTurn collect_turn(int max_wait_ms);
|
||||
|
||||
static Vocabulary vocabulary();
|
||||
|
||||
private:
|
||||
void fold(const SocketEvent& ev, ConvaiTurn& turn);
|
||||
|
||||
ConvaiOptions options_;
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
#include "vendors/audio/elevenlabs/creative.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
using C = ElevenCalls;
|
||||
|
||||
namespace {
|
||||
Query format_query(const AudioService& svc, const std::string& format) {
|
||||
Query q;
|
||||
if (!format.empty()) {
|
||||
if (!valid_output_format(format)) svc.bad("unknown output_format: " + format);
|
||||
q.add("output_format", format);
|
||||
}
|
||||
return q;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Json SoundEffects::build(const SfxRequest& r) const {
|
||||
require_text("text", r.text, 4100);
|
||||
Json o = Json::object();
|
||||
o.set("text", Json(r.text));
|
||||
if (r.loop >= 0) o.set("loop", Json(r.loop == 1));
|
||||
if (r.duration_seconds >= 0) {
|
||||
require_range("duration_seconds", r.duration_seconds, ElevenCaps::kMinSfxSeconds, ElevenCaps::kMaxSfxSeconds);
|
||||
o.set("duration_seconds", Json(r.duration_seconds));
|
||||
}
|
||||
if (r.prompt_influence >= 0) {
|
||||
require_range("prompt_influence", r.prompt_influence, 0, 1);
|
||||
o.set("prompt_influence", Json(r.prompt_influence));
|
||||
}
|
||||
if (!r.model_id.empty()) {
|
||||
require_text("model_id", r.model_id, 128);
|
||||
o.set("model_id", Json(r.model_id));
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
AudioBlob SoundEffects::generate(const SfxRequest& r) {
|
||||
Json body = build(r);
|
||||
llm::RawResponse raw = call_raw(C::kSfx, {}, &body, format_query(*this, r.output_format));
|
||||
return AudioBlob{raw.body, raw.content_type};
|
||||
}
|
||||
|
||||
Json Music::build(const MusicRequest& r) const {
|
||||
bool has_prompt = !r.prompt.empty();
|
||||
bool has_plan = r.composition_plan.is_object();
|
||||
if (has_prompt == has_plan) bad("give exactly one of prompt and composition_plan");
|
||||
Json o = Json::object();
|
||||
if (has_prompt) {
|
||||
require_text("prompt", r.prompt, ElevenCaps::kMaxPromptChars);
|
||||
o.set("prompt", Json(r.prompt));
|
||||
if (r.music_length_ms > 0) {
|
||||
require_range("music_length_ms", static_cast<double>(r.music_length_ms), ElevenCaps::kMinMusicMs,
|
||||
ElevenCaps::kMaxMusicMs);
|
||||
o.set("music_length_ms", Json(r.music_length_ms));
|
||||
}
|
||||
if (r.force_instrumental >= 0) o.set("force_instrumental", Json(r.force_instrumental == 1));
|
||||
if (r.seed >= 0) bad("seed is only valid with composition_plan");
|
||||
} else {
|
||||
o.set("composition_plan", r.composition_plan);
|
||||
if (r.music_length_ms > 0) bad("music_length_ms is only valid with prompt");
|
||||
if (r.force_instrumental >= 0) bad("force_instrumental is only valid with prompt");
|
||||
if (r.seed >= 0) {
|
||||
if (r.seed > 2147483647L) bad("seed must be 0..2147483647");
|
||||
o.set("seed", Json(r.seed));
|
||||
}
|
||||
}
|
||||
if (!r.model_id.empty()) {
|
||||
require_text("model_id", r.model_id, 128);
|
||||
o.set("model_id", Json(r.model_id));
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
AudioBlob Music::compose(const MusicRequest& r) {
|
||||
Json body = build(r);
|
||||
llm::RawResponse raw = call_raw(C::kMusic, {}, &body, format_query(*this, r.output_format));
|
||||
return AudioBlob{raw.body, raw.content_type};
|
||||
}
|
||||
|
||||
ByteStreamer::Result Music::compose_stream(const MusicRequest& r, const ChunkSink& sink) {
|
||||
Json body = build(r);
|
||||
return call_stream(C::kMusicStream, {}, &body, format_query(*this, r.output_format), sink);
|
||||
}
|
||||
|
||||
Json Music::plan(const std::string& prompt, long music_length_ms) {
|
||||
require_text("prompt", prompt, ElevenCaps::kMaxPromptChars);
|
||||
Json o = Json::object();
|
||||
o.set("prompt", Json(prompt));
|
||||
if (music_length_ms > 0) {
|
||||
require_range("music_length_ms", static_cast<double>(music_length_ms), ElevenCaps::kMinMusicMs,
|
||||
ElevenCaps::kMaxMusicMs);
|
||||
o.set("music_length_ms", Json(music_length_ms));
|
||||
}
|
||||
return call(C::kMusicPlan, {}, &o);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
#pragma once
|
||||
|
||||
// Sound effects and music generation. Both return audio bytes.
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
#include "vendors/audio/elevenlabs/tts.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct SfxRequest {
|
||||
std::string text; // required
|
||||
int loop = -1; // -1 unset
|
||||
double duration_seconds = -1; // 0.5..30, -1 = the model guesses
|
||||
double prompt_influence = -1; // 0..1
|
||||
std::string model_id; // "" = eleven_text_to_sound_v2
|
||||
std::string output_format; // query
|
||||
};
|
||||
|
||||
class SoundEffects : public ElevenService {
|
||||
public:
|
||||
using ElevenService::ElevenService;
|
||||
Json build(const SfxRequest& request) const;
|
||||
AudioBlob generate(const SfxRequest& request);
|
||||
};
|
||||
|
||||
struct MusicRequest {
|
||||
std::string prompt; // <= 4100 characters; exclusive with composition_plan
|
||||
Json composition_plan; // object from compose_plan, or null
|
||||
long music_length_ms = 0; // 3000..600000, only with prompt, 0 unset
|
||||
std::string model_id; // "" = music_v1
|
||||
long seed = -1; // 0..2147483647, only with composition_plan
|
||||
int force_instrumental = -1; // only with prompt
|
||||
std::string output_format; // query
|
||||
};
|
||||
|
||||
class Music : public ElevenService {
|
||||
public:
|
||||
using ElevenService::ElevenService;
|
||||
Json build(const MusicRequest& request) const;
|
||||
AudioBlob compose(const MusicRequest& request);
|
||||
ByteStreamer::Result compose_stream(const MusicRequest& request, const ChunkSink& sink);
|
||||
// Composition plan from a prompt (JSON, no audio).
|
||||
Json plan(const std::string& prompt, long music_length_ms = 0);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
#include "vendors/audio/elevenlabs/dubbing.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
using C = ElevenCalls;
|
||||
|
||||
clients::body::MultipartBody Dubbing::build(const DubbingRequest& r) const {
|
||||
if (r.media.empty() == r.source_url.empty()) bad("give exactly one of media and source_url");
|
||||
if (r.media.size() > ElevenCaps::kMaxFileBytes) bad("media is over the byte cap");
|
||||
require_text("source_url", r.source_url, AudioCaps::kMaxFieldChars, false);
|
||||
require_text("target_language", r.target_language, 16);
|
||||
require_text("source_language", r.source_language, 16, false);
|
||||
require_text("model_id", r.model_id, 128, false);
|
||||
if (r.keyterms.size() > AudioCaps::kMaxKeywords) bad("too many keyterms");
|
||||
clients::body::MultipartBody form;
|
||||
if (!r.media.empty()) form.add_file("file", r.filename, r.content_type, r.media);
|
||||
if (!r.source_url.empty()) form.add_text("source_url", r.source_url);
|
||||
form.add_text("target_language", r.target_language);
|
||||
if (!r.source_language.empty()) form.add_text("source_language", r.source_language);
|
||||
if (!r.model_id.empty()) form.add_text("model_id", r.model_id);
|
||||
for (const auto& k : r.keyterms) {
|
||||
require_text("keyterm", k, 100);
|
||||
form.add_text("keyterms", k);
|
||||
}
|
||||
return form;
|
||||
}
|
||||
|
||||
Json Dubbing::create(const DubbingRequest& r, bool confirm) {
|
||||
auto form = build(r);
|
||||
require_confirm("creating a dubbing project", confirm);
|
||||
return call_form(C::kDubCreate, {}, form);
|
||||
}
|
||||
|
||||
Json Dubbing::list(const Json& q) { return call(C::kDubList, {}, nullptr, query_from(C::kDubList, q)); }
|
||||
|
||||
Json Dubbing::get(const std::string& id) {
|
||||
check_id("dubbing project", id);
|
||||
return call(C::kDubGet, {id});
|
||||
}
|
||||
|
||||
Json Dubbing::languages(const std::string& id, const Json& q) {
|
||||
check_id("dubbing project", id);
|
||||
return call("dubbing_language_list", {id}, nullptr, query_from("dubbing_language_list", q));
|
||||
}
|
||||
|
||||
Json Dubbing::transcript(const std::string& id) {
|
||||
check_id("dubbing project", id);
|
||||
return call("dubbing_transcript_get", {id});
|
||||
}
|
||||
|
||||
Json Dubbing::remove(const std::string& id, bool confirm) {
|
||||
check_id("dubbing project", id);
|
||||
return call_destructive(C::kDubDelete, {id}, nullptr, Query(), confirm);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
#pragma once
|
||||
|
||||
// Dubbing projects: create (costs credits, confirm required), list, get,
|
||||
// languages, transcripts, delete.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct DubbingRequest {
|
||||
std::string media; // bytes of the audio or video, or use source_url
|
||||
std::string filename = "media";
|
||||
std::string content_type = "application/octet-stream";
|
||||
std::string source_url;
|
||||
std::string target_language; // required
|
||||
std::string source_language; // "" = detect
|
||||
std::string model_id;
|
||||
std::vector<std::string> keyterms;
|
||||
};
|
||||
|
||||
class Dubbing : public ElevenService {
|
||||
public:
|
||||
using ElevenService::ElevenService;
|
||||
clients::body::MultipartBody build(const DubbingRequest& request) const;
|
||||
// Starts a dub. Costs credits: confirm must be true.
|
||||
Json create(const DubbingRequest& request, bool confirm);
|
||||
Json list(const Json& query_object = Json());
|
||||
Json get(const std::string& project_id);
|
||||
Json languages(const std::string& project_id, const Json& query_object = Json());
|
||||
Json transcript(const std::string& project_id);
|
||||
Json remove(const std::string& project_id, bool confirm);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
#include "vendors/audio/elevenlabs/scribe_stream.h"
|
||||
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("elevenlabs", why); }
|
||||
|
||||
std::string build_query(const ScribeStreamOptions& o) {
|
||||
Query q;
|
||||
q.add_if("model_id", o.model_id);
|
||||
q.add_if("audio_format", o.audio_format);
|
||||
q.add_if("language_code", o.language_code);
|
||||
if (!o.commit_strategy.empty()) {
|
||||
if (o.commit_strategy != "manual" && o.commit_strategy != "vad") invalid("commit_strategy must be manual or vad");
|
||||
q.add("commit_strategy", o.commit_strategy);
|
||||
}
|
||||
if (o.include_timestamps >= 0) q.add_bool("include_timestamps", o.include_timestamps == 1);
|
||||
if (o.include_language_detection >= 0) q.add_bool("include_language_detection", o.include_language_detection == 1);
|
||||
if (o.keyterms.size() > AudioCaps::kMaxKeywords) invalid("too many keyterms");
|
||||
for (const auto& k : o.keyterms) {
|
||||
if (k.empty() || k.size() > 100) invalid("keyterms are 1..100 characters");
|
||||
q.add("keyterms", k);
|
||||
}
|
||||
std::string s = q.str();
|
||||
return s.empty() ? s : s.substr(1);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Vocabulary ScribeStream::vocabulary() {
|
||||
// AsyncAPI channel /v1/speech-to-text/realtime, discriminator message_type.
|
||||
return Vocabulary(
|
||||
"message_type", {"input_audio_chunk"},
|
||||
{"session_started", "partial_transcript", "committed_transcript",
|
||||
"committed_transcript_with_timestamps", "committed_transcript_entities", "edited_transcript",
|
||||
"warning", "error", "auth_error", "quota_exceeded", "transcriber_error", "input_error",
|
||||
"invalid_request", "rate_limited", "resource_exhausted", "queue_overflow",
|
||||
"commit_throttled", "chunk_size_exceeded", "insufficient_audio_activity",
|
||||
"session_time_limit_exceeded", "unaccepted_terms"});
|
||||
}
|
||||
|
||||
bool ScribeStream::is_error_type(const std::string& t) {
|
||||
static const char* kErrors[] = {"error", "auth_error", "quota_exceeded", "transcriber_error",
|
||||
"input_error", "invalid_request", "rate_limited",
|
||||
"resource_exhausted", "queue_overflow", "commit_throttled",
|
||||
"chunk_size_exceeded", "insufficient_audio_activity",
|
||||
"session_time_limit_exceeded", "unaccepted_terms"};
|
||||
for (const char* e : kErrors)
|
||||
if (t == e) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
ScribeStream::ScribeStream(std::string api_key, ScribeStreamOptions options) {
|
||||
std::string base = options.url.empty() ? ElevenEndpoints::ws_base() + "/v1/speech-to-text/realtime" : options.url;
|
||||
std::string url = AudioSocket::with_query(base, build_query(options));
|
||||
socket_ = std::make_unique<AudioSocket>("elevenlabs", url, ElevenEndpoints::ws_headers(api_key),
|
||||
vocabulary(), options.socket);
|
||||
}
|
||||
|
||||
void ScribeStream::send_audio(const std::string& audio, bool commit_now, int sample_rate,
|
||||
const std::string& previous_text) {
|
||||
if (audio.empty() || audio.size() > 3u * 1024u * 1024u) invalid("audio chunk must be 1 byte..3 MiB");
|
||||
Json m = Json::object();
|
||||
m.set("message_type", Json("input_audio_chunk"));
|
||||
m.set("audio_base_64", Json(security::crypto::Base64::encode(audio)));
|
||||
m.set("commit", Json(commit_now));
|
||||
if (sample_rate > 0) {
|
||||
if (sample_rate < 8000 || sample_rate > 96000) invalid("sample_rate must be 8000..96000");
|
||||
m.set("sample_rate", Json(sample_rate));
|
||||
}
|
||||
if (!previous_text.empty()) {
|
||||
if (previous_text.size() > 50000) invalid("previous_text is over 50000 characters");
|
||||
m.set("previous_text", Json(previous_text));
|
||||
}
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void ScribeStream::commit() {
|
||||
Json m = Json::object();
|
||||
m.set("message_type", Json("input_audio_chunk"));
|
||||
m.set("audio_base_64", Json(""));
|
||||
m.set("commit", Json(true));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
ScribeResult ScribeStream::collect(int max_wait_ms, std::size_t want_committed) {
|
||||
ScribeResult out;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto ev = socket_->next(max_wait_ms);
|
||||
if (!ev) return out;
|
||||
++out.events;
|
||||
if (ev->kind == SocketEvent::Kind::kClose) {
|
||||
out.closed = true;
|
||||
return out;
|
||||
}
|
||||
if (!ev->json_ok || !ev->json.is_object()) continue;
|
||||
const std::string& t = ev->type;
|
||||
auto text_of = [&](const char* k) {
|
||||
const Json* v = ev->json.find(k);
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
};
|
||||
if (t == "session_started") {
|
||||
out.session_id = text_of("session_id");
|
||||
} else if (t == "partial_transcript") {
|
||||
out.partial = text_of("text");
|
||||
} else if (t == "committed_transcript" || t == "committed_transcript_with_timestamps") {
|
||||
// With timestamps on the server may send both forms for one commit: keep one.
|
||||
if (t == "committed_transcript_with_timestamps" && !out.committed.empty() &&
|
||||
out.committed.back() == text_of("text"))
|
||||
continue;
|
||||
out.committed.push_back(text_of("text"));
|
||||
out.committed_text += (out.committed_text.empty() ? "" : " ") + out.committed.back();
|
||||
if (out.committed.size() >= want_committed) return out;
|
||||
} else if (t == "warning") {
|
||||
out.warnings.push_back(text_of("warning"));
|
||||
} else if (is_error_type(t)) {
|
||||
out.error_type = t;
|
||||
out.error_message = text_of("error");
|
||||
return out;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
#pragma once
|
||||
|
||||
// Realtime speech to text (Scribe) over WebSocket:
|
||||
// wss://api.elevenlabs.io/v1/speech-to-text/realtime. Send audio as
|
||||
// input_audio_chunk messages, receive partial and committed transcripts.
|
||||
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct ScribeStreamOptions {
|
||||
std::string model_id; // "" = server default
|
||||
std::string audio_format; // "" = default, else pcm_16000 etc (as the AsyncAPI lists)
|
||||
std::string language_code;
|
||||
std::string commit_strategy; // "", manual, vad
|
||||
int include_timestamps = -1;
|
||||
int include_language_detection = -1;
|
||||
std::vector<std::string> keyterms; // <= 100
|
||||
std::string url; // "" = wss://api.elevenlabs.io/v1/speech-to-text/realtime
|
||||
SocketOptions socket;
|
||||
};
|
||||
|
||||
struct ScribeResult {
|
||||
std::string committed_text; // committed transcripts joined by a space
|
||||
std::vector<std::string> committed; // each committed transcript
|
||||
std::string partial; // last partial
|
||||
std::string session_id;
|
||||
std::string error_type; // message_type of an error event, "" when none
|
||||
std::string error_message;
|
||||
std::vector<std::string> warnings;
|
||||
bool closed = false;
|
||||
std::size_t events = 0;
|
||||
};
|
||||
|
||||
class ScribeStream {
|
||||
public:
|
||||
ScribeStream(std::string api_key, ScribeStreamOptions options);
|
||||
std::string url() const { return socket_->url(); }
|
||||
void connect() { socket_->connect(); }
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
// Raw audio bytes (encoded as the audio_format says), 1..3 MiB per chunk.
|
||||
void send_audio(const std::string& audio, bool commit = false, int sample_rate = 0,
|
||||
const std::string& previous_text = "");
|
||||
void commit(); // empty chunk with commit=true
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
// Reads until a committed transcript, an error, close, or `max_wait_ms` of silence.
|
||||
ScribeResult collect(int max_wait_ms, std::size_t want_committed = 1);
|
||||
|
||||
static Vocabulary vocabulary();
|
||||
static bool is_error_type(const std::string& message_type);
|
||||
|
||||
private:
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
|
||||
#include <set>
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
bool valid_output_format(const std::string& format) {
|
||||
// OpenAPI /v1/text-to-speech/{voice_id} parameter output_format, enum as read 2026-10-25.
|
||||
static const std::set<std::string> kFormats = {
|
||||
"alaw_8000", "mp3_22050_32", "mp3_24000_48", "mp3_44100_128", "mp3_44100_192",
|
||||
"mp3_44100_32", "mp3_44100_64", "mp3_44100_96", "opus_48000_128", "opus_48000_192",
|
||||
"opus_48000_32", "opus_48000_64", "opus_48000_96", "pcm_16000", "pcm_22050",
|
||||
"pcm_24000", "pcm_32000", "pcm_44100", "pcm_48000", "pcm_8000",
|
||||
"ulaw_8000", "wav_16000", "wav_22050", "wav_24000", "wav_32000",
|
||||
"wav_44100", "wav_48000", "wav_8000"};
|
||||
return kFormats.count(format) > 0;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
#pragma once
|
||||
|
||||
// Base of every ElevenLabs service: the ElevenLabs call table plus the
|
||||
// ceilings that come from the ElevenLabs documentation.
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "vendors/audio/audio_service.h"
|
||||
#include "vendors/audio/elevenlabs/calls.h"
|
||||
#include "vendors/audio/elevenlabs/client.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct ElevenCaps {
|
||||
static constexpr std::size_t kMaxTtsChars = 40000; // largest model limit in the docs (flash / turbo)
|
||||
static constexpr std::size_t kMaxPromptChars = 4100; // music prompt (OpenAPI maxLength)
|
||||
static constexpr double kMinSfxSeconds = 0.5; // OpenAPI: at least 0.5, at most 30
|
||||
static constexpr double kMaxSfxSeconds = 30.0;
|
||||
static constexpr long kMinMusicMs = 3000; // OpenAPI: 3000..600000
|
||||
static constexpr long kMaxMusicMs = 600000;
|
||||
static constexpr int kMaxPageSize = 100;
|
||||
static constexpr std::size_t kMaxFileBytes = 128u * 1024u * 1024u;
|
||||
};
|
||||
|
||||
class ElevenService : public AudioService {
|
||||
public:
|
||||
explicit ElevenService(std::shared_ptr<const Hosts> hosts)
|
||||
: AudioService(calls(), std::move(hosts)) {}
|
||||
// From the environment (real HTTPS, key from ELEVENLABS_API_KEY).
|
||||
ElevenService() : ElevenService(ElevenEndpoints::from_env()) {}
|
||||
};
|
||||
|
||||
// Output formats the OpenAPI document lists for `output_format` (codec_rate[_bitrate]).
|
||||
bool valid_output_format(const std::string& format);
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+97
@@ -0,0 +1,97 @@
|
||||
#include "vendors/audio/elevenlabs/speech.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("elevenlabs", why); }
|
||||
|
||||
std::string format_name(const SpeechRequest& r) {
|
||||
int rate = r.sample_rate;
|
||||
if (r.format == "mp3") {
|
||||
if (rate == 0 || rate == 44100) return "mp3_44100_128";
|
||||
if (rate == 22050) return "mp3_22050_32";
|
||||
if (rate == 24000) return "mp3_24000_48";
|
||||
invalid("mp3 sample_rate must be 22050, 24000 or 44100");
|
||||
}
|
||||
if (rate == 0) rate = 24000;
|
||||
for (int ok : {8000, 16000, 22050, 24000, 32000, 44100, 48000})
|
||||
if (rate == ok) return r.format + "_" + std::to_string(rate);
|
||||
invalid(r.format + " sample_rate must be 8000, 16000, 22050, 24000, 32000, 44100 or 48000");
|
||||
}
|
||||
} // namespace
|
||||
|
||||
ElevenSynthesizer::ElevenSynthesizer(std::shared_ptr<const Hosts> hosts)
|
||||
: tts_(hosts), voices_(hosts) {}
|
||||
|
||||
TtsRequest ElevenSynthesizer::map(const SpeechRequest& r) const {
|
||||
validate(r);
|
||||
if (r.voice.empty()) invalid("a voice id is required (list voices first)");
|
||||
TtsRequest t;
|
||||
t.voice_id = r.voice;
|
||||
t.text = r.text;
|
||||
t.model_id = r.model;
|
||||
t.language_code = r.language.substr(0, 2);
|
||||
if (r.speed > 0) t.voice_settings.speed = r.speed;
|
||||
t.output_format = format_name(r);
|
||||
return t;
|
||||
}
|
||||
|
||||
SpeechResult ElevenSynthesizer::synthesize(const SpeechRequest& r) {
|
||||
AudioBlob blob = tts_.synthesize(map(r));
|
||||
SpeechResult out;
|
||||
out.audio = std::move(blob.bytes);
|
||||
out.content_type = blob.content_type.empty() ? content_type_for(r.format) : blob.content_type;
|
||||
out.format = r.format;
|
||||
out.provider = name();
|
||||
return out;
|
||||
}
|
||||
|
||||
Json ElevenSynthesizer::voices() { return voices_.list_v2(VoiceListQuery()); }
|
||||
|
||||
ElevenRecognizer::ElevenRecognizer(std::shared_ptr<const Hosts> hosts) : stt_(std::move(hosts)) {}
|
||||
|
||||
SttRequest ElevenRecognizer::map(const TranscribeRequest& r) const {
|
||||
validate(r);
|
||||
SttRequest s;
|
||||
if (!r.model.empty()) s.model_id = r.model;
|
||||
s.audio = r.audio;
|
||||
s.filename = r.filename;
|
||||
s.content_type = r.content_type;
|
||||
s.source_url = r.url;
|
||||
s.language_code = r.language;
|
||||
s.diarize = r.diarize;
|
||||
s.keyterms = r.keywords;
|
||||
return s;
|
||||
}
|
||||
|
||||
Transcript ElevenRecognizer::fold(const Json& doc) {
|
||||
Transcript t;
|
||||
t.provider = "elevenlabs";
|
||||
t.raw = doc;
|
||||
auto str = [&](const char* k) {
|
||||
const Json* v = doc.is_object() ? doc.find(k) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
};
|
||||
t.text = str("text");
|
||||
t.language = str("language_code");
|
||||
if (const Json* d = doc.is_object() ? doc.find("audio_duration_secs") : nullptr; d && d->is_number())
|
||||
t.duration = d->as_number();
|
||||
if (const Json* w = doc.is_object() ? doc.find("words") : nullptr; w && w->is_array()) {
|
||||
for (const Json& e : w->as_array()) {
|
||||
if (!e.is_object()) continue;
|
||||
const Json* type = e.find("type");
|
||||
if (type && type->is_string() && type->as_string() != "word") continue;
|
||||
TranscriptWord word;
|
||||
if (const Json* x = e.find("text"); x && x->is_string()) word.text = x->as_string();
|
||||
if (const Json* x = e.find("start"); x && x->is_number()) word.start = x->as_number();
|
||||
if (const Json* x = e.find("end"); x && x->is_number()) word.end = x->as_number();
|
||||
if (const Json* x = e.find("speaker_id"); x && x->is_string()) word.speaker = x->as_string();
|
||||
t.words.push_back(word);
|
||||
}
|
||||
}
|
||||
return t;
|
||||
}
|
||||
|
||||
Transcript ElevenRecognizer::transcribe(const TranscribeRequest& r) { return fold(stt_.transcribe(map(r))); }
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
#pragma once
|
||||
|
||||
// ElevenLabs behind the neutral speech seam.
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "vendors/audio/elevenlabs/account.h"
|
||||
#include "vendors/audio/elevenlabs/stt.h"
|
||||
#include "vendors/audio/elevenlabs/tts.h"
|
||||
#include "vendors/audio/elevenlabs/voices.h"
|
||||
#include "vendors/audio/speech.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
class ElevenSynthesizer : public SpeechSynthesizer {
|
||||
public:
|
||||
explicit ElevenSynthesizer(std::shared_ptr<const Hosts> hosts = ElevenEndpoints::from_env());
|
||||
std::string name() const override { return "elevenlabs"; }
|
||||
bool configured() const override { return tts_.configured(); }
|
||||
std::size_t max_chars() const override { return ElevenCaps::kMaxTtsChars; }
|
||||
std::vector<std::string> formats() const override { return {"mp3", "wav", "pcm"}; }
|
||||
SpeechResult synthesize(const SpeechRequest& request) override;
|
||||
Json voices() override;
|
||||
|
||||
// The provider request a neutral request maps to (no network).
|
||||
TtsRequest map(const SpeechRequest& request) const;
|
||||
|
||||
private:
|
||||
TextToSpeech tts_;
|
||||
Voices voices_;
|
||||
};
|
||||
|
||||
class ElevenRecognizer : public SpeechRecognizer {
|
||||
public:
|
||||
explicit ElevenRecognizer(std::shared_ptr<const Hosts> hosts = ElevenEndpoints::from_env());
|
||||
std::string name() const override { return "elevenlabs"; }
|
||||
bool configured() const override { return stt_.configured(); }
|
||||
Transcript transcribe(const TranscribeRequest& request) override;
|
||||
SttRequest map(const TranscribeRequest& request) const;
|
||||
// Folds a Scribe response document into the neutral transcript.
|
||||
static Transcript fold(const Json& doc);
|
||||
|
||||
private:
|
||||
SpeechToText stt_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
Vendored
+66
@@ -0,0 +1,66 @@
|
||||
#include "vendors/audio/elevenlabs/stt.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
using C = ElevenCalls;
|
||||
|
||||
clients::body::MultipartBody SpeechToText::build(const SttRequest& r) const {
|
||||
require_text("model_id", r.model_id, 128);
|
||||
int sources = (!r.audio.empty()) + (!r.source_url.empty()) + (!r.cloud_storage_url.empty());
|
||||
if (sources != 1) bad("give exactly one of audio, source_url and cloud_storage_url");
|
||||
if (r.audio.size() > ElevenCaps::kMaxFileBytes) bad("audio is over the byte cap");
|
||||
require_text("source_url", r.source_url, AudioCaps::kMaxFieldChars, false);
|
||||
require_text("cloud_storage_url", r.cloud_storage_url, AudioCaps::kMaxFieldChars, false);
|
||||
require_text("language_code", r.language_code, 16, false);
|
||||
if (r.num_speakers < 0 || r.num_speakers > 32) bad("num_speakers must be 0 or 1..32");
|
||||
const std::string& g = r.timestamps_granularity;
|
||||
if (!g.empty() && g != "none" && g != "word" && g != "character")
|
||||
bad("timestamps_granularity must be none, word or character");
|
||||
if (!r.file_format.empty() && r.file_format != "pcm_s16le_16" && r.file_format != "other")
|
||||
bad("file_format must be pcm_s16le_16 or other");
|
||||
if (r.keyterms.size() > AudioCaps::kMaxKeywords) bad("too many keyterms");
|
||||
if (r.temperature > 2.0) bad("temperature must be 0..2");
|
||||
clients::body::MultipartBody form;
|
||||
form.add_text("model_id", r.model_id);
|
||||
if (!r.audio.empty()) form.add_file("file", r.filename, r.content_type, r.audio);
|
||||
if (!r.source_url.empty()) form.add_text("source_url", r.source_url);
|
||||
if (!r.cloud_storage_url.empty()) form.add_text("cloud_storage_url", r.cloud_storage_url);
|
||||
if (!r.language_code.empty()) form.add_text("language_code", r.language_code);
|
||||
if (r.diarize) form.add_text("diarize", "true");
|
||||
if (r.num_speakers > 0) form.add_text("num_speakers", std::to_string(r.num_speakers));
|
||||
if (!g.empty()) form.add_text("timestamps_granularity", g);
|
||||
if (r.tag_audio_events >= 0) form.add_text("tag_audio_events", r.tag_audio_events ? "true" : "false");
|
||||
for (const auto& k : r.keyterms) {
|
||||
require_text("keyterm", k, 100);
|
||||
form.add_text("keyterms", k);
|
||||
}
|
||||
if (!r.file_format.empty()) form.add_text("file_format", r.file_format);
|
||||
if (r.use_multi_channel >= 0) form.add_text("use_multi_channel", r.use_multi_channel ? "true" : "false");
|
||||
if (r.seed >= 0) form.add_text("seed", std::to_string(r.seed));
|
||||
if (r.temperature >= 0) form.add_text("temperature", std::to_string(r.temperature));
|
||||
return form;
|
||||
}
|
||||
|
||||
Query SpeechToText::query(const SttRequest& r) const {
|
||||
Query q;
|
||||
if (r.enable_logging >= 0) q.add_bool("enable_logging", r.enable_logging == 1);
|
||||
return q;
|
||||
}
|
||||
|
||||
Json SpeechToText::transcribe(const SttRequest& r) {
|
||||
return call_form(C::kStt, {}, build(r), query(r));
|
||||
}
|
||||
|
||||
Json SpeechToText::get(const std::string& id) {
|
||||
check_id("transcription", id);
|
||||
return call(C::kSttGet, {id});
|
||||
}
|
||||
|
||||
Json SpeechToText::remove(const std::string& id, bool confirm) {
|
||||
check_id("transcription", id);
|
||||
return call_destructive(C::kSttDelete, {id}, nullptr, Query(), confirm);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
Vendored
+43
@@ -0,0 +1,43 @@
|
||||
#pragma once
|
||||
|
||||
// Speech to text (Scribe): POST /v1/speech-to-text as multipart/form-data,
|
||||
// plus fetching and deleting stored transcripts.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct SttRequest {
|
||||
std::string model_id = "scribe_v2"; // required by the API; scribe_v2 is the OpenAPI example
|
||||
std::string audio; // bytes; or use source_url / cloud_storage_url
|
||||
std::string filename = "audio";
|
||||
std::string content_type = "application/octet-stream";
|
||||
std::string source_url;
|
||||
std::string cloud_storage_url;
|
||||
std::string language_code; // ISO 639-1 / 639-3, "" = detect
|
||||
bool diarize = false;
|
||||
int num_speakers = 0; // 0 = unset, else 1..32
|
||||
std::string timestamps_granularity; // "", none, word, character
|
||||
int tag_audio_events = -1; // -1 unset
|
||||
std::vector<std::string> keyterms; // <= AudioCaps::kMaxKeywords
|
||||
std::string file_format; // "", pcm_s16le_16, other
|
||||
int use_multi_channel = -1;
|
||||
long seed = -1;
|
||||
double temperature = -1; // 0..2
|
||||
int enable_logging = -1; // query: -1 unset, 0 false
|
||||
};
|
||||
|
||||
class SpeechToText : public ElevenService {
|
||||
public:
|
||||
using ElevenService::ElevenService;
|
||||
clients::body::MultipartBody build(const SttRequest& request) const;
|
||||
Query query(const SttRequest& request) const;
|
||||
Json transcribe(const SttRequest& request);
|
||||
Json get(const std::string& transcription_id);
|
||||
Json remove(const std::string& transcription_id, bool confirm);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
Vendored
+110
@@ -0,0 +1,110 @@
|
||||
#include "vendors/audio/elevenlabs/tts.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
using C = ElevenCalls;
|
||||
|
||||
bool VoiceSettings::empty() const {
|
||||
return stability < 0 && similarity_boost < 0 && style < 0 && speed < 0 && use_speaker_boost < 0;
|
||||
}
|
||||
|
||||
Json VoiceSettings::to_json() const {
|
||||
auto unit = [](double v, const char* what) {
|
||||
if (v > 1.0) throw llm::ApiError::invalid("elevenlabs", std::string(what) + " must be 0..1");
|
||||
};
|
||||
Json o = Json::object();
|
||||
if (stability >= 0) { unit(stability, "stability"); o.set("stability", Json(stability)); }
|
||||
if (similarity_boost >= 0) { unit(similarity_boost, "similarity_boost"); o.set("similarity_boost", Json(similarity_boost)); }
|
||||
if (style >= 0) { unit(style, "style"); o.set("style", Json(style)); }
|
||||
if (speed >= 0) {
|
||||
if (speed == 0)
|
||||
throw llm::ApiError::invalid("elevenlabs", "voice speed must be above 0");
|
||||
o.set("speed", Json(speed));
|
||||
}
|
||||
if (use_speaker_boost >= 0) o.set("use_speaker_boost", Json(use_speaker_boost == 1));
|
||||
return o;
|
||||
}
|
||||
|
||||
Json TextToSpeech::build(const TtsRequest& r) const {
|
||||
check_id("voice", r.voice_id);
|
||||
require_text("text", r.text, ElevenCaps::kMaxTtsChars);
|
||||
require_text("model_id", r.model_id, 128, false);
|
||||
require_text("language_code", r.language_code, 16, false);
|
||||
require_text("previous_text", r.previous_text, ElevenCaps::kMaxTtsChars, false);
|
||||
require_text("next_text", r.next_text, ElevenCaps::kMaxTtsChars, false);
|
||||
Json o = Json::object();
|
||||
o.set("text", Json(r.text));
|
||||
if (!r.model_id.empty()) o.set("model_id", Json(r.model_id));
|
||||
if (!r.language_code.empty()) o.set("language_code", Json(r.language_code));
|
||||
if (!r.voice_settings.empty()) o.set("voice_settings", r.voice_settings.to_json());
|
||||
if (r.seed >= 0) {
|
||||
if (r.seed > 4294967295L) bad("seed must be 0..4294967295");
|
||||
o.set("seed", Json(r.seed));
|
||||
}
|
||||
if (!r.previous_text.empty()) o.set("previous_text", Json(r.previous_text));
|
||||
if (!r.next_text.empty()) o.set("next_text", Json(r.next_text));
|
||||
if (!r.apply_text_normalization.empty()) {
|
||||
const std::string& n = r.apply_text_normalization;
|
||||
if (n != "auto" && n != "on" && n != "off") bad("apply_text_normalization must be auto, on or off");
|
||||
o.set("apply_text_normalization", Json(n));
|
||||
}
|
||||
if (!r.pronunciation_dictionaries.empty()) {
|
||||
if (r.pronunciation_dictionaries.size() > 3) bad("at most 3 pronunciation dictionaries");
|
||||
Json arr = Json::array();
|
||||
for (const auto& [id, version] : r.pronunciation_dictionaries) {
|
||||
check_id("pronunciation dictionary", id);
|
||||
Json loc = Json::object();
|
||||
loc.set("pronunciation_dictionary_id", Json(id));
|
||||
if (!version.empty()) {
|
||||
check_id("pronunciation dictionary version", version);
|
||||
loc.set("version_id", Json(version));
|
||||
}
|
||||
arr.push(loc);
|
||||
}
|
||||
o.set("pronunciation_dictionary_locators", arr);
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
Query TextToSpeech::query(const TtsRequest& r) const {
|
||||
Query q;
|
||||
if (!r.output_format.empty()) {
|
||||
if (!valid_output_format(r.output_format)) bad("unknown output_format: " + r.output_format);
|
||||
q.add("output_format", r.output_format);
|
||||
}
|
||||
if (r.optimize_streaming_latency >= 0) {
|
||||
if (r.optimize_streaming_latency > 4) bad("optimize_streaming_latency must be 0..4");
|
||||
q.add_int("optimize_streaming_latency", r.optimize_streaming_latency);
|
||||
}
|
||||
if (r.enable_logging >= 0) q.add_bool("enable_logging", r.enable_logging == 1);
|
||||
return q;
|
||||
}
|
||||
|
||||
std::string TextToSpeech::describe(const TtsRequest& r, bool streaming) const {
|
||||
Json body = build(r);
|
||||
return preview(streaming ? C::kTtsStream : C::kTts, {r.voice_id}, &body, query(r));
|
||||
}
|
||||
|
||||
AudioBlob TextToSpeech::synthesize(const TtsRequest& r) {
|
||||
Json body = build(r);
|
||||
llm::RawResponse raw = call_raw(C::kTts, {r.voice_id}, &body, query(r));
|
||||
return AudioBlob{raw.body, raw.content_type};
|
||||
}
|
||||
|
||||
ByteStreamer::Result TextToSpeech::stream(const TtsRequest& r, const ChunkSink& sink) {
|
||||
Json body = build(r);
|
||||
return call_stream(C::kTtsStream, {r.voice_id}, &body, query(r), sink);
|
||||
}
|
||||
|
||||
Json TextToSpeech::synthesize_with_timestamps(const TtsRequest& r) {
|
||||
Json body = build(r);
|
||||
return call(C::kTtsTimestamps, {r.voice_id}, &body, query(r));
|
||||
}
|
||||
|
||||
ByteStreamer::Result TextToSpeech::stream_with_timestamps(const TtsRequest& r, const ChunkSink& sink) {
|
||||
Json body = build(r);
|
||||
// The streamed variant is declared "bytes" in the table (newline delimited JSON).
|
||||
return call_stream(C::kTtsStreamTimestamps, {r.voice_id}, &body, query(r), sink);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
Vendored
+64
@@ -0,0 +1,64 @@
|
||||
#pragma once
|
||||
|
||||
// Text to speech: buffered, streamed (HTTP chunked bytes) and with
|
||||
// character timestamps. WebSocket input streaming is TtsInputStream.
|
||||
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct VoiceSettings {
|
||||
// Negative = not set (the voice's stored settings apply).
|
||||
double stability = -1;
|
||||
double similarity_boost = -1;
|
||||
double style = -1;
|
||||
double speed = -1;
|
||||
int use_speaker_boost = -1; // -1 unset, 0 false, 1 true
|
||||
bool empty() const;
|
||||
Json to_json() const; // throws llm::ApiError::invalid when a value is out of range
|
||||
};
|
||||
|
||||
struct TtsRequest {
|
||||
std::string voice_id; // required, path segment
|
||||
std::string text; // 1..ElevenCaps::kMaxTtsChars
|
||||
std::string model_id; // "" = server default (eleven_multilingual_v2)
|
||||
std::string language_code; // ISO 639-1
|
||||
VoiceSettings voice_settings;
|
||||
long seed = -1; // 0..4294967295, -1 unset
|
||||
std::string previous_text, next_text;
|
||||
std::string apply_text_normalization; // "", auto, on, off
|
||||
std::vector<std::pair<std::string, std::string>> pronunciation_dictionaries; // (id, version) <= 3
|
||||
// Query parameters.
|
||||
std::string output_format; // "" = server default mp3_44100_128
|
||||
int optimize_streaming_latency = -1; // 0..4, -1 unset
|
||||
int enable_logging = -1; // -1 unset, 0 false (zero retention), 1 true
|
||||
};
|
||||
|
||||
struct AudioBlob {
|
||||
std::string bytes;
|
||||
std::string content_type;
|
||||
};
|
||||
|
||||
class TextToSpeech : public ElevenService {
|
||||
public:
|
||||
using ElevenService::ElevenService;
|
||||
|
||||
Json build(const TtsRequest& request) const; // body (validates)
|
||||
Query query(const TtsRequest& request) const; // query (validates)
|
||||
AudioBlob synthesize(const TtsRequest& request);
|
||||
// Incremental audio over HTTP (/stream); `sink` returns false to stop.
|
||||
ByteStreamer::Result stream(const TtsRequest& request, const ChunkSink& sink);
|
||||
// JSON with audio_base64 and alignment / normalized_alignment.
|
||||
Json synthesize_with_timestamps(const TtsRequest& request);
|
||||
// Newline delimited JSON objects, each with audio_base64 and alignment.
|
||||
ByteStreamer::Result stream_with_timestamps(const TtsRequest& request, const ChunkSink& sink);
|
||||
// Request preview without sending (path with query, body).
|
||||
std::string describe(const TtsRequest& request, bool streaming = false) const;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
#include "vendors/audio/elevenlabs/tts_stream.h"
|
||||
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("elevenlabs", why); }
|
||||
|
||||
std::string build_query(const TtsStreamOptions& o) {
|
||||
Query q;
|
||||
q.add_if("model_id", o.model_id);
|
||||
if (!o.output_format.empty()) {
|
||||
if (!valid_output_format(o.output_format)) invalid("unknown output_format: " + o.output_format);
|
||||
q.add("output_format", o.output_format);
|
||||
}
|
||||
q.add_if("language_code", o.language_code);
|
||||
if (o.inactivity_timeout != 0) {
|
||||
if (o.inactivity_timeout < 1 || o.inactivity_timeout > 180) invalid("inactivity_timeout must be 1..180");
|
||||
q.add_int("inactivity_timeout", o.inactivity_timeout);
|
||||
}
|
||||
if (o.sync_alignment >= 0) q.add_bool("sync_alignment", o.sync_alignment == 1);
|
||||
if (o.auto_mode >= 0) q.add_bool("auto_mode", o.auto_mode == 1);
|
||||
if (!o.apply_text_normalization.empty()) {
|
||||
const std::string& n = o.apply_text_normalization;
|
||||
if (n != "auto" && n != "on" && n != "off") invalid("apply_text_normalization must be auto, on or off");
|
||||
q.add("apply_text_normalization", n);
|
||||
}
|
||||
if (o.seed >= 0) {
|
||||
if (o.seed > 4294967295L) invalid("seed must be 0..4294967295");
|
||||
q.add_int("seed", o.seed);
|
||||
}
|
||||
if (o.enable_ssml_parsing >= 0) q.add_bool("enable_ssml_parsing", o.enable_ssml_parsing == 1);
|
||||
if (o.enable_logging >= 0) q.add_bool("enable_logging", o.enable_logging == 1);
|
||||
std::string s = q.str();
|
||||
return s.empty() ? s : s.substr(1);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Vocabulary TtsInputStream::vocabulary() {
|
||||
// The stream-input messages carry no discriminator field: they are plain
|
||||
// {"text": ...} objects, so both sets are empty and sends are untyped.
|
||||
return Vocabulary("type", {}, {});
|
||||
}
|
||||
|
||||
TtsInputStream::TtsInputStream(std::string api_key, TtsStreamOptions options)
|
||||
: options_(std::move(options)), api_key_(std::move(api_key)) {
|
||||
if (!llm::Service::valid_id(options_.voice_id)) invalid("bad voice id");
|
||||
std::string base = options_.url.empty() ? ElevenEndpoints::ws_base() + "/v1/text-to-speech/" +
|
||||
options_.voice_id + "/stream-input"
|
||||
: options_.url;
|
||||
std::string url = AudioSocket::with_query(base, build_query(options_));
|
||||
socket_ = std::make_unique<AudioSocket>("elevenlabs", url, ElevenEndpoints::ws_headers(api_key_),
|
||||
vocabulary(), options_.socket);
|
||||
}
|
||||
|
||||
std::string TtsInputStream::url() const { return socket_->url(); }
|
||||
|
||||
void TtsInputStream::connect() {
|
||||
socket_->connect();
|
||||
Json init = Json::object();
|
||||
init.set("text", Json(" "));
|
||||
if (!options_.voice_settings.empty()) init.set("voice_settings", options_.voice_settings.to_json());
|
||||
if (!options_.chunk_length_schedule.empty()) {
|
||||
if (options_.chunk_length_schedule.size() > 8) invalid("chunk_length_schedule has at most 8 entries");
|
||||
Json sched = Json::array();
|
||||
for (int v : options_.chunk_length_schedule) {
|
||||
if (v < 50 || v > 500) invalid("chunk_length_schedule entries must be 50..500");
|
||||
sched.push(Json(v));
|
||||
}
|
||||
Json gen = Json::object();
|
||||
gen.set("chunk_length_schedule", sched);
|
||||
init.set("generation_config", gen);
|
||||
}
|
||||
socket_->send_json(init, true);
|
||||
}
|
||||
|
||||
void TtsInputStream::send_text(const std::string& text, bool try_trigger_generation) {
|
||||
if (text.empty() || text.size() > ElevenCaps::kMaxTtsChars) invalid("text must be 1..40000 characters");
|
||||
if (text.find('\0') != std::string::npos) invalid("text has a NUL byte");
|
||||
Json m = Json::object();
|
||||
m.set("text", Json(text.back() == ' ' ? text : text + " "));
|
||||
if (try_trigger_generation) m.set("try_trigger_generation", Json(true));
|
||||
socket_->send_json(m, true);
|
||||
}
|
||||
|
||||
void TtsInputStream::flush() {
|
||||
Json m = Json::object();
|
||||
m.set("text", Json(""));
|
||||
m.set("flush", Json(true));
|
||||
socket_->send_json(m, true);
|
||||
}
|
||||
|
||||
void TtsInputStream::finish() {
|
||||
Json m = Json::object();
|
||||
m.set("text", Json(""));
|
||||
socket_->send_json(m, true);
|
||||
}
|
||||
|
||||
void TtsInputStream::fold(const SocketEvent& ev, TtsStreamResult& out) {
|
||||
if (ev.kind == SocketEvent::Kind::kClose) {
|
||||
out.closed = true;
|
||||
return;
|
||||
}
|
||||
if (ev.kind != SocketEvent::Kind::kText || !ev.json_ok || !ev.json.is_object()) return;
|
||||
const Json& j = ev.json;
|
||||
if (const Json* audio = j.find("audio"); audio && audio->is_string()) {
|
||||
std::string raw;
|
||||
if (!security::crypto::Base64::decode(audio->as_string(), raw))
|
||||
throw llm::ApiError("elevenlabs", 502, "BAD_RESPONSE", "audio chunk is not valid base64");
|
||||
if (out.audio.size() + raw.size() > AudioCaps::kMaxStreamAudioBytes) {
|
||||
out.truncated = true;
|
||||
} else {
|
||||
out.audio += raw;
|
||||
++out.chunks;
|
||||
}
|
||||
}
|
||||
if (const Json* al = j.find("alignment"); al && al->is_object()) out.alignments.push_back(*al);
|
||||
if (const Json* fin = j.find("isFinal"); fin && fin->is_bool() && fin->as_bool()) out.is_final = true;
|
||||
if (!j.find("audio") && !j.find("isFinal")) {
|
||||
for (const char* k : {"message", "error", "detail"}) {
|
||||
if (const Json* e = j.find(k); e && e->is_string()) {
|
||||
out.error = e->as_string();
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
TtsStreamResult TtsInputStream::collect(int max_wait_ms) {
|
||||
TtsStreamResult out;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto ev = socket_->next(max_wait_ms);
|
||||
if (!ev) return out;
|
||||
fold(*ev, out);
|
||||
if (out.is_final || out.closed || !out.error.empty() || out.truncated) return out;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
TtsStreamResult TtsInputStream::speak(const std::string& text, int max_wait_ms) {
|
||||
send_text(text);
|
||||
finish();
|
||||
return collect(max_wait_ms);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
#pragma once
|
||||
|
||||
// ElevenLabs text to speech over WebSocket (stream-input): send text as it
|
||||
// is produced (for example by an LLM), receive base64 audio chunks with
|
||||
// optional character alignment. Channel and message shapes follow
|
||||
// https://elevenlabs.io/docs/asyncapi/api-reference-2.json.
|
||||
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/elevenlabs/tts.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct TtsStreamOptions {
|
||||
std::string voice_id; // required
|
||||
std::string model_id; // "" = server default
|
||||
std::string output_format; // "" = mp3_44100_128
|
||||
std::string language_code;
|
||||
int inactivity_timeout = 0; // seconds, 0 unset, else 1..180
|
||||
int sync_alignment = -1;
|
||||
int auto_mode = -1;
|
||||
std::string apply_text_normalization; // "", auto, on, off
|
||||
long seed = -1;
|
||||
int enable_ssml_parsing = -1;
|
||||
int enable_logging = -1;
|
||||
VoiceSettings voice_settings; // sent in the first message
|
||||
std::vector<int> chunk_length_schedule; // generation_config, each 50..500, at most 8
|
||||
std::string url; // "" = wss://api.elevenlabs.io/v1/text-to-speech/{voice}/stream-input
|
||||
SocketOptions socket;
|
||||
};
|
||||
|
||||
struct TtsStreamResult {
|
||||
std::string audio; // concatenated decoded chunks
|
||||
std::size_t chunks = 0;
|
||||
std::vector<Json> alignments; // alignment objects when sync_alignment is on
|
||||
bool is_final = false; // {"isFinal":true} seen
|
||||
bool truncated = false; // audio cap hit
|
||||
std::string error; // server error message, "" when none
|
||||
bool closed = false;
|
||||
};
|
||||
|
||||
class TtsInputStream {
|
||||
public:
|
||||
TtsInputStream(std::string api_key, TtsStreamOptions options);
|
||||
|
||||
std::string url() const; // as opened, without any credential
|
||||
void connect(); // sends the initial " " message with voice settings
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
// Text must end with a space per the docs; added when missing. 1..40000 chars.
|
||||
void send_text(const std::string& text, bool try_trigger_generation = false);
|
||||
void flush(); // {"text":"","flush":true}
|
||||
void finish(); // {"text":""}: end of input, server closes after the last audio
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
// Reads until isFinal / error / close or `max_wait_ms` with no event.
|
||||
TtsStreamResult collect(int max_wait_ms);
|
||||
// send_text + finish + collect.
|
||||
TtsStreamResult speak(const std::string& text, int max_wait_ms);
|
||||
|
||||
const AudioSocket& socket() const { return *socket_; }
|
||||
|
||||
static Vocabulary vocabulary();
|
||||
|
||||
private:
|
||||
void fold(const SocketEvent& ev, TtsStreamResult& out);
|
||||
|
||||
TtsStreamOptions options_;
|
||||
std::string api_key_;
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+145
@@ -0,0 +1,145 @@
|
||||
#include "vendors/audio/elevenlabs/voices.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
using C = ElevenCalls;
|
||||
|
||||
namespace {
|
||||
Query list_query(const AudioService& svc, const VoiceListQuery& q) {
|
||||
Query out;
|
||||
if (q.page_size < 0 || q.page_size > ElevenCaps::kMaxPageSize) svc.bad("page_size must be 1..100");
|
||||
if (q.page_size > 0) out.add_int("page_size", q.page_size);
|
||||
out.add_if("next_page_token", q.next_page_token);
|
||||
out.add_if("search", q.search);
|
||||
if (!q.sort.empty() && q.sort != "created_at_unix" && q.sort != "name")
|
||||
svc.bad("sort must be created_at_unix or name");
|
||||
out.add_if("sort", q.sort);
|
||||
if (!q.sort_direction.empty() && q.sort_direction != "asc" && q.sort_direction != "desc")
|
||||
svc.bad("sort_direction must be asc or desc");
|
||||
out.add_if("sort_direction", q.sort_direction);
|
||||
out.add_if("voice_type", q.voice_type);
|
||||
out.add_if("category", q.category);
|
||||
if (q.include_total_count >= 0) out.add_bool("include_total_count", q.include_total_count == 1);
|
||||
return out;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Json Voices::list(bool show_legacy) {
|
||||
Query q;
|
||||
if (show_legacy) q.add_bool("show_legacy", true);
|
||||
return call(C::kVoices, {}, nullptr, q);
|
||||
}
|
||||
|
||||
Json Voices::list_v2(const VoiceListQuery& q) { return call(C::kVoicesV2, {}, nullptr, list_query(*this, q)); }
|
||||
|
||||
VoiceListing Voices::list_all_v2(VoiceListQuery q, std::size_t max_pages) {
|
||||
if (max_pages == 0 || max_pages > AudioCaps::kMaxPages) bad("max_pages must be 1.." + std::to_string(AudioCaps::kMaxPages));
|
||||
VoiceListing out;
|
||||
for (std::size_t page = 0; page < max_pages; ++page) {
|
||||
Json doc = list_v2(q);
|
||||
++out.pages;
|
||||
if (const Json* v = doc.find("voices"); v && v->is_array())
|
||||
for (const Json& e : v->as_array()) {
|
||||
if (out.voices.as_array().size() >= AudioCaps::kMaxListItems) {
|
||||
out.truncated = true;
|
||||
return out;
|
||||
}
|
||||
out.voices.push(e);
|
||||
}
|
||||
if (const Json* t = doc.find("total_count"); t && t->is_number()) out.total_count = t->as_int();
|
||||
const Json* more = doc.find("has_more");
|
||||
const Json* token = doc.find("next_page_token");
|
||||
if (!more || !more->is_bool() || !more->as_bool() || !token || !token->is_string() ||
|
||||
token->as_string().empty())
|
||||
return out;
|
||||
q.next_page_token = token->as_string();
|
||||
}
|
||||
out.truncated = true; // loud: the page cap stopped the walk while more pages existed
|
||||
return out;
|
||||
}
|
||||
|
||||
Json Voices::get(const std::string& id, bool with_settings) {
|
||||
check_id("voice", id);
|
||||
Query q;
|
||||
if (with_settings) q.add_bool("with_settings", true);
|
||||
return call(C::kVoiceGet, {id}, nullptr, q);
|
||||
}
|
||||
|
||||
Json Voices::settings(const std::string& id) {
|
||||
check_id("voice", id);
|
||||
return call(C::kVoiceSettings, {id});
|
||||
}
|
||||
|
||||
Json Voices::default_settings() { return call(C::kVoiceSettingsDefault); }
|
||||
|
||||
Json Voices::shared_library(const Json& query_object) {
|
||||
return call(C::kSharedVoices, {}, nullptr, query_from(C::kSharedVoices, query_object));
|
||||
}
|
||||
|
||||
Json Voices::design_build(const VoiceDesign& d) const {
|
||||
if (d.voice_description.size() < 20 || d.voice_description.size() > 1000)
|
||||
bad("voice_description must be 20..1000 characters");
|
||||
Json o = Json::object();
|
||||
o.set("voice_description", Json(d.voice_description));
|
||||
if (d.auto_generate_text) {
|
||||
if (!d.text.empty()) bad("give text or auto_generate_text, not both");
|
||||
o.set("auto_generate_text", Json(true));
|
||||
} else if (!d.text.empty()) {
|
||||
if (d.text.size() < 100 || d.text.size() > 1000) bad("text must be 100..1000 characters");
|
||||
o.set("text", Json(d.text));
|
||||
}
|
||||
if (!d.model_id.empty()) {
|
||||
if (d.model_id != "eleven_multilingual_ttv_v2" && d.model_id != "eleven_ttv_v3")
|
||||
bad("model_id must be eleven_multilingual_ttv_v2 or eleven_ttv_v3");
|
||||
o.set("model_id", Json(d.model_id));
|
||||
}
|
||||
if (d.loudness > -2) { require_range("loudness", d.loudness, -1, 1); o.set("loudness", Json(d.loudness)); }
|
||||
if (d.seed >= 0) {
|
||||
if (d.seed > 2147483647L) bad("seed must be 0..2147483647");
|
||||
o.set("seed", Json(d.seed));
|
||||
}
|
||||
if (d.guidance_scale >= 0) { require_range("guidance_scale", d.guidance_scale, 0, 100); o.set("guidance_scale", Json(d.guidance_scale)); }
|
||||
if (d.quality > -2) { require_range("quality", d.quality, -1, 1); o.set("quality", Json(d.quality)); }
|
||||
if (d.stream_previews >= 0) o.set("stream_previews", Json(d.stream_previews == 1));
|
||||
if (d.should_enhance >= 0) o.set("should_enhance", Json(d.should_enhance == 1));
|
||||
return o;
|
||||
}
|
||||
|
||||
Json Voices::design(const VoiceDesign& d) {
|
||||
Json body = design_build(d);
|
||||
Query q;
|
||||
if (!d.output_format.empty()) {
|
||||
if (!valid_output_format(d.output_format)) bad("unknown output_format: " + d.output_format);
|
||||
q.add("output_format", d.output_format);
|
||||
}
|
||||
return call(C::kVoiceDesign, {}, &body, q);
|
||||
}
|
||||
|
||||
AudioBlob Voices::design_preview_audio(const std::string& id) {
|
||||
check_id("generated voice", id);
|
||||
llm::RawResponse raw = call_raw(C::kVoiceDesignPreview, {id});
|
||||
return AudioBlob{raw.body, raw.content_type};
|
||||
}
|
||||
|
||||
Json Voices::save_design(const std::string& name, const std::string& description,
|
||||
const std::string& generated_voice_id, const Json& labels, bool confirm) {
|
||||
require_text("voice_name", name, 200);
|
||||
if (description.size() < 20 || description.size() > 1000) bad("voice_description must be 20..1000 characters");
|
||||
check_id("generated voice", generated_voice_id);
|
||||
require_confirm("saving a voice", confirm);
|
||||
Json o = Json::object();
|
||||
o.set("voice_name", Json(name));
|
||||
o.set("voice_description", Json(description));
|
||||
o.set("generated_voice_id", Json(generated_voice_id));
|
||||
if (labels.is_object()) o.set("labels", labels);
|
||||
return call(C::kVoiceDesignSave, {}, &o);
|
||||
}
|
||||
|
||||
Json Voices::remove(const std::string& id, bool confirm) {
|
||||
check_id("voice", id);
|
||||
return call_destructive(C::kVoiceDelete, {id}, nullptr, Query(), confirm);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
#pragma once
|
||||
|
||||
// Voices: the read side (list, get, settings, shared library), voice design
|
||||
// (text description to voice previews, then save) and the destructive
|
||||
// delete. Cloning (add_voice, PVC) creates resources and is reached through
|
||||
// the call table with confirm, not through a typed shortcut.
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/elevenlabs/service.h"
|
||||
#include "vendors/audio/elevenlabs/tts.h"
|
||||
|
||||
namespace molo::audio::elevenlabs {
|
||||
|
||||
struct VoiceListQuery {
|
||||
int page_size = 0; // 0 unset, else 1..100
|
||||
std::string next_page_token;
|
||||
std::string search, sort, sort_direction; // sort: created_at_unix | name; direction asc | desc
|
||||
std::string voice_type; // personal community default workspace non-default non-community saved
|
||||
std::string category; // premade cloned generated professional
|
||||
int include_total_count = -1;
|
||||
};
|
||||
|
||||
struct VoiceListing {
|
||||
Json voices = Json::array();
|
||||
bool truncated = false; // page cap reached while a next page existed
|
||||
std::size_t pages = 0;
|
||||
long total_count = -1;
|
||||
};
|
||||
|
||||
struct VoiceDesign {
|
||||
std::string voice_description; // 20..1000 characters
|
||||
std::string text; // "" with auto_generate_text, else 100..1000
|
||||
bool auto_generate_text = false;
|
||||
std::string model_id; // "", eleven_multilingual_ttv_v2, eleven_ttv_v3
|
||||
double loudness = -2; // -1..1, -2 unset
|
||||
long seed = -1; // 0..2147483647
|
||||
double guidance_scale = -1; // 0..100
|
||||
double quality = -2; // -1..1
|
||||
int stream_previews = -1;
|
||||
int should_enhance = -1;
|
||||
std::string output_format; // query
|
||||
};
|
||||
|
||||
class Voices : public ElevenService {
|
||||
public:
|
||||
using ElevenService::ElevenService;
|
||||
|
||||
Json list(bool show_legacy = false); // GET /v1/voices
|
||||
Json list_v2(const VoiceListQuery& query); // GET /v2/voices, one page
|
||||
VoiceListing list_all_v2(VoiceListQuery query, std::size_t max_pages);
|
||||
Json get(const std::string& voice_id, bool with_settings = false);
|
||||
Json settings(const std::string& voice_id);
|
||||
Json default_settings();
|
||||
Json shared_library(const Json& query_object); // GET /v1/shared-voices
|
||||
Json design_build(const VoiceDesign& design) const; // body
|
||||
Json design(const VoiceDesign& design); // POST /v1/text-to-voice/design
|
||||
AudioBlob design_preview_audio(const std::string& generated_voice_id);
|
||||
// Creates a voice from a design preview: a write, needs confirm.
|
||||
Json save_design(const std::string& name, const std::string& description,
|
||||
const std::string& generated_voice_id, const Json& labels, bool confirm);
|
||||
Json remove(const std::string& voice_id, bool confirm);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::elevenlabs
|
||||
Vendored
+153
@@ -0,0 +1,153 @@
|
||||
#include "vendors/audio/hosts.h"
|
||||
|
||||
#include "clients/transports/http_transport.h"
|
||||
#include "llm/config.h"
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
namespace {
|
||||
clients::ClientConfig stream_client_config() {
|
||||
clients::ClientConfig c;
|
||||
c.persist_cookies = false;
|
||||
return c;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
ByteStreamer::ByteStreamer(std::unique_ptr<clients::Transport> transport,
|
||||
llm::ApiClientConfig config, std::string key)
|
||||
: config_(std::move(config)),
|
||||
key_(std::move(key)),
|
||||
client_(std::move(transport), stream_client_config()) {}
|
||||
|
||||
ByteStreamer::Result ByteStreamer::stream(clients::Method method, const std::string& path,
|
||||
const Json* body, const ChunkSink& sink,
|
||||
const llm::Headers& extra) {
|
||||
if (!configured()) throw llm::NotConfiguredError(config_.vendor, config_.key_env);
|
||||
if (path.empty() || path.front() != '/' || path.find("..") != std::string::npos)
|
||||
throw llm::ApiError::invalid(config_.vendor, "bad request path");
|
||||
Result result;
|
||||
std::string probe; // leading bytes held back while they look like a JSON error
|
||||
bool probing = true;
|
||||
bool json_like = false;
|
||||
bool stopped = false;
|
||||
auto forward = [&](const std::string& chunk) -> bool {
|
||||
if (chunk.empty()) return true;
|
||||
if (result.bytes + chunk.size() > AudioCaps::kMaxStreamAudioBytes) {
|
||||
result.truncated = true;
|
||||
stopped = true;
|
||||
return false;
|
||||
}
|
||||
result.bytes += chunk.size();
|
||||
if (!sink(chunk)) {
|
||||
stopped = true;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
try {
|
||||
clients::ClientRequest req(method, clients::Url::parse(config_.base_url + path));
|
||||
if (config_.auth == llm::AuthStyle::kBearer) req.bearer(key_);
|
||||
else req.header(config_.auth_header, key_);
|
||||
for (const auto& [k, v] : config_.default_headers) req.header(k, v);
|
||||
for (const auto& [k, v] : extra) req.header(k, v);
|
||||
if (body) req.json(*body);
|
||||
client_.send_stream_blocking(std::move(req), [&](const std::string& chunk) -> bool {
|
||||
if (!probing) return forward(chunk);
|
||||
probe += chunk;
|
||||
if (!json_like) {
|
||||
std::size_t at = probe.find_first_not_of(" \t\r\n");
|
||||
if (at == std::string::npos) return true;
|
||||
json_like = probe[at] == '{';
|
||||
if (!json_like) {
|
||||
probing = false;
|
||||
std::string held;
|
||||
held.swap(probe);
|
||||
return forward(held);
|
||||
}
|
||||
}
|
||||
if (probe.size() > AudioCaps::kErrorProbeBytes) { // too big to be an error document
|
||||
probing = false;
|
||||
std::string held;
|
||||
held.swap(probe);
|
||||
return forward(held);
|
||||
}
|
||||
return true;
|
||||
});
|
||||
if (probing && !probe.empty()) {
|
||||
// The whole body fit the probe window. A JSON object with an error shape is the vendor's error.
|
||||
try {
|
||||
Json doc = Json::parse(probe);
|
||||
if (doc.is_object() && (doc.find("detail") || doc.find("error") || doc.find("message") ||
|
||||
doc.find("err_code") || doc.find("err_msg"))) {
|
||||
std::string msg = probe.substr(0, 300);
|
||||
throw llm::ApiError(config_.vendor, 400, "stream_error", msg);
|
||||
}
|
||||
} catch (const clients::body::JsonError&) {
|
||||
}
|
||||
forward(probe);
|
||||
}
|
||||
} catch (const llm::ApiError&) {
|
||||
throw;
|
||||
} catch (const std::exception& e) {
|
||||
if (!stopped) throw llm::ApiError(config_.vendor, 502, "UNAVAILABLE", e.what());
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
Hosts& Hosts::add(const std::string& name, AudioHost host) {
|
||||
if (hosts_.empty()) first_ = name;
|
||||
hosts_[name] = std::move(host);
|
||||
return *this;
|
||||
}
|
||||
|
||||
const AudioHost* Hosts::find(const std::string& name) const {
|
||||
auto it = hosts_.find(name);
|
||||
return it == hosts_.end() ? nullptr : &it->second;
|
||||
}
|
||||
|
||||
const AudioHost& Hosts::get(const std::string& name, const std::string& vendor) const {
|
||||
const AudioHost* h = find(name);
|
||||
if (!h) throw llm::ApiError::invalid(vendor, "no such host: " + name);
|
||||
return *h;
|
||||
}
|
||||
|
||||
bool Hosts::any_configured() const {
|
||||
for (const auto& [name, h] : hosts_)
|
||||
if (h.configured()) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
std::string allowlisted_base(const char* env_name, const std::string& fallback,
|
||||
std::initializer_list<const char*> allowed) {
|
||||
std::string v = llm::Config::get(env_name);
|
||||
while (!v.empty() && v.back() == '/') v.pop_back();
|
||||
for (const char* a : allowed)
|
||||
if (v == a) return v;
|
||||
return fallback;
|
||||
}
|
||||
|
||||
std::string to_ws_url(const std::string& http_url) {
|
||||
if (http_url.rfind("https://", 0) == 0) return "wss://" + http_url.substr(8);
|
||||
if (http_url.rfind("http://", 0) == 0) return "ws://" + http_url.substr(7);
|
||||
return http_url;
|
||||
}
|
||||
|
||||
AudioHost HostFactory::from_env(llm::ApiClientConfig config, const std::string& key_prefix) {
|
||||
std::string key = llm::Config::get(config.key_env.c_str());
|
||||
if (!key.empty()) key = key_prefix + key;
|
||||
return with_transports(std::make_unique<clients::transports::HttpTransport>(),
|
||||
std::make_unique<clients::transports::HttpTransport>(), std::move(config),
|
||||
key);
|
||||
}
|
||||
|
||||
AudioHost HostFactory::with_transports(std::unique_ptr<clients::Transport> api_transport,
|
||||
std::unique_ptr<clients::Transport> stream_transport,
|
||||
llm::ApiClientConfig config, const std::string& key) {
|
||||
AudioHost host;
|
||||
host.api = std::make_shared<llm::ApiClient>(std::move(api_transport), config, key);
|
||||
host.streamer = std::make_shared<ByteStreamer>(std::move(stream_transport), config, key);
|
||||
return host;
|
||||
}
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+91
@@ -0,0 +1,91 @@
|
||||
#pragma once
|
||||
|
||||
// One provider talks to one or more hosts (api, gateway, streaming, ...).
|
||||
// Each host has an ApiClient for buffered JSON / bytes calls and a
|
||||
// ByteStreamer for incremental bodies (streamed audio). Both authenticate the
|
||||
// same way and refuse to send without a key.
|
||||
|
||||
#include <functional>
|
||||
#include <initializer_list>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "clients/client.h"
|
||||
#include "llm/api_client.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
using Json = clients::body::Json;
|
||||
using ChunkSink = std::function<bool(const std::string& chunk)>;
|
||||
|
||||
// Incremental POST/GET bodies. The shared client reports no status for a
|
||||
// stream, so a small JSON document where audio was expected is treated as the
|
||||
// vendor's error (same rule llm::SseDecoder applies to SSE streams).
|
||||
class ByteStreamer {
|
||||
public:
|
||||
ByteStreamer(std::unique_ptr<clients::Transport> transport, llm::ApiClientConfig config,
|
||||
std::string key);
|
||||
bool configured() const { return !key_.empty(); }
|
||||
const llm::ApiClientConfig& config() const { return config_; }
|
||||
|
||||
// Streams the response body to `sink` (return false to stop). Total bytes
|
||||
// are capped by AudioCaps::kMaxStreamAudioBytes: past it the stream stops
|
||||
// and `truncated` is set. Throws llm::ApiError for a JSON error body.
|
||||
struct Result {
|
||||
std::size_t bytes = 0;
|
||||
bool truncated = false;
|
||||
};
|
||||
Result stream(clients::Method method, const std::string& path, const Json* body,
|
||||
const ChunkSink& sink, const llm::Headers& extra = {});
|
||||
|
||||
private:
|
||||
llm::ApiClientConfig config_;
|
||||
std::string key_;
|
||||
clients::Client client_;
|
||||
};
|
||||
|
||||
struct AudioHost {
|
||||
std::shared_ptr<llm::ApiClient> api;
|
||||
std::shared_ptr<ByteStreamer> streamer;
|
||||
bool configured() const { return api && api->configured(); }
|
||||
};
|
||||
|
||||
class Hosts {
|
||||
public:
|
||||
Hosts() = default;
|
||||
Hosts& add(const std::string& name, AudioHost host);
|
||||
const AudioHost& get(const std::string& name, const std::string& vendor) const;
|
||||
const AudioHost* find(const std::string& name) const;
|
||||
// The first host added: what llm::Service uses as its default client.
|
||||
const AudioHost& first() const { return hosts_.at(first_); }
|
||||
bool empty() const { return hosts_.empty(); }
|
||||
bool any_configured() const;
|
||||
const std::map<std::string, AudioHost>& all() const { return hosts_; }
|
||||
|
||||
private:
|
||||
std::map<std::string, AudioHost> hosts_;
|
||||
std::string first_;
|
||||
};
|
||||
|
||||
// Base URL override from the environment/config, accepted only when it is
|
||||
// exactly one of `allowed` (regional endpoints); anything else falls back to
|
||||
// `fallback` so a stray value can never redirect a credential to a new host.
|
||||
std::string allowlisted_base(const char* env_name, const std::string& fallback,
|
||||
std::initializer_list<const char*> allowed);
|
||||
// "https://h/x" -> "wss://h/x", "http://" -> "ws://".
|
||||
std::string to_ws_url(const std::string& http_url);
|
||||
|
||||
class HostFactory {
|
||||
public:
|
||||
// Real HTTPS transports. The key comes from llm::Config::get(config.key_env);
|
||||
// `key_prefix` is prepended when the key is non-empty (Deepgram "Token ").
|
||||
static AudioHost from_env(llm::ApiClientConfig config, const std::string& key_prefix = "");
|
||||
// Injected transports (tests): one transport per client.
|
||||
static AudioHost with_transports(std::unique_ptr<clients::Transport> api_transport,
|
||||
std::unique_ptr<clients::Transport> stream_transport,
|
||||
llm::ApiClientConfig config, const std::string& key);
|
||||
};
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+68
@@ -0,0 +1,68 @@
|
||||
#include "vendors/audio/hume/calls.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
// One row per REST call. tools/check_spec.py parses the block between the
|
||||
// markers: {id, method, path, host, body, fields, required, query, response, source}.
|
||||
// CALLS-BEGIN
|
||||
static const CallSpec kRows[] = {
|
||||
{"voices_list", "GET", "/v0/tts/voices", "api", "-", "", "", "provider,page_number,page_size,ascending_order,filter_tag", "json", "openapi-tts:voices_list"},
|
||||
{"voices_create", "POST", "/v0/tts/voices", "api", "json", "generation_id,name", "generation_id,name", "", "json", "openapi-tts:voices_create"},
|
||||
{"voices_delete", "DELETE", "/v0/tts/voices", "api", "-", "", "", "name", "json", "openapi-tts:voices_delete"},
|
||||
{"controlPlane_send", "POST", "/v0/evi/chat/{chat_id}/send", "api", "json", "audio,builtin_tools,context,custom_session_id,language_model_api_key,metadata,system_prompt,tools,type,variables,voice_id,text,content,tool_call_id,tool_name,tool_type,code,error,level", "", "", "json", "openapi-evi:controlPlane_send"},
|
||||
{"chatGroups_list-chat-groups", "GET", "/v0/evi/chat_groups", "api", "-", "", "", "page_number,page_size,ascending_order,config_id", "json", "openapi-evi:chatGroups_list-chat-groups"},
|
||||
{"chatGroups_get-chat-group", "GET", "/v0/evi/chat_groups/{id}", "api", "-", "", "", "status,page_size,page_number,ascending_order", "json", "openapi-evi:chatGroups_get-chat-group"},
|
||||
{"chatGroups_get-audio", "GET", "/v0/evi/chat_groups/{id}/audio", "api", "-", "", "", "page_number,page_size,ascending_order", "json", "openapi-evi:chatGroups_get-audio"},
|
||||
{"chatGroups_list-chat-group-events", "GET", "/v0/evi/chat_groups/{id}/events", "api", "-", "", "", "page_size,page_number,ascending_order", "json", "openapi-evi:chatGroups_list-chat-group-events"},
|
||||
{"chats_list-chats", "GET", "/v0/evi/chats", "api", "-", "", "", "page_number,page_size,ascending_order,config_id,status", "json", "openapi-evi:chats_list-chats"},
|
||||
{"chats_list-chat-events", "GET", "/v0/evi/chats/{id}", "api", "-", "", "", "page_size,page_number,ascending_order", "json", "openapi-evi:chats_list-chat-events"},
|
||||
{"chats_get-audio", "GET", "/v0/evi/chats/{id}/audio", "api", "-", "", "", "", "json", "openapi-evi:chats_get-audio"},
|
||||
{"configs_list-configs", "GET", "/v0/evi/configs", "api", "-", "", "", "page_number,page_size,restrict_to_most_recent,name", "json", "openapi-evi:configs_list-configs"},
|
||||
{"configs_create-config", "POST", "/v0/evi/configs", "api", "json", "builtin_tools,ellm_model,event_messages,evi_version,interruption,language_model,name,nudges,prompt,timeouts,tools,turn_detection,version_description,voice,webhooks", "evi_version,name", "", "json", "openapi-evi:configs_create-config"},
|
||||
{"configs_list-config-versions", "GET", "/v0/evi/configs/{id}", "api", "-", "", "", "page_number,page_size,restrict_to_most_recent", "json", "openapi-evi:configs_list-config-versions"},
|
||||
{"configs_create-config-version", "POST", "/v0/evi/configs/{id}", "api", "json", "builtin_tools,ellm_model,event_messages,evi_version,interruption,language_model,nudges,prompt,timeouts,tools,turn_detection,version_description,voice,webhooks", "evi_version", "", "json", "openapi-evi:configs_create-config-version"},
|
||||
{"configs_update-config-name", "PATCH", "/v0/evi/configs/{id}", "api", "json", "name", "name", "", "json", "openapi-evi:configs_update-config-name"},
|
||||
{"configs_delete-config", "DELETE", "/v0/evi/configs/{id}", "api", "-", "", "", "", "json", "openapi-evi:configs_delete-config"},
|
||||
{"configs_get-config-version", "GET", "/v0/evi/configs/{id}/version/{version}", "api", "-", "", "", "", "json", "openapi-evi:configs_get-config-version"},
|
||||
{"configs_update-config-description", "PATCH", "/v0/evi/configs/{id}/version/{version}", "api", "json", "version_description", "", "", "json", "openapi-evi:configs_update-config-description"},
|
||||
{"configs_delete-config-version", "DELETE", "/v0/evi/configs/{id}/version/{version}", "api", "-", "", "", "", "json", "openapi-evi:configs_delete-config-version"},
|
||||
{"prompts_list-prompts", "GET", "/v0/evi/prompts", "api", "-", "", "", "page_number,page_size,restrict_to_most_recent,name", "json", "openapi-evi:prompts_list-prompts"},
|
||||
{"prompts_create-prompt", "POST", "/v0/evi/prompts", "api", "json", "name,prompt_expansion,text,version_description", "name,text", "", "json", "openapi-evi:prompts_create-prompt"},
|
||||
{"prompts_list-prompt-versions", "GET", "/v0/evi/prompts/{id}", "api", "-", "", "", "page_number,page_size,restrict_to_most_recent", "json", "openapi-evi:prompts_list-prompt-versions"},
|
||||
{"prompts_create-prompt-version", "POST", "/v0/evi/prompts/{id}", "api", "json", "prompt_expansion,text,version_description", "text", "", "json", "openapi-evi:prompts_create-prompt-version"},
|
||||
{"prompts_update-prompt-name", "PATCH", "/v0/evi/prompts/{id}", "api", "json", "name", "name", "", "json", "openapi-evi:prompts_update-prompt-name"},
|
||||
{"prompts_delete-prompt", "DELETE", "/v0/evi/prompts/{id}", "api", "-", "", "", "", "json", "openapi-evi:prompts_delete-prompt"},
|
||||
{"prompts_get-prompt-version", "GET", "/v0/evi/prompts/{id}/version/{version}", "api", "-", "", "", "", "json", "openapi-evi:prompts_get-prompt-version"},
|
||||
{"prompts_update-prompt-description", "PATCH", "/v0/evi/prompts/{id}/version/{version}", "api", "json", "version_description", "", "", "json", "openapi-evi:prompts_update-prompt-description"},
|
||||
{"prompts_delete-prompt-version", "DELETE", "/v0/evi/prompts/{id}/version/{version}", "api", "-", "", "", "", "json", "openapi-evi:prompts_delete-prompt-version"},
|
||||
{"tools_list-tools", "GET", "/v0/evi/tools", "api", "-", "", "", "page_number,page_size,restrict_to_most_recent,name", "json", "openapi-evi:tools_list-tools"},
|
||||
{"tools_create-tool", "POST", "/v0/evi/tools", "api", "json", "description,fallback_content,name,parameters,version_description", "name,parameters", "", "json", "openapi-evi:tools_create-tool"},
|
||||
{"tools_list-tool-versions", "GET", "/v0/evi/tools/{id}", "api", "-", "", "", "page_number,page_size,restrict_to_most_recent", "json", "openapi-evi:tools_list-tool-versions"},
|
||||
{"tools_create-tool-version", "POST", "/v0/evi/tools/{id}", "api", "json", "description,fallback_content,parameters,version_description", "parameters", "", "json", "openapi-evi:tools_create-tool-version"},
|
||||
{"tools_update-tool-name", "PATCH", "/v0/evi/tools/{id}", "api", "json", "name", "name", "", "json", "openapi-evi:tools_update-tool-name"},
|
||||
{"tools_delete-tool", "DELETE", "/v0/evi/tools/{id}", "api", "-", "", "", "", "json", "openapi-evi:tools_delete-tool"},
|
||||
{"tools_get-tool-version", "GET", "/v0/evi/tools/{id}/version/{version}", "api", "-", "", "", "", "json", "openapi-evi:tools_get-tool-version"},
|
||||
{"tools_update-tool-description", "PATCH", "/v0/evi/tools/{id}/version/{version}", "api", "json", "version_description", "", "", "json", "openapi-evi:tools_update-tool-description"},
|
||||
{"tools_delete-tool-version", "DELETE", "/v0/evi/tools/{id}/version/{version}", "api", "-", "", "", "", "json", "openapi-evi:tools_delete-tool-version"},
|
||||
{"audio_measure", "POST", "/v1/expression/audio/file", "expression", "form", "config,file", "file", "", "json", "openapi-expression:audio_measure"},
|
||||
{"runs_list", "GET", "/v1/expression/run", "expression", "-", "", "", "page_size,cursor,user_id,status,mode,endpoint,started_before,started_after", "json", "openapi-expression:runs_list"},
|
||||
{"runs_get", "GET", "/v1/expression/run/{run_id}", "expression", "-", "", "", "", "json", "openapi-expression:runs_get"},
|
||||
{"runs_list-events", "GET", "/v1/expression/run/{run_id}/events", "expression", "-", "", "", "page_size,cursor", "json", "openapi-expression:runs_list-events"},
|
||||
{"video_measure", "POST", "/v1/expression/video/file", "expression", "form", "config,file", "file", "", "json", "openapi-expression:video_measure"},
|
||||
{"tts_synthesize_json", "POST", "/v0/tts", "api", "json", "utterances,context,format,include_timestamp_types,num_generations,split_utterances,strip_headers,temperature,version,instant_mode", "utterances", "", "json", "sdk:hume-python-sdk src/hume/tts/raw_client.py synthesize_json"},
|
||||
{"tts_synthesize_file", "POST", "/v0/tts/file", "api", "json", "utterances,context,format,include_timestamp_types,num_generations,split_utterances,strip_headers,temperature,version,instant_mode", "utterances", "", "bytes", "sdk:hume-python-sdk src/hume/tts/raw_client.py synthesize_file"},
|
||||
{"tts_synthesize_json_streaming", "POST", "/v0/tts/stream/json", "api", "json", "utterances,context,format,include_timestamp_types,num_generations,split_utterances,strip_headers,temperature,version,instant_mode", "utterances", "", "bytes", "sdk:hume-python-sdk src/hume/tts/raw_client.py synthesize_json_streaming"},
|
||||
{"tts_synthesize_file_streaming", "POST", "/v0/tts/stream/file", "api", "json", "utterances,context,format,include_timestamp_types,num_generations,split_utterances,strip_headers,temperature,version,instant_mode", "utterances", "", "bytes", "sdk:hume-python-sdk src/hume/tts/raw_client.py synthesize_file_streaming"},
|
||||
{"tts_convert_voice_json", "POST", "/v0/tts/voice_conversion/json", "api", "form", "strip_headers,audio,context,voice,format,include_timestamp_types", "", "", "bytes", "sdk:hume-python-sdk src/hume/tts/raw_client.py convert_voice_json"},
|
||||
{"tts_convert_voice_file", "POST", "/v0/tts/voice_conversion/file", "api", "form", "audio,strip_headers,context,voice,format,include_timestamp_types", "audio", "", "bytes", "sdk:hume-python-sdk src/hume/tts/raw_client.py convert_voice_file"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
static_assert(sizeof(kRows) / sizeof(kRows[0]) == 49, "hume call table size");
|
||||
|
||||
const CallTable& calls() {
|
||||
static const CallTable table("hume", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+45
@@ -0,0 +1,45 @@
|
||||
#pragma once
|
||||
|
||||
// The Hume call table: the three OpenAPI documents on dev.hume.ai
|
||||
// (text-to-speech-tts.json: voices; speech-to-speech-evi.json: EVI
|
||||
// configs, prompts, tools, chats, chat groups; expression-measurement-api.json:
|
||||
// batch runs) plus the TTS synthesis calls, which the published OpenAPI
|
||||
// document leaves out and which are taken from the official Python SDK
|
||||
// (src/hume/tts/raw_client.py). Hosts: api (api.hume.ai) and expression
|
||||
// (api.cloud.hume.ai).
|
||||
|
||||
#include "vendors/audio/call_spec.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
const CallTable& calls();
|
||||
|
||||
struct HumeCalls {
|
||||
static constexpr const char* kTts = "tts_synthesize_json";
|
||||
static constexpr const char* kTtsFile = "tts_synthesize_file";
|
||||
static constexpr const char* kTtsJsonStream = "tts_synthesize_json_streaming";
|
||||
static constexpr const char* kTtsFileStream = "tts_synthesize_file_streaming";
|
||||
static constexpr const char* kVoiceConvertJson = "tts_convert_voice_json";
|
||||
static constexpr const char* kVoiceConvertFile = "tts_convert_voice_file";
|
||||
static constexpr const char* kVoices = "voices_list";
|
||||
static constexpr const char* kVoiceCreate = "voices_create";
|
||||
static constexpr const char* kVoiceDelete = "voices_delete";
|
||||
static constexpr const char* kChats = "chats_list-chats";
|
||||
static constexpr const char* kChatEvents = "chats_list-chat-events";
|
||||
static constexpr const char* kChatAudio = "chats_get-audio";
|
||||
static constexpr const char* kChatGroups = "chatGroups_list-chat-groups";
|
||||
static constexpr const char* kChatGroup = "chatGroups_get-chat-group";
|
||||
static constexpr const char* kConfigs = "configs_list-configs";
|
||||
static constexpr const char* kConfigVersions = "configs_list-config-versions";
|
||||
static constexpr const char* kConfigVersion = "configs_get-config-version";
|
||||
static constexpr const char* kPrompts = "prompts_list-prompts";
|
||||
static constexpr const char* kTools = "tools_list-tools";
|
||||
static constexpr const char* kControlSend = "controlPlane_send";
|
||||
static constexpr const char* kRuns = "runs_list";
|
||||
static constexpr const char* kRun = "runs_get";
|
||||
static constexpr const char* kRunEvents = "runs_list-events";
|
||||
static constexpr const char* kAudioMeasure = "audio_measure";
|
||||
static constexpr const char* kVideoMeasure = "video_measure";
|
||||
};
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+35
@@ -0,0 +1,35 @@
|
||||
#include "vendors/audio/hume/client.h"
|
||||
|
||||
#include "llm/config.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
llm::ApiClientConfig HumeEndpoints::config(const std::string& base) {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = kVendor;
|
||||
c.base_url = base;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = kKeyHeader;
|
||||
return c;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> HumeEndpoints::from_env() {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", HostFactory::from_env(config(kApiBase)));
|
||||
hosts->add("expression", HostFactory::from_env(config(kExpressionBase)));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Hosts> HumeEndpoints::with(AudioHost api, AudioHost expression) {
|
||||
auto hosts = std::make_shared<Hosts>();
|
||||
hosts->add("api", std::move(api));
|
||||
hosts->add("expression", std::move(expression));
|
||||
return hosts;
|
||||
}
|
||||
|
||||
std::string HumeEndpoints::key() { return llm::Config::get(kKeyEnv); }
|
||||
|
||||
AudioSocket::Headers HumeEndpoints::ws_headers(const std::string& key) { return {{kKeyHeader, key}}; }
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+33
@@ -0,0 +1,33 @@
|
||||
#pragma once
|
||||
|
||||
// Hume endpoints and credentials.
|
||||
// api https://api.hume.ai header X-Hume-Api-Key: $HUME_API_KEY
|
||||
// expression https://api.cloud.hume.ai same key
|
||||
// WebSocket wss://api.hume.ai/v0/tts, wss://api.hume.ai/v0/evi same header
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/hosts.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
struct HumeEndpoints {
|
||||
static constexpr const char* kVendor = "hume";
|
||||
static constexpr const char* kKeyEnv = "HUME_API_KEY";
|
||||
static constexpr const char* kKeyHeader = "X-Hume-Api-Key";
|
||||
static constexpr const char* kApiBase = "https://api.hume.ai";
|
||||
static constexpr const char* kExpressionBase = "https://api.cloud.hume.ai";
|
||||
static constexpr const char* kTtsWs = "wss://api.hume.ai/v0/tts";
|
||||
static constexpr const char* kEviWs = "wss://api.hume.ai/v0/evi";
|
||||
|
||||
static llm::ApiClientConfig config(const std::string& base);
|
||||
static std::shared_ptr<const Hosts> from_env();
|
||||
// Injected hosts (tests): "api" and "expression".
|
||||
static std::shared_ptr<const Hosts> with(AudioHost api, AudioHost expression);
|
||||
static std::string key();
|
||||
static AudioSocket::Headers ws_headers(const std::string& key);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+110
@@ -0,0 +1,110 @@
|
||||
#include "vendors/audio/hume/evi.h"
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
using C = HumeCalls;
|
||||
|
||||
namespace {
|
||||
void page_checks(const AudioService& svc, const Json& q) {
|
||||
if (!q.is_object()) return;
|
||||
const Json* s = q.find("page_size");
|
||||
if (s && s->is_number() && (s->as_int() < 1 || s->as_int() > HumeCaps::kMaxPageSize)) svc.bad("page_size must be 1..100");
|
||||
const Json* p = q.find("page_number");
|
||||
if (p && p->is_number() && p->as_int() < 0) svc.bad("page_number must not be negative");
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Json HumeEvi::chats(const Json& q) {
|
||||
page_checks(*this, q);
|
||||
return call(C::kChats, {}, nullptr, query_from(C::kChats, q));
|
||||
}
|
||||
|
||||
Json HumeEvi::chat_events(const std::string& id, const Json& q) {
|
||||
check_id("chat", id);
|
||||
page_checks(*this, q);
|
||||
return call(C::kChatEvents, {id}, nullptr, query_from(C::kChatEvents, q));
|
||||
}
|
||||
|
||||
Json HumeEvi::chat_audio(const std::string& id) {
|
||||
check_id("chat", id);
|
||||
return call(C::kChatAudio, {id});
|
||||
}
|
||||
|
||||
Json HumeEvi::chat_groups(const Json& q) {
|
||||
page_checks(*this, q);
|
||||
return call(C::kChatGroups, {}, nullptr, query_from(C::kChatGroups, q));
|
||||
}
|
||||
|
||||
Json HumeEvi::chat_group(const std::string& id, const Json& q) {
|
||||
check_id("chat group", id);
|
||||
page_checks(*this, q);
|
||||
return call(C::kChatGroup, {id}, nullptr, query_from(C::kChatGroup, q));
|
||||
}
|
||||
|
||||
Json HumeEvi::configs(const Json& q) {
|
||||
page_checks(*this, q);
|
||||
return call(C::kConfigs, {}, nullptr, query_from(C::kConfigs, q));
|
||||
}
|
||||
|
||||
Json HumeEvi::config_versions(const std::string& id, const Json& q) {
|
||||
check_id("config", id);
|
||||
page_checks(*this, q);
|
||||
return call(C::kConfigVersions, {id}, nullptr, query_from(C::kConfigVersions, q));
|
||||
}
|
||||
|
||||
Json HumeEvi::config_version(const std::string& id, int version) {
|
||||
check_id("config", id);
|
||||
if (version < 0 || version > 100000) bad("version must be 0..100000");
|
||||
return call(C::kConfigVersion, {id, std::to_string(version)});
|
||||
}
|
||||
|
||||
Json HumeEvi::prompts(const Json& q) {
|
||||
page_checks(*this, q);
|
||||
return call(C::kPrompts, {}, nullptr, query_from(C::kPrompts, q));
|
||||
}
|
||||
|
||||
Json HumeEvi::tools(const Json& q) {
|
||||
page_checks(*this, q);
|
||||
return call(C::kTools, {}, nullptr, query_from(C::kTools, q));
|
||||
}
|
||||
|
||||
Json HumeEvi::control_send(const std::string& id, const Json& event, bool confirm) {
|
||||
check_id("chat", id);
|
||||
check_body(table_.get(C::kControlSend), &event);
|
||||
require_confirm("sending a control message to a live chat", confirm);
|
||||
return call(C::kControlSend, {id}, &event);
|
||||
}
|
||||
|
||||
Json HumeExpression::runs(const Json& q) { return call(C::kRuns, {}, nullptr, query_from(C::kRuns, q)); }
|
||||
|
||||
Json HumeExpression::run(const std::string& id) {
|
||||
check_id("run", id);
|
||||
return call(C::kRun, {id});
|
||||
}
|
||||
|
||||
Json HumeExpression::run_events(const std::string& id, const Json& q) {
|
||||
check_id("run", id);
|
||||
return call(C::kRunEvents, {id}, nullptr, query_from(C::kRunEvents, q));
|
||||
}
|
||||
|
||||
Json HumeExpression::measure_audio(const std::string& audio, const std::string& filename,
|
||||
const std::string& content_type, const Json& config, bool confirm) {
|
||||
if (audio.empty() || audio.size() > AudioCaps::kMaxAudioInBytes) bad("audio must be 1 byte..128 MiB");
|
||||
if (!config.is_object()) bad("config must be a JSON object");
|
||||
if (const Json* ms = config.find("measurement_timer_ms")) {
|
||||
if (!ms->is_number() || ms->as_int() < 3000 || ms->as_int() > 10000) bad("measurement_timer_ms must be 3000..10000");
|
||||
}
|
||||
for (const auto& [k, v] : config.as_object()) {
|
||||
(void)v;
|
||||
if (k != "measurement_timer_ms") bad("unknown config field '" + k + "' (AudioFileConfig has measurement_timer_ms)");
|
||||
}
|
||||
require_confirm("starting a measurement run", confirm);
|
||||
clients::body::MultipartBody form;
|
||||
form.add_text("config", config.dump());
|
||||
form.add_file("file", filename.empty() ? "audio" : filename, content_type, audio);
|
||||
return call_form(C::kAudioMeasure, {}, form);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+44
@@ -0,0 +1,44 @@
|
||||
#pragma once
|
||||
|
||||
// Empathic Voice Interface (EVI) management reads (chats, chat groups,
|
||||
// configs, prompts, tools) and Expression Measurement batch runs. Creating,
|
||||
// updating and deleting configs, prompts and tools is in the call table and
|
||||
// reached through call() / call_destructive().
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vendors/audio/hume/tts.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
class HumeEvi : public HumeService {
|
||||
public:
|
||||
using HumeService::HumeService;
|
||||
Json chats(const Json& query_object = Json());
|
||||
Json chat_events(const std::string& chat_id, const Json& query_object = Json());
|
||||
Json chat_audio(const std::string& chat_id);
|
||||
Json chat_groups(const Json& query_object = Json());
|
||||
Json chat_group(const std::string& group_id, const Json& query_object = Json());
|
||||
Json configs(const Json& query_object = Json());
|
||||
Json config_versions(const std::string& config_id, const Json& query_object = Json());
|
||||
Json config_version(const std::string& config_id, int version);
|
||||
Json prompts(const Json& query_object = Json());
|
||||
Json tools(const Json& query_object = Json());
|
||||
// Sends a control message (pause / resume assistant, session settings) to a live chat: confirm required.
|
||||
Json control_send(const std::string& chat_id, const Json& event, bool confirm);
|
||||
};
|
||||
|
||||
class HumeExpression : public HumeService {
|
||||
public:
|
||||
using HumeService::HumeService;
|
||||
Json runs(const Json& query_object = Json());
|
||||
Json run(const std::string& run_id);
|
||||
Json run_events(const std::string& run_id, const Json& query_object = Json());
|
||||
// Starts an audio measurement run (billed). The audio is 16-bit PCM, 16 kHz, mono: a WAV file or
|
||||
// headerless little-endian samples (the OpenAPI text). `config` is the AudioFileConfig object,
|
||||
// for example {"measurement_timer_ms":3000}, sent as the JSON `config` part.
|
||||
Json measure_audio(const std::string& audio, const std::string& filename, const std::string& content_type,
|
||||
const Json& config, bool confirm);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+252
@@ -0,0 +1,252 @@
|
||||
#include "vendors/audio/hume/sockets.h"
|
||||
|
||||
#include "security/crypto/Base64.h"
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("hume", why); }
|
||||
|
||||
std::string str_at(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
|
||||
Json typed(const char* type) {
|
||||
Json m = Json::object();
|
||||
m.set("type", Json(type));
|
||||
return m;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Vocabulary HumeTtsStream::vocabulary() {
|
||||
return Vocabulary("type", {}, {"audio", "timestamp", "error"});
|
||||
}
|
||||
|
||||
HumeTtsStream::HumeTtsStream(std::string api_key, HumeStreamOptions o) {
|
||||
Query q;
|
||||
q.add_if("format_type", o.format_type);
|
||||
if (!o.format_type.empty() && o.format_type != "mp3" && o.format_type != "wav" && o.format_type != "pcm")
|
||||
invalid("format_type must be mp3, wav or pcm");
|
||||
if (!o.context_generation_id.empty()) {
|
||||
if (!llm::Service::valid_id(o.context_generation_id)) invalid("bad generation id");
|
||||
q.add("context_generation_id", o.context_generation_id);
|
||||
}
|
||||
for (const auto& t : o.include_timestamp_types) {
|
||||
if (t != "word" && t != "phoneme") invalid("timestamp type must be word or phoneme");
|
||||
q.add("include_timestamp_types", t);
|
||||
}
|
||||
if (o.instant_mode >= 0) q.add_bool("instant_mode", o.instant_mode == 1);
|
||||
if (o.no_binary >= 0) q.add_bool("no_binary", o.no_binary == 1);
|
||||
if (o.strip_headers >= 0) q.add_bool("strip_headers", o.strip_headers == 1);
|
||||
if (!o.version.empty()) {
|
||||
if (o.version != "1" && o.version != "2") invalid("version must be 1 or 2");
|
||||
q.add("version", o.version);
|
||||
}
|
||||
std::string base = o.url.empty() ? std::string(HumeEndpoints::kTtsWs) + "/stream/input" : o.url;
|
||||
socket_ = std::make_unique<AudioSocket>("hume", base + q.str(), HumeEndpoints::ws_headers(api_key), vocabulary(),
|
||||
std::move(o.socket));
|
||||
}
|
||||
|
||||
void HumeTtsStream::send_text(const std::string& text, const std::string& voice_name, const std::string& description,
|
||||
double speed, double trailing_silence) {
|
||||
if (text.empty() || text.size() > HumeCaps::kMaxUtteranceChars) invalid("text must be 1..5000 characters");
|
||||
if (text.find('\0') != std::string::npos) invalid("text has a NUL byte");
|
||||
Json m = Json::object();
|
||||
m.set("text", Json(text));
|
||||
if (!voice_name.empty()) {
|
||||
if (voice_name.size() > 128) invalid("voice name is too long");
|
||||
Json v = Json::object();
|
||||
v.set("name", Json(voice_name));
|
||||
m.set("voice", v);
|
||||
}
|
||||
if (!description.empty()) {
|
||||
if (description.size() > HumeCaps::kMaxDescriptionChars) invalid("description is too long");
|
||||
m.set("description", Json(description));
|
||||
}
|
||||
if (speed != 0) {
|
||||
if (speed < 0) invalid("speed must be positive");
|
||||
m.set("speed", Json(speed));
|
||||
}
|
||||
if (trailing_silence >= 0) m.set("trailing_silence", Json(trailing_silence));
|
||||
socket_->send_json(m, true);
|
||||
}
|
||||
|
||||
void HumeTtsStream::flush() {
|
||||
Json m = Json::object();
|
||||
m.set("flush", Json(true));
|
||||
socket_->send_json(m, true);
|
||||
}
|
||||
|
||||
void HumeTtsStream::finish() {
|
||||
Json m = Json::object();
|
||||
m.set("close", Json(true));
|
||||
socket_->send_json(m, true);
|
||||
}
|
||||
|
||||
HumeStreamResult HumeTtsStream::collect(int max_wait_ms) {
|
||||
HumeStreamResult out;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto ev = socket_->next(max_wait_ms);
|
||||
if (!ev) return out;
|
||||
if (ev->kind == SocketEvent::Kind::kClose) {
|
||||
out.closed = true;
|
||||
return out;
|
||||
}
|
||||
if (ev->kind == SocketEvent::Kind::kBinary) {
|
||||
if (out.audio.size() + ev->bytes.size() > AudioCaps::kMaxStreamAudioBytes) {
|
||||
out.truncated = true;
|
||||
return out;
|
||||
}
|
||||
out.audio += ev->bytes;
|
||||
++out.chunks;
|
||||
continue;
|
||||
}
|
||||
if (!ev->json_ok || !ev->json.is_object()) continue;
|
||||
if (ev->type == "audio") {
|
||||
std::string raw;
|
||||
if (!security::crypto::Base64::decode(str_at(ev->json, "audio"), raw))
|
||||
throw llm::ApiError("hume", 502, "BAD_RESPONSE", "audio chunk is not valid base64");
|
||||
if (out.audio.size() + raw.size() > AudioCaps::kMaxStreamAudioBytes) {
|
||||
out.truncated = true;
|
||||
return out;
|
||||
}
|
||||
out.audio += raw;
|
||||
++out.chunks;
|
||||
out.generation_id = str_at(ev->json, "generation_id");
|
||||
if (const Json* l = ev->json.find("is_last_chunk"); l && l->is_bool() && l->as_bool()) {
|
||||
out.last_chunk = true;
|
||||
return out;
|
||||
}
|
||||
} else if (ev->type == "timestamp") {
|
||||
out.timestamps.push_back(ev->json);
|
||||
} else if (ev->type == "error" || ev->json.find("error")) {
|
||||
out.error = str_at(ev->json, "message");
|
||||
if (out.error.empty()) out.error = str_at(ev->json, "error");
|
||||
if (out.error.empty()) out.error = "stream error";
|
||||
return out;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
Vocabulary EviChat::vocabulary() {
|
||||
return Vocabulary("type",
|
||||
{"audio_input", "session_settings", "user_input", "assistant_input", "tool_response",
|
||||
"tool_error", "pause_assistant_message", "resume_assistant_message"},
|
||||
{"assistant_end", "assistant_message", "assistant_prosody", "audio_output", "chat_metadata",
|
||||
"error", "user_interruption", "user_message", "tool_call", "tool_response", "tool_error",
|
||||
"session_settings"});
|
||||
}
|
||||
|
||||
EviChat::EviChat(std::string api_key, EviOptions o) {
|
||||
Query q;
|
||||
if (!o.config_id.empty()) {
|
||||
if (!llm::Service::valid_id(o.config_id)) invalid("bad config id");
|
||||
q.add("config_id", o.config_id);
|
||||
}
|
||||
if (o.config_version >= 0) q.add_int("config_version", o.config_version);
|
||||
if (!o.resumed_chat_group_id.empty()) {
|
||||
if (!llm::Service::valid_id(o.resumed_chat_group_id)) invalid("bad chat group id");
|
||||
q.add("resumed_chat_group_id", o.resumed_chat_group_id);
|
||||
}
|
||||
if (o.verbose_transcription >= 0) q.add_bool("verbose_transcription", o.verbose_transcription == 1);
|
||||
if (o.event_limit < 0 || o.event_limit > 10000) invalid("event_limit must be 0..10000");
|
||||
if (o.event_limit > 0) q.add_int("event_limit", o.event_limit);
|
||||
std::string base = o.url.empty() ? std::string(HumeEndpoints::kEviWs) + "/chat" : o.url;
|
||||
socket_ = std::make_unique<AudioSocket>("hume", base + q.str(), HumeEndpoints::ws_headers(api_key), vocabulary(),
|
||||
std::move(o.socket));
|
||||
}
|
||||
|
||||
void EviChat::send_session_settings(const Json& settings) {
|
||||
if (!settings.is_object()) invalid("settings must be a JSON object");
|
||||
Json m = settings;
|
||||
m.set("type", Json("session_settings"));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void EviChat::send_audio(const std::string& audio) {
|
||||
if (audio.empty() || audio.size() > 1024u * 1024u) invalid("audio chunk must be 1 byte..1 MiB");
|
||||
Json m = typed("audio_input");
|
||||
m.set("data", Json(security::crypto::Base64::encode(audio)));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void EviChat::send_user_text(const std::string& text) {
|
||||
if (text.empty() || text.size() > 20000) invalid("text must be 1..20000 characters");
|
||||
Json m = typed("user_input");
|
||||
m.set("text", Json(text));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void EviChat::send_assistant_text(const std::string& text) {
|
||||
if (text.empty() || text.size() > 20000) invalid("text must be 1..20000 characters");
|
||||
Json m = typed("assistant_input");
|
||||
m.set("text", Json(text));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void EviChat::send_tool_response(const std::string& id, const std::string& content) {
|
||||
if (!llm::Service::valid_id(id)) invalid("bad tool_call_id");
|
||||
if (content.size() > 200000) invalid("content is over 200000 characters");
|
||||
Json m = typed("tool_response");
|
||||
m.set("tool_call_id", Json(id));
|
||||
m.set("content", Json(content));
|
||||
socket_->send_json(m);
|
||||
}
|
||||
|
||||
void EviChat::pause_assistant() { socket_->send_json(typed("pause_assistant_message")); }
|
||||
void EviChat::resume_assistant() { socket_->send_json(typed("resume_assistant_message")); }
|
||||
|
||||
EviTurn EviChat::collect_turn(int max_wait_ms) {
|
||||
EviTurn turn;
|
||||
for (std::size_t n = 0; n < AudioCaps::kMaxEventsPerCollect; ++n) {
|
||||
auto ev = socket_->next(max_wait_ms);
|
||||
if (!ev) return turn;
|
||||
if (ev->kind == SocketEvent::Kind::kClose) {
|
||||
turn.closed = true;
|
||||
return turn;
|
||||
}
|
||||
if (!ev->json_ok || !ev->json.is_object()) continue;
|
||||
const std::string& t = ev->type;
|
||||
auto content = [&](const char* key) {
|
||||
const Json* m = ev->json.find("message");
|
||||
return m ? str_at(*m, key) : std::string();
|
||||
};
|
||||
if (t == "chat_metadata") {
|
||||
turn.chat_id = str_at(ev->json, "chat_id");
|
||||
turn.chat_group_id = str_at(ev->json, "chat_group_id");
|
||||
} else if (t == "user_message") {
|
||||
turn.user_messages.push_back(content("content"));
|
||||
} else if (t == "assistant_message") {
|
||||
turn.assistant_messages.push_back(content("content"));
|
||||
} else if (t == "audio_output") {
|
||||
std::string raw;
|
||||
if (!security::crypto::Base64::decode(str_at(ev->json, "data"), raw))
|
||||
throw llm::ApiError("hume", 502, "BAD_RESPONSE", "audio_output is not valid base64");
|
||||
if (turn.audio.size() + raw.size() > AudioCaps::kMaxStreamAudioBytes) {
|
||||
turn.truncated = true;
|
||||
return turn;
|
||||
}
|
||||
turn.audio += raw;
|
||||
++turn.audio_chunks;
|
||||
} else if (t == "user_interruption") {
|
||||
turn.interrupted = true;
|
||||
} else if (t == "tool_call") {
|
||||
turn.tool_calls.push_back(ev->json);
|
||||
} else if (t == "assistant_end") {
|
||||
turn.assistant_end = true;
|
||||
return turn;
|
||||
} else if (t == "error") {
|
||||
turn.error = str_at(ev->json, "message");
|
||||
if (turn.error.empty()) turn.error = "chat error";
|
||||
return turn;
|
||||
}
|
||||
}
|
||||
return turn;
|
||||
}
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+111
@@ -0,0 +1,111 @@
|
||||
#pragma once
|
||||
|
||||
// Hume WebSockets: TTS stream input (wss://api.hume.ai/v0/tts/stream/input)
|
||||
// and the EVI chat (wss://api.hume.ai/v0/evi/chat). Message shapes follow the
|
||||
// Python SDK types (tts/types/publish_tts.py, snippet_audio_chunk.py,
|
||||
// empathic_voice/types/*.py).
|
||||
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/hume/tts.h"
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
struct HumeStreamOptions {
|
||||
std::string format_type; // "", mp3, wav, pcm (query parameter format_type)
|
||||
std::string context_generation_id;
|
||||
std::vector<std::string> include_timestamp_types; // word | phoneme
|
||||
int instant_mode = -1;
|
||||
int no_binary = -1; // ask for base64 JSON chunks only
|
||||
int strip_headers = -1;
|
||||
std::string version; // "", "1", "2"
|
||||
std::string url; // "" = wss://api.hume.ai/v0/tts/stream/input
|
||||
SocketOptions socket;
|
||||
};
|
||||
|
||||
struct HumeStreamResult {
|
||||
std::string audio; // decoded chunks
|
||||
std::size_t chunks = 0;
|
||||
std::vector<Json> timestamps;
|
||||
std::string generation_id;
|
||||
bool last_chunk = false; // is_last_chunk seen
|
||||
bool truncated = false;
|
||||
std::string error;
|
||||
bool closed = false;
|
||||
};
|
||||
|
||||
class HumeTtsStream {
|
||||
public:
|
||||
HumeTtsStream(std::string api_key, HumeStreamOptions options);
|
||||
std::string url() const { return socket_->url(); }
|
||||
void connect() { socket_->connect(); }
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
// PublishTts: text (and optionally voice name / description / speed / trailing_silence).
|
||||
void send_text(const std::string& text, const std::string& voice_name = "", const std::string& description = "",
|
||||
double speed = 0, double trailing_silence = -1);
|
||||
void flush(); // {"flush": true}
|
||||
void finish(); // {"close": true}: force generation and close
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
HumeStreamResult collect(int max_wait_ms);
|
||||
|
||||
static Vocabulary vocabulary();
|
||||
|
||||
private:
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
struct EviOptions {
|
||||
std::string config_id;
|
||||
int config_version = -1;
|
||||
std::string resumed_chat_group_id;
|
||||
int verbose_transcription = -1;
|
||||
int event_limit = 0;
|
||||
std::string url; // "" = wss://api.hume.ai/v0/evi/chat
|
||||
SocketOptions socket;
|
||||
};
|
||||
|
||||
struct EviTurn {
|
||||
std::string chat_id, chat_group_id;
|
||||
std::vector<std::string> user_messages; // user_message.message.content
|
||||
std::vector<std::string> assistant_messages; // assistant_message.message.content
|
||||
std::string audio; // decoded audio_output data
|
||||
std::size_t audio_chunks = 0;
|
||||
std::vector<Json> tool_calls;
|
||||
bool interrupted = false;
|
||||
bool assistant_end = false;
|
||||
bool truncated = false;
|
||||
std::string error;
|
||||
bool closed = false;
|
||||
};
|
||||
|
||||
class EviChat {
|
||||
public:
|
||||
EviChat(std::string api_key, EviOptions options);
|
||||
std::string url() const { return socket_->url(); }
|
||||
void connect() { socket_->connect(); }
|
||||
void close() { socket_->close(); }
|
||||
bool is_open() const { return socket_->is_open(); }
|
||||
|
||||
void send_session_settings(const Json& settings); // {"type":"session_settings",...}
|
||||
void send_audio(const std::string& audio); // {"type":"audio_input","data": base64}
|
||||
void send_user_text(const std::string& text); // user_input
|
||||
void send_assistant_text(const std::string& text); // assistant_input (assistant speaks this)
|
||||
void send_tool_response(const std::string& tool_call_id, const std::string& content);
|
||||
void pause_assistant();
|
||||
void resume_assistant();
|
||||
std::optional<SocketEvent> next(int wait_ms) { return socket_->next(wait_ms); }
|
||||
EviTurn collect_turn(int max_wait_ms);
|
||||
|
||||
static Vocabulary vocabulary();
|
||||
|
||||
private:
|
||||
std::unique_ptr<AudioSocket> socket_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+55
@@ -0,0 +1,55 @@
|
||||
#include "vendors/audio/hume/speech.h"
|
||||
|
||||
#include <cctype>
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("hume", why); }
|
||||
|
||||
bool looks_like_id(const std::string& s) {
|
||||
if (s.size() != 36) return false;
|
||||
for (std::size_t i = 0; i < s.size(); ++i) {
|
||||
bool dash = i == 8 || i == 13 || i == 18 || i == 23;
|
||||
if (dash ? s[i] != '-' : !std::isxdigit(static_cast<unsigned char>(s[i]))) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
HumeTtsRequest HumeSynthesizer::map(const SpeechRequest& r) const {
|
||||
validate(r);
|
||||
if (r.sample_rate != 0) invalid("Hume chooses the sample rate; leave sample_rate at 0");
|
||||
HumeTtsRequest h;
|
||||
Utterance u;
|
||||
u.text = r.text;
|
||||
if (!r.voice.empty()) {
|
||||
if (looks_like_id(r.voice)) u.voice_id = r.voice;
|
||||
else u.voice_name = r.voice;
|
||||
u.voice_provider = "HUME_AI"; // library voices; custom voices go through the typed API directly
|
||||
}
|
||||
if (r.speed > 0) u.speed = r.speed;
|
||||
h.utterances.push_back(u);
|
||||
h.format = r.format;
|
||||
if (r.model == "1" || r.model == "2") h.version = r.model;
|
||||
return h;
|
||||
}
|
||||
|
||||
SpeechResult HumeSynthesizer::synthesize(const SpeechRequest& r) {
|
||||
HumeAudio a = tts_.synthesize_file(map(r));
|
||||
SpeechResult out;
|
||||
out.audio = std::move(a.bytes);
|
||||
out.content_type = a.content_type.empty() ? content_type_for(r.format) : a.content_type;
|
||||
out.format = r.format;
|
||||
out.provider = name();
|
||||
return out;
|
||||
}
|
||||
|
||||
Json HumeSynthesizer::voices() {
|
||||
Json q = Json::object();
|
||||
q.set("provider", Json("HUME_AI"));
|
||||
q.set("page_size", Json(100));
|
||||
return tts_.voices(q);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+31
@@ -0,0 +1,31 @@
|
||||
#pragma once
|
||||
|
||||
// Hume Octave behind the neutral synthesizer seam. Hume has no
|
||||
// speech-to-text product (EVI is a voice agent, Expression Measurement
|
||||
// analyses prosody), so there is no recognizer.
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "vendors/audio/hume/tts.h"
|
||||
#include "vendors/audio/speech.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
class HumeSynthesizer : public SpeechSynthesizer {
|
||||
public:
|
||||
explicit HumeSynthesizer(std::shared_ptr<const Hosts> hosts = HumeEndpoints::from_env()) : tts_(std::move(hosts)) {}
|
||||
std::string name() const override { return "hume"; }
|
||||
bool configured() const override { return tts_.configured(); }
|
||||
std::size_t max_chars() const override { return HumeCaps::kMaxUtteranceChars; }
|
||||
std::vector<std::string> formats() const override { return {"mp3", "wav", "pcm"}; }
|
||||
SpeechResult synthesize(const SpeechRequest& request) override;
|
||||
Json voices() override;
|
||||
// The Hume request a neutral request maps to. `voice` is a voice name from the
|
||||
// library unless it looks like an id (36 characters with dashes).
|
||||
HumeTtsRequest map(const SpeechRequest& request) const;
|
||||
|
||||
private:
|
||||
HumeTts tts_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+134
@@ -0,0 +1,134 @@
|
||||
#include "vendors/audio/hume/tts.h"
|
||||
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
using C = HumeCalls;
|
||||
|
||||
Json HumeTts::build(const HumeTtsRequest& r) const {
|
||||
if (r.utterances.empty() || r.utterances.size() > HumeCaps::kMaxUtterances) bad("utterances must hold 1..100 entries");
|
||||
Json arr = Json::array();
|
||||
for (const Utterance& u : r.utterances) {
|
||||
require_text("text", u.text, HumeCaps::kMaxUtteranceChars);
|
||||
require_text("description", u.description, HumeCaps::kMaxDescriptionChars, false);
|
||||
Json j = Json::object();
|
||||
j.set("text", Json(u.text));
|
||||
if (!u.description.empty()) j.set("description", Json(u.description));
|
||||
if (!u.voice_id.empty() && !u.voice_name.empty()) bad("give a voice id or a voice name, not both");
|
||||
if (!u.voice_id.empty() || !u.voice_name.empty()) {
|
||||
Json v = Json::object();
|
||||
if (!u.voice_id.empty()) {
|
||||
check_id("voice", u.voice_id);
|
||||
v.set("id", Json(u.voice_id));
|
||||
} else {
|
||||
require_text("voice name", u.voice_name, 128);
|
||||
v.set("name", Json(u.voice_name));
|
||||
}
|
||||
if (!u.voice_provider.empty()) {
|
||||
if (u.voice_provider != "HUME_AI" && u.voice_provider != "CUSTOM_VOICE")
|
||||
bad("voice provider must be HUME_AI or CUSTOM_VOICE");
|
||||
v.set("provider", Json(u.voice_provider));
|
||||
}
|
||||
j.set("voice", v);
|
||||
}
|
||||
if (u.speed != 0) {
|
||||
if (u.speed < 0) bad("speed must be positive");
|
||||
j.set("speed", Json(u.speed));
|
||||
}
|
||||
if (u.trailing_silence >= 0) {
|
||||
require_range("trailing_silence", u.trailing_silence, 0, 60);
|
||||
j.set("trailing_silence", Json(u.trailing_silence));
|
||||
}
|
||||
arr.push(j);
|
||||
}
|
||||
Json o = Json::object();
|
||||
o.set("utterances", arr);
|
||||
if (r.format != "mp3" && r.format != "wav" && r.format != "pcm") bad("format must be mp3, wav or pcm");
|
||||
Json fmt = Json::object();
|
||||
fmt.set("type", Json(r.format));
|
||||
o.set("format", fmt);
|
||||
if (r.num_generations < 0 || r.num_generations > HumeCaps::kMaxGenerations) bad("num_generations must be 0 or 1..5");
|
||||
if (r.num_generations > 0) o.set("num_generations", Json(r.num_generations));
|
||||
if (r.split_utterances >= 0) o.set("split_utterances", Json(r.split_utterances == 1));
|
||||
if (r.strip_headers >= 0) o.set("strip_headers", Json(r.strip_headers == 1));
|
||||
if (r.instant_mode >= 0) o.set("instant_mode", Json(r.instant_mode == 1));
|
||||
if (!r.version.empty()) {
|
||||
if (r.version != "1" && r.version != "2") bad("version must be 1 or 2");
|
||||
o.set("version", Json(r.version));
|
||||
}
|
||||
if (!r.context_generation_id.empty()) {
|
||||
check_id("generation", r.context_generation_id);
|
||||
Json ctx = Json::object();
|
||||
ctx.set("generation_id", Json(r.context_generation_id));
|
||||
o.set("context", ctx);
|
||||
}
|
||||
if (!r.include_timestamp_types.empty()) {
|
||||
Json ts = Json::array();
|
||||
for (const auto& t : r.include_timestamp_types) {
|
||||
if (t != "word" && t != "phoneme") bad("timestamp type must be word or phoneme");
|
||||
ts.push(Json(t));
|
||||
}
|
||||
o.set("include_timestamp_types", ts);
|
||||
}
|
||||
check_body(table_.get(C::kTts), &o);
|
||||
return o;
|
||||
}
|
||||
|
||||
Json HumeTts::synthesize(const HumeTtsRequest& r) {
|
||||
Json body = build(r);
|
||||
return call(C::kTts, {}, &body);
|
||||
}
|
||||
|
||||
std::string HumeTts::decode_first_audio(const Json& result) {
|
||||
const Json* gens = result.is_object() ? result.find("generations") : nullptr;
|
||||
if (!gens || !gens->is_array() || gens->as_array().empty())
|
||||
throw llm::ApiError("hume", 502, "BAD_RESPONSE", "no generations in the result");
|
||||
const Json* audio = gens->as_array()[0].find("audio");
|
||||
std::string raw;
|
||||
if (!audio || !audio->is_string() || !security::crypto::Base64::decode(audio->as_string(), raw))
|
||||
throw llm::ApiError("hume", 502, "BAD_RESPONSE", "generation audio is missing or not valid base64");
|
||||
return raw;
|
||||
}
|
||||
|
||||
HumeAudio HumeTts::synthesize_file(const HumeTtsRequest& r) {
|
||||
Json body = build(r);
|
||||
llm::RawResponse raw = call_raw(C::kTtsFile, {}, &body);
|
||||
return HumeAudio{raw.body, raw.content_type};
|
||||
}
|
||||
|
||||
ByteStreamer::Result HumeTts::stream_json(const HumeTtsRequest& r, const ChunkSink& sink) {
|
||||
Json body = build(r);
|
||||
return call_stream(C::kTtsJsonStream, {}, &body, Query(), sink);
|
||||
}
|
||||
|
||||
ByteStreamer::Result HumeTts::stream_file(const HumeTtsRequest& r, const ChunkSink& sink) {
|
||||
Json body = build(r);
|
||||
return call_stream(C::kTtsFileStream, {}, &body, Query(), sink);
|
||||
}
|
||||
|
||||
Json HumeTts::voices(const Json& q) {
|
||||
if (q.is_object()) {
|
||||
const Json* s = q.find("page_size");
|
||||
if (s && s->is_number() && (s->as_int() < 1 || s->as_int() > HumeCaps::kMaxPageSize)) bad("page_size must be 1..100");
|
||||
const Json* p = q.find("provider");
|
||||
if (p && p->is_string() && p->as_string() != "HUME_AI" && p->as_string() != "CUSTOM_VOICE")
|
||||
bad("provider must be HUME_AI or CUSTOM_VOICE");
|
||||
}
|
||||
return call(C::kVoices, {}, nullptr, query_from(C::kVoices, q));
|
||||
}
|
||||
|
||||
Json HumeTts::create_voice(const Json& body, bool confirm) {
|
||||
check_body(table_.get(C::kVoiceCreate), &body);
|
||||
require_confirm("saving a voice", confirm);
|
||||
return call(C::kVoiceCreate, {}, &body);
|
||||
}
|
||||
|
||||
Json HumeTts::delete_voice(const std::string& name, bool confirm) {
|
||||
require_text("voice name", name, 128);
|
||||
Query q;
|
||||
q.add("name", name);
|
||||
return call_destructive(C::kVoiceDelete, {}, nullptr, q, confirm);
|
||||
}
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+76
@@ -0,0 +1,76 @@
|
||||
#pragma once
|
||||
|
||||
// Hume Octave text to speech and voices: synthesize (JSON with base64 audio,
|
||||
// or a file), stream, list voices. Utterances can carry acting directions
|
||||
// ("description"), a voice by id or name, speed and trailing silence.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/audio_service.h"
|
||||
#include "vendors/audio/hume/calls.h"
|
||||
#include "vendors/audio/hume/client.h"
|
||||
|
||||
namespace molo::audio::hume {
|
||||
|
||||
struct HumeCaps {
|
||||
static constexpr std::size_t kMaxUtterances = 100; // our ceiling
|
||||
static constexpr std::size_t kMaxUtteranceChars = 5000; // our ceiling per utterance
|
||||
static constexpr std::size_t kMaxDescriptionChars = 1000;
|
||||
static constexpr int kMaxPageSize = 100;
|
||||
static constexpr int kMaxGenerations = 5; // our ceiling for num_generations
|
||||
};
|
||||
|
||||
class HumeService : public AudioService {
|
||||
public:
|
||||
explicit HumeService(std::shared_ptr<const Hosts> hosts) : AudioService(calls(), std::move(hosts)) {}
|
||||
HumeService() : HumeService(HumeEndpoints::from_env()) {}
|
||||
};
|
||||
|
||||
struct Utterance {
|
||||
std::string text; // required
|
||||
std::string description; // acting directions, or a voice prompt when no voice is given
|
||||
std::string voice_id; // exactly one of voice_id and voice_name, or neither
|
||||
std::string voice_name;
|
||||
std::string voice_provider; // "", HUME_AI, CUSTOM_VOICE (default CUSTOM_VOICE per the SDK docs)
|
||||
double speed = 0; // 0 unset; 0.75..1.5 is stable per the SDK docs
|
||||
double trailing_silence = -1; // seconds, -1 unset
|
||||
};
|
||||
|
||||
struct HumeTtsRequest {
|
||||
std::vector<Utterance> utterances;
|
||||
std::string format = "mp3"; // mp3 | wav | pcm
|
||||
int num_generations = 0; // 0 unset, else 1..5
|
||||
int split_utterances = -1;
|
||||
int strip_headers = -1;
|
||||
int instant_mode = -1;
|
||||
std::string version; // "", "1", "2"
|
||||
std::string context_generation_id; // continue the prosody of an earlier generation
|
||||
std::vector<std::string> include_timestamp_types; // word | phoneme
|
||||
};
|
||||
|
||||
struct HumeAudio {
|
||||
std::string bytes;
|
||||
std::string content_type;
|
||||
};
|
||||
|
||||
class HumeTts : public HumeService {
|
||||
public:
|
||||
using HumeService::HumeService;
|
||||
Json build(const HumeTtsRequest& request) const;
|
||||
// JSON result: generations[].audio is base64 (see decode_first_audio).
|
||||
Json synthesize(const HumeTtsRequest& request);
|
||||
static std::string decode_first_audio(const Json& result);
|
||||
HumeAudio synthesize_file(const HumeTtsRequest& request);
|
||||
// Newline delimited JSON chunks ({"type":"audio","audio":base64,...}); raw bytes to `sink`.
|
||||
ByteStreamer::Result stream_json(const HumeTtsRequest& request, const ChunkSink& sink);
|
||||
ByteStreamer::Result stream_file(const HumeTtsRequest& request, const ChunkSink& sink);
|
||||
|
||||
// Voices.
|
||||
Json voices(const Json& query_object = Json()); // provider, page_number, page_size, ascending_order, filter_tag
|
||||
Json create_voice(const Json& body, bool confirm); // saves a generated voice: a write
|
||||
Json delete_voice(const std::string& name, bool confirm);
|
||||
};
|
||||
|
||||
} // namespace molo::audio::hume
|
||||
Vendored
+50
@@ -0,0 +1,50 @@
|
||||
#include "vendors/audio/providers.h"
|
||||
|
||||
#include "vendors/audio/assemblyai/calls.h"
|
||||
#include "vendors/audio/cartesia/calls.h"
|
||||
#include "vendors/audio/deepgram/calls.h"
|
||||
#include "vendors/audio/elevenlabs/calls.h"
|
||||
#include "vendors/audio/hume/calls.h"
|
||||
#include "vendors/audio/speechmatics/calls.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
const std::vector<ProviderInfo>& Providers::all() {
|
||||
static const std::vector<ProviderInfo> v = {
|
||||
{"elevenlabs", "ElevenLabs", "ELEVENLABS_API_KEY", "", "https://elevenlabs.io/app/settings/api-keys", true, true,
|
||||
"tts-stream-input,stt-realtime,convai-conversation", &elevenlabs::calls},
|
||||
{"assemblyai", "AssemblyAI", "ASSEMBLYAI_API_KEY", "", "https://www.assemblyai.com/dashboard/api-keys", false, true,
|
||||
"streaming-v3", &assemblyai::calls},
|
||||
{"deepgram", "Deepgram", "DEEPGRAM_API_KEY", "", "https://console.deepgram.com", true, true,
|
||||
"listen-v1,listen-v2,speak-v1,speak-v2,agent-converse", &deepgram::calls},
|
||||
{"cartesia", "Cartesia", "CARTESIA_API_KEY", "", "https://play.cartesia.ai/keys", true, true,
|
||||
"tts-websocket,stt-turns-websocket,stt-websocket", &cartesia::calls},
|
||||
{"speechmatics", "Speechmatics", "SPEECHMATICS_API_KEY", "SPEECHMATICS_MANAGEMENT_TOKEN",
|
||||
"https://portal.speechmatics.com/settings/api-keys/", true, true, "realtime-v2", &speechmatics::calls},
|
||||
{"hume", "Hume", "HUME_API_KEY", "", "https://app.hume.ai/keys", true, false, "tts-stream-input,evi-chat",
|
||||
&hume::calls},
|
||||
};
|
||||
return v;
|
||||
}
|
||||
|
||||
const ProviderInfo* Providers::find(const std::string& name) {
|
||||
for (const ProviderInfo& p : all())
|
||||
if (name == p.name) return &p;
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
std::size_t Providers::call_count() {
|
||||
std::size_t n = 0;
|
||||
for (const ProviderInfo& p : all()) n += p.table().size();
|
||||
return n;
|
||||
}
|
||||
|
||||
const std::vector<Providers::Absent>& Providers::absent() {
|
||||
static const std::vector<Absent> v = {
|
||||
{"playht",
|
||||
"PlayHT's API hosts do not resolve (api.play.ht and api.play.ai, 2026-10-25): no public API to wrap, none invented"},
|
||||
};
|
||||
return v;
|
||||
}
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+41
@@ -0,0 +1,41 @@
|
||||
#pragma once
|
||||
|
||||
// The provider table of this package: one descriptor per vendor (name,
|
||||
// credential variable(s), what it offers, the size of its call table). The
|
||||
// demo status route and the registry read from here, so a provider is
|
||||
// described in exactly one place.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/audio/call_spec.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
struct ProviderInfo {
|
||||
const char* name; // "elevenlabs"
|
||||
const char* title; // "ElevenLabs"
|
||||
const char* key_env; // "ELEVENLABS_API_KEY"
|
||||
const char* extra_env; // second credential ("" none)
|
||||
const char* key_page; // where to create the key
|
||||
bool synthesizer; // SpeechSynthesizer available
|
||||
bool recognizer; // SpeechRecognizer available
|
||||
const char* websockets; // comma list of WebSocket channels this package speaks
|
||||
const CallTable& (*table)(); // the call table
|
||||
};
|
||||
|
||||
class Providers {
|
||||
public:
|
||||
static const std::vector<ProviderInfo>& all();
|
||||
static const ProviderInfo* find(const std::string& name);
|
||||
// Total number of REST calls over all tables.
|
||||
static std::size_t call_count();
|
||||
// Providers asked about but without a public API (PlayHT), with the reason.
|
||||
struct Absent {
|
||||
const char* name;
|
||||
const char* reason;
|
||||
};
|
||||
static const std::vector<Absent>& absent();
|
||||
};
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+75
@@ -0,0 +1,75 @@
|
||||
#include "vendors/audio/query.h"
|
||||
|
||||
#include <cctype>
|
||||
#include <cmath>
|
||||
#include <stdexcept>
|
||||
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
std::string Query::encode(const std::string& s) {
|
||||
static const char* hex = "0123456789ABCDEF";
|
||||
std::string out;
|
||||
for (unsigned char c : s) {
|
||||
if (std::isalnum(c) || c == '-' || c == '_' || c == '~' || c == '.') {
|
||||
out.push_back(static_cast<char>(c));
|
||||
} else {
|
||||
out.push_back('%');
|
||||
out.push_back(hex[c >> 4]);
|
||||
out.push_back(hex[c & 15]);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
Query& Query::add(const std::string& name, const std::string& value) {
|
||||
pairs_.emplace_back(name, value);
|
||||
return *this;
|
||||
}
|
||||
Query& Query::add_int(const std::string& name, long value) { return add(name, std::to_string(value)); }
|
||||
Query& Query::add_bool(const std::string& name, bool value) { return add(name, value ? "true" : "false"); }
|
||||
Query& Query::add_if(const std::string& name, const std::string& value) {
|
||||
if (!value.empty()) add(name, value);
|
||||
return *this;
|
||||
}
|
||||
|
||||
namespace {
|
||||
bool scalar(const std::string& name, const clients::body::Json& v, Query& q) {
|
||||
if (v.is_string()) {
|
||||
q.add(name, v.as_string());
|
||||
} else if (v.is_bool()) {
|
||||
q.add_bool(name, v.as_bool());
|
||||
} else if (v.is_number()) {
|
||||
double d = v.as_number();
|
||||
if (d == std::floor(d) && std::fabs(d) < 9e15) q.add_int(name, static_cast<long>(d));
|
||||
else q.add(name, std::to_string(d));
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
bool Query::add_json(const std::string& name, const clients::body::Json& value) {
|
||||
if (value.is_null()) return true;
|
||||
if (value.is_array()) {
|
||||
if (value.as_array().size() > AudioCaps::kMaxListItems) return false;
|
||||
for (const auto& e : value.as_array())
|
||||
if (!scalar(name, e, *this)) return false;
|
||||
return true;
|
||||
}
|
||||
return scalar(name, value, *this);
|
||||
}
|
||||
|
||||
std::string Query::str() const {
|
||||
std::string out;
|
||||
for (const auto& [k, v] : pairs_) {
|
||||
out += out.empty() ? '?' : '&';
|
||||
out += encode(k) + "=" + encode(v);
|
||||
if (out.size() > AudioCaps::kMaxQueryChars) throw std::length_error("query string over the cap");
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+33
@@ -0,0 +1,33 @@
|
||||
#pragma once
|
||||
|
||||
// Percent-encoded query string builder with a hard size cap.
|
||||
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
class Query {
|
||||
public:
|
||||
Query& add(const std::string& name, const std::string& value);
|
||||
Query& add_int(const std::string& name, long value);
|
||||
Query& add_bool(const std::string& name, bool value);
|
||||
Query& add_if(const std::string& name, const std::string& value); // skips ""
|
||||
// Scalars become one pair, arrays one pair per element, null is skipped.
|
||||
// Objects are rejected (llm::ApiError::invalid is thrown by the caller).
|
||||
bool add_json(const std::string& name, const clients::body::Json& value);
|
||||
bool empty() const { return pairs_.empty(); }
|
||||
// "" or "?a=b&c=d". Throws std::length_error past AudioCaps::kMaxQueryChars.
|
||||
std::string str() const;
|
||||
const std::vector<std::pair<std::string, std::string>>& pairs() const { return pairs_; }
|
||||
|
||||
static std::string encode(const std::string& s);
|
||||
|
||||
private:
|
||||
std::vector<std::pair<std::string, std::string>> pairs_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+45
@@ -0,0 +1,45 @@
|
||||
#include "llm/error.h"
|
||||
#include "vendors/audio/assemblyai/speech.h"
|
||||
#include "vendors/audio/cartesia/speech.h"
|
||||
#include "vendors/audio/deepgram/speech.h"
|
||||
#include "vendors/audio/elevenlabs/speech.h"
|
||||
#include "vendors/audio/hume/speech.h"
|
||||
#include "vendors/audio/providers.h"
|
||||
#include "vendors/audio/speech.h"
|
||||
#include "vendors/audio/speechmatics/speech.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
std::vector<std::string> SpeechRegistry::synthesizer_names() {
|
||||
std::vector<std::string> out;
|
||||
for (const ProviderInfo& p : Providers::all())
|
||||
if (p.synthesizer) out.push_back(p.name);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<std::string> SpeechRegistry::recognizer_names() {
|
||||
std::vector<std::string> out;
|
||||
for (const ProviderInfo& p : Providers::all())
|
||||
if (p.recognizer) out.push_back(p.name);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::shared_ptr<SpeechSynthesizer> SpeechRegistry::synthesizer(const std::string& name) {
|
||||
if (name == "elevenlabs") return std::make_shared<elevenlabs::ElevenSynthesizer>();
|
||||
if (name == "deepgram") return std::make_shared<deepgram::DeepgramSynthesizer>();
|
||||
if (name == "cartesia") return std::make_shared<cartesia::CartesiaSynthesizer>();
|
||||
if (name == "speechmatics") return std::make_shared<speechmatics::SpeechmaticsSynthesizer>();
|
||||
if (name == "hume") return std::make_shared<hume::HumeSynthesizer>();
|
||||
throw llm::ApiError::invalid("audio", "no speech synthesizer named '" + name + "'");
|
||||
}
|
||||
|
||||
std::shared_ptr<SpeechRecognizer> SpeechRegistry::recognizer(const std::string& name) {
|
||||
if (name == "elevenlabs") return std::make_shared<elevenlabs::ElevenRecognizer>();
|
||||
if (name == "assemblyai") return std::make_shared<assemblyai::AssemblyRecognizer>();
|
||||
if (name == "deepgram") return std::make_shared<deepgram::DeepgramRecognizer>();
|
||||
if (name == "cartesia") return std::make_shared<cartesia::CartesiaRecognizer>();
|
||||
if (name == "speechmatics") return std::make_shared<speechmatics::SpeechmaticsRecognizer>();
|
||||
throw llm::ApiError::invalid("audio", "no speech recognizer named '" + name + "'");
|
||||
}
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+180
@@ -0,0 +1,180 @@
|
||||
#include "vendors/audio/socket.h"
|
||||
|
||||
#include <chrono>
|
||||
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
namespace gl = google::live;
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
long now_ms() {
|
||||
return std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
std::chrono::steady_clock::now().time_since_epoch())
|
||||
.count();
|
||||
}
|
||||
|
||||
llm::ApiError map_error(const std::string& vendor, const gl::WsError& e) {
|
||||
using K = gl::WsError::Kind;
|
||||
int status = e.http_status();
|
||||
if (status == 401 || status == 403)
|
||||
return llm::ApiError(vendor, status, "authentication_error", e.what());
|
||||
if (e.kind() == K::Invalid) return llm::ApiError(vendor, 400, "invalid_request_error", e.what());
|
||||
if (e.kind() == K::Timeout) return llm::ApiError(vendor, 504, "timeout", e.what());
|
||||
return llm::ApiError(vendor, status ? status : 502, "UNAVAILABLE", e.what());
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
AudioSocket::AudioSocket(std::string vendor, std::string url, Headers headers, Vocabulary vocabulary,
|
||||
SocketOptions options)
|
||||
: vendor_(std::move(vendor)),
|
||||
url_(std::move(url)),
|
||||
headers_(std::move(headers)),
|
||||
vocabulary_(std::move(vocabulary)),
|
||||
options_(std::move(options)) {}
|
||||
|
||||
AudioSocket::~AudioSocket() {
|
||||
try {
|
||||
close();
|
||||
} catch (...) {
|
||||
}
|
||||
}
|
||||
|
||||
std::string AudioSocket::with_query(const std::string& url, const std::string& query) {
|
||||
if (query.empty()) return url;
|
||||
return url + (url.find('?') == std::string::npos ? "?" : "&") + query;
|
||||
}
|
||||
|
||||
void AudioSocket::connect() {
|
||||
if (open_) throw llm::ApiError::invalid(vendor_, "socket already open");
|
||||
for (const auto& [name, value] : headers_) {
|
||||
if (value.empty() || value.find_first_of("\r\n") != std::string::npos ||
|
||||
name.find_first_of("\r\n:") != std::string::npos)
|
||||
throw llm::ApiError::invalid(vendor_, "missing or malformed credential header");
|
||||
}
|
||||
try {
|
||||
gl::WsUrl url = gl::WsUrl::parse(url_);
|
||||
ws_ = std::make_unique<gl::WsClient>(
|
||||
gl::open_stream(url, options_.connect_timeout_ms, AudioCaps::kIoTimeoutMs, options_.tls));
|
||||
gl::WsHeaders headers;
|
||||
for (const auto& h : headers_) headers.emplace_back(h.first, h.second);
|
||||
ws_->handshake(url, headers, options_.handshake_timeout_ms);
|
||||
ws_->set_read_timeout_ms(AudioCaps::kPollSliceMs);
|
||||
} catch (const gl::WsError& e) {
|
||||
if (ws_) ws_->shutdown();
|
||||
ws_.reset();
|
||||
throw map_error(vendor_, e);
|
||||
}
|
||||
open_ = true;
|
||||
started_ms_ = now_ms();
|
||||
}
|
||||
|
||||
void AudioSocket::send_json(const Json& message, bool allow_untyped) {
|
||||
if (!open_) throw llm::ApiError(vendor_, 409, "not_open", "socket is not open");
|
||||
const Json* type = message.is_object() ? message.find(vocabulary_.discriminator()) : nullptr;
|
||||
if (type) {
|
||||
if (!type->is_string() || !vocabulary_.known_client(type->as_string()))
|
||||
throw llm::ApiError::invalid(vendor_, "unknown client message type");
|
||||
} else if (!allow_untyped) {
|
||||
throw llm::ApiError::invalid(vendor_, "client message has no '" + vocabulary_.discriminator() + "'");
|
||||
}
|
||||
std::string text = message.dump();
|
||||
send_text(text);
|
||||
}
|
||||
|
||||
void AudioSocket::send_text(const std::string& text) {
|
||||
if (!open_) throw llm::ApiError(vendor_, 409, "not_open", "socket is not open");
|
||||
if (text.size() > AudioCaps::kMaxSocketFrameBytes)
|
||||
throw llm::ApiError::invalid(vendor_, "text frame over the byte cap");
|
||||
try {
|
||||
ws_->send_text(text);
|
||||
} catch (const gl::WsError& e) {
|
||||
throw map_error(vendor_, e);
|
||||
}
|
||||
++counters_.sent;
|
||||
}
|
||||
|
||||
void AudioSocket::send_binary(const std::string& bytes) {
|
||||
if (!open_) throw llm::ApiError(vendor_, 409, "not_open", "socket is not open");
|
||||
if (bytes.empty() || bytes.size() > AudioCaps::kMaxSocketFrameBytes)
|
||||
throw llm::ApiError::invalid(vendor_, "binary frame must be 1.." +
|
||||
std::to_string(AudioCaps::kMaxSocketFrameBytes) + " bytes");
|
||||
try {
|
||||
ws_->send_binary(bytes);
|
||||
} catch (const gl::WsError& e) {
|
||||
throw map_error(vendor_, e);
|
||||
}
|
||||
++counters_.sent;
|
||||
}
|
||||
|
||||
void AudioSocket::close() {
|
||||
if (!ws_) return;
|
||||
try {
|
||||
if (open_) ws_->close(1000, "");
|
||||
} catch (...) {
|
||||
}
|
||||
ws_->shutdown();
|
||||
open_ = false;
|
||||
}
|
||||
|
||||
SocketEvent AudioSocket::decode(const gl::WsMessage& m) {
|
||||
SocketEvent ev;
|
||||
if (m.kind == gl::WsMessage::Kind::Close) {
|
||||
ev.kind = SocketEvent::Kind::kClose;
|
||||
ev.text = m.data;
|
||||
ev.close_code = m.code;
|
||||
ev.type = "websocket.close";
|
||||
open_ = false;
|
||||
return ev;
|
||||
}
|
||||
++counters_.received;
|
||||
if (m.kind == gl::WsMessage::Kind::Binary) {
|
||||
ev.kind = SocketEvent::Kind::kBinary;
|
||||
ev.bytes = m.data;
|
||||
ev.type = "binary";
|
||||
++counters_.binary_frames;
|
||||
return ev;
|
||||
}
|
||||
ev.kind = SocketEvent::Kind::kText;
|
||||
ev.text = m.data;
|
||||
try {
|
||||
ev.json = Json::parse(m.data);
|
||||
ev.json_ok = true;
|
||||
} catch (const std::exception&) {
|
||||
++counters_.decode_errors;
|
||||
ev.known = false;
|
||||
return ev;
|
||||
}
|
||||
if (ev.json.is_object()) {
|
||||
if (const Json* t = ev.json.find(vocabulary_.discriminator()); t && t->is_string())
|
||||
ev.type = t->as_string();
|
||||
}
|
||||
if (!ev.type.empty()) ev.known = vocabulary_.known_server(ev.type);
|
||||
if (!ev.known) ++counters_.unknown_events;
|
||||
return ev;
|
||||
}
|
||||
|
||||
std::optional<SocketEvent> AudioSocket::next(int wait_ms) {
|
||||
if (!ws_) throw llm::ApiError(vendor_, 409, "not_open", "socket is not open");
|
||||
if (wait_ms < 0) wait_ms = 0;
|
||||
if (wait_ms > AudioCaps::kMaxWaitMs) wait_ms = AudioCaps::kMaxWaitMs;
|
||||
long deadline = now_ms() + wait_ms;
|
||||
for (;;) {
|
||||
if (now_ms() - started_ms_ > static_cast<long>(options_.max_session_seconds) * 1000)
|
||||
throw llm::ApiError(vendor_, 504, "max_duration", "session ran past the local time cap");
|
||||
try {
|
||||
auto m = ws_->receive();
|
||||
if (m) return decode(*m);
|
||||
} catch (const gl::WsError& e) {
|
||||
open_ = false;
|
||||
throw map_error(vendor_, e);
|
||||
}
|
||||
if (now_ms() >= deadline) return std::nullopt;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+112
@@ -0,0 +1,112 @@
|
||||
#pragma once
|
||||
|
||||
// One provider WebSocket: connect with explicit headers (the credential
|
||||
// travels in a header or a provider-defined sub-protocol, never logged and
|
||||
// never part of an error), send JSON text or binary frames, read decoded
|
||||
// events with a deadline. Built on the RFC 6455 client in src/google/live.
|
||||
// Protocol knowledge (message names, field names) lives in the per-provider
|
||||
// session classes and their Vocabulary, not here.
|
||||
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <set>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "google/live/ws_client.h"
|
||||
#include "google/live/ws_stream.h"
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
struct SocketOptions {
|
||||
int connect_timeout_ms = AudioCaps::kConnectTimeoutMs;
|
||||
int handshake_timeout_ms = AudioCaps::kHandshakeTimeoutMs;
|
||||
int max_session_seconds = AudioCaps::kMaxSessionSeconds;
|
||||
google::live::TlsFactory tls; // null: default wolfSSL session
|
||||
};
|
||||
|
||||
// A decoded server frame.
|
||||
struct SocketEvent {
|
||||
enum class Kind { kText, kBinary, kClose };
|
||||
Kind kind = Kind::kText;
|
||||
std::string text; // text frames and the close reason
|
||||
std::string bytes; // binary frames
|
||||
clients::body::Json json; // parsed text frame (null when it is not JSON)
|
||||
bool json_ok = false;
|
||||
std::string type; // value of the provider's discriminator field, "" if absent
|
||||
bool known = true; // false: type not in the provider's server vocabulary
|
||||
int close_code = 0;
|
||||
};
|
||||
|
||||
// Message names a provider speaks. Unknown server events are kept and counted,
|
||||
// unknown client messages are refused before sending.
|
||||
class Vocabulary {
|
||||
public:
|
||||
Vocabulary(std::string discriminator, std::set<std::string> client, std::set<std::string> server)
|
||||
: discriminator_(std::move(discriminator)), client_(std::move(client)), server_(std::move(server)) {}
|
||||
const std::string& discriminator() const { return discriminator_; }
|
||||
bool known_client(const std::string& t) const { return client_.count(t) > 0; }
|
||||
bool known_server(const std::string& t) const { return server_.count(t) > 0; }
|
||||
const std::set<std::string>& client() const { return client_; }
|
||||
const std::set<std::string>& server() const { return server_; }
|
||||
|
||||
private:
|
||||
std::string discriminator_; // "type", "message_type", "event", ...
|
||||
std::set<std::string> client_;
|
||||
std::set<std::string> server_;
|
||||
};
|
||||
|
||||
class AudioSocket {
|
||||
public:
|
||||
using Headers = std::vector<std::pair<std::string, std::string>>;
|
||||
|
||||
AudioSocket(std::string vendor, std::string url, Headers headers, Vocabulary vocabulary,
|
||||
SocketOptions options = {});
|
||||
~AudioSocket();
|
||||
AudioSocket(const AudioSocket&) = delete;
|
||||
AudioSocket& operator=(const AudioSocket&) = delete;
|
||||
|
||||
// Throws llm::ApiError (401 on a refused upgrade with 401/403, else 502/504).
|
||||
void connect();
|
||||
bool is_open() const { return open_; }
|
||||
// URL as given, without any header value: safe to log.
|
||||
const std::string& url() const { return url_; }
|
||||
|
||||
// `type` must be in the client vocabulary when the message has the
|
||||
// discriminator field; messages without it (provider specific raw JSON such
|
||||
// as {"text":"..."}) pass when allow_untyped is true.
|
||||
void send_json(const clients::body::Json& message, bool allow_untyped = false);
|
||||
void send_text(const std::string& text);
|
||||
void send_binary(const std::string& bytes);
|
||||
void close();
|
||||
|
||||
// Next server message, or nullopt when `wait_ms` elapsed. A close frame is
|
||||
// returned as kClose. Throws when the session ran past max_session_seconds.
|
||||
std::optional<SocketEvent> next(int wait_ms);
|
||||
|
||||
struct Counters {
|
||||
std::size_t sent = 0, received = 0, unknown_events = 0, binary_frames = 0, decode_errors = 0;
|
||||
};
|
||||
const Counters& counters() const { return counters_; }
|
||||
const Vocabulary& vocabulary() const { return vocabulary_; }
|
||||
|
||||
static std::string with_query(const std::string& url, const std::string& query);
|
||||
|
||||
private:
|
||||
SocketEvent decode(const google::live::WsMessage& m);
|
||||
|
||||
std::string vendor_;
|
||||
std::string url_;
|
||||
Headers headers_;
|
||||
Vocabulary vocabulary_;
|
||||
SocketOptions options_;
|
||||
std::unique_ptr<google::live::WsClient> ws_;
|
||||
bool open_ = false;
|
||||
long started_ms_ = 0;
|
||||
Counters counters_;
|
||||
};
|
||||
|
||||
} // namespace molo::audio
|
||||
Vendored
+162
@@ -0,0 +1,162 @@
|
||||
#include "vendors/audio/speech.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cctype>
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
#include <thread>
|
||||
|
||||
#include "llm/error.h"
|
||||
#include "vendors/audio/caps.h"
|
||||
|
||||
namespace molo::audio {
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void invalid(const std::string& why) { throw llm::ApiError::invalid("audio", why); }
|
||||
|
||||
bool has_nul(const std::string& s) { return s.find('\0') != std::string::npos; }
|
||||
|
||||
bool is_utf8_continuation(unsigned char c) { return (c & 0xC0) == 0x80; }
|
||||
} // namespace
|
||||
|
||||
std::string content_type_for(const std::string& format) {
|
||||
if (format == "mp3") return "audio/mpeg";
|
||||
if (format == "wav") return "audio/wav";
|
||||
if (format == "pcm") return "audio/pcm";
|
||||
return "";
|
||||
}
|
||||
|
||||
void validate(const SpeechRequest& r) {
|
||||
if (r.text.empty()) invalid("text is required");
|
||||
if (r.text.size() > AudioCaps::kMaxTextChars)
|
||||
invalid("text is over " + std::to_string(AudioCaps::kMaxTextChars) + " characters");
|
||||
if (has_nul(r.text)) invalid("text has a NUL byte");
|
||||
for (const std::string* s : {&r.voice, &r.model, &r.language}) {
|
||||
if (s->size() > 256) invalid("voice, model and language are at most 256 characters");
|
||||
if (has_nul(*s)) invalid("voice, model and language must not hold NUL bytes");
|
||||
}
|
||||
if (content_type_for(r.format).empty()) invalid("format must be mp3, wav or pcm");
|
||||
if (r.sample_rate != 0 && (r.sample_rate < 8000 || r.sample_rate > 96000))
|
||||
invalid("sample_rate must be 0 or 8000..96000");
|
||||
if (r.speed != 0 && !(r.speed >= 0.25 && r.speed <= 4.0)) invalid("speed must be 0 or 0.25..4.0");
|
||||
}
|
||||
|
||||
void validate(const TranscribeRequest& r) {
|
||||
if (r.audio.empty() == r.url.empty()) invalid("give exactly one of audio and url");
|
||||
if (r.audio.size() > AudioCaps::kMaxAudioInBytes) invalid("audio is over the byte cap");
|
||||
if (!r.url.empty()) {
|
||||
if (r.url.size() > AudioCaps::kMaxFieldChars) invalid("url is too long");
|
||||
if (r.url.rfind("https://", 0) != 0 && r.url.rfind("http://", 0) != 0) invalid("url must be http or https");
|
||||
if (r.url.find_first_of(" \r\n") != std::string::npos) invalid("url has whitespace");
|
||||
}
|
||||
if (r.filename.empty() || r.filename.size() > 255 || r.filename.find_first_of("\r\n\"/\\") != std::string::npos)
|
||||
invalid("filename is 1..255 characters without quotes, slashes or line breaks");
|
||||
if (r.content_type.empty() || r.content_type.size() > 128 || r.content_type.find_first_of("\r\n") != std::string::npos)
|
||||
invalid("content_type is required");
|
||||
if (r.language.size() > 32 || r.model.size() > 128) invalid("language or model is too long");
|
||||
if (r.keywords.size() > AudioCaps::kMaxKeywords)
|
||||
invalid("at most " + std::to_string(AudioCaps::kMaxKeywords) + " keywords");
|
||||
for (const auto& k : r.keywords)
|
||||
if (k.empty() || k.size() > 100 || has_nul(k)) invalid("keywords are 1..100 characters");
|
||||
if (r.max_wait_seconds < 1 || r.max_wait_seconds > AudioCaps::kMaxPollSeconds)
|
||||
invalid("max_wait_seconds must be 1.." + std::to_string(AudioCaps::kMaxPollSeconds));
|
||||
}
|
||||
|
||||
std::vector<std::string> split_text(const std::string& text, std::size_t max_chars) {
|
||||
if (max_chars == 0) invalid("max_chars must be positive");
|
||||
std::vector<std::string> out;
|
||||
std::size_t pos = 0;
|
||||
while (pos < text.size()) {
|
||||
while (pos < text.size() && std::isspace(static_cast<unsigned char>(text[pos]))) ++pos;
|
||||
if (pos >= text.size()) break;
|
||||
std::size_t remaining = text.size() - pos;
|
||||
std::size_t len = remaining;
|
||||
if (remaining > max_chars) {
|
||||
std::size_t window = max_chars;
|
||||
std::size_t cut = std::string::npos;
|
||||
// Prefer the last sentence end in the second half of the window, then any whitespace.
|
||||
for (std::size_t i = window; i > window / 2; --i) {
|
||||
char c = text[pos + i - 1];
|
||||
char next = pos + i < text.size() ? text[pos + i] : ' ';
|
||||
if ((c == '.' || c == '!' || c == '?' || c == '\n') && std::isspace(static_cast<unsigned char>(next))) {
|
||||
cut = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (cut == std::string::npos) {
|
||||
for (std::size_t i = window; i > window / 2; --i) {
|
||||
if (std::isspace(static_cast<unsigned char>(text[pos + i - 1]))) {
|
||||
cut = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (cut == std::string::npos) {
|
||||
cut = window;
|
||||
while (cut > 1 && is_utf8_continuation(static_cast<unsigned char>(text[pos + cut]))) --cut;
|
||||
}
|
||||
len = cut;
|
||||
}
|
||||
std::string chunk = text.substr(pos, len);
|
||||
while (!chunk.empty() && std::isspace(static_cast<unsigned char>(chunk.back()))) chunk.pop_back();
|
||||
if (!chunk.empty()) out.push_back(chunk);
|
||||
if (out.size() > AudioCaps::kMaxChunks)
|
||||
invalid("text needs more than " + std::to_string(AudioCaps::kMaxChunks) + " chunks");
|
||||
pos += len;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::size_t ChunkedSynthesizer::max_chars() const { return AudioCaps::kMaxTextChars; }
|
||||
|
||||
SpeechResult ChunkedSynthesizer::synthesize(const SpeechRequest& request) {
|
||||
validate(request);
|
||||
std::size_t per_call = std::min(inner_->max_chars(), AudioCaps::kMaxChunkChars * 8);
|
||||
if (request.text.size() <= per_call) return inner_->synthesize(request);
|
||||
std::vector<std::string> chunks = split_text(request.text, per_call);
|
||||
if (chunks.size() > 1 && request.format == "wav")
|
||||
invalid("wav cannot be joined across chunks: use mp3 or pcm for long text");
|
||||
SpeechResult joined;
|
||||
for (const std::string& c : chunks) {
|
||||
SpeechRequest part = request;
|
||||
part.text = c;
|
||||
SpeechResult r = inner_->synthesize(part);
|
||||
if (joined.audio.size() + r.audio.size() > AudioCaps::kMaxAudioOutBytes)
|
||||
invalid("joined audio is over the byte cap");
|
||||
joined.audio += r.audio;
|
||||
joined.content_type = r.content_type;
|
||||
joined.format = r.format;
|
||||
joined.provider = r.provider;
|
||||
}
|
||||
return joined;
|
||||
}
|
||||
|
||||
PollOutcome poll(const PollPolicy& requested, const std::function<bool(int)>& step,
|
||||
const std::function<void(int)>& sleep_ms) {
|
||||
PollPolicy policy = PollPolicy::clamped(requested.interval_ms, requested.max_seconds);
|
||||
PollOutcome out;
|
||||
const long cap_ms = static_cast<long>(policy.max_seconds) * 1000;
|
||||
for (;;) {
|
||||
++out.polls;
|
||||
if (step(out.polls)) {
|
||||
out.done = true;
|
||||
return out;
|
||||
}
|
||||
if (out.waited_ms + policy.interval_ms > cap_ms) {
|
||||
out.timed_out = true;
|
||||
return out;
|
||||
}
|
||||
if (sleep_ms) sleep_ms(policy.interval_ms);
|
||||
else std::this_thread::sleep_for(std::chrono::milliseconds(policy.interval_ms));
|
||||
out.waited_ms += policy.interval_ms;
|
||||
}
|
||||
}
|
||||
|
||||
PollPolicy PollPolicy::clamped(int interval_ms, int max_seconds) {
|
||||
PollPolicy p;
|
||||
p.interval_ms = std::clamp(interval_ms, AudioCaps::kMinPollIntervalMs, AudioCaps::kMaxPollIntervalMs);
|
||||
p.max_seconds = std::clamp(max_seconds, 1, AudioCaps::kMaxPollSeconds);
|
||||
return p;
|
||||
}
|
||||
|
||||
} // namespace molo::audio
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user