feat(clients): bedrock: SigV4 signer, Converse, event streams, guardrails, agents, KB, batch

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-25 15:08:00 +01:00
co-authored by Claude Sonnet 5.5
parent 3db67fcf1f
commit d2f783a6fe
133 changed files with 6394 additions and 3 deletions
+230
View File
@@ -0,0 +1,230 @@
# Amazon Bedrock (src/vendors/bedrock)
Namespace `molo::bedrock`. Amazon Bedrock over HTTPS with our own AWS Signature
V4 signer. Plan: `plans/18-breadth.md` (vendor rules of `plans/17`). Status
2026-10-25: implemented and tested locally; no AWS credential exists in this
environment, so only unauthenticated probes were verified live.
## Scope
Wrapped (18 operations, three hosts, one call table):
- bedrock-runtime (`bedrock-runtime.<region>.amazonaws.com`): Converse,
ConverseStream, InvokeModel, InvokeModelWithResponseStream, CountTokens,
ApplyGuardrail.
- bedrock-agent-runtime (`bedrock-agent-runtime.<region>.amazonaws.com`):
InvokeAgent (event stream), Retrieve (knowledge bases).
- bedrock control plane (`bedrock.<region>.amazonaws.com`): ListFoundationModels,
GetFoundationModel, ListInferenceProfiles, GetInferenceProfile, ListGuardrails,
GetGuardrail, batch inference (CreateModelInvocationJob, GetModelInvocationJob,
ListModelInvocationJobs, StopModelInvocationJob).
- `SigV4Signer` (own implementation over `security::crypto` Sha256 + Hmac),
event-stream decoder with CRC checks, `BedrockChatModel : llm::ChatModel`.
Not wrapped, said plainly (136 operations, each with its reason in
`tools/check_spec.py` `EXCLUDED`): async invoke (Nova Reel), bidirectional
streaming (needs h2), `InvokeGuardrailChecks`, sessions / flows / agent memory /
RetrieveAndGenerate / Rerank / inline agents of the agent runtime, and the
management plane (customization, import, evaluation, provisioned throughput,
marketplace, automated reasoning, guardrail and profile writes, logging, tags).
Bedrock Agents build-time APIs (`bedrock-agent`) are a separate service and are
not touched.
## Module map (one concern per file)
| File | Class | Concern |
|---|---|---|
| `caps.h` | `BedrockCaps` | static ceilings |
| `crc32.h/.cpp` | `Crc32` | IEEE CRC-32 for the event-stream framing |
| `sigv4.h/.cpp` | `SigV4Signer`, `AwsCredentials`, `SigV4Request/Result` | canonical request, string to sign, signing key, Authorization header |
| `credentials.h/.cpp` | `Credentials`, `Endpoints`, `AuthMode`, `AwsService` | env credentials, region, the three hosts |
| `calls.h/.cpp` | `Calls`, `CallSpec`, `SchemaFields` | the ONE call table (18 rows) and schema table (29 rows); path rendering |
| `event_stream.h/.cpp` | `EventStreamDecoder`, `EventMessage` | `application/vnd.amazon.eventstream` framing, both CRCs, exceptions |
| `client.h/.cpp` | `BedrockClient`, `BedrockService`, `CallInput`, `WirePreview` | builds, signs and sends a call from the table; error mapping |
| `converse.h/.cpp` | `Blocks`, `ConverseRequest`, `ConverseResult`, `ConverseStreamAccumulator`, `StreamDelta` | Converse bodies, results, stream folding |
| `runtime.h/.cpp` | `Runtime` | Converse, ConverseStream, InvokeModel (+ stream), CountTokens, ApplyGuardrail |
| `control.h/.cpp` | `Models`, `Guardrails`, `BatchJobs`, `BatchJobParams`, `ListResult` | control plane |
| `agent.h/.cpp` | `AgentRuntime`, `AgentInvokeParams`, `RetrieveParams` | InvokeAgent, Retrieve (+ capped paging) |
| `chat_model.h/.cpp` | `BedrockChatModel` | neutral `llm::ChatModel` over Converse |
| `tools/check_spec.py` | | checks the tables against the AWS service models |
`llm::ApiClient` is not used: it sends a fixed bearer or header key, while
SigV4 signs the final request (method, path, query, headers, body hash) per
call. `BedrockClient` does that job and keeps the foundation contracts: no
credential means `llm::NotConfiguredError` (503 `NOT_CONFIGURED`) before any
traffic, every failure is an `llm::ApiError`, the secret never appears in an
URL, an error or a preview. `BedrockService` replaces `llm::Service` for the
same reason (its constructor takes an `ApiClient`); `valid_id` is the same
rule widened to ARNs.
## Credentials
Read through `llm::Config` (environment, then `.env.json.local`, then
`.env.json`):
| Variable | Use |
|---|---|
| `AWS_BEARER_TOKEN_BEDROCK` | Bedrock API key, sent as `Authorization: Bearer`; nothing is signed |
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | SigV4 |
| `AWS_SESSION_TOKEN` | optional, signed and sent as `x-amz-security-token` |
| `AWS_REGION`, then `AWS_DEFAULT_REGION` | region; invalid or unset gives `us-east-1` |
| `BEDROCK_LIVE_MODEL` | live test only, model id of the one paid call |
When both an API key and SigV4 keys are set the API key is used for the calls
whose service model lists `httpBearerAuth` (bedrock, bedrock-runtime) and the
SigV4 keys for the agent runtime host, whose model lists SigV4 only. With only
an API key, an agent runtime call is refused before any traffic (503
`NOT_CONFIGURED`, message names the variables). Console: IAM access keys are
created at https://console.aws.amazon.com/iam/home#/security_credentials
(unverified: page not fetched); Bedrock API keys at the Bedrock console, "API
keys" (unverified, see https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html).
## Static caps (caps.h)
| Knob | Value |
|---|---|
| InvokeModel body | 25,000,000 bytes (spec blob max) |
| Response / stream bytes | 64 MiB / 64 MiB |
| Event message / headers | 16 MiB / 128 KiB |
| Stream events | 200,000 |
| Model id | 2048 characters (spec) |
| Messages / content blocks per message / tools | 4096 / 512 / 128 |
| Stop sequences | 2500 (spec) |
| Retrieve results / query | 100 / 20,000 characters (spec) |
| Agent input | 25,000,000 characters (spec) |
| List `maxResults` | 1..1000 (spec); walks stop at 20 pages, 5000 items, and say `truncated` |
| Batch timeout | 24..168 hours (spec); job name 63 characters |
| Signed headers / query params / path | 64 / 64 / 4096 |
## Contracts
- Every path, query parameter and header goes through the call table:
`BedrockClient` refuses a query or header name the row does not declare, a
path value outside `[A-Za-z0-9._:/-]`, `..`, an empty or over-long value.
Path values are percent-encoded as one segment (`:` becomes `%3A`, `/`
becomes `%2F`), so ARNs work.
- Canonical URI: the path as sent is encoded again (double encoding), the rule
for every non-S3 service; the suite and the Bedrock model-id test cover it.
- Validation runs before the credential check: bad input is 400
`INVALID_ARGUMENT`, never 503.
- Event streams: both CRC32s are checked before a message is delivered; a
message over 16 MiB, a header block over 128 KiB, a prelude that fails its
CRC, or a stream that ends inside a message throws. `:message-type`
`exception` / `error` becomes `ApiError` (type = exception name, status from
the service model: throttling 429, validation 400, modelStreamError 424, ...).
The transport hides the HTTP status of a stream, so a JSON error document in
place of a stream is thrown as status 502, type `STREAM_HTTP_ERROR`, with the
server's message.
- Errors: type from `x-amzn-ErrorType` (text after `:` dropped, `ns#` prefix
dropped), else `__type` / `code`; message from `message` / `Message`; request
id from `x-amzn-RequestId`.
- Lists never truncate silently: `ListResult.truncated`, `pages`, `next_token`.
- Create and stop (batch) are explicit methods; `BatchJobs::create` costs money
once the job runs. The demo makes them dry runs.
- `BedrockChatModel` mapping decisions are listed in `chat_model.h`.
## Sources
1. AWS service models (botocore `botocore/data/<service>/<version>/service-2.json`,
`develop` branch, fetched 2026-10-25): `bedrock` 2023-04-20 (108
operations), `bedrock-runtime` 2023-09-30 (11), `bedrock-agent-runtime`
2023-07-26 (35). Primary source for paths, members, enums, limits, error
status codes, auth lists, event-stream unions.
2. AWS Signature V4 test suite (`aws4_testsuite`, 24 cases, files `.req`
`.creq` `.sts` `.authz`), via the copy in the botocore repository
(`tests/unit/auth/aws4_testsuite`). Credentials of the suite are AWS's
published example ones.
3. AWS API reference pages fetched 2026-10-25: `API_runtime_Converse.html`,
`API_runtime_ConverseStream.html`, `API_agent-runtime_Retrieve.html`
(sample requests and responses used as fixtures).
4. AWS documentation "Examples of how to derive a signing key for Signature
Version 4" (key vector for `iam`, 20150830, us-east-1; recalled, matched by
our computation, not re-fetched).
5. Live unauthenticated probes (below).
Cross-checks: service models and reference pages agree on paths, members and
the Converse sample bodies; the suite and the key vector agree on the signing
chain.
## Verified against spec
`python3 -I src/vendors/bedrock/tools/check_spec.py` on 2026-10-25: 18 rows,
29 schema rows, 154 operations in the three models, 136 excluded with a reason,
0 problems. It checks, per row: operation exists, HTTP method, request URI,
URI placeholders equal the declared path params, every path / query / header /
body member exists with the right location and location name, every required
input member is covered, the response schema equals the operation output (the
event-stream union for stream rows), `bearer_ok` equals `httpBearerAuth` in the
model's auth list, `event_stream` equals the model, every member we read exists
(dotted paths descend); and fails on any operation neither in the table nor in
`EXCLUDED`. A mutated copy of `calls.cpp` was run to prove it fails.
SigV4: all 24 cases of the AWS test suite (96 files under
`tests/src/vendors/bedrock/sigv4_suite/`) give the suite's canonical request,
string to sign and Authorization header; the documented `iam` signing key
`c4afb1cc...a4b9` matches.
## Verified live (2026-10-25, no account)
`MOLO_LIVE_NET=1 ./build/ai_test` (tests `bedrock_live.*`):
- Control plane, `GET /foundation-models` with an invalid API key: 403
`AccessDeniedException`, "Invalid API Key format: Must start with pre-defined
prefix" (the host resolves, TLS works, the Bearer path is routed).
- With the documentation SigV4 key (`AKIDEXAMPLE`) on all three hosts
(`GET /foundation-models`, `POST /model/<id>/converse`,
`POST /knowledgebases/<id>/retrieve`): 403 `UnrecognizedClientException`,
"The security token included in the request is invalid." This is the answer
to a well-formed, correctly routed request with an unknown key, and shows the
error type header and message shape the client parses.
- `POST /model/<id>/converse-stream` with the same key: a JSON error document
instead of a stream, thrown as `STREAM_HTTP_ERROR`.
## Unverified
- No successful response was seen from any host. Converse / Retrieve success
bodies are the AWS reference samples (Converse) or shaped by the service
model (everything else); tests label which.
- The event-stream byte layout (prelude, CRCs, header types) was written from
the public description of the format, not compared with a recorded real
Bedrock stream. CRC-32 itself is checked with the standard check value
(`CBF43926`). The `:event-type` names are the union member names of the
service model.
- A real SigV4 signature accepted by AWS: the signer matches the published
suite, and the live probes got past routing, but only a key can prove an
accepted signature (the signed header set `content-type;host;x-amz-date` and
the double-encoded model path are the likely failure points).
- Whether Bedrock API keys work on `ListGuardrails`, batch and the other
control plane rows: the service model lists `httpBearerAuth` for the whole
`bedrock` model, AWS's documentation of key limits was not fetched.
- The agent runtime model lists no bearer auth; whether the host accepts API
keys anyway is unknown (the client refuses them by the model).
- Default model `us.amazon.nova-micro-v1:0` (used by the demo and
`BedrockChatModel`): model access and the inference profile must exist in
the account and region; not verified.
- The JSON-schema output (`outputConfig.textFormat`, schema as a string) and
`serviceTier` / `performanceConfig` members come from the model, not from a
call.
- `+` in a query string is treated as data when signing (`%2B`); our own
requests encode it, but a hand-written query with `+` meaning space would
sign differently from AWS's reading.
- Region endpoints other than `<prefix>.<region>.amazonaws.com[.cn]` (FIPS,
VPC endpoints, GovCloud variants) are not modelled.
## Gaps
- `src/llm` has no signing hook: `ApiClient` cannot sign per request, so this
package carries `BedrockClient` and `BedrockService` (foundation gap; a
`RequestSigner` seam in `ApiClient::build` would let vendors share it).
- The transport exposes no HTTP status or headers for streamed responses, so a
failed stream call reports 502 `STREAM_HTTP_ERROR` instead of the real status
and error type (`src/clients` `Transport::send_stream`).
- `InvokeModelWithBidirectionalStream` needs HTTP/2 framing the client does not
have; async invoke, sessions, flows, RetrieveAndGenerate, Rerank and inline
agents are listed in `EXCLUDED` and could be added row by row.
- Responses are mostly `Json` accessors, not typed classes (models, profiles,
guardrails, jobs, retrieval results).
- No credential provider chain (shared credentials file, IMDS, SSO, AssumeRole):
environment only.
- Requests are signed once; a clock skew over 5 minutes fails with the
server's own error, there is no resign-and-retry.
+166
View File
@@ -0,0 +1,166 @@
#include "vendors/bedrock/agent.h"
#include "security/crypto/Base64.h"
#include "vendors/bedrock/caps.h"
namespace molo::bedrock {
namespace {
constexpr std::size_t kMaxTraces = 200;
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("bedrock", why); }
bool alnum(const std::string& s, std::size_t min, std::size_t max, const char* extra = "") {
if (s.size() < min || s.size() > max) return false;
for (char c : s) {
if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')) continue;
if (*extra && std::string(extra).find(c) != std::string::npos) continue;
return false;
}
return true;
}
} // namespace
void AgentInvokeParams::validate() const {
if (!alnum(agent_id, 1, 10)) bad("agentId must be 1..10 letters or digits");
if (!alnum(alias_id, 1, 10)) bad("agentAliasId must be 1..10 letters or digits");
if (!alnum(session_id, 2, 100, "._:-")) bad("sessionId must be 2..100 of [0-9a-zA-Z._:-]");
if (input_text.empty() && session_state.is_object() && session_state.as_object().empty() && !end_session)
bad("inputText is required");
if (input_text.size() > BedrockCaps::kMaxAgentInputChars) bad("inputText too long");
if (!memory_id.empty() && !alnum(memory_id, 2, 100, "._:-")) bad("memoryId must be 2..100 of [0-9a-zA-Z._:-]");
if (!session_state.is_object()) bad("sessionState must be an object");
}
Json AgentInvokeParams::to_json() const {
validate();
Json b = Json::object();
if (!input_text.empty()) b.set("inputText", Json(input_text));
if (enable_trace) b.set("enableTrace", Json(true));
if (end_session) b.set("endSession", Json(true));
if (!memory_id.empty()) b.set("memoryId", Json(memory_id));
if (!session_state.as_object().empty()) b.set("sessionState", session_state);
if (stream_final_response) {
Json s = Json::object();
s.set("streamFinalResponse", Json(true));
b.set("streamingConfigurations", s);
}
return b;
}
void RetrieveParams::validate() const {
if (!alnum(knowledge_base_id, 10, 10)) bad("knowledgeBaseId must be 10 letters or digits");
if (query.empty() || query.size() > BedrockCaps::kMaxRetrieveQueryChars) bad("query must be 1..20000 characters");
if (number_of_results != 0 && (number_of_results < 1 || number_of_results > BedrockCaps::kMaxRetrieveResults))
bad("numberOfResults must be 1..100");
if (!search_type.empty() && search_type != "HYBRID" && search_type != "SEMANTIC") bad("overrideSearchType must be HYBRID or SEMANTIC");
if (!filter.is_null() && !filter.is_object()) bad("filter must be an object");
if (next_token.size() > 2048) bad("nextToken too long");
if (!guardrail_id.empty() && guardrail_version.empty()) bad("guardrail version is required with a guardrail");
}
Json RetrieveParams::to_json() const {
validate();
Json b = Json::object();
Json q = Json::object();
q.set("text", Json(query));
b.set("retrievalQuery", q);
Json vs = Json::object();
if (number_of_results != 0) vs.set("numberOfResults", Json(number_of_results));
if (!search_type.empty()) vs.set("overrideSearchType", Json(search_type));
if (filter.is_object()) vs.set("filter", filter);
if (!vs.as_object().empty()) {
Json rc = Json::object();
rc.set("vectorSearchConfiguration", vs);
b.set("retrievalConfiguration", rc);
}
if (!guardrail_id.empty()) {
Json g = Json::object();
g.set("guardrailId", Json(guardrail_id));
g.set("guardrailVersion", Json(guardrail_version));
b.set("guardrailConfiguration", g);
}
if (!next_token.empty()) b.set("nextToken", Json(next_token));
return b;
}
AgentResult AgentRuntime::invoke_agent(const AgentInvokeParams& params,
const std::function<bool(const std::string&)>& on_text) {
Json body = params.to_json();
CallInput in;
in.call = Call::kInvokeAgent;
in.path["agentId"] = params.agent_id;
in.path["agentAliasId"] = params.alias_id;
in.path["sessionId"] = params.session_id;
in.json = &body;
AgentResult result;
result.session_id = params.session_id;
EventStreamDecoder decoder(BedrockClient::vendor(), [&](const EventMessage& m) {
EventStreamDecoder::raise_if_failure(BedrockClient::vendor(), m);
if (!m.is_event()) return true;
Json p = m.payload.empty() ? Json::object() : Json::parse(m.payload);
std::string type = m.event_type();
if (type == "chunk") {
const Json* b = p.find("bytes");
std::string decoded;
if (!b || !b->is_string() || !security::crypto::Base64::decode(b->as_string(), decoded))
throw llm::ApiError(BedrockClient::vendor(), 502, "BAD_RESPONSE", "chunk without valid base64 bytes");
++result.chunks;
result.text += decoded;
if (const Json* a = p.find("attribution"); a && result.attributions.as_array().size() < kMaxTraces)
result.attributions.push(*a);
if (on_text && !on_text(decoded)) return false;
} else if (type == "trace") {
if (result.traces.as_array().size() < kMaxTraces) result.traces.push(p);
else result.traces_truncated = true;
} else if (type == "returnControl") {
result.return_control = p;
}
return true;
});
client_->stream(in, [&](const std::string& chunk) { return decoder.feed(chunk); });
decoder.finish();
return result;
}
RetrieveResult AgentRuntime::retrieve(const RetrieveParams& params) {
Json body = params.to_json();
CallInput in;
in.call = Call::kRetrieve;
in.path["knowledgeBaseId"] = params.knowledge_base_id;
in.json = &body;
Json r = client_->json(in);
RetrieveResult out;
out.pages = 1;
if (const Json* a = r.find("retrievalResults"); a && a->is_array()) out.results = *a;
out.next_token = string_at(r, "nextToken");
out.guardrail_action = string_at(r, "guardrailAction");
return out;
}
RetrieveResult AgentRuntime::retrieve_all(RetrieveParams params, std::size_t max_pages) {
if (max_pages < 1 || max_pages > BedrockCaps::kMaxPages) bad("max_pages must be 1..20");
RetrieveResult all;
for (std::size_t page = 0; page < max_pages; ++page) {
RetrieveResult r = retrieve(params);
++all.pages;
for (const Json& x : r.results.as_array()) {
if (all.results.as_array().size() >= BedrockCaps::kMaxItemsReturned) {
all.truncated = true;
all.next_token = r.next_token;
return all;
}
all.results.push(x);
}
all.guardrail_action = r.guardrail_action;
all.next_token = r.next_token;
if (r.next_token.empty()) return all;
params.next_token = r.next_token;
}
all.truncated = !all.next_token.empty();
return all;
}
} // namespace molo::bedrock
+70
View File
@@ -0,0 +1,70 @@
#pragma once
// bedrock-agent-runtime (host bedrock-agent-runtime.<region>.amazonaws.com):
// InvokeAgent (event stream) and knowledge base Retrieve. This host takes SigV4
// only; Bedrock API keys are refused before any traffic.
#include <functional>
#include <string>
#include "vendors/bedrock/client.h"
#include "vendors/bedrock/control.h"
namespace molo::bedrock {
struct AgentInvokeParams {
std::string agent_id; // [0-9a-zA-Z]{1,10}
std::string alias_id; // [0-9a-zA-Z]{1,10}
std::string session_id; // [0-9a-zA-Z._:-]{2,100}
std::string input_text; // up to 25,000,000 characters
bool enable_trace = false;
bool end_session = false;
std::string memory_id; // optional, same pattern as the session id
Json session_state = Json::object(); // sessionState passthrough, optional
bool stream_final_response = false; // streamingConfigurations.streamFinalResponse
void validate() const;
Json to_json() const;
};
struct AgentResult {
std::string text;
std::string session_id;
std::size_t chunks = 0;
Json traces = Json::array(); // trace events, capped
Json return_control = Json(); // returnControl payload when the agent asks the caller to act
Json attributions = Json::array();
bool traces_truncated = false;
};
struct RetrieveParams {
std::string knowledge_base_id; // [0-9a-zA-Z]{10}
std::string query; // 1..20000 characters
int number_of_results = 0; // 0: service default, else 1..100
std::string search_type; // "" | HYBRID | SEMANTIC
Json filter; // optional RetrievalFilter union, passthrough
std::string next_token;
std::string guardrail_id, guardrail_version;
void validate() const;
Json to_json() const;
};
struct RetrieveResult {
Json results = Json::array(); // retrievalResults
std::string next_token;
std::string guardrail_action;
bool truncated = false; // only set by retrieve_all
std::size_t pages = 0;
};
class AgentRuntime : public BedrockService {
public:
using BedrockService::BedrockService;
// on_text gets each decoded text chunk; return false to stop. Costs money.
AgentResult invoke_agent(const AgentInvokeParams& params,
const std::function<bool(const std::string&)>& on_text = nullptr);
RetrieveResult retrieve(const RetrieveParams& params);
// Follows nextToken up to `max_pages` (<= BedrockCaps::kMaxPages) pages.
RetrieveResult retrieve_all(RetrieveParams params, std::size_t max_pages);
};
} // namespace molo::bedrock
+122
View File
@@ -0,0 +1,122 @@
#include "vendors/bedrock/calls.h"
#include "clients/url.h"
#include "llm/error.h"
#include "vendors/bedrock/caps.h"
namespace molo::bedrock {
namespace {
constexpr AwsService R = AwsService::kRuntime;
constexpr AwsService C = AwsService::kControl;
constexpr AwsService A = AwsService::kAgentRuntime;
// One row per line: the checker reads this file with a regex. Quoted strings
// in order: op, http method, uri, path params, query params, header params,
// body fields, response schema.
const CallSpec kCalls[] = {
{Call::kConverse, R, "Converse", "POST", "/model/{modelId}/converse", "modelId", "", "", "messages,system,inferenceConfig,toolConfig,guardrailConfig,additionalModelRequestFields,promptVariables,additionalModelResponseFieldPaths,requestMetadata,performanceConfig,serviceTier,outputConfig", "ConverseResponse", true, false, true},
{Call::kConverseStream, R, "ConverseStream", "POST", "/model/{modelId}/converse-stream", "modelId", "", "", "messages,system,inferenceConfig,toolConfig,guardrailConfig,additionalModelRequestFields,promptVariables,additionalModelResponseFieldPaths,requestMetadata,performanceConfig,serviceTier,outputConfig", "ConverseStreamOutput", true, true, true},
{Call::kInvokeModel, R, "InvokeModel", "POST", "/model/{modelId}/invoke", "modelId", "", "Content-Type,Accept,X-Amzn-Bedrock-Trace,X-Amzn-Bedrock-GuardrailIdentifier,X-Amzn-Bedrock-GuardrailVersion,X-Amzn-Bedrock-PerformanceConfig-Latency,X-Amzn-Bedrock-Service-Tier", "body", "InvokeModelResponse", true, false, true},
{Call::kInvokeModelStream, R, "InvokeModelWithResponseStream", "POST", "/model/{modelId}/invoke-with-response-stream", "modelId", "", "Content-Type,X-Amzn-Bedrock-Accept,X-Amzn-Bedrock-Trace,X-Amzn-Bedrock-GuardrailIdentifier,X-Amzn-Bedrock-GuardrailVersion,X-Amzn-Bedrock-PerformanceConfig-Latency,X-Amzn-Bedrock-Service-Tier", "body", "ResponseStream", true, true, true},
{Call::kCountTokens, R, "CountTokens", "POST", "/model/{modelId}/count-tokens", "modelId", "", "", "input", "CountTokensResponse", true, false, false},
{Call::kApplyGuardrail, R, "ApplyGuardrail", "POST", "/guardrail/{guardrailIdentifier}/version/{guardrailVersion}/apply", "guardrailIdentifier,guardrailVersion", "", "", "source,content,outputScope", "ApplyGuardrailResponse", true, false, true},
{Call::kInvokeAgent, A, "InvokeAgent", "POST", "/agents/{agentId}/agentAliases/{agentAliasId}/sessions/{sessionId}/text", "agentId,agentAliasId,sessionId", "", "", "inputText,enableTrace,endSession,memoryId,sessionState,streamingConfigurations", "ResponseStream", false, true, true},
{Call::kRetrieve, A, "Retrieve", "POST", "/knowledgebases/{knowledgeBaseId}/retrieve", "knowledgeBaseId", "", "", "retrievalQuery,retrievalConfiguration,guardrailConfiguration,nextToken", "RetrieveResponse", false, false, false},
{Call::kListFoundationModels, C, "ListFoundationModels", "GET", "/foundation-models", "", "byProvider,byCustomizationType,byOutputModality,byInferenceType", "", "", "ListFoundationModelsResponse", true, false, false},
{Call::kGetFoundationModel, C, "GetFoundationModel", "GET", "/foundation-models/{modelIdentifier}", "modelIdentifier", "", "", "", "GetFoundationModelResponse", true, false, false},
{Call::kListInferenceProfiles, C, "ListInferenceProfiles", "GET", "/inference-profiles", "", "maxResults,nextToken,type", "", "", "ListInferenceProfilesResponse", true, false, false},
{Call::kGetInferenceProfile, C, "GetInferenceProfile", "GET", "/inference-profiles/{inferenceProfileIdentifier}", "inferenceProfileIdentifier", "", "", "", "GetInferenceProfileResponse", true, false, false},
{Call::kListGuardrails, C, "ListGuardrails", "GET", "/guardrails", "", "guardrailIdentifier,maxResults,nextToken", "", "", "ListGuardrailsResponse", true, false, false},
{Call::kGetGuardrail, C, "GetGuardrail", "GET", "/guardrails/{guardrailIdentifier}", "guardrailIdentifier", "guardrailVersion", "", "", "GetGuardrailResponse", true, false, false},
{Call::kCreateBatchJob, C, "CreateModelInvocationJob", "POST", "/model-invocation-job", "", "", "", "jobName,roleArn,clientRequestToken,modelId,inputDataConfig,outputDataConfig,timeoutDurationInHours,tags,modelInvocationType", "CreateModelInvocationJobResponse", true, false, true},
{Call::kGetBatchJob, C, "GetModelInvocationJob", "GET", "/model-invocation-job/{jobIdentifier}", "jobIdentifier", "", "", "", "GetModelInvocationJobResponse", true, false, false},
{Call::kListBatchJobs, C, "ListModelInvocationJobs", "GET", "/model-invocation-jobs", "", "statusEquals,nameContains,maxResults,nextToken,sortBy,sortOrder", "", "", "ListModelInvocationJobsResponse", true, false, false},
{Call::kStopBatchJob, C, "StopModelInvocationJob", "POST", "/model-invocation-job/{jobIdentifier}/stop", "jobIdentifier", "", "", "", "StopModelInvocationJobResponse", true, false, true},
};
// Members we read from replies; dotted names descend into structures.
const SchemaFields kSchemas[] = {
{"ConverseResponse", "bedrock-runtime", "output.message,stopReason,usage.inputTokens,usage.outputTokens,usage.totalTokens,usage.cacheReadInputTokens,usage.cacheWriteInputTokens,metrics.latencyMs,additionalModelResponseFields,trace,performanceConfig,serviceTier"},
{"ConverseStreamOutput", "bedrock-runtime", "messageStart,contentBlockStart,contentBlockDelta,contentBlockStop,messageStop,metadata,internalServerException,modelStreamErrorException,validationException,throttlingException,serviceUnavailableException"},
{"ContentBlockDeltaEvent", "bedrock-runtime", "delta,contentBlockIndex"},
{"ContentBlockStartEvent", "bedrock-runtime", "start,contentBlockIndex"},
{"MessageStopEvent", "bedrock-runtime", "stopReason,additionalModelResponseFields"},
{"ConverseStreamMetadataEvent", "bedrock-runtime", "usage.inputTokens,usage.outputTokens,usage.totalTokens,usage.cacheReadInputTokens,usage.cacheWriteInputTokens,metrics.latencyMs"},
{"InvokeModelResponse", "bedrock-runtime", "body,contentType"},
{"ResponseStream", "bedrock-runtime", "chunk,internalServerException,modelStreamErrorException,validationException,throttlingException,modelTimeoutException,serviceUnavailableException"},
{"PayloadPart", "bedrock-runtime", "bytes"},
{"CountTokensResponse", "bedrock-runtime", "inputTokens"},
{"ApplyGuardrailResponse", "bedrock-runtime", "usage,action,actionReason,outputs,assessments,guardrailCoverage"},
{"RetrieveResponse", "bedrock-agent-runtime", "retrievalResults,nextToken,guardrailAction"},
{"KnowledgeBaseRetrievalResult", "bedrock-agent-runtime", "content.text,content.type,content.byteContent,documentId,location.type,location.s3Location.uri,location.webLocation.url,metadata,score"},
{"ResponseStream", "bedrock-agent-runtime", "chunk,trace,returnControl,files,accessDeniedException,badGatewayException,conflictException,dependencyFailedException,internalServerException,modelNotReadyException,resourceNotFoundException,serviceQuotaExceededException,throttlingException,validationException"},
{"PayloadPart", "bedrock-agent-runtime", "bytes,attribution"},
{"ListFoundationModelsResponse", "bedrock", "modelSummaries"},
{"FoundationModelSummary", "bedrock", "modelArn,modelId,modelName,providerName,inputModalities,outputModalities,responseStreamingSupported,customizationsSupported,inferenceTypesSupported,modelLifecycle.status"},
{"GetFoundationModelResponse", "bedrock", "modelDetails"},
{"GetInferenceProfileResponse", "bedrock", "inferenceProfileName,inferenceProfileArn,inferenceProfileId,models,status,type,description"},
{"ListInferenceProfilesResponse", "bedrock", "inferenceProfileSummaries,nextToken"},
{"InferenceProfileSummary", "bedrock", "inferenceProfileName,inferenceProfileArn,inferenceProfileId,models,status,type,description"},
{"ListGuardrailsResponse", "bedrock", "guardrails,nextToken"},
{"GuardrailSummary", "bedrock", "id,arn,status,name,description,version,createdAt,updatedAt"},
{"GetGuardrailResponse", "bedrock", "name,description,guardrailId,guardrailArn,version,status,blockedInputMessaging,blockedOutputsMessaging,createdAt,updatedAt"},
{"CreateModelInvocationJobResponse", "bedrock", "jobArn"},
{"GetModelInvocationJobResponse", "bedrock", "jobArn,jobName,modelId,status,message,submitTime,lastModifiedTime,endTime,inputDataConfig,outputDataConfig,timeoutDurationInHours,totalRecordCount,processedRecordCount,successRecordCount,errorRecordCount"},
{"ListModelInvocationJobsResponse", "bedrock", "invocationJobSummaries,nextToken"},
{"ModelInvocationJobSummary", "bedrock", "jobArn,jobName,modelId,status,message,submitTime,lastModifiedTime,endTime,totalRecordCount,processedRecordCount,successRecordCount,errorRecordCount"},
{"StopModelInvocationJobResponse", "bedrock", ""},
};
constexpr std::size_t kCallCount = sizeof(kCalls) / sizeof(kCalls[0]);
constexpr std::size_t kSchemaCount = sizeof(kSchemas) / sizeof(kSchemas[0]);
} // namespace
const CallSpec& Calls::spec_of(Call call) {
for (const CallSpec& s : kCalls)
if (s.call == call) return s;
throw llm::ApiError::invalid("bedrock", "unknown call");
}
clients::Method Calls::method_of(Call call) {
return std::string(spec_of(call).http_method) == "GET" ? clients::Method::kGet : clients::Method::kPost;
}
std::size_t Calls::count() { return kCallCount; }
const CallSpec& Calls::at(std::size_t index) { return kCalls[index]; }
std::size_t Calls::schema_count() { return kSchemaCount; }
const SchemaFields& Calls::schema_at(std::size_t index) { return kSchemas[index]; }
bool Calls::valid_path_value(const std::string& value) {
if (value.empty() || value.size() > BedrockCaps::kMaxModelIdChars) return false;
if (value.find("..") != std::string::npos) return false;
for (char c : value)
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' ||
c == '_' || c == '.' || c == ':' || c == '/'))
return false;
return true;
}
std::string Calls::render(Call call, const PathArgs& args) {
std::string uri = spec_of(call).uri;
std::string out;
std::size_t i = 0;
while (i < uri.size()) {
if (uri[i] != '{') {
out.push_back(uri[i++]);
continue;
}
std::size_t close = uri.find('}', i);
std::string name = uri.substr(i + 1, close - i - 1);
auto it = args.find(name);
if (it == args.end()) throw llm::ApiError::invalid("bedrock", "missing " + name);
if (!valid_path_value(it->second)) throw llm::ApiError::invalid("bedrock", "bad " + name);
out += clients::Url::encode_component(it->second);
i = close + 1;
}
return out;
}
} // namespace molo::bedrock
+65
View File
@@ -0,0 +1,65 @@
#pragma once
// The ONE table of REST calls this module makes. tools/check_spec.py checks
// every row against the AWS service models (botocore service-2.json for
// bedrock, bedrock-runtime, bedrock-agent-runtime): operation name, HTTP
// method, request URI, path / query / header / body members, bearer support,
// and the response fields we read. Code builds every path from this table.
#include <cstddef>
#include <map>
#include <string>
#include "clients/request.h"
#include "vendors/bedrock/credentials.h"
namespace molo::bedrock {
enum class Call {
kConverse, kConverseStream, kInvokeModel, kInvokeModelStream, kCountTokens, kApplyGuardrail,
kInvokeAgent, kRetrieve,
kListFoundationModels, kGetFoundationModel, kListInferenceProfiles, kGetInferenceProfile,
kListGuardrails, kGetGuardrail,
kCreateBatchJob, kGetBatchJob, kListBatchJobs, kStopBatchJob,
};
struct CallSpec {
Call call;
AwsService service;
const char* op; // service model operation name
const char* http_method; // GET POST
const char* uri; // request URI template, "{name}" placeholders
const char* path_params; // comma list, same names as the placeholders
const char* query_params; // comma list of query string names we send
const char* header_params; // comma list of header names we send
const char* body_fields; // comma list of JSON body members we send ("body" = raw payload)
const char* response; // response schema name in the schema table
bool bearer_ok; // service model lists httpBearerAuth (Bedrock API keys)
bool event_stream; // the success reply is application/vnd.amazon.eventstream
bool changes_state; // creates, stops or costs money: demo needs confirm
};
struct SchemaFields {
const char* schema;
const char* service; // which service model: bedrock, bedrock-runtime, bedrock-agent-runtime
const char* fields; // comma list of members we read, dotted for nested ("usage.inputTokens")
};
using PathArgs = std::map<std::string, std::string>;
using QueryArgs = std::map<std::string, std::string>;
struct Calls {
static const CallSpec& spec_of(Call call);
static clients::Method method_of(Call call);
static std::size_t count();
static const CallSpec& at(std::size_t index);
static std::size_t schema_count();
static const SchemaFields& schema_at(std::size_t index);
// The URI with every {placeholder} replaced by the percent-encoded value;
// throws ApiError::invalid for a missing, empty, over-long or odd value.
static std::string render(Call call, const PathArgs& args);
// Allowed characters of an id placed in a path: [A-Za-z0-9._:/-], no "..".
static bool valid_path_value(const std::string& value);
};
} // namespace molo::bedrock
+42
View File
@@ -0,0 +1,42 @@
#pragma once
// Static ceilings for the Bedrock module. Never autotuned. Numbers marked
// (spec) come from the AWS service models (see SPEC.md ## Sources), the rest
// are our own hard caps.
#include <cstddef>
namespace molo::bedrock {
struct BedrockCaps {
static constexpr std::size_t kMaxInvokeBodyBytes = 25000000; // (spec) blob max
static constexpr std::size_t kMaxResponseBytes = 64u * 1024u * 1024u;
static constexpr std::size_t kMaxStreamBytes = 64u * 1024u * 1024u;
// Event stream framing: AWS documents 16 MiB per message and 128 KiB of headers.
static constexpr std::size_t kMaxEventMessageBytes = 16u * 1024u * 1024u;
static constexpr std::size_t kMaxEventHeaderBytes = 128u * 1024u;
static constexpr std::size_t kMaxStreamEvents = 200000;
static constexpr std::size_t kMaxModelIdChars = 2048; // (spec)
static constexpr std::size_t kMaxRegionChars = 32;
static constexpr std::size_t kMaxMessages = 4096;
static constexpr std::size_t kMaxTools = 128;
static constexpr std::size_t kMaxStopSequences = 2500; // (spec)
static constexpr std::size_t kMaxContentBlocks = 512;
static constexpr std::size_t kMaxSystemBlocks = 64;
static constexpr std::size_t kMaxSignedHeaders = 64;
static constexpr std::size_t kMaxQueryParams = 64;
static constexpr std::size_t kMaxPathChars = 4096;
static constexpr int kMaxResults = 1000; // (spec) list calls
static constexpr std::size_t kMaxPages = 20; // list walks
static constexpr std::size_t kMaxItemsReturned = 5000;
static constexpr int kMaxRetrieveResults = 100; // (spec)
static constexpr std::size_t kMaxRetrieveQueryChars = 20000; // (spec)
static constexpr std::size_t kMaxAgentInputChars = 25000000; // (spec)
static constexpr std::size_t kMaxGuardrailBlocks = 64;
static constexpr std::size_t kMaxGuardrailTextChars = 1000000;
static constexpr int kMinBatchTimeoutHours = 24; // (spec)
static constexpr int kMaxBatchTimeoutHours = 168; // (spec)
static constexpr std::size_t kMaxJobNameChars = 63; // (spec)
};
} // namespace molo::bedrock
+193
View File
@@ -0,0 +1,193 @@
#include "vendors/bedrock/chat_model.h"
namespace molo::bedrock {
namespace {
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("bedrock", why); }
std::string image_format(const std::string& mime) {
if (mime == "image/png") return "png";
if (mime == "image/jpeg" || mime == "image/jpg") return "jpeg";
if (mime == "image/gif") return "gif";
if (mime == "image/webp") return "webp";
bad("image mime must be image/png, image/jpeg, image/gif or image/webp");
}
const char* role_of(llm::Role r) { return r == llm::Role::kAssistant ? "assistant" : "user"; }
std::string str(const Json* v) { return v && v->is_string() ? v->as_string() : std::string(); }
} // namespace
BedrockChatModel::BedrockChatModel(std::shared_ptr<Runtime> runtime, std::string default_model)
: runtime_(std::move(runtime)), default_model_(std::move(default_model)) {}
std::shared_ptr<BedrockChatModel> BedrockChatModel::from_env(const std::string& default_model) {
return std::make_shared<BedrockChatModel>(std::make_shared<Runtime>(BedrockClient::from_env()), default_model);
}
std::string BedrockChatModel::finish_of(const std::string& s) {
if (s == "end_turn" || s == "stop_sequence") return "stop";
if (s == "max_tokens" || s == "model_context_window_exceeded") return "length";
if (s == "tool_use") return "tool_calls";
if (s == "guardrail_intervened" || s == "content_filtered") return "content_filter";
return s;
}
ConverseRequest BedrockChatModel::to_converse(const llm::ChatRequest& r) const {
ConverseRequest out;
out.model_id = r.model.empty() ? default_model_ : r.model;
if (!r.system.empty()) out.add_system(r.system);
// Build messages, merging consecutive same-role ones.
std::string last_role;
std::vector<Json> pending;
auto flush = [&] {
if (!pending.empty()) out.add(Blocks::message(last_role, pending));
pending.clear();
};
for (const llm::Message& m : r.messages) {
if (m.role == llm::Role::kSystem) {
if (!m.text().empty()) out.add_system(m.text());
continue;
}
std::string role = role_of(m.role);
std::vector<Json> blocks;
for (const llm::ContentPart& p : m.parts) {
if (!p.reasoning.empty()) {
if (m.role == llm::Role::kAssistant && !p.reasoning_signature.empty()) {
Json rt = Json::object();
rt.set("text", Json(p.reasoning));
rt.set("signature", Json(p.reasoning_signature));
Json rc = Json::object();
rc.set("reasoningText", rt);
Json b = Json::object();
b.set("reasoningContent", rc);
blocks.push_back(b);
}
}
if (!p.text.empty()) blocks.push_back(Blocks::text(p.text));
if (p.image) {
if (p.image->bytes.empty()) bad("Converse takes image bytes, not image URLs");
blocks.push_back(Blocks::image(image_format(p.image->mime), p.image->bytes));
}
if (p.tool_call) blocks.push_back(Blocks::tool_use(p.tool_call->id, p.tool_call->name, p.tool_call->arguments));
if (p.tool_result) blocks.push_back(Blocks::tool_result(p.tool_result->call_id, p.tool_result->content, p.tool_result->is_error));
}
if (blocks.empty()) continue;
if (role != last_role) flush();
last_role = role;
for (Json& b : blocks) pending.push_back(std::move(b));
}
flush();
if (r.max_tokens) out.max_tokens = r.max_tokens;
out.temperature = r.temperature;
out.top_p = r.top_p;
out.stop_sequences = r.stop;
if (r.tool_choice != "none") {
for (const llm::ToolSpec& t : r.tools) out.tools.push_back(Blocks::tool(t.name, t.description, t.parameters));
if (!out.tools.empty()) {
if (r.tool_choice == "required") out.tool_choice = "any";
else out.tool_choice = r.tool_choice; // "", "auto" or a tool name
}
}
if (r.json_mode && !r.response_schema.is_object())
bad("Converse has no JSON mode: pass response_schema (outputConfig.textFormat)");
if (r.response_schema.is_object()) out.output_schema = r.response_schema;
out.effort = r.reasoning_effort;
if (r.extra.is_object()) {
for (const auto& [key, value] : r.extra.as_object()) {
if (key == "additionalModelRequestFields") {
if (!value.is_object()) bad("additionalModelRequestFields must be an object");
out.additional_fields = value;
} else if (key == "guardrailIdentifier") out.guardrail_id = str(&value);
else if (key == "guardrailVersion") out.guardrail_version = str(&value);
else if (key == "guardrailTrace") out.guardrail_trace = str(&value);
else if (key == "serviceTier") out.service_tier = str(&value);
else if (key == "latency") out.latency = str(&value);
else if (key == "requestMetadata") {
if (!value.is_object()) bad("requestMetadata must be an object");
for (const auto& [k, v] : value.as_object()) {
if (!v.is_string()) bad("requestMetadata values must be strings");
out.request_metadata[k] = v.as_string();
}
} else {
bad("unsupported extra key: " + key);
}
}
}
return out;
}
llm::ChatResponse BedrockChatModel::to_response(const ConverseResult& result, const std::string& model) {
llm::ChatResponse out;
out.model = model;
out.message.role = llm::Role::kAssistant;
std::string reasoning = result.reasoning();
if (!reasoning.empty()) {
llm::ContentPart p;
p.reasoning = reasoning;
p.reasoning_signature = result.reasoning_signature();
out.message.parts.push_back(std::move(p));
}
std::string text = result.text();
if (!text.empty()) out.message.parts.push_back(llm::ContentPart::of_text(text));
for (const ConverseToolUse& u : result.tool_uses()) {
llm::ToolCall c;
c.id = u.id;
c.name = u.name;
c.arguments = u.input;
out.message.parts.push_back(llm::ContentPart::of_call(std::move(c)));
}
out.finish_reason = finish_of(result.stop_reason());
ConverseUsage u = result.usage();
out.usage.input_tokens = u.input_tokens;
out.usage.output_tokens = u.output_tokens;
out.usage.total_tokens = u.total_tokens;
out.usage.cached_tokens = u.cache_read_tokens;
out.raw = result.raw();
return out;
}
llm::ChatResponse BedrockChatModel::generate(const llm::ChatRequest& request) {
ConverseRequest wire = to_converse(request);
return to_response(runtime_->converse(wire), wire.model_id);
}
void BedrockChatModel::stream(const llm::ChatRequest& request,
const std::function<bool(const llm::ChatChunk&)>& on_chunk) {
ConverseRequest wire = to_converse(request);
runtime_->converse_stream(wire, [&](const StreamDelta& d) {
llm::ChatChunk c;
bool emit = false;
if (d.event == "contentBlockDelta") {
c.text_delta = d.text;
c.reasoning_delta = d.reasoning;
c.tool_args_delta = d.tool_args;
emit = !d.text.empty() || !d.reasoning.empty() || !d.tool_args.empty();
} else if (d.event == "contentBlockStart" && !d.tool_name.empty()) {
c.tool_call_id = d.tool_id;
c.tool_call_name = d.tool_name;
emit = true;
} else if (d.event == "messageStop") {
c.finish_reason = finish_of(d.stop_reason);
emit = true;
} else if (d.event == "metadata" && d.usage) {
llm::Usage u;
u.input_tokens = d.usage->input_tokens;
u.output_tokens = d.usage->output_tokens;
u.total_tokens = d.usage->total_tokens;
u.cached_tokens = d.usage->cache_read_tokens;
c.usage = u;
emit = true;
}
return emit ? on_chunk(c) : true;
});
}
} // namespace molo::bedrock
+56
View File
@@ -0,0 +1,56 @@
#pragma once
// BedrockChatModel: the llm::ChatModel seam over Converse / ConverseStream.
//
// Mapping notes (each one is a decision, not a guess):
// - system text and Role::kSystem messages become Converse `system` blocks.
// - Role::kTool messages (tool results) become user-role toolResult blocks;
// consecutive messages of the same role are merged, because Converse needs
// alternating roles.
// - Assistant reasoning is replayed as reasoningContent only when it has a
// signature; unsigned reasoning is dropped (the service would reject it).
// - Images must carry bytes (Converse takes no image URLs): a URL-only image
// is rejected loudly. mime image/png|jpeg|gif|webp maps to the format enum.
// - tool_choice: auto -> auto, required -> any, a tool name -> tool, "none"
// drops the tools from the request (Converse has no "none").
// - json_mode alone is rejected (Converse has no JSON mode); response_schema
// maps to outputConfig.textFormat json_schema; reasoning_effort to
// outputConfig.effort.
// - extra accepts: additionalModelRequestFields (object), guardrailIdentifier,
// guardrailVersion, guardrailTrace, serviceTier, latency, requestMetadata
// (object of strings). Any other key is rejected.
// - finish_reason: end_turn / stop_sequence -> stop; max_tokens /
// model_context_window_exceeded -> length; tool_use -> tool_calls;
// guardrail_intervened / content_filtered -> content_filter; other values
// pass through.
#include <memory>
#include "llm/chat_types.h"
#include "vendors/bedrock/runtime.h"
namespace molo::bedrock {
class BedrockChatModel : public llm::ChatModel {
public:
static constexpr const char* kDefaultModel = "us.amazon.nova-micro-v1:0";
explicit BedrockChatModel(std::shared_ptr<Runtime> runtime, std::string default_model = kDefaultModel);
static std::shared_ptr<BedrockChatModel> from_env(const std::string& default_model = kDefaultModel);
std::string vendor() const override { return "bedrock"; }
llm::ChatResponse generate(const llm::ChatRequest& request) override;
void stream(const llm::ChatRequest& request,
const std::function<bool(const llm::ChatChunk&)>& on_chunk) override;
// The wire request a neutral request maps to (also used by tests).
ConverseRequest to_converse(const llm::ChatRequest& request) const;
static llm::ChatResponse to_response(const ConverseResult& result, const std::string& model);
static std::string finish_of(const std::string& stop_reason);
private:
std::shared_ptr<Runtime> runtime_;
std::string default_model_;
};
} // namespace molo::bedrock
+248
View File
@@ -0,0 +1,248 @@
#include "vendors/bedrock/client.h"
#include <algorithm>
#include <cctype>
#include "clients/transports/http_transport.h"
#include "llm/caps.h"
#include "vendors/bedrock/caps.h"
namespace molo::bedrock {
using clients::body::Json;
namespace {
clients::ClientConfig make_config() {
clients::ClientConfig c;
c.persist_cookies = false;
return c;
}
std::string lower(std::string s) {
for (char& c : s) c = static_cast<char>(std::tolower(static_cast<unsigned char>(c)));
return s;
}
std::vector<std::string> split(const std::string& list) {
std::vector<std::string> out;
std::string cur;
for (char c : list) {
if (c == ',') {
if (!cur.empty()) out.push_back(cur);
cur.clear();
} else {
cur.push_back(c);
}
}
if (!cur.empty()) out.push_back(cur);
return out;
}
bool listed(const std::string& list, const std::string& name) {
for (const std::string& item : split(list))
if (lower(item) == lower(name)) return true;
return false;
}
// "ValidationException:http://internal.amazon.com/..." or "com.x#Name" -> "Name".
std::string clean_error_type(std::string t) {
auto colon = t.find(':');
if (colon != std::string::npos) t.resize(colon);
auto hash = t.rfind('#');
if (hash != std::string::npos) t.erase(0, hash + 1);
return t;
}
std::string time_text(std::time_t t) { return SigV4Signer::format_amz_date(t); }
} // namespace
BedrockClient::BedrockClient(std::unique_ptr<clients::Transport> transport, Credentials credentials,
Clock clock)
: credentials_(std::move(credentials)),
clock_(clock ? std::move(clock) : Clock([] { return std::time(nullptr); })),
client_(std::move(transport), make_config()) {}
std::shared_ptr<BedrockClient> BedrockClient::from_env() {
return std::make_shared<BedrockClient>(std::make_unique<clients::transports::HttpTransport>(),
Credentials::from_env());
}
void BedrockClient::guard(const CallSpec& spec) const {
if (!credentials_.configured()) throw llm::NotConfiguredError(vendor(), Credentials::env_hint());
bool bearer = spec.bearer_ok && !credentials_.bearer.empty();
if (!bearer && !credentials_.aws.valid())
throw llm::ApiError(vendor(), 503, "NOT_CONFIGURED",
std::string(spec.op) + " needs SigV4 credentials (set " +
Credentials::kEnvAccessKey + " and " + Credentials::kEnvSecretKey +
"); Bedrock API keys do not work on this host");
}
BedrockClient::Built BedrockClient::build(const CallInput& in) const {
const CallSpec& spec = Calls::spec_of(in.call);
guard(spec);
if (in.json && in.raw) throw llm::ApiError::invalid(vendor(), "a call has one body");
std::string path = Calls::render(in.call, in.path);
if (path.size() > BedrockCaps::kMaxPathChars) throw llm::ApiError::invalid(vendor(), "path too long");
// Query: only names of the table, encoded and sorted so the wire form equals
// the canonical form.
if (in.query.size() > BedrockCaps::kMaxQueryParams)
throw llm::ApiError::invalid(vendor(), "too many query parameters");
std::vector<std::pair<std::string, std::string>> q;
for (const auto& [k, v] : in.query) {
if (!listed(spec.query_params, k)) throw llm::ApiError::invalid(vendor(), "query parameter " + k + " is not part of " + spec.op);
q.emplace_back(clients::Url::encode_component(k), clients::Url::encode_component(v));
}
std::sort(q.begin(), q.end());
std::string query;
for (const auto& [k, v] : q) {
if (!query.empty()) query.push_back('&');
query += k + "=" + v;
}
std::string body;
bool has_body = false;
std::string content_type = in.content_type;
if (in.json) {
body = in.json->dump();
has_body = true;
} else if (in.raw) {
body = *in.raw;
has_body = true;
}
if (body.size() > BedrockCaps::kMaxInvokeBodyBytes)
throw llm::ApiError::invalid(vendor(), "request body over the byte cap");
std::string host = Endpoints::host(spec.service, credentials_.region);
HeaderList headers;
headers.emplace_back("Host", host);
if (has_body) headers.emplace_back("Content-Type", content_type);
for (const auto& [name, value] : in.headers) {
if (!listed(spec.header_params, name))
throw llm::ApiError::invalid(vendor(), "header " + name + " is not part of " + spec.op);
if (lower(name) == "content-type") {
// The table lists Content-Type for the raw calls: it replaces the default.
for (auto& h : headers)
if (lower(h.first) == "content-type") h.second = value;
content_type = value;
continue;
}
headers.emplace_back(name, value);
}
Built built;
built.preview.method = spec.http_method;
built.preview.url = "https://" + host + path + (query.empty() ? "" : "?" + query);
bool bearer = spec.bearer_ok && !credentials_.bearer.empty();
HeaderList wire = headers;
if (bearer) {
built.preview.auth_mode = "bearer";
} else {
SigV4Signer signer(credentials_.aws, credentials_.region, Endpoints::signing_name());
SigV4Request sr;
sr.method = spec.http_method;
sr.path = path;
sr.query = query;
sr.headers = headers;
sr.body = body;
SigV4Result sig = signer.sign(sr, time_text(clock_()));
for (const auto& h : sig.added_headers) wire.push_back(h);
built.preview.auth_mode = "sigv4";
built.preview.signed_headers = sig.signed_headers;
built.preview.canonical_request = sig.canonical_request;
}
clients::ClientRequest req(Calls::method_of(in.call), clients::Url::parse(built.preview.url));
for (const auto& [name, value] : wire) req.header(name, value);
if (bearer) req.bearer(credentials_.bearer);
if (has_body) req.bytes(std::move(body), content_type);
// A POST without a body still says "Content-Length: 0" (the transport only
// sends the header when a body string exists); no Content-Type is signed.
else if (Calls::method_of(in.call) == clients::Method::kPost) req.bytes(std::string(), "");
built.request = std::move(req);
return built;
}
WirePreview BedrockClient::preview(const CallInput& input) const { return build(input).preview; }
void BedrockClient::throw_http(const clients::ClientResponse& response) {
std::string type = "UNKNOWN";
std::string message = "HTTP " + std::to_string(response.status());
std::string request_id = response.headers().get_or("x-amzn-RequestId", "");
if (auto h = response.headers().get("x-amzn-ErrorType"); h && !h->empty()) type = clean_error_type(*h);
try {
Json doc = response.json();
if (doc.is_object()) {
for (const char* key : {"message", "Message", "errorMessage"})
if (const Json* m = doc.find(key); m && m->is_string()) {
message = m->as_string();
break;
}
if (type == "UNKNOWN")
for (const char* key : {"__type", "code", "Code"})
if (const Json* t = doc.find(key); t && t->is_string()) {
type = clean_error_type(t->as_string());
break;
}
}
} catch (const std::exception&) {
// Not JSON: keep the generic message.
}
throw llm::ApiError(vendor(), response.status(), type, message, request_id);
}
Reply BedrockClient::raw(const CallInput& input) {
try {
Built built = build(input);
clients::ClientResponse response = client_.send_blocking(std::move(built.request));
if (!response.is_success()) throw_http(response);
if (response.body().size() > BedrockCaps::kMaxResponseBytes)
throw llm::ApiError(vendor(), 502, "BAD_RESPONSE", "response over the byte cap");
Reply r;
r.status = response.status();
r.body = response.body();
r.content_type = response.content_type();
r.headers = response.headers();
return r;
} catch (const llm::ApiError&) {
throw;
} catch (const std::exception& e) {
throw llm::ApiError(vendor(), 502, "UNAVAILABLE", e.what());
}
}
Json BedrockClient::json(const CallInput& input) {
Reply r = raw(input);
if (r.body.find_first_not_of(" \t\r\n") == std::string::npos) return Json::object();
try {
return Json::parse(r.body);
} catch (const std::exception& e) {
throw llm::ApiError(vendor(), 502, "BAD_RESPONSE", e.what());
}
}
void BedrockClient::stream(const CallInput& input, const std::function<bool(const std::string&)>& sink) {
try {
Built built = build(input);
client_.send_stream_blocking(std::move(built.request), sink);
} catch (const llm::ApiError&) {
throw;
} catch (const std::exception& e) {
throw llm::ApiError(vendor(), 502, "UNAVAILABLE", e.what());
}
}
void BedrockService::check_id(const std::string& what, const std::string& id) const {
if (!valid_id(id)) throw llm::ApiError::invalid("bedrock", "bad " + what + " id");
}
std::string BedrockService::string_at(const Json& doc, const std::string& key, const std::string& fallback) {
const Json* v = doc.is_object() ? doc.find(key) : nullptr;
return v && v->is_string() ? v->as_string() : fallback;
}
} // namespace molo::bedrock
+111
View File
@@ -0,0 +1,111 @@
#pragma once
// BedrockClient: the one place the module talks HTTP. llm::ApiClient cannot be
// used because SigV4 signs the final request (method, path, query, headers and
// body hash) per call, which a fixed bearer / header-key client cannot do.
// This class does the same job for Bedrock: builds the request from the call
// table, signs it (SigV4) or sends the Bedrock API key (bearer), and maps every
// failure to llm::ApiError.
#include <ctime>
#include <functional>
#include <memory>
#include <string>
#include "clients/body/json.h"
#include "clients/client.h"
#include "llm/error.h"
#include "vendors/bedrock/calls.h"
#include "vendors/bedrock/credentials.h"
#include "vendors/bedrock/event_stream.h"
#include "vendors/bedrock/sigv4.h"
namespace molo::bedrock {
using Json = clients::body::Json;
struct CallInput {
Call call = Call::kConverse;
PathArgs path;
QueryArgs query;
HeaderList headers; // extra headers; names must be in the table
const clients::body::Json* json = nullptr; // JSON body
const std::string* raw = nullptr; // raw body (InvokeModel payload)
std::string content_type = "application/json";
};
struct Reply {
int status = 0;
std::string body;
std::string content_type;
clients::HeaderMap headers;
};
// What a call would put on the wire, without sending it. The secret is not in
// it; the Authorization header is shown as "<redacted>". Used by dry runs.
struct WirePreview {
std::string method;
std::string url;
std::string auth_mode;
std::string signed_headers;
std::string canonical_request; // empty for bearer
};
class BedrockClient {
public:
using Clock = std::function<std::time_t()>;
BedrockClient(std::unique_ptr<clients::Transport> transport, Credentials credentials,
Clock clock = nullptr);
static std::shared_ptr<BedrockClient> from_env();
bool configured() const { return credentials_.configured(); }
const Credentials& credentials() const { return credentials_; }
const std::string& region() const { return credentials_.region; }
clients::ClientStatsSnapshot stats() const { return client_.stats(); }
// JSON in, JSON out. Empty 2xx body gives {}.
clients::body::Json json(const CallInput& input);
Reply raw(const CallInput& input);
// Sink gets raw body chunks of an event-stream reply (feed an
// EventStreamDecoder); the decoder's finish() must be called by the caller.
void stream(const CallInput& input, const std::function<bool(const std::string&)>& sink);
// Builds, validates and signs without sending.
WirePreview preview(const CallInput& input) const;
static std::string vendor() { return "bedrock"; }
private:
struct Built {
clients::ClientRequest request;
WirePreview preview;
};
Built build(const CallInput& input) const;
void guard(const CallSpec& spec) const;
[[noreturn]] static void throw_http(const clients::ClientResponse& response);
Credentials credentials_;
Clock clock_;
clients::Client client_;
};
// Base for the service classes. Not llm::Service: that wraps an ApiClient.
class BedrockService {
public:
explicit BedrockService(std::shared_ptr<BedrockClient> client) : client_(std::move(client)) {}
virtual ~BedrockService() = default;
bool configured() const { return client_->configured(); }
BedrockClient& client() const { return *client_; }
// Same rule as llm::Service::valid_id but for ids that may be ARNs.
static bool valid_id(const std::string& id) { return Calls::valid_path_value(id); }
void check_id(const std::string& what, const std::string& id) const;
static std::string string_at(const clients::body::Json& doc, const std::string& key,
const std::string& fallback = "");
protected:
std::shared_ptr<BedrockClient> client_;
};
} // namespace molo::bedrock
+238
View File
@@ -0,0 +1,238 @@
#include "vendors/bedrock/control.h"
#include "vendors/bedrock/caps.h"
namespace molo::bedrock {
namespace {
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("bedrock", why); }
bool one_of(const std::string& v, std::initializer_list<const char*> allowed) {
for (const char* a : allowed)
if (v == a) return true;
return false;
}
bool alnum_dash(const std::string& s, const char* extra) {
for (char c : s) {
if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')) continue;
if (std::string(extra).find(c) != std::string::npos) continue;
return false;
}
return true;
}
void check_max(int max_results) {
if (max_results < 1 || max_results > BedrockCaps::kMaxResults) bad("maxResults must be 1..1000");
}
// Walks nextToken pages of `call` collecting `field`, up to the page and item caps.
ListResult walk(BedrockClient& client, Call call, QueryArgs query, const char* field) {
ListResult out;
std::string token;
for (std::size_t page = 0; page < BedrockCaps::kMaxPages; ++page) {
CallInput in;
in.call = call;
in.query = query;
if (!token.empty()) in.query["nextToken"] = token;
Json r = client.json(in);
++out.pages;
if (const Json* arr = r.find(field); arr && arr->is_array())
for (const Json& x : arr->as_array()) {
if (out.items.as_array().size() >= BedrockCaps::kMaxItemsReturned) {
out.truncated = true;
break;
}
out.items.push(x);
}
const Json* t = r.find("nextToken");
token = t && t->is_string() ? t->as_string() : "";
if (out.truncated || token.empty()) break;
}
if (!token.empty()) {
out.truncated = true;
out.next_token = token;
}
return out;
}
} // namespace
// ---- Models ---------------------------------------------------------------
ListResult Models::list(const ModelFilter& f) {
if (!f.customization_type.empty() && !one_of(f.customization_type, {"FINE_TUNING", "CONTINUED_PRE_TRAINING", "DISTILLATION"}))
bad("customization type must be FINE_TUNING, CONTINUED_PRE_TRAINING or DISTILLATION");
if (!f.output_modality.empty() && !one_of(f.output_modality, {"TEXT", "IMAGE", "EMBEDDING"}))
bad("output modality must be TEXT, IMAGE or EMBEDDING");
if (!f.inference_type.empty() && !one_of(f.inference_type, {"ON_DEMAND", "PROVISIONED"}))
bad("inference type must be ON_DEMAND or PROVISIONED");
if (f.provider.size() > 64 || !alnum_dash(f.provider, " .-_")) bad("bad provider name");
CallInput in;
in.call = Call::kListFoundationModels;
if (!f.provider.empty()) in.query["byProvider"] = f.provider;
if (!f.customization_type.empty()) in.query["byCustomizationType"] = f.customization_type;
if (!f.output_modality.empty()) in.query["byOutputModality"] = f.output_modality;
if (!f.inference_type.empty()) in.query["byInferenceType"] = f.inference_type;
Json r = client_->json(in);
ListResult out;
out.pages = 1;
if (const Json* arr = r.find("modelSummaries"); arr && arr->is_array()) {
for (const Json& x : arr->as_array()) {
if (out.items.as_array().size() >= BedrockCaps::kMaxItemsReturned) {
out.truncated = true;
break;
}
out.items.push(x);
}
}
return out;
}
Json Models::get(const std::string& model_identifier) {
check_id("model", model_identifier);
CallInput in;
in.call = Call::kGetFoundationModel;
in.path["modelIdentifier"] = model_identifier;
Json r = client_->json(in);
const Json* d = r.find("modelDetails");
return d ? *d : r;
}
ListResult Models::list_profiles(const std::string& type, int max_results) {
check_max(max_results);
if (!type.empty() && !one_of(type, {"SYSTEM_DEFINED", "APPLICATION"})) bad("type must be SYSTEM_DEFINED or APPLICATION");
QueryArgs q;
q["maxResults"] = std::to_string(max_results);
if (!type.empty()) q["type"] = type;
return walk(*client_, Call::kListInferenceProfiles, q, "inferenceProfileSummaries");
}
Json Models::get_profile(const std::string& id) {
check_id("inference profile", id);
CallInput in;
in.call = Call::kGetInferenceProfile;
in.path["inferenceProfileIdentifier"] = id;
return client_->json(in);
}
// ---- Guardrails -----------------------------------------------------------
ListResult Guardrails::list(const std::string& guardrail_identifier, int max_results) {
check_max(max_results);
QueryArgs q;
q["maxResults"] = std::to_string(max_results);
if (!guardrail_identifier.empty()) {
check_id("guardrail", guardrail_identifier);
q["guardrailIdentifier"] = guardrail_identifier;
}
return walk(*client_, Call::kListGuardrails, q, "guardrails");
}
Json Guardrails::get(const std::string& guardrail_identifier, const std::string& version) {
check_id("guardrail", guardrail_identifier);
CallInput in;
in.call = Call::kGetGuardrail;
in.path["guardrailIdentifier"] = guardrail_identifier;
if (!version.empty()) {
bool digits = version.size() <= 8 && version.find_first_not_of("0123456789") == std::string::npos && version[0] != '0';
if (!digits && version != "DRAFT") bad("version must be a number 1..99999999 or DRAFT");
in.query["guardrailVersion"] = version;
}
return client_->json(in);
}
// ---- Batch ----------------------------------------------------------------
void BatchJobParams::validate() const {
if (job_name.empty() || job_name.size() > BedrockCaps::kMaxJobNameChars) bad("jobName must be 1..63 characters");
if (!alnum_dash(job_name, "+-.")) bad("jobName may hold letters, digits, + - and .");
if (!((job_name[0] >= 'a' && job_name[0] <= 'z') || (job_name[0] >= 'A' && job_name[0] <= 'Z') ||
(job_name[0] >= '0' && job_name[0] <= '9')))
bad("jobName must start with a letter or digit");
if (role_arn.rfind("arn:aws", 0) != 0 || role_arn.find(":iam::") == std::string::npos || role_arn.find(":role/") == std::string::npos ||
role_arn.size() > 2048)
bad("roleArn must look like arn:aws:iam::<account>:role/<name>");
if (model_id.empty() || model_id.size() > BedrockCaps::kMaxModelIdChars || !Calls::valid_path_value(model_id)) bad("bad modelId");
for (const std::string* uri : {&input_s3_uri, &output_s3_uri})
if (uri->rfind("s3://", 0) != 0 || uri->size() < 8 || uri->size() > 1024) bad("S3 URIs must start with s3:// and name a bucket");
if (!invocation_type.empty() && !one_of(invocation_type, {"InvokeModel", "Converse"})) bad("modelInvocationType must be InvokeModel or Converse");
if (!client_request_token.empty() && (client_request_token.size() > 256 || !alnum_dash(client_request_token, "-")))
bad("clientRequestToken is 1..256 letters, digits and dashes");
if (timeout_hours != 0 && (timeout_hours < BedrockCaps::kMinBatchTimeoutHours || timeout_hours > BedrockCaps::kMaxBatchTimeoutHours))
bad("timeoutDurationInHours must be 24..168");
if (tags.size() > 200) bad("at most 200 tags");
}
Json BatchJobParams::to_json() const {
validate();
Json b = Json::object();
b.set("jobName", Json(job_name));
b.set("roleArn", Json(role_arn));
b.set("modelId", Json(model_id));
Json s3in = Json::object();
s3in.set("s3Uri", Json(input_s3_uri));
Json in = Json::object();
in.set("s3InputDataConfig", s3in);
b.set("inputDataConfig", in);
Json s3out = Json::object();
s3out.set("s3Uri", Json(output_s3_uri));
Json out = Json::object();
out.set("s3OutputDataConfig", s3out);
b.set("outputDataConfig", out);
if (!invocation_type.empty()) b.set("modelInvocationType", Json(invocation_type));
if (!client_request_token.empty()) b.set("clientRequestToken", Json(client_request_token));
if (timeout_hours != 0) b.set("timeoutDurationInHours", Json(timeout_hours));
if (!tags.empty()) {
Json t = Json::array();
for (const auto& [k, v] : tags) {
Json tag = Json::object();
tag.set("key", Json(k));
tag.set("value", Json(v));
t.push(tag);
}
b.set("tags", t);
}
return b;
}
std::string BatchJobs::create(const BatchJobParams& params) {
Json body = params.to_json();
CallInput in;
in.call = Call::kCreateBatchJob;
in.json = &body;
Json r = client_->json(in);
return string_at(r, "jobArn");
}
Json BatchJobs::get(const std::string& job_identifier) {
check_id("job", job_identifier);
CallInput in;
in.call = Call::kGetBatchJob;
in.path["jobIdentifier"] = job_identifier;
return client_->json(in);
}
ListResult BatchJobs::list(const std::string& status, const std::string& name_contains, int max_results) {
check_max(max_results);
if (!status.empty() &&
!one_of(status, {"Submitted", "InProgress", "Completed", "Failed", "Stopping", "Stopped", "PartiallyCompleted", "Expired", "Validating", "Scheduled"}))
bad("unknown job status");
if (name_contains.size() > 63) bad("nameContains is at most 63 characters");
QueryArgs q;
q["maxResults"] = std::to_string(max_results);
if (!status.empty()) q["statusEquals"] = status;
if (!name_contains.empty()) q["nameContains"] = name_contains;
return walk(*client_, Call::kListBatchJobs, q, "invocationJobSummaries");
}
void BatchJobs::stop(const std::string& job_identifier) {
check_id("job", job_identifier);
CallInput in;
in.call = Call::kStopBatchJob;
in.path["jobIdentifier"] = job_identifier;
client_->json(in);
}
} // namespace molo::bedrock
+77
View File
@@ -0,0 +1,77 @@
#pragma once
// bedrock (control plane, host bedrock.<region>.amazonaws.com): foundation
// models, inference profiles, guardrails (read only), batch inference jobs.
// Create and stop are explicit methods with no defaults and cost money or change
// state; everything else is read only.
#include <string>
#include <vector>
#include "vendors/bedrock/client.h"
namespace molo::bedrock {
// A list that may have been cut: `truncated` is true when more pages existed
// than the cap allowed (never silent).
struct ListResult {
Json items = Json::array();
bool truncated = false;
std::size_t pages = 0;
std::string next_token; // set when truncated, to continue by hand
};
struct ModelFilter {
std::string provider; // byProvider, e.g. "anthropic"
std::string customization_type; // FINE_TUNING | CONTINUED_PRE_TRAINING | DISTILLATION
std::string output_modality; // TEXT | IMAGE | EMBEDDING
std::string inference_type; // ON_DEMAND | PROVISIONED
};
class Models : public BedrockService {
public:
using BedrockService::BedrockService;
// modelSummaries (the call is not paginated).
ListResult list(const ModelFilter& filter = {});
// modelDetails
Json get(const std::string& model_identifier);
// type: "" | SYSTEM_DEFINED | APPLICATION. Walks pages up to BedrockCaps::kMaxPages.
ListResult list_profiles(const std::string& type = "", int max_results = 100);
Json get_profile(const std::string& profile_identifier);
};
class Guardrails : public BedrockService {
public:
using BedrockService::BedrockService;
// guardrail_identifier filters to the versions of one guardrail ("" lists all).
ListResult list(const std::string& guardrail_identifier = "", int max_results = 100);
// version: "" returns the DRAFT / working view; "1".."99999999" or "DRAFT".
Json get(const std::string& guardrail_identifier, const std::string& version = "");
};
struct BatchJobParams {
std::string job_name; // 1..63, [a-zA-Z0-9]... as the service model pattern
std::string role_arn; // arn:aws:iam::<account>:role/...
std::string model_id;
std::string input_s3_uri; // s3://bucket/key.jsonl
std::string output_s3_uri; // s3://bucket/prefix/
std::string invocation_type; // "" | InvokeModel | Converse
std::string client_request_token; // idempotency, optional
int timeout_hours = 0; // 0: service default, else 24..168
std::map<std::string, std::string> tags; // max 200
void validate() const;
Json to_json() const;
};
class BatchJobs : public BedrockService {
public:
using BedrockService::BedrockService;
// Returns the job ARN. Costs money once the job runs.
std::string create(const BatchJobParams& params);
Json get(const std::string& job_identifier);
ListResult list(const std::string& status = "", const std::string& name_contains = "", int max_results = 100);
// Stops a running job. No default arguments on purpose.
void stop(const std::string& job_identifier);
};
} // namespace molo::bedrock
+472
View File
@@ -0,0 +1,472 @@
#include "vendors/bedrock/converse.h"
#include <algorithm>
#include "llm/error.h"
#include "security/crypto/Base64.h"
#include "vendors/bedrock/caps.h"
namespace molo::bedrock {
namespace {
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("bedrock", why); }
bool one_of(const std::string& v, std::initializer_list<const char*> allowed) {
for (const char* a : allowed)
if (v == a) return true;
return false;
}
bool tool_name_ok(const std::string& n) {
if (n.empty() || n.size() > 64) return false;
for (char c : n)
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '_' || c == '-'))
return false;
return true;
}
std::string str(const Json* v) { return v && v->is_string() ? v->as_string() : std::string(); }
} // namespace
// ---- Blocks ---------------------------------------------------------------
Json Blocks::text(const std::string& t) {
Json b = Json::object();
b.set("text", Json(t));
return b;
}
Json Blocks::image(const std::string& format, const std::string& bytes) {
if (!one_of(format, {"png", "jpeg", "gif", "webp"})) bad("image format must be png, jpeg, gif or webp");
if (bytes.empty()) bad("image bytes are empty");
Json source = Json::object();
source.set("bytes", Json(security::crypto::Base64::encode(bytes)));
Json img = Json::object();
img.set("format", Json(format));
img.set("source", source);
Json b = Json::object();
b.set("image", img);
return b;
}
Json Blocks::document(const std::string& format, const std::string& name, const std::string& bytes) {
if (!one_of(format, {"pdf", "csv", "doc", "docx", "xls", "xlsx", "html", "txt", "md"}))
bad("document format must be pdf, csv, doc, docx, xls, xlsx, html, txt or md");
if (name.empty() || name.size() > 200) bad("document name must be 1..200 characters");
if (bytes.empty()) bad("document bytes are empty");
Json source = Json::object();
source.set("bytes", Json(security::crypto::Base64::encode(bytes)));
Json doc = Json::object();
doc.set("format", Json(format));
doc.set("name", Json(name));
doc.set("source", source);
Json b = Json::object();
b.set("document", doc);
return b;
}
Json Blocks::tool_use(const std::string& id, const std::string& name, const Json& input) {
if (id.empty() || id.size() > 64) bad("toolUseId must be 1..64 characters");
if (!tool_name_ok(name)) bad("tool name must be 1..64 of [A-Za-z0-9_-]");
Json t = Json::object();
t.set("toolUseId", Json(id));
t.set("name", Json(name));
t.set("input", input);
Json b = Json::object();
b.set("toolUse", t);
return b;
}
static Json tool_result_with(const std::string& id, const Json& content_block, bool is_error) {
if (id.empty() || id.size() > 64) bad("toolUseId must be 1..64 characters");
Json content = Json::array();
content.push(content_block);
Json r = Json::object();
r.set("toolUseId", Json(id));
r.set("content", content);
r.set("status", Json(is_error ? "error" : "success"));
Json b = Json::object();
b.set("toolResult", r);
return b;
}
Json Blocks::tool_result(const std::string& id, const std::string& text_value, bool is_error) {
return tool_result_with(id, text(text_value), is_error);
}
Json Blocks::tool_result_json(const std::string& id, const Json& value, bool is_error) {
Json j = Json::object();
j.set("json", value);
return tool_result_with(id, j, is_error);
}
Json Blocks::cache_point() {
Json p = Json::object();
p.set("type", Json("default"));
Json b = Json::object();
b.set("cachePoint", p);
return b;
}
Json Blocks::message(const std::string& role, const std::vector<Json>& content) {
if (role != "user" && role != "assistant") bad("message role must be user or assistant");
if (content.empty()) bad("a message needs at least one content block");
if (content.size() > BedrockCaps::kMaxContentBlocks) bad("too many content blocks in one message");
Json arr = Json::array();
for (const Json& c : content) arr.push(c);
Json m = Json::object();
m.set("role", Json(role));
m.set("content", arr);
return m;
}
Json Blocks::user(const std::string& t) { return message("user", {text(t)}); }
Json Blocks::assistant(const std::string& t) { return message("assistant", {text(t)}); }
Json Blocks::tool(const std::string& name, const std::string& description, const Json& json_schema, bool strict) {
if (!tool_name_ok(name)) bad("tool name must be 1..64 of [A-Za-z0-9_-]");
if (!json_schema.is_object()) bad("tool input schema must be a JSON object");
Json schema = Json::object();
schema.set("json", json_schema);
Json spec = Json::object();
spec.set("name", Json(name));
if (!description.empty()) spec.set("description", Json(description));
spec.set("inputSchema", schema);
if (strict) spec.set("strict", Json(true));
Json b = Json::object();
b.set("toolSpec", spec);
return b;
}
// ---- usage ----------------------------------------------------------------
ConverseUsage ConverseUsage::from_json(const Json& u) {
ConverseUsage out;
auto num = [&](const char* key) {
const Json* v = u.is_object() ? u.find(key) : nullptr;
return v && v->is_number() ? static_cast<int>(v->as_int()) : 0;
};
out.input_tokens = num("inputTokens");
out.output_tokens = num("outputTokens");
out.total_tokens = num("totalTokens");
out.cache_read_tokens = num("cacheReadInputTokens");
out.cache_write_tokens = num("cacheWriteInputTokens");
return out;
}
// ---- request --------------------------------------------------------------
ConverseRequest& ConverseRequest::add(Json message) {
if (messages.size() >= BedrockCaps::kMaxMessages) bad("too many messages");
messages.push_back(std::move(message));
return *this;
}
ConverseRequest& ConverseRequest::add_user(const std::string& t) { return add(Blocks::user(t)); }
ConverseRequest& ConverseRequest::add_assistant(const std::string& t) { return add(Blocks::assistant(t)); }
ConverseRequest& ConverseRequest::add_system(const std::string& t) {
if (t.empty()) bad("system text is empty");
if (system.size() >= BedrockCaps::kMaxSystemBlocks) bad("too many system blocks");
system.push_back(Blocks::text(t));
return *this;
}
void ConverseRequest::validate(bool for_stream) const {
if (model_id.empty() || model_id.size() > BedrockCaps::kMaxModelIdChars) bad("modelId must be 1..2048 characters");
if (messages.empty()) bad("at least one message is required");
if (messages.size() > BedrockCaps::kMaxMessages) bad("too many messages");
for (const Json& m : messages) {
if (!m.is_object()) bad("a message must be an object");
std::string role = str(m.find("role"));
if (role != "user" && role != "assistant") bad("message role must be user or assistant");
const Json* c = m.find("content");
if (!c || !c->is_array() || c->as_array().empty()) bad("a message needs a non-empty content array");
if (c->as_array().size() > BedrockCaps::kMaxContentBlocks) bad("too many content blocks in one message");
}
if (system.size() > BedrockCaps::kMaxSystemBlocks) bad("too many system blocks");
if (max_tokens && *max_tokens < 1) bad("maxTokens must be at least 1");
if (temperature && (*temperature < 0.0 || *temperature > 1.0)) bad("temperature must be between 0 and 1");
if (top_p && (*top_p < 0.0 || *top_p > 1.0)) bad("topP must be between 0 and 1");
if (stop_sequences.size() > BedrockCaps::kMaxStopSequences) bad("too many stop sequences");
for (const std::string& s : stop_sequences)
if (s.empty()) bad("a stop sequence must not be empty");
if (tools.size() > BedrockCaps::kMaxTools) bad("too many tools");
if (!tool_choice.empty() && tools.empty()) bad("toolChoice needs tools");
if (!tool_choice.empty() && tool_choice != "auto" && tool_choice != "any" && !tool_name_ok(tool_choice))
bad("toolChoice must be auto, any or a tool name");
if (!guardrail_id.empty() && guardrail_version.empty()) bad("guardrailVersion is required with a guardrail");
if (!guardrail_trace.empty() && !one_of(guardrail_trace, {"enabled", "disabled", "enabled_full"}))
bad("guardrail trace must be enabled, disabled or enabled_full");
if (!guardrail_stream_mode.empty()) {
if (!for_stream) bad("streamProcessingMode applies to ConverseStream only");
if (!one_of(guardrail_stream_mode, {"sync", "async"})) bad("streamProcessingMode must be sync or async");
}
if (!latency.empty() && !one_of(latency, {"standard", "optimized"})) bad("latency must be standard or optimized");
if (!service_tier.empty() && !one_of(service_tier, {"priority", "default", "flex", "reserved"}))
bad("service tier must be priority, default, flex or reserved");
if (request_metadata.size() > 16) bad("requestMetadata holds at most 16 entries");
if (additional_response_paths.size() > 10) bad("additionalModelResponseFieldPaths holds at most 10 entries");
if (!output_schema.is_null() && !output_schema.is_object()) bad("output schema must be a JSON object");
if (effort.size() > 16) bad("effort is at most 16 characters");
if (!additional_fields.is_object()) bad("additionalModelRequestFields must be an object");
}
Json ConverseRequest::to_json(bool for_stream) const {
validate(for_stream);
Json body = Json::object();
Json msgs = Json::array();
for (const Json& m : messages) msgs.push(m);
body.set("messages", msgs);
if (!system.empty()) {
Json s = Json::array();
for (const Json& b : system) s.push(b);
body.set("system", s);
}
Json inf = Json::object();
if (max_tokens) inf.set("maxTokens", Json(*max_tokens));
if (temperature) inf.set("temperature", Json(*temperature));
if (top_p) inf.set("topP", Json(*top_p));
if (!stop_sequences.empty()) {
Json s = Json::array();
for (const std::string& x : stop_sequences) s.push(Json(x));
inf.set("stopSequences", s);
}
if (!inf.as_object().empty()) body.set("inferenceConfig", inf);
if (!tools.empty()) {
Json cfg = Json::object();
Json t = Json::array();
for (const Json& x : tools) t.push(x);
cfg.set("tools", t);
if (!tool_choice.empty()) {
Json choice = Json::object();
if (tool_choice == "auto") choice.set("auto", Json::object());
else if (tool_choice == "any") choice.set("any", Json::object());
else {
Json n = Json::object();
n.set("name", Json(tool_choice));
choice.set("tool", n);
}
cfg.set("toolChoice", choice);
}
body.set("toolConfig", cfg);
}
if (!guardrail_id.empty()) {
Json g = Json::object();
g.set("guardrailIdentifier", Json(guardrail_id));
g.set("guardrailVersion", Json(guardrail_version));
if (!guardrail_trace.empty()) g.set("trace", Json(guardrail_trace));
if (for_stream && !guardrail_stream_mode.empty()) g.set("streamProcessingMode", Json(guardrail_stream_mode));
body.set("guardrailConfig", g);
}
if (!additional_fields.as_object().empty()) body.set("additionalModelRequestFields", additional_fields);
if (!additional_response_paths.empty()) {
Json p = Json::array();
for (const std::string& x : additional_response_paths) p.push(Json(x));
body.set("additionalModelResponseFieldPaths", p);
}
if (!request_metadata.empty()) {
Json m = Json::object();
for (const auto& [k, v] : request_metadata) m.set(k, Json(v));
body.set("requestMetadata", m);
}
if (!latency.empty()) {
Json p = Json::object();
p.set("latency", Json(latency));
body.set("performanceConfig", p);
}
if (!service_tier.empty()) {
Json t = Json::object();
t.set("type", Json(service_tier));
body.set("serviceTier", t);
}
Json out = Json::object();
if (output_schema.is_object()) {
Json def = Json::object();
def.set("schema", Json(output_schema.dump())); // the schema travels as a JSON string
def.set("name", Json(output_schema_name));
Json structure = Json::object();
structure.set("jsonSchema", def);
Json fmt = Json::object();
fmt.set("type", Json("json_schema"));
fmt.set("structure", structure);
out.set("textFormat", fmt);
}
if (!effort.empty()) out.set("effort", Json(effort));
if (!out.as_object().empty()) body.set("outputConfig", out);
return body;
}
Json ConverseRequest::to_count_json() const {
Json full = to_json(false);
Json c = Json::object();
for (const char* k : {"messages", "system", "toolConfig", "additionalModelRequestFields"})
if (const Json* v = full.find(k)) c.set(k, *v);
Json converse = Json::object();
converse.set("converse", c);
return converse;
}
// ---- result ---------------------------------------------------------------
ConverseResult::ConverseResult(Json raw) : raw_(std::move(raw)) {
if (const Json* u = raw_.is_object() ? raw_.find("usage") : nullptr) usage_ = ConverseUsage::from_json(*u);
}
const Json* ConverseResult::content() const {
const Json* out = raw_.is_object() ? raw_.find("output") : nullptr;
const Json* msg = out && out->is_object() ? out->find("message") : nullptr;
const Json* c = msg && msg->is_object() ? msg->find("content") : nullptr;
return c && c->is_array() ? c : nullptr;
}
std::string ConverseResult::stop_reason() const { return str(raw_.find("stopReason")); }
long ConverseResult::latency_ms() const {
const Json* m = raw_.find("metrics");
const Json* l = m && m->is_object() ? m->find("latencyMs") : nullptr;
return l && l->is_number() ? l->as_int() : 0;
}
std::string ConverseResult::text() const {
std::string out;
if (const Json* c = content())
for (const Json& b : c->as_array())
if (const Json* t = b.find("text"); t && t->is_string()) out += t->as_string();
return out;
}
std::string ConverseResult::reasoning() const {
std::string out;
if (const Json* c = content())
for (const Json& b : c->as_array())
if (const Json* r = b.find("reasoningContent"))
if (const Json* rt = r->find("reasoningText")) out += str(rt->find("text"));
return out;
}
std::string ConverseResult::reasoning_signature() const {
if (const Json* c = content())
for (const Json& b : c->as_array())
if (const Json* r = b.find("reasoningContent"))
if (const Json* rt = r->find("reasoningText"))
if (std::string s = str(rt->find("signature")); !s.empty()) return s;
return "";
}
std::vector<ConverseToolUse> ConverseResult::tool_uses() const {
std::vector<ConverseToolUse> out;
if (const Json* c = content())
for (const Json& b : c->as_array())
if (const Json* t = b.find("toolUse")) {
ConverseToolUse u;
u.id = str(t->find("toolUseId"));
u.name = str(t->find("name"));
if (const Json* in = t->find("input")) u.input = *in;
out.push_back(std::move(u));
}
return out;
}
Json ConverseResult::assistant_message() const {
if (const Json* out = raw_.find("output"))
if (const Json* m = out->find("message")) return *m;
return Json::object();
}
Json ConverseResult::trace() const {
const Json* t = raw_.find("trace");
return t ? *t : Json();
}
// ---- stream ---------------------------------------------------------------
StreamDelta ConverseStreamAccumulator::on_event(const std::string& event_type, const Json& p) {
++events_;
StreamDelta d;
d.event = event_type;
const Json* idx = p.is_object() ? p.find("contentBlockIndex") : nullptr;
if (idx && idx->is_number()) d.block_index = static_cast<int>(idx->as_int());
if (event_type == "messageStart") {
if (std::string r = str(p.find("role")); !r.empty()) role_ = r;
} else if (event_type == "contentBlockStart") {
const Json* start = p.find("start");
const Json* tu = start ? start->find("toolUse") : nullptr;
if (tu && d.block_index >= 0) {
Block& b = blocks_[d.block_index];
b.kind = "tool";
b.tool_id = d.tool_id = str(tu->find("toolUseId"));
b.tool_name = d.tool_name = str(tu->find("name"));
}
} else if (event_type == "contentBlockDelta") {
const Json* delta = p.find("delta");
if (delta && d.block_index >= 0) {
Block& b = blocks_[d.block_index];
if (const Json* t = delta->find("text")) {
if (b.kind.empty()) b.kind = "text";
b.text += d.text = str(t);
} else if (const Json* tu = delta->find("toolUse")) {
b.kind = "tool";
b.tool_args += d.tool_args = str(tu->find("input"));
} else if (const Json* rc = delta->find("reasoningContent")) {
b.kind = "reasoning";
if (const Json* rt = rc->find("text")) b.text += d.reasoning = str(rt);
if (const Json* sg = rc->find("signature")) b.signature += d.reasoning_signature = str(sg);
}
}
} else if (event_type == "messageStop") {
stop_reason_ = d.stop_reason = str(p.find("stopReason"));
} else if (event_type == "metadata") {
if (const Json* u = p.find("usage")) {
usage_ = *u;
d.usage = ConverseUsage::from_json(*u);
}
if (const Json* m = p.find("metrics")) metrics_ = *m;
if (const Json* t = p.find("trace")) trace_ = *t;
}
return d;
}
ConverseResult ConverseStreamAccumulator::result() const {
Json content = Json::array();
for (const auto& [index, b] : blocks_) {
(void)index;
if (b.kind == "tool") {
Json input = Json::object();
if (!b.tool_args.empty()) {
try {
input = Json::parse(b.tool_args);
} catch (const std::exception&) {
input = Json::object();
input.set("_invalid_json", Json(b.tool_args));
}
}
content.push(Blocks::tool_use(b.tool_id.empty() ? "tool" : b.tool_id, b.tool_name.empty() ? "tool" : b.tool_name, input));
} else if (b.kind == "reasoning") {
Json rt = Json::object();
rt.set("text", Json(b.text));
if (!b.signature.empty()) rt.set("signature", Json(b.signature));
Json rc = Json::object();
rc.set("reasoningText", rt);
Json blk = Json::object();
blk.set("reasoningContent", rc);
content.push(blk);
} else {
content.push(Blocks::text(b.text));
}
}
Json msg = Json::object();
msg.set("role", Json(role_));
msg.set("content", content);
Json out = Json::object();
out.set("message", msg);
Json raw = Json::object();
raw.set("output", out);
raw.set("stopReason", Json(stop_reason_));
raw.set("usage", usage_);
raw.set("metrics", metrics_);
if (!trace_.is_null()) raw.set("trace", trace_);
return ConverseResult(raw);
}
} // namespace molo::bedrock
+149
View File
@@ -0,0 +1,149 @@
#pragma once
// Converse / ConverseStream: request builder, result, and the stream
// accumulator. Wire shapes are the bedrock-runtime service model
// (ConverseRequest, ConverseResponse, ConverseStreamOutput).
#include <functional>
#include <map>
#include <optional>
#include <string>
#include <vector>
#include "clients/body/json.h"
namespace molo::bedrock {
using Json = clients::body::Json;
// Builders for the ContentBlock / SystemContentBlock / Tool unions.
struct Blocks {
static Json text(const std::string& text);
// format: png jpeg gif webp. Bytes are raw; they travel base64 encoded.
static Json image(const std::string& format, const std::string& bytes);
// format: pdf csv doc docx xls xlsx html txt md. name 1..200 chars.
static Json document(const std::string& format, const std::string& name, const std::string& bytes);
static Json tool_use(const std::string& id, const std::string& name, const Json& input);
static Json tool_result(const std::string& id, const std::string& text, bool is_error = false);
static Json tool_result_json(const std::string& id, const Json& value, bool is_error = false);
static Json cache_point();
static Json message(const std::string& role, const std::vector<Json>& content);
static Json user(const std::string& text);
static Json assistant(const std::string& text);
// Tool union: {"toolSpec":{name, description, inputSchema:{json}}}
static Json tool(const std::string& name, const std::string& description, const Json& json_schema,
bool strict = false);
};
struct ConverseUsage {
int input_tokens = 0;
int output_tokens = 0;
int total_tokens = 0;
int cache_read_tokens = 0;
int cache_write_tokens = 0;
static ConverseUsage from_json(const Json& usage);
};
struct ConverseToolUse {
std::string id;
std::string name;
Json input = Json::object();
};
class ConverseRequest {
public:
std::string model_id; // model id, inference profile id or ARN
std::vector<Json> messages;
std::vector<Json> system;
std::optional<int> max_tokens;
std::optional<double> temperature; // 0..1
std::optional<double> top_p; // 0..1
std::vector<std::string> stop_sequences;
std::vector<Json> tools; // Blocks::tool(...)
std::string tool_choice; // "", "auto", "any", or a tool name
Json additional_fields = Json::object(); // additionalModelRequestFields (model specific)
std::vector<std::string> additional_response_paths;
std::string guardrail_id, guardrail_version, guardrail_trace; // trace: enabled disabled enabled_full
std::string guardrail_stream_mode; // stream only: sync | async
std::string latency; // "" | standard | optimized
std::string service_tier; // "" | priority | default | flex | reserved
std::map<std::string, std::string> request_metadata; // max 16
Json output_schema; // null, or a JSON Schema object: outputConfig.textFormat json_schema
std::string output_schema_name = "response";
std::string effort; // outputConfig.effort (<=16 chars)
ConverseRequest& add(Json message);
ConverseRequest& add_user(const std::string& text);
ConverseRequest& add_assistant(const std::string& text);
ConverseRequest& add_system(const std::string& text);
// Throws ApiError::invalid (400 INVALID_ARGUMENT) naming the first rule broken.
void validate(bool for_stream = false) const;
// The JSON body (modelId is in the path, not here).
Json to_json(bool for_stream = false) const;
// The CountTokens `input.converse` body.
Json to_count_json() const;
};
class ConverseResult {
public:
ConverseResult() = default;
explicit ConverseResult(Json raw);
const Json& raw() const { return raw_; }
std::string stop_reason() const;
ConverseUsage usage() const { return usage_; }
long latency_ms() const;
// All text blocks of the assistant message, concatenated.
std::string text() const;
std::string reasoning() const;
std::string reasoning_signature() const;
std::vector<ConverseToolUse> tool_uses() const;
// The assistant message as sent back in the next request (history).
Json assistant_message() const;
Json trace() const;
private:
const Json* content() const;
Json raw_ = Json::object();
ConverseUsage usage_;
};
// One decoded stream event, in neutral form.
struct StreamDelta {
std::string event; // messageStart contentBlockStart contentBlockDelta contentBlockStop messageStop metadata
int block_index = -1;
std::string text;
std::string reasoning;
std::string reasoning_signature;
std::string tool_id; // contentBlockStart for a tool
std::string tool_name;
std::string tool_args; // partial JSON text
std::string stop_reason; // messageStop
std::optional<ConverseUsage> usage; // metadata
};
// Folds events into a ConverseResult that has the same shape as the
// non-streaming reply (output.message, stopReason, usage, metrics).
class ConverseStreamAccumulator {
public:
// `event_type` is the :event-type header, `payload` its JSON body.
StreamDelta on_event(const std::string& event_type, const Json& payload);
ConverseResult result() const;
std::size_t events() const { return events_; }
private:
struct Block {
std::string kind; // text | tool | reasoning
std::string text, signature, tool_id, tool_name, tool_args;
};
std::map<int, Block> blocks_;
std::string role_ = "assistant";
std::string stop_reason_;
Json usage_ = Json::object();
Json metrics_ = Json::object();
Json trace_;
std::size_t events_ = 0;
};
} // namespace molo::bedrock
+26
View File
@@ -0,0 +1,26 @@
#include "vendors/bedrock/crc32.h"
#include <array>
namespace molo::bedrock {
namespace {
std::array<std::uint32_t, 256> make_table() {
std::array<std::uint32_t, 256> t{};
for (std::uint32_t i = 0; i < 256; ++i) {
std::uint32_t c = i;
for (int k = 0; k < 8; ++k) c = (c & 1u) ? (0xEDB88320u ^ (c >> 1)) : (c >> 1);
t[i] = c;
}
return t;
}
} // namespace
std::uint32_t Crc32::of(const std::uint8_t* data, std::size_t len, std::uint32_t seed) {
static const std::array<std::uint32_t, 256> table = make_table();
std::uint32_t c = ~seed;
for (std::size_t i = 0; i < len; ++i) c = table[(c ^ data[i]) & 0xFFu] ^ (c >> 8);
return ~c;
}
} // namespace molo::bedrock
+22
View File
@@ -0,0 +1,22 @@
#pragma once
// CRC-32 (IEEE 802.3, reflected, polynomial 0xEDB88320): the checksum the
// vnd.amazon.eventstream framing uses for the prelude and for the message.
#include <cstddef>
#include <cstdint>
#include <string>
namespace molo::bedrock {
class Crc32 {
public:
// Check value: of("123456789") == 0xCBF43926. `seed` is a previous result,
// so a checksum can be built over several pieces.
static std::uint32_t of(const std::uint8_t* data, std::size_t len, std::uint32_t seed = 0);
static std::uint32_t of(const std::string& s, std::uint32_t seed = 0) {
return of(reinterpret_cast<const std::uint8_t*>(s.data()), s.size(), seed);
}
};
} // namespace molo::bedrock
+95
View File
@@ -0,0 +1,95 @@
#include "vendors/bedrock/credentials.h"
#include "llm/config.h"
#include "llm/error.h"
#include "vendors/bedrock/caps.h"
namespace molo::bedrock {
const char* Endpoints::prefix(AwsService service) {
switch (service) {
case AwsService::kRuntime: return "bedrock-runtime";
case AwsService::kControl: return "bedrock";
case AwsService::kAgentRuntime: return "bedrock-agent-runtime";
}
return "bedrock";
}
bool Endpoints::valid_region(const std::string& region) {
if (region.size() < 2 || region.size() > BedrockCaps::kMaxRegionChars) return false;
for (char c : region)
if (!((c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '-')) return false;
return true;
}
std::string Endpoints::host(AwsService service, const std::string& region) {
if (!valid_region(region)) throw llm::ApiError::invalid("bedrock", "bad region name");
std::string suffix = region.rfind("cn-", 0) == 0 ? ".amazonaws.com.cn" : ".amazonaws.com";
return std::string(prefix(service)) + "." + region + suffix;
}
namespace {
std::string pick_region() {
std::string r = llm::Config::get(Credentials::kEnvRegion);
if (r.empty()) r = llm::Config::get(Credentials::kEnvDefaultRegion);
if (r.empty() || !Endpoints::valid_region(r)) r = Credentials::kDefaultRegion;
return r;
}
} // namespace
Credentials Credentials::from_env() {
Credentials c;
c.region = pick_region();
std::string bearer = llm::Config::get(kEnvBearer);
if (!bearer.empty()) {
c.mode = AuthMode::kBearer;
c.bearer = std::move(bearer);
// Keep SigV4 keys too: the agent runtime host does not take API keys.
}
AwsCredentials aws;
aws.access_key_id = llm::Config::get(kEnvAccessKey);
aws.secret_access_key = llm::Config::get(kEnvSecretKey);
aws.session_token = llm::Config::get(kEnvSessionToken);
if (aws.valid()) {
c.aws = std::move(aws);
if (c.mode == AuthMode::kNone) c.mode = AuthMode::kSigV4;
}
return c;
}
Credentials Credentials::sigv4(AwsCredentials aws, std::string region) {
Credentials c;
c.mode = aws.valid() ? AuthMode::kSigV4 : AuthMode::kNone;
c.aws = std::move(aws);
c.region = std::move(region);
return c;
}
Credentials Credentials::bearer_token(std::string token, std::string region) {
Credentials c;
c.mode = token.empty() ? AuthMode::kNone : AuthMode::kBearer;
c.bearer = std::move(token);
c.region = std::move(region);
return c;
}
Credentials Credentials::none(std::string region) {
Credentials c;
c.region = std::move(region);
return c;
}
const char* Credentials::mode_name() const {
switch (mode) {
case AuthMode::kNone: return "none";
case AuthMode::kSigV4: return "sigv4";
case AuthMode::kBearer: return "bearer";
}
return "none";
}
std::string Credentials::env_hint() {
return std::string(kEnvBearer) + " or " + kEnvAccessKey + " and " + kEnvSecretKey;
}
} // namespace molo::bedrock
+60
View File
@@ -0,0 +1,60 @@
#pragma once
// Where the credential comes from and which host it talks to.
//
// AWS_BEARER_TOKEN_BEDROCK Bedrock API key, sent as "Authorization: Bearer"
// AWS_ACCESS_KEY_ID SigV4 access key
// AWS_SECRET_ACCESS_KEY SigV4 secret
// AWS_SESSION_TOKEN optional, signed as x-amz-security-token
// AWS_REGION region (then AWS_DEFAULT_REGION, then us-east-1)
//
// All are read through llm::Config (environment, then .env.json.local, then
// .env.json). When both a Bedrock API key and SigV4 keys exist the API key
// wins (the AWS SDKs do the same); the choice is reported by `mode_name()`.
// The secret never appears in an error, a log line or a URL.
#include <string>
#include "vendors/bedrock/sigv4.h"
namespace molo::bedrock {
enum class AuthMode { kNone, kSigV4, kBearer };
// The three hosts the module talks to. Signing name is "bedrock" for all of
// them (service models: metadata.signingName).
enum class AwsService { kRuntime, kControl, kAgentRuntime };
struct Endpoints {
static const char* prefix(AwsService service); // bedrock-runtime, bedrock, bedrock-agent-runtime
static const char* signing_name() { return "bedrock"; }
static std::string host(AwsService service, const std::string& region);
static bool valid_region(const std::string& region); // [a-z0-9-], 2..32
};
struct Credentials {
static constexpr const char* kEnvBearer = "AWS_BEARER_TOKEN_BEDROCK";
static constexpr const char* kEnvAccessKey = "AWS_ACCESS_KEY_ID";
static constexpr const char* kEnvSecretKey = "AWS_SECRET_ACCESS_KEY";
static constexpr const char* kEnvSessionToken = "AWS_SESSION_TOKEN";
static constexpr const char* kEnvRegion = "AWS_REGION";
static constexpr const char* kEnvDefaultRegion = "AWS_DEFAULT_REGION";
static constexpr const char* kDefaultRegion = "us-east-1";
AuthMode mode = AuthMode::kNone;
AwsCredentials aws;
std::string bearer;
std::string region = kDefaultRegion;
static Credentials from_env();
static Credentials sigv4(AwsCredentials aws, std::string region = kDefaultRegion);
static Credentials bearer_token(std::string token, std::string region = kDefaultRegion);
static Credentials none(std::string region = kDefaultRegion);
bool configured() const { return mode != AuthMode::kNone; }
const char* mode_name() const;
// Human text naming every variable that would configure the module.
static std::string env_hint();
};
} // namespace molo::bedrock
+192
View File
@@ -0,0 +1,192 @@
#include "vendors/bedrock/event_stream.h"
#include "clients/body/json.h"
#include "llm/error.h"
#include "vendors/bedrock/caps.h"
#include "vendors/bedrock/crc32.h"
namespace molo::bedrock {
namespace {
std::uint32_t be32(const std::string& s, std::size_t at) {
return (static_cast<std::uint32_t>(static_cast<unsigned char>(s[at])) << 24) |
(static_cast<std::uint32_t>(static_cast<unsigned char>(s[at + 1])) << 16) |
(static_cast<std::uint32_t>(static_cast<unsigned char>(s[at + 2])) << 8) |
static_cast<std::uint32_t>(static_cast<unsigned char>(s[at + 3]));
}
std::uint64_t be_n(const std::string& s, std::size_t at, int n) {
std::uint64_t v = 0;
for (int i = 0; i < n; ++i) v = (v << 8) | static_cast<unsigned char>(s[at + static_cast<std::size_t>(i)]);
return v;
}
void put32(std::string& out, std::uint32_t v) {
for (int shift = 24; shift >= 0; shift -= 8) out.push_back(static_cast<char>((v >> shift) & 0xFF));
}
[[noreturn]] void fail(const std::string& vendor, const std::string& type, const std::string& why) {
throw llm::ApiError(vendor, 502, type, why);
}
void parse_headers(const std::string& vendor, const std::string& block, EventMessage& out) {
std::size_t at = 0;
auto need = [&](std::size_t n) {
if (at + n > block.size()) fail(vendor, "EVENTSTREAM_HEADERS", "header block is truncated");
};
while (at < block.size()) {
need(1);
std::size_t name_len = static_cast<unsigned char>(block[at++]);
need(name_len + 1);
std::string name = block.substr(at, name_len);
at += name_len;
int type = static_cast<unsigned char>(block[at++]);
EventHeader h;
h.type = type;
switch (type) {
case 0: h.number = 1; break;
case 1: h.number = 0; break;
case 2: need(1); h.number = static_cast<std::int8_t>(block[at]); at += 1; break;
case 3: need(2); h.number = static_cast<std::int16_t>(be_n(block, at, 2)); at += 2; break;
case 4: need(4); h.number = static_cast<std::int32_t>(be_n(block, at, 4)); at += 4; break;
case 5:
case 8: need(8); h.number = static_cast<std::int64_t>(be_n(block, at, 8)); at += 8; break;
case 6:
case 7: {
need(2);
std::size_t len = static_cast<std::size_t>(be_n(block, at, 2));
at += 2;
need(len);
h.text = block.substr(at, len);
at += len;
break;
}
case 9: need(16); h.text = block.substr(at, 16); at += 16; break;
default: fail(vendor, "EVENTSTREAM_HEADERS", "unknown header value type " + std::to_string(type));
}
out.headers[name] = std::move(h);
}
}
} // namespace
std::string EventMessage::header(const std::string& name) const {
auto it = headers.find(name);
return it == headers.end() ? std::string() : it->second.text;
}
EventStreamDecoder::EventStreamDecoder(std::string vendor, Sink sink)
: vendor_(std::move(vendor)), sink_(std::move(sink)) {}
bool EventStreamDecoder::feed(const std::string& chunk) {
if (stopped_) return false;
total_ += chunk.size();
if (total_ > BedrockCaps::kMaxStreamBytes) fail(vendor_, "STREAM_TOO_LARGE", "stream over the byte cap");
buffer_ += chunk;
if (!started_ && !buffer_.empty()) {
started_ = true;
// A real frame starts with a 4 byte length; a JSON error body starts with '{'.
// 0x7B in the first length byte would mean a message over 2 GiB, far past the cap.
if (buffer_[0] == '{') json_error_ = true;
}
if (json_error_) {
if (buffer_.size() > 1024u * 1024u) fail(vendor_, "BAD_RESPONSE", "error body over 1 MiB");
return true;
}
return drain();
}
bool EventStreamDecoder::drain() {
while (buffer_.size() >= 12) {
std::uint32_t total = be32(buffer_, 0);
std::uint32_t headers_len = be32(buffer_, 4);
std::uint32_t prelude_crc = be32(buffer_, 8);
if (Crc32::of(buffer_.substr(0, 8)) != prelude_crc)
fail(vendor_, "EVENTSTREAM_CRC", "prelude CRC mismatch");
if (total < 16 + headers_len) fail(vendor_, "EVENTSTREAM_LENGTH", "message shorter than its headers");
if (total > BedrockCaps::kMaxEventMessageBytes) fail(vendor_, "EVENTSTREAM_LENGTH", "message over the size cap");
if (headers_len > BedrockCaps::kMaxEventHeaderBytes) fail(vendor_, "EVENTSTREAM_LENGTH", "headers over the size cap");
if (buffer_.size() < total) return true;
std::uint32_t message_crc = be32(buffer_, total - 4);
if (Crc32::of(buffer_.substr(0, total - 4)) != message_crc)
fail(vendor_, "EVENTSTREAM_CRC", "message CRC mismatch");
EventMessage msg;
parse_headers(vendor_, buffer_.substr(12, headers_len), msg);
msg.payload = buffer_.substr(12 + headers_len, total - 16 - headers_len);
buffer_.erase(0, total);
if (++events_ > BedrockCaps::kMaxStreamEvents) fail(vendor_, "STREAM_TOO_LONG", "too many events");
if (!sink_(msg)) {
stopped_ = true;
return false;
}
}
return true;
}
void EventStreamDecoder::finish() {
if (stopped_) return;
if (json_error_) {
std::string message = "the server answered with an error document instead of a stream";
std::string type = "STREAM_HTTP_ERROR";
try {
clients::body::Json doc = clients::body::Json::parse(buffer_);
for (const char* key : {"message", "Message"})
if (const auto* m = doc.find(key); m && m->is_string()) {
message = m->as_string();
break;
}
if (const auto* t = doc.find("__type"); t && t->is_string()) type = t->as_string();
} catch (const std::exception&) {
}
// The transport hides the HTTP status of a stream; 502 marks "not a stream".
throw llm::ApiError(vendor_, 502, type, message);
}
if (!buffer_.empty()) fail(vendor_, "EVENTSTREAM_TRUNCATED", "stream ended inside a message");
}
std::string EventStreamDecoder::encode(const std::map<std::string, std::string>& string_headers,
const std::string& payload) {
std::string headers;
for (const auto& [name, value] : string_headers) {
headers.push_back(static_cast<char>(name.size()));
headers += name;
headers.push_back(7);
headers.push_back(static_cast<char>((value.size() >> 8) & 0xFF));
headers.push_back(static_cast<char>(value.size() & 0xFF));
headers += value;
}
std::string out;
put32(out, static_cast<std::uint32_t>(16 + headers.size() + payload.size()));
put32(out, static_cast<std::uint32_t>(headers.size()));
put32(out, Crc32::of(out));
out += headers;
out += payload;
put32(out, Crc32::of(out));
return out;
}
void EventStreamDecoder::raise_if_failure(const std::string& vendor, const EventMessage& message) {
if (!message.is_failure()) return;
std::string type = message.message_type() == "exception" ? message.exception_type() : message.header(":error-code");
std::string text = message.header(":error-message");
try {
clients::body::Json doc = clients::body::Json::parse(message.payload);
if (const auto* m = doc.find("message"); m && m->is_string()) text = m->as_string();
} catch (const std::exception&) {
if (text.empty()) text = message.payload.substr(0, 512);
}
if (type.empty()) type = "EVENTSTREAM_ERROR";
int status = 502;
if (type == "throttlingException") status = 429;
else if (type == "validationException") status = 400;
else if (type == "accessDeniedException") status = 403;
else if (type == "resourceNotFoundException") status = 404;
else if (type == "serviceUnavailableException") status = 503;
else if (type == "internalServerException") status = 500;
else if (type == "modelTimeoutException") status = 408;
else if (type == "modelStreamErrorException") status = 424;
throw llm::ApiError(vendor, status, type, text);
}
} // namespace molo::bedrock
+77
View File
@@ -0,0 +1,77 @@
#pragma once
// Decoder for the application/vnd.amazon.eventstream binary framing used by
// ConverseStream, InvokeModelWithResponseStream and InvokeAgent.
//
// Message layout (all integers big endian):
// total length (4) | headers length (4) | prelude CRC32 (4, over the first 8 bytes)
// headers | payload | message CRC32 (4, over everything before it)
// Header: name length (1) | name | value type (1) | value
// types: 0 true, 1 false, 2 byte, 3 int16, 4 int32, 5 int64, 6 bytes (len16),
// 7 string (len16), 8 timestamp (ms, int64), 9 uuid (16 bytes)
//
// Both CRCs are checked before a message is delivered. A plain JSON body where
// a stream was expected (an HTTP error, the transport does not expose the
// status of a stream) is collected and thrown from finish() as an ApiError.
#include <cstdint>
#include <functional>
#include <map>
#include <string>
namespace molo::bedrock {
struct EventHeader {
int type = 7;
std::string text; // type 7 (string), 6 (raw bytes), 9 (raw uuid)
std::int64_t number = 0; // types 0-5, 8
};
struct EventMessage {
std::map<std::string, EventHeader> headers;
std::string payload;
std::string header(const std::string& name) const;
std::string message_type() const { return header(":message-type"); } // event | exception | error
std::string event_type() const { return header(":event-type"); }
std::string exception_type() const { return header(":exception-type"); }
bool is_event() const { return message_type() == "event"; }
bool is_failure() const { return message_type() == "exception" || message_type() == "error"; }
};
class EventStreamDecoder {
public:
using Sink = std::function<bool(const EventMessage&)>; // false: stop
EventStreamDecoder(std::string vendor, Sink sink);
// Returns false when the sink asked to stop. Throws ApiError on a CRC
// mismatch, an impossible length, a cap hit or a corrupt header block.
bool feed(const std::string& chunk);
// Throws when bytes of a message are left over (truncated stream) or when
// the body was a JSON error document instead of a stream.
void finish();
std::size_t events() const { return events_; }
// Encodes one message (used by tests and by anything that must build frames).
static std::string encode(const std::map<std::string, std::string>& string_headers,
const std::string& payload);
// Throws ApiError for an exception / error message; no-op for events.
static void raise_if_failure(const std::string& vendor, const EventMessage& message);
private:
bool drain();
std::string vendor_;
Sink sink_;
std::string buffer_;
std::size_t total_ = 0;
std::size_t events_ = 0;
bool stopped_ = false;
bool json_error_ = false;
bool started_ = false;
};
} // namespace molo::bedrock
+242
View File
@@ -0,0 +1,242 @@
#include "vendors/bedrock/runtime.h"
#include "security/crypto/Base64.h"
#include "vendors/bedrock/caps.h"
namespace molo::bedrock {
namespace {
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("bedrock", why); }
bool one_of(const std::string& v, std::initializer_list<const char*> allowed) {
for (const char* a : allowed)
if (v == a) return true;
return false;
}
// Drives an event-stream call: every decoded message goes to `on_message`.
void run_stream(BedrockClient& client, const CallInput& input,
const std::function<bool(const EventMessage&)>& on_message) {
EventStreamDecoder decoder(BedrockClient::vendor(), [&](const EventMessage& m) {
EventStreamDecoder::raise_if_failure(BedrockClient::vendor(), m);
return on_message(m);
});
client.stream(input, [&](const std::string& chunk) { return decoder.feed(chunk); });
decoder.finish();
}
Json parse_payload(const EventMessage& m) {
if (m.payload.empty()) return Json::object();
try {
return Json::parse(m.payload);
} catch (const std::exception& e) {
throw llm::ApiError(BedrockClient::vendor(), 502, "BAD_RESPONSE", std::string("event payload: ") + e.what());
}
}
void add_invoke_headers(CallInput& in, const InvokeOptions& o, bool stream) {
in.content_type = o.content_type;
in.headers.emplace_back("Content-Type", o.content_type);
if (!o.accept.empty()) in.headers.emplace_back(stream ? "X-Amzn-Bedrock-Accept" : "Accept", o.accept);
if (!o.trace.empty()) in.headers.emplace_back("X-Amzn-Bedrock-Trace", o.trace);
if (!o.guardrail_id.empty()) {
in.headers.emplace_back("X-Amzn-Bedrock-GuardrailIdentifier", o.guardrail_id);
in.headers.emplace_back("X-Amzn-Bedrock-GuardrailVersion", o.guardrail_version);
}
if (!o.latency.empty()) in.headers.emplace_back("X-Amzn-Bedrock-PerformanceConfig-Latency", o.latency);
if (!o.service_tier.empty()) in.headers.emplace_back("X-Amzn-Bedrock-Service-Tier", o.service_tier);
}
} // namespace
void InvokeOptions::validate() const {
if (content_type.empty() || content_type.find_first_of("\r\n") != std::string::npos) bad("bad content type");
if (accept.find_first_of("\r\n") != std::string::npos) bad("bad accept");
if (!trace.empty() && !one_of(trace, {"ENABLED", "DISABLED", "ENABLED_FULL"})) bad("trace must be ENABLED, DISABLED or ENABLED_FULL");
if (!guardrail_id.empty() && guardrail_version.empty()) bad("guardrail version is required with a guardrail");
if (guardrail_id.find_first_of("\r\n") != std::string::npos || guardrail_version.find_first_of("\r\n") != std::string::npos)
bad("bad guardrail header");
if (!latency.empty() && !one_of(latency, {"standard", "optimized"})) bad("latency must be standard or optimized");
if (!service_tier.empty() && !one_of(service_tier, {"priority", "default", "flex", "reserved"}))
bad("service tier must be priority, default, flex or reserved");
}
Json InvokeResult::json() const {
try {
return Json::parse(body);
} catch (const std::exception& e) {
throw llm::ApiError(BedrockClient::vendor(), 502, "BAD_RESPONSE", e.what());
}
}
bool GuardrailResult::intervened() const { return action() == "GUARDRAIL_INTERVENED"; }
std::string GuardrailResult::action() const {
const Json* a = raw.is_object() ? raw.find("action") : nullptr;
return a && a->is_string() ? a->as_string() : "";
}
std::string GuardrailResult::output_text() const {
std::string out;
const Json* o = raw.is_object() ? raw.find("outputs") : nullptr;
if (o && o->is_array())
for (const Json& x : o->as_array())
if (const Json* t = x.find("text"); t && t->is_string()) out += t->as_string();
return out;
}
ConverseResult Runtime::converse(const ConverseRequest& request) {
Json body = request.to_json(false);
check_id("model", request.model_id);
CallInput in;
in.call = Call::kConverse;
in.path["modelId"] = request.model_id;
in.json = &body;
return ConverseResult(client_->json(in));
}
ConverseResult Runtime::converse_stream(const ConverseRequest& request,
const std::function<bool(const StreamDelta&)>& sink) {
Json body = request.to_json(true);
check_id("model", request.model_id);
CallInput in;
in.call = Call::kConverseStream;
in.path["modelId"] = request.model_id;
in.json = &body;
ConverseStreamAccumulator acc;
run_stream(*client_, in, [&](const EventMessage& m) {
if (!m.is_event()) return true;
return sink(acc.on_event(m.event_type(), parse_payload(m)));
});
return acc.result();
}
InvokeResult Runtime::invoke_model_raw(const std::string& model_id, const std::string& body, const InvokeOptions& options) {
options.validate();
check_id("model", model_id);
CallInput in;
in.call = Call::kInvokeModel;
in.path["modelId"] = model_id;
in.raw = &body;
add_invoke_headers(in, options, false);
Reply r = client_->raw(in);
InvokeResult out;
out.body = std::move(r.body);
out.content_type = r.content_type;
return out;
}
InvokeResult Runtime::invoke_model(const std::string& model_id, const Json& body, const InvokeOptions& options) {
if (!body.is_object()) bad("an InvokeModel body must be a JSON object");
return invoke_model_raw(model_id, body.dump(), options);
}
std::size_t Runtime::invoke_model_stream(const std::string& model_id, const Json& body,
const std::function<bool(const std::string&)>& sink,
const InvokeOptions& options) {
options.validate();
if (!body.is_object()) bad("an InvokeModel body must be a JSON object");
check_id("model", model_id);
std::string raw = body.dump();
CallInput in;
in.call = Call::kInvokeModelStream;
in.path["modelId"] = model_id;
in.raw = &raw;
add_invoke_headers(in, options, true);
std::size_t chunks = 0;
run_stream(*client_, in, [&](const EventMessage& m) {
if (!m.is_event() || m.event_type() != "chunk") return true;
Json p = parse_payload(m);
const Json* b = p.find("bytes");
std::string decoded;
if (!b || !b->is_string() || !security::crypto::Base64::decode(b->as_string(), decoded))
throw llm::ApiError(BedrockClient::vendor(), 502, "BAD_RESPONSE", "chunk without valid base64 bytes");
++chunks;
return sink(decoded);
});
return chunks;
}
long Runtime::count_tokens(const ConverseRequest& request) {
Json input = request.to_count_json();
check_id("model", request.model_id);
Json body = Json::object();
body.set("input", input);
CallInput in;
in.call = Call::kCountTokens;
in.path["modelId"] = request.model_id;
in.json = &body;
Json r = client_->json(in);
const Json* n = r.find("inputTokens");
if (!n || !n->is_number()) throw llm::ApiError(BedrockClient::vendor(), 502, "BAD_RESPONSE", "no inputTokens in reply");
return n->as_int();
}
long Runtime::count_tokens_invoke(const std::string& model_id, const Json& model_body) {
check_id("model", model_id);
if (!model_body.is_object()) bad("an InvokeModel body must be a JSON object");
Json invoke = Json::object();
// The blob travels base64 encoded in this JSON field.
invoke.set("body", Json(security::crypto::Base64::encode(model_body.dump())));
Json input = Json::object();
input.set("invokeModel", invoke);
Json body = Json::object();
body.set("input", input);
CallInput in;
in.call = Call::kCountTokens;
in.path["modelId"] = model_id;
in.json = &body;
Json r = client_->json(in);
const Json* n = r.find("inputTokens");
if (!n || !n->is_number()) throw llm::ApiError(BedrockClient::vendor(), 502, "BAD_RESPONSE", "no inputTokens in reply");
return n->as_int();
}
Json Runtime::guardrail_text(const std::string& text, const std::vector<std::string>& qualifiers) {
Json t = Json::object();
t.set("text", Json(text));
if (!qualifiers.empty()) {
Json q = Json::array();
for (const std::string& x : qualifiers) {
if (!one_of(x, {"grounding_source", "query", "guard_content"})) bad("qualifier must be grounding_source, query or guard_content");
q.push(Json(x));
}
t.set("qualifiers", q);
}
Json b = Json::object();
b.set("text", t);
return b;
}
GuardrailResult Runtime::apply_guardrail(const std::string& guardrail_id, const std::string& version,
const std::string& source, const std::vector<Json>& blocks,
const std::string& scope) {
check_id("guardrail", guardrail_id);
check_id("guardrail version", version);
if (!one_of(source, {"INPUT", "OUTPUT"})) bad("source must be INPUT or OUTPUT");
if (blocks.empty()) bad("at least one content block is required");
if (blocks.size() > BedrockCaps::kMaxGuardrailBlocks) bad("too many content blocks");
if (!scope.empty() && !one_of(scope, {"INTERVENTIONS", "FULL"})) bad("outputScope must be INTERVENTIONS or FULL");
Json arr = Json::array();
for (const Json& b : blocks) {
if (const Json* t = b.find("text"); t && t->is_object())
if (const Json* s = t->find("text"); s && s->is_string() && s->as_string().size() > BedrockCaps::kMaxGuardrailTextChars)
bad("guardrail text too long");
arr.push(b);
}
Json body = Json::object();
body.set("source", Json(source));
body.set("content", arr);
if (!scope.empty()) body.set("outputScope", Json(scope));
CallInput in;
in.call = Call::kApplyGuardrail;
in.path["guardrailIdentifier"] = guardrail_id;
in.path["guardrailVersion"] = version;
in.json = &body;
GuardrailResult r;
r.raw = client_->json(in);
return r;
}
} // namespace molo::bedrock
+67
View File
@@ -0,0 +1,67 @@
#pragma once
// bedrock-runtime: Converse, ConverseStream, InvokeModel,
// InvokeModelWithResponseStream, CountTokens, ApplyGuardrail.
#include <functional>
#include <string>
#include <vector>
#include "vendors/bedrock/client.h"
#include "vendors/bedrock/converse.h"
namespace molo::bedrock {
struct InvokeOptions {
std::string content_type = "application/json";
std::string accept = "application/json"; // "" leaves it out
std::string trace; // ENABLED | DISABLED | ENABLED_FULL
std::string guardrail_id, guardrail_version;
std::string latency; // standard | optimized
std::string service_tier; // priority | default | flex | reserved
void validate() const;
};
struct InvokeResult {
std::string body; // the model's own format
std::string content_type;
Json json() const; // body parsed (throws ApiError BAD_RESPONSE when it is not JSON)
};
struct GuardrailResult {
Json raw;
bool intervened() const;
std::string action() const;
std::string output_text() const; // outputs[].text joined
};
class Runtime : public BedrockService {
public:
using BedrockService::BedrockService;
ConverseResult converse(const ConverseRequest& request);
// sink returns false to stop early. Returns the accumulated result (complete
// only when the stream ran to its end).
ConverseResult converse_stream(const ConverseRequest& request,
const std::function<bool(const StreamDelta&)>& sink);
InvokeResult invoke_model(const std::string& model_id, const Json& body, const InvokeOptions& options = {});
InvokeResult invoke_model_raw(const std::string& model_id, const std::string& body, const InvokeOptions& options);
// Each chunk is decoded from {"bytes": base64} to the model's own streamed JSON text.
// Returns the number of chunks delivered.
std::size_t invoke_model_stream(const std::string& model_id, const Json& body,
const std::function<bool(const std::string&)>& sink,
const InvokeOptions& options = {});
// Input tokens for a Converse-shaped request (no inference, no cost).
long count_tokens(const ConverseRequest& request);
long count_tokens_invoke(const std::string& model_id, const Json& body);
// source: INPUT | OUTPUT. blocks: GuardrailContentBlock unions. scope: "" | INTERVENTIONS | FULL.
GuardrailResult apply_guardrail(const std::string& guardrail_id, const std::string& version,
const std::string& source, const std::vector<Json>& blocks,
const std::string& scope = "");
static Json guardrail_text(const std::string& text, const std::vector<std::string>& qualifiers = {});
};
} // namespace molo::bedrock
+237
View File
@@ -0,0 +1,237 @@
#include "vendors/bedrock/sigv4.h"
#include <algorithm>
#include <cctype>
#include <map>
#include "llm/error.h"
#include "security/crypto/Hmac.h"
#include "security/crypto/Sha256.h"
#include "vendors/bedrock/caps.h"
namespace molo::bedrock {
namespace crypto = molo::security::crypto;
namespace {
std::string lower(std::string s) {
for (char& c : s) c = static_cast<char>(std::tolower(static_cast<unsigned char>(c)));
return s;
}
bool unreserved(unsigned char c) {
return (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') ||
c == '-' || c == '_' || c == '.' || c == '~';
}
int hex_val(char c) {
if (c >= '0' && c <= '9') return c - '0';
if (c >= 'a' && c <= 'f') return c - 'a' + 10;
if (c >= 'A' && c <= 'F') return c - 'A' + 10;
return -1;
}
// Percent-decoding without throwing; a stray '%' stays literal, '+' stays '+'.
std::string decode(const std::string& text) {
std::string out;
out.reserve(text.size());
for (std::size_t i = 0; i < text.size(); ++i) {
if (text[i] == '%' && i + 2 < text.size() + 0 && hex_val(text[i + 1]) >= 0 &&
hex_val(text[i + 2]) >= 0) {
out.push_back(static_cast<char>((hex_val(text[i + 1]) << 4) | hex_val(text[i + 2])));
i += 2;
} else {
out.push_back(text[i]);
}
}
return out;
}
std::string raw_string(const std::array<std::uint8_t, 32>& d) {
return std::string(reinterpret_cast<const char*>(d.data()), d.size());
}
std::string sha_hex(const std::string& s) { return crypto::Sha256::hex(crypto::Sha256::hash(s)); }
std::string trim_collapse(const std::string& v) {
std::string out;
bool space = false;
for (char c : v) {
if (c == ' ' || c == '\t' || c == '\n' || c == '\r') {
space = !out.empty();
continue;
}
if (space) out.push_back(' ');
space = false;
out.push_back(c);
}
return out;
}
// Removes "." and ".." segments and repeated slashes; keeps a trailing slash.
std::string normalize_path(const std::string& path) {
if (path.empty()) return "/";
std::vector<std::string> parts;
std::string cur;
auto flush = [&] {
if (cur == "..") {
if (!parts.empty()) parts.pop_back();
} else if (!cur.empty() && cur != ".") {
parts.push_back(cur);
}
cur.clear();
};
for (char c : path) {
if (c == '/') flush();
else cur.push_back(c);
}
bool trailing = cur.empty() || cur == "." || cur == "..";
flush();
std::string out = "/";
for (std::size_t i = 0; i < parts.size(); ++i) {
if (i) out.push_back('/');
out += parts[i];
}
if (trailing && out.size() > 1 && path.back() == '/') out.push_back('/');
return out;
}
} // namespace
SigV4Signer::SigV4Signer(AwsCredentials credentials, std::string region, std::string service)
: credentials_(std::move(credentials)), region_(std::move(region)), service_(std::move(service)) {
if (!credentials_.valid())
throw llm::ApiError::invalid("bedrock", "SigV4 needs an access key id and a secret access key");
}
std::string SigV4Signer::uri_encode(const std::string& text, bool keep_slash) {
static const char* kHex = "0123456789ABCDEF";
std::string out;
out.reserve(text.size());
for (char ch : text) {
unsigned char c = static_cast<unsigned char>(ch);
if (unreserved(c) || (keep_slash && c == '/')) {
out.push_back(ch);
} else {
out.push_back('%');
out.push_back(kHex[c >> 4]);
out.push_back(kHex[c & 0xF]);
}
}
return out;
}
std::string SigV4Signer::canonical_uri(const std::string& path) {
return uri_encode(normalize_path(path), true);
}
std::string SigV4Signer::canonical_query(const std::string& query) {
std::vector<std::pair<std::string, std::string>> params;
std::size_t pos = 0;
while (pos <= query.size() && !query.empty()) {
std::size_t amp = query.find('&', pos);
std::string pair = query.substr(pos, amp == std::string::npos ? std::string::npos : amp - pos);
pos = amp == std::string::npos ? query.size() + 1 : amp + 1;
if (pair.empty()) continue;
std::size_t eq = pair.find('=');
std::string k = eq == std::string::npos ? pair : pair.substr(0, eq);
std::string v = eq == std::string::npos ? "" : pair.substr(eq + 1);
params.emplace_back(uri_encode(decode(k), false), uri_encode(decode(v), false));
if (params.size() > BedrockCaps::kMaxQueryParams)
throw llm::ApiError::invalid("bedrock", "too many query parameters to sign");
}
std::sort(params.begin(), params.end());
std::string out;
for (const auto& [k, v] : params) {
if (!out.empty()) out.push_back('&');
out += k + "=" + v;
}
return out;
}
std::string SigV4Signer::format_amz_date(std::time_t when) {
std::tm tm{};
gmtime_r(&when, &tm);
char buf[32];
std::strftime(buf, sizeof(buf), "%Y%m%dT%H%M%SZ", &tm);
return buf;
}
std::string SigV4Signer::signing_key(const std::string& secret, const std::string& date8,
const std::string& region, const std::string& service) {
std::string k = raw_string(crypto::Hmac::sha256("AWS4" + secret, date8));
k = raw_string(crypto::Hmac::sha256(k, region));
k = raw_string(crypto::Hmac::sha256(k, service));
return raw_string(crypto::Hmac::sha256(k, "aws4_request"));
}
SigV4Result SigV4Signer::sign(const SigV4Request& request, const std::string& amz_date) const {
if (request.headers.size() > BedrockCaps::kMaxSignedHeaders)
throw llm::ApiError::invalid("bedrock", "too many headers to sign");
if (request.path.size() > BedrockCaps::kMaxPathChars)
throw llm::ApiError::invalid("bedrock", "path too long to sign");
HeaderList headers = request.headers;
HeaderList added;
auto has = [&](const std::string& name) {
return std::any_of(headers.begin(), headers.end(),
[&](const auto& h) { return lower(h.first) == name; });
};
auto value_of = [&](const std::string& name) {
for (const auto& h : headers)
if (lower(h.first) == name) return trim_collapse(h.second);
return std::string();
};
if (!has("host")) throw llm::ApiError::invalid("bedrock", "SigV4 needs a host header");
std::string date = amz_date;
if (has("x-amz-date")) {
date = value_of("x-amz-date");
} else {
headers.emplace_back("x-amz-date", date);
added.emplace_back("x-amz-date", date);
}
if (date.size() != 16 || date[8] != 'T' || date.back() != 'Z')
throw llm::ApiError::invalid("bedrock", "x-amz-date must look like 20150830T123600Z");
if (!credentials_.session_token.empty() && !has("x-amz-security-token")) {
headers.emplace_back("x-amz-security-token", credentials_.session_token);
added.emplace_back("x-amz-security-token", credentials_.session_token);
}
// Canonical headers: repeated names are joined in order of appearance.
std::map<std::string, std::string> canon;
for (const auto& [name, value] : headers) {
std::string n = lower(name);
std::string v = trim_collapse(value);
auto it = canon.find(n);
if (it == canon.end()) canon.emplace(n, v);
else it->second += "," + v;
}
std::string canonical_headers, signed_headers;
for (const auto& [n, v] : canon) {
canonical_headers += n + ":" + v + "\n";
if (!signed_headers.empty()) signed_headers.push_back(';');
signed_headers += n;
}
SigV4Result r;
r.amz_date = date;
std::string date8 = date.substr(0, 8);
r.credential_scope = date8 + "/" + region_ + "/" + service_ + "/aws4_request";
r.signed_headers = signed_headers;
r.canonical_request = request.method + "\n" + canonical_uri(request.path) + "\n" +
canonical_query(request.query) + "\n" + canonical_headers + "\n" +
signed_headers + "\n" + sha_hex(request.body);
r.string_to_sign = "AWS4-HMAC-SHA256\n" + date + "\n" + r.credential_scope + "\n" +
sha_hex(r.canonical_request);
std::string key = signing_key(credentials_.secret_access_key, date8, region_, service_);
r.signature = crypto::Hmac::hex(crypto::Hmac::sha256(key, r.string_to_sign));
r.authorization = "AWS4-HMAC-SHA256 Credential=" + credentials_.access_key_id + "/" +
r.credential_scope + ", SignedHeaders=" + signed_headers +
", Signature=" + r.signature;
added.emplace_back("Authorization", r.authorization);
r.added_headers = std::move(added);
return r;
}
} // namespace molo::bedrock
+78
View File
@@ -0,0 +1,78 @@
#pragma once
// AWS Signature Version 4 (header signing), our own implementation over
// security::crypto Sha256 and Hmac. Verified against AWS's published SigV4
// test suite (see SPEC.md ## Verified against spec).
//
// Rules implemented (docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv4-create-signed-request.html):
// - canonical URI: the path exactly as sent, dot segments and repeated
// slashes normalized, every byte outside the unreserved set (and '/')
// percent-encoded. A path that is already encoded is therefore encoded a
// second time, which is what every non-S3 service expects.
// - canonical query: names and values decoded, re-encoded, sorted by encoded
// name then value. A literal '+' is data, not a space.
// - canonical headers: lower-case names, values trimmed with inner runs of
// spaces collapsed, repeated names joined with ",", sorted by name.
// - every header in the request is signed; `host` is required; x-amz-date
// and x-amz-security-token are added when missing.
#include <ctime>
#include <string>
#include <utility>
#include <vector>
namespace molo::bedrock {
struct AwsCredentials {
std::string access_key_id;
std::string secret_access_key;
std::string session_token; // optional
bool valid() const { return !access_key_id.empty() && !secret_access_key.empty(); }
};
using HeaderList = std::vector<std::pair<std::string, std::string>>;
struct SigV4Request {
std::string method; // "GET", "POST", ...
std::string path; // as sent on the wire (already percent-encoded for real calls)
std::string query; // raw query string without '?'
HeaderList headers; // everything that will be sent and must be signed
std::string body;
};
struct SigV4Result {
std::string amz_date; // 20150830T123600Z
std::string credential_scope; // 20150830/us-east-1/service/aws4_request
std::string signed_headers; // host;x-amz-date
std::string canonical_request;
std::string string_to_sign;
std::string signature; // hex
std::string authorization; // full Authorization header value
HeaderList added_headers; // x-amz-date, x-amz-security-token, authorization (those not already present)
};
class SigV4Signer {
public:
SigV4Signer(AwsCredentials credentials, std::string region, std::string service);
// `amz_date` is "YYYYMMDDTHHMMSSZ" and wins over an x-amz-date header only
// when the request has none; when the header is present its value is used.
SigV4Result sign(const SigV4Request& request, const std::string& amz_date) const;
static std::string format_amz_date(std::time_t when);
static std::string signing_key(const std::string& secret, const std::string& date8,
const std::string& region, const std::string& service);
static std::string canonical_uri(const std::string& path);
static std::string canonical_query(const std::string& query);
static std::string uri_encode(const std::string& text, bool keep_slash);
const std::string& region() const { return region_; }
const std::string& service() const { return service_; }
private:
AwsCredentials credentials_;
std::string region_;
std::string service_;
};
} // namespace molo::bedrock
+341
View File
@@ -0,0 +1,341 @@
#!/usr/bin/env python3
"""Check src/vendors/bedrock/calls.cpp against the AWS service models.
AWS publishes the service definitions that generate its SDKs as JSON; the
botocore copy (boto/botocore, botocore/data/<service>/<version>/service-2.json)
is the pinned source here. Services read: bedrock (control plane),
bedrock-runtime, bedrock-agent-runtime. Exit 1 when:
row a declared call's operation, HTTP method or request URI differs from
the model, or a path / query / header / body member it names is not
a member of the operation input (by location and location name);
required a required input member of a declared call is not covered by the row;
response the row's response schema is not the operation output (or, for an
event stream row, not the event-stream union of its output);
auth bearer_ok disagrees with `httpBearerAuth` in the model's auth list;
stream event_stream disagrees with the model (an eventstream output member);
schema a member we read (dotted paths descend) is absent from its shape;
coverage an operation of the three services is neither in the table nor in
EXCLUDED below, or an EXCLUDED entry names an operation that does
not exist or is also in the table.
Python 3 stdlib only, no credentials, network required (or --dir DIR holding
bedrock.json, bedrock-runtime.json, bedrock-agent-runtime.json).
python3 -I src/vendors/bedrock/tools/check_spec.py [--dir DIR] [alternative calls.cpp]
"""
import json
import os
import re
import sys
import urllib.request
HERE = os.path.dirname(os.path.abspath(__file__))
CALLS = os.path.join(HERE, "..", "calls.cpp")
BASE = "https://raw.githubusercontent.com/boto/botocore/develop/botocore/data/"
SERVICES = {
"bedrock": "bedrock/2023-04-20/service-2.json",
"bedrock-runtime": "bedrock-runtime/2023-09-30/service-2.json",
"bedrock-agent-runtime": "bedrock-agent-runtime/2023-07-26/service-2.json",
}
SERVICE_OF = {"R": "bedrock-runtime", "C": "bedrock", "A": "bedrock-agent-runtime"}
# Operations the module does not wrap, each with the reason.
EXCLUDED = {
"bedrock-runtime": {
"GetAsyncInvoke": "async invoke (S3 output, Nova Reel video): not wrapped, see SPEC Gaps",
"InvokeGuardrailChecks": "newer guardrail checks API, shape not read: not wrapped, see SPEC Gaps",
"InvokeModelWithBidirectionalStream": "HTTP/2 bidirectional event stream (Nova Sonic): needs an h2 transport, not wrapped",
"ListAsyncInvokes": "async invoke (S3 output, Nova Reel video): not wrapped, see SPEC Gaps",
"StartAsyncInvoke": "async invoke (S3 output, Nova Reel video): not wrapped, see SPEC Gaps",
},
"bedrock-agent-runtime": {
"AgenticRetrieveStream": "other agent-runtime inference calls: not wrapped, see SPEC Gaps",
"CheckIngestedDocumentAcl": "knowledge base document ACL and content: not wrapped",
"CreateInvocation": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"CreateSession": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"DeleteAgentMemory": "agent memory: not wrapped, see SPEC Gaps",
"DeleteSession": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"EndSession": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"GenerateQuery": "other agent-runtime inference calls: not wrapped, see SPEC Gaps",
"GetAgentMemory": "agent memory: not wrapped, see SPEC Gaps",
"GetDocumentContent": "knowledge base document ACL and content: not wrapped",
"GetExecutionFlowSnapshot": "flows runtime (invoke, executions, snapshots): not wrapped, see SPEC Gaps",
"GetFlowExecution": "flows runtime (invoke, executions, snapshots): not wrapped, see SPEC Gaps",
"GetIngestedDocumentAcl": "knowledge base document ACL and content: not wrapped",
"GetInvocationStep": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"GetSession": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"InvokeFlow": "flows runtime (invoke, executions, snapshots): not wrapped, see SPEC Gaps",
"InvokeInlineAgent": "other agent-runtime inference calls: not wrapped, see SPEC Gaps",
"ListFlowExecutionEvents": "flows runtime (invoke, executions, snapshots): not wrapped, see SPEC Gaps",
"ListFlowExecutions": "flows runtime (invoke, executions, snapshots): not wrapped, see SPEC Gaps",
"ListInvocationSteps": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"ListInvocations": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"ListSessions": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"ListTagsForResource": "resource tags: not wrapped",
"OptimizePrompt": "other agent-runtime inference calls: not wrapped, see SPEC Gaps",
"PutInvocationStep": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
"Rerank": "other agent-runtime inference calls: not wrapped, see SPEC Gaps",
"RetrieveAndGenerate": "other agent-runtime inference calls: not wrapped, see SPEC Gaps",
"RetrieveAndGenerateStream": "other agent-runtime inference calls: not wrapped, see SPEC Gaps",
"StartFlowExecution": "flows runtime (invoke, executions, snapshots): not wrapped, see SPEC Gaps",
"StopFlowExecution": "flows runtime (invoke, executions, snapshots): not wrapped, see SPEC Gaps",
"TagResource": "resource tags: not wrapped",
"UntagResource": "resource tags: not wrapped",
"UpdateSession": "sessions API (create, list, steps, end): not wrapped, see SPEC Gaps",
},
"bedrock": {
"BatchDeleteAdvancedPromptOptimizationJob": "prompt optimization jobs: out of scope",
"BatchDeleteEvaluationJob": "model evaluation jobs: out of scope",
"CancelAutomatedReasoningPolicyBuildWorkflow": "automated reasoning policies: management plane, out of scope",
"CreateAdvancedPromptOptimizationJob": "prompt optimization jobs: out of scope",
"CreateAutomatedReasoningPolicy": "automated reasoning policies: management plane, out of scope",
"CreateAutomatedReasoningPolicyTestCase": "automated reasoning policies: management plane, out of scope",
"CreateAutomatedReasoningPolicyVersion": "automated reasoning policies: management plane, out of scope",
"CreateCustomModel": "model customization, import and copy: training workflows, out of scope",
"CreateCustomModelDeployment": "model customization, import and copy: training workflows, out of scope",
"CreateEvaluationJob": "model evaluation jobs: out of scope",
"CreateFoundationModelAgreement": "model access and agreements: console workflow, out of scope",
"CreateGuardrail": "guardrail writes and enforced configuration: out of scope (list and get are wrapped)",
"CreateGuardrailVersion": "guardrail writes and enforced configuration: out of scope (list and get are wrapped)",
"CreateInferenceProfile": "inference profile writes: out of scope (list and get are wrapped)",
"CreateMarketplaceModelEndpoint": "marketplace model endpoints: out of scope",
"CreateModelCopyJob": "model customization, import and copy: training workflows, out of scope",
"CreateModelCustomizationJob": "model customization, import and copy: training workflows, out of scope",
"CreateModelImportJob": "model customization, import and copy: training workflows, out of scope",
"CreatePromptRouter": "prompt routers: out of scope",
"CreateProvisionedModelThroughput": "provisioned throughput purchases: out of scope",
"DeleteAutomatedReasoningPolicy": "automated reasoning policies: management plane, out of scope",
"DeleteAutomatedReasoningPolicyBuildWorkflow": "automated reasoning policies: management plane, out of scope",
"DeleteAutomatedReasoningPolicyTestCase": "automated reasoning policies: management plane, out of scope",
"DeleteCustomModel": "model customization, import and copy: training workflows, out of scope",
"DeleteCustomModelDeployment": "model customization, import and copy: training workflows, out of scope",
"DeleteEnforcedGuardrailConfiguration": "guardrail writes and enforced configuration: out of scope (list and get are wrapped)",
"DeleteFoundationModelAgreement": "model access and agreements: console workflow, out of scope",
"DeleteGuardrail": "guardrail writes and enforced configuration: out of scope (list and get are wrapped)",
"DeleteImportedModel": "model customization, import and copy: training workflows, out of scope",
"DeleteInferenceProfile": "inference profile writes: out of scope (list and get are wrapped)",
"DeleteMarketplaceModelEndpoint": "marketplace model endpoints: out of scope",
"DeleteModelInvocationLoggingConfiguration": "account configuration, policies and tags: out of scope",
"DeletePromptRouter": "prompt routers: out of scope",
"DeleteProvisionedModelThroughput": "provisioned throughput purchases: out of scope",
"DeleteResourcePolicy": "account configuration, policies and tags: out of scope",
"DeregisterMarketplaceModelEndpoint": "marketplace model endpoints: out of scope",
"ExportAutomatedReasoningPolicyVersion": "automated reasoning policies: management plane, out of scope",
"GetAccountDataRetention": "account configuration, policies and tags: out of scope",
"GetAdvancedPromptOptimizationJob": "prompt optimization jobs: out of scope",
"GetAutomatedReasoningPolicy": "automated reasoning policies: management plane, out of scope",
"GetAutomatedReasoningPolicyAnnotations": "automated reasoning policies: management plane, out of scope",
"GetAutomatedReasoningPolicyBuildWorkflow": "automated reasoning policies: management plane, out of scope",
"GetAutomatedReasoningPolicyBuildWorkflowResultAssets": "automated reasoning policies: management plane, out of scope",
"GetAutomatedReasoningPolicyNextScenario": "automated reasoning policies: management plane, out of scope",
"GetAutomatedReasoningPolicyTestCase": "automated reasoning policies: management plane, out of scope",
"GetAutomatedReasoningPolicyTestResult": "automated reasoning policies: management plane, out of scope",
"GetCustomModel": "model customization, import and copy: training workflows, out of scope",
"GetCustomModelDeployment": "model customization, import and copy: training workflows, out of scope",
"GetEvaluationJob": "model evaluation jobs: out of scope",
"GetFoundationModelAvailability": "model access and agreements: console workflow, out of scope",
"GetImportedModel": "model customization, import and copy: training workflows, out of scope",
"GetMarketplaceModelEndpoint": "marketplace model endpoints: out of scope",
"GetModelCopyJob": "model customization, import and copy: training workflows, out of scope",
"GetModelCustomizationJob": "model customization, import and copy: training workflows, out of scope",
"GetModelImportJob": "model customization, import and copy: training workflows, out of scope",
"GetModelInvocationLoggingConfiguration": "account configuration, policies and tags: out of scope",
"GetPromptRouter": "prompt routers: out of scope",
"GetProvisionedModelThroughput": "provisioned throughput purchases: out of scope",
"GetResourcePolicy": "account configuration, policies and tags: out of scope",
"GetUseCaseForModelAccess": "model access and agreements: console workflow, out of scope",
"ListAdvancedPromptOptimizationJobs": "prompt optimization jobs: out of scope",
"ListAutomatedReasoningPolicies": "automated reasoning policies: management plane, out of scope",
"ListAutomatedReasoningPolicyBuildWorkflows": "automated reasoning policies: management plane, out of scope",
"ListAutomatedReasoningPolicyTestCases": "automated reasoning policies: management plane, out of scope",
"ListAutomatedReasoningPolicyTestResults": "automated reasoning policies: management plane, out of scope",
"ListCustomModelDeployments": "model customization, import and copy: training workflows, out of scope",
"ListCustomModels": "model customization, import and copy: training workflows, out of scope",
"ListEnforcedGuardrailsConfiguration": "guardrail writes and enforced configuration: out of scope (list and get are wrapped)",
"ListEvaluationJobs": "model evaluation jobs: out of scope",
"ListFoundationModelAgreementOffers": "model access and agreements: console workflow, out of scope",
"ListImportedModels": "model customization, import and copy: training workflows, out of scope",
"ListMarketplaceModelEndpoints": "marketplace model endpoints: out of scope",
"ListModelCopyJobs": "model customization, import and copy: training workflows, out of scope",
"ListModelCustomizationJobs": "model customization, import and copy: training workflows, out of scope",
"ListModelImportJobs": "model customization, import and copy: training workflows, out of scope",
"ListPromptRouters": "prompt routers: out of scope",
"ListProvisionedModelThroughputs": "provisioned throughput purchases: out of scope",
"ListTagsForResource": "account configuration, policies and tags: out of scope",
"PutAccountDataRetention": "account configuration, policies and tags: out of scope",
"PutEnforcedGuardrailConfiguration": "guardrail writes and enforced configuration: out of scope (list and get are wrapped)",
"PutModelInvocationLoggingConfiguration": "account configuration, policies and tags: out of scope",
"PutResourcePolicy": "account configuration, policies and tags: out of scope",
"PutUseCaseForModelAccess": "model access and agreements: console workflow, out of scope",
"RegisterMarketplaceModelEndpoint": "marketplace model endpoints: out of scope",
"StartAutomatedReasoningPolicyBuildWorkflow": "automated reasoning policies: management plane, out of scope",
"StartAutomatedReasoningPolicyTestWorkflow": "automated reasoning policies: management plane, out of scope",
"StopAdvancedPromptOptimizationJob": "prompt optimization jobs: out of scope",
"StopEvaluationJob": "model evaluation jobs: out of scope",
"StopModelCustomizationJob": "model customization, import and copy: training workflows, out of scope",
"TagResource": "account configuration, policies and tags: out of scope",
"UntagResource": "account configuration, policies and tags: out of scope",
"UpdateAutomatedReasoningPolicy": "automated reasoning policies: management plane, out of scope",
"UpdateAutomatedReasoningPolicyAnnotations": "automated reasoning policies: management plane, out of scope",
"UpdateAutomatedReasoningPolicyTestCase": "automated reasoning policies: management plane, out of scope",
"UpdateCustomModelDeployment": "model customization, import and copy: training workflows, out of scope",
"UpdateGuardrail": "guardrail writes and enforced configuration: out of scope (list and get are wrapped)",
"UpdateMarketplaceModelEndpoint": "marketplace model endpoints: out of scope",
"UpdateProvisionedModelThroughput": "provisioned throughput purchases: out of scope",
},
}
ROW = re.compile(
r'\{Call::(\w+), (\w), ' + r', '.join(['"([^"]*)"'] * 8) + r', (true|false), (true|false), (true|false)\}'
)
SCHEMA = re.compile(r'\{"(\w+)", "([\w-]+)", "([^"]*)"\}')
def load(svc, directory):
if directory:
with open(os.path.join(directory, svc + ".json"), encoding="utf-8") as f:
return json.load(f)
req = urllib.request.Request(BASE + SERVICES[svc], headers={"User-Agent": "molo-bedrock-spec-check/1"})
with urllib.request.urlopen(req, timeout=120) as r:
return json.loads(r.read().decode("utf-8"))
def csv(text):
return [x for x in text.split(",") if x]
def members(model, shape_name):
return model["shapes"][shape_name].get("members", {})
def main(argv):
directory = None
args = argv[1:]
if args[:1] == ["--dir"]:
directory, args = args[1], args[2:]
calls_path = args[0] if args else CALLS
text = open(calls_path, encoding="utf-8").read()
rows = ROW.findall(text)
schemas = SCHEMA.findall(text)
models = {svc: load(svc, directory) for svc in SERVICES}
problems = []
def bad(kind, msg):
problems.append(f"{kind}: {msg}")
used = {svc: set() for svc in SERVICES}
for (name, svc_letter, op, method, uri, path_p, query_p, header_p, body_p, response, bearer, stream, _state) in rows:
svc = SERVICE_OF[svc_letter]
model = models[svc]
used[svc].add(op)
operation = model["operations"].get(op)
if not operation:
bad("row", f"{name}: operation {op} not in {svc}")
continue
http = operation["http"]
if http["method"] != method:
bad("row", f"{op}: method {method} != model {http['method']}")
if http["requestUri"] != uri:
bad("row", f"{op}: uri {uri} != model {http['requestUri']}")
input_members = members(model, operation["input"]["shape"]) if "input" in operation else {}
required = set(model["shapes"][operation["input"]["shape"]].get("required", [])) if "input" in operation else set()
covered = set()
def find(location, wanted):
for m, spec in input_members.items():
loc = spec.get("location")
if loc == location and spec.get("locationName", m).lower() == wanted.lower():
return m
return None
for p in csv(path_p):
m = find("uri", p)
if not m:
bad("row", f"{op}: path param {p} is not a uri member")
else:
covered.add(m)
placeholders = re.findall(r"\{(\w+)\+?\}", uri)
if sorted(placeholders) != sorted(csv(path_p)):
bad("row", f"{op}: uri placeholders {placeholders} != path params {csv(path_p)}")
for q in csv(query_p):
m = find("querystring", q)
if not m:
bad("row", f"{op}: query param {q} is not a querystring member")
else:
covered.add(m)
for h in csv(header_p):
m = find("header", h)
if not m:
bad("row", f"{op}: header {h} is not a header member")
else:
covered.add(m)
for b in csv(body_p):
spec = input_members.get(b)
if not spec or spec.get("location"):
bad("row", f"{op}: body field {b} is not a body member")
else:
covered.add(b)
for r in sorted(required - covered):
bad("required", f"{op}: required member {r} is not covered by the row")
out = operation.get("output", {}).get("shape")
out_members = members(model, out) if out else {}
event_member = None
for m, spec in out_members.items():
if model["shapes"][spec["shape"]].get("eventstream"):
event_member = spec["shape"]
if bool(event_member) != (stream == "true"):
bad("stream", f"{op}: event_stream={stream} but model eventstream member is {event_member}")
expected = event_member if event_member else out
if response != expected:
bad("response", f"{op}: response schema {response} != model {expected}")
auth = model["metadata"].get("auth", [])
if ("smithy.api#httpBearerAuth" in auth) != (bearer == "true"):
bad("auth", f"{op}: bearer_ok={bearer} but model auth is {auth}")
for (schema, svc, fields) in schemas:
model = models.get(svc)
if not model or schema not in model["shapes"]:
bad("schema", f"{svc}/{schema}: shape not found")
continue
for path in csv(fields):
shape = schema
for part in path.split("."):
mem = members(model, shape)
if part not in mem:
bad("schema", f"{svc}/{schema}: {path}: {part} is not a member of {shape}")
break
shape = mem[part]["shape"]
# Lists of structures: descend through the member shape.
spec = model["shapes"][shape]
if spec["type"] == "list":
shape = spec["member"]["shape"]
for svc, model in models.items():
for op in sorted(model["operations"]):
if op not in used[svc] and op not in EXCLUDED.get(svc, {}):
bad("coverage", f"{svc}/{op} is neither in the call table nor in EXCLUDED")
for op in EXCLUDED.get(svc, {}):
if op not in model["operations"]:
bad("coverage", f"EXCLUDED {svc}/{op} is not an operation of the model")
if op in used[svc]:
bad("coverage", f"EXCLUDED {svc}/{op} is also in the call table")
total_ops = sum(len(m["operations"]) for m in models.values())
print(f"rows {len(rows)}, schemas {len(schemas)}, operations in models {total_ops}, "
f"excluded {sum(len(v) for v in EXCLUDED.values())}")
for svc, model in models.items():
print(f" {svc}: {len(model['operations'])} operations, apiVersion {model['metadata']['apiVersion']}, "
f"auth {model['metadata'].get('auth', ['(none listed)'])}")
if problems:
print(f"{len(problems)} problem(s):")
for p in problems:
print(" " + p)
return 1
print("OK")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))