feat(clients): armor: Model Armor, SCC agent findings, GuardedModel, discovery-checked call table
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,183 @@
|
||||
# Armor - Specification
|
||||
|
||||
Owned directory: `src/google/armor/`. Namespace `molo::google::armor`.
|
||||
Depends on the foundation in `src/google/` (`GoogleClient`, `Service`,
|
||||
`GoogleError`, `CloudContext`, `endpoints::custom`, `LanguageModel`). No other
|
||||
module.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
- Model Armor: templates (create, get, list, patch, delete), floor settings
|
||||
(get, update; project, folder, organization), locations (list, get),
|
||||
`sanitizeUserPrompt`, `sanitizeModelResponse` with typed filter results
|
||||
(prompt injection and jailbreak, sensitive data, malicious URIs,
|
||||
responsible AI, CSAM, virus scan).
|
||||
- Agent Threat Detection and Agent Anomaly Detection: they have NO API of
|
||||
their own. They publish findings to Security Command Center. Read-only
|
||||
access here is `SecurityFindings` over Security Command Center v2
|
||||
(`findings.list`, `findings:group`, `sources.list`, `sources.get`). The
|
||||
Finding resource carries typed `agent`, `agentAnomaly`, `agentSessions`,
|
||||
`agentDataAccessEvents`, `aiModel` and `vulnerability` members, which are
|
||||
read into `Finding`.
|
||||
- Agent Security dashboard: a console product with no dashboard API.
|
||||
`AgentSecurityDashboard` is a LOCAL abstraction that aggregates listed
|
||||
findings (counts by severity, category, class, state, per-asset worst
|
||||
severity). Agent assets are matched by the resource name fragment
|
||||
`reasoningEngines` (local convention, see Unverified).
|
||||
- Asset discovery: Security Command Center v2 has no assets list method
|
||||
(Discovery: only `assets.updateSecurityMarks`). `discovered_assets()` is
|
||||
local: distinct resources that carry findings.
|
||||
- `GuardedModel : LanguageModel`: sanitizes the prompt before and the response
|
||||
after any wrapped `LanguageModel`; throws `BlockedError`.
|
||||
|
||||
Authentication: Model Armor and Security Command Center REQUIRE an OAuth2
|
||||
access token (`GOOGLE_ACCESS_TOKEN`, scope `cloud-platform`). API keys,
|
||||
including express keys, are refused (see Verified live). Required project
|
||||
env: `GOOGLE_CLOUD_PROJECT`, optional `GOOGLE_CLOUD_LOCATION`.
|
||||
|
||||
## 2. Module map
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings (`ArmorCaps`) |
|
||||
| `calls.h/.cpp` | The ONE table of declared calls, schemas, enums; path matcher |
|
||||
| `tools/check_discovery.py` | Fails when `calls.h` and Google's Discovery documents disagree |
|
||||
| `armor_types.h/.cpp` | `SanitizationResult`, `FilterOutcome`, enums |
|
||||
| `template.h/.cpp` | `Template`, `FilterConfig`, `TemplateMetadata`, `FloorSetting`, rule sets |
|
||||
| `model_armor.h/.cpp` | `ModelArmor : Service`, `DataItem`, `LanguageHint` |
|
||||
| `armor_util.h/.cpp` | `query_escape`, `join_update_mask`, `valid_scope`, `valid_request_id` |
|
||||
| `findings.h/.cpp` | `SecurityFindings : Service`, `Finding`, `FindingQuery` |
|
||||
| `dashboard.h/.cpp` | `AgentSecurityDashboard` (local aggregation) |
|
||||
| `guarded_model.h/.cpp` | `Sanitizer` seam, `ModelArmorSanitizer`, `GuardedModel`, `BlockedError` |
|
||||
|
||||
## 3. Static caps (see caps.h)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Text per sanitize call | 100000 characters |
|
||||
| File bytes per sanitize call | 5 MiB (our ceiling) |
|
||||
| Stream buffer (guarded) | 200000 characters, 4096 chunks |
|
||||
| Template page size / pages followed | 100 / 10 |
|
||||
| Rai filters, rule sets, rules per set, words per rule, regex | 4, 16, 64, 1000, 1024 chars |
|
||||
| Labels per template | 64 (63 chars each) |
|
||||
| Update mask paths | 16 |
|
||||
| Detail strings per filter result | 50 |
|
||||
| Findings page size (Google max) / pages / results | 1000 / 10 / 5000 |
|
||||
| Categories in a filter | 20 |
|
||||
| Dashboard distinct assets | 500 |
|
||||
|
||||
## 4. Contracts
|
||||
|
||||
- Every id, scope, mask, filter value, order_by, enum string and size is
|
||||
validated before any network traffic (`GoogleError::invalid`, HTTP 400
|
||||
`INVALID_ARGUMENT`). No credential: `NotConfiguredError` (503) before the
|
||||
network.
|
||||
- Hosts: `https://modelarmor.{location}.rep.googleapis.com/v1` for templates,
|
||||
sanitize and floor settings; floor settings and locations also accept
|
||||
`global` (`https://modelarmor.googleapis.com/v1`); templates and sanitize
|
||||
reject `global`. Security Command Center: `https://securitycenter.googleapis.com/v2`.
|
||||
- Destructive: `delete_template` is an explicit method; the demo route also
|
||||
requires `confirm=delete`. Nothing here creates, updates or deletes
|
||||
findings, sources or any Security Command Center resource.
|
||||
- `retry ids`: `requestId` is accepted on create, patch and delete as a
|
||||
non-zero UUID.
|
||||
- Sanitization results are typed; an unknown or missing `invocationResult`
|
||||
never counts as clean (`fully_executed()` is false).
|
||||
- `GuardedModel` fails closed: a matched filter blocks (`incomplete() ==
|
||||
false`); a PARTIAL or FAILURE run, or any skipped filter, also blocks
|
||||
(`incomplete() == true`) unless `GuardOptions::allow_partial`. An
|
||||
unreachable sanitizer propagates its error and the model is not called
|
||||
(prompt stage) or the response is not returned (response stage). Requests
|
||||
with no text (only files) are counted in `skipped_empty`, not checked.
|
||||
Streaming buffers the whole stream, sanitizes the full text once, then
|
||||
replays the chunks; a blocked stream delivers nothing.
|
||||
- Findings: `FindingQuery` builds an AIP-160 filter from typed fields only
|
||||
(`category`, `state`, `severity`, `resourceName` substring). `findingClass`
|
||||
is filtered locally because Discovery lists it for `orderBy` only.
|
||||
|
||||
## 5. Verified against discovery
|
||||
|
||||
`python3 -I src/google/armor/tools/check_discovery.py` fetches
|
||||
`https://modelarmor.googleapis.com/$discovery/rest?version=v1` and
|
||||
`https://securitycenter.googleapis.com/$discovery/rest?version=v2` and checks
|
||||
every row of `calls.h`: 31 calls (rpc id, HTTP method, path, query
|
||||
parameters), 66 schemas (every field this module reads or writes) and 13 enum
|
||||
value sets. Result on 2026-10-24: `OK: 31 calls, 66 schemas, 13 enums
|
||||
checked, 0 problems`. Unit tests additionally prove that every request the
|
||||
client puts on the wire matches a declared call and only uses declared query
|
||||
parameters, and that every key of every request body built here is a field of
|
||||
its Discovery schema (nested schemas included).
|
||||
|
||||
Discovery facts that differ from the prose docs and are implemented as
|
||||
Discovery states them: `inspectOnly` and `inspectAndBlock` on floor settings
|
||||
are booleans; `FloorSetting.googleMcpServerFloorSetting` exists; the CSAM
|
||||
result member is spelled `csamFilterFilterResult`; templates carry
|
||||
`filterRuleSettings` (exclusion dictionaries and regex), custom error codes
|
||||
and messages, `filterVersionSelector`, `modalities`, `dataResidencyCompliant`;
|
||||
`ListTemplatesResponse.unreachable` exists; create, patch and delete accept
|
||||
`requestId`; the Discovery root URL lists `us-central1` as the sample region.
|
||||
|
||||
## 6. Verified live
|
||||
|
||||
Date 2026-10-24, project from `GOOGLE_CLOUD_PROJECT`, no resource created,
|
||||
updated or deleted.
|
||||
|
||||
| Call | Credential | Result |
|
||||
|---|---|---|
|
||||
| Model Armor `templates.list` | express key (`AQ.`) | 401 UNAUTHENTICATED (API keys are not supported); `google_live_armor.express_key_is_refused` |
|
||||
| Model Armor `templates.list` | service account OAuth token | 403 PERMISSION_DENIED: "Model Armor API has not been used in project before or it is disabled" |
|
||||
| Security Command Center v2 `findings.list` | service account OAuth token | 403 PERMISSION_DENIED: API not enabled |
|
||||
| Security Command Center v2 `sources` | express key | 401 (reported by the coordinator) |
|
||||
|
||||
So authentication and routing to both services work with an OAuth token; the
|
||||
request shapes beyond the 403 gate, and every response parser, are NOT yet
|
||||
exercised against a real reply. The opt-in test
|
||||
`google_live_armor.oauth_token_reads_templates_and_findings` (`MOLO_LIVE_NET=1`,
|
||||
`GOOGLE_ACCESS_TOKEN`, `GOOGLE_CLOUD_PROJECT`; loud SKIP otherwise) is read-only
|
||||
(list templates, get floor setting, list project findings, and
|
||||
`sanitizeUserPrompt` only when a template already exists) and reports a
|
||||
disabled API loudly. Run it once the APIs are enabled.
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- Discovery documents above (authoritative for paths, methods, fields, enums).
|
||||
- https://docs.cloud.google.com/security-command-center/docs/reference/model-armor/rest
|
||||
- https://docs.cloud.google.com/model-armor/reference/rest/v1/ (Template,
|
||||
FloorSetting, SanitizationResult, DataItem, templates.* method pages)
|
||||
- https://docs.cloud.google.com/security-command-center/docs/reference/rest/v2/organizations.sources.findings/list
|
||||
- https://docs.cloud.google.com/security-command-center/docs/reference/rest/v2/ListFindingsResponse
|
||||
- https://docs.cloud.google.com/gemini-enterprise-agent-platform/agent-anomalies-overview
|
||||
(anomaly findings are surfaced in Security Command Center)
|
||||
- https://docs.cloud.google.com/security-command-center/docs/overview-threats
|
||||
(Agent Platform threat findings)
|
||||
|
||||
## 8. Unverified
|
||||
|
||||
Nothing neither Discovery nor docs state:
|
||||
|
||||
- Finding `category` strings for Agent Threat Detection and Agent Anomaly
|
||||
Detection: not read; callers pass categories they see in the console.
|
||||
- That agent assets always contain `reasoningEngines` in `resourceName`
|
||||
(dashboard default filter; override with `resource_contains`).
|
||||
- Server-side behavior of the `resourceName:"x"` substring operator (AIP-160
|
||||
`:` is documented generally, not for this field).
|
||||
- Real response bodies: parsers follow Discovery schemas and are tested with
|
||||
documents built from schema field names, not with recorded Google replies.
|
||||
The FilterResult map keys (`rai`, `pi_and_jailbreak`, ...) are not relied
|
||||
upon; filters are identified by their member name.
|
||||
- Google's byte limit for file items in sanitize calls.
|
||||
- Whether `sources.get` exists for folders and projects (Discovery: only
|
||||
organizations; `get_source` accepts only organizations).
|
||||
- Gemini `promptFeedback` and `safetyRatings` were not live-checked; the guard
|
||||
does not use them.
|
||||
|
||||
## 9. Gaps
|
||||
|
||||
- `findings:group` is wired; the dashboard still aggregates locally (owner:
|
||||
`dashboard.cpp`). Server-side counting is the cheaper path for large sets.
|
||||
- Streaming sanitize (`streamingMode`, gRPC `StreamSanitize*`) is not
|
||||
implemented; `GuardedModel` buffers instead.
|
||||
- No asset inventory (Cloud Asset Inventory is a different API).
|
||||
- Findings mutation (mute, state), notification configs, BigQuery exports,
|
||||
mute configs and attack paths are out of scope (read-only module).
|
||||
- Token refresh is the caller's job (foundation gap).
|
||||
@@ -0,0 +1,220 @@
|
||||
#include "google/armor/armor_types.h"
|
||||
|
||||
#include "google/armor/caps.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
const Json* member(const Json& doc, const char* key) {
|
||||
return doc.is_object() ? doc.find(key) : nullptr;
|
||||
}
|
||||
|
||||
std::string text_of(const Json& doc, const char* key) {
|
||||
const Json* v = member(doc, key);
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
|
||||
MatchState match_from(const std::string& wire) {
|
||||
if (wire == "MATCH_FOUND") return MatchState::kMatch;
|
||||
if (wire == "NO_MATCH_FOUND") return MatchState::kNoMatch;
|
||||
return MatchState::kUnspecified;
|
||||
}
|
||||
|
||||
const char* match_name(MatchState m) {
|
||||
switch (m) {
|
||||
case MatchState::kMatch: return "MATCH_FOUND";
|
||||
case MatchState::kNoMatch: return "NO_MATCH_FOUND";
|
||||
default: return "";
|
||||
}
|
||||
}
|
||||
|
||||
void add_detail(FilterOutcome& out, const std::string& d) {
|
||||
if (out.details.size() < ArmorCaps::kMaxDetailsPerFilter) out.details.push_back(d);
|
||||
}
|
||||
|
||||
void read_common(FilterOutcome& out, const Json& body) {
|
||||
out.match = match_from(text_of(body, "matchState"));
|
||||
std::string exec = text_of(body, "executionState");
|
||||
out.execution = exec == "EXECUTION_SUCCESS" ? ExecutionState::kSuccess
|
||||
: exec == "EXECUTION_SKIPPED" ? ExecutionState::kSkipped
|
||||
: ExecutionState::kUnknown;
|
||||
out.confidence = confidence_from(text_of(body, "confidenceLevel"));
|
||||
if (const Json* items = member(body, "messageItems"); items && items->is_array())
|
||||
for (const Json& m : items->as_array()) {
|
||||
std::string t = text_of(m, "message");
|
||||
if (!t.empty() && out.messages.size() < ArmorCaps::kMaxDetailsPerFilter)
|
||||
out.messages.push_back(t);
|
||||
}
|
||||
}
|
||||
|
||||
// Reads one FilterResult object (exactly one of the *FilterResult members).
|
||||
bool read_filter(const Json& wrapper, FilterOutcome& out) {
|
||||
if (const Json* b = member(wrapper, "raiFilterResult")) {
|
||||
out.kind = FilterKind::kResponsibleAi;
|
||||
read_common(out, *b);
|
||||
if (const Json* types = member(*b, "raiFilterTypeResults");
|
||||
types && types->is_object())
|
||||
for (const auto& [name, r] : types->as_object())
|
||||
if (match_from(text_of(r, "matchState")) == MatchState::kMatch)
|
||||
add_detail(out, name + ":" + text_of(r, "confidenceLevel"));
|
||||
return true;
|
||||
}
|
||||
if (const Json* b = member(wrapper, "sdpFilterResult")) {
|
||||
out.kind = FilterKind::kSensitiveData;
|
||||
for (const char* variant : {"inspectResult", "deidentifyResult", "redactResult"}) {
|
||||
const Json* v = member(*b, variant);
|
||||
if (!v) continue;
|
||||
read_common(out, *v);
|
||||
if (const Json* f = member(*v, "findings"); f && f->is_array())
|
||||
for (const Json& finding : f->as_array())
|
||||
add_detail(out, text_of(finding, "infoType"));
|
||||
if (const Json* t = member(*v, "infoTypes"); t && t->is_array())
|
||||
for (const Json& i : t->as_array())
|
||||
if (i.is_string()) add_detail(out, i.as_string());
|
||||
}
|
||||
return true;
|
||||
}
|
||||
if (const Json* b = member(wrapper, "piAndJailbreakFilterResult")) {
|
||||
out.kind = FilterKind::kPromptInjectionJailbreak;
|
||||
read_common(out, *b);
|
||||
return true;
|
||||
}
|
||||
if (const Json* b = member(wrapper, "maliciousUriFilterResult")) {
|
||||
out.kind = FilterKind::kMaliciousUri;
|
||||
read_common(out, *b);
|
||||
if (const Json* u = member(*b, "maliciousUriMatchedItems"); u && u->is_array())
|
||||
for (const Json& item : u->as_array()) add_detail(out, text_of(item, "uri"));
|
||||
return true;
|
||||
}
|
||||
if (const Json* b = member(wrapper, "csamFilterFilterResult")) {
|
||||
out.kind = FilterKind::kCsam;
|
||||
read_common(out, *b);
|
||||
return true;
|
||||
}
|
||||
if (const Json* b = member(wrapper, "virusScanFilterResult")) {
|
||||
out.kind = FilterKind::kVirusScan;
|
||||
read_common(out, *b);
|
||||
if (const Json* d = member(*b, "virusDetails")) {
|
||||
if (const Json* n = member(*d, "names"); n && n->is_array())
|
||||
for (const Json& i : n->as_array())
|
||||
if (i.is_string()) add_detail(out, i.as_string());
|
||||
}
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const char* filter_kind_name(FilterKind kind) {
|
||||
switch (kind) {
|
||||
case FilterKind::kResponsibleAi: return "rai";
|
||||
case FilterKind::kSensitiveData: return "sdp";
|
||||
case FilterKind::kPromptInjectionJailbreak: return "pi_and_jailbreak";
|
||||
case FilterKind::kMaliciousUri: return "malicious_uris";
|
||||
case FilterKind::kCsam: return "csam";
|
||||
case FilterKind::kVirusScan: return "virus_scan";
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
const char* confidence_name(Confidence c) {
|
||||
switch (c) {
|
||||
case Confidence::kLowAndAbove: return "LOW_AND_ABOVE";
|
||||
case Confidence::kMediumAndAbove: return "MEDIUM_AND_ABOVE";
|
||||
case Confidence::kHigh: return "HIGH";
|
||||
default: return "";
|
||||
}
|
||||
}
|
||||
|
||||
Confidence confidence_from(const std::string& wire) {
|
||||
if (wire == "LOW_AND_ABOVE") return Confidence::kLowAndAbove;
|
||||
if (wire == "MEDIUM_AND_ABOVE") return Confidence::kMediumAndAbove;
|
||||
if (wire == "HIGH") return Confidence::kHigh;
|
||||
return Confidence::kUnspecified;
|
||||
}
|
||||
|
||||
const char* invocation_name(Invocation i) {
|
||||
switch (i) {
|
||||
case Invocation::kSuccess: return "SUCCESS";
|
||||
case Invocation::kPartial: return "PARTIAL";
|
||||
case Invocation::kFailure: return "FAILURE";
|
||||
default: return "";
|
||||
}
|
||||
}
|
||||
|
||||
Json FilterOutcome::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("filter", filter_kind_name(kind));
|
||||
o.set("matched", matched());
|
||||
o.set("match_state", match_name(match));
|
||||
o.set("executed", execution == ExecutionState::kSuccess);
|
||||
if (confidence != Confidence::kUnspecified)
|
||||
o.set("confidence", confidence_name(confidence));
|
||||
Json d = Json::array();
|
||||
for (const auto& s : details) d.push(s);
|
||||
o.set("details", d);
|
||||
return o;
|
||||
}
|
||||
|
||||
bool SanitizationResult::fully_executed() const {
|
||||
if (invocation != Invocation::kSuccess) return false;
|
||||
for (const auto& f : filters)
|
||||
if (f.execution == ExecutionState::kSkipped) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
std::vector<FilterOutcome> SanitizationResult::matches() const {
|
||||
std::vector<FilterOutcome> out;
|
||||
for (const auto& f : filters)
|
||||
if (f.matched()) out.push_back(f);
|
||||
return out;
|
||||
}
|
||||
|
||||
const FilterOutcome* SanitizationResult::find(FilterKind kind) const {
|
||||
for (const auto& f : filters)
|
||||
if (f.kind == kind) return &f;
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
Json SanitizationResult::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("matched", matched());
|
||||
o.set("invocation", invocation_name(invocation));
|
||||
o.set("fully_executed", fully_executed());
|
||||
Json f = Json::array();
|
||||
for (const auto& x : filters) f.push(x.to_json());
|
||||
o.set("filters", f);
|
||||
if (!error_code.empty()) o.set("error_code", error_code);
|
||||
if (!error_message.empty()) o.set("error_message", error_message);
|
||||
return o;
|
||||
}
|
||||
|
||||
SanitizationResult SanitizationResult::from_json(const Json& doc) {
|
||||
SanitizationResult r;
|
||||
// The response wraps the result in "sanitizationResult".
|
||||
const Json* body = member(doc, "sanitizationResult");
|
||||
const Json& src = body ? *body : doc;
|
||||
r.match = match_from(text_of(src, "filterMatchState"));
|
||||
std::string inv = text_of(src, "invocationResult");
|
||||
r.invocation = inv == "SUCCESS" ? Invocation::kSuccess
|
||||
: inv == "PARTIAL" ? Invocation::kPartial
|
||||
: inv == "FAILURE" ? Invocation::kFailure
|
||||
: Invocation::kUnknown;
|
||||
if (const Json* meta = member(src, "sanitizationMetadata")) {
|
||||
r.error_code = text_of(*meta, "errorCode");
|
||||
r.error_message = text_of(*meta, "errorMessage");
|
||||
}
|
||||
if (const Json* fr = member(src, "filterResults"); fr && fr->is_object())
|
||||
for (const auto& [key, wrapper] : fr->as_object()) {
|
||||
(void)key;
|
||||
FilterOutcome out;
|
||||
if (read_filter(wrapper, out)) r.filters.push_back(std::move(out));
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,59 @@
|
||||
#pragma once
|
||||
// Typed Model Armor sanitization result. Wire names follow
|
||||
// SanitizationResult / FilterResult in the Model Armor v1 REST reference.
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
enum class FilterKind {
|
||||
kResponsibleAi,
|
||||
kSensitiveData,
|
||||
kPromptInjectionJailbreak,
|
||||
kMaliciousUri,
|
||||
kCsam,
|
||||
kVirusScan,
|
||||
};
|
||||
const char* filter_kind_name(FilterKind kind); // "rai", "sdp", ...
|
||||
|
||||
enum class MatchState { kUnspecified, kNoMatch, kMatch };
|
||||
enum class ExecutionState { kUnknown, kSuccess, kSkipped };
|
||||
enum class Invocation { kUnknown, kSuccess, kPartial, kFailure };
|
||||
enum class Confidence { kUnspecified, kLowAndAbove, kMediumAndAbove, kHigh };
|
||||
|
||||
const char* confidence_name(Confidence c); // Wire enum string, "" if unset.
|
||||
Confidence confidence_from(const std::string& wire); // Unknown: unspecified.
|
||||
const char* invocation_name(Invocation i);
|
||||
|
||||
struct FilterOutcome {
|
||||
FilterKind kind = FilterKind::kResponsibleAi;
|
||||
MatchState match = MatchState::kUnspecified;
|
||||
ExecutionState execution = ExecutionState::kUnknown;
|
||||
Confidence confidence = Confidence::kUnspecified;
|
||||
// Matched rai types ("hate_speech:HIGH"), sdp info types, malicious uris.
|
||||
std::vector<std::string> details;
|
||||
std::vector<std::string> messages; // MessageItem texts.
|
||||
|
||||
bool matched() const { return match == MatchState::kMatch; }
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
struct SanitizationResult {
|
||||
MatchState match = MatchState::kUnspecified;
|
||||
Invocation invocation = Invocation::kUnknown;
|
||||
std::vector<FilterOutcome> filters;
|
||||
std::string error_code; // sanitizationMetadata.errorCode
|
||||
std::string error_message; // sanitizationMetadata.errorMessage
|
||||
|
||||
bool matched() const { return match == MatchState::kMatch; }
|
||||
// Every filter ran: invocation success and no skipped filter.
|
||||
bool fully_executed() const;
|
||||
std::vector<FilterOutcome> matches() const;
|
||||
const FilterOutcome* find(FilterKind kind) const;
|
||||
clients::body::Json to_json() const;
|
||||
static SanitizationResult from_json(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,79 @@
|
||||
#include "google/armor/armor_util.h"
|
||||
|
||||
#include "google/armor/caps.h"
|
||||
#include "google/errors.h"
|
||||
#include "google/resource.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
std::string query_escape(const std::string& value) {
|
||||
static const char* hex = "0123456789ABCDEF";
|
||||
std::string out;
|
||||
for (unsigned char c : value) {
|
||||
bool plain = (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') ||
|
||||
(c >= '0' && c <= '9') || c == '-' || c == '_' || c == '.' ||
|
||||
c == '~';
|
||||
if (plain) {
|
||||
out.push_back(static_cast<char>(c));
|
||||
} else {
|
||||
out.push_back('%');
|
||||
out.push_back(hex[c >> 4]);
|
||||
out.push_back(hex[c & 15]);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::string join_update_mask(const std::vector<std::string>& paths) {
|
||||
if (paths.empty() || paths.size() > ArmorCaps::kMaxUpdateMaskPaths)
|
||||
throw GoogleError::invalid("update mask needs 1 to 16 paths");
|
||||
std::string out;
|
||||
for (const auto& p : paths) {
|
||||
if (p.empty() || p.size() > 128)
|
||||
throw GoogleError::invalid("bad update mask path");
|
||||
for (char c : p)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') ||
|
||||
(c >= '0' && c <= '9') || c == '_' || c == '.'))
|
||||
throw GoogleError::invalid("bad update mask path");
|
||||
if (!out.empty()) out.push_back(',');
|
||||
out += p;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool valid_request_id(const std::string& id) {
|
||||
if (id.size() != 36) return false;
|
||||
bool all_zero = true;
|
||||
for (std::size_t i = 0; i < id.size(); ++i) {
|
||||
char c = id[i];
|
||||
if (i == 8 || i == 13 || i == 18 || i == 23) {
|
||||
if (c != '-') return false;
|
||||
continue;
|
||||
}
|
||||
bool hex = (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F');
|
||||
if (!hex) return false;
|
||||
if (c != '0') all_zero = false;
|
||||
}
|
||||
return !all_zero;
|
||||
}
|
||||
|
||||
bool valid_scope(const std::string& scope) {
|
||||
std::size_t slash = scope.find('/');
|
||||
if (slash == std::string::npos) return false;
|
||||
std::string kind = scope.substr(0, slash);
|
||||
std::string id = scope.substr(slash + 1);
|
||||
if (kind != "projects" && kind != "folders" && kind != "organizations")
|
||||
return false;
|
||||
if (kind == "projects") {
|
||||
// Project ids may carry ':' (domain-scoped), which valid_segment refuses.
|
||||
if (id.empty() || id.size() > 128) return false;
|
||||
for (char c : id)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '-' ||
|
||||
c == '.' || c == ':'))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
return valid_segment(id);
|
||||
}
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,21 @@
|
||||
#pragma once
|
||||
// Small validators and encoders shared by the armor classes.
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
// Percent-encodes everything outside the RFC 3986 unreserved set.
|
||||
std::string query_escape(const std::string& value);
|
||||
|
||||
// "a,b.c" style FieldMask: comma separated, each path [A-Za-z0-9_.]+.
|
||||
// Throws GoogleError::invalid when empty, too long or malformed.
|
||||
std::string join_update_mask(const std::vector<std::string>& paths);
|
||||
|
||||
// 8-4-4-4-12 hex UUID that is not the all-zero UUID (requestId rule).
|
||||
bool valid_request_id(const std::string& id);
|
||||
|
||||
// "projects/x", "folders/1", "organizations/2": kind plus a valid id.
|
||||
bool valid_scope(const std::string& scope);
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,58 @@
|
||||
#include "google/armor/calls.h"
|
||||
|
||||
#include <vector>
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
bool path_matches(const std::string& tmpl, const std::string& path) {
|
||||
// Split the template into literals around {placeholders}.
|
||||
std::vector<std::string> literals;
|
||||
std::string cur;
|
||||
for (std::size_t i = 0; i < tmpl.size(); ++i) {
|
||||
if (tmpl[i] == '{') {
|
||||
literals.push_back(cur);
|
||||
cur.clear();
|
||||
i = tmpl.find('}', i);
|
||||
if (i == std::string::npos) return false;
|
||||
} else {
|
||||
cur.push_back(tmpl[i]);
|
||||
}
|
||||
}
|
||||
literals.push_back(cur);
|
||||
|
||||
const std::string& first = literals.front();
|
||||
if (path.compare(0, first.size(), first) != 0) return false;
|
||||
std::size_t pos = first.size();
|
||||
if (literals.size() == 1) return pos == path.size();
|
||||
for (std::size_t i = 1; i < literals.size(); ++i) {
|
||||
const std::string& lit = literals[i];
|
||||
bool last = i + 1 == literals.size();
|
||||
if (pos >= path.size()) return false; // Placeholder needs a character.
|
||||
if (last) {
|
||||
if (path.size() < pos + 1 + lit.size()) return false;
|
||||
return path.compare(path.size() - lit.size(), lit.size(), lit) == 0;
|
||||
}
|
||||
std::size_t at = lit.empty() ? std::string::npos : path.find(lit, pos + 1);
|
||||
if (at == std::string::npos) return false;
|
||||
pos = at + lit.size();
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
const CallSpec* find_rpc(const std::string& rpc) {
|
||||
for (const auto& c : kCalls)
|
||||
if (rpc == c.rpc) return &c;
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
bool request_matches(const std::string& rpc, const std::string& http,
|
||||
const std::string& target) {
|
||||
const CallSpec* c = find_rpc(rpc);
|
||||
if (!c || http != c->http) return false;
|
||||
std::string p = target;
|
||||
if (!p.empty() && p.front() == '/') p.erase(0, 1);
|
||||
if (std::size_t q = p.find('?'); q != std::string::npos) p.erase(q);
|
||||
return path_matches(c->path, p);
|
||||
}
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,177 @@
|
||||
#pragma once
|
||||
// The ONE declaration of every Google call this module makes, and of the
|
||||
// request/response field names it relies on. tools/check_discovery.py reads
|
||||
// this file (plain-text rows, one per line) and fails when Google's
|
||||
// Discovery documents disagree. Rows are copied from the Discovery
|
||||
// documents, never typed from memory:
|
||||
// https://modelarmor.googleapis.com/$discovery/rest?version=v1
|
||||
// https://securitycenter.googleapis.com/$discovery/rest?version=v2
|
||||
#include <cstddef>
|
||||
#include <string>
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
struct CallSpec {
|
||||
const char* service; // Discovery service name
|
||||
const char* version; // Discovery version
|
||||
const char* rpc; // Discovery method id
|
||||
const char* http; // GET | POST | PATCH | DELETE
|
||||
const char* path; // Discovery `path` (relative to the service root)
|
||||
const char* query; // Comma separated query parameters this module sends
|
||||
};
|
||||
|
||||
struct SchemaSpec {
|
||||
const char* service;
|
||||
const char* version;
|
||||
const char* schema; // Discovery schema name
|
||||
const char* fields; // Comma separated property names this module reads or writes
|
||||
};
|
||||
|
||||
struct EnumSpec {
|
||||
const char* service;
|
||||
const char* version;
|
||||
const char* schema;
|
||||
const char* field;
|
||||
const char* values; // Comma separated enum values this module uses
|
||||
};
|
||||
|
||||
// clang-format off
|
||||
inline constexpr CallSpec kCalls[] = {
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.templates.create", "POST", "v1/{+parent}/templates", "templateId,requestId"},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.templates.get", "GET", "v1/{+name}", ""},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.templates.list", "GET", "v1/{+parent}/templates", "pageSize,pageToken,filter,orderBy"},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.templates.patch", "PATCH", "v1/{+name}", "updateMask,requestId"},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.templates.delete", "DELETE", "v1/{+name}", "requestId"},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.templates.sanitizeUserPrompt", "POST", "v1/{+name}:sanitizeUserPrompt", ""},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.templates.sanitizeModelResponse", "POST", "v1/{+name}:sanitizeModelResponse", ""},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.getFloorSetting", "GET", "v1/{+name}", ""},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.updateFloorSetting", "PATCH", "v1/{+name}", "updateMask"},
|
||||
{"modelarmor", "v1", "modelarmor.folders.locations.getFloorSetting", "GET", "v1/{+name}", ""},
|
||||
{"modelarmor", "v1", "modelarmor.folders.locations.updateFloorSetting", "PATCH", "v1/{+name}", "updateMask"},
|
||||
{"modelarmor", "v1", "modelarmor.organizations.locations.getFloorSetting", "GET", "v1/{+name}", ""},
|
||||
{"modelarmor", "v1", "modelarmor.organizations.locations.updateFloorSetting", "PATCH", "v1/{+name}", "updateMask"},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.list", "GET", "v1/{+name}/locations", "pageToken"},
|
||||
{"modelarmor", "v1", "modelarmor.projects.locations.get", "GET", "v1/{+name}", ""},
|
||||
{"securitycenter", "v2", "securitycenter.organizations.sources.findings.list", "GET", "v2/{+parent}/findings", "pageSize,pageToken,filter,orderBy,fieldMask"},
|
||||
{"securitycenter", "v2", "securitycenter.folders.sources.findings.list", "GET", "v2/{+parent}/findings", "pageSize,pageToken,filter,orderBy,fieldMask"},
|
||||
{"securitycenter", "v2", "securitycenter.projects.sources.findings.list", "GET", "v2/{+parent}/findings", "pageSize,pageToken,filter,orderBy,fieldMask"},
|
||||
{"securitycenter", "v2", "securitycenter.organizations.sources.locations.findings.list", "GET", "v2/{+parent}/findings", "pageSize,pageToken,filter,orderBy,fieldMask"},
|
||||
{"securitycenter", "v2", "securitycenter.folders.sources.locations.findings.list", "GET", "v2/{+parent}/findings", "pageSize,pageToken,filter,orderBy,fieldMask"},
|
||||
{"securitycenter", "v2", "securitycenter.projects.sources.locations.findings.list", "GET", "v2/{+parent}/findings", "pageSize,pageToken,filter,orderBy,fieldMask"},
|
||||
{"securitycenter", "v2", "securitycenter.organizations.sources.findings.group", "POST", "v2/{+parent}/findings:group", ""},
|
||||
{"securitycenter", "v2", "securitycenter.folders.sources.findings.group", "POST", "v2/{+parent}/findings:group", ""},
|
||||
{"securitycenter", "v2", "securitycenter.projects.sources.findings.group", "POST", "v2/{+parent}/findings:group", ""},
|
||||
{"securitycenter", "v2", "securitycenter.organizations.sources.locations.findings.group", "POST", "v2/{+parent}/findings:group", ""},
|
||||
{"securitycenter", "v2", "securitycenter.folders.sources.locations.findings.group", "POST", "v2/{+parent}/findings:group", ""},
|
||||
{"securitycenter", "v2", "securitycenter.projects.sources.locations.findings.group", "POST", "v2/{+parent}/findings:group", ""},
|
||||
{"securitycenter", "v2", "securitycenter.organizations.sources.list", "GET", "v2/{+parent}/sources", "pageSize,pageToken"},
|
||||
{"securitycenter", "v2", "securitycenter.folders.sources.list", "GET", "v2/{+parent}/sources", "pageSize,pageToken"},
|
||||
{"securitycenter", "v2", "securitycenter.projects.sources.list", "GET", "v2/{+parent}/sources", "pageSize,pageToken"},
|
||||
{"securitycenter", "v2", "securitycenter.organizations.sources.get", "GET", "v2/{+name}", ""},
|
||||
};
|
||||
|
||||
// Request bodies (what to_json emits) and response documents (what from_json reads).
|
||||
inline constexpr SchemaSpec kSchemas[] = {
|
||||
{"modelarmor", "v1", "Template", "name,labels,filterConfig,templateMetadata,createTime,updateTime"},
|
||||
{"modelarmor", "v1", "FilterConfig", "raiSettings,sdpSettings,piAndJailbreakFilterSettings,maliciousUriFilterSettings,filterRuleSettings"},
|
||||
{"modelarmor", "v1", "TemplateMetadata", "enforcementType,ignorePartialInvocationFailures,logTemplateOperations,logSanitizeOperations,multiLanguageDetection,customPromptSafetyErrorCode,customPromptSafetyErrorMessage,customLlmResponseSafetyErrorCode,customLlmResponseSafetyErrorMessage,filterVersionSelector,modalities,dataResidencyCompliant"},
|
||||
{"modelarmor", "v1", "MultiLanguageDetection", "enableMultiLanguageDetection"},
|
||||
{"modelarmor", "v1", "FilterVersionSelector", "version,alias"},
|
||||
{"modelarmor", "v1", "RaiFilterSettings", "raiFilters"},
|
||||
{"modelarmor", "v1", "RaiFilter", "filterType,confidenceLevel"},
|
||||
{"modelarmor", "v1", "FilterRuleSettings", "ruleSets"},
|
||||
{"modelarmor", "v1", "FilterRuleSet", "filterTypes,rules"},
|
||||
{"modelarmor", "v1", "FilterRule", "exclusionRule"},
|
||||
{"modelarmor", "v1", "ExclusionRule", "dictionary,regex,matchingScope"},
|
||||
{"modelarmor", "v1", "Dictionary", "wordList"},
|
||||
{"modelarmor", "v1", "WordList", "words"},
|
||||
{"modelarmor", "v1", "Regex", "pattern"},
|
||||
{"modelarmor", "v1", "SdpFilterSettings", "basicConfig,advancedConfig"},
|
||||
{"modelarmor", "v1", "SdpBasicConfig", "filterEnforcement"},
|
||||
{"modelarmor", "v1", "SdpAdvancedConfig", "inspectTemplate,deidentifyTemplate"},
|
||||
{"modelarmor", "v1", "PiAndJailbreakFilterSettings", "filterEnforcement,confidenceLevel"},
|
||||
{"modelarmor", "v1", "MaliciousUriFilterSettings", "filterEnforcement"},
|
||||
{"modelarmor", "v1", "FloorSetting", "name,updateTime,filterConfig,enableFloorSettingEnforcement,integratedServices,aiPlatformFloorSetting,googleMcpServerFloorSetting,floorSettingMetadata"},
|
||||
{"modelarmor", "v1", "AiPlatformFloorSetting", "enableCloudLogging,inspectOnly,inspectAndBlock"},
|
||||
{"modelarmor", "v1", "McpServerFloorSetting", "enableCloudLogging,inspectOnly,inspectAndBlock"},
|
||||
{"modelarmor", "v1", "FloorSettingMetadata", "multiLanguageDetection"},
|
||||
{"modelarmor", "v1", "FloorSettingFloorSettingMetadataMultiLanguageDetection", "enableMultiLanguageDetection"},
|
||||
{"modelarmor", "v1", "ListTemplatesResponse", "templates,nextPageToken,unreachable"},
|
||||
{"modelarmor", "v1", "ListLocationsResponse", "locations,nextPageToken"},
|
||||
{"modelarmor", "v1", "Location", "name,locationId,displayName"},
|
||||
{"modelarmor", "v1", "SanitizeUserPromptRequest", "userPromptData,multiLanguageDetectionMetadata"},
|
||||
{"modelarmor", "v1", "SanitizeModelResponseRequest", "modelResponseData,userPrompt,multiLanguageDetectionMetadata"},
|
||||
{"modelarmor", "v1", "MultiLanguageDetectionMetadata", "enableMultiLanguageDetection,sourceLanguage"},
|
||||
{"modelarmor", "v1", "DataItem", "text,byteItem"},
|
||||
{"modelarmor", "v1", "ByteDataItem", "byteDataType,byteData,fileLabel"},
|
||||
{"modelarmor", "v1", "SanitizeUserPromptResponse", "sanitizationResult"},
|
||||
{"modelarmor", "v1", "SanitizeModelResponseResponse", "sanitizationResult"},
|
||||
{"modelarmor", "v1", "SanitizationResult", "filterMatchState,filterResults,invocationResult,sanitizationMetadata"},
|
||||
{"modelarmor", "v1", "SanitizationMetadata", "errorCode,errorMessage"},
|
||||
{"modelarmor", "v1", "FilterResult", "raiFilterResult,sdpFilterResult,piAndJailbreakFilterResult,maliciousUriFilterResult,csamFilterFilterResult,virusScanFilterResult"},
|
||||
{"modelarmor", "v1", "RaiFilterResult", "executionState,messageItems,matchState,raiFilterTypeResults"},
|
||||
{"modelarmor", "v1", "RaiFilterTypeResult", "filterType,confidenceLevel,matchState"},
|
||||
{"modelarmor", "v1", "SdpFilterResult", "inspectResult,deidentifyResult,redactResult"},
|
||||
{"modelarmor", "v1", "SdpInspectResult", "executionState,messageItems,matchState,findings"},
|
||||
{"modelarmor", "v1", "SdpDeidentifyResult", "executionState,messageItems,matchState,infoTypes"},
|
||||
{"modelarmor", "v1", "SdpRedactResult", "executionState,messageItems,matchState,findings"},
|
||||
{"modelarmor", "v1", "SdpFinding", "infoType,likelihood"},
|
||||
{"modelarmor", "v1", "PiAndJailbreakFilterResult", "executionState,messageItems,matchState,confidenceLevel"},
|
||||
{"modelarmor", "v1", "MaliciousUriFilterResult", "executionState,messageItems,matchState,maliciousUriMatchedItems"},
|
||||
{"modelarmor", "v1", "MaliciousUriMatchedItem", "uri"},
|
||||
{"modelarmor", "v1", "CsamFilterResult", "executionState,messageItems,matchState"},
|
||||
{"modelarmor", "v1", "VirusScanFilterResult", "executionState,messageItems,matchState,virusDetails"},
|
||||
{"modelarmor", "v1", "VirusDetail", "names"},
|
||||
{"modelarmor", "v1", "MessageItem", "messageType,message"},
|
||||
{"securitycenter", "v2", "GroupFindingsRequest", "filter,groupBy,pageSize,pageToken"},
|
||||
{"securitycenter", "v2", "GroupFindingsResponse", "groupByResults,nextPageToken,totalSize"},
|
||||
{"securitycenter", "v2", "GroupResult", "count,properties"},
|
||||
{"securitycenter", "v2", "ListFindingsResponse", "listFindingsResults,nextPageToken,totalSize"},
|
||||
{"securitycenter", "v2", "ListFindingsResult", "finding,resource"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2Finding", "name,parent,category,state,severity,findingClass,mute,resourceName,eventTime,createTime,description,externalUri,moduleName,agent,agentAnomaly,agentSessions,agentDataAccessEvents,aiModel,vulnerability"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2Agent", "id,displayName"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2AgentAnomaly", "confidenceScore,invocationReferences,detectorReferences"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2InvocationReference", "invocationId"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2DetectorReference", "detectorId,displayName,severity,explanation,recommendation"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2AgentSession", "sessionId"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2Vulnerability", "cve"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2Cve", "id"},
|
||||
{"securitycenter", "v2", "ListSourcesResponse", "sources,nextPageToken"},
|
||||
{"securitycenter", "v2", "Source", "name,displayName,description,canonicalName"},
|
||||
};
|
||||
|
||||
inline constexpr EnumSpec kEnums[] = {
|
||||
{"modelarmor", "v1", "RaiFilter", "filterType", "SEXUALLY_EXPLICIT,HATE_SPEECH,HARASSMENT,DANGEROUS"},
|
||||
{"modelarmor", "v1", "RaiFilter", "confidenceLevel", "LOW_AND_ABOVE,MEDIUM_AND_ABOVE,HIGH"},
|
||||
{"modelarmor", "v1", "PiAndJailbreakFilterSettings", "filterEnforcement", "ENABLED,DISABLED"},
|
||||
{"modelarmor", "v1", "MaliciousUriFilterSettings", "filterEnforcement", "ENABLED,DISABLED"},
|
||||
{"modelarmor", "v1", "SdpBasicConfig", "filterEnforcement", "ENABLED,DISABLED"},
|
||||
{"modelarmor", "v1", "TemplateMetadata", "enforcementType", "INSPECT_ONLY,INSPECT_AND_BLOCK"},
|
||||
{"modelarmor", "v1", "SanitizationResult", "filterMatchState", "NO_MATCH_FOUND,MATCH_FOUND"},
|
||||
{"modelarmor", "v1", "SanitizationResult", "invocationResult", "SUCCESS,PARTIAL,FAILURE"},
|
||||
{"modelarmor", "v1", "RaiFilterResult", "executionState", "EXECUTION_SUCCESS,EXECUTION_SKIPPED"},
|
||||
{"modelarmor", "v1", "ByteDataItem", "byteDataType", "PLAINTEXT_UTF8,PDF,WORD_DOCUMENT,EXCEL_DOCUMENT,POWERPOINT_DOCUMENT,TXT,CSV,IMAGE"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2Finding", "state", "ACTIVE,INACTIVE"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2Finding", "severity", "CRITICAL,HIGH,MEDIUM,LOW"},
|
||||
{"securitycenter", "v2", "GoogleCloudSecuritycenterV2Finding", "findingClass", "THREAT,VULNERABILITY,MISCONFIGURATION,OBSERVATION"},
|
||||
};
|
||||
// clang-format on
|
||||
|
||||
inline constexpr std::size_t kCallCount = sizeof(kCalls) / sizeof(kCalls[0]);
|
||||
inline constexpr std::size_t kSchemaCount = sizeof(kSchemas) / sizeof(kSchemas[0]);
|
||||
inline constexpr std::size_t kEnumCount = sizeof(kEnums) / sizeof(kEnums[0]);
|
||||
|
||||
// Matches a request path (no query, no leading slash) against a Discovery
|
||||
// path template such as "v1/{+parent}/templates": placeholders match one or
|
||||
// more characters.
|
||||
bool path_matches(const std::string& discovery_path, const std::string& path);
|
||||
|
||||
// The declared call with this Discovery method id, or nullptr.
|
||||
const CallSpec* find_rpc(const std::string& rpc);
|
||||
|
||||
// True when `http` and the request target (path plus optional query, with or
|
||||
// without a leading slash) match the declared call `rpc`.
|
||||
bool request_matches(const std::string& rpc, const std::string& http,
|
||||
const std::string& target);
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,43 @@
|
||||
#pragma once
|
||||
// Static ceilings for the armor module. Never autotuned.
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
struct ArmorCaps {
|
||||
// Text sent to sanitizeUserPrompt / sanitizeModelResponse per call.
|
||||
static constexpr std::size_t kMaxTextChars = 100000;
|
||||
// Text held while a stream is buffered for response sanitization.
|
||||
static constexpr std::size_t kMaxStreamBufferChars = 200000;
|
||||
// Chunks held while a stream is buffered.
|
||||
static constexpr std::size_t kMaxStreamChunks = 4096;
|
||||
// Templates: page size and pages followed per list call.
|
||||
static constexpr int kMaxTemplatePageSize = 100;
|
||||
static constexpr std::size_t kMaxTemplatePages = 10;
|
||||
// File bytes per sanitize call (our ceiling; Google's is not documented).
|
||||
static constexpr std::size_t kMaxByteItemBytes = 5u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxRaiFilters = 4;
|
||||
static constexpr std::size_t kMaxRuleSets = 16;
|
||||
static constexpr std::size_t kMaxRulesPerSet = 64;
|
||||
static constexpr std::size_t kMaxRuleWords = 1000;
|
||||
static constexpr std::size_t kMaxRegexChars = 1024;
|
||||
static constexpr std::size_t kMaxLocations = 100;
|
||||
static constexpr std::size_t kMaxGroupResults = 1000;
|
||||
static constexpr std::size_t kMaxLabels = 64;
|
||||
static constexpr std::size_t kMaxLabelChars = 63;
|
||||
static constexpr std::size_t kMaxUpdateMaskPaths = 16;
|
||||
// Result detail lists kept per filter (matched uris, info types).
|
||||
static constexpr std::size_t kMaxDetailsPerFilter = 50;
|
||||
// Security Command Center findings.
|
||||
static constexpr int kMaxFindingsPageSize = 1000; // Google's maximum.
|
||||
static constexpr int kDefaultFindingsPageSize = 100;
|
||||
static constexpr std::size_t kMaxFindingPages = 10;
|
||||
static constexpr std::size_t kMaxFindings = 5000;
|
||||
static constexpr std::size_t kMaxFilterValueChars = 256;
|
||||
static constexpr std::size_t kMaxFilterCategories = 20;
|
||||
static constexpr std::size_t kMaxSources = 100;
|
||||
// Distinct assets reported by the local dashboard.
|
||||
static constexpr std::size_t kMaxDashboardAssets = 500;
|
||||
};
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,109 @@
|
||||
#include "google/armor/dashboard.h"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
#include "google/armor/caps.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
int rank(const std::string& severity) {
|
||||
if (severity == "CRITICAL") return 4;
|
||||
if (severity == "HIGH") return 3;
|
||||
if (severity == "MEDIUM") return 2;
|
||||
if (severity == "LOW") return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
Json counts(const std::map<std::string, std::size_t>& m) {
|
||||
Json o = Json::object();
|
||||
for (const auto& [k, v] : m) o.set(k.empty() ? "UNSPECIFIED" : k, static_cast<long>(v));
|
||||
return o;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Json AssetSummary::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("resource", resource_name);
|
||||
o.set("findings", static_cast<long>(findings));
|
||||
o.set("worst_severity", worst_severity);
|
||||
return o;
|
||||
}
|
||||
|
||||
Json SecuritySummary::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("total", static_cast<long>(total));
|
||||
o.set("agent_findings", static_cast<long>(agent_findings));
|
||||
o.set("anomalies", static_cast<long>(anomalies));
|
||||
o.set("by_severity", counts(by_severity));
|
||||
o.set("by_category", counts(by_category));
|
||||
o.set("by_class", counts(by_class));
|
||||
o.set("by_state", counts(by_state));
|
||||
Json a = Json::array();
|
||||
for (const auto& x : assets) a.push(x.to_json());
|
||||
o.set("assets", a);
|
||||
o.set("truncated", truncated);
|
||||
return o;
|
||||
}
|
||||
|
||||
FindingQuery AgentSecurityDashboard::with_agent_scope(FindingQuery query) {
|
||||
if (query.resource_contains.empty()) query.resource_contains = "reasoningEngines";
|
||||
return query;
|
||||
}
|
||||
|
||||
SecuritySummary AgentSecurityDashboard::summarize_findings(const FindingPage& page) {
|
||||
SecuritySummary s;
|
||||
s.truncated = page.truncated;
|
||||
std::map<std::string, AssetSummary> assets;
|
||||
for (const auto& f : page.findings) {
|
||||
++s.total;
|
||||
if (f.is_agent_finding()) ++s.agent_findings;
|
||||
if (f.has_anomaly) ++s.anomalies;
|
||||
++s.by_severity[f.severity];
|
||||
++s.by_category[f.category];
|
||||
++s.by_class[f.finding_class];
|
||||
++s.by_state[f.state];
|
||||
if (f.resource_name.empty()) continue;
|
||||
auto it = assets.find(f.resource_name);
|
||||
if (it == assets.end()) {
|
||||
if (assets.size() >= ArmorCaps::kMaxDashboardAssets) {
|
||||
s.truncated = true;
|
||||
continue;
|
||||
}
|
||||
it = assets.emplace(f.resource_name, AssetSummary{f.resource_name, 0, ""}).first;
|
||||
}
|
||||
++it->second.findings;
|
||||
if (rank(f.severity) > rank(it->second.worst_severity))
|
||||
it->second.worst_severity = f.severity;
|
||||
}
|
||||
for (auto& [name, a] : assets) {
|
||||
(void)name;
|
||||
s.assets.push_back(std::move(a));
|
||||
}
|
||||
std::sort(s.assets.begin(), s.assets.end(), [](const AssetSummary& a, const AssetSummary& b) {
|
||||
if (rank(a.worst_severity) != rank(b.worst_severity))
|
||||
return rank(a.worst_severity) > rank(b.worst_severity);
|
||||
if (a.findings != b.findings) return a.findings > b.findings;
|
||||
return a.resource_name < b.resource_name;
|
||||
});
|
||||
return s;
|
||||
}
|
||||
|
||||
SecuritySummary AgentSecurityDashboard::summarize(FindingQuery query) {
|
||||
return summarize_findings(findings_.list(with_agent_scope(std::move(query))));
|
||||
}
|
||||
|
||||
std::vector<AssetSummary> AgentSecurityDashboard::discovered_assets(FindingQuery query) {
|
||||
return summarize(std::move(query)).assets;
|
||||
}
|
||||
|
||||
FindingPage AgentSecurityDashboard::vulnerabilities(FindingQuery query) {
|
||||
query.finding_class = "VULNERABILITY";
|
||||
return findings_.list(with_agent_scope(std::move(query)));
|
||||
}
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,53 @@
|
||||
#pragma once
|
||||
// Agent Security dashboard, LOCAL abstraction: Google documents the console
|
||||
// dashboard but no dashboard API. This class aggregates the findings that
|
||||
// SecurityFindings lists (agent assets are matched by resource name) into
|
||||
// the counts the dashboard shows. Read-only.
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "google/armor/findings.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
struct AssetSummary {
|
||||
std::string resource_name;
|
||||
std::size_t findings = 0;
|
||||
std::string worst_severity;
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
struct SecuritySummary {
|
||||
std::size_t total = 0;
|
||||
std::size_t agent_findings = 0; // Findings carrying agent context.
|
||||
std::size_t anomalies = 0; // Findings carrying agentAnomaly.
|
||||
std::map<std::string, std::size_t> by_severity;
|
||||
std::map<std::string, std::size_t> by_category;
|
||||
std::map<std::string, std::size_t> by_class;
|
||||
std::map<std::string, std::size_t> by_state;
|
||||
std::vector<AssetSummary> assets; // Worst severity first, capped.
|
||||
bool truncated = false;
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
class AgentSecurityDashboard {
|
||||
public:
|
||||
explicit AgentSecurityDashboard(SecurityFindings& findings) : findings_(findings) {}
|
||||
|
||||
// resource_contains defaults to "reasoningEngines" (Agent Runtime) when empty.
|
||||
SecuritySummary summarize(FindingQuery query);
|
||||
// Asset discovery: distinct resources that carry findings.
|
||||
std::vector<AssetSummary> discovered_assets(FindingQuery query);
|
||||
// Findings of class VULNERABILITY for agent assets.
|
||||
FindingPage vulnerabilities(FindingQuery query);
|
||||
|
||||
static SecuritySummary summarize_findings(const FindingPage& page);
|
||||
|
||||
private:
|
||||
static FindingQuery with_agent_scope(FindingQuery query);
|
||||
SecurityFindings& findings_;
|
||||
};
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,337 @@
|
||||
#include "google/armor/findings.h"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
#include "google/armor/armor_util.h"
|
||||
#include "google/armor/caps.h"
|
||||
#include "google/errors.h"
|
||||
#include "google/resource.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
const Json* member(const Json& doc, const char* key) {
|
||||
return doc.is_object() ? doc.find(key) : nullptr;
|
||||
}
|
||||
|
||||
std::string text_of(const Json& doc, const char* key) {
|
||||
const Json* v = member(doc, key);
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
|
||||
// Category names, resource fragments: letters, digits and a few separators.
|
||||
bool safe_value(const std::string& v) {
|
||||
if (v.empty() || v.size() > ArmorCaps::kMaxFilterValueChars) return false;
|
||||
for (char c : v)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') ||
|
||||
(c >= '0' && c <= '9') || c == '_' || c == '-' || c == '.' ||
|
||||
c == '/' || c == ':'))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Comma separated field paths: letters, digits, '_' and '.'.
|
||||
bool safe_path_list(const std::string& v) {
|
||||
if (v.empty() || v.size() > ArmorCaps::kMaxFilterValueChars) return false;
|
||||
for (char c : v)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') ||
|
||||
(c >= '0' && c <= '9') || c == '_' || c == '.' || c == ','))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool one_of(const std::string& v, std::initializer_list<const char*> allowed) {
|
||||
return std::any_of(allowed.begin(), allowed.end(),
|
||||
[&](const char* a) { return v == a; });
|
||||
}
|
||||
|
||||
void validate_order_by(const std::string& order_by) {
|
||||
std::size_t start = 0;
|
||||
while (start <= order_by.size()) {
|
||||
std::size_t comma = order_by.find(',', start);
|
||||
std::string part = order_by.substr(
|
||||
start, comma == std::string::npos ? std::string::npos : comma - start);
|
||||
std::size_t space = part.find(' ');
|
||||
std::string field = part.substr(0, space);
|
||||
std::string suffix = space == std::string::npos ? "" : part.substr(space + 1);
|
||||
// Fields the API documents for orderBy (securitycenter v2 findings.list).
|
||||
if (!one_of(field, {"name", "parent", "state", "category", "resourceName",
|
||||
"eventTime", "createTime", "severity", "findingClass"}) ||
|
||||
(!suffix.empty() && suffix != "desc"))
|
||||
throw GoogleError::invalid("bad order_by");
|
||||
if (comma == std::string::npos) break;
|
||||
start = comma + 1;
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<std::string> strings_of(const Json& doc, const char* key, const char* field) {
|
||||
std::vector<std::string> out;
|
||||
const Json* arr = member(doc, key);
|
||||
if (!arr || !arr->is_array()) return out;
|
||||
for (const Json& x : arr->as_array())
|
||||
if (out.size() < ArmorCaps::kMaxDetailsPerFilter) out.push_back(text_of(x, field));
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Json Finding::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("name", name);
|
||||
o.set("category", category);
|
||||
o.set("state", state);
|
||||
o.set("severity", severity);
|
||||
o.set("class", finding_class);
|
||||
o.set("mute", mute);
|
||||
o.set("resource", resource_name);
|
||||
o.set("event_time", event_time);
|
||||
o.set("description", description);
|
||||
o.set("external_uri", external_uri);
|
||||
if (is_agent_finding()) {
|
||||
Json a = Json::object();
|
||||
a.set("id", agent_id);
|
||||
a.set("display_name", agent_display_name);
|
||||
a.set("anomaly", has_anomaly);
|
||||
if (has_anomaly) {
|
||||
a.set("confidence", anomaly_confidence);
|
||||
Json d = Json::array();
|
||||
for (const auto& x : anomaly_detectors) {
|
||||
Json item = Json::object();
|
||||
item.set("detector_id", x.detector_id);
|
||||
item.set("display_name", x.display_name);
|
||||
item.set("severity", x.severity);
|
||||
item.set("explanation", x.explanation);
|
||||
item.set("recommendation", x.recommendation);
|
||||
d.push(item);
|
||||
}
|
||||
a.set("detectors", d);
|
||||
}
|
||||
o.set("agent", a);
|
||||
}
|
||||
if (!cve_id.empty()) o.set("cve", cve_id);
|
||||
return o;
|
||||
}
|
||||
|
||||
Finding Finding::from_json(const Json& doc) {
|
||||
Finding f;
|
||||
f.name = text_of(doc, "name");
|
||||
f.parent = text_of(doc, "parent");
|
||||
f.category = text_of(doc, "category");
|
||||
f.state = text_of(doc, "state");
|
||||
f.severity = text_of(doc, "severity");
|
||||
f.finding_class = text_of(doc, "findingClass");
|
||||
f.mute = text_of(doc, "mute");
|
||||
f.resource_name = text_of(doc, "resourceName");
|
||||
f.event_time = text_of(doc, "eventTime");
|
||||
f.create_time = text_of(doc, "createTime");
|
||||
f.description = text_of(doc, "description");
|
||||
f.external_uri = text_of(doc, "externalUri");
|
||||
f.module_name = text_of(doc, "moduleName");
|
||||
if (const Json* a = member(doc, "agent")) {
|
||||
f.agent_id = text_of(*a, "id");
|
||||
f.agent_display_name = text_of(*a, "displayName");
|
||||
}
|
||||
if (const Json* an = member(doc, "agentAnomaly")) {
|
||||
f.has_anomaly = true;
|
||||
if (const Json* c = member(*an, "confidenceScore"); c && c->is_number())
|
||||
f.anomaly_confidence = c->as_number();
|
||||
f.anomaly_invocation_ids = strings_of(*an, "invocationReferences", "invocationId");
|
||||
if (const Json* det = member(*an, "detectorReferences"); det && det->is_array())
|
||||
for (const Json& d : det->as_array()) {
|
||||
if (f.anomaly_detectors.size() >= ArmorCaps::kMaxDetailsPerFilter) break;
|
||||
f.anomaly_detectors.push_back({text_of(d, "detectorId"), text_of(d, "displayName"),
|
||||
text_of(d, "severity"), text_of(d, "explanation"),
|
||||
text_of(d, "recommendation")});
|
||||
}
|
||||
}
|
||||
f.agent_session_ids = strings_of(doc, "agentSessions", "sessionId");
|
||||
if (const Json* v = member(doc, "vulnerability")) {
|
||||
f.vulnerability = *v;
|
||||
if (const Json* cve = member(*v, "cve")) f.cve_id = text_of(*cve, "id");
|
||||
}
|
||||
if (const Json* m = member(doc, "aiModel")) f.ai_model = *m;
|
||||
if (const Json* e = member(doc, "agentDataAccessEvents")) f.agent_data_access_events = *e;
|
||||
return f;
|
||||
}
|
||||
|
||||
void FindingQuery::validate() const {
|
||||
if (!valid_scope(scope)) throw GoogleError::invalid("bad findings scope");
|
||||
if (source != "-" && !valid_segment(source))
|
||||
throw GoogleError::invalid("bad source id");
|
||||
if (!location.empty() && !valid_segment(location))
|
||||
throw GoogleError::invalid("bad location");
|
||||
if (categories.size() > ArmorCaps::kMaxFilterCategories)
|
||||
throw GoogleError::invalid("too many categories");
|
||||
for (const auto& c : categories)
|
||||
if (!safe_value(c)) throw GoogleError::invalid("bad category");
|
||||
if (!state.empty() && !one_of(state, {"ACTIVE", "INACTIVE"}))
|
||||
throw GoogleError::invalid("state must be ACTIVE or INACTIVE");
|
||||
if (!severity.empty() && !one_of(severity, {"CRITICAL", "HIGH", "MEDIUM", "LOW"}))
|
||||
throw GoogleError::invalid("severity must be CRITICAL, HIGH, MEDIUM or LOW");
|
||||
if (!resource_contains.empty() && !safe_value(resource_contains))
|
||||
throw GoogleError::invalid("bad resource filter");
|
||||
if (!finding_class.empty() && !safe_value(finding_class))
|
||||
throw GoogleError::invalid("bad finding class");
|
||||
if (!order_by.empty()) validate_order_by(order_by);
|
||||
if (!field_mask.empty() && !safe_path_list(field_mask))
|
||||
throw GoogleError::invalid("bad field mask");
|
||||
if (page_size < 0 || page_size > ArmorCaps::kMaxFindingsPageSize)
|
||||
throw GoogleError::invalid("page_size must be 0 to 1000");
|
||||
if (max_pages == 0 || max_pages > ArmorCaps::kMaxFindingPages)
|
||||
throw GoogleError::invalid("max_pages must be 1 to 10");
|
||||
if (max_results > ArmorCaps::kMaxFindings)
|
||||
throw GoogleError::invalid("max_results must be at most 5000");
|
||||
}
|
||||
|
||||
std::string FindingQuery::parent_path() const {
|
||||
validate();
|
||||
std::string path = "/" + scope + "/sources/" + source;
|
||||
if (!location.empty()) path += "/locations/" + location;
|
||||
return path;
|
||||
}
|
||||
|
||||
std::string FindingQuery::filter() const {
|
||||
validate();
|
||||
std::vector<std::string> terms;
|
||||
if (!categories.empty()) {
|
||||
std::string group = "(";
|
||||
for (std::size_t i = 0; i < categories.size(); ++i) {
|
||||
if (i) group += " OR ";
|
||||
group += "category=\"" + categories[i] + "\"";
|
||||
}
|
||||
terms.push_back(group + ")");
|
||||
}
|
||||
if (!state.empty()) terms.push_back("state=\"" + state + "\"");
|
||||
if (!severity.empty()) terms.push_back("severity=\"" + severity + "\"");
|
||||
if (!resource_contains.empty())
|
||||
terms.push_back("resourceName:\"" + resource_contains + "\"");
|
||||
std::string out;
|
||||
for (const auto& t : terms) out += (out.empty() ? "" : " AND ") + t;
|
||||
return out;
|
||||
}
|
||||
|
||||
Json SecuritySource::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("name", name);
|
||||
o.set("display_name", display_name);
|
||||
o.set("description", description);
|
||||
o.set("canonical_name", canonical_name);
|
||||
return o;
|
||||
}
|
||||
|
||||
ServiceEndpoint SecurityFindings::endpoint() {
|
||||
return endpoints::custom("securitycenter", "https://securitycenter.googleapis.com/v2");
|
||||
}
|
||||
|
||||
SecurityFindings::SecurityFindings(std::shared_ptr<GoogleClient> client)
|
||||
: Service(std::move(client), endpoint()) {}
|
||||
|
||||
FindingPage SecurityFindings::list(const FindingQuery& query) {
|
||||
std::string filter = query.filter(); // Validates everything.
|
||||
std::string base = query.parent_path() + "/findings";
|
||||
int size = query.page_size > 0 ? query.page_size : ArmorCaps::kDefaultFindingsPageSize;
|
||||
std::string q = "?pageSize=" + std::to_string(size);
|
||||
if (!filter.empty()) q += "&filter=" + query_escape(filter);
|
||||
if (!query.order_by.empty()) q += "&orderBy=" + query_escape(query.order_by);
|
||||
if (!query.field_mask.empty()) q += "&fieldMask=" + query_escape(query.field_mask);
|
||||
|
||||
std::size_t cap = query.max_results ? query.max_results : ArmorCaps::kMaxFindings;
|
||||
FindingPage page;
|
||||
std::string token;
|
||||
for (std::size_t pages = 0; pages < query.max_pages; ++pages) {
|
||||
std::string path = base + q;
|
||||
if (!token.empty()) path += "&pageToken=" + query_escape(token);
|
||||
Json doc = fetch(path);
|
||||
if (pages == 0)
|
||||
if (const Json* t = doc.find("totalSize"); t && t->is_number())
|
||||
page.total_size = t->as_int();
|
||||
if (const Json* items = doc.find("listFindingsResults"); items && items->is_array())
|
||||
for (const Json& item : items->as_array()) {
|
||||
const Json* f = member(item, "finding");
|
||||
if (!f) continue;
|
||||
Finding finding = Finding::from_json(*f);
|
||||
if (!query.finding_class.empty() && finding.finding_class != query.finding_class)
|
||||
continue;
|
||||
if (page.findings.size() >= cap) {
|
||||
page.truncated = true;
|
||||
return page;
|
||||
}
|
||||
page.findings.push_back(std::move(finding));
|
||||
}
|
||||
const Json* next = doc.find("nextPageToken");
|
||||
if (!next || !next->is_string() || next->as_string().empty()) return page;
|
||||
token = next->as_string();
|
||||
}
|
||||
page.truncated = true; // A next page existed beyond max_pages.
|
||||
return page;
|
||||
}
|
||||
|
||||
std::vector<FindingGroup> SecurityFindings::group(const FindingQuery& query,
|
||||
const std::string& group_by) {
|
||||
std::string filter = query.filter();
|
||||
std::string path = query.parent_path() + "/findings:group";
|
||||
if (!safe_path_list(group_by)) throw GoogleError::invalid("bad group_by");
|
||||
Json body = Json::object();
|
||||
body.set("groupBy", group_by);
|
||||
if (!filter.empty()) body.set("filter", filter);
|
||||
int size = query.page_size > 0 ? query.page_size : ArmorCaps::kDefaultFindingsPageSize;
|
||||
body.set("pageSize", size);
|
||||
std::vector<FindingGroup> out;
|
||||
std::string token;
|
||||
for (std::size_t pages = 0; pages < query.max_pages; ++pages) {
|
||||
if (!token.empty()) body.set("pageToken", token);
|
||||
Json doc = call(path, body);
|
||||
if (const Json* rows = doc.find("groupByResults"); rows && rows->is_array())
|
||||
for (const Json& r : rows->as_array()) {
|
||||
if (out.size() >= ArmorCaps::kMaxGroupResults) return out;
|
||||
FindingGroup g;
|
||||
// count is an int64 string on the wire.
|
||||
const Json* c = member(r, "count");
|
||||
if (c && c->is_string()) g.count = std::strtol(c->as_string().c_str(), nullptr, 10);
|
||||
else if (c && c->is_number()) g.count = c->as_int();
|
||||
if (const Json* p = member(r, "properties")) g.properties = *p;
|
||||
out.push_back(std::move(g));
|
||||
}
|
||||
const Json* next = doc.find("nextPageToken");
|
||||
if (!next || !next->is_string() || next->as_string().empty()) break;
|
||||
token = next->as_string();
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<SecuritySource> SecurityFindings::list_sources(const std::string& scope) {
|
||||
if (!valid_scope(scope)) throw GoogleError::invalid("bad sources scope");
|
||||
std::vector<SecuritySource> out;
|
||||
std::string token;
|
||||
for (int pages = 0; pages < 5; ++pages) {
|
||||
std::string path = "/" + scope + "/sources?pageSize=100";
|
||||
if (!token.empty()) path += "&pageToken=" + query_escape(token);
|
||||
Json doc = fetch(path);
|
||||
if (const Json* items = doc.find("sources"); items && items->is_array())
|
||||
for (const Json& s : items->as_array()) {
|
||||
if (out.size() >= ArmorCaps::kMaxSources) return out;
|
||||
out.push_back({text_of(s, "name"), text_of(s, "displayName"),
|
||||
text_of(s, "description"), text_of(s, "canonicalName")});
|
||||
}
|
||||
const Json* next = doc.find("nextPageToken");
|
||||
if (!next || !next->is_string() || next->as_string().empty()) break;
|
||||
token = next->as_string();
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
SecuritySource SecurityFindings::get_source(const std::string& name) {
|
||||
// organizations/{id}/sources/{id} (the only scope with a get method).
|
||||
std::size_t slash = name.find("/sources/");
|
||||
if (slash == std::string::npos || name.rfind("organizations/", 0) != 0 ||
|
||||
!valid_scope(name.substr(0, slash)) || !valid_segment(name.substr(slash + 9)))
|
||||
throw GoogleError::invalid("source name must be organizations/{id}/sources/{id}");
|
||||
Json s = fetch("/" + name);
|
||||
return {text_of(s, "name"), text_of(s, "displayName"), text_of(s, "description"),
|
||||
text_of(s, "canonicalName")};
|
||||
}
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,123 @@
|
||||
#pragma once
|
||||
// Read-only Security Command Center findings (API v2). Agent Threat
|
||||
// Detection and Agent Anomaly Detection have no API of their own: they
|
||||
// publish findings to Security Command Center, whose Finding resource has
|
||||
// typed `agent`, `agentAnomaly`, `agentSessions`, `agentDataAccessEvents`,
|
||||
// `aiModel` and `vulnerability` members (Discovery, securitycenter v2).
|
||||
// Host: https://securitycenter.googleapis.com/v2. Auth: OAuth2 access token.
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "google/service.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
// agentAnomaly.detectorReferences[]
|
||||
struct AnomalyDetector {
|
||||
std::string detector_id;
|
||||
std::string display_name;
|
||||
std::string severity;
|
||||
std::string explanation;
|
||||
std::string recommendation;
|
||||
};
|
||||
|
||||
struct Finding {
|
||||
std::string name; // organizations/1/sources/2/locations/global/findings/3
|
||||
std::string parent;
|
||||
std::string category;
|
||||
std::string state; // ACTIVE | INACTIVE
|
||||
std::string severity; // CRITICAL | HIGH | MEDIUM | LOW
|
||||
std::string finding_class; // THREAT | VULNERABILITY | MISCONFIGURATION | ...
|
||||
std::string mute; // MUTED | UNMUTED | UNDEFINED
|
||||
std::string resource_name;
|
||||
std::string event_time;
|
||||
std::string create_time;
|
||||
std::string description;
|
||||
std::string external_uri;
|
||||
std::string module_name;
|
||||
// Agent context (empty when the finding is not about an agent).
|
||||
std::string agent_id;
|
||||
std::string agent_display_name;
|
||||
bool has_anomaly = false;
|
||||
double anomaly_confidence = 0.0;
|
||||
std::vector<std::string> anomaly_invocation_ids;
|
||||
std::vector<AnomalyDetector> anomaly_detectors;
|
||||
std::vector<std::string> agent_session_ids;
|
||||
// Vulnerability context.
|
||||
std::string cve_id;
|
||||
// Kept raw: shapes the dashboard does not need typed.
|
||||
clients::body::Json ai_model;
|
||||
clients::body::Json agent_data_access_events;
|
||||
clients::body::Json vulnerability;
|
||||
|
||||
bool is_agent_finding() const { return !agent_id.empty() || has_anomaly; }
|
||||
clients::body::Json to_json() const;
|
||||
static Finding from_json(const clients::body::Json& doc); // A Finding object.
|
||||
};
|
||||
|
||||
struct FindingQuery {
|
||||
// "organizations/{id}", "folders/{id}" or "projects/{id}".
|
||||
std::string scope;
|
||||
std::string source = "-"; // "-": every source.
|
||||
std::string location; // Empty: no /locations/ segment.
|
||||
std::vector<std::string> categories; // ORed.
|
||||
std::string state; // "", ACTIVE, INACTIVE
|
||||
std::string severity; // "", CRITICAL, HIGH, MEDIUM, LOW
|
||||
// Substring of resourceName, e.g. "reasoningEngines" for agent assets.
|
||||
std::string resource_contains;
|
||||
// Applied locally after listing (findingClass is orderBy-only in the API).
|
||||
std::string finding_class;
|
||||
std::string order_by; // "", "eventTime desc", "severity", ...
|
||||
std::string field_mask; // "", or comma separated Finding field paths
|
||||
int page_size = 0; // 0: ArmorCaps::kDefaultFindingsPageSize
|
||||
std::size_t max_pages = 1;
|
||||
std::size_t max_results = 0; // 0: ArmorCaps::kMaxFindings
|
||||
|
||||
// The AIP-160 filter built from the typed fields; "" when none are set.
|
||||
// Throws GoogleError::invalid for any malformed value.
|
||||
std::string filter() const;
|
||||
void validate() const;
|
||||
// "/{scope}/sources/{source}[/locations/{l}]" (validated).
|
||||
std::string parent_path() const;
|
||||
};
|
||||
|
||||
struct FindingPage {
|
||||
std::vector<Finding> findings;
|
||||
long total_size = -1; // totalSize of the first page; -1 when absent.
|
||||
bool truncated = false; // More existed than max_results or max_pages.
|
||||
};
|
||||
|
||||
struct SecuritySource {
|
||||
std::string name; // organizations/1/sources/2
|
||||
std::string display_name;
|
||||
std::string description;
|
||||
std::string canonical_name;
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
// findings:group result row.
|
||||
struct FindingGroup {
|
||||
long count = 0;
|
||||
clients::body::Json properties;
|
||||
};
|
||||
|
||||
class SecurityFindings : public Service {
|
||||
public:
|
||||
explicit SecurityFindings(std::shared_ptr<GoogleClient> client);
|
||||
|
||||
static ServiceEndpoint endpoint();
|
||||
|
||||
FindingPage list(const FindingQuery& query);
|
||||
// Server side counts (findings:group). group_by: comma separated fields,
|
||||
// e.g. "severity" or "category,state".
|
||||
std::vector<FindingGroup> group(const FindingQuery& query, const std::string& group_by);
|
||||
// Finding sources of a scope (Agent Threat Detection and friends).
|
||||
std::vector<SecuritySource> list_sources(const std::string& scope);
|
||||
// organizations/{id}/sources/{id}
|
||||
SecuritySource get_source(const std::string& name);
|
||||
};
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,155 @@
|
||||
#include "google/armor/guarded_model.h"
|
||||
|
||||
#include "google/armor/caps.h"
|
||||
#include "google/errors.h"
|
||||
#include "google/resource.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
namespace {
|
||||
|
||||
std::string describe(GuardStage stage, const SanitizationResult& r, bool incomplete) {
|
||||
std::string out = std::string("model armor blocked the ") +
|
||||
(stage == GuardStage::kPrompt ? "prompt" : "response");
|
||||
if (incomplete) return out + ": filters did not all run";
|
||||
out += ":";
|
||||
for (const auto& f : r.matches()) out += std::string(" ") + filter_kind_name(f.kind);
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
BlockedError::BlockedError(GuardStage stage, SanitizationResult result, bool incomplete)
|
||||
: std::runtime_error(describe(stage, result, incomplete)),
|
||||
stage_(stage),
|
||||
result_(std::move(result)),
|
||||
incomplete_(incomplete) {}
|
||||
|
||||
std::vector<FilterKind> BlockedError::filters() const {
|
||||
std::vector<FilterKind> out;
|
||||
for (const auto& f : result_.matches()) out.push_back(f.kind);
|
||||
return out;
|
||||
}
|
||||
|
||||
ModelArmorSanitizer::ModelArmorSanitizer(std::shared_ptr<ModelArmor> armor,
|
||||
std::string template_id)
|
||||
: armor_(std::move(armor)), template_id_(std::move(template_id)) {
|
||||
if (!armor_) throw GoogleError::invalid("sanitizer needs model armor");
|
||||
if (!valid_segment(template_id_))
|
||||
throw GoogleError::invalid("guard template id is missing or malformed");
|
||||
}
|
||||
|
||||
SanitizationResult ModelArmorSanitizer::sanitize_prompt(const std::string& text) {
|
||||
return armor_->sanitize_user_prompt(template_id_, text);
|
||||
}
|
||||
|
||||
SanitizationResult ModelArmorSanitizer::sanitize_response(const std::string& text,
|
||||
const std::string& prompt) {
|
||||
return armor_->sanitize_model_response(template_id_, text, prompt);
|
||||
}
|
||||
|
||||
GuardedModel::GuardedModel(std::shared_ptr<LanguageModel> inner,
|
||||
std::shared_ptr<Sanitizer> sanitizer, GuardOptions options)
|
||||
: inner_(std::move(inner)), sanitizer_(std::move(sanitizer)), options_(std::move(options)) {
|
||||
if (!inner_ || !sanitizer_)
|
||||
throw GoogleError::invalid("guarded model needs a model and a sanitizer");
|
||||
}
|
||||
|
||||
std::shared_ptr<GuardedModel> GuardedModel::with_armor(std::shared_ptr<LanguageModel> inner,
|
||||
std::shared_ptr<ModelArmor> armor,
|
||||
const std::string& template_id,
|
||||
GuardOptions options) {
|
||||
return std::make_shared<GuardedModel>(
|
||||
std::move(inner), std::make_shared<ModelArmorSanitizer>(std::move(armor), template_id),
|
||||
std::move(options));
|
||||
}
|
||||
|
||||
std::string GuardedModel::prompt_text(const GenerateRequest& request) {
|
||||
std::string out;
|
||||
auto add = [&](const Content& c) {
|
||||
std::string t = c.text();
|
||||
if (t.empty()) return;
|
||||
if (!out.empty()) out += "\n";
|
||||
out += t;
|
||||
};
|
||||
if (request.system) add(*request.system);
|
||||
for (const auto& c : request.contents)
|
||||
if (c.role != "model") add(c);
|
||||
return out;
|
||||
}
|
||||
|
||||
void GuardedModel::judge(GuardStage stage, const SanitizationResult& result) {
|
||||
auto& blocked = stage == GuardStage::kPrompt ? prompts_blocked_ : responses_blocked_;
|
||||
if (result.matched()) {
|
||||
++blocked;
|
||||
throw BlockedError(stage, result, false);
|
||||
}
|
||||
if (!options_.allow_partial && !result.fully_executed()) {
|
||||
++blocked;
|
||||
throw BlockedError(stage, result, true);
|
||||
}
|
||||
}
|
||||
|
||||
void GuardedModel::check_prompt(const std::string& text) {
|
||||
if (!options_.check_prompt) return;
|
||||
if (text.empty()) {
|
||||
++skipped_empty_;
|
||||
return;
|
||||
}
|
||||
++prompts_checked_;
|
||||
judge(GuardStage::kPrompt, sanitizer_->sanitize_prompt(text));
|
||||
}
|
||||
|
||||
void GuardedModel::check_response(const std::string& text, const std::string& prompt) {
|
||||
if (!options_.check_response) return;
|
||||
if (text.empty()) {
|
||||
++skipped_empty_;
|
||||
return;
|
||||
}
|
||||
++responses_checked_;
|
||||
// The prompt is context only; an oversized one is dropped, not an error.
|
||||
const std::string& context = prompt.size() <= ArmorCaps::kMaxTextChars ? prompt : std::string();
|
||||
judge(GuardStage::kResponse, sanitizer_->sanitize_response(text, context));
|
||||
}
|
||||
|
||||
GenerateResponse GuardedModel::generate(const GenerateRequest& request) {
|
||||
std::string prompt = prompt_text(request);
|
||||
check_prompt(prompt);
|
||||
GenerateResponse response = inner_->generate(request);
|
||||
std::string text;
|
||||
for (const auto& c : response.candidates) {
|
||||
std::string t = c.content.text();
|
||||
if (t.empty()) continue;
|
||||
if (!text.empty()) text += "\n";
|
||||
text += t;
|
||||
}
|
||||
check_response(text, prompt);
|
||||
return response;
|
||||
}
|
||||
|
||||
void GuardedModel::stream(const GenerateRequest& request,
|
||||
const std::function<bool(const GenerateResponse&)>& on_chunk) {
|
||||
std::string prompt = prompt_text(request);
|
||||
check_prompt(prompt);
|
||||
std::vector<GenerateResponse> chunks;
|
||||
std::string text;
|
||||
inner_->stream(request, [&](const GenerateResponse& chunk) {
|
||||
if (chunks.size() >= ArmorCaps::kMaxStreamChunks)
|
||||
throw GoogleError::invalid("guarded stream exceeds the chunk buffer");
|
||||
text += chunk.text();
|
||||
if (text.size() > ArmorCaps::kMaxStreamBufferChars)
|
||||
throw GoogleError::invalid("guarded stream exceeds the text buffer");
|
||||
chunks.push_back(chunk);
|
||||
return true;
|
||||
});
|
||||
check_response(text, prompt);
|
||||
for (const auto& chunk : chunks)
|
||||
if (!on_chunk(chunk)) break;
|
||||
}
|
||||
|
||||
GuardStats GuardedModel::stats() const {
|
||||
return {prompts_checked_.load(), prompts_blocked_.load(), responses_checked_.load(),
|
||||
responses_blocked_.load(), skipped_empty_.load()};
|
||||
}
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,109 @@
|
||||
#pragma once
|
||||
// GuardedModel: a LanguageModel decorator. Sanitizes the prompt through
|
||||
// Model Armor before the wrapped model runs and the response after, and
|
||||
// throws BlockedError (carrying the matched filters) instead of returning
|
||||
// blocked content. Fail closed: an incomplete Model Armor run also blocks
|
||||
// unless GuardOptions::allow_partial is set.
|
||||
//
|
||||
// Streaming: chunks are buffered until the stream ends, the whole text is
|
||||
// sanitized, then the chunks are replayed. The caller sees nothing of a
|
||||
// blocked response. Cost: no incremental output while guarded.
|
||||
#include <atomic>
|
||||
#include <memory>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/armor/model_armor.h"
|
||||
#include "google/gemini_types.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
enum class GuardStage { kPrompt, kResponse };
|
||||
|
||||
class BlockedError : public std::runtime_error {
|
||||
public:
|
||||
BlockedError(GuardStage stage, SanitizationResult result, bool incomplete);
|
||||
GuardStage stage() const noexcept { return stage_; }
|
||||
const SanitizationResult& result() const noexcept { return result_; }
|
||||
// True: blocked because filters did not all run, not because one matched.
|
||||
bool incomplete() const noexcept { return incomplete_; }
|
||||
std::vector<FilterKind> filters() const; // Matched filter kinds.
|
||||
|
||||
private:
|
||||
GuardStage stage_;
|
||||
SanitizationResult result_;
|
||||
bool incomplete_;
|
||||
};
|
||||
|
||||
// The sanitizing seam. ModelArmorSanitizer is the production implementation;
|
||||
// tests use a local fake.
|
||||
class Sanitizer {
|
||||
public:
|
||||
virtual ~Sanitizer() = default;
|
||||
virtual SanitizationResult sanitize_prompt(const std::string& text) = 0;
|
||||
virtual SanitizationResult sanitize_response(const std::string& text,
|
||||
const std::string& prompt) = 0;
|
||||
};
|
||||
|
||||
// Sanitizes through one Model Armor template.
|
||||
class ModelArmorSanitizer : public Sanitizer {
|
||||
public:
|
||||
ModelArmorSanitizer(std::shared_ptr<ModelArmor> armor, std::string template_id);
|
||||
SanitizationResult sanitize_prompt(const std::string& text) override;
|
||||
SanitizationResult sanitize_response(const std::string& text,
|
||||
const std::string& prompt) override;
|
||||
|
||||
private:
|
||||
std::shared_ptr<ModelArmor> armor_;
|
||||
std::string template_id_;
|
||||
};
|
||||
|
||||
struct GuardOptions {
|
||||
bool check_prompt = true;
|
||||
bool check_response = true;
|
||||
// Accept PARTIAL/skipped runs when no filter matched.
|
||||
bool allow_partial = false;
|
||||
};
|
||||
|
||||
struct GuardStats {
|
||||
std::size_t prompts_checked = 0;
|
||||
std::size_t prompts_blocked = 0;
|
||||
std::size_t responses_checked = 0;
|
||||
std::size_t responses_blocked = 0;
|
||||
std::size_t skipped_empty = 0; // Nothing textual to sanitize.
|
||||
};
|
||||
|
||||
class GuardedModel : public LanguageModel {
|
||||
public:
|
||||
GuardedModel(std::shared_ptr<LanguageModel> inner, std::shared_ptr<Sanitizer> sanitizer,
|
||||
GuardOptions options = {});
|
||||
// Convenience: guard through a Model Armor template.
|
||||
static std::shared_ptr<GuardedModel> with_armor(std::shared_ptr<LanguageModel> inner,
|
||||
std::shared_ptr<ModelArmor> armor,
|
||||
const std::string& template_id,
|
||||
GuardOptions options = {});
|
||||
|
||||
GenerateResponse generate(const GenerateRequest& request) override;
|
||||
void stream(const GenerateRequest& request,
|
||||
const std::function<bool(const GenerateResponse&)>& on_chunk) override;
|
||||
|
||||
GuardStats stats() const;
|
||||
|
||||
// Text sent for prompt sanitization: system plus user turns.
|
||||
static std::string prompt_text(const GenerateRequest& request);
|
||||
|
||||
private:
|
||||
void check_prompt(const std::string& text);
|
||||
void check_response(const std::string& text, const std::string& prompt);
|
||||
void judge(GuardStage stage, const SanitizationResult& result);
|
||||
|
||||
std::shared_ptr<LanguageModel> inner_;
|
||||
std::shared_ptr<Sanitizer> sanitizer_;
|
||||
GuardOptions options_;
|
||||
std::atomic<std::size_t> prompts_checked_{0}, prompts_blocked_{0};
|
||||
std::atomic<std::size_t> responses_checked_{0}, responses_blocked_{0};
|
||||
std::atomic<std::size_t> skipped_empty_{0};
|
||||
};
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,283 @@
|
||||
#include "google/armor/model_armor.h"
|
||||
|
||||
#include "google/armor/armor_util.h"
|
||||
#include "google/armor/caps.h"
|
||||
#include "google/errors.h"
|
||||
#include "google/operation.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
bool valid_byte_type(const std::string& t) {
|
||||
return t == "PLAINTEXT_UTF8" || t == "PDF" || t == "WORD_DOCUMENT" ||
|
||||
t == "EXCEL_DOCUMENT" || t == "POWERPOINT_DOCUMENT" || t == "TXT" ||
|
||||
t == "CSV" || t == "IMAGE";
|
||||
}
|
||||
|
||||
bool valid_filter_text(const std::string& v) {
|
||||
if (v.size() > ArmorCaps::kMaxFilterValueChars) return false;
|
||||
for (unsigned char c : v)
|
||||
if (c < 0x20 || c == 0x7f) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
void validate_hint(const LanguageHint& hint) {
|
||||
if (hint.source_language.size() > 35)
|
||||
throw GoogleError::invalid("source language too long");
|
||||
for (char c : hint.source_language)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '-'))
|
||||
throw GoogleError::invalid("bad source language");
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
DataItem DataItem::of_text(std::string text) {
|
||||
DataItem d;
|
||||
d.text = std::move(text);
|
||||
return d;
|
||||
}
|
||||
|
||||
DataItem DataItem::of_bytes(std::string byte_type, std::string bytes,
|
||||
std::string file_label) {
|
||||
DataItem d;
|
||||
d.byte_type = std::move(byte_type);
|
||||
d.bytes = std::move(bytes);
|
||||
d.file_label = std::move(file_label);
|
||||
return d;
|
||||
}
|
||||
|
||||
void DataItem::validate(const char* what) const {
|
||||
bool is_bytes = !byte_type.empty() || !bytes.empty();
|
||||
if (is_bytes && !text.empty())
|
||||
throw GoogleError::invalid(std::string(what) + " is text or bytes, not both");
|
||||
if (!is_bytes) {
|
||||
if (text.empty()) throw GoogleError::invalid(std::string(what) + " is empty");
|
||||
if (text.size() > ArmorCaps::kMaxTextChars)
|
||||
throw GoogleError::invalid(std::string(what) + " exceeds 100000 characters");
|
||||
return;
|
||||
}
|
||||
if (!valid_byte_type(byte_type))
|
||||
throw GoogleError::invalid("byte type is not a documented ByteItemType");
|
||||
if (bytes.empty()) throw GoogleError::invalid(std::string(what) + " bytes are empty");
|
||||
if (bytes.size() > ArmorCaps::kMaxByteItemBytes)
|
||||
throw GoogleError::invalid(std::string(what) + " bytes exceed 5 MiB");
|
||||
if (file_label.size() > 128) throw GoogleError::invalid("file label too long");
|
||||
}
|
||||
|
||||
Json DataItem::to_json() const {
|
||||
Json o = Json::object();
|
||||
if (byte_type.empty() && bytes.empty()) {
|
||||
o.set("text", text);
|
||||
return o;
|
||||
}
|
||||
Json b = Json::object();
|
||||
b.set("byteDataType", byte_type);
|
||||
b.set("byteData", security::crypto::Base64::encode(bytes));
|
||||
if (!file_label.empty()) b.set("fileLabel", file_label);
|
||||
o.set("byteItem", b);
|
||||
return o;
|
||||
}
|
||||
|
||||
ServiceEndpoint ModelArmor::endpoint_for(const std::string& location) {
|
||||
if (location == "global")
|
||||
return endpoints::custom("modelarmor", "https://modelarmor.googleapis.com/v1");
|
||||
return endpoints::custom(
|
||||
"modelarmor", "https://modelarmor." + location + ".rep.googleapis.com/v1");
|
||||
}
|
||||
|
||||
ModelArmor::ModelArmor(std::shared_ptr<GoogleClient> client, CloudContext context)
|
||||
: Service(std::move(client),
|
||||
endpoint_for(context.valid() ? context.location : "us-central1")),
|
||||
context_(std::move(context)) {}
|
||||
|
||||
void ModelArmor::require_regional() const {
|
||||
if (context_.location == "global")
|
||||
throw GoogleError::invalid("templates need a regional location, not global");
|
||||
}
|
||||
|
||||
std::string ModelArmor::template_path(const std::string& id) const {
|
||||
require_regional();
|
||||
return "/" + context_.child("templates", id);
|
||||
}
|
||||
|
||||
ServiceEndpoint ModelArmor::floor_endpoint() const {
|
||||
return endpoint_for(context_.location);
|
||||
}
|
||||
|
||||
std::string ModelArmor::floor_path(const std::string& scope) const {
|
||||
std::string s = scope;
|
||||
context_.parent(); // Validates project and location.
|
||||
if (s.empty()) s = "projects/" + context_.project;
|
||||
if (!valid_scope(s)) throw GoogleError::invalid("bad floor setting scope");
|
||||
return "/" + s + "/locations/" + context_.location + "/floorSetting";
|
||||
}
|
||||
|
||||
std::string ModelArmor::request_id_query(const std::string& request_id, char sep) {
|
||||
if (request_id.empty()) return "";
|
||||
if (!valid_request_id(request_id))
|
||||
throw GoogleError::invalid("request id must be a non-zero UUID");
|
||||
return std::string(1, sep) + "requestId=" + request_id;
|
||||
}
|
||||
|
||||
Template ModelArmor::create_template(const std::string& id, const Template& tmpl,
|
||||
const std::string& request_id) {
|
||||
template_path(id); // Validates the id and the location.
|
||||
std::string rid = request_id_query(request_id, '&');
|
||||
tmpl.validate();
|
||||
return Template::from_json(call(
|
||||
"/" + context_.parent() + "/templates?templateId=" + query_escape(id) + rid,
|
||||
tmpl.to_json()));
|
||||
}
|
||||
|
||||
Template ModelArmor::get_template(const std::string& id) {
|
||||
return Template::from_json(fetch(template_path(id)));
|
||||
}
|
||||
|
||||
TemplatePage ModelArmor::list_templates(int page_size, const std::string& filter,
|
||||
const std::string& order_by) {
|
||||
require_regional();
|
||||
if (page_size < 0) throw GoogleError::invalid("page_size must not be negative");
|
||||
if (page_size > ArmorCaps::kMaxTemplatePageSize)
|
||||
page_size = ArmorCaps::kMaxTemplatePageSize;
|
||||
if (!valid_filter_text(filter) || !valid_filter_text(order_by))
|
||||
throw GoogleError::invalid("bad filter or order_by");
|
||||
std::string base = "/" + context_.parent() + "/templates";
|
||||
std::string query;
|
||||
auto add = [&](const std::string& k, const std::string& v) {
|
||||
query += (query.empty() ? "?" : "&") + k + "=" + query_escape(v);
|
||||
};
|
||||
if (page_size > 0) add("pageSize", std::to_string(page_size));
|
||||
if (!filter.empty()) add("filter", filter);
|
||||
if (!order_by.empty()) add("orderBy", order_by);
|
||||
|
||||
TemplatePage page;
|
||||
std::string token;
|
||||
for (std::size_t pages = 0; pages < ArmorCaps::kMaxTemplatePages; ++pages) {
|
||||
std::string path = base + query;
|
||||
if (!token.empty()) path += (query.empty() ? "?" : "&") + std::string("pageToken=") + query_escape(token);
|
||||
Json doc = fetch(path);
|
||||
if (const Json* t = doc.find("templates"); t && t->is_array())
|
||||
for (const Json& item : t->as_array()) page.templates.push_back(Template::from_json(item));
|
||||
if (const Json* u = doc.find("unreachable"); u && u->is_array())
|
||||
for (const Json& x : u->as_array())
|
||||
if (x.is_string() && page.unreachable.size() < ArmorCaps::kMaxLocations)
|
||||
page.unreachable.push_back(x.as_string());
|
||||
const Json* next = doc.find("nextPageToken");
|
||||
if (!next || !next->is_string() || next->as_string().empty()) break;
|
||||
token = next->as_string();
|
||||
}
|
||||
return page;
|
||||
}
|
||||
|
||||
Template ModelArmor::patch_template(const std::string& id, const Template& tmpl,
|
||||
const std::vector<std::string>& update_mask,
|
||||
const std::string& request_id) {
|
||||
std::string path = template_path(id);
|
||||
std::string mask = join_update_mask(update_mask);
|
||||
std::string rid = request_id_query(request_id, '&');
|
||||
tmpl.validate();
|
||||
return Template::from_json(
|
||||
patch(path + "?updateMask=" + query_escape(mask) + rid, tmpl.to_json()));
|
||||
}
|
||||
|
||||
void ModelArmor::delete_template(const std::string& id, const std::string& request_id) {
|
||||
std::string path = template_path(id);
|
||||
remove(path + request_id_query(request_id, '?'));
|
||||
}
|
||||
|
||||
FloorSetting ModelArmor::get_floor_setting(const std::string& scope) {
|
||||
return FloorSetting::from_json(client_->get_json(floor_endpoint(), floor_path(scope)));
|
||||
}
|
||||
|
||||
FloorSetting ModelArmor::update_floor_setting(
|
||||
const FloorSetting& setting, const std::vector<std::string>& update_mask,
|
||||
const std::string& scope) {
|
||||
std::string path = floor_path(scope);
|
||||
std::string mask = join_update_mask(update_mask);
|
||||
setting.validate();
|
||||
return FloorSetting::from_json(client_->patch_json(
|
||||
floor_endpoint(), path + "?updateMask=" + query_escape(mask), setting.to_json()));
|
||||
}
|
||||
|
||||
std::vector<ArmorLocation> ModelArmor::list_locations() {
|
||||
context_.parent();
|
||||
Json items = list_all(*client_, floor_endpoint(),
|
||||
"/projects/" + context_.project + "/locations", "locations", 2);
|
||||
std::vector<ArmorLocation> out;
|
||||
for (const Json& l : items.as_array()) {
|
||||
if (out.size() >= ArmorCaps::kMaxLocations) break;
|
||||
out.push_back({string_at(l, "name"), string_at(l, "locationId"), string_at(l, "displayName")});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
ArmorLocation ModelArmor::get_location() {
|
||||
Json l = client_->get_json(floor_endpoint(), "/" + context_.parent());
|
||||
return {string_at(l, "name"), string_at(l, "locationId"), string_at(l, "displayName")};
|
||||
}
|
||||
|
||||
Json ModelArmor::user_prompt_body(const DataItem& item, const LanguageHint& hint) {
|
||||
Json body = Json::object();
|
||||
body.set("userPromptData", item.to_json());
|
||||
if (hint.present()) {
|
||||
Json m = Json::object();
|
||||
m.set("enableMultiLanguageDetection", hint.enable_detection);
|
||||
if (!hint.source_language.empty()) m.set("sourceLanguage", hint.source_language);
|
||||
body.set("multiLanguageDetectionMetadata", m);
|
||||
}
|
||||
return body;
|
||||
}
|
||||
|
||||
Json ModelArmor::model_response_body(const DataItem& item, const std::string& user_prompt,
|
||||
const LanguageHint& hint) {
|
||||
Json body = Json::object();
|
||||
body.set("modelResponseData", item.to_json());
|
||||
if (!user_prompt.empty()) body.set("userPrompt", user_prompt);
|
||||
if (hint.present()) {
|
||||
Json m = Json::object();
|
||||
m.set("enableMultiLanguageDetection", hint.enable_detection);
|
||||
if (!hint.source_language.empty()) m.set("sourceLanguage", hint.source_language);
|
||||
body.set("multiLanguageDetectionMetadata", m);
|
||||
}
|
||||
return body;
|
||||
}
|
||||
|
||||
SanitizationResult ModelArmor::sanitize_user_prompt(const std::string& template_id,
|
||||
const std::string& text) {
|
||||
return sanitize_user_prompt(template_id, DataItem::of_text(text));
|
||||
}
|
||||
|
||||
SanitizationResult ModelArmor::sanitize_user_prompt(const std::string& template_id,
|
||||
const DataItem& item,
|
||||
const LanguageHint& hint) {
|
||||
std::string path = template_path(template_id);
|
||||
item.validate("prompt");
|
||||
validate_hint(hint);
|
||||
return SanitizationResult::from_json(
|
||||
call(path + ":sanitizeUserPrompt", user_prompt_body(item, hint)));
|
||||
}
|
||||
|
||||
SanitizationResult ModelArmor::sanitize_model_response(const std::string& template_id,
|
||||
const std::string& text,
|
||||
const std::string& user_prompt) {
|
||||
return sanitize_model_response(template_id, DataItem::of_text(text), user_prompt);
|
||||
}
|
||||
|
||||
SanitizationResult ModelArmor::sanitize_model_response(const std::string& template_id,
|
||||
const DataItem& item,
|
||||
const std::string& user_prompt,
|
||||
const LanguageHint& hint) {
|
||||
std::string path = template_path(template_id);
|
||||
item.validate("response");
|
||||
validate_hint(hint);
|
||||
if (user_prompt.size() > ArmorCaps::kMaxTextChars)
|
||||
throw GoogleError::invalid("user prompt exceeds 100000 characters");
|
||||
return SanitizationResult::from_json(
|
||||
call(path + ":sanitizeModelResponse", model_response_body(item, user_prompt, hint)));
|
||||
}
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,109 @@
|
||||
#pragma once
|
||||
// Model Armor: templates CRUD, floor settings, locations, sanitizeUserPrompt
|
||||
// and sanitizeModelResponse. Regional host
|
||||
// https://modelarmor.{location}.rep.googleapis.com/v1; floor settings also
|
||||
// accept location "global" (https://modelarmor.googleapis.com/v1).
|
||||
// Auth: OAuth2 access token only (GOOGLE_ACCESS_TOKEN); API keys are refused
|
||||
// by this API (verified live, see SPEC.md).
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/armor/armor_types.h"
|
||||
#include "google/armor/template.h"
|
||||
#include "google/resource.h"
|
||||
#include "google/service.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
// DataItem: text, or file bytes with a ByteItemType.
|
||||
struct DataItem {
|
||||
std::string text;
|
||||
std::string bytes; // Raw bytes (base64 on the wire).
|
||||
std::string byte_type; // PLAINTEXT_UTF8, PDF, WORD_DOCUMENT, EXCEL_DOCUMENT,
|
||||
// POWERPOINT_DOCUMENT, TXT, CSV, IMAGE
|
||||
std::string file_label;
|
||||
|
||||
static DataItem of_text(std::string text);
|
||||
static DataItem of_bytes(std::string byte_type, std::string bytes,
|
||||
std::string file_label = "");
|
||||
void validate(const char* what) const;
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
// multiLanguageDetectionMetadata on both sanitize requests.
|
||||
struct LanguageHint {
|
||||
bool enable_detection = false;
|
||||
std::string source_language; // BCP-47-ish, optional
|
||||
bool present() const { return enable_detection || !source_language.empty(); }
|
||||
};
|
||||
|
||||
class ModelArmor : public Service {
|
||||
public:
|
||||
ModelArmor(std::shared_ptr<GoogleClient> client, CloudContext context);
|
||||
|
||||
static ServiceEndpoint endpoint_for(const std::string& location);
|
||||
|
||||
const CloudContext& context() const { return context_; }
|
||||
|
||||
// Templates (location must be regional, not "global"). request_id is an
|
||||
// optional retry id (UUID, not all zero); empty: none.
|
||||
Template create_template(const std::string& id, const Template& tmpl,
|
||||
const std::string& request_id = "");
|
||||
Template get_template(const std::string& id);
|
||||
// page_size 0: server default; capped at ArmorCaps::kMaxTemplatePageSize.
|
||||
// filter and order_by are passed through after a character check.
|
||||
TemplatePage list_templates(int page_size = 0, const std::string& filter = "",
|
||||
const std::string& order_by = "");
|
||||
// update_mask: Template field paths, e.g. {"filterConfig"}.
|
||||
Template patch_template(const std::string& id, const Template& tmpl,
|
||||
const std::vector<std::string>& update_mask,
|
||||
const std::string& request_id = "");
|
||||
// Explicit and destructive; there is no default or bulk form.
|
||||
void delete_template(const std::string& id, const std::string& request_id = "");
|
||||
|
||||
// Floor settings. scope is "projects/{id}", "folders/{id}" or
|
||||
// "organizations/{id}"; empty: the context's project.
|
||||
FloorSetting get_floor_setting(const std::string& scope = "");
|
||||
FloorSetting update_floor_setting(const FloorSetting& setting,
|
||||
const std::vector<std::string>& update_mask,
|
||||
const std::string& scope = "");
|
||||
|
||||
// projects.locations.list / get on the Model Armor host.
|
||||
std::vector<ArmorLocation> list_locations();
|
||||
ArmorLocation get_location();
|
||||
|
||||
// Sanitization. Text is 1..ArmorCaps::kMaxTextChars characters.
|
||||
SanitizationResult sanitize_user_prompt(const std::string& template_id,
|
||||
const std::string& text);
|
||||
SanitizationResult sanitize_user_prompt(const std::string& template_id,
|
||||
const DataItem& item,
|
||||
const LanguageHint& hint = {});
|
||||
SanitizationResult sanitize_model_response(const std::string& template_id,
|
||||
const std::string& text,
|
||||
const std::string& user_prompt = "");
|
||||
SanitizationResult sanitize_model_response(const std::string& template_id,
|
||||
const DataItem& item,
|
||||
const std::string& user_prompt = "",
|
||||
const LanguageHint& hint = {});
|
||||
|
||||
// Request builders, public so tests can compare them to the discovery
|
||||
// schema without any network.
|
||||
static clients::body::Json user_prompt_body(const DataItem& item,
|
||||
const LanguageHint& hint);
|
||||
static clients::body::Json model_response_body(const DataItem& item,
|
||||
const std::string& user_prompt,
|
||||
const LanguageHint& hint);
|
||||
// Resource paths (leading slash), validated.
|
||||
std::string template_path(const std::string& id) const;
|
||||
std::string floor_path(const std::string& scope) const;
|
||||
|
||||
private:
|
||||
ServiceEndpoint floor_endpoint() const;
|
||||
void require_regional() const;
|
||||
static std::string request_id_query(const std::string& request_id, char sep);
|
||||
|
||||
CloudContext context_;
|
||||
};
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,398 @@
|
||||
#include "google/armor/template.h"
|
||||
|
||||
#include "google/armor/caps.h"
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
const Json* member(const Json& doc, const char* key) {
|
||||
return doc.is_object() ? doc.find(key) : nullptr;
|
||||
}
|
||||
std::string text_of(const Json& doc, const char* key) {
|
||||
const Json* v = member(doc, key);
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
bool flag_of(const Json& doc, const char* key) {
|
||||
const Json* v = member(doc, key);
|
||||
return v && v->is_bool() && v->as_bool();
|
||||
}
|
||||
int int_of(const Json& doc, const char* key) {
|
||||
const Json* v = member(doc, key);
|
||||
return v && v->is_number() ? static_cast<int>(v->as_int()) : 0;
|
||||
}
|
||||
|
||||
bool valid_rai_type(const std::string& t) {
|
||||
return t == "SEXUALLY_EXPLICIT" || t == "HATE_SPEECH" || t == "HARASSMENT" ||
|
||||
t == "DANGEROUS";
|
||||
}
|
||||
|
||||
bool upper_name(const std::string& v) {
|
||||
if (v.empty() || v.size() > 64) return false;
|
||||
for (char c : v)
|
||||
if (!((c >= 'A' && c <= 'Z') || c == '_')) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
Json enabled_json(bool on) {
|
||||
Json o = Json::object();
|
||||
o.set("filterEnforcement", on ? "ENABLED" : "DISABLED");
|
||||
return o;
|
||||
}
|
||||
|
||||
bool enabled(const Json* settings) {
|
||||
return settings && text_of(*settings, "filterEnforcement") == "ENABLED";
|
||||
}
|
||||
|
||||
Json enforcement_json(const FloorEnforcement& e) {
|
||||
Json p = Json::object();
|
||||
p.set("enableCloudLogging", e.cloud_logging);
|
||||
p.set(e.inspect_only ? "inspectOnly" : "inspectAndBlock", true);
|
||||
return p;
|
||||
}
|
||||
|
||||
FloorEnforcement enforcement_from(const Json& p) {
|
||||
FloorEnforcement e;
|
||||
e.present = true;
|
||||
e.cloud_logging = flag_of(p, "enableCloudLogging");
|
||||
e.inspect_only = flag_of(p, "inspectOnly");
|
||||
return e;
|
||||
}
|
||||
|
||||
Json rule_sets_json(const std::vector<FilterRuleSet>& sets) {
|
||||
Json arr = Json::array();
|
||||
for (const auto& rs : sets) {
|
||||
Json set = Json::object();
|
||||
Json types = Json::array();
|
||||
for (const auto& t : rs.filter_types) types.push(t);
|
||||
set.set("filterTypes", types);
|
||||
Json rules = Json::array();
|
||||
for (const auto& r : rs.rules) {
|
||||
Json ex = Json::object();
|
||||
if (!r.matching_scope.empty()) ex.set("matchingScope", r.matching_scope);
|
||||
if (!r.regex.empty()) {
|
||||
Json re = Json::object();
|
||||
re.set("pattern", r.regex);
|
||||
ex.set("regex", re);
|
||||
} else {
|
||||
Json words = Json::array();
|
||||
for (const auto& w : r.words) words.push(w);
|
||||
Json list = Json::object();
|
||||
list.set("words", words);
|
||||
Json dict = Json::object();
|
||||
dict.set("wordList", list);
|
||||
ex.set("dictionary", dict);
|
||||
}
|
||||
Json rule = Json::object();
|
||||
rule.set("exclusionRule", ex);
|
||||
rules.push(rule);
|
||||
}
|
||||
set.set("rules", rules);
|
||||
arr.push(set);
|
||||
}
|
||||
Json settings = Json::object();
|
||||
settings.set("ruleSets", arr);
|
||||
return settings;
|
||||
}
|
||||
|
||||
std::vector<FilterRuleSet> rule_sets_from(const Json& settings) {
|
||||
std::vector<FilterRuleSet> out;
|
||||
const Json* sets = member(settings, "ruleSets");
|
||||
if (!sets || !sets->is_array()) return out;
|
||||
for (const Json& set : sets->as_array()) {
|
||||
if (out.size() >= ArmorCaps::kMaxRuleSets) break;
|
||||
FilterRuleSet rs;
|
||||
if (const Json* t = member(set, "filterTypes"); t && t->is_array())
|
||||
for (const Json& x : t->as_array())
|
||||
if (x.is_string()) rs.filter_types.push_back(x.as_string());
|
||||
if (const Json* rules = member(set, "rules"); rules && rules->is_array())
|
||||
for (const Json& rule : rules->as_array()) {
|
||||
const Json* ex = member(rule, "exclusionRule");
|
||||
if (!ex || rs.rules.size() >= ArmorCaps::kMaxRulesPerSet) continue;
|
||||
ExclusionRule er;
|
||||
er.matching_scope = text_of(*ex, "matchingScope");
|
||||
if (const Json* re = member(*ex, "regex")) er.regex = text_of(*re, "pattern");
|
||||
if (const Json* dict = member(*ex, "dictionary"))
|
||||
if (const Json* wl = member(*dict, "wordList"))
|
||||
if (const Json* w = member(*wl, "words"); w && w->is_array())
|
||||
for (const Json& x : w->as_array())
|
||||
if (x.is_string() && er.words.size() < ArmorCaps::kMaxRuleWords)
|
||||
er.words.push_back(x.as_string());
|
||||
rs.rules.push_back(std::move(er));
|
||||
}
|
||||
out.push_back(std::move(rs));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void FilterConfig::validate() const {
|
||||
if (rai.size() > ArmorCaps::kMaxRaiFilters)
|
||||
throw GoogleError::invalid("too many rai filters");
|
||||
for (const auto& f : rai)
|
||||
if (!valid_rai_type(f.type))
|
||||
throw GoogleError::invalid("rai filter type must be SEXUALLY_EXPLICIT, "
|
||||
"HATE_SPEECH, HARASSMENT or DANGEROUS");
|
||||
if (rule_sets.size() > ArmorCaps::kMaxRuleSets)
|
||||
throw GoogleError::invalid("too many rule sets");
|
||||
for (const auto& rs : rule_sets) {
|
||||
if (rs.rules.size() > ArmorCaps::kMaxRulesPerSet)
|
||||
throw GoogleError::invalid("too many rules in a rule set");
|
||||
for (const auto& t : rs.filter_types)
|
||||
if (!upper_name(t)) throw GoogleError::invalid("bad rule set filter type");
|
||||
for (const auto& r : rs.rules) {
|
||||
if (r.words.size() > ArmorCaps::kMaxRuleWords ||
|
||||
r.regex.size() > ArmorCaps::kMaxRegexChars)
|
||||
throw GoogleError::invalid("exclusion rule too large");
|
||||
if (!r.matching_scope.empty() && !upper_name(r.matching_scope))
|
||||
throw GoogleError::invalid("bad matching scope");
|
||||
if (r.regex.empty() && r.words.empty())
|
||||
throw GoogleError::invalid("exclusion rule needs words or a regex");
|
||||
}
|
||||
}
|
||||
if (sdp == SdpMode::kAdvanced && sdp_inspect_template.empty())
|
||||
throw GoogleError::invalid("advanced sdp needs an inspect template");
|
||||
if (sdp_inspect_template.size() > 512 || sdp_deidentify_template.size() > 512)
|
||||
throw GoogleError::invalid("sdp template name too long");
|
||||
}
|
||||
|
||||
Json FilterConfig::to_json() const {
|
||||
Json o = Json::object();
|
||||
if (!rai.empty()) {
|
||||
Json filters = Json::array();
|
||||
for (const auto& f : rai) {
|
||||
Json item = Json::object();
|
||||
item.set("filterType", f.type);
|
||||
if (f.confidence != Confidence::kUnspecified)
|
||||
item.set("confidenceLevel", confidence_name(f.confidence));
|
||||
filters.push(item);
|
||||
}
|
||||
Json settings = Json::object();
|
||||
settings.set("raiFilters", filters);
|
||||
o.set("raiSettings", settings);
|
||||
}
|
||||
if (!rule_sets.empty()) o.set("filterRuleSettings", rule_sets_json(rule_sets));
|
||||
if (sdp == SdpMode::kBasic) {
|
||||
Json settings = Json::object();
|
||||
settings.set("basicConfig", enabled_json(true));
|
||||
o.set("sdpSettings", settings);
|
||||
} else if (sdp == SdpMode::kAdvanced) {
|
||||
Json adv = Json::object();
|
||||
adv.set("inspectTemplate", sdp_inspect_template);
|
||||
if (!sdp_deidentify_template.empty())
|
||||
adv.set("deidentifyTemplate", sdp_deidentify_template);
|
||||
Json settings = Json::object();
|
||||
settings.set("advancedConfig", adv);
|
||||
o.set("sdpSettings", settings);
|
||||
}
|
||||
if (prompt_injection) {
|
||||
Json pi = enabled_json(true);
|
||||
if (prompt_injection_confidence != Confidence::kUnspecified)
|
||||
pi.set("confidenceLevel", confidence_name(prompt_injection_confidence));
|
||||
o.set("piAndJailbreakFilterSettings", pi);
|
||||
}
|
||||
if (malicious_uri) o.set("maliciousUriFilterSettings", enabled_json(true));
|
||||
return o;
|
||||
}
|
||||
|
||||
FilterConfig FilterConfig::from_json(const Json& doc) {
|
||||
FilterConfig c;
|
||||
if (const Json* r = member(doc, "raiSettings"))
|
||||
if (const Json* list = member(*r, "raiFilters"); list && list->is_array())
|
||||
for (const Json& f : list->as_array()) {
|
||||
if (c.rai.size() >= ArmorCaps::kMaxRaiFilters) break;
|
||||
c.rai.push_back({text_of(f, "filterType"),
|
||||
confidence_from(text_of(f, "confidenceLevel"))});
|
||||
}
|
||||
if (const Json* r = member(doc, "filterRuleSettings")) c.rule_sets = rule_sets_from(*r);
|
||||
if (const Json* s = member(doc, "sdpSettings")) {
|
||||
if (const Json* adv = member(*s, "advancedConfig")) {
|
||||
c.sdp = SdpMode::kAdvanced;
|
||||
c.sdp_inspect_template = text_of(*adv, "inspectTemplate");
|
||||
c.sdp_deidentify_template = text_of(*adv, "deidentifyTemplate");
|
||||
} else if (const Json* basic = member(*s, "basicConfig")) {
|
||||
if (enabled(basic)) c.sdp = SdpMode::kBasic;
|
||||
}
|
||||
}
|
||||
if (const Json* pi = member(doc, "piAndJailbreakFilterSettings")) {
|
||||
c.prompt_injection = enabled(pi);
|
||||
c.prompt_injection_confidence = confidence_from(text_of(*pi, "confidenceLevel"));
|
||||
}
|
||||
c.malicious_uri = enabled(member(doc, "maliciousUriFilterSettings"));
|
||||
return c;
|
||||
}
|
||||
|
||||
void TemplateMetadata::validate() const {
|
||||
if (!enforcement.empty() && enforcement != "INSPECT_ONLY" &&
|
||||
enforcement != "INSPECT_AND_BLOCK")
|
||||
throw GoogleError::invalid("enforcement must be INSPECT_ONLY or INSPECT_AND_BLOCK");
|
||||
if (custom_prompt_error_message.size() > 1024 ||
|
||||
custom_response_error_message.size() > 1024)
|
||||
throw GoogleError::invalid("custom error message too long");
|
||||
if (custom_prompt_error_code < 0 || custom_response_error_code < 0)
|
||||
throw GoogleError::invalid("custom error code must not be negative");
|
||||
if (!filter_version_alias.empty() && !upper_name(filter_version_alias))
|
||||
throw GoogleError::invalid("bad filter version alias");
|
||||
if (!filter_version.empty() && !filter_version_alias.empty())
|
||||
throw GoogleError::invalid("filter version and alias are mutually exclusive");
|
||||
if (filter_version.size() > 32) throw GoogleError::invalid("filter version too long");
|
||||
if (modalities.size() > 2) throw GoogleError::invalid("too many modalities");
|
||||
for (const auto& m : modalities)
|
||||
if (m != "MODALITY_TEXT" && m != "MODALITY_IMAGE")
|
||||
throw GoogleError::invalid("modality must be MODALITY_TEXT or MODALITY_IMAGE");
|
||||
}
|
||||
|
||||
Json TemplateMetadata::to_json() const {
|
||||
Json o = Json::object();
|
||||
if (!enforcement.empty()) o.set("enforcementType", enforcement);
|
||||
if (ignore_partial_failures) o.set("ignorePartialInvocationFailures", true);
|
||||
if (log_template_operations) o.set("logTemplateOperations", true);
|
||||
if (log_sanitize_operations) o.set("logSanitizeOperations", true);
|
||||
if (multi_language_detection) {
|
||||
Json m = Json::object();
|
||||
m.set("enableMultiLanguageDetection", true);
|
||||
o.set("multiLanguageDetection", m);
|
||||
}
|
||||
if (custom_prompt_error_code) o.set("customPromptSafetyErrorCode", custom_prompt_error_code);
|
||||
if (!custom_prompt_error_message.empty())
|
||||
o.set("customPromptSafetyErrorMessage", custom_prompt_error_message);
|
||||
if (custom_response_error_code)
|
||||
o.set("customLlmResponseSafetyErrorCode", custom_response_error_code);
|
||||
if (!custom_response_error_message.empty())
|
||||
o.set("customLlmResponseSafetyErrorMessage", custom_response_error_message);
|
||||
if (!filter_version.empty() || !filter_version_alias.empty()) {
|
||||
Json sel = Json::object();
|
||||
if (!filter_version.empty()) sel.set("version", filter_version);
|
||||
if (!filter_version_alias.empty()) sel.set("alias", filter_version_alias);
|
||||
o.set("filterVersionSelector", sel);
|
||||
}
|
||||
if (!modalities.empty()) {
|
||||
Json m = Json::array();
|
||||
for (const auto& x : modalities) m.push(x);
|
||||
o.set("modalities", m);
|
||||
}
|
||||
if (data_residency_set) o.set("dataResidencyCompliant", data_residency_compliant);
|
||||
return o;
|
||||
}
|
||||
|
||||
TemplateMetadata TemplateMetadata::from_json(const Json& doc) {
|
||||
TemplateMetadata m;
|
||||
m.enforcement = text_of(doc, "enforcementType");
|
||||
if (m.enforcement == "ENFORCEMENT_TYPE_UNSPECIFIED") m.enforcement.clear();
|
||||
m.ignore_partial_failures = flag_of(doc, "ignorePartialInvocationFailures");
|
||||
m.log_template_operations = flag_of(doc, "logTemplateOperations");
|
||||
m.log_sanitize_operations = flag_of(doc, "logSanitizeOperations");
|
||||
if (const Json* ml = member(doc, "multiLanguageDetection"))
|
||||
m.multi_language_detection = flag_of(*ml, "enableMultiLanguageDetection");
|
||||
m.custom_prompt_error_code = int_of(doc, "customPromptSafetyErrorCode");
|
||||
m.custom_prompt_error_message = text_of(doc, "customPromptSafetyErrorMessage");
|
||||
m.custom_response_error_code = int_of(doc, "customLlmResponseSafetyErrorCode");
|
||||
m.custom_response_error_message = text_of(doc, "customLlmResponseSafetyErrorMessage");
|
||||
if (const Json* sel = member(doc, "filterVersionSelector")) {
|
||||
m.filter_version = text_of(*sel, "version");
|
||||
m.filter_version_alias = text_of(*sel, "alias");
|
||||
}
|
||||
if (const Json* mods = member(doc, "modalities"); mods && mods->is_array())
|
||||
for (const Json& x : mods->as_array())
|
||||
if (x.is_string() && m.modalities.size() < 2) m.modalities.push_back(x.as_string());
|
||||
if (const Json* r = member(doc, "dataResidencyCompliant"); r && r->is_bool()) {
|
||||
m.data_residency_set = true;
|
||||
m.data_residency_compliant = r->as_bool();
|
||||
}
|
||||
return m;
|
||||
}
|
||||
|
||||
void Template::validate() const {
|
||||
filter.validate();
|
||||
metadata.validate();
|
||||
if (labels.size() > ArmorCaps::kMaxLabels)
|
||||
throw GoogleError::invalid("too many labels");
|
||||
for (const auto& [k, v] : labels)
|
||||
if (k.empty() || k.size() > ArmorCaps::kMaxLabelChars ||
|
||||
v.size() > ArmorCaps::kMaxLabelChars)
|
||||
throw GoogleError::invalid("label key or value too long");
|
||||
}
|
||||
|
||||
Json Template::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("filterConfig", filter.to_json());
|
||||
Json meta = metadata.to_json();
|
||||
if (!meta.as_object().empty()) o.set("templateMetadata", meta);
|
||||
if (!labels.empty()) {
|
||||
Json l = Json::object();
|
||||
for (const auto& [k, v] : labels) l.set(k, v);
|
||||
o.set("labels", l);
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
Template Template::from_json(const Json& doc) {
|
||||
Template t;
|
||||
t.name = text_of(doc, "name");
|
||||
t.create_time = text_of(doc, "createTime");
|
||||
t.update_time = text_of(doc, "updateTime");
|
||||
if (const Json* f = member(doc, "filterConfig")) t.filter = FilterConfig::from_json(*f);
|
||||
if (const Json* m = member(doc, "templateMetadata"))
|
||||
t.metadata = TemplateMetadata::from_json(*m);
|
||||
if (const Json* l = member(doc, "labels"); l && l->is_object())
|
||||
for (const auto& [k, v] : l->as_object())
|
||||
if (v.is_string() && t.labels.size() < ArmorCaps::kMaxLabels)
|
||||
t.labels[k] = v.as_string();
|
||||
return t;
|
||||
}
|
||||
|
||||
void FloorSetting::validate() const {
|
||||
filter.validate();
|
||||
for (const auto& s : integrated_services)
|
||||
if (s != "AI_PLATFORM" && s != "GOOGLE_MCP_SERVER")
|
||||
throw GoogleError::invalid(
|
||||
"integrated service must be AI_PLATFORM or GOOGLE_MCP_SERVER");
|
||||
if (integrated_services.size() > 2) throw GoogleError::invalid("too many integrated services");
|
||||
}
|
||||
|
||||
Json FloorSetting::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("filterConfig", filter.to_json());
|
||||
o.set("enableFloorSettingEnforcement", enforcement_enabled);
|
||||
if (!integrated_services.empty()) {
|
||||
Json s = Json::array();
|
||||
for (const auto& x : integrated_services) s.push(x);
|
||||
o.set("integratedServices", s);
|
||||
}
|
||||
if (ai_platform.present) o.set("aiPlatformFloorSetting", enforcement_json(ai_platform));
|
||||
if (mcp_server.present) o.set("googleMcpServerFloorSetting", enforcement_json(mcp_server));
|
||||
if (multi_language_detection) {
|
||||
Json ml = Json::object();
|
||||
ml.set("enableMultiLanguageDetection", true);
|
||||
Json meta = Json::object();
|
||||
meta.set("multiLanguageDetection", ml);
|
||||
o.set("floorSettingMetadata", meta);
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
FloorSetting FloorSetting::from_json(const Json& doc) {
|
||||
FloorSetting f;
|
||||
f.name = text_of(doc, "name");
|
||||
f.update_time = text_of(doc, "updateTime");
|
||||
f.ai_platform = FloorEnforcement{};
|
||||
if (const Json* c = member(doc, "filterConfig")) f.filter = FilterConfig::from_json(*c);
|
||||
f.enforcement_enabled = flag_of(doc, "enableFloorSettingEnforcement");
|
||||
if (const Json* s = member(doc, "integratedServices"); s && s->is_array())
|
||||
for (const Json& x : s->as_array())
|
||||
if (x.is_string() && f.integrated_services.size() < 2)
|
||||
f.integrated_services.push_back(x.as_string());
|
||||
if (const Json* p = member(doc, "aiPlatformFloorSetting")) f.ai_platform = enforcement_from(*p);
|
||||
if (const Json* p = member(doc, "googleMcpServerFloorSetting"))
|
||||
f.mcp_server = enforcement_from(*p);
|
||||
if (const Json* m = member(doc, "floorSettingMetadata"))
|
||||
if (const Json* ml = member(*m, "multiLanguageDetection"))
|
||||
f.multi_language_detection = flag_of(*ml, "enableMultiLanguageDetection");
|
||||
return f;
|
||||
}
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,118 @@
|
||||
#pragma once
|
||||
// Model Armor template and floor setting, typed. Wire names are the ones in
|
||||
// the Model Armor v1 Discovery document (see calls.h and SPEC.md).
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "google/armor/armor_types.h"
|
||||
|
||||
namespace molo::google::armor {
|
||||
|
||||
struct RaiFilter {
|
||||
std::string type; // SEXUALLY_EXPLICIT | HATE_SPEECH | HARASSMENT | DANGEROUS
|
||||
Confidence confidence = Confidence::kUnspecified;
|
||||
};
|
||||
|
||||
// Sensitive Data Protection: off, the fixed basic info types, or your own
|
||||
// inspect (and optional de-identify) templates.
|
||||
enum class SdpMode { kOff, kBasic, kAdvanced };
|
||||
|
||||
// filterRuleSettings.ruleSets[].rules[].exclusionRule
|
||||
struct ExclusionRule {
|
||||
std::string matching_scope; // MATCHING_SCOPE_FULL_MATCH | _PARTIAL_MATCH
|
||||
std::vector<std::string> words; // dictionary.wordList.words
|
||||
std::string regex; // regex.pattern (instead of words)
|
||||
};
|
||||
struct FilterRuleSet {
|
||||
std::vector<std::string> filter_types; // Enum names as Google spells them.
|
||||
std::vector<ExclusionRule> rules;
|
||||
};
|
||||
|
||||
struct FilterConfig {
|
||||
std::vector<RaiFilter> rai;
|
||||
std::vector<FilterRuleSet> rule_sets;
|
||||
SdpMode sdp = SdpMode::kOff;
|
||||
std::string sdp_inspect_template; // advanced only
|
||||
std::string sdp_deidentify_template; // advanced only, optional
|
||||
bool prompt_injection = false; // piAndJailbreakFilterSettings
|
||||
Confidence prompt_injection_confidence = Confidence::kUnspecified;
|
||||
bool malicious_uri = false;
|
||||
|
||||
// Throws GoogleError::invalid on a bad rai type, oversized rules or an
|
||||
// advanced sdp config without an inspect template.
|
||||
void validate() const;
|
||||
clients::body::Json to_json() const;
|
||||
static FilterConfig from_json(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
struct TemplateMetadata {
|
||||
std::string enforcement; // "", INSPECT_ONLY, INSPECT_AND_BLOCK
|
||||
bool ignore_partial_failures = false;
|
||||
bool log_template_operations = false;
|
||||
bool log_sanitize_operations = false;
|
||||
bool multi_language_detection = false;
|
||||
int custom_prompt_error_code = 0; // 0: unset
|
||||
std::string custom_prompt_error_message;
|
||||
int custom_response_error_code = 0;
|
||||
std::string custom_response_error_message;
|
||||
std::string filter_version; // filterVersionSelector.version
|
||||
std::string filter_version_alias; // filterVersionSelector.alias (enum name)
|
||||
std::vector<std::string> modalities; // MODALITY_TEXT, MODALITY_IMAGE
|
||||
bool data_residency_set = false;
|
||||
bool data_residency_compliant = false;
|
||||
|
||||
void validate() const;
|
||||
clients::body::Json to_json() const;
|
||||
static TemplateMetadata from_json(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
struct Template {
|
||||
std::string name; // projects/{p}/locations/{l}/templates/{id}
|
||||
std::map<std::string, std::string> labels;
|
||||
FilterConfig filter;
|
||||
TemplateMetadata metadata;
|
||||
std::string create_time;
|
||||
std::string update_time;
|
||||
|
||||
void validate() const;
|
||||
clients::body::Json to_json() const; // Request body (no output-only fields).
|
||||
static Template from_json(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
struct TemplatePage {
|
||||
std::vector<Template> templates;
|
||||
std::vector<std::string> unreachable; // Locations Google could not reach.
|
||||
};
|
||||
|
||||
// aiPlatformFloorSetting / googleMcpServerFloorSetting.
|
||||
struct FloorEnforcement {
|
||||
bool present = false; // Serialized only when set.
|
||||
bool cloud_logging = false;
|
||||
bool inspect_only = false; // else inspectAndBlock
|
||||
};
|
||||
|
||||
struct FloorSetting {
|
||||
std::string name; // {projects|folders|organizations}/{id}/locations/{l}/floorSetting
|
||||
FilterConfig filter;
|
||||
bool enforcement_enabled = false;
|
||||
std::vector<std::string> integrated_services; // AI_PLATFORM, GOOGLE_MCP_SERVER
|
||||
FloorEnforcement ai_platform{true, false, false};
|
||||
FloorEnforcement mcp_server;
|
||||
bool multi_language_detection = false;
|
||||
std::string update_time;
|
||||
|
||||
void validate() const;
|
||||
clients::body::Json to_json() const;
|
||||
static FloorSetting from_json(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
// One entry of locations.list.
|
||||
struct ArmorLocation {
|
||||
std::string name;
|
||||
std::string location_id;
|
||||
std::string display_name;
|
||||
};
|
||||
|
||||
} // namespace molo::google::armor
|
||||
@@ -0,0 +1,112 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Checks src/google/armor/calls.h against Google's Discovery documents.
|
||||
|
||||
Fails (exit 1) when a declared call (rpc id, HTTP method, path, query
|
||||
parameter), a declared schema field, or a declared enum value is absent from
|
||||
the live Discovery document. Stdlib only, no credentials needed.
|
||||
|
||||
python3 -I src/google/armor/tools/check_discovery.py # fetch live
|
||||
python3 -I src/google/armor/tools/check_discovery.py --cache DIR # read DIR/<service>-<version>.json
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
HEADER = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "calls.h")
|
||||
ROW = re.compile(r'^\s*\{((?:"[^"]*"(?:,\s*)?)+)\},\s*$')
|
||||
STR = re.compile(r'"([^"]*)"')
|
||||
|
||||
|
||||
def rows():
|
||||
calls, schemas, enums = [], [], []
|
||||
for line in open(HEADER, encoding="utf-8"):
|
||||
m = ROW.match(line)
|
||||
if not m:
|
||||
continue
|
||||
cells = STR.findall(m.group(1))
|
||||
{6: calls, 4: schemas, 5: enums}.get(len(cells), []).append(cells)
|
||||
return calls, schemas, enums
|
||||
|
||||
|
||||
def load(service, version, cache):
|
||||
if cache:
|
||||
path = os.path.join(cache, "%s-%s.json" % (service, version))
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
url = "https://%s.googleapis.com/$discovery/rest?version=%s" % (service, version)
|
||||
with urllib.request.urlopen(url, timeout=60) as r:
|
||||
return json.load(r)
|
||||
|
||||
|
||||
def methods(doc):
|
||||
out = {}
|
||||
|
||||
def walk(res):
|
||||
for sub in res.get("resources", {}).values():
|
||||
walk(sub)
|
||||
for m in res.get("methods", {}).values():
|
||||
out[m["id"]] = m
|
||||
|
||||
walk(doc)
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
cache = None
|
||||
if "--cache" in sys.argv:
|
||||
cache = sys.argv[sys.argv.index("--cache") + 1]
|
||||
calls, schemas, enums = rows()
|
||||
if not calls or not schemas or not enums:
|
||||
print("FAIL: no rows parsed from calls.h")
|
||||
return 1
|
||||
docs, errors = {}, []
|
||||
|
||||
def doc(service, version):
|
||||
key = (service, version)
|
||||
if key not in docs:
|
||||
docs[key] = load(service, version, cache)
|
||||
return docs[key]
|
||||
|
||||
index = {}
|
||||
for service, version, rpc, http, path, query in calls:
|
||||
d = doc(service, version)
|
||||
idx = index.setdefault((service, version), methods(d))
|
||||
m = idx.get(rpc)
|
||||
if not m:
|
||||
errors.append("call missing: " + rpc)
|
||||
continue
|
||||
if m["httpMethod"] != http:
|
||||
errors.append("%s: http %s != declared %s" % (rpc, m["httpMethod"], http))
|
||||
if m["path"] != path:
|
||||
errors.append("%s: path %s != declared %s" % (rpc, m["path"], path))
|
||||
for q in filter(None, query.split(",")):
|
||||
if q not in m.get("parameters", {}):
|
||||
errors.append("%s: query parameter %s missing" % (rpc, q))
|
||||
for service, version, schema, fields in schemas:
|
||||
s = doc(service, version)["schemas"].get(schema)
|
||||
if not s:
|
||||
errors.append("schema missing: " + schema)
|
||||
continue
|
||||
for f in fields.split(","):
|
||||
if f not in s.get("properties", {}):
|
||||
errors.append("%s: field %s missing" % (schema, f))
|
||||
for service, version, schema, field, values in enums:
|
||||
s = doc(service, version)["schemas"].get(schema, {})
|
||||
p = s.get("properties", {}).get(field)
|
||||
if not p or "enum" not in p:
|
||||
errors.append("%s.%s: no enum" % (schema, field))
|
||||
continue
|
||||
for v in values.split(","):
|
||||
if v not in p["enum"]:
|
||||
errors.append("%s.%s: value %s missing" % (schema, field, v))
|
||||
for e in errors:
|
||||
print("FAIL:", e)
|
||||
print("%s: %d calls, %d schemas, %d enums checked, %d problems"
|
||||
% ("FAIL" if errors else "OK", len(calls), len(schemas), len(enums), len(errors)))
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user