Merge branch 'worktree-agent-a2e298643b4505a43'

This commit is contained in:
2026-10-25 15:24:27 +01:00
26 changed files with 7030 additions and 3 deletions
+213
View File
@@ -0,0 +1,213 @@
# Google Discovery Engine - Specification
Owned directory: `src/google/discovery/`. Namespace `molo::google::discovery`.
Depends on the `src/google` foundation (`GoogleClient`, `GoogleError`,
`ServiceEndpoint`, `Operation`) and `clients::Client` for document fetches.
No other google package depends on it and it depends on none. Plan:
`plans/18-breadth.md` section 1.
## 1. Scope
One generic engine that reaches every API in Google's Discovery directory
(`https://discovery.googleapis.com/discovery/v1/apis`: 538 entries, 316
preferred, 2026-10-25) and calls any method by its Discovery id
(`serviceusage.services.list`). It fetches and caches a Discovery document,
lists APIs and methods, validates parameters and body against the document
before any traffic, builds the URL (path templates, query, repeated values,
regional host), chooses key or OAuth, walks `nextPageToken`, and polls
long-running operations. Typed classes in other packages are better for the
APIs they cover; this engine is the breadth.
Not here (each is classified Limited or Rejected, see section 5): media
upload and download bodies, resumable upload, bidirectional or client
streaming, non-JSON replies, the global batch endpoint (`batchPath`, shut
down by Google in 2020), and object-typed query parameters.
## 2. Module map (one concern per file)
| File | Concern |
|---|---|
| `caps.h` | Static ceilings (`DiscoveryCaps`) |
| `model.h/.cpp` | `Api`, `Method`, `Param`: parse a document, classify every method (`Support` Full / Limited / Rejected with notes), pagination and operation detection, host allowlist |
| `validate.h/.cpp` | `ParamValidator` (type, format, enum, pattern, range, repeated, path safety) and `BodyValidator` (schema walk: `$ref`, objects, maps, arrays, enums, int64 strings, free-form) |
| `planner.h/.cpp` | `Planner::plan`: `(Api, Method, CallArgs)` to `PreparedCall` (endpoint, encoded path and query, body, credential choice). No traffic |
| `catalog.h/.cpp` | `Directory`, `DocumentSource` (`HttpDocumentSource`, `FileDocumentSource`), `Catalog` (bounded LRU cache with age and refetch floor) |
| `engine.h/.cpp` | `Engine`: `list_apis`, `list_methods`, `describe`, `plan`, `call` (paging, long-running wait) |
| `tools/check_discovery_directory.py` | Fetches the whole directory and every preferred document, runs the engine's own classification over them, reports what it cannot handle |
| `tools/audit_main.cc` | The C++ half of that checker (built by the script; not globbed by `google.mk`) |
## 3. Static caps (see caps.h)
| Knob | Value |
|---|---|
| Document size | 8 MiB (largest real: compute v1, 6.4 MB) |
| Directory size / items | 2 MiB / 1000 |
| Cached documents / cached bytes | 32 / 96 MiB, least recently used evicted |
| Document max age | 24 h |
| Refetch floor | 60 s per document, even after `forget` |
| Methods per document | 10000 (largest real: compute v1, 1018; aiplatform v1, 1137) |
| Resource nesting | 16 |
| Parameter value | 2048 chars, no control characters |
| Repeated values / query values | 100 / 64 |
| Request target | 6000 chars |
| Request body | 4 MiB, schema depth 32, 200000 visited nodes |
| Pages per call / items per call | 50 / 10000 (default 1 page) |
| Listing | 500 rows |
| Operation wait | foundation `OperationCaps`: 400 polls, 2 s interval, 600 s |
## 4. Contracts
- Documents are fetched without a credential, over HTTPS, from hosts that
are `googleapis.com` or a subdomain only (`is_google_host_url`). A document
whose `rootUrl` or regional `endpointUrl` fails that test is rejected
(`BAD_DOCUMENT`), as is a document that is not the one the directory
listed, a wrong `kind` or `protocol`, a duplicate method id, or anything
over a cap.
- API reference: `name` (preferred version) or `name:version`. The method id
prefix is not always the API name (`admin` serves `reports.activities.list`),
so the API is always given explicitly.
- Validation order, all before traffic: API and method exist (404), method
not Rejected, `params` is an object, regional `location` exists in the
document, every param is declared (global params: only `fields`,
`quotaUser`, `prettyPrint`; `key`, `access_token`, `oauth_token`, `alt`,
`callback`, `uploadType`, `upload_protocol`, `$.xgafv` are refused by name),
values fit type / format / enum / pattern / range, required params are
present, path values have no dot segment, the body fits the request schema.
Failures are `GoogleError` 400 `INVALID_ARGUMENT` naming the parameter or
JSON path. After validation, no credential is `NotConfiguredError` (503).
- Path templates are `{x}` (value fully percent-encoded, `/` included) or
`{+x}` (reserved expansion: each `/` segment encoded, slashes kept).
Query parameters are sorted by name, repeated parameters repeat the key.
- Body: methods with no request schema take none; POST / PUT / PATCH without
a request schema or without a body send `{}`. Unknown properties are
rejected when the schema lists properties; free-form objects and `any`
pass; `null` passes (proto JSON unset); int64 / uint64 are digit strings
(an integral JSON number is accepted and converted).
- Credential: `AuthMode::kAuto` sends the API key for methods that declare no
OAuth scope (205 of 13194) and OAuth for the rest; `kKey` / `kOAuth` force
it through `ServiceEndpoint::prefer_key`. `GoogleClient` falls back to the
other credential only when the forced one is not configured.
- Regional hosts: `location` selects the `endpoints[]` row of the document
(69 documents have rows, e.g. `workflows.us-central1.rep.googleapis.com`).
An unknown location is rejected with examples; a document without rows
rejects any location. Nothing is guessed from the location string.
- Pagination: `max_pages > 1` needs a `pageToken` query param and a reply
schema with `nextPageToken`. The array to collect is the single array
property of the reply (`items_key` otherwise). The merged reply has the
collected array; `truncated` is true when a page or item cap stopped the
walk while a token remained (the token stays in the body).
- Long-running: a reply whose schema has `name` and boolean `done` is an
operation. With `wait`, the engine finds the API's `operations.get` method
whose `name` pattern matches, polls it with the foundation caps, throws
`OPERATION_FAILED` on an error result and `DEADLINE_EXCEEDED` at the cap.
APIs without an `operations.get` (111 methods) reject `wait` loudly.
Operations that use another shape (Compute `status` / `selfLink`) are
plain resources to this engine.
- Destructive or sending methods are ordinary methods here; the caller (the
demo) decides. The demo runs GET at once and dry-runs every other verb
unless `confirm:true`.
## 5. Shapes seen
Read from the 310 reachable preferred documents (13194 methods, 27285
schemas), 2026-10-25, and how each is handled:
| Shape | Seen | Handling |
|---|---|---|
| `path` templates `{x}` / `{+x}` only; every variable declared as a path param | 4712 / 10563 variables, no exception | Substituted and encoded; anything else Rejected |
| `flatPath` | 13034 methods | Ignored: `path` is authoritative and equivalent |
| `rootUrl` + `servicePath`; 12 documents have a non-empty `servicePath` | 310 | `base_url` = root (or regional root) + servicePath; `path` is relative to it |
| Param `location` | `path`, `query` only | Anything else Rejected |
| Param types | string, integer, boolean, number; `object` once (`discoveryengine ...dataConnector.mcp` `extensions`) | Object params Rejected (1 method) |
| Param `format` | int32, uint32, int64, uint64, double, float, byte, date-time, google-datetime, google-duration, google-fieldmask | Checked for integers and the 64-bit strings, date and duration shape; others are strings |
| `repeated` | 509 params | Array or scalar; cap 100 |
| `enum` | 710 params | Checked |
| `pattern` | 12636 params | `std::regex` ECMAScript; a pattern it cannot parse is left to Google |
| `minimum` / `maximum` (strings) | 306 / 61 | Checked |
| `$ref` request/response, nested `$ref`, `items`, `additionalProperties` (map types), `any`, readOnly | all | Body validator follows `$ref` (chain cap 16), maps, arrays; readOnly is not enforced |
| `mediaUpload` (simple, resumable) | 53 methods | Limited: the metadata call is made, no media bytes |
| `supportsMediaDownload` | 20 methods | Limited: JSON reply only, `alt=media` is refused |
| Reply `HttpBody` | 81 methods | Limited: the reply may not be JSON; a parse failure is `BAD_RESPONSE` |
| `streamingType` | `NONE` 4, `SERVER_SIDE` 2 (admob) | NONE ignored; SERVER_SIDE Limited (read whole); anything else Rejected |
| `batchPath` | 310 documents | Not used (Google removed the global batch endpoint) |
| `endpoints[]` | 69 documents | Regional hosts, see section 4 |
| `mtlsRootUrl`, `canonicalName`, `version_module`, `icons`, `auth` | many | Ignored |
| Operation-shaped replies (`name` + `done`) | 2359 methods | `wait`; 111 are in documents with no `operations.get` |
| `pageToken` + `nextPageToken` | 2386 methods | Paging |
| Other paging schemes (`start-index`, `nextLink`) | not counted | Not paged; one call per request |
## Required APIs
Enable these on the project (placeholder project `example-214512124`; full
list in `src/google/APIS.md`). The engine itself needs none: Discovery
documents are public. Calling an API needs that API enabled; a disabled one
answers 403 `SERVICE_DISABLED`. The live checks use:
- `serviceusage.googleapis.com`: https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=example-214512124
Scopes: `cloud-platform` covers the Cloud APIs. Workspace and other APIs
need the scopes named in each method's `scopes` (`describe` shows them); the
service-account credential in this repository supplies `cloud-platform` only.
## Sources
- `https://discovery.googleapis.com/discovery/v1/apis` (directory, 538
entries, 316 preferred) and every preferred `discoveryRestUrl`, fetched
2026-10-25.
- Discovery format reference: `https://developers.google.com/discovery/v1/reference/apis`
and `.../v1/using` (parameter and method fields).
## Verified against discovery
`python3 src/google/discovery/tools/check_discovery_directory.py`,
2026-10-25: 316 preferred APIs listed, 310 documents served and loaded by the
engine, 0 rejected documents. Six listed APIs are not served by Google
(`area120tables:v1alpha1`, `datalabeling:v1beta1`, `mybusinessqanda:v1` 404;
`contentwarehouse:v1`, `poly:v1`, `rapidmigrationassessment:v1` 502): not an
engine fault, listed by the script as UNREACHABLE. Of 13194 methods: 13037
Full, 156 Limited (81 HttpBody replies, 53 media upload, 20 media download,
2 server streaming), 1 Rejected (object query parameter). Dry plan of every
callable method with synthesized required parameters: 13185 planned, 8
skipped (synthetic value does not fit an exotic pattern), 0 failed.
## Verified live
`make google-live` as the service account on project `lisa-448004`,
2026-10-25 (read-only):
- Real directory over HTTPS: 538 items, 316 preferred; documents of
serviceusage, cloudresourcemanager, tasks, workflows parsed.
- Largest documents fit the caps: compute v1 (1018 methods, 6419010 bytes),
aiplatform v1 (1137 methods, 4027902 bytes).
- `serviceusage.services.get` for `serviceusage.googleapis.com` answered 200.
- `serviceusage.services.list` with `pageSize=3`, 3 pages: 9 items,
`truncated` true (the page cap, the token left in the body).
- `serviceusage.operations.get` with a missing name: 400 INVALID_ARGUMENT
(Google validates the name shape); the reply is recognised as an operation.
- Cloud Resource Manager answers `PERMISSION_DENIED` (API not enabled on the
project): reported by Google, not exercised.
## Unverified
- Long-running `wait` polling to completion: no safe read-only operation
exists on this project, so only detection and the `operations.get` lookup
are exercised live. The loop is the foundation's `wait_operation` logic
re-implemented over the planner.
- Key credential on a no-scope method live: no API key with a keyless-capable
API was used; the choice is verified at the request level (header).
- Regional endpoints are verified for URL construction only, not by a call.
- Methods Limited or Rejected are never called by design.
- Behaviour for the 8 methods whose required-parameter patterns the audit
could not synthesize is only the validation path.
## Gaps
- Media upload (simple, resumable), media download and non-JSON replies:
`Method::support` says Limited; the foundation `GoogleClient` is JSON only
(`src/google/client.h`).
- Batch (`batchPath`) is not offered.
- Paging schemes other than `pageToken` / `nextPageToken`.
- `AuthMode::kOAuth` falls back to the key when no OAuth credential exists
(foundation `GoogleClient::pick_auth`).
- Documents are cached in memory only; there is no on-disk cache.
- Test and audit coverage of `wait` against a real running operation.
+44
View File
@@ -0,0 +1,44 @@
#pragma once
// Static ceilings for the Discovery engine. Never autotuned. Measured
// against the 310 reachable preferred documents on 2026-10-25: largest
// document 6.4 MB (compute v1), most methods in one document is below
// kMaxMethods, directory 538 entries.
#include <chrono>
#include <cstddef>
namespace molo::google::discovery {
struct DiscoveryCaps {
// Documents and cache.
static constexpr std::size_t kMaxDocBytes = 8u * 1024u * 1024u;
static constexpr std::size_t kMaxDirectoryBytes = 2u * 1024u * 1024u;
static constexpr std::size_t kMaxDirectoryItems = 1000;
static constexpr std::size_t kMaxCachedApis = 32;
static constexpr std::size_t kMaxCachedBytes = 96u * 1024u * 1024u;
static constexpr std::chrono::seconds kMaxAge{24 * 3600};
// Floor: a document is never refetched more often than this, even when
// the caller forces a refresh.
static constexpr std::chrono::seconds kMinRefetch{60};
// Document structure.
static constexpr std::size_t kMaxMethods = 10000;
static constexpr std::size_t kMaxResourceDepth = 16;
static constexpr std::size_t kMaxSchemas = 20000;
// Listing.
static constexpr std::size_t kMaxListed = 500;
// Request validation.
static constexpr std::size_t kMaxParamChars = 2048;
static constexpr std::size_t kMaxRepeated = 100;
static constexpr std::size_t kMaxQueryParams = 64;
static constexpr std::size_t kMaxPathChars = 6000;
static constexpr std::size_t kMaxBodyBytes = 4u * 1024u * 1024u;
static constexpr std::size_t kMaxSchemaDepth = 32;
static constexpr std::size_t kMaxSchemaVisits = 200000;
// Pagination.
static constexpr std::size_t kDefaultPages = 1;
static constexpr std::size_t kMaxPages = 50;
static constexpr std::size_t kMaxItems = 10000;
};
} // namespace molo::google::discovery
+285
View File
@@ -0,0 +1,285 @@
#include "google/discovery/catalog.h"
#include <algorithm>
#include <cctype>
#include <fstream>
#include <sstream>
#include "clients/transports/http_transport.h"
#include "google/discovery/caps.h"
#include "google/errors.h"
namespace molo::google::discovery {
using clients::body::Json;
const char* const kDirectoryUrl = "https://discovery.googleapis.com/discovery/v1/apis";
namespace {
std::string str(const Json& o, const char* key) {
const Json* v = o.find(key);
return v && v->is_string() ? v->as_string() : "";
}
std::string lower(std::string s) {
for (char& c : s) c = static_cast<char>(std::tolower(static_cast<unsigned char>(c)));
return s;
}
bool valid_name(const std::string& s) {
if (s.empty() || s.size() > 64) return false;
for (char c : s)
if (!(std::isalnum(static_cast<unsigned char>(c)) || c == '_' || c == '-' || c == '.')) return false;
return true;
}
} // namespace
Json DirectoryItem::to_json() const {
Json o = Json::object();
o.set("id", Json(id));
o.set("name", Json(name));
o.set("version", Json(version));
o.set("title", Json(title));
o.set("description", Json(description));
o.set("discoveryRestUrl", Json(rest_url));
o.set("documentationLink", Json(documentation));
o.set("preferred", Json(preferred));
return o;
}
Directory Directory::parse(const std::string& text) {
if (text.size() > DiscoveryCaps::kMaxDirectoryBytes)
throw GoogleError(502, "BAD_DOCUMENT", "discovery directory is larger than the cap");
Json doc;
try {
doc = Json::parse(text);
} catch (const std::exception& e) {
throw GoogleError(502, "BAD_DOCUMENT", std::string("discovery directory: ") + e.what());
}
const Json* items = doc.is_object() ? doc.find("items") : nullptr;
if (!items || !items->is_array() || str(doc, "kind") != "discovery#directoryList")
throw GoogleError(502, "BAD_DOCUMENT", "discovery directory: not a directoryList");
if (items->as_array().size() > DiscoveryCaps::kMaxDirectoryItems)
throw GoogleError(502, "BAD_DOCUMENT", "discovery directory: more items than the cap");
Directory dir;
for (const Json& it : items->as_array()) {
DirectoryItem row;
row.id = str(it, "id");
row.name = str(it, "name");
row.version = str(it, "version");
row.title = str(it, "title");
row.description = str(it, "description");
row.rest_url = str(it, "discoveryRestUrl");
row.documentation = str(it, "documentationLink");
const Json* p = it.find("preferred");
row.preferred = p && p->is_bool() && p->as_bool();
if (row.id.empty() || row.name.empty() || row.version.empty() ||
!is_google_host_url(row.rest_url))
throw GoogleError(502, "BAD_DOCUMENT", "discovery directory: bad item " + row.id);
dir.items_.push_back(std::move(row));
}
return dir;
}
std::size_t Directory::preferred_count() const {
return static_cast<std::size_t>(std::count_if(items_.begin(), items_.end(),
[](const DirectoryItem& i) { return i.preferred; }));
}
const DirectoryItem* Directory::find(const std::string& name, const std::string& version) const {
const DirectoryItem* fallback = nullptr;
for (const DirectoryItem& i : items_) {
if (i.name != name) continue;
if (!version.empty()) {
if (i.version == version) return &i;
} else if (i.preferred) {
return &i;
} else if (!fallback) {
fallback = &i;
}
}
return version.empty() ? fallback : nullptr;
}
std::vector<const DirectoryItem*> Directory::search(const std::string& needle, std::size_t limit,
bool preferred_only) const {
std::vector<const DirectoryItem*> out;
std::string n = lower(needle);
limit = std::min(limit, DiscoveryCaps::kMaxListed);
for (const DirectoryItem& i : items_) {
if (out.size() >= limit) break;
if (preferred_only && !i.preferred) continue;
if (n.empty() || lower(i.name).find(n) != std::string::npos ||
lower(i.title).find(n) != std::string::npos)
out.push_back(&i);
}
return out;
}
namespace {
clients::ClientConfig source_config() {
clients::ClientConfig c;
c.auth = nullptr;
c.persist_cookies = false;
return c;
}
} // namespace
HttpDocumentSource::HttpDocumentSource()
: client_(std::make_unique<clients::transports::HttpTransport>(), source_config()) {}
HttpDocumentSource::HttpDocumentSource(std::unique_ptr<clients::Transport> transport)
: client_(std::move(transport), source_config()) {}
std::string HttpDocumentSource::fetch(const std::string& url, std::size_t max_bytes) {
if (!is_google_host_url(url))
throw GoogleError::invalid("discovery url is not an https googleapis.com url: " + url);
try {
clients::ClientResponse res = client_.get_blocking(url);
if (!res.is_success())
throw GoogleError(res.status(), res.status() == 404 ? "NOT_FOUND" : "UNAVAILABLE",
"discovery fetch failed with HTTP " + std::to_string(res.status()), "discovery");
if (res.body().size() > max_bytes)
throw GoogleError(502, "BAD_DOCUMENT", "discovery document is larger than the cap", "discovery");
return res.body();
} catch (const GoogleError&) {
throw;
} catch (const std::exception& e) {
throw GoogleError(502, "UNAVAILABLE", e.what(), "discovery");
}
}
std::string FileDocumentSource::fetch(const std::string& url, std::size_t max_bytes) {
std::string file;
if (url.rfind(kDirectoryUrl, 0) == 0) {
file = "directory.json";
} else {
if (index_.empty()) {
std::ifstream din(dir_ + "/directory.json", std::ios::binary);
if (!din) throw GoogleError(404, "NOT_FOUND", "saved directory.json missing", "discovery");
std::ostringstream dss;
dss << din.rdbuf();
for (const DirectoryItem& i : Directory::parse(dss.str()).items())
index_[i.rest_url] = i.name + "_" + i.version + ".json";
}
auto hit = index_.find(url);
if (hit == index_.end()) throw GoogleError::invalid("no saved document for url: " + url);
file = hit->second;
}
std::ifstream in(dir_ + "/" + file, std::ios::binary);
if (!in) throw GoogleError(404, "NOT_FOUND", "saved discovery document missing: " + file, "discovery");
std::ostringstream ss;
ss << in.rdbuf();
std::string text = ss.str();
if (text.size() > max_bytes)
throw GoogleError(502, "BAD_DOCUMENT", "discovery document is larger than the cap", "discovery");
return text;
}
Catalog::Catalog(std::shared_ptr<DocumentSource> source, Clock clock)
: source_(std::move(source)), clock_(std::move(clock)) {}
std::chrono::steady_clock::time_point Catalog::now() const {
return clock_ ? clock_() : std::chrono::steady_clock::now();
}
std::shared_ptr<const Directory> Catalog::directory() {
std::lock_guard<std::mutex> lock(mu_);
auto t = now();
if (directory_ && t - directory_at_ < DiscoveryCaps::kMaxAge) {
++stats_.hits;
return directory_;
}
++stats_.fetches;
std::string text = source_->fetch(kDirectoryUrl, DiscoveryCaps::kMaxDirectoryBytes);
directory_ = std::make_shared<const Directory>(Directory::parse(text));
directory_at_ = t;
return directory_;
}
std::shared_ptr<const Api> Catalog::api(const std::string& ref) {
std::string name = ref, version;
if (std::size_t c = ref.find(':'); c != std::string::npos) {
name = ref.substr(0, c);
version = ref.substr(c + 1);
}
if (!valid_name(name) || (!version.empty() && !valid_name(version)))
throw GoogleError::invalid("api must be name or name:version, got '" + ref + "'");
std::shared_ptr<const Directory> dir = directory();
const DirectoryItem* item = dir->find(name, version);
if (!item) throw GoogleError(404, "NOT_FOUND", "no such API in the Discovery directory: " + ref, "discovery");
std::lock_guard<std::mutex> lock(mu_);
auto t = now();
auto it = entries_.find(item->id);
if (it != entries_.end()) {
bool stale = t - it->second.fetched >= DiscoveryCaps::kMaxAge;
auto lf = last_fetch_.find(item->id);
bool floor = lf != last_fetch_.end() && t - lf->second < DiscoveryCaps::kMinRefetch;
if (!stale || floor) {
lru_.remove(item->id);
lru_.push_front(item->id);
++stats_.hits;
return it->second.api;
}
}
++stats_.fetches;
std::string text = source_->fetch(item->rest_url, DiscoveryCaps::kMaxDocBytes);
Json doc;
try {
doc = Json::parse(text);
} catch (const std::exception& e) {
throw GoogleError(502, "BAD_DOCUMENT", std::string("discovery document ") + item->id + ": " + e.what());
}
Api parsed = Api::parse(doc);
parsed.raw_bytes = text.size();
if (parsed.name() != item->name || parsed.version() != item->version)
throw GoogleError(502, "BAD_DOCUMENT", "document " + parsed.id() + " is not the one requested: " + item->id);
Entry e;
e.api = std::make_shared<const Api>(std::move(parsed));
e.fetched = t;
e.bytes = text.size();
entries_[item->id] = e;
last_fetch_[item->id] = t;
lru_.remove(item->id);
lru_.push_front(item->id);
evict_locked();
return e.api;
}
void Catalog::evict_locked() {
auto total = [&] {
std::size_t b = 0;
for (const auto& [k, v] : entries_) b += v.bytes;
return b;
};
while (entries_.size() > 1 &&
(entries_.size() > DiscoveryCaps::kMaxCachedApis || total() > DiscoveryCaps::kMaxCachedBytes)) {
entries_.erase(lru_.back());
lru_.pop_back();
++stats_.evictions;
}
}
void Catalog::forget(const std::string& ref) {
std::lock_guard<std::mutex> lock(mu_);
std::string name = ref, version;
if (std::size_t c = ref.find(':'); c != std::string::npos) {
name = ref.substr(0, c);
version = ref.substr(c + 1);
}
for (auto& [id, e] : entries_)
if (e.api->name() == name && (version.empty() || e.api->version() == version))
e.fetched = std::chrono::steady_clock::time_point{};
}
CatalogStats Catalog::stats() const {
std::lock_guard<std::mutex> lock(mu_);
CatalogStats s = stats_;
s.cached = entries_.size();
for (const auto& [k, v] : entries_) s.cached_bytes += v.bytes;
return s;
}
} // namespace molo::google::discovery
+125
View File
@@ -0,0 +1,125 @@
#pragma once
// Directory of Google APIs plus a bounded cache of parsed Discovery
// documents. Documents come from a DocumentSource: HTTPS in production
// (no credential, googleapis.com hosts only), a directory of saved real
// documents for offline audits.
#include <chrono>
#include <functional>
#include <list>
#include <map>
#include <memory>
#include <mutex>
#include <string>
#include <vector>
#include "clients/client.h"
#include "google/discovery/model.h"
namespace molo::google::discovery {
struct DirectoryItem {
std::string id; // bigquery:v2
std::string name;
std::string version;
std::string title;
std::string description;
std::string rest_url; // discoveryRestUrl
std::string documentation;
bool preferred = false;
clients::body::Json to_json() const;
};
class Directory {
public:
// Parses the body of https://discovery.googleapis.com/discovery/v1/apis.
static Directory parse(const std::string& text);
const std::vector<DirectoryItem>& items() const { return items_; }
std::size_t preferred_count() const;
// Version empty: the preferred version. nullptr when absent.
const DirectoryItem* find(const std::string& name, const std::string& version = "") const;
// Case-insensitive substring over name and title, at most `limit` rows.
std::vector<const DirectoryItem*> search(const std::string& needle, std::size_t limit,
bool preferred_only = true) const;
private:
std::vector<DirectoryItem> items_;
};
class DocumentSource {
public:
virtual ~DocumentSource() = default;
// Returns the body. Throws GoogleError on failure; never more than max_bytes.
virtual std::string fetch(const std::string& url, std::size_t max_bytes) = 0;
};
// HTTPS GET without credentials; refuses any host outside googleapis.com.
class HttpDocumentSource : public DocumentSource {
public:
HttpDocumentSource();
explicit HttpDocumentSource(std::unique_ptr<clients::Transport> transport);
std::string fetch(const std::string& url, std::size_t max_bytes) override;
private:
clients::Client client_;
};
// Serves saved real documents: "<dir>/<name>_<version>.json" for a document
// URL, "<dir>/directory.json" for the directory.
class FileDocumentSource : public DocumentSource {
public:
explicit FileDocumentSource(std::string dir) : dir_(std::move(dir)) {}
std::string fetch(const std::string& url, std::size_t max_bytes) override;
private:
std::string dir_;
std::map<std::string, std::string> index_; // rest url -> file
};
extern const char* const kDirectoryUrl;
struct CatalogStats {
std::size_t cached = 0;
std::size_t cached_bytes = 0;
std::size_t fetches = 0;
std::size_t hits = 0;
std::size_t evictions = 0;
};
class Catalog {
public:
using Clock = std::function<std::chrono::steady_clock::time_point()>;
explicit Catalog(std::shared_ptr<DocumentSource> source, Clock clock = nullptr);
// Cached up to DiscoveryCaps::kMaxAge; refetch is never more often than
// kMinRefetch.
std::shared_ptr<const Directory> directory();
// `ref` is "name" (preferred version) or "name:version".
std::shared_ptr<const Api> api(const std::string& ref);
// Drops one cached document (subject to the kMinRefetch floor on refetch).
void forget(const std::string& ref);
CatalogStats stats() const;
private:
struct Entry {
std::shared_ptr<const Api> api;
std::chrono::steady_clock::time_point fetched;
std::size_t bytes = 0;
};
std::chrono::steady_clock::time_point now() const;
void evict_locked();
std::shared_ptr<DocumentSource> source_;
Clock clock_;
mutable std::mutex mu_;
std::shared_ptr<const Directory> directory_;
std::chrono::steady_clock::time_point directory_at_{};
std::list<std::string> lru_; // most recent first, keys are item ids
std::map<std::string, Entry> entries_;
std::map<std::string, std::chrono::steady_clock::time_point> last_fetch_;
CatalogStats stats_;
};
} // namespace molo::google::discovery
+228
View File
@@ -0,0 +1,228 @@
#include "google/discovery/engine.h"
#include <regex>
#include <thread>
#include "google/discovery/caps.h"
#include "google/errors.h"
namespace molo::google::discovery {
using clients::body::Json;
namespace {
[[noreturn]] void bad(const std::string& why) { throw GoogleError::invalid(why); }
std::string str(const Json& o, const char* key) {
const Json* v = o.find(key);
return v && v->is_string() ? v->as_string() : "";
}
} // namespace
Json CallResult::to_json() const {
Json o = Json::object();
o.set("body", body);
o.set("pages", Json(static_cast<long>(pages)));
o.set("truncated", Json(truncated));
o.set("operationWaited", Json(operation_waited));
o.set("itemsKey", Json(items_key));
o.set("support", Json(support_name(support)));
Json n = Json::array();
for (const std::string& s : notes) n.push(Json(s));
o.set("notes", n);
return o;
}
Engine::Engine(std::shared_ptr<GoogleClient> client, std::shared_ptr<Catalog> catalog,
Sleeper sleeper)
: client_(std::move(client)), catalog_(std::move(catalog)), sleeper_(std::move(sleeper)) {
if (!sleeper_)
sleeper_ = [](std::chrono::milliseconds d) { std::this_thread::sleep_for(d); };
}
Json Engine::list_apis(const std::string& filter, std::size_t limit, bool preferred_only) {
auto dir = catalog_->directory();
Json out = Json::array();
for (const DirectoryItem* i : dir->search(filter, limit, preferred_only)) out.push(i->to_json());
return out;
}
Json Engine::list_methods(const std::string& api_ref, const std::string& filter,
std::size_t limit) {
auto api = catalog_->api(api_ref);
limit = std::min(limit, DiscoveryCaps::kMaxListed);
Json out = Json::array();
std::size_t matched = 0;
for (const Method& m : api->methods()) {
if (!filter.empty() && m.id.find(filter) == std::string::npos) continue;
++matched;
if (out.as_array().size() < limit) out.push(m.summary());
}
Json o = Json::object();
o.set("api", Json(api->id()));
o.set("total", Json(static_cast<long>(api->methods().size())));
o.set("matched", Json(static_cast<long>(matched)));
o.set("truncated", Json(matched > out.as_array().size()));
o.set("methods", out);
return o;
}
const Method& Engine::method_of(const Api& api, const std::string& method_id) const {
const Method* m = api.find(method_id);
if (!m) throw GoogleError(404, "NOT_FOUND", "api " + api.id() + " has no method " + method_id, "discovery");
return *m;
}
Json Engine::describe(const std::string& api_ref, const std::string& method_id) {
auto api = catalog_->api(api_ref);
const Method& m = method_of(*api, method_id);
Json o = m.summary();
Json params = Json::array();
for (const auto& [n, p] : m.params) {
Json pj = Json::object();
pj.set("name", Json(n));
pj.set("in", Json(p.location));
pj.set("type", Json(p.type));
if (!p.format.empty()) pj.set("format", Json(p.format));
pj.set("required", Json(p.required));
pj.set("repeated", Json(p.repeated));
if (!p.pattern.empty()) pj.set("pattern", Json(p.pattern));
if (!p.enum_values.empty()) {
Json e = Json::array();
for (const std::string& v : p.enum_values) e.push(Json(v));
pj.set("enum", e);
}
params.push(pj);
}
o.set("parameters", params);
Json scopes = Json::array();
for (const std::string& s : m.scopes) scopes.push(Json(s));
o.set("scopes", scopes);
o.set("request", Json(m.request_ref));
o.set("response", Json(m.response_ref));
o.set("pageable", Json(api->is_pageable(m)));
o.set("itemsKey", Json(api->is_pageable(m) ? api->items_key(m) : ""));
o.set("operation", Json(!m.response_ref.empty() && api->is_operation_schema(m.response_ref)));
Json locs = Json::array();
for (const EndpointRow& e : api->endpoints()) locs.push(Json(e.location));
o.set("locations", locs);
return o;
}
PreparedCall Engine::plan(const std::string& api_ref, const std::string& method_id,
const CallArgs& args) {
auto api = catalog_->api(api_ref);
return Planner::plan(*api, method_of(*api, method_id), args);
}
Json Engine::send(const PreparedCall& p) {
return client_->request(p.http, p.endpoint, p.path, p.has_body ? &p.body : nullptr);
}
CallResult Engine::call(const std::string& api_ref, const std::string& method_id,
const CallArgs& args) {
if (args.max_pages == 0 || args.max_pages > DiscoveryCaps::kMaxPages)
bad("max_pages must be 1 to " + std::to_string(DiscoveryCaps::kMaxPages));
auto api = catalog_->api(api_ref);
const Method& m = method_of(*api, method_id);
PreparedCall first = Planner::plan(*api, m, args); // all validation, no traffic
if (args.max_pages > 1) {
if (!first.pageable) bad("method " + m.id + " is not pageable (needs a pageToken parameter and a nextPageToken reply)");
if (args.items_key.empty() && api->items_key(m).empty())
bad("method " + m.id + " has several list fields: name one with items_key");
}
if (!client_->configured()) throw NotConfiguredError(api->name());
CallResult out;
out.support = first.support;
out.notes = first.notes;
if (args.max_pages > 1) {
walk_pages(*api, m, args, first, out);
return out;
}
out.body = send(first);
if (args.wait && first.operation) {
out.body = wait_for(*api, first, args, out.body);
out.operation_waited = true;
}
return out;
}
void Engine::walk_pages(const Api& api, const Method& m, const CallArgs& args,
const PreparedCall& first, CallResult& out) {
std::string key = !args.items_key.empty() ? args.items_key : api.items_key(m);
out.items_key = key;
Json items = Json::array();
Json last = Json::object();
PreparedCall call = first;
std::size_t count = 0;
for (std::size_t page = 0; page < args.max_pages; ++page) {
last = send(call);
out.pages = page + 1;
if (const Json* arr = last.find(key); arr && arr->is_array())
for (const Json& item : arr->as_array()) {
if (count >= DiscoveryCaps::kMaxItems) {
out.truncated = true;
break;
}
items.push(item);
++count;
}
const Json* next = last.find("nextPageToken");
std::string token = next && next->is_string() ? next->as_string() : "";
if (token.empty()) break;
if (out.truncated || page + 1 == args.max_pages) {
out.truncated = true;
break;
}
CallArgs again = args;
again.params.set("pageToken", Json(token));
call = Planner::plan(api, m, again);
}
if (last.is_object()) {
last.set(key, items);
if (!out.truncated) last.as_object().erase("nextPageToken");
}
out.body = last;
}
Json Engine::wait_for(const Api& api, const PreparedCall& origin, const CallArgs& args,
const Json& op_in) {
Json op = op_in;
std::string name = str(op, "name");
if (name.empty()) bad("operation reply has no name");
const Method* getter = nullptr;
for (const Method* g : api.operation_getters()) {
const Param& p = g->params.at("name");
if (p.pattern.empty() ? api.operation_getters().size() == 1
: std::regex_search(name, std::regex(p.pattern))) {
getter = g;
break;
}
}
if (!getter)
bad("no operations.get method of " + api.id() + " accepts the operation name " + name);
CallArgs poll;
poll.params.set("name", Json(name));
poll.location = args.location;
poll.auth = args.auth;
PreparedCall get = Planner::plan(api, *getter, poll);
(void)origin;
std::chrono::milliseconds waited{0};
for (std::size_t i = 0; i < OperationCaps::kMaxPolls; ++i) {
Operation parsed = Operation::from_json(op);
if (parsed.done) {
if (parsed.failed())
throw GoogleError(502, "OPERATION_FAILED", parsed.error_message, api.name());
return op;
}
if (waited >= OperationCaps::kMaxWait) break;
sleeper_(OperationCaps::kInterval);
waited += OperationCaps::kInterval;
op = send(get);
}
throw GoogleError(504, "DEADLINE_EXCEEDED",
"operation still running at the wait cap: " + name, api.name());
}
} // namespace molo::google::discovery
+68
View File
@@ -0,0 +1,68 @@
#pragma once
// The Discovery engine: reach any method of any Google Discovery API by id.
// Engine e(client, catalog);
// e.call("bigquery", "bigquery.datasets.list", args);
// Documents are fetched without credentials and cached (Catalog); the call
// itself goes through GoogleClient with the credential the planner chose.
#include <functional>
#include <memory>
#include <string>
#include "google/client.h"
#include "google/discovery/catalog.h"
#include "google/discovery/planner.h"
#include "google/operation.h"
namespace molo::google::discovery {
struct CallResult {
clients::body::Json body;
std::size_t pages = 1;
bool truncated = false; // page or item cap hit with more pages left
bool operation_waited = false;
std::string items_key; // set when pages were merged
Support support = Support::kFull;
std::vector<std::string> notes;
clients::body::Json to_json() const;
};
class Engine {
public:
Engine(std::shared_ptr<GoogleClient> client, std::shared_ptr<Catalog> catalog,
Sleeper sleeper = nullptr);
bool configured() const { return client_->configured(); }
Catalog& catalog() { return *catalog_; }
// Directory rows as Json; limit is capped at DiscoveryCaps::kMaxListed.
clients::body::Json list_apis(const std::string& filter, std::size_t limit,
bool preferred_only = true);
// Methods of one API (optionally those whose id contains `filter`).
clients::body::Json list_methods(const std::string& api_ref, const std::string& filter,
std::size_t limit);
clients::body::Json describe(const std::string& api_ref, const std::string& method_id);
// Validation only: no call to Google (the document fetch is not a call to
// the API). Throws GoogleError::invalid on any problem.
PreparedCall plan(const std::string& api_ref, const std::string& method_id,
const CallArgs& args);
CallResult call(const std::string& api_ref, const std::string& method_id,
const CallArgs& args);
private:
const Method& method_of(const Api& api, const std::string& method_id) const;
clients::body::Json send(const PreparedCall& p);
clients::body::Json wait_for(const Api& api, const PreparedCall& origin,
const CallArgs& args, const clients::body::Json& op);
void walk_pages(const Api& api, const Method& m, const CallArgs& args,
const PreparedCall& first, CallResult& out);
std::shared_ptr<GoogleClient> client_;
std::shared_ptr<Catalog> catalog_;
Sleeper sleeper_;
};
} // namespace molo::google::discovery
+338
View File
@@ -0,0 +1,338 @@
#include "google/discovery/model.h"
#include <algorithm>
#include <cctype>
#include <set>
#include "google/discovery/caps.h"
#include "google/errors.h"
namespace molo::google::discovery {
using clients::body::Json;
namespace {
[[noreturn]] void bad_doc(const std::string& why) {
throw GoogleError(502, "BAD_DOCUMENT", "discovery document: " + why);
}
std::string str(const Json& o, const char* key) {
const Json* v = o.find(key);
return v && v->is_string() ? v->as_string() : "";
}
bool flag(const Json& o, const char* key) {
const Json* v = o.find(key);
return v && v->is_bool() && v->as_bool();
}
std::optional<double> num_or_string(const Json& o, const char* key) {
const Json* v = o.find(key);
if (!v) return std::nullopt;
if (v->is_number()) return v->as_number();
if (v->is_string()) {
try {
return std::stod(v->as_string());
} catch (...) {
}
}
return std::nullopt;
}
std::string ref_of(const Json& o, const char* key) {
const Json* v = o.find(key);
return v && v->is_object() ? str(*v, "$ref") : "";
}
Param parse_param(const std::string& name, const Json& p) {
Param out;
out.name = name;
out.type = str(p, "type");
out.format = str(p, "format");
out.location = str(p, "location");
out.pattern = str(p, "pattern");
out.description = str(p, "description");
out.required = flag(p, "required");
out.repeated = flag(p, "repeated");
out.deprecated = flag(p, "deprecated");
if (const Json* e = p.find("enum"); e && e->is_array())
for (const Json& v : e->as_array())
if (v.is_string()) out.enum_values.push_back(v.as_string());
out.minimum = num_or_string(p, "minimum");
out.maximum = num_or_string(p, "maximum");
return out;
}
// Names of {var} / {+var} in a path template, in order.
std::vector<std::string> template_vars(const std::string& path, bool* malformed) {
std::vector<std::string> out;
for (std::size_t i = 0; i < path.size(); ++i) {
if (path[i] == '}') {
*malformed = true;
return out;
}
if (path[i] != '{') continue;
std::size_t close = path.find('}', i);
if (close == std::string::npos) {
*malformed = true;
return out;
}
std::string v = path.substr(i + 1, close - i - 1);
if (!v.empty() && v[0] == '+') v.erase(0, 1);
if (v.empty() || v.find('{') != std::string::npos) {
*malformed = true;
return out;
}
out.push_back(v);
i = close;
}
return out;
}
void classify(Method& m) {
auto reject = [&](const std::string& why) {
m.support = Support::kRejected;
m.notes.push_back(why);
};
auto limit = [&](const std::string& why) {
if (m.support == Support::kFull) m.support = Support::kLimited;
m.notes.push_back(why);
};
static const std::set<std::string> verbs = {"GET", "POST", "PUT", "PATCH", "DELETE"};
if (!verbs.count(m.http_method)) reject("http method " + m.http_method + " is not supported");
if (m.streaming_type == "SERVER_SIDE")
limit("server-streaming method: the reply is read whole, no incremental delivery");
else if (!m.streaming_type.empty() && m.streaming_type != "NONE")
reject("streaming method (" + m.streaming_type + ") needs a socket, not a request");
bool malformed = false;
std::set<std::string> in_path;
for (const std::string& v : template_vars(m.path, &malformed)) in_path.insert(v);
if (malformed) reject("path template is malformed");
for (const std::string& v : in_path) {
auto it = m.params.find(v);
if (it == m.params.end()) reject("path variable " + v + " has no declared parameter");
else if (it->second.location != "path")
reject("path variable " + v + " is declared as " + it->second.location);
}
for (const auto& [name, p] : m.params) {
if (p.location != "path" && p.location != "query")
reject("parameter " + name + " has location " + p.location);
else if (p.location == "path" && !in_path.count(name))
reject("path parameter " + name + " is not in the path template");
if (p.type == "object") reject("parameter " + name + " is an object (map) parameter");
else if (p.type != "string" && p.type != "integer" && p.type != "number" &&
p.type != "boolean")
reject("parameter " + name + " has unknown type " + p.type);
}
if (m.media.supported)
limit(std::string("media upload (") + (m.media.simple ? "simple" : "") +
(m.media.simple && m.media.resumable ? ", " : "") +
(m.media.resumable ? "resumable" : "") +
") not implemented: only the metadata call is made");
if (m.supports_download) limit("media download (alt=media) not implemented: JSON reply only");
if (m.response_ref == "HttpBody") limit("reply is google.api.HttpBody: may not be JSON");
}
void collect(const Json& node, std::vector<Method>& out, std::size_t depth) {
if (depth > DiscoveryCaps::kMaxResourceDepth) bad_doc("resources nested too deep");
if (const Json* ms = node.find("methods"); ms && ms->is_object()) {
for (const auto& [mname, mj] : ms->as_object()) {
(void)mname;
if (!mj.is_object()) bad_doc("method is not an object");
if (out.size() >= DiscoveryCaps::kMaxMethods) bad_doc("more methods than the cap");
Method m;
m.id = str(mj, "id");
m.http_method = str(mj, "httpMethod");
m.path = str(mj, "path");
m.flat_path = str(mj, "flatPath");
if (m.id.empty() || m.path.empty()) bad_doc("method without id or path");
m.request_ref = ref_of(mj, "request");
m.response_ref = ref_of(mj, "response");
m.supports_download = flag(mj, "supportsMediaDownload");
m.deprecated = flag(mj, "deprecated");
m.streaming_type = str(mj, "streamingType");
if (const Json* sc = mj.find("scopes"); sc && sc->is_array())
for (const Json& s : sc->as_array())
if (s.is_string()) m.scopes.push_back(s.as_string());
if (const Json* ps = mj.find("parameters"); ps && ps->is_object())
for (const auto& [pn, pj] : ps->as_object())
if (pj.is_object()) m.params[pn] = parse_param(pn, pj);
if (const Json* po = mj.find("parameterOrder"); po && po->is_array())
for (const Json& s : po->as_array())
if (s.is_string()) m.param_order.push_back(s.as_string());
if (const Json* mu = mj.find("mediaUpload"); mu && mu->is_object()) {
m.media.supported = true;
if (const Json* pr = mu->find("protocols"); pr && pr->is_object()) {
m.media.simple = pr->has("simple");
m.media.resumable = pr->has("resumable");
}
if (auto sz = num_or_string(*mu, "maxSize")) m.media.max_size = *sz;
}
classify(m);
out.push_back(std::move(m));
}
}
if (const Json* rs = node.find("resources"); rs && rs->is_object())
for (const auto& [rn, rj] : rs->as_object()) {
(void)rn;
if (rj.is_object()) collect(rj, out, depth + 1);
}
}
} // namespace
const char* support_name(Support s) {
switch (s) {
case Support::kFull: return "full";
case Support::kLimited: return "limited";
case Support::kRejected: return "rejected";
}
return "rejected";
}
bool is_google_host_url(const std::string& url) {
static const std::string kScheme = "https://";
if (url.rfind(kScheme, 0) != 0) return false;
std::size_t end = url.find_first_of("/?#", kScheme.size());
std::string host = url.substr(kScheme.size(), end == std::string::npos
? std::string::npos
: end - kScheme.size());
if (host.empty() || host.find('@') != std::string::npos) return false;
if (std::size_t colon = host.find(':'); colon != std::string::npos) {
if (host.substr(colon) != ":443") return false;
host.erase(colon);
}
static const std::string kSuffix = ".googleapis.com";
if (host.size() <= kSuffix.size() ||
host.compare(host.size() - kSuffix.size(), kSuffix.size(), kSuffix) != 0)
return false;
for (char c : host)
if (!(std::isalnum(static_cast<unsigned char>(c)) || c == '.' || c == '-')) return false;
return true;
}
Json Method::summary() const {
Json o = Json::object();
o.set("id", Json(id));
o.set("httpMethod", Json(http_method));
o.set("path", Json(path));
o.set("support", Json(support_name(support)));
o.set("hasBody", Json(!request_ref.empty()));
o.set("oauth", Json(needs_oauth()));
o.set("deprecated", Json(deprecated));
Json req = Json::array();
for (const auto& [n, p] : params)
if (p.required) req.push(Json(n));
o.set("required", req);
Json notes_j = Json::array();
for (const std::string& n : notes) notes_j.push(Json(n));
o.set("notes", notes_j);
return o;
}
Api Api::parse(const Json& doc) {
if (!doc.is_object()) bad_doc("not an object");
if (str(doc, "kind") != "discovery#restDescription") bad_doc("kind is not discovery#restDescription");
if (str(doc, "protocol") != "rest") bad_doc("protocol is not rest");
Api api;
api.id_ = str(doc, "id");
api.name_ = str(doc, "name");
api.version_ = str(doc, "version");
api.title_ = str(doc, "title");
api.root_url_ = str(doc, "rootUrl");
api.service_path_ = str(doc, "servicePath");
api.batch_path_ = str(doc, "batchPath");
if (api.id_.empty() || api.name_.empty() || api.version_.empty()) bad_doc("missing id, name or version");
if (!is_google_host_url(api.root_url_)) bad_doc("rootUrl is not an https googleapis.com host: " + api.root_url_);
if (api.root_url_.back() != '/') bad_doc("rootUrl has no trailing slash");
if (api.service_path_.find_first_of("?#{}") != std::string::npos) bad_doc("servicePath is not a plain path");
if (const Json* eps = doc.find("endpoints"); eps && eps->is_array())
for (const Json& e : eps->as_array()) {
EndpointRow row{str(e, "location"), str(e, "endpointUrl"), str(e, "description")};
if (row.location.empty() || !is_google_host_url(row.url) || row.url.back() != '/')
bad_doc("endpoint row is not a googleapis.com root URL: " + row.url);
api.endpoints_.push_back(std::move(row));
}
if (const Json* ps = doc.find("parameters"); ps && ps->is_object())
for (const auto& [n, p] : ps->as_object()) {
(void)p;
api.global_params_.push_back(n);
}
if (const Json* s = doc.find("schemas"); s && s->is_object()) {
if (s->as_object().size() > DiscoveryCaps::kMaxSchemas) bad_doc("more schemas than the cap");
api.schemas_ = *s;
}
collect(doc, api.methods_, 0);
for (std::size_t i = 0; i < api.methods_.size(); ++i)
if (!api.index_.emplace(api.methods_[i].id, i).second)
bad_doc("duplicate method id " + api.methods_[i].id);
return api;
}
std::string Api::base_url() const {
std::string b = root_url_ + service_path_;
while (!b.empty() && b.back() == '/') b.pop_back();
return b;
}
const Method* Api::find(const std::string& method_id) const {
auto it = index_.find(method_id);
return it == index_.end() ? nullptr : &methods_[it->second];
}
const Json* Api::schema(const std::string& name) const { return schemas_.find(name); }
std::size_t Api::schema_count() const { return schemas_.as_object().size(); }
bool Api::is_operation_schema(const std::string& ref) const {
const Json* s = schema(ref);
if (!s) return false;
const Json* props = s->find("properties");
if (!props || !props->is_object()) return false;
const Json* done = props->find("done");
return props->has("name") && done && done->is_object() && str(*done, "type") == "boolean";
}
bool Api::is_pageable(const Method& m) const {
auto it = m.params.find("pageToken");
if (it == m.params.end() || it->second.location != "query") return false;
const Json* s = schema(m.response_ref);
if (!s) return false;
const Json* props = s->find("properties");
return props && props->is_object() && props->has("nextPageToken");
}
std::string Api::items_key(const Method& m) const {
const Json* s = schema(m.response_ref);
const Json* props = s ? s->find("properties") : nullptr;
if (!props || !props->is_object()) return "";
std::string found;
for (const auto& [n, p] : props->as_object()) {
if (n == "nextPageToken" || n == "unreachable") continue;
if (p.is_object() && str(p, "type") == "array") {
if (!found.empty()) return ""; // ambiguous: caller names it
found = n;
}
}
return found;
}
std::vector<const Method*> Api::operation_getters() const {
std::vector<const Method*> out;
static const std::string kSuffix = "operations.get";
for (const Method& m : methods_) {
if (m.id.size() < kSuffix.size() ||
m.id.compare(m.id.size() - kSuffix.size(), kSuffix.size(), kSuffix) != 0)
continue;
if (m.http_method != "GET" || m.support == Support::kRejected) continue;
if (!is_operation_schema(m.response_ref)) continue;
std::size_t required = 0;
bool name_path = m.params.count("name") && m.params.at("name").location == "path";
for (const auto& [n, p] : m.params) required += p.required ? 1 : 0;
if (name_path && required == 1) out.push_back(&m);
}
return out;
}
} // namespace molo::google::discovery
+118
View File
@@ -0,0 +1,118 @@
#pragma once
// Parsed Discovery document: parameters, methods, schemas. Pure data plus
// classification of what the engine can and cannot do with each method.
// Every shape here was read from the real documents (see SPEC.md section
// "Shapes seen"); anything else is rejected loudly by Api::parse.
#include <map>
#include <optional>
#include <string>
#include <vector>
#include "clients/body/json.h"
namespace molo::google::discovery {
// Full: called as documented. Limited: callable, but part of the method's
// surface is not reachable (media bytes, non-JSON reply); notes say which.
// Rejected: never called; the notes say why.
enum class Support { kFull, kLimited, kRejected };
const char* support_name(Support s);
struct Param {
std::string name;
std::string type; // string integer number boolean (object: rejected)
std::string format;
std::string location; // path or query
std::string pattern;
std::string description;
bool required = false;
bool repeated = false;
bool deprecated = false;
std::vector<std::string> enum_values;
std::optional<double> minimum;
std::optional<double> maximum;
};
struct MediaUpload {
bool supported = false;
bool simple = false;
bool resumable = false;
double max_size = 0; // bytes, 0: not stated
};
struct Method {
std::string id; // bigquery.jobs.query
std::string http_method; // GET POST PUT PATCH DELETE
std::string path; // v1/{+name}:cancel (relative to base)
std::string flat_path;
std::string request_ref; // schema name, empty: no body
std::string response_ref; // schema name, empty: no body
std::map<std::string, Param> params;
std::vector<std::string> param_order;
std::vector<std::string> scopes;
MediaUpload media;
bool supports_download = false;
bool deprecated = false;
std::string streaming_type;
Support support = Support::kFull;
std::vector<std::string> notes; // why Limited or Rejected, one per cause
bool needs_oauth() const { return !scopes.empty(); }
clients::body::Json summary() const;
};
struct EndpointRow {
std::string location;
std::string url; // root URL, trailing slash
std::string description;
};
class Api {
public:
// Throws GoogleError (502 BAD_DOCUMENT) when the document is not one the
// engine can trust: wrong kind/protocol, host outside googleapis.com,
// over the method cap, duplicate method ids.
static Api parse(const clients::body::Json& doc);
const std::string& id() const { return id_; }
const std::string& name() const { return name_; }
const std::string& version() const { return version_; }
const std::string& title() const { return title_; }
const std::string& root_url() const { return root_url_; }
const std::string& service_path() const { return service_path_; }
// root_url + service_path without trailing slash.
std::string base_url() const;
const std::vector<EndpointRow>& endpoints() const { return endpoints_; }
const std::vector<Method>& methods() const { return methods_; }
const Method* find(const std::string& method_id) const;
const clients::body::Json* schema(const std::string& name) const;
std::size_t schema_count() const;
// Query parameters every method accepts, as declared by the document.
const std::vector<std::string>& global_params() const { return global_params_; }
bool batch_path_declared() const { return !batch_path_.empty(); }
bool is_operation_schema(const std::string& ref) const;
bool is_pageable(const Method& m) const;
// The array property to collect when paging m, or "" when none/ambiguous.
std::string items_key(const Method& m) const;
// Methods ending in "operations.get" usable to poll an operation.
std::vector<const Method*> operation_getters() const;
// Raw bytes of the parsed text, for cache accounting.
std::size_t raw_bytes = 0;
private:
std::string id_, name_, version_, title_, root_url_, service_path_, batch_path_;
std::vector<EndpointRow> endpoints_;
std::vector<Method> methods_;
std::map<std::string, std::size_t> index_;
clients::body::Json schemas_ = clients::body::Json::object();
std::vector<std::string> global_params_;
};
// True for an https URL whose host is googleapis.com or a subdomain of it.
bool is_google_host_url(const std::string& url);
} // namespace molo::google::discovery
+207
View File
@@ -0,0 +1,207 @@
#include "google/discovery/planner.h"
#include <algorithm>
#include <set>
#include "clients/url.h"
#include "google/discovery/caps.h"
#include "google/discovery/validate.h"
#include "google/errors.h"
namespace molo::google::discovery {
using clients::body::Json;
namespace {
[[noreturn]] void bad(const std::string& why) { throw GoogleError::invalid(why); }
// Document-level query parameters a caller may set. Everything else the
// document declares globally (alt, key, access_token, oauth_token, callback,
// uploadType, upload_protocol, $.xgafv) either changes the reply format,
// carries a credential, or switches protocol: rejected by name.
bool allowed_global(const std::string& name) {
return name == "fields" || name == "quotaUser" || name == "prettyPrint";
}
} // namespace
const char* auth_mode_name(AuthMode m) {
switch (m) {
case AuthMode::kAuto: return "auto";
case AuthMode::kKey: return "key";
case AuthMode::kOAuth: return "oauth";
}
return "auto";
}
AuthMode auth_mode_from(const std::string& name) {
if (name.empty() || name == "auto") return AuthMode::kAuto;
if (name == "key") return AuthMode::kKey;
if (name == "oauth") return AuthMode::kOAuth;
bad("auth must be auto, key or oauth");
}
std::string Planner::encode_path_value(const std::string& value, bool reserved) {
if (!reserved) return clients::Url::encode_component(value);
std::string out;
std::size_t start = 0;
while (true) {
std::size_t end = value.find('/', start);
out += clients::Url::encode_component(
value.substr(start, end == std::string::npos ? std::string::npos : end - start));
if (end == std::string::npos) break;
out += '/';
start = end + 1;
}
return out;
}
Json PreparedCall::to_json() const {
Json o = Json::object();
o.set("api", Json(api_id));
o.set("method", Json(method_id));
o.set("httpMethod", Json(clients::method_name(http)));
o.set("url", Json(endpoint.base_url + path));
o.set("credential", Json(endpoint.prefer_key ? "key" : "oauth"));
o.set("hasBody", Json(has_body));
o.set("support", Json(support_name(support)));
o.set("pageable", Json(pageable));
o.set("operation", Json(operation));
Json n = Json::array();
for (const std::string& s : notes) n.push(Json(s));
o.set("notes", n);
return o;
}
PreparedCall Planner::plan(const Api& api, const Method& m, const CallArgs& args) {
if (m.support == Support::kRejected) {
std::string why;
for (const std::string& n : m.notes) why += (why.empty() ? "" : "; ") + n;
bad("method " + m.id + " cannot be called by this engine: " + why);
}
if (!args.params.is_object()) bad("params must be an object");
// Endpoint: the global host, or a documented regional one.
std::string root = api.root_url();
if (!args.location.empty()) {
auto it = std::find_if(api.endpoints().begin(), api.endpoints().end(),
[&](const EndpointRow& e) { return e.location == args.location; });
if (it == api.endpoints().end()) {
std::string known;
for (std::size_t i = 0; i < api.endpoints().size() && i < 8; ++i)
known += (i ? ", " : "") + api.endpoints()[i].location;
bad("api " + api.id() + " has no endpoint for location '" + args.location + "'" +
(known.empty() ? " (it documents no regional endpoints)" : " (e.g. " + known + ")"));
}
root = it->url;
}
std::string base = root + api.service_path();
while (!base.empty() && base.back() == '/') base.pop_back();
// Parameters.
std::set<std::string> seen;
std::vector<std::pair<std::string, std::string>> query;
std::map<std::string, std::string> path_values;
for (const auto& [name, value] : args.params.as_object()) {
auto it = m.params.find(name);
if (it == m.params.end()) {
const auto& g = api.global_params();
if (std::find(g.begin(), g.end(), name) != g.end()) {
if (!allowed_global(name))
bad("parameter " + name + " is reserved: it is set by the engine or changes the reply format");
Param p;
p.name = name;
p.location = "query";
p.type = name == "prettyPrint" ? "boolean" : "string";
for (const std::string& v : ParamValidator::values(p, value)) query.emplace_back(name, v);
seen.insert(name);
continue;
}
std::string known;
for (const auto& [n, p] : m.params) known += (known.empty() ? "" : ", ") + n;
bad("method " + m.id + " has no parameter '" + name + "' (accepts: " + known + ")");
}
const Param& p = it->second;
std::vector<std::string> vals = ParamValidator::values(p, value);
seen.insert(name);
if (p.location == "path") {
if (vals.size() != 1) bad("path parameter " + name + " takes one value");
path_values[name] = vals[0];
} else {
for (const std::string& v : vals) query.emplace_back(name, v);
}
}
for (const auto& [name, p] : m.params)
if (p.required && !seen.count(name)) bad("required parameter " + name + " is missing");
if (query.size() > DiscoveryCaps::kMaxQueryParams)
bad("more than " + std::to_string(DiscoveryCaps::kMaxQueryParams) + " query values");
// Path: substitute {x} and {+x}.
std::string path;
for (std::size_t i = 0; i < m.path.size(); ++i) {
if (m.path[i] != '{') {
path += m.path[i];
continue;
}
std::size_t close = m.path.find('}', i);
std::string var = m.path.substr(i + 1, close - i - 1);
bool reserved = !var.empty() && var[0] == '+';
if (reserved) var.erase(0, 1);
auto v = path_values.find(var);
if (v == path_values.end()) bad("required path parameter " + var + " is missing");
path += encode_path_value(v->second, reserved);
i = close;
}
std::string qs;
for (const auto& [k, v] : query)
qs += (qs.empty() ? "?" : "&") + clients::Url::encode_component(k) + "=" +
clients::Url::encode_component(v);
path = "/" + path + qs;
if (path.size() > DiscoveryCaps::kMaxPathChars)
bad("request target is longer than " + std::to_string(DiscoveryCaps::kMaxPathChars));
PreparedCall out;
out.api_id = api.id();
out.method_id = m.id;
out.support = m.support;
out.notes = m.notes;
out.path = path;
if (m.http_method == "GET") out.http = clients::Method::kGet;
else if (m.http_method == "POST") out.http = clients::Method::kPost;
else if (m.http_method == "PUT") out.http = clients::Method::kPut;
else if (m.http_method == "PATCH") out.http = clients::Method::kPatch;
else out.http = clients::Method::kDelete;
out.endpoint.name = api.name();
out.endpoint.base_url = base;
switch (args.auth) {
case AuthMode::kKey: out.endpoint.prefer_key = true; break;
case AuthMode::kOAuth: out.endpoint.prefer_key = false; break;
case AuthMode::kAuto: out.endpoint.prefer_key = !m.needs_oauth(); break;
}
out.oauth_scoped = m.needs_oauth();
// Body.
bool verb_has_body = out.http == clients::Method::kPost || out.http == clients::Method::kPut ||
out.http == clients::Method::kPatch;
if (m.request_ref.empty()) {
if (!args.body.is_null()) bad("method " + m.id + " takes no request body");
if (verb_has_body) {
out.body = Json::object();
out.has_body = true;
}
} else if (!verb_has_body) {
bad("method " + m.id + " declares a body on a " + m.http_method);
} else {
out.body = args.body.is_null() ? Json::object() : args.body;
BodyValidator(api).validate(m.request_ref, out.body);
out.has_body = true;
}
out.pageable = api.is_pageable(m);
out.operation = !m.response_ref.empty() && api.is_operation_schema(m.response_ref);
return out;
}
} // namespace molo::google::discovery
+59
View File
@@ -0,0 +1,59 @@
#pragma once
// Turns (Api, Method, arguments) into one validated HTTP request. No
// network traffic happens here: every rule that can fail does so before a
// byte is sent.
#include <string>
#include <vector>
#include "clients/body/json.h"
#include "clients/request.h"
#include "google/discovery/model.h"
#include "google/endpoints.h"
namespace molo::google::discovery {
// Which credential reaches Google. Auto: methods that declare no OAuth scope
// take the API key, every other method takes OAuth. Key / OAuth force the
// choice (the client falls back to the other credential only when the forced
// one is not configured).
enum class AuthMode { kAuto, kKey, kOAuth };
const char* auth_mode_name(AuthMode mode);
AuthMode auth_mode_from(const std::string& name); // throws invalid
struct CallArgs {
clients::body::Json params = clients::body::Json::object(); // name -> value
clients::body::Json body; // null: none
std::string location; // regional endpoint (Api::endpoints) or empty
AuthMode auth = AuthMode::kAuto;
std::size_t max_pages = 1; // > 1 walks nextPageToken (list methods only)
std::string items_key; // array to collect when paging, else derived
bool wait = false; // poll a long-running operation to completion
};
struct PreparedCall {
std::string api_id;
std::string method_id;
clients::Method http = clients::Method::kGet;
ServiceEndpoint endpoint;
std::string path; // "/v1/..." with query string
clients::body::Json body;
bool has_body = false;
Support support = Support::kFull;
std::vector<std::string> notes;
bool pageable = false;
bool operation = false; // the reply is a google.longrunning.Operation
bool oauth_scoped = false;
clients::body::Json to_json() const;
};
class Planner {
public:
static PreparedCall plan(const Api& api, const Method& method, const CallArgs& args);
// Percent-encodes one value for a path template ({+x} keeps '/').
static std::string encode_path_value(const std::string& value, bool reserved);
};
} // namespace molo::google::discovery
+189
View File
@@ -0,0 +1,189 @@
// Audit tool: reads saved Discovery documents (the same files the engine
// would fetch) and prints, as JSON, which APIs and methods the engine
// cannot handle or only handles in part, and why. Driven by
// check_discovery_directory.py; one classification, the engine's own.
// discovery-audit <dir with directory.json and <name>_<version>.json>
#include <cstdio>
#include <fstream>
#include <iostream>
#include <map>
#include <memory>
#include <sstream>
#include "google/discovery/catalog.h"
#include "google/discovery/model.h"
#include "google/discovery/planner.h"
#include "google/errors.h"
using namespace molo::google;
using namespace molo::google::discovery;
using molo::clients::body::Json;
namespace {
// A value that satisfies a parameter's declaration, for a dry plan.
Json synth(const Param& p) {
if (!p.enum_values.empty()) return Json(p.enum_values[0]);
if (p.type == "boolean") return Json(true);
if (p.type == "integer") {
long v = p.minimum ? static_cast<long>(*p.minimum) : 1;
return p.format == "int64" ? Json(std::to_string(v)) : Json(v);
}
if (p.type == "number") return Json(p.minimum ? *p.minimum : 1.0);
if (p.format == "int64" || p.format == "uint64") return Json("1");
if (p.format == "google-datetime" || p.format == "date-time") return Json("2026-10-25T10:00:00Z");
if (p.format == "google-duration") return Json("1s");
std::string s = p.pattern;
auto rep = [&](const std::string& a, const std::string& b) {
for (std::size_t i = s.find(a); i != std::string::npos; i = s.find(a, i + b.size()))
s.replace(i, a.size(), b);
};
if (s.empty()) return Json("x1");
rep("[^/]+", "x1");
rep("[^/]*", "x1");
rep("[^/]", "x");
if (!s.empty() && s.front() == '^') s.erase(0, 1);
if (!s.empty() && s.back() == '$') s.pop_back();
return Json(s);
}
Json synth_args(const Method& m) {
Json params = Json::object();
for (const auto& [n, p] : m.params) {
if (!p.required) continue;
Json v = synth(p);
if (p.repeated) {
Json a = Json::array();
a.push(v);
v = a;
}
params.set(n, v);
}
return params;
}
} // namespace
int main(int argc, char** argv) {
if (argc != 2) {
std::fprintf(stderr, "usage: discovery-audit <dir>\n");
return 2;
}
auto source = std::make_shared<FileDocumentSource>(argv[1]);
Catalog catalog(source);
Json report = Json::object();
Json apis = Json::array();
Json failed = Json::array();
std::map<std::string, long> reasons;
long methods = 0, full = 0, limited = 0, rejected = 0, pageable = 0, lro = 0, lro_unpollable = 0,
regional = 0, batch = 0, planned = 0, plan_skipped = 0, plan_failed = 0;
Json plan_failures = Json::array();
std::shared_ptr<const Directory> dir;
try {
dir = catalog.directory();
} catch (const std::exception& e) {
std::fprintf(stderr, "directory: %s\n", e.what());
return 1;
}
long preferred = 0;
for (const DirectoryItem& item : dir->items()) {
if (!item.preferred) continue;
++preferred;
std::shared_ptr<const Api> api;
try {
api = catalog.api(item.name + ":" + item.version);
} catch (const std::exception& e) {
Json f = Json::object();
f.set("api", Json(item.id));
f.set("why", Json(std::string(e.what())));
failed.push(f);
continue;
}
bool has_getter = !api->operation_getters().empty();
Json a = Json::object();
a.set("api", Json(api->id()));
long a_full = 0, a_limited = 0, a_rejected = 0;
Json problems = Json::array();
for (const Method& m : api->methods()) {
++methods;
if (m.support == Support::kFull) ++full, ++a_full;
if (m.support == Support::kLimited) ++limited, ++a_limited;
if (m.support == Support::kRejected) ++rejected, ++a_rejected;
if (api->is_pageable(m)) ++pageable;
bool op = !m.response_ref.empty() && api->is_operation_schema(m.response_ref);
if (op) {
++lro;
if (!has_getter) {
++lro_unpollable;
reasons["long-running reply but the API has no operations.get (wait unavailable)"]++;
}
}
if (m.support != Support::kRejected) {
CallArgs args;
args.params = synth_args(m);
try {
(void)Planner::plan(*api, m, args);
++planned;
} catch (const GoogleError& e) {
std::string why = e.message();
if (why.find("does not match") != std::string::npos) {
++plan_skipped; // synthetic value did not fit an exotic pattern
} else {
++plan_failed;
Json f = Json::object();
f.set("method", Json(m.id));
f.set("why", Json(why));
plan_failures.push(f);
}
}
}
for (const std::string& n : m.notes) {
// Reason class: strip parameter and path names so counts group.
std::string cls = n.substr(0, n.find(" (") == std::string::npos ? n.size() : n.find(" ("));
if (cls.rfind("parameter ", 0) == 0 || cls.rfind("path ", 0) == 0) cls = n;
reasons[std::string(support_name(m.support)) + ": " + cls]++;
}
if (m.support != Support::kFull) {
Json p = Json::object();
p.set("method", Json(m.id));
p.set("support", Json(support_name(m.support)));
Json notes = Json::array();
for (const std::string& n : m.notes) notes.push(Json(n));
p.set("notes", notes);
problems.push(p);
}
}
if (!api->endpoints().empty()) ++regional;
if (api->batch_path_declared()) ++batch;
a.set("methods", Json(static_cast<long>(api->methods().size())));
a.set("full", Json(a_full));
a.set("limited", Json(a_limited));
a.set("rejected", Json(a_rejected));
a.set("operationGetter", Json(has_getter));
a.set("regionalEndpoints", Json(static_cast<long>(api->endpoints().size())));
a.set("problems", problems);
apis.push(a);
}
report.set("preferred", Json(preferred));
report.set("loaded", Json(static_cast<long>(preferred) - static_cast<long>(failed.as_array().size())));
report.set("failed", failed);
report.set("methods", Json(methods));
report.set("full", Json(full));
report.set("limited", Json(limited));
report.set("rejected", Json(rejected));
report.set("pageable", Json(pageable));
report.set("longRunning", Json(lro));
report.set("longRunningWithoutGetter", Json(lro_unpollable));
report.set("apisWithRegionalEndpoints", Json(regional));
report.set("apisDeclaringBatchPath", Json(batch));
Json rs = Json::object();
for (const auto& [k, v] : reasons) rs.set(k, Json(v));
report.set("reasons", rs);
report.set("planned", Json(planned));
report.set("planSkippedPattern", Json(plan_skipped));
report.set("planFailed", Json(plan_failed));
report.set("planFailures", plan_failures);
report.set("apis", apis);
std::cout << report.dump() << "\n";
return 0;
}
@@ -0,0 +1,152 @@
#!/usr/bin/env python3
"""Walk Google's whole Discovery directory and report what the engine cannot do.
python3 src/google/discovery/tools/check_discovery_directory.py [--cache DIR]
[--no-fetch] [--json FILE] [--max-rejected N] [--workers N]
1. Fetches https://discovery.googleapis.com/discovery/v1/apis and the
document of every preferred API into --cache (default
build/google/discovery-cache). Fetch failures are listed as UNREACHABLE:
Google lists them but does not serve them (not an engine fault).
2. Builds build/google/discovery-audit (src/google/discovery/tools/audit_main.cc)
against libgoogle.a and runs it on the saved files, so the verdict comes
from the engine's own classification, never from a second implementation.
3. Prints totals, a reason table and the APIs with rejected or limited
methods. Exit 1 when a fetched document does not parse, or when rejected
methods exceed --max-rejected (default: no ceiling).
python3 standard library only.
"""
import argparse
import concurrent.futures as cf
import json
import os
import subprocess
import sys
import urllib.request
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "..", ".."))
DIRECTORY = "https://discovery.googleapis.com/discovery/v1/apis"
AUDIT_SRC = os.path.join(ROOT, "src/google/discovery/tools/audit_main.cc")
AUDIT_BIN = os.path.join(ROOT, "build/google/discovery-audit")
LIBS = ["build/google/libgoogle.a", "build/clients/libclients.a", "build/liboperability.a"]
def get(url, tries=3):
last = None
for _ in range(tries):
try:
with urllib.request.urlopen(url, timeout=90) as r:
return r.read()
except Exception as e: # noqa: BLE001 - report whatever the network said
last = e
raise last
def fetch_all(cache, workers):
os.makedirs(cache, exist_ok=True)
raw = get(DIRECTORY)
with open(os.path.join(cache, "directory.json"), "wb") as f:
f.write(raw)
items = [i for i in json.loads(raw)["items"] if i.get("preferred")]
unreachable = []
def one(item):
path = os.path.join(cache, "%s_%s.json" % (item["name"], item["version"]))
try:
data = get(item["discoveryRestUrl"])
with open(path, "wb") as f:
f.write(data)
return None
except Exception as e: # noqa: BLE001
if os.path.exists(path):
os.remove(path)
return (item["id"], str(e))
with cf.ThreadPoolExecutor(workers) as ex:
for r in ex.map(one, items):
if r:
unreachable.append(r)
return len(items), unreachable
def build_audit():
subprocess.run(["make", "-s", "-j8", "google", "build/clients/libclients.a",
"build/liboperability.a"], cwd=ROOT, check=True)
cmd = ["g++", "-std=c++23", "-Wall", "-Wextra", "-Werror", "-DMOLO_HAS_WOLFSSL",
"-I" + os.path.join(ROOT, "src"), AUDIT_SRC] + [os.path.join(ROOT, l) for l in LIBS] + \
["-lwolfssl", "-o", AUDIT_BIN]
subprocess.run(cmd, cwd=ROOT, check=True)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--cache", default=os.path.join(ROOT, "build/google/discovery-cache"))
ap.add_argument("--no-fetch", action="store_true", help="reuse the files already in --cache")
ap.add_argument("--json", help="write the full audit report here")
ap.add_argument("--max-rejected", type=int, default=None)
ap.add_argument("--workers", type=int, default=16)
ap.add_argument("--list", type=int, default=40, help="APIs to list with problems")
args = ap.parse_args()
unreachable = []
listed = None
if not args.no_fetch:
listed, unreachable = fetch_all(args.cache, args.workers)
build_audit()
out = subprocess.run([AUDIT_BIN, args.cache], capture_output=True, text=True)
if out.returncode != 0:
print(out.stderr, file=sys.stderr)
return 1
rep = json.loads(out.stdout)
if args.json:
with open(args.json, "w") as f:
json.dump(rep, f, indent=1)
unreach_ids = {u[0] for u in unreachable}
parse_failures = [f for f in rep["failed"] if f["api"] not in unreach_ids]
gone = [f for f in rep["failed"] if f["api"] in unreach_ids or "NOT_FOUND" in f["why"]]
print("preferred APIs in the directory : %d" % rep["preferred"])
print("documents loaded by the engine : %d" % rep["loaded"])
print("UNREACHABLE at the source : %d" % len(gone))
for f in gone:
print(" %-40s %s" % (f["api"], f["why"][:90]))
parse_failures = [f for f in parse_failures if f not in gone]
print("documents the engine REJECTS : %d" % len(parse_failures))
for f in parse_failures:
print(" %-40s %s" % (f["api"], f["why"][:160]))
print("methods : %d" % rep["methods"])
print(" full / limited / rejected : %d / %d / %d" % (rep["full"], rep["limited"], rep["rejected"]))
print(" pageable (pageToken) : %d" % rep["pageable"])
print(" long-running replies : %d (%d in APIs with no operations.get)" %
(rep["longRunning"], rep["longRunningWithoutGetter"]))
print("dry plans (required params only): %d planned, %d skipped (synthetic value vs exotic pattern), %d FAILED" %
(rep["planned"], rep["planSkippedPattern"], rep["planFailed"]))
for f in rep["planFailures"][:20]:
print(" %s: %s" % (f["method"], f["why"][:140]))
print("APIs with regional endpoints : %d" % rep["apisWithRegionalEndpoints"])
print("APIs declaring a batchPath : %d (batch endpoints are not used)" % rep["apisDeclaringBatchPath"])
print("\nreasons (methods):")
for k, v in sorted(rep["reasons"].items(), key=lambda kv: -kv[1]):
print(" %6d %s" % (v, k))
bad = [a for a in rep["apis"] if a["rejected"] or a["limited"]]
bad.sort(key=lambda a: -(a["rejected"] * 1000 + a["limited"]))
print("\nAPIs with rejected or limited methods: %d of %d" % (len(bad), len(rep["apis"])))
for a in bad[:args.list]:
sample = a["problems"][0]
print(" %-34s rejected %-3d limited %-3d e.g. %s: %s" %
(a["api"], a["rejected"], a["limited"], sample["method"], sample["notes"][0][:70]))
if len(bad) > args.list:
print(" ... %d more (use --json)" % (len(bad) - args.list))
code = 0
if parse_failures or rep["planFailed"]:
code = 1
if args.max_rejected is not None and rep["rejected"] > args.max_rejected:
print("FAIL: %d rejected methods exceed --max-rejected %d" % (rep["rejected"], args.max_rejected))
code = 1
return code
if __name__ == "__main__":
sys.exit(main())
+220
View File
@@ -0,0 +1,220 @@
#include "google/discovery/validate.h"
#include <algorithm>
#include <cmath>
#include <regex>
#include "google/discovery/caps.h"
#include "google/errors.h"
namespace molo::google::discovery {
using clients::body::Json;
namespace {
[[noreturn]] void bad(const std::string& why) { throw GoogleError::invalid(why); }
std::string sval(const Json& o, const char* key) {
const Json* v = o.find(key);
return v && v->is_string() ? v->as_string() : "";
}
bool all_digits(const std::string& s, bool allow_sign) {
std::size_t i = allow_sign && !s.empty() && s[0] == '-' ? 1 : 0;
if (i >= s.size() || s.size() - i > 20) return false;
for (; i < s.size(); ++i)
if (s[i] < '0' || s[i] > '9') return false;
return true;
}
bool integral(double d) { return std::isfinite(d) && d == std::floor(d) && std::fabs(d) < 9.0e15; }
std::string int_text(double d) { return std::to_string(static_cast<long long>(d)); }
bool shape_ok(const std::string& format, const std::string& v) {
auto digit = [](char c) { return c >= '0' && c <= '9'; };
if (format == "date-time" || format == "google-datetime") {
// 2026-10-25T10:00:00Z shape: YYYY-MM-DDT
return v.size() >= 11 && digit(v[0]) && digit(v[1]) && digit(v[2]) && digit(v[3]) &&
v[4] == '-' && v[7] == '-' && (v[10] == 'T' || v[10] == 't');
}
if (format == "google-duration") {
if (v.size() < 2 || v.back() != 's') return false;
bool dot = false;
for (std::size_t i = v[0] == '-' ? 1 : 0; i + 1 < v.size(); ++i) {
if (v[i] == '.' && !dot) dot = true;
else if (!digit(v[i])) return false;
}
return v.size() > 1;
}
return true;
}
void control_free(const std::string& what, const std::string& v) {
for (unsigned char c : v)
if (c < 0x20 || c == 0x7f) bad(what + " contains a control character");
}
bool pattern_ok(const std::string& pattern, const std::string& v) {
try {
std::regex re(pattern, std::regex::ECMAScript | std::regex::optimize);
return std::regex_search(v, re);
} catch (const std::regex_error&) {
return true; // pattern syntax std::regex cannot read: Google checks it
}
}
std::string one(const Param& p, const Json& v) {
const std::string who = "parameter " + p.name;
std::string out;
if (p.type == "string") {
if (v.is_string()) out = v.as_string();
else if (v.is_number() && (p.format == "int64" || p.format == "uint64") && integral(v.as_number()))
out = int_text(v.as_number());
else bad(who + " must be a string");
if (p.format == "int64" && !all_digits(out, true)) bad(who + " must be a 64-bit integer");
if (p.format == "uint64" && !all_digits(out, false)) bad(who + " must be an unsigned 64-bit integer");
if (p.format == "uint64" && out.size() > 20) bad(who + " is too large");
if (!shape_ok(p.format, out)) bad(who + " is not a valid " + p.format);
} else if (p.type == "integer") {
double d;
if (v.is_number()) d = v.as_number();
else if (v.is_string() && all_digits(v.as_string(), true)) d = std::stod(v.as_string());
else bad(who + " must be an integer");
if (!integral(d)) bad(who + " must be an integer");
if (p.format == "int32" && (d < -2147483648.0 || d > 2147483647.0)) bad(who + " is outside int32");
if (p.format == "uint32" && (d < 0 || d > 4294967295.0)) bad(who + " is outside uint32");
if (p.minimum && d < *p.minimum) bad(who + " is below the minimum " + int_text(*p.minimum));
if (p.maximum && d > *p.maximum) bad(who + " is above the maximum " + int_text(*p.maximum));
out = int_text(d);
} else if (p.type == "number") {
if (!v.is_number() || !std::isfinite(v.as_number())) bad(who + " must be a number");
if (p.minimum && v.as_number() < *p.minimum) bad(who + " is below the minimum");
if (p.maximum && v.as_number() > *p.maximum) bad(who + " is above the maximum");
Json tmp(v.as_number());
out = tmp.dump();
} else if (p.type == "boolean") {
if (v.is_bool()) out = v.as_bool() ? "true" : "false";
else if (v.is_string() && (v.as_string() == "true" || v.as_string() == "false")) out = v.as_string();
else bad(who + " must be true or false");
} else {
bad(who + " has an unsupported type " + p.type);
}
if (out.size() > DiscoveryCaps::kMaxParamChars) bad(who + " is longer than the cap");
control_free(who, out);
if (!p.enum_values.empty() &&
std::find(p.enum_values.begin(), p.enum_values.end(), out) == p.enum_values.end())
bad(who + " is not one of its enum values");
if (!p.pattern.empty() && !pattern_ok(p.pattern, out)) bad(who + " does not match " + p.pattern);
if (p.location == "path") {
if (out.empty()) bad(who + " is empty");
// No dot segments: a path value must not climb out of its template.
std::size_t start = 0;
while (start <= out.size()) {
std::size_t end = out.find('/', start);
std::string seg = out.substr(start, end == std::string::npos ? std::string::npos : end - start);
if (seg == "." || seg == "..") bad(who + " contains a dot segment");
if (end == std::string::npos) break;
start = end + 1;
}
}
return out;
}
} // namespace
std::vector<std::string> ParamValidator::values(const Param& p, const Json& value) {
if (value.is_null()) bad("parameter " + p.name + " is null");
std::vector<std::string> out;
if (value.is_array()) {
if (!p.repeated) bad("parameter " + p.name + " is not repeated, got an array");
if (value.as_array().size() > DiscoveryCaps::kMaxRepeated)
bad("parameter " + p.name + " has more than " +
std::to_string(DiscoveryCaps::kMaxRepeated) + " values");
for (const Json& v : value.as_array()) out.push_back(one(p, v));
return out;
}
if (value.is_object()) bad("parameter " + p.name + " must be a scalar");
out.push_back(one(p, value));
return out;
}
struct BodyValidator::Walk {
std::size_t visits = 0;
};
void BodyValidator::validate(const std::string& ref, const Json& body) const {
if (body.dump().size() > DiscoveryCaps::kMaxBodyBytes) bad("request body is larger than the cap");
const Json* schema = api_.schema(ref);
if (!schema) bad("unknown request schema " + ref);
Walk walk;
node(*schema, body, ref, walk, 0);
}
void BodyValidator::node(const Json& schema_in, const Json& value, const std::string& at,
Walk& walk, std::size_t depth) const {
if (++walk.visits > DiscoveryCaps::kMaxSchemaVisits) bad("request body is too large to validate");
if (depth > DiscoveryCaps::kMaxSchemaDepth)
bad("request body is nested deeper than " + std::to_string(DiscoveryCaps::kMaxSchemaDepth));
const Json* schema = &schema_in;
for (int hops = 0; schema->find("$ref"); ++hops) {
if (hops > 16) bad("schema $ref chain too long at " + at);
const Json* next = api_.schema(sval(*schema, "$ref"));
if (!next) bad("unknown schema " + sval(*schema, "$ref") + " at " + at);
schema = next;
}
if (value.is_null()) return; // proto JSON: null is the unset value
std::string type = sval(*schema, "type");
if (type.empty() && schema->find("properties")) type = "object";
std::string format = sval(*schema, "format");
if (type == "any" || type.empty()) return;
if (type == "string") {
std::string text;
if (value.is_string()) text = value.as_string();
else if (value.is_number() && (format == "int64" || format == "uint64") && integral(value.as_number()))
text = int_text(value.as_number());
else bad(at + " must be a string");
if ((format == "int64" && !all_digits(text, true)) || (format == "uint64" && !all_digits(text, false)))
bad(at + " must be a " + format + " number");
if (const Json* e = schema->find("enum"); e && e->is_array()) {
bool hit = false;
for (const Json& v : e->as_array()) hit = hit || (v.is_string() && v.as_string() == text);
if (!hit) bad(at + " is not one of its enum values: " + text);
}
} else if (type == "integer") {
if (!value.is_number() || !integral(value.as_number())) bad(at + " must be an integer");
double d = value.as_number();
if (format == "int32" && (d < -2147483648.0 || d > 2147483647.0)) bad(at + " is outside int32");
if (format == "uint32" && (d < 0 || d > 4294967295.0)) bad(at + " is outside uint32");
} else if (type == "number") {
if (!value.is_number()) bad(at + " must be a number");
} else if (type == "boolean") {
if (!value.is_bool()) bad(at + " must be true or false");
} else if (type == "array") {
if (!value.is_array()) bad(at + " must be an array");
const Json* items = schema->find("items");
std::size_t i = 0;
for (const Json& v : value.as_array()) {
if (items && items->is_object()) node(*items, v, at + "/" + std::to_string(i), walk, depth + 1);
++i;
}
} else if (type == "object") {
if (!value.is_object()) bad(at + " must be an object");
const Json* props = schema->find("properties");
const Json* extra = schema->find("additionalProperties");
bool has_props = props && props->is_object() && !props->as_object().empty();
bool has_extra = extra && extra->is_object();
for (const auto& [k, v] : value.as_object()) {
const Json* ps = has_props ? props->find(k) : nullptr;
if (ps) node(*ps, v, at + "/" + k, walk, depth + 1);
else if (has_extra) node(*extra, v, at + "/" + k, walk, depth + 1);
else if (has_props) bad(at + " has no property " + k);
// neither: free-form object, anything goes
}
} else {
bad(at + " has unsupported schema type " + type);
}
}
} // namespace molo::google::discovery
+38
View File
@@ -0,0 +1,38 @@
#pragma once
// Validation before traffic: parameter values against their Discovery
// declaration, request bodies against the document's schemas. Every failure
// throws GoogleError::invalid naming the parameter or JSON path.
#include <string>
#include <vector>
#include "clients/body/json.h"
#include "google/discovery/model.h"
namespace molo::google::discovery {
class ParamValidator {
public:
// Canonical string form(s) of a value: one per element for repeated
// parameters. Checks type, format shape, enum, pattern, minimum, maximum,
// length and the repeated cap.
static std::vector<std::string> values(const Param& param,
const clients::body::Json& value);
};
class BodyValidator {
public:
explicit BodyValidator(const Api& api) : api_(api) {}
// Validates `body` against the schema named `ref`. Unknown properties,
// wrong types, values outside an enum, depth and size overruns throw.
void validate(const std::string& ref, const clients::body::Json& body) const;
private:
struct Walk;
void node(const clients::body::Json& schema, const clients::body::Json& value,
const std::string& at, Walk& walk, std::size_t depth) const;
const Api& api_;
};
} // namespace molo::google::discovery