Merge branch 'worktree-agent-a2e298643b4505a43'
This commit is contained in:
@@ -0,0 +1,213 @@
|
||||
# Google Discovery Engine - Specification
|
||||
|
||||
Owned directory: `src/google/discovery/`. Namespace `molo::google::discovery`.
|
||||
Depends on the `src/google` foundation (`GoogleClient`, `GoogleError`,
|
||||
`ServiceEndpoint`, `Operation`) and `clients::Client` for document fetches.
|
||||
No other google package depends on it and it depends on none. Plan:
|
||||
`plans/18-breadth.md` section 1.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
One generic engine that reaches every API in Google's Discovery directory
|
||||
(`https://discovery.googleapis.com/discovery/v1/apis`: 538 entries, 316
|
||||
preferred, 2026-10-25) and calls any method by its Discovery id
|
||||
(`serviceusage.services.list`). It fetches and caches a Discovery document,
|
||||
lists APIs and methods, validates parameters and body against the document
|
||||
before any traffic, builds the URL (path templates, query, repeated values,
|
||||
regional host), chooses key or OAuth, walks `nextPageToken`, and polls
|
||||
long-running operations. Typed classes in other packages are better for the
|
||||
APIs they cover; this engine is the breadth.
|
||||
|
||||
Not here (each is classified Limited or Rejected, see section 5): media
|
||||
upload and download bodies, resumable upload, bidirectional or client
|
||||
streaming, non-JSON replies, the global batch endpoint (`batchPath`, shut
|
||||
down by Google in 2020), and object-typed query parameters.
|
||||
|
||||
## 2. Module map (one concern per file)
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings (`DiscoveryCaps`) |
|
||||
| `model.h/.cpp` | `Api`, `Method`, `Param`: parse a document, classify every method (`Support` Full / Limited / Rejected with notes), pagination and operation detection, host allowlist |
|
||||
| `validate.h/.cpp` | `ParamValidator` (type, format, enum, pattern, range, repeated, path safety) and `BodyValidator` (schema walk: `$ref`, objects, maps, arrays, enums, int64 strings, free-form) |
|
||||
| `planner.h/.cpp` | `Planner::plan`: `(Api, Method, CallArgs)` to `PreparedCall` (endpoint, encoded path and query, body, credential choice). No traffic |
|
||||
| `catalog.h/.cpp` | `Directory`, `DocumentSource` (`HttpDocumentSource`, `FileDocumentSource`), `Catalog` (bounded LRU cache with age and refetch floor) |
|
||||
| `engine.h/.cpp` | `Engine`: `list_apis`, `list_methods`, `describe`, `plan`, `call` (paging, long-running wait) |
|
||||
| `tools/check_discovery_directory.py` | Fetches the whole directory and every preferred document, runs the engine's own classification over them, reports what it cannot handle |
|
||||
| `tools/audit_main.cc` | The C++ half of that checker (built by the script; not globbed by `google.mk`) |
|
||||
|
||||
## 3. Static caps (see caps.h)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Document size | 8 MiB (largest real: compute v1, 6.4 MB) |
|
||||
| Directory size / items | 2 MiB / 1000 |
|
||||
| Cached documents / cached bytes | 32 / 96 MiB, least recently used evicted |
|
||||
| Document max age | 24 h |
|
||||
| Refetch floor | 60 s per document, even after `forget` |
|
||||
| Methods per document | 10000 (largest real: compute v1, 1018; aiplatform v1, 1137) |
|
||||
| Resource nesting | 16 |
|
||||
| Parameter value | 2048 chars, no control characters |
|
||||
| Repeated values / query values | 100 / 64 |
|
||||
| Request target | 6000 chars |
|
||||
| Request body | 4 MiB, schema depth 32, 200000 visited nodes |
|
||||
| Pages per call / items per call | 50 / 10000 (default 1 page) |
|
||||
| Listing | 500 rows |
|
||||
| Operation wait | foundation `OperationCaps`: 400 polls, 2 s interval, 600 s |
|
||||
|
||||
## 4. Contracts
|
||||
|
||||
- Documents are fetched without a credential, over HTTPS, from hosts that
|
||||
are `googleapis.com` or a subdomain only (`is_google_host_url`). A document
|
||||
whose `rootUrl` or regional `endpointUrl` fails that test is rejected
|
||||
(`BAD_DOCUMENT`), as is a document that is not the one the directory
|
||||
listed, a wrong `kind` or `protocol`, a duplicate method id, or anything
|
||||
over a cap.
|
||||
- API reference: `name` (preferred version) or `name:version`. The method id
|
||||
prefix is not always the API name (`admin` serves `reports.activities.list`),
|
||||
so the API is always given explicitly.
|
||||
- Validation order, all before traffic: API and method exist (404), method
|
||||
not Rejected, `params` is an object, regional `location` exists in the
|
||||
document, every param is declared (global params: only `fields`,
|
||||
`quotaUser`, `prettyPrint`; `key`, `access_token`, `oauth_token`, `alt`,
|
||||
`callback`, `uploadType`, `upload_protocol`, `$.xgafv` are refused by name),
|
||||
values fit type / format / enum / pattern / range, required params are
|
||||
present, path values have no dot segment, the body fits the request schema.
|
||||
Failures are `GoogleError` 400 `INVALID_ARGUMENT` naming the parameter or
|
||||
JSON path. After validation, no credential is `NotConfiguredError` (503).
|
||||
- Path templates are `{x}` (value fully percent-encoded, `/` included) or
|
||||
`{+x}` (reserved expansion: each `/` segment encoded, slashes kept).
|
||||
Query parameters are sorted by name, repeated parameters repeat the key.
|
||||
- Body: methods with no request schema take none; POST / PUT / PATCH without
|
||||
a request schema or without a body send `{}`. Unknown properties are
|
||||
rejected when the schema lists properties; free-form objects and `any`
|
||||
pass; `null` passes (proto JSON unset); int64 / uint64 are digit strings
|
||||
(an integral JSON number is accepted and converted).
|
||||
- Credential: `AuthMode::kAuto` sends the API key for methods that declare no
|
||||
OAuth scope (205 of 13194) and OAuth for the rest; `kKey` / `kOAuth` force
|
||||
it through `ServiceEndpoint::prefer_key`. `GoogleClient` falls back to the
|
||||
other credential only when the forced one is not configured.
|
||||
- Regional hosts: `location` selects the `endpoints[]` row of the document
|
||||
(69 documents have rows, e.g. `workflows.us-central1.rep.googleapis.com`).
|
||||
An unknown location is rejected with examples; a document without rows
|
||||
rejects any location. Nothing is guessed from the location string.
|
||||
- Pagination: `max_pages > 1` needs a `pageToken` query param and a reply
|
||||
schema with `nextPageToken`. The array to collect is the single array
|
||||
property of the reply (`items_key` otherwise). The merged reply has the
|
||||
collected array; `truncated` is true when a page or item cap stopped the
|
||||
walk while a token remained (the token stays in the body).
|
||||
- Long-running: a reply whose schema has `name` and boolean `done` is an
|
||||
operation. With `wait`, the engine finds the API's `operations.get` method
|
||||
whose `name` pattern matches, polls it with the foundation caps, throws
|
||||
`OPERATION_FAILED` on an error result and `DEADLINE_EXCEEDED` at the cap.
|
||||
APIs without an `operations.get` (111 methods) reject `wait` loudly.
|
||||
Operations that use another shape (Compute `status` / `selfLink`) are
|
||||
plain resources to this engine.
|
||||
- Destructive or sending methods are ordinary methods here; the caller (the
|
||||
demo) decides. The demo runs GET at once and dry-runs every other verb
|
||||
unless `confirm:true`.
|
||||
|
||||
## 5. Shapes seen
|
||||
|
||||
Read from the 310 reachable preferred documents (13194 methods, 27285
|
||||
schemas), 2026-10-25, and how each is handled:
|
||||
|
||||
| Shape | Seen | Handling |
|
||||
|---|---|---|
|
||||
| `path` templates `{x}` / `{+x}` only; every variable declared as a path param | 4712 / 10563 variables, no exception | Substituted and encoded; anything else Rejected |
|
||||
| `flatPath` | 13034 methods | Ignored: `path` is authoritative and equivalent |
|
||||
| `rootUrl` + `servicePath`; 12 documents have a non-empty `servicePath` | 310 | `base_url` = root (or regional root) + servicePath; `path` is relative to it |
|
||||
| Param `location` | `path`, `query` only | Anything else Rejected |
|
||||
| Param types | string, integer, boolean, number; `object` once (`discoveryengine ...dataConnector.mcp` `extensions`) | Object params Rejected (1 method) |
|
||||
| Param `format` | int32, uint32, int64, uint64, double, float, byte, date-time, google-datetime, google-duration, google-fieldmask | Checked for integers and the 64-bit strings, date and duration shape; others are strings |
|
||||
| `repeated` | 509 params | Array or scalar; cap 100 |
|
||||
| `enum` | 710 params | Checked |
|
||||
| `pattern` | 12636 params | `std::regex` ECMAScript; a pattern it cannot parse is left to Google |
|
||||
| `minimum` / `maximum` (strings) | 306 / 61 | Checked |
|
||||
| `$ref` request/response, nested `$ref`, `items`, `additionalProperties` (map types), `any`, readOnly | all | Body validator follows `$ref` (chain cap 16), maps, arrays; readOnly is not enforced |
|
||||
| `mediaUpload` (simple, resumable) | 53 methods | Limited: the metadata call is made, no media bytes |
|
||||
| `supportsMediaDownload` | 20 methods | Limited: JSON reply only, `alt=media` is refused |
|
||||
| Reply `HttpBody` | 81 methods | Limited: the reply may not be JSON; a parse failure is `BAD_RESPONSE` |
|
||||
| `streamingType` | `NONE` 4, `SERVER_SIDE` 2 (admob) | NONE ignored; SERVER_SIDE Limited (read whole); anything else Rejected |
|
||||
| `batchPath` | 310 documents | Not used (Google removed the global batch endpoint) |
|
||||
| `endpoints[]` | 69 documents | Regional hosts, see section 4 |
|
||||
| `mtlsRootUrl`, `canonicalName`, `version_module`, `icons`, `auth` | many | Ignored |
|
||||
| Operation-shaped replies (`name` + `done`) | 2359 methods | `wait`; 111 are in documents with no `operations.get` |
|
||||
| `pageToken` + `nextPageToken` | 2386 methods | Paging |
|
||||
| Other paging schemes (`start-index`, `nextLink`) | not counted | Not paged; one call per request |
|
||||
|
||||
## Required APIs
|
||||
|
||||
Enable these on the project (placeholder project `example-214512124`; full
|
||||
list in `src/google/APIS.md`). The engine itself needs none: Discovery
|
||||
documents are public. Calling an API needs that API enabled; a disabled one
|
||||
answers 403 `SERVICE_DISABLED`. The live checks use:
|
||||
|
||||
- `serviceusage.googleapis.com`: https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=example-214512124
|
||||
|
||||
Scopes: `cloud-platform` covers the Cloud APIs. Workspace and other APIs
|
||||
need the scopes named in each method's `scopes` (`describe` shows them); the
|
||||
service-account credential in this repository supplies `cloud-platform` only.
|
||||
|
||||
## Sources
|
||||
|
||||
- `https://discovery.googleapis.com/discovery/v1/apis` (directory, 538
|
||||
entries, 316 preferred) and every preferred `discoveryRestUrl`, fetched
|
||||
2026-10-25.
|
||||
- Discovery format reference: `https://developers.google.com/discovery/v1/reference/apis`
|
||||
and `.../v1/using` (parameter and method fields).
|
||||
|
||||
## Verified against discovery
|
||||
|
||||
`python3 src/google/discovery/tools/check_discovery_directory.py`,
|
||||
2026-10-25: 316 preferred APIs listed, 310 documents served and loaded by the
|
||||
engine, 0 rejected documents. Six listed APIs are not served by Google
|
||||
(`area120tables:v1alpha1`, `datalabeling:v1beta1`, `mybusinessqanda:v1` 404;
|
||||
`contentwarehouse:v1`, `poly:v1`, `rapidmigrationassessment:v1` 502): not an
|
||||
engine fault, listed by the script as UNREACHABLE. Of 13194 methods: 13037
|
||||
Full, 156 Limited (81 HttpBody replies, 53 media upload, 20 media download,
|
||||
2 server streaming), 1 Rejected (object query parameter). Dry plan of every
|
||||
callable method with synthesized required parameters: 13185 planned, 8
|
||||
skipped (synthetic value does not fit an exotic pattern), 0 failed.
|
||||
|
||||
## Verified live
|
||||
|
||||
`make google-live` as the service account on project `lisa-448004`,
|
||||
2026-10-25 (read-only):
|
||||
|
||||
- Real directory over HTTPS: 538 items, 316 preferred; documents of
|
||||
serviceusage, cloudresourcemanager, tasks, workflows parsed.
|
||||
- Largest documents fit the caps: compute v1 (1018 methods, 6419010 bytes),
|
||||
aiplatform v1 (1137 methods, 4027902 bytes).
|
||||
- `serviceusage.services.get` for `serviceusage.googleapis.com` answered 200.
|
||||
- `serviceusage.services.list` with `pageSize=3`, 3 pages: 9 items,
|
||||
`truncated` true (the page cap, the token left in the body).
|
||||
- `serviceusage.operations.get` with a missing name: 400 INVALID_ARGUMENT
|
||||
(Google validates the name shape); the reply is recognised as an operation.
|
||||
- Cloud Resource Manager answers `PERMISSION_DENIED` (API not enabled on the
|
||||
project): reported by Google, not exercised.
|
||||
|
||||
## Unverified
|
||||
|
||||
- Long-running `wait` polling to completion: no safe read-only operation
|
||||
exists on this project, so only detection and the `operations.get` lookup
|
||||
are exercised live. The loop is the foundation's `wait_operation` logic
|
||||
re-implemented over the planner.
|
||||
- Key credential on a no-scope method live: no API key with a keyless-capable
|
||||
API was used; the choice is verified at the request level (header).
|
||||
- Regional endpoints are verified for URL construction only, not by a call.
|
||||
- Methods Limited or Rejected are never called by design.
|
||||
- Behaviour for the 8 methods whose required-parameter patterns the audit
|
||||
could not synthesize is only the validation path.
|
||||
|
||||
## Gaps
|
||||
|
||||
- Media upload (simple, resumable), media download and non-JSON replies:
|
||||
`Method::support` says Limited; the foundation `GoogleClient` is JSON only
|
||||
(`src/google/client.h`).
|
||||
- Batch (`batchPath`) is not offered.
|
||||
- Paging schemes other than `pageToken` / `nextPageToken`.
|
||||
- `AuthMode::kOAuth` falls back to the key when no OAuth credential exists
|
||||
(foundation `GoogleClient::pick_auth`).
|
||||
- Documents are cached in memory only; there is no on-disk cache.
|
||||
- Test and audit coverage of `wait` against a real running operation.
|
||||
@@ -0,0 +1,44 @@
|
||||
#pragma once
|
||||
|
||||
// Static ceilings for the Discovery engine. Never autotuned. Measured
|
||||
// against the 310 reachable preferred documents on 2026-10-25: largest
|
||||
// document 6.4 MB (compute v1), most methods in one document is below
|
||||
// kMaxMethods, directory 538 entries.
|
||||
|
||||
#include <chrono>
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
struct DiscoveryCaps {
|
||||
// Documents and cache.
|
||||
static constexpr std::size_t kMaxDocBytes = 8u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxDirectoryBytes = 2u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxDirectoryItems = 1000;
|
||||
static constexpr std::size_t kMaxCachedApis = 32;
|
||||
static constexpr std::size_t kMaxCachedBytes = 96u * 1024u * 1024u;
|
||||
static constexpr std::chrono::seconds kMaxAge{24 * 3600};
|
||||
// Floor: a document is never refetched more often than this, even when
|
||||
// the caller forces a refresh.
|
||||
static constexpr std::chrono::seconds kMinRefetch{60};
|
||||
// Document structure.
|
||||
static constexpr std::size_t kMaxMethods = 10000;
|
||||
static constexpr std::size_t kMaxResourceDepth = 16;
|
||||
static constexpr std::size_t kMaxSchemas = 20000;
|
||||
// Listing.
|
||||
static constexpr std::size_t kMaxListed = 500;
|
||||
// Request validation.
|
||||
static constexpr std::size_t kMaxParamChars = 2048;
|
||||
static constexpr std::size_t kMaxRepeated = 100;
|
||||
static constexpr std::size_t kMaxQueryParams = 64;
|
||||
static constexpr std::size_t kMaxPathChars = 6000;
|
||||
static constexpr std::size_t kMaxBodyBytes = 4u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxSchemaDepth = 32;
|
||||
static constexpr std::size_t kMaxSchemaVisits = 200000;
|
||||
// Pagination.
|
||||
static constexpr std::size_t kDefaultPages = 1;
|
||||
static constexpr std::size_t kMaxPages = 50;
|
||||
static constexpr std::size_t kMaxItems = 10000;
|
||||
};
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,285 @@
|
||||
#include "google/discovery/catalog.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cctype>
|
||||
#include <fstream>
|
||||
#include <sstream>
|
||||
|
||||
#include "clients/transports/http_transport.h"
|
||||
#include "google/discovery/caps.h"
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
const char* const kDirectoryUrl = "https://discovery.googleapis.com/discovery/v1/apis";
|
||||
|
||||
namespace {
|
||||
|
||||
std::string str(const Json& o, const char* key) {
|
||||
const Json* v = o.find(key);
|
||||
return v && v->is_string() ? v->as_string() : "";
|
||||
}
|
||||
|
||||
std::string lower(std::string s) {
|
||||
for (char& c : s) c = static_cast<char>(std::tolower(static_cast<unsigned char>(c)));
|
||||
return s;
|
||||
}
|
||||
|
||||
bool valid_name(const std::string& s) {
|
||||
if (s.empty() || s.size() > 64) return false;
|
||||
for (char c : s)
|
||||
if (!(std::isalnum(static_cast<unsigned char>(c)) || c == '_' || c == '-' || c == '.')) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Json DirectoryItem::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("id", Json(id));
|
||||
o.set("name", Json(name));
|
||||
o.set("version", Json(version));
|
||||
o.set("title", Json(title));
|
||||
o.set("description", Json(description));
|
||||
o.set("discoveryRestUrl", Json(rest_url));
|
||||
o.set("documentationLink", Json(documentation));
|
||||
o.set("preferred", Json(preferred));
|
||||
return o;
|
||||
}
|
||||
|
||||
Directory Directory::parse(const std::string& text) {
|
||||
if (text.size() > DiscoveryCaps::kMaxDirectoryBytes)
|
||||
throw GoogleError(502, "BAD_DOCUMENT", "discovery directory is larger than the cap");
|
||||
Json doc;
|
||||
try {
|
||||
doc = Json::parse(text);
|
||||
} catch (const std::exception& e) {
|
||||
throw GoogleError(502, "BAD_DOCUMENT", std::string("discovery directory: ") + e.what());
|
||||
}
|
||||
const Json* items = doc.is_object() ? doc.find("items") : nullptr;
|
||||
if (!items || !items->is_array() || str(doc, "kind") != "discovery#directoryList")
|
||||
throw GoogleError(502, "BAD_DOCUMENT", "discovery directory: not a directoryList");
|
||||
if (items->as_array().size() > DiscoveryCaps::kMaxDirectoryItems)
|
||||
throw GoogleError(502, "BAD_DOCUMENT", "discovery directory: more items than the cap");
|
||||
Directory dir;
|
||||
for (const Json& it : items->as_array()) {
|
||||
DirectoryItem row;
|
||||
row.id = str(it, "id");
|
||||
row.name = str(it, "name");
|
||||
row.version = str(it, "version");
|
||||
row.title = str(it, "title");
|
||||
row.description = str(it, "description");
|
||||
row.rest_url = str(it, "discoveryRestUrl");
|
||||
row.documentation = str(it, "documentationLink");
|
||||
const Json* p = it.find("preferred");
|
||||
row.preferred = p && p->is_bool() && p->as_bool();
|
||||
if (row.id.empty() || row.name.empty() || row.version.empty() ||
|
||||
!is_google_host_url(row.rest_url))
|
||||
throw GoogleError(502, "BAD_DOCUMENT", "discovery directory: bad item " + row.id);
|
||||
dir.items_.push_back(std::move(row));
|
||||
}
|
||||
return dir;
|
||||
}
|
||||
|
||||
std::size_t Directory::preferred_count() const {
|
||||
return static_cast<std::size_t>(std::count_if(items_.begin(), items_.end(),
|
||||
[](const DirectoryItem& i) { return i.preferred; }));
|
||||
}
|
||||
|
||||
const DirectoryItem* Directory::find(const std::string& name, const std::string& version) const {
|
||||
const DirectoryItem* fallback = nullptr;
|
||||
for (const DirectoryItem& i : items_) {
|
||||
if (i.name != name) continue;
|
||||
if (!version.empty()) {
|
||||
if (i.version == version) return &i;
|
||||
} else if (i.preferred) {
|
||||
return &i;
|
||||
} else if (!fallback) {
|
||||
fallback = &i;
|
||||
}
|
||||
}
|
||||
return version.empty() ? fallback : nullptr;
|
||||
}
|
||||
|
||||
std::vector<const DirectoryItem*> Directory::search(const std::string& needle, std::size_t limit,
|
||||
bool preferred_only) const {
|
||||
std::vector<const DirectoryItem*> out;
|
||||
std::string n = lower(needle);
|
||||
limit = std::min(limit, DiscoveryCaps::kMaxListed);
|
||||
for (const DirectoryItem& i : items_) {
|
||||
if (out.size() >= limit) break;
|
||||
if (preferred_only && !i.preferred) continue;
|
||||
if (n.empty() || lower(i.name).find(n) != std::string::npos ||
|
||||
lower(i.title).find(n) != std::string::npos)
|
||||
out.push_back(&i);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
namespace {
|
||||
clients::ClientConfig source_config() {
|
||||
clients::ClientConfig c;
|
||||
c.auth = nullptr;
|
||||
c.persist_cookies = false;
|
||||
return c;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
HttpDocumentSource::HttpDocumentSource()
|
||||
: client_(std::make_unique<clients::transports::HttpTransport>(), source_config()) {}
|
||||
|
||||
HttpDocumentSource::HttpDocumentSource(std::unique_ptr<clients::Transport> transport)
|
||||
: client_(std::move(transport), source_config()) {}
|
||||
|
||||
std::string HttpDocumentSource::fetch(const std::string& url, std::size_t max_bytes) {
|
||||
if (!is_google_host_url(url))
|
||||
throw GoogleError::invalid("discovery url is not an https googleapis.com url: " + url);
|
||||
try {
|
||||
clients::ClientResponse res = client_.get_blocking(url);
|
||||
if (!res.is_success())
|
||||
throw GoogleError(res.status(), res.status() == 404 ? "NOT_FOUND" : "UNAVAILABLE",
|
||||
"discovery fetch failed with HTTP " + std::to_string(res.status()), "discovery");
|
||||
if (res.body().size() > max_bytes)
|
||||
throw GoogleError(502, "BAD_DOCUMENT", "discovery document is larger than the cap", "discovery");
|
||||
return res.body();
|
||||
} catch (const GoogleError&) {
|
||||
throw;
|
||||
} catch (const std::exception& e) {
|
||||
throw GoogleError(502, "UNAVAILABLE", e.what(), "discovery");
|
||||
}
|
||||
}
|
||||
|
||||
std::string FileDocumentSource::fetch(const std::string& url, std::size_t max_bytes) {
|
||||
std::string file;
|
||||
if (url.rfind(kDirectoryUrl, 0) == 0) {
|
||||
file = "directory.json";
|
||||
} else {
|
||||
if (index_.empty()) {
|
||||
std::ifstream din(dir_ + "/directory.json", std::ios::binary);
|
||||
if (!din) throw GoogleError(404, "NOT_FOUND", "saved directory.json missing", "discovery");
|
||||
std::ostringstream dss;
|
||||
dss << din.rdbuf();
|
||||
for (const DirectoryItem& i : Directory::parse(dss.str()).items())
|
||||
index_[i.rest_url] = i.name + "_" + i.version + ".json";
|
||||
}
|
||||
auto hit = index_.find(url);
|
||||
if (hit == index_.end()) throw GoogleError::invalid("no saved document for url: " + url);
|
||||
file = hit->second;
|
||||
}
|
||||
std::ifstream in(dir_ + "/" + file, std::ios::binary);
|
||||
if (!in) throw GoogleError(404, "NOT_FOUND", "saved discovery document missing: " + file, "discovery");
|
||||
std::ostringstream ss;
|
||||
ss << in.rdbuf();
|
||||
std::string text = ss.str();
|
||||
if (text.size() > max_bytes)
|
||||
throw GoogleError(502, "BAD_DOCUMENT", "discovery document is larger than the cap", "discovery");
|
||||
return text;
|
||||
}
|
||||
|
||||
Catalog::Catalog(std::shared_ptr<DocumentSource> source, Clock clock)
|
||||
: source_(std::move(source)), clock_(std::move(clock)) {}
|
||||
|
||||
std::chrono::steady_clock::time_point Catalog::now() const {
|
||||
return clock_ ? clock_() : std::chrono::steady_clock::now();
|
||||
}
|
||||
|
||||
std::shared_ptr<const Directory> Catalog::directory() {
|
||||
std::lock_guard<std::mutex> lock(mu_);
|
||||
auto t = now();
|
||||
if (directory_ && t - directory_at_ < DiscoveryCaps::kMaxAge) {
|
||||
++stats_.hits;
|
||||
return directory_;
|
||||
}
|
||||
++stats_.fetches;
|
||||
std::string text = source_->fetch(kDirectoryUrl, DiscoveryCaps::kMaxDirectoryBytes);
|
||||
directory_ = std::make_shared<const Directory>(Directory::parse(text));
|
||||
directory_at_ = t;
|
||||
return directory_;
|
||||
}
|
||||
|
||||
std::shared_ptr<const Api> Catalog::api(const std::string& ref) {
|
||||
std::string name = ref, version;
|
||||
if (std::size_t c = ref.find(':'); c != std::string::npos) {
|
||||
name = ref.substr(0, c);
|
||||
version = ref.substr(c + 1);
|
||||
}
|
||||
if (!valid_name(name) || (!version.empty() && !valid_name(version)))
|
||||
throw GoogleError::invalid("api must be name or name:version, got '" + ref + "'");
|
||||
std::shared_ptr<const Directory> dir = directory();
|
||||
const DirectoryItem* item = dir->find(name, version);
|
||||
if (!item) throw GoogleError(404, "NOT_FOUND", "no such API in the Discovery directory: " + ref, "discovery");
|
||||
std::lock_guard<std::mutex> lock(mu_);
|
||||
auto t = now();
|
||||
auto it = entries_.find(item->id);
|
||||
if (it != entries_.end()) {
|
||||
bool stale = t - it->second.fetched >= DiscoveryCaps::kMaxAge;
|
||||
auto lf = last_fetch_.find(item->id);
|
||||
bool floor = lf != last_fetch_.end() && t - lf->second < DiscoveryCaps::kMinRefetch;
|
||||
if (!stale || floor) {
|
||||
lru_.remove(item->id);
|
||||
lru_.push_front(item->id);
|
||||
++stats_.hits;
|
||||
return it->second.api;
|
||||
}
|
||||
}
|
||||
++stats_.fetches;
|
||||
std::string text = source_->fetch(item->rest_url, DiscoveryCaps::kMaxDocBytes);
|
||||
Json doc;
|
||||
try {
|
||||
doc = Json::parse(text);
|
||||
} catch (const std::exception& e) {
|
||||
throw GoogleError(502, "BAD_DOCUMENT", std::string("discovery document ") + item->id + ": " + e.what());
|
||||
}
|
||||
Api parsed = Api::parse(doc);
|
||||
parsed.raw_bytes = text.size();
|
||||
if (parsed.name() != item->name || parsed.version() != item->version)
|
||||
throw GoogleError(502, "BAD_DOCUMENT", "document " + parsed.id() + " is not the one requested: " + item->id);
|
||||
Entry e;
|
||||
e.api = std::make_shared<const Api>(std::move(parsed));
|
||||
e.fetched = t;
|
||||
e.bytes = text.size();
|
||||
entries_[item->id] = e;
|
||||
last_fetch_[item->id] = t;
|
||||
lru_.remove(item->id);
|
||||
lru_.push_front(item->id);
|
||||
evict_locked();
|
||||
return e.api;
|
||||
}
|
||||
|
||||
void Catalog::evict_locked() {
|
||||
auto total = [&] {
|
||||
std::size_t b = 0;
|
||||
for (const auto& [k, v] : entries_) b += v.bytes;
|
||||
return b;
|
||||
};
|
||||
while (entries_.size() > 1 &&
|
||||
(entries_.size() > DiscoveryCaps::kMaxCachedApis || total() > DiscoveryCaps::kMaxCachedBytes)) {
|
||||
entries_.erase(lru_.back());
|
||||
lru_.pop_back();
|
||||
++stats_.evictions;
|
||||
}
|
||||
}
|
||||
|
||||
void Catalog::forget(const std::string& ref) {
|
||||
std::lock_guard<std::mutex> lock(mu_);
|
||||
std::string name = ref, version;
|
||||
if (std::size_t c = ref.find(':'); c != std::string::npos) {
|
||||
name = ref.substr(0, c);
|
||||
version = ref.substr(c + 1);
|
||||
}
|
||||
for (auto& [id, e] : entries_)
|
||||
if (e.api->name() == name && (version.empty() || e.api->version() == version))
|
||||
e.fetched = std::chrono::steady_clock::time_point{};
|
||||
}
|
||||
|
||||
CatalogStats Catalog::stats() const {
|
||||
std::lock_guard<std::mutex> lock(mu_);
|
||||
CatalogStats s = stats_;
|
||||
s.cached = entries_.size();
|
||||
for (const auto& [k, v] : entries_) s.cached_bytes += v.bytes;
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,125 @@
|
||||
#pragma once
|
||||
|
||||
// Directory of Google APIs plus a bounded cache of parsed Discovery
|
||||
// documents. Documents come from a DocumentSource: HTTPS in production
|
||||
// (no credential, googleapis.com hosts only), a directory of saved real
|
||||
// documents for offline audits.
|
||||
|
||||
#include <chrono>
|
||||
#include <functional>
|
||||
#include <list>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/client.h"
|
||||
#include "google/discovery/model.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
struct DirectoryItem {
|
||||
std::string id; // bigquery:v2
|
||||
std::string name;
|
||||
std::string version;
|
||||
std::string title;
|
||||
std::string description;
|
||||
std::string rest_url; // discoveryRestUrl
|
||||
std::string documentation;
|
||||
bool preferred = false;
|
||||
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
class Directory {
|
||||
public:
|
||||
// Parses the body of https://discovery.googleapis.com/discovery/v1/apis.
|
||||
static Directory parse(const std::string& text);
|
||||
const std::vector<DirectoryItem>& items() const { return items_; }
|
||||
std::size_t preferred_count() const;
|
||||
// Version empty: the preferred version. nullptr when absent.
|
||||
const DirectoryItem* find(const std::string& name, const std::string& version = "") const;
|
||||
// Case-insensitive substring over name and title, at most `limit` rows.
|
||||
std::vector<const DirectoryItem*> search(const std::string& needle, std::size_t limit,
|
||||
bool preferred_only = true) const;
|
||||
|
||||
private:
|
||||
std::vector<DirectoryItem> items_;
|
||||
};
|
||||
|
||||
class DocumentSource {
|
||||
public:
|
||||
virtual ~DocumentSource() = default;
|
||||
// Returns the body. Throws GoogleError on failure; never more than max_bytes.
|
||||
virtual std::string fetch(const std::string& url, std::size_t max_bytes) = 0;
|
||||
};
|
||||
|
||||
// HTTPS GET without credentials; refuses any host outside googleapis.com.
|
||||
class HttpDocumentSource : public DocumentSource {
|
||||
public:
|
||||
HttpDocumentSource();
|
||||
explicit HttpDocumentSource(std::unique_ptr<clients::Transport> transport);
|
||||
std::string fetch(const std::string& url, std::size_t max_bytes) override;
|
||||
|
||||
private:
|
||||
clients::Client client_;
|
||||
};
|
||||
|
||||
// Serves saved real documents: "<dir>/<name>_<version>.json" for a document
|
||||
// URL, "<dir>/directory.json" for the directory.
|
||||
class FileDocumentSource : public DocumentSource {
|
||||
public:
|
||||
explicit FileDocumentSource(std::string dir) : dir_(std::move(dir)) {}
|
||||
std::string fetch(const std::string& url, std::size_t max_bytes) override;
|
||||
|
||||
private:
|
||||
std::string dir_;
|
||||
std::map<std::string, std::string> index_; // rest url -> file
|
||||
};
|
||||
|
||||
extern const char* const kDirectoryUrl;
|
||||
|
||||
struct CatalogStats {
|
||||
std::size_t cached = 0;
|
||||
std::size_t cached_bytes = 0;
|
||||
std::size_t fetches = 0;
|
||||
std::size_t hits = 0;
|
||||
std::size_t evictions = 0;
|
||||
};
|
||||
|
||||
class Catalog {
|
||||
public:
|
||||
using Clock = std::function<std::chrono::steady_clock::time_point()>;
|
||||
explicit Catalog(std::shared_ptr<DocumentSource> source, Clock clock = nullptr);
|
||||
|
||||
// Cached up to DiscoveryCaps::kMaxAge; refetch is never more often than
|
||||
// kMinRefetch.
|
||||
std::shared_ptr<const Directory> directory();
|
||||
// `ref` is "name" (preferred version) or "name:version".
|
||||
std::shared_ptr<const Api> api(const std::string& ref);
|
||||
// Drops one cached document (subject to the kMinRefetch floor on refetch).
|
||||
void forget(const std::string& ref);
|
||||
CatalogStats stats() const;
|
||||
|
||||
private:
|
||||
struct Entry {
|
||||
std::shared_ptr<const Api> api;
|
||||
std::chrono::steady_clock::time_point fetched;
|
||||
std::size_t bytes = 0;
|
||||
};
|
||||
std::chrono::steady_clock::time_point now() const;
|
||||
void evict_locked();
|
||||
|
||||
std::shared_ptr<DocumentSource> source_;
|
||||
Clock clock_;
|
||||
mutable std::mutex mu_;
|
||||
std::shared_ptr<const Directory> directory_;
|
||||
std::chrono::steady_clock::time_point directory_at_{};
|
||||
std::list<std::string> lru_; // most recent first, keys are item ids
|
||||
std::map<std::string, Entry> entries_;
|
||||
std::map<std::string, std::chrono::steady_clock::time_point> last_fetch_;
|
||||
CatalogStats stats_;
|
||||
};
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,228 @@
|
||||
#include "google/discovery/engine.h"
|
||||
|
||||
#include <regex>
|
||||
#include <thread>
|
||||
|
||||
#include "google/discovery/caps.h"
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void bad(const std::string& why) { throw GoogleError::invalid(why); }
|
||||
|
||||
std::string str(const Json& o, const char* key) {
|
||||
const Json* v = o.find(key);
|
||||
return v && v->is_string() ? v->as_string() : "";
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Json CallResult::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("body", body);
|
||||
o.set("pages", Json(static_cast<long>(pages)));
|
||||
o.set("truncated", Json(truncated));
|
||||
o.set("operationWaited", Json(operation_waited));
|
||||
o.set("itemsKey", Json(items_key));
|
||||
o.set("support", Json(support_name(support)));
|
||||
Json n = Json::array();
|
||||
for (const std::string& s : notes) n.push(Json(s));
|
||||
o.set("notes", n);
|
||||
return o;
|
||||
}
|
||||
|
||||
Engine::Engine(std::shared_ptr<GoogleClient> client, std::shared_ptr<Catalog> catalog,
|
||||
Sleeper sleeper)
|
||||
: client_(std::move(client)), catalog_(std::move(catalog)), sleeper_(std::move(sleeper)) {
|
||||
if (!sleeper_)
|
||||
sleeper_ = [](std::chrono::milliseconds d) { std::this_thread::sleep_for(d); };
|
||||
}
|
||||
|
||||
Json Engine::list_apis(const std::string& filter, std::size_t limit, bool preferred_only) {
|
||||
auto dir = catalog_->directory();
|
||||
Json out = Json::array();
|
||||
for (const DirectoryItem* i : dir->search(filter, limit, preferred_only)) out.push(i->to_json());
|
||||
return out;
|
||||
}
|
||||
|
||||
Json Engine::list_methods(const std::string& api_ref, const std::string& filter,
|
||||
std::size_t limit) {
|
||||
auto api = catalog_->api(api_ref);
|
||||
limit = std::min(limit, DiscoveryCaps::kMaxListed);
|
||||
Json out = Json::array();
|
||||
std::size_t matched = 0;
|
||||
for (const Method& m : api->methods()) {
|
||||
if (!filter.empty() && m.id.find(filter) == std::string::npos) continue;
|
||||
++matched;
|
||||
if (out.as_array().size() < limit) out.push(m.summary());
|
||||
}
|
||||
Json o = Json::object();
|
||||
o.set("api", Json(api->id()));
|
||||
o.set("total", Json(static_cast<long>(api->methods().size())));
|
||||
o.set("matched", Json(static_cast<long>(matched)));
|
||||
o.set("truncated", Json(matched > out.as_array().size()));
|
||||
o.set("methods", out);
|
||||
return o;
|
||||
}
|
||||
|
||||
const Method& Engine::method_of(const Api& api, const std::string& method_id) const {
|
||||
const Method* m = api.find(method_id);
|
||||
if (!m) throw GoogleError(404, "NOT_FOUND", "api " + api.id() + " has no method " + method_id, "discovery");
|
||||
return *m;
|
||||
}
|
||||
|
||||
Json Engine::describe(const std::string& api_ref, const std::string& method_id) {
|
||||
auto api = catalog_->api(api_ref);
|
||||
const Method& m = method_of(*api, method_id);
|
||||
Json o = m.summary();
|
||||
Json params = Json::array();
|
||||
for (const auto& [n, p] : m.params) {
|
||||
Json pj = Json::object();
|
||||
pj.set("name", Json(n));
|
||||
pj.set("in", Json(p.location));
|
||||
pj.set("type", Json(p.type));
|
||||
if (!p.format.empty()) pj.set("format", Json(p.format));
|
||||
pj.set("required", Json(p.required));
|
||||
pj.set("repeated", Json(p.repeated));
|
||||
if (!p.pattern.empty()) pj.set("pattern", Json(p.pattern));
|
||||
if (!p.enum_values.empty()) {
|
||||
Json e = Json::array();
|
||||
for (const std::string& v : p.enum_values) e.push(Json(v));
|
||||
pj.set("enum", e);
|
||||
}
|
||||
params.push(pj);
|
||||
}
|
||||
o.set("parameters", params);
|
||||
Json scopes = Json::array();
|
||||
for (const std::string& s : m.scopes) scopes.push(Json(s));
|
||||
o.set("scopes", scopes);
|
||||
o.set("request", Json(m.request_ref));
|
||||
o.set("response", Json(m.response_ref));
|
||||
o.set("pageable", Json(api->is_pageable(m)));
|
||||
o.set("itemsKey", Json(api->is_pageable(m) ? api->items_key(m) : ""));
|
||||
o.set("operation", Json(!m.response_ref.empty() && api->is_operation_schema(m.response_ref)));
|
||||
Json locs = Json::array();
|
||||
for (const EndpointRow& e : api->endpoints()) locs.push(Json(e.location));
|
||||
o.set("locations", locs);
|
||||
return o;
|
||||
}
|
||||
|
||||
PreparedCall Engine::plan(const std::string& api_ref, const std::string& method_id,
|
||||
const CallArgs& args) {
|
||||
auto api = catalog_->api(api_ref);
|
||||
return Planner::plan(*api, method_of(*api, method_id), args);
|
||||
}
|
||||
|
||||
Json Engine::send(const PreparedCall& p) {
|
||||
return client_->request(p.http, p.endpoint, p.path, p.has_body ? &p.body : nullptr);
|
||||
}
|
||||
|
||||
CallResult Engine::call(const std::string& api_ref, const std::string& method_id,
|
||||
const CallArgs& args) {
|
||||
if (args.max_pages == 0 || args.max_pages > DiscoveryCaps::kMaxPages)
|
||||
bad("max_pages must be 1 to " + std::to_string(DiscoveryCaps::kMaxPages));
|
||||
auto api = catalog_->api(api_ref);
|
||||
const Method& m = method_of(*api, method_id);
|
||||
PreparedCall first = Planner::plan(*api, m, args); // all validation, no traffic
|
||||
if (args.max_pages > 1) {
|
||||
if (!first.pageable) bad("method " + m.id + " is not pageable (needs a pageToken parameter and a nextPageToken reply)");
|
||||
if (args.items_key.empty() && api->items_key(m).empty())
|
||||
bad("method " + m.id + " has several list fields: name one with items_key");
|
||||
}
|
||||
if (!client_->configured()) throw NotConfiguredError(api->name());
|
||||
|
||||
CallResult out;
|
||||
out.support = first.support;
|
||||
out.notes = first.notes;
|
||||
if (args.max_pages > 1) {
|
||||
walk_pages(*api, m, args, first, out);
|
||||
return out;
|
||||
}
|
||||
out.body = send(first);
|
||||
if (args.wait && first.operation) {
|
||||
out.body = wait_for(*api, first, args, out.body);
|
||||
out.operation_waited = true;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
void Engine::walk_pages(const Api& api, const Method& m, const CallArgs& args,
|
||||
const PreparedCall& first, CallResult& out) {
|
||||
std::string key = !args.items_key.empty() ? args.items_key : api.items_key(m);
|
||||
out.items_key = key;
|
||||
Json items = Json::array();
|
||||
Json last = Json::object();
|
||||
PreparedCall call = first;
|
||||
std::size_t count = 0;
|
||||
for (std::size_t page = 0; page < args.max_pages; ++page) {
|
||||
last = send(call);
|
||||
out.pages = page + 1;
|
||||
if (const Json* arr = last.find(key); arr && arr->is_array())
|
||||
for (const Json& item : arr->as_array()) {
|
||||
if (count >= DiscoveryCaps::kMaxItems) {
|
||||
out.truncated = true;
|
||||
break;
|
||||
}
|
||||
items.push(item);
|
||||
++count;
|
||||
}
|
||||
const Json* next = last.find("nextPageToken");
|
||||
std::string token = next && next->is_string() ? next->as_string() : "";
|
||||
if (token.empty()) break;
|
||||
if (out.truncated || page + 1 == args.max_pages) {
|
||||
out.truncated = true;
|
||||
break;
|
||||
}
|
||||
CallArgs again = args;
|
||||
again.params.set("pageToken", Json(token));
|
||||
call = Planner::plan(api, m, again);
|
||||
}
|
||||
if (last.is_object()) {
|
||||
last.set(key, items);
|
||||
if (!out.truncated) last.as_object().erase("nextPageToken");
|
||||
}
|
||||
out.body = last;
|
||||
}
|
||||
|
||||
Json Engine::wait_for(const Api& api, const PreparedCall& origin, const CallArgs& args,
|
||||
const Json& op_in) {
|
||||
Json op = op_in;
|
||||
std::string name = str(op, "name");
|
||||
if (name.empty()) bad("operation reply has no name");
|
||||
const Method* getter = nullptr;
|
||||
for (const Method* g : api.operation_getters()) {
|
||||
const Param& p = g->params.at("name");
|
||||
if (p.pattern.empty() ? api.operation_getters().size() == 1
|
||||
: std::regex_search(name, std::regex(p.pattern))) {
|
||||
getter = g;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!getter)
|
||||
bad("no operations.get method of " + api.id() + " accepts the operation name " + name);
|
||||
CallArgs poll;
|
||||
poll.params.set("name", Json(name));
|
||||
poll.location = args.location;
|
||||
poll.auth = args.auth;
|
||||
PreparedCall get = Planner::plan(api, *getter, poll);
|
||||
(void)origin;
|
||||
std::chrono::milliseconds waited{0};
|
||||
for (std::size_t i = 0; i < OperationCaps::kMaxPolls; ++i) {
|
||||
Operation parsed = Operation::from_json(op);
|
||||
if (parsed.done) {
|
||||
if (parsed.failed())
|
||||
throw GoogleError(502, "OPERATION_FAILED", parsed.error_message, api.name());
|
||||
return op;
|
||||
}
|
||||
if (waited >= OperationCaps::kMaxWait) break;
|
||||
sleeper_(OperationCaps::kInterval);
|
||||
waited += OperationCaps::kInterval;
|
||||
op = send(get);
|
||||
}
|
||||
throw GoogleError(504, "DEADLINE_EXCEEDED",
|
||||
"operation still running at the wait cap: " + name, api.name());
|
||||
}
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,68 @@
|
||||
#pragma once
|
||||
|
||||
// The Discovery engine: reach any method of any Google Discovery API by id.
|
||||
// Engine e(client, catalog);
|
||||
// e.call("bigquery", "bigquery.datasets.list", args);
|
||||
// Documents are fetched without credentials and cached (Catalog); the call
|
||||
// itself goes through GoogleClient with the credential the planner chose.
|
||||
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "google/client.h"
|
||||
#include "google/discovery/catalog.h"
|
||||
#include "google/discovery/planner.h"
|
||||
#include "google/operation.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
struct CallResult {
|
||||
clients::body::Json body;
|
||||
std::size_t pages = 1;
|
||||
bool truncated = false; // page or item cap hit with more pages left
|
||||
bool operation_waited = false;
|
||||
std::string items_key; // set when pages were merged
|
||||
Support support = Support::kFull;
|
||||
std::vector<std::string> notes;
|
||||
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
class Engine {
|
||||
public:
|
||||
Engine(std::shared_ptr<GoogleClient> client, std::shared_ptr<Catalog> catalog,
|
||||
Sleeper sleeper = nullptr);
|
||||
|
||||
bool configured() const { return client_->configured(); }
|
||||
Catalog& catalog() { return *catalog_; }
|
||||
|
||||
// Directory rows as Json; limit is capped at DiscoveryCaps::kMaxListed.
|
||||
clients::body::Json list_apis(const std::string& filter, std::size_t limit,
|
||||
bool preferred_only = true);
|
||||
// Methods of one API (optionally those whose id contains `filter`).
|
||||
clients::body::Json list_methods(const std::string& api_ref, const std::string& filter,
|
||||
std::size_t limit);
|
||||
clients::body::Json describe(const std::string& api_ref, const std::string& method_id);
|
||||
|
||||
// Validation only: no call to Google (the document fetch is not a call to
|
||||
// the API). Throws GoogleError::invalid on any problem.
|
||||
PreparedCall plan(const std::string& api_ref, const std::string& method_id,
|
||||
const CallArgs& args);
|
||||
CallResult call(const std::string& api_ref, const std::string& method_id,
|
||||
const CallArgs& args);
|
||||
|
||||
private:
|
||||
const Method& method_of(const Api& api, const std::string& method_id) const;
|
||||
clients::body::Json send(const PreparedCall& p);
|
||||
clients::body::Json wait_for(const Api& api, const PreparedCall& origin,
|
||||
const CallArgs& args, const clients::body::Json& op);
|
||||
void walk_pages(const Api& api, const Method& m, const CallArgs& args,
|
||||
const PreparedCall& first, CallResult& out);
|
||||
|
||||
std::shared_ptr<GoogleClient> client_;
|
||||
std::shared_ptr<Catalog> catalog_;
|
||||
Sleeper sleeper_;
|
||||
};
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,338 @@
|
||||
#include "google/discovery/model.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cctype>
|
||||
#include <set>
|
||||
|
||||
#include "google/discovery/caps.h"
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad_doc(const std::string& why) {
|
||||
throw GoogleError(502, "BAD_DOCUMENT", "discovery document: " + why);
|
||||
}
|
||||
|
||||
std::string str(const Json& o, const char* key) {
|
||||
const Json* v = o.find(key);
|
||||
return v && v->is_string() ? v->as_string() : "";
|
||||
}
|
||||
|
||||
bool flag(const Json& o, const char* key) {
|
||||
const Json* v = o.find(key);
|
||||
return v && v->is_bool() && v->as_bool();
|
||||
}
|
||||
|
||||
std::optional<double> num_or_string(const Json& o, const char* key) {
|
||||
const Json* v = o.find(key);
|
||||
if (!v) return std::nullopt;
|
||||
if (v->is_number()) return v->as_number();
|
||||
if (v->is_string()) {
|
||||
try {
|
||||
return std::stod(v->as_string());
|
||||
} catch (...) {
|
||||
}
|
||||
}
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
std::string ref_of(const Json& o, const char* key) {
|
||||
const Json* v = o.find(key);
|
||||
return v && v->is_object() ? str(*v, "$ref") : "";
|
||||
}
|
||||
|
||||
Param parse_param(const std::string& name, const Json& p) {
|
||||
Param out;
|
||||
out.name = name;
|
||||
out.type = str(p, "type");
|
||||
out.format = str(p, "format");
|
||||
out.location = str(p, "location");
|
||||
out.pattern = str(p, "pattern");
|
||||
out.description = str(p, "description");
|
||||
out.required = flag(p, "required");
|
||||
out.repeated = flag(p, "repeated");
|
||||
out.deprecated = flag(p, "deprecated");
|
||||
if (const Json* e = p.find("enum"); e && e->is_array())
|
||||
for (const Json& v : e->as_array())
|
||||
if (v.is_string()) out.enum_values.push_back(v.as_string());
|
||||
out.minimum = num_or_string(p, "minimum");
|
||||
out.maximum = num_or_string(p, "maximum");
|
||||
return out;
|
||||
}
|
||||
|
||||
// Names of {var} / {+var} in a path template, in order.
|
||||
std::vector<std::string> template_vars(const std::string& path, bool* malformed) {
|
||||
std::vector<std::string> out;
|
||||
for (std::size_t i = 0; i < path.size(); ++i) {
|
||||
if (path[i] == '}') {
|
||||
*malformed = true;
|
||||
return out;
|
||||
}
|
||||
if (path[i] != '{') continue;
|
||||
std::size_t close = path.find('}', i);
|
||||
if (close == std::string::npos) {
|
||||
*malformed = true;
|
||||
return out;
|
||||
}
|
||||
std::string v = path.substr(i + 1, close - i - 1);
|
||||
if (!v.empty() && v[0] == '+') v.erase(0, 1);
|
||||
if (v.empty() || v.find('{') != std::string::npos) {
|
||||
*malformed = true;
|
||||
return out;
|
||||
}
|
||||
out.push_back(v);
|
||||
i = close;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
void classify(Method& m) {
|
||||
auto reject = [&](const std::string& why) {
|
||||
m.support = Support::kRejected;
|
||||
m.notes.push_back(why);
|
||||
};
|
||||
auto limit = [&](const std::string& why) {
|
||||
if (m.support == Support::kFull) m.support = Support::kLimited;
|
||||
m.notes.push_back(why);
|
||||
};
|
||||
static const std::set<std::string> verbs = {"GET", "POST", "PUT", "PATCH", "DELETE"};
|
||||
if (!verbs.count(m.http_method)) reject("http method " + m.http_method + " is not supported");
|
||||
if (m.streaming_type == "SERVER_SIDE")
|
||||
limit("server-streaming method: the reply is read whole, no incremental delivery");
|
||||
else if (!m.streaming_type.empty() && m.streaming_type != "NONE")
|
||||
reject("streaming method (" + m.streaming_type + ") needs a socket, not a request");
|
||||
bool malformed = false;
|
||||
std::set<std::string> in_path;
|
||||
for (const std::string& v : template_vars(m.path, &malformed)) in_path.insert(v);
|
||||
if (malformed) reject("path template is malformed");
|
||||
for (const std::string& v : in_path) {
|
||||
auto it = m.params.find(v);
|
||||
if (it == m.params.end()) reject("path variable " + v + " has no declared parameter");
|
||||
else if (it->second.location != "path")
|
||||
reject("path variable " + v + " is declared as " + it->second.location);
|
||||
}
|
||||
for (const auto& [name, p] : m.params) {
|
||||
if (p.location != "path" && p.location != "query")
|
||||
reject("parameter " + name + " has location " + p.location);
|
||||
else if (p.location == "path" && !in_path.count(name))
|
||||
reject("path parameter " + name + " is not in the path template");
|
||||
if (p.type == "object") reject("parameter " + name + " is an object (map) parameter");
|
||||
else if (p.type != "string" && p.type != "integer" && p.type != "number" &&
|
||||
p.type != "boolean")
|
||||
reject("parameter " + name + " has unknown type " + p.type);
|
||||
}
|
||||
if (m.media.supported)
|
||||
limit(std::string("media upload (") + (m.media.simple ? "simple" : "") +
|
||||
(m.media.simple && m.media.resumable ? ", " : "") +
|
||||
(m.media.resumable ? "resumable" : "") +
|
||||
") not implemented: only the metadata call is made");
|
||||
if (m.supports_download) limit("media download (alt=media) not implemented: JSON reply only");
|
||||
if (m.response_ref == "HttpBody") limit("reply is google.api.HttpBody: may not be JSON");
|
||||
}
|
||||
|
||||
void collect(const Json& node, std::vector<Method>& out, std::size_t depth) {
|
||||
if (depth > DiscoveryCaps::kMaxResourceDepth) bad_doc("resources nested too deep");
|
||||
if (const Json* ms = node.find("methods"); ms && ms->is_object()) {
|
||||
for (const auto& [mname, mj] : ms->as_object()) {
|
||||
(void)mname;
|
||||
if (!mj.is_object()) bad_doc("method is not an object");
|
||||
if (out.size() >= DiscoveryCaps::kMaxMethods) bad_doc("more methods than the cap");
|
||||
Method m;
|
||||
m.id = str(mj, "id");
|
||||
m.http_method = str(mj, "httpMethod");
|
||||
m.path = str(mj, "path");
|
||||
m.flat_path = str(mj, "flatPath");
|
||||
if (m.id.empty() || m.path.empty()) bad_doc("method without id or path");
|
||||
m.request_ref = ref_of(mj, "request");
|
||||
m.response_ref = ref_of(mj, "response");
|
||||
m.supports_download = flag(mj, "supportsMediaDownload");
|
||||
m.deprecated = flag(mj, "deprecated");
|
||||
m.streaming_type = str(mj, "streamingType");
|
||||
if (const Json* sc = mj.find("scopes"); sc && sc->is_array())
|
||||
for (const Json& s : sc->as_array())
|
||||
if (s.is_string()) m.scopes.push_back(s.as_string());
|
||||
if (const Json* ps = mj.find("parameters"); ps && ps->is_object())
|
||||
for (const auto& [pn, pj] : ps->as_object())
|
||||
if (pj.is_object()) m.params[pn] = parse_param(pn, pj);
|
||||
if (const Json* po = mj.find("parameterOrder"); po && po->is_array())
|
||||
for (const Json& s : po->as_array())
|
||||
if (s.is_string()) m.param_order.push_back(s.as_string());
|
||||
if (const Json* mu = mj.find("mediaUpload"); mu && mu->is_object()) {
|
||||
m.media.supported = true;
|
||||
if (const Json* pr = mu->find("protocols"); pr && pr->is_object()) {
|
||||
m.media.simple = pr->has("simple");
|
||||
m.media.resumable = pr->has("resumable");
|
||||
}
|
||||
if (auto sz = num_or_string(*mu, "maxSize")) m.media.max_size = *sz;
|
||||
}
|
||||
classify(m);
|
||||
out.push_back(std::move(m));
|
||||
}
|
||||
}
|
||||
if (const Json* rs = node.find("resources"); rs && rs->is_object())
|
||||
for (const auto& [rn, rj] : rs->as_object()) {
|
||||
(void)rn;
|
||||
if (rj.is_object()) collect(rj, out, depth + 1);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const char* support_name(Support s) {
|
||||
switch (s) {
|
||||
case Support::kFull: return "full";
|
||||
case Support::kLimited: return "limited";
|
||||
case Support::kRejected: return "rejected";
|
||||
}
|
||||
return "rejected";
|
||||
}
|
||||
|
||||
bool is_google_host_url(const std::string& url) {
|
||||
static const std::string kScheme = "https://";
|
||||
if (url.rfind(kScheme, 0) != 0) return false;
|
||||
std::size_t end = url.find_first_of("/?#", kScheme.size());
|
||||
std::string host = url.substr(kScheme.size(), end == std::string::npos
|
||||
? std::string::npos
|
||||
: end - kScheme.size());
|
||||
if (host.empty() || host.find('@') != std::string::npos) return false;
|
||||
if (std::size_t colon = host.find(':'); colon != std::string::npos) {
|
||||
if (host.substr(colon) != ":443") return false;
|
||||
host.erase(colon);
|
||||
}
|
||||
static const std::string kSuffix = ".googleapis.com";
|
||||
if (host.size() <= kSuffix.size() ||
|
||||
host.compare(host.size() - kSuffix.size(), kSuffix.size(), kSuffix) != 0)
|
||||
return false;
|
||||
for (char c : host)
|
||||
if (!(std::isalnum(static_cast<unsigned char>(c)) || c == '.' || c == '-')) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
Json Method::summary() const {
|
||||
Json o = Json::object();
|
||||
o.set("id", Json(id));
|
||||
o.set("httpMethod", Json(http_method));
|
||||
o.set("path", Json(path));
|
||||
o.set("support", Json(support_name(support)));
|
||||
o.set("hasBody", Json(!request_ref.empty()));
|
||||
o.set("oauth", Json(needs_oauth()));
|
||||
o.set("deprecated", Json(deprecated));
|
||||
Json req = Json::array();
|
||||
for (const auto& [n, p] : params)
|
||||
if (p.required) req.push(Json(n));
|
||||
o.set("required", req);
|
||||
Json notes_j = Json::array();
|
||||
for (const std::string& n : notes) notes_j.push(Json(n));
|
||||
o.set("notes", notes_j);
|
||||
return o;
|
||||
}
|
||||
|
||||
Api Api::parse(const Json& doc) {
|
||||
if (!doc.is_object()) bad_doc("not an object");
|
||||
if (str(doc, "kind") != "discovery#restDescription") bad_doc("kind is not discovery#restDescription");
|
||||
if (str(doc, "protocol") != "rest") bad_doc("protocol is not rest");
|
||||
Api api;
|
||||
api.id_ = str(doc, "id");
|
||||
api.name_ = str(doc, "name");
|
||||
api.version_ = str(doc, "version");
|
||||
api.title_ = str(doc, "title");
|
||||
api.root_url_ = str(doc, "rootUrl");
|
||||
api.service_path_ = str(doc, "servicePath");
|
||||
api.batch_path_ = str(doc, "batchPath");
|
||||
if (api.id_.empty() || api.name_.empty() || api.version_.empty()) bad_doc("missing id, name or version");
|
||||
if (!is_google_host_url(api.root_url_)) bad_doc("rootUrl is not an https googleapis.com host: " + api.root_url_);
|
||||
if (api.root_url_.back() != '/') bad_doc("rootUrl has no trailing slash");
|
||||
if (api.service_path_.find_first_of("?#{}") != std::string::npos) bad_doc("servicePath is not a plain path");
|
||||
if (const Json* eps = doc.find("endpoints"); eps && eps->is_array())
|
||||
for (const Json& e : eps->as_array()) {
|
||||
EndpointRow row{str(e, "location"), str(e, "endpointUrl"), str(e, "description")};
|
||||
if (row.location.empty() || !is_google_host_url(row.url) || row.url.back() != '/')
|
||||
bad_doc("endpoint row is not a googleapis.com root URL: " + row.url);
|
||||
api.endpoints_.push_back(std::move(row));
|
||||
}
|
||||
if (const Json* ps = doc.find("parameters"); ps && ps->is_object())
|
||||
for (const auto& [n, p] : ps->as_object()) {
|
||||
(void)p;
|
||||
api.global_params_.push_back(n);
|
||||
}
|
||||
if (const Json* s = doc.find("schemas"); s && s->is_object()) {
|
||||
if (s->as_object().size() > DiscoveryCaps::kMaxSchemas) bad_doc("more schemas than the cap");
|
||||
api.schemas_ = *s;
|
||||
}
|
||||
collect(doc, api.methods_, 0);
|
||||
for (std::size_t i = 0; i < api.methods_.size(); ++i)
|
||||
if (!api.index_.emplace(api.methods_[i].id, i).second)
|
||||
bad_doc("duplicate method id " + api.methods_[i].id);
|
||||
return api;
|
||||
}
|
||||
|
||||
std::string Api::base_url() const {
|
||||
std::string b = root_url_ + service_path_;
|
||||
while (!b.empty() && b.back() == '/') b.pop_back();
|
||||
return b;
|
||||
}
|
||||
|
||||
const Method* Api::find(const std::string& method_id) const {
|
||||
auto it = index_.find(method_id);
|
||||
return it == index_.end() ? nullptr : &methods_[it->second];
|
||||
}
|
||||
|
||||
const Json* Api::schema(const std::string& name) const { return schemas_.find(name); }
|
||||
|
||||
std::size_t Api::schema_count() const { return schemas_.as_object().size(); }
|
||||
|
||||
bool Api::is_operation_schema(const std::string& ref) const {
|
||||
const Json* s = schema(ref);
|
||||
if (!s) return false;
|
||||
const Json* props = s->find("properties");
|
||||
if (!props || !props->is_object()) return false;
|
||||
const Json* done = props->find("done");
|
||||
return props->has("name") && done && done->is_object() && str(*done, "type") == "boolean";
|
||||
}
|
||||
|
||||
bool Api::is_pageable(const Method& m) const {
|
||||
auto it = m.params.find("pageToken");
|
||||
if (it == m.params.end() || it->second.location != "query") return false;
|
||||
const Json* s = schema(m.response_ref);
|
||||
if (!s) return false;
|
||||
const Json* props = s->find("properties");
|
||||
return props && props->is_object() && props->has("nextPageToken");
|
||||
}
|
||||
|
||||
std::string Api::items_key(const Method& m) const {
|
||||
const Json* s = schema(m.response_ref);
|
||||
const Json* props = s ? s->find("properties") : nullptr;
|
||||
if (!props || !props->is_object()) return "";
|
||||
std::string found;
|
||||
for (const auto& [n, p] : props->as_object()) {
|
||||
if (n == "nextPageToken" || n == "unreachable") continue;
|
||||
if (p.is_object() && str(p, "type") == "array") {
|
||||
if (!found.empty()) return ""; // ambiguous: caller names it
|
||||
found = n;
|
||||
}
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
std::vector<const Method*> Api::operation_getters() const {
|
||||
std::vector<const Method*> out;
|
||||
static const std::string kSuffix = "operations.get";
|
||||
for (const Method& m : methods_) {
|
||||
if (m.id.size() < kSuffix.size() ||
|
||||
m.id.compare(m.id.size() - kSuffix.size(), kSuffix.size(), kSuffix) != 0)
|
||||
continue;
|
||||
if (m.http_method != "GET" || m.support == Support::kRejected) continue;
|
||||
if (!is_operation_schema(m.response_ref)) continue;
|
||||
std::size_t required = 0;
|
||||
bool name_path = m.params.count("name") && m.params.at("name").location == "path";
|
||||
for (const auto& [n, p] : m.params) required += p.required ? 1 : 0;
|
||||
if (name_path && required == 1) out.push_back(&m);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,118 @@
|
||||
#pragma once
|
||||
|
||||
// Parsed Discovery document: parameters, methods, schemas. Pure data plus
|
||||
// classification of what the engine can and cannot do with each method.
|
||||
// Every shape here was read from the real documents (see SPEC.md section
|
||||
// "Shapes seen"); anything else is rejected loudly by Api::parse.
|
||||
|
||||
#include <map>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
// Full: called as documented. Limited: callable, but part of the method's
|
||||
// surface is not reachable (media bytes, non-JSON reply); notes say which.
|
||||
// Rejected: never called; the notes say why.
|
||||
enum class Support { kFull, kLimited, kRejected };
|
||||
const char* support_name(Support s);
|
||||
|
||||
struct Param {
|
||||
std::string name;
|
||||
std::string type; // string integer number boolean (object: rejected)
|
||||
std::string format;
|
||||
std::string location; // path or query
|
||||
std::string pattern;
|
||||
std::string description;
|
||||
bool required = false;
|
||||
bool repeated = false;
|
||||
bool deprecated = false;
|
||||
std::vector<std::string> enum_values;
|
||||
std::optional<double> minimum;
|
||||
std::optional<double> maximum;
|
||||
};
|
||||
|
||||
struct MediaUpload {
|
||||
bool supported = false;
|
||||
bool simple = false;
|
||||
bool resumable = false;
|
||||
double max_size = 0; // bytes, 0: not stated
|
||||
};
|
||||
|
||||
struct Method {
|
||||
std::string id; // bigquery.jobs.query
|
||||
std::string http_method; // GET POST PUT PATCH DELETE
|
||||
std::string path; // v1/{+name}:cancel (relative to base)
|
||||
std::string flat_path;
|
||||
std::string request_ref; // schema name, empty: no body
|
||||
std::string response_ref; // schema name, empty: no body
|
||||
std::map<std::string, Param> params;
|
||||
std::vector<std::string> param_order;
|
||||
std::vector<std::string> scopes;
|
||||
MediaUpload media;
|
||||
bool supports_download = false;
|
||||
bool deprecated = false;
|
||||
std::string streaming_type;
|
||||
Support support = Support::kFull;
|
||||
std::vector<std::string> notes; // why Limited or Rejected, one per cause
|
||||
|
||||
bool needs_oauth() const { return !scopes.empty(); }
|
||||
clients::body::Json summary() const;
|
||||
};
|
||||
|
||||
struct EndpointRow {
|
||||
std::string location;
|
||||
std::string url; // root URL, trailing slash
|
||||
std::string description;
|
||||
};
|
||||
|
||||
class Api {
|
||||
public:
|
||||
// Throws GoogleError (502 BAD_DOCUMENT) when the document is not one the
|
||||
// engine can trust: wrong kind/protocol, host outside googleapis.com,
|
||||
// over the method cap, duplicate method ids.
|
||||
static Api parse(const clients::body::Json& doc);
|
||||
|
||||
const std::string& id() const { return id_; }
|
||||
const std::string& name() const { return name_; }
|
||||
const std::string& version() const { return version_; }
|
||||
const std::string& title() const { return title_; }
|
||||
const std::string& root_url() const { return root_url_; }
|
||||
const std::string& service_path() const { return service_path_; }
|
||||
// root_url + service_path without trailing slash.
|
||||
std::string base_url() const;
|
||||
const std::vector<EndpointRow>& endpoints() const { return endpoints_; }
|
||||
const std::vector<Method>& methods() const { return methods_; }
|
||||
const Method* find(const std::string& method_id) const;
|
||||
const clients::body::Json* schema(const std::string& name) const;
|
||||
std::size_t schema_count() const;
|
||||
// Query parameters every method accepts, as declared by the document.
|
||||
const std::vector<std::string>& global_params() const { return global_params_; }
|
||||
bool batch_path_declared() const { return !batch_path_.empty(); }
|
||||
|
||||
bool is_operation_schema(const std::string& ref) const;
|
||||
bool is_pageable(const Method& m) const;
|
||||
// The array property to collect when paging m, or "" when none/ambiguous.
|
||||
std::string items_key(const Method& m) const;
|
||||
// Methods ending in "operations.get" usable to poll an operation.
|
||||
std::vector<const Method*> operation_getters() const;
|
||||
|
||||
// Raw bytes of the parsed text, for cache accounting.
|
||||
std::size_t raw_bytes = 0;
|
||||
|
||||
private:
|
||||
std::string id_, name_, version_, title_, root_url_, service_path_, batch_path_;
|
||||
std::vector<EndpointRow> endpoints_;
|
||||
std::vector<Method> methods_;
|
||||
std::map<std::string, std::size_t> index_;
|
||||
clients::body::Json schemas_ = clients::body::Json::object();
|
||||
std::vector<std::string> global_params_;
|
||||
};
|
||||
|
||||
// True for an https URL whose host is googleapis.com or a subdomain of it.
|
||||
bool is_google_host_url(const std::string& url);
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,207 @@
|
||||
#include "google/discovery/planner.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <set>
|
||||
|
||||
#include "clients/url.h"
|
||||
#include "google/discovery/caps.h"
|
||||
#include "google/discovery/validate.h"
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw GoogleError::invalid(why); }
|
||||
|
||||
// Document-level query parameters a caller may set. Everything else the
|
||||
// document declares globally (alt, key, access_token, oauth_token, callback,
|
||||
// uploadType, upload_protocol, $.xgafv) either changes the reply format,
|
||||
// carries a credential, or switches protocol: rejected by name.
|
||||
bool allowed_global(const std::string& name) {
|
||||
return name == "fields" || name == "quotaUser" || name == "prettyPrint";
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const char* auth_mode_name(AuthMode m) {
|
||||
switch (m) {
|
||||
case AuthMode::kAuto: return "auto";
|
||||
case AuthMode::kKey: return "key";
|
||||
case AuthMode::kOAuth: return "oauth";
|
||||
}
|
||||
return "auto";
|
||||
}
|
||||
|
||||
AuthMode auth_mode_from(const std::string& name) {
|
||||
if (name.empty() || name == "auto") return AuthMode::kAuto;
|
||||
if (name == "key") return AuthMode::kKey;
|
||||
if (name == "oauth") return AuthMode::kOAuth;
|
||||
bad("auth must be auto, key or oauth");
|
||||
}
|
||||
|
||||
std::string Planner::encode_path_value(const std::string& value, bool reserved) {
|
||||
if (!reserved) return clients::Url::encode_component(value);
|
||||
std::string out;
|
||||
std::size_t start = 0;
|
||||
while (true) {
|
||||
std::size_t end = value.find('/', start);
|
||||
out += clients::Url::encode_component(
|
||||
value.substr(start, end == std::string::npos ? std::string::npos : end - start));
|
||||
if (end == std::string::npos) break;
|
||||
out += '/';
|
||||
start = end + 1;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
Json PreparedCall::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("api", Json(api_id));
|
||||
o.set("method", Json(method_id));
|
||||
o.set("httpMethod", Json(clients::method_name(http)));
|
||||
o.set("url", Json(endpoint.base_url + path));
|
||||
o.set("credential", Json(endpoint.prefer_key ? "key" : "oauth"));
|
||||
o.set("hasBody", Json(has_body));
|
||||
o.set("support", Json(support_name(support)));
|
||||
o.set("pageable", Json(pageable));
|
||||
o.set("operation", Json(operation));
|
||||
Json n = Json::array();
|
||||
for (const std::string& s : notes) n.push(Json(s));
|
||||
o.set("notes", n);
|
||||
return o;
|
||||
}
|
||||
|
||||
PreparedCall Planner::plan(const Api& api, const Method& m, const CallArgs& args) {
|
||||
if (m.support == Support::kRejected) {
|
||||
std::string why;
|
||||
for (const std::string& n : m.notes) why += (why.empty() ? "" : "; ") + n;
|
||||
bad("method " + m.id + " cannot be called by this engine: " + why);
|
||||
}
|
||||
if (!args.params.is_object()) bad("params must be an object");
|
||||
|
||||
// Endpoint: the global host, or a documented regional one.
|
||||
std::string root = api.root_url();
|
||||
if (!args.location.empty()) {
|
||||
auto it = std::find_if(api.endpoints().begin(), api.endpoints().end(),
|
||||
[&](const EndpointRow& e) { return e.location == args.location; });
|
||||
if (it == api.endpoints().end()) {
|
||||
std::string known;
|
||||
for (std::size_t i = 0; i < api.endpoints().size() && i < 8; ++i)
|
||||
known += (i ? ", " : "") + api.endpoints()[i].location;
|
||||
bad("api " + api.id() + " has no endpoint for location '" + args.location + "'" +
|
||||
(known.empty() ? " (it documents no regional endpoints)" : " (e.g. " + known + ")"));
|
||||
}
|
||||
root = it->url;
|
||||
}
|
||||
std::string base = root + api.service_path();
|
||||
while (!base.empty() && base.back() == '/') base.pop_back();
|
||||
|
||||
// Parameters.
|
||||
std::set<std::string> seen;
|
||||
std::vector<std::pair<std::string, std::string>> query;
|
||||
std::map<std::string, std::string> path_values;
|
||||
for (const auto& [name, value] : args.params.as_object()) {
|
||||
auto it = m.params.find(name);
|
||||
if (it == m.params.end()) {
|
||||
const auto& g = api.global_params();
|
||||
if (std::find(g.begin(), g.end(), name) != g.end()) {
|
||||
if (!allowed_global(name))
|
||||
bad("parameter " + name + " is reserved: it is set by the engine or changes the reply format");
|
||||
Param p;
|
||||
p.name = name;
|
||||
p.location = "query";
|
||||
p.type = name == "prettyPrint" ? "boolean" : "string";
|
||||
for (const std::string& v : ParamValidator::values(p, value)) query.emplace_back(name, v);
|
||||
seen.insert(name);
|
||||
continue;
|
||||
}
|
||||
std::string known;
|
||||
for (const auto& [n, p] : m.params) known += (known.empty() ? "" : ", ") + n;
|
||||
bad("method " + m.id + " has no parameter '" + name + "' (accepts: " + known + ")");
|
||||
}
|
||||
const Param& p = it->second;
|
||||
std::vector<std::string> vals = ParamValidator::values(p, value);
|
||||
seen.insert(name);
|
||||
if (p.location == "path") {
|
||||
if (vals.size() != 1) bad("path parameter " + name + " takes one value");
|
||||
path_values[name] = vals[0];
|
||||
} else {
|
||||
for (const std::string& v : vals) query.emplace_back(name, v);
|
||||
}
|
||||
}
|
||||
for (const auto& [name, p] : m.params)
|
||||
if (p.required && !seen.count(name)) bad("required parameter " + name + " is missing");
|
||||
if (query.size() > DiscoveryCaps::kMaxQueryParams)
|
||||
bad("more than " + std::to_string(DiscoveryCaps::kMaxQueryParams) + " query values");
|
||||
|
||||
// Path: substitute {x} and {+x}.
|
||||
std::string path;
|
||||
for (std::size_t i = 0; i < m.path.size(); ++i) {
|
||||
if (m.path[i] != '{') {
|
||||
path += m.path[i];
|
||||
continue;
|
||||
}
|
||||
std::size_t close = m.path.find('}', i);
|
||||
std::string var = m.path.substr(i + 1, close - i - 1);
|
||||
bool reserved = !var.empty() && var[0] == '+';
|
||||
if (reserved) var.erase(0, 1);
|
||||
auto v = path_values.find(var);
|
||||
if (v == path_values.end()) bad("required path parameter " + var + " is missing");
|
||||
path += encode_path_value(v->second, reserved);
|
||||
i = close;
|
||||
}
|
||||
std::string qs;
|
||||
for (const auto& [k, v] : query)
|
||||
qs += (qs.empty() ? "?" : "&") + clients::Url::encode_component(k) + "=" +
|
||||
clients::Url::encode_component(v);
|
||||
path = "/" + path + qs;
|
||||
if (path.size() > DiscoveryCaps::kMaxPathChars)
|
||||
bad("request target is longer than " + std::to_string(DiscoveryCaps::kMaxPathChars));
|
||||
|
||||
PreparedCall out;
|
||||
out.api_id = api.id();
|
||||
out.method_id = m.id;
|
||||
out.support = m.support;
|
||||
out.notes = m.notes;
|
||||
out.path = path;
|
||||
if (m.http_method == "GET") out.http = clients::Method::kGet;
|
||||
else if (m.http_method == "POST") out.http = clients::Method::kPost;
|
||||
else if (m.http_method == "PUT") out.http = clients::Method::kPut;
|
||||
else if (m.http_method == "PATCH") out.http = clients::Method::kPatch;
|
||||
else out.http = clients::Method::kDelete;
|
||||
|
||||
out.endpoint.name = api.name();
|
||||
out.endpoint.base_url = base;
|
||||
switch (args.auth) {
|
||||
case AuthMode::kKey: out.endpoint.prefer_key = true; break;
|
||||
case AuthMode::kOAuth: out.endpoint.prefer_key = false; break;
|
||||
case AuthMode::kAuto: out.endpoint.prefer_key = !m.needs_oauth(); break;
|
||||
}
|
||||
out.oauth_scoped = m.needs_oauth();
|
||||
|
||||
// Body.
|
||||
bool verb_has_body = out.http == clients::Method::kPost || out.http == clients::Method::kPut ||
|
||||
out.http == clients::Method::kPatch;
|
||||
if (m.request_ref.empty()) {
|
||||
if (!args.body.is_null()) bad("method " + m.id + " takes no request body");
|
||||
if (verb_has_body) {
|
||||
out.body = Json::object();
|
||||
out.has_body = true;
|
||||
}
|
||||
} else if (!verb_has_body) {
|
||||
bad("method " + m.id + " declares a body on a " + m.http_method);
|
||||
} else {
|
||||
out.body = args.body.is_null() ? Json::object() : args.body;
|
||||
BodyValidator(api).validate(m.request_ref, out.body);
|
||||
out.has_body = true;
|
||||
}
|
||||
|
||||
out.pageable = api.is_pageable(m);
|
||||
out.operation = !m.response_ref.empty() && api.is_operation_schema(m.response_ref);
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,59 @@
|
||||
#pragma once
|
||||
|
||||
// Turns (Api, Method, arguments) into one validated HTTP request. No
|
||||
// network traffic happens here: every rule that can fail does so before a
|
||||
// byte is sent.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "clients/request.h"
|
||||
#include "google/discovery/model.h"
|
||||
#include "google/endpoints.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
// Which credential reaches Google. Auto: methods that declare no OAuth scope
|
||||
// take the API key, every other method takes OAuth. Key / OAuth force the
|
||||
// choice (the client falls back to the other credential only when the forced
|
||||
// one is not configured).
|
||||
enum class AuthMode { kAuto, kKey, kOAuth };
|
||||
const char* auth_mode_name(AuthMode mode);
|
||||
AuthMode auth_mode_from(const std::string& name); // throws invalid
|
||||
|
||||
struct CallArgs {
|
||||
clients::body::Json params = clients::body::Json::object(); // name -> value
|
||||
clients::body::Json body; // null: none
|
||||
std::string location; // regional endpoint (Api::endpoints) or empty
|
||||
AuthMode auth = AuthMode::kAuto;
|
||||
std::size_t max_pages = 1; // > 1 walks nextPageToken (list methods only)
|
||||
std::string items_key; // array to collect when paging, else derived
|
||||
bool wait = false; // poll a long-running operation to completion
|
||||
};
|
||||
|
||||
struct PreparedCall {
|
||||
std::string api_id;
|
||||
std::string method_id;
|
||||
clients::Method http = clients::Method::kGet;
|
||||
ServiceEndpoint endpoint;
|
||||
std::string path; // "/v1/..." with query string
|
||||
clients::body::Json body;
|
||||
bool has_body = false;
|
||||
Support support = Support::kFull;
|
||||
std::vector<std::string> notes;
|
||||
bool pageable = false;
|
||||
bool operation = false; // the reply is a google.longrunning.Operation
|
||||
bool oauth_scoped = false;
|
||||
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
class Planner {
|
||||
public:
|
||||
static PreparedCall plan(const Api& api, const Method& method, const CallArgs& args);
|
||||
// Percent-encodes one value for a path template ({+x} keeps '/').
|
||||
static std::string encode_path_value(const std::string& value, bool reserved);
|
||||
};
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,189 @@
|
||||
// Audit tool: reads saved Discovery documents (the same files the engine
|
||||
// would fetch) and prints, as JSON, which APIs and methods the engine
|
||||
// cannot handle or only handles in part, and why. Driven by
|
||||
// check_discovery_directory.py; one classification, the engine's own.
|
||||
// discovery-audit <dir with directory.json and <name>_<version>.json>
|
||||
#include <cstdio>
|
||||
#include <fstream>
|
||||
#include <iostream>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <sstream>
|
||||
|
||||
#include "google/discovery/catalog.h"
|
||||
#include "google/discovery/model.h"
|
||||
#include "google/discovery/planner.h"
|
||||
#include "google/errors.h"
|
||||
|
||||
using namespace molo::google;
|
||||
using namespace molo::google::discovery;
|
||||
using molo::clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
// A value that satisfies a parameter's declaration, for a dry plan.
|
||||
Json synth(const Param& p) {
|
||||
if (!p.enum_values.empty()) return Json(p.enum_values[0]);
|
||||
if (p.type == "boolean") return Json(true);
|
||||
if (p.type == "integer") {
|
||||
long v = p.minimum ? static_cast<long>(*p.minimum) : 1;
|
||||
return p.format == "int64" ? Json(std::to_string(v)) : Json(v);
|
||||
}
|
||||
if (p.type == "number") return Json(p.minimum ? *p.minimum : 1.0);
|
||||
if (p.format == "int64" || p.format == "uint64") return Json("1");
|
||||
if (p.format == "google-datetime" || p.format == "date-time") return Json("2026-10-25T10:00:00Z");
|
||||
if (p.format == "google-duration") return Json("1s");
|
||||
std::string s = p.pattern;
|
||||
auto rep = [&](const std::string& a, const std::string& b) {
|
||||
for (std::size_t i = s.find(a); i != std::string::npos; i = s.find(a, i + b.size()))
|
||||
s.replace(i, a.size(), b);
|
||||
};
|
||||
if (s.empty()) return Json("x1");
|
||||
rep("[^/]+", "x1");
|
||||
rep("[^/]*", "x1");
|
||||
rep("[^/]", "x");
|
||||
if (!s.empty() && s.front() == '^') s.erase(0, 1);
|
||||
if (!s.empty() && s.back() == '$') s.pop_back();
|
||||
return Json(s);
|
||||
}
|
||||
|
||||
Json synth_args(const Method& m) {
|
||||
Json params = Json::object();
|
||||
for (const auto& [n, p] : m.params) {
|
||||
if (!p.required) continue;
|
||||
Json v = synth(p);
|
||||
if (p.repeated) {
|
||||
Json a = Json::array();
|
||||
a.push(v);
|
||||
v = a;
|
||||
}
|
||||
params.set(n, v);
|
||||
}
|
||||
return params;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main(int argc, char** argv) {
|
||||
if (argc != 2) {
|
||||
std::fprintf(stderr, "usage: discovery-audit <dir>\n");
|
||||
return 2;
|
||||
}
|
||||
auto source = std::make_shared<FileDocumentSource>(argv[1]);
|
||||
Catalog catalog(source);
|
||||
Json report = Json::object();
|
||||
Json apis = Json::array();
|
||||
Json failed = Json::array();
|
||||
std::map<std::string, long> reasons;
|
||||
long methods = 0, full = 0, limited = 0, rejected = 0, pageable = 0, lro = 0, lro_unpollable = 0,
|
||||
regional = 0, batch = 0, planned = 0, plan_skipped = 0, plan_failed = 0;
|
||||
Json plan_failures = Json::array();
|
||||
std::shared_ptr<const Directory> dir;
|
||||
try {
|
||||
dir = catalog.directory();
|
||||
} catch (const std::exception& e) {
|
||||
std::fprintf(stderr, "directory: %s\n", e.what());
|
||||
return 1;
|
||||
}
|
||||
long preferred = 0;
|
||||
for (const DirectoryItem& item : dir->items()) {
|
||||
if (!item.preferred) continue;
|
||||
++preferred;
|
||||
std::shared_ptr<const Api> api;
|
||||
try {
|
||||
api = catalog.api(item.name + ":" + item.version);
|
||||
} catch (const std::exception& e) {
|
||||
Json f = Json::object();
|
||||
f.set("api", Json(item.id));
|
||||
f.set("why", Json(std::string(e.what())));
|
||||
failed.push(f);
|
||||
continue;
|
||||
}
|
||||
bool has_getter = !api->operation_getters().empty();
|
||||
Json a = Json::object();
|
||||
a.set("api", Json(api->id()));
|
||||
long a_full = 0, a_limited = 0, a_rejected = 0;
|
||||
Json problems = Json::array();
|
||||
for (const Method& m : api->methods()) {
|
||||
++methods;
|
||||
if (m.support == Support::kFull) ++full, ++a_full;
|
||||
if (m.support == Support::kLimited) ++limited, ++a_limited;
|
||||
if (m.support == Support::kRejected) ++rejected, ++a_rejected;
|
||||
if (api->is_pageable(m)) ++pageable;
|
||||
bool op = !m.response_ref.empty() && api->is_operation_schema(m.response_ref);
|
||||
if (op) {
|
||||
++lro;
|
||||
if (!has_getter) {
|
||||
++lro_unpollable;
|
||||
reasons["long-running reply but the API has no operations.get (wait unavailable)"]++;
|
||||
}
|
||||
}
|
||||
if (m.support != Support::kRejected) {
|
||||
CallArgs args;
|
||||
args.params = synth_args(m);
|
||||
try {
|
||||
(void)Planner::plan(*api, m, args);
|
||||
++planned;
|
||||
} catch (const GoogleError& e) {
|
||||
std::string why = e.message();
|
||||
if (why.find("does not match") != std::string::npos) {
|
||||
++plan_skipped; // synthetic value did not fit an exotic pattern
|
||||
} else {
|
||||
++plan_failed;
|
||||
Json f = Json::object();
|
||||
f.set("method", Json(m.id));
|
||||
f.set("why", Json(why));
|
||||
plan_failures.push(f);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const std::string& n : m.notes) {
|
||||
// Reason class: strip parameter and path names so counts group.
|
||||
std::string cls = n.substr(0, n.find(" (") == std::string::npos ? n.size() : n.find(" ("));
|
||||
if (cls.rfind("parameter ", 0) == 0 || cls.rfind("path ", 0) == 0) cls = n;
|
||||
reasons[std::string(support_name(m.support)) + ": " + cls]++;
|
||||
}
|
||||
if (m.support != Support::kFull) {
|
||||
Json p = Json::object();
|
||||
p.set("method", Json(m.id));
|
||||
p.set("support", Json(support_name(m.support)));
|
||||
Json notes = Json::array();
|
||||
for (const std::string& n : m.notes) notes.push(Json(n));
|
||||
p.set("notes", notes);
|
||||
problems.push(p);
|
||||
}
|
||||
}
|
||||
if (!api->endpoints().empty()) ++regional;
|
||||
if (api->batch_path_declared()) ++batch;
|
||||
a.set("methods", Json(static_cast<long>(api->methods().size())));
|
||||
a.set("full", Json(a_full));
|
||||
a.set("limited", Json(a_limited));
|
||||
a.set("rejected", Json(a_rejected));
|
||||
a.set("operationGetter", Json(has_getter));
|
||||
a.set("regionalEndpoints", Json(static_cast<long>(api->endpoints().size())));
|
||||
a.set("problems", problems);
|
||||
apis.push(a);
|
||||
}
|
||||
report.set("preferred", Json(preferred));
|
||||
report.set("loaded", Json(static_cast<long>(preferred) - static_cast<long>(failed.as_array().size())));
|
||||
report.set("failed", failed);
|
||||
report.set("methods", Json(methods));
|
||||
report.set("full", Json(full));
|
||||
report.set("limited", Json(limited));
|
||||
report.set("rejected", Json(rejected));
|
||||
report.set("pageable", Json(pageable));
|
||||
report.set("longRunning", Json(lro));
|
||||
report.set("longRunningWithoutGetter", Json(lro_unpollable));
|
||||
report.set("apisWithRegionalEndpoints", Json(regional));
|
||||
report.set("apisDeclaringBatchPath", Json(batch));
|
||||
Json rs = Json::object();
|
||||
for (const auto& [k, v] : reasons) rs.set(k, Json(v));
|
||||
report.set("reasons", rs);
|
||||
report.set("planned", Json(planned));
|
||||
report.set("planSkippedPattern", Json(plan_skipped));
|
||||
report.set("planFailed", Json(plan_failed));
|
||||
report.set("planFailures", plan_failures);
|
||||
report.set("apis", apis);
|
||||
std::cout << report.dump() << "\n";
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Walk Google's whole Discovery directory and report what the engine cannot do.
|
||||
|
||||
python3 src/google/discovery/tools/check_discovery_directory.py [--cache DIR]
|
||||
[--no-fetch] [--json FILE] [--max-rejected N] [--workers N]
|
||||
|
||||
1. Fetches https://discovery.googleapis.com/discovery/v1/apis and the
|
||||
document of every preferred API into --cache (default
|
||||
build/google/discovery-cache). Fetch failures are listed as UNREACHABLE:
|
||||
Google lists them but does not serve them (not an engine fault).
|
||||
2. Builds build/google/discovery-audit (src/google/discovery/tools/audit_main.cc)
|
||||
against libgoogle.a and runs it on the saved files, so the verdict comes
|
||||
from the engine's own classification, never from a second implementation.
|
||||
3. Prints totals, a reason table and the APIs with rejected or limited
|
||||
methods. Exit 1 when a fetched document does not parse, or when rejected
|
||||
methods exceed --max-rejected (default: no ceiling).
|
||||
|
||||
python3 standard library only.
|
||||
"""
|
||||
import argparse
|
||||
import concurrent.futures as cf
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "..", ".."))
|
||||
DIRECTORY = "https://discovery.googleapis.com/discovery/v1/apis"
|
||||
AUDIT_SRC = os.path.join(ROOT, "src/google/discovery/tools/audit_main.cc")
|
||||
AUDIT_BIN = os.path.join(ROOT, "build/google/discovery-audit")
|
||||
LIBS = ["build/google/libgoogle.a", "build/clients/libclients.a", "build/liboperability.a"]
|
||||
|
||||
|
||||
def get(url, tries=3):
|
||||
last = None
|
||||
for _ in range(tries):
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=90) as r:
|
||||
return r.read()
|
||||
except Exception as e: # noqa: BLE001 - report whatever the network said
|
||||
last = e
|
||||
raise last
|
||||
|
||||
|
||||
def fetch_all(cache, workers):
|
||||
os.makedirs(cache, exist_ok=True)
|
||||
raw = get(DIRECTORY)
|
||||
with open(os.path.join(cache, "directory.json"), "wb") as f:
|
||||
f.write(raw)
|
||||
items = [i for i in json.loads(raw)["items"] if i.get("preferred")]
|
||||
unreachable = []
|
||||
|
||||
def one(item):
|
||||
path = os.path.join(cache, "%s_%s.json" % (item["name"], item["version"]))
|
||||
try:
|
||||
data = get(item["discoveryRestUrl"])
|
||||
with open(path, "wb") as f:
|
||||
f.write(data)
|
||||
return None
|
||||
except Exception as e: # noqa: BLE001
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
return (item["id"], str(e))
|
||||
|
||||
with cf.ThreadPoolExecutor(workers) as ex:
|
||||
for r in ex.map(one, items):
|
||||
if r:
|
||||
unreachable.append(r)
|
||||
return len(items), unreachable
|
||||
|
||||
|
||||
def build_audit():
|
||||
subprocess.run(["make", "-s", "-j8", "google", "build/clients/libclients.a",
|
||||
"build/liboperability.a"], cwd=ROOT, check=True)
|
||||
cmd = ["g++", "-std=c++23", "-Wall", "-Wextra", "-Werror", "-DMOLO_HAS_WOLFSSL",
|
||||
"-I" + os.path.join(ROOT, "src"), AUDIT_SRC] + [os.path.join(ROOT, l) for l in LIBS] + \
|
||||
["-lwolfssl", "-o", AUDIT_BIN]
|
||||
subprocess.run(cmd, cwd=ROOT, check=True)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--cache", default=os.path.join(ROOT, "build/google/discovery-cache"))
|
||||
ap.add_argument("--no-fetch", action="store_true", help="reuse the files already in --cache")
|
||||
ap.add_argument("--json", help="write the full audit report here")
|
||||
ap.add_argument("--max-rejected", type=int, default=None)
|
||||
ap.add_argument("--workers", type=int, default=16)
|
||||
ap.add_argument("--list", type=int, default=40, help="APIs to list with problems")
|
||||
args = ap.parse_args()
|
||||
|
||||
unreachable = []
|
||||
listed = None
|
||||
if not args.no_fetch:
|
||||
listed, unreachable = fetch_all(args.cache, args.workers)
|
||||
build_audit()
|
||||
out = subprocess.run([AUDIT_BIN, args.cache], capture_output=True, text=True)
|
||||
if out.returncode != 0:
|
||||
print(out.stderr, file=sys.stderr)
|
||||
return 1
|
||||
rep = json.loads(out.stdout)
|
||||
if args.json:
|
||||
with open(args.json, "w") as f:
|
||||
json.dump(rep, f, indent=1)
|
||||
|
||||
unreach_ids = {u[0] for u in unreachable}
|
||||
parse_failures = [f for f in rep["failed"] if f["api"] not in unreach_ids]
|
||||
gone = [f for f in rep["failed"] if f["api"] in unreach_ids or "NOT_FOUND" in f["why"]]
|
||||
print("preferred APIs in the directory : %d" % rep["preferred"])
|
||||
print("documents loaded by the engine : %d" % rep["loaded"])
|
||||
print("UNREACHABLE at the source : %d" % len(gone))
|
||||
for f in gone:
|
||||
print(" %-40s %s" % (f["api"], f["why"][:90]))
|
||||
parse_failures = [f for f in parse_failures if f not in gone]
|
||||
print("documents the engine REJECTS : %d" % len(parse_failures))
|
||||
for f in parse_failures:
|
||||
print(" %-40s %s" % (f["api"], f["why"][:160]))
|
||||
print("methods : %d" % rep["methods"])
|
||||
print(" full / limited / rejected : %d / %d / %d" % (rep["full"], rep["limited"], rep["rejected"]))
|
||||
print(" pageable (pageToken) : %d" % rep["pageable"])
|
||||
print(" long-running replies : %d (%d in APIs with no operations.get)" %
|
||||
(rep["longRunning"], rep["longRunningWithoutGetter"]))
|
||||
print("dry plans (required params only): %d planned, %d skipped (synthetic value vs exotic pattern), %d FAILED" %
|
||||
(rep["planned"], rep["planSkippedPattern"], rep["planFailed"]))
|
||||
for f in rep["planFailures"][:20]:
|
||||
print(" %s: %s" % (f["method"], f["why"][:140]))
|
||||
print("APIs with regional endpoints : %d" % rep["apisWithRegionalEndpoints"])
|
||||
print("APIs declaring a batchPath : %d (batch endpoints are not used)" % rep["apisDeclaringBatchPath"])
|
||||
print("\nreasons (methods):")
|
||||
for k, v in sorted(rep["reasons"].items(), key=lambda kv: -kv[1]):
|
||||
print(" %6d %s" % (v, k))
|
||||
bad = [a for a in rep["apis"] if a["rejected"] or a["limited"]]
|
||||
bad.sort(key=lambda a: -(a["rejected"] * 1000 + a["limited"]))
|
||||
print("\nAPIs with rejected or limited methods: %d of %d" % (len(bad), len(rep["apis"])))
|
||||
for a in bad[:args.list]:
|
||||
sample = a["problems"][0]
|
||||
print(" %-34s rejected %-3d limited %-3d e.g. %s: %s" %
|
||||
(a["api"], a["rejected"], a["limited"], sample["method"], sample["notes"][0][:70]))
|
||||
if len(bad) > args.list:
|
||||
print(" ... %d more (use --json)" % (len(bad) - args.list))
|
||||
|
||||
code = 0
|
||||
if parse_failures or rep["planFailed"]:
|
||||
code = 1
|
||||
if args.max_rejected is not None and rep["rejected"] > args.max_rejected:
|
||||
print("FAIL: %d rejected methods exceed --max-rejected %d" % (rep["rejected"], args.max_rejected))
|
||||
code = 1
|
||||
return code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,220 @@
|
||||
#include "google/discovery/validate.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
#include <regex>
|
||||
|
||||
#include "google/discovery/caps.h"
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw GoogleError::invalid(why); }
|
||||
|
||||
std::string sval(const Json& o, const char* key) {
|
||||
const Json* v = o.find(key);
|
||||
return v && v->is_string() ? v->as_string() : "";
|
||||
}
|
||||
|
||||
bool all_digits(const std::string& s, bool allow_sign) {
|
||||
std::size_t i = allow_sign && !s.empty() && s[0] == '-' ? 1 : 0;
|
||||
if (i >= s.size() || s.size() - i > 20) return false;
|
||||
for (; i < s.size(); ++i)
|
||||
if (s[i] < '0' || s[i] > '9') return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool integral(double d) { return std::isfinite(d) && d == std::floor(d) && std::fabs(d) < 9.0e15; }
|
||||
|
||||
std::string int_text(double d) { return std::to_string(static_cast<long long>(d)); }
|
||||
|
||||
bool shape_ok(const std::string& format, const std::string& v) {
|
||||
auto digit = [](char c) { return c >= '0' && c <= '9'; };
|
||||
if (format == "date-time" || format == "google-datetime") {
|
||||
// 2026-10-25T10:00:00Z shape: YYYY-MM-DDT
|
||||
return v.size() >= 11 && digit(v[0]) && digit(v[1]) && digit(v[2]) && digit(v[3]) &&
|
||||
v[4] == '-' && v[7] == '-' && (v[10] == 'T' || v[10] == 't');
|
||||
}
|
||||
if (format == "google-duration") {
|
||||
if (v.size() < 2 || v.back() != 's') return false;
|
||||
bool dot = false;
|
||||
for (std::size_t i = v[0] == '-' ? 1 : 0; i + 1 < v.size(); ++i) {
|
||||
if (v[i] == '.' && !dot) dot = true;
|
||||
else if (!digit(v[i])) return false;
|
||||
}
|
||||
return v.size() > 1;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void control_free(const std::string& what, const std::string& v) {
|
||||
for (unsigned char c : v)
|
||||
if (c < 0x20 || c == 0x7f) bad(what + " contains a control character");
|
||||
}
|
||||
|
||||
bool pattern_ok(const std::string& pattern, const std::string& v) {
|
||||
try {
|
||||
std::regex re(pattern, std::regex::ECMAScript | std::regex::optimize);
|
||||
return std::regex_search(v, re);
|
||||
} catch (const std::regex_error&) {
|
||||
return true; // pattern syntax std::regex cannot read: Google checks it
|
||||
}
|
||||
}
|
||||
|
||||
std::string one(const Param& p, const Json& v) {
|
||||
const std::string who = "parameter " + p.name;
|
||||
std::string out;
|
||||
if (p.type == "string") {
|
||||
if (v.is_string()) out = v.as_string();
|
||||
else if (v.is_number() && (p.format == "int64" || p.format == "uint64") && integral(v.as_number()))
|
||||
out = int_text(v.as_number());
|
||||
else bad(who + " must be a string");
|
||||
if (p.format == "int64" && !all_digits(out, true)) bad(who + " must be a 64-bit integer");
|
||||
if (p.format == "uint64" && !all_digits(out, false)) bad(who + " must be an unsigned 64-bit integer");
|
||||
if (p.format == "uint64" && out.size() > 20) bad(who + " is too large");
|
||||
if (!shape_ok(p.format, out)) bad(who + " is not a valid " + p.format);
|
||||
} else if (p.type == "integer") {
|
||||
double d;
|
||||
if (v.is_number()) d = v.as_number();
|
||||
else if (v.is_string() && all_digits(v.as_string(), true)) d = std::stod(v.as_string());
|
||||
else bad(who + " must be an integer");
|
||||
if (!integral(d)) bad(who + " must be an integer");
|
||||
if (p.format == "int32" && (d < -2147483648.0 || d > 2147483647.0)) bad(who + " is outside int32");
|
||||
if (p.format == "uint32" && (d < 0 || d > 4294967295.0)) bad(who + " is outside uint32");
|
||||
if (p.minimum && d < *p.minimum) bad(who + " is below the minimum " + int_text(*p.minimum));
|
||||
if (p.maximum && d > *p.maximum) bad(who + " is above the maximum " + int_text(*p.maximum));
|
||||
out = int_text(d);
|
||||
} else if (p.type == "number") {
|
||||
if (!v.is_number() || !std::isfinite(v.as_number())) bad(who + " must be a number");
|
||||
if (p.minimum && v.as_number() < *p.minimum) bad(who + " is below the minimum");
|
||||
if (p.maximum && v.as_number() > *p.maximum) bad(who + " is above the maximum");
|
||||
Json tmp(v.as_number());
|
||||
out = tmp.dump();
|
||||
} else if (p.type == "boolean") {
|
||||
if (v.is_bool()) out = v.as_bool() ? "true" : "false";
|
||||
else if (v.is_string() && (v.as_string() == "true" || v.as_string() == "false")) out = v.as_string();
|
||||
else bad(who + " must be true or false");
|
||||
} else {
|
||||
bad(who + " has an unsupported type " + p.type);
|
||||
}
|
||||
if (out.size() > DiscoveryCaps::kMaxParamChars) bad(who + " is longer than the cap");
|
||||
control_free(who, out);
|
||||
if (!p.enum_values.empty() &&
|
||||
std::find(p.enum_values.begin(), p.enum_values.end(), out) == p.enum_values.end())
|
||||
bad(who + " is not one of its enum values");
|
||||
if (!p.pattern.empty() && !pattern_ok(p.pattern, out)) bad(who + " does not match " + p.pattern);
|
||||
if (p.location == "path") {
|
||||
if (out.empty()) bad(who + " is empty");
|
||||
// No dot segments: a path value must not climb out of its template.
|
||||
std::size_t start = 0;
|
||||
while (start <= out.size()) {
|
||||
std::size_t end = out.find('/', start);
|
||||
std::string seg = out.substr(start, end == std::string::npos ? std::string::npos : end - start);
|
||||
if (seg == "." || seg == "..") bad(who + " contains a dot segment");
|
||||
if (end == std::string::npos) break;
|
||||
start = end + 1;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::vector<std::string> ParamValidator::values(const Param& p, const Json& value) {
|
||||
if (value.is_null()) bad("parameter " + p.name + " is null");
|
||||
std::vector<std::string> out;
|
||||
if (value.is_array()) {
|
||||
if (!p.repeated) bad("parameter " + p.name + " is not repeated, got an array");
|
||||
if (value.as_array().size() > DiscoveryCaps::kMaxRepeated)
|
||||
bad("parameter " + p.name + " has more than " +
|
||||
std::to_string(DiscoveryCaps::kMaxRepeated) + " values");
|
||||
for (const Json& v : value.as_array()) out.push_back(one(p, v));
|
||||
return out;
|
||||
}
|
||||
if (value.is_object()) bad("parameter " + p.name + " must be a scalar");
|
||||
out.push_back(one(p, value));
|
||||
return out;
|
||||
}
|
||||
|
||||
struct BodyValidator::Walk {
|
||||
std::size_t visits = 0;
|
||||
};
|
||||
|
||||
void BodyValidator::validate(const std::string& ref, const Json& body) const {
|
||||
if (body.dump().size() > DiscoveryCaps::kMaxBodyBytes) bad("request body is larger than the cap");
|
||||
const Json* schema = api_.schema(ref);
|
||||
if (!schema) bad("unknown request schema " + ref);
|
||||
Walk walk;
|
||||
node(*schema, body, ref, walk, 0);
|
||||
}
|
||||
|
||||
void BodyValidator::node(const Json& schema_in, const Json& value, const std::string& at,
|
||||
Walk& walk, std::size_t depth) const {
|
||||
if (++walk.visits > DiscoveryCaps::kMaxSchemaVisits) bad("request body is too large to validate");
|
||||
if (depth > DiscoveryCaps::kMaxSchemaDepth)
|
||||
bad("request body is nested deeper than " + std::to_string(DiscoveryCaps::kMaxSchemaDepth));
|
||||
const Json* schema = &schema_in;
|
||||
for (int hops = 0; schema->find("$ref"); ++hops) {
|
||||
if (hops > 16) bad("schema $ref chain too long at " + at);
|
||||
const Json* next = api_.schema(sval(*schema, "$ref"));
|
||||
if (!next) bad("unknown schema " + sval(*schema, "$ref") + " at " + at);
|
||||
schema = next;
|
||||
}
|
||||
if (value.is_null()) return; // proto JSON: null is the unset value
|
||||
std::string type = sval(*schema, "type");
|
||||
if (type.empty() && schema->find("properties")) type = "object";
|
||||
std::string format = sval(*schema, "format");
|
||||
if (type == "any" || type.empty()) return;
|
||||
if (type == "string") {
|
||||
std::string text;
|
||||
if (value.is_string()) text = value.as_string();
|
||||
else if (value.is_number() && (format == "int64" || format == "uint64") && integral(value.as_number()))
|
||||
text = int_text(value.as_number());
|
||||
else bad(at + " must be a string");
|
||||
if ((format == "int64" && !all_digits(text, true)) || (format == "uint64" && !all_digits(text, false)))
|
||||
bad(at + " must be a " + format + " number");
|
||||
if (const Json* e = schema->find("enum"); e && e->is_array()) {
|
||||
bool hit = false;
|
||||
for (const Json& v : e->as_array()) hit = hit || (v.is_string() && v.as_string() == text);
|
||||
if (!hit) bad(at + " is not one of its enum values: " + text);
|
||||
}
|
||||
} else if (type == "integer") {
|
||||
if (!value.is_number() || !integral(value.as_number())) bad(at + " must be an integer");
|
||||
double d = value.as_number();
|
||||
if (format == "int32" && (d < -2147483648.0 || d > 2147483647.0)) bad(at + " is outside int32");
|
||||
if (format == "uint32" && (d < 0 || d > 4294967295.0)) bad(at + " is outside uint32");
|
||||
} else if (type == "number") {
|
||||
if (!value.is_number()) bad(at + " must be a number");
|
||||
} else if (type == "boolean") {
|
||||
if (!value.is_bool()) bad(at + " must be true or false");
|
||||
} else if (type == "array") {
|
||||
if (!value.is_array()) bad(at + " must be an array");
|
||||
const Json* items = schema->find("items");
|
||||
std::size_t i = 0;
|
||||
for (const Json& v : value.as_array()) {
|
||||
if (items && items->is_object()) node(*items, v, at + "/" + std::to_string(i), walk, depth + 1);
|
||||
++i;
|
||||
}
|
||||
} else if (type == "object") {
|
||||
if (!value.is_object()) bad(at + " must be an object");
|
||||
const Json* props = schema->find("properties");
|
||||
const Json* extra = schema->find("additionalProperties");
|
||||
bool has_props = props && props->is_object() && !props->as_object().empty();
|
||||
bool has_extra = extra && extra->is_object();
|
||||
for (const auto& [k, v] : value.as_object()) {
|
||||
const Json* ps = has_props ? props->find(k) : nullptr;
|
||||
if (ps) node(*ps, v, at + "/" + k, walk, depth + 1);
|
||||
else if (has_extra) node(*extra, v, at + "/" + k, walk, depth + 1);
|
||||
else if (has_props) bad(at + " has no property " + k);
|
||||
// neither: free-form object, anything goes
|
||||
}
|
||||
} else {
|
||||
bad(at + " has unsupported schema type " + type);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
@@ -0,0 +1,38 @@
|
||||
#pragma once
|
||||
|
||||
// Validation before traffic: parameter values against their Discovery
|
||||
// declaration, request bodies against the document's schemas. Every failure
|
||||
// throws GoogleError::invalid naming the parameter or JSON path.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "google/discovery/model.h"
|
||||
|
||||
namespace molo::google::discovery {
|
||||
|
||||
class ParamValidator {
|
||||
public:
|
||||
// Canonical string form(s) of a value: one per element for repeated
|
||||
// parameters. Checks type, format shape, enum, pattern, minimum, maximum,
|
||||
// length and the repeated cap.
|
||||
static std::vector<std::string> values(const Param& param,
|
||||
const clients::body::Json& value);
|
||||
};
|
||||
|
||||
class BodyValidator {
|
||||
public:
|
||||
explicit BodyValidator(const Api& api) : api_(api) {}
|
||||
// Validates `body` against the schema named `ref`. Unknown properties,
|
||||
// wrong types, values outside an enum, depth and size overruns throw.
|
||||
void validate(const std::string& ref, const clients::body::Json& body) const;
|
||||
|
||||
private:
|
||||
struct Walk;
|
||||
void node(const clients::body::Json& schema, const clients::body::Json& value,
|
||||
const std::string& at, Walk& walk, std::size_t depth) const;
|
||||
const Api& api_;
|
||||
};
|
||||
|
||||
} // namespace molo::google::discovery
|
||||
Reference in New Issue
Block a user