feat(clients): retrieval: Voyage, Jina, Pinecone, Qdrant, Weaviate, Mixedbread, Tavily, Exa, Brave
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Vendored
+224
@@ -0,0 +1,224 @@
|
||||
# Retrieval vendors - Specification
|
||||
|
||||
Owned directory: `src/vendors/retrieval/`. Namespace `molo::retrieval`. One
|
||||
subdirectory per provider. Depends on `molo::llm` (`ApiClient`, `Config`,
|
||||
`ApiError`) and `molo::clients`. Plan: `plans/18-breadth.md` (rules of
|
||||
`plans/17-llm-vendor-platforms.md` apply). `src/llm` is not edited.
|
||||
|
||||
Providers: Voyage AI, Jina AI, Pinecone, Qdrant, Weaviate, Mixedbread, Tavily,
|
||||
Exa, Brave Search.
|
||||
|
||||
## 1. Module map
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings, never autotuned (documented limits and our own) |
|
||||
| `call_table.h/.cpp` | `CallRow`, `CallTable`, `RowInfo` (flags, method, unverified) |
|
||||
| `gateway.h/.cpp` | `Gateway`: validates a call against its row (path ids, query, body fields, sizes), renders it, sends it. `call()` refuses destructive rows, `destroy()` only runs them, `plan()` renders without traffic |
|
||||
| `provider.h/.cpp` | `make_gateway` (one `ApiClient` per host), `check_host`, `Provider` base, `J` Json helpers |
|
||||
| `seams.h/.cpp` | Neutral `Embedder`, `Reranker`, `WebSearch` seams and their request validators |
|
||||
| `voyage/` | embeddings, multimodal, contextualized, rerank; `Embedder` + `Reranker` |
|
||||
| `jina/` | embeddings, rerank, models, batch (api.jina.ai), Reader, Search, Segmenter, Classifier; `Embedder` + `Reranker` + `WebSearch` |
|
||||
| `pinecone/` | control plane, data plane (index host), inference; `Embedder` + `Reranker` |
|
||||
| `qdrant/` | collections, points, query, snapshots, cluster |
|
||||
| `weaviate/` | schema, objects, batch, tenants, REST search, GraphQL over HTTP |
|
||||
| `mixedbread/` | embeddings, reranking, stores, files, parsing, extraction; `Embedder` + `Reranker` |
|
||||
| `tavily/` | search, extract, crawl, map, research, usage; `WebSearch` |
|
||||
| `exa/` | search, contents, findSimilar, answer, research; `WebSearch` |
|
||||
| `brave/` | web, news, images, videos, suggest, spellcheck, local, LLM context, answers; `WebSearch` |
|
||||
| `tools/check_spec.py` | Verifies every call table against the provider spec |
|
||||
|
||||
Each provider directory holds `calls.cpp` (the table, generated from the spec
|
||||
by `check_spec.py --rows <provider>` and then reviewed) and `<provider>.h/.cpp`
|
||||
(typed class). `calls.cpp` is the only place paths are written.
|
||||
|
||||
## 2. Seams
|
||||
|
||||
`Embedder::embed(EmbedRequest) -> EmbedResult`, `Reranker::rerank(RerankRequest)
|
||||
-> RerankResult`, `WebSearch::search(SearchRequest) -> SearchResult`. Shared
|
||||
validators reject blank or oversized input before any traffic. An adapter whose
|
||||
provider cannot honour a field says so (Jina search has no `recency`, throws).
|
||||
A reply with a vector count different from the input count is a `BAD_RESPONSE`.
|
||||
|
||||
## 3. Credentials
|
||||
|
||||
| Provider | Env | Auth header |
|
||||
|---|---|---|
|
||||
| Voyage | `VOYAGE_API_KEY` | `Authorization: Bearer` |
|
||||
| Jina | `JINA_API_KEY` | `Authorization: Bearer` |
|
||||
| Pinecone | `PINECONE_API_KEY` | `Api-Key` plus `X-Pinecone-Api-Version: 2026-07` |
|
||||
| Qdrant | `QDRANT_API_KEY` and `QDRANT_URL` | `api-key` |
|
||||
| Weaviate | `WEAVIATE_API_KEY` and `WEAVIATE_URL` (the `/v1` base is added) | `Authorization: Bearer` |
|
||||
| Mixedbread | `MXBAI_API_KEY` | `Authorization: Bearer` |
|
||||
| Tavily | `TAVILY_API_KEY` | `Authorization: Bearer` |
|
||||
| Exa | `EXA_API_KEY` | `x-api-key` |
|
||||
| Brave | `BRAVE_API_KEY` | `X-Subscription-Token` |
|
||||
|
||||
Key pages (unverified, the vendors' usual consoles): dash.voyageai.com,
|
||||
jina.ai/api-dashboard, app.pinecone.io (API keys), cloud.qdrant.io, console.weaviate.cloud,
|
||||
mixedbread.com (platform), app.tavily.com, dashboard.exa.ai, api-dashboard.search.brave.com.
|
||||
|
||||
## 4. Contracts
|
||||
|
||||
- No key (or no `QDRANT_URL` / `WEAVIATE_URL`): `configured()` is false, every
|
||||
call throws `NotConfiguredError` (503) before any traffic.
|
||||
- Rows carry flags: `d` destructive, `w` writes, `r` read-only POST, `p` paid.
|
||||
The demo runs `d`, `w` and `p` rows only with `"confirm":true`.
|
||||
- Destructive calls are explicit methods (`Qdrant::delete_collection`,
|
||||
`Pinecone::delete_index`, `Pinecone::delete_ids/delete_all/delete_by_filter`,
|
||||
`Weaviate::delete_class/batch_delete`, `Mixedbread::delete_store`, ...) that
|
||||
go through `Gateway::destroy`. `Gateway::call` refuses them.
|
||||
- Path ids pass `llm::Service::valid_id` (1..256 of `A-Za-z0-9._:-`); values are
|
||||
percent-encoded again. Query parameters and body fields must be listed in
|
||||
the row, otherwise `ApiError::invalid` (400 `INVALID_ARGUMENT`).
|
||||
- A Pinecone data plane host must be a bare DNS name. The data plane is only
|
||||
reachable on a `Pinecone` built with that host.
|
||||
- Hard caps: see `caps.h` (embed 1000 texts, rerank 1000 documents, upsert 1000
|
||||
records, top_k 10000, request body 8 MiB, query value 8 KiB, list limit 100,
|
||||
Mixedbread embed 256 inputs, Tavily extract 20 URLs, Brave web count 20).
|
||||
List walks are never unbounded: one call is one page.
|
||||
|
||||
## 5. Sources
|
||||
|
||||
| Provider | Source | Revision seen 2026-10-25 |
|
||||
|---|---|---|
|
||||
| Voyage | official `voyageai` Python SDK wheel (PyPI): resources `embeddings`, `multimodalembeddings`, `contextualizedembeddings`, `rerank`, base URL, parameters, response objects. Voyage publishes no OpenAPI document | voyageai 0.5.0 |
|
||||
| Jina | OpenAPI of `api.jina.ai`, `r.jina.ai`, `s.jina.ai`, `segment.jina.ai` (`/openapi.json`) | api 2026.10.13, reader/search 0.5.0, segment 0.1.0 |
|
||||
| Pinecone | OpenAPI YAML `db_control`, `db_data`, `inference` of `pinecone-io/pinecone-api` | 2026-07 |
|
||||
| Qdrant | `qdrant/qdrant` `docs/redoc/master/openapi.json` | master |
|
||||
| Weaviate | `weaviate/weaviate` `openapi-specs/schema.json` (Swagger 2, basePath `/v1`) | 1.41.0-dev |
|
||||
| Mixedbread | https://api.mixedbread.com/openapi.json | 0.1.0 |
|
||||
| Tavily | https://docs.tavily.com/documentation/api-reference/openapi.json | 1.0.0 |
|
||||
| Exa | `exa-labs/openapi-spec` `exa-openapi-spec.yaml` | 1.2.0 |
|
||||
| Brave | the rendered API reference, one markdown page per operation (`api-dashboard.search.brave.com/api-reference/<slug>/index.html.md`), plus `llms.txt` | 2026-10-25 |
|
||||
|
||||
Each path and field was cross-checked between the spec and the call table by
|
||||
`tools/check_spec.py`; typed request builders were written from the same
|
||||
schemas. The Voyage rows rest on one source (the SDK).
|
||||
|
||||
## 6. Verified against spec
|
||||
|
||||
`python3 -I src/vendors/retrieval/tools/check_spec.py` (stdlib only, network),
|
||||
run 2026-10-25: **0 problems**.
|
||||
|
||||
| Provider | Spec operations | Table rows | Excluded (with reason in the script) |
|
||||
|---|---|---|---|
|
||||
| Qdrant | 69 | 49 | 20 (health, telemetry, issues, locks, shard placement, cluster membership) |
|
||||
| Weaviate | 123 | 66 | 57 (authz, users, groups, replication, nodes, cluster, backups, classifications, tasks, experimental user export) |
|
||||
| Pinecone | 57 | 57 | 0 (assistant, admin, nexus, oauth and metrics specs are other products and were not read) |
|
||||
| Mixedbread | 192 | 80 | 112 (integrations and OAuth callbacks, admin, billing, buckets, events, api-keys, data sources, trace destinations) |
|
||||
| Jina | 34 | 15 (13 spec + 2 prose) | 21 (chat completions, reader helper routes, path and POST duplicates of GET rows) |
|
||||
| Tavily | 10 | 7 | 3 (`feedback`, `logs`, `org-usage`: no operationId, account logging) |
|
||||
| Exa | 7 | 7 | 0 |
|
||||
| Brave | 16 reference pages | 16 | 0 |
|
||||
| Voyage | 4 SDK resources | 4 | 0 |
|
||||
|
||||
The checker fails on a row absent from the spec, a body or query field absent
|
||||
from the spec, an operationId mismatch, and a spec operation that is neither in
|
||||
the table nor excluded. Its YAML reader (Pinecone and Exa are YAML) is a small
|
||||
stdlib parser verified against PyYAML on those files (`--selftest`: paths equal).
|
||||
|
||||
## 7. Verified live
|
||||
|
||||
Unauthenticated probes, 2026-10-25 (invalid or missing key; no success body
|
||||
was seen from any provider):
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| `GET api.pinecone.io/indexes` without key | 401 `{"error":{"code":"UNAUTHENTICATED","message":"Missing api-key header"},"status":401}` |
|
||||
| `POST api.tavily.com/search` without key | 401 `{"detail":{"error":"Unauthorized: missing or invalid API key."}}` |
|
||||
| `GET api.jina.ai/v1/embeddings` without key | 401 `{"detail":"Authentication required. ..."}` |
|
||||
| `GET api.voyageai.com/v1/embeddings` | 405 `{"detail":"Method Not Allowed"}` (route exists) |
|
||||
| `GET api.mixedbread.com/v1/embeddings` | 405 `{"detail":"Method Not Allowed"}` (route exists) |
|
||||
| `GET api.exa.ai/search` | 404 `{"error":"Not found","tag":"NOT_FOUND"}` (POST-only route) |
|
||||
| any `api.search.brave.com/res/v1/*` with an invalid token | 422 `SUBSCRIPTION_TOKEN_INVALID` (auth runs before routing: even `/nonexistent` answers 422, so the probe proves host and TLS, not paths) |
|
||||
| `openapi.json` of mixedbread, jina (4 hosts), tavily | 200 JSON (the specs above) |
|
||||
|
||||
Through our own classes (`MOLO_LIVE_NET=1 make ai-test`, test
|
||||
`retrieval_live.unauthenticated_probes_answer_with_an_auth_status`, 2026-10-25),
|
||||
an invalid key gives:
|
||||
|
||||
| Provider (call) | Result |
|
||||
|---|---|
|
||||
| Voyage (embed) | 401 `UNKNOWN` (body `{"detail":...}` is not parsed, see Gaps) |
|
||||
| Jina (embeddings) | 401 `AUTH_INVALID_API_KEY` |
|
||||
| Pinecone (list indexes) | 401 `UNAUTHENTICATED` "Invalid API key" |
|
||||
| Tavily (usage) | 401 `UNKNOWN` |
|
||||
| Exa (search) | 401, message "Invalid API key. Provide a valid key using 'Authorization: Bearer <key>' or 'x-api-key: <key>'" (so Exa accepts both headers) |
|
||||
| Brave (web search) | 422 `SUBSCRIPTION_TOKEN_INVALID` |
|
||||
| Mixedbread (list stores) | not reachable: `api.mixedbread.com` negotiates only TLS 1.2 (checked with `openssl s_client`) and `src/clients` pins TLS 1.3, so the handshake fails ("record layer version error"). `curl` reaches it and gets 401 `auth_credentials_invalid_error`. The test prints a `FINDING` line instead of failing |
|
||||
|
||||
Qdrant and Weaviate are your own servers and were not probed.
|
||||
|
||||
## 8. Unverified
|
||||
|
||||
Shapes inferred rather than read from a spec or seen:
|
||||
|
||||
- Every success response body: parsers follow the spec schemas (Pinecone,
|
||||
Qdrant, Mixedbread, Jina embeddings and rerank, Exa, Tavily request side) or
|
||||
the SDK (Voyage), and the tests use hand-built fixtures labelled FIXTURE,
|
||||
never recorded replies.
|
||||
- Tavily response schemas are `inline` in its OpenAPI (not named); the parser
|
||||
reads `answer`, `results[].title/url/content/score/published_date`,
|
||||
`failed_results[].url` from the docs' usual shape.
|
||||
- Jina Reader and Search JSON replies (`data.title/url/content/usage.tokens`,
|
||||
`data[]`): their OpenAPI documents describe only request parameters. Reader
|
||||
and Search are called as `GET /?url=` and `GET /search?q=` with
|
||||
`Accept: application/json`, as the spec's query parameters allow.
|
||||
- Jina Segmenter `POST /` and Classifier `POST /v1/classify` come from prose
|
||||
docs only (the segment OpenAPI lists only `/countToken`; `api.jina.ai` lists no
|
||||
classifier). They are the two `doc:` rows.
|
||||
- Voyage default model names (`voyage-3.5`, `rerank-2.5`, `voyage-multimodal-3`,
|
||||
`voyage-context-3`) and the Jina classifier default model are not in any
|
||||
machine-readable source; the SDK's own default is the old `voyage-2`. Voyage
|
||||
`return_documents` for rerank is not in the SDK and is not sent.
|
||||
- Pinecone 2026-07 index creation uses `schema.fields` and `deployment`; the
|
||||
vector field name `vector` and the data plane vector shape
|
||||
(`vectors[].id/values/metadata`) follow the 2026-07 `db_data` spec but were
|
||||
never exercised against a real index.
|
||||
- Mixedbread query prompt for `input_type=query`
|
||||
(`Represent this sentence for searching relevant passages:`) is the model
|
||||
card's, not in the OpenAPI.
|
||||
- Brave Answers non-streaming reply: the reference documents the streamed
|
||||
`choices[].delta`; we send `stream:false` and parse `choices[0].message.content`
|
||||
with a fallback to `delta`.
|
||||
- Weaviate REST search (`/v1/search/{collection}/...`) is new in the spec
|
||||
(1.41 dev); request fields are from the spec, behaviour on released servers
|
||||
is unknown.
|
||||
- Brave `summarizer` endpoints: not in today's API reference (Answers replaced
|
||||
them); not wrapped, not probed.
|
||||
|
||||
## 9. Gaps
|
||||
|
||||
Foundation (`src/llm`, `src/clients`, owner of those files):
|
||||
|
||||
- `src/clients/transports/tls_wolfssl.cpp` pins TLS 1.3 (`wolfTLSv1_3_client_method`, min version 1.3). `api.mixedbread.com` is TLS 1.2 only, so the whole Mixedbread package is unreachable until the transport accepts 1.2. Every other host in this package negotiates 1.3.
|
||||
- `tests/src/llm/test_llm.cpp` unsets `MOLO_LIVE_NET` in the middle of the run, so live tests registered after it (this package's) see the variable gone; `test_live.cpp` here reads it once at start-up to work around that.
|
||||
|
||||
- `ApiClient` needs a key to be "configured": a self-hosted Qdrant or Weaviate
|
||||
without authentication cannot be called.
|
||||
- `request()` parses every success body as JSON: Jina batch output downloads,
|
||||
Mixedbread file downloads and Qdrant snapshot downloads (raw bytes) are not
|
||||
reachable through `Gateway`; `request_raw` exists but the table engine does
|
||||
not use it.
|
||||
- No multipart in the table engine: Qdrant snapshot upload and recover from an
|
||||
uploaded file, Mixedbread file upload and `upload_store_file`, Pinecone bulk
|
||||
import from a file are table rows but cannot carry their file.
|
||||
- `error_from_response` drops the `detail` envelope (Voyage, Jina, Tavily
|
||||
`{"detail":...}`): the status survives, the message becomes `HTTP <code>`.
|
||||
- No streaming for Tavily/Brave Answers/Exa answer.
|
||||
- `Url::encode_component` percent-encodes space as `%20`; both forms are fine.
|
||||
|
||||
Package:
|
||||
|
||||
- gRPC (Weaviate, Qdrant, Pinecone `proto`) is out of scope: the gRPC APIs are
|
||||
not HTTP and Weaviate has no HTTP gateway for them.
|
||||
- Pinecone assistant, admin, nexus, oauth and metrics products are not wrapped.
|
||||
- Mixedbread parsing, extraction, chat and responses rows exist in the table
|
||||
but have no typed methods; use the generic `/ai/retrieval/call` route or
|
||||
`Gateway::call`.
|
||||
- The typed classes cover the main flows; every other row is reachable through
|
||||
`Gateway::call/destroy` with the same validation.
|
||||
- No retry or rate-limit handling (each provider documents its own).
|
||||
- Live success paths are unverified until keys exist: `MOLO_LIVE_NET=1
|
||||
MOLO_LIVE_PAID=1 make ai-test` with the keys in `.env.json.local`.
|
||||
+172
@@ -0,0 +1,172 @@
|
||||
#include "vendors/retrieval/brave/brave.h"
|
||||
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
llm::ApiClientConfig Brave::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "brave";
|
||||
c.base_url = kBaseUrl;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = "X-Subscription-Token";
|
||||
c.default_headers["Accept"] = "application/json";
|
||||
return c;
|
||||
}
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("brave", why); }
|
||||
|
||||
void check_query(const std::string& q, std::size_t max_chars) {
|
||||
if (q.find_first_not_of(" \t\r\n") == std::string::npos) bad("query is blank");
|
||||
if (q.size() > max_chars) bad("query is over the documented length");
|
||||
}
|
||||
void check_count(int count, int max) {
|
||||
if (count < 1 || count > max) bad("count must be 1.." + std::to_string(max));
|
||||
}
|
||||
CallArgs query_args(const std::string& q, int count, const std::string& freshness) {
|
||||
CallArgs a;
|
||||
a.query["q"] = q;
|
||||
a.query["count"] = Query::number(count);
|
||||
if (!freshness.empty()) a.query["freshness"] = freshness;
|
||||
return a;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Brave Brave::from_env() { return Brave(make_gateway(brave_calls(), config(), {{"main", kBaseUrl}}, "", nullptr)); }
|
||||
Brave Brave::with(const TransportFactory& factory, std::string key) {
|
||||
return Brave(make_gateway(brave_calls(), config(), {{"main", kBaseUrl}}, std::move(key), factory));
|
||||
}
|
||||
|
||||
std::string Brave::freshness_of(const std::string& r) {
|
||||
if (r == "day") return "pd";
|
||||
if (r == "week") return "pw";
|
||||
if (r == "month") return "pm";
|
||||
if (r == "year") return "py";
|
||||
return "";
|
||||
}
|
||||
|
||||
SearchResult Brave::parse_web(const Json& doc) {
|
||||
SearchResult out;
|
||||
const Json* web = J::at(doc, "web");
|
||||
if (!web) return out;
|
||||
for (const Json& r : J::array(*web, "results")) {
|
||||
SearchHit h;
|
||||
h.title = J::str(r, "title");
|
||||
h.url = J::str(r, "url");
|
||||
h.snippet = J::str(r, "description");
|
||||
h.published = J::str(r, "page_age");
|
||||
out.hits.push_back(std::move(h));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
SearchResult Brave::parse_results(const Json& doc) {
|
||||
SearchResult out;
|
||||
for (const Json& r : J::array(doc, "results")) {
|
||||
SearchHit h;
|
||||
h.title = J::str(r, "title");
|
||||
h.url = J::str(r, "url");
|
||||
h.snippet = J::str(r, "description");
|
||||
h.published = J::str(r, "page_age");
|
||||
if (h.published.empty()) h.published = J::str(r, "age");
|
||||
out.hits.push_back(std::move(h));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<BraveImage> Brave::parse_images(const Json& doc) {
|
||||
std::vector<BraveImage> out;
|
||||
for (const Json& r : J::array(doc, "results")) {
|
||||
BraveImage i;
|
||||
i.title = J::str(r, "title");
|
||||
i.page_url = J::str(r, "url");
|
||||
i.source = J::str(r, "source");
|
||||
if (const Json* p = J::at(r, "properties")) i.image_url = J::str(*p, "url");
|
||||
out.push_back(std::move(i));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<std::string> Brave::parse_suggest(const Json& doc) {
|
||||
std::vector<std::string> out;
|
||||
for (const Json& r : J::array(doc, "results")) out.push_back(J::str(r, "query"));
|
||||
return out;
|
||||
}
|
||||
|
||||
std::string Brave::parse_answer(const Json& doc) {
|
||||
const Json::Array& choices = J::array(doc, "choices");
|
||||
if (choices.empty()) return "";
|
||||
if (const Json* m = J::at(choices.front(), "message")) return J::str(*m, "content");
|
||||
if (const Json* d = J::at(choices.front(), "delta")) return J::str(*d, "content");
|
||||
return "";
|
||||
}
|
||||
|
||||
SearchResult Brave::search(const SearchRequest& request) {
|
||||
WebSearch::validate("brave", request);
|
||||
check_query(request.query, 600);
|
||||
check_count(request.max_results, 20);
|
||||
return parse_web(gw_->call("web_search", query_args(request.query, request.max_results, freshness_of(request.recency))));
|
||||
}
|
||||
|
||||
SearchResult Brave::news(const SearchRequest& request) {
|
||||
WebSearch::validate("brave", request);
|
||||
check_query(request.query, RetrievalCaps::kBraveQueryChars);
|
||||
check_count(request.max_results, RetrievalCaps::kBraveMaxCount);
|
||||
return parse_results(gw_->call("news_search", query_args(request.query, request.max_results, freshness_of(request.recency))));
|
||||
}
|
||||
|
||||
SearchResult Brave::videos(const SearchRequest& request) {
|
||||
WebSearch::validate("brave", request);
|
||||
check_query(request.query, RetrievalCaps::kBraveQueryChars);
|
||||
check_count(request.max_results, RetrievalCaps::kBraveMaxCount);
|
||||
return parse_results(gw_->call("video_search", query_args(request.query, request.max_results, freshness_of(request.recency))));
|
||||
}
|
||||
|
||||
std::vector<BraveImage> Brave::images(const std::string& query, int count) {
|
||||
check_query(query, RetrievalCaps::kBraveQueryChars);
|
||||
check_count(count, 200);
|
||||
return parse_images(gw_->call("image_search", query_args(query, count, "")));
|
||||
}
|
||||
|
||||
std::vector<std::string> Brave::suggest(const std::string& query, int count) {
|
||||
check_query(query, RetrievalCaps::kBraveQueryChars);
|
||||
check_count(count, 20);
|
||||
CallArgs a;
|
||||
a.query["q"] = query;
|
||||
a.query["count"] = Query::number(count);
|
||||
return parse_suggest(gw_->call("suggest", a));
|
||||
}
|
||||
|
||||
Json Brave::spellcheck(const std::string& query) {
|
||||
check_query(query, RetrievalCaps::kBraveQueryChars);
|
||||
CallArgs a;
|
||||
a.query["q"] = query;
|
||||
return gw_->call("spellcheck", a);
|
||||
}
|
||||
|
||||
Json Brave::llm_context(const std::string& query) {
|
||||
check_query(query, RetrievalCaps::kBraveQueryChars);
|
||||
CallArgs a;
|
||||
a.query["q"] = query;
|
||||
return gw_->call("llm_context", a);
|
||||
}
|
||||
|
||||
std::string Brave::answer(const std::string& prompt) {
|
||||
if (prompt.find_first_not_of(" \t\r\n") == std::string::npos || prompt.size() > RetrievalCaps::kMaxQueryChars)
|
||||
bad("prompt must be 1..8192 chars");
|
||||
Json msg = Json::object();
|
||||
msg.set("role", Json("user"));
|
||||
msg.set("content", Json(prompt));
|
||||
Json msgs = Json::array();
|
||||
msgs.push(msg);
|
||||
Json body = Json::object();
|
||||
body.set("messages", msgs);
|
||||
body.set("stream", Json(false));
|
||||
CallArgs a;
|
||||
a.body = body;
|
||||
return parse_answer(gw_->call("answers", a));
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+56
@@ -0,0 +1,56 @@
|
||||
#pragma once
|
||||
|
||||
// Brave Search API: web, news, images, videos, suggest, spellcheck, local POIs
|
||||
// and descriptions, place search, LLM context and Answers (chat completions).
|
||||
// Implements the WebSearch seam over web search. Brave's separate Summarizer
|
||||
// endpoints are no longer in the published API reference (Answers replaces
|
||||
// them), so they are not wrapped.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/retrieval/provider.h"
|
||||
#include "vendors/retrieval/seams.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallTable& brave_calls();
|
||||
|
||||
struct BraveImage {
|
||||
std::string title;
|
||||
std::string page_url;
|
||||
std::string image_url;
|
||||
std::string source;
|
||||
};
|
||||
|
||||
class Brave : public Provider, public WebSearch {
|
||||
public:
|
||||
static constexpr const char* kKeyEnv = "BRAVE_API_KEY";
|
||||
static constexpr const char* kBaseUrl = "https://api.search.brave.com/res";
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
static Brave from_env();
|
||||
static Brave with(const TransportFactory& factory, std::string key);
|
||||
explicit Brave(std::shared_ptr<Gateway> gateway) : Provider(std::move(gateway)) {}
|
||||
|
||||
std::string provider() const override { return "brave"; }
|
||||
SearchResult search(const SearchRequest& request) override; // web, count <= 20
|
||||
SearchResult news(const SearchRequest& request); // count <= 50
|
||||
SearchResult videos(const SearchRequest& request); // count <= 50
|
||||
std::vector<BraveImage> images(const std::string& query, int count); // count <= 200
|
||||
std::vector<std::string> suggest(const std::string& query, int count); // count <= 20
|
||||
Json spellcheck(const std::string& query);
|
||||
Json llm_context(const std::string& query);
|
||||
// One cited answer. Sends stream=false; the reply shape of the non-streaming
|
||||
// form is not documented (see SPEC.md ## Unverified).
|
||||
std::string answer(const std::string& prompt);
|
||||
|
||||
static SearchResult parse_web(const Json& doc);
|
||||
static SearchResult parse_results(const Json& doc); // news and videos
|
||||
static std::vector<BraveImage> parse_images(const Json& doc);
|
||||
static std::vector<std::string> parse_suggest(const Json& doc);
|
||||
static std::string parse_answer(const Json& doc);
|
||||
static std::string freshness_of(const std::string& recency);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
#include "vendors/retrieval/brave/brave.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// name, host, http method, path, spec operation id (or reference page / "doc:"),
|
||||
// body fields, query parameters, flags (d destructive, w writes, r read-only POST,
|
||||
// p paid), response schema)
|
||||
static const CallRow kRows[] = {
|
||||
{"web_search", "main", "GET", "/v1/web/search", "web/search/get", "", "count,country,enable_rich_callback,extra_snippets,freshness,goggles,goggles_id,include_fetch_metadata,offset,operators,q,result_filter,safesearch,search_lang,spellcheck,summary,text_decorations,ui_lang,units", "p", "-"},
|
||||
{"web_search_post", "main", "POST", "/v1/web/search", "web/search/post", "count,country,enable_rich_callback,extra_snippets,freshness,goggles,goggles_id,include_fetch_metadata,offset,operators,q,result_filter,safesearch,search_lang,spellcheck,summary,text_decorations,ui_lang,units", "", "rp", "-"},
|
||||
{"local_pois", "main", "GET", "/v1/local/pois", "web/local_pois", "", "ids,search_lang,ui_lang,units", "p", "-"},
|
||||
{"local_descriptions", "main", "GET", "/v1/local/descriptions", "web/poi_descriptions", "", "ids", "p", "-"},
|
||||
{"rich_search", "main", "GET", "/v1/web/rich", "web/rich_search", "", "callback_key", "p", "-"},
|
||||
{"place_search", "main", "GET", "/v1/local/place_search", "web/place_search", "", "count,country,geoloc,latitude,location,longitude,q,radius,safesearch,search_lang,spellcheck,ui_lang,units", "p", "-"},
|
||||
{"news_search", "main", "GET", "/v1/news/search", "news/news_search/get", "", "count,country,extra_snippets,freshness,goggles,include_fetch_metadata,offset,operators,q,safesearch,search_lang,spellcheck,ui_lang", "p", "-"},
|
||||
{"news_search_post", "main", "POST", "/v1/news/search", "news/news_search/post", "count,country,extra_snippets,freshness,goggles,include_fetch_metadata,offset,operators,q,safesearch,search_lang,spellcheck,ui_lang", "", "rp", "-"},
|
||||
{"image_search", "main", "GET", "/v1/images/search", "images/image_search", "", "count,country,q,safesearch,search_lang,spellcheck", "p", "-"},
|
||||
{"video_search", "main", "GET", "/v1/videos/search", "videos/video_search/get", "", "count,country,freshness,include_fetch_metadata,offset,operators,q,safesearch,search_lang,spellcheck,ui_lang", "p", "-"},
|
||||
{"video_search_post", "main", "POST", "/v1/videos/search", "videos/video_search/post", "count,country,freshness,include_fetch_metadata,offset,operators,q,safesearch,search_lang,spellcheck,ui_lang", "", "rp", "-"},
|
||||
{"suggest", "main", "GET", "/v1/suggest/search", "other/suggestions", "", "count,country,lang,q,rich", "p", "-"},
|
||||
{"spellcheck", "main", "GET", "/v1/spellcheck/search", "other/spell_check", "", "country,lang,q", "p", "-"},
|
||||
{"answers", "main", "POST", "/v1/chat/completions", "ai/answers", "country,enable_citations,enable_entities,enable_research,language,max_completion_tokens,messages,metadata,model,research_allow_thinking,research_maximum_number_of_iterations,research_maximum_number_of_queries,research_maximum_number_of_results_per_query,research_maximum_number_of_seconds,research_maximum_number_of_tokens_per_query,safesearch,seed,stream,web_search_options", "", "rp", "-"},
|
||||
{"llm_context", "main", "GET", "/v1/llm/context", "ai/llm_context/get", "", "context_threshold_mode,count,country,enable_local,enable_source_metadata,freshness,goggles,maximum_number_of_snippets,maximum_number_of_snippets_per_url,maximum_number_of_tokens,maximum_number_of_tokens_per_url,maximum_number_of_urls,operators,operators_fallback,q,safesearch,search_lang,spellcheck", "p", "-"},
|
||||
{"llm_context_post", "main", "POST", "/v1/llm/context", "ai/llm_context/post", "context_threshold_mode,count,country,enable_local,enable_source_metadata,freshness,goggles,maximum_number_of_snippets,maximum_number_of_snippets_per_url,maximum_number_of_tokens,maximum_number_of_tokens_per_url,maximum_number_of_urls,operators,operators_fallback,q,safesearch,search_lang,spellcheck", "", "rp", "-"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
const CallTable& brave_calls() {
|
||||
static const CallTable table{kRows, sizeof(kRows) / sizeof(kRows[0])};
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+41
@@ -0,0 +1,41 @@
|
||||
#include "vendors/retrieval/call_table.h"
|
||||
|
||||
#include <cstring>
|
||||
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallRow* CallTable::find(const std::string& name) const {
|
||||
for (std::size_t i = 0; i < count; ++i)
|
||||
if (name == rows[i].name) return &rows[i];
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
bool RowInfo::has(const CallRow& row, char flag) { return std::strchr(row.flags, flag) != nullptr && flag != '\0'; }
|
||||
|
||||
clients::Method RowInfo::method(const CallRow& row) {
|
||||
const char* m = row.method;
|
||||
if (!std::strcmp(m, "GET")) return clients::Method::kGet;
|
||||
if (!std::strcmp(m, "POST")) return clients::Method::kPost;
|
||||
if (!std::strcmp(m, "PUT")) return clients::Method::kPut;
|
||||
if (!std::strcmp(m, "PATCH")) return clients::Method::kPatch;
|
||||
if (!std::strcmp(m, "DELETE")) return clients::Method::kDelete;
|
||||
throw llm::ApiError::invalid("retrieval", std::string("bad method in call table: ") + m);
|
||||
}
|
||||
|
||||
bool RowInfo::listed(const char* csv, const std::string& field) {
|
||||
std::string s = csv;
|
||||
std::size_t pos = 0;
|
||||
while (pos <= s.size()) {
|
||||
std::size_t end = s.find(',', pos);
|
||||
if (end == std::string::npos) end = s.size();
|
||||
if (s.compare(pos, end - pos, field) == 0 && end - pos == field.size()) return !field.empty();
|
||||
pos = end + 1;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
bool RowInfo::unverified(const CallRow& row) { return std::strncmp(row.op, "doc:", 4) == 0; }
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+50
@@ -0,0 +1,50 @@
|
||||
#pragma once
|
||||
|
||||
// The one table of REST calls per provider. Every provider owns a calls.cpp
|
||||
// with a `CALLS-BEGIN ... CALLS-END` block of CallRow lines that
|
||||
// tools/check_spec.py compares with the provider's published spec. Code builds
|
||||
// every path from a row, never from a literal.
|
||||
|
||||
#include <cstddef>
|
||||
#include <string>
|
||||
|
||||
#include "clients/request.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// flags: d destructive, w writes, r read-only POST, p paid / metered.
|
||||
// A GET row has no write flag. `op` starting with "doc:" means the row comes
|
||||
// from prose documentation only (listed as unverified by check_spec.py).
|
||||
struct CallRow {
|
||||
const char* name; // snake_case id used by the demo and by tests
|
||||
const char* host; // which ApiClient serves it ("main", "control", ...)
|
||||
const char* method; // GET POST PUT PATCH DELETE
|
||||
const char* path; // from the host root, {placeholders} for path params
|
||||
const char* op; // spec operation id (or a reference page slug)
|
||||
const char* body; // comma list of top-level body fields, "*" = free form
|
||||
const char* query; // comma list of query parameters
|
||||
const char* flags;
|
||||
const char* resp; // response schema name in the spec, "-" when none
|
||||
};
|
||||
|
||||
struct CallTable {
|
||||
const CallRow* rows = nullptr;
|
||||
std::size_t count = 0;
|
||||
|
||||
const CallRow* find(const std::string& name) const;
|
||||
const CallRow& at(std::size_t index) const { return rows[index]; }
|
||||
std::size_t size() const { return count; }
|
||||
};
|
||||
|
||||
struct RowInfo {
|
||||
static bool has(const CallRow& row, char flag);
|
||||
static bool destructive(const CallRow& row) { return has(row, 'd'); }
|
||||
static bool paid(const CallRow& row) { return has(row, 'p'); }
|
||||
static bool writes(const CallRow& row) { return has(row, 'w') || has(row, 'd'); }
|
||||
static clients::Method method(const CallRow& row);
|
||||
// True when `field` is in a comma list.
|
||||
static bool listed(const char* csv, const std::string& field);
|
||||
static bool unverified(const CallRow& row);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+47
@@ -0,0 +1,47 @@
|
||||
#pragma once
|
||||
|
||||
// Static ceilings for the retrieval package. Never autotuned. "doc" numbers
|
||||
// come from the provider's published limits (see SPEC.md ## Sources), "own"
|
||||
// numbers are our hard caps.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
struct RetrievalCaps {
|
||||
// Request shape (own)
|
||||
static constexpr std::size_t kMaxBodyBytes = 8u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxQueryParams = 32;
|
||||
static constexpr std::size_t kMaxQueryValueBytes = 8192;
|
||||
static constexpr std::size_t kMaxPathParams = 4;
|
||||
static constexpr std::size_t kMaxHostChars = 255;
|
||||
// Embedding, rerank, search seams
|
||||
static constexpr std::size_t kMaxEmbedTexts = 1000; // Voyage doc: 1000 inputs
|
||||
static constexpr std::size_t kMaxTextBytes = 1024u * 1024u; // own, per text
|
||||
static constexpr std::size_t kMaxRerankDocs = 1000; // Voyage doc: 1000 documents
|
||||
static constexpr std::size_t kMaxQueryChars = 8192; // own (Brave doc: 400)
|
||||
static constexpr std::size_t kMaxSearchResults = 100; // own
|
||||
static constexpr std::size_t kMaxUrls = 100; // own (Tavily doc: 20)
|
||||
static constexpr int kMaxDimensions = 20000; // Pinecone doc: 20000
|
||||
static constexpr int kMaxTopN = 1000;
|
||||
// Vector databases
|
||||
static constexpr std::size_t kMaxRecordsPerUpsert = 1000; // Pinecone doc: 1000
|
||||
static constexpr std::size_t kMaxIdsPerFetch = 1000; // Pinecone doc: 1000
|
||||
static constexpr int kMaxTopK = 10000; // Pinecone doc: 10000
|
||||
static constexpr int kMaxListLimit = 100; // Pinecone doc: 100
|
||||
static constexpr std::size_t kMaxPointsPerUpsert = 1000; // own
|
||||
static constexpr int kMaxPointLimit = 10000; // own
|
||||
static constexpr std::size_t kMaxPages = 20; // list walks
|
||||
static constexpr std::size_t kMaxItemsReturned = 5000;
|
||||
// Brave
|
||||
static constexpr int kBraveMaxCount = 50; // doc: web 20, news 50
|
||||
static constexpr int kBraveMaxOffset = 9; // doc
|
||||
static constexpr std::size_t kBraveQueryChars = 400; // doc
|
||||
// Tavily and Exa
|
||||
static constexpr int kMaxCrawlLimit = 500; // own
|
||||
static constexpr int kMaxCrawlDepth = 5; // doc
|
||||
static constexpr int kTavilyMaxResults = 20; // doc
|
||||
static constexpr int kExaMaxResults = 100; // own
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+25
@@ -0,0 +1,25 @@
|
||||
#include "vendors/retrieval/exa/exa.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// name, host, http method, path, spec operation id (or reference page / "doc:"),
|
||||
// body fields, query parameters, flags (d destructive, w writes, r read-only POST,
|
||||
// p paid), response schema)
|
||||
static const CallRow kRows[] = {
|
||||
{"search", "main", "POST", "/search", "search", "additionalQueries,category,contents,context,endCrawlDate,endPublishedDate,excludeDomains,excludeText,includeDomains,includeText,moderation,numResults,outputSchema,query,startCrawlDate,startPublishedDate,type,userLocation", "", "rp", "-"},
|
||||
{"find_similar", "main", "POST", "/findSimilar", "findSimilar", "contents,context,endCrawlDate,endPublishedDate,excludeDomains,excludeSourceDomain,excludeText,includeDomains,includeText,moderation,numResults,startCrawlDate,startPublishedDate,url", "", "rp", "-"},
|
||||
{"contents", "main", "POST", "/contents", "getContents", "context,extras,highlights,ids,livecrawl,livecrawlTimeout,maxAgeHours,subpageTarget,subpages,summary,text,urls", "", "rp", "-"},
|
||||
{"answer", "main", "POST", "/answer", "answer", "outputSchema,query,stream,text", "", "rp", "-"},
|
||||
{"research_create", "main", "POST", "/research/v0/tasks", "research-tasks-create", "instructions,model,output", "", "rp", "-"},
|
||||
{"research_list", "main", "GET", "/research/v0/tasks", "research-tasks-list", "", "cursor,limit", "", "-"},
|
||||
{"research_get", "main", "GET", "/research/v0/tasks/{id}", "ResearchControllerV0_getResearchTask", "", "", "", "-"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
const CallTable& exa_calls() {
|
||||
static const CallTable table{kRows, sizeof(kRows) / sizeof(kRows[0])};
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+173
@@ -0,0 +1,173 @@
|
||||
#include "vendors/retrieval/exa/exa.h"
|
||||
|
||||
#include <chrono>
|
||||
#include <ctime>
|
||||
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
llm::ApiClientConfig Exa::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "exa";
|
||||
c.base_url = kBaseUrl;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = "x-api-key";
|
||||
return c;
|
||||
}
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("exa", why); }
|
||||
CallArgs with_body(Json body) {
|
||||
CallArgs a;
|
||||
a.body = std::move(body);
|
||||
return a;
|
||||
}
|
||||
void check_url(const std::string& url) {
|
||||
bool http = url.rfind("http://", 0) == 0 || url.rfind("https://", 0) == 0;
|
||||
if (!http || url.size() > 8192 || url.find_first_of(" \t\r\n") != std::string::npos)
|
||||
bad("url must be http(s) without spaces, at most 8192 chars");
|
||||
}
|
||||
Json text_contents(int max_characters) {
|
||||
Json text = Json::object();
|
||||
text.set("maxCharacters", Json(max_characters));
|
||||
Json c = Json::object();
|
||||
c.set("text", text);
|
||||
return c;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Exa Exa::from_env() { return Exa(make_gateway(exa_calls(), config(), {{"main", kBaseUrl}}, "", nullptr)); }
|
||||
Exa Exa::with(const TransportFactory& factory, std::string key) {
|
||||
return Exa(make_gateway(exa_calls(), config(), {{"main", kBaseUrl}}, std::move(key), factory));
|
||||
}
|
||||
|
||||
int Exa::recency_days(const std::string& r) {
|
||||
if (r == "day") return 1;
|
||||
if (r == "week") return 7;
|
||||
if (r == "month") return 31;
|
||||
if (r == "year") return 365;
|
||||
return 0;
|
||||
}
|
||||
|
||||
std::string Exa::days_ago_iso(long now_epoch_seconds, int days) {
|
||||
std::time_t t = static_cast<std::time_t>(now_epoch_seconds - static_cast<long>(days) * 86400L);
|
||||
std::tm tm{};
|
||||
gmtime_r(&t, &tm);
|
||||
char buf[32];
|
||||
std::strftime(buf, sizeof buf, "%Y-%m-%dT%H:%M:%S.000Z", &tm);
|
||||
return buf;
|
||||
}
|
||||
|
||||
SearchResult Exa::parse_results(const Json& doc) {
|
||||
SearchResult out;
|
||||
for (const Json& r : J::array(doc, "results")) {
|
||||
SearchHit h;
|
||||
h.title = J::str(r, "title");
|
||||
h.url = J::str(r, "url");
|
||||
h.published = J::str(r, "publishedDate");
|
||||
const Json::Array& hl = J::array(r, "highlights");
|
||||
if (!hl.empty() && hl.front().is_string()) h.snippet = hl.front().as_string();
|
||||
if (h.snippet.empty()) h.snippet = J::str(r, "summary");
|
||||
if (h.snippet.empty()) h.snippet = J::str(r, "text").substr(0, 500);
|
||||
if (const Json* s = J::at(r, "score"); s && s->is_number()) {
|
||||
h.score = s->as_number();
|
||||
h.has_score = true;
|
||||
}
|
||||
out.hits.push_back(std::move(h));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
ExaAnswer Exa::parse_answer(const Json& doc) {
|
||||
ExaAnswer out;
|
||||
out.answer = J::str(doc, "answer");
|
||||
for (const Json& c : J::array(doc, "citations")) {
|
||||
SearchHit h;
|
||||
h.title = J::str(c, "title");
|
||||
h.url = J::str(c, "url");
|
||||
h.published = J::str(c, "publishedDate");
|
||||
h.snippet = J::str(c, "text").substr(0, 500);
|
||||
out.citations.push_back(std::move(h));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
SearchResult Exa::search(const SearchRequest& request) { return search_typed(request, ""); }
|
||||
|
||||
SearchResult Exa::search_typed(const SearchRequest& request, const std::string& type) {
|
||||
WebSearch::validate("exa", request);
|
||||
if (request.max_results > RetrievalCaps::kExaMaxResults) bad("max_results is at most 100");
|
||||
static const char* types[] = {"", "neural", "fast", "auto", "deep", "deep-reasoning", "instant"};
|
||||
bool ok = false;
|
||||
for (const char* t : types) ok = ok || type == t;
|
||||
if (!ok) bad("type must be neural, fast, auto, deep, deep-reasoning or instant");
|
||||
Json body = Json::object();
|
||||
body.set("query", Json(request.query));
|
||||
body.set("numResults", Json(request.max_results));
|
||||
if (!type.empty()) body.set("type", Json(type));
|
||||
body.set("contents", text_contents(500));
|
||||
if (int days = recency_days(request.recency); days > 0) {
|
||||
long now = static_cast<long>(std::chrono::duration_cast<std::chrono::seconds>(
|
||||
std::chrono::system_clock::now().time_since_epoch()).count());
|
||||
body.set("startPublishedDate", Json(days_ago_iso(now, days)));
|
||||
}
|
||||
return parse_results(gw_->call("search", with_body(body)));
|
||||
}
|
||||
|
||||
SearchResult Exa::find_similar(const std::string& url, int num_results) {
|
||||
check_url(url);
|
||||
if (num_results < 1 || num_results > RetrievalCaps::kExaMaxResults) bad("num_results must be 1..100");
|
||||
Json body = Json::object();
|
||||
body.set("url", Json(url));
|
||||
body.set("numResults", Json(num_results));
|
||||
body.set("contents", text_contents(500));
|
||||
return parse_results(gw_->call("find_similar", with_body(body)));
|
||||
}
|
||||
|
||||
SearchResult Exa::contents(const std::vector<std::string>& urls, int max_characters) {
|
||||
if (urls.empty() || urls.size() > RetrievalCaps::kMaxUrls) bad("urls must hold 1..100 entries");
|
||||
for (const std::string& u : urls) check_url(u);
|
||||
if (max_characters < 1 || max_characters > 1000000) bad("max_characters must be 1..1000000");
|
||||
Json body = Json::object();
|
||||
body.set("urls", J::strings(urls));
|
||||
body.set("text", [&] {
|
||||
Json t = Json::object();
|
||||
t.set("maxCharacters", Json(max_characters));
|
||||
return t;
|
||||
}());
|
||||
return parse_results(gw_->call("contents", with_body(body)));
|
||||
}
|
||||
|
||||
ExaAnswer Exa::answer(const std::string& query) {
|
||||
if (query.empty() || query.size() > RetrievalCaps::kMaxQueryChars) bad("query must be 1..8192 chars");
|
||||
Json body = Json::object();
|
||||
body.set("query", Json(query));
|
||||
body.set("text", Json(false));
|
||||
return parse_answer(gw_->call("answer", with_body(body)));
|
||||
}
|
||||
|
||||
Json Exa::research_create(const std::string& instructions, const std::string& model) {
|
||||
if (instructions.empty() || instructions.size() > 4096) bad("instructions must be 1..4096 chars");
|
||||
if (!model.empty() && model != "exa-research" && model != "exa-research-pro") bad("model must be exa-research or exa-research-pro");
|
||||
Json body = Json::object();
|
||||
body.set("instructions", Json(instructions));
|
||||
if (!model.empty()) body.set("model", Json(model));
|
||||
return gw_->call("research_create", with_body(body));
|
||||
}
|
||||
|
||||
Json Exa::research_get(const std::string& id) {
|
||||
CallArgs a;
|
||||
a.path["id"] = id;
|
||||
return gw_->call("research_get", a);
|
||||
}
|
||||
|
||||
Json Exa::research_list(int limit) {
|
||||
if (limit < 1 || limit > 200) bad("limit must be 1..200");
|
||||
CallArgs a;
|
||||
a.query["limit"] = Query::number(limit);
|
||||
return gw_->call("research_list", a);
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+49
@@ -0,0 +1,49 @@
|
||||
#pragma once
|
||||
|
||||
// Exa: search, contents, findSimilar, answer and research tasks. Implements
|
||||
// the WebSearch seam.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/retrieval/provider.h"
|
||||
#include "vendors/retrieval/seams.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallTable& exa_calls();
|
||||
|
||||
struct ExaAnswer {
|
||||
std::string answer;
|
||||
std::vector<SearchHit> citations;
|
||||
};
|
||||
|
||||
class Exa : public Provider, public WebSearch {
|
||||
public:
|
||||
static constexpr const char* kKeyEnv = "EXA_API_KEY";
|
||||
static constexpr const char* kBaseUrl = "https://api.exa.ai";
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
static Exa from_env();
|
||||
static Exa with(const TransportFactory& factory, std::string key);
|
||||
explicit Exa(std::shared_ptr<Gateway> gateway) : Provider(std::move(gateway)) {}
|
||||
|
||||
std::string provider() const override { return "exa"; }
|
||||
// `type`: "" (auto), neural, fast, auto, deep, deep-reasoning, instant.
|
||||
SearchResult search(const SearchRequest& request) override;
|
||||
SearchResult search_typed(const SearchRequest& request, const std::string& type);
|
||||
SearchResult find_similar(const std::string& url, int num_results);
|
||||
SearchResult contents(const std::vector<std::string>& urls, int max_characters);
|
||||
ExaAnswer answer(const std::string& query);
|
||||
Json research_create(const std::string& instructions, const std::string& model);
|
||||
Json research_get(const std::string& id);
|
||||
Json research_list(int limit);
|
||||
|
||||
static SearchResult parse_results(const Json& doc);
|
||||
static ExaAnswer parse_answer(const Json& doc);
|
||||
// ISO-8601 UTC timestamp `days` before `now_epoch_seconds` (testable).
|
||||
static std::string days_ago_iso(long now_epoch_seconds, int days);
|
||||
static int recency_days(const std::string& recency);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+120
@@ -0,0 +1,120 @@
|
||||
#include "vendors/retrieval/gateway.h"
|
||||
|
||||
#include "clients/url.h"
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void bad(const std::string& vendor, const std::string& why) {
|
||||
throw llm::ApiError::invalid(vendor, why);
|
||||
}
|
||||
|
||||
std::vector<std::string> placeholders(const std::string& path) {
|
||||
std::vector<std::string> out;
|
||||
std::size_t pos = 0;
|
||||
while ((pos = path.find('{', pos)) != std::string::npos) {
|
||||
std::size_t end = path.find('}', pos);
|
||||
if (end == std::string::npos) break;
|
||||
out.push_back(path.substr(pos + 1, end - pos - 1));
|
||||
pos = end + 1;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool bad_text(const std::string& v) {
|
||||
for (unsigned char c : v)
|
||||
if (c < 0x20 || c == 0x7f) return true;
|
||||
return false;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Gateway::Gateway(std::string vendor, CallTable table, Hosts hosts)
|
||||
: vendor_(std::move(vendor)), table_(table), hosts_(std::move(hosts)) {}
|
||||
|
||||
bool Gateway::configured() const {
|
||||
if (hosts_.empty() || !missing_setting_.empty()) return false;
|
||||
for (const auto& [name, client] : hosts_)
|
||||
if (!client || !client->configured()) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
PlannedCall Gateway::plan(const std::string& name, const CallArgs& args) const {
|
||||
const CallRow* row = table_.find(name);
|
||||
if (!row) bad(vendor_, "unknown call: " + name);
|
||||
|
||||
// path parameters: exactly the placeholders, each a valid id segment
|
||||
std::string path = row->path;
|
||||
std::vector<std::string> wanted = placeholders(path);
|
||||
if (wanted.size() > RetrievalCaps::kMaxPathParams) bad(vendor_, "too many path parameters in the table");
|
||||
for (const auto& [k, v] : args.path) {
|
||||
bool known = false;
|
||||
for (const std::string& w : wanted) known = known || w == k;
|
||||
if (!known) bad(vendor_, "unexpected path parameter: " + k);
|
||||
}
|
||||
for (const std::string& w : wanted) {
|
||||
auto it = args.path.find(w);
|
||||
if (it == args.path.end()) bad(vendor_, "missing path parameter: " + w);
|
||||
if (!llm::Service::valid_id(it->second)) bad(vendor_, "bad " + w + " (1..256 of A-Z a-z 0-9 . _ : -)");
|
||||
std::string token = "{" + w + "}";
|
||||
path.replace(path.find(token), token.size(), clients::Url::encode_component(it->second));
|
||||
}
|
||||
|
||||
// query parameters: listed in the row, bounded
|
||||
if (args.query.size() + args.query_multi.size() > RetrievalCaps::kMaxQueryParams + RetrievalCaps::kMaxIdsPerFetch)
|
||||
bad(vendor_, "too many query parameters");
|
||||
if (args.query.size() > RetrievalCaps::kMaxQueryParams) bad(vendor_, "too many query parameters");
|
||||
std::vector<std::pair<std::string, std::string>> pairs(args.query.begin(), args.query.end());
|
||||
pairs.insert(pairs.end(), args.query_multi.begin(), args.query_multi.end());
|
||||
std::string qs;
|
||||
for (const auto& [k, v] : pairs) {
|
||||
if (!RowInfo::listed(row->query, k)) bad(vendor_, "query parameter not in the call table: " + k);
|
||||
if (v.size() > RetrievalCaps::kMaxQueryValueBytes) bad(vendor_, "query value too long: " + k);
|
||||
if (bad_text(v)) bad(vendor_, "query value has control characters: " + k);
|
||||
qs += qs.empty() ? '?' : '&';
|
||||
qs += clients::Url::encode_component(k) + "=" + clients::Url::encode_component(v);
|
||||
}
|
||||
|
||||
// body: object whose keys are listed, or free form
|
||||
std::string body_spec = row->body;
|
||||
if (!args.body.is_null()) {
|
||||
if (body_spec.empty()) bad(vendor_, "this call takes no body");
|
||||
if (!args.body.is_object() && body_spec != "*") bad(vendor_, "body must be a JSON object");
|
||||
if (args.body.is_object() && body_spec != "*")
|
||||
for (const auto& [k, v] : args.body.as_object())
|
||||
if (!RowInfo::listed(row->body, k)) bad(vendor_, "body field not in the call table: " + k);
|
||||
if (args.body.dump().size() > RetrievalCaps::kMaxBodyBytes) bad(vendor_, "body over the byte cap");
|
||||
}
|
||||
|
||||
if (hosts_.find(row->host) == hosts_.end()) bad(vendor_, std::string("no client for host ") + row->host);
|
||||
|
||||
PlannedCall p;
|
||||
p.name = row->name;
|
||||
p.vendor = vendor_;
|
||||
p.host = row->host;
|
||||
p.method = row->method;
|
||||
p.target = path + qs;
|
||||
p.body = args.body;
|
||||
p.flags = row->flags;
|
||||
p.destructive = RowInfo::destructive(*row);
|
||||
p.paid = RowInfo::paid(*row);
|
||||
p.writes = RowInfo::writes(*row);
|
||||
return p;
|
||||
}
|
||||
|
||||
Json Gateway::send(const std::string& name, const CallArgs& args, bool destructive) {
|
||||
PlannedCall p = plan(name, args);
|
||||
if (!missing_setting_.empty()) throw llm::NotConfiguredError(vendor_, missing_setting_);
|
||||
if (p.destructive && !destructive)
|
||||
bad(vendor_, name + " is destructive: use destroy()");
|
||||
if (!p.destructive && destructive) bad(vendor_, name + " is not destructive: use call()");
|
||||
const CallRow* row = table_.find(name);
|
||||
std::shared_ptr<llm::ApiClient> client = hosts_.at(row->host);
|
||||
const Json* body = args.body.is_null() ? nullptr : &args.body;
|
||||
return client->request(RowInfo::method(*row), p.target, body, args.headers);
|
||||
}
|
||||
|
||||
Json Gateway::call(const std::string& name, const CallArgs& args) { return send(name, args, false); }
|
||||
Json Gateway::destroy(const std::string& name, const CallArgs& args) { return send(name, args, true); }
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+81
@@ -0,0 +1,81 @@
|
||||
#pragma once
|
||||
|
||||
// Gateway: the table-driven engine behind every provider class. It validates a
|
||||
// call against its CallRow (path parameters, query parameters, body fields,
|
||||
// sizes), renders the request, and sends it through llm::ApiClient. Rows
|
||||
// flagged destructive are refused by call() and only reachable through the
|
||||
// explicit destroy(). plan() renders without sending (dry runs).
|
||||
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "llm/api_client.h"
|
||||
#include "vendors/retrieval/call_table.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
using Json = clients::body::Json;
|
||||
|
||||
struct CallArgs {
|
||||
std::map<std::string, std::string> path; // {placeholder} values, ids only
|
||||
std::map<std::string, std::string> query; // already text; see Query helpers
|
||||
// Repeated parameters (`ids=a&ids=b`); counted with `query` against the cap.
|
||||
std::vector<std::pair<std::string, std::string>> query_multi;
|
||||
Json body; // null = no body
|
||||
llm::Headers headers; // extra per-call headers
|
||||
};
|
||||
|
||||
struct PlannedCall {
|
||||
std::string name;
|
||||
std::string vendor;
|
||||
std::string host;
|
||||
std::string method;
|
||||
std::string target; // path and query string
|
||||
Json body;
|
||||
std::string flags;
|
||||
bool destructive = false;
|
||||
bool paid = false;
|
||||
bool writes = false;
|
||||
};
|
||||
|
||||
class Gateway {
|
||||
public:
|
||||
using Hosts = std::map<std::string, std::shared_ptr<llm::ApiClient>>;
|
||||
|
||||
Gateway(std::string vendor, CallTable table, Hosts hosts);
|
||||
|
||||
const std::string& vendor() const { return vendor_; }
|
||||
const CallTable& table() const { return table_; }
|
||||
// True when every host client has a credential and no setting is missing.
|
||||
bool configured() const;
|
||||
// Marks the gateway unusable until a setting exists (for example QDRANT_URL):
|
||||
// send() then throws NotConfiguredError(vendor, env) before any traffic.
|
||||
void require_setting(const std::string& env_name) { missing_setting_ = env_name; }
|
||||
const std::string& missing_setting() const { return missing_setting_; }
|
||||
const Hosts& hosts() const { return hosts_; }
|
||||
|
||||
// Validates and renders; throws ApiError::invalid (400 INVALID_ARGUMENT).
|
||||
PlannedCall plan(const std::string& name, const CallArgs& args = {}) const;
|
||||
// Sends a non-destructive row.
|
||||
Json call(const std::string& name, const CallArgs& args = {});
|
||||
// Sends a destructive row. No default arguments, on purpose.
|
||||
Json destroy(const std::string& name, const CallArgs& args);
|
||||
|
||||
private:
|
||||
Json send(const std::string& name, const CallArgs& args, bool destructive);
|
||||
|
||||
std::string vendor_;
|
||||
CallTable table_;
|
||||
Hosts hosts_;
|
||||
std::string missing_setting_;
|
||||
};
|
||||
|
||||
// Query text helpers: one place that decides how values are written.
|
||||
struct Query {
|
||||
static std::string flag(bool v) { return v ? "true" : "false"; }
|
||||
static std::string number(long v) { return std::to_string(v); }
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+33
@@ -0,0 +1,33 @@
|
||||
#include "vendors/retrieval/jina/jina.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// name, host, http method, path, spec operation id (or reference page / "doc:"),
|
||||
// body fields, query parameters, flags (d destructive, w writes, r read-only POST,
|
||||
// p paid), response schema)
|
||||
static const CallRow kRows[] = {
|
||||
{"embeddings", "api", "POST", "/v1/embeddings", "embeddings_v1_embeddings_post", "dimensions,embedding_type,input,input_type,late_chunking,model,normalized,return_multivector,return_tokenized_input,task,truncate", "", "rp", "EmbeddingResponse"},
|
||||
{"rerank", "api", "POST", "/v1/rerank", "rerank_v1_rerank_post", "documents,max_doc_length,model,query,return_documents,return_embeddings,top_n", "", "rp", "RerankingResponse"},
|
||||
{"list_models", "api", "GET", "/v1/models", "list_models_v1_models_get", "", "", "", "ModelListResponse"},
|
||||
{"get_model", "api", "GET", "/v1/models/{model_id}", "get_model_v1_models__model_id__get", "", "", "", "ModelInfo"},
|
||||
{"create_batch", "api", "POST", "/v1/batch/embeddings", "create_batch_job_v1_batch_embeddings_post", "dimensions,input,input_url,model,normalized,task,webhook_url", "", "wp", "BatchStatus"},
|
||||
{"get_batch", "api", "GET", "/v1/batch/{batch_id}", "get_batch_status_v1_batch__batch_id__get", "", "", "", "BatchStatus"},
|
||||
{"cancel_batch", "api", "DELETE", "/v1/batch/{batch_id}", "cancel_batch_job_v1_batch__batch_id__delete", "", "", "d", "BatchStatus"},
|
||||
{"list_batches", "api", "GET", "/v1/batches", "list_batch_jobs_v1_batches_get", "", "limit", "", "inline"},
|
||||
{"batch_output", "api", "GET", "/v1/batch/{batch_id}/output", "download_batch_output_v1_batch__batch_id__output_get", "", "", "", "-"},
|
||||
{"batch_errors", "api", "GET", "/v1/batch/{batch_id}/errors", "download_batch_errors_v1_batch__batch_id__errors_get", "", "", "", "-"},
|
||||
{"read", "reader", "GET", "/", "", "", "assertStatusCode,base,cacheTolerance,detachInvisibles,engine,exportStorageState,file,html,injectFrameScript,injectPageScript,instruction,jsonSchema,keepImgDataUrl,locale,markdownChunking,maxTokens,noCache,noGfm,noServiceWorker,page,pdf,preloadUrl,preset,proxy,proxyResources,proxyUrl,referer,removeOverlay,removeSelector,respondTiming,respondWith,retainImages,retainLinks,retainMedia,robotsTxt,setCookies,skipCertVerification,targetSelector,timeout,tokenBudget,url,userAgent,waitForSelector,webBotAuth,withGeneratedAlt,withIframe,withImagesSummary,withLinksSummary,withShadowDom,zeroDataRetention", "p", "-"},
|
||||
{"search", "search", "GET", "/search", "", "", "assertStatusCode,base,cacheTolerance,count,detachInvisibles,engine,exportStorageState,ext,fallback,file,filetype,gl,hl,html,injectFrameScript,injectPageScript,instruction,intitle,jsonSchema,keepImgDataUrl,loc,locale,location,markdownChunking,maxTokens,nfpr,noCache,noGfm,noServiceWorker,num,page,pdf,preloadUrl,preset,provider,proxy,proxyResources,proxyUrl,q,referer,removeOverlay,removeSelector,respondTiming,respondWith,retainImages,retainLinks,retainMedia,robotsTxt,setCookies,site,skipCertVerification,targetSelector,timeout,tokenBudget,type,url,userAgent,waitForSelector,webBotAuth,withGeneratedAlt,withIframe,withImagesSummary,withLinksSummary,withShadowDom,zeroDataRetention", "p", "inline"},
|
||||
{"count_token", "segment", "POST", "/countToken", "", "content,head,max_chunk_length,return_chunks,return_tokens,tail,tokenizer", "content,head,max_chunk_length,return_chunks,return_tokens,tail,tokenizer", "rp", "-"},
|
||||
{"segment", "segment", "POST", "/", "doc:segmenter", "content,tokenizer,return_tokens,return_chunks,max_chunk_length,head,tail", "", "p", "-"},
|
||||
{"classify", "api", "POST", "/v1/classify", "doc:classifier", "model,input,labels", "", "p", "-"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
const CallTable& jina_calls() {
|
||||
static const CallTable table{kRows, sizeof(kRows) / sizeof(kRows[0])};
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+217
@@ -0,0 +1,217 @@
|
||||
#include "vendors/retrieval/jina/jina.h"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
llm::ApiClientConfig Jina::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "jina";
|
||||
c.base_url = kApiUrl;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kBearer;
|
||||
return c;
|
||||
}
|
||||
|
||||
namespace {
|
||||
std::vector<HostSpec> hosts() {
|
||||
return {{"api", Jina::kApiUrl}, {"reader", Jina::kReaderUrl}, {"search", Jina::kSearchUrl}, {"segment", Jina::kSegmentUrl}};
|
||||
}
|
||||
llm::Headers json_accept() { return {{"Accept", "application/json"}}; }
|
||||
} // namespace
|
||||
|
||||
Jina Jina::from_env() { return Jina(make_gateway(jina_calls(), config(), hosts(), "", nullptr)); }
|
||||
Jina Jina::with(const TransportFactory& factory, std::string key) {
|
||||
return Jina(make_gateway(jina_calls(), config(), hosts(), std::move(key), factory));
|
||||
}
|
||||
|
||||
void Jina::check_url(const std::string& url) {
|
||||
bool http = url.rfind("http://", 0) == 0 || url.rfind("https://", 0) == 0;
|
||||
if (!http || url.size() > 8192 || url.find_first_of(" \t\r\n") != std::string::npos)
|
||||
throw llm::ApiError::invalid("jina", "url must be http(s) without spaces, at most 8192 chars");
|
||||
}
|
||||
|
||||
EmbedResult Jina::parse_embeddings(const Json& doc) {
|
||||
EmbedResult out;
|
||||
out.model = J::str(doc, "model");
|
||||
std::vector<std::pair<int, std::vector<float>>> rows;
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
const Json* e = J::at(d, "embedding");
|
||||
rows.emplace_back(static_cast<int>(J::num(d, "index", static_cast<double>(rows.size()))),
|
||||
e ? J::floats(*e) : std::vector<float>{});
|
||||
}
|
||||
std::stable_sort(rows.begin(), rows.end(), [](const auto& a, const auto& b) { return a.first < b.first; });
|
||||
for (auto& r : rows) out.vectors.push_back(std::move(r.second));
|
||||
if (const Json* u = J::at(doc, "usage")) out.total_tokens = static_cast<long>(J::num(*u, "total_tokens"));
|
||||
return out;
|
||||
}
|
||||
|
||||
RerankResult Jina::parse_rerank(const Json& doc) {
|
||||
RerankResult out;
|
||||
out.model = J::str(doc, "model");
|
||||
for (const Json& d : J::array(doc, "results")) {
|
||||
RerankHit h;
|
||||
h.index = static_cast<int>(J::num(d, "index"));
|
||||
h.score = J::num(d, "relevance_score");
|
||||
out.hits.push_back(h);
|
||||
}
|
||||
std::stable_sort(out.hits.begin(), out.hits.end(), [](const RerankHit& a, const RerankHit& b) { return a.score > b.score; });
|
||||
if (const Json* u = J::at(doc, "usage")) out.total_tokens = static_cast<long>(J::num(*u, "total_tokens"));
|
||||
return out;
|
||||
}
|
||||
|
||||
JinaPage Jina::parse_page(const Json& doc) {
|
||||
JinaPage p;
|
||||
const Json* data = J::at(doc, "data");
|
||||
const Json& d = data && data->is_object() ? *data : doc;
|
||||
p.title = J::str(d, "title");
|
||||
p.url = J::str(d, "url");
|
||||
p.description = J::str(d, "description");
|
||||
p.content = J::str(d, "content");
|
||||
if (const Json* u = J::at(d, "usage")) p.tokens = static_cast<long>(J::num(*u, "tokens"));
|
||||
return p;
|
||||
}
|
||||
|
||||
SearchResult Jina::parse_search(const Json& doc) {
|
||||
SearchResult out;
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
SearchHit h;
|
||||
h.title = J::str(d, "title");
|
||||
h.url = J::str(d, "url");
|
||||
h.snippet = J::str(d, "description");
|
||||
if (h.snippet.empty()) h.snippet = J::str(d, "content").substr(0, 500);
|
||||
out.hits.push_back(h);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
JinaSegments Jina::parse_segments(const Json& doc) {
|
||||
JinaSegments s;
|
||||
s.num_tokens = static_cast<long>(J::num(doc, "num_tokens"));
|
||||
s.num_chunks = static_cast<long>(J::num(doc, "num_chunks"));
|
||||
for (const Json& c : J::array(doc, "chunks"))
|
||||
if (c.is_string()) s.chunks.push_back(c.as_string());
|
||||
return s;
|
||||
}
|
||||
|
||||
std::vector<JinaPrediction> Jina::parse_classification(const Json& doc) {
|
||||
std::vector<JinaPrediction> out;
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
JinaPrediction p;
|
||||
p.index = static_cast<int>(J::num(d, "index", static_cast<double>(out.size())));
|
||||
p.label = J::str(d, "prediction");
|
||||
p.score = J::num(d, "score");
|
||||
out.push_back(p);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
EmbedResult Jina::embed(const EmbedRequest& request) {
|
||||
Embedder::validate("jina", request);
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(request.model.empty() ? std::string(kDefaultEmbedModel) : request.model));
|
||||
body.set("input", J::strings(request.texts));
|
||||
if (request.input_type == "query") body.set("task", Json("retrieval.query"));
|
||||
if (request.input_type == "document") body.set("task", Json("retrieval.passage"));
|
||||
if (request.dimensions > 0) body.set("dimensions", Json(request.dimensions));
|
||||
CallArgs args;
|
||||
args.body = body;
|
||||
EmbedResult r = parse_embeddings(gw_->call("embeddings", args));
|
||||
if (r.vectors.size() != request.texts.size())
|
||||
throw llm::ApiError("jina", 502, "BAD_RESPONSE", "embedding count differs from input count");
|
||||
return r;
|
||||
}
|
||||
|
||||
RerankResult Jina::rerank(const RerankRequest& request) {
|
||||
Reranker::validate("jina", request);
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(request.model.empty() ? std::string(kDefaultRerankModel) : request.model));
|
||||
body.set("query", Json(request.query));
|
||||
body.set("documents", J::strings(request.documents));
|
||||
body.set("return_documents", Json(false));
|
||||
if (request.top_n > 0) body.set("top_n", Json(request.top_n));
|
||||
CallArgs args;
|
||||
args.body = body;
|
||||
return parse_rerank(gw_->call("rerank", args));
|
||||
}
|
||||
|
||||
SearchResult Jina::search(const SearchRequest& request) {
|
||||
WebSearch::validate("jina", request);
|
||||
if (!request.recency.empty()) throw llm::ApiError::invalid("jina", "jina search has no recency filter");
|
||||
CallArgs args;
|
||||
args.query["q"] = request.query;
|
||||
args.headers = json_accept();
|
||||
return parse_search(gw_->call("search", args));
|
||||
}
|
||||
|
||||
JinaPage Jina::read(const std::string& url) {
|
||||
check_url(url);
|
||||
CallArgs args;
|
||||
args.query["url"] = url;
|
||||
args.headers = json_accept();
|
||||
return parse_page(gw_->call("read", args));
|
||||
}
|
||||
|
||||
JinaSegments Jina::segment(const std::string& content, const std::string& tokenizer, int max_chunk_length) {
|
||||
if (content.empty() || content.size() > 65536) throw llm::ApiError::invalid("jina", "content must be 1..65536 bytes");
|
||||
if (max_chunk_length < 0 || max_chunk_length > 8192) throw llm::ApiError::invalid("jina", "max_chunk_length out of range");
|
||||
Json body = Json::object();
|
||||
body.set("content", Json(content));
|
||||
if (!tokenizer.empty()) body.set("tokenizer", Json(tokenizer));
|
||||
body.set("return_chunks", Json(true));
|
||||
if (max_chunk_length > 0) body.set("max_chunk_length", Json(max_chunk_length));
|
||||
CallArgs args;
|
||||
args.body = body;
|
||||
return parse_segments(gw_->call("segment", args));
|
||||
}
|
||||
|
||||
long Jina::count_tokens(const std::string& content, const std::string& tokenizer) {
|
||||
if (content.empty() || content.size() > 65536) throw llm::ApiError::invalid("jina", "content must be 1..65536 bytes");
|
||||
CallArgs args;
|
||||
args.query["content"] = content;
|
||||
if (!tokenizer.empty()) args.query["tokenizer"] = tokenizer;
|
||||
Json doc = gw_->call("count_token", args);
|
||||
return static_cast<long>(J::num(doc, "num_tokens"));
|
||||
}
|
||||
|
||||
std::vector<JinaPrediction> Jina::classify(const std::vector<std::string>& texts,
|
||||
const std::vector<std::string>& labels, const std::string& model) {
|
||||
if (texts.empty() || texts.size() > RetrievalCaps::kMaxEmbedTexts) throw llm::ApiError::invalid("jina", "texts must hold 1..1000 entries");
|
||||
if (labels.size() < 2 || labels.size() > 256) throw llm::ApiError::invalid("jina", "labels must hold 2..256 entries");
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(model.empty() ? std::string(kDefaultClassifyModel) : model));
|
||||
body.set("input", J::strings(texts));
|
||||
body.set("labels", J::strings(labels));
|
||||
CallArgs args;
|
||||
args.body = body;
|
||||
return parse_classification(gw_->call("classify", args));
|
||||
}
|
||||
|
||||
Json Jina::list_models() { return gw_->call("list_models"); }
|
||||
|
||||
Json Jina::create_batch(const std::string& model, const std::string& input_url, const std::string& task) {
|
||||
check_url(input_url);
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(model));
|
||||
body.set("input_url", Json(input_url));
|
||||
if (!task.empty()) body.set("task", Json(task));
|
||||
CallArgs args;
|
||||
args.body = body;
|
||||
return gw_->call("create_batch", args);
|
||||
}
|
||||
|
||||
Json Jina::get_batch(const std::string& batch_id) {
|
||||
CallArgs args;
|
||||
args.path["batch_id"] = batch_id;
|
||||
return gw_->call("get_batch", args);
|
||||
}
|
||||
|
||||
Json Jina::cancel_batch(const std::string& batch_id) {
|
||||
CallArgs args;
|
||||
args.path["batch_id"] = batch_id;
|
||||
return gw_->destroy("cancel_batch", args);
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+80
@@ -0,0 +1,80 @@
|
||||
#pragma once
|
||||
|
||||
// Jina AI: embeddings, rerank, models and batch embeddings (api.jina.ai), the
|
||||
// Reader (r.jina.ai), Search (s.jina.ai), the Segmenter (segment.jina.ai) and
|
||||
// the Classifier. Implements Embedder, Reranker and WebSearch.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/retrieval/provider.h"
|
||||
#include "vendors/retrieval/seams.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallTable& jina_calls();
|
||||
|
||||
struct JinaPage {
|
||||
std::string title;
|
||||
std::string url;
|
||||
std::string description;
|
||||
std::string content; // markdown
|
||||
long tokens = 0;
|
||||
};
|
||||
|
||||
struct JinaSegments {
|
||||
long num_tokens = 0;
|
||||
long num_chunks = 0;
|
||||
std::vector<std::string> chunks;
|
||||
};
|
||||
|
||||
struct JinaPrediction {
|
||||
int index = 0;
|
||||
std::string label;
|
||||
double score = 0;
|
||||
};
|
||||
|
||||
class Jina : public Provider, public Embedder, public Reranker, public WebSearch {
|
||||
public:
|
||||
static constexpr const char* kKeyEnv = "JINA_API_KEY";
|
||||
static constexpr const char* kApiUrl = "https://api.jina.ai";
|
||||
static constexpr const char* kReaderUrl = "https://r.jina.ai";
|
||||
static constexpr const char* kSearchUrl = "https://s.jina.ai";
|
||||
static constexpr const char* kSegmentUrl = "https://segment.jina.ai";
|
||||
static constexpr const char* kDefaultEmbedModel = "jina-embeddings-v3";
|
||||
static constexpr const char* kDefaultRerankModel = "jina-reranker-v3";
|
||||
static constexpr const char* kDefaultClassifyModel = "jina-embeddings-v3";
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
static Jina from_env();
|
||||
static Jina with(const TransportFactory& factory, std::string key);
|
||||
explicit Jina(std::shared_ptr<Gateway> gateway) : Provider(std::move(gateway)) {}
|
||||
|
||||
std::string provider() const override { return "jina"; }
|
||||
EmbedResult embed(const EmbedRequest& request) override;
|
||||
RerankResult rerank(const RerankRequest& request) override;
|
||||
SearchResult search(const SearchRequest& request) override;
|
||||
|
||||
JinaPage read(const std::string& url);
|
||||
JinaSegments segment(const std::string& content, const std::string& tokenizer, int max_chunk_length);
|
||||
long count_tokens(const std::string& content, const std::string& tokenizer);
|
||||
std::vector<JinaPrediction> classify(const std::vector<std::string>& texts,
|
||||
const std::vector<std::string>& labels, const std::string& model);
|
||||
Json list_models();
|
||||
Json create_batch(const std::string& model, const std::string& input_url, const std::string& task);
|
||||
Json get_batch(const std::string& batch_id);
|
||||
// Destructive: cancels a running batch job.
|
||||
Json cancel_batch(const std::string& batch_id);
|
||||
|
||||
static EmbedResult parse_embeddings(const Json& doc);
|
||||
static RerankResult parse_rerank(const Json& doc);
|
||||
static JinaPage parse_page(const Json& doc);
|
||||
static SearchResult parse_search(const Json& doc);
|
||||
static JinaSegments parse_segments(const Json& doc);
|
||||
static std::vector<JinaPrediction> parse_classification(const Json& doc);
|
||||
|
||||
// http(s) URL, 1..8192 chars; throws ApiError::invalid.
|
||||
static void check_url(const std::string& url);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
#include "vendors/retrieval/mixedbread/mixedbread.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// name, host, http method, path, spec operation id (or reference page / "doc:"),
|
||||
// body fields, query parameters, flags (d destructive, w writes, r read-only POST,
|
||||
// p paid), response schema)
|
||||
static const CallRow kRows[] = {
|
||||
{"create_file", "main", "POST", "/v1/files", "create_file", "file", "", "w", "FileObject"},
|
||||
{"list_files", "main", "GET", "/v1/files", "list_files", "", "after,before,include_total,limit,q", "", "FileListResponse"},
|
||||
{"list_multipart_uploads", "main", "GET", "/v1/files/uploads", "list_multipart_uploads", "", "", "", "MultipartUploadListResponse"},
|
||||
{"create_multipart_upload", "main", "POST", "/v1/files/uploads", "create_multipart_upload", "file_size,filename,mime_type,part_count", "", "w", "CreateMultipartUploadResponse"},
|
||||
{"complete_multipart_upload", "main", "POST", "/v1/files/uploads/{upload_id}/complete", "complete_multipart_upload", "parts", "", "w", "FileObject"},
|
||||
{"abort_multipart_upload", "main", "POST", "/v1/files/uploads/{upload_id}/abort", "abort_multipart_upload", "", "", "d", "FileDeleted"},
|
||||
{"get_multipart_upload", "main", "GET", "/v1/files/uploads/{upload_id}", "get_multipart_upload", "", "", "", "MultipartUploadDetailResponse"},
|
||||
{"retrieve_file", "main", "GET", "/v1/files/{file_id}", "retrieve_file", "", "", "", "FileObject"},
|
||||
{"update_file", "main", "POST", "/v1/files/{file_id}", "update_file", "file", "", "w", "FileObject"},
|
||||
{"delete_file", "main", "DELETE", "/v1/files/{file_id}", "delete_file", "", "", "d", "FileDeleted"},
|
||||
{"download_file", "main", "GET", "/v1/files/{file_id}/content", "download_file", "", "", "", "inline"},
|
||||
{"list_store_events", "main", "GET", "/v1/stores/{store_identifier}/events", "list_store_events", "", "after,before,event_type,filter_after,filter_before,include_total,limit", "", "StoreEventListResponse"},
|
||||
{"get_store_event_histogram", "main", "POST", "/v1/stores/{store_identifier}/events/histogram", "get_store_event_histogram", "bucket_seconds,end_time,event_types,start_time", "", "r", "StoreEventHistogramResponse"},
|
||||
{"create_store_file", "main", "POST", "/v1/stores/{store_identifier}/files", "create_store_file", "config,experimental,external_id,file_id,metadata,overwrite", "", "w", "StoreFile"},
|
||||
{"list_store_files", "main", "GET", "/v1/stores/{store_identifier}/files", "list_store_files", "", "after,before,include_total,limit,sort_by,sort_order,statuses", "", "StoreFileListResponse"},
|
||||
{"upload_store_file", "main", "POST", "/v1/stores/{store_identifier}/files/upload", "upload_store_file", "file,params", "", "w", "StoreFile"},
|
||||
{"list_store_files_with_metadata_filter", "main", "POST", "/v1/stores/{store_identifier}/files/list", "list_store_files_with_metadata_filter", "after,before,include_total,limit,metadata_filter,q,sort_by,sort_order,statuses", "", "r", "StoreFileListResponse"},
|
||||
{"retrieve_store_file", "main", "GET", "/v1/stores/{store_identifier}/files/{file_identifier}", "retrieve_store_file", "", "return_chunks", "", "StoreFile"},
|
||||
{"update_store_file", "main", "PATCH", "/v1/stores/{store_identifier}/files/{file_identifier}", "update_store_file", "metadata", "", "w", "StoreFile"},
|
||||
{"delete_store_file", "main", "DELETE", "/v1/stores/{store_identifier}/files/{file_identifier}", "delete_store_file", "", "", "d", "StoreFileDeleted"},
|
||||
{"create_search_rule", "main", "POST", "/v1/stores/{store_identifier}/rules", "create_search_rule", "query,rules", "", "r", "mxbai_omni__api__routes__v1__stores__rules__models__SearchRuleResponse"},
|
||||
{"retrieve_search_rule", "main", "GET", "/v1/stores/{store_identifier}/rules/{rule_id}", "retrieve_search_rule", "", "", "", "mxbai_omni__api__routes__v1__stores__rules__models__SearchRuleResponse"},
|
||||
{"update_search_rule", "main", "PUT", "/v1/stores/{store_identifier}/rules/{rule_id}", "update_search_rule", "query,rules", "", "r", "mxbai_omni__api__routes__v1__stores__rules__models__SearchRuleResponse"},
|
||||
{"delete_search_rule", "main", "DELETE", "/v1/stores/{store_identifier}/rules/{rule_id}", "delete_search_rule", "", "", "d", "mxbai_omni__api__routes__v1__stores__rules__models__SearchRuleDeleted"},
|
||||
{"delete_specific_search_rule", "main", "DELETE", "/v1/stores/{store_identifier}/rules/{rule_id}/specific", "delete_specific_search_rule", "rule", "", "d", "mxbai_omni__api__routes__v1__stores__rules__models__SearchRuleSpecificDeleted"},
|
||||
{"create_store", "main", "POST", "/v1/stores", "create_store", "config,description,expires_after,file_ids,is_public,license,metadata,name,tags", "", "w", "Store"},
|
||||
{"list_stores", "main", "GET", "/v1/stores", "list_stores", "", "after,before,include_total,limit,q,tags", "", "StoreListResponse"},
|
||||
{"retrieve_store", "main", "GET", "/v1/stores/{store_identifier}", "retrieve_store", "", "", "", "Store"},
|
||||
{"update_store", "main", "PUT", "/v1/stores/{store_identifier}", "update_store", "description,expires_after,is_public,license,metadata,name,tags", "", "w", "Store"},
|
||||
{"delete_store", "main", "DELETE", "/v1/stores/{store_identifier}", "delete_store", "", "", "d", "StoreDeleted"},
|
||||
{"copy_store", "main", "POST", "/v1/stores/{store_identifier}/copy", "copy_store", "description,metadata,name,tags", "", "w", "Store"},
|
||||
{"train_store_lsf", "main", "POST", "/v1/stores/{store_identifier}/lsf/train", "train_store_lsf", "", "", "w", "StoreLsfTrainingQueued"},
|
||||
{"clear_store_lsf", "main", "DELETE", "/v1/stores/{store_identifier}/lsf", "clear_store_lsf", "", "", "d", "StoreLsfCleared"},
|
||||
{"search_store_chunks", "main", "POST", "/v1/stores/search", "search_store_chunks", "file_ids,filters,query,search_options,store_identifiers,stream,top_k", "", "rp", "StoreSearchResponse"},
|
||||
{"grep_store_chunks", "main", "POST", "/v1/stores/grep", "grep_store_chunks", "case_sensitive,file_ids,filters,pattern,return_metadata,store_identifiers,targets,top_k", "", "r", "StoreGrepResponse"},
|
||||
{"list_store_chunks", "main", "POST", "/v1/stores/list-chunks", "list_store_chunks", "file_ids,filters,search_options,sort_by,store_identifiers,top_k", "", "r", "StoreListChunksResponse"},
|
||||
{"create_question_answering", "main", "POST", "/v1/stores/question-answering", "create_question_answering", "file_ids,filters,instructions,qa_options,query,search_options,store_identifiers,stream,top_k", "", "rp", "StoreQAResults"},
|
||||
{"enhance_query", "main", "POST", "/v1/stores/queries/enhance", "enhance_query", "file_ids,filters,instructions,query,search_options,store_identifiers,top_k", "", "rp", "QueryEnhanceResults"},
|
||||
{"get_metadata_facets", "main", "POST", "/v1/stores/metadata-facets", "get_metadata_facets", "facets,file_ids,filters,max_fields,max_files,max_values_per_field,query,search_options,store_identifiers,top_k", "", "r", "MetadataFacets"},
|
||||
{"create_vector_store_file", "main", "POST", "/v1/vector_stores/{vector_store_identifier}/files", "create_vector_store_file", "config,experimental,external_id,file_id,metadata,overwrite", "", "w", "VectorStoreFile"},
|
||||
{"list_vector_store_files", "main", "GET", "/v1/vector_stores/{vector_store_identifier}/files", "list_vector_store_files", "", "after,before,include_total,limit,sort_by,sort_order,statuses", "", "VectorStoreFileListResponse"},
|
||||
{"list_vector_store_files_with_metadata_filter", "main", "POST", "/v1/vector_stores/{vector_store_identifier}/files/list", "list_vector_store_files_with_metadata_filter", "after,before,include_total,limit,metadata_filter,sort_by,sort_order,statuses", "", "r", "VectorStoreFileListResponse"},
|
||||
{"retrieve_vector_store_file", "main", "GET", "/v1/vector_stores/{vector_store_identifier}/files/{file_id}", "retrieve_vector_store_file", "", "return_chunks", "", "VectorStoreFile"},
|
||||
{"delete_vector_store_file", "main", "DELETE", "/v1/vector_stores/{vector_store_identifier}/files/{file_id}", "delete_vector_store_file", "", "", "d", "VectorStoreFileDeleted"},
|
||||
{"deprecated_create_search_rule", "main", "POST", "/v1/vector_stores/{vector_store_identifier}/rules", "deprecated_create_search_rule", "query,rules", "", "r", "mxbai_omni__api__routes__v1__deprecated_vector_stores__models__SearchRuleResponse"},
|
||||
{"deprecated_retrieve_search_rule", "main", "GET", "/v1/vector_stores/{vector_store_identifier}/rules/{rule_id}", "deprecated_retrieve_search_rule", "", "", "", "mxbai_omni__api__routes__v1__deprecated_vector_stores__models__SearchRuleResponse"},
|
||||
{"deprecated_update_search_rule", "main", "PUT", "/v1/vector_stores/{vector_store_identifier}/rules/{rule_id}", "deprecated_update_search_rule", "query,rules", "", "r", "mxbai_omni__api__routes__v1__deprecated_vector_stores__models__SearchRuleResponse"},
|
||||
{"deprecated_delete_search_rule", "main", "DELETE", "/v1/vector_stores/{vector_store_identifier}/rules/{rule_id}", "deprecated_delete_search_rule", "", "", "d", "mxbai_omni__api__routes__v1__deprecated_vector_stores__models__SearchRuleDeleted"},
|
||||
{"deprecated_delete_specific_search_rule", "main", "DELETE", "/v1/vector_stores/{vector_store_identifier}/rules/{rule_id}/specific", "deprecated_delete_specific_search_rule", "rule", "", "d", "mxbai_omni__api__routes__v1__deprecated_vector_stores__models__SearchRuleSpecificDeleted"},
|
||||
{"create_vector_store", "main", "POST", "/v1/vector_stores", "create_vector_store", "description,expires_after,file_ids,is_public,metadata,name", "", "w", "VectorStore"},
|
||||
{"list_vector_stores", "main", "GET", "/v1/vector_stores", "list_vector_stores", "", "after,before,include_total,limit,q", "", "VectorStoreListResponse"},
|
||||
{"retrieve_vector_store", "main", "GET", "/v1/vector_stores/{vector_store_identifier}", "retrieve_vector_store", "", "", "", "VectorStore"},
|
||||
{"update_vector_store", "main", "PUT", "/v1/vector_stores/{vector_store_identifier}", "update_vector_store", "description,expires_after,is_public,metadata,name", "", "w", "VectorStore"},
|
||||
{"delete_vector_store", "main", "DELETE", "/v1/vector_stores/{vector_store_identifier}", "delete_vector_store", "", "", "d", "VectorStoreDeleted"},
|
||||
{"search_vector_store_chunks", "main", "POST", "/v1/vector_stores/search", "search_vector_store_chunks", "file_ids,filters,query,search_options,top_k,vector_store_identifiers", "", "rp", "VectorStoreSearchResponse"},
|
||||
{"deprecated_create_question_answering", "main", "POST", "/v1/vector_stores/question-answering", "deprecated_create_question_answering", "file_ids,filters,qa_options,query,search_options,stream,top_k,vector_store_identifiers", "", "rp", "VectorStoreQAResults"},
|
||||
{"create_chat_completion", "main", "POST", "/v1/chat/completions", "create_chat_completion", "context_management,include,max_completion_tokens,max_tokens,max_tool_calls,messages,metadata,model,parallel_tool_calls,previous_completion_id,previous_messages,response_format,store,stream,temperature,terminal_tool_name,tool_choice,tools,top_p", "", "wp", "ChatCompletion"},
|
||||
{"list_chat_completions", "main", "GET", "/v1/chat/completions", "list_chat_completions", "", "after,include,limit,q", "", "ChatCompletionList"},
|
||||
{"retrieve_chat_completion", "main", "GET", "/v1/chat/completions/{completion_id}", "retrieve_chat_completion", "", "include", "", "ChatCompletion"},
|
||||
{"delete_chat_completion", "main", "DELETE", "/v1/chat/completions/{completion_id}", "delete_chat_completion", "", "", "d", "ChatCompletionDeleted"},
|
||||
{"list_chat_completion_turns", "main", "GET", "/v1/chat/completions/{completion_id}/turns", "list_chat_completion_turns", "", "include", "", "ChatCompletionList"},
|
||||
{"list_chat_completion_messages", "main", "GET", "/v1/chat/completions/{completion_id}/messages", "list_chat_completion_messages", "", "", "", "CompletionMessageList"},
|
||||
{"close_chat_completion_transcript", "main", "POST", "/v1/chat/completions/{completion_id}/transcript/close", "close_chat_completion_transcript", "answer,tool_call_id,tool_content", "", "w", "ChatCompletion"},
|
||||
{"create_response", "main", "POST", "/v1/responses", "create_response", "background,client_metadata,context_management,include,input,instructions,max_output_tokens,max_tool_calls,metadata,model,parallel_tool_calls,previous_response_id,prompt_cache_key,reasoning,safety_identifier,service_tier,store,stream,stream_options,temperature,text,tool_choice,tools,top_p,truncation,user", "", "wp", "Response"},
|
||||
{"retrieve_response", "main", "GET", "/v1/responses/{response_id}", "retrieve_response", "", "include[],starting_after,stream", "", "Response"},
|
||||
{"delete_response", "main", "DELETE", "/v1/responses/{response_id}", "delete_response", "", "", "d", "ResponseDeleted"},
|
||||
{"list_response_input_items", "main", "GET", "/v1/responses/{response_id}/input_items", "list_response_input_items", "", "after,limit,order", "", "ResponseItemList"},
|
||||
{"create_parsing_job", "main", "POST", "/v1/parsing/jobs", "create_parsing_job", "chunking_strategy,element_types,file_id,mode,return_format", "", "wp", "ParsingJobResponse"},
|
||||
{"list_parsing_jobs", "main", "GET", "/v1/parsing/jobs", "list_parsing_jobs", "", "after,before,include_total,limit,q,statuses", "", "ParsingJobListResponse"},
|
||||
{"retrieve_parsing_job", "main", "GET", "/v1/parsing/jobs/{job_id}", "retrieve_parsing_job", "", "", "", "ParsingJobResponse"},
|
||||
{"cancel_parsing_job", "main", "PATCH", "/v1/parsing/jobs/{job_id}", "cancel_parsing_job", "", "", "d", "ParsingJobResponse"},
|
||||
{"delete_parsing_job", "main", "DELETE", "/v1/parsing/jobs/{job_id}", "delete_parsing_job", "", "", "d", "ParsingJobDeleted"},
|
||||
{"create_extraction_job", "main", "POST", "/v1/extractions/jobs", "create_extraction_job", "file_id,json_schema", "", "rp", "ExtractionJob"},
|
||||
{"retrieve_extraction_job", "main", "GET", "/v1/extractions/jobs/{job_id}", "retrieve_extraction_job", "", "", "", "ExtractionJob"},
|
||||
{"create_schema", "main", "POST", "/v1/extractions/schema", "create_schema", "description", "", "w", "CreatedJsonSchema"},
|
||||
{"enhance_schema", "main", "POST", "/v1/extractions/schema/enhance", "enhance_schema", "json_schema", "", "r", "EnhancedJsonSchema"},
|
||||
{"validate_schema", "main", "POST", "/v1/extractions/schema/validate", "validate_schema", "json_schema", "", "r", "ValidatedJsonSchema"},
|
||||
{"extract_content", "main", "POST", "/v1/extractions/content", "extract_content", "content,instructions,json_schema", "", "rp", "ExtractionResult"},
|
||||
{"create_embeddings", "main", "POST", "/v1/embeddings", "create_embeddings", "dimensions,encoding_format,input,model,normalized,prompt", "", "rp", "EmbeddingCreateResponse"},
|
||||
{"rerank", "main", "POST", "/v1/reranking", "rerank", "input,model,query,rank_fields,return_input,rewrite_query,top_k", "", "rp", "RerankResponse"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
const CallTable& mixedbread_calls() {
|
||||
static const CallTable table{kRows, sizeof(kRows) / sizeof(kRows[0])};
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+160
@@ -0,0 +1,160 @@
|
||||
#include "vendors/retrieval/mixedbread/mixedbread.h"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
llm::ApiClientConfig Mixedbread::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "mixedbread";
|
||||
c.base_url = kBaseUrl;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kBearer;
|
||||
return c;
|
||||
}
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("mixedbread", why); }
|
||||
CallArgs with_body(Json body) {
|
||||
CallArgs a;
|
||||
a.body = std::move(body);
|
||||
return a;
|
||||
}
|
||||
CallArgs store_arg(const std::string& store) {
|
||||
CallArgs a;
|
||||
a.path["store_identifier"] = store;
|
||||
return a;
|
||||
}
|
||||
void check_stores(const std::vector<std::string>& stores) {
|
||||
if (stores.empty() || stores.size() > 16) bad("stores must hold 1..16 entries");
|
||||
for (const std::string& s : stores)
|
||||
if (!llm::Service::valid_id(s)) bad("bad store identifier");
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Mixedbread Mixedbread::from_env() {
|
||||
return Mixedbread(make_gateway(mixedbread_calls(), config(), {{"main", kBaseUrl}}, "", nullptr));
|
||||
}
|
||||
Mixedbread Mixedbread::with(const TransportFactory& factory, std::string key) {
|
||||
return Mixedbread(make_gateway(mixedbread_calls(), config(), {{"main", kBaseUrl}}, std::move(key), factory));
|
||||
}
|
||||
|
||||
EmbedResult Mixedbread::parse_embeddings(const Json& doc) {
|
||||
EmbedResult out;
|
||||
out.model = J::str(doc, "model");
|
||||
std::vector<std::pair<int, std::vector<float>>> rows;
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
const Json* e = J::at(d, "embedding");
|
||||
rows.emplace_back(static_cast<int>(J::num(d, "index", static_cast<double>(rows.size()))),
|
||||
e ? J::floats(*e) : std::vector<float>{});
|
||||
}
|
||||
std::stable_sort(rows.begin(), rows.end(), [](const auto& a, const auto& b) { return a.first < b.first; });
|
||||
for (auto& r : rows) out.vectors.push_back(std::move(r.second));
|
||||
if (const Json* u = J::at(doc, "usage")) out.total_tokens = static_cast<long>(J::num(*u, "total_tokens"));
|
||||
return out;
|
||||
}
|
||||
|
||||
RerankResult Mixedbread::parse_rerank(const Json& doc) {
|
||||
RerankResult out;
|
||||
out.model = J::str(doc, "model");
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
RerankHit h;
|
||||
h.index = static_cast<int>(J::num(d, "index"));
|
||||
h.score = J::num(d, "score");
|
||||
out.hits.push_back(h);
|
||||
}
|
||||
std::stable_sort(out.hits.begin(), out.hits.end(), [](const RerankHit& a, const RerankHit& b) { return a.score > b.score; });
|
||||
if (const Json* u = J::at(doc, "usage")) out.total_tokens = static_cast<long>(J::num(*u, "total_tokens"));
|
||||
return out;
|
||||
}
|
||||
|
||||
EmbedResult Mixedbread::embed(const EmbedRequest& request) {
|
||||
Embedder::validate("mixedbread", request);
|
||||
if (request.texts.size() > kMaxEmbedInputs) bad("at most 256 texts per request");
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(request.model.empty() ? std::string(kDefaultEmbedModel) : request.model));
|
||||
body.set("input", J::strings(request.texts));
|
||||
if (request.dimensions > 0) body.set("dimensions", Json(request.dimensions));
|
||||
if (request.input_type == "query") body.set("prompt", Json("Represent this sentence for searching relevant passages:"));
|
||||
EmbedResult r = parse_embeddings(gw_->call("create_embeddings", with_body(body)));
|
||||
if (r.vectors.size() != request.texts.size())
|
||||
throw llm::ApiError("mixedbread", 502, "BAD_RESPONSE", "embedding count differs from input count");
|
||||
return r;
|
||||
}
|
||||
|
||||
RerankResult Mixedbread::rerank(const RerankRequest& request) {
|
||||
Reranker::validate("mixedbread", request);
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(request.model.empty() ? std::string(kDefaultRerankModel) : request.model));
|
||||
body.set("query", Json(request.query));
|
||||
body.set("input", J::strings(request.documents));
|
||||
body.set("return_input", Json(false));
|
||||
if (request.top_n > 0) body.set("top_k", Json(request.top_n));
|
||||
return parse_rerank(gw_->call("rerank", with_body(body)));
|
||||
}
|
||||
|
||||
Json Mixedbread::list_stores() { return gw_->call("list_stores"); }
|
||||
|
||||
Json Mixedbread::create_store(const std::string& name, const std::string& description) {
|
||||
if (name.empty() || name.size() > 128) bad("store name must be 1..128 chars");
|
||||
Json body = Json::object();
|
||||
body.set("name", Json(name));
|
||||
if (!description.empty()) body.set("description", Json(description));
|
||||
return gw_->call("create_store", with_body(body));
|
||||
}
|
||||
|
||||
Json Mixedbread::retrieve_store(const std::string& store) { return gw_->call("retrieve_store", store_arg(store)); }
|
||||
Json Mixedbread::delete_store(const std::string& store) { return gw_->destroy("delete_store", store_arg(store)); }
|
||||
|
||||
Json Mixedbread::search_stores(const std::vector<std::string>& stores, const std::string& query, int top_k) {
|
||||
check_stores(stores);
|
||||
if (query.empty() || query.size() > RetrievalCaps::kMaxQueryChars) bad("query must be 1..8192 chars");
|
||||
if (top_k < 1 || top_k > 100) bad("top_k must be 1..100");
|
||||
Json body = Json::object();
|
||||
body.set("query", Json(query));
|
||||
body.set("store_identifiers", J::strings(stores));
|
||||
body.set("top_k", Json(top_k));
|
||||
return gw_->call("search_store_chunks", with_body(body));
|
||||
}
|
||||
|
||||
Json Mixedbread::answer(const std::vector<std::string>& stores, const std::string& query) {
|
||||
check_stores(stores);
|
||||
if (query.empty() || query.size() > RetrievalCaps::kMaxQueryChars) bad("query must be 1..8192 chars");
|
||||
Json body = Json::object();
|
||||
body.set("query", Json(query));
|
||||
body.set("store_identifiers", J::strings(stores));
|
||||
return gw_->call("create_question_answering", with_body(body));
|
||||
}
|
||||
|
||||
Json Mixedbread::add_file_to_store(const std::string& store, const std::string& file_id) {
|
||||
if (!llm::Service::valid_id(file_id)) bad("bad file id");
|
||||
CallArgs a = store_arg(store);
|
||||
Json body = Json::object();
|
||||
body.set("file_id", Json(file_id));
|
||||
a.body = body;
|
||||
return gw_->call("create_store_file", a);
|
||||
}
|
||||
|
||||
Json Mixedbread::delete_store_file(const std::string& store, const std::string& file) {
|
||||
CallArgs a = store_arg(store);
|
||||
a.path["file_identifier"] = file;
|
||||
return gw_->destroy("delete_store_file", a);
|
||||
}
|
||||
|
||||
Json Mixedbread::list_files() { return gw_->call("list_files"); }
|
||||
|
||||
Json Mixedbread::retrieve_file(const std::string& file_id) {
|
||||
CallArgs a;
|
||||
a.path["file_id"] = file_id;
|
||||
return gw_->call("retrieve_file", a);
|
||||
}
|
||||
|
||||
Json Mixedbread::delete_file(const std::string& file_id) {
|
||||
CallArgs a;
|
||||
a.path["file_id"] = file_id;
|
||||
return gw_->destroy("delete_file", a);
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
#pragma once
|
||||
|
||||
// Mixedbread: embeddings, reranking, stores (hosted vector stores with
|
||||
// files, chunk search and question answering), files, parsing and extraction.
|
||||
// Implements Embedder and Reranker. Organisation administration, integrations
|
||||
// and OAuth callbacks are not wrapped (see EXCLUDED in tools/check_spec.py).
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/retrieval/provider.h"
|
||||
#include "vendors/retrieval/seams.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallTable& mixedbread_calls();
|
||||
|
||||
class Mixedbread : public Provider, public Embedder, public Reranker {
|
||||
public:
|
||||
static constexpr const char* kKeyEnv = "MXBAI_API_KEY";
|
||||
static constexpr const char* kBaseUrl = "https://api.mixedbread.com";
|
||||
static constexpr const char* kDefaultEmbedModel = "mixedbread-ai/mxbai-embed-large-v1";
|
||||
static constexpr const char* kDefaultRerankModel = "mixedbread-ai/mxbai-rerank-v3.1-listwise";
|
||||
static constexpr std::size_t kMaxEmbedInputs = 256; // spec: input maxItems 256
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
static Mixedbread from_env();
|
||||
static Mixedbread with(const TransportFactory& factory, std::string key);
|
||||
explicit Mixedbread(std::shared_ptr<Gateway> gateway) : Provider(std::move(gateway)) {}
|
||||
|
||||
std::string provider() const override { return "mixedbread"; }
|
||||
EmbedResult embed(const EmbedRequest& request) override;
|
||||
RerankResult rerank(const RerankRequest& request) override;
|
||||
|
||||
// Stores
|
||||
Json list_stores();
|
||||
Json create_store(const std::string& name, const std::string& description);
|
||||
Json retrieve_store(const std::string& store);
|
||||
Json delete_store(const std::string& store); // destructive
|
||||
Json search_stores(const std::vector<std::string>& stores, const std::string& query, int top_k);
|
||||
Json answer(const std::vector<std::string>& stores, const std::string& query);
|
||||
Json add_file_to_store(const std::string& store, const std::string& file_id);
|
||||
Json delete_store_file(const std::string& store, const std::string& file); // destructive
|
||||
// Files
|
||||
Json list_files();
|
||||
Json retrieve_file(const std::string& file_id);
|
||||
Json delete_file(const std::string& file_id); // destructive
|
||||
|
||||
static EmbedResult parse_embeddings(const Json& doc);
|
||||
static RerankResult parse_rerank(const Json& doc);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
#include "vendors/retrieval/pinecone/pinecone.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// name, host, http method, path, spec operation id (or reference page / "doc:"),
|
||||
// body fields, query parameters, flags (d destructive, w writes, r read-only POST,
|
||||
// p paid), response schema)
|
||||
static const CallRow kRows[] = {
|
||||
{"list_indexes", "control", "GET", "/indexes", "list_indexes", "", "", "", "IndexList"},
|
||||
{"create_index", "control", "POST", "/indexes", "create_index", "cmek_id,deletion_protection,deployment,name,read_capacity,schema,tags", "", "w", "IndexModel"},
|
||||
{"describe_index", "control", "GET", "/indexes/{index_name}", "describe_index", "", "", "", "IndexModel"},
|
||||
{"delete_index", "control", "DELETE", "/indexes/{index_name}", "delete_index", "", "", "d", "-"},
|
||||
{"configure_index", "control", "PATCH", "/indexes/{index_name}", "configure_index", "deletion_protection,deployment,read_capacity,schema,tags", "", "w", "IndexModel"},
|
||||
{"list_index_backups", "control", "GET", "/indexes/{index_name}/backups", "list_index_backups", "", "include_deleted,limit,paginationToken", "", "BackupList"},
|
||||
{"create_backup", "control", "POST", "/indexes/{index_name}/backups", "create_backup", "description,name", "", "w", "BackupModel"},
|
||||
{"list_backup_schedules", "control", "GET", "/indexes/{index_name}/backup-schedules", "list_backup_schedules", "", "limit,paginationToken", "", "BackupScheduleList"},
|
||||
{"create_backup_schedule", "control", "POST", "/indexes/{index_name}/backup-schedules", "create_backup_schedule", "name,retention,schedule", "", "w", "BackupScheduleResponse"},
|
||||
{"create_index_for_model", "control", "POST", "/indexes/create-for-model", "create_index_for_model", "cloud,deletion_protection,embed,name,read_capacity,region,schema,tags", "", "w", "IndexModel"},
|
||||
{"list_project_backups", "control", "GET", "/backups", "list_project_backups", "", "limit,paginationToken", "", "BackupList"},
|
||||
{"describe_backup", "control", "GET", "/backups/{backup_id}", "describe_backup", "", "", "", "BackupModel"},
|
||||
{"delete_backup", "control", "DELETE", "/backups/{backup_id}", "delete_backup", "", "", "d", "-"},
|
||||
{"create_index_from_backup", "control", "POST", "/backups/{backup_id}/create-index", "create_index_from_backup_operation", "deletion_protection,name,read_capacity,tags", "", "w", "CreateIndexFromBackupResponse"},
|
||||
{"describe_backup_schedule", "control", "GET", "/backup-schedules/{schedule_id}", "describe_backup_schedule", "", "", "", "BackupScheduleResponse"},
|
||||
{"delete_backup_schedule", "control", "DELETE", "/backup-schedules/{schedule_id}", "delete_backup_schedule", "", "", "d", "-"},
|
||||
{"update_backup_schedule", "control", "PATCH", "/backup-schedules/{schedule_id}", "update_backup_schedule", "enabled,frequency,retention", "", "w", "BackupScheduleResponse"},
|
||||
{"list_backup_schedule_history", "control", "GET", "/backup-schedules/{schedule_id}/history", "list_backup_schedule_history", "", "limit,paginationToken", "", "BackupScheduleHistoryList"},
|
||||
{"list_restore_jobs", "control", "GET", "/restore-jobs", "list_restore_jobs", "", "limit,paginationToken", "", "RestoreJobList"},
|
||||
{"describe_restore_job", "control", "GET", "/restore-jobs/{job_id}", "describe_restore_job", "", "", "", "RestoreJobModel"},
|
||||
{"list_collections", "control", "GET", "/collections", "list_collections", "", "", "", "CollectionList"},
|
||||
{"create_collection", "control", "POST", "/collections", "create_collection", "name,source", "", "w", "CollectionModel"},
|
||||
{"describe_collection", "control", "GET", "/collections/{collection_name}", "describe_collection", "", "", "", "CollectionModel"},
|
||||
{"delete_collection", "control", "DELETE", "/collections/{collection_name}", "delete_collection", "", "", "d", "-"},
|
||||
{"list_bulk_imports", "data", "GET", "/bulk/imports", "listBulkImports", "", "limit,paginationToken", "", "ListImportsResponse"},
|
||||
{"start_bulk_import", "data", "POST", "/bulk/imports", "startBulkImport", "errorMode,integrationId,uri", "", "w", "StartImportResponse"},
|
||||
{"describe_bulk_import", "data", "GET", "/bulk/imports/{id}", "describeBulkImport", "", "", "", "ImportModel"},
|
||||
{"cancel_bulk_import", "data", "DELETE", "/bulk/imports/{id}", "cancelBulkImport", "", "", "d", "CancelImportResponse"},
|
||||
{"describe_index_stats", "data", "POST", "/describe_index_stats", "describeIndexStats", "filter", "", "r", "IndexDescription"},
|
||||
{"query_vectors", "data", "POST", "/query", "queryVectors", "filter,id,includeMetadata,includeValues,maxCandidates,namespace,scanFactor,sparseVector,topK,vector", "", "r", "QueryResponse"},
|
||||
{"delete_vectors", "data", "POST", "/vectors/delete", "deleteVectors", "deleteAll,filter,ids,namespace", "", "d", "DeleteResponse"},
|
||||
{"fetch_vectors", "data", "GET", "/vectors/fetch", "fetchVectors", "", "ids,namespace", "", "FetchResponse"},
|
||||
{"fetch_vectors_by_metadata", "data", "POST", "/vectors/fetch_by_metadata", "fetch_vectors_by_metadata", "filter,limit,namespace,paginationToken", "", "r", "FetchByMetadataResponse"},
|
||||
{"list_vectors", "data", "GET", "/vectors/list", "listVectors", "", "limit,namespace,paginationToken,prefix", "", "ListResponse"},
|
||||
{"update_vector", "data", "POST", "/vectors/update", "updateVector", "dryRun,filter,id,namespace,setMetadata,sparseValues,values", "", "w", "UpdateResponse"},
|
||||
{"upsert_vectors", "data", "POST", "/vectors/upsert", "upsertVectors", "namespace,vectors", "", "w", "UpsertResponse"},
|
||||
{"list_namespaces", "data", "GET", "/namespaces", "listNamespacesOperation", "", "limit,paginationToken,prefix", "", "ListNamespacesResponse"},
|
||||
{"create_namespace", "data", "POST", "/namespaces", "createNamespace", "name,schema", "", "w", "NamespaceDescription"},
|
||||
{"describe_namespace", "data", "GET", "/namespaces/{namespace}", "describeNamespace", "", "", "", "NamespaceDescription"},
|
||||
{"delete_namespace", "data", "DELETE", "/namespaces/{namespace}", "deleteNamespace", "", "", "d", "DeleteResponse"},
|
||||
{"list_namespace_aliases", "data", "GET", "/namespace-aliases", "listNamespaceAliases", "", "limit,paginationToken,prefix,target_namespace", "", "ListNamespaceAliasesResponse"},
|
||||
{"create_namespace_alias", "data", "POST", "/namespace-aliases", "createNamespaceAlias", "name,target_namespace", "", "w", "NamespaceAliasDescription"},
|
||||
{"describe_namespace_alias", "data", "GET", "/namespace-aliases/{alias_name}", "describeNamespaceAlias", "", "", "", "NamespaceAliasDescription"},
|
||||
{"delete_namespace_alias", "data", "DELETE", "/namespace-aliases/{alias_name}", "deleteNamespaceAlias", "", "", "d", "DeleteResponse"},
|
||||
{"repoint_namespace_alias", "data", "PATCH", "/namespace-aliases/{alias_name}", "repointNamespaceAlias", "target_namespace", "", "w", "NamespaceAliasDescription"},
|
||||
{"upsert_records_namespace", "data", "POST", "/records/namespaces/{namespace}/upsert", "upsertRecordsNamespace", "*", "", "wp", "-"},
|
||||
{"search_records_namespace", "data", "POST", "/records/namespaces/{namespace}/search", "searchRecordsNamespace", "fields,query,rerank", "", "rp", "SearchRecordsResponse"},
|
||||
{"upsert_documents", "data", "POST", "/namespaces/{namespace}/documents/upsert", "upsertDocuments", "documents", "", "w", "UpsertDocumentsResponse"},
|
||||
{"search_documents", "data", "POST", "/namespaces/{namespace}/documents/search", "searchDocuments", "filter,include_fields,score_by,top_k", "", "r", "SearchDocumentsResponse"},
|
||||
{"fetch_documents", "data", "POST", "/namespaces/{namespace}/documents/fetch", "fetchDocuments", "filter,ids,include_fields,limit,pagination_token", "", "r", "FetchDocumentsResponse"},
|
||||
{"delete_documents", "data", "POST", "/namespaces/{namespace}/documents/delete", "deleteDocuments", "delete_all,filter,ids", "", "d", "DeleteDocumentsResponse"},
|
||||
{"update_documents", "data", "POST", "/namespaces/{namespace}/documents/update", "updateDocuments", "documents,filter,remove_fields,set_fields", "", "w", "UpdateDocumentsResponse"},
|
||||
{"list_documents", "data", "POST", "/namespaces/{namespace}/documents/list", "listDocuments", "limit,pagination_token,prefix", "", "r", "ListDocumentsResponse"},
|
||||
{"embed", "inference", "POST", "/embed", "embed", "inputs,model,parameters", "", "rp", "EmbeddingsList"},
|
||||
{"rerank", "inference", "POST", "/rerank", "rerank", "documents,model,parameters,query,rank_fields,return_documents,top_n", "", "rp", "RerankResult"},
|
||||
{"list_models", "inference", "GET", "/models", "list_models", "", "type,vector_type", "", "ModelInfoList"},
|
||||
{"get_model", "inference", "GET", "/models/{model_name}", "get_model", "", "", "", "ModelInfo"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
const CallTable& pinecone_calls() {
|
||||
static const CallTable table{kRows, sizeof(kRows) / sizeof(kRows[0])};
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+312
@@ -0,0 +1,312 @@
|
||||
#include "vendors/retrieval/pinecone/pinecone.h"
|
||||
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
llm::ApiClientConfig Pinecone::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "pinecone";
|
||||
c.base_url = kControlUrl;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = "Api-Key";
|
||||
c.default_headers["X-Pinecone-Api-Version"] = kApiVersion;
|
||||
return c;
|
||||
}
|
||||
|
||||
namespace {
|
||||
std::vector<HostSpec> hosts_for(const std::string& index_host) {
|
||||
std::vector<HostSpec> h = {{"control", Pinecone::kControlUrl}, {"inference", Pinecone::kControlUrl}};
|
||||
if (!index_host.empty()) {
|
||||
check_host("pinecone", index_host);
|
||||
h.push_back({"data", "https://" + index_host});
|
||||
}
|
||||
return h;
|
||||
}
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("pinecone", why); }
|
||||
CallArgs with_body(Json body) {
|
||||
CallArgs a;
|
||||
a.body = std::move(body);
|
||||
return a;
|
||||
}
|
||||
CallArgs with_path(const char* key, const std::string& value) {
|
||||
CallArgs a;
|
||||
a.path[key] = value;
|
||||
return a;
|
||||
}
|
||||
void put_ns(Json& body, const std::string& ns) {
|
||||
if (!ns.empty()) body.set("namespace", Json(ns));
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Pinecone Pinecone::from_env(const std::string& index_host) {
|
||||
return Pinecone(make_gateway(pinecone_calls(), config(), hosts_for(index_host), "", nullptr));
|
||||
}
|
||||
|
||||
Pinecone Pinecone::with(const TransportFactory& factory, std::string key, const std::string& index_host) {
|
||||
return Pinecone(make_gateway(pinecone_calls(), config(), hosts_for(index_host), std::move(key), factory));
|
||||
}
|
||||
|
||||
// ---- parsers ---------------------------------------------------------------
|
||||
|
||||
EmbedResult Pinecone::parse_embeddings(const Json& doc) {
|
||||
EmbedResult out;
|
||||
out.model = J::str(doc, "model");
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
const Json* v = J::at(d, "values");
|
||||
out.vectors.push_back(v ? J::floats(*v) : std::vector<float>{});
|
||||
}
|
||||
if (const Json* u = J::at(doc, "usage")) out.total_tokens = static_cast<long>(J::num(*u, "total_tokens"));
|
||||
return out;
|
||||
}
|
||||
|
||||
RerankResult Pinecone::parse_rerank(const Json& doc) {
|
||||
RerankResult out;
|
||||
out.model = J::str(doc, "model");
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
RerankHit h;
|
||||
h.index = static_cast<int>(J::num(d, "index"));
|
||||
h.score = J::num(d, "score");
|
||||
out.hits.push_back(h);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<PineconeMatch> Pinecone::parse_matches(const Json& doc) {
|
||||
std::vector<PineconeMatch> out;
|
||||
for (const Json& m : J::array(doc, "matches")) {
|
||||
PineconeMatch x;
|
||||
x.id = J::str(m, "id");
|
||||
x.score = J::num(m, "score");
|
||||
if (const Json* v = J::at(m, "values")) x.values = J::floats(*v);
|
||||
if (const Json* md = J::at(m, "metadata")) x.metadata = *md;
|
||||
out.push_back(std::move(x));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
PineconePage Pinecone::parse_list(const Json& doc) {
|
||||
PineconePage p;
|
||||
for (const Json& v : J::array(doc, "vectors")) p.ids.push_back(J::str(v, "id"));
|
||||
if (const Json* pg = J::at(doc, "pagination")) p.next_token = J::str(*pg, "next");
|
||||
return p;
|
||||
}
|
||||
|
||||
Json Pinecone::vector_json(const PineconeVector& v) {
|
||||
if (v.id.empty() || v.id.size() > 512) bad("vector id must be 1..512 chars");
|
||||
if (v.values.empty() || v.values.size() > static_cast<std::size_t>(RetrievalCaps::kMaxDimensions))
|
||||
bad("vector values must hold 1..20000 numbers");
|
||||
Json o = Json::object();
|
||||
o.set("id", Json(v.id));
|
||||
Json vals = Json::array();
|
||||
for (float f : v.values) vals.push(Json(static_cast<double>(f)));
|
||||
o.set("values", vals);
|
||||
if (!v.metadata.is_null()) {
|
||||
if (!v.metadata.is_object()) bad("metadata must be an object");
|
||||
o.set("metadata", v.metadata);
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
// ---- inference -------------------------------------------------------------
|
||||
|
||||
EmbedResult Pinecone::embed(const EmbedRequest& request) {
|
||||
Embedder::validate("pinecone", request);
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(request.model.empty() ? std::string(kDefaultEmbedModel) : request.model));
|
||||
Json inputs = Json::array();
|
||||
for (const std::string& t : request.texts) {
|
||||
Json i = Json::object();
|
||||
i.set("text", Json(t));
|
||||
inputs.push(i);
|
||||
}
|
||||
body.set("inputs", inputs);
|
||||
Json params = Json::object();
|
||||
if (!request.input_type.empty()) params.set("input_type", Json(request.input_type == "query" ? "query" : "passage"));
|
||||
params.set("truncate", Json("END"));
|
||||
body.set("parameters", params);
|
||||
EmbedResult r = parse_embeddings(gw_->call("embed", with_body(body)));
|
||||
if (r.vectors.size() != request.texts.size())
|
||||
throw llm::ApiError("pinecone", 502, "BAD_RESPONSE", "embedding count differs from input count");
|
||||
return r;
|
||||
}
|
||||
|
||||
RerankResult Pinecone::rerank(const RerankRequest& request) {
|
||||
Reranker::validate("pinecone", request);
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(request.model.empty() ? std::string(kDefaultRerankModel) : request.model));
|
||||
body.set("query", Json(request.query));
|
||||
Json docs = Json::array();
|
||||
for (const std::string& d : request.documents) {
|
||||
Json i = Json::object();
|
||||
i.set("text", Json(d));
|
||||
docs.push(i);
|
||||
}
|
||||
body.set("documents", docs);
|
||||
body.set("return_documents", Json(false));
|
||||
if (request.top_n > 0) body.set("top_n", Json(request.top_n));
|
||||
return parse_rerank(gw_->call("rerank", with_body(body)));
|
||||
}
|
||||
|
||||
// ---- control plane ---------------------------------------------------------
|
||||
|
||||
Json Pinecone::list_indexes() { return gw_->call("list_indexes"); }
|
||||
Json Pinecone::describe_index(const std::string& name) { return gw_->call("describe_index", with_path("index_name", name)); }
|
||||
|
||||
Json Pinecone::create_index(const std::string& name, int dimension, const std::string& metric,
|
||||
const std::string& cloud, const std::string& region) {
|
||||
if (name.empty() || name.size() > 45) bad("index name must be 1..45 chars");
|
||||
if (dimension < 1 || dimension > RetrievalCaps::kMaxDimensions) bad("dimension must be 1..20000");
|
||||
if (metric != "cosine" && metric != "dotproduct" && metric != "euclidean") bad("metric must be cosine, dotproduct or euclidean");
|
||||
if (cloud.empty() || region.empty()) bad("cloud and region are required");
|
||||
Json field = Json::object();
|
||||
field.set("type", Json("dense_vector"));
|
||||
field.set("dimension", Json(dimension));
|
||||
field.set("metric", Json(metric));
|
||||
Json fields = Json::object();
|
||||
fields.set("vector", field);
|
||||
Json schema = Json::object();
|
||||
schema.set("fields", fields);
|
||||
Json deployment = Json::object();
|
||||
deployment.set("deployment_type", Json("managed"));
|
||||
deployment.set("cloud", Json(cloud));
|
||||
deployment.set("region", Json(region));
|
||||
Json body = Json::object();
|
||||
body.set("name", Json(name));
|
||||
body.set("schema", schema);
|
||||
body.set("deployment", deployment);
|
||||
return gw_->call("create_index", with_body(body));
|
||||
}
|
||||
|
||||
Json Pinecone::configure_index(const std::string& name, const Json& body) {
|
||||
if (!body.is_object()) bad("body must be an object");
|
||||
CallArgs a = with_path("index_name", name);
|
||||
a.body = body;
|
||||
return gw_->call("configure_index", a);
|
||||
}
|
||||
|
||||
Json Pinecone::delete_index(const std::string& name) { return gw_->destroy("delete_index", with_path("index_name", name)); }
|
||||
Json Pinecone::list_collections() { return gw_->call("list_collections"); }
|
||||
Json Pinecone::describe_collection(const std::string& name) {
|
||||
return gw_->call("describe_collection", with_path("collection_name", name));
|
||||
}
|
||||
Json Pinecone::create_collection(const std::string& name, const std::string& source_index) {
|
||||
if (name.empty() || source_index.empty()) bad("name and source are required");
|
||||
Json body = Json::object();
|
||||
body.set("name", Json(name));
|
||||
body.set("source", Json(source_index));
|
||||
return gw_->call("create_collection", with_body(body));
|
||||
}
|
||||
Json Pinecone::delete_collection(const std::string& name) {
|
||||
return gw_->destroy("delete_collection", with_path("collection_name", name));
|
||||
}
|
||||
Json Pinecone::list_backups(const std::string& index_name) {
|
||||
return gw_->call("list_index_backups", with_path("index_name", index_name));
|
||||
}
|
||||
Json Pinecone::create_backup(const std::string& index_name, const std::string& name, const std::string& description) {
|
||||
Json body = Json::object();
|
||||
if (!name.empty()) body.set("name", Json(name));
|
||||
if (!description.empty()) body.set("description", Json(description));
|
||||
CallArgs a = with_path("index_name", index_name);
|
||||
a.body = body;
|
||||
return gw_->call("create_backup", a);
|
||||
}
|
||||
Json Pinecone::describe_backup(const std::string& backup_id) { return gw_->call("describe_backup", with_path("backup_id", backup_id)); }
|
||||
Json Pinecone::delete_backup(const std::string& backup_id) { return gw_->destroy("delete_backup", with_path("backup_id", backup_id)); }
|
||||
|
||||
// ---- data plane ------------------------------------------------------------
|
||||
|
||||
Json Pinecone::upsert(const std::vector<PineconeVector>& vectors, const std::string& ns) {
|
||||
if (vectors.empty() || vectors.size() > RetrievalCaps::kMaxRecordsPerUpsert) bad("vectors must hold 1..1000 entries");
|
||||
Json arr = Json::array();
|
||||
for (const PineconeVector& v : vectors) arr.push(vector_json(v));
|
||||
Json body = Json::object();
|
||||
body.set("vectors", arr);
|
||||
put_ns(body, ns);
|
||||
return gw_->call("upsert_vectors", with_body(body));
|
||||
}
|
||||
|
||||
std::vector<PineconeMatch> Pinecone::query(const PineconeQuery& q) {
|
||||
if (q.top_k < 1 || q.top_k > RetrievalCaps::kMaxTopK) bad("top_k must be 1..10000");
|
||||
if (q.vector.empty() == q.id.empty()) bad("give exactly one of vector and id");
|
||||
Json body = Json::object();
|
||||
body.set("topK", Json(q.top_k));
|
||||
if (!q.vector.empty()) {
|
||||
if (q.vector.size() > static_cast<std::size_t>(RetrievalCaps::kMaxDimensions)) bad("vector too long");
|
||||
Json v = Json::array();
|
||||
for (float f : q.vector) v.push(Json(static_cast<double>(f)));
|
||||
body.set("vector", v);
|
||||
} else {
|
||||
body.set("id", Json(q.id));
|
||||
}
|
||||
put_ns(body, q.ns);
|
||||
if (!q.filter.is_null()) {
|
||||
if (!q.filter.is_object()) bad("filter must be an object");
|
||||
body.set("filter", q.filter);
|
||||
}
|
||||
body.set("includeValues", Json(q.include_values));
|
||||
body.set("includeMetadata", Json(q.include_metadata));
|
||||
return parse_matches(gw_->call("query_vectors", with_body(body)));
|
||||
}
|
||||
|
||||
Json Pinecone::fetch(const std::vector<std::string>& ids, const std::string& ns) {
|
||||
if (ids.empty() || ids.size() > kFetchIdsPerCall) bad("ids must hold 1..100 entries per call");
|
||||
CallArgs a;
|
||||
for (const std::string& id : ids) a.query_multi.emplace_back("ids", id);
|
||||
if (!ns.empty()) a.query["namespace"] = ns;
|
||||
return gw_->call("fetch_vectors", a);
|
||||
}
|
||||
|
||||
PineconePage Pinecone::list(const std::string& prefix, const std::string& ns, int limit, const std::string& token) {
|
||||
if (limit < 1 || limit > RetrievalCaps::kMaxListLimit) bad("limit must be 1..100");
|
||||
CallArgs a;
|
||||
a.query["limit"] = Query::number(limit);
|
||||
if (!prefix.empty()) a.query["prefix"] = prefix;
|
||||
if (!ns.empty()) a.query["namespace"] = ns;
|
||||
if (!token.empty()) a.query["paginationToken"] = token;
|
||||
return parse_list(gw_->call("list_vectors", a));
|
||||
}
|
||||
|
||||
Json Pinecone::stats(const Json& filter) {
|
||||
Json body = Json::object();
|
||||
if (!filter.is_null()) body.set("filter", filter);
|
||||
return gw_->call("describe_index_stats", with_body(body));
|
||||
}
|
||||
|
||||
Json Pinecone::list_namespaces(int limit, const std::string& token) {
|
||||
if (limit < 1 || limit > RetrievalCaps::kMaxListLimit) bad("limit must be 1..100");
|
||||
CallArgs a;
|
||||
a.query["limit"] = Query::number(limit);
|
||||
if (!token.empty()) a.query["paginationToken"] = token;
|
||||
return gw_->call("list_namespaces", a);
|
||||
}
|
||||
|
||||
Json Pinecone::describe_namespace(const std::string& ns) { return gw_->call("describe_namespace", with_path("namespace", ns)); }
|
||||
|
||||
Json Pinecone::delete_ids(const std::vector<std::string>& ids, const std::string& ns) {
|
||||
if (ids.empty() || ids.size() > RetrievalCaps::kMaxIdsPerFetch) bad("ids must hold 1..1000 entries");
|
||||
Json body = Json::object();
|
||||
body.set("ids", J::strings(ids));
|
||||
put_ns(body, ns);
|
||||
return gw_->destroy("delete_vectors", with_body(body));
|
||||
}
|
||||
|
||||
Json Pinecone::delete_all(const std::string& ns) {
|
||||
Json body = Json::object();
|
||||
body.set("deleteAll", Json(true));
|
||||
put_ns(body, ns);
|
||||
return gw_->destroy("delete_vectors", with_body(body));
|
||||
}
|
||||
|
||||
Json Pinecone::delete_by_filter(const Json& filter, const std::string& ns) {
|
||||
if (!filter.is_object() || filter.as_object().empty()) bad("filter must be a non-empty object");
|
||||
Json body = Json::object();
|
||||
body.set("filter", filter);
|
||||
put_ns(body, ns);
|
||||
return gw_->destroy("delete_vectors", with_body(body));
|
||||
}
|
||||
|
||||
Json Pinecone::delete_namespace(const std::string& ns) { return gw_->destroy("delete_namespace", with_path("namespace", ns)); }
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+107
@@ -0,0 +1,107 @@
|
||||
#pragma once
|
||||
|
||||
// Pinecone: control plane (indexes, collections, backups), data plane
|
||||
// (upsert, query, fetch, list, delete, namespaces, stats) and inference
|
||||
// (embed, rerank). API version 2026-07. The data plane lives on the host of an
|
||||
// index, so a Pinecone object is built with that host (empty = control plane
|
||||
// and inference only). Implements Embedder and Reranker through the hosted
|
||||
// inference models.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/retrieval/provider.h"
|
||||
#include "vendors/retrieval/seams.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallTable& pinecone_calls();
|
||||
|
||||
struct PineconeVector {
|
||||
std::string id;
|
||||
std::vector<float> values;
|
||||
Json metadata; // null = none
|
||||
};
|
||||
|
||||
struct PineconeMatch {
|
||||
std::string id;
|
||||
double score = 0;
|
||||
std::vector<float> values;
|
||||
Json metadata;
|
||||
};
|
||||
|
||||
struct PineconeQuery {
|
||||
std::vector<float> vector; // or `id`
|
||||
std::string id;
|
||||
int top_k = 10;
|
||||
std::string ns; // namespace; empty = the default namespace
|
||||
Json filter; // null = none
|
||||
bool include_values = false;
|
||||
bool include_metadata = true;
|
||||
};
|
||||
|
||||
struct PineconePage {
|
||||
std::vector<std::string> ids;
|
||||
std::string next_token; // empty = last page
|
||||
bool truncated = false; // the page walk hit the cap
|
||||
};
|
||||
|
||||
class Pinecone : public Provider, public Embedder, public Reranker {
|
||||
public:
|
||||
static constexpr const char* kKeyEnv = "PINECONE_API_KEY";
|
||||
static constexpr const char* kControlUrl = "https://api.pinecone.io";
|
||||
static constexpr const char* kApiVersion = "2026-07";
|
||||
static constexpr const char* kDefaultEmbedModel = "multilingual-e5-large";
|
||||
static constexpr const char* kDefaultRerankModel = "bge-reranker-v2-m3";
|
||||
static constexpr std::size_t kFetchIdsPerCall = 100; // ids travel in the URL
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
// index_host: the `host` field of describe_index, e.g. "my-index-abc.svc.pinecone.io".
|
||||
static Pinecone from_env(const std::string& index_host = "");
|
||||
static Pinecone with(const TransportFactory& factory, std::string key, const std::string& index_host = "");
|
||||
explicit Pinecone(std::shared_ptr<Gateway> gateway) : Provider(std::move(gateway)) {}
|
||||
|
||||
std::string provider() const override { return "pinecone"; }
|
||||
EmbedResult embed(const EmbedRequest& request) override;
|
||||
RerankResult rerank(const RerankRequest& request) override;
|
||||
|
||||
// Control plane
|
||||
Json list_indexes();
|
||||
Json describe_index(const std::string& name);
|
||||
// A dense-vector index on Pinecone managed infrastructure.
|
||||
Json create_index(const std::string& name, int dimension, const std::string& metric,
|
||||
const std::string& cloud, const std::string& region);
|
||||
Json configure_index(const std::string& name, const Json& body);
|
||||
// Destructive.
|
||||
Json delete_index(const std::string& name);
|
||||
Json list_collections();
|
||||
Json describe_collection(const std::string& name);
|
||||
Json create_collection(const std::string& name, const std::string& source_index);
|
||||
Json delete_collection(const std::string& name); // destructive
|
||||
Json list_backups(const std::string& index_name);
|
||||
Json create_backup(const std::string& index_name, const std::string& name, const std::string& description);
|
||||
Json describe_backup(const std::string& backup_id);
|
||||
Json delete_backup(const std::string& backup_id); // destructive
|
||||
|
||||
// Data plane (needs the index host)
|
||||
Json upsert(const std::vector<PineconeVector>& vectors, const std::string& ns);
|
||||
std::vector<PineconeMatch> query(const PineconeQuery& query);
|
||||
Json fetch(const std::vector<std::string>& ids, const std::string& ns);
|
||||
PineconePage list(const std::string& prefix, const std::string& ns, int limit, const std::string& token);
|
||||
Json stats(const Json& filter);
|
||||
Json list_namespaces(int limit, const std::string& token);
|
||||
Json describe_namespace(const std::string& ns);
|
||||
// Destructive.
|
||||
Json delete_ids(const std::vector<std::string>& ids, const std::string& ns);
|
||||
Json delete_all(const std::string& ns);
|
||||
Json delete_by_filter(const Json& filter, const std::string& ns);
|
||||
Json delete_namespace(const std::string& ns);
|
||||
|
||||
static EmbedResult parse_embeddings(const Json& doc);
|
||||
static RerankResult parse_rerank(const Json& doc);
|
||||
static std::vector<PineconeMatch> parse_matches(const Json& doc);
|
||||
static PineconePage parse_list(const Json& doc);
|
||||
static Json vector_json(const PineconeVector& v);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+75
@@ -0,0 +1,75 @@
|
||||
#include "vendors/retrieval/provider.h"
|
||||
|
||||
#include "clients/transports/http_transport.h"
|
||||
#include "llm/config.h"
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
std::shared_ptr<Gateway> make_gateway(const CallTable& table, const llm::ApiClientConfig& base,
|
||||
const std::vector<HostSpec>& hosts, std::string key,
|
||||
const TransportFactory& factory) {
|
||||
if (key.empty()) key = llm::Config::get(base.key_env.c_str());
|
||||
Gateway::Hosts clients;
|
||||
for (const HostSpec& h : hosts) {
|
||||
llm::ApiClientConfig cfg = base;
|
||||
cfg.base_url = h.base_url;
|
||||
std::unique_ptr<clients::Transport> transport =
|
||||
factory ? factory() : std::make_unique<clients::transports::HttpTransport>();
|
||||
clients[h.host] = std::make_shared<llm::ApiClient>(std::move(transport), cfg, key);
|
||||
}
|
||||
return std::make_shared<Gateway>(base.vendor, table, std::move(clients));
|
||||
}
|
||||
|
||||
void check_host(const std::string& vendor, const std::string& host) {
|
||||
if (host.empty() || host.size() > RetrievalCaps::kMaxHostChars)
|
||||
throw llm::ApiError::invalid(vendor, "host is empty or too long");
|
||||
for (unsigned char c : host) {
|
||||
bool ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' || c == '.';
|
||||
if (!ok) throw llm::ApiError::invalid(vendor, "host must be a bare DNS name (no scheme, port or path)");
|
||||
}
|
||||
if (host.front() == '.' || host.front() == '-' || host.find("..") != std::string::npos)
|
||||
throw llm::ApiError::invalid(vendor, "malformed host");
|
||||
}
|
||||
|
||||
Json J::strings(const std::vector<std::string>& v) {
|
||||
Json a = Json::array();
|
||||
for (const std::string& s : v) a.push(Json(s));
|
||||
return a;
|
||||
}
|
||||
|
||||
std::string J::str(const Json& doc, const std::string& key, const std::string& fallback) {
|
||||
const Json* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
if (!v) return fallback;
|
||||
if (v->is_string()) return v->as_string();
|
||||
return fallback;
|
||||
}
|
||||
|
||||
double J::num(const Json& doc, const std::string& key, double fallback) {
|
||||
const Json* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
return v && v->is_number() ? v->as_number() : fallback;
|
||||
}
|
||||
|
||||
bool J::has_array(const Json& doc, const std::string& key) {
|
||||
const Json* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
return v && v->is_array();
|
||||
}
|
||||
|
||||
const Json* J::at(const Json& doc, const std::string& key) { return doc.is_object() ? doc.find(key) : nullptr; }
|
||||
|
||||
std::vector<float> J::floats(const Json& arr) {
|
||||
std::vector<float> out;
|
||||
if (!arr.is_array()) return out;
|
||||
out.reserve(arr.as_array().size());
|
||||
for (const Json& x : arr.as_array())
|
||||
if (x.is_number()) out.push_back(static_cast<float>(x.as_number()));
|
||||
return out;
|
||||
}
|
||||
|
||||
const Json::Array& J::array(const Json& doc, const char* key) {
|
||||
static const Json::Array empty;
|
||||
const Json* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
return v && v->is_array() ? v->as_array() : empty;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+60
@@ -0,0 +1,60 @@
|
||||
#pragma once
|
||||
|
||||
// Shared plumbing of the provider classes: building a Gateway over one
|
||||
// ApiClient per host, and small Json helpers so each provider file only holds
|
||||
// what is specific to the provider.
|
||||
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/transport.h"
|
||||
#include "llm/api_client.h"
|
||||
#include "vendors/retrieval/gateway.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
struct HostSpec {
|
||||
std::string host; // name used in CallRow::host
|
||||
std::string base_url; // https://..., no trailing slash
|
||||
};
|
||||
|
||||
using TransportFactory = std::function<std::unique_ptr<clients::Transport>()>;
|
||||
|
||||
// One ApiClient per host. `factory` null = real HTTPS. `key` empty = look up
|
||||
// base.key_env through llm::Config (empty there too = not configured).
|
||||
std::shared_ptr<Gateway> make_gateway(const CallTable& table, const llm::ApiClientConfig& base,
|
||||
const std::vector<HostSpec>& hosts, std::string key,
|
||||
const TransportFactory& factory);
|
||||
|
||||
// A host name from the data plane of a database must be a bare https host.
|
||||
void check_host(const std::string& vendor, const std::string& host);
|
||||
|
||||
class Provider {
|
||||
public:
|
||||
explicit Provider(std::shared_ptr<Gateway> gateway) : gw_(std::move(gateway)) {}
|
||||
virtual ~Provider() = default;
|
||||
Gateway& gateway() const { return *gw_; }
|
||||
std::shared_ptr<Gateway> shared_gateway() const { return gw_; }
|
||||
bool configured() const { return gw_->configured(); }
|
||||
const CallTable& table() const { return gw_->table(); }
|
||||
|
||||
protected:
|
||||
std::shared_ptr<Gateway> gw_;
|
||||
};
|
||||
|
||||
// Json helpers
|
||||
struct J {
|
||||
static Json strings(const std::vector<std::string>& v);
|
||||
static Json object() { return Json::object(); }
|
||||
static std::string str(const Json& doc, const std::string& key, const std::string& fallback = "");
|
||||
static double num(const Json& doc, const std::string& key, double fallback = 0);
|
||||
static bool has_array(const Json& doc, const std::string& key);
|
||||
static const Json* at(const Json& doc, const std::string& key);
|
||||
static std::vector<float> floats(const Json& arr);
|
||||
// The array at `key`, or an empty static array.
|
||||
static const Json::Array& array(const Json& doc, const char* key);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
#include "vendors/retrieval/qdrant/qdrant.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// name, host, http method, path, spec operation id (or reference page / "doc:"),
|
||||
// body fields, query parameters, flags (d destructive, w writes, r read-only POST,
|
||||
// p paid), response schema)
|
||||
static const CallRow kRows[] = {
|
||||
{"create_shard_key", "main", "PUT", "/collections/{collection_name}/shards", "create_shard_key", "initial_state,placement,replication_factor,shard_key,shards_number", "timeout", "w", "inline"},
|
||||
{"list_shard_keys", "main", "GET", "/collections/{collection_name}/shards", "list_shard_keys", "", "", "", "inline"},
|
||||
{"cluster_status", "main", "GET", "/cluster", "cluster_status", "", "", "", "inline"},
|
||||
{"cluster_telemetry", "main", "GET", "/cluster/telemetry", "cluster_telemetry", "", "details_level,timeout", "", "inline"},
|
||||
{"get_quotas", "main", "GET", "/quotas", "get_quotas", "", "", "", "inline"},
|
||||
{"update_quotas", "main", "PUT", "/quotas", "update_quotas", "enabled,max_disk_usage_percent,max_resident_memory_percent,release_margin_percent", "wait", "w", "inline"},
|
||||
{"get_collections", "main", "GET", "/collections", "get_collections", "", "", "", "inline"},
|
||||
{"get_collection", "main", "GET", "/collections/{collection_name}", "get_collection", "", "", "", "inline"},
|
||||
{"create_collection", "main", "PUT", "/collections/{collection_name}", "create_collection", "hnsw_config,id_tracker,metadata,on_disk_payload,optimizers_config,payload,quantization_config,replication_factor,shard_number,sharding_method,sparse_vectors,strict_mode_config,vectors,wal_config,write_consistency_factor", "timeout", "w", "inline"},
|
||||
{"update_collection", "main", "PATCH", "/collections/{collection_name}", "update_collection", "hnsw_config,metadata,optimizers_config,params,quantization_config,sparse_vectors,strict_mode_config,vectors", "timeout", "w", "inline"},
|
||||
{"delete_collection", "main", "DELETE", "/collections/{collection_name}", "delete_collection", "", "timeout", "d", "inline"},
|
||||
{"update_aliases", "main", "POST", "/collections/aliases", "update_aliases", "actions", "timeout", "w", "inline"},
|
||||
{"create_field_index", "main", "PUT", "/collections/{collection_name}/index", "create_field_index", "field_name,field_schema", "ordering,timeout,wait", "w", "inline"},
|
||||
{"collection_exists", "main", "GET", "/collections/{collection_name}/exists", "collection_exists", "", "", "", "inline"},
|
||||
{"delete_field_index", "main", "DELETE", "/collections/{collection_name}/index/{field_name}", "delete_field_index", "", "ordering,timeout,wait", "d", "inline"},
|
||||
{"create_vector_name", "main", "PUT", "/collections/{collection_name}/vectors/{vector_name}", "create_vector_name", "dense,sparse", "ordering,timeout,wait", "w", "inline"},
|
||||
{"delete_vector_name", "main", "DELETE", "/collections/{collection_name}/vectors/{vector_name}", "delete_vector_name", "", "ordering,timeout,wait", "d", "inline"},
|
||||
{"get_optimizations", "main", "GET", "/collections/{collection_name}/optimizations", "get_optimizations", "", "completed_limit,with", "", "inline"},
|
||||
{"get_collection_aliases", "main", "GET", "/collections/{collection_name}/aliases", "get_collection_aliases", "", "", "", "inline"},
|
||||
{"get_collections_aliases", "main", "GET", "/aliases", "get_collections_aliases", "", "", "", "inline"},
|
||||
{"recover_from_uploaded_snapshot", "main", "POST", "/collections/{collection_name}/snapshots/upload", "recover_from_uploaded_snapshot", "snapshot", "checksum,priority,wait", "d", "inline"},
|
||||
{"recover_from_snapshot", "main", "PUT", "/collections/{collection_name}/snapshots/recover", "recover_from_snapshot", "api_key,checksum,location,priority", "wait", "d", "inline"},
|
||||
{"list_snapshots", "main", "GET", "/collections/{collection_name}/snapshots", "list_snapshots", "", "", "", "inline"},
|
||||
{"create_snapshot", "main", "POST", "/collections/{collection_name}/snapshots", "create_snapshot", "", "wait", "w", "inline"},
|
||||
{"delete_snapshot", "main", "DELETE", "/collections/{collection_name}/snapshots/{snapshot_name}", "delete_snapshot", "", "wait", "d", "inline"},
|
||||
{"get_snapshot", "main", "GET", "/collections/{collection_name}/snapshots/{snapshot_name}", "get_snapshot", "", "", "", "inline"},
|
||||
{"list_full_snapshots", "main", "GET", "/snapshots", "list_full_snapshots", "", "", "", "inline"},
|
||||
{"create_full_snapshot", "main", "POST", "/snapshots", "create_full_snapshot", "", "wait", "w", "inline"},
|
||||
{"delete_full_snapshot", "main", "DELETE", "/snapshots/{snapshot_name}", "delete_full_snapshot", "", "wait", "d", "inline"},
|
||||
{"get_full_snapshot", "main", "GET", "/snapshots/{snapshot_name}", "get_full_snapshot", "", "", "", "inline"},
|
||||
{"get_point", "main", "GET", "/collections/{collection_name}/points/{id}", "get_point", "", "consistency,timeout", "", "inline"},
|
||||
{"get_points", "main", "POST", "/collections/{collection_name}/points", "get_points", "ids,shard_key,with_payload,with_vector", "consistency,timeout", "r", "inline"},
|
||||
{"upsert_points", "main", "PUT", "/collections/{collection_name}/points", "upsert_points", "batch,points,shard_key,update_filter,update_mode", "ordering,timeout,wait", "w", "inline"},
|
||||
{"delete_points", "main", "POST", "/collections/{collection_name}/points/delete", "delete_points", "filter,points,shard_key", "ordering,timeout,wait", "d", "inline"},
|
||||
{"update_vectors", "main", "PUT", "/collections/{collection_name}/points/vectors", "update_vectors", "points,shard_key,update_filter", "ordering,timeout,wait", "w", "inline"},
|
||||
{"delete_vectors", "main", "POST", "/collections/{collection_name}/points/vectors/delete", "delete_vectors", "filter,points,shard_key,vector", "ordering,timeout,wait", "d", "inline"},
|
||||
{"set_payload", "main", "POST", "/collections/{collection_name}/points/payload", "set_payload", "filter,key,payload,points,shard_key", "ordering,timeout,wait", "w", "inline"},
|
||||
{"overwrite_payload", "main", "PUT", "/collections/{collection_name}/points/payload", "overwrite_payload", "filter,key,payload,points,shard_key", "ordering,timeout,wait", "w", "inline"},
|
||||
{"delete_payload", "main", "POST", "/collections/{collection_name}/points/payload/delete", "delete_payload", "filter,keys,points,shard_key", "ordering,timeout,wait", "d", "inline"},
|
||||
{"clear_payload", "main", "POST", "/collections/{collection_name}/points/payload/clear", "clear_payload", "filter,points,shard_key", "ordering,timeout,wait", "d", "inline"},
|
||||
{"batch_update", "main", "POST", "/collections/{collection_name}/points/batch", "batch_update", "operations", "ordering,timeout,wait", "w", "inline"},
|
||||
{"scroll_points", "main", "POST", "/collections/{collection_name}/points/scroll", "scroll_points", "filter,limit,offset,order_by,shard_key,with_payload,with_vector", "consistency,timeout", "r", "inline"},
|
||||
{"count_points", "main", "POST", "/collections/{collection_name}/points/count", "count_points", "exact,filter,shard_key", "consistency,timeout", "r", "inline"},
|
||||
{"facet", "main", "POST", "/collections/{collection_name}/facet", "facet", "exact,filter,key,limit,shard_key", "consistency,timeout", "r", "inline"},
|
||||
{"query_points", "main", "POST", "/collections/{collection_name}/points/query", "query_points", "filter,limit,lookup_from,offset,params,prefetch,query,score_threshold,shard_key,using,with_payload,with_vector", "consistency,timeout", "r", "inline"},
|
||||
{"query_batch_points", "main", "POST", "/collections/{collection_name}/points/query/batch", "query_batch_points", "searches", "consistency,timeout", "r", "inline"},
|
||||
{"query_points_groups", "main", "POST", "/collections/{collection_name}/points/query/groups", "query_points_groups", "filter,group_by,group_size,limit,lookup_from,params,prefetch,query,score_threshold,shard_key,using,with_lookup,with_payload,with_vector", "consistency,timeout", "r", "inline"},
|
||||
{"search_matrix_pairs", "main", "POST", "/collections/{collection_name}/points/search/matrix/pairs", "search_matrix_pairs", "filter,limit,sample,shard_key,using", "consistency,timeout", "r", "inline"},
|
||||
{"search_matrix_offsets", "main", "POST", "/collections/{collection_name}/points/search/matrix/offsets", "search_matrix_offsets", "filter,limit,sample,shard_key,using", "consistency,timeout", "r", "inline"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
const CallTable& qdrant_calls() {
|
||||
static const CallTable table{kRows, sizeof(kRows) / sizeof(kRows[0])};
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+195
@@ -0,0 +1,195 @@
|
||||
#include "vendors/retrieval/qdrant/qdrant.h"
|
||||
|
||||
#include "llm/config.h"
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
llm::ApiClientConfig Qdrant::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "qdrant";
|
||||
c.base_url = "";
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = "api-key";
|
||||
return c;
|
||||
}
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("qdrant", why); }
|
||||
|
||||
std::string trim_url(std::string url) {
|
||||
while (!url.empty() && url.back() == '/') url.pop_back();
|
||||
return url;
|
||||
}
|
||||
|
||||
Qdrant build(const std::string& url, std::string key, const TransportFactory& factory) {
|
||||
std::string base = trim_url(url);
|
||||
bool ok = base.rfind("https://", 0) == 0 || base.rfind("http://", 0) == 0;
|
||||
auto gw = make_gateway(qdrant_calls(), Qdrant::config(), {{"main", ok ? base : "https://qdrant.invalid"}},
|
||||
std::move(key), factory);
|
||||
if (!ok) gw->require_setting(Qdrant::kUrlEnv);
|
||||
return Qdrant(gw);
|
||||
}
|
||||
|
||||
CallArgs coll(const std::string& name) {
|
||||
CallArgs a;
|
||||
a.path["collection_name"] = name;
|
||||
return a;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Qdrant Qdrant::from_env() { return build(llm::Config::get(kUrlEnv), "", nullptr); }
|
||||
Qdrant Qdrant::with(const TransportFactory& factory, std::string key, const std::string& base_url) {
|
||||
return build(base_url, std::move(key), factory);
|
||||
}
|
||||
|
||||
const Json& Qdrant::result_of(const Json& doc) {
|
||||
static const Json empty = Json::object();
|
||||
const Json* r = doc.is_object() ? doc.find("result") : nullptr;
|
||||
return r ? *r : empty;
|
||||
}
|
||||
|
||||
Json Qdrant::point_json(const QdrantPoint& p) {
|
||||
bool id_ok = (p.id.is_number() && p.id.as_number() >= 0) || (p.id.is_string() && !p.id.as_string().empty() && p.id.as_string().size() <= 64);
|
||||
if (!id_ok) bad("point id must be an unsigned integer or a UUID string");
|
||||
if (p.vector.empty() || p.vector.size() > static_cast<std::size_t>(RetrievalCaps::kMaxDimensions)) bad("vector must hold 1..20000 numbers");
|
||||
Json o = Json::object();
|
||||
o.set("id", p.id);
|
||||
Json v = Json::array();
|
||||
for (float f : p.vector) v.push(Json(static_cast<double>(f)));
|
||||
o.set("vector", v);
|
||||
if (!p.payload.is_null()) {
|
||||
if (!p.payload.is_object()) bad("payload must be an object");
|
||||
o.set("payload", p.payload);
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
std::vector<QdrantHit> Qdrant::parse_hits(const Json& doc) {
|
||||
std::vector<QdrantHit> out;
|
||||
const Json& r = result_of(doc);
|
||||
for (const Json& p : J::array(r, "points")) {
|
||||
QdrantHit h;
|
||||
if (const Json* id = J::at(p, "id")) h.id = *id;
|
||||
h.score = J::num(p, "score");
|
||||
if (const Json* pl = J::at(p, "payload")) h.payload = *pl;
|
||||
out.push_back(std::move(h));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
Json Qdrant::list_collections() { return gw_->call("get_collections"); }
|
||||
|
||||
bool Qdrant::collection_exists(const std::string& name) {
|
||||
Json doc = gw_->call("collection_exists", coll(name));
|
||||
const Json* e = J::at(result_of(doc), "exists");
|
||||
return e && e->is_bool() && e->as_bool();
|
||||
}
|
||||
|
||||
Json Qdrant::get_collection(const std::string& name) { return gw_->call("get_collection", coll(name)); }
|
||||
|
||||
Json Qdrant::create_collection(const std::string& name, int size, const std::string& distance, bool on_disk_payload) {
|
||||
if (size < 1 || size > RetrievalCaps::kMaxDimensions) bad("size must be 1..20000");
|
||||
if (distance != "Cosine" && distance != "Euclid" && distance != "Dot" && distance != "Manhattan")
|
||||
bad("distance must be Cosine, Euclid, Dot or Manhattan");
|
||||
Json vectors = Json::object();
|
||||
vectors.set("size", Json(size));
|
||||
vectors.set("distance", Json(distance));
|
||||
Json body = Json::object();
|
||||
body.set("vectors", vectors);
|
||||
body.set("on_disk_payload", Json(on_disk_payload));
|
||||
CallArgs a = coll(name);
|
||||
a.body = body;
|
||||
return gw_->call("create_collection", a);
|
||||
}
|
||||
|
||||
Json Qdrant::delete_collection(const std::string& name) { return gw_->destroy("delete_collection", coll(name)); }
|
||||
|
||||
Json Qdrant::upsert(const std::string& collection, const std::vector<QdrantPoint>& points, bool wait) {
|
||||
if (points.empty() || points.size() > RetrievalCaps::kMaxPointsPerUpsert) bad("points must hold 1..1000 entries");
|
||||
Json arr = Json::array();
|
||||
for (const QdrantPoint& p : points) arr.push(point_json(p));
|
||||
Json body = Json::object();
|
||||
body.set("points", arr);
|
||||
CallArgs a = coll(collection);
|
||||
a.body = body;
|
||||
a.query["wait"] = Query::flag(wait);
|
||||
return gw_->call("upsert_points", a);
|
||||
}
|
||||
|
||||
std::vector<QdrantHit> Qdrant::query(const std::string& collection, const std::vector<float>& vector, int limit,
|
||||
const Json& filter, bool with_payload) {
|
||||
if (vector.empty() || vector.size() > static_cast<std::size_t>(RetrievalCaps::kMaxDimensions)) bad("vector must hold 1..20000 numbers");
|
||||
if (limit < 1 || limit > RetrievalCaps::kMaxPointLimit) bad("limit must be 1..10000");
|
||||
Json v = Json::array();
|
||||
for (float f : vector) v.push(Json(static_cast<double>(f)));
|
||||
Json body = Json::object();
|
||||
body.set("query", v);
|
||||
body.set("limit", Json(limit));
|
||||
body.set("with_payload", Json(with_payload));
|
||||
if (!filter.is_null()) body.set("filter", filter);
|
||||
return parse_hits(query_raw(collection, body));
|
||||
}
|
||||
|
||||
Json Qdrant::query_raw(const std::string& collection, const Json& body) {
|
||||
CallArgs a = coll(collection);
|
||||
a.body = body;
|
||||
return gw_->call("query_points", a);
|
||||
}
|
||||
|
||||
Json Qdrant::scroll(const std::string& collection, int limit, const Json& offset, const Json& filter) {
|
||||
if (limit < 1 || limit > RetrievalCaps::kMaxPointLimit) bad("limit must be 1..10000");
|
||||
Json body = Json::object();
|
||||
body.set("limit", Json(limit));
|
||||
if (!offset.is_null()) body.set("offset", offset);
|
||||
if (!filter.is_null()) body.set("filter", filter);
|
||||
CallArgs a = coll(collection);
|
||||
a.body = body;
|
||||
return gw_->call("scroll_points", a);
|
||||
}
|
||||
|
||||
long Qdrant::count(const std::string& collection, const Json& filter, bool exact) {
|
||||
Json body = Json::object();
|
||||
body.set("exact", Json(exact));
|
||||
if (!filter.is_null()) body.set("filter", filter);
|
||||
CallArgs a = coll(collection);
|
||||
a.body = body;
|
||||
return static_cast<long>(J::num(result_of(gw_->call("count_points", a)), "count"));
|
||||
}
|
||||
|
||||
Json Qdrant::delete_points(const std::string& collection, const std::vector<Json>& ids) {
|
||||
if (ids.empty() || ids.size() > RetrievalCaps::kMaxPointsPerUpsert) bad("ids must hold 1..1000 entries");
|
||||
Json arr = Json::array();
|
||||
for (const Json& id : ids) {
|
||||
if (!id.is_number() && !id.is_string()) bad("an id must be a number or a string");
|
||||
arr.push(id);
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("points", arr);
|
||||
CallArgs a = coll(collection);
|
||||
a.body = body;
|
||||
a.query["wait"] = "true";
|
||||
return gw_->destroy("delete_points", a);
|
||||
}
|
||||
|
||||
Json Qdrant::delete_points_by_filter(const std::string& collection, const Json& filter) {
|
||||
if (!filter.is_object() || filter.as_object().empty()) bad("filter must be a non-empty object");
|
||||
Json body = Json::object();
|
||||
body.set("filter", filter);
|
||||
CallArgs a = coll(collection);
|
||||
a.body = body;
|
||||
a.query["wait"] = "true";
|
||||
return gw_->destroy("delete_points", a);
|
||||
}
|
||||
|
||||
Json Qdrant::create_snapshot(const std::string& collection) { return gw_->call("create_snapshot", coll(collection)); }
|
||||
Json Qdrant::list_snapshots(const std::string& collection) { return gw_->call("list_snapshots", coll(collection)); }
|
||||
Json Qdrant::delete_snapshot(const std::string& collection, const std::string& snapshot) {
|
||||
CallArgs a = coll(collection);
|
||||
a.path["snapshot_name"] = snapshot;
|
||||
return gw_->destroy("delete_snapshot", a);
|
||||
}
|
||||
Json Qdrant::cluster_status() { return gw_->call("cluster_status"); }
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
#pragma once
|
||||
|
||||
// Qdrant REST API: collections, points (upsert, query, scroll, count, delete),
|
||||
// snapshots and cluster status. The server is yours (QDRANT_URL, for example
|
||||
// https://xyz.cloud.qdrant.io:6333), so there is no vendor host.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/retrieval/provider.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallTable& qdrant_calls();
|
||||
|
||||
struct QdrantPoint {
|
||||
Json id; // unsigned integer or UUID string
|
||||
std::vector<float> vector;
|
||||
Json payload; // null = none
|
||||
};
|
||||
|
||||
struct QdrantHit {
|
||||
Json id;
|
||||
double score = 0;
|
||||
Json payload;
|
||||
};
|
||||
|
||||
class Qdrant : public Provider {
|
||||
public:
|
||||
static constexpr const char* kKeyEnv = "QDRANT_API_KEY";
|
||||
static constexpr const char* kUrlEnv = "QDRANT_URL";
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
static Qdrant from_env();
|
||||
static Qdrant with(const TransportFactory& factory, std::string key, const std::string& base_url);
|
||||
explicit Qdrant(std::shared_ptr<Gateway> gateway) : Provider(std::move(gateway)) {}
|
||||
|
||||
// Collections
|
||||
Json list_collections();
|
||||
bool collection_exists(const std::string& name);
|
||||
Json get_collection(const std::string& name);
|
||||
Json create_collection(const std::string& name, int size, const std::string& distance, bool on_disk_payload);
|
||||
Json delete_collection(const std::string& name); // destructive
|
||||
|
||||
// Points
|
||||
Json upsert(const std::string& collection, const std::vector<QdrantPoint>& points, bool wait);
|
||||
std::vector<QdrantHit> query(const std::string& collection, const std::vector<float>& vector, int limit,
|
||||
const Json& filter, bool with_payload);
|
||||
Json query_raw(const std::string& collection, const Json& body);
|
||||
Json scroll(const std::string& collection, int limit, const Json& offset, const Json& filter);
|
||||
long count(const std::string& collection, const Json& filter, bool exact);
|
||||
Json delete_points(const std::string& collection, const std::vector<Json>& ids); // destructive
|
||||
Json delete_points_by_filter(const std::string& collection, const Json& filter); // destructive
|
||||
|
||||
// Snapshots and cluster
|
||||
Json create_snapshot(const std::string& collection);
|
||||
Json list_snapshots(const std::string& collection);
|
||||
Json delete_snapshot(const std::string& collection, const std::string& snapshot); // destructive
|
||||
Json cluster_status();
|
||||
|
||||
static std::vector<QdrantHit> parse_hits(const Json& doc);
|
||||
static Json point_json(const QdrantPoint& p);
|
||||
static const Json& result_of(const Json& doc);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+48
@@ -0,0 +1,48 @@
|
||||
#include "vendors/retrieval/seams.h"
|
||||
|
||||
#include "llm/error.h"
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void bad(const std::string& vendor, const std::string& why) {
|
||||
throw llm::ApiError::invalid(vendor, why);
|
||||
}
|
||||
bool blank(const std::string& s) { return s.find_first_not_of(" \t\r\n") == std::string::npos; }
|
||||
} // namespace
|
||||
|
||||
void Embedder::validate(const std::string& vendor, const EmbedRequest& r) {
|
||||
if (r.texts.empty()) bad(vendor, "texts is empty");
|
||||
if (r.texts.size() > RetrievalCaps::kMaxEmbedTexts) bad(vendor, "too many texts");
|
||||
for (const std::string& t : r.texts) {
|
||||
if (blank(t)) bad(vendor, "a text is blank");
|
||||
if (t.size() > RetrievalCaps::kMaxTextBytes) bad(vendor, "a text is over the byte cap");
|
||||
}
|
||||
if (!r.input_type.empty() && r.input_type != "query" && r.input_type != "document")
|
||||
bad(vendor, "input_type must be query or document");
|
||||
if (r.dimensions < 0 || r.dimensions > RetrievalCaps::kMaxDimensions) bad(vendor, "dimensions out of range");
|
||||
}
|
||||
|
||||
void Reranker::validate(const std::string& vendor, const RerankRequest& r) {
|
||||
if (blank(r.query)) bad(vendor, "query is blank");
|
||||
if (r.query.size() > RetrievalCaps::kMaxQueryChars) bad(vendor, "query too long");
|
||||
if (r.documents.empty()) bad(vendor, "documents is empty");
|
||||
if (r.documents.size() > RetrievalCaps::kMaxRerankDocs) bad(vendor, "too many documents");
|
||||
for (const std::string& d : r.documents) {
|
||||
if (blank(d)) bad(vendor, "a document is blank");
|
||||
if (d.size() > RetrievalCaps::kMaxTextBytes) bad(vendor, "a document is over the byte cap");
|
||||
}
|
||||
if (r.top_n < 0 || r.top_n > RetrievalCaps::kMaxTopN) bad(vendor, "top_n out of range");
|
||||
}
|
||||
|
||||
void WebSearch::validate(const std::string& vendor, const SearchRequest& r) {
|
||||
if (blank(r.query)) bad(vendor, "query is blank");
|
||||
if (r.query.size() > RetrievalCaps::kMaxQueryChars) bad(vendor, "query too long");
|
||||
if (r.max_results < 1 || r.max_results > static_cast<int>(RetrievalCaps::kMaxSearchResults))
|
||||
bad(vendor, "max_results out of range");
|
||||
if (!r.recency.empty() && r.recency != "day" && r.recency != "week" && r.recency != "month" && r.recency != "year")
|
||||
bad(vendor, "recency must be day, week, month or year");
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Vendored
+92
@@ -0,0 +1,92 @@
|
||||
#pragma once
|
||||
|
||||
// Neutral seams of the retrieval package. Provider adapters implement them;
|
||||
// callers (RAG pipelines, the demo) depend on these only. src/llm is not
|
||||
// touched: these types are retrieval's own.
|
||||
//
|
||||
// Embedder texts -> vectors (Voyage, Jina, Pinecone, Mixedbread)
|
||||
// Reranker query + docs -> order (Voyage, Jina, Pinecone, Mixedbread)
|
||||
// WebSearch query -> hits (Jina, Tavily, Exa, Brave)
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
struct EmbedRequest {
|
||||
std::vector<std::string> texts;
|
||||
std::string model; // empty = the adapter's default
|
||||
std::string input_type; // "", "query" or "document"
|
||||
int dimensions = 0; // 0 = the model's default
|
||||
};
|
||||
|
||||
struct EmbedResult {
|
||||
std::vector<std::vector<float>> vectors; // one per input text, in order
|
||||
std::string model;
|
||||
long total_tokens = 0;
|
||||
};
|
||||
|
||||
class Embedder {
|
||||
public:
|
||||
virtual ~Embedder() = default;
|
||||
virtual std::string provider() const = 0;
|
||||
virtual EmbedResult embed(const EmbedRequest& request) = 0;
|
||||
// Caps shared by all adapters; throws llm::ApiError::invalid.
|
||||
static void validate(const std::string& vendor, const EmbedRequest& request);
|
||||
};
|
||||
|
||||
struct RerankRequest {
|
||||
std::string query;
|
||||
std::vector<std::string> documents;
|
||||
std::string model; // empty = the adapter's default
|
||||
int top_n = 0; // 0 = all documents
|
||||
};
|
||||
|
||||
struct RerankHit {
|
||||
int index = 0; // position in RerankRequest::documents
|
||||
double score = 0;
|
||||
};
|
||||
|
||||
struct RerankResult {
|
||||
std::vector<RerankHit> hits; // best first
|
||||
std::string model;
|
||||
long total_tokens = 0;
|
||||
};
|
||||
|
||||
class Reranker {
|
||||
public:
|
||||
virtual ~Reranker() = default;
|
||||
virtual std::string provider() const = 0;
|
||||
virtual RerankResult rerank(const RerankRequest& request) = 0;
|
||||
static void validate(const std::string& vendor, const RerankRequest& request);
|
||||
};
|
||||
|
||||
struct SearchRequest {
|
||||
std::string query;
|
||||
int max_results = 5;
|
||||
std::string recency; // "", "day", "week", "month", "year" (where supported)
|
||||
};
|
||||
|
||||
struct SearchHit {
|
||||
std::string title;
|
||||
std::string url;
|
||||
std::string snippet;
|
||||
std::string published; // as the provider gave it, may be empty
|
||||
double score = 0;
|
||||
bool has_score = false;
|
||||
};
|
||||
|
||||
struct SearchResult {
|
||||
std::vector<SearchHit> hits;
|
||||
std::string answer; // only when the provider returned one
|
||||
};
|
||||
|
||||
class WebSearch {
|
||||
public:
|
||||
virtual ~WebSearch() = default;
|
||||
virtual std::string provider() const = 0;
|
||||
virtual SearchResult search(const SearchRequest& request) = 0;
|
||||
static void validate(const std::string& vendor, const SearchRequest& request);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
#include "vendors/retrieval/tavily/tavily.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// name, host, http method, path, spec operation id (or reference page / "doc:"),
|
||||
// body fields, query parameters, flags (d destructive, w writes, r read-only POST,
|
||||
// p paid), response schema)
|
||||
static const CallRow kRows[] = {
|
||||
{"search", "main", "POST", "/search", "search", "auto_parameters,chunks_per_source,country,end_date,exact_match,exclude_domains,filter_by_language,filter_by_published_date,include_answer,include_domains,include_domains_mode,include_favicon,include_image_descriptions,include_images,include_published_date,include_raw_content,include_usage,language,max_results,query,safe_search,search_depth,start_date,time_range,topic", "", "rp", "inline"},
|
||||
{"extract", "main", "POST", "/extract", "extract", "chunks_per_source,extract_depth,format,include_favicon,include_images,include_usage,query,timeout,urls", "", "rp", "inline"},
|
||||
{"crawl", "main", "POST", "/crawl", "crawl", "allow_external,chunks_per_source,exclude_domains,exclude_paths,extract_depth,format,include_favicon,include_images,include_usage,instructions,limit,max_breadth,max_depth,select_domains,select_paths,timeout,url", "", "rp", "inline"},
|
||||
{"map", "main", "POST", "/map", "map", "allow_external,exclude_domains,exclude_paths,include_usage,instructions,limit,max_breadth,max_depth,select_domains,select_paths,timeout,url", "", "rp", "inline"},
|
||||
{"create_research", "main", "POST", "/research", "createResearch", "citation_format,exclude_domains,files,include_domains,input,model,output_length,output_schema,stream", "", "rp", "inline"},
|
||||
{"get_research", "main", "GET", "/research/{request_id}", "getResearch", "", "include_usage", "", "-"},
|
||||
{"get_usage", "main", "GET", "/usage", "getUsage", "", "", "", "inline"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
const CallTable& tavily_calls() {
|
||||
static const CallTable table{kRows, sizeof(kRows) / sizeof(kRows[0])};
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
#include "vendors/retrieval/tavily/tavily.h"
|
||||
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
llm::ApiClientConfig Tavily::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "tavily";
|
||||
c.base_url = kBaseUrl;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kBearer;
|
||||
return c;
|
||||
}
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("tavily", why); }
|
||||
CallArgs with_body(Json body) {
|
||||
CallArgs a;
|
||||
a.body = std::move(body);
|
||||
return a;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Tavily Tavily::from_env() { return Tavily(make_gateway(tavily_calls(), config(), {{"main", kBaseUrl}}, "", nullptr)); }
|
||||
Tavily Tavily::with(const TransportFactory& factory, std::string key) {
|
||||
return Tavily(make_gateway(tavily_calls(), config(), {{"main", kBaseUrl}}, std::move(key), factory));
|
||||
}
|
||||
|
||||
void Tavily::check_url(const std::string& url) {
|
||||
bool http = url.rfind("http://", 0) == 0 || url.rfind("https://", 0) == 0;
|
||||
if (!http || url.size() > 8192 || url.find_first_of(" \t\r\n") != std::string::npos)
|
||||
bad("url must be http(s) without spaces, at most 8192 chars");
|
||||
}
|
||||
|
||||
SearchResult Tavily::parse_search(const Json& doc) {
|
||||
SearchResult out;
|
||||
out.answer = J::str(doc, "answer");
|
||||
for (const Json& r : J::array(doc, "results")) {
|
||||
SearchHit h;
|
||||
h.title = J::str(r, "title");
|
||||
h.url = J::str(r, "url");
|
||||
h.snippet = J::str(r, "content");
|
||||
h.published = J::str(r, "published_date");
|
||||
if (const Json* s = J::at(r, "score"); s && s->is_number()) {
|
||||
h.score = s->as_number();
|
||||
h.has_score = true;
|
||||
}
|
||||
out.hits.push_back(std::move(h));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
TavilyExtract Tavily::parse_extract(const Json& doc) {
|
||||
TavilyExtract out;
|
||||
for (const Json& r : J::array(doc, "results")) out.pages.push_back({J::str(r, "url"), J::str(r, "raw_content")});
|
||||
for (const Json& r : J::array(doc, "failed_results")) out.failed.push_back(J::str(r, "url"));
|
||||
return out;
|
||||
}
|
||||
|
||||
TavilyCrawl Tavily::parse_crawl(const Json& doc) {
|
||||
TavilyCrawl out;
|
||||
out.base_url = J::str(doc, "base_url");
|
||||
for (const Json& r : J::array(doc, "results")) out.pages.push_back({J::str(r, "url"), J::str(r, "raw_content")});
|
||||
return out;
|
||||
}
|
||||
|
||||
SearchResult Tavily::search(const SearchRequest& request) { return search(request, TavilyOptions{}); }
|
||||
|
||||
SearchResult Tavily::search(const SearchRequest& request, const TavilyOptions& o) {
|
||||
WebSearch::validate("tavily", request);
|
||||
if (request.max_results > RetrievalCaps::kTavilyMaxResults) bad("max_results is at most 20");
|
||||
static const char* depths[] = {"", "basic", "advanced", "fast", "ultra-fast"};
|
||||
bool depth_ok = false;
|
||||
for (const char* d : depths) depth_ok = depth_ok || o.search_depth == d;
|
||||
if (!depth_ok) bad("search_depth must be basic, advanced, fast or ultra-fast");
|
||||
if (!o.topic.empty() && o.topic != "general" && o.topic != "news" && o.topic != "finance") bad("topic must be general, news or finance");
|
||||
Json body = Json::object();
|
||||
body.set("query", Json(request.query));
|
||||
body.set("max_results", Json(request.max_results));
|
||||
if (!o.search_depth.empty()) body.set("search_depth", Json(o.search_depth));
|
||||
if (!o.topic.empty()) body.set("topic", Json(o.topic));
|
||||
if (o.include_answer) body.set("include_answer", Json(true));
|
||||
if (o.include_raw_content) body.set("include_raw_content", Json(true));
|
||||
if (!o.include_domains.empty()) body.set("include_domains", J::strings(o.include_domains));
|
||||
if (!o.exclude_domains.empty()) body.set("exclude_domains", J::strings(o.exclude_domains));
|
||||
if (!request.recency.empty()) body.set("time_range", Json(request.recency));
|
||||
return parse_search(gw_->call("search", with_body(body)));
|
||||
}
|
||||
|
||||
TavilyExtract Tavily::extract(const std::vector<std::string>& urls, bool advanced) {
|
||||
if (urls.empty() || urls.size() > kMaxExtractUrls) bad("urls must hold 1..20 entries");
|
||||
for (const std::string& u : urls) check_url(u);
|
||||
Json body = Json::object();
|
||||
body.set("urls", J::strings(urls));
|
||||
if (advanced) body.set("extract_depth", Json("advanced"));
|
||||
return parse_extract(gw_->call("extract", with_body(body)));
|
||||
}
|
||||
|
||||
TavilyCrawl Tavily::crawl(const std::string& url, int limit, int max_depth, const std::string& instructions) {
|
||||
check_url(url);
|
||||
if (limit < 1 || limit > RetrievalCaps::kMaxCrawlLimit) bad("limit must be 1..500");
|
||||
if (max_depth < 1 || max_depth > RetrievalCaps::kMaxCrawlDepth) bad("max_depth must be 1..5");
|
||||
Json body = Json::object();
|
||||
body.set("url", Json(url));
|
||||
body.set("limit", Json(limit));
|
||||
body.set("max_depth", Json(max_depth));
|
||||
if (!instructions.empty()) body.set("instructions", Json(instructions));
|
||||
return parse_crawl(gw_->call("crawl", with_body(body)));
|
||||
}
|
||||
|
||||
std::vector<std::string> Tavily::map(const std::string& url, int limit, int max_depth) {
|
||||
check_url(url);
|
||||
if (limit < 1 || limit > RetrievalCaps::kMaxCrawlLimit) bad("limit must be 1..500");
|
||||
if (max_depth < 1 || max_depth > RetrievalCaps::kMaxCrawlDepth) bad("max_depth must be 1..5");
|
||||
Json body = Json::object();
|
||||
body.set("url", Json(url));
|
||||
body.set("limit", Json(limit));
|
||||
body.set("max_depth", Json(max_depth));
|
||||
Json doc = gw_->call("map", with_body(body));
|
||||
std::vector<std::string> out;
|
||||
for (const Json& r : J::array(doc, "results"))
|
||||
if (r.is_string()) out.push_back(r.as_string());
|
||||
return out;
|
||||
}
|
||||
|
||||
Json Tavily::research(const std::string& input) {
|
||||
if (input.empty() || input.size() > RetrievalCaps::kMaxQueryChars) bad("input must be 1..8192 chars");
|
||||
Json body = Json::object();
|
||||
body.set("input", Json(input));
|
||||
return gw_->call("create_research", with_body(body));
|
||||
}
|
||||
|
||||
Json Tavily::get_research(const std::string& request_id) {
|
||||
CallArgs a;
|
||||
a.path["request_id"] = request_id;
|
||||
return gw_->call("get_research", a);
|
||||
}
|
||||
|
||||
Json Tavily::usage() { return gw_->call("get_usage"); }
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
#pragma once
|
||||
|
||||
// Tavily: search, extract, crawl, map, research and usage. Implements the
|
||||
// WebSearch seam.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/retrieval/provider.h"
|
||||
#include "vendors/retrieval/seams.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallTable& tavily_calls();
|
||||
|
||||
struct TavilyOptions {
|
||||
std::string search_depth; // "", basic, advanced, fast, ultra-fast
|
||||
std::string topic; // "", general, news, finance
|
||||
bool include_answer = false;
|
||||
bool include_raw_content = false;
|
||||
std::vector<std::string> include_domains;
|
||||
std::vector<std::string> exclude_domains;
|
||||
};
|
||||
|
||||
struct TavilyPage {
|
||||
std::string url;
|
||||
std::string raw_content;
|
||||
};
|
||||
|
||||
struct TavilyExtract {
|
||||
std::vector<TavilyPage> pages;
|
||||
std::vector<std::string> failed; // urls
|
||||
};
|
||||
|
||||
struct TavilyCrawl {
|
||||
std::string base_url;
|
||||
std::vector<TavilyPage> pages;
|
||||
};
|
||||
|
||||
class Tavily : public Provider, public WebSearch {
|
||||
public:
|
||||
static constexpr const char* kKeyEnv = "TAVILY_API_KEY";
|
||||
static constexpr const char* kBaseUrl = "https://api.tavily.com";
|
||||
static constexpr std::size_t kMaxExtractUrls = 20; // doc: 20 urls per extract
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
static Tavily from_env();
|
||||
static Tavily with(const TransportFactory& factory, std::string key);
|
||||
explicit Tavily(std::shared_ptr<Gateway> gateway) : Provider(std::move(gateway)) {}
|
||||
|
||||
std::string provider() const override { return "tavily"; }
|
||||
SearchResult search(const SearchRequest& request) override;
|
||||
SearchResult search(const SearchRequest& request, const TavilyOptions& options);
|
||||
|
||||
TavilyExtract extract(const std::vector<std::string>& urls, bool advanced);
|
||||
TavilyCrawl crawl(const std::string& url, int limit, int max_depth, const std::string& instructions);
|
||||
std::vector<std::string> map(const std::string& url, int limit, int max_depth);
|
||||
Json research(const std::string& input);
|
||||
Json get_research(const std::string& request_id);
|
||||
Json usage();
|
||||
|
||||
static SearchResult parse_search(const Json& doc);
|
||||
static TavilyExtract parse_extract(const Json& doc);
|
||||
static TavilyCrawl parse_crawl(const Json& doc);
|
||||
static void check_url(const std::string& url);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Binary file not shown.
+649
@@ -0,0 +1,649 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check the retrieval call tables against each provider's published spec.
|
||||
|
||||
Sources (fetched live, python3 stdlib only, no credentials):
|
||||
|
||||
qdrant OpenAPI 3 JSON qdrant/qdrant master docs/redoc/master/openapi.json
|
||||
weaviate Swagger 2 JSON weaviate/weaviate main openapi-specs/schema.json
|
||||
pinecone OpenAPI 3 YAML pinecone-io/pinecone-api 2026-07: db_control, db_data, inference
|
||||
mixedbread OpenAPI 3 JSON https://api.mixedbread.com/openapi.json
|
||||
jina OpenAPI 3 JSON api.jina.ai, r.jina.ai, s.jina.ai, segment.jina.ai (/openapi.json)
|
||||
tavily OpenAPI 3 JSON https://docs.tavily.com/documentation/api-reference/openapi.json
|
||||
exa OpenAPI 3.1 YAML exa-labs/openapi-spec master exa-openapi-spec.yaml
|
||||
brave rendered API reference, one markdown page per operation
|
||||
(api-dashboard.search.brave.com/api-reference/<slug>/index.html.md)
|
||||
voyage official Python SDK wheel on PyPI (voyageai): resource names, base URL, parameters
|
||||
|
||||
For every provider it fails (exit 1) when
|
||||
row a declared call's method + path (+ operationId) is not in the spec;
|
||||
field a request body field or query parameter we send is not in the spec;
|
||||
coverage a spec operation is neither in the table nor in EXCLUDED with a reason.
|
||||
Rows whose source is prose documentation only (op starting with "doc:") are
|
||||
counted and listed as unverified, never silently accepted.
|
||||
|
||||
python3 -I src/vendors/retrieval/tools/check_spec.py [provider ...]
|
||||
python3 -I src/vendors/retrieval/tools/check_spec.py --rows provider # print table rows
|
||||
"""
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
import zipfile
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
CALLS_DIR = os.path.join(HERE, "..")
|
||||
RAW = "https://raw." + "githubusercontent.com/"
|
||||
BRAVE = "https://api-dashboard.search.brave.com"
|
||||
|
||||
SOURCES = {
|
||||
"qdrant": [("main", RAW + "qdrant/qdrant/master/docs/redoc/master/openapi.json")],
|
||||
"weaviate": [("main", RAW + "weaviate/weaviate/main/openapi-specs/schema.json")],
|
||||
"pinecone": [
|
||||
("control", RAW + "pinecone-io/pinecone-api/main/2026-07/db_control_2026-07.oas.yaml"),
|
||||
("data", RAW + "pinecone-io/pinecone-api/main/2026-07/db_data_2026-07.oas.yaml"),
|
||||
("inference", RAW + "pinecone-io/pinecone-api/main/2026-07/inference_2026-07.oas.yaml"),
|
||||
],
|
||||
"mixedbread": [("main", "https://api.mixedbread.com/openapi.json")],
|
||||
"jina": [
|
||||
("api", "https://api.jina.ai/openapi.json"),
|
||||
("reader", "https://r.jina.ai/openapi.json"),
|
||||
("search", "https://s.jina.ai/openapi.json"),
|
||||
("segment", "https://segment.jina.ai/openapi.json"),
|
||||
],
|
||||
"tavily": [("main", "https://docs.tavily.com/documentation/api-reference/openapi.json")],
|
||||
"exa": [("main", RAW + "exa-labs/openapi-spec/master/exa-openapi-spec.yaml")],
|
||||
}
|
||||
|
||||
# Spec operations that are deliberately not in the tables. (regex on
|
||||
# "HOST METHOD path", reason). Nothing is excluded silently.
|
||||
EXCLUDED = {
|
||||
"qdrant": [
|
||||
(r"^main (GET|POST) /(metrics|telemetry|healthz|livez|readyz)$|^main GET /$", "service health and telemetry, not retrieval"),
|
||||
(r"^main \w+ /(issues|debugger|locks)", "operator and debugging surface"),
|
||||
(r"^main \w+ /cluster/(recover|metadata)|^main \w+ /cluster/peer", "cluster membership operations stay with the operator"),
|
||||
(r"^main \w+ /collections/\{collection_name\}/shards/|^main \w+ /collections/\{collection_name\}/cluster", "shard placement is operator work"),
|
||||
],
|
||||
"weaviate": [
|
||||
(r"^main \w+ /(authz|users|groups|replication|nodes|cluster|meta|\.well-known|classifications|tasks|backups)", "operator, auth and cluster administration"),
|
||||
(r"^main GET /$", "discovery links"),
|
||||
(r"^main \w+ /experimental", "database user administration"),
|
||||
],
|
||||
"pinecone": [],
|
||||
"mixedbread": [
|
||||
(r"^main \w+ /v1/(integrations|admin|billing|trace-destinations|buckets|events|api-keys|data_sources|schemas)", "organisation administration, integrations and OAuth callbacks"),
|
||||
(r"^main GET /$", "service info"),
|
||||
],
|
||||
"jina": [
|
||||
(r"^api \w+ /v1/chat/completions", "chat model surface, not retrieval"),
|
||||
(r"^reader \w+ /(echo|ephemeral|bot|\.well-known)", "internal helper routes of the reader service"),
|
||||
(r"^reader \w+ /\{url\}|^search \w+ /\{q\}", "path form of the same call as `GET /` and `GET /search`"),
|
||||
(r"^reader POST /$|^search POST /search$", "POST form of the same call as the GET row"),
|
||||
],
|
||||
"tavily": [
|
||||
(r"^main POST /(feedback|logs|org-usage)$", "no operationId, account logging and organisation usage, not retrieval"),
|
||||
],
|
||||
"exa": [],
|
||||
}
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Minimal YAML (block mappings and sequences, scalars, one-line flow, block
|
||||
# scalars). Enough for the published OpenAPI files; verified against PyYAML
|
||||
# by tests/ tooling when PyYAML is present (see --selftest).
|
||||
|
||||
|
||||
def _scalar(text):
|
||||
t = text.strip()
|
||||
if t == "":
|
||||
return None
|
||||
if t[0] == '"' and t[-1] == '"' and len(t) >= 2:
|
||||
try:
|
||||
return json.loads(t)
|
||||
except ValueError:
|
||||
return t[1:-1]
|
||||
if t[0] == "'" and t[-1] == "'" and len(t) >= 2:
|
||||
return t[1:-1].replace("''", "'")
|
||||
if t[0] in "[{":
|
||||
try:
|
||||
return json.loads(t)
|
||||
except ValueError:
|
||||
try:
|
||||
return _flow(t)
|
||||
except Exception:
|
||||
return t
|
||||
low = t.lower()
|
||||
if low in ("true", "false"):
|
||||
return low == "true"
|
||||
if low in ("null", "~"):
|
||||
return None
|
||||
if re.fullmatch(r"-?\d+", t):
|
||||
return int(t)
|
||||
if re.fullmatch(r"-?\d+\.\d+", t):
|
||||
return float(t)
|
||||
return t
|
||||
|
||||
|
||||
def _split_flow(body):
|
||||
parts, depth, cur, q = [], 0, "", None
|
||||
for ch in body:
|
||||
if q:
|
||||
cur += ch
|
||||
if ch == q:
|
||||
q = None
|
||||
elif ch in "\"'":
|
||||
q = ch
|
||||
cur += ch
|
||||
elif ch in "[{":
|
||||
depth += 1
|
||||
cur += ch
|
||||
elif ch in "]}":
|
||||
depth -= 1
|
||||
cur += ch
|
||||
elif ch == "," and depth == 0:
|
||||
parts.append(cur)
|
||||
cur = ""
|
||||
else:
|
||||
cur += ch
|
||||
if cur.strip():
|
||||
parts.append(cur)
|
||||
return parts
|
||||
|
||||
|
||||
def _flow(t):
|
||||
t = t.strip()
|
||||
if t[0] == "[":
|
||||
return [_scalar(p) for p in _split_flow(t[1:-1])]
|
||||
out = {}
|
||||
for p in _split_flow(t[1:-1]):
|
||||
k, _, v = p.partition(":")
|
||||
out[_scalar(k)] = _scalar(v)
|
||||
return out
|
||||
|
||||
|
||||
def _strip_comment(line):
|
||||
if "#" not in line:
|
||||
return line
|
||||
q = None
|
||||
for i, ch in enumerate(line):
|
||||
if q:
|
||||
if ch == q:
|
||||
q = None
|
||||
elif ch in "\"'":
|
||||
q = ch
|
||||
elif ch == "#" and (i == 0 or line[i - 1] in " \t"):
|
||||
return line[:i].rstrip()
|
||||
return line
|
||||
|
||||
|
||||
def _bracket_depth(t):
|
||||
d, q = 0, None
|
||||
for ch in t:
|
||||
if q:
|
||||
if ch == q:
|
||||
q = None
|
||||
elif ch in "\"'":
|
||||
q = ch
|
||||
elif ch in "[{":
|
||||
d += 1
|
||||
elif ch in "]}":
|
||||
d -= 1
|
||||
return d
|
||||
|
||||
|
||||
def mini_yaml(text):
|
||||
raw = text.split("\n")
|
||||
lines = [] # (indent, content, original index)
|
||||
i = 0
|
||||
while i < len(raw):
|
||||
line = raw[i]
|
||||
if not line.strip() or line.lstrip().startswith("#") or line.startswith("---"):
|
||||
i += 1
|
||||
continue
|
||||
indent = len(line) - len(line.lstrip(" "))
|
||||
content = _strip_comment(line.strip())
|
||||
# multi-line flow collection: join until the brackets balance
|
||||
if re.match(r"^(?:(?:[^\s\[{\"'][^:]*?|\S):\s+|-\s+)?[\[{]", content) and not re.match(r"^.*?:\s+[|>]", content):
|
||||
while _bracket_depth(content) > 0 and i + 1 < len(raw):
|
||||
i += 1
|
||||
content += " " + _strip_comment(raw[i].strip())
|
||||
m = re.match(r"^(.*?:|-)\s+[|>][-+]?\d*$", content) or re.match(r"^()[|>][-+]?\d*$", content)
|
||||
lines.append([indent, content, None])
|
||||
if m:
|
||||
j = i + 1
|
||||
block = []
|
||||
while j < len(raw) and (not raw[j].strip() or len(raw[j]) - len(raw[j].lstrip(" ")) > indent):
|
||||
block.append(raw[j].strip())
|
||||
j += 1
|
||||
lines[-1][2] = "\n".join(block)
|
||||
i = j
|
||||
continue
|
||||
i += 1
|
||||
pos = [0]
|
||||
|
||||
def parse(indent):
|
||||
if pos[0] >= len(lines):
|
||||
return None
|
||||
ind, content, _ = lines[pos[0]]
|
||||
if content[0] in "[{":
|
||||
pos[0] += 1
|
||||
return _scalar(content)
|
||||
if content.startswith("- ") or content == "-":
|
||||
return parse_seq(ind)
|
||||
return parse_map(ind)
|
||||
|
||||
def split_key(content):
|
||||
if content[0] in "\"'":
|
||||
q = content[0]
|
||||
end = content.index(q, 1)
|
||||
rest = content[end + 1:]
|
||||
if rest.startswith(":"):
|
||||
return content[1:end], rest[1:].strip()
|
||||
m = re.match(r"^([^\s\[{][^:]*?|\S):(?:\s+(.*))?$", content)
|
||||
if m:
|
||||
return m.group(1), (m.group(2) or "").strip()
|
||||
return None, None
|
||||
|
||||
anchors = {}
|
||||
|
||||
def value_after(key_indent, rest, block):
|
||||
if block is not None:
|
||||
return block
|
||||
if rest.startswith("*"):
|
||||
return anchors.get(rest[1:].strip())
|
||||
if rest.startswith("&"):
|
||||
name, _, rest = rest[1:].partition(" ")
|
||||
val = value_after(key_indent, rest.strip(), None)
|
||||
anchors[name] = val
|
||||
return val
|
||||
if rest != "":
|
||||
return _scalar(rest)
|
||||
if pos[0] < len(lines):
|
||||
nind, ncontent, _ = lines[pos[0]]
|
||||
if nind > key_indent or (nind == key_indent and (ncontent.startswith("- ") or ncontent == "-")):
|
||||
return parse(nind)
|
||||
return None
|
||||
|
||||
def parse_map(ind):
|
||||
out = {}
|
||||
while pos[0] < len(lines):
|
||||
cind, content, block = lines[pos[0]]
|
||||
if cind != ind or content.startswith("- "):
|
||||
break
|
||||
key, rest = split_key(content)
|
||||
if key is None:
|
||||
raise ValueError("yaml: cannot parse line: " + content)
|
||||
pos[0] += 1
|
||||
if block is not None:
|
||||
rest = ""
|
||||
out[key] = value_after(ind, rest, block)
|
||||
return out
|
||||
|
||||
def parse_seq(ind):
|
||||
out = []
|
||||
while pos[0] < len(lines):
|
||||
cind, content, block = lines[pos[0]]
|
||||
if cind != ind or not (content.startswith("- ") or content == "-"):
|
||||
break
|
||||
item = content[1:].strip()
|
||||
if item == "":
|
||||
pos[0] += 1
|
||||
out.append(parse(ind + 1) if pos[0] < len(lines) and lines[pos[0]][0] > ind else None)
|
||||
continue
|
||||
key, rest = split_key(item)
|
||||
if key is not None and not item.startswith(("[", "{")):
|
||||
# inline mapping start: treat "- k: v" as a map at indent ind+2
|
||||
lines[pos[0]] = [ind + 2, item, block]
|
||||
out.append(parse_map(ind + 2))
|
||||
else:
|
||||
pos[0] += 1
|
||||
out.append(block if block is not None else _scalar(item))
|
||||
return out
|
||||
|
||||
return parse(0)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def fetch(url):
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "molo-retrieval-spec-check/1"})
|
||||
with urllib.request.urlopen(req, timeout=90) as r:
|
||||
return r.read().decode("utf-8", "replace")
|
||||
|
||||
|
||||
_cache = {}
|
||||
|
||||
|
||||
def load_doc(url):
|
||||
if url not in _cache:
|
||||
text = fetch(url)
|
||||
_cache[url] = json.loads(text) if url.endswith(".json") else mini_yaml(text)
|
||||
return _cache[url]
|
||||
|
||||
|
||||
def deref(doc, node):
|
||||
seen = 0
|
||||
while isinstance(node, dict) and "$ref" in node and seen < 20:
|
||||
ref = node["$ref"]
|
||||
cur = doc
|
||||
for part in ref.lstrip("#/").split("/"):
|
||||
cur = cur[part.replace("~1", "/").replace("~0", "~")]
|
||||
node = cur
|
||||
seen += 1
|
||||
return node
|
||||
|
||||
|
||||
def props(doc, schema, depth=0):
|
||||
"""Top-level property names a schema accepts ('*' when free-form)."""
|
||||
schema = deref(doc, schema)
|
||||
if not isinstance(schema, dict) or depth > 8:
|
||||
return set()
|
||||
out = set()
|
||||
for key in ("allOf", "anyOf", "oneOf"):
|
||||
for sub in schema.get(key, []) or []:
|
||||
out |= props(doc, sub, depth + 1)
|
||||
out |= set((schema.get("properties") or {}).keys())
|
||||
if schema.get("type") == "array":
|
||||
out.add("*")
|
||||
if not out and (schema.get("additionalProperties") is True or schema.get("type") == "object" and not schema.get("properties")):
|
||||
out.add("*")
|
||||
return out
|
||||
|
||||
|
||||
def response_name(op):
|
||||
for code in ("200", "201", "202", 200, 201, 202):
|
||||
r = (op.get("responses") or {}).get(code)
|
||||
if isinstance(r, dict):
|
||||
content = (r.get("content") or {})
|
||||
for ct, body in content.items():
|
||||
sch = (body or {}).get("schema") or {}
|
||||
if "$ref" in sch:
|
||||
return sch["$ref"].split("/")[-1]
|
||||
if sch.get("type"):
|
||||
return "inline"
|
||||
if r.get("schema", {}).get("$ref"):
|
||||
return r["schema"]["$ref"].split("/")[-1]
|
||||
return "-"
|
||||
return "-"
|
||||
|
||||
|
||||
def spec_ops(provider):
|
||||
"""{ 'host METHOD path': dict(op_id, body:set, query:set, response) }."""
|
||||
ops = {}
|
||||
for host, url in SOURCES[provider]:
|
||||
doc = load_doc(url)
|
||||
for path, item in doc["paths"].items():
|
||||
common = item.get("parameters", []) or []
|
||||
for method, op in item.items():
|
||||
if method not in ("get", "post", "put", "patch", "delete"):
|
||||
continue
|
||||
params = [deref(doc, p) for p in common + (op.get("parameters") or [])]
|
||||
query = {p["name"] for p in params if p.get("in") == "query"}
|
||||
headers = {p["name"] for p in params if p.get("in") == "header"}
|
||||
body = set()
|
||||
rb = deref(doc, op.get("requestBody") or {})
|
||||
for ct, c in (rb.get("content") or {}).items():
|
||||
body |= props(doc, (c or {}).get("schema") or {})
|
||||
for p in params: # swagger 2
|
||||
if p.get("in") == "body":
|
||||
body |= props(doc, p.get("schema") or {})
|
||||
if p.get("in") == "formData":
|
||||
body.add(p["name"])
|
||||
ops["%s %s %s" % (host, method.upper(), path)] = {
|
||||
"op_id": op.get("operationId") or "",
|
||||
"body": body, "query": query, "headers": headers,
|
||||
"response": response_name(op),
|
||||
}
|
||||
return ops
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Call tables (parsed from the C++ sources)
|
||||
|
||||
STR = re.compile(r'"((?:[^"\\]|\\.)*)"')
|
||||
|
||||
|
||||
def table_rows(provider):
|
||||
path = os.path.join(CALLS_DIR, provider, "calls.cpp")
|
||||
text = open(path).read()
|
||||
a, b = text.index("CALLS-BEGIN"), text.index("CALLS-END")
|
||||
rows = []
|
||||
for line in text[a:b].splitlines()[1:]:
|
||||
s = line.strip()
|
||||
if s.startswith("{") and '"' in s:
|
||||
f = STR.findall(s)
|
||||
rows.append(dict(zip(("name", "host", "method", "path", "op", "body", "query", "flags", "resp"), f)))
|
||||
return rows
|
||||
|
||||
|
||||
def split(csv):
|
||||
return {x for x in csv.split(",") if x}
|
||||
|
||||
|
||||
def snake(s):
|
||||
s = re.sub(r"[^A-Za-z0-9]+", "_", s)
|
||||
s = re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", s)
|
||||
return re.sub(r"_+", "_", s).strip("_").lower()
|
||||
|
||||
|
||||
READ_POST = re.compile(r"find_similar|contents|graphql|tokenize|aggregate|mcp_post|validate|search|query|scroll|count|fetch|list|recommend|discover|facet|matrix|retrieve|describe|get|embed|rerank|grep|read|extract|crawl|map|answer|enhance|validate|segment|classify|suggest|context|completions|stats")
|
||||
DESTRUCTIVE = re.compile(r"delete|remove|clear|recover|drop|truncate|reset|revoke|cancel|abort|disconnect|purge")
|
||||
PAID = {
|
||||
"voyage": re.compile(r".*"),
|
||||
"jina": re.compile(r"embeddings|rerank|read|search|segment|classify|count_token|batch_embeddings|create_batch"),
|
||||
"pinecone": re.compile(r"embed|rerank|search_records|upsert_records"),
|
||||
"mixedbread": re.compile(r"^(create_embeddings|rerank|create_question_answering|deprecated_create_question_answering|search_store_chunks|search_vector_store_chunks|enhance_query|create_parsing_job|create_extraction_job|extract_content|create_chat_completion|create_response)$"),
|
||||
"tavily": re.compile(r"search|extract|crawl|map|research"),
|
||||
"exa": re.compile(r"search|find_similar|contents|answer|research"),
|
||||
"brave": re.compile(r".*"),
|
||||
}
|
||||
|
||||
|
||||
RENAMES = {
|
||||
"jina": {
|
||||
"embeddings_v1_embeddings_post": "embeddings", "rerank_v1_rerank_post": "rerank",
|
||||
"list_models_v1_models_get": "list_models", "get_model_v1_models_model_id_get": "get_model",
|
||||
"create_batch_job_v1_batch_embeddings_post": "create_batch",
|
||||
"get_batch_status_v1_batch_batch_id_get": "get_batch",
|
||||
"cancel_batch_job_v1_batch_batch_id_delete": "cancel_batch",
|
||||
"list_batch_jobs_v1_batches_get": "list_batches",
|
||||
"download_batch_output_v1_batch_batch_id_output_get": "batch_output",
|
||||
"download_batch_errors_v1_batch_batch_id_errors_get": "batch_errors",
|
||||
"get": "read", "get_search": "search", "post_count_token": "count_token",
|
||||
},
|
||||
"exa": {
|
||||
"get_contents": "contents", "research_tasks_create": "research_create",
|
||||
"research_tasks_list": "research_list", "research_controller_v0_get_research_task": "research_get",
|
||||
},
|
||||
"pinecone": {
|
||||
"list_namespaces_operation": "list_namespaces",
|
||||
"create_index_from_backup_operation": "create_index_from_backup",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def op_name(provider, host, method, path, op_id):
|
||||
base = snake(op_id) if op_id else snake(method + "_" + path)
|
||||
return RENAMES.get(provider, {}).get(base, base)
|
||||
|
||||
|
||||
def make_flags(provider, name, method):
|
||||
f = ""
|
||||
if method == "DELETE" or DESTRUCTIVE.search(name):
|
||||
f += "d"
|
||||
elif method != "GET":
|
||||
f += "r" if READ_POST.search(name) else "w"
|
||||
if PAID.get(provider) and PAID[provider].search(name) and (method not in ("GET", "DELETE") or provider == "jina"):
|
||||
f += "p"
|
||||
return f
|
||||
|
||||
|
||||
def excluded(provider, key):
|
||||
for rx, why in EXCLUDED.get(provider, []):
|
||||
if re.search(rx, key):
|
||||
return why
|
||||
return None
|
||||
|
||||
|
||||
def gen_rows(provider):
|
||||
ops = spec_ops(provider)
|
||||
seen = set()
|
||||
out = []
|
||||
for key, o in ops.items():
|
||||
if excluded(provider, key):
|
||||
continue
|
||||
host, method, path = key.split(" ", 2)
|
||||
name = op_name(provider, host, method, path, o["op_id"])
|
||||
while name in seen:
|
||||
name += "_" + method.lower()
|
||||
seen.add(name)
|
||||
body = ",".join(sorted(o["body"]))
|
||||
out.append(' {"%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s"},' % (
|
||||
name, host, method, path, o["op_id"], body, ",".join(sorted(o["query"])), make_flags(provider, name, method), o["response"]))
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def check_openapi(provider, errors, notes):
|
||||
ops = spec_ops(provider)
|
||||
rows = table_rows(provider)
|
||||
keys = set()
|
||||
unverified = 0
|
||||
for r in rows:
|
||||
if r["op"].startswith("doc:"):
|
||||
unverified += 1
|
||||
notes.append("%s: unverified (prose docs only): %s %s %s" % (provider, r["name"], r["method"], r["path"]))
|
||||
continue
|
||||
key = "%s %s %s" % (r["host"], r["method"], r["path"])
|
||||
keys.add(key)
|
||||
o = ops.get(key)
|
||||
if not o:
|
||||
errors.append("%s row %s: %s not in the spec" % (provider, r["name"], key))
|
||||
continue
|
||||
if o["op_id"] != r["op"]:
|
||||
errors.append("%s row %s: operationId %r, spec says %r" % (provider, r["name"], r["op"], o["op_id"]))
|
||||
if r["body"] != "*":
|
||||
extra = split(r["body"]) - o["body"] - ({"*"} if "*" in o["body"] else set())
|
||||
if extra and "*" not in o["body"]:
|
||||
errors.append("%s row %s: body fields not in spec: %s" % (provider, r["name"], sorted(extra)))
|
||||
extra = split(r["query"]) - o["query"]
|
||||
if extra:
|
||||
errors.append("%s row %s: query parameters not in spec: %s" % (provider, r["name"], sorted(extra)))
|
||||
for key in ops:
|
||||
if key not in keys and not excluded(provider, key):
|
||||
errors.append("%s coverage: spec operation %s is neither in the table nor excluded" % (provider, key))
|
||||
notes.append("%s: %d spec operations, %d table rows (%d unverified), %d excluded" % (
|
||||
provider, len(ops), len(rows), unverified, sum(1 for k in ops if excluded(provider, k))))
|
||||
|
||||
|
||||
def check_brave(errors, notes):
|
||||
rows = table_rows("brave")
|
||||
html = fetch(BRAVE + "/api-reference/web/local_pois")
|
||||
slugs = sorted(set(re.findall(r'href="/api-reference/([^"#?]+)"', html)))
|
||||
covered = set()
|
||||
for r in rows:
|
||||
slug = r["op"]
|
||||
covered.add(slug)
|
||||
if slug not in slugs:
|
||||
errors.append("brave row %s: reference page %s not listed" % (r["name"], slug))
|
||||
continue
|
||||
page = fetch("%s/api-reference/%s/index.html.md" % (BRAVE, slug))
|
||||
m = re.search(r"`(GET|POST)` `([^`]+)`", page)
|
||||
if not m or m.group(1) != r["method"] or m.group(2) != r["path"]:
|
||||
errors.append("brave row %s: page says %s" % (r["name"], m.groups() if m else None))
|
||||
tables = {}
|
||||
cur = None
|
||||
for line in page.splitlines():
|
||||
if line.startswith("## "):
|
||||
cur = line[3:].strip()
|
||||
elif line.startswith("| `") and cur in ("Query Parameters", "Request Body", "Headers"):
|
||||
tables.setdefault(cur, set()).add(line.split("`")[1])
|
||||
for f in split(r["query"]):
|
||||
if f not in tables.get("Query Parameters", set()):
|
||||
errors.append("brave row %s: query parameter %s not on page" % (r["name"], f))
|
||||
if r["body"] != "*":
|
||||
for f in split(r["body"]):
|
||||
if f not in tables.get("Request Body", set()):
|
||||
errors.append("brave row %s: body field %s not on page" % (r["name"], f))
|
||||
for s in slugs:
|
||||
if s not in covered:
|
||||
errors.append("brave coverage: reference page %s is not in the table" % s)
|
||||
notes.append("brave: %d reference pages, %d table rows" % (len(slugs), len(rows)))
|
||||
|
||||
|
||||
def check_voyage(errors, notes):
|
||||
meta = json.loads(fetch("https://pypi.org/pypi/voyageai/json"))
|
||||
wheel = next(f for f in meta["urls"] if f["filename"].endswith(".whl"))
|
||||
with urllib.request.urlopen(urllib.request.Request(wheel["url"], headers={"User-Agent": "molo/1"}), timeout=90) as r:
|
||||
z = zipfile.ZipFile(io.BytesIO(r.read()))
|
||||
src = {n: z.read(n).decode("utf-8", "replace") for n in z.namelist() if n.endswith(".py")}
|
||||
blob = "\n".join(src.values())
|
||||
resources = set(re.findall(r'OBJECT_NAME = "([a-z]+)"', blob))
|
||||
base = re.search(r'"(https://api\.voyageai\.com/v1)"', blob)
|
||||
rows = table_rows("voyage")
|
||||
if not base:
|
||||
errors.append("voyage: base URL not found in the SDK")
|
||||
for r in rows:
|
||||
if r["method"] != "POST" or r["path"].lstrip("/").replace("v1/", "") not in resources:
|
||||
errors.append("voyage row %s: resource %s not in SDK resources %s" % (r["name"], r["path"], sorted(resources)))
|
||||
for f in split(r["body"]):
|
||||
if not re.search(r"\b%s\b" % re.escape(f), blob):
|
||||
errors.append("voyage row %s: field %s not in the SDK source" % (r["name"], f))
|
||||
missing = resources - {r["path"].replace("/v1/", "").lstrip("/") for r in rows}
|
||||
for m in missing:
|
||||
errors.append("voyage coverage: SDK resource %s not in the table" % m)
|
||||
notes.append("voyage: SDK voyageai %s, resources %s, %d table rows" % (meta["info"]["version"], sorted(resources), len(rows)))
|
||||
|
||||
|
||||
def selftest():
|
||||
try:
|
||||
import yaml # noqa
|
||||
except ImportError:
|
||||
print("selftest: PyYAML absent, skipped")
|
||||
return 0
|
||||
bad = 0
|
||||
for provider in ("pinecone", "exa"):
|
||||
for host, url in SOURCES[provider]:
|
||||
text = fetch(url)
|
||||
mine = mini_yaml(text)
|
||||
ref = yaml.safe_load(text)
|
||||
def strip(x): # examples are free text; only the structure is compared
|
||||
if isinstance(x, dict):
|
||||
return {k: strip(v) for k, v in x.items() if k not in ("example", "examples", "x-codeSamples", "description", "default")}
|
||||
if isinstance(x, list):
|
||||
return [strip(v) for v in x]
|
||||
return x
|
||||
same = json.dumps(strip(mine["paths"]), sort_keys=True, default=str) == json.dumps(strip(ref["paths"]), sort_keys=True, default=str)
|
||||
print("selftest", provider, host, "paths equal" if same else "paths DIFFER")
|
||||
bad += 0 if same else 1
|
||||
return bad
|
||||
|
||||
|
||||
def main(argv):
|
||||
if argv and argv[0] == "--rows":
|
||||
print("\n".join(gen_rows(argv[1])))
|
||||
return 0
|
||||
if argv and argv[0] == "--selftest":
|
||||
return 1 if selftest() else 0
|
||||
wanted = argv or ["qdrant", "weaviate", "pinecone", "mixedbread", "jina", "tavily", "exa", "brave", "voyage"]
|
||||
errors, notes = [], []
|
||||
for p in wanted:
|
||||
if p == "brave":
|
||||
check_brave(errors, notes)
|
||||
elif p == "voyage":
|
||||
check_voyage(errors, notes)
|
||||
else:
|
||||
check_openapi(p, errors, notes)
|
||||
for n in notes:
|
||||
print("note:", n)
|
||||
for e in errors:
|
||||
print("FAIL:", e)
|
||||
print("check_spec: %d problems" % len(errors))
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
#include "vendors/retrieval/voyage/voyage.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// name, host, http method, path, spec operation id (or reference page / "doc:"),
|
||||
// body fields, query parameters, flags (d destructive, w writes, r read-only POST,
|
||||
// p paid), response schema)
|
||||
static const CallRow kRows[] = {
|
||||
{"embed", "main", "POST", "/v1/embeddings", "sdk:Embedding", "input,model,input_type,truncation,output_dimension,output_dtype,encoding_format", "", "p", "-"},
|
||||
{"multimodal_embed", "main", "POST", "/v1/multimodalembeddings", "sdk:MultimodalEmbedding", "inputs,model,input_type,truncation,output_dimension,output_dtype", "", "p", "-"},
|
||||
{"contextualized_embed", "main", "POST", "/v1/contextualizedembeddings", "sdk:ContextualizedEmbedding", "inputs,model,input_type,output_dimension,output_dtype,encoding_format", "", "p", "-"},
|
||||
{"rerank", "main", "POST", "/v1/rerank", "sdk:Reranking", "query,documents,model,top_k,truncation", "", "p", "-"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
const CallTable& voyage_calls() {
|
||||
static const CallTable table{kRows, sizeof(kRows) / sizeof(kRows[0])};
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+169
@@ -0,0 +1,169 @@
|
||||
#include "vendors/retrieval/voyage/voyage.h"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
llm::ApiClientConfig Voyage::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "voyage";
|
||||
c.base_url = kBaseUrl;
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kBearer;
|
||||
return c;
|
||||
}
|
||||
|
||||
Voyage Voyage::from_env() {
|
||||
return Voyage(make_gateway(voyage_calls(), config(), {{"main", kBaseUrl}}, "", nullptr));
|
||||
}
|
||||
|
||||
Voyage Voyage::with(const TransportFactory& factory, std::string key) {
|
||||
return Voyage(make_gateway(voyage_calls(), config(), {{"main", kBaseUrl}}, std::move(key), factory));
|
||||
}
|
||||
|
||||
EmbedResult Voyage::parse_embeddings(const Json& doc) {
|
||||
EmbedResult out;
|
||||
out.model = J::str(doc, "model");
|
||||
std::vector<std::pair<int, std::vector<float>>> rows;
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
const Json* e = J::at(d, "embedding");
|
||||
rows.emplace_back(static_cast<int>(J::num(d, "index", static_cast<double>(rows.size()))),
|
||||
e ? J::floats(*e) : std::vector<float>{});
|
||||
}
|
||||
std::stable_sort(rows.begin(), rows.end(), [](const auto& a, const auto& b) { return a.first < b.first; });
|
||||
for (auto& r : rows) out.vectors.push_back(std::move(r.second));
|
||||
if (const Json* u = J::at(doc, "usage")) out.total_tokens = static_cast<long>(J::num(*u, "total_tokens"));
|
||||
return out;
|
||||
}
|
||||
|
||||
RerankResult Voyage::parse_rerank(const Json& doc) {
|
||||
RerankResult out;
|
||||
out.model = J::str(doc, "model");
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
RerankHit h;
|
||||
h.index = static_cast<int>(J::num(d, "index"));
|
||||
h.score = J::num(d, "relevance_score");
|
||||
out.hits.push_back(h);
|
||||
}
|
||||
std::stable_sort(out.hits.begin(), out.hits.end(), [](const RerankHit& a, const RerankHit& b) { return a.score > b.score; });
|
||||
if (const Json* u = J::at(doc, "usage")) out.total_tokens = static_cast<long>(J::num(*u, "total_tokens"));
|
||||
return out;
|
||||
}
|
||||
|
||||
VoyageContextualResult Voyage::parse_contextual(const Json& doc) {
|
||||
VoyageContextualResult out;
|
||||
out.model = J::str(doc, "model");
|
||||
for (const Json& d : J::array(doc, "data")) {
|
||||
std::vector<std::vector<float>> chunks;
|
||||
for (const Json& c : J::array(d, "data")) {
|
||||
const Json* e = J::at(c, "embedding");
|
||||
chunks.push_back(e ? J::floats(*e) : std::vector<float>{});
|
||||
}
|
||||
out.embeddings.push_back(std::move(chunks));
|
||||
}
|
||||
if (const Json* u = J::at(doc, "usage")) out.total_tokens = static_cast<long>(J::num(*u, "total_tokens"));
|
||||
return out;
|
||||
}
|
||||
|
||||
EmbedResult Voyage::embed(const EmbedRequest& request) {
|
||||
Embedder::validate("voyage", request);
|
||||
Json body = Json::object();
|
||||
body.set("input", J::strings(request.texts));
|
||||
body.set("model", Json(request.model.empty() ? std::string(kDefaultEmbedModel) : request.model));
|
||||
if (!request.input_type.empty()) body.set("input_type", Json(request.input_type));
|
||||
if (request.dimensions > 0) body.set("output_dimension", Json(request.dimensions));
|
||||
CallArgs args;
|
||||
args.body = body;
|
||||
EmbedResult r = parse_embeddings(gw_->call("embed", args));
|
||||
if (r.vectors.size() != request.texts.size())
|
||||
throw llm::ApiError("voyage", 502, "BAD_RESPONSE", "embedding count differs from input count");
|
||||
return r;
|
||||
}
|
||||
|
||||
RerankResult Voyage::rerank(const RerankRequest& request) {
|
||||
Reranker::validate("voyage", request);
|
||||
Json body = Json::object();
|
||||
body.set("query", Json(request.query));
|
||||
body.set("documents", J::strings(request.documents));
|
||||
body.set("model", Json(request.model.empty() ? std::string(kDefaultRerankModel) : request.model));
|
||||
if (request.top_n > 0) body.set("top_k", Json(request.top_n));
|
||||
CallArgs args;
|
||||
args.body = body;
|
||||
return parse_rerank(gw_->call("rerank", args));
|
||||
}
|
||||
|
||||
EmbedResult Voyage::multimodal_embed(const std::vector<std::vector<VoyageSegment>>& inputs,
|
||||
const std::string& model, const std::string& input_type) {
|
||||
if (inputs.empty() || inputs.size() > 1000) throw llm::ApiError::invalid("voyage", "inputs must hold 1..1000 entries");
|
||||
if (!input_type.empty() && input_type != "query" && input_type != "document")
|
||||
throw llm::ApiError::invalid("voyage", "input_type must be query or document");
|
||||
Json arr = Json::array();
|
||||
for (const auto& segments : inputs) {
|
||||
if (segments.empty() || segments.size() > 64) throw llm::ApiError::invalid("voyage", "an input needs 1..64 segments");
|
||||
Json content = Json::array();
|
||||
for (const VoyageSegment& s : segments) {
|
||||
if (s.value.empty() || s.value.size() > RetrievalCaps::kMaxBodyBytes)
|
||||
throw llm::ApiError::invalid("voyage", "a segment is empty or too large");
|
||||
Json seg = Json::object();
|
||||
switch (s.kind) {
|
||||
case VoyageSegment::Kind::kText:
|
||||
seg.set("type", Json("text"));
|
||||
seg.set("text", Json(s.value));
|
||||
break;
|
||||
case VoyageSegment::Kind::kImageUrl:
|
||||
seg.set("type", Json("image_url"));
|
||||
seg.set("image_url", Json(s.value));
|
||||
break;
|
||||
case VoyageSegment::Kind::kImageBase64:
|
||||
seg.set("type", Json("image_base64"));
|
||||
seg.set("image_base64", Json(s.value));
|
||||
break;
|
||||
case VoyageSegment::Kind::kVideoUrl:
|
||||
seg.set("type", Json("video_url"));
|
||||
seg.set("video_url", Json(s.value));
|
||||
break;
|
||||
case VoyageSegment::Kind::kVideoBase64:
|
||||
seg.set("type", Json("video_base64"));
|
||||
seg.set("video_base64", Json(s.value));
|
||||
break;
|
||||
}
|
||||
content.push(seg);
|
||||
}
|
||||
Json item = Json::object();
|
||||
item.set("content", content);
|
||||
arr.push(item);
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("inputs", arr);
|
||||
body.set("model", Json(model.empty() ? std::string(kDefaultMultimodalModel) : model));
|
||||
if (!input_type.empty()) body.set("input_type", Json(input_type));
|
||||
CallArgs args;
|
||||
args.body = body;
|
||||
return parse_embeddings(gw_->call("multimodal_embed", args));
|
||||
}
|
||||
|
||||
VoyageContextualResult Voyage::contextualized_embed(const std::vector<std::vector<std::string>>& documents,
|
||||
const std::string& model, const std::string& input_type,
|
||||
int dimensions) {
|
||||
if (documents.empty() || documents.size() > 1000) throw llm::ApiError::invalid("voyage", "documents must hold 1..1000 entries");
|
||||
if (!input_type.empty() && input_type != "query" && input_type != "document")
|
||||
throw llm::ApiError::invalid("voyage", "input_type must be query or document");
|
||||
if (dimensions < 0 || dimensions > RetrievalCaps::kMaxDimensions) throw llm::ApiError::invalid("voyage", "dimensions out of range");
|
||||
Json inputs = Json::array();
|
||||
for (const auto& chunks : documents) {
|
||||
if (chunks.empty() || chunks.size() > 16000) throw llm::ApiError::invalid("voyage", "a document needs 1..16000 chunks");
|
||||
inputs.push(J::strings(chunks));
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("inputs", inputs);
|
||||
body.set("model", Json(model.empty() ? std::string(kDefaultContextModel) : model));
|
||||
if (!input_type.empty()) body.set("input_type", Json(input_type));
|
||||
if (dimensions > 0) body.set("output_dimension", Json(dimensions));
|
||||
CallArgs args;
|
||||
args.body = body;
|
||||
return parse_contextual(gw_->call("contextualized_embed", args));
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
#pragma once
|
||||
|
||||
// Voyage AI: embeddings, multimodal embeddings, contextualized chunk
|
||||
// embeddings and rerank. Implements the Embedder and Reranker seams.
|
||||
// Source: the official voyageai Python SDK (Voyage publishes no OpenAPI
|
||||
// document), see SPEC.md.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/retrieval/provider.h"
|
||||
#include "vendors/retrieval/seams.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallTable& voyage_calls();
|
||||
|
||||
// One multimodal input is a list of segments (text, image URL, image base64,
|
||||
// video URL, video base64), embedded together as one vector.
|
||||
struct VoyageSegment {
|
||||
enum class Kind { kText, kImageUrl, kImageBase64, kVideoUrl, kVideoBase64 };
|
||||
Kind kind = Kind::kText;
|
||||
std::string value;
|
||||
};
|
||||
|
||||
struct VoyageContextualResult {
|
||||
// embeddings[document][chunk]
|
||||
std::vector<std::vector<std::vector<float>>> embeddings;
|
||||
std::string model;
|
||||
long total_tokens = 0;
|
||||
};
|
||||
|
||||
class Voyage : public Provider, public Embedder, public Reranker {
|
||||
public:
|
||||
static constexpr const char* kKeyEnv = "VOYAGE_API_KEY";
|
||||
static constexpr const char* kBaseUrl = "https://api.voyageai.com";
|
||||
static constexpr const char* kDefaultEmbedModel = "voyage-3.5";
|
||||
static constexpr const char* kDefaultRerankModel = "rerank-2.5";
|
||||
static constexpr const char* kDefaultMultimodalModel = "voyage-multimodal-3";
|
||||
static constexpr const char* kDefaultContextModel = "voyage-context-3";
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
static Voyage from_env();
|
||||
static Voyage with(const TransportFactory& factory, std::string key);
|
||||
explicit Voyage(std::shared_ptr<Gateway> gateway) : Provider(std::move(gateway)) {}
|
||||
|
||||
std::string provider() const override { return "voyage"; }
|
||||
EmbedResult embed(const EmbedRequest& request) override;
|
||||
RerankResult rerank(const RerankRequest& request) override;
|
||||
|
||||
EmbedResult multimodal_embed(const std::vector<std::vector<VoyageSegment>>& inputs,
|
||||
const std::string& model, const std::string& input_type);
|
||||
// Each document is the list of its chunks; the model embeds chunks in the
|
||||
// context of the whole document.
|
||||
VoyageContextualResult contextualized_embed(const std::vector<std::vector<std::string>>& documents,
|
||||
const std::string& model, const std::string& input_type,
|
||||
int dimensions);
|
||||
|
||||
// Pure parsers (public so tests can run them on documented shapes).
|
||||
static EmbedResult parse_embeddings(const Json& doc);
|
||||
static RerankResult parse_rerank(const Json& doc);
|
||||
static VoyageContextualResult parse_contextual(const Json& doc);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
#include "vendors/retrieval/weaviate/weaviate.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// name, host, http method, path, spec operation id (or reference page / "doc:"),
|
||||
// body fields, query parameters, flags (d destructive, w writes, r read-only POST,
|
||||
// p paid), response schema)
|
||||
static const CallRow kRows[] = {
|
||||
{"objects_list", "main", "GET", "/objects", "objects.list", "", "after,class,include,limit,offset,order,sort,tenant", "", "ObjectsListResponse"},
|
||||
{"objects_create", "main", "POST", "/objects", "objects.create", "additional,class,creationTimeUnix,id,lastUpdateTimeUnix,properties,tenant,vector,vectorWeights,vectors", "consistency_level", "w", "Object"},
|
||||
{"objects_delete", "main", "DELETE", "/objects/{id}", "objects.delete", "", "consistency_level,tenant", "d", "-"},
|
||||
{"objects_get", "main", "GET", "/objects/{id}", "objects.get", "", "include", "", "Object"},
|
||||
{"objects_patch", "main", "PATCH", "/objects/{id}", "objects.patch", "additional,class,creationTimeUnix,id,lastUpdateTimeUnix,properties,tenant,vector,vectorWeights,vectors", "consistency_level", "w", "-"},
|
||||
{"objects_update", "main", "PUT", "/objects/{id}", "objects.update", "additional,class,creationTimeUnix,id,lastUpdateTimeUnix,properties,tenant,vector,vectorWeights,vectors", "consistency_level", "w", "Object"},
|
||||
{"objects_class_get", "main", "GET", "/objects/{className}/{id}", "objects.class.get", "", "consistency_level,include,node_name,tenant", "", "Object"},
|
||||
{"objects_class_delete", "main", "DELETE", "/objects/{className}/{id}", "objects.class.delete", "", "consistency_level,tenant", "d", "-"},
|
||||
{"objects_class_put", "main", "PUT", "/objects/{className}/{id}", "objects.class.put", "additional,class,creationTimeUnix,id,lastUpdateTimeUnix,properties,tenant,vector,vectorWeights,vectors", "consistency_level", "w", "Object"},
|
||||
{"objects_class_patch", "main", "PATCH", "/objects/{className}/{id}", "objects.class.patch", "additional,class,creationTimeUnix,id,lastUpdateTimeUnix,properties,tenant,vector,vectorWeights,vectors", "consistency_level", "w", "-"},
|
||||
{"objects_references_create", "main", "POST", "/objects/{id}/references/{propertyName}", "objects.references.create", "beacon,class,classification,href,schema", "tenant", "w", "-"},
|
||||
{"objects_references_update", "main", "PUT", "/objects/{id}/references/{propertyName}", "objects.references.update", "*", "tenant", "w", "-"},
|
||||
{"objects_references_delete", "main", "DELETE", "/objects/{id}/references/{propertyName}", "objects.references.delete", "beacon,class,classification,href,schema", "tenant", "d", "-"},
|
||||
{"objects_class_references_create", "main", "POST", "/objects/{className}/{id}/references/{propertyName}", "objects.class.references.create", "beacon,class,classification,href,schema", "consistency_level,tenant", "w", "-"},
|
||||
{"objects_class_references_put", "main", "PUT", "/objects/{className}/{id}/references/{propertyName}", "objects.class.references.put", "*", "consistency_level,tenant", "w", "-"},
|
||||
{"objects_class_references_delete", "main", "DELETE", "/objects/{className}/{id}/references/{propertyName}", "objects.class.references.delete", "beacon,class,classification,href,schema", "consistency_level,tenant", "d", "-"},
|
||||
{"objects_validate", "main", "POST", "/objects/validate", "objects.validate", "additional,class,creationTimeUnix,id,lastUpdateTimeUnix,properties,tenant,vector,vectorWeights,vectors", "", "r", "-"},
|
||||
{"batch_objects_create", "main", "POST", "/batch/objects", "batch.objects.create", "fields,objects", "consistency_level", "w", "-"},
|
||||
{"batch_objects_delete", "main", "DELETE", "/batch/objects", "batch.objects.delete", "deletionTimeUnixMilli,dryRun,match,output", "consistency_level,tenant", "d", "BatchDeleteResponse"},
|
||||
{"batch_references_create", "main", "POST", "/batch/references", "batch.references.create", "*", "consistency_level", "w", "-"},
|
||||
{"graphql_post", "main", "POST", "/graphql", "graphql.post", "operationName,query,variables", "", "r", "GraphQLResponse"},
|
||||
{"graphql_batch", "main", "POST", "/graphql/batch", "graphql.batch", "*", "", "r", "GraphQLResponses"},
|
||||
{"tokenize", "main", "POST", "/tokenize", "tokenize", "analyzerConfig,stopwordPresets,stopwords,text,tokenization", "", "r", "TokenizeResponse"},
|
||||
{"schema_dump", "main", "GET", "/schema", "schema.dump", "", "", "", "Schema"},
|
||||
{"schema_objects_create", "main", "POST", "/schema", "schema.objects.create", "class,description,invertedIndexConfig,moduleConfig,multiTenancyConfig,objectTtlConfig,properties,replicationConfig,shardingConfig,vectorConfig,vectorIndexConfig,vectorIndexType,vectorizer", "", "w", "Class"},
|
||||
{"schema_objects_get", "main", "GET", "/schema/{className}", "schema.objects.get", "", "", "", "Class"},
|
||||
{"schema_objects_delete", "main", "DELETE", "/schema/{className}", "schema.objects.delete", "", "", "d", "-"},
|
||||
{"schema_objects_update", "main", "PUT", "/schema/{className}", "schema.objects.update", "class,description,invertedIndexConfig,moduleConfig,multiTenancyConfig,objectTtlConfig,properties,replicationConfig,shardingConfig,vectorConfig,vectorIndexConfig,vectorIndexType,vectorizer", "", "w", "Class"},
|
||||
{"schema_objects_properties_add", "main", "POST", "/schema/{className}/properties", "schema.objects.properties.add", "bucketGeneration,dataType,description,disableDuplicatedReferences,indexFilterable,indexInverted,indexRangeFilters,indexSearchable,moduleConfig,name,nestedProperties,searchableBlockmax,textAnalyzer,tokenization", "", "w", "Property"},
|
||||
{"schema_objects_indexes_get", "main", "GET", "/schema/{className}/indexes", "schema.objects.indexes.get", "", "", "", "IndexStatusResponse"},
|
||||
{"schema_objects_index_upsert", "main", "PUT", "/schema/{className}/properties/{propertyName}/index/{indexName}", "schema.objects.index.upsert", "algorithm,tokenization", "tenants", "w", "IndexUpdateResponse"},
|
||||
{"schema_objects_properties_delete", "main", "DELETE", "/schema/{className}/properties/{propertyName}/index/{indexName}", "schema.objects.properties.delete", "", "", "d", "-"},
|
||||
{"schema_objects_index_rebuild", "main", "POST", "/schema/{className}/properties/{propertyName}/index/{indexName}/rebuild", "schema.objects.index.rebuild", "", "tenants", "w", "IndexUpdateResponse"},
|
||||
{"schema_objects_index_cancel", "main", "POST", "/schema/{className}/properties/{propertyName}/index/{indexName}/cancel", "schema.objects.index.cancel", "", "", "d", "IndexUpdateResponse"},
|
||||
{"schema_objects_properties_tokenize", "main", "POST", "/schema/{className}/properties/{propertyName}/tokenize", "schema.objects.properties.tokenize", "text", "", "r", "TokenizeResponse"},
|
||||
{"schema_objects_vectors_delete", "main", "DELETE", "/schema/{className}/vectors/{vectorIndexName}/index", "schema.objects.vectors.delete", "", "", "d", "-"},
|
||||
{"schema_objects_shards_get", "main", "GET", "/schema/{className}/shards", "schema.objects.shards.get", "", "tenant", "", "ShardStatusList"},
|
||||
{"schema_objects_shards_update", "main", "PUT", "/schema/{className}/shards/{shardName}", "schema.objects.shards.update", "status", "", "w", "ShardStatus"},
|
||||
{"tenants_create", "main", "POST", "/schema/{className}/tenants", "tenants.create", "*", "", "w", "-"},
|
||||
{"tenants_update", "main", "PUT", "/schema/{className}/tenants", "tenants.update", "*", "", "w", "-"},
|
||||
{"tenants_delete", "main", "DELETE", "/schema/{className}/tenants", "tenants.delete", "*", "", "d", "-"},
|
||||
{"tenants_get", "main", "GET", "/schema/{className}/tenants", "tenants.get", "", "", "", "-"},
|
||||
{"tenants_get_one", "main", "GET", "/schema/{className}/tenants/{tenantName}", "tenants.get.one", "", "", "", "Tenant"},
|
||||
{"aliases_get", "main", "GET", "/aliases", "aliases.get", "", "class", "", "AliasResponse"},
|
||||
{"aliases_create", "main", "POST", "/aliases", "aliases.create", "alias,class", "", "w", "Alias"},
|
||||
{"aliases_get_alias", "main", "GET", "/aliases/{aliasName}", "aliases.get.alias", "", "", "", "Alias"},
|
||||
{"aliases_update", "main", "PUT", "/aliases/{aliasName}", "aliases.update", "class", "", "w", "Alias"},
|
||||
{"aliases_delete", "main", "DELETE", "/aliases/{aliasName}", "aliases.delete", "", "", "d", "-"},
|
||||
{"list_namespaces", "main", "GET", "/namespaces", "listNamespaces", "", "", "", "NamespaceListResponse"},
|
||||
{"create_namespace", "main", "POST", "/namespaces/{namespace_id}", "createNamespace", "home_node", "", "w", "Namespace"},
|
||||
{"get_namespace", "main", "GET", "/namespaces/{namespace_id}", "getNamespace", "", "", "", "Namespace"},
|
||||
{"update_namespace", "main", "PUT", "/namespaces/{namespace_id}", "updateNamespace", "home_node", "", "w", "Namespace"},
|
||||
{"delete_namespace", "main", "DELETE", "/namespaces/{namespace_id}", "deleteNamespace", "", "", "d", "-"},
|
||||
{"suspend_namespace", "main", "POST", "/namespaces/{namespace_id}/suspend", "suspendNamespace", "", "", "w", "-"},
|
||||
{"resume_namespace", "main", "POST", "/namespaces/{namespace_id}/resume", "resumeNamespace", "", "", "w", "-"},
|
||||
{"export_create", "main", "POST", "/export/{backend}", "export.create", "exclude,file_format,id,include", "", "w", "ExportCreateResponse"},
|
||||
{"export_status", "main", "GET", "/export/{backend}/{id}", "export.status", "", "", "", "ExportStatusResponse"},
|
||||
{"export_cancel", "main", "DELETE", "/export/{backend}/{id}", "export.cancel", "", "", "d", "-"},
|
||||
{"mcp_post", "main", "POST", "/mcp", "mcp.post", "", "", "r", "-"},
|
||||
{"mcp_delete", "main", "DELETE", "/mcp", "mcp.delete", "", "", "d", "-"},
|
||||
{"aggregate", "main", "POST", "/aggregate/{collection}", "aggregate", "groupBy,limit,objectLimit,over,returnMetrics,tenant,where", "", "r", "AggregateResponse"},
|
||||
{"search_bm25", "main", "POST", "/search/{collection}/bm25", "search.bm25", "autoLimit,consistencyLevel,groupBy,groupedTask,limit,numberOfGroups,objectsPerGroup,offset,query,queryProperties,rerank,returnMetadata,returnProperties,returnReferences,singlePrompt,tenant,where", "", "r", "SearchResponse"},
|
||||
{"search_hybrid", "main", "POST", "/search/{collection}/hybrid", "search.hybrid", "alpha,autoLimit,consistencyLevel,fusionType,groupBy,groupedTask,limit,maxVectorDistance,numberOfGroups,objectsPerGroup,offset,query,queryProperties,rerank,returnMetadata,returnProperties,returnReferences,singlePrompt,targetVector,tenant,where", "", "r", "SearchResponse"},
|
||||
{"search_near_object", "main", "POST", "/search/{collection}/near-object", "search.nearObject", "autoLimit,certainty,consistencyLevel,distance,groupBy,groupedTask,id,limit,numberOfGroups,objectsPerGroup,offset,rerank,returnMetadata,returnProperties,returnReferences,singlePrompt,targetVector,tenant,where", "", "r", "SearchResponse"},
|
||||
{"search_near_text", "main", "POST", "/search/{collection}/near-text", "search.nearText", "autoLimit,certainty,consistencyLevel,distance,groupBy,groupedTask,limit,numberOfGroups,objectsPerGroup,offset,query,rerank,returnMetadata,returnProperties,returnReferences,singlePrompt,targetVector,tenant,where", "", "r", "SearchResponse"},
|
||||
{"search_near_vector", "main", "POST", "/search/{collection}/near-vector", "search.nearVector", "autoLimit,certainty,consistencyLevel,distance,groupBy,groupedTask,limit,numberOfGroups,objectsPerGroup,offset,rerank,returnMetadata,returnProperties,returnReferences,singlePrompt,targetVector,tenant,vector,where", "", "r", "SearchResponse"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
const CallTable& weaviate_calls() {
|
||||
static const CallTable table{kRows, sizeof(kRows) / sizeof(kRows[0])};
|
||||
return table;
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+162
@@ -0,0 +1,162 @@
|
||||
#include "vendors/retrieval/weaviate/weaviate.h"
|
||||
|
||||
#include "llm/config.h"
|
||||
#include "vendors/retrieval/caps.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
llm::ApiClientConfig Weaviate::config() {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "weaviate";
|
||||
c.base_url = "";
|
||||
c.key_env = kKeyEnv;
|
||||
c.auth = llm::AuthStyle::kBearer;
|
||||
return c;
|
||||
}
|
||||
|
||||
namespace {
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("weaviate", why); }
|
||||
|
||||
Weaviate build(std::string url, std::string key, const TransportFactory& factory) {
|
||||
while (!url.empty() && url.back() == '/') url.pop_back();
|
||||
bool ok = url.rfind("https://", 0) == 0 || url.rfind("http://", 0) == 0;
|
||||
auto gw = make_gateway(weaviate_calls(), Weaviate::config(), {{"main", ok ? url + "/v1" : "https://weaviate.invalid/v1"}},
|
||||
std::move(key), factory);
|
||||
if (!ok) gw->require_setting(Weaviate::kUrlEnv);
|
||||
return Weaviate(gw);
|
||||
}
|
||||
|
||||
CallArgs with_body(Json body) {
|
||||
CallArgs a;
|
||||
a.body = std::move(body);
|
||||
return a;
|
||||
}
|
||||
CallArgs cls(const std::string& name) {
|
||||
CallArgs a;
|
||||
a.path["className"] = Weaviate::validate_class_name(name);
|
||||
return a;
|
||||
}
|
||||
Json search_body(const std::string& query, int limit) {
|
||||
if (query.empty() || query.size() > RetrievalCaps::kMaxQueryChars) bad("query must be 1..8192 chars");
|
||||
if (limit < 1 || limit > RetrievalCaps::kMaxPointLimit) bad("limit must be 1..10000");
|
||||
Json b = Json::object();
|
||||
b.set("query", Json(query));
|
||||
b.set("limit", Json(limit));
|
||||
return b;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
Weaviate Weaviate::from_env() { return build(llm::Config::get(kUrlEnv), "", nullptr); }
|
||||
Weaviate Weaviate::with(const TransportFactory& factory, std::string key, const std::string& base_url) {
|
||||
return build(base_url, std::move(key), factory);
|
||||
}
|
||||
|
||||
std::string Weaviate::validate_class_name(const std::string& name) {
|
||||
// Weaviate class names start with an uppercase letter: /[A-Z][_0-9A-Za-z]*/
|
||||
bool ok = !name.empty() && name.size() <= 255 && name[0] >= 'A' && name[0] <= 'Z';
|
||||
for (char c : name) ok = ok && ((c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '_');
|
||||
if (!ok) bad("class name must match [A-Z][_0-9A-Za-z]* (max 255)");
|
||||
return name;
|
||||
}
|
||||
|
||||
Json Weaviate::schema() { return gw_->call("schema_dump"); }
|
||||
Json Weaviate::get_class(const std::string& n) { return gw_->call("schema_objects_get", cls(n)); }
|
||||
|
||||
Json Weaviate::create_class(const Json& def) {
|
||||
if (!def.is_object()) bad("class definition must be an object");
|
||||
const Json* c = def.find("class");
|
||||
if (!c || !c->is_string()) bad("class definition needs a class name");
|
||||
validate_class_name(c->as_string());
|
||||
return gw_->call("schema_objects_create", with_body(def));
|
||||
}
|
||||
|
||||
Json Weaviate::delete_class(const std::string& n) { return gw_->destroy("schema_objects_delete", cls(n)); }
|
||||
|
||||
Json Weaviate::create_object(const std::string& class_name, const Json& properties, const std::vector<float>& vector) {
|
||||
if (!properties.is_object()) bad("properties must be an object");
|
||||
Json body = Json::object();
|
||||
body.set("class", Json(validate_class_name(class_name)));
|
||||
body.set("properties", properties);
|
||||
if (!vector.empty()) {
|
||||
if (vector.size() > static_cast<std::size_t>(RetrievalCaps::kMaxDimensions)) bad("vector too long");
|
||||
Json v = Json::array();
|
||||
for (float f : vector) v.push(Json(static_cast<double>(f)));
|
||||
body.set("vector", v);
|
||||
}
|
||||
return gw_->call("objects_create", with_body(body));
|
||||
}
|
||||
|
||||
Json Weaviate::get_object(const std::string& class_name, const std::string& id) {
|
||||
CallArgs a = cls(class_name);
|
||||
a.path["id"] = id;
|
||||
return gw_->call("objects_class_get", a);
|
||||
}
|
||||
|
||||
Json Weaviate::delete_object(const std::string& class_name, const std::string& id) {
|
||||
CallArgs a = cls(class_name);
|
||||
a.path["id"] = id;
|
||||
return gw_->destroy("objects_class_delete", a);
|
||||
}
|
||||
|
||||
Json Weaviate::batch_create(const std::vector<Json>& objects) {
|
||||
if (objects.empty() || objects.size() > RetrievalCaps::kMaxPointsPerUpsert) bad("objects must hold 1..1000 entries");
|
||||
Json arr = Json::array();
|
||||
for (const Json& o : objects) {
|
||||
if (!o.is_object()) bad("an object must be a JSON object");
|
||||
arr.push(o);
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("objects", arr);
|
||||
return gw_->call("batch_objects_create", with_body(body));
|
||||
}
|
||||
|
||||
Json Weaviate::batch_delete(const Json& match, bool dry_run) {
|
||||
if (!match.is_object() || !match.has("class")) bad("match needs a class");
|
||||
Json body = Json::object();
|
||||
body.set("match", match);
|
||||
body.set("dryRun", Json(dry_run));
|
||||
body.set("output", Json("verbose"));
|
||||
return gw_->destroy("batch_objects_delete", with_body(body));
|
||||
}
|
||||
|
||||
Json Weaviate::near_text(const std::string& collection, const std::string& query, int limit) {
|
||||
CallArgs a = with_body(search_body(query, limit));
|
||||
a.path["collection"] = collection;
|
||||
return gw_->call("search_near_text", a);
|
||||
}
|
||||
|
||||
Json Weaviate::bm25(const std::string& collection, const std::string& query, int limit) {
|
||||
CallArgs a = with_body(search_body(query, limit));
|
||||
a.path["collection"] = collection;
|
||||
return gw_->call("search_bm25", a);
|
||||
}
|
||||
|
||||
Json Weaviate::hybrid(const std::string& collection, const std::string& query, int limit, double alpha) {
|
||||
if (alpha < 0 || alpha > 1) bad("alpha must be 0..1");
|
||||
Json b = search_body(query, limit);
|
||||
b.set("alpha", Json(alpha));
|
||||
CallArgs a = with_body(b);
|
||||
a.path["collection"] = collection;
|
||||
return gw_->call("search_hybrid", a);
|
||||
}
|
||||
|
||||
Json Weaviate::graphql(const std::string& query, const Json& variables) {
|
||||
if (query.empty() || query.size() > RetrievalCaps::kMaxQueryChars * 8) bad("query must be 1..65536 chars");
|
||||
Json b = Json::object();
|
||||
b.set("query", Json(query));
|
||||
if (!variables.is_null()) b.set("variables", variables);
|
||||
return gw_->call("graphql_post", with_body(b));
|
||||
}
|
||||
|
||||
Json Weaviate::tenants(const std::string& class_name) { return gw_->call("tenants_get", cls(class_name)); }
|
||||
|
||||
Json Weaviate::delete_tenants(const std::string& class_name, const std::vector<std::string>& names) {
|
||||
if (names.empty() || names.size() > 100) bad("names must hold 1..100 entries");
|
||||
return gw_->destroy("tenants_delete", [&] {
|
||||
CallArgs a = cls(class_name);
|
||||
a.body = J::strings(names);
|
||||
return a;
|
||||
}());
|
||||
}
|
||||
|
||||
} // namespace molo::retrieval
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
#pragma once
|
||||
|
||||
// Weaviate REST (/v1): schema, objects, batch, tenants, aliases, the REST
|
||||
// search endpoints (near-text, near-vector style) and GraphQL over HTTP.
|
||||
// Weaviate's gRPC API is not HTTP and has no gateway: it is out of scope (see
|
||||
// SPEC.md). The server is yours (WEAVIATE_URL).
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/retrieval/provider.h"
|
||||
|
||||
namespace molo::retrieval {
|
||||
|
||||
const CallTable& weaviate_calls();
|
||||
|
||||
class Weaviate : public Provider {
|
||||
public:
|
||||
static constexpr const char* kKeyEnv = "WEAVIATE_API_KEY";
|
||||
static constexpr const char* kUrlEnv = "WEAVIATE_URL";
|
||||
|
||||
static llm::ApiClientConfig config();
|
||||
static Weaviate from_env();
|
||||
static Weaviate with(const TransportFactory& factory, std::string key, const std::string& base_url);
|
||||
explicit Weaviate(std::shared_ptr<Gateway> gateway) : Provider(std::move(gateway)) {}
|
||||
|
||||
// Schema
|
||||
Json schema();
|
||||
Json get_class(const std::string& class_name);
|
||||
Json create_class(const Json& class_definition);
|
||||
Json delete_class(const std::string& class_name); // destructive: drops the class and its objects
|
||||
|
||||
// Objects
|
||||
Json create_object(const std::string& class_name, const Json& properties, const std::vector<float>& vector);
|
||||
Json get_object(const std::string& class_name, const std::string& id);
|
||||
Json delete_object(const std::string& class_name, const std::string& id); // destructive
|
||||
Json batch_create(const std::vector<Json>& objects);
|
||||
// Destructive. `match` is {"class":..., "where":{...}}; dry_run reports without deleting.
|
||||
Json batch_delete(const Json& match, bool dry_run);
|
||||
|
||||
// Search
|
||||
Json near_text(const std::string& collection, const std::string& query, int limit);
|
||||
Json bm25(const std::string& collection, const std::string& query, int limit);
|
||||
Json hybrid(const std::string& collection, const std::string& query, int limit, double alpha);
|
||||
Json graphql(const std::string& query, const Json& variables);
|
||||
|
||||
// Multi-tenancy
|
||||
Json tenants(const std::string& class_name);
|
||||
Json delete_tenants(const std::string& class_name, const std::vector<std::string>& names); // destructive
|
||||
|
||||
static std::string validate_class_name(const std::string& name);
|
||||
};
|
||||
|
||||
} // namespace molo::retrieval
|
||||
Reference in New Issue
Block a user