feat(clients): huggingface: Hub, Inference Providers, Endpoints, TGI, TEI, chat model, demo
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Vendored
+234
@@ -0,0 +1,234 @@
|
||||
# Hugging Face - Specification
|
||||
|
||||
Owned directory: `src/vendors/huggingface/`. Namespace `molo::huggingface`.
|
||||
Depends on `molo::llm` (`ApiClient`, `Service`, `SseDecoder`, `ChatModel`,
|
||||
`Config`) and `molo::clients`. Built by `make ai` (the vendor glob in
|
||||
`src/ai.mk` picks the directory up). Demo: `example/app/ai/HuggingfaceDemo.cpp`,
|
||||
routes under `/ai/huggingface`. Plans: `plans/17-llm-vendor-platforms.md`,
|
||||
`plans/18-breadth.md`.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
Hugging Face is five HTTP surfaces on four hosts. All are covered:
|
||||
|
||||
| Surface | Host | Class | What |
|
||||
|---|---|---|---|
|
||||
| Hub | `https://huggingface.co` | `Hub`, `Discussions`, `Collections`, `Webhooks` | models, datasets, spaces (list, info, revision), repo tree / refs / commits / compare / paths-info / likers / safetensors metadata, file download (`/resolve`), search, trending, tags, tasks, papers, Spaces catalogs, Inference Provider mappings (`/api/partners/{provider}/models`), whoami, discussions, collections, webhooks |
|
||||
| Router (Inference Providers) | `https://router.huggingface.co` | `RouterChat`, `InferenceTasks` | OpenAI-compatible chat (`/v1/chat/completions`, routing suffix `:fastest` `:cheapest` `:preferred` `:<provider>`), `/v1/models`, hf-inference task APIs (25 tasks, the classic Inference API), hf-inference chat |
|
||||
| Inference Endpoints | `https://api.endpoints.huggingface.cloud` | `Endpoints` | the whole management API v2 / v3: endpoints (list, get, create, update, pause, resume, scale to zero, delete), logs, 15 metric series, replicas, vendors and quotas, search, CIDR lists, custom domains, private services, audit logs |
|
||||
| TGI server | any URL | `TgiServer` | native (`/generate`, `/generate_stream`, `/tokenize`, `/info`, `/health`, `/metrics`) and OpenAI-compatible (`/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/invocations`) |
|
||||
| TEI server | any URL | `TeiServer` | `/embed`, `/embed_all`, `/embed_sparse`, `/predict`, `/rerank`, `/similarity`, `/tokenize`, `/decode`, `/v1/embeddings`, `/info`, `/health`, `/metrics` |
|
||||
|
||||
Plus `HuggingfaceChatModel : llm::ChatModel` over the router (default), a TGI
|
||||
server, or any `ChatEndpoint`.
|
||||
|
||||
Not covered (explicit, see `tools/check_spec.py` EXCLUDED and section 8):
|
||||
repository writes (commit, upload, LFS, branches, tags, settings), organizations
|
||||
and users, billing, Jobs, buckets / containers / kernels, gated-access requests,
|
||||
SCIM, OAuth, `datasets-server.huggingface.co`, the `huggingface.co/api/blog`
|
||||
and posts comment APIs, the provider-specific router paths.
|
||||
|
||||
## 2. Module map (one class per concern)
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | `HfCaps`: static ceilings |
|
||||
| `calls.h/.cpp` | `Call` enum and `kCalls`: the ONE table of 132 HTTP calls (surface, method, path, spec kind, reference, auth, flags, body fields, query parameters) |
|
||||
| `request_plan.h/.cpp` | `RequestPlan`, `PathArgs`, `Query`, `RepoRef`, `RepoType`: renders and validates one call from the table |
|
||||
| `platform.h/.cpp` | `Platform`, `Access`, `RequestBudget`: hosts, `HF_TOKEN`, anonymous clients, shared call budget, server URL check |
|
||||
| `hf_service.h/.cpp` | `HfService : llm::Service`: surface check, token check, budget, JSON / raw / bytes / paged / SSE calls |
|
||||
| `hub.h/.cpp` | `Hub` |
|
||||
| `community.h/.cpp` | `Discussions`, `Collections`, `Webhooks` (+ `CollectionItem`, `WebhookSpec`) |
|
||||
| `providers.h/.cpp` | `RouterChat`, `RouterModel`, `TaskCatalog`, `InferenceTasks` |
|
||||
| `endpoints.h/.cpp` | `Endpoints`, `MetricKind` |
|
||||
| `servers.h/.cpp` | `TgiServer`, `TeiServer` |
|
||||
| `chat_wire.h/.cpp` | `ChatWire`, `ChatEndpoint`: neutral chat <-> OpenAI-compatible wire |
|
||||
| `chat_model.h/.cpp` | `HuggingfaceChatModel` |
|
||||
| `tools/check_spec.py` | table vs published documents |
|
||||
|
||||
## 3. Static caps (caps.h)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Path segment / repo-relative file path / segments | 256 / 1024 chars / 32 |
|
||||
| Query parameters / one query value | 32 / 2048 chars |
|
||||
| Model id / server URL | 256 / 2048 chars |
|
||||
| Local call budget | 1000 calls per 300 s per process (observed server limit: 500 per 300 s, `RateLimit-Policy: "fixed window";"api";q=500;w=300`) |
|
||||
| Page walk | 20 pages, 5000 items, page `limit` 1000 |
|
||||
| `paths-info` paths | 256 |
|
||||
| Download | 64 MiB (the `llm` response cap; larger fails loudly) |
|
||||
| Inference input (binary) | 16 MiB |
|
||||
| Batch inputs (TEI, embeddings) | 256 entries, 4 MiB text |
|
||||
| Stream events | 200000 |
|
||||
| Messages / tools / stop sequences | 4096 / 128 / 4 |
|
||||
| Webhook domains | 8 |
|
||||
|
||||
## 4. Contracts
|
||||
|
||||
- **Token.** `HF_TOKEN` (or the legacy `HUGGING_FACE_HUB_TOKEN`) via
|
||||
`llm::Config`. It travels as `Authorization: Bearer` only, never in a URL or an
|
||||
error. A row with `auth: token` and no token throws `NotConfiguredError`
|
||||
(503 `NOT_CONFIGURED`) before any traffic. Rows with `auth: none` or
|
||||
`optional` (public Hub reads, `GET /v1/models`, `GET /v2/provider`, TGI / TEI)
|
||||
run without a token.
|
||||
- **Anonymous requests.** `llm::ApiClient` cannot send an unauthenticated
|
||||
request, so without a token `Platform` builds the client with a marker header
|
||||
`X-Molo-Anonymous: 1` instead of `Authorization`. Foundation gap, worked
|
||||
around inside the package.
|
||||
- **One table.** Every path, query parameter and body field comes from `kCalls`.
|
||||
`RequestPlan::make` refuses a query parameter or body field the row does not
|
||||
list, an unused or missing path argument, a body on a bodiless row, and rows
|
||||
flagged `unsupported`.
|
||||
- **Ids.** `namespace`, `repo`, `name`: `[A-Za-z0-9._-]`, 1..256, not `.` / `..`.
|
||||
Repo ids must be `owner/name`; legacy ids without an owner (`gpt2`) are refused
|
||||
because the Hub answers them with a redirect to the canonical id. A revision or
|
||||
tag is one percent-encoded segment (`refs/pr/1` -> `refs%2Fpr%2F1`). File
|
||||
paths are validated per segment and percent-encoded. Query values are
|
||||
percent-encoded; a value containing `..` has its dots encoded because
|
||||
`ApiClient` refuses `..` anywhere in the request path.
|
||||
- **Lists.** Hub list calls follow `Link: <...>; rel="next"` cursors, accept only
|
||||
the `cursor` of a Link to the same host, and return `Page` with `truncated`,
|
||||
`truncated_by` (`items` or `pages`) and `next_cursor`. Never silent.
|
||||
- **Writes and deletes.** Rows flagged `write` need a token. Deletes are
|
||||
`remove_*` methods with no defaults, flagged `destructive` in the table
|
||||
(tested: 7 DELETE rows, all flagged). The demo runs every paid or changing
|
||||
action as a dry run unless the body has `"confirm":true`.
|
||||
- **Chat wire.** `ChatWire` documents its mapping in `chat_wire.h`: tool
|
||||
arguments that are not JSON arrive as `{"_invalid_json": raw}`; reasoning is
|
||||
read from `reasoning_content` then `reasoning`; `response_schema` maps to a
|
||||
strict `json_schema`; usage is requested in streams.
|
||||
- **Errors.** Everything is an `ApiError`; the Hub error body
|
||||
`{"error":"..."}` and the Endpoints body
|
||||
`{"error":"401 Unauthorized","code":"UNAUTHORIZED"}` keep their message and
|
||||
status. A local budget stop is `429 LOCAL_BUDGET`.
|
||||
- **Tests.** `MOLO_ENV_FILES=off`; `MOLO_LIVE_NET=1` enables the live tests;
|
||||
`MOLO_LIVE_PAID=1` additionally allows the one chat completion.
|
||||
|
||||
## 5. Credentials
|
||||
|
||||
| Env var | Page |
|
||||
|---|---|
|
||||
| `HF_TOKEN` | https://huggingface.co/settings/tokens (docs: https://huggingface.co/docs/hub/security-tokens). A fine-grained token needs "Make calls to Inference Providers" for the router, "Make calls to Inference Endpoints" and "Manage Inference Endpoints" for the endpoints API, and write scopes for discussions / collections / webhooks. Inference Providers bills the token owner (or the organization named by `RouterChat::set_bill_to`) |
|
||||
|
||||
No key exists in the development environment: see sections 7 and 8.
|
||||
|
||||
## 6. Sources
|
||||
|
||||
Machine-readable documents, fetched 2026-10-25:
|
||||
|
||||
- Hub OpenAPI 3.1 `https://huggingface.co/.well-known/openapi.json` (374
|
||||
operations, 307 paths; also `/.well-known/openapi.md`). It omits the
|
||||
list / info routes of models, datasets and spaces (`GET /api/models`,
|
||||
`/api/models/{ns}/{repo}`, ...), `/api/tasks` and `/api/spaces/{ns}/{repo}/runtime`;
|
||||
those 11 Hub rows cite `huggingface_hub` `hf_api.py` (spec kind `src`); the
|
||||
three hf-inference router rows cite `hf_inference.py` (also `src`).
|
||||
- Inference Endpoints OpenAPI 2.0.0 `https://api.endpoints.huggingface.cloud/openapi.json`
|
||||
(48 operations). The old `/api-doc/openapi.json` answers 404; the swagger page
|
||||
at `/` points at `/openapi.json`.
|
||||
- TGI `docs/openapi.json` (3.3.6-dev0, 12 operations) and TEI `docs/openapi.json`
|
||||
(1.9.4, 12 operations) from the `huggingface/text-generation-inference` and
|
||||
`huggingface/text-embeddings-inference` repositories, branch `main`.
|
||||
- `huggingface_hub` `src/huggingface_hub/hf_api.py`, `constants.py`,
|
||||
`inference/_providers/__init__.py` and `hf_inference.py`: endpoints and
|
||||
parameters of the routes the OpenAPI omits, the router URL template
|
||||
`https://router.huggingface.co/{provider}`, the 25-task hf-inference map, the
|
||||
feature-extraction / sentence-similarity `/pipeline/{task}` path, binary-input
|
||||
tasks, base64 form with parameters.
|
||||
- `huggingface.js` `packages/tasks/src/tasks/<task>/spec/input.json` for the
|
||||
chat-completion body fields (26 tasks fetched; `audio-to-audio` and
|
||||
`tabular-classification` have no schema file).
|
||||
- Unauthenticated probes, section 8.
|
||||
|
||||
## 7. Verified against spec
|
||||
|
||||
`python3 -I src/vendors/huggingface/tools/check_spec.py --probe`, 2026-10-25:
|
||||
|
||||
```
|
||||
132 rows checked; spec operations hub=374 ie=48 tgi=12 tei=12; 0 problems
|
||||
```
|
||||
|
||||
For `ie`, `tgi`, `tei`: every document operation, query parameter and body
|
||||
field is in the table (equality). For `hub`: every row's method, path (expanded
|
||||
over models / datasets / spaces where the document has one path per type),
|
||||
query parameters and body fields exist in the document; the other 314
|
||||
operations are excluded by explicit rules in `EXCLUDED`, counted per reason in
|
||||
the output (jobs 23, buckets / containers / kernels 33, organizations / users /
|
||||
SCIM 77, ...). The chat row is checked against the huggingface.js
|
||||
chat-completion schema (equality of fields). `--probe` GETs the `src` and
|
||||
`none` rows without a credential: all answered 200 (public) or 401 (the Hub's
|
||||
answer for an unknown or private repo id), none 404.
|
||||
|
||||
Local tests: `make ai-test` runs 66 `huggingface_*` tests (table integrity, id
|
||||
and path rules, query and body checks, caps, Link walks, budget window, token
|
||||
header, anonymous header, error mapping, chat wire, SSE transcript, every
|
||||
surface's request shapes). Replies in tests are `RECORDED` (copied from a probe,
|
||||
trimmed) or `SHAPE` (written to the documented schema). Nothing is a scripted
|
||||
claim about vendor behavior. Example route tests: `tests/example/ai/huggingface/`.
|
||||
|
||||
## 8. Verified live
|
||||
|
||||
Run on 2026-10-25 with no credential unless stated. All through the real HTTP
|
||||
transport (`MOLO_LIVE_NET=1 make ai-test`: 4 live tests, 3 run without a key).
|
||||
|
||||
| Request | Result |
|
||||
|---|---|
|
||||
| `GET huggingface.co/api/models?limit=2`, `/api/datasets?limit=2`, `/api/spaces?limit=2` | 200 JSON array; `Link: <...&cursor=...>; rel="next"`; `RateLimit-Policy: "fixed window";"api";q=500;w=300` |
|
||||
| `GET /api/models/gpt2`, `/api/datasets/squad`, `/api/models/gpt2/refs` | 307 to the canonical `openai-community/gpt2` / `rajpurkar/squad` (the reason legacy ids are refused) |
|
||||
| `GET /api/models/openai-community/gpt2`, `/refs`, `/tree/main`, `/discussions`; `POST /paths-info/main`; `GET /openai-community/gpt2/resolve/main/config.json` (665 bytes, `X-Repo-Commit` header) | 200 |
|
||||
| `GET /api/spaces/gradio/hello_world`, `/api/spaces/hardware`, `/api/trending`, `/api/models-tags-by-type`, `/api/tasks`, `/api/daily_papers`, `/api/quicksearch`, `/api/collections?limit=2`, `/api/collections/nvidia/flora-6ad6a8986320a55467825cf5`, `/api/partners/groq/models` | 200 |
|
||||
| `GET /api/whoami-v2`, `/api/settings/webhooks` without a token | 401 `{"error":"Invalid username or password."}` |
|
||||
| `GET router.huggingface.co/v1/models` | 200, 134 models, per model a `providers` array (status, context_length, pricing, supports_tools, latency) |
|
||||
| `POST router.huggingface.co/v1/chat/completions`, `/v1/embeddings`, `/hf-inference/models/gpt2`, any other path | 401 `text/html` without a token, including a made-up path. The router answers 401 before routing, so these probes do NOT prove a path exists |
|
||||
| `GET router.huggingface.co/hf-inference/models/gpt2` | 401 JSON `{"error":"Invalid username or password."}` |
|
||||
| `GET api.endpoints.huggingface.cloud/v2/provider` | 200 (public vendor / region / compute / price list) |
|
||||
| `GET .../v2/endpoint/huggingface`, `/v2/provider/huggingface`, `/v2/search/huggingface`, `/v2/endpoint/huggingface/x/logs` | 401 `{"error":"401 Unauthorized","code":"UNAUTHORIZED"}` |
|
||||
| Invalid token `hf_invalid_molo_probe` on `whoami` and `endpoints list` | 401 with the two envelopes above (live test) |
|
||||
| `api-inference.huggingface.co` | host does not resolve (the classic host is gone; hf-inference lives behind the router) |
|
||||
| `datasets-server.huggingface.co/is-valid?dataset=squad` | 404 `{"error":"The dataset has been renamed..."}` (out of scope, noted) |
|
||||
|
||||
## 9. Unverified
|
||||
|
||||
No token was available, so no authenticated success body was seen:
|
||||
|
||||
- every `auth: token` call: whoami, space runtime, discussions create / comment /
|
||||
status / title / delete, collections create / update / add item / delete,
|
||||
webhooks (all), Endpoints (all except `GET /v2/provider`);
|
||||
- the router chat completion, its routing suffixes, the `X-HF-Bill-To` header
|
||||
(taken from huggingface_hub / docs, unprobed), streaming usage in the last
|
||||
chunk, `reasoning` vs `reasoning_content` per provider;
|
||||
- every hf-inference task call and reply shape: the `/hf-inference/models/{model}`
|
||||
and `/pipeline/{task}` paths and the binary / base64 input forms come from
|
||||
`huggingface_hub` source, not from a response;
|
||||
- the response shapes our code reads (`choices`, `usage`, tool call fields) are
|
||||
the OpenAI-compatible shapes of the chat-completion spec, not seen from the
|
||||
router;
|
||||
- TGI and TEI: no server was reachable; request shapes follow their OpenAPI
|
||||
documents; `ready()` treats 503 as "starting" per TGI behavior and the
|
||||
documents, unobserved;
|
||||
- `Endpoints::validate_spec` checks only the required top-level keys, the name
|
||||
rule from the schema description and the `type` enum;
|
||||
- ZeroGPU / pro quotas and per-token rate limits other than the observed
|
||||
anonymous `q=500;w=300`;
|
||||
- text ranges in `/resolve` (no range support), xet / LFS downloads of files
|
||||
over 64 MiB.
|
||||
|
||||
## 10. Gaps
|
||||
|
||||
- `llm::ApiClient` cannot send an unauthenticated request (workaround: marker
|
||||
header), has no GET stream (the three Endpoints SSE routes `GET .../sse`,
|
||||
`.../logs/sse` and `/v3/.../logs/sse` are in the table with flag
|
||||
`unsupported` and refused), no per-call timeout, and drops response headers on
|
||||
`request()` (paging uses `request_raw`).
|
||||
- `llm::ApiClient::build` refuses `..` in any request path including the query;
|
||||
query values with `..` are dot-encoded as a workaround.
|
||||
- Provider-specific router paths (`router.huggingface.co/{provider}/...`, for
|
||||
example text-to-image on fal-ai or Replicate) have no per-provider schema
|
||||
published; only `hf-inference` tasks and the generic chat route are wrapped.
|
||||
- Collection item update / delete (`.../items/{itemId}`) are not wrapped: the
|
||||
document names the path with the `{slug}-{id}` split.
|
||||
- Repo writes (commit, upload, branches, tags, LFS) are out of scope; `Hub`
|
||||
reads and downloads only. Legacy ids without owner are refused.
|
||||
- `kernels` repo type exists on the Hub and in the discussions document; not
|
||||
wrapped.
|
||||
- Services `Discussions`, `Collections`, `Webhooks` share `Surface::kHub`.
|
||||
- Google services used: none.
|
||||
Vendored
+197
@@ -0,0 +1,197 @@
|
||||
#include "vendors/huggingface/calls.h"
|
||||
|
||||
#include <cstring>
|
||||
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
const char* surface_name(Surface surface) {
|
||||
switch (surface) {
|
||||
case Surface::kHub: return "hub";
|
||||
case Surface::kRouter: return "router";
|
||||
case Surface::kEndpoints: return "endpoints";
|
||||
case Surface::kTgi: return "tgi";
|
||||
case Surface::kTei: return "tei";
|
||||
}
|
||||
return "?";
|
||||
}
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// http method, path, spec, ref, auth, flags, body fields, query params)
|
||||
static const CallSpec kCalls[] = {
|
||||
{Call::kWhoami, Surface::kHub, "GET", "/api/whoami-v2", "hub", "-", "token", "", "", ""},
|
||||
{Call::kModelsTagsByType, Surface::kHub, "GET", "/api/models-tags-by-type", "hub", "-", "optional", "", "", "type"},
|
||||
{Call::kDatasetsTagsByType, Surface::kHub, "GET", "/api/datasets-tags-by-type", "hub", "-", "optional", "", "", "type"},
|
||||
{Call::kTrending, Surface::kHub, "GET", "/api/trending", "hub", "-", "optional", "", "", "limit,type"},
|
||||
{Call::kQuicksearch, Surface::kHub, "GET", "/api/quicksearch", "hub", "-", "optional", "", "", "discussionCollectionName,discussionId,exclude,includeInvitees,lang,library,limit,namespace,orgsFilter,pipelines,q,repoName,repoType,reposFilter,spacesTags,type"},
|
||||
{Call::kFullTextSearch, Surface::kHub, "GET", "/api/search/full-text", "hub", "-", "optional", "", "", "filter,limit,q,skip,type"},
|
||||
{Call::kRepoLikers, Surface::kHub, "GET", "/api/{repoType}/{namespace}/{repo}/likers", "hub", "-", "optional", "", "", "expand"},
|
||||
{Call::kRepoTree, Surface::kHub, "GET", "/api/{repoType}/{namespace}/{repo}/tree/{rev}/{path}", "hub", "-", "optional", "optpath", "", "cursor,expand,limit,recursive"},
|
||||
{Call::kRepoTreeSize, Surface::kHub, "GET", "/api/{repoType}/{namespace}/{repo}/treesize/{rev}/{path}", "hub", "-", "optional", "optpath", "", ""},
|
||||
{Call::kRepoRefs, Surface::kHub, "GET", "/api/{repoType}/{namespace}/{repo}/refs", "hub", "-", "optional", "", "", "include_prs"},
|
||||
{Call::kRepoCommits, Surface::kHub, "GET", "/api/{repoType}/{namespace}/{repo}/commits/{rev}", "hub", "-", "optional", "", "", "expand,limit,p"},
|
||||
{Call::kRepoCompare, Surface::kHub, "GET", "/api/{repoType}/{namespace}/{repo}/compare/{compare}", "hub", "-", "optional", "", "", "raw"},
|
||||
{Call::kRepoPathsInfo, Surface::kHub, "POST", "/api/{repoType}/{namespace}/{repo}/paths-info/{rev}", "hub", "-", "optional", "", "expand,paths", ""},
|
||||
{Call::kModelSafetensors, Surface::kHub, "GET", "/api/models/{namespace}/{repo}/safetensors-metadata/{path}", "hub", "-", "optional", "", "", ""},
|
||||
{Call::kFileResolve, Surface::kHub, "GET", "{typePrefix}/{namespace}/{repo}/resolve/{rev}/{path}", "hub", "-", "optional", "raw", "", ""},
|
||||
{Call::kDiscussionList, Surface::kHub, "GET", "/api/{repoType}/{namespace}/{repo}/discussions", "hub", "-", "optional", "", "", "author,p,search,sort,status,type"},
|
||||
{Call::kDiscussionCreate, Surface::kHub, "POST", "/api/{repoType}/{namespace}/{repo}/discussions", "hub", "-", "token", "write", "description,pullRequest,title", ""},
|
||||
{Call::kDiscussionGet, Surface::kHub, "GET", "/api/{repoType}/{namespace}/{repo}/discussions/{num}", "hub", "-", "optional", "", "", ""},
|
||||
{Call::kDiscussionDelete, Surface::kHub, "DELETE", "/api/{repoType}/{namespace}/{repo}/discussions/{num}", "hub", "-", "token", "write,destructive", "", ""},
|
||||
{Call::kDiscussionComment, Surface::kHub, "POST", "/api/{repoType}/{namespace}/{repo}/discussions/{num}/comment", "hub", "-", "token", "write", "comment", ""},
|
||||
{Call::kDiscussionStatus, Surface::kHub, "POST", "/api/{repoType}/{namespace}/{repo}/discussions/{num}/status", "hub", "-", "token", "write", "comment,status", ""},
|
||||
{Call::kDiscussionTitle, Surface::kHub, "POST", "/api/{repoType}/{namespace}/{repo}/discussions/{num}/title", "hub", "-", "token", "write", "title", ""},
|
||||
{Call::kCollectionsList, Surface::kHub, "GET", "/api/collections", "hub", "-", "optional", "", "", "cursor,expand,item,limit,owner,q,sort"},
|
||||
{Call::kCollectionGet, Surface::kHub, "GET", "/api/collections/{namespace}/{slug}", "hub", "-", "optional", "", "", ""},
|
||||
{Call::kCollectionCreate, Surface::kHub, "POST", "/api/collections", "hub", "-", "token", "write", "description,item,namespace,private,resourceGroupId,title", ""},
|
||||
{Call::kCollectionUpdate, Surface::kHub, "PATCH", "/api/collections/{namespace}/{slug}", "hub", "-", "token", "write", "description,gating,position,private,theme,title", ""},
|
||||
{Call::kCollectionDelete, Surface::kHub, "DELETE", "/api/collections/{namespace}/{slug}", "hub", "-", "token", "write,destructive", "", ""},
|
||||
{Call::kCollectionAddItem, Surface::kHub, "POST", "/api/collections/{namespace}/{slug}/items", "hub", "-", "token", "write", "item,note", ""},
|
||||
{Call::kWebhooksList, Surface::kHub, "GET", "/api/settings/webhooks", "hub", "-", "token", "", "", ""},
|
||||
{Call::kWebhookCreate, Surface::kHub, "POST", "/api/settings/webhooks", "hub", "-", "token", "write", "domains,job,jobSourceId,secret,secrets,url,watched", ""},
|
||||
{Call::kWebhookGet, Surface::kHub, "GET", "/api/settings/webhooks/{webhookId}", "hub", "-", "token", "", "", ""},
|
||||
{Call::kWebhookUpdate, Surface::kHub, "POST", "/api/settings/webhooks/{webhookId}", "hub", "-", "token", "write", "domains,job,jobSourceId,secret,secrets,url,watched", ""},
|
||||
{Call::kWebhookDelete, Surface::kHub, "DELETE", "/api/settings/webhooks/{webhookId}", "hub", "-", "token", "write,destructive", "", ""},
|
||||
{Call::kWebhookToggle, Surface::kHub, "POST", "/api/settings/webhooks/{webhookId}/{action}", "hub", "-", "token", "write", "", ""},
|
||||
{Call::kWebhookReplay, Surface::kHub, "POST", "/api/settings/webhooks/{webhookId}/replay/{logId}", "hub", "-", "token", "write", "", ""},
|
||||
{Call::kDailyPapers, Surface::kHub, "GET", "/api/daily_papers", "hub", "-", "optional", "", "", "date,limit,month,p,sort,submitter,week"},
|
||||
{Call::kPapersList, Surface::kHub, "GET", "/api/papers", "hub", "-", "optional", "", "", "cursor,limit"},
|
||||
{Call::kPapersSearch, Surface::kHub, "GET", "/api/papers/search", "hub", "-", "optional", "", "", "limit,q"},
|
||||
{Call::kPaperGet, Surface::kHub, "GET", "/api/papers/{paperId}", "hub", "-", "optional", "", "", "field"},
|
||||
{Call::kSpacesHardware, Surface::kHub, "GET", "/api/spaces/hardware", "hub", "-", "optional", "", "", ""},
|
||||
{Call::kSpacesTemplates, Surface::kHub, "GET", "/api/spaces/templates", "hub", "-", "optional", "", "", ""},
|
||||
{Call::kSpacesSemanticSearch, Surface::kHub, "GET", "/api/spaces/semantic-search", "hub", "-", "optional", "", "", "agents,category,filter,includeNonRunning,q,sdk"},
|
||||
{Call::kSpacesFeatured, Surface::kHub, "GET", "/api/spaces/featured", "hub", "-", "optional", "", "", "date"},
|
||||
{Call::kPartnerModels, Surface::kHub, "GET", "/api/partners/{provider}/models", "hub", "-", "optional", "", "", "status"},
|
||||
{Call::kModelsList, Surface::kHub, "GET", "/api/models", "src", "{self.endpoint}/api/models\"", "optional", "", "", "filter,author,apps,gated,inference,inference_provider,pipeline_tag,num_parameters,search,sort,limit,full,config,cardData,expand,cursor,direction"},
|
||||
{Call::kDatasetsList, Surface::kHub, "GET", "/api/datasets", "src", "{self.endpoint}/api/datasets\"", "optional", "", "", "filter,author,gated,search,sort,limit,full,expand,cursor,direction"},
|
||||
{Call::kSpacesList, Surface::kHub, "GET", "/api/spaces", "src", "{self.endpoint}/api/spaces\"", "optional", "", "", "filter,author,search,sort,limit,full,expand,cursor,direction,datasets,models,linked"},
|
||||
{Call::kModelInfo, Surface::kHub, "GET", "/api/models/{namespace}/{repo}", "src", "/api/models/{repo_id}\"", "optional", "", "", "securityStatus,blobs,expand"},
|
||||
{Call::kModelRevision, Surface::kHub, "GET", "/api/models/{namespace}/{repo}/revision/{rev}", "src", "/api/models/{repo_id}/revision/", "optional", "", "", "securityStatus,blobs,expand"},
|
||||
{Call::kDatasetInfo, Surface::kHub, "GET", "/api/datasets/{namespace}/{repo}", "src", "/api/datasets/{repo_id}\"", "optional", "", "", "blobs,expand"},
|
||||
{Call::kDatasetRevision, Surface::kHub, "GET", "/api/datasets/{namespace}/{repo}/revision/{rev}", "src", "/api/datasets/{repo_id}/revision/", "optional", "", "", "blobs,expand"},
|
||||
{Call::kSpaceInfo, Surface::kHub, "GET", "/api/spaces/{namespace}/{repo}", "src", "/api/spaces/{repo_id}\"", "optional", "", "", "expand"},
|
||||
{Call::kSpaceRevision, Surface::kHub, "GET", "/api/spaces/{namespace}/{repo}/revision/{rev}", "src", "/api/spaces/{repo_id}/revision/", "optional", "", "", "expand"},
|
||||
{Call::kSpaceRuntime, Surface::kHub, "GET", "/api/spaces/{namespace}/{repo}/runtime", "src", "/api/spaces/{repo_id}/runtime", "token", "", "", ""},
|
||||
{Call::kTasks, Surface::kHub, "GET", "/api/tasks", "src", "{constants.ENDPOINT}/api/tasks", "optional", "", "", ""},
|
||||
{Call::kRouterChat, Surface::kRouter, "POST", "/v1/chat/completions", "task", "chat-completion", "token", "", "frequency_penalty,logit_bias,logprobs,max_tokens,messages,model,n,presence_penalty,reasoning_effort,response_format,seed,stop,stream,stream_options,temperature,tool_choice,tool_prompt,tools,top_logprobs,top_p", ""},
|
||||
{Call::kRouterModels, Surface::kRouter, "GET", "/v1/models", "none", "probe", "none", "", "", ""},
|
||||
{Call::kHfInferenceTask, Surface::kRouter, "POST", "/hf-inference/models/{model}", "src", "models/{mapped_model}", "token", "", "inputs,parameters,stream", ""},
|
||||
{Call::kHfInferencePipeline, Surface::kRouter, "POST", "/hf-inference/models/{model}/pipeline/{task}", "src", "pipeline/{self.task}", "token", "", "inputs,parameters,normalize,prompt_name,truncate,truncation_direction", ""},
|
||||
{Call::kHfInferenceChat, Surface::kRouter, "POST", "/hf-inference/models/{model}/v1/chat/completions", "src", "\"/v1/chat/completions\"", "token", "", "frequency_penalty,logit_bias,logprobs,max_tokens,messages,model,n,presence_penalty,reasoning_effort,response_format,seed,stop,stream,stream_options,temperature,tool_choice,tool_prompt,tools,top_logprobs,top_p", ""},
|
||||
{Call::kIeGetAuditLogs, Surface::kEndpoints, "GET", "/v2/audit-logs/{namespace}", "ie", "get_audit_logs", "token", "", "", "cursor,endpoint,limit,username"},
|
||||
{Call::kIeListEndpoint, Surface::kEndpoints, "GET", "/v2/endpoint/{namespace}", "ie", "list_endpoint", "token", "", "", "cursor,direction,engine,instanceSize,instanceType,limit,search,sortBy,state,tags,task,type"},
|
||||
{Call::kIeCreateEndpoint, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}", "ie", "create_endpoint", "token", "write", "cacheHttpResponses,compute,customRouter,experimentalFeatures,model,name,network,notifications,privateService,provider,route,tags,type", ""},
|
||||
{Call::kIeGetEndpoint, Surface::kEndpoints, "GET", "/v2/endpoint/{namespace}/{name}", "ie", "get_endpoint", "token", "", "", ""},
|
||||
{Call::kIeUpdateEndpoint, Surface::kEndpoints, "PUT", "/v2/endpoint/{namespace}/{name}", "ie", "update_endpoint", "token", "write", "compute,customRouter,experimentalFeatures,model,network,notifications,provider,route,tags,type", ""},
|
||||
{Call::kIeDeleteEndpoint, Surface::kEndpoints, "DELETE", "/v2/endpoint/{namespace}/{name}", "ie", "delete_endpoint", "token", "write,destructive", "", ""},
|
||||
{Call::kIeDebugAgent, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/agent/debug", "ie", "debug_agent", "token", "write", "", ""},
|
||||
{Call::kIeGetEndpointLogs, Surface::kEndpoints, "GET", "/v2/endpoint/{namespace}/{name}/logs", "ie", "get_endpoint_logs", "token", "raw", "", "follow,line_max_length,replica,tail"},
|
||||
{Call::kIeGetEndpointLogsSse, Surface::kEndpoints, "GET", "/v2/endpoint/{namespace}/{name}/logs/sse", "ie", "get_endpoint_logs_sse", "token", "unsupported", "", "line_max_length,replica,tail"},
|
||||
{Call::kIeGetEndpointMetrics, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics", "ie", "get_endpoint_metrics", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetHardwareCpu, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/hardware-cpu", "ie", "get_hardware_cpu", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetHardwareGpu, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/hardware-gpu", "ie", "get_hardware_gpu", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetInferenceItl, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/inference/itl", "ie", "get_inference_itl", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetInferenceKvCache, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/inference/kv-cache", "ie", "get_inference_kv_cache", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetInferencePrefixCache, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/inference/prefix-cache", "ie", "get_inference_prefix_cache", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetInferenceRequests, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/inference/requests", "ie", "get_inference_requests", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetInferenceTtft, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/inference/ttft", "ie", "get_inference_ttft", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetPendingRequest, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/pending-request", "ie", "get_pending_request", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetReplicaIds, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/replica-ids", "ie", "get_replica_ids", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetReplicaRunning, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/replica-running", "ie", "get_replica_running", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetReplicaStatus, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/replica-status", "ie", "get_replica_status", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetResponseElapsed, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/response-elapsed", "ie", "get_response_elapsed", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetResponseStatusCode, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/response-status-code", "ie", "get_response_status_code", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetStatusCodeGrouped, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/metrics/response-status-code-grouped", "ie", "get_status_code_grouped", "token", "", "replicaId,start,stop", ""},
|
||||
{Call::kIeGetEndpointOpenMetrics, Surface::kEndpoints, "GET", "/v2/endpoint/{namespace}/{name}/open-metrics", "ie", "get_endpoint_open_metrics", "token", "raw", "", ""},
|
||||
{Call::kIePauseEndpoint, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/pause", "ie", "pause_endpoint", "token", "write,destructive", "", ""},
|
||||
{Call::kIeGetEndpointReplica, Surface::kEndpoints, "GET", "/v2/endpoint/{namespace}/{name}/replica", "ie", "get_endpoint_replica", "token", "", "", ""},
|
||||
{Call::kIeResumeEndpoint, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/resume", "ie", "resume_endpoint", "token", "write", "", ""},
|
||||
{Call::kIeScaleToZeroEndpoint, Surface::kEndpoints, "POST", "/v2/endpoint/{namespace}/{name}/scale-to-zero", "ie", "scale_to_zero_endpoint", "token", "write,destructive", "", ""},
|
||||
{Call::kIeGetEndpointSse, Surface::kEndpoints, "GET", "/v2/endpoint/{namespace}/{name}/sse", "ie", "get_endpoint_sse", "token", "unsupported", "", ""},
|
||||
{Call::kIeListCidrsList, Surface::kEndpoints, "GET", "/v2/network/cidr-list/{namespace}", "ie", "list_cidrs_list", "token", "", "", ""},
|
||||
{Call::kIeCreateCidrList, Surface::kEndpoints, "POST", "/v2/network/cidr-list/{namespace}", "ie", "create_cidr_list", "token", "write", "cidrList,id,name", ""},
|
||||
{Call::kIeUpdateCidrList, Surface::kEndpoints, "POST", "/v2/network/cidr-list/{namespace}/{cidr_id}", "ie", "update_cidr_list", "token", "write", "cidrList,id,name", ""},
|
||||
{Call::kIeDeleteCidrList, Surface::kEndpoints, "DELETE", "/v2/network/cidr-list/{namespace}/{cidr_id}", "ie", "delete_cidr_list", "token", "write,destructive", "", ""},
|
||||
{Call::kIeListDomains, Surface::kEndpoints, "GET", "/v2/network/domain/{namespace}", "ie", "list_domains", "token", "", "", ""},
|
||||
{Call::kIeCreateDomain, Surface::kEndpoints, "POST", "/v2/network/domain/{namespace}", "ie", "create_domain", "token", "write", "domain", ""},
|
||||
{Call::kIeDeleteDomain, Surface::kEndpoints, "DELETE", "/v2/network/domain/{namespace}/{domain}", "ie", "delete_domain", "token", "write,destructive", "", ""},
|
||||
{Call::kIeListPrivateServices, Surface::kEndpoints, "GET", "/v2/network/private-service/{namespace}", "ie", "list_private_services", "token", "", "", "account-id,shared"},
|
||||
{Call::kIeDeletePrivateService, Surface::kEndpoints, "DELETE", "/v2/network/private-service/{namespace}/{service-id}", "ie", "delete_private_service", "token", "write,destructive", "", ""},
|
||||
{Call::kIeListVendors, Surface::kEndpoints, "GET", "/v2/provider", "ie", "list_vendors", "none", "", "", ""},
|
||||
{Call::kIeListVendorsQuotasNamespace, Surface::kEndpoints, "GET", "/v2/provider/quotas/{namespace}", "ie", "list_vendors_quotas_namespace", "token", "", "", ""},
|
||||
{Call::kIeListVendorsNamespace, Surface::kEndpoints, "GET", "/v2/provider/{namespace}", "ie", "list_vendors_namespace", "token", "", "", ""},
|
||||
{Call::kIeSearchEndpoint, Surface::kEndpoints, "GET", "/v2/search/{namespace}", "ie", "search_endpoint", "token", "", "", "limit,search,tags"},
|
||||
{Call::kIeCountEndpointStates, Surface::kEndpoints, "GET", "/v2/search/{namespace}/states", "ie", "count_endpoint_states", "token", "", "", ""},
|
||||
{Call::kIeListEndpointTags, Surface::kEndpoints, "GET", "/v2/search/{namespace}/tags", "ie", "list_endpoint_tags", "token", "", "", "limit,prefix"},
|
||||
{Call::kIeGetEndpointLogsV3, Surface::kEndpoints, "GET", "/v3/endpoint/{namespace}/{name}/logs", "ie", "get_endpoint_logs_v3", "token", "", "", "after,before,level,limit,line_max_length,order,replica,search,since,until"},
|
||||
{Call::kIeDownloadEndpointLogsV3, Surface::kEndpoints, "GET", "/v3/endpoint/{namespace}/{name}/logs/download", "ie", "download_endpoint_logs_v3", "token", "raw", "", "level,line_max_length,replica,search,since,until"},
|
||||
{Call::kIeGetEndpointLogsSseV3, Surface::kEndpoints, "GET", "/v3/endpoint/{namespace}/{name}/logs/sse", "ie", "get_endpoint_logs_sse_v3", "token", "unsupported", "", "after,level,line_max_length,replica,search,since,tail"},
|
||||
{Call::kTgiCompatGenerate, Surface::kTgi, "POST", "/", "tgi", "compat_generate", "optional", "", "inputs,parameters,stream", ""},
|
||||
{Call::kTgiGetChatTokenize, Surface::kTgi, "POST", "/chat_tokenize", "tgi", "get_chat_tokenize", "optional", "", "frequency_penalty,logit_bias,logprobs,max_tokens,messages,model,n,presence_penalty,response_format,seed,stop,stream,stream_options,temperature,tool_choice,tool_prompt,tools,top_logprobs,top_p", ""},
|
||||
{Call::kTgiGenerate, Surface::kTgi, "POST", "/generate", "tgi", "generate", "optional", "", "inputs,parameters", ""},
|
||||
{Call::kTgiGenerateStream, Surface::kTgi, "POST", "/generate_stream", "tgi", "generate_stream", "optional", "stream", "inputs,parameters", ""},
|
||||
{Call::kTgiHealth, Surface::kTgi, "GET", "/health", "tgi", "health", "optional", "", "", ""},
|
||||
{Call::kTgiGetModelInfo, Surface::kTgi, "GET", "/info", "tgi", "get_model_info", "optional", "", "", ""},
|
||||
{Call::kTgiSagemakerCompatibility, Surface::kTgi, "POST", "/invocations", "tgi", "sagemaker_compatibility", "optional", "", "frequency_penalty,inputs,logit_bias,logprobs,max_tokens,messages,model,n,parameters,presence_penalty,prompt,repetition_penalty,response_format,seed,stop,stream,stream_options,suffix,temperature,tool_choice,tool_prompt,tools,top_logprobs,top_p", ""},
|
||||
{Call::kTgiMetrics, Surface::kTgi, "GET", "/metrics", "tgi", "metrics", "optional", "raw", "", ""},
|
||||
{Call::kTgiTokenize, Surface::kTgi, "POST", "/tokenize", "tgi", "tokenize", "optional", "", "inputs,parameters", ""},
|
||||
{Call::kTgiChatCompletions, Surface::kTgi, "POST", "/v1/chat/completions", "tgi", "chat_completions", "optional", "", "frequency_penalty,logit_bias,logprobs,max_tokens,messages,model,n,presence_penalty,response_format,seed,stop,stream,stream_options,temperature,tool_choice,tool_prompt,tools,top_logprobs,top_p", ""},
|
||||
{Call::kTgiCompletions, Surface::kTgi, "POST", "/v1/completions", "tgi", "completions", "optional", "", "frequency_penalty,max_tokens,model,prompt,repetition_penalty,seed,stop,stream,suffix,temperature,top_p", ""},
|
||||
{Call::kTgiOpenaiGetModelInfo, Surface::kTgi, "GET", "/v1/models", "tgi", "openai_get_model_info", "optional", "", "", ""},
|
||||
{Call::kTeiDecode, Surface::kTei, "POST", "/decode", "tei", "decode", "optional", "", "ids,skip_special_tokens", ""},
|
||||
{Call::kTeiEmbed, Surface::kTei, "POST", "/embed", "tei", "embed", "optional", "", "dimensions,inputs,normalize,prompt_name,truncate,truncation_direction", ""},
|
||||
{Call::kTeiEmbedAll, Surface::kTei, "POST", "/embed_all", "tei", "embed_all", "optional", "", "inputs,prompt_name,truncate,truncation_direction", ""},
|
||||
{Call::kTeiEmbedSparse, Surface::kTei, "POST", "/embed_sparse", "tei", "embed_sparse", "optional", "", "inputs,prompt_name,truncate,truncation_direction", ""},
|
||||
{Call::kTeiHealth, Surface::kTei, "GET", "/health", "tei", "health", "optional", "", "", ""},
|
||||
{Call::kTeiGetModelInfo, Surface::kTei, "GET", "/info", "tei", "get_model_info", "optional", "", "", ""},
|
||||
{Call::kTeiMetrics, Surface::kTei, "GET", "/metrics", "tei", "metrics", "optional", "raw", "", ""},
|
||||
{Call::kTeiPredict, Surface::kTei, "POST", "/predict", "tei", "predict", "optional", "", "inputs,raw_scores,truncate,truncation_direction", ""},
|
||||
{Call::kTeiRerank, Surface::kTei, "POST", "/rerank", "tei", "rerank", "optional", "", "query,raw_scores,return_text,texts,truncate,truncation_direction", ""},
|
||||
{Call::kTeiSimilarity, Surface::kTei, "POST", "/similarity", "tei", "similarity", "optional", "", "inputs,parameters", ""},
|
||||
{Call::kTeiTokenize, Surface::kTei, "POST", "/tokenize", "tei", "tokenize", "optional", "", "add_special_tokens,inputs,prompt_name", ""},
|
||||
{Call::kTeiOpenaiEmbed, Surface::kTei, "POST", "/v1/embeddings", "tei", "openai_embed", "optional", "", "dimensions,encoding_format,input,model,user", ""},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
bool CallSpec::has_flag(const char* flag) const { return Calls::in_list(flags, flag); }
|
||||
|
||||
bool Calls::in_list(const char* list, const std::string& name) {
|
||||
const std::size_t n = name.size();
|
||||
const char* p = list;
|
||||
while (*p) {
|
||||
const char* end = std::strchr(p, ',');
|
||||
std::size_t len = end ? static_cast<std::size_t>(end - p) : std::strlen(p);
|
||||
if (len == n && std::strncmp(p, name.c_str(), n) == 0) return true;
|
||||
if (!end) break;
|
||||
p = end + 1;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
std::size_t Calls::count() { return sizeof(kCalls) / sizeof(kCalls[0]); }
|
||||
|
||||
const CallSpec& Calls::at(std::size_t index) {
|
||||
if (index >= count()) throw llm::ApiError::invalid("huggingface", "call index out of range");
|
||||
return kCalls[index];
|
||||
}
|
||||
|
||||
const CallSpec& Calls::spec_of(Call call) {
|
||||
const std::size_t i = static_cast<std::size_t>(call);
|
||||
if (i >= count() || kCalls[i].call != call)
|
||||
throw llm::ApiError::invalid("huggingface", "call table out of order");
|
||||
return kCalls[i];
|
||||
}
|
||||
|
||||
clients::Method Calls::method_of(Call call) {
|
||||
const char* m = spec_of(call).http_method;
|
||||
if (!std::strcmp(m, "GET")) return clients::Method::kGet;
|
||||
if (!std::strcmp(m, "POST")) return clients::Method::kPost;
|
||||
if (!std::strcmp(m, "PUT")) return clients::Method::kPut;
|
||||
if (!std::strcmp(m, "PATCH")) return clients::Method::kPatch;
|
||||
if (!std::strcmp(m, "DELETE")) return clients::Method::kDelete;
|
||||
throw llm::ApiError::invalid("huggingface", "unknown http method in the call table");
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+189
@@ -0,0 +1,189 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of HTTP calls this module makes (every surface). Code builds
|
||||
// every path, query and body check from it, never from a literal.
|
||||
// tools/check_spec.py verifies each row against the published documents:
|
||||
// spec "hub" -> https://huggingface.co/.well-known/openapi.json (method, path, query, body fields)
|
||||
// spec "ie" -> https://api.endpoints.huggingface.cloud/openapi.json
|
||||
// spec "tgi" -> text-generation-inference docs/openapi.json
|
||||
// spec "tei" -> text-embeddings-inference docs/openapi.json
|
||||
// spec "src" -> pinned huggingface_hub source (the Hub OpenAPI omits these): `ref` must occur in it
|
||||
// spec "task" -> huggingface.js task input schema named by `ref`
|
||||
// spec "none" -> not in any published document; confirmed by an unauthenticated probe (see SPEC)
|
||||
// Fields: auth is "none" (public), "optional" (public, a token widens it) or
|
||||
// "token" (needs HF_TOKEN). flags: write, destructive, raw (non JSON reply),
|
||||
// stream, optpath (a trailing {path} may be empty), unsupported (listed for
|
||||
// coverage, refused at call time with the reason in SPEC.md).
|
||||
|
||||
#include <cstddef>
|
||||
#include <string>
|
||||
|
||||
#include "clients/request.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
enum class Surface { kHub, kRouter, kEndpoints, kTgi, kTei };
|
||||
|
||||
const char* surface_name(Surface surface);
|
||||
|
||||
enum class Call {
|
||||
kWhoami,
|
||||
kModelsTagsByType,
|
||||
kDatasetsTagsByType,
|
||||
kTrending,
|
||||
kQuicksearch,
|
||||
kFullTextSearch,
|
||||
kRepoLikers,
|
||||
kRepoTree,
|
||||
kRepoTreeSize,
|
||||
kRepoRefs,
|
||||
kRepoCommits,
|
||||
kRepoCompare,
|
||||
kRepoPathsInfo,
|
||||
kModelSafetensors,
|
||||
kFileResolve,
|
||||
kDiscussionList,
|
||||
kDiscussionCreate,
|
||||
kDiscussionGet,
|
||||
kDiscussionDelete,
|
||||
kDiscussionComment,
|
||||
kDiscussionStatus,
|
||||
kDiscussionTitle,
|
||||
kCollectionsList,
|
||||
kCollectionGet,
|
||||
kCollectionCreate,
|
||||
kCollectionUpdate,
|
||||
kCollectionDelete,
|
||||
kCollectionAddItem,
|
||||
kWebhooksList,
|
||||
kWebhookCreate,
|
||||
kWebhookGet,
|
||||
kWebhookUpdate,
|
||||
kWebhookDelete,
|
||||
kWebhookToggle,
|
||||
kWebhookReplay,
|
||||
kDailyPapers,
|
||||
kPapersList,
|
||||
kPapersSearch,
|
||||
kPaperGet,
|
||||
kSpacesHardware,
|
||||
kSpacesTemplates,
|
||||
kSpacesSemanticSearch,
|
||||
kSpacesFeatured,
|
||||
kPartnerModels,
|
||||
kModelsList,
|
||||
kDatasetsList,
|
||||
kSpacesList,
|
||||
kModelInfo,
|
||||
kModelRevision,
|
||||
kDatasetInfo,
|
||||
kDatasetRevision,
|
||||
kSpaceInfo,
|
||||
kSpaceRevision,
|
||||
kSpaceRuntime,
|
||||
kTasks,
|
||||
kRouterChat,
|
||||
kRouterModels,
|
||||
kHfInferenceTask,
|
||||
kHfInferencePipeline,
|
||||
kHfInferenceChat,
|
||||
kIeGetAuditLogs,
|
||||
kIeListEndpoint,
|
||||
kIeCreateEndpoint,
|
||||
kIeGetEndpoint,
|
||||
kIeUpdateEndpoint,
|
||||
kIeDeleteEndpoint,
|
||||
kIeDebugAgent,
|
||||
kIeGetEndpointLogs,
|
||||
kIeGetEndpointLogsSse,
|
||||
kIeGetEndpointMetrics,
|
||||
kIeGetHardwareCpu,
|
||||
kIeGetHardwareGpu,
|
||||
kIeGetInferenceItl,
|
||||
kIeGetInferenceKvCache,
|
||||
kIeGetInferencePrefixCache,
|
||||
kIeGetInferenceRequests,
|
||||
kIeGetInferenceTtft,
|
||||
kIeGetPendingRequest,
|
||||
kIeGetReplicaIds,
|
||||
kIeGetReplicaRunning,
|
||||
kIeGetReplicaStatus,
|
||||
kIeGetResponseElapsed,
|
||||
kIeGetResponseStatusCode,
|
||||
kIeGetStatusCodeGrouped,
|
||||
kIeGetEndpointOpenMetrics,
|
||||
kIePauseEndpoint,
|
||||
kIeGetEndpointReplica,
|
||||
kIeResumeEndpoint,
|
||||
kIeScaleToZeroEndpoint,
|
||||
kIeGetEndpointSse,
|
||||
kIeListCidrsList,
|
||||
kIeCreateCidrList,
|
||||
kIeUpdateCidrList,
|
||||
kIeDeleteCidrList,
|
||||
kIeListDomains,
|
||||
kIeCreateDomain,
|
||||
kIeDeleteDomain,
|
||||
kIeListPrivateServices,
|
||||
kIeDeletePrivateService,
|
||||
kIeListVendors,
|
||||
kIeListVendorsQuotasNamespace,
|
||||
kIeListVendorsNamespace,
|
||||
kIeSearchEndpoint,
|
||||
kIeCountEndpointStates,
|
||||
kIeListEndpointTags,
|
||||
kIeGetEndpointLogsV3,
|
||||
kIeDownloadEndpointLogsV3,
|
||||
kIeGetEndpointLogsSseV3,
|
||||
kTgiCompatGenerate,
|
||||
kTgiGetChatTokenize,
|
||||
kTgiGenerate,
|
||||
kTgiGenerateStream,
|
||||
kTgiHealth,
|
||||
kTgiGetModelInfo,
|
||||
kTgiSagemakerCompatibility,
|
||||
kTgiMetrics,
|
||||
kTgiTokenize,
|
||||
kTgiChatCompletions,
|
||||
kTgiCompletions,
|
||||
kTgiOpenaiGetModelInfo,
|
||||
kTeiDecode,
|
||||
kTeiEmbed,
|
||||
kTeiEmbedAll,
|
||||
kTeiEmbedSparse,
|
||||
kTeiHealth,
|
||||
kTeiGetModelInfo,
|
||||
kTeiMetrics,
|
||||
kTeiPredict,
|
||||
kTeiRerank,
|
||||
kTeiSimilarity,
|
||||
kTeiTokenize,
|
||||
kTeiOpenaiEmbed,
|
||||
};
|
||||
|
||||
struct CallSpec {
|
||||
Call call;
|
||||
Surface surface;
|
||||
const char* http_method; // GET POST PUT PATCH DELETE
|
||||
const char* path; // template: {name} placeholders
|
||||
const char* spec; // none, hub, ie, tgi, tei, src, task
|
||||
const char* ref; // operationId, source needle, or task name
|
||||
const char* auth; // none, optional, token
|
||||
const char* flags; // comma list
|
||||
const char* body_fields; // comma list: accepted top-level body fields (empty: no body)
|
||||
const char* query_params; // comma list: accepted query parameters
|
||||
|
||||
bool has_flag(const char* flag) const;
|
||||
bool has_body() const { return body_fields[0] != '\0'; }
|
||||
};
|
||||
|
||||
struct Calls {
|
||||
static const CallSpec& spec_of(Call call);
|
||||
static clients::Method method_of(Call call);
|
||||
static std::size_t count();
|
||||
static const CallSpec& at(std::size_t index);
|
||||
// True when `name` is in the comma list.
|
||||
static bool in_list(const char* list, const std::string& name);
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+46
@@ -0,0 +1,46 @@
|
||||
#pragma once
|
||||
|
||||
// Static ceilings for the Hugging Face module. Never autotuned. The numbers
|
||||
// marked "observed" come from response headers seen on 2026-10-25 (see
|
||||
// SPEC.md ## Verified live); the rest are our own hard caps.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
struct HfCaps {
|
||||
// Path and query rendering.
|
||||
static constexpr std::size_t kMaxSegmentChars = 256; // one path segment (id, rev, tag)
|
||||
static constexpr std::size_t kMaxFilePathChars = 1024; // a path inside a repo
|
||||
static constexpr std::size_t kMaxPathSegments = 32;
|
||||
static constexpr std::size_t kMaxQueryParams = 32;
|
||||
static constexpr std::size_t kMaxQueryValueChars = 2048;
|
||||
static constexpr std::size_t kMaxModelChars = 256;
|
||||
static constexpr std::size_t kMaxServerUrlChars = 2048;
|
||||
|
||||
// Request budget. observed: RateLimit-Policy "fixed window";"api";q=500;w=300
|
||||
// on huggingface.co. We allow twice that per process so a single process
|
||||
// never trips the server limit by accident, and stop locally beyond it.
|
||||
static constexpr int kBudgetWindowSeconds = 300;
|
||||
static constexpr int kBudgetMaxCalls = 1000;
|
||||
|
||||
// Page walks (Link: rel="next" cursors). Truncation is reported, never silent.
|
||||
static constexpr std::size_t kMaxPages = 20;
|
||||
static constexpr std::size_t kMaxItems = 5000;
|
||||
static constexpr int kMaxPageLimit = 1000;
|
||||
static constexpr std::size_t kMaxPathsInfo = 256;
|
||||
|
||||
// Payloads.
|
||||
static constexpr std::size_t kMaxDownloadBytes = 64u * 1024u * 1024u; // matches llm::Caps response cap
|
||||
static constexpr std::size_t kMaxInferenceInputBytes = 16u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxBatchInputs = 256; // TEI embed / rerank batches
|
||||
static constexpr std::size_t kMaxStreamEvents = 200000;
|
||||
static constexpr std::size_t kMaxStopSequences = 16;
|
||||
static constexpr std::size_t kMaxMessages = 4096;
|
||||
static constexpr std::size_t kMaxTools = 128;
|
||||
static constexpr int kMaxTokensCeiling = 1048576;
|
||||
static constexpr std::size_t kMaxWebhookDomains = 8;
|
||||
static constexpr std::size_t kMaxTextChars = 4u * 1024u * 1024u;
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
#include "vendors/huggingface/chat_model.h"
|
||||
|
||||
#include "vendors/huggingface/providers.h"
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
HuggingfaceChatModel::HuggingfaceChatModel(std::shared_ptr<ChatEndpoint> endpoint, std::string default_model)
|
||||
: endpoint_(std::move(endpoint)), default_model_(std::move(default_model)) {
|
||||
if (!endpoint_) throw llm::ApiError::invalid("huggingface", "chat model needs an endpoint");
|
||||
}
|
||||
|
||||
std::shared_ptr<HuggingfaceChatModel> HuggingfaceChatModel::from_env(const std::string& default_model) {
|
||||
return over_router(Platform::from_env(), default_model);
|
||||
}
|
||||
|
||||
std::shared_ptr<HuggingfaceChatModel> HuggingfaceChatModel::over_router(const Platform& platform,
|
||||
const std::string& default_model) {
|
||||
return std::make_shared<HuggingfaceChatModel>(std::make_shared<RouterChat>(RouterChat::from_platform(platform)),
|
||||
default_model);
|
||||
}
|
||||
|
||||
clients::body::Json HuggingfaceChatModel::to_body(const llm::ChatRequest& request, bool stream) const {
|
||||
return ChatWire::to_body(request, default_model_, stream);
|
||||
}
|
||||
|
||||
llm::ChatResponse HuggingfaceChatModel::generate(const llm::ChatRequest& request) {
|
||||
return ChatWire::to_response(endpoint_->create(to_body(request, false)));
|
||||
}
|
||||
|
||||
void HuggingfaceChatModel::stream(const llm::ChatRequest& request,
|
||||
const std::function<bool(const llm::ChatChunk&)>& on_chunk) {
|
||||
ChatWire::StreamState state;
|
||||
bool stopped = false;
|
||||
endpoint_->stream(to_body(request, true), [&](const llm::SseEvent& e) {
|
||||
if (stopped) return false;
|
||||
if (!ChatWire::on_event(e.data, state, on_chunk)) {
|
||||
stopped = true;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+40
@@ -0,0 +1,40 @@
|
||||
#pragma once
|
||||
|
||||
// HuggingfaceChatModel: the llm::ChatModel seam over any Hugging Face chat
|
||||
// endpoint that speaks the OpenAI-compatible chat wire: the Inference
|
||||
// Providers router (default), hf-inference, or a TGI server. The mapping rules
|
||||
// are in chat_wire.h. The default model must be given by the caller: the
|
||||
// router has no default (model ids change; GET /v1/models lists them).
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "vendors/huggingface/chat_wire.h"
|
||||
#include "vendors/huggingface/platform.h"
|
||||
#include "llm/chat_types.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
class HuggingfaceChatModel : public llm::ChatModel {
|
||||
public:
|
||||
HuggingfaceChatModel(std::shared_ptr<ChatEndpoint> endpoint, std::string default_model);
|
||||
// Router with HF_TOKEN from the environment.
|
||||
static std::shared_ptr<HuggingfaceChatModel> from_env(const std::string& default_model);
|
||||
// Router over an explicit platform (tests).
|
||||
static std::shared_ptr<HuggingfaceChatModel> over_router(const Platform& platform,
|
||||
const std::string& default_model);
|
||||
|
||||
std::string vendor() const override { return "huggingface"; }
|
||||
llm::ChatResponse generate(const llm::ChatRequest& request) override;
|
||||
void stream(const llm::ChatRequest& request,
|
||||
const std::function<bool(const llm::ChatChunk&)>& on_chunk) override;
|
||||
|
||||
// The wire body a neutral request maps to (tests, dry runs).
|
||||
clients::body::Json to_body(const llm::ChatRequest& request, bool stream) const;
|
||||
|
||||
private:
|
||||
std::shared_ptr<ChatEndpoint> endpoint_;
|
||||
std::string default_model_;
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
+328
@@ -0,0 +1,328 @@
|
||||
#include "vendors/huggingface/chat_wire.h"
|
||||
|
||||
#include "vendors/huggingface/caps.h"
|
||||
#include "llm/error.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("huggingface", why); }
|
||||
|
||||
constexpr std::size_t kMaxStop = 4;
|
||||
|
||||
Json image_url_part(const llm::ImageData& img) {
|
||||
std::string url = img.url;
|
||||
if (url.empty()) {
|
||||
if (img.bytes.empty()) bad("image has neither url nor bytes");
|
||||
if (img.mime.empty()) bad("image bytes need a mime type");
|
||||
url = "data:" + img.mime + ";base64," + security::crypto::Base64::encode(img.bytes);
|
||||
}
|
||||
Json u = Json::object();
|
||||
u.set("url", Json(url));
|
||||
Json p = Json::object();
|
||||
p.set("type", Json("image_url"));
|
||||
p.set("image_url", u);
|
||||
return p;
|
||||
}
|
||||
|
||||
Json text_part(const std::string& t) {
|
||||
Json p = Json::object();
|
||||
p.set("type", Json("text"));
|
||||
p.set("text", Json(t));
|
||||
return p;
|
||||
}
|
||||
|
||||
Json tool_call_json(const llm::ToolCall& c) {
|
||||
Json fn = Json::object();
|
||||
fn.set("name", Json(c.name));
|
||||
fn.set("arguments", Json(c.arguments.is_null() ? std::string("{}") : c.arguments.dump()));
|
||||
Json o = Json::object();
|
||||
o.set("id", Json(c.id));
|
||||
o.set("type", Json("function"));
|
||||
o.set("function", fn);
|
||||
return o;
|
||||
}
|
||||
|
||||
void append_message(Json& messages, const llm::Message& m) {
|
||||
switch (m.role) {
|
||||
case llm::Role::kSystem: {
|
||||
Json o = Json::object();
|
||||
o.set("role", Json("system"));
|
||||
o.set("content", Json(m.text()));
|
||||
if (!m.name.empty()) o.set("name", Json(m.name));
|
||||
messages.push(o);
|
||||
return;
|
||||
}
|
||||
case llm::Role::kUser: {
|
||||
Json o = Json::object();
|
||||
o.set("role", Json("user"));
|
||||
bool has_image = false;
|
||||
for (const llm::ContentPart& p : m.parts) has_image = has_image || p.image.has_value();
|
||||
if (!has_image) {
|
||||
o.set("content", Json(m.text()));
|
||||
} else {
|
||||
Json parts = Json::array();
|
||||
for (const llm::ContentPart& p : m.parts) {
|
||||
if (!p.text.empty()) parts.push(text_part(p.text));
|
||||
if (p.image) parts.push(image_url_part(*p.image));
|
||||
}
|
||||
o.set("content", parts);
|
||||
}
|
||||
if (!m.name.empty()) o.set("name", Json(m.name));
|
||||
messages.push(o);
|
||||
return;
|
||||
}
|
||||
case llm::Role::kAssistant: {
|
||||
Json o = Json::object();
|
||||
o.set("role", Json("assistant"));
|
||||
Json calls = Json::array();
|
||||
for (const llm::ContentPart& p : m.parts)
|
||||
if (p.tool_call) calls.push(tool_call_json(*p.tool_call));
|
||||
const std::string text = m.text();
|
||||
if (!text.empty() || calls.as_array().empty()) o.set("content", Json(text));
|
||||
if (!calls.as_array().empty()) o.set("tool_calls", calls);
|
||||
messages.push(o);
|
||||
return;
|
||||
}
|
||||
case llm::Role::kTool: {
|
||||
bool any = false;
|
||||
for (const llm::ContentPart& p : m.parts) {
|
||||
if (!p.tool_result) continue;
|
||||
Json o = Json::object();
|
||||
o.set("role", Json("tool"));
|
||||
o.set("tool_call_id", Json(p.tool_result->call_id));
|
||||
o.set("content", Json(p.tool_result->content));
|
||||
messages.push(o);
|
||||
any = true;
|
||||
}
|
||||
if (!any) bad("a tool message needs a tool result part");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
llm::Usage usage_of(const Json* u) {
|
||||
llm::Usage out;
|
||||
if (!u || !u->is_object()) return out;
|
||||
auto num = [&](const Json& o, const char* k) {
|
||||
const Json* v = o.find(k);
|
||||
return v && v->is_number() ? static_cast<int>(v->as_int()) : 0;
|
||||
};
|
||||
out.input_tokens = num(*u, "prompt_tokens");
|
||||
out.output_tokens = num(*u, "completion_tokens");
|
||||
out.total_tokens = num(*u, "total_tokens");
|
||||
if (const Json* d = u->find("completion_tokens_details"); d && d->is_object())
|
||||
out.reasoning_tokens = num(*d, "reasoning_tokens");
|
||||
if (const Json* d = u->find("prompt_tokens_details"); d && d->is_object())
|
||||
out.cached_tokens = num(*d, "cached_tokens");
|
||||
return out;
|
||||
}
|
||||
|
||||
std::string str(const Json& o, const char* k) {
|
||||
const Json* v = o.is_object() ? o.find(k) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : std::string();
|
||||
}
|
||||
|
||||
std::string reasoning_of(const Json& holder) {
|
||||
std::string r = str(holder, "reasoning_content");
|
||||
if (r.empty()) r = str(holder, "reasoning");
|
||||
return r;
|
||||
}
|
||||
|
||||
Json parse_arguments(const std::string& raw) {
|
||||
if (raw.empty()) return Json::object();
|
||||
try {
|
||||
Json j = Json::parse(raw);
|
||||
if (j.is_object()) return j;
|
||||
} catch (const std::exception&) {
|
||||
}
|
||||
Json o = Json::object();
|
||||
o.set("_invalid_json", Json(raw));
|
||||
return o;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Json ChatWire::to_body(const llm::ChatRequest& r, const std::string& default_model, bool stream) {
|
||||
const std::string model = r.model.empty() ? default_model : r.model;
|
||||
if (model.empty() || model.size() > HfCaps::kMaxModelChars) bad("model is empty or too long");
|
||||
if (r.messages.empty()) bad("messages must not be empty");
|
||||
if (r.messages.size() > HfCaps::kMaxMessages) bad("too many messages");
|
||||
if (r.tools.size() > HfCaps::kMaxTools) bad("too many tools");
|
||||
if (r.stop.size() > kMaxStop) bad("at most 4 stop sequences");
|
||||
if (r.json_mode && !r.response_schema.is_null()) bad("json_mode and response_schema are exclusive");
|
||||
if (r.max_tokens && (*r.max_tokens < 1 || *r.max_tokens > HfCaps::kMaxTokensCeiling)) bad("max_tokens out of range");
|
||||
if (r.temperature && (*r.temperature < 0.0 || *r.temperature > 2.0)) bad("temperature must be 0..2");
|
||||
if (r.top_p && (*r.top_p <= 0.0 || *r.top_p > 1.0)) bad("top_p must be in (0, 1]");
|
||||
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(model));
|
||||
Json messages = Json::array();
|
||||
if (!r.system.empty()) {
|
||||
Json s = Json::object();
|
||||
s.set("role", Json("system"));
|
||||
s.set("content", Json(r.system));
|
||||
messages.push(s);
|
||||
}
|
||||
for (const llm::Message& m : r.messages) append_message(messages, m);
|
||||
body.set("messages", messages);
|
||||
if (!r.tools.empty()) {
|
||||
Json tools = Json::array();
|
||||
for (const llm::ToolSpec& t : r.tools) {
|
||||
if (t.name.empty() || t.name.size() > 128) bad("tool name is empty or too long");
|
||||
Json fn = Json::object();
|
||||
fn.set("name", Json(t.name));
|
||||
if (!t.description.empty()) fn.set("description", Json(t.description));
|
||||
fn.set("parameters", t.parameters);
|
||||
Json o = Json::object();
|
||||
o.set("type", Json("function"));
|
||||
o.set("function", fn);
|
||||
tools.push(o);
|
||||
}
|
||||
body.set("tools", tools);
|
||||
}
|
||||
if (!r.tool_choice.empty()) {
|
||||
if (r.tool_choice == "auto" || r.tool_choice == "none" || r.tool_choice == "required") {
|
||||
body.set("tool_choice", Json(r.tool_choice));
|
||||
} else {
|
||||
Json n = Json::object();
|
||||
n.set("name", Json(r.tool_choice));
|
||||
Json c = Json::object();
|
||||
c.set("type", Json("function"));
|
||||
c.set("function", n);
|
||||
body.set("tool_choice", c);
|
||||
}
|
||||
}
|
||||
if (r.max_tokens) body.set("max_tokens", Json(*r.max_tokens));
|
||||
if (r.temperature) body.set("temperature", Json(*r.temperature));
|
||||
if (r.top_p) body.set("top_p", Json(*r.top_p));
|
||||
if (!r.stop.empty()) {
|
||||
Json s = Json::array();
|
||||
for (const std::string& x : r.stop) s.push(Json(x));
|
||||
body.set("stop", s);
|
||||
}
|
||||
if (r.json_mode) {
|
||||
Json f = Json::object();
|
||||
f.set("type", Json("json_object"));
|
||||
body.set("response_format", f);
|
||||
}
|
||||
if (!r.response_schema.is_null()) {
|
||||
Json cfg = Json::object();
|
||||
cfg.set("name", Json("response"));
|
||||
cfg.set("schema", r.response_schema);
|
||||
cfg.set("strict", Json(true));
|
||||
Json f = Json::object();
|
||||
f.set("type", Json("json_schema"));
|
||||
f.set("json_schema", cfg);
|
||||
body.set("response_format", f);
|
||||
}
|
||||
if (!r.reasoning_effort.empty()) body.set("reasoning_effort", Json(r.reasoning_effort));
|
||||
if (r.extra.is_object())
|
||||
for (const auto& [k, v] : r.extra.as_object()) body.set(k, v);
|
||||
if (stream) {
|
||||
body.set("stream", Json(true));
|
||||
Json so = Json::object();
|
||||
so.set("include_usage", Json(true));
|
||||
body.set("stream_options", so);
|
||||
}
|
||||
return body;
|
||||
}
|
||||
|
||||
llm::ChatResponse ChatWire::to_response(const Json& doc) {
|
||||
llm::ChatResponse out;
|
||||
out.raw = doc;
|
||||
out.id = str(doc, "id");
|
||||
out.model = str(doc, "model");
|
||||
out.message.role = llm::Role::kAssistant;
|
||||
const Json* choices = doc.is_object() ? doc.find("choices") : nullptr;
|
||||
if (!choices || !choices->is_array() || choices->as_array().empty())
|
||||
throw llm::ApiError("huggingface", 502, "BAD_RESPONSE", "reply has no choices");
|
||||
const Json& choice = choices->as_array().front();
|
||||
const Json* msg = choice.find("message");
|
||||
out.finish_reason = str(choice, "finish_reason");
|
||||
if (msg && msg->is_object()) {
|
||||
std::string reasoning = reasoning_of(*msg);
|
||||
if (!reasoning.empty()) {
|
||||
llm::ContentPart r;
|
||||
r.reasoning = reasoning;
|
||||
out.message.parts.push_back(r);
|
||||
}
|
||||
std::string content = str(*msg, "content");
|
||||
const Json* calls = msg->find("tool_calls");
|
||||
const bool has_calls = calls && calls->is_array() && !calls->as_array().empty();
|
||||
if (!content.empty() || !has_calls) out.message.parts.push_back(llm::ContentPart::of_text(content));
|
||||
if (has_calls) {
|
||||
for (const Json& c : calls->as_array()) {
|
||||
llm::ToolCall tc;
|
||||
tc.id = str(c, "id");
|
||||
const Json* fn = c.find("function");
|
||||
if (fn && fn->is_object()) {
|
||||
tc.name = str(*fn, "name");
|
||||
const Json* args = fn->find("arguments");
|
||||
if (args && args->is_string()) tc.arguments = parse_arguments(args->as_string());
|
||||
else if (args && args->is_object()) tc.arguments = *args; // TGI may send an object
|
||||
}
|
||||
out.message.parts.push_back(llm::ContentPart::of_call(tc));
|
||||
}
|
||||
}
|
||||
}
|
||||
out.usage = usage_of(doc.find("usage"));
|
||||
return out;
|
||||
}
|
||||
|
||||
bool ChatWire::on_event(const Json& event, StreamState& state,
|
||||
const std::function<bool(const llm::ChatChunk&)>& sink) {
|
||||
if (const Json* err = event.find("error")) {
|
||||
std::string msg = err->is_string() ? err->as_string() : str(*err, "message");
|
||||
throw llm::ApiError("huggingface", 502, "STREAM_ERROR", msg.empty() ? "error event in stream" : msg);
|
||||
}
|
||||
const Json* choices = event.find("choices");
|
||||
if (choices && choices->is_array() && !choices->as_array().empty()) {
|
||||
const Json& choice = choices->as_array().front();
|
||||
if (const Json* delta = choice.find("delta"); delta && delta->is_object()) {
|
||||
llm::ChatChunk c;
|
||||
c.text_delta = str(*delta, "content");
|
||||
c.reasoning_delta = reasoning_of(*delta);
|
||||
if (!c.text_delta.empty() || !c.reasoning_delta.empty())
|
||||
if (!sink(c)) return false;
|
||||
const Json* calls = delta->find("tool_calls");
|
||||
if (calls && calls->is_array()) {
|
||||
for (const Json& f : calls->as_array()) {
|
||||
int index = 0;
|
||||
if (const Json* i = f.find("index"); i && i->is_number()) index = static_cast<int>(i->as_int());
|
||||
std::string id = str(f, "id");
|
||||
if (!id.empty()) state.ids[index] = id;
|
||||
const Json* fn = f.find("function");
|
||||
std::string name = fn ? str(*fn, "name") : "";
|
||||
if (!name.empty()) state.names[index] = name;
|
||||
llm::ChatChunk t;
|
||||
t.tool_call_id = state.ids[index];
|
||||
t.tool_call_name = state.names[index];
|
||||
if (fn) {
|
||||
const Json* a = fn->find("arguments");
|
||||
if (a && a->is_string()) t.tool_args_delta = a->as_string();
|
||||
}
|
||||
if (!sink(t)) return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
std::string fin = str(choice, "finish_reason");
|
||||
if (!fin.empty()) {
|
||||
llm::ChatChunk c;
|
||||
c.finish_reason = fin;
|
||||
if (!sink(c)) return false;
|
||||
}
|
||||
}
|
||||
if (const Json* u = event.find("usage"); u && u->is_object()) {
|
||||
llm::ChatChunk c;
|
||||
c.usage = usage_of(u);
|
||||
if (!sink(c)) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+51
@@ -0,0 +1,51 @@
|
||||
#pragma once
|
||||
|
||||
// OpenAI-compatible chat wire used by every Hugging Face chat surface: the
|
||||
// Inference Providers router, hf-inference and a TGI server share the shape
|
||||
// (huggingface.js tasks/chat-completion/spec/input.json). ChatWire converts
|
||||
// the vendor-neutral llm::ChatRequest into that body and the replies back.
|
||||
//
|
||||
// Decisions:
|
||||
// - tool call arguments that are not valid JSON arrive as {"_invalid_json": raw}
|
||||
// - reasoning text is read from "reasoning_content", else "reasoning" (providers differ)
|
||||
// - response_schema maps to response_format json_schema (strict); json_mode to json_object;
|
||||
// both together are refused
|
||||
// - at most 4 stop sequences (the task spec's documented limit)
|
||||
// - ChatRequest.extra is merged into the body; keys the call table does not list are refused
|
||||
|
||||
#include <functional>
|
||||
#include <map>
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "vendors/huggingface/calls.h"
|
||||
#include "llm/chat_types.h"
|
||||
#include "llm/sse.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
// Where a chat body goes: the router, hf-inference, or a TGI server.
|
||||
class ChatEndpoint {
|
||||
public:
|
||||
virtual ~ChatEndpoint() = default;
|
||||
virtual clients::body::Json create(const clients::body::Json& body) = 0;
|
||||
virtual void stream(const clients::body::Json& body, const llm::SseDecoder::Sink& sink) = 0;
|
||||
};
|
||||
|
||||
struct ChatWire {
|
||||
static clients::body::Json to_body(const llm::ChatRequest& request, const std::string& default_model,
|
||||
bool stream);
|
||||
static llm::ChatResponse to_response(const clients::body::Json& doc);
|
||||
|
||||
// Tracks tool call fragments across stream events (index -> id, name).
|
||||
struct StreamState {
|
||||
std::map<int, std::string> ids;
|
||||
std::map<int, std::string> names;
|
||||
};
|
||||
// Converts one SSE event into chunks; the sink returns false to stop.
|
||||
// Returns false when the sink asked to stop. Throws ApiError for an error event.
|
||||
static bool on_event(const clients::body::Json& event, StreamState& state,
|
||||
const std::function<bool(const llm::ChatChunk&)>& sink);
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
+243
@@ -0,0 +1,243 @@
|
||||
#include "vendors/huggingface/community.h"
|
||||
|
||||
#include <set>
|
||||
|
||||
#include "vendors/huggingface/caps.h"
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("huggingface", why); }
|
||||
|
||||
void add_if(Query& q, const char* name, const std::string& value) {
|
||||
if (!value.empty()) q.add(name, value);
|
||||
}
|
||||
|
||||
PathArgs discussion_args(RepoType type, const RepoRef& repo, long num) {
|
||||
if (num < 1 || num > 999999999) bad("discussion number must be 1..999999999");
|
||||
PathArgs a;
|
||||
a.repo(repo).repo_type(type).set("num", std::to_string(num));
|
||||
return a;
|
||||
}
|
||||
|
||||
PathArgs repo_args(RepoType type, const RepoRef& repo) {
|
||||
PathArgs a;
|
||||
a.repo(repo).repo_type(type);
|
||||
return a;
|
||||
}
|
||||
|
||||
void text_in(const std::string& what, const std::string& v, std::size_t min, std::size_t max) {
|
||||
if (v.size() < min || v.size() > max)
|
||||
bad(what + " must be " + std::to_string(min) + ".." + std::to_string(max) + " characters");
|
||||
}
|
||||
|
||||
PathArgs collection_args(const std::string& owner, const std::string& slug) {
|
||||
PathArgs a;
|
||||
a.set("namespace", owner).set("slug", slug);
|
||||
return a;
|
||||
}
|
||||
|
||||
const std::set<std::string>& item_types() {
|
||||
static const std::set<std::string> s = {"paper", "collection", "space", "model", "dataset", "bucket"};
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Json Discussions::list(RepoType type, const RepoRef& repo, const DiscussionFilter& f) {
|
||||
if (f.page < 0 || f.page > 100000) bad("page must be 0..100000");
|
||||
Query q;
|
||||
if (f.page > 0) q.add("p", static_cast<long>(f.page));
|
||||
add_if(q, "type", f.kind);
|
||||
add_if(q, "status", f.status);
|
||||
add_if(q, "author", f.author);
|
||||
add_if(q, "search", f.search);
|
||||
add_if(q, "sort", f.sort);
|
||||
return call(Call::kDiscussionList, repo_args(type, repo), q);
|
||||
}
|
||||
|
||||
Json Discussions::get(RepoType type, const RepoRef& repo, long num) {
|
||||
return call(Call::kDiscussionGet, discussion_args(type, repo, num));
|
||||
}
|
||||
|
||||
Json Discussions::create(RepoType type, const RepoRef& repo, const std::string& title,
|
||||
const std::string& description, bool pull_request) {
|
||||
text_in("title", title, 3, 200);
|
||||
text_in("description", description, 0, 65536);
|
||||
Json body = Json::object();
|
||||
body.set("title", Json(title));
|
||||
body.set("description", Json(description));
|
||||
if (pull_request) body.set("pullRequest", Json(true));
|
||||
return call(Call::kDiscussionCreate, repo_args(type, repo), Query(), &body);
|
||||
}
|
||||
|
||||
Json Discussions::comment(RepoType type, const RepoRef& repo, long num, const std::string& text) {
|
||||
text_in("comment", text, 1, 65536);
|
||||
Json body = Json::object();
|
||||
body.set("comment", Json(text));
|
||||
return call(Call::kDiscussionComment, discussion_args(type, repo, num), Query(), &body);
|
||||
}
|
||||
|
||||
Json Discussions::change_status(RepoType type, const RepoRef& repo, long num, const std::string& status,
|
||||
const std::string& comment) {
|
||||
if (status != "open" && status != "closed") bad("status must be open or closed");
|
||||
text_in("comment", comment, 0, 65536);
|
||||
Json body = Json::object();
|
||||
body.set("status", Json(status));
|
||||
if (!comment.empty()) body.set("comment", Json(comment));
|
||||
return call(Call::kDiscussionStatus, discussion_args(type, repo, num), Query(), &body);
|
||||
}
|
||||
|
||||
Json Discussions::rename(RepoType type, const RepoRef& repo, long num, const std::string& title) {
|
||||
text_in("title", title, 3, 200);
|
||||
Json body = Json::object();
|
||||
body.set("title", Json(title));
|
||||
return call(Call::kDiscussionTitle, discussion_args(type, repo, num), Query(), &body);
|
||||
}
|
||||
|
||||
Json Discussions::remove_discussion(RepoType type, const RepoRef& repo, long num) {
|
||||
return call(Call::kDiscussionDelete, discussion_args(type, repo, num));
|
||||
}
|
||||
|
||||
Json CollectionItem::to_json() const {
|
||||
if (!item_types().count(type)) bad("collection item type must be paper, collection, space, model, dataset or bucket");
|
||||
text_in("collection item id", id, 1, HfCaps::kMaxSegmentChars);
|
||||
Json o = Json::object();
|
||||
o.set("type", Json(type));
|
||||
o.set("id", Json(id));
|
||||
return o;
|
||||
}
|
||||
|
||||
Query CollectionListParams::to_query() const {
|
||||
if (limit < 0 || limit > HfCaps::kMaxPageLimit) bad("limit out of range");
|
||||
Query query;
|
||||
add_if(query, "owner", owner);
|
||||
add_if(query, "q", q);
|
||||
add_if(query, "sort", sort);
|
||||
if (limit > 0) query.add("limit", static_cast<long>(limit));
|
||||
if (!item.id.empty()) {
|
||||
item.to_json();
|
||||
query.add("item", item.type + "/" + item.id);
|
||||
}
|
||||
return query;
|
||||
}
|
||||
|
||||
Page Collections::list(const CollectionListParams& params, std::size_t max_items, std::size_t max_pages) {
|
||||
return walk(Call::kCollectionsList, PathArgs(), params.to_query(), max_items, max_pages);
|
||||
}
|
||||
|
||||
Json Collections::get(const std::string& owner, const std::string& slug) {
|
||||
return call(Call::kCollectionGet, collection_args(owner, slug));
|
||||
}
|
||||
|
||||
Json Collections::create(const std::string& owner, const std::string& title, const std::string& description,
|
||||
bool is_private) {
|
||||
if (!RequestPlan::valid_name(owner)) bad("bad owner");
|
||||
text_in("title", title, 1, 60);
|
||||
text_in("description", description, 0, 150);
|
||||
Json body = Json::object();
|
||||
body.set("title", Json(title));
|
||||
body.set("namespace", Json(owner));
|
||||
if (!description.empty()) body.set("description", Json(description));
|
||||
if (is_private) body.set("private", Json(true));
|
||||
return call(Call::kCollectionCreate, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json Collections::update(const std::string& owner, const std::string& slug, const Json& fields) {
|
||||
if (!fields.is_object() || fields.as_object().empty()) bad("fields must be a non-empty object");
|
||||
return call(Call::kCollectionUpdate, collection_args(owner, slug), Query(), &fields);
|
||||
}
|
||||
|
||||
Json Collections::add_item(const std::string& owner, const std::string& slug, const CollectionItem& item,
|
||||
const std::string& note) {
|
||||
text_in("note", note, 0, 500);
|
||||
Json body = Json::object();
|
||||
body.set("item", item.to_json());
|
||||
if (!note.empty()) body.set("note", Json(note));
|
||||
return call(Call::kCollectionAddItem, collection_args(owner, slug), Query(), &body);
|
||||
}
|
||||
|
||||
Json Collections::remove_collection(const std::string& owner, const std::string& slug) {
|
||||
return call(Call::kCollectionDelete, collection_args(owner, slug));
|
||||
}
|
||||
|
||||
Json WebhookSpec::to_json() const {
|
||||
static const std::set<std::string> kinds = {"dataset", "model", "space", "bucket", "kernel",
|
||||
"container", "user", "org"};
|
||||
if (url.size() < 8 || url.size() > HfCaps::kMaxServerUrlChars || url.rfind("http", 0) != 0)
|
||||
bad("webhook url must be an http(s) URL");
|
||||
if (watched.empty() || watched.size() > 64) bad("watched must hold 1..64 entries");
|
||||
if (domains.empty() || domains.size() > HfCaps::kMaxWebhookDomains) bad("domains must hold 1.. entries");
|
||||
Json w = Json::array();
|
||||
for (const Watched& x : watched) {
|
||||
if (!kinds.count(x.type)) bad("watched type must be dataset, model, space, bucket, kernel, container, user or org");
|
||||
if (x.name.empty() || x.name.size() > 256) bad("watched name is empty or too long");
|
||||
Json e = Json::object();
|
||||
e.set("type", Json(x.type));
|
||||
e.set("name", Json(x.name));
|
||||
w.push(e);
|
||||
}
|
||||
Json d = Json::array();
|
||||
for (const std::string& dom : domains) {
|
||||
if (dom != "repo" && dom != "discussion") bad("domain must be repo or discussion");
|
||||
d.push(Json(dom));
|
||||
}
|
||||
Json o = Json::object();
|
||||
o.set("url", Json(url));
|
||||
o.set("watched", w);
|
||||
o.set("domains", d);
|
||||
if (!secret.empty()) {
|
||||
for (unsigned char c : secret)
|
||||
if (c < 0x20 || c > 0x7f) bad("secret must be printable ASCII");
|
||||
o.set("secret", Json(secret));
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
Json Webhooks::list() { return call(Call::kWebhooksList); }
|
||||
|
||||
Json Webhooks::get(const std::string& id) {
|
||||
PathArgs a;
|
||||
a.set("webhookId", id);
|
||||
return call(Call::kWebhookGet, a);
|
||||
}
|
||||
|
||||
Json Webhooks::create(const WebhookSpec& spec) {
|
||||
Json body = spec.to_json();
|
||||
return call(Call::kWebhookCreate, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json Webhooks::update(const std::string& id, const WebhookSpec& spec) {
|
||||
Json body = spec.to_json();
|
||||
PathArgs a;
|
||||
a.set("webhookId", id);
|
||||
return call(Call::kWebhookUpdate, a, Query(), &body);
|
||||
}
|
||||
|
||||
Json Webhooks::enable(const std::string& id) {
|
||||
PathArgs a;
|
||||
a.set("webhookId", id).set("action", "enable");
|
||||
return call(Call::kWebhookToggle, a);
|
||||
}
|
||||
|
||||
Json Webhooks::disable(const std::string& id) {
|
||||
PathArgs a;
|
||||
a.set("webhookId", id).set("action", "disable");
|
||||
return call(Call::kWebhookToggle, a);
|
||||
}
|
||||
|
||||
Json Webhooks::replay(const std::string& id, const std::string& log_id) {
|
||||
PathArgs a;
|
||||
a.set("webhookId", id).set("logId", log_id);
|
||||
return call(Call::kWebhookReplay, a);
|
||||
}
|
||||
|
||||
Json Webhooks::remove_webhook(const std::string& id) {
|
||||
PathArgs a;
|
||||
a.set("webhookId", id);
|
||||
return call(Call::kWebhookDelete, a);
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+94
@@ -0,0 +1,94 @@
|
||||
#pragma once
|
||||
|
||||
// Discussions (and pull request threads), Collections and Webhooks on the Hub.
|
||||
// Reads are public where the Hub makes them public; every write needs a token
|
||||
// and is an explicit method. Deletes have no defaults and are named remove_*.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/huggingface/hf_service.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
struct DiscussionFilter {
|
||||
int page = 0; // p
|
||||
std::string kind; // all, discussion, pull_request
|
||||
std::string status; // all, open, closed
|
||||
std::string author;
|
||||
std::string search;
|
||||
std::string sort; // recently-created, trending, reactions
|
||||
};
|
||||
|
||||
class Discussions : public HfService {
|
||||
public:
|
||||
explicit Discussions(Access access) : HfService(std::move(access), Surface::kHub) {}
|
||||
Json list(RepoType type, const RepoRef& repo, const DiscussionFilter& filter = {});
|
||||
Json get(RepoType type, const RepoRef& repo, long num);
|
||||
|
||||
// Writes (token required).
|
||||
Json create(RepoType type, const RepoRef& repo, const std::string& title, const std::string& description,
|
||||
bool pull_request);
|
||||
Json comment(RepoType type, const RepoRef& repo, long num, const std::string& text);
|
||||
Json change_status(RepoType type, const RepoRef& repo, long num, const std::string& status,
|
||||
const std::string& comment = "");
|
||||
Json rename(RepoType type, const RepoRef& repo, long num, const std::string& title);
|
||||
Json remove_discussion(RepoType type, const RepoRef& repo, long num);
|
||||
};
|
||||
|
||||
struct CollectionItem {
|
||||
std::string type; // paper, collection, space, model, dataset, bucket
|
||||
std::string id;
|
||||
Json to_json() const; // validates
|
||||
};
|
||||
|
||||
struct CollectionListParams {
|
||||
std::string owner;
|
||||
std::string q;
|
||||
std::string sort;
|
||||
int limit = 0;
|
||||
CollectionItem item; // optional filter: collections containing this item (id empty: none)
|
||||
Query to_query() const;
|
||||
};
|
||||
|
||||
class Collections : public HfService {
|
||||
public:
|
||||
explicit Collections(Access access) : HfService(std::move(access), Surface::kHub) {}
|
||||
Page list(const CollectionListParams& params, std::size_t max_items = 100, std::size_t max_pages = 5);
|
||||
// slug is "name-<24 hex id>" exactly as the Hub prints it after the owner.
|
||||
Json get(const std::string& owner, const std::string& slug);
|
||||
|
||||
Json create(const std::string& owner, const std::string& title, const std::string& description,
|
||||
bool is_private);
|
||||
Json update(const std::string& owner, const std::string& slug, const Json& fields);
|
||||
Json add_item(const std::string& owner, const std::string& slug, const CollectionItem& item,
|
||||
const std::string& note = "");
|
||||
Json remove_collection(const std::string& owner, const std::string& slug);
|
||||
};
|
||||
|
||||
struct WebhookSpec {
|
||||
struct Watched {
|
||||
std::string type; // dataset, model, space, bucket, kernel, container, user, org
|
||||
std::string name;
|
||||
};
|
||||
std::string url;
|
||||
std::vector<Watched> watched;
|
||||
std::vector<std::string> domains; // repo, discussion
|
||||
std::string secret; // printable ASCII, optional
|
||||
Json to_json() const; // validates
|
||||
};
|
||||
|
||||
class Webhooks : public HfService {
|
||||
public:
|
||||
explicit Webhooks(Access access) : HfService(std::move(access), Surface::kHub) {}
|
||||
Json list();
|
||||
Json get(const std::string& id);
|
||||
Json create(const WebhookSpec& spec);
|
||||
Json update(const std::string& id, const WebhookSpec& spec);
|
||||
Json enable(const std::string& id);
|
||||
Json disable(const std::string& id);
|
||||
Json replay(const std::string& id, const std::string& log_id);
|
||||
Json remove_webhook(const std::string& id);
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
+235
@@ -0,0 +1,235 @@
|
||||
#include "vendors/huggingface/endpoints.h"
|
||||
|
||||
#include <cctype>
|
||||
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("huggingface", why); }
|
||||
|
||||
struct MetricRow {
|
||||
MetricKind kind;
|
||||
const char* name;
|
||||
Call call;
|
||||
};
|
||||
|
||||
const MetricRow kMetrics[] = {
|
||||
{MetricKind::kAll, "all", Call::kIeGetEndpointMetrics},
|
||||
{MetricKind::kHardwareCpu, "hardware-cpu", Call::kIeGetHardwareCpu},
|
||||
{MetricKind::kHardwareGpu, "hardware-gpu", Call::kIeGetHardwareGpu},
|
||||
{MetricKind::kItl, "itl", Call::kIeGetInferenceItl},
|
||||
{MetricKind::kKvCache, "kv-cache", Call::kIeGetInferenceKvCache},
|
||||
{MetricKind::kPrefixCache, "prefix-cache", Call::kIeGetInferencePrefixCache},
|
||||
{MetricKind::kRequests, "requests", Call::kIeGetInferenceRequests},
|
||||
{MetricKind::kTtft, "ttft", Call::kIeGetInferenceTtft},
|
||||
{MetricKind::kPendingRequest, "pending-request", Call::kIeGetPendingRequest},
|
||||
{MetricKind::kReplicaIds, "replica-ids", Call::kIeGetReplicaIds},
|
||||
{MetricKind::kReplicaRunning, "replica-running", Call::kIeGetReplicaRunning},
|
||||
{MetricKind::kReplicaStatus, "replica-status", Call::kIeGetReplicaStatus},
|
||||
{MetricKind::kResponseElapsed, "response-elapsed", Call::kIeGetResponseElapsed},
|
||||
{MetricKind::kResponseStatusCode, "response-status-code", Call::kIeGetResponseStatusCode},
|
||||
{MetricKind::kResponseStatusCodeGrouped, "response-status-code-grouped", Call::kIeGetStatusCodeGrouped},
|
||||
};
|
||||
|
||||
PathArgs ns_args(const std::string& ns) {
|
||||
PathArgs a;
|
||||
a.set("namespace", ns);
|
||||
return a;
|
||||
}
|
||||
|
||||
PathArgs ep_args(const std::string& ns, const std::string& name) {
|
||||
PathArgs a = ns_args(ns);
|
||||
a.set("name", name);
|
||||
return a;
|
||||
}
|
||||
|
||||
void check_limit(int limit, int max) {
|
||||
if (limit < 0 || limit > max) bad("limit must be 0.." + std::to_string(max));
|
||||
}
|
||||
|
||||
bool rfc3339_like(const std::string& s) {
|
||||
if (s.size() < 20 || s.size() > 40) return false;
|
||||
return std::isdigit(static_cast<unsigned char>(s[0])) && s[4] == '-' && s[7] == '-' && s[10] == 'T';
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const char* metric_kind_name(MetricKind kind) {
|
||||
for (const MetricRow& r : kMetrics)
|
||||
if (r.kind == kind) return r.name;
|
||||
return "all";
|
||||
}
|
||||
|
||||
bool parse_metric_kind(const std::string& text, MetricKind& out) {
|
||||
for (const MetricRow& r : kMetrics)
|
||||
if (text == r.name) {
|
||||
out = r.kind;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
void Endpoints::validate_name(const std::string& name) {
|
||||
if (name.empty() || name.size() > 32) bad("endpoint name must be 1..32 characters");
|
||||
for (char c : name)
|
||||
if (!(std::islower(static_cast<unsigned char>(c)) || std::isdigit(static_cast<unsigned char>(c)) || c == '-'))
|
||||
bad("endpoint name: lowercase letters, digits and - only");
|
||||
}
|
||||
|
||||
void Endpoints::validate_spec(const Json& spec, bool creating) {
|
||||
if (!spec.is_object()) bad("endpoint spec must be a JSON object");
|
||||
if (creating) {
|
||||
for (const char* k : {"name", "type", "provider", "compute", "model"})
|
||||
if (!spec.find(k)) bad(std::string("endpoint spec needs ") + k);
|
||||
const Json* n = spec.find("name");
|
||||
if (!n->is_string()) bad("name must be a string");
|
||||
validate_name(n->as_string());
|
||||
}
|
||||
if (const Json* t = spec.find("type")) {
|
||||
if (!t->is_string() || (t->as_string() != "public" && t->as_string() != "authenticated" &&
|
||||
t->as_string() != "private"))
|
||||
bad("type must be public, authenticated or private");
|
||||
}
|
||||
}
|
||||
|
||||
Json Endpoints::list(const std::string& ns, const EndpointListParams& p) {
|
||||
check_limit(p.limit, 100);
|
||||
Query q;
|
||||
if (!p.search.empty()) q.add("search", p.search);
|
||||
if (!p.tags.empty()) q.add("tags", p.tags);
|
||||
if (!p.type.empty()) q.add("type", p.type);
|
||||
if (!p.task.empty()) q.add("task", p.task);
|
||||
if (!p.state.empty()) q.add("state", p.state);
|
||||
if (!p.cursor.empty()) q.add("cursor", p.cursor);
|
||||
if (p.limit > 0) q.add("limit", static_cast<long>(p.limit));
|
||||
return call(Call::kIeListEndpoint, ns_args(ns), q);
|
||||
}
|
||||
|
||||
Json Endpoints::get(const std::string& ns, const std::string& name) {
|
||||
return call(Call::kIeGetEndpoint, ep_args(ns, name));
|
||||
}
|
||||
|
||||
Json Endpoints::replicas(const std::string& ns, const std::string& name) {
|
||||
return call(Call::kIeGetEndpointReplica, ep_args(ns, name));
|
||||
}
|
||||
|
||||
std::string Endpoints::logs(const std::string& ns, const std::string& name, int tail, const std::string& replica) {
|
||||
check_limit(tail, 1000);
|
||||
Query q;
|
||||
if (tail > 0) q.add("tail", static_cast<long>(tail));
|
||||
if (!replica.empty()) q.add("replica", replica);
|
||||
return call_raw(Call::kIeGetEndpointLogs, ep_args(ns, name), q).body;
|
||||
}
|
||||
|
||||
Json Endpoints::logs_v3(const std::string& ns, const std::string& name, int limit) {
|
||||
check_limit(limit, 1000);
|
||||
Query q;
|
||||
if (limit > 0) q.add("limit", static_cast<long>(limit));
|
||||
return call(Call::kIeGetEndpointLogsV3, ep_args(ns, name), q);
|
||||
}
|
||||
|
||||
Json Endpoints::metrics(MetricKind kind, const std::string& ns, const std::string& name,
|
||||
const std::string& start, const std::string& stop, const std::string& replica_id) {
|
||||
if (!rfc3339_like(start) || !rfc3339_like(stop)) bad("start and stop must be RFC 3339 (2021-01-01T00:00:00Z)");
|
||||
Json body = Json::object();
|
||||
body.set("start", Json(start));
|
||||
body.set("stop", Json(stop));
|
||||
if (!replica_id.empty()) body.set("replicaId", Json(replica_id));
|
||||
for (const MetricRow& r : kMetrics)
|
||||
if (r.kind == kind) return call(r.call, ep_args(ns, name), Query(), &body);
|
||||
bad("unknown metric kind");
|
||||
}
|
||||
|
||||
std::string Endpoints::open_metrics(const std::string& ns, const std::string& name) {
|
||||
return call_raw(Call::kIeGetEndpointOpenMetrics, ep_args(ns, name)).body;
|
||||
}
|
||||
|
||||
Json Endpoints::search(const std::string& ns, const std::string& text, int limit) {
|
||||
check_limit(limit, 100);
|
||||
Query q;
|
||||
if (!text.empty()) q.add("search", text);
|
||||
if (limit > 0) q.add("limit", static_cast<long>(limit));
|
||||
return call(Call::kIeSearchEndpoint, ns_args(ns), q);
|
||||
}
|
||||
|
||||
Json Endpoints::state_counts(const std::string& ns) { return call(Call::kIeCountEndpointStates, ns_args(ns)); }
|
||||
|
||||
Json Endpoints::tags(const std::string& ns, const std::string& prefix, int limit) {
|
||||
check_limit(limit, 100);
|
||||
Query q;
|
||||
if (!prefix.empty()) q.add("prefix", prefix);
|
||||
if (limit > 0) q.add("limit", static_cast<long>(limit));
|
||||
return call(Call::kIeListEndpointTags, ns_args(ns), q);
|
||||
}
|
||||
|
||||
Json Endpoints::vendors() { return call(Call::kIeListVendors); }
|
||||
Json Endpoints::vendors_for(const std::string& ns) { return call(Call::kIeListVendorsNamespace, ns_args(ns)); }
|
||||
Json Endpoints::quotas(const std::string& ns) { return call(Call::kIeListVendorsQuotasNamespace, ns_args(ns)); }
|
||||
|
||||
Json Endpoints::audit_logs(const std::string& ns, int limit) {
|
||||
check_limit(limit, 100);
|
||||
Query q;
|
||||
if (limit > 0) q.add("limit", static_cast<long>(limit));
|
||||
return call(Call::kIeGetAuditLogs, ns_args(ns), q);
|
||||
}
|
||||
|
||||
Json Endpoints::cidr_lists(const std::string& ns) { return call(Call::kIeListCidrsList, ns_args(ns)); }
|
||||
Json Endpoints::domains(const std::string& ns) { return call(Call::kIeListDomains, ns_args(ns)); }
|
||||
Json Endpoints::private_services(const std::string& ns) { return call(Call::kIeListPrivateServices, ns_args(ns)); }
|
||||
|
||||
Json Endpoints::create(const std::string& ns, const Json& spec) {
|
||||
validate_spec(spec, true);
|
||||
return call(Call::kIeCreateEndpoint, ns_args(ns), Query(), &spec);
|
||||
}
|
||||
|
||||
Json Endpoints::update(const std::string& ns, const std::string& name, const Json& spec) {
|
||||
validate_spec(spec, false);
|
||||
return call(Call::kIeUpdateEndpoint, ep_args(ns, name), Query(), &spec);
|
||||
}
|
||||
|
||||
Json Endpoints::pause(const std::string& ns, const std::string& name) {
|
||||
return call(Call::kIePauseEndpoint, ep_args(ns, name));
|
||||
}
|
||||
Json Endpoints::resume(const std::string& ns, const std::string& name) {
|
||||
return call(Call::kIeResumeEndpoint, ep_args(ns, name));
|
||||
}
|
||||
Json Endpoints::scale_to_zero(const std::string& ns, const std::string& name) {
|
||||
return call(Call::kIeScaleToZeroEndpoint, ep_args(ns, name));
|
||||
}
|
||||
Json Endpoints::remove_endpoint(const std::string& ns, const std::string& name) {
|
||||
return call(Call::kIeDeleteEndpoint, ep_args(ns, name));
|
||||
}
|
||||
|
||||
Json Endpoints::create_cidr_list(const std::string& ns, const Json& spec) {
|
||||
return call(Call::kIeCreateCidrList, ns_args(ns), Query(), &spec);
|
||||
}
|
||||
|
||||
Json Endpoints::remove_cidr_list(const std::string& ns, const std::string& id) {
|
||||
PathArgs a = ns_args(ns);
|
||||
a.set("cidr_id", id);
|
||||
return call(Call::kIeDeleteCidrList, a);
|
||||
}
|
||||
|
||||
Json Endpoints::create_domain(const std::string& ns, const std::string& domain) {
|
||||
if (domain.empty() || domain.size() > 253) bad("domain is empty or too long");
|
||||
Json body = Json::object();
|
||||
body.set("domain", Json(domain));
|
||||
return call(Call::kIeCreateDomain, ns_args(ns), Query(), &body);
|
||||
}
|
||||
|
||||
Json Endpoints::remove_domain(const std::string& ns, const std::string& domain) {
|
||||
PathArgs a = ns_args(ns);
|
||||
a.set("domain", domain);
|
||||
return call(Call::kIeDeleteDomain, a);
|
||||
}
|
||||
|
||||
Json Endpoints::remove_private_service(const std::string& ns, const std::string& id) {
|
||||
PathArgs a = ns_args(ns);
|
||||
a.set("service-id", id);
|
||||
return call(Call::kIeDeletePrivateService, a);
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+79
@@ -0,0 +1,79 @@
|
||||
#pragma once
|
||||
|
||||
// Inference Endpoints management API (https://api.endpoints.huggingface.cloud,
|
||||
// OpenAPI 2.0.0): endpoints (list, get, create, update, pause, resume, scale
|
||||
// to zero, delete), logs, metrics, replicas, providers and quotas, search,
|
||||
// network (CIDR lists, domains, private services), audit logs.
|
||||
// Creating an endpoint costs money; deleting, pausing and scaling to zero
|
||||
// disrupt service. All of those are explicit methods and the demo only runs
|
||||
// them with "confirm":true. The two GET SSE streams are listed in the call
|
||||
// table as unsupported (ApiClient.stream is POST only).
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vendors/huggingface/hf_service.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
enum class MetricKind {
|
||||
kAll, kHardwareCpu, kHardwareGpu, kItl, kKvCache, kPrefixCache, kRequests, kTtft,
|
||||
kPendingRequest, kReplicaIds, kReplicaRunning, kReplicaStatus, kResponseElapsed,
|
||||
kResponseStatusCode, kResponseStatusCodeGrouped,
|
||||
};
|
||||
|
||||
const char* metric_kind_name(MetricKind kind);
|
||||
bool parse_metric_kind(const std::string& text, MetricKind& out);
|
||||
|
||||
struct EndpointListParams {
|
||||
std::string search;
|
||||
std::string tags;
|
||||
std::string type; // public, authenticated, private
|
||||
std::string task;
|
||||
std::string state;
|
||||
std::string cursor;
|
||||
int limit = 0; // 1..100
|
||||
};
|
||||
|
||||
class Endpoints : public HfService {
|
||||
public:
|
||||
explicit Endpoints(Access access) : HfService(std::move(access), Surface::kEndpoints) {}
|
||||
static Endpoints from_platform(const Platform& p) { return Endpoints(p.endpoints); }
|
||||
|
||||
// Reads.
|
||||
Json list(const std::string& ns, const EndpointListParams& params = {});
|
||||
Json get(const std::string& ns, const std::string& name);
|
||||
Json replicas(const std::string& ns, const std::string& name);
|
||||
std::string logs(const std::string& ns, const std::string& name, int tail = 100, const std::string& replica = "");
|
||||
Json logs_v3(const std::string& ns, const std::string& name, int limit = 100);
|
||||
Json metrics(MetricKind kind, const std::string& ns, const std::string& name, const std::string& start,
|
||||
const std::string& stop, const std::string& replica_id = "");
|
||||
std::string open_metrics(const std::string& ns, const std::string& name);
|
||||
Json search(const std::string& ns, const std::string& text, int limit = 20);
|
||||
Json state_counts(const std::string& ns);
|
||||
Json tags(const std::string& ns, const std::string& prefix = "", int limit = 50);
|
||||
Json vendors(); // public, no token
|
||||
Json vendors_for(const std::string& ns);
|
||||
Json quotas(const std::string& ns);
|
||||
Json audit_logs(const std::string& ns, int limit = 50);
|
||||
Json cidr_lists(const std::string& ns);
|
||||
Json domains(const std::string& ns);
|
||||
Json private_services(const std::string& ns);
|
||||
|
||||
// Writes. create() costs money while the endpoint runs.
|
||||
Json create(const std::string& ns, const Json& spec);
|
||||
Json update(const std::string& ns, const std::string& name, const Json& spec);
|
||||
Json pause(const std::string& ns, const std::string& name);
|
||||
Json resume(const std::string& ns, const std::string& name);
|
||||
Json scale_to_zero(const std::string& ns, const std::string& name);
|
||||
Json remove_endpoint(const std::string& ns, const std::string& name);
|
||||
Json create_cidr_list(const std::string& ns, const Json& spec);
|
||||
Json remove_cidr_list(const std::string& ns, const std::string& id);
|
||||
Json create_domain(const std::string& ns, const std::string& domain);
|
||||
Json remove_domain(const std::string& ns, const std::string& domain);
|
||||
Json remove_private_service(const std::string& ns, const std::string& id);
|
||||
|
||||
static void validate_name(const std::string& name); // lowercase alnum and '-', 1..32
|
||||
static void validate_spec(const Json& spec, bool creating);
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
#include "vendors/huggingface/hf_service.h"
|
||||
|
||||
#include <cstring>
|
||||
|
||||
#include "clients/url.h"
|
||||
#include "vendors/huggingface/caps.h"
|
||||
#include "llm/caps.h"
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
HfService::HfService(Access access, Surface surface)
|
||||
: llm::Service(access.client), access_(std::move(access)), surface_(surface) {}
|
||||
|
||||
const CallSpec& HfService::checked(Call call, bool) const {
|
||||
const CallSpec& spec = Calls::spec_of(call);
|
||||
if (spec.surface != surface_)
|
||||
throw llm::ApiError::invalid("huggingface", std::string("call ") + spec.path + " belongs to the " +
|
||||
surface_name(spec.surface) + " surface");
|
||||
if (!std::strcmp(spec.auth, "token") && !access_.has_token)
|
||||
throw llm::NotConfiguredError("huggingface", Platform::kKeyEnv);
|
||||
if (access_.budget) access_.budget->spend();
|
||||
return spec;
|
||||
}
|
||||
|
||||
Json HfService::call(Call c, const PathArgs& args, const Query& query, const Json* body,
|
||||
const llm::Headers& extra) {
|
||||
checked(c);
|
||||
Plan plan = RequestPlan::make(c, args, query, body);
|
||||
return client().request(plan.method, plan.target, body, extra);
|
||||
}
|
||||
|
||||
llm::RawResponse HfService::call_raw(Call c, const PathArgs& args, const Query& query, const Json* body,
|
||||
const llm::Headers& extra) {
|
||||
checked(c);
|
||||
Plan plan = RequestPlan::make(c, args, query, body);
|
||||
std::string text;
|
||||
if (body) text = body->dump();
|
||||
return client().request_raw(plan.method, plan.target, body ? &text : nullptr,
|
||||
body ? "application/json" : "", extra);
|
||||
}
|
||||
|
||||
llm::RawResponse HfService::call_bytes(Call c, const PathArgs& args, const std::string& body,
|
||||
const std::string& content_type, const llm::Headers& extra) {
|
||||
const CallSpec& spec = checked(c);
|
||||
if (!spec.has_body()) throw llm::ApiError::invalid("huggingface", "this call takes no body");
|
||||
if (body.size() > HfCaps::kMaxInferenceInputBytes)
|
||||
throw llm::ApiError::invalid("huggingface", "input is over the byte cap");
|
||||
Plan plan = RequestPlan::make(c, args, Query());
|
||||
return client().request_raw(plan.method, plan.target, &body, content_type, extra);
|
||||
}
|
||||
|
||||
std::string next_cursor_of(const std::string& link, const std::string& expected_host) {
|
||||
// <https://huggingface.co/api/models?limit=2&cursor=...>; rel="next"
|
||||
std::size_t pos = 0;
|
||||
while (pos < link.size()) {
|
||||
std::size_t lt = link.find('<', pos);
|
||||
if (lt == std::string::npos) return "";
|
||||
std::size_t gt = link.find('>', lt);
|
||||
if (gt == std::string::npos) return "";
|
||||
std::size_t next_entry = link.find('<', gt);
|
||||
std::string params = link.substr(gt + 1, (next_entry == std::string::npos ? link.size() : next_entry) - gt - 1);
|
||||
if (params.find("rel=\"next\"") != std::string::npos) {
|
||||
clients::Url url = clients::Url::parse(link.substr(lt + 1, gt - lt - 1));
|
||||
if (url.host() != expected_host)
|
||||
throw llm::ApiError("huggingface", 502, "BAD_RESPONSE", "Link header points at another host");
|
||||
auto cur = url.query_value("cursor");
|
||||
return cur ? *cur : "";
|
||||
}
|
||||
if (next_entry == std::string::npos) break;
|
||||
pos = next_entry;
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
Page HfService::walk(Call c, const PathArgs& args, const Query& query, std::size_t max_items,
|
||||
std::size_t max_pages) {
|
||||
const CallSpec& spec = Calls::spec_of(c);
|
||||
if (std::strcmp(spec.http_method, "GET") != 0)
|
||||
throw llm::ApiError::invalid("huggingface", "only GET lists can be walked");
|
||||
if (max_items == 0 || max_items > HfCaps::kMaxItems) max_items = HfCaps::kMaxItems;
|
||||
if (max_pages == 0 || max_pages > HfCaps::kMaxPages) max_pages = HfCaps::kMaxPages;
|
||||
clients::Url base = clients::Url::parse(client().config().base_url);
|
||||
Page page;
|
||||
Query q = query;
|
||||
while (true) {
|
||||
checked(c);
|
||||
Plan plan = RequestPlan::make(c, args, q);
|
||||
llm::RawResponse raw = client().request_raw(plan.method, plan.target);
|
||||
Json doc;
|
||||
try {
|
||||
doc = Json::parse(raw.body);
|
||||
} catch (const std::exception& e) {
|
||||
throw llm::ApiError("huggingface", 502, "BAD_RESPONSE", e.what());
|
||||
}
|
||||
if (!doc.is_array()) throw llm::ApiError("huggingface", 502, "BAD_RESPONSE", "list reply is not an array");
|
||||
++page.pages;
|
||||
std::string cursor = next_cursor_of(raw.headers.get_or("link", ""), base.host());
|
||||
for (const Json& item : doc.as_array()) {
|
||||
if (page.items.as_array().size() >= max_items) {
|
||||
page.truncated = true;
|
||||
page.truncated_by = "items";
|
||||
page.next_cursor = "";
|
||||
return page;
|
||||
}
|
||||
page.items.push(item);
|
||||
}
|
||||
if (cursor.empty()) return page;
|
||||
if (page.pages >= max_pages) {
|
||||
page.truncated = true;
|
||||
page.truncated_by = "pages";
|
||||
page.next_cursor = cursor;
|
||||
return page;
|
||||
}
|
||||
if (!Calls::in_list(spec.query_params, "cursor"))
|
||||
throw llm::ApiError("huggingface", 502, "BAD_RESPONSE", "server paged a call that has no cursor");
|
||||
q.replace("cursor", cursor);
|
||||
}
|
||||
}
|
||||
|
||||
void HfService::stream(Call c, const PathArgs& args, const Json& body, const llm::SseDecoder::Sink& sink,
|
||||
const llm::Headers& extra) {
|
||||
const CallSpec& spec = checked(c);
|
||||
if (std::strcmp(spec.http_method, "POST") != 0)
|
||||
throw llm::ApiError::invalid("huggingface", "streaming is POST only (llm gap)");
|
||||
Plan plan = RequestPlan::make(c, args, Query(), &body);
|
||||
std::size_t events = 0;
|
||||
client().stream(plan.target, body,
|
||||
[&](const llm::SseEvent& e) {
|
||||
if (++events > HfCaps::kMaxStreamEvents) return false;
|
||||
return sink(e);
|
||||
},
|
||||
extra);
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+61
@@ -0,0 +1,61 @@
|
||||
#pragma once
|
||||
|
||||
// HfService: base of every service in this package. One method per kind of
|
||||
// call (JSON, raw bytes, JSON body to raw reply, page walk, SSE), all driven by
|
||||
// the call table through RequestPlan. Enforces: the call belongs to this
|
||||
// service's surface, a token exists when the row demands one, the shared
|
||||
// request budget, destructive/write rows only through the explicit methods
|
||||
// that pass `allow_write`.
|
||||
|
||||
#include <functional>
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "vendors/huggingface/calls.h"
|
||||
#include "vendors/huggingface/platform.h"
|
||||
#include "vendors/huggingface/request_plan.h"
|
||||
#include "llm/api_client.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
using Json = clients::body::Json;
|
||||
|
||||
// One walked list: items plus the loud truncation report.
|
||||
struct Page {
|
||||
Json items = Json::array();
|
||||
bool truncated = false; // stopped by a cap while the server had more
|
||||
std::size_t pages = 0;
|
||||
std::string next_cursor; // set when truncated: resume point
|
||||
std::string truncated_by; // "items", "pages" or ""
|
||||
};
|
||||
|
||||
class HfService : public llm::Service {
|
||||
public:
|
||||
HfService(Access access, Surface surface);
|
||||
Surface surface() const { return surface_; }
|
||||
bool has_token() const { return access_.has_token; }
|
||||
|
||||
protected:
|
||||
Json call(Call call, const PathArgs& args = {}, const Query& query = {}, const Json* body = nullptr,
|
||||
const llm::Headers& extra = {});
|
||||
llm::RawResponse call_raw(Call call, const PathArgs& args = {}, const Query& query = {},
|
||||
const Json* body = nullptr, const llm::Headers& extra = {});
|
||||
// Raw request body (binary inference inputs).
|
||||
llm::RawResponse call_bytes(Call call, const PathArgs& args, const std::string& body,
|
||||
const std::string& content_type, const llm::Headers& extra = {});
|
||||
// GET list that follows Link: rel="next" cursors up to the caps.
|
||||
Page walk(Call call, const PathArgs& args, const Query& query, std::size_t max_items,
|
||||
std::size_t max_pages);
|
||||
void stream(Call call, const PathArgs& args, const Json& body, const llm::SseDecoder::Sink& sink,
|
||||
const llm::Headers& extra = {});
|
||||
|
||||
const CallSpec& checked(Call call, bool body_expected = false) const;
|
||||
|
||||
Access access_;
|
||||
Surface surface_;
|
||||
};
|
||||
|
||||
// The cursor in a Link header's rel="next" target, or empty.
|
||||
std::string next_cursor_of(const std::string& link_header, const std::string& expected_host);
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+244
@@ -0,0 +1,244 @@
|
||||
#include "vendors/huggingface/hub.h"
|
||||
|
||||
#include "vendors/huggingface/caps.h"
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("huggingface", why); }
|
||||
|
||||
void check_limit(int limit) {
|
||||
if (limit < 0 || limit > HfCaps::kMaxPageLimit) bad("limit must be 0.." + std::to_string(HfCaps::kMaxPageLimit));
|
||||
}
|
||||
|
||||
void add_if(Query& q, const char* name, const std::string& value) {
|
||||
if (!value.empty()) q.add(name, value);
|
||||
}
|
||||
|
||||
Call info_call(RepoType t, bool revision) {
|
||||
switch (t) {
|
||||
case RepoType::kModel: return revision ? Call::kModelRevision : Call::kModelInfo;
|
||||
case RepoType::kDataset: return revision ? Call::kDatasetRevision : Call::kDatasetInfo;
|
||||
case RepoType::kSpace: return revision ? Call::kSpaceRevision : Call::kSpaceInfo;
|
||||
}
|
||||
return Call::kModelInfo;
|
||||
}
|
||||
|
||||
PathArgs repo_args(RepoType type, const RepoRef& ref) {
|
||||
PathArgs a;
|
||||
a.repo(ref).repo_type(type);
|
||||
return a;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Query RepoListParams::to_query() const {
|
||||
check_limit(limit);
|
||||
Query q;
|
||||
add_if(q, "search", search);
|
||||
add_if(q, "author", author);
|
||||
add_if(q, "sort", sort);
|
||||
add_if(q, "direction", direction);
|
||||
add_if(q, "pipeline_tag", pipeline_tag);
|
||||
add_if(q, "inference_provider", inference_provider);
|
||||
for (const std::string& f : filter) q.add("filter", f);
|
||||
for (const std::string& e : expand) q.add("expand", e);
|
||||
if (limit > 0) q.add("limit", static_cast<long>(limit));
|
||||
if (full) q.flag("full", true);
|
||||
return q;
|
||||
}
|
||||
|
||||
Call Hub::list_call(RepoType t) {
|
||||
switch (t) {
|
||||
case RepoType::kModel: return Call::kModelsList;
|
||||
case RepoType::kDataset: return Call::kDatasetsList;
|
||||
case RepoType::kSpace: return Call::kSpacesList;
|
||||
}
|
||||
return Call::kModelsList;
|
||||
}
|
||||
|
||||
Json Hub::whoami() { return call(Call::kWhoami); }
|
||||
|
||||
Page Hub::list_repos(RepoType type, const RepoListParams& params, std::size_t max_items,
|
||||
std::size_t max_pages) {
|
||||
return walk(list_call(type), PathArgs(), params.to_query(), max_items, max_pages);
|
||||
}
|
||||
|
||||
Json Hub::repo_info(RepoType type, const RepoRef& ref, const std::string& revision,
|
||||
const std::vector<std::string>& expand) {
|
||||
PathArgs a;
|
||||
a.set("namespace", ref.owner()).set("repo", ref.name());
|
||||
if (!revision.empty()) a.set("rev", revision);
|
||||
Query q;
|
||||
for (const std::string& e : expand) q.add("expand", e);
|
||||
return call(info_call(type, !revision.empty()), a, q);
|
||||
}
|
||||
|
||||
Json Hub::inference_provider_mapping(const RepoRef& model) {
|
||||
return repo_info(RepoType::kModel, model, "", {"inferenceProviderMapping"});
|
||||
}
|
||||
|
||||
Json Hub::tree(RepoType type, const RepoRef& ref, const std::string& rev, const std::string& path,
|
||||
bool recursive, int limit) {
|
||||
check_limit(limit);
|
||||
PathArgs a = repo_args(type, ref);
|
||||
a.set("rev", rev).set("path", path);
|
||||
Query q;
|
||||
if (recursive) q.flag("recursive", true);
|
||||
if (limit > 0) q.add("limit", static_cast<long>(limit));
|
||||
return call(Call::kRepoTree, a, q);
|
||||
}
|
||||
|
||||
Page Hub::tree_all(RepoType type, const RepoRef& ref, const std::string& rev, const std::string& path,
|
||||
bool recursive, std::size_t max_items, std::size_t max_pages) {
|
||||
PathArgs a = repo_args(type, ref);
|
||||
a.set("rev", rev).set("path", path);
|
||||
Query q;
|
||||
if (recursive) q.flag("recursive", true);
|
||||
return walk(Call::kRepoTree, a, q, max_items, max_pages);
|
||||
}
|
||||
|
||||
Json Hub::refs(RepoType type, const RepoRef& ref, bool include_prs) {
|
||||
Query q;
|
||||
if (include_prs) q.flag("include_prs", true);
|
||||
return call(Call::kRepoRefs, repo_args(type, ref), q);
|
||||
}
|
||||
|
||||
Json Hub::commits(RepoType type, const RepoRef& ref, const std::string& rev, int limit) {
|
||||
check_limit(limit);
|
||||
PathArgs a = repo_args(type, ref);
|
||||
a.set("rev", rev);
|
||||
Query q;
|
||||
if (limit > 0) q.add("limit", static_cast<long>(limit));
|
||||
return call(Call::kRepoCommits, a, q);
|
||||
}
|
||||
|
||||
Json Hub::compare(RepoType type, const RepoRef& ref, const std::string& cmp) {
|
||||
PathArgs a = repo_args(type, ref);
|
||||
a.set("compare", cmp);
|
||||
return call(Call::kRepoCompare, a);
|
||||
}
|
||||
|
||||
Json Hub::paths_info(RepoType type, const RepoRef& ref, const std::string& rev,
|
||||
const std::vector<std::string>& paths) {
|
||||
if (paths.empty() || paths.size() > HfCaps::kMaxPathsInfo)
|
||||
bad("paths must hold 1.." + std::to_string(HfCaps::kMaxPathsInfo) + " entries");
|
||||
Json arr = Json::array();
|
||||
for (const std::string& p : paths) {
|
||||
RequestPlan::encode_file_path(p); // validates
|
||||
arr.push(Json(p));
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("paths", arr);
|
||||
PathArgs a = repo_args(type, ref);
|
||||
a.set("rev", rev);
|
||||
return call(Call::kRepoPathsInfo, a, Query(), &body);
|
||||
}
|
||||
|
||||
Json Hub::likers(RepoType type, const RepoRef& ref) { return call(Call::kRepoLikers, repo_args(type, ref)); }
|
||||
|
||||
Json Hub::safetensors_metadata(const RepoRef& model, const std::string& path) {
|
||||
PathArgs a;
|
||||
a.repo(model).set("path", path);
|
||||
return call(Call::kModelSafetensors, a);
|
||||
}
|
||||
|
||||
Download Hub::download(RepoType type, const RepoRef& ref, const std::string& rev, const std::string& path) {
|
||||
PathArgs a = repo_args(type, ref);
|
||||
a.set("rev", rev).set("path", path);
|
||||
llm::RawResponse raw = call_raw(Call::kFileResolve, a);
|
||||
Download d;
|
||||
d.bytes = std::move(raw.body);
|
||||
d.content_type = raw.content_type;
|
||||
d.commit = raw.headers.get_or("x-repo-commit", "");
|
||||
d.etag = raw.headers.get_or("etag", "");
|
||||
return d;
|
||||
}
|
||||
|
||||
Json Hub::quicksearch(const std::string& q, int limit, const std::string& type) {
|
||||
check_limit(limit);
|
||||
if (q.empty()) bad("q must not be empty");
|
||||
Query query;
|
||||
query.add("q", q);
|
||||
if (limit > 0) query.add("limit", static_cast<long>(limit));
|
||||
add_if(query, "type", type);
|
||||
return call(Call::kQuicksearch, PathArgs(), query);
|
||||
}
|
||||
|
||||
Json Hub::full_text_search(const std::string& q, int limit, const std::string& type) {
|
||||
check_limit(limit);
|
||||
if (q.empty()) bad("q must not be empty");
|
||||
Query query;
|
||||
query.add("q", q);
|
||||
if (limit > 0) query.add("limit", static_cast<long>(limit));
|
||||
add_if(query, "type", type);
|
||||
return call(Call::kFullTextSearch, PathArgs(), query);
|
||||
}
|
||||
|
||||
Json Hub::trending(const std::string& type, int limit) {
|
||||
check_limit(limit);
|
||||
Query q;
|
||||
add_if(q, "type", type);
|
||||
if (limit > 0) q.add("limit", static_cast<long>(limit));
|
||||
return call(Call::kTrending, PathArgs(), q);
|
||||
}
|
||||
|
||||
Json Hub::tags_by_type(RepoType type, const std::string& tag_type) {
|
||||
Query q;
|
||||
add_if(q, "type", tag_type);
|
||||
if (type == RepoType::kModel) return call(Call::kModelsTagsByType, PathArgs(), q);
|
||||
if (type == RepoType::kDataset) return call(Call::kDatasetsTagsByType, PathArgs(), q);
|
||||
bad("tags exist for models and datasets only");
|
||||
}
|
||||
|
||||
Json Hub::tasks() { return call(Call::kTasks); }
|
||||
Json Hub::spaces_hardware() { return call(Call::kSpacesHardware); }
|
||||
Json Hub::spaces_featured() { return call(Call::kSpacesFeatured); }
|
||||
|
||||
Json Hub::spaces_semantic_search(const std::string& q) {
|
||||
if (q.empty()) bad("q must not be empty");
|
||||
Query query;
|
||||
query.add("q", q);
|
||||
return call(Call::kSpacesSemanticSearch, PathArgs(), query);
|
||||
}
|
||||
|
||||
Json Hub::space_runtime(const RepoRef& space) {
|
||||
PathArgs a;
|
||||
a.repo(space);
|
||||
return call(Call::kSpaceRuntime, a);
|
||||
}
|
||||
|
||||
Json Hub::daily_papers(int limit, const std::string& date) {
|
||||
check_limit(limit);
|
||||
Query q;
|
||||
if (limit > 0) q.add("limit", static_cast<long>(limit));
|
||||
add_if(q, "date", date);
|
||||
return call(Call::kDailyPapers, PathArgs(), q);
|
||||
}
|
||||
|
||||
Json Hub::papers_search(const std::string& q, int limit) {
|
||||
check_limit(limit);
|
||||
if (q.empty()) bad("q must not be empty");
|
||||
Query query;
|
||||
query.add("q", q);
|
||||
if (limit > 0) query.add("limit", static_cast<long>(limit));
|
||||
return call(Call::kPapersSearch, PathArgs(), query);
|
||||
}
|
||||
|
||||
Json Hub::paper(const std::string& paper_id) {
|
||||
PathArgs a;
|
||||
a.set("paperId", paper_id);
|
||||
return call(Call::kPaperGet, a);
|
||||
}
|
||||
|
||||
Json Hub::partner_models(const std::string& provider, const std::string& status) {
|
||||
PathArgs a;
|
||||
a.set("provider", provider);
|
||||
Query q;
|
||||
add_if(q, "status", status);
|
||||
return call(Call::kPartnerModels, a, q);
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+93
@@ -0,0 +1,93 @@
|
||||
#pragma once
|
||||
|
||||
// Hub reads over https://huggingface.co/api: models, datasets, spaces, repo
|
||||
// trees / refs / commits / files, search, trending, tags, papers, Spaces
|
||||
// catalogs, Inference Provider mappings, whoami, and file download.
|
||||
// Public reads work without a token (Platform builds an anonymous client).
|
||||
// Writes to repositories (commit, upload, delete, settings, branches, LFS) are
|
||||
// outside this module; see SPEC.md ## Scope.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/huggingface/hf_service.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
// Query for GET /api/{models,datasets,spaces}. Fields a repo type does not
|
||||
// accept are refused by the call table, loudly.
|
||||
struct RepoListParams {
|
||||
std::string search;
|
||||
std::string author;
|
||||
std::string sort; // downloads, likes, trendingScore, createdAt, lastModified, ...
|
||||
std::string direction; // "-1" descending
|
||||
std::string pipeline_tag; // models only
|
||||
std::string inference_provider; // models only (provider name or "all")
|
||||
std::vector<std::string> filter;
|
||||
std::vector<std::string> expand; // repeated expand=
|
||||
int limit = 0; // page size, 0: server default; max HfCaps::kMaxPageLimit
|
||||
bool full = false;
|
||||
|
||||
Query to_query() const;
|
||||
};
|
||||
|
||||
struct Download {
|
||||
std::string bytes;
|
||||
std::string content_type;
|
||||
std::string commit; // X-Repo-Commit when sent
|
||||
std::string etag;
|
||||
};
|
||||
|
||||
class Hub : public HfService {
|
||||
public:
|
||||
explicit Hub(Access access) : HfService(std::move(access), Surface::kHub) {}
|
||||
static Hub from_platform(const Platform& p) { return Hub(p.hub); }
|
||||
|
||||
// The call table row that lists this repo type.
|
||||
static Call list_call(RepoType type);
|
||||
|
||||
Json whoami(); // needs a token
|
||||
|
||||
// Lists (page walks, truncation reported).
|
||||
Page list_repos(RepoType type, const RepoListParams& params, std::size_t max_items = 100,
|
||||
std::size_t max_pages = 5);
|
||||
Json repo_info(RepoType type, const RepoRef& ref, const std::string& revision = "",
|
||||
const std::vector<std::string>& expand = {});
|
||||
Json inference_provider_mapping(const RepoRef& model); // model info expanded to the mapping
|
||||
|
||||
// Repo contents.
|
||||
Json tree(RepoType type, const RepoRef& ref, const std::string& rev, const std::string& path = "",
|
||||
bool recursive = false, int limit = 0);
|
||||
Page tree_all(RepoType type, const RepoRef& ref, const std::string& rev, const std::string& path,
|
||||
bool recursive, std::size_t max_items = 1000, std::size_t max_pages = 10);
|
||||
Json refs(RepoType type, const RepoRef& ref, bool include_prs = false);
|
||||
Json commits(RepoType type, const RepoRef& ref, const std::string& rev, int limit = 0);
|
||||
Json compare(RepoType type, const RepoRef& ref, const std::string& compare);
|
||||
Json paths_info(RepoType type, const RepoRef& ref, const std::string& rev,
|
||||
const std::vector<std::string>& paths);
|
||||
Json likers(RepoType type, const RepoRef& ref);
|
||||
Json safetensors_metadata(const RepoRef& model, const std::string& path);
|
||||
// File bytes through /{repo}/resolve/{rev}/{path}; over the byte cap fails loudly.
|
||||
Download download(RepoType type, const RepoRef& ref, const std::string& rev, const std::string& path);
|
||||
|
||||
// Search and catalogs.
|
||||
Json quicksearch(const std::string& q, int limit = 10, const std::string& type = "");
|
||||
Json full_text_search(const std::string& q, int limit = 10, const std::string& type = "");
|
||||
Json trending(const std::string& type = "", int limit = 0);
|
||||
Json tags_by_type(RepoType type, const std::string& tag_type = ""); // models or datasets
|
||||
Json tasks();
|
||||
Json spaces_hardware();
|
||||
Json spaces_featured();
|
||||
Json spaces_semantic_search(const std::string& q);
|
||||
Json space_runtime(const RepoRef& space); // needs a token
|
||||
|
||||
// Papers.
|
||||
Json daily_papers(int limit = 0, const std::string& date = "");
|
||||
Json papers_search(const std::string& q, int limit = 0);
|
||||
Json paper(const std::string& paper_id);
|
||||
|
||||
// Inference Providers catalog: which models a provider serves.
|
||||
Json partner_models(const std::string& provider, const std::string& status = "");
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+130
@@ -0,0 +1,130 @@
|
||||
#include "vendors/huggingface/platform.h"
|
||||
|
||||
#include <chrono>
|
||||
|
||||
#include "clients/transports/http_transport.h"
|
||||
#include "clients/url.h"
|
||||
#include "vendors/huggingface/caps.h"
|
||||
#include "llm/config.h"
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
RequestBudget::RequestBudget(int max_calls, int window_seconds, Clock clock)
|
||||
: max_calls_(max_calls), window_(window_seconds), clock_(std::move(clock)) {
|
||||
if (!clock_) {
|
||||
clock_ = [] {
|
||||
return static_cast<long>(std::chrono::duration_cast<std::chrono::seconds>(
|
||||
std::chrono::steady_clock::now().time_since_epoch())
|
||||
.count());
|
||||
};
|
||||
}
|
||||
window_start_ = clock_();
|
||||
}
|
||||
|
||||
void RequestBudget::spend() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const long now = clock_();
|
||||
if (now - window_start_ >= window_) {
|
||||
window_start_ = now;
|
||||
used_ = 0;
|
||||
}
|
||||
if (used_ >= max_calls_)
|
||||
throw llm::ApiError("huggingface", 429, "LOCAL_BUDGET",
|
||||
"local request budget spent (" + std::to_string(max_calls_) + " calls per " +
|
||||
std::to_string(window_) + " s); wait for the next window");
|
||||
++used_;
|
||||
}
|
||||
|
||||
int RequestBudget::used() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return used_;
|
||||
}
|
||||
|
||||
std::string Platform::token_from_env() {
|
||||
std::string t = llm::Config::get(kKeyEnv);
|
||||
if (t.empty()) t = llm::Config::get(kLegacyKeyEnv);
|
||||
return t;
|
||||
}
|
||||
|
||||
llm::ApiClientConfig Platform::config(const std::string& base_url, bool has_token) {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = "huggingface";
|
||||
c.base_url = base_url;
|
||||
c.key_env = kKeyEnv;
|
||||
if (has_token) {
|
||||
c.auth = llm::AuthStyle::kBearer;
|
||||
} else {
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = "X-Molo-Anonymous"; // marker, not a credential
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
Access make_access(std::unique_ptr<clients::Transport> transport, const std::string& base_url,
|
||||
const std::string& token, std::shared_ptr<RequestBudget> budget) {
|
||||
Access a;
|
||||
a.has_token = !token.empty();
|
||||
a.client = std::make_shared<llm::ApiClient>(std::move(transport), Platform::config(base_url, a.has_token),
|
||||
a.has_token ? token : std::string("1"));
|
||||
a.budget = std::move(budget);
|
||||
return a;
|
||||
}
|
||||
|
||||
std::shared_ptr<RequestBudget> default_budget() {
|
||||
return std::make_shared<RequestBudget>(HfCaps::kBudgetMaxCalls, HfCaps::kBudgetWindowSeconds);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Platform Platform::from_env() {
|
||||
return with(std::make_unique<clients::transports::HttpTransport>(),
|
||||
std::make_unique<clients::transports::HttpTransport>(),
|
||||
std::make_unique<clients::transports::HttpTransport>(), token_from_env());
|
||||
}
|
||||
|
||||
Platform Platform::with(std::unique_ptr<clients::Transport> hub_transport,
|
||||
std::unique_ptr<clients::Transport> router_transport,
|
||||
std::unique_ptr<clients::Transport> endpoints_transport,
|
||||
const std::string& token, std::shared_ptr<RequestBudget> budget) {
|
||||
if (!budget) budget = default_budget();
|
||||
Platform p;
|
||||
p.hub = make_access(std::move(hub_transport), kHubHost, token, budget);
|
||||
p.router = make_access(std::move(router_transport), kRouterHost, token, budget);
|
||||
p.endpoints = make_access(std::move(endpoints_transport), kEndpointsHost, token, budget);
|
||||
return p;
|
||||
}
|
||||
|
||||
std::string Platform::check_server_url(const std::string& url) {
|
||||
if (url.empty() || url.size() > HfCaps::kMaxServerUrlChars)
|
||||
throw llm::ApiError::invalid("huggingface", "server url is empty or too long");
|
||||
clients::Url parsed;
|
||||
try {
|
||||
parsed = clients::Url::parse(url);
|
||||
} catch (const std::exception& e) {
|
||||
throw llm::ApiError::invalid("huggingface", std::string("bad server url: ") + e.what());
|
||||
}
|
||||
if ((parsed.scheme() != "http" && parsed.scheme() != "https") || !parsed.has_host())
|
||||
throw llm::ApiError::invalid("huggingface", "server url must be http(s)://host[:port]");
|
||||
if (url.find('@') != std::string::npos || url.find('?') != std::string::npos ||
|
||||
url.find('#') != std::string::npos)
|
||||
throw llm::ApiError::invalid("huggingface", "server url must not carry credentials, a query or a fragment");
|
||||
std::string out = url;
|
||||
while (!out.empty() && out.back() == '/') out.pop_back();
|
||||
return out;
|
||||
}
|
||||
|
||||
Access Platform::server(const std::string& base_url, const std::string& token,
|
||||
std::shared_ptr<RequestBudget> budget) {
|
||||
return server_with(std::make_unique<clients::transports::HttpTransport>(), base_url, token, std::move(budget));
|
||||
}
|
||||
|
||||
Access Platform::server_with(std::unique_ptr<clients::Transport> transport, const std::string& base_url,
|
||||
const std::string& token, std::shared_ptr<RequestBudget> budget) {
|
||||
if (!budget) budget = default_budget();
|
||||
return make_access(std::move(transport), check_server_url(base_url), token, std::move(budget));
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+83
@@ -0,0 +1,83 @@
|
||||
#pragma once
|
||||
|
||||
// Wiring for the Hugging Face hosts and one shared request budget.
|
||||
//
|
||||
// One credential, HF_TOKEN (the legacy name HUGGING_FACE_HUB_TOKEN is also
|
||||
// read), through llm::Config. The Hub serves public reads without any token,
|
||||
// but llm::ApiClient cannot send an unauthenticated request. Workaround kept
|
||||
// inside this package: without a token the clients are built with a harmless
|
||||
// marker header instead of Authorization (reported in SPEC.md ## Gaps).
|
||||
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
|
||||
#include "clients/transport.h"
|
||||
#include "llm/api_client.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
// Fixed-window call counter. Local ceiling so one process never trips the
|
||||
// server limit; the clock is injectable for tests.
|
||||
class RequestBudget {
|
||||
public:
|
||||
using Clock = std::function<long()>; // monotonic seconds
|
||||
RequestBudget(int max_calls, int window_seconds, Clock clock = nullptr);
|
||||
// Throws ApiError 429 LOCAL_BUDGET when the window is spent.
|
||||
void spend();
|
||||
int used() const;
|
||||
int max_calls() const { return max_calls_; }
|
||||
|
||||
private:
|
||||
mutable std::mutex mutex_;
|
||||
int max_calls_;
|
||||
int window_;
|
||||
Clock clock_;
|
||||
long window_start_ = 0;
|
||||
int used_ = 0;
|
||||
};
|
||||
|
||||
// What a service needs: a client, whether a token backs it, the shared budget.
|
||||
struct Access {
|
||||
std::shared_ptr<llm::ApiClient> client;
|
||||
bool has_token = false;
|
||||
std::shared_ptr<RequestBudget> budget;
|
||||
};
|
||||
|
||||
struct Platform {
|
||||
static constexpr const char* kHubHost = "https://huggingface.co";
|
||||
static constexpr const char* kRouterHost = "https://router.huggingface.co";
|
||||
static constexpr const char* kEndpointsHost = "https://api.endpoints.huggingface.cloud";
|
||||
static constexpr const char* kKeyEnv = "HF_TOKEN";
|
||||
static constexpr const char* kLegacyKeyEnv = "HUGGING_FACE_HUB_TOKEN";
|
||||
|
||||
Access hub;
|
||||
Access router;
|
||||
Access endpoints;
|
||||
|
||||
// HF_TOKEN, else HUGGING_FACE_HUB_TOKEN, else empty.
|
||||
static std::string token_from_env();
|
||||
// Real HTTPS transports. A missing token still builds working anonymous
|
||||
// clients (public Hub reads); calls that need a token fail NOT_CONFIGURED.
|
||||
static Platform from_env();
|
||||
static Platform with(std::unique_ptr<clients::Transport> hub_transport,
|
||||
std::unique_ptr<clients::Transport> router_transport,
|
||||
std::unique_ptr<clients::Transport> endpoints_transport,
|
||||
const std::string& token,
|
||||
std::shared_ptr<RequestBudget> budget = nullptr);
|
||||
bool configured() const { return hub.has_token; }
|
||||
|
||||
// A TGI / TEI server (Inference Endpoint URL, localhost, any host). The token
|
||||
// is optional: self-hosted servers usually have none.
|
||||
static Access server(const std::string& base_url, const std::string& token,
|
||||
std::shared_ptr<RequestBudget> budget = nullptr);
|
||||
static Access server_with(std::unique_ptr<clients::Transport> transport, const std::string& base_url,
|
||||
const std::string& token, std::shared_ptr<RequestBudget> budget = nullptr);
|
||||
// http(s) URL with a host, no userinfo, query or fragment; trailing / removed.
|
||||
static std::string check_server_url(const std::string& url);
|
||||
|
||||
static llm::ApiClientConfig config(const std::string& base_url, bool has_token);
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
#include "vendors/huggingface/providers.h"
|
||||
|
||||
#include "vendors/huggingface/caps.h"
|
||||
#include "llm/error.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("huggingface", why); }
|
||||
|
||||
// hf-inference task map, from huggingface_hub inference/_providers/__init__.py
|
||||
// ("hf-inference" entry) and hf_inference.py (binary input classes).
|
||||
const TaskInfo kTasks[] = {
|
||||
{"text-to-image", false, false, "binary image reply"},
|
||||
{"text-generation", false, false, ""},
|
||||
{"text-classification", false, false, ""},
|
||||
{"question-answering", false, false, ""},
|
||||
{"audio-classification", true, false, ""},
|
||||
{"automatic-speech-recognition", true, false, ""},
|
||||
{"fill-mask", false, false, ""},
|
||||
{"feature-extraction", false, true, "parameters at the top level"},
|
||||
{"image-classification", true, false, ""},
|
||||
{"image-segmentation", true, false, ""},
|
||||
{"document-question-answering", false, false, ""},
|
||||
{"image-to-text", true, false, ""},
|
||||
{"object-detection", true, false, ""},
|
||||
{"audio-to-audio", true, false, ""},
|
||||
{"zero-shot-image-classification", true, false, ""},
|
||||
{"zero-shot-classification", false, false, ""},
|
||||
{"image-to-image", true, false, "binary image reply"},
|
||||
{"sentence-similarity", false, true, ""},
|
||||
{"table-question-answering", false, false, ""},
|
||||
{"tabular-classification", false, false, ""},
|
||||
{"text-to-speech", false, false, "binary audio reply"},
|
||||
{"token-classification", false, false, ""},
|
||||
{"translation", false, false, ""},
|
||||
{"summarization", false, false, ""},
|
||||
{"visual-question-answering", true, false, ""},
|
||||
};
|
||||
|
||||
bool is_json_type(const std::string& ct) {
|
||||
return ct.find("json") != std::string::npos;
|
||||
}
|
||||
|
||||
InferenceResult to_result(const llm::RawResponse& raw) {
|
||||
InferenceResult r;
|
||||
r.content_type = raw.content_type;
|
||||
if (is_json_type(raw.content_type) || raw.content_type.empty()) {
|
||||
try {
|
||||
r.json = Json::parse(raw.body);
|
||||
r.is_json = true;
|
||||
return r;
|
||||
} catch (const std::exception&) {
|
||||
if (!raw.content_type.empty())
|
||||
throw llm::ApiError("huggingface", 502, "BAD_RESPONSE", "JSON reply does not parse");
|
||||
}
|
||||
}
|
||||
r.bytes = raw.body;
|
||||
return r;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
RouterModel RouterModel::parse(const std::string& text) {
|
||||
if (text.empty() || text.size() > HfCaps::kMaxModelChars) bad("model is empty or too long");
|
||||
RouterModel m;
|
||||
const std::size_t colon = text.find(':');
|
||||
m.id = text.substr(0, colon);
|
||||
if (colon != std::string::npos) {
|
||||
m.selector = text.substr(colon + 1);
|
||||
if (!llm::Service::valid_id(m.selector) || m.selector.find(':') != std::string::npos)
|
||||
bad("bad routing selector after ':' (fastest, cheapest, preferred or a provider id)");
|
||||
}
|
||||
RequestPlan::check_model(m.id);
|
||||
return m;
|
||||
}
|
||||
|
||||
void RouterChat::set_bill_to(const std::string& organization) {
|
||||
if (!organization.empty() && !RequestPlan::valid_name(organization)) bad("bad organization name");
|
||||
bill_to_ = organization;
|
||||
}
|
||||
|
||||
llm::Headers RouterChat::headers() const {
|
||||
llm::Headers h;
|
||||
if (!bill_to_.empty()) h["X-HF-Bill-To"] = bill_to_;
|
||||
return h;
|
||||
}
|
||||
|
||||
Json RouterChat::create(const Json& body) {
|
||||
const Json* model = body.is_object() ? body.find("model") : nullptr;
|
||||
if (!model || !model->is_string()) bad("model is required");
|
||||
RouterModel::parse(model->as_string());
|
||||
if (const Json* s = body.find("stream"); s && s->is_bool() && s->as_bool()) bad("use stream() for streaming");
|
||||
return call(Call::kRouterChat, PathArgs(), Query(), &body, headers());
|
||||
}
|
||||
|
||||
void RouterChat::stream(const Json& body, const llm::SseDecoder::Sink& sink) {
|
||||
const Json* model = body.is_object() ? body.find("model") : nullptr;
|
||||
if (!model || !model->is_string()) bad("model is required");
|
||||
RouterModel::parse(model->as_string());
|
||||
HfService::stream(Call::kRouterChat, PathArgs(), body, sink, headers());
|
||||
}
|
||||
|
||||
Json RouterChat::models() { return call(Call::kRouterModels); }
|
||||
|
||||
Json RouterChat::create_hf_inference(const std::string& model, const Json& body) {
|
||||
PathArgs a;
|
||||
a.set("model", model);
|
||||
return call(Call::kHfInferenceChat, a, Query(), &body, headers());
|
||||
}
|
||||
|
||||
std::size_t TaskCatalog::count() { return sizeof(kTasks) / sizeof(kTasks[0]); }
|
||||
const TaskInfo& TaskCatalog::at(std::size_t i) {
|
||||
if (i >= count()) bad("task index out of range");
|
||||
return kTasks[i];
|
||||
}
|
||||
const TaskInfo* TaskCatalog::find(const std::string& name) {
|
||||
for (const TaskInfo& t : kTasks)
|
||||
if (name == t.name) return &t;
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
InferenceResult InferenceTasks::run_json(const std::string& task, const std::string& model, const Json& inputs,
|
||||
const Json& parameters) {
|
||||
const TaskInfo* info = TaskCatalog::find(task);
|
||||
if (!info) bad("unknown hf-inference task " + task);
|
||||
if (info->binary_input) bad("task " + task + " takes a file input; use run_binary");
|
||||
if (inputs.is_null()) bad("inputs is required");
|
||||
RequestPlan::check_model(model);
|
||||
Json body = Json::object();
|
||||
body.set("inputs", inputs);
|
||||
if (info->pipeline_path) {
|
||||
if (parameters.is_object())
|
||||
for (const auto& [k, v] : parameters.as_object()) body.set(k, v);
|
||||
} else if (!parameters.is_null()) {
|
||||
if (!parameters.is_object()) bad("parameters must be an object");
|
||||
if (!parameters.as_object().empty()) body.set("parameters", parameters);
|
||||
}
|
||||
PathArgs a;
|
||||
a.set("model", model);
|
||||
if (info->pipeline_path) {
|
||||
a.set("task", task);
|
||||
return to_result(call_raw(Call::kHfInferencePipeline, a, Query(), &body));
|
||||
}
|
||||
return to_result(call_raw(Call::kHfInferenceTask, a, Query(), &body));
|
||||
}
|
||||
|
||||
InferenceResult InferenceTasks::run_binary(const std::string& task, const std::string& model,
|
||||
const std::string& bytes, const std::string& mime,
|
||||
const Json& parameters) {
|
||||
const TaskInfo* info = TaskCatalog::find(task);
|
||||
if (!info) bad("unknown hf-inference task " + task);
|
||||
if (!info->binary_input) bad("task " + task + " takes JSON input; use run_json");
|
||||
if (bytes.empty()) bad("input bytes are empty");
|
||||
if (bytes.size() > HfCaps::kMaxInferenceInputBytes) bad("input is over the byte cap");
|
||||
RequestPlan::check_model(model);
|
||||
PathArgs a;
|
||||
a.set("model", model);
|
||||
const bool has_params = parameters.is_object() && !parameters.as_object().empty();
|
||||
if (!has_params) {
|
||||
if (mime.empty() || mime.find('\n') != std::string::npos) bad("mime type required");
|
||||
return to_result(call_bytes(Call::kHfInferenceTask, a, bytes, mime));
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("inputs", Json(security::crypto::Base64::encode(bytes)));
|
||||
body.set("parameters", parameters);
|
||||
return to_result(call_raw(Call::kHfInferenceTask, a, Query(), &body));
|
||||
}
|
||||
|
||||
Json InferenceTasks::embed(const std::string& model, const std::vector<std::string>& texts, bool normalize) {
|
||||
if (texts.empty() || texts.size() > HfCaps::kMaxBatchInputs) bad("texts must hold 1..256 entries");
|
||||
Json arr = Json::array();
|
||||
std::size_t total = 0;
|
||||
for (const std::string& t : texts) {
|
||||
total += t.size();
|
||||
if (total > HfCaps::kMaxTextChars) bad("texts are over the size cap");
|
||||
arr.push(Json(t));
|
||||
}
|
||||
Json params = Json::object();
|
||||
if (normalize) params.set("normalize", Json(true));
|
||||
InferenceResult r = run_json("feature-extraction", model, texts.size() == 1 ? Json(texts[0]) : arr, params);
|
||||
if (!r.is_json) throw llm::ApiError("huggingface", 502, "BAD_RESPONSE", "embedding reply is not JSON");
|
||||
return r.json;
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+85
@@ -0,0 +1,85 @@
|
||||
#pragma once
|
||||
|
||||
// Inference Providers (https://router.huggingface.co): the OpenAI-compatible
|
||||
// chat router plus the hf-inference task APIs (the classic Inference API).
|
||||
// The classic host api-inference.huggingface.co no longer resolves (probe
|
||||
// 2026-10-25); the same tasks live under /hf-inference/models/{model}.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/huggingface/chat_wire.h"
|
||||
#include "vendors/huggingface/hf_service.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
// "owner/name" or "owner/name:selector" where selector is fastest, cheapest,
|
||||
// preferred or a provider id (the router's routing suffix).
|
||||
struct RouterModel {
|
||||
std::string id;
|
||||
std::string selector;
|
||||
static RouterModel parse(const std::string& text); // throws ApiError::invalid
|
||||
std::string str() const { return selector.empty() ? id : id + ":" + selector; }
|
||||
};
|
||||
|
||||
class RouterChat : public HfService, public ChatEndpoint {
|
||||
public:
|
||||
explicit RouterChat(Access access) : HfService(std::move(access), Surface::kRouter) {}
|
||||
static RouterChat from_platform(const Platform& p) { return RouterChat(p.router); }
|
||||
|
||||
// Chat completions through the router. `bill_to` sets X-HF-Bill-To (an
|
||||
// organization that pays); empty: the token owner pays.
|
||||
Json create(const Json& body) override;
|
||||
void stream(const Json& body, const llm::SseDecoder::Sink& sink) override;
|
||||
void set_bill_to(const std::string& organization);
|
||||
|
||||
// GET /v1/models: public, no token needed.
|
||||
Json models();
|
||||
|
||||
// Chat straight to hf-inference for one model (no provider routing).
|
||||
Json create_hf_inference(const std::string& model, const Json& body);
|
||||
|
||||
private:
|
||||
llm::Headers headers() const;
|
||||
std::string bill_to_;
|
||||
};
|
||||
|
||||
// One hf-inference task.
|
||||
struct TaskInfo {
|
||||
const char* name; // pipeline tag: text-classification, ...
|
||||
bool binary_input; // the input is a file (image, audio), not JSON
|
||||
bool pipeline_path; // /pipeline/{task} (feature-extraction, sentence-similarity)
|
||||
const char* note;
|
||||
};
|
||||
|
||||
struct TaskCatalog {
|
||||
static std::size_t count();
|
||||
static const TaskInfo& at(std::size_t index);
|
||||
static const TaskInfo* find(const std::string& name); // nullptr when unknown
|
||||
};
|
||||
|
||||
struct InferenceResult {
|
||||
bool is_json = false;
|
||||
Json json; // when is_json
|
||||
std::string bytes; // images, audio
|
||||
std::string content_type;
|
||||
};
|
||||
|
||||
class InferenceTasks : public HfService {
|
||||
public:
|
||||
explicit InferenceTasks(Access access) : HfService(std::move(access), Surface::kRouter) {}
|
||||
static InferenceTasks from_platform(const Platform& p) { return InferenceTasks(p.router); }
|
||||
|
||||
// JSON-input tasks: {"inputs": inputs, "parameters": parameters}. For the
|
||||
// pipeline tasks the parameters go at the top level, as huggingface_hub does.
|
||||
InferenceResult run_json(const std::string& task, const std::string& model, const Json& inputs,
|
||||
const Json& parameters = Json());
|
||||
// File-input tasks: raw bytes when there are no parameters, else the
|
||||
// base64 JSON form huggingface_hub sends.
|
||||
InferenceResult run_binary(const std::string& task, const std::string& model, const std::string& bytes,
|
||||
const std::string& mime, const Json& parameters = Json());
|
||||
// feature-extraction convenience: embeddings for 1..HfCaps::kMaxBatchInputs texts.
|
||||
Json embed(const std::string& model, const std::vector<std::string>& texts, bool normalize = false);
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
+271
@@ -0,0 +1,271 @@
|
||||
#include "vendors/huggingface/request_plan.h"
|
||||
|
||||
#include <cctype>
|
||||
#include <set>
|
||||
|
||||
#include "vendors/huggingface/caps.h"
|
||||
#include "llm/api_client.h"
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("huggingface", why); }
|
||||
|
||||
bool has_control(const std::string& s) {
|
||||
for (unsigned char c : s)
|
||||
if (c < 0x20 || c == 0x7f) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
bool all_digits(const std::string& s) {
|
||||
if (s.empty()) return false;
|
||||
for (char c : s)
|
||||
if (!std::isdigit(static_cast<unsigned char>(c))) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string segment(const std::string& name, const std::string& value) {
|
||||
if (value.empty()) bad(name + " must not be empty");
|
||||
if (value.size() > HfCaps::kMaxSegmentChars) bad(name + " is too long");
|
||||
if (has_control(value)) bad(name + " has control characters");
|
||||
if (value == "." || value == "..") bad(name + " must not be . or ..");
|
||||
return RequestPlan::encode(value);
|
||||
}
|
||||
|
||||
std::string render_value(const std::string& name, const std::string& value) {
|
||||
if (name == "namespace" || name == "repo" || name == "name") {
|
||||
if (!RequestPlan::valid_name(value)) bad("bad " + name + " (letters, digits, - _ . only)");
|
||||
return value;
|
||||
}
|
||||
if (name == "repoType") {
|
||||
RepoType t;
|
||||
if (!parse_repo_type(value, t)) bad("repoType must be models, datasets or spaces");
|
||||
return repo_type_plural(t);
|
||||
}
|
||||
if (name == "typePrefix") {
|
||||
RepoType t;
|
||||
if (!parse_repo_type(value, t)) bad("repoType must be models, datasets or spaces");
|
||||
return t == RepoType::kModel ? "" : std::string("/") + repo_type_plural(t);
|
||||
}
|
||||
if (name == "path") return RequestPlan::encode_file_path(value);
|
||||
if (name == "model") return RequestPlan::check_model(value);
|
||||
if (name == "num") {
|
||||
if (!all_digits(value) || value.size() > 9) bad("num must be a discussion number");
|
||||
return value;
|
||||
}
|
||||
if (name == "action") {
|
||||
if (value != "enable" && value != "disable") bad("action must be enable or disable");
|
||||
return value;
|
||||
}
|
||||
if (name == "task" || name == "provider") {
|
||||
if (!llm::Service::valid_id(value)) bad("bad " + name);
|
||||
return value;
|
||||
}
|
||||
return segment(name, value);
|
||||
}
|
||||
|
||||
std::string query_encode_value(const std::string& v) {
|
||||
std::string out = RequestPlan::encode(v);
|
||||
// ApiClient refuses ".." anywhere in the request path, query included.
|
||||
if (v.find("..") != std::string::npos) {
|
||||
std::string fixed;
|
||||
for (char c : out) {
|
||||
if (c == '.') fixed += "%2E";
|
||||
else fixed += c;
|
||||
}
|
||||
return fixed;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const char* repo_type_plural(RepoType type) {
|
||||
switch (type) {
|
||||
case RepoType::kModel: return "models";
|
||||
case RepoType::kDataset: return "datasets";
|
||||
case RepoType::kSpace: return "spaces";
|
||||
}
|
||||
return "models";
|
||||
}
|
||||
|
||||
bool parse_repo_type(const std::string& text, RepoType& out) {
|
||||
if (text == "models" || text == "model") out = RepoType::kModel;
|
||||
else if (text == "datasets" || text == "dataset") out = RepoType::kDataset;
|
||||
else if (text == "spaces" || text == "space") out = RepoType::kSpace;
|
||||
else return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
RepoRef RepoRef::parse(const std::string& id) {
|
||||
const std::size_t slash = id.find('/');
|
||||
if (slash == std::string::npos)
|
||||
bad("repo id must be owner/name (for example openai-community/gpt2); legacy ids without an owner redirect");
|
||||
RepoRef r;
|
||||
r.owner_ = id.substr(0, slash);
|
||||
r.name_ = id.substr(slash + 1);
|
||||
if (!RequestPlan::valid_name(r.owner_) || !RequestPlan::valid_name(r.name_))
|
||||
bad("bad repo id (owner/name, letters, digits, - _ . only)");
|
||||
return r;
|
||||
}
|
||||
|
||||
PathArgs& PathArgs::set(const std::string& name, std::string value) {
|
||||
values_[name] = std::move(value);
|
||||
return *this;
|
||||
}
|
||||
|
||||
PathArgs& PathArgs::repo(const RepoRef& ref) {
|
||||
values_["namespace"] = ref.owner();
|
||||
values_["repo"] = ref.name();
|
||||
return *this;
|
||||
}
|
||||
|
||||
PathArgs& PathArgs::repo_type(RepoType type) {
|
||||
values_["repoType"] = repo_type_plural(type);
|
||||
values_["typePrefix"] = repo_type_plural(type);
|
||||
return *this;
|
||||
}
|
||||
|
||||
const std::string* PathArgs::find(const std::string& name) const {
|
||||
auto it = values_.find(name);
|
||||
return it == values_.end() ? nullptr : &it->second;
|
||||
}
|
||||
|
||||
Query& Query::add(const std::string& name, const std::string& value) {
|
||||
if (items_.size() >= HfCaps::kMaxQueryParams) bad("too many query parameters");
|
||||
if (value.size() > HfCaps::kMaxQueryValueChars) bad("query parameter " + name + " is too long");
|
||||
if (has_control(value)) bad("query parameter " + name + " has control characters");
|
||||
items_.emplace_back(name, value);
|
||||
return *this;
|
||||
}
|
||||
|
||||
Query& Query::add(const std::string& name, long value) { return add(name, std::to_string(value)); }
|
||||
|
||||
Query& Query::flag(const std::string& name, bool value) { return add(name, value ? "true" : "false"); }
|
||||
|
||||
bool Query::has(const std::string& name) const {
|
||||
for (const auto& kv : items_)
|
||||
if (kv.first == name) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
Query& Query::replace(const std::string& name, const std::string& value) {
|
||||
std::vector<std::pair<std::string, std::string>> kept;
|
||||
for (auto& kv : items_)
|
||||
if (kv.first != name) kept.push_back(kv);
|
||||
items_ = std::move(kept);
|
||||
return add(name, value);
|
||||
}
|
||||
|
||||
std::string RequestPlan::encode(const std::string& text) {
|
||||
static const char* hex = "0123456789ABCDEF";
|
||||
std::string out;
|
||||
for (unsigned char c : text) {
|
||||
if (std::isalnum(c) || c == '-' || c == '.' || c == '_' || c == '~') {
|
||||
out += static_cast<char>(c);
|
||||
} else {
|
||||
out += '%';
|
||||
out += hex[c >> 4];
|
||||
out += hex[c & 15];
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool RequestPlan::valid_name(const std::string& text) {
|
||||
if (text.empty() || text.size() > HfCaps::kMaxSegmentChars || text == "." || text == "..") return false;
|
||||
for (char c : text)
|
||||
if (!(std::isalnum(static_cast<unsigned char>(c)) || c == '-' || c == '_' || c == '.')) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string RequestPlan::encode_file_path(const std::string& path) {
|
||||
if (path.empty()) bad("path must not be empty");
|
||||
if (path.size() > HfCaps::kMaxFilePathChars) bad("path is too long");
|
||||
if (path.front() == '/' || path.back() == '/') bad("path must not start or end with /");
|
||||
std::string out;
|
||||
std::size_t segments = 0, start = 0;
|
||||
while (start <= path.size()) {
|
||||
std::size_t end = path.find('/', start);
|
||||
if (end == std::string::npos) end = path.size();
|
||||
std::string seg = path.substr(start, end - start);
|
||||
if (seg.empty()) bad("path has an empty segment");
|
||||
if (++segments > HfCaps::kMaxPathSegments) bad("path has too many segments");
|
||||
if (!out.empty()) out += '/';
|
||||
out += segment("path segment", seg);
|
||||
start = end + 1;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::string RequestPlan::check_model(const std::string& model) {
|
||||
if (model.empty() || model.size() > HfCaps::kMaxModelChars) bad("model id is empty or too long");
|
||||
const std::size_t slash = model.find('/');
|
||||
if (slash == std::string::npos) {
|
||||
if (!valid_name(model)) bad("bad model id");
|
||||
return model;
|
||||
}
|
||||
if (model.find('/', slash + 1) != std::string::npos) bad("model id must be owner/name");
|
||||
if (!valid_name(model.substr(0, slash)) || !valid_name(model.substr(slash + 1))) bad("bad model id");
|
||||
return model;
|
||||
}
|
||||
|
||||
Plan RequestPlan::make(Call call, const PathArgs& args, const Query& query, const clients::body::Json* body) {
|
||||
const CallSpec& spec = Calls::spec_of(call);
|
||||
if (spec.has_flag("unsupported"))
|
||||
bad(std::string("call is listed but not supported: ") + spec.path + " (see SPEC.md ## Gaps)");
|
||||
std::string out;
|
||||
std::set<std::string> used;
|
||||
const std::string tpl = spec.path;
|
||||
for (std::size_t i = 0; i < tpl.size();) {
|
||||
if (tpl[i] != '{') {
|
||||
out += tpl[i++];
|
||||
continue;
|
||||
}
|
||||
const std::size_t close = tpl.find('}', i);
|
||||
if (close == std::string::npos) bad("malformed call table path");
|
||||
const std::string name = tpl.substr(i + 1, close - i - 1);
|
||||
i = close + 1;
|
||||
const std::string* v = args.find(name);
|
||||
const bool last_optional = name == "path" && spec.has_flag("optpath") && i == tpl.size();
|
||||
if (!v || (v->empty() && last_optional)) {
|
||||
if (last_optional) {
|
||||
if (!out.empty() && out.back() == '/') out.pop_back();
|
||||
used.insert(name);
|
||||
continue;
|
||||
}
|
||||
bad("missing path argument " + name);
|
||||
}
|
||||
used.insert(name);
|
||||
out += render_value(name, *v);
|
||||
}
|
||||
for (const auto& kv : args.all()) {
|
||||
// repo_type() sets both repoType and typePrefix; only the ones the path uses count.
|
||||
if (!used.count(kv.first) && kv.first != "repoType" && kv.first != "typePrefix")
|
||||
bad("path argument " + kv.first + " is not used by " + spec.path);
|
||||
}
|
||||
if (spec.path[0] == '{' && out.empty()) out = "/";
|
||||
std::string qs;
|
||||
for (const auto& kv : query.items()) {
|
||||
if (!Calls::in_list(spec.query_params, kv.first))
|
||||
bad("query parameter " + kv.first + " is not accepted by " + spec.http_method + " " + spec.path);
|
||||
if (!qs.empty()) qs += '&';
|
||||
qs += encode(kv.first) + "=" + query_encode_value(kv.second);
|
||||
}
|
||||
if (body) {
|
||||
if (!spec.has_body()) bad(std::string(spec.http_method) + " " + spec.path + " takes no body");
|
||||
if (!body->is_object()) bad("body must be a JSON object");
|
||||
for (const auto& kv : body->as_object())
|
||||
if (!Calls::in_list(spec.body_fields, kv.first))
|
||||
bad("body field " + kv.first + " is not accepted by " + spec.http_method + " " + spec.path);
|
||||
}
|
||||
Plan plan;
|
||||
plan.call = call;
|
||||
plan.method = Calls::method_of(call);
|
||||
plan.target = out + (qs.empty() ? "" : "?" + qs);
|
||||
return plan;
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
#pragma once
|
||||
|
||||
// RequestPlan: turns one row of the call table plus caller arguments into a
|
||||
// validated request target (path and query). Every id is checked before it is
|
||||
// placed in a path, every query parameter and body field must be one the table
|
||||
// lists, every size is capped (caps.h). Nothing here touches the network.
|
||||
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "clients/request.h"
|
||||
#include "vendors/huggingface/calls.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
// Hub repository kinds the module reads. Kernels exist on the Hub but are
|
||||
// outside this module (SPEC.md ## Gaps).
|
||||
enum class RepoType { kModel, kDataset, kSpace };
|
||||
|
||||
const char* repo_type_plural(RepoType type); // models, datasets, spaces
|
||||
bool parse_repo_type(const std::string& text, RepoType& out); // accepts singular or plural
|
||||
|
||||
// "owner/name" validated once. Legacy ids without an owner (gpt2) are refused
|
||||
// loudly: the Hub answers them with a redirect to the canonical id.
|
||||
class RepoRef {
|
||||
public:
|
||||
static RepoRef parse(const std::string& id); // throws ApiError::invalid
|
||||
const std::string& owner() const { return owner_; }
|
||||
const std::string& name() const { return name_; }
|
||||
std::string id() const { return owner_ + "/" + name_; }
|
||||
|
||||
private:
|
||||
std::string owner_;
|
||||
std::string name_;
|
||||
};
|
||||
|
||||
class PathArgs {
|
||||
public:
|
||||
PathArgs& set(const std::string& name, std::string value);
|
||||
PathArgs& repo(const RepoRef& ref); // namespace + repo
|
||||
PathArgs& repo_type(RepoType type); // repoType + typePrefix
|
||||
const std::string* find(const std::string& name) const;
|
||||
const std::map<std::string, std::string>& all() const { return values_; }
|
||||
|
||||
private:
|
||||
std::map<std::string, std::string> values_;
|
||||
};
|
||||
|
||||
class Query {
|
||||
public:
|
||||
Query& add(const std::string& name, const std::string& value); // repeats allowed
|
||||
Query& add(const std::string& name, long value);
|
||||
Query& flag(const std::string& name, bool value);
|
||||
bool has(const std::string& name) const;
|
||||
// Replaces every value of `name` (page walks set the cursor).
|
||||
Query& replace(const std::string& name, const std::string& value);
|
||||
const std::vector<std::pair<std::string, std::string>>& items() const { return items_; }
|
||||
|
||||
private:
|
||||
std::vector<std::pair<std::string, std::string>> items_;
|
||||
};
|
||||
|
||||
struct Plan {
|
||||
Call call;
|
||||
clients::Method method;
|
||||
std::string target; // "/path?query", percent-encoded
|
||||
};
|
||||
|
||||
struct RequestPlan {
|
||||
// Throws ApiError::invalid (400 INVALID_ARGUMENT) naming the offending part.
|
||||
static Plan make(Call call, const PathArgs& args, const Query& query,
|
||||
const clients::body::Json* body = nullptr);
|
||||
|
||||
static std::string encode(const std::string& text); // RFC 3986 unreserved kept
|
||||
// True for a Hub-style name: 1..256 of [A-Za-z0-9._-] and not "." or "..".
|
||||
static bool valid_name(const std::string& text);
|
||||
// Validates a repo-relative file path and returns it percent-encoded per segment.
|
||||
static std::string encode_file_path(const std::string& path);
|
||||
// Validates "owner/name" or a bare "name" model id (router, hf-inference).
|
||||
static std::string check_model(const std::string& model);
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+183
@@ -0,0 +1,183 @@
|
||||
#include "vendors/huggingface/servers.h"
|
||||
|
||||
#include "vendors/huggingface/caps.h"
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("huggingface", why); }
|
||||
|
||||
Json with_inputs(const Json& inputs, const Json& extra) {
|
||||
Json body = Json::object();
|
||||
body.set("inputs", inputs);
|
||||
if (extra.is_object())
|
||||
for (const auto& [k, v] : extra.as_object()) body.set(k, v);
|
||||
return body;
|
||||
}
|
||||
|
||||
bool health_of(HfService& s, const std::function<llm::RawResponse()>& get) {
|
||||
(void)s;
|
||||
try {
|
||||
get();
|
||||
return true;
|
||||
} catch (const llm::ApiError& e) {
|
||||
if (e.http_status() == 503) return false;
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
bool TgiServer::ready() {
|
||||
return health_of(*this, [&] { return call_raw(Call::kTgiHealth); });
|
||||
}
|
||||
Json TgiServer::info() { return call(Call::kTgiGetModelInfo); }
|
||||
std::string TgiServer::metrics() { return call_raw(Call::kTgiMetrics).body; }
|
||||
Json TgiServer::models() { return call(Call::kTgiOpenaiGetModelInfo); }
|
||||
|
||||
Json TgiServer::generate(const std::string& inputs, const Json& parameters) {
|
||||
if (inputs.empty() || inputs.size() > HfCaps::kMaxTextChars) bad("inputs is empty or over the size cap");
|
||||
Json body = Json::object();
|
||||
body.set("inputs", Json(inputs));
|
||||
if (parameters.is_object() && !parameters.as_object().empty()) body.set("parameters", parameters);
|
||||
return call(Call::kTgiGenerate, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
void TgiServer::generate_stream(const std::string& inputs, const Json& parameters,
|
||||
const llm::SseDecoder::Sink& sink) {
|
||||
if (inputs.empty() || inputs.size() > HfCaps::kMaxTextChars) bad("inputs is empty or over the size cap");
|
||||
Json body = Json::object();
|
||||
body.set("inputs", Json(inputs));
|
||||
if (parameters.is_object() && !parameters.as_object().empty()) body.set("parameters", parameters);
|
||||
HfService::stream(Call::kTgiGenerateStream, PathArgs(), body, sink);
|
||||
}
|
||||
|
||||
Json TgiServer::tokenize(const std::string& inputs, const Json& parameters) {
|
||||
if (inputs.empty() || inputs.size() > HfCaps::kMaxTextChars) bad("inputs is empty or over the size cap");
|
||||
Json body = Json::object();
|
||||
body.set("inputs", Json(inputs));
|
||||
if (parameters.is_object() && !parameters.as_object().empty()) body.set("parameters", parameters);
|
||||
return call(Call::kTgiTokenize, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json TgiServer::chat_tokenize(const Json& chat_body) {
|
||||
return call(Call::kTgiGetChatTokenize, PathArgs(), Query(), &chat_body);
|
||||
}
|
||||
|
||||
Json TgiServer::create(const Json& body) {
|
||||
if (const Json* s = body.is_object() ? body.find("stream") : nullptr; s && s->is_bool() && s->as_bool())
|
||||
bad("use stream() for streaming");
|
||||
return call(Call::kTgiChatCompletions, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
void TgiServer::stream(const Json& body, const llm::SseDecoder::Sink& sink) {
|
||||
HfService::stream(Call::kTgiChatCompletions, PathArgs(), body, sink);
|
||||
}
|
||||
|
||||
Json TgiServer::completions(const Json& body) { return call(Call::kTgiCompletions, PathArgs(), Query(), &body); }
|
||||
Json TgiServer::invocations(const Json& body) { return call(Call::kTgiSagemakerCompatibility, PathArgs(), Query(), &body); }
|
||||
|
||||
void TeiServer::check_inputs(const Json& inputs) {
|
||||
std::size_t chars = 0, count = 1;
|
||||
if (inputs.is_string()) {
|
||||
chars = inputs.as_string().size();
|
||||
} else if (inputs.is_array()) {
|
||||
count = inputs.as_array().size();
|
||||
if (count == 0 || count > HfCaps::kMaxBatchInputs) bad("inputs must hold 1..256 entries");
|
||||
for (const Json& x : inputs.as_array()) {
|
||||
if (x.is_string()) chars += x.as_string().size();
|
||||
else if (x.is_array()) {
|
||||
for (const Json& y : x.as_array())
|
||||
if (y.is_string()) chars += y.as_string().size();
|
||||
} else bad("inputs entries must be strings (or pairs of strings for predict)");
|
||||
}
|
||||
} else {
|
||||
bad("inputs must be a string or an array");
|
||||
}
|
||||
if (chars == 0 || chars > HfCaps::kMaxTextChars) bad("inputs are empty or over the size cap");
|
||||
}
|
||||
|
||||
bool TeiServer::ready() { return health_of(*this, [&] { return call_raw(Call::kTeiHealth); }); }
|
||||
Json TeiServer::info() { return call(Call::kTeiGetModelInfo); }
|
||||
std::string TeiServer::metrics() { return call_raw(Call::kTeiMetrics).body; }
|
||||
|
||||
Json TeiServer::embed(const Json& inputs, const Json& options) {
|
||||
check_inputs(inputs);
|
||||
Json body = with_inputs(inputs, options);
|
||||
return call(Call::kTeiEmbed, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json TeiServer::embed_all(const Json& inputs, const Json& options) {
|
||||
check_inputs(inputs);
|
||||
Json body = with_inputs(inputs, options);
|
||||
return call(Call::kTeiEmbedAll, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json TeiServer::embed_sparse(const Json& inputs, const Json& options) {
|
||||
check_inputs(inputs);
|
||||
Json body = with_inputs(inputs, options);
|
||||
return call(Call::kTeiEmbedSparse, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json TeiServer::predict(const Json& inputs, const Json& options) {
|
||||
check_inputs(inputs);
|
||||
Json body = with_inputs(inputs, options);
|
||||
return call(Call::kTeiPredict, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json TeiServer::rerank(const std::string& query, const std::vector<std::string>& texts, bool return_text,
|
||||
bool raw_scores) {
|
||||
if (query.empty() || query.size() > HfCaps::kMaxTextChars) bad("query is empty or over the size cap");
|
||||
if (texts.empty() || texts.size() > HfCaps::kMaxBatchInputs) bad("texts must hold 1..256 entries");
|
||||
Json arr = Json::array();
|
||||
std::size_t chars = query.size();
|
||||
for (const std::string& t : texts) {
|
||||
chars += t.size();
|
||||
arr.push(Json(t));
|
||||
}
|
||||
if (chars > HfCaps::kMaxTextChars) bad("texts are over the size cap");
|
||||
Json body = Json::object();
|
||||
body.set("query", Json(query));
|
||||
body.set("texts", arr);
|
||||
if (return_text) body.set("return_text", Json(true));
|
||||
if (raw_scores) body.set("raw_scores", Json(true));
|
||||
return call(Call::kTeiRerank, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json TeiServer::similarity(const Json& inputs, const Json& parameters) {
|
||||
if (!inputs.is_object()) bad("similarity inputs must be an object {source_sentence, sentences}");
|
||||
Json body = Json::object();
|
||||
body.set("inputs", inputs);
|
||||
if (parameters.is_object() && !parameters.as_object().empty()) body.set("parameters", parameters);
|
||||
return call(Call::kTeiSimilarity, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json TeiServer::tokenize(const Json& inputs, const Json& options) {
|
||||
check_inputs(inputs);
|
||||
Json body = with_inputs(inputs, options);
|
||||
return call(Call::kTeiTokenize, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json TeiServer::decode(const std::vector<long>& ids, bool skip_special_tokens) {
|
||||
if (ids.empty() || ids.size() > 100000) bad("ids must hold 1..100000 entries");
|
||||
Json arr = Json::array();
|
||||
for (long id : ids) arr.push(Json(id));
|
||||
Json body = Json::object();
|
||||
body.set("ids", arr);
|
||||
body.set("skip_special_tokens", Json(skip_special_tokens));
|
||||
return call(Call::kTeiDecode, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
Json TeiServer::openai_embeddings(const Json& input, const std::string& model, int dimensions) {
|
||||
check_inputs(input);
|
||||
if (dimensions < 0 || dimensions > 65536) bad("dimensions out of range");
|
||||
Json body = Json::object();
|
||||
body.set("input", input);
|
||||
if (!model.empty()) body.set("model", Json(model));
|
||||
if (dimensions > 0) body.set("dimensions", Json(dimensions));
|
||||
return call(Call::kTeiOpenaiEmbed, PathArgs(), Query(), &body);
|
||||
}
|
||||
|
||||
} // namespace molo::huggingface
|
||||
Vendored
+64
@@ -0,0 +1,64 @@
|
||||
#pragma once
|
||||
|
||||
// Text Generation Inference (TGI) and Text Embeddings Inference (TEI) server
|
||||
// APIs. Both run anywhere: an Inference Endpoint URL, a container on
|
||||
// localhost, a Space. The base URL is given by the caller (Platform::server
|
||||
// validates it); the token is optional because self-hosted servers usually
|
||||
// have none. Request shapes follow the servers' own OpenAPI documents.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/huggingface/chat_wire.h"
|
||||
#include "vendors/huggingface/hf_service.h"
|
||||
|
||||
namespace molo::huggingface {
|
||||
|
||||
class TgiServer : public HfService, public ChatEndpoint {
|
||||
public:
|
||||
explicit TgiServer(Access access) : HfService(std::move(access), Surface::kTgi) {}
|
||||
|
||||
bool ready(); // GET /health: true on 2xx, false on 503, throws otherwise
|
||||
Json info(); // GET /info
|
||||
std::string metrics(); // GET /metrics (Prometheus text)
|
||||
Json models(); // GET /v1/models
|
||||
|
||||
// Native API.
|
||||
Json generate(const std::string& inputs, const Json& parameters = Json());
|
||||
// Streams token events; sink returns false to stop.
|
||||
void generate_stream(const std::string& inputs, const Json& parameters, const llm::SseDecoder::Sink& sink);
|
||||
Json tokenize(const std::string& inputs, const Json& parameters = Json());
|
||||
Json chat_tokenize(const Json& chat_body);
|
||||
|
||||
// OpenAI-compatible API.
|
||||
Json create(const Json& body) override; // POST /v1/chat/completions
|
||||
void stream(const Json& body, const llm::SseDecoder::Sink& sink) override;
|
||||
Json completions(const Json& body); // POST /v1/completions
|
||||
Json invocations(const Json& body); // POST /invocations (SageMaker compatibility)
|
||||
};
|
||||
|
||||
class TeiServer : public HfService {
|
||||
public:
|
||||
explicit TeiServer(Access access) : HfService(std::move(access), Surface::kTei) {}
|
||||
|
||||
bool ready();
|
||||
Json info();
|
||||
std::string metrics();
|
||||
|
||||
// inputs: one string or an array of strings (1..HfCaps::kMaxBatchInputs).
|
||||
Json embed(const Json& inputs, const Json& options = Json());
|
||||
Json embed_all(const Json& inputs, const Json& options = Json());
|
||||
Json embed_sparse(const Json& inputs, const Json& options = Json());
|
||||
Json predict(const Json& inputs, const Json& options = Json());
|
||||
Json rerank(const std::string& query, const std::vector<std::string>& texts, bool return_text = false,
|
||||
bool raw_scores = false);
|
||||
Json similarity(const Json& inputs, const Json& parameters = Json());
|
||||
Json tokenize(const Json& inputs, const Json& options = Json());
|
||||
Json decode(const std::vector<long>& ids, bool skip_special_tokens = true);
|
||||
Json openai_embeddings(const Json& input, const std::string& model = "", int dimensions = 0);
|
||||
|
||||
// Counts and size of an `inputs` value: refuses anything over the caps.
|
||||
static void check_inputs(const Json& inputs);
|
||||
};
|
||||
|
||||
} // namespace molo::huggingface
|
||||
+315
@@ -0,0 +1,315 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Checks calls.cpp (the module's one call table) against Hugging Face's
|
||||
published documents. Python 3 stdlib only, no credentials.
|
||||
|
||||
python3 -I src/vendors/huggingface/tools/check_spec.py [--cache DIR] [--probe]
|
||||
|
||||
Documents (all fetched unless found in --cache):
|
||||
hub https://huggingface.co/.well-known/openapi.json (Hub OpenAPI 3.1)
|
||||
ie https://api.endpoints.huggingface.cloud/openapi.json (Inference Endpoints 2.0.0)
|
||||
tgi text-generation-inference docs/openapi.json (main)
|
||||
tei text-embeddings-inference docs/openapi.json (main)
|
||||
src huggingface_hub hf_api.py, hf_inference.py, constants.py (main; the Hub OpenAPI
|
||||
omits the repo list / info routes, so those rows cite the SDK source)
|
||||
task huggingface.js packages/tasks/src/tasks/<task>/spec/input.json
|
||||
|
||||
Per row it requires: the HTTP method and path exist (a {repoType} row is
|
||||
expanded to models, datasets, spaces; {typePrefix} to "", /datasets, /spaces),
|
||||
the operationId matches when the document has one, every query parameter and
|
||||
body field in the row is present in the document. Coverage: for ie, tgi and tei
|
||||
every document operation, query parameter and body field must be in the table;
|
||||
for hub every operation must be in the table or match an explicit EXCLUDED rule
|
||||
(counted per reason in the output); for hub the unlisted query parameters and
|
||||
body fields are reported (not failed: the Hub document is large and
|
||||
the table lists what the module sends). With --probe, rows whose spec is
|
||||
"src" or "none" are probed without a credential: a 404 fails.
|
||||
Exit 0: all rows verified. Exit 1: at least one mismatch (printed).
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
TABLE = os.path.join(HERE, "..", "calls.cpp")
|
||||
RAW = "https://raw.githubusercontent.com/huggingface/"
|
||||
URLS = {
|
||||
"hub.json": "https://huggingface.co/.well-known/openapi.json",
|
||||
"ie.json": "https://api.endpoints.huggingface.cloud/openapi.json",
|
||||
"tgi.json": RAW + "text-generation-inference/main/docs/openapi.json",
|
||||
"tei.json": RAW + "text-embeddings-inference/main/docs/openapi.json",
|
||||
"hf_api.py": RAW + "huggingface_hub/main/src/huggingface_hub/hf_api.py",
|
||||
"hf_inference.py": RAW + "huggingface_hub/main/src/huggingface_hub/inference/_providers/hf_inference.py",
|
||||
"constants.py": RAW + "huggingface_hub/main/src/huggingface_hub/constants.py",
|
||||
}
|
||||
TASK_URL = RAW + "huggingface.js/main/packages/tasks/src/tasks/%s/spec/input.json"
|
||||
HOSTS = {
|
||||
"Hub": "https://huggingface.co",
|
||||
"Router": "https://router.huggingface.co",
|
||||
"Endpoints": "https://api.endpoints.huggingface.cloud",
|
||||
}
|
||||
S = r'"((?:[^"\\]|\\.)*)"'
|
||||
ROW = re.compile(r"^\s*\{Call::(\w+), Surface::k(\w+), " + ", ".join([S] * 8) + r"\},?\s*$")
|
||||
METHODS = ("get", "post", "put", "patch", "delete")
|
||||
REPO_TYPES = ["models", "datasets", "spaces"]
|
||||
PREFIXES = ["", "/datasets", "/spaces"]
|
||||
|
||||
# Hub operations deliberately not wrapped: (reason, regex over "METHOD /path").
|
||||
EXCLUDED = [
|
||||
("repository writes (commit, upload, LFS, branches, tags, squash, duplicate, settings, move, create)",
|
||||
r"^(POST|PUT|DELETE) /api/(models|datasets|spaces)/\{namespace\}/\{repo\}/(commit|preupload|lfs-files|tag|branch|super-squash|settings|resource-group|volumes|hardware|sleeptime|dev-mode|custom-domain|secrets|variables)"),
|
||||
("repository writes (commit, upload, LFS, branches, tags, squash, duplicate, settings, move, create)",
|
||||
r"^(POST|PUT|DELETE|GET) /api/(repos|\{repoType\})/.*(create|move|duplicate)"),
|
||||
("repository admin reads (LFS file list, xet tokens, jwt, scan, notebook, leaderboard, resource group)",
|
||||
r"^GET /api/(models|datasets|spaces)/\{namespace\}/\{repo\}/(lfs-files|xet-write-token|xet-read-token|jwt|scan|notebook|leaderboard|resource-group|logs|events|metrics)"),
|
||||
("gated repo access requests", r"(ask-access|user-access|access-request)"),
|
||||
("resolve-cache and viewer helpers", r"^GET /api/resolve-cache/"),
|
||||
("secrets and variables of a Space (credentials)", r"^(GET|POST|DELETE) /api/spaces/\{namespace\}/\{repo\}/(secrets|variables)"),
|
||||
("rarely used discussion moderation (pin, merge, ignore, storage, ref delete, comment edit, hide, reaction)",
|
||||
r"^(GET|POST|DELETE) /api/\{repoType\}/\{namespace\}/\{repo\}/discussions/\{num\}/(pin|merge|ignore|storage|ref|comment/)"),
|
||||
("collection item update and delete (need the slug-id split, see SPEC Gaps) and resource groups",
|
||||
r"^(PATCH|DELETE|POST|GET) /api/collections/.*(items/|items/batch|resource-group|\{slug\}-\{id\})"),
|
||||
("comments on blogs, papers, posts", r"^(POST|DELETE) /api/(blog|papers|posts)/"),
|
||||
("paper curation (index, links, media, claim)", r"^POST /api/(papers/index|papers/\{paperId\}/(links|media)|settings/papers/claim)"),
|
||||
("account settings, billing and notifications", r"^(GET|POST|PUT|PATCH|DELETE) /api/(settings/(?!webhooks)|notifications|credentials)"),
|
||||
("organizations, users, avatars, SCIM, service accounts, resource groups",
|
||||
r"^[A-Z]+ /api/(orgs|organizations|users|avatars|scim|service-accounts|resource-groups)"),
|
||||
("Hub Jobs and scheduled jobs (separate product)", r"^[A-Z]+ /api/(jobs|scheduled-jobs)"),
|
||||
("buckets, containers, kernels (separate repo kinds)", r"^[A-Z]+ /(api/)?(buckets|containers|kernels)"),
|
||||
("Spaces runtime control", r"^[A-Z]+ /api/spaces/(zero-gpu|\{namespace\}/\{repo\}/(runtime|restart|pause))"),
|
||||
("partner provider mapping writes and inference-provider account settings",
|
||||
r"^(POST|PUT) /api/(partners|settings/inference-providers)"),
|
||||
("Inference Endpoints auth-check helper (internal)", r"/api/inference-endpoints/"),
|
||||
("OAuth, docs search, agents, sql console, registry token, repo search POST variant",
|
||||
r"^[A-Z]+ /(oauth|api/(docs|agent-harnesses|registry|sql-console|\{repoType\}/\{namespace\}/\{repo\}/sql-console|quicksearch$))"),
|
||||
("blog, MCP settings, likes", r"^[A-Z]+ /api/(blog|settings|trending-.*)"),
|
||||
]
|
||||
|
||||
|
||||
def load(cache, name):
|
||||
if cache:
|
||||
p = os.path.join(cache, name)
|
||||
if os.path.exists(p):
|
||||
return open(p, "rb").read()
|
||||
url = URLS.get(name) or name
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "molo-check-spec/1"})
|
||||
with urllib.request.urlopen(req, timeout=120) as r:
|
||||
data = r.read()
|
||||
if cache:
|
||||
os.makedirs(os.path.dirname(os.path.join(cache, name)), exist_ok=True)
|
||||
open(os.path.join(cache, name), "wb").write(data)
|
||||
return data
|
||||
|
||||
|
||||
def resolve(doc, node):
|
||||
seen = 0
|
||||
while isinstance(node, dict) and "$ref" in node and seen < 20:
|
||||
cur = doc
|
||||
for p in node["$ref"].lstrip("#/").split("/"):
|
||||
cur = cur[p]
|
||||
node = cur
|
||||
seen += 1
|
||||
return node
|
||||
|
||||
|
||||
def props(doc, schema):
|
||||
schema = resolve(doc, schema)
|
||||
if not isinstance(schema, dict):
|
||||
return {}
|
||||
out = dict(schema.get("properties", {}))
|
||||
for key in ("allOf", "anyOf", "oneOf"):
|
||||
for sub in schema.get(key, []):
|
||||
out.update(props(doc, sub))
|
||||
return out
|
||||
|
||||
|
||||
def op_fields(doc, op):
|
||||
q = set()
|
||||
for p in op.get("parameters", []):
|
||||
p = resolve(doc, p)
|
||||
if p.get("in") == "query":
|
||||
q.add(p["name"])
|
||||
body = set()
|
||||
rb = resolve(doc, op.get("requestBody", {}))
|
||||
if rb:
|
||||
content = rb.get("content", {})
|
||||
ct = next(iter(content), None)
|
||||
if ct:
|
||||
body = set(props(doc, content[ct].get("schema")).keys())
|
||||
return q, body
|
||||
|
||||
|
||||
def unesc(s):
|
||||
return s.replace('\\"', '"').replace("\\\\", "\\")
|
||||
|
||||
|
||||
def variants(path, doc_paths=()):
|
||||
# The document may carry {repoType} literally (discussions) or one path per
|
||||
# repo type (everything else); a literal match wins.
|
||||
if path in doc_paths:
|
||||
return [path]
|
||||
if "{repoType}" in path:
|
||||
return [path.replace("{repoType}", t) for t in REPO_TYPES]
|
||||
if "{typePrefix}" in path:
|
||||
return [path.replace("{typePrefix}", p) for p in PREFIXES]
|
||||
return [path]
|
||||
|
||||
|
||||
def csv(s):
|
||||
return [x for x in s.split(",") if x]
|
||||
|
||||
|
||||
def probe_url(surface, path):
|
||||
subs = {"{namespace}": "openai-community", "{repo}": "gpt2", "{rev}": "main", "{path}": "config.json",
|
||||
"{model}": "openai-community/gpt2", "{task}": "feature-extraction", "{repoType}": "models"}
|
||||
for k, v in subs.items():
|
||||
path = path.replace(k, v)
|
||||
return HOSTS[surface] + path
|
||||
|
||||
|
||||
def main():
|
||||
cache = None
|
||||
if "--cache" in sys.argv:
|
||||
cache = sys.argv[sys.argv.index("--cache") + 1]
|
||||
text = open(TABLE).read()
|
||||
begin, end = text.index("// CALLS-BEGIN"), text.index("// CALLS-END")
|
||||
rows = []
|
||||
for line in text[begin:end].splitlines():
|
||||
m = ROW.match(line)
|
||||
if m:
|
||||
g = m.groups()
|
||||
rows.append(dict(name=g[0], surface=g[1], method=unesc(g[2]), path=unesc(g[3]), spec=unesc(g[4]),
|
||||
ref=unesc(g[5]), auth=unesc(g[6]), flags=unesc(g[7]), body=csv(unesc(g[8])),
|
||||
query=csv(unesc(g[9]))))
|
||||
if not rows:
|
||||
print("no rows parsed from calls.cpp")
|
||||
return 1
|
||||
docs = {}
|
||||
shas = {}
|
||||
for key in ("hub", "ie", "tgi", "tei"):
|
||||
raw = load(cache, key + ".json")
|
||||
shas[key] = hashlib.sha256(raw).hexdigest()[:12]
|
||||
docs[key] = json.loads(raw)
|
||||
sources = ""
|
||||
for name in ("hf_api.py", "hf_inference.py", "constants.py"):
|
||||
raw = load(cache, name)
|
||||
shas[name] = hashlib.sha256(raw).hexdigest()[:12]
|
||||
sources += raw.decode("utf-8", "replace")
|
||||
ops = {}
|
||||
for key, doc in docs.items():
|
||||
ops[key] = {}
|
||||
for path, item in doc["paths"].items():
|
||||
for method, op in item.items():
|
||||
if method in METHODS:
|
||||
ops[key][(method.upper(), path)] = op
|
||||
bad = []
|
||||
names = set()
|
||||
covered = {k: set() for k in docs}
|
||||
notes = {"hub_unlisted_query": 0, "hub_unlisted_body": 0}
|
||||
for r in rows:
|
||||
if r["name"] in names:
|
||||
bad.append("duplicate row %s" % r["name"])
|
||||
names.add(r["name"])
|
||||
spec = r["spec"]
|
||||
if spec in docs:
|
||||
union_q, union_b, found = set(), set(), False
|
||||
for v in variants(r["path"], docs[spec]["paths"]):
|
||||
op = ops[spec].get((r["method"], v))
|
||||
if op is None:
|
||||
bad.append("%s: %s %s not in the %s document" % (r["name"], r["method"], v, spec))
|
||||
continue
|
||||
found = True
|
||||
covered[spec].add((r["method"], v))
|
||||
if spec != "hub" and r["ref"] != "-" and op.get("operationId") != r["ref"]:
|
||||
bad.append("%s: operationId %s, table says %s" % (r["name"], op.get("operationId"), r["ref"]))
|
||||
q, b = op_fields(docs[spec], op)
|
||||
union_q |= q
|
||||
union_b |= b
|
||||
if not found:
|
||||
continue
|
||||
for f in r["query"]:
|
||||
if f not in union_q:
|
||||
bad.append("%s: query parameter %s absent from the document" % (r["name"], f))
|
||||
for f in r["body"]:
|
||||
if f not in union_b:
|
||||
bad.append("%s: body field %s absent from the document" % (r["name"], f))
|
||||
if spec == "hub":
|
||||
notes["hub_unlisted_query"] += len(union_q - set(r["query"]))
|
||||
notes["hub_unlisted_body"] += len(union_b - set(r["body"]))
|
||||
else:
|
||||
for f in union_q - set(r["query"]):
|
||||
bad.append("%s: document query parameter %s not in the table" % (r["name"], f))
|
||||
for f in union_b - set(r["body"]):
|
||||
bad.append("%s: document body field %s not in the table" % (r["name"], f))
|
||||
elif spec == "src":
|
||||
if r["ref"] not in sources:
|
||||
bad.append("%s: source needle %r not found in huggingface_hub" % (r["name"], r["ref"]))
|
||||
elif spec == "task":
|
||||
try:
|
||||
schema = json.loads(load(cache, "tasks/%s.json" % r["ref"]) if cache and os.path.exists(os.path.join(cache, "tasks/%s.json" % r["ref"])) else load(None, TASK_URL % r["ref"]))
|
||||
except Exception as e: # noqa
|
||||
bad.append("%s: cannot load the %s task schema (%s)" % (r["name"], r["ref"], e))
|
||||
continue
|
||||
allowed = set(schema.get("properties", {}).keys())
|
||||
for f in r["body"]:
|
||||
if f not in allowed:
|
||||
bad.append("%s: body field %s absent from the %s task schema" % (r["name"], f, r["ref"]))
|
||||
for f in allowed - set(r["body"]):
|
||||
bad.append("%s: task schema field %s not in the table" % (r["name"], f))
|
||||
elif spec != "none":
|
||||
bad.append("%s: unknown spec kind %s" % (r["name"], spec))
|
||||
# Coverage.
|
||||
for key in ("ie", "tgi", "tei"):
|
||||
for k in ops[key]:
|
||||
if k not in covered[key]:
|
||||
bad.append("unwrapped %s operation %s %s" % (key, k[0], k[1]))
|
||||
excluded_count = {}
|
||||
unwrapped_hub = 0
|
||||
for (method, path), op in sorted(ops["hub"].items()):
|
||||
if (method, path) in covered["hub"]:
|
||||
continue
|
||||
label = "%s %s" % (method, path)
|
||||
reason = None
|
||||
for why, rx in EXCLUDED:
|
||||
if re.search(rx, label):
|
||||
reason = why
|
||||
break
|
||||
if reason is None:
|
||||
bad.append("unwrapped hub operation %s (tags %s)" % (label, op.get("tags")))
|
||||
else:
|
||||
excluded_count[reason] = excluded_count.get(reason, 0) + 1
|
||||
unwrapped_hub += 1
|
||||
if "--probe" in sys.argv:
|
||||
for r in rows:
|
||||
if r["spec"] not in ("src", "none") or r["surface"] not in HOSTS or r["method"] != "GET":
|
||||
continue
|
||||
if "{" in r["path"].replace("{namespace}", "").replace("{repo}", "").replace("{rev}", "").replace("{path}", "").replace("{model}", ""):
|
||||
continue
|
||||
url = probe_url(r["surface"], r["path"])
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "molo-check-spec/1"})
|
||||
try:
|
||||
status = urllib.request.urlopen(req, timeout=60).status
|
||||
except urllib.error.HTTPError as e:
|
||||
status = e.code
|
||||
print("probe GET %s -> %d" % (url, status))
|
||||
if status == 404:
|
||||
bad.append("%s: probe answered 404" % r["name"])
|
||||
for b in bad:
|
||||
print("FAIL " + b)
|
||||
print("sources: " + ", ".join("%s=%s" % kv for kv in sorted(shas.items())))
|
||||
print("hub operations excluded (%d):" % unwrapped_hub)
|
||||
for why, n in sorted(excluded_count.items()):
|
||||
print(" %3d %s" % (n, why))
|
||||
print("hub columns the module does not send: %d query parameters, %d body fields (information)"
|
||||
% (notes["hub_unlisted_query"], notes["hub_unlisted_body"]))
|
||||
counts = {k: len(v) for k, v in ops.items()}
|
||||
print("%d rows checked; spec operations hub=%d ie=%d tgi=%d tei=%d; %d problems"
|
||||
% (len(rows), counts["hub"], counts["ie"], counts["tgi"], counts["tei"], len(bad)))
|
||||
return 1 if bad else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user