235 lines
15 KiB
Markdown
235 lines
15 KiB
Markdown
# Hugging Face - Specification
|
|||
|
|
|
||
|
|
Owned directory: `src/vendors/huggingface/`. Namespace `molo::huggingface`.
|
||
|
|
Depends on `molo::llm` (`ApiClient`, `Service`, `SseDecoder`, `ChatModel`,
|
||
|
|
`Config`) and `molo::clients`. Built by `make ai` (the vendor glob in
|
||
|
|
`src/ai.mk` picks the directory up). Demo: `example/app/ai/HuggingfaceDemo.cpp`,
|
||
|
|
routes under `/ai/huggingface`. Plans: `plans/17-llm-vendor-platforms.md`,
|
||
|
|
`plans/18-breadth.md`.
|
||
|
|
|
||
|
|
## 1. Scope
|
||
|
|
|
||
|
|
Hugging Face is five HTTP surfaces on four hosts. All are covered:
|
||
|
|
|
||
|
|
| Surface | Host | Class | What |
|
||
|
|
|---|---|---|---|
|
||
|
|
| Hub | `https://huggingface.co` | `Hub`, `Discussions`, `Collections`, `Webhooks` | models, datasets, spaces (list, info, revision), repo tree / refs / commits / compare / paths-info / likers / safetensors metadata, file download (`/resolve`), search, trending, tags, tasks, papers, Spaces catalogs, Inference Provider mappings (`/api/partners/{provider}/models`), whoami, discussions, collections, webhooks |
|
||
|
|
| Router (Inference Providers) | `https://router.huggingface.co` | `RouterChat`, `InferenceTasks` | OpenAI-compatible chat (`/v1/chat/completions`, routing suffix `:fastest` `:cheapest` `:preferred` `:<provider>`), `/v1/models`, hf-inference task APIs (25 tasks, the classic Inference API), hf-inference chat |
|
||
|
|
| Inference Endpoints | `https://api.endpoints.huggingface.cloud` | `Endpoints` | the whole management API v2 / v3: endpoints (list, get, create, update, pause, resume, scale to zero, delete), logs, 15 metric series, replicas, vendors and quotas, search, CIDR lists, custom domains, private services, audit logs |
|
||
|
|
| TGI server | any URL | `TgiServer` | native (`/generate`, `/generate_stream`, `/tokenize`, `/info`, `/health`, `/metrics`) and OpenAI-compatible (`/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/invocations`) |
|
||
|
|
| TEI server | any URL | `TeiServer` | `/embed`, `/embed_all`, `/embed_sparse`, `/predict`, `/rerank`, `/similarity`, `/tokenize`, `/decode`, `/v1/embeddings`, `/info`, `/health`, `/metrics` |
|
||
|
|
|
||
|
|
Plus `HuggingfaceChatModel : llm::ChatModel` over the router (default), a TGI
|
||
|
|
server, or any `ChatEndpoint`.
|
||
|
|
|
||
|
|
Not covered (explicit, see `tools/check_spec.py` EXCLUDED and section 8):
|
||
|
|
repository writes (commit, upload, LFS, branches, tags, settings), organizations
|
||
|
|
and users, billing, Jobs, buckets / containers / kernels, gated-access requests,
|
||
|
|
SCIM, OAuth, `datasets-server.huggingface.co`, the `huggingface.co/api/blog`
|
||
|
|
and posts comment APIs, the provider-specific router paths.
|
||
|
|
|
||
|
|
## 2. Module map (one class per concern)
|
||
|
|
|
||
|
|
| File | Concern |
|
||
|
|
|---|---|
|
||
|
|
| `caps.h` | `HfCaps`: static ceilings |
|
||
|
|
| `calls.h/.cpp` | `Call` enum and `kCalls`: the ONE table of 132 HTTP calls (surface, method, path, spec kind, reference, auth, flags, body fields, query parameters) |
|
||
|
|
| `request_plan.h/.cpp` | `RequestPlan`, `PathArgs`, `Query`, `RepoRef`, `RepoType`: renders and validates one call from the table |
|
||
|
|
| `platform.h/.cpp` | `Platform`, `Access`, `RequestBudget`: hosts, `HF_TOKEN`, anonymous clients, shared call budget, server URL check |
|
||
|
|
| `hf_service.h/.cpp` | `HfService : llm::Service`: surface check, token check, budget, JSON / raw / bytes / paged / SSE calls |
|
||
|
|
| `hub.h/.cpp` | `Hub` |
|
||
|
|
| `community.h/.cpp` | `Discussions`, `Collections`, `Webhooks` (+ `CollectionItem`, `WebhookSpec`) |
|
||
|
|
| `providers.h/.cpp` | `RouterChat`, `RouterModel`, `TaskCatalog`, `InferenceTasks` |
|
||
|
|
| `endpoints.h/.cpp` | `Endpoints`, `MetricKind` |
|
||
|
|
| `servers.h/.cpp` | `TgiServer`, `TeiServer` |
|
||
|
|
| `chat_wire.h/.cpp` | `ChatWire`, `ChatEndpoint`: neutral chat <-> OpenAI-compatible wire |
|
||
|
|
| `chat_model.h/.cpp` | `HuggingfaceChatModel` |
|
||
|
|
| `tools/check_spec.py` | table vs published documents |
|
||
|
|
|
||
|
|
## 3. Static caps (caps.h)
|
||
|
|
|
||
|
|
| Knob | Value |
|
||
|
|
|---|---|
|
||
|
|
| Path segment / repo-relative file path / segments | 256 / 1024 chars / 32 |
|
||
|
|
| Query parameters / one query value | 32 / 2048 chars |
|
||
|
|
| Model id / server URL | 256 / 2048 chars |
|
||
|
|
| Local call budget | 1000 calls per 300 s per process (observed server limit: 500 per 300 s, `RateLimit-Policy: "fixed window";"api";q=500;w=300`) |
|
||
|
|
| Page walk | 20 pages, 5000 items, page `limit` 1000 |
|
||
|
|
| `paths-info` paths | 256 |
|
||
|
|
| Download | 64 MiB (the `llm` response cap; larger fails loudly) |
|
||
|
|
| Inference input (binary) | 16 MiB |
|
||
|
|
| Batch inputs (TEI, embeddings) | 256 entries, 4 MiB text |
|
||
|
|
| Stream events | 200000 |
|
||
|
|
| Messages / tools / stop sequences | 4096 / 128 / 4 |
|
||
|
|
| Webhook domains | 8 |
|
||
|
|
|
||
|
|
## 4. Contracts
|
||
|
|
|
||
|
|
- **Token.** `HF_TOKEN` (or the legacy `HUGGING_FACE_HUB_TOKEN`) via
|
||
|
|
`llm::Config`. It travels as `Authorization: Bearer` only, never in a URL or an
|
||
|
|
error. A row with `auth: token` and no token throws `NotConfiguredError`
|
||
|
|
(503 `NOT_CONFIGURED`) before any traffic. Rows with `auth: none` or
|
||
|
|
`optional` (public Hub reads, `GET /v1/models`, `GET /v2/provider`, TGI / TEI)
|
||
|
|
run without a token.
|
||
|
|
- **Anonymous requests.** `llm::ApiClient` cannot send an unauthenticated
|
||
|
|
request, so without a token `Platform` builds the client with a marker header
|
||
|
|
`X-Molo-Anonymous: 1` instead of `Authorization`. Foundation gap, worked
|
||
|
|
around inside the package.
|
||
|
|
- **One table.** Every path, query parameter and body field comes from `kCalls`.
|
||
|
|
`RequestPlan::make` refuses a query parameter or body field the row does not
|
||
|
|
list, an unused or missing path argument, a body on a bodiless row, and rows
|
||
|
|
flagged `unsupported`.
|
||
|
|
- **Ids.** `namespace`, `repo`, `name`: `[A-Za-z0-9._-]`, 1..256, not `.` / `..`.
|
||
|
|
Repo ids must be `owner/name`; legacy ids without an owner (`gpt2`) are refused
|
||
|
|
because the Hub answers them with a redirect to the canonical id. A revision or
|
||
|
|
tag is one percent-encoded segment (`refs/pr/1` -> `refs%2Fpr%2F1`). File
|
||
|
|
paths are validated per segment and percent-encoded. Query values are
|
||
|
|
percent-encoded; a value containing `..` has its dots encoded because
|
||
|
|
`ApiClient` refuses `..` anywhere in the request path.
|
||
|
|
- **Lists.** Hub list calls follow `Link: <...>; rel="next"` cursors, accept only
|
||
|
|
the `cursor` of a Link to the same host, and return `Page` with `truncated`,
|
||
|
|
`truncated_by` (`items` or `pages`) and `next_cursor`. Never silent.
|
||
|
|
- **Writes and deletes.** Rows flagged `write` need a token. Deletes are
|
||
|
|
`remove_*` methods with no defaults, flagged `destructive` in the table
|
||
|
|
(tested: 7 DELETE rows, all flagged). The demo runs every paid or changing
|
||
|
|
action as a dry run unless the body has `"confirm":true`.
|
||
|
|
- **Chat wire.** `ChatWire` documents its mapping in `chat_wire.h`: tool
|
||
|
|
arguments that are not JSON arrive as `{"_invalid_json": raw}`; reasoning is
|
||
|
|
read from `reasoning_content` then `reasoning`; `response_schema` maps to a
|
||
|
|
strict `json_schema`; usage is requested in streams.
|
||
|
|
- **Errors.** Everything is an `ApiError`; the Hub error body
|
||
|
|
`{"error":"..."}` and the Endpoints body
|
||
|
|
`{"error":"401 Unauthorized","code":"UNAUTHORIZED"}` keep their message and
|
||
|
|
status. A local budget stop is `429 LOCAL_BUDGET`.
|
||
|
|
- **Tests.** `MOLO_ENV_FILES=off`; `MOLO_LIVE_NET=1` enables the live tests;
|
||
|
|
`MOLO_LIVE_PAID=1` additionally allows the one chat completion.
|
||
|
|
|
||
|
|
## 5. Credentials
|
||
|
|
|
||
|
|
| Env var | Page |
|
||
|
|
|---|---|
|
||
|
|
| `HF_TOKEN` | https://huggingface.co/settings/tokens (docs: https://huggingface.co/docs/hub/security-tokens). A fine-grained token needs "Make calls to Inference Providers" for the router, "Make calls to Inference Endpoints" and "Manage Inference Endpoints" for the endpoints API, and write scopes for discussions / collections / webhooks. Inference Providers bills the token owner (or the organization named by `RouterChat::set_bill_to`) |
|
||
|
|
|
||
|
|
No key exists in the development environment: see sections 7 and 8.
|
||
|
|
|
||
|
|
## 6. Sources
|
||
|
|
|
||
|
|
Machine-readable documents, fetched 2026-10-25:
|
||
|
|
|
||
|
|
- Hub OpenAPI 3.1 `https://huggingface.co/.well-known/openapi.json` (374
|
||
|
|
operations, 307 paths; also `/.well-known/openapi.md`). It omits the
|
||
|
|
list / info routes of models, datasets and spaces (`GET /api/models`,
|
||
|
|
`/api/models/{ns}/{repo}`, ...), `/api/tasks` and `/api/spaces/{ns}/{repo}/runtime`;
|
||
|
|
those 11 Hub rows cite `huggingface_hub` `hf_api.py` (spec kind `src`); the
|
||
|
|
three hf-inference router rows cite `hf_inference.py` (also `src`).
|
||
|
|
- Inference Endpoints OpenAPI 2.0.0 `https://api.endpoints.huggingface.cloud/openapi.json`
|
||
|
|
(48 operations). The old `/api-doc/openapi.json` answers 404; the swagger page
|
||
|
|
at `/` points at `/openapi.json`.
|
||
|
|
- TGI `docs/openapi.json` (3.3.6-dev0, 12 operations) and TEI `docs/openapi.json`
|
||
|
|
(1.9.4, 12 operations) from the `huggingface/text-generation-inference` and
|
||
|
|
`huggingface/text-embeddings-inference` repositories, branch `main`.
|
||
|
|
- `huggingface_hub` `src/huggingface_hub/hf_api.py`, `constants.py`,
|
||
|
|
`inference/_providers/__init__.py` and `hf_inference.py`: endpoints and
|
||
|
|
parameters of the routes the OpenAPI omits, the router URL template
|
||
|
|
`https://router.huggingface.co/{provider}`, the 25-task hf-inference map, the
|
||
|
|
feature-extraction / sentence-similarity `/pipeline/{task}` path, binary-input
|
||
|
|
tasks, base64 form with parameters.
|
||
|
|
- `huggingface.js` `packages/tasks/src/tasks/<task>/spec/input.json` for the
|
||
|
|
chat-completion body fields (26 tasks fetched; `audio-to-audio` and
|
||
|
|
`tabular-classification` have no schema file).
|
||
|
|
- Unauthenticated probes, section 8.
|
||
|
|
|
||
|
|
## 7. Verified against spec
|
||
|
|
|
||
|
|
`python3 -I src/vendors/huggingface/tools/check_spec.py --probe`, 2026-10-25:
|
||
|
|
|
||
|
|
```
|
||
|
|
132 rows checked; spec operations hub=374 ie=48 tgi=12 tei=12; 0 problems
|
||
|
|
```
|
||
|
|
|
||
|
|
For `ie`, `tgi`, `tei`: every document operation, query parameter and body
|
||
|
|
field is in the table (equality). For `hub`: every row's method, path (expanded
|
||
|
|
over models / datasets / spaces where the document has one path per type),
|
||
|
|
query parameters and body fields exist in the document; the other 314
|
||
|
|
operations are excluded by explicit rules in `EXCLUDED`, counted per reason in
|
||
|
|
the output (jobs 23, buckets / containers / kernels 33, organizations / users /
|
||
|
|
SCIM 77, ...). The chat row is checked against the huggingface.js
|
||
|
|
chat-completion schema (equality of fields). `--probe` GETs the `src` and
|
||
|
|
`none` rows without a credential: all answered 200 (public) or 401 (the Hub's
|
||
|
|
answer for an unknown or private repo id), none 404.
|
||
|
|
|
||
|
|
Local tests: `make ai-test` runs 66 `huggingface_*` tests (table integrity, id
|
||
|
|
and path rules, query and body checks, caps, Link walks, budget window, token
|
||
|
|
header, anonymous header, error mapping, chat wire, SSE transcript, every
|
||
|
|
surface's request shapes). Replies in tests are `RECORDED` (copied from a probe,
|
||
|
|
trimmed) or `SHAPE` (written to the documented schema). Nothing is a scripted
|
||
|
|
claim about vendor behavior. Example route tests: `tests/example/ai/huggingface/`.
|
||
|
|
|
||
|
|
## 8. Verified live
|
||
|
|
|
||
|
|
Run on 2026-10-25 with no credential unless stated. All through the real HTTP
|
||
|
|
transport (`MOLO_LIVE_NET=1 make ai-test`: 4 live tests, 3 run without a key).
|
||
|
|
|
||
|
|
| Request | Result |
|
||
|
|
|---|---|
|
||
|
|
| `GET huggingface.co/api/models?limit=2`, `/api/datasets?limit=2`, `/api/spaces?limit=2` | 200 JSON array; `Link: <...&cursor=...>; rel="next"`; `RateLimit-Policy: "fixed window";"api";q=500;w=300` |
|
||
|
|
| `GET /api/models/gpt2`, `/api/datasets/squad`, `/api/models/gpt2/refs` | 307 to the canonical `openai-community/gpt2` / `rajpurkar/squad` (the reason legacy ids are refused) |
|
||
|
|
| `GET /api/models/openai-community/gpt2`, `/refs`, `/tree/main`, `/discussions`; `POST /paths-info/main`; `GET /openai-community/gpt2/resolve/main/config.json` (665 bytes, `X-Repo-Commit` header) | 200 |
|
||
|
|
| `GET /api/spaces/gradio/hello_world`, `/api/spaces/hardware`, `/api/trending`, `/api/models-tags-by-type`, `/api/tasks`, `/api/daily_papers`, `/api/quicksearch`, `/api/collections?limit=2`, `/api/collections/nvidia/flora-6ad6a8986320a55467825cf5`, `/api/partners/groq/models` | 200 |
|
||
|
|
| `GET /api/whoami-v2`, `/api/settings/webhooks` without a token | 401 `{"error":"Invalid username or password."}` |
|
||
|
|
| `GET router.huggingface.co/v1/models` | 200, 134 models, per model a `providers` array (status, context_length, pricing, supports_tools, latency) |
|
||
|
|
| `POST router.huggingface.co/v1/chat/completions`, `/v1/embeddings`, `/hf-inference/models/gpt2`, any other path | 401 `text/html` without a token, including a made-up path. The router answers 401 before routing, so these probes do NOT prove a path exists |
|
||
|
|
| `GET router.huggingface.co/hf-inference/models/gpt2` | 401 JSON `{"error":"Invalid username or password."}` |
|
||
|
|
| `GET api.endpoints.huggingface.cloud/v2/provider` | 200 (public vendor / region / compute / price list) |
|
||
|
|
| `GET .../v2/endpoint/huggingface`, `/v2/provider/huggingface`, `/v2/search/huggingface`, `/v2/endpoint/huggingface/x/logs` | 401 `{"error":"401 Unauthorized","code":"UNAUTHORIZED"}` |
|
||
|
|
| Invalid token `hf_invalid_molo_probe` on `whoami` and `endpoints list` | 401 with the two envelopes above (live test) |
|
||
|
|
| `api-inference.huggingface.co` | host does not resolve (the classic host is gone; hf-inference lives behind the router) |
|
||
|
|
| `datasets-server.huggingface.co/is-valid?dataset=squad` | 404 `{"error":"The dataset has been renamed..."}` (out of scope, noted) |
|
||
|
|
|
||
|
|
## 9. Unverified
|
||
|
|
|
||
|
|
No token was available, so no authenticated success body was seen:
|
||
|
|
|
||
|
|
- every `auth: token` call: whoami, space runtime, discussions create / comment /
|
||
|
|
status / title / delete, collections create / update / add item / delete,
|
||
|
|
webhooks (all), Endpoints (all except `GET /v2/provider`);
|
||
|
|
- the router chat completion, its routing suffixes, the `X-HF-Bill-To` header
|
||
|
|
(taken from huggingface_hub / docs, unprobed), streaming usage in the last
|
||
|
|
chunk, `reasoning` vs `reasoning_content` per provider;
|
||
|
|
- every hf-inference task call and reply shape: the `/hf-inference/models/{model}`
|
||
|
|
and `/pipeline/{task}` paths and the binary / base64 input forms come from
|
||
|
|
`huggingface_hub` source, not from a response;
|
||
|
|
- the response shapes our code reads (`choices`, `usage`, tool call fields) are
|
||
|
|
the OpenAI-compatible shapes of the chat-completion spec, not seen from the
|
||
|
|
router;
|
||
|
|
- TGI and TEI: no server was reachable; request shapes follow their OpenAPI
|
||
|
|
documents; `ready()` treats 503 as "starting" per TGI behavior and the
|
||
|
|
documents, unobserved;
|
||
|
|
- `Endpoints::validate_spec` checks only the required top-level keys, the name
|
||
|
|
rule from the schema description and the `type` enum;
|
||
|
|
- ZeroGPU / pro quotas and per-token rate limits other than the observed
|
||
|
|
anonymous `q=500;w=300`;
|
||
|
|
- text ranges in `/resolve` (no range support), xet / LFS downloads of files
|
||
|
|
over 64 MiB.
|
||
|
|
|
||
|
|
## 10. Gaps
|
||
|
|
|
||
|
|
- `llm::ApiClient` cannot send an unauthenticated request (workaround: marker
|
||
|
|
header), has no GET stream (the three Endpoints SSE routes `GET .../sse`,
|
||
|
|
`.../logs/sse` and `/v3/.../logs/sse` are in the table with flag
|
||
|
|
`unsupported` and refused), no per-call timeout, and drops response headers on
|
||
|
|
`request()` (paging uses `request_raw`).
|
||
|
|
- `llm::ApiClient::build` refuses `..` in any request path including the query;
|
||
|
|
query values with `..` are dot-encoded as a workaround.
|
||
|
|
- Provider-specific router paths (`router.huggingface.co/{provider}/...`, for
|
||
|
|
example text-to-image on fal-ai or Replicate) have no per-provider schema
|
||
|
|
published; only `hf-inference` tasks and the generic chat route are wrapped.
|
||
|
|
- Collection item update / delete (`.../items/{itemId}`) are not wrapped: the
|
||
|
|
document names the path with the `{slug}-{id}` split.
|
||
|
|
- Repo writes (commit, upload, branches, tags, LFS) are out of scope; `Hub`
|
||
|
|
reads and downloads only. Legacy ids without owner are refused.
|
||
|
|
- `kernels` repo type exists on the Hub and in the discussions document; not
|
||
|
|
wrapped.
|
||
|
|
- Services `Discussions`, `Collections`, `Webhooks` share `Surface::kHub`.
|
||
|
|
- Google services used: none.
|