Files
moloapp/src/vendors/huggingface/SPEC.md
T

15 KiB

Hugging Face - Specification

Owned directory: src/vendors/huggingface/. Namespace molo::huggingface. Depends on molo::llm (ApiClient, Service, SseDecoder, ChatModel, Config) and molo::clients. Built by make ai (the vendor glob in src/ai.mk picks the directory up). Demo: example/app/ai/HuggingfaceDemo.cpp, routes under /ai/huggingface. Plans: plans/17-llm-vendor-platforms.md, plans/18-breadth.md.

1. Scope

Hugging Face is five HTTP surfaces on four hosts. All are covered:

Surface Host Class What
Hub https://huggingface.co Hub, Discussions, Collections, Webhooks models, datasets, spaces (list, info, revision), repo tree / refs / commits / compare / paths-info / likers / safetensors metadata, file download (/resolve), search, trending, tags, tasks, papers, Spaces catalogs, Inference Provider mappings (/api/partners/{provider}/models), whoami, discussions, collections, webhooks
Router (Inference Providers) https://router.huggingface.co RouterChat, InferenceTasks OpenAI-compatible chat (/v1/chat/completions, routing suffix :fastest :cheapest :preferred :<provider>), /v1/models, hf-inference task APIs (25 tasks, the classic Inference API), hf-inference chat
Inference Endpoints https://api.endpoints.huggingface.cloud Endpoints the whole management API v2 / v3: endpoints (list, get, create, update, pause, resume, scale to zero, delete), logs, 15 metric series, replicas, vendors and quotas, search, CIDR lists, custom domains, private services, audit logs
TGI server any URL TgiServer native (/generate, /generate_stream, /tokenize, /info, /health, /metrics) and OpenAI-compatible (/v1/chat/completions, /v1/completions, /v1/models, /invocations)
TEI server any URL TeiServer /embed, /embed_all, /embed_sparse, /predict, /rerank, /similarity, /tokenize, /decode, /v1/embeddings, /info, /health, /metrics

Plus HuggingfaceChatModel : llm::ChatModel over the router (default), a TGI server, or any ChatEndpoint.

Not covered (explicit, see tools/check_spec.py EXCLUDED and section 8): repository writes (commit, upload, LFS, branches, tags, settings), organizations and users, billing, Jobs, buckets / containers / kernels, gated-access requests, SCIM, OAuth, datasets-server.huggingface.co, the huggingface.co/api/blog and posts comment APIs, the provider-specific router paths.

2. Module map (one class per concern)

File Concern
caps.h HfCaps: static ceilings
calls.h/.cpp Call enum and kCalls: the ONE table of 132 HTTP calls (surface, method, path, spec kind, reference, auth, flags, body fields, query parameters)
request_plan.h/.cpp RequestPlan, PathArgs, Query, RepoRef, RepoType: renders and validates one call from the table
platform.h/.cpp Platform, Access, RequestBudget: hosts, HF_TOKEN, anonymous clients, shared call budget, server URL check
hf_service.h/.cpp HfService : llm::Service: surface check, token check, budget, JSON / raw / bytes / paged / SSE calls
hub.h/.cpp Hub
community.h/.cpp Discussions, Collections, Webhooks (+ CollectionItem, WebhookSpec)
providers.h/.cpp RouterChat, RouterModel, TaskCatalog, InferenceTasks
endpoints.h/.cpp Endpoints, MetricKind
servers.h/.cpp TgiServer, TeiServer
chat_wire.h/.cpp ChatWire, ChatEndpoint: neutral chat <-> OpenAI-compatible wire
chat_model.h/.cpp HuggingfaceChatModel
tools/check_spec.py table vs published documents

3. Static caps (caps.h)

Knob Value
Path segment / repo-relative file path / segments 256 / 1024 chars / 32
Query parameters / one query value 32 / 2048 chars
Model id / server URL 256 / 2048 chars
Local call budget 1000 calls per 300 s per process (observed server limit: 500 per 300 s, RateLimit-Policy: "fixed window";"api";q=500;w=300)
Page walk 20 pages, 5000 items, page limit 1000
paths-info paths 256
Download 64 MiB (the llm response cap; larger fails loudly)
Inference input (binary) 16 MiB
Batch inputs (TEI, embeddings) 256 entries, 4 MiB text
Stream events 200000
Messages / tools / stop sequences 4096 / 128 / 4
Webhook domains 8

4. Contracts

  • Token. HF_TOKEN (or the legacy HUGGING_FACE_HUB_TOKEN) via llm::Config. It travels as Authorization: Bearer only, never in a URL or an error. A row with auth: token and no token throws NotConfiguredError (503 NOT_CONFIGURED) before any traffic. Rows with auth: none or optional (public Hub reads, GET /v1/models, GET /v2/provider, TGI / TEI) run without a token.
  • Anonymous requests. llm::ApiClient cannot send an unauthenticated request, so without a token Platform builds the client with a marker header X-Molo-Anonymous: 1 instead of Authorization. Foundation gap, worked around inside the package.
  • One table. Every path, query parameter and body field comes from kCalls. RequestPlan::make refuses a query parameter or body field the row does not list, an unused or missing path argument, a body on a bodiless row, and rows flagged unsupported.
  • Ids. namespace, repo, name: [A-Za-z0-9._-], 1..256, not . / ... Repo ids must be owner/name; legacy ids without an owner (gpt2) are refused because the Hub answers them with a redirect to the canonical id. A revision or tag is one percent-encoded segment (refs/pr/1 -> refs%2Fpr%2F1). File paths are validated per segment and percent-encoded. Query values are percent-encoded; a value containing .. has its dots encoded because ApiClient refuses .. anywhere in the request path.
  • Lists. Hub list calls follow Link: <...>; rel="next" cursors, accept only the cursor of a Link to the same host, and return Page with truncated, truncated_by (items or pages) and next_cursor. Never silent.
  • Writes and deletes. Rows flagged write need a token. Deletes are remove_* methods with no defaults, flagged destructive in the table (tested: 7 DELETE rows, all flagged). The demo runs every paid or changing action as a dry run unless the body has "confirm":true.
  • Chat wire. ChatWire documents its mapping in chat_wire.h: tool arguments that are not JSON arrive as {"_invalid_json": raw}; reasoning is read from reasoning_content then reasoning; response_schema maps to a strict json_schema; usage is requested in streams.
  • Errors. Everything is an ApiError; the Hub error body {"error":"..."} and the Endpoints body {"error":"401 Unauthorized","code":"UNAUTHORIZED"} keep their message and status. A local budget stop is 429 LOCAL_BUDGET.
  • Tests. MOLO_ENV_FILES=off; MOLO_LIVE_NET=1 enables the live tests; MOLO_LIVE_PAID=1 additionally allows the one chat completion.

5. Credentials

Env var Page
HF_TOKEN https://huggingface.co/settings/tokens (docs: https://huggingface.co/docs/hub/security-tokens). A fine-grained token needs "Make calls to Inference Providers" for the router, "Make calls to Inference Endpoints" and "Manage Inference Endpoints" for the endpoints API, and write scopes for discussions / collections / webhooks. Inference Providers bills the token owner (or the organization named by RouterChat::set_bill_to)

No key exists in the development environment: see sections 7 and 8.

6. Sources

Machine-readable documents, fetched 2026-10-25:

  • Hub OpenAPI 3.1 https://huggingface.co/.well-known/openapi.json (374 operations, 307 paths; also /.well-known/openapi.md). It omits the list / info routes of models, datasets and spaces (GET /api/models, /api/models/{ns}/{repo}, ...), /api/tasks and /api/spaces/{ns}/{repo}/runtime; those 11 Hub rows cite huggingface_hub hf_api.py (spec kind src); the three hf-inference router rows cite hf_inference.py (also src).
  • Inference Endpoints OpenAPI 2.0.0 https://api.endpoints.huggingface.cloud/openapi.json (48 operations). The old /api-doc/openapi.json answers 404; the swagger page at / points at /openapi.json.
  • TGI docs/openapi.json (3.3.6-dev0, 12 operations) and TEI docs/openapi.json (1.9.4, 12 operations) from the huggingface/text-generation-inference and huggingface/text-embeddings-inference repositories, branch main.
  • huggingface_hub src/huggingface_hub/hf_api.py, constants.py, inference/_providers/__init__.py and hf_inference.py: endpoints and parameters of the routes the OpenAPI omits, the router URL template https://router.huggingface.co/{provider}, the 25-task hf-inference map, the feature-extraction / sentence-similarity /pipeline/{task} path, binary-input tasks, base64 form with parameters.
  • huggingface.js packages/tasks/src/tasks/<task>/spec/input.json for the chat-completion body fields (26 tasks fetched; audio-to-audio and tabular-classification have no schema file).
  • Unauthenticated probes, section 8.

7. Verified against spec

python3 -I src/vendors/huggingface/tools/check_spec.py --probe, 2026-10-25:

132 rows checked; spec operations hub=374 ie=48 tgi=12 tei=12; 0 problems

For ie, tgi, tei: every document operation, query parameter and body field is in the table (equality). For hub: every row's method, path (expanded over models / datasets / spaces where the document has one path per type), query parameters and body fields exist in the document; the other 314 operations are excluded by explicit rules in EXCLUDED, counted per reason in the output (jobs 23, buckets / containers / kernels 33, organizations / users / SCIM 77, ...). The chat row is checked against the huggingface.js chat-completion schema (equality of fields). --probe GETs the src and none rows without a credential: all answered 200 (public) or 401 (the Hub's answer for an unknown or private repo id), none 404.

Local tests: make ai-test runs 66 huggingface_* tests (table integrity, id and path rules, query and body checks, caps, Link walks, budget window, token header, anonymous header, error mapping, chat wire, SSE transcript, every surface's request shapes). Replies in tests are RECORDED (copied from a probe, trimmed) or SHAPE (written to the documented schema). Nothing is a scripted claim about vendor behavior. Example route tests: tests/example/ai/huggingface/.

8. Verified live

Run on 2026-10-25 with no credential unless stated. All through the real HTTP transport (MOLO_LIVE_NET=1 make ai-test: 4 live tests, 3 run without a key).

Request Result
GET huggingface.co/api/models?limit=2, /api/datasets?limit=2, /api/spaces?limit=2 200 JSON array; Link: <...&cursor=...>; rel="next"; RateLimit-Policy: "fixed window";"api";q=500;w=300
GET /api/models/gpt2, /api/datasets/squad, /api/models/gpt2/refs 307 to the canonical openai-community/gpt2 / rajpurkar/squad (the reason legacy ids are refused)
GET /api/models/openai-community/gpt2, /refs, /tree/main, /discussions; POST /paths-info/main; GET /openai-community/gpt2/resolve/main/config.json (665 bytes, X-Repo-Commit header) 200
GET /api/spaces/gradio/hello_world, /api/spaces/hardware, /api/trending, /api/models-tags-by-type, /api/tasks, /api/daily_papers, /api/quicksearch, /api/collections?limit=2, /api/collections/nvidia/flora-6ad6a8986320a55467825cf5, /api/partners/groq/models 200
GET /api/whoami-v2, /api/settings/webhooks without a token 401 {"error":"Invalid username or password."}
GET router.huggingface.co/v1/models 200, 134 models, per model a providers array (status, context_length, pricing, supports_tools, latency)
POST router.huggingface.co/v1/chat/completions, /v1/embeddings, /hf-inference/models/gpt2, any other path 401 text/html without a token, including a made-up path. The router answers 401 before routing, so these probes do NOT prove a path exists
GET router.huggingface.co/hf-inference/models/gpt2 401 JSON {"error":"Invalid username or password."}
GET api.endpoints.huggingface.cloud/v2/provider 200 (public vendor / region / compute / price list)
GET .../v2/endpoint/huggingface, /v2/provider/huggingface, /v2/search/huggingface, /v2/endpoint/huggingface/x/logs 401 {"error":"401 Unauthorized","code":"UNAUTHORIZED"}
Invalid token hf_invalid_molo_probe on whoami and endpoints list 401 with the two envelopes above (live test)
api-inference.huggingface.co host does not resolve (the classic host is gone; hf-inference lives behind the router)
datasets-server.huggingface.co/is-valid?dataset=squad 404 {"error":"The dataset has been renamed..."} (out of scope, noted)

9. Unverified

No token was available, so no authenticated success body was seen:

  • every auth: token call: whoami, space runtime, discussions create / comment / status / title / delete, collections create / update / add item / delete, webhooks (all), Endpoints (all except GET /v2/provider);
  • the router chat completion, its routing suffixes, the X-HF-Bill-To header (taken from huggingface_hub / docs, unprobed), streaming usage in the last chunk, reasoning vs reasoning_content per provider;
  • every hf-inference task call and reply shape: the /hf-inference/models/{model} and /pipeline/{task} paths and the binary / base64 input forms come from huggingface_hub source, not from a response;
  • the response shapes our code reads (choices, usage, tool call fields) are the OpenAI-compatible shapes of the chat-completion spec, not seen from the router;
  • TGI and TEI: no server was reachable; request shapes follow their OpenAPI documents; ready() treats 503 as "starting" per TGI behavior and the documents, unobserved;
  • Endpoints::validate_spec checks only the required top-level keys, the name rule from the schema description and the type enum;
  • ZeroGPU / pro quotas and per-token rate limits other than the observed anonymous q=500;w=300;
  • text ranges in /resolve (no range support), xet / LFS downloads of files over 64 MiB.

10. Gaps

  • llm::ApiClient cannot send an unauthenticated request (workaround: marker header), has no GET stream (the three Endpoints SSE routes GET .../sse, .../logs/sse and /v3/.../logs/sse are in the table with flag unsupported and refused), no per-call timeout, and drops response headers on request() (paging uses request_raw).
  • llm::ApiClient::build refuses .. in any request path including the query; query values with .. are dot-encoded as a workaround.
  • Provider-specific router paths (router.huggingface.co/{provider}/..., for example text-to-image on fal-ai or Replicate) have no per-provider schema published; only hf-inference tasks and the generic chat route are wrapped.
  • Collection item update / delete (.../items/{itemId}) are not wrapped: the document names the path with the {slug}-{id} split.
  • Repo writes (commit, upload, branches, tags, LFS) are out of scope; Hub reads and downloads only. Legacy ids without owner are refused.
  • kernels repo type exists on the Hub and in the discussions document; not wrapped.
  • Services Discussions, Collections, Webhooks share Surface::kHub.
  • Google services used: none.