Merge branch 'worktree-agent-a1e8651ad784c651f'
This commit is contained in:
Vendored
+254
@@ -0,0 +1,254 @@
|
||||
# Package media - Specification
|
||||
|
||||
Owned directory: `src/vendors/media/`. Namespace `molo::media`. One subdirectory
|
||||
per provider, one shared `common/`. Depends on `molo::llm` (`ApiClient`,
|
||||
`Service`, `ApiError`, `Config`), `molo::clients` and `security/crypto`
|
||||
(Base64, Sha256, Hmac). Plan: `plans/18-breadth.md` section 2, method of
|
||||
`plans/17-llm-vendor-platforms.md`. Demo: `example/app/ai/MediaDemo.cpp`,
|
||||
routes `/ai/media`. Nothing in `src/llm` was edited.
|
||||
|
||||
## 1. Scope and module map
|
||||
|
||||
Twelve providers, one call table each (`<p>/calls.cpp`, checked by
|
||||
`tools/check_spec.py`), one client class per host, generator adapters on a
|
||||
neutral seam.
|
||||
|
||||
| Directory | Classes | Covers |
|
||||
|---|---|---|
|
||||
| `common/` | `CallTable`, `MediaService`, `PreparedCall`, `ImageGenerator`, `VideoGenerator`, `JobTicket`, `JobStatus`, `JobRunner`, `Downloader`, `OutputScanner`, `MediaRegistry`, `ClientFactory`, `AdapterSupport`, `MediaCaps` | the seam, request building, polling, downloads, provider list |
|
||||
| `stability/` | `StabilityClient`, `StabilityImageGenerator` | v2beta Stable Image (generate ultra / core / sd3, edit x7, upscale x3, control x4), 3D (2), Stable Audio (6 + results), v1 engines list, account, balance, v1 text-to-image / image-to-image / masking |
|
||||
| `bfl/` | `BflClient`, `BflImageGenerator`, `BflVideoGenerator` | all 25 FLUX model and tool endpoints, get_result polling, credits, fine-tune list / details / delete |
|
||||
| `runway/` | `RunwayClient`, `RunwayImageGenerator`, `RunwayVideoGenerator` | every SDK route: generation (18), recipes (7), tasks, uploads, organization, avatars, voices, documents, routers, workflows, realtime sessions |
|
||||
| `luma/` | `LumaClient`, `LumaVideoGenerator`, `LumaImageGenerator` | Ray video, Photon image, reframe, modify, upscale, audio, list, delete, concepts, credits |
|
||||
| `replicate/` | `ReplicateClient`, `ReplicateWebhook`, `ReplicateJobs`, `ReplicateImageGenerator`, `ReplicateVideoGenerator` | all 36 operations: predictions (model, version, deployment), models, versions, deployments, files, trainings, hardware, collections, search, account, webhook secret; Standard Webhooks verification |
|
||||
| `fal/` | `FalQueue`, `FalPlatform`, `FalImageGenerator`, `FalVideoGenerator` | queue API (submit, submit with path, status, result, cancel) and the model-related Platform API (search, pricing, estimate, usage, analytics, billing events, request listing, payload deletion, meta) |
|
||||
| `ideogram/` | `IdeogramClient`, `IdeogramImageGenerator`, `IdeogramVideoGenerator` | all 70 v2 operations (image, SVG, video, ad tools, generations, account); free price quote through `dry_run=true` |
|
||||
| `recraft/` | `RecraftClient`, `RecraftImageGenerator` | all 22 operations: generation (3 routes), image tools, styles, prompt enhancement, user |
|
||||
| `leonardo/` | `LeonardoClient`, `LeonardoImageGenerator`, `LeonardoVideoGenerator` | v2 async and sync generations, models; v1 generation get and `me` (prose only) |
|
||||
| `minimax/` | `MinimaxClient`, `MinimaxVideoGenerator` | Hailuo video v1 (create, query, file retrieve) and v2 (create, query, list, delete) |
|
||||
| `kling/` | `KlingJwt`, `KlingClient`, `KlingVideoGenerator` | text2video, image2video, image generation, task queries; JWT auth |
|
||||
| `higgsfield/` | `HiggsfieldClient`, `HiggsfieldJobs`, `HiggsfieldImageGenerator`, `HiggsfieldVideoGenerator` | the shared lifecycle (submit by model path, status, cancel, delete) for every model |
|
||||
|
||||
Products without a public API (not implemented, shown in
|
||||
`MediaRegistry::unsupported()` and on the demo page):
|
||||
|
||||
- **Midjourney**: no public API. Its docs describe the web and Discord products
|
||||
only; third-party relays are unofficial and break its terms.
|
||||
- **Pika**: `pika.art/api` advertises a membership API that resells other
|
||||
vendors' models, but no endpoint, schema or auth document exists
|
||||
(2026-10-25). Pika models are reachable through fal.
|
||||
|
||||
## 2. The seam (`common/job.h`)
|
||||
|
||||
```
|
||||
ImageRequest / VideoRequest neutral request, validate() with caps
|
||||
ImageGenerator::plan(req) exact request as JSON, no network, no credential
|
||||
ImageGenerator::submit(req, confirm) JobTicket (confirm has no default)
|
||||
MediaGenerator::poll(ticket) JobStatus {queued|running|succeeded|failed|cancelled}
|
||||
MediaGenerator::cancel(ticket, confirm) false when the API has no cancel
|
||||
JobRunner::wait(generator, ticket) polls under hard caps
|
||||
Downloader::fetch(asset) bytes of a URL or an inline base64 output
|
||||
```
|
||||
|
||||
Synchronous providers (Stability, Recraft, Ideogram by default) return an
|
||||
`immediate` ticket that already carries its assets; `JobRunner` answers those
|
||||
without a poll. A failed job is a `JobStatus`, a transport or auth failure is an
|
||||
`ApiError`.
|
||||
|
||||
## 3. Static caps (see `common/caps.h`)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Prompt | 32000 chars |
|
||||
| Body fields / one text field | 128 / 1 MiB |
|
||||
| Uploaded file / files per call | 16 MiB / 16 |
|
||||
| Query args / value | 16 / 1024 chars |
|
||||
| Path argument / segments of a multi-segment id | 256 chars / 6 |
|
||||
| Extra headers | 8, 512 chars each |
|
||||
| Images per request / video seconds / dimension | 8 / 60 / 8192 |
|
||||
| Poll interval floor / ceiling | 500 ms / 30000 ms |
|
||||
| Poll attempts / total wait | 240 / 1800 s (demo: 30 / 60 s) |
|
||||
| Assets per job / download / URL | 16 / 128 MiB / 4096 chars |
|
||||
| Scan depth / nodes | 8 / 4096 |
|
||||
| Page walk / items returned | 20 / 1000 |
|
||||
| Webhook tolerance / body | 300 s / 1 MiB |
|
||||
| Call table | 256 rows |
|
||||
|
||||
`src/llm` caps still apply (32 MiB request, 64 MiB response).
|
||||
|
||||
## 4. Contracts
|
||||
|
||||
- Every call lives in the provider's table: key, method, path, spec operation
|
||||
id, body kind (`none|json|multipart|either`), accepted body fields, accepted
|
||||
query parameters, effect (`read|paid|write|delete`).
|
||||
- `MediaService::prepare` is pure. It rejects unknown keys, missing or extra
|
||||
path arguments, ids outside `[A-Za-z0-9._:=~-]` (and `.` / `..`), unknown query
|
||||
parameters, unknown body fields, files on a JSON row, empty or oversized
|
||||
files, bad header names. A rejected request never reaches the network.
|
||||
- `MediaService::execute` refuses any non-read effect unless `confirmed` is
|
||||
true. Typed methods that delete or cancel take `confirm` with no default.
|
||||
The only two lowered effects are named and documented: Ideogram `quote`
|
||||
(`dry_run=true`, defined by the spec as not billed) and Runway
|
||||
`organization_usage` (a POST that only queries).
|
||||
- No key: `configured()` is false and every call throws `NotConfiguredError`
|
||||
(503) before any traffic. Keys travel in headers only.
|
||||
- Lists report truncation (`"truncated": true`), never silently.
|
||||
- Poll caps: the runner never goes below the interval floor, throws
|
||||
`WAIT_EXPIRED` (504) naming the job when attempts or total wait run out, and
|
||||
never cancels on its own.
|
||||
- Errors stay `ApiError` with the vendor's status. Envelopes that hide an error
|
||||
in a 200 are converted: MiniMax `base_resp.status_code`, Kling `code`.
|
||||
|
||||
## 5. Credentials
|
||||
|
||||
| Provider | Variables | Page |
|
||||
|---|---|---|
|
||||
| Stability AI | `STABILITY_API_KEY` | https://platform.stability.ai/account/keys (read: redirects to sign-in) |
|
||||
| Black Forest Labs | `BFL_API_KEY` | https://dashboard.bfl.ai/ (read: redirects to sign-in) |
|
||||
| Runway | `RUNWAY_API_KEY` (also `RUNWAYML_API_SECRET`, the SDK name) | https://dev.runwayml.com/ (read: 200) |
|
||||
| Luma | `LUMA_API_KEY` | https://lumalabs.ai/api/keys (read: 200) |
|
||||
| Replicate | `REPLICATE_API_TOKEN` | https://replicate.com/account/api-tokens (read: redirects to sign-in) |
|
||||
| fal | `FAL_KEY` | https://fal.ai/dashboard/keys (read: redirects to sign-in) |
|
||||
| Ideogram | `IDEOGRAM_API_KEY` | https://ideogram.ai/manage-api (unverified: Cloudflare challenge blocked the fetch) |
|
||||
| Recraft | `RECRAFT_API_TOKEN` | https://www.recraft.ai/profile/api (read: redirects to sign-in with that callback) |
|
||||
| Leonardo | `LEONARDO_API_KEY` | https://app.leonardo.ai/api-access (docs; fetch answered 429) |
|
||||
| MiniMax | `MINIMAX_API_KEY` | https://platform.minimax.io/user-center/basic-information/interface-key (read: redirects to /console/access) |
|
||||
| Kling | `KLING_ACCESS_KEY` and `KLING_SECRET_KEY` | https://app.klingai.com/global/dev/api-key (read: redirect) |
|
||||
| Higgsfield | `HF_API_KEY_ID` and `HF_API_KEY_SECRET` | https://cloud.higgsfield.ai/ (docs; read: redirect) |
|
||||
|
||||
Add these rows to `src/llm/SPEC.md` section 5 at integration.
|
||||
|
||||
## 6. Sources
|
||||
|
||||
Per provider, the machine-readable source `tools/check_spec.py` reads.
|
||||
|
||||
| Provider | Source |
|
||||
|---|---|
|
||||
| Stability | https://docs.stability.ai/openapi.json (v2beta) and https://docs.stability.ai/openapi-v1.json; prose index https://docs.stability.ai/llms.txt |
|
||||
| BFL | https://api.bfl.ai/openapi.json (version 0.0.1) |
|
||||
| Replicate | https://api.replicate.com/openapi.json (server `https://api.replicate.com/v1`) |
|
||||
| fal | queue: https://fal.ai/docs/api-reference/inference-api/openapi/queue.json; platform: https://api.fal.ai/v1/openapi.json; index https://docs.fal.ai/llms.txt |
|
||||
| Ideogram | https://developer.ideogram.ai/v2/openapi.json |
|
||||
| Recraft | https://www.recraft.ai/docs/api-reference/openapi.json |
|
||||
| Higgsfield | https://docs.higgsfield.ai/docs/openapi.json (declared supplementary by the vendor) plus https://docs.higgsfield.ai/docs/llms.txt |
|
||||
| Leonardo | OpenAPI embedded in https://docs.leonardo.ai/reference/creategeneration.md, `createsyncgeneration.md`, `getmodels.md`; quick start `docs/getting-started.md`, FAQ `docs/api-faq.md` |
|
||||
| MiniMax | OpenAPI embedded in the `platform.minimax.io/docs/api-reference/video-generation-*.md` pages and `file-management-retrieve.md`; index `llms.txt` |
|
||||
| Runway | npm `@runwayml/sdk` 4.24.0: `resources/**/*.js` (paths), `*.d.ts` (fields), `client.js` (base URL, `X-Runway-Version` default 2024-11-06) |
|
||||
| Luma | npm `lumaai` 1.19.1: `resources/**/*.js`, `*.d.ts`, `index.js` (base URL) |
|
||||
| Kling | none machine-readable; community sources (a third-party OpenAPI listing, a skill pack) plus live probes |
|
||||
|
||||
Cross-check: Stability, BFL, Replicate, Ideogram and Recraft are also covered by
|
||||
live unauthenticated probes of the same paths (section 8). Standard Webhooks
|
||||
vector for the Replicate verifier comes from the Standard Webhooks project.
|
||||
|
||||
## 7. Verified against spec
|
||||
|
||||
`python3 -I src/vendors/media/tools/check_spec.py`, run 2026-10-25 (twelve
|
||||
providers, all green; a deliberately corrupted BFL row failed it with both a row
|
||||
error and a coverage error, then was restored):
|
||||
|
||||
| Provider | Rows | Source operations | Result |
|
||||
|---|---|---|---|
|
||||
| stability | 34 | 37 (3 alpha operations excluded) | ok |
|
||||
| bfl | 30 | 30 | ok |
|
||||
| replicate | 36 | 36 | ok |
|
||||
| fal | 16 | every queue and model-related platform operation (1 SSE stream route excluded; 78 platform operations outside model inference out of scope) | ok |
|
||||
| ideogram | 70 | 70 | ok |
|
||||
| recraft | 22 | 22 | ok |
|
||||
| leonardo | 6 | 3 (3 v1 rows prose only, listed by the checker) | ok |
|
||||
| runway | 66 | 66 SDK calls | ok |
|
||||
| luma | 13 | 13 SDK calls | ok |
|
||||
| minimax | 7 | 7 (5 reference pages listed as not implemented) | ok |
|
||||
| higgsfield | 10 | 9 (1 generic row by design) | ok |
|
||||
| kling | 6 | probe only | UNVERIFIED |
|
||||
|
||||
The checker verifies: each row's method and path exist, operation ids match,
|
||||
body kind matches the spec content types, every body field and query parameter
|
||||
we send exists, each response schema field we read is reachable, and every
|
||||
source operation is in the table or in the explicit exclusion list.
|
||||
|
||||
## 8. Verified live
|
||||
|
||||
Unauthenticated probes, 2026-10-25, recorded by `probe` runs and by
|
||||
`tests/src/vendors/media/test_live.cpp` (`MOLO_LIVE_NET=1`, 13 providers, green):
|
||||
|
||||
| Host and path | Invalid key answered |
|
||||
|---|---|
|
||||
| api.stability.ai `/v1/user/balance` | 401 `{"id","message","name"}` |
|
||||
| api.stability.ai `/v2beta/results/00000000` | 400 `{"errors":["id: string must contain exactly 64 character(s)"],...}` (the result id is 64 characters) |
|
||||
| api.bfl.ai `/v1/credits` | 422 `{"detail":"Invalid API key format"}` |
|
||||
| api.replicate.com `/v1/account` | 401 `{"title","detail","status"}` |
|
||||
| api.fal.ai `/v1/models/pricing` | 401 `{"error":{"type":"authorization_error",...}}` |
|
||||
| queue.fal.run `/fal-ai/flux/requests/<id>/status` | 401 `{"detail":"Authentication is required"}` |
|
||||
| api.fal.ai `/v1/models`, `/v1/meta` | 200 without any key (public) |
|
||||
| api.ideogram.ai `/v2/account/api-keys` | 401 `{"message":"Access denied..."}`; `/v2/account/usage` answers 400 for the missing `start_time` before auth |
|
||||
| external.api.recraft.ai `/v1/users/me` | 401 `request unauthorized` (plain text) |
|
||||
| cloud.leonardo.ai `/api/rest/v2/models`, `/v1/me` | 401 `{"error","path","code":"access-denied"}` |
|
||||
| api.dev.runwayml.com `/v1/organization` | 401 `{"error":"...did not start with key_","docUrl"}` |
|
||||
| api.lumalabs.ai `/dream-machine/v1/credits` | 403 `{"detail":"Not authenticated"}` |
|
||||
| api.minimax.io `/v1/query/video_generation` | 200 with `base_resp.status_code 1004` (converted to an ApiError) |
|
||||
| api-singapore.klingai.com `/v1/videos/text2video/...` | 401 `{"code":1002,...}`; api.klingai.com answers `{"code":1000,"message":"Auth failed"}` |
|
||||
| api.higgsfield.ai `/requests/<id>/status` | 401 `{"detail":"Invalid credentials"}` |
|
||||
|
||||
No success body of any provider has been seen: no key exists here. Live tests
|
||||
with keys (`MOLO_LIVE_NET=1` plus the provider variables) run one free read per
|
||||
provider (`MediaRegistry::account`). MiniMax, Kling and Higgsfield have no free
|
||||
account read, so a key is never proven for them without a paid or resource call.
|
||||
|
||||
## 9. Unverified
|
||||
|
||||
Every shape inferred rather than read from a spec or seen:
|
||||
|
||||
- All success and result bodies, for every provider (no key). Status words and
|
||||
result locations come from the specs or SDK types where they exist (BFL
|
||||
`StatusResponse` enum and `result.sample`, fal `QueueStatus`, Ideogram
|
||||
`GenerationResponse`, Higgsfield `RequestStatus`, Luma and Runway SDK
|
||||
types, MiniMax pages) and are exercised by local mapper tests only.
|
||||
- **Kling**: all six paths, the request fields, the `{code,data:{task_id,
|
||||
task_status,task_result}}` envelope, the statuses and the JWT claims
|
||||
(`iss`, `exp`, `nbf`, 1800 s) come from community sources, not from Kling.
|
||||
Only the paths and the auth refusal were probed. The JWT builder matches an
|
||||
independent HS256 implementation but Kling's acceptance of it was never seen.
|
||||
- **Leonardo**: the polling route `GET /api/rest/v1/generations/{id}`, its
|
||||
`generations_by_pk.{status,generated_images[].url}` shape, `GET /v1/me`, and
|
||||
the v1 create body are described by quick start and FAQ prose only (the v1
|
||||
reference pages were removed). Whether a v2 `generationId` is pollable on the
|
||||
v1 route was not confirmed.
|
||||
- **Higgsfield**: only five model paths are in the OpenAPI document; any other
|
||||
path goes through the generic row (`model.submit`) with an open body.
|
||||
- **fal**: result URL locations differ per model; `OutputScanner` finds URLs
|
||||
generically. A failed request is assumed to answer the result fetch with an
|
||||
HTTP 4xx (the queue spec says `COMPLETED` also covers failure).
|
||||
- **BFL**: `polling_url` may point at a regional host; polling uses
|
||||
`get_result?id=` on the configured host instead.
|
||||
- **Recraft** default raster output is lossless WebP (docs); inline results are
|
||||
labelled `image/webp` on that basis.
|
||||
- **Replicate** model names listed by `models()` and the other adapters'
|
||||
`models()` lists are names seen in the docs and specs, not a live catalogue.
|
||||
- Runway `X-Runway-Version` 2024-11-06 is the SDK default; newer versions may
|
||||
change bodies.
|
||||
- Console URLs in section 5 marked as such.
|
||||
|
||||
## 10. Gaps
|
||||
|
||||
- Foundation (`src/llm`, report to its owner): no unauthenticated requests (so
|
||||
fal's public model search needs a key through our client); no GET stream (fal
|
||||
status stream, excluded); per-call timeouts missing; `error_from_response`
|
||||
leaves `type` empty for `{"message"}`, `{"detail"}` and `{"errors"}`
|
||||
envelopes (Stability, BFL, Replicate, Higgsfield, Luma, Runway answer
|
||||
`UNKNOWN`); the 32 MiB request ceiling blocks large video uploads (package
|
||||
files are capped at 16 MiB each); `post_multipart` is bypassed through
|
||||
`request_raw`.
|
||||
- Another suite in the shared test binary unsets `MOLO_LIVE_NET`
|
||||
(`tests/src/google/optimize/test_gaps.cpp`); the media live tests read it once
|
||||
at startup to be immune.
|
||||
- Not implemented: Stability gRPC and v2beta image-to-video (absent from the
|
||||
published spec); MiniMax Video Agent, H3-Context-IR and regeneration tasks;
|
||||
fal Platform assets / serverless / compute / keys / workflows; Replicate list
|
||||
cursors (first page only, `truncated` is set); Ideogram and Replicate webhook
|
||||
receivers (Replicate verification exists); WebSocket / realtime routes of
|
||||
Runway beyond session creation.
|
||||
- Recipes and avatar conversations of Runway are table rows only, with no typed
|
||||
wrapper: use `MediaService::call_json`.
|
||||
- Fetching a result needs a second step: `Downloader` is separate from
|
||||
`poll()` so that callers choose when to spend bandwidth.
|
||||
Vendored
+146
@@ -0,0 +1,146 @@
|
||||
#include "vendors/media/bfl/bfl.h"
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs with_body(const Json& body) {
|
||||
CallArgs a;
|
||||
a.body = body;
|
||||
return a;
|
||||
}
|
||||
|
||||
JobTicket ticket_from(const Json& out, const std::string& model, MediaKind kind) {
|
||||
JobTicket t;
|
||||
t.provider = "bfl";
|
||||
t.model = model;
|
||||
t.kind = kind;
|
||||
t.id = AdapterSupport::str(out, "id");
|
||||
if (t.id.empty()) throw llm::ApiError("bfl", 502, "BAD_RESPONSE", "no id in the submit reply");
|
||||
t.meta = Json::object();
|
||||
AdapterSupport::put(t.meta, "cost", AdapterSupport::str(out, "cost"));
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus status_of(BflClient& client, const JobTicket& ticket, MediaKind kind) {
|
||||
CallArgs a;
|
||||
a.query["id"] = ticket.id;
|
||||
Json out = client.call_json("get_result", a);
|
||||
JobStatus s;
|
||||
s.raw_state = AdapterSupport::str(out, "status");
|
||||
s.state = BflImageGenerator::state_of(s.raw_state);
|
||||
if (const Json* p = out.find("progress"); p && p->is_number()) s.progress = p->as_number();
|
||||
if (s.state == JobState::kSucceeded) {
|
||||
if (const Json* r = out.find("result")) s.assets = OutputScanner::scan(*r, kind);
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "Ready but no result URL in the reply";
|
||||
}
|
||||
} else if (s.state == JobState::kFailed) {
|
||||
s.error = s.raw_state;
|
||||
if (const Json* d = out.find("details"); d && d->is_object()) s.error += ": " + d->dump();
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
PreparedCall BflClient::plan(const std::string& key, const Json& body) const { return prepare(key, with_body(body)); }
|
||||
|
||||
Json BflClient::submit(const std::string& key, const Json& body, bool confirm) {
|
||||
return call_json(key, with_body(body), confirm);
|
||||
}
|
||||
|
||||
Json BflClient::get_result(const std::string& id) {
|
||||
CallArgs a;
|
||||
a.query["id"] = id;
|
||||
return call_json("get_result", a);
|
||||
}
|
||||
|
||||
Json BflClient::credits() { return call_json("credits"); }
|
||||
Json BflClient::my_finetunes() { return call_json("my_finetunes"); }
|
||||
|
||||
Json BflClient::finetune_details(const std::string& finetune_id) {
|
||||
CallArgs a;
|
||||
a.query["finetune_id"] = finetune_id;
|
||||
return call_json("finetune_details", a);
|
||||
}
|
||||
|
||||
Json BflClient::delete_finetune(const std::string& finetune_id, bool confirm) {
|
||||
Json b = Json::object();
|
||||
b.set("finetune_id", Json(finetune_id));
|
||||
return call_json("delete_finetune", with_body(b), confirm);
|
||||
}
|
||||
|
||||
JobState BflImageGenerator::state_of(const std::string& status) {
|
||||
if (status == "Ready") return JobState::kSucceeded;
|
||||
if (status == "Pending" || status == "Reasoning") return JobState::kQueued;
|
||||
if (status == "Generating") return JobState::kRunning;
|
||||
if (status == "Task not found" || status == "Error" || status == "Request Moderated" ||
|
||||
status == "Content Moderated" || status == "No Card Generated")
|
||||
return JobState::kFailed;
|
||||
return JobState::kRunning;
|
||||
}
|
||||
|
||||
std::vector<std::string> BflImageGenerator::models() const {
|
||||
return {"flux-2-pro", "flux-2-max", "flux-2-flex", "flux-2-klein-9b", "flux-2-klein-4b", "flux-kontext-pro",
|
||||
"flux-kontext-max", "flux-pro-1.1", "flux-pro-1.1-ultra", "flux-dev", "flux-3-image"};
|
||||
}
|
||||
|
||||
Json BflImageGenerator::body(const ImageRequest& r, std::string& key) const {
|
||||
r.validate();
|
||||
key = r.model.empty() ? "flux-2-pro" : r.model;
|
||||
if (r.count != 1) throw llm::ApiError::invalid("bfl", "one image per call (count must be 1)");
|
||||
Json b = Json::object();
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
AdapterSupport::put_int(b, "width", r.width);
|
||||
AdapterSupport::put_int(b, "height", r.height);
|
||||
AdapterSupport::put(b, "aspect_ratio", r.aspect_ratio);
|
||||
AdapterSupport::put_seed(b, "seed", r.seed);
|
||||
AdapterSupport::put(b, "input_image", r.source_url);
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json BflImageGenerator::plan(const ImageRequest& request) const {
|
||||
std::string key;
|
||||
Json b = body(request, key);
|
||||
return client_->plan(key, b).summary();
|
||||
}
|
||||
|
||||
JobTicket BflImageGenerator::submit(const ImageRequest& request, bool confirm) {
|
||||
std::string key;
|
||||
Json b = body(request, key);
|
||||
AdapterSupport::need_confirm(confirm, "bfl", "generate");
|
||||
return ticket_from(client_->submit(key, b, true), key, MediaKind::kImage);
|
||||
}
|
||||
|
||||
JobStatus BflImageGenerator::poll(const JobTicket& ticket) { return status_of(*client_, ticket, MediaKind::kImage); }
|
||||
|
||||
Json BflVideoGenerator::body(const VideoRequest& r) const {
|
||||
r.validate();
|
||||
Json b = Json::object();
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
AdapterSupport::put(b, "aspect_ratio", r.aspect_ratio);
|
||||
AdapterSupport::put_int(b, "duration", r.duration_seconds);
|
||||
AdapterSupport::put(b, "resolution", r.resolution);
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json BflVideoGenerator::plan(const VideoRequest& request) const {
|
||||
return client_->plan("flux-3-video", body(request)).summary();
|
||||
}
|
||||
|
||||
JobTicket BflVideoGenerator::submit(const VideoRequest& request, bool confirm) {
|
||||
Json b = body(request);
|
||||
AdapterSupport::need_confirm(confirm, "bfl", "generate");
|
||||
return ticket_from(client_->submit("flux-3-video", b, true), "flux-3-video", MediaKind::kVideo);
|
||||
}
|
||||
|
||||
JobStatus BflVideoGenerator::poll(const JobTicket& ticket) { return status_of(*client_, ticket, MediaKind::kVideo); }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+76
@@ -0,0 +1,76 @@
|
||||
#pragma once
|
||||
|
||||
// Black Forest Labs (FLUX): submit to a model endpoint, poll /v1/get_result.
|
||||
// Auth is the x-key header. Models are endpoints (flux-2-pro, flux-kontext-pro,
|
||||
// flux-pro-1.1-ultra, ...), tools are endpoints under /v1/flux-tools/.
|
||||
|
||||
#include "vendors/media/bfl/calls.h"
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class BflClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api.bfl.ai";
|
||||
static constexpr const char* kKeyEnv = "BFL_API_KEY";
|
||||
|
||||
explicit BflClient(std::shared_ptr<llm::ApiClient> client) : MediaService(std::move(client), BflCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::header("bfl", kHost, kKeyEnv, "x-key"); }
|
||||
static std::shared_ptr<BflClient> from_env() {
|
||||
return std::make_shared<BflClient>(ClientFactory::real(config(), ClientFactory::env({kKeyEnv})));
|
||||
}
|
||||
static std::shared_ptr<BflClient> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<BflClient>(ClientFactory::with(std::move(t), config(), key));
|
||||
}
|
||||
|
||||
// key = a POST model / tool row ("flux-2-pro", "tools.erase-v1"). Returns {id, polling_url, cost, ...}.
|
||||
Json submit(const std::string& key, const Json& body, bool confirm);
|
||||
PreparedCall plan(const std::string& key, const Json& body) const;
|
||||
Json get_result(const std::string& id);
|
||||
Json credits();
|
||||
Json my_finetunes();
|
||||
Json finetune_details(const std::string& finetune_id);
|
||||
// Destructive: no default for confirm.
|
||||
Json delete_finetune(const std::string& finetune_id, bool confirm);
|
||||
};
|
||||
|
||||
class BflImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit BflImageGenerator(std::shared_ptr<BflClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "bfl"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override;
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket&, bool) override { return false; } // no cancel endpoint
|
||||
int suggested_poll_ms() const override { return 1500; }
|
||||
|
||||
// Maps the BFL status word onto JobState (shared with the video adapter).
|
||||
static JobState state_of(const std::string& status);
|
||||
|
||||
private:
|
||||
Json body(const ImageRequest& request, std::string& key) const;
|
||||
std::shared_ptr<BflClient> client_;
|
||||
};
|
||||
|
||||
class BflVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit BflVideoGenerator(std::shared_ptr<BflClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "bfl"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override { return {"flux-3-video"}; }
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket&, bool) override { return false; }
|
||||
int suggested_poll_ms() const override { return 4000; }
|
||||
|
||||
private:
|
||||
Json body(const VideoRequest& request) const;
|
||||
std::shared_ptr<BflClient> client_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+57
@@ -0,0 +1,57 @@
|
||||
#include "vendors/media/bfl/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"finetune_details", "GET", "/v1/finetune_details", "finetune_details_v1_finetune_details_get", "none", "", "finetune_id", "read"},
|
||||
{"my_finetunes", "GET", "/v1/my_finetunes", "my_finetunes_v1_my_finetunes_get", "none", "", "", "read"},
|
||||
{"delete_finetune", "POST", "/v1/delete_finetune", "delete_finetune_v1_delete_finetune_post", "json", "finetune_id", "", "delete"},
|
||||
{"flux-pro-1.1", "POST", "/v1/flux-pro-1.1", "flux_pro_1_1_v1_flux_pro_1_1_post", "json", "user,prompt,image_prompt,width,height,prompt_upsampling,seed,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-dev", "POST", "/v1/flux-dev", "flux_dev_v1_flux_dev_post", "json", "user,prompt,image_prompt,width,height,steps,prompt_upsampling,seed,guidance,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-pro-1.1-ultra", "POST", "/v1/flux-pro-1.1-ultra", "generate_flux_1_1_ultra_v1_flux_pro_1_1_ultra_post", "json", "user,prompt,prompt_upsampling,seed,aspect_ratio,safety_tolerance,output_format,raw,image_prompt,image_prompt_strength,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-pro-1.0-fill", "POST", "/v1/flux-pro-1.0-fill", "fill_v1_flux_pro_1_0_fill_post", "json", "user,image,mask,prompt,steps,prompt_upsampling,seed,guidance,output_format,safety_tolerance,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-pro-1.0-expand", "POST", "/v1/flux-pro-1.0-expand", "expand_v1_flux_pro_1_0_expand_post", "json", "user,image,top,bottom,left,right,prompt,steps,prompt_upsampling,seed,guidance,output_format,safety_tolerance,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-pro-1.1-ultra-finetuned", "POST", "/v1/flux-pro-1.1-ultra-finetuned", "generate_bigblue_finetuned_v1_flux_pro_1_1_ultra_finetuned_post", "json", "user,finetune_id,finetune_strength,prompt,seed,aspect_ratio,safety_tolerance,output_format,image_prompt,image_prompt_strength,prompt_upsampling,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-2-flex", "POST", "/v1/flux-2-flex", "generate_flux_2_flex_v1_flux_2_flex_post", "json", "user,prompt,prompt_upsampling,input_image,input_image_2,input_image_3,input_image_4,input_image_5,input_image_6,input_image_7,input_image_8,input_image_blob_path,seed,width,height,guidance,steps,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-2-klein-9b", "POST", "/v1/flux-2-klein-9b", "generate_flux_2_klein_9b_v1_flux_2_klein_9b_post", "json", "user,prompt,input_image,input_image_2,input_image_3,input_image_4,seed,width,height,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-2-klein-9b-preview", "POST", "/v1/flux-2-klein-9b-preview", "generate_flux_2_klein_9b_kv_v1_flux_2_klein_9b_preview_post", "json", "user,prompt,input_image,input_image_2,input_image_3,input_image_4,seed,width,height,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-2-klein-4b", "POST", "/v1/flux-2-klein-4b", "generate_flux_2_klein_4b_v1_flux_2_klein_4b_post", "json", "user,prompt,input_image,input_image_2,input_image_3,input_image_4,seed,width,height,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-2-max", "POST", "/v1/flux-2-max", "generate_flux_2_max_v1_flux_2_max_post", "json", "user,prompt,disable_pup,input_image,input_image_2,input_image_3,input_image_4,input_image_5,input_image_6,input_image_7,input_image_8,seed,width,height,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-2-pro-preview", "POST", "/v1/flux-2-pro-preview", "generate_flux_2_pro_preview_v1_flux_2_pro_preview_post", "json", "user,prompt,disable_pup,input_image,input_image_2,input_image_3,input_image_4,input_image_5,input_image_6,input_image_7,input_image_8,seed,width,height,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-2-pro", "POST", "/v1/flux-2-pro", "generate_flux_2_pro_v1_flux_2_pro_post", "json", "user,prompt,disable_pup,input_image,input_image_2,input_image_3,input_image_4,input_image_5,input_image_6,input_image_7,input_image_8,seed,width,height,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-3-video", "POST", "/v1/flux-3-video", "flux_3_video_v1_flux_3_video_post", "json", "user,prompt,aspect_ratio,duration,resolution,version,generate_audio,safety_tolerance,draft,mode,keyframes,start_video,draft_cache", "", "paid"},
|
||||
{"flux-3-image", "POST", "/v1/flux-3-image", "flux_3_image_v1_flux_3_image_post", "json", "user,prompt,images,aspect_ratio,resolution,width,height,safety_tolerance,grounding,version", "", "paid"},
|
||||
{"flux-3-image-klein", "POST", "/v1/flux-3-image-klein", "flux_3_image_klein_v1_flux_3_image_klein_post", "json", "user,prompt,images,aspect_ratio,resolution,width,height,seed,safety_tolerance,output_format,steps", "", "paid"},
|
||||
{"tools.video-upscale-v1", "POST", "/v1/flux-tools/video-upscale-v1", "generate_flux_tools_video_upscale_v1_v1_flux_tools_video_upscale_v1_post", "json", "user,input_video,prompt,creativity,upscale_factor,safety_tolerance,webhook_url,webhook_secret", "", "paid"},
|
||||
{"tools.video-edit-v1", "POST", "/v1/flux-tools/video-edit-v1", "generate_flux_tools_video_edit_v1_v1_flux_tools_video_edit_v1_post", "json", "user,video,prompt,safety_tolerance", "", "paid"},
|
||||
{"tools.outpainting-v1", "POST", "/v1/flux-tools/outpainting-v1", "generate_flux_tools_outpainting_v1_v1_flux_tools_outpainting_v1_post", "json", "user,input_image,width,height,auto_crop,safety_tolerance,output_format,prompt,reference_offset_x,reference_offset_y,mode,disable_pup", "", "paid"},
|
||||
{"tools.vto-v1", "POST", "/v1/flux-tools/vto-v1", "generate_flux_tools_vto_v1_v1_flux_tools_vto_v1_post", "json", "user,prompt,person,garment,seed,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"tools.vto-v2", "POST", "/v1/flux-tools/vto-v2", "generate_flux_tools_vto_v2_v1_flux_tools_vto_v2_post", "json", "user,prompt,person,garment,seed,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"tools.erase-v1", "POST", "/v1/flux-tools/erase-v1", "generate_flux_tools_erase_v1_v1_flux_tools_erase_v1_post", "json", "user,image,mask,dilate_pixels,seed,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"tools.deblur-v1", "POST", "/v1/flux-tools/deblur-v1", "generate_flux_tools_deblur_v1_v1_flux_tools_deblur_v1_post", "json", "user,image,seed,safety_tolerance,output_format,webhook_url,webhook_secret", "", "paid"},
|
||||
{"flux-kontext-max", "POST", "/v1/flux-kontext-max", "generate_flux_kontext_max_v1_flux_kontext_max_post", "json", "user,prompt,input_image,input_image_2,input_image_3,input_image_4,seed,aspect_ratio,output_format,webhook_url,webhook_secret,prompt_upsampling,safety_tolerance", "", "paid"},
|
||||
{"flux-kontext-pro", "POST", "/v1/flux-kontext-pro", "generate_flux_kontext_pro_v1_flux_kontext_pro_post", "json", "user,prompt,input_image,input_image_2,input_image_3,input_image_4,seed,aspect_ratio,output_format,webhook_url,webhook_secret,prompt_upsampling,safety_tolerance", "", "paid"},
|
||||
{"get_result", "GET", "/v1/get_result", "get_result_v1_get_result_get", "none", "", "id", "read"},
|
||||
{"credits", "GET", "/v1/credits", "get_credits_v1_credits_get", "none", "", "", "read"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
{"AsyncResponse", "id,polling_url,cost,input_mp,output_mp"},
|
||||
{"ResultResponse", "id,status,result,progress,details,preview"},
|
||||
{"CreditsResponse", "credits"},
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& BflCalls::table() {
|
||||
static const CallTable t("bfl", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t BflCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* BflCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct BflCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+50
@@ -0,0 +1,50 @@
|
||||
#pragma once
|
||||
|
||||
// Small helpers every generator adapter shares: the confirmation gate, body
|
||||
// assembly that skips unset values, and extra-field merging.
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "llm/error.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct AdapterSupport {
|
||||
// Every submit / cancel passes through here first: no confirm, no call.
|
||||
static void need_confirm(bool confirm, const std::string& provider, const std::string& what) {
|
||||
if (!confirm)
|
||||
throw llm::ApiError::invalid(provider, what + " needs explicit confirmation (it costs money or changes a resource)");
|
||||
}
|
||||
|
||||
static void put(clients::body::Json& obj, const std::string& key, const std::string& value) {
|
||||
if (!value.empty()) obj.set(key, clients::body::Json(value));
|
||||
}
|
||||
static void put_int(clients::body::Json& obj, const std::string& key, int value) {
|
||||
if (value > 0) obj.set(key, clients::body::Json(value));
|
||||
}
|
||||
static void put_seed(clients::body::Json& obj, const std::string& key, long seed) {
|
||||
if (seed >= 0) obj.set(key, clients::body::Json(static_cast<double>(seed)));
|
||||
}
|
||||
|
||||
// Provider fields from `extra` win over derived ones (the caller asked).
|
||||
static void merge_extra(clients::body::Json& obj, const clients::body::Json& extra) {
|
||||
if (!extra.is_object()) return;
|
||||
for (const auto& [k, v] : extra.as_object()) obj.set(k, v);
|
||||
}
|
||||
|
||||
static std::string str(const clients::body::Json& doc, const std::string& key) {
|
||||
const clients::body::Json* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
if (v && v->is_string()) return v->as_string();
|
||||
if (v && v->is_number()) return v->dump();
|
||||
return "";
|
||||
}
|
||||
|
||||
static const clients::body::Json* obj_at(const clients::body::Json& doc, const std::string& key) {
|
||||
const clients::body::Json* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
return v && v->is_object() ? v : nullptr;
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+162
@@ -0,0 +1,162 @@
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
#include <cstring>
|
||||
#include <stdexcept>
|
||||
|
||||
#include "llm/error.h"
|
||||
#include "vendors/media/common/caps.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
bool one_of(const char* value, std::initializer_list<const char*> options) {
|
||||
for (const char* o : options)
|
||||
if (std::strcmp(value, o) == 0) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
CallTable::CallTable(const char* vendor, const CallRow* rows, std::size_t count)
|
||||
: vendor_(vendor), rows_(rows), count_(count) {
|
||||
if (count > kMaxRows) throw std::logic_error(vendor_ + ": call table over " + std::to_string(kMaxRows) + " rows");
|
||||
for (std::size_t i = 0; i < count; ++i) {
|
||||
const CallRow& r = rows[i];
|
||||
if (!index_.emplace(r.key, i).second) throw std::logic_error(vendor_ + ": duplicate call key " + r.key);
|
||||
if (!one_of(r.http_method, {"GET", "POST", "PUT", "PATCH", "DELETE"}))
|
||||
throw std::logic_error(vendor_ + ": bad method on " + r.key);
|
||||
if (!one_of(r.body_kind, {"none", "json", "multipart", "either", "form"}))
|
||||
throw std::logic_error(vendor_ + ": bad body kind on " + r.key);
|
||||
if (!one_of(r.effect, {"read", "paid", "write", "delete"}))
|
||||
throw std::logic_error(vendor_ + ": bad effect on " + r.key);
|
||||
if (r.path[0] != '/') throw std::logic_error(vendor_ + ": path must start with / on " + r.key);
|
||||
}
|
||||
}
|
||||
|
||||
const CallRow& CallTable::at(std::size_t index) const {
|
||||
if (index >= count_) throw llm::ApiError::invalid(vendor_, "call index out of range");
|
||||
return rows_[index];
|
||||
}
|
||||
|
||||
const CallRow* CallTable::find(const std::string& key) const {
|
||||
auto it = index_.find(key);
|
||||
return it == index_.end() ? nullptr : &rows_[it->second];
|
||||
}
|
||||
|
||||
const CallRow& CallTable::get(const std::string& key) const {
|
||||
const CallRow* row = find(key);
|
||||
if (!row) throw llm::ApiError::invalid(vendor_, "unknown call '" + key + "'");
|
||||
return *row;
|
||||
}
|
||||
|
||||
Effect CallTable::effect_of(const CallRow& row) {
|
||||
if (std::strcmp(row.effect, "paid") == 0) return Effect::kPaid;
|
||||
if (std::strcmp(row.effect, "write") == 0) return Effect::kWrite;
|
||||
if (std::strcmp(row.effect, "delete") == 0) return Effect::kDelete;
|
||||
return Effect::kRead;
|
||||
}
|
||||
|
||||
const char* CallTable::effect_name(Effect effect) {
|
||||
switch (effect) {
|
||||
case Effect::kPaid: return "paid";
|
||||
case Effect::kWrite: return "write";
|
||||
case Effect::kDelete: return "delete";
|
||||
default: return "read";
|
||||
}
|
||||
}
|
||||
|
||||
clients::Method CallTable::method_of(const CallRow& row) {
|
||||
const char* m = row.http_method;
|
||||
if (std::strcmp(m, "POST") == 0) return clients::Method::kPost;
|
||||
if (std::strcmp(m, "PUT") == 0) return clients::Method::kPut;
|
||||
if (std::strcmp(m, "PATCH") == 0) return clients::Method::kPatch;
|
||||
if (std::strcmp(m, "DELETE") == 0) return clients::Method::kDelete;
|
||||
return clients::Method::kGet;
|
||||
}
|
||||
|
||||
std::vector<std::string> CallTable::split(const char* csv) {
|
||||
std::vector<std::string> out;
|
||||
std::string cur;
|
||||
for (const char* p = csv; *p; ++p) {
|
||||
if (*p == ',') {
|
||||
if (!cur.empty()) out.push_back(cur);
|
||||
cur.clear();
|
||||
} else {
|
||||
cur.push_back(*p);
|
||||
}
|
||||
}
|
||||
if (!cur.empty()) out.push_back(cur);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<std::string> CallTable::placeholders(const char* path) {
|
||||
std::vector<std::string> out;
|
||||
std::string p(path);
|
||||
std::size_t pos = 0;
|
||||
while ((pos = p.find('{', pos)) != std::string::npos) {
|
||||
std::size_t end = p.find('}', pos);
|
||||
if (end == std::string::npos) break;
|
||||
out.push_back(p.substr(pos + 1, end - pos - 1));
|
||||
pos = end + 1;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool CallTable::body_allows(const CallRow& row, const std::string& field) const {
|
||||
for (const std::string& f : split(row.body_fields))
|
||||
if (f == "*" || f == field) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
bool CallTable::query_allows(const CallRow& row, const std::string& name) const {
|
||||
for (const std::string& f : split(row.query_params))
|
||||
if (f == name) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
bool CallTable::valid_segment(const std::string& id) {
|
||||
if (id.empty() || id.size() > MediaCaps::kMaxPathArgChars || id == "." || id == "..") return false;
|
||||
for (char c : id)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' ||
|
||||
c == '_' || c == '.' || c == ':' || c == '=' || c == '~'))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string CallTable::render_path(const CallRow& row,
|
||||
const std::map<std::string, std::string>& args) const {
|
||||
std::vector<std::string> names = placeholders(row.path);
|
||||
for (const auto& [name, value] : args) {
|
||||
bool known = false;
|
||||
for (const std::string& n : names) known = known || n == name;
|
||||
if (!known) throw llm::ApiError::invalid(vendor_, "call '" + std::string(row.key) + "' has no path argument '" + name + "'");
|
||||
(void)value;
|
||||
}
|
||||
std::string path = row.path;
|
||||
for (const std::string& name : names) {
|
||||
auto it = args.find(name);
|
||||
if (it == args.end()) throw llm::ApiError::invalid(vendor_, "call '" + std::string(row.key) + "' needs path argument '" + name + "'");
|
||||
const std::string& value = it->second;
|
||||
bool multi = name == "endpoint_id" || name == "model_path" || name == "path";
|
||||
if (multi) {
|
||||
std::size_t segments = 0, start = 0;
|
||||
while (true) {
|
||||
std::size_t slash = value.find('/', start);
|
||||
std::string seg = value.substr(start, slash == std::string::npos ? std::string::npos : slash - start);
|
||||
if (!valid_segment(seg)) throw llm::ApiError::invalid(vendor_, "bad " + name);
|
||||
if (++segments > MediaCaps::kMaxPathSegments) throw llm::ApiError::invalid(vendor_, name + " has too many segments");
|
||||
if (slash == std::string::npos) break;
|
||||
start = slash + 1;
|
||||
}
|
||||
} else if (!valid_segment(value)) {
|
||||
throw llm::ApiError::invalid(vendor_, "bad " + name);
|
||||
}
|
||||
std::string token = "{" + name + "}";
|
||||
std::size_t at = path.find(token);
|
||||
path.replace(at, token.size(), value);
|
||||
}
|
||||
return path;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
#pragma once
|
||||
|
||||
// One table row per REST call. Every provider has exactly one table (calls.cpp
|
||||
// in its directory); tools/check_spec.py checks each row against the
|
||||
// provider's published spec. Code never builds a path from a literal.
|
||||
|
||||
#include <cstddef>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/request.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// What a call does to the account. Anything but kRead needs explicit
|
||||
// confirmation at execution time (MediaService::execute).
|
||||
enum class Effect { kRead, kPaid, kWrite, kDelete };
|
||||
|
||||
struct CallRow {
|
||||
const char* key; // short stable name, unique within the provider
|
||||
const char* http_method; // GET POST PUT PATCH DELETE
|
||||
const char* path; // from the host, "{placeholder}" segments
|
||||
const char* op; // spec operation id when the spec has one
|
||||
const char* body_kind; // none | json | multipart | either | form
|
||||
const char* body_fields; // comma list of accepted body fields; "*" = open
|
||||
const char* query_params; // comma list of accepted query parameters
|
||||
const char* effect; // read | paid | write | delete
|
||||
};
|
||||
|
||||
class CallTable {
|
||||
public:
|
||||
static constexpr std::size_t kMaxRows = 256;
|
||||
|
||||
// Throws std::logic_error on a duplicate key, a bad method / kind / effect,
|
||||
// or more than kMaxRows rows (a table bug, caught by the unit tests).
|
||||
CallTable(const char* vendor, const CallRow* rows, std::size_t count);
|
||||
|
||||
const std::string& vendor() const { return vendor_; }
|
||||
std::size_t size() const { return count_; }
|
||||
const CallRow& at(std::size_t index) const;
|
||||
const CallRow* find(const std::string& key) const;
|
||||
// Throws llm::ApiError::invalid for an unknown key.
|
||||
const CallRow& get(const std::string& key) const;
|
||||
|
||||
static Effect effect_of(const CallRow& row);
|
||||
static const char* effect_name(Effect effect);
|
||||
static clients::Method method_of(const CallRow& row);
|
||||
static std::vector<std::string> split(const char* csv);
|
||||
static std::vector<std::string> placeholders(const char* path);
|
||||
|
||||
bool body_allows(const CallRow& row, const std::string& field) const;
|
||||
bool query_allows(const CallRow& row, const std::string& name) const;
|
||||
|
||||
// One URL path segment: 1..256 of [A-Za-z0-9._:=~-], not "." or "..".
|
||||
static bool valid_segment(const std::string& id);
|
||||
// Replaces every {placeholder}; each must be supplied, none may be extra.
|
||||
// Placeholders named endpoint_id, model_path or path accept 1..6 "/"-joined
|
||||
// segments, every other placeholder exactly one. Throws ApiError::invalid.
|
||||
std::string render_path(const CallRow& row,
|
||||
const std::map<std::string, std::string>& args) const;
|
||||
|
||||
private:
|
||||
std::string vendor_;
|
||||
const CallRow* rows_;
|
||||
std::size_t count_;
|
||||
std::map<std::string, std::size_t> index_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+47
@@ -0,0 +1,47 @@
|
||||
#pragma once
|
||||
|
||||
// Static ceilings for package media. Never autotuned. Request and response
|
||||
// byte ceilings that come from src/llm (32 MiB / 64 MiB) are not repeated here;
|
||||
// ours are the ones the media workflows add: files, polls, downloads, pages.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct MediaCaps {
|
||||
// Request shaping.
|
||||
static constexpr std::size_t kMaxPromptChars = 32000;
|
||||
static constexpr std::size_t kMaxBodyFields = 128;
|
||||
static constexpr std::size_t kMaxFieldChars = 1u << 20; // one text field, 1 MiB
|
||||
static constexpr std::size_t kMaxFileBytes = 16u * 1024u * 1024u; // one uploaded file
|
||||
static constexpr std::size_t kMaxFiles = 16;
|
||||
static constexpr std::size_t kMaxQueryArgs = 16;
|
||||
static constexpr std::size_t kMaxQueryValueChars = 1024;
|
||||
static constexpr std::size_t kMaxPathArgChars = 256;
|
||||
static constexpr std::size_t kMaxPathSegments = 6; // multi-segment ids (fal endpoints)
|
||||
static constexpr std::size_t kMaxHeaders = 8;
|
||||
static constexpr std::size_t kMaxHeaderValueChars = 512;
|
||||
// Generation limits we enforce before spending money.
|
||||
static constexpr int kMaxImagesPerRequest = 8;
|
||||
static constexpr int kMaxVideoSeconds = 60;
|
||||
static constexpr int kMaxDimension = 8192;
|
||||
// Job polling: floors and ceilings (the machine never goes below / above).
|
||||
static constexpr int kMinPollIntervalMs = 500;
|
||||
static constexpr int kMaxPollIntervalMs = 30000;
|
||||
static constexpr int kMaxPollAttempts = 240;
|
||||
static constexpr int kMaxWaitSeconds = 1800;
|
||||
// Outputs.
|
||||
static constexpr std::size_t kMaxAssetsPerJob = 16;
|
||||
static constexpr std::size_t kMaxDownloadBytes = 128u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxUrlChars = 4096;
|
||||
static constexpr std::size_t kMaxScanDepth = 8;
|
||||
static constexpr std::size_t kMaxScanNodes = 4096;
|
||||
// Listings.
|
||||
static constexpr std::size_t kMaxPages = 20;
|
||||
static constexpr std::size_t kMaxItemsReturned = 1000;
|
||||
// Webhooks.
|
||||
static constexpr long kWebhookToleranceSeconds = 300;
|
||||
static constexpr std::size_t kMaxWebhookBodyBytes = 1u * 1024u * 1024u;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
#pragma once
|
||||
|
||||
// Builds the llm::ApiClient every provider client wraps. One place, so the
|
||||
// credential lookup (environment, then .env.json.local, then .env.json via
|
||||
// llm::Config) and the transport choice are identical for all providers.
|
||||
|
||||
#include <initializer_list>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "clients/transports/http_transport.h"
|
||||
#include "llm/api_client.h"
|
||||
#include "llm/config.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct ClientFactory {
|
||||
// First non-empty value among `names` (llm::Config::get), else "".
|
||||
static std::string env(std::initializer_list<const char*> names) {
|
||||
for (const char* n : names) {
|
||||
std::string v = llm::Config::get(n);
|
||||
if (!v.empty()) return v;
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
// Real HTTPS. An empty key leaves the client "not configured".
|
||||
static std::shared_ptr<llm::ApiClient> real(llm::ApiClientConfig config, const std::string& key) {
|
||||
return std::make_shared<llm::ApiClient>(std::make_unique<clients::transports::HttpTransport>(),
|
||||
std::move(config), key);
|
||||
}
|
||||
|
||||
// Any transport (tests record the requests the real client builds).
|
||||
static std::shared_ptr<llm::ApiClient> with(std::unique_ptr<clients::Transport> transport,
|
||||
llm::ApiClientConfig config, const std::string& key) {
|
||||
return std::make_shared<llm::ApiClient>(std::move(transport), std::move(config), key);
|
||||
}
|
||||
|
||||
static llm::ApiClientConfig bearer(const std::string& vendor, const std::string& base_url,
|
||||
const std::string& key_env) {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = vendor;
|
||||
c.base_url = base_url;
|
||||
c.key_env = key_env;
|
||||
c.auth = llm::AuthStyle::kBearer;
|
||||
return c;
|
||||
}
|
||||
|
||||
static llm::ApiClientConfig header(const std::string& vendor, const std::string& base_url,
|
||||
const std::string& key_env, const std::string& header_name) {
|
||||
llm::ApiClientConfig c = bearer(vendor, base_url, key_env);
|
||||
c.auth = llm::AuthStyle::kHeader;
|
||||
c.auth_header = header_name;
|
||||
return c;
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+74
@@ -0,0 +1,74 @@
|
||||
#include "vendors/media/common/downloader.h"
|
||||
|
||||
#include "clients/transports/http_transport.h"
|
||||
#include "llm/error.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
#include "vendors/media/common/caps.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
clients::ClientConfig plain_config() {
|
||||
clients::ClientConfig c;
|
||||
c.persist_cookies = false;
|
||||
return c;
|
||||
}
|
||||
|
||||
std::unique_ptr<clients::Transport> real_or(std::unique_ptr<clients::Transport> t) {
|
||||
if (t) return t;
|
||||
return std::make_unique<clients::transports::HttpTransport>();
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Downloader::Downloader(std::unique_ptr<clients::Transport> transport)
|
||||
: client_(real_or(std::move(transport)), plain_config()) {}
|
||||
|
||||
void Downloader::validate_url(const std::string& url) {
|
||||
auto bad = [](const std::string& why) { throw llm::ApiError::invalid("media", why); };
|
||||
if (url.empty() || url.size() > MediaCaps::kMaxUrlChars) bad("download url length");
|
||||
if (url.rfind("https://", 0) != 0) bad("download url must be https");
|
||||
for (unsigned char c : url)
|
||||
if (c <= 0x20 || c == 0x7f) bad("download url has whitespace or control characters");
|
||||
std::size_t host_end = url.find_first_of("/?#", 8);
|
||||
std::string authority = url.substr(8, host_end == std::string::npos ? std::string::npos : host_end - 8);
|
||||
if (authority.empty()) bad("download url has no host");
|
||||
if (authority.find('@') != std::string::npos) bad("download url must not carry credentials");
|
||||
}
|
||||
|
||||
std::string Downloader::fetch_url(const std::string& url, std::string* content_type) {
|
||||
validate_url(url);
|
||||
try {
|
||||
clients::ClientRequest req(clients::Method::kGet, clients::Url::parse(url));
|
||||
clients::ClientResponse res = client_.send_blocking(std::move(req));
|
||||
if (!res.is_success())
|
||||
throw llm::ApiError("media", res.status(), "DOWNLOAD_FAILED", "download answered HTTP " + std::to_string(res.status()));
|
||||
if (res.body().size() > MediaCaps::kMaxDownloadBytes)
|
||||
throw llm::ApiError("media", 502, "BAD_RESPONSE", "download over the byte cap");
|
||||
if (content_type) *content_type = res.content_type();
|
||||
return res.body();
|
||||
} catch (const llm::ApiError&) {
|
||||
throw;
|
||||
} catch (const std::exception& e) {
|
||||
throw llm::ApiError("media", 502, "UNAVAILABLE", e.what());
|
||||
}
|
||||
}
|
||||
|
||||
FetchedAsset Downloader::fetch(const MediaAsset& asset) {
|
||||
FetchedAsset out;
|
||||
out.asset = asset;
|
||||
if (!asset.data_base64.empty()) {
|
||||
if (!security::crypto::Base64::decode(asset.data_base64, out.bytes))
|
||||
throw llm::ApiError::invalid("media", "inline asset is not valid base64");
|
||||
if (out.bytes.size() > MediaCaps::kMaxDownloadBytes)
|
||||
throw llm::ApiError("media", 502, "BAD_RESPONSE", "inline asset over the byte cap");
|
||||
out.content_type = asset.mime;
|
||||
return out;
|
||||
}
|
||||
if (asset.url.empty()) throw llm::ApiError::invalid("media", "asset has neither url nor data");
|
||||
out.bytes = fetch_url(asset.url, &out.content_type);
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
#pragma once
|
||||
|
||||
// Downloader: fetch an output (https URL or inline base64) as bytes under the
|
||||
// download cap. Sends no credential: result URLs are public or pre-signed.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "clients/client.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct FetchedAsset {
|
||||
MediaAsset asset;
|
||||
std::string bytes;
|
||||
std::string content_type;
|
||||
};
|
||||
|
||||
class Downloader {
|
||||
public:
|
||||
// Null transport = real HTTPS.
|
||||
explicit Downloader(std::unique_ptr<clients::Transport> transport = nullptr);
|
||||
|
||||
// https only, 1..4096 chars, no userinfo, no whitespace or control chars.
|
||||
// Throws ApiError::invalid("media", ...).
|
||||
static void validate_url(const std::string& url);
|
||||
|
||||
// Decodes inline data or downloads the URL. Throws ApiError on a bad status
|
||||
// (the body is never returned for a failure) or when the cap is exceeded.
|
||||
FetchedAsset fetch(const MediaAsset& asset);
|
||||
std::string fetch_url(const std::string& url, std::string* content_type = nullptr);
|
||||
|
||||
private:
|
||||
clients::Client client_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+145
@@ -0,0 +1,145 @@
|
||||
#include "vendors/media/common/job.h"
|
||||
|
||||
#include "llm/error.h"
|
||||
#include "vendors/media/common/caps.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
const char* media_kind_name(MediaKind kind) {
|
||||
switch (kind) {
|
||||
case MediaKind::kImage: return "image";
|
||||
case MediaKind::kVideo: return "video";
|
||||
case MediaKind::kAudio: return "audio";
|
||||
case MediaKind::kModel3d: return "model3d";
|
||||
default: return "other";
|
||||
}
|
||||
}
|
||||
|
||||
const char* job_state_name(JobState state) {
|
||||
switch (state) {
|
||||
case JobState::kQueued: return "queued";
|
||||
case JobState::kRunning: return "running";
|
||||
case JobState::kSucceeded: return "succeeded";
|
||||
case JobState::kFailed: return "failed";
|
||||
default: return "cancelled";
|
||||
}
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
MediaKind kind_from(const std::string& s) {
|
||||
if (s == "image") return MediaKind::kImage;
|
||||
if (s == "video") return MediaKind::kVideo;
|
||||
if (s == "audio") return MediaKind::kAudio;
|
||||
if (s == "model3d") return MediaKind::kModel3d;
|
||||
return MediaKind::kOther;
|
||||
}
|
||||
|
||||
std::string str(const Json& doc, const char* key) {
|
||||
const Json* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : "";
|
||||
}
|
||||
|
||||
Json assets_json(const std::vector<MediaAsset>& assets) {
|
||||
Json a = Json::array();
|
||||
for (const MediaAsset& x : assets) a.push(x.to_json());
|
||||
return a;
|
||||
}
|
||||
|
||||
[[noreturn]] void bad(const std::string& why) { throw llm::ApiError::invalid("media", why); }
|
||||
|
||||
void check_text(const std::string& what, const std::string& v, std::size_t cap) {
|
||||
if (v.size() > cap) bad(what + " is over " + std::to_string(cap) + " characters");
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Json MediaAsset::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("kind", Json(media_kind_name(kind)));
|
||||
if (!url.empty()) o.set("url", Json(url));
|
||||
if (!data_base64.empty()) o.set("data_base64_bytes", Json(static_cast<double>(data_base64.size())));
|
||||
if (!mime.empty()) o.set("mime", Json(mime));
|
||||
if (!label.empty()) o.set("label", Json(label));
|
||||
return o;
|
||||
}
|
||||
|
||||
Json JobTicket::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("provider", Json(provider));
|
||||
o.set("id", Json(id));
|
||||
o.set("model", Json(model));
|
||||
o.set("kind", Json(media_kind_name(kind)));
|
||||
o.set("immediate", Json(immediate));
|
||||
o.set("assets", assets_json(assets));
|
||||
if (!meta.is_null()) o.set("meta", meta);
|
||||
return o;
|
||||
}
|
||||
|
||||
JobTicket JobTicket::from_json(const Json& doc) {
|
||||
if (!doc.is_object()) bad("ticket must be a JSON object");
|
||||
JobTicket t;
|
||||
t.provider = str(doc, "provider");
|
||||
t.id = str(doc, "id");
|
||||
t.model = str(doc, "model");
|
||||
t.kind = kind_from(str(doc, "kind"));
|
||||
if (t.provider.empty() || t.id.empty()) bad("ticket needs provider and id");
|
||||
if (const Json* m = doc.find("meta"); m && m->is_object()) t.meta = *m;
|
||||
if (const Json* im = doc.find("immediate"); im && im->is_bool()) t.immediate = im->as_bool();
|
||||
// Only URL assets travel back: inline (base64) outputs are not echoed by
|
||||
// to_json(), so an immediate ticket restored from JSON carries its URLs only.
|
||||
if (const Json* a = doc.find("assets"); a && a->is_array())
|
||||
for (const Json& x : a->as_array()) {
|
||||
std::string url = str(x, "url");
|
||||
if (url.empty() || t.assets.size() >= MediaCaps::kMaxAssetsPerJob) continue;
|
||||
MediaAsset asset;
|
||||
asset.url = url;
|
||||
asset.kind = kind_from(str(x, "kind"));
|
||||
asset.mime = str(x, "mime");
|
||||
asset.label = str(x, "label");
|
||||
t.assets.push_back(asset);
|
||||
}
|
||||
return t;
|
||||
}
|
||||
|
||||
Json JobStatus::to_json() const {
|
||||
Json o = Json::object();
|
||||
o.set("state", Json(job_state_name(state)));
|
||||
o.set("raw_state", Json(raw_state));
|
||||
if (progress >= 0) o.set("progress", Json(progress));
|
||||
if (!error.empty()) o.set("error", Json(error));
|
||||
o.set("assets", assets_json(assets));
|
||||
return o;
|
||||
}
|
||||
|
||||
void ImageRequest::validate() const {
|
||||
if (prompt.empty()) bad("prompt is required");
|
||||
check_text("prompt", prompt, MediaCaps::kMaxPromptChars);
|
||||
check_text("negative_prompt", negative_prompt, MediaCaps::kMaxPromptChars);
|
||||
check_text("model", model, 256);
|
||||
check_text("source_url", source_url, MediaCaps::kMaxUrlChars);
|
||||
if (count < 1 || count > MediaCaps::kMaxImagesPerRequest)
|
||||
bad("count must be 1.." + std::to_string(MediaCaps::kMaxImagesPerRequest));
|
||||
if (width < 0 || width > MediaCaps::kMaxDimension || height < 0 || height > MediaCaps::kMaxDimension)
|
||||
bad("width and height must be 0.." + std::to_string(MediaCaps::kMaxDimension));
|
||||
if ((width == 0) != (height == 0)) bad("width and height go together");
|
||||
if (seed < -1 || seed > 4294967295L) bad("seed must be -1 or 0..4294967295");
|
||||
if (!extra.is_null() && !extra.is_object()) bad("extra must be a JSON object");
|
||||
}
|
||||
|
||||
void VideoRequest::validate() const {
|
||||
if (prompt.empty() && image_url.empty()) bad("prompt or image_url is required");
|
||||
check_text("prompt", prompt, MediaCaps::kMaxPromptChars);
|
||||
check_text("negative_prompt", negative_prompt, MediaCaps::kMaxPromptChars);
|
||||
check_text("model", model, 256);
|
||||
check_text("image_url", image_url, MediaCaps::kMaxUrlChars);
|
||||
check_text("end_image_url", end_image_url, MediaCaps::kMaxUrlChars);
|
||||
if (duration_seconds < 0 || duration_seconds > MediaCaps::kMaxVideoSeconds)
|
||||
bad("duration_seconds must be 0.." + std::to_string(MediaCaps::kMaxVideoSeconds));
|
||||
if (seed < -1 || seed > 4294967295L) bad("seed must be -1 or 0..4294967295");
|
||||
if (!extra.is_null() && !extra.is_object()) bad("extra must be a JSON object");
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+124
@@ -0,0 +1,124 @@
|
||||
#pragma once
|
||||
|
||||
// The neutral generation seam of package media: submit a request, poll the
|
||||
// job, fetch the outputs. Every provider adapter implements ImageGenerator
|
||||
// and / or VideoGenerator; callers (the demo, tests, other packages) code
|
||||
// against these types only. Nothing here knows a provider's wire format.
|
||||
//
|
||||
// Money rule: submit() and cancel() take an explicit confirm flag with no
|
||||
// default. Without it they throw before any network traffic.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
enum class MediaKind { kImage, kVideo, kAudio, kModel3d, kOther };
|
||||
enum class JobState { kQueued, kRunning, kSucceeded, kFailed, kCancelled };
|
||||
|
||||
const char* media_kind_name(MediaKind kind);
|
||||
const char* job_state_name(JobState state);
|
||||
|
||||
// One output. Exactly one of url / data_base64 is set.
|
||||
struct MediaAsset {
|
||||
MediaKind kind = MediaKind::kOther;
|
||||
std::string url;
|
||||
std::string data_base64;
|
||||
std::string mime;
|
||||
std::string label; // provider note: seed, finish reason, model
|
||||
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
// Handle of a submitted job. `immediate` jobs (synchronous providers) carry
|
||||
// their assets already; poll() answers from the ticket without a network call.
|
||||
struct JobTicket {
|
||||
std::string provider;
|
||||
std::string id;
|
||||
std::string model;
|
||||
MediaKind kind = MediaKind::kImage;
|
||||
bool immediate = false;
|
||||
std::vector<MediaAsset> assets;
|
||||
clients::body::Json meta; // provider extras the poll needs (endpoint, urls, cost)
|
||||
|
||||
clients::body::Json to_json() const;
|
||||
// Rebuilds a ticket from to_json() output (the demo round-trips tickets).
|
||||
static JobTicket from_json(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
struct JobStatus {
|
||||
JobState state = JobState::kQueued;
|
||||
std::string raw_state; // the provider's own word
|
||||
double progress = -1; // 0..1, -1 = unknown
|
||||
std::string error;
|
||||
std::vector<MediaAsset> assets;
|
||||
clients::body::Json meta;
|
||||
|
||||
bool terminal() const { return state == JobState::kSucceeded || state == JobState::kFailed || state == JobState::kCancelled; }
|
||||
clients::body::Json to_json() const;
|
||||
};
|
||||
|
||||
struct ImageRequest {
|
||||
std::string prompt;
|
||||
std::string negative_prompt;
|
||||
std::string model; // provider model / endpoint id; empty = adapter default
|
||||
int width = 0; // 0 = provider default
|
||||
int height = 0;
|
||||
std::string aspect_ratio; // "16:9"; empty = default
|
||||
int count = 1;
|
||||
long seed = -1; // -1 = random
|
||||
std::string source_url; // reference / source image for edit styles
|
||||
clients::body::Json extra; // provider fields merged last, checked against the call table
|
||||
|
||||
// Throws llm::ApiError::invalid(vendor "media") on a bad value or cap breach.
|
||||
void validate() const;
|
||||
};
|
||||
|
||||
struct VideoRequest {
|
||||
std::string prompt;
|
||||
std::string negative_prompt;
|
||||
std::string model;
|
||||
int duration_seconds = 0; // 0 = provider default
|
||||
std::string aspect_ratio;
|
||||
std::string resolution; // "720p", "1080p"; empty = default
|
||||
long seed = -1;
|
||||
std::string image_url; // start frame for image-to-video
|
||||
std::string end_image_url; // end frame
|
||||
clients::body::Json extra;
|
||||
|
||||
void validate() const;
|
||||
};
|
||||
|
||||
class MediaGenerator {
|
||||
public:
|
||||
virtual ~MediaGenerator() = default;
|
||||
virtual std::string provider() const = 0;
|
||||
virtual bool configured() const = 0;
|
||||
// Model / endpoint ids the adapter knows by name (not a live catalogue).
|
||||
virtual std::vector<std::string> models() const = 0;
|
||||
// One status check. A failed job is a JobStatus with state kFailed, a
|
||||
// transport or auth failure is an ApiError.
|
||||
virtual JobStatus poll(const JobTicket& ticket) = 0;
|
||||
// Cancels where the provider supports it; returns false (without any call)
|
||||
// where it does not. Explicit confirm, no default.
|
||||
virtual bool cancel(const JobTicket& ticket, bool confirm) = 0;
|
||||
// Suggested first poll interval; JobRunner clamps it to the caps.
|
||||
virtual int suggested_poll_ms() const { return 2000; }
|
||||
};
|
||||
|
||||
class ImageGenerator : public MediaGenerator {
|
||||
public:
|
||||
// Exact request the submit would send, as JSON; no network, no credential.
|
||||
virtual clients::body::Json plan(const ImageRequest& request) const = 0;
|
||||
virtual JobTicket submit(const ImageRequest& request, bool confirm) = 0;
|
||||
};
|
||||
|
||||
class VideoGenerator : public MediaGenerator {
|
||||
public:
|
||||
virtual clients::body::Json plan(const VideoRequest& request) const = 0;
|
||||
virtual JobTicket submit(const VideoRequest& request, bool confirm) = 0;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
#include "vendors/media/common/job_runner.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
#include <thread>
|
||||
|
||||
#include "llm/error.h"
|
||||
#include "vendors/media/common/caps.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
void WaitPolicy::validate() const {
|
||||
auto bad = [](const std::string& why) { throw llm::ApiError::invalid("media", why); };
|
||||
if (first_interval_ms < 0 || first_interval_ms > MediaCaps::kMaxPollIntervalMs) bad("first_interval_ms out of range");
|
||||
if (backoff < 1.0 || backoff > 3.0) bad("backoff must be 1.0..3.0");
|
||||
if (max_interval_ms < MediaCaps::kMinPollIntervalMs || max_interval_ms > MediaCaps::kMaxPollIntervalMs)
|
||||
bad("max_interval_ms must be " + std::to_string(MediaCaps::kMinPollIntervalMs) + ".." +
|
||||
std::to_string(MediaCaps::kMaxPollIntervalMs));
|
||||
if (max_attempts < 1 || max_attempts > MediaCaps::kMaxPollAttempts)
|
||||
bad("max_attempts must be 1.." + std::to_string(MediaCaps::kMaxPollAttempts));
|
||||
if (max_wait_seconds < 1 || max_wait_seconds > MediaCaps::kMaxWaitSeconds)
|
||||
bad("max_wait_seconds must be 1.." + std::to_string(MediaCaps::kMaxWaitSeconds));
|
||||
}
|
||||
|
||||
JobRunner::JobRunner(WaitPolicy policy, Sleeper sleeper, Clock clock)
|
||||
: policy_(policy), sleeper_(std::move(sleeper)), clock_(std::move(clock)) {
|
||||
policy_.validate();
|
||||
if (!sleeper_)
|
||||
sleeper_ = [](int ms) { std::this_thread::sleep_for(std::chrono::milliseconds(ms)); };
|
||||
if (!clock_)
|
||||
clock_ = [] {
|
||||
return static_cast<long>(std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
std::chrono::steady_clock::now().time_since_epoch())
|
||||
.count());
|
||||
};
|
||||
}
|
||||
|
||||
JobStatus JobRunner::wait(MediaGenerator& generator, const JobTicket& ticket) const {
|
||||
if (ticket.immediate) {
|
||||
JobStatus s;
|
||||
s.state = JobState::kSucceeded;
|
||||
s.raw_state = "immediate";
|
||||
s.progress = 1;
|
||||
s.assets = ticket.assets;
|
||||
return s;
|
||||
}
|
||||
const long start = clock_();
|
||||
const long deadline_ms = static_cast<long>(policy_.max_wait_seconds) * 1000L;
|
||||
int interval = policy_.first_interval_ms > 0 ? policy_.first_interval_ms : generator.suggested_poll_ms();
|
||||
interval = std::clamp(interval, MediaCaps::kMinPollIntervalMs, policy_.max_interval_ms);
|
||||
for (int attempt = 1; attempt <= policy_.max_attempts; ++attempt) {
|
||||
JobStatus s = generator.poll(ticket);
|
||||
if (s.terminal()) return s;
|
||||
if (clock_() - start + interval > deadline_ms) break;
|
||||
sleeper_(interval);
|
||||
interval = std::clamp(static_cast<int>(interval * policy_.backoff), MediaCaps::kMinPollIntervalMs,
|
||||
policy_.max_interval_ms);
|
||||
}
|
||||
throw llm::ApiError("media", 504, "WAIT_EXPIRED",
|
||||
generator.provider() + " job " + ticket.id + " is not finished after the poll caps (" +
|
||||
std::to_string(policy_.max_attempts) + " polls, " + std::to_string(policy_.max_wait_seconds) +
|
||||
" s); it was NOT cancelled, check it with poll");
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
#pragma once
|
||||
|
||||
// JobRunner: poll a job to a terminal state under hard caps. The caps are a
|
||||
// floor on the interval (never hammer a provider) and ceilings on attempts
|
||||
// and total wait (never hang). When a cap is hit the runner throws; it never
|
||||
// cancels the job on its own (a human decides, the money may already be spent).
|
||||
|
||||
#include <functional>
|
||||
|
||||
#include "vendors/media/common/job.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct WaitPolicy {
|
||||
int first_interval_ms = 0; // 0 = the generator's suggestion
|
||||
double backoff = 1.5; // interval multiplier per attempt, 1.0 .. 3.0
|
||||
int max_interval_ms = 15000; // <= MediaCaps::kMaxPollIntervalMs
|
||||
int max_attempts = 120; // <= MediaCaps::kMaxPollAttempts
|
||||
int max_wait_seconds = 900; // <= MediaCaps::kMaxWaitSeconds
|
||||
|
||||
// Throws ApiError::invalid when a value is outside the caps.
|
||||
void validate() const;
|
||||
};
|
||||
|
||||
class JobRunner {
|
||||
public:
|
||||
using Sleeper = std::function<void(int milliseconds)>;
|
||||
using Clock = std::function<long()>; // monotonic milliseconds
|
||||
|
||||
// Null sleeper / clock use the real ones. Tests inject fakes.
|
||||
explicit JobRunner(WaitPolicy policy = {}, Sleeper sleeper = nullptr, Clock clock = nullptr);
|
||||
|
||||
const WaitPolicy& policy() const { return policy_; }
|
||||
|
||||
// Immediate tickets answer without a poll. Otherwise polls until terminal.
|
||||
// Throws ApiError (504, WAIT_EXPIRED) when attempts or wait are exhausted.
|
||||
JobStatus wait(MediaGenerator& generator, const JobTicket& ticket) const;
|
||||
|
||||
private:
|
||||
WaitPolicy policy_;
|
||||
Sleeper sleeper_;
|
||||
Clock clock_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+171
@@ -0,0 +1,171 @@
|
||||
#include "vendors/media/common/media_service.h"
|
||||
|
||||
#include <cstring>
|
||||
|
||||
#include "clients/body/form.h"
|
||||
#include "clients/url.h"
|
||||
#include "llm/error.h"
|
||||
#include "vendors/media/common/caps.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
bool valid_header_name(const std::string& n) {
|
||||
if (n.empty() || n.size() > 64) return false;
|
||||
for (char c : n)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-')) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string field_text(const Json& v) {
|
||||
if (v.is_string()) return v.as_string();
|
||||
if (v.is_bool()) return v.as_bool() ? "true" : "false";
|
||||
return v.dump(); // numbers, arrays, objects travel as JSON text
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Json PreparedCall::summary() const {
|
||||
Json o = Json::object();
|
||||
o.set("vendor", Json(vendor));
|
||||
o.set("key", Json(key));
|
||||
o.set("method", Json(method));
|
||||
o.set("target", Json(target));
|
||||
o.set("effect", Json(CallTable::effect_name(effect)));
|
||||
if (!content_type.empty()) o.set("content_type", Json(content_type.substr(0, content_type.find(';'))));
|
||||
if (!preview.is_null()) o.set("body", preview);
|
||||
return o;
|
||||
}
|
||||
|
||||
MediaService::MediaService(std::shared_ptr<llm::ApiClient> client, const CallTable& table)
|
||||
: llm::Service(std::move(client)), table_(table) {}
|
||||
|
||||
PreparedCall MediaService::prepare(const std::string& key, const CallArgs& args) const {
|
||||
const std::string v = table_.vendor();
|
||||
const CallRow& row = table_.get(key);
|
||||
PreparedCall out;
|
||||
out.vendor = v;
|
||||
out.key = key;
|
||||
out.method = row.http_method;
|
||||
out.effect = CallTable::effect_of(row);
|
||||
|
||||
std::string target = table_.render_path(row, args.path);
|
||||
|
||||
if (args.query.size() > MediaCaps::kMaxQueryArgs) throw llm::ApiError::invalid(v, "too many query parameters");
|
||||
std::string qs;
|
||||
for (const auto& [name, value] : args.query) {
|
||||
if (!table_.query_allows(row, name))
|
||||
throw llm::ApiError::invalid(v, "call '" + key + "' does not accept query parameter '" + name + "'");
|
||||
if (value.size() > MediaCaps::kMaxQueryValueChars) throw llm::ApiError::invalid(v, "query value too long: " + name);
|
||||
qs += qs.empty() ? '?' : '&';
|
||||
qs += clients::Url::encode_component(name) + "=" + clients::Url::encode_component(value);
|
||||
}
|
||||
out.target = target + qs;
|
||||
|
||||
if (args.headers.size() > MediaCaps::kMaxHeaders) throw llm::ApiError::invalid(v, "too many headers");
|
||||
for (const auto& [k, val] : args.headers) {
|
||||
if (!valid_header_name(k) || val.size() > MediaCaps::kMaxHeaderValueChars ||
|
||||
val.find_first_of("\r\n") != std::string::npos)
|
||||
throw llm::ApiError::invalid(v, "bad header '" + k + "'");
|
||||
out.headers[k] = val;
|
||||
}
|
||||
|
||||
const std::string kind = row.body_kind;
|
||||
const bool has_body = !args.body.is_null() && !(args.body.is_object() && args.body.as_object().empty());
|
||||
if (kind == "none") {
|
||||
if (has_body || !args.files.empty()) throw llm::ApiError::invalid(v, "call '" + key + "' takes no body");
|
||||
return out;
|
||||
}
|
||||
if (kind == "form") throw llm::ApiError::invalid(v, "form bodies are not supported by this package");
|
||||
|
||||
if (!args.body.is_null() && !args.body.is_object()) throw llm::ApiError::invalid(v, "body must be a JSON object");
|
||||
Json fields = args.body.is_null() ? Json::object() : args.body;
|
||||
if (fields.as_object().size() > MediaCaps::kMaxBodyFields) throw llm::ApiError::invalid(v, "too many body fields");
|
||||
for (const auto& [name, value] : fields.as_object()) {
|
||||
if (!table_.body_allows(row, name))
|
||||
throw llm::ApiError::invalid(v, "call '" + key + "' does not accept body field '" + name + "'");
|
||||
if (value.is_string() && value.as_string().size() > MediaCaps::kMaxFieldChars)
|
||||
throw llm::ApiError::invalid(v, "field '" + name + "' is over the size cap");
|
||||
}
|
||||
|
||||
const bool multipart = kind == "multipart" || (kind == "either" && !args.files.empty());
|
||||
if (!args.files.empty() && !multipart) throw llm::ApiError::invalid(v, "call '" + key + "' does not take files");
|
||||
if (args.files.size() > MediaCaps::kMaxFiles) throw llm::ApiError::invalid(v, "too many files");
|
||||
|
||||
if (!multipart) {
|
||||
out.body = fields.dump();
|
||||
out.content_type = "application/json";
|
||||
out.preview = fields;
|
||||
return out;
|
||||
}
|
||||
|
||||
clients::body::MultipartBody form;
|
||||
Json preview = Json::object();
|
||||
for (const auto& [name, value] : fields.as_object()) {
|
||||
form.add_text(name, field_text(value));
|
||||
preview.set(name, value);
|
||||
}
|
||||
Json files = Json::array();
|
||||
for (const FilePart& f : args.files) {
|
||||
if (!table_.body_allows(row, f.field))
|
||||
throw llm::ApiError::invalid(v, "call '" + key + "' does not accept file field '" + f.field + "'");
|
||||
if (f.data.empty()) throw llm::ApiError::invalid(v, "file '" + f.field + "' is empty");
|
||||
if (f.data.size() > MediaCaps::kMaxFileBytes) throw llm::ApiError::invalid(v, "file '" + f.field + "' is over the size cap");
|
||||
if (f.filename.find_first_of("\"\r\n") != std::string::npos || f.content_type.find_first_of("\r\n") != std::string::npos)
|
||||
throw llm::ApiError::invalid(v, "bad file name or content type");
|
||||
form.add_file(f.field, f.filename.empty() ? f.field : f.filename,
|
||||
f.content_type.empty() ? "application/octet-stream" : f.content_type, f.data);
|
||||
Json info = Json::object();
|
||||
info.set("field", Json(f.field));
|
||||
info.set("filename", Json(f.filename));
|
||||
info.set("bytes", Json(static_cast<double>(f.data.size())));
|
||||
files.push(info);
|
||||
}
|
||||
if (!args.files.empty()) preview.set("_files", files);
|
||||
out.body = form.encode();
|
||||
out.content_type = form.content_type();
|
||||
out.preview = preview;
|
||||
return out;
|
||||
}
|
||||
|
||||
llm::RawResponse MediaService::execute(const PreparedCall& call, bool confirmed) {
|
||||
if (call.effect != Effect::kRead && !confirmed)
|
||||
throw llm::ApiError::invalid(table_.vendor(), "call '" + call.key + "' has effect " +
|
||||
CallTable::effect_name(call.effect) +
|
||||
" and needs explicit confirmation");
|
||||
const CallRow& row = table_.get(call.key);
|
||||
const bool has_body = !call.body.empty();
|
||||
return client_->request_raw(CallTable::method_of(row), call.target, has_body ? &call.body : nullptr,
|
||||
call.content_type, call.headers);
|
||||
}
|
||||
|
||||
llm::RawResponse MediaService::call_raw(const std::string& key, const CallArgs& args, bool confirmed) {
|
||||
return execute(prepare(key, args), confirmed);
|
||||
}
|
||||
|
||||
Json MediaService::parse_json(const std::string& vendor, const llm::RawResponse& raw) {
|
||||
if (raw.body.find_first_not_of(" \t\r\n") == std::string::npos) return Json::object();
|
||||
try {
|
||||
return Json::parse(raw.body);
|
||||
} catch (const std::exception& e) {
|
||||
throw llm::ApiError(vendor, 502, "BAD_RESPONSE", std::string("not JSON: ") + e.what());
|
||||
}
|
||||
}
|
||||
|
||||
Json MediaService::call_json(const std::string& key, const CallArgs& args, bool confirmed) {
|
||||
return parse_json(table_.vendor(), call_raw(key, args, confirmed));
|
||||
}
|
||||
|
||||
const Json* MediaService::member(const Json& doc, const std::string& key) {
|
||||
return doc.is_object() ? doc.find(key) : nullptr;
|
||||
}
|
||||
|
||||
std::string MediaService::text(const Json& doc, const std::string& key, const std::string& fallback) {
|
||||
const Json* v = member(doc, key);
|
||||
if (v && v->is_string()) return v->as_string();
|
||||
if (v && v->is_number()) return v->dump();
|
||||
return fallback;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
#pragma once
|
||||
|
||||
// MediaService: the validated path from "I want call X with these arguments"
|
||||
// to bytes on the wire. prepare() is pure (no network): it checks the key,
|
||||
// path arguments, query parameters, body fields and files against the call
|
||||
// table row and builds the exact request. execute() sends it and refuses any
|
||||
// call whose effect is not "read" unless the caller confirms explicitly.
|
||||
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "llm/api_client.h"
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
using Json = clients::body::Json;
|
||||
|
||||
struct FilePart {
|
||||
std::string field;
|
||||
std::string filename;
|
||||
std::string content_type;
|
||||
std::string data;
|
||||
};
|
||||
|
||||
struct CallArgs {
|
||||
std::map<std::string, std::string> path;
|
||||
std::map<std::string, std::string> query;
|
||||
Json body; // null = no body; otherwise a JSON object
|
||||
std::vector<FilePart> files; // multipart rows only
|
||||
llm::Headers headers; // extra request headers (Accept, Prefer, ...)
|
||||
};
|
||||
|
||||
struct PreparedCall {
|
||||
std::string vendor;
|
||||
std::string key;
|
||||
std::string method; // GET POST ...
|
||||
std::string target; // path plus query string
|
||||
std::string body; // exact bytes ("" = none)
|
||||
std::string content_type; // "" when no body
|
||||
llm::Headers headers;
|
||||
Effect effect = Effect::kRead;
|
||||
Json preview; // body as JSON, or the multipart part list
|
||||
|
||||
// What a dry run shows: method, target, effect, content type, body preview.
|
||||
Json summary() const;
|
||||
};
|
||||
|
||||
class MediaService : public llm::Service {
|
||||
public:
|
||||
MediaService(std::shared_ptr<llm::ApiClient> client, const CallTable& table);
|
||||
|
||||
const CallTable& table() const { return table_; }
|
||||
const std::string& vendor() const { return table_.vendor(); }
|
||||
|
||||
PreparedCall prepare(const std::string& key, const CallArgs& args = {}) const;
|
||||
|
||||
// Non-read effects throw ApiError::invalid unless confirmed is true. The
|
||||
// missing-credential error surfaces here, before any network traffic.
|
||||
llm::RawResponse execute(const PreparedCall& call, bool confirmed);
|
||||
|
||||
// prepare + execute + parse. Empty body -> {}. Non-JSON body -> ApiError.
|
||||
Json call_json(const std::string& key, const CallArgs& args = {}, bool confirmed = false);
|
||||
llm::RawResponse call_raw(const std::string& key, const CallArgs& args = {}, bool confirmed = false);
|
||||
|
||||
// Helpers for typed wrappers.
|
||||
static Json parse_json(const std::string& vendor, const llm::RawResponse& raw);
|
||||
static const Json* member(const Json& doc, const std::string& key);
|
||||
static std::string text(const Json& doc, const std::string& key, const std::string& fallback = "");
|
||||
|
||||
private:
|
||||
const CallTable& table_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <set>
|
||||
|
||||
#include "vendors/media/common/caps.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
std::string lower_ext(const std::string& name) {
|
||||
std::string n = name.substr(0, name.find_first_of("?#"));
|
||||
std::size_t dot = n.rfind('.');
|
||||
if (dot == std::string::npos || n.size() - dot > 8) return "";
|
||||
std::string e = n.substr(dot + 1);
|
||||
std::transform(e.begin(), e.end(), e.begin(), [](unsigned char c) { return static_cast<char>(std::tolower(c)); });
|
||||
return e;
|
||||
}
|
||||
|
||||
struct Walk {
|
||||
std::vector<MediaAsset> out;
|
||||
std::set<std::string> seen;
|
||||
std::size_t nodes = 0;
|
||||
MediaKind fallback;
|
||||
|
||||
void add(const std::string& s, const std::string& key) {
|
||||
if (out.size() >= MediaCaps::kMaxAssetsPerJob) return;
|
||||
bool http = s.rfind("https://", 0) == 0 || s.rfind("http://", 0) == 0;
|
||||
bool data = s.rfind("data:", 0) == 0;
|
||||
if (!http && !data) return;
|
||||
if (s.size() > MediaCaps::kMaxUrlChars && !data) return;
|
||||
if (!seen.insert(s).second) return;
|
||||
MediaAsset a;
|
||||
if (data) {
|
||||
std::size_t comma = s.find(',');
|
||||
std::size_t semi = s.find(';');
|
||||
if (comma == std::string::npos) return;
|
||||
a.mime = s.substr(5, std::min(semi, comma) - 5);
|
||||
a.data_base64 = s.substr(comma + 1);
|
||||
a.kind = fallback;
|
||||
if (a.mime.rfind("video/", 0) == 0) a.kind = MediaKind::kVideo;
|
||||
else if (a.mime.rfind("image/", 0) == 0) a.kind = MediaKind::kImage;
|
||||
else if (a.mime.rfind("audio/", 0) == 0) a.kind = MediaKind::kAudio;
|
||||
} else {
|
||||
a.url = s;
|
||||
a.kind = OutputScanner::kind_from_name(s, fallback);
|
||||
if (a.kind == fallback) {
|
||||
if (key.find("video") != std::string::npos) a.kind = MediaKind::kVideo;
|
||||
else if (key.find("image") != std::string::npos) a.kind = MediaKind::kImage;
|
||||
else if (key.find("audio") != std::string::npos) a.kind = MediaKind::kAudio;
|
||||
}
|
||||
a.mime = OutputScanner::mime_from_name(s);
|
||||
}
|
||||
out.push_back(a);
|
||||
}
|
||||
|
||||
void go(const Json& v, const std::string& key, std::size_t depth) {
|
||||
if (depth > MediaCaps::kMaxScanDepth || ++nodes > MediaCaps::kMaxScanNodes) return;
|
||||
if (v.is_string()) {
|
||||
add(v.as_string(), key);
|
||||
} else if (v.is_array()) {
|
||||
for (const Json& x : v.as_array()) go(x, key, depth + 1);
|
||||
} else if (v.is_object()) {
|
||||
for (const auto& [k, x] : v.as_object()) {
|
||||
if (OutputScanner::skipped_key(k)) continue;
|
||||
go(x, k, depth + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
bool OutputScanner::skipped_key(const std::string& key) {
|
||||
static const std::set<std::string> skip = {
|
||||
"input", "request", "urls", "logs", "metrics", "webhook", "webhook_url", "callback_url",
|
||||
"status_url", "cancel_url", "response_url", "polling_url", "get", "cancel", "stream", "web",
|
||||
"params", "parameters", "meta", "links", "self", "thumbnail_url", "model_url"};
|
||||
return skip.count(key) > 0;
|
||||
}
|
||||
|
||||
MediaKind OutputScanner::kind_from_name(const std::string& name, MediaKind fallback) {
|
||||
std::string e = lower_ext(name);
|
||||
if (e == "mp4" || e == "webm" || e == "mov" || e == "mkv" || e == "m4v") return MediaKind::kVideo;
|
||||
if (e == "png" || e == "jpg" || e == "jpeg" || e == "webp" || e == "gif" || e == "svg" || e == "avif") return MediaKind::kImage;
|
||||
if (e == "mp3" || e == "wav" || e == "flac" || e == "ogg" || e == "m4a" || e == "aac") return MediaKind::kAudio;
|
||||
if (e == "glb" || e == "gltf" || e == "obj" || e == "ply" || e == "usdz" || e == "fbx" || e == "stl") return MediaKind::kModel3d;
|
||||
return fallback;
|
||||
}
|
||||
|
||||
std::string OutputScanner::mime_from_name(const std::string& name) {
|
||||
std::string e = lower_ext(name);
|
||||
if (e == "mp4" || e == "m4v") return "video/mp4";
|
||||
if (e == "webm") return "video/webm";
|
||||
if (e == "mov") return "video/quicktime";
|
||||
if (e == "png") return "image/png";
|
||||
if (e == "jpg" || e == "jpeg") return "image/jpeg";
|
||||
if (e == "webp") return "image/webp";
|
||||
if (e == "gif") return "image/gif";
|
||||
if (e == "svg") return "image/svg+xml";
|
||||
if (e == "mp3") return "audio/mpeg";
|
||||
if (e == "wav") return "audio/wav";
|
||||
if (e == "flac") return "audio/flac";
|
||||
if (e == "ogg") return "audio/ogg";
|
||||
if (e == "glb") return "model/gltf-binary";
|
||||
return "";
|
||||
}
|
||||
|
||||
std::vector<MediaAsset> OutputScanner::scan(const Json& doc, MediaKind fallback) {
|
||||
Walk w;
|
||||
w.fallback = fallback;
|
||||
w.go(doc, "", 0);
|
||||
return std::move(w.out);
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
#pragma once
|
||||
|
||||
// OutputScanner: finds result URLs / inline data in a provider's result JSON.
|
||||
// Providers shape results differently (fal: images[].url, Replicate: output as
|
||||
// string or array, Runway: output[]), so the adapters share one bounded walk
|
||||
// instead of seven hand-rolled ones. Keys that carry request echoes or
|
||||
// control links (input, request, urls, webhook, status_url, ...) are skipped.
|
||||
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/media/common/job.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class OutputScanner {
|
||||
public:
|
||||
// Walks `doc` (depth and node caps in MediaCaps) and returns de-duplicated
|
||||
// assets in document order, at most MediaCaps::kMaxAssetsPerJob.
|
||||
static std::vector<MediaAsset> scan(const clients::body::Json& doc, MediaKind fallback);
|
||||
|
||||
// Kind from a URL / file name extension, else `fallback`.
|
||||
static MediaKind kind_from_name(const std::string& name, MediaKind fallback);
|
||||
// MIME for a kind + extension, "" when unknown.
|
||||
static std::string mime_from_name(const std::string& name);
|
||||
// True for keys whose subtree is request echo or control metadata.
|
||||
static bool skipped_key(const std::string& key);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+195
@@ -0,0 +1,195 @@
|
||||
#include "vendors/media/common/registry.h"
|
||||
|
||||
#include <ctime>
|
||||
|
||||
#include "llm/config.h"
|
||||
#include "llm/error.h"
|
||||
#include "vendors/media/bfl/bfl.h"
|
||||
#include "vendors/media/fal/fal.h"
|
||||
#include "vendors/media/higgsfield/higgsfield.h"
|
||||
#include "vendors/media/ideogram/ideogram.h"
|
||||
#include "vendors/media/kling/kling.h"
|
||||
#include "vendors/media/leonardo/leonardo.h"
|
||||
#include "vendors/media/luma/luma.h"
|
||||
#include "vendors/media/minimax/minimax.h"
|
||||
#include "vendors/media/recraft/recraft.h"
|
||||
#include "vendors/media/replicate/replicate.h"
|
||||
#include "vendors/media/runway/runway.h"
|
||||
#include "vendors/media/stability/stability.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
const std::vector<ProviderInfo>& MediaRegistry::providers() {
|
||||
static const std::vector<ProviderInfo> list = {
|
||||
{"stability", "Stability AI", {"STABILITY_API_KEY"}, true, false, Evidence::kOpenApi,
|
||||
"https://platform.stability.ai/account/keys", "v1.user.balance",
|
||||
"v2beta image (generate, edit, upscale, control), 3D, audio; v1 engines. No video route in the published spec."},
|
||||
{"bfl", "Black Forest Labs", {"BFL_API_KEY"}, true, true, Evidence::kOpenApi,
|
||||
"https://dashboard.bfl.ai/", "credits", "FLUX models and tools, polling via get_result, fine-tunes."},
|
||||
{"replicate", "Replicate", {"REPLICATE_API_TOKEN"}, true, true, Evidence::kOpenApi,
|
||||
"https://replicate.com/account/api-tokens", "account.get",
|
||||
"Predictions, models, deployments, files, trainings, webhook verification. Any model by owner/name."},
|
||||
{"fal", "fal", {"FAL_KEY"}, true, true, Evidence::kOpenApi, "https://fal.ai/dashboard/keys",
|
||||
"platform.models.pricing", "Queue API (any endpoint id) and the model-related Platform API."},
|
||||
{"ideogram", "Ideogram", {"IDEOGRAM_API_KEY"}, true, true, Evidence::kOpenApi,
|
||||
"https://ideogram.ai/manage-api", "account.usage",
|
||||
"v2 image, SVG, video and ad tools; dry_run=true gives a free price quote."},
|
||||
{"recraft", "Recraft", {"RECRAFT_API_TOKEN"}, true, false, Evidence::kOpenApi,
|
||||
"https://www.recraft.ai/profile/api", "users.me", "Generation, vector, background, upscale, styles."},
|
||||
{"leonardo", "Leonardo.Ai", {"LEONARDO_API_KEY"}, true, true, Evidence::kDocs,
|
||||
"https://app.leonardo.ai/api-access", "v2.models",
|
||||
"v2 generations for image, video, audio and 3D models. v1 reference pages are gone from the docs."},
|
||||
{"runway", "Runway", {"RUNWAY_API_KEY", "RUNWAYML_API_SECRET"}, true, true, Evidence::kSdk,
|
||||
"https://dev.runwayml.com/", "organization.get",
|
||||
"Text / image / video generation, tasks, uploads, avatars, voices, documents, routers, workflows."},
|
||||
{"luma", "Luma Dream Machine", {"LUMA_API_KEY"}, true, true, Evidence::kSdk,
|
||||
"https://lumalabs.ai/api/keys", "credits.get", "Ray video, Photon image, reframe, modify, upscale, audio."},
|
||||
{"minimax", "MiniMax Hailuo", {"MINIMAX_API_KEY"}, false, true, Evidence::kDocs,
|
||||
"https://platform.minimax.io/user-center/basic-information/interface-key", "",
|
||||
"Hailuo video v1 and v2, file retrieval. No free account read exists."},
|
||||
{"kling", "Kling AI", {"KLING_ACCESS_KEY", "KLING_SECRET_KEY"}, false, true, Evidence::kProbeOnly,
|
||||
"https://app.klingai.com/global/dev/api-key", "",
|
||||
"Official API with JWT auth; docs are a JS app with no machine-readable spec: paths are unverified."},
|
||||
{"higgsfield", "Higgsfield", {"HF_API_KEY_ID", "HF_API_KEY_SECRET"}, true, true, Evidence::kOpenApi,
|
||||
"https://cloud.higgsfield.ai/", "",
|
||||
"One lifecycle for every model path; the OpenAPI document is explicitly supplementary."},
|
||||
};
|
||||
return list;
|
||||
}
|
||||
|
||||
const std::vector<UnsupportedProduct>& MediaRegistry::unsupported() {
|
||||
static const std::vector<UnsupportedProduct> list = {
|
||||
{"Midjourney",
|
||||
"No public API. Midjourney documents only its web and Discord products; third-party relays are unofficial "
|
||||
"and violate its terms, so none is implemented."},
|
||||
{"Pika",
|
||||
"pika.art/api advertises a membership API that resells other vendors' models, but no endpoint, schema or "
|
||||
"authentication document was found (2026-10-25), so nothing is implemented. Pika models are reachable "
|
||||
"through fal."},
|
||||
};
|
||||
return list;
|
||||
}
|
||||
|
||||
const ProviderInfo* MediaRegistry::find(const std::string& id) {
|
||||
for (const ProviderInfo& p : providers())
|
||||
if (p.id == id) return &p;
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
const char* MediaRegistry::evidence_name(Evidence e) {
|
||||
switch (e) {
|
||||
case Evidence::kOpenApi: return "openapi";
|
||||
case Evidence::kSdk: return "sdk";
|
||||
case Evidence::kDocs: return "docs";
|
||||
default: return "probe-only";
|
||||
}
|
||||
}
|
||||
|
||||
const CallTable& MediaRegistry::table(const std::string& id) {
|
||||
if (id == "stability") return StabilityCalls::table();
|
||||
if (id == "bfl") return BflCalls::table();
|
||||
if (id == "replicate") return ReplicateCalls::table();
|
||||
if (id == "fal") return FalCalls::table();
|
||||
if (id == "ideogram") return IdeogramCalls::table();
|
||||
if (id == "recraft") return RecraftCalls::table();
|
||||
if (id == "leonardo") return LeonardoCalls::table();
|
||||
if (id == "runway") return RunwayCalls::table();
|
||||
if (id == "luma") return LumaCalls::table();
|
||||
if (id == "minimax") return MinimaxCalls::table();
|
||||
if (id == "kling") return KlingCalls::table();
|
||||
if (id == "higgsfield") return HiggsfieldCalls::table();
|
||||
throw llm::ApiError::invalid("media", "unknown provider '" + id + "'");
|
||||
}
|
||||
|
||||
bool MediaRegistry::configured(const std::string& id) {
|
||||
const ProviderInfo* p = find(id);
|
||||
if (!p) throw llm::ApiError::invalid("media", "unknown provider '" + id + "'");
|
||||
if (id == "runway") return !llm::Config::get("RUNWAY_API_KEY").empty() || !llm::Config::get("RUNWAYML_API_SECRET").empty();
|
||||
for (const std::string& e : p->env)
|
||||
if (llm::Config::get(e.c_str()).empty()) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
std::shared_ptr<MediaService> MediaRegistry::service(const std::string& id, const std::string& key) {
|
||||
std::string slot = id;
|
||||
if (id == "fal") slot = key.rfind("platform.", 0) == 0 ? "fal.platform" : "fal.queue";
|
||||
auto it = services_.find(slot);
|
||||
if (it != services_.end()) return it->second;
|
||||
std::shared_ptr<MediaService> s;
|
||||
if (id == "stability") s = StabilityClient::from_env();
|
||||
else if (id == "bfl") s = BflClient::from_env();
|
||||
else if (id == "replicate") s = ReplicateClient::from_env();
|
||||
else if (id == "fal") s = slot == "fal.platform" ? std::shared_ptr<MediaService>(FalPlatform::from_env())
|
||||
: std::shared_ptr<MediaService>(FalQueue::from_env());
|
||||
else if (id == "ideogram") s = IdeogramClient::from_env();
|
||||
else if (id == "recraft") s = RecraftClient::from_env();
|
||||
else if (id == "leonardo") s = LeonardoClient::from_env();
|
||||
else if (id == "runway") s = RunwayClient::from_env();
|
||||
else if (id == "luma") s = LumaClient::from_env();
|
||||
else if (id == "minimax") s = MinimaxClient::from_env();
|
||||
else if (id == "kling") s = KlingClient::from_env();
|
||||
else if (id == "higgsfield") s = HiggsfieldClient::from_env();
|
||||
else throw llm::ApiError::invalid("media", "unknown provider '" + id + "'");
|
||||
services_[slot] = s;
|
||||
return s;
|
||||
}
|
||||
|
||||
std::shared_ptr<ImageGenerator> MediaRegistry::image(const std::string& id) {
|
||||
auto it = images_.find(id);
|
||||
if (it != images_.end()) return it->second;
|
||||
std::shared_ptr<ImageGenerator> g;
|
||||
if (id == "stability") g = std::make_shared<StabilityImageGenerator>(std::static_pointer_cast<StabilityClient>(service(id)));
|
||||
else if (id == "bfl") g = std::make_shared<BflImageGenerator>(std::static_pointer_cast<BflClient>(service(id)));
|
||||
else if (id == "replicate") g = std::make_shared<ReplicateImageGenerator>(std::static_pointer_cast<ReplicateClient>(service(id)));
|
||||
else if (id == "fal") g = std::make_shared<FalImageGenerator>(std::static_pointer_cast<FalQueue>(service(id)));
|
||||
else if (id == "ideogram") g = std::make_shared<IdeogramImageGenerator>(std::static_pointer_cast<IdeogramClient>(service(id)));
|
||||
else if (id == "recraft") g = std::make_shared<RecraftImageGenerator>(std::static_pointer_cast<RecraftClient>(service(id)));
|
||||
else if (id == "leonardo") g = std::make_shared<LeonardoImageGenerator>(std::static_pointer_cast<LeonardoClient>(service(id)));
|
||||
else if (id == "runway") g = std::make_shared<RunwayImageGenerator>(std::static_pointer_cast<RunwayClient>(service(id)));
|
||||
else if (id == "luma") g = std::make_shared<LumaImageGenerator>(std::static_pointer_cast<LumaClient>(service(id)));
|
||||
else if (id == "higgsfield") g = std::make_shared<HiggsfieldImageGenerator>(std::static_pointer_cast<HiggsfieldClient>(service(id)));
|
||||
else throw llm::ApiError::invalid("media", "provider '" + id + "' has no image generator");
|
||||
images_[id] = g;
|
||||
return g;
|
||||
}
|
||||
|
||||
std::shared_ptr<VideoGenerator> MediaRegistry::video(const std::string& id) {
|
||||
auto it = videos_.find(id);
|
||||
if (it != videos_.end()) return it->second;
|
||||
std::shared_ptr<VideoGenerator> g;
|
||||
if (id == "bfl") g = std::make_shared<BflVideoGenerator>(std::static_pointer_cast<BflClient>(service(id)));
|
||||
else if (id == "replicate") g = std::make_shared<ReplicateVideoGenerator>(std::static_pointer_cast<ReplicateClient>(service(id)));
|
||||
else if (id == "fal") g = std::make_shared<FalVideoGenerator>(std::static_pointer_cast<FalQueue>(service(id)));
|
||||
else if (id == "ideogram") g = std::make_shared<IdeogramVideoGenerator>(std::static_pointer_cast<IdeogramClient>(service(id)));
|
||||
else if (id == "leonardo") g = std::make_shared<LeonardoVideoGenerator>(std::static_pointer_cast<LeonardoClient>(service(id)));
|
||||
else if (id == "runway") g = std::make_shared<RunwayVideoGenerator>(std::static_pointer_cast<RunwayClient>(service(id)));
|
||||
else if (id == "luma") g = std::make_shared<LumaVideoGenerator>(std::static_pointer_cast<LumaClient>(service(id)));
|
||||
else if (id == "minimax") g = std::make_shared<MinimaxVideoGenerator>(std::static_pointer_cast<MinimaxClient>(service(id)));
|
||||
else if (id == "kling") g = std::make_shared<KlingVideoGenerator>(std::static_pointer_cast<KlingClient>(service(id)));
|
||||
else if (id == "higgsfield") g = std::make_shared<HiggsfieldVideoGenerator>(std::static_pointer_cast<HiggsfieldClient>(service(id)));
|
||||
else throw llm::ApiError::invalid("media", "provider '" + id + "' has no video generator");
|
||||
videos_[id] = g;
|
||||
return g;
|
||||
}
|
||||
|
||||
Json MediaRegistry::account(const std::string& id) {
|
||||
const ProviderInfo* p = find(id);
|
||||
if (!p) throw llm::ApiError::invalid("media", "unknown provider '" + id + "'");
|
||||
if (p->account_key.empty())
|
||||
throw llm::ApiError::invalid("media", id + " has no free account read; use a generation poll to prove the credential");
|
||||
std::shared_ptr<MediaService> s = service(id, p->account_key);
|
||||
CallArgs args;
|
||||
if (id == "fal") args.query["endpoint_id"] = "fal-ai/flux/dev";
|
||||
if (id == "stability") args.headers["Accept"] = "application/json";
|
||||
if (id == "ideogram") { // usage requires start_time (spec: required, date-time)
|
||||
std::time_t t = std::time(nullptr) - 24 * 3600;
|
||||
std::tm tm{};
|
||||
gmtime_r(&t, &tm);
|
||||
char buf[32];
|
||||
std::strftime(buf, sizeof buf, "%Y-%m-%dT%H:%M:%SZ", &tm);
|
||||
args.query["start_time"] = buf;
|
||||
}
|
||||
return s->call_json(p->account_key, args);
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+72
@@ -0,0 +1,72 @@
|
||||
#pragma once
|
||||
|
||||
// MediaRegistry: the one list of providers in package media, what each one
|
||||
// can do, which environment variables it needs and how well its API surface
|
||||
// is verified. Builds generators and call services from the environment on
|
||||
// demand and caches them. Products without a public API are listed too, with
|
||||
// the reason, so nothing is silently missing.
|
||||
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// How the provider's call table was checked (see each SPEC).
|
||||
enum class Evidence {
|
||||
kOpenApi, // vendor-published OpenAPI document, checked by tools/check_spec.py
|
||||
kSdk, // official SDK source, checked by tools/check_spec.py
|
||||
kDocs, // vendor doc pages, checked by tools/check_spec.py
|
||||
kProbeOnly // no machine-readable source: community sources plus live probes
|
||||
};
|
||||
|
||||
struct ProviderInfo {
|
||||
std::string id;
|
||||
std::string name;
|
||||
std::vector<std::string> env; // every variable the credential needs
|
||||
bool image = false;
|
||||
bool video = false;
|
||||
Evidence evidence = Evidence::kOpenApi;
|
||||
std::string console_url; // where the key is created
|
||||
std::string account_key; // table key of a free read that proves the credential ("" = none)
|
||||
std::string note;
|
||||
};
|
||||
|
||||
struct UnsupportedProduct {
|
||||
std::string name;
|
||||
std::string reason;
|
||||
};
|
||||
|
||||
class MediaRegistry {
|
||||
public:
|
||||
static const std::vector<ProviderInfo>& providers();
|
||||
static const std::vector<UnsupportedProduct>& unsupported();
|
||||
static const ProviderInfo* find(const std::string& id);
|
||||
static const char* evidence_name(Evidence evidence);
|
||||
// Table of one provider (no credential needed). Throws ApiError::invalid.
|
||||
static const CallTable& table(const std::string& id);
|
||||
|
||||
// True when every env variable the provider needs is set (llm::Config).
|
||||
static bool configured(const std::string& id);
|
||||
|
||||
// Built from the environment on first use, then cached for this registry.
|
||||
// `key` picks the host for providers with two (fal: queue.* / platform.*).
|
||||
std::shared_ptr<MediaService> service(const std::string& id, const std::string& key = "");
|
||||
std::shared_ptr<ImageGenerator> image(const std::string& id);
|
||||
std::shared_ptr<VideoGenerator> video(const std::string& id);
|
||||
|
||||
// Free read that proves the credential works (balance, account, models).
|
||||
// Returns the provider's JSON. Throws NotConfiguredError without a key.
|
||||
Json account(const std::string& id);
|
||||
|
||||
private:
|
||||
std::map<std::string, std::shared_ptr<MediaService>> services_;
|
||||
std::map<std::string, std::shared_ptr<ImageGenerator>> images_;
|
||||
std::map<std::string, std::shared_ptr<VideoGenerator>> videos_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+41
@@ -0,0 +1,41 @@
|
||||
#include "vendors/media/fal/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"queue.submit", "POST", "/{owner}/{app}", "submitQueueRequest", "json", "*", "fal_webhook,fal_max_queue_length", "paid"},
|
||||
{"queue.submit_path", "POST", "/{owner}/{app}/{path}", "submitQueueRequestWithPath", "json", "*", "fal_webhook,fal_max_queue_length", "paid"},
|
||||
{"queue.status", "GET", "/{owner}/{app}/requests/{request_id}/status", "getQueueRequestStatus", "none", "", "logs", "read"},
|
||||
{"queue.result", "GET", "/{owner}/{app}/requests/{request_id}", "getQueueRequestResult", "none", "", "", "read"},
|
||||
{"queue.cancel", "PUT", "/{owner}/{app}/requests/{request_id}/cancel", "cancelQueueRequest", "none", "", "", "write"},
|
||||
{"platform.models.insights", "GET", "/v1/models/insights", "getModelInsights", "none", "", "endpoint_id,q,limit", "read"},
|
||||
{"platform.models.search", "GET", "/v1/models", "getModels", "none", "", "limit,cursor,endpoint_id,q,category,status,sort,expand", "read"},
|
||||
{"platform.models.pricing", "GET", "/v1/models/pricing", "getPricing", "none", "", "endpoint_id", "read"},
|
||||
{"platform.models.estimate", "POST", "/v1/models/pricing/estimate", "estimatePricing", "json", "estimate_type,endpoints", "", "read"},
|
||||
{"platform.models.usage", "GET", "/v1/models/usage", "getUsage", "none", "", "limit,cursor,start,end,timezone,timeframe,bound_to_timeframe,source,endpoint_id,api_key_id,login_username,tag,expand", "read"},
|
||||
{"platform.models.analytics", "GET", "/v1/models/analytics", "getAnalytics", "none", "", "limit,cursor,start,end,timezone,timeframe,bound_to_timeframe,endpoint_id,expand", "read"},
|
||||
{"platform.models.billing_events", "GET", "/v1/models/billing-events", "getBillingEvents", "none", "", "limit,cursor,start,end,source,endpoint_id,request_id,tag,api_key_id,login_username,expand", "read"},
|
||||
{"platform.models.delete_payloads", "DELETE", "/v1/models/requests/{request_id}/payloads", "deleteRequestPayloads", "none", "", "", "delete"},
|
||||
{"platform.models.requests_by_endpoint", "GET", "/v1/models/requests/by-endpoint", "listRequestsByEndpoint", "none", "", "limit,cursor,endpoint_id,start,end,status,request_id,expand,sort_by", "read"},
|
||||
{"platform.models.requests_search", "GET", "/v1/models/requests/search", "searchRequests", "none", "", "limit,cursor,query,image_url,video_url,endpoint_id,endpoint,exclude_api_requests,only_api_requests,min_similarity", "read"},
|
||||
{"platform.meta", "GET", "/v1/meta", "getMeta", "none", "", "", "read"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
{"QueueStatus", "status,request_id,response_url,status_url,cancel_url,logs,metrics,queue_position"},
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& FalCalls::table() {
|
||||
static const CallTable t("fal", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t FalCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* FalCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct FalCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+260
@@ -0,0 +1,260 @@
|
||||
#include "vendors/media/fal/fal.h"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/caps.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs ids(const FalEndpoint& e, const std::string& request_id = "") {
|
||||
CallArgs a;
|
||||
a.path["owner"] = e.owner;
|
||||
a.path["app"] = e.app;
|
||||
if (!request_id.empty()) a.path["request_id"] = request_id;
|
||||
return a;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
FalEndpoint FalEndpoint::parse(const std::string& id) {
|
||||
std::vector<std::string> parts;
|
||||
std::size_t start = 0;
|
||||
while (true) {
|
||||
std::size_t slash = id.find('/', start);
|
||||
parts.push_back(id.substr(start, slash == std::string::npos ? std::string::npos : slash - start));
|
||||
if (slash == std::string::npos) break;
|
||||
start = slash + 1;
|
||||
}
|
||||
if (parts.size() < 2 || parts.size() > 5)
|
||||
throw llm::ApiError::invalid("fal", "endpoint must look like owner/app or owner/app/path");
|
||||
for (const std::string& p : parts)
|
||||
if (!CallTable::valid_segment(p)) throw llm::ApiError::invalid("fal", "bad endpoint id");
|
||||
FalEndpoint e;
|
||||
e.owner = parts[0];
|
||||
e.app = parts[1];
|
||||
// Everything below owner/app is the {path} of the submit_path row; the
|
||||
// call table accepts several "/"-joined segments for it.
|
||||
for (std::size_t i = 2; i < parts.size(); ++i) e.path += (i > 2 ? "/" : "") + parts[i];
|
||||
return e;
|
||||
}
|
||||
|
||||
PreparedCall FalQueue::plan(const FalEndpoint& e, const Json& body, const FalSubmitOptions& o) const {
|
||||
CallArgs a = ids(e);
|
||||
a.body = body;
|
||||
if (!e.path.empty()) {
|
||||
a.path["path"] = e.path;
|
||||
}
|
||||
if (!o.webhook.empty()) a.query["fal_webhook"] = o.webhook;
|
||||
if (o.max_queue_length > 0) a.query["fal_max_queue_length"] = std::to_string(o.max_queue_length);
|
||||
if (!o.priority.empty()) {
|
||||
if (o.priority != "low" && o.priority != "normal" && o.priority != "high")
|
||||
throw llm::ApiError::invalid("fal", "priority must be low, normal or high");
|
||||
a.headers["X-Fal-Queue-Priority"] = o.priority;
|
||||
}
|
||||
return prepare(e.path.empty() ? "queue.submit" : "queue.submit_path", a);
|
||||
}
|
||||
|
||||
Json FalQueue::submit(const FalEndpoint& e, const Json& body, const FalSubmitOptions& o, bool confirm) {
|
||||
return parse_json("fal", execute(plan(e, body, o), confirm));
|
||||
}
|
||||
|
||||
Json FalQueue::status(const FalEndpoint& e, const std::string& request_id, bool logs) {
|
||||
CallArgs a = ids(e, request_id);
|
||||
if (logs) a.query["logs"] = "1";
|
||||
return call_json("queue.status", a);
|
||||
}
|
||||
|
||||
Json FalQueue::result(const FalEndpoint& e, const std::string& request_id) {
|
||||
return call_json("queue.result", ids(e, request_id));
|
||||
}
|
||||
|
||||
Json FalQueue::cancel(const FalEndpoint& e, const std::string& request_id, bool confirm) {
|
||||
return call_json("queue.cancel", ids(e, request_id), confirm);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- platform
|
||||
|
||||
Json FalPlatform::models(const std::string& query, const std::string& category, std::size_t max_pages,
|
||||
std::size_t max_items) {
|
||||
if (max_pages < 1 || max_pages > MediaCaps::kMaxPages) throw llm::ApiError::invalid("fal", "max_pages out of range");
|
||||
if (max_items < 1 || max_items > MediaCaps::kMaxItemsReturned) throw llm::ApiError::invalid("fal", "max_items out of range");
|
||||
Json all = Json::array();
|
||||
std::string cursor;
|
||||
bool more = false;
|
||||
std::size_t pages = 0;
|
||||
while (pages < max_pages) {
|
||||
CallArgs a;
|
||||
a.query["limit"] = std::to_string(std::min<std::size_t>(max_items - all.as_array().size(), 100));
|
||||
if (!query.empty()) a.query["q"] = query;
|
||||
if (!category.empty()) a.query["category"] = category;
|
||||
if (!cursor.empty()) a.query["cursor"] = cursor;
|
||||
Json page = call_json("platform.models.search", a);
|
||||
++pages;
|
||||
if (const Json* m = page.find("models"); m && m->is_array())
|
||||
for (const Json& x : m->as_array())
|
||||
if (all.as_array().size() < max_items) all.push(x);
|
||||
const Json* hm = page.find("has_more");
|
||||
more = hm && hm->is_bool() && hm->as_bool();
|
||||
cursor = AdapterSupport::str(page, "next_cursor");
|
||||
if (!more || cursor.empty() || all.as_array().size() >= max_items) break;
|
||||
}
|
||||
Json out = Json::object();
|
||||
out.set("models", all);
|
||||
out.set("pages", Json(static_cast<double>(pages)));
|
||||
out.set("truncated", Json(more));
|
||||
return out;
|
||||
}
|
||||
|
||||
Json FalPlatform::pricing(const std::string& endpoint_id) {
|
||||
CallArgs a;
|
||||
a.query["endpoint_id"] = endpoint_id;
|
||||
return call_json("platform.models.pricing", a);
|
||||
}
|
||||
|
||||
Json FalPlatform::estimate(const Json& body) {
|
||||
CallArgs a;
|
||||
a.body = body;
|
||||
return call_json("platform.models.estimate", a);
|
||||
}
|
||||
|
||||
Json FalPlatform::usage(const std::map<std::string, std::string>& query) {
|
||||
CallArgs a;
|
||||
a.query = query;
|
||||
return call_json("platform.models.usage", a);
|
||||
}
|
||||
|
||||
Json FalPlatform::analytics(const std::map<std::string, std::string>& query) {
|
||||
CallArgs a;
|
||||
a.query = query;
|
||||
return call_json("platform.models.analytics", a);
|
||||
}
|
||||
|
||||
Json FalPlatform::meta() { return call_json("platform.meta"); }
|
||||
|
||||
Json FalPlatform::delete_payloads(const std::string& request_id, bool confirm) {
|
||||
CallArgs a;
|
||||
a.path["request_id"] = request_id;
|
||||
return call_json("platform.models.delete_payloads", a, confirm);
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------- jobs
|
||||
|
||||
JobTicket FalJobs::ticket_from(const Json& reply, const FalEndpoint& e, MediaKind kind) {
|
||||
JobTicket t;
|
||||
t.provider = "fal";
|
||||
t.model = e.full();
|
||||
t.kind = kind;
|
||||
t.id = AdapterSupport::str(reply, "request_id");
|
||||
if (t.id.empty()) throw llm::ApiError("fal", 502, "BAD_RESPONSE", "no request_id in the submit reply");
|
||||
t.meta = Json::object();
|
||||
t.meta.set("endpoint", Json(e.full()));
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus FalJobs::poll(FalQueue& queue, const JobTicket& ticket) {
|
||||
FalEndpoint e = FalEndpoint::parse(AdapterSupport::str(ticket.meta, "endpoint"));
|
||||
Json st = queue.status(e, ticket.id);
|
||||
JobStatus s;
|
||||
s.raw_state = AdapterSupport::str(st, "status");
|
||||
if (s.raw_state == "IN_QUEUE") {
|
||||
s.state = JobState::kQueued;
|
||||
return s;
|
||||
}
|
||||
if (s.raw_state != "COMPLETED") {
|
||||
s.state = JobState::kRunning;
|
||||
return s;
|
||||
}
|
||||
// COMPLETED is final for success, failure and cancellation alike: the
|
||||
// result fetch tells which (a failed request answers with an HTTP error).
|
||||
try {
|
||||
Json res = queue.result(e, ticket.id);
|
||||
s.state = JobState::kSucceeded;
|
||||
s.assets = OutputScanner::scan(res, ticket.kind);
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "completed without a result URL: " + res.dump().substr(0, 300);
|
||||
}
|
||||
} catch (const llm::ApiError& err) {
|
||||
if (err.http_status() == 401 || err.http_status() == 403 || err.http_status() >= 500 || err.http_status() == 429) throw;
|
||||
s.state = JobState::kFailed;
|
||||
s.error = err.message();
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
bool FalJobs::cancel(FalQueue& queue, const JobTicket& ticket, bool confirm) {
|
||||
AdapterSupport::need_confirm(confirm, "fal", "cancel");
|
||||
queue.cancel(FalEndpoint::parse(AdapterSupport::str(ticket.meta, "endpoint")), ticket.id, true);
|
||||
return true;
|
||||
}
|
||||
|
||||
Json FalImageGenerator::body(const ImageRequest& r) const {
|
||||
r.validate();
|
||||
Json b = Json::object();
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
AdapterSupport::put(b, "negative_prompt", r.negative_prompt);
|
||||
if (r.width && r.height) {
|
||||
Json size = Json::object();
|
||||
size.set("width", Json(r.width));
|
||||
size.set("height", Json(r.height));
|
||||
b.set("image_size", size);
|
||||
}
|
||||
AdapterSupport::put_seed(b, "seed", r.seed);
|
||||
if (r.count > 1) b.set("num_images", Json(r.count));
|
||||
AdapterSupport::put(b, "image_url", r.source_url);
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json FalImageGenerator::plan(const ImageRequest& r) const {
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("fal", "model (endpoint id) is required");
|
||||
return queue_->plan(FalEndpoint::parse(r.model), body(r), {}).summary();
|
||||
}
|
||||
|
||||
JobTicket FalImageGenerator::submit(const ImageRequest& r, bool confirm) {
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("fal", "model (endpoint id) is required");
|
||||
FalEndpoint e = FalEndpoint::parse(r.model);
|
||||
Json b = body(r);
|
||||
AdapterSupport::need_confirm(confirm, "fal", "queue submit");
|
||||
return FalJobs::ticket_from(queue_->submit(e, b, {}, true), e, MediaKind::kImage);
|
||||
}
|
||||
|
||||
JobStatus FalImageGenerator::poll(const JobTicket& t) { return FalJobs::poll(*queue_, t); }
|
||||
bool FalImageGenerator::cancel(const JobTicket& t, bool confirm) { return FalJobs::cancel(*queue_, t, confirm); }
|
||||
|
||||
Json FalVideoGenerator::body(const VideoRequest& r) const {
|
||||
r.validate();
|
||||
Json b = Json::object();
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
AdapterSupport::put(b, "negative_prompt", r.negative_prompt);
|
||||
if (r.duration_seconds > 0) b.set("duration", Json(std::to_string(r.duration_seconds)));
|
||||
AdapterSupport::put(b, "aspect_ratio", r.aspect_ratio);
|
||||
AdapterSupport::put(b, "resolution", r.resolution);
|
||||
AdapterSupport::put_seed(b, "seed", r.seed);
|
||||
AdapterSupport::put(b, "image_url", r.image_url);
|
||||
AdapterSupport::put(b, "end_image_url", r.end_image_url);
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json FalVideoGenerator::plan(const VideoRequest& r) const {
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("fal", "model (endpoint id) is required");
|
||||
return queue_->plan(FalEndpoint::parse(r.model), body(r), {}).summary();
|
||||
}
|
||||
|
||||
JobTicket FalVideoGenerator::submit(const VideoRequest& r, bool confirm) {
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("fal", "model (endpoint id) is required");
|
||||
FalEndpoint e = FalEndpoint::parse(r.model);
|
||||
Json b = body(r);
|
||||
AdapterSupport::need_confirm(confirm, "fal", "queue submit");
|
||||
return FalJobs::ticket_from(queue_->submit(e, b, {}, true), e, MediaKind::kVideo);
|
||||
}
|
||||
|
||||
JobStatus FalVideoGenerator::poll(const JobTicket& t) { return FalJobs::poll(*queue_, t); }
|
||||
bool FalVideoGenerator::cancel(const JobTicket& t, bool confirm) { return FalJobs::cancel(*queue_, t, confirm); }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+128
@@ -0,0 +1,128 @@
|
||||
#pragma once
|
||||
|
||||
// fal: the queue API (queue.fal.run: submit, status, result, cancel) and the
|
||||
// model-related Platform API (api.fal.ai/v1: model search, pricing, estimate,
|
||||
// usage, analytics, billing events, request listing). Auth is
|
||||
// "Authorization: Key <FAL_KEY>" on both hosts. Platform groups outside model
|
||||
// inference (assets, serverless, compute, keys, organization) are out of scope
|
||||
// and listed as exclusions in tools/check_spec.py.
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/fal/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// "owner/app" or "owner/app/path" (fal-ai/flux/dev, fal-ai/flux/dev/image-to-image).
|
||||
struct FalEndpoint {
|
||||
std::string owner, app, path;
|
||||
static FalEndpoint parse(const std::string& id); // throws ApiError::invalid
|
||||
std::string app_id() const { return owner + "/" + app; }
|
||||
std::string full() const { return path.empty() ? app_id() : app_id() + "/" + path; }
|
||||
};
|
||||
|
||||
struct FalSubmitOptions {
|
||||
std::string priority; // "", "low", "normal", "high" -> X-Fal-Queue-Priority
|
||||
std::string webhook; // https URL -> fal_webhook
|
||||
int max_queue_length = 0; // 0 = unset -> fal_max_queue_length
|
||||
};
|
||||
|
||||
class FalQueue : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://queue.fal.run";
|
||||
static constexpr const char* kKeyEnv = "FAL_KEY";
|
||||
|
||||
explicit FalQueue(std::shared_ptr<llm::ApiClient> client) : MediaService(std::move(client), FalCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::header("fal", kHost, kKeyEnv, "Authorization"); }
|
||||
// fal sends the key as "Key <value>".
|
||||
static std::string header_value(const std::string& key) { return key.empty() ? "" : "Key " + key; }
|
||||
static std::shared_ptr<FalQueue> from_env() {
|
||||
return std::make_shared<FalQueue>(ClientFactory::real(config(), header_value(ClientFactory::env({kKeyEnv}))));
|
||||
}
|
||||
static std::shared_ptr<FalQueue> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<FalQueue>(ClientFactory::with(std::move(t), config(), header_value(key)));
|
||||
}
|
||||
|
||||
PreparedCall plan(const FalEndpoint& endpoint, const Json& body, const FalSubmitOptions& options) const;
|
||||
// Paid. Returns {request_id, response_url, status_url, cancel_url, ...}.
|
||||
Json submit(const FalEndpoint& endpoint, const Json& body, const FalSubmitOptions& options, bool confirm);
|
||||
Json status(const FalEndpoint& endpoint, const std::string& request_id, bool logs = false);
|
||||
Json result(const FalEndpoint& endpoint, const std::string& request_id);
|
||||
Json cancel(const FalEndpoint& endpoint, const std::string& request_id, bool confirm);
|
||||
};
|
||||
|
||||
class FalPlatform : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api.fal.ai";
|
||||
|
||||
explicit FalPlatform(std::shared_ptr<llm::ApiClient> client) : MediaService(std::move(client), FalCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::header("fal", kHost, FalQueue::kKeyEnv, "Authorization"); }
|
||||
static std::shared_ptr<FalPlatform> from_env() {
|
||||
return std::make_shared<FalPlatform>(
|
||||
ClientFactory::real(config(), FalQueue::header_value(ClientFactory::env({FalQueue::kKeyEnv}))));
|
||||
}
|
||||
static std::shared_ptr<FalPlatform> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<FalPlatform>(ClientFactory::with(std::move(t), config(), FalQueue::header_value(key)));
|
||||
}
|
||||
|
||||
// Search; walks next_cursor up to max_pages / max_items and says so loudly
|
||||
// ("truncated": true) when the server had more.
|
||||
Json models(const std::string& query, const std::string& category, std::size_t max_pages = 3,
|
||||
std::size_t max_items = 200);
|
||||
Json pricing(const std::string& endpoint_id);
|
||||
Json estimate(const Json& body); // pure computation on fal's side, no charge
|
||||
Json usage(const std::map<std::string, std::string>& query);
|
||||
Json analytics(const std::map<std::string, std::string>& query);
|
||||
Json meta();
|
||||
Json delete_payloads(const std::string& request_id, bool confirm); // destructive, no default
|
||||
};
|
||||
|
||||
class FalImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit FalImageGenerator(std::shared_ptr<FalQueue> q) : queue_(std::move(q)) {}
|
||||
std::string provider() const override { return "fal"; }
|
||||
bool configured() const override { return queue_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"fal-ai/flux/dev", "fal-ai/flux/schnell", "fal-ai/flux-pro/v1.1", "fal-ai/recraft/v3/text-to-image",
|
||||
"fal-ai/ideogram/v3", "fal-ai/stable-diffusion-v35-large"};
|
||||
}
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override;
|
||||
int suggested_poll_ms() const override { return 1500; }
|
||||
|
||||
private:
|
||||
Json body(const ImageRequest& request) const;
|
||||
std::shared_ptr<FalQueue> queue_;
|
||||
};
|
||||
|
||||
class FalVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit FalVideoGenerator(std::shared_ptr<FalQueue> q) : queue_(std::move(q)) {}
|
||||
std::string provider() const override { return "fal"; }
|
||||
bool configured() const override { return queue_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"fal-ai/kling-video/v2.1/master/text-to-video", "fal-ai/minimax/hailuo-02/standard/text-to-video",
|
||||
"fal-ai/veo3", "fal-ai/luma-dream-machine"};
|
||||
}
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override;
|
||||
int suggested_poll_ms() const override { return 4000; }
|
||||
|
||||
private:
|
||||
Json body(const VideoRequest& request) const;
|
||||
std::shared_ptr<FalQueue> queue_;
|
||||
};
|
||||
|
||||
// Shared queue polling for both fal adapters.
|
||||
struct FalJobs {
|
||||
static JobTicket ticket_from(const Json& submit_reply, const FalEndpoint& endpoint, MediaKind kind);
|
||||
static JobStatus poll(FalQueue& queue, const JobTicket& ticket);
|
||||
static bool cancel(FalQueue& queue, const JobTicket& ticket, bool confirm);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
#include "vendors/media/higgsfield/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"higgsfield-ai.soul.standard", "POST", "/higgsfield-ai/soul/standard", "post_higgsfield_ai_soul_standard", "json", "prompt,num_images,resolution,aspect_ratio", "", "paid"},
|
||||
{"requests.id.status", "GET", "/requests/{request_id}/status", "get_request_status", "none", "", "", "read"},
|
||||
{"requests.id.cancel", "POST", "/requests/{request_id}/cancel", "cancel_request", "none", "", "", "write"},
|
||||
{"kling-video.v2.5-turbo.pro.image-to-video", "POST", "/kling-video/v2.5-turbo/pro/image-to-video", "post_kling_video_v2_5_turbo_pro_image_to_video", "json", "prompt,duration,cfg_scale,image_url,negative_prompt", "", "paid"},
|
||||
{"kling-video.v2.5-turbo.pro.text-to-video", "POST", "/kling-video/v2.5-turbo/pro/text-to-video", "post_kling_video_v2_5_turbo_pro_text_to_video", "json", "prompt,duration,cfg_scale,negative_prompt", "", "paid"},
|
||||
{"kling-video.v2.5-turbo.standard.image-to-video", "POST", "/kling-video/v2.5-turbo/standard/image-to-video", "post_kling_video_v2_5_turbo_standard_image_to_video", "json", "prompt,duration,cfg_scale,image_url,negative_prompt", "", "paid"},
|
||||
{"minimax.hailuo-2.3.standard.image-to-video", "POST", "/minimax/hailuo-2.3/standard/image-to-video", "post_minimax_hailuo_2_3_standard_image_to_video", "json", "prompt,duration,image_url,prompt_optimizer", "", "paid"},
|
||||
{"minimax.hailuo-2.3.standard.text-to-video", "POST", "/minimax/hailuo-2.3/standard/text-to-video", "post_minimax_hailuo_2_3_standard_text_to_video", "json", "prompt,duration,prompt_optimizer", "", "paid"},
|
||||
{"requests.id", "DELETE", "/requests/{request_id}", "delete_request", "none", "", "", "delete"},
|
||||
{"model.submit", "POST", "/{model_path}", "", "json", "*", "", "paid"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
{"RequestStatus", "status,request_id,status_url,cancel_url,error,images,video,url"},
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& HiggsfieldCalls::table() {
|
||||
static const CallTable t("higgsfield", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t HiggsfieldCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* HiggsfieldCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct HiggsfieldCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
#include "vendors/media/higgsfield/higgsfield.h"
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs for_model(const std::string& model_path, const Json& body) {
|
||||
CallArgs a;
|
||||
a.body = body;
|
||||
if (HiggsfieldClient::row_for(model_path) == "model.submit") a.path["model_path"] = model_path;
|
||||
return a;
|
||||
}
|
||||
|
||||
CallArgs by_request(const std::string& id) {
|
||||
CallArgs a;
|
||||
a.path["request_id"] = id;
|
||||
return a;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string HiggsfieldClient::row_for(const std::string& model_path) {
|
||||
std::string key = model_path;
|
||||
for (char& c : key)
|
||||
if (c == '/') c = '.';
|
||||
return HiggsfieldCalls::table().find(key) ? key : "model.submit";
|
||||
}
|
||||
|
||||
PreparedCall HiggsfieldClient::plan(const std::string& model_path, const Json& body) const {
|
||||
return prepare(row_for(model_path), for_model(model_path, body));
|
||||
}
|
||||
|
||||
Json HiggsfieldClient::submit(const std::string& model_path, const Json& body, bool confirm) {
|
||||
return call_json(row_for(model_path), for_model(model_path, body), confirm);
|
||||
}
|
||||
|
||||
Json HiggsfieldClient::status(const std::string& id) { return call_json("requests.id.status", by_request(id)); }
|
||||
Json HiggsfieldClient::cancel(const std::string& id, bool confirm) {
|
||||
return call_json("requests.id.cancel", by_request(id), confirm);
|
||||
}
|
||||
Json HiggsfieldClient::remove(const std::string& id, bool confirm) {
|
||||
return call_json("requests.id", by_request(id), confirm);
|
||||
}
|
||||
|
||||
JobState HiggsfieldJobs::state_of(const std::string& status) {
|
||||
if (status == "completed") return JobState::kSucceeded;
|
||||
if (status == "failed" || status == "nsfw") return JobState::kFailed;
|
||||
if (status == "canceled") return JobState::kCancelled;
|
||||
if (status == "in_progress") return JobState::kRunning;
|
||||
return JobState::kQueued; // queued
|
||||
}
|
||||
|
||||
JobTicket HiggsfieldJobs::ticket_from(const Json& reply, const std::string& model, MediaKind kind) {
|
||||
JobTicket t;
|
||||
t.provider = "higgsfield";
|
||||
t.model = model;
|
||||
t.kind = kind;
|
||||
t.id = AdapterSupport::str(reply, "request_id");
|
||||
if (t.id.empty()) throw llm::ApiError("higgsfield", 502, "BAD_RESPONSE", "no request_id in the reply");
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus HiggsfieldJobs::poll(HiggsfieldClient& client, const JobTicket& ticket) {
|
||||
Json st = client.status(ticket.id);
|
||||
JobStatus s;
|
||||
s.raw_state = AdapterSupport::str(st, "status");
|
||||
s.state = state_of(s.raw_state);
|
||||
if (s.raw_state == "nsfw") s.error = "rejected by the content filter (nsfw)";
|
||||
if (s.state == JobState::kFailed && s.error.empty()) s.error = AdapterSupport::str(st, "error");
|
||||
if (s.state == JobState::kSucceeded) {
|
||||
for (const char* key : {"images", "video"})
|
||||
if (const Json* x = st.find(key)) {
|
||||
auto found = OutputScanner::scan(*x, ticket.kind);
|
||||
s.assets.insert(s.assets.end(), found.begin(), found.end());
|
||||
}
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "completed without images or video";
|
||||
}
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
Json HiggsfieldImageGenerator::body(const ImageRequest& r, std::string& model) const {
|
||||
r.validate();
|
||||
model = r.model.empty() ? "higgsfield-ai/soul/standard" : r.model;
|
||||
Json b = Json::object();
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
if (r.count > 1) b.set("num_images", Json(r.count));
|
||||
AdapterSupport::put(b, "aspect_ratio", r.aspect_ratio);
|
||||
AdapterSupport::put_seed(b, "seed", r.seed);
|
||||
AdapterSupport::put(b, "image_url", r.source_url);
|
||||
if (r.width || r.height || !r.negative_prompt.empty())
|
||||
throw llm::ApiError::invalid("higgsfield", "width / height / negative_prompt are model-specific: pass them in extra");
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json HiggsfieldImageGenerator::plan(const ImageRequest& request) const {
|
||||
std::string model;
|
||||
Json b = body(request, model);
|
||||
return client_->plan(model, b).summary();
|
||||
}
|
||||
|
||||
JobTicket HiggsfieldImageGenerator::submit(const ImageRequest& request, bool confirm) {
|
||||
std::string model;
|
||||
Json b = body(request, model);
|
||||
AdapterSupport::need_confirm(confirm, "higgsfield", "submit");
|
||||
return HiggsfieldJobs::ticket_from(client_->submit(model, b, true), model, MediaKind::kImage);
|
||||
}
|
||||
|
||||
bool HiggsfieldImageGenerator::cancel(const JobTicket& t, bool confirm) {
|
||||
AdapterSupport::need_confirm(confirm, "higgsfield", "cancel");
|
||||
client_->cancel(t.id, true);
|
||||
return true;
|
||||
}
|
||||
|
||||
Json HiggsfieldVideoGenerator::body(const VideoRequest& r, std::string& model) const {
|
||||
r.validate();
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("higgsfield", "model path is required (see models())");
|
||||
model = r.model;
|
||||
Json b = Json::object();
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
AdapterSupport::put(b, "negative_prompt", r.negative_prompt);
|
||||
AdapterSupport::put(b, "aspect_ratio", r.aspect_ratio);
|
||||
AdapterSupport::put(b, "resolution", r.resolution);
|
||||
AdapterSupport::put_int(b, "duration", r.duration_seconds);
|
||||
AdapterSupport::put(b, "image_url", r.image_url);
|
||||
AdapterSupport::put(b, "end_image_url", r.end_image_url);
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json HiggsfieldVideoGenerator::plan(const VideoRequest& request) const {
|
||||
std::string model;
|
||||
Json b = body(request, model);
|
||||
return client_->plan(model, b).summary();
|
||||
}
|
||||
|
||||
JobTicket HiggsfieldVideoGenerator::submit(const VideoRequest& request, bool confirm) {
|
||||
std::string model;
|
||||
Json b = body(request, model);
|
||||
AdapterSupport::need_confirm(confirm, "higgsfield", "submit");
|
||||
return HiggsfieldJobs::ticket_from(client_->submit(model, b, true), model, MediaKind::kVideo);
|
||||
}
|
||||
|
||||
bool HiggsfieldVideoGenerator::cancel(const JobTicket& t, bool confirm) {
|
||||
AdapterSupport::need_confirm(confirm, "higgsfield", "cancel");
|
||||
client_->cancel(t.id, true);
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+94
@@ -0,0 +1,94 @@
|
||||
#pragma once
|
||||
|
||||
// Higgsfield cloud API (https://api.higgsfield.ai): every model is an
|
||||
// endpoint path ("higgsfield-ai/soul/standard", "kling-video/v2.5-turbo/pro/
|
||||
// text-to-video", ...); the shared lifecycle is submit -> GET
|
||||
// /requests/{id}/status -> optional cancel / delete. Auth is
|
||||
// "Authorization: Key <HF_API_KEY_ID>:<HF_API_KEY_SECRET>". The published
|
||||
// OpenAPI document is explicitly supplementary: a model absent from it is not
|
||||
// unavailable, so submit() accepts any model path through the generic row.
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/higgsfield/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class HiggsfieldClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api.higgsfield.ai";
|
||||
static constexpr const char* kIdEnv = "HF_API_KEY_ID";
|
||||
static constexpr const char* kSecretEnv = "HF_API_KEY_SECRET";
|
||||
|
||||
explicit HiggsfieldClient(std::shared_ptr<llm::ApiClient> client)
|
||||
: MediaService(std::move(client), HiggsfieldCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::header("higgsfield", kHost, kIdEnv, "Authorization"); }
|
||||
// "Key id:secret"; empty when either half is missing.
|
||||
static std::string header_value(const std::string& id, const std::string& secret) {
|
||||
return id.empty() || secret.empty() ? "" : "Key " + id + ":" + secret;
|
||||
}
|
||||
static std::shared_ptr<HiggsfieldClient> from_env() {
|
||||
return std::make_shared<HiggsfieldClient>(ClientFactory::real(
|
||||
config(), header_value(ClientFactory::env({kIdEnv}), ClientFactory::env({kSecretEnv}))));
|
||||
}
|
||||
static std::shared_ptr<HiggsfieldClient> with(std::unique_ptr<clients::Transport> t, const std::string& id,
|
||||
const std::string& secret) {
|
||||
return std::make_shared<HiggsfieldClient>(ClientFactory::with(std::move(t), config(), header_value(id, secret)));
|
||||
}
|
||||
|
||||
// Uses the spec's typed row when the path is one of them, else model.submit.
|
||||
Json submit(const std::string& model_path, const Json& body, bool confirm);
|
||||
PreparedCall plan(const std::string& model_path, const Json& body) const;
|
||||
Json status(const std::string& request_id);
|
||||
Json cancel(const std::string& request_id, bool confirm); // queued requests only
|
||||
Json remove(const std::string& request_id, bool confirm); // deletes files and parameters
|
||||
static std::string row_for(const std::string& model_path);
|
||||
};
|
||||
|
||||
class HiggsfieldJobs {
|
||||
public:
|
||||
static JobTicket ticket_from(const Json& reply, const std::string& model, MediaKind kind);
|
||||
static JobStatus poll(HiggsfieldClient& client, const JobTicket& ticket);
|
||||
static JobState state_of(const std::string& status);
|
||||
};
|
||||
|
||||
class HiggsfieldImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit HiggsfieldImageGenerator(std::shared_ptr<HiggsfieldClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "higgsfield"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override { return {"higgsfield-ai/soul/standard"}; }
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override { return HiggsfieldJobs::poll(*client_, ticket); }
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override;
|
||||
int suggested_poll_ms() const override { return 3000; }
|
||||
|
||||
private:
|
||||
Json body(const ImageRequest& request, std::string& model) const;
|
||||
std::shared_ptr<HiggsfieldClient> client_;
|
||||
};
|
||||
|
||||
class HiggsfieldVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit HiggsfieldVideoGenerator(std::shared_ptr<HiggsfieldClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "higgsfield"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"kling-video/v2.5-turbo/pro/text-to-video", "kling-video/v2.5-turbo/pro/image-to-video",
|
||||
"kling-video/v2.5-turbo/standard/image-to-video", "minimax/hailuo-2.3/standard/text-to-video",
|
||||
"minimax/hailuo-2.3/standard/image-to-video"};
|
||||
}
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override { return HiggsfieldJobs::poll(*client_, ticket); }
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override;
|
||||
int suggested_poll_ms() const override { return 5000; }
|
||||
|
||||
private:
|
||||
Json body(const VideoRequest& request, std::string& model) const;
|
||||
std::shared_ptr<HiggsfieldClient> client_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+96
@@ -0,0 +1,96 @@
|
||||
#include "vendors/media/ideogram/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"image.precise-edit.ideogram-4-5", "POST", "/v2/image/precise-edit/ideogram-4-5", "imagesPreciseEdit_post_precise_edit_image_v2_ideogram45", "multipart", "prompt,image,reference_images,mask,context_window,quality,seed,num_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.auto", "POST", "/v2/image/generate/auto", "autoModel_post_generate_image_v2_auto", "multipart", "prompt,images,negative_prompt,aspect_ratio,resolution,resolution_tier,magic_prompt,num_images,seed,private,async", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-4-5", "POST", "/v2/image/generate/ideogram-4-5", "imagesGenerate_post_generate_image_v2_ideogram45", "multipart", "prompt,magic_prompt,images,mask,size,background,quality,seed,num_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-4", "POST", "/v2/image/generate/ideogram-4", "imagesGenerate_post_generate_image_v2_ideogram_v4", "json", "prompt,magic_prompt,seed,num_images,resolution,rendering_speed,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-4-custom-model", "POST", "/v2/image/generate/ideogram-4-custom-model", "imagesGenerate_post_generate_image_v2_ideogram_v4_custom_model", "json", "prompt,custom_model_uri,transparent_background,magic_prompt,seed,num_images,resolution,rendering_speed,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-4-transparent", "POST", "/v2/image/generate/ideogram-4-transparent", "imagesGenerate_post_generate_image_v2_ideogram_v4_transparent", "json", "prompt,magic_prompt,seed,num_images,aspect_ratio,output_resolution,rendering_speed,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-3", "POST", "/v2/image/generate/ideogram-3", "imagesGenerate_post_generate_image_v2_ideogram_v3", "multipart", "prompt,negative_prompt,seed,resolution,aspect_ratio,rendering_speed,magic_prompt,num_images,color_palette,style_codes,style_type,style_preset,style_reference_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-3-character", "POST", "/v2/image/generate/ideogram-3-character", "imagesGenerate_post_generate_image_v2_ideogram_v3_character", "multipart", "prompt,negative_prompt,character_reference_images,character_reference_mask,seed,resolution,aspect_ratio,rendering_speed,magic_prompt,num_images,style_type,style_reference_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-3-custom-model", "POST", "/v2/image/generate/ideogram-3-custom-model", "imagesGenerate_post_generate_image_v2_ideogram_v3_custom_model", "multipart", "prompt,custom_model_uri,transparent_background,negative_prompt,seed,resolution,aspect_ratio,rendering_speed,magic_prompt,num_images,color_palette,style_codes,style_preset,style_reference_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-3-transparent", "POST", "/v2/image/generate/ideogram-3-transparent", "imagesGenerate_post_generate_image_v2_ideogram_v3_transparent", "json", "prompt,negative_prompt,seed,upscale_factor,aspect_ratio,rendering_speed,magic_prompt,num_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-2a", "POST", "/v2/image/generate/ideogram-2a", "imagesGenerate_post_generate_image_v2_ideogram_v2_a", "json", "prompt,seed,aspect_ratio,rendering_speed,magic_prompt,num_images,style_type,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.ideogram-2", "POST", "/v2/image/generate/ideogram-2", "imagesGenerate_post_generate_image_v2_ideogram_v2", "json", "prompt,negative_prompt,seed,aspect_ratio,rendering_speed,magic_prompt,num_images,style_type,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.gpt-image-2-5-flare", "POST", "/v2/image/generate/gpt-image-2-5-flare", "imagesGenerate_post_generate_image_v2_gpt_image25_flare", "multipart", "prompt,images,mask,background,quality,num_images,seed,aspect_ratio,resolution,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.gpt-image-2-5-sunburst", "POST", "/v2/image/generate/gpt-image-2-5-sunburst", "imagesGenerate_post_generate_image_v2_gpt_image25_sunburst", "multipart", "prompt,images,mask,background,quality,num_images,seed,aspect_ratio,resolution,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.gpt-image-2", "POST", "/v2/image/generate/gpt-image-2", "imagesGenerate_post_generate_image_v2_gpt_image2", "multipart", "prompt,images,mask,num_images,seed,aspect_ratio,resolution,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.nano-banana-2", "POST", "/v2/image/generate/nano-banana-2", "imagesGenerate_post_generate_image_v2_nano_banana2", "multipart", "prompt,images,aspect_ratio,resolution_tier,num_images,seed,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.nano-banana-2-1", "POST", "/v2/image/generate/nano-banana-2-1", "imagesGenerate_post_generate_image_v2_nano_banana21", "multipart", "prompt,images,aspect_ratio,resolution_tier,num_images,seed,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.nano-banana-pro", "POST", "/v2/image/generate/nano-banana-pro", "imagesGenerate_post_generate_image_v2_nano_banana_pro", "multipart", "prompt,images,aspect_ratio,resolution_tier,num_images,seed,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.p-image-ideogram", "POST", "/v2/image/generate/p-image-ideogram", "imagesGenerate_post_generate_image_v2_p_image_ideogram", "json", "prompt,magic_prompt,seed,num_images,quality,resolution,aspect_ratio,custom_width,custom_height,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.generate.z-image", "POST", "/v2/image/generate/z-image", "imagesGenerate_post_generate_image_v2_z_image", "json", "prompt,resolution,num_inference_steps,seed,num_images,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.remix.auto", "POST", "/v2/image/remix/auto", "autoModel_post_tool_remix", "multipart", "prompt,image,image_weight,negative_prompt,resolution,aspect_ratio,resolution_tier,magic_prompt,seed,style_preset,color_palette,style_codes,style_type,num_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.remix.ideogram-4", "POST", "/v2/image/remix/ideogram-4", "imagesRemix_post_remix_image_v2_ideogram_v4", "multipart", "prompt,image,image_weight,num_images,resolution,rendering_speed,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.remix.ideogram-3", "POST", "/v2/image/remix/ideogram-3", "imagesRemix_post_remix_image_v2_ideogram_v3", "multipart", "prompt,image,image_weight,negative_prompt,seed,resolution,aspect_ratio,rendering_speed,magic_prompt,num_images,color_palette,style_codes,style_type,style_preset,style_reference_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.remix.ideogram-3-character", "POST", "/v2/image/remix/ideogram-3-character", "imagesRemix_post_remix_image_v2_ideogram_v3_character", "multipart", "prompt,image,image_weight,negative_prompt,character_reference_images,character_reference_mask,seed,resolution,aspect_ratio,rendering_speed,magic_prompt,num_images,style_type,style_reference_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.remix.ideogram-3-custom-model", "POST", "/v2/image/remix/ideogram-3-custom-model", "imagesRemix_post_remix_image_v2_ideogram_v3_custom_model", "multipart", "prompt,custom_model_uri,image,image_weight,negative_prompt,seed,resolution,aspect_ratio,rendering_speed,magic_prompt,num_images,color_palette,style_codes,style_preset,style_reference_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.inpaint.ideogram-3", "POST", "/v2/image/inpaint/ideogram-3", "imagesInpaint_post_inpaint_image_v2_ideogram_v3", "multipart", "prompt,image,mask,magic_prompt,num_images,seed,rendering_speed,style_type,style_preset,style_codes,style_reference_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.inpaint.ideogram-3-character", "POST", "/v2/image/inpaint/ideogram-3-character", "imagesInpaint_post_inpaint_image_v2_ideogram_v3_character", "multipart", "prompt,image,mask,character_reference_images,character_reference_mask,magic_prompt,num_images,seed,rendering_speed,style_type,style_codes,style_reference_images,enable_copyright_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.inpaint.ideogram-3-custom-model", "POST", "/v2/image/inpaint/ideogram-3-custom-model", "imagesInpaint_post_inpaint_image_v2_ideogram_v3_custom_model", "multipart", "prompt,custom_model_uri,image,mask,magic_prompt,num_images,seed,rendering_speed,style_preset,style_codes,style_reference_images,enable_copyright_detection,async", "dry_run", "paid"},
|
||||
{"image.reframe.ideogram-3", "POST", "/v2/image/reframe/ideogram-3", "reframe_ideogramV3", "multipart", "image,resolution,num_images,seed,rendering_speed,style_preset,color_palette,style_codes,style_reference_images,async", "dry_run", "paid"},
|
||||
{"image.reframe.nano-banana-2", "POST", "/v2/image/reframe/nano-banana-2", "reframe_nanoBanana2", "multipart", "image,aspect_ratio,private,num_images,seed", "dry_run", "paid"},
|
||||
{"image.upscale.auto", "POST", "/v2/image/upscale/auto", "tools_post_upscale", "multipart", "image,upscale_factor,seed,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.upscale.nano-banana-pro", "POST", "/v2/image/upscale/nano-banana-pro", "imagesUpscale_post_upscale_image_nano_banana_pro", "multipart", "image,prompt,resolution_tier,seed,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.upscale.topaz-bloom-2", "POST", "/v2/image/upscale/topaz-bloom-2", "imagesUpscale_post_upscale_image_topaz_bloom2", "multipart", "image,upscale_factor,prompt,creativity,color_preservation,seed,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.upscale.topaz-redefine", "POST", "/v2/image/upscale/topaz-redefine", "imagesUpscale_post_upscale_image_topaz_redefine", "multipart", "image,upscale_factor,prompt,creativity,autoprompt,texture,detail,detail_strength,sharpen,denoise,face_enhancement,face_enhancement_strength,face_enhancement_creativity,subject_detection,seed,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.upscale.topaz-standard-2", "POST", "/v2/image/upscale/topaz-standard-2", "imagesUpscale_post_upscale_image_topaz_standard_v2", "multipart", "image,upscale_factor,strength,sharpen,denoise,fix_compression,face_enhancement,face_enhancement_strength,face_enhancement_creativity,subject_detection,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.upscale.topaz-text-refine", "POST", "/v2/image/upscale/topaz-text-refine", "imagesUpscale_post_upscale_image_topaz_text_refine", "multipart", "image,upscale_factor,strength,sharpen,denoise,fix_compression,face_enhancement,face_enhancement_strength,face_enhancement_creativity,subject_detection,seed,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.upscale.topaz-wonder-3-5", "POST", "/v2/image/upscale/topaz-wonder-3-5", "imagesUpscale_post_upscale_image_topaz_wonder35", "multipart", "image,upscale_factor,enhancement_strength,seed,async,webhook_url", "dry_run", "paid"},
|
||||
{"image.replace-background.ideogram-3", "POST", "/v2/image/replace-background/ideogram-3", "tool_replaceBackgroundIdeogramV3", "multipart", "image,prompt,rendering_speed,num_images,private,webhook_url", "dry_run", "paid"},
|
||||
{"image.replace-background.gpt-image-2", "POST", "/v2/image/replace-background/gpt-image-2", "tool_replaceBackground", "multipart", "image,prompt,quality,num_images,private,webhook_url", "dry_run", "paid"},
|
||||
{"image.remove-background.ideogram-1", "POST", "/v2/image/remove-background/ideogram-1", "tool_removeBackground", "multipart", "image,private,async", "dry_run", "paid"},
|
||||
{"image.remove-object.ideogram-1", "POST", "/v2/image/remove-object/ideogram-1", "objectRemoval_post_remove_object_from_v2_assets", "multipart", "image,mask,seed", "dry_run", "paid"},
|
||||
{"image.describe.ideogram-4", "POST", "/v2/image/describe/ideogram-4", "imageDescribe_post_describe_image_ideogram_v4", "multipart", "image,include_bbox,include_style_descriptions,include_tags", "", "paid"},
|
||||
{"image.describe.ideogram-3", "POST", "/v2/image/describe/ideogram-3", "imageDescribe_post_describe_image_ideogram_v3", "multipart", "image", "", "paid"},
|
||||
{"design.layerize.ideogram-3", "POST", "/v2/design/layerize/ideogram-3", "design_post_layerize_design_ideogram_v3", "multipart", "image,prompt,seed,font_candidate_files,async,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.kling-3-standard-image-to-video", "POST", "/v2/video/generate/kling-3-standard-image-to-video", "videoGenerate_post_generate_video_kling_v3_standard_image_to_video", "multipart", "image,end_image,prompt,negative_prompt,duration,generate_audio,shot_type,cfg_scale,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.kling-3-standard-text-to-video", "POST", "/v2/video/generate/kling-3-standard-text-to-video", "videoGenerate_post_generate_video_kling_v3_standard_text_to_video", "json", "prompt,negative_prompt,aspect_ratio,duration,generate_audio,shot_type,cfg_scale,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.minimax-h3-image-to-video", "POST", "/v2/video/generate/minimax-h3-image-to-video", "videoGenerate_post_generate_video_minimax_h3_image_to_video", "multipart", "image,end_image,prompt,resolution,duration,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.minimax-h3-reference-to-video", "POST", "/v2/video/generate/minimax-h3-reference-to-video", "videoEdit_post_edit_video_minimax_h3_reference_to_video", "multipart", "prompt,reference_images,reference_audios,aspect_ratio,resolution,duration,prompt_expansion_mode,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.minimax-h3-text-to-video", "POST", "/v2/video/generate/minimax-h3-text-to-video", "videoGenerate_post_generate_video_minimax_h3_text_to_video", "json", "prompt,aspect_ratio,resolution,duration,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.seedance-2-5-image-to-video", "POST", "/v2/video/generate/seedance-2-5-image-to-video", "videoGenerate_post_generate_video_seed_dance25_image_to_video", "multipart", "image,end_image,prompt,resolution,duration,generate_audio,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.seedance-2-5-reference-to-video", "POST", "/v2/video/generate/seedance-2-5-reference-to-video", "videoGenerate_post_generate_video_seed_dance25_reference_to_video", "multipart", "prompt,reference_images,reference_audios,aspect_ratio,resolution,duration,generate_audio,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.seedance-2-5-text-to-video", "POST", "/v2/video/generate/seedance-2-5-text-to-video", "videoGenerate_post_generate_video_seed_dance25_text_to_video", "json", "prompt,aspect_ratio,resolution,duration,generate_audio,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.seedance-2-image-to-video", "POST", "/v2/video/generate/seedance-2-image-to-video", "videoGenerate_post_generate_video_seed_dance2_image_to_video", "multipart", "image,end_image,prompt,aspect_ratio,resolution,duration,generate_audio,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.seedance-2-reference-to-video", "POST", "/v2/video/generate/seedance-2-reference-to-video", "videoGenerate_post_generate_video_seed_dance2_reference_to_video", "multipart", "prompt,reference_images,reference_audios,aspect_ratio,resolution,duration,generate_audio,webhook_url", "dry_run", "paid"},
|
||||
{"video.generate.seedance-2-text-to-video", "POST", "/v2/video/generate/seedance-2-text-to-video", "videoGenerate_post_generate_video_seed_dance2_text_to_video", "json", "prompt,aspect_ratio,resolution,duration,generate_audio,webhook_url", "dry_run", "paid"},
|
||||
{"svg.generate.ideogram-vector-1", "POST", "/v2/svg/generate/ideogram-vector-1", "svgGenerate_post_generate_svg", "multipart", "prompt,image_asset_identifiers,images,aspect_ratio,quality,num_images,seed,private,target_collection_id,webhook_url", "dry_run", "paid"},
|
||||
{"svg.vectorize.ideogram-vector-1", "POST", "/v2/svg/vectorize/ideogram-vector-1", "svgGenerate_post_vectorizer", "multipart", "image,image_asset_identifier,mode,private,target_collection_id,webhook_url", "dry_run", "paid"},
|
||||
{"tool.ad-localizer", "POST", "/v2/tool/ad-localizer", "editWorkflow_post_ad_localizer", "multipart", "image,language,exact_copy,prompt,quality,seed,num_images,webhook_url", "dry_run", "paid"},
|
||||
{"tool.ad-resizer", "POST", "/v2/tool/ad-resizer", "editWorkflow_post_ad_resizer", "multipart", "image,resolution,platform,prompt,quality,num_images,webhook_url", "dry_run", "paid"},
|
||||
{"tool.ad-variations", "POST", "/v2/tool/ad-variations", "editWorkflow_post_ad_variations", "multipart", "image,variation_type,prompt,quality,num_images,webhook_url", "", "paid"},
|
||||
{"tool.colorways", "POST", "/v2/tool/colorways", "editWorkflow_post_colorways", "multipart", "image,masks,colors,aspect_ratio,quality,webhook_url", "dry_run", "paid"},
|
||||
{"tool.ghost-mannequin", "POST", "/v2/tool/ghost-mannequin", "editWorkflow_post_ghost_mannequin", "multipart", "garment_images,front_image,back_image,left_image,right_image,top_image,bottom_image,output_reference_asset_identifier,output_reference_image,view,instruction,metadata,aspect_ratio,quality,seed,webhook_url", "dry_run", "paid"},
|
||||
{"tool.material-swap", "POST", "/v2/tool/material-swap", "editWorkflow_post_material_swap", "multipart", "image,masks,materials,aspect_ratio,quality,webhook_url", "dry_run", "paid"},
|
||||
{"tool.model-pose-variants", "POST", "/v2/tool/model-pose-variants", "editWorkflow_post_model_pose_variants", "multipart", "source_image,instruction,aspect_ratio,quality,seed,num_images,webhook_url", "", "paid"},
|
||||
{"tool.sketch-to-render", "POST", "/v2/tool/sketch-to-render", "editWorkflow_post_sketch_to_render", "multipart", "sketch_image,instruction,aspect_ratio,quality,seed,num_images,webhook_url", "", "paid"},
|
||||
{"generations.id", "GET", "/v2/generations/{generation_id}", "generations_get_generation_v2", "none", "", "", "read"},
|
||||
{"account.api-keys", "GET", "/v2/account/api-keys", "account_get_account_api_keys", "none", "", "", "read"},
|
||||
{"account.invoices", "GET", "/v2/account/invoices", "account_get_account_invoices", "none", "", "", "read"},
|
||||
{"account.usage", "GET", "/v2/account/usage", "account_get_account_usage", "none", "", "start_time,end_time,bucket_width,sources", "read"},
|
||||
{"image.replace-background.ideogram-4-5", "POST", "/v2/image/replace-background/ideogram-4-5", "tool_replaceBackgroundIdeogram45", "multipart", "image,background_reference,num_images,private,webhook_url", "dry_run", "paid"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
{"GenerateImageIdeogramV3Response", "generation_id,data,seed"},
|
||||
{"GenerationResponse", "generation_id,status,created,response_type,usage_cost_usd_micros,failure_reason,data"},
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& IdeogramCalls::table() {
|
||||
static const CallTable t("ideogram", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t IdeogramCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* IdeogramCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct IdeogramCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+174
@@ -0,0 +1,174 @@
|
||||
#include "vendors/media/ideogram/ideogram.h"
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs args(const Json& fields, const std::vector<FilePart>& files, bool dry_run) {
|
||||
CallArgs a;
|
||||
a.body = fields;
|
||||
a.files = files;
|
||||
if (dry_run) a.query["dry_run"] = "true";
|
||||
return a;
|
||||
}
|
||||
|
||||
// Image routes take `async` in the body; video routes are always asynchronous.
|
||||
bool is_async(const Json& fields) {
|
||||
const Json* a = fields.find("async");
|
||||
return a && a->is_bool() && a->as_bool();
|
||||
}
|
||||
|
||||
JobStatus status_from(const Json& g, MediaKind kind) {
|
||||
JobStatus s;
|
||||
s.raw_state = AdapterSupport::str(g, "status");
|
||||
if (s.raw_state == "completed") {
|
||||
s.state = JobState::kSucceeded;
|
||||
if (const Json* d = g.find("data")) s.assets = OutputScanner::scan(*d, kind);
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "completed without a URL (images that fail the safety check have an empty url)";
|
||||
}
|
||||
} else if (s.raw_state == "failed") {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = AdapterSupport::str(g, "failure_reason");
|
||||
} else {
|
||||
s.state = s.raw_state == "pending" ? JobState::kRunning : JobState::kQueued;
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
PreparedCall IdeogramClient::plan(const std::string& key, const Json& fields, const std::vector<FilePart>& files,
|
||||
bool dry_run) const {
|
||||
return prepare(key, args(fields, files, dry_run));
|
||||
}
|
||||
|
||||
Json IdeogramClient::generate(const std::string& key, const Json& fields, const std::vector<FilePart>& files,
|
||||
bool confirm) {
|
||||
return call_json(key, args(fields, files, false), confirm);
|
||||
}
|
||||
|
||||
Json IdeogramClient::quote(const std::string& key, const Json& fields, const std::vector<FilePart>& files) {
|
||||
PreparedCall c = plan(key, fields, files, true);
|
||||
c.effect = Effect::kRead; // dry_run=true is defined by the spec as not billed
|
||||
return parse_json("ideogram", execute(c, false));
|
||||
}
|
||||
|
||||
Json IdeogramClient::generation(const std::string& id) {
|
||||
CallArgs a;
|
||||
a.path["generation_id"] = id;
|
||||
return call_json("generations.id", a);
|
||||
}
|
||||
|
||||
Json IdeogramClient::usage(const std::map<std::string, std::string>& query) {
|
||||
CallArgs a;
|
||||
a.query = query;
|
||||
return call_json("account.usage", a);
|
||||
}
|
||||
Json IdeogramClient::api_keys() { return call_json("account.api-keys"); }
|
||||
Json IdeogramClient::invoices() { return call_json("account.invoices"); }
|
||||
|
||||
std::string IdeogramImageGenerator::key_for(const std::string& model) {
|
||||
std::string m = model.empty() ? "ideogram-3" : model;
|
||||
return "image.generate." + m;
|
||||
}
|
||||
|
||||
Json IdeogramImageGenerator::fields(const ImageRequest& r, std::string& key) const {
|
||||
r.validate();
|
||||
key = key_for(r.model);
|
||||
Json f = Json::object();
|
||||
AdapterSupport::put(f, "prompt", r.prompt);
|
||||
AdapterSupport::put(f, "negative_prompt", r.negative_prompt);
|
||||
AdapterSupport::put_seed(f, "seed", r.seed > 2147483647L ? 2147483647L : r.seed);
|
||||
if (r.count > 1) f.set("num_images", Json(r.count));
|
||||
AdapterSupport::put(f, "aspect_ratio", r.aspect_ratio);
|
||||
if (r.width && r.height) f.set("resolution", Json(std::to_string(r.width) + "x" + std::to_string(r.height)));
|
||||
if (!r.source_url.empty())
|
||||
throw llm::ApiError::invalid("ideogram", "source images are files: use IdeogramClient::generate with a FilePart and an edit row");
|
||||
AdapterSupport::merge_extra(f, r.extra);
|
||||
return f;
|
||||
}
|
||||
|
||||
Json IdeogramImageGenerator::plan(const ImageRequest& request) const {
|
||||
std::string key;
|
||||
Json f = fields(request, key);
|
||||
return client_->plan(key, f, {}).summary();
|
||||
}
|
||||
|
||||
JobTicket IdeogramImageGenerator::submit(const ImageRequest& request, bool confirm) {
|
||||
std::string key;
|
||||
Json f = fields(request, key);
|
||||
AdapterSupport::need_confirm(confirm, "ideogram", "generate");
|
||||
Json out = client_->generate(key, f, {}, true);
|
||||
JobTicket t;
|
||||
t.provider = "ideogram";
|
||||
t.model = request.model.empty() ? "ideogram-3" : request.model;
|
||||
t.kind = MediaKind::kImage;
|
||||
t.id = AdapterSupport::str(out, "generation_id");
|
||||
if (is_async(f)) {
|
||||
if (t.id.empty()) throw llm::ApiError("ideogram", 502, "BAD_RESPONSE", "no generation_id in the async reply");
|
||||
return t;
|
||||
}
|
||||
if (t.id.empty()) t.id = "immediate";
|
||||
t.immediate = true;
|
||||
if (const Json* d = out.find("data")) t.assets = OutputScanner::scan(*d, MediaKind::kImage);
|
||||
if (t.assets.empty()) throw llm::ApiError("ideogram", 502, "BAD_RESPONSE", "no image URL in the reply (safety check?)");
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus IdeogramImageGenerator::poll(const JobTicket& ticket) {
|
||||
if (ticket.immediate) {
|
||||
JobStatus s;
|
||||
s.state = JobState::kSucceeded;
|
||||
s.raw_state = "immediate";
|
||||
s.assets = ticket.assets;
|
||||
return s;
|
||||
}
|
||||
return status_from(client_->generation(ticket.id), MediaKind::kImage);
|
||||
}
|
||||
|
||||
Json IdeogramVideoGenerator::fields(const VideoRequest& r, std::string& key) const {
|
||||
r.validate();
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("ideogram", "model is required (for example kling-3-standard-text-to-video)");
|
||||
key = "video.generate." + r.model;
|
||||
Json f = Json::object();
|
||||
AdapterSupport::put(f, "prompt", r.prompt);
|
||||
AdapterSupport::put(f, "negative_prompt", r.negative_prompt);
|
||||
AdapterSupport::put(f, "aspect_ratio", r.aspect_ratio);
|
||||
AdapterSupport::put(f, "resolution", r.resolution);
|
||||
AdapterSupport::put_int(f, "duration", r.duration_seconds);
|
||||
if (!r.image_url.empty())
|
||||
throw llm::ApiError::invalid("ideogram", "start frames are files: use IdeogramClient::generate with an image-to-video row");
|
||||
AdapterSupport::merge_extra(f, r.extra);
|
||||
return f;
|
||||
}
|
||||
|
||||
Json IdeogramVideoGenerator::plan(const VideoRequest& request) const {
|
||||
std::string key;
|
||||
Json f = fields(request, key);
|
||||
return client_->plan(key, f, {}).summary();
|
||||
}
|
||||
|
||||
JobTicket IdeogramVideoGenerator::submit(const VideoRequest& request, bool confirm) {
|
||||
std::string key;
|
||||
Json f = fields(request, key);
|
||||
AdapterSupport::need_confirm(confirm, "ideogram", "generate");
|
||||
Json out = client_->generate(key, f, {}, true);
|
||||
JobTicket t;
|
||||
t.provider = "ideogram";
|
||||
t.model = request.model;
|
||||
t.kind = MediaKind::kVideo;
|
||||
t.id = AdapterSupport::str(out, "generation_id");
|
||||
if (t.id.empty()) throw llm::ApiError("ideogram", 502, "BAD_RESPONSE", "no generation_id in the reply");
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus IdeogramVideoGenerator::poll(const JobTicket& ticket) {
|
||||
return status_from(client_->generation(ticket.id), MediaKind::kVideo);
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
#pragma once
|
||||
|
||||
// Ideogram API v2: image generate / remix / inpaint / reframe / upscale /
|
||||
// background tools, SVG, video models, ad tools, generation polling, account.
|
||||
// Auth is the Api-Key header. Every POST accepts dry_run=true, which the spec
|
||||
// defines as "validated and priced but not run: nothing generated, stored or
|
||||
// billed": quote() uses it, so a price check is free.
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/ideogram/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class IdeogramClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api.ideogram.ai";
|
||||
static constexpr const char* kKeyEnv = "IDEOGRAM_API_KEY";
|
||||
|
||||
explicit IdeogramClient(std::shared_ptr<llm::ApiClient> client)
|
||||
: MediaService(std::move(client), IdeogramCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::header("ideogram", kHost, kKeyEnv, "Api-Key"); }
|
||||
static std::shared_ptr<IdeogramClient> from_env() {
|
||||
return std::make_shared<IdeogramClient>(ClientFactory::real(config(), ClientFactory::env({kKeyEnv})));
|
||||
}
|
||||
static std::shared_ptr<IdeogramClient> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<IdeogramClient>(ClientFactory::with(std::move(t), config(), key));
|
||||
}
|
||||
|
||||
// Paid: any POST row of the table (image.*, video.*, svg.*, tool.*).
|
||||
Json generate(const std::string& key, const Json& fields, const std::vector<FilePart>& files, bool confirm);
|
||||
PreparedCall plan(const std::string& key, const Json& fields, const std::vector<FilePart>& files,
|
||||
bool dry_run = false) const;
|
||||
// Free price quote (dry_run=true): the only non-read call that needs no confirm.
|
||||
Json quote(const std::string& key, const Json& fields, const std::vector<FilePart>& files);
|
||||
Json generation(const std::string& generation_id);
|
||||
Json usage(const std::map<std::string, std::string>& query);
|
||||
Json api_keys();
|
||||
Json invoices();
|
||||
};
|
||||
|
||||
class IdeogramImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit IdeogramImageGenerator(std::shared_ptr<IdeogramClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "ideogram"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"ideogram-4-5", "ideogram-4", "ideogram-3", "ideogram-3-transparent", "ideogram-2a", "ideogram-2", "auto"};
|
||||
}
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket&, bool) override { return false; } // no cancel in the API
|
||||
int suggested_poll_ms() const override { return 2000; }
|
||||
|
||||
static std::string key_for(const std::string& model);
|
||||
|
||||
private:
|
||||
Json fields(const ImageRequest& request, std::string& key) const;
|
||||
std::shared_ptr<IdeogramClient> client_;
|
||||
};
|
||||
|
||||
class IdeogramVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit IdeogramVideoGenerator(std::shared_ptr<IdeogramClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "ideogram"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"kling-3-standard-text-to-video", "minimax-h3-text-to-video", "seedance-2-text-to-video",
|
||||
"seedance-2-5-text-to-video"};
|
||||
}
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket&, bool) override { return false; }
|
||||
int suggested_poll_ms() const override { return 4000; }
|
||||
|
||||
private:
|
||||
Json fields(const VideoRequest& request, std::string& key) const;
|
||||
std::shared_ptr<IdeogramClient> client_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+30
@@ -0,0 +1,30 @@
|
||||
#include "vendors/media/kling/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"text2video.create", "POST", "/v1/videos/text2video", "", "json", "model_name,prompt,negative_prompt,cfg_scale,mode,aspect_ratio,duration,callback_url,external_task_id", "", "paid"},
|
||||
{"text2video.get", "GET", "/v1/videos/text2video/{id}", "", "none", "", "", "read"},
|
||||
{"image2video.create", "POST", "/v1/videos/image2video", "", "json", "model_name,image,image_tail,prompt,negative_prompt,cfg_scale,mode,duration,callback_url,external_task_id", "", "paid"},
|
||||
{"image2video.get", "GET", "/v1/videos/image2video/{id}", "", "none", "", "", "read"},
|
||||
{"images.create", "POST", "/v1/images/generations", "", "json", "model_name,prompt,negative_prompt,image,image_reference,image_fidelity,n,aspect_ratio,callback_url", "", "paid"},
|
||||
{"images.get", "GET", "/v1/images/generations/{id}", "", "none", "", "", "read"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& KlingCalls::table() {
|
||||
static const CallTable t("kling", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t KlingCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* KlingCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct KlingCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+156
@@ -0,0 +1,156 @@
|
||||
#include "vendors/media/kling/kling.h"
|
||||
|
||||
#include <ctime>
|
||||
#include <functional>
|
||||
|
||||
#include "security/crypto/Base64.h"
|
||||
#include "security/crypto/Hmac.h"
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
std::shared_ptr<llm::ApiClient> build_client(const std::string& access, const std::string& secret, long now,
|
||||
const std::function<std::unique_ptr<clients::Transport>()>& factory) {
|
||||
std::string token = KlingClient::credentials_present(access, secret) ? KlingJwt::make(access, secret, now) : "";
|
||||
if (factory) return ClientFactory::with(factory(), KlingClient::config(), token);
|
||||
return ClientFactory::real(KlingClient::config(), token);
|
||||
}
|
||||
|
||||
CallArgs with_body(const Json& body) {
|
||||
CallArgs a;
|
||||
a.body = body;
|
||||
return a;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string KlingJwt::make(const std::string& access_key, const std::string& secret_key, long now) {
|
||||
Json header = Json::object();
|
||||
header.set("alg", Json("HS256"));
|
||||
header.set("typ", Json("JWT"));
|
||||
Json payload = Json::object();
|
||||
payload.set("iss", Json(access_key));
|
||||
payload.set("exp", Json(static_cast<double>(now + kTtlSeconds)));
|
||||
payload.set("nbf", Json(static_cast<double>(now - 5)));
|
||||
std::string signing = security::crypto::Base64::encode_url(header.dump()) + "." +
|
||||
security::crypto::Base64::encode_url(payload.dump());
|
||||
auto mac = security::crypto::Hmac::sha256(secret_key, signing);
|
||||
return signing + "." + security::crypto::Base64::encode_url(reinterpret_cast<const std::uint8_t*>(mac.data()), mac.size());
|
||||
}
|
||||
|
||||
KlingClient::KlingClient(std::string access_key, std::string secret_key,
|
||||
std::function<std::unique_ptr<clients::Transport>()> transport)
|
||||
: MediaService(build_client(access_key, secret_key, static_cast<long>(std::time(nullptr)), transport),
|
||||
KlingCalls::table()),
|
||||
access_(std::move(access_key)),
|
||||
secret_(std::move(secret_key)),
|
||||
transport_(std::move(transport)),
|
||||
minted_at_(static_cast<long>(std::time(nullptr))) {}
|
||||
|
||||
std::shared_ptr<KlingClient> KlingClient::from_env() {
|
||||
return std::make_shared<KlingClient>(ClientFactory::env({kAccessEnv}), ClientFactory::env({kSecretEnv}));
|
||||
}
|
||||
|
||||
void KlingClient::renew() {
|
||||
long now = static_cast<long>(std::time(nullptr));
|
||||
if (!credentials_present(access_, secret_) || now - minted_at_ < KlingJwt::kRenewAfterSeconds) return;
|
||||
client_ = build_client(access_, secret_, now, transport_);
|
||||
minted_at_ = now;
|
||||
}
|
||||
|
||||
void KlingClient::check_envelope(const Json& doc) {
|
||||
const Json* code = doc.is_object() ? doc.find("code") : nullptr;
|
||||
if (!code || !code->is_number() || code->as_int() == 0) return;
|
||||
throw llm::ApiError("kling", 400, std::to_string(code->as_int()), AdapterSupport::str(doc, "message"),
|
||||
AdapterSupport::str(doc, "request_id"));
|
||||
}
|
||||
|
||||
PreparedCall KlingClient::plan(const std::string& key, const Json& body) const { return prepare(key, with_body(body)); }
|
||||
|
||||
Json KlingClient::submit(const std::string& key, const Json& body, bool confirm) {
|
||||
renew();
|
||||
Json out = call_json(key, with_body(body), confirm);
|
||||
check_envelope(out);
|
||||
return out;
|
||||
}
|
||||
|
||||
Json KlingClient::get(const std::string& key, const std::string& task_id) {
|
||||
renew();
|
||||
CallArgs a;
|
||||
a.path["id"] = task_id;
|
||||
Json out = call_json(key, a);
|
||||
check_envelope(out);
|
||||
return out;
|
||||
}
|
||||
|
||||
JobState KlingVideoGenerator::state_of(const std::string& status) {
|
||||
if (status == "succeed") return JobState::kSucceeded;
|
||||
if (status == "failed") return JobState::kFailed;
|
||||
if (status == "processing") return JobState::kRunning;
|
||||
return JobState::kQueued; // submitted
|
||||
}
|
||||
|
||||
Json KlingVideoGenerator::body(const VideoRequest& r) const {
|
||||
r.validate();
|
||||
Json b = Json::object();
|
||||
b.set("model_name", Json(r.model.empty() ? "kling-v1-6" : r.model));
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
AdapterSupport::put(b, "negative_prompt", r.negative_prompt);
|
||||
AdapterSupport::put(b, "aspect_ratio", r.aspect_ratio);
|
||||
if (r.duration_seconds > 0) b.set("duration", Json(std::to_string(r.duration_seconds)));
|
||||
AdapterSupport::put(b, "image", r.image_url);
|
||||
AdapterSupport::put(b, "image_tail", r.end_image_url);
|
||||
if (r.seed >= 0) throw llm::ApiError::invalid("kling", "no seed parameter is documented");
|
||||
if (!r.resolution.empty()) throw llm::ApiError::invalid("kling", "resolution is set through mode (std / pro): pass mode in extra");
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json KlingVideoGenerator::plan(const VideoRequest& request) const {
|
||||
return client_->plan(request.image_url.empty() ? "text2video.create" : "image2video.create", body(request)).summary();
|
||||
}
|
||||
|
||||
JobTicket KlingVideoGenerator::submit(const VideoRequest& request, bool confirm) {
|
||||
Json b = body(request);
|
||||
AdapterSupport::need_confirm(confirm, "kling", "generate");
|
||||
const bool image = !request.image_url.empty();
|
||||
Json out = client_->submit(image ? "image2video.create" : "text2video.create", b, true);
|
||||
const Json* data = out.find("data");
|
||||
JobTicket t;
|
||||
t.provider = "kling";
|
||||
t.model = b.find("model_name")->as_string();
|
||||
t.kind = MediaKind::kVideo;
|
||||
t.id = data ? AdapterSupport::str(*data, "task_id") : "";
|
||||
if (t.id.empty()) throw llm::ApiError("kling", 502, "BAD_RESPONSE", "no data.task_id in the reply");
|
||||
t.meta = Json::object();
|
||||
t.meta.set("route", Json(image ? "image2video" : "text2video"));
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus KlingVideoGenerator::poll(const JobTicket& ticket) {
|
||||
std::string route = AdapterSupport::str(ticket.meta, "route");
|
||||
if (route != "image2video" && route != "text2video") throw llm::ApiError::invalid("kling", "ticket has no route");
|
||||
Json out = client_->get(route + ".get", ticket.id);
|
||||
const Json* data = out.find("data");
|
||||
JobStatus s;
|
||||
if (!data) {
|
||||
s.state = JobState::kQueued;
|
||||
return s;
|
||||
}
|
||||
s.raw_state = AdapterSupport::str(*data, "task_status");
|
||||
s.state = state_of(s.raw_state);
|
||||
if (s.state == JobState::kFailed) s.error = AdapterSupport::str(*data, "task_status_msg");
|
||||
if (s.state == JobState::kSucceeded) {
|
||||
if (const Json* r = data->find("task_result")) s.assets = OutputScanner::scan(*r, MediaKind::kVideo);
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "succeed without videos";
|
||||
}
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+80
@@ -0,0 +1,80 @@
|
||||
#pragma once
|
||||
|
||||
// Kling AI (Kuaishou) API. Kling publishes its developer docs as a JavaScript
|
||||
// app with no machine-readable spec, so this adapter is built from community
|
||||
// sources and live probes only: see SPEC "Unverified". Auth is a short-lived
|
||||
// HS256 JWT built from KLING_ACCESS_KEY (iss) and KLING_SECRET_KEY. Hosts
|
||||
// api-singapore.klingai.com and api.klingai.com both answered the documented
|
||||
// path with `{"code":1001,"message":"Authorization is empty"}` unauthenticated.
|
||||
|
||||
#include <chrono>
|
||||
#include <functional>
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/kling/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class KlingJwt {
|
||||
public:
|
||||
static constexpr long kTtlSeconds = 1800; // token lifetime
|
||||
static constexpr long kRenewAfterSeconds = 1500; // the client rebuilds before this
|
||||
// header {"alg":"HS256","typ":"JWT"}, payload {iss, exp, nbf}; base64url, no padding.
|
||||
static std::string make(const std::string& access_key, const std::string& secret_key, long now_seconds);
|
||||
};
|
||||
|
||||
class KlingClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api-singapore.klingai.com";
|
||||
static constexpr const char* kAccessEnv = "KLING_ACCESS_KEY";
|
||||
static constexpr const char* kSecretEnv = "KLING_SECRET_KEY";
|
||||
|
||||
// The token is fixed per ApiClient, so a KlingClient holds a factory and
|
||||
// renews the ApiClient when the token nears expiry.
|
||||
KlingClient(std::string access_key, std::string secret_key,
|
||||
std::function<std::unique_ptr<clients::Transport>()> transport = nullptr);
|
||||
static llm::ApiClientConfig config() { return ClientFactory::bearer("kling", kHost, kAccessEnv); }
|
||||
static std::shared_ptr<KlingClient> from_env();
|
||||
static bool credentials_present(const std::string& access, const std::string& secret) {
|
||||
return !access.empty() && !secret.empty();
|
||||
}
|
||||
|
||||
// Paid submits: text2video.create, image2video.create, images.create.
|
||||
Json submit(const std::string& key, const Json& body, bool confirm);
|
||||
PreparedCall plan(const std::string& key, const Json& body) const;
|
||||
Json get(const std::string& key, const std::string& task_id); // *.get rows
|
||||
// Throws ApiError when the envelope's code is not 0.
|
||||
static void check_envelope(const Json& doc);
|
||||
|
||||
private:
|
||||
void renew();
|
||||
std::string access_, secret_;
|
||||
std::function<std::unique_ptr<clients::Transport>()> transport_;
|
||||
long minted_at_ = 0;
|
||||
};
|
||||
|
||||
class KlingVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit KlingVideoGenerator(std::shared_ptr<KlingClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "kling"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"kling-v1", "kling-v1-6", "kling-v2-master", "kling-v2-1", "kling-v2-5-turbo"};
|
||||
}
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket&, bool) override { return false; } // not documented
|
||||
int suggested_poll_ms() const override { return 6000; }
|
||||
|
||||
// submitted / processing / succeed / failed (community sources).
|
||||
static JobState state_of(const std::string& status);
|
||||
|
||||
private:
|
||||
Json body(const VideoRequest& request) const;
|
||||
std::shared_ptr<KlingClient> client_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+30
@@ -0,0 +1,30 @@
|
||||
#include "vendors/media/leonardo/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"v2.generations.create", "POST", "/v2/generations", "createGeneration", "json", "model,parameters,public", "", "paid"},
|
||||
{"v2.generations.sync", "POST", "/v2/generationssync", "createSyncGeneration", "json", "model,parameters,public", "", "paid"},
|
||||
{"v2.models", "GET", "/v2/models", "getModels", "none", "", "", "read"},
|
||||
{"v1.generations.create", "POST", "/v1/generations", "", "json", "prompt,modelId,width,height,num_images,alchemy,ultra,contrast,styleUUID,negative_prompt,seed,public", "", "paid"},
|
||||
{"v1.generations.get", "GET", "/v1/generations/{id}", "getGenerationById", "none", "", "", "read"},
|
||||
{"v1.me", "GET", "/v1/me", "getUserSelf", "none", "", "", "read"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& LeonardoCalls::table() {
|
||||
static const CallTable t("leonardo", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t LeonardoCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* LeonardoCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct LeonardoCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
#include "vendors/media/leonardo/leonardo.h"
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs with_body(const Json& body) {
|
||||
CallArgs a;
|
||||
a.body = body;
|
||||
return a;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
PreparedCall LeonardoClient::plan(const Json& body) const { return prepare("v2.generations.create", with_body(body)); }
|
||||
Json LeonardoClient::create(const Json& body, bool confirm) {
|
||||
return call_json("v2.generations.create", with_body(body), confirm);
|
||||
}
|
||||
Json LeonardoClient::create_sync(const Json& body, bool confirm) {
|
||||
return call_json("v2.generations.sync", with_body(body), confirm);
|
||||
}
|
||||
Json LeonardoClient::models() { return call_json("v2.models"); }
|
||||
Json LeonardoClient::generation(const std::string& id) {
|
||||
CallArgs a;
|
||||
a.path["id"] = id;
|
||||
return call_json("v1.generations.get", a);
|
||||
}
|
||||
Json LeonardoClient::me() { return call_json("v1.me"); }
|
||||
|
||||
Json LeonardoJobs::body(const std::string& model, const Json& parameters, bool is_public) {
|
||||
Json b = Json::object();
|
||||
b.set("model", Json(model));
|
||||
b.set("parameters", parameters);
|
||||
b.set("public", Json(is_public)); // never publish to the community feed by default
|
||||
return b;
|
||||
}
|
||||
|
||||
JobTicket LeonardoJobs::ticket_from(const Json& reply, const std::string& model, MediaKind kind) {
|
||||
JobTicket t;
|
||||
t.provider = "leonardo";
|
||||
t.model = model;
|
||||
t.kind = kind;
|
||||
t.id = AdapterSupport::str(reply, "generationId");
|
||||
if (t.id.empty()) throw llm::ApiError("leonardo", 502, "BAD_RESPONSE", "no generationId in the reply");
|
||||
t.meta = Json::object();
|
||||
AdapterSupport::put(t.meta, "apiCreditCost", AdapterSupport::str(reply, "apiCreditCost"));
|
||||
return t;
|
||||
}
|
||||
|
||||
// v1 GET /generations/{id}: {"generations_by_pk": {"status": "PENDING|COMPLETE|FAILED",
|
||||
// "generated_images": [{"url": ...}], ...}} (status words from the FAQ; shape unverified).
|
||||
JobStatus LeonardoJobs::status_of(const Json& doc, MediaKind kind) {
|
||||
const Json* g = doc.find("generations_by_pk");
|
||||
const Json& gen = g && g->is_object() ? *g : doc;
|
||||
JobStatus s;
|
||||
s.raw_state = AdapterSupport::str(gen, "status");
|
||||
if (s.raw_state == "COMPLETE") {
|
||||
s.state = JobState::kSucceeded;
|
||||
for (const char* key : {"generated_images", "generated_videos", "generated_assets"})
|
||||
if (const Json* a = gen.find(key)) {
|
||||
auto found = OutputScanner::scan(*a, kind);
|
||||
s.assets.insert(s.assets.end(), found.begin(), found.end());
|
||||
}
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "COMPLETE without generated images";
|
||||
}
|
||||
} else if (s.raw_state == "FAILED") {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "FAILED";
|
||||
} else {
|
||||
s.state = JobState::kRunning; // PENDING
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
JobStatus LeonardoJobs::poll(LeonardoClient& client, const JobTicket& ticket) {
|
||||
return status_of(client.generation(ticket.id), ticket.kind);
|
||||
}
|
||||
|
||||
Json LeonardoImageGenerator::body(const ImageRequest& r) const {
|
||||
r.validate();
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("leonardo", "model is required (see LeonardoClient::models)");
|
||||
Json p = Json::object();
|
||||
AdapterSupport::put(p, "prompt", r.prompt);
|
||||
AdapterSupport::put_int(p, "width", r.width);
|
||||
AdapterSupport::put_int(p, "height", r.height);
|
||||
p.set("quantity", Json(r.count));
|
||||
AdapterSupport::put_seed(p, "seed", r.seed);
|
||||
if (!r.aspect_ratio.empty() || !r.negative_prompt.empty() || !r.source_url.empty())
|
||||
throw llm::ApiError::invalid("leonardo", "aspect_ratio / negative_prompt / source images are model-specific: pass them in extra (parameters)");
|
||||
AdapterSupport::merge_extra(p, r.extra);
|
||||
return LeonardoJobs::body(r.model, p, false);
|
||||
}
|
||||
|
||||
Json LeonardoImageGenerator::plan(const ImageRequest& request) const { return client_->plan(body(request)).summary(); }
|
||||
|
||||
JobTicket LeonardoImageGenerator::submit(const ImageRequest& request, bool confirm) {
|
||||
Json b = body(request);
|
||||
AdapterSupport::need_confirm(confirm, "leonardo", "generate");
|
||||
return LeonardoJobs::ticket_from(client_->create(b, true), request.model, MediaKind::kImage);
|
||||
}
|
||||
|
||||
Json LeonardoVideoGenerator::body(const VideoRequest& r) const {
|
||||
r.validate();
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("leonardo", "model is required (see LeonardoClient::models)");
|
||||
Json p = Json::object();
|
||||
AdapterSupport::put(p, "prompt", r.prompt);
|
||||
AdapterSupport::put_int(p, "duration", r.duration_seconds);
|
||||
AdapterSupport::put(p, "resolution", r.resolution);
|
||||
AdapterSupport::put_seed(p, "seed", r.seed);
|
||||
if (!r.aspect_ratio.empty() || !r.negative_prompt.empty() || !r.image_url.empty() || !r.end_image_url.empty())
|
||||
throw llm::ApiError::invalid("leonardo", "frames / aspect_ratio / negative_prompt are model-specific: pass them in extra (parameters)");
|
||||
AdapterSupport::merge_extra(p, r.extra);
|
||||
return LeonardoJobs::body(r.model, p, false);
|
||||
}
|
||||
|
||||
Json LeonardoVideoGenerator::plan(const VideoRequest& request) const { return client_->plan(body(request)).summary(); }
|
||||
|
||||
JobTicket LeonardoVideoGenerator::submit(const VideoRequest& request, bool confirm) {
|
||||
Json b = body(request);
|
||||
AdapterSupport::need_confirm(confirm, "leonardo", "generate");
|
||||
return LeonardoJobs::ticket_from(client_->create(b, true), request.model, MediaKind::kVideo);
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
#pragma once
|
||||
|
||||
// Leonardo.Ai production API. v2 (https://cloud.leonardo.ai/api/rest/v2):
|
||||
// POST /generations (async, any image / video / audio / 3D model by id),
|
||||
// POST /generationssync, GET /models. Polling uses the documented v1
|
||||
// GET /generations/{id} (the v1 reference pages are gone from the docs; the
|
||||
// shape is described in prose only, see SPEC Unverified). Bearer auth with
|
||||
// LEONARDO_API_KEY.
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/leonardo/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class LeonardoClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://cloud.leonardo.ai/api/rest";
|
||||
static constexpr const char* kKeyEnv = "LEONARDO_API_KEY";
|
||||
|
||||
explicit LeonardoClient(std::shared_ptr<llm::ApiClient> client)
|
||||
: MediaService(std::move(client), LeonardoCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::bearer("leonardo", kHost, kKeyEnv); }
|
||||
static std::shared_ptr<LeonardoClient> from_env() {
|
||||
return std::make_shared<LeonardoClient>(ClientFactory::real(config(), ClientFactory::env({kKeyEnv})));
|
||||
}
|
||||
static std::shared_ptr<LeonardoClient> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<LeonardoClient>(ClientFactory::with(std::move(t), config(), key));
|
||||
}
|
||||
|
||||
// body = {"model": "<id>", "parameters": {...}, "public": false}
|
||||
Json create(const Json& body, bool confirm);
|
||||
PreparedCall plan(const Json& body) const;
|
||||
Json create_sync(const Json& body, bool confirm); // up to ~27 s, finished result in the reply
|
||||
Json models(); // v2, parameter specs per model
|
||||
Json generation(const std::string& id); // v1
|
||||
Json me(); // v1, account and token balance
|
||||
};
|
||||
|
||||
// Shared by the image and video adapters (the API does not tell them apart).
|
||||
class LeonardoJobs {
|
||||
public:
|
||||
static Json body(const std::string& model, const Json& parameters, bool is_public);
|
||||
static JobTicket ticket_from(const Json& reply, const std::string& model, MediaKind kind);
|
||||
static JobStatus poll(LeonardoClient& client, const JobTicket& ticket);
|
||||
static JobStatus status_of(const Json& generation, MediaKind kind);
|
||||
};
|
||||
|
||||
class LeonardoImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit LeonardoImageGenerator(std::shared_ptr<LeonardoClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "leonardo"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"flux-pro-2.0", "flux-kontext-pro", "phoenix-v1.0", "lucid-origin", "ideogram-v3.0", "gpt-image-2",
|
||||
"nano-banana-2", "seedream-4.5"};
|
||||
}
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override { return LeonardoJobs::poll(*client_, ticket); }
|
||||
bool cancel(const JobTicket&, bool) override { return false; } // no cancel in the v2 spec
|
||||
int suggested_poll_ms() const override { return 3000; }
|
||||
|
||||
private:
|
||||
Json body(const ImageRequest& request) const;
|
||||
std::shared_ptr<LeonardoClient> client_;
|
||||
};
|
||||
|
||||
class LeonardoVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit LeonardoVideoGenerator(std::shared_ptr<LeonardoClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "leonardo"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"motion_2.0", "veo-3.1-fast-generate-001", "kling-3.0", "hailuo-03", "seedance-2.0", "wan-2.7"};
|
||||
}
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override { return LeonardoJobs::poll(*client_, ticket); }
|
||||
bool cancel(const JobTicket&, bool) override { return false; }
|
||||
int suggested_poll_ms() const override { return 5000; }
|
||||
|
||||
private:
|
||||
Json body(const VideoRequest& request) const;
|
||||
std::shared_ptr<LeonardoClient> client_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+39
@@ -0,0 +1,39 @@
|
||||
#include "vendors/media/luma/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"generations.create_video", "POST", "/generations/video", "", "json", "model,aspect_ratio,callback_url,concepts,duration,generation_type,keyframes,loop,prompt,resolution", "", "paid"},
|
||||
{"generations.create_image", "POST", "/generations/image", "", "json", "model,aspect_ratio,callback_url,character_ref,format,generation_type,image_ref,modify_image_ref,prompt,style_ref,sync,sync_timeout", "", "paid"},
|
||||
{"generations.reframe_image", "POST", "/generations/image/reframe", "", "json", "aspect_ratio,generation_type,media,model,callback_url,format,grid_position_x,grid_position_y,prompt,resized_height,resized_width,x_end,x_start,y_end,y_start", "", "paid"},
|
||||
{"generations.reframe_video", "POST", "/generations/video/reframe", "", "json", "aspect_ratio,generation_type,media,model,callback_url,first_frame,grid_position_x,grid_position_y,prompt,resized_height,resized_width,x_end,x_start,y_end,y_start", "", "paid"},
|
||||
{"generations.modify_video", "POST", "/generations/video/modify", "", "json", "generation_type,media,mode,model,callback_url,first_frame,prompt", "", "paid"},
|
||||
{"generations.upscale", "POST", "/generations/{id}/upscale", "", "json", "callback_url,generation_type,resolution", "", "paid"},
|
||||
{"generations.audio", "POST", "/generations/{id}/audio", "", "json", "callback_url,generation_type,negative_prompt,prompt", "", "paid"},
|
||||
{"generations.get", "GET", "/generations/{id}", "", "none", "", "", "read"},
|
||||
{"generations.list", "GET", "/generations", "", "none", "", "limit,offset", "read"},
|
||||
{"generations.delete", "DELETE", "/generations/{id}", "", "none", "", "", "delete"},
|
||||
{"concepts.list", "GET", "/generations/concepts/list", "", "none", "", "", "read"},
|
||||
{"credits.get", "GET", "/credits", "", "none", "", "", "read"},
|
||||
{"ping", "GET", "/ping", "", "none", "", "", "read"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
{"Generation", "id,state,failure_reason,created_at,assets,generation_type,model,request,image,video,progress_video"},
|
||||
{"CreditGetResponse", "credit_balance"},
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& LumaCalls::table() {
|
||||
static const CallTable t("luma", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t LumaCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* LumaCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct LumaCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+174
@@ -0,0 +1,174 @@
|
||||
#include "vendors/media/luma/luma.h"
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs with_body(const Json& body) {
|
||||
CallArgs a;
|
||||
a.body = body;
|
||||
return a;
|
||||
}
|
||||
|
||||
CallArgs by_id(const std::string& id, const Json& body = Json()) {
|
||||
CallArgs a;
|
||||
a.path["id"] = id;
|
||||
a.body = body;
|
||||
return a;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
PreparedCall LumaClient::plan(const std::string& key, const Json& body) const { return prepare(key, with_body(body)); }
|
||||
Json LumaClient::create(const std::string& key, const Json& body, bool confirm) {
|
||||
return call_json(key, with_body(body), confirm);
|
||||
}
|
||||
Json LumaClient::upscale(const std::string& id, const Json& body, bool confirm) {
|
||||
return call_json("generations.upscale", by_id(id, body), confirm);
|
||||
}
|
||||
Json LumaClient::add_audio(const std::string& id, const Json& body, bool confirm) {
|
||||
return call_json("generations.audio", by_id(id, body), confirm);
|
||||
}
|
||||
Json LumaClient::generation(const std::string& id) { return call_json("generations.get", by_id(id)); }
|
||||
|
||||
Json LumaClient::list(int limit, int offset) {
|
||||
CallArgs a;
|
||||
if (limit > 0) a.query["limit"] = std::to_string(limit);
|
||||
if (offset > 0) a.query["offset"] = std::to_string(offset);
|
||||
return call_json("generations.list", a);
|
||||
}
|
||||
|
||||
Json LumaClient::remove(const std::string& id, bool confirm) { return call_json("generations.delete", by_id(id), confirm); }
|
||||
Json LumaClient::credits() { return call_json("credits.get"); }
|
||||
Json LumaClient::concepts() { return call_json("concepts.list"); }
|
||||
Json LumaClient::ping() { return call_json("ping"); }
|
||||
|
||||
JobTicket LumaJobs::ticket_from(const Json& g, const std::string& model, MediaKind kind) {
|
||||
JobTicket t;
|
||||
t.provider = "luma";
|
||||
t.model = model;
|
||||
t.kind = kind;
|
||||
t.id = AdapterSupport::str(g, "id");
|
||||
if (t.id.empty()) throw llm::ApiError("luma", 502, "BAD_RESPONSE", "no id in the generation");
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus LumaJobs::status_of(const Json& g, MediaKind kind) {
|
||||
JobStatus s;
|
||||
s.raw_state = AdapterSupport::str(g, "state");
|
||||
if (s.raw_state == "completed") {
|
||||
s.state = JobState::kSucceeded;
|
||||
if (const Json* a = g.find("assets")) s.assets = OutputScanner::scan(*a, kind);
|
||||
// progress_video is a preview, not the result
|
||||
std::vector<MediaAsset> keep;
|
||||
for (const MediaAsset& x : s.assets)
|
||||
if (x.url.find("progress") == std::string::npos) keep.push_back(x);
|
||||
if (!keep.empty()) s.assets = keep;
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "completed without assets";
|
||||
}
|
||||
} else if (s.raw_state == "failed") {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = AdapterSupport::str(g, "failure_reason");
|
||||
} else if (s.raw_state == "dreaming") {
|
||||
s.state = JobState::kRunning;
|
||||
} else {
|
||||
s.state = JobState::kQueued;
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
JobStatus LumaJobs::poll(LumaClient& client, const JobTicket& ticket) {
|
||||
return status_of(client.generation(ticket.id), ticket.kind);
|
||||
}
|
||||
|
||||
Json LumaVideoGenerator::body(const VideoRequest& r) const {
|
||||
r.validate();
|
||||
Json b = Json::object();
|
||||
b.set("model", Json(r.model.empty() ? "ray-2" : r.model));
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
AdapterSupport::put(b, "aspect_ratio", r.aspect_ratio);
|
||||
AdapterSupport::put(b, "resolution", r.resolution);
|
||||
if (r.duration_seconds > 0) b.set("duration", Json(std::to_string(r.duration_seconds) + "s"));
|
||||
if (!r.image_url.empty() || !r.end_image_url.empty()) {
|
||||
Json kf = Json::object();
|
||||
if (!r.image_url.empty()) {
|
||||
Json f0 = Json::object();
|
||||
f0.set("type", Json("image"));
|
||||
f0.set("url", Json(r.image_url));
|
||||
kf.set("frame0", f0);
|
||||
}
|
||||
if (!r.end_image_url.empty()) {
|
||||
Json f1 = Json::object();
|
||||
f1.set("type", Json("image"));
|
||||
f1.set("url", Json(r.end_image_url));
|
||||
kf.set("frame1", f1);
|
||||
}
|
||||
b.set("keyframes", kf);
|
||||
}
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json LumaVideoGenerator::plan(const VideoRequest& request) const {
|
||||
return client_->plan("generations.create_video", body(request)).summary();
|
||||
}
|
||||
|
||||
JobTicket LumaVideoGenerator::submit(const VideoRequest& request, bool confirm) {
|
||||
Json b = body(request);
|
||||
AdapterSupport::need_confirm(confirm, "luma", "generate");
|
||||
return LumaJobs::ticket_from(client_->create("generations.create_video", b, true),
|
||||
b.find("model")->as_string(), MediaKind::kVideo);
|
||||
}
|
||||
|
||||
JobStatus LumaVideoGenerator::poll(const JobTicket& ticket) { return LumaJobs::poll(*client_, ticket); }
|
||||
|
||||
bool LumaVideoGenerator::cancel(const JobTicket& ticket, bool confirm) {
|
||||
AdapterSupport::need_confirm(confirm, "luma", "delete generation");
|
||||
client_->remove(ticket.id, true);
|
||||
return true;
|
||||
}
|
||||
|
||||
Json LumaImageGenerator::body(const ImageRequest& r) const {
|
||||
r.validate();
|
||||
if (r.count != 1) throw llm::ApiError::invalid("luma", "one image per call (count must be 1)");
|
||||
if (r.width || r.height) throw llm::ApiError::invalid("luma", "use aspect_ratio, not width / height");
|
||||
Json b = Json::object();
|
||||
b.set("model", Json(r.model.empty() ? "photon-1" : r.model));
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
AdapterSupport::put(b, "aspect_ratio", r.aspect_ratio);
|
||||
if (!r.source_url.empty()) {
|
||||
Json ref = Json::array();
|
||||
Json one = Json::object();
|
||||
one.set("url", Json(r.source_url));
|
||||
ref.push(one);
|
||||
b.set("image_ref", ref);
|
||||
}
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json LumaImageGenerator::plan(const ImageRequest& request) const {
|
||||
return client_->plan("generations.create_image", body(request)).summary();
|
||||
}
|
||||
|
||||
JobTicket LumaImageGenerator::submit(const ImageRequest& request, bool confirm) {
|
||||
Json b = body(request);
|
||||
AdapterSupport::need_confirm(confirm, "luma", "generate");
|
||||
return LumaJobs::ticket_from(client_->create("generations.create_image", b, true),
|
||||
b.find("model")->as_string(), MediaKind::kImage);
|
||||
}
|
||||
|
||||
JobStatus LumaImageGenerator::poll(const JobTicket& ticket) { return LumaJobs::poll(*client_, ticket); }
|
||||
|
||||
bool LumaImageGenerator::cancel(const JobTicket& ticket, bool confirm) {
|
||||
AdapterSupport::need_confirm(confirm, "luma", "delete generation");
|
||||
client_->remove(ticket.id, true);
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+82
@@ -0,0 +1,82 @@
|
||||
#pragma once
|
||||
|
||||
// Luma Dream Machine API (https://api.lumalabs.ai/dream-machine/v1): video
|
||||
// (Ray models), images (Photon), reframe, modify, upscale, audio, concepts,
|
||||
// credits. Bearer auth with LUMA_API_KEY. States: queued, dreaming,
|
||||
// completed, failed (from the official SDK types).
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/luma/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class LumaClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api.lumalabs.ai/dream-machine/v1";
|
||||
static constexpr const char* kKeyEnv = "LUMA_API_KEY";
|
||||
|
||||
explicit LumaClient(std::shared_ptr<llm::ApiClient> client) : MediaService(std::move(client), LumaCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::bearer("luma", kHost, kKeyEnv); }
|
||||
static std::shared_ptr<LumaClient> from_env() {
|
||||
return std::make_shared<LumaClient>(ClientFactory::real(config(), ClientFactory::env({kKeyEnv})));
|
||||
}
|
||||
static std::shared_ptr<LumaClient> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<LumaClient>(ClientFactory::with(std::move(t), config(), key));
|
||||
}
|
||||
|
||||
// key: generations.create_video | create_image | reframe_image | reframe_video | modify_video.
|
||||
Json create(const std::string& key, const Json& body, bool confirm);
|
||||
PreparedCall plan(const std::string& key, const Json& body) const;
|
||||
Json upscale(const std::string& generation_id, const Json& body, bool confirm);
|
||||
Json add_audio(const std::string& generation_id, const Json& body, bool confirm);
|
||||
Json generation(const std::string& id);
|
||||
Json list(int limit, int offset);
|
||||
Json remove(const std::string& id, bool confirm); // destructive, no default
|
||||
Json credits();
|
||||
Json concepts();
|
||||
Json ping();
|
||||
};
|
||||
|
||||
class LumaVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit LumaVideoGenerator(std::shared_ptr<LumaClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "luma"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override { return {"ray-2", "ray-flash-2"}; }
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override; // deletes the generation
|
||||
int suggested_poll_ms() const override { return 4000; }
|
||||
|
||||
private:
|
||||
Json body(const VideoRequest& request) const;
|
||||
std::shared_ptr<LumaClient> client_;
|
||||
};
|
||||
|
||||
class LumaImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit LumaImageGenerator(std::shared_ptr<LumaClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "luma"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override { return {"photon-1", "photon-flash-1"}; }
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override;
|
||||
int suggested_poll_ms() const override { return 2000; }
|
||||
|
||||
private:
|
||||
Json body(const ImageRequest& request) const;
|
||||
std::shared_ptr<LumaClient> client_;
|
||||
};
|
||||
|
||||
struct LumaJobs {
|
||||
static JobTicket ticket_from(const Json& generation, const std::string& model, MediaKind kind);
|
||||
static JobStatus poll(LumaClient& client, const JobTicket& ticket);
|
||||
static JobStatus status_of(const Json& generation, MediaKind kind);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+31
@@ -0,0 +1,31 @@
|
||||
#include "vendors/media/minimax/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"video.create", "POST", "/v1/video_generation", "videoGeneration", "json", "model,prompt,first_frame_image,last_frame_image,subject_reference,prompt_optimizer,fast_pretreatment,duration,resolution,callback_url", "", "paid"},
|
||||
{"video.query", "GET", "/v1/query/video_generation", "", "none", "", "task_id", "read"},
|
||||
{"files.retrieve", "GET", "/v1/files/retrieve", "", "none", "", "file_id", "read"},
|
||||
{"v2.video.create", "POST", "/v2/video_generation", "", "json", "model,content,resolution,duration,ratio,extra,callback_url", "", "paid"},
|
||||
{"v2.video.query", "GET", "/v2/query/video_generation/{task_id}", "", "none", "", "", "read"},
|
||||
{"v2.video.list", "GET", "/v2/query/video_generation", "", "none", "", "page_num,page_size", "read"},
|
||||
{"v2.video.delete", "DELETE", "/v2/video_generation/{task_id}", "", "none", "", "", "delete"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& MinimaxCalls::table() {
|
||||
static const CallTable t("minimax", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t MinimaxCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* MinimaxCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct MinimaxCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+138
@@ -0,0 +1,138 @@
|
||||
#include "vendors/media/minimax/minimax.h"
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs with_body(const Json& body) {
|
||||
CallArgs a;
|
||||
a.body = body;
|
||||
return a;
|
||||
}
|
||||
|
||||
CallArgs task(const std::string& id) {
|
||||
CallArgs a;
|
||||
a.path["task_id"] = id;
|
||||
return a;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void MinimaxClient::check_base_resp(const Json& doc) {
|
||||
const Json* br = doc.is_object() ? doc.find("base_resp") : nullptr;
|
||||
if (!br || !br->is_object()) return;
|
||||
const Json* code = br->find("status_code");
|
||||
if (!code || !code->is_number() || code->as_int() == 0) return;
|
||||
throw llm::ApiError("minimax", 400, std::to_string(code->as_int()), AdapterSupport::str(*br, "status_msg"));
|
||||
}
|
||||
|
||||
PreparedCall MinimaxClient::plan_video(const Json& body) const { return prepare("video.create", with_body(body)); }
|
||||
|
||||
Json MinimaxClient::create_video(const Json& body, bool confirm) {
|
||||
Json out = call_json("video.create", with_body(body), confirm);
|
||||
check_base_resp(out);
|
||||
return out;
|
||||
}
|
||||
|
||||
Json MinimaxClient::query_video(const std::string& task_id) {
|
||||
CallArgs a;
|
||||
a.query["task_id"] = task_id;
|
||||
Json out = call_json("video.query", a);
|
||||
check_base_resp(out);
|
||||
return out;
|
||||
}
|
||||
|
||||
Json MinimaxClient::retrieve_file(const std::string& file_id) {
|
||||
CallArgs a;
|
||||
a.query["file_id"] = file_id;
|
||||
Json out = call_json("files.retrieve", a);
|
||||
check_base_resp(out);
|
||||
return out;
|
||||
}
|
||||
|
||||
Json MinimaxClient::create_video_v2(const Json& body, bool confirm) {
|
||||
return call_json("v2.video.create", with_body(body), confirm);
|
||||
}
|
||||
Json MinimaxClient::query_video_v2(const std::string& id) { return call_json("v2.video.query", task(id)); }
|
||||
|
||||
Json MinimaxClient::list_video_v2(int page_num, int page_size) {
|
||||
CallArgs a;
|
||||
if (page_num > 0) a.query["page_num"] = std::to_string(page_num);
|
||||
if (page_size > 0) a.query["page_size"] = std::to_string(page_size);
|
||||
return call_json("v2.video.list", a);
|
||||
}
|
||||
|
||||
Json MinimaxClient::delete_video_v2(const std::string& id, bool confirm) {
|
||||
return call_json("v2.video.delete", task(id), confirm);
|
||||
}
|
||||
|
||||
JobState MinimaxVideoGenerator::state_of(const std::string& status) {
|
||||
if (status == "Success") return JobState::kSucceeded;
|
||||
if (status == "Fail") return JobState::kFailed;
|
||||
if (status == "Processing") return JobState::kRunning;
|
||||
return JobState::kQueued; // Preparing, Queueing
|
||||
}
|
||||
|
||||
Json MinimaxVideoGenerator::body(const VideoRequest& r) const {
|
||||
r.validate();
|
||||
Json b = Json::object();
|
||||
b.set("model", Json(r.model.empty() ? "MiniMax-Hailuo-2.3" : r.model));
|
||||
AdapterSupport::put(b, "prompt", r.prompt);
|
||||
AdapterSupport::put_int(b, "duration", r.duration_seconds);
|
||||
if (!r.resolution.empty()) {
|
||||
std::string res = r.resolution;
|
||||
for (char& c : res) c = static_cast<char>(std::toupper(static_cast<unsigned char>(c)));
|
||||
b.set("resolution", Json(res)); // docs: 768P / 1080P
|
||||
}
|
||||
AdapterSupport::put(b, "first_frame_image", r.image_url);
|
||||
AdapterSupport::put(b, "last_frame_image", r.end_image_url);
|
||||
if (!r.aspect_ratio.empty()) throw llm::ApiError::invalid("minimax", "v1 video_generation has no aspect_ratio");
|
||||
if (r.seed >= 0) throw llm::ApiError::invalid("minimax", "v1 video_generation has no seed");
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json MinimaxVideoGenerator::plan(const VideoRequest& request) const {
|
||||
return client_->plan_video(body(request)).summary();
|
||||
}
|
||||
|
||||
JobTicket MinimaxVideoGenerator::submit(const VideoRequest& request, bool confirm) {
|
||||
Json b = body(request);
|
||||
AdapterSupport::need_confirm(confirm, "minimax", "generate");
|
||||
Json out = client_->create_video(b, true);
|
||||
JobTicket t;
|
||||
t.provider = "minimax";
|
||||
t.model = b.find("model")->as_string();
|
||||
t.kind = MediaKind::kVideo;
|
||||
t.id = AdapterSupport::str(out, "task_id");
|
||||
if (t.id.empty()) throw llm::ApiError("minimax", 502, "BAD_RESPONSE", "no task_id in the reply");
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus MinimaxVideoGenerator::poll(const JobTicket& ticket) {
|
||||
Json q = client_->query_video(ticket.id);
|
||||
JobStatus s;
|
||||
s.raw_state = AdapterSupport::str(q, "status");
|
||||
s.state = state_of(s.raw_state);
|
||||
if (s.state == JobState::kFailed) s.error = "Fail";
|
||||
if (s.state == JobState::kSucceeded) {
|
||||
std::string file_id = AdapterSupport::str(q, "file_id");
|
||||
if (file_id.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "Success without a file_id";
|
||||
return s;
|
||||
}
|
||||
Json f = client_->retrieve_file(file_id);
|
||||
if (const Json* file = f.find("file")) s.assets = OutputScanner::scan(*file, MediaKind::kVideo);
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "file retrieved without a download_url";
|
||||
}
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+66
@@ -0,0 +1,66 @@
|
||||
#pragma once
|
||||
|
||||
// MiniMax (Hailuo) video API on https://api.minimax.io: v1 video_generation
|
||||
// (text, image, first/last frame, subject reference), the query endpoint, file
|
||||
// retrieval for the download URL, and the v2 content-array endpoints.
|
||||
// Bearer auth with MINIMAX_API_KEY. Errors can arrive as HTTP 200 with
|
||||
// base_resp.status_code != 0: MinimaxClient turns those into ApiError.
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/minimax/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class MinimaxClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api.minimax.io";
|
||||
static constexpr const char* kKeyEnv = "MINIMAX_API_KEY";
|
||||
|
||||
explicit MinimaxClient(std::shared_ptr<llm::ApiClient> client)
|
||||
: MediaService(std::move(client), MinimaxCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::bearer("minimax", kHost, kKeyEnv); }
|
||||
static std::shared_ptr<MinimaxClient> from_env() {
|
||||
return std::make_shared<MinimaxClient>(ClientFactory::real(config(), ClientFactory::env({kKeyEnv})));
|
||||
}
|
||||
static std::shared_ptr<MinimaxClient> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<MinimaxClient>(ClientFactory::with(std::move(t), config(), key));
|
||||
}
|
||||
|
||||
Json create_video(const Json& body, bool confirm); // v1, paid
|
||||
PreparedCall plan_video(const Json& body) const;
|
||||
Json query_video(const std::string& task_id); // v1
|
||||
Json retrieve_file(const std::string& file_id); // v1
|
||||
Json create_video_v2(const Json& body, bool confirm);
|
||||
Json query_video_v2(const std::string& task_id);
|
||||
Json list_video_v2(int page_num, int page_size);
|
||||
Json delete_video_v2(const std::string& task_id, bool confirm); // no default
|
||||
|
||||
// Throws ApiError when base_resp.status_code is present and not 0.
|
||||
static void check_base_resp(const Json& doc);
|
||||
};
|
||||
|
||||
class MinimaxVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit MinimaxVideoGenerator(std::shared_ptr<MinimaxClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "minimax"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"MiniMax-Hailuo-2.3", "MiniMax-Hailuo-02", "T2V-01-Director", "T2V-01"};
|
||||
}
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket&, bool) override { return false; } // v1 has no cancel
|
||||
int suggested_poll_ms() const override { return 8000; }
|
||||
|
||||
// Preparing / Queueing / Processing / Success / Fail (v1 query).
|
||||
static JobState state_of(const std::string& status);
|
||||
|
||||
private:
|
||||
Json body(const VideoRequest& request) const;
|
||||
std::shared_ptr<MinimaxClient> client_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+48
@@ -0,0 +1,48 @@
|
||||
#include "vendors/media/recraft/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"images.creativeUpscale", "POST", "/v1/images/creativeUpscale", "CreativeUpscale", "either", "image_format,response_format,upscale,zdr,image_url,image", "", "paid"},
|
||||
{"images.crispUpscale", "POST", "/v1/images/crispUpscale", "CrispUpscale", "either", "image_format,response_format,upscale,zdr,image_url,image", "", "paid"},
|
||||
{"images.enhance", "POST", "/v1/images/enhance", "EnhanceImage", "either", "image_format,refinement,response_format,zdr,image_url,image", "", "paid"},
|
||||
{"images.eraseRegion", "POST", "/v1/images/eraseRegion", "EraseRegion", "either", "image_format,response_format,zdr,image_url,mask_url,image,mask", "", "paid"},
|
||||
{"images.generateBackground", "POST", "/v1/images/generateBackground", "GenerateBackground", "either", "controls,image_format,model,n,negative_prompt,prompt,random_seed,response_format,style,style_id,style_match,text_layout,zdr,image_url,mask_url,image,mask", "", "paid"},
|
||||
{"images.generations", "POST", "/v1/images/generations", "GenerateImage", "either", "controls,image_format,model,n,negative_prompt,prompt,random_seed,response_format,size,style,style_id,style_match,text_layout,upscale,zdr,style_reference_urls,style_references", "", "paid"},
|
||||
{"images.generations.raster", "POST", "/v1/images/generations/raster", "GenerateImageRaster", "either", "controls,image_format,model,n,negative_prompt,prompt,random_seed,response_format,size,style,style_id,style_match,text_layout,upscale,zdr,style_reference_urls,style_references", "", "paid"},
|
||||
{"images.generations.vector", "POST", "/v1/images/generations/vector", "GenerateImageVector", "either", "controls,image_format,model,n,negative_prompt,prompt,random_seed,response_format,size,style,style_id,style_match,text_layout,upscale,zdr,style_reference_urls,style_references", "", "paid"},
|
||||
{"images.imageToImage", "POST", "/v1/images/imageToImage", "ImageToImage", "either", "controls,image_format,model,n,negative_prompt,prompt,random_seed,response_format,style,style_id,style_match,text_layout,zdr,image_url,strength,image", "", "paid"},
|
||||
{"images.inpaint", "POST", "/v1/images/inpaint", "InpaintImage", "either", "controls,image_format,model,n,negative_prompt,prompt,random_seed,response_format,style,style_id,style_match,text_layout,zdr,image_url,mask_url,image,mask", "", "paid"},
|
||||
{"images.outpaint", "POST", "/v1/images/outpaint", "OutpaintImage", "either", "controls,image_format,model,n,negative_prompt,prompt,random_seed,response_format,style,style_id,style_match,text_layout,zdr,expand_bottom,expand_left,expand_right,expand_top,size,zoom_out_percentage,image_url,image", "", "paid"},
|
||||
{"images.removeBackground", "POST", "/v1/images/removeBackground", "RemoveBackground", "either", "image_format,response_format,upscale,zdr,image_url,image", "", "paid"},
|
||||
{"images.replaceBackground", "POST", "/v1/images/replaceBackground", "ReplaceBackground", "either", "controls,image_format,model,n,negative_prompt,prompt,random_seed,response_format,style,style_id,style_match,text_layout,zdr,image_url,image", "", "paid"},
|
||||
{"images.variateImage", "POST", "/v1/images/variateImage", "VariateImage", "either", "image_format,model,n,random_seed,response_format,size,zdr,image_url,image", "", "paid"},
|
||||
{"images.vectorize", "POST", "/v1/images/vectorize", "VectorizeImage", "either", "image_format,response_format,upscale,zdr,color_reduction,limit_num_shapes,max_num_colors,max_num_shapes,return_gradients,shape_stacking,small_shape_filter,strict_color_palette,svg_compression,image_url,image", "", "paid"},
|
||||
{"styles", "GET", "/v1/styles", "ListStyles", "none", "", "", "read"},
|
||||
{"styles.create", "POST", "/v1/styles", "CreateStyle", "either", "image_weights,match,mix_policy,model,palette,private,prompt,source_style_weights,source_styles,style,image_urls,images", "", "paid"},
|
||||
{"styles.id.delete", "DELETE", "/v1/styles/{style_id}", "DeleteStyle", "none", "", "", "delete"},
|
||||
{"styles.id", "GET", "/v1/styles/{style_id}", "GetStyle", "none", "", "", "read"},
|
||||
{"styles.basic", "GET", "/v1/styles/basic", "ListBasicStyles", "none", "", "", "read"},
|
||||
{"users.me", "GET", "/v1/users/me", "GetCurrentUser", "none", "", "", "read"},
|
||||
{"prompts.enhance", "POST", "/v1/prompts/enhance", "EnhancePrompt", "json", "prompt", "", "paid"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
{"GenerateImageResponse", "created,data,credits,style_id"},
|
||||
{"User", "id,email,name,credits"},
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& RecraftCalls::table() {
|
||||
static const CallTable t("recraft", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t RecraftCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* RecraftCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct RecraftCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
#include "vendors/media/recraft/recraft.h"
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs args(const Json& fields, const std::vector<FilePart>& files) {
|
||||
CallArgs a;
|
||||
a.body = fields;
|
||||
a.files = files;
|
||||
return a;
|
||||
}
|
||||
|
||||
CallArgs style_id(const std::string& id) {
|
||||
CallArgs a;
|
||||
a.path["style_id"] = id;
|
||||
return a;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
PreparedCall RecraftClient::plan(const std::string& key, const Json& fields, const std::vector<FilePart>& files) const {
|
||||
return prepare(key, args(fields, files));
|
||||
}
|
||||
|
||||
Json RecraftClient::run(const std::string& key, const Json& fields, const std::vector<FilePart>& files, bool confirm) {
|
||||
return call_json(key, args(fields, files), confirm);
|
||||
}
|
||||
|
||||
Json RecraftClient::me() { return call_json("users.me"); }
|
||||
Json RecraftClient::styles() { return call_json("styles"); }
|
||||
Json RecraftClient::basic_styles() { return call_json("styles.basic"); }
|
||||
Json RecraftClient::style(const std::string& id) { return call_json("styles.id", style_id(id)); }
|
||||
Json RecraftClient::create_style(const Json& fields, const std::vector<FilePart>& files, bool confirm) {
|
||||
return call_json("styles.create", args(fields, files), confirm);
|
||||
}
|
||||
Json RecraftClient::delete_style(const std::string& id, bool confirm) {
|
||||
return call_json("styles.id.delete", style_id(id), confirm);
|
||||
}
|
||||
|
||||
Json RecraftImageGenerator::fields(const ImageRequest& r) const {
|
||||
r.validate();
|
||||
Json f = Json::object();
|
||||
AdapterSupport::put(f, "prompt", r.prompt);
|
||||
AdapterSupport::put(f, "negative_prompt", r.negative_prompt);
|
||||
AdapterSupport::put(f, "model", r.model);
|
||||
if (r.count > 1) f.set("n", Json(r.count));
|
||||
if (r.width && r.height) f.set("size", Json(std::to_string(r.width) + "x" + std::to_string(r.height)));
|
||||
AdapterSupport::put_seed(f, "random_seed", r.seed);
|
||||
if (!r.aspect_ratio.empty())
|
||||
throw llm::ApiError::invalid("recraft", "Recraft sizes images with size (WxH), not aspect_ratio");
|
||||
if (!r.source_url.empty())
|
||||
throw llm::ApiError::invalid("recraft", "source images use the images.* edit rows: RecraftClient::run with image_url");
|
||||
AdapterSupport::merge_extra(f, r.extra);
|
||||
return f;
|
||||
}
|
||||
|
||||
Json RecraftImageGenerator::plan(const ImageRequest& request) const {
|
||||
return client_->plan("images.generations", fields(request), {}).summary();
|
||||
}
|
||||
|
||||
JobTicket RecraftImageGenerator::submit(const ImageRequest& request, bool confirm) {
|
||||
Json f = fields(request);
|
||||
AdapterSupport::need_confirm(confirm, "recraft", "generate");
|
||||
Json out = client_->run("images.generations", f, {}, true);
|
||||
JobTicket t;
|
||||
t.provider = "recraft";
|
||||
t.model = request.model;
|
||||
t.kind = MediaKind::kImage;
|
||||
t.id = "immediate";
|
||||
t.immediate = true;
|
||||
if (const Json* d = out.find("data"); d && d->is_array()) {
|
||||
for (const Json& item : d->as_array()) {
|
||||
std::string b64 = AdapterSupport::str(item, "b64_json");
|
||||
if (!b64.empty()) {
|
||||
MediaAsset a;
|
||||
a.kind = MediaKind::kImage;
|
||||
a.data_base64 = b64;
|
||||
a.mime = "image/webp"; // Recraft's default raster format is lossless WebP
|
||||
t.assets.push_back(a);
|
||||
}
|
||||
}
|
||||
if (t.assets.empty()) t.assets = OutputScanner::scan(*d, MediaKind::kImage);
|
||||
}
|
||||
if (t.assets.empty()) throw llm::ApiError("recraft", 502, "BAD_RESPONSE", "no image in the reply");
|
||||
t.meta = Json::object();
|
||||
AdapterSupport::put(t.meta, "credits", AdapterSupport::str(out, "credits"));
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus RecraftImageGenerator::poll(const JobTicket& ticket) {
|
||||
JobStatus s;
|
||||
s.state = JobState::kSucceeded;
|
||||
s.raw_state = "immediate";
|
||||
s.assets = ticket.assets;
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+61
@@ -0,0 +1,61 @@
|
||||
#pragma once
|
||||
|
||||
// Recraft external API: generation (general, raster, vector), image tools
|
||||
// (vectorize, background, upscale, erase, remix, inpaint, outpaint, enhance),
|
||||
// styles, prompt enhancement and the user endpoint. Bearer auth with
|
||||
// RECRAFT_API_TOKEN, base https://external.api.recraft.ai. Every image call
|
||||
// is synchronous: the reply carries the result, so tickets are immediate.
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/recraft/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class RecraftClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://external.api.recraft.ai";
|
||||
static constexpr const char* kKeyEnv = "RECRAFT_API_TOKEN";
|
||||
|
||||
explicit RecraftClient(std::shared_ptr<llm::ApiClient> client)
|
||||
: MediaService(std::move(client), RecraftCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::bearer("recraft", kHost, kKeyEnv); }
|
||||
static std::shared_ptr<RecraftClient> from_env() {
|
||||
return std::make_shared<RecraftClient>(ClientFactory::real(config(), ClientFactory::env({kKeyEnv})));
|
||||
}
|
||||
static std::shared_ptr<RecraftClient> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<RecraftClient>(ClientFactory::with(std::move(t), config(), key));
|
||||
}
|
||||
|
||||
// Paid image rows (images.*) and prompts.enhance. JSON bodies take image_url
|
||||
// fields; multipart is used when files are given.
|
||||
Json run(const std::string& key, const Json& fields, const std::vector<FilePart>& files, bool confirm);
|
||||
PreparedCall plan(const std::string& key, const Json& fields, const std::vector<FilePart>& files) const;
|
||||
Json me(); // credits + identity: the free read
|
||||
Json styles();
|
||||
Json basic_styles();
|
||||
Json style(const std::string& style_id);
|
||||
Json create_style(const Json& fields, const std::vector<FilePart>& files, bool confirm); // $0.005
|
||||
Json delete_style(const std::string& style_id, bool confirm); // destructive
|
||||
};
|
||||
|
||||
class RecraftImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit RecraftImageGenerator(std::shared_ptr<RecraftClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "recraft"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"recraftv4_1", "recraftv4_1_pro", "recraftv4", "recraftv4_pro", "recraftv3", "recraftv3_vector", "recraftv2"};
|
||||
}
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket&, bool) override { return false; } // synchronous API
|
||||
|
||||
private:
|
||||
Json fields(const ImageRequest& request) const;
|
||||
std::shared_ptr<RecraftClient> client_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
#include "vendors/media/replicate/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"account.get", "GET", "/account", "account.get", "none", "", "", "read"},
|
||||
{"collections.list", "GET", "/collections", "collections.list", "none", "", "", "read"},
|
||||
{"collections.get", "GET", "/collections/{collection_slug}", "collections.get", "none", "", "", "read"},
|
||||
{"deployments.list", "GET", "/deployments", "deployments.list", "none", "", "", "read"},
|
||||
{"deployments.create", "POST", "/deployments", "deployments.create", "json", "hardware,max_instances,min_instances,model,name,version", "", "write"},
|
||||
{"deployments.delete", "DELETE", "/deployments/{deployment_owner}/{deployment_name}", "deployments.delete", "none", "", "", "delete"},
|
||||
{"deployments.get", "GET", "/deployments/{deployment_owner}/{deployment_name}", "deployments.get", "none", "", "", "read"},
|
||||
{"deployments.update", "PATCH", "/deployments/{deployment_owner}/{deployment_name}", "deployments.update", "json", "hardware,max_instances,min_instances,version", "", "write"},
|
||||
{"deployments.predictions.create", "POST", "/deployments/{deployment_owner}/{deployment_name}/predictions", "deployments.predictions.create", "json", "input,stream,webhook,webhook_events_filter", "", "paid"},
|
||||
{"files.list", "GET", "/files", "files.list", "none", "", "", "read"},
|
||||
{"files.create", "POST", "/files", "files.create", "multipart", "content,filename,metadata,type", "", "write"},
|
||||
{"files.delete", "DELETE", "/files/{file_id}", "files.delete", "none", "", "", "delete"},
|
||||
{"files.get", "GET", "/files/{file_id}", "files.get", "none", "", "", "read"},
|
||||
{"files.download", "GET", "/files/{file_id}/download", "files.download", "none", "", "owner,expiry,signature", "read"},
|
||||
{"hardware.list", "GET", "/hardware", "hardware.list", "none", "", "", "read"},
|
||||
{"models.list", "GET", "/models", "models.list", "none", "", "sort_by,sort_direction", "read"},
|
||||
{"models.create", "POST", "/models", "models.create", "json", "cover_image_url,description,github_url,hardware,license_url,name,owner,paper_url,visibility", "", "write"},
|
||||
{"models.delete", "DELETE", "/models/{model_owner}/{model_name}", "models.delete", "none", "", "", "delete"},
|
||||
{"models.get", "GET", "/models/{model_owner}/{model_name}", "models.get", "none", "", "", "read"},
|
||||
{"models.update", "PATCH", "/models/{model_owner}/{model_name}", "models.update", "json", "description,github_url,license_url,paper_url,readme,weights_url", "", "write"},
|
||||
{"models.examples.list", "GET", "/models/{model_owner}/{model_name}/examples", "models.examples.list", "none", "", "", "read"},
|
||||
{"models.predictions.create", "POST", "/models/{model_owner}/{model_name}/predictions", "models.predictions.create", "json", "input,stream,webhook,webhook_events_filter", "", "paid"},
|
||||
{"models.readme.get", "GET", "/models/{model_owner}/{model_name}/readme", "models.readme.get", "none", "", "", "read"},
|
||||
{"models.versions.list", "GET", "/models/{model_owner}/{model_name}/versions", "models.versions.list", "none", "", "", "read"},
|
||||
{"models.versions.delete", "DELETE", "/models/{model_owner}/{model_name}/versions/{version_id}", "models.versions.delete", "none", "", "", "delete"},
|
||||
{"models.versions.get", "GET", "/models/{model_owner}/{model_name}/versions/{version_id}", "models.versions.get", "none", "", "", "read"},
|
||||
{"trainings.create", "POST", "/models/{model_owner}/{model_name}/versions/{version_id}/trainings", "trainings.create", "json", "destination,input,webhook,webhook_events_filter", "", "paid"},
|
||||
{"predictions.list", "GET", "/predictions", "predictions.list", "none", "", "created_after,created_before,source", "read"},
|
||||
{"predictions.create", "POST", "/predictions", "predictions.create", "json", "input,stream,version,webhook,webhook_events_filter", "", "paid"},
|
||||
{"predictions.get", "GET", "/predictions/{prediction_id}", "predictions.get", "none", "", "", "read"},
|
||||
{"predictions.cancel", "POST", "/predictions/{prediction_id}/cancel", "predictions.cancel", "none", "", "", "write"},
|
||||
{"search", "GET", "/search", "search", "none", "", "query,limit", "read"},
|
||||
{"trainings.list", "GET", "/trainings", "trainings.list", "none", "", "", "read"},
|
||||
{"trainings.get", "GET", "/trainings/{training_id}", "trainings.get", "none", "", "", "read"},
|
||||
{"trainings.cancel", "POST", "/trainings/{training_id}/cancel", "trainings.cancel", "none", "", "", "write"},
|
||||
{"webhooks.default.secret.get", "GET", "/webhooks/default/secret", "webhooks.default.secret.get", "none", "", "", "read"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
{"schemas_prediction_response", "id,model,version,input,output,status,error,logs,metrics,created_at,started_at,completed_at,urls,get,cancel"},
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& ReplicateCalls::table() {
|
||||
static const CallTable t("replicate", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t ReplicateCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* ReplicateCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct ReplicateCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+278
@@ -0,0 +1,278 @@
|
||||
#include "vendors/media/replicate/replicate.h"
|
||||
|
||||
#include <cstdlib>
|
||||
#include <sstream>
|
||||
|
||||
#include "security/crypto/Base64.h"
|
||||
#include "security/crypto/Hmac.h"
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/caps.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs args_for(const ReplicateClient::ModelRef& m, const Json& input, const Json& options) {
|
||||
CallArgs a;
|
||||
Json body = Json::object();
|
||||
body.set("input", input.is_null() ? Json::object() : input);
|
||||
if (options.is_object())
|
||||
for (const auto& [k, v] : options.as_object()) body.set(k, v); // webhook, webhook_events_filter, stream
|
||||
if (m.version.empty()) {
|
||||
a.path["model_owner"] = m.owner;
|
||||
a.path["model_name"] = m.name;
|
||||
} else {
|
||||
body.set("version", Json(m.version));
|
||||
}
|
||||
a.body = body;
|
||||
return a;
|
||||
}
|
||||
|
||||
CallArgs one(const char* name, const std::string& value) {
|
||||
CallArgs a;
|
||||
a.path[name] = value;
|
||||
return a;
|
||||
}
|
||||
|
||||
Json mark_truncated(Json page) {
|
||||
const Json* next = page.is_object() ? page.find("next") : nullptr;
|
||||
if (next && next->is_string() && !next->as_string().empty()) page.set("truncated", Json(true));
|
||||
return page;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
ReplicateClient::ModelRef ReplicateClient::ModelRef::parse(const std::string& ref) {
|
||||
ModelRef m;
|
||||
std::string head = ref;
|
||||
std::size_t colon = ref.find(':');
|
||||
if (colon != std::string::npos) {
|
||||
head = ref.substr(0, colon);
|
||||
m.version = ref.substr(colon + 1);
|
||||
}
|
||||
std::size_t slash = head.find('/');
|
||||
if (slash == std::string::npos || head.find('/', slash + 1) != std::string::npos)
|
||||
throw llm::ApiError::invalid("replicate", "model must look like owner/name or owner/name:version");
|
||||
m.owner = head.substr(0, slash);
|
||||
m.name = head.substr(slash + 1);
|
||||
if (!CallTable::valid_segment(m.owner) || !CallTable::valid_segment(m.name) ||
|
||||
(colon != std::string::npos && !CallTable::valid_segment(m.version)))
|
||||
throw llm::ApiError::invalid("replicate", "bad model reference");
|
||||
return m;
|
||||
}
|
||||
|
||||
PreparedCall ReplicateClient::plan_prediction(const ModelRef& m, const Json& input, const Json& options) const {
|
||||
return prepare(m.version.empty() ? "models.predictions.create" : "predictions.create", args_for(m, input, options));
|
||||
}
|
||||
|
||||
Json ReplicateClient::create_prediction(const ModelRef& m, const Json& input, const Json& options, bool confirm) {
|
||||
return call_json(m.version.empty() ? "models.predictions.create" : "predictions.create",
|
||||
args_for(m, input, options), confirm);
|
||||
}
|
||||
|
||||
Json ReplicateClient::create_deployment_prediction(const std::string& owner, const std::string& name,
|
||||
const Json& input, const Json& options, bool confirm) {
|
||||
CallArgs a;
|
||||
a.path["deployment_owner"] = owner;
|
||||
a.path["deployment_name"] = name;
|
||||
Json body = Json::object();
|
||||
body.set("input", input);
|
||||
if (options.is_object())
|
||||
for (const auto& [k, v] : options.as_object()) body.set(k, v);
|
||||
a.body = body;
|
||||
return call_json("deployments.predictions.create", a, confirm);
|
||||
}
|
||||
|
||||
Json ReplicateClient::get_prediction(const std::string& id) { return call_json("predictions.get", one("prediction_id", id)); }
|
||||
Json ReplicateClient::cancel_prediction(const std::string& id, bool confirm) {
|
||||
return call_json("predictions.cancel", one("prediction_id", id), confirm);
|
||||
}
|
||||
Json ReplicateClient::list_predictions() { return mark_truncated(call_json("predictions.list")); }
|
||||
Json ReplicateClient::account() { return call_json("account.get"); }
|
||||
Json ReplicateClient::hardware() { return call_json("hardware.list"); }
|
||||
Json ReplicateClient::models_list() { return mark_truncated(call_json("models.list")); }
|
||||
|
||||
Json ReplicateClient::model_get(const std::string& owner, const std::string& name) {
|
||||
CallArgs a;
|
||||
a.path["model_owner"] = owner;
|
||||
a.path["model_name"] = name;
|
||||
return call_json("models.get", a);
|
||||
}
|
||||
|
||||
Json ReplicateClient::versions_list(const std::string& owner, const std::string& name) {
|
||||
CallArgs a;
|
||||
a.path["model_owner"] = owner;
|
||||
a.path["model_name"] = name;
|
||||
return mark_truncated(call_json("models.versions.list", a));
|
||||
}
|
||||
|
||||
Json ReplicateClient::deployments_list() { return mark_truncated(call_json("deployments.list")); }
|
||||
|
||||
Json ReplicateClient::deployment_get(const std::string& owner, const std::string& name) {
|
||||
CallArgs a;
|
||||
a.path["deployment_owner"] = owner;
|
||||
a.path["deployment_name"] = name;
|
||||
return call_json("deployments.get", a);
|
||||
}
|
||||
|
||||
Json ReplicateClient::files_list() { return mark_truncated(call_json("files.list")); }
|
||||
Json ReplicateClient::file_get(const std::string& id) { return call_json("files.get", one("file_id", id)); }
|
||||
Json ReplicateClient::file_delete(const std::string& id, bool confirm) {
|
||||
return call_json("files.delete", one("file_id", id), confirm);
|
||||
}
|
||||
|
||||
Json ReplicateClient::file_upload(const std::string& filename, const std::string& content_type,
|
||||
const std::string& data, bool confirm) {
|
||||
CallArgs a;
|
||||
a.body = Json::object();
|
||||
a.body.set("filename", Json(filename));
|
||||
a.files.push_back(FilePart{"content", filename, content_type, data});
|
||||
return call_json("files.create", a, confirm);
|
||||
}
|
||||
|
||||
Json ReplicateClient::collections_list() { return mark_truncated(call_json("collections.list")); }
|
||||
|
||||
Json ReplicateClient::search(const std::string& query) {
|
||||
CallArgs a;
|
||||
a.query["query"] = query;
|
||||
return call_json("search", a);
|
||||
}
|
||||
|
||||
Json ReplicateClient::webhook_secret() { return call_json("webhooks.default.secret.get"); }
|
||||
Json ReplicateClient::trainings_list() { return mark_truncated(call_json("trainings.list")); }
|
||||
|
||||
// ---------------------------------------------------------------- webhooks
|
||||
|
||||
std::string ReplicateWebhook::sign(const std::string& secret, const std::string& id, const std::string& timestamp,
|
||||
const std::string& body) {
|
||||
std::string b64 = secret.rfind("whsec_", 0) == 0 ? secret.substr(6) : secret;
|
||||
std::string key;
|
||||
if (b64.empty() || !security::crypto::Base64::decode(b64, key) || key.empty()) return "";
|
||||
if (body.size() > MediaCaps::kMaxWebhookBodyBytes) return "";
|
||||
auto mac = security::crypto::Hmac::sha256(key, id + "." + timestamp + "." + body);
|
||||
return security::crypto::Base64::encode(reinterpret_cast<const std::uint8_t*>(mac.data()), mac.size());
|
||||
}
|
||||
|
||||
bool ReplicateWebhook::verify(const std::string& secret, const std::string& id, const std::string& timestamp,
|
||||
const std::string& signature_header, const std::string& body, long now_seconds) {
|
||||
if (id.empty() || timestamp.empty() || timestamp.size() > 12 || signature_header.size() > 1024) return false;
|
||||
for (char c : timestamp)
|
||||
if (c < '0' || c > '9') return false;
|
||||
long ts = std::strtol(timestamp.c_str(), nullptr, 10);
|
||||
long skew = now_seconds > ts ? now_seconds - ts : ts - now_seconds;
|
||||
if (skew > MediaCaps::kWebhookToleranceSeconds) return false;
|
||||
std::string expected = sign(secret, id, timestamp, body);
|
||||
if (expected.empty()) return false;
|
||||
std::istringstream in(signature_header);
|
||||
std::string part;
|
||||
while (in >> part) {
|
||||
if (part.rfind("v1,", 0) != 0) continue;
|
||||
std::string got = part.substr(3);
|
||||
if (got.size() != expected.size()) continue;
|
||||
unsigned char diff = 0; // constant time over the whole string
|
||||
for (std::size_t i = 0; i < got.size(); ++i) diff |= static_cast<unsigned char>(got[i] ^ expected[i]);
|
||||
if (diff == 0) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------- jobs
|
||||
|
||||
JobTicket ReplicateJobs::ticket_from(const Json& p, const std::string& model, MediaKind kind) const {
|
||||
JobTicket t;
|
||||
t.provider = "replicate";
|
||||
t.model = model;
|
||||
t.kind = kind;
|
||||
t.id = AdapterSupport::str(p, "id");
|
||||
if (t.id.empty()) throw llm::ApiError("replicate", 502, "BAD_RESPONSE", "no id in the prediction");
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus ReplicateJobs::poll(const JobTicket& ticket, MediaKind kind) {
|
||||
Json p = client_->get_prediction(ticket.id);
|
||||
JobStatus s;
|
||||
s.raw_state = AdapterSupport::str(p, "status");
|
||||
if (s.raw_state == "succeeded") {
|
||||
s.state = JobState::kSucceeded;
|
||||
if (const Json* o = p.find("output")) s.assets = OutputScanner::scan(*o, kind);
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "succeeded without a downloadable output (non-URL output)";
|
||||
}
|
||||
} else if (s.raw_state == "failed") {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = AdapterSupport::str(p, "error");
|
||||
} else if (s.raw_state == "canceled" || s.raw_state == "aborted") {
|
||||
s.state = JobState::kCancelled;
|
||||
} else if (s.raw_state == "processing") {
|
||||
s.state = JobState::kRunning;
|
||||
} else {
|
||||
s.state = JobState::kQueued;
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
bool ReplicateJobs::cancel(const JobTicket& ticket, bool confirm) {
|
||||
AdapterSupport::need_confirm(confirm, "replicate", "cancel");
|
||||
client_->cancel_prediction(ticket.id, true);
|
||||
return true;
|
||||
}
|
||||
|
||||
Json ReplicateImageGenerator::input(const ImageRequest& r) const {
|
||||
r.validate();
|
||||
Json in = Json::object();
|
||||
AdapterSupport::put(in, "prompt", r.prompt);
|
||||
AdapterSupport::put(in, "negative_prompt", r.negative_prompt);
|
||||
AdapterSupport::put_int(in, "width", r.width);
|
||||
AdapterSupport::put_int(in, "height", r.height);
|
||||
AdapterSupport::put(in, "aspect_ratio", r.aspect_ratio);
|
||||
AdapterSupport::put_seed(in, "seed", r.seed);
|
||||
if (r.count > 1) in.set("num_outputs", Json(r.count));
|
||||
AdapterSupport::put(in, "image", r.source_url);
|
||||
AdapterSupport::merge_extra(in, r.extra);
|
||||
return in;
|
||||
}
|
||||
|
||||
Json ReplicateImageGenerator::plan(const ImageRequest& r) const {
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("replicate", "model is required (owner/name)");
|
||||
return jobs_.client().plan_prediction(ReplicateClient::ModelRef::parse(r.model), input(r), Json::object()).summary();
|
||||
}
|
||||
|
||||
JobTicket ReplicateImageGenerator::submit(const ImageRequest& r, bool confirm) {
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("replicate", "model is required (owner/name)");
|
||||
auto ref = ReplicateClient::ModelRef::parse(r.model);
|
||||
Json in = input(r);
|
||||
AdapterSupport::need_confirm(confirm, "replicate", "create prediction");
|
||||
return jobs_.ticket_from(jobs_.client().create_prediction(ref, in, Json::object(), true), r.model, MediaKind::kImage);
|
||||
}
|
||||
|
||||
Json ReplicateVideoGenerator::input(const VideoRequest& r) const {
|
||||
r.validate();
|
||||
Json in = Json::object();
|
||||
AdapterSupport::put(in, "prompt", r.prompt);
|
||||
AdapterSupport::put(in, "negative_prompt", r.negative_prompt);
|
||||
AdapterSupport::put_int(in, "duration", r.duration_seconds);
|
||||
AdapterSupport::put(in, "aspect_ratio", r.aspect_ratio);
|
||||
AdapterSupport::put(in, "resolution", r.resolution);
|
||||
AdapterSupport::put_seed(in, "seed", r.seed);
|
||||
AdapterSupport::put(in, "image", r.image_url);
|
||||
AdapterSupport::put(in, "last_frame_image", r.end_image_url);
|
||||
AdapterSupport::merge_extra(in, r.extra);
|
||||
return in;
|
||||
}
|
||||
|
||||
Json ReplicateVideoGenerator::plan(const VideoRequest& r) const {
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("replicate", "model is required (owner/name)");
|
||||
return jobs_.client().plan_prediction(ReplicateClient::ModelRef::parse(r.model), input(r), Json::object()).summary();
|
||||
}
|
||||
|
||||
JobTicket ReplicateVideoGenerator::submit(const VideoRequest& r, bool confirm) {
|
||||
if (r.model.empty()) throw llm::ApiError::invalid("replicate", "model is required (owner/name)");
|
||||
auto ref = ReplicateClient::ModelRef::parse(r.model);
|
||||
Json in = input(r);
|
||||
AdapterSupport::need_confirm(confirm, "replicate", "create prediction");
|
||||
return jobs_.ticket_from(jobs_.client().create_prediction(ref, in, Json::object(), true), r.model, MediaKind::kVideo);
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
#pragma once
|
||||
|
||||
// Replicate: predictions (official models, versions, deployments), models,
|
||||
// files, account, hardware, collections, trainings and webhook verification.
|
||||
// Bearer auth, base https://api.replicate.com/v1.
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/replicate/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class ReplicateClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api.replicate.com/v1"; // the spec server url; table paths are relative to it
|
||||
static constexpr const char* kKeyEnv = "REPLICATE_API_TOKEN";
|
||||
|
||||
explicit ReplicateClient(std::shared_ptr<llm::ApiClient> client)
|
||||
: MediaService(std::move(client), ReplicateCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::bearer("replicate", kHost, kKeyEnv); }
|
||||
static std::shared_ptr<ReplicateClient> from_env() {
|
||||
return std::make_shared<ReplicateClient>(ClientFactory::real(config(), ClientFactory::env({kKeyEnv})));
|
||||
}
|
||||
static std::shared_ptr<ReplicateClient> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<ReplicateClient>(ClientFactory::with(std::move(t), config(), key));
|
||||
}
|
||||
|
||||
// "owner/name" (official model) or "owner/name:version" (pinned version).
|
||||
struct ModelRef {
|
||||
std::string owner, name, version;
|
||||
static ModelRef parse(const std::string& ref); // throws ApiError::invalid
|
||||
};
|
||||
|
||||
// Paid. Official model -> models.predictions.create; with version -> predictions.create.
|
||||
Json create_prediction(const ModelRef& model, const Json& input, const Json& options, bool confirm);
|
||||
PreparedCall plan_prediction(const ModelRef& model, const Json& input, const Json& options) const;
|
||||
Json create_deployment_prediction(const std::string& owner, const std::string& name, const Json& input,
|
||||
const Json& options, bool confirm);
|
||||
Json get_prediction(const std::string& id);
|
||||
Json cancel_prediction(const std::string& id, bool confirm); // write, no default
|
||||
// First page only; "truncated" is set when the server has a next page.
|
||||
Json list_predictions();
|
||||
|
||||
Json account();
|
||||
Json hardware();
|
||||
Json models_list();
|
||||
Json model_get(const std::string& owner, const std::string& name);
|
||||
Json versions_list(const std::string& owner, const std::string& name);
|
||||
Json deployments_list();
|
||||
Json deployment_get(const std::string& owner, const std::string& name);
|
||||
Json files_list();
|
||||
Json file_get(const std::string& id);
|
||||
Json file_upload(const std::string& filename, const std::string& content_type, const std::string& data, bool confirm);
|
||||
Json file_delete(const std::string& id, bool confirm); // destructive, no default
|
||||
Json collections_list();
|
||||
Json search(const std::string& query);
|
||||
Json webhook_secret();
|
||||
Json trainings_list();
|
||||
};
|
||||
|
||||
// Standard Webhooks verification (id, timestamp, signature headers).
|
||||
class ReplicateWebhook {
|
||||
public:
|
||||
// secret = the "whsec_..." value from webhooks.default.secret.get.
|
||||
// signature_header = "v1,<base64> v1,<base64>" (space separated).
|
||||
// now_seconds: injected clock. Returns false on any mismatch, stale or
|
||||
// malformed input; never throws on attacker-controlled data.
|
||||
static bool verify(const std::string& secret, const std::string& webhook_id, const std::string& timestamp,
|
||||
const std::string& signature_header, const std::string& body, long now_seconds);
|
||||
// base64(HMAC-SHA256(key, id + "." + timestamp + "." + body)); "" on a bad secret.
|
||||
static std::string sign(const std::string& secret, const std::string& webhook_id, const std::string& timestamp,
|
||||
const std::string& body);
|
||||
};
|
||||
|
||||
// kind-parametrised adapter core shared by the image and video generators.
|
||||
class ReplicateJobs {
|
||||
public:
|
||||
explicit ReplicateJobs(std::shared_ptr<ReplicateClient> c) : client_(std::move(c)) {}
|
||||
JobStatus poll(const JobTicket& ticket, MediaKind kind);
|
||||
bool cancel(const JobTicket& ticket, bool confirm);
|
||||
JobTicket ticket_from(const Json& prediction, const std::string& model, MediaKind kind) const;
|
||||
ReplicateClient& client() { return *client_; }
|
||||
const ReplicateClient& client() const { return *client_; }
|
||||
|
||||
private:
|
||||
std::shared_ptr<ReplicateClient> client_;
|
||||
};
|
||||
|
||||
class ReplicateImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit ReplicateImageGenerator(std::shared_ptr<ReplicateClient> c) : jobs_(std::move(c)) {}
|
||||
std::string provider() const override { return "replicate"; }
|
||||
bool configured() const override { return jobs_.client().configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"black-forest-labs/flux-schnell", "black-forest-labs/flux-dev", "black-forest-labs/flux-1.1-pro",
|
||||
"stability-ai/stable-diffusion-3.5-large", "ideogram-ai/ideogram-v3-turbo", "recraft-ai/recraft-v3"};
|
||||
}
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override { return jobs_.poll(ticket, MediaKind::kImage); }
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override { return jobs_.cancel(ticket, confirm); }
|
||||
int suggested_poll_ms() const override { return 1500; }
|
||||
|
||||
private:
|
||||
Json input(const ImageRequest& request) const;
|
||||
ReplicateJobs jobs_;
|
||||
};
|
||||
|
||||
class ReplicateVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit ReplicateVideoGenerator(std::shared_ptr<ReplicateClient> c) : jobs_(std::move(c)) {}
|
||||
std::string provider() const override { return "replicate"; }
|
||||
bool configured() const override { return jobs_.client().configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"minimax/video-01", "luma/ray", "kwaivgi/kling-v1.6-standard", "wan-video/wan-2.1-t2v-480p"};
|
||||
}
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override { return jobs_.poll(ticket, MediaKind::kVideo); }
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override { return jobs_.cancel(ticket, confirm); }
|
||||
int suggested_poll_ms() const override { return 3000; }
|
||||
|
||||
private:
|
||||
Json input(const VideoRequest& request) const;
|
||||
ReplicateJobs jobs_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+91
@@ -0,0 +1,91 @@
|
||||
#include "vendors/media/runway/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"text_to_image", "POST", "/v1/text_to_image", "", "json", "model,promptText,ratio,referenceImages,contentModeration,seed,background,outputCount,quality,outputFormat,grounding,edit,maskUri,preciseEdit,thinkingLevel,backgroundColor,colors", "", "paid"},
|
||||
{"text_to_video", "POST", "/v1/text_to_video", "", "json", "duration,model,promptText,ratio,contentModeration,outputFormat,proresProfile,seed,audio,negativePrompt,providerSettings,referenceAudio,references,referenceVideos,resolution,cfgScale,promptExpansionMode,draft,draftTaskId", "", "paid"},
|
||||
{"image_to_video", "POST", "/v1/image_to_video", "", "json", "duration,model,promptImage,promptText,ratio,contentModeration,outputFormat,proresProfile,seed,audio,negativePrompt,providerSettings,references,referenceAudio,resolution,cfgScale,referenceVideos,promptExpansionMode,draft", "", "paid"},
|
||||
{"video_to_video", "POST", "/v1/video_to_video", "", "json", "model,videoUri,contentModeration,keyframes,outputFormat,promptText,proresProfile,ratio,seed,targetAspectRatio,promptVideo,providerSettings,references,duration,referenceAudio,referenceVideos,resolution,audio,mode,draft", "", "paid"},
|
||||
{"image_upscale", "POST", "/v1/image_upscale", "", "json", "imageUri,model,flavor,scaleFactor,sharpen,smartGrain,ultraDetail", "", "paid"},
|
||||
{"video_upscale", "POST", "/v1/video_upscale", "", "json", "model,videoUri,creativity,flavor,fpsBoost,resolution,sharpen,smartGrain,targetFramerate", "", "paid"},
|
||||
{"video_to_hdr", "POST", "/v1/video_to_hdr", "", "json", "model,videoUri,outputFormat,proresProfile", "", "paid"},
|
||||
{"video_to_audio", "POST", "/v1/video_to_audio", "", "json", "duration,model,videoUri,format,negativePrompt,numVariants,preserveSpeech,promptText,seed", "", "paid"},
|
||||
{"character_performance", "POST", "/v1/character_performance", "", "json", "type,uri,publicFigureThreshold,character,model,reference,bodyControl,contentModeration,expressionIntensity,ratio,seed", "", "paid"},
|
||||
{"sound_effect", "POST", "/v1/sound_effect", "", "json", "model,promptText,loudnessRate,outputFormat,pitchRate,referenceAudios,sampleRate,speechRate,duration,format,negativePrompt,numVariants,seed,loop", "", "paid"},
|
||||
{"speech_to_speech", "POST", "/v1/speech_to_speech", "", "json", "type,uri,presetId,media,model,voice,removeBackgroundNoise", "", "paid"},
|
||||
{"text_to_speech", "POST", "/v1/text_to_speech", "", "json", "model,promptText,loudnessRate,outputFormat,pitchRate,sampleRate,speechRate,voice,languageCode,nextText,previousText,seed,similarityBoost,speed,stability,style,useSpeakerBoost,applyTextNormalization", "", "paid"},
|
||||
{"voice_dubbing", "POST", "/v1/voice_dubbing", "", "json", "audioUri,model,targetLang,disableVoiceCloning,dropBackgroundAudio,numSpeakers", "", "paid"},
|
||||
{"voice_isolation", "POST", "/v1/voice_isolation", "", "json", "audioUri,model", "", "paid"},
|
||||
{"avatar_videos", "POST", "/v1/avatar_videos", "", "json", "presetId,type,avatarId,audio,text,voice,avatar,model,speech", "", "paid"},
|
||||
{"generate.image", "POST", "/v1/generate/image", "", "json", "promptText,aspectRatio,contentModeration,outputCount,referenceImages,resolution,seed,configId,input", "", "paid"},
|
||||
{"generate.video", "POST", "/v1/generate/video", "", "json", "aspectRatio,audio,contentModeration,duration,keyframes,negativePrompt,promptText,referenceAudio,referenceImages,referenceVideos,resolution,seed,configId,input", "", "paid"},
|
||||
{"generate.audio", "POST", "/v1/generate/audio", "", "json", "promptText,type,duration,loop,referenceAudios,voice,configId,input", "", "paid"},
|
||||
{"recipes.ad_localization", "POST", "/v1/recipes/ad_localization", "", "json", "uri,mode,prompt,version,audio,duration,firstFrame,ratio,shots,view,referenceImage,targetLanguage,outputCount,quality,productImages,productInfo,styleImages,userConcept,image,newProductImages,originalProductImage,referenceVideo,resolution,characterImage,productImage", "", "paid"},
|
||||
{"recipes.marketing_stock_image", "POST", "/v1/recipes/marketing_stock_image", "", "json", "uri,mode,prompt,version,audio,duration,firstFrame,ratio,shots,view,referenceImage,targetLanguage,outputCount,quality,productImages,productInfo,styleImages,userConcept,image,newProductImages,originalProductImage,referenceVideo,resolution,characterImage,productImage", "", "paid"},
|
||||
{"recipes.multi_shot_video", "POST", "/v1/recipes/multi_shot_video", "", "json", "uri,mode,prompt,version,audio,duration,firstFrame,ratio,shots,view,referenceImage,targetLanguage,outputCount,quality,productImages,productInfo,styleImages,userConcept,image,newProductImages,originalProductImage,referenceVideo,resolution,characterImage,productImage", "", "paid"},
|
||||
{"recipes.product_ad", "POST", "/v1/recipes/product_ad", "", "json", "uri,mode,prompt,version,audio,duration,firstFrame,ratio,shots,view,referenceImage,targetLanguage,outputCount,quality,productImages,productInfo,styleImages,userConcept,image,newProductImages,originalProductImage,referenceVideo,resolution,characterImage,productImage", "", "paid"},
|
||||
{"recipes.product_campaign_image", "POST", "/v1/recipes/product_campaign_image", "", "json", "uri,mode,prompt,version,audio,duration,firstFrame,ratio,shots,view,referenceImage,targetLanguage,outputCount,quality,productImages,productInfo,styleImages,userConcept,image,newProductImages,originalProductImage,referenceVideo,resolution,characterImage,productImage", "", "paid"},
|
||||
{"recipes.product_swap", "POST", "/v1/recipes/product_swap", "", "json", "uri,mode,prompt,version,audio,duration,firstFrame,ratio,shots,view,referenceImage,targetLanguage,outputCount,quality,productImages,productInfo,styleImages,userConcept,image,newProductImages,originalProductImage,referenceVideo,resolution,characterImage,productImage", "", "paid"},
|
||||
{"recipes.product_ugc", "POST", "/v1/recipes/product_ugc", "", "json", "uri,mode,prompt,version,audio,duration,firstFrame,ratio,shots,view,referenceImage,targetLanguage,outputCount,quality,productImages,productInfo,styleImages,userConcept,image,newProductImages,originalProductImage,referenceVideo,resolution,characterImage,productImage", "", "paid"},
|
||||
{"tasks.get", "GET", "/v1/tasks/{id}", "", "none", "", "", "read"},
|
||||
{"tasks.delete", "DELETE", "/v1/tasks/{id}", "", "none", "", "", "delete"},
|
||||
{"organization.get", "GET", "/v1/organization", "", "none", "", "", "read"},
|
||||
{"organization.usage", "POST", "/v1/organization/usage", "", "json", "beforeDate,startDate", "", "read"},
|
||||
{"uploads.create", "POST", "/v1/uploads", "", "json", "file,fileMetadata", "", "write"},
|
||||
{"avatars.create", "POST", "/v1/avatars", "", "json", "presetId,type,id,name,personality,referenceImage,voice,documentIds,imageProcessing,startScript,endDate,startDate", "", "write"},
|
||||
{"avatars.list", "GET", "/v1/avatars", "", "none", "", "", "read"},
|
||||
{"avatars.get", "GET", "/v1/avatars/{id}", "", "none", "", "", "read"},
|
||||
{"avatars.update", "PATCH", "/v1/avatars/{id}", "", "json", "presetId,type,id,name,personality,referenceImage,voice,documentIds,imageProcessing,startScript,endDate,startDate", "", "write"},
|
||||
{"avatars.delete", "DELETE", "/v1/avatars/{id}", "", "none", "", "", "delete"},
|
||||
{"avatar_usage", "GET", "/v1/avatar_usage", "", "none", "", "", "read"},
|
||||
{"avatar_conversations.list", "GET", "/v1/avatar_conversations", "", "none", "", "", "read"},
|
||||
{"avatar_conversations.get", "GET", "/v1/avatar_conversations/{id}", "", "none", "", "", "read"},
|
||||
{"avatar_conversations.delete", "DELETE", "/v1/avatar_conversations/{id}", "", "none", "", "", "delete"},
|
||||
{"documents.create", "POST", "/v1/documents", "", "json", "content,name", "", "write"},
|
||||
{"documents.list", "GET", "/v1/documents", "", "none", "", "", "read"},
|
||||
{"documents.get", "GET", "/v1/documents/{id}", "", "none", "", "", "read"},
|
||||
{"documents.update", "PATCH", "/v1/documents/{id}", "", "json", "content,name", "", "write"},
|
||||
{"documents.delete", "DELETE", "/v1/documents/{id}", "", "none", "", "", "delete"},
|
||||
{"voices.create", "POST", "/v1/voices", "", "json", "audio,type,model,prompt,from,name,description", "", "write"},
|
||||
{"voices.preview", "POST", "/v1/voices/preview", "", "json", "audio,type,model,prompt,from,name,description", "", "paid"},
|
||||
{"voices.list", "GET", "/v1/voices", "", "none", "", "", "read"},
|
||||
{"voices.get", "GET", "/v1/voices/{id}", "", "none", "", "", "read"},
|
||||
{"voices.update", "PATCH", "/v1/voices/{id}", "", "json", "audio,type,model,prompt,from,name,description", "", "write"},
|
||||
{"voices.delete", "DELETE", "/v1/voices/{id}", "", "none", "", "", "delete"},
|
||||
{"realtime_sessions.create", "POST", "/v1/realtime_sessions", "", "json", "presetId,type,avatarId,signedUrl,token,roomName,url,agentIdentity,description,name,parameters,timeoutSeconds,avatar,model,integration,livekit,maxDuration,personality,startScript,tools", "", "paid"},
|
||||
{"realtime_sessions.get", "GET", "/v1/realtime_sessions/{id}", "", "none", "", "", "read"},
|
||||
{"realtime_sessions.delete", "DELETE", "/v1/realtime_sessions/{id}", "", "none", "", "", "delete"},
|
||||
{"routers.create", "POST", "/v1/routers", "", "json", "fallback,maxCreditsPerGeneration,models,optimizeFor,schemaVersion,slug,description,name,settings", "", "write"},
|
||||
{"routers.list", "GET", "/v1/routers", "", "none", "", "", "read"},
|
||||
{"routers.get", "GET", "/v1/routers/{id}", "", "none", "", "", "read"},
|
||||
{"routers.update", "PATCH", "/v1/routers/{id}", "", "json", "fallback,maxCreditsPerGeneration,models,optimizeFor,schemaVersion,slug,description,name,settings", "", "write"},
|
||||
{"routers.delete", "DELETE", "/v1/routers/{id}", "", "none", "", "", "delete"},
|
||||
{"routers.requests", "GET", "/v1/routers/{id}/requests", "", "none", "", "", "read"},
|
||||
{"workflows.list", "GET", "/v1/workflows", "", "none", "", "", "read"},
|
||||
{"workflows.get", "GET", "/v1/workflows/{id}", "", "none", "", "", "read"},
|
||||
{"workflows.invoke", "POST", "/v1/workflows/{id}", "", "json", "type,value,uri,nodeOutputs", "", "paid"},
|
||||
{"workflow_invocations.get", "GET", "/v1/workflow_invocations/{id}", "", "none", "", "", "read"},
|
||||
{"audit_logs.list", "GET", "/v1/organization/webapp/audit_logs", "", "none", "", "", "read"},
|
||||
{"audit_logs.get", "GET", "/v1/organization/webapp/audit_logs/{id}", "", "none", "", "", "read"},
|
||||
{"webapp_usage.list", "GET", "/v1/organization/webapp/usage", "", "none", "", "", "read"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
{"TaskRetrieveResponse", "id,status,createdAt,output,failure,failureCode,progress"},
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& RunwayCalls::table() {
|
||||
static const CallTable t("runway", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t RunwayCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* RunwayCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct RunwayCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+163
@@ -0,0 +1,163 @@
|
||||
#include "vendors/media/runway/runway.h"
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
#include "vendors/media/common/output_scanner.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs with_body(const Json& body) {
|
||||
CallArgs a;
|
||||
a.body = body;
|
||||
return a;
|
||||
}
|
||||
|
||||
CallArgs by_id(const std::string& id) {
|
||||
CallArgs a;
|
||||
a.path["id"] = id;
|
||||
return a;
|
||||
}
|
||||
|
||||
std::string ratio_of(const std::string& aspect, int width, int height) {
|
||||
if (!aspect.empty()) return aspect;
|
||||
if (width && height) return std::to_string(width) + ":" + std::to_string(height);
|
||||
return "";
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
PreparedCall RunwayClient::plan(const std::string& key, const Json& body) const { return prepare(key, with_body(body)); }
|
||||
Json RunwayClient::submit(const std::string& key, const Json& body, bool confirm) {
|
||||
return call_json(key, with_body(body), confirm);
|
||||
}
|
||||
Json RunwayClient::task(const std::string& id) { return call_json("tasks.get", by_id(id)); }
|
||||
Json RunwayClient::cancel_or_delete_task(const std::string& id, bool confirm) {
|
||||
return call_json("tasks.delete", by_id(id), confirm);
|
||||
}
|
||||
Json RunwayClient::organization() { return call_json("organization.get"); }
|
||||
|
||||
Json RunwayClient::organization_usage(const Json& body) {
|
||||
PreparedCall c = prepare("organization.usage", with_body(body));
|
||||
c.effect = Effect::kRead; // a usage query: the POST creates nothing and costs nothing
|
||||
return parse_json("runway", execute(c, false));
|
||||
}
|
||||
|
||||
JobTicket RunwayJobs::ticket_from(const Json& reply, const std::string& model, MediaKind kind) {
|
||||
JobTicket t;
|
||||
t.provider = "runway";
|
||||
t.model = model;
|
||||
t.kind = kind;
|
||||
t.id = AdapterSupport::str(reply, "id");
|
||||
if (t.id.empty()) throw llm::ApiError("runway", 502, "BAD_RESPONSE", "no id in the task reply");
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus RunwayJobs::status_of(const Json& task, MediaKind kind) {
|
||||
JobStatus s;
|
||||
s.raw_state = AdapterSupport::str(task, "status");
|
||||
if (s.raw_state == "SUCCEEDED") {
|
||||
s.state = JobState::kSucceeded;
|
||||
if (const Json* o = task.find("output")) s.assets = OutputScanner::scan(*o, kind);
|
||||
if (s.assets.empty()) {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = "succeeded without output URLs";
|
||||
}
|
||||
} else if (s.raw_state == "FAILED") {
|
||||
s.state = JobState::kFailed;
|
||||
s.error = AdapterSupport::str(task, "failure");
|
||||
std::string code = AdapterSupport::str(task, "failureCode");
|
||||
if (!code.empty()) s.error += " (" + code + ")";
|
||||
} else if (s.raw_state == "CANCELLED") {
|
||||
s.state = JobState::kCancelled;
|
||||
} else if (s.raw_state == "RUNNING") {
|
||||
s.state = JobState::kRunning;
|
||||
if (const Json* p = task.find("progress"); p && p->is_number()) s.progress = p->as_number();
|
||||
} else {
|
||||
s.state = JobState::kQueued; // PENDING, THROTTLED
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
JobStatus RunwayJobs::poll(RunwayClient& client, const JobTicket& ticket) {
|
||||
return status_of(client.task(ticket.id), ticket.kind);
|
||||
}
|
||||
|
||||
Json RunwayImageGenerator::body(const ImageRequest& r) const {
|
||||
r.validate();
|
||||
if (r.count != 1) throw llm::ApiError::invalid("runway", "one image per task (count must be 1)");
|
||||
Json b = Json::object();
|
||||
b.set("model", Json(r.model.empty() ? "gen4_image" : r.model));
|
||||
AdapterSupport::put(b, "promptText", r.prompt);
|
||||
AdapterSupport::put(b, "ratio", ratio_of(r.aspect_ratio, r.width, r.height));
|
||||
AdapterSupport::put_seed(b, "seed", r.seed);
|
||||
if (!r.source_url.empty()) {
|
||||
Json refs = Json::array();
|
||||
Json one = Json::object();
|
||||
one.set("uri", Json(r.source_url));
|
||||
one.set("tag", Json("source"));
|
||||
refs.push(one);
|
||||
b.set("referenceImages", refs);
|
||||
}
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json RunwayImageGenerator::plan(const ImageRequest& request) const {
|
||||
return client_->plan("text_to_image", body(request)).summary();
|
||||
}
|
||||
|
||||
JobTicket RunwayImageGenerator::submit(const ImageRequest& request, bool confirm) {
|
||||
Json b = body(request);
|
||||
AdapterSupport::need_confirm(confirm, "runway", "generate");
|
||||
return RunwayJobs::ticket_from(client_->submit("text_to_image", b, true), b.find("model")->as_string(), MediaKind::kImage);
|
||||
}
|
||||
|
||||
JobStatus RunwayImageGenerator::poll(const JobTicket& t) { return RunwayJobs::poll(*client_, t); }
|
||||
|
||||
bool RunwayImageGenerator::cancel(const JobTicket& t, bool confirm) {
|
||||
AdapterSupport::need_confirm(confirm, "runway", "cancel task");
|
||||
client_->cancel_or_delete_task(t.id, true);
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string RunwayVideoGenerator::key_for(const VideoRequest& r) {
|
||||
return r.image_url.empty() ? "text_to_video" : "image_to_video";
|
||||
}
|
||||
|
||||
Json RunwayVideoGenerator::body(const VideoRequest& r) const {
|
||||
r.validate();
|
||||
Json b = Json::object();
|
||||
b.set("model", Json(r.model.empty() ? (r.image_url.empty() ? "gen4.5" : "gen4_turbo") : r.model));
|
||||
AdapterSupport::put(b, "promptText", r.prompt);
|
||||
AdapterSupport::put(b, "ratio", r.aspect_ratio);
|
||||
AdapterSupport::put_int(b, "duration", r.duration_seconds);
|
||||
AdapterSupport::put_seed(b, "seed", r.seed);
|
||||
AdapterSupport::put(b, "negativePrompt", r.negative_prompt);
|
||||
AdapterSupport::put(b, "resolution", r.resolution);
|
||||
AdapterSupport::put(b, "promptImage", r.image_url);
|
||||
if (!r.end_image_url.empty())
|
||||
throw llm::ApiError::invalid("runway", "end frames use per-model promptImage arrays: pass them in extra");
|
||||
AdapterSupport::merge_extra(b, r.extra);
|
||||
return b;
|
||||
}
|
||||
|
||||
Json RunwayVideoGenerator::plan(const VideoRequest& request) const {
|
||||
return client_->plan(key_for(request), body(request)).summary();
|
||||
}
|
||||
|
||||
JobTicket RunwayVideoGenerator::submit(const VideoRequest& request, bool confirm) {
|
||||
Json b = body(request);
|
||||
AdapterSupport::need_confirm(confirm, "runway", "generate");
|
||||
return RunwayJobs::ticket_from(client_->submit(key_for(request), b, true), b.find("model")->as_string(), MediaKind::kVideo);
|
||||
}
|
||||
|
||||
JobStatus RunwayVideoGenerator::poll(const JobTicket& t) { return RunwayJobs::poll(*client_, t); }
|
||||
|
||||
bool RunwayVideoGenerator::cancel(const JobTicket& t, bool confirm) {
|
||||
AdapterSupport::need_confirm(confirm, "runway", "cancel task");
|
||||
client_->cancel_or_delete_task(t.id, true);
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+93
@@ -0,0 +1,93 @@
|
||||
#pragma once
|
||||
|
||||
// Runway developer API (https://api.dev.runwayml.com): text / image / video
|
||||
// generation, tasks, uploads, organization, avatars, voices, documents,
|
||||
// routers, workflows, realtime sessions. Bearer auth plus the required
|
||||
// X-Runway-Version header. The key is RUNWAY_API_KEY (the official SDK reads
|
||||
// RUNWAYML_API_SECRET; both are accepted). Runway publishes no OpenAPI
|
||||
// document: the table is checked against the official SDK source.
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/runway/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class RunwayClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api.dev.runwayml.com";
|
||||
static constexpr const char* kKeyEnv = "RUNWAY_API_KEY";
|
||||
static constexpr const char* kKeyEnvSdk = "RUNWAYML_API_SECRET";
|
||||
static constexpr const char* kApiVersion = "2024-11-06"; // SDK default X-Runway-Version
|
||||
|
||||
explicit RunwayClient(std::shared_ptr<llm::ApiClient> client)
|
||||
: MediaService(std::move(client), RunwayCalls::table()) {}
|
||||
static llm::ApiClientConfig config() {
|
||||
llm::ApiClientConfig c = ClientFactory::bearer("runway", kHost, kKeyEnv);
|
||||
c.default_headers["X-Runway-Version"] = kApiVersion;
|
||||
return c;
|
||||
}
|
||||
static std::shared_ptr<RunwayClient> from_env() {
|
||||
return std::make_shared<RunwayClient>(ClientFactory::real(config(), ClientFactory::env({kKeyEnv, kKeyEnvSdk})));
|
||||
}
|
||||
static std::shared_ptr<RunwayClient> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<RunwayClient>(ClientFactory::with(std::move(t), config(), key));
|
||||
}
|
||||
|
||||
// Paid generation rows (text_to_image, text_to_video, image_to_video, ...). Returns {id, ...}.
|
||||
Json submit(const std::string& key, const Json& body, bool confirm);
|
||||
PreparedCall plan(const std::string& key, const Json& body) const;
|
||||
Json task(const std::string& id);
|
||||
Json cancel_or_delete_task(const std::string& id, bool confirm); // no default
|
||||
Json organization();
|
||||
Json organization_usage(const Json& body); // POST but a query: effect read
|
||||
};
|
||||
|
||||
class RunwayImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit RunwayImageGenerator(std::shared_ptr<RunwayClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "runway"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"gen4_image", "gen4_image_turbo", "gemini_2.5_flash", "ideogram_4_5", "gpt_image_2", "seedream5_pro"};
|
||||
}
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override;
|
||||
int suggested_poll_ms() const override { return 3000; }
|
||||
|
||||
private:
|
||||
Json body(const ImageRequest& request) const;
|
||||
std::shared_ptr<RunwayClient> client_;
|
||||
};
|
||||
|
||||
class RunwayVideoGenerator : public VideoGenerator {
|
||||
public:
|
||||
explicit RunwayVideoGenerator(std::shared_ptr<RunwayClient> c) : client_(std::move(c)) {}
|
||||
std::string provider() const override { return "runway"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override {
|
||||
return {"gen4.5", "gen4_turbo", "veo3.1", "veo3.1_fast", "kling3.0_pro", "seedance2", "hailuo3"};
|
||||
}
|
||||
Json plan(const VideoRequest& request) const override;
|
||||
JobTicket submit(const VideoRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket& ticket, bool confirm) override;
|
||||
int suggested_poll_ms() const override { return 5000; }
|
||||
|
||||
static std::string key_for(const VideoRequest& request);
|
||||
|
||||
private:
|
||||
Json body(const VideoRequest& request) const;
|
||||
std::shared_ptr<RunwayClient> client_;
|
||||
};
|
||||
|
||||
struct RunwayJobs {
|
||||
static JobStatus poll(RunwayClient& client, const JobTicket& ticket);
|
||||
static JobStatus status_of(const Json& task, MediaKind kind);
|
||||
static JobTicket ticket_from(const Json& reply, const std::string& model, MediaKind kind);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
#include "vendors/media/stability/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line, strings in order:
|
||||
// key, http method, path, spec operation id, body kind, body fields, query params, effect)
|
||||
static const CallRow kRows[] = {
|
||||
{"3d.stable-fast-3d", "POST", "/v2beta/3d/stable-fast-3d", "", "multipart", "image,texture_resolution,foreground_ratio,remesh,vertex_count", "", "paid"},
|
||||
{"3d.stable-point-aware-3d", "POST", "/v2beta/3d/stable-point-aware-3d", "", "multipart", "image,texture_resolution,foreground_ratio,remesh,target_type,target_count,guidance_scale,seed", "", "paid"},
|
||||
{"results.id", "GET", "/v2beta/results/{id}", "", "none", "", "", "read"},
|
||||
{"audio.stable-audio-2.text-to-audio", "POST", "/v2beta/audio/stable-audio-2/text-to-audio", "", "multipart", "prompt,duration,seed,steps,cfg_scale,model,output_format", "", "paid"},
|
||||
{"audio.stable-audio-2.audio-to-audio", "POST", "/v2beta/audio/stable-audio-2/audio-to-audio", "", "multipart", "prompt,duration,seed,steps,cfg_scale,model,output_format,audio,strength", "", "paid"},
|
||||
{"audio.stable-audio-2.inpaint", "POST", "/v2beta/audio/stable-audio-2/inpaint", "", "multipart", "prompt,duration,seed,steps,output_format,audio,mask_start,mask_end", "", "paid"},
|
||||
{"audio.stable-audio.text-to-audio", "POST", "/v2beta/audio/stable-audio/text-to-audio", "", "multipart", "prompt,model,duration,seed,steps,cfg_scale,output_format", "", "paid"},
|
||||
{"audio.stable-audio.audio-to-audio", "POST", "/v2beta/audio/stable-audio/audio-to-audio", "", "multipart", "prompt,model,duration,seed,steps,cfg_scale,output_format,strength,audio", "", "paid"},
|
||||
{"audio.stable-audio.inpaint", "POST", "/v2beta/audio/stable-audio/inpaint", "", "multipart", "prompt,model,duration,seed,steps,cfg_scale,output_format,mask_start,mask_end,audio", "", "paid"},
|
||||
{"audio.results.id", "GET", "/v2beta/audio/results/{id}", "fetchAudioResult", "none", "", "", "read"},
|
||||
{"stable-image.upscale.conservative", "POST", "/v2beta/stable-image/upscale/conservative", "", "multipart", "image,prompt,negative_prompt,seed,output_format,creativity", "", "paid"},
|
||||
{"stable-image.upscale.creative", "POST", "/v2beta/stable-image/upscale/creative", "", "multipart", "image,prompt,negative_prompt,output_format,seed,creativity,style_preset", "", "paid"},
|
||||
{"stable-image.upscale.creative.result.id", "GET", "/v2beta/stable-image/upscale/creative/result/{id}", "", "none", "", "", "read"},
|
||||
{"stable-image.upscale.fast", "POST", "/v2beta/stable-image/upscale/fast", "", "multipart", "image,output_format", "", "paid"},
|
||||
{"stable-image.edit.erase", "POST", "/v2beta/stable-image/edit/erase", "", "multipart", "image,mask,grow_mask,seed,output_format", "", "paid"},
|
||||
{"stable-image.edit.inpaint", "POST", "/v2beta/stable-image/edit/inpaint", "", "multipart", "image,prompt,negative_prompt,mask,grow_mask,seed,output_format,style_preset", "", "paid"},
|
||||
{"stable-image.edit.outpaint", "POST", "/v2beta/stable-image/edit/outpaint", "", "multipart", "image,left,right,up,down,creativity,prompt,seed,output_format,style_preset", "", "paid"},
|
||||
{"stable-image.edit.search-and-replace", "POST", "/v2beta/stable-image/edit/search-and-replace", "", "multipart", "image,prompt,search_prompt,negative_prompt,grow_mask,seed,output_format,style_preset", "", "paid"},
|
||||
{"stable-image.edit.search-and-recolor", "POST", "/v2beta/stable-image/edit/search-and-recolor", "", "multipart", "image,prompt,select_prompt,negative_prompt,grow_mask,seed,output_format,style_preset", "", "paid"},
|
||||
{"stable-image.edit.remove-background", "POST", "/v2beta/stable-image/edit/remove-background", "", "multipart", "image,output_format", "", "paid"},
|
||||
{"stable-image.edit.replace-background-and-relight", "POST", "/v2beta/stable-image/edit/replace-background-and-relight", "", "multipart", "subject_image,background_reference,background_prompt,foreground_prompt,negative_prompt,preserve_original_subject,original_background_depth,keep_original_background,light_source_direction,light_reference,light_source_strength,seed,output_format", "", "paid"},
|
||||
{"stable-image.generate.ultra", "POST", "/v2beta/stable-image/generate/ultra", "", "multipart", "prompt,negative_prompt,aspect_ratio,seed,output_format,image,style_preset,strength", "", "paid"},
|
||||
{"stable-image.generate.core", "POST", "/v2beta/stable-image/generate/core", "", "multipart", "prompt,aspect_ratio,negative_prompt,seed,style_preset,output_format", "", "paid"},
|
||||
{"stable-image.generate.sd3", "POST", "/v2beta/stable-image/generate/sd3", "", "multipart", "prompt,mode,image,strength,aspect_ratio,model,seed,output_format,style_preset,negative_prompt,cfg_scale", "", "paid"},
|
||||
{"stable-image.control.sketch", "POST", "/v2beta/stable-image/control/sketch", "", "multipart", "prompt,image,control_strength,negative_prompt,seed,output_format,style_preset", "", "paid"},
|
||||
{"stable-image.control.structure", "POST", "/v2beta/stable-image/control/structure", "", "multipart", "prompt,image,control_strength,negative_prompt,seed,output_format,style_preset", "", "paid"},
|
||||
{"stable-image.control.style", "POST", "/v2beta/stable-image/control/style", "", "multipart", "prompt,image,negative_prompt,aspect_ratio,fidelity,seed,output_format,style_preset", "", "paid"},
|
||||
{"stable-image.control.style-transfer", "POST", "/v2beta/stable-image/control/style-transfer", "", "multipart", "init_image,style_image,prompt,negative_prompt,seed,style_strength,composition_fidelity,change_strength,output_format", "", "paid"},
|
||||
{"v1.generation.id.text-to-image", "POST", "/v1/generation/{engine_id}/text-to-image", "textToImage", "json", "height,width,text_prompts,cfg_scale,clip_guidance_preset,sampler,samples,seed,steps,style_preset,extras", "", "paid"},
|
||||
{"v1.generation.id.image-to-image", "POST", "/v1/generation/{engine_id}/image-to-image", "imageToImage", "multipart", "text_prompts,init_image,init_image_mode,image_strength,step_schedule_start,step_schedule_end,cfg_scale,clip_guidance_preset,sampler,samples,seed,steps,style_preset,extras", "", "paid"},
|
||||
{"v1.generation.id.image-to-image.masking", "POST", "/v1/generation/{engine_id}/image-to-image/masking", "masking", "multipart", "init_image,mask_source,mask_image,text_prompts,cfg_scale,clip_guidance_preset,sampler,samples,seed,steps,style_preset,extras", "", "paid"},
|
||||
{"v1.engines.list", "GET", "/v1/engines/list", "listEngines", "none", "", "", "read"},
|
||||
{"v1.user.account", "GET", "/v1/user/account", "userAccount", "none", "", "", "read"},
|
||||
{"v1.user.balance", "GET", "/v1/user/balance", "userBalance", "none", "", "", "read"},
|
||||
};
|
||||
// CALLS-END
|
||||
|
||||
// FIELDS-BEGIN (parsed by tools/check_spec.py; one row per line: schema, fields we read)
|
||||
static const char* const kFields[][2] = {
|
||||
{"AsyncGenerationResponse", "id"},
|
||||
{"ImageJSONResponse", "image,finish_reason,seed"},
|
||||
{"AudioJSONResponse", "audio,finish_reason,seed"},
|
||||
};
|
||||
// FIELDS-END
|
||||
|
||||
const CallTable& StabilityCalls::table() {
|
||||
static const CallTable t("stability", kRows, sizeof(kRows) / sizeof(kRows[0]));
|
||||
return t;
|
||||
}
|
||||
|
||||
std::size_t StabilityCalls::field_count() { return sizeof(kFields) / sizeof(kFields[0]); }
|
||||
const char* const* StabilityCalls::field_row(std::size_t i) { return kFields[i]; }
|
||||
|
||||
} // namespace molo::media
|
||||
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls for this provider. tools/check_spec.py checks
|
||||
// every row against the published spec. Code builds every path from here.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "vendors/media/common/call_table.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
struct StabilityCalls {
|
||||
static const CallTable& table();
|
||||
// Response fields our parsers read, per schema (checked by check_spec.py).
|
||||
static std::size_t field_count();
|
||||
static const char* const* field_row(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+141
@@ -0,0 +1,141 @@
|
||||
#include "vendors/media/stability/stability.h"
|
||||
|
||||
#include "vendors/media/common/adapter_support.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
namespace {
|
||||
|
||||
CallArgs form(const Json& fields, const std::vector<FilePart>& files) {
|
||||
CallArgs a;
|
||||
a.body = fields;
|
||||
a.files = files;
|
||||
a.headers["Accept"] = "application/json";
|
||||
return a;
|
||||
}
|
||||
|
||||
CallArgs by_id(const std::string& id) {
|
||||
CallArgs a;
|
||||
a.path["id"] = id;
|
||||
a.headers["Accept"] = "application/json";
|
||||
return a;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
PreparedCall StabilityClient::plan(const std::string& key, const Json& fields,
|
||||
const std::vector<FilePart>& files) const {
|
||||
return prepare(key, form(fields, files));
|
||||
}
|
||||
|
||||
Json StabilityClient::submit(const std::string& key, const Json& fields, const std::vector<FilePart>& files,
|
||||
bool confirm) {
|
||||
return call_json(key, form(fields, files), confirm);
|
||||
}
|
||||
|
||||
Json StabilityClient::result(const std::string& id) { return call_json("results.id", by_id(id)); }
|
||||
Json StabilityClient::audio_result(const std::string& id) { return call_json("audio.results.id", by_id(id)); }
|
||||
Json StabilityClient::creative_upscale_result(const std::string& id) {
|
||||
return call_json("stable-image.upscale.creative.result.id", by_id(id));
|
||||
}
|
||||
Json StabilityClient::engines() { return call_json("v1.engines.list"); }
|
||||
Json StabilityClient::account() { return call_json("v1.user.account"); }
|
||||
Json StabilityClient::balance() { return call_json("v1.user.balance"); }
|
||||
|
||||
Json StabilityClient::v1_text_to_image(const std::string& engine_id, const Json& body, bool confirm) {
|
||||
CallArgs a;
|
||||
a.path["engine_id"] = engine_id;
|
||||
a.body = body;
|
||||
a.headers["Accept"] = "application/json";
|
||||
return call_json("v1.generation.id.text-to-image", a, confirm);
|
||||
}
|
||||
|
||||
std::vector<std::string> StabilityImageGenerator::models() const {
|
||||
return {"core", "ultra", "sd3.5-large", "sd3.5-large-turbo", "sd3.5-medium", "sd3.5-flash"};
|
||||
}
|
||||
|
||||
std::string StabilityImageGenerator::key_for(const std::string& model) {
|
||||
if (model.empty() || model == "core") return "stable-image.generate.core";
|
||||
if (model == "ultra") return "stable-image.generate.ultra";
|
||||
if (model.rfind("sd3", 0) == 0) return "stable-image.generate.sd3";
|
||||
throw llm::ApiError::invalid("stability", "unknown model '" + model + "' (core, ultra, sd3*)");
|
||||
}
|
||||
|
||||
Json StabilityImageGenerator::fields(const ImageRequest& r, std::string& key) const {
|
||||
r.validate();
|
||||
key = key_for(r.model);
|
||||
if (r.count != 1) throw llm::ApiError::invalid("stability", "one image per call (count must be 1)");
|
||||
if (r.width || r.height) throw llm::ApiError::invalid("stability", "use aspect_ratio, not width / height");
|
||||
if (!r.source_url.empty())
|
||||
throw llm::ApiError::invalid("stability", "source images are uploaded as files: use StabilityClient::submit with a FilePart");
|
||||
Json f = Json::object();
|
||||
AdapterSupport::put(f, "prompt", r.prompt);
|
||||
AdapterSupport::put(f, "negative_prompt", r.negative_prompt);
|
||||
AdapterSupport::put(f, "aspect_ratio", r.aspect_ratio);
|
||||
if (r.seed >= 0) f.set("seed", Json(static_cast<double>(r.seed > 4294967294L ? 4294967294L : r.seed)));
|
||||
if (key == "stable-image.generate.sd3" && r.model != "sd3") f.set("model", Json(r.model));
|
||||
f.set("output_format", Json("png"));
|
||||
AdapterSupport::merge_extra(f, r.extra);
|
||||
return f;
|
||||
}
|
||||
|
||||
Json StabilityImageGenerator::plan(const ImageRequest& request) const {
|
||||
std::string key;
|
||||
Json f = fields(request, key);
|
||||
return client_->plan(key, f, {}).summary();
|
||||
}
|
||||
|
||||
JobTicket StabilityImageGenerator::submit(const ImageRequest& request, bool confirm) {
|
||||
std::string key;
|
||||
Json f = fields(request, key);
|
||||
AdapterSupport::need_confirm(confirm, "stability", "generate");
|
||||
Json out = client_->submit(key, f, {}, true);
|
||||
JobTicket t;
|
||||
t.provider = "stability";
|
||||
t.model = request.model.empty() ? "core" : request.model;
|
||||
t.kind = MediaKind::kImage;
|
||||
if (const Json* id = out.find("id"); id && id->is_string()) {
|
||||
t.id = id->as_string();
|
||||
return t;
|
||||
}
|
||||
std::string b64 = AdapterSupport::str(out, "image");
|
||||
if (b64.empty()) throw llm::ApiError("stability", 502, "BAD_RESPONSE", "no image and no id in the reply");
|
||||
MediaAsset a;
|
||||
a.kind = MediaKind::kImage;
|
||||
a.data_base64 = b64;
|
||||
a.mime = "image/" + (f.find("output_format") ? f.find("output_format")->as_string() : std::string("png"));
|
||||
a.label = "seed=" + AdapterSupport::str(out, "seed") + " finish_reason=" + AdapterSupport::str(out, "finish_reason");
|
||||
t.id = "immediate";
|
||||
t.immediate = true;
|
||||
t.assets.push_back(a);
|
||||
return t;
|
||||
}
|
||||
|
||||
JobStatus StabilityImageGenerator::poll(const JobTicket& ticket) {
|
||||
JobStatus s;
|
||||
if (ticket.immediate) {
|
||||
s.state = JobState::kSucceeded;
|
||||
s.raw_state = "immediate";
|
||||
s.assets = ticket.assets;
|
||||
return s;
|
||||
}
|
||||
Json out = client_->result(ticket.id);
|
||||
std::string b64 = AdapterSupport::str(out, "image");
|
||||
if (b64.empty()) {
|
||||
s.state = JobState::kRunning;
|
||||
s.raw_state = AdapterSupport::str(out, "status");
|
||||
if (s.raw_state.empty()) s.raw_state = "in-progress";
|
||||
return s;
|
||||
}
|
||||
MediaAsset a;
|
||||
a.kind = MediaKind::kImage;
|
||||
a.data_base64 = b64;
|
||||
a.mime = "image/png";
|
||||
a.label = "seed=" + AdapterSupport::str(out, "seed") + " finish_reason=" + AdapterSupport::str(out, "finish_reason");
|
||||
s.state = JobState::kSucceeded;
|
||||
s.raw_state = "complete";
|
||||
s.assets.push_back(a);
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace molo::media
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
#pragma once
|
||||
|
||||
// Stability AI: v2beta Stable Image (generate, edit, upscale, control), 3D,
|
||||
// Stable Audio, and the v1 engines / account surface. Every v2beta call is
|
||||
// multipart/form-data; we always ask for JSON (base64 images) so errors and
|
||||
// results parse the same way.
|
||||
|
||||
#include "vendors/media/common/client_factory.h"
|
||||
#include "vendors/media/common/job.h"
|
||||
#include "vendors/media/common/media_service.h"
|
||||
#include "vendors/media/stability/calls.h"
|
||||
|
||||
namespace molo::media {
|
||||
|
||||
class StabilityClient : public MediaService {
|
||||
public:
|
||||
static constexpr const char* kHost = "https://api.stability.ai";
|
||||
static constexpr const char* kKeyEnv = "STABILITY_API_KEY";
|
||||
|
||||
explicit StabilityClient(std::shared_ptr<llm::ApiClient> client)
|
||||
: MediaService(std::move(client), StabilityCalls::table()) {}
|
||||
static llm::ApiClientConfig config() { return ClientFactory::bearer("stability", kHost, kKeyEnv); }
|
||||
static std::shared_ptr<StabilityClient> from_env() {
|
||||
return std::make_shared<StabilityClient>(ClientFactory::real(config(), ClientFactory::env({kKeyEnv})));
|
||||
}
|
||||
static std::shared_ptr<StabilityClient> with(std::unique_ptr<clients::Transport> t, const std::string& key) {
|
||||
return std::make_shared<StabilityClient>(ClientFactory::with(std::move(t), config(), key));
|
||||
}
|
||||
|
||||
// Any v2beta POST row (stable-image.*, 3d.*, audio.*). Paid: confirm required.
|
||||
// Returns {image|audio, finish_reason, seed} or, for async rows, {id}.
|
||||
Json submit(const std::string& key, const Json& fields, const std::vector<FilePart>& files, bool confirm);
|
||||
PreparedCall plan(const std::string& key, const Json& fields, const std::vector<FilePart>& files) const;
|
||||
|
||||
// Polls: HTTP 202 {id,status} while running, 200 with the result when done.
|
||||
Json result(const std::string& id); // /v2beta/results/{id}
|
||||
Json audio_result(const std::string& id); // /v2beta/audio/results/{id}
|
||||
Json creative_upscale_result(const std::string& id); // legacy creative upscale path
|
||||
|
||||
// v1 reads.
|
||||
Json engines();
|
||||
Json account();
|
||||
Json balance();
|
||||
// v1 text-to-image (JSON body, artifacts[] with base64).
|
||||
Json v1_text_to_image(const std::string& engine_id, const Json& body, bool confirm);
|
||||
};
|
||||
|
||||
class StabilityImageGenerator : public ImageGenerator {
|
||||
public:
|
||||
explicit StabilityImageGenerator(std::shared_ptr<StabilityClient> client) : client_(std::move(client)) {}
|
||||
std::string provider() const override { return "stability"; }
|
||||
bool configured() const override { return client_->configured(); }
|
||||
std::vector<std::string> models() const override;
|
||||
Json plan(const ImageRequest& request) const override;
|
||||
JobTicket submit(const ImageRequest& request, bool confirm) override;
|
||||
JobStatus poll(const JobTicket& ticket) override;
|
||||
bool cancel(const JobTicket&, bool) override { return false; } // no cancel in the API
|
||||
|
||||
// model -> table key (ultra, core, sd3*), "" model = core.
|
||||
static std::string key_for(const std::string& model);
|
||||
|
||||
private:
|
||||
Json fields(const ImageRequest& request, std::string& key) const;
|
||||
std::shared_ptr<StabilityClient> client_;
|
||||
};
|
||||
|
||||
} // namespace molo::media
|
||||
+578
@@ -0,0 +1,578 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check the media call tables against what each vendor publishes.
|
||||
|
||||
For every provider the checker reads the rows between CALLS-BEGIN / CALLS-END
|
||||
in src/vendors/media/<provider>/calls.cpp (and the response fields between
|
||||
FIELDS-BEGIN / FIELDS-END) and compares them with the vendor's own source:
|
||||
|
||||
openapi Stability (v2beta + v1), Black Forest Labs, Replicate, fal (queue
|
||||
spec from the docs, Platform API), Ideogram v2, Recraft, Higgsfield
|
||||
sdk Runway and Luma: the official npm package (resources/*.js for the
|
||||
paths, resources/*.d.ts for the request fields)
|
||||
docs Leonardo v2 (the OpenAPI document embedded in three reference
|
||||
pages), MiniMax (the OpenAPI embedded in the API reference pages)
|
||||
probe Kling (no machine-readable source exists): the path must answer
|
||||
401/400/403 unauthenticated, never 404. Reported as UNVERIFIED.
|
||||
|
||||
It fails (exit 1) when
|
||||
row a declared (method, path) is not an operation of the source
|
||||
field a request body field or query parameter we send is not in the source
|
||||
kind the declared body kind disagrees with the source content types
|
||||
schema a response field we read is not reachable from the named schema
|
||||
coverage a source operation is neither in the table nor in EXCLUSIONS
|
||||
|
||||
Python 3 standard library only. No credentials. Network required.
|
||||
|
||||
python3 -I src/vendors/media/tools/check_spec.py [provider ...]
|
||||
"""
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import tarfile
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
ROOT = os.path.normpath(os.path.join(HERE, ".."))
|
||||
STR = re.compile(r'"((?:[^"\\]|\\.)*)"')
|
||||
UA = {"User-Agent": "molo-media-spec-check/1"}
|
||||
|
||||
# (provider, source operation) -> reason. Everything else in a source must be in the table.
|
||||
EXCLUSIONS = {
|
||||
"stability": {
|
||||
("POST", "/v2alpha/generation/stable-image/upscale"): "alpha endpoint, superseded by v2beta upscale",
|
||||
("GET", "/v2alpha/generation/stable-image/upscale/result/{id}"): "alpha endpoint",
|
||||
("POST", "/v2alpha/generation/stable-image/inpaint"): "alpha endpoint, superseded by v2beta edit/inpaint",
|
||||
},
|
||||
"fal": {
|
||||
("GET", "/{owner}/{app}/requests/{request_id}/status/stream"): "server-sent events; ApiClient streams POST only (gap)",
|
||||
},
|
||||
}
|
||||
# fal Platform API operations outside model inference: out of scope for package media.
|
||||
FAL_PLATFORM_SKIP_PREFIXES = ("/workflows", "/assets", "/storage", "/serverless", "/compute", "/keys", "/account",
|
||||
"/organization")
|
||||
# Rows whose path is generic by design (the vendor says its OpenAPI file is supplementary).
|
||||
GENERIC_ROWS = {"higgsfield": {"model.submit"}}
|
||||
# Rows checked by documentation prose only (the reference page was removed). Listed, never silent.
|
||||
PROSE_ONLY = {"leonardo": {"v1.generations.create", "v1.generations.get", "v1.me"}}
|
||||
|
||||
problems = []
|
||||
notes = []
|
||||
cache = {}
|
||||
|
||||
|
||||
def fail(provider, kind, msg):
|
||||
problems.append("%s: %s: %s" % (provider, kind, msg))
|
||||
|
||||
|
||||
def fetch(url, binary=False):
|
||||
if url not in cache:
|
||||
req = urllib.request.Request(url, headers=UA)
|
||||
with urllib.request.urlopen(req, timeout=90) as r:
|
||||
cache[url] = r.read()
|
||||
data = cache[url]
|
||||
return data if binary else data.decode("utf-8", "replace")
|
||||
|
||||
|
||||
def rows_of(provider):
|
||||
text = open(os.path.join(ROOT, provider, "calls.cpp")).read()
|
||||
a, b = text.index("// CALLS-BEGIN"), text.index("// CALLS-END")
|
||||
rows = []
|
||||
for line in text[a:b].splitlines()[2:]:
|
||||
s = line.strip()
|
||||
if s.startswith("{") and '"' in s:
|
||||
f = STR.findall(s)
|
||||
assert len(f) == 8, (provider, s)
|
||||
rows.append(dict(zip(("key", "method", "path", "op", "kind", "body", "query", "effect"), f)))
|
||||
c, d = text.index("// FIELDS-BEGIN"), text.index("// FIELDS-END")
|
||||
fields = []
|
||||
for line in text[c:d].splitlines()[1:]:
|
||||
s = line.strip()
|
||||
if s.startswith("{") and '"' in s:
|
||||
fields.append(STR.findall(s))
|
||||
for r in rows:
|
||||
r["body"] = [x for x in r["body"].split(",") if x]
|
||||
r["query"] = [x for x in r["query"].split(",") if x]
|
||||
return rows, fields
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- OpenAPI ----
|
||||
class Oas:
|
||||
def __init__(self, doc):
|
||||
self.d = doc
|
||||
|
||||
def ref(self, node):
|
||||
n = 0
|
||||
while isinstance(node, dict) and "$ref" in node and n < 25:
|
||||
cur = self.d
|
||||
for part in node["$ref"].lstrip("#/").split("/"):
|
||||
cur = cur[part.replace("~1", "/").replace("~0", "~")]
|
||||
node = cur
|
||||
n += 1
|
||||
return node
|
||||
|
||||
def props(self, schema, depth=0):
|
||||
schema = self.ref(schema)
|
||||
out = []
|
||||
if not isinstance(schema, dict) or depth > 6:
|
||||
return out
|
||||
for name in schema.get("properties") or {}:
|
||||
out.append(name)
|
||||
for key in ("allOf", "oneOf", "anyOf"):
|
||||
for sub in schema.get(key) or []:
|
||||
out.extend(self.props(sub, depth + 1))
|
||||
return out
|
||||
|
||||
def reachable(self, schema, depth=0, seen=None):
|
||||
"""Every property name reachable from a schema (for response fields)."""
|
||||
seen = seen if seen is not None else set()
|
||||
out = set()
|
||||
if depth > 6:
|
||||
return out
|
||||
if isinstance(schema, dict) and "$ref" in schema:
|
||||
if schema["$ref"] in seen:
|
||||
return out
|
||||
seen.add(schema["$ref"])
|
||||
schema = self.ref(schema)
|
||||
if not isinstance(schema, dict):
|
||||
return out
|
||||
for name, sub in (schema.get("properties") or {}).items():
|
||||
out.add(name)
|
||||
out |= self.reachable(sub, depth + 1, seen)
|
||||
for key in ("allOf", "oneOf", "anyOf"):
|
||||
for sub in schema.get(key) or []:
|
||||
out |= self.reachable(sub, depth + 1, seen)
|
||||
if "items" in schema:
|
||||
out |= self.reachable(schema["items"], depth + 1, seen)
|
||||
ap = schema.get("additionalProperties")
|
||||
if isinstance(ap, dict):
|
||||
out |= self.reachable(ap, depth + 1, seen)
|
||||
return out
|
||||
|
||||
def ops(self):
|
||||
for path, item in self.d["paths"].items():
|
||||
for m, op in item.items():
|
||||
if m in ("get", "post", "put", "patch", "delete"):
|
||||
yield m.upper(), path, op
|
||||
|
||||
def op(self, method, path):
|
||||
item = self.d["paths"].get(path)
|
||||
return item.get(method.lower()) if item else None
|
||||
|
||||
def params(self, path, op, where):
|
||||
out = []
|
||||
for p in (self.d["paths"][path].get("parameters") or []) + (op.get("parameters") or []):
|
||||
p = self.ref(p)
|
||||
if p.get("in") == where:
|
||||
out.append(p["name"])
|
||||
return out
|
||||
|
||||
def body(self, op):
|
||||
rb = self.ref(op.get("requestBody") or {})
|
||||
content = rb.get("content") or {}
|
||||
kinds, fields = [], []
|
||||
for ct, k in (("application/json", "json"), ("multipart/form-data", "multipart"),
|
||||
("application/x-www-form-urlencoded", "form")):
|
||||
if ct in content:
|
||||
kinds.append(k)
|
||||
fields.extend(self.props(content[ct].get("schema") or {}))
|
||||
return kinds, fields
|
||||
|
||||
|
||||
def check_rows_against_oas(provider, rows, oas, strip="", only=lambda r: True, path_filter=lambda p: True):
|
||||
"""Compare table rows with one OpenAPI document. Returns the (method, path) set covered."""
|
||||
covered = set()
|
||||
for r in rows:
|
||||
if not only(r):
|
||||
continue
|
||||
if r["key"] in GENERIC_ROWS.get(provider, ()):
|
||||
notes.append("%s: row %s is generic by design (not in the spec)" % (provider, r["key"]))
|
||||
continue
|
||||
path = r["path"][len(strip):] if strip and r["path"].startswith(strip) else r["path"]
|
||||
op = oas.op(r["method"], path)
|
||||
if op is None:
|
||||
fail(provider, "row", "%s %s (%s) is not in the spec" % (r["method"], r["path"], r["key"]))
|
||||
continue
|
||||
covered.add((r["method"], path))
|
||||
if r["op"] and op.get("operationId") != r["op"]:
|
||||
fail(provider, "row", "%s declares operation id %s, spec says %s" % (r["key"], r["op"], op.get("operationId")))
|
||||
kinds, fields = oas.body(op)
|
||||
want = r["kind"]
|
||||
have = "none" if not kinds else (kinds[0] if len(kinds) == 1 else "either")
|
||||
if want != have:
|
||||
fail(provider, "kind", "%s declares body kind %s, spec has %s" % (r["key"], want, have))
|
||||
for f in r["body"]:
|
||||
if f != "*" and f not in fields:
|
||||
fail(provider, "field", "%s sends body field %s, absent from the spec" % (r["key"], f))
|
||||
q = oas.params(path, op, "query")
|
||||
for f in r["query"]:
|
||||
if f not in q:
|
||||
fail(provider, "field", "%s sends query parameter %s, absent from the spec" % (r["key"], f))
|
||||
return covered
|
||||
|
||||
|
||||
def check_coverage(provider, oas, covered, strip="", path_filter=lambda p: True):
|
||||
excl = EXCLUSIONS.get(provider, {})
|
||||
n = 0
|
||||
for m, p, _op in oas.ops():
|
||||
if not path_filter(p):
|
||||
continue
|
||||
n += 1
|
||||
if (m, p) in covered or (m, p) in excl:
|
||||
continue
|
||||
fail(provider, "coverage", "spec operation %s %s is neither in the table nor excluded" % (m, p))
|
||||
return n
|
||||
|
||||
|
||||
def check_fields(provider, fields, oas_list):
|
||||
for schema, flist in fields:
|
||||
want = [x for x in flist.split(",") if x]
|
||||
ok = False
|
||||
for oas in oas_list:
|
||||
schemas = oas.d.get("components", {}).get("schemas", {})
|
||||
if schema in schemas:
|
||||
have = oas.reachable({"$ref": "#/components/schemas/" + schema})
|
||||
missing = [f for f in want if f not in have]
|
||||
for f in missing:
|
||||
fail(provider, "schema", "response field %s is not reachable from schema %s" % (f, schema))
|
||||
ok = True
|
||||
break
|
||||
if not ok:
|
||||
fail(provider, "schema", "schema %s not found in the spec" % schema)
|
||||
|
||||
|
||||
def summarize(provider, rows, ops, kind):
|
||||
nb = sum(len(r["body"]) + len(r["query"]) for r in rows)
|
||||
print(" %-11s %-8s rows=%-3d fields=%-4d source ops=%s" % (provider, kind, len(rows), nb, ops))
|
||||
|
||||
|
||||
# ------------------------------------------------------------- providers ----
|
||||
def stability():
|
||||
rows, fields = rows_of("stability")
|
||||
v2 = Oas(json.loads(fetch("https://docs.stability.ai/openapi.json")))
|
||||
v1 = Oas(json.loads(fetch("https://docs.stability.ai/openapi-v1.json")))
|
||||
c2 = check_rows_against_oas("stability", rows, v2, only=lambda r: r["path"].startswith("/v2beta"))
|
||||
c1 = check_rows_against_oas("stability", rows, v1, only=lambda r: r["path"].startswith("/v1"))
|
||||
n = check_coverage("stability", v2, c2) + check_coverage("stability", v1, c1)
|
||||
check_fields("stability", fields, [v2, v1])
|
||||
summarize("stability", rows, n, "openapi")
|
||||
|
||||
|
||||
def bfl():
|
||||
rows, fields = rows_of("bfl")
|
||||
oas = Oas(json.loads(fetch("https://api.bfl.ai/openapi.json")))
|
||||
cov = check_rows_against_oas("bfl", rows, oas)
|
||||
n = check_coverage("bfl", oas, cov)
|
||||
check_fields("bfl", fields, [oas])
|
||||
summarize("bfl", rows, n, "openapi")
|
||||
|
||||
|
||||
def replicate():
|
||||
rows, fields = rows_of("replicate")
|
||||
oas = Oas(json.loads(fetch("https://api.replicate.com/openapi.json")))
|
||||
servers = [s.get("url") for s in oas.d.get("servers", [])]
|
||||
if servers != ["https://api.replicate.com/v1"]:
|
||||
fail("replicate", "row", "spec server url changed: %s (kHost assumes https://api.replicate.com/v1)" % servers)
|
||||
cov = check_rows_against_oas("replicate", rows, oas)
|
||||
n = check_coverage("replicate", oas, cov)
|
||||
check_fields("replicate", fields, [oas])
|
||||
summarize("replicate", rows, n, "openapi")
|
||||
|
||||
|
||||
def fal():
|
||||
rows, fields = rows_of("fal")
|
||||
queue = Oas(json.loads(fetch("https://fal.ai/docs/api-reference/inference-api/openapi/queue.json")))
|
||||
plat = Oas(json.loads(fetch("https://api.fal.ai/v1/openapi.json")))
|
||||
cq = check_rows_against_oas("fal", rows, queue, only=lambda r: r["key"].startswith("queue."))
|
||||
cp = check_rows_against_oas("fal", rows, plat, strip="/v1", only=lambda r: r["key"].startswith("platform."))
|
||||
n = check_coverage("fal", queue, cq) + check_coverage(
|
||||
"fal", plat, cp, path_filter=lambda p: not p.startswith(FAL_PLATFORM_SKIP_PREFIXES))
|
||||
skipped = [p for _m, p, _o in plat.ops() if p.startswith(FAL_PLATFORM_SKIP_PREFIXES)]
|
||||
notes.append("fal: %d Platform API operations outside model inference are out of scope (assets, storage, "
|
||||
"serverless, compute, keys, account, organization, workflows)" % len(skipped))
|
||||
check_fields("fal", fields, [queue, plat])
|
||||
summarize("fal", rows, n, "openapi")
|
||||
|
||||
|
||||
def ideogram():
|
||||
rows, fields = rows_of("ideogram")
|
||||
oas = Oas(json.loads(fetch("https://developer.ideogram.ai/v2/openapi.json")))
|
||||
cov = check_rows_against_oas("ideogram", rows, oas)
|
||||
n = check_coverage("ideogram", oas, cov)
|
||||
check_fields("ideogram", fields, [oas])
|
||||
summarize("ideogram", rows, n, "openapi")
|
||||
|
||||
|
||||
def recraft():
|
||||
rows, fields = rows_of("recraft")
|
||||
oas = Oas(json.loads(fetch("https://www.recraft.ai/docs/api-reference/openapi.json")))
|
||||
cov = check_rows_against_oas("recraft", rows, oas)
|
||||
n = check_coverage("recraft", oas, cov)
|
||||
check_fields("recraft", fields, [oas])
|
||||
summarize("recraft", rows, n, "openapi")
|
||||
|
||||
|
||||
def higgsfield():
|
||||
rows, fields = rows_of("higgsfield")
|
||||
oas = Oas(json.loads(fetch("https://docs.higgsfield.ai/docs/openapi.json")))
|
||||
cov = check_rows_against_oas("higgsfield", rows, oas)
|
||||
n = check_coverage("higgsfield", oas, cov)
|
||||
check_fields("higgsfield", fields, [oas])
|
||||
summarize("higgsfield", rows, n, "openapi")
|
||||
|
||||
|
||||
# ------------------------------------------------------- docs-embedded ----
|
||||
def md_openapi(url):
|
||||
text = fetch(url)
|
||||
m = re.search(r"```json\n(.*?)\n```", text, re.S)
|
||||
return Oas(json.loads(m.group(1)))
|
||||
|
||||
|
||||
def leonardo():
|
||||
rows, fields = rows_of("leonardo")
|
||||
pages = {"createGeneration": "creategeneration", "createSyncGeneration": "createsyncgeneration", "getModels": "getmodels"}
|
||||
docs = [md_openapi("https://docs.leonardo.ai/reference/%s.md" % p) for p in pages.values()]
|
||||
covered = set()
|
||||
n = 0
|
||||
for oas in docs:
|
||||
for m, p, _op in oas.ops():
|
||||
n += 1
|
||||
covered.add((m, "/v2" + p))
|
||||
prose = PROSE_ONLY["leonardo"]
|
||||
for r in rows:
|
||||
if r["key"] in prose:
|
||||
continue
|
||||
hit = None
|
||||
for oas in docs:
|
||||
op = oas.op(r["method"], r["path"][len("/v2"):])
|
||||
if op is not None:
|
||||
hit = (oas, op)
|
||||
if hit is None:
|
||||
fail("leonardo", "row", "%s %s (%s) is not in the v2 reference" % (r["method"], r["path"], r["key"]))
|
||||
continue
|
||||
oas, op = hit
|
||||
if r["op"] and op.get("operationId") != r["op"]:
|
||||
fail("leonardo", "row", "%s operation id %s != %s" % (r["key"], r["op"], op.get("operationId")))
|
||||
kinds, have = oas.body(op)
|
||||
if r["body"] and not (set(r["body"]) <= set(have) | {"model", "parameters", "public"}):
|
||||
fail("leonardo", "field", "%s body fields %s" % (r["key"], r["body"]))
|
||||
for key in sorted(prose):
|
||||
notes.append("leonardo: row %s is UNVERIFIED against a spec (v1 reference pages were removed); it is "
|
||||
"documented in the quick start (POST /v1/generations) and FAQ prose (GET generation by id)" % key)
|
||||
quick = fetch("https://docs.leonardo.ai/docs/getting-started.md")
|
||||
if "api/rest/v1/generations" not in quick:
|
||||
fail("leonardo", "row", "quick start no longer shows POST /api/rest/v1/generations")
|
||||
summarize("leonardo", rows, n, "docs")
|
||||
|
||||
|
||||
MINIMAX_PAGES = ["video-generation-t2v", "video-generation-i2v", "video-generation-fl2v", "video-generation-s2v",
|
||||
"video-generation-query", "video-generation-v2-create", "video-generation-v2-query",
|
||||
"video-generation-v2-list", "video-generation-v2-delete"]
|
||||
MINIMAX_FILE_PAGE = "file-management-retrieve"
|
||||
MINIMAX_UNSUPPORTED_PAGES = {
|
||||
"video-generation-v2-h3-context-ir": "H3-Context-IR task (gap)",
|
||||
"video-generation-v2-regeneration": "video regeneration task (gap)",
|
||||
"video-agent-create": "Video Agent templates (gap)",
|
||||
"video-agent-query": "Video Agent templates (gap)",
|
||||
"video-generation-download": "download helper page, covered by files.retrieve",
|
||||
}
|
||||
|
||||
|
||||
def minimax():
|
||||
rows, _fields = rows_of("minimax")
|
||||
base = "https://platform.minimax.io/docs/api-reference/"
|
||||
texts = {p: fetch(base + p + ".md") for p in MINIMAX_PAGES + [MINIMAX_FILE_PAGE]}
|
||||
heads = set()
|
||||
page_text = {}
|
||||
for p, t in texts.items():
|
||||
for m in re.finditer(r"^````yaml \S+ (get|post|put|patch|delete) (/\S+)$", t, re.M | re.I):
|
||||
heads.add((m.group(1).upper(), m.group(2)))
|
||||
page_text.setdefault((m.group(1).upper(), m.group(2)), "")
|
||||
page_text[(m.group(1).upper(), m.group(2))] += t
|
||||
for r in rows:
|
||||
key = (r["method"], r["path"])
|
||||
if key not in heads:
|
||||
fail("minimax", "row", "%s %s (%s) is not an operation in the API reference pages" % (r["method"], r["path"], r["key"]))
|
||||
continue
|
||||
t = page_text[key]
|
||||
for f in r["body"]:
|
||||
if not re.search(r"^\s+%s:\s*$" % re.escape(f), t, re.M):
|
||||
fail("minimax", "field", "%s body field %s not found in the reference" % (r["key"], f))
|
||||
for f in r["query"]:
|
||||
if not re.search(r"- name: %s\b" % re.escape(f), t):
|
||||
fail("minimax", "field", "%s query parameter %s not found in the reference" % (r["key"], f))
|
||||
declared = {(r["method"], r["path"]) for r in rows}
|
||||
for op in sorted(heads):
|
||||
if op not in declared and op not in EXCLUSIONS["minimax"]:
|
||||
fail("minimax", "coverage", "reference operation %s %s is neither in the table nor excluded" % op)
|
||||
index = fetch("https://platform.minimax.io/docs/llms.txt")
|
||||
for page in re.findall(r"api-reference/((?:video-generation|video-agent)[a-z0-9-]*)\.md", index):
|
||||
if page in texts or page in MINIMAX_UNSUPPORTED_PAGES:
|
||||
continue
|
||||
fail("minimax", "coverage", "index lists reference page %s that is neither checked nor excluded" % page)
|
||||
for page, why in sorted(MINIMAX_UNSUPPORTED_PAGES.items()):
|
||||
notes.append("minimax: reference page %s not implemented: %s" % (page, why))
|
||||
summarize("minimax", rows, len(heads), "docs")
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- SDK ----
|
||||
def npm_tarball(pkg):
|
||||
meta = json.loads(fetch("https://registry.npmjs.org/%s/latest" % pkg))
|
||||
data = fetch(meta["dist"]["tarball"], binary=True)
|
||||
files = {}
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as tf:
|
||||
for m in tf.getmembers():
|
||||
if m.isfile() and m.name.startswith("package/") and m.size < 4_000_000:
|
||||
files[m.name[len("package/"):]] = tf.extractfile(m).read().decode("utf-8", "replace")
|
||||
return meta["version"], files
|
||||
|
||||
|
||||
def sdk_ops(files, prefix, js_ext=".js"):
|
||||
"""(METHOD, normalized path) -> resource file, from `_client.<verb>(...)` calls."""
|
||||
out = {}
|
||||
pat = re.compile(r"_client\.(get|post|put|patch|delete|getAPIList)\(\s*(?:\(0,\s*path_1\.path\)\s*`([^`]*)`|'([^']*)'|`([^`]*)`)")
|
||||
for name, text in files.items():
|
||||
if not name.startswith(prefix) or not name.endswith(js_ext) or name.endswith(".d.ts"):
|
||||
continue
|
||||
for m in pat.finditer(text):
|
||||
verb = m.group(1)
|
||||
p = m.group(2) or m.group(3) or m.group(4)
|
||||
p = re.sub(r"\$\{[^}]*\}", "{id}", p)
|
||||
out[("GET" if verb == "getAPIList" else verb.upper(), p)] = name
|
||||
return out
|
||||
|
||||
|
||||
def norm(path):
|
||||
return re.sub(r"\{[^}]*\}", "{id}", path)
|
||||
|
||||
|
||||
def runway_fields(text):
|
||||
out = []
|
||||
for m in re.finditer(r"export declare namespace (\w+Params) \{\n(.*?)\n\}\n", text, re.S):
|
||||
out += re.findall(r"^ (\w+)\??:", m.group(2), re.M)
|
||||
for m in re.finditer(r"export interface (\w+Params) \{\n(.*?)\n\}\n", text, re.S):
|
||||
out += re.findall(r"^ (\w+)\??:", m.group(2), re.M)
|
||||
return set(out)
|
||||
|
||||
|
||||
def luma_fields(text):
|
||||
out = []
|
||||
for m in re.finditer(r"export interface (\w+Params) \{\n(.*?)\n\}\n", text, re.S):
|
||||
out += re.findall(r"^ (\w+)\??:", m.group(2), re.M)
|
||||
return set(out)
|
||||
|
||||
|
||||
def sdk_check(provider, pkg, rows, prefix, base_strip, field_fn):
|
||||
version, files = npm_tarball(pkg)
|
||||
ops = sdk_ops(files, prefix)
|
||||
notes.append("%s: official SDK %s %s (%d operations found)" % (provider, pkg, version, len(ops)))
|
||||
declared = set()
|
||||
for r in rows:
|
||||
path = r["path"]
|
||||
key = (r["method"], norm(path))
|
||||
declared.add(key)
|
||||
if key not in ops:
|
||||
fail(provider, "row", "%s %s (%s) is not called by the SDK" % (r["method"], path, r["key"]))
|
||||
continue
|
||||
if r["body"] and r["body"] != ["*"]:
|
||||
js = ops[key]
|
||||
dts = files.get(js[:-3] + ".d.ts", "")
|
||||
have = field_fn(dts)
|
||||
for f in r["body"]:
|
||||
if f not in have:
|
||||
fail(provider, "field", "%s sends body field %s, absent from %s" % (r["key"], f, js[:-3] + ".d.ts"))
|
||||
for key in sorted(ops):
|
||||
if key not in declared and key not in EXCLUSIONS.get(provider, {}):
|
||||
fail(provider, "coverage", "SDK operation %s %s is neither in the table nor excluded" % key)
|
||||
return version, ops, files
|
||||
|
||||
|
||||
def runway():
|
||||
rows, fields = rows_of("runway")
|
||||
version, ops, files = sdk_check("runway", "@runwayml/sdk", rows, "resources/", "", runway_fields)
|
||||
tasks = files.get("resources/tasks.d.ts", "")
|
||||
for schema, flist in fields:
|
||||
for f in [x for x in flist.split(",") if x]:
|
||||
if not re.search(r"\b%s\??:" % re.escape(f), tasks):
|
||||
fail("runway", "schema", "task field %s not found in resources/tasks.d.ts" % f)
|
||||
for line in ("X-Runway-Version",):
|
||||
if not any(line in t for n, t in files.items() if n.endswith("client.js")):
|
||||
fail("runway", "row", "SDK no longer sends %s" % line)
|
||||
m = re.search(r"runwayVersion = '([^']+)'", files.get("client.js", ""))
|
||||
notes.append("runway: SDK default X-Runway-Version %s (table assumes 2024-11-06)" % (m.group(1) if m else "?"))
|
||||
if m and m.group(1) != "2024-11-06":
|
||||
fail("runway", "row", "SDK default X-Runway-Version is %s, RunwayClient::kApiVersion says 2024-11-06" % m.group(1))
|
||||
summarize("runway", rows, len(ops), "sdk")
|
||||
|
||||
|
||||
def luma():
|
||||
rows, fields = rows_of("luma")
|
||||
version, ops, files = sdk_check("luma", "lumaai", rows, "resources/", "", luma_fields)
|
||||
gen = files.get("resources/generations/generations.d.ts", "")
|
||||
for schema, flist in fields:
|
||||
for f in [x for x in flist.split(",") if x]:
|
||||
if not any(re.search(r"\b%s\??:" % re.escape(f), t) for n, t in files.items() if n.endswith(".d.ts")):
|
||||
fail("luma", "schema", "field %s not found in the SDK types" % f)
|
||||
if "dream-machine/v1" not in files.get("index.js", ""):
|
||||
fail("luma", "row", "SDK base URL is no longer .../dream-machine/v1")
|
||||
if "'queued' | 'dreaming' | 'completed' | 'failed'" not in gen:
|
||||
fail("luma", "schema", "generation states changed in the SDK types")
|
||||
summarize("luma", rows, len(ops), "sdk")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ probe ----
|
||||
def kling():
|
||||
rows, _f = rows_of("kling")
|
||||
bad = 0
|
||||
for r in rows:
|
||||
url = "https://api-singapore.klingai.com" + r["path"].replace("{id}", "0")
|
||||
data = b"{}" if r["method"] == "POST" else None
|
||||
req = urllib.request.Request(url, data=data, method=r["method"], headers=dict(UA, **{"Content-Type": "application/json"}))
|
||||
try:
|
||||
urllib.request.urlopen(req, timeout=30)
|
||||
code = 200
|
||||
except urllib.error.HTTPError as e:
|
||||
code = e.code
|
||||
except Exception as e: # network down: do not pretend
|
||||
fail("kling", "row", "probe of %s failed: %s" % (r["path"], e))
|
||||
continue
|
||||
if code == 404:
|
||||
fail("kling", "row", "%s %s answered 404 on the documented host" % (r["method"], r["path"]))
|
||||
else:
|
||||
bad += 0
|
||||
notes.append("kling: UNVERIFIED. No spec or SDK source exists; %d paths only probed (not 404 unauthenticated). "
|
||||
"Request and response fields come from community sources." % len(rows))
|
||||
summarize("kling", rows, "-", "probe")
|
||||
|
||||
|
||||
CHECKS = {"stability": stability, "bfl": bfl, "replicate": replicate, "fal": fal, "ideogram": ideogram,
|
||||
"recraft": recraft, "leonardo": leonardo, "runway": runway, "luma": luma, "minimax": minimax,
|
||||
"kling": kling, "higgsfield": higgsfield}
|
||||
|
||||
|
||||
def main():
|
||||
wanted = sys.argv[1:] or list(CHECKS)
|
||||
print("media call tables against vendor sources")
|
||||
for name in wanted:
|
||||
if name not in CHECKS:
|
||||
print("unknown provider", name)
|
||||
return 2
|
||||
try:
|
||||
CHECKS[name]()
|
||||
except Exception as e: # a vendor source that cannot be read is a failure, loudly
|
||||
fail(name, "source", "%s: %s" % (type(e).__name__, e))
|
||||
for n in notes:
|
||||
print(" note:", n)
|
||||
if problems:
|
||||
print("\nFAILED (%d)" % len(problems))
|
||||
for p in problems:
|
||||
print(" ", p)
|
||||
return 1
|
||||
print("\nOK: every declared row, field and schema matches its source; every source operation is covered or excluded")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user