retoor/devplacepy
1
0
Fork 1
Code Issues 44 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat: add embeddings endpoint and config for OpenAI-compatible text embeddings via gateway

Browse Source 
Add POST /openai/v1/embeddings route in openai_gateway router, new config fields for embedding upstream URL/model/key/enabled toggle with defaults pointing to OpenRouter Qwen3 8B, INTERNAL_EMBED_MODEL constant in config.py, documentation in docs_api.py and README.md describing the molodetz~embed model mapping, and embed-call tracking in gateway metrics alongside existing chat/vision counters.
This commit is contained in:
retoor
2026-06-14 01:06:18 +00:00
parent 1076696dec
commit c7770ee21a
17 changed files with 662 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+1 -1
View File
@@ -144,7 +144,7 @@ Batch helpers (`get_users_by_uids`, `get_comment_counts_by_post_uids`, `get_vote
### Background services (`devplacepy/services/`)
`BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation. `ServiceManager` is a singleton that registers, starts, and stops services. `NewsService` fetches articles from `news_api_url`, grades each via `news_ai_url`, and inserts ALL of them into `news` with `status="published"` or `"draft"` based on the configurable `news_grade_threshold` - nothing is silently skipped. The `/admin/services` page polls live status and the log tail.
`GatewayService` (`/openai/v1/*`) is the **single point of truth for AI**. Every other AI consumer (news, bots, Devii guests) defaults its endpoint to `config.INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override via `DEVPLACE_INTERNAL_BASE_URL`), sends the generic model `molodetz`, and authenticates with the auto-generated `gateway_internal_key` (via `database.internal_gateway_key()`) when its own key is unset. **Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default on); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key. Real provider keys/URLs/models live only in the gateway; `database.migrate_ai_gateway_settings()` (run in `init_db()`) generates the internal key, migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into the editable, non-secret `gateway_api_key`/`gateway_vision_key` settings, and rewrites old external AI URLs to the gateway. The gateway is `default_enabled=True`. The gateway records every upstream call to `gateway_usage_ledger` (cost from the upstream native `cost` field when present, else computed from per-1M pricing) and samples `gateway_concurrency_samples`; `services/openai_gateway/analytics.py` rolls these into per-hour/24h token, cost, latency, error, and behavior metrics shown on `/admin/ai-usage` (`AiUsageMonitor.js`, schema `GatewayUsageOut`, Devii action `ai_usage`). It also retries transient upstream failures and trips a circuit breaker (`services/openai_gateway/reliability.py`). TTFT/inter-token latency are not tracked because the upstream is called non-streaming.
`GatewayService` (`/openai/v1/*`) is the **single point of truth for AI**. Every other AI consumer (news, bots, Devii guests) defaults its endpoint to `config.INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override via `DEVPLACE_INTERNAL_BASE_URL`), sends the generic model `molodetz`, and authenticates with the auto-generated `gateway_internal_key` (via `database.internal_gateway_key()`) when its own key is unset. **Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default on); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key. Real provider keys/URLs/models live only in the gateway; `database.migrate_ai_gateway_settings()` (run in `init_db()`) generates the internal key, migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into the editable, non-secret `gateway_api_key`/`gateway_vision_key` settings, and rewrites old external AI URLs to the gateway. The gateway is `default_enabled=True`. The gateway also serves OpenAI-compatible text **embeddings** at `POST /openai/v1/embeddings`: clients send the client model `molodetz~embed`, which `handle_embeddings` remaps to the configured embedding model (default `qwen/qwen3-embedding-8b` on OpenRouter, $0.01/1M input tokens), tracked per call as `backend="embed"` in `gateway_usage_ledger`. The gateway records every upstream call to `gateway_usage_ledger` (cost from the upstream native `cost` field when present, else computed from per-1M pricing) and samples `gateway_concurrency_samples`; `services/openai_gateway/analytics.py` rolls these into per-hour/24h token, cost, latency, error, and behavior metrics shown on `/admin/ai-usage` (`AiUsageMonitor.js`, schema `GatewayUsageOut`, Devii action `ai_usage`). It also retries transient upstream failures and trips a circuit breaker (`services/openai_gateway/reliability.py`). TTFT/inter-token latency are not tracked because the upstream is called non-streaming.
To add a service: extend `BaseService`, override `async def run_once(self) -> None`, register in `main.py`'s startup with `service_manager.register(YourService())`. It auto-appears on the services page.