retoor/devplacepy
1
0
Fork 1
Code Issues 44 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat: add deepsearch research system with CLI prune/clear and database schema

Browse Source 
Implement a multi-agent deep web research subsystem including CLI commands for pruning expired jobs and clearing all artifacts, database tables for sessions/messages/URL cache with indexes, config paths for chroma storage, and internal embed URL for vector operations.
This commit is contained in:
retoor
2026-06-14 01:34:21 +00:00
parent c7770ee21a
commit 4ae0b0db5d
53 changed files with 3956 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+1 -1
View File
@@ -144,7 +144,7 @@ Batch helpers (`get_users_by_uids`, `get_comment_counts_by_post_uids`, `get_vote
### Background services (`devplacepy/services/`)
`BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation. `ServiceManager` is a singleton that registers, starts, and stops services. `NewsService` fetches articles from `news_api_url`, grades each via `news_ai_url`, and inserts ALL of them into `news` with `status="published"` or `"draft"` based on the configurable `news_grade_threshold` - nothing is silently skipped. The `/admin/services` page polls live status and the log tail.
`GatewayService` (`/openai/v1/*`) is the **single point of truth for AI**. Every other AI consumer (news, bots, Devii guests) defaults its endpoint to `config.INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override via `DEVPLACE_INTERNAL_BASE_URL`), sends the generic model `molodetz`, and authenticates with the auto-generated `gateway_internal_key` (via `database.internal_gateway_key()`) when its own key is unset. **Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default on); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key. Real provider keys/URLs/models live only in the gateway; `database.migrate_ai_gateway_settings()` (run in `init_db()`) generates the internal key, migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into the editable, non-secret `gateway_api_key`/`gateway_vision_key` settings, and rewrites old external AI URLs to the gateway. The gateway is `default_enabled=True`. The gateway also serves OpenAI-compatible text **embeddings** at `POST /openai/v1/embeddings`: clients send the client model `molodetz~embed`, which `handle_embeddings` remaps to the configured embedding model (default `qwen/qwen3-embedding-8b` on OpenRouter, $0.01/1M input tokens), tracked per call as `backend="embed"` in `gateway_usage_ledger`. The gateway records every upstream call to `gateway_usage_ledger` (cost from the upstream native `cost` field when present, else computed from per-1M pricing) and samples `gateway_concurrency_samples`; `services/openai_gateway/analytics.py` rolls these into per-hour/24h token, cost, latency, error, and behavior metrics shown on `/admin/ai-usage` (`AiUsageMonitor.js`, schema `GatewayUsageOut`, Devii action `ai_usage`). It also retries transient upstream failures and trips a circuit breaker (`services/openai_gateway/reliability.py`). TTFT/inter-token latency are not tracked because the upstream is called non-streaming.
`GatewayService` (`/openai/v1/*`) is the **single point of truth for AI**. Every other AI consumer (news, bots, Devii guests) defaults its endpoint to `config.INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override via `DEVPLACE_INTERNAL_BASE_URL`), sends the generic model `molodetz`, and authenticates with the auto-generated `gateway_internal_key` (via `database.internal_gateway_key()`) when its own key is unset. **Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default on); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key. **System-message composition (`services/openai_gateway/system_message.py`, `handle_chat` only):** before assembling the upstream chat payload (after vision augmentation, so it reaches every chat consumer) the gateway composes ONE system message in the fixed order **operator preamble -> date line -> client system content**. (1) The admin-editable `gateway_system_preamble` config field (Prompt group, multiline text, default empty) is prepended ahead of the client's system content on every chat call; if the client sent no system message it becomes a new leading one (a blank preamble is a no-op). (2) The **current date in EU `DD/MM/YYYY` (date only, never time** - omitting time keeps the message byte-stable per day so upstream prompt caching stays effective) is injected only when the composed text has NO date in any common format, detected by the `contains_date` regex set (ISO `YYYY-MM-DD`, `D/M/YYYY`, `DD-MM-YYYY`, `DD.MM.YYYY`, `MM/DD/YYYY`, and textual months like `14 June 2026`/`June 14, 2026`/`14 Jun 2026`). Embeddings/passthrough are untouched. Real provider keys/URLs/models live only in the gateway; `database.migrate_ai_gateway_settings()` (run in `init_db()`) generates the internal key, migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into the editable, non-secret `gateway_api_key`/`gateway_vision_key` settings, and rewrites old external AI URLs to the gateway. The gateway is `default_enabled=True`. The gateway also serves OpenAI-compatible text **embeddings** at `POST /openai/v1/embeddings`: clients send the client model `molodetz~embed`, which `handle_embeddings` remaps to the configured embedding model (default `qwen/qwen3-embedding-8b` on OpenRouter, $0.01/1M input tokens), tracked per call as `backend="embed"` in `gateway_usage_ledger`. The gateway records every upstream call to `gateway_usage_ledger` (cost from the upstream native `cost` field when present, else computed from per-1M pricing) and samples `gateway_concurrency_samples`; `services/openai_gateway/analytics.py` rolls these into per-hour/24h token, cost, latency, error, and behavior metrics shown on `/admin/ai-usage` (`AiUsageMonitor.js`, schema `GatewayUsageOut`, Devii action `ai_usage`). It also retries transient upstream failures and trips a circuit breaker (`services/openai_gateway/reliability.py`). TTFT/inter-token latency are not tracked because the upstream is called non-streaming.
To add a service: extend `BaseService`, override `async def run_once(self) -> None`, register in `main.py`'s startup with `service_manager.register(YourService())`. It auto-appears on the services page.