feat: add admin/internal database API with CRUD, read-only query, and natural-language SQL endpoints

Add a new `/dbapi` router package providing a generic database API over `dataset`, restricted to admin sessions, admin API keys, and the internal gateway key. Includes:

- `tables.py`: list all tables and inspect table schemas
- `crud.py`: full CRUD operations (GET, POST, PATCH, DELETE) with soft-delete awareness, born-live inserts, `?include_deleted`, `.../restore`, and `?hard=true` purge
- `query.py`: validated read-only SELECT execution via sqlglot parsing, classification, and EXPLAIN dry-run; async query jobs with WebSocket streaming via `DbApiJobService`
- `nl.py`: natural-language-to-SQL conversion using the platform AI gateway with re-prompting until validation passes

Also register `DbApiJobService` and `PubSubService` in the service manager, add `DBAPI_DIR` to config data paths, and force cleartext `http://` connections to HTTP/1.1 in `curl_transport` to fix large request failures against uvicorn's HTTP/1.1-only internal gateway.
This commit is contained in:
2026-06-14 23:00:30 +00:00
parent 96521e8757
commit d31ccba6c1
69 changed files with 3278 additions and 12 deletions
+3 -1
View File
@@ -113,6 +113,8 @@ Routers in `devplacepy/routers/` are organised as a **directory tree that mirror
| `/p/{slug}` | proxy.py - public ingress reverse proxy (HTTP + WebSocket) to a running container instance's published host port, opt-in per instance via `ingress_slug` |
| `/xmlrpc` | xmlrpc.py - reverse-proxies XML-RPC calls to the forking XML-RPC bridge (`services/xmlrpc/`, supervised by `XmlrpcService` on loopback `config.XMLRPC_PORT`), which generates one XML-RPC method per documented REST endpoint from `docs_api.API_GROUPS` (`posts.create`, `feed.list`, ...). One struct of named params per call; auth via in-band `api_key`, `X-API-KEY`/`Bearer` header, or `http://user:pass@host/xmlrpc` Basic. Full introspection + `system.multicall`; REST errors become XML-RPC faults. Exempt from the rate limiter (enforced on the forwarded internal hop). See AGENTS.md -> "XML-RPC bridge" |
| `/api` | devrant/ package - devRant-compatible REST protocol (`auth.py`, `rants.py`, `comments.py`, `notifs.py`). Translates devRant requests onto DevPlace data: rants<->posts, comments/votes onto the native engagement layer, devRant integer ids onto each table's auto-increment `id`. Token auth via `devrant_tokens`. See `### devRant compatibility API` and AGENTS.md -> "devRant compatibility API" |
| `/dbapi` | dbapi/ package - **admin/internal-only** generic database API over `dataset`. `tables.py` (`GET /tables`, `GET /{table}/schema`), `crud.py` (`GET/POST/PATCH/DELETE /{table}[/{key}/{value}]`, soft-delete aware, born-live inserts, `?include_deleted`, `.../restore`), `query.py` (`POST /query` hard SELECT-only validated read-only run; `POST /query/async` + `GET /query/{uid}` + `WS /query/{uid}/ws` via `DbApiJobService` kind `dbquery`), `nl.py` (`POST /nl` natural-language->SQL via the AI gateway, returns validated SQL, `execute=true` runs it read-only). Auth = admin session/api_key OR `internal_gateway_key()` (`services/dbapi/policy.py`); SQL validated by `services/dbapi/validate.py` (sqlglot classify + suspicious-flag + read-only `EXPLAIN` dry-run). Devii tools `db_*` (admin; mutations confirm-gated). See AGENTS.md -> "Database API service" |
| `/pubsub` | pubsub.py - **database-free** publish/subscribe bus. `WS /pubsub/ws` (subscribe/unsubscribe/publish frames, `foo.*` wildcards), `POST /pubsub/publish` + `GET /pubsub/topics` (admin/internal). Lock-owner-gated WS (`4013` retry) so subscribers converge on one worker; topic authz in `services/pubsub/policy.py` (`user.{uid}.*` private, `public.*` shared, admin/internal anywhere, guests opt-in). In-process `services.pubsub.publish(topic, data)` for backends; frontend `app.pubsub`. See AGENTS.md -> "Pub/Sub service" |
| (none) | seo.py - `/robots.txt`, `/sitemap.xml` |
Docs (`routers/docs/pages.py` `DOCS_PAGES`, re-exported from the `routers/docs` package; handlers in `routers/docs/views.py`) entries take optional `admin: True` (hidden + 404 for non-admins, but still indexed and surfaced only to admins by `docs_search`) and `section: "..."` (a nested sidebar group rendered by `docs_base.html`). Sidebar `section`s are grouped under four ordered **audience tiers** (`AUDIENCES` in `pages.py`: `Start here`, `Build with the API`, `Contribute and internals`, `Operate`); `nav_groups(visible_pages)` builds the `[(audience, [(section, [pages])])]` tree and `views.py` passes it as `nav`, so `docs_base.html` renders an audience super-header (`.sidebar-tier`) above each section. The tiering is sidebar-only - `DOCS_PAGES` stays canonical for search/export/routing. The public `getting-started` page (`SECTION_GENERAL`) is the new-contributor on-ramp; gate its deep-internals links with `{% if is_admin(user) %}` so guests get no 404s. Keep one canonical home per concept (the `auth` API group defers auth-method detail to the `authentication` prose page). The member-facing `devii` prose page is functional; admins also get a `Devii internals` section of `devii-*` technical subpages. Prose docs pages are rendered to HTML **server-side** (`docs_prose.py`, mistune GFM): the handler converts the `data-render` markdown block to HTML before sending and drops the `data-render` attribute, so the client never re-renders it (no markdown-to-HTML flicker, faster first paint). Authored example markup inside that block is therefore still HTML-escaped (`&lt;dp-...&gt;`); content outside the block (component live-demo blocks and their `<script type="module">`) passes through untouched. User-generated content elsewhere still uses the client `data-render` pipeline. The public `Components` section (`component-*` prose pages) documents the custom web components with a **live, interactive example** on each page. The public `Styles` section (`styles`, `styles-colors`, `styles-layout`, `styles-responsiveness`, `styles-consistency`) is the design-system reference: the colour tokens and their meaning, the approved page layouts, the responsive breakpoint ladder, and the HARD structural rules every page must follow (taken from the feed/posts page as the canonical implementation). It uses the same live-demo convention (real demo markup OUTSIDE the `data-render` block, example markup inside it entity-escaped). A prose page's `<div class="docs-content" data-render>` is rendered client-side via marked + DOMPurify on `textContent`, so any example markup shown as code inside it MUST be HTML-escaped (`&lt;dp-dialog&gt;`) or the browser parses it as a real element before render; the live demo itself goes in a separate block OUTSIDE the `data-render` div, where a `<script type="module">` (which imports its own component module, since it executes before `Application.js`) wires it up.
@@ -209,7 +211,7 @@ A second REST surface mounted at `/api` that mimics the public devRant API on De
- **`RedirectResponse(url=..., status_code=302)`** - always pass `status_code` as an integer literal, never `status.HTTP_302_FOUND`.
- **Ownership checks:** compare `resource["user_uid"] == user["uid"]` before edit/delete (use `content.is_owner`). **Deletes additionally allow any admin:** every content delete guards `is_owner(...) or is_admin(user)` on the endpoint (posts/gists/projects in `content.delete_content_item`, `comments.delete_comment`, `project_files.project_file_delete`, `media.delete_media`, `uploads.delete_attachment_route`; news is admin-only), so an administrator (including via Devii, which only calls the API) may delete any member's content while members are limited to their own. Edits stay owner-only. Deletes are **soft** (`deleted_at`/`deleted_by`) and cascade comments → votes → engagement → resource under one shared stamp.
- **All dates shown to users are DD/MM/YYYY** (European). Use `format_date()` from `utils.py` (template global), or `time_ago()` which auto-switches to DD/MM/YYYY past 30 days.
- **Slug + UUID lookup:** resources with slugs (posts, gists, news, projects) accept either the slug or the bare UUID - see `resolve_by_slug()` in `database.py`. Slugs are generated via `make_combined_slug(title, uid)` which prefixes the first 8 chars of the UUID.
- **Slug + UUID lookup:** resources with slugs (posts, gists, news, projects) accept either the slug or the bare UUID - see `resolve_by_slug()` in `database.py`. Slugs are generated via `make_combined_slug(title, uid)` which prefixes the last 12 hex chars of the UUID (the uuid7 **random tail**, NOT its leading bytes - the first 48 bits are a millisecond timestamp, so a head prefix is identical for all rows written in the same ~65s window and collides with the same title, making `resolve_by_slug` ambiguous; same reasoning as the blob-shard tail rule).
- **Username:** 3-32 chars, letters/numbers/hyphens/underscores. **Password:** min 6 chars.
- **Roles are stored capitalized:** the `users.role` value is exactly `"Admin"` or `"Member"` (the first registered user becomes `"Admin"`; `auth.py`). Always test admin-ness through the `is_admin(user)` global (`utils.py`, also a Jinja global), which compares `user.get("role") == "Admin"` case-sensitively - never hand-roll a lowercase compare. The **only** lowercase surface is the CLI (`devplace role set <username> <member|admin>` accepts lowercase but writes `role.capitalize()`; `role get` prints `.lower()` for display). A lowercase role in the DB silently fails every admin check.
- **Never pass a `respond()` context key that collides with a Jinja global.** `respond(request, template, ctx, model=XOut)` feeds the **same** `ctx` dict to both the Pydantic model (JSON) and the template render. Jinja context variables shadow `env.globals` across the whole inheritance chain, so a key like `is_admin`/`avatar_url`/`format_date`/`is_self`/`owns`/`guest_disabled` holding a non-callable value turns `base.html`'s `{% if is_admin(user) %}` into `False(user)` -> `TypeError: 'bool' object is not callable`, a 500 that only fires for the branch that calls the global (e.g. logged-in users, not guests). Name viewer/permission flags distinctly (`viewer_is_admin`, not `is_admin`) in both the schema and the context. This was a real issue on `/issues/{number}`; `tests/api/issues/create.py::test_issue_detail_renders_for_{member,admin}` guard it.