feat: restrict backup archive download to primary admin and hide admin-hidden projects from other admins
DevPlace CI / test (push) Failing after 22m57s

- Add `get_admin_uids()` and `get_primary_admin_uid()` to database.py for resolving the earliest-created admin
- Modify `can_view_project()` in content.py so a project hidden by an admin is invisible to other admins (both web UI and REST API)
- Update `_download_url()` and `_backup_payload()` in admin/backups.py to accept a `can_download` flag, gating the download endpoint with `is_primary_admin()`
- Remove `role` from `_user_facts()` in docs_live.py to avoid leaking admin status in live docs
- Update doc summaries in docs_api.py to reflect the new admin-visibility and backup-download semantics
This commit is contained in:
2026-06-17 14:08:28 +00:00
parent 6b5347103b
commit 0a554ebc32
71 changed files with 1868 additions and 527 deletions
+19 -11
View File
@@ -59,13 +59,13 @@ devplacepy/
| `/posts` | Post detail, creation |
| `/gists` | Code gist listing, detail, creation, and editing; left panel offers language filtering and free-text `search` (title and description), public read |
| `/comments` | Comment creation, owner editing (`POST /comments/edit/{comment_uid}`), deletion |
| `/projects` | Project listing (left panel offers type filtering and free-text `search` over title and description), creation, owner editing (`POST /projects/edit/{slug}`), and per-project visibility toggles: `POST /projects/{slug}/private` (owner-only visibility) and `POST /projects/{slug}/readonly` (immutable files) |
| `/projects` | Project listing (left panel offers type filtering and free-text `search` over title and description), creation, owner editing (`POST /projects/edit/{slug}`), and per-project visibility toggles: `POST /projects/{slug}/private` (owner-only visibility) and `POST /projects/{slug}/readonly` (immutable files). A project hidden by a member stays visible to administrators, but a project hidden by an administrator is visible only to that owner administrator - other administrators cannot see it, its files, or its containers (web UI and REST API alike) |
| `/projects/{slug}/files` | Per-project filesystem: directory and file CRUD, upload, inline editing, and line-range operations (`lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append`) for surgical edits to large text files (public read, owner write; all writes refused while the project is read-only) |
| `/zips` | Zip job status (`/zips/{uid}`) and archive download (`/zips/{uid}/download`); archives are queued via `/projects/{slug}/zip` and `/projects/{slug}/files/zip` |
| `/forks` | Fork job status (`/forks/{uid}`); forks are queued via `/projects/{slug}/fork`. Any signed-in user can fork a project they can view into a new project they own; once the job finishes the response carries the new project URL |
| `/tools` | Public developer tools. `/tools/seo` is **SEO Diagnostics**: audit any URL or sitemap and stream live progress over a websocket. Queue with `POST /tools/seo/run`, poll `GET /tools/seo/{uid}`, read the full report at `GET /tools/seo/{uid}/report`. `/tools/deepsearch` is **DeepSearch**: a multi-agent deep web researcher that crawls and indexes sources, synthesises a cited report with confidence and gap analysis, and lets you chat over the gathered evidence. Queue with `POST /tools/deepsearch/run`, poll `GET /tools/deepsearch/{uid}`, read the report at `GET /tools/deepsearch/{uid}/session`, export at `/export.{md,json,pdf}` |
| `/projects/{slug}/containers` | Admin per-project container manager: Dockerfile CRUD with immutable versions, async image builds, and container instance creation. Reachable from the project page via the admin-only **Containers** button |
| `/admin/containers` | Admin **Containers** manager: list, create, edit, and control every container instance across all projects. The list (`/admin/containers`) has inline start/stop/restart/terminal/edit/delete on each row and a create form (pick a project, optionally a run-as user, a boot language with a source editor, restart policy, start-on-boot, plus env/ports/limits/ingress). Each instance has a detail page (`/admin/containers/{uid}`) with lifecycle controls, live logs and metrics, an interactive terminal, schedules, ingress, workspace sync, and a status history, and an edit page (`/admin/containers/{uid}/edit`) |
| `/admin/containers` | Admin **Containers** manager: list, create, edit, and control every container instance across all projects (scoped by project visibility - instances attached to another administrator's hidden project are excluded). The list (`/admin/containers`) has inline start/stop/restart/terminal/edit/delete on each row and a create form (pick a project, optionally a run-as user, a boot language with a source editor, restart policy, start-on-boot, plus env/ports/limits/ingress). Each instance has a detail page (`/admin/containers/{uid}`) with lifecycle controls, live logs and metrics, an interactive terminal, schedules, ingress, workspace sync, and a status history, and an edit page (`/admin/containers/{uid}/edit`) |
| `/p/{slug}` | Public ingress proxy (HTTP + WebSocket) to a running container instance's published port, opt-in per instance via `ingress_slug` |
| `/profile` | Profile view, editing, and a public **Media** tab (`?tab=media`) showing every attachment a user uploaded, newest first |
| `/media` | Per-attachment soft delete and restore: `POST /media/{uid}/delete` (owner or admin), `POST /media/{uid}/restore` (admin) |
@@ -118,6 +118,10 @@ Every AI gateway response (`/openai/v1/*`) also returns per-call `X-Gateway-*` h
XP awards are wired at the existing content-creation, vote, and follow hook points in the routers and centralized in `award_xp()` / `check_milestone_badges()` (`devplacepy/utils.py`). Existing accounts have their XP and levels backfilled once from prior activity at startup (`init_db()`).
## Vibe coding (Alpha, admin only)
Build software by talking to an AI agent instead of typing every line. Create a project for storage, attach a container to it (the shared `ppy` image, your files mounted at `/app`), start it, and open a terminal. The whole flow is drivable conversationally through Devii. Inside every container three agents ship preinstalled and run on **your own API key**, so all AI usage is metered to your account: **DevPlace Code (`dpc`)**, a coding agent in the same class as Claude Code; **`botje.py`**, a plug-and-play DevPlace bot you can copy and customise; and **`pagent`**, a minimal zero-dependency agent. Each container is launched with `PRAVDA_BASE_URL`, `PRAVDA_OPENAI_URL`, `PRAVDA_API_KEY`, `PRAVDA_USER_UID`, `PRAVDA_CONTAINER_NAME`, `PRAVDA_CONTAINER_UID`, and `PRAVDA_INGRESS_URL` already set. Publish a container port to a public URL at `/p/<slug>` by setting an `ingress_slug` and `ingress_port` (ask Devii to do it at create time). The feature is in **Alpha** and currently limited to administrators; the full walkthrough, including a tutorial that vibes a web app and puts it online, is at `/docs/getting-started-vibing.html`.
## Admin: Audit Log
Every state-changing action on the platform is recorded to an append-only audit log: account and authentication events, content and engagement, projects and files, news curation, admin governance, background services, container operations, ingress traffic, AI gateway calls, the Devii agent, the CLI, gamification side-effects, and security denials. Each row captures who acted, in which role, from where (origin, IP, user-agent), on which objects, what changed (old to new value), a sanitised summary, event-specific metadata, and whether the action succeeded, failed, or was denied. A companion table links every related object with a labelled relation. The complete event catalogue lives in `events.md`.
@@ -340,7 +344,7 @@ and its full configuration are documented automatically - including future servi
`DeepsearchService` powers the public **Tools -> DeepSearch** researcher. Given a single research question it plans a set of diverse web search queries, crawls and reads the most relevant sources in a subprocess (plain HTTP first, headless-browser fallback for JavaScript-heavy pages, PDF documents streamed and text-extracted, every URL SSRF-guarded), de-duplicates content, and indexes everything into a per-session ChromaDB vector collection (embeddings via the AI gateway with a local fallback). A chain of agents (summarizer, critic, linker) then synthesises a cited report with a confidence score, source diversity and explicit gap analysis. Progress streams live over `WS /tools/deepsearch/{uid}/ws`; the report is at `/tools/deepsearch/{uid}/session` (HTML or JSON) and can be exported as Markdown, JSON or PDF. A grounded chat over the session's evidence runs at `WS /tools/deepsearch/{uid}/chat` using hybrid retrieval (vector + keyword/BM25). Runs can be paused, resumed or cancelled. A cross-session URL cache avoids re-fetching pages seen by earlier runs. CLI: `devplace deepsearch prune` / `devplace deepsearch clear`. ChromaDB, weasyprint, pypdf and Playwright are core dependencies.
`BackupService` powers the admin **Admin -> Backups** dashboard, an enterprise-grade backup system that runs entirely as asynchronous jobs so it never impacts the running server. An administrator can back up one of four targets: the **database** (a consistent SQLite snapshot of the main database and the Devii task/lesson databases, taken with SQLite's online backup API so it is consistent under WAL), **uploads** (every attachment and project file), **keys and config** (VAPID keys), or the **full data directory** (database snapshot, uploads, and keys in one archive, excluding regenerable staging, locks, caches, and container workspaces). Each backup is compressed to a `tar.gz` in a stdlib subprocess off the request path and recorded with its size, file count, and a SHA-256 checksum. Archives live under `data/backups/` (sharded on the random uuid tail) and are served only through the admin-guarded `/admin/backups/{uid}/download` route. The dashboard reports detailed storage usage - the size and file count of every major data area, the total data-directory footprint, the total size and count of stored backups, and disk usage (total, used, free, percent), computed in a worker thread and cached briefly so the page never blocks. Backups can be **scheduled** (CRUD) on an interval or 5-field cron expression with a `keep_last` rotation count that prunes older backups of the same schedule; the service evaluates schedules only on the lock-owning worker so each fires exactly once. Backup archives are permanent operational artifacts: job retention only removes the tracking row, never the archive, which is deleted only by an administrator, by schedule rotation, or via the CLI. CLI: `devplace backups list` / `devplace backups run <target>` / `devplace backups prune` / `devplace backups clear`. Devii tools: `backups_overview`, `backup_run`, `backup_status`, `backup_delete`, `backup_schedule_create`, `backup_schedule_delete` (all admin-only). The service creates and stores backups but does not restore them into a live server; restore is a documented manual procedure (stop the server, unpack the archive over the data directory, verify the checksum, restart).
`BackupService` powers the admin **Admin -> Backups** dashboard, an enterprise-grade backup system that runs entirely as asynchronous jobs so it never impacts the running server. An administrator can back up one of four targets: the **database** (a consistent SQLite snapshot of the main database and the Devii task/lesson databases, taken with SQLite's online backup API so it is consistent under WAL), **uploads** (every attachment and project file), **keys and config** (VAPID keys), or the **full data directory** (database snapshot, uploads, and keys in one archive, excluding regenerable staging, locks, caches, and container workspaces). Each backup is compressed to a `tar.gz` in a stdlib subprocess off the request path and recorded with its size, file count, and a SHA-256 checksum. Archives live under `data/backups/` (sharded on the random uuid tail) and are served only through `/admin/backups/{uid}/download`, which is restricted to the **primary administrator** - the first user created with the Admin role. Every other administrator receives a 403 from the endpoint and sees the Download button disabled with the tooltip `Not available`; creating, running, deleting, and scheduling backups remain available to all administrators. The dashboard reports detailed storage usage - the size and file count of every major data area, the total data-directory footprint, the total size and count of stored backups, and disk usage (total, used, free, percent), computed in a worker thread and cached briefly so the page never blocks. Backups can be **scheduled** (CRUD) on an interval or 5-field cron expression with a `keep_last` rotation count that prunes older backups of the same schedule; the service evaluates schedules only on the lock-owning worker so each fires exactly once. Backup archives are permanent operational artifacts: job retention only removes the tracking row, never the archive, which is deleted only by an administrator, by schedule rotation, or via the CLI. CLI: `devplace backups list` / `devplace backups run <target>` / `devplace backups prune` / `devplace backups clear`. Devii tools: `backups_overview`, `backup_run`, `backup_status`, `backup_delete`, `backup_schedule_create`, `backup_schedule_delete` (all admin-only). The service creates and stores backups but does not restore them into a live server; restore is a documented manual procedure (stop the server, unpack the archive over the data directory, verify the checksum, restart).
### Adding a service
@@ -460,6 +464,7 @@ Configuration on the Services tab:
| `gateway_price_cache_hit_per_m` / `_cache_miss_per_m` / `_output_per_m` | 0.0028 / 0.14 / 0.28 | Chat cost per 1M tokens, used when the upstream returns no native cost (DeepSeek) |
| `gateway_vision_price_input_per_m` / `_output_per_m` | 0 / 0 | Vision cost per 1M tokens, used only when the vision upstream returns no native cost |
| `gateway_embed_price_input_per_m` | 0.01 | Embeddings cost per 1M input tokens, used only when the embeddings upstream returns no native cost |
| `gateway_rsearch_cost_per_call` | 0.0 | Flat cost attributed to each external `rsearch` call (web search / AI answer / chat / image describe), recorded under backend `rsearch` so external AI spend appears in AI usage |
| `gateway_max_retries` / `gateway_retry_backoff_ms` | 2 / 250 | Retry attempts and linear backoff on timeout, connection error, or upstream 5xx |
| `gateway_circuit_threshold` / `gateway_circuit_cooldown_seconds` | 5 / 30 | Consecutive failures before the circuit breaker opens, and its cooldown |
| `gateway_usage_retention_hours` | 720 | How long per-call usage rows are kept before pruning (30 days) |
@@ -477,8 +482,11 @@ shows live request/error/latency/vision-call counters.
Every upstream call (chat, vision, and passthrough) is recorded to `gateway_usage_ledger`
with its tokens, cost, latency breakdown, status, and caller. Cost is taken from the
upstream native `cost` field when present (OpenRouter) and computed from the configured
per-million pricing otherwise (DeepSeek, which reports no cost). The admin **AI usage**
page (`/admin/ai-usage`) reports per-hour and 24h metrics - request volume and throughput,
per-million pricing otherwise (DeepSeek, which reports no cost). External `rsearch` web
tools do not pass through the gateway upstream, so each call is also recorded to the same
ledger under backend `rsearch` (zero tokens, the flat `gateway_rsearch_cost_per_call`),
keeping the ledger a complete record of platform AI spend. The admin **AI usage**
page (`/admin/ai-usage`) reports per-hour, 24h, and **all-time** metrics - request volume and throughput,
token usage with averages and percentiles (p50/p90/p95/p99), latency (upstream round-trip,
gateway overhead, semaphore queue wait, connection establishment), error rates by category,
cost (per model, per caller, input vs output, projected monthly burn, caching savings),
@@ -708,13 +716,13 @@ A member may delete only their own content, but an **administrator may delete an
### Database API (`/dbapi`, admin or internal only)
An administrator (session or admin API key) or an internal service (the gateway internal key) can read and update any table through a single, safe API; members and guests get `403`.
An administrator (session or admin API key) or an internal service (the gateway internal key) can **read** any table through a single, safe API; members and guests get `403`. The database API is **strictly read-only** - it can never insert, update, replace, delete, or restore data in any way.
- **CRUD per table:** `GET /dbapi/{table}` (filtered, searchable, keyset pagination), `GET /dbapi/{table}/{key}/{value}`, `POST /dbapi/{table}`, `PATCH /dbapi/{table}/{key}/{value}`, `DELETE /dbapi/{table}/{key}/{value}` (soft delete, `?hard=true` to purge), and `.../restore`. Inserts are born-live; unknown columns are rejected; deny-listed tables (sessions, password resets) are never exposed.
- **`query()` is read-only:** `POST /dbapi/query` runs a single validated SELECT and returns rows. Every query is parsed (sqlglot), classified, and dry-run with `EXPLAIN` on a read-only connection before execution; non-SELECT statements are refused and a SELECT with no WHERE/JOIN/LIMIT is flagged as suspicious. All writes go through the structured CRUD, never raw SQL.
- **Ask in plain language:** `POST /dbapi/nl` turns a question such as *"all users registered longer than three days"* into a validated SELECT (auto-adding `deleted_at IS NULL` for soft-delete tables), using the platform AI gateway and re-prompting until the SQL validates; pass `execute=true` to also run it.
- **Async:** `POST /dbapi/query/async` runs a heavy query off the request path and streams progress over `WS /dbapi/query/{uid}/ws`.
- The Devii assistant exposes the same capability to administrators, with every insert/update/delete requiring explicit confirmation.
- **Read per table:** `GET /dbapi/{table}` (filtered, searchable, keyset pagination) and `GET /dbapi/{table}/{key}/{value}`. There are no write endpoints; deny-listed tables (sessions, password resets) are never exposed.
- **`query()` is read-only:** `POST /dbapi/query` runs a single validated SELECT and returns rows. Every query is parsed (sqlglot), classified, and dry-run with `EXPLAIN` on a read-only connection before execution; non-SELECT statements (INSERT/UPDATE/DELETE/DDL) are refused, and a SELECT with no WHERE/JOIN/LIMIT is flagged as suspicious.
- **Ask in plain language:** `POST /dbapi/nl` turns a question such as *"all users registered longer than three days"* into a validated SELECT (auto-adding `deleted_at IS NULL` for soft-delete tables), using the platform AI gateway and re-prompting until the SQL validates; pass `execute=true` to also run it read-only.
- **Async:** `POST /dbapi/query/async` runs a heavy read query off the request path and streams progress over `WS /dbapi/query/{uid}/ws`.
- The Devii assistant exposes the same read-only capability to administrators (list, get, query, and natural-language SELECT); it cannot change data through the database API.
### Pub/Sub bus (`/pubsub`)