retoor/devplacepy
1
0
Fork 1
Code Issues 44 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat: add project fork async job service with cli prune/clear commands and shared container image build target

Browse Source
This commit is contained in:
retoor
2026-06-09 14:06:02 +00:00
parent c565e3b55c
commit 05c6fa7f3b
90 changed files with 6889 additions and 1146 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+50 -9
View File
@@ -59,7 +59,9 @@ devplacepy/
| `/projects` | Project listing, creation, and per-project visibility toggles: `POST /projects/{slug}/private` (owner-only visibility) and `POST /projects/{slug}/readonly` (immutable files) |
| `/projects/{slug}/files` | Per-project filesystem: directory and file CRUD, upload, inline editing, and line-range operations (`lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append`) for surgical edits to large text files (public read, owner write; all writes refused while the project is read-only) |
| `/zips` | Zip job status (`/zips/{uid}`) and archive download (`/zips/{uid}/download`); archives are queued via `/projects/{slug}/zip` and `/projects/{slug}/files/zip` |
| `/projects/{slug}/containers` | Admin container manager: Dockerfile CRUD with immutable versions, async image builds, container instance lifecycle, logs, exec, metrics, and schedules |
| `/forks` | Fork job status (`/forks/{uid}`); forks are queued via `/projects/{slug}/fork`. Any signed-in user can fork a project they can view into a new project they own; once the job finishes the response carries the new project URL |
| `/projects/{slug}/containers` | Admin per-project container manager: Dockerfile CRUD with immutable versions, async image builds, and container instance creation. Reachable from the project page via the admin-only **Containers** button |
| `/admin/containers` | Admin **Containers** section: a list of every container instance across all projects (`/admin/containers`), each linking to a dedicated instance detail page (`/admin/containers/{uid}`) with lifecycle controls, live logs and metrics, an interactive terminal, schedules, ingress, and workspace sync |
| `/p/{slug}` | Public ingress proxy (HTTP + WebSocket) to a running container instance's published port, opt-in per instance via `ingress_slug` |
| `/profile` | Profile view, editing |
| `/messages` | Direct messaging |
@@ -127,6 +129,8 @@ Operational behavior is tunable live from `/admin/settings` (stored in `site_set
| `registration_open` | `1` | When `0`, new sign-ups are rejected |
| `maintenance_mode` | `0` | When `1`, non-admins see the maintenance page; admins retain access |
| `maintenance_message` | scheduled-maintenance text | Message shown during maintenance |
| `customization_enabled` | `1` | When `0`, no user CSS/JS customization is injected on any page |
| `customization_js_enabled` | `1` | When `0`, user custom CSS is still served but custom JavaScript is suppressed |
Numeric values are floored to safe minimums so an invalid entry cannot lock out writes or stall services. Consumers read via `get_setting`/`get_int_setting`, which fall back to these defaults when a row is absent.
@@ -210,16 +214,16 @@ and its full configuration are documented automatically - including future servi
- **`NewsService`** - fetches news from `news.app.molodetz.nl/api`, grades with AI, stores articles >= threshold
- **`BotsService`** - Playwright fleet of AI personas that browse and interact with a DevPlace instance, with live cost/usage metrics (opt-in; install the `bots` extra)
- **`GatewayService`** - OpenAI-compatible LLM gateway at `/openai/v1/*`, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default)
- **`JobService` / `ZipService`** - generic async job framework (`services/jobs/`) for heavy, blocking work run off the request path; `ZipService` is the first consumer, building project zip archives in a subprocess
- **`ContainerBuildService` / `ContainerService`** - the admin container manager (`services/containers/`): async Docker image builds and a reconciling supervisor for container instances
- **`JobService` / `ZipService` / `ForkService`** - generic async job framework (`services/jobs/`) for heavy, blocking work run off the request path; `ZipService` builds project zip archives in a subprocess, `ForkService` copies a project into a new project owned by the forking user
- **`ContainerService`** - the admin container manager (`services/containers/`): a reconciling supervisor for container instances, all running one shared prebuilt image
### Container manager (admin only)
`services/containers/` builds versioned Docker images for projects and runs supervised container instances, all from the web UI, the HTTP API, and Devii. It drives the `docker` CLI via async subprocesses behind a pluggable `Backend` interface (a `DockerCliBackend` plus a `FakeBackend` for tests). `ContainerBuildService` (a `JobService`) builds `<name>:<build-number>` images asynchronously with streamed logs; `ContainerService` reconciles desired instance state against `docker ps` each tick (containers are labeled `devplace.instance=<uid>`, so orphans are reaped and no state is lost), applies restart policies, fires cron/interval/one-time schedules, and samples metrics. Dockerfiles have immutable versions; saving new content auto-builds the next tag (builds run with `--network=host` by default so pip can reach PyPI). The container's `/app` is bind-mounted to a persistent project workspace (materialized from the project files) and can be synced back. A running instance can be **published** with an `ingress_slug`, making its service reachable (HTTP and WebSocket) at `/p/<slug>` through DevPlace. A high-quality default Dockerfile (Debian python-slim) is provided.
`services/containers/` runs supervised container instances from the web UI, the HTTP API, and Devii. It drives the `docker` CLI via async subprocesses behind a pluggable `Backend` interface (a `DockerCliBackend` plus a `FakeBackend` for tests). **There is no in-app image building.** Every instance runs ONE shared prebuilt image, `ppy:latest` (override `DEVPLACE_CONTAINER_IMAGE`), built once with **`make ppy`** from `ppy.Dockerfile`: a Python + Playwright base with a broad set of common libraries preinstalled, the `pravda` (uid 1000) user, the sudo superclone, and `pagent` at `/usr/bin/pagent.py` all baked in. Creating an instance is then an instant `docker run` (it fails fast with a clear error if the `ppy` image has not been built yet). `ContainerService` reconciles desired instance state against `docker ps` each tick (containers are labeled `devplace.instance=<uid>`, so orphans are reaped and no state is lost), applies restart policies, fires cron/interval/one-time schedules, and samples metrics. The container's `/app` is bind-mounted to a persistent project workspace (materialized from the project files) and can be synced back; projects that need extra packages use runtime `pip install` (pravda owns the site-packages, no sudo needed) or add the library to `ppy.Dockerfile` and rerun `make ppy`. A running instance can be **published** with an `ingress_slug`, making its service reachable (HTTP and WebSocket) at `/p/<slug>` through DevPlace. The manager is reached two ways: the admin **Containers** sidebar entry (`/admin/containers`) lists every instance across all projects and opens a dedicated detail page per instance, and each project page carries an admin-only **Containers** button to its own instance manager.
**Security:** this requires mounting the Docker socket, which grants host root. Every build, run, exec, lifecycle, and schedule operation is administrator-only; `--privileged` is never used and all docker calls are argument-list subprocesses. The services are disabled by default; an admin enables them on `/admin/services`. CLI: `devplace containers list | reconcile | prune | gc-workspaces`.
**Security:** this requires mounting the Docker socket, which grants host root. Every run, exec, lifecycle, and schedule operation is administrator-only; `--privileged` is never used and all docker calls are argument-list subprocesses. The service is disabled by default; an admin enables **Containers** on `/admin/services`. CLI: `devplace containers list | reconcile | prune | prune-builds | gc-workspaces` (`prune-builds` is a one-time cleanup that removes legacy per-project images and the old dockerfiles/builds tables).
**Runtime data** (container workspaces and zip archives) lives in `DEVPLACE_DATA_DIR` (default `var/`), **outside the package and never served via `/static`**; ephemeral build contexts use the OS temp dir. The docker daemon must be able to bind-mount the data dir for `/app`. In development `make dev` scopes `uvicorn --reload` to `--reload-dir devplacepy` so writing build contexts does not restart the server mid-build.
**Runtime data** (container workspaces and zip archives) lives in `DEVPLACE_DATA_DIR` (default `var/`), **outside the package and never served via `/static`**. The docker daemon must be able to bind-mount the data dir for `/app`.
### Async job framework and zip downloads
@@ -227,6 +231,8 @@ and its full configuration are documented automatically - including future servi
`ZipService` archives a project (or a file/folder subtree) into `{crc32}.{slug}.zip` using the standalone `zip_worker` subprocess, with a CRC32 over the output. The frontend `app.zipDownloader` (any `data-zip-download` element, plus the files context menu) enqueues, polls `/zips/{uid}`, and downloads from `/zips/{uid}/download`. The job uid is the capability token, so anyone with the link can download. CLI: `devplace zips prune` (delete expired) / `devplace zips clear` (delete all).
`ForkService` copies a project into a new project owned by the forking user. The **Fork** button on the project page (any signed-in user) prompts for a name; the job creates the destination project, duplicates the entire virtual filesystem, and records a directional `project_forks` relation so each fork shows a "Forked from X" link. The frontend `app.projectForker` enqueues, polls `/forks/{uid}`, and redirects to the new project once it is done; on failure the partially created project is rolled back. The forked project is permanent, so retention removes only the job tracking row. CLI: `devplace forks prune` / `devplace forks clear` (job rows only).
### Adding a service
Create a class extending `BaseService`, declare its `config_fields`, override `run_once()` (read parameters via `self.get_config()`), and register in `main.py`:
@@ -419,6 +425,42 @@ devii --api-key <your DevPlace api_key> --base-url https://your-host
devii -p "List my unread notifications as a bullet list." # one-shot
```
### Site customization (per-user CSS/JS)
Each user can reshape the site to taste by injecting their own **CSS** (look) and
**JavaScript** (behaviour), scoped either to a single **page type** (every post page, every
project page) or **globally** (every page). Customizations apply only to that user's own
sessions; a signed-out guest gets customizations scoped to their `devii_guest` session cookie.
This is the user's own code running in their own browser, like a userscript, so it never affects
anyone else's view.
It is configured conversationally through Devii. Ask for a change and Devii previews it live in
your browser, then before saving it asks whether to apply it to the current page type or the
whole site, and only persists once you confirm. Overrides are stored in the `user_customizations`
table (one row per owner, scope, and language; cached with cross-worker invalidation) and injected
server-side: custom CSS is the last `<style>` in `<head>` so it overrides the design system, and
custom JavaScript runs after the application boots with access to the global `app`. Devii tools:
`customize_list`, `customize_get`, `customize_set_css`, `customize_set_js`, and `customize_reset`
(restores the default look and behaviour). Operators can disable the feature site-wide with the
`customization_enabled` / `customization_js_enabled` settings.
### User-defined tools (vibe new functionality)
Users can invent new Devii capabilities by describing them in natural language, with no code. Telling
Devii "when I say woeii, search the web for a cat picture matching my description and set it as my
page background" makes it call `tool_create` and store a virtual tool. The tool then appears in
Devii's own toolset, and on the next turn "woeii orange" calls it: the handler re-runs Devii itself
(a self-evaluating sub-agent) on the stored prompt plus the user's input, with full tool access, and
returns the result. Each tool takes a single free-form `input` string; the stored prompt directs what
to do with it.
Virtual tools are owner-scoped (persistent in the DB for signed-in users in `devii_virtual_tools`,
session-only for guests) and never run in anyone else's session. The full CRUD is exposed as Devii
tools: `tool_create`, `tool_list`, `tool_get`, `tool_update` (including enable/disable), and
`tool_delete`. A general `eval(prompt)` tool runs Devii on any prompt and returns the result; it is
the same engine that powers virtual tools. Self-evaluation depth is bounded so a tool cannot loop by
calling itself.
## Push notifications & PWA
Authenticated users can receive native web push notifications, and the site is an
@@ -538,7 +580,7 @@ make docker-up # start (creates ./var, mounts the socket)
Open `http://<host>:${PORT}` (default 10500). `make docker-logs` tails output; `make docker-down` stops.
`make docker-build` and `make docker-up` always apply the `docker-compose.containers.yml` overlay, so the admin-only Container Manager works with no extra setup. Mounting the docker socket grants the app **root on the host**; the feature itself stays disabled until an admin enables **Container builds** and **Containers** on `/admin/services`, but treat the whole deployment as trusted-admins-only. Always update through the make targets - a plain `docker compose up -d` drops the overlay and silently removes the socket and CLI.
`make docker-build` and `make docker-up` always apply the `docker-compose.containers.yml` overlay, so the admin-only Container Manager works with no extra setup. Mounting the docker socket grants the app **root on the host**; the feature itself stays disabled until an admin enables **Containers** on `/admin/services` (and the `ppy` image is built once with `make ppy`), but treat the whole deployment as trusted-admins-only. Always update through the make targets - a plain `docker compose up -d` drops the overlay and silently removes the socket and CLI.
### Updating
@@ -597,8 +639,7 @@ Changes are promoted through automated DTAP streets: Development (`make dev`), T
2. Validate each touched language (Python compiles/imports, JS parses, CSS and HTML balance)
3. `make test` - run all tests (fail-fast)
4. Add Playwright tests in `tests/test_*.py` for new functionality
5. For visual features: `falcon take --output /tmp/verify.png && falcon describe /tmp/verify.png`
6. Update `AGENTS.md` and `README.md` if new conventions were introduced
5. Update `AGENTS.md` and `README.md` if new conventions were introduced
## License