blindxfish/devplacepy
1
0
Fork 0
forked from retoor/devplacepy
Code Pull Requests Activity

feat: add container manager CLI commands and docker-compose overlay for admin container lifecycle

Browse Source
This commit is contained in:
retoor
2026-06-09 04:41:27 +00:00
parent 317126efc2
commit c565e3b55c
83 changed files with 6078 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+39 -7
View File
@@ -56,9 +56,11 @@ devplacepy/
| `/news` | Developer news listing, detail page with comments |
| `/posts` | Post detail, creation |
| `/comments` | Comment creation, deletion |
| `/projects` | Project listing, creation |
| `/projects/{slug}/files` | Per-project filesystem: directory and file CRUD, upload, inline editing (public read, owner write) |
| `/projects` | Project listing, creation, and per-project visibility toggles: `POST /projects/{slug}/private` (owner-only visibility) and `POST /projects/{slug}/readonly` (immutable files) |
| `/projects/{slug}/files` | Per-project filesystem: directory and file CRUD, upload, inline editing, and line-range operations (`lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append`) for surgical edits to large text files (public read, owner write; all writes refused while the project is read-only) |
| `/zips` | Zip job status (`/zips/{uid}`) and archive download (`/zips/{uid}/download`); archives are queued via `/projects/{slug}/zip` and `/projects/{slug}/files/zip` |
| `/projects/{slug}/containers` | Admin container manager: Dockerfile CRUD with immutable versions, async image builds, container instance lifecycle, logs, exec, metrics, and schedules |
| `/p/{slug}` | Public ingress proxy (HTTP + WebSocket) to a running container instance's published port, opt-in per instance via `ingress_slug` |
| `/profile` | Profile view, editing |
| `/messages` | Direct messaging |
| `/notifications` | Notification list, mark read, live unread counts (`/notifications/counts`) |
@@ -95,6 +97,8 @@ Member progression is driven by activity and peer recognition.
- **Emoji reactions** - a fixed palette of reactions on posts, comments, gists, and projects, separate from voting and carrying no ranking weight.
- **Polls** - a post can carry a poll (question plus up to six options); results appear as live bars once the viewer votes, one vote per member. A poll can be attached when the post is created or added later by editing a post that has none.
- **Bookmarks** - save posts, gists, projects, and news to a personal list at `/bookmarks/saved`.
- **Private projects** - an owner can mark a project private so it is visible only to them (and administrators) and excluded from listings, profiles, search, the sitemap, and zip access. Set at creation or toggled later from the project page.
- **Read-only projects** - an owner can mark a project read-only, making its entire virtual filesystem immutable: every write, edit, line-edit, move, delete, and upload is refused from all paths (the web UI, the HTTP API, the Devii agent, and container workspace sync) until read-only is turned off. Devii may toggle read-only only after the user explicitly confirms.
XP awards are wired at the existing content-creation, vote, and follow hook points in the routers and centralized in `award_xp()` / `check_milestone_badges()` (`devplacepy/utils.py`). Existing accounts have their XP and levels backfilled once from prior activity at startup (`init_db()`).
@@ -103,6 +107,7 @@ XP awards are wired at the existing content-creation, vote, and follow hook poin
| Env var | Default | Purpose |
|---------|---------|---------|
| `DEVPLACE_DATABASE_URL` | `sqlite:///devplace.db` | Database connection string |
| `DEVPLACE_DATA_DIR` | `<repo>/var` | Persistent runtime data dir, outside the package and not served via `/static` (zip archives, container workspaces). Point at a volume in production |
| `SECRET_KEY` | hardcoded fallback | Session signing key |
| `DEVPLACE_VAPID_SUB` | `mailto:retoor@molodetz.nl` | Contact address in the VAPID JWT `sub` claim |
| `DEVPLACE_INTERNAL_BASE_URL` | `http://localhost:10500` | Base URL the platform's own services dial for the AI gateway |
@@ -114,6 +119,7 @@ Operational behavior is tunable live from `/admin/settings` (stored in `site_set
| Setting | Default | Effect |
|---------|---------|--------|
| `site_url` | empty | Public origin for absolute links (SEO, container ingress `/p/<slug>`); empty derives from the request or `DEVPLACE_SITE_URL` |
| `rate_limit_per_minute` | `60` | Mutating requests allowed per IP per window |
| `rate_limit_window_seconds` | `60` | Rolling window for the request limit |
| `session_max_age_days` | `7` | Standard session cookie lifetime |
@@ -205,6 +211,15 @@ and its full configuration are documented automatically - including future servi
- **`BotsService`** - Playwright fleet of AI personas that browse and interact with a DevPlace instance, with live cost/usage metrics (opt-in; install the `bots` extra)
- **`GatewayService`** - OpenAI-compatible LLM gateway at `/openai/v1/*`, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default)
- **`JobService` / `ZipService`** - generic async job framework (`services/jobs/`) for heavy, blocking work run off the request path; `ZipService` is the first consumer, building project zip archives in a subprocess
- **`ContainerBuildService` / `ContainerService`** - the admin container manager (`services/containers/`): async Docker image builds and a reconciling supervisor for container instances
### Container manager (admin only)
`services/containers/` builds versioned Docker images for projects and runs supervised container instances, all from the web UI, the HTTP API, and Devii. It drives the `docker` CLI via async subprocesses behind a pluggable `Backend` interface (a `DockerCliBackend` plus a `FakeBackend` for tests). `ContainerBuildService` (a `JobService`) builds `<name>:<build-number>` images asynchronously with streamed logs; `ContainerService` reconciles desired instance state against `docker ps` each tick (containers are labeled `devplace.instance=<uid>`, so orphans are reaped and no state is lost), applies restart policies, fires cron/interval/one-time schedules, and samples metrics. Dockerfiles have immutable versions; saving new content auto-builds the next tag (builds run with `--network=host` by default so pip can reach PyPI). The container's `/app` is bind-mounted to a persistent project workspace (materialized from the project files) and can be synced back. A running instance can be **published** with an `ingress_slug`, making its service reachable (HTTP and WebSocket) at `/p/<slug>` through DevPlace. A high-quality default Dockerfile (Debian python-slim) is provided.
**Security:** this requires mounting the Docker socket, which grants host root. Every build, run, exec, lifecycle, and schedule operation is administrator-only; `--privileged` is never used and all docker calls are argument-list subprocesses. The services are disabled by default; an admin enables them on `/admin/services`. CLI: `devplace containers list | reconcile | prune | gc-workspaces`.
**Runtime data** (container workspaces and zip archives) lives in `DEVPLACE_DATA_DIR` (default `var/`), **outside the package and never served via `/static`**; ephemeral build contexts use the OS temp dir. The docker daemon must be able to bind-mount the data dir for `/app`. In development `make dev` scopes `uvicorn --reload` to `--reload-dir devplacepy` so writing build contexts does not restart the server mid-build.
### Async job framework and zip downloads
@@ -517,25 +532,42 @@ This requires prod and dev to be on the **same host/filesystem**; SQLite is a lo
```bash
cp .env.example .env # set SECRET_KEY; adjust PORT, DEVPLACE_SITE_URL
make docker-up # docker compose up -d (builds on first run)
make docker-build # build the image (installs the docker CLI)
make docker-up # start (creates ./var, mounts the socket)
```
Open `http://<host>:${PORT}` (default 10500). `make docker-logs` tails output; `make docker-down` stops.
`make docker-build` and `make docker-up` always apply the `docker-compose.containers.yml` overlay, so the admin-only Container Manager works with no extra setup. Mounting the docker socket grants the app **root on the host**; the feature itself stays disabled until an admin enables **Container builds** and **Containers** on `/admin/services`, but treat the whole deployment as trusted-admins-only. Always update through the make targets - a plain `docker compose up -d` drops the overlay and silently removes the socket and CLI.
### Updating
```bash
git pull
docker compose up -d # restart with new code (bind-mounted, no rebuild)
docker compose build && \
docker compose up -d # only when dependencies in pyproject.toml change
make docker-up # restart with new code (bind-mounted, no rebuild)
make docker-build && \
make docker-up # only when dependencies in pyproject.toml change
```
The `make deploy` target fast-forwards the `production` branch (`git checkout production && git merge master && git push origin production`); pull that branch on the server.
### Container Manager wiring (what the overlay does)
The Container Manager drives the host Docker daemon, so `make docker-build`/`make docker-up` apply `docker-compose.containers.yml` automatically. The make targets derive everything the overlay needs - `DOCKER_GID` from the socket and `DEVPLACE_DATA_DIR` as the project's own `./var` at its real host path - so no `sudo`, no `/srv` dir, and no manual `.env` editing are required. (Override `DEVPLACE_DATA_DIR` or `DOCKER_GID` on the make command line to point elsewhere.)
What the overlay (`docker-compose.containers.yml`) changes:
- **Docker CLI in the image** via the `INSTALL_DOCKER_CLI=true` build arg (the base image stays lean).
- **Docker socket** mounted into the app container. This grants the app **root on the host** - every build/run/exec is admin-only, `--privileged` is never used, and all docker calls are argument-list subprocesses, but treat the whole feature as trusted-admins-only.
- **Socket permissions:** the app runs as UID 1000, so the overlay adds the host `docker` group via `group_add`. `make` reads the gid straight from `/var/run/docker.sock` (`stat -c '%g'`), the exact group that owns the socket.
- **Data dir at a consistent path (critical).** When the app (in its container) runs `docker run -v <path>:/app`, the daemon resolves `<path>` against the **host**, not the app container. So the workspace/data dir must be mounted at the **same absolute path** on host and in the container - the make targets set `DEVPLACE_DATA_DIR` to the project's `./var` (an absolute host path) and mount it at that identical path on both sides. (Build contexts go through the docker API as a tarball, so they can stay in the container's temp dir - only the `/app` bind mount needs path consistency.)
- **Ingress reach:** published container ports live on the **host**, so the overlay sets `DEVPLACE_CONTAINER_PROXY_HOST=host.docker.internal` (with `extra_hosts: host-gateway`) so the `/p/<slug>` proxy can reach them. On a bare-metal `make prod` deploy the app is already on the host, so the default `127.0.0.1` works and no overlay is needed (just install the docker CLI and run the services).
Then enable **Container builds** and **Containers** on `/admin/services`. Builds default to `--network=host` (configurable on the service) so pip can reach PyPI; set the build network to empty to use the docker default.
### nginx specifics
- **WebSockets:** `/devii/ws` (the Devii terminal) is proxied with `Upgrade`/`Connection` headers and a 1h read timeout.
- **WebSockets:** `/devii/ws` (the Devii terminal) and `/p/` (container ingress) are proxied with `Upgrade`/`Connection` headers and 1h timeouts; `/p/` also disables buffering for streaming.
- **Uploads:** `/static/uploads/` is served with `X-Content-Type-Options: nosniff` and a `Content-Disposition` that mirrors the app's `UploadStaticFiles` control: known-safe media (images, video, audio) is served `inline` so it embeds and plays in the page, while every other type is forced to `attachment` so it downloads instead of rendering (stored-XSS defense). SVG is deliberately excluded from the inline set.
- **Upload size:** `NGINX_MAX_BODY_SIZE` (default `50m`) must be **>= the admin-configurable `max_upload_size_mb`** (Admin -> Settings), or large uploads are rejected with HTTP 413 before reaching the app.
- **Micro-cache:** off by default; enable with `NGINX_CACHE_ENABLED=true`.