retoor/devplacepy
1
0
Fork 1
Code Issues 44 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

chore: consolidate runtime data layout under single data/ root directory

Browse Source 
Migrate all runtime artifacts (database, uploads, VAPID keys, locks, bot state, container workspaces, zip staging) from scattered locations (`var/`, `devplacepy/static/uploads/`) into a unified `data/` directory. Update `config.py` as single source of truth with `DATA_PATHS` registry and `ensure_data_dirs()`, add `devplace migrate-data` CLI command with CRC verification and idempotent relocation, adjust `.dockerignore`, `.env.example`, `.gitignore`, `Dockerfile`, `Makefile`, `README.md`, `AGENTS.md`, `CLAUDE.md`, and all import paths in `attachments.py` to reference `config.UPLOADS_DIR`/`ATTACHMENTS_DIR` instead of computing from `STATIC_DIR`.
This commit is contained in:
retoor
2026-06-13 19:06:43 +00:00
parent de745f7a75
commit 1294c95bed
24 changed files with 378 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+13 -9
View File
@@ -120,8 +120,8 @@ The log is **administrator-only**. `/admin/audit-log` is a paginated, filterable
| Env var | Default | Purpose |
|---------|---------|---------|
| `DEVPLACE_DATABASE_URL` | `sqlite:///devplace.db` | Database connection string |
| `DEVPLACE_DATA_DIR` | `<repo>/var` | Persistent runtime data dir, outside the package and not served via `/static` (zip archives, container workspaces). Point at a volume in production |
| `DEVPLACE_DATABASE_URL` | `sqlite:///<repo>/data/devplace.db` | Database connection string |
| `DEVPLACE_DATA_DIR` | `<repo>/data` | Single root for every runtime/user-generated artifact (DB, uploads, VAPID keys, locks, bot state, job staging, container workspaces), outside the package and not served via `/static`. Point at a volume in production. Defined once in `config.py` (`DATA_PATHS` registry, created by `ensure_data_dirs()`) |
| `SECRET_KEY` | hardcoded fallback | Session signing key |
| `DEVPLACE_VAPID_SUB` | `mailto:retoor@molodetz.nl` | Contact address in the VAPID JWT `sub` claim |
| `DEVPLACE_INTERNAL_BASE_URL` | `http://localhost:10500` | Base URL the platform's own services dial for the AI gateway |
@@ -239,7 +239,7 @@ and its full configuration are documented automatically - including future servi
**Security:** this requires mounting the Docker socket, which grants host root. Every run, exec, lifecycle, and schedule operation is administrator-only; `--privileged` is never used and all docker calls are argument-list subprocesses. The service is disabled by default; an admin enables **Containers** on `/admin/services`. CLI: `devplace containers list | reconcile | prune | prune-builds | gc-workspaces` (`prune-builds` is a one-time cleanup that removes legacy per-project images and the old dockerfiles/builds tables).
**Runtime data** (container workspaces and zip archives) lives in `DEVPLACE_DATA_DIR` (default `var/`), **outside the package and never served via `/static`**. The docker daemon must be able to bind-mount the data dir for `/app`.
**Runtime data** (container workspaces and zip archives) lives in `DEVPLACE_DATA_DIR` (default `data/`), **outside the package and never served via `/static`**. The docker daemon must be able to bind-mount the data dir for `/app`.
### Async job framework and zip downloads
@@ -606,8 +606,8 @@ channel is enabled, and `push.notify_user` is scheduled only when the push chann
### VAPID keys
The server identity is three PEM files generated once at startup in the repository
root: `notification-private.pem`, `notification-private.pkcs8.pem`,
The server identity is three PEM files generated once at startup under
`data/keys/`: `notification-private.pem`, `notification-private.pkcs8.pem`,
`notification-public.pem`. They are git-ignored.
**These keys are the application's identity to the push services. If they are lost or
@@ -691,16 +691,20 @@ Production runs on a single host via Docker Compose: an **app** container (Uvico
### Shared database and uploads
The app container bind-mounts the host project directory (`./` to `/app`) and runs as the host user (`DEVPLACE_UID`/`DEVPLACE_GID`, default `1000`). Because `config.py` resolves the database to an absolute path under the project root, the container reads and writes the **same `devplace.db`** as `make dev`, plus the same `static/uploads/`, `devii_lessons.db`, `devii_tasks.db`, VAPID keys, and `devplace-services.lock`. SQLite WAL mode allows the concurrent access, and the file lock (`fcntl.flock` on `devplace-services.lock`) guarantees only one process - dev or prod - runs the background services (news, bots, Devii hub), so they never double-fire.
Every runtime artifact lives under one consolidated `data/` directory (`DEVPLACE_DATA_DIR`, default `<repo>/data`), defined once in `config.py` via the `DATA_PATHS` registry and created at startup by `ensure_data_dirs()`: `data/devplace.db` (+ `-wal`/`-shm`), `data/devii_tasks.db`, `data/devii_lessons.db`, `data/uploads/` (attachments and project files), `data/keys/` (VAPID), `data/locks/`, `data/bot/`, and the job staging / container workspace dirs. Nothing runtime is written inside the package any more.
The app container bind-mounts the host project directory (`./` to `/app`) and runs as the host user (`DEVPLACE_UID`/`DEVPLACE_GID`, default `1000`). Because `config.py` resolves every path to an absolute location under the project's `data/` dir, the container reads and writes the **same `data/devplace.db`** as `make dev`, plus the same `data/uploads/`, `data/devii_*.db`, `data/keys/`, and `data/locks/devplace-services.lock`. SQLite WAL mode allows the concurrent access, and the file lock (`fcntl.flock` on `data/locks/devplace-services.lock`) guarantees only one process - dev or prod - runs the background services (news, bots, Devii hub), so they never double-fire.
This requires prod and dev to be on the **same host/filesystem**; SQLite is a local-file database and cannot be shared across machines. No `DEVPLACE_DATABASE_URL` override is set, so nothing diverges.
To migrate an older install whose data was scattered across the repo root, the package, `var/`, and `$HOME`, stop the app and run `devplace migrate-data --dry-run` (review the plan), then `devplace migrate-data`. It is idempotent and copy-before-delete (verify CRC, atomic replace, then unlink the source), so it is safe to re-run.
### First deploy
```bash
cp .env.example .env # set SECRET_KEY; adjust PORT, DEVPLACE_SITE_URL
make docker-build # build the image (installs the docker CLI)
make docker-up # start (creates ./var, mounts the socket)
make docker-up # start (creates ./data, mounts the socket)
```
Open `http://<host>:${PORT}` (default 10500). `make docker-logs` tails output; `make docker-down` stops.
@@ -720,14 +724,14 @@ The `make deploy` target fast-forwards the `production` branch (`git checkout pr
### Container Manager wiring (what the overlay does)
The Container Manager drives the host Docker daemon, so `make docker-build`/`make docker-up` apply `docker-compose.containers.yml` automatically. The make targets derive everything the overlay needs - `DOCKER_GID` from the socket and `DEVPLACE_DATA_DIR` as the project's own `./var` at its real host path - so no `sudo`, no `/srv` dir, and no manual `.env` editing are required. (Override `DEVPLACE_DATA_DIR` or `DOCKER_GID` on the make command line to point elsewhere.)
The Container Manager drives the host Docker daemon, so `make docker-build`/`make docker-up` apply `docker-compose.containers.yml` automatically. The make targets derive everything the overlay needs - `DOCKER_GID` from the socket and `DEVPLACE_DATA_DIR` as the project's own `./data` at its real host path - so no `sudo`, no `/srv` dir, and no manual `.env` editing are required. (Override `DEVPLACE_DATA_DIR` or `DOCKER_GID` on the make command line to point elsewhere.)
What the overlay (`docker-compose.containers.yml`) changes:
- **Docker CLI in the image** via the `INSTALL_DOCKER_CLI=true` build arg (the base image stays lean).
- **Docker socket** mounted into the app container. This grants the app **root on the host** - every build/run/exec is admin-only, `--privileged` is never used, and all docker calls are argument-list subprocesses, but treat the whole feature as trusted-admins-only.
- **Socket permissions:** the app runs as UID 1000, so the overlay adds the host `docker` group via `group_add`. `make` reads the gid straight from `/var/run/docker.sock` (`stat -c '%g'`), the exact group that owns the socket.
- **Data dir at a consistent path (critical).** When the app (in its container) runs `docker run -v <path>:/app`, the daemon resolves `<path>` against the **host**, not the app container. So the workspace/data dir must be mounted at the **same absolute path** on host and in the container - the make targets set `DEVPLACE_DATA_DIR` to the project's `./var` (an absolute host path) and mount it at that identical path on both sides. (Build contexts go through the docker API as a tarball, so they can stay in the container's temp dir - only the `/app` bind mount needs path consistency.)
- **Data dir at a consistent path (critical).** When the app (in its container) runs `docker run -v <path>:/app`, the daemon resolves `<path>` against the **host**, not the app container. So the workspace/data dir must be mounted at the **same absolute path** on host and in the container - the make targets set `DEVPLACE_DATA_DIR` to the project's `./data` (an absolute host path) and mount it at that identical path on both sides. (Build contexts go through the docker API as a tarball, so they can stay in the container's temp dir - only the `/app` bind mount needs path consistency.)
- **Ingress reach:** published container ports live on the **host**, so the overlay sets `DEVPLACE_CONTAINER_PROXY_HOST=host.docker.internal` (with `extra_hosts: host-gateway`) so the `/p/<slug>` proxy can reach them. On a bare-metal `make prod` deploy the app is already on the host, so the default `127.0.0.1` works and no overlay is needed (just install the docker CLI and run the services).
Then enable **Container builds** and **Containers** on `/admin/services`. Builds default to `--network=host` (configurable on the service) so pip can reach PyPI; set the build network to empty to use the docker default.