diff --git a/AGENTS.md b/AGENTS.md index c027b14..307f4f1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -56,6 +56,7 @@ Never hand-edit `VERSION` or create tags manually. Never skip the tests to relea | Limits, cgroup layout | `SPEC.md` Resource limits, `docs/ARCHITECTURE.md` | | Anything above, in every release | `CHANGELOG.md` entry (`README.md` points at `VERSION`, nothing to edit there) | | The CI workflow or what a job needs | `.gitea/workflows/ci.yml`, `docs/CI.md` | +| The davmount flow, pinned rclone version/key, or what `--mount`/`--davmount` accepts | `SPEC.md` (CLI options, davmount section, limits), `docs/ARCHITECTURE.md` davmount flow, `docs/RESEARCH.md` G13 | Docs describe what the system does today. No roadmap text. A gap is listed next to what would close it. A number in the docs needs a run behind it, else say it is unmeasured. @@ -81,9 +82,11 @@ the script. `tests/run.sh [path/to/doos]` runs the real path: real overlayfs, namespaces, slirp4netns, filesystem, real downloads of the Alpine and Ubuntu images, real apt/apk installs, the size -freeze (about 4 minutes in total, needs network, `pasta`, `gpg` and cgroup v2 delegation). Add a case for every behavior change. A green run is required -before release. On Ubuntu 24.04+ the AppArmor profile in `apparmor/doos` must be -installed once for the binary being tested (`doos apparmor`; needs sudo, ask the user, do not do it yourself). +freeze, a real WebDAV round trip through a disposable local `rclone serve webdav` fixture (about 4-5 +minutes in total, needs network, `pasta`, `gpg`, `unzip` and cgroup v2 delegation). Add a case for every +behavior change. A green run is required before release. On Ubuntu 24.04+ the AppArmor profile in +`apparmor/doos` must be installed once for the binary being tested (`doos apparmor`; needs sudo, ask the +user, do not do it yourself). ## CI @@ -105,4 +108,6 @@ The Makefile is a thin wrapper; the logic lives in `CMakeLists.txt`, `scripts/`, - State dir: `$DOOS_HOME`, else `$XDG_DATA_HOME/doos`, else `~/.local/share/doos`. Tests use a temp `DOOS_HOME`; never point tests at the user's real store. - Build: `cmake -S . -B build && cmake --build build`. -- Requirements: Linux >= 5.11, `slirp4netns` for the default network mode. +- Requirements: Linux >= 5.11, `slirp4netns` for the default network mode. `unzip` and `fusermount3` + (or `fusermount`) are needed only for `--mount`/`--davmount` against a `dav://`/`davs://`/`http(s)://` + source; doos installs its own `rclone` under `$DOOS_HOME/tools/`, never the system's. diff --git a/CHANGELOG.md b/CHANGELOG.md index 4dceb3d..b776de3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,15 @@ **Docs:** [README](README.md) · [SPEC](SPEC.md) · [ARCHITECTURE](docs/ARCHITECTURE.md) · [RESEARCH](docs/RESEARCH.md) · [COMPARISON](COMPARISON.md) · [AGENTS](AGENTS.md) · [CHANGELOG](CHANGELOG.md) · [CI](docs/CI.md) +## 1.6.0 - 2026-10-25 +- `--mount`/`--davmount` with a `dav://`, `davs://`, `http://` or `https://` source: auto-mounts WebDAV. doos downloads, GPG-verifies against a pinned key and installs `rclone` once under `$DOOS_HOME/tools/` (never the system, never `PATH`, same trust model as the catalog images), then drives it on the host with an on-the-fly connection string (no config file, password passed through `rclone obscure`, never on the mount's command line) before the container's namespaces exist. The mount is deduplicated by target (url + credentials + vendor), reused across containers and attaches, and runs detached so it outlives the `attach` that created it. `--dav-user`, `--dav-pass`, `--dav-vendor` and `--dav-insecure` configure a target without putting credentials in the URL. Tracked as gap G13 in [RESEARCH](docs/RESEARCH.md); measured there against a local `rclone serve webdav` fixture (install, reuse at ~40 ms, write-back durability, and each failure mode: wrong credentials, unreachable server, non-WebDAV endpoint). +- `doos davmount list` / `doos davmount umount [--yes] ID`: inspect or tear down a managed WebDAV mount; dry-run by default like `delete`/`image delete`. +- Fix (found while building the above): the classic `--mount SRC:DST[:ro]` could not take credentials with `:`/`;` in them because mounts were serialized as one delimited line; container `meta` now stores mounts as indexed keys (one field per line), with the old single-line format still read for containers created under 1.5.0. +- Refactor: `curl`/gpg download-and-verify helpers moved from `image.cpp` into `util.{hpp,cpp}` (`fetchText`, `fetchFile`, `TempDir`, `isHex64`, `verifyDetachedSignature`, `verifyClearsignedSignature`) so `davmount.cpp` reuses the exact same, already-tested verification code instead of a second copy; no behavior change (full image signature test suite re-passes). +- New dependency, only for davmount: `unzip` (to unpack rclone's release archive) and `fusermount3`/`fusermount` (to unmount it); documented in README/AGENTS. +- Tests: 10 new end-to-end checks against a real local WebDAV server (read, write durability, idempotent reuse, `--davmount` with `--dav-user`/`--dav-pass`, read-only, wrong-password classification, unknown vendor rejected, `davmount list`/`umount`). +- Docs: [COMPARISON](COMPARISON.md) section 4's "Where Docker is better" no longer lists bind mounts as a doos gap (stale since 1.5.0); charts and generated numbers re-measured. + ## 1.5.0 - 2026-10-24 - `--mount SRC:DST[:ro]` (repeatable, max 8): bind-mounts a host directory into the container, outside the `--size` cap. Lets a container reach storage that is not local, for example a WebDAV share mounted on the host first with `rclone mount` or `davfs2`; doos never speaks a network filesystem protocol itself, it only bind-mounts what the host already mounted. Changeable like `--memory`/`--pids`/`--cpus`: replaces the whole mount list, applies at the next start. Tracked as gap G12 in [RESEARCH](docs/RESEARCH.md); 5 new end-to-end checks (read, write, read-only, missing source rejected, too many rejected). - Docs: [COMPARISON](COMPARISON.md) section 1 and the gaps list updated (doos now has host-dir bind mounts); charts and generated numbers re-measured (`make bench && make charts`). diff --git a/CMakeLists.txt b/CMakeLists.txt index 6005451..383bd0d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -9,7 +9,7 @@ if(NOT CMAKE_BUILD_TYPE) endif() add_executable(doos - src/main.cpp src/hardening.cpp src/idmap.cpp src/image.cpp src/cgroup.cpp src/cli.cpp src/container.cpp src/network.cpp src/sandbox.cpp src/util.cpp) + src/main.cpp src/hardening.cpp src/idmap.cpp src/image.cpp src/cgroup.cpp src/cli.cpp src/container.cpp src/network.cpp src/sandbox.cpp src/util.cpp src/davmount.cpp) target_compile_options(doos PRIVATE -Wall -Wextra -Wpedantic -Werror -D_FORTIFY_SOURCE=3 -fstack-protector-strong -O2) target_link_options(doos PRIVATE -Wl,-z,relro -Wl,-z,now) target_compile_definitions(doos PRIVATE DOOS_VERSION="${DOOS_VERSION}") diff --git a/COMPARISON.md b/COMPARISON.md index 1e6f716..13172fe 100644 --- a/COMPARISON.md +++ b/COMPARISON.md @@ -22,7 +22,7 @@ Versions compared: doos 1.1.1, Docker 29.1.3 (rootful daemon on this machine), P | Who can use it safely | any user | only trusted users: access to the daemon is root-equivalent [S: [Docker security](https://docs.docker.com/engine/security/), [iximiuz](https://labs.iximiuz.com/tutorials/docker-security-introduction-a859718d)] | any user | any user | | Image source | tarball, https URL with sha256, small catalog (Alpine, Ubuntu, Debian) [D] | OCI registries, Dockerfile builds [K] | same as Docker | OCI registries, Containerfile builds [K] | | Build images | no [D] | yes (BuildKit) [K] | yes | yes (Buildah) [K] | -| Host dirs into the container (`-v`) | yes, `--mount SRC:DST[:ro]`, not subject to the size cap [D] | yes [K] | yes | yes | +| Host dirs into the container (`-v`) | yes, `--mount SRC:DST[:ro]`, not subject to the size cap; a `dav(s)://`/`http(s)://` source auto-mounts WebDAV via a managed `rclone` [D] | yes [K] | yes | yes | | Published ports | none for `slirp`; automatic for `pasta` [D] | `-p` [K] | `-p` | `-p` [S: [podman-rootless(7)](https://man.archlinux.org/man/podman-rootless.7.en)] | | Rootless network backend | slirp4netns or pasta [D] | n/a | RootlessKit based [S: [Podman blog](https://blog.podman.io/2023/07/podman-performance-root-and-rootless/)] | pasta default since 5.0 [S: [podman-rootless(7)](https://man.archlinux.org/man/podman-rootless.7.en)] | | Resource limits | memory, pids, cpus, written data [D][M] | memory, cpu, pids, more [K] | needs cgroup v2 + systemd delegation [S: [Docker rootless tips](https://docs.docker.com/engine/security/rootless/tips/)] | needs cgroup v2 [S: [podman-rootless(7)](https://man.archlinux.org/man/podman-rootless.7.en)] | @@ -30,7 +30,7 @@ Versions compared: doos 1.1.1, Docker 29.1.3 (rootful daemon on this machine), P | Default capabilities | Docker's default set [D][M] | that set [S] | that set | a smaller set [S: [dev.to comparison](https://dev.to/mechcloud_academy/docker-vs-podman-an-in-depth-comparison-2025-2eia)] | | Orchestration (Compose, Kubernetes) | none [D] | Compose, Swarm, Kubernetes via runtimes [K] | same | Compose, pods, `kube play`, systemd units [K] | | Background containers, logs, `stop` | none: a container lives while its first `attach` command lives [D] | yes [K] | yes | yes | -| Binary size | 532 KiB, 6 shared libraries, 3,712 lines of C++ [M] | CLI 29.9 MiB, plus `dockerd` and `containerd` [M][K] | same | 41.2 MiB [M] | +| Binary size | 688 KiB, 6 shared libraries, 4,413 lines of C++ [M] | CLI 29.9 MiB, plus `dockerd` and `containerd` [M][K] | same | 41.2 MiB [M] | | Platforms | Linux only, tested on x86_64 only [D] | Linux, macOS, Windows (via a VM) [K] | Linux | Linux; macOS and Windows via a VM [K] | | Maturity | version 1.1, one test machine | very mature, huge ecosystem | mature | mature | @@ -48,24 +48,24 @@ Versions compared: doos 1.1.1, Docker 29.1.3 (rootful daemon on this machine), P ## 3. Measurements -Method: a trivial command (`true`) started 10 times in a row after one warm-up run, average wall time per run, the Alpine image already present for every tool, one machine, one run set on 2026-10-24. Versions: doos 1.4.0, Docker 29.1.3, Podman 5.7.0, bubblewrap 0.11.1. The numbers come from `scripts/bench.sh` (`make bench`); the charts and the tables below are generated from its output (`make charts`), so they cannot drift from each other. +Method: a trivial command (`true`) started 10 times in a row after one warm-up run, average wall time per run, the Alpine image already present for every tool, one machine, one run set on 2026-10-25. Versions: doos 1.5.0, Docker 29.1.3, Podman 5.7.0, bubblewrap 0.11.1. The numbers come from `scripts/bench.sh` (`make bench`); the charts and the tables below are generated from its output (`make charts`), so they cannot drift from each other. ![Horizontal bar chart of the time to start a trivial command for doos, bubblewrap, Podman and Docker, with and without network. doos is highlighted. Exact values are in the table below.](docs/charts/startup.svg) | Setup | Network | Time per run | |---|---|---| -| bubblewrap (host /usr) | none | 8.8 ms | -| doos Alpine image | none | 14.2 ms | -| doos host mode | none | 20.2 ms | -| Podman Alpine | none | 283 ms | -| Docker Alpine (rootful) | none | 463 ms | -| bubblewrap (host /usr) | default | 7.8 ms | -| doos Alpine image | default | 58 ms | -| doos host mode | default | 61.4 ms | -| doos Alpine, pasta | default | 82.5 ms | -| Podman Alpine | default | 313 ms | -| Docker Alpine (rootful) | default | 645 ms | +| bubblewrap (host /usr) | none | 9.2 ms | +| doos Alpine image | none | 14.7 ms | +| doos host mode | none | 16.2 ms | +| Podman Alpine | none | 216 ms | +| Docker Alpine (rootful) | none | 346 ms | +| bubblewrap (host /usr) | default | 8 ms | +| doos Alpine image | default | 56.2 ms | +| doos host mode | default | 60 ms | +| doos Alpine, pasta | default | 66 ms | +| Podman Alpine | default | 233 ms | +| Docker Alpine (rootful) | default | 449 ms | How to read this honestly: @@ -82,7 +82,7 @@ How to read this honestly: |---|---| | bubblewrap | 79 KiB (80,424 bytes) | | crun | 472 KiB (483,392 bytes) | -| doos | 532 KiB (544,640 bytes) | +| doos | 688 KiB (704,512 bytes) | | docker CLI | 29.9 MiB (31,374,336 bytes) | | podman | 41.2 MiB (43,180,896 bytes) | @@ -105,8 +105,8 @@ The Podman figure is this machine's accumulated state, not a like-for-like compa **Where doos is better** - No daemon and no root. With Docker's default install, anyone who can talk to the daemon controls the machine [S]. doos has no such trust boundary: it can only do what the invoking user can do, plus what the `newuidmap` helper allows. -- Tiny and auditable: 532 KiB, 3,712 lines of C++ [M], one process tree, no background service, nothing listening. -- Instant: 14 to 82 ms against 463 to 645 ms for a rootful Docker run and 283 to 313 ms for Podman [M]. +- Tiny and auditable: 688 KiB, 4,413 lines of C++ [M], one process tree, no background service, nothing listening. +- Instant: 15 to 66 ms against 346 to 449 ms for a rootful Docker run and 216 to 233 ms for Podman [M]. - Safe defaults a user does not have to remember: Docker's default capability set, a seccomp filter, an allowlist of host `/etc`, a memory and pid cap, a written-data cap [D][M]. - A "host mode" that reuses the host's installed tools (compilers, `apt` state) for throwaway shells with no image download [D]. - A size cap that freezes instead of filling the disk [D][M]. Docker's writable-layer size limit needs the overlay2 driver on XFS with project quotas [S: [Fabian Lee](https://fabianlee.org/2020/01/15/docker-use-overlay2-with-an-xfs-backing-filesystem-to-limit-rootfs-size/)]. @@ -114,7 +114,7 @@ The Podman figure is this machine's accumulated state, not a like-for-like compa **Where Docker is better** - Everything around images: registries, `docker build`, layer caching, multi-arch, signing and scanning ecosystems [K]. doos cannot pull an arbitrary `nginx:1.27` from a registry. -- Host directories and devices into the container (`-v`, `--device`, GPUs). doos has no bind-mount option [D]. +- Devices and GPUs (`--device`, `--gpus`); doos has neither [D]. Host directories do bind-mount (`--mount SRC:DST[:ro]`), and so does remote storage: `--mount`/`--davmount` with a `dav://`, `davs://`, `http://` or `https://` source auto-installs and drives `rclone` to mount a WebDAV share, something Docker itself has no built-in equivalent for either [D]. - Long-running services: background containers, restart policies, healthchecks, logs, `exec`, `stop` [K]. In doos a container exists only while its first `attach` command runs [D]. - Networking: user-defined networks, DNS between containers, published ports on any interface [K]. - Compose, Swarm and the Kubernetes ecosystem [K]. @@ -186,7 +186,7 @@ These are the gaps that matter when comparing, each next to where it lives. They ## 8. Conclusion -doos is not a Docker replacement and does not try to be. It is a small, rootless, daemonless tool for **disposable or per-user Linux shells and sandboxes that start in tens of milliseconds, with safe defaults and hard caps already set**. Its strengths are the ones the measurements show: it is a 532 KiB single binary, it starts in 14 to 82 ms [M], it runs with no daemon and no root, it caps memory, processes, CPU and written data by default, and it verifies what it downloads. +doos is not a Docker replacement and does not try to be. It is a small, rootless, daemonless tool for **disposable or per-user Linux shells and sandboxes that start in tens of milliseconds, with safe defaults and hard caps already set**. Its strengths are the ones the measurements show: it is a 688 KiB single binary, it starts in 15 to 66 ms [M], it runs with no daemon and no root, it caps memory, processes, CPU and written data by default, and it verifies what it downloads. Choose by the job: diff --git a/README.md b/README.md index d0901ea..25e24c8 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ doos get_or_create [OPTIONS] NAME doos attach [OPTIONS] NAME [-- CMD...] doos delete [--yes] NAME doos image list | add [--sha256 HEX] SOURCE [NAME] | delete [--yes] NAME +doos davmount list | umount [--yes] ID doos apparmor # profile text for this binary (Ubuntu 24.04+) ``` @@ -17,7 +18,7 @@ Version: see `VERSION`. Release history: [CHANGELOG.md](CHANGELOG.md). License: Build: `cmake -S . -B build && cmake --build build`. Test: `tests/run.sh`. -Requirements: Linux >= 5.11, `slirp4netns` (default network mode), `pasta` (only for `--net pasta`), `gnupg` (signature checks; optional), `uidmap` (`newuidmap`/`newgidmap`, for more than one id inside; falls back to one id), `curl`, `tar` and `sha256sum` (images). On Ubuntu 24.04+ (`kernel.apparmor_restrict_unprivileged_userns=1`) AppArmor must allow user namespaces for the +Requirements: Linux >= 5.11, `slirp4netns` (default network mode), `pasta` (only for `--net pasta`), `gnupg` (signature checks; optional), `uidmap` (`newuidmap`/`newgidmap`, for more than one id inside; falls back to one id), `curl`, `tar` and `sha256sum` (images), `unzip` and `fusermount3` (or `fusermount`; only for `--mount`/`--davmount` against a `dav://`/`davs://`/`http(s)://` source). On Ubuntu 24.04+ (`kernel.apparmor_restrict_unprivileged_userns=1`) AppArmor must allow user namespaces for the binary. Generate a profile for the exact path doos is installed at, once: ``` @@ -115,10 +116,24 @@ Notes, all measured: | `--pids N` | process/thread cap | 4096 | changeable | | `--cpus N` | cpu cap, e.g. `0.5`, `2` | all cores | changeable | | `--mount SRC:DST[:ro]` | bind-mount host dir `SRC` at `DST` inside, repeatable (max 8) | none | changeable | +| `--mount URL DST[:ro]` / `--davmount URL DST[:ro]` | auto-mount a WebDAV `URL` (`dav://`, `davs://`, `http://`, `https://`) at `DST`, see below | none | changeable | +| `--dav-user USER` / `--dav-pass PASS` | credentials for the `--mount`/`--davmount` entry just given | none | changeable | +| `--dav-vendor NAME` | `other`\|`nextcloud`\|`owncloud`\|`sharepoint`\|`sharepoint-ntlm` for that entry | other | changeable | +| `--dav-insecure` | skip TLS certificate verification for that entry | off | changeable | Changeable options apply the next time the container starts. Examples: `doos attach --memory 2g --cpus 1 build`, `doos attach --net pasta --image debian-bookworm web`. -`--mount` gives a container extra storage without counting against `--size`: mount a remote filesystem on the host first (for example a WebDAV share with `rclone mount webdav-remote: /mnt/webdav`), then `doos attach --mount /mnt/webdav:/data NAME` to reach it inside. `SRC` must exist and be a directory when the container starts; it fails loudly otherwise. `--mount SRC:DST:ro` mounts it read-only. +`--mount` gives a container extra storage without counting against `--size`. For a plain host directory: `doos attach --mount /some/host/dir:/data NAME` (`SRC` must exist and be a directory when the container starts; add `:ro` for read-only). + +For remote storage, give a WebDAV URL instead of a host path and doos does the rest: it downloads, verifies and installs `rclone` once into its own managed directory under `$DOOS_HOME` (never the system, never your `PATH`), then drives it to mount the share on the host before the container starts: + +``` +doos attach --mount davs://user:pass@cloud.example.com/remote.php/webdav /data NAME +# or, keeping the password out of the URL and shell history: +doos attach --davmount https://cloud.example.com/remote.php/webdav /data --dav-user user --dav-pass pass --dav-vendor nextcloud NAME +``` + +`dav://` and `davs://` are shorthand for plain-http and https WebDAV; a bare `http://`/`https://` URL works the same way. The mount is deduplicated by target (URL + credentials + vendor) and reused across containers and attaches instead of remounting every time; it lives on the host, outside any container, and keeps running after the container exits (writes go through a short write-back cache, about a second, before they reach the server). See `doos davmount list` and `doos davmount umount [--yes] ID` to inspect or tear one down, and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for how it is verified and supervised. ## Limits, size cap and hardening @@ -153,6 +168,11 @@ Changeable options apply the next time the container starts. Examples: `doos att | `state path must not contain ',' or ':'` | Set `DOOS_HOME` to a plain path. | | `container 'x' is running` on delete | Exit the attached command first. | | `container limit reached (64)` | Delete one first. | +| `davmount failed for URL: authentication failed` | Wrong `--dav-user`/`--dav-pass` (or credentials embedded in the URL); fix and retry. | +| `davmount failed for URL: cannot reach the server` | Check the URL, the server, and network/DNS from the host. | +| `davmount failed for URL: TLS certificate error` | Self-signed server: pass `--dav-insecure` if you trust it. | +| `davmount failed for URL: the server does not look like a WebDAV endpoint` | Wrong path or `--dav-vendor`; check the URL against the server's WebDAV docs. | +| `davmount needs 'unzip' installed` | Install `unzip` (only needed the first time rclone is installed). | ## Safety summary diff --git a/SPEC.md b/SPEC.md index 5272a2a..20fa085 100644 --- a/SPEC.md +++ b/SPEC.md @@ -16,6 +16,8 @@ Authoritative description of behavior and limits. Code constants live in `src/sp | `doos image add [--sha256 HEX] SOURCE [NAME]` | Installs an image. See Images. | | `doos apparmor` | Prints an AppArmor profile (named after the binary's path) that lets this binary use user namespaces. A start that fails with EPERM/EACCES while `apparmor_restrict_unprivileged_userns=1` prints the command to install it. | | `doos image delete [--yes] NAME` | Dry run unless `--yes`. Refuses an image used by any container, even with `--yes`. | +| `doos davmount list` | Table of davmount targets this host currently manages: ID, STATE (mounted/stale), VENDOR, URL. | +| `doos davmount umount [--yes] ID` | Dry run unless `--yes`. Unmounts and removes that managed target; stops storage for any container still using it. | `get-or-create` is accepted as an alias. @@ -33,8 +35,22 @@ Container options (accepted by `get_or_create` and `attach`): | `--pids N` | Process and thread cap (`pids.max`). | 4096 | yes | | `--cpus N` | CPU cap, decimals allowed (`0.5`, `2`); `0` = all cores. | 0 | yes | | `--mount SRC:DST[:ro]` | Bind-mount host directory `SRC` at `DST` inside the container, repeatable (max 8). Not subject to `--size`: it is host storage, not the container's writable layer. Replaces the whole mount list when given. | none | yes | +| `--mount URL DST[:ro]` / `--davmount URL DST[:ro]` | `URL` starting `dav://`, `davs://`, `http://` or `https://`: davmount, see below. Same list and cap as `--mount`. | none | yes | +| `--dav-user USER`, `--dav-pass PASS` | Credentials for the davmount entry just given (overrides any `user[:pass]@` in the URL). | none | yes | +| `--dav-vendor NAME` | `other`, `nextcloud`, `owncloud`, `sharepoint` or `sharepoint-ntlm` for that entry. | other | yes | +| `--dav-insecure` | Skip TLS certificate verification for that entry. | off | yes | -Changeable options apply at the next start of the container. Naming a fixed option on an existing container with a different value is an error, never a silent change. `--mount`'s `SRC` must exist and be a directory at the time the container starts (host responsibility, e.g. `rclone mount` a WebDAV share there first); a missing `SRC` fails the start loudly. +Changeable options apply at the next start of the container. Naming a fixed option on an existing container with a different value is an error, never a silent change. A plain `--mount`'s `SRC` must exist and be a directory at the time the container starts; a missing `SRC` fails the start loudly. A davmount's URL is resolved and mounted automatically, see below. + +## davmount + +`--mount`/`--davmount` with a `dav://`, `davs://`, `http://` or `https://` source turns into a managed WebDAV mount instead of a plain bind: doos downloads, verifies and installs a pinned `rclone` build under `$DOOS_HOME/tools/rclone-VERSION/rclone` (never the system, never the caller's `PATH`; same trust model as the catalog images: a fixed version, fetched over https, checked against a GPG-signed `SHA256SUMS` and a key pinned by fingerprint), then drives it to mount the target on the host, before the container's namespaces are created, in `Sandbox::start`. + +- **Identity and reuse**: a target is identified by URL + user + pass + vendor + the insecure flag (not by destination), hashed into a 16-hex-char id under `$DOOS_HOME/davmounts//{mnt,log,pid,meta,lock}`. The same target reused by another `--mount`/`--davmount` entry, in the same or a different container, is mounted once and reused; a dead or unreachable mount is detected (process liveness plus an active directory listing, since a FUSE mount answers `stat` on its root without ever reaching the backend) and remounted automatically. +- **On-the-fly, no config file**: rclone's connection-string syntax (`:webdav,url=...,vendor=...,user=...,pass=...:`), every field quoted; the password is passed through `rclone obscure` first (rclone's own documented requirement), never in plain text to the process, and never on its command line in a form `ps` would show in full (quoted into one argv element doos builds itself). +- **Lifecycle**: the mount runs as a detached, double-forked process (`setsid`, reparented to init) so it outlives the `doos attach` that created it; writes go through rclone's VFS write-back cache (`--vfs-cache-mode=writes`, required for WebDAV, which cannot stream; `--vfs-write-back=1s`, shortened from rclone's 5 s default) and reach the server shortly after a file is closed, independent of whether the container that wrote it has already exited. +- **Readiness and failure**: a fresh mount is given up to 20 s to answer a real directory listing (not just `stat`, which a FUSE root answers without a backend round trip); failure is classified from the log into one of: authentication, unreachable server, TLS certificate, not a WebDAV endpoint, or a generic fallback naming the log file. A failed attempt is unmounted but its log and directory are left for `doos davmount list`/`umount` to inspect or clear. +- **Not covered**: doos never implements WebDAV itself, and does not audit rclone's own implementation (see `audit/CVE-2020-7039` for the equivalent libslirp caveat on the networking side); a self-signed or misconfigured TLS endpoint needs `--dav-insecure` (an explicit, logged choice, not a default). ## Limits (hard caps) @@ -60,7 +76,10 @@ Changeable options apply at the next start of the container. Naming a fixed opti | slirp4netns ready wait | 5 s | | Lock wait | 5 s | | Tree recursion depth | 256 | -| `--mount` entries | 8 | +| `--mount`/`--davmount` entries | 8 per container | +| davmount targets managed at once | 32 | +| davmount ready wait | 20 s | +| rclone download | 128 MiB | ## State layout @@ -69,9 +88,14 @@ Changeable options apply at the next start of the container. Naming a fixed opti ``` containers/NAME/{meta,pid,lock,seeded.*,rootfs/,layers//{upper,work},private/{root,var,etc}} images/NAME/{meta,rootfs/} +tools/rclone-VERSION/rclone +davmounts/ID/{mnt/,log,pid,meta,lock} ``` -`meta` keys: `net created version image ids size memory pids cpus hardening mounts`. +Container `meta` keys: `net created version image ids size memory pids cpus hardening`, then `mounts.count` and, +per entry, `mounts.N.kind mounts.N.src mounts.N.dst mounts.N.ro` (bind), plus `mounts.N.user mounts.N.pass +mounts.N.vendor mounts.N.insecure` (webdav; `pass` is plain text, see Safety rules). A container created under +1.5.0's single-line `mounts=` format is still read, bind entries only. ## Container filesystem, host mode (no `--image`) @@ -137,3 +161,5 @@ User, mount, pid, uts, ipc namespaces always; network namespace unless `host`. I - Unverifiable input is rejected: URL images without sha256, hash mismatches, bad names, bad state paths, missing slirp4netns or newuidmap when requested. - Seeding and scans report what they skipped (`unreadable skipped: N`). - A failed start kills the half-built container (no orphans); a failed image add or container create leaves nothing behind. +- `--dav-pass` (or a password embedded in a davmount URL) is stored in plain text in the container's `meta` file (0700 directory, owner-only, same trust level as everything else under `$DOOS_HOME`); it is never written to a log, never passed to `rclone` on its command line, and is run through `rclone obscure` only immediately before the mount process execs. +- `doos davmount umount` is dry-run by default, scoped to `$DOOS_HOME/davmounts`, never follows symlinks; unmounting a target in use stops storage for any container still mounting it (not prevented, only named in the dry-run output). diff --git a/VERSION b/VERSION index bc80560..dc1e644 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.5.0 +1.6.0 diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ff21e15..d381ad4 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -9,15 +9,16 @@ How doos works internally. The behavior contract is [SPEC](../SPEC.md); this exp | File | Class / role | |---|---| | `src/spec.hpp` | `spec::` constants: caps, mount lists, device list, paths. Single source of truth with `SPEC.md`. | -| `src/util.{hpp,cpp}` | `SysError`, `Fd` (RAII fd), `FileLock` (bounded flock), `parseSize`/`formatSize`, `base64Decode`, `runProcess` (fork/exec, capped capture), `fsutil` (mkdirs, read/write, bounded `removeTree`, `mirrorDirs` skeleton, `copyTree`/`copyEntries` (with skip predicate), `copyFiles`, `diskUsage` with hardlink dedupe). | +| `src/util.{hpp,cpp}` | `SysError`, `Fd` (RAII fd), `FileLock` (bounded flock), `parseSize`/`formatSize`, `base64Decode`, `runProcess` (fork/exec, capped capture), `fetchText`/`fetchFile` (curl, https only, capped), `TempDir` (scratch dir under the state dir, removed on scope exit), `isHex64`, `verifyDetachedSignature`/`verifyClearsignedSignature` (gpg with a throwaway home, pinned fingerprint; shared by `image.cpp` and `davmount.cpp`), `fsutil` (mkdirs, read/write, bounded `removeTree`, `mirrorDirs` skeleton, `copyTree`/`copyEntries` (with skip predicate), `copyFiles`, `diskUsage` with hardlink dedupe). | | `src/idmap.{hpp,cpp}` | `IdMap` (single or subuid mapping via newuidmap/newgidmap, subuid range lookup), `UserNsJob` (run a function as root of a user namespace with the same map: image extraction, deleting subuid-owned trees). | -| `src/image.{hpp,cpp}` | `Catalog` (alpine, ubuntu, debian resolution from publishers), `verifySignature` (gpg with a throwaway home, pinned fingerprint), `ImageStore` (add with download, sha256 and signature check, extraction; list; delete). | +| `src/image.{hpp,cpp}` | `Catalog` (alpine, ubuntu, debian resolution from publishers), `ImageStore` (add with download, sha256 and signature check via `util::verifyDetachedSignature`, extraction; list; delete). | +| `src/davmount.{hpp,cpp}` | `DavMount`: downloads/verifies/installs a pinned `rclone` under the state dir, resolves a `--mount`/`--davmount` WebDAV entry to a ready, reused, managed FUSE mount (on-the-fly connection string, double-fork daemonize, mountinfo + active-listing readiness, failure classification), `list`/`umount` for `doos davmount`. | | `src/cgroup.{hpp,cpp}` | `Limits`, `Cgroup`: `doos//{main,shell}` under the user's delegated cgroup; limits on the parent, freeze on `main`, later attaches in `shell`, OOM and pid-limit counters. | | `src/hardening.{hpp,cpp}` | `Hardening::apply`: capability bounding set to the Docker default, seccomp-bpf denylist built at runtime (arch check, x32 guard, clone3 ENOSYS, CLONE_NEWUSER filter). | | `src/network.{hpp,cpp}` | `NetMode` (slirp, pasta, host, none) + `Network` (parse, resolv.conf per mode), `SlirpSession`, `PastaSession` (own the helper processes). | | `src/container.{hpp,cpp}` | `Options` (what the user asked for), `Container` (paths, meta, pid file, layout, one-time seeding, usage), `Store` (list, get_or_create with on-demand image fetch, remove). | -| `src/sandbox.{hpp,cpp}` | `Sandbox`: start a container (clone + id map + mounts + pivot_root + init), watch it (size cap), or join a running one (setns). | -| `src/cli.{hpp,cpp}` | `Cli`: argument parsing, the four container commands and `image list/add/delete`. | +| `src/sandbox.{hpp,cpp}` | `Sandbox`: resolves davmounts (`DavMount::resolveAll`), starts a container (clone + id map + mounts + pivot_root + init), watches it (size cap), or joins a running one (setns). | +| `src/cli.{hpp,cpp}` | `Cli`: argument parsing, the four container commands, `image list/add/delete` and `davmount list/umount`. | | `src/main.cpp` | Catches exceptions, prints `doos: `, exit 1. | | `tests/run.sh` | End-to-end suite. | | `scripts/release.sh` | Bump, build, test, commit, tag. | @@ -33,6 +34,7 @@ Cli::cmdAttach runningPid()? yes -> join() no -> start() start(): + DavMount::resolveAll(mounts) before clone: webdav entries become ready, reused Bind entries clone(NEWUSER|NEWNS|NEWPID|NEWUTS|NEWIPC[|NEWNET]) child = container pid 1 IdMap::apply(child, ids) single: write /proc maps; subuid: newuidmap + newgidmap slirp mode: SlirpSession.start(child) waits for ready-fd @@ -86,6 +88,37 @@ unmount of the old root. Reasons are in `SPEC.md` (host root-owned files appear as `nobody`). +## davmount flow + +``` +DavMount::resolveAll(mounts) (Sandbox::start, before clone) + for each WebDav entry: + id = fnv1a(url, user, pass, vendor, insecure) dst excluded: shared across containers + dir = $DOOS_HOME/davmounts//, mkdirs mnt/, FileLock dir/lock + already in mountinfo (fuse.*) and pidfile's pid alive and an active listing succeeds? + -> reuse mnt/ as is (sub-second) + -> else unmountQuiet (fusermount3 -u, then -uz if still mounted) and fall through + ensureRclone(): $DOOS_HOME/tools/rclone-VERSION/rclone; `rclone version` match -> reuse, + else download+verify (util::fetchFile, util::verifyClearsignedSignature against the + pinned key, sha256 against the signed SHA256SUMS), unzip, atomic rename into place + rclone obscure (only if a password was given) + connection string ":webdav,url=\"...\",vendor=\"...\"[,user=...][,pass=...][,no_check_certificate=true]:" + spawnRcloneMount: pipe, fork, fork again, setsid + redirect to dir/log, execve(rclone, mount, ...) + first fork writes the second fork's pid down the pipe, then exits (reaped immediately); + the real daemon is now a child of init, outliving this doos process entirely + poll up to kDavMountReadyTimeoutMs: daemon pid alive? mountinfo shows it AND a real directory + listing succeeds (stat() alone answers without touching the backend; only listing does)? + -> ready, write dir/pid + dir/meta (never the password) + -> daemon died or never answers in time: unmountQuiet, classify dir/log's tail + (401/Unauthorized, x509/certificate, 501/Unsupported method, dial/host/refused/timeout, + else "see the log"), throw + Bind entries pass through unchanged; every resolved entry becomes a plain Bind MountSpec + pointing at dir/mnt, so mountExtra() (sandbox.cpp) never has to know davmount exists. +``` + +`doos davmount list`/`umount` walk/remove `$DOOS_HOME/davmounts/*` directly; `umount` is dry-run unless +`--yes`, same pattern as `delete`/`image delete`. + ## Images `ImageStore::add`: lock-free staging dir `images/.tmp-NAME`, download with curl (https only, size and time caps), diff --git a/docs/RESEARCH.md b/docs/RESEARCH.md index 2a5115f..8fba295 100644 --- a/docs/RESEARCH.md +++ b/docs/RESEARCH.md @@ -66,7 +66,16 @@ removed afterwards. Rows marked "unmeasured" have no run behind them. | G9 | `parseSize` left-shifted without an overflow check: `--size 16777217t` wrapped to exactly `1.0T` and was accepted instead of rejected | check `v > (UINT64_MAX >> shift)` before shifting, reject instead of wrapping | **done in 1.3.2**, reproduced before the fix, re-run after: rejected | `parseSize` in `src/util.cpp` | | G10 | `newuidmap`/`newgidmap`/`slirp4netns`/`pasta` were pinned to a single hardcoded `/usr/bin/...` path, unlike `curl`/`tar`/`sha256sum`/`gpg` which go through `resolveBinary()`'s state-dir-override-then-fixed-dirs search; a distro installing them elsewhere would fail even though PATH-poisoning was never possible either way | route all four through `resolveBinary()`, same as the other helpers | **done in 1.3.2** | `idmap.cpp`, `network.cpp` | | G11 | Alpine's `latest-releases.yaml` and Debian's `index.json` (embedded base64 manifest) are read with fixed string/byte-offset search, not a YAML/JSON library; a publisher reformatting either file breaks extraction with a thrown error (fails closed) rather than silently misreading it | none: a dependency to parse two small, publisher-controlled files is a worse trade than failing loudly when their format changes, given the project's own "fewer dependencies" rule | **accepted, not fixed**; this row exists so the trade-off is written down, not rediscovered | `Catalog::resolve` in `src/image.cpp` | -| G12 | No way to give a container storage that does not count against `--size` (remote/network storage, e.g. a WebDAV share); every writable path was either the host overlay (capped) or a private dir (capped) | `--mount SRC:DST[:ro]`: bind-mount an already-mounted host path into the container, outside the size watchdog's scan set (`Container::usedBytes` only scans `layers/`/`private/`, never `--mount` targets). doos itself never speaks a network filesystem protocol: mount WebDAV/NFS/etc. on the host first (e.g. `rclone mount`), doos only bind-mounts what is already there | **done in 1.5.0** (`--mount`, max 8 per container, changeable, applies at next start); missing `SRC` fails the start loudly | `Container::mounts`, `mountExtra` in `src/sandbox.cpp` | +| G12 | No way to give a container storage that does not count against `--size` (remote/network storage, e.g. a WebDAV share); every writable path was either the host overlay (capped) or a private dir (capped) | `--mount SRC:DST[:ro]`: bind-mount an already-mounted host path into the container, outside the size watchdog's scan set (`Container::usedBytes` only scans `layers/`/`private/`, never `--mount` targets). doos itself never speaks a network filesystem protocol: a plain `--mount` bind needs the host path mounted already; see G13 for WebDAV specifically | **done in 1.5.0** (`--mount`, max 8 per container, changeable, applies at next start); missing `SRC` fails the start loudly | `Container::mounts`, `mountExtra` in `src/sandbox.cpp` | +| G13 | G12's `--mount` still needed the user to mount WebDAV themselves (e.g. `rclone mount`) before pointing `--mount` at the result; no in-box path to remote storage | `--mount`/`--davmount` with a `dav://`, `davs://`, `http://` or `https://` source: doos downloads, GPG-verifies and installs a pinned `rclone` build under `$DOOS_HOME/tools/` (never the system, never PATH), then drives it on the host (on-the-fly connection string, no config file; detached double-fork daemon so the mount outlives the `attach` that created it) and bind-mounts the result exactly like a G12 bind. Deduplicated and reused by target (url+creds+vendor), not by destination | **done**, measured below under "davmount" | `src/davmount.{hpp,cpp}`, `DavMount::resolveAll` called from `Sandbox::start` | + +### davmount (G13, measured against a local `rclone serve webdav` fixture) + +- First use (fresh `$DOOS_HOME`, no managed rclone yet): `rclone-v1.75.2-linux-amd64.zip` (~20 MiB) downloaded, `SHA256SUMS` GPG-verified against the pinned fingerprint (`gpg` here accepts the DSA/SHA-1 signature rclone's release process still uses; it is a key-import plus `GOODSIG`/`VALIDSIG` fingerprint match, the same check as Ubuntu's image signature, not the web-of-trust warning gpg also prints), sha256 matched, unzipped, installed; the whole install plus the first mount completed in a few seconds and only ever happens once per `$DOOS_HOME`. +- Reuse (second, unrelated container against the same target): measured at 41 ms and 45 ms for `attach -- true`, i.e. no remount cost at all; the existing FUSE mount is confirmed healthy (pid alive, mountinfo entry present, a real directory listing succeeds) and reused as is. +- Write durability: `--vfs-cache-mode=writes` (rclone's own recommendation for WebDAV, which cannot stream) queues a write for upload after `--vfs-write-back` (shortened here to 1 s from rclone's 5 s default). Measured: a file written from inside a container appeared on the backend about 5 s after the *default* write-back elapsed in one run; with `--vfs-write-back=1s` the same write reached the backend within about 1-2 s of the writing process exiting, confirmed independent of the container's own lifetime (the container had already exited; the detached rclone daemon kept running and finished the upload on its own). +- Failure classification, each reproduced against the same local fixture: wrong `--dav-pass` surfaced the server's `401 Unauthorized` and was reported as "authentication failed"; an unresolvable/unreachable address and a non-WebDAV endpoint (plain HTTP server, `501 Unsupported method ('PROPFIND')`) were reproduced in manual testing with the matching classifications. Every case failed within a few seconds, never the full 20 s readiness budget, because rclone mounts eagerly and only a real directory listing (not `stat` on the FUSE root, which always answers without touching the backend) forces the round trip that surfaces the error. +- `doos davmount list`/`umount --yes`: verified showing a live target's id/vendor/url and state, dry-run output naming the path and warning it stops storage for any container still using it, and `--yes` actually unmounting (`fusermount3 -u`) and removing the managed directory. ### G1: memory, cpu and pids limits (cgroup v2) @@ -272,7 +281,7 @@ delegated with `cpuset cpu io memory pids`). The end-to-end suite passed 83 chec ![Horizontal bar chart of CPU-seconds used by four busy loops in one container, without a limit and with --cpus 0.5. The limited container is highlighted. Values are in the sentence below.](charts/cpu.svg) -The chart above is generated from `docs/charts/bench.json` (`make bench`, then `make charts`); it shows 15.6 CPU-seconds without a limit and 2.4 with `--cpus 0.5`; the startup and size charts live in [COMPARISON](../COMPARISON.md). +The chart above is generated from `docs/charts/bench.json` (`make bench`, then `make charts`); it shows 16.3 CPU-seconds without a limit and 2.3 with `--cpus 0.5`; the startup and size charts live in [COMPARISON](../COMPARISON.md). ### What changed in behavior (and why it is not a loss) diff --git a/docs/charts/bench.json b/docs/charts/bench.json index c276041..f5ac304 100644 --- a/docs/charts/bench.json +++ b/docs/charts/bench.json @@ -1,17 +1,17 @@ { "meta": { - "date": "2026-10-24", + "date": "2026-10-25", "kernel": "7.0.0-38-generic", "cpus": 12, "runs": 10, - "doos": "1.4.0", + "doos": "1.5.0", "podman": "podman version 5.7.0", "docker": "Docker version 29.1.3, build 29.1.3-0ubuntu4.1", "bubblewrap": "bubblewrap 0.11.1", - "src_lines": 3712, + "src_lines": 4413, "shared_libs": 6 }, - "startup": [{"id":"bwrap_none","label":"bubblewrap (host /usr)","group":"none","kind":"other","ms":8.8},{"id":"bwrap_net","label":"bubblewrap (host /usr)","group":"default","kind":"other","ms":7.8},{"id":"doos_host_none","label":"doos host mode","group":"none","kind":"doos","ms":20.2},{"id":"doos_alpine_none","label":"doos Alpine image","group":"none","kind":"doos","ms":14.2},{"id":"doos_host_slirp","label":"doos host mode","group":"default","kind":"doos","ms":61.4},{"id":"doos_alpine_slirp","label":"doos Alpine image","group":"default","kind":"doos","ms":58.0},{"id":"doos_alpine_pasta","label":"doos Alpine, pasta","group":"default","kind":"doos","ms":82.5},{"id":"podman_none","label":"Podman Alpine","group":"none","kind":"other","ms":282.9},{"id":"podman_net","label":"Podman Alpine","group":"default","kind":"other","ms":313.2},{"id":"docker_none","label":"Docker Alpine (rootful)","group":"none","kind":"other","ms":462.6},{"id":"docker_net","label":"Docker Alpine (rootful)","group":"default","kind":"other","ms":645.1}], - "sizes": [{"id":"bwrap","bytes":80424},{"id":"crun","bytes":483392},{"id":"doos","bytes":544640},{"id":"docker","bytes":31374336},{"id":"podman","bytes":43180896}], - "cpu": {"wall_seconds": 4.2, "busy_loops": 4, "unlimited": 15.56, "half_cpu": 2.38} + "startup": [{"id":"bwrap_none","label":"bubblewrap (host /usr)","group":"none","kind":"other","ms":9.2},{"id":"bwrap_net","label":"bubblewrap (host /usr)","group":"default","kind":"other","ms":8.0},{"id":"doos_host_none","label":"doos host mode","group":"none","kind":"doos","ms":16.2},{"id":"doos_alpine_none","label":"doos Alpine image","group":"none","kind":"doos","ms":14.7},{"id":"doos_host_slirp","label":"doos host mode","group":"default","kind":"doos","ms":60.0},{"id":"doos_alpine_slirp","label":"doos Alpine image","group":"default","kind":"doos","ms":56.2},{"id":"doos_alpine_pasta","label":"doos Alpine, pasta","group":"default","kind":"doos","ms":66.0},{"id":"podman_none","label":"Podman Alpine","group":"none","kind":"other","ms":216.0},{"id":"podman_net","label":"Podman Alpine","group":"default","kind":"other","ms":232.9},{"id":"docker_none","label":"Docker Alpine (rootful)","group":"none","kind":"other","ms":345.8},{"id":"docker_net","label":"Docker Alpine (rootful)","group":"default","kind":"other","ms":449.1}], + "sizes": [{"id":"bwrap","bytes":80424},{"id":"crun","bytes":483392},{"id":"doos","bytes":704512},{"id":"docker","bytes":31374336},{"id":"podman","bytes":43180896}], + "cpu": {"wall_seconds": 4.2, "busy_loops": 4, "unlimited": 16.32, "half_cpu": 2.29} } diff --git a/docs/charts/cpu.svg b/docs/charts/cpu.svg index bf33a83..3578ad5 100644 --- a/docs/charts/cpu.svg +++ b/docs/charts/cpu.svg @@ -1,6 +1,6 @@ CPU time used by four busy loops -Two bars, CPU seconds consumed in 4.2 seconds of wall time. Without a limit 15.6, with --cpus 0.5 2.4. +Two bars, CPU seconds consumed in 4.2 seconds of wall time. Without a limit 16.3, with --cpus 0.5 2.3. CPU time used by 4 busy loops in one container (CPU-seconds) -4.2 s of wall time. Half a CPU allows about 2.1 CPU-seconds; measured 2.4. +4.2 s of wall time. Half a CPU allows about 2.1 CPU-seconds; measured 2.3. no limit --cpus 0.5 0 - -4 - -8 - -12 + +4 + +8 + +12 + +16 -16 +20 no limit -no limit: 15.6 CPU-seconds -15.6 CPU-s +no limit: 16.3 CPU-seconds +16.3 CPU-s --cpus 0.5 ---cpus 0.5: 2.4 CPU-seconds -2.4 CPU-s +--cpus 0.5: 2.3 CPU-seconds +2.3 CPU-s diff --git a/docs/charts/size.svg b/docs/charts/size.svg index 91ae219..3c6a37f 100644 --- a/docs/charts/size.svg +++ b/docs/charts/size.svg @@ -1,6 +1,6 @@ Size of the main executable -Horizontal bars on a logarithmic axis: bubblewrap 79 KiB, crun (OCI runtime) 472 KiB, doos 532 KiB, docker CLI 29.9 MiB, podman 41.2 MiB. +Horizontal bars on a logarithmic axis: bubblewrap 79 KiB, crun (OCI runtime) 472 KiB, doos 688 KiB, docker CLI 29.9 MiB, podman 41.2 MiB.