retoor/doos
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

doos 1.0.0: images, size cap, subuid ids, working apt in host mode

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
retoor
2026-10-23 21:41:28 +02:00
co-authored by Claude Sonnet 5.5
parent fdb96c7b1c
commit d446c29372
24 changed files with 1709 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+47 -16
View File
@@ -7,11 +7,14 @@ How doos works internally. Behavior contract is `SPEC.md`; this explains the cod
| File | Class / role |
|---|---|
| `src/spec.hpp` | `spec::` constants: caps, mount lists, device list, paths. Single source of truth with `SPEC.md`. |
| `src/util.{hpp,cpp}` | `SysError` (errno-carrying exception), `Fd` (RAII fd), `FileLock` (bounded flock), `fsutil` (mkdirs, read/write, bounded `removeTree`). |
| `src/util.{hpp,cpp}` | `SysError`, `Fd` (RAII fd), `FileLock` (bounded flock), `parseSize`/`formatSize`, `runProcess` (fork/exec, capped capture), `fsutil` (mkdirs, read/write, bounded `removeTree`, `mirrorDirs` skeleton, `copyTree`, `copyFiles`, `diskUsage` with hardlink dedupe). |
| `src/idmap.{hpp,cpp}` | `IdMap` (single or subuid mapping via newuidmap/newgidmap, subuid range lookup), `UserNsJob` (run a function as root of a user namespace with the same map: image extraction, deleting subuid-owned trees). |
| `src/image.{hpp,cpp}` | `Catalog` (alpine, ubuntu resolution from publishers), `ImageStore` (add with download, sha256 check, extraction; list; delete). |
| `src/cgroup.{hpp,cpp}` | `Cgroup`: one cgroup v2 leaf per running container under the user's delegated cgroup, used for `cgroup.freeze`. |
| `src/network.{hpp,cpp}` | `NetMode` + `Network` (parse, resolv.conf per mode), `SlirpSession` (owns the slirp4netns process). |
| `src/container.{hpp,cpp}` | `Container` (one container's paths, meta, pid file, layout), `Store` (list, get_or_create, remove). |
| `src/sandbox.{hpp,cpp}` | `Sandbox`: start a container (clone + mounts + pivot_root + init) or join a running one (setns). |
| `src/cli.{hpp,cpp}` | `Cli`: argument parsing and the four commands. |
| `src/container.{hpp,cpp}` | `Options` (what the user asked for), `Container` (paths, meta, pid file, layout, one-time seeding, usage), `Store` (list, get_or_create with on-demand image fetch, remove). |
| `src/sandbox.{hpp,cpp}` | `Sandbox`: start a container (clone + id map + mounts + pivot_root + init), watch it (size cap), or join a running one (setns). |
| `src/cli.{hpp,cpp}` | `Cli`: argument parsing, the four container commands and `image list/add/delete`. |
| `src/main.cpp` | Catches exceptions, prints `doos: <message>`, exit 1. |
| `tests/run.sh` | End-to-end suite. |
| `scripts/release.sh` | Bump, build, test, commit, tag. |
@@ -28,13 +31,15 @@ Cli::cmdAttach
start():
clone(NEWUSER|NEWNS|NEWPID|NEWUTS|NEWIPC[|NEWNET]) child = container pid 1
parent writes /proc/<child>/{setgroups,uid_map,gid_map} 0 -> invoking uid, size 1
IdMap::apply(child, ids) single: write /proc maps; subuid: newuidmap + newgidmap
slirp mode: SlirpSession.start(child) waits for ready-fd
Cgroup::create + add(child) best effort; without it the size cap only warns
recordPid(child) (pid + starttime, defeats pid reuse)
lock.unlock() joiners must not wait for container lifetime
write 'G' on sync pipe child continues
parent: ignore SIGINT/SIGQUIT, forward SIGTERM/SIGHUP, waitpid(child)
on exit: pid file cleared, slirp stopped (RAII)
parent: ignore SIGINT/SIGQUIT, forward SIGTERM/SIGHUP, watch(child)
watch: poll(pidfd, 10 s); each wake scans usage -> warn 80%, freeze 100%, thaw <95%
on exit: pid file cleared, slirp stopped, cgroup removed (RAII)
child (containerInit, pid 1):
PDEATHSIG=SIGKILL, close inherited lock fd, wait for 'G'
@@ -45,15 +50,41 @@ child (containerInit, pid 1):
## setupRoot order
1. `MS_REC|MS_PRIVATE` on `/` so nothing propagates to the host.
2. Bind the container `rootfs/` onto itself (pivot_root needs a mountpoint).
3. Write `resolv.conf` into the `/etc` upper dir (matches network mode every start).
4. Overlay mounts (`lowerdir=<host dir>,upperdir,workdir,userxattr`, nosuid, nodev).
5. Compat dirs: recreate host symlinks (usr-merge) or overlay real dirs.
6. Bind private dirs (`/root`, `/var`), mount tmpfs (`/tmp`, `/run`).
7. `/dev`: tmpfs, bind host device nodes, new devpts, `/dev/shm`, symlinks.
8. `/proc`, then read-only remount and masks from `spec.hpp`.
9. `chdir(rootfs)`, `pivot_root(".", ".")`, lazy unmount of old root.
Common start: `MS_REC|MS_PRIVATE` on `/`, then `resolv.conf` is written into the container's `/etc` location
(`Container::etcDir`: `private/etc` in host mode, the root upper layer in image mode), and on a single-id
container with apt present an `APT::Sandbox::User "root"` override is written next to it.
Host mode:
1. Bind the container `rootfs/` onto itself (pivot_root needs a mountpoint).
2. Overlay mounts for `/usr`, `/opt` and real compat dirs (`lowerdir=<host dir>,upperdir,workdir,userxattr`).
3. Compat symlinks recreated (usr-merge).
4. Bind the private dirs `/root`, `/var`, `/etc`; then the overlay for `/var/lib/dpkg` inside the private `/var`.
Image mode:
1. One overlay on `rootfs/`: lower = `images/NAME/rootfs`, upper = `layers/root/upper`.
2. Mountpoints (`/proc /dev /tmp /run ...`) created inside the merged tree.
Both modes then: tmpfs `/tmp` and `/run`; `/dev` (tmpfs, bound host nodes, new devpts, `/dev/shm`, symlinks);
`/proc` with read-only remounts and masks from `spec.hpp`; `chdir(rootfs)`, `pivot_root(".", ".")`, lazy
unmount of the old root.
## Seeding (host mode, once per container)
`Container::prepareLayout` runs on create and as a heal on every later load; each step leaves a
`seeded.<name>` marker so it runs once and prints what it did and what it skipped:
1. Directory skeleton of every overlaid host dir into its upper layer (`fsutil::mirrorDirs`).
2. Regular files of `/var/lib/dpkg`, `triggers`, `alternatives` copied into the upper (`copyFiles`), then empty lock files.
3. `/etc` copied into `private/etc` (`copyTree`).
Reasons are in `SPEC.md` (host root-owned files appear as `nobody`).
## Images
`ImageStore::add`: lock-free staging dir `images/.tmp-NAME`, download with curl (https only, size and time caps),
sha256 check, `tar -x -p --numeric-owner` executed by `UserNsJob` with the image's id mode (`--no-same-owner`
for single), usage measured, `meta` written, atomic `rename` into place. Any failure removes the staging dir
from inside the id map, so a failed add leaves nothing. `Store::getOrCreate` calls it when a catalog image is
missing, and pins the container's id mode to the image's.
## join flow
docs/RESEARCH.md
+70 -9
View File
@@ -42,19 +42,19 @@ Single run set, one machine; no other hardware measured.
## Known gaps and researched solutions
Status of everything in this section: **researched, not implemented**. Probes ran on the
Status per gap is in the table below and in "Resolution log (1.0.0)" at the end. Probes ran on the
machine above (cgroup v2, systemd user manager, ext4 root, uid 1000). Probe cgroups were
removed afterwards. Rows marked "unmeasured" have no run behind them.
| Id | Gap | Proposed fix | Bump | Code that would change |
| Id | Gap | Fix | Status | Code |
|---|---|---|---|---|
| G1 | No memory/cpu/pids limits | Per-container cgroup v2 leaf under the delegated user service | minor | `sandbox.cpp` start/join, new `Cgroup` class, `spec.hpp` |
| G2 | No disk cap on upper dirs | Usage watchdog with `cgroup.freeze` at the hard limit; optional tmpfs upper | minor | `Sandbox::start` parent loop, `spec.hpp` |
| G3 | Single uid mapping | `newuidmap`/`newgidmap` with `/etc/subuid`, fallback to single map | minor, delete path changes | `sandbox.cpp` writeMap, `Store::remove` |
| G4 | Host `/etc` readable | Allowlist-only private `/etc` instead of an overlay of host `/etc` | minor | `spec.hpp` lists, `setupRoot` |
| G5 | `--net host` shares abstract sockets | Keep opt-in; evaluate pasta as an isolated replacement | patch or minor | `network.cpp` |
| G6 | No seccomp, nested userns allowed | Hand-built seccomp-bpf denylist after `no_new_privs` | minor | new `Seccomp` class, `sandbox.cpp` |
| G7 | Host-owned lower dirs are not writable; `chown --reference` of host files fails; `/var` is empty so apt has no databases | Image-based lower layer (extracted rootfs, owned by mapped ids) plus G3; short term: pre-seeded directory skeleton | major (changes layout) | `Container::prepareLayout`, `setupRoot`, new image import |
| G1 | Memory/cpu/pids limits | Cgroup v2 leaf under the delegated user service | **partly done in 1.0.0**: the leaf exists and is used for `cgroup.freeze`; `memory.max`, `pids.max`, `cpu.max` are not set yet | `cgroup.cpp` |
| G2 | Disk cap on upper dirs | Usage watchdog with freeze | **done in 1.0.0** (`--size`) | `Sandbox::watch` |
| G3 | Single uid mapping | `newuidmap`/`newgidmap` with `/etc/subuid`, fallback single | **done in 1.0.0** (`--ids`, default subuid) | `idmap.cpp`, `Store::remove` |
| G4 | Host `/etc` readable | Allowlist-only private `/etc` | **open** for host mode: `/etc` is now a private copy of all readable files (stable, still world-readable secrets); **closed in image mode** (no host files) | `container.cpp` |
| G5 | `--net host` shares abstract sockets | Keep opt-in; evaluate pasta | **open** | `network.cpp` |
| G6 | No seccomp, nested userns allowed | Hand-built seccomp-bpf denylist | **open** | new `Seccomp` class |
| G7 | Host-owned lower dirs unwritable, `chown` of host files fails, empty `/var` | Image layer + subuid; skeleton and seeding for host mode | **done in 1.0.0** | `image.cpp`, `container.cpp` |
### G1: memory, cpu and pids limits (cgroup v2)
@@ -161,3 +161,64 @@ Researched ways forward, in order of effort:
- Short term, writable skeleton: at create time mirror every directory of the lower dirs into the upper dir with the same mode (directories owned by the user are writable; merged view is unchanged). Directories unreadable to the user (for example `/etc/ssl/private`, mode 700, root-owned) are skipped and stay as they are. Measured effect: writes into `/usr/bin` work, attach time unchanged (41 ms). Unmeasured: C++ implementation speed, and it does not fix the `chown` failure.
- Proper fix, image layer: use an extracted distro rootfs (for example from `podman export` or `skopeo copy` plus unpack, both installed here; `mmdebstrap` and `debootstrap` are not) as the lower dir, with ownership shifted into the subuid range (G3). Then the tree is owned by mapped ids, `/var/lib/dpkg` exists, `chown` works, `_apt` exists, and host `/etc` stops leaking (G4). This also removes the dependence on the host distro. Cost: image import step, disk per image (shared lower, so once), and a layout change, hence a major bump.
- Alternative to evaluate: `fuse-overlayfs` (installed) supports uid mappings between layers. Unmeasured; adds a FUSE process per container.
## Resolution log (1.0.0)
Everything below was built and measured on the machine above (Ubuntu 26.04, kernel 7.0.0-38, ext4 root,
uid 1000, subuid range 100000:65536). The end-to-end suite (`tests/run.sh`) passed 56 of 56.
### What the field trial broke, and what fixed it
| Symptom (0.1.3) | Root cause | Fix in 1.0.0 | Result |
|---|---|---|---|
| `mkdir/touch` in `/usr/bin` fails | host dirs show as `nobody`; container root has no right over unmapped owners | `fsutil::mirrorDirs`: every host directory recreated in the upper layer, owned by the container | 13,163 dirs for `/usr` in about 1.1 s (the shell loop took 38 s); `touch /usr/bin/x` works, attach time unchanged |
| `/var/lib/apt/lists/partial is missing`, no dpkg lock | `/var` was empty | `/var` pre-seeded with the apt/dpkg dirs (`kVarDirs`); host `/var/lib/dpkg` overlaid | apt starts |
| `open lock-frontend: Permission denied` | host lock files owned by nobody, mode 640 | empty lock files pre-created in the upper (`kUpperLockFiles`) | lock acquired |
| `dpkg: error creating backup file status-old: Operation not permitted` | dpkg hard-links `status`; kernel `protected_hardlinks` refuses links to files you do not own and cannot write | regular files of `/var/lib/dpkg`, `triggers`, `alternatives` copied into the upper (`kUpperCopyDirs`) | dpkg state writable |
| `chown: changing ownership of '/etc/shells.tmp': Invalid argument` | `chown --reference` of a file owned by an unmapped uid | `/etc` is a private copy owned by the container root (`kCopiedDirs`) | `stat -c %u /etc/shells` is 0 |
| `setegid 65534 failed`, `seteuid 42 failed` (apt `_apt` user) | one mappable id | subuid mapping through `newuidmap`/`newgidmap` (ids 1..65535) | `apt-get update` works with no override; with `--ids single` an `APT::Sandbox::User "root"` override is written instead |
| `apt-get install cowsay` not configured | all of the above | all of the above | install completes, `dpkg --audit` clean, `/usr/games/cowsay` runs, host has no `/usr/games/cowsay` |
### Image layer
- Why images are the clean fix: the extracted tree is owned by mapped ids, so nothing needs a skeleton, copies or overrides. Measured: `apk add jq` in Alpine and `apt-get install cowsay curl ca-certificates` in Ubuntu 26.04 complete with a clean `dpkg --audit`; `curl https://example.com` returns 200.
- Sources chosen (checked live with curl, 2026-10-23):
- Alpine: `latest-releases.yaml` lists file name and sha256 per flavor (`alpine-minirootfs-3.24.3-x86_64.tar.gz`, 3.7 MB).
- Ubuntu: `cdimage.ubuntu.com/ubuntu-base/releases/<ver>/release/SHA256SUMS` plus tarballs (26.04.1 and 24.04.5 present).
- Debian: no usable tarball path found (`debuerreotype/docker-debian-artifacts` returned 404 for `rootfs.tar.xz`; only `.sha256` files resolved), so Debian is supported through `image add URL --sha256`, not through the catalog.
- Not used: `podman export` / `skopeo` (installed here, but one more dependency; a registry client is out of scope).
- Trust model: HTTPS to the publisher plus the checksum file from the same publisher. The Ubuntu `SHA256SUMS.gpg` signature is not checked (unimplemented, listed below).
- Extraction runs inside a user namespace with the id map the containers will use (`UserNsJob`): subuid images keep the real ownership; single-id images use `tar --no-same-owner`. A container's id mode is pinned to its image's.
- Cleanup: files owned by subuids cannot be removed by the host user (`Permission denied`). `Store::remove` and `ImageStore::remove` therefore delete from inside the id map. Measured: deleting containers and images with subuid-owned files works, and a failed add leaves no staging directory.
- Image sizes (hardlinks counted once): Alpine 8.3M, Ubuntu base 97M. The first measurement said 1.3G because uutils-coreutils hard-links one 11 MB binary about 100 times; `diskUsage` now deduplicates by (device, inode).
### Size cap
- Mechanism: `Sandbox::watch` polls the init's pidfd with a 10 s timeout and scans usage on every wake. 80% warns, 100% freezes the container's cgroup, below 95% it thaws.
- Measured: `--size 64m`, a 100 MB write: `FROZEN` message at 100M of 64M, tick output stopped; `doos attach cap -- rm /root/big` ran (a second attach is not in the frozen cgroup), the container thawed (`thawed (4.0K used)`) and continued.
- Freezing needs no cgroup controller, only the delegated `user@UID.service` subtree. Without it the cap warns and says so.
- Chosen over a size-capped tmpfs because tmpfs is memory-backed and not persistent; chosen over project quota because that needs root.
### Timings (5 sequential `attach -- true`, this machine, subuid, one run set)
| Mode | Per attach |
|---|---|
| host, slirp | about 50 ms |
| image alpine, slirp | about 40 ms |
| image ubuntu, slirp | about 40 ms |
| host, `--net none` | about 10 ms |
Subuid mapping added about 15 to 25 ms per start compared with 0.1.x (two setuid helper processes). Creating a host-mode container (skeleton, `/etc` copy, dpkg files) took about 1.2 s once. First `--image alpine` use including download and extraction: 0.3 s end to end on this network.
### Still open, stated plainly
- G1: `memory.max`, `pids.max`, `cpu.max` are not set. Measured earlier that they work here; not wired in.
- G4: host mode copies every readable file of host `/etc` at creation. Use an image to avoid host files entirely.
- G5, G6: unchanged (see above).
- Size accounting sees only what the host user can read. Directories of other container uids with mode 700 inside the writable layer are not counted (reported only as an internal skip count, not shown). Unmeasured.
- Overshoot: up to one scan interval (10 s) of writes can pass the cap before the freeze; measured 100M written against a 64M cap.
- A second `attach` is outside the frozen cgroup and outside any limit by design (it is the rescue path).
- Host-mode containers drift from the host: the `/etc` copy, dpkg state and skeleton are snapshots at creation. A host package upgrade is visible in `/usr` but not in the container's dpkg database.
- Ubuntu image signature (`SHA256SUMS.gpg`) is not verified; Alpine's yaml is not signature-checked either.
- `image add FILE` without `--sha256` is accepted and prints a note, since the user chose the file.
- Catalog covers x86_64 and aarch64 only; aarch64 is untested.