doos/docs/RESEARCH.md

# Research: user-level containers with overlayfs

**Docs:** [README](../README.md) · [SPEC](../SPEC.md) · [ARCHITECTURE](ARCHITECTURE.md) · [RESEARCH](RESEARCH.md) · [COMPARISON](../COMPARISON.md) · [AGENTS](../AGENTS.md) · [CHANGELOG](../CHANGELOG.md) · [CI](CI.md)

Measured on this machine: Ubuntu, kernel 7.0.0-38, AppArmor 5.0.2, slirp4netns 1.3.3, uid 1000, no root at runtime.

Memory-safety of doos's own code (not its behavior) is checked separately with `make valgrind`
against a fixed set of invocations; see [valgrind/README.md](../valgrind/README.md) for what it
covers and its suppressions. Known CVEs in software doos depends on or resembles are tracked in
[audit/README.md](../audit/README.md) (`make audit`).

## What works unprivileged

1. `clone(CLONE_NEWUSER|NEWNS|NEWPID|NEWUTS|NEWIPC[|NEWNET])`, then the parent writes `setgroups=deny`, `uid_map`, `gid_map` (single entry, own uid). No setuid helpers (`newuidmap`) needed.
2. overlayfs mounts inside the userns (kernel >= 5.11) with the `userxattr` option. Upper and work dirs must be on a filesystem with user xattrs (ext4, tmpfs >= 6.6 verified for ours).
3. Mounts that work inside: tmpfs, proc (needs own pidns), devpts (new instance, no `gid=` since gid 5 is unmapped), bind mounts of host `/dev/{null,zero,full,random,urandom,tty}` (no `mknod` in a userns), read-only remount of `/proc` subpaths (must copy locked flags from `statvfs`).
4. `pivot_root(".", ".")` plus lazy unmount of the old root.

## What does not

- Writing mapped uid ranges beyond own uid needs `newuidmap` + `/etc/subuid`. Skipped: single mapping keeps the design dumb. Consequence: no chown/su to other uids inside, so `apt install` as non-root users does not work.
- cgroups: no memory/cpu/pid limits without systemd delegation. Not implemented (gap).
- Ubuntu 24.04+: `kernel.apparmor_restrict_unprivileged_userns=1` lets any binary create a userns but strips capabilities inside it. Measured: uid_map write succeeds, first `mount` fails with EPERM. Fix is an AppArmor profile granting `userns` to the binary (`apparmor/doos`, install once with sudo). Turning the sysctl off is global and weaker.

## Networking options

| Option | Result |
|---|---|
| slirp4netns (chosen default) | Works unprivileged. Needs only the netns of the child pid. `--disable-host-loopback` blocks the container from reaching host 127.0.0.1 (measured: blocked). DNS via 10.0.2.3 works (measured). |
| pasta | Installed here but not used: slirp4netns suffices, one dependency. |
| veth + bridge | Needs CAP_NET_ADMIN in the host netns. Not possible unprivileged. |
| host netns | Works, zero isolation. Offered as explicit `--net host`. |

## Host safety design

- No host paths writable: overlay lower dirs are only reachable through overlay (writes copy up to the container upper). Verified: write to `/usr` and `/etc` inside leaves host untouched.
- Only `/usr /etc /opt` and compat dirs are exposed read-through. `/home`, `/var`, `/sys`, `/root` of the host are not present.
- Old root detached after pivot, `no_new_privs`, `/proc` hardened, nosuid/nodev on tmpfs/devpts/overlay.
- Residual exposure (gaps G1 to G6 below): no resource limits, no disk cap, single uid, host `/etc` readable, abstract unix sockets shared in `--net host`, no seccomp filter.

## Measurements (this machine, 5 sequential `attach -- true`)

| Mode | Per attach |
|---|---|
| slirp | ~26 ms |
| none | ~6 ms |

Single run set, one machine; no other hardware measured.

## Known gaps and researched solutions

Status per gap is in the table below and in "Resolution log (1.0.0)" at the end. Probes ran on the
machine above (cgroup v2, systemd user manager, ext4 root, uid 1000). Probe cgroups were
removed afterwards. Rows marked "unmeasured" have no run behind them.

| Id | Gap | Fix | Status | Code |
|---|---|---|---|---|
| G1 | Memory/cpu/pids limits | Cgroup v2 limits on `doos/<name>` | **done in 1.1.0** (`--memory`, `--pids`, `--cpus`) | `cgroup.cpp` |
| G2 | Disk cap on upper dirs | Usage watchdog with freeze | **done in 1.0.0**, accounting and overshoot improved in 1.1.0 | `Sandbox::watch` |
| G3 | Single uid mapping | `newuidmap`/`newgidmap` with `/etc/subuid`, fallback single | **done in 1.0.0** (`--ids`, default subuid) | `idmap.cpp`, `Store::remove` |
| G4 | Host `/etc` readable | Allowlist-only private `/etc` | **done in 1.1.0** for host mode (`--hardening relaxed` restores the full copy); image mode never had host files | `container.cpp` |
| G5 | `--net host` shares abstract sockets | pasta mode | **done in 1.1.0** (`--net pasta`); `host` stays as an explicit, noted opt-in | `network.cpp` |
| G6 | No seccomp, nested userns allowed | Capability drop + seccomp-bpf denylist | **done in 1.1.0** (`--hardening`) | `hardening.cpp` |
| G7 | Host-owned lower dirs unwritable, `chown` of host files fails, empty `/var` | Image layer + subuid; skeleton and seeding for host mode | **done in 1.0.0** | `image.cpp`, `container.cpp` |
| G8 | `fsutil::mkdirs` followed a symlink planted at a container's path under `$DOOS_HOME/containers/<name>`, same bug class as [`audit/CVE-2021-30465`](../audit/CVE-2021-30465/README.md) | walk the path component by component from a pinned base fd with `mkdirat`/`openat(O_NOFOLLOW\|O_DIRECTORY)`, refusing any symlink instead of following it | **done in 1.3.0**, re-measured: `make -C audit/CVE-2021-30465 test` now PASSes | `fsutil::mkdirs` in `src/util.cpp` |
| G9 | `parseSize` left-shifted without an overflow check: `--size 16777217t` wrapped to exactly `1.0T` and was accepted instead of rejected | check `v > (UINT64_MAX >> shift)` before shifting, reject instead of wrapping | **done in 1.3.2**, reproduced before the fix, re-run after: rejected | `parseSize` in `src/util.cpp` |
| G10 | `newuidmap`/`newgidmap`/`slirp4netns`/`pasta` were pinned to a single hardcoded `/usr/bin/...` path, unlike `curl`/`tar`/`sha256sum`/`gpg` which go through `resolveBinary()`'s state-dir-override-then-fixed-dirs search; a distro installing them elsewhere would fail even though PATH-poisoning was never possible either way | route all four through `resolveBinary()`, same as the other helpers | **done in 1.3.2** | `idmap.cpp`, `network.cpp` |
| G11 | Alpine's `latest-releases.yaml` and Debian's `index.json` (embedded base64 manifest) are read with fixed string/byte-offset search, not a YAML/JSON library; a publisher reformatting either file breaks extraction with a thrown error (fails closed) rather than silently misreading it | none: a dependency to parse two small, publisher-controlled files is a worse trade than failing loudly when their format changes, given the project's own "fewer dependencies" rule | **accepted, not fixed**; this row exists so the trade-off is written down, not rediscovered | `Catalog::resolve` in `src/image.cpp` |

### G1: memory, cpu and pids limits (cgroup v2)

Findings, measured:
- `/sys/fs/cgroup` is cgroup2 with `nsdelegate`. `user@1000.service/cgroup.controllers` lists `cpuset cpu io memory pids`, and its `subtree_control` already enables all five. A plain user can `mkdir` below `/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/`.
- A leaf with `pids.max=20`: the 21st fork failed with `Cannot fork`.
- A leaf with `memory.max=64m`: touching 200 MiB ended in `Killed`, `memory.events` showed `oom_kill 1`. An untouched `bytearray(200MiB)` is not charged (lazy pages), so the first attempt proved nothing; only the touched run counts.
- Gotcha, measured: enabling `+memory +pids` in `subtree_control` of a cgroup makes it a non-leaf; writing a pid into its `cgroup.procs` then fails with `I/O error`. The layout must be `.../doos/` (controllers enabled here, no processes) with one leaf per container `.../doos/<name>/` (limits set here, processes here).
- `RLIMIT_NPROC` is not an option: it counts per real uid across the whole host user, not per container.

Design:
1. Discover the base: parse `/proc/self/cgroup`, take the path up to and including `user@<uid>.service`.
2. Parent creates `doos/<name>` after `clone`, writes `memory.max`, `memory.swap.max=0`, `pids.max`, `cpu.max`, then writes the child pid to `cgroup.procs` before the sync byte `G` (child never runs outside limits). `join()` children enter via the same `cgroup.procs` write before `setns`.
3. On exit: write `1` to `cgroup.kill` (kernel >= 5.14), then `rmdir`.
4. Defaults as constants in `spec.hpp` (static ceilings, no autotuning): for example memory 2g, pids 512, cpu unlimited. Flags `--memory`, `--pids`, `--cpus` may lower or raise within a hard cap.
5. No delegation (no systemd user manager, cgroup v1): fail loudly with a counter and a clear reason; `--no-limits` is an explicit opt-out, never silent.

Rejected: `systemd-run --user --scope -p MemoryMax=...` (needs D-Bus, one more moving part). Unmeasured: behavior when doos itself runs inside a scope that is not under `user@.service` (the tmux scope here is, but other launchers may not be).

Sources: [rootlesscontaine.rs cgroup v2](https://rootlesscontaine.rs/getting-started/common/cgroup2/), [runc cgroup v2](https://github.com/thaJeztah/runc/blob/main/docs/cgroup-v2.md), [Docker rootless tips](https://docs.docker.com/engine/security/rootless/tips/).

### G2: disk usage of upper dirs

Findings:
- Project quotas (XFS `pquota`, ext4 project quota) need root to set. This machine's root is ext4 on `/dev/md2`. Not usable unprivileged. Docker's overlay2 size limit also only works on XFS with `pquota` ([source](https://fabianlee.org/2020/01/15/docker-use-overlay2-with-an-xfs-backing-filesystem-to-limit-rootfs-size/)).
- Loop-mounted ext4 images need root; the kernel does not allow ext4 mounts from a user namespace. `fuse2fs` is not installed here. `fuse-overlayfs` is installed but has no quota either.
- A `size=` tmpfs mounted inside the container userns is enforced, measured: `size=1m`, writing 5 MiB ended in `No space left on device`.

Design (follows the rule "machines hold, humans analyze"):
1. Default: the parent process (already alive in `Sandbox::start`) measures upper-dir usage on a fixed interval with a floor (for example every 30 s, never faster). Soft limit: warning on stderr. Hard limit: write `1` to the container's `cgroup.freeze` (needs G1). Freeze stops processes but deletes nothing; the human frees space or raises the limit. Never auto-delete.
2. Opt-in `--ephemeral`: upper dirs on a size-capped tmpfs. Hard cap by the kernel, but memory-backed, counted in the memory cgroup, and lost on exit.
3. Constants (soft, hard, interval) in `spec.hpp`.

Unmeasured: cost of scanning a large upper dir (needs a run on a populated container); overshoot between two scans.

### G3: more than one uid inside

Findings, measured with `podman unshare` (map `0 -> 1000`, `1..65536 -> 100000..`): overlayfs with `userxattr` mounts fine under a multi-id map. `/etc/subuid` and `/etc/subgid` contain `retoor:100000:65536`, `newuidmap` and `newgidmap` are installed.
- Host root-owned files in the lower dirs show as `65534 nobody`. That is already true with the single map, so nothing gets worse.
- Gain: `chown`, `su`, `apt` sandbox users, daemons that drop privileges.
- Cost, important: files created by container uids 1..65535 are owned by host uids 100000+. The host user cannot delete them. `Store::remove` as written would fail with `Permission denied`. Delete must run inside a user namespace with the same map (re-enter and `rm` there, or `podman unshare`-style helper) before removing the directory.

Design: after `clone`, parent execs `newuidmap <pid> 0 <uid> 1 1 <sub> 65536` and the gid equivalent, `setgroups` allowed. Fallback to the current single map when the user has no subuid range, with a printed notice. Store the chosen mode in `meta` (`idmap=single|subuid`) so delete knows which path to take.

Unmeasured: `apt install` end to end.

### G4: host `/etc` visible inside

Finding, measured: `find /etc -type f -perm -004` returns 1116 world-readable files on this host, including names such as `/etc/fail2ban/action.d/cloudflare-token.conf` (content not read). Anything world-readable in host `/etc` is readable in every container.

Design: replace the `/etc` overlay by a private `/etc`. At create time, populate `private/etc` from an allowlist in `spec.hpp` (`kEtcAllow`): `passwd group nsswitch.conf ld.so.cache ld.so.conf ld.so.conf.d ssl/certs ca-certificates alternatives bash.bashrc profile profile.d localtime timezone os-release protocols services hosts hostname`, plus a generated `resolv.conf`. Two ways: copy (stable, can go stale, for example CA certs) or read-only bind of each allowed path (always current, more mounts). Recommendation: read-only bind, since doos already carries the mount machinery.

Unmeasured: whether this allowlist is enough for `apt`, `curl`, `python3`, `git`; needs a run per tool before fixing the list.

### G5: `--net host` and abstract unix sockets

Fact: abstract `AF_UNIX` sockets belong to the network namespace, so `--net host` exposes host abstract sockets (some X11 and D-Bus setups use them) and host loopback services. This is inherent to sharing the namespace.

Researched alternative, pasta (installed, `pasta 0.0~git20260120`): it attaches to an existing netns by pid (`pasta --config-net <pid>`), gives outbound access, and mirrors host addresses into the namespace. Measured: outbound TCP worked, and an inside connection to `127.0.0.1:22` reached the host (default flags, once). A second run with `--no-splice --no-map-gw --map-host-loopback none` printed `Couldn't get any nameserver address`, and I could not confirm the loopback result from it. Conclusion: pasta is not proven to match slirp4netns' `--disable-host-loopback` guarantee. Slirp stays the default. Next step is a clean test matrix of pasta flags with a listener on a known host port.

Interim: `host` remains an explicit opt-in documented as unisolated.

### G6: kernel attack surface (seccomp, nested user namespaces)

Findings:
- A seccomp-bpf filter installs unprivileged once `no_new_privs` is set (already done). Measured with a raw `prctl(PR_SET_SECCOMP)` program: a denied syscall (`keyctl`) returned `-1 EPERM`. No `libseccomp` headers on this machine, and "fewer dependencies" argues for hand-assembled BPF.
- `/proc/sys` is read-only in the container, so `user.max_user_namespaces=0` cannot be set from inside; seccomp is the lever.

Design: denylist modelled on the Docker default profile, applied after `setupRoot` and before `exec`, and also in the `join()` path. Check `AUDIT_ARCH` first and kill on a foreign ABI (blocks the x32 bypass). Deny with `EPERM`: `kexec_load kexec_file_load bpf perf_event_open userfaultfd keyctl add_key request_key open_by_handle_at process_vm_readv process_vm_writev reboot swapon swapoff init_module finit_module delete_module`. Nested userns: deny `unshare` and `setns` with `CLONE_NEWUSER`, deny `clone` with `CLONE_NEWUSER`, and return `ENOSYS` for `clone3` (flags live in a struct, cannot be inspected; libc falls back to `clone`). Trade-off: breaks nested containers and Chromium-style sandboxes; make it a documented constant, not a switch that hides in code.

Unmeasured: compatibility impact on real workloads.

### Priority

1. G1 (limits): cheapest, measured to work, unlocks G2's freeze.
2. G6 (seccomp): self-contained, no new dependency.
3. G4 (private `/etc`): needs the per-tool allowlist runs first.
4. G2, then G3 (G3 changes the delete path, do it deliberately).
5. G5: needs a test matrix before any change.

## Field trial: apt and common tools (v0.1.2, measured)

Run in a fresh `--net slirp` container on the host above (Ubuntu with apt 3.2.0, dpkg 1.23.7).

| Step | Result |
|---|---|
| curl https, git ls-remote, gcc compile+run, python3 (`ssl`, `sqlite3`), pip, ping 1.1.1.1 | all work |
| `apt-get update` as shipped | fails: `/var/lib/apt/lists/partial is missing` (`/var` is an empty private dir) |
| `apt-get install` as shipped | fails: `/var/lib/dpkg/lock-frontend` missing, no dpkg status database |
| after seeding `/var` (dirs plus copy of host `dpkg/status`, `info`, `alternatives`, 40 MB) | apt starts, but the download method dies: `setegid 65534 failed`, `seteuid 42 failed` (sandbox user `_apt` cannot exist with a single id) |
| with `-o APT::Sandbox::User=root` | `apt-get update` works: 58.9 MB fetched in 3 s |
| `apt-get install cowsay` | download works; unpack fails: `unable to create '/usr/games/cowsay.dpkg-new': Permission denied` |
| after pre-creating the host directory tree in the upper dirs (13164 dirs for `/usr`, done with a shell loop, 38 s; not yet implemented in C++) | unpack works, `/usr/games/cowsay hoi` runs; host `/usr/games/cowsay` still absent (isolation holds) |
| configure step | fails: `chown: changing ownership of '/etc/shells.tmp': Invalid argument` (debianutils postinst copies owner of `/etc/shells`, which shows as `65534`), so dpkg reports 4 packages not fully installed |

Root causes:
1. Host files owned by host root appear as `nobody` (uid 65534). Container root has `CAP_DAC_OVERRIDE` only over inodes whose owner is mapped, so it cannot write into host-root-owned directories of the lower layer, and cannot `chown` to or from an unmapped owner.
2. `/var` has no package database.
3. apt wants a second uid (G3).

Conclusion: the current design (host `/usr`, `/etc`, `/opt` as lower layer) is good for running tools and for isolated experiments, not for package management. Bare tools and compilers work. Honest status: **`apt-get install` does not complete today.**

Researched ways forward, in order of effort:
- Short term, writable skeleton: at create time mirror every directory of the lower dirs into the upper dir with the same mode (directories owned by the user are writable; merged view is unchanged). Directories unreadable to the user (for example `/etc/ssl/private`, mode 700, root-owned) are skipped and stay as they are. Measured effect: writes into `/usr/bin` work, attach time unchanged (41 ms). Unmeasured: C++ implementation speed, and it does not fix the `chown` failure.
- Proper fix, image layer: use an extracted distro rootfs (for example from `podman export` or `skopeo copy` plus unpack, both installed here; `mmdebstrap` and `debootstrap` are not) as the lower dir, with ownership shifted into the subuid range (G3). Then the tree is owned by mapped ids, `/var/lib/dpkg` exists, `chown` works, `_apt` exists, and host `/etc` stops leaking (G4). This also removes the dependence on the host distro. Cost: image import step, disk per image (shared lower, so once), and a layout change, hence a major bump.
- Alternative to evaluate: `fuse-overlayfs` (installed) supports uid mappings between layers. Unmeasured; adds a FUSE process per container.

## Resolution log (1.0.0)

Everything below was built and measured on the machine above (Ubuntu 26.04, kernel 7.0.0-38, ext4 root,
uid 1000, subuid range 100000:65536). The end-to-end suite (`tests/run.sh`) passed 56 of 56.

### What the field trial broke, and what fixed it

| Symptom (0.1.3) | Root cause | Fix in 1.0.0 | Result |
|---|---|---|---|
| `mkdir/touch` in `/usr/bin` fails | host dirs show as `nobody`; container root has no right over unmapped owners | `fsutil::mirrorDirs`: every host directory recreated in the upper layer, owned by the container | 13,163 dirs for `/usr` in about 1.1 s (the shell loop took 38 s); `touch /usr/bin/x` works, attach time unchanged |
| `/var/lib/apt/lists/partial is missing`, no dpkg lock | `/var` was empty | `/var` pre-seeded with the apt/dpkg dirs (`kVarDirs`); host `/var/lib/dpkg` overlaid | apt starts |
| `open lock-frontend: Permission denied` | host lock files owned by nobody, mode 640 | empty lock files pre-created in the upper (`kUpperLockFiles`) | lock acquired |
| `dpkg: error creating backup file status-old: Operation not permitted` | dpkg hard-links `status`; kernel `protected_hardlinks` refuses links to files you do not own and cannot write | regular files of `/var/lib/dpkg`, `triggers`, `alternatives` copied into the upper (`kUpperCopyDirs`) | dpkg state writable |
| `chown: changing ownership of '/etc/shells.tmp': Invalid argument` | `chown --reference` of a file owned by an unmapped uid | `/etc` is a private copy owned by the container root (`kCopiedDirs`) | `stat -c %u /etc/shells` is 0 |
| `setegid 65534 failed`, `seteuid 42 failed` (apt `_apt` user) | one mappable id | subuid mapping through `newuidmap`/`newgidmap` (ids 1..65535) | `apt-get update` works with no override; with `--ids single` an `APT::Sandbox::User "root"` override is written instead |
| `apt-get install cowsay` not configured | all of the above | all of the above | install completes, `dpkg --audit` clean, `/usr/games/cowsay` runs, host has no `/usr/games/cowsay` |

### Image layer

- Why images are the clean fix: the extracted tree is owned by mapped ids, so nothing needs a skeleton, copies or overrides. Measured: `apk add jq` in Alpine and `apt-get install cowsay curl ca-certificates` in Ubuntu 26.04 complete with a clean `dpkg --audit`; `curl https://example.com` returns 200.
- Sources chosen (checked live with curl, 2026-10-23):
  - Alpine: `latest-releases.yaml` lists file name and sha256 per flavor (`alpine-minirootfs-3.24.3-x86_64.tar.gz`, 3.7 MB).
  - Ubuntu: `cdimage.ubuntu.com/ubuntu-base/releases/<ver>/release/SHA256SUMS` plus tarballs (26.04.1 and 24.04.5 present).
  - Debian: no usable tarball path found (`debuerreotype/docker-debian-artifacts` returned 404 for `rootfs.tar.xz`; only `.sha256` files resolved), so Debian is supported through `image add URL --sha256`, not through the catalog.
  - Not used: `podman export` / `skopeo` (installed here, but one more dependency; a registry client is out of scope).
- Trust model: HTTPS to the publisher plus the checksum file from the same publisher. The Ubuntu `SHA256SUMS.gpg` signature is not checked (unimplemented, listed below).
- Extraction runs inside a user namespace with the id map the containers will use (`UserNsJob`): subuid images keep the real ownership; single-id images use `tar --no-same-owner`. A container's id mode is pinned to its image's.
- Cleanup: files owned by subuids cannot be removed by the host user (`Permission denied`). `Store::remove` and `ImageStore::remove` therefore delete from inside the id map. Measured: deleting containers and images with subuid-owned files works, and a failed add leaves no staging directory.
- Image sizes (hardlinks counted once): Alpine 8.3M, Ubuntu base 97M. The first measurement said 1.3G because uutils-coreutils hard-links one 11 MB binary about 100 times; `diskUsage` now deduplicates by (device, inode).

### Size cap

- Mechanism: `Sandbox::watch` polls the init's pidfd with a 10 s timeout and scans usage on every wake. 80% warns, 100% freezes the container's cgroup, below 95% it thaws.
- Measured: `--size 64m`, a 100 MB write: `FROZEN` message at 100M of 64M, tick output stopped; `doos attach cap -- rm /root/big` ran (a second attach is not in the frozen cgroup), the container thawed (`thawed (4.0K used)`) and continued.
- Freezing needs no cgroup controller, only the delegated `user@UID.service` subtree. Without it the cap warns and says so.
- Chosen over a size-capped tmpfs because tmpfs is memory-backed and not persistent; chosen over project quota because that needs root.

### Timings (5 sequential `attach -- true`, this machine, subuid, one run set)

| Mode | Per attach |
|---|---|
| host, slirp | about 50 ms |
| image alpine, slirp | about 40 ms |
| image ubuntu, slirp | about 40 ms |
| host, `--net none` | about 10 ms |

Subuid mapping added about 15 to 25 ms per start compared with 0.1.x (two setuid helper processes). Creating a host-mode container (skeleton, `/etc` copy, dpkg files) took about 1.2 s once. First `--image alpine` use including download and extraction: 0.3 s end to end on this network.

### Still open after 1.0.0 (see "Resolution log (1.1.0)" for what changed since)

- G1: `memory.max`, `pids.max`, `cpu.max` are not set. Measured earlier that they work here; not wired in.
- G4: host mode copies every readable file of host `/etc` at creation. Use an image to avoid host files entirely.
- G5, G6: unchanged (see above).
- Size accounting sees only what the host user can read. Directories of other container uids with mode 700 inside the writable layer are not counted (reported only as an internal skip count, not shown). Unmeasured.
- Overshoot: up to one scan interval (10 s) of writes can pass the cap before the freeze; measured 100M written against a 64M cap.
- A second `attach` is outside the frozen cgroup and outside any limit by design (it is the rescue path).
- Host-mode containers drift from the host: the `/etc` copy, dpkg state and skeleton are snapshots at creation. A host package upgrade is visible in `/usr` but not in the container's dpkg database.
- Ubuntu image signature (`SHA256SUMS.gpg`) is not verified; Alpine's yaml is not signature-checked either.
- `image add FILE` without `--sha256` is accepted and prints a note, since the user chose the file.
- Catalog covers x86_64 and aarch64 only; aarch64 is untested.

## Resolution log (1.1.0)

Built and measured on the same machine (Ubuntu 26.04, kernel 7.0.0-38, 12 cores, ext4, uid 1000, cgroup v2
delegated with `cpuset cpu io memory pids`). The end-to-end suite passed 83 checks.

### Research behind each decision

| Question | Finding | Source | Decision |
|---|---|---|---|
| Which capabilities to keep | Docker's default set: CHOWN DAC_OVERRIDE FSETID FOWNER MKNOD NET_RAW SETGID SETUID SETFCAP SETPCAP NET_BIND_SERVICE SYS_CHROOT KILL AUDIT_WRITE | [Docker run reference via OneUptime](https://oneuptime.com/blog/post/2026-01-25-docker-container-capabilities/markdown), [Red Hat container security guide](https://access.redhat.com/documentation/en/red_hat_enterprise_linux_atomic_host/7/html/container_security_guide/linux_capabilities_and_seccomp) | Same set. Measured: `CapEff` inside is `a80425fb`, the Docker default mask. |
| Which syscalls to deny | Docker's profile blocks keyring calls (not namespaced) and io_uring (container escapes); kexec and bpf are also capability-gated | [Docker seccomp docs](https://docs.docker.com/engine/security/seccomp/) | Denylist in `hardening.cpp`; capability drop covers the capability-gated ones twice. |
| x32 and architecture | A denylist keyed on the x86-64 arch can be dodged with `__X32_SYSCALL_BIT`; kernels >= 5.4 answer such numbers with ENOSYS | [seccomp(2) man page patch](https://lkml.iu.edu/hypermail/linux/kernel/2007.1/03947.html) | Arch check first (kill on mismatch), x32 numbers answered ENOSYS. |
| `clone3` | Flags live in a struct seccomp cannot read; ENOSYS (not EPERM) makes glibc fall back to `clone` | [QEMU seccomp patch](https://qemu.googlesource.com/qemu/+/c542b302707628609f2a33bcbd3e442282806b8e), [libc-alpha thread](https://public-inbox.org/libc-alpha/20210727102222.r2hys526mfkpt4xo@senku/) | `clone3` -> ENOSYS, `clone`/`unshare` with `CLONE_NEWUSER` -> EPERM. Measured both. |
| cgroup layout | Delegation covers a subtree; limit files exist only when the parent enables the controller; a cgroup that has children cannot hold processes | [kernel cgroup v2 docs](https://docs.kernel.org/admin-guide/cgroup-v2.html), [rootlesscontaine.rs](https://rootlesscontaine.rs/getting-started/common/cgroup2/) | `doos/` enables `+memory +pids +cpu`; limits on `doos/<name>` (no processes); leaves `main` and `shell`. |
| Host-like networking without sharing the namespace | pasta attaches to an existing namespace by PID and splices local TCP/UDP between namespaces | [passt(1) man page](https://manpages.ubuntu.com/manpages/resolute/man1/passt.1.html) | `--net pasta`. Measured below. |
| Image signatures | Alpine publishes the release key fingerprint on its downloads page; Ubuntu publishes the CD image key fingerprint in its verification tutorial | [alpinelinux.org/downloads](https://alpinelinux.org/downloads/), [ubuntu.com verify tutorial](https://ubuntu.com/tutorials/how-to-verify-ubuntu) | Fingerprints pinned in `spec.hpp`; keys fetched at install and refused unless the fingerprint matches. |
| Debian rootfs | No `rootfs.tar.xz` served; the OCI layer is at `oci/blobs/rootfs.tar.gz` and its digest is in the manifest embedded (base64) in `oci/index.json` | measured with curl against `debuerreotype/docker-debian-artifacts` | `debian`, `debian-RELEASE` in the catalog; the digest is the integrity check. |

### Measurements

| Check | Result |
|---|---|
| Capabilities | `CapEff a80425fb`, `Seccomp: 2` (filter) inside a default container; relaxed: `CapEff 000001ffffffffff`, `Seccomp: 0` |
| seccomp | `keyctl`, `bpf`, `mount`, `unshare(CLONE_NEWUSER)` -> EPERM; `clone3` -> ENOSYS; `unshare -U` and `mount -t tmpfs` fail; relaxed container runs `unshare -U` fine |
| Tools under hardening | gcc, curl https, git, python3 `ssl`, ping, apt and apk installs all still work |
| `/etc` | 50 entries under the allowlist versus 215 in a relaxed copy; no `shadow`, `sudoers`, `ssh_host_*` |
| Memory | `--memory 256m`, touching 400 MB: whole container killed, doos prints `killed by its memory limit` |
| Pids | `--pids 32`, 80 forks: `can't fork`, doos prints `hit its process limit (32)` |
| CPU | 4 busy loops for 4.3 s wall: unlimited 16.2 CPU-seconds, `--cpus 0.5` 2.3 CPU-seconds (0.54 of one core) |
| pasta | host `127.0.0.1` listener reachable, host abstract unix socket blocked (host mode: reachable), outbound TCP, DNS and https 200, inbound port forwarded to the host |
| Signatures | Alpine and Ubuntu 24.04 verified with the pinned keys; a stand-in `gpg` that reports a valid signature by another key, or `BADSIG`, is refused and leaves no image |
| Debian | `debian` (trixie) image installs and `apt-get install cowsay` completes, `dpkg --audit` clean; image 82M |
| Size accounting | 80 MB written by container uid 5 in a mode-700 directory (invisible to the host user) against `--size 64m`: frozen; cleanup from a second attach thawed it |
| Attach time (5 runs) | host or image, slirp: about 40 ms each; pasta: about 65 ms; `--net none`: about 10 ms |

![Horizontal bar chart of CPU-seconds used by four busy loops in one container, without a limit and with --cpus 0.5. The limited container is highlighted. Values are in the sentence below.](charts/cpu.svg)

The chart above is generated from `docs/charts/bench.json` (`make bench`, then `make charts`); it shows <!--bench:cpu_free-->16.3<!--/bench--> CPU-seconds without a limit and <!--bench:cpu_half-->2.3<!--/bench--> with `--cpus 0.5`; the startup and size charts live in [COMPARISON](../COMPARISON.md).

### What changed in behavior (and why it is not a loss)

- Hardening can be switched off per container (`--hardening relaxed`), so nothing that needs the denied calls is lost, it is opt-in now.
- Host `/etc` content outside the allowlist is absent under default hardening. `relaxed` restores the full copy; images never had it.
- The 10 s scan interval is gone: scans start every 2 s at the earliest but are paced by their own cost, so large trees are not scanned in a tight loop.
- pasta replaces the need for `--net host` in almost every case; `host` remains.

### Still open, stated plainly

- Memory limits count `/tmp` and `/dev/shm` (both RAM-backed) and page cache charged to the container; a build that fills `/tmp` can hit `--memory`. Measured only with anonymous memory.
- `--pids` counts threads. Thread-heavy programs need a higher value.
- `memory.high` is not used, so there is no soft throttle before the OOM kill.
- Size accounting is a scan, not a quota: writes can pass the cap by up to one scan interval, bounded per file by `RLIMIT_FSIZE`. A process can still write many small files faster than a scan.
- A later `doos attach` shares the memory, pids and cpu limits and the per-file size limit, but is never frozen. That is the rescue path by design, so it can also fill the disk past the cap.
- Host-mode containers still drift from the host (snapshots of `/etc`, dpkg state, skeleton). Images do not.
- Debian images have no publisher signature to check; trust is HTTPS to the Debian Docker maintainers' repository plus the content digest. The Alpine key is pinned from alpinelinux.org, not from a second out-of-band channel beyond the fingerprint shown on the same site and in public mailing lists.
- aarch64 paths (Alpine `aarch64`, Debian `dist-arm64v8`, seccomp numbers for aarch64) compile and are guarded but were not run: no aarch64 machine here.
- Without `gpg` installed, signature checks are skipped with a printed note.
- pasta is not signalled by doos (it drops uid); it exits by itself within about a second after the container's namespace is gone. Measured: no pasta process remained after 1.5 s.
- seccomp is a denylist. A new kernel syscall is allowed until it is added to the list.

For how doos stacks up against Docker, Podman and others, with these measurements next to theirs, see [COMPARISON](../COMPARISON.md). Behavior contract: [SPEC](../SPEC.md). Code map: [ARCHITECTURE](ARCHITECTURE.md). Usage: [README](../README.md).