doos/docs/RESEARCH.md

Research: user-level containers with overlayfs

Measured on this machine: Ubuntu, kernel 7.0.0-38, AppArmor 5.0.2, slirp4netns 1.3.3, uid 1000, no root at runtime.

What works unprivileged

1. clone(CLONE_NEWUSER|NEWNS|NEWPID|NEWUTS|NEWIPC[|NEWNET]), then the parent writes setgroups=deny, uid_map, gid_map (single entry, own uid). No setuid helpers (newuidmap) needed.
2. overlayfs mounts inside the userns (kernel >= 5.11) with the userxattr option. Upper and work dirs must be on a filesystem with user xattrs (ext4, tmpfs >= 6.6 verified for ours).
3. Mounts that work inside: tmpfs, proc (needs own pidns), devpts (new instance, no gid= since gid 5 is unmapped), bind mounts of host /dev/{null,zero,full,random,urandom,tty} (no mknod in a userns), read-only remount of /proc subpaths (must copy locked flags from statvfs).
4. pivot_root(".", ".") plus lazy unmount of the old root.

What does not

• Writing mapped uid ranges beyond own uid needs newuidmap + /etc/subuid. Skipped: single mapping keeps the design dumb. Consequence: no chown/su to other uids inside, so apt install as non-root users does not work.
• cgroups: no memory/cpu/pid limits without systemd delegation. Not implemented (gap).
• Ubuntu 24.04+: kernel.apparmor_restrict_unprivileged_userns=1 lets any binary create a userns but strips capabilities inside it. Measured: uid_map write succeeds, first mount fails with EPERM. Fix is an AppArmor profile granting userns to the binary (apparmor/doos, install once with sudo). Turning the sysctl off is global and weaker.

Networking options

Option	Result
slirp4netns (chosen default)	Works unprivileged. Needs only the netns of the child pid. --disable-host-loopback blocks the container from reaching host 127.0.0.1 (measured: blocked). DNS via 10.0.2.3 works (measured).
pasta	Installed here but not used: slirp4netns suffices, one dependency.
veth + bridge	Needs CAP_NET_ADMIN in the host netns. Not possible unprivileged.
host netns	Works, zero isolation. Offered as explicit --net host.

Host safety design

• No host paths writable: overlay lower dirs are only reachable through overlay (writes copy up to the container upper). Verified: write to /usr and /etc inside leaves host untouched.
• Only /usr /etc /opt and compat dirs are exposed read-through. /home, /var, /sys, /root of the host are not present.
• Old root detached after pivot, no_new_privs, /proc hardened, nosuid/nodev on tmpfs/devpts/overlay.
• Residual exposure (gaps G1 to G6 below): no resource limits, no disk cap, single uid, host /etc readable, abstract unix sockets shared in --net host, no seccomp filter.

Measurements (this machine, 5 sequential attach -- true)

Mode	Per attach
slirp	~26 ms
none	~6 ms

Single run set, one machine; no other hardware measured.

Known gaps and researched solutions

Status per gap is in the table below and in "Resolution log (1.0.0)" at the end. Probes ran on the machine above (cgroup v2, systemd user manager, ext4 root, uid 1000). Probe cgroups were removed afterwards. Rows marked "unmeasured" have no run behind them.

Id	Gap	Fix	Status	Code
G1	Memory/cpu/pids limits	Cgroup v2 leaf under the delegated user service	partly done in 1.0.0: the leaf exists and is used for cgroup.freeze; memory.max, pids.max, cpu.max are not set yet	cgroup.cpp
G2	Disk cap on upper dirs	Usage watchdog with freeze	done in 1.0.0 (--size)	Sandbox::watch
G3	Single uid mapping	newuidmap/newgidmap with /etc/subuid, fallback single	done in 1.0.0 (--ids, default subuid)	idmap.cpp, Store::remove
G4	Host /etc readable	Allowlist-only private /etc	open for host mode: /etc is now a private copy of all readable files (stable, still world-readable secrets); closed in image mode (no host files)	container.cpp
G5	--net host shares abstract sockets	Keep opt-in; evaluate pasta	open	network.cpp
G6	No seccomp, nested userns allowed	Hand-built seccomp-bpf denylist	open	new Seccomp class
G7	Host-owned lower dirs unwritable, chown of host files fails, empty /var	Image layer + subuid; skeleton and seeding for host mode	done in 1.0.0	image.cpp, container.cpp

G1: memory, cpu and pids limits (cgroup v2)

Findings, measured:

• /sys/fs/cgroup is cgroup2 with nsdelegate. user@1000.service/cgroup.controllers lists cpuset cpu io memory pids, and its subtree_control already enables all five. A plain user can mkdir below /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/.
• A leaf with pids.max=20: the 21st fork failed with Cannot fork.
• A leaf with memory.max=64m: touching 200 MiB ended in Killed, memory.events showed oom_kill 1. An untouched bytearray(200MiB) is not charged (lazy pages), so the first attempt proved nothing; only the touched run counts.
• Gotcha, measured: enabling +memory +pids in subtree_control of a cgroup makes it a non-leaf; writing a pid into its cgroup.procs then fails with I/O error. The layout must be .../doos/ (controllers enabled here, no processes) with one leaf per container .../doos/<name>/ (limits set here, processes here).
• RLIMIT_NPROC is not an option: it counts per real uid across the whole host user, not per container.

Design:

1. Discover the base: parse /proc/self/cgroup, take the path up to and including user@<uid>.service.
2. Parent creates doos/<name> after clone, writes memory.max, memory.swap.max=0, pids.max, cpu.max, then writes the child pid to cgroup.procs before the sync byte G (child never runs outside limits). join() children enter via the same cgroup.procs write before setns.
3. On exit: write 1 to cgroup.kill (kernel >= 5.14), then rmdir.
4. Defaults as constants in spec.hpp (static ceilings, no autotuning): for example memory 2g, pids 512, cpu unlimited. Flags --memory, --pids, --cpus may lower or raise within a hard cap.
5. No delegation (no systemd user manager, cgroup v1): fail loudly with a counter and a clear reason; --no-limits is an explicit opt-out, never silent.

Rejected: systemd-run --user --scope -p MemoryMax=... (needs D-Bus, one more moving part). Unmeasured: behavior when doos itself runs inside a scope that is not under user@.service (the tmux scope here is, but other launchers may not be).

Sources: rootlesscontaine.rs cgroup v2, runc cgroup v2, Docker rootless tips.

G2: disk usage of upper dirs

Findings:

• Project quotas (XFS pquota, ext4 project quota) need root to set. This machine's root is ext4 on /dev/md2. Not usable unprivileged. Docker's overlay2 size limit also only works on XFS with pquota (source).
• Loop-mounted ext4 images need root; the kernel does not allow ext4 mounts from a user namespace. fuse2fs is not installed here. fuse-overlayfs is installed but has no quota either.
• A size= tmpfs mounted inside the container userns is enforced, measured: size=1m, writing 5 MiB ended in No space left on device.

Design (follows the rule "machines hold, humans analyze"):

1. Default: the parent process (already alive in Sandbox::start) measures upper-dir usage on a fixed interval with a floor (for example every 30 s, never faster). Soft limit: warning on stderr. Hard limit: write 1 to the container's cgroup.freeze (needs G1). Freeze stops processes but deletes nothing; the human frees space or raises the limit. Never auto-delete.
2. Opt-in --ephemeral: upper dirs on a size-capped tmpfs. Hard cap by the kernel, but memory-backed, counted in the memory cgroup, and lost on exit.
3. Constants (soft, hard, interval) in spec.hpp.

Unmeasured: cost of scanning a large upper dir (needs a run on a populated container); overshoot between two scans.

G3: more than one uid inside

Findings, measured with podman unshare (map 0 -> 1000, 1..65536 -> 100000..): overlayfs with userxattr mounts fine under a multi-id map. /etc/subuid and /etc/subgid contain retoor:100000:65536, newuidmap and newgidmap are installed.

• Host root-owned files in the lower dirs show as 65534 nobody. That is already true with the single map, so nothing gets worse.
• Gain: chown, su, apt sandbox users, daemons that drop privileges.
• Cost, important: files created by container uids 1..65535 are owned by host uids 100000+. The host user cannot delete them. Store::remove as written would fail with Permission denied. Delete must run inside a user namespace with the same map (re-enter and rm there, or podman unshare-style helper) before removing the directory.

Design: after clone, parent execs newuidmap <pid> 0 <uid> 1 1 <sub> 65536 and the gid equivalent, setgroups allowed. Fallback to the current single map when the user has no subuid range, with a printed notice. Store the chosen mode in meta (idmap=single|subuid) so delete knows which path to take.

Unmeasured: apt install end to end.

G4: host /etc visible inside

Finding, measured: find /etc -type f -perm -004 returns 1116 world-readable files on this host, including names such as /etc/fail2ban/action.d/cloudflare-token.conf (content not read). Anything world-readable in host /etc is readable in every container.

Design: replace the /etc overlay by a private /etc. At create time, populate private/etc from an allowlist in spec.hpp (kEtcAllow): passwd group nsswitch.conf ld.so.cache ld.so.conf ld.so.conf.d ssl/certs ca-certificates alternatives bash.bashrc profile profile.d localtime timezone os-release protocols services hosts hostname, plus a generated resolv.conf. Two ways: copy (stable, can go stale, for example CA certs) or read-only bind of each allowed path (always current, more mounts). Recommendation: read-only bind, since doos already carries the mount machinery.

Unmeasured: whether this allowlist is enough for apt, curl, python3, git; needs a run per tool before fixing the list.

G5: --net host and abstract unix sockets

Fact: abstract AF_UNIX sockets belong to the network namespace, so --net host exposes host abstract sockets (some X11 and D-Bus setups use them) and host loopback services. This is inherent to sharing the namespace.

Researched alternative, pasta (installed, pasta 0.0~git20260120): it attaches to an existing netns by pid (pasta --config-net <pid>), gives outbound access, and mirrors host addresses into the namespace. Measured: outbound TCP worked, and an inside connection to 127.0.0.1:22 reached the host (default flags, once). A second run with --no-splice --no-map-gw --map-host-loopback none printed Couldn't get any nameserver address, and I could not confirm the loopback result from it. Conclusion: pasta is not proven to match slirp4netns' --disable-host-loopback guarantee. Slirp stays the default. Next step is a clean test matrix of pasta flags with a listener on a known host port.

Interim: host remains an explicit opt-in documented as unisolated.

G6: kernel attack surface (seccomp, nested user namespaces)

Findings:

• A seccomp-bpf filter installs unprivileged once no_new_privs is set (already done). Measured with a raw prctl(PR_SET_SECCOMP) program: a denied syscall (keyctl) returned -1 EPERM. No libseccomp headers on this machine, and "fewer dependencies" argues for hand-assembled BPF.
• /proc/sys is read-only in the container, so user.max_user_namespaces=0 cannot be set from inside; seccomp is the lever.

Design: denylist modelled on the Docker default profile, applied after setupRoot and before exec, and also in the join() path. Check AUDIT_ARCH first and kill on a foreign ABI (blocks the x32 bypass). Deny with EPERM: kexec_load kexec_file_load bpf perf_event_open userfaultfd keyctl add_key request_key open_by_handle_at process_vm_readv process_vm_writev reboot swapon swapoff init_module finit_module delete_module. Nested userns: deny unshare and setns with CLONE_NEWUSER, deny clone with CLONE_NEWUSER, and return ENOSYS for clone3 (flags live in a struct, cannot be inspected; libc falls back to clone). Trade-off: breaks nested containers and Chromium-style sandboxes; make it a documented constant, not a switch that hides in code.

Unmeasured: compatibility impact on real workloads.

Priority

1. G1 (limits): cheapest, measured to work, unlocks G2's freeze.
2. G6 (seccomp): self-contained, no new dependency.
3. G4 (private /etc): needs the per-tool allowlist runs first.
4. G2, then G3 (G3 changes the delete path, do it deliberately).
5. G5: needs a test matrix before any change.

Field trial: apt and common tools (v0.1.2, measured)

Run in a fresh --net slirp container on the host above (Ubuntu with apt 3.2.0, dpkg 1.23.7).

Step	Result
curl https, git ls-remote, gcc compile+run, python3 (ssl, sqlite3), pip, ping 1.1.1.1	all work
apt-get update as shipped	fails: /var/lib/apt/lists/partial is missing (/var is an empty private dir)
apt-get install as shipped	fails: /var/lib/dpkg/lock-frontend missing, no dpkg status database
after seeding /var (dirs plus copy of host dpkg/status, info, alternatives, 40 MB)	apt starts, but the download method dies: setegid 65534 failed, seteuid 42 failed (sandbox user _apt cannot exist with a single id)
with -o APT::Sandbox::User=root	apt-get update works: 58.9 MB fetched in 3 s
apt-get install cowsay	download works; unpack fails: unable to create '/usr/games/cowsay.dpkg-new': Permission denied
after pre-creating the host directory tree in the upper dirs (13164 dirs for /usr, done with a shell loop, 38 s; not yet implemented in C++)	unpack works, /usr/games/cowsay hoi runs; host /usr/games/cowsay still absent (isolation holds)
configure step	fails: chown: changing ownership of '/etc/shells.tmp': Invalid argument (debianutils postinst copies owner of /etc/shells, which shows as 65534), so dpkg reports 4 packages not fully installed

Root causes:

1. Host files owned by host root appear as nobody (uid 65534). Container root has CAP_DAC_OVERRIDE only over inodes whose owner is mapped, so it cannot write into host-root-owned directories of the lower layer, and cannot chown to or from an unmapped owner.
2. /var has no package database.
3. apt wants a second uid (G3).

Conclusion: the current design (host /usr, /etc, /opt as lower layer) is good for running tools and for isolated experiments, not for package management. Bare tools and compilers work. Honest status: apt-get install does not complete today.

Researched ways forward, in order of effort:

• Short term, writable skeleton: at create time mirror every directory of the lower dirs into the upper dir with the same mode (directories owned by the user are writable; merged view is unchanged). Directories unreadable to the user (for example /etc/ssl/private, mode 700, root-owned) are skipped and stay as they are. Measured effect: writes into /usr/bin work, attach time unchanged (41 ms). Unmeasured: C++ implementation speed, and it does not fix the chown failure.
• Proper fix, image layer: use an extracted distro rootfs (for example from podman export or skopeo copy plus unpack, both installed here; mmdebstrap and debootstrap are not) as the lower dir, with ownership shifted into the subuid range (G3). Then the tree is owned by mapped ids, /var/lib/dpkg exists, chown works, _apt exists, and host /etc stops leaking (G4). This also removes the dependence on the host distro. Cost: image import step, disk per image (shared lower, so once), and a layout change, hence a major bump.
• Alternative to evaluate: fuse-overlayfs (installed) supports uid mappings between layers. Unmeasured; adds a FUSE process per container.

Resolution log (1.0.0)

Everything below was built and measured on the machine above (Ubuntu 26.04, kernel 7.0.0-38, ext4 root, uid 1000, subuid range 100000:65536). The end-to-end suite (tests/run.sh) passed 56 of 56.

What the field trial broke, and what fixed it

Symptom (0.1.3)	Root cause	Fix in 1.0.0	Result
mkdir/touch in /usr/bin fails	host dirs show as nobody; container root has no right over unmapped owners	fsutil::mirrorDirs: every host directory recreated in the upper layer, owned by the container	13,163 dirs for /usr in about 1.1 s (the shell loop took 38 s); touch /usr/bin/x works, attach time unchanged
/var/lib/apt/lists/partial is missing, no dpkg lock	/var was empty	/var pre-seeded with the apt/dpkg dirs (kVarDirs); host /var/lib/dpkg overlaid	apt starts
open lock-frontend: Permission denied	host lock files owned by nobody, mode 640	empty lock files pre-created in the upper (kUpperLockFiles)	lock acquired
dpkg: error creating backup file status-old: Operation not permitted	dpkg hard-links status; kernel protected_hardlinks refuses links to files you do not own and cannot write	regular files of /var/lib/dpkg, triggers, alternatives copied into the upper (kUpperCopyDirs)	dpkg state writable
chown: changing ownership of '/etc/shells.tmp': Invalid argument	chown --reference of a file owned by an unmapped uid	/etc is a private copy owned by the container root (kCopiedDirs)	stat -c %u /etc/shells is 0
setegid 65534 failed, seteuid 42 failed (apt _apt user)	one mappable id	subuid mapping through newuidmap/newgidmap (ids 1..65535)	apt-get update works with no override; with --ids single an APT::Sandbox::User "root" override is written instead
apt-get install cowsay not configured	all of the above	all of the above	install completes, dpkg --audit clean, /usr/games/cowsay runs, host has no /usr/games/cowsay

Image layer

• Why images are the clean fix: the extracted tree is owned by mapped ids, so nothing needs a skeleton, copies or overrides. Measured: apk add jq in Alpine and apt-get install cowsay curl ca-certificates in Ubuntu 26.04 complete with a clean dpkg --audit; curl https://example.com returns 200.
• Sources chosen (checked live with curl, 2026-10-23):
  • Alpine: latest-releases.yaml lists file name and sha256 per flavor (alpine-minirootfs-3.24.3-x86_64.tar.gz, 3.7 MB).
  • Ubuntu: cdimage.ubuntu.com/ubuntu-base/releases/<ver>/release/SHA256SUMS plus tarballs (26.04.1 and 24.04.5 present).
  • Debian: no usable tarball path found (debuerreotype/docker-debian-artifacts returned 404 for rootfs.tar.xz; only .sha256 files resolved), so Debian is supported through image add URL --sha256, not through the catalog.
  • Not used: podman export / skopeo (installed here, but one more dependency; a registry client is out of scope).
• Trust model: HTTPS to the publisher plus the checksum file from the same publisher. The Ubuntu SHA256SUMS.gpg signature is not checked (unimplemented, listed below).
• Extraction runs inside a user namespace with the id map the containers will use (UserNsJob): subuid images keep the real ownership; single-id images use tar --no-same-owner. A container's id mode is pinned to its image's.
• Cleanup: files owned by subuids cannot be removed by the host user (Permission denied). Store::remove and ImageStore::remove therefore delete from inside the id map. Measured: deleting containers and images with subuid-owned files works, and a failed add leaves no staging directory.
• Image sizes (hardlinks counted once): Alpine 8.3M, Ubuntu base 97M. The first measurement said 1.3G because uutils-coreutils hard-links one 11 MB binary about 100 times; diskUsage now deduplicates by (device, inode).

Size cap

• Mechanism: Sandbox::watch polls the init's pidfd with a 10 s timeout and scans usage on every wake. 80% warns, 100% freezes the container's cgroup, below 95% it thaws.
• Measured: --size 64m, a 100 MB write: FROZEN message at 100M of 64M, tick output stopped; doos attach cap -- rm /root/big ran (a second attach is not in the frozen cgroup), the container thawed (thawed (4.0K used)) and continued.
• Freezing needs no cgroup controller, only the delegated user@UID.service subtree. Without it the cap warns and says so.
• Chosen over a size-capped tmpfs because tmpfs is memory-backed and not persistent; chosen over project quota because that needs root.

Timings (5 sequential attach -- true, this machine, subuid, one run set)

Mode	Per attach
host, slirp	about 50 ms
image alpine, slirp	about 40 ms
image ubuntu, slirp	about 40 ms
host, --net none	about 10 ms

Subuid mapping added about 15 to 25 ms per start compared with 0.1.x (two setuid helper processes). Creating a host-mode container (skeleton, /etc copy, dpkg files) took about 1.2 s once. First --image alpine use including download and extraction: 0.3 s end to end on this network.

Still open, stated plainly

• G1: memory.max, pids.max, cpu.max are not set. Measured earlier that they work here; not wired in.
• G4: host mode copies every readable file of host /etc at creation. Use an image to avoid host files entirely.
• G5, G6: unchanged (see above).
• Size accounting sees only what the host user can read. Directories of other container uids with mode 700 inside the writable layer are not counted (reported only as an internal skip count, not shown). Unmeasured.
• Overshoot: up to one scan interval (10 s) of writes can pass the cap before the freeze; measured 100M written against a 64M cap.
• A second attach is outside the frozen cgroup and outside any limit by design (it is the rescue path).
• Host-mode containers drift from the host: the /etc copy, dpkg state and skeleton are snapshots at creation. A host package upgrade is visible in /usr but not in the container's dpkg database.
• Ubuntu image signature (SHA256SUMS.gpg) is not verified; Alpine's yaml is not signature-checked either.
• image add FILE without --sha256 is accepted and prints a note, since the user chose the file.
• Catalog covers x86_64 and aarch64 only; aarch64 is untested.