doos/docs/RESEARCH.md

Research: user-level containers with overlayfs

Measured on this machine: Ubuntu, kernel 7.0.0-38, AppArmor 5.0.2, slirp4netns 1.3.3, uid 1000, no root at runtime.

What works unprivileged

1. clone(CLONE_NEWUSER|NEWNS|NEWPID|NEWUTS|NEWIPC[|NEWNET]), then the parent writes setgroups=deny, uid_map, gid_map (single entry, own uid). No setuid helpers (newuidmap) needed.
2. overlayfs mounts inside the userns (kernel >= 5.11) with the userxattr option. Upper and work dirs must be on a filesystem with user xattrs (ext4, tmpfs >= 6.6 verified for ours).
3. Mounts that work inside: tmpfs, proc (needs own pidns), devpts (new instance, no gid= since gid 5 is unmapped), bind mounts of host /dev/{null,zero,full,random,urandom,tty} (no mknod in a userns), read-only remount of /proc subpaths (must copy locked flags from statvfs).
4. pivot_root(".", ".") plus lazy unmount of the old root.

What does not

• Writing mapped uid ranges beyond own uid needs newuidmap + /etc/subuid. Skipped: single mapping keeps the design dumb. Consequence: no chown/su to other uids inside, so apt install as non-root users does not work.
• cgroups: no memory/cpu/pid limits without systemd delegation. Not implemented (gap).
• Ubuntu 24.04+: kernel.apparmor_restrict_unprivileged_userns=1 lets any binary create a userns but strips capabilities inside it. Measured: uid_map write succeeds, first mount fails with EPERM. Fix is an AppArmor profile granting userns to the binary (apparmor/doos, install once with sudo). Turning the sysctl off is global and weaker.

Networking options

Option	Result
slirp4netns (chosen default)	Works unprivileged. Needs only the netns of the child pid. --disable-host-loopback blocks the container from reaching host 127.0.0.1 (measured: blocked). DNS via 10.0.2.3 works (measured).
pasta	Installed here but not used: slirp4netns suffices, one dependency.
veth + bridge	Needs CAP_NET_ADMIN in the host netns. Not possible unprivileged.
host netns	Works, zero isolation. Offered as explicit --net host.

Host safety design

• No host paths writable: overlay lower dirs are only reachable through overlay (writes copy up to the container upper). Verified: write to /usr and /etc inside leaves host untouched.
• Only /usr /etc /opt and compat dirs are exposed read-through. /home, /var, /sys, /root of the host are not present.
• Old root detached after pivot, no_new_privs, /proc hardened, nosuid/nodev on tmpfs/devpts/overlay.
• Residual exposure (gaps G1 to G6 below): no resource limits, no disk cap, single uid, host /etc readable, abstract unix sockets shared in --net host, no seccomp filter.

Measurements (this machine, 5 sequential attach -- true)

Mode	Per attach
slirp	~26 ms
none	~6 ms

Single run set, one machine; no other hardware measured.

Known gaps and researched solutions

Status of everything in this section: researched, not implemented. Probes ran on the machine above (cgroup v2, systemd user manager, ext4 root, uid 1000). Probe cgroups were removed afterwards. Rows marked "unmeasured" have no run behind them.

Id	Gap	Proposed fix	Bump	Code that would change
G1	No memory/cpu/pids limits	Per-container cgroup v2 leaf under the delegated user service	minor	sandbox.cpp start/join, new Cgroup class, spec.hpp
G2	No disk cap on upper dirs	Usage watchdog with cgroup.freeze at the hard limit; optional tmpfs upper	minor	Sandbox::start parent loop, spec.hpp
G3	Single uid mapping	newuidmap/newgidmap with /etc/subuid, fallback to single map	minor, delete path changes	sandbox.cpp writeMap, Store::remove
G4	Host /etc readable	Allowlist-only private /etc instead of an overlay of host /etc	minor	spec.hpp lists, setupRoot
G5	--net host shares abstract sockets	Keep opt-in; evaluate pasta as an isolated replacement	patch or minor	network.cpp
G6	No seccomp, nested userns allowed	Hand-built seccomp-bpf denylist after no_new_privs	minor	new Seccomp class, sandbox.cpp
G7	Host-owned lower dirs are not writable; chown --reference of host files fails; /var is empty so apt has no databases	Image-based lower layer (extracted rootfs, owned by mapped ids) plus G3; short term: pre-seeded directory skeleton	major (changes layout)	Container::prepareLayout, setupRoot, new image import

G1: memory, cpu and pids limits (cgroup v2)

Findings, measured:

• /sys/fs/cgroup is cgroup2 with nsdelegate. user@1000.service/cgroup.controllers lists cpuset cpu io memory pids, and its subtree_control already enables all five. A plain user can mkdir below /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/.
• A leaf with pids.max=20: the 21st fork failed with Cannot fork.
• A leaf with memory.max=64m: touching 200 MiB ended in Killed, memory.events showed oom_kill 1. An untouched bytearray(200MiB) is not charged (lazy pages), so the first attempt proved nothing; only the touched run counts.
• Gotcha, measured: enabling +memory +pids in subtree_control of a cgroup makes it a non-leaf; writing a pid into its cgroup.procs then fails with I/O error. The layout must be .../doos/ (controllers enabled here, no processes) with one leaf per container .../doos/<name>/ (limits set here, processes here).
• RLIMIT_NPROC is not an option: it counts per real uid across the whole host user, not per container.

Design:

1. Discover the base: parse /proc/self/cgroup, take the path up to and including user@<uid>.service.
2. Parent creates doos/<name> after clone, writes memory.max, memory.swap.max=0, pids.max, cpu.max, then writes the child pid to cgroup.procs before the sync byte G (child never runs outside limits). join() children enter via the same cgroup.procs write before setns.
3. On exit: write 1 to cgroup.kill (kernel >= 5.14), then rmdir.
4. Defaults as constants in spec.hpp (static ceilings, no autotuning): for example memory 2g, pids 512, cpu unlimited. Flags --memory, --pids, --cpus may lower or raise within a hard cap.
5. No delegation (no systemd user manager, cgroup v1): fail loudly with a counter and a clear reason; --no-limits is an explicit opt-out, never silent.

Rejected: systemd-run --user --scope -p MemoryMax=... (needs D-Bus, one more moving part). Unmeasured: behavior when doos itself runs inside a scope that is not under user@.service (the tmux scope here is, but other launchers may not be).

Sources: rootlesscontaine.rs cgroup v2, runc cgroup v2, Docker rootless tips.

G2: disk usage of upper dirs

Findings:

• Project quotas (XFS pquota, ext4 project quota) need root to set. This machine's root is ext4 on /dev/md2. Not usable unprivileged. Docker's overlay2 size limit also only works on XFS with pquota (source).
• Loop-mounted ext4 images need root; the kernel does not allow ext4 mounts from a user namespace. fuse2fs is not installed here. fuse-overlayfs is installed but has no quota either.
• A size= tmpfs mounted inside the container userns is enforced, measured: size=1m, writing 5 MiB ended in No space left on device.

Design (follows the rule "machines hold, humans analyze"):

1. Default: the parent process (already alive in Sandbox::start) measures upper-dir usage on a fixed interval with a floor (for example every 30 s, never faster). Soft limit: warning on stderr. Hard limit: write 1 to the container's cgroup.freeze (needs G1). Freeze stops processes but deletes nothing; the human frees space or raises the limit. Never auto-delete.
2. Opt-in --ephemeral: upper dirs on a size-capped tmpfs. Hard cap by the kernel, but memory-backed, counted in the memory cgroup, and lost on exit.
3. Constants (soft, hard, interval) in spec.hpp.

Unmeasured: cost of scanning a large upper dir (needs a run on a populated container); overshoot between two scans.

G3: more than one uid inside

Findings, measured with podman unshare (map 0 -> 1000, 1..65536 -> 100000..): overlayfs with userxattr mounts fine under a multi-id map. /etc/subuid and /etc/subgid contain retoor:100000:65536, newuidmap and newgidmap are installed.

• Host root-owned files in the lower dirs show as 65534 nobody. That is already true with the single map, so nothing gets worse.
• Gain: chown, su, apt sandbox users, daemons that drop privileges.
• Cost, important: files created by container uids 1..65535 are owned by host uids 100000+. The host user cannot delete them. Store::remove as written would fail with Permission denied. Delete must run inside a user namespace with the same map (re-enter and rm there, or podman unshare-style helper) before removing the directory.

Design: after clone, parent execs newuidmap <pid> 0 <uid> 1 1 <sub> 65536 and the gid equivalent, setgroups allowed. Fallback to the current single map when the user has no subuid range, with a printed notice. Store the chosen mode in meta (idmap=single|subuid) so delete knows which path to take.

Unmeasured: apt install end to end.

G4: host /etc visible inside

Finding, measured: find /etc -type f -perm -004 returns 1116 world-readable files on this host, including names such as /etc/fail2ban/action.d/cloudflare-token.conf (content not read). Anything world-readable in host /etc is readable in every container.

Design: replace the /etc overlay by a private /etc. At create time, populate private/etc from an allowlist in spec.hpp (kEtcAllow): passwd group nsswitch.conf ld.so.cache ld.so.conf ld.so.conf.d ssl/certs ca-certificates alternatives bash.bashrc profile profile.d localtime timezone os-release protocols services hosts hostname, plus a generated resolv.conf. Two ways: copy (stable, can go stale, for example CA certs) or read-only bind of each allowed path (always current, more mounts). Recommendation: read-only bind, since doos already carries the mount machinery.

Unmeasured: whether this allowlist is enough for apt, curl, python3, git; needs a run per tool before fixing the list.

G5: --net host and abstract unix sockets

Fact: abstract AF_UNIX sockets belong to the network namespace, so --net host exposes host abstract sockets (some X11 and D-Bus setups use them) and host loopback services. This is inherent to sharing the namespace.

Researched alternative, pasta (installed, pasta 0.0~git20260120): it attaches to an existing netns by pid (pasta --config-net <pid>), gives outbound access, and mirrors host addresses into the namespace. Measured: outbound TCP worked, and an inside connection to 127.0.0.1:22 reached the host (default flags, once). A second run with --no-splice --no-map-gw --map-host-loopback none printed Couldn't get any nameserver address, and I could not confirm the loopback result from it. Conclusion: pasta is not proven to match slirp4netns' --disable-host-loopback guarantee. Slirp stays the default. Next step is a clean test matrix of pasta flags with a listener on a known host port.

Interim: host remains an explicit opt-in documented as unisolated.

G6: kernel attack surface (seccomp, nested user namespaces)

Findings:

• A seccomp-bpf filter installs unprivileged once no_new_privs is set (already done). Measured with a raw prctl(PR_SET_SECCOMP) program: a denied syscall (keyctl) returned -1 EPERM. No libseccomp headers on this machine, and "fewer dependencies" argues for hand-assembled BPF.
• /proc/sys is read-only in the container, so user.max_user_namespaces=0 cannot be set from inside; seccomp is the lever.

Design: denylist modelled on the Docker default profile, applied after setupRoot and before exec, and also in the join() path. Check AUDIT_ARCH first and kill on a foreign ABI (blocks the x32 bypass). Deny with EPERM: kexec_load kexec_file_load bpf perf_event_open userfaultfd keyctl add_key request_key open_by_handle_at process_vm_readv process_vm_writev reboot swapon swapoff init_module finit_module delete_module. Nested userns: deny unshare and setns with CLONE_NEWUSER, deny clone with CLONE_NEWUSER, and return ENOSYS for clone3 (flags live in a struct, cannot be inspected; libc falls back to clone). Trade-off: breaks nested containers and Chromium-style sandboxes; make it a documented constant, not a switch that hides in code.

Unmeasured: compatibility impact on real workloads.

Priority

1. G1 (limits): cheapest, measured to work, unlocks G2's freeze.
2. G6 (seccomp): self-contained, no new dependency.
3. G4 (private /etc): needs the per-tool allowlist runs first.
4. G2, then G3 (G3 changes the delete path, do it deliberately).
5. G5: needs a test matrix before any change.

Field trial: apt and common tools (v0.1.2, measured)

Run in a fresh --net slirp container on the host above (Ubuntu with apt 3.2.0, dpkg 1.23.7).

Step	Result
curl https, git ls-remote, gcc compile+run, python3 (ssl, sqlite3), pip, ping 1.1.1.1	all work
apt-get update as shipped	fails: /var/lib/apt/lists/partial is missing (/var is an empty private dir)
apt-get install as shipped	fails: /var/lib/dpkg/lock-frontend missing, no dpkg status database
after seeding /var (dirs plus copy of host dpkg/status, info, alternatives, 40 MB)	apt starts, but the download method dies: setegid 65534 failed, seteuid 42 failed (sandbox user _apt cannot exist with a single id)
with -o APT::Sandbox::User=root	apt-get update works: 58.9 MB fetched in 3 s
apt-get install cowsay	download works; unpack fails: unable to create '/usr/games/cowsay.dpkg-new': Permission denied
after pre-creating the host directory tree in the upper dirs (13164 dirs for /usr, done with a shell loop, 38 s; not yet implemented in C++)	unpack works, /usr/games/cowsay hoi runs; host /usr/games/cowsay still absent (isolation holds)
configure step	fails: chown: changing ownership of '/etc/shells.tmp': Invalid argument (debianutils postinst copies owner of /etc/shells, which shows as 65534), so dpkg reports 4 packages not fully installed

Root causes:

1. Host files owned by host root appear as nobody (uid 65534). Container root has CAP_DAC_OVERRIDE only over inodes whose owner is mapped, so it cannot write into host-root-owned directories of the lower layer, and cannot chown to or from an unmapped owner.
2. /var has no package database.
3. apt wants a second uid (G3).

Conclusion: the current design (host /usr, /etc, /opt as lower layer) is good for running tools and for isolated experiments, not for package management. Bare tools and compilers work. Honest status: apt-get install does not complete today.

Researched ways forward, in order of effort:

• Short term, writable skeleton: at create time mirror every directory of the lower dirs into the upper dir with the same mode (directories owned by the user are writable; merged view is unchanged). Directories unreadable to the user (for example /etc/ssl/private, mode 700, root-owned) are skipped and stay as they are. Measured effect: writes into /usr/bin work, attach time unchanged (41 ms). Unmeasured: C++ implementation speed, and it does not fix the chown failure.
• Proper fix, image layer: use an extracted distro rootfs (for example from podman export or skopeo copy plus unpack, both installed here; mmdebstrap and debootstrap are not) as the lower dir, with ownership shifted into the subuid range (G3). Then the tree is owned by mapped ids, /var/lib/dpkg exists, chown works, _apt exists, and host /etc stops leaking (G4). This also removes the dependence on the host distro. Cost: image import step, disk per image (shared lower, so once), and a layout change, hence a major bump.
• Alternative to evaluate: fuse-overlayfs (installed) supports uid mappings between layers. Unmeasured; adds a FUSE process per container.