diff --git a/CHANGELOG.md b/CHANGELOG.md index e769952..3337eb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,8 @@ # Changelog +## 0.1.2 - 2026-10-23 +- Docs: researched solutions for all known gaps (G1 to G6) in `docs/RESEARCH.md`, with measurements. Nothing implemented yet. + ## 0.1.1 - 2026-10-23 - Docs: AGENTS.md release workflow, ARCHITECTURE.md, expanded README, this changelog. - Tooling: `scripts/release.sh` (bump, build, test, commit, tag). diff --git a/README.md b/README.md index 95bd503..d70fbdb 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # doos -Version: see `VERSION` (currently 0.1.1). Release history: `CHANGELOG.md`. +Version: see `VERSION` (currently 0.1.2). Release history: `CHANGELOG.md`. User-level containers in C++20: overlayfs over the host's `/usr`, `/etc`, `/opt`, user/mount/pid namespaces, networking through slirp4netns. No root at runtime. @@ -60,7 +60,7 @@ join it. Data persists until `delete --yes`. Host files are never written: system dirs are reached only through overlayfs. Host `/home`, `/sys` and host loopback are not reachable in the default mode. `--net host` removes network isolation and is opt-in. Known gaps (no cgroups, readable host `/etc`) -are listed in `docs/RESEARCH.md`. +are listed with researched solutions (G1 to G6, not implemented) in `docs/RESEARCH.md`. ## Project docs diff --git a/VERSION b/VERSION index 17e51c3..d917d3e 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.1.1 +0.1.2 diff --git a/docs/RESEARCH.md b/docs/RESEARCH.md index 4009e0c..ab5c93b 100644 --- a/docs/RESEARCH.md +++ b/docs/RESEARCH.md @@ -29,7 +29,7 @@ Measured on this machine: Ubuntu, kernel 7.0.0-38, AppArmor 5.0.2, slirp4netns 1 - No host paths writable: overlay lower dirs are only reachable through overlay (writes copy up to the container upper). Verified: write to `/usr` and `/etc` inside leaves host untouched. - Only `/usr /etc /opt` and compat dirs are exposed read-through. `/home`, `/var`, `/sys`, `/root` of the host are not present. - Old root detached after pivot, `no_new_privs`, `/proc` hardened, nosuid/nodev on tmpfs/devpts/overlay. -- Residual exposure (gaps): host `/etc` is readable inside (any world-readable secret in it); abstract unix sockets are shared in `--net host`; no resource cgroups, so a container can fill the disk through its upper dir or fork-bomb within the user's `RLIMIT_NPROC`; kernel attack surface of userns is that of the host kernel. +- Residual exposure (gaps G1 to G6 below): no resource limits, no disk cap, single uid, host `/etc` readable, abstract unix sockets shared in `--net host`, no seccomp filter. ## Measurements (this machine, 5 sequential `attach -- true`) @@ -39,3 +39,97 @@ Measured on this machine: Ubuntu, kernel 7.0.0-38, AppArmor 5.0.2, slirp4netns 1 | none | ~6 ms | Single run set, one machine; no other hardware measured. + +## Known gaps and researched solutions + +Status of everything in this section: **researched, not implemented**. Probes ran on the +machine above (cgroup v2, systemd user manager, ext4 root, uid 1000). Probe cgroups were +removed afterwards. Rows marked "unmeasured" have no run behind them. + +| Id | Gap | Proposed fix | Bump | Code that would change | +|---|---|---|---|---| +| G1 | No memory/cpu/pids limits | Per-container cgroup v2 leaf under the delegated user service | minor | `sandbox.cpp` start/join, new `Cgroup` class, `spec.hpp` | +| G2 | No disk cap on upper dirs | Usage watchdog with `cgroup.freeze` at the hard limit; optional tmpfs upper | minor | `Sandbox::start` parent loop, `spec.hpp` | +| G3 | Single uid mapping | `newuidmap`/`newgidmap` with `/etc/subuid`, fallback to single map | minor, delete path changes | `sandbox.cpp` writeMap, `Store::remove` | +| G4 | Host `/etc` readable | Allowlist-only private `/etc` instead of an overlay of host `/etc` | minor | `spec.hpp` lists, `setupRoot` | +| G5 | `--net host` shares abstract sockets | Keep opt-in; evaluate pasta as an isolated replacement | patch or minor | `network.cpp` | +| G6 | No seccomp, nested userns allowed | Hand-built seccomp-bpf denylist after `no_new_privs` | minor | new `Seccomp` class, `sandbox.cpp` | + +### G1: memory, cpu and pids limits (cgroup v2) + +Findings, measured: +- `/sys/fs/cgroup` is cgroup2 with `nsdelegate`. `user@1000.service/cgroup.controllers` lists `cpuset cpu io memory pids`, and its `subtree_control` already enables all five. A plain user can `mkdir` below `/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/`. +- A leaf with `pids.max=20`: the 21st fork failed with `Cannot fork`. +- A leaf with `memory.max=64m`: touching 200 MiB ended in `Killed`, `memory.events` showed `oom_kill 1`. An untouched `bytearray(200MiB)` is not charged (lazy pages), so the first attempt proved nothing; only the touched run counts. +- Gotcha, measured: enabling `+memory +pids` in `subtree_control` of a cgroup makes it a non-leaf; writing a pid into its `cgroup.procs` then fails with `I/O error`. The layout must be `.../doos/` (controllers enabled here, no processes) with one leaf per container `.../doos//` (limits set here, processes here). +- `RLIMIT_NPROC` is not an option: it counts per real uid across the whole host user, not per container. + +Design: +1. Discover the base: parse `/proc/self/cgroup`, take the path up to and including `user@.service`. +2. Parent creates `doos/` after `clone`, writes `memory.max`, `memory.swap.max=0`, `pids.max`, `cpu.max`, then writes the child pid to `cgroup.procs` before the sync byte `G` (child never runs outside limits). `join()` children enter via the same `cgroup.procs` write before `setns`. +3. On exit: write `1` to `cgroup.kill` (kernel >= 5.14), then `rmdir`. +4. Defaults as constants in `spec.hpp` (static ceilings, no autotuning): for example memory 2g, pids 512, cpu unlimited. Flags `--memory`, `--pids`, `--cpus` may lower or raise within a hard cap. +5. No delegation (no systemd user manager, cgroup v1): fail loudly with a counter and a clear reason; `--no-limits` is an explicit opt-out, never silent. + +Rejected: `systemd-run --user --scope -p MemoryMax=...` (needs D-Bus, one more moving part). Unmeasured: behavior when doos itself runs inside a scope that is not under `user@.service` (the tmux scope here is, but other launchers may not be). + +Sources: [rootlesscontaine.rs cgroup v2](https://rootlesscontaine.rs/getting-started/common/cgroup2/), [runc cgroup v2](https://github.com/thaJeztah/runc/blob/main/docs/cgroup-v2.md), [Docker rootless tips](https://docs.docker.com/engine/security/rootless/tips/). + +### G2: disk usage of upper dirs + +Findings: +- Project quotas (XFS `pquota`, ext4 project quota) need root to set. This machine's root is ext4 on `/dev/md2`. Not usable unprivileged. Docker's overlay2 size limit also only works on XFS with `pquota` ([source](https://fabianlee.org/2020/01/15/docker-use-overlay2-with-an-xfs-backing-filesystem-to-limit-rootfs-size/)). +- Loop-mounted ext4 images need root; the kernel does not allow ext4 mounts from a user namespace. `fuse2fs` is not installed here. `fuse-overlayfs` is installed but has no quota either. +- A `size=` tmpfs mounted inside the container userns is enforced, measured: `size=1m`, writing 5 MiB ended in `No space left on device`. + +Design (follows the rule "machines hold, humans analyze"): +1. Default: the parent process (already alive in `Sandbox::start`) measures upper-dir usage on a fixed interval with a floor (for example every 30 s, never faster). Soft limit: warning on stderr. Hard limit: write `1` to the container's `cgroup.freeze` (needs G1). Freeze stops processes but deletes nothing; the human frees space or raises the limit. Never auto-delete. +2. Opt-in `--ephemeral`: upper dirs on a size-capped tmpfs. Hard cap by the kernel, but memory-backed, counted in the memory cgroup, and lost on exit. +3. Constants (soft, hard, interval) in `spec.hpp`. + +Unmeasured: cost of scanning a large upper dir (needs a run on a populated container); overshoot between two scans. + +### G3: more than one uid inside + +Findings, measured with `podman unshare` (map `0 -> 1000`, `1..65536 -> 100000..`): overlayfs with `userxattr` mounts fine under a multi-id map. `/etc/subuid` and `/etc/subgid` contain `retoor:100000:65536`, `newuidmap` and `newgidmap` are installed. +- Host root-owned files in the lower dirs show as `65534 nobody`. That is already true with the single map, so nothing gets worse. +- Gain: `chown`, `su`, `apt` sandbox users, daemons that drop privileges. +- Cost, important: files created by container uids 1..65535 are owned by host uids 100000+. The host user cannot delete them. `Store::remove` as written would fail with `Permission denied`. Delete must run inside a user namespace with the same map (re-enter and `rm` there, or `podman unshare`-style helper) before removing the directory. + +Design: after `clone`, parent execs `newuidmap 0 1 1 65536` and the gid equivalent, `setgroups` allowed. Fallback to the current single map when the user has no subuid range, with a printed notice. Store the chosen mode in `meta` (`idmap=single|subuid`) so delete knows which path to take. + +Unmeasured: `apt install` end to end. + +### G4: host `/etc` visible inside + +Finding, measured: `find /etc -type f -perm -004` returns 1116 world-readable files on this host, including names such as `/etc/fail2ban/action.d/cloudflare-token.conf` (content not read). Anything world-readable in host `/etc` is readable in every container. + +Design: replace the `/etc` overlay by a private `/etc`. At create time, populate `private/etc` from an allowlist in `spec.hpp` (`kEtcAllow`): `passwd group nsswitch.conf ld.so.cache ld.so.conf ld.so.conf.d ssl/certs ca-certificates alternatives bash.bashrc profile profile.d localtime timezone os-release protocols services hosts hostname`, plus a generated `resolv.conf`. Two ways: copy (stable, can go stale, for example CA certs) or read-only bind of each allowed path (always current, more mounts). Recommendation: read-only bind, since doos already carries the mount machinery. + +Unmeasured: whether this allowlist is enough for `apt`, `curl`, `python3`, `git`; needs a run per tool before fixing the list. + +### G5: `--net host` and abstract unix sockets + +Fact: abstract `AF_UNIX` sockets belong to the network namespace, so `--net host` exposes host abstract sockets (some X11 and D-Bus setups use them) and host loopback services. This is inherent to sharing the namespace. + +Researched alternative, pasta (installed, `pasta 0.0~git20260120`): it attaches to an existing netns by pid (`pasta --config-net `), gives outbound access, and mirrors host addresses into the namespace. Measured: outbound TCP worked, and an inside connection to `127.0.0.1:22` reached the host (default flags, once). A second run with `--no-splice --no-map-gw --map-host-loopback none` printed `Couldn't get any nameserver address`, and I could not confirm the loopback result from it. Conclusion: pasta is not proven to match slirp4netns' `--disable-host-loopback` guarantee. Slirp stays the default. Next step is a clean test matrix of pasta flags with a listener on a known host port. + +Interim: `host` remains an explicit opt-in documented as unisolated. + +### G6: kernel attack surface (seccomp, nested user namespaces) + +Findings: +- A seccomp-bpf filter installs unprivileged once `no_new_privs` is set (already done). Measured with a raw `prctl(PR_SET_SECCOMP)` program: a denied syscall (`keyctl`) returned `-1 EPERM`. No `libseccomp` headers on this machine, and "fewer dependencies" argues for hand-assembled BPF. +- `/proc/sys` is read-only in the container, so `user.max_user_namespaces=0` cannot be set from inside; seccomp is the lever. + +Design: denylist modelled on the Docker default profile, applied after `setupRoot` and before `exec`, and also in the `join()` path. Check `AUDIT_ARCH` first and kill on a foreign ABI (blocks the x32 bypass). Deny with `EPERM`: `kexec_load kexec_file_load bpf perf_event_open userfaultfd keyctl add_key request_key open_by_handle_at process_vm_readv process_vm_writev reboot swapon swapoff init_module finit_module delete_module`. Nested userns: deny `unshare` and `setns` with `CLONE_NEWUSER`, deny `clone` with `CLONE_NEWUSER`, and return `ENOSYS` for `clone3` (flags live in a struct, cannot be inspected; libc falls back to `clone`). Trade-off: breaks nested containers and Chromium-style sandboxes; make it a documented constant, not a switch that hides in code. + +Unmeasured: compatibility impact on real workloads. + +### Priority + +1. G1 (limits): cheapest, measured to work, unlocks G2's freeze. +2. G6 (seccomp): self-contained, no new dependency. +3. G4 (private `/etc`): needs the per-tool allowlist runs first. +4. G2, then G3 (G3 changes the delete path, do it deliberately). +5. G5: needs a test matrix before any change.