|
|
|
@@ -29,7 +29,7 @@ Measured on this machine: Ubuntu, kernel 7.0.0-38, AppArmor 5.0.2, slirp4netns 1
|
|
|
|
|
- No host paths writable: overlay lower dirs are only reachable through overlay (writes copy up to the container upper). Verified: write to `/usr` and `/etc` inside leaves host untouched.
|
|
|
|
|
- Only `/usr /etc /opt` and compat dirs are exposed read-through. `/home`, `/var`, `/sys`, `/root` of the host are not present.
|
|
|
|
|
- Old root detached after pivot, `no_new_privs`, `/proc` hardened, nosuid/nodev on tmpfs/devpts/overlay.
|
|
|
|
|
- Residual exposure (gaps): host `/etc` is readable inside (any world-readable secret in it); abstract unix sockets are shared in `--net host`; no resource cgroups, so a container can fill the disk through its upper dir or fork-bomb within the user's `RLIMIT_NPROC`; kernel attack surface of userns is that of the host kernel.
|
|
|
|
|
- Residual exposure (gaps G1 to G6 below): no resource limits, no disk cap, single uid, host `/etc` readable, abstract unix sockets shared in `--net host`, no seccomp filter.
|
|
|
|
|
|
|
|
|
|
## Measurements (this machine, 5 sequential `attach -- true`)
|
|
|
|
|
|
|
|
|
@@ -39,3 +39,125 @@ Measured on this machine: Ubuntu, kernel 7.0.0-38, AppArmor 5.0.2, slirp4netns 1
|
|
|
|
|
| none | ~6 ms |
|
|
|
|
|
|
|
|
|
|
Single run set, one machine; no other hardware measured.
|
|
|
|
|
|
|
|
|
|
## Known gaps and researched solutions
|
|
|
|
|
|
|
|
|
|
Status of everything in this section: **researched, not implemented**. Probes ran on the
|
|
|
|
|
machine above (cgroup v2, systemd user manager, ext4 root, uid 1000). Probe cgroups were
|
|
|
|
|
removed afterwards. Rows marked "unmeasured" have no run behind them.
|
|
|
|
|
|
|
|
|
|
| Id | Gap | Proposed fix | Bump | Code that would change |
|
|
|
|
|
|---|---|---|---|---|
|
|
|
|
|
| G1 | No memory/cpu/pids limits | Per-container cgroup v2 leaf under the delegated user service | minor | `sandbox.cpp` start/join, new `Cgroup` class, `spec.hpp` |
|
|
|
|
|
| G2 | No disk cap on upper dirs | Usage watchdog with `cgroup.freeze` at the hard limit; optional tmpfs upper | minor | `Sandbox::start` parent loop, `spec.hpp` |
|
|
|
|
|
| G3 | Single uid mapping | `newuidmap`/`newgidmap` with `/etc/subuid`, fallback to single map | minor, delete path changes | `sandbox.cpp` writeMap, `Store::remove` |
|
|
|
|
|
| G4 | Host `/etc` readable | Allowlist-only private `/etc` instead of an overlay of host `/etc` | minor | `spec.hpp` lists, `setupRoot` |
|
|
|
|
|
| G5 | `--net host` shares abstract sockets | Keep opt-in; evaluate pasta as an isolated replacement | patch or minor | `network.cpp` |
|
|
|
|
|
| G6 | No seccomp, nested userns allowed | Hand-built seccomp-bpf denylist after `no_new_privs` | minor | new `Seccomp` class, `sandbox.cpp` |
|
|
|
|
|
| G7 | Host-owned lower dirs are not writable; `chown --reference` of host files fails; `/var` is empty so apt has no databases | Image-based lower layer (extracted rootfs, owned by mapped ids) plus G3; short term: pre-seeded directory skeleton | major (changes layout) | `Container::prepareLayout`, `setupRoot`, new image import |
|
|
|
|
|
|
|
|
|
|
### G1: memory, cpu and pids limits (cgroup v2)
|
|
|
|
|
|
|
|
|
|
Findings, measured:
|
|
|
|
|
- `/sys/fs/cgroup` is cgroup2 with `nsdelegate`. `user@1000.service/cgroup.controllers` lists `cpuset cpu io memory pids`, and its `subtree_control` already enables all five. A plain user can `mkdir` below `/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/`.
|
|
|
|
|
- A leaf with `pids.max=20`: the 21st fork failed with `Cannot fork`.
|
|
|
|
|
- A leaf with `memory.max=64m`: touching 200 MiB ended in `Killed`, `memory.events` showed `oom_kill 1`. An untouched `bytearray(200MiB)` is not charged (lazy pages), so the first attempt proved nothing; only the touched run counts.
|
|
|
|
|
- Gotcha, measured: enabling `+memory +pids` in `subtree_control` of a cgroup makes it a non-leaf; writing a pid into its `cgroup.procs` then fails with `I/O error`. The layout must be `.../doos/` (controllers enabled here, no processes) with one leaf per container `.../doos/<name>/` (limits set here, processes here).
|
|
|
|
|
- `RLIMIT_NPROC` is not an option: it counts per real uid across the whole host user, not per container.
|
|
|
|
|
|
|
|
|
|
Design:
|
|
|
|
|
1. Discover the base: parse `/proc/self/cgroup`, take the path up to and including `user@<uid>.service`.
|
|
|
|
|
2. Parent creates `doos/<name>` after `clone`, writes `memory.max`, `memory.swap.max=0`, `pids.max`, `cpu.max`, then writes the child pid to `cgroup.procs` before the sync byte `G` (child never runs outside limits). `join()` children enter via the same `cgroup.procs` write before `setns`.
|
|
|
|
|
3. On exit: write `1` to `cgroup.kill` (kernel >= 5.14), then `rmdir`.
|
|
|
|
|
4. Defaults as constants in `spec.hpp` (static ceilings, no autotuning): for example memory 2g, pids 512, cpu unlimited. Flags `--memory`, `--pids`, `--cpus` may lower or raise within a hard cap.
|
|
|
|
|
5. No delegation (no systemd user manager, cgroup v1): fail loudly with a counter and a clear reason; `--no-limits` is an explicit opt-out, never silent.
|
|
|
|
|
|
|
|
|
|
Rejected: `systemd-run --user --scope -p MemoryMax=...` (needs D-Bus, one more moving part). Unmeasured: behavior when doos itself runs inside a scope that is not under `user@.service` (the tmux scope here is, but other launchers may not be).
|
|
|
|
|
|
|
|
|
|
Sources: [rootlesscontaine.rs cgroup v2](https://rootlesscontaine.rs/getting-started/common/cgroup2/), [runc cgroup v2](https://github.com/thaJeztah/runc/blob/main/docs/cgroup-v2.md), [Docker rootless tips](https://docs.docker.com/engine/security/rootless/tips/).
|
|
|
|
|
|
|
|
|
|
### G2: disk usage of upper dirs
|
|
|
|
|
|
|
|
|
|
Findings:
|
|
|
|
|
- Project quotas (XFS `pquota`, ext4 project quota) need root to set. This machine's root is ext4 on `/dev/md2`. Not usable unprivileged. Docker's overlay2 size limit also only works on XFS with `pquota` ([source](https://fabianlee.org/2020/01/15/docker-use-overlay2-with-an-xfs-backing-filesystem-to-limit-rootfs-size/)).
|
|
|
|
|
- Loop-mounted ext4 images need root; the kernel does not allow ext4 mounts from a user namespace. `fuse2fs` is not installed here. `fuse-overlayfs` is installed but has no quota either.
|
|
|
|
|
- A `size=` tmpfs mounted inside the container userns is enforced, measured: `size=1m`, writing 5 MiB ended in `No space left on device`.
|
|
|
|
|
|
|
|
|
|
Design (follows the rule "machines hold, humans analyze"):
|
|
|
|
|
1. Default: the parent process (already alive in `Sandbox::start`) measures upper-dir usage on a fixed interval with a floor (for example every 30 s, never faster). Soft limit: warning on stderr. Hard limit: write `1` to the container's `cgroup.freeze` (needs G1). Freeze stops processes but deletes nothing; the human frees space or raises the limit. Never auto-delete.
|
|
|
|
|
2. Opt-in `--ephemeral`: upper dirs on a size-capped tmpfs. Hard cap by the kernel, but memory-backed, counted in the memory cgroup, and lost on exit.
|
|
|
|
|
3. Constants (soft, hard, interval) in `spec.hpp`.
|
|
|
|
|
|
|
|
|
|
Unmeasured: cost of scanning a large upper dir (needs a run on a populated container); overshoot between two scans.
|
|
|
|
|
|
|
|
|
|
### G3: more than one uid inside
|
|
|
|
|
|
|
|
|
|
Findings, measured with `podman unshare` (map `0 -> 1000`, `1..65536 -> 100000..`): overlayfs with `userxattr` mounts fine under a multi-id map. `/etc/subuid` and `/etc/subgid` contain `retoor:100000:65536`, `newuidmap` and `newgidmap` are installed.
|
|
|
|
|
- Host root-owned files in the lower dirs show as `65534 nobody`. That is already true with the single map, so nothing gets worse.
|
|
|
|
|
- Gain: `chown`, `su`, `apt` sandbox users, daemons that drop privileges.
|
|
|
|
|
- Cost, important: files created by container uids 1..65535 are owned by host uids 100000+. The host user cannot delete them. `Store::remove` as written would fail with `Permission denied`. Delete must run inside a user namespace with the same map (re-enter and `rm` there, or `podman unshare`-style helper) before removing the directory.
|
|
|
|
|
|
|
|
|
|
Design: after `clone`, parent execs `newuidmap <pid> 0 <uid> 1 1 <sub> 65536` and the gid equivalent, `setgroups` allowed. Fallback to the current single map when the user has no subuid range, with a printed notice. Store the chosen mode in `meta` (`idmap=single|subuid`) so delete knows which path to take.
|
|
|
|
|
|
|
|
|
|
Unmeasured: `apt install` end to end.
|
|
|
|
|
|
|
|
|
|
### G4: host `/etc` visible inside
|
|
|
|
|
|
|
|
|
|
Finding, measured: `find /etc -type f -perm -004` returns 1116 world-readable files on this host, including names such as `/etc/fail2ban/action.d/cloudflare-token.conf` (content not read). Anything world-readable in host `/etc` is readable in every container.
|
|
|
|
|
|
|
|
|
|
Design: replace the `/etc` overlay by a private `/etc`. At create time, populate `private/etc` from an allowlist in `spec.hpp` (`kEtcAllow`): `passwd group nsswitch.conf ld.so.cache ld.so.conf ld.so.conf.d ssl/certs ca-certificates alternatives bash.bashrc profile profile.d localtime timezone os-release protocols services hosts hostname`, plus a generated `resolv.conf`. Two ways: copy (stable, can go stale, for example CA certs) or read-only bind of each allowed path (always current, more mounts). Recommendation: read-only bind, since doos already carries the mount machinery.
|
|
|
|
|
|
|
|
|
|
Unmeasured: whether this allowlist is enough for `apt`, `curl`, `python3`, `git`; needs a run per tool before fixing the list.
|
|
|
|
|
|
|
|
|
|
### G5: `--net host` and abstract unix sockets
|
|
|
|
|
|
|
|
|
|
Fact: abstract `AF_UNIX` sockets belong to the network namespace, so `--net host` exposes host abstract sockets (some X11 and D-Bus setups use them) and host loopback services. This is inherent to sharing the namespace.
|
|
|
|
|
|
|
|
|
|
Researched alternative, pasta (installed, `pasta 0.0~git20260120`): it attaches to an existing netns by pid (`pasta --config-net <pid>`), gives outbound access, and mirrors host addresses into the namespace. Measured: outbound TCP worked, and an inside connection to `127.0.0.1:22` reached the host (default flags, once). A second run with `--no-splice --no-map-gw --map-host-loopback none` printed `Couldn't get any nameserver address`, and I could not confirm the loopback result from it. Conclusion: pasta is not proven to match slirp4netns' `--disable-host-loopback` guarantee. Slirp stays the default. Next step is a clean test matrix of pasta flags with a listener on a known host port.
|
|
|
|
|
|
|
|
|
|
Interim: `host` remains an explicit opt-in documented as unisolated.
|
|
|
|
|
|
|
|
|
|
### G6: kernel attack surface (seccomp, nested user namespaces)
|
|
|
|
|
|
|
|
|
|
Findings:
|
|
|
|
|
- A seccomp-bpf filter installs unprivileged once `no_new_privs` is set (already done). Measured with a raw `prctl(PR_SET_SECCOMP)` program: a denied syscall (`keyctl`) returned `-1 EPERM`. No `libseccomp` headers on this machine, and "fewer dependencies" argues for hand-assembled BPF.
|
|
|
|
|
- `/proc/sys` is read-only in the container, so `user.max_user_namespaces=0` cannot be set from inside; seccomp is the lever.
|
|
|
|
|
|
|
|
|
|
Design: denylist modelled on the Docker default profile, applied after `setupRoot` and before `exec`, and also in the `join()` path. Check `AUDIT_ARCH` first and kill on a foreign ABI (blocks the x32 bypass). Deny with `EPERM`: `kexec_load kexec_file_load bpf perf_event_open userfaultfd keyctl add_key request_key open_by_handle_at process_vm_readv process_vm_writev reboot swapon swapoff init_module finit_module delete_module`. Nested userns: deny `unshare` and `setns` with `CLONE_NEWUSER`, deny `clone` with `CLONE_NEWUSER`, and return `ENOSYS` for `clone3` (flags live in a struct, cannot be inspected; libc falls back to `clone`). Trade-off: breaks nested containers and Chromium-style sandboxes; make it a documented constant, not a switch that hides in code.
|
|
|
|
|
|
|
|
|
|
Unmeasured: compatibility impact on real workloads.
|
|
|
|
|
|
|
|
|
|
### Priority
|
|
|
|
|
|
|
|
|
|
1. G1 (limits): cheapest, measured to work, unlocks G2's freeze.
|
|
|
|
|
2. G6 (seccomp): self-contained, no new dependency.
|
|
|
|
|
3. G4 (private `/etc`): needs the per-tool allowlist runs first.
|
|
|
|
|
4. G2, then G3 (G3 changes the delete path, do it deliberately).
|
|
|
|
|
5. G5: needs a test matrix before any change.
|
|
|
|
|
|
|
|
|
|
## Field trial: apt and common tools (v0.1.2, measured)
|
|
|
|
|
|
|
|
|
|
Run in a fresh `--net slirp` container on the host above (Ubuntu with apt 3.2.0, dpkg 1.23.7).
|
|
|
|
|
|
|
|
|
|
| Step | Result |
|
|
|
|
|
|---|---|
|
|
|
|
|
| curl https, git ls-remote, gcc compile+run, python3 (`ssl`, `sqlite3`), pip, ping 1.1.1.1 | all work |
|
|
|
|
|
| `apt-get update` as shipped | fails: `/var/lib/apt/lists/partial is missing` (`/var` is an empty private dir) |
|
|
|
|
|
| `apt-get install` as shipped | fails: `/var/lib/dpkg/lock-frontend` missing, no dpkg status database |
|
|
|
|
|
| after seeding `/var` (dirs plus copy of host `dpkg/status`, `info`, `alternatives`, 40 MB) | apt starts, but the download method dies: `setegid 65534 failed`, `seteuid 42 failed` (sandbox user `_apt` cannot exist with a single id) |
|
|
|
|
|
| with `-o APT::Sandbox::User=root` | `apt-get update` works: 58.9 MB fetched in 3 s |
|
|
|
|
|
| `apt-get install cowsay` | download works; unpack fails: `unable to create '/usr/games/cowsay.dpkg-new': Permission denied` |
|
|
|
|
|
| after pre-creating the host directory tree in the upper dirs (13164 dirs for `/usr`, done with a shell loop, 38 s; not yet implemented in C++) | unpack works, `/usr/games/cowsay hoi` runs; host `/usr/games/cowsay` still absent (isolation holds) |
|
|
|
|
|
| configure step | fails: `chown: changing ownership of '/etc/shells.tmp': Invalid argument` (debianutils postinst copies owner of `/etc/shells`, which shows as `65534`), so dpkg reports 4 packages not fully installed |
|
|
|
|
|
|
|
|
|
|
Root causes:
|
|
|
|
|
1. Host files owned by host root appear as `nobody` (uid 65534). Container root has `CAP_DAC_OVERRIDE` only over inodes whose owner is mapped, so it cannot write into host-root-owned directories of the lower layer, and cannot `chown` to or from an unmapped owner.
|
|
|
|
|
2. `/var` has no package database.
|
|
|
|
|
3. apt wants a second uid (G3).
|
|
|
|
|
|
|
|
|
|
Conclusion: the current design (host `/usr`, `/etc`, `/opt` as lower layer) is good for running tools and for isolated experiments, not for package management. Bare tools and compilers work. Honest status: **`apt-get install` does not complete today.**
|
|
|
|
|
|
|
|
|
|
Researched ways forward, in order of effort:
|
|
|
|
|
- Short term, writable skeleton: at create time mirror every directory of the lower dirs into the upper dir with the same mode (directories owned by the user are writable; merged view is unchanged). Directories unreadable to the user (for example `/etc/ssl/private`, mode 700, root-owned) are skipped and stay as they are. Measured effect: writes into `/usr/bin` work, attach time unchanged (41 ms). Unmeasured: C++ implementation speed, and it does not fix the `chown` failure.
|
|
|
|
|
- Proper fix, image layer: use an extracted distro rootfs (for example from `podman export` or `skopeo copy` plus unpack, both installed here; `mmdebstrap` and `debootstrap` are not) as the lower dir, with ownership shifted into the subuid range (G3). Then the tree is owned by mapped ids, `/var/lib/dpkg` exists, `chown` works, `_apt` exists, and host `/etc` stops leaking (G4). This also removes the dependence on the host distro. Cost: image import step, disk per image (shared lower, so once), and a layout change, hence a major bump.
|
|
|
|
|
- Alternative to evaluate: `fuse-overlayfs` (installed) supports uid mappings between layers. Unmeasured; adds a FUSE process per container.
|
|
|
|
|
retoorandClaude Sonnet 5.5