doos/docs/ARCHITECTURE.md

# Architecture

**Docs:** [README](../README.md) · [SPEC](../SPEC.md) · [ARCHITECTURE](ARCHITECTURE.md) · [RESEARCH](RESEARCH.md) · [COMPARISON](../COMPARISON.md) · [AGENTS](../AGENTS.md) · [CHANGELOG](../CHANGELOG.md) · [CI](CI.md)

How doos works internally. The behavior contract is [SPEC](../SPEC.md); this explains the code. Why it was built this way and what was measured: [RESEARCH](RESEARCH.md); how it compares with other tools: [COMPARISON](../COMPARISON.md).

## File map

| File | Class / role |
|---|---|
| `src/spec.hpp` | `spec::` constants: caps, mount lists, device list, paths. Single source of truth with `SPEC.md`. |
| `src/util.{hpp,cpp}` | `SysError`, `Fd` (RAII fd), `FileLock` (bounded flock), `parseSize`/`formatSize`, `base64Decode`, `runProcess` (fork/exec, capped capture), `fetchText`/`fetchFile` (curl, https only, capped), `TempDir` (scratch dir under the state dir, removed on scope exit), `isHex64`, `verifyDetachedSignature`/`verifyClearsignedSignature` (gpg with a throwaway home, pinned fingerprint; shared by `image.cpp` and `davmount.cpp`), `fsutil` (mkdirs, read/write, bounded `removeTree`, `mirrorDirs` skeleton, `copyTree`/`copyEntries` (with skip predicate), `copyFiles`, `diskUsage` with hardlink dedupe). |
| `src/idmap.{hpp,cpp}` | `IdMap` (single or subuid mapping via newuidmap/newgidmap, subuid range lookup), `UserNsJob` (run a function as root of a user namespace with the same map: image extraction, deleting subuid-owned trees). |
| `src/image.{hpp,cpp}` | `Catalog` (alpine, ubuntu, debian resolution from publishers), `ImageStore` (add with download, sha256 and signature check via `util::verifyDetachedSignature`, extraction; list; delete). |
| `src/davmount.{hpp,cpp}` | `DavMount`: downloads/verifies/installs a pinned `rclone` under the state dir, resolves a `--mount`/`--davmount` WebDAV entry to a ready, reused, managed FUSE mount (on-the-fly connection string, double-fork daemonize, mountinfo + active-listing readiness, failure classification), `list`/`umount` for `doos davmount`. |
| `src/cgroup.{hpp,cpp}` | `Limits`, `Cgroup`: `doos/<name>/{main,shell}` under the user's delegated cgroup; limits on the parent, freeze on `main`, later attaches in `shell`, OOM and pid-limit counters. |
| `src/hardening.{hpp,cpp}` | `Hardening::apply`: capability bounding set to the Docker default, seccomp-bpf denylist built at runtime (arch check, x32 guard, clone3 ENOSYS, CLONE_NEWUSER filter). |
| `src/network.{hpp,cpp}` | `NetMode` (slirp, pasta, host, none) + `Network` (parse, resolv.conf per mode), `SlirpSession`, `PastaSession` (own the helper processes). |
| `src/container.{hpp,cpp}` | `Options` (what the user asked for), `Container` (paths, meta, pid file, layout, one-time seeding, usage), `Store` (list, get_or_create with on-demand image fetch, remove). |
| `src/sandbox.{hpp,cpp}` | `Sandbox`: resolves davmounts (`DavMount::resolveAll`), starts a container (clone + id map + mounts + pivot_root + init), watches it (size cap), or joins a running one (setns). |
| `src/cli.{hpp,cpp}` | `Cli`: argument parsing, the four container commands, `image list/add/delete` and `davmount list/umount`. |
| `src/main.cpp` | Catches exceptions, prints `doos: <message>`, exit 1. |
| `tests/run.sh` | End-to-end suite. |
| `scripts/release.sh` | Bump, build, test, commit, tag. |
| `apparmor/doos` | Profile granting `userns` on Ubuntu 24.04+. |

## attach flow

```
Cli::cmdAttach
  Store::getOrCreate           lock, create layout + meta if missing
  Sandbox::run
    FileLock (start-or-join decision)
    runningPid()?  yes -> join()   no -> start()

start():
  DavMount::resolveAll(mounts)    before clone: webdav entries become ready, reused Bind entries
  clone(NEWUSER|NEWNS|NEWPID|NEWUTS|NEWIPC[|NEWNET])      child = container pid 1
  IdMap::apply(child, ids)        single: write /proc maps; subuid: newuidmap + newgidmap
  slirp mode: SlirpSession.start(child)                    waits for ready-fd
  slirp/pasta session for the netns of child
  Cgroup::create(limits) + addMain(child)   best effort; without it nothing is enforced and doos says so
  recordPid(child)  (pid + starttime, defeats pid reuse)
  lock.unlock()     joiners must not wait for container lifetime
  write 'G' on sync pipe                                    child continues
  parent: ignore SIGINT/SIGQUIT, forward SIGTERM/SIGHUP, watch(child)
  watch: poll(pidfd, until next scan); scan = usage from inside the id map (UserNsJob) -> warn 80%, freeze 100%,
         thaw <95%; next scan after max(2 s, 10 x scan time)
  after exit: report OOM kills / pid-limit hits
  on exit: pid file cleared, slirp stopped, cgroup removed (RAII)

child (containerInit, pid 1):
  PDEATHSIG=SIGKILL, close inherited lock fd, wait for 'G'
  setupRoot -> sethostname -> loopback up (isolated netns) -> rlimits/env/no_new_privs
  fork worker -> applyLimitsAndEnv (RLIMIT_FSIZE = size cap, Hardening::apply unless relaxed) -> exec command
  pid 1 loop: forward signals, reap all, on worker exit kill(-1, SIGKILL), exit with its code
```

## setupRoot order

Common start: `MS_REC|MS_PRIVATE` on `/`, then `resolv.conf` is written into the container's `/etc` location
(`Container::etcDir`: `private/etc` in host mode, the root upper layer in image mode), and on a single-id
container with apt present an `APT::Sandbox::User "root"` override is written next to it.

Host mode:
1. Bind the container `rootfs/` onto itself (pivot_root needs a mountpoint).
2. Overlay mounts for `/usr`, `/opt` and real compat dirs (`lowerdir=<host dir>,upperdir,workdir,userxattr`).
3. Compat symlinks recreated (usr-merge).
4. Bind the private dirs `/root`, `/var`, `/etc`; then the overlay for `/var/lib/dpkg` inside the private `/var`.

Image mode:
1. One overlay on `rootfs/`: lower = `images/NAME/rootfs`, upper = `layers/root/upper`.
2. Mountpoints (`/proc /dev /tmp /run ...`) created inside the merged tree.

Both modes then: tmpfs `/tmp` and `/run`; `/dev` (tmpfs, bound host nodes, new devpts, `/dev/shm`, symlinks);
`/proc` with read-only remounts and masks from `spec.hpp`; `mountExtra` binds each `Container::mounts()`
entry (`--mount SRC:DST[:ro]`) at `DST`, creating the mountpoint first (`fsutil::mkdirs`), read-only ones
remounted the same way `/proc`'s read-only paths are; `chdir(rootfs)`, `pivot_root(".", ".")`, lazy
unmount of the old root.

## Seeding (host mode, once per container)

`Container::prepareLayout` runs on create and as a heal on every later load; each step leaves a
`seeded.<name>` marker so it runs once and prints what it did and what it skipped:
1. Directory skeleton of every overlaid host dir into its upper layer (`fsutil::mirrorDirs`).
2. Regular files of `/var/lib/dpkg`, `triggers`, `alternatives` copied into the upper (`copyFiles`), then empty lock files.
3. `/etc` copied into `private/etc`: the allowlist (`copyEntries` with the credential deny predicate) under default hardening, everything readable (`copyTree`) under relaxed.

Reasons are in `SPEC.md` (host root-owned files appear as `nobody`).

## davmount flow

```
DavMount::resolveAll(mounts)                               (Sandbox::start, before clone)
  for each WebDav entry:
    id = fnv1a(url, user, pass, vendor, insecure)          dst excluded: shared across containers
    dir = $DOOS_HOME/davmounts/<id>/, mkdirs mnt/, FileLock dir/lock
    already in mountinfo (fuse.*) and pidfile's pid alive and an active listing succeeds?
        -> reuse mnt/ as is (sub-second)
        -> else unmountQuiet (fusermount3 -u, then -uz if still mounted) and fall through
    ensureRclone(): $DOOS_HOME/tools/rclone-VERSION/rclone; `rclone version` match -> reuse,
        else download+verify (util::fetchFile, util::verifyClearsignedSignature against the
        pinned key, sha256 against the signed SHA256SUMS), unzip, atomic rename into place
    rclone obscure <pass> (only if a password was given)
    connection string ":webdav,url=\"...\",vendor=\"...\"[,user=...][,pass=...][,no_check_certificate=true]:"
    spawnRcloneMount: pipe, fork, fork again, setsid + redirect to dir/log, execve(rclone, mount, ...)
        first fork writes the second fork's pid down the pipe, then exits (reaped immediately);
        the real daemon is now a child of init, outliving this doos process entirely
    poll up to kDavMountReadyTimeoutMs: daemon pid alive? mountinfo shows it AND a real directory
        listing succeeds (stat() alone answers without touching the backend; only listing does)?
        -> ready, write dir/pid + dir/meta (never the password)
        -> daemon died or never answers in time: unmountQuiet, classify dir/log's tail
           (401/Unauthorized, x509/certificate, 501/Unsupported method, dial/host/refused/timeout,
           else "see the log"), throw
  Bind entries pass through unchanged; every resolved entry becomes a plain Bind MountSpec
  pointing at dir/mnt, so mountExtra() (sandbox.cpp) never has to know davmount exists.
```

`doos davmount list`/`umount` walk/remove `$DOOS_HOME/davmounts/*` directly; `umount` is dry-run unless
`--yes`, same pattern as `delete`/`image delete`.

## Images

`ImageStore::add`: lock-free staging dir `images/.tmp-NAME`, download with curl (https only, size and time caps),
sha256 check, `tar -x -p --numeric-owner` executed by `UserNsJob` with the image's id mode (`--no-same-owner`
for single), usage measured, `meta` written, atomic `rename` into place. Any failure removes the staging dir
from inside the id map, so a failed add leaves nothing. `Store::getOrCreate` calls it when a catalog image is
missing, and pins the container's id mode to the image's.

## join flow

Open `/proc/<pid>/ns/{user,mnt,uts,ipc,net,pid}` and `/proc/<pid>/root` first, fork, move the child into the
container's `shell` cgroup (same limits, never frozen), `setns` user first (grants capability for the rest), `fchdir`+`chroot` to the container
root, fork again (pid ns applies to children), apply env and limits, exec.
In `host` network mode the net namespace is not joined.

## Lifetime model

The namespaces live as long as the first `attach` command. When it exits, pid 1 kills
the rest and the container is `stopped`. State (upper dirs, `/root`, `/var`) persists on
disk until `delete --yes`. There is no daemon.

## Error handling

All failures throw (`SysError` or `std::runtime_error`); `main` prints one line.
Inside the child, errors print `doos: <message>` and exit 125. A parent-side failure
after `clone` kills the child (`Reaper`), so no half-built container survives.