doos/docs/ARCHITECTURE.md

# Architecture

How doos works internally. Behavior contract is `SPEC.md`; this explains the code.

## File map

| File | Class / role |
|---|---|
| `src/spec.hpp` | `spec::` constants: caps, mount lists, device list, paths. Single source of truth with `SPEC.md`. |
| `src/util.{hpp,cpp}` | `SysError`, `Fd` (RAII fd), `FileLock` (bounded flock), `parseSize`/`formatSize`, `base64Decode`, `runProcess` (fork/exec, capped capture), `fsutil` (mkdirs, read/write, bounded `removeTree`, `mirrorDirs` skeleton, `copyTree`/`copyEntries` (with skip predicate), `copyFiles`, `diskUsage` with hardlink dedupe). |
| `src/idmap.{hpp,cpp}` | `IdMap` (single or subuid mapping via newuidmap/newgidmap, subuid range lookup), `UserNsJob` (run a function as root of a user namespace with the same map: image extraction, deleting subuid-owned trees). |
| `src/image.{hpp,cpp}` | `Catalog` (alpine, ubuntu, debian resolution from publishers), `verifySignature` (gpg with a throwaway home, pinned fingerprint), `ImageStore` (add with download, sha256 and signature check, extraction; list; delete). |
| `src/cgroup.{hpp,cpp}` | `Limits`, `Cgroup`: `doos/<name>/{main,shell}` under the user's delegated cgroup; limits on the parent, freeze on `main`, later attaches in `shell`, OOM and pid-limit counters. |
| `src/hardening.{hpp,cpp}` | `Hardening::apply`: capability bounding set to the Docker default, seccomp-bpf denylist built at runtime (arch check, x32 guard, clone3 ENOSYS, CLONE_NEWUSER filter). |
| `src/network.{hpp,cpp}` | `NetMode` (slirp, pasta, host, none) + `Network` (parse, resolv.conf per mode), `SlirpSession`, `PastaSession` (own the helper processes). |
| `src/container.{hpp,cpp}` | `Options` (what the user asked for), `Container` (paths, meta, pid file, layout, one-time seeding, usage), `Store` (list, get_or_create with on-demand image fetch, remove). |
| `src/sandbox.{hpp,cpp}` | `Sandbox`: start a container (clone + id map + mounts + pivot_root + init), watch it (size cap), or join a running one (setns). |
| `src/cli.{hpp,cpp}` | `Cli`: argument parsing, the four container commands and `image list/add/delete`. |
| `src/main.cpp` | Catches exceptions, prints `doos: <message>`, exit 1. |
| `tests/run.sh` | End-to-end suite. |
| `scripts/release.sh` | Bump, build, test, commit, tag. |
| `apparmor/doos` | Profile granting `userns` on Ubuntu 24.04+. |

## attach flow

```
Cli::cmdAttach
  Store::getOrCreate           lock, create layout + meta if missing
  Sandbox::run
    FileLock (start-or-join decision)
    runningPid()?  yes -> join()   no -> start()

start():
  clone(NEWUSER|NEWNS|NEWPID|NEWUTS|NEWIPC[|NEWNET])      child = container pid 1
  IdMap::apply(child, ids)        single: write /proc maps; subuid: newuidmap + newgidmap
  slirp mode: SlirpSession.start(child)                    waits for ready-fd
  slirp/pasta session for the netns of child
  Cgroup::create(limits) + addMain(child)   best effort; without it nothing is enforced and doos says so
  recordPid(child)  (pid + starttime, defeats pid reuse)
  lock.unlock()     joiners must not wait for container lifetime
  write 'G' on sync pipe                                    child continues
  parent: ignore SIGINT/SIGQUIT, forward SIGTERM/SIGHUP, watch(child)
  watch: poll(pidfd, until next scan); scan = usage from inside the id map (UserNsJob) -> warn 80%, freeze 100%,
         thaw <95%; next scan after max(2 s, 10 x scan time)
  after exit: report OOM kills / pid-limit hits
  on exit: pid file cleared, slirp stopped, cgroup removed (RAII)

child (containerInit, pid 1):
  PDEATHSIG=SIGKILL, close inherited lock fd, wait for 'G'
  setupRoot -> sethostname -> loopback up (isolated netns) -> rlimits/env/no_new_privs
  fork worker -> applyLimitsAndEnv (RLIMIT_FSIZE = size cap, Hardening::apply unless relaxed) -> exec command
  pid 1 loop: forward signals, reap all, on worker exit kill(-1, SIGKILL), exit with its code
```

## setupRoot order

Common start: `MS_REC|MS_PRIVATE` on `/`, then `resolv.conf` is written into the container's `/etc` location
(`Container::etcDir`: `private/etc` in host mode, the root upper layer in image mode), and on a single-id
container with apt present an `APT::Sandbox::User "root"` override is written next to it.

Host mode:
1. Bind the container `rootfs/` onto itself (pivot_root needs a mountpoint).
2. Overlay mounts for `/usr`, `/opt` and real compat dirs (`lowerdir=<host dir>,upperdir,workdir,userxattr`).
3. Compat symlinks recreated (usr-merge).
4. Bind the private dirs `/root`, `/var`, `/etc`; then the overlay for `/var/lib/dpkg` inside the private `/var`.

Image mode:
1. One overlay on `rootfs/`: lower = `images/NAME/rootfs`, upper = `layers/root/upper`.
2. Mountpoints (`/proc /dev /tmp /run ...`) created inside the merged tree.

Both modes then: tmpfs `/tmp` and `/run`; `/dev` (tmpfs, bound host nodes, new devpts, `/dev/shm`, symlinks);
`/proc` with read-only remounts and masks from `spec.hpp`; `chdir(rootfs)`, `pivot_root(".", ".")`, lazy
unmount of the old root.

## Seeding (host mode, once per container)

`Container::prepareLayout` runs on create and as a heal on every later load; each step leaves a
`seeded.<name>` marker so it runs once and prints what it did and what it skipped:
1. Directory skeleton of every overlaid host dir into its upper layer (`fsutil::mirrorDirs`).
2. Regular files of `/var/lib/dpkg`, `triggers`, `alternatives` copied into the upper (`copyFiles`), then empty lock files.
3. `/etc` copied into `private/etc`: the allowlist (`copyEntries` with the credential deny predicate) under default hardening, everything readable (`copyTree`) under relaxed.

Reasons are in `SPEC.md` (host root-owned files appear as `nobody`).

## Images

`ImageStore::add`: lock-free staging dir `images/.tmp-NAME`, download with curl (https only, size and time caps),
sha256 check, `tar -x -p --numeric-owner` executed by `UserNsJob` with the image's id mode (`--no-same-owner`
for single), usage measured, `meta` written, atomic `rename` into place. Any failure removes the staging dir
from inside the id map, so a failed add leaves nothing. `Store::getOrCreate` calls it when a catalog image is
missing, and pins the container's id mode to the image's.

## join flow

Open `/proc/<pid>/ns/{user,mnt,uts,ipc,net,pid}` and `/proc/<pid>/root` first, fork, move the child into the
container's `shell` cgroup (same limits, never frozen), `setns` user first (grants capability for the rest), `fchdir`+`chroot` to the container
root, fork again (pid ns applies to children), apply env and limits, exec.
In `host` network mode the net namespace is not joined.

## Lifetime model

The namespaces live as long as the first `attach` command. When it exits, pid 1 kills
the rest and the container is `stopped`. State (upper dirs, `/root`, `/var`) persists on
disk until `delete --yes`. There is no daemon.

## Error handling

All failures throw (`SysError` or `std::runtime_error`); `main` prints one line.
Inside the child, errors print `doos: <message>` and exit 125. A parent-side failure
after `clone` kills the child (`Reaper`), so no half-built container survives.