- New rootless doos container runtime: sandbox, cgroup, network, images, hardening, davmount, CLI plus molo integration - 18 CVE audit reproductions with READMEs and Makefiles - Example DoosDemo routes, IndexCatalog, static assets, spec update - CLI serve on 0.0.0.0:8000, arg parser, dev server improvements - Runtime HTTP fixes, supervisor isolation backend - Tests for doos, example doos routes, CLI serve; suite count 261 to 265
22 KiB
doos specification
Docs: README · SPEC · ARCHITECTURE · RESEARCH · COMPARISON · PROVENANCE · CHANGELOG · CI
Authoritative description of behavior and limits. Code constants live in src/doos/caps.h (class Caps, values identical to doos 1.6.0); change both together. How the code implements it: ARCHITECTURE. Measurements and remaining limits: RESEARCH. How it differs from other tools: COMPARISON. Usage: README. The MoloApp move: PROVENANCE.
CLI
| Command | Behavior |
|---|---|
doos list |
Table of NAME, STATE (running/stopped), NET, IMAGE (host when none), SIZE (cap), AGE. |
doos get_or_create [OPTIONS] NAME |
Creates the container if missing. Prints created NAME or exists NAME. |
doos attach [OPTIONS] NAME [-- CMD...] |
get_or_create, then run CMD (default /bin/bash, else /bin/sh). Starts the container if stopped, joins it if running. Exit code is the command's. |
doos delete [--yes] NAME |
Dry run unless --yes. Refuses a running container. |
doos image list |
Installed images (SIZE, IDS) plus catalog entries marked available. |
doos image add [--sha256 HEX] SOURCE [NAME] |
Installs an image. See Images. |
doos apparmor |
Prints an AppArmor profile (named after the binary's path) that lets this binary use user namespaces. A start that fails with EPERM/EACCES while apparmor_restrict_unprivileged_userns=1 prints the command to install it. |
doos image delete [--yes] NAME |
Dry run unless --yes. Refuses an image used by any container, even with --yes. |
doos davmount list |
Table of davmount targets this host currently manages: ID, STATE (mounted/stale), VENDOR, URL. |
doos davmount umount [--yes] ID |
Dry run unless --yes. Unmounts and removes that managed target; stops storage for any container still using it. |
get-or-create is accepted as an alias.
molo doos (thin front end)
molo doos ... forwards to the same libdoos the standalone binary uses. Text
output is byte-identical to doos ...; exit codes stay doos's (0/1/2, never
69). Forwarding drops argv[0], the molo-global flags (--format plus value,
-v/--verbose, -q/--quiet) and the resolved command path, and keeps
everything else verbatim: flag order, repeats (repeatable --mount), and the
-- CMD tail. Consequence: molo globals keep their molo meaning under
molo doos (a doos command literally containing -v is expressed through
the standalone binary). molo doos --help and molo help doos ... render
from the one molo command tree.
--format json renders the same data as one JSON object per command (stdout
is always parseable, including on failure, where the body is
{"ok":false,"error":"..."}):
| Command | Success body |
|---|---|
list |
{"ok":true,"containers":[{"name","state","net","image","size_bytes","created"}]} |
get_or_create |
{"ok":true,"name","created"} |
delete, image delete, davmount umount |
{"ok":true,"name"/"id","dry_run","report"} |
image list |
{"ok":true,"images":[{"name","bytes","ids","state"}]} (bytes/ids null when available) |
image add |
{"ok":true,"name","bytes","ids"} |
davmount list |
{"ok":true,"davmounts":[{"id","url","vendor","mountpoint","mounted"}]} |
apparmor |
{"ok":true,"profile"} |
version |
{"ok":true,"version"} |
attach is interactive and has no JSON form (exit 2 with a JSON error).
Unknown trailing input is rejected (strict); exit codes mirror the text CLI.
Container options (accepted by get_or_create and attach):
| Option | Meaning | Default | Changeable later |
|---|---|---|---|
--image NAME |
Start from an image instead of the host system. Missing catalog images are fetched on demand. | none (host mode) | no |
--net slirp|pasta|host|none |
Network mode, see Networking. | slirp | no |
--ids single|subuid |
Id mapping. | subuid when available, else single | no |
--hardening default|relaxed |
relaxed = no capability drop, no seccomp filter, full readable host /etc (host mode). relaxed and --net host also require --i-understand on every get_or_create and attach call, and print a warning once given. |
default | no |
--i-understand |
Confirmation flag (no value) for the reduced sandboxing described above. Without it, --hardening relaxed or --net host is an error. |
absent | n/a |
--size SIZE |
Cap for what the container writes (512m, 8g, 1t; k/m/g/t). |
4g | yes |
--memory SIZE |
Memory cap (cgroup memory.max, swap 0, whole-container OOM kill). |
8g | yes |
--pids N |
Process and thread cap (pids.max). |
4096 | yes |
--cpus N |
CPU cap, decimals allowed (0.5, 2); 0 = all cores. |
0 | yes |
--mount SRC:DST[:ro] |
Bind-mount host directory SRC at DST inside the container, repeatable (max 8). Not subject to --size: it is host storage, not the container's writable layer. Replaces the whole mount list when given. |
none | yes |
--mount URL DST[:ro] / --davmount URL DST[:ro] |
URL starting dav://, davs://, http:// or https://: davmount, see below. Same list and cap as --mount. |
none | yes |
--dav-user USER, --dav-pass PASS |
Credentials for the davmount entry just given (overrides any user[:pass]@ in the URL). |
none | yes |
--dav-vendor NAME |
other, nextcloud, owncloud, sharepoint or sharepoint-ntlm for that entry. |
other | yes |
--dav-insecure |
Skip TLS certificate verification for that entry. | off | yes |
Changeable options apply at the next start of the container. Naming a fixed option on an existing container with a different value is an error, never a silent change. A plain --mount's SRC must exist and be a directory at the time the container starts; a missing SRC fails the start loudly. A davmount's URL is resolved and mounted automatically, see below.
davmount
--mount/--davmount with a dav://, davs://, http:// or https:// source turns into a managed WebDAV mount instead of a plain bind: doos downloads, verifies and installs a pinned rclone build under $DOOS_HOME/tools/rclone-VERSION/rclone (never the system, never the caller's PATH; same trust model as the catalog images: a fixed version, fetched over https, checked against a GPG-signed SHA256SUMS and a key pinned by fingerprint), then drives it to mount the target on the host, before the container's namespaces are created, in Sandbox::start.
- Identity and reuse: a target is identified by URL + user + pass + vendor + the insecure flag (not by destination), hashed into a 16-hex-char id under
$DOOS_HOME/davmounts/<id>/{mnt,log,pid,meta,lock}. The same target reused by another--mount/--davmountentry, in the same or a different container, is mounted once and reused; a dead or unreachable mount is detected (process liveness plus an active directory listing, since a FUSE mount answersstaton its root without ever reaching the backend) and remounted automatically. - On-the-fly, no config file: rclone's connection-string syntax (
:webdav,url=...,vendor=...,user=...,pass=...:), every field quoted; the password is passed throughrclone obscurefirst (rclone's own documented requirement), never in plain text to the process, and never on its command line in a formpswould show in full (quoted into one argv element doos builds itself). - Lifecycle: the mount runs as a detached, double-forked process (
setsid, reparented to init) so it outlives thedoos attachthat created it; writes go through rclone's VFS write-back cache (--vfs-cache-mode=writes, required for WebDAV, which cannot stream;--vfs-write-back=1s, shortened from rclone's 5 s default) and reach the server shortly after a file is closed, independent of whether the container that wrote it has already exited. - Readiness and failure: a fresh mount is given up to 20 s to answer a real directory listing (not just
stat, which a FUSE root answers without a backend round trip); failure is classified from the log into one of: authentication, unreachable server, TLS certificate, not a WebDAV endpoint, or a generic fallback naming the log file. A failed attempt is unmounted but its log and directory are left fordoos davmount list/umountto inspect or clear. - Not covered: doos never implements WebDAV itself, and does not audit rclone's own implementation (see
audit/CVE-2020-7039for the equivalent libslirp caveat on the networking side); a self-signed or misconfigured TLS endpoint needs--dav-insecure(an explicit, logged choice, not a default). - Why rclone stays (decided): the in-house
src/realtime/webdavmodule is a protocol client (request/response objects, no transport, no FUSE, no daemon, no NTLM); doos needs a kernel-visible FUSE filesystem with write-back caching and vendor quirks that a directory bind-mount can consume. Replacing rclone would mean writing that filesystem from scratch whilefusermountremains an external helper either way, so the pinned, verified, managed rclone stays. The two modules compose (a doos mount can consume a share the framework serves) but never merge.
Limits (hard caps)
| Item | Cap |
|---|---|
| Containers | 64 |
| Images | 16 |
| Name | containers [a-z0-9][a-z0-9_-]{0,31}, images [a-z0-9][a-z0-9._-]{0,31} (: becomes -) |
| Command arguments | 256 |
| Open files inside | 4096 |
/tmp, /run, /dev/shm |
256m, 64m, 64m (tmpfs) |
--size |
64m to 1t |
| Size scan interval | 2 s floor, and at least 10x the duration of the last scan |
--memory |
64m to 1t |
--pids |
16 to 4,194,304 |
--cpus |
0.1 to 1024 |
| Single file size inside | the --size cap (RLIMIT_FSIZE) |
| Size scan entries | 5,000,000 |
| Image download | 2 GiB, 900 s |
| Skeleton directories | 200,000 |
/etc copy |
50,000 files, 256 MiB |
| Id range per container | 65536 ids (0 plus up to 65535 from subuid) |
| slirp4netns ready wait | 5 s |
| Lock wait | 5 s |
| Tree recursion depth | 256 |
--mount/--davmount entries |
8 per container |
| davmount targets managed at once | 32 |
| davmount ready wait | 20 s |
| rclone download | 128 MiB |
State layout
$DOOS_HOME, else $XDG_DATA_HOME/doos, else ~/.local/share/doos. Path must not contain , or :.
containers/NAME/{meta,pid,lock,seeded.*,rootfs/,layers/<dir>/{upper,work},private/{root,var,etc}}
images/NAME/{meta,rootfs/}
tools/rclone-VERSION/rclone
davmounts/ID/{mnt/,log,pid,meta,lock}
Container meta keys: net created version image ids size memory pids cpus hardening, then mounts.count and,
per entry, mounts.N.kind mounts.N.src mounts.N.dst mounts.N.ro (bind), plus mounts.N.user mounts.N.pass mounts.N.vendor mounts.N.insecure (webdav; pass is plain text, see Safety rules). A container created under
1.5.0's single-line mounts= format is still read, bind entries only.
Container filesystem, host mode (no --image)
| Path | Source |
|---|---|
/usr /opt (and /bin /sbin /lib /lib32 /lib64 when real dirs) |
overlayfs, lower = host dir, upper = layers/<dir>/upper, with a directory skeleton in the upper |
/bin /sbin /lib... when host symlinks |
same symlink recreated |
/etc |
private copy, made once at creation: under --hardening default only the allowlist in Caps::kEtcAllow (passwd, group, nsswitch, ld.so*, ssl, ca-certificates, alternatives, apt, dpkg, shell and tool defaults...) minus names that may hold credentials (private, shadow, gshadow, sudoers*, auth.conf*, ssh_host_*, *.key); under relaxed every readable file |
/root, /var |
persistent private dirs; /var pre-seeded with the apt/dpkg directories |
/var/lib/dpkg |
overlay of the host dpkg database (when the host has it), regular files copied into the upper, lock files pre-created |
/tmp /run /dev/shm |
tmpfs |
/dev |
tmpfs + bound null zero full random urandom tty, new devpts instance |
/proc |
own pid namespace; sys sysrq-trigger irq bus fs read-only; kcore keys timer_list masked |
/sys, /home, everything else on host |
absent |
Why the skeleton and copies: host root-owned files appear as nobody inside the container. Container root cannot write into directories it does not own, nor open files it does not own for writing, nor hard-link them. The skeleton recreates every host directory as a container-owned directory in the upper layer (same mode, owner rwx added); the copies make the few files package managers rewrite container-owned.
Container filesystem, image mode (--image)
/ is one overlay: lower = images/NAME/rootfs (shared, never written), upper = layers/root/upper. /tmp /run /dev /proc are mounted over it as in host mode. The image is extracted under the same id map the container uses, so image files owned by root are owned by the container's root and nothing needs a skeleton or copies. LANG is C.UTF-8.
Images
SOURCE for image add:
| Source | Resolution | Verification |
|---|---|---|
alpine |
latest-stable minirootfs from dl-cdn.alpinelinux.org, via latest-releases.yaml |
sha256 from the yaml, and the detached .asc signature checked against Alpine's pinned release key (0482D840 22F52DF1 C4E7CD43 293ACD09 07D9495A) |
ubuntu, ubuntu-VERSION |
newest ubuntu-base-*-base-ARCH.tar.gz of that release from cdimage.ubuntu.com (default 26.04) |
SHA256SUMS checked with SHA256SUMS.gpg against Ubuntu's pinned CD image key (84393 8DF 228D 22F7 B374 2BC0 D94A A3F0 EFE2 1092), then sha256 |
debian, debian-RELEASE |
debuerreotype/docker-debian-artifacts OCI layer for RELEASE (default trixie), slim variant |
sha256 = the layer digest in the OCI manifest; no publisher signature exists |
https://... |
downloaded | --sha256 HEX required, otherwise rejected |
| a file path | read locally | verified when --sha256 is given; otherwise used with a printed note |
http:// is refused. Signature keys are fetched over https and rejected unless their fingerprint equals the pinned one; a bad, missing or foreign signature rejects the image. Without gpg installed the signature step is skipped with a printed note and the checksum still applies. A mismatch deletes everything it downloaded. The tarball is extracted with tar -p --numeric-owner, excluding dev/*, inside a user namespace with the image's id mode. Architectures: x86_64 and aarch64 for the catalog.
Size cap
The parent process measures the container's writable data (upper layers and private dirs, hard links counted once) from inside the container's id map, so files and mode-700 directories of other container ids are counted. A scan runs at most every 2 s and never uses more than 1/10 of the time (the wait is at least 10x the last scan). At 80% it warns once on stderr. At 100% it freezes the container's main cgroup (cgroup.freeze); processes stop, nothing is deleted. It thaws on its own once usage is below 95%. A second doos attach runs in the shell cgroup and is not frozen, so the user can free space from inside. No file can exceed the cap on its own (RLIMIT_FSIZE), which bounds the overshoot between scans. Without cgroup v2 delegation the cap only warns and says so at start.
Resource limits
Each running container gets doos/<name>/ below the cgroup delegated to the user's systemd manager, with leaves main (init and its descendants) and shell (later attaches). Limits sit on doos/<name> and cover both leaves: memory.max and memory.swap.max=0 with memory.oom.group=1 (an OOM kill takes the whole container), pids.max, cpu.max. After exit doos reports OOM kills and process-limit hits. A missing controller is named on stderr and the other limits still apply. Without delegation no limit is enforced, and doos says so at start. The group is removed when the container stops.
Hardening (--hardening default)
Every process that runs a container command gets, right before exec: no_new_privs, core dumps off, the capability bounding set reduced to the Docker default set (CHOWN DAC_OVERRIDE FOWNER FSETID KILL SETGID SETUID SETPCAP NET_BIND_SERVICE NET_RAW SYS_CHROOT MKNOD AUDIT_WRITE SETFCAP, mask a80425fb), and a seccomp-bpf filter: wrong architecture kills the process, x32 numbers get ENOSYS, clone3 gets ENOSYS (libc falls back to clone), clone/unshare with CLONE_NEWUSER get EPERM, and these get EPERM: the mount family and pivot_root, setns, keyring calls, bpf, perf_event_open, userfaultfd, io_uring_*, kexec_*, module loading, open_by_handle_at, process_vm_*, kcmp, reboot, swapon/off, clock and time setting, acct, quotactl and old obsolete calls. Supported on x86_64 and aarch64; elsewhere only the capability drop applies, with a printed note. Tools that need the denied calls (container runtimes inside a container, Chromium's sandbox, io_uring users) need --hardening relaxed.
Isolation
User, mount, pid, uts, ipc namespaces always; network namespace unless host. Id map: container 0 is the invoking user; with subuid ids 1..65535 map to the user's /etc/subuid range through newuidmap/newgidmap; with single nothing else is mappable. no_new_privs, core dumps off. pid 1 is a reaping init that forwards SIGTERM/SIGINT/SIGHUP/SIGQUIT and kills the container when the command exits. Container lifetime = the first attach command; later attaches join via setns.
Networking
slirp: private netns,slirp4netns --configure --disable-host-loopback, DNS 10.0.2.3, outbound only, host 127.0.0.1 unreachable.none: private netns with loopback only.slirphas no inbound path (nothing outside reaches a port served inside).pasta: private netns that behaves like the host network: addresses and routes mirrored, host loopback reachable, ports bound inside are forwarded to the host, DNS from the host's resolver. The host's abstract unix sockets are not shared. Needspasta. Newly bound ports are picked up after about 2 s; only loopback and the default gateway address reach other containers' forwarded ports, not the host's public address.host: shares the host network namespace including its abstract unix sockets (explicit opt-in; a note recommendspasta).
Safety rules
delete(containers and images) is dry-run by default, scoped to the store, refuses running containers and used images, never follows symlinks. Trees with subuid-owned files are removed from inside the id map.- Unverifiable input is rejected: URL images without sha256, hash mismatches, bad names, bad state paths, missing slirp4netns or newuidmap when requested.
- Seeding and scans report what they skipped (
unreadable skipped: N). - A failed start kills the half-built container (no orphans); a failed image add or container create leaves nothing behind.
--dav-pass(or a password embedded in a davmount URL) is stored in plain text in the container'smetafile (0700 directory, owner-only, same trust level as everything else under$DOOS_HOME); it is never written to a log, never passed torcloneon its command line, and is run throughrclone obscureonly immediately before the mount process execs.doos davmount umountis dry-run by default, scoped to$DOOS_HOME/davmounts, never follows symlinks; unmounting a target in use stops storage for any container still mounting it (not prevented, only named in the dry-run output).
Framework seams
How other MoloApp parts consume libdoos (all additive, all default-off):
- Supervisor isolation (
supervisor/common/isolation_backend.hpp, implemented bydoos/seam/isolation.hpp): an optionalProcessSpecbackend applying cgroup v2 limits (memory/pids/cpus) to the supervised child plus optional pre-exec hardening. All-or-nothing: when any requested limit cannot attach, the child is stopped and the worker reports failure instead of running degraded. Hardening enters the child's own user namespace plus a self id map first (capability drops need caps the child only holds there); setup failure exits 126. - Deploy strategy (
doos/deploy/doos_strategy.hpp, implementsmolo::deploy::Strategy): provisions one container per instance and verifies each with a probe command in batches, mirroring Rolling's protocol, caps (plus the 64-container ceiling), and event/counter names. Probes run serially insideready_timeout_ms(stuck probes get SIGTERM, then SIGKILL, race-free via pidfds); provisioning is not transactional (containers persist, retries reuse them). - Tenancy: doos is an opt-in OS-level execution isolation behind
tenant plus supervisor, never a tenancy strategy value; quotas, flags
and metrics stay in
molo::tenancy. - Observability: the seam layer on the molo side emits lifecycle signals through molo events/metrics; libdoos itself gains no observability dependency and the standalone binary keeps stderr text.
- Example: hermetic
/doosroutes (page, status, create, delete on a demo store) live inexample/app/DoosDemo.cpp; real container starts stay in the opt-indoos-livetarget, never in the default suite.
Contributor rules
Folded from the doos 1.6.0 AGENTS.md (frozen original:
docs/history/AGENTS.md); the root AGENTS.md pipeline stays
authoritative:
- C++23,
-Wall -Wextra -Wpedantic -Werror, warning-free; one class per file, keep the deep structure (cli/,seam/,deploy/). - Every limit lives in
Caps(caps.h) and in this SPEC; change both together, never one alone. - Behavior changes update this SPEC first, then code, then the example route, then tests, then the changelog entry (via the release tool).
- Deletes stay dry-run by default; destructive operations are confirmed explicitly and scoped as narrowly as possible.
- Never use em dashes anywhere: code, docs, comments, commits.
- No new third-party dependency without retiring the trust it replaces;
helpers stay pinned, verified, and managed under
$DOOS_HOME. src/doos/VERSIONis stamped into the binary and containermeta; only the release flow updates it, never a hand edit.