ad/molohttp_deploy.md

<!-- retoor <retoor@molodetz.nl> -->
# molohttp deploy and operations

Production server `molodetz`. molohttp 0.9.1. Facts below were read from the live system on 2026-10-05. This is a production box: follow the backup specification before every change.

## Access

Operator account is `retoor`. Root login is never needed.

- `retoor` is in the `sudo` group with `(ALL) NOPASSWD: ALL` (verified with `sudo -n true` and `sudo -n -l`).
- Every privileged command in this guide runs under `sudo` and also works non-interactively in scripts.
- `/etc/molohttp` is `molohttp:molohttp 750`: reads and edits both require `sudo`.

## How it works

molohttp is the single front door on ports 80 and 443. One systemd unit runs one process as user `molohttp` (24+ days uptime at time of writing).

- Binary: `/usr/local/bin/molohttp` (`root:root 755`).
- Unit: `/etc/systemd/system/molohttp.service` (`root:root 644`), enabled and active.
- Start: `molohttp run --config /etc/molohttp/config.json --env /etc/molohttp/.env.json`, working directory `/etc/molohttp`.
- Reload: `ExecReload=/bin/kill -HUP $MAINPID`, so `sudo systemctl reload molohttp` applies config without dropping connections. There is no config-validate subcommand; validate JSON by hand (see change procedure).
- Privileged ports are bound via `CAP_NET_BIND_SERVICE`; the process itself is unprivileged with `ProtectSystem=strict` and `ProtectHome=true`.
- `config.json` holds 94 sites, all `enabled`, all `reverse_proxy` to `http://127.0.0.1:<port>` with one exception: a single upstream on `http://pravda.education:10500`. One site carries a basic-auth middleware. `site-094` is `staging.app.molodetz.nl`, the Molodetz docker dev server (see Staging below).
- TLS: minimum TLS 1.2, HSTS one year with subdomains. Certificates live per hostname as `/etc/molohttp/certs/<host>.pem` plus `<host>.key`.
- ACME: production Let's Encrypt, account `retoor@molodetz.nl`, renewal check every 12 hours starting 30 days before expiry, state in `/var/lib/molohttp/acme`.
- Logging: JSON to stdout, collected by journald (`SyslogIdentifier=molohttp`). The `access_log_path`/`error_log_path` settings are not materialized as live files.
- Admin: the admin UI is deliberately not exposed. `admin.molodetz.nl` is not hosted, routed, or resolvable: no site entry, no DNS record. Exposing it would be a security flaw. The only admin path is the local CLI: `molohttp admin user-list`, `molohttp admin hash`. `molohttp install`/`uninstall` must run as root and are out of scope for routine work.

## Files and owners

Canonical state. Verify with `sudo find /etc/molohttp /var/lib/molohttp /var/log/molohttp -maxdepth 2 -exec stat -c '%U:%G %a %n' {} +`.

| Path | Owner | Mode | Notes |
|---|---|---|---|
| `/etc/molohttp` | `molohttp:molohttp` | 750 | config root, sudo to enter |
| `/etc/molohttp/config.json` | `molohttp:molohttp` | 640 | site table, exact path read |
| `/etc/molohttp/.env.json` | `molohttp:molohttp` | 640 | global settings, holds password hash |
| `/etc/molohttp/certs` | `molohttp:molohttp` | 700 | per-host pairs |
| `/etc/molohttp/certs/*.key` | `molohttp:molohttp` | 600 | private keys |
| `/etc/molohttp/certs/*.pem` | `molohttp:molohttp` | 644 | certificates |
| `/etc/molohttp/*.bak-*` | `root:root` | 640 | operator backups, this spec only |
| `/var/lib/molohttp` | `molohttp:molohttp` | 750 | state root |
| `/var/lib/molohttp/acme` | `molohttp:molohttp` | 750 | live ACME state |
| `/var/log/molohttp` | `molohttp:molohttp` | 750 | empty dir, daemon logs to journal |
| `/var/www/.well-known` | `molohttp:molohttp` | 750 | ACME challenges |
| `/etc/systemd/system/molohttp.service` | `root:root` | 644 | unit file |
| `/usr/local/bin/molohttp` | `root:root` | 755 | binary |

Known anomalies, observed 2026-10-05, left untouched pending a decision:

- Six key pairs under `certs/` are owned `caddy:caddy` (the `pravda.education` hosts plus `mail.molodetz.nl`). The `.key` files at mode 600 are unreadable by the `molohttp` user; confirm those hosts serve TLS correctly, then normalize with `sudo chown molohttp:molohttp` on the pair.
- `/var/lib/molohttp/analytics` plus `/var/lib/molohttp/stats` (`caddy:caddy`, stale since Jul 27) are leftovers from a previous server. Nothing reads them. (A stale 2.5 GB `caddy`-owned `access.log` in `/var/log/molohttp` was already removed on 2026-10-05 after confirming nothing held it open.)
- Older backups in `/etc/molohttp` use four different naming schemes and mixed owners. They are harmless history; all new backups follow the one format below.

## Backup specification

Always back up before editing. One format, no variants:

`<file>.bak-<reason>-<YYYYMMDDHHMMSS>`, owned `root:root`, mode 640, next to the original. Siblings are safe: the daemon reads `config.json` and `.env.json` by exact path only, never by glob.

```
FILE=/etc/molohttp/config.json
REASON=site-add-example
STAMP=$(date +%Y%m%d%H%M%S)
sudo cp -p "$FILE" "$FILE.bak-$REASON-$STAMP"
sudo chown root:root "$FILE.bak-$REASON-$STAMP"
sudo chmod 640 "$FILE.bak-$REASON-$STAMP"
ls -la "$FILE"*
```

Rules: reason is a required lowercase slug (`site-add-<host>`, `site-remove-<host>`, `upstream-<host>`, `tls-<host>`, `env-<what>`). Never delete old backups; at about 52 KB each they are the audit trail. Never invent a second format.

## Change procedure

1. Back up per the specification above.
2. Edit with `sudo` (for example `sudo -e /etc/molohttp/config.json`).
3. Restore ownership, since root-written files lock the daemon out: `sudo chown molohttp:molohttp /etc/molohttp/config.json`, `sudo chmod 640 /etc/molohttp/config.json`.
4. Validate: `sudo -n cat /etc/molohttp/config.json | python3 -m json.tool > /dev/null`.
5. Apply: `sudo systemctl reload molohttp`.
6. Verify: `systemctl is-active molohttp`, `curl -s -o /dev/null -w "%{http_code}\n" https://<host>/`, and `sudo journalctl -u molohttp --no-pager -n 20`.
7. Roll back on any failure: `sudo cp -p <backup> /etc/molohttp/config.json`, `sudo chown molohttp:molohttp /etc/molohttp/config.json`, `sudo chmod 640 /etc/molohttp/config.json`, then reload again and re-verify.

Use `restart` instead of `reload` only when the process itself must be replaced. Never `stop` without `start` on this box.

## Add or remove a site

Append one object to the `sites` array. Keep `id` unique (`site-094`, `site-095`, ...); `name` equals the primary hostname.

```
{
  "id": "site-094",
  "name": "new.molodetz.nl",
  "enabled": true,
  "matchers": {"hosts": ["new.molodetz.nl"], "headers": {}, "paths": []},
  "handler": {
    "type": "reverse_proxy",
    "upstreams": [{"url": "http://127.0.0.1:8XXX", "weight": 1}],
    "load_balancing": "round_robin",
    "websocket": false,
    "connection_timeout_ms": 5000,
    "read_timeout_ms": 30000
  }
}
```

Pick a free localhost port for the upstream and keep the app itself bound to 127.0.0.1. Certificates are provisioned automatically through ACME within seconds of the first HTTPS traffic; verify with `curl` and `openssl s_client -connect <host>:443 -servername <host>`, and check the journal on any TLS error. Per-host files under `certs/` may materialize later on molohttp's own schedule (observed with staging: valid certificate served immediately, files not yet present), so their absence right after creation is not a failure. Removal is the reverse: back up, delete the object, validate, reload, verify the hostname no longer routes.

## Staging

`staging.app.molodetz.nl` (`site-094`) proxies to `http://127.0.0.1:19847`, the Molodetz docker dev server from this repo (see Docker in `README.md`). It carries a normal Let's Encrypt certificate. Every saved change under `./molodetz` is live on staging within seconds through uvicorn reload; app edits never need a molohttp change. Staging data is the file mount `./data/staging`, refreshed from holy production (`./data/production`) with `make staging-refresh` (see `bin/staging-refresh.sh`). All environments live under `./data/<env>`; named volumes are never used.

## Static publishing

`static.molodetz.nl` proxies to `http://127.0.0.1:8117`, served by `rserver` running as `retoor` with working directory `/home/retoor/projects/static`. Publishing static content needs no molohttp change: write files under that directory and they are live (verified end to end with `/pizdetz/`, which redirects to `/pizdetz/index.html`). The Molodetz screenshot gallery is produced this way; see the Screenshots section in `README.md`.

## Ownership ensure

Run after any manual intervention to restore the canonical state. It does not touch the documented anomalies.

```
sudo chown molohttp:molohttp /etc/molohttp /etc/molohttp/config.json /etc/molohttp/.env.json
sudo chmod 750 /etc/molohttp
sudo chmod 640 /etc/molohttp/config.json /etc/molohttp/.env.json
sudo chown molohttp:molohttp /etc/molohttp/certs /var/lib/molohttp /var/log/molohttp /var/www/.well-known
sudo chmod 700 /etc/molohttp/certs
sudo chmod 750 /var/lib/molohttp /var/log/molohttp /var/www/.well-known
sudo find /etc/molohttp/certs -name '*.key' -user molohttp -exec chmod 600 {} +
sudo find /etc/molohttp/certs -name '*.pem' -user molohttp -exec chmod 644 {} +
sudo find /etc/molohttp -maxdepth 1 -name '*.bak-*' -exec chown root:root {} + -exec chmod 640 {} +
```

## Quick commands

| Task | Command |
|---|---|
| Status | `systemctl is-active molohttp`, `molohttp service-status` |
| Logs | `sudo journalctl -u molohttp -f`, `molohttp service-logs` |
| Reload config | `sudo systemctl reload molohttp` |
| Restart | `sudo systemctl restart molohttp` |
| Site table | `sudo -n cat /etc/molohttp/config.json \| python3 -m json.tool \| less` |
| Port owners | `sudo -n ss -tlnp` |
| Version | `molohttp --version` |