<!-- retoor <retoor@molodetz.nl> -->
# molohttp deploy and operations

Production server `molodetz`. molohttp 0.9.1. Facts below were read from the live system on 2026-10-05. This is a production box: follow the backup specification before every change.

## Access

Operator account is `retoor`. Root login is never needed.

- `retoor` is in the `sudo` group with `(ALL) NOPASSWD: ALL` (verified with `sudo -n true` and `sudo -n -l`).
- Every privileged command in this guide runs under `sudo` and also works non-interactively in scripts.
- `/etc/molohttp` is `molohttp:molohttp 750`: reads and edits both require `sudo`.

## How it works

molohttp is the single front door on ports 80 and 443. One systemd unit runs one process as user `molohttp` (24+ days uptime at time of writing).

- Binary: `/usr/local/bin/molohttp` (`root:root 755`).
- Unit: `/etc/systemd/system/molohttp.service` (`root:root 644`), enabled and active.
- Start: `molohttp run --config /etc/molohttp/config.json --env /etc/molohttp/.env.json`, working directory `/etc/molohttp`.
- Reload: `ExecReload=/bin/kill -HUP $MAINPID`, so `sudo systemctl reload molohttp` applies config without dropping connections. There is no config-validate subcommand; validate JSON by hand (see change procedure).
- Privileged ports are bound via `CAP_NET_BIND_SERVICE`; the process itself is unprivileged with `ProtectSystem=strict` and `ProtectHome=true`.
- `config.json` holds 95 sites, all `enabled`, all `reverse_proxy` to `http://127.0.0.1:<port>` with one exception: a single upstream on `http://pravda.education:10500`. One site carries a basic-auth middleware. Molodetz owns three: `site-022` `molodetz.nl` (production, see Cutover), `site-094` `staging.app.molodetz.nl` (see Staging), `site-095` `molodetz-test.app.molodetz.nl` (see Test).
- Hostnames must be unique across sites. On reload molohttp keeps the first site per hostname and silently drops later duplicates, persisting the result. Always assert the hostname is free before appending (observed 2026-10-05: a duplicate `test.app.molodetz.nl` entry vanished on reload because `site-085` already claimed it).
- TLS: minimum TLS 1.2, HSTS one year with subdomains. Certificates live per hostname as `/etc/molohttp/certs/<host>.pem` plus `<host>.key`.
- ACME: production Let's Encrypt, account `retoor@molodetz.nl`, renewal check every 12 hours starting 30 days before expiry, state in `/var/lib/molohttp/acme`.
- Logging: JSON to stdout, collected by journald (`SyslogIdentifier=molohttp`). The `access_log_path`/`error_log_path` settings are not materialized as live files.
- Admin: the admin UI is deliberately not exposed. `admin.molodetz.nl` is not hosted, routed, or resolvable: no site entry, no DNS record. Exposing it would be a security flaw. The only admin path is the local CLI: `molohttp admin user-list`, `molohttp admin hash`. `molohttp install`/`uninstall` must run as root and are out of scope for routine work.

## Files and owners

Canonical state. Verify with `sudo find /etc/molohttp /var/lib/molohttp /var/log/molohttp -maxdepth 2 -exec stat -c '%U:%G %a %n' {} +`.

| Path | Owner | Mode | Notes |
|---|---|---|---|
| `/etc/molohttp` | `molohttp:molohttp` | 750 | config root, sudo to enter |
| `/etc/molohttp/config.json` | `molohttp:molohttp` | 640 | site table, exact path read |
| `/etc/molohttp/.env.json` | `molohttp:molohttp` | 640 | global settings, holds password hash |
| `/etc/molohttp/certs` | `molohttp:molohttp` | 700 | per-host pairs |
| `/etc/molohttp/certs/*.key` | `molohttp:molohttp` | 600 | private keys |
| `/etc/molohttp/certs/*.pem` | `molohttp:molohttp` | 644 | certificates |
| `/etc/molohttp/*.bak-*` | `root:root` | 640 | operator backups, this spec only |
| `/var/lib/molohttp` | `molohttp:molohttp` | 750 | state root |
| `/var/lib/molohttp/acme` | `molohttp:molohttp` | 750 | live ACME state |
| `/var/log/molohttp` | `molohttp:molohttp` | 750 | empty dir, daemon logs to journal |
| `/var/www/.well-known` | `molohttp:molohttp` | 750 | ACME challenges |
| `/etc/systemd/system/molohttp.service` | `root:root` | 644 | unit file |
| `/usr/local/bin/molohttp` | `root:root` | 755 | binary |

Known anomalies, observed 2026-10-05, left untouched pending a decision:

- Six key pairs under `certs/` are owned `caddy:caddy` (the `pravda.education` hosts plus `mail.molodetz.nl`). The `.key` files at mode 600 are unreadable by the `molohttp` user; confirm those hosts serve TLS correctly, then normalize with `sudo chown molohttp:molohttp` on the pair.
- `/var/lib/molohttp/analytics` plus `/var/lib/molohttp/stats` (`caddy:caddy`, stale since Jul 27) are leftovers from a previous server. Nothing reads them. (A stale 2.5 GB `caddy`-owned `access.log` in `/var/log/molohttp` was already removed on 2026-10-05 after confirming nothing held it open.)
- Older backups in `/etc/molohttp` use four different naming schemes and mixed owners. They are harmless history; all new backups follow the one format below.

## Backup specification

Always back up before editing. One format, no variants:

`<file>.bak-<reason>-<YYYYMMDDHHMMSS>`, owned `root:root`, mode 640, next to the original. Siblings are safe: the daemon reads `config.json` and `.env.json` by exact path only, never by glob.

```
FILE=/etc/molohttp/config.json
REASON=site-add-example
STAMP=$(date +%Y%m%d%H%M%S)
sudo cp -p "$FILE" "$FILE.bak-$REASON-$STAMP"
sudo chown root:root "$FILE.bak-$REASON-$STAMP"
sudo chmod 640 "$FILE.bak-$REASON-$STAMP"
ls -la "$FILE"*
```

Rules: reason is a required lowercase slug (`site-add-<host>`, `site-remove-<host>`, `upstream-<host>`, `tls-<host>`, `env-<what>`). Never delete old backups; at about 52 KB each they are the audit trail. Never invent a second format.

## Change procedure

1. Back up per the specification above.
2. Edit with `sudo` (for example `sudo -e /etc/molohttp/config.json`).
3. Restore ownership, since root-written files lock the daemon out: `sudo chown molohttp:molohttp /etc/molohttp/config.json`, `sudo chmod 640 /etc/molohttp/config.json`.
4. Validate: `sudo -n cat /etc/molohttp/config.json | python3 -m json.tool > /dev/null`.
5. Apply: `sudo systemctl reload molohttp`.
6. Verify: `systemctl is-active molohttp`, `curl -s -o /dev/null -w "%{http_code}\n" https://<host>/`, and `sudo journalctl -u molohttp --no-pager -n 20`.
7. Roll back on any failure: `sudo cp -p <backup> /etc/molohttp/config.json`, `sudo chown molohttp:molohttp /etc/molohttp/config.json`, `sudo chmod 640 /etc/molohttp/config.json`, then reload again and re-verify.

Use `restart` instead of `reload` only when the process itself must be replaced. Never `stop` without `start` on this box.

## Add or remove a site

Append one object to the `sites` array. Keep `id` unique (`site-094`, `site-095`, ...); `name` equals the primary hostname.

```
{
  "id": "site-094",
  "name": "new.molodetz.nl",
  "enabled": true,
  "matchers": {"hosts": ["new.molodetz.nl"], "headers": {}, "paths": []},
  "handler": {
    "type": "reverse_proxy",
    "upstreams": [{"url": "http://127.0.0.1:8XXX", "weight": 1}],
    "load_balancing": "round_robin",
    "websocket": false,
    "connection_timeout_ms": 5000,
    "read_timeout_ms": 30000
  }
}
```

Pick a free localhost port for the upstream and keep the app itself bound to 127.0.0.1. Certificates are provisioned automatically through ACME within seconds of the first HTTPS traffic; verify with `curl` and `openssl s_client -connect <host>:443 -servername <host>`, and check the journal on any TLS error. Per-host files under `certs/` may materialize later on molohttp's own schedule (observed with staging: valid certificate served immediately, files not yet present), so their absence right after creation is not a failure. Removal is the reverse: back up, delete the object, validate, reload, verify the hostname no longer routes.

## Staging

`staging.app.molodetz.nl` (`site-094`) proxies to `http://127.0.0.1:19847`, the Molodetz docker dev server from this repo (see Docker in `README.md`). It carries a normal Let's Encrypt certificate. Every saved change under `./molodetz` is live on staging within seconds through uvicorn reload; app edits never need a molohttp change. Staging data is the file mount `./data/staging`, refreshed from holy production (`./data/production`) with `make staging-refresh` (see `bin/staging-refresh.sh`). All environments live under `./data/<env>`; named volumes are never used.

## Test

`molodetz-test.app.molodetz.nl` (`site-095`) proxies to `http://127.0.0.1:19849`, the Molodetz docker test server (`docker-compose.test.yml`, dev server with reload, data in `./data/test`, default admin password `test-admin-pass-1`). `test.app.molodetz.nl` belongs to another app (`site-085`) and must not be touched.

## Production cutover

`molodetz.nl` (`site-022`) was cut over from molodev (`127.0.0.1:8084`) to the Molodetz prod container (`127.0.0.1:19848`) on 2026-10-05, backup `config.json.bak-site-cutover-prod-20261005145603`. Production data is holy in `./data/production`; the container runs `docker-compose.prod.yml` (2 workers, no reload) with secrets from `PROD_SECRET_KEY` and `PROD_ADMIN_PASSWORD`, which have no defaults and fail loudly when unset.

Toggle back to the old app (molodev is still running on `:8084` for exactly this): back up per the specification with reason `site-cutover-rollback`, set `site-022` upstreams back to `[{"url": "http://127.0.0.1:8084", "weight": 1}]`, restore ownership, validate JSON, reload, verify `https://molodetz.nl/` answers the old site. Toggle forward again with the same steps in reverse (`:19848`).

## Static publishing

`static.molodetz.nl` proxies to `http://127.0.0.1:8117`, served by `rserver` running as `retoor` with working directory `/home/retoor/projects/static`. Publishing static content needs no molohttp change: write files under that directory and they are live (verified end to end with `/pizdetz/`, which redirects to `/pizdetz/index.html`). The Molodetz screenshot gallery is produced this way; see the Screenshots section in `README.md`.

## Ownership ensure

Run after any manual intervention to restore the canonical state. It does not touch the documented anomalies.

```
sudo chown molohttp:molohttp /etc/molohttp /etc/molohttp/config.json /etc/molohttp/.env.json
sudo chmod 750 /etc/molohttp
sudo chmod 640 /etc/molohttp/config.json /etc/molohttp/.env.json
sudo chown molohttp:molohttp /etc/molohttp/certs /var/lib/molohttp /var/log/molohttp /var/www/.well-known
sudo chmod 700 /etc/molohttp/certs
sudo chmod 750 /var/lib/molohttp /var/log/molohttp /var/www/.well-known
sudo find /etc/molohttp/certs -name '*.key' -user molohttp -exec chmod 600 {} +
sudo find /etc/molohttp/certs -name '*.pem' -user molohttp -exec chmod 644 {} +
sudo find /etc/molohttp -maxdepth 1 -name '*.bak-*' -exec chown root:root {} + -exec chmod 640 {} +
```

## Quick commands

| Task | Command |
|---|---|
| Status | `systemctl is-active molohttp`, `molohttp service-status` |
| Logs | `sudo journalctl -u molohttp -f`, `molohttp service-logs` |
| Reload config | `sudo systemctl reload molohttp` |
| Restart | `sudo systemctl restart molohttp` |
| Site table | `sudo -n cat /etc/molohttp/config.json \| python3 -m json.tool \| less` |
| Port owners | `sudo -n ss -tlnp` |
| Version | `molohttp --version` |