molohttp deploy and operations

Production server molodetz. molohttp 0.9.1. Facts below were read from the live system on 2026-10-05. This is a production box: follow the backup specification before every change.

Access

Operator account is retoor. Root login is never needed.

retoor is in the sudo group with (ALL) NOPASSWD: ALL (verified with sudo -n true and sudo -n -l).
Every privileged command in this guide runs under sudo and also works non-interactively in scripts.
/etc/molohttp is molohttp:molohttp 750: reads and edits both require sudo.

How it works

molohttp is the single front door on ports 80 and 443. One systemd unit runs one process as user molohttp (24+ days uptime at time of writing).

Binary: /usr/local/bin/molohttp (root:root 755).
Unit: /etc/systemd/system/molohttp.service (root:root 644), enabled and active.
Start: molohttp run --config /etc/molohttp/config.json --env /etc/molohttp/.env.json, working directory /etc/molohttp.
Reload: ExecReload=/bin/kill -HUP $MAINPID, so sudo systemctl reload molohttp applies config without dropping connections. There is no config-validate subcommand; validate JSON by hand (see change procedure).
Privileged ports are bound via CAP_NET_BIND_SERVICE; the process itself is unprivileged with ProtectSystem=strict and ProtectHome=true.
config.json holds 94 sites, all enabled, all reverse_proxy to http://127.0.0.1:<port> with one exception: a single upstream on http://pravda.education:10500. One site carries a basic-auth middleware. site-094 is staging.app.molodetz.nl, the Molodetz docker dev server (see Staging below).
TLS: minimum TLS 1.2, HSTS one year with subdomains. Certificates live per hostname as /etc/molohttp/certs/<host>.pem plus <host>.key.
ACME: production Let's Encrypt, account retoor@molodetz.nl, renewal check every 12 hours starting 30 days before expiry, state in /var/lib/molohttp/acme.
Logging: JSON to stdout, collected by journald (SyslogIdentifier=molohttp). The access_log_path/error_log_path settings are not materialized as live files.
Admin: the admin UI is deliberately not exposed. admin.molodetz.nl is not hosted, routed, or resolvable: no site entry, no DNS record. Exposing it would be a security flaw. The only admin path is the local CLI: molohttp admin user-list, molohttp admin hash. molohttp install/uninstall must run as root and are out of scope for routine work.

Files and owners

Canonical state. Verify with sudo find /etc/molohttp /var/lib/molohttp /var/log/molohttp -maxdepth 2 -exec stat -c '%U:%G %a %n' {} +.

Path	Owner	Mode	Notes
`/etc/molohttp`	`molohttp:molohttp`	750	config root, sudo to enter
`/etc/molohttp/config.json`	`molohttp:molohttp`	640	site table, exact path read
`/etc/molohttp/.env.json`	`molohttp:molohttp`	640	global settings, holds password hash
`/etc/molohttp/certs`	`molohttp:molohttp`	700	per-host pairs
`/etc/molohttp/certs/*.key`	`molohttp:molohttp`	600	private keys
`/etc/molohttp/certs/*.pem`	`molohttp:molohttp`	644	certificates
`/etc/molohttp/.bak-`	`root:root`	640	operator backups, this spec only
`/var/lib/molohttp`	`molohttp:molohttp`	750	state root
`/var/lib/molohttp/acme`	`molohttp:molohttp`	750	live ACME state
`/var/log/molohttp`	`molohttp:molohttp`	750	empty dir, daemon logs to journal
`/var/www/.well-known`	`molohttp:molohttp`	750	ACME challenges
`/etc/systemd/system/molohttp.service`	`root:root`	644	unit file
`/usr/local/bin/molohttp`	`root:root`	755	binary

Known anomalies, observed 2026-10-05, left untouched pending a decision:

Six key pairs under certs/ are owned caddy:caddy (the pravda.education hosts plus mail.molodetz.nl). The .key files at mode 600 are unreadable by the molohttp user; confirm those hosts serve TLS correctly, then normalize with sudo chown molohttp:molohttp on the pair.
/var/lib/molohttp/analytics plus /var/lib/molohttp/stats (caddy:caddy, stale since Jul 27) are leftovers from a previous server. Nothing reads them. (A stale 2.5 GB caddy-owned access.log in /var/log/molohttp was already removed on 2026-10-05 after confirming nothing held it open.)
Older backups in /etc/molohttp use four different naming schemes and mixed owners. They are harmless history; all new backups follow the one format below.

Backup specification

Always back up before editing. One format, no variants:

<file>.bak-<reason>-<YYYYMMDDHHMMSS>, owned root:root, mode 640, next to the original. Siblings are safe: the daemon reads config.json and .env.json by exact path only, never by glob.

FILE=/etc/molohttp/config.json
REASON=site-add-example
STAMP=$(date +%Y%m%d%H%M%S)
sudo cp -p "$FILE" "$FILE.bak-$REASON-$STAMP"
sudo chown root:root "$FILE.bak-$REASON-$STAMP"
sudo chmod 640 "$FILE.bak-$REASON-$STAMP"
ls -la "$FILE"*

Rules: reason is a required lowercase slug (site-add-<host>, site-remove-<host>, upstream-<host>, tls-<host>, env-<what>). Never delete old backups; at about 52 KB each they are the audit trail. Never invent a second format.

Change procedure

Back up per the specification above.
Edit with sudo (for example sudo -e /etc/molohttp/config.json).
Restore ownership, since root-written files lock the daemon out: sudo chown molohttp:molohttp /etc/molohttp/config.json, sudo chmod 640 /etc/molohttp/config.json.
Validate: sudo -n cat /etc/molohttp/config.json | python3 -m json.tool > /dev/null.
Apply: sudo systemctl reload molohttp.
Verify: systemctl is-active molohttp, curl -s -o /dev/null -w "%{http_code}\n" https://<host>/, and sudo journalctl -u molohttp --no-pager -n 20.
Roll back on any failure: sudo cp -p <backup> /etc/molohttp/config.json, sudo chown molohttp:molohttp /etc/molohttp/config.json, sudo chmod 640 /etc/molohttp/config.json, then reload again and re-verify.

Use restart instead of reload only when the process itself must be replaced. Never stop without start on this box.

Add or remove a site

Append one object to the sites array. Keep id unique (site-094, site-095, ...); name equals the primary hostname.

{
  "id": "site-094",
  "name": "new.molodetz.nl",
  "enabled": true,
  "matchers": {"hosts": ["new.molodetz.nl"], "headers": {}, "paths": []},
  "handler": {
    "type": "reverse_proxy",
    "upstreams": [{"url": "http://127.0.0.1:8XXX", "weight": 1}],
    "load_balancing": "round_robin",
    "websocket": false,
    "connection_timeout_ms": 5000,
    "read_timeout_ms": 30000
  }
}

Pick a free localhost port for the upstream and keep the app itself bound to 127.0.0.1. Certificates are provisioned automatically through ACME within seconds of the first HTTPS traffic; verify with curl and openssl s_client -connect <host>:443 -servername <host>, and check the journal on any TLS error. Per-host files under certs/ may materialize later on molohttp's own schedule (observed with staging: valid certificate served immediately, files not yet present), so their absence right after creation is not a failure. Removal is the reverse: back up, delete the object, validate, reload, verify the hostname no longer routes.

Staging

staging.app.molodetz.nl (site-094) proxies to http://127.0.0.1:19847, the Molodetz docker dev server from this repo (see Docker in README.md). It carries a normal Let's Encrypt certificate. Every saved change under ./molodetz is live on staging within seconds through uvicorn reload; app edits never need a molohttp change. Staging data is the file mount ./data/staging, refreshed from holy production (./data/production) with make staging-refresh (see bin/staging-refresh.sh). All environments live under ./data/<env>; named volumes are never used.

Static publishing

static.molodetz.nl proxies to http://127.0.0.1:8117, served by rserver running as retoor with working directory /home/retoor/projects/static. Publishing static content needs no molohttp change: write files under that directory and they are live (verified end to end with /pizdetz/, which redirects to /pizdetz/index.html). The Molodetz screenshot gallery is produced this way; see the Screenshots section in README.md.

Ownership ensure

Run after any manual intervention to restore the canonical state. It does not touch the documented anomalies.

sudo chown molohttp:molohttp /etc/molohttp /etc/molohttp/config.json /etc/molohttp/.env.json
sudo chmod 750 /etc/molohttp
sudo chmod 640 /etc/molohttp/config.json /etc/molohttp/.env.json
sudo chown molohttp:molohttp /etc/molohttp/certs /var/lib/molohttp /var/log/molohttp /var/www/.well-known
sudo chmod 700 /etc/molohttp/certs
sudo chmod 750 /var/lib/molohttp /var/log/molohttp /var/www/.well-known
sudo find /etc/molohttp/certs -name '*.key' -user molohttp -exec chmod 600 {} +
sudo find /etc/molohttp/certs -name '*.pem' -user molohttp -exec chmod 644 {} +
sudo find /etc/molohttp -maxdepth 1 -name '*.bak-*' -exec chown root:root {} + -exec chmod 640 {} +

Quick commands

Task	Command
Status	`systemctl is-active molohttp`, `molohttp service-status`
Logs	`sudo journalctl -u molohttp -f`, `molohttp service-logs`
Reload config	`sudo systemctl reload molohttp`
Restart	`sudo systemctl restart molohttp`
Site table	`sudo -n cat /etc/molohttp/config.json \| python3 -m json.tool \| less`
Port owners	`sudo -n ss -tlnp`
Version	`molohttp --version`

9.5 KiB Raw Blame History