ad/molohttp_deploy.md

molohttp deploy and operations

Production server molodetz. molohttp 0.9.1. Facts below were read from the live system on 2026-10-05. This is a production box: follow the backup specification before every change.

Access

Operator account is retoor. Root login is never needed.

• retoor is in the sudo group with (ALL) NOPASSWD: ALL (verified with sudo -n true and sudo -n -l).
• Every privileged command in this guide runs under sudo and also works non-interactively in scripts.
• /etc/molohttp is molohttp:molohttp 750: reads and edits both require sudo.

How it works

molohttp is the single front door on ports 80 and 443. One systemd unit runs one process as user molohttp (24+ days uptime at time of writing).

• Binary: /usr/local/bin/molohttp (root:root 755).
• Unit: /etc/systemd/system/molohttp.service (root:root 644), enabled and active.
• Start: molohttp run --config /etc/molohttp/config.json --env /etc/molohttp/.env.json, working directory /etc/molohttp.
• Reload: ExecReload=/bin/kill -HUP $MAINPID, so sudo systemctl reload molohttp applies config without dropping connections. There is no config-validate subcommand; validate JSON by hand (see change procedure).
• Privileged ports are bound via CAP_NET_BIND_SERVICE; the process itself is unprivileged with ProtectSystem=strict and ProtectHome=true.
• config.json holds 95 sites, all enabled, all reverse_proxy to http://127.0.0.1:<port> with one exception: a single upstream on http://pravda.education:10500. One site carries a basic-auth middleware. Molodetz owns three: site-022 molodetz.nl (production, see Cutover), site-094 staging.app.molodetz.nl (see Staging), site-095 molodetz-test.app.molodetz.nl (see Test).
• Hostnames must be unique across sites. On reload molohttp keeps the first site per hostname and silently drops later duplicates, persisting the result. Always assert the hostname is free before appending (observed 2026-10-05: a duplicate test.app.molodetz.nl entry vanished on reload because site-085 already claimed it).
• TLS: minimum TLS 1.2, HSTS one year with subdomains. Certificates live per hostname as /etc/molohttp/certs/<host>.pem plus <host>.key.
• ACME: production Let's Encrypt, account retoor@molodetz.nl, renewal check every 12 hours starting 30 days before expiry, state in /var/lib/molohttp/acme.
• Logging: JSON to stdout, collected by journald (SyslogIdentifier=molohttp). The access_log_path/error_log_path settings are not materialized as live files.
• Admin: the admin UI is deliberately not exposed. admin.molodetz.nl is not hosted, routed, or resolvable: no site entry, no DNS record. Exposing it would be a security flaw. The only admin path is the local CLI: molohttp admin user-list, molohttp admin hash. molohttp install/uninstall must run as root and are out of scope for routine work.

Files and owners

Canonical state. Verify with sudo find /etc/molohttp /var/lib/molohttp /var/log/molohttp -maxdepth 2 -exec stat -c '%U:%G %a %n' {} +.

Path	Owner	Mode	Notes
/etc/molohttp	molohttp:molohttp	750	config root, sudo to enter
/etc/molohttp/config.json	molohttp:molohttp	640	site table, exact path read
/etc/molohttp/.env.json	molohttp:molohttp	640	global settings, holds password hash
/etc/molohttp/certs	molohttp:molohttp	700	per-host pairs
/etc/molohttp/certs/*.key	molohttp:molohttp	600	private keys
/etc/molohttp/certs/*.pem	molohttp:molohttp	644	certificates
/etc/molohttp/*.bak-*	root:root	640	operator backups, this spec only
/var/lib/molohttp	molohttp:molohttp	750	state root
/var/lib/molohttp/acme	molohttp:molohttp	750	live ACME state
/var/log/molohttp	molohttp:molohttp	750	empty dir, daemon logs to journal
/var/www/.well-known	molohttp:molohttp	750	ACME challenges
/etc/systemd/system/molohttp.service	root:root	644	unit file
/usr/local/bin/molohttp	root:root	755	binary

Known anomalies, observed 2026-10-05, left untouched pending a decision:

• Six key pairs under certs/ are owned caddy:caddy (the pravda.education hosts plus mail.molodetz.nl). The .key files at mode 600 are unreadable by the molohttp user; confirm those hosts serve TLS correctly, then normalize with sudo chown molohttp:molohttp on the pair.
• /var/lib/molohttp/analytics plus /var/lib/molohttp/stats (caddy:caddy, stale since Jul 27) are leftovers from a previous server. Nothing reads them. (A stale 2.5 GB caddy-owned access.log in /var/log/molohttp was already removed on 2026-10-05 after confirming nothing held it open.)
• Older backups in /etc/molohttp use four different naming schemes and mixed owners. They are harmless history; all new backups follow the one format below.

Backup specification

Always back up before editing. One format, no variants:

<file>.bak-<reason>-<YYYYMMDDHHMMSS>, owned root:root, mode 640, next to the original. Siblings are safe: the daemon reads config.json and .env.json by exact path only, never by glob.

FILE=/etc/molohttp/config.json
REASON=site-add-example
STAMP=$(date +%Y%m%d%H%M%S)
sudo cp -p "$FILE" "$FILE.bak-$REASON-$STAMP"
sudo chown root:root "$FILE.bak-$REASON-$STAMP"
sudo chmod 640 "$FILE.bak-$REASON-$STAMP"
ls -la "$FILE"*

Rules: reason is a required lowercase slug (site-add-<host>, site-remove-<host>, upstream-<host>, tls-<host>, env-<what>). Never delete old backups; at about 52 KB each they are the audit trail. Never invent a second format.

Change procedure

1. Back up per the specification above.
2. Edit with sudo (for example sudo -e /etc/molohttp/config.json).
3. Restore ownership, since root-written files lock the daemon out: sudo chown molohttp:molohttp /etc/molohttp/config.json, sudo chmod 640 /etc/molohttp/config.json.
4. Validate: sudo -n cat /etc/molohttp/config.json | python3 -m json.tool > /dev/null.
5. Apply: sudo systemctl reload molohttp.
6. Verify: systemctl is-active molohttp, curl -s -o /dev/null -w "%{http_code}\n" https://<host>/, and sudo journalctl -u molohttp --no-pager -n 20.
7. Roll back on any failure: sudo cp -p <backup> /etc/molohttp/config.json, sudo chown molohttp:molohttp /etc/molohttp/config.json, sudo chmod 640 /etc/molohttp/config.json, then reload again and re-verify.

Use restart instead of reload only when the process itself must be replaced. Never stop without start on this box.

Add or remove a site

Append one object to the sites array. Keep id unique (site-094, site-095, ...); name equals the primary hostname.

{
  "id": "site-094",
  "name": "new.molodetz.nl",
  "enabled": true,
  "matchers": {"hosts": ["new.molodetz.nl"], "headers": {}, "paths": []},
  "handler": {
    "type": "reverse_proxy",
    "upstreams": [{"url": "http://127.0.0.1:8XXX", "weight": 1}],
    "load_balancing": "round_robin",
    "websocket": false,
    "connection_timeout_ms": 5000,
    "read_timeout_ms": 30000
  }
}

Pick a free localhost port for the upstream and keep the app itself bound to 127.0.0.1. Certificates are provisioned automatically through ACME within seconds of the first HTTPS traffic; verify with curl and openssl s_client -connect <host>:443 -servername <host>, and check the journal on any TLS error. Per-host files under certs/ may materialize later on molohttp's own schedule (observed with staging: valid certificate served immediately, files not yet present), so their absence right after creation is not a failure. Removal is the reverse: back up, delete the object, validate, reload, verify the hostname no longer routes.

Staging

staging.app.molodetz.nl (site-094) proxies to http://127.0.0.1:19847, the Molodetz docker dev server from this repo (see Docker in README.md). It carries a normal Let's Encrypt certificate. Every saved change under ./molodetz is live on staging within seconds through uvicorn reload; app edits never need a molohttp change. Staging data is the file mount ./data/staging, refreshed from holy production (./data/production) with make staging-refresh (see bin/staging-refresh.sh). All environments live under ./data/<env>; named volumes are never used.

Test

molodetz-test.app.molodetz.nl (site-095) proxies to http://127.0.0.1:19849, the Molodetz docker test server (docker-compose.test.yml, dev server with reload, data in ./data/test, default admin password test-admin-pass-1). test.app.molodetz.nl belongs to another app (site-085) and must not be touched.

Production cutover

molodetz.nl (site-022) was cut over from molodev (127.0.0.1:8084) to the Molodetz prod container (127.0.0.1:19848) on 2026-10-05, backup config.json.bak-site-cutover-prod-20261005145603. Production data is holy in ./data/production; the container runs docker-compose.prod.yml (2 workers, no reload) with secrets from PROD_SECRET_KEY and PROD_ADMIN_PASSWORD, which have no defaults and fail loudly when unset.

Toggle back to the old app (molodev is still running on :8084 for exactly this): back up per the specification with reason site-cutover-rollback, set site-022 upstreams back to [{"url": "http://127.0.0.1:8084", "weight": 1}], restore ownership, validate JSON, reload, verify https://molodetz.nl/ answers the old site. Toggle forward again with the same steps in reverse (:19848).

Static publishing

static.molodetz.nl proxies to http://127.0.0.1:8117, served by rserver running as retoor with working directory /home/retoor/projects/static. Publishing static content needs no molohttp change: write files under that directory and they are live (verified end to end with /pizdetz/, which redirects to /pizdetz/index.html). The Molodetz screenshot gallery is produced this way; see the Screenshots section in README.md.

Ownership ensure

Run after any manual intervention to restore the canonical state. It does not touch the documented anomalies.

sudo chown molohttp:molohttp /etc/molohttp /etc/molohttp/config.json /etc/molohttp/.env.json
sudo chmod 750 /etc/molohttp
sudo chmod 640 /etc/molohttp/config.json /etc/molohttp/.env.json
sudo chown molohttp:molohttp /etc/molohttp/certs /var/lib/molohttp /var/log/molohttp /var/www/.well-known
sudo chmod 700 /etc/molohttp/certs
sudo chmod 750 /var/lib/molohttp /var/log/molohttp /var/www/.well-known
sudo find /etc/molohttp/certs -name '*.key' -user molohttp -exec chmod 600 {} +
sudo find /etc/molohttp/certs -name '*.pem' -user molohttp -exec chmod 644 {} +
sudo find /etc/molohttp -maxdepth 1 -name '*.bak-*' -exec chown root:root {} + -exec chmod 640 {} +

Quick commands

Task	Command
Status	systemctl is-active molohttp, molohttp service-status
Logs	sudo journalctl -u molohttp -f, molohttp service-logs
Reload config	sudo systemctl reload molohttp
Restart	sudo systemctl restart molohttp
Site table	sudo -n cat /etc/molohttp/config.json | python3 -m json.tool | less
Port owners	sudo -n ss -tlnp
Version	molohttp --version