Reverse proxy / expose specification (reverse tunnels)

Part: reverse tunnel server + molo expose client. Owned: src/realtime/ (tunnel protocol) + src/cli/ (expose CLI). Plan: plans/06-realtime.md § tunnel. Version: 0.2.0.

1. Purpose

Any application (in a container or elsewhere) can publish one or more of its local endpoints through the MoloApp server so the public internet reaches them via claimed public routes. Transport between the local apps ("exposer") and the server is always one outbound connection initiated by the exposer; inbound TCP to the exposer is never required. The server then reverse-proxies public HTTP traffic arriving at the claimed routes into that tunnel. One tunnel serves many concurrent upstreams: each public request is routed inside the tunnel to the matching upstream entry.

This system is HTTP/2 (RFC 9113, src/runtime/http/v2/). The tunnel rides the same H2 stack: one H2 connection carries many concurrent streams, one stream per proxied request. There is no HTTP/1-only or single-flight assumption anywhere in this spec.

2. Roles and terms

Server: the MoloApp instance accepting tunnels and public traffic.
Exposer / agent: the molo expose process running near the local apps.
Upstream: one local origin the agent forwards to. One of: host:port (plain TCP), http(s)://... URL, ws(s)://... URL. A tunnel holds an upstream table: {path_prefix -> upstream} plus a default entry. Many upstreams per tunnel, all concurrent.
Claim: a public route reservation. Forms:
1. relative path only, e.g. /hook1 — served on all server domains;
2. domain + path, e.g. example.com/hook1 — served on that host only;
3. bare domain, e.g. example.com or example.com/ — whole-host claim.
Key: per-claim pre-shared token authorizing the agent.

molo expose <upstream>... [<key>@<server>] [<claim>]
molo expose --upstream <upstream> [--upstream <upstream>...] --server <server> --claim <claim> --key <key>

<upstream>: [scheme://]host[:port][/path] where scheme is tcp | http | https | ws | wss. No scheme defaults to http. Examples: 8080, 127.0.0.1:3000, http://127.0.0.1:3000, wss://127.0.0.1:8443/socket. Repeatable. Each entry may carry a sub-path selector: <upstream>=<subpath>, e.g. http://127.0.0.1:3001=/api, http://127.0.0.1:3002=/static. Entries without a selector share the default entry; longest sub-path prefix wins per request.
<server>: server base URL, http(s)://host[:port] or ws(s)://host[:port]. Path prefix other than / is rejected (exit 2). Default: the server the CLI is configured against (dev default http://127.0.0.1:8080).
[key@]<server>: when key@ prefix is present it is the claim key; otherwise --key flag or MOLO_EXPOSE_KEY env supplies it.
<claim> ("reverse proxy url" in shorthand): [/path] or [host][/path]. Host part never contains a scheme or port. Path must start with / (bare host implies /). Normalization: lowercase host, collapse duplicate slashes, strip trailing slash except root, reject .., %2e, backslash, NUL. Max 256 chars host, 1024 chars path.

Examples:

molo expose 8080 mykey@https://srv.example.com /hook1
molo expose http://127.0.0.1:3000 --server https://srv.example.com --claim acme.example.com/ --key mykey
molo expose http://127.0.0.1:3001=/api http://127.0.0.1:3002=/ mykey@https://srv.example.com /app

4. Tunnel handshake

Agent dials the server's H2 endpoint and opens the tunnel as one H2 connection (h2c upgrade or prior-knowledge/h2-TLS per server config). The legacy single-WebSocket form (GET /__molo/expose with Upgrade: websocket, headers X-Molo-Claim + Authorization: Bearer, subprotocol molo-expose-v1) remains accepted as a fallback transport, but the normative transport is H2: one connection, many streams. Query form ?claim=&key= is accepted for non-CLI agents and logged as deprecated.
Server validates: unknown/expired key → 403; malformed claim → 400; claim already held and live → 409 (unless --takeover with a newer key generation, then old tunnel gets GOAWAY + 30 s drain); claim path collides with a static route, /_molo, /healthz, /readyz, /startupz, /metrics, /csp-report or any reserved prefix → 409.
Success: H2 settings handshake completes; agent sends JSON hello on stream 1 (or first WS message on fallback transport): {"v":2,"claim":"...","upstreams":[{"upstream":"...","prefix":"..."}], "agent":"molo-cli/<ver>"}. The v1 single-upstream hello ({"v":1,"claim","upstream","agent"}) is accepted and treated as a one-entry table. Server answers {"v":2,"status":"ok","claim":"...","tunnel_id":"..."} or {"v":2,"status":"error","code":"...","message":"..."} then RST_STREAM / close with 4001-analog on error.

Heartbeat: H2 PING every 25 s, agent must ACK within 20 s or tunnel closes and the claim is released after the drain window (WS fallback: server ping / agent pong, same intervals, close code 1001).

Reconnect: agent backoff 1/2/4/8 s capped at 30 s, jitter ±20 %, holds the claim for 30 s after disconnect (late reconnect with same key + tunnel id reattaches, in-flight streams replay once, idempotency is upstream's job).

5. Proxying: one H2 stream per request, N upstreams per tunnel

No JSON-envelope multiplexing layer. Multiplexing IS HTTP/2 streams on the single tunnel connection (src/runtime/http/v2/connection.{h,cpp}, stream.{h,cpp}): each public request becomes one H2 stream head + DATA frames; flow control (connection + per-stream windows), HPACK, RST_STREAM (REFUSED_STREAM on overload), and GOAWAY are the H2 native mechanisms. The WS-fallback transport keeps the v1 envelope only for that fallback.

Public request arrives at claimed route → server strips the claim prefix (relative claims) or keeps full path (whole-host claims), copies method, query, headers, body (max per caps::kBodyMaxDefault = 10 MiB per request; larger → 413), assigns the next H2 stream id on the tunnel connection, forwards HEADERS + DATA frames. The agent routes the stream to the upstream entry with the longest matching prefix.
Agent performs the upstream fetch (plain TCP relay, HTTP request, or WS pass-through depending on the matched upstream scheme), streams response chunks back as H2 DATA frames on the same stream, finishes with END_STREAM (status, headers, trailers).
Hop-by-hop headers (Connection, Transfer-Encoding, Keep-Alive, Upgrade except WS pass-through) are stripped; X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, X-Molo-Tunnel-Id are appended.
Timeouts (from src/runtime/caps.h): header 10 s, body 30 s, idle keep-alive 75 s, handler budget 30 s, drain 30 s. Agent dial 10 s, upstream response header 30 s. Upstream unreachable → server answers 502; tunnel down → 503 with Retry-After: 5.
WebSocket public clients: server forwards the WS upgrade through the tunnel as an H2 extended-CONNECT stream (fallback transport: opaque bidirectional data envelopes, ping/pong preserved end to end); upstream plain-TCP mode tunnels raw bytes after the 101.
Concurrency per tunnel connection follows the H2 stack: caps::kH2MaxConcurrentStreamsDefault = 100 default, raisable to caps::kH2MaxConcurrentStreamsCeiling = 1000. At the cap the server sends RST_STREAM REFUSED_STREAM; the public side gets 503 + Retry-After. (V1 text "max 256" is superseded by these caps.)

6. Routing and precedence

Longest exact host+path match wins. Whole-host claim beats path claim on another host only when Host matches exactly. Two agents may hold overlapping claims only if neither is a prefix of the other at a path-segment boundary; otherwise second gets 409. Within one tunnel, longest upstream prefix wins; ties reject the hello with 400. Unclaimed paths fall through to normal routing (static, app routes, 404). Host matching is case-insensitive; trailing-dot FQDN normalized away.

7. Security

Keys: opaque 128-bit random, stored only as SHA-256 on the server, rotation via molo expose-key rotate --claim ... (old key drains 5 min). Constant-time compare. Failed auth rate-limited 10/min/IP, logged once.
TLS: public side uses the server's normal TLS config. Tunnel side SHOULD be h2-TLS/wss/https in production; cleartext h2c/ws/http allowed only with --allow-insecure (agent) / expose.allow_insecure=true (server) and a startup warning. MOLO_EXPOSE_KEY never echoed; --format json redacts it.
SSRF guard: upstream must resolve to loopback/RFC1918/link-local by default; public IPs need --allow-public-upstream (agent refuses without explicit flag). DNS re-resolved per stream; TTL honored. Applies per upstream table entry.
Abuse caps: 64 tunnels per server (configurable), 16 per key, 128 connections per public IP per route (shared with realtime caps), request body 10 MiB (§5), response streaming unbounded but handler-budget 30 s cap, claim label lengths per §3.

8. CLI surface

molo expose <upstream>... [key@server] [claim]   open (and hold) a tunnel
molo expose --list                               list own live tunnels (server query)
molo unexpose <claim>                            close one tunnel
molo expose-key rotate|revoke --claim <claim>    key management
molo stats [expose]                              literally all data the current
                                                 instance collected (see §9)

expose runs in foreground, prints claim -> upstream[,...] via tunnel <id>, reconnects until SIGINT/SIGTERM. Flags: --takeover, --allow-insecure, --allow-public-upstream, --name, --format json|text, -v. --upstream is repeatable; positional upstreams use [url=]prefix selectors.

9. `molo stats` contract

molo stats prints everything the running instance (server when pointed at one with --server, else the local agent) has collected:

tunnels: id, claim, upstream table (scheme per entry), key fingerprint (sha256[:16], never the key), connected_at, reconnects, streams active/total, bytes in/out, last error;
per-stream: id, tunnel, method/path or WS flag, matched upstream prefix, status, duration, bytes, upstream latency;
server rollups: uptime, tunnels live/peak, claims held, 502/503 counts, auth failures, heartbeat timeouts, ping p99;
agent view: upstream dial errors per table entry, reconnect count, backoff state.

Text default is human tables; --format json emits one parseable JSON document on stdout (logs on stderr). --watch <secs> refreshes in place. Exit 0 ok, 1 no data/server unreachable, 2 usage.

10. Observability

Log lines on open/close/takeover/auth-failure with claim + tunnel id + key fingerprint (never the key); per-upstream-entry stream counters. Prometheus counters molo_expose_tunnels_live, molo_expose_streams_total{status}, molo_expose_bytes_{in,out}_total, molo_expose_auth_failures_total, molo_expose_reconnects_total. Health endpoints unaffected; a held claim never shadows them (§4 rule 2).

11. V1 scope decisions (normative, no deferrals)

(see full text in prior revision; unchanged.)

12. OO design mapping (v0.2.0 implementation)

Spec concept	Code
Claim grammar (§3)	`Claim`, `ParseClaim` in `src/realtime/expose/claim.{h,cpp}`
Upstream entry + longest-prefix table, SSRF guard (§5–§7)	`UpstreamEntry`, `UpstreamTable`, `UpstreamHostAllowed` in `upstream.{h,cpp}`
Tunnel + claim registry: auth, caps, drain, routing (§4,§6,§7)	`Tunnel`, `ClaimRegistry` in `tunnel.{h,cpp}`
Hello v1/v2 compat (§4 rule 3)	`Hello`, `ParseHello` in `hello.{h,cpp}`
H2 transport glue: 1 conn, 1 stream/req (§5)	`TunnelProxy` helpers in `proxy.{h,cpp}` over `src/runtime/http/v2/` (`H2Connection::Run/StreamTask/WriterTask`, `H2Stream::OnData/OnRst`, `flow_control`)
Route caps (§7: 128 conn/IP)	`PerIpCounter` in `proxy.{h,cpp}`
H2 error mapping (§5 overload)	`kH2RefusedStream (0x7)`, `kH2Cancel (0x8)`; `REFUSED_STREAM` on overload (`stream.cpp:kRefusedStream`)
Timeouts/caps (§5)	`src/runtime/caps.h`: `kHeaderTimeout` 10 s, `kIdleKeepAliveTimeout` 75 s, `kHandlerBudget` 30 s, `kBodyMaxDefault` 10 MiB, `kH2InitialWindow` 65535
Heartbeat/reconnect (§4)	`kPingIntervalSec` 25, `kPingAckTimeoutSec` 20, `kClaimHoldSec` 30, `kTakeoverDrainSec` 30 in `tunnel.h`
CLI surface (§8)	`src/cli/cli/expose_cli.{hpp,cpp}`

H2 audit note: H2Connection owns the connection; each proxied request runs on its own stream (StreamTask), responses stream via WriterTask / FileBodyTask, backpressure via flow_control, overload refuses with RST_STREAM REFUSED_STREAM (kRefusedStream), takeover drains with GOAWAY + 30 s. No JSON-envelope multiplexing: multiplexing IS H2 streams.

13 KiB Raw Blame History