moloapp/reverse.md

# Reverse proxy / expose specification (reverse tunnels)

Part: reverse tunnel server + `molo expose` client. Owned: `src/realtime/` (tunnel protocol) + `src/cli/` (expose CLI). Plan: `plans/06-realtime.md` § tunnel. Version: 0.2.0.

## 1. Purpose

Any application (in a container or elsewhere) can publish one or more of its
local endpoints through the MoloApp server so the public internet reaches
them via claimed public routes. Transport between the local apps
("exposer") and the server is always one outbound connection initiated by
the exposer; inbound TCP to the exposer is never required. The server then
reverse-proxies public HTTP traffic arriving at the claimed routes into
that tunnel. One tunnel serves many concurrent upstreams: each public
request is routed inside the tunnel to the matching upstream entry.

This system is HTTP/2 (RFC 9113, `src/runtime/http/v2/`). The tunnel rides
the same H2 stack: one H2 connection carries many concurrent streams, one
stream per proxied request. There is no HTTP/1-only or single-flight
assumption anywhere in this spec.

## 2. Roles and terms

- **Server**: the MoloApp instance accepting tunnels and public traffic.
- **Exposer / agent**: the `molo expose` process running near the local apps.
- **Upstream**: one local origin the agent forwards to. One of:
  `host:port` (plain TCP), `http(s)://...` URL, `ws(s)://...` URL.
  A tunnel holds an upstream **table**: `{path_prefix -> upstream}` plus a
  default entry. Many upstreams per tunnel, all concurrent.
- **Claim**: a public route reservation. Forms:
  1. relative path only, e.g. `/hook1` — served on all server domains;
  2. `domain + path`, e.g. `example.com/hook1` — served on that host only;
  3. bare domain, e.g. `example.com` or `example.com/` — whole-host claim.
- **Key**: per-claim pre-shared token authorizing the agent.

## 3. Claim syntax (CLI + server share one grammar)

```
molo expose <upstream>... [<key>@<server>] [<claim>]
molo expose --upstream <upstream> [--upstream <upstream>...] --server <server> --claim <claim> --key <key>
```

- `<upstream>`: `[scheme://]host[:port][/path]` where scheme is
  `tcp | http | https | ws | wss`. No scheme defaults to `http`.
  Examples: `8080`, `127.0.0.1:3000`, `http://127.0.0.1:3000`,
  `wss://127.0.0.1:8443/socket`.
  Repeatable. Each entry may carry a sub-path selector:
  `<upstream>=<subpath>`, e.g. `http://127.0.0.1:3001=/api`,
  `http://127.0.0.1:3002=/static`. Entries without a selector share the
  default entry; longest sub-path prefix wins per request.
- `<server>`: server base URL, `http(s)://host[:port]` or `ws(s)://host[:port]`.
  Path prefix other than `/` is rejected (exit 2). Default: the server the
  CLI is configured against (dev default `http://127.0.0.1:8080`).
- `[key@]<server>`: when `key@` prefix is present it is the claim key;
  otherwise `--key` flag or `MOLO_EXPOSE_KEY` env supplies it.
- `<claim>` ("reverse proxy url" in shorthand): `[/path]` or `[host][/path]`.
  Host part never contains a scheme or port. Path must start with `/`
  (bare host implies `/`). Normalization: lowercase host, collapse duplicate
  slashes, strip trailing slash except root, reject `..`, `%2e`, backslash,
  NUL. Max 256 chars host, 1024 chars path.

Examples:

```
molo expose 8080 mykey@https://srv.example.com /hook1
molo expose http://127.0.0.1:3000 --server https://srv.example.com --claim acme.example.com/ --key mykey
molo expose http://127.0.0.1:3001=/api http://127.0.0.1:3002=/ mykey@https://srv.example.com /app
```

## 4. Tunnel handshake

1. Agent dials the server's H2 endpoint and opens the tunnel as one H2
   connection (h2c upgrade or prior-knowledge/h2-TLS per server config).
   The legacy single-WebSocket form (`GET /__molo/expose` with
   `Upgrade: websocket`, headers `X-Molo-Claim` + `Authorization: Bearer`,
   subprotocol `molo-expose-v1`) remains accepted as a fallback transport,
   but the normative transport is H2: one connection, many streams.
   Query form `?claim=&key=` is accepted for non-CLI agents and logged as
   deprecated.
2. Server validates: unknown/expired key → `403`; malformed claim → `400`;
   claim already held and live → `409` (unless `--takeover` with a newer
   key generation, then old tunnel gets GOAWAY + 30 s drain);
   claim path collides with a static route, `/_molo`, `/healthz`, `/readyz`,
   `/startupz`, `/metrics`, `/csp-report` or any reserved prefix → `409`.
3. Success: H2 settings handshake completes; agent sends JSON `hello` on
   stream 1 (or first WS message on fallback transport):
   `{"v":2,"claim":"...","upstreams":[{"upstream":"...","prefix":"..."}],
   "agent":"molo-cli/<ver>"}`.
   The v1 single-upstream hello (`{"v":1,"claim","upstream","agent"}`)
   is accepted and treated as a one-entry table.
   Server answers `{"v":2,"status":"ok","claim":"...","tunnel_id":"..."}`
   or `{"v":2,"status":"error","code":"...","message":"..."}` then
   RST_STREAM / close with 4001-analog on error.

Heartbeat: H2 PING every 25 s, agent must ACK within 20 s or tunnel
closes and the claim is released after the drain window (WS fallback:
server ping / agent pong, same intervals, close code 1001).

Reconnect: agent backoff 1/2/4/8 s capped at 30 s, jitter ±20 %, holds the
claim for 30 s after disconnect (late reconnect with same key + tunnel id
reattaches, in-flight streams replay once, idempotency is upstream's job).

## 5. Proxying: one H2 stream per request, N upstreams per tunnel

No JSON-envelope multiplexing layer. Multiplexing IS HTTP/2 streams on the
single tunnel connection (`src/runtime/http/v2/connection.{h,cpp}`,
`stream.{h,cpp}`): each public request becomes one H2 stream head +
DATA frames; flow control (connection + per-stream windows), HPACK,
RST_STREAM (`REFUSED_STREAM` on overload), and GOAWAY are the H2
native mechanisms. The WS-fallback transport keeps the v1 envelope only
for that fallback.

- Public request arrives at claimed route → server strips the claim prefix
  (relative claims) or keeps full path (whole-host claims), copies method,
  query, headers, body (max per `caps::kBodyMaxDefault` = 10 MiB per
  request; larger → `413`), assigns the next H2 stream id on the tunnel
  connection, forwards HEADERS + DATA frames. The agent routes the stream
  to the upstream entry with the longest matching `prefix`.
- Agent performs the upstream fetch (plain TCP relay, HTTP request, or WS
  pass-through depending on the matched upstream scheme), streams response
  chunks back as H2 DATA frames on the same stream, finishes with
  END_STREAM (status, headers, trailers).
- Hop-by-hop headers (`Connection`, `Transfer-Encoding`, `Keep-Alive`,
  `Upgrade` except WS pass-through) are stripped; `X-Forwarded-For`,
  `X-Forwarded-Host`, `X-Forwarded-Proto`, `X-Molo-Tunnel-Id` are appended.
- Timeouts (from `src/runtime/caps.h`): header 10 s, body 30 s, idle
  keep-alive 75 s, handler budget 30 s, drain 30 s. Agent dial 10 s,
  upstream response header 30 s. Upstream unreachable → server answers
  `502`; tunnel down → `503` with `Retry-After: 5`.
- WebSocket public clients: server forwards the WS upgrade through the
  tunnel as an H2 extended-CONNECT stream (fallback transport: opaque
  bidirectional `data` envelopes, ping/pong preserved end to end);
  upstream plain-TCP mode tunnels raw bytes after the 101.
- Concurrency per tunnel connection follows the H2 stack:
  `caps::kH2MaxConcurrentStreamsDefault` = 100 default, raisable to
  `caps::kH2MaxConcurrentStreamsCeiling` = 1000. At the cap the server
  sends RST_STREAM `REFUSED_STREAM`; the public side gets `503` +
  `Retry-After`. (V1 text "max 256" is superseded by these caps.)

## 6. Routing and precedence

Longest exact host+path match wins. Whole-host claim beats path claim on
another host only when `Host` matches exactly. Two agents may hold
overlapping claims only if neither is a prefix of the other at a path-segment
boundary; otherwise second gets `409`. Within one tunnel, longest upstream
`prefix` wins; ties reject the hello with `400`. Unclaimed paths fall
through to normal routing (static, app routes, 404). `Host` matching is
case-insensitive; trailing-dot FQDN normalized away.

## 7. Security

- Keys: opaque 128-bit random, stored only as SHA-256 on the server,
  rotation via `molo expose-key rotate --claim ...` (old key drains 5 min).
  Constant-time compare. Failed auth rate-limited 10/min/IP, logged once.
- TLS: public side uses the server's normal TLS config. Tunnel side SHOULD
  be h2-TLS/`wss`/`https` in production; cleartext h2c/`ws`/`http` allowed
  only with `--allow-insecure` (agent) / `expose.allow_insecure=true`
  (server) and a startup warning. `MOLO_EXPOSE_KEY` never echoed;
  `--format json` redacts it.
- SSRF guard: upstream must resolve to loopback/RFC1918/link-local by
  default; public IPs need `--allow-public-upstream` (agent refuses without
  explicit flag). DNS re-resolved per stream; TTL honored. Applies per
  upstream table entry.
- Abuse caps: 64 tunnels per server (configurable), 16 per key, 128
  connections per public IP per route (shared with realtime caps), request
  body 10 MiB (§5), response streaming unbounded but handler-budget 30 s
  cap, claim label lengths per §3.

## 8. CLI surface

```
molo expose <upstream>... [key@server] [claim]   open (and hold) a tunnel
molo expose --list                               list own live tunnels (server query)
molo unexpose <claim>                            close one tunnel
molo expose-key rotate|revoke --claim <claim>    key management
molo stats [expose]                              literally all data the current
                                                 instance collected (see §9)
```

`expose` runs in foreground, prints `claim -> upstream[,...] via tunnel <id>`,
reconnects until SIGINT/SIGTERM. Flags: `--takeover`, `--allow-insecure`,
`--allow-public-upstream`, `--name`, `--format json|text`, `-v`.
`--upstream` is repeatable; positional upstreams use `[url=]prefix` selectors.

## 9. `molo stats` contract

`molo stats` prints everything the running instance (server when pointed at
one with `--server`, else the local agent) has collected:

- tunnels: id, claim, upstream table (scheme per entry), key fingerprint
  (sha256[:16], never the key), connected_at, reconnects, streams
  active/total, bytes in/out, last error;
- per-stream: id, tunnel, method/path or WS flag, matched upstream prefix,
  status, duration, bytes, upstream latency;
- server rollups: uptime, tunnels live/peak, claims held, 502/503 counts,
  auth failures, heartbeat timeouts, ping p99;
- agent view: upstream dial errors per table entry, reconnect count,
  backoff state.

Text default is human tables; `--format json` emits one parseable JSON
document on stdout (logs on stderr). `--watch <secs>` refreshes in place.
Exit 0 ok, 1 no data/server unreachable, 2 usage.

## 10. Observability

Log lines on open/close/takeover/auth-failure with claim + tunnel id +
key fingerprint (never the key); per-upstream-entry stream counters.
Prometheus counters
`molo_expose_tunnels_live`, `molo_expose_streams_total{status}`,
`molo_expose_bytes_{in,out}_total`, `molo_expose_auth_failures_total`,
`molo_expose_reconnects_total`. Health endpoints unaffected; a held claim
never shadows them (§4 rule 2).

## 11. V1 scope decisions (normative, no deferrals)
(see full text in prior revision; unchanged.)

## 12. OO design mapping (v0.2.0 implementation)

| Spec concept | Code |
|---|---|
| Claim grammar (§3) | `Claim`, `ParseClaim` in `src/realtime/expose/claim.{h,cpp}` |
| Upstream entry + longest-prefix table, SSRF guard (§5–§7) | `UpstreamEntry`, `UpstreamTable`, `UpstreamHostAllowed` in `upstream.{h,cpp}` |
| Tunnel + claim registry: auth, caps, drain, routing (§4,§6,§7) | `Tunnel`, `ClaimRegistry` in `tunnel.{h,cpp}` |
| Hello v1/v2 compat (§4 rule 3) | `Hello`, `ParseHello` in `hello.{h,cpp}` |
| H2 transport glue: 1 conn, 1 stream/req (§5) | `TunnelProxy` helpers in `proxy.{h,cpp}` over `src/runtime/http/v2/` (`H2Connection::Run/StreamTask/WriterTask`, `H2Stream::OnData/OnRst`, `flow_control`) |
| Route caps (§7: 128 conn/IP) | `PerIpCounter` in `proxy.{h,cpp}` |
| H2 error mapping (§5 overload) | `kH2RefusedStream (0x7)`, `kH2Cancel (0x8)`; `REFUSED_STREAM` on overload (`stream.cpp:kRefusedStream`) |
| Timeouts/caps (§5) | `src/runtime/caps.h`: `kHeaderTimeout` 10 s, `kIdleKeepAliveTimeout` 75 s, `kHandlerBudget` 30 s, `kBodyMaxDefault` 10 MiB, `kH2InitialWindow` 65535 |
| Heartbeat/reconnect (§4) | `kPingIntervalSec` 25, `kPingAckTimeoutSec` 20, `kClaimHoldSec` 30, `kTakeoverDrainSec` 30 in `tunnel.h` |
| CLI surface (§8) | `src/cli/cli/expose_cli.{hpp,cpp}` |

H2 audit note: `H2Connection` owns the connection; each proxied request
runs on its own stream (`StreamTask`), responses stream via `WriterTask` /
`FileBodyTask`, backpressure via `flow_control`, overload refuses with
`RST_STREAM REFUSED_STREAM` (`kRefusedStream`), takeover drains with
GOAWAY + 30 s. No JSON-envelope multiplexing: multiplexing IS H2 streams.