fix(runtime): cap H2 control-frame output to stop PING-flood DoS/crash
MOLO-H2-004 (CVE-2019-9511/9512/9515/9518 class): a client that floods PING frames without draining its socket crashed runtime_interop_server after ~100-150k frames. Root cause, confirmed with an ASan build, is a stack overflow, not heap growth: EventLoop::post()/the Task<T> continuation resume synchronously when work is already buffered, so H2Connection::ReadLoop's frame loop never returns to EventLoop::run_once()'s flat dispatch while a burst of already-read frames remains; each frame nests the native stack one level deeper (ReadLoop -> Ensure -> ReadLoop -> ...) until it overflows. ReadLoop now forces a co_await loop().yield() every caps::kH2FramesPerYield frames, unwinding the stack back to run_once() regardless of burst size. Separately, EnqueueControl/EnqueueData/CompleteResponse had no ceiling on out_queued_bytes_ for H2, unlike the H1 path's out_.size() check in runtime/http/connection.cpp. Added EnforceWriteCap(), using a new, tight caps::kH2MaxPendingControlBytes for control output (PING/SETTINGS acks, RST, WINDOW_UPDATE, GOAWAY) and the existing kWriteBufferCeiling for DATA/response output, same as H1. Because the kernel's own TCP send buffer keeps draining small control frames even while the peer never reads, out_queued_bytes_ alone can stay near zero through a flood, so also added a monotonic caps::kH2MaxControlFramesPerConnection rate cap in EnqueueControl as the mechanism that actually bounds this class of abuse; exceeding either cap drops the connection via Fatal(). audit/MOLO-H2-004 now PASSes repeatably: the connection is cut well before all 300k PINGs are accepted, and the server stays up with bounded RSS. All 134 runtime unit tests (including the H2 suite) still pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -45,11 +45,48 @@ inline constexpr std::size_t kBodyMaxCeiling = 100 * 1024 * 1024;
|
||||
inline constexpr std::size_t kReadChunkSize = 16384;
|
||||
inline constexpr std::size_t kWriteHighWatermark = 256 * 1024;
|
||||
inline constexpr std::size_t kWriteLowWatermark = 64 * 1024;
|
||||
// Hard floor on a connection's pending-write bytes: H1 checks out_.size()
|
||||
// against this in runtime/http/connection.cpp; H2's EnforceWriteCap()
|
||||
// (runtime/http/v2/connection.cpp) checks out_queued_bytes_ against it for
|
||||
// DATA (response bodies, legitimately large). Either side treats crossing
|
||||
// it as an abusive/unresponsive peer and drops the connection.
|
||||
inline constexpr std::size_t kWriteBufferCeiling = 16 * 1024 * 1024;
|
||||
// Same floor, but for H2 control-frame output only (PING/SETTINGS acks,
|
||||
// RST, WINDOW_UPDATE, GOAWAY): these never need more than a few bytes per
|
||||
// stream, so a peer that stops draining and leaves this much unread control
|
||||
// output queued (out_queued_bytes_ in EnforceWriteCap()) is flooding, not
|
||||
// legitimately slow. Kept far tighter than kWriteBufferCeiling on purpose;
|
||||
// see audit/MOLO-H2-004 (PING-flood DoS class, CVE-2019-9511/9512/9515/9518).
|
||||
inline constexpr std::size_t kH2MaxPendingControlBytes = 128 * 1024;
|
||||
// Per-connection rate cap on control frames the server has queued to send
|
||||
// (PING/SETTINGS acks, RST, WINDOW_UPDATE, GOAWAY), tracked as a monotonic
|
||||
// lifetime count in H2Connection::EnqueueControl, independent of
|
||||
// out_queued_bytes_/kH2MaxPendingControlBytes: the kernel's own TCP send
|
||||
// buffer (SO_SNDBUF, autotuned up to several MB on most hosts) happily
|
||||
// drains 17-byte PING acks indefinitely even while the peer never actually
|
||||
// reads, so out_queued_bytes_ alone can stay near zero through an entire
|
||||
// flood and never trip. This counter is what actually bounds a PING-flood
|
||||
// style DoS (audit/MOLO-H2-004, CVE-2019-9511/9512/9515/9518 class): far
|
||||
// beyond anything a legitimate long-lived connection generates (keepalive
|
||||
// PINGs, per-request WINDOW_UPDATEs/RSTs), but decisively below an abusive
|
||||
// flood.
|
||||
inline constexpr std::size_t kH2MaxControlFramesPerConnection = 20000;
|
||||
// 6.7 Static: bounded streaming chunk for file-backed response bodies.
|
||||
inline constexpr std::size_t kFileChunkSize = 64 * 1024;
|
||||
|
||||
// 6.4 HTTP/2.
|
||||
// Floor on how many frames ReadLoop (runtime/http/v2/connection.cpp) may
|
||||
// process back to back, already-buffered, before it must co_await a real
|
||||
// yield. EventLoop::post()/the Task<T> continuation resume synchronously
|
||||
// when work is already ready (see EventLoop::post, event_loop.cpp), so a
|
||||
// tight burst of already-buffered frames (e.g. a PING flood landing in one
|
||||
// read) never truly returns to EventLoop::run_once()'s flat dispatch loop;
|
||||
// each frame nests the native call stack one level deeper (ReadLoop ->
|
||||
// Ensure -> ReadLoop -> ...) until it overflows. Forcing a yield every
|
||||
// kH2FramesPerYield frames unwinds that stack back to run_once() regardless
|
||||
// of frame count. Confirmed by an ASan stack-overflow repro; see
|
||||
// audit/MOLO-H2-004.
|
||||
inline constexpr std::size_t kH2FramesPerYield = 64;
|
||||
inline constexpr std::size_t kH2InitialWindow = 65535;
|
||||
inline constexpr std::size_t kH2MaxFrameDefault = 16384;
|
||||
inline constexpr std::size_t kH2MaxFrameCeiling = 16777215;
|
||||
|
||||
@@ -39,6 +39,7 @@ constexpr uint32_t kFrameSizeError = 0x6;
|
||||
constexpr uint32_t kRefusedStream = 0x7;
|
||||
constexpr uint32_t kCancel = 0x8;
|
||||
constexpr uint32_t kCompressionError = 0x9;
|
||||
constexpr uint32_t kEnhanceYourCalm = 0xb;
|
||||
|
||||
bool IsH2HopHeader(std::string_view name) {
|
||||
return EqCaseInsensitive(name, "connection") ||
|
||||
@@ -84,6 +85,22 @@ std::string H2Connection::FrameBytes(uint8_t type, uint8_t flags,
|
||||
return out;
|
||||
}
|
||||
|
||||
void H2Connection::EnforceWriteCap(std::size_t ceiling) {
|
||||
// A non-draining peer (never reads its socket) must not be able to grow
|
||||
// out_queue_ without bound, whether the bytes came from PING acks or any
|
||||
// other control/data frame. Mirrors the H1 out_.size() > write_ceiling
|
||||
// check in runtime/http/connection.cpp: once a connection stops draining
|
||||
// past the ceiling for the kind of output just queued, treat it as
|
||||
// abusive and drop it. Control output (PING/SETTINGS acks, RST,
|
||||
// WINDOW_UPDATE, GOAWAY) uses the much tighter
|
||||
// caps::kH2MaxPendingControlBytes; DATA (response bodies, legitimately
|
||||
// large) uses caps::kWriteBufferCeiling.
|
||||
if (fatal_) return;
|
||||
if (out_queued_bytes_ > ceiling) {
|
||||
Fatal(kEnhanceYourCalm);
|
||||
}
|
||||
}
|
||||
|
||||
void H2Connection::EnqueueControl(uint32_t stream_id,
|
||||
const std::string& frame_bytes) {
|
||||
OutItem item;
|
||||
@@ -93,6 +110,13 @@ void H2Connection::EnqueueControl(uint32_t stream_id,
|
||||
item.end_stream = false;
|
||||
out_queued_bytes_ += item.bytes.size();
|
||||
out_queue_.push_back(std::move(item));
|
||||
EnforceWriteCap(caps::kH2MaxPendingControlBytes);
|
||||
// Rate cap: see caps::kH2MaxControlFramesPerConnection. Monotonic, so it
|
||||
// still bounds a flood even when the kernel send buffer keeps draining
|
||||
// out_queued_bytes_ back to near zero.
|
||||
if (!fatal_ && ++control_frames_sent_ > caps::kH2MaxControlFramesPerConnection) {
|
||||
Fatal(kEnhanceYourCalm);
|
||||
}
|
||||
KickWriter();
|
||||
}
|
||||
|
||||
@@ -105,6 +129,7 @@ void H2Connection::EnqueueData(uint32_t stream_id, std::string payload,
|
||||
item.end_stream = end_stream;
|
||||
out_queued_bytes_ += item.bytes.size();
|
||||
out_queue_.push_back(std::move(item));
|
||||
EnforceWriteCap(caps::kWriteBufferCeiling);
|
||||
KickWriter();
|
||||
}
|
||||
|
||||
@@ -261,6 +286,11 @@ Task<void> H2Connection::ReadLoop(bool first_must_be_settings) {
|
||||
auto self = shared_from_this();
|
||||
bool first = first_must_be_settings;
|
||||
std::string& in = conn_.inbox();
|
||||
// Stack-depth floor: see caps::kH2FramesPerYield. Without this, a burst
|
||||
// of frames already sitting in `in` (e.g. a PING flood) processes with no
|
||||
// real suspension, and each loop turn nests the native stack one level
|
||||
// deeper instead of returning to EventLoop::run_once()'s flat dispatch.
|
||||
std::size_t frames_since_yield = 0;
|
||||
for (;;) {
|
||||
if (conn_.closed() || fatal_) co_return;
|
||||
if ((peer_goaway_ || draining_) && streams_.empty()) co_return;
|
||||
@@ -296,6 +326,11 @@ Task<void> H2Connection::ReadLoop(bool first_must_be_settings) {
|
||||
}
|
||||
}
|
||||
if (!ProcessFrame(fr, payload.data())) co_return;
|
||||
if (++frames_since_yield >= caps::kH2FramesPerYield) {
|
||||
frames_since_yield = 0;
|
||||
co_await conn_.loop().yield();
|
||||
if (conn_.closed() || fatal_) co_return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -811,6 +846,7 @@ bool H2Connection::CompleteResponse(uint32_t stream_id, FullResponse resp,
|
||||
ctl.end_stream = no_body;
|
||||
out_queued_bytes_ += ctl.bytes.size();
|
||||
out_queue_.push_back(std::move(ctl));
|
||||
EnforceWriteCap(caps::kWriteBufferCeiling);
|
||||
if (no_body) {
|
||||
KickWriter();
|
||||
return false;
|
||||
|
||||
@@ -87,6 +87,13 @@ class H2Connection : public std::enable_shared_from_this<H2Connection> {
|
||||
|
||||
void EnqueueControl(uint32_t stream_id, const std::string& frame_bytes);
|
||||
void EnqueueData(uint32_t stream_id, std::string payload, bool end_stream);
|
||||
// Backpressure floor: a peer that stops draining (e.g. a PING flood that
|
||||
// never reads the socket) must not be allowed to grow out_queue_ without
|
||||
// bound. Checked after every append to out_queued_bytes_; trips Fatal()
|
||||
// once `ceiling` is crossed. Callers pass caps::kH2MaxPendingControlBytes
|
||||
// for control output or caps::kWriteBufferCeiling (shared with the H1
|
||||
// path in runtime/http/connection.cpp) for DATA/response output.
|
||||
void EnforceWriteCap(std::size_t ceiling);
|
||||
void KickWriter();
|
||||
void WakeWriter();
|
||||
void SendSettingsAck();
|
||||
@@ -127,6 +134,9 @@ class H2Connection : public std::enable_shared_from_this<H2Connection> {
|
||||
std::deque<OutItem> out_queue_;
|
||||
// Payload bytes currently queued; bounds file-body streaming.
|
||||
std::size_t out_queued_bytes_ = 0;
|
||||
// Lifetime count of control frames queued to send; see
|
||||
// caps::kH2MaxControlFramesPerConnection.
|
||||
std::size_t control_frames_sent_ = 0;
|
||||
void UnqueueBytes(std::size_t n) {
|
||||
out_queued_bytes_ -=
|
||||
n > out_queued_bytes_ ? out_queued_bytes_ : n;
|
||||
|
||||
Reference in New Issue
Block a user