OpenSSL has quietly fixed and backported a denial-of-service flaw that researchers are calling HollowByte: a single 11-byte TLS handshake message, repeated across enough connections, is enough to force a server to allocate memory it will never use. No CVE identifier was assigned, but the fix landed across every actively maintained OpenSSL branch, which is itself a signal of how broadly the underlying pattern applies.
What Happened
Every TLS handshake message starts with a 4-byte header, three bytes of which declare the length of the handshake data that follows. In vulnerable OpenSSL versions, the library trusted that length claim immediately: it allocated a buffer sized to the declared length before it had received the corresponding payload or validated that the payload would ever arrive.
Okta’s red team, which is credited with the finding, describes the exploit as trivially cheap to run: open a TLS connection, send an 11-byte message whose header claims a payload of, say, several megabytes, and then simply stop sending data. The server-side worker thread allocates the claimed buffer up front and then blocks, waiting indefinitely for bytes that are never coming. Repeat that across many concurrent connections and the memory adds up fast relative to the bandwidth spent — a classic asymmetric DoS, where the attacker’s cost is a fraction of the defender’s.
Okta’s tests against NGINX built on a vulnerable OpenSSL found that low-capacity or resource-constrained deployments could be driven to memory exhaustion outright, while higher-spec servers still lost up to roughly 25% of available memory under the attack — all while the actual network bandwidth consumed stayed low enough to slide under typical volumetric DDoS alerting thresholds. That last point is what makes HollowByte more than a footnote: it doesn’t look like a flood on the wire, so bandwidth- and packet-rate-based detections won’t catch it.
Why It Matters
OpenSSL isn’t a niche dependency — it’s embedded in NGINX and Apache, in language runtimes (Node.js, Python, Ruby, PHP all commonly link against it or a fork), in MySQL and PostgreSQL’s TLS layers, and it ships as the default TLS/certificate stack on most Linux distributions. Anything terminating TLS with a vulnerable OpenSSL — public-facing web servers, API gateways, load balancers, mail servers, internal service-mesh sidecars — is a candidate target. The attack requires no authentication and no valid TLS session to complete; it only needs the handshake to begin.
Because the exhaustion happens per-connection and scales with connection count rather than raw throughput, an attacker doesn’t need botnet-scale bandwidth to have an effect — a modest number of hosts opening many concurrent connections against a resource-constrained target (small VPS instances, edge nodes, IoT-adjacent appliances) can be sufficient.
Affected Versions and Fix
The issue is fixed in OpenSSL 4.0.1, with backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21. The fix changes the allocation strategy so the receive buffer grows incrementally as data actually arrives, instead of being pre-sized to whatever the untrusted header claims. Versions prior to these backport targets on each branch remain vulnerable.
No CVE was assigned to HollowByte — OpenSSL classified and shipped the fix without the usual public advisory workflow, which means it won’t show up in CVE-driven vulnerability scanners or KEV-style tracking. Inventory and patch management processes that gate purely on CVE feeds will miss this one; you need to track OpenSSL’s own release notes and changelog directly.
Mitigation
- Upgrade OpenSSL to 4.0.1, or the appropriate backport (3.6.3, 3.5.7, 3.4.6, 3.0.21) for whichever 3.x branch you run. Check both your OS package manager’s OpenSSL package and any statically-linked or vendored copies bundled inside containers, appliances, or language runtime distributions — those often lag behind distro patches.
- Rebuild and redeploy container images that bundle OpenSSL rather than relying on the host’s shared library; a patched host OpenSSL does nothing for an application container with its own static or vendored copy.
- Apply connection- and memory-based rate limiting in front of TLS-terminating services (per-source-IP concurrent connection caps, worker memory ceilings) as a stopgap on systems that can’t patch immediately — this won’t stop the flaw but bounds the blast radius of a single attacker.
- Don’t rely on bandwidth-based DDoS detection alone for this class of attack; monitor server-side memory and connection-table growth directly, since the wire-level traffic volume stays deceptively low.
- Audit embedded/OEM devices — network appliances, IoT management interfaces, and any product that vendors OpenSSL internally are often slow to rebuild against upstream fixes; check vendor advisories rather than assuming inherited patches.