A fresh vulnerability in OpenSSL, dubbed “HollowByte,” has been uncovered that allows a remote, unauthenticated attacker to trigger a denial-of-service (DoS) simply by sending an 11-byte payload. This exploit abuses how certain OpenSSL versions pre-allocate memory during the TLS handshake based on a claimed message size—even before any real data arrives. Servers running the flawed versions reserve large chunks of RAM but never receive what was promised by the attacker, leaving workers hung indefinitely.
What’s Going Wrong
TLS handshakes begin when a client sends a ClientHello message wrapped in a record. That record includes a four-byte header: one byte for message type and three bytes declaring the length of the following message body. In older OpenSSL releases, once the header is read, the server immediately allocates a buffer sized to the attacker-declared length—even if none of the body has arrived. These versions trust the header without any early validation.
In one test, sending a mere 11-byte record caused OpenSSL to allocate up to 131 KB of memory, followed by the server blocking as it waited for a body that never showed up. This behavior demands just a minimal payload yet forces systems into excessive memory allocation cycles that can’t complete, straining both connection resources and server threads.
After the Attack: Locked Memory & Unrecoverable Usage
When the malicious connection finally closes, OpenSSL frees its buffer—but the system’s memory allocator (glibc, on Linux) doesn’t always return that RAM to the OS immediately. Instead, it keeps freed blocks around for reuse. Exploiters can send many connections claiming different oversized lengths so that these fragments don’t align evenly enough to be reusable. The result? Resident Set Size (RSS) climbs and stays high—even after the attack ends.
In lab tests, an unpatched server with only 1 GB of RAM was killed by the out-of-memory (OOM) killer after 547 MB of frozen, fragmented memory accumulated. On a 16 GB test server, around 25% of system memory became locked up, even with standard limits on connections. Typical connection throttling or rate-limits didn’t stop the problem.
The Broader Risk & Who’s Impacted
Almost any service using vulnerable OpenSSL versions is at risk—including web servers like Nginx and Apache, scripting runtimes such as Node.js, Python, PHP, and Ruby, and databases including MySQL and PostgreSQL. Because OpenSSL is so widely embedded, the vulnerability surface is large.
How It’s Fixed & What to Do Now
OpenSSL released updates that fix this issue by changing allocation behavior: instead of trusting length declared in headers immediately, the buffer is now grown in small increments as data actually arrives (typically in chunks no larger than one TLS record). This means a malicious claimed size no longer forces large upfront allocation. The fix landed in version 4.0.1, and has been backported silently to 3.6.3, 3.5.7, 3.4.6, and 3.0.21.
Organizations should upgrade to these or later versions wherever possible. Also, defenses in the application stack matter: set handshake and idle timeouts, limit how many concurrent connections can stall, enable non-blocking I/O, and rate-limit per source. These are standard protections against slow or stalled client attacks—regardless of the TLS library.
Why It Matters: HollowByte isn’t just another memory leak—it’s a reminder that even minimal payloads can trigger severe resource exhaustion when libraries trust unverified metadata. It underscores the importance of hardening in foundational components like TLS libraries. For operators, this patch isn’t optional. Going forward, watch how your allocator handles freed memory, and ensure that any service exposed to the internet has robust timeouts and connection hygiene. That’s what makes this fix more than just a patch—it’s a tightening of our digital defenses.
