A recently disclosed vulnerability in OpenSSL, termed ‘HollowByte,’ enables attackers to execute denial-of-service (DoS) attacks using minimal TLS handshake requests. This flaw affects OpenSSL versions prior to the patched releases and can lead to significant memory exhaustion on servers.
In a standard TLS handshake, each message includes a 4-byte header, with three bytes indicating the message’s length. Older OpenSSL versions would allocate memory based on this declared length immediately upon receiving the header, without waiting for the actual message body. This behavior allows an attacker to send a small, crafted ClientHello message that claims a large size, causing the server to allocate up to 131 KB of memory per connection. If the attacker doesn’t send the corresponding message body, the server’s worker thread remains blocked, awaiting data that never arrives.
On systems using the GNU C Library (glibc), this issue is exacerbated. When the attacker drops the connection, OpenSSL releases the allocated buffer. However, glibc retains these freed memory chunks for reuse rather than returning them to the operating system. By varying the claimed sizes in each connection, an attacker can prevent the allocator from efficiently reusing memory, leading to heap fragmentation and a persistent increase in the server’s resident memory usage. In tests conducted by Okta’s Red Team, a server with 1 GB of RAM was terminated due to out-of-memory conditions after 547 MB of memory became fragmented and unusable. On a 16 GB server, the attack consumed 25% of system memory without exceeding the server’s connection limits, rendering standard connection-limiting defenses ineffective.
OpenSSL addressed this issue in June by releasing patched versions: 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21. Notably, these updates were issued without assigning a Common Vulnerabilities and Exposures (CVE) identifier, publishing an advisory, or including a changelog entry specific to this fix. The OpenSSL security team categorized the issue as a ‘bug or hardening’ fix, which, according to their policy, does not warrant a CVE or advisory. This decision has raised concerns within the security community, as the lack of formal documentation may hinder organizations from recognizing the necessity of applying these updates promptly.
Given the widespread use of OpenSSL in securing internet communications, it’s imperative for organizations to assess their systems and apply the relevant patches to mitigate the risk posed by the HollowByte vulnerability. The absence of a CVE or advisory underscores the importance of proactive monitoring and maintenance of security infrastructure to address such silent fixes.
The HollowByte vulnerability highlights the critical need for transparency in the disclosure and documentation of security flaws. While OpenSSL’s decision to treat this as a ‘bug or hardening’ fix may align with their internal policies, it potentially leaves many systems vulnerable due to the lack of formal advisories. Organizations must remain vigilant, regularly monitor for updates, and apply patches promptly, even in the absence of explicit vulnerability disclosures. This incident serves as a reminder of the complexities involved in managing open-source security and the importance of clear communication between maintainers and users.
