OpenSSL HollowByte: How an 11-Byte Payload Causes Stealthy Memory Exhaustion and DoS
Executive Summary
The cybersecurity landscape has recently been alerted to a significant flaw within OpenSSL, dubbed "HollowByte." Disclosed by the Okta Red Team in mid-July 2026, this vulnerability demonstrates how remarkably small, incomplete TLS handshake fragments can disrupt web infrastructure. By sending a payload of just 11 bytes, attackers can trick the OpenSSL state machine into allocating disproportionately large memory buffers without ever completing a connection. This behavior results in severe heap fragmentation, system slowdowns, and eventual Denial of Service (DoS) Okta Security Research.
The mechanism allows for stealthy exhaustion of resources, leading to a state where memory is not released back to the operating system. Consequently, critical services face performance degradation as the Resident Set Size (RSS) of the process permanently bloats Security Affairs.
Deep-Dive Technical Analysis
The 11-Byte Mechanism
The HollowByte vulnerability exploits the way OpenSSL handles incoming TLS records. An attacker initiates a connection and sends an incomplete 11-byte TLS Client Hello record header. As the OpenSSL state machine parses these initial bytes, it reads the record length field. If this field indicates a large record, OpenSSL pre-allocates a buffer to accommodate the expected data—up to 131 KB per socket connection. By sending only the header and no subsequent data, the attacker forces the system to commit memory for a payload that never arrives.
Heap Fragmentation and Permanent RSS Bloat
The danger of HollowByte lies in its long-term impact on system memory. By repeatedly opening sockets, delivering the 11-byte header, and then abruptly terminating or abandoning the connections, an attacker induces extreme glibc heap fragmentation. Because standard memory allocators often struggle to consolidate or return fragmented blocks to the kernel, the Linux Resident Set Size (RSS) becomes permanently bloated. In observed cases, this fragmentation has led to up to a 25% drop in available system memory, as the allocations remain "stuck" within the process even after the connections are closed.
Evasion of Rate Limiting and DoS
HollowByte is particularly difficult to defend against because it bypasses traditional security perimeters. Most Web Application Firewalls (WAFs) and rate-limiters are tuned to detect volume-based attacks or malformed protocol sequences. However, an 11-byte transmission falls far below typical volume-based thresholds. Furthermore, because no full HTTP or TLS session is ever established, layer-7 inspection tools remain blind to the attack, as they typically wait for a completed handshake before performing deep packet analysis The Hacker News.
Silent Patch Releases and Vulnerability Management Challenges
OpenSSL addressed the HollowByte flaw by releasing maintenance updates across several branches, including versions 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21. These updates modify the buffer pre-allocation behavior to mitigate the exhaustion risk.
However, the nature of these releases has introduced operational friction. OpenSSL initially deployed these fixes as maintenance updates without assigning a formal CVE ID or issuing a high-severity alert. This "stealth patch" approach presents significant hurdles for enterprise security teams. Without a CVE identifier, automated vulnerability scanners may fail to flag the vulnerability, and security compliance teams may struggle to prioritize the necessary updates, as their workflows are often dependent on standardized vulnerability scores and identifiers Security Affairs.
Industry Impact and Actionable Mitigations
System Administrator Playbook
To defend against HollowByte, system administrators should implement the following technical controls:
Action Item
Technical Implementation
Immediate Upgrade
Transition all production environments to OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, or 3.0.21.
Timeout Configuration
Enforce strict TCP keep-alive and connection timeout rules. In NGINX, adjust client_header_timeout; in HAProxy, utilize timeout header.
Resource Monitoring
Track glibc memory allocation patterns and monitor for unusual RSS bloat metrics specifically on edge load balancers.
SOC and Incident Response Guidance
Security Operations Centers (SOC) should update their detection logic to identify the unique signature of this attack. Network anomaly rules should be configured to flag high volumes of incomplete TLS handshake attempts. Specifically, teams should monitor for socket connections where fewer than 20 bytes are transmitted before the connection is either abandoned or closed, as this is a primary indicator of the HollowByte exploitation attempt.