The Crowdstrike Incident
Summary
In July 2024, a faulty software update from CrowdStrike, a leading cybersecurity firm, caused widespread system crashes on millions of Windows computers globally. This incident disrupted critical sectors such as transportation, healthcare, and finance, highlighting the vulnerabilities inherent in modern IT systems. This article gives us explanations about the technical backgrounds which are leading finally to the incident.
What is CrowdStrike?
CrowdStrike is the name of a publicly traded US company that provides information security and cybersecurity solutions. The company was founded in 2011 and has also assisted in investigating various cases of economic cyber espionage, including on behalf of the United States Department of Justice. Google is one of the main investors, and the company now has a market capitalization of about 65 billion US dollars. In 2013, the software product CrowdStrike Falcon was launched. CrowdStrike Falcon is a widely used Enterprise Detection and Response (EDR) protection software against cyber attacks. The product uses AI and machine learning technologies, among others, to ensure the protection of computer systems.
The Incident
On July 19, 2024, CrowdStrike released a update — Channel File 291 — for its Falcon Sensor software. This update introduced an additional input parameter (21 instead of the expected 20), which caused an out-of-bounds memory access. The error led to system crashes characterized by the “Blue Screen of Death” (BSOD) on affected Windows systems.
Technical Background
CrowdStrike Falcon
Like any software product, CrowdStrike Falcon requires maintenance. The central security scanner agent is called “CrowdStrike Falcon Sensor” by CrowdStrike. This needs constant updates to be able to respond appropriately to current threat situations and security vulnerabilities. During operation, software updates are regularly rolled out using so-called Channel files. CrowdStrike distributes dynamic updates and detection rules using the Channel files.
Chronology
On July 19, 2024, numerous system failures were detected worldwide. It was quickly determined that the affected systems were using CrowdStrike Falcon as protection software. However, this incident was not a classic cyberattack. Rather, this error occurred in connection with a regular standard procedure - the software update process.
In February 2024, a new feature was rolled out for the “Falcon Sensor”. This new function was intended to improve the detection of threats and attacks using Windows mechanisms. As part of internal stress tests, Rapid Response Updates (designation for special updates for Falcon) were tested. No abnormalities were detected here. The tests took place in March and April 2024.
On July 19, 2024, the Rapid Response Content Updates were then deployed to certain Windows hosts. Specifically, this was Channel File 291. During the root cause analysis, it was determined that the sensor expected 20 input fields, but the update provided 21 input fields. This led to a memory access outside the specified memory addresses, which ultimately resulted in a system crash - the Blue Screen of Death (BSOD).
Software Architecture of Falcon
We now want to discuss the technical background of the Rapid Response Updates. For further understanding, here is an explanation of how the Falcon Sensor software works. The CrowdStrike Falcon Sensor is an advanced security system that uses artificial intelligence and machine learning to detect and defend against malicious activities on computers. It continuously collects data from protected devices and analyzes it to identify suspicious behaviors. Both known and new attack patterns are detected. To stay up-to-date, the sensor’s protection mechanisms are regularly updated and improved through insights from the analysis of real threats.
An important component of the sensor is the so-called “Rapid Response Content”, which is deployed in the form of ChannelFile updates. These are flexible rules that allow the system to quickly respond to new threats without requiring manual adjustment of the system. These rules are regularly updated and enable the sensor to adapt to constantly changing threat landscapes.
The new function rolled out in February 2024 - “IPC Template Type” - is intended to enable the analysis of certain Windows interprocess communications (“IPC”). IPC Template instances were deployed as Rapid Response Content under Channel File 291 as intended. In that - now known as faulty update Channel File - 21 input parameters were defined, whereas the sensor software expected 20. The error was not detected during the test routines because wildcards were used for the 21st input here.
What happened specifically on July 19? Two additional IPC Template instances were rolled out. One of these had a non-wildcard criterion as the 21st input parameter. In previous versions, the 21st input parameter was not used and was not intended. Sensors that now received the new version performed a comparison between the input parameters. The system interpreter expected 20 but received 21. The attempted access to the 21st value caused an out-of-bounds memory write access outside the data field and resulted in a system crash.
In this context, it’s important to understand why an out-of-bounds error led to a complete system crash. The reason lies in the system architecture of the Falcon Agent. It doesn’t run as a separately isolated process, but instead accesses the requested memory resources directly through the Windows kernel as a kernel process. From a technical perspective, Falcon uses a device driver that operates in kernel mode - also known as “Ring Zero”. This allows Falcon full access to operating system resources such as Windows components from Microsoft itself. Falcon has an integrated quarantine function. This function inspects suspicious and potentially problematic files in an isolated area and makes them inaccessible to the operating system.
However, in this case, this function led to a race condition - the faulty file was loaded at startup because it was a software-specific component. As a result, the agent essentially overtook itself, leading to a Blue Screen of Death (BSOD).
Political Background
Why was such a malfunction even possible? Direct kernel access is normally not intended for third-party software. The fact that this access was possible stems from a European Commission regulation from 2009. This regulation mandated access to the operating system kernel for third-party software. The background of this decision was Microsoft’s market power and, in this context, to ensure the competitiveness of other companies.
Critics argued that Microsoft took the easy way out here. Apple’s MacOS operating system was not affected because an API was implemented instead of a direct access method. Through this API, third-party providers can use kernel functions. Microsoft could now use this incident as a political issue to argue against such decisions. On the other hand, critics see a need for action from Microsoft. They could also improve security in their own system through better implementations.
The incident clearly illustrates the tension between ensuring fair competition and the need to guarantee the security and stability of critical systems.
Impact
Approximately 8.5 million systems were affected globally. Key disruptions included:
- Airports: Flight delays and cancellations.
- Healthcare: Hospital systems were disrupted, delaying critical operations.
- Financial Services: ATM networks and payment systems experienced outages.
- Media: Sky News temporarily ceased broadcasting.
Linux and macOS systems were unaffected due to their different security architectures.
Response and Mitigation
- Manual Recovery: Affected systems required booting into safe mode to delete the faulty file manually.
- Patch Deployment: CrowdStrike released a corrected update within an hour of identifying the issue.
- Lessons Learned: The incident underscored the need for rigorous testing protocols and robust emergency response plans.