Blog

The recent issues caused by a CrowdStrike update have led to significant disruptions across various sectors, including air travel, healthcare, and financial services. 

This incident highlights the critical balance between software security and system stability, as well as the potential consequences of software updates that are not thoroughly vetted.

Overview of the Incident

According to Forbes, on July 19, 2024, a faulty update from CrowdStrike, specifically affecting its Falcon Sensor product, triggered widespread computer malfunctions, primarily on Windows systems. The update led to a series of crashes known as the “Blue Screen of Death,” causing systems to enter a reboot loop. 

This malfunction affected numerous organizations globally, including airlines, hospitals, and media outlets, resulting in grounded flights, disrupted operations, and significant operational challenges.

Key Points of Failure

Defective Kernel Driver

The root cause of the issue was identified as a defect in a kernel driver released by CrowdStrike. 

This driver is integral to the functioning of the Falcon Sensor, which operates at a deep system level to monitor and protect against cyber threats 

The defective update initiated a chain reaction of reboots, affecting critical infrastructure worldwide.

Impact on Various Sectors

The incident had a profound impact across various sectors, demonstrating the vulnerabilities of interconnected digital infrastructure. Here’s a detailed look at how different industries were affected:

Aviation

The airline industry faced significant disruptions due to the CrowdStrike outage. Thousands of flights were canceled, with major carriers like Delta reporting over 600 cancellations alone. Passengers experienced extensive delays and confusion as systems went offline, leading to a chaotic travel environment globally.

Healthcare

Healthcare facilities encountered serious operational challenges. Hospitals struggled to access critical patient data, resulting in delays for medical procedures, including surgeries. While some hospitals, like Pittsburgh’s UPMC, reported minimal disruptions, others faced significant hurdles that impacted patient care.

Broadcasting

Major broadcasting networks experienced technical difficulties, causing interruptions in news coverage. Outlets such as NBC, MSNBC, and Sky News had to rely on backup systems to continue broadcasting, affecting the flow of information to the public during a critical time.

Financial Services

The financial sector also felt the repercussions, with some stock markets temporarily halting trading. The London Stock Exchange experienced disruptions, although others like the New York Stock Exchange continued operations. This incident highlighted the fragility of financial systems reliant on interconnected technologies.

Government and Corporate Sectors

The outage impacted numerous government agencies and Fortune 500 companies that rely on CrowdStrike’s cybersecurity solutions. With over half of these companies affected, the incident raised concerns about the resilience of critical infrastructure and the concentration of risk in the cybersecurity industry.

Response from CrowdStrike 

Following the incident, CrowdStrike’s CEO, George Kurtz, acknowledged the issue and emphasized that it was not the result of a cyberattack. The company quickly mobilized its engineering team to address the defect and released a corrective update. However, many systems required manual intervention to apply the fix, complicating recovery efforts for IT teams.

Mitigation and Recovery Efforts

CrowdStrike provided a series of workaround steps for affected users, which included booting systems in Safe Mode, deleting the problematic driver file, and ensuring systems were connected via wired networks for better stability during recovery. For environments like virtual servers, additional steps were outlined to detach and manage disk volumes.

Communication and Support

The incident underscored the necessity for effective communication from CrowdStrike to its customers. Cybersecurity professionals emphasized the importance of clear guidance on applying fixes and the need to halt further updates until the situation was fully resolved. Organizations were encouraged to consider recovery options such as restoring from backups or using bootable media tools to expedite recovery processes.

Possible Causes

Inadequate quality checks before deployment

The update appears to have skipped important testing and vetting procedures before being rolled out to customers. As Patrick Wardle, a security researcher, noted: “Ideally, this would have been rolled out to a limited pool first. That is a safer approach to avoid a big mess like this.

Frequency of updates

CrowdStrike frequently updates its Falcon Sensor software to detect new threats. However, this rapid update cadence may have led to less rigorous testing. As Wardle explained: “It’s very common that security products update their signatures, like once a day… The frequency of updates is probably the reason why (CrowdStrike) didn’t test it as much.”

Defective code in the update

The root cause was traced back to a specific file in the CrowdStrike update that contained either configuration information or malware signatures. This file had a flaw that caused Windows systems to crash when the update was applied.

Troubleshooting Steps

Identify the Issue

Determine the extent and nature of the issue by checking affected systems and gathering error messages or logs.

Check CrowdStrike Status

Verify that all CrowdStrike services are running as expected. Use the CrowdStrike console to check for any alerts or errors.

Update CrowdStrike

Ensure that CrowdStrike is updated to the latest version, as updates often include bug fixes and performance improvements.

Disable/Enable CrowdStrike Temporarily

Temporarily disable CrowdStrike to see if the issue persists. If disabling resolves the issue, it is likely related to CrowdStrike.

Check for Conflicts

Identify and resolve any conflicts with other security software or system management tools. Disable other security software temporarily to test for conflicts.

Review System Logs

Examine system and application logs for relevant error messages or warnings that could indicate the cause of the issue.

Contact CrowdStrike Support

If the problem persists, reach out to CrowdStrike support for professional assistance and provide them with detailed information about the issue.

Rollback Updates

Consider rolling back to a previous version of CrowdStrike if the issue began after a recent update.

Network Configuration

 Ensure that network settings and configurations are not blocking CrowdStrike. Check firewall settings and network policies to allow CrowdStrike traffic.

System Restore

As a last resort, perform a system restore to revert the system to a point before the issue started.

Integrating Better Cybersecurity Services

To address and prevent such incidents in the future, organizations should consider integrating robust cybersecurity services such as offshore software testing services and platform testing services. These services can provide thorough testing and validation before deployment, ensuring updates do not cause system failures. Offshore QA testing services and offshore software testing companies can offer dedicated teams for continuous testing and monitoring, reducing the likelihood of unvetted updates reaching production environments.

Agile testing service providers can help maintain a balance between rapid updates and rigorous testing, ensuring that frequent updates do not compromise system stability. 

Additionally, automated software testing services from reputable automation testing companies can streamline the testing process, identifying potential issues quickly and efficiently.

Conclusion

While CrowdStrike remains a vital tool in the cybersecurity arsenal, recent reports of system issues highlight the importance of regular maintenance, proper configuration, and prompt troubleshooting. 

By understanding the potential causes and following the recommended steps, organizations can mitigate the impact of such issues and maintain the integrity of their cybersecurity defenses.