On July 19, 2024, CrowdStrike–the cybersecurity company behind a widely-used endpoint protection platform– released a faulty update that caused a widespread outage of Microsoft Windows devices around the world. In this article, you will learn more about what happened in the CrowdStrike incident and what Microserve learned from remediating the incident for our clients.
What Happened During the CrowdStrike Incident?
CrowdStrike released an update to the Rapid Response Content sensor for Falcon, CrowdStrike’s platform that uses a “unified set of cloud-delivered technologies that prevent all types of attacks—including malware and much more.” According to the Incident Root Cause Analysis, “Template Instances are delivered as Rapid Response Content to sensors via a corresponding Channel File numbered 291.”
The CrowdStrike incident was caused by a new inter-process communication (IPC) Template Type that was deployed with the update. This IPC Template Type expected 21 input values to process the Rapid Response Content via Channel 291, however, this was unprecedented. Previous Template Types required only 20 input values. So, in production, the mismatched input values caused Windows devices to reboot and crash, leaving Windows devices with a blue screen of death.
As a result, IT managers scrambled to remediate the incident and get devices up and running. Following the CrowdStrike incident, Microsoft released a Microsoft Recovery Tool, which can be delivered via USB or PXE network.
What Did We Learn from the CrowdStrike Incident?
It’s important to understand cybersecurity incidents, like the CrowdStrike incident, to improve procedures and strengthen IT security posture against evolving threats. Microserve worked tirelessly with its clients to remediate the CrowdStrike issue and is happy to share the lessons learned from the CrowdStrike incident.
Be prepared: Cybersecurity incidents can happen at any time
The timeline of the CrowdStrike incident teaches us our first lesson: be prepared to remediate an incident at any time.
- On July 19, at 04:09 UTC (9:09PM PST), CrowdStrike released the faulty update
- On July 19, at 05:27 UTC (10:27PM PST), CrowdStrike reverted the update
- On July 19, at 06:48 UTC (11:48PM PST), Google Compute Engine reported the issue
- On July 19, at 23:37 UTC (July 20, 4:37PM PST), Microsoft released the Microsoft Recovery Tool
Cybersecurity incidents can happen at any hour of the day. Updates typically take place outside normal working hours to minimize disruption, but all too often it can result in a late-night call to your IT Manager.
Make a plan: Incident response planning can help you get back online faster
An incident response plan is a powerful cybersecurity tool to help organizations prepare for incidents and remediate issues as fast as possible. An incident response plan addresses who is responsible for what tasks, what platforms and services should be engaged in a cybersecurity incident, and more. The more detailed and tested your incident response plan is, the more prepared you are to deal with outages and incidents like the CrowdStrike incident.
Act quickly: Limit the damage and reduce your downtime
The CrowdStrike incident was costly to organizations worldwide that has a real-world impact. Fortune 500 companies faced a collective $5.4 billion in losses. The incident cost Delta Airlines $500 million with over 5,000 cancelled flights. Not only were flights affected in the CrowdStrike incident, but the NHS also experienced issues with their appointment management systems, causing doctors to work offline from certain records and systems. Acting quickly in a cybersecurity incident can save you hundreds of thousands of dollars in operating costs and business reputation. The faster you get your systems online, the less risk and cost you face.
Know your devices: Maintain an accurate device inventory
Security teams with the highest cybersecurity posture know what devices they have, where they are, and who is using them. Maintaining an accurate inventory of the devices in your network is especially important when dealing with a cybersecurity incident. Visibility into your device fleet helps you understand which devices are affected and informs how you should remediate incidents. In the CrowdStrike incident, knowing which OS devices were running and knowing when they last checked in helped IT teams determine which machines were affected.
How Did Microserve Remediate the CrowdStrike Incident?
Behind the scenes, Microserve technicians worked tirelessly around the clock to update machines and remediate the CrowdStrike incident. Working well into the evenings and all weekend, around 20 Microsoft technicians helped our clients recover BitLocker codes, deploy updates via USB, and more.
An unnamed Microserve customer had this to say after CrowdStrike hit their organization:
“I want to thank you and the Microserve team, as I know many of your folks supported our efforts to remedy devices across the province since last Friday. Please share my thanks and gratitude to the team.”
Do you want to work with Microserve to develop your incident response plan, get an accurate device inventory, and respond quickly to cybersecurity threats? Get in touch with the cybersecurity experts at Microserve today to discuss how we can help.
