Commentary from Semperis on today’s IT outages
July 2024 by Guido Grillenmeier, Principal Technologist, EMEA, Semperis
The commentary from Guido Grillenmeier, Principal Technologist, Semperis on today’s IT outages.
The impact of the Crowdstrike update and subsequent IT outage is having a profound impact as hospitals, airlines and production lines at manufacturing companies are down. Even POS systems in small businesses using Windows are down.
After receiving the Crowdstrike update the Windows resources were exhausted causing the systems to boot and then get stuck in the Blue Screen of Death due to issues loading the drivers after the update.
While we assume this not the result of a cyberattack on the internal network of Crowdstrike, the impact to the victims is indistinguishable from a well-planned supply chain attack, such as SolarWinds, Kaseya and NotPetya.
The immediate impact today for global companies is devastating, their business is partially or completely disrupted based on which systems are using Crowdstrike’s software. Fixing this requires every single system to be rebooted into Safe mode and removing the problematic files after it became evident their software was the root cause. In many cases the reboot can’t be done remotely as the systems are stuck in the boot phase. Instead, an IT person will have to visit each and every system and fix it manually.
Can we prevent outages like this in the future? Due to the increased interdependencies between systems and software that controls them, be it to protect them or to merely update or monitor them, there are no guarantees these outages won’t happen.
The one constant that does exist regardless of whether there is criminal intent or software updates gone bad, is proper disaster recovery planning for the different types of outages. Disaster recovery planning must consider all outage scenarios to ensure that the core business can continue. The next outage might take out your identity system, in most companies still Microsoft Active Directory. It is essential to build operational resiliency into your business plans so when disaster strikes you can limit disruptions and keep systems running.