The CrowdStrike Incident: What Happened & Software Quality Concerns, SolarWinds Lawsuit

[00:00:00] It's Monday, July 22nd, 2024, and I'm Dave Sobel. Two things to know today. Well, we have to talk about Friday's CrowdStrike incident to discuss where we go from here. And let's update on SolarWinds and their SEC lawsuit. This is the Business of Tech.

[00:00:23] So I was rather unplugged on Friday. Anything happen in the world of IT? Jokes aside, the top story is clearly the CrowdStrike update. In the early hours of Friday morning, CrowdStrike released a software update to their Falcon

[00:00:38] platform that caused Windows machines to enter a boot loop, ending in the blue screen of death. CrowdStrike has kernel-level access, allowing the system-level failure. The fix was released, and booting to safe mode is required to either work around the issue or apply the fix.

[00:00:54] Microsoft confirmed that the issue also impacted Windows 365 PCs. That's the technical side of the story. The impact was significant. According to a Microsoft blog post, around 8.5 million Windows devices, less than 1% of Windows machines globally, were impacted. Across industries, it affected banks, retailers, brokerage companies, airlines and airports,

[00:01:17] rail networks alone, and all globally. Because the fix is so manual, despite its simplicity, it's a massive manual labor effort at scale. Microsoft has released a recovery tool to help IT admins repair Windows machines that were impacted by the update.

[00:01:34] The tool creates a bootable USB drive that allows IT admins to quickly recover affected machines by automatically deleting the problematic CrowdStrike file. Separate recovery steps are available for Windows Virtual Machines running on Azure, and Microsoft has published recovery steps for all Windows 10 and Windows 11 devices

[00:01:52] on its support site. Shares of the company closed down 11% Friday, but are up over 93% over the last 12 months. CrowdStrike CEO George Kurtz issued an apology for the incident. Kurtz was also involved in a similar tech debacle as the CTO of McAfee in 2010.

[00:02:11] And for those saying it didn't happen on Linux, it did, back in June with a different kernel system crash from CrowdStrike. Why do we care? Well, breaking format slightly, I want to highlight two other perspectives on why we care.

[00:02:27] Megan McArdle in the Washington Post had an opinion piece about the blame, our drive for efficiency. Quoting the article, It's natural to ask how the heck this could have happened and how we can make sure it never does again. But another question needs to come first.

[00:02:42] How much might we be willing to sacrifice to protect against the risk? At the moment, this might seem absurd. To people stranded by cancelled flights or missing doctor's appointment, it's worth spending almost anything to avert the chaos that has beset airlines, hospitals, first responders and other critical services.

[00:03:02] But this is like saying it's worth insuring your car to the absolute maximum after you've gotten into a big accident. Many car owners decide not to buy the most generous possible coverage because that coverage also carries higher premiums.

[00:03:17] So too with maximally insuring against meltdowns like the CrowdStrike mess, because doing so would require paying a price in economic efficiency. So we have to decide what trade-offs we're willing to make. Averting this particular disaster might not have cost a ton of money, to be clear.

[00:03:35] But CrowdStrike is only one of approximately 1 zillion points of failure in our thoroughly networked and globalized economy. Over the past 50 years, the market's relentless drive for efficiency and reach has made such mass failure nodes more numerous, more potentially catastrophic and harder to see before they

[00:03:54] fail while also giving us instant access to all the world's culture and most of its information. Plus more cheaper and better goods and services and a global economy that every year lifts tens of millions more out of poverty.

[00:04:08] And another key perspective, Jen Easterly, director of CISA from a post on LinkedIn. Quote, While it wasn't malicious, it was a serious mistake, one for which CEO George Kurtz took full responsibility, apologized and committed to resolving collaboratively.

[00:04:24] Needless to say, in a world of opportunistic punitry and schadenfreude, George and his team came under pretty heavy fire. I'll just say two things on that. First, from very early morning until late in the evening, George and his entire team were transparent, responsive and professional with my team.

[00:04:42] Second, anyone who's ever been part of managing a major incident knows that gleefully throwing metaphorically flaming piles of poo at anyone, particularly in the middle of a massive crisis response is generally asshole behavior. And the second passage.

[00:04:57] To channel my alter ego Bob Lord, we don't have a cybersecurity problem, we have a software quality problem. Now, before you start throwing flaming poo at me, yes, I further recognize the irony of a cybersecurity vendor creating a defective update that temporarily crippled the operations

[00:05:14] of the world's biggest software company. And to be clear, this was not a Microsoft issue. As I said at the top, we don't know yet fully what happened or why. But one thing I do know is that any company that builds any kind of software should design,

[00:05:29] test and deliver it with a priority on dramatically driving down the number of flaws. Flaws which can be intentionally exploited by bad actors or flaws that can unintentionally take down critical services across the globe.

[00:05:43] The other thing I know is that anyone who consumes tech, yep, that's basically all of us, should demand that those technology and software manufacturers do exactly that. Why we've been working with technology companies large and small, including CrowdStrike and

[00:05:57] Microsoft, to voluntarily commit to the Secure by Design pledge. So why do we really care? Now, I want to push back on some of the buzz I saw online, where the call was if you're not supporting to be silent.

[00:06:11] Empathy is not the only response to the incident and constructive criticism is also required. I want to applaud the technology community, both MSP and security, for working together to solve customers' problems. But IT services and MSPs shouldn't have to bear all the weight of incidents such as this.

[00:06:30] Easterly is right, we do have a software quality problem. And as such, it's appropriate to be asking the question, how did this happen again? And what can we do differently to prevent it? Multiple security experts are noting that the product wasn't sufficiently tested or

[00:06:49] that it should have been rolled out onto a limited pool first. And the CEO himself has experience with outages and like everyone, memory fades. Consider McArdle's argument, how many customers would actually spend the money to create the necessary redundancies across those one zillion points of failure?

[00:07:08] Asking the customer to spend more because systems are fragile accepts that software vendors can take all the value and the money and the downside is pushed outward. Replacing the entire workload on the implementers, IT departments and service providers is ridiculous.

[00:07:26] Software vendors should have a non-zero level of responsibility for the products they release. Easterly is right, anyone who consumes tech should demand that those technology and software manufacturers design, test and deliver with a priority on driving down the number of flaws.

[00:07:42] And there should be consequences beyond the stock price. Compare this to other industries, aviation, automotive, legal and medical, where there are real consequences when things go wrong. IT departments and service providers must advocate for more stringent testing protocols and phased rollouts from their vendors.

[00:08:02] This includes demanding transparency in testing procedures and results. Encourage and support regulatory frameworks that hold software vendors accountable for the reliability and security of their products. And speaking of consequences... This episode sponsored by Skykick, new sponsor for MSP Radio.

[00:08:26] Skykick has been helping over 30,000 MSPs for the past 10 years be more successful in the cloud, migrating, protecting, securing and managing their Microsoft 365 customers. A highlight in their offerings is their Microsoft 365 data protection solution, Cloud Backup. They've recently enhanced it with a new feature called Smart Insights.

[00:08:46] This feature delivers visual insights, empowering partners to engage more efficiently with customers on Microsoft 365 data protection. And MSP Radio listeners get a special offer. Get a free 2M365 email migration for a customer when you bundle it with backup. Visit skykick.com slash MSP Radio to learn details.

[00:09:10] A U.S. judge has dismissed most of the Security and Exchange Commission's lawsuit against SolarWinds, stating that the claims were based on hindsight and speculation. The lawsuit accused SolarWinds of concealing security weaknesses before and after a cyber attack linked to Russia.

[00:09:27] The judge also dismissed claims against the Chief Information Security Officer Tim Brown. The SEC case filed last October was the first to target a company that fell victim to a cyber attack without a simultaneous settlement. Why do we care?

[00:09:42] Well, the ruling is seen as a positive development for CISOs and IT leaders facing legal scrutiny after cyber attacks. Except I'll counter-argue that it shows a stunning lack of responsibility in our current legal framework. Even a layperson knows SolarWinds 1.2.3 highlights a problem.

[00:10:00] And so far, there's no responsibility assumed by the vendor, which results in millions of dollars of damage paid for by customers and providers, and in this case, the U.S. taxpayer directly. Now, disclosure, I'm a SolarWinds shareholder, so I know the journey here.

[00:10:16] The stock plummets upon notice of the breach, and after 18 months of purgatory, begins the climb upward. And customers still run their Orion products. I see here instead a vision of the future of CrowdStrike. There won't be a mass exodus.

[00:10:32] There will be no individual consequences for leadership in the software companies who made the mistake. So what's the advice? Zig while others zag. Your question isn't about replacing CrowdStrike or being lucky that you didn't run it.

[00:10:46] How can you both simplify systems and push vendors to take responsibility for their part in the chain? Today's episode is supported by CoreView. Your customers need your Microsoft 365 expertise, and CoreView has the only M365 management platform designed for MSPs.

[00:11:07] Manage hundreds of tenants, automate manual tasks, and monitor compliance, all while intelligently comparing to the baseline. With a no-code control approach, CoreView revolutionizes your Microsoft 365 administration. This powerful platform enables automatic reporting and remediation, ensuring optimal performance and security. The best part?

[00:11:29] You achieve this high level of service without the need for a large workforce, allowing you to focus on growing your business through efficiency. Want to know more? Visit coreview.com slash MSP and find out more.

[00:11:43] It's National Hammock Day, which I think a lot of IT administrators need after that whole incident. The show is also celebrating achieving 1 million podcast downloads over the weekend. So thank you all for listening.

[00:11:59] With listenership all over the world, it's a pleasure for me every day to deliver some insights to you. If you've got a question you want answered, we take those, send them in as question at MSPradio.com.

[00:12:10] There'll be a live show this week, Wednesday 3 p.m. Eastern on YouTube and LinkedIn. And if you like the show, share it with a colleague. Oh, and are you a software vendor? Want to reach an MSP audience? This is a great show to do it on.

[00:12:24] MSPradio.com slash engage. Talk to you again tomorrow.