Quality Assurance in Software: Lessons from the CrowdStrike Incident with Ryan Morris

[00:00:12] [SPEAKER_02]: Well, we have to talk about CrowdStrike.

[00:00:15] [SPEAKER_02]: But instead of looking to the past, we'll talk about where we go from here.

[00:00:20] [SPEAKER_02]: Ryan Morris, channel expert and my co-host on Killing It joins me today.

[00:00:25] [SPEAKER_02]: Welcome to the Business of Tech Lounge, the live version of the Business of Tech podcast.

[00:00:30] [SPEAKER_02]: It's Wednesday, July 24th, 2024.

[00:00:34] [SPEAKER_02]: And I'm Dave Sobel.

[00:00:35] [SPEAKER_02]: We'll be taking questions and comments throughout the show to make sure to put them in chat,

[00:00:40] [SPEAKER_02]: in particular with a topic like this.

[00:00:42] [SPEAKER_02]: I would love to hear from you.

[00:00:43] [SPEAKER_02]: I think we've got a lot of thoughts on where we might go, and it'll be a lively discussion.

[00:00:48] [SPEAKER_02]: There is a dedicated questions section in the show with listeners submitted questions,

[00:00:54] [SPEAKER_02]: but we will take those comments and questions at any point during the show.

[00:00:58] [SPEAKER_02]: I want to thank SalesBuilder, our Patreon sponsor, who support makes this show possible.

[00:01:04] [SPEAKER_02]: Focus on your IT sales workflow with the power of automation, and visit them at salesbuilder.com.

[00:01:11] [SPEAKER_02]: That's B-U-I-L-D-R dot com.

[00:01:14] [SPEAKER_02]: And a reminder, I am watching that chat.

[00:01:18] [SPEAKER_02]: So let's dive into our top story.

[00:01:20] [SPEAKER_02]: We have to talk about the CrowdStrike Incident.

[00:01:23] [SPEAKER_02]: Not because we want to spend a lot of time on the history.

[00:01:26] [SPEAKER_02]: I already covered that on Monday.

[00:01:28] [SPEAKER_02]: But for those of you who might not have known, those 78 minutes where an update from CrowdStrike was out in the wild

[00:01:34] [SPEAKER_02]: and it took over 8.5 million PCs,

[00:01:38] [SPEAKER_02]: because one of the largest IT outages we've ever seen.

[00:01:42] [SPEAKER_02]: Recovery is still ongoing as Delta Airlines, among others, is still working on cleaning that all up.

[00:01:48] [SPEAKER_02]: But I don't want to dive into the what happened.

[00:01:50] [SPEAKER_02]: If you missed it, as on Monday's show I did a deep dive on what happened with the particular incident.

[00:01:56] [SPEAKER_02]: But instead, I want to talk about Jen Easterly's comments on LinkedIn.

[00:02:03] [SPEAKER_02]: Now, Easterly is the head of CISA and gave a lot of thought in her ode to an outage

[00:02:08] [SPEAKER_02]: of exactly where we might go as an industry and what started this.

[00:02:13] [SPEAKER_02]: Her basic premise?

[00:02:14] [SPEAKER_02]: We have a software quality problem, not a hacking one.

[00:02:18] [SPEAKER_02]: Let's pull from her comments that I covered on Monday's episode.

[00:03:07] [SPEAKER_02]: The other thing I know is that anyone who consumes tech, yep that's basically all of us,

[00:03:13] [SPEAKER_02]: should demand that those technology and software manufacturers do exactly that.

[00:03:20] [SPEAKER_02]: So, to have a conversation about this I want to bring on my co-host from Killing It, Ryan Morris.

[00:03:26] [SPEAKER_02]: Ryan, thanks for joining me today.

[00:03:30] [SPEAKER_03]: I just kind of had to do a little bit of that because it feels when I see you on a digital

[00:03:34] [SPEAKER_03]: screen like this that's built into my DNA now so I'm very glad to be here sir.

[00:03:41] [SPEAKER_03]: Thank you very much for having me.

[00:03:42] [SPEAKER_02]: Well, I appreciate you joining me because in particular I thought your expertise around

[00:03:47] [SPEAKER_02]: thinking about channel and channel delivery would be super helpful for this conversation.

[00:03:52] [SPEAKER_02]: One of the things that I kind of want to start with a basic idea if we're talking about

[00:03:57] [SPEAKER_02]: software quality and we're thinking about the level of quality that's coming out of

[00:04:03] [SPEAKER_02]: manufacturers and if IT service providers, MSPs, whatever we want to talk about are

[00:04:07] [SPEAKER_02]: the delivery mechanisms for that they don't actually control the software.

[00:04:11] [SPEAKER_02]: How much control do you think MSPs and channel companies really have over the quality?

[00:04:18] [SPEAKER_03]: You know what, that's a really good question because it gets to the heart of why would somebody

[00:04:24] [SPEAKER_03]: agree to be in business with me. If I am a technology vendor, I try to recruit partners,

[00:04:30] [SPEAKER_03]: I try to sell through partners, I try to get them to take my products to market

[00:04:35] [SPEAKER_03]: that will benefit them and it will benefit me. We've asked that question extensively in research

[00:04:41] [SPEAKER_03]: and really tried to get down to the heart of the issue. Why would an independent company that could

[00:04:47] [SPEAKER_03]: choose to work with any vendor, why would they choose to work with you? There's a lot of people

[00:04:53] [SPEAKER_03]: who gut level, they think well because I have the best technology, cutting edge, most advanced

[00:04:58] [SPEAKER_03]: feature function rich, I have the best solution that's why people work with me. That is

[00:05:03] [SPEAKER_03]: absolutely not true. The A number one reason why a channel partner would agree to work with a vendor

[00:05:10] [SPEAKER_03]: is that the software or technology, whatever it is that they are getting into business with

[00:05:16] [SPEAKER_03]: does what it promises it will do reliably and professionally without the risk of embarrassing

[00:05:24] [SPEAKER_03]: me with my install base as the channel partner. I cannot tolerate bleeding edge technology

[00:05:32] [SPEAKER_03]: that I will deploy in my install base especially as an MSP with a fixed cost contract agreement

[00:05:38] [SPEAKER_03]: with those end users. I can't bring in unproven things, I can't bring in the latest

[00:05:45] [SPEAKER_03]: most advanced capabilities and just pebble in the pond and see what happens with the ripples.

[00:05:53] [SPEAKER_03]: That's malpractice, right? You just can't do that. Don't embarrass me with my customers.

[00:05:58] [SPEAKER_03]: That's the very first thing that we have to do. I believe that speaks to the quality of the product

[00:06:06] [SPEAKER_03]: and to the support that is available for the people in the channel who take our technologies

[00:06:12] [SPEAKER_03]: to market as vendors. If you cannot build a perfect product, I fully admit that's not possible,

[00:06:20] [SPEAKER_03]: right? I get it. This is why we all have jobs in the channel. Technology is not perfect and

[00:06:25] [SPEAKER_03]: never will be perfect, but if you can't do that then you must provide the downstream capabilities

[00:06:33] [SPEAKER_03]: to react and recover as effectively as possible. That's what we should expect. I will call that

[00:06:41] [SPEAKER_03]: the channel partners Bill of Rights. You deserve partners who will build good product,

[00:06:48] [SPEAKER_03]: support it extensively and accelerate your access as the channel partner to escalated levels of

[00:06:56] [SPEAKER_03]: support. If we can't count on that from a vendor, doesn't matter how cool their software is,

[00:07:03] [SPEAKER_03]: doesn't matter what the margins on it are, doesn't matter how broadly distributed.

[00:07:17] [SPEAKER_02]: Interestingly, if I think about what's going on then from a CrowdStrike perspective,

[00:07:24] [SPEAKER_02]: I don't think CrowdStrike necessarily did anything wrong after the mistake, right? After the point

[00:07:33] [SPEAKER_02]: where they hit, where they realized they made a mistake. I think where the problem was is the

[00:07:38] [SPEAKER_02]: earlier part. It is we don't know and I want to be super clear that I'm speculating, but I can

[00:07:44] [SPEAKER_02]: feel reasonably comfortable in speculating that well something broke in the quality assurance

[00:07:49] [SPEAKER_02]: testing process, right? Because I can know that at some level it didn't go the way that they were

[00:07:55] [SPEAKER_02]: expecting it to. By the way, if we dig into some of the way that very large companies do this,

[00:08:00] [SPEAKER_02]: they often deploy in a small geography or a small number of clients, not everybody, right? We can

[00:08:08] [SPEAKER_02]: definitely recognize that there is an issue there from a delivery of quality perspective.

[00:08:13] [SPEAKER_02]: Let's talk industry-wide. What can we do industry-wide to focus on quality here and get better

[00:08:21] [SPEAKER_02]: visibility into that? Because if I'm a channel partner and I'm working with CrowdStrike,

[00:08:26] [SPEAKER_02]: I don't know how their QA process works. I don't get visibility into that until it breaks.

[00:08:33] [SPEAKER_02]: Is there a technique from it? Let's start industry-wise before we get to individual

[00:08:37] [SPEAKER_02]: provider. Is there something as an industry we can do here to make this product,

[00:08:41] [SPEAKER_03]: to increase quality here? You know what? You're exactly right. It has to begin at the industry

[00:08:47] [SPEAKER_03]: level because you and I, we can tilt at windmills all day long about what my issues are as an

[00:08:54] [SPEAKER_03]: individual provider and a very large globally distributed company, they're not going to change

[00:08:59] [SPEAKER_03]: for my request, right? It has to begin at the industry level and I am really glad you brought

[00:09:04] [SPEAKER_03]: up the Jen Easterly commentary on this because she says something in there that I agree with

[00:09:11] [SPEAKER_03]: very passionately. Security by design or quality by design, right? A pledge from an industry

[00:09:20] [SPEAKER_03]: perspective that says it's not cool when you make a software product and then offload the QA and

[00:09:27] [SPEAKER_03]: testing and support costs onto the customer or onto the channel. I believe that that is industry

[00:09:36] [SPEAKER_03]: practice for a vendor to sell a product that is not ready for prime time. Now, I get why they do.

[00:09:44] [SPEAKER_03]: It's faster to market. We don't want to be the last one to take advantage of this.

[00:09:48] [SPEAKER_03]: It's less expensive. It's less overhead and operating expense for me as the vendor.

[00:09:54] [SPEAKER_03]: I get why it happens but it's not acceptable. I think that as an industry we need to take

[00:10:01] [SPEAKER_03]: a step back from bringing the fastest to market with the most features with the lowest cost and

[00:10:08] [SPEAKER_03]: that whole you want it now, you want it fast, you want it great quality, you want a great price.

[00:10:14] [SPEAKER_03]: And we always say you can have two but you can't have all three of those points of the value

[00:10:18] [SPEAKER_03]: conversation. As an industry we've been lying to ourselves saying no, you can get it fast

[00:10:25] [SPEAKER_03]: and quality and cheap. You can have everything you want. No, you can't and let's be honest about that.

[00:10:33] [SPEAKER_03]: So I think that initiatives like the idea of security, secure by design, quality by design,

[00:10:42] [SPEAKER_03]: I think that holistically we all need to buy into that and begin to push back on vendors who say,

[00:10:50] [SPEAKER_03]: well, I make the product and then you just make sure it works.

[00:10:54] [SPEAKER_03]: No, the value chain is an interdependent ecosystem. If one part of it gets sick, everybody pays a

[00:11:03] [SPEAKER_03]: price. If the vendor sells substandard technology and passes along the responsibility to the MSP

[00:11:13] [SPEAKER_03]: or the partner channel and then the partner channel has to fix it without additional compensation

[00:11:21] [SPEAKER_03]: with their shouldering all the risk for radically lower margins in that conversation,

[00:11:27] [SPEAKER_03]: that's not healthy and it can't sustain. And if you don't fix it,

[00:11:32] [SPEAKER_03]: dot, dot, dot channel partners either drop you or they go out of business because they've

[00:11:37] [SPEAKER_03]: had a risk realization or they just can't make enough money to shoulder all of that responsibility.

[00:11:44] [SPEAKER_03]: At which point, everybody thinks, oh, that's sad for the channel partner, they paid the price.

[00:11:50] [SPEAKER_03]: Yeah, but I've been a vendor for very many years and if my partners go away for any reason

[00:11:56] [SPEAKER_03]: without forewarning, without my ability to shift and reallocate that coverage

[00:12:01] [SPEAKER_03]: to another participant, by definition, I pay the price as the vendor. I'm not isolated from

[00:12:09] [SPEAKER_03]: that sickness. I pay the price because I just got diminished capacity. Some of my customers are

[00:12:14] [SPEAKER_03]: left uncovered and I'm the one who's going to suffer the brand impact for that lack of coverage

[00:12:21] [SPEAKER_03]: in the marketplace. So I believe we start there as an industry and we say let's be good to each

[00:12:28] [SPEAKER_03]: other. Mutually beneficial ecosystems will make everybody more money. We start there and then

[00:12:36] [SPEAKER_03]: let's have a scorecard. Let's have, you know, you and I, we've grown up in a world of magic

[00:12:42] [SPEAKER_03]: quadrants and waves and all of these different ways of measuring the quality of a vendor offering.

[00:12:50] [SPEAKER_03]: You notice how most of that is pre-market, right? It's do you have the best technology

[00:12:56] [SPEAKER_03]: and do you have the ability to execute on developing and delivering that technology?

[00:13:01] [SPEAKER_03]: Do you notice how those things don't include aftermarket? That's not okay. Like that, that's just

[00:13:07] [SPEAKER_02]: not acceptable. In fact, this is what I want to ask you. So you're somebody that spends a lot of

[00:13:12] [SPEAKER_02]: time advising channel companies on the way to go to market. And I, you know, so I'm going to say

[00:13:16] [SPEAKER_02]: I'm out of that side of the game long enough as just an independent analyst to say,

[00:13:21] [SPEAKER_02]: look, I can see the actual opportunity here for a couple of channel vendors. The channel vendor

[00:13:26] [SPEAKER_02]: that says, you know what? We're not going to just leave you hanging. We're actually going to

[00:13:33] [SPEAKER_02]: assume some non-zero responsibility in terms of labor restoration that we will help you and

[00:13:40] [SPEAKER_02]: guarantee at some level, not just be like, oh, we might give you a month's credit or

[00:13:45] [SPEAKER_02]: something along those lines. But we're actually going to say in the event of outage,

[00:13:49] [SPEAKER_02]: we're going to stand with you with some warranty. Right? You'll get, we may not be able to make

[00:13:54] [SPEAKER_02]: you whole, but we'll make you, we'll invest in you. We make a mistake. We're going to help

[00:13:59] [SPEAKER_02]: you with some monetary credit. Are you ready to advise channel businesses the vendor side that

[00:14:06] [SPEAKER_02]: this is a competitive advantage? Am I right in highlighting that like, there are some vendors

[00:14:11] [SPEAKER_02]: here that could move to make a real difference with an insurance style approach.

[00:14:17] [SPEAKER_03]: You are 100% correct in that strategy. Think about it. If I raise my expectations, right? My advice

[00:14:25] [SPEAKER_03]: to the channel community is to have higher expectations of the vendor partners that you

[00:14:31] [SPEAKER_03]: get into business with and insist that they demonstrate their quality assurance to you

[00:14:38] [SPEAKER_03]: and insist that they convince you that if and when something goes wrong, this is the way that we

[00:14:46] [SPEAKER_03]: will handle it. And we will make sure that you will be okay. If in the process of that conversation,

[00:14:51] [SPEAKER_03]: a vendor says, you know what? I'm not willing to warranty my solution to the extent that I

[00:14:56] [SPEAKER_03]: will make you whole if and when things go wrong. My inevitable reaction to that is

[00:15:04] [SPEAKER_03]: so apparently you are afraid that something serious is going to go wrong and that the implications of

[00:15:11] [SPEAKER_03]: that are going to be big and you don't want to shoulder the expense. You know what? Thank you

[00:15:16] [SPEAKER_03]: very much for telling me you don't have confidence in your own product. I wouldn't have

[00:15:21] [SPEAKER_03]: confidence in it if you don't either, right? I believe that that is a phenomenal opportunity

[00:15:27] [SPEAKER_03]: for distinguishing. Don't tell me you've got three more features. Don't tell me that you've

[00:15:32] [SPEAKER_03]: got a special pricing promotion for the month of May. Don't tell me that our brand is the biggest

[00:15:37] [SPEAKER_03]: and the best and customers are clamoring to buy my solution. Tell me we are in this together

[00:15:44] [SPEAKER_03]: and what that means is not only will I give you good technology, I will enable you to sell it

[00:15:51] [SPEAKER_03]: and support it effectively yourself. I will advise you on how to build a stable lucrative

[00:15:56] [SPEAKER_03]: and growing business around the platform of my technology and I am a grown-up. I acknowledge

[00:16:05] [SPEAKER_03]: that things might go wrong and if they do, here is precisely how I will make you okay in that process.

[00:16:13] [SPEAKER_03]: If vendor A says that and vendor B says no, no, no, I won't warranty this stuff,

[00:16:18] [SPEAKER_03]: the decision on which vendor to go with just got really easy.

[00:16:22] [SPEAKER_02]: I'm going to laugh and go, there's now a video of us talking about this that can be shared with

[00:16:26] [SPEAKER_02]: vendors of our approach. The other thing I'm going to highlight that is the SISA has put the list of

[00:16:32] [SPEAKER_02]: companies that have signed their Secure By Design pledge online. That's available. We'll put the

[00:16:40] [SPEAKER_02]: URL in the description for those watching the recording and throw it up here on screen

[00:16:45] [SPEAKER_02]: in a banner for those that are watching. What we'll actually say is you can now look and

[00:16:51] [SPEAKER_02]: find the vendors that have signed it. From a channel perspective, kudos to them. Huntress has already

[00:16:59] [SPEAKER_02]: signed, enable his sign disclosure. I'm a shareholder. I always feel important to be transparent with

[00:17:06] [SPEAKER_02]: that. There's only a handful of channel companies that have actually signed the Secure By Design

[00:17:14] [SPEAKER_02]: pledge. Right? I'm going to ask, we've talked a lot from an industry like a wide industry

[00:17:20] [SPEAKER_02]: perspective, but a lot of our listeners, people listening to show right now are smaller or mid-sized

[00:17:26] [SPEAKER_02]: organizations that we consider themselves a channel company or an MSP. Someone that's designing

[00:17:31] [SPEAKER_02]: that and you alluded to it right at the beginning. A big multinational company is not going to

[00:17:36] [SPEAKER_02]: change their policy for that solution provider. What can that solution provider do in the

[00:17:43] [SPEAKER_02]: short term in your opinion to like zig while others are zagging right now? What can they

[00:17:50] [SPEAKER_02]: do differently based on this quality problem? That's a great question and I'm going to admit

[00:17:56] [SPEAKER_03]: that my answer is going to, it's going to sound like I'm putting extra work on your shoulders

[00:18:01] [SPEAKER_03]: because a little bit that's true. The old adage always be closing. Always be evolving

[00:18:09] [SPEAKER_03]: is the thing. As an industry we are infatuated with the concept of standardization and

[00:18:16] [SPEAKER_03]: stability, packaged offerings with best practice designed in with continuity over time. I get it

[00:18:24] [SPEAKER_03]: right? I understand why that is true but the minute you stop evaluating where you could push

[00:18:31] [SPEAKER_03]: the edges of your current offering, where you might improve in the margins, where you might

[00:18:36] [SPEAKER_03]: get a little bit better at the offering you do, the minute you stop doing that is the

[00:18:42] [SPEAKER_03]: minute you volunteer to be a dinosaur in your own industry. You can't sell the same thing you

[00:18:48] [SPEAKER_03]: sold five years ago unless you can validate that it is still best practice. God bless them.

[00:18:57] [SPEAKER_03]: There are some vendors who year after year after year continue to lead the way and they are

[00:19:03] [SPEAKER_03]: indeed still best practice in their category. Kudos, that's great but don't assume,

[00:19:08] [SPEAKER_03]: don't hope that that's going to be true. Always be researching and testing out the vendors that you

[00:19:16] [SPEAKER_03]: might swap in and out of your packaged offering. What we have to do is to admit

[00:19:23] [SPEAKER_03]: you and I don't sell dot matrix printers anymore, you and I don't sell 2001 versions of systems

[00:19:30] [SPEAKER_03]: admin software. What was best in class at one point is not permanent and in order to acknowledge that

[00:19:40] [SPEAKER_03]: and live within that, you and I did a project years ago where we argued for one of the

[00:19:47] [SPEAKER_03]: fundamental principles of remaining perpetually relevant which is the idea that I refer to as

[00:19:53] [SPEAKER_03]: argue in the alternative. What I mean by that is argue against your own brand offering, be your own

[00:20:02] [SPEAKER_03]: competitor and say what are your weaknesses and why would a customer not choose you and what is

[00:20:08] [SPEAKER_03]: the gray area that you haven't yet fully solved for your customers? If you're the one challenging

[00:20:15] [SPEAKER_03]: yourself like that, you're going to find out what's a problem before it's embarrassing in

[00:20:20] [SPEAKER_03]: front of a customer or causes a loss to a competitor. You've got to research the alternatives, you have

[00:20:26] [SPEAKER_03]: to test your own offering and then you have to be willing to do the change management even though we

[00:20:34] [SPEAKER_03]: all cringe at that concept, unplugging vendor A and replacing them with vendor B at any component

[00:20:41] [SPEAKER_03]: level of the stack that we've baked our cake 100 years ago and we don't want to mess with

[00:20:49] [SPEAKER_03]: the recipe yet except that it's going to get stale eventually unless you are conscientiously proving

[00:20:57] [SPEAKER_03]: that it is still best in class. So in addition to just operating your business and working with the

[00:21:05] [SPEAKER_03]: vendors you're currently in business with to ensure that you have the best deployment

[00:21:08] [SPEAKER_03]: and the best support mechanisms, you must also actually do the things that would justify

[00:21:17] [SPEAKER_03]: when you want to actually make a change and then be very good at onboarding that new vendor doing the

[00:21:26] [SPEAKER_03]: two systems in parallel to verify that they are stable, doing a cutover process that minimizes

[00:21:33] [SPEAKER_03]: disruption and risk potential for your customers. I guarantee there's half of the audience that

[00:21:39] [SPEAKER_03]: will listen to this that just literally broke out in a sweat when I described that. That's

[00:21:45] [SPEAKER_03]: the way we used to do things and as MSPs we said to ourselves we don't need to do that because we

[00:21:50] [SPEAKER_03]: have standardization and stabilization until you leave it just a little too long and then it becomes

[00:21:58] [SPEAKER_03]: yesterday's news and it smells like fish and you can't rest on your laurels. Well implied in what

[00:22:05] [SPEAKER_02]: you just said is something that I want to call out explicitly and in some cases that's removal

[00:22:09] [SPEAKER_02]: of vendors. I think that oftentimes too many solution providers are leaning into having

[00:22:15] [SPEAKER_02]: everything, must have as many solutions as possible and one of the things I've been thinking

[00:22:21] [SPEAKER_02]: about a lot in terms of that you brought up kind of cake and cooking. I want to actually

[00:22:25] [SPEAKER_02]: refer to beer making for a moment here and there's a reason why the Germans have purity laws

[00:22:29] [SPEAKER_02]: right? That there are only four ingredients in beer is because if you make it overly complicated

[00:22:36] [SPEAKER_02]: it will ruin the batch and you can dilute it or in our case from a technology perspective

[00:22:42] [SPEAKER_02]: introduce other risks that are not necessarily ones that are worth bringing in. I've been thinking

[00:22:47] [SPEAKER_02]: a lot about this particularly when we talk about cybersecurity oftentimes I say it and I want to

[00:22:52] [SPEAKER_02]: say it a little bit differently this time is simplicity is good right? Being really great

[00:22:58] [SPEAKER_02]: at backup and disaster recovery and having a good incident plan might just mean literally

[00:23:03] [SPEAKER_02]: having a really good backup tool and not a lot else right? And then you may lean into multifactor

[00:23:09] [SPEAKER_02]: authentication and good training and not a million other tools and you might be decide to be really

[00:23:16] [SPEAKER_02]: good at just a few things and keep your recipe very, very simple because you've also not introduced

[00:23:24] [SPEAKER_02]: a whole bunch of things. I'm not meaning to knock on CrowdStrike here I'm just observing that

[00:23:29] [SPEAKER_02]: it was one more thing in the mix that could then break at a kernel level and that's what

[00:23:35] [SPEAKER_02]: happened. Yeah it could happen to any vendor but the way to reduce the possibility is reduce the

[00:23:41] [SPEAKER_02]: number of vendors in the mix that's just literally one way of making this better is by removing

[00:23:47] [SPEAKER_02]: that and I think it's important to think about that. I'm going to call out that we are going to

[00:23:51] [SPEAKER_02]: continue taking conversation and discussion here you know Ryan the one sort of to roll

[00:23:56] [SPEAKER_02]: up a little bit and end on this topic before we get into some general questions from the

[00:24:00] [SPEAKER_02]: audience is you know from a you know if you now that we have a moment to try to step back

[00:24:07] [SPEAKER_02]: and look at this I have come to the conclusion I think the quality conversation is the top one

[00:24:12] [SPEAKER_02]: is there something here or do you disagree with that or is there something else that we should be

[00:24:17] [SPEAKER_02]: focusing on as the learning coming out of this? I don't want to spend a bunch of we're not

[00:24:22] [SPEAKER_02]: blaming people because I think the response was very good it's how can we get better to

[00:24:26] [SPEAKER_03]: make sure these things happen less? You know what again I will agree with the analysis that Jen

[00:24:32] [SPEAKER_03]: Easterly had it's naive and childish to assume that everything is perfect and nothing will ever

[00:24:38] [SPEAKER_03]: break. We all wish that were true and some of us insist on that oh if you can't guarantee

[00:24:42] [SPEAKER_03]: absolute 100 quality don't be a child right like that's literally not possible.

[00:24:49] [SPEAKER_03]: Things will break and when they do what matters most is the quality of your response and

[00:24:54] [SPEAKER_03]: your reaction and these guys yes they had an outage and not a cyber hack it was just a good old

[00:25:01] [SPEAKER_03]: fashioned IT outage right? It happened but they responded professionally they've responded aggressively

[00:25:09] [SPEAKER_03]: they are doing very good things and we can probably learn some lessons from the way that

[00:25:14] [SPEAKER_03]: they are managing this stuff. I think that you are right and I'll roll it up like a half step

[00:25:21] [SPEAKER_03]: further quality is the issue secure by design security is a function of quality the fact that

[00:25:28] [SPEAKER_03]: your stuff is not vulnerable is equally your responsibility as a software vendor as when I

[00:25:35] [SPEAKER_03]: click on that button does it do what the code intended it to do right? Software quality must

[00:25:41] [SPEAKER_03]: also include not just performance but reliability stability security manageability each of these

[00:25:50] [SPEAKER_03]: factors is a component of a scorecard that I would use as an MSP to evaluate all of the

[00:25:57] [SPEAKER_03]: offers that are out there and insist that vendors step up and own their thing. You've heard me say

[00:26:04] [SPEAKER_03]: it before right? I think a vendor who ships a product that is inherently not stable or inherently

[00:26:10] [SPEAKER_03]: not secure right equal quality importance there that's malpractice on their part right?

[00:26:16] [SPEAKER_03]: That's akin to I sold you a car that had a phenomenal engine and great handling and really

[00:26:23] [SPEAKER_03]: comfortable bucket seats but you know what it didn't have brakes because you know what the safety part

[00:26:28] [SPEAKER_03]: of it that's my responsibility you can manage that yourself with an aftermarket bolt-on. No,

[00:26:35] [SPEAKER_03]: that's not okay we can't allow our industry to operate at a level that accepts that vendors

[00:26:42] [SPEAKER_03]: ship cars without brake systems and then it's okay because we'll step in and make it secure. No,

[00:26:49] [SPEAKER_03]: what we have to do is to insist on quality and vote with your feet right? Like vote with your

[00:26:57] [SPEAKER_03]: get up and leave if the vendor won't give you something that you can be confident in that you

[00:27:01] [SPEAKER_03]: can really bet your brand reputation and your revenue stream on find another vendor who could

[00:27:07] [SPEAKER_03]: and you know what the worst time to try and figure that out is right after the shit hit the fan.

[00:27:14] [SPEAKER_03]: That's not the right time to research. The right time to research, they always say the right time

[00:27:19] [SPEAKER_03]: to planetary was 20 years ago the second best time to planetary is right now start researching

[00:27:26] [SPEAKER_03]: have alternatives know what you would do if vendor A, B or C went kaput temporarily or

[00:27:34] [SPEAKER_03]: permanently right things happen if that happens what will you do that risk management and scenario

[00:27:40] [SPEAKER_03]: planning that's a component that McKinsey will charge you five million dollars to go through

[00:27:47] [SPEAKER_03]: that exercise as a consulting project even as SMBs we must admit that part of the professional

[00:27:55] [SPEAKER_03]: discipline is anticipating what could go wrong planning for how we will respond and scripting

[00:28:02] [SPEAKER_03]: the process from A to B that's well that's what we mean by being profession. We've given everybody

[00:28:09] [SPEAKER_02]: the homework from there right I'm going to ask you to stick around for questions we're going to

[00:28:12] [SPEAKER_02]: continue to take questions and comments in the chat if you're watching live remember if you

[00:28:16] [SPEAKER_02]: bring your question live you'll get a live response we really do enjoy taking the questions

[00:28:22] [SPEAKER_02]: and of course if you're watching the recording or you have it after time but you can always

[00:28:26] [SPEAKER_02]: submit them ahead of time. I've got our first question right I think you're going to like

[00:28:30] [SPEAKER_02]: this one it comes from Mr. Incredible on YouTube let's hear the question what are your thoughts on

[00:28:36] [SPEAKER_00]: how AI will impact IT careers like sysadmin network admin help desk etc or can you have a

[00:28:43] [SPEAKER_02]: guest on that would share their thoughts? Ryan I knew I'd have a guest on who probably had

[00:28:49] [SPEAKER_02]: some thoughts on this one do you want to take Mr. Incredible's question first?

[00:28:53] [SPEAKER_03]: Yeah absolutely I six months ago literally six months ago the world had a con niption fit and

[00:29:02] [SPEAKER_03]: said oh no AI will eliminate the need for software developers it will eliminate the need for help

[00:29:08] [SPEAKER_03]: desk it will eliminate the need for a lot of the jobs that we do A. that's really over

[00:29:15] [SPEAKER_03]: promising and under delivering for what is essentially literally like a gen 1.3 product

[00:29:21] [SPEAKER_03]: offering I know they call it 4.0 right but it is still gen 1.3 of the technology evolution

[00:29:29] [SPEAKER_03]: it's a tool it will not replace strategy it will not replace the understanding of a customer

[00:29:37] [SPEAKER_03]: environment it will not replace the interpersonal characteristics of trust and credibility it

[00:29:45] [SPEAKER_03]: will not eliminate the intuition or as the smart people say the heuristics of being able to

[00:29:53] [SPEAKER_03]: intuit like this seems like a different situation than what it might look like on the surface so I

[00:30:00] [SPEAKER_03]: need to respond differently I believe help desk there's a FAQs exist right like they always have

[00:30:06] [SPEAKER_03]: and when if you can use AI to make your FAQs more searchable and interactive cool that you're

[00:30:13] [SPEAKER_03]: raise the bar on what is automatable that's great but you will never eliminate the crisis

[00:30:20] [SPEAKER_03]: response responsibility you will never eliminate the I know I think what is the right answer but

[00:30:27] [SPEAKER_03]: I'm just not sure can you reassure me right you can't eliminate those aspects of the jobs that we

[00:30:34] [SPEAKER_03]: do AI is a tool and it will make good humans that are good at their jobs even better and more

[00:30:41] [SPEAKER_03]: productive at their jobs I do not believe in the foreseeable future that it will eliminate any of

[00:30:48] [SPEAKER_03]: these jobs it will make a software developer faster more productive it'll eliminate some quality

[00:30:55] [SPEAKER_03]: issues in the code that they deliver cool we all benefit from that but it will not eliminate the

[00:31:01] [SPEAKER_03]: job of the software developer because it's not just about I need to write good code you know

[00:31:07] [SPEAKER_03]: that whole requirements discovery part of the software development process I know where I have

[00:31:14] [SPEAKER_03]: to ask the customer no really like what is it you actually want this thing to do AI will never be

[00:31:22] [SPEAKER_03]: able to do that in the foreseeable future because it does not understand who we're talking to

[00:31:28] [SPEAKER_03]: and how to intuit tone of voice and to interpret all of those non-language symbols and

[00:31:37] [SPEAKER_03]: and the unspoken parts of reactions and body language that is your job that's my job right

[00:31:44] [SPEAKER_03]: like we will be better at our jobs when AT enhance when when AI enhances us but it's not going to

[00:31:52] [SPEAKER_02]: take our jobs yeah and I'm falling into a similar camp on this and what's interesting I've been

[00:31:57] [SPEAKER_02]: having a lot of conversations with AI people that are deploying products and I find that the closer

[00:32:02] [SPEAKER_02]: they are to the model developers meaning the people at open AI for example or anthropic are the

[00:32:08] [SPEAKER_02]: ones that are talking about completely replacing humans but when I get to people that are actually

[00:32:13] [SPEAKER_02]: building products on top of the models the ones that are actually implementing models in

[00:32:18] [SPEAKER_02]: businesses they're having conversations that look much more like we are augmenting humans we

[00:32:23] [SPEAKER_02]: are making them better we are not replacing it I think that's important because by the way let's

[00:32:28] [SPEAKER_02]: observe we're all people that believe in that sort of delivery of advice through a channel

[00:32:34] [SPEAKER_02]: if you believe in that model which most of us do and history has proven us to be right

[00:32:40] [SPEAKER_02]: we will find then that those people are too far separated from customers to understand the

[00:32:45] [SPEAKER_02]: way technology works in the field in actual production so for mr. Grebel specifically to

[00:32:51] [SPEAKER_02]: answer your question how do I think it's going to impact it I think it's going to make great

[00:32:56] [SPEAKER_02]: engineers network designers network help us people help those people it's going to make great ones

[00:33:01] [SPEAKER_02]: even better it's going to make average ones better and it's going to increase the on ramp

[00:33:08] [SPEAKER_02]: for those that are less skilled to get to average now we will need to continue to invest in

[00:33:15] [SPEAKER_02]: those non-skilled people as they become average they will need to understand critical thinking

[00:33:20] [SPEAKER_02]: better but then that is also a good opportunity for senior people to be mentoring and and coaching

[00:33:26] [SPEAKER_02]: and working with junior ones so I look at this and say it seems all like good opportunity but

[00:33:33] [SPEAKER_03]: not to worry about replacement well and and think about it like this every technology

[00:33:40] [SPEAKER_03]: from the steam engine to the wheel to the incline plane right every technology that has

[00:33:48] [SPEAKER_03]: ever been invented and deployed by humankind was it was feared in the beginning was going to

[00:33:55] [SPEAKER_03]: eliminate human jobs and responsibilities but what happened in the end was it allowed the

[00:34:00] [SPEAKER_03]: people who have that responsibility to be more productive get more done with the same amount

[00:34:05] [SPEAKER_03]: of effort and resource and to focus on what's new what's next what's even more valuable

[00:34:12] [SPEAKER_03]: and it's exactly what you say what's more valuable is the human dynamic and that's just not something

[00:34:20] [SPEAKER_03]: that we have figured out how to program program into software can it happen at some point in the

[00:34:28] [SPEAKER_03]: future sure I'm a technology optimist I believe technology will do phenomenal things that we

[00:34:35] [SPEAKER_03]: can't yet imagine but not in your career time frame right like not mine not yours what it will be is a

[00:34:43] [SPEAKER_03]: tool that makes us better but you know what if you have a tool and you just deploy it and you don't

[00:34:48] [SPEAKER_03]: know how to use it effectively you've heard me say this before right my my old example is you

[00:34:54] [SPEAKER_03]: give me the world's best table saw and I'm less likely to use that very sophisticated implement

[00:35:01] [SPEAKER_03]: to make you some heirloom quality furniture than I am to cut off my own thumbs right because I don't

[00:35:06] [SPEAKER_03]: know how to use that tool that's not what I do AI is exactly the same thing do not expect it to do

[00:35:12] [SPEAKER_03]: your job be a conscientious adopter and a professional at using that tool it'll make you

[00:35:20] [SPEAKER_03]: better but if you assume I can just replace a person with that tool and it'll be good enough

[00:35:27] [SPEAKER_03]: you know what just tell me who you who your customers are because I'd like to follow up and

[00:35:32] [SPEAKER_03]: maybe have a customer support conversation and Ryan I'm not gonna give you a table saw I won't

[00:35:38] [SPEAKER_02]: give you a table saw so I think we covered that one I want to address a couple of trio of questions

[00:35:44] [SPEAKER_02]: from YouTube this time I'm going to paraphrase them a little bit because not sure 7874 asked

[00:35:50] [SPEAKER_02]: three targeted questions about my if I was starting an msp video and I wanted to address them

[00:35:56] [SPEAKER_02]: first not sure wanted to know about my use of in tune and then my statement of remote control

[00:36:02] [SPEAKER_02]: tools based on into I want to take a moment and clarify my thinking when I'm talking about building

[00:36:08] [SPEAKER_02]: a person when I was thinking about the solution if I was building an msp today and where would

[00:36:13] [SPEAKER_02]: base my tool is is I would base my technology around into and I would use them as my rmm and

[00:36:20] [SPEAKER_02]: my thinking is that if there are other tools that I decide I need in the business they have to be

[00:36:28] [SPEAKER_02]: compatible with into I'm not necessarily saying you must include other tools what I'm saying is a

[00:36:34] [SPEAKER_02]: pre-requisite of adding tools is that they must be compatible with the solution into that I have

[00:36:41] [SPEAKER_02]: selected for this so that's what I'm thinking is it's not necessarily saying you definitely

[00:36:46] [SPEAKER_02]: need more tools what I'm saying is if I decide I need more tools as the business evolves they must

[00:36:53] [SPEAKER_02]: be compatible with into also I made a comment in the video about blocking competitors by offering

[00:37:00] [SPEAKER_02]: infrastructure services in this case I don't mean technically block them not sure took this to mean

[00:37:07] [SPEAKER_02]: in in tune specifically no absolutely not Microsoft is very explicit that you can't block people

[00:37:14] [SPEAKER_02]: you can't let you know you can't take them from allowing them to move to another client

[00:37:18] [SPEAKER_02]: what I'm saying is that I would make sure my new modern solution provider where remember my premise

[00:37:25] [SPEAKER_02]: is that I'm going to work very heavily in the productivity stack I'm going to make sure that

[00:37:31] [SPEAKER_02]: I offer infrastructure services and if I don't want to build those out and likely me Dave Sobel

[00:37:38] [SPEAKER_02]: does not want to do so I'm going to bring in a partner another msp that wants that work or

[00:37:43] [SPEAKER_02]: with more some of my more modern thinking on this is I would bring in a distributor

[00:37:47] [SPEAKER_02]: as part of this if you watch my future of distribution video I've got a whole section

[00:37:52] [SPEAKER_02]: on using how I would bring in distributors to fix to fill the holes in my offering

[00:37:58] [SPEAKER_02]: this is the way that I would do that and finally the third question not sure asked me was about

[00:38:04] [SPEAKER_02]: wanting Zapier integration I want to say that first off I was using Zapier as an example

[00:38:10] [SPEAKER_02]: I tend to like Zapier I use it in my own business as an integration platform but the key

[00:38:16] [SPEAKER_02]: is that I'm selecting tools that have modern API driven interfaces one of the things that we see

[00:38:23] [SPEAKER_02]: in a lot of older tools is that it is not something that is easy to link with with APIs

[00:38:29] [SPEAKER_02]: I was talking with a guest just this week and we mentioned the idea of working with line of

[00:38:35] [SPEAKER_02]: business applications for example there's a lot of EMR and dental practice ones for dentists

[00:38:40] [SPEAKER_02]: and what we want to make sure is we want to make sure that we can get to the data contained within

[00:38:44] [SPEAKER_02]: that but most legacy solutions don't have really good API's that allow us to pull that data

[00:38:52] [SPEAKER_02]: and so what we're looking for in modern solutions as much as possible

[00:38:56] [SPEAKER_02]: is that we're looking for those API driven ones that allow us to extract data and then do things

[00:39:03] [SPEAKER_02]: with it particularly in our own business as an MSP solution provider we're going to run our own

[00:39:10] [SPEAKER_02]: business on ones that are very modern now I want to make one last statement about this

[00:39:15] [SPEAKER_02]: I want to emphasize that I Dave Sobel and my role as host of the business of tech

[00:39:19] [SPEAKER_02]: do not spend a lot of time reviewing and selecting technical tools my job is not to

[00:39:26] [SPEAKER_02]: understand every intricacy of every RMM in the market it is instead to understand what those

[00:39:32] [SPEAKER_02]: companies are doing to take advantage of market opportunities where are they moving where are they

[00:39:39] [SPEAKER_02]: investing where do they believe their next growth opportunity is and advise on their business

[00:39:45] [SPEAKER_02]: strategies there's lots other great people out there in the MSP space that will help you with

[00:39:50] [SPEAKER_02]: tool selection when I talk about tools I'm doing so in a general sense in order to give you a

[00:39:57] [SPEAKER_02]: sense of why I might pick a particular category of tools and then I will use one as an example

[00:40:03] [SPEAKER_02]: not sure I hope that I'm going to post this in the YouTube video I hope this helps you with your

[00:40:07] [SPEAKER_02]: answers and we will continue to take questions online as much as possible Ryan this has been

[00:40:13] [SPEAKER_02]: great fun I appreciate you joining me for the Q&A discussions as well look forward to

[00:40:18] [SPEAKER_02]: having you on for future conversations absolutely thank you very much for having me sir

[00:40:24] [SPEAKER_02]: well Shreya Kushek of Rackspace is my guest on this coming weekend's episode in the interview we dive

[00:40:31] [SPEAKER_02]: into why sustainability matters but in this clip which I think is relevant to our conversation

[00:40:37] [SPEAKER_02]: today he talks about the AI framework that Rackspace uses here's a preview of that interview

[00:40:43] [SPEAKER_01]: three s's that we talked about symbiotic secure and sustainable is what we call

[00:40:49] [SPEAKER_01]: responsible AI right and for us as Rackspace we felt like you know what we're known for in the market

[00:40:58] [SPEAKER_01]: is we manage and run production workloads we've done that for 25 plus years right we know how these

[00:41:06] [SPEAKER_01]: systems run in production and we're trying to bring that expertise to customers to say let us implement

[00:41:14] [SPEAKER_01]: those AI solutions that are responsible in these and meet these three criteria that I just talked

[00:41:21] [SPEAKER_02]: about that's that's kind of what we do now it's a cool fascinating conversation about how Rackspace

[00:41:27] [SPEAKER_02]: is using AI but also how they're leaning into sustainability and what they're doing in terms

[00:41:33] [SPEAKER_02]: of that benefits and you can understand that as a data center provider they think a lot about

[00:41:38] [SPEAKER_02]: energy usage now my patreon supporters already have this video if you want to listen to it now

[00:41:43] [SPEAKER_02]: you can support the show at patreon.com slash msp radio and get access to that video right now

[00:41:50] [SPEAKER_02]: and all my interview episodes early it'll drop on the weekend on youtube and the podcast feed

[00:41:56] [SPEAKER_02]: you're interested I encourage you to go listen to it now I want to thank salesbuilder who's

[00:42:08] [SPEAKER_02]: the power of automation and visit them at salesbuilder.com that's buildr.com and vendors you too can get

[00:42:17] [SPEAKER_02]: your name mentioned on the live show it's a simple monthly subscription visit patreon.com slash msp radio

[00:42:24] [SPEAKER_02]: to learn more and if you're listening this you can support the show most easily by just liking

[00:42:30] [SPEAKER_02]: sharing and following on your favorite platforms that is the number one thing that makes a

[00:42:36] [SPEAKER_02]: difference in the production and value of this show make sure you tell a colleague if you want to

[00:42:42] [SPEAKER_02]: support directly and get access to those videos early you can support with our give what you want

[00:42:47] [SPEAKER_02]: model you set what you think the content is worth and you can visit patreon.com slash msp radio

[00:42:53] [SPEAKER_02]: to sign up if you have a question and you're listening to the recording send it in at question

[00:42:59] [SPEAKER_02]: at msp radio.com really appreciate you joining me for the business of tech lounge

[00:43:04] [SPEAKER_02]: and I will see you next time