We have discussed what happens when a vendor is just gone. What happens when a vendor has an outage or some business disruption? Do you jump to another vendor? Do you plan to weather the storm, or do you pivot? Eric Hanson of Inland Productivity Solutions and I sit down and discuss the challenges faced with business disruptions and failures that might require a pivot... Then again, they might not.
[00:00:06] Welcome to MSP 1337. I'm your host, Chris Johnson, a show dedicated to cybersecurity challenges, solutions, a journey together, not alone.
[00:00:21] Welcome everybody to this episode of MSP 1337. Today, we are talking about the interim.
[00:00:30] We had this great conversation a few weeks back where we talked about what happens if your vendor's gone, like permanently.
[00:00:37] We were doing incident response conversations around it. A lot of MSPs I know did lose some sleep over it.
[00:00:45] And I think that it really just lends itself to doing a better job of having the tabletop exercise of being able to make good decisions around if you were to have to switch vendors.
[00:00:56] But one thing we didn't talk about, and I want to say welcome to the show, Eric Hansen of Inland Productivity.
[00:01:02] I should probably have done that at the beginning, but Eric, welcome to the show.
[00:01:04] Well, thanks for having me, Chris.
[00:01:08] The interim problem.
[00:01:10] The interim.
[00:01:12] For those of you that have listened to some of the banter that have gone back and forth with Eric and I,
[00:01:17] sometimes we explore the things that maybe we shouldn't because that just causes others to have undue stress.
[00:01:24] But I think this is an important one because we left something out in that previous conversation,
[00:01:29] which brings us to today, and that is if you are in a situation where your cloud vendor's down or your on-prem vendor solution has been locked up,
[00:01:42] you know, we can think of CDK and some others, what do you do during the interim?
[00:01:47] You're making a decision to wait it out.
[00:01:49] You're making a decision to shift to another vendor.
[00:01:54] I'm not sure I can think of a really good answer to this one, Eric.
[00:01:57] Like, what do you do in the interim?
[00:02:01] Well, first off, fly the airplane.
[00:02:03] Don't panic.
[00:02:06] That's one.
[00:02:07] Black stuff.
[00:02:09] There you go.
[00:02:10] Something like that.
[00:02:11] Get the thing to best glide.
[00:02:14] Right.
[00:02:14] And, yeah, I think that it is an important thought, maybe written procedure to have what happens in that interim.
[00:02:29] You know, we've seen outages with Microsoft and other vendors.
[00:02:38] I think if you're in AWS East, you lived a lot of pain at one point.
[00:02:42] Sure.
[00:02:45] Where, okay, well, you're down.
[00:02:48] So what do you do?
[00:02:49] Well, you're down for an hour.
[00:02:51] You're down for two hours.
[00:02:53] That's inconvenient.
[00:02:54] It's annoying.
[00:02:57] But is it business ending or, you know, significant downturn for a quarter or two ending?
[00:03:08] No.
[00:03:09] That's an inconvenience.
[00:03:10] The reality is.
[00:03:13] Well, let's talk about the inconvenience.
[00:03:15] I've actually had this conversation before.
[00:03:16] I've used an example where I was on the internal IT side where you were the external IT.
[00:03:24] Do you remember when a car hit a power pole and popped a few transformers and we suddenly didn't have power in the office?
[00:03:32] And we weren't going to get power back very quickly.
[00:03:36] Yeah.
[00:03:37] And we had a similar connectivity problem too.
[00:04:09] Right.
[00:04:10] We immediately had a conversation that took action to put things in place so that that particular problem couldn't happen again.
[00:04:17] The thing that I found interesting about it, though, is at what point, I mean, we don't have power back.
[00:04:24] Do we continue to have people hanging out at the water cooler before we just send them home?
[00:04:29] And I think that was part of the conversation we had on a previous episode, which is like sometimes inconveniences can cause pretty catastrophic results if you continue to wait too long to do something about it.
[00:04:43] I think similarly, anything in healthcare, if you have any sort of scenario where you can't operate as normal, you have to have a plan B for the temporary, which can be temporary might be a long, you know, our definition of temporary is not all equal.
[00:05:02] Yeah, I know for sure.
[00:05:04] I mean, temporary, we actually had a client that had a vault fire and their temporary internet out.
[00:05:11] It melted all the fiber optic cable.
[00:05:14] Their temporary outage was months.
[00:05:21] Fortunately, they were right across the street from us.
[00:05:23] They had power back in a couple of days, but it was literally months before they got connectivity back.
[00:05:33] We got completely lucky and we had a plan B for them and the hardware to do it.
[00:05:37] But it was literally pure luck that we had everything in place.
[00:05:43] But, you know, we think about these temporary outages as issues of infrastructure.
[00:05:50] You don't have an internet connection.
[00:05:52] You don't have power.
[00:05:53] Okay, well, take a CDK level problem with your hosted PSA or your hosted RMM.
[00:06:05] Now, what do you do?
[00:06:08] Well, and I think up to this point, we've always, and this isn't specific to our industry and the tools that we use.
[00:06:16] I don't care what EHR or EMR system you might use.
[00:06:19] There's always the possibility that exporting the data and trying to plug it into something else is anything but smooth.
[00:06:28] It's a lot of other key word phrases that you might use to describe it.
[00:06:33] Well, anybody who's been through a data migration knows that.
[00:06:36] Right.
[00:06:38] I've even seen where there's examples of like to import data into our platform.
[00:06:42] Here is the template.
[00:06:43] You're like, well, that template doesn't match what you're looking for on the import.
[00:06:47] These aren't the same data fields.
[00:06:49] You gave me the template and you had me export a sample, you know, example.
[00:06:54] And those columns don't match the one you want to see for import.
[00:06:57] So I've seen that too with vendors, right?
[00:06:59] It raises the question, though, it's not just about having my data or having, you have to have a functioning working tool in order for this to work.
[00:07:16] So it makes me raise the question of interim of if I were, as we had talked about, you know, switching before, like if I had to go from RMM, you know, something warrior to, yeah.
[00:07:30] To fill in the blank.
[00:07:32] To do a test.
[00:07:34] To like, and even go so far as I wonder if there are vendors that would actually be willing to have this conversation that says, we know that we're not perfect and that there isn't the possibility that someone could take us out.
[00:07:47] But what if we had a path as a vendor to another vendor where we could actually have your data go into a different platform, allow you to see it, feel it, touch it, test it, and call it a day.
[00:08:01] So like, if we're out, if we're the CDK, here's Car Software X that can actually facilitate you for the next 60 days.
[00:08:11] And I realized that was a very pie in the sky trick on the quest.
[00:08:15] But isn't that kind of what we're talking about?
[00:08:18] Like, it's not like you're going to build something on-prem out of thin air.
[00:08:21] No, no.
[00:08:23] If you have something on-prem, you probably have a plan to bring that back.
[00:08:31] You know, if it's on-prem, you've probably got it on some vendor's system that can virtualize it locally.
[00:08:40] So that maybe takes care of your problem, although there are some of those on-prem systems that reach out to your vendor's cloud.
[00:08:50] I won't say who.
[00:08:52] Licensing not available offline.
[00:08:55] Exactly.
[00:08:56] Licensing not available or authentication not available.
[00:09:02] We've seen both of those.
[00:09:04] Yeah, this hurts SSO, doesn't it?
[00:09:05] This kills SSO.
[00:09:09] And MSA and?
[00:09:12] And, and, and, and, right.
[00:09:14] Yeah.
[00:09:14] So, you know, I'm not necessarily advocating that on-prem or off-prem is better or worse.
[00:09:22] It's just something that you need to have thought through.
[00:09:25] But I think the current competitive landscape to some degree probably removes the incentive.
[00:09:39] Yeah.
[00:09:40] The raise the tide for all boats.
[00:09:42] For, for, yeah.
[00:09:44] I mean, I don't, I don't think, you know, vendor A is going to want to build that plan to get to vendor B.
[00:09:50] Right.
[00:09:51] And to take some lessons from the CDK mess, you know, you had basically half of the dealerships in the country couldn't transact business for a couple of weeks.
[00:10:01] Right.
[00:10:01] That's a significant.
[00:10:02] What was in this country?
[00:10:04] Well, right.
[00:10:05] Right.
[00:10:06] But in this country, it took out about half of the, about half of the vendors or about half of the dealerships.
[00:10:13] And what did they do?
[00:10:14] They found workarounds.
[00:10:15] I mean, they were kludgy as hell.
[00:10:17] I know of one dealership that was, okay, yeah, we can sell you this car.
[00:10:21] Hang on.
[00:10:21] Let me get my car, drive down the street to the Mazda store that my cousin works at so I can run your credit.
[00:10:28] Right.
[00:10:29] Right.
[00:10:30] And then they'd come back and I mean, they did sell cars that way.
[00:10:34] They did handwritten contracts.
[00:10:35] They did a bunch of things, but we are in an industry that is entirely reliant on tech.
[00:10:41] Yeah.
[00:10:42] You know, if you're, okay, you lose your RMM.
[00:10:46] Well, that's cool.
[00:10:47] If all of your customers are local.
[00:10:49] I mean, it's not cool, but sucks bunch, but at least you can still go to that customer.
[00:10:54] Yeah.
[00:10:55] If you're a luxury car dealership that specializes in getting, you know, rare cars for someone all over the country, that's a whole different.
[00:11:02] That's a whole different ballgame.
[00:11:04] If you are a, you know, if you're a regional MSP and you don't have boots on the ground and relationships to get those boots on the ground in the places where you need the boots on the ground, because now I don't have, you know, let's say, let's say it's the RMM.
[00:11:22] You lose the RMM.
[00:11:23] The RMM is going to be down.
[00:11:26] You know, there's lots of different ones.
[00:11:29] Do they meet my operational criteria perfectly?
[00:11:33] Probably not.
[00:11:34] But can I do the break fix and things that have to happen with another tool quickly?
[00:11:44] Yeah.
[00:11:45] You know, you may, if.
[00:11:47] It's the paper world, right?
[00:11:48] This goes back to healthcare and having to do responsive systems on a piece of paper because the EMR systems offline.
[00:11:56] But it.
[00:11:58] I guess the question that I have is, and I have this really bad scenario go through my head that I'm going to say is more or less not likely, but to the.
[00:12:07] Until it is.
[00:12:08] I was thinking so.
[00:12:10] So 365 is offline.
[00:12:13] And my laptop reboots and it's prompting me to put in the key or decryption.
[00:12:23] I now have no path forward.
[00:12:27] Because I need, well, the key that's living in 365 that it's not local to where I am.
[00:12:33] Anyway, they just had this.
[00:12:35] Right.
[00:12:35] Right.
[00:12:35] So that's a, right.
[00:12:37] So that's a documentation problem, right?
[00:12:40] Should you be documenting the key in two places, right?
[00:12:43] It can, it can absolutely live two places, but you know, and, and we had an example of that.
[00:12:49] What?
[00:12:50] A month after, less than a month after CDK, you had an example of that where people couldn't get to those keys.
[00:12:57] Right.
[00:12:57] Because they couldn't log in whatever it is they needed to log into.
[00:13:03] To get those keys.
[00:13:05] These are all bad things that we don't want to have to deal with.
[00:13:08] So I was thinking about to the premise of this episode.
[00:13:11] We're in the business of bad things.
[00:13:14] Right.
[00:13:14] So we're, we're in this interim space and I think you described one scenario, which is like,
[00:13:19] what's the bare minimum, regardless of what tools are going to be used,
[00:13:23] that would allow you to function when another set of tools is down.
[00:13:27] And we talked about some of this on the previous episode about,
[00:13:29] you know, you're using TeamView or some other app to allow you to remote connect to an end user machine.
[00:13:35] Great.
[00:13:36] And a lot of those things are pretty low on the value add.
[00:13:40] Because if we're talking about things like automated patching and that kind of thing,
[00:13:44] well, now you're talking about 30 days plus have gone by and you still don't have access to this,
[00:13:48] say, RMM tool.
[00:13:50] But the, the internal piece, which I think kind of is how this all started is,
[00:13:55] how do I function internally?
[00:13:57] If one of those systems, whether it's cloud or on-prem goes away,
[00:14:02] what does pivot look like without sort of reinventing the whole thing?
[00:14:09] Because that's the part we talked about last time was you're just moving to another vendor.
[00:14:15] That's not necessarily a solution to the problem.
[00:14:19] Right.
[00:14:20] Right.
[00:14:21] And you're treating the symptom.
[00:14:24] Right.
[00:14:24] Because more than.
[00:14:27] Yeah.
[00:14:27] I think though, when we talk about the tool and I leave one vendor to go to the next,
[00:14:33] it, it may, it may be more than solving for a symptom, but, but the,
[00:14:38] the kicker is when they come back online, do you want to go back?
[00:14:42] Or are you going to say, I'm never going back because this happened to this vendor.
[00:14:45] And now we're back into that knee jerk reaction space that says this vendor was bad.
[00:14:49] So I'm moving to this vendor that hasn't had bad thing happen to them.
[00:14:53] Yeah.
[00:14:54] Yeah.
[00:14:55] Right.
[00:14:56] Well, that's why I said, I left it kind of hanging because yet or not,
[00:14:59] you have no way of knowing it may never happen.
[00:15:02] It might happen tomorrow.
[00:15:03] Right.
[00:15:04] Right.
[00:15:04] Well, I mean, there was a password vendor a couple of years ago that had a pretty significant problem.
[00:15:10] And, um, it wasn't handled well, you know, at first it really didn't affect them up front.
[00:15:16] No, but.
[00:15:18] Why is it no?
[00:15:20] Yeah.
[00:15:20] Well, I mean, I got one of those, you know, I can, I can get you one.
[00:15:26] Let me, let me, let me order you one off Amazon.
[00:15:29] It comes all down.
[00:15:31] This is password manager.
[00:15:32] It's one that says password.
[00:15:33] It's password.
[00:15:34] Yeah.
[00:15:34] Password, password book.
[00:15:36] Yeah.
[00:15:36] Yeah.
[00:15:37] Just, uh, you know, add an asterisk onto the end of all those passwords.
[00:15:41] Leave that part out.
[00:15:42] That's right.
[00:15:45] I mean, I can see that one being your, your break glass binder that sits in a physical
[00:15:51] story.
[00:15:51] Right.
[00:15:52] Your password manager is what's gone today.
[00:15:54] What gives you the ability to go change anything?
[00:15:57] Well, right.
[00:15:58] You know, do you, when was the last time you printed out a list of all your customers?
[00:16:01] And I think you guys talked about that on the, on that episode as well.
[00:16:05] We did.
[00:16:06] You know, when, and, and where is that?
[00:16:10] And how often should you print?
[00:16:12] So we've, it's funny.
[00:16:13] We had this conversation today about, um, vendor management and how often the vendor management
[00:16:18] should be done.
[00:16:19] And I said, one of the things that, that I would want to know is, um, what's the phone
[00:16:23] number if I have a problem?
[00:16:26] Does it work?
[00:16:27] And, and seriously, like, does it work?
[00:16:29] Cause I've had an example given to me where they did a tabletop exercise for their cybersecurity
[00:16:34] insurance, took 72 hours to finally get to the right department and the right person
[00:16:38] to help them finish the tabletop exercise.
[00:16:41] Cause they had, oh, we changed phone systems and no one has called in since that change
[00:16:46] that we know of, uh, to test the, to find out that it doesn't work.
[00:16:52] Yeah.
[00:16:52] Right.
[00:16:53] Yeah.
[00:16:53] Yeah.
[00:16:54] Well, I, that, that was literally going to be the example I was going to use the cybersecurity
[00:16:58] insurance vendor, because that's often your first stop now.
[00:17:02] Right.
[00:17:03] You got to call them before you do anything.
[00:17:06] Yeah.
[00:17:06] And we even had, we, we actually went down the rabbit hole a little bit with this.
[00:17:09] Like what's the responsibility matrix?
[00:17:11] Because one person's probably not managing 70 vendors by themselves.
[00:17:15] You know, who's the, who's the, you know, the, the liaison kind of like account management,
[00:17:20] only not the vendor.
[00:17:21] It's the MSP side, you know, to be able to say, I've taken ownership of Datto or Kasey
[00:17:26] or whoever it is.
[00:17:27] And these are our account reps and all those details.
[00:17:30] But what about when something happens and who's my backup in case I'm not here when something
[00:17:35] happens, who would have, who do I go to next?
[00:17:38] And I think this kind of goes to the pivot.
[00:17:40] Like sometimes you could, I think you could make a decision to go to another vendor and
[00:17:46] you've made a very poor decision because the communication path with the existing vendor
[00:17:52] that's having the problem might've actually already had a path in place to solve for where
[00:17:57] you're currently at right now.
[00:17:59] But they weren't on your priority list to get the phone call off to because we're, we're
[00:18:03] back in that hole.
[00:18:04] My vendor screwed up.
[00:18:05] We just don't say that about Microsoft, AWS, Google, AT&T, Verizon, fill in the blank.
[00:18:11] We don't because we assume that because of how big they are, we're at their mercy anyways.
[00:18:16] Right.
[00:18:17] You're at their mercy and, and they're going to get it fixed eventually.
[00:18:20] And eventually it's hopefully not very long.
[00:18:24] Right.
[00:18:25] But, you know.
[00:18:26] Someday and not today are not things you want to hear.
[00:18:30] Yeah.
[00:18:30] Unless you're that, you know, with that cloud vendor that accidentally deleted your account
[00:18:35] and thank God you had a backup.
[00:18:37] But.
[00:18:38] So should you have pivot, a pivot for all of your vendors?
[00:18:43] Like how do you, how does one even weigh this from a.
[00:18:49] From business perspective.
[00:18:52] Yeah.
[00:18:52] I mean, I think we could talk about risk all day long and critical and probabilities,
[00:18:57] but I don't think to what we've talked about so far, none of the things that we've talked
[00:19:03] about have been, would have been foreseen based on say, doing a risk assessment of the
[00:19:09] vendor.
[00:19:11] Right.
[00:19:12] No, I agree.
[00:19:13] I agree.
[00:19:14] I don't know.
[00:19:15] I think you probably need to look at your, you know, define what your critical vendors
[00:19:23] are.
[00:19:24] You know, is your.
[00:19:26] Core four.
[00:19:27] Okay.
[00:19:28] And what core four are those in your mind?
[00:19:31] If I was, if I was as a, as a no longer an MSP, they've changed.
[00:19:36] I would, I would say, so my financial package, so that would be QuickBooks or whatever it is
[00:19:41] that we're using to address billing and payroll.
[00:19:44] I would say my 365 or whatever my communication core communication tool is.
[00:19:51] So usually that's the one that comes to mind.
[00:19:54] I would have my PSA.
[00:19:56] Okay.
[00:19:56] And then probably my RMM tool or something adjacent to what would be traditionally called
[00:20:02] an RMM.
[00:20:04] Okay.
[00:20:05] So there are obviously others, but there are, oh no, there are obviously others and many
[00:20:10] others, depending upon what your stack looks like.
[00:20:13] There could be many, many, many of them.
[00:20:15] Like I'm probably not jumping to the SIM and sword tools out of the gate, but like, no,
[00:20:20] I agree.
[00:20:21] I don't have the core four in place.
[00:20:22] They're not very useful.
[00:20:23] I would argue there's core five.
[00:20:25] You got to have your backup tool, whatever that is.
[00:20:28] And that's a mixed environment.
[00:20:29] That's fair.
[00:20:30] I think.
[00:20:32] So you've got those five, you know, if your, if your DNS tool goes offline, yeah, it's
[00:20:40] inconvenient, but I can fix that.
[00:20:42] Yeah.
[00:20:43] Um, you know, I've got lots of, lots of paths to, to address that.
[00:20:48] It's the things you don't have easy paths to, you know, let's say a, one of the big PSA
[00:20:56] vendors just breaks.
[00:20:59] And it seems like it's not going to be fixed real quick.
[00:21:04] You probably have, you know, if, unless you're hardcore, you know, diehard, this is, this is
[00:21:12] my vendor and this is who my vendors always going to be.
[00:21:16] Um, you've at least looked at others.
[00:21:19] You have an idea of where you might go if it hits the fan.
[00:21:25] Um, and let's say it, that entire vendor gets taken out.
[00:21:31] Okay.
[00:21:31] So I'm gonna, I can't use my, my, um, the tool, no working, the tool, no working from
[00:21:41] my Florida company.
[00:21:43] Cause they just got hit with a hurricane.
[00:21:44] Right.
[00:21:45] Sure.
[00:21:46] Um, and, and there were some core thing that there was, you know, one, one machine and it's
[00:21:52] irrecoverable.
[00:21:53] So they're going to be out for three weeks.
[00:21:55] Do you pick that up and move it to another tool?
[00:22:00] Do you work around it for what could be an unknown period of time?
[00:22:06] Sure.
[00:22:07] Um, or do you go, you know, I was looking to make that change to these other guys anyway.
[00:22:14] So now you and everybody else or many other people using that same tool need to make a
[00:22:22] move.
[00:22:23] Does vendor B or vendor C, or even a combination of vendor B and C, can they onboard that many
[00:22:31] people?
[00:22:32] Do they have, does their cloud have the capacity to do it?
[00:22:37] Do they have the capacity to work through the issues that require conversation?
[00:22:44] We're just moving the data.
[00:22:46] Well, yeah.
[00:22:46] Yeah.
[00:22:47] Oh, that's just magic, isn't it?
[00:22:49] I mean, it's just computers.
[00:22:50] I was going to say, you know, back up a little bit and magic hackers.
[00:22:53] Yeah.
[00:22:53] I just think about, you know, backing stuff up today is largely, you know, a second, we
[00:22:58] don't even give it a second thought.
[00:22:59] Data moves relatively fast in most cases are fast enough because we don't need it.
[00:23:04] Right.
[00:23:04] It doesn't matter what speed is, as long as it gets to completion at some point in time
[00:23:08] that I'm comfortable with.
[00:23:09] The other direction though, we're never comfortable with it.
[00:23:12] Right.
[00:23:12] It's never fast enough.
[00:23:13] I don't care.
[00:23:14] I can't imagine anybody saying, oh yeah, it was great.
[00:23:17] Like we got it back within the 15 minutes that I wanted it to.
[00:23:20] No, it might've been 45 minutes and we're probably like, fingers crossed, we got it all
[00:23:24] right.
[00:23:26] But when you start moving data long distances and they have to traverse more than just from
[00:23:31] an endpoint to the cloud, they now have to go cloud back to another endpoint at another
[00:23:35] location.
[00:23:36] Things start to get squirrely no matter how good.
[00:23:39] The vendors are.
[00:23:40] And like, to your point, it raises questions though.
[00:23:43] Like, okay.
[00:23:43] To your point about if I migrate, well, now we're back to that whole knee jerk reaction.
[00:23:48] I don't care if it's, that's the plan.
[00:23:49] It feels like unless the plan is really written out and says, if we know that they've already
[00:23:55] committed to, you're going to be down for more than 72 hours.
[00:23:58] And now you want to kick in this sort of, um, IR strategy that's going to move.
[00:24:05] Yes.
[00:24:05] You're breaking glass.
[00:24:06] Probably a bad idea.
[00:24:07] Yeah.
[00:24:08] Yeah.
[00:24:08] Well, I mean, just think about what, what we do in the event of a, you know, a hardware
[00:24:15] outage is everybody's first response.
[00:24:18] I'm assuming there's a BDR on, on site in this case is everybody's first response spin
[00:24:24] up the VM.
[00:24:26] Right.
[00:24:27] Or is it, all right, let's see what this is going to take to fix.
[00:24:31] Because if this can be fixed with a part in four hours or five hours from pick your favorite
[00:24:37] hardware vendor, is it worth the organizational headache of then having to repopulate all that
[00:24:44] data back?
[00:24:47] And that kind of went back to the example I gave that you and I went through after we were restoring
[00:24:52] things.
[00:24:53] I mean, cause we really weren't truly back to real operational function for months.
[00:25:01] Well, no, it was longer than that.
[00:25:03] There were a couple of things that were just development that, yeah, we got to start that
[00:25:07] over again.
[00:25:07] I was going to say there were things that I had built that took a lot longer to bring
[00:25:11] back to life because there were so many moving parts that had been built over so many years.
[00:25:16] It couldn't just be, you know, boop and it's done.
[00:25:20] Yeah.
[00:25:21] Yeah.
[00:25:21] I mean, we would be closer to that today, I think, but still.
[00:25:25] Sure.
[00:25:27] You know.
[00:25:27] Which goes back to how long, like what makes sense, what impact on business for pivoting during
[00:25:34] the inconvenience window.
[00:25:36] When does inconvenience become not an inconvenience anymore?
[00:25:41] And I think that was the, you know, the example I gave with power outage and some others is
[00:25:44] that we're not good at even having that conversation.
[00:25:47] Yep.
[00:25:47] Time for everybody to go home or, Hey, you're working from home.
[00:25:50] So if all your employees work from home, at what point when they don't have internet and
[00:25:55] power at home, do you not encourage them to say, I don't know, go somewhere else?
[00:25:58] Assuming that it's not because of, you know, that that's some regional disaster.
[00:26:04] Yeah.
[00:26:04] Um, but we see this kind of thing happen all the time.
[00:26:08] And when you talk to the organization, they often didn't really have a plan.
[00:26:11] So they just kind of accepted whatever the employee did.
[00:26:15] Right.
[00:26:16] Well, I, and that, that goes back to planning in general for these types of outages, you
[00:26:22] know, whether it's our organizations or customer organizations, I think very few really, really
[00:26:32] think about, okay, I can't get into this building for a week.
[00:26:38] What do we do?
[00:26:41] Right.
[00:26:42] I'll bet you.
[00:26:44] I mean, I would bet it's in single digits, the number of companies that truly have an
[00:26:50] executable.
[00:26:53] And I don't even want to say tested, but an executable plan to do that.
[00:27:03] Yeah.
[00:27:03] I think that's kind of where it's a different interim than where we started.
[00:27:08] Yeah.
[00:27:09] Well, I think it all comes down to one thing.
[00:27:12] I think your CDK, the scenarios there, what we saw with CrowdStrike, we've seen examples
[00:27:19] of this happen with other vendors or just even things that aren't even tied to the vendor,
[00:27:25] but more what is impacting the MSP based on geography or any number of things.
[00:27:28] But the MSP has never really considered what does the model look like?
[00:27:34] Not when I think about switching vendors because of the situation, but thinking about
[00:27:39] what are, what is the game plan for, for weathering the storm?
[00:27:43] And I think that kind of really, to me is what interim means is how long can we weather this
[00:27:49] storm before we recognize we've made a really bad decision and figuring out how to weather
[00:27:54] the storm to a point that's a whole lot less than that.
[00:27:59] Yeah.
[00:28:00] No, there's, there's a lot to think about here that I think has been, has been missed.
[00:28:07] Yeah.
[00:28:08] Because, you know, here happened.
[00:28:10] Go ahead.
[00:28:11] Oh, I was going to say, you just, you can't think of every potential scenario.
[00:28:16] No.
[00:28:18] I had an ISP here that they decided to not be an ISP anymore.
[00:28:25] So they sold their infrastructure.
[00:28:28] Sure.
[00:28:28] No.
[00:28:28] Well, funny.
[00:28:29] They sold their infrastructure to another service provider.
[00:28:33] And basically all of the legacy fiber that had been deployed 20 plus years ago, the new
[00:28:40] ISP really doesn't care about it.
[00:28:43] They just want the existing customers that are on, whether it's legacy or not, that are
[00:28:48] on active fiber.
[00:28:49] So any dark fiber that's legacy, they're essentially going to abandon.
[00:28:53] Well, for a place that I used to work, that's somewhat problematic because there's a lot of
[00:28:59] legal contracts around who can and can't touch this fiber in the event that strands are going
[00:29:05] to get used.
[00:29:06] Well, there's another fiber company that knows that all of this fiber exists in the ground.
[00:29:11] It's abandoned.
[00:29:14] Who has access to bring things back up?
[00:29:17] So the point of anybody that would have had ownership of the dark fiber, but have to use
[00:29:23] the carrier for servicing the fiber, they're now in this weird legal predicament of can't
[00:29:28] use.
[00:29:28] It's not quite the same as what we're talking about for the examples given, but can you
[00:29:34] imagine going, okay, so we're getting ready to roll out another fiber redundancy for our
[00:29:41] environment.
[00:29:41] And then you find out that you're in pivot mode because you can either go with a completely
[00:29:45] different carrier or you can see how long can you wait, possibly get it for free.
[00:29:53] Because you already have the fiber.
[00:29:55] You just need someone to buy the service or to terminate it where you need it terminated
[00:30:00] to have the resources to do that.
[00:30:03] So it's a small cost, right?
[00:30:04] But we're not talking about running fiber for miles because the fiber already exists.
[00:30:09] Because the fiber already exists.
[00:30:11] Yeah.
[00:30:12] Yeah.
[00:30:12] All over in this distributed environment that you may have worked in.
[00:30:17] Yeah.
[00:30:18] Right.
[00:30:18] Yeah.
[00:30:19] Where you want to take advantage of that dark fiber.
[00:30:22] Right.
[00:30:22] I think this is the same problem that you have when as an MSP, you're making decisions
[00:30:27] because something has happened with a vendor.
[00:30:30] They're acquired.
[00:30:32] They have, they're going through, they're talking about going, filing an IPO.
[00:30:37] I mean, there's so many different things that can happen with a vendor that you go up or
[00:30:42] switching vendors.
[00:30:43] What's your pivot window that says, I am, you know, shelter in place right out the storm.
[00:30:49] Don't make those decisions just because of what is happening with a vendor in 15 minutes
[00:30:55] into the situation is putting you in the, well, I'm going to go over here where it's
[00:30:59] today safer than where I'm at right now.
[00:31:02] The grass seems to be greener on this side of the fence.
[00:31:05] The problem is that some other dude painted it green from your side of the fence.
[00:31:10] That's right.
[00:31:11] And you don't know it's dead grass.
[00:31:12] In fact, it's actually a mirror just looking at your own yard.
[00:31:16] Yeah.
[00:31:17] That's even worse because you get on the other side of that mirror and you got problems.
[00:31:22] Yeah.
[00:31:23] Who's been watching?
[00:31:26] Right.
[00:31:26] Well, or, you know, you're, you're making a decision to go to somebody who's, you know,
[00:31:31] on their, on their third round of funding and they're running out and there may not be a
[00:31:34] fourth round of funding.
[00:31:36] Um, you know, do you organizationally want to go through that pain?
[00:31:41] So how do you work through the middle?
[00:31:45] I, I think that's the kicker.
[00:31:48] So obviously today we've come up with no answers to these problems.
[00:31:52] However, well, you know, there you go.
[00:31:55] The way you get an answer to the problems first recognizing you have it.
[00:32:01] Yeah.
[00:32:02] And I think the reality is we could probably do five more episodes on this topic and have
[00:32:09] very different conversations about it because I think there's really no one right answer,
[00:32:14] right?
[00:32:14] Like what we're saying is that you have to, uh, take the time, put a plan in place, call
[00:32:21] it pivot, call it shelter in place, whatever you want to call it.
[00:32:23] But, but to just make a decision around a vendor having a, an outage or a vendor having
[00:32:33] a substantial impact to their organization doesn't necessarily mean you shouldn't wait
[00:32:40] a little bit before making a move.
[00:32:42] I I've heard people call it, well, don't procrastinate.
[00:32:45] It's not what we're proposing either because you actually have a plan in place and it's not
[00:32:49] procrastination because you know what that timeline, that time limit that you've placed
[00:32:53] on it actually is.
[00:32:54] Do you have a time limit?
[00:32:56] If you don't, here's where I got to rip the bandaid off.
[00:32:58] Yeah, exactly.
[00:33:00] Uh, what's the point of no return and knowing what that is rather than preemptively just
[00:33:04] doing it because that feels right right now because you're mad.
[00:33:08] Right.
[00:33:09] Yeah.
[00:33:10] You definitely can't be making these, these decisions emotionally for sure.
[00:33:15] I think that, I think that covers it.
[00:33:17] Uh, I think for those of you listening, this has been an episode of MSP 1337.
[00:33:21] Thanks and have a great week.