Many MSPs and businesses believe they're prepared for an outage because they have backups, redundant internet connections, or cloud-based applications. In reality, most recovery strategies fall apart when tested in real-world scenarios. In this episode of MSP 1337, Chris Johnson sits down with Charles Love of Showtech Solutions to explore the dangerous gap between perceived resilience and actual preparedness. They discuss why incident response plans must come before tabletop exercises, how internet redundancy often fails under scrutiny, and why MSPs should treat their own operational dependencies with the same rigor they apply to clients. From protecting the "core four" business applications to safeguarding documentation, passwords, and recovery keys, this conversation delivers practical guidance for building recovery plans that work when systems fail and chaos begins.
[00:00:06] Welcome to MSP 1337. I'm your host Chris Johnson, a show dedicated to cybersecurity challenges solutions, a journey together, not alone. A little bit late as I have been under the weather and that just kind of gets in the way of I had no voice to say the least. This week I'm joined again by Charles Love of Showtech Solutions. Charles, welcome to the show.
[00:00:37] Hey, thanks Chris. Appreciate it. We're, as we were kind of talking before I hit the record button, I got asked about sort of this tabletop exercise scenario and I thought more importantly than the tabletop itself was what kind of backup or retention of data does my business have for the things that are critical to my organization?
[00:01:01] Because whether it's a tabletop exercise, whether we're dealing with an actual power outage or an incident, you know, going down this rabbit hole of doing a tabletop exercise without an IR plan, without some of those other things that are called out in the Trustmark program or really CIS Top 18. So, you know, almost any framework has some sort of logical linear approach to if you don't have A, it's really hard to solve for B.
[00:01:28] And so I just thought maybe it'd be interesting to have this conversation. We haven't talked about it in a long time. Maybe thinking about what I would refer to as the core four. It varies depending on the MSP or the business you're dealing with. But I think of like your accounting package, your communications package, professional services, maybe ERP, CRM, PSA, that type of thing. And there's probably another one in there.
[00:01:56] Maybe if we took comms or communications a little bit further to say we separate phones out from that. But like there's somewhere between three and five, give or take, that one could probably settle on pretty quickly for most businesses of like these are the things that we can't live without. And then they all have varying degrees of time and point in time of recovery that they still are relevant before things become catastrophic.
[00:02:25] Is that fair if we think about how we look at protecting the business? Yeah. And, you know, as being an MSP, I'll speak to the MSP world. Right. And the things we see with our customers. But a lot of times we see customers do ready firing. Right. I kid you not. Last week we had a call and said we're sorry. We called the customer.
[00:02:55] And of course, their phones are dead. Right. Because they have no power. And then when they got power, they called us matter than hell. Right. And we're like, wait a minute. Wait a minute. What's what's why are you so upset? You know, I can't believe it, Charles. You guys, you told me to get a secondary Internet connection. So they're going on and on and on. And I go, we don't understand how when we have no power. Our Internet doesn't work. We have two Internet connections.
[00:03:24] The other one is Starlink. Starlink has no connection to our building. I'm like, it's I'm like, is this are we are we doing a tick tock? Like what's what's going on? Right. Like you, you have equipment in your room. You have switches. You have computers. You have a firewall and you have no power. Yes.
[00:03:52] And we had to be very, you know, like, no, no, no. Like you have to understand. Remember how we talked about generator and all that stuff? And you don't want to do that. Yeah. So we literally had one of those calls that I cannot believe we have two Internet connections and we don't have any Internet. I'm like, yeah, but you're you're sitting in the dark. Right. Of course. And of course, you're going to do with the Internet in the dark. Yeah. So, like, you know, you can't print, can't do all that stuff.
[00:04:19] Like, you know, we have we have UPS is like, oh, you mean the thirty five dollar thing under your desk? That last maybe five minutes. Like, yeah, it's a real. This is a really good example of running a scenario through a customer when there's no power. Right. So that same. By the way, that's the best scenario. By the way, I have. It's funny you said that. Yeah, I actually shared the power scenario on the call yesterday.
[00:04:49] I said, this is the probably the easiest scenario that everyone can quickly, you know, relate to of powers out 15 minutes, an hour, four hours a day. At what point in that timeline do you need to make more significant changes to the approach of how you're going to finish out that day? Yeah. And it's just this is one of those things. And this literally happened last week. Right. And we know we've had it before, but it was like last week.
[00:05:20] You know, and Internet is one of those things where customers think they don't need it until they do. Right. Right. And so like when you spec out a building. All right. So how important is Internet? It's not important at all. My employees can deal without the Internet. All right. So let me just make sure we don't want to firewalls fail over to two separate circuits. Right. Two different UPS is we don't want to have full redundancy. Right. No, that stuff costs too much. OK, cool.
[00:05:49] But boy, when the Internet goes out for five seconds. Yeah. You know, the flood of calls. I have an important meeting to attend to. Well, you have one firewall, one UPS. You have one Internet connection. And Frontier says it's going to be down until tomorrow. Well, that's unacceptable. You guys suck. Yep. Yep. Yeah. Because we're Frontier. I didn't realize that just now, but that's who we are. Sorry.
[00:06:16] I mean, we have other there's other MSPs who are like really aggressive where they will sell to firewalls and they will sell to UPS and they will sell standby switches. We you know, we just don't push that on customers probably should. But but that's a really good example of almost the most critical thing. But from the ownership level of of a business, they can live without Internet for 10 minutes. Right. They can live without Internet for a day. But that doesn't trickle down to the employees.
[00:06:46] Right. I need to watch my Netflix. You know, like this is unacceptable. You know, like this is unacceptable. I need to understand what's happening, you know, with, you know, I don't know the world. Name your name your show. Yes. Right. I can't think of it right now. Like, America's of our lives. Yeah. Yeah. Like, I need to understand. So it's just it's just one of those things. Right. So even like in my own house, I will tell you when I don't have Internet, there is chaos. Right. Because the fish tanks don't go off.
[00:07:15] The alarm is all messed up. The doors don't auto lock. Like there's there's all I have. I now have two Internet connections. But still the to talk about two Internet connections, that really doesn't mean anything. Right. Because what a lot of where's the bad guys? Well, no, the multiple vendors typically use the same box on the corner. Right. So if a car plows into the box and in Tampa, we have boxes on corners. Yeah.
[00:07:45] The pedestals. Yeah. Yeah. So like people don't realize there's like 17 different ISPs in those boxes. Right. So in Florida, where I am, we don't have underground wiring. Everything is up in the trees and, you know, hurricane season. Yeah. So, yeah. When when the trees are blowing, cables get ripped. I don't care if you have wow or you have spectrum. That's two vendors that use coax cables in our and I don't have fiber yet. I can have either one of those.
[00:08:15] They're sharing the same poles. Yeah. Right. And I actually just had similar where I now have, I started with aerial. And now there are three other carriers that've come down my street that are all direct burial. So I'm like, ooh, I could have redundancy with two different ways to get it to me. That would basically be the probability of both of them being cut is not very high.
[00:08:38] But to your point, this is the area of explanation that so many miss is that the client's not asking you to explain how it's delivered to the building. They're just saying, we want you to ensure that we have redundancy, but they don't want to pay for the added cost of just because you have two circuits doesn't mean you get to take advantage of both of them when the failures happen. Yeah. And then and then I'm the bad guy because like that, I this has happened right.
[00:09:08] Internet's down for a customer right for whatever reason they the it's underground burial fiber. They've cut the fiber, you know, it takes a while to get that fixed. Well, we just went to Verizon and bought a hotspot. I was like, oh, that's that's cool. So you can work. He's like, no, no, no, we bought it for the whole company. You have 45 employees. Right. Like I go, have you ever? Have you ever like shared a hotspot when you're on vacation with your family? Oh, my God, it sucks. Yes.
[00:09:37] You now want to share what sucks for three or four people to 50 60 100 device. Oh, I didn't think about that. Right. So like we try to have one of the scenarios we try to walk people through is let's talk about the Internet scenario. Right. So in some cases we are bringing Starlink into the equation because we a lot of areas here can't get far. But fiber, it's getting better. Right.
[00:10:04] But like even at my own home there now, I didn't know this, but T-Mobile now does fiber. It's odd. Never would have thought that. So they've been burying cable all through our neighborhood, you know, the fiber lines for T-Mobile. So I will now have a fiber option at home, which is great. But the bad news is those knuckleheads keep cutting water lines. Right. Right. And sewer lines and it's causing so much trouble.
[00:10:31] But in a scenario where a customer can't get that type of connection, we're leveraging, say, traditional fiber or coax and we're pairing it with a Starlink type service, which is not great. Right. Right. But it's it's a hotspot. You know, it's you can get the 300 meg connection, you can get the gig connection, whatever. It's not really gig there. But now you've got to have somebody like mount the thing on the roof, run the cabling down.
[00:11:00] Like it's a little bit of a buildup. But that is a great scenario to talk to customers about just for the Internet. Like, let's not even talk about all the other things. Right. Like what about printing? Right. Right. Because if you do it right and you have a firewall and you're using Starlink or using whatever, if an ISP goes down, you know, you can fail over back and forth all as well. Certain things ain't going to work because you may be whitelisted to IPs, you know, all kinds of fun stuff like that.
[00:11:29] But but that's something that customers don't really realize because it's I.T. We should be in control of the Internet. Yeah. Right. So so that that's a really good scenario. And then going back to what we talked about in the beginning, let's talk about in the MSP space. What is my you know, my recovery time? Say if one of my one of my apps were to go down, I think last time we talked about what happened with my RMM tool. Sure goes down.
[00:11:59] Well, we got a little aggressive and said forever. And that was like, well, that's we should have those answers. But I think in the context of doing initially, especially when you're doing this for the first time, we don't want to get too aggressive because then it's like, well, that'll never happen. And now you get people not wanting to really participate in the exercise. Yeah. But but remote access is the biggest one. Right. So, like, I'll give you three scenarios that the listeners can kind of think about. Right. So scenario number one, what happens if the RMM tool is unavailable?
[00:12:28] Right. Well, we run a secondary. Right. We we have the ability if our primary RMM is unavailable, we have a secondary remote access tool that we can jump on boxes with. Not as not as fancy, but it works. Right. So that's how we can support our customers and event the RMM tools down. We'll be blind. Right. To like bar charge, things like that. But at least we'll have something. So that's scenario number one.
[00:12:58] But I think that's I think that's more to the point. Right. It's what are the appropriate redundancies that you have? So going back to going back to the power example, I remember when I was working in construction management, when we built out the complex, we had redundancy. Redundant power coming from two different locations, utility companies. They had redundant Internet, multiple egress points.
[00:13:24] They didn't take into consideration what would happen when because they decided not to do generators because they're like, it's too expensive. It's like I think it's going to cost like 30, $30,000 for the one building that they wanted it for. And a lady fell asleep at the wheel, took out a power pole, pop the transformer, and then it popped the transformer on the next block and the next block took out four transformers. And it was going to be a full day before they could get all of the power poles back up.
[00:13:51] No big deal, but they had to get they didn't have the transformers needed for that part of the city were in like another. It was just it was going to take them a day to get the transformers to the to do the repairs. Well, that was unacceptable. And I'm like, this was the scenario we played out for you when we were putting the building was being built. Yeah, you know, twenty five, thirty thousand dollars. Now you're like, this is unacceptable.
[00:14:16] You're going to spend fifty to seventy five thousand dollars because now we've got to bring a crane back to put the generator on the roof. We've got to bring a you know, all these things had to happen that were tied to not actually walking through a scenario of why this is critical, why this is important, why the things that you think are redundancies are not necessarily in fact unlikely to be the scenarios that would have triggered the problem in the world. In the first place. Like, sure. Hey, cut down a power pole. What happens?
[00:14:46] Well, why would someone do that? OK, maybe they didn't use a chainsaw, but like the car is a pretty good tool to get a power pole, you know, clean shaved right off at the ground. Like it just works just fine just as well. So, yeah. So, yeah. So, MSP scenario number two, the digital brain. Right. The amount of MSPs I talked to who don't export their digital brain, you know, like their their document, their documentation system is mind boggling.
[00:15:16] I, I run a run book once a week. Right. Are the vendor that we use has been down granted only for like a couple of hours, but man, during a couple of hours, my guys are playing solitaire. Right. Yeah, because like they can't MFA anything. They can't log in. Nobody knows passwords. I don't know any of our passwords. The damn things are 300. All right. Okay. They're 30 characters long.
[00:15:45] They're like, look, if I can't spell my own name, you think I can type a password? Yeah. Yeah. But like we've, we've become so accustomed to our digital brain. Right. So we, we run that export and bonus tip. We also run a export weekly as well of the BitLocker decryption codes, put it in. Oh yeah. The keys. Right. Because sometimes like God forbid something's happening and then the knowledge base is down. I can, I can pull the BitLocker codes.
[00:16:15] Now here's a cool thing. Sometimes the BitLocker code changes due to whatever reason. And because I have a weekly export of it all. Um, I can go back in history and see if that code ever changed. Right. So just another. Got it. Another fun tip. Um, third thing. And this is Charles tip number three is going to be your PSA. Um, I've been burned a few times in my years being an MSP.
[00:16:42] And I twin every help desk email that comes into the PSA to a shared mailbox and exchange. Never look at it. It's just there. It's collecting every once in a while. I get a wild hair. I go and delete that stuff. Nothing worse. And again, this is all about RTO RPO. Right. Uh, return. Uh, help me with RTO. Uh, return to office. Real time. Oh, it's real time.
[00:17:09] Well, it's, it's the real time objective versus real point in time objective. So one is how much time can, can lapse between getting back to putting data in. And one is how much data can you lose and still come back? So you need up to the minute file level, or do you need the last 72 hours a week? Wouldn't be the end of the world, which goes back to, we know, like, uh, the, the idea behind back everything up versus back nothing up somewhere in the middle.
[00:17:38] Right. You don't need to probably back up this server, uh, the server OS level heart, you know, you don't need a full image of your server every day being backed up. Right. The server is not changing dramatically from. I don't agree. Eh, I'm a huge fan of server level backup. So fair. I'm just saying that whatever the balance is between what changes are happening versus not happening.
[00:18:05] Cause yes, with backups today and a full full version doesn't necessarily mean you had to do bare metal, uh, from zero to full server. You may have just had to do delta differences to ensure that you have the latest of the backup. But, but to your point, like what do I need and how old can it be before it no longer matters?
[00:18:28] For sure. For sure. And that's why I was bringing up the, the, the last point here, right? The, the whole, the, the PSA thing, right? So your PSA is dead, right? Cause we talked about what happens if your RMM dies, what happens if your PSA dies now customers can't communicate with you. Right? Right. So, so what we do is we send everything that comes in to that twin mailbox, that shared mailbox, which also goes to teams. And then it sends over to our PSA vendor.
[00:18:51] So when, when MSPs are sitting there and doing their, their own tabletop exercises, like I always say, you know, you, you gotta, you gotta do your own thing first before you talk to customers about it. Right? So three things, what happens if your PSA goes unavailable? What are you going to do? Right? Uh, crying is not an option. It will occur, but you know, it could be a thing. Uh, RMM, PSA, internet. Those are the three things that matter the most.
[00:19:20] And you, you know, this is an example of where I heard it said by another MSP, you know, we, we always talk about the, you know, drink your own champagne kind of thing of like, you're using the same services that you're making in your client's. Cause this is where I had the example given of where they don't use the same. So, uh, this particular MSP, like we don't use the same phone solutions that our clients use because if our clients are having a phone problem, guess who's not having a phone problem?
[00:19:47] Uh, if we're having a phone problem, guess who's probably not having a phone problem. So I was like, that's an interesting model. I don't know how probable that logic is necessarily, but it points to the fact that that is creating a level of reduction. Of risk by ensuring that, Hey, if my system's having a problem, not everybody is, but it goes back to your internet point of all the pedals stole. That that's a double-edged sword, right?
[00:20:16] That was good, but. And, and let me, yeah. And let me explain that, right? Because if you're not using the same solution your customers are using, you're not going to be an expert in it. Right. And granted, there's ways to front your phone system so that you can redirect the calls, right? That's what we do. Um, we actually do use the same phone system our customers uses. And when, when there's an issue with the vendor, we're already aware. You know what I mean?
[00:20:42] Right. But, but, but at the, I don't know what it's called, but the provider level, I can redirect our inbound number to like a cell phone if I had to. Sure. Um, right. There, there are ways, this goes back to knowing what you're, you're getting into, you know, what does your IR plan look like? And this kind of goes back to the beginning of this conversation of like, doing a tabletop exercise or playing out scenarios.
[00:21:05] Without an IR plan is really just kind of like a, uh, playing dice in the backyard just for fun. Like the IR plan is so critical, even a rudimentary one, but sometimes doing sort of that water cooler exercise makes the client or your staff start thinking about the fact like, Hey, boy, it'd be a really good idea to have an IR plan.
[00:21:28] So that we have this stuff documented so that we're not doing a knee jerk reaction of like, Hmm, hadn't thought about what would happen if my RMM tool was down or if I couldn't use my phone system or how those, those next pieces fall into place to ensure that chaos doesn't ensue. Yeah. And I'll give you a funny one. Uh, if you want to see how your employees react, like I have TVs are all around the office. Um, go to YouTube. There is a BitLocker 10 hour video.
[00:21:57] Play that video, play that video, put it full screen, see how your employees react. Right. It's the, it's the screen that your PC, you know, your computer has been BitLockered. You have 24 hours to pay me. Yeah. See, see if they follow, if you do have an IR plan, see if they follow it. Yeah. Right. If they remote into that PC, that's, that's no bueno. Right. If they find out it's, it's a YouTube video, well, then they didn't follow the IR plan. Right.
[00:22:23] Right. Cause the IR plan is typically it's like, let's call a meeting. Let's open a team's channel, whatever it may be. You should have a structure. So it's a good, it's a surprise tabletop. I may or may not have done that in the past. I may or may not have contacted our SOC team to initiate a fake alert. Right. Um, right.
[00:22:45] Right. But, uh, yeah, there's all kinds of things like every, you know, if you don't have a plan, uh, like it's just, it's just chaos. Right. But sometimes the plans are, I think Mike Tyson said, you know, you, you have a plan until you get punched in the face. Right. Um, but you have something to go to.
[00:23:07] Right. And if, and if you have a good plan for your own stuff, now, when your customer has that issue, they just follow the plan, call it a workbook, call it a cookbook, call it a recipe. I don't care. Right. But you have to protect your house first.
[00:23:26] Right. The, the, you know, I just, if you don't allow your employee, your customers to have admin access, why in the world do you have admin access to your own PC? Right. Things like that. Yeah. Do what you say.
[00:23:39] Sure. I like it. I think that's the good, I think that's probably a good stopping point. Like I think what I really wanted to establish, we covered at least twice. And that is don't overanalyze what tabletop exercises are meant to do.
[00:23:57] But they're not very useful if you aren't thinking about and building out an IR plan. And, and a good, a good case for this is don't try to build an IR plan for your entire organization all at once on day one. You can create IR plans for your core business applications first, and then use those to create your bigger IR plan for everything else. Right.
[00:24:27] And it's just that it's going to take a really long time, which is why IR plans often can take upwards of six months to a year to fully get fleshed out. You should be able to have a pretty solid IR plan for your RMM tool within, you know, a few hours of effort. Like, Hey, what's, how are we going to create redundancy?
[00:24:45] How are we going to inform the staff and make sure that everybody understands what those protocols are? And you are in essence done. If phones can be very similar, right? Hey, if our phone network, it goes down, what do we do? Oh, someone has to remote in and change the call to route to someone's cell phone number and boom, we're done. Right. Like, and test them out, make sure that they work, make sure that they do the thing that it's supposed to do for that scenario.
[00:25:10] And again, no matter how many scenarios you and I come up with going to be, we've exhausted all scenarios. Yeah. And just test it. Right. So like a, a IR plan, like just, let's just talk about like MSP continual operations. Right. Uh, you could just have a document of what happens if the, if, if the PSA is offline, what do we do? How do we communicate? Right. We use a third party client portal. Right.
[00:25:38] So for us, it's not that big of a deal. Uh, so they can at least still create tickets and things like that, but we could put a banner at the top. So in our PSA is down, uh, policy, we have, uh, send a message through pipe drive. Right. We have put a, put a banner in our, in our client portal, letting them know that there's an issue. Right. So we have established procedures so that it's not,
[00:26:03] uh, Oh, the PSA is down. It's like, yeah, that's, that's terrible. I'm sure they'll get it back up, but let's, let's just continue to, you know, uh, step three, keep tabs on the shared mailbox. Right. To, to call those customers. Right. So that's not going to be a big long thing is what I'm trying to say. I, yeah, I'm with you. Um, so there you have it. Uh, this has been an episode of MSP 1337. Uh, thanks and have a great week.