Overcoming a Ransomware Attack - How??

[00:00:00] The existence of our company was at stake here and the likely outcome was that when I go home this weekend,

[00:00:09] I won't have a company anymore. The 28 years that I had put into this business with my business partner,

[00:00:16] you know, that multi-million dollar business was now worth zero.

[00:00:25] Hey guys, welcome to this episode of MSP Mindset.

[00:00:28] I'm your host Damien Stevens and I'm really excited to talk about a really tough topic with Robert Choffy

[00:00:35] and just as a spoiler, if you're an MSP ever wondered what happens if?

[00:00:42] It happens if a lot of my clients or maybe more than a lot are attacked, encrypted, etc.

[00:00:49] We're going to answer that and we're going to actually get an practical things of what do you do?

[00:00:55] Before we get any further, just to introduce Robert, he has been an MSP for decades.

[00:01:01] He is currently the CTO and co-founder of Progressive Computing.

[00:01:06] He's served on the advisory board of many tech companies, axiants, exigents, Otakama, and others.

[00:01:13] He is currently the chairperson of the Comp-TIA Emergency Response Team.

[00:01:19] He knows a few things about emergencies.

[00:01:23] Is it involved pure group facilitator at IT Nation?

[00:01:27] And, as much of his creative nature and problem solving skills

[00:01:32] to the thousands of hours playing Dungeons & Dragons?

[00:01:36] I have to admit that publicly because it's on my profile, but yes, I've reviewed much of my success in life

[00:01:42] to being able to defeat monsters and dragons made up in our heads.

[00:01:47] I love that.

[00:01:49] I love to have some color, right?

[00:01:52] We're all human and despite all the really cool things that you've accomplished,

[00:01:57] it helps me connect to those real human under there.

[00:02:01] Let's start at the beginning, Robert.

[00:02:07] Sure.

[00:02:08] You've had an MSP for a long time

[00:02:10] and you were telling me about a story where I'm going to let you tell it,

[00:02:18] but there was a zero day in an RMM and all sorts of things started unfold.

[00:02:25] So take us back to the...

[00:02:28] Yeah, it was worth it.

[00:02:30] We're worse than any made up monster we ever encountered while we're all playing

[00:02:34] in my best friend's basement throughout my teenager years.

[00:02:38] But thank you first. Let me say that, Damian, for inviting me onto your show

[00:02:44] to leverage your platform to talk about what I think is a very important topic in our industry.

[00:02:52] Cybercrime, ransomware, what does an MSP do when they are 100% compromised?

[00:03:01] And I'll tell you what the answer is not.

[00:03:04] The answer is not hiding. The answer is not staying quiet.

[00:03:08] There is no shame in victimhood and there is a lot of strength and vulnerability,

[00:03:13] which I hope humbly that you will agree with me by the time we get to the end of this conversation.

[00:03:18] So if I may set the stage and probably many listeners already know,

[00:03:24] and I know Damian that you already know, but on July 2nd of 2021,

[00:03:28] a threat actor exploited a zero-day exploit in Kaseya VSA's RMM tool,

[00:03:34] which we were using to manage 100% of our customers.

[00:03:37] They were able to use that exploit to create an administrator login for themselves,

[00:03:44] uploaded RevalSodeNoKeeB and pushed a script out to install that to 100% of our endpoints.

[00:03:51] On that day, it was a total of 250 servers that were completely encrypted as well as about 2,200 endpoints,

[00:04:02] which I would classify as just your standard Windows desktops and laptops.

[00:04:06] So yeah, Friday, July 2nd, just before Independence Day weekend.

[00:04:11] What am I doing this weekend anyway?

[00:04:14] Let's start recovering all of our customers.

[00:04:18] I want to unpack a couple things you said there.

[00:04:22] First of all, everybody uses an RMM.

[00:04:26] Second of all, it's one of the largest.

[00:04:30] And so my point is in third of all, there's this zero-day.

[00:04:33] So my point is from everything I've understood about this,

[00:04:37] sometimes I've talked to others, MSPs that have assumed the guys that got popped or are sleep at the wheel.

[00:04:46] They were the guys that never put in patches.

[00:04:49] They don't have any cybersecurity.

[00:04:52] But here you are using one of the top vendors, one updating things,

[00:04:58] having a good cybersecurity culture and hygiene and doing really nothing outside of best practices.

[00:05:09] And that's what I want to highlight.

[00:05:12] Right, none of us are prepared for zero-day by definition.

[00:05:17] You know, this is not a defensive answer by any stretch, but when the attack happened,

[00:05:25] one of the things that I thought about was what sort of culpability might I have?

[00:05:31] Did I leave ourselves exposed or did somebody here do something to ourselves in that state?

[00:05:40] Do we leave any doors or windows open, right?

[00:05:43] And what I looked and I saw the evidence which I have hard evidence to indicate this,

[00:05:51] we were not only following their best practices, but we were fully patched on their very latest patch.

[00:05:57] I should say we were only one patch level behind, and the patch level that we were behind on was simply a feature update.

[00:06:04] It wasn't their latest security update.

[00:06:06] So there were less security updates. I think it was at either the end of May or beginning of June,

[00:06:11] we patched our system then and we got hit on July 2nd.

[00:06:15] So it's not like we, and I imagine the other 50 or 60 some odd victims around the world who were affected exactly the way we were,

[00:06:24] were using a two or three-year-old version.

[00:06:27] This is not about, you know, believe me, the negligence conversation went through my head all day on Friday, July 2nd.

[00:06:35] But I, you know, I found some solace if there was any to be found on that day knowing that a we were not alone in this attack and b

[00:06:45] there was little if anything that we could have done to prevent it.

[00:06:49] Yeah, yeah, that I don't know who wouldn't beat themselves up about it, right?

[00:06:54] What do we do? Did we just leave RDP open to the internet?

[00:06:57] Did we, you know, are we too many patches behind, you know, do we get, because it, you know,

[00:07:02] I think sometimes, right?

[00:07:04] Blind password, you know, whatever, right?

[00:07:07] You know somebody leaves some door open somewhere, you know, because, you know, by nature of what we do,

[00:07:13] we sometimes take these little shortcuts to just like kind of get through the day or and then forget that I left, you know,

[00:07:20] the back door open.

[00:07:23] And that's not the case.

[00:07:24] I mean, I thought it might have been I was actually kind of expecting it to be that since the evidence would show that well how

[00:07:31] did they just march in and do this right?

[00:07:34] No one here, including myself, would have thought that a zero day exploit would have been the culprit.

[00:07:42] Now, I mean, listen, all things being equal as many layers of hell that we walk through Damian.

[00:07:50] You know, that's a one of the alternate realities of this universe, of the multiverse that we live in.

[00:07:59] And that version of the universe, we would have done something negligent and it could have been far worse.

[00:08:04] So people will often say to me like, oh my god, that's like the worst nightmare.

[00:08:09] And I say, you know, don't let me fool you into thinking that it was a walk in the park because it was not.

[00:08:15] Hell was very hot for us, but all things considered it could have been worse.

[00:08:21] Imagine if we had done something negligent right and it was only us or if there was data expiltration or a lot of ores.

[00:08:31] You know, we can talk for a half an hour just on the ores.

[00:08:35] But we were fortunate in some ways, which is really weird for me to say that it could have been worse.

[00:08:46] So let's talk about the you mentioned 60 companies.

[00:08:53] You mean more than likely 60 MSPs?

[00:08:57] Yeah, so there's for certain.

[00:09:00] Well, we don't the estimate was at the time and I'm sure I can go find you know, there's tons of news articles out there that I'll say quote unquote document that this or at least reported it.

[00:09:14] There were and I believe Kasey had come out with a statement that there were approximately 50 of their customers that were affected by this.

[00:09:23] One is huge or at least one of the customers of an MSP was a customer by the name of co op and they're in the news.

[00:09:33] Like I'm not not revealing anything secret because you can go look them up there Swedish supermarket with 800 stores throughout Europe and 500 of which were hit by this because their MSP who I've forgotten their name and it's probably not important for me even say who they are, but you could find out.

[00:09:51] They used their Kasey instance to get to all the co op supermarkets so 500 out of 800 stores shut down for about a week because all those cash registers were being protected or I should say being monitored with Kasey of VSA and guess what operating system they were all running that's right windows and guess what got encrypted that day.

[00:10:14] Transact I think about all the people who lost wages I think about all of the perishables that might have had to have been disposed of the loss of revenue.

[00:10:26] I mean, you know there are big corporation they probably could survive things better than most small businesses like your typical MSP including us right I put myself in that category.

[00:10:40] You know they had probably had more resources to be able to withstand something like this but still I think about them and I wonder you know gee like that was a big impact right there alone just one customer not even one Kaseya customer but a customer of a Kaseya customer so.

[00:10:59] You know there were kindergarten and New Zealand that were affected by this they were municipalities all across the United States that made the news because of this and I'm sure there are thousands of untold stories that you just won't hear about you'll hear me and you'll talk you'll hear me talk generically about my 80 customers on that day but what about the other 40 to 50 ish MSPs who aren't talking about this.

[00:11:28] I don't fault them so if you are one of them in your listening to me please by all means it's not a criticism and I completely understand why you might want to go look this is behind me I don't want to talk about it it's just let's put it aside but okay I'm choosing a different path and I'm not looking for applause I'm just looking for a little bit of support getting the word out there because it's really important we talk about this.

[00:11:55] That's right it starts with talking about it right because emotionally from a business perspective all community perspective like the worst thing we could do is not try to get the word out and help.

[00:12:10] So take me back to how did you even realize you're encrypted take me back to that July second day yeah so it was it was a beautiful day here in New York you know if you live in the northeast or in the north.

[00:12:25] In the upper Midwest at least my friends in the upper Midwest tell me it's the same thing there you know winters can be shitty they can be cold icy you know dark like I say in New York you know for about three months out of the year the sky is gray and the ground is brown on sometimes it's white with ice so

[00:12:48] you know days are short you know when when it's July second and it's 80 plus degrees out there isn't a cloud in the sky and there's zero humidity those are the days that make you feel like that dried up potted plant that finally gets a good gulp of water right you spring back to life you know you breathe deeply the air I was just so beautiful forecast was the same for the next couple of days we had plans personal plans at home to spend time with friends

[00:13:18] we were going on a boat ride in a pride you know friends bars have a boat we were gonna go on a boat ride in a Hudson River maybe go down to the statue of Liberty or cruise around Manhattan a little bit it was the mood was jovial it was light hearted everybody was looking forward to that weekend so it was about noon I was sitting in my kitchen and I saw my director of operations send the staircase to the second floor where I was sitting turns the corner down the way I was sitting

[00:13:48] in the hall now walking towards me as I'm sitting in the kitchen looking at him through the kitchen threshold and I knew something was instantly wrong by his body language is gate even me being near sighted I could see that at that distance that there was just something wrong with his complexion

[00:14:07] in the space and so I got up and as I passed another individual in the kitchen I said I holy shit I think somebody just died I mean that was exactly what I was thinking just by looking at him I knew instantly and I'm not normally an intuitive person in that way but his body language spoke volumes to me

[00:14:30] and I was bracing myself for like oh my god did one of my family members just call the office with terrible news did something happen in my business partner did something happen to one of our engineers in the field

[00:14:45] like what terrible news am I about to I was stealing myself for the catastrophic news and while I don't want to necessarily liken this to being either better or worse than somebody dying

[00:15:03] because all things considered nobody died in this thankfully

[00:15:09] but you know the look on jays face you know spoke those that sort of set that tone for me and I you know when I approach him we got within you know like very close we are only about two feet apart and you know jana have an excellent personal and professional relationship but he wouldn't look me in the eye

[00:15:30] and that's when I really felt like okay like this is bad like what's going on here and when I finally got him to utter the words he said you know all of our customers are ransomware and then he finally started to make eye contact with me

[00:15:44] when you hear those words they're too surreal to process I mean it's almost like a bad joke like come on like all of them what the hell like so I rejected his reality and try to insert my own which was no no no no

[00:16:00] that's that's not real like which ones or which one right and all of them that's an impossibility right and he was like no it's all of them and and again I refuted which he retorted back with and to this day we have a debate jana about whether he did this subconsciously or intentionally we don't know which

[00:16:25] however he started to rattle off all of our customers and I think he chose to give them to me in order of revenue by largest to smallest now we didn't name all of them and I'm kind of giggling here and I don't mean to laugh about it and make anybody think that I'm flipping or nonchalant about this.

[00:16:46] I do have a philosophy in life that you have only two choices laugh for cry so believe me on the inside there's a maelstrom of tears that are ready to kind of come out but I kind of like to find the humor and even the worst of times and it was a little funny that he did this at least I found that humor.

[00:17:07] But those words every time he named one of those clients there are all big names representing all in excess of ten thousand a month an m r r for us.

[00:17:18] And they were like pan-caking down upon me like you know collapsing dominoes not just the little ones on a table but like you know this dominoes the size of houses just slamming down on me each one just kind of pushing me further down into the earth.

[00:17:36] Until I kind of felt like all that beautiful weather that I just talked about you know that proverbially inside the office it felt like the walls were closing in and that there was no air and that the room darkened.

[00:17:53] I mean it's I know that's not what actually physically happened but that's the way it felt.

[00:17:59] But one of the things we did is we went right back to my office where you see me sitting right now with the same computer and the same three monitors that I have the only thing that's different is my mic in camera.

[00:18:13] And sat here frantically trying to figure out what the hell was wrong I think even at that moment Damian I wasn't I was still incredulous like this is just too unbelievable to be a reality.

[00:18:27] Right and my by the way meanwhile like you know when Jane I were talking out in the hall outside the kitchen and as we maneuvered over into my office I could hear and see people on the phone I could see people's body language is getting serious I could hear words like you know I'm sorry like we'll get a technician I don't know what's going on you know calm please calm down.

[00:18:49] You know all kinds of alarm kind of conversation just little snippets that were enough for me to piece into the story of what was happening or unfolding before me as further evidence that no this was a really serious situation.

[00:19:05] So as we were as I was trying to log in to Kaseya at my desk and trying to figure out what the hell was going on I had one of my chief engineers on my cell phone.

[00:19:16] On speaker phone and I finally got I think the final nail in the coffin for me was when I watched all the icons on my desktop flip to that white box.

[00:19:29] Now I know this is a technical audience and you probably already know what that means but if anyone listening doesn't understand what that means that means when windows displays an icon it knows that a DOC X is a word document and displays the icon for right Damian so.

[00:19:44] And same thing with Excel and XLSX or a PDF it shows the appropriate icon for it when a file gets encrypted windows doesn't know what that file is anymore and puts a white box in its place so as I watched all those icons flip to white boxes that's when.

[00:20:00] It really solidified in my mind that we were dead we were beyond repair we were we were hanging on the edge about to fall into the abyss are our very the existence of our company was.

[00:20:18] Was at stake here and the likely outcome was that when I go home this weekend I won't have a company anymore the 28 years that I had put into this business with my business partner you know that multi million dollar business was now worth zero if.

[00:20:36] And it might have even been worth negative considering all the potential lawsuits and other costs that might be coming our way I thought about recalling my daughter back from college and telling her you know she made it halfway through with you know awesome grades but she wouldn't be able to return because I wouldn't be able to afford it that maybe I might need to sell my house.

[00:21:00] The personal impact of me thinking about my life plan that I work so hard on with my wife our financial plan our retirement plan and family vacations and all the things are important to us personally i thought this i felt this tremendous guilt Damian.

[00:21:18] Of that I failed everybody somehow that I felt my family I failed our employees I felt my partner i don't know why it may be its Catholic guilt i don't know but but.

[00:21:31] But I felt that and it's unreasonable especially now to say why are you feeling that way it's just what I felt I felt I felt the world I thought about even all the local business relationships as well as the community relationships that we've had and I felt embarrassed i felt like like i was worthless which is and these were feelings that I never feel I'm an a type on the lion I am.

[00:22:01] You know always in charge I'm the guy that everybody looks to to like fix the problem or figure crap out.

[00:22:09] And here I was thinking i'm dead like i can't get my own mortality at that moment Damian it was it was I was frozen and it was scary as.

[00:22:20] I want to say the f word but it was scariest you know what and i don't know what to do i mean we were i was frozen for about ten minutes speechless thoughtless i mean it wasn't.

[00:22:33] You know like most technicians like me who built a business you always even when you're having a difficult problem you're always kind of like thinking about or you know puzzling around or or ruminating on a potential solution or different ideas are you stretching but i'm telling you my mind what.

[00:22:50] I went completely black it went totally dark and i couldn't even formulate a coherent thought that's how bad this was.

[00:23:01] And so emotionally I was in such a horrible place that I knew I needed to stamp myself out of this I needed to defibrillate right I screwed up that word right I needed to defibrillate here to get my my heart rhythm back to start breathing and thinking.

[00:23:20] Again because there was this was just you know otherwise I had no other choice other than to tell everybody sorry everybody go home let's disconnect the phones i'll send an email to all our clients to say we're out of business good luck right.

[00:23:35] You know that's an option it was an option and believe me the thought crossed my mind for about five milliseconds but you know I arrested that thought in fact I killed that thought immediately but that's where your head goes right.

[00:23:50] I don't even feel bad and guilty about even thinking that right yeah it was not fun man not fun so how just for context for everybody how many years in business when this happened.

[00:24:05] At the time it was 28 years so this February so it is right at the end of January when we're recording this and so next month and about a month from now we'll be hitting our 31st year.

[00:24:16] So it was we had just passed our 28th year at that point.

[00:24:21] Yeah so I is just another fellow human and just some it's been an MSP and had the.

[00:24:31] Opportunity to help some MSPs through this i don't know why you wouldn't beat yourself up right we want to solve the problem and there's usually two or three different ways we can solve it especially coming from a technical background.

[00:24:46] But also just the gravity of you took the risk and built a business and built those business community relationships and hired the people.

[00:24:55] And I know you feel like I do that depend on you right and they did and they still do and I felt that sense there are like children to me right I mean I don't mean them in that in that sense that but like I felt like I had an obligation to provide for them right right and that's.

[00:25:12] You know I want to just talk about that because if you're not feeling that you're not really admitting it I think and.

[00:25:18] 28 years in business like you said that's that's what you did for retirement that's what you did to put your daughter through college that's you know the in a lot of ways the American dream.

[00:25:29] And you know now you're you know who when you're faced with that just.

[00:25:34] Is gone you know in.

[00:25:36] In the.

[00:25:38] Stole it from me right like how could I put 28 years into something and then within seconds it begun and actually the void that it left was actually still a bill I had to pay right because like I said of what I was kind of thinking at that moment like oh my god I'm going to get my ass

[00:25:54] suit off like by different companies right like how does one survive that right.

[00:26:00] Right.

[00:26:02] So what the where did you go from there what the heck do you do next.

[00:26:07] Well you can't see but up here on the shelf over there I got all kinds of alcoholic beverages some got some hard stuff up there I'm kidding I actually do have that stuff here but that's not why it's here.

[00:26:21] Not at all there mainly gifts from people who probably commiserated commiserated with me on that on that day.

[00:26:27] There's only one that's cracked open and just a little bit.

[00:26:31] But anyway in all seriousness I told you I try to laugh whenever I can.

[00:26:37] But I felt like the first thing I needed to do was to rally the troops this was war right and so I had a few I'm a big history buff so if you've seen my presentation and hopefully I'll get that book out one of these days it's coming along.

[00:26:54] I channeled my inner Winston Churchill and my general patent at the same time and the general patent came came out and said you know stand up like March start and I just started barking orders like I mean it's what I think best around here.

[00:27:11] You know if there's anybody who's a hard ass in this company it's going to be me but I channeled that that what I believe is a strength at that moment and I told Jay who is with me in my office as I was you know.

[00:27:24] Having a mental short circuit there for ten minutes.

[00:27:30] I was like Jay you know get downstairs like get the entire help this team in the conference room I started pointing around people you get off that tell people get off the phone get your asses in the conference room call all the text that are in the field or anybody who's in the field and get them to dial into the conference bridge because they just knew that we needed to regroup.

[00:27:47] And quite frankly Damien even when people were filing in and chiming in on the conference bridge I sat in one of those chairs in that conference room and said what the F am I going to tell these people.

[00:28:05] Sure.

[00:28:07] They're all eyes are on me what am I going to say do you know what to say I don't know what to say like this is like I'm fruit now here I am in one of those states again like even though there's this outward general patent putting on a face but on the inside I was still like crumbling and I looked at my business partner who has got the opposite personality than me right now.

[00:28:35] He is you know he is smooth he's easy going he is you know you might even say laissez faire he's embodies that sort of southern Italian culture of just like everything's going to be fine just sift the espresso and kick your feet up and we're going to be good don't worry right me I'm sort of the hot head of Italian anyway so maybe that's why we make a good team here I mean after all these years we haven't killed each other yet so.

[00:29:02] I would think that's a positive thing so big thumbs up to clashing but compliment complimentary type of personalities but I looked at him and I kind of remembered something.

[00:29:16] Because in him I see culture I see our company culture I think he helped influence greatly by his demeanor and his mannerisms and his way of thinking and acting and behaving.

[00:29:32] Not general patent right and our four core values which we derive from from EOS is what I start to think about I started to think about our culture and then I remembered that line from and I believe it's Peter Drucker don't crucify me if I'm wrong I believe it was Peter Drucker who said culture each strategy so please don't call me out on that if I'm wrong but you know what I mean so I sort of our core values I thought about our culture and I said

[00:30:02] that's it that's the ticket right there right if we're going to survive this if we're going to rely on anything we cannot get through this unless we remain true to our core values that we hold the line on our integrity and what makes us take as an organization so I'll recite them for you very quickly there's four team together we get it done and that was I think you know wow like more than ever do we need that one it sounds maybe not so grand the O

[00:30:32] OOS on a normal average every day but team together we get it done we really invest strongly in developing a cohesive team here humble confidence service without ego right we've got to have a lot of humility right now but we've also got to be experts in our fields here in our field here and what we know we can and cannot do and execute well commitment to do what it takes oh my goodness you're going

[00:31:00] to need to have a lot of commitment and determination in those dark days which we did as was evidenced by the Herkulean efforts of our team as we recovered and then the last is respect right respect always is the always is the subtext of our core value of respect

[00:31:18] and I knew that we needed to behave and navigate through this situation with that sort of that integrity that I talked about about respecting respecting people even though you may not get the respect back from them the anger the vitriol the frustrations that we may be faced with

[00:31:39] that we were going to maintain who we were in a respectful way so I recited them and I expanded upon them in sort of those ways that I just explained to remind them of what makes us us

[00:31:54] and that's essentially the the raw raw speech that I gave in those few moments but I was upfront honest with them there's a lot that we don't know there is a lot that we can't say so please limit what you say but please get back to your desks, get back on the phone start answering those emails

[00:32:15] and just assure people that we're working on it and as soon as we have information to provide we will just keep checking your email and that was the best that I could do in that moment in time I mean literally it was like you know and I really in a lot of ways it despised Hollywood but sometimes like for the dramatic effect of stealing a scene out of a Hollywood movie it was like that car driving down the road everything is happy and peaceful singing kids in the back seat

[00:32:43] it's sunny out and then bam there's the T-bone you know your car gets T-bone the car flips there's glass everywhere there's blood splatter I mean that's the way it felt right like we were happy go lucky going down the road and suddenly at a nowhere our world went upside down

[00:33:00] what do you say and do in those situations are you prepared for that and I would venture a guess that most organizations are not we were not we totally were not

[00:33:12] and there's nothing wrong with being unprepared for something like that however if you're hearing my story I would implore you to consider some conversation around that it's not not an expensive conversation you don't have to hire expensive consultants or get an expensive tool or just sit with your team and talk about role play if I can go back to play in D and D

[00:33:39] role play out some of those scenarios about what would you do where where are your important things what tools do you're going to need what communication systems what how are you going to behave who you're going to call right do you have an insurance policy

[00:33:54] right those sorts of things you don't have to go through a lab or tabletops to help start preparing for something like this.

[00:34:03] Have you ever wondered if you could recover your backups let me ask a better question have you ever had a backup failed to recover have you ever lost data yeah that's me here's what's crazy 58% of recoveries fail to recover so if you think it's just you you couldn't be more wrong

[00:34:22] what are we going to do about that well you've got two options enter servicity what we do what we exist to do is test your backups and manage them for you we test every volume every single day we do weekly testing monthly testing and quarterly testing

[00:34:37] and you can handle the treasury and babysitting of backups and the data management to our team that monitors your backups for you it's time to level up your backup game one way you can do that is visit servicity.com slash learn more if you'd like to take the process that I've spent the last 18 years building

[00:34:55] and steal it and apply it to your msp so that you can level up your backup game check out the link below in the description or visit servicity.com slash learn more

[00:35:07] And one thing I want to highlight Robert is I'm sure I'm not sure anybody could go through this and not feel completely unprepared just hearing you know what you went through and just a little bit of the tone and emotion

[00:35:23] I think feeling unprepared but there's one thing I've heard from people that made it through this that I heard from you which is culture so maybe you want to prepare in terms of talked about this and the exercises and I'm not taking away because that's a really good recommendation

[00:35:41] I don't know that I don't know how this would have ended if you hadn't already invested in your team and built such a culture so my count argument is you are better prepared than you thought in some ways

[00:35:57] I'll take it right I've learned gracious enough to take a compliment when they come my way not it's not my nature to accept compliments that don't feel like I don't deserve them but I will take that one thing and thank you for saying that and you really make an excellent point there is

[00:36:17] I'm going to butcher another statement around you know during good times you know it's easy for people to be on their best behavior but it's during the bad times when you really see people's character come out right so if you're an MSP owner or leader and you're thinking about maybe you know 25% of your staff or people who you wouldn't trust with your own children

[00:36:41] maybe you have a problem right and maybe you won't be in a position like I was to survive and even thrive as well as we've done post attack I'm not you know that's not a criticism but I think you bring up a really good point here Damian is the value of our culture is so undervalued yeah

[00:37:04] and the investments that we make there in people and being a good leader right listen I was I was literally the asshole here for years and my business partner had to pull me aside and tell me as much right in his own respectful nice way of doing so but I didn't realize that I was behaving that way

[00:37:26] I thought I had the best interests of me and our company and everybody else involved and I just realized it was my delivery it wasn't in my heart be a jerk but it came across that way sometimes and it wasn't maybe I'm making myself sound like a terrible human being but I was likened to like yeah sometimes you're like angry dad right

[00:37:46] and I had to learn that angry dad had no place here right and dad has a place here but not angry dad right dad could be stern and he could be demanding but dad could be loving to and supportive right as he should be

[00:38:01] and so therefore like sometimes I guess the point I'm trying to make too is sometimes you have to look in the mirror and say what's wrong with this culture well why don't you start there taking a value it was self evaluation first are you the problem

[00:38:15] right right are you hiring the right people are you investing in them are you are you the problem there's so many questions

[00:38:23] and you know I've had the opportunity to use the way I say it like I was going to say pleasure it's been my pleasure to help but I hate that any MSPs ever been put in this situation to help some through it in certain ways and there's the ones that have crumbled

[00:38:38] and the ones that haven't and that's what I've taken away and it's one of my court beliefs the people are we build the business you know they're what's important and you know make I just want to make that point you made that early investments

[00:38:50] so this I think everybody's listening to those I'm not just here to blow sunshine this is just like this is a core thing if you invest in your team you're more prepared than you realize

[00:39:01] totally so right and and it's not just about you know giving them gifts or throwing parties or you know getting a foosball table it's it's you know it's getting I want to say you know being at their level and being respectful of them and treating them with you know we we um

[00:39:19] um I don't like it when people call me the boss here like I got one one employee long time employee an engineer that's been with us 25 years actually have to by the way that

[00:39:28] have that longevity with us and he affectionately calls me boss man right and it's like the only person I allow to call me the boss I'm like stop calling me boss like it's uh we're a team right

[00:39:39] you know I'm your team member right and so it's these little subtle things that help like recognition that people are human beings and they have their own needs

[00:39:49] whenever anyone on my team has a personal issue um we are you know maybe maybe even a little too generous sometimes uh but I think that generosity um comes back and

[00:40:01] you know the uh the dividends that that pays is immense because you buy so much um loyalty and respect other people that you're leading to show them that you're a compassionate

[00:40:14] leader that you care about them that it's not just about how much money I can squeeze out of your employment here

[00:40:20] exactly so what happened next and how did the team respond well uh so uh we did exactly what everybody didn't want me to do

[00:40:32] I told everybody to sit on their hands uh why because I got the legal advice to do so and it was the correct

[00:40:39] advice we did not know what we were dealing with so we needed to and I'm really shortening up the story here

[00:40:46] otherwise I could just blow v8 on for hours um uh we needed to get some forensic evidence um completed uh in order

[00:40:54] to inform our recovery plan we didn't know what we were dealing with at the point right we still

[00:40:58] didn't know were they there for months weeks years a day an hour a minute like we had we had so many

[00:41:04] questions not enough answers so while my technical team was like already suited up you know for war

[00:41:11] you know they uh you know their backpacks on their rifles locked and loaded and they were ready to go and I'm

[00:41:17] like calm down um um um um um um and uh so it wasn't until something yeah it was

[00:41:34] it was very difficult for them and um I want to say there were some minor transgressions here

[00:41:39] in there but nothing nothing that worked out uh negatively uh there was uh some of the text

[00:41:44] decided that they were going to do some some um sort of sandbox testing on this like on our some

[00:41:50] of our recovery efforts which is good right like gave them something to do something to focus on

[00:41:54] and uh it was kind of limited and very limited in what they were doing uh but you know by uh Sunday

[00:42:01] afternoon we understood what our recovery uh point objective was we knew that we could recover systems

[00:42:08] as late as 1048 AM Friday morning because it was at 1049 AM when the attacker had entered our

[00:42:17] system for the first time uh so we had conclusive proof of that forensically uh double verified

[00:42:23] meaning two independent analysis said the same exact thing um and so at that point we knew

[00:42:30] well we have every server backed up now it's just a question of nuking every desktop laptop and

[00:42:35] reinstalling windows and going from there um so uh that's easy uh yeah while it is on paper uh

[00:42:45] except we had a math problem here Damian because you know I had 250 servers 2200 endpoints that needed

[00:42:51] to be recovered uh and I only had about you know maybe uh odd or no 15 18 technical hands to do the work

[00:42:59] um so how do you do that right it's like having a time share and having every time share holder

[00:43:05] show up on the same week to vacation you can't do that right uh as we know an msp does not um uh

[00:43:14] you know is built sort of on that kind of time share model um and uh that's where the story of

[00:43:22] community comes in because news started to spread like wildfire amongst our friends uh so it's

[00:43:28] another piece of advice that I would give anybody listening if you are not invested and I'm going

[00:43:33] to restate that word invested and you need to invest not spend but invest in peer groups

[00:43:41] and IT conferences and your relationships with solution partners notice how it didn't save vendors

[00:43:48] right um they're not just squeezing you for money or trying to get over on you right you need to

[00:43:53] build relationships in all of those areas in the entire community ecosystem uh we did so

[00:44:00] heavily uh since joining a peer group in 2010 and that peer group really taught us the

[00:44:09] go-giver mentality although I think it was within my partner or not to be that way anyway um but

[00:44:15] I instinctually knew that by making friends in the industry that somehow some way that would help

[00:44:20] me over time and I thought it was going to be like business advice right uh or you know uh help me

[00:44:26] R&D or help me uh fast track R&D because you did all the work and you're just going to share the

[00:44:31] results with me or or give me those best practices or teach me how to do something that I don't know

[00:44:35] how to do um we never expected that they would get on airplanes owners of other msp's with some of

[00:44:42] their best engineers and spend a week in New York on their own dime helping us recover um and that

[00:44:49] helped came not just from our direct peers but also from solution partners that offered us a

[00:44:56] myriad of help in a lot of different ways um uh Axian ended up giving us some dedicated

[00:45:04] team on the back end and sent one of their center top sales engineer very technical guy

[00:45:11] out to New York from I forget where he was living at the time might have been Denver uh to come out

[00:45:16] and spend like a week here to help us with recovery efforts um and then even the subtle sort of

[00:45:22] help where you get like you know huntress and insisted despite my 10 denials of their assistance

[00:45:28] to send food right you guys got to eat right um um somebody a friend of mine that works there

[00:45:35] put Kyle hand Sloan on the phone their CEO I had never known before I didn't know who this guy was

[00:45:39] in fact we were barely even a customer we only had like a hundred of their agents as a trial rolled

[00:45:45] out and here he was giving me some pretty awesome advice about um you know uh some things to think

[00:45:51] about and he gave me some uh encouragement um not as somebody trying to sell me a product

[00:45:59] but is almost like a friend like a friend that out of the blue like I am here to help you brother

[00:46:05] I am here he gave me some advice that really stuck with me uh that helped me navigate some situations

[00:46:10] his words echoed in my head so axi and huntress connect wise is a huge help um in giving us an

[00:46:16] RMM tool for free for like the rest of the year just they were like we don't want any money let's

[00:46:21] just get you on this thing because you we need because you need to be made whole they didn't say

[00:46:26] we need to make you whole they said you need to be made whole right there was a difference there

[00:46:33] um and so these are the lessons that I think that come out of that community experience like

[00:46:39] be a go-giver go out into the community invest in relationships don't ask for what like I all

[00:46:47] often ask a solution partner hey how do I help you and like what like nobody asks us that

[00:46:51] the quiet I want to know how I can help you right is there do you want an introduction do you

[00:46:56] do you like what do you need some help well like maybe a case study it like I want to know how

[00:47:00] I can help you too not just like just because I'm buying something from you doesn't mean I can't

[00:47:05] help you back right right and that's why I think um and I don't want to take all the credit here

[00:47:11] right I keep saying the word i but my business partner also of the same mentality also making

[00:47:18] relationships with people that we leveraged uh during that time you know he was super instrumental

[00:47:25] in you know while I was in that operational um um mode of recovery and you know again channeling

[00:47:34] my inner general patent uh he was making sure that everybody was taken care of that everybody

[00:47:38] was fed right and that there was a little bit of celebration some nights and a little bit of

[00:47:43] you know let's let's uh let's relax a little bit let's you know enjoy ourselves a little bit as

[00:47:48] much as this is a hard time like he focused on the people where I focused on the process

[00:47:53] right yeah yeah so um but the community support that we received was um and

[00:48:03] again if I can make a slight dnd reference uh uh invoked Tolkien uh it was like the the fires

[00:48:12] of gondor being lit one member said get your asses to New York in an email to a bunch of other guys

[00:48:19] and uh it was like the riders of Rohan showing up uh right at that pivotal moment just when we

[00:48:26] felt defeated and we would never be able to get through this uh help arrived and more and more

[00:48:31] help arrived behind that and then even behind that more and more help behind that arrived and it

[00:48:36] was an overwhelming um and incredibly humbling experience um and I can almost feel the tears well

[00:48:44] in up in my eyes um to think about all those people who came to our aid selflessly um and without

[00:48:52] any expectations of anything in return other than to say this is terrible situation and if we don't

[00:48:59] help you you're gonna die right yeah um so it was just an amazing experience in that way

[00:49:06] you know we don't think about a ransomware attack uh we think about all the negative sides of it

[00:49:11] and it was truly negative please don't mistake me I don't want anybody to ever go through this um

[00:49:18] but I really grasped hard to look for those positive things um and so the dichotomy

[00:49:27] of the highs and the lows throughout this whole ordeal were so stark there was nothing in the middle

[00:49:34] right there was nothing average or transactional about anything that happened it was either we're on

[00:49:40] cloud nine or we were on the ninth level of hell um and it was just it was it was it changed me

[00:49:48] Damien it changed me Robert you were unfortunately I think an outlier in terms of talking about this

[00:49:57] in terms of you had a solid culture of foundation I'm not saying you're the only one but

[00:50:04] when you add that and you add the peer group and the strong foundation of peers you had already

[00:50:11] built and just this really strong community I think you're in my opinion based on at least

[00:50:17] my personal experiences were better prepared than at least many others not everybody but many

[00:50:25] and um I think that's amazing so I want to talk about community I know we want to lead into

[00:50:31] something and what folks can do yeah before I get there help us understand what happened in terms

[00:50:38] of how did you keep your team going how long did this take you you had backups you could recover

[00:50:45] it sounds like they did work unfortunately that is not a lot of the actual case

[00:50:51] yeah we we bid our nails wondering are these damn backups that you know we get the green light

[00:50:57] reports on but are they you know now the true test came right of of of of restoring servers from

[00:51:07] backup now thankfully we had a perfect record in restoration we had and I'm gonna put a little

[00:51:15] little asterisk on that we had one sbs small business server 2011 server out there this was

[00:51:21] remember this attack happened in 2021 I know what you're all saying and thinking right like how could

[00:51:25] you have a customer that's still on such ancient well you know we all have that one or two that

[00:51:30] just has the white knuckle death grip on some something old that they don't want to get rid of so

[00:51:35] right we had it was impossible to get that thing to restore back because of all the old technology

[00:51:41] and it was backing it up great it just wouldn't restore so we know that have to spin up that

[00:51:46] instance in the cloud instead of a local restore and then move them to the cloud so was that successful

[00:51:52] I mean you know you can argue that a lot of different ways but um we got them open running if you

[00:51:57] don't allow running right yeah they didn't lose their data and it was sbs that's that's a pretty

[00:52:04] that's why I say 100% yeah but it took us 17 calendar days and I really mean the word calendar

[00:52:13] because remember we worked around the clock some of these guys on my team got blessed our hearts they

[00:52:17] worked like 18 plus hour days I only learned recently because I was interviewing my team individually

[00:52:25] on the events of the day just to you know for documentation and recording and I found out that one

[00:52:32] of our texts left uh one of our clients on the very first day of recovery at like midnight or 2am

[00:52:39] and was supposed to drive home to get someplace by 6am but ended up driving to that client slept

[00:52:44] in his parking lot for like two hours and then and then entered the guys building at like 4am like

[00:52:51] had I know and he intentionally kept it quiet because he didn't want anybody to know that like you

[00:52:55] know he violated orders of go home and go sleep right yeah uh but that's the dedication of some

[00:53:01] of these guys was just like it brings me to tears even thinking about it like you just want to hug

[00:53:06] the guy and just thank him for being such an awesome human um but you want to strangle him at the same

[00:53:12] time to say why'd you put yourself at such risk right um but it's all water into the bridge but

[00:53:18] you know the recovery took us about 17 calendar days to get most systems fully operational

[00:53:25] uh but getting to that 100% mark like that was then getting everybody to that 95% mark but getting

[00:53:31] everybody to 100% took months right because think about all the settings and configurations and you

[00:53:39] forgot this printer and what about that one application that I only use once uh uh once every blue

[00:53:44] moon or some tax software that I only use in January like it was just we got like it was getting you

[00:53:51] know uh uh death by a thousand cuts on uh just all the little crap right and you know it that was

[00:54:01] just as frustrating uh in some well maybe I'm over emphasizing here it was frustrating it wasn't

[00:54:08] as frustrating as going through the main recovery efforts but it it it was like that nagging like

[00:54:13] constant reminder of what happened to us like we all just wanted to get to a point where we could

[00:54:18] say are we are we just back to normal um and that day I think came somewhere in maybe October of

[00:54:25] that year where the like I kept asking the team almost on a daily basis like is it normally yet

[00:54:32] like what kind of tickets are coming in like what kind of issues you're facing and of course I'm

[00:54:36] doing a lot of account management stuff so I'm hearing it from that end as well but it was somewhere

[00:54:41] around aug uh I'm sorry October that we felt like all right I think we've hit a stride now of normal

[00:54:46] say right um so I think it's an important lesson in this too is just because you restore a system

[00:54:53] doesn't mean that really things are working the way they should be yeah and I can't imagine there was

[00:54:59] any normalcy after working around the clock for 17 calendar days now people are exhausted people

[00:55:04] delayed or even canceled vacations um you know it put a lot of stress on on both individuals working

[00:55:11] here as well as family members um you know Damien not to go too deep into this but I had a one of my

[00:55:17] key star players a guy a 25 year veteran lost his dad he died right in the middle of this right

[00:55:24] a week into recovery offers while his wife was about to give birth to their uh second child like

[00:55:31] any day now the day of his day of his dad's funeral um he left the funeral in a mad rush

[00:55:38] to get to the hospital so it could be as with his wife as she was giving birth right so it was like

[00:55:44] what the hell like are you kidding me like you know first of all getting hit with this whole thing was

[00:55:51] bet like you know it's terrible and now like this thing has to happen to this poor man and

[00:55:57] and then go through like he was denied seeing his dad in the last days of his life and then he was

[00:56:02] denied really to be there emotionally for his wife and the birth of their son because you know where

[00:56:07] his head was it wasn't work as much as we told him stay away like forget here we got you he was

[00:56:14] like how I can't walk away like we're in the middle of war right so yeah and every good MSP

[00:56:22] you know engineer that I've ever met at all and you good they did just care they just really

[00:56:27] care like you know I know that they love solving problems I know that I do and you do Robert but

[00:56:33] the the reality is that they just can't walk away you know they just care too much and

[00:56:40] I think there's some of them like you know you could say you're fired

[00:56:44] and they would still go restore them or something you know to me like not that you would do that but

[00:56:47] it they just they couldn't walk away and leave it they don't walk away they just care too much

[00:56:51] and sometimes I don't even care as much about you as they care let me restate that that's not what

[00:56:57] I'm really meant it's like they almost care for the client just as much as they care for the

[00:57:02] organization that is employees them right like they look at that client as their employer as well

[00:57:08] right they're they're just by extension a part of of of us and it's really honestly the relationship

[00:57:16] that we try to instill with our clients is that we aren't extension of their business and I think

[00:57:21] our engineers take that to heart so it's a good point yeah I did the day they're the people

[00:57:27] who are supporting their people so it's hard when you had years supporting them to just you know

[00:57:32] to not take that personally right um I want to make sure that we get to this part that I'm really

[00:57:38] excited for you to talk about which is what you learned and what you did afterward and there's this

[00:57:46] great response and there's a great resource that I was unaware of until I met E. Robert

[00:57:52] that I think is not well known so tell us tell us what drove you to create this or so I'll try

[00:58:01] to give you an abbreviated version of this but it was in August of that year when I was sort of

[00:58:06] uh reministing is not really the right word maybe ruminating on what had happened to us

[00:58:12] in the prior month and a half and feeling grateful right um for all of the help we received

[00:58:21] and just kind of reflecting on everything that happened you're feeling grateful a month and a half

[00:58:27] later I was feeling grateful for the support and help that we received for sure for me yeah

[00:58:33] yeah uh I had a lot of gratitude in my heart and appreciation which you know I uh learned

[00:58:39] to make sure that it was very vocal about and it was in that moment that I had real like

[00:58:46] I started to question myself like why why why was I that fortunate what did I do to

[00:58:51] to deserve or to receive such help and then that's when I was kind of thinking about the investments

[00:58:57] that I had spoken about earlier into community and then it dawned on me that there's a lot of MSPs

[00:59:03] out there that don't make similar investments for whatever variety of reasons they don't see the

[00:59:07] value they don't have the time they don't have the money a combination of things whatever the case

[00:59:13] may be and said how awful would it be had we've been in the same scenario but had not made those

[00:59:21] investments right and I didn't want anybody in the world to go through that and it was also born

[00:59:30] so while I had some positive emotions like gratitude in my heart also this hatred of the people

[00:59:36] that had conducted this attack against us um was also omnipresent at that moment and I said I

[00:59:45] got to do something with these two powerful emotions of both gratitude and hatred and how do I

[00:59:49] make something good out of this um gee put these two ingredients in a bowl mix uh stir well

[00:59:55] bake at 350 for 25 minutes see what happens uh well what came out of the oven was this idea um which

[01:00:02] I'll cut right to the chase I registered the domain name MSP 911 dot org uh the vision that I had

[01:00:08] with that website it would be the giant red button that you can hit in the event you have an emergency

[01:00:13] or that sort of break glass in case of emergency um because we were frozen in those first few hours

[01:00:19] we didn't know who to call we made a lot of different calls through a lot of different organizations

[01:00:23] we were a little rudderless because of the um the initial shell shock right of what happened to us

[01:00:31] and I just had this vision in my head that even after the shell shock would somebody be able to

[01:00:36] survive an event like this without the help that we got and in my mind the answer was probably not

[01:00:45] but how can I sit by and allow that to happen to the next guy or gal running an MSP out there

[01:00:54] as a strong believer in the community even though I might not have met you or maybe you don't

[01:00:58] make the investments that I've made I still feel a camaraderie with you even though we've never met

[01:01:03] we may never meet uh but I still feel like you know this is part of our community and I think our

[01:01:08] country is under attack too right that flag hangs behind me for good reason because I'm very patriotic

[01:01:14] in that way and um so MSP 911 dot org um was the idea and then I came to the horrible realization

[01:01:22] there's just no way I'm gonna be able to do this on my own I can't staff this I can't like and I

[01:01:27] wasn't looking to make money on it so fast forward to itination connect I meet MJ Shore over at

[01:01:33] Comtea he had a similar similar kind of idea rattling around in his head and I told him about my

[01:01:40] experience and then my idea with the website totally fell in love with it put resources behind

[01:01:48] what is now known as the Comtea emergency response team so if you go to that website site it's

[01:01:53] branded as the Comtea emergency response team you'll see me listed there is the chair

[01:01:59] matley is my co-chair there's a bunch of wonderful smart and amazing humans we are all 100%

[01:02:06] volunteer based we don't monetize this we don't make money on it I'm just doing this because I feel

[01:02:14] altruistically this is me paying it forward back to the community and it's my way of fighting

[01:02:21] back against cyber criminals I don't have the resources to go attack them I'm not smart enough

[01:02:27] technically to be able to work them and it'll some of you out there probably can go toe-to-toe with

[01:02:33] with a thread actor I can't technically I don't have that skill if they wanted to own me they would

[01:02:41] but that's not the point the point is that what I can do is I can hopefully be the first wave

[01:02:47] in a building wave of people of community that are rallying around together that are

[01:02:54] willing to lock arms together to say you know what when you attack one of us you attack all of us

[01:03:00] so um so that's you know the fast forward to today is that the website is active there's a form

[01:03:07] on the page that if you are compromised in any kind of way or you need help because of some sort

[01:03:13] of security incident you can fill out the form it's going to open a case in our online system

[01:03:19] there's a whole notification process somebody will get back to you I'll give you a couple of

[01:03:24] for instance is Damian since uh november since we made this last change where it's now web form

[01:03:31] we've taken three calls um they've all been very different I'll talk a little bit about the call

[01:03:38] that I'm personally handling because I just happen to be on call um and uh this is a case

[01:03:44] of an individual who bought an msp uh it is a female she has great business experience she invested

[01:03:52] a ton of money into an msp that she bought a great solid uh profitable business uh here she is

[01:03:58] humming along and then bam somebody owns their data center uh wipe and installs ransomware on all

[01:04:05] hosted servers which is their entire business model right um and now she's like looking at

[01:04:11] you know the face of annihilation staring her in the face she's like I'm gonna lose my multi-million

[01:04:16] dollar investment I've been doing this for a year and a half I'm dead I've got lawsuits coming

[01:04:21] at me left right in center you know what do I do now by the time she logged the call with us we

[01:04:27] she was already a couple of weeks into recovery so we really couldn't guide her there but the kind

[01:04:33] of coaching and assistance and um a camaraderie that we can offer through the emergency response team

[01:04:39] in that particular case was me just saying to her hey guess what I went through this um exactly

[01:04:46] I just just in sharing my story with her and allowing her to sort of cry on my shoulder

[01:04:51] I think provided her some comfort it actually provided me a little bit of therapy myself

[01:04:56] right um as a as as a former victim to be able to just be able to two victims be able to get

[01:05:03] together and one of them to say I've been there I've done this here's the advice I got

[01:05:07] I'll give you the same advice consult with your attorney about this stuff but you know

[01:05:12] here are some things that I think you ought to be doing things that you ought to say to your team

[01:05:16] things that i communications that you should be sending to your customers um and here's just some

[01:05:21] personal advice right I'd like to say that she has um uh warmly received um that uh friendship from

[01:05:30] me and I you know hope to continue to be able to provide that service to her over the weeks and

[01:05:36] months that unfold for her at least i'm a voice out there that she can reach out to and go

[01:05:40] robberton having a really awful day today it just needs somebody to talk to

[01:05:43] hmm that's that's so amazing the ultimate vision here Damian and we're not quite there yet is to

[01:05:50] be able to replicate what happened with us to be able to um uh put an alarm or to alert out there

[01:05:56] and rally volunteers who are willing to come to your place of business and help you with recovery

[01:06:03] efforts the way we received help that's the old excuse me that's the ultimate vision um legally

[01:06:09] there's some challenges with that that we're working through um but um why come to you because

[01:06:14] there's the Switzerland of uh the msp space uh they don't compete with anybody they're not

[01:06:19] you know it's not like an rm tool or a ps a tool or a cyber security tool uh they're not competing with

[01:06:25] anybody so yeah everybody loves come to you so uh it seemed like the right place to put it

[01:06:31] yeah i love that it's not vendor owned up vendor controlled i love that correct um yeah i think

[01:06:37] the more you said community but count us in count you in count other msp the more people we get united

[01:06:46] the more will be closer you know maybe in some cases not always folks getting on a plane the more

[01:06:52] folks that connect to what you're doing then we might be able to help in different ways and have

[01:06:56] different experiences in different expertise um and uh i love the reference you know to a my to

[01:07:04] to uh to the flag you know and uh because you talked about if we get together we can defeat the

[01:07:09] bad guys right maybe think of united we stand divided we fall right and um and so if i'm an msp

[01:07:17] should i do this proactively should i get registered and signed up just in case do i hit this button

[01:07:22] do i only break in the case of ransomware attack do i rally the troops and get restores going and

[01:07:29] call the others and then you guys after you know legal and instant response help me understand uh

[01:07:37] the intention is that you would uh put in that call to us you know first because we're going to

[01:07:42] ask you questions like our intake form as as a for instance we'll ask you questions hey like

[01:07:48] do you have cyber liability insurance yes did you notify them no okay hang up with me go call

[01:07:54] them first right because you need legal representation insurance needs to be notified uh you know

[01:08:00] there they're gonna be able to help you with that but once you get settled with them give us call back

[01:08:04] tell us how it went right now i got the uh kyle handsloven in my head right about the the advice what

[01:08:10] did they tell you are you comfortable with them right because if you're not comfortable with them

[01:08:14] then you need to push back right and you need to you need to stand your ground like don't

[01:08:19] like i know that you're feel like you're in the er room and you're just gonna listen to whatever

[01:08:24] the doctors are gonna tell you you gotta stand up for yourself and say hold on a second i don't think

[01:08:28] that's the right course action here and here's why right um our goal and mission is not to

[01:08:35] uh not not to influence them but just to open their eyes up to the possibilities before them right

[01:08:42] we are not there to act as their legal representation or their forensic people uh but we can help

[01:08:47] validate things we can help share experiences uh so that they feel and just to be able to feel like

[01:08:55] most msp owners are what single owner i happen to have a business partner so i leveraged you know we

[01:09:01] leveraged each other in in that time but most msp owners are are they're it's lonely at the top by

[01:09:08] hear them say a lot as a facilitator right um it's like it's just me i make all the decisions

[01:09:13] i got 20 people who count on me and they're gonna look at me like you know like blinking you know

[01:09:20] robots like what do we do boss um it's all on you like don't you want somebody you can call and say

[01:09:27] i got your man i got your back um we're gonna help you i don't know you you don't know me but

[01:09:32] like we're gonna help each other right um just to be that voice uh there is um you know mental

[01:09:39] health is becoming a bigger and bigger issue in our space and even in business

[01:09:45] but in our space in particular Damian and i want to say we're kind of bleeding into that area

[01:09:51] of this mental health the psychological and emotional impact that's something like this can have on

[01:09:56] you it could be very devastating uh that alone other than the financial and the reputational

[01:10:02] damage that can unfold but just that other the human side of this uh could be enough to

[01:10:09] you know make you know take you from a strong leader that's gonna get through something and survive

[01:10:14] to someone who fails miserably because you just didn't believe in yourself

[01:10:18] and you feel like you're attacked on all sides right yeah it's not just the ransom attack right

[01:10:24] you you you feel responsible to your team you feel responsible to your clients you feel all these

[01:10:29] emotion you want it to be done immediately and of course it won't right so there's just so many

[01:10:34] of those i'm glad you brought up that because yeah if you can never monster vibe but even just

[01:10:42] step up and i don't mean that and you know you're going to feel this way like you know if anything

[01:10:48] i want people to have permission and give them some grace to feel that way and if they can reach out

[01:10:54] and you know contact you through msp9on.org contact me contact you know contact somebody in peer group

[01:11:04] and they they've either been there or helped others through it um it's all the difference in the world

[01:11:09] because you're you're not alone and you don't feel maybe like you felt this guilt or shame

[01:11:16] for how could i let this happen how could i let you know and it gets deep right like there's

[01:11:22] how did i let this happen to my team or clients which is really really deep then you know

[01:11:27] like you were saying how does this happen to my family how do i you know what you said hit me like

[01:11:33] do you call your daughter back home from college right to have that thought is to be human

[01:11:38] but like that's a that's a horrible thought that i had that you know that you had to go through

[01:11:43] and um and just that community is it's just it's an amazing gift i will say that you know

[01:11:50] being that somebody that that helps msp exclusively with back of a disaster recovery we see

[01:11:56] ransomware all the time unfortunately and in all these years i've been asked what do you do

[01:12:03] and not the technical angle it's really just like what do you do and the best answer i had before

[01:12:08] was build those foundations with not only us but most importantly your peer groups right that's

[01:12:14] who you can call kind of what what you ended up doing before i knew you robert and um so i i'm just

[01:12:22] completely over the moon about uh msp911.org comptia emergency response um

[01:12:30] um all volunteer staffed right you're you're you're not making money they're not making money

[01:12:35] but more importantly you're not paying for this and this is worth its weight in gold

[01:12:40] so one of the most like a lot of things life won't you know some of the best things are free

[01:12:44] and um and so i just you know we're going to do everything possible to spread the word um

[01:12:49] great and uh and connect people to this because this is just an amazing resource if anybody out there

[01:12:55] feels like that they could be uh you know we're not just asking for you know random volunteers

[01:13:00] everybody wants to help i get that but if you have a specific skill set in cyber security incident

[01:13:07] uh response crisis management you know that's kind of some of the skills that we're looking for

[01:13:12] everybody will raise their hand and say yeah yeah yeah i'll help you know put my we're really kind

[01:13:17] of looking for people with that sort of experience and even if you've been a victim believe it or not

[01:13:22] that's i hate to say it this way but it's resume man um and so we will take that so at the very

[01:13:30] very bottom of that web page there's a little uh kind of uh un uh ceremonious link at the

[01:13:36] bottom it says click here to fill out a volunteer form not the forum that you see on the page that

[01:13:41] will actually alert us that there's a problem uh but way at the bottom and uh we'll just exercise

[01:13:48] in patience please we've got a lot of incoming requests where like i said we're all running our own

[01:13:53] businesses and working hard to squeeze in an hour of time a week to help move the the the needle a little

[01:14:01] bit forward it's been two years in the making so far uh but we're finally kind of operational and

[01:14:08] delivering on these services so if you volunteer or you do volunteer you know here's back right away

[01:14:14] trust me we got your information we will get to um and you can always contact me you can find me

[01:14:20] on LinkedIn i'm not hard to find i'm very visible out there um uh and just shoot me a note and let

[01:14:28] me know uh what you know uh what your concerns or what your issues are about it really about anything

[01:14:36] honestly this has been a huge gift Robert thank you um we'll make sure to put msp on on one

[01:14:42] dot org and if you're listening you already know where to go in the links we'll put a link to your

[01:14:47] LinkedIn as well um so um Robert anything else you would advise people as we close out today

[01:14:55] now just uh you know listen uh you've heard this before you don't need to hear me say it again but

[01:15:02] i think it's worthwhile to note that remember cyber security is a journey it's not a destination

[01:15:08] uh and how do you eat an elephant one bite at a time so if you feel like you're behind on things

[01:15:13] uh it's probably because you are but you know what take a breath it's okay just take a step forward

[01:15:18] just do something small today or next week that improves your cyber security hygiene or posture

[01:15:25] or end or for your customers and just keep that continuous improvement mindset uh at work so

[01:15:33] um Rome was not built in a day uh you will you two will get there

[01:15:37] absolutely yeah on that note anybody's interested we um have a resource that's a slightly

[01:15:45] unusual but cyber security of backups we give it away so doesn't matter what tool you use if

[01:15:50] you'd like to apply that and basically the the lens of this is what do i do to protect my backups

[01:15:56] so that i can recover from local and not always from cloud and so there's it's a 1% thing it's what

[01:16:03] you said you know obviously they've got the basics um you know like not reusing credentials or not

[01:16:08] domain joining things that shouldn't be um and then you've got things like separate v-lans and other

[01:16:13] things that might take more time and know that like you said Rome wasn't built in a day so um

[01:16:18] so make sure to take advantage of that um speaking of time i know we could probably talk about this

[01:16:23] all day Robert i want to thank you for this gift of your time and sharing this incredibly traumatic

[01:16:30] experience and how you've grown your company has in terms of culture and community and you know the

[01:16:38] fact that you've uh built the comp tia emergency response it's just amazing thank you so much

[01:16:44] well i appreciate your your time and uh your uh platform to spread the word even further i don't

[01:16:50] do this for me i'm doing it for us and i really i mean that sounds crazy but you got to know me to

[01:16:56] understand how genuine i am an authentic i am about being that uh everything that i say and do

[01:17:03] it's really about us uh it's anything that i get behind is um something that i genuinely and

[01:17:09] authentically and passionately feel like this is important right um um otherwise just just spend my

[01:17:17] time promoting myself to actual paying customers maybe do some of that too right i try um and

[01:17:25] i do some of that uh but i could ignore all of this right i could have said i'm going to put this

[01:17:31] a hundred percent behind me never thought think about it never talk about it again

[01:17:35] but i just felt like it was not the patriotic thing to do for me for me it was i saw an opportunity to

[01:17:42] uh kind of raise that middle finger up to the cyber criminals in a way that maybe that is not

[01:17:47] expected right and that is like i said uh to to to move the community forward together

[01:17:54] yeah absolutely well thank you for doing that it takes courage to share this and courage to do

[01:17:59] something about it um robert we may have to to have you on and talk more about this uh especially

[01:18:04] as the emergency response center grows uh well and there are other parts of the story are still

[01:18:10] unfolding there's the case against jr slava synski um their sentencing is on march 1st of 2024

[01:18:16] i'm sure i'll have more news to share about that okay well thank you so much robert all right thank you

[01:18:23] all right bye