Certification Without Accountability: Adwait Nadkarni on the Liability Gaps Facing MSPs

[00:00:01] The security products you buy, the certifications you trust, the gear you put on a client's network, a lot of those signals are weaker than the label suggests. Adwait Nadkarni has spent years proving it. In this episode we talk about what a vendor's weekend scan everything is actually worth, why certified can hide deliberate evasion, and what an MSP should actually do when the signals they rely on can't be trusted.

[00:00:30] This is the Business of Tech. Adwait Nadkarni, you are an Associate Professor of Computer Science at William & Mary and the Director of the William & Mary Cybersecurity Center. Welcome to the Business of Tech. Happy to be here. Now you're in a whole field of diving into cybersecurity and understanding why, and I really want to help make our operator audience think about this. These are the people that are buying, deploying, and standing behind these technologies.

[00:00:59] Your research really shows that these security tools that everybody trusts, like the ones that are baked into the products they buy, and the pipelines that the vendors brag about, well, they miss basic decades-old flaws. For an operator who can't read the source code and is taking a vendor's WeScan everything on faith, what is their claim worth and how do you measure it?

[00:01:25] What we have observed in the wild is a lot of these times these tools are usually sold via demos, right? You have someone who wants to sell a tool come to you with a demo and they do it on a certain small test set of vulnerabilities. And as that is proprietary code, you as the operator have no access to it. You know exactly how good they are, how good they would be in the wild.

[00:01:50] So mostly what happens is you end up buying something based on reputation or based on what your manager tells you or based on what your friends in the market are saying, right? But one thing that's that stood out in our interviews with operators and we are mainly coming from the product side and not really the IT side is there are some companies that perform hostile reviews of the tools as they come to them.

[00:02:16] So think about, and again, my examples are mostly going to be on the product side, but you will see parallels on your side as well. You'll find everybody has those vulnerabilities that have burned them in the past or that were very close to burning them in the past. So one strategy we've seen people use very successfully is inserting those very vulnerabilities into live production pieces of code before deployment.

[00:02:43] So alternate shadow copies of code and then using those copies of the product to evaluate the vendor that's trying to sell a product to you. That way, you know if they could catch something that has burned you before and you just know how good they are. And some of the tools that we have developed actually allow you to do that at scale and with varied levels of complexity to sort of truly challenge the detection tool that you're being sold.

[00:03:08] Is there like a particular way, particularly for those that are not necessarily developers that are those that are those implementers? Are there research toolkits or ways that they can help test the products that they're deploying themselves rather than purely kind of testing? And at the same time, you don't want to just let it loose on a client environment. Like how can you put it through some of the rigors to get a better sense?

[00:03:32] Some of it comes down to the documentation itself, looking at exactly what the product is trying to claim. And when we say product here, we mostly mean the evaluation toolkit, right? The security product that is being sold to you. So examining those claims a little more critically than just giving it to the hype is really the first step.

[00:03:56] Once you're past that, there are several research toolkits that are out there, some that we've developed that allow you to see vulnerabilities into products and then evaluate existing tools or the tools that are being sold to you on those products, mostly automatically with very little manual intervention. And with language models, in fact, the amount of manual intervention that you would have needed in the very recent past, you need that at all. So you'd be able to automate most of that.

[00:04:24] Now, one of the things that fascinated me is you found that certified IoT products were with code written to look secure while quietly using vulnerable parameters. Like, you know, this is kind of a deliberate evasion that still passes certification. You know, MSPs put certified gear on clients networks all the time. Like when certified can hide things, what can an operator realistically check before a device goes on a client network?

[00:04:54] Yeah, that's a really tricky question, you know, because it also is borderline illegal to hide vulnerabilities. I'm saying borderline illegal because the regulations that go into that back certification still do not spell out what liability looks like in case of failure. So we've got some other research looking at that. But so we've seen examples mainly on the product side, right?

[00:05:19] We have seen developers hide vulnerabilities and essentially make a vulnerable parameter look like a secure parameter. We, we, we hesitate to say it's intentional backdooring of the software because again, you don't really have window into the developers intent.

[00:05:38] So you can't say that confidently, but these are unequivocally cases of evading mostly static analysis tools and by extension also human eyes, human code reviews, because even as a person, when you look at the code, you won't really understand. You won't really see anything wrong with that piece of code unless you look much more closely.

[00:06:00] So for MSPs, the lesson to take from here is really that you have to take certification with a pinch of salt. I really hope that certified products, hardware products, especially in the MSP landscape are much better off than IoT applications. So fingers crossed. But I wouldn't really assume that as a guarantee because essentially what this is, is cheating. Right.

[00:06:24] And if you found application developers cheat in IoT products, it wouldn't be a stretch to say that they would be cheating in other domains as well. Well, tell me a little bit more about that, because a lot of the operators really do live on these badges. Right. The idea of certified or sock to or pass to pen test. But you've done specific work focusing on the labs and how they forum shop and compete on speed. Tell me a little bit about how that process works and what your research shows.

[00:06:50] Generally, when it comes to compliance, what ends up happening is you want to get a product certified with a certain regulation in mind. You shop around for a lab that can perform the certification and the lab certifies a product for you. You pay the lab and that's that's the entire loop. Now, if you notice here, you see that the regulators that make these standards are not really in the compliance loop itself.

[00:07:16] And they start off staring at you from outside the compliance loop and they represent consumers who are really doing the same because all the regulator does is licensing labs that can make this compliance scale. Right. So they're really out of this loop and they don't have a say in exactly how the certification occurs. There are some guidelines, but it's impossible for regulators. It's impossible to have an FDA like ecosystem where the FDA goes through every single product that goes out there.

[00:07:44] We can't have that in tech. We just won't be able to scale like that. So what ends up happening is I'm thinking about from the lab's perspective. If you as a vendor want to get something certified, you're more likely to go to a lab that's going to give you a quick route to certification, regardless of how rigorously it looks at your product. Right. You'll probably hesitate going to a lab that's going to return a product 50 different times for multiple vulnerabilities. And how you fix every single one. And again, this is really a question of how much you value your product.

[00:08:13] If you just want to ship products continuously and especially when it comes to software, you're more likely to go to a vendor that's just going to quickly pass your product. So when it comes to labs, if they see vendors pivoting, other labs that offer quick certification, they also have no incentive to rigorously certify products. And this leads to a situation where there is no accountability.

[00:08:37] And because there's no accountability, labs just offer quick certification and vendors flock to labs and certification essentially just becomes a liability shield where all you're really doing is having something to show a badge to get people to buy a product, but nothing real, nothing technical to actually back. We'll be right back after this message. If you've been watching the backup market, you know, pricing has gotten complicated.

[00:09:07] Veeam feels like legacy overhead. Some of the newer platforms have gotten expensive fast. Comet Backup is what I keep seeing MSPs land on when they want modern backup without the modern price tag. Bring your own storage, control your costs and run it your way. Comet Backup is built for MSPs who want flexibility without the vendor dependency. Check them out at Comet Backup.com. And we're back.

[00:09:37] I'm trying to sort through and understand the various products and offerings that I got. Can you give any guidance on what the research says, like the way that an operator can sort through this? I think operators should mainly look for legal guarantees at this point because that is what I would do if I was purchasing something like this.

[00:09:55] Regardless of what the certification says or regardless of whether a product is certified, the contracts between whether they're between operators and the products they purchase or between the product vendors and the labs that perform the certification. Those contracts determine who is liable and who bears the cost in case a product fails and in case there is harm when a product fails.

[00:10:20] So at the end of the day, what really matters is liability because that is the stick you can eventually beat people with, right? Metaphorically. So as an operator, what you really need to look at are the contracts you're signing. Regardless of what the certification says, someone sells you, it's just like buying a house.

[00:10:38] If someone sells you a house that has gone through inspection, but then there is a line under the inspection report that says this inspection doesn't mean anything and you can't hold us to anything if anything in the inspection fails. Then at the end of the day, you have no guarantees. So you really want to go for some hard guarantees in the contracts where there's agreements with the product vendors whose products you're buying or with the regulators who design the standards or whether it's the labs that offer certification.

[00:11:06] Analyzing, looking at those agreements before you get into business matters. Okay. Now knowing that we're both not lawyers in this conversation, right? Like you, you've done some specific law school work that found vendors are able to contract their way out of liability. The idea of use TLS when possible, right? The nest disclaimer. Say a little bit more because MSPs end up are the ones that are caught in the middle, right? They're the ones that have to accept the vendor and user license agreements that disclaim everything.

[00:11:33] But on the other side, they have to deal with clients where they're promising protection. Like talk me through kind of the specific things that you're the researchers found or the ways, you know, the vendors are kind of get around this and what you can do to counter that. Yeah. So then the disclaimer about use TLS where possible, it was in I believe it was in a standard. There's a standard called IOXT. It's also an alliance of vendors and labs as well as the standard has the same name.

[00:12:02] And that was a disclaimer that was the language used in one of a part of the standard that dealt with TLS use. Essentially, they required everything. And at the end, they suffixed it with where possible, which made the whole preceding requirement completely meaningless. So that's a gap in the standard itself. What we have seen vendors do has been mainly through the analysis of licensing agreements.

[00:12:28] We've seen everything from vendors disclaiming harm resulting in death and injury. And we're not lawyers, but that is illegal under US law. It's pretty clear any court would throw it out. We've seen situations where they're not so brazen as to have an illegal statement of the EULA. But they would still have, you know, for example, the smoke detector and it's a very, very popular brand.

[00:12:57] And in the licensing agreement of the smoke detectors and the security alarm from the brand, you see a statement that says these devices should not be used in for anything that is time sensitive and anything that is safety sensitive. And this again is for, for devices that are smoke detectors, security cameras, smoke alarms and so on and so forth.

[00:13:20] Right. So we've seen situations like that where the, in fact, where your typical non smart smoke detector, just the $10 smoke detectors that you can put up there. They probably don't disclaim those use cases. They don't disclaim safety as much as smart smoke detectors do. So that is one of the things that stood out.

[00:13:42] Now, among other things, you've also seen slightly subtle, subtle liability waivers where vendors would essentially make things very ambiguous.

[00:13:52] For example, there's a, there's a EULA where the vendor essentially said something like, as long as you use the product, if you do not use the product with the, with the appropriate set of batteries, the product would essentially, you would void all, all live, all warranties.

[00:14:17] And there's no specification of what the appropriate or high quality set of batteries is. So no matter what you use, they could just say, well, there's not high quality set of batteries, all warranties thrown out. And again, these are like, they're actually disclaiming liability for JAM in general, and also disclaiming all possible warranties. Besides this, again, there are many, many cases.

[00:14:39] I mean, I could, I have to jog my memory, but there are situations where vendors would essentially, this is very important when it comes to labs, because labs, again, essentially say that since vendors are eventually responsible for developing their products. Well, we are not liable for any harm, regardless of whether the product is certified. So that is another thing that we've seen.

[00:15:04] But yeah, so in general, what I can say is that vendors disclaim all or most liability, and you can't even trust that vendors would back the bare functionality of the product in the licensing agreement itself. We'll be right back after this message. We'll be right back after this. This message is by Chris who will be right back after the purchase of MSPs, and I'm going to go to the next page. Here's what I keep hearing from MSPs. The security tools are fine. It's the work underneath them that's breaking people.

[00:15:32] Too many alerts, nobody on staff to triage them, and client reporting that eats the whole week. That's the part Guards is going after. A consolidated security platform built for protecting SMB clients, endpoint, email identity, but with an autonomous analyst working underneath it. it, doing the triage and the reporting that an MSP usually can't afford to hire for.

[00:15:58] Real security operations at the scale you actually run at. Visit guardswithaz.com. And we're back. An MSP only ever sees the sales pitch. Like, are there particular questions to ask? Are there answers that you should worry about that help separate one vendor who actually cares versus one that's performing it? Like, give us some guidance on the way to engage.

[00:16:28] So this is very tricky. So Samsung, I interned there way back, I believe, 12 years ago. And I actually observed in the company itself a very good security culture. In fact, very few companies have a culture where products go through design security reviews, right? Right when you're designing the product, security is integrated into the product at inception. And that is something that I saw that was really cool. And that was one of the very few places that did that. But in my

[00:16:57] research, I've also seen places where, you know, you could point out to a vulnerability to a vendor. And they would essentially go like, well, is this exploitable? If it's not exploitable, I don't want to fix it. It's like saying, you know, like, it's like me pointing out that there's a bomb under your house. And you're like, well, can you light a fuse to it? If you can't light a fuse to it? Well, I'm not going to worry about it until you can, right? Because it's a very, it's a ridiculous reason to, to not fix vulnerability, because you could have a

[00:17:25] developer come in, say six months later, who had no idea how the most of the code base works, and who would then connect to that vulnerable piece of code. And there that's an exploitable part to the vulnerability. So you can't really just keep existing vulnerabilities in there and not fix them. Now in this case, and this is a really cool story is like, we found a vulnerability. We then demonstrated it was exploitable. We reported the vulnerability to the vendor. The vendor said they fixed it. And then when we looked at the newer version of the

[00:17:55] product, we found that they hadn't fixed it. They had moved it to a slightly different location. So the vulnerability still remained, still exploitable, literally in the same class. And all they did was refactored code a little bit to make it look like it went away. And I don't know what to think about it. Like they could have fixed it. It's not hard to fix, not hard to exploit. So for MSPs, again, it's very difficult from,

[00:18:20] from you from the vantage point at which MSPs stand, it's very difficult for them to directly evaluate things or directly fix things. At least that is what I get from, from a lot of our conversations, right? Because you don't have access to source code. And you may or may not have enough pull with the vendors to make them fix things. But what you could do, like I said, is legally shield yourself very, very well. Look at what you're signing. Look at what the vendor's

[00:18:49] promising, especially when it comes to security. If they offer a certified product, see what actually backs the certification. If there are, if there is a way to seek damages, or if they accept liability, if the certified product fails in certain ways, then you should probably go for it. But if not, those guarantees mean nothing. So really knowing the value of what promises are being made to you is the most you can do at this point. And this makes you equivalent to consumers

[00:19:18] as well, because as consumers, that is what we can do as well. When you buy an IoT product, you look at what's being promised to you. You look at how reputed the vendor is, and then you make your call. Now, give me a little sense, like broadly speaking, I mean, I'm getting, you know, I think I know the answer here, but I'd like to hear from somebody who spends their time thinking about it. This doesn't feel like a technical problem. This feels like a policy problem. Like, I'm sure you've given some thought to the way this would get addressed. Like if we thought about it more broadly, how would you address this with policy?

[00:19:47] This is a policy problem. Absolutely. It is because again, a lot of the vulnerabilities we've found, a lot of the, those that vendors were hiding, they were not difficult to fix. Why they weren't fixed is a good question. In some cases, some things cause actual delay at runtime. Let's say you're a smartphone vendor, and you're trying to make a phone boot up where your users value you based on how fast

[00:20:12] you can make it boot up. If you use a certain kinds of encryption, or if you use certain cryptographic hashes that slow it down by a couple of seconds, your customers may or may not tolerate it, or you may or may not be able to sell it to your customers. Right? So that is one reason performance is one reason why people use bad cryptography. At least that is the one thing we found. That's not the only reason a lot of times the ecosystem where you've got third party contractors

[00:20:42] were absolutely no stake in the product. They just want to go from job to job to job. And if you put a lot of QA on them, they're not going to do it. There's going to quickly wrap it up, send it to you. Right? So it is a problem that's caused by how fast the software ecosystem evolves. And again, I'm speaking only for the software ecosystem, because that's what we've looked at. Hardware is a

[00:21:06] little slower. So you may or may not see problems of this scale and seriousness in hardware, but you might still see it. But in software, at least it is the pace at which things evolve a pace at which companies have to release products. That is what causes a lot of these problems. As consumers, we can go for and especially MSPs being consumers for some of these products can together, even if they can't

[00:21:36] do it individually, they can probably form a consortium of sorts to evaluate products and to back products that are actually secure. I don't know if a consortium of sorts exists, but if it doesn't, that is something that really needs to happen because you see that those kinds of movements in the general software engineering industry as well. I would I would I would see is a great example, although it didn't really work out well.

[00:22:02] To my knowledge, IFC is again is a consortium of vendors and labs, but the certifications, really certified products have vulnerabilities, but it doesn't always have to end that way. If you have a consortium of MSPs that can evaluate products, or that can at least guide what products and what security standards are adopted and how rigorously they're adopted, that is one way the industry can help

[00:22:28] itself. Because it's very hard to expect the government to help the industry in this regard. It's a big industry, but it's still not exactly the focus of the regulatory regime right now. So you just have to deal with that unless you can somehow motivate someone in government to take notice and you know regulate this. If you want that regulation, that's another question, right? As an industry,

[00:22:55] do you really want to be regulated? Do you want additional regulations that just turn out to be more work and more red tape without any benefit? So I believe the way one way to solve this problem is for consumers like MSPs to band together and develop, at least take a stand on exactly how secure you want the product that is for you to be. Well, that's the call to action right there is for the industry to band together.

[00:23:21] Adwait Nadkarni is the class of 1953 associate professor at William & Mary and director of the William & Mary Cybersecurity Center where he runs the Secura Platforms Lab. If people are interested in learning more, seeing some of your research, what's the best way to keep an eye on that and keep in touch? The best way, honestly, is to just watch my website. I usually post papers that are published right there. Also, William & Mary news often covers the research that we do. Not just my research,

[00:23:46] but the CA's research done by everybody who works on cybersecurity in CS. I feel free to reach out to the William & Mary Cybersecurity Center. Awesome. Well, thanks so much for joining me. It's been a great conversation. All right. Thanks. Want more from the Business of Tech? Join Business of Tech Plus for ad-free episodes, early interviews, extended cuts, subscriber-only shows, and exclusive member perks and analysis.

[00:24:12] Sign up at businessof.tech slash plus. And follow this show on your podcast app. And if you're on YouTube, hit subscribe and the bell so you never miss a story. Reviews and comments help spread the word too. Interested in advertising? Head to mspradio.com slash engage. The Business of Tech is written and produced by me, Dave Sobel, under ethics guidelines posted at businessof.tech. Thanks for listening.

[00:24:41] I'll see you on the next episode. Produced by Picture This Video. Part of the MSP Radio Network.