Deepfakes, Data Poisoning, and the Governance Gap

[00:00:06] Welcome to MSP 1337. I'm your host Chris Johnson, a show dedicated to cybersecurity challenges solutions, a journey together, not alone. MSP 1337 I'm joined this week by a very unique guest to the show and I say unique because we don't often have someone with this level of expertise or experience.

[00:00:34] And so I'm excited to welcome Dr. Stephen Wright of Macadamia Solutions to the show. Dr. Wright, welcome to the show. DR. Thanks for having me. DR. So one of the things that caught my interest when it was put in front of me that you would be a great guest was the book that I think you recently wrote around the intersections of technology governance.

[00:01:03] DR. The Venn diagram that you used to illustrate showed the three circles of business, law and technology and that intersection being governance. And ironically, the timing is quite perfect because the world that I spend a lot of time supporting is tied to the delivery of IP services into predominantly the S&B, but it's the mid-market space even as well.

[00:01:31] DR. And there's this really scary spot when we look at the maturity of organizations providing these services and it's the area of governance. And particularly since we're dealing with the technology space, the governance of technology. DR. And so I would just like maybe if you could give us a little bit of background. I know you've written multiple books, you've got patents, you do a lot of things that, you know, around data supply chain and protecting data.

[00:02:00] I mean, correct me if I'm wrong, you know, walk me through a little bit about, you know, your history and kind of where you landed on and why you decided to write this book. DR. So my background, I've been working in the technology space for many years in four different countries, actually. The US, Australia, the UK and Belgium. DR. I've worked for a number of different companies.

[00:02:31] And I've worked at, I like to joke that I started at the bottom layer designing circuitry and I've been working my way up the protocol stack ever since. DR. Wow. DR. And then I went to law school and trying to connect the two together.

[00:02:52] DR. So one of the things I've learned over the years is that particularly in big companies, it's very easy to get very siloed. And so you have folks that are worried about, you know, the legal side of things, they're protecting the business. They're worried about other folks who are on the technology side. They're trying to push the company's technology forward and the business folks are trying to figure out how to make money out of all of this. DR. Sure.

[00:03:21] DR. And they don't always talk to each other in a language that makes sense. DR. Or at least... DR. Or even in the same language. DR. Or there's some sort of partial translation going on. DR. Sure. DR. And they both think they go away with some understanding, but it's really not there.

[00:03:40] DR. And that, I think, becomes even more problematic with the rise of AI as a sort of general tool suite for bad actors to do bad things. DR. So where in the past your business might tend to delegate things down to the IT to take care of the technology side,

[00:04:08] DR. And there had been very limited risks to the business from that. DR. With the advent of Sarbanes-Oxley and for bigger companies at least, you've now got board level responsibility for cybersecurity events and things like that going on. So it really does have an impact on the business that there are failures that can happen.

[00:04:36] DR. It's not just Sarbanes actually in the public company space. DR. If you are a government contractor, particularly in the defense space, there's things like the CMMC. DR. Yeah. DR. And if you're a defense contractor and you can't prove that your security arrangements meet the levels of CMMC that you say you have, then you could be in for a world of hurt.

[00:05:06] DR. Yeah, that would get you real quickly into potentially violating False Claims Act and a number of things. DR. You know, as I was thinking about what you were saying, you were kind of going down this interesting path that I like very much because this came up today in a conversation. DR. Actually, it was a presentation that I did on the opportunity of selling beyond risk with regards to AI and what does that look like.

[00:05:29] DR. And one of the questions that came up, someone brought up the recent exploitation of hugging face. DR. And the first one was like, you know, this happened and we were at this AI got out of the cage and it did this thing to the company. DR. And then you've got another AI company saying, and we did it to two companies. DR. And then there was the, and we did it to three companies.

[00:05:57] DR. And the part that was really interesting to me is like no one's talking about the fact that these are crimes that are being committed by the very nature of what was done. DR. And yet, we're hearing about this through like marketing spokespeople, not like, hey, there's an investigation going on and, you know, legal and law enforcement are at work here. DR. How are you seeing, you know, the transformation of the things that we're seeing play out in the news?

[00:06:24] DR. And obviously, our tools that did what they were intended to do and we're not surprised, but we're trying to make people think that they should be surprised at what has happened.

[00:06:33] DR. And on where this goes next, because I think we have at least a few more years of chaotic disruption as we learn to catch up and rein in the things that we are building, largely by putting in place the things that we have been hyping or promoting and saying that need to be done for the last 20 plus years, 30 years of security best practices that now more than ever are a critical component to anything that we do.

[00:07:03] DR. What are you thinking or seeing? DR. What are you thinking or seeing maybe this is probably more opinion than fact today, because we're looking at the future, but what do you see is coming down the pike, if you will, on that front? DR. Oh, there's a number of things coming on. DR. Yeah, a few. DR. I think there's, firstly, there's a sort of broad trend towards increasing legalization of broadly unbearing.

[00:07:32] DR. I think there's a lot of things coming up with the IT and generally cyber security type issues. That's increasingly, there's more and more legislation, both at a federal and state level, that's impacting companies.

[00:07:48] DR. So, you know, the Europeans keep inventing new extraterritorial legislation to protect their citizens as well. And if you have customers out of the country, you could be impacted by that. DR. So, while everyone likes to make lawyer jokes, your company lawyers do have a real job to try and keep a track of all of this stuff and figure out what applies to your company. DR. Sure.

[00:08:17] DR. And so there is a real and rising concern there. And there's pros and cons of that. On the one hand, it is setting the bar higher for the standards for competence for professionals in the world. DR. So the systems are generally hardening over time. And in that sense, things are getting better. But the offense side is also getting better.

[00:08:45] DR. And they're just as happy to use AI tools as anybody else to streamline their code development and so forth. DR. Do you think there's a risk in sort of what we're seeing in this approach? And I liken it to when Metasploit, the tool Metasploit came out and there was a lot of talk around, we shouldn't let this get into the public's hands because of all the bad things that you can do with it.

[00:09:14] DR. And it's like, well, the threat actors are already using this to do bad things. If we don't allow this into the hands of the public, then how do you create the defense side of this? DR. Like we want to at least be able to defend with the same level of tools as the adversary is using to attack us. DR. I feel like we're at a crossroads right now of as AI models continue to mature, this sort of same concept is coming back to the forefront along those same lines.

[00:09:44] DR. Yeah, and that's I think a particular issue with the legislative side of things because often you get the different legislatures rushing to put their stamp on something to constrain what they see as bad behavior.

[00:10:09] DR. But that may not be the final form of the technology. DR. Mm-hmm. DR. On a Tuesday or a day that ends in Y. DR. Yeah, they've tried to outlaw a thing that morphs into something else and the industry moves on and now you've created this gateway to regulation for something that doesn't exist.

[00:10:35] DR. Yeah, it doesn't look the same as it did. So to recognize it as a, yeah, it doesn't look, yeah, exactly. We're looking for the wrong person, right? Like you're looking for the wrong identity. DR. You get that or else you get this, again, regulatory compliance burden to deal with imagined horrors that don't exist. And so there is that on the one hand. And on the other hand, you also have some companies that really use it.

[00:11:04] DR. They seem to be aiming for regulatory capture in the sense that they really want to create a environment or legal environment where they're treated special and once they get through the door, then that stops others from coming in. DR. Yeah.

[00:11:24] DR. So with that as a corporate strategy, then you're playing this weird game of trying to play up the dangers of your own product so that you can get the appropriate regulations in place to stop others from coming in. DR. Yeah. DR. And then you make the regulations so complicated that nobody could be in you enter into that space.

[00:11:48] DR. That reminds me of aspirational policies where you ask AI to help you build a good policy because you don't want to write it yourself. And then it writes an amazing policy because it found all the things that should go into this policy. And so it did just that and now nobody can follow it. DR. Yes. Yes, you can. You can. Yeah, you can end up in situations like that.

[00:12:12] DR. So thinking about some of the books that you've written and just kind of thinking as I was looking at the different copies, you know, supply chain, your data supply chain, you then get into blockchain, smart contracts, ethics, law, technology adoption, market research for small business, math or market research math for small business. DR. And then now looking at the technology governance, I feel like there's a pattern here.

[00:12:38] DR. Like I feel like you've kind of had this sort of like progressive seeing something missing from what we need to be educated on. DR. You know, I feel like your books are written to fill gaps that we all should be tuning into. DR. That's my limited observation. DR. Help me better understand that because I feel like you're on to something that we need to know about. DR. Well, thank you. DR. I hope it does fill a gap for somebody.

[00:13:06] DR. Yeah, so my background is perhaps a little unusual in the sense that I have an MBA, a PhD in computing and that law degree, which is an odd mix and goes back to that sort of Venn diagram of the business legal and technology overlapping. So that's the things that are in the center there are where I can add the most value to people. DR. I just thought maybe you didn't know what you wanted to do, so you just stayed in school longer than the rest of us.

[00:13:35] DR. Well, I have had folks joke and tell me that if I get one more degree, I'll be a thermometer, but that's fair. DR. That's fair. DR. And probably able to quickly convert Celsius and Fahrenheit considering the number of countries that you've lived in. DR. What is it? Multiply by 1.8, add 30 or something to that effect to get one direction or the other. I'm not very good at math, so.

[00:14:02] DR. Anyway, so the most recent book on data supply chain management is focused on a couple of types of bad behavior that aren't easily addressed by the current cybersecurity systems. DR. So things like deepfakes, data poisoning and data suppression. DR. So deepfakes, I think folks are fairly familiar with the word in other contexts.

[00:14:33] DR. So you may be familiar with folks using AI tools to generate images of somebody and put them in an inappropriate situation for that person. DR. So think of your favorite politician. DR. And even early on, catfishing would have been something along those lines, right? DR. The earlier versions before we got into the deepfakes were how to create the illusion that I look differently than I actually do, or I am somebody that I'm not.

[00:15:03] DR. Yep. Yeah. DR. And it plays into the notion of phishing to a certain extent that you're trying to make some misrepresentation. DR. So in some cases, the artifact itself is the misrepresentation. DR. You're trying to create a fake image in the style of a famous artist's work, for example. DR. Mm-hmm. DR. It's one thing. DR. But these have become much more sophisticated, and it's not just still images.

[00:15:33] You can get voices. You can get video with sound that is quite realistic. DR. Well, with our midterms coming up, it has been, I don't know if you've mentioned, I actually subscribe to a YouTube channel, and because it's YouTube, the ads that come up a lot are political ads. And it's been really interesting, like if you're not paying attention, and looking like in the top left-hand corner that says,

[00:15:57] this was a digital synthesization or a, it is not real. Like essentially it is admitting to not being the real person saying these things, yet they're still getting away with using these advertisements.

[00:16:13] DR. Yeah, it's becoming a real problem, both in the political space and in the social media space, and in the business space.

[00:16:29] DR. There have been cases where, for example, company executives have been on a video conference call, where all of the other parties on the call were these faked out entities, resulting in the business sending money somewhere for some deal that was an obvious scam. DR. Right.

[00:16:50] DR. So, this is not something that was going to be caught by your firewalls and your traditional IT systems. And you really have to have processes and procedures and people be aware of these kinds of issues. And maybe there's some contractual solutions as well.

[00:17:15] DR. So, deepfakes is one thing data poisoning is when you realize that most companies are very reliant on data to move their business forward. So, what happens if malicious data is introduced in there or data is suppressed that you don't get data that you should be getting? DR. Okay.

[00:17:39] DR. This data poisoning term comes from AI poisoning the data to use to train the AI system. DR. Sure. DR. But it's more widely applicable. It's not just AI systems that consume poison data. It's particularly the humans in the loop that are receiving this malicious data that need to be aware of the possibility of the data.

[00:18:06] DR. So, you know, I don't have systems in place to check, you know, where's this data coming from? DR. How do I verify that it is? DR. And do some sort of out-of-band check to validate what's going on. DR. It's a particularly pernicious problem because it spreads through the supply chain. So, you're getting your data from somebody upstream. You're passing your information on to your customers, clients, and regulators, and so forth.

[00:18:34] DR. And you don't want to be causing problems downstream to those folks with the data you're putting out. DR. But you're relying on getting clean data coming in to drive your decisions. DR. And so, do you even have contractual constraints on your data suppliers to notify you if they've been poisoned or attacked in some way in their data? DR. Because, I mean, it's hard enough to detect these things on data that's generated internally. DR. Sure.

[00:19:03] DR. If you're getting data coming in from somebody else, how would you know whether there's something faked or suppressed or whatever in that data? DR. It reminds... DR. I don't know if you've read the book, but the first case of a nation-state threat actor getting into a system back in the... I'm going to say late 70s, early 80s. It was Cliff Stoll wrote the book, The Cuckoo's Egg. DR. Yeah. DR. I think... Yeah, Cliff.

[00:19:31] DR. And I'm just thinking about how the... what the threat actor was doing was essentially moving small, like, pennies, moving pennies around to manipulate and basically siphon money off of an organization, basically through manipulation of data, right?

[00:19:48] DR. Like, by changing the numbers just a little bit, by not enough to be, like, more than an anomaly, you know, you're talking about rounding, you know, two and three cents, not tens of thousands of dollars shifting accounts or moving in a ledger. DR. This is like... so fast forward 40-plus years, and we're essentially talking about the same thing with a more sophisticated approach and a much better name.

[00:20:12] DR. Yeah, in this case, it's... the danger is that these manipulated data streams tend to be targeting the humans rather than the underlying systems. I mean, you're... if you're bringing in data to process in some sort of big data system, you probably have some at least syntactic checks on the data coming in to check that it's in the right format, that there's no...

[00:20:41] obviously missing gaps in the data stream or whatever. DR. Validation's harder now, though, too, right? Because we have data flowing in so fast, we're like, hey, give me the short long of it, I don't have time... too long didn't read, the TLDR, and then realizing that if we don't look closely over time, that data may be becoming less and less accurate just because of how it's being brought in. DR. Yeah, yeah, you've got plenty of issues at that level as well.

[00:21:11] DR. So, yeah, it becomes a real challenge in the same way that using AI tools can help your software development by doing a consistent check across your code base.

[00:21:32] DR. That same notion of checking your whole organizational space of how do you check to see where data is coming in and where it's being... where... DR. Where it's glowing. DR. Where it's going. DR. I said glowing, I meant going. DR. Not using it as, but... DR. Yeah. DR. But how do you detect bad data coming in and going out? DR. Sure.

[00:21:56] DR. That's a bigger challenge because most of it is not entirely documented and digitized, which is the main production chain. DR. You bring up an interesting point that I think from a maturity standpoint, most organizations, if they were to really, you know, get it down to brass tacks, it's like, DR. Do you understand your inventory, your data inventory, your asset inventory?

[00:22:24] DR. And then what is your strategy around ensuring that where that data goes and who has access to that data and how that data comes back? DR. You know, we're surprisingly at a space or place in time where there is sophistication of tools to enable us to be successful or at least monitor for anomalies that would be getting into that. DR. Is it potentially being poisoned or not? That we didn't really have five, ten years ago.

[00:22:52] DR. Yes, granted today we have AI doing a lot of things good, bad or otherwise, but I mean tools even without AI to build them or AI to manipulate them, we still are at a much more sophisticated place than we ever were 10, even two years ago. DR. The tools are available to do a much better job. It's how many companies are actually doing that better job. DR. Exactly.

[00:23:17] DR. It's, for many organizations, whether it's through mergers and acquisitions or whatever, you end up with a heterogeneous mess of assets that you don't have the money to rationalize. DR. And so you're, or at least you're rationalizing some of them over time.

[00:23:42] DR. And so that leaves you in this state where it's very hard to understand the whole scope, the scope of the whole problem. You've got lots of pinpoint solutions all over the place. You may have firewalls on all your ins and outs, but that doesn't solve your data governance problem. DR. Even at the basic level that we only want one format for this thing. DR. Yeah.

[00:24:10] DR. You know, it's funny. Governance is probably that component, right? DR. That most organizations tend to avoid focusing on, right? DR. Like they, does leadership, does the leadership team have commitment to making this a reality? DR. Yes, okay, great. That's a great starting point. But does every employee understand the why behind we're doing it a certain way so that they understand that it's in their best interest to also look out for, to also raise awareness for?

[00:24:39] DR. Like it's, you know, we, I joked about it, you know, during COVID we saw this really transpire where we shifted the workforce into a residential model that definitely did not have the sophisticated tools and resources that we could arguably say existed in the four walls of the brick and mortar office.

[00:24:57] DR. But we also exposed that you all now have to participate in the outcome. It's no longer something that we can easily say, oh, well, the IT department or, you know, Dr. Wright's going to take care of that because he told me he would. DR. And we were already conditioned to allow things like MFA and other types of security tools be in place from a consumer standpoint. DR. We didn't tell the bank, we're not going to bank with them anymore when they turned on us requirements for MFA.

[00:25:26] But we still whined and complained when we had to do the same thing at work because we again wanted it to be somebody else's problem. DR. And I think COVID, for as bad as it was, I think it started to expose some of those things that said, hey, we have the human element, the human element has to participate, whether they want to or not, they're part of the final outcome, good, bad or otherwise. DR. Yeah, I think it brings up some interesting examples that forced a lot of remote work.

[00:25:57] DR. Yeah. DR. And that I think also highlighted that security procedures aren't always IT functions. So one example was during COVID, there were companies who were classified into essential businesses and non-essential businesses. DR. Sure.

[00:26:21] DR. And so real estate transactions were one of the things that were supposed to still be able to happen. And that was a bit of a problem to do remote real estate transactions. Because in many states you actually require physical wet ink signatures on various documents as part of this, which you couldn't do remotely. DR. And so it forced some evolution in that direction for folks to be able to make those kind of procedures.

[00:26:52] But those signatures are a, and the validation of those signatures is a sort of out-of-band security procedure to validate those documents, right? DR. Yeah. DR. Yeah, actually. DR. Same for like wills and things like documents where you need to have the test data render, and a witness or maybe two depending on the state you're in and so forth. DR. So you bring up a funny, this is totally along those lines that just made me think of it.

[00:27:22] We still are dealing with the antiquated tool we call the fax machine. Like even today, I don't know how many times it's like, well, in order for us to protect the privacy of the patient, you have to fax it to us. DR. It's like, right, but it's going into a digital machine that you have set up to automatically print out in a room that is full of people that have no business looking at this. Yet our fallback is, but it's analog. And that's really what I hear when they say that.

[00:27:52] DR. So that's a great idea of the technology being used by its very nature is tied to an analog model for privacy. So that when we look at it through the security rule, it's like, oh yeah, well, it's not digital. And it's like, well, okay, but my fax became digital. Now what? DR. And it's mind boggling. I still find it mind boggling that we still to this day, and again, I think to some degree, one could say the fax machine is somewhat bulletproof.

[00:28:21] Because if you have a fax machine from 1960, and I have a fax machine that was built yesterday, they still perform the exact same physical function. Both of them would do the same job probably just as well. One might have more memory in case it doesn't go through, but I mean, the idea behind what it's going to do is the same.

[00:28:40] Versus if you run, you know, Google Mail and I run Outlook, we have very different tools trying to do the same sort of communication protocols than we would if we said we had a physical fax machine that both have physical buttons to dial a physical phone number to send a piece of paper across a modem line that hasn't probably really been built in how many decades?

[00:29:04] DR. Yeah, and most of them actually these days put it in, put it over IP and send it to. DR. Right, it's getting converted before it ever leaves my office to digital. I actually have an app on my phone called fax. And the part that's scary is for a week subscription, it's like $10 for a week worth of usage. DR. And then I think about it and I'm like, remember back in the day when you paid per page to send a fax?

[00:29:30] DR. Like we've come so far, I can send as many pages as I want and it's only $10 for seven days. DR. Yep. DR. All right, we've kind of gone a few different directions. We got a couple minutes left. DR. I will make sure that we put in the show notes the books that you have out there. I think they all have a place on your bookshelf.

[00:29:52] I call it the, and maybe you do this too as an attorney. I am no attorney by any means, but I have books that sit on my bookshelf and people have given me a hard time about them. DR. One of them is, it's an AI governance policy book. It's ridiculously long. It's way too small font. I have used it eight times in the last year.

[00:30:12] I can't find when I need something and I go to the index in that book, I can find the document that helps me solve for something that I can't yet get through the digital world of AI or Google or some of those things. DR. So I started to assemble this sort of bookshelf. It's got about a dozen books on it now that I would call it if I was an attorney, those are my legal reference books that until they're available to my AI, I'm going to have to still pull the book off the shelf and turn those pages.

[00:30:44] DR. Well, you're, you're, yeah, so the legal profession is struggling with AI as many other professions are. DR. Sure. And so it's, there, there, there are specialized IT services that lawyers use for case research and so forth. DR. Sure.

[00:31:08] DR. And they have various evolutions of those tools that are available to the field that incorporate more and more AI. DR. Yeah. DR. So they are using it, just not your off the shelf. DR. Sure. DR. Right. DR. It's coming along. DR. You're right, right, right. DR. So last question, I asked this to everybody. DR. I don't know about you.

[00:31:34] I don't read stuff that I write, but is there a book that you're reading today that you think would be beneficial to our audience? DR. It can be, it can be fiction, nonfiction. DR. It does not have to do anything with the field that we talked about today, just a book that you found of value that you think our readers, our listeners would enjoy. DR. Um, book that you enjoy. DR. Oh, let me go back to The Lord of the Rings. DR. Nice. DR. Nice.

[00:32:02] DR. It's something that starts off with a children's double fairy tale, ends up with darker adult themes, and the, uh… DR. There's, um… DR. Lots of interesting material along the way. DR. Um… DR. The movies are not bad, and even if you, um, see the airline advertisements for Air New Zealand, they are quite hilarious. DR. That's… DR. I have seen those. DR. Uh… DR. It's funny you say that. DR. So I… DR. I read the books over and over again growing up.

[00:32:31] DR. In fact, my dad read them to me when I was little. DR. I now have as a… DR. As a Christmas tradition with my kids, we marathon starting around Thanksgiving. DR. Usually it's the day after Thanksgiving, and we start, and we work our way through the Hobbit movies, and then we work our way through the Lord of the Rings. DR. In my early 20s, before we had kids, they were released as the collector's editions that were like the uncut. DR. So instead of it being just three hours per movie, it was like five and six hours.

[00:33:01] DR. We only did that one time. DR. That was… DR. That was too many… DR. That was just too much… DR. Too much. DR. But now my kids have seen those uncut versions, and every year they're like, DR. Dad, can we watch the uncut edition this year? DR. I'm like… DR. You can watch that on your own at your house. DR. This is what we're watching at our house. DR. But yes, thank you for bringing up The Lord of the Rings. DR. Those are cult classics. DR. Anything fantasy literature was built because of what, you know, Tolkien did.

[00:33:31] DR. Well, Dr. Wright, I appreciate you being on the show. DR. For those of you listening, this has been an episode of MSP 1337. DR. Thanks and have a great week. DR. Thank you. DR. Thank you.