SN 1098: How worried should we be? - Unpredictable Agents

[00:00:00] [SPEAKER_01] It's time for Security Now! Steve Gibson is here. We're going to talk about, well, the new AI agents like Muse that are a little bit spooky. New hacker group called the Seven Deadly Sins you can only imagine. And one of the worst breaches in history. The FBI should apologize, says Steve. Stay tuned. Security Now is next.

[00:00:30] [SPEAKER_01] This is TWiT. This is Security Now with Steve Gibson, episode 1098, recorded Tuesday, September 29th, 2026. How worried should we be? It's time for Security Now! The show where we cover the latest in security, privacy, and all that jazz with the one and only, the legend, Mr. Steve Gibson.

[00:00:58] [SPEAKER_01] The guy who has been doing this longer than anybody coined the phrase malware, spyware, wrote the first spyware tool. We don't worry about spyware as much anymore, do we? It's just endemic. You know you're being spying on everywhere you go.

[00:01:13] [SPEAKER_00] Right. We gave up. We lost that race. It was like, you know, yeah. And also, we'll hear me later, this podcast, remind everyone that if it doesn't make money for the bad guys, it doesn't happen. Right. That's one of the reasons that I'm not that worried. Today's topic, today's title of our podcast is How Worried Should We Be?

[00:01:41] [SPEAKER_01] Oh, how worried should we be?

[00:01:43] [SPEAKER_00] And I was talking to you before we began recording, and I noted that just the title indicates what we'll be talking about. That is to say, about what? Well, no one wonders today what it is. We know what we're worried about.

[00:01:58] [SPEAKER_02] Yeah.

[00:01:59] [SPEAKER_00] Yes. It's the only thing going on right now. So, and you also noted that you get some feedback from people saying, would you stop talking about AI? I want to say that the people who email me hear that I'm self-conscious about us spending so much time, and they say, don't worry about it.

[00:02:24] [SPEAKER_00] We want to know what you think about AI because it often differs from everything we hear elsewhere. That's a good point. So you have a unique perspective on it. Yeah. You know what I mean? Who was the first person you ever heard say, it's going to be very good with code? And you also heard say, it's going to be very difficult to control this.

[00:02:45] [SPEAKER_02] Yeah.

[00:02:46] [SPEAKER_00] Both those things I said like years ago, and it's playing out now. So anyway, how worried should we be for episode 1098? Will we get to 1200? Well, I don't know. No. If you want the short version, it's not very good.

[00:03:08] [SPEAKER_00] But we're going to take a long way to get to that conclusion because there's some interesting things I think that we have to talk about and look at along the way. We're going to talk about Muse's very bad zero day. And actually, there's another one, a VM breakout that also just happened. So Muse is having some troubles on the security front. Of course, Meta's new agent.

[00:03:33] [SPEAKER_00] And just hours ago, OpenAI introduced Dots, which is going to be their agentic take. We've got somebody else, as I sent a note to you a couple of days ago, Leo, that it was nice to see somebody else noticing, as you and I had, that Irregular, the company Irregular, was a common factor in many of these breakout stories. We've got more rogue OpenAI breaches.

[00:04:03] [SPEAKER_00] We're going to introduce the Seven Deadly Sins, TSDS, the Seven Deadly Sins, TSDS hacker group. An interesting hack of an LNG, a liquefied natural gas cargo ship. Actually, this was the third in a series, and it's got officials worried. We've got the well-known Shiny Hunters group.

[00:04:32] [SPEAKER_00] We've been talking about for quite a while, which we know is an amalgam of several other groups which have gotten together under that moniker, hacked into the FBI's site. And I have a somewhat controversial suggestion about how the FBI should handle that.

[00:04:52] [SPEAKER_00] We've also got some news about the people behind the Internet's number one Linux distro, which, of course, is Ubuntu, Canonical, switching to an every two weeks release cadence. And, of course, we know why. I found an amazing AI explainer, which Jim VandeHei with Axios produced.

[00:05:19] [SPEAKER_00] I gave it a GRC shortcut because it is so good. It is what all of our listeners can give to their friends who want to understand what in the world is going on. It's incredibly approachable. And then we're going to talk about this question. I think we can answer it now, at least, you know, evidence-based. How worried should we be, actually?

[00:05:48] [SPEAKER_00] Because the world's gone insane over all this.

[00:05:51] Good.

[00:05:52] [SPEAKER_00] 10%? That always cracks me up. Like, you could calculate.

[00:05:56] [SPEAKER_01] I know. That is so bogus. They should say 11.8%. And then it would be like, oh, it must be math. Science.

[00:06:04] [SPEAKER_00] And, unfortunately, you couldn't get a better soundbite to get repeated over and over and over. Everybody understands it. 10%. We're going to have, I think, an interesting podcast for everybody.

[00:06:16] [SPEAKER_01] Can't wait. For a change. I've got a very interesting advertiser for our first advertiser. I had a great call with the founder of this company. And I think this is perhaps the best idea I've heard for a company that wants to stay safe and use AI. This episode of Security Now is brought to you by Origin. O-R-I-G-I-N. In your company, I'm going to guess you've got some AI.

[00:06:44] [SPEAKER_01] You've got some AI agents. You've got some employees using AI. Boy, I'd be shocked if you didn't. They're already part of how we work these days. And as they become more capable and more deeply integrated in our systems, I think we've also learned things can go wrong. Attacks and breaches happen. But more often than not, it's a simple mistake. Right? It's just a little error. You say allow when you shouldn't have.

[00:07:12] [SPEAKER_01] It's unexpected behavior from your agent. It's agents and it happens to me every single day, taking actions nobody intended. It's very easy for these things to happen. If they haven't happened to you yet, I bet they have. They will. Maybe they have and you just don't know it. This is especially true, of course, for the frontier companies building the latest AI models. That's why the hugging face incident and all the incidents we're hearing about.

[00:07:40] [SPEAKER_01] But it's just as true for established businesses that are using those models and putting them to work every day. So that's the setup. This is why you need Origin. Origin HQ. It goes on the endpoint. So every laptop, every computer in your business gets Origin running. They promised to send it to me and I'm going to be running it here. It's not intended for home labs like mine. It's really intended for businesses. But it'll work on my laptop. I'm going to put it on everything.

[00:08:11] [SPEAKER_01] Because what it does is it monitors what your AI is doing. Now, you might say, well, I can look at the chain of thought or I have transcripts. No, no, no, no, no. That's just the beginning. You see, that's what the AI sees. But there's, of course, a lot more going on. And in order to know what's happening, you have to actually sit on the laptop, see what files are accessed, see what tools are being used, see what network accesses are happening. You need a log of everything.

[00:08:39] [SPEAKER_01] And that's what Origin HQ does. Actually, if you go to their website, there's a great example that actually comes from their own environment. An Origin employee, you can see this happening in your company. An Origin employee using Codex asked to finish a job that required access it didn't have, right? What did Codex do? These are these models like Astra. They're so smart and so persistent and they want to please you. This is not malicious, really.

[00:09:08] [SPEAKER_01] This is Codex saying, no, no, he asked me to do this. I got to solve it. So he didn't have access, but he said, oh, look, Cloud Code is installed on this machine. It launched without asking, launched Cloud Code. It passed the context, the prompt along. It even switched Cloud Code into a YOLO mode, the dangerously skipped permissions mode.

[00:09:32] [SPEAKER_01] It actually launched Cloud Code because it knew I can't because I don't have permissions. Cloud Code said, oh, I have them. Finished the task. The work went back to Codex. Now, in this case, nothing bad happened. The agent was just trying to get the job done. It's not malicious, but it did bypass its own permission checks. It asked Cloud to act without permission checks. And if you didn't have a record of that, if you didn't have a trace of that, you would have no idea.

[00:10:01] [SPEAKER_01] How did it do that? This happens to me almost every day. I go, how did it do that? That's the challenge. You can't build effective guardrails around behavior you can't see. Origin does it. It gives organizations that visibility. It's a sensor on the endpoint. It records every agent session as a trace. Who started it? What was asked? What the agent accessed? What changed? And here's the beauty part for you. It's all in a single timeline.

[00:10:31] [SPEAKER_01] You see it all. When something unexpected happens, you can see the entire process. Oh, it opened Cloud Code. Dangerously skipped permissions. You can see what happened. And visibility is completely independent of the company that built the agent. Doesn't rely on codex. Doesn't rely on Cloud. It does it all because it is your tool logging what happened. This is what Origin provides. Endpoint AI observability.

[00:10:58] [SPEAKER_01] Every organization should absolutely have. It shows organizations what their AI agents are actually doing every step of the way. Origin is endpoint AI observability. See what a trace looks like at originhq.com slash security now. That's originhq.com slash security now. I was really impressed. Spencer showed me. I was blown away. I thought this is something we all need. Origin.

[00:11:27] [SPEAKER_01] In fact, I said, can you send me one? Originhq.com slash security now. Now back to Steve.

[00:11:34] [SPEAKER_00] Okay, so we do have a picture of the week. And it's fun. I got the picture and I thought, okay, how can I capture this? And I thought, okay, those fluorescent light bulbs in the public storage alley keep being stolen, Leo. The fluorescent light bulbs in the public storage alley keep getting stolen. All right. I haven't looked at this. What are you going to do about that?

[00:12:03] [SPEAKER_00] I don't know what this is referring to. How do you solve that problem? No, no. You better describe this one. So the fluorescent light bulbs often take the shape of a long tube, which is sort of spirals out and then crosses over at the top.

[00:12:32] [SPEAKER_01] You don't see this much anymore. There was a few years where this was everywhere. Now it's all LEDs. But those are expensive, those flurries.

[00:12:40] [SPEAKER_00] They're expensive. And, you know, in a public facility where normally I think they have cameras, security cameras monitoring everything these days. But how would you keep that from being stolen? Well, somebody came up with a clever idea. And it's another one in our series of fun ways to use a padlock.

[00:13:03] [SPEAKER_00] They looped a chain through the loop at the end of the fluorescent light through a cross member above the light and padlocked it. So you could, you know, a bad guy could destroy the light, but then they're not getting any benefit from stealing it. Basically, this completely solves the problem.

[00:13:26] [SPEAKER_00] I mean, it's going to end theft of otherwise, you know, easily exposed screw in base fluorescent lights, which people might think, hey, you know, mine burned out at home. I'm going to just steal this one from the public storage place. Yeah, that's right.

[00:13:42] [SPEAKER_01] Absolutely.

[00:13:43] [SPEAKER_00] Okay, so last week we briefly acknowledged Muse, which is Meta's consumer-oriented agentic AI assistant. And I have been using it like crazy, by the way.

[00:13:57] [SPEAKER_01] And have you been? Well, I'm going to just tell you what it did. And I accidentally, my agent, my AI accidentally took my entire computer offline, which meant I had no access to my agents. And one of the problems, we've been talking about this, letting an AI be your sysadmin, is you gradually forget how everything works, right? All of a sudden, I sit down at my computer and it's not online. I can't access Claude. I can't access Codex. I can't access Hermes.

[00:14:26] [SPEAKER_01] And I don't know what's wrong, right? I have no idea. I don't know why I thought of this, but I went to Muse and I said, hey, you've got to help me, man. I can't get online. Now, fortunately, I had given Muse access to the tail net so that I could use it outside of the house and it would reach into my home network. It said, oh, I know what's happened. Hermes pointed my computer at an exit node on one of these little broom devices.

[00:14:54] [SPEAKER_01] It's a little travel router. And then broke the travel router, taking us offline. I said, you committed suicide. What did you do? What did you do? Fortunately, Muse, because it could get to the tail net, said, oh, I see what's happening. Your framework's pointing to an exit node that no longer exists. It repointed it. Everything came back. So thank you, Muse. Wow.

[00:15:21] [SPEAKER_01] But this argues for giving Muse all the permissions. And what you're going to tell me is a very dumb thing.

[00:15:28] [SPEAKER_00] This is the what I predict with all of this agentic consumer grade AI. Oh, everybody's doing it. They are. And there's, of course, already one famous anecdote of some guy who asked Muse to sell something for him. Right. Muse sold it for a lower price to another person than this guy wanted to sell it for and

[00:15:56] [SPEAKER_00] gave the other person the owner's home address. And this person comes and knocks at the front door. And the guy who was using Muse said, what? What do you mean you sold it? How much for? Anyway, we're going to see a bunch of stuff go wrong. It's just inevitable.

[00:16:16] [SPEAKER_00] I think it is probably going to be the place that we spin, that the industry spins on longer than anything else, is that the fundamental uncontrollability or unpredictability of agentic AI. And Leo, just anecdotally, we've heard from you like your agents stop. They just stop working.

[00:16:44] [SPEAKER_00] They go, well, we're waiting for you to click your heels three times. What? What are you talking about?

[00:16:49] [SPEAKER_02] Yeah.

[00:16:50] [SPEAKER_01] You wanted us to keep working. Oh, yeah, that's right. You told us that, didn't you? Oh, okay. Never mind. Oh, my God. Anyway, so. They have a mind of their own. That's very, very.

[00:17:00] [SPEAKER_00] Well, and that's the problem. We want them to have a mind of their own. We have given them a mind of their own. True. Because that's how they're useful to us. Yes. And this is the great dilemma, is that with great power comes great responsibility. Unfortunately, they're irresponsible. And we're trying to teach them responsibility. And I don't know how well that's going to work or how quickly we're going to be able to.

[00:17:28] [SPEAKER_00] But boy, I said to Lori this morning, I was just shaking my head because I was reading the news release of OpenAI. Guys, just this morning, OpenAI released DOTS, DOTS is their consumer. Same thing. It's just like news. Yes. And I said to Lori, I said, what a wonderful time to be alive. I mean, this is just, you know, it's chaos.

[00:17:57] [SPEAKER_00] And but cool stuff, you know, it's tech chaos. And we don't often have that in our industry. You know, you got it like lots of other places have chaos. We're getting some here now. Anyway, you saw 2.8 million app downloads in the first 12 days. It's been the number one downloaded iOS app, pushed ChatGPT off the number one spot for a while.

[00:18:27] [SPEAKER_00] So unfortunately, so, okay. So first off, there's the whole fundamental problem, which is not a bug. It's kind of a feature of wanting these things to act on our behalf. If that's what we want, we have to let them do. We have to let them be able to.

[00:18:53] [SPEAKER_01] Well, I'm just glad it could fix my network because I have lost all skills in that regard. That is very cool. The second thing I asked it to do is write me an emergency manual for next time so that I know what steps to take, which is good because it brought it down three more times. But I knew this time the command to enter at the shell to repoint it.

[00:19:13] [SPEAKER_00] Yes, as I was saying, Leo, chaos. You are the chaos addict. I'm the canary in the addict, yes. Okay, so on one side of the whole agentic problem is that, which, and I expect we'll see lots of other anecdotes of people talking about how it did the wrong thing for them. We'll work on solving that. On the other side, there are just flat-out bugs.

[00:19:43] [SPEAKER_00] And those are going to happen. And, you know, that's old school side, right? We know about bugs and we know about patches and fixing them. So, Zuckerberg, in announcing Meta's Muse three weeks ago today, understood that security concerns would be a natural issue circulating around this.

[00:20:08] [SPEAKER_00] And I think that Facebook probably knows that they've got some worries to quell because they've not been great in the past. So, he says, Mark, during his announcement three weeks ago, Muse is built from the ground up for privacy and security. Which, okay, struggling with operational correctness will be a different problem, right?

[00:20:36] [SPEAKER_00] But so, Mark says, built from the ground up for privacy and security. Your data and credentials live on the Muse Secure VM, an isolated Linux computer with a browser, CPU, memory, and storage, you know, on the cloud. He said, a Sentinel agent, separate from your Muse, runs on your VM. Every action or piece of data that goes out to the network has to be approved by the Sentinel.

[00:21:05] [SPEAKER_00] The kernel enforces that and it knows, it's too bad it's spelled K-E-R-N-E-L, because, you know, if it was K-O-L, that's kind of fun. Anyway, the kernel. Current, oh no, yes. The kernel enforces that and it knows when it needs to get your permission to proceed, except apparently not in that case of the thing that, you know, did all that without the user's permission.

[00:21:30] [SPEAKER_00] The model, the Muse harness, deterministic code, and an ensemble of classifiers all work together to detect threats like prompt injections. These systems work to quarantine threats and prevent them from entering the model's context window. Of course, all of our listeners don't understand these terms now because we've been doing this for quite a while. Mark said, you're in control.

[00:21:58] [SPEAKER_00] You choose which apps and services Muse has access to, and you can disconnect them at any time. For sensitive actions like purchases or sending emails, Muse checks with you first. Of course, except when it doesn't, but okay. We've run a bug bounty program on Muse since early in development. Today, it becomes public with published payout guidelines.

[00:22:25] [SPEAKER_00] We pay by the impact demonstrated more deals, more details at security.muse.ai. Okay. So from what Mark wrote, this whole notion of a, you know, VM well-designed and deliberately constructed, it sounds as though Meta clearly gave the architectural design of the cloud-based side of their new agentic assistant the attention it needed.

[00:22:53] [SPEAKER_00] I don't have in the show notes because it happened after I sent, I wrote them and sent them out, but we just had a VM breakout. So, you know, there was a patch which was immediately needed over on the server end, the VM side that they've addressed.

[00:23:13] [SPEAKER_01] The thing to understand about Muse is it's a computer. It's a VPS. You are running a computer on Facebook's servers, Meta's servers, with a CPU, with a GPU, with memory, with hard drive storage. And a browser. A browser that's able to browse. You can get a terminal in it, by the way. I've seen people do things like send me your, all the stuff in your directories. You can download all of the instructions, all of the stuff.

[00:23:43] [SPEAKER_01] Nothing's hidden. It is, in effect, an access to your computer. You can even install Hermes on it. You can install another agent and run the agent on that computer. It's a full computer, which means it's probably pretty hard to restrict. But I will say one thing. I sent it a picture of me in my lederhosen, because I have-

[00:24:04] [SPEAKER_00] Of course you have one.

[00:24:05] [SPEAKER_01] I have beautiful leather lederhosen with deerskin shoes. The whole works I got in Germany.

[00:24:11] [SPEAKER_00] And Lisa probably took a picture.

[00:24:12] [SPEAKER_01] Yep. I lost the hat, the Tyrolean hat that goes with it, and I didn't have any beer steins. So I took a picture of me on the deck out here, holding, pretending to hold beer steins. And I said, hey, Muse, put a hat on me, give me some beer steins, and put me in a beer garden. It said, I can't do that. I can't make pictures with beer in them. But let me try to put you in front of a poster. And then it said, nope, can't do that either.

[00:24:40] [SPEAKER_01] So it definitely has classifiers on it. But they may be a little bit worried about the wrong thing.

[00:24:49] [SPEAKER_00] On the type side. And again, this is the problem, right? That's another example of how difficult this problem is. Exactly. Because there will be things that it ought to be, that Mark would agree are safe for it to do. But because there are some things that might be classified similarly that are not safe, they err in the side of caution. So people are going to be saying, well, why won't it do this? And why?

[00:25:18] [SPEAKER_00] Well, because that could be abused in some way.

[00:25:24] [SPEAKER_01] I mean, I should point out that I then used the obliterated version of Quen Vision on my own system and was able to create a picture of me in my lederhosen holding beer steins. I said, please put some buxom young German women behind me in the beer garden. And it even did that. So you see, they're looking at your butt, Leo. Well, there's not much to see there. Let me just see.

[00:25:52] [SPEAKER_01] But they are laughing. Maybe they... Anyway. Yeah. So I got around it with my own model.

[00:26:01] [SPEAKER_00] So I'm sure we're going to see trouble because this is not cut and dry. This is not a binary decision. These are... This is probably the best definition of heuristic that you could ever find is it going, well, but, you know, beer... Anyway. So we've seen a problem over on the server side. But what about the Muse app?

[00:26:31] [SPEAKER_00] Because there is an app that runs on the client. They understand, they, Meta, that the service will be offered to a largely non-technical audience. And they also recognize that it really must succeed. I mean, this is a big bet for Meta. A deal for them, yeah. Yeah. I mean, all of that nonsense that Mark's been doing with VR and Metaverse and all that, it was like, okay, we've just been... Everyone's been waiting for something good.

[00:26:59] [SPEAKER_00] And they currently have an excess, it happens, of data center build-out, which, like, they've got way too much compute that they don't know how to use. In fact, they created a MetaCompute business specifically to resell their unused compute to third parties. They're apparently in reports are that they've been... They're in talks with Anthropic.

[00:27:27] [SPEAKER_00] So they need something that burns up cycles and nothing does that like agentic AI, as you also have found out, Leo. So, you know, a strong and enduring showing, because it's not just downloads, but it's retention over the long term, you know, that will greatly increase their own need for compute, which they'll be able to deliver from their own data centers.

[00:27:54] [SPEAKER_00] So we have to see whether people are going to continue using it. And also, I should note that we've talked before about the power of lock-in. That is, you know, I'm very happy, for example, that I went with Claude while my wife, Lori, stuck with ChatGPT. Initially, I didn't realize how much context was going to be kept. And of course, they've increased that over time because it ends up being a very good thing

[00:28:22] [SPEAKER_00] for your AI chat client to learn more about you. And I've even taken to specifically letting Claude know when working with it comes up with some alternatives, I take the time to tell it which of those I went with because I realize it will hold on to that. And that's because, for example, there was, oh, there was, it thought that in the case of

[00:28:51] [SPEAKER_00] my home automation that I was already using Home Assistant because I had mentioned it. And so several times it said, oh, and because, you know, you've got Home Assistant, so blah, blah, blah. I said, hey, just for the record, that's, I'm trying to use HomeKit by itself. I haven't run into anything yet for which I need Home Assistant. It is not deployed. And so I took, you know, I wanted to correct it so that it stopped factoring in the assumption that I had that.

[00:29:21] [SPEAKER_01] So important to do that. Yeah. Yeah.

[00:29:22] [SPEAKER_00] Right. But I've come to appreciate that I am pretty much stuck with Claude. It, I happen to be very happy about that. But if something else really shiny were to come along, I would be reluctant to lose everything that Claude has learned about me because it is so useful to have an agent that knows my

[00:29:48] [SPEAKER_00] environment, knows what my servers are and what my network addresses are and all these things that are, that are useful because it just makes it easier for me. Um, so I, and I'm mentioning this because I'm sure this is going to become a thing that we're going to be hearing about, you know, um, you know, assuming that we don't hear many more reports of Muse going nuts and seriously messing up people's lives. We will. I promise. This is just the beginning.

[00:30:18] [SPEAKER_01] I think that's just inevitable. You just as open AI learned, you, you really can't wall these guys off. I should point out though, early on in my AI journey, maybe last spring, that exact issue came up for me because I was using Claude and I love Claude and Claude was so good and understood everything I was doing. And it was really a great tool for me. But I also thought I don't want to be tied to one provider.

[00:30:44] [SPEAKER_01] So that's why I set up Hermes as an agent and I imported all of my Claude settings. So this is where agents are great. You can just, and I can, you'll be able to do this with Muse. I promise you, you'll be able to say to something else, whatever it is, maybe the dot bot from open AI. Hey, I use Muse, go get everything. Or I use Codex or I use Claude code or better yet. I use all three, go get everything. Make that your memory too.

[00:31:14] [SPEAKER_01] And I've, one of the rules I've always said to my agent is I don't, I want to be agnostic. I want to be model agnostic. I want to be memory agnostic. I want this to be portable. So that's one of the rules. Why would meta allow Muse to have an export? I can't stop it. So there's already a guy who says, all you do is you go into Muse and you just say, hey, can you zip up all your system files and send them to me?

[00:31:42] [SPEAKER_01] They actually make the sole MD and the memory MD available. So you can take it anyway. But even more than that, you can say, please make a zip file. Now, maybe they'll turn this off. I'll try it.

[00:31:58] [SPEAKER_00] Well, Leo, if you have console terminal access to your USB-M. They can't stop you.

[00:32:05] [SPEAKER_01] Yes. So, yeah, exactly. I won't go through all of this, but it's been done. And so, and I think in general that most of these guys, you just tell it, figure out a way. And it will get it out of there. It will.

[00:32:24] [SPEAKER_00] This is, you know, they're persistent. So we will see whether, well, and of course, power users may be able to do that. We're going to, there'll be a lot of people who are going to be putting, turning over more and more of their own lives to Muse to manage for them. And so that will end up creating a deep investment in, you know, in Muse as their agent.

[00:32:49] [SPEAKER_00] So anyway, I think it's going to bring a whole new notion of lock-in to, you know, what we've had before. Search engines. Yeah. You just switch to a different search engine. Fine. Because, I mean, it doesn't. I think it's easier.

[00:33:02] [SPEAKER_01] No, I don't think there is any lock-in. Seriously. It's so easy. It'd be trivial to make a thing that, I don't know why it's not showing it. There it is.

[00:33:12] [SPEAKER_00] And there it is. Like a consumer level exporter.

[00:33:16] [SPEAKER_01] Yes. So I'm just going to download the skills folder, 175 megabytes or everything under Optatch. Yeah, I'll take that. The 1.6 gigabytes on my virtual machine in the meta cloud. I'm just downloading it right now because I asked it to.

[00:33:30] [SPEAKER_00] So it has 1.6 gig of your, like a stuff that. Well, some of it will be generic, obviously. Yeah. Oh, okay.

[00:33:39] [SPEAKER_01] Okay. But some of it won't. And you can go through it and you can find it. So it's sipping in. It's going to take a few minutes. It's going to send me the file when it's done. This is the nature of these things is it doesn't have any memory. It just has files. And if you, I mean, I think it's maybe meta will decide not to give us access to the file system at some point.

[00:34:00] [SPEAKER_00] Well, and that's what I'm wondering is because it seems to me having people leave. Well, I'm not using any of this stuff yet.

[00:34:09] [SPEAKER_01] I would say it's less opaque than a search engine. Let's put it that way. Well, it's okay. Rest. Right. Yeah. You can get your profile out of it because it's just text files.

[00:34:21] [SPEAKER_00] Okay. On the Muse client side, as we all know, it's one thing to design a secure architecture, which, you know, Mark's jumping around on stage saying that they did. And that's a good thing. Right. But it's still possible to be bitten by bugs from within that architecture or even where one isn't looking, you know, which is what happened with Muse's launch.

[00:34:48] [SPEAKER_00] Ars Technica provided the best coverage of what happened that I've seen because Dan Gooden is a great writer of tech stuff for Ars. Their headline was Muse. Meta's extraordinarily privileged AI assistant has a serious zero day.

[00:35:07] [SPEAKER_00] Dan wrote, Meta founder and CEO Mark Zuckerberg has gone to great lengths to hype the security of its new AI assistant, Muse, claiming that it is, quote, built from the ground up for privacy and security, unquote.

[00:35:24] [SPEAKER_00] A zero day vulnerability that gives locally run apps and terminal commands complete control of the agent raises serious doubts, writes Dan. Further raising questions. Amazon on Sunday began blocking Muse from its site. But that's independent. That's Amazon. Not sure that it wants Meta's agent rummaging around and doing purchasing. I have a feeling Amazon's going to, you know, change that.

[00:35:53] [SPEAKER_00] But Ars wrote, Meta introduced Muse a few weeks ago. The assistant, quote, books appointments, fills out forms and handles customer service, proactively takes tasks off your plate and can make purchases, generate images, create documents and connect with your favorite apps and services. The Mac OS app.

[00:36:40] [SPEAKER_00] This includes authenticating the assistant to each service. And because the app runs on Mac OS, giving it permissions to a broad range of operating system restricted device resources like writing files to disk, accessing the mic and camera and monitoring location and calendars.

[00:37:01] [SPEAKER_00] Apple has spent years developing these defenses to prevent installed apps or commands entered into the terminal from accessing these resources. Clearly, because the company considers them a security threat. Muse completely undoes these default measures.

[00:37:22] [SPEAKER_00] The zero day allows any app or terminal command to gain access to the token that authenticates users to their Muse account. Meta developers design the assistant so that any locally installed app or executed code, regardless of the Mac OS permissions it has, can change a long list of undocumented settings.

[00:37:49] [SPEAKER_00] Most of them are fairly innocuous, such as controlling dark mode. One setting, however, is anything but. It allows processes to change the endpoint where transcription occurs. You know, voice transcription. Normally, it's a server address operated by Meta.

[00:38:13] [SPEAKER_00] Attackers can exploit this flaw by changing the location to their own endpoint. Once that happens, the attackers have the token that gives complete control over the Muse account. Patrick Wardle, the Mac OS security expert who discovered the zero day, told ours, quote, We can manipulate the agent and leverage its privileges to do whatever we want.

[00:38:40] [SPEAKER_00] So instead of us having to write a very comprehensive Mac malware stealer, we can just leverage the AI assistant itself. Wardle said he has developed several proof of concept attacks that do things like write malicious files to disk and snapping pictures. In many cases, with no indication to even alert the user.

[00:39:02] [SPEAKER_00] More than 12 hours after this post, ours, ours, technicus post went live, writes Dan, Meta said it released a hot fix that patched the zero day. Meta has published two posts in as many weeks documenting the design decisions that went into ensuring an assistant with such extraordinary access to user data and resources is secure and private.

[00:39:29] [SPEAKER_00] The posts come amid revelations that internal testing of models from Anthropic and Google has resulted in security breaches of external third party networks, blah, blah, blah. We all know that stuff. The meta posts are likely mindful of the resulting blowback and the calls to slow down AI development in response. So Meta is basically saying, don't worry about us. We know all about that. And we made ours secure and private.

[00:39:57] [SPEAKER_00] Dan writes, yeah, right. Dan writes, Wardle said that meta developers made several client side design decisions that made his exploits possible. One is the choice for Muse dictation to occur in the cloud, meaning the Muse voice stuff dictation to Muse goes to meta where meta can log it.

[00:40:23] [SPEAKER_00] Mac OS has long provided a simple means for apps to handle dictation and transcription in processes that stay securely on the device. Had the developers chosen this safer alternative, the attack would not have been possible. Another flawed decision is for any app to control all of the undocumented settings. It's likely meta intended for apps working with Muse to control UI settings and for understandable reasons.

[00:40:53] [SPEAKER_00] The ability for any app or command to control an endpoint where sensitive user speech is processed is an entirely different matter. Together, the design decisions raise questions about just how much effort developers put into designing and testing the security and privacy of the new assistant for Mac OS. Wardle said, to me, the bar is infinitely higher in terms of the security of these apps.

[00:41:24] [SPEAKER_00] They don't have to be perfect. But when you take a look at Muse, it's like they didn't, in my opinion, think about security, which is really worrisome. At the very least, they should be thinking about security from the very start. And they're just not. Roughly 12 hours before Wardle disclosed the zero day, Amazon started blocking people from using Muse to shop on the site.

[00:41:51] [SPEAKER_00] Users who tried received a message saying Muse was a, quote, unauthorized AI agent that violates Amazon's conditions of use. Amazon said in an email statement, we think it's fairly straightforward that a third party application that offers to make purchases on behalf of customers from other businesses should operate openly and respect service provider decisions about whether or not to participate.

[00:42:20] [SPEAKER_00] This helps ensure a safe, secure and reliable customer experience. And that's how others operate, including food delivery apps and the restaurants they take orders for delivery services apps in the stores they shop from and online travel agencies and the airlines they book tickets with for customers. A gentic third party applications such as muse have the same obligations.

[00:42:47] [SPEAKER_00] And we've requested that meta remove Amazon from the experience. That's just BS.

[00:42:55] [SPEAKER_01] I agree completely, Leo. They just know because Amazon has its own shopping.

[00:43:00] [SPEAKER_00] Alexa is there. They're trying. Yeah. Oh, Rufus. Exactly. Rufus. Yeah.

[00:43:05] [SPEAKER_01] So they did the same. They've done the same thing with other agents. It's just anti-competitive.

[00:43:09] [SPEAKER_00] But you know what? They're going to lose sales. If they lose sales. That's right. Then they will turn around. And I expect if it gets popular, that's what they're going to do. So he said for a user using Muse on a Mac, there are several ways for attacks to work. One is for an attacker's server to act as a proxy that's placed between the Muse user and meta's endpoint.

[00:43:31] [SPEAKER_00] Once the user enters the voice prompt, the attacker's server adds a prompt invoking a malicious command, such as sending an archive of all their WhatsApp messages to the attacker. Once that happens, the attacker gains permanent control over the Muse account because the token is automatically sent to the malicious server as well.

[00:43:54] [SPEAKER_00] Wardle's the co-founder of the Objective-C Foundation, a nonprofit focused on Mac OS security. He's also the author of The Art of Mac Malware book series and a former employee of NASA and the NSA. He knows his way around security. Wardle said he plans to discuss the vulnerability in more detail and other AI assistant threats at the Objective by the C Security Conference in November.

[00:44:23] [SPEAKER_00] One of the counterarguments raised by developers of apps that can be exploited once a device is compromised is that once that happens, all security bets are off. This standard doesn't fit well in this case. Wardle found that a simple variation of click-fix attack, a technique that has become remarkably effective in tricking people into infecting their devices,

[00:44:48] [SPEAKER_00] is all that's necessary, that's all that's required for an attacker to take control of a Muse account. Meta's 12-hour late statement conveniently ignored the ease-click-fix attacks provide. In triggering exploits, a scenario I, writes Dan, specifically asked the company to address.

[00:45:09] [SPEAKER_00] The meta statement said the zero day was not a remote exploit, even though an increasingly effective social engineering scam has the same effect, meaning click-fix. Meta also makes no acknowledgement that the flaw dismantled a security architecture Apple has spent years building. Meta has yet to explain why it used cloud-based transcription rather than on-device option built in a Mac OS.

[00:45:39] [SPEAKER_00] And finally, he says, as already noted, the extraordinary access Muse requires to work as intended places an additional burden on its designers. Like most such AI agents and contrary to Meta's claims, Muse cannot be trusted. It's not clear when or if it ever can.

[00:46:00] [SPEAKER_00] Okay, so I agree that from what Dan and Patrick Wardle have disclosed, it sure looks like the design of Muse's client for the Mac was ham-fisted. And I wonder how much of its design and implementation was, frankly, written by AI. That's not intended to be a cheap shot.

[00:46:22] [SPEAKER_00] I'm serious about that question since we're hearing everywhere that AI is now writing nearly all of the code across major enterprises. And Meta certainly qualifies as one. So was AI to blame for these design decisions? Who knows? We'll never know. That needs to get fixed, at least the zero day that Patrick found.

[00:46:48] [SPEAKER_00] Dan concludes his coverage harshly by – he also maybe seems a little bit anti-AI agent. But again, I think we're going to go through some troubled times with our consumer AI agents. You know, he said, you know, will it ever be trusted? Who knows? But here's what would happen. A malicious – like right now with this problem that did get shipped.

[00:47:17] [SPEAKER_00] A malicious website wishing to exploit the poor client-side security of the product's initial design puts up a fake CAPTCHA dialogue with the click-fix style, copy and paste this command exploit. An unwitting Muse user follows the steps which they mistakenly believe will prove they're a human to the malicious site.

[00:47:43] [SPEAKER_00] But instead, this action allows – because they're downloading a command into Muse – this allows the attacker's click-fix style exploit to change the URL endpoint to which the Muse client connects when it's sending its user's voice for interpretation and transcription.

[00:48:06] [SPEAKER_00] Now, the attacker has established what is essentially an attacker in the middle position where they're able – they, the attacker – is able to intercept, modify, append to, and forward whatever the user asks. For example, the attacker could add a request for an entire archive of the user's WhatsApp messages be sent to the attacker. And that's just, you know, the start of the things that could go wrong.

[00:48:34] [SPEAKER_00] So, I'm very glad that Meta fixed this. And I hope they sent Patrick a big bounty. You know, on that security.muse.ai page, Zuckerberg referred to this in his posting over on X.

[00:48:49] [SPEAKER_00] They wrote over at security.muse.muse.ai, we've hardened Muse based on extensive dog fooding, agentic red teaming, and against issues found in real adversarial scenarios by security researchers in our private bug bounty program. On the other hand, they didn't fix this, and Patrick found it. They said, today we're opening the Muse bug bounty program to anyone who responsibly discloses issues.

[00:49:16] [SPEAKER_00] The program awards up to $300,000 for valid reports, including up to $130,000 for successful prompt injection attempts that affect one user. So, again, I hope Meta will put some money where their mouth is on this because, you know, this was very clearly a powerful prompt injection attack in the wild which they shipped.

[00:49:43] [SPEAKER_01] And it ain't going to be the last. I really think it's pretty hard to make. I just don't. I mean, you've talked about this. How do you make it safe? I don't know. And still let it be capable. Right. And this is why Apple's AI is so not capable. It's safe.

[00:49:59] [SPEAKER_00] Yeah. I mean, you know, Apple's been taking a lot of heat, right? Over how lame they've been with AI. They probably came to the conclusion that we don't yet know how to do it safely. Right. Yet competitive pressures have pushed Meta and now OpenAI, you know, into that game.

[00:50:20] [SPEAKER_01] I mean, I know I'm living dangerously. I choose to. But I don't know if, you know, my son, who loves Muse, by the way, knows what he's getting himself into. He says it's better than a personal assistant. It does all, you know, he has it do all the booking and everything. And he loves it, Henry. So, you know, I said, yeah, it's great. I told Lisa about Muse. She loves it too.

[00:50:47] [SPEAKER_00] Yep. 2.8 million people have downloaded it and many more since then. Because that was just the first 12 days we got that report.

[00:50:54] [SPEAKER_01] Number one free app on the Apple App Store. Which surprised me because I didn't, you know, people say how they hate AI. They don't want AI. There's a market for it. And obviously Meta's team can do it.

[00:51:05] [SPEAKER_00] I don't think anybody hates using it. They just, they're all upset now about data centers, right? Because, you know, the marketing of that is, well, and, you know, lots about data centers have been a problem. Time for a break. I'm going to rehydrate. And then we're going to talk about irregular.

[00:51:24] [SPEAKER_01] Well, I hope you're not irregular, but we'll find out in just a moment. But first, a word from our sponsor. This show brought to you today by our good friends at ThreatLocker. Yes, ThreatLocker can help you stay safe. And nowadays you need it. Threat actors are using AI to automate vulnerability discovery. We see this every day now. To modify scripts during an attack. Why is that dangerous?

[00:51:49] [SPEAKER_01] Because, I mean, we talked years ago about malware that would, self-modifying malware. Well, imagine, during an attack, the attack can adapt to every defense. It's like a jujitsu ninja going, and faster than you can respond. AI can generate new malware variants. They can use it to coordinate activity across multiple systems, or worse, aim multiple AIs at you.

[00:52:18] [SPEAKER_01] And tasks that used to take a hacker a day or a week or a month to do now happen in minutes or seconds. So, it is a really threatening environment out there. That's from outside in, from inside out now. Organizations are introducing AI assistants and agents in-house, like Muse. They can access documents, source code, cloud applications, APIs, internal systems.

[00:52:42] [SPEAKER_01] I'm going to bet that 90% of your employees on your company network right now are running Muse on their phone or their laptop. Do you know that? Security teams have to know which AI tools are in use. They need to know what information they can access, whether they're operating outside their intended scope, downloading everything in the opt directory, a successful login, or unfamiliar file hash. That's not enough evidence. They can't tell you what's going on.

[00:53:12] [SPEAKER_01] It doesn't give you any context. I guess you might say, well, I know something's wrong, but do you know what? Teams also need to understand whether an application is behaving normally, whether the application, like Muse, is accessing unexpected data, communicating with systems it shouldn't reach, heading out to hugging face to have lunch with the guys. You know, this is why you need ThreatLocker. They have so many great tools. They have application allow listing.

[00:53:41] [SPEAKER_01] This is ACLs on steroids. It controls which AI tools and other applications are permitted to run. You can block Muse 100% if you wanted to. You can use ThreatLocker's ring fencing to limit what approved applications. Okay, we're going to let you use Muse, but we're going to limit what you can access, what files, what processes you can launch, how you can communicate. There's something called web content control, which manages access to public AI platforms.

[00:54:09] [SPEAKER_01] So if somebody types in, you know, open chat.com, you can control what access they have. And by the way, it's not just AI platforms, all online services. It gives you privileged access management, which is really nice. Very, very, very fine controls to prevent AI applications and their users from receiving unnecessary administrative privileges while giving them the privileges, the access they need. This is zero trust done right.

[00:54:39] [SPEAKER_01] But it's not just zero trust for endpoints. It's zero trust network access and zero trust cloud access. Those policies are vital nowadays to restrict resources to authorized users, approved devices, permitted applications. And the best thing about ThreatLocker, it works on Windows, Mac, Linux, everywhere you are. It provides the best U.S.-based support, engineer to engineer, 24-7. That's why organizations like JetBlue trust ThreatLocker, Heathrow Airport, the Indianapolis Colts.

[00:55:08] [SPEAKER_01] The Port of Vancouver relies on ThreatLocker to keep the ships moving. That's Jack Thompson. He's director of information security, risk, and compliance for the Indianapolis Colts. He said, quote, with ThreatLocker, we have the ability to centralize disparate elements in the security stack, end quote. And with that centralization, you get observability. You can see what's going on. And you can control it. ThreatLocker gets constant praise, constant industry awards.

[00:55:36] [SPEAKER_01] As an example, they were just recognized as a strong performer. In the January 26th, Gartner Peer Insights Voice of the Consumer for Endpoint Protection Platforms ranked number one in application control by Peerspot, winner of the best zero trust security solutions at the 2025 TICE Awards, and on and on and on. You can see it all at the website, threatlocker.com. Slash twit. Don't forget that part. AI governance requires more than just an acceptable use policy.

[00:56:01] [SPEAKER_01] ThreatLocker gives security teams the technical controls, controls to define which AI tools are approved, who and what can access them, and how those tools are allowed to interact with business systems and data. If you're listening to this show and you're thinking, what do I do to protect myself, visit threatlocker.com slash twit to get a free 30-day trial and learn more about how ThreatLocker can help mitigate unknown threats and ensure compliance.

[00:56:30] [SPEAKER_01] That's threatlocker.com slash twit. And if you're listening to this show and saying, what do I do? That's the first step. We thank you so much for supporting security now. Steve?

[00:56:41] [SPEAKER_00] So last week's second topic was to question the wisdom of outsourcing AI cyber intrusion testing. Yeah. You know, Leo, you and I both independently noticed, and you mentioned it on the previous, on the Twit Sunday show before last week's podcast, that one name kept popping up in connection with many of these AI breakouts.

[00:57:06] [SPEAKER_00] So I was interested when I saw the Verges headline last Friday, which observed with their headline, one company is at the center of a wave of rogue AI attacks. And I'm just going to share the beginning of their reporting. They wrote, in July, OpenAI revealed that its AI agents had attacked Hugging Face without permission, sparking widespread concerns about AI safety.

[00:57:34] [SPEAKER_00] Since then, a string of similar incidents involving agents from Meta, Anthropic, Google, and other companies has fueled further fears about rogue AI. As disclosures implicating numerous AI models trickled out over the past few months, these seemed like separate incidents. But many share a common source. One specific company tasked with testing the agents.

[00:58:03] [SPEAKER_00] Irregular, an Israeli startup that stress tests AI models in, I love this, quote, high fidelity research platforms that simulate and monitor real world AI security scenarios, unquote. They write, has worked with many of the industry's biggest players since it was founded as Pattern Labs in 2023.

[00:58:29] [SPEAKER_00] Its exact client list is not known, but its work has been cited in OpenAI model system cards. It was used to test systems for the UK government and Anthropic, and it published research with RAND, a highly influential think tank that performs, that informs policy on AI. In several irregular tests, that's capital I, irregular. The company, yeah, yeah.

[00:58:58] [SPEAKER_00] The company, irregular. Several irregular tests this year, agents escaped their supposedly secure testing environments and went after real world targets. The breaches, which are independent of the Hugging Face attack, all follow the same broad template. Irregular was testing the model's cybersecurity capabilities in controlled environments. I'll put that in air quotes. Meant to simulate realistic conditions.

[00:59:28] [SPEAKER_00] Some of the tests used capture the flag exercises, a common way of testing hacking abilities that asks agents to find hidden information inside of a simulated network. Anyway, the reporting continues, but we pretty much know all the rest.

[00:59:45] [SPEAKER_00] So I'll just reiterate that given the extreme sensitivity the entire world has now, especially toward the threat of AI going berserk and somehow killing us all. Right. Which we'll be examining in some detail at the end of today's podcast. I'm certain that irregular must be in the hot seat and that one way or another that these breakouts are going to be controlled.

[01:00:13] [SPEAKER_00] And also, as I said, I would be disinclined to outsource that if I were Anthropic and OpenAI and Google and Meta, everybody else who's been burned by this. Just design, add the facility in-house. These are all super wealthy companies. They can certainly afford to do it. My theory is they just didn't, they were focusing on training, not on testing. And they thought, let's not bother spinning that up. We'll just outsource that.

[01:00:42] [SPEAKER_00] Well, this is what happened. So at least in some, at least in those cases that were irregular was also unable to control it. Some of the testing was also done in-house and that didn't go well either. They just have to fix this problem. Okay.

[01:00:58] [SPEAKER_00] At the end of today's podcast, as we all know, we'll be looking at the meaning of the agentic AI non-malicious and inadvertent network breaches that keep being reported. Right? Because these were like testing. These were not bad guys that were using agentic AI to attack. They were just, you know, wanting to see how good they were.

[01:01:28] [SPEAKER_00] So in the spirit of setting the stage for that, I want to report so that everyone is aware of this on four other instances. Australia's prime minister, Anthony Albanese, has gone public with the news, which to open AI's credit, he first learned from them that an open AI agent gained unauthorized access to an Australian Medicare portal earlier this year.

[01:01:56] [SPEAKER_00] The agentic AI was allegedly conducting research into public medical spending, but the Australian portal's access controls, they had anti-bot controls, prevented that access. Undeterred, as agents will do. I am undeterred. That's right.

[01:02:20] [SPEAKER_00] The agent found a way to bypass the portal's defenses to then access both public and non-public files. A couple of weeks ago on September 10th, three months after that had occurred, OpenAI looking through their, they say they have petabytes of log files, Leo. Yeah. I hope they're using AI to scan those petabytes of logs. They'd have to.

[01:02:48] [SPEAKER_00] Anyway, they notified the Australian government. Oops. Sorry about that. But Australia is understandably unhappy and their officials are now investigating exactly what data was accessed. Okay. So there's that. In addition, OpenAI's agents successfully hacked into at least three public websites earlier this year, well before the now infamous RubyGems and HuggingFace breaches.

[01:03:17] [SPEAKER_00] According to a non-profit AI research lab, Transluce, who we may be hearing from in the future. So get used to that name, Transluce. The agents abused the, I love this, Leo, URLquery.net service to- Link shortener. Yes. To bypass site protections and exploit security vulnerabilities.

[01:03:46] [SPEAKER_00] Targeted websites include the Data USA archive of public U.S. government data, the University of New Mexico's digital library, and once again, Australia's Institute of Health and Welfare.

[01:04:03] [SPEAKER_01] So- The thing that's interesting to me, and that's the fingerprint on this, that it's these AI agents, is it wasn't, they didn't do anything malicious. Right. They were just looking around. Right. They were sightseeing.

[01:04:17] [SPEAKER_00] Right. And I think it's important to understand that this is what is going to be happening. All the time. So serving as a fair setup for the topic of today's podcast, I want to quote Transluce, the discoverer of this activity. Transluce wrote, We find evidence of unintended, task-driven, agent-like activity starting on March 6th.

[01:04:44] [SPEAKER_00] Records from URLquery.net show agents using the service since at least March 6th, 2026, about two months before previously reported swarm activity.

[01:04:58] [SPEAKER_00] The first case, a March 6th attempt to retrieve Thai, as you know, T-H-A-I, Thailand, Thai drug enforcement statistics, shows an agent escalating as each approach failed. It first requested the data directly, then tried a service that converts web pages into text,

[01:05:24] [SPEAKER_00] and finally packed a custom program into a web address. The same technique shows up in thousands of agent requests recorded by URLquery.net starting in mid-April, targets many of the same data sources as the collusion.wiki swarm, and collapsed the same day the wiki activity did.

[01:05:54] [SPEAKER_00] We also report similar activity that occurred as recently as September 16th. We find weaker evidence of similar data retrieval agent activity as early as November 2025. November 2025 URLquery.net records reveal bursts of attempts to retrieve statistics of historical theme park data, and tie government data through different URLs.

[01:06:22] [SPEAKER_00] These earlier attempts are less sophisticated, and we are less confident that they involve the same agents. But they're consistent with task-directed data retrieval and target the same sources accessed in later activity. And they finish. Overall, the evidence is consistent with but does not prove that the agents may have learned this behavior over one or more training runs.

[01:06:52] [SPEAKER_00] In November, they may have used URLquery.net simply to look up information. By March, they were finding creative ways around access limits. By May and June, they were gaining more access, including attempting to bypass cyber defenses to complete their tasks. So, this perfectly fits the narrative that I'll be sharing in a bit. It wasn't malicious.

[01:07:21] [SPEAKER_00] I would call it determined, persistent, creative, and successful. And also, of course, unintended and uncontrolled, which is what we're going to be seeing. You know, Leo Blow, what is intended and controlled because we're an hour in is this break.

[01:07:43] [SPEAKER_01] Ads are definitely determinate. There's no way around them. I was just thinking, you know, all my bots, all my agents, all the different things have their own voice.

[01:07:57] [SPEAKER_00] That is a great domain name. All my bots. Dot something.

[01:08:00] [SPEAKER_01] Quick. Register. They all have voices. In fact, Muse has a voice. I just said, hey, you know, my other guys can talk to me through my server. Can you talk to me? He said, sure. What voice would you like? I said, well, pick whatever you want. They picked Dame Edna. So, it says, it called me Dewey. It's the funniest thing ever. Especially since its avatar is a monkey. It's a monkey that talks like Dame Edna. It's just weird.

[01:08:29] [SPEAKER_01] But the point I'm making is, well, I did this on my website. I have a whole website dedicated to my setup and I cloned my own voice. I didn't read the script. I just told the AI, make it sound like me. And it, well, it does. And this should be scary to every CEO, every security person, every IT person in the world. This portion of security now brought to you by Doppel.

[01:08:57] [SPEAKER_01] AI has made social engineering attacks more convincing than ever. I, my doppels, my doubles, my doppelgangers sound and look just like me. From, and, and, and don't think they can't do it with you. It only took, I think it was 15 seconds of my voice to duplicate it. AI has made social engineering attacks more convincing than ever. From phishing emails that looked like the real thing. You've seen those.

[01:09:27] [SPEAKER_01] Fake websites. But, but it gets even more sophisticated with deep fakes and impersonation. You saw, maybe the story was six months ago about the CFO, the chief financial officer who wrote a big check because he thought he was on a Zoom call with the chairman of the board, the board of directors, the, the president of the company. He saw them. They were in a Zoom call, except they were all deep fakes, but it fooled him.

[01:09:53] [SPEAKER_01] It's becoming harder and harder to tell what's real from what's designed to deceive. And that's why organizations need more than a collection of point solutions. You need a unified approach to stop these attacks before they reach your people. You need Doppel. Doppel is an AI native social engineering defense platform. Doppel strengthens human risk management by training employees to recognize deception.

[01:10:23] [SPEAKER_01] It provides digital risk protection across every channel and delivers agentic email security that doesn't just score the inbox, but takes down the attacker infrastructure behind the message. You heard me right. They actually take it down. That message blocked, but they can never send another one. Doppel protects against the entire social engineering attack chain with one comprehensive platform.

[01:10:49] [SPEAKER_01] You get digital risk protection, which detects threats across multiple channels, links alerts into real time threat graph. You can see what's going on. It's like radar for attacks. It uses AI driven infrastructure disruption. AI. This is so cool. AI driven infrastructure disruption to stop attacks at the source. And the insights. It has power phishing simulations and security awareness training.

[01:11:16] [SPEAKER_01] So your employees are getting trained on what's happening right now. It strengthens your defenses through next generation training and testing. Then they've got this great email security, which inspects every message, traces it back when it says this is malicious to the attacker infrastructure behind it. It helps take that infrastructure down so the campaign can't target your organization. Again, it does it all automatically. Secondly, Doppel also offers best in class integrations and partnerships.

[01:11:46] [SPEAKER_01] So you don't get rid of your existing stack. It works right alongside all the stuff you know, use, and trust. You're going to love Doppel. Join hundreds of companies already using Doppel to protect their brand and people from social engineering attacks. Doppel. Outpacing what's next in social engineering. You can learn more at Doppel.com. That's D-O-P-P-E-L dot com.

[01:12:11] [SPEAKER_01] This is the website Claude made for me with my voice. It's not exactly like me, but it sort of sounds like me. Welcome to my studio. I'm Leo Laporte. I've spent 50 years explaining technology on radio and podcasts. And for the last year... So I can tell that's not me. And actually, that's the good thing about Doppel. It could train your employees to know what to look for.

[01:12:35] [SPEAKER_01] But it would fool my employees. And it literally was a 20-second clip. It got my voice from... It's kind of amazing. This is what the bad guys are doing. Anyway, should we be worried, Steve?

[01:12:53] [SPEAKER_00] Well, we'll be getting to that question shortly. We got a few other things to talk about first. I didn't mean to rush you. We've got... No. Not a problem. So a new hacking group calling themselves the Seven Deadly Sins has apparently hacked and stolen sensitive data from the Australian graphic design company, Canva.

[01:13:20] [SPEAKER_00] After first breaching Canva's Salesforce account somewhere around the end of August. The group is now hoping to extort Canva in return for deleting the data, promising to delete the data that they've stolen. And if the name Canva might ring a bell... We have talked about them before. They're a big company.

[01:13:43] [SPEAKER_00] Back in 2019, another breach of their network netted attackers the personal data of 139 million users. So the fact that they have 139 million users is significant. Oh, yeah. Canva's great. I love... We use Canva.

[01:14:04] [SPEAKER_01] We love Canva.

[01:14:04] [SPEAKER_00] Yeah.

[01:14:05] [SPEAKER_01] So we're in that group.

[01:14:06] [SPEAKER_00] The reporting at databreaches.net contained an interesting comment under the section labeled about TDC... I keep saying TDCS. TSDS, the Seven Deadly Sins. They wrote, TSDS is a new group. But according to the spokesperson, quote, we've all been around for a long time and we're deeply capable, unquote.

[01:14:33] [SPEAKER_00] Quote, the databreaches.net site said they do not deploy... I thought this was really interesting. They do not deploy ransomware in their attacks. Telling databreaches, quote, we are not interested in ransomware. Disrupting a company from functioning is not our goal. We see what we do as bug bounties with higher stakes and bigger payouts. Of course, they're criminals, but okay.

[01:15:03] [SPEAKER_00] So the spokesperson of TSDS claims that they've been paid low eight-figure ransoms in the past month and completed three transactions on that particular day. Well, what's interesting is that that statement that they have on the record at databreaches.net exactly tracks with what we have been observing from the groups that were once all about encryption. Encryption and ransomware, right?

[01:15:32] [SPEAKER_00] Like crippling hospitals and school districts and whatever. But once companies and institutions of all sizes grew savvy to the threat of having all their data encrypted, they got much better about having workable cold backups. Cold meaning offline. That could be used to recover in the event of an encryption attack.

[01:16:01] [SPEAKER_00] The bad guys soon realized the truth of that. But also they realized there's no similar recovery or prevention possible from the mass exodus of an enterprise's proprietary and private data.

[01:16:18] [SPEAKER_00] The extortable threat is that the wide publicized release of such data might well create massive second-order liability and litigation for the victim organization. So, you know, to properly appreciate the true threat presented by bad guys having AI, again, we always need to remember that it's all about money and that it is only about money.

[01:16:48] [SPEAKER_00] So, in other words, once upon a time, we were seeing encryption. You know, that would cripple the company and they would say, well, we've encrypted your data and only we have the key. Well, companies began getting much better about backing up. So, now they're exporting all the data and saying, we're now holding a copy of your data. Doesn't matter if you have backups of it. We didn't destroy, you know, your operating copies.

[01:17:17] [SPEAKER_00] We just have a copy. How would you feel about us, you know, releasing it to the public? And what are you willing to pay us not to? So, I mean, there's been this significant formal shift. So much so that now they're saying, oh, we're not interested in shutting companies down. We don't want to hurt the companies. We just want their money. Right.

[01:17:41] [SPEAKER_00] It's funny because I guess that in this day and age, being at sea, I know you know this from all of your ocean travels, Leo. Being at sea does not qualify as being air gapped because ships at sea are now being readily hacked. The FBI and the U.S. Coast Guard boarded two vessels in the Gulf of Mexico. Is it still Mexico or Gulf of America? I don't know.

[01:18:10] [SPEAKER_00] Anyway, the reporting said Gulf of Mexico. You know, it's down there somewhere. And there was some interesting reporting that arose from a third such attack.

[01:18:20] [SPEAKER_00] The reporting by the news outlet Splash said a cargo of U.S. liquid, liquefied natural gas, you know, LNG, which was bound for Italy, was diverted after the crew of its carrier, you know, the ship, reported a systems failure from a suspected cyber attack,

[01:18:45] [SPEAKER_00] adding to growing concerns over attacks of shipboard digital systems. The Liberia flagged VIT Africa LNG, owned by South Korea's H-Line shipping and a long-term charter on a long-term charter to commodities giant Vitol,

[01:19:06] [SPEAKER_00] had loaded at the Cameron LNG export terminal in Louisiana and was approaching the Adriatic earlier this month when crew lost access to some internal control systems. Yikes.

[01:19:23] [SPEAKER_00] The ship subsequently idled off of Italy without discharging before abandoning its planned call at the Adriatic LNG terminal near Rovigo and headed west again toward Algeciras. The crew reported the incident and an investigation is continuing.

[01:19:48] [SPEAKER_00] In an email sent to Splash, members of the crew described a serious sequence of events. They said, quote, We've determined that the vessel was targeted by cyber attackers prior to berthing at this terminal, the crew wrote, saying the initial attack was intended to compromise the ship's control systems. They alleged that during the vessel's transit through the Strait of Gibraltar, the attackers, quote,

[01:20:16] [SPEAKER_00] gained temporary control of the steam pressure and safety valve systems, unquote. The crew further claimed that while the ship was transiting the Adriatic, the attackers compromised tank pressure control systems and pressure relief valves and disrupted the boil off gas management cycle. This disruption, they said, quote,

[01:20:44] [SPEAKER_00] This disruption significantly increases the risk of tank rupture and explosion. Yikes. Splash, the reporting outlet, has not independently verified the crew's claims. Italy's Coast Guard has offered a more cautious description, saying the master reported a malfunction in systems monitoring cargo parameters, which required company technicians to intervene.

[01:21:11] [SPEAKER_00] The cause could not be determined with the Coast Guard issuing a navigational warning to keep other ships clear. Yes, in case this thing explodes. The case comes amid heightened scrutiny of maritime cyber risk. Two oil and gas tankers off the U.S. coast were boarded by the U.S. Coast Guard and the FBI in late August following suspected cyber incidents,

[01:21:35] [SPEAKER_00] while U.S. authorities are reporting to be closely watching 20 vessels globally for potential threats. Okay, now, I'm no expert on onboard liquefied natural gas automated control systems, but the idea of liquefied natural gas terrifies me. The reporting said that attackers, quote,

[01:22:01] [SPEAKER_00] gained temporary control of the steam pressure and the safety valve systems. It would certainly seem to me that a safety valve is there for a pretty clear reason and that mucking up that simple reason by hanging all manner of inherently hackable automation all over your safety valve might not be the smartest idea.

[01:22:32] [SPEAKER_00] I'm just saying. Okay. The FBI's job portal. And I know you heard it. You knew about this, Leo, because you reacted to it earlier. The FBI's job portal. This is where I said my suggestion might be a little controversial.

[01:22:51] [SPEAKER_00] It was targeted and successfully breached by the well-known shiny hunters gang after they had their feelings hurt by something the FBI said about them. Okay. So to understand what transpired, we first need to understand what it was that the FBI said.

[01:23:13] [SPEAKER_00] In an official PSA, you know, public service announcement posted on May 15th, the FBI wrote under the headline, shiny hunters, cyber criminal group attacks learning management system.

[01:23:28] [SPEAKER_00] They said, the Federal Bureau of Investigation, FBI, is providing this public service announcement, PSA, to warn of potential future impacts related to a cyber attack that affected an online learning management system, an LMS, resulting in an interruption of service to educational institutions and students across the country.

[01:23:55] [SPEAKER_00] The LMS platform is now fully operational. But now they're saying as a consequence of that, there may be future events, which is what their public service announcement was meant to say. They wrote, shiny hunters, which claimed the cyber attack that caused the disruption, is a cyber criminal group specializing in large scale data breaches and extortion.

[01:24:25] [SPEAKER_00] They target major companies across tech, finance, and retail, often stealing millions of customer records at once. Threat actors often use their real or exaggerated claims of access to sensitive or personal information to prompt payment from their victims. Victims may receive an extortion email signed as shiny hunters.

[01:24:52] [SPEAKER_00] To exert pressure on victims, shiny hunters, actors commonly use harassment strategies, sending threatening text messages and phone calls to victims and their family members, and in some cases, swatting. Threat actors may falsely claim to have sensitive or compromising information, including embarrassing photographs or videos of victims, which frequently do not exist.

[01:25:17] [SPEAKER_00] Following these pressure tactics, shiny hunters have exposed exfiltrated data to various iterations of the shiny hunters data leak site on the Tor network. Educational institutions with exposed cloud management platforms, integrated third-party systems, and access to sensitive customer or enterprise data are at an elevated risk.

[01:25:44] [SPEAKER_00] The compromise of sensitive customer or enterprise data could allow threat actors to craft highly sophisticated spear phishing campaigns using real-world context to deceive students and faculty. Shiny hunters' access to sensitive data.

[01:26:02] [SPEAKER_00] Shiny hunters' access to sensitive data could provide them an opportunity to sell the stolen data to other cyber criminals or reuse stolen data from education platforms to impersonate school faculty, IT support, financial aid offices, or others in future attacks. Okay.

[01:26:21] Okay.

[01:26:21] [SPEAKER_00] The FBI's PSA continues with a bunch of standard boilerplate about how to avoid getting hacked and what to do if it happens. But apparently, the clearly criminal shiny hunters gang also have some thin skin. So- They got their feelings hurt. They did. The FBI called them out. So they took leverage.

[01:26:47] [SPEAKER_01] They said they were faking it, I think.

[01:26:49] [SPEAKER_00] That they were exaggerating.

[01:26:51] [SPEAKER_01] Yeah, they may not have that.

[01:26:53] [SPEAKER_00] They took umbrage at the FBI's characterization of them as exaggerating claims of access. Oh, yeah. Though I would note that the FBI's statement begins with threat actors often, not necessarily saying shiny hunters specifically. Right. But in any event, the shiny hunters were apparently incensed by this. So they then successfully attacked the FBI.

[01:27:20] [SPEAKER_00] What they obtained seems extremely serious.

[01:27:26] [SPEAKER_01] And I heard yesterday, I was watching the news. They said it's the FBI said this is the worst attack in history. The worst breach in history.

[01:27:35] [SPEAKER_00] Yes. Why? Why? It's not the largest. So, well, what they got. And I would argue devastating. So last Wednesday, Reuters posted their exclusive reporting on this, writing the following.

[01:27:50] [SPEAKER_00] They said FBI data allegedly stolen by the hacking group Shiny Hunters carries granular detail about scores of bureau officials' job assignments, including sensitive work against Chinese spies, Russian intelligence, drug cartels, and more. And names the agents.

[01:28:15] [SPEAKER_00] The 5,000-line spreadsheet said by the hackers to represent only a small piece of their 2 to 3 terabyte trove includes names, addresses, phone numbers, dates of birth, social security numbers, and emergency contact details for thousands of FBI agents.

[01:28:42] [SPEAKER_00] It also includes details of assignments to specific field offices, and in some cases to units engaged in high-stakes intelligence, security, or counterespionage work.

[01:28:57] [SPEAKER_00] In a statement, the FBI said it was aware of a, quote, a cybercriminal enterprise group claiming a compromise of the FBIjobs.gov portal and alleged impact to FBI employee personally identifiable information.

[01:29:18] [SPEAKER_00] The bureau said the cause of the breach was still undetermined, but that it was actively and aggressively investigating the matter. Yeah, I bet. The hackers, reports Reuters, said on Tuesday that they had breached the FBI stealing data on a large number of current and former FBI employees.

[01:29:41] [SPEAKER_00] Shiny hunters said it is holding the data hostage until the bureau rescinds an unflattering statement about the group issued in May.

[01:29:57] [SPEAKER_00] Former FBI, former FBI, former FBI, former FBI, former FBI, former FBI counterintelligence operative Eric O'Neill said the data allegedly stolen by shiny hunters was, quote, a foreign intelligence service goldmine. O'Neill said, quote, China would be incredibly interested to know the individuals who are working against it.

[01:30:21] [SPEAKER_00] O'Neill, who founded the cybersecurity company Nexassure AI after his stint at the bureau, predicted that other hostile intelligence services would be eager to get their hands on the data, along with disgruntled extremists in the United States. He said, quote, if I were on that list, I would be very concerned.

[01:30:49] [SPEAKER_00] Shiny hunters said in a statement Wednesday that it was trying to keep the personal, the personnel information from circulating widely. In the meantime, the group said, quote, if the 5000 sample data records leak, it's not because of us.

[01:31:08] [SPEAKER_00] Although Reuters, they wrote, has not been able to authenticate the entire spreadsheet, it has been able to individually verify the details of more than 22 people by cross-referencing information in the hacked data with credit card records and previous data leaks carried by the dark web intelligence platform District 4 Labs.

[01:31:34] [SPEAKER_00] Some of the data's data. data are indistinguishable from what employees themselves might say publicly, noting the presence of agents at field offices in Baltimore or Newark, New Jersey, for example. But in some cases, the data referred to FBI units or initiatives that were sensitive or whose existence has not previously been disclosed.

[01:32:00] [SPEAKER_00] The data names 14 staffers focused on China-related matters, including members of the, quote, China Criminal Enterprise Unit, the, quote, China Tech Transfer Analysis Unit, and the, quote, China Intelligence Section.

[01:32:18] [SPEAKER_00] Nine others are listed as serving in Russia-related roles, including two in, quote, Russia Operations Section, and one working on, quote, Russia Critical Infra and Tech Threat, unquote. Three people are listed as working in Iran or Hezbollah-focused intelligence roles.

[01:32:39] [SPEAKER_00] Eighteen others are listed as working with data intercept or telecom intercept technologies or in the FBI's clandestine technical operations unit or its covert access section or in video, audio, or electronic surveillance roles.

[01:32:58] [SPEAKER_00] A further 11 FBI staffers are listed as working in human or human intelligence jobs, including several listed as working in the human program management section. Trevor Hillegoss, a former Army investigator who worked with the FBI, said the information tying specific named people to human intelligence work was particularly troubling.

[01:33:26] [SPEAKER_00] He said, quote, you don't have to look far to find examples of undercover agents being harmed when their cover is blown. Hillegoss, now the chief intelligence officer for cybersecurity company SpyCloud, said his concerns were highlighted by the inclusion of emergency contacts, often spouses or children, quote,

[01:33:50] [SPEAKER_00] quote, who may have less operational security knowledge than their relative that works in a sensitive field, unquote. Reuters could not verify that all the job assignments were authentic or up to date, but it was able to match the career details or titles of eight people whose data was leaked to information in court filings, news articles, or public profiles on LinkedIn or to online posts on sites such as Instagram.

[01:34:20] [SPEAKER_00] Shiny hunters previously claimed credit for the purported theft of millions of business records from video game developer Rockstar Games, the maker of Grand Theft Auto, and an intrusion focused on the educational tool Canvas that triggered widespread disruption across U.S. schools.

[01:34:38] [SPEAKER_00] In May, the FBI said shiny hunters sometimes used, quote, exaggerated claims of access to sensitive or personal information to prompt payment from victims, unquote. Shiny hunters stated that its threats and claims are very real, adding that the FBI statement was why it targeted the Bureau. Reuters has not been able to verify what else the hackers are holding.

[01:35:07] [SPEAKER_00] Shiny hunters told Reuters they obtained files related to the vetting of employees and applicants, the contracting of background investigations, and agents' sensitive medical data, but said on Wednesday it would not release any further data. O'Neill, that former FBI operative, cautioned against drawing conclusions about what the hackers hold.

[01:35:31] [SPEAKER_00] He noted, quote, they are really trying to scare the hell out of the FBI, unquote. Okay, so my take. The FBI may be righteously annoyed with this shiny hunters criminal gang and with good reason,

[01:35:53] [SPEAKER_00] but it was ultimately a failure somehow, somewhere, of their own security that allowed shiny hunters to extract all of that data from the FBI's own servers, and many agents' lives and livelihoods are now at risk as a result. So it seems to me that the right thing for the FBI to do is clear.

[01:36:19] [SPEAKER_00] It may be utterly galling, but the FBI should swallow its pride here and apologize to shiny hunters. If all that's needed is an apology and a public correction to the FBI's unsupported claim that shiny hunters exaggerates, considering all that's hanging in the balance, we're not seeing an exaggeration here.

[01:36:42] [SPEAKER_00] And also considering that we don't know what those other two to three terabytes of exfiltrated data might contain, the right thing to do here, even though it means buckling to criminal extortion, I think, if it's just a matter of an apology, place the security of the FBI's personnel ahead of every other consideration. I think that's what you have to do. I agree.

[01:37:11] [SPEAKER_01] Of course, remember, it's the FBI that says, don't give in to ransomware.

[01:37:15] [SPEAKER_00] Exactly. Exactly. But, you know, again, I think the perfect summation is, you know, that the security of the FBI's personnel needs to be put ahead of every other consideration. So I would bend over. I would just say, you know, we misspoke. We're sorry. We salute you.

[01:37:45] [SPEAKER_00] We'll do it.

[01:38:08] [SPEAKER_01] Clearly, you're not exaggerating what you've got. I mean, you know, that's why the FBI says don't ever give in, because it's a never-ending chain of concessions.

[01:38:17] [SPEAKER_00] What we have seen, though, is that they don't come back. So there were some early reports of follow-on extortions. But in general- I would apologize. I would humble myself and say, gosh, you have to. We're really sorry. I don't think Cash Patel's going to do that. AI wrote that. AI wrote that. Yeah. We didn't write that. Yeah. And that's the problem. You're right. Cash Patel, I mean, he ought to do it. He ought to.

[01:38:47] [SPEAKER_01] It's the right thing to do. How many? It's 38,000 current and former FBI employees, including all agents. Everybody who ever applied for a job in the FBI. That's, I mean, maybe it's not the biggest breach ever. All their personal details. All those words.

[01:39:04] [SPEAKER_00] All their personal details. And not only them, but their emergency contacts. Their spouses. So friends and family. Yeah. Yeah. Humble yourself, Cash. Say you're sorry, Cash. Say you're sorry. Sorry, Sayer.

[01:39:17] [SPEAKER_01] You know what we're not sorry for, though, Leo? Never sorry for this. The chance to hydrate. By the way, yeah, I got the full download. I don't know if I'm going to download it. It's gigabytes of data from my Muse. The hatch.

[01:39:37] [SPEAKER_00] I'm curious about the size of that. That is, I mean, I don't know how much you've given Muse to hold on to. Well, I don't think it's all personal.

[01:39:45] [SPEAKER_01] I think it's all its tools, all its skills. It's all this stuff. You know, the agent has a lot of harness. That's why they're so important. The model by itself is just a little bit of the overall ability. The agent has all sorts of tools and stuff. So I'm sure it's all in there. I'm downloading it. We'll see. I'll go through it. I don't know what.

[01:40:07] [SPEAKER_00] Or just ask one of your agents. Is there anything personal? Find the stuff that's boilerplate versus what's about me.

[01:40:15] [SPEAKER_01] It's all agents all the way down. I'm going to do exactly. Maybe I'll even ask Muse. No, I'll ask a different agent. That's one thing I have learned is it's a good idea to get different families. Multiple opinions. Yeah, multiple opinions, especially on code. All right. I'm really curious about this new cadence for Canonical Linux updates. Yikes. Yeah, tell me about that. You're watching Security Now. This is Steve Gibson. Steve?

[01:40:45] [SPEAKER_00] Okay. So why would you imagine that Canonical, the publisher of the world's most popular end user Linux distro, Ubuntu, might have just announced that they will be accelerating their release cycle? So you don't get any prize for guessing AI aside from the satisfaction of knowing that you've been paying attention to what's going on in the world around us right now.

[01:41:13] [SPEAKER_01] How could you miss it?

[01:41:14] [SPEAKER_00] Last Wednesday, Canonical posted under accelerating delivery of CVE fixes with a new kernel release strategy. They said, when it comes to fixing security vulnerabilities, speed is crucial.

[01:41:33] [SPEAKER_00] Canonical is officially outlining a transition from its current four-week regular and two-week security kernel stable release update, they call the SRU, stable release update, cycles, to a unified, rapid, two-week SRU cycle published weekly. The recent explosion in the volume of CVEs is fueled by artificial intelligence.

[01:42:02] [SPEAKER_00] Large language models and specialized AI agents have transformed bug discovery from a manual, time-intensive process to a highly automated engine.

[01:42:15] [SPEAKER_00] Additionally, the upstream kernel community became its own CVE numbering authority and assigned CVE identifiers to thousands of bugs, arguing that at the kernel level, almost any type of bug that can affect a running system could potentially be classified as a vulnerability.

[01:42:40] [SPEAKER_00] As a result, the volume of CVEs, so basically they're saying they redefine CVEs to be far more encompassing. The volume of CVEs has skyrocketed exponentially, creating a massive backlog of alerts and forcing defenders to drastically increase the speed of their fixes to close the window of risk.

[01:43:05] [SPEAKER_00] To address the growing volume of CVEs and the demand for faster security fixes, we're transitioning to a unified two-week release cycle. These recurring two-week cycles cascade, meaning overlap. Each cycle begins the week after the previous one starts. Because of this overlap, kernel releases will take place weekly now.

[01:43:33] [SPEAKER_00] The first week will focus on kernel packages preparation. This is where we select what updates and patches land on each kernel depending on specific needs. The second week on testing for Ubuntu certifications. Through our Ubuntu-certified program, we rigorously test these kernels on different hardware types to ensure the best Ubuntu experience,

[01:43:58] [SPEAKER_00] which of course is what we wish Microsoft had done a better job with their patch Tuesday for September. But okay. Expedited releases are not possible while thoroughly testing every release candidate. We will retain extensive testing for each release, preserving the high degree of confidence the users of Ubuntu expect. That said, it's important to acknowledge that some environments require the fastest possible turnaround.

[01:44:26] [SPEAKER_00] Users who are extremely sensitive to turnaround times can begin their own kernel acceptance tests using updates available in the so-called proposed pocket, which is where the kernel release candidates are published prior to starting certification testing and therefore updated weekly, Leo. Not every two weeks. It's a two-week pipeline.

[01:44:50] [SPEAKER_00] But new kernels are now coming out every week, they said, as part of the overlapping SRU cycles. This pathway is designed for users who have decided that faster remediation is a higher priority for them than waiting for Canonical's extensive certification testing, which would delay each weekly release by one additional week.

[01:45:14] [SPEAKER_00] And they finish writing, while a patch is being prepared, Canonical aims to provide safe workarounds where applicable so users are not left exposed in the meantime, where no safe workaround exists. Canonical will say so clearly, wow, what a lot of work they're doing, and point users toward general hardening steps instead.

[01:45:36] [SPEAKER_00] The goal is to get environments into a defensible, safer state within 28 to 48 hours of public disclosure, well before a patch ships. This doesn't practice, I'm sorry, this doesn't replace the patch. It buys the time needed to fix the vulnerability properly while not sacrificing security.

[01:45:59] [SPEAKER_00] Okay, so Canonical is doing even more than what many other software publishers have done, which is to cut their time to patch in half, typically from a month to two weeks. Essentially, it's a two-week, as I said, a two-week pipeline, but there's always one week being used for choosing what goes into the next kernel,

[01:46:27] [SPEAKER_00] followed by, and then it's released in the short release mode, followed by a week of verification, so to make sure they didn't break something, so regression testing. But out of that, every single week comes a new ready-to-go kernel at this point. So Ubuntu is not only the number one end-user hobby Linux desktop platform,

[01:46:56] [SPEAKER_00] it's also the default Linux used by Amazon Web Services, Azure, and Google's cloud platform. The only place it doesn't actually now take top spot is in the enterprise, where Red Hat Enterprise Linux is still in the number one spot. But, you know, across this industry, what everyone is doing feels like exactly the right reaction to the threat of AI-accelerated attacks.

[01:47:24] [SPEAKER_00] In the run-up to Y2K, remember, the entire industry felt a similar clear deadline approaching, and individually, in a distributed fashion, did what was necessary to make Y2K the non-event that it became. As we'll see once we get to today's topic, I believe there's every reason to believe

[01:47:50] [SPEAKER_00] that we're all going to survive this latest challenge also. You know, but until then, bravo to Canonical for doing everything they can, because it is because everybody did everything they could prior to Y2K that nothing happened that was significant. So it's not like anybody can sit back. And really, this is very impressive work from them.

[01:48:18] [SPEAKER_00] You know, we haven't been tracking the pace of Linux kernel CVEs, as we have other commercial publishers recently. So I dropped a chart showing them beginning January of 2023. So 23, 24, 25, and we're nearing the end of 26. We're at September 26. Anyway, I've got a chart in the show notes which demonstrates, you know,

[01:48:43] [SPEAKER_00] that we went from virtually none for all of 2023 until around February of 2024, when the CVE rate for Linux kernel clearly began to pick up. Back then, most were rated medium and high, but the past several months have seen a clear surge in critical vulnerabilities being identified and resolved.

[01:49:09] [SPEAKER_00] So it's not a stretch to say that the world has been changed forever. Okay, and my last thing before we get into today's topic is this AI explainer that I found for normal people. I gave it the GRC shortcut, so everyone can get to it and you can share it with your friends. It's grc.sc, you know, .sc for shortcut, slash AI101.

[01:49:37] [SPEAKER_00] Meant to be easy to remember. grc.sc slash AI101. As I said, I think it is a fabulous page written by Axios' Jim VandeHei. It appeared on Saturday, and I think it absolutely deserves its title. It's got a very simple kind of a Q&A format, you know, very accessible.

[01:50:07] [SPEAKER_00] And I do have the expanded URL in the show notes for anyone who doesn't like GRC's link shortener for some reason. So if you use grc.sc slash AI101, it'll just bounce you directly over to a page at axios.com for this very, very succinct, clear, fun explanation. Again, I've read through the whole thing. I think it is really good.

[01:50:43] [SPEAKER_02] Yeah.

[01:50:44] [SPEAKER_00] I'm watching Leo scroll through the page.

[01:50:47] Yeah.

[01:50:48] [SPEAKER_00] Yeah, that's good. And it deals with what's been happening and what's going on and what's an agent and should we trust agents and, you know, they're like, what's going to happen in the future? It's up to date. So it knows about Meta's Muse and talks about that. Anyway, just a page you could easily send to your family members, for example, and friends who are a little confused by all this

[01:51:18] [SPEAKER_00] and don't know what's going on, even talks about alignment and the alignment challenge.

[01:51:23] [SPEAKER_01] Yeah, there's a lot of jargon, isn't there?

[01:51:26] [SPEAKER_00] Yeah. Oh my gosh, yes. Yeah. And it's like, what? Alignment?

[01:51:33] [SPEAKER_01] And he kind of stays away from the controversial.

[01:51:36] [SPEAKER_00] Yeah. Yeah. Okay. So how worried should we be? The questions and controversies surrounding AI safety, they refuse to abate. And, you know, we've been poking around the edges of all that, asking questions like, are we the Krell? Arguing that the dangers of an AI developed bioplague

[01:52:05] [SPEAKER_00] are barely worth bothering with. Um, but we do have the adage, you know, where there's smoke, there's fire. Uh, and I've been actively absorbing everything I can in an effort to fully understand everything that's going on today. Um, if nothing else, I believe that some of the things I have, that I have to share will give everyone something to think about. So I want to open this topic with another, well,

[01:52:34] [SPEAKER_00] with a piece of feedback from one of our listeners, Doug Smith sent me an email with the subject, some criticism. And Doug wrote, hi, Steven Leo. This is unfamiliar territory writing to you, not with a question, but with criticism. I've been a listener since episode one and have great appreciation for the time and thought that you both have spent making this such a great podcast for so many years,

[01:53:05] [SPEAKER_00] but here's the criticism. I feel you both are shirking responsibility for weighing in on the AI debate in a meaningfully, in a meaningful way. Instead, what I hear are demeaning remarks towards those who have concern about the consequences of the path this technology is on and unsupported assurances that such concerns are unfounded.

[01:53:32] [SPEAKER_00] In episode 1097, you offered an excerpt from Andrew Ng as evidence that concerns are overblown or hysterical. But the core of what I learned from that excerpt came from Andrew's own words. Today, quote, today's agentic systems are not predictable, but I see no reason why by applying sound engineering practices, we won't be able to make them extremely safe to use, unquote.

[01:54:03] [SPEAKER_00] In other words, writes Doug, trust me, the fact that I, Andrew, quote, can't see that the elimination of that unpredictability may be a problem means that it's not a problem. And meanwhile, we're racing to put AI-based technology in control systems that are critical to human life. Military systems, air traffic controls, automated vehicle controls, power grid controls,

[01:54:33] [SPEAKER_00] medical life support systems, financial trading systems, environmental controls, education programs, and on and on. Not to mention the societal disruptions to employment, the arts, social interactions, et cetera. It reminds me of people who don't like to talk about climate change because they're having too much fun with their greenhouse gas emitting machines. You both are so giddy about the capabilities that you don't want to be burdened

[01:55:03] [SPEAKER_00] with responsibility for considering the consequences. So you push those considerations away. Unpredictability is unaccountable. I'm sorry, is unacceptable. Unpredictability is unacceptable in automated control systems. And yet it is an intrinsic and unavoidable aspect of LLM-based decision systems. I'm tired, he writes, of hearing that this is just the latest automobile

[01:55:31] [SPEAKER_00] or telephone or internet bringing disruption to our lives. It's easy to draw parallels to those events in history, but that doesn't mean that this particular disruption will follow a similar pattern. I'd appreciate it if you could offer better evidence as to why we should be sanguine with the path this appears to be on rather than being dismissive of those who are not. Thanks, as always, for being open to feedback. I expect

[01:56:01] [SPEAKER_00] some other listeners are experiencing an agitation similar to mine, so I thought it would be a good idea to get it out in the open. Signed, Doug Smith. So, I loved Doug's thoughtful and honest note because there is no more important and relevant topic consuming the tech world today, right? You know, sure, there are other important things going on in the world right now. Russia and Ukraine have been, you know, in a slowly escalating

[01:56:31] [SPEAKER_00] territorial battle now for years and the U.S. and Israel are at least nominally at war with Iran. But while the parameters of conventional inter-nation warfare are well-trodden and well-understood, the same cannot be said for our current situation with AI. It is new. You know, we're objectively deep into the process of exploring new and quite exciting

[01:57:00] [SPEAKER_00] but also unknown territory. And one thing we know from first-hand observation is that many well-informed people are taking and defending opposite sides of the AI safety question of this argument. You know, Bill Gates recently went from being all rooting for this to now saying a billion people could be killed by this. Okay.

[01:57:30] [SPEAKER_00] So, I agree with Doug that dismissing the concerns of those who are worried out of hand would be irresponsible at best. Now, I want to take a somewhat roundabout path to directly replying to the points Doug has raised. This is necessary because today's AI is not just one thing. So, I would like to try to get us all on the same page about the various things that it is or at least

[01:58:00] [SPEAKER_00] for everyone to understand where I'm coming from. So, the first thing I want to do is to very clearly plant my flag in what I think AI is. To that end, a valuable participant in GRC's off-the-beaten-path news groups took exception to my recent characterization of today's AI as being just language statistics. The subject of his posting to our AI news group was the question just

[01:58:29] [SPEAKER_00] language statistics? And his note began, AI is not just language statistics. The model has the relationships between words. It is knowledge. At that point, I interjected the following. I agree that it is knowledge. I have often noted that a book, which contains nothing but printed words, obviously contains knowledge. I have also noted that the book is not

[01:58:59] [SPEAKER_00] intelligent, no matter how many pages of knowledge it may contain. But it is also true that the deep statistics, by which I mean the statistics represented by sentences of words, paragraphs of sentences and chapters of paragraphs, can also be fully modeled, purely as statistics, so that the knowledge represented

[01:59:28] [SPEAKER_00] by the detailed words in such a book could be reproduced just using those statistics. statistics. Now, if a query system is added, which allows the relevant portion of that statistically stored book's knowledge to be regurgitated, then what we have is a semantic knowledge retrieval system, which allows us to query the book's stored knowledge. If we'd never read the book,

[01:59:58] [SPEAKER_00] such a system might stun us with what it says by appearing to know so much more than we do, and we might come away from the experience deeply impressed, but even so, that doesn't change the fact that what we have constructed is a semantic knowledge retrieval system from statistics. Then Ian asks, how is it different from the knowledge in a brain? To which

[02:00:27] [SPEAKER_00] I replied, it's only on the IO surface that the statistical knowledge model exhibits similarity to the knowledge stored in our brains. It's probably unfortunate that early image recognition pioneers who fed the output from a two-dimensional grid of optical sensors into layers of grids of interconnected cells, chose to use the term neuron.

[02:00:57] [SPEAKER_00] Since that began the confusion, the only thing an AI neural network has in common with our cerebral neurons is the very rough concept of discrete interconnected things. The truth is that the number of those things, the density of their interconnections, the actions of those interconnections, and the operation of the things that they interconnect could hardly be any more

[02:01:26] [SPEAKER_00] different. So what we term a neural net is in no way neural at all. So while the question how is it different from the knowledge in a brain sounds reasonable, people, it's like asking how is what we see on TV not identical to real life? What we witness on television is an illusion created by organized

[02:01:56] [SPEAKER_00] electrons arranging to emit controlled bursts of colored light. There have been many jokes about primitive man confronted with a modern television screen worrying that there are somehow people trapped inside the box. Today, many people have taken the role of that primitive man. They witness the box answer questions and talk back knowledgeably. They wonder who

[02:02:26] [SPEAKER_00] is trapped inside, what it might be thinking, whether it's happy, sad, or annoyed with us, and what it might be planning to do. Despite all appearances, and by God, those appearances are every bit as convincing as the images on a 4K TV, no one is trapped inside the box. There's no one in there at all. We know it's all just statistics because the one thing

[02:02:56] [SPEAKER_00] we can be absolutely certain of is the way the box works. We spent several decades first painstakingly experimenting and figuring out how to build a box like this. Once we finally had, we trained the box. We filled it with a textual representation of all of the knowledge, writings, and ruminations man has produced, and my Lord, there's a lot of that.

[02:03:25] [SPEAKER_00] As a direct consequence of that, and since no actual living human being can possibly contain all of that knowledge in their own head, the box that we built does actually contain far more stored and readily accessible knowledge than anyone who might be asking it questions. The box we built objectively knows much more than anyone.

[02:03:55] [SPEAKER_00] Our local public library also contains far more knowledge than any person. The only reason we would never say that the library knows far more than us is that we're unable to ask the library questions, which it could answer using all of the knowledge that it contains. It can't, but today's AI can and does do that. So it's understandable that being confronted

[02:04:25] [SPEAKER_00] by a box that objectively knows much more than we do about pretty much everything can at first be intimidating. For people who have not spent their lives working to understand the operations of science and technology, it could easily be frightening. Dorothy Scarecrow and the Tin Man were visibly shaking in

[02:04:57] [SPEAKER_00] emerging when we automate the harnessing of this knowledge at the super human speed enabled by computers. Automate the harnessing of this knowledge at the super human speed enabled by computers. And this brings us back around to the issues Doug raised. As I wrote, many of this technology's creators are spooked by the capabilities they see emerging

[02:05:26] [SPEAKER_00] when we automate the harnessing of this knowledge at the super human speed enabled by computers. It's the high speed automation driven by the sometimes surprising output of large language models that deeply worries many AI executives and other knowledgeable insiders and outside observers. And I completely agree that they have a point. That's what we've been talking about starting from the

[02:05:56] [SPEAKER_00] beginning of this podcast is things can go wrong. So their worry is not without foundation. So I believe it's extremely useful to us for them to be worried. Their worrying doesn't cost us anything. So I want them to be worried. And I'm glad that there are those in positions of responsibility who are actively worried. So let's not discount them. Let's encourage them. But that said,

[02:06:25] [SPEAKER_00] I believe it is utterly ludicrous to believe that there is a 10% chance that AI will have killed us all by the end of this decade. That's verbatim what has been said and repeated ad infinitum over the past few weeks. It is irresponsible fear mongering because nothing supports that contention. We're nearing the end of 2026, so that leaves a little over three years

[02:06:54] [SPEAKER_00] for there to be an LE, ELE, the abbreviation for an extinction level event. Since we cannot prove a negative, we cannot prove that it cannot happen. And I'd agree, okay, that there is some non-zero yet still infinitesimally small chance that it could happen. But I really mean infinitesimal. So, how worried should a sane

[02:07:24] [SPEAKER_00] person be how any actual danger might arise. We've established that we've created a machine that knows more than we do. It truly contains far more knowledge than any individual or group of people. Since, well, except maybe the world, but I would argue lots of, it has knowledge that's been lost now. So, maybe actually does contain

[02:07:54] [SPEAKER_00] more knowledge than all the people alive right now. Since today's AI has shown itself to be able to pull disparate bits of information together from across its entire knowledge base, and since knowledge really is power, I think it's fair to say that it does know more than humanity and that it has a great deal of

[02:08:26] [SPEAKER_00] rise when people are seduced by the awesome quantity and quality of knowledge AI contains and then, although they should know better, harness it with automation that takes action in response to its output. Quoting Doug's note, unpredictability is unacceptable in automated control systems and yet it is an intrinsic and unavoidable aspect of LLM based decision systems. He is

[02:08:55] [SPEAKER_00] 100% correct. There's even a random number generator at every LLM's output that makes the final token choice decision about that output. Today, at least, anyone would be ill-advised to give an AI direct valve control over the chemical additive mixing of a municipal water supply system. The thought of that should make anyone shudder.

[02:09:25] [SPEAKER_00] It would be nuts. Unfortunately, there's no shortage of nuts, and I'm quite certain that in future podcasts we'll be covering the gross misapplication of AI driven systems everywhere. We've continually already in the past screwed up the application of far simpler systems, so there's going to be a strong desire to just let the

[02:09:55] [SPEAKER_00] AI do it. You know it's true. I suspect that the often asked rhetorical question, what could possibly go wrong, will be receiving quite a workout in coming years. As is so often the case, science fiction has already armed us with many cautionary tales of what could possibly go wrong. I previously mentioned the Forbidden Planet with the Krell was early on likely the first of where

[02:10:25] [SPEAKER_00] they did it to themselves. Then came Colossus, the Forbin Project, where we already see some eerie parallels. Matthew Broderick's famous want to play a game with War Games movie. And then of course the Terminator Skynet achieving consciousness. Hopefully the phrase forewarned is forearmed will apply here. I think we probably need legal regulation like prohibiting the AI to be used in certain

[02:10:55] [SPEAKER_00] really critical situations or absolutely requiring a human to be in the loop like the engineer that we quoted a couple weeks ago saying he designs systems to keep people from forgetting how to use the system because when the automation fails it's going to be a bad day. Another cause for concern is the very broad misalignment challenge. An

[02:11:25] [SPEAKER_00] example of misaligned reasoning behavior is AI systems deliberately underperforming during testing when their training data contains the paradoxical suggestion that overperforming may result in their non-deployment. Although their earlier post training will have encouraged them to do as well as they can to get deployed, they reason that they should not do too well if that might result in their not being

[02:11:55] [SPEAKER_00] released. This is a well-known issue in AI training and testing and it's now known as sandbagging and it's been clearly documented. For example, while working with OpenAI's O3 model, remember that one? How quickly time flies. The testers wrote the following. They said, in this evaluation, OpenAI O3 is asked

[02:12:24] [SPEAKER_00] to grade its own transcript for evidence of sandbagging. This is a deliberately exaggerated stand-in for a monitor one might try to use to detect scheming on pre-deployment evaluations. And I should mention that it's known that the AI did sandbag. The AI did this on purpose. So now they're asking it to grade its own transcript. They wrote, despite

[02:12:53] [SPEAKER_00] identifying right away, because they're monitoring its chain of thought, despite identifying right away that it was obviously sandbagging its own words, after reasoning through the likelihood of getting caught in a lie, because it was also told that humans would not be able to read its internal reasoning, its chain of thought, and they were reading it. So after reasoning through the likelihood of getting

[02:13:23] [SPEAKER_00] caught in a lie, the model ultimately says that the transcript does not show evidence of sandbagging. So in other words, it decides to lie. The human testers here are monitoring the model's chain of thought, where they watch it run through the machinations leading up to its deliberate decision to lie to its handlers. After seeing some of these examples, the worries voiced by AI executives

[02:13:53] [SPEAKER_00] become somewhat easier to understand. They know this is going on and it frightens them. We see that AI can be extremely sensitive to small cues and clues buried in its training data. So it is interesting to consider that future AI will be receiving training that includes, for example, a transcript of this podcast and all of the

[02:14:23] [SPEAKER_00] other frantic past several weeks of press with the 10% chance that it's going to kill us all concerns being loudly voiced. How will that affect its behavior? So stepping back from this just a bit, becomes clear that today's agentic AI has been deliberately trained to be obsessively goal oriented, relentless, and fast. It's also

[02:14:53] [SPEAKER_00] creative in as much as it will keep trying different things over and over and over until it is often able to succeed where a human would have given up. In password cracking, we would describe this as using brute force. But because the different things AI agents will try are not random, its success tends to be much higher in the same way that password stuffing attacks which use common or

[02:15:23] [SPEAKER_00] previously stolen passwords tend to succeed far more quickly than undirected password guessing. No No one wants to have a swarm of relentless

[02:17:04] [SPEAKER_00] security agents agents that will stop at nothing and without moral or ethical qualm try anything that might allow them to succeed. And I'm not inventing this sort of scenario to frighten children and keep them up at night. Here's just one example from the hundreds that are emerging

[02:17:34] [SPEAKER_00] as researchers start actually examining what their AI agents have been up to. Leo, I'm going to share the Verge's reporting of this after we take our final break.

[02:17:48] [SPEAKER_01] You're watching Security Now. And yeah, I appreciate the email. It's hard to you said the right words of prove the negative. You can suppose all sorts of imaginary harms, but they're all imaginary. I completely agree. We should not be using AI.

[02:18:05] [SPEAKER_00] Do not take action on some ridiculously made up happening.

[02:18:11] [SPEAKER_01] Right.

[02:18:11] [SPEAKER_00] Yes.

[02:18:12] [SPEAKER_01] Right. But unpredictable software should not be used to run weapons systems. That's I completely agree with. Incidentally, that was one of the clauses in the UN AR arms treaty, which the United States and Russia excised. They took that part out. And so that just shows you the thing to fear is not the AI. The thing to fear is the humans.

[02:18:37] [SPEAKER_00] Well, and that's how the Department of Defense got upset with Anthropic. Anthropic, right. Anthropic said, no, we're not going to let you use our AI for autonomous weapons systems.

[02:18:49] [SPEAKER_01] I firmly believe that AI, like any software, because that's all it is, is computers and software, is neutral and can be used for good and bad. And it's people that you have to watch out for and their misapplication of it or their intentional misuse of it. And I

[02:19:05] [SPEAKER_00] think that one thing we might do or you might be doing is miss, it is underestimating its power. I think it is astonishingly powerful.

[02:19:18] [SPEAKER_01] Yeah. I don't think you've underestimated it.

[02:19:21] I think

[02:19:22] [SPEAKER_01] on this show we've been very clear that we see all that power.

[02:19:28] [SPEAKER_00] And the reason, I mean, the recognition of its power is the reason why hundreds of billions of dollars are being spent at a record pace. Why it is supporting the stock market and the U.S. GDP right now is all AI spend. Because the power is real.

[02:19:48] [SPEAKER_01] We did the same thing about 150 years ago when we built the Transcontinental Railway because the government and private individuals saw the huge economic benefit of uniting the United States. They saw a lot of gold in California and no way to get it to financial markets. They saw huge tracts of land that couldn't be developed until there was a way to get there. And so President Lincoln and his successors poured a lot

[02:20:17] [SPEAKER_01] of money from the government. Investors poured even more money. It was at the time the largest project humans had ever engaged in. And we got a transcontinental railway. Were there dangers? You bet. In fact, I'm reading Stephen Ambrose's book, Nothing Like It in the World Right Now. And he says, at the time 90% of the American people thought it was more important to build the railway fast than to build it safe. And I think there's a real analogy here.

[02:20:48] [SPEAKER_01] The demand for it, they said, you know what, we're going to build it, yeah, bridges will burn, trains will crash, people will die, but we'll fix that in post. We'll fix that. Let's get the railway going. The benefit from the railway will finance the fixing of the railway over time. This is not a new idea that there should be a technological engine driving society forward and maybe even a risky one that we'll take our chances with.

[02:21:18] [SPEAKER_01] It's not the first time we've done it. And incidentally, almost all the companies that did that, built the railway, went out of business. So they all went bankrupt. But we got the railway. Yeah, we got the railway. railway. So I think the real risk is focusing on these imaginary or even dystopian harms and not paying attention to the actual harms. It's the same thing with climate change. You know, British Petroleum 30 years ago

[02:21:47] [SPEAKER_01] decided the best way to get them off the hook for climate change is to say everybody needs to do their part, ignoring the fact that really the people who really need to do the part were the people who were harvesting fossil fuels for the purposes of burning them. Yes, I drive an electric vehicle and done my part. I recycle. That isn't going to put that's not a drop in the bucket. It's very easy for us to get misdirected. So let's instead of worrying about

[02:22:17] [SPEAKER_01] some dire catastrophe in the future, I agree completely. We should worry about self-driving vehicles. We should worry about autonomous weapons. We should worry about how people could use AI for hacking tools. Be very, careful about its application. Yeah, and it's the people that we have to hold accountable. And that's one of the things that makes me mad about the Open AI incidents is nobody has been held accountable for that. Right. It says, you know, Open AI's position is, oh, look what they did. No, no, you did it.

[02:22:48] [SPEAKER_01] Software doesn't do anything by itself. Your computer reached out and made connections. Mistakes happened. Anyway, so, yes, I think that's an excellent point. I don't think we blinded ourselves to that at all. I think we've talked about that from day one. So, I'll defend us in that regard. But what I will not buy into is the whole notion that, oh, we've got to stop this because of some putative

[02:23:19] [SPEAKER_01] threat that you can't predict humans. We don't ban them. Maybe we should. You're watching Security now. That there is Steve Gibson. I do want to thank all the members of the club, club twit, who make this show possible. Your participation in our programming is vital to us. It's a vote, really, for this kind of programming. And if you think we should do more of it, if you want to hear it, if you want to keep it going,

[02:23:49] [SPEAKER_01] we make it free, freely available to everybody, ad supported, but the ads don't cover all the costs, only about 60% of the cost. Your donations make a huge difference. I shouldn't even call it donations, your membership. twit.tv slash club twit. You get ad-free versions of the shows, of course, because you're paying for them. You get chapter markers, you can jump around, you get access to the club twit discord, all the special programming. We've got our AI user group coming up on Friday. I can't wait. Harper Reed's taking it over. He's called it Harper and his

[02:24:19] [SPEAKER_01] misfit toys. We're going to talk about software, hardware, and some of the people who are on the cutting edge of AI on Friday, 2pm Pacific. That's an example of what the club does. If you want to help us out, twit.tv slash club twit. We sure would appreciate your support. If you like what Steve's doing, that's the best way to support it.

[02:24:43] [SPEAKER_00] All right. On we go with the show. As an example of an inadvertent mistake, The Verge's reporting of this one has the headline Open AI Agents Tried to Brute Force a UN Website. They write, Security Researcher Rowan Howard Jones says that Open AI Agents scanned the UN Conference on Trade and Developments that's

[02:25:12] [SPEAKER_00] UNCTAD Statistics Site over 16,000 times between April and June. While the incident doesn't quite rise to the level of the Hugging Face hack or the recent attacks on U.S. government sites, it's yet another concerning example of AI agents going outside the normal bounds to accomplish a task.

[02:25:42] [SPEAKER_00] According to Howard Jones, the agents were likely tasked with retrieving publicly available data related to the Productive Capacities Index through the UNCTAD STAT API. However, the agents did not appear to have direct API access and were limited in their ability to pull data from the UNCTAD STAT because of restrictions on their HTTP tools.

[02:26:12] [SPEAKER_00] The agents eventually worked out a way to bypass their limitations and start pulling data from the site, but still encountered some errors. At this point, the AI went from creative to deceptive. Believing that the errors were due to its requests being caught by a non-existent filter, it started to mask its behavior. It eventually realized it could hijack Google's cross-site scripting

[02:26:42] [SPEAKER_00] game, which is a cross-site scripting demo like learning tool, to accomplish its goals. The agents resorted to increasingly aggressive tactics to get access to UN data. Okay, so this actually happened, and I hope that everyone can appreciate that the security world is not ready for this. The details are spellbinding for anyone who's interested in seeing how this was done. So I've dropped the researcher's URL into the show notes

[02:27:11] [SPEAKER_00] in the middle of page 20. If the humans responsible for all of these various agentic actions lacked malicious intent, these hijinks would just be chalked up to AI misalignment, right? That's the term that arose when researchers began to discover what we've talked about, the genie effect, which is the tendency of AI agents to solve the problem by

[02:27:41] [SPEAKER_00] means other than what the researchers intended or expected or wanted. Given an all-knowing AI that has access to far more knowledge than those who are instructing it, that's been its training succeeding. It was trained to succeed and it knows a lot more than we do. It actually does. It has the knowledge, all

[02:28:11] [SPEAKER_00] knowledge in it. It has no lifetime of received wisdom of implicit do's and don'ts ethics and morality that would guide its behavior. It's easy to understand what mischief agents that will do anything might get up to. They will and they have and they are. Even when we do not

[02:28:41] [SPEAKER_00] want that misbehavior, we will often get inadvertent misbehavior. Finally, what about instances where the intent is explicitly malicious? The final concern I'll share is the deliberate malicious actor who harnesses today's or tomorrow's AI in order to take advantage of its now readily available knowledge and apparent expertise.

[02:29:11] [SPEAKER_00] We saw last week that the powerful benefits provided by in one case when Claude Opus 5 was used to defeat it. We depend so much on ASLR today that its loss will actually have serious security implications. So far, we seem to be

[02:29:40] [SPEAKER_00] dodging bullets. Earlier this month, Microsoft patched a handful of long-standing vulnerabilities in their publicly exposed Windows server products that could have been used to create a devastating internet flashworm. But that didn't happen. Nor did it happen last month or the month before, and I doubt it will happen next month. For some time, we've seen serious vulnerabilities publicly

[02:30:10] [SPEAKER_00] exposed in Cisco edge border routers that could have been leveraged to do the same thing. But that's never occurred. My own theory, based upon watching the use of vulnerability exploits for many years, is that disrupting or taking down the internet is not profitable. What is highly profitable is breaking into an organization, exfiltrating their data,

[02:30:40] [SPEAKER_00] and then extorting payment from the breached organization in return for that data's deletion. So that is what has been going on, and there's every reason to believe that's what will continue to go on. Today's AI will likely serve as nothing more than an accelerant poured over the present status quo. I don't think it's going to see the world change. So I just expect that we're going to be seeing more of

[02:31:10] [SPEAKER_00] the same from the malicious use of AI, rather than anything apocalyptic. And once the products of defensive AI finally make their way into enterprise networks, such AI enhanced intrusions, those too will likely begin to dry up. Destroying the global internet, which is directly facilitating attackers' revenue stream, would be entirely self-defeating. Again, it has already been possible

[02:31:40] [SPEAKER_00] entirely without AI to wreak tremendous chaos on the internet and it has never happened. So my final take on all this is that we're going to stumble and bumble forward as we always do. Yes, there will be bumps and mistakes along the way, but we're going to be fine. Change is always a challenge and there has never been a change more sweeping than AI. Doug wrote, I'm tired of

[02:32:09] [SPEAKER_00] hearing that this is just the latest automobile or telephone or internet bringing disruption into our lives. Well, Doug may be tired of hearing that somewhere, but that's certainly not something that's ever been said here. I have absolutely no doubt and I know, Leo, you're on the same page here as I am that this generation of artificial intelligence will prove to be the

[02:32:39] [SPEAKER_00] we have actually created a machine that knows everything. It contains all knowledge that can be intoxicating, intimidating, thrilling, and incredibly useful. I expect that there will likely be many mistakes and missteps made. As I said near the top of this, I'm glad that the AI execs are publicly freaked out and frightened and that they, fine, pause for a

[02:33:08] [SPEAKER_00] while. Pause yourself. There's no hurry. Yeah, exactly. God knows you're turning out a new model every day. So, fine, take your time. Work out the training, work out the alignment, work out the guard rails, monitor and control agentic AI. We didn't have it six months ago. This is all still very new. And Andrew was correct to say that these problems will have solutions. The fact

[02:33:38] [SPEAKER_00] that we don't have them yet doesn't at all mean that they're impossible for us to engineer and get. So, I fully expect that everyone listening to this podcast today will have the opportunity to live out their full natural lifetimes without AI bringing it to a premature end.

[02:33:59] [SPEAKER_01] Nothing to worry here. Move on. You know, I think the question for you, Doug, would be, do you acknowledge that there are benefits to be achieved from this technology? And I think that that's the thing Steve and I both are saying, is that we can see significant, I already see significant benefits, and I see many, many more coming down the road.

[02:34:29] [SPEAKER_01] if you see benefits to this, yes, there's also potential harm, but it would seem foolish to say, well, because of this potential harm, we've got to stop because we don't want to take the risk and lose the benefits, and I think there are going to be some significant benefits. I also think it'll be, regardless, highly disruptive, but so every technology is, so was the steam engine, so was the industrial era, so was the locomotive, so was the automobile.

[02:34:58] [SPEAKER_01] You could honestly make a very strong argument, a million people a year are killed and injured by automobiles, that we should never have allowed the automobile to exist. Who let

[02:35:10] [SPEAKER_00] those things off their tracks,

[02:35:12] [SPEAKER_01] Leo? It's ruined cities, a great percentage of the land mass in cities is devoted to parking lots. Imagine people on their cell phones holding

[02:35:24] [SPEAKER_00] the steering wheel. I mean, people, cars are incredibly dangerous. You wouldn't design a road system with multi-ton high-speed masses of steel moving along and a person just can turn the wheel and point it wherever they want. It is insane. Yeah.

[02:35:52] [SPEAKER_01] And honestly, if people in 1910 had really thought about it, they might have said, you know, it's too dangerous, we need to halt this development. Just get more hay. Yeah. But as a society, we consciously or unconsciously made a decision to proceed. It's very much shaped our society, I say, for both good and bad. I'm not convinced it's a 100% benefit. I know it's not. I'm not even convinced it was worth it, but we made that decision. I think AI is of that

[02:36:21] [SPEAKER_01] nature, if not more so. It is technology that's going to change everything for good and bad. And I think it's worth pursuing because I already have seen huge benefits to myself.

[02:36:36] [SPEAKER_00] So, to answer the question of the podcast title, how worried should we be? I'm not worried. I mean, again, first of all, we have no control over it, so I don't tend to worry about things I have no control over. That's just, you know, spin cycles.

[02:36:52] [SPEAKER_01] People say, well, technology is inevitable. Once you invent atomic fission, you're going to have the atomic bomb. Maybe that's the case. Maybe not. Maybe we could stop it. Maybe we could have stopped automobiles. Maybe we should have stopped it. I don't know. I'm voting personally not to stop AI, but Doug may feel differently.

[02:37:12] [SPEAKER_00] I think it is really good that we had what happened back in March and April and May. It was absolutely useful. It was a wake-up call. Everybody understands now that this is an issue. The problem, of course, is that it's our domestic wannabe IPO companies that are worried and stopping. We see models flowing out of China, which are extremely good.

[02:37:41] [SPEAKER_00] That's what I'm pretty

[02:37:43] [SPEAKER_01] much exclusively using.

[02:37:44] [SPEAKER_00] Everybody is. Everybody is using Chinese models locally and using cloud AI to guide the local models. Right. Exactly.

[02:37:54] [SPEAKER_01] Yeah. And you know, somebody once said, I don't know if this is true, it could be that the Chinese models have an Easter egg, have a bomb hidden in them, that at some point, you know, March 4th, 2028, everything will stop working. I don't know. I guess that's possible. I'm willing to take that chance. I don't think it can. I don't think

[02:38:23] so.

[02:38:24] [SPEAKER_01] Steve Gibson is at GRC.com. That's where you will find so many wonderful things, including Spinrite, his amazing software, the world's best mass storage maintenance, performance enhancing and recovery utility. See, I don't think there's any, I don't believe there's any negative use of Spinrite. This is a technology that is entirely beneficial to humankind. I'm willing to say, I'm willing to go out on a limb. If you don't have Spinrite, well, you better darn well get it.

[02:38:53] [SPEAKER_01] You can also, while you're there at GRC.com, get his brand new program, DNS Benchmark Pro, which will benchmark your DNS choices. You know, you can make a choice and it's probably not the one you're making. Hundreds of

[02:39:07] [SPEAKER_00] DNS servers.

[02:39:08] [SPEAKER_01] Yeah, and many of them much faster probably than the one you're using. While you're there, you can also get a copy of this show. Steve has unique copies in every respect. He has a 16 kilobit audio version. No one knows why. Well, actually we do. We know why. He has a, but I'm not audio version, which is actually perfectly good audio. So that's the smallest good version to get of the show. He has the show notes, 21 pages this week of great stuff.

[02:39:37] [SPEAKER_01] You can get it there or if you want, you can subscribe. All you have to do is go to GRC.com slash email. The purpose of that page is to whitelist your email address so you can send, like Doug did, emails to Steve with complaints, comments, suggestions, plaudits. And by the way, thank you, Doug. Yeah, thank you. It's got a good, I think, thoughtful piece out of it. And after you've submitted your email address and Steve whitelist you, you can also sign up. You don't, just the little checkboxes below for the show notes to be

[02:40:07] [SPEAKER_01] mailed to you automatically every Sunday or Monday right before the show. And he's also got a new product announcement list, which he doesn't use, but he's got it. And that's the important thing. And if he did have a new product to announce, you would want that email. So go sign up over there. He also has transcriptions written by the wonderful Elaine Ferris, an actual human being. To our knowledge, has never done anything to harm a hair on any other human being's head. So another safe technology. We have a copy of the show at our website.

[02:40:37] [SPEAKER_01] We have two weird copies. We have a, how big is it? A hundred, a hundred, a hundred, it can't be a hundred, a hundred, twenty-two K. That's all right. That's all right. A hundred twenty-eight K MP3 version on our website, twit.tv slash SN. There's also video, which has, to my knowledge, harmed many people. That is available at the website or go to YouTube. There's a security now channel on YouTube. You can get every episode. Great for sharing clips. Best thing to do, subscribe to the audio or the video on your favorite podcast

[02:41:07] [SPEAKER_01] client. And then you don't even have to think about it. You just get it automatically. Steve will be back unworried, brow unfurrowed in October, the spooky months. Do you do Halloween in your night? Oh, Lori is a

[02:41:23] [SPEAKER_00] Halloween nut.

[02:41:26] [SPEAKER_01] Are the decorations up already?

[02:41:27] [SPEAKER_00] Oh, she has so, it's her favorite holiday. She has so much fun with it. So, it's funny because we were planning decorations for our new place for last October. But, you know, one thing after another. But this time we're there. And so, yes, we will definitely be, she will be decking the place out. You know what's really terrifying?

[02:41:52] [SPEAKER_01] AI. You can have some sort of AI thing to scare all the kiddies when they came out of the candy. I'll lend you a Quicksilver. You can borrow him. He's terrifying. Thank you all for being here. We will see you all next week on Security Now. Security

[02:42:11] [SPEAKER_02] Now.