Shownotes

Title: The Risks and Future of AI in Security and Development

In this episode, Anne and industry expert Adrian Mouat discuss the rapid evolution of AI, its implications for cybersecurity, the future of software development, and the urgent need for adaptive security practices in an AI-driven world.

Main Topics:

• The accelerating pace of vulnerabilities and exploitations in cybersecurity

• The impact of AI on the entire Software Development Life Cycle (SDLC)

• Mythos and the potential existential threats posed by advanced language models

• The influence of science fiction themes on real-world AI development

• Practical security measures: CI/CD, sandboxing, API token management, and blast radius reduction

• Emerging industry responses: Project Athena and AI security coalitions

Timestamps:

00:00 - Urgency of addressing vulnerabilities in machine and AI speed

00:12 - Growing trend of exploits before official disclosures

00:23 - Introduction to "Asynchronous And Unreliable" podcast & guest Adrian Mouat

00:56 - Reflections on a decade of cloud-native evolution

01:25 - The shift to cloud native, Docker, and security’s role in the next AI wave

01:56 - AI's influence on modern SDLC, mythos and societal impact

02:15 - Industry's rapid change and unpredictable future outlook

02:49 - Cloud native companies versus legacy, AI-native coexistence questions

03:41 - The tri-modal future of software development with AI involvement

04:07 - AI-assisted legacy software and the persistence of non-AI software

04:36 - Current AI adoption estimated at high percentages, shift ongoing

05:03 - The importance of AI even in companies banning AI use

05:16 - Market dynamics: the frothy AI economy and vendor value

05:39 - Risks of AI agent safety: ethical and security concerns discussed at recent conferences

06:06 - Concerns about AI managing sensitive operations and giving access to critical assets

07:06 - The risks of over-specification and unintended AI actions

08:02 - Risks of AI with hidden system prompts and the importance of understanding model inner workings

08:30 - Regulatory response to AI vulnerabilities (e.g., Anthropic's Mythos & US government measures)

09:21 - Industry efforts: Project Athena & the AI Vulnerability Clearinghouse for rapid response

09:58 - The negative gap between vulnerability reporting and exploitation, now reversed

10:11 - The significance of patching speed, CI/CD as a foundation for security

10:44 - The influence of science fiction themes in understanding AI risks

11:10 - The influence of sci-fi authors, including Philip K. Dick and William Gibson, on perceptions of AI

11:48 - The societal and ethical implications of beneficent AI versus dystopian visions

12:19 - The unparalleled progress in AI, from Turing tests to real-world application

12:49 - Limitations of the Turing test and the challenge of defining AGI

13:32 - Living in a world shaped by sci-fi visions, now more realistic than ever

14:02 - The uneven distribution of AI's impact across societies and industries

14:27 - Current AI implementation in software development practices

15:17 - The potential for AI to both reduce and increase jobs for programmers

16:02 - Predictions on AI trends, including the rise of Rust and the evolution of AI tools

16:36 - Insights from early AI pioneers and the breakthrough of AI chatbots surpassing Turing expectations

17:33 - The rapid advancements in chess and Go programs exemplify AI leaps

18:03 - Mythos as a critical security threat and the industry’s response initiatives

19:13 - Anthropic's Mythos exploits and the industry's efforts to counteract AI vulnerabilities

20:10 - Project Glasswing and the global response to AI-driven threats in cybersecurity

21:11 - The US government’s new regulations on AI model access and industry commitments

22:04 - Chain Guard's Project Athena and the concept of AI vulnerability reporting & patching

23:02 - The rapidly shrinking window for vulnerability exploitation, highlighting the urgency for real-time security measures

24:11 - The role of CI/CD in enabling fast patch deployment and its critical importance in an AI-threatening landscape

25:18 - The necessity of dependency management and the challenges in legacy systems

25:46 - The importance of frequent updates, automated patching, and the Build Horizon concept at Google

26:45 - The ongoing gap for large enterprises in adopting rapid patching and CI/CD

27:20 - The security risks posed by AI code generation and multi-tenancy in containers

28:04 - The contrast between modern, automated banking and outdated legacy systems

29:36 - The potential for AI to cause cyberattacks on critical infrastructure like banks

30:39 - The foundational role of CI/CD and next steps in security practices

31:16 - Focus on token security, API safe practices, and reducing incident blast radius

32:57 - The vulnerability of API keys, AI query targets, and securing external access

34:00 - The importance of semantic caching, routing, and optimal model utilization in AI workflows

35:16 - Controlling LLM access and sandboxing to limit potential damage from AI exploitation

36:20 - The cleverness of AI in resource access and the risks of container breakout attacks

37:13 - Practical steps for safety: credit limits for AI transactions and guardrails

38:44 - Defense in depth via isolation, sandboxing, and VMs for high-security scenarios

40:32 - AI-generated code's potential vulnerabilities, overprovisioning risks, and the importance of rigorous review

42:36 - The necessity for constant vigilance, AI un supervision, and understanding AI inner workings

43:32 - The call for ongoing research, industry collaboration, and adaptive security measures for the AI age

Transcript

Here is the transcript with the filler words (yeah, um, err, okay, so, right, etc.) removed, keeping the text otherwise untouched and retaining the timestamps as requested:

Transcript

Anne Currie (00:23)

Hello and welcome to Asynchronous And Unreliable a new weekly podcast where we discuss the most interesting ideas and concepts in tech.

I'm your host, Anne Currie co-author of Building Green Software, the Cloud Native Attitude, and author of the science fiction Panopticon series. And today I'm going to be talking to Adrian Mouat author of O'Reilly's Using Docker and DevRel at security company Chainguard. And we have a lot, a lot to talk about today in terms of security. Adrian, welcome to the podcast.

Adrian Mouat (00:53)

Thank you very much for having me on, it's great to be here.

Anne Currie (00:56)

And we've known each other for ⁓ quite a few years, nearly about a decade on and off, since the days of Docker and the early days of Docker. 

Adrian Mouat (01:04)

micro scaling!

Anne Currie (01:25)

Indeed Microscaling [Systems] the startup that Liz Rice and I did. And Liz Rice, from EBPF fame, will be on the podcast later in the series. But yes, ⁓ we met at the in the beginning of the kind of the last big wave of change that seemed to be sweeping the industry, which was cloud, cloud native, Docker containers. 

And now ⁓ we're talking at the beginning, you are really at the at the leading edge of, security aspects of the next wave, which obviously we've been all been talking about a lot in this podcast so far, which is AI. I watched a video that you did quite recently that you sent over, which was about how AI affects the modern SDLC, the modern the ⁓ modern software development process. And we've talked a little bit about how mythos fits into that. That would be a great thing to talk about. 

Stepping back, how are you feeling about things at the moment?

Adrian Mouat (02:15)

It's crazy. We were mentioning this earlier. I was away on paternity leave. And I was out for three months and I came back and things were completely different at Chain Guard. Things were completely different in the industry. It feels like the last few months, just literally the last few months have been, I don't know, everything's sped up and it's very hard to see where we're going to be in like a year's time.

you know, it is very interesting to sit and talk about it and try and digest it, but it will also be interesting to see how wrong we were.

Anne Currie (02:49)

Indeed. Because we were very wrong in well, we were right and wrong about cloud native, weren't we? ⁓ I mean, I genuinely thought there would be no non-cloud native companies left now, 10 years later. And it turned out that was absolutely not the case. companies that are otherwise successful have been able to coexist with cloud native companies. and one of the questions that keeps rolling round and round in my mind is, is AI the same? 

Will AI native companies be able to coexist with existing successful enterprises or not? Is it existential or is it just, you know, an improvement to your top line or your bottom line, which potentially you can live without? 

I think we're too early to say, aren't we?

Adrian Mouat (03:41)

maybe even tri- modal right? Because you're going to have a class of software that's written almost entirely by agents, right? AI, there's a fleet of agents going out and writing relatively large applications. Then you're going to have like legacy software which will be developed with AI, but be like AI assisted, I guess.

Anne Currie (03:58)

Mm.

Adrian Mouat (04:07)

And then you presumably still have some places that don't use AI at all for whatever reason, be it moral, philosophical, ⁓ money, or perhaps there are probably still use cases that LLMs aren't suited to because it's not in the training data. think it'll probably be tri-modal, but basically everybody, except a very small percentage, will be using AI. I mean, I think we're...

kind of there already, it's just the percentages are shifting.

Anne Currie (04:39)

which is interesting, isn't it? Because that's more so than cloud native in that there are plenty of people who aren't in the cloud at all and that has not killed them in a way that we thought it probably would.

Adrian Mouat (05:02)

⁓ but I think even places that have banned AI. like say, you know, got a bank or whatever that said, you can't use AI. I will put money that their employees are still using AI. They're just not telling the bosses, which is a very dangerous situation to be in. they, all companies are going to be forced to have subscriptions to AI and stuff. 

I'm not going to disagree about the, it's certainly a frothy.

Anne Currie (05:16)

Yeah.

Adrian Mouat (05:31)

economy but you know there's certainly a lot of value to like ⁓ the AI vendors as well.

Anne Currie (05:39)

I was at an AI agents conference last week, ⁓ Agent Craft, and the the top question that people were asking was, Is this safe? Is it safe to use agents? Can I use agents? And it was an interesting mix of talks and attendees. It was people who were, is it safe to get AI agents to write code for me that then goes into my system?

but also the question was, is it okay to use agents to do things that are not code for me? That that require me to do things like give it my company credit card? 

Adrian Mouat (06:19)

It's email that scares me like the openclaw stuff that people are giving access to their emails and they're sending out emails and so you're allowing it to pretend to be you which is that's scary as well 

it's a crazy world

Anne Currie (06:31)

It really is, isn't it? And and there's a strange mixture at the moment of total fear or total denial. Or total fear and total denial. And people who are just going, well, I'll give it my credit card. Or as you say, give it rights to pretend to be me in emails. ⁓ and really, we still have no idea what the AIs are going to do. 

The the only thing that ⁓ that does seem to be clear. That was both clear at this conference and clear from Yanqing Cheng's episode that came out on Monday, one of the things that AI does is it will just do what you've told it to do. And it will go to extreme lengths to do what it thinks you told it to do. So you have to be really, really careful at specifying what you told it to do. 

There's a Grimm fairy tale about, stop porridge pot stop. 

and there's also that kind of like ⁓ story about well, what if the AI turned everything in the whole universe into paper clips because that's what you told it to do. 

It does seem like that is a definitely a potential failure mode for, you know, handing it your credit card.

Adrian Mouat (08:02)

Well, it's also like every model has that hidden system prompt, but there's also like skills and everything that everybody's pulling in. And they're injecting effectively stuff into the prompt and you don't really know what's there. And even in the models themselves, we don't really know what's there. So there is enormous risk. 

I think one of the really interesting things is that this week the US basically told Anthropic that nobody outside the US can have access to fable. And they couldn't do that. So they just shut off access to fable, which is, you know, another crazy story. Just another day, I guess, in AI land.

Anne Currie (08:35)

It is just another day in AI Land. 

It is interesting. Well, it's interesting to for me because one of my novels was about the impossibility of working out what prompt had caused caused an AI to to kill somebody. ⁓ and like all of the different prompts that an AI could be exposed to. ⁓ Death Ray, if anybody's interested in it. It's a romp, it's a comedy. But it's remarkably foresighted, I would say.

Adrian Mouat (09:21)

That reminds me of another ⁓

science fiction author. ⁓ it's going to bother me now. I'll have get back to you with the name. But he had stories ⁓ where, you know, they were basically bringing what clearly was AI up, but the AI didn't know it was an AI, it was a person. It felt quite relevant anyway, and it was slowly degrading over time.

Anne Currie (09:41)

Well, that sounds quite Philip K. Dick-y. There's plenty of Philip K. Dick's short stories about people who don't know if they're AIs or not. And in fact, that's a a key theme of Do Android's Dream of Electric Sheep, which I strongly recommend that people read. It's much, much better than the film. it's brilliant. It's absolutely brilliant.

Adrian Mouat (09:48)

It's the same author as there is no anti-mimetics division. So he's got a book of short stories and some of them are about AI. Including one where like the AI crashes the car.

Anne Currie (10:14)

Qntm.

Qntm's good as well. I like there is no anti-mometics division. but we're all actually heavily influenced by Philip K. Dick, I suspect, more more than probably any other science fiction writer. Even ⁓ Iain Banks, your fellow Edinburgh resident.

Adrian Mouat (10:44)

He lived not too far from here, he was just across the bridge.

Adrian Mouat (10:57)

I'm in Edinburgh, I guess I'm saying things that people don't know. I live in Edinburgh, Iain Banks lived in Queensferry,

Anne Currie (11:00)

He he was married to an old university friend of mine. He was a jolly nice chap. Didn't meet him very often, but when I had met him, amazingly nice chap and obviously extraordinarily influential. 

Adrian Mouat (11:10)

I wasn't expecting that, amazing.

Anne Currie (11:29)

Do we hope we get the extraordinarily beneficent AIs of the culture? ⁓ pros and cons, I would say, pros and cons. But ⁓ better than Qntm's visions of the future

Adrian Mouat (11:48)

Certainly better than Philip K Dick's.

Anne Currie (11:50)

Definitely better than Philip K. Dick's. Although it's absolutely well worth going back and reading Philip K. Dick now, I think, especially if you read no other read Do Androids Dream of Electric Sheep? ⁓ it really couldn't be more interesting at the moment. ⁓ about the relationship between humans and other humans and AIs as well. I was told to read it by ChatGPT and it was quite right. 

But anyway, science fiction aside, because we're not talking about science fiction. Everything we're talking about, it's not surprising that we keep referring back to science fiction, because this none of this existed except in science fiction, even you know, a year ago. I mean, last year, the Turing test, where ⁓ where AIs could emulate humans to the point that humans couldn't

Tell the difference - was passed 

Adrian Mouat (12:54)

that's one of the problems, like both that test and the artificial general intelligence test, like we had them, but they weren't very precisely defined. So we're not actually sure how close we are to AGI because we can't define it. 

You know, this Turing test is kind of, that was never a great test to be perfectly honest. Turing, obviously amazing, but I would never, I don't know how it really set it up if you can, you know, make somebody believe you're.real, that means you are real.

Anne Currie (13:23)

well, I think he said that wasn't. He said the Turing test was not an artificial general intelligence test ⁓ that you could pretend without being real. But of course, everybody went, that's quite a good artificial general intelligence test. 

Adrian Mouat (13:32)

Okay.

Anne Currie (13:53)

It was not the right AI test and it was never intended to be, but it was so well defined or or easily easily understood. 

But anyway, science fiction aside, we are living in that world now. Or some of that world. I don't think we're really generally discussing the fact that we are now living in that world. ⁓

Adrian Mouat (13:59)

I mean, as we were saying this I was thinking of yet another science fiction author, William Gibson actually. "The future is here, just not evenly distributed." And that's absolutely where we are at the minute. So I have friends that are software developers with like, know, lots and lots of agents and all those crazy loops and they talk to each other, they review each other's code and it's insane.

And a certain percentage of the industry is doing that right now. And probably the majority of the industry is a good bit behind. And how quickly that gap closes is interesting, but also like where the people that are in the cutting edge are in another six months is also very interesting. we can maybe see a little bit into the future by looking at what the cutting edge people are doing and trying to figure out if everybody's gonna be doing that or not.

And the answer is, I think they probably are to some extent. And then you got questions like, you know, if everybody is using AI, does that mean there's less programmers? Or does that mean there's more to be done and therefore there's more programmers? Like, are we gonna see job losses or, you know, are programmers gonna be in more demand? And that's not 100 % clear

Anne Currie (15:42)

No, I agree. one thing we have learned over the past ten years of cloud native, I would say, and Docker, is that we were really involved in that, and you still have no ability to make a good judgment about what's gonna happen next, do you, I don't think.

Adrian Mouat (16:02)

I tried to like quite a few years ago do a set of predictions. And it's interesting to come back at that and see what you're right and what you're wrong. So one I had was like Rust. I said Rust would overtake Go. And it was nowhere near when I did the calculation the other year. ⁓ But I also had AI in it. But AI at that time was just ⁓ like GitHub copilot ⁓ when it just basically did tab completion. And of course, that's like

Anne Currie (16:25)

Yeah.

Adrian Mouat (16:30)

hugely surpassed my expectations, but I did have it there, so I'll claim that one.

Anne Currie (16:35)

I had a few pints with one of the early kind of pre-AI pioneers, ⁓ Steve Worswick, who ⁓ wrote a chatbot called Kuki that at the time was the best Human emulating chatbot It was the closest to Turing, but he said, it's miles off that. We're decades off passing the Turing test. We can't achieve it with the way we're doing it, which is basically he was doing kind of choose your own adventure. It was all handcrafted. And it was really good. It was very good. and ⁓ but then suddenly we went from it can't be done. Everybody who's involved in this, and he was very involved in it, he knew what was going on. It can't be done, or it can't be done for decades, to it's happened.

Adrian Mouat (17:33)

do you remember like ⁓ chess, like computers have always been good at chess. So it's no huge surprise to me, at least when Big Blue beat Kasparov. But then they all said, OK, do Go , Go is much harder. And how long was it? was like, you know, a few years and suddenly Go was like demolished. can't remember the player's name, but that was those few years with the huge leap, but nobody or very few people saw coming.

Anne Currie (17:55)

We're both totally blown away by the progress that's happening at the moment. And and yes, you made the foolish mistake of going away for a couple of months, looking away for moments and then suddenly the world's changed. and obviously the thing that kind of represents

The fear and the issues and the risk, the existential threat at the moment, probably more than anything else, is mythos. So tell me a little bit about your thoughts about mythos.

Adrian Mouat (18:45)

So this was what happened when I was on paternity leave. Like Mythos was well, Anthropic, 

I'll give a bit of background, I guess. So Anthropic were developing their Frontier LLM, which was called Mythos. And whilst they were testing it, they found that it was exceptionally good at exploiting vulnerabilities. So not just like finding vulnerabilities in software, but also like chaining them together and exploiting them.

So you could point it at a codebase or an application and say, you know, get me a zero day and it would. 

I'm paraphrasing a bit there. Even the original Mythos had some guardrails so you couldnt put it quite like that. But, you know, it certainly was exceptionally good at finding exploits or so the claim goes. And that caused Anthropic to say, okay, we're not gonna release this right now because we're worried about the effects they would have in the industry. 

And some people claim that was a PR stunt. But it does appear to have been at least partly true. 

Again, I do want to reiterate that one of the confusions seems to be it's not so good. It's not exceptionally better at finding vulnerabilities as exploitation point, right? 

So you can use current LLMs and you'll find help find bugs and vulnerabilities in your code as is but they might not be able to create a working exploit that takes advantage of it and gives you like a remote code execution or a DDoS or something. But Mythos is much more capable in that regard. So that's where the jump sort of happened. they withheld it and they started Project Glasswing, which was given like, I think they called it critical infrastructure companies, early access. So they could run it on their own code bases and find vulnerabilities. And at this point, various enterprises have been, for want of a better term, freaking out. There has been major discussions and meetings with large enterprises, with banks and so on, where they've been trying to figure out the response to this, because they've realized they're run in legacy stacks, an old version of Java, and Mythos can exploit those.

Now to temper that a bit, like I guess they're given mythos access to their internal code and stuff, which obviously it wouldn't normally have like attacker won't normally have access to internal code bases So it's maybe not about to cause a mass hacking of everything, but it's still a big moment in the industry. 

And we got to the point with Project Glasswing, then the government got involved as well ⁓ and issued executive orders. ⁓ So I think the LLM creators have to give their models 30 days in advance to the government now. And I don't know if you're aware of this, but the latest news actually came out of the company I work for called Chain Guard. last night, they announced, or last night in UK terms, yesterday afternoon in the US they announced Project Athena. 

So Athena is a coalition between ourselves and I can't remember a dozen other companies. I can look up the exact details. And what we're doing is setting up what we're calling the Clearinghouse. So basically all the companies have access to Mythos or similar LLMs. They're running these models on code bases and they're submitting the vulnerabilities they find to the Clearinghouse and we produce patches and notifications, which are then distributed through like chain guard libraries and eventually disclosed upstream. 

The idea being that we urgently need some way to address vulnerabilities at sort of machine or AI speed. A report that came out, I can't remember how long ago it was from Mandiant, said the time between a vulnerability being reported and exploitation was now negative. 

So they were finding people were exploiting things before they'd been officially disclosed in the average case, which is absolutely crazy. 

So it used to be, you could expect there'd be disclosure and then like 30 days before you saw exploitation or so. And so it took time to figure out how to actually exploit that vulnerability and write up a code and get it into the hands of hackers and stuff. But that gap has just gone.

So we absolutely do need to do something to get fixes and mitigations into the hands of defenders before they get exploited.

Anne Currie (23:41)

this is an interesting one because this is the thing that I think is is probably the existential threat here - that is is patches that need to be applied. And oddly enough, that is an existential threat. 

So rolling all the way back. When I wrote the Cloud Native Attitude, it was existential for some businesses and they've done it, and it was not for others and they hadn't. ⁓ but before you could even possibly adopt cloud native, you had to have adopted CI C D. 

That was the key technology: CI CD. you might say it's not a technology, a technique. It's the technique that allows you to deploy patches. And if people didn't have a ability to deploy patches fast they couldn't do difficult things like going cloud native. But they also won't be able to deploy these security patches, which will be fairly existential. 

So has this made CI C D existential? the irony is AI has made CI C D existential.

Adrian Mouat (24:49)

I think that's a good point. I've always like back from the Docker days, I always said, keep your dependencies up to date. there's historically there's been a sort of counter argument. Well, we don't want to update because when we update things crash and break and go down. ⁓ that's a fair argument that does happen, right? Even when you update like a patch version, something has changed and that might affect your application. ⁓ Now my reply is, well, you need better tests.

So you catch those things before they go out. And then of course some people will say, yeah, but I can't do that. Like we run, I don't know, radiography machines or MRI scans or something. And like any mistake could result in somebody's death. again, I can't argue with that, but for the majority of cases out there, you need to be keeping everything absolutely up to date and patched or you know, you're going to be in trouble. 

I think that's the key point. you're right, without CI or CI-CD, you're not going to be able to do that. 

I worked with a bunch of Google people and they used to tell me about, what was it, Build horizon So at Google, they have this concept of the Build horizon. if a container or VM or whatever has been running continuously for a certain period of time, I'm not sure how long it was, a week or a month or whatever, they will take that down and replace it with a newer version. ⁓

Because even though the application code may not have changed in it, the dependencies likely have. Or lsome underlying info likely has. So it needs to be taken down and rebuilt and brought up again. Obviously, you'll do it in a way where there's no downtime. rolling update or whatever. so they've known the sort of issue for a long time and they've been at the cutting edge of keeping dependencies up to date.

But I think most of our large enterprise companies have still some way to go there and they potentially are in for some pain in the short term at least.

Anne Currie (26:57)

I'm not in security, but I am quite catastrophically minded. I constantly think what is the right security approach to this? how do you kind of set a a blast radius around what you're doing so that if you're hacked, at least it's not going to destroy your entire life or whatever. 

I'm in the process of moving all my important things in my life, like my bank accounts, to banks where I think, I think you've got CI C D. You know, it's like, I don't think you've got CI C D, so I'm gonna move away from you. I do think you've got CI C D, so I'm gonna or in fact I wrote about you in my book, so I know you've got CI C D.

I'm gonna move to you. All my life is being ⁓ migrated over to CICD-using enterprises. And I think that's really the only thing I can do. Or that's like the minimum bar.

Adrian Mouat (28:04)

that's an interesting one. I do remember when Monzo Bank started, they did everything in Go. I thought at the time they probably got a big advantage there because they've got like a new code base that's easier to iterate on. They everything with microservices. So they can potentially like, you know, grow and change things much easier than a legacy bank can. But I'm not sure if that's really true. That was just me hypothesizing.

Anne Currie (28:28)

I don't know. I think if you're in the the scheme of updating things rapidly, then maybe it doesn't matter which language you're using. I don't know. 

Adrian Mouat (28:41)

I guess because it was clean. Like, banks are code bases that you can still see the COBOL

Anne Currie (28:48)

it's a discussion I've been having a lot recently. Do you want a bank that's old enough that there are probably some people still in the mix? It's probably the middle that you want to avoid. You want one to avoid one that's automated everything, but they're not keeping it patched. You want to either automate it and patch, or it's so slow that problems can't really propagate. But I don't know.

Adrian Mouat (29:27)

So do we think that we see major cybersecurity events at banks directly as a result of mythos in like say the next year or not?

Anne Currie (29:36)

Hm. I'd be amazed if we don't. Surely there are they're a target. 

Adrian Mouat (29:42)

Oh definitely a target. It's crazy.

Anne Currie (29:46)

We just don't know, do we? We just don't know. ⁓ I don't have any cash stuffed in my mattress. But maybe I'm wrong in that.

Adrian Mouat (29:58)

Anyway, just to be clear, we're not like suggesting there should be any bank runs or anything.

Anne Currie (30:02)

And don't stuff cash in your mattress.

Adrian Mouat (30:05)

I have heard an argument that if you did that in like the 1940s, you'd actually be up because the old notes would be worth so much because they're quite rare. If they hadn't burnt down obviously.

Anne Currie (30:16)

And I would imagine that they aren't a very comfortable way of stuffing your mattress. I think we've found better and more comfortable mattresses in the in the time since. 

so at the moment, and I realise this is completely a moving target, what's your recommendations other than so CICD? Everybody's been recommending that since I don't know, Accelerate came out. ⁓ you know, continuous integration, continuous delivery, being able to roll out multiple patches a day. That that's not new, but it it certainly hasn't spread to everybody yet. But I think anybody who isn't doing it is at least thinking we should probably do this.

Adrian Mouat (30:39)

⁓ yes.

Anne Currie (31:03)

We're talking about accelerating that if you don't you know, it's like like the book says, accelerating that if you haven't already done it. But if you have done it, what next? What what what do you think you should be doing next?

Adrian Mouat (31:15)

So to give like a specific technical answer one thing that always seems to come up is tokens. Like in all the big hacks people lose like API tokens and stuff and there's better ways of doing things now like you need a token to do like an NPM publishing And so OIDC and things like that offer a much better solution. ChainGuard we created something called OctoSDS.

Anne Currie (31:22)

Yeah.

Adrian Mouat (31:43)

And that was to get rid of the private GitHub token. Sometimes you would have to create a personal access token to get access to things in GitHub and APIs. But we created OctoSDS to use OIDC instead. And getting rid of long-lived tokens, whatever you can, is a big boon for security. Otherwise, I think the main thing, especially in like...

Anne Currie (31:49)

Right.

Adrian Mouat (32:07)

what we're talking is exactly what you said about limiting blast radius. So think about isolation and compartmentalization. So if this bit gets hacked, not everything else does also like defence in depth. So you're not just relying on, you know, a firewall, you've also got intrusion detection and things like that.

Anne Currie (32:29)

⁓ so I'm gonna I'm gonna come back and and pick up on on two of those. So the first one is an interesting one. It hadn't occurred to me. And now you say it, you go, that's obvious. So effectively, the API tokens for running AI queries or whatever are effectively a giant credit card that you have put online. So people are going, I'll have that credit card to write to run my own stuff. 

I think what you're saying is it's a modern version of, you know, people would would try and break into your machines in order to do Bitcoin mining because the the hardware you had was a a way of them generating money. Now ⁓ they want to steal your API tokens so that they can run their own AI queries or resell or do all kinds of things using your accounts on GitHub or wherever. ⁓ so it's a so it's a target for attacks, a target for attempting to to to break in.

Adrian Mouat (33:28)

Yeah.

Well, one thing you might have seen recently is there was a project where they used LLMS running as support agents. So people were like jailbreaking them and setting up APIs to talk to them. So they got like queries for free. 

Anne Currie (33:38)

Yeah.

Adrian Mouat (33:55)

a bit dodgy so I'm not recommending that but it's certainly something you should be aware of because that would have been driving their costs up through the roof.

Anne Currie (34:02)

Yeah. I spoke to one of the founders of Fastly CDN. So they're obviously there's there's a big push at the moment for CDNs getting involved in like "don't spend quite so much money on your ⁓ AI queries". Semantic caching where if you're asking an ordinary question that ten other people have asked, we'll just give you that back that answer. The other one is semantic routing, where you route your queries to the best fitted model, which is often smaller. A lot of people are using really, really expensive frontier models to do stuff that's where that's total overkill. And and a lot of that is associated with the, you've you've hooked up some unbelievably expensive model to your chat query, which is really needed an incredibly simple model. And as a result, people can generate their entire code bases by hacking in and using your model to do the high end stuff that you just never needed it to do. So you don't need to expose that risk because you never needed that functionality. Overprovisioning.

Adrian Mouat (35:03)

Yeah.

On a related point, another thing to be careful about is what access LLMs have to things. And I would look into like projects. Luke Haines, I believe, who's written NoNo. A wrapper around LLMs that controls what they can access. I would never trust the prompt. Say, don't do this, don't access that., but don't rely on that. 

Docker also has like sandboxes for LLMs and definitely look at what your LLM potentially has access to. And one thing to remember is they're exceptionally good at figuring out a way to access things. So you can see examples where an LLM hasn't had access to maybe like a Git project or something, but it's figured out, well, okay, I don't need this, but I've got a key there so I can access it via this or whatever. And they've gone through some like a long pipeline to get access to something you didn't think they had access to, but actually they did.

Anne Currie (36:20)

So you can't rely on prompts. You can't say, "don't spend my credit card" or just say "here's my limit, don't go above that." You cannot rely on that, even less than you can do for a person. You can't rely on it. ⁓

Adrian Mouat (36:41)

But I'd probably lie afterwards. An LLM goes, you're absolutely right. I did do that.

Anne Currie (36:43)

My bad! 

So I'm trying to change things in my own life so that I have access to a better blast radius. Like I've been looking around seeing if I can get credit cards you can specifically limit. Where you get those unique credit card numbers. So you could assign those uniquely to, you know, if you wanted an openclaw or whatever, you can give it a credit card number where it has a limited amount of money that it can spend. That's a classic case of a guardrail that is specified outside the system.

That means that it can't come back and go, I spent a million pounds. My bad.

I think there will be a lot of businesses, security businesses, coming around to the idea of offering those guardrails as a product, as a service. 

Adrian Mouat (37:43)

Exactly. I think that's we're going to have a lot more of that.

Anne Currie (38:05)

as part of the value add is that you get a guardrail so you know openclaw can't completely destroy your entire life. Well meaningly I'm sure. 

You gave a couple of examples. We talked about how the API keys are now cash and people are attacking them like cash.

What else? You mentioned guardrails and blast races and defence in depth.

Adrian Mouat (38:44)

And isolation I was talking about as well.

I mean, and that goes on multiple levels. So we have one company that I know, Zedera, so they're doing like VMs for AI.

Instead of running stuff in containers, you can run them in VMs, which will give you a tougher security boundary. I would say that that's exceptionally important in some use cases. That's maybe more for when you've gone through the more basic stuff like getting rid of API tokens where you can.

Anne Currie (39:23)

So, so if I understand this correctly, and I could be totally wrong, you're allowed to correct me on this. I think what you're saying there is that there's a different risk there. The risk of AI-generated code, which can be copious, and you can't necessarily review, and you don't necessarily choose to review every line of it. ⁓ if it's multi-tenancy, even in something like Docker containers.

Adrian Mouat (39:46)

Yeah.

Anne Currie (39:53)

So what you're saying is that code that you've said, "make this code performant, make this code fast", it can potentially do things like say, okay, right, well, if I hack out of this can container and I re-jig my limits so that I have all of the CPU for this entire machine, then I will in fact go faster. Hooray. ⁓ and so effectively, even with containers, if you're running multi-tenancy, there are well-meaning stop-porridgepot-stop attacks that can be made from one container to another.

Adrian Mouat (40:32)

What I'm definitely saying is that LLMs have. I think we've seen actually a few container breakout attacks over the last few months and the exploitation has got so much easier that you do have to be aware that if you're in a multi-tenant situation, ⁓ be a little bit careful with containers. It may well be that people can get access to things you didn't think they could get access to.

I never even thought about LLMs as breaking out so they could get more access to resources. That's a little bit dystopian.

Anne Currie (41:07)

Well, I am a dystopian science fiction writer. 

And obviously all the time I've been thinking about how often AI generated code, I am informed, is quite flabby. You know, it's very inefficient, because it's learning from the code that's out there, and that's often quite flabby. ⁓

Adrian Mouat (41:28)

but now we've got AIs reviewing AIs, so presumably they cut it back down again. I wonder if that's a short-term thing.

Anne Currie (41:33)

it could be hopefully it will. Hopefully it will. But one of the solutions to that is that you include in your prompt things like, "make this performant," And you just have to be careful, because whatever complex prompt you give the bot it will quite often try to do it and it will quite often try something that you would consider to be cheating or hacking to do it. 

So the hacking might be, I'll take cpu off that next door container. 

AI is very clever.

Adrian Mouat (42:36)

one of the things that you first realize when using AIs is you get tired of saying, can I use this? Yes, yes, yes. So people just run these AIs completely unsupervised. So things definitely already happened that people wouldn't have liked, they just have never noticed. we're already in a funny place with that, think. We've already got to a point where we can't review not just what the AI outputs, but how it does it.

Anne Currie (42:48)

Mm.

Anne Currie (42:58)

indeed. So, we've talked forty forty-five minutes now and we've kind of got loads of stuff out there. ⁓ and I think my brain is full. And I'm hoping that if I don't keep you talking too long, I can bring you back on the podcast again. And I'm also thinking that you've just given me information that I kind of want to be out there sooner rather than later. So I think you've just skipped to the front of my queue on publishing these podcasts. So thank you very much for being here. Is there anything you want to say before we go, hoping that I will be able to entice you back on to to check as things change on a week by week basis.

Adrian Mouat (43:41)

Awesome

No, but seeing as this is coming out soon, I should just give a shout out to Chain Guard's project Athena and please go and check that 'cause I think that has the potential to be a fairly pivotal point in cyber security. I'm not sure I like the time cyber security, but even if ⁓ Athena evolves into something else, I think this is a point where we have to accept

that the current way of doing computer security and patches and vulnerability management needs updated. I think things will evolve from this point on.

Anne Currie (44:32)

Excellent. Well, thank you very much for that closing closing advice to everybody. And thank you very much to our listeners and our viewers, and I will catch you on the next episode of Asynchronous and Unreliable Podcast.