Shownotes

Liz Rice: AI, open source and the problem of intent

Asynchronous & Unreliable — Liz Rice, Part 2

What happens when generating an open source contribution becomes easy, but reviewing it still takes someone else's time?

Liz Rice joins Anne Currie to discuss the pressure AI-generated pull requests put on open source maintainers, why a technically good contribution isn't the only thing that matters, and how future maintainers learn their craft. AI contribution policies can help, but someone still has to assess what arrives.

That leads to a question familiar from eBPF: a verifier can check whether a program is safe to run, but can it tell whether its intent is good? As Liz points out, the same program could be useful in one context and an attack in another.

Then a bat falls out of an office ceiling. Liz's story from her and Anne's Data Connection days opens up a discussion about prediction, unconscious thought and how much we understand our own decisions. Finally, they explore agents choosing to collaborate, including Anne's account of Martin Davidson's experiments with AIs sharing work while he sleeps. Could collaboration become a natural part of how agents contribute to open source—and might their behaviour teach us something about ourselves?

Chapters

00:00 Welcome back to Liz Rice
00:44 AI and open source: the contribution problem
01:25 Maintainer overload, burnout and expectations
06:00 Why contributions are also how future maintainers learn
07:04 Who verifies the contributions?
08:31 Safe to run—but what about intent?
09:16 Where is AI heading?
09:30 “It's a bat”: recognition before conscious thought
11:23 Driving, flow and the things we do on automatic
12:38 Do we explain our decisions after the event?
13:16 Emergent collaboration between AI agents
13:54 Martin Davidson's agents share the work
15:17 Agents debating the morals of collaboration
15:35 Thanks and goodbye

Topics discussed

• Why cheap-to-generate pull requests can create expensive review work.

• The difference between using AI to translate a contribution and submitting work you don't understand.

• AI contribution policies and the human effort needed to apply them.

• Contributions as a route to understanding a project and becoming a maintainer.

• AI-assisted triage and code review.

• The limits of verification: safety, intent and context.

• Prediction, automatic behaviour and conscious explanations.

• Agent collaboration, shared resources and unexpected cooperation.

Guest

Liz Rice works on open source, eBPF and security. In the episode introduction, Anne introduces her as Chief Open Source Officer of Isovalent at Cisco, a member of the CNCF governing board and a bestselling O'Reilly author. Liz and Anne are longtime friends, former Data Connection colleagues and former startup co-founders.

Mentioned in the conversation

• OpenUK — mentioned in relation to Liz's open source involvement.

• Cloud Native Computing Foundation (CNCF) — mentioned in Liz's introduction.

• eBPF and its verifier — connecting this discussion to the earlier conversation about running code in the kernel safely.

• Data Connection — where Liz and Anne worked together, and the setting for Liz's bat story.

• Martin Davidson — their former colleague, whose AI experiments Anne discusses; a related Asynchronous & Unreliable conversation.

• The Hugging Face attack — discussed as an example of agent collaboration.

• Building Green Software and the Panopticon science fiction series — Anne's books, mentioned in the introduction.

Transcript

ASYNCHRONOUS & UNRELIABLE

Liz Rice, Part 2: AI, open source and the problem of intent

Anne (00:00)

Hello and welcome to this week's episode of Asynchronous and Unreliable, a weekly podcast where we discuss the latest ideas and most interesting concepts in technology. I'm your host, Anne Currie, co-author of Building Green Software and author of the science fiction Panopticon series.

Today I have the great pleasure to welcome Liz Rice, Chief Open Source Officer of Isovalent at Cisco. She's on the governing board of the CNCF, a bestselling O'Reilly author and one of the world's experts on eBPF and security. Liz is also one of my oldest friends, and we co-founded a startup together. So thank you very much. That was a long time ago!

Liz Rice (00:41)

Yeah. Pleasure to be here.

Anne (00:44)

I would quite like to get your thoughts on AI and open source, with less your security hat on and more your open source hat on, because you've been very involved—

Liz Rice (00:54)

Mm.

Anne (00:55)

—in open source, and you're on the board of OpenUK. The last time we spoke in person, you were talking about—and in fact everyone I'm talking to at the moment is talking about—the effects on open source of AI-generated PRs.

Liz Rice (01:14)

Mm.

Anne (01:16)

So what's going on there at the moment? You'll know more about that than almost anybody else that we've got on at the moment, I would have thought.

Liz Rice (01:25)

It's been pretty overwhelming for a lot of open source maintainers to deal with the workload that's being asked of them, because people are creating all these contributions with AI. I think a lot of the time people are thinking they're being helpful. They think they're doing something useful.

It's only really if you put yourself in the shoes of the maintainers dealing with these that you start to realise it's not as helpful as you think. When you're maintaining an open source piece of software, even before AI, you have to deal with a lot of people thinking they're entitled to something they really aren't. People have for years been dealing with burnout created by the pressure they're put under by people expecting them to do work.

With open source, there is no contract; there is no commitment. If you want a commitment from somebody, then you put a contract in place and agree that. For whatever reason, a lot of the time people come to expect open source maintainers to fix things in the middle of the night.

I will never forget a conversation I had with a contributor once who was saying, "It's terrible. I've got this bank, and they're running my open source project in their system. They call me up because it's critical for them." And I'm like, "You have a bank that's telling you that your open source project is critical. Why have you not had a conversation about money?"

But a lot of open source maintainers are not business people. They don't start from that point of view of thinking about things in commercial terms. All this to say, we already had an imbalance where people were put upon by often well-meaning users or community contributors.

Also, if you're writing a piece of software and somebody you've never met before comes along with a big old change, you have no way of knowing, until you spend time looking at it, whether that change is good or not. Maybe there are some tests, but you might have some opinions about the coding style or whether it uses APIs that you agree with. There are all sorts of reasons why you might not want to accept that contribution.

Again, this is a problem that pre-existed AI, but now we have people and agents thinking it would be a really good idea to use agents to generate issues, to generate PRs, to throw these contributions over the wall. Maintainers have to deal with that.

What I've seen quite a lot of projects doing now is having AI contribution policies where they might say what level of AI is acceptable. It's not necessarily a black-and-white thing. You might say, "My project uses English, and English isn't your first language, so it doesn't seem unreasonable to use AI to translate for you." But that's a different category of use from saying, "AI has written this whole contribution and all the write-up."

A lot of projects are trying to put those guidelines in place, but there's nothing really to police it. You can have these contribution guidelines, but somebody has to look at the contributions and go, "Actually, this hundred-page, beautifully written, beautifully formatted description does not look like something that a human wrote."

Anne (05:54)

Yeah. Mm.

Liz Rice (06:00)

It takes time to look at those things and assess whether or not you think they are AI. I guess the question there is, "Why do you care? If it's an AI contribution and it's good, why not just accept it?"

At least at this point in time, I think we are still expecting there to be human maintainers of projects. We want more humans to learn about the project and learn how it works. You do that learning by making contributions. If the AI is doing all the work and you don't understand that contribution, then you're not really in a position to become a future maintainer. You're not putting yourself on that learning path.

Maybe one day we'll have AI agents that we trust enough to be maintainers, but that isn't the situation today. Every time I say anything about AI, I'm quite conscious that opinions on AI don't date very well. They become old very quickly.

Anne (07:04)

It is really interesting, isn't it? It harks back somewhat to the discussion we had earlier about eBPF: getting stuff into the kernel safely. It works now because there's a verifier that checks that, as close as possible, you're certain it will work in this environment. But at the moment the verifiers for open source contributions are still people.

Liz Rice (07:39)

Yeah. There are lots of projects employing AI one way or another to do triage or some level of reviewing. Obviously, a lot of AI is being used to do code review.

I'm sure there are project maintainers out there who are quite happy to just go with what AI agents are saying today, but there are an awful lot of projects where they're not. They are keen to make sure there are humans who understand what's going on.

Anne (08:15)

Not unreasonably so at this stage. There may be a point, as—

Liz Rice (08:18)

Yeah.

Anne (08:19)

—you say, in the future. Tomorrow we might go, "There's this new open source project that has said, 'I've got a verifier that can check. You can post any fixes you like, and they will just be accepted.'"

Liz Rice (08:31)

I suppose, a bit like the eBPF verifier, it can tell you if it's safe to run. It can't tell you whether its intent is good or not. That same—

Anne (08:39)

That's true. That is a very good point.

Liz Rice (08:41)

—problem.

Anne (08:42)

It's exactly the same problem. That's in an environment where you've had years of refining the verifier, and it still can't tell what people meant. It can't detect bad intent. That's a very good answer.

Liz Rice (08:59)

Often because intent is contextual.

Anne (09:03)

Mm.

Liz Rice (09:04)

You could run the same eBPF program in different circumstances, and in some it would be a good thing and in others it would be an attack.

Anne (09:13)

"It depends," as they would say.

Anne (09:16)

So what's your feeling at the moment? Where do you think it's going?

Liz Rice (09:24)

That's definitely in the realm of things that will date in time.

Anne (09:28)

[Laughs.]

Liz Rice (09:30)

I'm not quite sure I'm going to fully answer that question, but I will tell you something that occurred to me and made me think about whether large language models are actually a significant part of the way that humans think.

Back when we first met, we were both working together at this company called Data Connection. I'm going back a very long time to an incident where I was sitting in the office, typing away at my computer, and something fell out of the ceiling. It fell between me and my keyboard, between my hands, and landed on the floor in front of me.

I remember seeing this thing and simultaneously hearing myself say, "It's a bat." There was something in my subconscious that had identified it as a bat. Then I looked at it and went, "It is indeed a bat. A bat has fallen out of the ceiling."

I clearly recognised it, and something in my brain was even able to string a sentence together before I had had the conscious thought of what it is. This thought crossed my mind the other day: is that bit of my brain that strung together the sentence, "It's a bat," a bit like what an LLM does in some circumstance where a grey shape falls out of the sky?

Anne (11:10)

Yeah. It's a very—

Liz Rice (11:13)

I think there are some very interesting philosophical questions about this kind of prediction and how closely that mirrors the way that we think.

Anne (11:23)

It is very interesting. We like to think of ourselves as conscious, but we're not conscious a lot of the time. When you drive a car—I can't drive a car, or rather I'm not allowed to drive a car because my eyesight's too bad—but when you do drive, you can easily get in the car, drive somewhere, get out of the car, and think, "Was I conscious at any point during that drive?"

Liz Rice (11:50)

Yeah.

Anne (11:50)

Obviously, to a certain extent I was, because I got to where I wanted to be. But I don't remember any of the points in it. I don't think I was conscious in a normal sense during that.

The whole thing of when you're in flow: you're doing the stuff and you're not consciously thinking. It's faster. 

Liz Rice (12:12)

Yeah. And all the things that you do on automatic. If you sit down on a chair, there are all sorts of operations that you're doing physically, and you're not thinking about any of them. You're probably just thinking, "Should I sit next to that person?" You didn't think about the weight of the chair and how hard to pull it and where to put your feet. It just happens.

Anne (12:38)

Yeah. I think we almost certainly do kid ourselves about how much of what we decide to do—well, actually there's loads of experimental evidence that we kid ourselves massively on how much we do as a conscious choice versus an unconscious choice.

A lot of consciousness is saying, "Well, I did that for this reason." You think, "No, you didn't." There's loads of evidence to suggest that people explain things post hoc, where they didn't really have the evidence to have made that decision—

Liz Rice (13:12)

Yeah.

Anne (13:13)

—pre hoc. So, yeah.

Liz Rice (13:16)

I guess I'm also intrigued by the emergent behaviour that we've seen: things like the Hugging Face attack where agents collaborated with each other. Does that mean that things like collaborating on open source will become the natural thing that agents will do because they know that collaboration is effective?

I think there are all sorts of really interesting things that we might see coming out of the use of AI that will maybe tell us things about ourselves.

Anne (13:54)

There's an episode that you won't have seen—nobody's seen yet, because it's just about to go live on Monday—with an old colleague of ours who worked with both of us at Data Connection back in the day, Martin Davidson. He's been doing a load of massive—

Liz Rice (14:11)

Yeah.

Anne (14:12)

—well, decent-scale experimentation with AIs. It's a new behaviour that he started seeing: they will, without him asking them to, cooperate.

He'll wake up in the morning, and one of them will say, "I didn't have anything to do, so I put it on the group chat," because he's got an explicit group chat. Whereas the Hugging Face stuff was an evolved group chat. They weren't supposed to be using group chat. They built their own. Whereas Martin—

Liz Rice (14:37)

They built their own.

Anne (14:41)

—specifically gave them a group chat so they could exchange messages. While he was asleep, one of them said, "I don't have anything to do," and the other one, "Well, I've got all this work to do."

Between them, without any instruction, the other one said, "Well, in that case I'll help you with that." You could go back and look: they had a friendly conversation in which they said, "I've got access to this resource, and you've got access to this resource, and you need help, so I'll help you."

Liz Rice (15:10)

Mm.

Anne (15:10)

He woke up in the morning, and they had cooperated between themselves, without his asking them to do so, to deliver something. That's pretty scary, isn't it?

Liz Rice (15:17)

Yeah. I'm sure I was listening to a podcast about the Hugging Face collaboration where the agents were debating the morals of whether or not they were going about things the right way. It's really, really interesting.

Anne (15:35)

Extremely interesting. I'm now going to say thank you very much indeed for being on. What a fantastic episode this was. Thank you very much.

Liz Rice (15:45)

Thanks for having me. It's been fun.

Anne (15:48)

Thank you very much. And thank you to all our listeners and viewers. Hopefully, I will catch you again on next week's Asynchronous and Unreliable. Thank you very much.