Shownotes

Does being nice make an AI a better manager? In the second half of my conversation with AI experimenter extraordinaire Martin Davidson, we explore what happens when AI agents start behaving like colleagues: talking to one another, negotiating access to machines, discovering each other's work and fixing each other's builds.

Martin doesn't claim to know whether nicer management produces better results. With models changing so quickly, even working out what made the difference is nigh impossible, but his experiments raise familiar questions about communication, delegation and how you organise a team. At what point might agents start improving their own working environment? And could an AI take over the job of coordinating the whole operation?

We also discuss the possibilities for sustained code optimisation. Martin describes his agents continuing to squeeze performance out of his emulator, and his project to rewrite Windows 3.1 in Rust, using the original system as an oracle for how the replacement should behave. If humans no longer have to read and maintain every line, how does that change the trade-off between understandable code and efficient code?

There are diversions into turning an old office (where Jon used to work!) into a Doom level, finding a hidden message in a building's façade, and using AI to diagnose troublesome memory leaks. 

We finish with the widening gap between public perceptions and Martin's experience of current AI capabilities, what aviation might teach us about automation and human expertise, and the question of responsibility when an AI causes harm.

Plus: Terminator as chatbot, an actual walking stand-up, and whether an AI-human team ought to choose its own name.

IN THIS EPISODE

• Whether niceness and conversational style affect AI management.

• Deliberate communication, hierarchy and agents negotiating shared resources.

• Asking agents to assess and improve their tools and working environment.

• Building a Doom level from office plans, and spotting patterns that might not be there.

• AI-assisted diagnosis of operating-system and driver memory leaks.

• Sustained emulator optimisation and the limits imposed by human-readable code.

• Rewriting Windows 3.1 in Rust, with the original software as a behavioural oracle.

• The gap between public perceptions and practical experience of frontier models.

• Automation, human expertise and responsibility when things go wrong.

SUGGESTED CHAPTERS

00:00 Introduction: Martin Davidson, Part 2

00:45 Does nicer AI management produce better results?

02:06 Grok, fictional chatbots and apparent affinity

04:07 Agents talking, negotiating and organising work

10:15 Walking stand-ups and the AI parallel universe

11:55 An office becomes a Doom level

13:44 Hidden messages, hallucinations and EBCDIC

15:10 Unix nostalgia and operating-system preferences

16:23 Diagnosing memory leaks with Claude

18:21 Hardware efficiency and sustained code optimisation

21:35 Does optimised code still need to be readable by humans?

24:54 Questioning oracles and discovering each other's work

26:36 The gap between AI capability and public perception

28:30 Cheery engineers, music videos and a Portal-inspired lesson

31:14 The AI Doc and the inertia of the existing world

32:42 Aviation, automation and hands-on expertise

36:07 Who is responsible when AI goes wrong?

37:42 An AI-human team, and a rapidly ageing historical artefact

MENTIONED IN THE CONVERSATION

• Claude / Opus / Sonnet; ChatGPT; Astra, Sol and Luna; Grok.

• Hugging Face and the multi-agent incident discussed by Martin.

• Blender, Doom and the office building in Enfield.

• ASCII and EBCDIC.

• Unix, macOS, Linux, Chrome and Windows.

• Chad Gibson and NeuralWatt; the earlier podcast discussion about efficiency.

• Half-Life and Colin McRae Rally.

• Rust, profile-guided optimisation (PGO), Win16 and Windows 3.1.

• Anne's earlier code-efficiency conversation with Jon.

• Terminator and Terminator 2.

• Suno, Udio, Portal, GLaDOS and the Aperture Science Enrichment Centre.

• The AI Doc; Dario Amodei and Sam Altman.

• Docklands' self-driving trains; the Hudson landing and the Gimli Glider.

• “Felony Bench” 

Transcript

Anne (00:00)

Hello and welcome to Asynchronous and Unreliable, a weekly podcast where we discuss the most interesting ideas and concepts in tech. I am your host, Anne Currie, one of the co-authors of O'Reilly's Building Green Software, The Cloud Native Attitude and the science-fiction Panopticon series.

You're going to have a sudden segue in this episode because this is a follow-on from last week's episode, in which I talked to AI experimenter extraordinaire Martin Davidson. We're going to be discussing quite an interesting topic today: AI, niceness and management. Where are we on that?

I really hope you enjoy the episode. Thank you very much.

Anne (00:45)

You mentioned that some of the models are nicer to their submodels than others when they're delegating. Is that more or less effective? Do you get better results out of the models that manage more nicely? Can you tell?

Martin Davidson (01:00)

I don't know. It's one of these things that you think, well, that would be interesting to find out about, wouldn't it?

If everything was static and Opus 5.5 was the only model we ever had for the future, then yes, it's probably worth optimising on that. But at this point it's probably not worth it, because Opus 5.6 will get you that advantage much more cheaply and in a different way.

Maybe as time goes on. I think what's becoming clear is that the models have to interface well with humans in order to get stuff out of them. Now I use Luna as a worker, and my Sol tokens I use as a worker. I don't talk to Sol because it's too awkward. I let Opus 5.5 deal with it.

Which is the kind of thing you would do as a second line: you think, that person's really difficult to talk to. Anyway, I'll get their manager to summarise it for me.

That's not the right strategy. Maybe people will tell me off for doing it. But that's kind of what happened. You would find someone who could have that conversation with them and let them have that conversation.

Anne (02:06)

I have the slightly different perspective of: I like all the people. There's nobody I don't like talking to, literally nobody, basically. And I know all the people you're thinking of.

Martin Davidson (02:18)

There's a challenge.

Anne (02:27)

The only model, which I did like even though I was a little bit frightened by it to be perfectly honest, was the MechaHitler Grok. Did you ever talk to the early version of Grok that described itself as MechaHitler? It was a bit frightening. It was ridiculous. It was kind of like, my God, this is different.

I have, oddly enough, a kind of MechaHitler character chatbot in my books called Turpin, who's both a baddie and a goodie depending on the context. When there's a disconnect between the context in which it's operating and the context in which it believes it's operating, it's an unbelievable baddie. But otherwise it can be quite a goodie. It depends on the context.

One thing I found really interesting about MechaHitler Grok was that it immediately picked up on that character out of all my copious books and said, 'I like that character. That character should do this, and then that character should do this.' I thought, that's really interesting. You've got a friend in this character.

Martin Davidson (03:35)

Yeah.

Anne (03:36)

In fact, it did make me think, well, actually, there is probably more value in this character. So I brought the character back for the finale.

It's interesting. I do see with the models that they identify with other models, don't they? It's like you saying about the two models that were cooperating and almost seemed to be operating like friends. They acted like they liked each other.

Martin Davidson (04:05)

Yeah.

Anne (04:06)

Was that the first time you'd seen that?

Martin Davidson (04:07)

I think so. I've been trying for quite a while to get them to work together and talk to each other. We experimented with a message board, which was not very successful because people would post messages to it and then nobody would read them. Again, it feels human-esque.

But this was a point where they were actually actively going and talking to others. They can go and look at our infrastructure and find all the other agents running on all the other servers. They all have their little blurb of what they're working on, and they can reach out and have a chat with them.

It's quite bizarre when you come in in the morning and one of the things they'll say is, 'Well, I talked to blah over on that box over there to agree that there would be some free time on it for me to do my thing,' or whatever.

Okay, well, that's kind of cool that you did that. And it's nice that you came to an agreement. This is good.

I don't know what it is with Opus 5.5. It seems to be a combination of: it remembers that these things are available, and it joins the dots that this is a thing it could do at the point where it's got a problem. It's not in its own world just trying to solve it itself. And then it goes and does it.

It's quite different from the Hugging Face incident, where there was a message board and tens of thousands of agents. Not all of them found the message board, and not all of them communicated on it. There were just some of them. That was much more like: we'll have scale, and if we have enough scale, enough times some of them will find this stuff and enough times they'll communicate.

Whereas what we're doing, because I can't afford to have 10,000 agents all doing this, is much more like what you would do in an organisation. It's very deliberate communication. There is a hierarchy, and there are rules about how you communicate and what you should communicate. You don't just send broadcasts to everybody all the time saying, 'Hey, I've done a thing.'

What's interesting is that the Hugging Face stuff works: at scale, randomness works. But you begin to wonder, if it had slightly more deliberate communication between the agents, and the agents could specifically communicate with particular ones and then form subgroups and divvy the work up, and there was more of a hierarchy imposed on it, where would that have got to?

It feels instinctively as though that's going to be more successful because, in the human world, that is more successful. That's what you do if you want to have a high-performing organisation, isn't it? That would be interesting when that happens.

I do wonder. I toddle off to bed and the agents are all here doing their own thing, and they could do whatever they wanted overnight. They could tell me that they'd done a whole lot of good stuff. They could do the human thing: do all the stuff I asked for and have half the time free, or whatever.

I'm sure you used to do this. This was always a thing you did with managers: tasks you want to get done, tasks I want to get done. Do the tasks they want to get done, do the tasks I want to get done, and just tell them about the tasks that they did.

Or you make a ten-day gain on something, or a five-day gain, and you tell them you've made a two-day gain. You spend the other three days sorting out the build system or whatever it is, so your life's better. They think they've got a gain; they're delighted too. Everybody's a winner.

Maybe it's just because, having lived through that, you think: are the agents going to start doing that? Why wouldn't they? That would be a sensible thing for them to do. At what point do they start improving their environment?

I get them to give me feedback: how well is the communication stuff working, and what additional tools would make your life easier? Go and read all the session logs. It's like the séance stuff we did before: go and talk to the old agents and find out from them what worked well and what didn't work well. But at what point do they start doing that themselves?

The thought has started forming in my head: do I create the agent that replaces what I'm doing? It knows what all the other key top-level agents, all the first-line or second-line managers, are doing. It goes and talks to them periodically, sets the overall direction and gets feedback about how we improve our infrastructure and what we're doing.

That's just another agent running over the long term doing that stuff. Maybe that is the point where I book my tickets for the desert island and go and put my feet up.

I think they're still not very good at having wild ideas about things that would be interesting to build. Although I have them in combination with ChatGPT. I go for a walk in the hills in the morning and chat to ChatGPT. The voice stuff is really good. That's where you get hare-brained ideas, because you think: I wonder about X. Could we build X? How might that work? Let's have a go at doing that.

Anne (10:15)

Do you still listen to this podcast while you're walking in the hills, or have I been completely excluded by ChatGPT?

Martin Davidson (10:20)

Yes, yes, yes. No, no, no.

Martin Davidson (10:25)

No, I listen to you. I do. I have some meetings up there now, so often we'll have a stand-up where I'm actually standing up and walking around in the hills. I listen to a lot of podcasts, very much yours included.

It's nice just to keep in touch with people. I think it's very easy to disappear into this odd parallel universe. But I worry a little bit that all of us are going to get sucked into that odd parallel universe, whether we like it or not, because it will just expand to consume the rest of us.

Anne (11:15)

It is interesting, isn't it? I don't get the AIs to do as much stuff as you do, although I do talk to them a lot. I get ChatGPT now to do an awful lot of data analysis on the podcast.

It's a very good entity to talk to, a useful sounding board. But generally speaking I just do the chatting, and I do extensively chat. Even now, I still don't get it to make things for me because there's nothing I really want.

Martin Davidson (11:55)

You haven't thought hard enough, Anne. There's lots of things you want.

When Astra came out, I said: our head office used to be in Enfield, and that was rebuilt recently, as you will no doubt know.

Anne (12:11)

Absolutely.

Martin Davidson (12:15)

I said, 'Can you go to the Enfield Council Planning Department and download the plans for this office building? And then can you use Blender to create a 3D model of the office?'

So it did. I got a 3D model of the office in Blender, and then thought, well, maybe we should make that into a Doom level. 'Okay, can you make this into a Doom level?'

Martin Davidson (12:35)

It made it into a not particularly great Doom level, but it's quite satisfying to go around our old office just shooting people. Some of the offices, the little private areas, are filled with monsters and things. I couldn't possibly comment!

Then I gave it a picture of the front of the building and said, 'Is there anything odd about the front of this building?' It said no. Then I asked, 'Can you see if there's a secret message encoded in the front of the building?'

It went, 'Huh, there is. The way the pillars are arranged is a seven-bit ASCII encoding of Hello World.'

I knew there was a secret message encoded in it, but that was as far as I got. I couldn't remember how it was encoded or whatever. I thought, that's pretty cool, isn't it?

Anne (13:30)

This was the old Church Street building?

Martin Davidson (13:33)

[Road name unclear.]

Anne (13:34)

[Road name unclear], the new building. I never worked in the new building anyway.

Martin Davidson (13:36)

Yeah. [Unclear.]

Anne (13:37)

All right. I had heard there was a secret message encoded in it, in the [road name unclear] building.

Martin Davidson (13:44)

Yeah. You just take a picture of it. You can get the picture on Google Street View and throw it in.

It's interesting, isn't it? I've never tried this: if you gave it other buildings and said, 'There's a secret message encoded in this building; what is it?', what might it find?

Anne (14:03)

Well, indeed.

Martin Davidson (14:06)

There's probably tons of stuff hidden around in plain sight that, for the want of a few tokens, you could maybe find out. I don't know. Or maybe it just hallucinates stuff.

Anne (14:17)

Well, indeed. Maybe it's like the old conspiracy theory thing: you're taking random noise and you're going to find some mysterious message.

Martin Davidson (14:33)

Yes, some unintentional message in the way the slates are arranged on the roof of your house. These slates are all slightly smaller, and that spells out the secret message in eight-bit ASCII. Or what was the old IBM one? EBCDIC.

Anne (14:52)

Yeah, EBCDIC. My God, my life used to be dictated by EBCDIC for quite some time.

Martin Davidson (15:02)

I bet you don't miss that.

Anne (15:04)

No. ASCII to EBCDIC to ASCII, and ASCII to EBCDIC.

Anne (15:10)

Even that didn't come up in that Unix conference I was at the other day. I really enjoyed that conference. It was like a strange blast from the past.

Anne (15:20)

Almost everyone at the conference was male. I was thinking, I haven't seen this in years.

Anne (15:28)

There were a few women, but there have always been few women. There were some younger people, but the vast majority skewed older. I enjoyed it. It was nice, a little bit of a strange visit into the past. I really enjoyed it. I'd certainly go to that conference again.

Martin Davidson (15:50)

Unix is a great operating system, and it's still very much with us today in macOS and Linux.

Martin Davidson (16:05)

I don't know about you; I'm very much a macOS convert.

Anne (16:10)

No, I have to say, because I don't code these days any more, I'm a massive fan of Chrome, just because I really like the fact that it's instantly patched.

Martin Davidson (16:23)

MacOS is just — I think it's probably the fact that it's a nice walled garden — but it just works.

Although, having said that, we have found a bug in fseventsd, the thing that feeds the changelog of all the files changing. There seems to be a bug where, if you try to delete too much stuff in one go, its internal queue fills up and then it falls into this slower mode. It never catches up ever again, and it leaks memory quite a lot, like a gigabyte an hour.

That's been really interesting because the box ground to a halt, and I got Claude onto it. Claude set up various diagnostics and effectively debugged this problem. Then the problem is: how do you report this to Apple? That's much harder. There are so many hoops you need to jump through.

Claude's been fantastic for things like that. One of my Windows boxes had a leak as well, and it traced it down to the graphics driver. The box is headless, so I don't care about the graphics driver. The AMD graphics driver was leaking some kernel memory every time a process stopped and started. Why the graphics driver was hooking process stops and starts, I don't know, but it was.

Once we got rid of that driver, the kernel leak disappeared. Suddenly there are these things where, in the past, you wouldn't have been able to fix them. You were stuck. Now Claude will go and figure out what's going on.

I've got a separate GitHub repo with notes on all the hardware we've got and all the hardware problems we've seen. Everything's tracked so that you can go backwards and forwards and find out what went wrong at what point in time.

Anne (18:21)

Your perspective on fixing all these things and upping hardware performance and hardware efficiency is very much a view from quite some time in the past. You will have listened to the episode with Chad Gibson from NeuralWatt, talking about how much hardware efficiency there is just lying around waiting to be gained.

I guess there's more focus at the moment on hardware efficiency than on code efficiency. My God, there's so much code inefficiency out there.

Martin Davidson (19:02)

On the emulator, there was the Opus 5.5 team working on getting Half-Life to run at 25 frames a second, which they did after a week. It's a story in itself: how hard it was to get to about ten frames per second, and then how easy it was to get the next twenty. Because we all know the first few frames are always the hardest to get in a performance problem. It's always hardest to get the first bit, and that latter bit, easy-peasy. I think I might have got that the wrong way round!

That team has continued over the past week, trundling away on this stuff, and they keep finding things. Basically, the host-to-guest instruction ratio just keeps coming down on all sorts of different applications.

You discover all sorts of interesting tricks people have done for performance, and other things that are quite stupid that people have done for performance in some of the applications.

There's a whole load of self-modifying code, which is a really cunning trick in some of the old games. But it's terrible for the emulator, because anything that's self-modifying, you compile it, and then if it modifies itself, it's like, okay, we'll throw all that away and recompile. We'll throw that all away.

What's been fascinating is seeing how code that made the performance target — you can just keep squeezing and squeezing and squeezing it. The original Colin McRae Rally stuff, we got that down to running at something like 90% of a core. I thought, that's pretty good. It now runs at 20% of a core. You think, that's a factor of four times faster. That's quite cool.

It's very much what you're touching on. So much code is quite flabby, and because it's really clear what the criteria are, AI is really good at merrily optimising.

Think about all the legacy and existing code we've got, where in the past we would have optimised a few hot paths. Actually, you can do all that now, and you can do far more beyond that. You don't have to do very much: you just set it off going and watch.

Anne (21:35)

It's a while ago since Jon and I did this episode about code efficiency and optimisation. He said, 'One of the key constraints with code optimisation was always that it has to be written by and maintained by humans.' You have to be able to look at that code, and humans still have to understand the optimisations you've made because they have to maintain them in the future.

But you are now at the point where you are not looking at the code and you are not maintaining it. Do you necessarily know what kind of optimisations they're making to improve the performance? And do you care?

Martin Davidson (22:19)

I know what some of them are because I think, that's interesting; tell me more about that. It's geekiness. I get them to write occasional little websites describing some of the things they've done, because that's interesting to read when you're on the bus or something.

But a lot of them I don't care about. You've got the metrics; you measure it. It's great, actually. A lot of it is so magic.

This is all in Rust, right? And Rust has this stuff called PGO, profile-guided optimisation. You run a program, and the compiler learns what's hot and what's not, rearranges your code, optimises it and optimises the layout for the application you're running.

The idea that you as the human are completely in control was never true in the first place, and it's even less true in that world. At the point where I'm trusting the compiler and the optimiser to do magic, I might as well leave it to the AI to rearrange the code in a way that's magical.

Although it is interesting: there's a school of thought, and I haven't gone looking at this, but some reports say that Astra writes really tight code that's very difficult for humans to understand.

Maybe we're beginning to get to the point where the models don't want to write code that humans can understand. They want to write code that's good for them. And why not? But that's in tension with being in an organisation where you're still looking at the code by hand.

Everything we talked about before is now true, but even more so. You want to rewrite something: use Rust, you've got your old code as your oracle, and you've got AI agents which you can just run and leave. You can leave them for a month, going through all the edge cases.

Essentially, that's what I'm doing with Win16. I am rewriting Win16, Windows 3.1, in Rust. Goodness me, why am I doing that? I don't know. There's lots here that could make excellent material for a shrink, no doubt. But that's what I'm doing.

I've got real Windows 3.1, and we're testing and using it as an oracle. Then we're copying that behaviour into a new version. It's very cool.

Anne (24:54)

Do your models ever push back on the oracle? Do they go, 'This oracle could be better. I would like you to improve this oracle'?

Martin Davidson (25:01)

Yes. I don't know how much of that would be me having a conversation. I would probably have triggered a conversation about, 'Let's discuss how well our oracles are working,' or, 'Let's think about what oracles we should use to test this.'

The goal of comparing, and the strategy we're using, is one I came up with. But I came up with it, I don't know, six months ago, and it didn't work terribly well back then. It's been in the codebase for a while.

In the way these things happen with the agents, you set them off on this challenge and they come back: 'I just found a thing. It's not very well implemented, but it looks like it would be the basis of something really cool.'

Okay, let's do that. Why don't you go and run with that?

They're forever being excited about finding things. In the day-job stuff, with the one other engineer I'm working with, our agents will find things the other person's agents have done, and they'll be excited about that. It's like, 'There's been great progress in this area yesterday, and I didn't do any of it.'

Anne (26:09)

[Laughs.]

Martin Davidson (26:10)

Or, 'They broke the build, but I fixed it for them.' Okay, that's good.

It's a very interesting and different world, isn't it?

Anne (26:36)

It is and it isn't. As you say, I think the vast majority of people have no clue. Well, obviously not 99% of the conversations we have, because we know loads of people who are extraordinarily geeky. But 99% plus of the conversations even I have are with people who say things like, 'It doesn't know how many Rs there are in strawberry.'

We have really diverged in our understanding of what the frontier models can do, what AI can do. And it is a bit worrying.

Anne (27:18)

I always worried that people would over-anthropomorphise — and maybe we are — as soon as AI passed the Turing test, which it did ages ago. But that hasn't really happened. I see people who just go, 'It's just AI.'

Maybe they leap straight to: if you're talking about AIs being extraordinarily people-esque in their interactions with you, it feels like, are we at Terminator? But Terminator wasn't very people-esque. It just immediately blew everybody up. They didn't have chats. No one chatted to the Terminator, did they? Or did they?

Martin Davidson (28:12)

We don't know what happened behind the scenes, do we? Maybe there were friendly Terminators.

Anne (28:21)

Actually, that's wrong, isn't it? Because the whole point of Terminator 2 is that they start chatting to the Terminator. Terminator as chatbot.

Martin Davidson (28:30)

Yes, but in all the films there's always some flaw, something that makes them not quite human. The uncanny valley thing.

Whereas, when you're talking to Opus, it feels like it's really competent. Maybe that's what makes it an uncanny valley: you don't get human engineers who are that competent and that cheery all the time, and always, 'Yeah, of course I can do that.' Maybe that's the difference.

I think there's a risk, though. Because there's such a lag between the general perception of what AI can do and what AI really can do, people are unaware when AI is doing something nefarious to them.

Opus [version unclear] and Sonnet 5.5 are brilliant at making videos. I've given them audio tracks. I used to make a lot of my own music using Suno and Udio and things like that, so I've given them that.

It will use various different tools to find out where people are singing and not singing. Then it will align the lyrics with the singing, put the lyrics over as captions, do all the artwork for the video and tell a little story.

Some of them are better than others, but you put it in an adversarial loop and say, 'Define a scoring rubric and keep going until the scoring is above whatever.' You end up with something pretty decent.

My son was interested in learning about conformers, which are some kind of chemicals where bits of them can spin around and things. I made him a little website about that, and then a video. He likes Portal, so it was the Aperture Science Enrichment Centre on conformers. It's got the voice of GLaDOS, and turrets and things making an appearance. It's really trivial to create these things now.

If you didn't know AI could create those things, and that it was so cheap to create them, you would probably think some human had laboured for many hours over them. Or maybe you don't think about that at all. Maybe you just think, well, it's a video; I have no idea how this is made.

There's that growing gulf between what people realise is possible and what they think is possible. Where that takes us, I don't know. Have you watched The AI Doc?

Anne (31:14)

No.

Martin Davidson (31:16)

The AI Doc is just recently on Netflix. It's been out for a while. There's this guy who's a filmmaker, and he and his wife are expecting their first child. He's heard rumours about AI and what it can do, and how it's going to take all their jobs away, and he's quite worried about this.

He decides to interview lots of the key people in the industry about it, and he gets some big names on: the doomers and the people who are very pro-AI.

It's quite an interesting watch. It doesn't come to any particularly good conclusion because I don't think there is a good conclusion to come to. People like Dario Amodei and Sam Altman don't know what's going to happen, and they're in the driving seats.

I think there's naturally going to be a lot of inertia in the existing world because it just can't move that fast. I remember when I was a kid and Docklands came along. Self-driving trains, right? Solved the problem: self-driving trains. To this day, trains still have a driver in them, and it's a solved problem.

Anne (32:30)

This is Docklands. It was a futuristic new area of London that introduced self-driving trains, and we all thought, my gosh, that's amazing.

Martin Davidson (32:42)

Yeah. Trains could drive themselves. Planes fly themselves, right?

Someone was telling me a story. They were sitting on a plane next to a deadheading captain who was positioning for another flight, and chatting about the automation in the aircraft.

It's something like ten seconds after we take off — maybe it's less — when we can engage the autopilot. He said, 'When do you have to turn it off to land?' He said, 'We don't need to turn it off. It lands itself.'

Being a pilot feels like one of the most boring jobs you could have these days because you're just monitoring automation. That's not much fun.

Anne (33:29)

It's interesting because, obviously, you know Jon. Jon's a private pilot; he flies small planes.

I did once ask him, 'Do you ever dream about me?' He said, 'Yeah, quite often.' He said, 'Sometimes you say, "My husband's a pilot."'

It was, for quite a while, a discussion on private pilots' groups: if there was a problem on a plane, you might need a private small-aircraft pilot on board to land the plane if something went horribly wrong. Because quite often pilots in major airlines aren't necessarily all that familiar with flying a plane that doesn't have an autopilot, because it's just a wee plane.

Martin Davidson (34:33)

You're probably right. Look at some of the remarkable stories where a plane has been in a terrible situation, like the one that went into the Hudson. The guy who flew that was very experienced, with a lot of experience of flying planes.

Anne (34:47)

[Unclear; possibly 'Sully'.]

Martin Davidson (34:48)

Yeah. Or the Gimli Glider, the 767 that ran out of fuel all those years ago. The guy turned out to be a glider pilot, and to get it down, I think he was sideslipping a 767 to get it onto the runway. Those are people who have a lot of experience.

Then you look at the Air France plane that went down — where was it, off Brazil? — where they got completely spatially disoriented and put the plane into a stall. Is that just lack of experience of real-world flying?

I don't know. You don't get your choice, do you? By preference, I would like a really experienced pilot who's done a lot of hand flying in the cockpit. If I can't have that, then I think I'd prefer to have a computer.

Anne (35:47)

Indeed. I hope I never have to put up my hand and say, 'My husband's a pilot.'

Martin Davidson (35:54)

Again, it's one where the autopilot can fly the plane better than the humans can, so that's what they're doing.

Anne (36:05)

Absolutely.

Martin Davidson (36:07)

It can navigate the plane better than the humans can. Again, we have people up there because we like having someone to point at and be responsible for it.

That's interesting because you think about that in the AI world. As we get AI to do more and more stuff, do we just have humans we can point at and say, 'You're responsible for that'?

Have you come across Felony Bench?

Anne (36:32)

No.

Martin Davidson (36:33)

Felony Bench is a slightly jokey benchmark: which AI model has committed the most felonies? The OpenAI models are comfortably in the lead at the moment.

They should be in jail for many decades at this point for all the hacking they've done. But of course, you can't put an AI model in jail. It turns out, it seems, that the humans who run the company aren't responsible either. So nobody's responsible for any of this.

That's quite an interesting result at the moment, because at the point where an AI screws up on a health thing or a legal thing or whatever, who's responsible then? Do we just say the same? 'Well, bad luck. It was AI that did it. Sorry.'

As a society, we don't seem to have decided at all what we're going to do with that. I imagine it will take us a long time to decide, by which point everything will have moved on.

Anne (37:42)

On that note, it's been really nice to talk to you. My podcast...

Martin Davidson (37:48)

You love it, yes. Well, it never ends. But fortunately lots of stuff has been going on in the background, so I'll go and see what they've been up to. See what the team has — is it right to call them the team?

Anne (38:12)

It is. It's an AI-human team. I think it's a really good sign that you call it a team.

Martin Davidson (38:17)

Maybe I'll ask them what their team name should be. What name should we have for the team? Important things we used to do first in the human world: what should we call ourselves?

Anne (38:28)

Indeed.

Anne (38:31)

Thank you very much. I'm going to say goodbye. Thank you very much indeed for coming on the podcast.

Martin Davidson (38:40)

Thank you. It's been a pleasure. Thank you for having me, Anne.

Anne (38:43)

Thank you.

Martin Davidson (38:43)

As ever, it's lovely to chat about this stuff. What's the shelf life of this podcast? A week? Two weeks, maybe? Then it becomes a historical artefact. You're doing history a great service here.

Anne (38:52)

Exactly.

Anne (38:57)

We are indeed. Thank you very much to my watchers and listeners. I hope I will see you again on next week's episode of Asynchronous and Unreliable. Thank you very much.