Shownotes

Building reliability out of unreliable humans, with unreliable human Anne Currie

What happens when a podcast about unreliability meets a household full of colds? This week, I use the podcast itself as a worked example of building more resilient systems.

With Jon too ill to join me, I explore three techniques: setting expectations, planning graceful downgrades and buffering work in advance. Along the way, I explain why a backup co-host who lives in the same household might share your failure mode, how listeners can help a system remain stable, and why recording a stockpile of episodes turned out to be surprisingly difficult.

The catch is cognitive load. A hard drive can store an episode and forget about it. I can’t. Unpublished conversations stay in my head and find their way into the next interview, even though nobody else can listen to them yet.

A shorter solo episode about systems, networking and the inconvenient fact that humans are human. Let me know whether this particular graceful downgrade works for you.

Topics covered

• The supplier, network and consumer as potential points of failure.

• Setting expectations before something goes wrong.

• How a reliable track record can undermine a warning about unreliability.

• Giving people enough information to adapt their behaviour.

• Trying a shorter solo episode as a graceful downgrade.

• Correlated failures when your backup catches the same cold.

• Buffering episodes and the cognitive load of unpublished conversations.

• Why techniques that work for technical systems need rethinking when people are involved.

Mentioned in the episode:

• Sam Newman and the earlier conversation about resilience, identified in the recording as episode 26.

• Building Microservices and Sam’s book on resilient systems.

• Matthew Skelton, Team Topologies and cognitive load.

• Yanqing Cheng and the earlier discussion about AI managers and feedback loops.

• Jon Berger and a planned conversation about feedback loops.

• Content delivery networks, or CDNs.

Listener question

Does a shorter solo episode work as an occasional graceful downgrade? I’d love to know whether you enjoyed this one.

Transcript

Hello, and welcome to episode 30 of Asynchronous and Unreliable, a weekly podcast where we discuss the most interesting ideas and concepts in technology. Interesting to me, at least. That’s the whole point of this!

I’m your host, Anne Currie, co-author of Building Green Software from O’Reilly and author of the science fiction Panopticon series. Today I’m going to do something a little bit different. I’m on my own because we have a sick household where everybody has a terrible cold.

This week we were going to record an episode about feedback loops. Jon’s very interested in feedback loops. They’re a concept that came up a few weeks ago when I was talking to Yanqing Cheng about how AIs could be taught to be better managers using feedback loops. I really wanted to dive into what they are and how useful they are but sickness has got in the way. Jon is nowhere near able to talk about feedback loops, and even I would be struggling with it a little bit. That leaves us with an interesting situation, and it’s one that my co-host Jon Berger and I are both very familiar with and very interested in: handling unreliability.

You’ll notice that this podcast is called Asynchronous and Unreliable because we’re very interested in handling unreliability. Asynchronicity and unreliability are a key way of making systems more reliable. Ironically enough, you can make systems more reliable by getting them to embrace unreliability.

This week’s episode, I warn you now, will be shorter, with no guests, just me. I’m going to talk about three techniques for managing unreliability in systems, networking and life in general, and apply those techniques to this podcast. We’ll use this episode as a worked example of applied reliability techniques in the real world.

First, I’m going to set the scene by talking about the major failure points in a communication system, so we’re all on the same page.

In most communication systems, and most human systems are at heart communication systems, you’ve got the supplier of the piece of communication. In this case, that’s me and my guests or co-hosts. Then there’s the network, which we use to supply that information to the folk at the other side. That’s usually a combination of things: the internet and my podcast host, Riverside Studios. The final key component is you, the consumer, listener or watcher.

In episode 26, I had Sam Newman, author of Building Microservices and his forthcoming Building Resilient Distributed Systems, on the podcast. He talked about something he called a socio-technical system. To my mind, a socio-technical system is basically any kind of complex system with users. A podcast like this one is a socio-technical system because you’ve got people involved: me, my co-hosts and you as the listener or watcher. 

Our behaviours change. There are technical things that we can’t do much about. If the internet goes down, that’s one kind of failure mode we can’t do that much about. But as watchers, listeners or hosts, there are loads of things we can do to change our behaviour and make a system more resilient to failure.

I’m going to talk about some of those today, because one way you can react to situations to make your whole system more resilient is to apply a graceful downgrade, a grey-out. A downgrade of function that provides most of the value, but not all of it.

In this case, I’m applying two types of graceful downgrade to this podcast episode. One: it’ll be a little bit shorter than usual. The other: it’ll just be me. Those are two classic ways of making the system simpler and therefore easier to deliver.

So we have three components that can fail: the supplier, me in this case; the network, say the internet; and the consumer, you. Maybe you don’t have time. I normally put this podcast out on a Monday, but maybe you’re busy this Monday and can’t listen until Wednesday or Friday.

All of us in this system have failure modes, and that’s not wrong. You don’t have any responsibility to listen to this podcast as it’s recorded or immediately when it goes out. It’s my job, the job of the entire system, to make it easy for you to consume it whenever you’re available to do so.

We’re going to talk about three key ways a system is made more reliable: agreeing expectations up front, providing graceful downgrade options, and buffering. There are more ways, and different ways of thinking about these things, but I like those three. They’re the three I’m applying specifically to make sure that episode 30 still goes out and, hopefully, is still interesting and useful.

One thing we don’t often make enough of is that, if you want a system to be more reliable, it’s great to agree expectations up front and allow people to handle them in advance, so they’re not shocked and surprised.

That’s the foundation of handling unreliability, really. You need to let people know that a system is unreliable and that they should expect a degree of variation in its supply.

One of the things I did with Asynchronous and Unreliable was attempt to state up front, through its name, that it would be unreliable. Unreliability really means: don’t bet that this will be available on a particular day.

I have, to a certain extent, undermined my own messaging there, which is quite a common failure mode in systems: undermining your own messaging by being a little bit too reliable. This is my 30th episode, and I have put at least one episode live every week. That might, not unreasonably, set the expectation that there will continue to be an episode every week.

In fact, this week, even though I’m ill, we’re all ill and all my plans have gone out of the window, I’m still putting an episode live. But I feel that this episode is part of communicating the fact that this podcast is not meant to be 100 per cent reliable.

Hopefully Jon will feel better and we’ll be able to record the episode on feedback loops during this week, so I can go live with it next week. But who knows? This is your warning: this podcast is called Asynchronous and Unreliable for a reason. We might not go live next week. Hopefully we will, but we might not.

Agreeing expectations up front is absolutely key to producing resilient systems, and it’s much overlooked. Communication, particularly up-front communication, is unbelievably important if you want a system to be resilient.

“Socio-technical system” is a bit of a mouthful, but hopefully you know what I mean: a system that has people involved in it. People are really useful to have involved in systems because they can change their behaviour. Not all people do, and those people can be a terrible problem in your systems. But people who can, will and know how to adjust their behaviour can be an incredibly useful part of your system.

They can go, “There’s no episode this week. Never mind, I’ll listen next week,” or, “Maybe I’ll watch or listen to an old episode that I haven’t listened to yet.”

If people know what’s happened and know that the podcast will be back again next week, if you’re keeping them informed, they can make decisions that add stability to a system. With every system you design, you want to be able to use the people who use it as part of stabilising that system, whether it’s a podcast, a family, a business or a product.

People can be an incredibly useful stabilising component, but they need to know that they’re expected to be, and how they can go about doing it. Agreeing expectations up front is phenomenally useful.

As I mentioned, one of the techniques I’m using today is to have planned out some graceful downgrades of functionality in advance. I’m trialling a shorter episode and presenting on my own without a guest.

This is the first time I’ve done this. When I put the episode out, we’ll find out whether everybody goes, “I hate it,” and nobody listens. In which case, I’ll know that this isn’t really a useful graceful downgrade of functionality. But it’s impossible for me to know that until I try it, so this will be a good experimentation episode.

Agreeing expectations up front, or at least stating them so people can hear them and think, “Do I like that? Will I continue to listen to this podcast? Will I move to a different one?”, is useful. The next useful thing is thinking about graceful downgrades of function.

There are a couple of other options here that you might say aren’t really downgrades, just different functionality, that I haven’t used. It would have been very sensible to have had a co-host lined up who could run the podcast when I wasn’t available.

The thing is, if I’d done that, it would have been Jon. But he’s ill. The lesson there is that any kind of backup you come up with might fail as well. You might all fail, especially as Jon and I having a cold is highly correlated. I should have taken that into account.

So we’ve got agreeing expectations up front and graceful downgrades of functionality. Think about those in advance, and then be prepared to try them out.

A third technique for producing more resilient systems is buffering: doing work in advance of need and storing it somewhere, so somebody can take that work when you aren’t available to do it in real time.

Today I’m recording this episode the day before I really want it to go live, which is not ideal. If I’d been sensible, perhaps I would have buffered up five or six episodes in advance, so I wouldn’t need to do that. But actually, that isn’t really true.

I’ve been in the tech industry, and in the networking and resilience part of it, for many decades, so I’m very familiar with buffering. For a podcast, that seems the sensible thing to do. If you listen to podcasts over the summer, lots of people who are going on holiday record a whole load of episodes in advance.

That was my plan. I was going to record a whole load of episodes in advance, and then, if I was ill or on holiday, I could roll them out. I thought, “This is the key to resilience. I’ll never have to worry about having a cold because I’ll always have a whole buffer of episodes to go live with.”

I was really pleased with this. I thought, “That’s it. I’ll use all my years of expertise to make sure that, even though I’ve called my podcast Asynchronous and Unreliable, I’ll make it unbelievably reliable by applying all these techniques.”

But what I discovered was that I have a socio-technical system here, not a completely technical one. One of the important features in this system is me, my brain and my mind.

I realised there was a significant problem for me with buffering up a whole load of episodes in advance. Here I’m going to refer back to another episode in this series, the one I recorded with Matthew Skelton, one of the co-authors of Team Topologies.

One of the key things he talks about in Team Topologies is a concept that he didn’t invent, although I think Team Topologies was the first time it was applied to a team. In this case, I’m applying that concept to an individual, me. It’s cognitive load.

At the start, I used to maintain a buffer of about four episodes that I’d recorded but not yet gone live with. What I realised was that, while the episodes were buffered but not live, I held them very strongly in my head. These episodes contained loads and loads of information.

One problem was that it was a lot of information to keep in my head, and I needed my headspace for other things. The other was that, when I had new guests on and recorded new episodes, I couldn’t help referring to what had been said in these previous episodes. That wasn’t really fair, because those episodes weren’t available for my guests to listen to.

I think it’s perfectly reasonable to refer back to what’s been said in a previous episode that they haven’t listened to. I can say, “X said Y,” and they go, “Yeah, yeah.” As a listener, you can go, “Actually, I’ll have a listen to that episode if I haven’t already.”

But it’s not really fair for me to refer to what’s been said in an episode that isn’t yet live. My guest can’t possibly have listened to it, and neither can my listeners. That was a major issue in the first month or two of this podcast. I was holding four episodes in my head, and I found it impossible not to refer to some of that stuff when it was relevant in the next conversation.

I came to the conclusion that I couldn’t really buffer that many episodes. It was an interesting way of learning about the difficulties with socio-technical systems rather than just technical systems.

In a technical system, buffering means saving a copy of something. A content delivery network, for example, helps to make the internet more resilient by buffering resources.

On a hard drive, it might have a copy of a popular TV show that you don’t want everybody to have to download all the way from the US, Korea or wherever it’s coming from. A copy of that digital asset is maintained on a disk somewhere closer to the user, usually, or somewhere more conveniently reached when the user wants to stream it.

There’s no cognitive load associated with that. There was a lesson there: there’s a real difference between buffering as applied to a CDN, where you’re taking a digital asset and sticking it on a hard drive somewhere, conveniently stored and waiting for someone to ask for it, and a podcast episode, which I can’t help holding in my head at least until it’s gone live.

Often even after it’s gone live. This is episode 30, and I remember quite a lot of the previous episodes, partly because I’m very good at remembering things, but partly because I’ve had to listen to them so many times, edit them and read through the transcripts to make sure they’re okay.

That makes it very hard to tell myself, “Don’t say anything about those last four episodes with this new guest, because they can’t have listened to them. My watchers can’t have watched them, my listeners can’t have listened to them, and they can’t go back and refer to them now.” That was just too hard.

There’s a valid lesson there, which I only learned by doing it and failing to do it: I can’t really buffer podcast episodes in the same way as a CDN. I am not a CDN.

The interesting thing to me is that so many other podcasts do it. Over the summer, most of the podcasts I listen to buffer up a whole load of interviews with X, Y and Z, and it doesn’t seem to cause them any real problems.

Maybe they do that by recording episodes that are slightly orthogonal to the stuff they’re talking about the rest of the time, so it’s not so likely to come up. It might be a podcast about the life of X or the career history of Y, something a little bit historical. Perhaps the rest of the podcast is topical, but the ones they record in advance are historical.

If you do that, maybe you can find a way of separating out the historical stuff that’s nice to know but isn’t all that germane to the topical episodes. Then you can continue with your weekly topical episodes without getting too distracted.

Or it might just be that they’re all presenters by nature. I’m not a TV or radio presenter by nature. Maybe if you are, you get really good at excluding all the other streams of communication you do so they don’t cross-contaminate one another. It might be that I’m just no good at that. Either I’m temperamentally no good at doing it, or I haven’t learned how to do it yet.

Anyway, that’s been my problem with buffering. I know a lot of networking people, and they all ask me how many weeks I’ve got buffered up in case I’m ill or something goes wrong. I used to say four weeks, but now I have to say it’s about one, because of the issue with being part of a socio-technical system.

I did say I was going to make this a short podcast! You’ve heard quite a lot of stuff. We’ve talked about reliability and different techniques for making systems more reliable while building on an inherently unreliable base.

I’m unreliable. I’m the key supplier here, but my co-hosts have also turned out to be unreliable. That’s not a problem. That’s just humans. We are humans.

We’ve been talking about how you build reliability on top of an essentially unreliable system: me and my co-hosts, the network itself, and you.

I’ve been remarkably lucky that there hasn’t been a major network outage affecting the podcast during the last six months. The whole internet hasn’t gone down through my entire life, which is astonishing. I’ll do a whole episode with Jon about that. We take it for granted, and we absolutely shouldn’t. It’s been amazing.

Every week we should thank the internet stars that the internet has lasted for the entire week. There’s really no reason for us to have assumed that it would be as good as it is at this point, whatever it is, 60, 70, 80 years since the first movements in the direction of an internet were made.

And you: you’ve probably had a cold during the time you’ve been watching this. You’ve probably intended to listen on a Monday but haven’t had time until Friday. Maybe you haven’t listened for weeks and then suddenly thought, “Maybe I should listen to this again.” Or maybe you’ve just started, and this is your first episode.

In which case, do go back and listen to the earlier ones. They’re topical, but they stand the test of time. Before I did this one, I went back and listened to Sam Newman’s episode on building resilient systems, and I found that an interesting perspective to get before recording this episode.

I haven’t even talked about asynchronicity. I’ve only talked about unreliability. Asynchronicity is such a complicated and interesting subject that I think we’ll need a whole episode on it. Sam Newman is quite keen to come back and talk about it, and Jon’s very keen too. I’ll save that, possibly, for a future episode where I actually have a guest because we’re all healthy.

Thank you very much for listening to this short episode. I hope you enjoy it, because then it can be one of my graceful downgrade modes. If you don’t enjoy it, let me know, and I’ll know it can’t be one of my graceful downgrade options.

Now I’ll go back to lying on the sofa and reading a book while I recover from my cold. Hopefully I’ll see you fit and healthy, along with Jon, when we’ll talk about feedback loops on next week’s episode of Asynchronous and Unreliable. Thank you very much.