The Change Resilient Podcast Episode 5: Courtney Nash on Sociotechnical Systems and the Pure Joy of
Human Learning
For episode five, I'm joined by the amazing, thoughtful, fun, and precise, Courtney Nash.
Courtney is the co-founder and CEO of The VOID, where she researches system safety, resilience, and failure in complex sociotechnical systems. She’s also vice president of the board of the Resilience in Software Foundation.
A former cognitive neuroscientist, Courtney has spent much of her career exploring how people learn, make decisions, and respond when complex systems behave in unexpected ways. Over the past two decades, she’s held roles at organizations including Verica, Fastly, O’Reilly, Microsoft, and Amazon. Today Courtney is one of the most influential voices in resilience engineering and modern incident analysis.
Listen on Spotify Apple, Amazon, YouTube, and wherever you enjoy your podcasts.
Links from show references include:
Resilience in Software Foundation
Paper: Detecting and Preventing Latent Risk Accumulation
Show transcript (slightly edited for clarity, machine transcribed, apologies for errors):
[00:03] Steve Semelsberger: Hello, everybody! Welcome to the Change Resilient Podcast. My guest today is Courtney Nash. Courtney's the co-founder and CEO of The Void, where she researches system safety, resilience, and failure in complex socio-technical systems. She's also Vice President of the board of the Resilient and Software Foundation. She's a former cognitive neuroscientist, and she spent a good chunk of her career exploring how people learn, how they make decisions, and how humans respond when complex systems behave in unexpected ways. Over the past two decades, she's held roles at organizations including Verica, Fastly, O'Reilly, Microsoft, and Amazon. Today, Courtney is one of the more influential, maybe the most, we'll see, influential voices in resilience engineering. She waved me off when I said that with a big smile. So, welcome to the Change Resilient podcast, Courtney. It's great to have you.
[00:56] Courtney Nash: Thank you so much, it's a pleasure to be here.
[00:58] Steve Semelsberger: So let's dive in. So when you use the word resilience as applied to complex software systems, the stuff that, you know, we talk about and you and I often focus on, what do you mean by the term resilience?
[01:13] Courtney Nash: Yeah, so I am going to say that I am cribbing from a longer history of… researchers in this domain who have been at this a lot longer than I have, and my favorite person to quote when I'm asked about this is Dr. David Woods, who is, a researcher at the Ohio State, I don't get that wrong anymore. Yeah, and, he is, in part, known for his theory of graceful extensibility, which is a very nerdy academic way of talking about what we mean when we say resilience. And so I would say the… CliffNote's version is that resilience is the ability to respond to and adapt to and recover from unexpected happenings. Disruptions, when things don't go the way you expect them to go. So, it's not just having a failover, it's not just having a disaster recovery plan, right? It's not auto-scaling, it's, an… there's words that we use sometimes very similarly for those, which I have a feeling you're gonna ask me about.
[02:31] Steve Semelsberger: I might…
[02:33] Courtney Nash: Yeah, so it's very distinct from other things that we think about in technology, in terms of, like, site reliability engineering, for example, which has another R word in it. And I think, in particular, because resilience, that ability to adapt, is a… Particularly in socio-technical systems, software world, it's a very uniquely human thing.
[02:59] Steve Semelsberger: Yes.
[03:00] Courtney Nash: One of the key things about resilience versus, like, reliability, for example.
[03:04] Steve Semelsberger: Yeah, so that's a great definition. I wrote down graceful extensibility. Shout out to David Woods, and one of my other guests, as you know, is John Allspaw, and he thinks about resilience a whole lot, John does. And so, what would your definition of reliability be?
[03:22] Courtney Nash: So, I mean, if you think about that word, I think the colloquial use of the word is a fairly accurate one, but we've tended to expand its boundaries too much. Right? So, like, reliable… reliability, the state of being reliable, is doing what you said you would do. Is a thing doing what it said it would do. So, your friend who doesn't make your lunch dates regularly would not be considered to be reliable, right? Something you feel like does what it says it's gonna do. You can depend on it. It's trustworthy. Now, I'm… I'm anthropomorphizing a little bit, maybe, but I think that it's this notion that it's… the thing is going to do what it says it will do, or that we say it will do, right? Things… We're in a weird world now, where things… We attribute feelings to things now, or cognition to things, but it's just performing as expected. And then… that can… we've, I think, expanded the boundaries of what that actually means, but so, the narrowest definition for me is that it does… the thing does what it says it does, and it's the what it says it should do, or what we say it should do, where we get into trouble, I feel like, right? Because in a very simple system. X happens, Y happens. Being able to say it's going to do what it does. you can model that pretty well, you can expect what's going to happen, but that's, like, not how our software works anymore. So, people think, for example, we have five nines of reliability, right? five nines in what environment? Given what constraints, given what other things might happen, right? So… I'm not saying reliability is impossible. in our environment, but I'm saying that we, we… We expect more of that word than I think we can deliver on in the current world of how software works.
[05:18] Steve Semelsberger: I like the way that you broke it down. I once worked for a startup, and the values I still remember, they were the three I's. Intensity and intelligence and integrity. And integrity for that company was not a moral judgment. It wasn't about doing right, it was actually the way you described reliability. it was doing the thing that you said you would do, because that's often very, very hard for various systems, including we humans. So then, what about… what about robustness, Courtney? That's another word. People like robust systems. How does that relate in this lexicon that we're playing around with?
[05:55] Courtney Nash: Yeah, so robustness is… I like to think of as having defenses, right? Or having capacity planning for me, and this is John Elspot, I'm… I'm looking at his book on my bookshelf, from when he wrote it and I worked at O'Reilly Media. I feel like capacity planning is the quintessential act of trying to create robustness, right? You think, what are all the ways we could quote-unquote, make this system reliable, right? So, robustness is an attempt at making things reliable. We will have autoscalers, we will have, backup, you know, data centers, or we'll be able to have failovers to different zones and regions and availability zones and all of that kind of stuff, right? Again, within the known. So, right, so robustness exists within what we think we can know about a system. So we're gonna add this much because historically, we've seen traffic peak over the holidays up to this level, like, we want to handle that. We want to… when, you know, when U.S. East 1 goes down, we can fail, like, so it's… extra defenses or extra resources, if you will, knowing that things will not go as planned, but within, I would say, the boundaries of what we can know. Right? You could take robustness to the edge of the Earth, but these things exist within constraints. of organizations that have budgets, and, you know, boards of directors, and stockholders, and… right? So, robustness also exists within constraints. I think that's one of the other really important things, is that you could theoretically make a system, I don't know, potentially, infinitely robust, but nobody has the money for that, right? Or, you know, the ability to really do that. So that's the other piece of it, is I think it's defenses, resources, extra capacity within a known set of constraints. And once things are outside the known, Robustness ain't gonna help you.
[08:04] Steve Semelsberger: Yeah. Do all systems inherently have some degree of latent risk of things going wrong?
[08:11] Courtney Nash: All complex systems do.
[08:13] Steve Semelsberger: How do you figure out the degree of latent risk?
[08:19] Courtney Nash: You… Can't? People hate that answer, right? We… there are degrees of risk that you can try to figure out. Latent, that the very word you used is what's fascinating about that.
[08:35] Steve Semelsberger: Yeah, there's a group of researchers who have published a paper, where they built a formula for a latent risk index. you know, where they're attempting to look at a variety of different contributing factors, and I think they've observed something like 200 or 300 different software systems, and they've built a formula. And I'm not an expert, so I need to go check on this. You're arguing that latent risk by definition, is non-calculable.
[09:03] Courtney Nash: I… I… I think there will be too much certainty about that, then, compared to reality. I mean, all models are wrong.
[09:12] Steve Semelsberger: Yes, they are.
[09:14] Courtney Nash: Some are just more useful.
[09:15] Steve Semelsberger: Well put.
[09:18] Courtney Nash: So… I think the interesting thing about people who research in this space, who play in this space, who understand this space in a way that other people don't, is they are… I wouldn't say comfortable, but they are willing to accept that in… The nature of these kinds of complex systems, you cannot fully model what will happen. That is the very nature of the way they function, and the way… at this point, the way we've designed software to work, and we didn't design it to… be that way. We designed it to make money, and to move fast at making money. And so we could have theoretically designed systems that were not the way they are now, but we didn't, and so here we are. But then we want to apply social models, economic models, business models to that, that have some inherent expectation of certainty. Right? And like, so the… the way I like to describe it is, like, when I started in this industry at working at Microsoft, like, we shipped stuff on a disk. That's how old I am, right?
[10:35] Steve Semelsberger: I'm there with you, my first startup, I have my CD-ROM framed, it was a one-year anniversary, it was called iChat, so we were building, you know, all sorts of message boards and real-time communication stuff, Web 1.0, and shipping CDs, Courtney, I'm there with you.
[10:51] Courtney Nash: And what happened in that environment was still more complex than, obviously, what had come before, and before computers, and before software, but that compared to now is so wildly different in terms of how much you could say about what that system was going to do, and how much control you had over that, and like, once you shift it, nothing else really touched it, right? And these things were barely touching the internet, even, when we had them. And so… and the mental models we still have in our industry, Like, for most people, especially on the business side of things. come from that… that… that manufacturing model, right? A linear… for lack of, like, Taylorism world. But our systems don't function or exist in that. And I often get asked about, like, metrics, like, what should we measure? What should we measure? And I was like, well, you are trying to… measure a system with metrics that don't reflect a system like that very well. And so these very rigid expectations of business metrics and complex software systems are not good. Bedfellows, that's a weird word, but…
[11:58] Steve Semelsberger: Yeah, I got your bedfellow approach, too. So, you sound a little bit like a student of Zen, or maybe a practitioner. Have you studied Zen?
[12:07] Courtney Nash: No. No, I am a… I am a psychologist wet-wear neuroscientist by trade, right? Like, even David Woods gives me shit for this. Can I swear on your podcast?
[12:18] Steve Semelsberger: Oh yeah, you can swear.
[12:19] Courtney Nash: Okay. Because…
[12:21] Steve Semelsberger: Yeah, parents, if you're driving, there's little listeners in the car, apologies.
[12:25] Courtney Nash: I'm bad, but but, like, even David would give me crap for this, because he'd be like, oh, you black box psychologists, you'd put people in a thing, and see what they do, and think, you know… and I was like, you know, fair. I really do kind of come, you know, from that kind of a world. But I don't think… it's necessarily Zen-like. I mean, if you have to try to approach a Zen perspective when dealing with systems like that.
[12:50] Steve Semelsberger: Yeah, these words, like, words can mean many different things. There's a particular, story that's told, where there's the student, and then there's the old sage, and I always imagine, like, there's a mountain or something in the background, you know, and the old sage…
[13:05] Courtney Nash: ...seeing uncertainty, I feel like, is a shared thing across those two, perhaps?
[13:11] Steve Semelsberger: Bingo, that's where I was going, and the thing is, you know, the words we use of the thing, the models, are not the thing. So the quick story, and the reason why I asked, was that, you know, there's the wise master, and there's a set of students, and there's a wooden pitcher of water on the table. The master says, what is this? And, you know, one student says, well, it's… it's a pitcher of water, no, it's wood, it's held it together, no. And the other student tips over the pitcher, and the water spills everywhere, and the Zen Master smiles. Because it's the thing, it's not the model of the thing, or the language, or the concept of the thing. And I think this is a lot of what you're getting into. And it helps me kind of see this connection as a cognitive neuroscientist, as to how maybe you found yourself in this field of resilience. Was it the human journey of resilience, possibly, Courtney, that kind of led you to this? Is it the complexity of systems? You know, and how, you know, the brain may operate kind of like some systems? Or… tell me a little bit about your background and how you got to where you are today.
[14:18] Courtney Nash: I mean, the real answer is pretty funny and pretty silly. The real answer is when I went to work at O'Reilly, and I could get how I got to there from here or from there. And I… and I was working for a… I was working on a series of books, called the Head First series. that had a cognitive neuroscience underpinning to how they were designed. That's how I got to O'Reilly. None of this was on my radar yet. Literally none. And then one day, they were like, hey, we need an editor to go cover this DevOps stuff, and could you just go to this conference? And then I looked, I was like. System Administration?
[14:55] Steve Semelsberger: Yeah.
[14:57] Courtney Nash: Aw, man, I don't know. Right? I was like, I don't know anything about that. Like, ha. I was literally told, go do this. Okay? That's how I started down the rabbit hole I'm now in. And so, I… I mean, I do have a long history of picking up new things, and career jumping, and topic shifting, and I like to learn about new stuff, so I went and learned about what the heck DevOps was, and I actually really fell in love with the way that came to be, and I've obviously met John Alspot at that point, we co-chaired the Velocity conference together, but what happened during that time was… DevOps was really focused on the human aspect of software and software development, right? And how people communicate, and how people learn, and how they share that learning, and that… was very squarely related to my academic background, and sort of acquisition of expertise, if you will, for a way of describing what I was particularly interested in. And so then I was like, wait a minute… like, all these people in the software industry have no idea about how any of this kind of stuff works, and so that it was a sheer happenstance, and great luck. I still, you know, consider myself to have found that place and that community, to have met John, and then that… we all trundled together with John down this rabbit hole of him discovering this world that I already knew about, right? Of cognitive systems work, of cognitive science, of… human factors, where I was like, oh, wait, you guys, this is new… oh, this is new to you? Oh, oh, okay, cool, cool, we can do the thing, we can, like, you know, and so… and that led us to researchers like David Woods, and Eric Holnagel and Sidney Decker, and names that I will… would mention throughout this entire conversation. Nancy Levison and Lassain Bainbridge, all these people who've been studying this work in other domains. And so, happenstance, that's how I ended up here, and it just was a… a perfect fit.
[17:05] Steve Semelsberger: Such a good fit, and part of your work is with The Void, which we'll talk about in a minute, and then part of your work, and I know you do many, many things, but at least a chunk of what you do is the Resilience and Software Foundation. Tell our listeners a little bit about RISF. Is that how I say it, Courtney?
[17:24] Courtney Nash: Yeah, I know. We were all like, wow, this is a really bad acronym, but we don't have any, you know, like, we don't have anything better. There was a group of folks, John Allspaw was one of the original, you know, sort of founding members that were in this community called Learning From Incidents. That was called LFI. It runs off the tongue way better than RISF, but this growing group of folks who were recognizing how valuable software incidents were for learning about how systems work, for understanding, you know, complex software systems, all this stuff that we're talking about more and more now. And that community just sort of morphed and shifted, and, you know, sometimes, like, someone starts a Slack, and then the Slack doesn't, like, and then they move on. like, it was kind of one of those situations, but… so, one of the other very active members of that community, was Colette Alexander, and we just hit a point where we were like. We need something to talk about all of this stuff that is… going to hopefully persist longer than any of us, or our job affiliations, or, like, whatever's going on, and so she actually started RASF, the Resilient and Software Foundation, and I was her partner in crime to most of that, but she… Colette really deserves the lion's share of the credit, like, all of the horrible parts of, like, starting a nonprofit and doing all the forms and figuring that all out, was definitely all Colette. But we wanted a place That could grow, and that we could start to have these kinds of conversations around resilience engineering, which is a domain outside of software, right? There is a Resilient Engineering Association that has their own conference, and their own thing. These are people who are interested in, like, safety and aviation, and nuclear power, and shipping, and electric… you know, grid stuff, I mean, like… truly safety-critical industries, nuclear power work, which is where David Wood's Three Mile Island got his start, we wanted to bring what we were learning from that community into the software industry and have it be something that can grow as an organization, and hopefully really have an impact on the industry. I will say most of us are of this, like, we started when you were shipping software on CDs, and we're all like, wait a minute. This… this didn't work out how we were hoping, right? And now this thing's running the world, and we want to make sure that it is safe, and that, you know, people can experience this in a way that will be… helpful to them. So that's largely where the… where RISF.
[20:10] Steve Semelsberger: My own shout out to Colette. So, I joined, full disclosure, I joined RISF as an individual member. came in, and about a month after I joined, there was a new member welcome. And so there were 6 of us, newbies, who showed up on Zoom, and guess who was there to welcome us? It was Colette and 3 others, and she was dynamite, and super warm....
[20:30] Courtney Nash: I have not made nearly as many as she has, so…
[20:35] Steve Semelsberger: Well, it's great, and to me, it feels like… it feels like a moment. It feels like with the acceleration of AI, everything. that this is a really important moment to learn from other industries and to think deeply and explore together this concept of resilience. I shout out to the Foundation. If you're curious about the things that Courtney and I are talking about, take a look at it. I think it's a really, really special group of people. Super welcoming and open. It gets really heady and nerdy fast, so if you like that stuff, you'll feel very much at home, but It's also the kind of place that you can come into with whatever background perspective and, you know, find your way into it as well, too. So thank you for sharing that.
[21:15] Courtney Nash: mostly software practitioners in that community, but we do have a few researchers, and a few sort of lurking academic folks, and other industry folks, so… and we are working on… Develop me more materials to help people the learning curve is steep and dense. The stuff we read, the materials, the papers that I'm even name-checking as we chat are non-trivial stuff to start to wrap your brain around, so one of our goals is to help distribute that kind of information, help people get their arms around it better, help them bring that into their own organizations. Right? And start to spread that information throughout the industry.
[21:55] Steve Semelsberger: You mentioned the origination of learning from incidents, and many people who know you know your work with The Void deeply, but for some of our listeners, this might be their first exposure to you. So, tell us a bit about what The Void is as well, Courtney.
[22:10] Courtney Nash: Yeah, The Void started out as I was brought on to a small startup that had a chaos engineering-like product. They were working largely with Kubernetes and Kafka, because, you know, those work great all the time. And they brought me in weird background, right, to sort of do product research, but They didn't want me to do traditional product research. And so, my… one of the first things I started doing was, first of all, running incident chats inside the company. And so I would go find Kubernetes and Kafka incidents that people had written up about, and we would sit down and we would talk about them, and we were, like, looking for what we could learn, right? So I was looking for failure in the wild. of these technologies, and I started getting… I was like, alright, I read all the marketing materials, you know, I watched all the talks, and I was like, this is insufficient. for us to understand, like, how these things fail in ways. And so I started collecting incident reports, and once word got out that I was collecting them, like, Dan Liu back then had a Kubernetes.af, so he had a bit of a repo of Kubernetes incident reports, and then John Alspaugh, who we've talked about multiple times now, he had been scraping all of… shout out to Lex Neva, SRA weeklies for, like, years, and Lex used to actually include incidents a lot in his newsletter. He doesn't do it as much anymore, and so I had this scrapes database, and then the next thing you know, I turned around and had, like, 2,000 incident reports.
[23:32] Steve Semelsberger: Wow.
[23:32] Courtney Nash: Then I was… we were like. this is interesting, we're onto something here, and there's definitely information here that is not really surfaced or present anywhere else other than the occasional luck of a conference talk where somebody, you know, pulls back the curtains, right? So, we kept going, kept collecting, kept scraping, kept… and then I had, like, 10,000 incident reports, public incident reports in this database. And I started doing research on that, so I wanted to pick at… things that we collectively, the Learning from Incident Resilient folks. Knew for reasons that we didn't held quite deeply, based on other research, were not helping us as an industry. So things like MTTR as a metric, as a representation of Resilience, or how well your incident process is working. And I had seen some other work that other people had already done, so, Stepan at, Google, who has the… O'Reilly little short book on MTTR, and I set out to replicate his results, which we did, and then we started chasing research around, like, the prevalence of automation in incidents. So, just using the data that was in public incident reports, which, admitted, is not necessarily the same as internal incident reports or that kind of internal data, but it's still… gave us so much information about the way we perceive incidents, what we can learn from them, you know, and so that is the general scope of what the void was for quite a while. I also do some, like, consulting and other kinds of things with folks through that organization, so the goal is just to surface… What we can learn from incidents that, you know, and to help people… not be afraid of that, and to help people use that information, and mostly, I mean, the initial goal was to get people to share their incidents more, right? More people to write up their incidents, to share the stuff that we don't normally share in public write-ups that is so useful and valuable to the rest of us as an industry. And I think other industries have been Somewhat better about that than we have. So… so that's been the goal, and I took the void, that startup, turned it into a different startup, turned it into something else, you know, as things go. So I've sort of pulled it off separately, and it's orbiting itself right now. Trying to figure out what I'm gonna fully do with it, but… That's the hope, is that we all see incidents as ways of understanding how our systems actually work.
[26:06] Steve Semelsberger: We'll put the URL in the show notes if folks want to visit The Void and learn more.
[26:12] Courtney Nash: Cool.
[26:12] Steve Semelsberger: Yeah, okay, so you said a number of things there that, that got me even more interested in some of these topics. So… One thread. We've been talking about people and systems. Do you see correlation between the resilience of the humans in an organization and the resilience of the systems that they're responsible for?
[26:38] Courtney Nash: Mmm. This is a really good one. This is where our use of the word can get really interesting, because when… when many of us talk about resilience in the technological social-technical construct, right, in the sense of software. We… and I say that it is a uniquely human generated thing. it is still… the resilience is the system. Resilient is, as David Woods likes to say, a verb. It is not a… you are not suddenly resilient. You are not… it is not a state. That, right? And so, you can, even as an organization, or as a team, or as a person, be more or less resilient. the notion that one person is more resilient, and therefore the system is more resilient, I don't think that's true. And I actually think that sometimes… highly… what look like Resilient systems. Sit atop some very, at risk of no longer being resilient humans.
[27:43] Steve Semelsberger: Hmmm.
[27:44] Courtney Nash: Be… because… We are the ones who step in and resolve these problems, right? individual resilient… individual resilience doesn't exist either. We are… we are only resilient in the context of the system that we were in, right? And so, like, we'd like to think of resilience, this human resilience, like, I'm gonna get knocked down and get back up, right?
[28:07] Steve Semelsberger: Yeah, grit, fortitude, things like that. Sometimes that’s how we use the word.
[28:12] Courtney Nash: Yes, and those are not the same thing. And, like, a highly resilient system should not… Burnout. It's components. Ultimately, right? In my opinion, a truly resilient thing, to be able to withstand those changes, that capacity to respond to unexpected things, we have largely done that in our industry by burning people out. And I would argue that is not actually technically a truly resilient system. Because at some point, then, that piece… no longer can contribute to the system, and then it devolves in terms of its resilience. And so that's the phenomenon of incident commanders and responders and people burning out, I'm not directly answering your question, but I think you can also be maybe not a resilient person, whatever that means, but you could be in a system that supports you, and that system could be more resilient. I mean, it's the same in the sense that burnout is not an individual problem. Burnout is a systemic problem within an organization. that's a whole other… but… and I have a lot of opinions and feelings and research about that. So, I feel like I diverged from your initial question.
[29:30] Steve Semelsberger: No, I… what I took away… let me… let me see if I captured some of your intent, is… low, maybe negative correlation, because I was probing for a lot of things with that question, as you picked up. You know, one is how do we use these words? And then two, you know, when the system is the people and the technology, if you split those two things apart, which is a false thing, but you can kind of do it as a thought exercise. Certainly, we have all seen heroic efforts of the human, and when we use grit, and like, I'm just gonna stay up all night and keep plugging away, and when the phone rings, I answer, and I do the things, and those… Really gritty people keep fragile systems going.
[30:10] Courtney Nash: Yes.
[30:11] Steve Semelsberger: So the whole thing seems resilient, and the adaptive capacity is, like, again, this stuff that can cause real harm to the humans who are propping it up. Reminds me a little bit of the statue in front of Rockefeller Center of, you know, one of the gods holding the globe on them, and we've seen people doing that, and it's a… man, it's an interesting time, too. So go deeper into this, if you would, Courtney, because you know, what we see a lot, and, you know, I spend time in my day job talking with practitioners and engineering leaders, and there's such an expectation of increased velocity. Humans doing more with less right now. AIs power everything. Go faster, change more, update, push into production, go, go. go.
[30:54] Courtney Nash: I chaired a conference called Velocity for 5 years.
[30:56] Steve Semelsberger: Yeah, you felt it.
[30:59] Steve Semelsberger: And it's even more now, isn't it, in this moment that we're in, wouldn't you say?
[31:04] Courtney Nash: Yeah, we are definitely at an inflection point with that. I almost feel like we were making some interesting progress with all of the DevOps work and all of that, that we have just… not just erased, but… but… you know, just run backwards past, in terms of… acknowledging the roles that humans play in technology, in supporting them, in acknowledging how important expertise is. We are definitely charging towards a very dangerous reality, in my opinion, but and some of that comes from this thing that you just said, that you said, oh, we could do that as a thought experiment, and I would argue our industry does this as a thing all the time, which is to separate humans and machines.
[31:49] Steve Semelsberger: Yeah.
[31:50] Courtney Nash: Right? And so, this notion, it started with all the automation stuff, it's now… gone wild with all of the AI stuff, that, like, oh, humans are good at this stuff, and machines are good at this stuff, so let's just give the stuff to machines that they're good at, and we'll have the humans do this other stuff, and then, because the machines have the stuff that they're good at, the humans can just do more of the other stuff. And I would ask anyone who's listening to this if that is your lived experience, because I would argue, you're like, you're all like, no, no, no, no, no, that's not how this is working at all. And it's because the premise is false. The premise that humans are good at, machines are good at, separable, cognitive-like things, and you can just cleave that and then give one to one, and give one to the other. Like, that's not how complex systems work. And when did we ever, like, you know, so it's this old… Idea that comes from cognitive science. Unfortunately, Paul Fitz, the Fitz List. machines are better at, humans are better at. It actually was… men are better at. We… we've re-ricked that one. And that is another, sort of like the tailoristic line, you know, assembly line notion of software that we still cling to. We still cling to this, like, separation of what quote-unquote humans are good at and quote-unquote machines are good at. And that… Is a pervasively problematic framing. as we go into this, like, world of AI, in part, and because this expertise bit Right? Like, humans have expertise with the system. The boundary conditions, the weirdnesses of it, the way that… and they only have that expertise by having their hands in it all the time. And we started doing this with automation, but now we're going crazy with it with AI, right? And so we're like, no, we'll just have the AI do all the stuff. And it can't do all the stuff the way we want it to do it yet, and it can't do it ever in my… I'm one of those people. Not a lot of AI hater, but the framework we use is putting us in a place where it's not gonna do what we think we want it to do, right? To go back to that. And now, then, you have AI doing so much more stuff, and when it doesn't work, the humans step in, and they have… no context, right? And they don't have the acquired, accumulated expertise, and they can't even introspect the system. And actually, the Frontier AI Labs, OpenAI, is now actively developing technology to make… chain of thought. harder to… C. So now we're literally doing the thing! So, yeah, I think, I think that particular framework, humans are better at, machines are better at, and then piling AI on that… I think we're gonna have a reckoning of some sort. I'm not like, oh, the AI bubble will burst, but I think we're already seeing companies saying that, like, this investment in AI isn't… It's not giving us what we want, and people are having to come in and spend more time figuring out what the heck is going on than we'd promised ourselves we would.
[35:04] Steve Semelsberger: Yeah, yeah, massive data sets, massive amounts of complexity. Turns out that as great as the language models are, it is really hard to get inference to behave repeatedly and trustably, let alone be governed and compliant.
[35:22] Courtney Nash: Yeah, and humans' ability to improvise and adapt Right? And I don't mean AI starting message boards, adapt. I don't mean that. I mean something else, right? Humans' ability to be slightly unpredictable and, like, base… Because they have this inherently intuitively built expertise. is what tends to solve those problems, and tends to help in those situations. And it's not like pattern matching an LLM's, you know, over 4,000 more incidents. It's not gonna get you to the same place.
[36:00] Steve Semelsberger: Hmm. We humans do love to create boundaries and separation, too. There's a book that was written, I believe in 1972, by Ken Wilber called No Boundary, that some of what you were talking about, Courtney, reminded me of, and one of our guests on the podcast is an author named Byron Reese. He's a futurist, and he wrote a book called We Are Agora. Which is the concept that humans might just be superorganisms in some of the ways that we behave together. So he's a beekeeper, and you should listen to the episode, because he'll do…
[36:31] Courtney Nash: Writing it down because I'm interested in it based on the title.
[36:36] Steve Semelsberger: Agora was a Greek marketplace, people come together. So he's a beekeeper...
[36:38] Courtney Nash: Oh yeah, I'm a liberal arts college kid.
[36:42] Steve Semelsberger: Wonderful, you’d love Byron. He was asking a lot of really good questions about… you know, when and how do humans behave as superorganisms? You know, when is our collective activity more than any of us as individuals? And, you know, if you mash Byron Reese and Ken Wilbert together, you know, you then work on eroding some of these false boundaries that reductionist tendencies of our Western brains especially love to do to compartmentalize. So… Some of our listeners are pragmatic, and it's interesting because, you know, we read and we pay attention, and for example, a couple weeks ago, Gartner put some research out, and in the world of what they would roughly call AI ops, whatever that means, but in that world, so they, you know, they ask questions of technology practitioners in that construct. And the… the third… most important thing that they received an answer to. Top initiative of the year, 30-something percent said, decrease MTTR. So, a lot of people in big corporations, you know, they've heard it, and it kind of makes sense. And so, given your perspectives on the challenges of MTTR, if somebody brought you in to consult, what would you say is a better way to measure? If somebody feels like, look, you know, the board and the finance team, they want us to have some numbers?
[38:02] Courtney Nash: They got to have numbers.
[38:03] Steve Semelsberger: Like, what are the numbers? Like, what are some of the key numbers?
[38:06] Courtney Nash: Yeah, so the things that I encourage people to start looking at is…MTTR is supposed to be, like, are we good at incident response, right? Like, are we good at handling incidents? And a lot of them also kind of want to say, like, we're having fewer incidents. And if I could put a pin in that, we'll come back to that one, right? Incident count, okay? So they're using MTTR as a proxy for incident response, and sometimes often more broadly for reliability. and I've talked about, like, we can point to Stephan's work and my work, and, like, the statistics of why that's not true. Like, statistically, MTTR cannot tell people what they wanted to tell them. But they don't like that answer, because then the answer is always like, okay, what else can I do? And so the goal is, like, are we learning from these incidents, right? Like, are we getting better at handling incidents? And the way you get better at handling incidents is by learning from your other incidents, right? So, there's some baseline stuff that companies can do, all the DORA DevOps stuff, right? Like, have a CICD pipeline, have, you know, SLOs, have all… there's technological stuff that can help you get to base level of, like, how are we handling incidents? And beyond that, the numbers are harder to help you there. But what I do tell people, then, is look at your incident Write-up situation. How are you… disseminating what you're learning from your incidents. Are you disseminating what you're learning from your incidents? So you can look at things like, how are incident write-ups being shared? Are people using them in architectural, you know, diagrams, or in code design situations? This is hard, because all of my examples sound like I'm 20 years behind the curve, and so, like, how, like, but are you factoring them into all of your AI work, like, is AI aware of… if you are using AI to design your products, is AI aware of your incident reports, and what is… Right? Are people sharing them? Are people reading them? Are people coming to your incident, you know, post-mortem meetings, and that kind of a thing? That is the space in which you get better at these things. And then, you can also look at… and this is where things get a little dangerous, right? What you don't want to become is, like, an action item factory. Alright, so what happens is people have, you know, you have your incident report, and learning and fixing things are two different domains, right? So I'm asking… these are metrics around, is your organization learning, right? Are they sharing this information? Are people using this information? Are they refer… The… are you allowing the people who say there's a problem over here To go fix that problem. Because what happens a lot, and I see this all the time, is… you have incident reports, you get action items out of them, right? I guarantee you the teams involved in those systems already know everything in those action items. There's people who've been saying for years, hey guys. we keep having these scary database incidents, and so, like, maybe we should take some time and, like, go fix the thing over there that, like, keeps falling over, right? So the other side that you can look at is, are you funding? And prioritizing the work that's coming out of these. Now, not how many tickets closed, how many action items closed, right? But are you paying attention to the things that people are saying? Are problems coming out of your incidents. Because those are often… The things that are involved in the future ones. So what I see a lot, the perversion I see a lot is, like, all of these things come out of incident reports, but it's so hard for the teams to prioritize that work, because it's not feature work, it's not product work, it's not… What have you. And so, designing metrics around that, are we… Are we carving enough time and effort out for the teams to do this, what some people just, you know, put a bubble around called reliability work? If you are not… Allowing people to do the work that's coming out of the incidents, then… Like, you're… your MTTR is still gonna be all over the place, no matter what, too. That's the thing, because you won't know if you're getting better. so the last thing I tell management teams to do is to regularly and consistently survey the people at the sharp end, right? So you… do you have… you probably have employee satisfaction surveys, right? You know? do you ask your incident responders, your SREs, everybody, like, how things are going, how confident they are, how resourced they are? Like, are you… that stuff, it's not… it's qualitative data made quantitative. So that's the third thing, is… responder, Surveys. It's real. That's probably the best source of information about how you're doing, but a lot of companies just tend not to see that as valid data as these other theoretically quantitative things.
[43:14] Steve Semelsberger: Does some of that, Courtney, fit within the S of SPACE? If you look at satisfaction of the engineer, whether it's a DevOps engineer or a developer, some of my understanding of space is it's an attempt to address some of what you're saying.
[43:26] Courtney Nash: Some of that, but if you're, like, really homed in on the MTTR reliability, you need to ask the people at the, like I said, the tip of the spear about specifically reliability-related satisfaction, right?
[43:39] Steve Semelsberger: Yeah.
[43:39] Courtney Nash: Not just developer satisfaction, or speed, or resources, like, that kind of thing. So you could… you could fork a bit more out of that, in my opinion, yeah.
[43:48] Steve Semelsberger: Not to quote the analysts too much, but I will. So, we were talking with one of the lead observability analysts from Forester, and they said something like…
[43:58] Courtney Nash: Which one?
[43:59] Steve Semelsberger: Carlos (Casanova).
[44:00] Courtney Nash: Okay.
[44:02] Steve Semelsberger: So Carlos, shout out, because I really appreciated what you said, and hope it's cool that I quote this. But… but my memory of the conversation was. Man, if organizations could put 5-10% of their total engineering capacity towards prevention. Yeah, and it was the construct we were using in that conversation. You know, we kind of averted a minute ago to reliability, but his notion was the team knows the stuff to do to prevent the incidents from happening, and if we could put 10%, 10% of our capacity towards that, you know, many organizations would have much stronger, more reliable, and hopefully more resilient systems as well, too. Have you seen companies take that on, Courtney? And maybe you can go back and tell me if you agree with it first, and then you can tell me if you've seen companies try something like this.
[44:49] Courtney Nash: Yeah, I mean, I think… there are places that tend to have that in their DNA more than others. I mean, Google is largely… not always, but is, like, probably… but it's a stupid example, right? Be like Google! Sorry, you know, and I'm the one who signed the SRE book, so I know. But, but yeah, I mean, I… I don't have great examples to point to, but I… and I think the problem that happens is people then try to have, like, centers of excellence or something, they over-index on it, right? And then they create these bureaucratic structures that have no effectiveness. But there are… I think Honeycomb is a really great example of a company that… and obviously we're talking about observability, I think they're doing a lot of really interesting and cool things. They're not perfect at it, I know that. I know that there are people who are there who are like, well, we really should be doing… but, like, I think they are an example. of companies that sort of put their money where their mouth is around those types of… of efforts. And oh, God, I just had a thought at the beginning of that that I lost. You're asking if I had seen… oh, oh, no, the other thing I was gonna say was. I do want to say, if you do that work, right? If you do talk to those people, and you hear what they say, and you give them the 10%, and you do all of that, two things. Your MTTR is still gonna be weird, so, like, that's the worst part. Ditch it. Commit to ditching it, and then do this other work, because then if you're like, oh, we do this, our reliability will get better, and… Accept the fact that that metric doesn't tell you anything, but two, you're not gonna have fewer incidents.
[46:22] Courtney Nash: You're just not, like, I think that's the crazy part. Like, I analyzed… granted, there's scale, right? Like, Cloudflare has thousands of incidents a year, y'all. So does Google, so does Microsoft. so does AWS, but they don't tell you, right? Like, the people who are good at this don't have… they have scale, for starters, right? So it's sort of an unfair comparison. But being good at reliability or incident management or whatever does not mean you necessarily have fewer incidents. I think that's one of the other metrics-based misconceptions that's out there. It just… it just… they're not going away.
[47:02] Steve Semelsberger: (Laughs). So, what about time to detect? Is TTD and put an M or not in front of it, is time to detect valuable?
[47:11] Courtney Nash: I don't have… see, the thing is, is people ask me this question all the time, and I'm like, I mean, I… someone has the data, Google has the data for this, I'm sure. I don't have access to the data. Time to detect might be normally distributed. So let's, let's just say… okay, so the problem with MTTR is it's not normally distributed, so you can't take a mean. The mean is meaningless, blah blah blah blah, long tail. Statistics 101. So, maybe MTTD… is. Let's, let's just say it is, okay?
[47:45] Courtney Nash: What does… driving that number down, focusing on driving that number down. Like, what are you gonna do? And the answer I feel like I'm always gonna get is you're gonna do all the things you should do anyways. Regardless of whether you have that metric or not. Right? Because it's like, oh, well, if we're gonna reduce our MTTD, then you're gonna do things like having good observability, having SLOs, like, what does your monitoring look like? All of this sort of software hygiene stuff. Right? if you were to run an experiment, this is sort of classic DORA stuff, even. Like, one organization that isn't doing all of these things, and one organization that is. the organization that is doing all the good stuff probably has, if it's just normally distributed, a lower MTTD. Okay. But then the incident itself… could be a slam dunk, it could be the weirdest one you've ever had, like, so the time to detect doesn't necessarily mean that you're gonna be any faster at responding to it, or that it's gonna be any more predictable, or that it's gonna be any easier or not. The organization that doesn't all that, have all that stuff, might have a longer one. And so that might sort of delay things over time, and then they could go do the basic software organization stuff, and their MTT would go down. but then all the other incident stuff is… right? So I'm just like, what are we doing? Like, why… I honestly sort of been baffled by this. Like, we're so obsessed with having these very quantifiable sort of business metrics. that we're spending all this time talking about those, and I'm just like, go… go do the thing! Like, Liz, Fong Jones at Honeycomb, I'm on a rant now, sorry.
[49:27] Steve Semelsberger: That's good, that's okay, go, go.
[49:28] Courtney Nash: has this amazing post, which is basically saying exactly what I'm saying, because they have gone down this crazy road of, like, 3X-ing their PRs using AI, right? And all of us who know them really well for a while were like, what are you guys doing? Like, it's crazy. crazy cakes, right? But any… and Liz, of all people, was probably an early non-believer, right? And… but her whole point, I'll go find the post, is, look, don't do this stuff if you're bad at software. Like, like, don't do this! What we're doing… if you don't have all the fundamentals nailed, and I mean, like, nailed, right? And so, if you don't have… well-established SLOs that you revisit. If you don't have the culture of experimenting with this stuff, if you don't have the ways to move fast and change rapidly, which is actually a good thing, not a bad thing, if you don't have all this stuff in place, and then you layer a bunch of AI on top of that, you are accelerating disaster. And so, you know, so that point was, go nail the fundamentals. And then the metrics honestly don't… matter as much, those traditional sort of metrics, which they don't really use.
[50:45] Steve Semelsberger: Good points, good points, Courtney. You told me at the start, before we began recording, that there are some humans in your life that need you to pick them up, so I want to respect those humans.
[50:57] Courtney Nash: I'm okay, thank you.
[50:58] Steve Semelsberger: Okay, okay, cool. Well, let's, let's look ahead a little bit. This has been awesome, we've covered a lot of ground, we've been precise with language, we've looked at concept, we've looked at history, we've talked about your background and the crazy journey that has you with me today, which I really appreciate very much. As you look out ahead, you know, you are clearly one who values learning. So what are some of the things that you would like to learn over the course of the next year or so, Courtney?
[51:30] Courtney Nash: That's a really good question.
[51:35] Courtney Nash: Well, one of the things that I had hoped to do with The Void that I haven't done yet, I've designed a pretty large survey. I spend a lot of my time talking to people who are really good at this stuff. How's everybody else… how are y'all doing out there? Like, big organizations. And people ask me these questions, and I feel like me, as a researcher, and as someone who's continually learning, I don't know what John Deere. does. You know what I mean? Like, I… and so these big places that have a huge outsized impact on our lives, I think more than a Honeycomb does. No offense, y'all. Love you. I really wanted to learn more about what's happening there, how those folks can be helped. You know, like, how we can take all of this really obtuse, nerdy. Science and research and… and put that into a place that's actually beneficial, you know, to the industry as a much larger whole. I mean, I think in the long run, that's why I got involved with RISF, and why I think what that organization is doing is so… important. Especially as we… continue to add AI, to this. I think what's interesting about it is it's… I am seeing people have conversations about expertise, and human value, and de-skilling. I have seen that word in a, like, a media article, and I was like. Okay, that's straight out of, like, the cognitive science literature, right? That we are de-skilling experts by, adding AI into so many things. So, I think that's what I'm the most… I'm honestly wanting to see how we can have a larger impact on the industry, and I don't… truly feel like I understand the reality of… Big organizations that aren't software started, like, I understand the reality at Google. I understand the reality at Cloudflare, sort of. I mean, does anybody really understand, that? But… they do. So, so yeah, so I think there's… and I've, I've… it's the… that's where I'm curious about, and what the next year is going to look like, because this stuff is… is disgust and… And experienced most in a very niche place, just in our industry still. And as I've, over my career, have watched how things flesh out and distill and eventually reach these enterprises and whatnot, I think that is where my curiosity Is landing for the next… the near-term future.
[54:18] Steve Semelsberger: Possible GoFundMe bus tour coming up, Courtney.
[54:21] Courtney Nash: Yeah. I don't know, we'll see.
[54:25] Steve Semelsberger: Oh, I think a lot of organizations would love to have you on the inside if that was a possibility. So, what you put into the universe, who knows what sort of invitations that might invoke for you. I love your generosity, your precision, your perspective, your willingness to take positions on things. This has been a ton of fun for me. I've learned a lot. Any final thoughts? Anything you want to, put on the table that we didn't get to today, Courtney?
[54:56] Courtney Nash: I don't… I don't think so, I just always come back to the same thing, which is that humans are amazing. Like, humans are amazing! So, that's… that's the, you know, people… I started giving these talks on automation a couple years ago, and people were like, well, didn't you use automation to put you… you know, I was like, of course, all software is some form of automation, right? I'm not anti-automation, I'm not as, like… but I am very much more… pro-human. and I want us to hold on to that. Like, the ability to learn something, to get good at something, to learn about something new, is… so special, and for us to celebrate that, to hang on to that, to continue to encourage that, to not… lose that, is… what I hope we can all focus on.
[55:45] Steve Semelsberger: Couldn't agree more. Like many, I have maybe a mixed perspective on Las Vegas, but my youngest son had a basketball tournament in Vegas this summer. And I did something really different. I walked the Vegas Strip a whole bunch to appreciate the people. And to appreciate what people have actually built in Las Vegas. If you just look, like, whether or not you like to gamble, I didn't place a bet, it wasn't really my thing, but I just spent time really in this, like, place of deep… appreciation, gratitude, people watching. It's one of my favorite things about big cities. I agree with you, people are amazing. Every single, every single human is so incredibly special. And there's so many of us around, it's easy to lose sight of that, but when you really reflect on who people are, and what they're doing, and what we're so lucky to be a part of. And yes, there's a lot of uncertainty in the world, but there's a lot of reason for hope as well, too.
[56:45] Courtney Nash: I would leave it at that, I agree.
[56:47] Steve Semelsberger: Courtney Nash, great to have you. The different sites that you reference will be in the show notes. Thank you so much for the work that you're doing. Any places that you're going to? Conferences? Bus tours?
[57:00] Courtney Nash: Well, I'm about to get on a plane to Salt Lake City tomorrow, but that's because my daughter is a, phenomenal mountain bike racer, so we're going to have the last bike race of this season. A couple of things are happening. I am the new… I didn't mention this when we were chatting earlier, I am the new two-year co-chair of SRECON, so for, 27 and 28. And so that… my responsibilities for that are just ticking up, and I am going to present at SRECon EMEA with a crew of RISF folks talking about incident response and incident analysis, and how you can get better at that stuff. We are also doing a full-day training at QCon in San Francisco, a little more accessible to folks than maybe, perhaps, Dublin in short order.
[57:45] Steve Semelsberger: Depends on where our listeners are, of course, you know, to those in Dublin, make sure you see Courtney, as well as to those in San Francisco.
[57:51] Courtney Nash: Please, please come say hi to us. Those are the, those are the fun things that are, that are currently happening.
[57:55] Steve Semelsberger: Well, congratulations, and thank you for your track and conference share work with us, and all that you do in the world, too, and thank you again so much for your time today, Courtney. It was great to have you on the podcast.
[58:06] Courtney Nash: Thanks, it was a real treat.