Transcript: Lessons from the Mountains: Retrospectives and the Secret Sauce of High Performance Teams
0:00 Brad I always try to say, like, look, what really matters is we need to try to figure out what the root cause is.
0:05 Brad But many things probably went wrong and they all have their own root causes.
0:28 Scott place yesterday that do you like cocktails do you have a cocktail guy have
0:32 Brad A hundred percent, dude.
0:34 Scott you been to this place called cannon on 12th avenue by seattle university i'd
0:38 Brad Yes, that's a classic.
0:39 Brad The
0:40 Scott never been there's a classic how long has it been there oh
0:43 Brad 20 years.
0:44 Scott really well
0:46 Brad More than 20 years, maybe.
0:48 Scott i went there for the first time last night and i ordered a gin drink
0:54 Scott And they literally serve it in a bathtub-shaped cup.
0:56 Brad love it yep that's one of their yep that's one of their their things i think they also have like uh
1:04 Brad they got like a drink that's like the they serve in a um blood bag
1:09 Scott I didn't see that.
1:10 Scott I saw a drink they served
1:11 Brad yeah
1:11 Scott in a Chinese takeout thing.
1:14 Scott So it was super fun.
1:15 Scott We had a really old whiskey.
1:17 Scott Well,
1:18 Scott actually, it wasn't that old. It was just 23 years old.
1:21 Scott I wanted the whiskey that was like 130 years old, but I couldn't bring myself to spending $2,000 on a single drink whiskey.
1:30 Brad Yeah, that's special occasion whiskey.
1:33 Scott Yeah,
1:33 Scott I don't know if there's an occasion special enough in my life for that.
1:40 Brad It's all a matter of scale,
1:41 Brad I guess.
1:41 Scott That's right.
1:43 Brad Welcome to another episode of How Many CTOs Does It Take podcast.
1:46 Brad I'm Brad Hefta-Gaub and my co-host.
1:48 Scott My name is Scott Poirier.
1:50 Brad Awesome.
1:50 Brad And today I think we talked about on text,
1:54 Brad we chatted with each other a little bit about the idea of talking about retrospectives.
1:59 Scott Retrospectives. Love it.
2:00 Scott The most important part of the Agile process.
2:05 Brad I think so.
2:06 Scott Yeah, I do too.
2:08 Brad So I will tell you part of my,
2:12 Brad I have very strong opinions about retrospectives.
2:16 Brad And part of those opinions come from the,
2:21 Brad one of my hobbies slash one of the things I spend a lot of time doing, which is traveling in the mountains.
2:31 Brad And in the winter, of course, that means traveling in avalanche terrain.
2:36 Brad And as you know, Scott,
2:39 Brad I'm a ski patroller and a SAR unit member.
2:41 Brad And one of the things I do is I'm a professional member of the American Avalanche Association and I'm an avalanche educator.
2:49 Brad So I teach avalanche skills to search and rescue unit members and ski patrollers.
2:56 Brad And the notion of a debrief after you travel through avalanche terrain and the notion of a high quality after action discussion and after action report on SAR missions or traveling in the mountains is like an essential component.
3:19 Brad to making sure that when you travel in low information, high consequence environments,
3:28 Brad you get better and hopefully prevent dire consequences,
3:37 Brad right?
3:38 Brad So the idea of really sort of trying to interrogate, well,
3:42 Brad what decisions did we make?
3:45 Brad And did we just get lucky or were those decisions,
3:49 Brad you know, proactive in our success or were they things that if we interrogate them and evaluate,
3:57 Brad like maybe we could make you could get better at how we did that?
4:02 Brad And so.
4:04 Brad I see incredible overlap between decision making in the mountains and decision making as it relates to writing software.
4:14 Scott Yeah.
4:15 Scott I agree.
4:17 Scott I worked at a company once which had some core values and they were really well done core values amongst all the companies I've worked at.
4:26 Scott It's I'd say the top two.
4:28 Scott It's number one and number two. And one of its core values was simply get better every day.
4:34 Brad Yeah.
4:35 Scott And I think this fits right with that, which is.
4:40 Scott The spirit of the retro is getting better and
4:44 Brad Yeah.
4:45 Scott the best teams I've been a part of are striving to get better all the time.
4:50 Brad Yeah.
4:51 Brad Growth mindset.
4:52 Scott Yeah,
4:52 Scott growth mindset.
4:54 Scott Once we tried to have a retro for every error,
5:01 Scott we stopped after a few weeks because it was just like, oh, it was too much.
5:05 Scott But if you think about it, why not?
5:07 Brad You mean like every 500 in the web server log?
5:11 Scott something like that i think it was for every alert that went off actually is what we did because we asked ourselves let's do a retro on this and it ended up being like multiple retros every day and it was it didn't work it was a good idea but we were asking ourselves the question yeah every 500 in the logs everything in the logs that's an error
5:35 Scott Every time the alarms go off like these are problems that are happening,
5:38 Scott they might not be customer impacting problems,
5:42 Scott but why wouldn't you do a retro and understand what is it that caused this error to happen?
5:48 Scott Any bug,
5:49 Scott why wouldn't you do if there were enough time in the day,
5:51 Scott why wouldn't you do a retro for every single bug?
5:54 Scott Say, well,
5:55 Scott how did it come to be that this happened?
5:58 Brad Totally. I think that there's a spirit to that, that if you did have enough time, you would want to interrogate every negative consequence and ask yourself like what caused it.
6:09 Brad And I certainly support that idea.
6:11 Brad And I think that's one of the first things that I mean, to your point, like given enough time, right? Like I think also it benefits being able to like step back and kind of look at a collection of things that happened and say.
6:23 Brad and try to interrogate them as a collection and say like well what what do these all have in common what were our common behaviors that might have led to them right
6:32 Scott Now there's a thing I think I know you're going to say I'm going to set you up. I'll give you a softball here, Brad.
6:37 Brad Yeah,
6:37 Scott I know that you're going to say, Scott,
6:40 Scott you were just interrogating the things that went wrong.
6:43 Brad exactly.
6:43 Scott Why didn't you do a retro on every single PR where nothing went wrong and ask what did we do on this PR to make it so that it was successful?
6:53 Brad 100%. That's exactly what I was about to say to you, which is, of course, comes from traveling in the mountains, right?
7:00 Brad every time you get home every time you get back like that was a success and those are the times that you really have to ask yourself well why did we what was why were we successful like what did we do differently that was successful and I think that in software engineering like the teams that have the amazing sprint where they got all of their tasks done and they delivered a bunch of value and
7:24 Brad And then if they just if they if they end their sprint and their closing ceremonies with,
7:30 Brad you know, you know, popping champagne corks and slapping each other on the back, congratulating each other, that kind of misses the most important point, I think, to the to the retro,
7:42 Brad which is.
7:44 Brad Yeah, but why? Why did we succeed this time?
7:46 Brad What was it that we did that made us successful?
7:49 Brad Were we particularly good at our estimates? Did we have was it was it just some heroic effort by part of the team to accomplish this goal or had we actually set ourselves up for success?
8:04 Brad From the things that we had done during the planning process. Right. And trying to really understand that, because if it's if the answer is, yay, we did it, but it only happened because of Scott's heroic efforts.
8:16 Brad Well.
8:18 Brad That's maybe looks like a win on paper,
8:21 Brad but is not a very sustainable win.
8:25 Scott Yes.
8:25 Brad Right.
8:25 Scott Yeah.
8:26 Brad Like we can't count on Scott always doing the heroic thing every sprint.
8:30 Scott That's right.
8:30 Scott That's right.
8:31 Scott So, okay,
8:32 Scott I want to get into this a little bit more practical here, though.
8:35 Scott We could talk about this theoretically all day long.
8:38 Scott I can think of three different types of retros, sprint retro.
8:43 Scott So do that every one week, two weeks, three weeks, however often you have your sprint cadence.
8:47 Scott A project retro.
8:50 Scott So, or maybe you might call it an epic retro, but a project retro,
8:54 Scott which is something might last multiple sprints, but now we've sort of completed this major milestone.
8:58 Scott Let's go back and look on the whole thing.
9:00 Scott And then I'll call it an incident retro, which is, well, there was some sort of problem in our operational environment.
9:06 Scott Let's look at that.
9:07 Scott I have a fairly strong opinion about how the last one should go.
9:14 Scott The postmortem type incident retro,
9:17 Scott right?
9:18 Brad Mm-hmm.
9:20 Scott I don't have as strong of an opinion on how the first two should go.
9:24 Scott I'd love to hear,
9:24 Scott like, how do you do your sprint retros? How have you found that those are most effective?
9:29 Brad Yeah, I'll tell you how we're doing it today,
9:31 Brad and I feel like it works really, really well for this team that I'm working with now.
9:36 Brad I think I would.
9:38 Brad Definitely lump them into the category of a high performance engineering team.
9:42 Brad They're all very growth mindset,
9:44 Brad very open minded. They're all very introspective about and they thrive in interrogating what went well and what didn't go well.
9:53 Brad And so here's how we do it.
9:55 Brad Our sprints are two weeks.
9:57 Brad We have 90 minutes dedicated on the second Friday. Our sprints start on Mondays and ends on Friday,
10:06 Brad you know, 10 business days later.
10:09 Brad We happen to be using a tool that is a plug-in to Trello for our, it's called retrospectives, it's a plug-in to Trello, but all it really is is a.
10:18 Brad everybody gets to put in little cards that are either in the column of what went well or what could have improved.
10:29 Brad And so here's our process.
10:31 Brad We start the retro and we say,
10:34 Brad all right, everybody,
10:34 Brad boom,
10:35 Brad start adding your cards.
10:37 Brad And when you're done adding cards,
10:39 Brad raise your hand, right?
10:41 Brad This tool happens to have a little button to say I'm ready.
10:44 Brad And we tend to get, you know, it takes us about,
10:49 Brad I would say.
10:50 Brad somewhere between five to ten minutes for everybody to kind of get all their typing in and get kind of done like they almost everybody almost comes in with like a they know what they're going to type and we actually will open up the tool ahead of time like sometimes I'll just like I'll open it up on Thursday and just like hey start adding stuff you know and so it takes about five to ten minutes for everybody add their items and then when everyone's ready we
11:15 Brad click the next button and everyone can see all the cards at that point and it actually does show the people who added the cards so it's not anonymous but with my team that works okay because we've got such a high level of trust with each other that nobody minds that like brad said oh we completely screwed that up right
11:34 Brad And the next step is we group the things together into kind of topics and so and and what went well and what could have gone better can be grouped together into the same topic. We don't we don't make it so that there is a lot of times you have something where it's like, let's say the topic was like this past retrospective we had a topic that was.
11:55 Brad fire drill slash new feature x and what had happened in that case was it was a heroic effort to get the feature over the finish line um and people thought felt like it was a fire drill but in that grouping of the fire drill there was a whole bunch of like what didn't go well or what could have gone better and then there was several people that were like it went well like people jumped in and did a heroic thing right so we grouped them into categories
12:20 Brad And we usually do that live,
12:23 Brad like anybody can group things together.
12:25 Brad I tend to be the one kind of leading
12:28 Scott Mm
12:28 Brad it, so-hmm. I group them together.
12:31 Brad And that usually takes about two or three minutes.
12:35 Brad And then when we are all happy with, yeah,
12:37 Brad those feel like the right groups,
12:38 Brad then we click the next button.
12:41 Brad And we go, you know, kind of to anonymous mode again and everybody gets we've chosen four votes. They get to vote for four things, four of the topics that they really want to make sure we talk about.
12:52 Brad Right.
12:53 Brad And there tend to be like somewhere between eight or 10 groupings once we're, you know, once we've got,
12:57 Brad we have like,
12:59 Brad you know, I'd say like 50 or 60 items that get grouped into 10 things.
13:03 Brad And so now you're, you're voting on what are the four of these 10 things that I think are most important to talk about?
13:10 Brad And you can vote multiple times on the same topic. So if you're like, I really want to make sure that this fire drill topic is the thing we talk about.
13:19 Brad You can put all your votes on that.
13:22 Brad Then when everyone's done voting,
13:24 Brad we switch into back into kind of shared interactive mode.
13:31 Brad We talk about it. We talk about the topic. The one that gets the highest votes is the first one that we talk about.
13:36 Brad And we try really hard to always like sometimes there's a little bit like, hey,
13:41 Brad can you remind us exactly what happened? Hey,
13:43 Brad you know, Brad,
13:44 Brad when you said this, what did you mean by that in your in your description of the problem?
13:50 Brad But we mostly say let's focus on start stop keeps.
13:55 Brad Right.
13:56 Brad So if it's a good thing or a bad thing, like if it didn't go well or it could have gone better or went well, we still ask the question,
14:03 Brad okay,
14:03 Brad well,
14:04 Brad what about this do we want to start doing?
14:06 Brad What about this do we want to stop doing?
14:08 Brad And what about this do we want to keep doing?
14:11 Brad And we generate start-stop keeps for every one of our topics that we talk about.
14:17 Brad And then when we're done with all that, that's it.
14:19 Brad We have our list of start-stop keeps,
14:20 Brad and then we turn around and file those back into our process.
14:24 Scott Okay, I have two questions.
14:26 Scott Actually, I'm sorry, I have three questions.
14:28 Scott Where do the stop-start keeps go?
14:31 Scott Do you have any mechanism?
14:34 Scott For, you know, I guess I was going to say, do you have any mechanism for tracking if anything happens with them, which there's another flavor of that, which is how do you make it such that things happen?
14:44 Brad Yeah.
14:45 Scott And the third, which is this is the one where I often get puzzled, sometimes a start,
14:52 Scott especially,
14:52 Scott or sometimes a stop, but especially a start is pretty big.
14:59 Scott For you know, it might be I'm just making this example up.
15:02 Scott We really need to replace the whole payment system
15:05 Brad Yeah.
15:06 Scott Right like
15:06 Brad Yeah,
15:06 Brad for sure.
15:07 Scott somebody
15:07 Brad Right.
15:08 Scott Somebody
15:08 Brad We can't use that third party off provider. It's caused three of these these outages in the last month.
15:13 Scott Which is not like
15:16 Scott So that's not that's different than like,
15:17 Scott hey,
15:18 Scott we need to agree to have two people do code reviews very simple where we can put that in place.
15:22 Scott Hey, we need to do this giant and I always get stuck on the giant project that comes out of this.
15:27 Scott So how do you handle those?
15:28 Scott So those are my three questions.
15:29 Scott My three questions are, what do you do with the start stop keeps?
15:33 Scott How do you kind of
15:35 Scott prioritize or track to see if something's happening because nothing's happening with them why are you doing it in the first Yeah. place and then how do you handle the
15:41 Brad The
15:41 Scott big ones
15:42 Brad big monster ones.
15:42 Brad Yeah.
15:43 Brad Well,
15:44 Brad um,
15:45 Brad So let me answer those like like actually practically what we're doing today and
15:50 Scott Sure.
15:50 Brad then what my sort of general opinions on this are.
15:53 Brad So first of all, the tool that we use happens to be integrated with Trello, which is what we're using for project task management anyway.
16:00 Brad And it actually has a feature where you can go create cards out of all these things.
16:05 Scott Cool.
16:06 Brad Now, we tend to actually not do that. Instead,
16:10 Brad what we do is.
16:12 Brad And we,
16:12 Brad it's kind of on me as the, you know, leader,
16:18 Brad CTO,
16:19 Brad engineering manager,
16:20 Brad scrum master,
16:21 Brad right? We're a small enough team that I kind of wear all those hats,
16:23 Brad right?
16:23 Brad And it's on me to basically keep the team honest that we did it.
16:28 Brad A lot of times I will go back and we'll just, we'll just say,
16:31 Brad okay, great. Like, you know, the stuff like have two people do code reviews,
16:34 Brad that's like going,
16:34 Brad okay,
16:34 Brad great. We all agree that's what we're going to do.
16:36 Brad Cool.
16:37 Brad And then I, we have a document, which is our SDLC. So it's, you know, it's a couple pages long, and I just go and update that. And we all agree that's what we're going to do now.
16:48 Brad On.
16:50 Brad And so I think that we have just barely enough process to keep us honest that we really do do the start stop keeps and
17:02 Brad We don't currently use a sort of external tracking system to force that.
17:08 Brad But I do think that...
17:11 Brad A lot of times you do need some kind of a accountability measure, an accountability roadblock slash process.
17:20 Brad And so putting it in your backlog is a good way to keep that accountable,
17:27 Brad putting it on a.
17:31 Brad You know,
17:31 Brad a report that you share with the leadership team is a good way of sort of forcing some kind of accountability.
17:40 Brad And I think you have to judge like.
17:44 Brad It's it depends type thing,
17:46 Brad right?
17:46 Brad Like you have to judge the stage and the size and the maturity of the team and the project to decide what type of accountability you want.
17:54 Brad But I totally think you should have some kind of an accountability forcing function.
18:01 Brad So I think that's the answer to the first two questions you had.
18:03 Scott Yep, yep.
18:04 Brad And then the second question, the third question, which is the doozy of the question is when it's a big monster task,
18:10 Brad then what do you do?
18:12 Brad And, you know, I think that that's kind of the age old question.
18:16 Brad You have to do that risk reward judgment conversation and you have to.
18:22 Brad make the appropriate business case for why the thing has to happen and if you can't make that business case in a convincing way to your stakeholders and your leadership team then probably it's not appropriate to do it right
18:39 Scott Though, let me give you a real-world example from my life in the last two days.
18:46 Scott So my team did something I'm very proud of. We're working on rolling out this new feature.
18:51 Scott It launches on Monday.
18:53 Scott It's pretty big.
18:54 Scott A lot of people have been working on it for a long time.
18:56 Scott It's a whole new offering that we're going to be offering.
18:59 Scott And the thing I'm especially proud of is actually yesterday they went and tested it in production.
19:10 Scott actually placed some real orders in production.
19:14 Scott uh we spent real money on it i mean it's gonna you know goes to companies expensing whatever but like you know we really did it and the engineering director leading this really pushed hard on it um and i'm really proud of her for doing so because and the others there were others advocating for it as well so that when we roll this thing out on monday we really know it's going to work and it found at least one pretty serious problem that we had not found in our staging environment
19:40 Scott And so there was a little discussion about that today,
19:43 Scott and the call came once again for a better staging environment.
19:48 Scott And in particular, when they talk about the better staging environment,
19:51 Scott there's not just they,
19:53 Scott I mean, we all talk about it.
19:54 Scott There's really a few things.
19:56 Scott One is the data that's in the staging environment.
19:59 Brad The reason we found this in production is because it was an edge case, not terribly edge case,
20:04 Brad but it was an edge case that the ordinary set of tests that you would do in a staging environment,
20:09 Brad you wouldn't think to test this scenario.
20:13 Brad That's the first thing.
20:14 Brad The second is integration with third-party systems,
20:18 Brad right?
20:21 Brad Stripe, Twilio, whatever. Those often aren't really live in your staging environment.
20:28 Brad This particular issue had to do with the particular user who was using the thing.
20:33 Brad Had their account had to be in a certain kind of way that would happen ordinarily through the course of business in our production environment. You just wouldn't think of doing it and staging.
20:44 Brad So okay,
20:45 Brad so we're having a little retro on this today,
20:47 Brad not a formal retro just like
20:49 Brad some people talking about like what went wrong yesterday and it was came back to what we really need to talk about the staging environment and I've heard that I hear that every six weeks for the last 20 years of my career
21:00 Scott Right.
21:01 Brad but
21:02 Scott That's why it was one of the problems I wanted to solve right out of the gate with this current project I'm on.
21:08 Brad it's a tough one it's a big one you know getting good seed data good staging data is real real tough
21:13 Brad and you mess it up and then you got to go reset it it's just an expensive hard difficult problem it's hard to make a business case for because what did it really cost us well it cost us some heartache today or if we had shipped this without testing it would have cost us maybe a little bit of bug in production but it's not like a multi-million dollar savings I guess over time it probably adds up but it's a tough one to make a case for
21:38 Scott Yep.
21:38 Scott I think, yeah,
21:40 Scott of course. Right.
21:42 Scott Like, I think that.
21:44 Scott You know, this is a good one of those, you know, we talked about this in probably one of our very first episodes, this notion of like, you know, tech debt compounds in a negative way and tech investment compounds in a positive way.
21:57 Brad Yeah, that's great.
21:59 Scott And just today in the dev team meeting,
22:03 Scott I had a team meeting where somebody made the case for like,
22:06 Scott oh, we've got this one file that's too freaking large. You know, it's it's it's a it's.
22:11 Scott it's a big huge file file that's just a mess and it's so big that like the llms don't like to even load it in you know because it complains about the number of tokens that get blown up in the llm when you try to load it and so like ai assisted coding is almost impossible on this file right
22:28 Brad How long is that file?
22:30 Scott it's like 4 000 lines it's a mess it should be that big
22:34 Scott Well, it's not for an 18-month-old code base. It's one of the oldest files, and it's already got,
22:42 Scott you know, it's the source of a bunch of tech debt,
22:44 Scott you know.
22:44 Brad I've seen,
22:45 Scott But
22:45 Brad yeah,
22:45 Scott to
22:45 Brad I've seen some long
22:46 Scott me,
22:46 Brad files.
22:46 Scott that's a good, yeah,
22:48 Scott sure.
22:49 Scott Yeah, in my career,
22:51 Scott that's not the longest file I've ever worked on, for sure.
22:55 Scott But, you know, that's another example where it's kind of like.
23:00 Scott if that shows up in the retrospective like yeah we really got to refactor that thing you know yeah it's a hard those are hard cases to make um coming out of the retrospective so yeah
23:12 Brad So that's sprint retrospective.
23:15 Brad Now, do you ever do sort of like project retrospectives?
23:20 Scott Um,
23:20 Scott we haven't I've done those before. Yeah.
23:22 Scott And I haven't done them in my current incarnation.
23:26 Brad Mm-hmm.
23:27 Scott But but yeah, I think project retrospectives are kind of also a really natural step and they but you know, I'll tell you what I was gonna, I think what's interesting is
23:45 Scott In all of these types of retrospectives, I feel like they're on a spectrum where the spectrum is likelihood of negative feelings.
23:58 Scott Right.
23:58 Scott Likelihood of stressful feelings where the sprint one is on the lowest likelihood of stressful feelings and the project one is probably in the middle and the, you know, the incident outage retrospective is on the far end of very likely negative feelings.
24:16 Brad It's true.
24:17 Brad Yeah.
24:20 Scott And so this to me brings up the other really important point, I think, and the other strong opinion I have about retrospectives is how important it is to have real trust and,
24:35 Scott you know, like real honest,
24:40 Scott authentic trust amongst the team.
24:44 Scott in all of these retrospectives and
24:47 Brad Yes.
24:47 Scott um i think that again this kind of you know this is another example of something that i feel like has really strong overlap with the traveling in the mountains part of my life which is it's so important that when you're in one of these low information high consequence environments
25:11 Scott It it's so important that you have as many human observers and human brains and, you know, working together and bringing their best selves to the table. And that means they've got to be able to be able to comfortably say like, hey,
25:32 Scott like Brad,
25:35 Scott like you made this decision.
25:38 Scott It negatively affected the project.
25:40 Brad Mm-hmm,
25:41 Brad mm
25:41 Scott And do that in a way, you know,
25:43 Scott I always tell you, I like to use the phrase egoless retrospective instead of blameless retrospective because,
25:47 Brad -hmm.
25:48 Scott you know, I need you to be able to say to me,
25:50 Scott hey,
25:51 Scott Brad,
25:51 Scott this is the facts of what happened, right?
25:55 Scott And the facts include people who made decisions.
25:59 Scott And
25:59 Brad Yeah,
25:59 Scott people who did things.
26:01 Scott And so we have to get to this place where people are comfortable saying like admitting that they did something that had a consequence or that the boss did something that had a consequence or the other team did something that had a consequence and not turn it into attacking. Right.
26:19 Brad yeah
26:19 Scott And not turn it into.
26:23 Scott an insult an injury right it needs to be looked at as a a truly healthy thing for all of us to interrogate the good and the bad so that we can be better
26:38 Brad Yeah,
26:39 Brad this doesn't work in the mountain scenario,
26:41 Brad but in the software scenario,
26:42 Brad if I can, I like it when the person who caused the problem,
26:48 Brad if they know they caused the problem.
26:50 Brad is the leader of the retrospective because they're saying look i did this and i'm trying to get better i'm trying for all of us to get better but like to some degree that gives them the ownership as opposed to sitting in a room where everyone's eyes are on them their eyes get to be on everybody else so to speak a little bit um doesn't always work out that way so when we when i do incident retrospectives i do have an opinion on this it's got three parts to it
27:15 Brad You always start with a statement.
27:17 Brad We're here to get better.
27:18 Brad We're not here to assign blame.
27:21 Brad And the way we get better is by inspecting this and by developing we're searching for working agreements and action items that we could put in place to get better.
27:29 Brad That's why we're here.
27:31 Brad You just start with that statement just to set the tone.
27:36 Brad Next step is timeline.
27:39 Brad And we're not inspecting for the problem.
27:42 Brad What happened? Literally what happened?
27:44 Brad Start with when did the alarm go off? An alarm went off at 1.51 p.m.
27:48 Brad Okay, then what happened next?
27:51 Brad Brian found it. Brian was the first one to see the alarm.
27:54 Brad Okay,
27:54 Brad what would he do?
27:55 Brad He paged Brad.
27:56 Brad Just go through it in a minute-by-minute basis, just literally just what were the steps that happened?
28:04 Brad And often there's a lot of confusion about what happened.
28:07 Scott Exactly.
28:08 Brad No,
28:09 Brad no, no, no.
28:09 Brad Brad didn't see the alarm.
28:11 Brad It was Alice over here that saw it first.
28:14 Brad Oh,
28:14 Brad I see. Okay.
28:15 Brad And she was already working on the incident by the time Brad got to it, you know, so like that unpacking is actually kind of valuable because a lot of times in these incidents, there is confusion, you know, nobody is quite sure what's going on.
28:29 Brad with systems and people and all sorts of stuff so start with just getting that written down and a lot of times people don't like to do that part they want to jump
28:36 Scott Oh,
28:37 Brad right into
28:37 Scott that's
28:37 Brad yeah
28:37 Scott the most no like that. I so that is the most one of the most important parts.
28:42 Scott You've got to have a full accounting of what happened. Right.
28:47 Scott And by the way, again,
28:48 Scott strong overlap with, you know, mountain environments like this is a critical component.
28:54 Scott component to any search and rescue after action report the very first thing that we do is we outline the timeline like when did the page come out when was the first team at the trailhead what did they take into the trailhead with them what did they you know when did they reach the subject you know it's just it's very important to have that that timeline because to your point in the moment
29:20 Scott It's very frenetic.
29:22 Brad Mm
29:23 Scott It's
29:23 Brad -hmm
29:23 Scott very high-paced. There's a lot of information that...
29:27 Scott And there's a lot of information that not everybody is aware of in the moment.
29:33 Scott And so being able to then really do that retrospective where you say,
29:36 Scott hey, these were all the streams that were happening at the same time,
29:39 Brad Mm-hmm.
29:39 Scott right?
29:40 Scott While Alice was working on solving,
29:42 Scott you know, patching the system, you know, Brad thought he was patching the system.
29:48 Scott And they both patched two different components that counteracted each other.
29:53 Brad Yeah,
29:53 Scott Right.
29:53 Brad yeah.
29:54 Scott Wow.
29:54 Scott That's really,
29:55 Scott that's an interesting insight.
29:57 Scott What, what,
29:58 Scott okay.
29:58 Scott So forget that the thing happened. In the thing happening, the way we responded possibly made things worse.
30:07 Brad Okay,
30:08 Brad so you've touched on something key, actually, that I forgot to mention.
30:12 Brad When I do these retrospectives, I always tell the team we're actually looking,
30:16 Brad we're solving,
30:16 Brad we're trying to solve for two things.
30:19 Brad not just how do we prevent this problem from happening again or get we're also looking at our response right and so when we're looking for action items and working agreements we're not only looking about where did the problem occur but how do we respond to it faster and you need the timeline for that also okay so after you get the timeline established then I like the five wise approach
30:43 Brad And I shouldn't say I like the five wise words. I pretty much demand it. Oh, there's something else.
30:48 Brad I really think it's important.
30:50 Brad You talked about it's frenetic.
30:52 Brad I try to get people next day.
30:54 Brad Next morning is when the retro happens,
30:56 Brad right?
30:57 Brad Don't wait. Make them wait till Monday if it's on a Friday or something like that. But don't wait a week because people just forget stuff,
31:03 Brad right?
31:04 Brad You forget details.
31:05 Scott At the very minimum, start the retrospective document as
31:08 Brad Yeah,
31:08 Scott soon
31:08 Brad that's
31:09 Scott as possible.
31:09 Brad right.
31:09 Scott Like there's been times when I've started the retrospective document mid-incident, right?
31:14 Scott Like there's an incident happening,
31:16 Scott boom,
31:16 Scott let's
31:16 Brad That's
31:16 Scott create the
31:17 Brad right,
31:17 Scott retrospective
31:17 Brad for sure.
31:18 Scott document right now and start documenting the timeline right now as it's happening.
31:22 Brad Yeah.
31:23 Brad And,
31:23 Brad you know, people sometimes use a Slack channel.
31:27 Brad Often I encourage people to get into a video call like this or whatever.
31:32 Brad Because the communication is better.
31:34 Brad Now I'm glad that at least we use Google Meet and you've got this Gemini note taker, it can start keeping track of that stuff for us as opposed to having to write it down.
31:43 Scott And by the way, can I speak to that for a second real quick? So one of the practices that we instituted at Health Tech Company,
31:51 Scott which I thought was very useful,
31:54 Scott was we essentially...
31:57 Scott created a role that we called like the incident incident commander basically
32:02 Brad Yeah, exactly.
32:02 Scott almost take and that that that role was um designed to uh
32:10 Scott be much like an incident commander is in the um in the star world of incident command system where it's like the incident commander is like sure they're the kind of like final arbiter of what's going to happen but they need to be above this they need to be above the fray and they need to be able to step back and take the 50 000 foot view um and make sure that they're delegating and deploying the experts to actually solve the problem but
32:35 Scott but they're sitting there going okay great cool 8 p.m alarm sounded and they can start they can basically kind of be in uh in these software environments they can tend to be on that zoom call kind of keeping track of everything
32:49 Brad Now,
32:49 Scott and being available to sort of like you know make calls when it's needed yeah
32:55 Brad just for the normies and muggles at home, you've said SAR a bunch of times.
33:01 Brad But
33:01 Scott search and rescue
33:02 Brad search and rescue,
33:03 Brad search and rescue,
33:04 Scott yeah
33:04 Brad which is very noble that you do that, by the way, I do not have the stomach to do that.
33:09 Brad Well,
33:10 Brad I do not have the stomach to show up at the end when you get to the rescue part of it.
33:15 Brad Okay,
33:16 Brad so yeah,
33:18 Brad so I like to start with the five whys and people sometimes start too far in the process.
33:23 Brad I like to start way up.
33:24 Brad Why did you get? Why did the alert go off?
33:27 Brad Sometimes it's just a customer service or a person,
33:29 Brad why didn't Sally from customer service call you, Brad?
33:33 Brad Well, she called me because she thought the site was down.
33:36 Brad Why did she think the site was down?
33:37 Brad And just be,
33:39 Brad I try to really encourage you,
33:41 Brad be as plaintive and almost like stupid as possible. Why did she think the site was down?
33:46 Brad Well,
33:46 Scott Right.
33:46 Brad because their browser said 504 gateway timeout.
33:52 Scott Yeah.
33:53 Brad So, and really push that. And then along the way, and keep on pushing until you get to true causes.
34:00 Brad And I find that it has taken some training in my experience. I learned and then I've had to teach other people what is really the root cause.
34:10 Brad A lot of times people stop before they get to the real root cause.
34:13 Scott What's an example of a team that stops too early or, you know?
34:18 Brad Okay,
34:18 Brad here we had something like this happen recently.
34:20 Brad So we have a number of staging environments and you can kind of reserve a staging environment to test something.
34:26 Brad And we had a confluence of things happen recently.
34:28 Brad There's kind of a little bit of a comedy of errors.
34:33 Brad Person A was doing a demo for some stakeholders on a staging environment.
34:38 Brad Person B was doing some maintenance on the staging environment,
34:42 Brad but person C was the person who had the staging environment checked out.
34:47 Scott Oh, no.
34:48 Brad So it was just a total comedy of errors. So you could say.
34:54 Brad Well,
34:55 Brad why did the demo,
34:56 Brad why did the staging server crash during the demo?
34:59 Brad Oh,
34:59 Brad it's because I'll just use some names. It's because Mike rebooted it as part of this maintenance and you're done.
35:07 Brad But really there was some other thing.
35:09 Brad Well, wait a second.
35:11 Brad You know, we had to get to, well, it was Brian that had the staging server reserve.
35:15 Brad Why was Mike even working on that thing in the first place, right?
35:18 Brad What was it about our, this is maybe not the best example,
35:21 Brad but to me, that's not the real root cause that, you know, Mike rebooted it. The root cause is that somehow our reservation system failed,
35:30 Brad right?
35:31 Scott Yeah, and I actually, so that's really interesting,
35:33 Scott Scott,
35:34 Scott because I think that kind of dovetails into one of the things I have.
35:38 Scott Like, I'm not going to say I don't like the five wise process.
35:41 Scott I do like the five wise process, the idea of like,
35:44 Scott let's keep pushing on the wise until
35:46 Brad Digging,
35:46 Scott we figure
35:47 Brad keep digging,
35:47 Scott out a true,
35:47 Brad yeah.
35:47 Scott keep digging.
35:49 Scott But one of the things I see that teams struggle with is picking the first wrong,
35:54 Scott the first why they start with is the wrong thing to start with.
35:58 Scott Right.
35:59 Scott And so like, for example, in your story where, you know, person A was doing a demo, person B rebooted the system and person C was the one that had it checked out.
36:09 Brad Yeah.
36:09 Scott Like, I feel like one of the root causes you're probably have to get to is to your point, like something went wrong with the reservation system and somehow like I guess you can get there with the five whys, but I feel like you end up.
36:23 Scott end up having to probably do,
36:26 Scott you know.
36:27 Scott uh at least five different starts and you know five wise off of each of them so it's like 25 wise to have found that like because one of the wise might have been well well why did why did mike think that it was okay to do a demo in the demo environment or why did mike think it wasn't going to reboot in the middle of his demo oh well because he was doing a demo but why did he think he was the one doing you know who had the space
36:54 Scott in space you know oh because actually brian checked it out and told mike that brian that he could use brian's checkout right
37:02 Brad Yeah.
37:03 Scott or whatever you know what i mean and it's like to me that's what i've i've always kind of like i always try to say like look what really matters is we need to try to figure out what the root cause is of
37:15 Scott But many things probably went wrong, and they all have their own root causes.
37:19 Brad Yes, there are branches.
37:20 Brad Frequently there are branches.
37:22 Brad In fact, my experience, most problems that are more than just a typical bug are two things happen simultaneously when the right hand didn't know what the left was doing,
37:34 Brad you
37:35 Scott Yep, which is
37:36 Brad know?
37:36 Scott why the timeline is so important.
37:39 Brad That's right.
37:40 Scott Right.
37:41 Scott And it's the integrated timeline.
37:43 Brad And then you end up with some working agreements and action items and you put them somewhere,
37:49 Brad you know, like we like to put, you know, we use Jira, like just stick them in Jira. You tag them as retroactive items,
37:56 Brad retro items so that you can.
37:59 Brad measure your progress um like you uh we're the company i'm at now we're we have an sdlc document that needs a lot of refinements we're rebooting that process to refine it but i think those working agreements go in the sdlc and um that's the cycle and it's worked pretty well for me i've been pretty happy with it kind of like the five wires process i guess you don't like it okay i guess there's one there's one area we disagree red we finally found it
38:26 Scott it's not that i don't like it i don't i think it needs there's more to it than that
38:31 Brad What
38:31 Scott so i
38:31 Brad do you think? Do you have a different approach you like?
38:35 Scott don't have a better one so maybe the reason i'm putting out there that i i feel like the five wise uh is insufficient is a plea to hey folks do you have a better idea out there like how do we make sure that it gets you know that we really cover the ground that
38:52 Scott is the very rich nature of why cascading failures happen in the first place.
38:57 Brad I think what you're trying to say,
38:59 Brad Brett,
38:59 Brad is after you smash the subscribe button,
39:01 Brad when you're leaving that five star review,
39:03 Brad leave a comment in the five star review for a better way to
39:08 Scott I
39:08 Brad do retros.
39:08 Scott agree.
39:08 Scott Let's do that.
39:10 Scott And I'll say one last thing about this.
39:13 Scott I feel like this conversation about retros.
39:17 Scott dovetails really nicely with our conversation several episodes back about building trust because you just can't do these you can't really do good retros if you don't have a culture of trust
39:34 Brad Yeah.
39:36 Brad Yeah.
39:36 Brad And I would say it is harder to build a culture of trust when you're fully remote.
39:44 Brad which makes these retros a little bit harder yeah
39:49 Scott It's hard to build a culture of trust.
39:51 Brad it is hard to build a culture of trust yeah
39:53 Scott You could stop there.
39:56 Brad do you think it's hard well let's let's just go up a second do you think it's not as
39:59 Brad It's not harder to build a culture of trust virtually.
40:02 Scott Well, I've told you this, and we kind of talked about this in the building trust conversation we had.
40:09 Scott I feel like remote.
40:12 Scott forces you to be way more intentional about
40:14 Brad Yeah, for sure.
40:15 Scott building trust.
40:17 Brad Yeah.
40:17 Scott And I think the key to building trust is being really intentional.
40:22 Scott And so even in a in a in person world or a hybrid world,
40:26 Scott if you're not intentional,
40:28 Scott then it doesn't help you that you're in person.
40:31 Scott In fact, it might hurt you that you're in person.
40:34 Scott And so to me.
40:36 Scott Maybe I think it's
40:40 Scott almost as if forcing yourself to be virtual means you have to be much more intentional and that means you're going to put the investment in and it's going to get the payoff.
40:53 Scott So that's my take.
40:54 Scott I think I'm not going to say it's easier,
40:56 Scott but the tools that really work in building trust.
41:02 Scott Those tools are available to you in the virtual world as much as they are in the in-person world.
41:10 Scott That's my opinion.
41:12 Brad All right.
41:13 Brad Is that where we should leave it?
41:15 Scott That's where we should leave it.
41:17 Brad All right. Well, thanks for listening,
41:18 Brad everybody.
41:19 Brad It's been fun to have you don't
41:20 Scott Yeah.
41:21 Brad forget to smash that subscribe button
41:25 Scott Rate and review.
41:26 Scott Follow us on all the social media platforms.
41:28 Brad All the social media platforms and tell all your friends
41:31 Scott All right.
41:32 Scott Good
41:32 Brad Talk
41:32 Scott talking to to you,
41:33 Brad you later,
41:33 Scott Scott.
41:33 Brad bud.
41:33 Brad Bye
41:35 Scott This episode of How Many CTOs Does It Take podcast was edited by Safran Epagow.
41:40 Scott Our theme song was AI Generated by Brad.
41:43 Scott Please follow us on all the social media platforms at at how many CTOs pod you can find us on YouTube LinkedIn Instagram Facebook threads X tick-tock and patreon or at our website at how many CTOs pod calm and remember to rate review and subscribe on the podcast platform of your choice thanks for listening