Lessons from the Mountains: Retrospectives and the Secret Sauce of High Performance Teams

0:00 Brad: I always try to say, like, look, what really matters is we need to try to figure out what the root cause is.

0:05 But many things probably went wrong and they all have their own root causes.

0:28 Scott: place yesterday that do you like cocktails do you have a cocktail guy have

0:32 Brad: A hundred percent, dude.

0:34 Scott: you been to this place called cannon on 12th avenue by seattle university i'd

0:38 Brad: Yes, that's a classic.

0:39 The

0:40 Scott: never been there's a classic how long has it been there oh

0:43 Brad: 20 years.

0:44 Scott: really well

0:46 Brad: More than 20 years, maybe.

0:48 Scott: i went there for the first time last night and i ordered a gin drink

0:54 And they literally serve it in a bathtub-shaped cup.

0:56 Brad: love it yep that's one of their yep that's one of their their things i think they also have like uh

1:04 they got like a drink that's like the they serve in a um blood bag

1:09 Scott: I didn't see that.

1:10 I saw a drink they served

1:11 Brad: yeah

1:11 Scott: in a Chinese takeout thing.

1:14 So it was super fun.

1:15 We had a really old whiskey.

1:17 Well,

1:18 actually, it wasn't that old. It was just 23 years old.

1:21 I wanted the whiskey that was like 130 years old, but I couldn't bring myself to spending $2,000 on a single drink whiskey.

1:30 Brad: Yeah, that's special occasion whiskey.

1:33 Scott: Yeah,

1:33 I don't know if there's an occasion special enough in my life for that.

1:40 Brad: It's all a matter of scale,

1:41 I guess.

1:41 Scott: That's right.

1:43 Brad: Welcome to another episode of How Many CTOs Does It Take podcast.

1:46 I'm Brad Hefta-Gaub and my co-host.

1:48 Scott: My name is Scott Poirier.

1:50 Brad: Awesome.

1:50 And today I think we talked about on text,

1:54 we chatted with each other a little bit about the idea of talking about retrospectives.

1:59 Scott: Retrospectives. Love it.

2:00 The most important part of the Agile process.

2:05 Brad: I think so.

2:06 Scott: Yeah, I do too.

2:08 Brad: So I will tell you part of my,

2:12 I have very strong opinions about retrospectives.

2:16 And part of those opinions come from the,

2:21 one of my hobbies slash one of the things I spend a lot of time doing, which is traveling in the mountains.

2:31 And in the winter, of course, that means traveling in avalanche terrain.

2:36 And as you know, Scott,

2:39 I'm a ski patroller and a SAR unit member.

2:41 And one of the things I do is I'm a professional member of the American Avalanche Association and I'm an avalanche educator.

2:49 So I teach avalanche skills to search and rescue unit members and ski patrollers.

2:56 And the notion of a debrief after you travel through avalanche terrain and the notion of a high quality after action discussion and after action report on SAR missions or traveling in the mountains is like an essential component.

3:19 to making sure that when you travel in low information, high consequence environments,

3:28 you get better and hopefully prevent dire consequences,

3:37 right?

3:38 So the idea of really sort of trying to interrogate, well,

3:42 what decisions did we make?

3:45 And did we just get lucky or were those decisions,

3:49 you know, proactive in our success or were they things that if we interrogate them and evaluate,

3:57 like maybe we could make you could get better at how we did that?

4:02 And so.

4:04 I see incredible overlap between decision making in the mountains and decision making as it relates to writing software.

4:14 Scott: Yeah.

4:15 I agree.

4:17 I worked at a company once which had some core values and they were really well done core values amongst all the companies I've worked at.

4:26 It's I'd say the top two.

4:28 It's number one and number two. And one of its core values was simply get better every day.

4:34 Brad: Yeah.

4:35 Scott: And I think this fits right with that, which is.

4:40 The spirit of the retro is getting better and

4:44 Brad: Yeah.

4:45 Scott: the best teams I've been a part of are striving to get better all the time.

4:50 Brad: Yeah.

4:51 Growth mindset.

4:52 Scott: Yeah,

4:52 growth mindset.

4:54 Once we tried to have a retro for every error,

5:01 we stopped after a few weeks because it was just like, oh, it was too much.

5:05 But if you think about it, why not?

5:07 Brad: You mean like every 500 in the web server log?

5:11 Scott: something like that i think it was for every alert that went off actually is what we did because we asked ourselves let's do a retro on this and it ended up being like multiple retros every day and it was it didn't work it was a good idea but we were asking ourselves the question yeah every 500 in the logs everything in the logs that's an error

5:35 Every time the alarms go off like these are problems that are happening,

5:38 they might not be customer impacting problems,

5:42 but why wouldn't you do a retro and understand what is it that caused this error to happen?

5:48 Any bug,

5:49 why wouldn't you do if there were enough time in the day,

5:51 why wouldn't you do a retro for every single bug?

5:54 Say, well,

5:55 how did it come to be that this happened?

5:58 Brad: Totally. I think that there's a spirit to that, that if you did have enough time, you would want to interrogate every negative consequence and ask yourself like what caused it.

6:09 And I certainly support that idea.

6:11 And I think that's one of the first things that I mean, to your point, like given enough time, right? Like I think also it benefits being able to like step back and kind of look at a collection of things that happened and say.

6:23 and try to interrogate them as a collection and say like well what what do these all have in common what were our common behaviors that might have led to them right

6:32 Scott: Now there's a thing I think I know you're going to say I'm going to set you up. I'll give you a softball here, Brad.

6:37 Brad: Yeah,

6:37 Scott: I know that you're going to say, Scott,

6:40 you were just interrogating the things that went wrong.

6:43 Brad: exactly.

6:43 Scott: Why didn't you do a retro on every single PR where nothing went wrong and ask what did we do on this PR to make it so that it was successful?

6:53 Brad: 100%. That's exactly what I was about to say to you, which is, of course, comes from traveling in the mountains, right?

7:00 every time you get home every time you get back like that was a success and those are the times that you really have to ask yourself well why did we what was why were we successful like what did we do differently that was successful and I think that in software engineering like the teams that have the amazing sprint where they got all of their tasks done and they delivered a bunch of value and

7:24 And then if they just if they if they end their sprint and their closing ceremonies with,

7:30 you know, you know, popping champagne corks and slapping each other on the back, congratulating each other, that kind of misses the most important point, I think, to the to the retro,

7:42 which is.

7:44 Yeah, but why? Why did we succeed this time?

7:46 What was it that we did that made us successful?

7:49 Were we particularly good at our estimates? Did we have was it was it just some heroic effort by part of the team to accomplish this goal or had we actually set ourselves up for success?

8:04 From the things that we had done during the planning process. Right. And trying to really understand that, because if it's if the answer is, yay, we did it, but it only happened because of Scott's heroic efforts.

8:16 Well.

8:18 That's maybe looks like a win on paper,

8:21 but is not a very sustainable win.

8:25 Scott: Yes.

8:25 Brad: Right.

8:25 Scott: Yeah.

8:26 Brad: Like we can't count on Scott always doing the heroic thing every sprint.

8:30 Scott: That's right.

8:30 That's right.

8:31 So, okay,

8:32 I want to get into this a little bit more practical here, though.

8:35 We could talk about this theoretically all day long.

8:38 I can think of three different types of retros, sprint retro.

8:43 So do that every one week, two weeks, three weeks, however often you have your sprint cadence.

8:47 A project retro.

8:50 So, or maybe you might call it an epic retro, but a project retro,

8:54 which is something might last multiple sprints, but now we've sort of completed this major milestone.

8:58 Let's go back and look on the whole thing.

9:00 And then I'll call it an incident retro, which is, well, there was some sort of problem in our operational environment.

9:06 Let's look at that.

9:07 I have a fairly strong opinion about how the last one should go.

9:14 The postmortem type incident retro,

9:17 right?

9:18 Brad: Mm-hmm.

9:20 Scott: I don't have as strong of an opinion on how the first two should go.

9:24 I'd love to hear,

9:24 like, how do you do your sprint retros? How have you found that those are most effective?

9:29 Brad: Yeah, I'll tell you how we're doing it today,

9:31 and I feel like it works really, really well for this team that I'm working with now.

9:36 I think I would.

9:38 Definitely lump them into the category of a high performance engineering team.

9:42 They're all very growth mindset,

9:44 very open minded. They're all very introspective about and they thrive in interrogating what went well and what didn't go well.

9:53 And so here's how we do it.

9:55 Our sprints are two weeks.

9:57 We have 90 minutes dedicated on the second Friday. Our sprints start on Mondays and ends on Friday,

10:06 you know, 10 business days later.

10:09 We happen to be using a tool that is a plug-in to Trello for our, it's called retrospectives, it's a plug-in to Trello, but all it really is is a.

10:18 everybody gets to put in little cards that are either in the column of what went well or what could have improved.

10:29 And so here's our process.

10:31 We start the retro and we say,

10:34 all right, everybody,

10:34 boom,

10:35 start adding your cards.

10:37 And when you're done adding cards,

10:39 raise your hand, right?

10:41 This tool happens to have a little button to say I'm ready.

10:44 And we tend to get, you know, it takes us about,

10:49 I would say.

10:50 somewhere between five to ten minutes for everybody to kind of get all their typing in and get kind of done like they almost everybody almost comes in with like a they know what they're going to type and we actually will open up the tool ahead of time like sometimes I'll just like I'll open it up on Thursday and just like hey start adding stuff you know and so it takes about five to ten minutes for everybody add their items and then when everyone's ready we

11:15 click the next button and everyone can see all the cards at that point and it actually does show the people who added the cards so it's not anonymous but with my team that works okay because we've got such a high level of trust with each other that nobody minds that like brad said oh we completely screwed that up right

11:34 And the next step is we group the things together into kind of topics and so and and what went well and what could have gone better can be grouped together into the same topic. We don't we don't make it so that there is a lot of times you have something where it's like, let's say the topic was like this past retrospective we had a topic that was.

11:55 fire drill slash new feature x and what had happened in that case was it was a heroic effort to get the feature over the finish line um and people thought felt like it was a fire drill but in that grouping of the fire drill there was a whole bunch of like what didn't go well or what could have gone better and then there was several people that were like it went well like people jumped in and did a heroic thing right so we grouped them into categories

12:20 And we usually do that live,

12:23 like anybody can group things together.

12:25 I tend to be the one kind of leading

12:28 Scott: Mm

12:28 Brad: it, so-hmm. I group them together.

12:31 And that usually takes about two or three minutes.

12:35 And then when we are all happy with, yeah,

12:37 those feel like the right groups,

12:38 then we click the next button.

12:41 And we go, you know, kind of to anonymous mode again and everybody gets we've chosen four votes. They get to vote for four things, four of the topics that they really want to make sure we talk about.

12:52 Right.

12:53 And there tend to be like somewhere between eight or 10 groupings once we're, you know, once we've got,

12:57 we have like,

12:59 you know, I'd say like 50 or 60 items that get grouped into 10 things.

13:03 And so now you're, you're voting on what are the four of these 10 things that I think are most important to talk about?

13:10 And you can vote multiple times on the same topic. So if you're like, I really want to make sure that this fire drill topic is the thing we talk about.

13:19 You can put all your votes on that.

13:22 Then when everyone's done voting,

13:24 we switch into back into kind of shared interactive mode.

13:31 We talk about it. We talk about the topic. The one that gets the highest votes is the first one that we talk about.

13:36 And we try really hard to always like sometimes there's a little bit like, hey,

13:41 can you remind us exactly what happened? Hey,

13:43 you know, Brad,

13:44 when you said this, what did you mean by that in your in your description of the problem?

13:50 But we mostly say let's focus on start stop keeps.

13:55 Right.

13:56 So if it's a good thing or a bad thing, like if it didn't go well or it could have gone better or went well, we still ask the question,

14:03 okay,

14:03 well,

14:04 what about this do we want to start doing?

14:06 What about this do we want to stop doing?

14:08 And what about this do we want to keep doing?

14:11 And we generate start-stop keeps for every one of our topics that we talk about.

14:17 And then when we're done with all that, that's it.

14:19 We have our list of start-stop keeps,

14:20 and then we turn around and file those back into our process.

14:24 Scott: Okay, I have two questions.

14:26 Actually, I'm sorry, I have three questions.

14:28 Where do the stop-start keeps go?

14:31 Do you have any mechanism?

14:34 For, you know, I guess I was going to say, do you have any mechanism for tracking if anything happens with them, which there's another flavor of that, which is how do you make it such that things happen?

14:44 Brad: Yeah.

14:45 Scott: And the third, which is this is the one where I often get puzzled, sometimes a start,

14:52 especially,

14:52 or sometimes a stop, but especially a start is pretty big.

14:59 For you know, it might be I'm just making this example up.

15:02 We really need to replace the whole payment system

15:05 Brad: Yeah.

15:06 Scott: Right like

15:06 Brad: Yeah,

15:06 for sure.

15:07 Scott: somebody

15:07 Brad: Right.

15:08 Scott: Somebody

15:08 Brad: We can't use that third party off provider. It's caused three of these these outages in the last month.

15:13 Scott: Which is not like

15:16 So that's not that's different than like,

15:17 hey,

15:18 we need to agree to have two people do code reviews very simple where we can put that in place.

15:22 Hey, we need to do this giant and I always get stuck on the giant project that comes out of this.

15:27 So how do you handle those?

15:28 So those are my three questions.

15:29 My three questions are, what do you do with the start stop keeps?

15:33 How do you kind of

15:35 prioritize or track to see if something's happening because nothing's happening with them why are you doing it in the first Yeah. place and then how do you handle the

15:41 Brad: The

15:41 Scott: big ones

15:42 Brad: big monster ones.

15:42 Yeah.

15:43 Well,

15:44 um,

15:45 So let me answer those like like actually practically what we're doing today and

15:50 Scott: Sure.

15:50 Brad: then what my sort of general opinions on this are.

15:53 So first of all, the tool that we use happens to be integrated with Trello, which is what we're using for project task management anyway.

16:00 And it actually has a feature where you can go create cards out of all these things.

16:05 Scott: Cool.

16:06 Brad: Now, we tend to actually not do that. Instead,

16:10 what we do is.

16:12 And we,

16:12 it's kind of on me as the, you know, leader,

16:18 CTO,

16:19 engineering manager,

16:20 scrum master,

16:21 right? We're a small enough team that I kind of wear all those hats,

16:23 right?

16:23 And it's on me to basically keep the team honest that we did it.

16:28 A lot of times I will go back and we'll just, we'll just say,

16:31 okay, great. Like, you know, the stuff like have two people do code reviews,

16:34 that's like going,

16:34 okay,

16:34 great. We all agree that's what we're going to do.

16:36 Cool.

16:37 And then I, we have a document, which is our SDLC. So it's, you know, it's a couple pages long, and I just go and update that. And we all agree that's what we're going to do now.

16:48 On.

16:50 And so I think that we have just barely enough process to keep us honest that we really do do the start stop keeps and

17:02 We don't currently use a sort of external tracking system to force that.

17:08 But I do think that...

17:11 A lot of times you do need some kind of a accountability measure, an accountability roadblock slash process.

17:20 And so putting it in your backlog is a good way to keep that accountable,

17:27 putting it on a.

17:31 You know,

17:31 a report that you share with the leadership team is a good way of sort of forcing some kind of accountability.

17:40 And I think you have to judge like.

17:44 It's it depends type thing,

17:46 right?

17:46 Like you have to judge the stage and the size and the maturity of the team and the project to decide what type of accountability you want.

17:54 But I totally think you should have some kind of an accountability forcing function.

18:01 So I think that's the answer to the first two questions you had.

18:03 Scott: Yep, yep.

18:04 Brad: And then the second question, the third question, which is the doozy of the question is when it's a big monster task,

18:10 then what do you do?

18:12 And, you know, I think that that's kind of the age old question.

18:16 You have to do that risk reward judgment conversation and you have to.

18:22 make the appropriate business case for why the thing has to happen and if you can't make that business case in a convincing way to your stakeholders and your leadership team then probably it's not appropriate to do it right

18:39 Scott: Though, let me give you a real-world example from my life in the last two days.

18:46 So my team did something I'm very proud of. We're working on rolling out this new feature.

18:51 It launches on Monday.

18:53 It's pretty big.

18:54 A lot of people have been working on it for a long time.

18:56 It's a whole new offering that we're going to be offering.

18:59 And the thing I'm especially proud of is actually yesterday they went and tested it in production.

19:10 actually placed some real orders in production.

19:14 uh we spent real money on it i mean it's gonna you know goes to companies expensing whatever but like you know we really did it and the engineering director leading this really pushed hard on it um and i'm really proud of her for doing so because and the others there were others advocating for it as well so that when we roll this thing out on monday we really know it's going to work and it found at least one pretty serious problem that we had not found in our staging environment

19:40 And so there was a little discussion about that today,

19:43 and the call came once again for a better staging environment.

19:48 And in particular, when they talk about the better staging environment,

19:51 there's not just they,

19:53 I mean, we all talk about it.

19:54 There's really a few things.

19:56 One is the data that's in the staging environment.

19:59 Brad: The reason we found this in production is because it was an edge case, not terribly edge case,

20:04 but it was an edge case that the ordinary set of tests that you would do in a staging environment,

20:09 you wouldn't think to test this scenario.

20:13 That's the first thing.

20:14 The second is integration with third-party systems,

20:18 right?

20:21 Stripe, Twilio, whatever. Those often aren't really live in your staging environment.

20:28 This particular issue had to do with the particular user who was using the thing.

20:33 Had their account had to be in a certain kind of way that would happen ordinarily through the course of business in our production environment. You just wouldn't think of doing it and staging.

20:44 So okay,

20:45 so we're having a little retro on this today,

20:47 not a formal retro just like

20:49 some people talking about like what went wrong yesterday and it was came back to what we really need to talk about the staging environment and I've heard that I hear that every six weeks for the last 20 years of my career

21:00 Scott: Right.

21:01 Brad: but

21:02 Scott: That's why it was one of the problems I wanted to solve right out of the gate with this current project I'm on.

21:08 Brad: it's a tough one it's a big one you know getting good seed data good staging data is real real tough

21:13 and you mess it up and then you got to go reset it it's just an expensive hard difficult problem it's hard to make a business case for because what did it really cost us well it cost us some heartache today or if we had shipped this without testing it would have cost us maybe a little bit of bug in production but it's not like a multi-million dollar savings I guess over time it probably adds up but it's a tough one to make a case for

21:38 Scott: Yep.

21:38 I think, yeah,

21:40 of course. Right.

21:42 Like, I think that.

21:44 You know, this is a good one of those, you know, we talked about this in probably one of our very first episodes, this notion of like, you know, tech debt compounds in a negative way and tech investment compounds in a positive way.

21:57 Brad: Yeah, that's great.

21:59 Scott: And just today in the dev team meeting,

22:03 I had a team meeting where somebody made the case for like,

22:06 oh, we've got this one file that's too freaking large. You know, it's it's it's a it's.

22:11 it's a big huge file file that's just a mess and it's so big that like the llms don't like to even load it in you know because it complains about the number of tokens that get blown up in the llm when you try to load it and so like ai assisted coding is almost impossible on this file right

22:28 Brad: How long is that file?

22:30 Scott: it's like 4 000 lines it's a mess it should be that big

22:34 Well, it's not for an 18-month-old code base. It's one of the oldest files, and it's already got,

22:42 you know, it's the source of a bunch of tech debt,

22:44 you know.

22:44 Brad: I've seen,

22:45 Scott: But

22:45 Brad: yeah,

22:45 Scott: to

22:45 Brad: I've seen some long

22:46 Scott: me,

22:46 Brad: files.

22:46 Scott: that's a good, yeah,

22:48 sure.

22:49 Yeah, in my career,

22:51 that's not the longest file I've ever worked on, for sure.

22:55 But, you know, that's another example where it's kind of like.

23:00 if that shows up in the retrospective like yeah we really got to refactor that thing you know yeah it's a hard those are hard cases to make um coming out of the retrospective so yeah

23:12 Brad: So that's sprint retrospective.

23:15 Now, do you ever do sort of like project retrospectives?

23:20 Scott: Um,

23:20 we haven't I've done those before. Yeah.

23:22 And I haven't done them in my current incarnation.

23:26 Brad: Mm-hmm.

23:27 Scott: But but yeah, I think project retrospectives are kind of also a really natural step and they but you know, I'll tell you what I was gonna, I think what's interesting is

23:45 In all of these types of retrospectives, I feel like they're on a spectrum where the spectrum is likelihood of negative feelings.

23:58 Right.

23:58 Likelihood of stressful feelings where the sprint one is on the lowest likelihood of stressful feelings and the project one is probably in the middle and the, you know, the incident outage retrospective is on the far end of very likely negative feelings.

24:16 Brad: It's true.

24:17 Yeah.

24:20 Scott: And so this to me brings up the other really important point, I think, and the other strong opinion I have about retrospectives is how important it is to have real trust and,

24:35 you know, like real honest,

24:40 authentic trust amongst the team.

24:44 in all of these retrospectives and

24:47 Brad: Yes.

24:47 Scott: um i think that again this kind of you know this is another example of something that i feel like has really strong overlap with the traveling in the mountains part of my life which is it's so important that when you're in one of these low information high consequence environments

25:11 It it's so important that you have as many human observers and human brains and, you know, working together and bringing their best selves to the table. And that means they've got to be able to be able to comfortably say like, hey,

25:32 like Brad,

25:35 like you made this decision.

25:38 It negatively affected the project.

25:40 Brad: Mm-hmm,

25:41 mm

25:41 Scott: And do that in a way, you know,

25:43 I always tell you, I like to use the phrase egoless retrospective instead of blameless retrospective because,

25:47 Brad: -hmm.

25:48 Scott: you know, I need you to be able to say to me,

25:50 hey,

25:51 Brad,

25:51 this is the facts of what happened, right?

25:55 And the facts include people who made decisions.

25:59 And

25:59 Brad: Yeah,

25:59 Scott: people who did things.

26:01 And so we have to get to this place where people are comfortable saying like admitting that they did something that had a consequence or that the boss did something that had a consequence or the other team did something that had a consequence and not turn it into attacking. Right.

26:19 Brad: yeah

26:19 Scott: And not turn it into.

26:23 an insult an injury right it needs to be looked at as a a truly healthy thing for all of us to interrogate the good and the bad so that we can be better

26:38 Brad: Yeah,

26:39 this doesn't work in the mountain scenario,

26:41 but in the software scenario,

26:42 if I can, I like it when the person who caused the problem,

26:48 if they know they caused the problem.

26:50 is the leader of the retrospective because they're saying look i did this and i'm trying to get better i'm trying for all of us to get better but like to some degree that gives them the ownership as opposed to sitting in a room where everyone's eyes are on them their eyes get to be on everybody else so to speak a little bit um doesn't always work out that way so when we when i do incident retrospectives i do have an opinion on this it's got three parts to it

27:15 You always start with a statement.

27:17 We're here to get better.

27:18 We're not here to assign blame.

27:21 And the way we get better is by inspecting this and by developing we're searching for working agreements and action items that we could put in place to get better.

27:29 That's why we're here.

27:31 You just start with that statement just to set the tone.

27:36 Next step is timeline.

27:39 And we're not inspecting for the problem.

27:42 What happened? Literally what happened?

27:44 Start with when did the alarm go off? An alarm went off at 1.51 p.m.

27:48 Okay, then what happened next?

27:51 Brian found it. Brian was the first one to see the alarm.

27:54 Okay,

27:54 what would he do?

27:55 He paged Brad.

27:56 Just go through it in a minute-by-minute basis, just literally just what were the steps that happened?

28:04 And often there's a lot of confusion about what happened.

28:07 Scott: Exactly.

28:08 Brad: No,

28:09 no, no, no.

28:09 Brad didn't see the alarm.

28:11 It was Alice over here that saw it first.

28:14 Oh,

28:14 I see. Okay.

28:15 And she was already working on the incident by the time Brad got to it, you know, so like that unpacking is actually kind of valuable because a lot of times in these incidents, there is confusion, you know, nobody is quite sure what's going on.

28:29 with systems and people and all sorts of stuff so start with just getting that written down and a lot of times people don't like to do that part they want to jump

28:36 Scott: Oh,

28:37 Brad: right into

28:37 Scott: that's

28:37 Brad: yeah

28:37 Scott: the most no like that. I so that is the most one of the most important parts.

28:42 You've got to have a full accounting of what happened. Right.

28:47 And by the way, again,

28:48 strong overlap with, you know, mountain environments like this is a critical component.

28:54 component to any search and rescue after action report the very first thing that we do is we outline the timeline like when did the page come out when was the first team at the trailhead what did they take into the trailhead with them what did they you know when did they reach the subject you know it's just it's very important to have that that timeline because to your point in the moment

29:20 It's very frenetic.

29:22 Brad: Mm

29:23 Scott: It's

29:23 Brad: -hmm

29:23 Scott: very high-paced. There's a lot of information that...

29:27 And there's a lot of information that not everybody is aware of in the moment.

29:33 And so being able to then really do that retrospective where you say,

29:36 hey, these were all the streams that were happening at the same time,

29:39 Brad: Mm-hmm.

29:39 Scott: right?

29:40 While Alice was working on solving,

29:42 you know, patching the system, you know, Brad thought he was patching the system.

29:48 And they both patched two different components that counteracted each other.

29:53 Brad: Yeah,

29:53 Scott: Right.

29:53 Brad: yeah.

29:54 Scott: Wow.

29:54 That's really,

29:55 that's an interesting insight.

29:57 What, what,

29:58 okay.

29:58 So forget that the thing happened. In the thing happening, the way we responded possibly made things worse.

30:07 Brad: Okay,

30:08 so you've touched on something key, actually, that I forgot to mention.

30:12 When I do these retrospectives, I always tell the team we're actually looking,

30:16 we're solving,

30:16 we're trying to solve for two things.

30:19 not just how do we prevent this problem from happening again or get we're also looking at our response right and so when we're looking for action items and working agreements we're not only looking about where did the problem occur but how do we respond to it faster and you need the timeline for that also okay so after you get the timeline established then I like the five wise approach

30:43 And I shouldn't say I like the five wise words. I pretty much demand it. Oh, there's something else.

30:48 I really think it's important.

30:50 You talked about it's frenetic.

30:52 I try to get people next day.

30:54 Next morning is when the retro happens,

30:56 right?

30:57 Don't wait. Make them wait till Monday if it's on a Friday or something like that. But don't wait a week because people just forget stuff,

31:03 right?

31:04 You forget details.

31:05 Scott: At the very minimum, start the retrospective document as

31:08 Brad: Yeah,

31:08 Scott: soon

31:08 Brad: that's

31:09 Scott: as possible.

31:09 Brad: right.

31:09 Scott: Like there's been times when I've started the retrospective document mid-incident, right?

31:14 Like there's an incident happening,

31:16 boom,

31:16 let's

31:16 Brad: That's

31:16 Scott: create the

31:17 Brad: right,

31:17 Scott: retrospective

31:17 Brad: for sure.

31:18 Scott: document right now and start documenting the timeline right now as it's happening.

31:22 Brad: Yeah.

31:23 And,

31:23 you know, people sometimes use a Slack channel.

31:27 Often I encourage people to get into a video call like this or whatever.

31:32 Because the communication is better.

31:34 Now I'm glad that at least we use Google Meet and you've got this Gemini note taker, it can start keeping track of that stuff for us as opposed to having to write it down.

31:43 Scott: And by the way, can I speak to that for a second real quick? So one of the practices that we instituted at Health Tech Company,

31:51 which I thought was very useful,

31:54 was we essentially...

31:57 created a role that we called like the incident incident commander basically

32:02 Brad: Yeah, exactly.

32:02 Scott: almost take and that that that role was um designed to uh

32:10 be much like an incident commander is in the um in the star world of incident command system where it's like the incident commander is like sure they're the kind of like final arbiter of what's going to happen but they need to be above this they need to be above the fray and they need to be able to step back and take the 50 000 foot view um and make sure that they're delegating and deploying the experts to actually solve the problem but

32:35 but they're sitting there going okay great cool 8 p.m alarm sounded and they can start they can basically kind of be in uh in these software environments they can tend to be on that zoom call kind of keeping track of everything

32:49 Brad: Now,

32:49 Scott: and being available to sort of like you know make calls when it's needed yeah

32:55 Brad: just for the normies and muggles at home, you've said SAR a bunch of times.

33:01 But

33:01 Scott: search and rescue

33:02 Brad: search and rescue,

33:03 search and rescue,

33:04 Scott: yeah

33:04 Brad: which is very noble that you do that, by the way, I do not have the stomach to do that.

33:09 Well,

33:10 I do not have the stomach to show up at the end when you get to the rescue part of it.

33:15 Okay,

33:16 so yeah,

33:18 so I like to start with the five whys and people sometimes start too far in the process.

33:23 I like to start way up.

33:24 Why did you get? Why did the alert go off?

33:27 Sometimes it's just a customer service or a person,

33:29 why didn't Sally from customer service call you, Brad?

33:33 Well, she called me because she thought the site was down.

33:36 Why did she think the site was down?

33:37 And just be,

33:39 I try to really encourage you,

33:41 be as plaintive and almost like stupid as possible. Why did she think the site was down?

33:46 Well,

33:46 Scott: Right.

33:46 Brad: because their browser said 504 gateway timeout.

33:52 Scott: Yeah.

33:53 Brad: So, and really push that. And then along the way, and keep on pushing until you get to true causes.

34:00 And I find that it has taken some training in my experience. I learned and then I've had to teach other people what is really the root cause.

34:10 A lot of times people stop before they get to the real root cause.

34:13 Scott: What's an example of a team that stops too early or, you know?

34:18 Brad: Okay,

34:18 here we had something like this happen recently.

34:20 So we have a number of staging environments and you can kind of reserve a staging environment to test something.

34:26 And we had a confluence of things happen recently.

34:28 There's kind of a little bit of a comedy of errors.

34:33 Person A was doing a demo for some stakeholders on a staging environment.

34:38 Person B was doing some maintenance on the staging environment,

34:42 but person C was the person who had the staging environment checked out.

34:47 Scott: Oh, no.

34:48 Brad: So it was just a total comedy of errors. So you could say.

34:54 Well,

34:55 why did the demo,

34:56 why did the staging server crash during the demo?

34:59 Oh,

34:59 it's because I'll just use some names. It's because Mike rebooted it as part of this maintenance and you're done.

35:07 But really there was some other thing.

35:09 Well, wait a second.

35:11 You know, we had to get to, well, it was Brian that had the staging server reserve.

35:15 Why was Mike even working on that thing in the first place, right?

35:18 What was it about our, this is maybe not the best example,

35:21 but to me, that's not the real root cause that, you know, Mike rebooted it. The root cause is that somehow our reservation system failed,

35:30 right?

35:31 Scott: Yeah, and I actually, so that's really interesting,

35:33 Scott,

35:34 because I think that kind of dovetails into one of the things I have.

35:38 Like, I'm not going to say I don't like the five wise process.

35:41 I do like the five wise process, the idea of like,

35:44 let's keep pushing on the wise until

35:46 Brad: Digging,

35:46 Scott: we figure

35:47 Brad: keep digging,

35:47 Scott: out a true,

35:47 Brad: yeah.

35:47 Scott: keep digging.

35:49 But one of the things I see that teams struggle with is picking the first wrong,

35:54 the first why they start with is the wrong thing to start with.

35:58 Right.

35:59 And so like, for example, in your story where, you know, person A was doing a demo, person B rebooted the system and person C was the one that had it checked out.

36:09 Brad: Yeah.

36:09 Scott: Like, I feel like one of the root causes you're probably have to get to is to your point, like something went wrong with the reservation system and somehow like I guess you can get there with the five whys, but I feel like you end up.

36:23 end up having to probably do,

36:26 you know.

36:27 uh at least five different starts and you know five wise off of each of them so it's like 25 wise to have found that like because one of the wise might have been well well why did why did mike think that it was okay to do a demo in the demo environment or why did mike think it wasn't going to reboot in the middle of his demo oh well because he was doing a demo but why did he think he was the one doing you know who had the space

36:54 in space you know oh because actually brian checked it out and told mike that brian that he could use brian's checkout right

37:02 Brad: Yeah.

37:03 Scott: or whatever you know what i mean and it's like to me that's what i've i've always kind of like i always try to say like look what really matters is we need to try to figure out what the root cause is of

37:15 But many things probably went wrong, and they all have their own root causes.

37:19 Brad: Yes, there are branches.

37:20 Frequently there are branches.

37:22 In fact, my experience, most problems that are more than just a typical bug are two things happen simultaneously when the right hand didn't know what the left was doing,

37:34 you

37:35 Scott: Yep, which is

37:36 Brad: know?

37:36 Scott: why the timeline is so important.

37:39 Brad: That's right.

37:40 Scott: Right.

37:41 And it's the integrated timeline.

37:43 Brad: And then you end up with some working agreements and action items and you put them somewhere,

37:49 you know, like we like to put, you know, we use Jira, like just stick them in Jira. You tag them as retroactive items,

37:56 retro items so that you can.

37:59 measure your progress um like you uh we're the company i'm at now we're we have an sdlc document that needs a lot of refinements we're rebooting that process to refine it but i think those working agreements go in the sdlc and um that's the cycle and it's worked pretty well for me i've been pretty happy with it kind of like the five wires process i guess you don't like it okay i guess there's one there's one area we disagree red we finally found it

38:26 Scott: it's not that i don't like it i don't i think it needs there's more to it than that

38:31 Brad: What

38:31 Scott: so i

38:31 Brad: do you think? Do you have a different approach you like?

38:35 Scott: don't have a better one so maybe the reason i'm putting out there that i i feel like the five wise uh is insufficient is a plea to hey folks do you have a better idea out there like how do we make sure that it gets you know that we really cover the ground that

38:52 is the very rich nature of why cascading failures happen in the first place.

38:57 Brad: I think what you're trying to say,

38:59 Brett,

38:59 is after you smash the subscribe button,

39:01 when you're leaving that five star review,

39:03 leave a comment in the five star review for a better way to

39:08 Scott: I

39:08 Brad: do retros.

39:08 Scott: agree.

39:08 Let's do that.

39:10 And I'll say one last thing about this.

39:13 I feel like this conversation about retros.

39:17 dovetails really nicely with our conversation several episodes back about building trust because you just can't do these you can't really do good retros if you don't have a culture of trust

39:34 Brad: Yeah.

39:36 Yeah.

39:36 And I would say it is harder to build a culture of trust when you're fully remote.

39:44 which makes these retros a little bit harder yeah

39:49 Scott: It's hard to build a culture of trust.

39:51 Brad: it is hard to build a culture of trust yeah

39:53 Scott: You could stop there.

39:56 Brad: do you think it's hard well let's let's just go up a second do you think it's not as

39:59 It's not harder to build a culture of trust virtually.

40:02 Scott: Well, I've told you this, and we kind of talked about this in the building trust conversation we had.

40:09 I feel like remote.

40:12 forces you to be way more intentional about

40:14 Brad: Yeah, for sure.

40:15 Scott: building trust.

40:17 Brad: Yeah.

40:17 Scott: And I think the key to building trust is being really intentional.

40:22 And so even in a in a in person world or a hybrid world,

40:26 if you're not intentional,

40:28 then it doesn't help you that you're in person.

40:31 In fact, it might hurt you that you're in person.

40:34 And so to me.

40:36 Maybe I think it's

40:40 almost as if forcing yourself to be virtual means you have to be much more intentional and that means you're going to put the investment in and it's going to get the payoff.

40:53 So that's my take.

40:54 I think I'm not going to say it's easier,

40:56 but the tools that really work in building trust.

41:02 Those tools are available to you in the virtual world as much as they are in the in-person world.

41:10 That's my opinion.

41:12 Brad: All right.

41:13 Is that where we should leave it?

41:15 Scott: That's where we should leave it.

41:17 Brad: All right. Well, thanks for listening,

41:18 everybody.

41:19 It's been fun to have you don't

41:20 Scott: Yeah.

41:21 Brad: forget to smash that subscribe button

41:25 Scott: Rate and review.

41:26 Follow us on all the social media platforms.

41:28 Brad: All the social media platforms and tell all your friends

41:31 Scott: All right.

41:32 Good

41:32 Brad: Talk

41:32 Scott: talking to to you,

41:33 Brad: you later,

41:33 Scott: Scott.

41:33 Brad: bud.

41:33 Bye

41:35 Scott: This episode of How Many CTOs Does It Take podcast was edited by Safran Epagow.

41:40 Our theme song was AI Generated by Brad.

41:43 Please follow us on all the social media platforms at at how many CTOs pod you can find us on YouTube LinkedIn Instagram Facebook threads X tick-tock and patreon or at our website at how many CTOs pod calm and remember to rate review and subscribe on the podcast platform of your choice thanks for listening