Deploys, Demos, and the Elusive Metric of Engineering Productivity

0:00 Scott: Do you agree with the statement that as an industry,

0:02 we don't have a way to measure productivity?

0:04 And if you do agree with that statement,

0:06 can you think of a better way than number of deploys to measure it?

0:10 Brad: I have to say yes to the first statement because every single owner,

0:16 every business collaborator, every stakeholder has always said to me,

0:21 why don't I get stuff done faster?

0:38 stories untold.

0:40 Scott: Welcome,

0:40 Brad,

0:41 to this week's episode of How Many CTOs Does It Take to Screw In One Light Bulb?

0:49 Brad: How many CTOs does it take to screw in one light bulb?

0:53 Scott: Well, it just depends on how many light bulbs you have.

0:56 Brad: How many light bulbs you have?

0:59 Scott: I don't know. I don't even know where that comes from.

1:02 Brad: I'm Brad Hefta-Gaub.

1:04 Scott: Yep, he's Brad Hefta-Gaub. My name is Scott Borat.

1:07 We are here to help try to demystify and figure out some of the most puzzling.

1:15 aspects software engineering management and today we have the topic of all topics to discuss i'm

1:22 Brad: What's that topic?

1:23 Scott: not i'm going to try to not use loaded words i'm

1:27 Brad: Okay.

1:27 Don't say urgency.

1:30 Scott: not going to say urgency and i'm not going to say velocity i'm

1:33 Brad: Try not to say velocity.

1:34 Scott: just going to say people want our teams to get stuff done faster

1:40 Brad: Yeah.

1:41 get wait we we talked a little bit about this in an episode in the past and

1:46 Scott: I'm sure we have.

1:47 Brad: what and and you you actually did this great job where we talked about this for a long time and then you go you know what i think it is it's get stuff done faster so

2:01 Scott: That's right.

2:02 Brad: like we got to be clear on the stuff we got to be clear when it's done and we want to do that faster

2:08 Scott: And let's talk about the faster,

2:11 because

2:11 Brad: Yeah.

2:11 Scott: to my knowledge, nobody in this industry has figured out how to measure engineering productivity either.

2:24 we make what I'll broadly call process improvements,

2:28 but they could be improvements to the technology, we make improvements that allow us to get stuff done faster.

2:33 And then the effect of that is projects that were large now become medium and projects that were extra large now become large.

2:41 And we don't see any of the gains because the bucket just gets filled up again,

2:46 right?

2:46 Brad: Right.

2:47 Scott: And then the purists, like the scrum purists say,

2:51 well,

2:51 Well,

2:51 you know, a story point really isn't a measure of time, it's a measure of business value.

2:57 So if we're doing that the right way, then you would see that your team is delivering more business value per unit of time.

3:03 But then it comes down to, well, who's picking the projects and are we picking the right projects? And that's not really a very good measure in my mind either.

3:12 Though I have had a hypothesis in my mind for quite some time that I have not had the chance to test out,

3:21 but I might get the chance to test it out in the near future, my current place of employment.

3:28 And I want to run that hypothesis by you and see what you think.

3:31 Brad: I'm all ears.

3:33 Scott: So to get to this hypothesis,

3:35 first,

3:35 I have to take a big step back and I have to invite everyone to come read my e-book that is not yet written.

3:40 But just remember to check back on this podcast every few months to see if the e-book is out there.

3:46 And it is how it's called how to turn around an engineering team.

3:50 And I sort of have a process I follow. I've done it a few times.

3:54 I'm not trying to toot my own horn, but I've seemed to have figured out something that works a little bit.

3:58 Step one is operations.

4:00 Is your system stable?

4:02 There's no point in trying to build new features if your sites or your app or whatever is always crashed.

4:07 Brad: Yep.

4:07 Scott: Step two is be predictable. Try and have good estimation accuracy,

4:12 deliver on time.

4:13 There's no point in trying to be early. You can't be early if you're still, if you're not, you can't be early if you're not even on time, right?

4:21 So first get your situation stable.

4:24 Second.

4:25 learn how to deliver on schedule then once you're getting there on time try to get there a little bit faster

4:32 Brad: Right. You said something recently to me on a previous episode. You said there's no point in going fast if you arrive late.

4:40 Scott: Exactly.

4:41 Exactly.

4:42 So and,

4:43 you know, I have a few key metrics that I pay attention to for each of these in the stability space.

4:50 It's pretty simple.

4:51 Raw number of incidents,

4:53 however you want, however you want to define an incident,

4:55 raw number of incidents and mean time to response,

4:58 right?

4:59 How many problems are we having and are we getting there quickly?

5:01 And if you're not having very many problems and when you have them, you're getting to the fire quickly.

5:05 To quickly to put it out.

5:06 that's good you might also want to look at meantime to recovery if it's taking you a long time to fix them but typically it doesn't take people that long to fix them it's just like do we have the operational awareness to have the pagers and all these things to like set us off and do we have the prevention to to prevent them from happening in the first place so kind of look at those kpis and then for for accuracy i pretty much just look at estimation accuracy

5:33 Um,

5:33 for whatever given period of time,

5:36 um,

5:37 what did you say you were going to do and how much of it did you get done?

5:42 And I personally aim for 80%.

5:45 100% is easy.

5:46 Everyone just sandbags, no problem.

5:48 80% you're still striving a little bit and you've got to give your stakeholder, you got to coach your stakeholders and say,

5:54 hey, we're going to miss like that's part of the plan is we're going to miss. So you need to have some grace with the team.

5:59 If you want 100%, we can give you 100%, but you won't get as much done because we're

6:03 Brad: Yeah,

6:03 Scott: going to put

6:03 Brad: get

6:04 Scott: tons of buffer in

6:04 Brad: super

6:04 Scott: there.

6:04 Brad: conservative.

6:06 Scott: Right, exactly.

6:07 So primarily look at estimation accuracy. You can look at a sprint level.

6:11 You could look at it on the quarterly level.

6:13 There's a lot of ways you can look at it, but just how good are you at predicting things are going to get done?

6:18 And that forces a lot of good behaviors you have to know.

6:21 What do you want us to do? You have to understand your system.

6:24 So you know, there's a lot of good behaviors. We've probably talked about that bunch here.

6:28 I could talk about it a bunch more.

6:30 And then the last step is going faster.

6:32 And nobody knows how to do this. We just talked about this. But my hypothesis is this.

6:37 Just count the number of deploys.

6:41 Or to be more precise,

6:43 deploys per person.

6:45 because i can get more deploys by just hiring more people and you get more deploys but just count the number of deploys that's one of the dora metrics for people who know dora metrics i think it's the number one dora metric is deploy frequency so you count your number of deploys and you also have to balance that by looking at you know your

7:02 incident rate and your bug rate and you know are you maintaining your quality but here's my hypothesis you count the number of deploys because what will happen is if you count the number of deploys is that will incentivize people deploying more frequently which means they'll have to do smaller things

7:16 Brad: Mm-hmm.

7:17 Scott: and just like in nature smaller stuff goes faster right and you'll get

7:24 a bunch of velocity gains and either you're getting smaller stuff because you're designing smaller projects or you still have big projects but your engineers are shipping smaller bits and putting them behind a feature flag which has a whole other set of benefits behind it you're getting quality improvements because smaller bits are easier to test and if there's a problem they're easier to understand probably to get a whole bunch of whole host of health benefits from smaller

7:49 particles flying through the system and so just measure how many deploys per person i mean not like individually brad how many employees you did but you just have to account for the fact there's

7:59 Brad: Yeah,

8:00 Scott: big large

8:00 Brad: average deploys

8:00 Scott: teams yeah

8:01 Brad: per team member or whatever.

8:02 Scott: whatever and include all your people your qa people your product managers include everybody right because you just you're just you know because if you add 10 more product managers you must be doing it because you think it'll help you go faster right

8:15 So average deploys for people and then start asking questions.

8:18 Well,

8:19 what would it take us to have more deploys,

8:20 Brad: Mm

8:21 Scott: right?

8:21 Brad: hmm.

8:22 Scott: We need to ship smaller bits.

8:24 We need to make it so it takes less time to ship whatever.

8:28 You can start asking those questions.

8:29 So that is my – I had this idea in mind for quite a while.

8:33 I really haven't – in several years haven't got to a spot where I needed to pull this tool out of my toolkit,

8:38 and I just want to know what you think of it.

8:40 Brad: Well,

8:40 let's see.

8:41 I think it's.

8:42 It's an interesting idea.

8:44 Scott: When you throw your thinking, kind of throw one other thing in there,

8:47 it goes back to urgency.

8:50 Brad: Yeah.

8:51 Scott: Brad, have you ever had a CEO or an SLT stakeholder say to you, I just don't feel like there's enough urgency?

8:56 I don't know what that means.

8:58 Does it mean you don't see people running up down the

9:00 Brad: Yeah.

9:00 Scott: hall screaming,

9:01 saying the house on fire?

9:04 I have this other part of the hypothesis is,

9:06 and this is not the,

9:08 this is just vanity,

9:09 but it's like if it's a machine gun of deployments, pew,

9:12 pew,

9:13 pew,

9:13 just stuff's going out over, look, we ship this, we ship.

9:15 we ship this we

9:16 Brad: yeah

9:16 Scott: ship that like people will feel like oh my goodness that team they're just firing off pistons they're just shipping stuff 25 times a day there's stuff going out it's crazy yeah

9:24 Brad: right well we okay so we literally had an episode recently about this it was all like urgency and we even called it like CTO therapy all about urgency right you know but let me let me see if I can play back what you're saying that I think it there's definitely this feels like there's I feel like I'm getting a little bit of a

9:42 the i can i can see some purchase with this idea so let me just see if i can play it back to see if i get it let's start with human nature human nature is that people that know they're being measured

9:55 tend to respond to the thing they're being measured against and they try to you know they it's a score it's a score in the game called evaluation evaluation

10:08 Right.

10:08 Scott: Yeah

10:09 Brad: And so you always have to be careful when you like create a measurement because.

10:13 it will have the side effect of that measurement is the thing that's going to get optimized right so if so we always we i love to joke about how like engineering efficiency should be measured in thousand lines of code right clocks Mm like

10:25 Scott: -hmm.

10:25 Brad: that's the classic joke and i it's such a great joke right because like it's the classic you can see immediately the problem with measuring clocks because then you just get lots and lots of code and if that code is really inefficient or does stuff that you don't care about or is

10:39 is you know buggy then you just get lots more bad stuff right so number of lines of code is a really bad measurement of value but if you called that your if you included that in your in your you know like your engineering leveling guidelines then oh boy are people gonna you know they're gonna they're gonna like drive that number

11:02 Scott: That's right.

11:02 Brad: So we know that when we set a marker of we're going to start measuring something,

11:07 people are going to respond to it, right?

11:11 So you're saying like,

11:13 hey, I think that I want to put a marker on number of deploys.

11:21 And the theory that you have is.

11:24 There's more upside to number of deploys than there is downside to number of deploys. So when people start focusing on that number,

11:34 the benefits of kind of thinking of themselves as being judged by that, the benefits of that,

11:40 and you enumerated a few of those benefits,

11:43 smaller,

11:44 basically smaller changes.

11:48 which all have their own benefits you know easier to test easier to deploy easier to reason about easier easier easier so you're saying like hey the benefits of if they choose the strategy of optimizing that number by going small the benefits of small lots of small is going to be is probably going to be a very is going to create actually the thing that we're looking at getting

12:14 stuff done faster now i guess the question i have is as i articulate that it it sounds pretty reasonable what are the hidden what are the what are the what are the hidden consequences of that

12:30 Scott: Okay,

12:31 I can tell you one.

12:31 The team and organization will

12:35 start to do smaller things.

12:38 I

12:38 Brad: Yeah.

12:38 Scott: don't mean smaller deploys.

12:40 I mean like they won't try to, you won't try to do big bold changes.

12:44 Brad: Yeah,

12:44 they'll be afraid to do a big change.

12:46 Or they're not afraid,

12:48 but they'll recognize that the big, bold thing that they haven't reasoned about how to break down into small things,

12:57 they'll instinctually know that that will hurt their number.

13:02 Whether they say it out loud or voice it, they'll have in their gut like, oh, that big thing,

13:08 that's going to drive our number down.

13:12 Scott: Well,

13:12 I was speaking slightly different.

13:14 If there's a big thing you want to do and you're not breaking it down,

13:17 well,

13:17 then you should break it down and use a feature flag to roll it out in a bunch of little bits and hide it and so on and so forth.

13:22 I was actually thinking more like culturally.

13:25 like you just would stop thinking about big bets like let's just compare babe ruth to ichiro right

13:31 Brad: Yeah.

13:32 Scott: like people that are swinging for the fences and missing a lot versus people who are not swinging for the fences but

13:41 Brad: Getting on

13:41 Scott: getting

13:41 Brad: base.

13:41 Scott: a lot of singles and you could create a culture of singles and as a business then miss

13:53 Brad: It could be, yeah,

13:53 Scott: not even have the comfort of like,

13:56 not even see, not even see the opportunities to swing for the fences,

14:00 setting aside that you might also, if you do have a swing for the fences idea,

14:05 really hard time figuring out how to break it down.

14:08 Brad: I agree. That's one of the that's definitely one of the potential pitfalls of this approach.

14:14 We cannot blow this idea out of the water 14 minutes into talking about it.

14:20 There's got to be a better idea here than that.

14:23 I mean, we

14:24 Scott: What do you mean?

14:24 Brad: got to be able to sell.

14:26 I like your idea,

14:27 small.

14:28 Scott: Yeah.

14:28 Brad: So we can't be scared.

14:30 We have to think about how do we also address swing for the fences.

14:33 Scott: Well,

14:33 to me, the way you address screwing within the fences is the size of the units you deploy should be unrelated to your idea.

14:43 Brad: You're saying that.

14:44 Then you just got to get good at like breaking things down and figuring out

14:47 Scott: Yeah.

14:48 Brad: how to put stuff behind feature flags

14:49 Scott: Yeah.

14:50 Brad: Yeah

14:51 Scott: I mean, actually,

14:51 there's kind of a, just as a.

14:54 tangent but maybe it's not a tangent at all to something we talked about in a previous episode about what we were seeing AI do with the size of tasks we complete right

15:04 Brad: Yeah.

15:04 Scott: and that whatever research we were looking

15:06 Brad: Yeah.

15:06 Scott: at was like look it can successfully complete a complete a 12-minute task

15:10 Brad: Right.

15:10 Scott: right

15:10 Brad: Right. Right. Give it to

15:11 Scott: so

15:11 Brad: me as essentially fast as

15:12 Scott: it's doing

15:12 Brad: it does.

15:12 Scott: a deploy every 12 minutes right

15:14 Brad: Yeah.

15:14 Totally.

15:15 Totally.

15:15 Scott: so that's

15:19 That's what we got to get to.

15:20 Brad: yeah and i think that like i mean it's so funny i can think about this in most my most recent past you know we had this big feature that we were working on at um you know my main team and um it was hard it was a big feature it was and the whole team worked on it for you know for for us this was a big feature it was like the whole team worked on it for like three sprints and it felt endless you know and it was like

15:46 And in each of the sprint retros, we were like, oh, man,

15:51 like we bit off too much or gosh,

15:55 we could have broken this down smaller or gosh, you know.

15:59 and you know the good thing is we have a really strong relationship between product and engineering so that like the product owners were like yeah that's cool man maybe i should have thought about how i could have broken this down even more like what would i have accepted as a partial implementation of this you know um and of course we used feature flags and we kind of like kept shipping even though

16:20 you know even though the feature wasn't done you know so there were lots of little deploys that were kind of behind feature flag so we're certainly doing that but um i think one of the lessons that the team took away from that was you know they did sort of say like hey when we got it all done they were like yeah we're really happy with that and so it's good that we didn't get scared of trying to do big things right like it's good that we were like oh okay even though it was hard like for us the relative

16:45 pain of this felt felt

16:47 Scott: Mm-hmm.

16:47 Brad: big we were really happy at the end right that we didn't we didn't get scared off by it because it was going to be a bigger type of project than we would normally have taken on and so that was one great outcome

17:02 that kind of speaks to your we always have this much you know small pieces multiple people working on it get it merged get it tested get it deployed you know that that um that philosophy is very consistent with how we work so i think you just have to be really intentional about that like i think that i think that one of the really big risks is scaring the team away from doing big things and so i feel like some scaffolding

17:27 around yes we really want to see lots of pew pew pew pew pew small deploys and we believe in you that you can also do big things in that framework and we're not going to let we're not going to get scared of big things in that framework

17:46 Scott: Yeah, I think we don't want to let the tail wag the dog.

17:50 Brad: Right.

17:50 Scott: Right,

17:51 which is this is a metric to help us have a framework for both conveying to our stakeholders that we're going faster,

18:02 but also a tool for us to ask ourselves,

18:05 what would it?

18:07 But let's just say, what would it take for us to go faster?

18:09 Let's just say I have the Dora metrics. We use a tool that gives us those metrics.

18:14 I looked at them earlier today. I don't remember the numbers.

18:16 Brad: Are

18:16 Scott: But you let's

18:16 Brad: using

18:16 Scott: just say,

18:17 Brad: that using that B that B one yeah

18:20 Scott: no,

18:20 we used to use, I think it was called B linear B.

18:23 Brad: be linear

18:25 Scott: Oh, no,

18:25 we don't. We use that. We have switched to a tool called Jellyfish.

18:28 Brad: Oh.

18:29 Scott: I don't have commentary if it's good or bad,

18:31 actually.

18:31 I haven't spent a ton of time with it, but I was able to go right in there today and see the number of deploys we've done on day, week, month for the last year, as long as we've had it.

18:42 And I don't remember the exact number, but let's just say for argument's sake it was.

18:49 one deploy per person per month including

18:52 Brad: Oh, okay.

18:54 Scott: our product managers and designers and everything right i'm just using a rule of thumb here it would i think it would be okay to say what if we got that to

19:03 Brad: Yeah, totally.

19:05 Scott: Right.

19:06 You know, it'd be

19:07 Brad: What

19:07 Scott: a tool.

19:07 Brad: does it take to get that number to go up? Number go up.

19:09 Scott: What does it take to get it to go up? Yeah, that's the question we're trying to ask ourselves.

19:14 Not, well,

19:15 we don't really want to do this project because it would ruin our velocity or deploy numbers.

19:20 Brad: Yeah, for sure. You definitely don't want it. You want to make sure that those words don't ever get uttered. But those are the things that are like in the back of the brain that people don't even say out loud.

19:28 They don't even know they're thinking it, but they somehow become cautious,

19:31 right?

19:31 Scott: Yeah, probably.

19:33 Brad: Okay, so like I said, I like this idea.

19:35 And I'm just doing I'm doing the thing that I do when I try to understand an idea better.

19:41 I try to think out loud and I try to ask questions about it. And those sometimes come off as

19:46 adversarial so hear me that i'm merely exploring this idea the part of me that i feel and we talked a little bit about this in the in the episode on urgency but the part

19:59 Scott: To me, that's like, man,

20:01 how do we get better at telling our stakeholders that the big thing is coming,

20:12 right?

20:13 Brad: Mm-hmm.

20:15 Scott: You know, when we talked about urgency, one of the things that we talked a little bit about was we did kind of like talk about this idea and it's very similar to where you're talking about this small deploy.

20:24 So I think there's a synergy here between these two ideas is this idea of like,

20:28 well,

20:28 you know, like if you basically have something to show every sprint,

20:32 like this is the big problem, right? When you have these big projects, it's like, man, that thing was on the roadmap for six months and then I finally got it.

20:39 But you just were in the back room cooking this thing up and I never saw any progress against it.

20:46 And we talked about this idea in that even if you haven't actually changed anything about how fast you're building stuff,

20:54 if you just are showing regular updates that are meaningful,

20:58 like that are really meaningful updates,

21:00 then I think it's going to create this sense of momentum.

21:04 and progress and so the what so one of the things i really love about your idea here is if you could basically say man look every one of those you know boom boom boom little bullets that were shooting out here those are all value and so you just like you said you'll just get the sense of like man that team is really cranking out value right and i guess the question i so yes i love all that and as a developer you do feel like

21:31 like man i just made progress like like literally today it was like i had this i i was coaching this developer i had a pr that was similar to this i was coaching and i keep telling my team i'm like okay you got two great ideas make that two prs don't make that one pr right like do do the thing that you said you were going to do and while you were doing that you came up with another idea of a thing that would make it even better well finish the first thing then make it better

21:57 then add the second thing right and

21:59 Brad: Mm-hmm,

21:59 Scott: and

21:59 Brad: mm

21:59 Scott: it's

21:59 Brad: -hmm.

22:00 Scott: so uh you know there are two uh forces that pull us as engineers one of these is like i can always see more that i can do right there's always more code that i can write there's always a way for me to make it better and so many engineers are driven towards like make it polish it polish it polish it polish it and so coaching them to be like no no

22:22 I want you to do this progressively. Get this part done.

22:26 Get the next part done. Get the next part done.

22:28 Get the next part done.

22:29 You'll feel better. We'll feel better. We'll see progress.

22:31 We'll see these, you know, we'll move number up on this new deployment concept that our CTO Scott wants us to be measured against.

22:41 But it's a pull.

22:43 It's a pull, this gravitational force towards I really want to tidy it up. I really want to polish it.

22:49 Brad: Right?

22:49 Scott: mm-hmm

22:49 Brad: Progressive

22:50 Scott: progressive

22:51 Brad: rendering.

22:52 Scott: rendering exactly i used to like what like when i was a like early days like was like some engineer called it stepwise enhancement so i like i'd always use that term stepwise enhancement

23:03 Brad: You know, the only way I have ever found.

23:09 to for stakeholders to have the feeling that something is coming along fast enough it has not been burned down charts it has not been status updates it's the only thing is demos because they they see it with their own eyes we talked about different types of learners early on but for some reason this is the only one that's worked i've ever seen work for all types learners as they see it

23:35 they feel it they get excited about it right and they're along the journey with you as opposed to being on the second that's my career that is the only way i've ever really seen it to work

23:47 Scott: Well, the hard part, I love that. I love that idea.

23:51 But you also like to use the analogy of like building a house.

23:55 Every remodel I've ever had participated in or

23:59 house build or whatever it's like the the general contractor that like walks you through when you know at each step of the way right

24:09 Brad: Mm-hmm.

24:10 Scott: like the demo is finished right i've never walked into a kitchen remodel where the demo was finished that

24:18 The contractor hadn't like done something like put tape on the ground of where the new stove was going to be,

24:24 you

24:24 Brad: Mm

24:25 Scott: know, or

24:25 Brad: hmm.

24:25 Scott: the outline of where the cabinets are going to be, you know, is on the uppers are there's like there's like, you know, markings on the wall.

24:34 If the drywall is up, there's marking on the wall where the uppers are going to go in. Right.

24:38 and they walk me through it and they go you see we've done this part but you but you've got to picture the next part you got a picture i need you to believe that we're going to get to that next phase right and

24:51 Brad: Yeah.

24:51 Scott: i think that works for some people that have imagination but there's also some stakeholders that are just like look until you give me the finished product i'm not satisfied

25:03 Brad: Okay, I have two responses for you, and one is just a digression. Which do you want first?

25:07 Scott: So yeah,

25:07 go for it. This is your podcast.

25:10 I'm

25:10 Brad: It's

25:10 Scott: just a guest

25:10 Brad: not my podcast.

25:11 Scott: here.

25:13 Brad: Well, I think the way the sequence in which you build things is different.

25:18 Let's just say you're doing a screen in an app.

25:21 You might first make that screen and have the buttons not do anything.

25:28 But at least the...

25:29 the demo watchers can say okay it's here like it doesn't work yet next week this button's going to work and next

25:37 Scott: Yeah.

25:37 Brad: the week after that this other button's going to work and the week after that this picture is going to be it's going to slowly you can see it's just it's the placeholder text right but you're going to see it come together and that's a different mindset for how you build and how you sequence the construction than

25:52 And it might be slightly less efficient. I'm not 100% sure. I actually have a hypothesis on that, but it might be less efficient, but it's definitely a different mindset.

26:00 Now, my hypothesis is that it's not less efficient because it keeps you focused on actually building the stuff that's going to make it work.

26:06 Scott: Can I respond to that real quick?

26:09 Brad: Of course you can.

26:09 Scott: First, I love that idea.

26:10 Brad: It's your podcast.

26:18 Scott: well done sir um uh i i like that idea and but i will also say that i'm not sure that you have to build it outside in i think you can build it inside out also you

26:32 Brad: Mm

26:33 Scott: know like

26:33 Brad: -hmm.

26:33 Scott: i've seen the demo where it's like

26:36 you know it's got some back-end stuff it's got some front-end stuff you know the team got the back-end stuff working and they had a tiny little shell of the front end you know didn't match the didn't match the wires didn't have the right text but it did you know it had a light that went from red to green when you pushed a button and and then they can say in the demo see that actually was the credit card transaction you know or whatever

27:01 Brad: Do you want to know the coolest demo I was ever part of?

27:02 or part of this

27:03 Scott: Yeah, tell me.

27:04 Brad: is way back when i was at rover and we were launching this whole new product i don't know if rover still has it um but it was a it was on demand dog walking push a button and a dog walker would be there in 30 minutes maybe it was 60 minutes but real fast and because right now or at least previously to that it had been scheduled you had to schedule it and

27:24 I don't know if you can schedule same day,

27:26 but anyway,

27:27 not only was the engineering team demoing what was getting built,

27:31 there was a bunch of other stuff that had to happen. Marketing was happening.

27:34 Scott: Oh

27:34 Brad: Operations was happening.

27:36 Customer service was happening.

27:37 And they participated in the demos also,

27:40 right?

27:41 Here's the scripts that the agents are going to use. Look,

27:45 we've got the templates for Zendesk figured out. Oh, here's the marketing copy. So the whole.

27:51 whole team as we were trying to lift up this barn of this new project everybody was doing their part of showing it was really uh it was really fantastic it

28:00 Scott: That's pretty cool.

28:01 Brad: was

28:01 Scott: Yeah.

28:02 Brad: cool it was the coolest thing i've ever seen actually i mean not in my whole life but in terms of demos oh can i tell you the digressions for

28:10 Scott: Yeah,

28:10 Brad: us i want to tell

28:10 Scott: tell me me the digression

28:10 Brad: digression sorry

28:11 Scott: yeah

28:11 Brad: we were talking about um building from the inside out this related to houses two cool things i've seen one

28:18 was a one-time thing and one is reusable I have a friend who's a home builder he builds fancy homes for rich people and he had this client where they basically mocked up using like cardboard and wood and plywood in a warehouse a kitchen for one of their clients their clients could actually like walk through

28:39 Scott: Walk

28:39 Brad: it and

28:39 Scott: through it.

28:39 Brad: feel

28:41 And it was a lot of money. It was like the client spent like $50,000 to like get this mock up.

28:45 But they're building like a bajillion dollar house. So it was a drop in the bucket.

28:49 And that's in fact what the builder said. He's like,

28:52 hey, you're going to build this really expensive house.

28:54 You're not confident about this part.

28:55 Wouldn't it be worth it to you to like get to actually experience it? So they built they built a kitchen,

29:01 which was kind of cool.

29:02 And then I've seen this thing online,

29:04 which is similar.

29:06 It is also it's in a warehouse and they.

29:08 And they use they take your floor plans and they project from the ceiling,

29:12 probably just using, you know, stock projectors, they project your floor plan on the floor in actual size so you can go in and walk around your

29:22 Scott: Yeah.

29:22 Brad: floor plan with your architect and your builder,

29:24 et cetera, to see if it feels right,

29:26 which I think is pretty cool.

29:28 Scott: Yeah.

29:29 Brad: I actually don't even, you know, for all of our listeners,

29:31 I'm up here in Seattle.

29:32 I don't even know if we have that here,

29:34 but if we don't, maybe that's a business you and I should start,

29:36 Brad.

29:37 Scott: Yeah,

29:37 that sounds like a cool idea.

29:38 I agree.

29:39 And it does speak to that whole question of like, you know, you've got these points along the way that you know you have stuff done and you want to somehow demo that you're done with it and test it,

29:53 you know.

29:55 The

29:55 Brad: For sure, before you start building, it's way cheaper to change those architectural plans.

29:59 In fact, the same builder friend told me about a house he was working on and they were building it and it went up and then was pretty well built.

30:08 I mean, it framed up and stuff and the client was like, you know, this would have a better view if we had a third floor.

30:14 Scott: that's a big change they

30:16 Brad: changed it like that's a that's a big change right would have been better to have that change in the drawings than when it was built here's a quick so back on topic i have a question for you two

30:28 Scott: Yeah, sure.

30:28 Brad: questions for you do you agree with the statement that as an industry we don't have a way to measure productivity and if you do agree with that statement

30:37 Can you think of a better way the number of deploys to measure it?

30:41 Scott: I have to say yes to the first statement because every single owner,

30:47 every business collaborator, every stakeholder has always said to me, why don't I get stuff done faster?

30:55 right

30:55 Brad: Right.

30:55 Scott: like like that's the age-old thing right like you know and and how many times have you you know like you said like you know as as a cto how many times have you brought in with the sort of this vague sense of like gosh can the engineering team get stuff done faster right and you're always like and and i love your i love your playbook right like i think your playbook is exactly the right playbook which is first let's make sure that it's operating correctly then let's make sure that we're predicting

31:20 predictable and then we can talk about ways to become faster and i 100

31:24 Brad: Mm

31:24 Scott: agree

31:24 Brad: -hmm.

31:24 Scott: with that playbook and i don't you know and it's like i can't count the number of times that you know some recruiter has called me or some client has called me and it's like yeah we got to like get the team to move faster and i'm like well

31:37 look man there's a lot that could be behind that you know and so first step is let me go figure out what's behind whatever you're doing you know and one of the things that could be behind it and you've said this in the past is there could be just an unrealistic appetite for stuff to get done in a timeline

31:56 based on the configuration in the environment,

32:00 right?

32:00 Like it could just be an unrealistic appetite.

32:03 Brad: Or not even the configuration in the environment,

32:05 just the fact that sometimes things are more complicated. I remember a time again coming back to Rover where we wanted to change something.

32:13 It wanted to change.

32:15 how far in the future you

32:16 Scott: Right.

32:17 Brad: could book something i've mentioned this before

32:19 Scott: Yeah,

32:20 I think you did. It's like 60 places in the code where there's this number.

32:24 Brad: was 20 like 20 but i was i remember talking to you i was like why is this taking so long it just seems like isn't there a constant somewhere we have to change he's like well no let me show you all the stuff that has to change to make all these date pickers and all this stuff i was like oh actually it's a lot of stuff it sounds so easy someone in a meeting just says

32:39 Says, hey,

32:40 we should change that from 90 days to 120 days,

32:43 right?

32:44 And easy done,

32:46 right?

32:47 Scott: Yeah.

32:47 Brad: That's simple.

32:48 And

32:48 Scott: Right.

32:48 Brad: but it's hard.

32:51 Scott: Yeah.

32:51 Well,

32:52 and so first of all, I 100% agree with you that I have never.

32:57 I mean, there are so many books on this. There's so many different white papers. There's so many different tools out there. There's so many different philosophies out there that since there's so much diversity in solutions,

33:11 it's clear we haven't solved the problem,

33:14 right?

33:15 Brad: Right.

33:15 Scott: Like as an industry, we have not solved this problem.

33:18 Brad: Mm hmm.

33:19 Scott: And there's a lot of evidence that small deploys have so many positive benefits.

33:28 There's a lot of evidence for that.

33:30 Actually,

33:31 you just made me think of a framing of this that I want to let's

33:35 Brad: I

33:35 Scott: play around with this a saw little bit.

33:36 Brad: a light bulb go off over your head.

33:38 Scott: Yeah,

33:38 a light bulb just went off my head.

33:41 Okay,

33:41 so let's be really frank with each other about the

33:45 Brad: Mm-hmm.

33:46 Scott: reality of the 10x engineer.

33:48 I've known them.

33:50 Brad: Yes.

33:50 Scott: They exist.

33:51 They are a real thing.

33:52 Brad: Yes.

33:53 Scott: I think that there is probably a common set of attributes for those engineers.

34:00 Like, I think that when our business owners say,

34:05 I want stuff done faster.

34:07 Like they imagine that if every engineer performed like a 10X engineer,

34:13 then that's the itch that they're trying to scratch.

34:16 So let me just start with that.

34:17 That's my, that's my,

34:18 I'm going to pause at that.

34:20 Like if we could get, if,

34:21 if we could get everyone to perform like a 10X engineer, then maybe that's, that would finally scratch that itch.

34:27 Brad: What percentage of engineers do you think are 10x engineers?

34:31 Scott: one-tenth of one percent one in a thousand well

34:33 Brad: Oh, that's way smaller than I would have thought.

34:36 Scott: you think the numbers okay

34:38 Brad: One,

34:38 1%. I mean, I'm just thinking of a group of 100 engineers. There's one that's really talented.

34:44 Scott: i'll go one percent i'm

34:45 Brad: I mean, I don't know.

34:46 Scott: not going 10 i won't go 10

34:48 Brad: No, not 10 at all. No way.

34:50 I mean, I think one is maybe high.

34:51 I just think, look,

34:53 there's also,

34:53 you know, there's Michael Jordan and Kobe Bryant, right? There's

34:57 there's basketball players that are just more talented too right

34:59 Scott: Yeah, totally.

35:00 Brad: those people some people are just better

35:02 Scott: Yeah.

35:03 Brad: and i'm not one of them by the way okay

35:05 Scott: Well,

35:05 I mean,

35:07 all right, well,

35:08 at the risk of tooting my own horn,

35:09 I'm a 10X engineer easily.

35:11 I've

35:12 Brad: yeah

35:12 Scott: always been.

35:12 Brad: i

35:13 Scott: I've always been.

35:13 Brad: believe it

35:14 Scott: I guess actually I should probably say I probably should say at least one percent because of the of the I've worked with at least you know a thousand engineers in my career and

35:23 Brad: Oh,

35:24 Scott: yeah probably 10 10 to 20 of them but I think I was in places that had exceptional luck and getting 10x engineers so probably 10 to 20 of them were 10x

35:34 Brad: it attracts

35:34 Scott: engineers

35:34 Brad: people.

35:35 Scott: Yeah.

35:36 Talent attracts talent.

35:38 Brad: Time tracks talent.

35:39 Scott: Yeah.

35:39 So but let's talk about like what is it about the 10x engineer?

35:43 And you're you've kind of made a point that like maybe it's maybe it's just raw talent and that's hard to say.

35:49 But but I think there are commonalities and I think one of those commonalities is the ability for those people to break tasks down in and just think of problems in an incremental way.

36:02 Brad: Mm-hmm.

36:03 Scott: Right.

36:04 Like even the ones that would, you know, go away at night and come back in the morning and have the massive implementation of the whole system,

36:12 Brad: Mm

36:12 Scott: when

36:13 Brad: -hmm.

36:13 Scott: you really interrogated them,

36:15 they didn't just generate that in whole cloth.

36:18 It was actually that they they did it in a systematic and an incremental way. They were just really good at at piling those things on quickly.

36:28 Brad: Yeah.

36:28 Scott: What do you think about that?

36:30 Brad: I think that's true.

36:31 I think I want to save for an entirely different episode the attributes of the 10X engineer,

36:35 but I think that's one,

36:37 I think that's because I think it's easily an hour conversation.

36:40 I think that is one of them for

36:41 Scott: But do we,

36:42 Brad: sure.

36:42 Scott: so,

36:42 okay,

36:42 so of the attributes of a 10X engineer,

36:45 we do think that probably one of them is they would naturally excel at this deploy more often.

36:52 Brad: Well, yes, I would say that, but I would say the inverse also,

36:56 which is engineers who are not talented at breaking their work into smaller tasks have a hard time going fast.

37:03 Scott: Yeah.

37:04 Brad: That's right.

37:06 And so I don't know if it's the 10x engine,

37:08 I don't know if it's the breaking it down that makes you 10x or 10x that allows you to break it down.

37:14 Scott: Okay.

37:14 Brad: But I just know that people who take on a big blob of work.

37:19 You know, the story,

37:21 right? You're looking at the Kanban board and they got one ticket and it's been sitting there for two weeks,

37:25 right?

37:26 And I think that's because they're not breaking it down and it's just this big giant blob of stuff.

37:33 Scott: Okay, so.

37:34 The light bulb that went off that you saw me react to was

37:37 Brad: Yeah.

37:37 Scott: essentially what you just said,

37:40 which is I don't know about causation or correlation.

37:43 I don't care about that. But what I can tell you is there is definitely correlation with 10x engineers and a number of deploys.

37:53 Brad: Yeah, that's probably true.

37:54 Scott: And probably that's because that is a correlation of being able to break things down into small pieces.

38:01 Brad: Yeah, part

38:02 Scott: and

38:02 Brad: of it.

38:02 Scott: move and move fast on those small

38:03 Brad: It's

38:03 Scott: pieces one

38:04 Brad: one aspect of what makes it 10x or I think for sure.

38:07 Scott: x it was one aspect that makes it 10x or so my point is though and you even gave me the core the counter argument to that which is guess what the the non 10xers it's very rare that they're doing lots of small deploys

38:22 Brad: Well, and the other kind of argument that people who are doing large deploys, mega deploys, aren't going fast,

38:31 or

38:31 Scott: Yep.

38:31 Brad: at least they're not giving the appearance of going fast.

38:34 Scott: Right.

38:35 So this is the this is that was the light bulb is I was like, oh,

38:38 yeah,

38:39 if I frame this as like all the 10x engineers I've worked with in my career.

38:45 What

38:45 Brad: Mm hmm.

38:45 Scott: is one of the traits that they all have in common?

38:48 Not all their traits, but one of the traits they all have in common is they would immediately excel at your rubric of more deploys equals number go up.

38:58 Brad: Yeah,

38:59 there's another.

39:00 saying about this this is this is off the 10x for a little bit but about you're when we were talking about the 10x it reminded me of a specific engineer I work with who would always say the fastest feature to the fastest thing to ship is no code at all

39:14 Scott: Well, I had a colleague that I really admired and respected and he would always be like, he'd say something really similar.

39:22 He's like, there's no bugs in the lines of code I didn't write.

39:25 Brad: Right.

39:26 Similar idea.

39:27 And so if what our business wants us to do is go really fast,

39:31 well,

39:31 the first thing we should do is not write things,

39:33 but then the smaller,

39:34 the smaller the bit, the faster you can go, right? If it's a one liner.

39:38 boom you can go fast right yeah and

39:41 Scott: Yeah.

39:41 Brad: part of that an extra thing or maybe it's going fast thing i don't know which again correlation causation is building and designing systems where you can make impactful changes without changing much code

39:56 Scott: Yeah.

39:58 Yeah.

39:58 Brad: which think it's

39:59 Scott: sorts of design attributes right but like they're the somehow a talent for that of knowing how to build and design systems where you can get a lot of leverage of like you don't have to change much to make stuff different

40:13 Brad: Yeah, it's so, yes,

40:15 100% agree.

40:16 And it feels like we've made some progress and we've gone a couple steps forward on this question of how do we get people to.

40:25 build stuff faster get stuff done faster stuff done faster um and then i also have the the fear of the but what ifs or the the counter arguments which is the you know you can also write code in such a way that you've told yourself you're following all the patterns

40:46 And then, but then, you know, like small PRs, like you and I've talked about that,

40:51 you know, if the PR is over 200 lines, it's probably going to be very difficult for someone to go review

40:55 it.

40:56 And,

40:57 and then do you have like literalists on the team that basically then refuse to do code reviews for anything over 200 lines of code and then next thing you know, you're, you're slowing down because of that or you're ending up, you know, you're forcing yourself to break things down into too small of a task.

41:12 And that results in essentially clunkier code that then is harder to modify later,

41:16 you know, so there's all these like there's all these subtle it's these subtle like cultural anti patterns that get created even when we're trying to do a thing that we know in general is going to produce better results.

41:33 Scott: See, this is where engineers are good.

41:35 It's when they...

41:37 Take something that is asking them to change their behavior and they push it to an extreme and then say,

41:45 look,

41:45 it's absurd without

41:49 recognizing.

41:50 We're in a craft, not a science here,

41:52 Brad: Yeah.

41:52 Scott: right?

41:54 And so it's a craft that looks a lot like science and definitely has some scientific elements to it.

42:00 But yeah, from time to time, you're going to say, look,

42:03 I got this PR and it's 2,600 lines.

42:06 But frankly,

42:07 there is just no way to make this change without making it across the

42:10 Brad: I

42:11 Scott: whole system.

42:11 Brad: just reviewed a PR where someone was like, I am sorry,

42:14 this looks big.

42:15 but literally i'm just changing the name of a function but it was in like 1200 files i had

42:21 Scott: Right.

42:21 Brad: to change you know yeah what's the difference between an engineer and a kleptomaniac in

42:25 Scott: I don't know, actually.

42:27 Brad: engineers take things literally kleptomaniacs take things literally

42:32 Scott: So I think isn't the answer to that riddle, what's the difference between an engineer and a kleptomaniac? The answer is a comma.

42:41 Brad: That's

42:47 even better.

42:47 That's even better.

42:49 That's even better.

42:51 The linter found the missing comma.

42:54 Scott: Okay, I think, did we get somewhere today?

42:56 If

42:56 Brad: I

42:57 Scott: we feel didn't

42:57 Brad: like we got

42:57 Scott: get,

42:57 Brad: somewhere

42:57 Scott: we're

42:57 Brad: today.

42:58 Scott: on episode like whatever,

42:59 episode X, I don't know whatever show we're on,

43:02 and we've actually made some progress. If we make a little bit of progress every,

43:04 Brad: I feel like we made progress.

43:05 Scott: well,

43:06 here's the deal.

43:06 I've been talking to my team about it.

43:08 I'm going to try it out.

43:09 I have not tried it yet, meaning I haven't gotten to the point where I might get pushback.

43:15 I've gotten to the point where everyone I've talked to says,

43:18 that seems like a good idea. We could try that.

43:21 and the tooling is easy we've already got the measurements and measuring the ploys is about the easiest possible thing to measure right

43:27 Brad: Yeah.

43:27 Scott: there's no there's no judgment it's just like did the bot push code out or did it not push code out right um so we'll see we'll see what we can figure out

43:37 Brad: Love it.

43:38 I like it.

43:39 Scott: here's

43:39 Brad: I feel like

43:40 Scott: one

43:40 Brad: we got somewhere.

43:41 Scott: Yeah, now here's one other dimension.

43:42 Actually, I could use your help on this one. I've never had to do this part either.

43:45 How do I sell this to non-technical people?

43:47 How do I convince them that this is the right way to measure productivity?

43:52 Brad: Well, I think that the.

43:55 This is kind of what I was trying to ask, you know, at minute 20.

43:59 If you already are doing the thing that we talked about,

44:02 the idea of having some regular recurring,

44:06 not once a quarter, not once a year checkpoints that include demos of progress,

44:11 if

44:12 Scott: Mm

44:12 Brad: you've

44:12 Scott: -hmm.

44:12 Brad: got that in your culture,

44:14 then a simple explanation of, hey.

44:20 Sometimes you're going to see demos that represent progress.

44:25 i will you know i you know we we will we'll explain why they are progress but when you see the demo it might not be the end point it might not be the final thing like sometimes it makes sense for you to walk through the floor plan walking through the floor plan if it's just projected on the floor is still progress we've made progress on that floor plan and that's a good thing so you've got if you if you are if you don't yet have a culture of

44:50 of regular recurring demonstrations,

44:56 I would encourage that as part of this process.

44:59 I feel like that's a pretty important communication tool.

45:03 Scott: Let's say you don't have it, though.

45:05 Let's say you can't have it, but you just have to explain to somebody, a CFO or somebody like you're going to say, hey, so let's just say Jill.

45:15 Hey, Jill,

45:16 as a business person.

45:18 If you want to see if your engineering team is being more productive,

45:22 pay attention to how frequently they deploy and if that frequency is moving,

45:30 generally speaking,

45:31 up and to the right.

45:33 And they're going to say,

45:34 why?

45:34 Why is that a signal of how fast stuff is being done?

45:38 Brad: This is an analogy that I would be going to play this out before you use this one, but maybe it's a compounding interest thing.

45:46 Maybe you say like which what would you rather have would you rather have check in six months for $1,000 or would you rather I give you a dollar a day between now and six months from now and you've got interest compounding on both of them.

46:00 Scott: That's definitely one way you could say it. Another way you could say it is, well, look,

46:03 Jill,

46:04 big things don't go fast.

46:08 Brad: Yeah,

46:09 I like that.

46:10 until they say what about a rhinoceros pretty

46:12 Scott: I mean, how fast is a rhinoceros?

46:16 Brad: fast i think they can get up to like 60 plus miles an hour yeah

46:21 Scott: That is indeed pretty fast.

46:23 I mean, yeah, they could say, well,

46:25 what about a spaceship?

46:28 Brad: Yeah.

46:29 Scott: Okay.

46:29 Brad: So anyway,

46:30 I like I like the point that like in general,

46:32 big things go slower than small things and it's just going to depend on if you've got the you know, like how much of a curmudgeon you're you know, or how much of an antagonistic, you know, response. That's why I feel like that's why I feel like, you know, the proof's in the pudding,

46:45 right?

46:45 Like, I feel like, man,

46:46 if there's a way to get like if you if you could kind of God,

46:50 there's so many awesome benefits of if you can get a spirit.

46:54 and like you described that that rover demo god damn man like a culture where the whole team is like chipping in on a demo and like cheering for each other when they get to the other side of that demo

47:08 man

47:09 Scott: That would

47:09 Brad: that's

47:09 Scott: be really

47:09 Brad: like good that'd

47:10 Scott: demo.

47:10 Brad: be amazing

47:11 Scott: We had some really, when I was primarily working on Dolly, I think our listeners know Dolly was acquired by TaskRabbit. We had some really great demos at Dolly.

47:20 I want to say this because I want people to learn from it.

47:24 We would do demos on Friday morning and we would do a dress rehearsal.

47:31 at two o'clock on thursday afternoon because the first few demos were really rough they were just

47:37 Brad: Yeah.

47:37 Scott: like it was it was no bueno it was like hey this is embarrassing guys we got to get together the night before kind of like and we had an mc there was one specific person he was the demo mc he figured out the script each week and we practiced it and frequently between like 2 30 on thursday and 10 a.m on friday when we did the demo there was a lot of

47:57 A lot of scrambling to fix stuff. But that demo,

47:59 that dress rehearsal was so powerful because it got the team focused like,

48:05 oh, these are the gaps in demonstrating this thing.

48:08 Brad: Yeah,

48:08 Scott: And

48:08 Brad: totally.

48:09 Scott: it was really, really powerful.

48:10 Brad: I love that story and 100% agree that like a demo that goes,

48:16 a failed demo is way worse than no demo at all.

48:19 Scott: Very embarrassing. Yeah, so we got together and did that and it turned out the dress rehearsal was the really.

48:23 really powerful part than

48:24 Brad: Yeah.

48:25 Scott: the demo was just showing it off i'm wondering now i'm thinking let's say you start to say to your stakeholders the jills of the world focus on deploys could there be any downsides to that

48:36 Brad: Well,

48:37 I mean, I think there's always the downsides of you mean.

48:40 did they say that to you like what are the downsides

48:42 Scott: No,

48:42 Brad: oh yeah

48:43 Scott: no. Now I've got a CEO running around saying to everybody,

48:47 deploy faster,

48:48 deploy faster,

48:49 deploy more often,

48:50 Brad: yeah

48:50 Scott: deploy more

48:50 Brad: then

48:50 Scott: often.

48:50 Brad: you get then you get like bullshit deploys or you get like you you over optimize on that and you have people that you know we already talked about that you know people get afraid of doing big things the people that struggle at breaking things down they they they feel like they're unable you know whatever they they feel like they're

49:07 like they're dragging the team down the team feels like they're dragging them down you know you have the classic you have the classic problems with anything that's measured is going to change behavior

49:16 Scott: Yeah, but I'm thinking more of the CEO person being the one pounding that drum as opposed to if that's coming from,

49:24 I don't know, I just wonder if that could create a whole different.

49:29 But if there's a Pandora's box you're opening when you suddenly

49:33 Brad: I

49:33 Scott: have.

49:33 Brad: think the only – yeah,

49:35 I think that I'm going to stand by it that that is the normal Pandora's box you get whenever you pick any metric,

49:43 right?

49:47 You just got to try to think through your your unintended consequences. And so this is why this is why I said, like, I think that there's a lot of strong evidence that smaller PRs, smaller things like that are definitely there's a lot of evidence that that makes you generally generally a better practice,

50:04 safer,

50:04 Scott: Yeah

50:04 Brad: easier to test, easier to reason about all those reasons. So.

50:09 The side effect that we talked that I think is genuinely the worst possible side effect is sort of this big project paralysis that can happen.

50:21 And that's

50:22 Scott: Yeah, yeah.

50:22 Brad: the harder thing to teach engineering teams to do.

50:26 Scott: Well, yeah.

50:27 And the good news is, at least in the organization I'm at right now,

50:30 we have the opposite problem.

50:32 The organization has successfully delivered several large projects over the last number of years,

50:37 which has been great,

50:38 but actually needs to shift to a little bit more of a, we need to iterate on these, we need to iterate on these and make them better.

50:46 And so it naturally fits with what the business wants to do right now.

50:50 So it'll be good.

50:51 Brad: That sounds like a good thing.

50:53 Scott: All right.

50:53 Well, this has felt good,

50:55 Brad.

50:56 Brad: Yeah,

50:57 well,

50:57 I'm glad that you came to this session of CTO therapy.

51:01 Scott: Yeah.

51:01 Brad: Talk about the V

51:03 Scott: Thanks,

51:03 Brad: word

51:04 Scott: Doc.

51:04 Thanks,

51:05 Doc.

51:05 All right.

51:07 Until next time, smash

51:09 Brad: Until next time what do we say?

51:11 Scott: rate and subscribe.

51:12 All right. Thanks, everybody.

51:13 Brad: See you later

51:15 This episode of how many CTOs does it take podcast was edited by saffron hepigal our theme song was AI generated by Brad

51:23 Please follow us on all the social media platforms at at HowManyCTOsPod.

51:28 You can find us on YouTube,

51:30 LinkedIn,

51:31 Instagram,

51:31 Facebook,

51:32 Threads, X,

51:33 TikTok,

51:34 and Patreon,

51:35 or at our website at HowManyCTOsPod.com.

51:39 And remember to rate, review,

51:41 and subscribe on the podcast platform of your choice.