Pioneer or Perish: How and When to Evolve Your Engineering Org Toward the AI Development Life Cycle

0:00 Brad: Forget about AI for a second.

0:01 Like, forget that AI is even in this conversation.

0:05 What would you do normally?

0:07 Like, what would you do as an engineering team?

0:28 Welcome, everybody, to another episode of the How Many CTOs Does It Take podcast.

0:33 My name is Brad Hefta-Gaub,

0:34 and my trusty co-host is...

0:36 Scott: Scott Porad.

0:38 How are you doing, Brad?

0:39 Brad: I'm doing well. How are you doing today?

0:41 Scott: I'm doing good.

0:43 Brad,

0:44 I think our listeners know that

0:46 Brad: Yes.

0:46 Scott: we prepare.

0:50 For hours and hours

0:51 Brad: hours

0:51 Scott: and hours before

0:52 Brad: hours because we can

0:53 Scott: these

0:53 Brad: yes calls, we do yes we

0:55 Scott: no, not on the days when we don't have a guest,

0:58 sometimes we just roll the dice.

1:00 Brad: do we just roll the dice we have no idea

1:01 Scott: And today

1:02 Brad: what we're going

1:02 Scott: we're

1:02 Brad: to talk about

1:02 Scott: going to spin the wheel today and it's going to be Brad's choice.

1:06 Brad: well let's see i have had i had an interesting uh week this week where i was working with one of my

1:16 top high performance teams one of my top high performance engineers and he basically blew my mind on what he was doing how he was using AI and I was like oh okay

1:29 this i need to pay attention to this like this

1:31 Scott: He is blew your mind.

1:33 Brad: yeah he blew my mind because

1:35 Scott: i That's

1:35 Brad: because i

1:35 Scott: pretty

1:36 Brad: thought explosive. i okay so and and i'll just say that like the context here is that this guy and this team is doing the way i i've basically kind of been coding the way they've been coding with ai and incredible performance like this team has basically

1:55 completely consumed the product backlog of their product owner like there's nothing left for them to work on and they're they're the teams that are cranking out and of course we don't care about lines of code that's not the right way to measure business value but they're the ones that are like every single person on this team is doing 3 000 lines of code a day per developer consistently every day like they're just cranking out business value

2:19 And so I was a little bit like, hey,

2:22 man, hard,

2:23 hard to hard to optimize past that.

2:25 Right.

2:25 Scott: Mm

2:26 Brad: Like they're doing pretty they're doing pretty great stuff.

2:29 And, you know, I kind of mentioned a couple of times ago we met and talked about this,

2:35 how I kind of went viral a little bit by saying like,

2:37 hey,

2:38 you know.

2:40 nobody's running agents overnight that's not how this is done they're like real serious engineers are actually like guiding the agents very very clear plan modes very clear like guard rails on quality and um and this is one of the engineers that i was thinking about when i said that and i noticed he was doing code reviews on people's prs

3:05 like really fast and he was getting some really good quality out of these code reviews like real insights and so i was like and when i say really fast what i mean is they were kind of appearing out of nowhere like no one was asking him to do the code review and they were appearing and i was like what is going on and i asked him and he it turns out he is running the agents overnight

3:30 And he has built up a system for that that is quite impressive.

3:36 Scott: Okay, before I respond to that,

3:37 Brad: Yeah.

3:37 Scott: I just have to, I know you're a music guy.

3:39 When you talked about getting to the end of the product backlog that the product owner didn't have anything left,

3:45 it reminded me of a song by a rapper I like, a woman.

3:51 the refrain of this song is no one ever really knows what happens at the end of the road but it sounds like that team has gotten to the end of the road okay

4:00 Brad: I mean, they've at least gotten to the point where now their product owner needs to start cranking out new features or decide how to use that value.

4:11 Scott: so

4:12 Coming back to the guy who's doing code reviews all the time. So is what's going on there is he has like something listening for a web hook on GitHub and every time somebody submits a PR.

4:27 wakes up an agent and goes and fetches the data

4:30 Brad: Yeah,

4:30 Scott: and

4:30 Brad: that's

4:31 Scott: when I say

4:31 Brad: part of

4:31 Scott: the

4:31 Brad: it.

4:31 Scott: data goes and fetches the PR and

4:33 Brad: Yep, that's a big part of it.

4:34 Scott: is

4:34 Brad: So he.

4:35 Scott: that what he's got going on

4:36 Brad: Yeah,

4:36 so yeah,

4:37 and we should get him on the pod and I chatted with him. He's interested in coming on and giving so so we won't go into all the details because he'll be able to do it better than I will.

4:45 But effectively,

4:46 yeah,

4:46 he he's essentially he's not using quad code to do this, but another once we start.

4:52 started talking about this another person on the team goes oh yeah i started using the new claude code uh loop feature the scheduled scheduled agent feature that and it can do the same thing and i'm having it do that but i'm just having it doing on my prs so basically yeah the guy's got a harness and we should talk about like the components that make all this work but he's got his own harness that um has a whole collection of skills and markdown files that represent what he described

5:17 he describes as a world-class code review so like you know he's given it you know he he's he's got years he's you know a senior engineer he's got years of his own doing code reviews against code he's got years of his own writing code and so he's developed he's got his own rag setup which is like his his his opinionated view on how to do code reviews and all of that is arming his

5:42 work his harness that pulls at the harness is able to like pull down the latest pr for the repos that he's interested in looking at he's got them all listed something new comes into that repo it pulls it down runs a code review against using his opinionated way of doing code reviews and then submits comments back to the pr

6:01 Scott: I think what's really interesting about that whole thing is the part you said about the opinionated.

6:09 code review and in particular you've probably heard the expression before about pornography I can't describe it but I know it when I see it I

6:19 Brad: Yeah, describe

6:20 Scott: think what I'm taking from the story of this guy is that to use agents and skills you can't treat it like pornography you actually have to even if you have an opinion you have to be able to describe

6:35 Brad: your opinion.

6:36 Scott: your opinion right

6:37 Brad: Yeah.

6:38 Scott: and and in some you know sort of concrete kinds of ways you couldn't just tell an agent well you'll know it if you see it right

6:46 Brad: Yeah.

6:46 And that being said,

6:47 and again,

6:48 like we'll have to like if we can get him on the pod and we can talk with him, have him tell more about what he feels comfortable sharing.

6:56 I do think that you can.

6:58 You can get a long ways with most of the LLMs by saying,

7:02 you know, you are a security expert.

7:05 Your job

7:05 Scott: Mm

7:06 Brad: is

7:06 Scott: -hmm.

7:06 Brad: to look for the following types of concerns, and then you enumerate those concerns.

7:11 And so I think it's very possible that what part of his opinion is an enumeration of the types of things that matter to him when he does code reviews.

7:23 Right.

7:24 Scott: A hundred percent. What I'm simply getting at is that for a lot of engineers all throughout time,

7:30 or at least the time that I've worked, which is not all of time,

7:34 but most of it.

7:35 And well,

7:36 that gets at the nature of time. So let's not go there.

7:40 But a lot of things we do, we have a hard time describing, you know, a lot of the judgments we make, we sort of have a hard time.

7:49 describing as a matter of fact now I'll use an example from a conversation I just had an hour and a half ago with one of my engineers on my team we were talking about a certain part of the system and some aspect I don't want to go into the details but I just said I was like my gut is just telling me that's you know we want to do it this way I couldn't I couldn't tell him why I thought that I was just like I don't know I've been doing this a long time and my I'd say

8:14 I just feel like the way you're describing is not as good as this other way.

8:19 I couldn't probably if I really thought about it, I could break it down and try and explain.

8:23 And I just think there's a lot of that that we do in our field. And then when you start to use agents,

8:30 you have to be able to start to articulate that a little bit better.

8:34 Brad: Yeah, I think that's fair.

8:35 I think the thing that I was struck by.

8:38 is that another thing he said was he then started to describe because once we got into it, I was like, oh, okay, tell me more about what your framework is doing and what elements are you using?

8:55 What are the skills that you've set up?

8:59 And there's a couple things that he said that I think are also interesting threads to pull on.

9:05 The first one is that, and this is already been,

9:09 like, this should be well known to most of our listeners who are paying attention to AI.

9:14 You know, you can swap out the models pretty easily with most of these tools,

9:21 and you can...

9:23 It's more and more clear that the frontier models might be a little bit ahead,

9:29 but the open source models are getting so good that

9:33 Scott: Mm

9:33 Brad: there's a real question about whether the model is doing that much. How much does the model do versus how much is the framework and tooling around the model actually

9:44 Scott: -hmm.

9:45 Brad: doing the thing, right? And so, for example.

9:49 um,

9:50 quad code is the model that a lot of people are using the framework that a lot of people are using. And you can, you know, swap out different versions of the LLM behind it and it still does all of its stuff,

10:02 right?

10:03 Cursor as an IDE has toggleable different, um, models.

10:09 You can pick different models and the cursor team basically released their own model so that they didn't have to, you know, rent tokens from.

10:16 from the

10:16 Scott: Amen.

10:16 Brad: frontier models and you know i i would tell you i've told you before that i think that their composer model is um absolutely performing just as well and it's cheaper than than the other frontier models i rarely ever ask cursor to use one of the frontier models because

10:32 composers just fine right

10:34 Scott: Mm

10:34 Brad: um

10:34 Scott: -hmm, mm

10:35 Brad: so there's the ip of the model and there's this big huge gargantuan battle between the the frontier you know models to try to make a better model but maybe the framework is what matters and

10:48 Scott: -hmm.

10:49 Brad: so then the question is okay well do you he had essentially built his own tooling because he started on it like you know 12 months ago

11:00 Where open CLAW didn't exist yet,

11:03 CLAW code didn't quite have the same tooling to it.

11:07 OpenClaw and ClaudeCode and his own little tooling thing,

11:12 they're all also rapidly approaching commoditization, right?

11:16 So the models themselves are essentially approaching commoditization.

11:20 And his thesis,

11:22 which I, you know, in chatting with him about it, I was like,

11:24 oh yeah,

11:24 that makes sense, is that the framework around the models,

11:28 those things are also

11:32 approaching commoditization right

11:34 Scott: Mm-hmm.

11:34 Brad: so especially now that open you know there's like there's there's a dozen different variants of open claw and and a bunch of them are open source and they're they're all kind of competing for that framework space um and so his thesis is like hey the thing that matters is probably more than anything is

11:55 As he thinks about being an AI architect,

11:58 AI engineer is like,

12:00 hey,

12:00 it really matters like how do you put all these pieces together like the model that the models themselves are going to be interchangeable.

12:07 Maybe you're going to choose different ones based on the type of task.

12:10 The frameworks are going to be commoditized. So what really matters is like your ability to sort of like.

12:17 frame up, piece these things together and create opinionated skills about how to do the thing that you want done in a professional manner.

12:28 Scott: Yeah, yeah,

12:30 I don't have anything to add to that.

12:31 I agree.

12:32 Brad: So then the other thread that I think you can pull on is,

12:35 okay,

12:35 what does this mean for engineering teams?

12:38 Like, how should they be using

12:40 Quad code? How should they be using frameworks?

12:43 How should they be using models?

12:45 And how should they be integrating whatever they're doing into their software development lifecycle?

12:51 Scott: Well, this is, I feel like you've teed up the topic on my mind and I don't have any answer for it.

12:57 Maybe our 39 listeners can leave a comment in the comments wherever this is posted.

13:01 But I think that's actually where we're at right now, which is we've got all sorts of people experimenting, doing different things. And I can't remember if I've talked about this in the podcast.

13:11 I went a couple of weeks ago to a two-day workshop led by the AI Tinkerers Group on setting up a software factory.

13:19 Brad: Yeah.

13:19 Scott: And we worked through that for a couple of days.

13:22 And what I took from that was those were the two days that we kind of entered a new era.

13:30 We went from when I did that,

13:32 I went from, oh, OK,

13:34 we've got a bunch of people. There's a whole spectrum of how people on our team are doing it. Some people are really far ahead. Some people are resistant.

13:42 You know, most people are in the middle.

13:44 And what I took from that was, oh,

13:48 okay,

13:48 we need to kind of set up a, I'm going to try not to use loaded words, we need to develop a common approach amongst ourselves so that the roughly 100 or so people on this team are.

14:01 doing it similarly because right now it's all over the map and it's not all over the map in a bad way because all that experimentation is good it even occurred to me in a conversation I had with some of my team about starting to standardize this there wasn't even really agreement

14:23 like what should be a skill what should we what should do how should the parts all fit together basically you know forget forget what i said what should be a skill just how should the parts all fit together and it's not that they were it wasn't even it wasn't like people who were butting heads and disagreeing it was more like huh should we do it that way or

14:39 Brad: Yeah.

14:39 Scott: should we do it this way you know it's all it's all very open and greenfield

14:43 And so what we're kind of thinking of, this is the idea we're toying with.

14:46 You know the notion of long-term support with software,

14:49 right?

14:50 Companies will have, you know, version one,

14:52 version two, version three, and they'll have some version where they give you long-term support and they'll tell you we're going to support this for a while.

14:57 We're kind of thinking of like, maybe we do this long-term.

15:01 support approach where it's like well this is how we're going to do it and we're going to do it that way for say three months and then we'll we'll learn during that time and then we'll revise it and the reason i bring up the long-term support is one of the challenges we're having that's moving so fast right

15:15 Brad: Yeah.

15:15 Scott: people are saying we you just you just suggested that i do it this other way like last week and now you're telling me to do it this way i'm just trying to get my job done i'm just trying to get some software built here like i can't be changing my whole development

15:26 world every day.

15:27 And so you can kind of have people experimenting,

15:29 but then you sort of roll that into your LTS.

15:34 let's say quarterly or monthly i don't know if and this fits back with that the aldc that we talked about many episodes ago it's like that's kind of it's kind of what we need to do is build our agent development life cycle like how their ai development life cycle what are all the parts and how do they fit together i think this is the name of the game right now

15:50 Brad: Yeah, yeah, I think so.

15:51 And I think so.

15:52 So that really resonates with me and I have a couple I have a couple of reactions to it. The first reaction I'm going to say is.

16:01 you know this developer that i was talking about um one of the things he said which i was like this is this was the moment where i was kind of like i didn't feel like my i was necessarily my mind was blown but this was where i was kind of like oh yeah yeah old guy here is getting left behind i gotta get i gotta up my game was he basically said look man if i do something consistently multiple times a day i need to give that to my agent like if i'm a if i'm doing the same thing multiple times a day

16:29 So that's a me teach the agent how to do it so it does it.

16:32 Scott: Mm-hmm.

16:33 Brad: I teach the agent how to do it the way I do it so that it does it my way.

16:37 Scott: Yeah.

16:37 Brad: And so this was a simple example, and this was the reason that like the beginning of the thread that I pulled on was,

16:45 hey, wow,

16:46 you're doing code reviews like we all do code reviews multiple times a day,

16:50 right?

16:51 This team is cranking out 3,000 lines of code per developer per day.

16:54 like there's you know every single person has got to you know go do these code reviews right and so so he was just like i like even when i'm using agents to do things like do code reviews i was still like i am still like saying oh now it's time for you to go look do look at this pr hey agent go look at this pr do a code review for it

17:18 And what he's gotten past is he doesn't even have to tell the agent to go do that.

17:23 Scott: Right, right.

17:24 Brad: He has the automatic queue getting created.

17:27 But because it's paying attention to the stuff that matters to him, it's sucking it in as a task for the harness to do.

17:36 And then it knows and he's told it, you are allowed to do these things without my supervision.

17:42 And then these are the gates you come to where I will supervise you.

17:46 Scott: Mm

17:46 Brad: And

17:47 Scott: -hmm.

17:47 Brad: this is the artifact I want you to give me.

17:49 And then I will tell you to proceed or not.

17:51 Scott: So that, and you mentioned earlier, there's another person on that team who was doing something similar and had devised it in their own way.

17:59 Brad: And they had devised it in their own way using a different framework using it. And I effectively am doing it my own way using a slightly different tool, but a more, even though again,

18:09 I'm like this.

18:10 And then there was another developer in the conversation,

18:12 he was like,

18:12 Oh yeah,

18:13 I'm kind of doing it Brad's way like I'm also two manual and

18:16 And they were both like,

18:17 oh yeah,

18:18 we need to figure out how to do it more automated so we can speed up even more.

18:23 Scott: Well, let me ask you a question.

18:24 My point of view about talking about the whole ALDC and LTS and stuff a minute ago was that thinking,

18:33 OK,

18:33 it's not good that everybody's doing it their own way because some people are doing it better than others.

18:42 And some people are the fact and because of the rate of change,

18:47 some people aren't even really able to keep up.

18:49 So I guess I want you I want to pressure test an assumption I have.

18:52 Brad: Yeah.

18:53 Scott: When I hear you say you've got three or four people on your team all doing this differently,

18:56 I'm saying to myself,

18:57 oh,

18:58 that's I would rather like that group of people come along and have one skill they all share across their team.

19:07 But I can also see.

19:08 some benefit maybe in the diversity so like I'm just curious to know like maybe I'm heading down the wrong path of trying to structure this and make it consistent

19:16 Brad: I will.

19:17 OK,

19:18 that's that was kind of one of the other threads I wanted to pull on because I knew you wanted to go down that that direction a little bit.

19:23 And I'm going to I'm going to preface this by saying.

19:27 I bet you this is a it depends answer.

19:30 Scott: Of course it depends.

19:32 Where is Kevin when we need him?

19:34 Brad: Where's Kevin when we need him?

19:35 So I think this one's going to be an it depends. First of all, the diversity and the experimentation and the willingness to let smart people innovate,

19:49 especially in the rapid.

19:53 rapidly changing times that we're in feels

19:55 Scott: Mm-hmm.

19:56 Brad: like it's got to be the right call like oh

19:59 Because standardization is optimization and premature optimization is the is the death of so much software.

20:07 Scott: Well, that's why I'm trying to blend it together here.

20:09 I'm trying to say, look,

20:10 we're going to have a standard approach that everybody can use.

20:14 And anybody can go innovate on their own. You don't have to use the standard approach because there's people where I can't stop them.

20:20 They're

20:20 Brad: Yeah.

20:20 Scott: going to they're going to they're

20:21 Brad: Exactly.

20:22 Scott: going to they're sending me text messages on, you know, 8 p.m. on Saturday night telling me the cool stuff they're doing.

20:27 Brad: Yeah.

20:27 Scott: Right. So, like, I can't stop them.

20:29 So it's like, well,

20:30 go do it. Great.

20:32 And then let's learn what's really working and then let's put that into the standard version because not everybody on this team is going to be as excited about you and use their Saturday night.

20:40 night to figure out how to make some cool thing work yeah

20:42 Brad: Well, so first of all, it sounds like we're aligned on the idea that like having a freedom to innovate, you know, is the American way and let's let people do that. Okay,

20:55 so that we're

20:55 Scott: capitalism for the victory

20:56 Brad: aligned on that.

20:58 I like I get the so the other part about your comment that resonates for me, though,

21:03 is that

21:05 If you are feeling stuck,

21:07 if you are feeling like you're being left behind,

21:10 then man,

21:13 that sure would help if someone would give you the recipe to

21:16 Scott: Yeah.

21:16 Brad: tell you how to set it up to not be stuck.

21:19 Scott: Mm hmm.

21:19 Absolutely.

21:19 Brad: So that also makes sense and resonates for me.

21:24 But then the other tension there is, man,

21:26 this stuff is changing so fast. Even if you document it, it might be obsolete.

21:30 It certainly will be obsolete in three months. It might be obsolete in

21:35 Scott: Yeah.

21:35 Brad: six weeks.

21:35 Scott: I'm not even talking about documenting.

21:37 I'm just like, let's talk about your code reviewer skill.

21:40 It's like, OK,

21:41 gang, here's here's our common code reviewer skill.

21:44 We've all put our best thoughts on how to make it.

21:45 make a really good code reviewer skill here,

21:47 right?

21:48 Brad: Yeah, I think there's value in capturing that.

21:51 By the way, I will say the other thing about this is,

21:55 and I keep saying this to people,

21:56 I literally just said it to a team that I was doing some AI coaching with earlier today where they were asking me like,

22:04 well,

22:04 what should we do here?

22:06 And I basically said,

22:07 forget about AI for a second.

22:09 Like forget that AI is even in this conversation.

22:13 what would you do normally like what would you do as an engineering team i think the question was like i think the question was like oh um uh we're going to use spec kit sdd to produce uh the do we're going to do spec you know spec kit specify and it produces this like essentially prd and then we run clarify on it it's it's refined that it's really good and they were like at this point should we

22:39 have other developers review it or should we go to the implement should go to the plan phase before and then give them both artifacts right and I was just like the answer to that question has nothing to do with AI

22:51 Scott: yeah that's right someone at our company this week or last week it must have been last week i don't recall gave a presentation about this exact same thing he now we got a lot of people using spec kit i don't know if folks on the

23:06 audience or know what SpecKit is, it's sort of an STD framework.

23:10 it helps you develop your spec and your implementation but it's kind of basically plan mode you know in cursor and he just mapped each of the things to each of the steps in our sdlc and he's like gang this is no different than what we're already doing you're just getting the ai to do it for you and you're double checking the work like

23:28 Brad: Yeah

23:28 Scott: and in fact it's doing it better because lots of times we'd skip a bunch of these steps because we would think it's easy or because we were lazy or because someone was impatient

23:36 and this because it's so fast they don't have to skip them anymore

23:39 Brad: Yeah.

23:39 Scott: so to me that really resonated and then i'm building on that with what you just said which is like forget about the ai like it's just our sdlc what would you do right well i wouldn't send it

23:51 thing out there without having so the ship some code without having it reviewed okay who's going to review it you or this robot over here

23:58 Brad: Right.

23:59 Exactly.

23:59 And so what I think is interesting about that to apply to the question of standardization and or like the, you know, kind of like the guidance that you want to give a team like, hey,

24:12 like, let's get let's start doing it this way.

24:14 The way the what is the way you're going to do it? Right.

24:19 That is definitely,

24:20 Scott: that's the thing

24:21 Brad: that's

24:21 Scott: yeah

24:22 Brad: like the way you would have done it independent of the AI.

24:25 Scott: but there's this is this is where it gets complicated because you kind of have to agree but also you have to figure out like how do we map these all the tools i'm just going to broadly say all the tools out there right all this stuff out there we kind of got to figure out how to map our stuff onto the

24:42 Things out of the tools that are out there those tools are changing really rapidly.

24:47 Brad: Right.

24:47 Scott: So that's where it gets tricky

24:49 Brad: I agree with that statement. So but but again, like if we like let's just pause,

24:54 pause the film for a second here.

24:56 Right.

24:57 We're going to pause the film and we're going to say,

24:58 OK,

24:59 what happened here?

24:59 What was what was Brad like, you know, impressed by?

25:03 And the answer to that question is this is a team who has a very strong SDLC.

25:10 it's as long as i've worked with them i've worked with them for like two and a half years now they've always had a so well before the early ai stuff like they were already they had a strong sdlc it was like we get a ticket from product we think about that we write up our plan on that

25:29 we start to code it when we're done coding it we put up a pull request that pull request gets reviewed that you know that it already went through all the quality gates of automated testing blah blah and you know when we we merge it then it's in our staging environment we run our uat you know blah blah it's like that was just their sdlc the thing and and along the way

25:56 AI has gotten better and better.

25:57 And each of these little parts of it, they've been adding a little AI spice to each little part.

26:05 Oh,

26:05 Scott: Mm-hmm.

26:05 Brad: the agents have gotten good enough that if I just give them a good quality ticket,

26:10 they will write a great implementation plan.

26:14 I trust that.

26:16 Let's do it, right?

26:17 Before that, there was the if I write the implementation plan.

26:21 And I give it to the agent. It does a good job of coding it.

26:24 Scott: Mm

26:24 Brad: But

26:24 Scott: hmm.

26:24 Brad: now they went to, okay,

26:25 you're not just good at coding it. You're actually good at writing the implementation plan.

26:28 I just need to read that.

26:29 Oh,

26:30 and now it's like,

26:31 oh, wait, they're actually really good at writing the specification for me better than I am at writing the specification,

26:37 right?

26:37 And now I'm just in it. And they're really good at doing code reviews. And so this team has already been putting in the agents at each of these places.

26:47 Scott: Just use the existing SDLC as the skeleton and start to add the agents to it.

26:52 Brad: And then what the next step, after you've done that, where manually at each place in the SDLC,

26:59 a human being sits down and says,

27:01 hey, Mr.

27:02 Agent,

27:02 now it's your time to write the spec.

27:04 Great,

27:05 write the spec.

27:06 Come back 10 minutes later.

27:08 Hey,

27:09 Agent,

27:09 I've read your spec.

27:10 I like it. Now it's time for you to write the implementation plan.

27:13 Right?

27:14 Come back five minutes later,

27:16 read the implementation plan.

27:17 Oh, hey, good job,

27:18 agent.

27:19 I trust that implementation plan.

27:22 Now it's your time to write the code,

27:24 dot, dot, dot, dot, dot.

27:26 What I'm saying is like.

27:28 This is the part that I feel like I went viral a couple of weeks ago saying like, no one's doing this overnight, right?

27:36 Now I'm like,

27:38 oh, wait a second,

27:39 this engineer that I actually really trust is doing it overnight. And he has just figured out that some of those steps in his process can now be automated so long as he has.

27:54 Based on the framework and in his mind of what the definition of quality is,

28:00 he's able to get the checkpoints that matter.

28:05 Scott: Yeah. So it's really the quality gates are the key.

28:09 Brad: Yeah, and he even said that. He was like,

28:11 look,

28:11 I've gotten to the point now where I can be really clear about what the definition of a correct looking acceptance criteria is for each of these steps along the way.

28:27 Scott: So we had an engineer at our weekly engineering show and tell show something off sort of similar this week.

28:33 I just thought of it as you mentioned this.

28:35 He was doing something slightly different.

28:38 Every time we have a test fail, an automated test failure,

28:43 he was triggering something.

28:48 you can imagine the steps basically having the AI figure out the solution and create a PR and then he was looking up on code owners who owned that area and just assigning it to that team say I think I have you know effectively saying this could be the fix to the problem and

29:02 Brad: Right.

29:03 Scott: he said he's just testing it locally right now like just within himself he isn't assigning them to the teams yet he's seeing if he can get it there

29:11 which is pretty cool and someone else in our meeting was like could we just start feeding that same thing bugs bug tickets it's

29:18 Brad: Yeah.

29:18 Scott: like computer errors from data dog he's like yeah i don't see why not

29:21 Brad: Well,

29:22 okay,

29:22 cool.

29:23 I love that story,

29:24 and I'm going to double down on that. Again,

29:29 this is super...

29:31 compelling story from this engineer that i trust one of the things he mentioned is yeah i have an overnight one of my jobs that i very clearly specified is the agents are to get 100 test coverage for

29:45 Scott: Oh yeah,

29:46 for sure.

29:47 Brad: but for old code that doesn't have test coverage so

29:51 Scott: Oh, interesting.

29:52 Brad: he has agents that are just

29:56 preemptively going to find places in the code that are missing test coverage well

30:03 Scott: I wonder how it does that,

30:04 actually.

30:05 Does it go execute the tests and then do some sort

30:08 Brad: you

30:08 Scott: of coverage

30:08 Brad: yeah

30:08 Scott: reporting?

30:09 Brad: i mean your your coverage your coverage report at least you

30:11 Scott: It's

30:11 Brad: know it

30:12 Scott: giving it the coverage report.

30:13 Brad: gives you the coverage report and

30:14 Scott: Got

30:14 Brad: it tells

30:14 Scott: it.

30:14 Brad: you tells you the functions and the lines and the branches that haven't been covered and so he's just like look i like

30:23 this is i mean this is a good example where it's like it's pretty hard to go wrong on writing a test i mean you can write tests that don't actually test the thing but if you run every line of code right in the test then you definitely got more coverage than if you didn't run every line of code and so that's like a goal seeking like that's a very straightforward thing for a a goal seeking framework to be like

30:49 Keep going until

30:50 Scott: Yeah.

30:51 Brad: you get 100% coverage.

30:53 Scott: Yeah.

30:53 Yeah, that's right.

30:56 Brad, it's crazy.

30:59 I am now squarely in the camp that in five years,

31:02 we're not even going to recognize this industry.

31:04 I mean, it's just going to be so different.

31:06 Like my career, which has lasted roughly the last 30 years, a little bit more.

31:10 you know has been the internet era like i didn't i never worked on non-internet i didn't work on any client server software that was that was pre i came into this with the start of the internet 1995

31:20 is when i got my first job as a doing

31:23 Brad: Yeah.

31:23 Scott: programming it's been you know the same

31:27 So what's changed?

31:29 Cloud, that's the main thing that changed.

31:31 Cloud and mobile,

31:32 those are the two big things that changed over my time.

31:34 But it's all been kind of the same refining of the same thing.

31:39 Brad: Yeah.

31:39 Scott: Yeah, you know, and

31:40 Brad: Yeah,

31:40 Scott: to me, this is like,

31:41 oh,

31:41 this is just now we on one hand, it's like, oh, this is changing everything. But then we just talked about it's the same as DLC.

31:49 Brad: that's the thing that I'm holding on to. And I also think that this is a great example where the engineer that I'm talking about here,

31:58 he's one of the most successful,

32:01 compelling users of this stuff, and he's a great engineer.

32:05 And he has been.

32:05 has been and he's pretty young actually and he's been kind of like oh man what's going to happen to the industry right but what i see him doing is like exactly what i would tell every engineer to do like you just gotta get great at giving your giving your opinion to the to the tooling and your opinion still matters your taste still matters

32:33 Scott: Just leave it at that.

32:34 Brad: Your taste still matters.

32:36 Scott: Your taste still matters.

32:38 Brad: There you go.

32:39 Scott: All right.

32:40 Brad: All right.

32:40 Scott: Brad, it's been nice catching up this week.

32:41 Brad: Great catching up.

32:43 What do we say?

32:45 Scott: Smash rate and review.

32:46 Brad: Smash that subscribe button.

32:48 All right, see you guys next week.

32:49 Scott: All right, bye.

32:50 Brad: This episode of the How Many CTOs Does It Take podcast was produced and edited by Safran Heptekau.

32:56 Our theme song was AI generated by Brat. Please follow us on all the social media platforms at HowManyCTOsPod. You can find us on YouTube,

33:07 LinkedIn,

33:07 Instagram,

33:08 Facebook,

33:08 Threads, X,

33:09 TikTok,

33:10 and Patreon.

33:11 or at our website at HowManyCTOsPod.com.

33:15 And remember to rate, review,

33:16 and subscribe on the podcast platform of your choice.

33:19 Thanks for listening.