Transcript: Pioneer or Perish: How and When to Evolve Your Engineering Org Toward the AI Development Life Cycle
0:00 Brad Forget about AI for a second.
0:01 Brad Like, forget that AI is even in this conversation.
0:05 Brad What would you do normally?
0:07 Brad Like, what would you do as an engineering team?
0:28 Brad Welcome, everybody, to another episode of the How Many CTOs Does It Take podcast.
0:33 Brad My name is Brad Hefta-Gaub,
0:34 Brad and my trusty co-host is...
0:36 Scott Scott Porad.
0:38 Scott How are you doing, Brad?
0:39 Brad I'm doing well. How are you doing today?
0:41 Scott I'm doing good.
0:43 Scott Brad,
0:44 Scott I think our listeners know that
0:46 Brad Yes.
0:46 Scott we prepare.
0:50 Scott For hours and hours
0:51 Brad hours
0:51 Scott and hours before
0:52 Brad hours because we can
0:53 Scott these
0:53 Brad yes calls, we do yes we
0:55 Scott no, not on the days when we don't have a guest,
0:58 Scott sometimes we just roll the dice.
1:00 Brad do we just roll the dice we have no idea
1:01 Scott And today
1:02 Brad what we're going
1:02 Scott we're
1:02 Brad to talk about
1:02 Scott going to spin the wheel today and it's going to be Brad's choice.
1:06 Brad well let's see i have had i had an interesting uh week this week where i was working with one of my
1:16 Brad top high performance teams one of my top high performance engineers and he basically blew my mind on what he was doing how he was using AI and I was like oh okay
1:29 Brad this i need to pay attention to this like this
1:31 Scott He is blew your mind.
1:33 Brad yeah he blew my mind because
1:35 Scott i That's
1:35 Brad because i
1:35 Scott pretty
1:36 Brad thought explosive. i okay so and and i'll just say that like the context here is that this guy and this team is doing the way i i've basically kind of been coding the way they've been coding with ai and incredible performance like this team has basically
1:55 Brad completely consumed the product backlog of their product owner like there's nothing left for them to work on and they're they're the teams that are cranking out and of course we don't care about lines of code that's not the right way to measure business value but they're the ones that are like every single person on this team is doing 3 000 lines of code a day per developer consistently every day like they're just cranking out business value
2:19 Brad And so I was a little bit like, hey,
2:22 Brad man, hard,
2:23 Brad hard to hard to optimize past that.
2:25 Brad Right.
2:25 Scott Mm
2:26 Brad Like they're doing pretty they're doing pretty great stuff.
2:29 Brad And, you know, I kind of mentioned a couple of times ago we met and talked about this,
2:35 Brad how I kind of went viral a little bit by saying like,
2:37 Brad hey,
2:38 Brad you know.
2:40 Brad nobody's running agents overnight that's not how this is done they're like real serious engineers are actually like guiding the agents very very clear plan modes very clear like guard rails on quality and um and this is one of the engineers that i was thinking about when i said that and i noticed he was doing code reviews on people's prs
3:05 Brad like really fast and he was getting some really good quality out of these code reviews like real insights and so i was like and when i say really fast what i mean is they were kind of appearing out of nowhere like no one was asking him to do the code review and they were appearing and i was like what is going on and i asked him and he it turns out he is running the agents overnight
3:30 Brad And he has built up a system for that that is quite impressive.
3:36 Scott Okay, before I respond to that,
3:37 Brad Yeah.
3:37 Scott I just have to, I know you're a music guy.
3:39 Scott When you talked about getting to the end of the product backlog that the product owner didn't have anything left,
3:45 Scott it reminded me of a song by a rapper I like, a woman.
3:51 Scott the refrain of this song is no one ever really knows what happens at the end of the road but it sounds like that team has gotten to the end of the road okay
4:00 Brad I mean, they've at least gotten to the point where now their product owner needs to start cranking out new features or decide how to use that value.
4:11 Scott so
4:12 Scott Coming back to the guy who's doing code reviews all the time. So is what's going on there is he has like something listening for a web hook on GitHub and every time somebody submits a PR.
4:27 Scott wakes up an agent and goes and fetches the data
4:30 Brad Yeah,
4:30 Scott and
4:30 Brad that's
4:31 Scott when I say
4:31 Brad part of
4:31 Scott the
4:31 Brad it.
4:31 Scott data goes and fetches the PR and
4:33 Brad Yep, that's a big part of it.
4:34 Scott is
4:34 Brad So he.
4:35 Scott that what he's got going on
4:36 Brad Yeah,
4:36 Brad so yeah,
4:37 Brad and we should get him on the pod and I chatted with him. He's interested in coming on and giving so so we won't go into all the details because he'll be able to do it better than I will.
4:45 Brad But effectively,
4:46 Brad yeah,
4:46 Brad he he's essentially he's not using quad code to do this, but another once we start.
4:52 Brad started talking about this another person on the team goes oh yeah i started using the new claude code uh loop feature the scheduled scheduled agent feature that and it can do the same thing and i'm having it do that but i'm just having it doing on my prs so basically yeah the guy's got a harness and we should talk about like the components that make all this work but he's got his own harness that um has a whole collection of skills and markdown files that represent what he described
5:17 Brad he describes as a world-class code review so like you know he's given it you know he he's he's got years he's you know a senior engineer he's got years of his own doing code reviews against code he's got years of his own writing code and so he's developed he's got his own rag setup which is like his his his opinionated view on how to do code reviews and all of that is arming his
5:42 Brad work his harness that pulls at the harness is able to like pull down the latest pr for the repos that he's interested in looking at he's got them all listed something new comes into that repo it pulls it down runs a code review against using his opinionated way of doing code reviews and then submits comments back to the pr
6:01 Scott I think what's really interesting about that whole thing is the part you said about the opinionated.
6:09 Scott code review and in particular you've probably heard the expression before about pornography I can't describe it but I know it when I see it I
6:19 Brad Yeah, describe
6:20 Scott think what I'm taking from the story of this guy is that to use agents and skills you can't treat it like pornography you actually have to even if you have an opinion you have to be able to describe
6:35 Brad your opinion.
6:36 Scott your opinion right
6:37 Brad Yeah.
6:38 Scott and and in some you know sort of concrete kinds of ways you couldn't just tell an agent well you'll know it if you see it right
6:46 Brad Yeah.
6:46 Brad And that being said,
6:47 Brad and again,
6:48 Brad like we'll have to like if we can get him on the pod and we can talk with him, have him tell more about what he feels comfortable sharing.
6:56 Brad I do think that you can.
6:58 Brad You can get a long ways with most of the LLMs by saying,
7:02 Brad you know, you are a security expert.
7:05 Brad Your job
7:05 Scott Mm
7:06 Brad is
7:06 Scott -hmm.
7:06 Brad to look for the following types of concerns, and then you enumerate those concerns.
7:11 Brad And so I think it's very possible that what part of his opinion is an enumeration of the types of things that matter to him when he does code reviews.
7:23 Brad Right.
7:24 Scott A hundred percent. What I'm simply getting at is that for a lot of engineers all throughout time,
7:30 Scott or at least the time that I've worked, which is not all of time,
7:34 Scott but most of it.
7:35 Scott And well,
7:36 Scott that gets at the nature of time. So let's not go there.
7:40 Scott But a lot of things we do, we have a hard time describing, you know, a lot of the judgments we make, we sort of have a hard time.
7:49 Scott describing as a matter of fact now I'll use an example from a conversation I just had an hour and a half ago with one of my engineers on my team we were talking about a certain part of the system and some aspect I don't want to go into the details but I just said I was like my gut is just telling me that's you know we want to do it this way I couldn't I couldn't tell him why I thought that I was just like I don't know I've been doing this a long time and my I'd say
8:14 Scott I just feel like the way you're describing is not as good as this other way.
8:19 Scott I couldn't probably if I really thought about it, I could break it down and try and explain.
8:23 Scott And I just think there's a lot of that that we do in our field. And then when you start to use agents,
8:30 Scott you have to be able to start to articulate that a little bit better.
8:34 Brad Yeah, I think that's fair.
8:35 Brad I think the thing that I was struck by.
8:38 Brad is that another thing he said was he then started to describe because once we got into it, I was like, oh, okay, tell me more about what your framework is doing and what elements are you using?
8:55 Brad What are the skills that you've set up?
8:59 Brad And there's a couple things that he said that I think are also interesting threads to pull on.
9:05 Brad The first one is that, and this is already been,
9:09 Brad like, this should be well known to most of our listeners who are paying attention to AI.
9:14 Brad You know, you can swap out the models pretty easily with most of these tools,
9:21 Brad and you can...
9:23 Brad It's more and more clear that the frontier models might be a little bit ahead,
9:29 Brad but the open source models are getting so good that
9:33 Scott Mm
9:33 Brad there's a real question about whether the model is doing that much. How much does the model do versus how much is the framework and tooling around the model actually
9:44 Scott -hmm.
9:45 Brad doing the thing, right? And so, for example.
9:49 Brad um,
9:50 Brad quad code is the model that a lot of people are using the framework that a lot of people are using. And you can, you know, swap out different versions of the LLM behind it and it still does all of its stuff,
10:02 Brad right?
10:03 Brad Cursor as an IDE has toggleable different, um, models.
10:09 Brad You can pick different models and the cursor team basically released their own model so that they didn't have to, you know, rent tokens from.
10:16 Brad from the
10:16 Scott Amen.
10:16 Brad frontier models and you know i i would tell you i've told you before that i think that their composer model is um absolutely performing just as well and it's cheaper than than the other frontier models i rarely ever ask cursor to use one of the frontier models because
10:32 Brad composers just fine right
10:34 Scott Mm
10:34 Brad um
10:34 Scott -hmm, mm
10:35 Brad so there's the ip of the model and there's this big huge gargantuan battle between the the frontier you know models to try to make a better model but maybe the framework is what matters and
10:48 Scott -hmm.
10:49 Brad so then the question is okay well do you he had essentially built his own tooling because he started on it like you know 12 months ago
11:00 Brad Where open CLAW didn't exist yet,
11:03 Brad CLAW code didn't quite have the same tooling to it.
11:07 Brad OpenClaw and ClaudeCode and his own little tooling thing,
11:12 Brad they're all also rapidly approaching commoditization, right?
11:16 Brad So the models themselves are essentially approaching commoditization.
11:20 Brad And his thesis,
11:22 Brad which I, you know, in chatting with him about it, I was like,
11:24 Brad oh yeah,
11:24 Brad that makes sense, is that the framework around the models,
11:28 Brad those things are also
11:32 Brad approaching commoditization right
11:34 Scott Mm-hmm.
11:34 Brad so especially now that open you know there's like there's there's a dozen different variants of open claw and and a bunch of them are open source and they're they're all kind of competing for that framework space um and so his thesis is like hey the thing that matters is probably more than anything is
11:55 Brad As he thinks about being an AI architect,
11:58 Brad AI engineer is like,
12:00 Brad hey,
12:00 Brad it really matters like how do you put all these pieces together like the model that the models themselves are going to be interchangeable.
12:07 Brad Maybe you're going to choose different ones based on the type of task.
12:10 Brad The frameworks are going to be commoditized. So what really matters is like your ability to sort of like.
12:17 Brad frame up, piece these things together and create opinionated skills about how to do the thing that you want done in a professional manner.
12:28 Scott Yeah, yeah,
12:30 Scott I don't have anything to add to that.
12:31 Scott I agree.
12:32 Brad So then the other thread that I think you can pull on is,
12:35 Brad okay,
12:35 Brad what does this mean for engineering teams?
12:38 Brad Like, how should they be using
12:40 Brad Quad code? How should they be using frameworks?
12:43 Brad How should they be using models?
12:45 Brad And how should they be integrating whatever they're doing into their software development lifecycle?
12:51 Scott Well, this is, I feel like you've teed up the topic on my mind and I don't have any answer for it.
12:57 Scott Maybe our 39 listeners can leave a comment in the comments wherever this is posted.
13:01 Scott But I think that's actually where we're at right now, which is we've got all sorts of people experimenting, doing different things. And I can't remember if I've talked about this in the podcast.
13:11 Scott I went a couple of weeks ago to a two-day workshop led by the AI Tinkerers Group on setting up a software factory.
13:19 Brad Yeah.
13:19 Scott And we worked through that for a couple of days.
13:22 Scott And what I took from that was those were the two days that we kind of entered a new era.
13:30 Scott We went from when I did that,
13:32 Scott I went from, oh, OK,
13:34 Scott we've got a bunch of people. There's a whole spectrum of how people on our team are doing it. Some people are really far ahead. Some people are resistant.
13:42 Scott You know, most people are in the middle.
13:44 Scott And what I took from that was, oh,
13:48 Scott okay,
13:48 Scott we need to kind of set up a, I'm going to try not to use loaded words, we need to develop a common approach amongst ourselves so that the roughly 100 or so people on this team are.
14:01 Scott doing it similarly because right now it's all over the map and it's not all over the map in a bad way because all that experimentation is good it even occurred to me in a conversation I had with some of my team about starting to standardize this there wasn't even really agreement
14:23 Scott like what should be a skill what should we what should do how should the parts all fit together basically you know forget forget what i said what should be a skill just how should the parts all fit together and it's not that they were it wasn't even it wasn't like people who were butting heads and disagreeing it was more like huh should we do it that way or
14:39 Brad Yeah.
14:39 Scott should we do it this way you know it's all it's all very open and greenfield
14:43 Scott And so what we're kind of thinking of, this is the idea we're toying with.
14:46 Scott You know the notion of long-term support with software,
14:49 Scott right?
14:50 Scott Companies will have, you know, version one,
14:52 Scott version two, version three, and they'll have some version where they give you long-term support and they'll tell you we're going to support this for a while.
14:57 Scott We're kind of thinking of like, maybe we do this long-term.
15:01 Scott support approach where it's like well this is how we're going to do it and we're going to do it that way for say three months and then we'll we'll learn during that time and then we'll revise it and the reason i bring up the long-term support is one of the challenges we're having that's moving so fast right
15:15 Brad Yeah.
15:15 Scott people are saying we you just you just suggested that i do it this other way like last week and now you're telling me to do it this way i'm just trying to get my job done i'm just trying to get some software built here like i can't be changing my whole development
15:26 Scott world every day.
15:27 Scott And so you can kind of have people experimenting,
15:29 Scott but then you sort of roll that into your LTS.
15:34 Scott let's say quarterly or monthly i don't know if and this fits back with that the aldc that we talked about many episodes ago it's like that's kind of it's kind of what we need to do is build our agent development life cycle like how their ai development life cycle what are all the parts and how do they fit together i think this is the name of the game right now
15:50 Brad Yeah, yeah, I think so.
15:51 Brad And I think so.
15:52 Brad So that really resonates with me and I have a couple I have a couple of reactions to it. The first reaction I'm going to say is.
16:01 Brad you know this developer that i was talking about um one of the things he said which i was like this is this was the moment where i was kind of like i didn't feel like my i was necessarily my mind was blown but this was where i was kind of like oh yeah yeah old guy here is getting left behind i gotta get i gotta up my game was he basically said look man if i do something consistently multiple times a day i need to give that to my agent like if i'm a if i'm doing the same thing multiple times a day
16:29 Brad So that's a me teach the agent how to do it so it does it.
16:32 Scott Mm-hmm.
16:33 Brad I teach the agent how to do it the way I do it so that it does it my way.
16:37 Scott Yeah.
16:37 Brad And so this was a simple example, and this was the reason that like the beginning of the thread that I pulled on was,
16:45 Brad hey, wow,
16:46 Brad you're doing code reviews like we all do code reviews multiple times a day,
16:50 Brad right?
16:51 Brad This team is cranking out 3,000 lines of code per developer per day.
16:54 Brad like there's you know every single person has got to you know go do these code reviews right and so so he was just like i like even when i'm using agents to do things like do code reviews i was still like i am still like saying oh now it's time for you to go look do look at this pr hey agent go look at this pr do a code review for it
17:18 Brad And what he's gotten past is he doesn't even have to tell the agent to go do that.
17:23 Scott Right, right.
17:24 Brad He has the automatic queue getting created.
17:27 Brad But because it's paying attention to the stuff that matters to him, it's sucking it in as a task for the harness to do.
17:36 Brad And then it knows and he's told it, you are allowed to do these things without my supervision.
17:42 Brad And then these are the gates you come to where I will supervise you.
17:46 Scott Mm
17:46 Brad And
17:47 Scott -hmm.
17:47 Brad this is the artifact I want you to give me.
17:49 Brad And then I will tell you to proceed or not.
17:51 Scott So that, and you mentioned earlier, there's another person on that team who was doing something similar and had devised it in their own way.
17:59 Brad And they had devised it in their own way using a different framework using it. And I effectively am doing it my own way using a slightly different tool, but a more, even though again,
18:09 Brad I'm like this.
18:10 Brad And then there was another developer in the conversation,
18:12 Brad he was like,
18:12 Brad Oh yeah,
18:13 Brad I'm kind of doing it Brad's way like I'm also two manual and
18:16 Brad And they were both like,
18:17 Brad oh yeah,
18:18 Brad we need to figure out how to do it more automated so we can speed up even more.
18:23 Scott Well, let me ask you a question.
18:24 Scott My point of view about talking about the whole ALDC and LTS and stuff a minute ago was that thinking,
18:33 Scott OK,
18:33 Scott it's not good that everybody's doing it their own way because some people are doing it better than others.
18:42 Scott And some people are the fact and because of the rate of change,
18:47 Scott some people aren't even really able to keep up.
18:49 Scott So I guess I want you I want to pressure test an assumption I have.
18:52 Brad Yeah.
18:53 Scott When I hear you say you've got three or four people on your team all doing this differently,
18:56 Scott I'm saying to myself,
18:57 Scott oh,
18:58 Scott that's I would rather like that group of people come along and have one skill they all share across their team.
19:07 Scott But I can also see.
19:08 Scott some benefit maybe in the diversity so like I'm just curious to know like maybe I'm heading down the wrong path of trying to structure this and make it consistent
19:16 Brad I will.
19:17 Brad OK,
19:18 Brad that's that was kind of one of the other threads I wanted to pull on because I knew you wanted to go down that that direction a little bit.
19:23 Brad And I'm going to I'm going to preface this by saying.
19:27 Brad I bet you this is a it depends answer.
19:30 Scott Of course it depends.
19:32 Scott Where is Kevin when we need him?
19:34 Brad Where's Kevin when we need him?
19:35 Brad So I think this one's going to be an it depends. First of all, the diversity and the experimentation and the willingness to let smart people innovate,
19:49 Brad especially in the rapid.
19:53 Brad rapidly changing times that we're in feels
19:55 Scott Mm-hmm.
19:56 Brad like it's got to be the right call like oh
19:59 Brad Because standardization is optimization and premature optimization is the is the death of so much software.
20:07 Scott Well, that's why I'm trying to blend it together here.
20:09 Scott I'm trying to say, look,
20:10 Scott we're going to have a standard approach that everybody can use.
20:14 Scott And anybody can go innovate on their own. You don't have to use the standard approach because there's people where I can't stop them.
20:20 Scott They're
20:20 Brad Yeah.
20:20 Scott going to they're going to they're
20:21 Brad Exactly.
20:22 Scott going to they're sending me text messages on, you know, 8 p.m. on Saturday night telling me the cool stuff they're doing.
20:27 Brad Yeah.
20:27 Scott Right. So, like, I can't stop them.
20:29 Scott So it's like, well,
20:30 Scott go do it. Great.
20:32 Scott And then let's learn what's really working and then let's put that into the standard version because not everybody on this team is going to be as excited about you and use their Saturday night.
20:40 Scott night to figure out how to make some cool thing work yeah
20:42 Brad Well, so first of all, it sounds like we're aligned on the idea that like having a freedom to innovate, you know, is the American way and let's let people do that. Okay,
20:55 Brad so that we're
20:55 Scott capitalism for the victory
20:56 Brad aligned on that.
20:58 Brad I like I get the so the other part about your comment that resonates for me, though,
21:03 Brad is that
21:05 Brad If you are feeling stuck,
21:07 Brad if you are feeling like you're being left behind,
21:10 Brad then man,
21:13 Brad that sure would help if someone would give you the recipe to
21:16 Scott Yeah.
21:16 Brad tell you how to set it up to not be stuck.
21:19 Scott Mm hmm.
21:19 Scott Absolutely.
21:19 Brad So that also makes sense and resonates for me.
21:24 Brad But then the other tension there is, man,
21:26 Brad this stuff is changing so fast. Even if you document it, it might be obsolete.
21:30 Brad It certainly will be obsolete in three months. It might be obsolete in
21:35 Scott Yeah.
21:35 Brad six weeks.
21:35 Scott I'm not even talking about documenting.
21:37 Scott I'm just like, let's talk about your code reviewer skill.
21:40 Scott It's like, OK,
21:41 Scott gang, here's here's our common code reviewer skill.
21:44 Scott We've all put our best thoughts on how to make it.
21:45 Scott make a really good code reviewer skill here,
21:47 Scott right?
21:48 Brad Yeah, I think there's value in capturing that.
21:51 Brad By the way, I will say the other thing about this is,
21:55 Brad and I keep saying this to people,
21:56 Brad I literally just said it to a team that I was doing some AI coaching with earlier today where they were asking me like,
22:04 Brad well,
22:04 Brad what should we do here?
22:06 Brad And I basically said,
22:07 Brad forget about AI for a second.
22:09 Brad Like forget that AI is even in this conversation.
22:13 Brad what would you do normally like what would you do as an engineering team i think the question was like i think the question was like oh um uh we're going to use spec kit sdd to produce uh the do we're going to do spec you know spec kit specify and it produces this like essentially prd and then we run clarify on it it's it's refined that it's really good and they were like at this point should we
22:39 Brad have other developers review it or should we go to the implement should go to the plan phase before and then give them both artifacts right and I was just like the answer to that question has nothing to do with AI
22:51 Scott yeah that's right someone at our company this week or last week it must have been last week i don't recall gave a presentation about this exact same thing he now we got a lot of people using spec kit i don't know if folks on the
23:06 Scott audience or know what SpecKit is, it's sort of an STD framework.
23:10 Scott it helps you develop your spec and your implementation but it's kind of basically plan mode you know in cursor and he just mapped each of the things to each of the steps in our sdlc and he's like gang this is no different than what we're already doing you're just getting the ai to do it for you and you're double checking the work like
23:28 Brad Yeah
23:28 Scott and in fact it's doing it better because lots of times we'd skip a bunch of these steps because we would think it's easy or because we were lazy or because someone was impatient
23:36 Scott and this because it's so fast they don't have to skip them anymore
23:39 Brad Yeah.
23:39 Scott so to me that really resonated and then i'm building on that with what you just said which is like forget about the ai like it's just our sdlc what would you do right well i wouldn't send it
23:51 Scott thing out there without having so the ship some code without having it reviewed okay who's going to review it you or this robot over here
23:58 Brad Right.
23:59 Brad Exactly.
23:59 Brad And so what I think is interesting about that to apply to the question of standardization and or like the, you know, kind of like the guidance that you want to give a team like, hey,
24:12 Brad like, let's get let's start doing it this way.
24:14 Brad The way the what is the way you're going to do it? Right.
24:19 Brad That is definitely,
24:20 Scott that's the thing
24:21 Brad that's
24:21 Scott yeah
24:22 Brad like the way you would have done it independent of the AI.
24:25 Scott but there's this is this is where it gets complicated because you kind of have to agree but also you have to figure out like how do we map these all the tools i'm just going to broadly say all the tools out there right all this stuff out there we kind of got to figure out how to map our stuff onto the
24:42 Scott Things out of the tools that are out there those tools are changing really rapidly.
24:47 Brad Right.
24:47 Scott So that's where it gets tricky
24:49 Brad I agree with that statement. So but but again, like if we like let's just pause,
24:54 Brad pause the film for a second here.
24:56 Brad Right.
24:57 Brad We're going to pause the film and we're going to say,
24:58 Brad OK,
24:59 Brad what happened here?
24:59 Brad What was what was Brad like, you know, impressed by?
25:03 Brad And the answer to that question is this is a team who has a very strong SDLC.
25:10 Brad it's as long as i've worked with them i've worked with them for like two and a half years now they've always had a so well before the early ai stuff like they were already they had a strong sdlc it was like we get a ticket from product we think about that we write up our plan on that
25:29 Brad we start to code it when we're done coding it we put up a pull request that pull request gets reviewed that you know that it already went through all the quality gates of automated testing blah blah and you know when we we merge it then it's in our staging environment we run our uat you know blah blah it's like that was just their sdlc the thing and and along the way
25:56 Brad AI has gotten better and better.
25:57 Brad And each of these little parts of it, they've been adding a little AI spice to each little part.
26:05 Brad Oh,
26:05 Scott Mm-hmm.
26:05 Brad the agents have gotten good enough that if I just give them a good quality ticket,
26:10 Brad they will write a great implementation plan.
26:14 Brad I trust that.
26:16 Brad Let's do it, right?
26:17 Brad Before that, there was the if I write the implementation plan.
26:21 Brad And I give it to the agent. It does a good job of coding it.
26:24 Scott Mm
26:24 Brad But
26:24 Scott hmm.
26:24 Brad now they went to, okay,
26:25 Brad you're not just good at coding it. You're actually good at writing the implementation plan.
26:28 Brad I just need to read that.
26:29 Brad Oh,
26:30 Brad and now it's like,
26:31 Brad oh, wait, they're actually really good at writing the specification for me better than I am at writing the specification,
26:37 Brad right?
26:37 Brad And now I'm just in it. And they're really good at doing code reviews. And so this team has already been putting in the agents at each of these places.
26:47 Scott Just use the existing SDLC as the skeleton and start to add the agents to it.
26:52 Brad And then what the next step, after you've done that, where manually at each place in the SDLC,
26:59 Brad a human being sits down and says,
27:01 Brad hey, Mr.
27:02 Brad Agent,
27:02 Brad now it's your time to write the spec.
27:04 Brad Great,
27:05 Brad write the spec.
27:06 Brad Come back 10 minutes later.
27:08 Brad Hey,
27:09 Brad Agent,
27:09 Brad I've read your spec.
27:10 Brad I like it. Now it's time for you to write the implementation plan.
27:13 Brad Right?
27:14 Brad Come back five minutes later,
27:16 Brad read the implementation plan.
27:17 Brad Oh, hey, good job,
27:18 Brad agent.
27:19 Brad I trust that implementation plan.
27:22 Brad Now it's your time to write the code,
27:24 Brad dot, dot, dot, dot, dot.
27:26 Brad What I'm saying is like.
27:28 Brad This is the part that I feel like I went viral a couple of weeks ago saying like, no one's doing this overnight, right?
27:36 Brad Now I'm like,
27:38 Brad oh, wait a second,
27:39 Brad this engineer that I actually really trust is doing it overnight. And he has just figured out that some of those steps in his process can now be automated so long as he has.
27:54 Brad Based on the framework and in his mind of what the definition of quality is,
28:00 Brad he's able to get the checkpoints that matter.
28:05 Scott Yeah. So it's really the quality gates are the key.
28:09 Brad Yeah, and he even said that. He was like,
28:11 Brad look,
28:11 Brad I've gotten to the point now where I can be really clear about what the definition of a correct looking acceptance criteria is for each of these steps along the way.
28:27 Scott So we had an engineer at our weekly engineering show and tell show something off sort of similar this week.
28:33 Scott I just thought of it as you mentioned this.
28:35 Scott He was doing something slightly different.
28:38 Scott Every time we have a test fail, an automated test failure,
28:43 Scott he was triggering something.
28:48 Scott you can imagine the steps basically having the AI figure out the solution and create a PR and then he was looking up on code owners who owned that area and just assigning it to that team say I think I have you know effectively saying this could be the fix to the problem and
29:02 Brad Right.
29:03 Scott he said he's just testing it locally right now like just within himself he isn't assigning them to the teams yet he's seeing if he can get it there
29:11 Scott which is pretty cool and someone else in our meeting was like could we just start feeding that same thing bugs bug tickets it's
29:18 Brad Yeah.
29:18 Scott like computer errors from data dog he's like yeah i don't see why not
29:21 Brad Well,
29:22 Brad okay,
29:22 Brad cool.
29:23 Brad I love that story,
29:24 Brad and I'm going to double down on that. Again,
29:29 Brad this is super...
29:31 Brad compelling story from this engineer that i trust one of the things he mentioned is yeah i have an overnight one of my jobs that i very clearly specified is the agents are to get 100 test coverage for
29:45 Scott Oh yeah,
29:46 Scott for sure.
29:47 Brad but for old code that doesn't have test coverage so
29:51 Scott Oh, interesting.
29:52 Brad he has agents that are just
29:56 Brad preemptively going to find places in the code that are missing test coverage well
30:03 Scott I wonder how it does that,
30:04 Scott actually.
30:05 Scott Does it go execute the tests and then do some sort
30:08 Brad you
30:08 Scott of coverage
30:08 Brad yeah
30:08 Scott reporting?
30:09 Brad i mean your your coverage your coverage report at least you
30:11 Scott It's
30:11 Brad know it
30:12 Scott giving it the coverage report.
30:13 Brad gives you the coverage report and
30:14 Scott Got
30:14 Brad it tells
30:14 Scott it.
30:14 Brad you tells you the functions and the lines and the branches that haven't been covered and so he's just like look i like
30:23 Brad this is i mean this is a good example where it's like it's pretty hard to go wrong on writing a test i mean you can write tests that don't actually test the thing but if you run every line of code right in the test then you definitely got more coverage than if you didn't run every line of code and so that's like a goal seeking like that's a very straightforward thing for a a goal seeking framework to be like
30:49 Brad Keep going until
30:50 Scott Yeah.
30:51 Brad you get 100% coverage.
30:53 Scott Yeah.
30:53 Scott Yeah, that's right.
30:56 Scott Brad, it's crazy.
30:59 Scott I am now squarely in the camp that in five years,
31:02 Scott we're not even going to recognize this industry.
31:04 Scott I mean, it's just going to be so different.
31:06 Scott Like my career, which has lasted roughly the last 30 years, a little bit more.
31:10 Scott you know has been the internet era like i didn't i never worked on non-internet i didn't work on any client server software that was that was pre i came into this with the start of the internet 1995
31:20 Scott is when i got my first job as a doing
31:23 Brad Yeah.
31:23 Scott programming it's been you know the same
31:27 Scott So what's changed?
31:29 Scott Cloud, that's the main thing that changed.
31:31 Scott Cloud and mobile,
31:32 Scott those are the two big things that changed over my time.
31:34 Scott But it's all been kind of the same refining of the same thing.
31:39 Brad Yeah.
31:39 Scott Yeah, you know, and
31:40 Brad Yeah,
31:40 Scott to me, this is like,
31:41 Scott oh,
31:41 Scott this is just now we on one hand, it's like, oh, this is changing everything. But then we just talked about it's the same as DLC.
31:49 Brad that's the thing that I'm holding on to. And I also think that this is a great example where the engineer that I'm talking about here,
31:58 Brad he's one of the most successful,
32:01 Brad compelling users of this stuff, and he's a great engineer.
32:05 Brad And he has been.
32:05 Brad has been and he's pretty young actually and he's been kind of like oh man what's going to happen to the industry right but what i see him doing is like exactly what i would tell every engineer to do like you just gotta get great at giving your giving your opinion to the to the tooling and your opinion still matters your taste still matters
32:33 Scott Just leave it at that.
32:34 Brad Your taste still matters.
32:36 Scott Your taste still matters.
32:38 Brad There you go.
32:39 Scott All right.
32:40 Brad All right.
32:40 Scott Brad, it's been nice catching up this week.
32:41 Brad Great catching up.
32:43 Brad What do we say?
32:45 Scott Smash rate and review.
32:46 Brad Smash that subscribe button.
32:48 Brad All right, see you guys next week.
32:49 Scott All right, bye.
32:50 Brad This episode of the How Many CTOs Does It Take podcast was produced and edited by Safran Heptekau.
32:56 Brad Our theme song was AI generated by Brat. Please follow us on all the social media platforms at HowManyCTOsPod. You can find us on YouTube,
33:07 Brad LinkedIn,
33:07 Brad Instagram,
33:08 Brad Facebook,
33:08 Brad Threads, X,
33:09 Brad TikTok,
33:10 Brad and Patreon.
33:11 Brad or at our website at HowManyCTOsPod.com.
33:15 Brad And remember to rate, review,
33:16 Brad and subscribe on the podcast platform of your choice.
33:19 Brad Thanks for listening.