Method to the Madness: A CTO Framework for Systemic AI Adoption

0:00 Scott: You need a little bit of method to the madness,

0:01 where at least you say we're going to do this way, but that one's still going to be people's choice.

0:22 Brad: Welcome everybody to another

0:24 episode of the how many CTOs does it take podcast my name is Brad Hefta-Gaub and this is my trusty co-host nice

0:32 Scott: Scott Paul Brad,

0:33 nice to see you.

0:35 Brad: to see you I heard you went paddle boarding this morning

0:39 Scott: I did. I did indeed go paddleboarding this morning. I've been having a fun week, like I told you,

0:44 Brad: because Yeah, I love

0:45 Scott: it. the company that I work for right now gives,

0:48 I don't say gives isn't the right word, but closes for a week in the summer.

0:53 And as a result, everybody takes the week off and comes back. We call it recharge week,

1:00 in fact, and comes back recharged. You know, when you take a week off and everyone else is working, you just come back to more work.

1:05 work right

1:06 Brad: Right.

1:06 Scott: so and i've been a lot like i told you also i've been watching a lot of world cup

1:10 Brad: I mean, this is we don't have to talk about it because this is going to come out a month later. So everybody will know what the answer to the question is.

1:16 But it's exciting to watch the World Cup. It's exciting that it's happening here.

1:20 A bunch of it's happening here in Seattle.

1:22 And so, yeah,

1:23 that's been that's been a fun little fun little experience for all of us.

1:27 Scott: yesterday i went to pioneer square

1:31 Where for those of you who don't live in Seattle,

1:33 that's like our old timey neighborhood with lots of bars and art galleries and stuff like that.

1:38 And I watched the USA soccer game.

1:43 It was super fun just to be out with the people. I'm a man of the people.

1:48 Brad: You're a man of the people and the paddleboard.

1:50 Cool.

1:50 Well, you said you had some important CTO conversations to have, and that's why we're recording this.

1:56 So what is...

1:57 Scott: not important, but let me give a little bit of context, two bits of context for this.

2:02 The first bit of context is as CTO's job is to sort of

2:09 Set the story,

2:10 frame the story, frame the narrative so that people kind of know what they're doing.

2:15 Did you ever read the book Sapiens by Yuval Harari?

2:22 Brad: I have not read that book.

2:24 Scott: I

2:24 Brad: I think you've mentioned it in the past. It's definitely a book title that's familiar,

2:28 so I should probably

2:29 Scott: strong recommend strong recommend easy read if you'd like that kind of stuff if you don't like it whatever so he's you've all Harari is a historian and he wrote this book called sapiens

2:42 You know, you and I, we are Homo sapiens. We

2:44 Brad: Mm-hmm.

2:44 Scott: are from the, I don't remember the exact, like we're from the genus Homo and the species sapiens, right?

2:50 And there were other species in our genus. There was Homo erectus and Homo, whatever the Neanderthalus and all these different,

3:00 but somehow we survived,

3:02 right? We Homo sapiens.

3:06 And he writes about this and he says everyone thinks it's because of the agricultural revolution and that was certainly real but he said there was a revolution before that which he referred to as the cognitive revolution and he said that's what really separated us from the other animals because at that point we were just animals like everybody else we're still animals but like we were just equal and we're not at the top of the food chain at that time.

3:31 Brad: Yeah.

3:32 Scott: And the cognitive revolution fundamentally was our ability to believe in things that didn't exist.

3:41 Brad: Right.

3:42 Scott: So an example he gives is the country of France.

3:46 France doesn't exist,

3:47 right?

3:48 I mean, it's drawn on a map. You can see over my shoulder, there's an atlas,

3:52 right? But like, you know, there's no such thing as France in the world. But...

3:58 When people believe in this idea of France, what are those three words, you know, fraternity,

4:02 egality,

4:02 I can't remember the last word of their moderate.

4:05 When people believe in that idea of France, believe in the idea of the borders of France,

4:09 then they do things like fight for France, right, all these kinds of things.

4:13 Another example from the more modern times,

4:15 a corporation isn't a real thing.

4:18 It doesn't exist in nature. It doesn't exist anywhere.

4:21 It's just a figment of our mind. We made up this idea of a corporation,

4:24 which

4:24 Brad: Right.

4:25 Scott: is this entity,

4:26 right?

4:26 So anyway,

4:27 his argument about Homo sapiens is we developed the ability to believe,

4:32 so we developed the ability to believe,

4:34 to have this cognitive belief in things that didn't exist,

4:37 which allowed us to organize in bigger and bigger groups of people and thus accomplish.

4:43 More complex things than our competitors could accomplish.

4:46 Sure. For instance,

4:47 if we were going to go fight against another.

4:49 species, whether it was a Homo species or some other, we could organize bigger groups of people to defend ourselves.

4:55 Okay, so what's this have to do with being a CTO?

4:56 Well, part of our job as a CTO is to tell the story to frame the story so that our teams can believe in some idea that isn't really real, that doesn't

5:05 Brad: really Right.

5:05 Scott: exist,

5:05 but they can understand some concepts that they can organize around it and then go accomplish something.

5:10 So that's the first bit of context. The second bit of context is I went to a baseball game earlier this week.

5:14 we and

5:16 Brad: You

5:16 Scott: you

5:16 Brad: got to believe in balls and strikes.

5:18 Scott: got to believe in balls and strikes and um well no that's actually a real thing like the ball does or doesn't go over the plate that's actually a actual thing that happens so anyway i'm sitting with my buddy who took me to the game and

5:32 He is not in the tech business and he's in real estate and he just says, what are you guys doing with AI these days?

5:39 And so I had to think kind of quick on my feet,

5:42 like, how do I tell him what I'm doing with AI in a way that he can understand?

5:46 Because if I got into the nuts and bolts of like how these agents work and stuff like that, I would have bored him.

5:52 And I said, well,

5:54 you know, we're kind of trying to do three things.

5:58 Brad: Okay

5:58 Scott: And this was and I just did this from the off the cuff.

6:01 I didn't plan it.

6:03 But the fact that I had to explain it in simple terms helped me do it off the cuff and explain it in a way that all of a sudden I realized like,

6:10 oh,

6:10 this is actually the story that I should be telling my team.

6:14 And not just me,

6:16 I think any CTO at a

6:19 reasonably sized company needs to tell a team so we're trying to do three things let me see if i can remember it off the top of my head i actually wrote it down in my phone so i should bring up my notes here because as soon as i said it to him we're at the baseball game and i was like oh i gotta write that down that was that was i was just smart oh

6:33 Brad: It's also very AI of you to come up with three things, right?

6:36 Like that's another one of the AI writing tips.

6:41 Scott: is that right the first thing is we're trying to do is figure out how to get

6:45 get AI to do complex engineering,

6:47 right?

6:48 So we've talked about a lot about the shift from vibe coding to whether it's spec driven development or whatever it is, as an industry, as an industry of professional software engineers,

6:57 we're trying to figure out how do we get this to build real stuff for us, not just,

7:02 you know, toy projects.

7:03 And so we've, that's like one, one work stream we're working on, one track we're working on.

7:08 The second thing we're working on, and I think you and I have talked about this,

7:12 is we're also trying to figure out how to fully automate this stuff.

7:15 You've got,

7:16 we've talked about your.

7:17 crew of robots that you got working for you.

7:20 So we got different work streams.

7:22 People are trying to figure out how do we totally make this thing autonomous or as close to autonomous as we can for either simple work streams or complex work streams. We start with simple ones and we work our way up.

7:33 And then the third piece that I think we're working on is how do we scale it?

7:37 Meaning how do we get

7:39 lots of people in an organization to be using this and that's where sort of standards and systems are coming into play i said this very succinctly right to you but i i kind of just said it off the cuff to my friend as i was trying to explain to him well this is what we're doing with ai because if i explained to him well i got an engineer who's trying to do this blah blah he would have just his eyes would have glassed over he wouldn't know what i was talking

8:01 Brad: Right.

8:01 Scott: about but he thought that was very interesting so i want to know what you think of that

8:05 Is that it? Is that what we're all doing right now is kind of going on those three work streams?

8:10 They're kind of intertwined. They're not as nice and clean and tidy as I just described them. But fundamentally,

8:16 it seems like that's what we're doing.

8:19 Brad: Yeah, let me see if I can play that back to see if I understood what you're saying.

8:23 So the first one is just like.

8:25 more and more like investing in the engineers using AI to do complex engineering tasks like get them get the engineers educated get the tooling set up correctly and just kind of like leaning into if I would have had an engineer do it have an engineer do it with with AI to go bigger faster better

8:46 That's

8:46 Scott: Yeah,

8:47 Brad: item one,

8:47 Scott: let's even go further back to when Microsoft introduced IntelliSense, right?

8:54 That was like a very crude form of AI,

8:57 right? To try and predict what the end of this function,

9:00 autocomplete should be,

9:01 right?

9:02 And so it's just been an evolution of their doing more and more and more complex engineering tasks with some computer assistance.

9:08 Brad: Right, but just to be clear.

9:11 at most organizations are not building those complex systems they're they're just using the ones that have been built

9:18 Scott: Correct.

9:19 Trying to build other complex systems with those,

9:21 Brad: with

9:21 Scott: right? We're trying to build

9:22 Brad: those exactly

9:22 Scott: a mobile

9:22 Brad: so yeah

9:23 Scott: app or a soft banking platform or whatever you're trying to build, right?

9:28 Brad: but there is like yeah but to your point like you're doing that and you're using ai that way right like you're using ai you're trying to get your team to use ai more you're trying to

9:37 and it's interesting to me i'll just pause on that one for a second the whole like token maxing thing where like people were panicked about their token bills and stuff like that i think that's an interesting piece that's got to be in that category it's probably a combination of that category one and your second one which is category two where um where like

10:00 The leadership is trying to tell teams like I want you using AI and we're going to incentivize you to use AI and we're going to do a leaderboard on who's used the most tokens.

10:11 So I think that's pretty consistently people are in that space,

10:15 not necessarily trying to do bad metrics like leaderboards on tokens, but get our engineers to use the best.

10:24 most advanced tools to write the software that is building that we're building okay

10:29 Scott: Well, that's not the first point, really.

10:31 Brad: so maybe I'm

10:32 Scott: I

10:32 Brad: missing

10:32 Scott: mean, I think that's a thing we're doing.

10:34 Okay,

10:35 so the first things to say I could do and, you know,

10:37 OpenAI really released their first chat to people.

10:42 It could help you write a function or maybe it could make a web page, right?

10:45 Brad: Yeah.

10:46 Scott: But it couldn't build a complex trading system for a bank,

10:49 right?

10:49 Brad: Yeah.

10:50 Or even like a complete system,

10:52 like a whole feature,

10:53 like brand

10:54 Scott: Right,

10:54 Brad: new feature called,

10:55 I want downloadable reports from this dashboard.

11:00 Scott: right.

11:01 And so I feel like part of what we're doing is trying to push the AI.

11:05 How can we build increasingly complex things?

11:08 it's

11:08 Brad: Hmm.

11:09 Scott: nice that you can help me autocomplete a function but i want you to build the whole feature and then i want to go from building a whole feature to building a whole system right

11:16 Brad: Yes.

11:16 Scott: and so we're that's that's me that's the first work stream that we're working on is how do we get this to more effectively build more and more increasingly complex things

11:25 Brad: Okay, cool.

11:26 I agree with that statement. I think that's what I'm seeing within all my teams. That's certainly what I'm seeing.

11:30 You know, we talked a couple episodes back about.

11:34 the um my beaker stack um template

11:37 Scott: Yeah.

11:37 Brad: and my my uh skein uh ai memory backplane and my my desire to build sass apps every six weeks by the time this one comes out there'll be another sass app out i actually have it almost ready to release um and so um yeah and i and basically i haven't yet just turned to the ai and said

11:58 here's the spec for the whole app build the whole thing i haven't done that yet but i am at the place where i'm like literally like big feature by big feature like this that the app i'm i'm going to release shortly is essentially like a um

12:13 a retirement calculator it's literally a retirement calculator app that you can kind of plug in your your current you know whatever your savings it's like that and it runs monte carlo and it runs back history back testing and stuff like that it's got reports it's got all these different like compare and contrast like oh what if i delay my retirement by a year what happens you know all this stuff and literally in that case i'm i'm just given the ai like big chunky features one at a time

12:39 And so I didn't have it build the whole thing, but I have it build pretty big chunky features in

12:43 Scott: Mm

12:43 Brad: one

12:43 Scott: -hmm.

12:44 Brad: shot, right?

12:45 So yeah,

12:45 that's a great example of like moving to more and more complex tasks.

12:49 And maybe by my third SaaS app, I'll be brave enough to just say,

12:53 here's my framework,

12:55 here's my well-written spec,

12:56 build me the entire thing and let me see what it looks like. And there's certainly people out there that are kind of pushing those boundaries,

13:02 right?

13:03 So yeah,

13:04 I totally agree with that first bucket.

13:05 The second bucket I heard you talk about.

13:07 is the do it in the dark factory do it automatically do it autonomously as opposed to like you know lots of human steering did i get that one right so the

13:18 Scott: That's the second one,

13:19 Brad: the

13:19 Scott: kind

13:19 Brad: second

13:19 Scott: of, yeah,

13:19 Brad: bucket is like

13:20 Scott: trying to figure out how to get to a software factory

13:22 Brad: how do you get to an automatic uh in you know dark factory meaning like the humans aren't having to do the steering or they're doing the steering way way way high up you know yeah i agree

13:32 Scott: just checking in.

13:33 From time

13:34 Brad: checking

13:34 Scott: to time on

13:34 Brad: in from time to time to see how it's doing the

13:36 Scott: the human in the loop.

13:37 Brad: human taking the human out of the loop or moving the human to a much much further distant um able to and yeah and for sure lots of experiments are happening in that space you know lots of people are going down that road you know um ai tinkerers joe heitzberg he's his team's like constantly having these you know

13:59 Scott: We got to get Joe on here.

14:00 Brad: yeah we should get joe on yeah for

14:02 Scott: I'll

14:02 Brad: sure yeah

14:02 Scott: call Joe. I'm friends with Joe. I'll call him up.

14:04 Brad: um so yeah he's ai tinkerers are just doing lots and lots of stuff around like more and more automation you know and so yeah totally that's a that's a great bucket and then when i think i heard your third bucket this is what i thought i heard but this might have been confirmation bias on my part the tooling and the framework and the systems and the processes around all that to keep everything

14:28 doing

14:28 Scott: No,

14:28 Brad: quality is that what

14:29 Scott: sort

14:29 Brad: you meant or

14:29 Scott: of.

14:29 Brad: what did you mean by that

14:30 Scott: I

14:30 Brad: i

14:30 Scott: mean this. Within a software engineering organization,

14:34 once you get to two people in your software engineering organization,

14:38 you start to have some sort of standards and systems.

14:41 For instance,

14:42 you might agree,

14:43 well,

14:44 first you might agree that you're going to use source control, and then you might agree which source control you're going to use.

14:50 Brad: feel like this was our second episode we talked about what are the bare minimums of

14:53 So of a software team,

14:55 and I think source control was the first one we came up with.

14:58 Scott: Right.

14:58 So like as soon as you have more than one person,

15:00 you start to.

15:02 put some of those things in place and we have all sorts of standards within our industry that people don't even question anymore like source control nobody comes in you don't hire anybody if you're a company they say why are you guys using source control that's you know so um but you know source control code reviews we have we have a convention that we agree that everybody gets their code reviewed in fact it's built into our

15:28 door source control right because

15:30 Brad: Yeah.

15:30 Scott: we use so there's all these conventions we have for how we do software because we've understood we understand as a industry or as a community or as a team that they help us do better work and so i think what

15:46 I'm really emphasizing with my team right now is we need to be doing that within our team as it relates to AI,

15:56 right?

15:57 So for instance,

15:58 we already have one AI agent code review tool,

16:03 right? We've agreed this is we're going to put it

16:05 Brad: Yeah.

16:05 Scott: into our build pipeline and we've agreed that this is part of it.

16:08 We've got other parts of people using AI that are very inconsistent and not to say some are better or worse than others,

16:16 but I think just like every other aspect of software development,

16:19 it is wise for us to look at what people are experimenting with and make decisions about,

16:25 okay,

16:25 that's good.

16:26 We need to make that universal systemic across our whole.

16:31 What everybody does, we've got 150 engineers,

16:33 what everybody does here,

16:35 they all, I'll use the same example,

16:37 everybody all has their code reviewed by an AI code review agent.

16:41 Brad: Yeah.

16:41 Scott: That's already happening.

16:42 Well, what else is there,

16:43 right?

16:44 I think part of that, part of this work stream that I was talking about is trying to build those standards.

16:51 And I think there's two

16:52 Brad: In

16:52 Scott: reasons.

16:53 Brad: an AI world.

16:55 Scott: Yeah.

16:55 Brad: to how to adjust and build uh okay so in other words let me see if i can play that back then so you're saying like all software engineering teams have some set of processes and standards that they've been following whatever those were and they like they have pretty common shapes and we can like but there's always a little bit of tweaks here and there and you know typically it's whatever you know code review it's you know it's it's you know whatever the rules and the processes that the team follow for the process

17:22 And what I think I hear you saying is AI is changing how we work. So we need to make sure that we're improving and optimizing our whatever our workflows are to be good workflows in the new AI coding world.

17:38 So like the ADLC we talked about a couple months ago.

17:41 Scott: Yes, and I would say...

17:45 Not only good in the AI world,

17:46 but we need to bring consistency. We need to shift from organic adoption to systemic usage,

17:54 right?

17:55 In certain aspects of what we don't organically adopt GitHub.

17:59 Some people don't decide to use GitHub a lot and some people decide to use GitHub a little,

18:03 right?

18:03 We are systemic about how we use GitHub.

18:06 in our organization.

18:07 Brad: You know, it's interesting that you say that,

18:09 and I, a lot of the work I'm doing now is on pretty small tight-knit teams, and so it's by default, like,

18:17 it's already systemic, you know, it's already systematic if, if,

18:21 you know, the team is small,

18:22 they've kind of like organically came with the way they work, and that is effectively the system. But in an organization as large,

18:28 and you and I have worked in much,

18:29 much larger organizations, you're currently at a.

18:31 about a larger organization in that it's 100 plus engineers and so you've got lots of different teams um when we had ted neward on like almost a year ago he talked a lot about kind of like how there is no like best practice there's only the practice that's best for your team um and i'm kind of curious how you're thinking about the how i mean i i it resonates with me the idea that you'd want

18:58 want to have system-wide processes and policies.

19:02 Scott: Mm-hmm.

19:03 Brad: And I also can picture a world where variations amongst teams is not necessarily a bad thing.

19:09 Scott: Okay.

19:10 Yes,

19:11 I'm not saying it should all be totally uniform.

19:13 I'm saying let's figure out what is good, what should be uniform.

19:18 Source control is such a great example because it's not such not a debate.

19:22 Brad: Right.

19:23 Scott: We figured out that should be uniform and we just do it.

19:26 But there's other parts of software engineering that we don't think should be uniform.

19:30 So

19:30 Brad: Yeah,

19:30 Scott: we add to those things.

19:30 So what

19:31 Brad: that's fair enough.

19:31 Scott: are the parts that we should start to make uniform?

19:34 And we've already decided a code review agent is something we should make uniform,

19:38 Brad: Yeah.

19:38 Scott: right?

19:38 Are there other things we should make uniform?

19:40 Brad: Yeah.

19:41 Scott: And there's going to be lots of things that we shouldn't make uniform.

19:44 And we definitely want people pushing ahead because it's changing so fast.

19:49 We definitely want people pushing ahead and experimenting with all the new stuff and like coming back to the home base.

19:54 And I have this idea of like scouts going.

19:56 Just going out and scouting

19:57 Brad: Yeah.

19:57 Scott: out and then coming back and saying, guys, look what I found.

19:59 we should all do this thing and

20:01 Brad: Yeah,

20:01 Scott: then then maybe more people will try it out and if we decide it's good so yeah now we're going to make it our our convention our agreement and our team we all do whatever the thing is and that has two benefits well maybe it's the same side of both it just raises all boats right i feel like what's changed for me in the last year is this organic adoption to systemic usage thing we've been doing organic adoption for the last year all sorts of people are pushing ahead and

20:24 in all sorts of different ways of course there's some people who aren't pushing ahead but you know all sorts of people are pushing ahead bringing ideas back saying we should do it this way we should do it that way and you need a little bit of method to the madness at least you say we're gonna do this way but that one's still going to be people's choice right

20:43 Brad: Yeah.

20:44 God damn it, Scott.

20:45 I thought we were going to have an argument.

20:47 Scott: Ha, ha, ha, ha, ha.

20:48 Brad: But yeah,

20:49 Scott: Yeah, a really interesting one in this, like I'll give you a real example in this one.

20:52 We've probably talked here before about spec-driven development and

20:56 Brad: mm

20:56 Scott: spec kit versus plan mode,

20:59 which are both versions of spec-driven development,

21:02 right?

21:02 And there are camps at our company which are like spec kit for everything.

21:06 And there are camps which are like,

21:09 well,

21:09 spec kit's overkill, spec kit for nothing.

21:12 And there's a camp which is,

21:13 well,

21:13 it depends what you're...

21:14 that's what you're working on right and

21:14 Brad: Right.

21:15 Scott: so it's like okay well maybe we could just set up some guidelines to say under these conditions yolo and under these conditions use

21:22 Brad: Yeah.

21:23 Scott: plan mode you got to use plan mode under these conditions you gotta you gotta use spec kit i don't know what those conditions would be but that's how you start to make a bring some method to the madness

21:33 Brad: I'll throw out another thing along that spectrum that you might want to start thinking about, which is I think this is halfway in between spec.

21:39 spec kit and plan mode which is still write basically write a either have the agent or write yourself but probably have the agent write a markdown file which is the feature brief of the thing that's being built and have that checked into the code

22:01 you know next to the code right this is very much what you're already doing with spec kit

22:05 Scott: A markdown file that's a speech or brief.

22:07 Yeah.

22:07 Brad: yeah

22:07 Scott: Okay.

22:07 Brad: feature

22:08 Scott: That's

22:08 Brad: brief

22:08 Scott: more

22:08 Brad: base

22:08 Scott: than plan mode, but less than spec mode.

22:11 Brad: more than plan mode but less than spec mode and what you do is you write you have the agent write the feature brief you save it in line in the file with the with the code and then you have the agent run plan off of that

22:25 That feature brief.

22:26 Scott: Yeah.

22:26 Brad: It was just like a mini version of SpecKit, but it doesn't have all the extra layers of SpecKit of like the Constitution and the,

22:33 you know, anyway,

22:35 that's what I'm seeing. That's what I'm seeing more and more of my teams kind of that's where they've landed. Right.

22:41 So they they kind of like they tried they were they were already like using plan mode that they were really comfortable with.

22:48 They tried Spec Kit.

22:49 It felt like overkill,

22:51 but the thing that they liked about Spec Kit was saving the, having the feature brief,

22:57 having the kind of detail,

22:58 the spec part.

23:01 saved into the code and then running plan off of that but anyway to your point that's

23:07 Scott: wait

23:07 Brad: a great

23:07 Scott: wait oh before before we go off of that we need a brand name for that middle ground right

23:11 Brad: i think it's just called i don't know what the brand name for it is i don't know i i see more and more teams doing it it's definitely like or like the the industry has organically come up with this as an as a approach

23:23 Scott: and if we could just brand it

23:25 Brad: i

23:25 Scott: then

23:25 Brad: know we try

23:25 Scott: we'll be

23:26 Brad: to call it the adlc we got the

23:28 Got the animals even

23:29 Scott: I'm going to have to come up with a name for that.

23:30 Okay.

23:30 So yeah. So here's another example, another example, two independent examples, but they're the same thing.

23:38 My security team is like,

23:41 hey, I want to make like some sort of security agent that can like review the.

23:49 effectively review the plan they want to review what we you know like the tech spec like the technical design doc like

23:56 Because they're reviewing a lot of those with their own eyes and they're like, well, could we write an agent to do that, right?

24:01 Brad: Especially with Fable.

24:03 Scott: Is Fable back out yet?

24:04 Brad: Fable's back out.

24:05 Yeah.

24:06 Scott: Oh, the architect.

24:07 Brad: I mean, I don't know if by the time this comes out, it might be not out again,

24:10 but yeah,

24:11 they reopened it up yesterday.

24:13 Scott: Architects want to do the same thing.

24:14 Like, hey, often someone comes ask us to review stuff.

24:18 Could we just have an agent review?

24:21 their plans and then okay so then there this begs a question

24:25 If we do that, should that become part of our system like you just always do it? And if you do a thing like you were talking about before where you write a feature brief and then you have plan mode made the technical plan,

24:37 should it always go through the architect agent and the security agent,

24:40 right?

24:40 Brad: Yeah.

24:41 Scott: Those are the kinds of things that I mean when I'm thinking about going from organic to systemic.

24:46 Brad: Yeah.

24:46 Scott: And I don't know what they're going to be and I think, well,

24:50 you know what I think they're going to be, I think they're going to be a lot of the same.

24:52 The same things that we do already that we just find ways that we can sort of automate with agents.

24:57 Brad: i'll tell you so yeah i love this i love the the it's you're scratching right now like there's there's some there's definitely some stuff there and i'll i'll tell you i'll tell a little story about an example from one of my teams that i work with um and this is a team that's like this is like they're definitely like um very much

25:16 ai pilled they're using the agents they're they're you know they're the team that i'm talking about that are kind of more and more checking in feature briefs and then building off of it and they're really really productive they're like 3 000 plus lines of code a day per developer you know again clocks are a bad metric but it's like real business value where they're just like adding feature feature feature feature feature right and they um

25:41 had a moment where all of a sudden they got we all got they all got notices that their co-pilot github co-pilot tokens had reached their max usage on github and they were like what happened right and this team is basically they all spend about they all have essentially a stipend of like 200 bucks a month per developer like that's and they're

26:09 And that's all they've been spending and they're super productive.

26:12 So like nowhere near this token maxing like,

26:15 oh my God, we're spending like our whole,

26:16 you know, we're spending an employee salary worth of tokens,

26:20 right?

26:20 And they don't need it because they're still super fast at that budget.

26:25 But we so the DevOps guy was like digging into the reports on GitHub and we're like, this is weird, like because we thought the only thing that was happening on GitHub was just the code reviews.

26:37 Right.

26:39 And the report came back and there was just one like line where it's like almost all the usage was like the copilot code review model.

26:49 But then there was this one like weird spiky line that was essentially Opus 4.8 and we're like,

26:58 wait a second, like.

27:00 nobody's using github copilot to do coding it like all of us they all have their own you know local agents that do coding and another person ran the same the same it looked like the exact same report like literally the lines were the same color and the lines were the same shape but then if you looked at the if you looked at the legend it was the name of one developer was perfectly matched to the opus for

27:27 is 4.8 and everyone else's name were matched to the other things and we were like what the heck what are you what are you doing and it turned out that he had written a code review his own custom code review skill that he was using and so there that team would always like click co-pilot code review and then they also would as individuals they'd run their own code reviews using their own local

27:54 own local agents but his script was specifically using copilot uh github tokens and was specifically using opus 4.8 now ultimately

28:05 his his co his extra code review was doing good stuff and everyone liked that he was doing it but it was funny because to your point it was an example where he had innovated in a way that the rest of the team hadn't and then we later discovered he was doing it different and because of like a weird spike in our bill now not a crazy spike like uh instead of a 50 budget that we had for copilot we

28:31 We upped it to $200 for Copilot.

28:34 Scott: Well, and what's interesting about that, now it begs the question,

28:37 should we adopt this thing this guy made? Should everybody be using this?

28:40 Brad: Exactly. Exactly. That's the question.

28:42 Scott: And if you think about what has made software so one of the aspects that have made software so effective,

28:50 I think that's the right word to use,

28:52 is our ability to encapsulate and reuse.

28:57 My friend who I just mentioned I went to the baseball game with is in real estate. He's a home builder.

29:02 And every time he builds a house, they start from scratch. They're building bathrooms and wiring and kitchens and all that stuff

29:08 Brad: Yeah.

29:08 Scott: again.

29:09 Whereas we build a code review agent once and a whole bunch of people get to use it over and over and over again. And our whole industry is just layer upon layer upon layer upon layer upon layer upon layer of innovation.

29:22 right

29:22 Brad: Yeah.

29:22 Scott: that where it's truly like standing on the shoulders of giants and giants and giants and giants right and it's what it's something that's really pretty cool and magical about what we do and

29:34 we're in this like hyper phase of like where we have all these decisions to make about that so that's another work stream then you bring up the last one which you added to this pile which is yeah the last work stream we're working on is figuring out how to manage tokens and

29:47 Brad: Yeah.

29:48 Scott: it's really in the water right now in the air i should say in the water i don't know people are really thinking about

29:54 how to manage tokens that's actually a little frustrating to me I feel like if you have the money you shouldn't be thinking about it at all you should be figuring out how to make this stuff valuable and at some point down the road optimize but it feels like a premature optimization to me right now but that's the fourth work stream I think is token management

30:12 Brad: I think that, well, here's my take on token management.

30:15 I think that for.

30:17 teams that were already behaving rationally about their use of AI,

30:21 where they were essentially,

30:24 I guess I'm going to say rationally, I think my teams are doing this rationally, right?

30:29 Like they're getting a lot of productivity out of 200 bucks a month.

30:33 If they need to bump to 500 bucks a month,

30:35 I'm happy with that,

30:36 Scott: Mm

30:36 Brad: right?

30:37 Scott: hmm.

30:37 Brad: Like no big deal.

30:38 You guys are

30:39 Scott: Right.

30:39 Brad: doing great, right?

30:40 I think that where the reason that token spend is so in the water right now or in the air, as you said,

30:48 it's in zeitgeist, is this token maxing thing that went on where a bunch of, I'm going to say.

30:56 not you know nearsighted or my maybe myopic is the right way to describe this leaders decided to incentivize their engineers to to have token leaderboards and anything that you measure is going to turn you know like people are going to optimize it you know so if you tell them like i want you to spend more tokens i'm going to do a leaderboard of tokens you're going to get massive token spend like that's just human nature

31:21 Scott: Every time we talk about token maxing, it reminds me of a story from one of my children who was about seven years old.

31:27 And it's adorable little story. Can I share it?

31:29 Brad: Sure.

31:31 Scott: There used to be a video game called Club Penguin.

31:34 Brad: Yeah.

31:34 Scott: Do you remember that?

31:35 Brad: Yes.

31:37 Scott: OK,

31:37 so Club Penguin was this online game for people who don't know that was probably run by Disney or something.

31:43 It was for kids and it was a little virtual world and you went in there and you played games and did stuff and you could get.

31:48 you could get tokens and then you could go buy stuff from the store and there was no money i mean you paid ten dollars a month for it or whatever but there was no like you couldn't buy coins and stuff like that one of the games that was in there was that you could like play the dj in

32:01 Brad: Hmm.

32:01 Scott: the club like there's a whole little virtual world and there was like a little dance club a discotheque in there and you can play the dj and you got tokens you earn tokens for being the dj

32:13 So one day I noticed my kid fiddling with the timer on the oven and I'm like,

32:18 hey,

32:18 what are you doing?

32:19 And he's like,

32:20 well, I figured out that as long as you're in the DJ booth,

32:24 you earn tokens.

32:26 You just have to be active every at least every 10 minutes.

32:29 So I've been setting the timer for nine minutes and then I just go over and move the mouse.

32:35 just so I can do something else so they're just earning me tokens all day and every time I just hear this phrase token maxing I think of that little story it was just such an adorable little story for like a seven-year-old

32:45 Brad: Eddie

32:45 Scott: boy

32:45 Brad: hacked the system. He figured it out.

32:46 Scott: he figured out that he didn't have to do it and so when I hear the phrase token maxing it's a totally different use of the word token maxing I just think about like adorable little story when my son was like seven years old

32:58 Brad: So I'm with you on let's not like if you're being already if you're already being rational about your your the way you're using the AI agents, it's not that big deal.

33:08 Now, I have two.

33:10 I have two additional things to say about the token problem.

33:13 The first one I'll say is I think where the crazy token spends are happening is this whole concept of looping.

33:20 And this is something that is also in the zeitgeist and you're hearing like the anthropic team is claiming that they're all about doing looping and you hear the OpenQuad team talking about they're all doing looping. And what looping is, is the idea that you give the agent an end goal,

33:35 like an acceptance criteria.

33:36 And then you literally just say,

33:38 keep running until you hit this acceptance criteria.

33:41 Scott: Mm-hmm.

33:41 Brad: And it just does essentially like, you know, brute force implementation of over and over and over and over and over and over and over and over and over again until it hits the defined goal.

33:53 And I just feel like.

33:57 Of course, that's going to burn a bunch of tokens and maybe at the end you got something that finished, but I'm pretty certain that if you'd human steered that thing, you would have gotten the same thing with a lot less token spend and probably even faster. And so I'm a little, I'm, I'm, mark me down, you know.

34:21 July 2026 I am I am skeptical on loops I'm skeptical now that might come back to haunt me loops might get better you know there's certainly some big smart people saying it's all about loops nowadays but that just feels like just I mean brute force is not the way to get most things done yeah

34:39 Scott: Yeah, that is the million monkey theory in action,

34:43 right?

34:43 Brad: it's a million monkey theory in action exactly

34:46 Scott: It's interesting,

34:47 right,

34:49 if we guide it, you're essentially when you guide it, you're just chopping off a bunch of loops,

34:52 Brad: Yeah, you're chopping off a bunch of loops.

34:54 Scott: you're chopping off a bunch of big, expensive loops,

34:57 right?

34:58 Big, big course ones and really narrowing it down to a line.

35:04 There's probably somewhere in between.

35:06 Brad: Yeah, that's one thing I would say about tokens.

35:09 Now, the next thing I'm going to say is I have installed local inference.

35:13 inference now now i'm not actively using it but i set up local inference um and the model that i picked was an open open weight open source model and the benchmarks for it it performs about as good as haiku um now

35:30 So that to me means I wouldn't quite use it for a daily driver coding agent because I'm looking at, you know, I want something closer to, you know, Opus performance or GPT-5 performance.

35:45 But the open weight model I have on the machine that I'm able to run it at only performs about as good as Haiku.

35:53 But I am.

35:56 Seriously considering investing in a couple of Sparks that would allow me to run an Opus grade,

36:05 a GPT-5 grade local inference model. And then then my tokens become.

36:12 infinite free modulo the electricity putting into my machine i think there's a couple reasons for that not because necessarily token spend but because i want the privacy of it i want the in i want the autonomy of it i want the independence i don't want to be beholden to the cloud provider saying oops sorry you've reached the max for

36:37 for today wait wait another 12 hours or or we've decided to increase the price of these tokens and you're you're you know you've you're addicted to it so we know you're going to pay it

36:49 Scott: It's really interesting because they're very subtly in that.

36:55 When what you're doing, there's a shift from a technical problem to a business problem.

37:00 And it brings up a lot of very interesting business questions like if you're anthropic, should you be scared to death of these local models?

37:09 If you're Apple,

37:10 are you smart to have been way behind the curve here?

37:13 You know, it's real interesting.

37:15 Brad: You and I've been saying that for a while like I've been saying like I feel like Apple's strategy has always been like

37:21 focus on the hardware and

37:23 Scott: Mm-hmm.

37:23 Brad: know that local inference is coming right and so i think they may end up being the ones that were the smart ones in the end and um is anthropic nervous about it go listen to dario's public statements about open weights the guy looks nervous right he's he's he's uh you know dancing dancing a little fast you know um and of course he's got his reasons oh you know

37:48 it's not as easy as people think you know like you might get inference working but optimizing inference to run quickly that's like that's where the real optimizing the the hardware to do it you know or whatever and I'm just like okay dude like that's that's definitely for

38:03 Scott: Well, he's right.

38:04 He's not wrong, right? He's for the masses.

38:06 Brad: the masses he's not wrong

38:08 Scott: The masses he's not wrong.

38:10 Probably for even for big enterprises,

38:12 he's not wrong.

38:13 Brad: I think it's all going to come down to you can't like enterprises don't like enterprises always are going to make a decision based on like, you know, what's the what's kind of the moment that matters to me now?

38:27 Like,

38:27 I need to go quickly.

38:28 I, you know, I'd rather off offload this accountability onto a vendor.

38:33 I can have a good, you know, contractual agreement with the vendor to behave in a certain way.

38:40 way great those are all things that enterprises are used to doing but what enterprises don't like is they don't like being beholden to a single vendor they

38:47 Scott: Mm-hmm.

38:47 Brad: don't like vendor lock-in right and how many times as ctos have we had to face that question of like wait a second is this technical choice i'm making is that going to create a case of vendor lock-in and then what am i what's my plan if that becomes problematic

39:02 Scott: We should do a whole episode on vendor lock-in.

39:05 Brad: Yeah.

39:05 Scott: I mean, I feel like I'm a contrarian on that one.

39:08 Brad: Which is to say you're fine with just go with the vendor,

39:10 don't worry about lock-in?

39:12 Scott: I mean, a lot of energy,

39:14 yeah, ballpark, yes.

39:15 A lot of energy has been spent in my career avoiding vendor lock-in, but we've actually changed vendors a few times, right?

39:22 Like typically you just don't, because the costs of changing rarely outweigh the savings you get from being with some other vendor.

39:32 Brad: I hear you on that, and I think that's a valid point.

39:37 And in the case of AI,

39:38 I think my

39:41 Scott: You know what you said?

39:42 Brad: only statement on that would be these guys are in a fight to the death and they're going to do everything they can to maximize their leverage. And to me, that's a.

40:00 Scott: You just don't want to get stuck holding the ball or you don't want to get stuck, you know, without the chair or something like that. You know what I mean? So that I'm not advising like everybody should set up local inference. I'm not saying that.

40:11 But I but I think it's super I think that if you are a engineer who's thoughtful about like,

40:16 where's AI going?

40:18 I think setting up a weekend project, doing a weekend project where you set up local inference just so you understand the mechanics of it.

40:25 of it you don't necessarily have to use it but you should definitely set it up to understand the mechanics of it because i think that um

40:32 it's a tool that we're going to need to have in our back pocket right yeah

40:36 Brad: Do you know much about AWS Bedrock?

40:39 Scott: i mean it's it's a great platform that's it's kind of like that uh i think there's a ton of value in in bedrock it's like lets you it's it lets you basically kind of solve that if you your vendor locked into aws but you are able to then toggle which back-end provider you're using you're able to route

40:58 Brad: Mm hmm.

40:58 Scott: The other cool thing about Bedrock is it's kind of got its own like layers of like prompt injection protection and safety and kind of like it's got these quality in and out gates.

41:07 So that's a pretty cool feature.

41:08 Brad: Yeah.

41:08 Scott: It's got a bunch of the open source models running behind it. So you can essentially use those and you're just paying you're just renting the hardware.

41:16 So,

41:16 Brad: Yeah.

41:16 Scott: yeah,

41:16 I think Bedrock's pretty cool thing.

41:19 Brad: I'm glad to hear you say that because to me, it seems like the first order solution to vendor lock in is going to be.

41:25 going to be azure

41:25 Scott: bedrock.

41:25 Brad: google and amazon are going to put better a bedrock like solution in place i'm sure they've already got them and

41:32 Scott: Well, it's also the case that like the.

41:34 Like it's pretty easy to write your code and such like even if you didn't want to use bedrock

41:39 It's pretty easy to write your code to have a you know, the open AI API is a little different than the anthropic one But they're close enough that you can

41:47 Brad: Mm

41:47 Scott: you

41:47 Brad: -hmm.

41:48 Scott: can have a very simple and you know API contract layer that in your code that can easily toggle between the two So that that that's like pretty easy code.

41:56 So there you go How do we summarize this what we're doing as CTOs around?

42:01 around AI.

42:01 Brad: Well,

42:02 I want to summarize it in two ways.

42:05 Actually, I want to summarize more than two.

42:07 The first is, this is really more now for the CTO audience.

42:11 Having to explain what you are doing as a CTO to somebody who is not technical can sometimes help you be more effective at explaining it to your own team.

42:22 Scott: Yeah, that's a great call out.

42:23 Brad: The second thing is, a big part,

42:27 why is it important to explain?

42:28 To explain it to our own teams.

42:29 Well,

42:29 a big part of our job as CTOs is to create the narrative, the story that people can buy into the thing that doesn't really exist like I was talking about this book Sapiens and the cognitive revolution to create an idea that everybody can attach to or connect with or get in line with or I don't know be part of so that we can organize more and more people.

42:55 to do something and that is key to our jobs as CTO is

43:01 being able to tell those stories both to our own teams and to our stakeholders i guess those are the main lessons for ctos and then i guess maybe there's a framework here that other ctos could use to start to think about ai which is at least on our team what i'm thinking about is uh just going from memory here because i don't have my phone with my notes open but how do we manage tokens how do we scale it how do we create standards to scale it across a lot of

43:26 it's a lot of people how do we figure out how to totally automate this thing and how do we build more and more complex solutions with it and those are kind of the different work streams that we're working down right now they're all jumbled together right so it's there it's not one or the other but like those are kind of the four problems that we're fundamentally trying to solve right now in

43:46 Scott: Yeah.

43:47 Brad: terms of how we use this AI and I think if there was the last thing out of this

43:53 or whatever your existing LDS LDLC is.

43:58 whatever project you're working on whether it's one person or a hundred people or a thousand people that's the backbone we're not skirting from scratch here we're starting from where we're at and

44:08 Scott: Yeah.

44:08 Brad: saying okay how do we take this and level it up one more level i this is this is no different than refactoring versus rewriting and i guess fundamentally i didn't articulate this before as well but i think we're fundamentally working at refactoring our stlc not right

44:24 you're not rewriting our SDLC and

44:25 Scott: Yep.

44:25 Brad: some of these refactors might be really big sometimes when you're done refactoring it looks totally different let's

44:30 Scott: Love it. And I agree.

44:31 We didn't get into an argument again today.

44:33 We couldn't do it. We're too aligned.

44:35 Brad: get some real idiots on this call and people who really just have terrible ideas and then we can just really disagree with them

44:42 Scott: That sounds great.

44:44 Brad: all right

44:45 Scott: They should smash that subscribe button. Rate and review.

44:49 Brad: That's right.

44:49 All right.

44:50 Scott: All right.

44:50 Brad: Talk to you soon, Red.

44:51 Scott: Bye.

44:51 This episode of the How Many CTOs Does It Take podcast was produced and edited by Saffron Tepdekow.

44:58 Our theme song was AI Generated by Brad.

45:01 Please follow us on all the social media platforms at HowManyCTOsPod.

45:06 You can find us on YouTube,

45:08 LinkedIn,

45:09 Instagram,

45:10 Facebook,

45:10 ThreadsX,

45:11 TikTok and Patreon or at our website at HowManyCTOsPod.

45:15 pod.com and remember to rate, review,

45:18 and subscribe on the podcast platform of your choice.

45:21 Thanks for listening!