Transcript: Method to the Madness: A CTO Framework for Systemic AI Adoption
0:00 Scott You need a little bit of method to the madness,
0:01 Scott where at least you say we're going to do this way, but that one's still going to be people's choice.
0:22 Brad Welcome everybody to another
0:24 Brad episode of the how many CTOs does it take podcast my name is Brad Hefta-Gaub and this is my trusty co-host nice
0:32 Scott Scott Paul Brad,
0:33 Scott nice to see you.
0:35 Brad to see you I heard you went paddle boarding this morning
0:39 Scott I did. I did indeed go paddleboarding this morning. I've been having a fun week, like I told you,
0:44 Brad because Yeah, I love
0:45 Scott it. the company that I work for right now gives,
0:48 Scott I don't say gives isn't the right word, but closes for a week in the summer.
0:53 Scott And as a result, everybody takes the week off and comes back. We call it recharge week,
1:00 Scott in fact, and comes back recharged. You know, when you take a week off and everyone else is working, you just come back to more work.
1:05 Scott work right
1:06 Brad Right.
1:06 Scott so and i've been a lot like i told you also i've been watching a lot of world cup
1:10 Brad I mean, this is we don't have to talk about it because this is going to come out a month later. So everybody will know what the answer to the question is.
1:16 Brad But it's exciting to watch the World Cup. It's exciting that it's happening here.
1:20 Brad A bunch of it's happening here in Seattle.
1:22 Brad And so, yeah,
1:23 Brad that's been that's been a fun little fun little experience for all of us.
1:27 Scott yesterday i went to pioneer square
1:31 Scott Where for those of you who don't live in Seattle,
1:33 Scott that's like our old timey neighborhood with lots of bars and art galleries and stuff like that.
1:38 Scott And I watched the USA soccer game.
1:43 Scott It was super fun just to be out with the people. I'm a man of the people.
1:48 Brad You're a man of the people and the paddleboard.
1:50 Brad Cool.
1:50 Brad Well, you said you had some important CTO conversations to have, and that's why we're recording this.
1:56 Brad So what is...
1:57 Scott not important, but let me give a little bit of context, two bits of context for this.
2:02 Scott The first bit of context is as CTO's job is to sort of
2:09 Scott Set the story,
2:10 Scott frame the story, frame the narrative so that people kind of know what they're doing.
2:15 Scott Did you ever read the book Sapiens by Yuval Harari?
2:22 Brad I have not read that book.
2:24 Scott I
2:24 Brad I think you've mentioned it in the past. It's definitely a book title that's familiar,
2:28 Brad so I should probably
2:29 Scott strong recommend strong recommend easy read if you'd like that kind of stuff if you don't like it whatever so he's you've all Harari is a historian and he wrote this book called sapiens
2:42 Scott You know, you and I, we are Homo sapiens. We
2:44 Brad Mm-hmm.
2:44 Scott are from the, I don't remember the exact, like we're from the genus Homo and the species sapiens, right?
2:50 Scott And there were other species in our genus. There was Homo erectus and Homo, whatever the Neanderthalus and all these different,
3:00 Scott but somehow we survived,
3:02 Scott right? We Homo sapiens.
3:06 Scott And he writes about this and he says everyone thinks it's because of the agricultural revolution and that was certainly real but he said there was a revolution before that which he referred to as the cognitive revolution and he said that's what really separated us from the other animals because at that point we were just animals like everybody else we're still animals but like we were just equal and we're not at the top of the food chain at that time.
3:31 Brad Yeah.
3:32 Scott And the cognitive revolution fundamentally was our ability to believe in things that didn't exist.
3:41 Brad Right.
3:42 Scott So an example he gives is the country of France.
3:46 Scott France doesn't exist,
3:47 Scott right?
3:48 Scott I mean, it's drawn on a map. You can see over my shoulder, there's an atlas,
3:52 Scott right? But like, you know, there's no such thing as France in the world. But...
3:58 Scott When people believe in this idea of France, what are those three words, you know, fraternity,
4:02 Scott egality,
4:02 Scott I can't remember the last word of their moderate.
4:05 Scott When people believe in that idea of France, believe in the idea of the borders of France,
4:09 Scott then they do things like fight for France, right, all these kinds of things.
4:13 Scott Another example from the more modern times,
4:15 Scott a corporation isn't a real thing.
4:18 Scott It doesn't exist in nature. It doesn't exist anywhere.
4:21 Scott It's just a figment of our mind. We made up this idea of a corporation,
4:24 Scott which
4:24 Brad Right.
4:25 Scott is this entity,
4:26 Scott right?
4:26 Scott So anyway,
4:27 Scott his argument about Homo sapiens is we developed the ability to believe,
4:32 Scott so we developed the ability to believe,
4:34 Scott to have this cognitive belief in things that didn't exist,
4:37 Scott which allowed us to organize in bigger and bigger groups of people and thus accomplish.
4:43 Scott More complex things than our competitors could accomplish.
4:46 Scott Sure. For instance,
4:47 Scott if we were going to go fight against another.
4:49 Scott species, whether it was a Homo species or some other, we could organize bigger groups of people to defend ourselves.
4:55 Scott Okay, so what's this have to do with being a CTO?
4:56 Scott Well, part of our job as a CTO is to tell the story to frame the story so that our teams can believe in some idea that isn't really real, that doesn't
5:05 Brad really Right.
5:05 Scott exist,
5:05 Scott but they can understand some concepts that they can organize around it and then go accomplish something.
5:10 Scott So that's the first bit of context. The second bit of context is I went to a baseball game earlier this week.
5:14 Scott we and
5:16 Brad You
5:16 Scott you
5:16 Brad got to believe in balls and strikes.
5:18 Scott got to believe in balls and strikes and um well no that's actually a real thing like the ball does or doesn't go over the plate that's actually a actual thing that happens so anyway i'm sitting with my buddy who took me to the game and
5:32 Scott He is not in the tech business and he's in real estate and he just says, what are you guys doing with AI these days?
5:39 Scott And so I had to think kind of quick on my feet,
5:42 Scott like, how do I tell him what I'm doing with AI in a way that he can understand?
5:46 Scott Because if I got into the nuts and bolts of like how these agents work and stuff like that, I would have bored him.
5:52 Scott And I said, well,
5:54 Scott you know, we're kind of trying to do three things.
5:58 Brad Okay
5:58 Scott And this was and I just did this from the off the cuff.
6:01 Scott I didn't plan it.
6:03 Scott But the fact that I had to explain it in simple terms helped me do it off the cuff and explain it in a way that all of a sudden I realized like,
6:10 Scott oh,
6:10 Scott this is actually the story that I should be telling my team.
6:14 Scott And not just me,
6:16 Scott I think any CTO at a
6:19 Scott reasonably sized company needs to tell a team so we're trying to do three things let me see if i can remember it off the top of my head i actually wrote it down in my phone so i should bring up my notes here because as soon as i said it to him we're at the baseball game and i was like oh i gotta write that down that was that was i was just smart oh
6:33 Brad It's also very AI of you to come up with three things, right?
6:36 Brad Like that's another one of the AI writing tips.
6:41 Scott is that right the first thing is we're trying to do is figure out how to get
6:45 Scott get AI to do complex engineering,
6:47 Scott right?
6:48 Scott So we've talked about a lot about the shift from vibe coding to whether it's spec driven development or whatever it is, as an industry, as an industry of professional software engineers,
6:57 Scott we're trying to figure out how do we get this to build real stuff for us, not just,
7:02 Scott you know, toy projects.
7:03 Scott And so we've, that's like one, one work stream we're working on, one track we're working on.
7:08 Scott The second thing we're working on, and I think you and I have talked about this,
7:12 Scott is we're also trying to figure out how to fully automate this stuff.
7:15 Scott You've got,
7:16 Scott we've talked about your.
7:17 Scott crew of robots that you got working for you.
7:20 Scott So we got different work streams.
7:22 Scott People are trying to figure out how do we totally make this thing autonomous or as close to autonomous as we can for either simple work streams or complex work streams. We start with simple ones and we work our way up.
7:33 Scott And then the third piece that I think we're working on is how do we scale it?
7:37 Scott Meaning how do we get
7:39 Scott lots of people in an organization to be using this and that's where sort of standards and systems are coming into play i said this very succinctly right to you but i i kind of just said it off the cuff to my friend as i was trying to explain to him well this is what we're doing with ai because if i explained to him well i got an engineer who's trying to do this blah blah he would have just his eyes would have glassed over he wouldn't know what i was talking
8:01 Brad Right.
8:01 Scott about but he thought that was very interesting so i want to know what you think of that
8:05 Scott Is that it? Is that what we're all doing right now is kind of going on those three work streams?
8:10 Scott They're kind of intertwined. They're not as nice and clean and tidy as I just described them. But fundamentally,
8:16 Scott it seems like that's what we're doing.
8:19 Brad Yeah, let me see if I can play that back to see if I understood what you're saying.
8:23 Brad So the first one is just like.
8:25 Brad more and more like investing in the engineers using AI to do complex engineering tasks like get them get the engineers educated get the tooling set up correctly and just kind of like leaning into if I would have had an engineer do it have an engineer do it with with AI to go bigger faster better
8:46 Brad That's
8:46 Scott Yeah,
8:47 Brad item one,
8:47 Scott let's even go further back to when Microsoft introduced IntelliSense, right?
8:54 Scott That was like a very crude form of AI,
8:57 Scott right? To try and predict what the end of this function,
9:00 Scott autocomplete should be,
9:01 Scott right?
9:02 Scott And so it's just been an evolution of their doing more and more and more complex engineering tasks with some computer assistance.
9:08 Brad Right, but just to be clear.
9:11 Brad at most organizations are not building those complex systems they're they're just using the ones that have been built
9:18 Scott Correct.
9:19 Scott Trying to build other complex systems with those,
9:21 Brad with
9:21 Scott right? We're trying to build
9:22 Brad those exactly
9:22 Scott a mobile
9:22 Brad so yeah
9:23 Scott app or a soft banking platform or whatever you're trying to build, right?
9:28 Brad but there is like yeah but to your point like you're doing that and you're using ai that way right like you're using ai you're trying to get your team to use ai more you're trying to
9:37 Brad and it's interesting to me i'll just pause on that one for a second the whole like token maxing thing where like people were panicked about their token bills and stuff like that i think that's an interesting piece that's got to be in that category it's probably a combination of that category one and your second one which is category two where um where like
10:00 Brad The leadership is trying to tell teams like I want you using AI and we're going to incentivize you to use AI and we're going to do a leaderboard on who's used the most tokens.
10:11 Brad So I think that's pretty consistently people are in that space,
10:15 Brad not necessarily trying to do bad metrics like leaderboards on tokens, but get our engineers to use the best.
10:24 Brad most advanced tools to write the software that is building that we're building okay
10:29 Scott Well, that's not the first point, really.
10:31 Brad so maybe I'm
10:32 Scott I
10:32 Brad missing
10:32 Scott mean, I think that's a thing we're doing.
10:34 Scott Okay,
10:35 Scott so the first things to say I could do and, you know,
10:37 Scott OpenAI really released their first chat to people.
10:42 Scott It could help you write a function or maybe it could make a web page, right?
10:45 Brad Yeah.
10:46 Scott But it couldn't build a complex trading system for a bank,
10:49 Scott right?
10:49 Brad Yeah.
10:50 Brad Or even like a complete system,
10:52 Brad like a whole feature,
10:53 Brad like brand
10:54 Scott Right,
10:54 Brad new feature called,
10:55 Brad I want downloadable reports from this dashboard.
11:00 Scott right.
11:01 Scott And so I feel like part of what we're doing is trying to push the AI.
11:05 Scott How can we build increasingly complex things?
11:08 Scott it's
11:08 Brad Hmm.
11:09 Scott nice that you can help me autocomplete a function but i want you to build the whole feature and then i want to go from building a whole feature to building a whole system right
11:16 Brad Yes.
11:16 Scott and so we're that's that's me that's the first work stream that we're working on is how do we get this to more effectively build more and more increasingly complex things
11:25 Brad Okay, cool.
11:26 Brad I agree with that statement. I think that's what I'm seeing within all my teams. That's certainly what I'm seeing.
11:30 Brad You know, we talked a couple episodes back about.
11:34 Brad the um my beaker stack um template
11:37 Scott Yeah.
11:37 Brad and my my uh skein uh ai memory backplane and my my desire to build sass apps every six weeks by the time this one comes out there'll be another sass app out i actually have it almost ready to release um and so um yeah and i and basically i haven't yet just turned to the ai and said
11:58 Brad here's the spec for the whole app build the whole thing i haven't done that yet but i am at the place where i'm like literally like big feature by big feature like this that the app i'm i'm going to release shortly is essentially like a um
12:13 Brad a retirement calculator it's literally a retirement calculator app that you can kind of plug in your your current you know whatever your savings it's like that and it runs monte carlo and it runs back history back testing and stuff like that it's got reports it's got all these different like compare and contrast like oh what if i delay my retirement by a year what happens you know all this stuff and literally in that case i'm i'm just given the ai like big chunky features one at a time
12:39 Brad And so I didn't have it build the whole thing, but I have it build pretty big chunky features in
12:43 Scott Mm
12:43 Brad one
12:43 Scott -hmm.
12:44 Brad shot, right?
12:45 Brad So yeah,
12:45 Brad that's a great example of like moving to more and more complex tasks.
12:49 Brad And maybe by my third SaaS app, I'll be brave enough to just say,
12:53 Brad here's my framework,
12:55 Brad here's my well-written spec,
12:56 Brad build me the entire thing and let me see what it looks like. And there's certainly people out there that are kind of pushing those boundaries,
13:02 Brad right?
13:03 Brad So yeah,
13:04 Brad I totally agree with that first bucket.
13:05 Brad The second bucket I heard you talk about.
13:07 Brad is the do it in the dark factory do it automatically do it autonomously as opposed to like you know lots of human steering did i get that one right so the
13:18 Scott That's the second one,
13:19 Brad the
13:19 Scott kind
13:19 Brad second
13:19 Scott of, yeah,
13:19 Brad bucket is like
13:20 Scott trying to figure out how to get to a software factory
13:22 Brad how do you get to an automatic uh in you know dark factory meaning like the humans aren't having to do the steering or they're doing the steering way way way high up you know yeah i agree
13:32 Scott just checking in.
13:33 Scott From time
13:34 Brad checking
13:34 Scott to time on
13:34 Brad in from time to time to see how it's doing the
13:36 Scott the human in the loop.
13:37 Brad human taking the human out of the loop or moving the human to a much much further distant um able to and yeah and for sure lots of experiments are happening in that space you know lots of people are going down that road you know um ai tinkerers joe heitzberg he's his team's like constantly having these you know
13:59 Scott We got to get Joe on here.
14:00 Brad yeah we should get joe on yeah for
14:02 Scott I'll
14:02 Brad sure yeah
14:02 Scott call Joe. I'm friends with Joe. I'll call him up.
14:04 Brad um so yeah he's ai tinkerers are just doing lots and lots of stuff around like more and more automation you know and so yeah totally that's a that's a great bucket and then when i think i heard your third bucket this is what i thought i heard but this might have been confirmation bias on my part the tooling and the framework and the systems and the processes around all that to keep everything
14:28 Brad doing
14:28 Scott No,
14:28 Brad quality is that what
14:29 Scott sort
14:29 Brad you meant or
14:29 Scott of.
14:29 Brad what did you mean by that
14:30 Scott I
14:30 Brad i
14:30 Scott mean this. Within a software engineering organization,
14:34 Scott once you get to two people in your software engineering organization,
14:38 Scott you start to have some sort of standards and systems.
14:41 Scott For instance,
14:42 Scott you might agree,
14:43 Scott well,
14:44 Scott first you might agree that you're going to use source control, and then you might agree which source control you're going to use.
14:50 Brad feel like this was our second episode we talked about what are the bare minimums of
14:53 Brad So of a software team,
14:55 Brad and I think source control was the first one we came up with.
14:58 Scott Right.
14:58 Scott So like as soon as you have more than one person,
15:00 Scott you start to.
15:02 Scott put some of those things in place and we have all sorts of standards within our industry that people don't even question anymore like source control nobody comes in you don't hire anybody if you're a company they say why are you guys using source control that's you know so um but you know source control code reviews we have we have a convention that we agree that everybody gets their code reviewed in fact it's built into our
15:28 Scott door source control right because
15:30 Brad Yeah.
15:30 Scott we use so there's all these conventions we have for how we do software because we've understood we understand as a industry or as a community or as a team that they help us do better work and so i think what
15:46 Scott I'm really emphasizing with my team right now is we need to be doing that within our team as it relates to AI,
15:56 Scott right?
15:57 Scott So for instance,
15:58 Scott we already have one AI agent code review tool,
16:03 Scott right? We've agreed this is we're going to put it
16:05 Brad Yeah.
16:05 Scott into our build pipeline and we've agreed that this is part of it.
16:08 Scott We've got other parts of people using AI that are very inconsistent and not to say some are better or worse than others,
16:16 Scott but I think just like every other aspect of software development,
16:19 Scott it is wise for us to look at what people are experimenting with and make decisions about,
16:25 Scott okay,
16:25 Scott that's good.
16:26 Scott We need to make that universal systemic across our whole.
16:31 Scott What everybody does, we've got 150 engineers,
16:33 Scott what everybody does here,
16:35 Scott they all, I'll use the same example,
16:37 Scott everybody all has their code reviewed by an AI code review agent.
16:41 Brad Yeah.
16:41 Scott That's already happening.
16:42 Scott Well, what else is there,
16:43 Scott right?
16:44 Scott I think part of that, part of this work stream that I was talking about is trying to build those standards.
16:51 Scott And I think there's two
16:52 Brad In
16:52 Scott reasons.
16:53 Brad an AI world.
16:55 Scott Yeah.
16:55 Brad to how to adjust and build uh okay so in other words let me see if i can play that back then so you're saying like all software engineering teams have some set of processes and standards that they've been following whatever those were and they like they have pretty common shapes and we can like but there's always a little bit of tweaks here and there and you know typically it's whatever you know code review it's you know it's it's you know whatever the rules and the processes that the team follow for the process
17:22 Brad And what I think I hear you saying is AI is changing how we work. So we need to make sure that we're improving and optimizing our whatever our workflows are to be good workflows in the new AI coding world.
17:38 Brad So like the ADLC we talked about a couple months ago.
17:41 Scott Yes, and I would say...
17:45 Scott Not only good in the AI world,
17:46 Scott but we need to bring consistency. We need to shift from organic adoption to systemic usage,
17:54 Scott right?
17:55 Scott In certain aspects of what we don't organically adopt GitHub.
17:59 Scott Some people don't decide to use GitHub a lot and some people decide to use GitHub a little,
18:03 Scott right?
18:03 Scott We are systemic about how we use GitHub.
18:06 Scott in our organization.
18:07 Brad You know, it's interesting that you say that,
18:09 Brad and I, a lot of the work I'm doing now is on pretty small tight-knit teams, and so it's by default, like,
18:17 Brad it's already systemic, you know, it's already systematic if, if,
18:21 Brad you know, the team is small,
18:22 Brad they've kind of like organically came with the way they work, and that is effectively the system. But in an organization as large,
18:28 Brad and you and I have worked in much,
18:29 Brad much larger organizations, you're currently at a.
18:31 Brad about a larger organization in that it's 100 plus engineers and so you've got lots of different teams um when we had ted neward on like almost a year ago he talked a lot about kind of like how there is no like best practice there's only the practice that's best for your team um and i'm kind of curious how you're thinking about the how i mean i i it resonates with me the idea that you'd want
18:58 Brad want to have system-wide processes and policies.
19:02 Scott Mm-hmm.
19:03 Brad And I also can picture a world where variations amongst teams is not necessarily a bad thing.
19:09 Scott Okay.
19:10 Scott Yes,
19:11 Scott I'm not saying it should all be totally uniform.
19:13 Scott I'm saying let's figure out what is good, what should be uniform.
19:18 Scott Source control is such a great example because it's not such not a debate.
19:22 Brad Right.
19:23 Scott We figured out that should be uniform and we just do it.
19:26 Scott But there's other parts of software engineering that we don't think should be uniform.
19:30 Scott So
19:30 Brad Yeah,
19:30 Scott we add to those things.
19:30 Scott So what
19:31 Brad that's fair enough.
19:31 Scott are the parts that we should start to make uniform?
19:34 Scott And we've already decided a code review agent is something we should make uniform,
19:38 Brad Yeah.
19:38 Scott right?
19:38 Scott Are there other things we should make uniform?
19:40 Brad Yeah.
19:41 Scott And there's going to be lots of things that we shouldn't make uniform.
19:44 Scott And we definitely want people pushing ahead because it's changing so fast.
19:49 Scott We definitely want people pushing ahead and experimenting with all the new stuff and like coming back to the home base.
19:54 Scott And I have this idea of like scouts going.
19:56 Scott Just going out and scouting
19:57 Brad Yeah.
19:57 Scott out and then coming back and saying, guys, look what I found.
19:59 Scott we should all do this thing and
20:01 Brad Yeah,
20:01 Scott then then maybe more people will try it out and if we decide it's good so yeah now we're going to make it our our convention our agreement and our team we all do whatever the thing is and that has two benefits well maybe it's the same side of both it just raises all boats right i feel like what's changed for me in the last year is this organic adoption to systemic usage thing we've been doing organic adoption for the last year all sorts of people are pushing ahead and
20:24 Scott in all sorts of different ways of course there's some people who aren't pushing ahead but you know all sorts of people are pushing ahead bringing ideas back saying we should do it this way we should do it that way and you need a little bit of method to the madness at least you say we're gonna do this way but that one's still going to be people's choice right
20:43 Brad Yeah.
20:44 Brad God damn it, Scott.
20:45 Brad I thought we were going to have an argument.
20:47 Scott Ha, ha, ha, ha, ha.
20:48 Brad But yeah,
20:49 Scott Yeah, a really interesting one in this, like I'll give you a real example in this one.
20:52 Scott We've probably talked here before about spec-driven development and
20:56 Brad mm
20:56 Scott spec kit versus plan mode,
20:59 Scott which are both versions of spec-driven development,
21:02 Scott right?
21:02 Scott And there are camps at our company which are like spec kit for everything.
21:06 Scott And there are camps which are like,
21:09 Scott well,
21:09 Scott spec kit's overkill, spec kit for nothing.
21:12 Scott And there's a camp which is,
21:13 Scott well,
21:13 Scott it depends what you're...
21:14 Scott that's what you're working on right and
21:14 Brad Right.
21:15 Scott so it's like okay well maybe we could just set up some guidelines to say under these conditions yolo and under these conditions use
21:22 Brad Yeah.
21:23 Scott plan mode you got to use plan mode under these conditions you gotta you gotta use spec kit i don't know what those conditions would be but that's how you start to make a bring some method to the madness
21:33 Brad I'll throw out another thing along that spectrum that you might want to start thinking about, which is I think this is halfway in between spec.
21:39 Brad spec kit and plan mode which is still write basically write a either have the agent or write yourself but probably have the agent write a markdown file which is the feature brief of the thing that's being built and have that checked into the code
22:01 Brad you know next to the code right this is very much what you're already doing with spec kit
22:05 Scott A markdown file that's a speech or brief.
22:07 Scott Yeah.
22:07 Brad yeah
22:07 Scott Okay.
22:07 Brad feature
22:08 Scott That's
22:08 Brad brief
22:08 Scott more
22:08 Brad base
22:08 Scott than plan mode, but less than spec mode.
22:11 Brad more than plan mode but less than spec mode and what you do is you write you have the agent write the feature brief you save it in line in the file with the with the code and then you have the agent run plan off of that
22:25 Brad That feature brief.
22:26 Scott Yeah.
22:26 Brad It was just like a mini version of SpecKit, but it doesn't have all the extra layers of SpecKit of like the Constitution and the,
22:33 Brad you know, anyway,
22:35 Brad that's what I'm seeing. That's what I'm seeing more and more of my teams kind of that's where they've landed. Right.
22:41 Brad So they they kind of like they tried they were they were already like using plan mode that they were really comfortable with.
22:48 Brad They tried Spec Kit.
22:49 Brad It felt like overkill,
22:51 Brad but the thing that they liked about Spec Kit was saving the, having the feature brief,
22:57 Brad having the kind of detail,
22:58 Brad the spec part.
23:01 Brad saved into the code and then running plan off of that but anyway to your point that's
23:07 Scott wait
23:07 Brad a great
23:07 Scott wait oh before before we go off of that we need a brand name for that middle ground right
23:11 Brad i think it's just called i don't know what the brand name for it is i don't know i i see more and more teams doing it it's definitely like or like the the industry has organically come up with this as an as a approach
23:23 Scott and if we could just brand it
23:25 Brad i
23:25 Scott then
23:25 Brad know we try
23:25 Scott we'll be
23:26 Brad to call it the adlc we got the
23:28 Brad Got the animals even
23:29 Scott I'm going to have to come up with a name for that.
23:30 Scott Okay.
23:30 Scott So yeah. So here's another example, another example, two independent examples, but they're the same thing.
23:38 Scott My security team is like,
23:41 Scott hey, I want to make like some sort of security agent that can like review the.
23:49 Scott effectively review the plan they want to review what we you know like the tech spec like the technical design doc like
23:56 Scott Because they're reviewing a lot of those with their own eyes and they're like, well, could we write an agent to do that, right?
24:01 Brad Especially with Fable.
24:03 Scott Is Fable back out yet?
24:04 Brad Fable's back out.
24:05 Brad Yeah.
24:06 Scott Oh, the architect.
24:07 Brad I mean, I don't know if by the time this comes out, it might be not out again,
24:10 Brad but yeah,
24:11 Brad they reopened it up yesterday.
24:13 Scott Architects want to do the same thing.
24:14 Scott Like, hey, often someone comes ask us to review stuff.
24:18 Scott Could we just have an agent review?
24:21 Scott their plans and then okay so then there this begs a question
24:25 Scott If we do that, should that become part of our system like you just always do it? And if you do a thing like you were talking about before where you write a feature brief and then you have plan mode made the technical plan,
24:37 Scott should it always go through the architect agent and the security agent,
24:40 Scott right?
24:40 Brad Yeah.
24:41 Scott Those are the kinds of things that I mean when I'm thinking about going from organic to systemic.
24:46 Brad Yeah.
24:46 Scott And I don't know what they're going to be and I think, well,
24:50 Scott you know what I think they're going to be, I think they're going to be a lot of the same.
24:52 Scott The same things that we do already that we just find ways that we can sort of automate with agents.
24:57 Brad i'll tell you so yeah i love this i love the the it's you're scratching right now like there's there's some there's definitely some stuff there and i'll i'll tell you i'll tell a little story about an example from one of my teams that i work with um and this is a team that's like this is like they're definitely like um very much
25:16 Brad ai pilled they're using the agents they're they're you know they're the team that i'm talking about that are kind of more and more checking in feature briefs and then building off of it and they're really really productive they're like 3 000 plus lines of code a day per developer you know again clocks are a bad metric but it's like real business value where they're just like adding feature feature feature feature feature right and they um
25:41 Brad had a moment where all of a sudden they got we all got they all got notices that their co-pilot github co-pilot tokens had reached their max usage on github and they were like what happened right and this team is basically they all spend about they all have essentially a stipend of like 200 bucks a month per developer like that's and they're
26:09 Brad And that's all they've been spending and they're super productive.
26:12 Brad So like nowhere near this token maxing like,
26:15 Brad oh my God, we're spending like our whole,
26:16 Brad you know, we're spending an employee salary worth of tokens,
26:20 Brad right?
26:20 Brad And they don't need it because they're still super fast at that budget.
26:25 Brad But we so the DevOps guy was like digging into the reports on GitHub and we're like, this is weird, like because we thought the only thing that was happening on GitHub was just the code reviews.
26:37 Brad Right.
26:39 Brad And the report came back and there was just one like line where it's like almost all the usage was like the copilot code review model.
26:49 Brad But then there was this one like weird spiky line that was essentially Opus 4.8 and we're like,
26:58 Brad wait a second, like.
27:00 Brad nobody's using github copilot to do coding it like all of us they all have their own you know local agents that do coding and another person ran the same the same it looked like the exact same report like literally the lines were the same color and the lines were the same shape but then if you looked at the if you looked at the legend it was the name of one developer was perfectly matched to the opus for
27:27 Brad is 4.8 and everyone else's name were matched to the other things and we were like what the heck what are you what are you doing and it turned out that he had written a code review his own custom code review skill that he was using and so there that team would always like click co-pilot code review and then they also would as individuals they'd run their own code reviews using their own local
27:54 Brad own local agents but his script was specifically using copilot uh github tokens and was specifically using opus 4.8 now ultimately
28:05 Brad his his co his extra code review was doing good stuff and everyone liked that he was doing it but it was funny because to your point it was an example where he had innovated in a way that the rest of the team hadn't and then we later discovered he was doing it different and because of like a weird spike in our bill now not a crazy spike like uh instead of a 50 budget that we had for copilot we
28:31 Brad We upped it to $200 for Copilot.
28:34 Scott Well, and what's interesting about that, now it begs the question,
28:37 Scott should we adopt this thing this guy made? Should everybody be using this?
28:40 Brad Exactly. Exactly. That's the question.
28:42 Scott And if you think about what has made software so one of the aspects that have made software so effective,
28:50 Scott I think that's the right word to use,
28:52 Scott is our ability to encapsulate and reuse.
28:57 Scott My friend who I just mentioned I went to the baseball game with is in real estate. He's a home builder.
29:02 Scott And every time he builds a house, they start from scratch. They're building bathrooms and wiring and kitchens and all that stuff
29:08 Brad Yeah.
29:08 Scott again.
29:09 Scott Whereas we build a code review agent once and a whole bunch of people get to use it over and over and over again. And our whole industry is just layer upon layer upon layer upon layer upon layer upon layer of innovation.
29:22 Scott right
29:22 Brad Yeah.
29:22 Scott that where it's truly like standing on the shoulders of giants and giants and giants and giants right and it's what it's something that's really pretty cool and magical about what we do and
29:34 Scott we're in this like hyper phase of like where we have all these decisions to make about that so that's another work stream then you bring up the last one which you added to this pile which is yeah the last work stream we're working on is figuring out how to manage tokens and
29:47 Brad Yeah.
29:48 Scott it's really in the water right now in the air i should say in the water i don't know people are really thinking about
29:54 Scott how to manage tokens that's actually a little frustrating to me I feel like if you have the money you shouldn't be thinking about it at all you should be figuring out how to make this stuff valuable and at some point down the road optimize but it feels like a premature optimization to me right now but that's the fourth work stream I think is token management
30:12 Brad I think that, well, here's my take on token management.
30:15 Brad I think that for.
30:17 Brad teams that were already behaving rationally about their use of AI,
30:21 Brad where they were essentially,
30:24 Brad I guess I'm going to say rationally, I think my teams are doing this rationally, right?
30:29 Brad Like they're getting a lot of productivity out of 200 bucks a month.
30:33 Brad If they need to bump to 500 bucks a month,
30:35 Brad I'm happy with that,
30:36 Scott Mm
30:36 Brad right?
30:37 Scott hmm.
30:37 Brad Like no big deal.
30:38 Brad You guys are
30:39 Scott Right.
30:39 Brad doing great, right?
30:40 Brad I think that where the reason that token spend is so in the water right now or in the air, as you said,
30:48 Brad it's in zeitgeist, is this token maxing thing that went on where a bunch of, I'm going to say.
30:56 Brad not you know nearsighted or my maybe myopic is the right way to describe this leaders decided to incentivize their engineers to to have token leaderboards and anything that you measure is going to turn you know like people are going to optimize it you know so if you tell them like i want you to spend more tokens i'm going to do a leaderboard of tokens you're going to get massive token spend like that's just human nature
31:21 Scott Every time we talk about token maxing, it reminds me of a story from one of my children who was about seven years old.
31:27 Scott And it's adorable little story. Can I share it?
31:29 Brad Sure.
31:31 Scott There used to be a video game called Club Penguin.
31:34 Brad Yeah.
31:34 Scott Do you remember that?
31:35 Brad Yes.
31:37 Scott OK,
31:37 Scott so Club Penguin was this online game for people who don't know that was probably run by Disney or something.
31:43 Scott It was for kids and it was a little virtual world and you went in there and you played games and did stuff and you could get.
31:48 Scott you could get tokens and then you could go buy stuff from the store and there was no money i mean you paid ten dollars a month for it or whatever but there was no like you couldn't buy coins and stuff like that one of the games that was in there was that you could like play the dj in
32:01 Brad Hmm.
32:01 Scott the club like there's a whole little virtual world and there was like a little dance club a discotheque in there and you can play the dj and you got tokens you earn tokens for being the dj
32:13 Scott So one day I noticed my kid fiddling with the timer on the oven and I'm like,
32:18 Scott hey,
32:18 Scott what are you doing?
32:19 Scott And he's like,
32:20 Scott well, I figured out that as long as you're in the DJ booth,
32:24 Scott you earn tokens.
32:26 Scott You just have to be active every at least every 10 minutes.
32:29 Scott So I've been setting the timer for nine minutes and then I just go over and move the mouse.
32:35 Scott just so I can do something else so they're just earning me tokens all day and every time I just hear this phrase token maxing I think of that little story it was just such an adorable little story for like a seven-year-old
32:45 Brad Eddie
32:45 Scott boy
32:45 Brad hacked the system. He figured it out.
32:46 Scott he figured out that he didn't have to do it and so when I hear the phrase token maxing it's a totally different use of the word token maxing I just think about like adorable little story when my son was like seven years old
32:58 Brad So I'm with you on let's not like if you're being already if you're already being rational about your your the way you're using the AI agents, it's not that big deal.
33:08 Brad Now, I have two.
33:10 Brad I have two additional things to say about the token problem.
33:13 Brad The first one I'll say is I think where the crazy token spends are happening is this whole concept of looping.
33:20 Brad And this is something that is also in the zeitgeist and you're hearing like the anthropic team is claiming that they're all about doing looping and you hear the OpenQuad team talking about they're all doing looping. And what looping is, is the idea that you give the agent an end goal,
33:35 Brad like an acceptance criteria.
33:36 Brad And then you literally just say,
33:38 Brad keep running until you hit this acceptance criteria.
33:41 Scott Mm-hmm.
33:41 Brad And it just does essentially like, you know, brute force implementation of over and over and over and over and over and over and over and over and over again until it hits the defined goal.
33:53 Brad And I just feel like.
33:57 Brad Of course, that's going to burn a bunch of tokens and maybe at the end you got something that finished, but I'm pretty certain that if you'd human steered that thing, you would have gotten the same thing with a lot less token spend and probably even faster. And so I'm a little, I'm, I'm, mark me down, you know.
34:21 Brad July 2026 I am I am skeptical on loops I'm skeptical now that might come back to haunt me loops might get better you know there's certainly some big smart people saying it's all about loops nowadays but that just feels like just I mean brute force is not the way to get most things done yeah
34:39 Scott Yeah, that is the million monkey theory in action,
34:43 Scott right?
34:43 Brad it's a million monkey theory in action exactly
34:46 Scott It's interesting,
34:47 Scott right,
34:49 Scott if we guide it, you're essentially when you guide it, you're just chopping off a bunch of loops,
34:52 Brad Yeah, you're chopping off a bunch of loops.
34:54 Scott you're chopping off a bunch of big, expensive loops,
34:57 Scott right?
34:58 Scott Big, big course ones and really narrowing it down to a line.
35:04 Scott There's probably somewhere in between.
35:06 Brad Yeah, that's one thing I would say about tokens.
35:09 Brad Now, the next thing I'm going to say is I have installed local inference.
35:13 Brad inference now now i'm not actively using it but i set up local inference um and the model that i picked was an open open weight open source model and the benchmarks for it it performs about as good as haiku um now
35:30 Brad So that to me means I wouldn't quite use it for a daily driver coding agent because I'm looking at, you know, I want something closer to, you know, Opus performance or GPT-5 performance.
35:45 Brad But the open weight model I have on the machine that I'm able to run it at only performs about as good as Haiku.
35:53 Brad But I am.
35:56 Brad Seriously considering investing in a couple of Sparks that would allow me to run an Opus grade,
36:05 Brad a GPT-5 grade local inference model. And then then my tokens become.
36:12 Brad infinite free modulo the electricity putting into my machine i think there's a couple reasons for that not because necessarily token spend but because i want the privacy of it i want the in i want the autonomy of it i want the independence i don't want to be beholden to the cloud provider saying oops sorry you've reached the max for
36:37 Brad for today wait wait another 12 hours or or we've decided to increase the price of these tokens and you're you're you know you've you're addicted to it so we know you're going to pay it
36:49 Scott It's really interesting because they're very subtly in that.
36:55 Scott When what you're doing, there's a shift from a technical problem to a business problem.
37:00 Scott And it brings up a lot of very interesting business questions like if you're anthropic, should you be scared to death of these local models?
37:09 Scott If you're Apple,
37:10 Scott are you smart to have been way behind the curve here?
37:13 Scott You know, it's real interesting.
37:15 Brad You and I've been saying that for a while like I've been saying like I feel like Apple's strategy has always been like
37:21 Brad focus on the hardware and
37:23 Scott Mm-hmm.
37:23 Brad know that local inference is coming right and so i think they may end up being the ones that were the smart ones in the end and um is anthropic nervous about it go listen to dario's public statements about open weights the guy looks nervous right he's he's he's uh you know dancing dancing a little fast you know um and of course he's got his reasons oh you know
37:48 Brad it's not as easy as people think you know like you might get inference working but optimizing inference to run quickly that's like that's where the real optimizing the the hardware to do it you know or whatever and I'm just like okay dude like that's that's definitely for
38:03 Scott Well, he's right.
38:04 Scott He's not wrong, right? He's for the masses.
38:06 Brad the masses he's not wrong
38:08 Scott The masses he's not wrong.
38:10 Scott Probably for even for big enterprises,
38:12 Scott he's not wrong.
38:13 Brad I think it's all going to come down to you can't like enterprises don't like enterprises always are going to make a decision based on like, you know, what's the what's kind of the moment that matters to me now?
38:27 Brad Like,
38:27 Brad I need to go quickly.
38:28 Brad I, you know, I'd rather off offload this accountability onto a vendor.
38:33 Brad I can have a good, you know, contractual agreement with the vendor to behave in a certain way.
38:40 Brad way great those are all things that enterprises are used to doing but what enterprises don't like is they don't like being beholden to a single vendor they
38:47 Scott Mm-hmm.
38:47 Brad don't like vendor lock-in right and how many times as ctos have we had to face that question of like wait a second is this technical choice i'm making is that going to create a case of vendor lock-in and then what am i what's my plan if that becomes problematic
39:02 Scott We should do a whole episode on vendor lock-in.
39:05 Brad Yeah.
39:05 Scott I mean, I feel like I'm a contrarian on that one.
39:08 Brad Which is to say you're fine with just go with the vendor,
39:10 Brad don't worry about lock-in?
39:12 Scott I mean, a lot of energy,
39:14 Scott yeah, ballpark, yes.
39:15 Scott A lot of energy has been spent in my career avoiding vendor lock-in, but we've actually changed vendors a few times, right?
39:22 Scott Like typically you just don't, because the costs of changing rarely outweigh the savings you get from being with some other vendor.
39:32 Brad I hear you on that, and I think that's a valid point.
39:37 Brad And in the case of AI,
39:38 Brad I think my
39:41 Scott You know what you said?
39:42 Brad only statement on that would be these guys are in a fight to the death and they're going to do everything they can to maximize their leverage. And to me, that's a.
40:00 Scott You just don't want to get stuck holding the ball or you don't want to get stuck, you know, without the chair or something like that. You know what I mean? So that I'm not advising like everybody should set up local inference. I'm not saying that.
40:11 Scott But I but I think it's super I think that if you are a engineer who's thoughtful about like,
40:16 Scott where's AI going?
40:18 Scott I think setting up a weekend project, doing a weekend project where you set up local inference just so you understand the mechanics of it.
40:25 Scott of it you don't necessarily have to use it but you should definitely set it up to understand the mechanics of it because i think that um
40:32 Scott it's a tool that we're going to need to have in our back pocket right yeah
40:36 Brad Do you know much about AWS Bedrock?
40:39 Scott i mean it's it's a great platform that's it's kind of like that uh i think there's a ton of value in in bedrock it's like lets you it's it lets you basically kind of solve that if you your vendor locked into aws but you are able to then toggle which back-end provider you're using you're able to route
40:58 Brad Mm hmm.
40:58 Scott The other cool thing about Bedrock is it's kind of got its own like layers of like prompt injection protection and safety and kind of like it's got these quality in and out gates.
41:07 Scott So that's a pretty cool feature.
41:08 Brad Yeah.
41:08 Scott It's got a bunch of the open source models running behind it. So you can essentially use those and you're just paying you're just renting the hardware.
41:16 Scott So,
41:16 Brad Yeah.
41:16 Scott yeah,
41:16 Scott I think Bedrock's pretty cool thing.
41:19 Brad I'm glad to hear you say that because to me, it seems like the first order solution to vendor lock in is going to be.
41:25 Brad going to be azure
41:25 Scott bedrock.
41:25 Brad google and amazon are going to put better a bedrock like solution in place i'm sure they've already got them and
41:32 Scott Well, it's also the case that like the.
41:34 Scott Like it's pretty easy to write your code and such like even if you didn't want to use bedrock
41:39 Scott It's pretty easy to write your code to have a you know, the open AI API is a little different than the anthropic one But they're close enough that you can
41:47 Brad Mm
41:47 Scott you
41:47 Brad -hmm.
41:48 Scott can have a very simple and you know API contract layer that in your code that can easily toggle between the two So that that that's like pretty easy code.
41:56 Scott So there you go How do we summarize this what we're doing as CTOs around?
42:01 Scott around AI.
42:01 Brad Well,
42:02 Brad I want to summarize it in two ways.
42:05 Brad Actually, I want to summarize more than two.
42:07 Brad The first is, this is really more now for the CTO audience.
42:11 Brad Having to explain what you are doing as a CTO to somebody who is not technical can sometimes help you be more effective at explaining it to your own team.
42:22 Scott Yeah, that's a great call out.
42:23 Brad The second thing is, a big part,
42:27 Brad why is it important to explain?
42:28 Brad To explain it to our own teams.
42:29 Brad Well,
42:29 Brad a big part of our job as CTOs is to create the narrative, the story that people can buy into the thing that doesn't really exist like I was talking about this book Sapiens and the cognitive revolution to create an idea that everybody can attach to or connect with or get in line with or I don't know be part of so that we can organize more and more people.
42:55 Brad to do something and that is key to our jobs as CTO is
43:01 Brad being able to tell those stories both to our own teams and to our stakeholders i guess those are the main lessons for ctos and then i guess maybe there's a framework here that other ctos could use to start to think about ai which is at least on our team what i'm thinking about is uh just going from memory here because i don't have my phone with my notes open but how do we manage tokens how do we scale it how do we create standards to scale it across a lot of
43:26 Brad it's a lot of people how do we figure out how to totally automate this thing and how do we build more and more complex solutions with it and those are kind of the different work streams that we're working down right now they're all jumbled together right so it's there it's not one or the other but like those are kind of the four problems that we're fundamentally trying to solve right now in
43:46 Scott Yeah.
43:47 Brad terms of how we use this AI and I think if there was the last thing out of this
43:53 Brad or whatever your existing LDS LDLC is.
43:58 Brad whatever project you're working on whether it's one person or a hundred people or a thousand people that's the backbone we're not skirting from scratch here we're starting from where we're at and
44:08 Scott Yeah.
44:08 Brad saying okay how do we take this and level it up one more level i this is this is no different than refactoring versus rewriting and i guess fundamentally i didn't articulate this before as well but i think we're fundamentally working at refactoring our stlc not right
44:24 Brad you're not rewriting our SDLC and
44:25 Scott Yep.
44:25 Brad some of these refactors might be really big sometimes when you're done refactoring it looks totally different let's
44:30 Scott Love it. And I agree.
44:31 Scott We didn't get into an argument again today.
44:33 Scott We couldn't do it. We're too aligned.
44:35 Brad get some real idiots on this call and people who really just have terrible ideas and then we can just really disagree with them
44:42 Scott That sounds great.
44:44 Brad all right
44:45 Scott They should smash that subscribe button. Rate and review.
44:49 Brad That's right.
44:49 Brad All right.
44:50 Scott All right.
44:50 Brad Talk to you soon, Red.
44:51 Scott Bye.
44:51 Scott This episode of the How Many CTOs Does It Take podcast was produced and edited by Saffron Tepdekow.
44:58 Scott Our theme song was AI Generated by Brad.
45:01 Scott Please follow us on all the social media platforms at HowManyCTOsPod.
45:06 Scott You can find us on YouTube,
45:08 Scott LinkedIn,
45:09 Scott Instagram,
45:10 Scott Facebook,
45:10 Scott ThreadsX,
45:11 Scott TikTok and Patreon or at our website at HowManyCTOsPod.
45:15 Scott pod.com and remember to rate, review,
45:18 Scott and subscribe on the podcast platform of your choice.
45:21 Scott Thanks for listening!