Beyond Cost Reduction: AI in Revenue Generation and Innovation with Andy Boettcher

0:00 Scott: It's a curious way of always assuming you're wrong.

0:20 Brad: Welcome,

0:20 everybody,

0:21 to another episode of the How Many CTOs Does It Take podcast.

0:24 My name is Brad Hefta-Gaub,

0:26 and this is my trusty co-host.

0:28 Scott: Scott Porad,

0:29 Brad,

0:30 how are you doing?

0:31 Brad: I'm doing really well today.

0:32 I'm excited.

0:33 We have a guest with us,

0:35 Andy Boettcher.

0:38 Andy,

0:38 would you like to introduce yourself?

0:40 Maybe tell us a little bit about what brought you here.

0:44 Not to the pod exactly,

0:46 but like what, you know?

0:47 What brought you to this point in your career and give the listeners a little bit of intro about who you are.

0:52 Scott: All right, will do.

0:54 Brad,

0:54 Scott,

0:54 thanks for having me on today,

0:56 guys.

0:56 Just to preface so I don't go too deep, I've got grandkids, so I'll keep the story somewhat to a dull roar.

1:05 Elevator pitch, I usually say,

1:07 so I've been doing this stuff for 33 years.

1:09 I started out system side of the house, went to development, did all kinds of.

1:12 kinds of things i did salesforce for 13 years and i looked back one day right around where chat gpt dropped back in december of i think that was 22 when it dropped and i said you know everything i've done in my career has to do with data and i'm an engineer by trade but figured out that i can actually talk to people or people said i can talk to people's more of it right and uh you know doing some selling over the years and

1:38 Someone said,

1:38 you know, Andy,

1:39 you should go talk to people. You have a unique perspective on this data and AI stuff,

1:43 specifically how actually to make it work and make a complex and a weird topic less weird.

1:50 So that's how I kind of started my gig.

1:53 I started going into data and AI as a full-time deal back in 2023, and that kind of leads us here to 2026 with a couple of years under my belt.

2:03 developed in a couple scars and a

2:05 Brad: Yeah.

2:06 Scott: couple successes and a couple points of view so again brad scott happy to be here very much looking forward to our conversation today love

2:15 Brad: Okay,

2:15 I have a question about your career history.

2:17 Scott: oh boy

2:19 Brad: You know, before our call, I did look at your LinkedIn and I saw you worked for a company called Micro Estimating Systems.

2:26 Scott: Yeah.

2:26 Andy: And we have talked a lot about estimation on this show,

2:30 Scott: Yep.

2:30 Andy: and I am dying to know what micro, there's actually, what is micro estimating systems?

2:35 Do they help us estimate?

2:36 Scott: Oh,

2:37 that is an interesting story.

2:41 Micro estimating systems was a machine shop estimation software.

2:45 Andy: Okay,

2:46 Scott: So

2:46 Andy: interesting.

2:46 Scott: you think of CAD drawings, you think of machine shops.

2:49 I mean,

2:50 I live in Minnesota. We have mining and manufacturing.

2:53 manufacturing is our history up here right so everyone in their career at some point gets into something to do with manufacturing i happen to get into manufacturing technology and manufacturers they run down to the fifth precision of a decimal for profit

3:07 Brad: Yeah,

3:08 Scott: right

3:08 Brad: totally.

3:08 Scott: like is electricity labor cost you pretty much name it the the grease in the machine in the cnc machine that software actually uh was able to predict

3:20 at different volumes down to five or six precision points of a cent of how much parts would cost to run your shop.

3:27 So I was a CTO there at the time. Technic, I think I was VP of technology, but it was a small company. So, you know, I'm not a title guy.

3:37 But yeah,

3:37 I was software guy there.

3:39 I picked up where the previous software guy picked up and or left off and built some pretty cool software.

3:45 Really brought to Scott,

3:47 you had mentioned something earlier,

3:49 as we were introduced many,

3:50 many, many hours ago here,

3:53 that running 2026 stuff on 1990s hardware,

3:59 that was kind of microestimating systems at the time.

4:02 That's the whole story.

4:04 I think I need other beverages to really get into some of the depths of that.

4:07 But it was a really fun company to work with, but really teaches you the value of estimation.

4:12 information and the ability to pull that stuff together months ahead of time before you get to the point.

4:17 Andy: Are there? Okay,

4:18 now I have a serious question about this.

4:21 in the software business,

4:23 and our best days were maybe mediocre estimators. Is there things we can learn from those manufacturing operations when they're being so precise about estimation? Is there some lessons we can take from that?

4:34 Scott: Oh, that's a really good question,

4:36 Scott.

4:36 The biggest difference is you know your inputs in manufacturing.

4:44 Andy: Mm.

4:44 Scott: Your inputs rarely change.

4:46 You may have stainless steel versus plastic versus brass versus any amount of hard stuff you're feeding into the machine.

4:56 You know how much electricity costs.

4:57 You know how much your labor costs.

4:59 You know how much your maintenance fees are,

5:00 how often you have to change the bit,

5:02 which machines you go to.

5:03 Estimation becomes a lot easier when you control your inputs. We as software people.

5:10 Our inputs, yeah,

5:11 there's some system inputs,

5:13 right?

5:13 We know we get upstream data.

5:15 We know we have to push data downstream.

5:17 We know we have to hook to things.

5:19 Nowadays,

5:19 you know, we have AI which is built by design not to answer the same question twice,

5:24 but I digress.

5:26 Andy: Mm

5:26 Scott: But

5:26 Andy: -hmm.

5:26 Scott: our biggest input,

5:28 guys, is people.

5:30 Andy: Mm

5:30 Scott: And

5:31 Andy: hmm.

5:31 Scott: are people, I hesitate to use the word trustworthy,

5:36 that's not what I mean, but are people regular?

5:38 No.

5:38 So that's why on the best day we're mediocre estimators.

5:42 Brad: We can get into the big what's deterministic debate,

5:46 but people are definitely not.

5:48 Scott: People are definitely not.

5:52 Andy: Yeah,

5:52 that's that's pretty interesting,

5:54 isn't it?

5:55 Okay,

5:55 so if we could just fast forward through the career for a second,

5:58 what are you doing today?

5:58 What's going on these days?

6:00 Scott: Yeah, these days, so I'm here,

6:03 I'm one of the owners here and the Chief Innovation Officer for DoubleTrack.

6:08 DoubleTrack is a data and AI consultancy, and we like, or a boutique consulting firm that likes to tackle the weird icky sticky stuff.

6:16 Andy: Oh.

6:17 Scott: uh yeah you guys have you guys hear this all the time you hear companies you work with peers colleagues say our data sucks or we just bought this thing and it was supposed to fix it and it didn't and now it's sitting on the shelf and i have to pay for it for the next three years like you hear all the stories right i'm not quite sure you guys tell me here for a second how deep have you guys got into playing with ai use cases like real ai use

6:43 use cases when i say real we're actually trying to move a revenue needle for a company with ai not necessarily cost takeout but like we're trying to increase the revenue how far have you guys gone so far

6:54 Andy: That's a great question.

6:56 I think Brad's gone very deep on the software engineering side.

6:59 Brad: Yeah.

7:00 Andy: He's got a whole army of robots over there building stuff for him.

7:03 I've spent more time on our product side trying to figure out how we use.

7:08 use AI to help drive task rabbit business and our results have been fair to middling we're still learning how to make it work

7:17 Scott: Yeah, that's exactly it.

7:19 We're still trying to figure out how to make it work.

7:22 The best of us.

7:23 are still trying to figure out how to make it work i

7:25 Brad: Yeah.

7:26 Scott: chose to go into this full time to get more of that right we all got some crazy brains you know we all we all look like we've been around the block once or twice we have definitely a number of failures in our career that's what we learned from right not everything can be a success um i wanted to kind of take that forward into some interesting organizations and do exactly talking about

7:48 try things out adjust bring new technologies in apply some good information architecture to be able to really power ai and move it forward honoring the work we've done in the past but at the same time also realize that what we've been architecting for years doesn't cut the mustard anymore like it's a whole new world right so scott what i've been doing the past couple of years is you

8:11 trying to answer that question or at least get better at it on my best of days.

8:15 Brad: I was going to say, well, what are some examples that I mean, like, I agree,

8:17 I think I understand the thread that you're pulling on is a lot of the people talk about AI as sort of a cost reduction,

8:23 like automate things you have humans doing,

8:25 that's going to make things cheaper,

8:27 right?

8:27 And I guess you could argue that, you know, I am aggressively using AI and all the teams that I work with are all software engineering teams, and they all use AI.

8:36 And they've gotten,

8:37 you know, they're very much like.

8:40 using it as an acceleration right and so you can sort of say like well acceleration is a little bit like the cost reduction right or you're getting more stuff done faster so it would have taken you more people or more time to do that same amount of stuff so that's kind of in the cost reduction and i think the thread you're pulling on is okay so we got cost reduction ideas but what about actual like new sources of revenue ideas or increasing existing revenue stream ideas

9:08 So

9:08 Scott: Yep.

9:08 Brad: the top line, not the bottom line.

9:10 The top line, not the middle line.

9:12 Scott: Yeah.

9:13 So that's where I've been trying to play.

9:15 Like it's, I'll kind of walk my way into it. So Brad,

9:18 you said your teams use AI,

9:20 right?

9:20 So you have implemented this, you work with your team,

9:24 you help them understand how to use it,

9:26 you teach them how to use skills or you try to teach them how to use skills,

9:29 right?

9:30 Brad: Absolutely.

9:31 Scott: I've got a product that I try to put out because I try to do this to other companies as well.

9:36 Brad,

9:37 would you agree it's not the technical capability of that developer, it's the mindset,

9:41 right?

9:42 Brad: Oh, for sure.

9:43 Well,

9:44 I mean, I, yeah,

9:45 I mean, I guess I would have to.

9:47 we'd have to kind of like figure out what you mean by the technical ability of the developer but for sure it's the processes and the systems and the sort of mindset like in particular i just was talking to a engineer that's not on one of the teams i work with and they were adamant like i'm not going to ever use ai i'm all i'm better than the ai i write code better i don't need it and the teams i work with that are like using it it's like it they none of them would turn back at this point none of them would ever go back

10:14 ever go back they're like this is really really powerful um now i think they also happen to be really strong engineers to start with right like they they actually

10:23 Andy: Doesn't hurt.

10:24 Brad: right like in fact the ones that were strong engineers i think were the ones that embraced the ai fastest because they saw they saw the potential they saw how to control it and manage it and use it to let you know 10x themselves

10:39 Scott: So I'm going to kind of start walking into that question you asked me, right?

10:43 But that's where you have to start.

10:44 The only way that you can really start to activate this thing for revenue generation or new product ideas is to have a different mindset than we've had.

10:55 I'm not going to say it's easy for cost takeout because it's not,

10:58 right?

10:58 Like just for example, my organization, I actually measure the firm project to project with team members on AI.

11:06 guy tools how much more effective are we how much can i use that in a sales capacity to talk to my customers and say hey i am and i'll tell you guys i am at last check i think i just checked it last night thinking about this conversation i think i'm at like 28 i'm like a 0.28 multiplier forward so 1.28 faster faster

11:27 with ai and you hear like brad you said like 10x right we all say 10x 100x in little spurts we can be incredibly faster but if i look across an entire body of work i'm 28 faster that ain't too bad to shake a stick at right but you have to measure that stuff but that's all about cost takeout but you think about how we can use this how you use ai as a partner or a peer if i go into pure programming lingo it's a pure

11:53 peer programmer a lot of other engineers use this as a tool they think of it as a tool and they talk to it like a tool and they expect things out of it like a tool but you have to think about using it as a peer and bouncing your own ideas off of it and that's how you accelerate yourself right you don't accelerate yourself with a tool you accelerate yourself with a peer so you can take and turn that into generating massive skills libraries to take out the manual work you were doing to make things more

12:20 things more predictable to make things more regular coming out like i use it for all i have a brand kit for all of my documentation so any one of my organization is the same voice and the same look and feel on things that go out the door simple things like that that's cost takeout right but when you get to true revenue generation it's how can i use these things now to answer questions that somebody took a week to put together or they were

12:46 they deployed 100 people go out to figure this thing out you could look at that and think it's cost takeout but if you really think about the power of these tools i'll give you an example i'm working with a customer to ingest imaging imaging at scale and just do basic image image recognition which has been around for decades at this point it got better but using that to determine if they should sell one of their certain products into a certain

13:12 certain market and that's an example it's just one very basic example of revenue lift I can go push in that area without even do traditional traditional market research or things like that I could go on I've got a list of stuff that we've done for revenue generation but the only way you can unlock some of these is by having a different way to look at this stuff and really look at it as a peer and an accelerator versus a cost takeout tool

13:35 Andy: If you were to describe what that new or different mindset.

13:39 mindset is compared to the old mindset like i think you know well actually i'm not i was about to shape that answer i just think that question there how would you describe the new mindset

13:48 Scott: It's a curious way of always assuming you're wrong, right?

13:54 I mean, I'm crawling up on 50.

13:56 I've done a bunch of stuff and I ain't too shabby without toot my horn,

14:00 right?

14:02 But at the same time, a lot of these tools help me think differently and I have to be open minded.

14:09 I need to be able to ask open ended questions. I need it to ask open ended questions of me and I need to like have it help me think differently.

14:19 But really, Scott,

14:21 it's it's just raw curiosity and having an open enough mind to realize that what I've done for.

14:27 the last 33 years is great from a base skill standpoint but i can do something different it's amazing the individuals that that are there they they are really good they are so good but brad i'll go back to your your example of one of one of the other team members on a different team he had he's like nope i am really good at this and i'm going to do this and i don't need that to help me

14:50 That person will be replaced by somebody who uses these tools.

14:54 Brad: Yeah.

14:55 Scott: To be very blunt,

14:56 that's how it's going to happen.

14:58 Brad: Yeah.

14:58 You know, when I'm when I'm pondering as I hear you talk about this is that I'll acknowledge like I'm I'm pretty AI pilled at this point, like the people around me in my professional career are all.

15:12 pretty all in on

15:14 Andy: Mm

15:14 Brad: AI.

15:14 Andy: -hmm.

15:14 Brad: So I was a little surprised when I talked to this engineer that was like, I ain't going to use it. And but I wonder maybe what you're speaking to,

15:24 Andy,

15:25 is that what you and your team do is help people who aren't necessarily yet AI pilled understand.

15:35 how to think differently, but maybe I'm misreading what you're saying.

15:40 Scott: No, you're pretty close, Brad.

15:42 It's even people who think they're AI-pilled.

15:44 Right?

15:45 I've run into people who think they're wizards with chat GPT, and they're making really cool documents.

15:51 They're doing some baseline coding.

15:54 In their mind, they're like, this is awesome.

15:56 This has changed my life. And I'm like,

15:57 oh, wait.

15:59 Check out this, check out that.

16:00 Look at this different way to do it. There's this different tooling.

16:03 Have you asked it these types of questions?

16:05 The leveling that AI pilling up is great. I'll give you a real personal example of mine.

16:11 So my youngest daughter,

16:13 she's almost 18.

16:14 Right. And she's trying to figure out what she wants to do with a career.

16:18 I've started to introduce her into this. And Brad,

16:21 your statement about my daughter is definitely not AI-pilled, even though she grew up in technology and she's now,

16:29 I mean.

16:30 She's spent the last half of her teen years in an AI-enabled world.

16:35 She is not AI-pilled yet.

16:37 A lot of the things that people fall back on is, you know, take the weird things about like water and electricity,

16:42 which aren't not real, but for the sake of this conversation,

16:46 we're going to park all that and say, you know, my daughter is an artist and she says,

16:50 hey,

16:51 I don't like AI because it takes the human creativity out.

16:55 So I am against it, which,

16:58 by the way, in that specific lens,

17:00 I think the human mind is one of the most creative things in the entire galaxy.

17:07 showing her how these tools can help with different things other than artistic creativity has opened her eyes to see that oh maybe this isn't so bad it's really fun to see somebody's eyes light up with all this new tooling and learn how they can literally change their life and make a better living and help those around them and help companies make more money and all that kind of stuff

17:29 Andy: You know, I think one of the things you were saying about mindset that really resonated with me.

17:33 It would be kind of fit with what you were also saying about having not thinking of the AI as a tool, though it is a tool, but thinking of it as a partner to talk to.

17:45 Is learning how to talk to it differently.

17:49 I just had an experience the other day where I almost went into blog and said, I want to write an app to do this.

18:00 And I caught myself and I said, this is the problem I'm having.

18:07 you know this is this is what i'm trying to accomplish and it responded as like well turns out and i said i'm thinking i said i think i must have said i'm thinking of writing an app let me tell you can you help me with that this is the problem i'm having this and i already had my mind how i wanted to do it but i don't know what made me think of catching myself and framing it differently and and it said well turns out there's already a few apps in the app store to do that you don't even have to write this app

18:32 this app sidebar

18:33 Brad: Mm-hmm.

18:34 Andy: and this is a perfect AI story neither of those apps actually did what I want to do so now I'm back to square one however I did it was it didn't and it did resonate with me like oh interesting you have to you have to learn a different way of talking to the AI

18:57 especially if you're like have been in engineering for a while and you have a mindset of like oh I've got this problem I'm going to write an app to solve that problem hey Claude can you help me write an app I needed to do this and I wanted to have this back end I wanted to be and I was already thinking about that I was like do I want to do this on a mobile app or should be able to think about all those things right I'll tell you guys what I was doing I've got like a thousand CDs

19:21 From the old days, from back when we put music on a physical thing, and I'm going to get rid of them.

19:28 But I thought, I'm going to make one giant Spotify playlist of like all the CDs that I have.

19:35 But going in and doing that manually would take a really long time. So I thought maybe I can just take a picture,

19:40 use the camera on my phone, just have it look at the album cover,

19:43 figure out what it is, and then add it to a Spotify playlist.

19:47 And that's what I was trying to do.

19:49 And apparently if I had the barcode for all of those CDs,

19:54 it could do that.

19:56 But I made a stupid choice.

19:57 I wanted to get rid of these CDs,

19:59 so I got rid of them.

19:59 Scott: of them all but you know i saved the little i don't you call it it isn't an album cover you know it's the little thing in there that you can open it up or the insert i saved all the inserts i've got a shoebox full of inserts right now and i wish i'd saved all the cds because then i would have been able to just scan those barcodes and there's two apps out there where you can scan barcodes to do that anyway but my main point is i've gotten way of course my main point is what you're saying about the mindset what you're saying about treating it like a partner and how you talk to it

20:27 I do think is something we have to learn, and it's closer to product management.

20:32 We always talk about engineers,

20:34 product managers just tell us the problem and we'll figure out the solution.

20:36 It's closer to product management than engineering,

20:40 I think.

20:41 That's my,

20:42 now I'm getting off my soapbox now,

20:43 but what you're saying,

20:44 what you're lying down and picking up is what I'm trying to say,

20:46 Andy.

20:46 Brad: No, Skye, you're singing the right tune,

20:49 man,

20:49 and I'll add a layer.

20:51 So you're talking single-threaded.

20:54 You pick up, I'm a product manager.

20:56 This is what I'm trying.

20:57 Like you may, you know,

20:58 a lot of these answers,

20:59 but you're trying to give the AI session context.

21:02 Scott: Mm-hmm.

21:02 Brad: Right.

21:03 And like I'm a clod guy.

21:05 Right. So that's my frontier model of choice.

21:08 So I'm sitting mostly in Opus 5 most of these long and my learning with talking to Opus is having a document along the way so it doesn't just fill the context window.

21:19 he learned how to talk to it right but when you get into once you get past that overall architecture and ideation and you put your outcomes in and you start to stitch this stuff forward you start to build and you build with with agents and networks of agents and sub agents and all of that um i did a podcast a while ago about kind of the chaotic brain i was also always a managed service guy kind of growing up

21:46 growing up I started to manage service IT business in my past and in that type of world you're not just working on one project you're working on multiple projects and multiple customers right and you're context switching darn near constantly which right this is not a not a judgment call of anybody some people can context switch fast some can't

22:08 Right.

22:08 My mind is a way where I can't be on a single thing because my brain automatically starts to wander and look for the next thing. And my work product suffers when I'm on one thing. My work product is awesome when I'm on like 12 things because I can jump around.

22:23 It's the same type of deal when you get deep into building with these things.

22:27 Your brain is now tracking not just one context window of maybe a fable architect that is leading a bunch of opus and sonnet subagents. I have multiple tabs open.

22:42 And I'm running multiple orchestrators with multiple levels of agents and I'm all working on the same code base and different work trees and things like that and jumping around between them all and helping direct.

22:53 That is where your real accelerator comes in. But it takes, again,

22:57 a bit of a crazy brain to be able to track all of that together as it's going because it doesn't take a break.

23:04 Scott: Yeah.

23:05 Andy: It's funny you're saying that. For a while,

23:07 I've been saying that traditional software engineers who use agents need to start thinking like engineering managers,

23:15 okay?

23:16 Or in other words, they're thinking about like...

23:19 I've got a team of engineers.

23:21 I know what their capabilities are.

23:23 I know how to give them well-described problems.

23:26 I know how to check on them at appropriate times and determine if they're getting off track and to redirect them if they're off track.

23:33 Right. That's the job. That's effectively the job of an engineering manager.

23:36 Brad: Mm

23:37 Andy: And

23:37 Brad: -hmm.

23:37 Andy: I think that, you know.

23:39 The next layer up is,

23:41 oh,

23:42 I'm now not just running a team of agents, but I'm actually running teams of agent managers managing agents.

23:51 Well, now you're now you're kind of going up to that director of engineering level or that VP of engineering level, you know, or that CTO level where you're actually you got to be able to step back far enough. You got to have systems and processes in place for reporting.

24:05 guarding guardrails etc to be like yes i can trust this team to move and when i go back to it when i context switch back to it it will you know either be on track

24:17 Or not be so far off track that I won't be able to redirect it.

24:19 Brad: But we all know reality comes in play with this stuff, and I'll guarantee you 30% of your agents have completely gone crazy.

24:26 Like they've they're doing stuff that is not aligned and you have to be able to pop in and correct them ahead of time as there, you know, there was a term somebody told me one time it's like they called it cyber shoring.

24:39 Right.

24:39 Scott: Hmm

24:40 Brad: I've got an entire team of like traditionally I'd have a whole team of engineers who are working on things and they're working at the human speed and which is not slow,

24:49 but it's not.

24:51 agentic fast right

24:52 Andy: Yeah.

24:52 Brad: but now you have to be watching these teams and have other agents that are also watching those teams and putting testing frameworks in place and things like that you're doing it all out of your vision which frankly can be exhausting

25:08 Andy: Yeah,

25:08 totally.

25:09 Scott: Brett, can I go back to something earlier?

25:10 You mentioned, Andy,

25:11 that you've been working with teams on the revenue side of the equation.

25:16 And I think there's a lot of people that would be very interested.

25:17 interesting in hearing about that if you can share with us without you know sharing anything proprietary or anything that you're not allowed to share for your clients I think there's a I think there's a lot of people trying to figure that out and I'd love to hear you know success story what's what's working what's uh you know maybe what's not working but like I think there's a lot of people trying to crack that nut and would love to hear what you've learned so far you

25:42 Brad: Yeah, I will.

25:44 I'll share a pattern.

25:45 You're right. I can't get into too many details on too many specifics,

25:48 but I can share a pattern.

25:50 You guys both know of things like all the name of the software just escaped me. It's a green alligator.

25:57 It'll come to me in a second, but it's for basically CX, right? It's CX management.

26:02 And part of that is understanding the intentions and temperatures of your.

26:09 of your customers if you go if you follow that follow that rabbit down the rabbit hole you start to get into in SAS companies or product companies get into renewals you get into upselling you get to attrition management you get to those type of areas in there right and trying to understand the customers you have and a lot of software packages over the years have used things like

26:33 like call recordings and other data points to be able to start to feel temperature and but the interesting part of it is none of that stuff is truly real time

26:46 Right before the advents of AI,

26:48 it was, you know, I wouldn't even call it batch.

26:52 Like if you go like, you know, real time near real time batch and time scale,

26:56 right?

26:56 It's somewhere between near real time and batch, more toward the batch side,

27:01 where you're catching things a day or two or the next time somebody runs a report or something.

27:06 But it's nowhere near real time.

27:09 One of the patterns that I'm chasing with my customers right now is I call it.

27:13 I call it scraping the corners.

27:15 It is near real time, if not real time while calls are going on,

27:19 understanding product usage,

27:21 understanding inflection of voice,

27:24 looking at payment patterns,

27:26 looking at even all the way out to company like real time company research.

27:31 Are they filing warrant notices?

27:32 Are they doing Edgar filings? Are they an M&A talks with somebody? It's understanding.

27:37 All of that data at scale to better enable Brad sitting in that renewal seat to say, hey, you know what?

27:45 I heard you're doing this over here. Do you ever think about this product we have?

27:52 not having that direct conversation in the fly but taking it back to your product team and saying hey I got a couple customers that are falling into this it's a product to be able to build but

27:59 Scott: Mm-hmm.

27:59 Brad: the amount of computations and logic leaps that you have to take with that was never available until AI became commercially viable in December of 22.

28:12 Scott: Yeah,

28:12 I talked to another CTO at a pretty big company and they were doing, yeah,

28:17 inline, real-time cross-sell.

28:22 for their CX agent because they were listening to the audio.

28:24 Remember,

28:24 Brad,

28:25 we had someone on here who was doing

28:26 Brad: Yep.

28:26 Scott: voice.

28:26 They were doing voice masks.

28:28 That was made a little bit different.

28:29 These guys are listening to the audio.

28:31 They know the account because the customer is right there.

28:33 And so these are people calling. They basically turned their customer support center into a profit center from a cost center because basically people were calling in with problems. You know, they were hopefully able to fix the problem,

28:46 but in real time, then it was able to recommend.

28:47 recommend here's what you should try to cross sell to this person or upsell to this person and it was working and I thought that was pretty cool there's probably a lot of uses that could besides just upselling there's probably a lot of uses for that real-time listen to the audio and recommend like right in the person's ear hey this is what you should say right now

29:07 Brad: The little crazy big brother,

29:09 Scott,

29:09 right? It's, it's, I mean, there,

29:11 there's a.

29:12 Ah,

29:12 this, I'm going to use the wrong word, but take my intention out the word. There's a bit, I mean, we're playing with a little bit of ethics, right?

29:19 We're playing with a little bit of ethics, a little bit of appropriateness, a little bit of humanist, right?

29:24 About arming individuals with that type of knowledge that fast in a very vulnerable moment that somebody is sharing something with us and trying to take advantage of that.

29:34 But there is a real part of that when you, when you.

29:38 correctly and ethically apply it it's it's incredible and the i mean the human brain could probably do this i bet you can we've all i tongue in cheek will say y'all seen the movie lucy right you

29:52 Scott: I haven't.

29:53 Brad: haven't oh uh it it's totally far-fetched super science fiction but watch it it's it's interesting and uh i mean it gets a little

30:03 go a little uh far-fetched at the end but it's an interesting parallel to how ai is growing today it really is um so it's got scarlett johansson in it you can look it up it's it's actually you know i like those type of movies i like those anyway um but yeah it's it's like this cx place and frankly i'm a guy who when i tell this to my prospects i don't just talk

30:25 part of possible if i'm telling you something it's because either i've done it i'm doing it or i know somebody really close to me that has and i see it actually work a lot of the crazy stuff i go into the market with and a lot of the things that are talking about i'm actually dog fooding to myself like the cx piece of listening to calls my delivery team is they're wonderful people they are i i

30:52 love the problem solving the customer relationship building like my staff is amazing and they're in the moment and they're solving problems Scott right they're calling it like we're talking to somebody somebody's telling us their problems we're trying to help them work through their problems we're proposing solutions and we're in the moment but they don't always catch everything

31:14 So, you know, we just got done talking about the agents of agents of sub agents and that whole kind of piece.

31:20 Now you introduce a synchronous agents that are always out here doing something and informing you on the side to

31:26 Andy: Sure.

31:27 Brad: be able to have you start with something. I have asynchronous agents that are going to review those and saying,

31:31 hey, I did all this other research.

31:33 There's an angle here.

31:34 And it's, you know, the idea,

31:37 my personal idea,

31:39 this is where my humanist piece comes in.

31:40 like i'm always trying to help somebody yes i make money for what i do right but i'm always trying to start start the conversation with i'm trying to help you out if you see value in it let's do some business together and if you don't shake your hand and you let me know when i can help you out again right but like using those asynchronous agents to do that stuff it's real it's working i've i've experienced it

32:00 Andy: Yeah,

32:01 I was going to say that it's interesting,

32:02 like I think that.

32:04 You know, one of the things that you just touched on is clearly like the.

32:08 you know what level of use of ai are you and when you're using your when your agents are running essentially and and looking for work to do like when they know how to do a job and you set them up in a way that they are able to look for opportunities to do that job then that's like that's like the next level right it's not i'm not prompting the agent to do something for me i've got it set up and it knows its job and it knows how to

32:35 was how to go do it and in software like you know i i've been writing software for you know been writing code for more than 40 years so like i can think of all those things that are jobs that yeah i'm going to tell you how to do the job and you just go there and do it in the middle of the night for me it'd be great you know simple examples like you know go improve test coverage on every single line of code like every line of code needs to have test coverage on it you know um or

33:00 go you know replace i guess there was just some press release that came out um uh about atlassian and open ai crowing about the fact that some major test harness upgrade they did you know that was estimated five years of work they did it in two weeks or something like that you know now

33:20 I rolled my eyes a little bit about like, really,

33:22 five years of work.

33:23 I'm pretty sure the person that said that overestimated. But the point is that there's always these opportunities out there that are and it's this interesting question about like, you know, your point about cost reduction versus revenue generation.

33:37 every job is doing something right you're doing something in that job and hopefully you know that job is leading towards the success of the business that the job is part of right and so in some sense hopefully we're all part of like building towards the success of the business and yeah like i said in in software engineering i can just think of things like every crash that ever happens in the software

34:01 You know, you got your you got your real user monitoring, you got your observability running,

34:05 just have an agent that's like every time a crash happens.

34:09 It goes and looks at the code and figures out a bug fix and then boom, that crash doesn't happen again.

34:13 And that's like one less unhappy customer,

34:16 right?

34:16 I just saw a demo of a guy that's, you know, he's a small indie developer. He's like building casual games and he's got all his agents set up so that like any customer service complaint,

34:27 any bug report,

34:29 any one star review,

34:31 his agents are like scraping that stuff and coming back with solutions too.

34:36 To address it, you

34:37 Brad: Yep.

34:37 Andy: know, and he wakes up in the morning and he sees pull requests to fix the bugs that were mentioned in the in the, you know.

34:45 in the customer service ticket.

34:46 Brad: Now, Brad,

34:47 you touch on something very,

34:49 very important to me.

34:50 I mentioned to you guys a while ago,

34:52 I think I said I was 28% was my lift on using AI,

34:58 right?

34:58 And Brad,

34:59 right before that,

35:00 you said, oh, 10x lift,

35:01 right?

35:02 And you hear all these stories and, you know, you kind of cut them in half and that's really where the truth is,

35:07 right? Part of this stuff, when I first started playing,

35:10 I was sending.

35:11 sending agents off into the ether to do crazy things right like all kinds of different things we all did that when we first started playing with it like go do the thing and it comes back and gives you a login prompt trust is hard to earn and easy to burn right and those of us that have been playing with this whether you have been confronted with it directly or indirectly

35:36 AI is starting to erode some of that trust,

35:40 right?

35:41 You guys go on LinkedIn. You guys looked at LinkedIn,

35:43 right?

35:43 Like, Scott,

35:44 you said you Google stalked me before I came in here,

35:46 right?

35:46 Obviously, you went to my LinkedIn and looked at that, right?

35:49 Scott: Mm-hmm.

35:49 Brad: What is the LinkedIn feed full of now?

35:53 AI slop.

35:54 from people who would write good articles in the past but because we're trying to there's so much noise out there we're trying to compete with the noise there's some trust that gets lost because you write something and you clearly see a bunch of AI flags in it you know somebody didn't review the whole thing we're all guilty of it I mean humans are lazy by nature that's what we do right so you know we put this stuff out there and you start to burn that trust

36:20 I made a pretty hard cut and decision probably about six months ago, kind of around when when Anthropic finally figured out their go to market and put together an actual suite that somebody can use and more than like one facet using my phone,

36:36 multiple machines like when they figured it out pretty well. Right.

36:40 But the human in the middle stuff is.

36:44 Given a lot of lip service,

36:46 but...

36:48 I'll tell you, once you embrace it and you force it,

36:51 Brad,

36:52 the statement you said that got me on this path was your buddy who's doing the different gaming stuff and the one star reviews,

36:58 there's a bunch of PR sitting for him in the morning.

37:01 Does he have an agent that put them in?

37:02 No,

37:03 because he wants to make sure that they're right and he wants to make sure they go in with a full amount of data fidelity and customer fidelity and trust,

37:12 right?

37:13 That is something that I always try to weed through all of the stories is what is somebody just shoving stuff out for a cool stat to talk about versus have you done something that or that have you used have used AI to deliver something that has increased the amount of trust that somebody has in you that is a powerful piece and that's why I'm only at a 28.

37:38 only eight percent lift you know you learn how to do some of that stuff better and eventually i'll get my lift to go up but um i mean the first first couple days of it i was ripping stuff out and i look i look back on some of the deliverables i did and i cringe i'm like oh my god i let that out the door as i was super excited at the time

37:58 Andy: Yeah, but I,

37:59 you know, it's interesting you say that,

38:01 Andy, because I...

38:02 I agree.

38:03 First of all, I agree with the idea that like you don't have systems to do quality gates and verification and validation like that, I think, is required.

38:13 I actually think that one of the things that we should be learning from the agents is that it turns out that just have another just have a human read it or just have a human sign their name to it.

38:30 That's actually not a very good quality gate.

38:35 Brad: Not

38:35 Andy: And

38:35 Brad: by itself,

38:35 Andy: no,

38:36 Brad: no.

38:36 Andy: right?

38:37 That's not a good quality gate,

38:38 right?

38:39 And I think that I have a feeling that a lot of the energy around,

38:45 I mean, yes,

38:46 it's because we're trying to move faster, but I sometimes feel like I wonder if the reason that people are getting bit by

38:54 I'll use the vernacular AI slop.

38:57 Brad: Mm-hmm.

38:57 Andy: Is it actually more that like, well,

39:01 they would have written it poorly themselves anyway,

39:06 even if they didn't have the AI and they would have put their name on it and put it out anyway.

39:11 Right.

39:12 And so before we had AI slop,

39:14 we had.

39:16 humans

39:16 Scott: Slop.

39:16 Andy: we had slop you know and you know and so when i think about that with software it's like there are there are there are things that we do as software engineers that make a huge difference in the quality and yeah we all do code reviews we all have human code reviews and you know what i'm i've said it before i'm gonna say it again that's not what's causing that's not the actual thing that gets us quality

39:42 Like it's a it's a ritual we do,

39:44 but if anybody just is a, you know, a green check clicker, you know, they're just click and approve,

39:51 then you're not getting that quality.

39:53 It's do you have automated tests? Do you have CICD that prevents the clicking,

39:58 you know, if it.

39:59 Scott: fails right um that that's that's really where i think and then and then you got the quality on the other side like is it even the thing that we even built was it even something that was useful to build right that that takes quality gates as well

40:15 I can see

40:16 Brad: You

40:16 Scott: your crinkling your brow a bit there, Scott.

40:19 Is it possible we have an argument on our hands?

40:23 Brad: We do not have an argument. You've made me think about something differently than I thought about before.

40:28 Okay,

40:29 so I, to use Andy's phrase,

40:33 toot my horn, I'm not trying to toot my horn, but I was early in my career, I was a big proponent of everything gets code reviewed.

40:41 And I remember having arguments with people about this and they would say stuff like, it's just a one liner.

40:47 And I'm like, then it won't take that long to code review.

40:52 Scott: we already spent more time arguing about whether you would do the code review than if you just do the code review

40:56 Brad: And the argument I would always make is we're going to have a policy. And when I got to the point where I could set rules for an organization, I set a rule which said.

41:05 Everything will have a code review.

41:07 Because if it's big,

41:08 it's going to need a code review.

41:10 And if it's small,

41:12 it only take a second and the place where we get in trouble is the stuff that's in the middle where we make the judgment call okay so now the industry's moved along and that's kind of become a standard thing and what you got me to thinking Brad the reason I was furrowing my brow a minute ago or whatever because I was thinking huh maybe I should put in a new rule which is the opposite rule

41:35 No human code reviews allowed anymore.

41:37 You can only have agents review this code.

41:39 Scott: By the way, I will tell you right now in the teams that I'm the team that is going the fastest that is producing the highest quality code at a phenomenal rate they.

41:50 don't they have humans click the button to have their personal agent go do the code review but

41:56 Brad: Mm-hmm.

41:57 Scott: the but they are they're not looking at the code anymore they're they're look they're looking at the reports from the agents they're looking at what the agent says about the code what is what did what did the five adversarial agents because like scott writes code andy brad bob and janet all send their adversarial agents at scott's code you

42:17 And Brad's accountable for his adversarial agents take on it and reading it and going,

42:22 yep, I agree with my adversarial agent.

42:24 I approve.

42:24 Right.

42:25 Andy's accountable for his adversarial agent.

42:27 But I don't think Brad and Andy are reading the code anymore.

42:30 Brad: And if they have if the agent says,

42:33 well, this isn't good or something like that,

42:35 how does that just very practically, does that just get put in a in a pull request comment?

42:39 Like, how does that actually what's the feedback loop like there?

42:42 Andy: I mean,

42:42 Brett,

42:43 I.

42:43 Brad: Yes and no.

42:44 So my personal view on this, we've all seen the agents come back and say exactly that. I don't agree with this. We should do it differently.

42:53 The question of why?

42:56 Andy: You know, I will challenge it and make it produce

42:59 Scott: Oh, absolutely.

42:59 Andy: evidence.

43:01 Scott: Yeah,

43:01 Andy: And

43:01 Scott: it's got to

43:01 Andy: then,

43:01 Scott: give you

43:02 Andy: yes,

43:02 Scott: a good reason.

43:02 Andy: and then Scott, that goes in the poll request,

43:05 right?

43:06 That evidence to be able to have the person after you or the room next door be able to look at that without context and say, yeah, Andy made a good decision here.

43:15 Scott: Yeah, that's what I'm saying. Yeah,

43:17 I'll just let me echo that as well.

43:19 So the.

43:21 I mean, it's a shame since GitHub has struggled so much lately. It's like, oh my God,

43:25 am I going to have to move to GitLab or something,

43:27 you know, but the reality is that artifact,

43:30 it's the GitHub pull request. It's the comments in the GitHub pull request.

43:34 That's like the most important artifact at this point.

43:36 Like that's where the agents report their findings in telemetry. That's where the agent goes and does some telemetry and comes back and goes, I see exactly what happened to this customer,

43:45 blah, blah, blah, blah, you know, and it's all documented in.

43:48 a github issue that then is linked to the github pull request and then the code review that happens has inline comments that says oh you uh didn't include the uh you should have used a finally block here because the logging you know is not going to get logged in the case of a error being thrown or something like that you know like you got to explain you know it's not just like change this line it's like it's got to have like clear explanation and those are all these nice little artifacts

44:15 facts that the humans when they look at it can go and and again this is as if if i'm if i'm a vp of engineering of at a hundred person team 100 engineering team how what's my quality guards at this point it's like let me go do a spot check i'm gonna go just do a spot check of the last 15 prs that were merged what is the what is the conversation that's happening between these engineers are they taking their job seriously right are they

44:42 really thinking critically about the code that they're reviewing and they're approving.

44:49 And to me, the artifact is GitHub.

44:51 Andy: But let me throw a curveball in here. I think, like Scott,

44:55 this might actually speak into where your brain's thinking here too.

45:01 These tools are pretty smart,

45:02 right?

45:04 One thing that I did learn probably in January, started putting in practice is all of my skills, all of my reviewers,

45:12 all of this stuff, you know, you go and say review my code and look for best practices, right?

45:16 And you let it do its thing, right?

45:19 I have introduced intentional scoring rubrics.

45:24 in all of these that say you know on an A to F scale like you have to be an A to ship it doesn't have to be perfect but it has to produce the evidence that got me there in a way that my organization agrees the way it should be done and doesn't introduce other things every single one of I have 64 skills in my in my company's cloud library for different things and they every single one of them have a rubric

45:49 to allow something to pass and exit the agent or not.

45:54 That decision right there has helped me have trust in these different AI processes and agents and skills that are out there to put out a product that I can then expect somebody else to trust to put back in, not just blindly say,

46:10 hey, go do a code review and look for best practice stuff.

46:13 I make it.

46:15 create evidence to support it and that's what goes again scott that's what goes in the pr for

46:20 Brad: Mm hmm.

46:21 Andy: me pretty

46:22 Brad: And then I guess you just have another agent pick up all those things and fix it.

46:25 Andy: much

46:26 Brad: But I'm really I'm really pontificating this idea of actually,

46:29 should we put an opposite policy in place?

46:32 No human code reviews allowed anymore.

46:35 You know, it's all about with quality capes. I never even heard the word quality capes before like a year ago. And now it's all we talk about, you know, do we create the structures to force ourselves to actually create,

46:47 no pun intended,

46:49 quality,

46:50 quality capes.

46:51 Scott: Mm

46:51 Brad: as

46:51 Scott: hmm.

46:51 Brad: opposed to, you know, green checkmark on a PR and we think it's a quality gate.

46:57 Scott: Yeah.

46:58 Well,

46:58 and also,

46:58 I think that, you know, one of the you know, we talked a lot about this with Uncle Bob when he was on the pod and I think we've we've talked about it with other engineering leaders that, you know, there are a lot of really powerful deterministic tools that you can run, you know, like if you want you want to if you've got, you know, I.

47:16 I used to say this before AI was taking off when I had engineers on these teams where in their human code review,

47:24 they would be blocking on grammatical errors in comment blocks,

47:29 right?

47:30 Or they would block PRs on you didn't have the type doc header correctly,

47:35 right?

47:36 And Tabs tabs,

47:37 Andy: versus spaces.

47:38 Tabs versus spaces.

47:39 Scott: whatever,

47:39 right? And I'm like...

47:41 I do not want if you are blocking a PR based on anything like that, we're making it a lint rule.

47:47 We're going to have a deterministic tool that runs that is which is a pre-commit hook like you're wasting everybody's time by blocking a PR on something that can be checked by a deterministic tool.

48:00 Right.

48:01 And so just take it to the next level,

48:03 deterministic tools as much as possible. If you've got if you've got a definition of what quality is,

48:09 build a deterministic tool that that validates that and put that in your CICD pipeline. And guess what?

48:15 Your agents won't be able to commit either.

48:17 They have to trust they have to they have to get the pre hook or the pre commit hook run to,

48:23 you know.

48:25 Brad: Andy,

48:25 we kind of need to wrap.

48:26 up here, but I'd love to leave the final word to you.

48:30 As you have been working with all these companies and experimenting, you know, just trying a lot of new things,

48:36 learning about new things, if you were to,

48:38 you know, someone just getting started,

48:40 what's the words of wisdom you would offer them?

48:42 Andy: Have an open mind.

48:44 That's the best, that's the best thing I can say.

48:47 Ask Mm open-ended

48:48 Brad: -hmm.

48:48 Andy: questions,

48:48 have an open mind.

48:50 Let the thing help you form new neural pathways on how to use this stuff.

48:57 It's a, I haven't been this excited in my career, boys,

49:00 for decades.

49:02 Scott: Yeah.

49:02 Andy: The stuff that we're able to do now,

49:04 but you have to have an open mind and you have to experiment and you have to be open to fail.

49:11 Brad: Mm-hmm.

49:11 Andy: That's

49:11 Scott: Love the it.

49:12 Andy: thing I would tell people getting into this.

49:14 Scott: Sky's the limit.

49:15 Andy: Yep.

49:17 Brad: Love it. Reality is.

49:18 Scott: Well,

49:18 it was nice meeting you, Andy.

49:20 Thanks for taking the time to come on the pod.

49:23 It

49:24 Andy: Absolutely.

49:24 Scott: was great conversation.

49:24 Brad: Appreciate it.

49:25 Andy: I appreciate the invite.

49:26 Brad Scott,

49:27 this has been a fun time, guys.

49:28 I feel we could keep talking for hours on random other things. And you guys didn't even start me on jeeps. So there you go.

49:38 Brad: All right.

49:39 Well, thank you very much for everyone who's listening.

49:42 Smash the subscribe button,

49:44 rate,

49:44 review,

49:45 tell your mom.

49:45 Scott: Tell your mom.

49:47 She's going to love it.

49:48 All

49:48 Brad: Okay,

49:49 Scott: right.

49:49 Brad: bye everyone.

49:50 Scott: This episode of the How Many CTOs Does It Take podcast was produced and edited by Saffron Eptical.

49:56 Our theme song was AI generated by Brad.

49:59 Please follow us on all the social media platforms at HowManyCTOsPod.

50:05 You can find us on YouTube,

50:06 LinkedIn,

50:07 Instagram,

50:08 Facebook,

50:08 ThreadsX, TikTok,

50:10 and Patreon,

50:11 or at our website at HowManyCTOsPod.com.

50:14 Um, and remember to rate, review,

50:16 and subscribe on the podcast platform of your choice.

50:19 Thanks for listening.