Transcript: The State of AI in 2025: What the DORA Report Reveals

0:00 Brad These themes probably existed independent of AI assistance.

0:04 Brad These are also attributes of what high-performance teams have available to them.

0:27 Scott All right, Brad,

0:27 Scott welcome back to another episode of How Many CTOs Does It Take?

0:33 Scott And I'm very excited to actually interview you today because earlier this week you sent me a link to an article that you said you're really fired up about and

0:44 Brad Yeah

0:44 Scott contemporaneously.

0:49 Scott I received something similar on the same topic from another person, and I don't know what it is.

0:55 Scott I mean, I know the headline, which has something to do with Dora Metrics and AI,

0:59 Scott but I actually don't even think we've introduced the show. This is the How Many CTOs Does It Take podcast.

1:04 Scott My name is Scott Porad. This is my trusty co-host.

1:07 Brad I'm Brad Hefta-Gaub, and in this episode, we're going to talk about the newest Dora report on the state of AI.

1:16 Brad Hi,

1:16 Brad software development for 2025.

1:18 Scott before we start we got to take a quick look at hacker news how many of the top 10 articles are about ai one only one the

1:30 Scott ai hype is over all right dora metrics and ai let's hear about it tell me about this article that you found brad or this thing that's going on it's more than one article because i got two of them

1:41 Scott Something's happening out there.

1:42 Brad Well, so first let's start out. We all know what the Dora program is. It's essentially,

1:48 Brad you know.

1:51 Brad You know, it's a door is all about research program,

1:54 Brad it's kind of supported by Google and it's all about metrics for kind of measuring the overall productivity of your engineering team. Right.

2:01 Brad And so we talk about the door metrics are essentially like typically it's the, you know, how quickly you do releases, how frequently you're doing releases, how much downtime you have from releases, how quickly you repair any downtime. Those are kind of the classic.

2:15 Brad sector metrics that are out there and yeah so the Dora the Dora group working group basically went out and did a very very extensive research paper so it's essentially like a you know a think tank type you know analysis paper that you might get from a you know a consulting group or any kind of a working group and they looked at

2:38 Brad Kind of where are we at in adoption of AI?

2:43 Brad Where are we at in how that adoption is? Is it positively or negatively impacting team performance?

2:51 Brad What types of teams are being more or less successful with adopting AI?

2:55 Brad And it's an extensive report.

2:58 Brad So it's 142 page long report.

3:02 Scott And I see that into.

3:05 Scott chat GPT or Gemini and just get a summary.

3:08 Brad Probably.

3:09 Brad But they did a good job like all good reports.

3:12 Brad They lead with the executive summary and they kind of enumerate their key findings.

3:18 Brad And I think that's certainly one place that we could start is kind of like what are the key findings that they came with.

3:24 Scott Let's do it. Let's start wherever you want. What are the key findings that they came with?

3:27 Brad All right. So right out of the gate, one of the key findings is that AI adoption has become nearly universal.

3:35 Brad So basically, in the people they surveyed, 90% of respondents are saying they use AI as a consistent part of their workflow.

3:43 Brad And more than 80% believe that it has increased their productivity.

3:49 Scott I'm skeptical already.

3:51 Brad Well,

3:52 Brad there's a really interesting chart that I found on like page 32 of the report.

4:00 Brad And then I realized it was actually called out on page,

4:04 Brad you know, on that key findings part of the report, which is it's this awesome like the landscape of AI's impact.

4:11 Brad And I'm going to just read these sort of the vectors that they kind of got some results.

4:18 Brad results on,

4:18 Brad and you'll see something interesting in it. So vectors that AI impact,

4:23 Brad individual effectiveness.

4:25 Brad Okay.

4:26 Brad Top impact all the way far to the right.

4:31 Brad Yay.

4:31 Brad Everybody feels more productive.

4:34 Brad Next most impactful,

4:36 Brad next most highest impact.

4:41 Brad Software delivery instability.

4:46 Brad And it is increased with AI adoption.

4:49 Brad And the chart says,

4:51 Brad note,

4:51 Brad an increase here is not a desirable outcome.

4:57 Scott Right.

4:59 Brad uh organizational performance uh positive high impact uh valuable uh work positive high impact code quality positive high impact product performance positive high impact these are all kind of getting less and less impactful um team performance software delivery throughput and then burnout actually above zero so actually increasing burnout and then friction decreasing friction so basically of these various

5:25 Brad areas metrics on the impact the vast majority of them are generally considered positive if they are if ai is improving them and three of them would be considered negative if they were increasing and like the the second top increase is software delivery instability is being reported as a real actual outcome of ai

5:48 Scott Let's have an impact.

5:50 Brad it's having an impact uh not necessarily the one that we would want so um so that's kind of a really interesting so i i feel like that's certainly worth like the aha 90 of people are using it 80 of people saying they're more productive with it and um software instability delivery instability is being reported at a

6:15 Brad uh estimated effect standardized at 0.1 increase i don't know what the the unit of measure of that is would have to like go dig into that but clearly not a good thing well

6:30 Scott Yeah, I'm just,

6:31 Scott look,

6:32 Scott I'm going to trust the report that that's what it says, that it doesn't jive with my experience of the engineers I talk to on a regular basis.

6:40 Scott I see more. I just hear more skepticism. That's all.

6:43 Brad Actually, they've got some great charts on that. And they actually did something really interesting. And again,

6:48 Brad this is like 140 page report. And so there's there's probably something in there for anyone to find,

6:53 Brad right?

6:54 Brad Like what's the saying lies, damn lies and statistics,

6:56 Brad right?

6:57 Brad So we can make these statistics say whatever we want them to say. And at 142 pages of statistics,

7:02 Brad I'm sure any team could find something in there to support their particular.

7:06 Brad political agenda.

7:08 Brad One thing I do think is interesting is they did a really interesting deep dive on types of organizations and they categorize these types of organizations into seven different clusters of organizations and those different clusters of organizations were experiencing very different results.

7:28 Brad And so they have a whole chapter called understanding your software delivery.

7:31 Brad delivery performance.

7:31 Brad A look at seven team profiles.

7:34 Scott Mm

7:34 Brad And

7:34 Scott -hmm.

7:34 Brad they basically clustered these groups into,

7:41 Brad like I said, seven groups. So cluster one,

7:45 Brad foundational changes.

7:46 Brad These teams are stuck in survival mode,

7:48 Brad facing significant challenges with fundamental gaps in their processes,

7:51 Brad environments,

7:52 Brad and outcomes.

7:53 Brad And about 10% of their survey respondents were in that category.

7:56 Scott Mm-hmm.

7:57 Brad Cluster two,

7:58 Brad the legacy bottleneck. Teams in this cluster are a constant state of reaction where unstable systems dictate their work and undermine their morale.

8:06 Brad And this is about 11% of their respondents.

8:09 Brad Constrained by process,

8:11 Brad these teams are running on the treadmill despite working on stable systems. Their efforts are consumed in inefficient processes leading to high burnout and low impact.

8:20 Brad 17% of respondents were in that category.

8:26 Brad High impact low cadence. So these teams are producing high impact work reflected in strong product performance and high individual effectiveness.

8:33 Brad However,

8:33 Brad these are coupled with low cadence deliveries more model characterized by low software delivery throughput and high instability. So in other words.

8:43 Brad and seven percent are in this so these are sort of like they're making high impact but they're but they're kind of trudging along and from a dora perspective right the dora metrics are all about like fast iteration and fast you know delivery so this team would kind of score on a on a sort of like they might feel like they're making impact but they're kind of slow to move

9:04 Scott Yeah, well,

9:04 Scott they're aiming well and hitting their target.

9:08 Scott They just don't shoot very often.

9:10 Brad Yeah, exactly.

9:11 Brad And let's see, I'll rattle off the next three here.

9:14 Brad Stable and methodical teams working on steady artisans, delivering high quality code at a deliberate and sustainable pace.

9:21 Brad 15% of the respondents are there.

9:23 Brad Pragmatic performers team consistently delivering work with impressive speed and stability,

9:28 Brad even if their environment hasn't reached a state of peak engagement,

9:31 Brad 20% of teams there. And then the all hail greatest cluster here,

9:36 Brad the harmonious.

9:37 Brad Monious high achievers, this is what excellence looks like and something like 20% of the door responders are in that category and each of those different segments clearly are and this is one of the things of this door report that.

9:49 Brad It's not a tools question.

9:51 Brad AI, integrating AI into your development cycle is not a tools question.

9:56 Brad It's a systems level question.

9:58 Brad And you've got to set your systems up correctly. And systems, of course, we mean like all the systems, like, you know, your cultural systems, your process systems,

10:07 Brad your technology systems,

10:09 Brad right?

10:09 Brad So that's kind of one of the other key things that came out of this report.

10:13 Scott That is fascinating.

10:15 Scott You remember John Helm, who was on this podcast.

10:19 Brad Yeah

10:19 Scott He has a saying that I picked up from him 20 years ago,

10:24 Scott which to me is very relevant to that little piece right there,

10:28 Scott which is he always would say software is just a function of the people and processes who created it.

10:36 Scott And so what I hear.

10:40 Scott I would adapt it. I'll call it John Helm's AI corollary.

10:45 Scott AI effectiveness is only a function.

10:50 Scott of the people and processes in which it is embedded.

10:53 Brad Mm hmm.

10:54 Scott Yeah.

10:54 Brad Mm hmm. And and one of the this is this is this was its own key finding here.

11:00 Brad So successful adoption of AI requires more than just tools.

11:05 Brad Our new DORA AI capabilities model identifies seven foundational practices,

11:11 Brad including clear AI policy,

11:12 Brad a healthy data ecosystem and a user centric focus.

11:16 Brad that are proven to amplify the positive impact of AI on organizational performance.

11:21 Brad And so they have a whole chapter on sort of like the things they've found that the team's reporting high performance all had in common.

11:34 Scott Okay, that's handy.

11:36 Brad Yeah, those.

11:36 Scott Now, are those things they found,

11:38 Scott are they...

11:40 Scott tangible or are they fuzzy like

11:44 Brad Good

11:44 Scott are they things

11:44 Brad question.

11:44 Scott are they things like are they things like

11:47 Scott The team has a great culture,

11:49 Scott which is very fuzzy,

11:50 Scott or they very tangible like,

11:52 Scott oh, the team always,

11:54 Scott you know, has code reviews done within 20 minutes or making that

11:58 Brad Yeah,

11:58 Scott example up.

11:58 Brad yeah, that's great.

12:00 Brad Well,

12:01 Brad again,

12:01 Brad it's like a whole chapter in this 140-page report,

12:05 Brad so I haven't digested all of their recommendations,

12:07 Brad but let me rattle off the seven key themes that they identified.

12:13 Brad Clear and communicated AI stance.

12:16 Brad So a very clear stance in our organization.

12:18 Brad What is our stance on AI?

12:19 Scott Easy, okay.

12:21 Brad Working in small batches. I want to drill into that one. That one sounds like a theme that we've seen in all of our discussions and use of AI.

12:30 Brad Healthy data ecosystems.

12:33 Brad Not sure what that means, but I would imagine that their

12:37 Scott Exactly what

12:37 Brad data makes

12:38 Scott that means.

12:38 Brad all.

12:39 Brad Data, you know what that means?

12:41 Scott Well,

12:42 Scott I mean, I've worked in unhealthy data ecosystems.

12:45 Brad Okay.

12:47 Scott So, you know, if your data pipeline, what's your data ecosystem?

12:51 Scott At least in my world, I work in an online transaction business,

12:55 Scott right?

12:55 Scott Is your transactional data clean?

12:58 Scott Is it getting into, is the pipeline's functioning? Is there a semantic layer that's abstracting your raw data into data that people can do stuff with?

13:06 Scott with and then ultimately it's asking the question i think which is or not asking the question but it's saying look if if those parts aren't in place it's really hard to get your ai to do magic stuff because it's operating against the garbage that comes up

13:22 Brad Yeah, so they say that that's exactly what they say about health data systems and they say in our analysis,

13:29 Brad healthiness of a data ecosystem is measured as a single factor comprised of three individual indicators measuring respondents'

13:37 Brad perception of the overall quality of internal data sources,

13:40 Brad the accessibility of internal data sources,

13:43 Brad and the degree to which the internal data sources are siloed.

13:47 Brad load or divided from one another so

13:50 Scott Can we sidebar here for

13:51 Brad yeah

13:51 Scott a second?

13:51 Scott Can I give you all an example of an unclean data system?

13:54 Scott This happened to me within the last month.

13:58 Scott So an important thing to know is the company I work for has a fiscal year that ends September 1st, and then there's an audit thereafter.

14:06 Scott And for a hot minute, there was a panic because the auditors were saying.

14:13 Scott some very material number the revenue is off by some very material number from what we reported to what they were seeing like like you know in this business like you know like like a one two three kind of percent like a big number yeah

14:28 Brad That's a big number, yeah.

14:29 Scott well

14:32 Scott It turns out there was this operational workflow,

14:35 Scott which was some people, so,

14:37 Scott you know, we run a two-sided marketplace and sometimes there'd be a problem where I can't remember exactly the details.

14:42 Scott Let's just say that the worker wasn't getting paid on time.

14:45 Scott Maybe they weren't getting paid on time because the customer's credit card failed or something.

14:49 Scott I don't know the details.

14:50 Scott So Ops, in order to pay that worker,

14:54 Scott would create a second order.

14:56 Scott And so here's what I'm going to do. I'm going to create a second order for you. I'm going to mark this as approved and you'll get paid.

15:00 Scott And then eventually we'll collect the money from the first order and it'll all work out.

15:04 Scott But what it made it look like in our system is that there were two orders.

15:09 Scott And what happened is when the credit card actually ran or whatever,

15:13 Scott the second order,

15:14 Scott the first order looked like it got paid because the operator manually did it. And the second order got paid because the money actually came through. And so.

15:22 Scott So the auditors are like, well, wait a second, you look, it says you have too much money here. You don't have as much money as you say you have.

15:27 Scott And we ultimately tracked it down to that.

15:30 Scott So it was a great example.

15:31 Scott Oh,

15:32 Scott to me, it's an example of two things.

15:33 Scott The data wasn't clean.

15:34 Scott And

15:35 Brad Yep.

15:35 Scott also something I call, you know, I sometimes refer to as cry sourcefulness,

15:39 Scott which is you have operators just trying to be resourceful and try to solve a problem.

15:43 Brad Yeah, totally.

15:44 Scott And actually,

15:46 Scott this wasn't really an example of cry sourcefulness. We'll save that for another.

15:48 Scott for another podcast but they were just being resourceful and trying to solve a problem for customers and conceptually in their head they thought well the money's all going to work out it'll be fine but from an accounting perspective it didn't so that's an example of sort of unclean data

16:02 Brad Well, and to sort of go back, that makes a ton of sense. And to go back to that question of like,

16:07 Brad what are the what are these attributes that make you more likely to have productivity in your AI empowerment?

16:13 Brad And after healthy data ecosystems,

16:15 Brad another item that's listed is AI accessible internal data,

16:19 Brad right?

16:19 Scott Yeah,

16:19 Brad Like the AI needs to be able to access those healthy ecosystems of data, right?

16:23 Scott that's

16:23 Brad Like that's, that's,

16:24 Scott right.

16:24 Brad that's a key part. And, and, you know, I feel like I feel like this report.

16:29 Brad You know, I bet you if we went back in time, like to the 2021 DORA metrics reports,

16:35 Brad these themes probably existed independent of AI assistance.

16:41 Brad Like these are these are also attributes of what high performance teams have and have available to

16:47 Scott Well,

16:48 Brad them.

16:48 Scott you get teams that can effectively automate things.

16:52 Brad Yeah,

16:52 Brad exactly.

16:53 Scott Can I, before you go any further, can I sidebar my sidebar for one more second?

16:57 Brad Yeah,

16:57 Scott That particular problem I told you about,

16:59 Scott the sort of audit thing,

17:00 Scott it ended up getting resolved and it was fine.

17:03 Scott And it actually exposes another thing that drives me crazy.

17:06 Scott People are like,

17:07 Scott oh, there's a bug in the system, all the revenue's off and all the engineers are out there looking for the bug and nobody can find a bug,

17:12 Scott right?

17:12 Scott Because there was no bug, right? The system was working.

17:15 Scott Just there was like a bug kind of like it was like a people process kind of bug.

17:20 Scott But it's always this thing that drives me bananas. And I guess I just take it.

17:23 Scott take it personally like people start blaming us oh the system's buggy it's not working and then it sort of makes it way up and it's getting reported in the meetings oh there's a bug in the accounting system and that's why we had this thing that was off but we fix it like no no it wasn't a bug it wasn't

17:35 Brad Yeah,

17:36 Scott a bug it

17:36 Brad it's

17:37 Scott worked just fine

17:38 Brad so funny you say this because I just experienced something that's very similar to that where there's a subsystem that we have that has generally been.

17:49 Brad and plagued with bugs and um the theory of the case was like oh there's a bug over here and i'm sure it has to be this flavor of

18:00 Scott Mm

18:00 Brad bug and because here's my here's here's my empirical observations i've made and

18:07 Brad and they were like oh we must have broke something because it worked in september and it doesn't work now you know and i was like hold on a second let's go let's like capture what we know what we know we know versus what we think we know and um after kind of like looking at it there was indeed a bug

18:30 Brad But the bug actually always existed, and it was just that we didn't see the bug in September because this particular bug related to.

18:40 Brad longer and longer time periods expose the Yeah, bug where

18:43 Scott yeah.

18:44 Brad shorter time periods don't and and so it was a real interesting example it's it's not exactly the same thing that you're describing but it's a similar thing where it's so easy for us especially when we've we've kind of like

18:59 Brad bumped our head against a particular you know doorframe walking through it every time we're like convinced when we have a headache we must have hit that doorframe again and

19:08 Scott that's right well

19:09 Brad it it turns out we could have a headache for other reasons oh

19:12 Scott i just think i i take that word bug so personally because it fundamentally means like a person made an error right

19:19 Scott And anyway,

19:20 Scott okay,

19:21 Scott so number

19:21 Brad So, yeah,

19:22 Scott four, number four accessible,

19:23 Scott accessible internal data.

19:26 Brad number five, no surprise here, strong version control practices.

19:31 Brad Right.

19:31 Scott That

19:31 Brad Again.

19:32 Scott one, that one we've got nailed.

19:33 Brad Right.

19:34 Brad Right.

19:36 Brad Working small batches mentioned that one already.

19:38 Brad Users.

19:38 Scott Oh, actually,

19:39 Scott hang on a second.

19:40 Scott You know where I take that back.

19:43 Scott that one we've got nailed that

19:44 Brad But yeah

19:45 Scott one we've got nailed when it comes to code but for a lot of these models I don't think as an industry I think I think some of the machine learning models and stuff have gotten better but I don't know where that the state of the art is I know that in in the sort of

19:59 Brad Data modeling world,

20:01 Brad we have these machine learning models and AI models now.

20:04 Brad I don't know there's the maturity of versioning the models.

20:09 Scott Well, you know what's interesting?

20:10 Brad It might have developed in the last few years.

20:12 Scott Maybe.

20:13 Scott I actually think this is a really interesting question of strong version control practices. And again,

20:20 Scott this is a report that's looking at an industry-wide report.

20:24 Scott And they kind of identified these seven clusters of organizational kind of characteristics.

20:31 Scott And what they're saying here is that, you know, this is a really nice line here.

20:36 Scott A key aspect of mature version control is its function as a psychological safety net.

20:42 Scott This safety net allows development teams to experiment and innovate with confidence,

20:47 Scott knowing they can easily revert to a stable state if something goes wrong.

20:52 Scott One of our most tangible examples of this is the reliance on rollback and revert features,

20:58 Scott the ability to undo changes swiftly and without.

21:01 Scott without the fuss, uh,

21:02 Scott is not just a convenience. It's a critical enabler of speed and resilience.

21:06 Scott And I got to be honest with you, you said we've nailed that.

21:10 Scott And I think that you and I come from a long history of a school of hard knocks learning,

21:15 Scott learning that you, you want high quality version control system and you want high quality version control best practices.

21:22 Scott Right.

21:22 Scott And, you know, but it's a, it's a learning experience for some developers. I was just.

21:28 Scott It's just I was literally just talking to my friend who's starting a two guys in the garage project and he doesn't come from a classic software engineering background.

21:36 Scott And I was, you know, I was talking to him about, you know, his current challenges and he was saying like,

21:41 Scott oh, yeah,

21:42 Scott my my buddy says my my my pull requests are way too large.

21:45 Scott I'm like, well, how large are your pull requests?

21:47 Scott And he was like,

21:48 Scott oh,

21:48 Scott you know.

21:49 Scott a dozen files and like 2000 lines of code changed and i'm like yeah like that's that you're not making it easier

21:59 Brad No.

21:59 Scott on your your your co-founder here you know that's that don't do that um and you know i think that that i would not be surprised if uh there aren't

22:11 Scott organizations out there that for whatever reason either because they don't have you know principles don't have the experience to kind of demand this or for whatever reason they started running fast and haven't like kind of standardized on these things that i could imagine strong version control best practices are might might be something that teams struggle with and

22:35 Scott AI would,

22:35 Scott that would definitely, if you don't have good quality version control systems, and then you're running hard with AI writing lots of code for you, then I am not surprised that that increase in product instability line would go up for those teams.

22:50 Brad Yeah, yeah.

22:51 Brad I just, I have a hypothesis that software development practice,

22:55 Brad there's like software development practices within a data organization.

23:01 Brad on average are not as strong software development practices in the software development organizations are very strong and then within the data organizations are not but that's okay that's a good one that's a real good one i'm glad you brought you dug into that one more

23:14 Scott Yeah, and I blew past this one, but say working in small batches, I want to double click on that one a little bit. That's another good example where this is all about like the degree to which the teams break down their changes into manageable units that can be quickly tested and evaluated,

23:29 Scott right?

23:30 Scott So how many times have you and I like talked about this concept in engineering efficiency or how many times have we been dropped into an organization and

23:39 Scott And, you know, one of the key things that we're looking at is,

23:41 Scott well, how do you break down your work? How do you how do you how to make sure your work is bite sized?

23:47 Scott You know, that's the that's the way you get quality estimates.

23:50 Scott That's the way you get quality reproducibility. That's the way you get, you know, faster turnaround time.

23:56 Scott It's such a it's such a kind of core element to what I consider to be high performance engineering teams.

24:05 Brad Mm-hmm.

24:05 Scott Right.

24:06 Scott The high performance engineering teams are better at breaking tasks down.

24:09 Brad The name of the game.

24:11 Scott And again,

24:12 Scott I feel like this is a good example where if you kind of run into,

24:16 Scott you know, headlong into I'm I'm empowering agents,

24:20 Scott I'm I'm doing all this AI stuff,

24:22 Scott but you don't have like a kind of a system level thinking around we break things down into small pieces.

24:30 Scott then the AI agents are happy to follow your lead and write 2,000 line PRs,

24:35 Brad Yeah.

24:36 Scott right?

24:37 Brad And I really want to do an episode with you just to talk about how to train and incentivize teams and individuals to break things into smaller pieces.

24:49 Brad It's like a thing that either you get it or you don't.

24:52 Brad And the people who don't, how do we teach them to do it?

24:57 Brad Different

24:57 Scott Yeah,

24:57 Brad episode,

24:57 Scott that's a question.

24:58 Brad different episode.

24:58 Scott We'll have to think about that.

25:00 Scott So the next attribute they have on teams, teams with these attributes are more likely to be successful adopting AI.

25:06 Scott A user-centric focus is also measured,

25:10 Scott has been included in past iterations of our annual survey.

25:15 Scott A user-centric focus refers to the degree to which the teams think about the experiences of the end users of their primary application service.

25:23 Scott So a seven-point Likert scale indicators,

25:30 Scott apparently.

25:30 Scott All right, so that

25:32 Brad because

25:34 Scott sounds to me like,

25:36 Scott sure,

25:38 Scott it makes sense.

25:38 Scott It makes sense to me that if you have a user-centric model,

25:42 Scott you're probably going to get more results out of it.

25:45 Brad to some degree AI

25:48 Brad feels like one of those things is a solution in search of a problem and if your team is more focused on the problems you're going to find more effective solutions

26:00 Scott That's fair.

26:01 Scott That's fair.

26:02 Scott Last item that they list is quality internal platforms.

26:07 Scott So and again,

26:09 Scott I think this is the quality of these platforms.

26:12 Scott you know, set of capabilities that is shared across multiple applications or services,

26:18 Scott you know, these are kind of like your classic platform conversation,

26:24 Scott you know, that you and I'd like to have,

26:26 Scott like,

26:27 Scott are your internal platforms high quality?

26:31 Brad Yeah, well look we'll just do all that stuff and they'll just be great

26:36 Scott Yeah.

26:36 Scott And I think that that I mean, so here's so OK,

26:39 Scott so let's zoom the camera out for a second. OK,

26:40 Scott so what does this mean?

26:41 Scott Right.

26:42 Scott So if those are all the so we have this data set where we have 90 percent of people,

26:48 Scott 90 percent of organizations are adopting AI.

26:50 Scott Right.

26:51 Scott A whole bunch of people are like,

26:53 Scott yes,

26:53 Scott this makes me more effective. Right.

26:57 Scott And one of the metrics is a,

27:00 Scott you know.

27:02 Scott 0.1 units increase in instability of software releases oh my god right like that's horrible that's like that's like a really bad outcome yeah

27:12 Brad You know, you outlined those seven different archetypes of organizations earlier.

27:17 Brad I think it was seven.

27:19 Brad Does it do a crosstab of those? And you said the scores of the things that like were better or worse and there's the organizations that do a crosstab. It's like I have a hypothesis based on everything you've said so far.

27:32 Brad Here's my hypothesis.

27:34 Brad The organizations that are higher performing are getting more out of AI.

27:39 Brad Because all the things

27:40 Scott Yeah.

27:40 Brad you just said, which were attributes of an organization that's getting a lot out of AI,

27:45 Brad sounded to me like things of high performing organizations.

27:49 Brad right and so I suspect that if you're already an organization that I'm just trying to remember the things you said has clean data and has good internal systems and all these things AI is probably really helping you out and if you're an organization that doesn't have that stuff already then you're down in the bottom I don't know what you called it the hapless and helpless or whatever the first group was I don't remember what they were but

28:12 Brad You said, what was the first group you described of those groups?

28:16 Brad Because they got a great description for the name of it. I can't remember the name.

28:19 Scott Cluster one is stuck in survival mode,

28:22 Scott facing significant challenges with fundamental gaps in their processes,

28:26 Scott environments and outcomes like that sounds like a bad place to be.

28:29 Scott There's two other groups that also sound like they're in pretty bad spaces. The legacy bottleneck are in a constant state of reaction where unstable systems.

28:38 Scott dictate their work and undermine

28:40 Brad their Mm-hmm.

28:40 Scott morale and then there's the constrained by process these teams are running on a treadmill despite working on stable systems their efforts are consumed by inefficient processes leading to high burnout and low impact those are the three those are the three like you know um you know heroes to zero scale those are the three in the closer to zero scale you

29:04 Brad In my hypothesis,

29:05 Brad I have to look at this report myself, or maybe they've made the data available, is that those three categories,

29:11 Brad which I think were about 35% of the people,

29:14 Scott Yeah, 28 to 38%.

29:17 Brad 38.

29:18 Brad So 38, that's a lot, that's more than a third.

29:22 Brad Right?

29:23 Brad That's two out of five, almost people.

29:28 Brad Those organizations are not having a lot of success.

29:33 Scott They're not.

29:34 Brad They're

29:35 Scott And yeah,

29:35 Brad not.

29:35 Brad But

29:35 Scott well,

29:36 Brad they're not having a lot of success with anything.

29:38 Brad It doesn't have anything to do with AI.

29:40 Scott but here's the thing that may and again,

29:43 Scott I, you know, I haven't digested all 142 pages of this report,

29:49 Scott but I would not be surprised if.

29:52 Scott They're going nowhere faster,

29:54 Scott right?

29:55 Scott Like in other words, they're already struggling and AI just makes them go faster.

30:00 Scott They're just struggling faster.

30:01 Brad Yeah. So going slower, faster,

30:04 Brad the

30:05 Scott Yeah,

30:05 Scott they're going in the wrong direction faster.

30:09 Brad direction faster.

30:10 Brad Do you think that this report and the things you've outlined could be used almost as a

30:18 Brad The scorecard for organizational maturity assessment.

30:26 Scott Yeah, I think that, you know, I don't speak for the Dora group,

30:30 Scott but I think that Dora generally considers themselves as trying to quantify in a very measurable way.

30:42 Scott what they deem as the highest priority attributes of mature software organizations.

30:50 Scott So I think that's consistent with their vision statement anyway.

30:54 Brad Pretty cool.

30:55 Brad Well, you've been calling this a report.

30:58 Scott Yeah.

30:59 Brad It is not a report.

31:00 Brad It is a book.

31:02 Scott It's a book.

31:03 Scott 142 pages.

31:04 Scott It's

31:05 Brad It's

31:05 Scott a book.

31:05 Brad been 42 pages.

31:06 Brad Some people argue that the old man in the sea is the greatest American novel ever, and it was only like 120 pages.

31:13 Scott There's some pretty cool charts in here.

31:15 Scott Time spent using AI on a recent workday.

31:20 Scott So we've got this nice clean distribution where the axis is hours.

31:25 Scott and the number of respondents that report those hours.

31:28 Scott So what do you think the median number of hours that the several thousand people that respond to this report said they use AI in a recent workday?

31:40 Brad Well, before I tell you my answer,

31:43 Brad and I have a number in mind,

31:44 Brad I want to tell you that I think on all these AI surveys right now.

31:50 Brad The answers are inflated.

31:53 Brad And I think directionally, it all tells us the same thing. This is my very first skepticism at the beginning.

31:58 Brad I think right now, and this isn't exactly right,

32:01 Brad but like it's politically incorrect to say I'm not using AI or AI isn't helping me.

32:07 Brad Right.

32:07 Scott Yeah.

32:08 Brad So.

32:09 Brad I think the answer is 5.75 hours per day.

32:13 Scott Oh, interesting.

32:14 Scott Well,

32:15 Scott the median was two hours,

32:17 Brad Oh,

32:17 Scott but it's got a pretty long tail going all the way to 12 hours.

32:23 Brad does that include the agents you have working for you?

32:28 Scott Right?

32:29 Scott No, this is time spent using AI.

32:32 Scott I mean, this person, someone says 12 hours, they're saying they're working 12 hour days and they're using AI.

32:38 Scott Here's another one,

32:39 Scott reflexive use of AI.

32:41 Scott How often do users turn to AI when encountering a problem or task?

32:46 Scott So always,

32:48 Scott most of the time, about half the time, sometimes,

32:50 Scott never.

32:51 Scott Where do you think the pig and the python on that one is?

32:54 Brad Sometimes.

32:56 Scott Yeah,

32:56 Scott 39% of respondents said sometimes that's the pig and the python there.

33:01 Brad pig in the python what does that expression mean oh

33:03 Scott The lump in the curve,

33:05 Scott right?

33:05 Brad gotcha uh yeah that well that jives with the two hours to me right right if that if that number was 5.75 hours i would they would have said most of the time

33:19 Scott It's interesting.

33:20 Scott This is another,

33:21 Scott and I'm not quite sure how I would have answered this question.

33:24 Scott General reliance on AI at work, distribution of reliance levels among AI users.

33:29 Scott So 8% of respondents say a great deal.

33:31 Scott 20% say a lot.

33:34 Scott 37% say a moderate amount.

33:38 Scott A little 30%.

33:41 Scott reliance on AI by task number one largest reliance on AI by task is writing new code

33:48 Brad Hmm.

33:49 Scott so that's a little surprising to me um because we would have you know code reviews is down at 56 percent and writing new code is at 71 percent so that's kind

34:01 Brad You

34:01 Scott of interesting

34:01 Brad know, we were just talking about breaking work down.

34:05 Brad I was talking to one of our engineering managers last week.

34:08 Brad trying to name Tom and he told me what he was finding AI really effective for was take we kind of talked about this again we'll call back to Michael Natkin and sort of telling the AI to make you a plan he was saying it was actually being really effective for taking his product requirements document and his technical design document and breaking it down into tasks actually turning those tasks into JIRA tickets it was helping him and his team decompose their work

34:36 Scott Yeah, that's interesting.

34:37 Scott So like at least on their survey,

34:39 Scott things that sound very similar to that are creating specifications at 56% and analyzing requirements at 49%.

34:53 Brad Yeah.

34:54 Scott So definitely some some,

34:57 Scott you know, people are using it for those types of things.

35:01 Brad Yeah, you told me about a client that you have, I don't know if it's an active client or past client,

35:06 Brad where you used AI to take their requirements and turn them into more of a spec, a written spec for the engineers

35:12 Scott Yeah,

35:13 Brad and

35:13 Scott feature briefs.

35:13 Brad everybody loved it.

35:16 Scott Yeah.

35:16 Scott Yeah.

35:17 Scott And in fact, I kind of did a little mini teaching tutorial session with that product owner and say, here's how I'm doing.

35:23 Scott Here's

35:23 Brad Yeah.

35:23 Scott how I did it.

35:24 Scott And their reaction was,

35:26 Scott oh, great.

35:26 Scott I'll start doing that too.

35:27 Scott And they've been happy with the results.

35:29 Brad Okay.

35:29 Scott So yeah.

35:29 Brad So can I finish with an AI story today?

35:32 Brad It's a minor one,

35:33 Brad but it's a cautionary tale.

35:35 Brad It's an instructive lesson for all of us.

35:39 Brad Our CEO recently sent an all company email,

35:43 Brad not an unusual thing to happen.

35:48 Brad She observed,

35:49 Brad and we used Google Workspace at our company, and at the top of the email, there was the Google Workspace Gmail AI summary.

35:59 Brad And she was annoyed because she said,

36:01 Brad oh, this AI summary isn't accurately reflecting what I said in this email. And in a particular thing that was referring to,

36:10 Brad there was a number in there.

36:12 Brad It was referring to 33 people.

36:15 Brad And it listed it as 33%. And that's what we say 33% of the people and she was like, no, no, it's 33 people.

36:21 Brad Okay, well, another person on our team said, well,

36:23 Brad my AI summary says something totally different.

36:27 Brad Not only does it not make a mistake,

36:29 Brad it says something totally different,

36:30 Brad and there's an instructive lesson in there.

36:32 Brad I said this to our senior leadership team where I'm the only technical person.

36:36 Brad AI is not deterministic.

36:38 Brad It is probabilistic. You're

36:39 Scott Yep.

36:39 Brad going to get a different answer every time.

36:43 Brad And it taught us a little bit about how that, you know, for instance,

36:47 Brad Gmail,

36:47 Brad they're not.

36:48 Brad basing it off the source email they're basing it off the received email but everybody gets a different answer and it's a it's a real lesson with this AI you know it was a good it was a good opportunity to teach my senior leadership colleagues co-workers

37:04 Brad counterparts a little bit about how AI works this week with that AI summary

37:08 Scott Yeah.

37:10 Scott That is that's a great story. And for sure, those of us that have been plugging stuff into these LLMs for a while, we've noticed this exact same question two times in a row, two totally different answers.

37:21 Scott So there you go. That's the state of Dora for AI.

37:26 Brad We'll stick it in the show notes.

37:28 Scott Yeah,

37:28 Scott we'll definitely link this in the show notes.

37:30 Brad I love it. Now I'm going to go read that book.

37:33 Scott Yeah.

37:34 Brad All right. It was nice to talk to you, Brad.

37:36 Scott All right. Good talking to you, Scott.

37:37 Brad Till next week.

37:39 Scott This episode of How Many CTOs Does It Take podcast was edited by Saffron Epigal.

37:44 Scott Our theme song was AI generated by Brad.

37:47 Brad Please follow us on all the social media platforms at at how many CTOs pod You can find us on YouTube LinkedIn Instagram Facebook threads X TikTok and patreon or at our website at how many CTOs pod calm and remember to rate review and subscribe on the podcast platform of your choice thanks for listening