Transcript: From Prompts to Product: How AI is Changing Software Development
0:00 Brad The robot product manager did a way better job of specifying to the robot software engineer what I, as the chief product officer,
0:09 Brad just described as, I just want transcripts on my podcast website.
0:31 Brad Another episode of the How Many CTOs Does It Take podcast?
0:36 Brad I'm Brad Hefta-Gaub, and my trusty co-host is...
0:40 Scott got all rad and we are here today for a little mini episode to talk to you about a bit of AI data that I saw recently
0:50 Brad AI data,
0:51 Brad the latest and greatest on AI.
0:53 Scott the latest and greatest on AI which
0:55 Brad By
0:55 Scott will yes
0:55 Brad the way, by the time this episode comes out, GPT-5 will have been released.
0:59 Brad So that will be interesting.
1:01 Scott this data will be still but I saw this brand I felt like I had to call you right away and get your take on this
1:05 Brad Yes.
1:05 Brad Tell me.
1:07 Scott I saw a chart, a piece of data on the internet on someone's blog.
1:12 Scott I don't have it in front of me handy,
1:14 Scott but it basically showed the x-axis was time.
1:19 Scott The Y axis was the number of minutes,
1:21 Scott the duration of a human task that AI was able to complete successfully.
1:30 Scott So, you know, the X axis is 2020,
1:33 Scott 2021,
1:34 Scott 2022,
1:34 Scott et cetera, et cetera. You know, by 2023.
1:39 Scott AI could complete a three,
1:42 Scott a two minute task that a human could complete.
1:45 Scott And by 2024,
1:46 Scott it could complete a three minute task.
1:50 Scott And now we're in mid 2025 and it can complete a task of, you know, say 12 minutes or maybe 15 minutes at the most, right?
1:58 Brad So can we assume that like a length of time of the task is
2:04 Brad some kind of a rough estimate for the complexity of the task?
2:07 Scott I think that is an implication here,
2:11 Scott which is longer tasks just are more complex and humans can do more complex things.
2:17 Scott So, you know, can the computer do what the human can do?
2:20 Scott And so this was pointed out, this data was pointed out to me by someone who was kind of talking about how it really illustrates.
2:31 Scott for all this AI-assisted coding and all the agentic coding,
2:36 Scott which
2:37 Brad I know you love that word.
2:38 Scott I've been thinking about that word actually.
2:40 Scott Agentic, is agentic really just a fancy word for asynchronous?
2:43 Brad I don't know.
2:45 Brad I think it was just somebody wanted to say agent,
2:51 Brad but sound fancier.
2:52 Brad You should.
2:53 Scott So,
2:54 Scott look,
2:55 Scott I know you spend a lot of time dealing with AI,
2:57 Scott exploring it,
2:58 Scott et cetera,
2:58 Scott et cetera.
2:59 Scott So I want to get your take.
3:00 Scott To me, this really spoke and what this personal conversation I was having was like, how small, it's a signal of how small we have to break down the tasks for the AI or the AI agents to succeed at them.
3:13 Scott Like if you want to code something,
3:16 Scott you can't just say.
3:17 Scott make me a mobile app you got to just break it down into like three minute four minute five minute tasks for the agent to do and has that been your experience
3:27 Brad Okay, so I have a couple things to say about this.
3:31 Brad So first of all,
3:32 Scott Let's
3:32 Brad I
3:32 Scott hear
3:32 Brad hadn't never put it in the form of like,
3:37 Brad do I think this task is going to take the agent X minutes to solve?
3:42 Brad But it has definitely been the case that I have found that you want to break things down into small enough tasks that when it starts to go off the rails,
3:51 Brad it doesn't go too far off.
3:52 Brad are off the rails and
3:54 Scott Mm-hmm.
3:54 Brad that's kind of the way i think about it and in fact oftentimes i would give my like here's a here's a here's a really i think funny example of where i've seen the ais go off the rails where i have to give it a time box and say if you
4:11 Brad If you bump up against this time box,
4:13 Brad then stop what you're doing and come and ask me for help.
4:17 Brad And that is, no joke,
4:19 Brad fixing linter errors.
4:22 Brad So,
4:22 Scott Hmm.
4:22 Brad you know, a lot of times if you have really strict lint rules,
4:26 Brad you can have these almost rules that are competitive with each other where it's like I fix one rule and the easy way to fix the rule is essentially to break another rule.
4:36 Brad But if you have both those rules in place, then coming up with the correctly formatted code that achieves that meets both requirements sometimes is hard.
4:45 Brad Right.
4:46 Brad And I have seen I've seen.
4:48 Brad seen i just like i've got like cursor with the agent running in the in the in the sidebar and i'll just watch it it will like and it's like literally flipping between two different implementations over and over and over again going like oh i i fixed the lint error oh i added more lint errors wait i fixed those lint errors i came and it's just like it it's you know even today um it's still pretty bad at that
5:15 Brad Um,
5:16 Brad and so that's an example where I've kind of stopped it wasting its own time and stopped it wasting my time by giving it very clear instructions about don't go off the rail.
5:28 Brad If you, if you, if you may, if you fail to fix this in two attempts,
5:33 Brad stop what you're doing and ask me for help.
5:35 Brad Right.
5:40 Brad So I think that kind of reads on the point about break things down into small pieces.
5:46 Brad Now, of course, we've also seen that, you know, Michael Natkin and.
5:53 Brad Kevin,
5:54 Brad our friend from Pioneer Square Labs,
5:57 Brad they often are using this technique of giving the agents, asking the agent to produce a checklist and then asking the agent to work through that checklist.
6:06 Brad And I think that's a really good paradigm that's essentially achieving that same goal of like, have it only focus on small pieces at a time.
6:16 Scott Mm-hmm.
6:17 Brad Another thing that is kind of, I think, related to this, two years ago or two,
6:24 Brad three years ago, one of the first like prompt engineering hacks that somebody discovered and then announced to the world and everyone immediately adopted it was if you gave it the prompt where you asked it the question,
6:38 Brad but then said, think about the steps you're going to take.
6:43 Brad to answer this question and complete those steps one by one like that original prompt all
6:51 Scott Mm
6:52 Brad of
6:52 Scott -hmm.
6:52 Brad a sudden unlocked a whole bunch of results on the exact same llms right so it wasn't that the llm got better it was that somebody just discovered that if you pre-prompted the llm to say think this through step by step
7:08 Brad It just performed better. And I think that's very much in line with this.
7:14 Brad It's just something about the nature of the way the LLMs work where telling it to go step by step keeps it from hallucinating basically or
7:25 Scott Mm-hmm.
7:26 Brad makes it less likely to hallucinate.
7:31 Scott It's fascinating.
7:33 Scott how these llms like how we're learning to interact with them you
7:39 Brad Right?
7:40 Scott know they're they're obviously not living beings but they're almost like they're they're like alien life forms where we don't speak the same language entirely or we don't we don't have the same way of interacting with each other
7:52 Scott And we're kind of learning, which is it's just fascinating.
7:56 Scott Your point about the this is a tangent,
7:58 Scott if I could just share a tangent for a second, your point about the linter.
8:02 Scott Have you seen the videos like on Instagram or wherever YouTube where the Waymo traffic jams?
8:09 Brad I have not, but I can picture them in my mind's eye.
8:14 Scott Yeah,
8:14 Scott so I've seen a few of these, you know, it'll be like, you know, two Waymos like get, you know, Waymos are very bright,
8:20 Scott they know how to not crash into stuff and they'll get, you know, get in some space with another Waymo and they'll just get stuck.
8:27 Scott because neither Waymo will neither Waymo will move and you know if you if you or I get at a four-way stop and there's a little confusion eventually one person waves the other one on but the Waymo doesn't know how to do
8:36 Brad They
8:36 Scott that and
8:37 Brad don't have arms to stick their arms out of waves.
8:40 Scott and I've if you go on YouTube or Instagram you can find these videos of like parking lots full of Waymos like all the Waymos are trying to like leave in the morning to go do their routes but somehow something just got off a little bit
8:52 Scott bit in their system you know they're programmed to leave all the parking lot in some order but they've gotten in a total traffic jam i'm going to find one and give it to you to put in the show notes
9:01 Brad I love it. I love it.
9:02 Scott it'll be terrific
9:03 Brad So I want to hear something I think would be interesting,
9:07 Brad Scott.
9:07 Brad So.
9:09 Brad I, you know, I told you a while ago that one of the things I've been using as like my experimentation and play around with to try to learn how to better interact with these coding agents is our podcast website.
9:23 Scott Yeah.
9:23 Brad And so yesterday or yesterday or two days ago,
9:26 Brad I got a bee in my bonnet and I decided I came up with like three features I wanted to add to our podcast website.
9:32 Scott Oh,
9:32 Brad And
9:32 Scott we got to do another live coding then.
9:34 Brad well,
9:34 Brad I already did them.
9:35 Brad So but we'll use these as we'll use these
9:37 Scott So you're
9:38 Brad as.
9:38 Scott like my AI agent.
9:40 Brad Yeah,
9:40 Brad so let me tell you an example of three things that it successfully did.
9:45 Brad And and I think that we can kind of think through the scope of it.
9:49 Brad Okay, so here is a prompt that I gave.
9:52 Brad I was using Codex, but I think this would have worked just as easily with GitHub Copilot or Quadcode.
9:59 Brad Like this, this feels like a very in the sweet spot.
10:02 Brad So I asked Codex, the episode component.
10:06 Brad This is my exact prompt.
10:07 Brad The episode component uses the episode description from the podcast RSS feed,
10:12 Brad which often,
10:13 Brad not guaranteed,
10:14 Brad will include time codes in the format of mm:ss for topics in the episode.
10:21 Brad I would like to make those time codes be hyperlinks to fast forward the audio player to that time stamp in the audio file.
10:29 Scott Okay.
10:30 Brad So let me just ask you.
10:33 Brad If you gave that instruction to a junior engineer,
10:36 Brad how many story points do you think they would estimate that at? How hard would they say that was?
10:42 Brad And, you know, do you feel like that's an appropriately described specification?
10:48 Scott I feel like it is certainly enough.
10:51 Scott for the engineer to have an outline of what you want, and then to maybe go ask about some edge cases they could potentially think of.
11:01 Scott Like, okay, well, what about this brand?
11:03 Scott What about that? What about this, you know, et cetera, et cetera.
11:05 Brad Okay.
11:05 Scott It's a great starting point.
11:07 Brad Okay.
11:08 Scott Actually,
11:09 Scott let's even put it this way.
11:11 Scott In regular work at many of the companies I've been at,
11:14 Scott that is...
11:16 Scott That is more specificity than most of the tickets that get written.
11:21 Brad Yeah, fair enough.
11:23 Brad All right.
11:24 Brad So code X worked.
11:27 Brad So my codec setup,
11:28 Brad it has direct access to our podcast GitHub account.
11:33 Brad It
11:33 Scott Mm-hmm.
11:33 Brad has the development environment for the podcast website is set up with lint. It has end-to-end tests. It has its just, you know, node NPM packages. So it's like a pretty robust, straightforward,
11:49 Brad you know, any developer that we.
11:51 Brad hired off the street to work on it. They'd read the readme, they'd know how to work in it, right?
11:56 Brad So it has all that access to it.
11:58 Scott Mm-hmm, mm
11:59 Brad With that prompt, it worked for 5 minutes and 28 seconds and produced a pull request that added 31 lines of code and subtracted one line of code.
12:15 Scott -hmm.
12:16 Brad and ran fully linted, fully compiled,
12:20 Brad and fully passed end-to-end tests.
12:24 Scott Pretty good,
12:24 Scott I'd say.
12:25 Brad And I opened the pull request that generated a ephemeral environment for me to test in.
12:33 Brad It worked exactly as I imagined it would work. And...
12:39 Brad Boom,
12:39 Brad I accepted the change.
12:40 Scott I think if they're a human,
12:43 Scott that would have taken 30 to 45 minutes.
12:47 Brad Testing that PR,
12:49 Brad our episodes page, by the way, has like all the episodes listed and they all have audio players that you can go click on and play.
12:59 Brad In testing that PR,
13:00 Brad I noticed a bug that I had never noticed.
13:03 Brad And that is.
13:05 Brad If if you wanted to on our episodes page,
13:08 Brad you could just start clicking and playing all the episodes at once and then you would have this like cacophony of chorus of Brad and Scott talking in your ears.
13:16 Brad And I thought to myself,
13:17 Brad that's weird.
13:19 Brad That's probably a bug. Like if you start playing one of the episodes, it probably should just pause the one that you were listening to. Right.
13:26 Brad It feels like.
13:26 Scott Probably, yeah.
13:27 Brad All right. So I said.
13:30 Brad Gave Codex a new command.
13:32 Brad I said, it is possible for multiple episode components to be visible on the same page.
13:37 Brad For example, on the episodes page,
13:39 Brad we show all the episodes as episode components.
13:42 Brad Episode components have audio players on them.
13:45 Brad Right now, it is possible to click or...
13:48 Brad Click or to click play or click a time code link on multiple episodes at the same time and end up with multiple audio players playing at the same time.
14:00 Brad We would prefer that if you start playing an episode or a time code of an episode,
14:04 Brad that it stops any other episode audio player that is already playing.
14:09 Scott Very specific.
14:10 Brad How do you feel about that?
14:12 Scott I feel great about it. I thought you were going to say we would prefer that only one is allowed to be played at a time and let it figure out how to handle that scenario.
14:20 Scott But that was great. That was a perfect ticket.
14:22 Scott Somebody should be able to figure that out.
14:24 Brad All right, what do you based on based on the previous example,
14:28 Brad how long do you think chat GPT codecs worked on that problem?
14:33 Scott I want to say one thing first.
14:36 Scott You know, when we write bugs, at least the way I was taught,
14:41 Scott you sort of write a background and then you write repo steps and then you write actual and then you write expected,
14:47 Scott you know, that's sort of a format.
14:49 Scott And then there's other formats like I think there's like the Gherkin format for like a way of describing features.
14:55 Scott And I wonder if that type of.
14:59 Scott structure. There might be different kinds of structures that are needed for LLMs,
15:02 Scott but like a more formal type of structure is not only going to be valuable, but our industry will start to coalesce around that as just a more effective way of communicating.
15:14 Scott But to answer your specific question,
15:18 Scott I'm going to say your first one was about five and a half minutes.
15:22 Scott And so just a little like strategizing here,
15:26 Scott it could have been a lot faster.
15:28 Scott So you were impressed. So I could say 30 seconds or it could have been abysmal and I could say, you know, 90 minutes. But since since earlier, I think you just told me sometimes you.
15:44 Scott Given a time limit,
15:45 Scott I'm going to say 90 seconds.
15:49 Brad I think that's a pretty good guess, but I will tell you that you're wrong,
15:53 Brad but not because of why I think you're wrong. I think it's impractical for it to take 90 or 30 seconds to do something only because it's got to check out all the code.
16:02 Brad Because every single one of these,
16:03 Brad it sets up a whole new environment.
16:05 Scott Okay, understood.
16:06 Scott I did not totally rock that part.
16:08 Scott Okay,
16:08 Scott I'm going to go back and I'm going to say seven minutes.
16:13 Brad It took four minutes and 10 seconds.
16:17 Brad um and
16:18 Scott That's that's another 45-minute task.
16:22 Brad right and here's what's interesting by the way about this one um this one is a little complex right because it had to solve a like the episode component doesn't know about the existence of other episodes so
16:36 Scott Yeah,
16:36 Scott I was thinking about that
16:37 Brad it had to solve this problem of well how am i going to make the other the episode know about other episodes
16:44 Brad Right.
16:44 Brad And honestly,
16:45 Brad it did it in a really clever way.
16:48 Brad It actually introduced a new audio player Svelte store,
16:54 Brad which is basically Svelte stores allow you to kind of exchange information between two different pieces of code.
17:00 Brad And so it just basically changed it so that every time.
17:04 Brad the the episode starts to play an audio player it records it puts in the story like i'm the audio player you know and
17:12 Scott Yeah.
17:12 Brad so then it's able to know the last audio player and so it solved the problem in a pretty ingenious way and i honestly think that had i given had i given this to like a junior engineer they would have had to think kind of hard about that part of the problem like how am i gonna how am i gonna let these things know about each other
17:29 Brad without like making the code really ugly and complicated of like having some list of all audio players bullshit or
17:36 Scott Well,
17:37 Brad something like that
17:37 Scott it's interesting.
17:38 Scott Or do you have a design which says there is one audio player on this page,
17:42 Scott and as you click play,
17:44 Scott play, play, it updates the source in the single audio player.
17:50 Brad why that maybe would be the answer that it would have had another another but
17:53 Scott answer
17:54 Brad then i would but then i would have been like i still want there to be a little play button down there like i want each of the episodes to be able to have a nice little play button that you could just you know start playing it
18:03 Scott it's interesting what this is making me think of something do you think
18:06 Scott Okay, there's a lot of potential solutions to that problem,
18:09 Scott each of which are fine,
18:10 Scott pros and cons to each, but do you think that this kind of coding is going to lead to standardization?
18:17 Scott Like one of the complaints,
18:18 Scott I don't know if complaints is the right word, one of the observations I've made about software engineering is, I mean, we have standards, but so little standards, right?
18:27 Scott I mean, you go build a house.
18:30 Scott Every single way the
18:31 Brad Yeah.
18:31 Scott wires and plumbing and everything is standard, we've got nothing relatively speaking.
18:35 Scott And I wonder if all of this agentic coding is going to lead to more standardization.
18:41 Brad I don't know if I can answer that question,
18:42 Brad but let me give you a part of the answer that I think maybe I think this speaks to what you're asking.
18:49 Brad So you recall a couple episodes ago we talked to Vince Dineen about design
18:55 Scott Mm-hmm.
18:55 Brad and,
18:57 Brad you know, he kind of talked a lot about design systems, you know, like tailwind or or skeleton or whatever,
19:03 Brad get a design system. And Greg Conklin, when we interviewed him, he.
19:06 Brad And he he mentioned that as one of like the early things that startups should do is like get a design system early.
19:13 Brad So design systems are an example of if you are using it in your development organization, then it standardizes the work within your development organization.
19:22 Brad So a button is always a button.
19:24 Brad Right.
19:24 Scott Mm-hmm.
19:26 Brad This particular code base that we have for our podcast,
19:29 Brad I've chosen to use Svelte, which is a framework not dissimilar from React.
19:36 Brad I could have used React,
19:38 Brad but I used Svelte instead.
19:39 Brad And there is a paradigm for doing Svelte development,
19:44 Brad just like there's a paradigm for React development,
19:46 Brad right?
19:47 Brad And Angular has its own paradigm and, you know, all the different frameworks kind of have their own paradigms.
19:53 Brad And what is interesting is that the AI code.
19:59 Brad assistance,
20:00 Brad I believe are quite good at following programmatic paradigms for the particular framework that the code base is.
20:15 Scott Mm-hmm.
20:15 Brad So Svelte is a relatively opinionated framework.
20:21 Brad There's ways to do Svelte things right.
20:25 Brad And there's ways to, and if you're not doing it the Svelte-like way,
20:29 Brad it's obviously, to anybody who's familiar with Svelte, it's obviously not the Svelte-like way to do it.
20:37 Scott Mm-hmm.
20:38 Brad I didn't have to tell ChatGPT Codex that this was a Svelte code base.
20:46 Brad I did use the word component,
20:48 Brad so I kind of hinted to it, but it downloads the code, looks at it and goes,
20:51 Brad ah,
20:52 Brad this is spelled and immediately then.
20:55 Brad So so to answer your question about like standardization, I think out of the box, the agents write code consistent with the paradigm of the framework that the code exists as.
21:11 Brad And it's quite good at that.
21:13 Scott So it almost takes like linting to the next level.
21:16 Brad Or it's, you know, it's almost like taking,
21:20 Brad like you don't have to teach it.
21:25 Brad Like if a brand new junior developer shows up on my code base and they've never coded in Svelte before,
21:29 Brad their first couple projects are all about a senior engineer saying,
21:34 Brad don't do it that way. That's not the Svelte way to do it, right?
21:38 Scott Right,
21:38 Brad You
21:38 Scott right.
21:38 Brad don't spend any time with the coding agents to teach them how to code in Svelte in the Svelte way.
21:45 Scott Yeah, it's interesting.
21:46 Scott I mean, how many times in your career,
21:48 Scott Brad,
21:49 Scott have you had to go,
21:50 Scott there's been some piece of software that wasn't working right and the person who wrote it isn't around and it's in some language you're not all that familiar with like,
21:57 Scott oh, you know.
22:00 Scott there's a shell script a batch file like okay well i've done these before but i'm not a shell script or batch file like expert um and you go in there and you hack some stuff in and you solve the problem and like you know you maybe do a little stack overflow to figure out how to do the thing you want to do and like then you move on right you're done and how ai is going to be a game changer for that
22:25 Brad 100% agree.
22:26 Scott Yeah.
22:27 Brad All right. I have one last example I want to share with you.
22:31 Scott Let's hear it.
22:31 Brad You want to hear this one?
22:33 Scott Yeah.
22:33 Brad Okay.
22:33 Scott Okay, let me guess. Before you even tell me,
22:36 Scott 15 minutes.
22:38 Brad Ooh, good question.
22:40 Brad Nope.
22:42 Scott Okay, let's hear it.
22:43 Brad Okay.
22:44 Brad All right. So this one.
22:46 Brad I got to be in my bonnet to say, what if we added transcripts to our show episodes?
22:52 Scott Mm-hmm.
22:53 Brad You know,
22:53 Brad we use Descript as our recording platform,
22:57 Brad and Descript has the ability to output transcripts. And our hosting platform has a little link when we publish episodes where it's like add your transcript here.
23:10 Brad And I don't really know that much about transcripts, but apparently they're...
23:13 Brad Apparently, there's a standardization around like VTT files or video transcripts and something called an SRT file is essentially an audio transcript and it includes time codes, name of the speaker and then whatever they said, right?
23:26 Brad And I thought to myself, would that be cool if we could have in that episode thing,
23:29 Brad if we have a transcript for it.
23:32 Brad have it like have a little fold down that shows the transcript scott said this brad said this vince said this and um it should i don't know it like it should follow along it should be clickable all that stuff right but i didn't know anything about how transcripts work like i honestly didn't know so the first thing i did was i went to chat gpt and said
23:57 Brad explain teach me everything about your
23:59 Scott Right.
23:59 Brad audio podcast transcripts you know and the stuff i just told you about srt files that was stuff that it it did it told me about and
24:07 Scott And now that's going to clutter your brain for the rest of your life.
24:10 Brad exactly i won't be able to get rid of it so um so it turns out that like
24:15 Brad dscript will not output srt files or vtt files it just has these uh so it doesn't follow the standards so that kind of sucks but i went ahead and exported a transcript and i asked chat gpt like what's the best way to get me a transcript that would be easiest to convert to one of these srt files so it kind of told me you know i gave screenshots of dscript said here's some options tell me which ones to choose and so it said choose this and this and then it wrote me
24:41 Brad me a Perl script or Python script to convert that output into an SRT file and I was like okay cool and
24:50 Scott I'm liking it so far.
24:51 Brad so then I'm like okay I want to ask my coding agent to add this and so what I did was I asked ChatGPT after having this big long conversation with it about transcripts I said will you please
25:04 Brad Give me instructions that I can give my AI coding agent to add the transcript feature to our website.
25:12 Scott Very meta.
25:14 Brad Very meta.
25:15 Brad Now, I am not going to read this to you because it's really long.
25:20 Brad It's like, you know, a page at like 10 point font of instructions,
25:26 Brad but it included.
25:30 Brad feature name transcript display and playback sync with podcast episodes it included two paragraphs of context it gave requirements around fetching the so it turns out the transcript is embedded in the rss feed as another url that you've got to go download and then it's going to be in this srt format and you're going to have to convert that so it gave it like explaining how to fetch it
25:54 Brad It gave it rendering instructions,
25:57 Brad right? Display the transcript and a scrollable list of blocks,
26:00 Brad human readable timescale,
26:02 Brad speaker's name in bold,
26:04 Brad transcript text.
26:06 Brad You know, use standard styling,
26:08 Brad audio player integration.
26:09 Brad When the person clicks on a piece of the transcript,
26:12 Brad jump the audio player to that point.
26:15 Brad Accessibility, it had excessive,
26:17 Brad I didn't come up with this, it came up with accessibility instructions to make sure that it's using article style blocks, which are HTML meta tags,
26:25 Brad keyboard navigation,
26:26 Brad performance requirements.
26:29 Brad It added all that.
26:31 Scott So what I'm taking from this is compared to the human-created tickets and specifications,
26:39 Scott it does a really good job of specifying exactly what to build,
26:45 Scott including a variety of considerations that...
26:50 Scott We often forget that we should always be considering that are often a pain for companies to go back and deal with.
26:56 Scott I've been involved in accessibility lawsuits where we've been sued because our product was not accessible.
27:03 Scott We have to go back and fix these things.
27:05 Scott But the robots get that stuff out of
27:08 Brad The
27:08 Scott the gate.
27:08 Brad robot thought of it.
27:09 Scott Yep. Amazing.
27:10 Brad The robot product manager did a way better job.
27:13 Brad of specifying to the robot software engineer what I, as the chief product officer, just described as,
27:21 Brad I just want transcripts on my podcast website,
27:24 Brad right?
27:24 Scott Wow.
27:26 Brad Now, what is your guess on how long the robot coder took to solve this problem?
27:33 Scott 15 minutes.
27:36 Brad Five minutes and 29 seconds.
27:40 Brad Wait, there's a little bit more to the story.
27:43 Brad Five minutes and 29 seconds,
27:45 Brad it produced a pull request.
27:47 Brad It was a brand new transcript component,
27:52 Brad so it did a great job of being svelt-like.
27:55 Brad The transcript component was about 171 lines of code.
27:59 Brad It modified the episode component to embed the transcript component,
28:02 Brad and it had to modify the sort of like the fetching functions that download that podcast data to recognize transcript URL.
28:11 Brad Opened up the pull request.
28:13 Brad launched the ephemeral domain it didn't write it didn't write new end-to-end tests for me which is interesting open
28:21 Scott Hmm.
28:21 Brad the pull request launched
28:23 Scott Bad robot.
28:23 Brad the uh bad
28:24 Scott Bad robot.
28:25 Brad robot i didn't tell it to make new tests uh launched the domain and it did not work it did not show the transcript it showed no transcript at all for the episode that i knew had a transcript because i specifically added a transcript so it could do it
28:42 Brad and i was like what the heck like it clearly didn't maybe you know i was like ah maybe the rss feed doesn't yet include it so then i had to go debugging and i downloaded the rss feed oh yeah no it's in there i double check yeah it's there there was a bug in the original code where it did it did not know how to pull out the transcript url out of the rss feed
29:07 Brad Um,
29:07 Brad but so it was obvious to me that it was, that was the bug.
29:11 Brad So then I basically brought that code base,
29:15 Brad that pull request,
29:16 Brad that, that branch into cursor.
29:18 Brad And I told cursor,
29:19 Brad Hey,
29:20 Brad this code has a bug.
29:21 Brad I know there's a transcript on episode one.
29:27 Brad Tell me why is this not working?
29:31 Brad And cursor figured out.
29:34 Brad that cursor probably Claude Sonnet figured out that ChatGPT 3.5 or whatever it codex used
29:43 Brad didn't know how to do that correctly and cursor fixed that bug and that took it because now I was kind of driving it took a couple minutes for me to get it to fix that bug um and
29:55 Scott Did you actually have one model talk shit about another model to you?
29:59 Scott Is that what they're doing? They're pointing fingers at each other?
30:02 Brad They were basically pointing fingers at each other.
30:05 Scott I have two observations
30:07 Brad But
30:07 Scott to
30:07 Brad I always
30:07 Scott make.
30:07 Brad – I want to keep the robots happy with me and happy with each other, so I never name names.
30:14 Brad I always ever say,
30:15 Brad like, this was written by another expert.
30:18 Brad Please tell me what that expert did wrong.
30:20 Scott All right, so I have two observations before we wrap.
30:25 Scott The first is, and I'm not trying to toot my horn by saying this, I'm trying to just say this is the data point to go with it.
30:32 Scott I've kind of envisioned for a long time that all,
30:36 Scott when I say a long time, a long time in the AI landscape,
30:40 Scott which is like a couple of years at the most, right?
30:44 Scott All software engineering,
30:45 Scott not all software engineering, a lot of software engineering is going to actually become QA engineering.
30:51 Scott And that's what I just heard you talking about,
30:53 Scott which is like what's going to become product management is going to become QA, right?
30:57 Brad Yep.
30:59 Scott Now, there will be lots of people working to make the middle part.
31:02 Scott We're going to have to write the robots,
31:03 Scott although maybe the robots will start writing the robots,
31:05 Scott right?
31:05 Scott And it's turtles all the way down.
31:07 Scott So that seems to be true,
31:09 Scott you know, and that gets to the point about how like you're a very experienced engineer and so you know how to do this. How is our...
31:18 Scott How are the new entrants to our field going to get this experience?
31:22 Scott But the other observation I want to make before we wrap is, it seems like you're having fun.
31:29 Brad Yeah.
31:30 Scott And I think I hear a lot of people being
31:34 Scott are feeling threatened,
31:36 Scott their jobs and their livelihoods and their passion feeling threatened by AI,
31:41 Scott but I also hear you having a lot of fun making stuff with it.
31:45 Scott I don't know. I think that's a good thing.
31:47 Brad Yeah,
31:48 Brad I think that I'm having fun. The engineers that are on my team that have really embraced AI,
31:54 Brad I think they're having a lot of fun too.
31:56 Brad I think they feel empowered and they feel more powerful.
32:01 Scott Yeah.
32:02 Brad And that feels fun.
32:04 Scott So overall soup to nuts that last one,
32:07 Scott how long did it take you?
32:08 Brad Probably total time.
32:10 Brad Well,
32:10 Brad starting out me learning what how how they actually
32:15 Scott You
32:16 Brad worked,
32:17 Brad I'm going to say 30 minutes total.
32:21 Scott easily did a day's worth of work, a half day to a day's worth of work.
32:27 Scott including coffee breaks and lunch and the whole thing right in 45 minutes yeah
32:33 Brad And it was, you mean for all three of those projects?
32:35 Scott yeah
32:36 Brad I would say it was probably two days worth of work in 45 minutes because I had to go learn about like if I hadn't had ChatGPT to teach me about transcripts, that would have been even more time, right?
32:52 Brad that that was that was a big piece that i would have to go learn a lot about that like how are we going to learn about srt and vtt files like
32:59 Scott You would have just been perusing blog posts and reading RFCs and stuff like
33:03 Brad and
33:03 Scott that.
33:03 Brad plus i'm horrible with like writing like you know text processing things you know like you you know like the fact that it was able to convert some random text file into a specific format for me that was like
33:14 Brad That
33:14 Scott It's
33:15 Brad was definitely
33:15 Scott change
33:15 Brad extra expert.
33:16 Scott changer.
33:16 Brad Yeah,
33:17 Scott it's making it made you powerful.
33:18 Brad For sure. It was easily a day to two days worth of work across three different disciplines,
33:25 Brad product engineering and QA, that I spent 45 minutes on.
33:28 Scott So the same joy that we got in the seventh grade when we saw the logo turtle scored across the screen.
33:36 Brad Yep.
33:37 Scott And we're like, look,
33:38 Scott cool,
33:38 Scott I made this, right?
33:40 Scott And when we made a little video game, which was stupid,
33:43 Scott you know, made something that added two numbers together or whatever in seventh grade computer class. And we're aging myself probably just like you age yourself.
33:51 Scott But that was fun.
33:53 Scott and because you were like as a kid you were being like this is cool check out what i can do and you know as a result of that now we've got self-driving cars but like um if somebody
34:04 Brad Stuck
34:04 Scott could
34:04 Brad in a parking lot.
34:05 Scott get stuck in a parking lot so um yeah maybe this is going to bring some of the joy back to what we're doing because i sense this is maybe this is me projecting i feel like in the last
34:20 Scott It's 15 years, a lot of the joy has been squeezed out of what we do.
34:25 Scott As I know why exactly, but it seems that's a topic for another time,
34:29 Scott but maybe this is going to bring the joy back.
34:31 Brad There you go. Bring the joy back.
34:33 Brad I love it. Well,
34:34 Brad thanks for talking to me about this one. This was fun.
34:36 Brad I think it's always great that we do these recaps on kind of the state of AI and yeah.
34:42 Brad As you know, I'm quite bullish on it.
34:44 Scott Well, it was nice talking to you, Brad.
34:46 Scott Let's talk again soon.
34:48 Brad All right.
34:49 Brad Bye. Have a good one.
34:50 Brad This episode of How Many CTOs Does It Take Podcast was edited by Safran Epigow.
34:55 Brad Our theme song was AI Generated by Brad.
34:58 Brad Please follow us on all the social media platforms at at how many CTOs pod you can find us on YouTube LinkedIn Instagram Facebook threads X tick-tock and patreon or at our website at how many CTOs pod comm and remember to rate review and subscribe on the podcast platform of your choice thanks for listening