Transcript: Bonus Episode: Challenges and Insights From Using Cursor to Add a Feature to Our Website

0:00 Brad It puts, in my mind, more pressure on the test automation.

0:16 Scott And these CCOs will share their stories untold.

0:20 Brad Hey, Brad.

0:21 Brad Welcome to a little bonus episode of how many CTOs does it take?

0:25 Scott Yeah.

0:26 Brad Yeah,

0:27 Brad so everyone out there listening,

0:28 Brad we just spent a little bit of time doing some coding with an AI agent.

0:35 Brad You can go over to our YouTube channel,

0:37 Brad just go over to YouTube,

0:39 Brad look for how many CTOs does it take?

0:41 Brad I don't think there's more than one of them out there,

0:43 Brad so you'll find

0:44 Scott How

0:44 Brad us.

0:44 Scott many CTOs does it take, Pod?

0:46 Brad Okay,

0:46 Scott But

0:46 Brad how many we'll CTOs

0:47 Scott link

0:47 Brad does it take,

0:47 Scott the

0:47 Brad Pod?

0:48 Scott video in the show notes of this little mini bonus episode that we're going to record here as a podcast.

0:55 Brad We're just going to tell you about what we just experienced.

0:57 Brad We just tried to take the How Many CTOs podcast website and update it a little bit.

1:03 Brad And it did,

1:05 Brad the AI did well at first, and then we asked it to do a second task and it got a little,

1:10 Brad it kind of went down a rabbit hole.

1:12 Scott Yeah.

1:13 Brad Right.

1:14 Scott Yeah.

1:15 Brad And, you know, I thought to myself,

1:18 Brad this is interesting.

1:21 Brad You could see an engineer spending a lot of time hugging

1:28 Brad at the AI trying to get it to do the thing as opposed to just stopping and saying,

1:35 Brad all right, I'm just going to code this part myself.

1:38 Scott Yeah.

1:38 Brad I can't seem to get the AI to do what I wanted to do, but then that brings up something interesting,

1:45 Brad which you talked about, which is getting the AI to do writing it myself means I have to know what I want to do and how to do it.

1:56 Scott Yeah. And so,

1:57 Scott you know, we did this little experiment as kind of a follow up from our interview with Michael Nacken. And one of the things he said, which I thought was really resonated with me, was this notion that the AI can get stuck.

2:12 Scott It can get it can get grounded on the thing it gave you.

2:15 Scott And the more you press on it to try to do something different,

2:18 Scott it just kind of gets more and more entrenched in it.

2:21 Scott And sometimes you have to say, forget that,

2:23 Scott I'm going to give you a new instruction.

2:26 Scott And to me, I think there's two things that are required from the engineer,

2:32 Scott that was us in this case, to know how to operate the AI agent.

2:38 Scott And the first one is knowing that.

2:41 Scott If you just keep pestering it and badgering it, it will just get more and more stuck in a rut.

2:48 Scott And so knowing that you have to stop when that happens and reset,

2:51 Scott that's like thing number one.

2:54 Scott And then thing number two is kind of what I think you just said, which is you also have to know really what you want and you need to be able to give it clear instructions about how you want it to go actually solve the thing that it's stuck on.

3:08 Brad Mm

3:08 Scott Um,

3:08 Brad -hmm.

3:08 Scott and,

3:10 Scott uh,

3:10 Scott you guys can watch the video,

3:12 Scott but in this particular case,

3:13 Scott the thing that is an interesting thing that neither Scott or I had thought about, I don't, because I've never tried to build a website for a podcast before.

3:21 Scott And so the idea of going, we wanted to basically add a link to listen to the episode on Apple podcast or listen to the episode on Spotify.

3:33 Scott And the AI didn't immediately know given our that we gave it the page, the show page on Spotify and the show page on Apple podcast,

3:43 Scott it didn't know how to map that onto the episodes.

3:47 Scott Right.

3:48 Scott And that's probably a case where as engineers,

3:52 Scott we need to go understand how that was going to work anyway.

3:56 Scott And if we were going to code it ourselves,

3:58 Scott we'd have to understand that and come up with a plan for that.

4:01 Scott And the AI wasn't any better at knowing how to solve that problem without us giving it better instructions.

4:07 Brad Yeah,

4:08 Brad there's like some,

4:08 Brad I don't know how to describe what the limitation is, but there's some sort of limitation about what it doesn't know everything about the world. It isn't like omniscient.

4:19 Scott Right,

4:19 Scott right.

4:21 Brad So there's another thought,

4:22 Brad you know, you shared.

4:24 Brad this idea of you know stopping and starting over again and really when you're AI or no AI sometimes you have to do that sometimes you have to say I need to stop you kind of get in a tug-of-war with what you're doing you get stuck on it and you my experience was I would have to stop and go do something else work on something else take a day sometimes take a day or two

4:49 Brad Sometimes you got to like let your subconscious mind do its work to see the problem in a different way as opposed to just continuing to pound,

4:59 Brad pound, pound, pound in

5:00 Scott Right. Totally,

5:01 Brad order to

5:01 Scott totally.

5:01 Brad get to the answer.

5:04 Scott Yeah.

5:04 Scott So in some sense, you're saying it's also as human, like, you know, flesh and blood intelligence, we also sometimes need to stop and do a reset as well.

5:15 Brad For sure.

5:16 Brad I've had that experience many times,

5:18 Brad which

5:18 Scott Yeah.

5:18 Brad is like, I don't, I don't, I can't figure this out. I'm just going to stop and walk away and just, I don't know. I find when I'm either walking the dog or doing the dishes,

5:27 Brad that seems to be when problems get solved.

5:29 Scott Epiphanies come.

5:31 Brad Yeah,

5:32 Brad I don't know. I'm just not thinking about it, but the brain, you know, the background processor is thinking about it or human background processor.

5:40 Scott Another thing that happened in our little coding session that we had there that I thought was really interesting for you,

5:47 Scott like I was just like,

5:48 Scott hey, let's just give it some instructions and hit the go button and see what it comes up with,

5:51 Scott because I kind of just wanted to see it run and just kind of.

5:55 Scott give it you know kind of free reign to go do whatever it wanted to and I

6:00 Brad Mm-hmm.

6:00 Scott felt confident that whatever it came up with I would be able to evaluate to be like is that make sense you know but you you had a reaction to that you were you were like a little bit you want to you know what was that reaction that you were having

6:14 Scott You know, when you were like, hey,

6:16 Scott I want to wait, wait,

6:17 Scott what code is it changing and what's it doing?

6:19 Brad Yeah, I guess, you know, in my work, in my career as an engineer, you know, an engineer coding.

6:28 Brad I don't think I was ever the best engineer,

6:30 Brad but I think one of the things I learned or I had come naturally but I'm not sure which one was to pay attention to detail and to know every line of my code and how it was working and

6:43 Scott Yeah.

6:44 Brad that was a way I guess that I tried to ensure quality is to be really thoroughly familiar with what was there.

6:54 Brad And then I think it gave me an advantage.

6:58 Brad you know as I say because you know moved into more like a lead type of role or whatever which is when people wanted to change the system I knew it really well how it worked in detail

7:07 Scott Yep.

7:07 Brad and so when we just asked the bot like go do this thing and it made a bunch of changes I felt uncomfortable I was like well wait a second how I don't know what this code does anymore and so then when like you know if if somebody comes to me and says hey Scott can the web

7:24 Brad the website do this now,

7:26 Brad you know, in that world where I was very intimately familiar with the code,

7:30 Brad I could say, well, yes, because I could already see in my mind how I would change it to make it do that.

7:35 Scott Yeah.

7:37 Brad Now, to be fair,

7:38 Brad I'd never seen this website before you wrote it.

7:41 Brad So you knew it was there.

7:42 Brad This was the first time I saw

7:43 Scott I

7:43 Brad it.

7:43 Scott mean, I didn't really write very much of it. It's a very much a very thin template of a blog platform using Svelte. That's it. It's not a lot of genius going on there.

7:54 Brad I understand.

7:55 Brad But even still, I did feel a little bit of like feeling it was a feeling in me, which is like, oh, I'm feeling uncomfortable with not knowing.

8:02 Brad what this is doing and

8:05 Scott Yeah, totally.

8:05 Brad I think if I use this now there's still going to be a pull request right in a merge and so I think in a world where there was more of this going on I would find myself wanting to spend more time with pull requests I worked with a place once where we had an expectation or working agreement we called it that if you were doing a PR

8:28 Brad I'm sorry, if you were reviewing a PR,

8:30 Brad you would actually pull the code down and run it on your machine locally.

8:34 Brad That was kind of just a, you can think of it as a form of QA that we put in place, right?

8:39 Scott Sure.

8:40 Brad And it didn't happen always. It was an expectation, but you can imagine there were lots of cases where someone just looked at it and said, this is fine and didn't feel compelled to do that.

8:49 Brad And, but you could see the importance of practices like that.

8:53 Scott Yeah.

8:54 Brad Now.

8:55 Scott Yeah.

8:56 Scott Well,

8:56 Scott and I think that that's a it's interesting that you say that tension like it makes a ton of sense that and this I feel like this is a little bit of what we talked about with Michael that that idea of, you know, the maybe junior engineer, but maybe this happens even with.

9:14 Scott uh veteran engineers like i don't consider myself a junior engineer yeah i've been doing this a little bit

9:20 Brad You

9:21 Scott a

9:21 Brad just look

9:21 Scott little while

9:21 Brad it it's just the

9:22 Scott i

9:22 Brad way you

9:22 Scott just

9:22 Brad look look the

9:23 Scott at i just look like

9:24 Brad baby

9:24 Scott a junior

9:24 Brad face

9:24 Scott engineer yeah um but uh

9:29 Scott Yeah,

9:29 Scott I had a reaction of like,

9:30 Scott I have trust in this thing. I've played around with it enough. I know what I just asked it to do. And it gave us a really good plan,

9:37 Scott right?

9:37 Scott Like you kind of said, like before we asked it to do what we asked it to do, right?

9:42 Scott Which was just to, you know, essentially embed and display the RSS feed for our podcast instead of just having a link to our podcast host.

9:53 Scott it came up with the plan that seemed like the right plan right like you and i've talked about this like Mm when you're going to go

9:58 Brad -hmm.

9:58 Scott write some code the first thing you got to do is you got to have a plan like what are we going to probably do it's going to download the rss and then parse it and put that in some html right so it had the right plan and that was believable and then it coded it but it wrote like 300 lines of code and your reaction to that was like

10:17 Scott What?

10:18 Scott Like,

10:18 Brad Yeah.

10:19 Scott yeah.

10:19 Brad Another thing it taught me, it puts in my mind more pressure on the test automation or whatever phrase differently. It puts more pressure on the QA practices that you have,

10:29 Brad whatever they are, whether they're test automation or human or whatever you got,

10:33 Brad however you do QA.

10:34 Brad Right.

10:35 Brad It puts more pressure on that.

10:37 Scott That's fair.

10:39 Scott You know, another thing that is interesting is in its first implementation,

10:43 Scott right?

10:43 Brad Mm

10:43 Scott Like

10:44 Brad -hmm.

10:44 Scott if you can go check the video because it's interesting,

10:47 Scott like it gave us some code and like we went and loaded the web page first. I like, no,

10:51 Scott Scott,

10:52 Scott let's just let's just run it. I'm just going to run it. Let's see what it does. And it generated an error and the error message was,

10:58 Scott oh, something's went something has gone wrong.

11:01 Scott Here's a link to the hosted site.

11:04 Scott And I actually thought that was kind of clever and I'm not even sure like we didn't ask it to do that and I'm not sure if I had coded it myself if I would have necessarily thought to do that that error handling that way,

11:15 Scott you know, but

11:17 Brad No, but it said it was going to do that.

11:19 Scott oh,

11:20 Scott it did say that in its It plans.

11:21 Brad did say in the plan that it would handle errors.

11:24 Brad Yeah.

11:25 Scott right i wasn't sure how it was going to handle the errors you know and

11:28 Brad That wasn't

11:28 Scott so yeah

11:29 Brad either,

11:29 Brad yeah.

11:29 Scott but you know but you know to me i do think that this is just one more example of like um i mean i think it did better work than i would have expected a junior engineer to do with the same level of instruction that we gave it it basically

11:43 Brad That's for sure.

11:45 Scott right

11:45 Brad We gave it three sentences.

11:47 Scott Yeah, we gave it three sentences,

11:49 Scott and it came back with a pretty solid plan.

11:52 Brad Mm-hmm.

11:53 Brad And then we gave it a slightly more complicated ask, which technically was probably impossible and

11:59 Scott Might have been impossible with what we gave it,

12:02 Brad couldn't

12:02 Scott and

12:02 Brad do it.

12:02 Scott it couldn't do it.

12:03 Brad All right.

12:05 Brad Well,

12:05 Scott All right,

12:06 Brad until

12:06 Scott this was fun.

12:06 Brad next time.

12:08 Scott All right.

12:08 Brad This is how many CTOs does it take?

12:11 Scott All right, Scott,

12:12 Scott talk to you later.

12:13 Scott This episode of How Many CTOs Does It Take podcast was edited by Saffron Epigop. Our theme song was AI Generated by Brad.

12:22 Scott Please follow us on all the social media platforms at at HowManyCTOsPod.

12:27 Scott You can find us on YouTube,

12:29 Scott LinkedIn,

12:30 Scott Instagram,

12:30 Scott Facebook,

12:31 Scott Threads, X,

12:32 Scott TikTok,

12:32 Scott and Patreon,

12:33 Scott or at our website at HowManyCTOsPod.com.

12:38 Scott And remember to rate, review,

12:40 Scott and subscribe on the podcast platform of your choice.

12:44 Scott Thanks for listening.