Bonus Episode: Challenges and Insights From Using Cursor to Add a Feature to Our Website

0:00 Brad: It puts, in my mind, more pressure on the test automation.

0:16 Scott: And these CCOs will share their stories untold.

0:20 Brad: Hey, Brad.

0:21 Welcome to a little bonus episode of how many CTOs does it take?

0:25 Scott: Yeah.

0:26 Brad: Yeah,

0:27 so everyone out there listening,

0:28 we just spent a little bit of time doing some coding with an AI agent.

0:35 You can go over to our YouTube channel,

0:37 just go over to YouTube,

0:39 look for how many CTOs does it take?

0:41 I don't think there's more than one of them out there,

0:43 so you'll find

0:44 Scott: How

0:44 Brad: us.

0:44 Scott: many CTOs does it take, Pod?

0:46 Brad: Okay,

0:46 Scott: But

0:46 Brad: how many we'll CTOs

0:47 Scott: link

0:47 Brad: does it take,

0:47 Scott: the

0:47 Brad: Pod?

0:48 Scott: video in the show notes of this little mini bonus episode that we're going to record here as a podcast.

0:55 Brad: We're just going to tell you about what we just experienced.

0:57 We just tried to take the How Many CTOs podcast website and update it a little bit.

1:03 And it did,

1:05 the AI did well at first, and then we asked it to do a second task and it got a little,

1:10 it kind of went down a rabbit hole.

1:12 Scott: Yeah.

1:13 Brad: Right.

1:14 Scott: Yeah.

1:15 Brad: And, you know, I thought to myself,

1:18 this is interesting.

1:21 You could see an engineer spending a lot of time hugging

1:28 at the AI trying to get it to do the thing as opposed to just stopping and saying,

1:35 all right, I'm just going to code this part myself.

1:38 Scott: Yeah.

1:38 Brad: I can't seem to get the AI to do what I wanted to do, but then that brings up something interesting,

1:45 which you talked about, which is getting the AI to do writing it myself means I have to know what I want to do and how to do it.

1:56 Scott: Yeah. And so,

1:57 you know, we did this little experiment as kind of a follow up from our interview with Michael Nacken. And one of the things he said, which I thought was really resonated with me, was this notion that the AI can get stuck.

2:12 It can get it can get grounded on the thing it gave you.

2:15 And the more you press on it to try to do something different,

2:18 it just kind of gets more and more entrenched in it.

2:21 And sometimes you have to say, forget that,

2:23 I'm going to give you a new instruction.

2:26 And to me, I think there's two things that are required from the engineer,

2:32 that was us in this case, to know how to operate the AI agent.

2:38 And the first one is knowing that.

2:41 If you just keep pestering it and badgering it, it will just get more and more stuck in a rut.

2:48 And so knowing that you have to stop when that happens and reset,

2:51 that's like thing number one.

2:54 And then thing number two is kind of what I think you just said, which is you also have to know really what you want and you need to be able to give it clear instructions about how you want it to go actually solve the thing that it's stuck on.

3:08 Brad: Mm

3:08 Scott: Um,

3:08 Brad: -hmm.

3:08 Scott: and,

3:10 uh,

3:10 you guys can watch the video,

3:12 but in this particular case,

3:13 the thing that is an interesting thing that neither Scott or I had thought about, I don't, because I've never tried to build a website for a podcast before.

3:21 And so the idea of going, we wanted to basically add a link to listen to the episode on Apple podcast or listen to the episode on Spotify.

3:33 And the AI didn't immediately know given our that we gave it the page, the show page on Spotify and the show page on Apple podcast,

3:43 it didn't know how to map that onto the episodes.

3:47 Right.

3:48 And that's probably a case where as engineers,

3:52 we need to go understand how that was going to work anyway.

3:56 And if we were going to code it ourselves,

3:58 we'd have to understand that and come up with a plan for that.

4:01 And the AI wasn't any better at knowing how to solve that problem without us giving it better instructions.

4:07 Brad: Yeah,

4:08 there's like some,

4:08 I don't know how to describe what the limitation is, but there's some sort of limitation about what it doesn't know everything about the world. It isn't like omniscient.

4:19 Scott: Right,

4:19 right.

4:21 Brad: So there's another thought,

4:22 you know, you shared.

4:24 this idea of you know stopping and starting over again and really when you're AI or no AI sometimes you have to do that sometimes you have to say I need to stop you kind of get in a tug-of-war with what you're doing you get stuck on it and you my experience was I would have to stop and go do something else work on something else take a day sometimes take a day or two

4:49 Sometimes you got to like let your subconscious mind do its work to see the problem in a different way as opposed to just continuing to pound,

4:59 pound, pound, pound in

5:00 Scott: Right. Totally,

5:01 Brad: order to

5:01 Scott: totally.

5:01 Brad: get to the answer.

5:04 Scott: Yeah.

5:04 So in some sense, you're saying it's also as human, like, you know, flesh and blood intelligence, we also sometimes need to stop and do a reset as well.

5:15 Brad: For sure.

5:16 I've had that experience many times,

5:18 which

5:18 Scott: Yeah.

5:18 Brad: is like, I don't, I don't, I can't figure this out. I'm just going to stop and walk away and just, I don't know. I find when I'm either walking the dog or doing the dishes,

5:27 that seems to be when problems get solved.

5:29 Scott: Epiphanies come.

5:31 Brad: Yeah,

5:32 I don't know. I'm just not thinking about it, but the brain, you know, the background processor is thinking about it or human background processor.

5:40 Scott: Another thing that happened in our little coding session that we had there that I thought was really interesting for you,

5:47 like I was just like,

5:48 hey, let's just give it some instructions and hit the go button and see what it comes up with,

5:51 because I kind of just wanted to see it run and just kind of.

5:55 give it you know kind of free reign to go do whatever it wanted to and I

6:00 Brad: Mm-hmm.

6:00 Scott: felt confident that whatever it came up with I would be able to evaluate to be like is that make sense you know but you you had a reaction to that you were you were like a little bit you want to you know what was that reaction that you were having

6:14 You know, when you were like, hey,

6:16 I want to wait, wait,

6:17 what code is it changing and what's it doing?

6:19 Brad: Yeah, I guess, you know, in my work, in my career as an engineer, you know, an engineer coding.

6:28 I don't think I was ever the best engineer,

6:30 but I think one of the things I learned or I had come naturally but I'm not sure which one was to pay attention to detail and to know every line of my code and how it was working and

6:43 Scott: Yeah.

6:44 Brad: that was a way I guess that I tried to ensure quality is to be really thoroughly familiar with what was there.

6:54 And then I think it gave me an advantage.

6:58 you know as I say because you know moved into more like a lead type of role or whatever which is when people wanted to change the system I knew it really well how it worked in detail

7:07 Scott: Yep.

7:07 Brad: and so when we just asked the bot like go do this thing and it made a bunch of changes I felt uncomfortable I was like well wait a second how I don't know what this code does anymore and so then when like you know if if somebody comes to me and says hey Scott can the web

7:24 the website do this now,

7:26 you know, in that world where I was very intimately familiar with the code,

7:30 I could say, well, yes, because I could already see in my mind how I would change it to make it do that.

7:35 Scott: Yeah.

7:37 Brad: Now, to be fair,

7:38 I'd never seen this website before you wrote it.

7:41 So you knew it was there.

7:42 This was the first time I saw

7:43 Scott: I

7:43 Brad: it.

7:43 Scott: mean, I didn't really write very much of it. It's a very much a very thin template of a blog platform using Svelte. That's it. It's not a lot of genius going on there.

7:54 Brad: I understand.

7:55 But even still, I did feel a little bit of like feeling it was a feeling in me, which is like, oh, I'm feeling uncomfortable with not knowing.

8:02 what this is doing and

8:05 Scott: Yeah, totally.

8:05 Brad: I think if I use this now there's still going to be a pull request right in a merge and so I think in a world where there was more of this going on I would find myself wanting to spend more time with pull requests I worked with a place once where we had an expectation or working agreement we called it that if you were doing a PR

8:28 I'm sorry, if you were reviewing a PR,

8:30 you would actually pull the code down and run it on your machine locally.

8:34 That was kind of just a, you can think of it as a form of QA that we put in place, right?

8:39 Scott: Sure.

8:40 Brad: And it didn't happen always. It was an expectation, but you can imagine there were lots of cases where someone just looked at it and said, this is fine and didn't feel compelled to do that.

8:49 And, but you could see the importance of practices like that.

8:53 Scott: Yeah.

8:54 Brad: Now.

8:55 Scott: Yeah.

8:56 Well,

8:56 and I think that that's a it's interesting that you say that tension like it makes a ton of sense that and this I feel like this is a little bit of what we talked about with Michael that that idea of, you know, the maybe junior engineer, but maybe this happens even with.

9:14 uh veteran engineers like i don't consider myself a junior engineer yeah i've been doing this a little bit

9:20 Brad: You

9:21 Scott: a

9:21 Brad: just look

9:21 Scott: little while

9:21 Brad: it it's just the

9:22 Scott: i

9:22 Brad: way you

9:22 Scott: just

9:22 Brad: look look the

9:23 Scott: at i just look like

9:24 Brad: baby

9:24 Scott: a junior

9:24 Brad: face

9:24 Scott: engineer yeah um but uh

9:29 Yeah,

9:29 I had a reaction of like,

9:30 I have trust in this thing. I've played around with it enough. I know what I just asked it to do. And it gave us a really good plan,

9:37 right?

9:37 Like you kind of said, like before we asked it to do what we asked it to do, right?

9:42 Which was just to, you know, essentially embed and display the RSS feed for our podcast instead of just having a link to our podcast host.

9:53 it came up with the plan that seemed like the right plan right like you and i've talked about this like Mm when you're going to go

9:58 Brad: -hmm.

9:58 Scott: write some code the first thing you got to do is you got to have a plan like what are we going to probably do it's going to download the rss and then parse it and put that in some html right so it had the right plan and that was believable and then it coded it but it wrote like 300 lines of code and your reaction to that was like

10:17 What?

10:18 Like,

10:18 Brad: Yeah.

10:19 Scott: yeah.

10:19 Brad: Another thing it taught me, it puts in my mind more pressure on the test automation or whatever phrase differently. It puts more pressure on the QA practices that you have,

10:29 whatever they are, whether they're test automation or human or whatever you got,

10:33 however you do QA.

10:34 Right.

10:35 It puts more pressure on that.

10:37 Scott: That's fair.

10:39 You know, another thing that is interesting is in its first implementation,

10:43 right?

10:43 Brad: Mm

10:43 Scott: Like

10:44 Brad: -hmm.

10:44 Scott: if you can go check the video because it's interesting,

10:47 like it gave us some code and like we went and loaded the web page first. I like, no,

10:51 Scott,

10:52 let's just let's just run it. I'm just going to run it. Let's see what it does. And it generated an error and the error message was,

10:58 oh, something's went something has gone wrong.

11:01 Here's a link to the hosted site.

11:04 And I actually thought that was kind of clever and I'm not even sure like we didn't ask it to do that and I'm not sure if I had coded it myself if I would have necessarily thought to do that that error handling that way,

11:15 you know, but

11:17 Brad: No, but it said it was going to do that.

11:19 Scott: oh,

11:20 it did say that in its It plans.

11:21 Brad: did say in the plan that it would handle errors.

11:24 Yeah.

11:25 Scott: right i wasn't sure how it was going to handle the errors you know and

11:28 Brad: That wasn't

11:28 Scott: so yeah

11:29 Brad: either,

11:29 yeah.

11:29 Scott: but you know but you know to me i do think that this is just one more example of like um i mean i think it did better work than i would have expected a junior engineer to do with the same level of instruction that we gave it it basically

11:43 Brad: That's for sure.

11:45 Scott: right

11:45 Brad: We gave it three sentences.

11:47 Scott: Yeah, we gave it three sentences,

11:49 and it came back with a pretty solid plan.

11:52 Brad: Mm-hmm.

11:53 And then we gave it a slightly more complicated ask, which technically was probably impossible and

11:59 Scott: Might have been impossible with what we gave it,

12:02 Brad: couldn't

12:02 Scott: and

12:02 Brad: do it.

12:02 Scott: it couldn't do it.

12:03 Brad: All right.

12:05 Well,

12:05 Scott: All right,

12:06 Brad: until

12:06 Scott: this was fun.

12:06 Brad: next time.

12:08 Scott: All right.

12:08 Brad: This is how many CTOs does it take?

12:11 Scott: All right, Scott,

12:12 talk to you later.

12:13 This episode of How Many CTOs Does It Take podcast was edited by Saffron Epigop. Our theme song was AI Generated by Brad.

12:22 Please follow us on all the social media platforms at at HowManyCTOsPod.

12:27 You can find us on YouTube,

12:29 LinkedIn,

12:30 Instagram,

12:30 Facebook,

12:31 Threads, X,

12:32 TikTok,

12:32 and Patreon,

12:33 or at our website at HowManyCTOsPod.com.

12:38 And remember to rate, review,

12:40 and subscribe on the podcast platform of your choice.

12:44 Thanks for listening.