The Reality of AI in Software Engineering and the Future of Work
- 00:00 Introduction: The Next Revolution
- 00:20 AI Coding Experiment on Upwork
- 02:35 Results and Analysis of AI Performance
- 03:30 Challenges and Limitations of AI
- 06:09 AI in the Workplace: A New Mandate
- 08:12 The Rise of Vibe Coding
- 11:00 AI Natives vs. AI Immigrants
- 12:41 The Hype Machine and Future of AI
- 15:22 Energy Consumption and Environmental Impact
- 19:50 Conclusion: The AI Revolution Continues
Resources:
- How Many CTOs Pod: https://howmanyctospod.com
- Scott Porad: https://www.linkedin.com/in/scottporad/
- Brad Hefta-Gaub: https://www.linkedin.com/in/bradheftagaub/
- OpenAI Research Paper: https://openai.com/index/swe-lancer/
#podcast #technology #leadership #ctos #technicaldebt #management #leadership #problemsolvingtechniques #problemsolvingstrategies #engineering #entrepreneurship #engineeringmanagement #ai #aicoding #vibecoding #aiassistedcoding
Brad: And this is the next revolution.
Scott: We're gonna kill them, protect me,
and throw a message, and tell us every tale in the comments,
so we'll start a few big names and a few times for how many CTOs will share the stories untold.
Brett,
Scott.
Brad: An article came across my desk this week about AI coding that I thought was very fascinating.
Scott: Yes.
Tell me about it.
Brad: Okay,
so OpenAI,
the company behind ChatGPT, wanted to find out how effective their models were.
So they went out to Upwork and they got...
1,488 jobs, freelance software engineering jobs off of Upwork.
These were worth a collective amount,
a million dollars.
They ranged from $50 bug fixes to $32,000 like entire projects.
And they fed these jobs into two of their models and into Claude.
to see how effective AI could be doing this.
And they did this because a lot of the tests are versus sort of contrived greenfield type of projects and they wanted more real world examples,
more diverse examples to really test their model.
Would you care to venture a guess at what percentage of the jobs were successfully completed?
By the models,
the AI coding agents.
Scott: Let me think about this.
I'm going to try to use the hype scale and the new trend towards vibe coding to predict the success of this.
Seeing how OpenAI just raised.
more billions of dollars on an even larger market cap and they keep saying that uh you know we're going to replace all engineers you know with this i gotta imagine it was 100 right scott like they were completely successful in all of these things
Brad: It was not 100%,
Scott: wait
Brad: but
Scott: what i
Brad: I think you're joking around.
So what do you think it really was?
Scott: don't know in my experience if i ask it to do small well we did a test on this right we asked we asked ai to update our website in three sentences it did a very simple thing very quickly and well and then we gave it a slightly more complicated thing and it went off the rails so at least 50 50
Brad: Okay,
so credit to OpenAI.
I'm just going to read you what it says.
Results indicate that the real world freelance work in our benchmark remains challenging for frontier language models.
The best performing model,
Claude 3.5 Sonnet, so not their model,
resolves 26.2.
percent of the issues yeah
Scott: 26% of the issues,
a
Brad: a
Scott: quarter of the time it succeeded.
Brad: quarter of the time it succeeded higher reliability is needed for trustworthy deployment they
Scott: Yep.
Brad: said that right here in the paper that's pretty interesting if i had an engineer who only completed 26 of their tickets successfully
Scott: And the others, other 74%.
What caused
Brad: unsuccessfully.
Scott: the bugs?
Unsuccessfully? Couldn't do it?
Brad: Yeah, I don't think they would last very long.
Scott: They'd be on a pip.
Brad: They would be on a pip.
So here's a little bit further data just digging into it.
It was most successful at server-side logic.
That's 42%. It was least successful at system-wide quality and reliability tasks.
Scott: Hmm.
Brad: Now, that's a small sample set.
There was only,
I can't tell if it was either three or five in that sample set,
so that's not very many.
Then next up at 24% was client-side application logic.
At 32% was UI UX.
Those were on bigger samples.
Scott: Interesting.
Well,
what I was going to say.
This maybe speaks to this.
Um,
so I've been using a ton of cursor lately,
um,
Brad: Mm
Scott: in agentic
Brad: -hmm.
Scott: mode.
And, um,
so what that means of course, is that when I get it, when I ask it to do something,
it kind of goes off and starts doing it and try and do things like run its own tests or run its own linting or run its own compile. And then whatever it sees the problem is, it says,
Oh, let me fix that. And it kind of runs in these loops where it tries to.
to kind of keep going until it completely solves the problem.
And one of the things I've started doing now when I use it is I explicitly tell it, give me your proposal,
do not implement it until I agree with your proposal.
because and and because it often will get stuck in a loop like i've seen it where like it like for example for whatever reason in our code base we have pretty strict linting rules and it's you know a lot of typescript and um the agent will like write some code it'll lint it
the lint will fail with a bunch of errors about you know some of our strict rules like you know unsafe any's or whatever and it'll go and try to fix the linting error and it'll create more lint errors so then it'll go try to fix those lint errors by reverting back to the original code which then has the original lint errors and then this gets stuck in this infinite loop of like beating itself up trying to
undo what it already did and so much so that I've like I said I've started this practice of basically saying do not proceed do not pass go do not collect $200 until I tell you that you can proceed
Brad: Well, yeah,
that's smart because it needs that.
And we would do that with our junior engineers also.
Go make me a plan. Let me look at it before you go.
You know, work on it.
Scott: Yep.
Brad: So I saw this,
it came across my desk. And when I say it came across my desk, I mean, I saw it in my LinkedIn feed.
And I thought it was interesting in regard to the context of the announcement made by Spotify last, not Spotify,
Shopify last
Scott: Yeah.
Brad: week, where the guy who was the CTO says, or maybe it was the CEO,
I can't remember.
Scott: It was CEO,
yeah.
Brad: Prove before you hire someone you have to prove AI can't do it.
Scott: Oh, I didn't see this one.
Brad: Yeah,
that's what they said before you hire someone you have to prove that AI can't do it
Scott: Oh,
interesting.
When I saw another memo that was something from the Shopify CEO that was like, I use AI and I've become using it for everything I do every day, and I expect it.
You know, you all of you should be doing that.
You should all be using AI for everything because learning how to use AI is a skill. It doesn't do it on its own. You need
Brad: Yeah
Scott: to know how you need to have the skill to know how to use it.
And it's, you know, that memo was kind of like I had previously encouraged people to research this.
Now I am mandating you will be using it in all of your.
in
Brad: Yeah,
Scott: all your tasks.
Brad: it is a skill. And that's absolutely right.
I don't know that it's going to be I'm not yet convinced it's going to replace people in the same way that others are.
I've definitely found it saved me tons of time on some stuff.
I still have to edit and own what it's doing.
I can't go blind.
I've
Scott: Yeah, 100%.
Brad: used the chat GBT deep research for some interesting things.
It's surprised me, but not for coding.
You know, when you do it to get yourself some information,
there's it doesn't have to be perfectly right.
Code is exacting. It has to be right.
There's no or it's a bug. Right.
So but this this shows that it's not there yet to replace people,
I don't think.
Scott: I think it's a long ways away from.
replacing um engineers and i think that let's talk about vibe coding for a second right so uh vibe coding is all the rage now where um a combination of people that are not technical at all are using things like lovable or other vibe coding tools and they're just kind of just chatting with the ai telling it to write stuff and they're getting code and then they run it in their sample you know the ai will run it you know like
Brad: Mm
Scott: literally
Brad: -hmm.
Scott: with curse
cursor in agentic mode it's like you could start with a blank directory and tell it to build a you know a full stack web app and it'll set up docker containers and set everything up and go boom go go to your local host you know 5147 and and here's your web app right and
Brad: Yeah
Scott: it it definitely if you're not a developer who's familiar with things like where do you put your api keys don't put your api keys in your source
source code you know in your javascript right uh and you don't know that it that's what it does for its first implementation it just does that right and so there was a i just saw this post on um
I don't know if it was on X or LinkedIn,
but somebody who basically in 47 minutes, who's not necessarily a, you know, a red team or a hacker type person in 47 minutes basically found exploits in the top 20 lovable apps that are out there on the website. And it was things like API keys were exposed,
you know, prompts were exposed,
people's personal.
personal information was exposed including bank account information and it's it's these types of things that I think are the other side of this coin which is you can go really fast to get a prototype up and running
Because the agents will do that for you. But if you don't have the wherewithal and the technical acumen to say,
wait a second,
don't put my API keys in the source code.
You don't know what you don't know.
Brad: Yeah,
that's exactly right.
Or I've been finding this. I was doing a little thing yesterday. I was trying to get Gemini to write some code for me. I wasn't trying. I was having it do it.
And it just made a mistake.
And this thing I was trying to run it wasn't working and I went back to the Gemini chat I was like, are you sure that's right?
And they were like, oh, yeah,
that good catch.
That's that's not right.
And not I knew a little bit of what I was doing.
It saved me a ton of time because I got me the scaffolding. But when I tried to run it and then it wasn't it was giving me a strange error.
I knew how to sort of debug it a little bit. So.
We'll see.
I do agree with whatever that was saying. It is a tool that people are going to have to learn.
I've been talking recently to people,
you maybe remember this, but I'm going to have to dig up the article from the olden times about digital natives versus digital immigrants.
Scott: Right.
Brad: And that was when people were talking about the internet.
And for anybody listening who maybe doesn't know,
a digital native is somebody who grew up with the internet.
and a digital immigrant is somebody who sort of came of age before the internet and then immigrated to the internet.
Brad,
you and I are both sort of digital immigrants.
Well,
we're bread on the cusp actually.
It's like we were brought to the country at a young age.
But like my dad,
so immigrants have accents.
My dad, of course, he's 80 and he is a digital immigrant for sure and he has an accent. You know what his accent is?
He types in all caps too often,
you
Scott: Right.
Brad: know.
um because he writes in all caps that's how he writes he writes in all caps why wouldn't i write why wouldn't i type that way so um well we're doing the same thing again with ai right
Scott: Right.
Brad: you know uh i am going to be in you and i are going to be ai immigrants right we spent a whole life whole career before ai came along there are people who are in the eighth grade today who are going to be ai native and um it's just gonna be part of their lives and they're going to know how it works
And so it's an interesting thing, I think.
So the point, the reason I think it's interesting is coming back to what you were saying about,
I don't remember what you're reading saying,
I use this every day,
I'm learning how to use it.
Just like an immigrant has to learn the language and learn the society and the norms and all these things, it's kind of a similar kind of thing.
Scott: Yeah, I mean, it's interesting.
There's a part of me that wonders.
will the you know there's no doubt that there's the huge hype machine right the hype machine talks
Brad: I've
Scott: about how
Brad: never seen so much hype.
Scott: yeah it's it's the hype machine um and you know i mean i i've told you this before too like when folks come to me and say oh what are we going to do about the national debt you know eventually everything the economy is going to collapse because of the national debt and i'm like
I, you know, I think I think a major revolution in work that comes from AI is more likely to arrive sooner than national debt.
Right.
Like and so now,
Brad: Yeah.
Scott: is that part of the hype machine?
Is that me falling for the hype or is that me just saying, wow,
man, I I can see what's happened in the last two years of using this stuff.
That trend line is moving very quickly and it's reshaping the way people work.
It's reshaping the way that kind of white collar work is done.
Brad: So a commenter on a post of mine on LinkedIn described it as the industrial revolution.
So we had the Industrial Revolution, of course, prior to that we had the Stone Age, the Iron Age and all these things, right?
And then we had the Information Revolution. That's what we were just talking about a few minutes ago in terms of the Internet.
And this is the next revolution. It's not the industrial revolution.
Maybe you and I can become the guys that coin the name.
A
Scott: That's
Brad: revolution.
Scott: what it will be.
Brad: Because AI revolution isn't sexy enough. We need a better name. Remember there was the information superhighway. We need to come up with something like that for this.
Scott: It's a bunch of tubes.
Brad: It's a tube revolution.
I want to ask you something about cursor.
Do they really call it agentic mode?
Scott: Yes.
Brad: Okay, I've got some grammar issues with that. It should really be agent mode.
Scott: Uh,
let me just double check.
I'm pretty sure they call it Agenic Mode,
but um...
But
Brad: Because
Scott: you
Brad: in
Scott: might be right.
Brad: our language that I see that tick at the end means like, right?
It's like an agent.
But this
Scott: Oh,
Brad: is an agent.
Scott: maybe they call
Brad: It should
Scott: it
Brad: just be,
Scott: agent.
Brad: they call it agent.
Scott: They call it agent.
Brad: Good for the grammar officials at Cursor.
I was worried about that business.
Well,
the problem is we've heard Cursor has been made like by three people,
right?
And a bunch of agents.
You hear about them. They got a billion dollars of revenue with three people.
people the problem is they didn't actually go hire a grammar person and that's Yeah, the problem
Scott: maybe.
That could be.
Brad: so i want to since we're on this ai topic in our little mini bonus episode here i want to tell you the thing that's actually been really bugging me lately
it's i mean i worry about the jobs and careers and what this is going to mean for our industry but the thing that actually really is making me uncomfortable is is the climate change piece of it the energy consumption i mean every one of these queries is you know like running a light bulb for three hours and you know i can't only imagine how much one of these deep research queries costs that's
Scott: Yeah.
Brad: cost a fortune and now we're just sitting here typing it all day long just burning energy
Scott: Yeah. Well,
I agree that's concerning.
I don't have an answer to what to do about that.
It's a little bit like the genie's out of the bottle.
Brad: She knows
Scott: I
Brad: the bottle.
Scott: think that, you know, maybe one of the things to consider is that this does, maybe this is part of the answer of.
How do we really commit to these reusable forms of energy,
you know,
Brad: For
Scott: or,
Brad: sure.
Scott: you know, some of the less polluting forms of energy, you know, without not trying to get political on that, but I do think that that's one of the one of the things we know is that technology.
almost always drives other technology,
right?
Brad: Yeah, that is true.
Well, look, if the cost of carbon emission were in the price of energy,
it would sell itself in an instant.
Scott: That's true.
Brad: And I think we're all getting away on a free ride where we're not having to pay the full price.
I think that stepping away from energy for a second, I think we're.
we'll start to see a correction is at some point these companies are going to have to make money i mean google and microsoft they can fund this stuff forever but open ai can't raise 80 billion dollars a year i can only imagine how much they're burning every month they must be burning four billion four billion a month
Scott: Yeah, I don't know. I don't I haven't looked at their financials, obviously,
but it's it is interesting.
And that's kind of the point about the hype machine. Right. So like, I think that the
This is a classic case where the middle always loses,
right?
Like these types of technology revolutions, the big established players are the ones that have the advantage to actually be the providers of the platform or the technology.
And then there are real benefits for the bottom of the curve where it's like those of us that just get to use it,
Brad: right? Mm
Scott: So the
Brad: -hmm.
Scott: two guys in the garage or, you know, whatever,
the one artist basically gets to use it.
use it and the technology makes them become more effective and potentially you know have success that way so those are the two ends like the huge corporations of the Googles and the Microsofts and the bottom of the curve with the individuals who get the benefit but anybody that's in the middle is going to get squashed because they're you just can't compete with the big guys right
Brad: Yeah, or they're going to find some way to make it more energy efficient.
Right,
and they're gonna the big guys in the middle.
Scott: Yeah, I don't know.
I think the second that the second use anybody that solves any problem in the middle that's in any way interesting in scale.
immediately usurped by the big guys right
Brad: That's true.
That's true.
Scott: that's the challenge and that's part of the reason i would argue that's one of the reasons for the hype cycle right the one reason for the hype cycle is to further protect the big guys and make it that much you know create these you know essentially these barriers to entry to becoming bigger because if they can hype it up then
Brad: I see the most hype coming from people in the middle who have something to sell.
Scott: You're kind of considering OpenAI to be in the middle in this case because they're too small.
Brad: Yeah.
Scott: They're not the Microsoft.
Brad: No,
I mean, they're huge.
No,
but like if I just look at my LinkedIn feed,
the people who are posting the most about AI is going to replace everything are,
if you look, what company does this person work for?
They typically work for some company that sells some AI.
thing so
Scott: Yeah.
Brad: they
Scott: Yeah.
Brad: got a vested interest there it's
Scott: Yeah,
for sure. Well,
that's called capitalism.
Brad: called capitalism all right should we call it a day
Scott: Let's call it a day on the mini episode,
the mini bonus episode on this week in AI.
Brad: And this week in AI,
I almost feel like we could do this every week, do a little this week in AI.
I love it. A bonus episode
Scott: this All right.
Brad: week in AI only for the Patreon supporters.
Gang,
I want everyone who's listening to know that you still have the opportunity to be our first Patreon supporter.
And it costs nothing.
We are free on Patreon right now.
So you could be number one. And the first three people.
where our Patreon supporters,
if I recall,
get a boat ride this summer.
Scott: They get a boat ride on your boat.
Brad: That's right.
Scott: All right. Great talking to you,
Brad: See
Scott: Scott.
Brad: you, Bye. Brad.
Bye.
Scott: This episode of How Many CTOs Does It Take podcast was edited by Safran Epigal.
Our theme song was AI Generated by Brad.
Please follow us on all the social media platforms at at HowManyCTOsPod.
You can find us on YouTube,
LinkedIn,
Instagram,
Facebook,
Threads, X,
TikTok,
and Patreon,
or at our website at HowManyCTOsPod.com.
And remember to rate, review,
and subscribe on the podcast platform of your choice.
Thanks for listening.