Embracing Failure and Innovation: Lessons from Punk Rock with Kevin Goldsmith
- 00:00 Introduction and Guest Welcome
- 00:55 Record Label Stories
- 02:42 Kevin's Career Journey
- 07:34 Writing 'It Depends'
- 13:19 Lessons from Failure
- 18:04 Spotify Retrospectives
- 25:02 Feature Requests and Fun Stories
- 27:42 Exploring Music Algorithms
- 28:36 Technical Challenges and Business Implications
- 29:23 The Social Aspect of Music Sharing
- 31:29 Managing Multiple Spotify Accounts
- 33:00 Tackling Tech Debt in Monolithic Systems
- 42:23 AI Assisted Coding: A Tool for the Future
- 48:40 Music Recommendations and Closing Remarks
Resources:
- How Many CTOs Pod: https://howmanyctospod.com
- Scott Porad: https://www.linkedin.com/in/scottporad/
- Brad Hefta-Gaub: https://www.linkedin.com/in/bradheftagaub/
- Kevin Goldsmith: https://www.linkedin.com/in/goldsmith/
- It Depends Book: https://itdependsbook.net/
- Tacocat: http://tacocatdotcom.com/
- The Telescreens: https://linktr.ee/Telescreens
Guest
Brad: But when you make a mistake,
how do you then take the time to learn the lessons from it?
Kevin: Welcome to the show,
protect me from the rest of the universe never tells in the coming song Which are two big names and a vision time for how many CTOs will share their stories untold
Hello, everybody.
Welcome to.
This episode of How Many CTOs Does It Take?
We have got a very interesting guest today. I have got two guests today. I've got my regular co-host Brad Hefta-Gaub, who when he was younger started his own punk rock record label. And we've got with us...
Kevin Goldsmith who also when he was younger started a record label and has had a notorious career after music in the tech industry and has written a book that we're going to get to hear about today called It Depends but I think the place we've got to start with Kevin before we have you introduce yourself is what was the name of your record label and what kind of music did you have on it
Brad: So the record label was called Unicircle Records because it was kind of punk rock,
R-E-K-K-I-D-S,
and it featured,
it was, the motto is called Sound Art,
Art Rock.
So the most famous band on the label was this band called Bethany Curve from Santa Cruz that was,
or I guess kind of is.
more shoegazy but there was a lot of experimental stuff a lot of kind of art rock stuff bands from all over the country so it was pretty active for through the 90s
Kevin: That's cool.
Scott: Very cool.
Kevin: And Brad,
how about your record label?
Scott: My record label was called Carving Knife Records,
started as a classic.
you know i was in a band called three blind mice and so started the whole diy do-it-yourself label for your own band uh but um then i realized that my friends were actually much better musicians than i was and i started putting out their records so it was all pretty much punk it was in the late 80s early 90s of seattle scene so it was like pre grunge um and we were on the more punk side of things so probably the biggest band was a band called monster
your truck driver and
Kevin: Oh, of course.
Scott: another great band called cat food which was an all-girl punk band
Oh,
Kevin: I love it. Well,
I'm the only guy in the room without a record label.
I'm going to have to get to work on that.
So, Kevin, welcome.
Tell us about yourself.
So what's your how did you arrive on this podcast?
You started you're born somewhere and you're here today.
What happened in between?
Brad: Oh,
gosh.
OK,
well,
that's all right. Let's see how many seconds I can put 55 years into.
I got into computers when I was a little kid and my school got a TRSAT model one.
And so that was my first exposure,
really got into programming, really enjoyed it, decided that was going to be when I found out I could actually have a job doing it. That was like the best news I ever got.
Got really focused on it,
went to college for it, ended up originally.
In college I got into graphics, so I was doing a lot of computer graphics stuff,
ended up working for a company,
my first job after college was at this company called Silicon Graphics, so it was a pretty good place to be doing graphics in the early 90s.
And then went into commercial animation. So I worked on TV shows and films for a little while and then ended up working at Microsoft in the Microsoft research team for a while and then on Windows Media.
And then that was kind of 2000,
early 2000 left like everybody,
a lot of people did if Microsoft went to go do a startup.
Also in that media encoding space. So I did a couple.
a couple startups that all fell apart uh around uh when all of it fell apart ended up back at microsoft and then uh like michael neck and who you just had i was at adobe i left microsoft went to adobe it was there for nine years and so i've been
Kind of a, I'd been a lead and I'd been kind of, I kept going back and forth between like,
oh, I'm going to be a senior developer and I guess I'm managing people now,
but it was never like a choice.
It was just like, Kevin, you now manage people.
So I wasn't a great manager,
but at Adobe, I got much more serious about being a manager.
So I was there nine years,
started as just a developer or senior developer.
left as a director and then ended up at Spotify and so then I was at Spotify in Sweden for a few years started there as a director ended up leaving as VP and then came had came back to Seattle and started as a started being a CTO and so I've been a CTO for the last nine years at different companies so
After kind of music,
I went, I had my journey in the wilderness where I did sort of legal tech and edutech and fintech and data science.
So I was at different companies and then kind of ended up having opportunity to come back to music.
And so for the last two and a half years, I've been CTO at a company called DistroKid where we do music.
music distribution.
So kind of got found my way back into music, which which I've been really digging.
Scott: I love it. You said you went had a journey into the wilderness and then you listed like a whole bunch of random fields.
I feel like I've done that before.
Brad: It was a thing where, you know, I'd been like you, I'd been in a lot of kind of media. I'd been in media, I'd been in 3D graphics and digital audio and digital video for a long time.
And but coming back to Seattle.
because you know this is where you know my family is it is um come back to Seattle the choices were there wasn't a lot of choices right especially around that time like 2016 especially if you wanted to be in a startup and so yeah I I did my time in in these other fields but it was really it ended up being really
really valuable and really useful and so I'm glad I did it but it's kind of nice to be back you know be back in my home kind of area again
Scott: Totally.
Kevin: I was just thinking today,
I can't even remember what the topic was, but I was just thinking about how if I hadn't had this engineering leadership role at three or four different companies, I guess this is the fourth one now.
I just don't think I would, I mean, I'm not trying to toot my own horn, but I just don't think I'd be as good at it as I am.
Because each one you make, each one's different.
So
Scott: Yep.
Kevin: you have to learn how to like apply the principles differently.
And you make mistakes in each one. So you kind of get a little bit of a redo,
a little bit each one.
And I think that jumping around does have some value.
Scott: Totally.
Kevin: Okay,
so Kevin,
along the way, I know that you wrote a book, which I'm embarrassed to admit,
I never bought or read called It Depends.
And tell us all about that, because I think all of us as engineering leaders, somebody have said, asked us some question and we have answered,
it depends.
And so you nailed it on the head with the title.
What is what's that all about?
Brad: You mean, there
Kevin: Right
Brad: you go.
Kevin: there.
Brad: I mean, this book, Scott?
Kevin: That book.
That's
Brad: This
Kevin: what looks like a hardcover.
Brad: is a hardcover.
Kevin: Where can someone get the hardcover?
Scott: Signed by the author.
Brad: There is ways to get one signed by the author.
So what this book came about for I've been blogging as a kind of especially I started really at Adobe kind of doing my.
there was a decision I made where instead of being sort of an accidental manager or having management thrust upon me there was a point there where I actually decided you know what actually this is I want to make the switch like I'm going to get as serious about being a manager as I was about being a developer and and really build you know build my skills and build a craft here and so that I started blogging because at the time
So this is like 2005-ish is when I started doing this. There really wasn't a lot of stuff.
There was Michael Lop's blog and like he had his book out, but the books were really weak,
really lame, very kind of 80s style old school.
And there was a couple of people besides Michael Lop blogging, but not a lot.
And so I thought,
OK,
well,
I'm going to document sort of the things I learn as I go. So I started blogging then. I'm not a huge, to be fair, I'm not like a massive.
as a writer there's other people now that write daily and stuff i don't i don't like writing i like writing code not words but you know when i would find something that i would feel like oh i just really learned something or
this just happened and and this is maybe interesting to other people i would write it down and share it and then a few years ago when we all had a lot more time than we were used to having um in addition to like you know buying way too much euro rack gear um i started actually realizing like you know what i've actually written a lot of stuff and this might be valuable
to kind of put together so if you think of like really you know there's really well-known books that are really just collections of blog posts and like oh i could i've i've got a blog post i could do that
And so that was the thought is I hadn't written a book. I'd thought about writing a book, but writing a book was really daunting and thought, well,
you know, maybe a good way to start is to take the things I've already written, clean them up a little bit. I'm a much better writer now than I was 10 years ago.
So clean them up a little bit and then collect them and have that be the first book.
So that's what it really is. It's a collection of.
My blog posts from 2012 to 2022,
and the name came about, like you said, like.
People would ask me, so I mentor a lot of people over the years, like I really like mentoring kind of up and coming folks, especially folks who are kind of not coming out of non-traditional backgrounds or are from underrepresented groups.
And so, you know, I would be talking to them and they'd say,
oh, you should write a book someday or they'd say, or if you were going to write a book, what would it be called? And I'd say, well.
well probably be called it depends and i'd said that for years and years as a joke because that is like my number one answer to any question is oh how do you how many developers you know what's a qa to developer ratio on a team like well totally depends right what kind of team is it what's it doing and
Scott: Right.
Brad: so it's such a it's for all of us it's just like the number one answer we say is so every answer is so
content uh context dependent all right well it just made sense as a book and so that's where the title comes from my
Kevin: So I would imagine each blog post is sort of its own chapter Do you have a favorite?
Brad: favorite one in there is uh it's is it the first gosh it should be i think it's the first which is uh it's uh yeah fail safe fail smart succeed
Um,
and
Kevin: Mm
Brad: the
Kevin: -hmm.
Brad: title is really lame. I'm really bad at titles, but the title is really lame, but it's essentially I did for, I also give talks and I gave for years and years, I gave this keynote talk that was essentially talking about.
how we learn well from failure and how to fail well right so if we're if we have to fail to learn and we have to fail to succeed to do anything interesting like how do you fail well like how do you set up the conditions in your team to fail well and it uses a lot of examples of stuff we did at spotify because spotify was we that was really well set up like we were really good at failing smartly and so uh i decided you know i
I've been giving it for a long time. I'm like, oh, I should probably write it down so that, you know, I can refer to that.
And that took me.
like a year and a half to take i started with oh i'll just take a transcription of one of the keynote talks i gave and i'll just edit it and that that was the hardest chapter to write which is the saddest because it was it was a talk i'd given probably a dozen times but and it's and it is an edited transcription and just took forever it was so painful but i'm really happy with how it came out
Scott: Yeah.
Kevin: I mean, I have to ask, what are the conditions that
create successful failure.
Brad: So.
The important stuff is, I think, would be obvious to anybody, which is like, how do you fail fast?
How do you fail fast while being confident that you're not failing too fast, right?
So part of it is setting up how do you do feature development?
Some of it comes from a lot of kind of lean startup thinking,
but we had it set up at Spotify where product development was set up into very specific structured stages.
pages and it was important enough at the company it was sort of one of the company mottos think it build it ship it tweak it and so
Kevin: the Hmm.
Brad: idea was
you know how fast could you if you think about investment in a feature or investment in a product when you're just thinking about and you're talking about it you like you a product manager a couple people okay well that's not a lot of investment it's not a lot of time spent then you start building it now you're actually starting to do a lot of investment and then you know up until from the time you think of it to the time it's actually in front of real customers that's just all
all lost money that's all just invested cash and so the question was how do you if you visualize it as a curve which is what i do in the chapter how do you minimize the area under that curve and so there's a bunch of tools that are real obvious like how do you ship an mvp what's the right mvp to ship what's the minimum version of this that you can get in front of people and start getting data back that's an important part of it but then it's also
How do you decide as you go forward,
when are we done investing in this?
And then how do you know whether you're hitting sort of a local maxima or whether you really actually got the value out of it?
So there's a lot behind that thinking, but the core of it is how do you do things in such a way so that you are constantly validating your hypotheses? And how do you also the hardest part?
in any company is this,
this isn't working.
and then the product manager is like oh no it really is like no no you did it wrong now maybe we need to keep working on we need to keep working on it's like no like how do you make that decision of no this is it we're done like hey we thought it was a good idea it wasn't let's make sure we've learned from it that's the other part is when you fail it's really easy to or you you want to sweep it under the rug like no one likes to dwell on their failures everybody likes to think about how
how great they are.
But when you make a mistake,
how do you then take the time to learn the lessons from it?
And I think another part of how do you fail well is making failure not a big deal.
So one of the things that happened when I joined Spotify,
sorry,
I'm rambling, but I feel
Scott: No,
Brad: very
Scott: you're not. This is great,
Brad: passionate
Scott: though.
Brad: about this.
One of the things that I remember,
so right after I joined Spotify,
I think it was right when we got,
was it Led Zeppelin? I think it was when we got Led Zeppelin on the platform.
So that happened kind of right after in my team,
right after I joined.
so i was there for part of it you know a lot of the negotiation part i wasn't there for but the actual like oh they had a bunch of requirements and we got them all done we launched them on the platform because they were one of these at the time you know a lot of bands didn't want to be on streaming um and so we had to do this a bunch of times with different kind of major artists but they the result was huge success
Right. So I can't take all the credit for it because I wasn't there for most of it. But I got there massive success.
The label sent us like cake and liquor and like they were so overjoyed with how it went, like unqualified success because of music.
Of course, they sent us liquor.
And then I'm like,
okay, cool.
Like, yeah, we had the big party.
We're awesome.
Like, let's move on.
And then the next day I got a big meeting, like time to like do the retrospective.
And we then spent like four hours going through everything that went wrong.
in this highly successful process and this highly successful launch because we wanted to make sure like the next time we did it we wouldn't make those same we'd make new mistakes we wouldn't make the same mistakes again and I'd never seen that any company right you you do the post mortems or you do the after actions or whatever when something goes wrong but to have some to go through that process when something goes right I'd never seen it
company before,
but that was sort of the essence of the way Spotify deals with stuff like that.
Scott: I think that's such a great story,
Kevin.
I am, you know, even on the small scale with the small team that I'm working with now, we're like pretty religious about doing our retrospectives and I'm always reminding them like.
And when the sprint goes great,
Yeah. that's when it's the most important for us to do the retrospective because we really need to ask ourselves, did we just get lucky or did we actually do something that made this sprint go better than the last sprint or this sprint stand out? And that's something I kind of learned that from I do a bunch of mountain sports and traveling in avalanche terrain and in, you know, in,
you know, low information, high consequence.
environments
Brad: Yeah,
Scott: you
Brad: yeah
Scott: just got to ask and i and i feel like software and business is a low information high consequence you know environment and you have to you really have to interrogate like
what actually happened here and what can we learn from
Brad: Absolutely. And the other important part of that that Spotify and I've done ever since was really good at is all that information was captured after every project and put into a library.
And that was a big reference.
Like a couple years later,
one of the things I talk about in the chapter,
like my biggest professional failure was something called Spotify Now,
which you don't remember.
But in 2015,
oh man,
it was super big deal. It was like the biggest thing Spotify had ever launched.
And it was very much an anticipation of Apple Music launching.
We knew Apple Music was launching, we wanted to come out first and kind of take the wind out of their sails.
And
It was, we didn't follow our own rules like we,
there's a couple places there where we didn't follow some of our own rules about doing stuff like these big bang releases which it really was encapsulated a lot of stuff was not the way we ever did stuff, did things we also didn't launch, we launched in a tiny.
test because they didn't want the press to find out stuff like that so but i drove it like i was the technical lead of this it was the biggest project spotify had ever done to that date and yeah totally sank in the market like it it hurt retention instead of helping it like not even oh we hoped it was going to do a big retention help no it actually we this this set of features was making people leave and because it was such a big set of features
we didn't know which part of it was done because we hadn't been able
Kevin: to Oh,
Brad: test the
Kevin: yeah.
Brad: way we always would so it's a lot of it is about how when we felt like okay so when you have that big failure and by the way the other thing we never did and I learned all these things that we never did because I had that library of prior projects and prior failures and prior successes to look at and like there were all these rules like
don't do this don't do this don't do this and then we decide to do them anyway and guess what it didn't work again yeah
Kevin: Can I ask a question about that library?
I actually have two questions, a minor question and a major question.
The major question is I want to hear about this library and I do want to know what features Led Zeppelin wanted in Spotify. That's kind of curious.
But may I spell the library before you answer those?
Describe for us what that library looked like, what it shaped like, how it was organized,
because lots of times you make documentation that goes off and nobody ever looks at it again.
So I'm curious to know if I wanted to make a library like that, how would you advise doing it?
Brad: Well, this was the genius part of it, which is it was a Google drive folder with a bunch of Google docs in it.
There was there was nothing special about that library except that.
Everyone had access to it and everyone knew it existed and we were constantly adding to it.
that was this massive cornucopia of like projects big and small what went well what didn't go well and what what have we kind of learned about things afterwards and but there wasn't anything special about it all it wasn't like organized in any kind of special way it was just you you knew like right before I joined they'd done this other kind of big bang release and they did the same mistake which they
they pre-announced it and then took like a year to roll out something like that so that was a very famous failed project so i spent a lot of time reading about what they did wrong in that project and we didn't make any of those mistakes we were right we we launched while they like the second they announced it on stage we started rolling out but of course we didn't roll out to 100 because we would never do that we rolled out to like five percent
But the minimal testing we'd been allowed to do beforehand and then the testing in the actual world,
it turned out very big different results.
So that was that was part of it.
Scott: What are your feelings about the idea that your after action report of Spotify now is being read by product managers or, you know,
VPs of engineering at Spotify?
Brad: I, you know, I got my name on a, on a bunch of my names and a bunch of those both successful and unsuccessful projects, you kind of just feel like, you know, this is my contribution to future, future Spotify projects, I guess,
you
Scott: Yeah.
Brad: know, it is weird how many people I've met who,
you know, I've been gone from there like nine years now,
but how many people I met, I meet who like
like at a conference or something like i know your name oh i read your name in this like in the spotify now uh thing oh you can't even believe how many uh retrospectives we had on spotify now after after that launch
Kevin: You know, you're giving me a few thoughts here.
The first is.
As engineering teams, we develop playbooks for how to do the same thing over and over again.
And some of those playbooks are very simple,
Brad: Yeah.
Kevin: like merge code review release,
right?
But for bigger projects,
those playbooks are more complicated.
And one of the things we've started doing or just beginning an effort to do where I'm at now is actually write that playbook down.
And so when we do a retrospective.
I always describe a retrospective,
the outcomes we want from this retrospective are action items and working agreements.
And really the playbook is the set of working agreements that we've come Yeah. up with for how to be successful.
And I really like what you were saying about the retrospective after a successful project.
We have a few big projects launching over the next couple of months, and I think I'm going to I don't think I know I now know I'm going to bring that to my team and do that because knock wood,
I think they're going to be successful. Well, at least the execution and delivery is going to be successful. Who knows if the customers are going to want them, but we believe they are going to want them, but you never know.
But at least in the execution and delivery, I think I'm going to really bring that in. I think that's a great idea.
Okay,
so I've got to go back to the minor question for a second.
Brad: Yeah. What did Led Zeppelin like?
Kevin: I did have a chance to meet Robert Plant once very quickly,
and I'm very curious to know what features Robert Plant wanted in Spotify.
Brad: Robert,
I don't know which of these things Robert Plant wanted.
They were actually pretty mellow. What I remember is that it wasn't that they needed new features added.
I think it was more about how there was like a Led Zeppelin takeover.
over in the in the ui um was i think the first time we did that and uh they had to be kind of coordinated in how they were released um but it was more about that it was more that there was a lot of fear from the label from the band about okay now we're going into streaming and they wanted it to be kind of a suitable both in-app kind of takeover experience led zeppelin is here you
but also make sure that the albums were looked right and sounded right they weren't as bad as the beatles the beatles i mean were bad but like everything got remastered for for spotify um so that was a big project when we launched the beatles on spotify no
Scott: I was really hoping for like a classic rock star writer story.
It was like only green M&Ms.
Brad: oh no
Scott: That's bad.
Brad: no i
Kevin: I was thinking you were going to say something like they don't they didn't really like how the fast forward worked and they wanted that fixed before.
Brad: i have been yelled at by by musicians about features in spotify like when they would come to the office and stuff um but no unfortunately well you know the sad part is uh like any great
great software company yeah we were the ones doing all the work but we weren't the ones hanging out with Robert Plant or whatever
Scott: Right.
Brad: um you know and London having drinks negotiating the the licensing that was that was the content people they they were the lucky ones there we did we they did send us the cake that was pretty nice and
Scott: And the booze and the booze apparently.
Brad: the booze yeah
Kevin: Now, I do have a feature idea for Spotify.
What if I do still have any friends over there?
Brad: I strangely enough I have a lot of friends over there still okay
Kevin: Can you tell them this idea?
Scott: I'm sure this is like well they're gonna be listening aren't they listening to this podcast it's on Spotify right
Kevin: Oh,
Brad: yeah
Scott: we're on isn't it?
Kevin: in the podcast.
Brad: okay
Kevin: This is I'm actually serious about this. This is the real feature I want I want what's called impersonation
So I have an algorithm,
you have an algorithm,
you have an algorithm.
They're all different.
Brad: Yeah.
Kevin: I want to hear what you hear.
I want you to be able to I want to be able to invite you to listen to my algorithm.
And when you're listening to my algorithm and fast forwarding or liking or whatever, it doesn't it doesn't affect my algorithm.
Brad: Mm-hmm.
Kevin: I'm just curious to hear what you know what your what is your daily mix like right what is your release whatever radar called whatever it is right I'm just very curious to because I have friends and we look I we exchange music or whatever and uh the way I thought of this is one time someone forwarded me like a auto-generated playlist but when I opened it
It was what my algorithm made
Brad: Yeah.
Kevin: not what was on their version of that playlist.
I was like, oh interesting I want to be able to listen to this guy's name was Joel.
I want to be able to listen to Joel's daily mix one not my daily mix one and
I think that'd be really cool
Brad: Yes.
there's a lot there's a yeah there's a lot there that it's
Kevin: I
Brad: not
Kevin: didn't say it
Brad: like
Kevin: was easy.
Brad: a debate no
Scott: love
Brad: no no it
Scott: it this is
Brad: technically technically it's
Scott: a classic though
Brad: not hard yeah yeah
Scott: it's the classic like right this is the cto right like boom you get this you know whatever random idea comes in from somewhere and immediately your brain goes click click wait a second how would we code that
Brad: So
Scott: how
Brad: what
Scott: that is
Brad: are the that implications
Scott: is
Brad: from the business?
Scott: What is the what is this? How does that impact, you know, user retention? Like I got to go I got to go from the code to the strategy back to the code back to the strategy.
And then you go into the political mode of like, how important is this person that's asking for this feature?
How do I how do I explain this to them?
Kevin: Kevin,
you need to explain to your friends at Spotify that I'm very important.
Brad: Yeah, social and music is really hard.
So it's not a technical problem at all. In fact, if you wanted to do that today,
you could tell your friend to copy, you know, to select all on a playlist,
copy, paste to a new playlist,
and then that is a static copy of any auto-generated
Kevin: Yes
Brad: playlist.
So I have .
uh just i don't know i started doing it when long before we launched uh discover weekly and release a radar every week i have a script that just um snapshots it because i didn't like that it went away i really liked sometimes so i have you know hundreds and actually yeah many hundreds of of those playlists that i do that basically does exactly that so there is it's not a problem to do it
but social if you if if you ever use the desktop client remember there used to be you could see what your friends were listening to um
Kevin: Yeah, it isn't there, is it anymore?
Brad: uh they finally killed it um that was the a matter of a lot of debate um i ended up owning it like through my organization at one point and the whole idea of like could we make social work and spotify and
and big debates so it's more of a political product discussion like that's why it won't happen and i can tell you now it won't happen because because the fact that that features long they finally killed that feature is their way of saying yeah we're not going to do this um but i fought for social for a while no
Kevin: I want to see what it's like in your world,
in your musical world,
not my musical world.
Brad: no and yeah it makes sense the what you
What you can do today,
I don't know why I'm selling Spotify here,
but one of the things you can do is if you have a family account,
you have mixed playlists where there's all these things that will generate and you'll kind of combine your favorites and like the rest of your family's favorites. And I actually really like that. My kid is amazing based.
And so that and you can follow people and you can see their playlists.
not the auto-generated one.
Yeah.
Kevin: I do have a family account and there's five accounts in the family account. There are four people in our family.
Brad: Yeah.
Kevin: And so I've claimed the fifth one. I actually have two Spotify accounts.
And because the algorithm was so dialed in on one kind of music for me,
Brad: Yeah,
Kevin: and I wanted to start exploring different kinds of music, and it wasn't really changing for me. So now I've got one that's kind of like rock indie guitar kind of music and one that's like electronic hip hop kind of music and almost like two separate algorithms.
Brad: that's interesting that is actually interesting the your taste clusters right so you get six six
Kevin: Yes.
Brad: daily you know daily mix playlists you have six taste clusters primary taste clusters right so those do update pretty regularly so if you find yourself um oh you listen to a lot of indie rock you know those will start to get dominated by various flavors of indie rock
But then if you start listening to hip hop and it's not like exclusively you just start bringing in more of it into your daily listening,
Kevin: Yeah.
Brad: you'll start to see those taste clusters kind of evolve that way.
I think it's the same thing with DJ.
You'll see it with DJ.
You'll see it with your daily or the I forget what it's called.
I was just listening to the new daily mixing that is generated the day part.
player yeah
Kevin: Yeah,
Brad: yes
Kevin: okay, can we go back to the book?
Scott: Yeah.
Brad: i haven't worked there in nine years
Kevin: You
Brad: yeah
Kevin: got any chapters in there on tech debt?
Brad: i don't have any chapters in the book that is specifically about tech debt but i have but tech debt touches so much of what we do it's kind of inferred or referenced in lots of chapters
Kevin: So the mother of all questions of tech debt
Brad: Yeah,
Kevin: is
Brad: yeah.
Kevin: when you've got the monolith,
you got a bunch of people been hacking on it for a long time, it's turned into a ball of mud. And the question that I find myself facing almost everywhere I've worked is we need to start disentangling them.
And have you figured out that one yet?
Brad: Well, Scott,
it depends on it depends on kind of the how you got to that ball of mud.
What's the what's the delays or what's what's the problem with the ball of mud?
So at AVO, we're a CTO at AVO, which is where I was after I left Spotify.
There we had the classic monolith problem had been, you know, small team team grew bigger.
never got around to like doing the work the monolith was getting bigger and bigger and bigger uh we did a lot of work there to disentangle that but the primary problem we were trying to solve is deploys took
like most of a day so we wanted to be able to deploy faster so we put a lot of energy into figuring out like what are the pieces we need to do and so we left a lot of the monolith there
because it was stable it was just slow to deploy but moved a lot of new development into new services and so for that way like most of the active development could be done and deployed really quick so our average deploy time went down from like five hours and it had to be done by like the DevOps team to like 10 minutes and it could be done by any team it could happen incrementally but we still had a
had a pretty big monolith there that a lot of people were scared to deal with but the biggest problem we had at that point wasn't the performance of that monolith it was really how its existence was hurting our build time
at on Fido where I went a bit later same same kind of problem but there it was very specific bottlenecks that were slowing the performance of the application
application and so over there we actually went and very systematically kind of looked at where these bottlenecks were and then tried to disentangle them that way to to reduce the the hard coupling between pieces and so over time in both these cases kind of over time the monolith kind of kind of got smaller and smaller and smaller um but you know in both of these it was never
really an option for us to like okay we're going to spend a year two years and just really focus on blowing it up because it was just the business that wasn't going to be the right thing for the business right let's solve the problems we have to solve yeah
Scott: Yeah, I think, Kevin, that your answer there is so interesting because as soon as you started talking about the examples that you successfully did some decoupling, you led with the business reason behind it.
And I think that's the fundamental challenge that, you know.
And developers, maybe the, you know, obviously as CTOs, it's our responsibility to sell the idea to decide if it's a good idea and
Brad: Yeah
Scott: then to sell the idea.
And the developer might be just like, hey,
like, I want to refactor this stuff because I don't like it, you know, and,
you know, we've got it. We're the ones that have got to be like,
look,
I can't justify that expense.
I need a reason that we're going to do it. And in those two examples you had.
very clear business impact that
Brad: Yeah.
Scott: justified doing some of the work.
Kevin: One of the challenges I've seen there is the business problem is it's hard for engineers to reason about the whole thing.
And as a result, that just slows things down and often causes unintended consequences.
And that's a lot less quantifiable than build time or system performance.
Brad: That's absolutely true.
Yeah, that's right.
I think.
You know, in both these cases, I was lucky enough to have a set of developers from early days who all knew the system pretty well. And that was obviously,
you know, a weakness as well as a strength.
But that allowed us to say, okay,
well,
in the case of Avvo, if we're moving the new stuff over here, so most people don't have to touch the old stuff.
You know, anymore.
The good news was,
OK,
well, that improved speed of development and kind of let people have.
a better model in their heads of at least the stuff they worked on but it also meant that no there's this thing over here which is like really scary it's the haunted house that you you know dare each other to kind of go in the front gate of um and it was okay it was okay because we had the people that that were knew it but if you know someone happens and they were all on a bus together or something then we would have been in in real trouble that was a good
Again,
there's just a calculated risk, like, all right, let's take this more important to the business.
I have to say the.
My favorite, I haven't used it, but I try and recommend it to other people and see if it works for them,
is I was at some conference and the chief architect at, what's the cool eyeglass, there was this cool eyeglass startup,
Warby Parker.
Kevin: Warby Parker. Warby Parker.
They should advertise with us.
Brad: They should.
The chief architect talked about their monolithic composition story and he did, and this was a bunch of years ago,
but he did it, they did it in a way where they kind of took their monoliths, found the split points, split it into four.
So they made it from one monolith into four monoliths and then from four monoliths into 16 and this very kind
Kevin: Yeah.
Brad: of I loved that and I've wanted to do that. I just never have. And instead,
I recommend it to other people and wait to see if it works for them.
Kevin: Yeah.
The sort of fractal approach.
Brad: Yeah.
Kevin: I've taken a slightly similar approach where
We,
I can't remember exactly how we did it technically, but we ran the same code base in different clusters.
So we didn't actually break the monolith up,
but I guess we did it at the network layer.
We routed different routes, different clusters.
So at least where we're starting to say like, look, I'll just use some stupid examples of counts versus checkout or something like that, right?
We're in different places so we can start seating with the team,
right? Like this is how we're starting to think about breaking this.
Brad: Yep.
But it was still all one blob and
Kevin: Yeah, that's
Brad: then it can sort of get decoupled.
Kevin: really smart.
That's really smart.
I like that idea.
Brad: We had another reason for doing it too,
which was when you have a monolith,
if you have, and this was actually a bigger driver,
frankly,
if you have one part of the system that is sub performing suboptimally, it is contagious to the rest of the system.
And so what that did was actually isolate for us if the account system was performing. If you ship something
Kevin: yeah
Brad: that was slowed down the account system,
you only screwed with the accounts.
You didn't screw with all the rest of the system.
Kevin: but the code was all still in the so the code the monolith was the code base was still a monolith you just had four instances of it and and split the the what what functions were running in which instance that was what you did to it
Brad: I'm trying to remember how we did it, but I think we actually simply did it.
they did it at the router level which is like
Kevin: oh Yeah.
Brad: if you come in on this route you know slash account you're going to go over to this cluster over here and if you come in on slash users you're going to go to this one over here and um so it started to help us think about how to break it up and then we got that other benefit which was it helped us isolate problems
Kevin: Performance. And I mean, that gives you even one of the benefits of like a microservice architecture where you could then scale each of those clusters separately.
Right. So if user
Brad: We
Kevin: in SWOT isn't really slow.
Yeah, that's smart.
I like that.
Brad: it solved another problem too so we had some pretty strong slas around endpoint performance and i remember getting into a i mean it wasn't an argument we weren't arguing but like a debate with an engineering manager i was like hey you got to deal with this these endpoints are out of sla he's like you know the support team hits these like once a day like who cares like i don't care about this this is not worth the time to go fix this and so that was another another
aspect of it is that you could think about how different user segments had different standards you
Kevin: Yeah,
Brad: know the support team didn't need to have every endpoint come back in 100 milliseconds or 500 milliseconds whatever it was they could wait a little bit longer if they had to okay
Kevin: Yeah. that's cool yeah
Brad: so one more question about your book before we wrap up is there a chapter about AI assisted coding in
Kevin: there No, because this ends it's for it's 2012 to 2022
So I managed to dodge that subject completely.
Yeah.
What are your opinions on AI-assisted coding?
I did use OpenAI to generate the index for the book so I wouldn't have to go through it by myself.
Cool.
Yeah.
My opinion on AI-assisted coding has kind of gone from one place to another and then actually coming back to my original idea,
which is AI-assisted coding is a tool.
um like any other tool like when they launched basic the entire point of basic was to get rid of computer programmers right it was going to be a light it was it was it was a language for every person right which is why you know that's how i learned how to code because it was easy approachable to me and but that was the the point oh we'll get rid of computer programmers and the number of times so you know we're all about the same age but you know i've been in industry third
30 some year 33 years the number of times i've heard oh this is the end of software development like all software developers are going to lose their jobs now this is just the latest one what ends up happening is we get to take advantage of this tool we get to use it like a tool
software become we can now do much more complicated and much more powerful things than we used to be able to do because we have this new tool and that's only if we think about it as how do we adapt how do we take this tool adapt it into our workflow and just let us do better stuff because now we don't have to do the old stuff we used to do that's kind of the where i'm feeling it about it
But is it going to be a problem in the short term as we kind of figure out we get from where we are now to there?
Yeah,
absolutely.
Am I scared for junior developers and how did they learn?
Yeah, totally.
The number of people I've had approach me like,
hey, my kid or my cousin is graduating school.
There's no one hiring junior developers anymore.
Yeah, that's pretty freaking scary for us as an industry.
But I do think we will eventually figure this part out and it'll it'll just make us it'll just enable more complicated, more sophisticated, more interesting kind of experiences in the future.
That's where I'm at today.
Brad: I have seen anecdotal data that it is the engineers earlier in their career who are embracing it more.
They're not as rigid,
Kevin: Yeah.
Brad: you know, I mean, if you've been in it 30 years, you've got, you have some opinions about this, right?
If you're newer,
you're not as entrenched.
I've also heard the opposite.
not exactly opposite it is the more experienced engineers who are getting more out of it because they know how to instruct the robot what to do
Kevin: That is exactly my experience.
The founder of District Kid who was the original developer,
but he's really pragmatic,
really open to trying stuff,
was still contributing code when I joined the company.
he was a big proponent of this and as was I I'm like this is a really interesting tool we should be looking at the hardest people to convince were all the senior developers because they're like this isn't going to do better you know this is going to generate worse code than me or whatever and once they got into it and once they realized oh you know what actually this means I don't have to code boilerplate anymore which is a waste of time instead I can do this other stuff yeah they're
They all figured it out and really are very, very powerful with it.
Brad: Can I read you some stats that are just hot off the press that I just got today?
Kevin: Yes, please.
Brad: This person surveyed thousands of developers in 2024,
saw 30 to 40% increase in the code generated.
But 25% of that code required rewriting and bug fixes,
but
Kevin: Yeah.
Brad: the net was a 15 to 20% increase in productivity.
So the senior engineers are right.
It doesn't write as good code as they do,
but net works out better.
Kevin: I am like.
we my team is a pretty small team and a handful like probably 50 50 like more senior more junior and it's so interesting I try to watch these I'm trying to watch the team really closely and I was telling them today in a team meeting I'm like you need to treat this
And when you if you ask the AI to do something, you need to treat it like another developer you asked to do something and now you are going into the role of the code reviewer.
You're going into the role of the of the auditor of this other code and you you cannot think of it as code you wrote.
You need to think of it as code something else wrote that now I'm putting my name on and therefore I need to like look at it interrogate it the same way I would.
Because we have a very strong culture of code reviews,
and so same way I would when I look at another developer's
Yeah
code.
Right
I think, you know, going back to this idea of the monolith, I think one of the biggest problems of these monolith code bases is it's impossible to find anything.
And you end up with a lot of duplication because it's impossible to find anything. So somebody's like,
oh, I need a function or I need a class that does this. I can't find one. I'm just going to make a new one.
And all of a sudden you've got business logic duplicated all over the place.
This is actually one thing that we've been looking into,
which is at DistroKid because we have a pretty large code base.
And yeah,
like it's, I'm finding like developers are writing new things that duplicate things that already existed because they didn't know it existed.
And so that is one thing we're looking for is like, hey,
I need a function or I need a class that does this or I need a function that does this. Is there one in the code base and actually we're getting pretty good results out of that.
So just also that value of helping with large code bases, helping you understand your own code base.
and and write and not even write code for you but help you write better code is is really interesting as well yeah
Yeah, yeah,
Brad: All the teachers here.
Kevin: love it
Brad: It seems like that's a wrap for today.
As two music guys,
I'm going to finish on this note.
One recommended album for our audience to listen to from each of you.
We'll go with you first,
Brad,
and we'll let Kevin have the final word.
Kevin: Uh,
I'm not going to go with an album,
but I will go with a new to me band that my wife and I just discovered last Monday.
We were we went to go see Franz Ferdinand here in Seattle and the opening band was this band we'd never heard of before called the telescreens and they absolutely rocked the house.
And then my wife's such a super fan of Franz Ferdinand, we flew down to San Francisco to see them again.
And on Friday, and then telescreens played a different set and they were even more amazing and chatted with the band afterwards, you know, at the merch table, I gave them a Velvet Underground meets the Gun Club meets Replacements meets Fugazi and they just just brought this amazing high energy poetic.
Punk sound that was just amazing.
The Telescreens.
Brad: All right, Kevin.
Kevin: I'm going to just say Taco Cat.
is my one of my new favorite Seattle bands I do not get out as much to shows as I used to but uh I still pay attention and um hadn't heard them and just picked them up they've been around they're not new but uh that's my new favorite band I found listening to them a lot
Brad: That's fun to hear about,
Kevin,
because when I worked at Cheeseburger like 15 years ago now,
a few of them.
And the taco cat came out of Cheeseburger.
The lead
Kevin: Is
Brad: singer
Kevin: that right?
Brad: and a few of the other members were working at Cheeseburger and then they all quit to go on tour to be on their band.
So it's awesome that they're still around.
Kevin: I don't know if they're still around.
It's more I just came across them.
But that's awesome.
That was 15 years ago?
Brad: Well, yeah,
I joined Cheeseburger in 2007.
Kevin: Yeah, they started in 2007. According to Wikipedia,
there's still a thing.
There you go.
Oh, awesome.
I was still playing a lot of shows around then. I don't know how I missed them.
But all right.
Good to know.
Brad: Well,
Kevin,
thank you for your time today.
Where can people get your book if they want it?
Kevin: My book is available at itdependspook.net.
It depends,
but all one word. So there's links to all the bookstores in the world that I know of that carry it there and the audiobook and all the various versions of it.
We will make sure to link that in the show notes.
Brad: Cool.
Kevin: Thank you. Thank you so much for having me on. It was a lot of fun.
Great meeting you,
Brad: Thank
Kevin: Kevin.
Brad: you, Kevin.
Scott: This episode of How Many CTOs Does It Take Podcast was edited by Saffron Heppegau. Our theme song was AI Generated by Brad.
Please follow us on all the social media platforms at at howmanyctospod. You can find us on YouTube,
LinkedIn,
Instagram,
Facebook,
Threads, X,
TikTok,
and Patreon,
or at our website at howmanyctospod.com.
And remember to rate, review, and subscribe on the podcast platform of your choice.
Thanks for listening.