Transcript: Embracing Failure and Innovation: Lessons from Punk Rock with Kevin Goldsmith
0:00 Brad But when you make a mistake,
0:01 Brad how do you then take the time to learn the lessons from it?
0:10 Kevin Welcome to the show,
0:11 Kevin protect me from the rest of the universe never tells in the coming song Which are two big names and a vision time for how many CTOs will share their stories untold
0:20 Kevin Hello, everybody.
0:22 Kevin Welcome to.
0:24 Kevin This episode of How Many CTOs Does It Take?
0:28 Kevin We have got a very interesting guest today. I have got two guests today. I've got my regular co-host Brad Hefta-Gaub, who when he was younger started his own punk rock record label. And we've got with us...
0:41 Kevin Kevin Goldsmith who also when he was younger started a record label and has had a notorious career after music in the tech industry and has written a book that we're going to get to hear about today called It Depends but I think the place we've got to start with Kevin before we have you introduce yourself is what was the name of your record label and what kind of music did you have on it
1:05 Brad So the record label was called Unicircle Records because it was kind of punk rock,
1:10 Brad R-E-K-K-I-D-S,
1:12 Brad and it featured,
1:14 Brad it was, the motto is called Sound Art,
1:18 Brad Art Rock.
1:19 Brad So the most famous band on the label was this band called Bethany Curve from Santa Cruz that was,
1:28 Brad or I guess kind of is.
1:31 Brad more shoegazy but there was a lot of experimental stuff a lot of kind of art rock stuff bands from all over the country so it was pretty active for through the 90s
1:48 Kevin That's cool.
1:49 Scott Very cool.
1:50 Kevin And Brad,
1:51 Kevin how about your record label?
1:52 Scott My record label was called Carving Knife Records,
1:56 Scott started as a classic.
1:59 Scott you know i was in a band called three blind mice and so started the whole diy do-it-yourself label for your own band uh but um then i realized that my friends were actually much better musicians than i was and i started putting out their records so it was all pretty much punk it was in the late 80s early 90s of seattle scene so it was like pre grunge um and we were on the more punk side of things so probably the biggest band was a band called monster
2:26 Scott your truck driver and
2:27 Kevin Oh, of course.
2:27 Scott another great band called cat food which was an all-girl punk band
2:35 Scott Oh,
2:35 Kevin I love it. Well,
2:36 Kevin I'm the only guy in the room without a record label.
2:39 Kevin I'm going to have to get to work on that.
2:40 Kevin So, Kevin, welcome.
2:42 Kevin Tell us about yourself.
2:43 Kevin So what's your how did you arrive on this podcast?
2:46 Kevin You started you're born somewhere and you're here today.
2:48 Kevin What happened in between?
2:51 Brad Oh,
2:51 Brad gosh.
2:52 Brad OK,
2:52 Brad well,
2:53 Brad that's all right. Let's see how many seconds I can put 55 years into.
3:01 Brad I got into computers when I was a little kid and my school got a TRSAT model one.
3:09 Brad And so that was my first exposure,
3:11 Brad really got into programming, really enjoyed it, decided that was going to be when I found out I could actually have a job doing it. That was like the best news I ever got.
3:21 Brad Got really focused on it,
3:23 Brad went to college for it, ended up originally.
3:27 Brad In college I got into graphics, so I was doing a lot of computer graphics stuff,
3:31 Brad ended up working for a company,
3:33 Brad my first job after college was at this company called Silicon Graphics, so it was a pretty good place to be doing graphics in the early 90s.
3:44 Brad And then went into commercial animation. So I worked on TV shows and films for a little while and then ended up working at Microsoft in the Microsoft research team for a while and then on Windows Media.
3:57 Brad And then that was kind of 2000,
4:00 Brad early 2000 left like everybody,
4:02 Brad a lot of people did if Microsoft went to go do a startup.
4:06 Brad Also in that media encoding space. So I did a couple.
4:10 Brad a couple startups that all fell apart uh around uh when all of it fell apart ended up back at microsoft and then uh like michael neck and who you just had i was at adobe i left microsoft went to adobe it was there for nine years and so i've been
4:29 Brad Kind of a, I'd been a lead and I'd been kind of, I kept going back and forth between like,
4:35 Brad oh, I'm going to be a senior developer and I guess I'm managing people now,
4:38 Brad but it was never like a choice.
4:40 Brad It was just like, Kevin, you now manage people.
4:42 Brad So I wasn't a great manager,
4:44 Brad but at Adobe, I got much more serious about being a manager.
4:47 Brad So I was there nine years,
4:48 Brad started as just a developer or senior developer.
4:54 Brad left as a director and then ended up at Spotify and so then I was at Spotify in Sweden for a few years started there as a director ended up leaving as VP and then came had came back to Seattle and started as a started being a CTO and so I've been a CTO for the last nine years at different companies so
5:21 Brad After kind of music,
5:22 Brad I went, I had my journey in the wilderness where I did sort of legal tech and edutech and fintech and data science.
5:35 Brad So I was at different companies and then kind of ended up having opportunity to come back to music.
5:41 Brad And so for the last two and a half years, I've been CTO at a company called DistroKid where we do music.
5:47 Brad music distribution.
5:48 Brad So kind of got found my way back into music, which which I've been really digging.
5:53 Scott I love it. You said you went had a journey into the wilderness and then you listed like a whole bunch of random fields.
6:01 Scott I feel like I've done that before.
6:04 Brad It was a thing where, you know, I'd been like you, I'd been in a lot of kind of media. I'd been in media, I'd been in 3D graphics and digital audio and digital video for a long time.
6:17 Brad And but coming back to Seattle.
6:20 Brad because you know this is where you know my family is it is um come back to Seattle the choices were there wasn't a lot of choices right especially around that time like 2016 especially if you wanted to be in a startup and so yeah I I did my time in in these other fields but it was really it ended up being really
6:43 Brad really valuable and really useful and so I'm glad I did it but it's kind of nice to be back you know be back in my home kind of area again
6:53 Scott Totally.
6:54 Kevin I was just thinking today,
6:56 Kevin I can't even remember what the topic was, but I was just thinking about how if I hadn't had this engineering leadership role at three or four different companies, I guess this is the fourth one now.
7:08 Kevin I just don't think I would, I mean, I'm not trying to toot my own horn, but I just don't think I'd be as good at it as I am.
7:14 Kevin Because each one you make, each one's different.
7:18 Kevin So
7:18 Scott Yep.
7:18 Kevin you have to learn how to like apply the principles differently.
7:21 Kevin And you make mistakes in each one. So you kind of get a little bit of a redo,
7:26 Kevin a little bit each one.
7:27 Kevin And I think that jumping around does have some value.
7:32 Scott Totally.
7:34 Kevin Okay,
7:34 Kevin so Kevin,
7:34 Kevin along the way, I know that you wrote a book, which I'm embarrassed to admit,
7:38 Kevin I never bought or read called It Depends.
7:43 Kevin And tell us all about that, because I think all of us as engineering leaders, somebody have said, asked us some question and we have answered,
7:51 Kevin it depends.
7:52 Kevin And so you nailed it on the head with the title.
7:54 Kevin What is what's that all about?
7:58 Brad You mean, there
7:59 Kevin Right
7:59 Brad you go.
7:59 Kevin there.
7:59 Brad I mean, this book, Scott?
8:02 Kevin That book.
8:03 Kevin That's
8:03 Brad This
8:03 Kevin what looks like a hardcover.
8:04 Brad is a hardcover.
8:05 Kevin Where can someone get the hardcover?
8:07 Scott Signed by the author.
8:09 Brad There is ways to get one signed by the author.
8:13 Brad So what this book came about for I've been blogging as a kind of especially I started really at Adobe kind of doing my.
8:25 Brad there was a decision I made where instead of being sort of an accidental manager or having management thrust upon me there was a point there where I actually decided you know what actually this is I want to make the switch like I'm going to get as serious about being a manager as I was about being a developer and and really build you know build my skills and build a craft here and so that I started blogging because at the time
8:50 Brad So this is like 2005-ish is when I started doing this. There really wasn't a lot of stuff.
8:57 Brad There was Michael Lop's blog and like he had his book out, but the books were really weak,
9:03 Brad really lame, very kind of 80s style old school.
9:09 Brad And there was a couple of people besides Michael Lop blogging, but not a lot.
9:13 Brad And so I thought,
9:13 Brad OK,
9:14 Brad well,
9:14 Brad I'm going to document sort of the things I learn as I go. So I started blogging then. I'm not a huge, to be fair, I'm not like a massive.
9:22 Brad as a writer there's other people now that write daily and stuff i don't i don't like writing i like writing code not words but you know when i would find something that i would feel like oh i just really learned something or
9:37 Brad this just happened and and this is maybe interesting to other people i would write it down and share it and then a few years ago when we all had a lot more time than we were used to having um in addition to like you know buying way too much euro rack gear um i started actually realizing like you know what i've actually written a lot of stuff and this might be valuable
10:03 Brad to kind of put together so if you think of like really you know there's really well-known books that are really just collections of blog posts and like oh i could i've i've got a blog post i could do that
10:15 Brad And so that was the thought is I hadn't written a book. I'd thought about writing a book, but writing a book was really daunting and thought, well,
10:22 Brad you know, maybe a good way to start is to take the things I've already written, clean them up a little bit. I'm a much better writer now than I was 10 years ago.
10:31 Brad So clean them up a little bit and then collect them and have that be the first book.
10:35 Brad So that's what it really is. It's a collection of.
10:39 Brad My blog posts from 2012 to 2022,
10:43 Brad and the name came about, like you said, like.
10:47 Brad People would ask me, so I mentor a lot of people over the years, like I really like mentoring kind of up and coming folks, especially folks who are kind of not coming out of non-traditional backgrounds or are from underrepresented groups.
11:04 Brad And so, you know, I would be talking to them and they'd say,
11:08 Brad oh, you should write a book someday or they'd say, or if you were going to write a book, what would it be called? And I'd say, well.
11:13 Brad well probably be called it depends and i'd said that for years and years as a joke because that is like my number one answer to any question is oh how do you how many developers you know what's a qa to developer ratio on a team like well totally depends right what kind of team is it what's it doing and
11:32 Scott Right.
11:32 Brad so it's such a it's for all of us it's just like the number one answer we say is so every answer is so
11:38 Brad content uh context dependent all right well it just made sense as a book and so that's where the title comes from my
11:44 Kevin So I would imagine each blog post is sort of its own chapter Do you have a favorite?
11:49 Brad favorite one in there is uh it's is it the first gosh it should be i think it's the first which is uh it's uh yeah fail safe fail smart succeed
12:02 Brad Um,
12:03 Brad and
12:03 Kevin Mm
12:03 Brad the
12:03 Kevin -hmm.
12:03 Brad title is really lame. I'm really bad at titles, but the title is really lame, but it's essentially I did for, I also give talks and I gave for years and years, I gave this keynote talk that was essentially talking about.
12:20 Brad how we learn well from failure and how to fail well right so if we're if we have to fail to learn and we have to fail to succeed to do anything interesting like how do you fail well like how do you set up the conditions in your team to fail well and it uses a lot of examples of stuff we did at spotify because spotify was we that was really well set up like we were really good at failing smartly and so uh i decided you know i
12:46 Brad I've been giving it for a long time. I'm like, oh, I should probably write it down so that, you know, I can refer to that.
12:52 Brad And that took me.
12:54 Brad like a year and a half to take i started with oh i'll just take a transcription of one of the keynote talks i gave and i'll just edit it and that that was the hardest chapter to write which is the saddest because it was it was a talk i'd given probably a dozen times but and it's and it is an edited transcription and just took forever it was so painful but i'm really happy with how it came out
13:19 Scott Yeah.
13:19 Kevin I mean, I have to ask, what are the conditions that
13:21 Kevin create successful failure.
13:24 Brad So.
13:25 Brad The important stuff is, I think, would be obvious to anybody, which is like, how do you fail fast?
13:30 Brad How do you fail fast while being confident that you're not failing too fast, right?
13:36 Brad So part of it is setting up how do you do feature development?
13:41 Brad Some of it comes from a lot of kind of lean startup thinking,
13:45 Brad but we had it set up at Spotify where product development was set up into very specific structured stages.
13:52 Brad pages and it was important enough at the company it was sort of one of the company mottos think it build it ship it tweak it and so
14:00 Kevin the Hmm.
14:00 Brad idea was
14:02 Brad you know how fast could you if you think about investment in a feature or investment in a product when you're just thinking about and you're talking about it you like you a product manager a couple people okay well that's not a lot of investment it's not a lot of time spent then you start building it now you're actually starting to do a lot of investment and then you know up until from the time you think of it to the time it's actually in front of real customers that's just all
14:27 Brad all lost money that's all just invested cash and so the question was how do you if you visualize it as a curve which is what i do in the chapter how do you minimize the area under that curve and so there's a bunch of tools that are real obvious like how do you ship an mvp what's the right mvp to ship what's the minimum version of this that you can get in front of people and start getting data back that's an important part of it but then it's also
14:53 Brad How do you decide as you go forward,
14:56 Brad when are we done investing in this?
14:59 Brad And then how do you know whether you're hitting sort of a local maxima or whether you really actually got the value out of it?
15:06 Brad So there's a lot behind that thinking, but the core of it is how do you do things in such a way so that you are constantly validating your hypotheses? And how do you also the hardest part?
15:20 Brad in any company is this,
15:22 Brad this isn't working.
15:23 Brad and then the product manager is like oh no it really is like no no you did it wrong now maybe we need to keep working on we need to keep working on it's like no like how do you make that decision of no this is it we're done like hey we thought it was a good idea it wasn't let's make sure we've learned from it that's the other part is when you fail it's really easy to or you you want to sweep it under the rug like no one likes to dwell on their failures everybody likes to think about how
15:49 Brad how great they are.
15:50 Brad But when you make a mistake,
15:52 Brad how do you then take the time to learn the lessons from it?
15:57 Brad And I think another part of how do you fail well is making failure not a big deal.
16:06 Brad So one of the things that happened when I joined Spotify,
16:09 Brad sorry,
16:09 Brad I'm rambling, but I feel
16:12 Scott No,
16:12 Brad very
16:12 Scott you're not. This is great,
16:13 Brad passionate
16:13 Scott though.
16:13 Brad about this.
16:15 Brad One of the things that I remember,
16:17 Brad so right after I joined Spotify,
16:19 Brad I think it was right when we got,
16:22 Brad was it Led Zeppelin? I think it was when we got Led Zeppelin on the platform.
16:25 Brad So that happened kind of right after in my team,
16:29 Brad right after I joined.
16:31 Brad so i was there for part of it you know a lot of the negotiation part i wasn't there for but the actual like oh they had a bunch of requirements and we got them all done we launched them on the platform because they were one of these at the time you know a lot of bands didn't want to be on streaming um and so we had to do this a bunch of times with different kind of major artists but they the result was huge success
16:56 Brad Right. So I can't take all the credit for it because I wasn't there for most of it. But I got there massive success.
17:02 Brad The label sent us like cake and liquor and like they were so overjoyed with how it went, like unqualified success because of music.
17:13 Brad Of course, they sent us liquor.
17:15 Brad And then I'm like,
17:17 Brad okay, cool.
17:18 Brad Like, yeah, we had the big party.
17:19 Brad We're awesome.
17:21 Brad Like, let's move on.
17:22 Brad And then the next day I got a big meeting, like time to like do the retrospective.
17:28 Brad And we then spent like four hours going through everything that went wrong.
17:33 Brad in this highly successful process and this highly successful launch because we wanted to make sure like the next time we did it we wouldn't make those same we'd make new mistakes we wouldn't make the same mistakes again and I'd never seen that any company right you you do the post mortems or you do the after actions or whatever when something goes wrong but to have some to go through that process when something goes right I'd never seen it
17:57 Brad company before,
17:57 Brad but that was sort of the essence of the way Spotify deals with stuff like that.
18:03 Scott I think that's such a great story,
18:05 Scott Kevin.
18:05 Scott I am, you know, even on the small scale with the small team that I'm working with now, we're like pretty religious about doing our retrospectives and I'm always reminding them like.
18:16 Scott And when the sprint goes great,
18:18 Scott Yeah. that's when it's the most important for us to do the retrospective because we really need to ask ourselves, did we just get lucky or did we actually do something that made this sprint go better than the last sprint or this sprint stand out? And that's something I kind of learned that from I do a bunch of mountain sports and traveling in avalanche terrain and in, you know, in,
18:40 Scott you know, low information, high consequence.
18:43 Scott environments
18:44 Brad Yeah,
18:44 Scott you
18:44 Brad yeah
18:45 Scott just got to ask and i and i feel like software and business is a low information high consequence you know environment and you have to you really have to interrogate like
18:55 Scott what actually happened here and what can we learn from
19:00 Brad Absolutely. And the other important part of that that Spotify and I've done ever since was really good at is all that information was captured after every project and put into a library.
19:12 Brad And that was a big reference.
19:16 Brad Like a couple years later,
19:18 Brad one of the things I talk about in the chapter,
19:19 Brad like my biggest professional failure was something called Spotify Now,
19:23 Brad which you don't remember.
19:25 Brad But in 2015,
19:26 Brad oh man,
19:27 Brad it was super big deal. It was like the biggest thing Spotify had ever launched.
19:31 Brad And it was very much an anticipation of Apple Music launching.
19:34 Brad We knew Apple Music was launching, we wanted to come out first and kind of take the wind out of their sails.
19:40 Brad And
19:41 Brad It was, we didn't follow our own rules like we,
19:44 Brad there's a couple places there where we didn't follow some of our own rules about doing stuff like these big bang releases which it really was encapsulated a lot of stuff was not the way we ever did stuff, did things we also didn't launch, we launched in a tiny.
19:59 Brad test because they didn't want the press to find out stuff like that so but i drove it like i was the technical lead of this it was the biggest project spotify had ever done to that date and yeah totally sank in the market like it it hurt retention instead of helping it like not even oh we hoped it was going to do a big retention help no it actually we this this set of features was making people leave and because it was such a big set of features
20:26 Brad we didn't know which part of it was done because we hadn't been able
20:30 Kevin to Oh,
20:31 Brad test the
20:31 Kevin yeah.
20:31 Brad way we always would so it's a lot of it is about how when we felt like okay so when you have that big failure and by the way the other thing we never did and I learned all these things that we never did because I had that library of prior projects and prior failures and prior successes to look at and like there were all these rules like
20:51 Brad don't do this don't do this don't do this and then we decide to do them anyway and guess what it didn't work again yeah
20:58 Kevin Can I ask a question about that library?
21:01 Kevin I actually have two questions, a minor question and a major question.
21:04 Kevin The major question is I want to hear about this library and I do want to know what features Led Zeppelin wanted in Spotify. That's kind of curious.
21:13 Kevin But may I spell the library before you answer those?
21:16 Kevin Describe for us what that library looked like, what it shaped like, how it was organized,
21:20 Kevin because lots of times you make documentation that goes off and nobody ever looks at it again.
21:25 Kevin So I'm curious to know if I wanted to make a library like that, how would you advise doing it?
21:30 Brad Well, this was the genius part of it, which is it was a Google drive folder with a bunch of Google docs in it.
21:37 Brad There was there was nothing special about that library except that.
21:41 Brad Everyone had access to it and everyone knew it existed and we were constantly adding to it.
21:48 Brad that was this massive cornucopia of like projects big and small what went well what didn't go well and what what have we kind of learned about things afterwards and but there wasn't anything special about it all it wasn't like organized in any kind of special way it was just you you knew like right before I joined they'd done this other kind of big bang release and they did the same mistake which they
22:13 Brad they pre-announced it and then took like a year to roll out something like that so that was a very famous failed project so i spent a lot of time reading about what they did wrong in that project and we didn't make any of those mistakes we were right we we launched while they like the second they announced it on stage we started rolling out but of course we didn't roll out to 100 because we would never do that we rolled out to like five percent
22:40 Brad But the minimal testing we'd been allowed to do beforehand and then the testing in the actual world,
22:46 Brad it turned out very big different results.
22:50 Brad So that was that was part of it.
22:52 Scott What are your feelings about the idea that your after action report of Spotify now is being read by product managers or, you know,
23:03 Scott VPs of engineering at Spotify?
23:05 Brad I, you know, I got my name on a, on a bunch of my names and a bunch of those both successful and unsuccessful projects, you kind of just feel like, you know, this is my contribution to future, future Spotify projects, I guess,
23:20 Brad you
23:20 Scott Yeah.
23:20 Brad know, it is weird how many people I've met who,
23:24 Brad you know, I've been gone from there like nine years now,
23:27 Brad but how many people I met, I meet who like
23:30 Brad like at a conference or something like i know your name oh i read your name in this like in the spotify now uh thing oh you can't even believe how many uh retrospectives we had on spotify now after after that launch
23:43 Kevin You know, you're giving me a few thoughts here.
23:46 Kevin The first is.
23:50 Kevin As engineering teams, we develop playbooks for how to do the same thing over and over again.
23:55 Kevin And some of those playbooks are very simple,
23:56 Brad Yeah.
23:56 Kevin like merge code review release,
23:58 Kevin right?
23:59 Kevin But for bigger projects,
24:00 Kevin those playbooks are more complicated.
24:03 Kevin And one of the things we've started doing or just beginning an effort to do where I'm at now is actually write that playbook down.
24:14 Kevin And so when we do a retrospective.
24:17 Kevin I always describe a retrospective,
24:19 Kevin the outcomes we want from this retrospective are action items and working agreements.
24:24 Kevin And really the playbook is the set of working agreements that we've come Yeah. up with for how to be successful.
24:31 Kevin And I really like what you were saying about the retrospective after a successful project.
24:37 Kevin We have a few big projects launching over the next couple of months, and I think I'm going to I don't think I know I now know I'm going to bring that to my team and do that because knock wood,
24:49 Kevin I think they're going to be successful. Well, at least the execution and delivery is going to be successful. Who knows if the customers are going to want them, but we believe they are going to want them, but you never know.
24:57 Kevin But at least in the execution and delivery, I think I'm going to really bring that in. I think that's a great idea.
25:02 Kevin Okay,
25:03 Kevin so I've got to go back to the minor question for a second.
25:05 Brad Yeah. What did Led Zeppelin like?
25:06 Kevin I did have a chance to meet Robert Plant once very quickly,
25:09 Kevin and I'm very curious to know what features Robert Plant wanted in Spotify.
25:14 Brad Robert,
25:14 Brad I don't know which of these things Robert Plant wanted.
25:18 Brad They were actually pretty mellow. What I remember is that it wasn't that they needed new features added.
25:24 Brad I think it was more about how there was like a Led Zeppelin takeover.
25:29 Brad over in the in the ui um was i think the first time we did that and uh they had to be kind of coordinated in how they were released um but it was more about that it was more that there was a lot of fear from the label from the band about okay now we're going into streaming and they wanted it to be kind of a suitable both in-app kind of takeover experience led zeppelin is here you
25:55 Brad but also make sure that the albums were looked right and sounded right they weren't as bad as the beatles the beatles i mean were bad but like everything got remastered for for spotify um so that was a big project when we launched the beatles on spotify no
26:15 Scott I was really hoping for like a classic rock star writer story.
26:18 Scott It was like only green M&Ms.
26:20 Brad oh no
26:20 Scott That's bad.
26:21 Brad no i
26:24 Kevin I was thinking you were going to say something like they don't they didn't really like how the fast forward worked and they wanted that fixed before.
26:31 Brad i have been yelled at by by musicians about features in spotify like when they would come to the office and stuff um but no unfortunately well you know the sad part is uh like any great
26:46 Brad great software company yeah we were the ones doing all the work but we weren't the ones hanging out with Robert Plant or whatever
26:53 Scott Right.
26:53 Brad um you know and London having drinks negotiating the the licensing that was that was the content people they they were the lucky ones there we did we they did send us the cake that was pretty nice and
27:07 Scott And the booze and the booze apparently.
27:09 Brad the booze yeah
27:10 Kevin Now, I do have a feature idea for Spotify.
27:13 Kevin What if I do still have any friends over there?
27:14 Brad I strangely enough I have a lot of friends over there still okay
27:18 Kevin Can you tell them this idea?
27:21 Scott I'm sure this is like well they're gonna be listening aren't they listening to this podcast it's on Spotify right
27:27 Kevin Oh,
27:27 Brad yeah
27:27 Scott we're on isn't it?
27:28 Kevin in the podcast.
27:28 Brad okay
27:30 Kevin This is I'm actually serious about this. This is the real feature I want I want what's called impersonation
27:36 Kevin So I have an algorithm,
27:37 Kevin you have an algorithm,
27:38 Kevin you have an algorithm.
27:39 Kevin They're all different.
27:40 Brad Yeah.
27:40 Kevin I want to hear what you hear.
27:42 Kevin I want you to be able to I want to be able to invite you to listen to my algorithm.
27:47 Kevin And when you're listening to my algorithm and fast forwarding or liking or whatever, it doesn't it doesn't affect my algorithm.
27:52 Brad Mm-hmm.
27:54 Kevin I'm just curious to hear what you know what your what is your daily mix like right what is your release whatever radar called whatever it is right I'm just very curious to because I have friends and we look I we exchange music or whatever and uh the way I thought of this is one time someone forwarded me like a auto-generated playlist but when I opened it
28:16 Kevin It was what my algorithm made
28:17 Brad Yeah.
28:17 Kevin not what was on their version of that playlist.
28:20 Kevin I was like, oh interesting I want to be able to listen to this guy's name was Joel.
28:24 Kevin I want to be able to listen to Joel's daily mix one not my daily mix one and
28:29 Kevin I think that'd be really cool
28:31 Brad Yes.
28:33 Brad there's a lot there's a yeah there's a lot there that it's
28:36 Kevin I
28:37 Brad not
28:37 Kevin didn't say it
28:37 Brad like
28:37 Kevin was easy.
28:37 Brad a debate no
28:38 Scott love
28:38 Brad no no it
28:38 Scott it this is
28:39 Brad technically technically it's
28:39 Scott a classic though
28:40 Brad not hard yeah yeah
28:41 Scott it's the classic like right this is the cto right like boom you get this you know whatever random idea comes in from somewhere and immediately your brain goes click click wait a second how would we code that
28:56 Brad So
28:56 Scott how
28:56 Brad what
28:56 Scott that is
28:56 Brad are the that implications
28:57 Scott is
28:57 Brad from the business?
28:58 Scott What is the what is this? How does that impact, you know, user retention? Like I got to go I got to go from the code to the strategy back to the code back to the strategy.
29:07 Scott And then you go into the political mode of like, how important is this person that's asking for this feature?
29:12 Scott How do I how do I explain this to them?
29:16 Kevin Kevin,
29:16 Kevin you need to explain to your friends at Spotify that I'm very important.
29:23 Brad Yeah, social and music is really hard.
29:27 Brad So it's not a technical problem at all. In fact, if you wanted to do that today,
29:31 Brad you could tell your friend to copy, you know, to select all on a playlist,
29:37 Brad copy, paste to a new playlist,
29:39 Brad and then that is a static copy of any auto-generated
29:43 Kevin Yes
29:43 Brad playlist.
29:44 Brad So I have .
29:45 Brad uh just i don't know i started doing it when long before we launched uh discover weekly and release a radar every week i have a script that just um snapshots it because i didn't like that it went away i really liked sometimes so i have you know hundreds and actually yeah many hundreds of of those playlists that i do that basically does exactly that so there is it's not a problem to do it
30:11 Brad but social if you if if you ever use the desktop client remember there used to be you could see what your friends were listening to um
30:19 Kevin Yeah, it isn't there, is it anymore?
30:21 Brad uh they finally killed it um that was the a matter of a lot of debate um i ended up owning it like through my organization at one point and the whole idea of like could we make social work and spotify and
30:36 Brad and big debates so it's more of a political product discussion like that's why it won't happen and i can tell you now it won't happen because because the fact that that features long they finally killed that feature is their way of saying yeah we're not going to do this um but i fought for social for a while no
30:54 Kevin I want to see what it's like in your world,
30:56 Kevin in your musical world,
30:57 Kevin not my musical world.
30:58 Brad no and yeah it makes sense the what you
31:00 Brad What you can do today,
31:02 Brad I don't know why I'm selling Spotify here,
31:05 Brad but one of the things you can do is if you have a family account,
31:09 Brad you have mixed playlists where there's all these things that will generate and you'll kind of combine your favorites and like the rest of your family's favorites. And I actually really like that. My kid is amazing based.
31:20 Brad And so that and you can follow people and you can see their playlists.
31:25 Brad not the auto-generated one.
31:27 Brad Yeah.
31:29 Kevin I do have a family account and there's five accounts in the family account. There are four people in our family.
31:35 Brad Yeah.
31:35 Kevin And so I've claimed the fifth one. I actually have two Spotify accounts.
31:39 Kevin And because the algorithm was so dialed in on one kind of music for me,
31:44 Brad Yeah,
31:44 Kevin and I wanted to start exploring different kinds of music, and it wasn't really changing for me. So now I've got one that's kind of like rock indie guitar kind of music and one that's like electronic hip hop kind of music and almost like two separate algorithms.
32:01 Brad that's interesting that is actually interesting the your taste clusters right so you get six six
32:08 Kevin Yes.
32:09 Brad daily you know daily mix playlists you have six taste clusters primary taste clusters right so those do update pretty regularly so if you find yourself um oh you listen to a lot of indie rock you know those will start to get dominated by various flavors of indie rock
32:27 Brad But then if you start listening to hip hop and it's not like exclusively you just start bringing in more of it into your daily listening,
32:35 Kevin Yeah.
32:35 Brad you'll start to see those taste clusters kind of evolve that way.
32:38 Brad I think it's the same thing with DJ.
32:40 Brad You'll see it with DJ.
32:41 Brad You'll see it with your daily or the I forget what it's called.
32:47 Brad I was just listening to the new daily mixing that is generated the day part.
32:52 Brad player yeah
32:54 Kevin Yeah,
32:54 Brad yes
32:55 Kevin okay, can we go back to the book?
32:57 Scott Yeah.
32:58 Brad i haven't worked there in nine years
32:59 Kevin You
32:59 Brad yeah
32:59 Kevin got any chapters in there on tech debt?
33:01 Brad i don't have any chapters in the book that is specifically about tech debt but i have but tech debt touches so much of what we do it's kind of inferred or referenced in lots of chapters
33:15 Kevin So the mother of all questions of tech debt
33:18 Brad Yeah,
33:18 Kevin is
33:18 Brad yeah.
33:19 Kevin when you've got the monolith,
33:20 Kevin you got a bunch of people been hacking on it for a long time, it's turned into a ball of mud. And the question that I find myself facing almost everywhere I've worked is we need to start disentangling them.
33:36 Kevin And have you figured out that one yet?
33:39 Brad Well, Scott,
33:40 Brad it depends on it depends on kind of the how you got to that ball of mud.
33:46 Brad What's the what's the delays or what's what's the problem with the ball of mud?
33:50 Brad So at AVO, we're a CTO at AVO, which is where I was after I left Spotify.
33:56 Brad There we had the classic monolith problem had been, you know, small team team grew bigger.
34:02 Brad never got around to like doing the work the monolith was getting bigger and bigger and bigger uh we did a lot of work there to disentangle that but the primary problem we were trying to solve is deploys took
34:15 Brad like most of a day so we wanted to be able to deploy faster so we put a lot of energy into figuring out like what are the pieces we need to do and so we left a lot of the monolith there
34:30 Brad because it was stable it was just slow to deploy but moved a lot of new development into new services and so for that way like most of the active development could be done and deployed really quick so our average deploy time went down from like five hours and it had to be done by like the DevOps team to like 10 minutes and it could be done by any team it could happen incrementally but we still had a
34:55 Brad had a pretty big monolith there that a lot of people were scared to deal with but the biggest problem we had at that point wasn't the performance of that monolith it was really how its existence was hurting our build time
35:09 Brad at on Fido where I went a bit later same same kind of problem but there it was very specific bottlenecks that were slowing the performance of the application
35:22 Brad application and so over there we actually went and very systematically kind of looked at where these bottlenecks were and then tried to disentangle them that way to to reduce the the hard coupling between pieces and so over time in both these cases kind of over time the monolith kind of kind of got smaller and smaller and smaller um but you know in both of these it was never
35:47 Brad really an option for us to like okay we're going to spend a year two years and just really focus on blowing it up because it was just the business that wasn't going to be the right thing for the business right let's solve the problems we have to solve yeah
36:03 Scott Yeah, I think, Kevin, that your answer there is so interesting because as soon as you started talking about the examples that you successfully did some decoupling, you led with the business reason behind it.
36:14 Scott And I think that's the fundamental challenge that, you know.
36:20 Scott And developers, maybe the, you know, obviously as CTOs, it's our responsibility to sell the idea to decide if it's a good idea and
36:27 Brad Yeah
36:27 Scott then to sell the idea.
36:29 Scott And the developer might be just like, hey,
36:31 Scott like, I want to refactor this stuff because I don't like it, you know, and,
36:36 Scott you know, we've got it. We're the ones that have got to be like,
36:39 Scott look,
36:39 Scott I can't justify that expense.
36:41 Scott I need a reason that we're going to do it. And in those two examples you had.
36:45 Scott very clear business impact that
36:47 Brad Yeah.
36:47 Scott justified doing some of the work.
36:51 Kevin One of the challenges I've seen there is the business problem is it's hard for engineers to reason about the whole thing.
36:59 Kevin And as a result, that just slows things down and often causes unintended consequences.
37:07 Kevin And that's a lot less quantifiable than build time or system performance.
37:13 Brad That's absolutely true.
37:14 Brad Yeah, that's right.
37:16 Brad I think.
37:18 Brad You know, in both these cases, I was lucky enough to have a set of developers from early days who all knew the system pretty well. And that was obviously,
37:30 Brad you know, a weakness as well as a strength.
37:33 Brad But that allowed us to say, okay,
37:36 Brad well,
37:36 Brad in the case of Avvo, if we're moving the new stuff over here, so most people don't have to touch the old stuff.
37:43 Brad You know, anymore.
37:45 Brad The good news was,
37:46 Brad OK,
37:47 Brad well, that improved speed of development and kind of let people have.
37:51 Brad a better model in their heads of at least the stuff they worked on but it also meant that no there's this thing over here which is like really scary it's the haunted house that you you know dare each other to kind of go in the front gate of um and it was okay it was okay because we had the people that that were knew it but if you know someone happens and they were all on a bus together or something then we would have been in in real trouble that was a good
38:18 Brad Again,
38:18 Brad there's just a calculated risk, like, all right, let's take this more important to the business.
38:24 Brad I have to say the.
38:26 Brad My favorite, I haven't used it, but I try and recommend it to other people and see if it works for them,
38:32 Brad is I was at some conference and the chief architect at, what's the cool eyeglass, there was this cool eyeglass startup,
38:43 Brad Warby Parker.
38:44 Kevin Warby Parker. Warby Parker.
38:46 Kevin They should advertise with us.
38:47 Brad They should.
38:50 Brad The chief architect talked about their monolithic composition story and he did, and this was a bunch of years ago,
38:56 Brad but he did it, they did it in a way where they kind of took their monoliths, found the split points, split it into four.
39:03 Brad So they made it from one monolith into four monoliths and then from four monoliths into 16 and this very kind
39:10 Kevin Yeah.
39:10 Brad of I loved that and I've wanted to do that. I just never have. And instead,
39:16 Brad I recommend it to other people and wait to see if it works for them.
39:19 Kevin Yeah.
39:21 Kevin The sort of fractal approach.
39:23 Brad Yeah.
39:23 Kevin I've taken a slightly similar approach where
39:27 Kevin We,
39:27 Kevin I can't remember exactly how we did it technically, but we ran the same code base in different clusters.
39:35 Kevin So we didn't actually break the monolith up,
39:37 Kevin but I guess we did it at the network layer.
39:41 Kevin We routed different routes, different clusters.
39:44 Kevin So at least where we're starting to say like, look, I'll just use some stupid examples of counts versus checkout or something like that, right?
39:53 Kevin We're in different places so we can start seating with the team,
39:57 Kevin right? Like this is how we're starting to think about breaking this.
39:59 Brad Yep.
40:00 Brad But it was still all one blob and
40:01 Kevin Yeah, that's
40:02 Brad then it can sort of get decoupled.
40:05 Kevin really smart.
40:06 Kevin That's really smart.
40:07 Kevin I like that idea.
40:08 Brad We had another reason for doing it too,
40:10 Brad which was when you have a monolith,
40:12 Brad if you have, and this was actually a bigger driver,
40:16 Brad frankly,
40:17 Brad if you have one part of the system that is sub performing suboptimally, it is contagious to the rest of the system.
40:25 Brad And so what that did was actually isolate for us if the account system was performing. If you ship something
40:32 Kevin yeah
40:32 Brad that was slowed down the account system,
40:33 Brad you only screwed with the accounts.
40:35 Brad You didn't screw with all the rest of the system.
40:37 Kevin but the code was all still in the so the code the monolith was the code base was still a monolith you just had four instances of it and and split the the what what functions were running in which instance that was what you did to it
40:50 Brad I'm trying to remember how we did it, but I think we actually simply did it.
40:52 Brad they did it at the router level which is like
40:54 Kevin oh Yeah.
40:54 Brad if you come in on this route you know slash account you're going to go over to this cluster over here and if you come in on slash users you're going to go to this one over here and um so it started to help us think about how to break it up and then we got that other benefit which was it helped us isolate problems
41:11 Kevin Performance. And I mean, that gives you even one of the benefits of like a microservice architecture where you could then scale each of those clusters separately.
41:18 Kevin Right. So if user
41:19 Brad We
41:19 Kevin in SWOT isn't really slow.
41:20 Kevin Yeah, that's smart.
41:21 Kevin I like that.
41:24 Brad it solved another problem too so we had some pretty strong slas around endpoint performance and i remember getting into a i mean it wasn't an argument we weren't arguing but like a debate with an engineering manager i was like hey you got to deal with this these endpoints are out of sla he's like you know the support team hits these like once a day like who cares like i don't care about this this is not worth the time to go fix this and so that was another another
41:50 Brad aspect of it is that you could think about how different user segments had different standards you
41:56 Kevin Yeah,
41:57 Brad know the support team didn't need to have every endpoint come back in 100 milliseconds or 500 milliseconds whatever it was they could wait a little bit longer if they had to okay
42:05 Kevin Yeah. that's cool yeah
42:06 Brad so one more question about your book before we wrap up is there a chapter about AI assisted coding in
42:11 Kevin there No, because this ends it's for it's 2012 to 2022
42:16 Kevin So I managed to dodge that subject completely.
42:22 Kevin Yeah.
42:23 Kevin What are your opinions on AI-assisted coding?
42:25 Kevin I did use OpenAI to generate the index for the book so I wouldn't have to go through it by myself.
42:33 Kevin Cool.
42:33 Kevin Yeah.
42:36 Kevin My opinion on AI-assisted coding has kind of gone from one place to another and then actually coming back to my original idea,
42:44 Kevin which is AI-assisted coding is a tool.
42:49 Kevin um like any other tool like when they launched basic the entire point of basic was to get rid of computer programmers right it was going to be a light it was it was it was a language for every person right which is why you know that's how i learned how to code because it was easy approachable to me and but that was the the point oh we'll get rid of computer programmers and the number of times so you know we're all about the same age but you know i've been in industry third
43:16 Kevin 30 some year 33 years the number of times i've heard oh this is the end of software development like all software developers are going to lose their jobs now this is just the latest one what ends up happening is we get to take advantage of this tool we get to use it like a tool
43:32 Kevin software become we can now do much more complicated and much more powerful things than we used to be able to do because we have this new tool and that's only if we think about it as how do we adapt how do we take this tool adapt it into our workflow and just let us do better stuff because now we don't have to do the old stuff we used to do that's kind of the where i'm feeling it about it
43:57 Kevin But is it going to be a problem in the short term as we kind of figure out we get from where we are now to there?
44:03 Kevin Yeah,
44:03 Kevin absolutely.
44:05 Kevin Am I scared for junior developers and how did they learn?
44:08 Kevin Yeah, totally.
44:11 Kevin The number of people I've had approach me like,
44:13 Kevin hey, my kid or my cousin is graduating school.
44:17 Kevin There's no one hiring junior developers anymore.
44:20 Kevin Yeah, that's pretty freaking scary for us as an industry.
44:24 Kevin But I do think we will eventually figure this part out and it'll it'll just make us it'll just enable more complicated, more sophisticated, more interesting kind of experiences in the future.
44:39 Kevin That's where I'm at today.
44:40 Brad I have seen anecdotal data that it is the engineers earlier in their career who are embracing it more.
44:51 Brad They're not as rigid,
44:53 Kevin Yeah.
44:53 Brad you know, I mean, if you've been in it 30 years, you've got, you have some opinions about this, right?
44:57 Brad If you're newer,
44:58 Brad you're not as entrenched.
45:00 Brad I've also heard the opposite.
45:02 Brad not exactly opposite it is the more experienced engineers who are getting more out of it because they know how to instruct the robot what to do
45:10 Kevin That is exactly my experience.
45:14 Kevin The founder of District Kid who was the original developer,
45:18 Kevin but he's really pragmatic,
45:19 Kevin really open to trying stuff,
45:21 Kevin was still contributing code when I joined the company.
45:25 Kevin he was a big proponent of this and as was I I'm like this is a really interesting tool we should be looking at the hardest people to convince were all the senior developers because they're like this isn't going to do better you know this is going to generate worse code than me or whatever and once they got into it and once they realized oh you know what actually this means I don't have to code boilerplate anymore which is a waste of time instead I can do this other stuff yeah they're
45:52 Kevin They all figured it out and really are very, very powerful with it.
46:00 Brad Can I read you some stats that are just hot off the press that I just got today?
46:05 Kevin Yes, please.
46:07 Brad This person surveyed thousands of developers in 2024,
46:11 Brad saw 30 to 40% increase in the code generated.
46:18 Brad But 25% of that code required rewriting and bug fixes,
46:23 Brad but
46:23 Kevin Yeah.
46:24 Brad the net was a 15 to 20% increase in productivity.
46:29 Brad So the senior engineers are right.
46:31 Brad It doesn't write as good code as they do,
46:34 Brad but net works out better.
46:38 Kevin I am like.
46:42 Kevin we my team is a pretty small team and a handful like probably 50 50 like more senior more junior and it's so interesting I try to watch these I'm trying to watch the team really closely and I was telling them today in a team meeting I'm like you need to treat this
47:00 Kevin And when you if you ask the AI to do something, you need to treat it like another developer you asked to do something and now you are going into the role of the code reviewer.
47:09 Kevin You're going into the role of the of the auditor of this other code and you you cannot think of it as code you wrote.
47:17 Kevin You need to think of it as code something else wrote that now I'm putting my name on and therefore I need to like look at it interrogate it the same way I would.
47:26 Kevin Because we have a very strong culture of code reviews,
47:28 Kevin and so same way I would when I look at another developer's
47:31 Kevin Yeah
47:32 Kevin code.
47:32 Kevin Right
47:33 Kevin I think, you know, going back to this idea of the monolith, I think one of the biggest problems of these monolith code bases is it's impossible to find anything.
47:41 Kevin And you end up with a lot of duplication because it's impossible to find anything. So somebody's like,
47:46 Kevin oh, I need a function or I need a class that does this. I can't find one. I'm just going to make a new one.
47:52 Kevin And all of a sudden you've got business logic duplicated all over the place.
47:56 Kevin This is actually one thing that we've been looking into,
47:59 Kevin which is at DistroKid because we have a pretty large code base.
48:04 Kevin And yeah,
48:06 Kevin like it's, I'm finding like developers are writing new things that duplicate things that already existed because they didn't know it existed.
48:13 Kevin And so that is one thing we're looking for is like, hey,
48:16 Kevin I need a function or I need a class that does this or I need a function that does this. Is there one in the code base and actually we're getting pretty good results out of that.
48:24 Kevin So just also that value of helping with large code bases, helping you understand your own code base.
48:30 Kevin and and write and not even write code for you but help you write better code is is really interesting as well yeah
48:36 Kevin Yeah, yeah,
48:37 Brad All the teachers here.
48:39 Kevin love it
48:39 Brad It seems like that's a wrap for today.
48:43 Brad As two music guys,
48:44 Brad I'm going to finish on this note.
48:48 Brad One recommended album for our audience to listen to from each of you.
48:55 Brad We'll go with you first,
48:56 Brad Brad,
48:57 Brad and we'll let Kevin have the final word.
48:59 Kevin Uh,
48:59 Kevin I'm not going to go with an album,
49:01 Kevin but I will go with a new to me band that my wife and I just discovered last Monday.
49:07 Kevin We were we went to go see Franz Ferdinand here in Seattle and the opening band was this band we'd never heard of before called the telescreens and they absolutely rocked the house.
49:19 Kevin And then my wife's such a super fan of Franz Ferdinand, we flew down to San Francisco to see them again.
49:24 Kevin And on Friday, and then telescreens played a different set and they were even more amazing and chatted with the band afterwards, you know, at the merch table, I gave them a Velvet Underground meets the Gun Club meets Replacements meets Fugazi and they just just brought this amazing high energy poetic.
49:49 Kevin Punk sound that was just amazing.
49:51 Kevin The Telescreens.
49:53 Brad All right, Kevin.
49:54 Kevin I'm going to just say Taco Cat.
49:55 Kevin is my one of my new favorite Seattle bands I do not get out as much to shows as I used to but uh I still pay attention and um hadn't heard them and just picked them up they've been around they're not new but uh that's my new favorite band I found listening to them a lot
50:14 Brad That's fun to hear about,
50:15 Brad Kevin,
50:16 Brad because when I worked at Cheeseburger like 15 years ago now,
50:19 Brad a few of them.
50:21 Brad And the taco cat came out of Cheeseburger.
50:24 Brad The lead
50:25 Kevin Is
50:25 Brad singer
50:25 Kevin that right?
50:25 Brad and a few of the other members were working at Cheeseburger and then they all quit to go on tour to be on their band.
50:31 Brad So it's awesome that they're still around.
50:34 Kevin I don't know if they're still around.
50:35 Kevin It's more I just came across them.
50:38 Kevin But that's awesome.
50:40 Kevin That was 15 years ago?
50:42 Brad Well, yeah,
50:43 Brad I joined Cheeseburger in 2007.
50:45 Kevin Yeah, they started in 2007. According to Wikipedia,
50:49 Kevin there's still a thing.
50:50 Kevin There you go.
50:51 Kevin Oh, awesome.
50:53 Kevin I was still playing a lot of shows around then. I don't know how I missed them.
50:56 Kevin But all right.
50:57 Kevin Good to know.
50:58 Brad Well,
50:59 Brad Kevin,
50:59 Brad thank you for your time today.
51:00 Brad Where can people get your book if they want it?
51:03 Kevin My book is available at itdependspook.net.
51:08 Kevin It depends,
51:08 Kevin but all one word. So there's links to all the bookstores in the world that I know of that carry it there and the audiobook and all the various versions of it.
51:19 Kevin We will make sure to link that in the show notes.
51:21 Brad Cool.
51:22 Kevin Thank you. Thank you so much for having me on. It was a lot of fun.
51:26 Kevin Great meeting you,
51:26 Brad Thank
51:26 Kevin Kevin.
51:26 Brad you, Kevin.
51:29 Scott This episode of How Many CTOs Does It Take Podcast was edited by Saffron Heppegau. Our theme song was AI Generated by Brad.
51:37 Scott Please follow us on all the social media platforms at at howmanyctospod. You can find us on YouTube,
51:44 Scott LinkedIn,
51:45 Scott Instagram,
51:45 Scott Facebook,
51:46 Scott Threads, X,
51:46 Scott TikTok,
51:47 Scott and Patreon,
51:48 Scott or at our website at howmanyctospod.com.
51:53 Scott And remember to rate, review, and subscribe on the podcast platform of your choice.
51:59 Scott Thanks for listening.