Transcript: The Ball of Mud - Tackling Software Monoliths and Decomposing Tech Debt: Strategies and Challenges
0:00 Scott Maybe actually we could come up with an argument or a narrative or something to convince our business stakeholders actually that is the best thing to do is just invest X percent of your dollars every year in building putting new parts on the boat.
0:33 Brad I'm Brad Hefta-Gaub,
0:35 Brad that's Scott Porat, and this is the How Many CTOs Does It Take podcast.
0:38 Scott Don't forget to subscribe,
0:40 Scott rate,
0:41 Scott and review.
0:42 Brad Hit that smash that subscribe button.
0:45 Scott Smash it.
0:47 Brad Uh,
0:47 Brad so you just mentioned to me that you had a topic that you wanted to talk about. What was your topic that you feel like you want to attack today?
0:56 Scott I want to not just today for the rest of my life. There's actually three things in my mind where if somebody can solve these three things and these three things are the following.
1:08 Scott Measuring engineering productivity.
1:10 Brad Yep.
1:11 Scott measuring the impact of tech debt.
1:14 Brad Mm-hmm.
1:16 Brad So that you can make a case one way or the other about addressing it?
1:22 Scott Sure, or just, yeah,
1:24 Scott exactly.
1:25 Brad Just measuring.
1:26 Scott Quantitating the impact of it.
1:29 Brad Yep.
1:29 Scott Quantitating the impact of it.
1:32 Scott And I've developed some very expensive ways to very labor-intensive ways to do that,
1:37 Scott but developing some cheaper ways to do that.
1:40 Scott The topic for today is decomposing a monolith, and I feel like we could talk to every...
1:47 Scott engineering leader in the history of the world and try to get to the bottom of how to do this and let me set the scene for you I think you've probably been in this scene before you not only have seen this play before you've probably been in this play generally speaking web application web back end of a web application
2:05 Scott number of years old in particular the one I'm dealing with is
2:08 Scott a decade old although i've dealt with ones that have been five years old eight years old etc using some kind of framework you know could be a node kind of thing could be a you know django i happen to be working on a rails thing right now but it could apply to dot net you know asp whatever
2:25 Brad Yeah, totally.
2:26 Scott they're on these days it could apply to anything you know i've we've got about 100 engineers working on
2:33 Scott A monolith, when we say monolith, we're talking about a back-end, a giant back-end service that does many things,
2:38 Brad Mm-hmm.
2:39 Scott that has been adapted and evolved over the years as a business has grown and changed.
2:46 Scott And it has a whole host of well-intentioned yet poor technical decisions and just plain old poor technical decisions,
2:55 Scott you know, has reached the point where...
2:57 Scott engineers say hey it just takes us twice as long to do everything because
3:03 Brad Yeah.
3:03 Scott there is so much tech that there's so many places where there are if else statements because somebody wanted to hack something in or you have unintended consequences where you change something here and it breaks something over there and no one knows why etc etc etc
3:23 Scott And so, you know, the general strategy that I've seen is,
3:27 Scott hey,
3:27 Scott company,
3:28 Scott well, what are the two strategies? Strategy one is, well, we're going to write it from scratch. We're going to rewrite this thing.
3:33 Brad Second system.
3:35 Scott Second system could work, but it's really hard because you've got a decade of requirements in there that nobody really understands.
3:40 Brad Right.
3:41 Brad Nobody even remembers them.
3:43 Scott None of the people work here anymore,
3:44 Scott right?
3:47 Scott So it takes a really, really long time to do that.
3:50 Scott highly risky etc etc the classic Netscape 4 problem right they
3:55 Brad Yep.
3:55 Scott took you know it's kind of a joke but they took like two years to write Netscape 4 and what Internet Explorer came along and crushed them or something like that Internet Explorer is dead in fact I use Microsoft Edge for the first time last night that's beside the point
4:11 Scott the other strategy is well we're going to incrementally rewrite this
4:15 Scott And we're going to somehow create some separation of concerns so that we don't have such a ball of mud,
4:22 Scott right?
4:22 Scott The classic ball of mud paper,
4:24 Scott spaghetti code.
4:25 Scott And, you know, people try to do that either by enforcing boundaries within the monolith or by separating into different services or.
4:31 Scott components that can be imported from things like there's a different bunch of different strategies but they ultimately try to reduce the complexity by reducing by creating some boundaries and
4:41 Brad Mm-hmm.
4:41 Scott it seems like a slog and there's one other thing that i've seen that's common in these systems there's usually at least one but usually a few kind of
4:54 Scott I'll call them god objects or octopus objects right
4:56 Brad Mm-hmm.
4:57 Scott they have their tentacles into everything it
4:59 Brad Yep.
5:00 Scott could be the user object or if you have a transaction system it could be the order object like there's a thing which like everything touches and
5:08 Brad Yep.
5:08 Scott that tends to actually be the real bottleneck because nobody can actually figure out how to that's the obvious first piece to take out oh let's extract the order
5:18 Brad Mm
5:18 Scott from
5:18 Brad hmm.
5:18 Scott the system so
5:19 Brad Mm hmm.
5:20 Scott that it's really clean but that's actually the most obvious thing to get out or the user is the most obvious thing to get out those then become the hardest thing to get out because everything touches it
5:30 Brad Yeah,
5:30 Scott so you
5:31 Brad right.
5:31 Scott actually have to take out all the other stuff first and just leave it there but what then happens is you're just peeling off little parts that don't really make a difference okay so i peeled off the address surface well whoop-dee-doo it like nobody ever touches the address surface so it's like you know and so people start to peel off the little parts around the edge and then what you get is you still have the monolith it's slightly smaller but now you got new code in this new system that's nice and fresh and clean
5:56 Scott But you still got the monolith and
5:58 Brad Yeah.
5:58 Scott you can, unless your organization has a ton of resources and a ton of discipline.
6:04 Scott You put some effort into chiseling some stuff off, but you never really make a ton of progress. You just it's a grind.
6:09 Scott I'm on my third company like this,
6:11 Brad Yeah,
6:11 Scott by the way.
6:11 Brad exactly.
6:12 Brad Well,
6:12 Brad I okay,
6:13 Brad so first of all, it's funny because you started by saying or you kind of mentioned to me recently, you said, it's all about decomposing the monolith, right?
6:23 Scott Yeah.
6:23 Brad And that's the phraseology you use and the example you're giving right now of the ball of mud is that it's a monolith ball of mud and it's got like a
6:30 Brad like a god object or two in it and and so that's you know the scenario that you've just described is very is is kind of the the state that you're in you start your starting state um
6:41 Scott That's the state.
6:41 Brad but can i i guess i want to double check something or challenge something like i've i've been in that same play where it wasn't a monolith but it was
6:55 Brad a bunch of microservices but it was the same ball of mud and it was the same there was a god object or multiple god objects that were over you know they were the areas of concern weren't well defined or and so i guess is the problem actually the monolith or is the problem the 10 years worth of
7:17 Brad poorly specified user require poorly specified business requirements that have basically been you know glued and pasted and baling wired and and you know bubble gummed together whether it was a monolith or not
7:37 Scott Yeah,
7:37 Scott it isn't necessarily that it's a monolith.
7:39 Scott I think the way I would say it is you have a system that is resistant to change.
7:45 Scott and it's resistant to change because it has many
7:48 Brad international needs
7:49 Scott many yeah many areas that are tightly coupled um
7:52 Brad yeah
7:52 Scott so even if you just want to go do an ordinary feature oh and and the footprint of the system whether it's a monolith or many services or whatever is large so if you want if you want to go oh and typically
8:07 Scott you got a lot of new people right so
8:08 Brad Yeah, sure.
8:09 Scott because any organization that's 10 years old has a lot of new people right so if i'm an engineer and you want me to go make have a simple feature right hey display the user's birth date right well why is it resistant to change first i have to do a ton of investigation just to figure out how to do it or i don't and it maybe has unintended consequences or i do and it still hasn't
8:32 Scott I don't think that consequences
8:33 Brad Yeah,
8:33 Scott because I didn't even know about that service over there.
8:36 Brad right,
8:37 Scott Nobody mentioned that to me, right?
8:38 Brad right,
8:39 Scott And so somebody comes along and says,
8:42 Scott why is it going to take two months to display? We have a database here which has a birth date in it and I just want to put a birth date on the user's profile.
8:51 Scott And why is that going to take two months exactly?
8:53 Scott You
8:53 Brad right.
8:54 Scott know?
8:56 Scott And it's because, well, there's these 74 services in between we got to change.
9:00 Scott I'll give you actually a real,
9:03 Scott real concrete example of this one.
9:05 Scott I remember from my time at Rover,
9:07 Scott we wanted to change the number of days in the future that somebody could schedule.
9:14 Scott a dog sit and I thought well that's going to be easy there's just a constant somewhere that's right now it's 120 and we just need to make it say 400 or whatever the number we need to be is right and then the estimate comes back like at two months and I sat down with the engineer and I was like okay he was a he was a really good engineer so I trusted him but I was like I just need you to explain to me what why is this taking so long and he walked me through it
9:39 Scott And the answer was, I remember this number specifically, there were 21 places in the system that depended on this thing.
9:47 Scott and a lot of them were kind of hard some of them were hard coded and some of them looked at the constant and so on and so forth and it just wasn't as simple as you thought right because
9:56 Brad Yeah, yeah
9:57 Scott of all this time so to me it's the interdependencies or the tight coupling which is the source of the issue and you get tight coupling from poor design intentional poor design or unintentional poor design what I mean by intentional is just like
10:14 Scott like somebody did it and they weren't doing a good job or unintentional as they did it and they made a mistake we all make mistakes right
10:21 Brad Yes,
10:21 Scott you
10:21 Brad really.
10:21 Scott know that we come back and look at and look oh that was a bad way to design that right and that's ultimately I think where it comes from and then I think it gets compounded in times of stress you know where a business is asking us to do something quickly and we just think well I'll just jam this in here
10:40 Scott And it really doesn't seem like that big of a deal, but, you know,
10:44 Scott 10 years, everybody does one of those a year and you've got 100 years on your team and you do it for a decade.
10:49 Scott That's a thousand of those things.
10:50 Brad Yeah, totally.
10:52 Brad Well,
10:52 Brad so the question at hand is, how do you solve this?
10:57 Brad Like, how do you make it so that you're 10 years into this thing?
11:04 Brad And what do you do to make it so that you're not every single small request is a month long project?
11:13 Scott Yeah.
11:13 Brad Okay, so let's, is there,
11:17 Brad do we have examples of teams that have done it really well?
11:25 Brad Like, or is this just everyone always struggles with this?
11:29 Scott Well, that is a great question. And if we had examples of teams, anybody who is listening to this podcast knows of anybody in
11:38 Brad who doesn't
11:39 Scott the world
11:39 Brad get into this who after
11:40 Scott done
11:40 Brad 10
11:40 Scott well,
11:40 Brad years doesn't get into this totally
11:42 Scott well, okay,
11:44 Scott there's two questions, people who don't get into this, but people who've gotten out of it, there's two different things. How do you, okay,
11:49 Scott so
11:50 Scott The fastest way out of a hole is to stop digging.
11:53 Brad Yep.
11:54 Scott So who's figured out how to stop digging,
11:56 Scott right,
11:56 Scott and at least get out of it? And that's a different question than how do you never get into it in the first place.
12:01 Brad Yeah,
12:02 Brad agreed.
12:03 Brad Agreed.
12:06 Scott Now what I've also seen well-intentioned teams draw figuratively speaking into diagrams,
12:11 Scott but they want the system to look like.
12:13 Brad Mm
12:13 Scott right
12:13 Brad hmm.
12:14 Scott well we're going to take this order object this god on you we're going to extract it out we're going to extract the user out right that sounds nice but the devil's in the details right you have to build some sort of bridge i guess
12:28 Brad Yeah.
12:28 Brad And I think that's usually the right that tends to be the technique that most teams attempt.
12:34 Brad Right. Is they pick the.
12:36 Brad You know, you did a great job of teeing this up in the introduction where it's like you kind of like pick the the object that's the least constrained object and you,
12:45 Brad you know, slice it and cleanly put it to the side,
12:49 Brad you know, and you never get to the place where.
12:53 Brad The over-dependent object is the one you're attacking.
13:00 Scott That's right, because the most constrained object is the one that people use the most.
13:08 Scott There's one other thing I want to throw into this sauce,
13:11 Scott a little seasoning in the soup.
13:13 Scott I could be wrong about this because you mentioned the situation with having it be.
13:18 Scott you know a service-based architecture I've seen this problem when you're using an ORM right it's so easy to just go connect this object to that object with an ORM and
13:30 Brad Yeah.
13:30 Scott it's all behind the scenes and then your whole system uses an ORM but you want to extract one piece of it to say not use an ORM
13:40 Brad Well,
13:41 Brad I've seen it in another example I've seen it is somebody starts out where they're like, oh, well,
13:45 Brad we're going to like not get into this mess. We're going to keep ourselves from getting into this mess and we're going to we've got this great idea.
13:52 Brad We're going to have a I don't know, I'll pick some example. We're going to have a.
13:58 Brad message bus and we're
14:00 Scott Mm-hmm.
14:01 Brad going to keep everything all the microservices cleanly separated and they just use this message bus to talk to each other right
14:09 Brad But then next thing you know, it's like the message bus is overly type specific and the message bus becomes the thing that's over constrained and you,
14:18 Brad oh,
14:19 Brad oopsie, I added a new message type,
14:22 Brad but I didn't update all the other components to know that this new parameter exists on this message type and boom,
14:28 Scott Yeah.
14:29 Brad crashes the message bus or all the components that didn't know about it start crashing,
14:34 Brad right?
14:35 Scott You want to hear a doozy?
14:36 Brad Sure.
14:38 Scott Organization I heard of. They were using an ORM.
14:41 Scott They were using the on change event from the ORM to spawn messages into the message bus that then other objects or whatever would listen to other parts of the system would listen to.
14:55 Scott So including the analytics system would use those events to.
15:00 Scott collect you know telemetry about the app and um so then it's even worse because you're the putting of messages on the bus oh and so by tapping into the unchange event it's getting the raw data and putting it on the message bus so by by it's so now people don't actually even really know
15:23 Scott when they're breaking if I have a message and I do it and then something breaks oh I changed that message that must have been me right
15:29 Scott But when you're just changing an object and the ORM is saving it and then something is breaking,
15:33 Scott nobody knows. It's
15:34 Brad Right.
15:34 Scott like, well, why is this breaking all of a sudden?
15:35 Brad Right.
15:37 Brad We're we're our this is a classic tar baby.
15:40 Brad Right.
15:41 Brad Like we just it starts getting stickier and we start getting more stuck to it. Right. So it's not even just, oh,
15:48 Brad hey,
15:49 Brad you know, the answer is decompose your monolith because if you use microservices, you'll prevent yourself.
15:54 Brad yourself from getting into this mess nope nope you can you can still be in this mess with microservices oh
15:58 Scott Mm
15:58 Brad um
15:59 Scott -hmm.
15:59 Brad let's use like strategies of of um you know decoupling like you know message buses and things like that oh well no you can still get yourself into this mess and so here we are we're still stuck with this problem i the more i think about it um what's the what's the what's the uh um
16:21 Brad The thought experiment of the ship that I replace one piece at a time, if I replace every piece of the ship one piece at a time, do I still have the same ship at the end? It's
16:33 Scott Mm
16:33 Brad not Prometheus, but one of the, you know, one of the Iliad, right?
16:38 Brad Like one of the Greek or,
16:41 Brad you know, epic poems.
16:44 Brad The teams that I've seen that I'm not going to say they're going to hit it out of the park. I'm not saying they've solved this problem,
16:51 Brad but the ones that I've seen manage this problem the most are the ones that essentially don't try to rewrite the whole system. Don't fall for second system effect, but they do basically rewrite pieces one at a time.
17:07 Scott Yeah.
17:10 Brad try to be as intentional as possible that's the only that's the only path i've seen have
17:16 Scott Yeah.
17:16 Brad a hope
17:18 Scott Yeah.
17:18 Scott So, okay.
17:19 Scott This would be a great topic to talk to people about. Let me ask you another question.
17:23 Scott Okay.
17:23 Scott So team I'm working with now recently embarked on something like this. They, the commerce portion of what they're doing, you know, payments and stuff like that,
17:32 Scott they extracted.
17:34 Scott And one of the engineers on the team that I like, he spent four,
17:40 Scott I talked about the octopus problem,
17:41 Brad Yeah.
17:41 Scott he spent four months figuring
17:45 Brad Researching.
17:45 Scott out where all the touch points were.
17:47 Brad Yeah. How many legs this octopus had.
17:49 Scott So, yeah,
17:50 Scott that's right.
17:51 Scott He wrote actual, he actually wrote some software to profile what was happening so he could understand where this thing.
17:59 Scott interactive because one of the things that happened was you know there would be like the virtual sort of dead code and you wouldn't even know if you needed to do it so we spent all this time doing this they did it it succeeded it was a big huge effort they had to have he and he and three other people spent four months figuring out what to after the new service was built four people spent four months figuring out what the touch points were going to be able to fix and then 20 people spent six months doing it
18:26 Scott This huge effort,
18:27 Scott which gets me to item number two, item number one on my list, maybe it's item number two, the impact of measuring the impact of these things.
18:36 Brad Yeah.
18:37 Scott So all our business partners are saying, well,
18:39 Scott that's great.
18:40 Scott I now have a cleaner version of what I had before.
18:43 Scott Exactly how did that help me out?
18:45 Brad Exactly.
18:45 Scott Well, it's making everything go faster. In fact, the team is already reporting that projects are going faster.
18:52 Scott You know, but to my business partners,
18:55 Scott stakeholders,
18:56 Scott I mean, that doesn't mean anything to them.
18:59 Scott They understand the words and they have some trust, but they don't feel it, right?
19:03 Brad Right,
19:04 Scott They don't feel it. They don't feel it like they feel money on the balance sheet.
19:07 Brad right,
19:07 Scott That's for sure.
19:08 Brad right.
19:08 Scott So how do we measure the benefits of these things? Let's say that it really does take a long time and a lot of hard, discipline, patient work.
19:17 Scott It's the only way.
19:19 Scott It's really hard to get a business to fund that for the long term because they're just saying, I don't understand what I'm getting for this,
19:25 Scott given
19:25 Brad Yeah,
19:26 Brad right.
19:26 Scott that we can't measure engineering productivity,
19:28 Scott even though this is the kind of thing that is improving the productivity.
19:32 Brad Yeah,
19:32 Brad I mean, I think that I think that the right.
19:34 Brad So we talked a little bit about this in a previous episode where we were talking about the idea that if you're going to measure the.
19:43 Brad performance or the value being added by a platform team for example one of the ways to measure that is are the application teams sitting on top of that platform team are they able to move faster are they they're
19:59 Scott Do their, you know, projects take less whatever unit of measure you use to measure productivity?
20:08 Scott And so it's really hard to say this will definitely produce this result.
20:14 Scott But once you've done the work, you ought to be able to have some metric of projects that touch this now go faster.
20:24 Scott Right.
20:25 Brad In theory,
20:26 Brad yeah,
20:26 Brad that's what we're hearing, you know, go to lay for sure.
20:29 Scott Right.
20:30 Scott Right.
20:31 Brad What do you think about, you know, a few years ago, Google came out with the Dora metrics and
20:36 Scott Yeah, yeah, I like the Dora metrics.
20:38 Brad one of those metrics is cycle time,
20:40 Brad if I recall.
20:41 Scott Yeah.
20:42 Brad But it called for lead time for changes, what they call the amount of time it takes to get commit to production.
20:46 Scott Yep.
20:47 Brad Let's take a look. What does that mean exactly?
20:51 Brad lead time for changes how long does it take to get to go from code committed to code successfully running in production that's really not cycle time that's i mean to go from code committed to code successfully running is nothing it's really like from someone has an idea to when the code is live how
21:08 Scott Yeah, exactly.
21:08 Scott That's what I want.
21:09 Scott I want time from idea to production.
21:14 Scott That's what I want to measure.
21:16 Brad do you think you would control for scope
21:18 Brad I mean,
21:19 Brad a small idea and a big idea
21:21 Scott Sure.
21:21 Brad are different.
21:22 Scott Totally.
21:23 Scott Well, okay.
21:24 Scott I had this question for you. I actually, I remember sending this to you between our last two episodes. I was like,
21:32 Scott how do you, I'm going to get to your question about Dora and how do you control for scope?
21:37 Scott I'm going to get there for a second, in a second.
21:38 Scott On a team of 100,
21:40 Scott right?
21:40 Scott Okay, so you're the CTO, you've been airdropped in,
21:42 Scott you're getting to know this team.
21:44 Scott There's 120 engineers there.
21:46 Scott Do all of your teams.
21:48 Scott use the same process like do they use the same metrics do they use the same whatever mechanism you use for say estimates whatever mechanisms you use for tracking progress are all the teams using the same process
22:07 Brad I can't tell you whether they are right now,
22:10 Brad but.
22:12 Brad given my choice,
22:13 Brad they would be.
22:14 Scott Right.
22:16 Scott So the reason I ask that question is because.
22:19 Scott One of the things I've seen often is the sort of like organic nature of the organization or small d democratic nature of the organization is you somehow said,
22:32 Scott yeah,
22:32 Scott yeah, let the teams come up with their own processes.
22:35 Scott Right.
22:35 Brad Yeah.
22:35 Scott And now you're sitting here at the, you know, at the CTO level and you're like,
22:41 Scott well.
22:44 Scott I have no idea if Team A is, I can't measure Team A's performance relative to a benchmark because they're not even following the same process as the benchmark.
22:59 Scott So
22:59 Brad Right.
22:59 Scott I don't know,
23:01 Scott like you're over there using story points, you know, and everyone else is using.
23:09 Scott I don't know, T-shirt sizes,
23:11 Scott right?
23:11 Scott So now we have this problem where we don't even have the same language,
23:15 Scott right?
23:16 Brad That's right.
23:17 Brad Well,
23:17 Scott How
23:17 Brad what's interesting is that,
23:19 Brad so for that problem I found a way to correct for
23:22 Scott do you correct
23:23 Brad that.
23:23 Scott for that?
23:24 Brad To me, I could do 10 episodes on this.
23:28 Brad I probably already talked about it in every one we've talked about.
23:31 Brad To me, the key measurement is estimation accuracy.
23:36 Scott Right.
23:36 Brad And as a team, I don't really care how you want to estimate.
23:41 Brad I'm just going to measure how good you are at it,
23:43 Brad which is when did you say you were going to get something done and when did it get done,
23:46 Brad right? And if you want to use...
23:49 Brad planning poker or you want to make numbers up or you just want to let the boss put it on a schedule it doesn't matter everybody's the boss the team manager put it on a schedule it doesn't matter to me i mean it doesn't matter i prefer the teams i'll do it the same way but ultimately what matters is where they accurate right but usually the way our industry does it is we do despite the effort of story points trying to be units of value you
24:17 Brad We do estimate in time.
24:19 Brad Let's say you've got this project takes you four weeks.
24:24 Brad And now you're going to do some work so that a large project only takes you, you know, you fix your system that's all broken.
24:33 Brad So you do some work so that a large project only takes you two weeks.
24:39 Scott That's fantastic.
24:40 Brad Yeah,
24:41 Brad it's great. You made a tremendous investment.
24:43 Brad But now when the next project that comes along,
24:46 Brad you're not going to call it a large project anymore.
24:48 Brad You're going to call it a medium project,
24:50 Brad right?
24:50 Scott Right.
24:51 Brad Because it only takes you two weeks.
24:53 Brad And so...
24:56 Brad you'd like to be able to say well look the engineering team did this work and we used to only be able to do 10 large projects in a quarter and now we can do 20 large projects in a quarter Yep. but what happens is we used to be able to do 10 large projects and we can still do 10 large projects they happen to just be bigger projects but we don't really get that signal right because we
25:18 Scott I adjust think that.
25:18 Brad you know it's like we don't really have a way of a unit of engineering work we need that we need to
25:24 Brad We the what would we call that a gates now
25:29 Scott A clock,
25:30 Scott of course. No, I'm just kidding.
25:33 Scott Thousand lines of code.
25:34 Scott Well, look,
25:35 Brad I get it. I get it. I was trying to I was trying to think of that made sense,
25:38 Brad but it doesn't make any sense
25:39 Scott it's the classic worst possible metric.
25:43 Brad Maybe we should call it a brook didn't wasn't he did the mythical man month was not brook didn't he write that we should call it a brook a unit of engineering work is a brook
25:51 Scott All right.
25:52 Scott Well,
25:52 Scott OK,
25:52 Scott so I think that like, OK,
25:54 Scott so let me just like.
25:56 Scott Try to rewind and pull on the thread a little bit here.
25:58 Scott So, okay.
25:58 Scott So remember,
25:59 Scott we start out with, we're starting with this challenge of how do we measure that the cleanup of the technical debt,
26:09 Scott the investment that we put into making this thing better?
26:12 Scott Why?
26:13 Scott How do we prove that that was useful? And the answer that I think we all agree with, we agree is it should either fixate a stability.
26:22 Scott a stability issue that we've been haunted by,
26:25 Scott or it should make us more efficient as an engineering team.
26:28 Brad should reduce that cycle time which
26:29 Scott It should reduce that cycle time from idea to production.
26:34 Scott And the only
26:35 Brad is
26:35 Scott way
26:35 Brad crazy
26:36 Scott we,
26:36 Brad because if I have more ideas the time is longer right if you can do if you like if you can do one idea if you can do one idea a month and I can do one idea a month but you've got ten ideas and I've got two ideas
26:48 Brad your cycle time will be longer well
26:51 Scott Well,
26:51 Scott if I am if I if I can get twice as many ideas done in the same amount of time that you can get an idea done,
27:01 Scott then I get twice as much work done. Right.
27:04 Scott Like so
27:05 Brad what
27:05 Scott the if same.
27:05 Brad your if your ideas are half the size the reason i have two and you have ten is it's five times bigger
27:10 Scott Well,
27:10 Scott this is the thing. Right.
27:11 Scott Then now you have to have a way of measuring the value of the idea.
27:16 Scott Right?
27:16 Brad Well, value is interesting because that's a subtle choice of word.
27:19 Brad You could have chosen size of the idea.
27:21 Scott No,
27:21 Scott value of the idea.
27:23 Scott I want to deliver more value to the customer more quickly.
27:27 Scott So
27:27 Brad that's
27:27 Scott that's what I
27:27 Brad dollar
27:27 Scott would want.
27:28 Brad it's incremental dollars per unit of time oh holy shazal is the metric incremental is it essentially i don't know if this is quite right is it revenue growth per engineer salary dollar
27:43 Scott I mean, that's not a bad way of measuring like, is it worthwhile?
27:46 Scott Like that would be the measurement for me,
27:48 Scott like, should I build this software myself or should I just buy something off the shelf?
27:52 Scott shelf right like
27:53 Brad Right. Basically,
27:54 Scott revenue
27:54 Brad last year we could
27:55 Scott amortization of however i got the software built right well
28:00 Brad generate a million dollars of revenue with 10 people and this year we can generate $2 million of revenue with 10 people.
28:06 Brad But it's the tricky one because there are so many different factors that go into what caused that revenue to increase.
28:12 Scott another another one of our problems here that we have to be honest with ourselves about is
28:17 Scott What if the ideas coming out of our product team are just dumb ideas and aren't going to drive any revenue?
28:24 Brad that's what i was saying is that there's a lot of things that go into that it could be the dumb ideas it could be the marketing maybe marketing did a better job this year they had a better ad campaign and they drove a bunch of revenue like there's a bunch of different reasons it's really really hard to connect it to revenue or whatever metric it is
28:41 Scott Right.
28:42 Scott Right.
28:43 Scott Yeah.
28:44 Scott I mean, this is this is the fundamental. This is hard.
28:46 Scott This is hard.
28:47 Scott Well,
28:47 Scott so I guess the question here is, given that this is definitely going to be hard,
28:54 Scott what are the areas that we OK,
28:56 Scott so I want I want to throw this out. Right. I asked you the probative question of are all your teams even following the same process?
29:03 Scott Right.
29:03 Brad Right, yeah,
29:04 Scott And
29:04 Brad yeah.
29:04 Scott I and I think that that's like, I mean.
29:08 Scott Not only do you, what if you have teams that not only are not following the same process,
29:14 Scott but they're actually following a bad process?
29:18 Scott Like they're actually like not doing something that if you were to set a break point and look at what they're doing,
29:26 Scott you and every other like reasonably experienced engineering leader would say, what are you doing? That's not the way to write software.
29:36 Scott Right.
29:37 Scott How do we solve for that?
29:39 Scott Are you?
29:41 Brad it's just freaking i don't know if we're shaving a yak or pulling the thread on a sweater but we're doing one of them right now i mean it's just all crazy well i think this is yeah i mean what do you do what do you do about that i don't know either um
29:53 Brad it's I mean it's all it's all intertwined I guess let's just can I tweak the subject for
29:59 Scott Yeah,
29:59 Brad a second go
29:59 Scott totally,
30:00 Scott totally.
30:00 Brad back to where we started a little bit let's just we only reason we went down this productivity path was when you said well the only really way to do this is to just sort of chisel work away at it for a long period of time like I don't think he said Prometheus's boat I don't know whose boat that is he doesn't sound like a sailor seems like a sailing kind of situation but
30:20 Brad you don't make it like you know the only way is to be like a sailboat and just carefully just go through the whole thing and replace every part one part at a time before you know it you got a new boat right and the only reason i went down the productivity route there was to say well if you're going to take that strategy
30:35 Brad it's going to be it's going to be tough for a business to digest endless investment for no visible return right
30:45 Brad But maybe, this is where I wanted to change the subject to,
30:47 Brad maybe actually we could come up with an argument or a narrative or something to convince our business stakeholders actually that is the best thing to do is just invest X percent of your dollars every year in building, putting new parts in the boat because that's just the only way to not have your boat fall apart in the middle of the ocean.
31:11 Brad Right.
31:12 Scott I mean, that would be true for hard infrastructure,
31:17 Scott right?
31:18 Scott Like if you own a factory,
31:21 Scott you can't not put maintenance into it.
31:25 Brad So, yeah, I mean, it's interesting. A factory is very different in the sense of, I mean, I think one of the things that makes.
31:33 Brad software complicated is it's under constant change the
31:36 Scott Yeah.
31:37 Brad factory is not under nearly as much change maybe it is maybe it's getting used i don't know i went through the ballard locks this week they close it for either two or four weeks a year i think two weeks a year for maintenance so
31:53 Scott For maintenance.
31:54 Brad two weeks out of 52 and it's getting used all day every day and it's over 100 years old
31:59 Scott Yeah.
31:59 Brad So let's think about that for a second.
32:01 Brad Software typically gets capitalized on a three-year schedule.
32:05 Brad Does that mean every year we should be spending a third of our resources to build this to maintain it?
32:13 Scott I mean, isn't that the theory behind what depreciation and capitalization is all about?
32:20 Scott Like you're basically saying like the lifetime of this thing is three years.
32:26 Brad but let's think about that for a second if we've got 10 engineers let's just make it nine engineers to make our math easier nine engineers and
32:34 Scott Three of them are constantly working on fixing a third of the system.
32:38 Brad yeah but six of them are making new system right so it actually starts compounding so do you end up in some kind of situation where like after
32:50 Brad Five years like 75% of your team is actually working on replacing parts of the boat
32:54 Scott Well,
32:55 Scott again,
32:56 Scott like this is the this is kind of like what I was trying to say is a it's got nothing to do with what the code architecture was,
33:02 Scott whether it was a monolith or microservice.
33:03 Scott It has much more to do with the 10 years of product requirements.
33:08 Scott Right.
33:09 Brad Well, in deferred maintenance,
33:10 Scott And
33:11 Brad it sounds like.
33:11 Scott years ago I worked with a developer and his his like claim to fame was he always wanted to go look at his Git.
33:21 Scott um history and have more deleted lines of code than added lines of code
33:26 Brad Mm
33:26 Scott like-hmm. that was his measure
33:27 Brad Yeah.
33:27 Scott of that was that was what he said like i am a great developer because i have always deleted more code than i've added right so it was funny because like anybody that ever like you know added some module or something like that this guy was the first person to be like i'll i'll take the ticket to delete that module that we decided we don't need anymore
33:47 Scott more because you know he would just be he would be deleting code that other people added that was you know easy you know he was the guy that was kept doing appendectomies on on the code you know and um and and it was funny it was like it was comical and also he's not wrong like if you're the person that can if you're if you're actively getting rid of pieces of the boat that we're no longer using
34:15 Scott That's very valuable to the organization.
34:18 Brad Right. I mean, effectively,
34:19 Brad if you think of your code as your plant and equipment and there's less you have to maintain,
34:23 Brad because maybe the strategy here is actually more of a financial argument to get your business people and your finance people to buy into just not to be looking for an ROI,
34:34 Brad but to just be like, well, of course, I mean, we just have to maintain our system and that that should be something like a third of your resources.
34:43 Brad for any given level of system let's just think of a team with three people let's just think of one person in year one that person writes one year of new code in
34:52 Scott Mm-hmm.
34:52 Brad year two they write a second year of new code in year three they write a third year of new code now in year four they write they have to refactor the first year new code so they didn't write any new code
35:08 Brad So that means over a six year, and that happens again for three years,
35:10 Brad so over a six year period, it's been 50% of their time has been new code versus.
35:17 Brad So is that it? Is it 50%?
35:19 Scott Well,
35:20 Scott I think in realistically what actually happens,
35:22 Scott right,
35:22 Scott is on year one, you write new code,
35:24 Scott year two, you write new code, year three, you write new code,
35:26 Scott year four, you delete half of the code from year one, you rewrite half the code from year one, and you add another half a year code.
35:35 Brad Got
35:35 Scott Right.
35:35 Brad you. So at that point, at that point, what have you got? You've got three
35:38 Scott Now you've got now you've got three and
35:41 Brad and
35:41 Scott a half
35:41 Brad a half.
35:41 Scott years of fresh code.
35:44 Brad Well,
35:44 Brad but half of the half of those three and a half, one half a year is tech debt.
35:48 Scott Exactly.
35:48 Brad is deferred one half year is deferred maintenance and
35:52 Scott Right.
35:52 Brad so let's say you do that again the next year now you've got four years with a year of tech debt and let's say you do that again you've got four
35:58 Scott Four and a half years with it.
35:59 Brad and a half years
36:00 Scott A
36:00 Brad a
36:00 Scott year
36:00 Brad year
36:00 Scott and a half.
36:00 Brad half and so you know one more year you've got five years of two years of tech debt one more year you got five so you're starting to catch up right
36:09 Brad But it's probably something like 50%. Now,
36:12 Scott Yeah.
36:13 Brad how do we convince our business partners to invest 50% of their precious engineering resources in maintenance?
36:19 Scott Yeah,
36:20 Scott that's a big ask.
36:21 Scott That's a pretty big ask.
36:23 Brad It is a big ask.
36:25 Brad it's kind of the thing they need right and then if you're in a hole if we're in that situation where you've got the five years of tech debt because you've been doing it that way now you not only have to ask for the 50 to keep up you gotta you have more to dig out it's
36:39 Scott Yeah. Well,
36:39 Brad kind of insane yeah
36:41 Scott and that's why what second system effect is really about is declaring bankruptcy,
36:47 Scott right?
36:49 Brad Yeah.
36:49 Scott Like I'm just I'm just picking up and moving away from the factory that's got all the broken down equipment in it.
36:57 Brad It's just like when people tear down an old house, take their car to the junkyard instead of fixing it.
37:01 Scott Yeah, I'm notoriously bad.
37:03 Scott I, you know, I try really hard not to ever let this happen at work, but my personal email inbox is just like, you know.
37:11 Scott gets gets overloaded i'm not i'm really horrible at zero inbox right and so every once in a while i just declare inbox bankruptcy i just like move all of the entire inbox into a folder called bankrupt bankrupt inbox and now i got a zero inbox nice
37:26 Brad Well, that's interesting.
37:27 Brad I am the opposite of you. I'm an inbox zero guy. I was an inbox zero before we got on this call today.
37:33 Scott so how much of your day is spent
37:37 Scott keeping your inbox at zero right
37:39 Brad Not much.
37:39 Scott well that's good then maybe that's a is there a clue here what's
37:44 Brad Yeah.
37:45 Brad First clue two,
37:46 Scott don't
37:46 Brad don't send emails because
37:48 Scott write when new code
37:49 Brad you send emails,
37:51 Brad people
37:51 Scott when you
37:51 Brad respond
37:51 Scott write new code
37:52 Brad to them.
37:52 Scott when you write new code you have code to maintain that's
37:56 Brad Don't respond to emails.
37:58 Brad Don't respond to emails.
38:00 Brad Because when you respond to emails,
38:01 Brad people think it's a way to communicate with you.
38:04 Brad And if you don't respond, they'll find other ways.
38:06 Brad I'm just joking.
38:07 Brad I would say the answer is I don't check email that often.
38:11 Brad I don't stay glued to it all day.
38:13 Brad And I have found some systems, I would be happy to talk about this at another time,
38:17 Brad but I've found some systems for processing quickly through a lot of email.
38:22 Scott Here's my point, by the way. I wasn't actually trying to raise the topic of inbox email inboxes.
38:27 Brad Yeah,
38:27 Scott I'm saying, is there a corollary there to the developer problem that we're facing?
38:32 Brad right.
38:33 Scott Right. Like I just saw this really interesting meme on the Internet. It was like a ADHD tip.
38:38 Scott Instead of putting it down,
38:41 Scott put it away.
38:43 Scott Right.
38:44 Scott Like people that have ADHD is super hard.
38:46 Scott Like I put it down like my now my office is completely strewn with stuff.
38:50 Scott Right.
38:50 Scott No, no. Instead of putting it down, put it away.
38:53 Scott Right.
38:55 Scott So, again,
38:55 Brad Yeah
38:55 Scott like I'm not trying to simplify the notion of strategies for dealing with ADHD,
39:03 Scott but I think there's an interesting corollary here between our problem of technical debt.
39:11 Brad Hmm?
39:11 Scott Right.
39:12 Scott It's not dissimilar from the problem of inbox debt.
39:18 Brad No.
39:19 Brad But I'd say the difference is you can still receive email when you declare bankruptcy versus you don't have to go reconstruct all those messages and all those communications that somebody sent to you.
39:34 Brad You know, it seems to him.
39:36 Scott Yeah.
39:37 Brad Well, to me, this is the this is the most interesting topic of our time.
39:41 Scott Right.
39:42 Brad If we can solve this, we can solve climate change.
39:44 Brad That's just what I'm saying,
39:45 Brad because it's the vexing problem that I see every organization getting into and then they got to get out of it because their system. And that's what I'm seeing now in my third organization in a row.
39:59 Scott actually really the fourth in a row it gets to this point where the resistance the system's resistance to change becomes commercially impractical for the company right essentially
40:10 Brad Right.
40:10 Scott they all essentially they are becoming tech debt bankrupt and they're they're hiring us to get them out of bankruptcy to do a restructuring as one would might say right and the question is how do you do it
40:24 Scott Right. And I hear you saying here the only way to do it is to patiently go through bit by bit.
40:29 Scott And
40:30 Brad Well,
40:30 Scott then I
40:30 Brad and I think and I think and the other thing I don't know if I specify I didn't put enough emphasis on this particular syllable,
40:37 Brad but I think that like being willing to cut things is
40:43 Scott Yeah.
40:43 Brad a really critical part of that.
40:46 Scott Yeah.
40:46 Scott People have a really hard time with that.
40:49 Brad Especially
40:49 Scott Right.
40:50 Brad when they don't know what you're asking them to cut.
40:54 Brad Right.
40:54 Scott Yo, I would actually say there's even another hard thing in there.
40:56 Scott Well,
40:57 Scott while you're in there problem.
40:58 Brad Yeah.
40:59 Brad Right.
41:00 Scott Oh,
41:00 Brad It's
41:00 Scott well,
41:00 Brad good.
41:01 Scott if you're going to be if you're going to be refactoring that thing,
41:03 Scott could we just make this change to it?
41:06 Brad Right.
41:07 Brad Right.
41:08 Brad Exactly.
41:08 Brad Exactly.
41:11 Scott That just makes it go backwards.
41:12 Scott Well,
41:13 Scott this is we have got to find we got to get some guests on here.
41:15 Brad Yeah, they that have that have tackled this problem.
41:19 Scott either succeeded or failed and help us figure out how to do it i
41:23 Brad I love it. I love it.
41:24 Scott will say one thing a company that i was at haven't been there for several years but they're about six or 12 months ago somewhere that company the cto did call me i was talking to him the new guy and they had been trying to refactor one of their mobile apps
41:41 Scott and he called to tell me they were done and it took us we had started together and it took four years to
41:46 Brad Wow.
41:46 Scott do a to do a mobile app so
41:49 Brad Wow.
41:50 Brad Yeah.
41:51 Scott it's it takes time yeah
41:54 Brad Yeah,
41:54 Brad it's hard.
41:55 Brad It's way easier if you basically start with a fresh slate of requirements.
42:01 Brad And you go, I don't care about the old requirements.
42:03 Brad I'm starting fresh with my requirements.
42:05 Brad And that gives you a ton of freedom.
42:08 Brad If you if you can do that.
42:10 Scott Yeah,
42:11 Scott I feel like that my apprehension there is I've never seen a business,
42:16 Scott I've yet to come across a business that is willing to invest the time and money to declare bankruptcy.
42:23 Brad Yeah,
42:23 Brad it's
42:24 Scott There's
42:24 Brad hard.
42:24 Scott some sort of six to 12 month time horizon,
42:28 Scott which is all that they can really imagine.
42:31 Brad Yeah,
42:31 Scott Maybe it's because they operate in fiscal years and they're like, I can't invest multiple fiscal years in this thing.
42:38 Brad exactly.
42:39 Brad Exactly.
42:42 Scott All right. Well,
42:43 Brad All
42:43 Scott that's all I got for
42:43 Brad right, now. man.
42:43 Scott It didn't seem like we got anywhere today.
42:45 Brad We didn't get anywhere.
42:46 Brad Well, let's declare bankruptcy on this episode of this podcast.
42:51 Scott All right.
42:52 Brad That's the answer.
42:53 Brad All right.
42:53 Brad Have a good one. I'll talk to you next week.
42:56 Brad This episode of How Many CTOs Does It Take Podcast was edited by Saffron Epigal. Our theme song was AI Generated by Brad.
43:04 Brad Please follow us on all the social media platforms at at HowManyCTOsPod.
43:09 Brad You can find us on YouTube,
43:12 Brad LinkedIn,
43:12 Brad Instagram,
43:13 Brad Facebook,
43:13 Brad Threads, X,
43:14 Brad TikTok,
43:15 Brad and Patreon or at our website at HowManyCTOsPod.com.
43:20 Brad And remember to rate, review,
43:23 Brad and subscribe on the podcast platform of your choice.