The Ball of Mud - Tackling Software Monoliths and Decomposing Tech Debt: Strategies and Challenges
- 00:00 Introduction and Podcast Setup
- 00:47 Key Challenges in Engineering
- 01:40 Decomposing the Monolith
- 04:10 Strategies for Incremental Improvement
- 06:12 Real-World Examples and Challenges
- 18:31 Measuring Engineering Productivity
- 21:54 The Challenge of Consistent Processes
- 23:25 Estimation Accuracy as a Key Metric
- 25:58 The Complexity of Measuring Value
- 30:44 The Financial Argument for Maintenance
- 36:41 Declaring Technical Debt Bankruptcy
- 42:41 Concluding Thoughts and Outro
Resources:
- How Many CTOs Pod: https://howmanyctospod.com
- Scott Porad: https://www.linkedin.com/in/scottporad/
- Brad Hefta-Gaub: https://www.linkedin.com/in/bradheftagaub/
Scott: Maybe actually we could come up with an argument or a narrative or something to convince our business stakeholders actually that is the best thing to do is just invest X percent of your dollars every year in building putting new parts on the boat.
Brad: I'm Brad Hefta-Gaub,
that's Scott Porat, and this is the How Many CTOs Does It Take podcast.
Scott: Don't forget to subscribe,
rate,
and review.
Brad: Hit that smash that subscribe button.
Scott: Smash it.
Brad: Uh,
so you just mentioned to me that you had a topic that you wanted to talk about. What was your topic that you feel like you want to attack today?
Scott: I want to not just today for the rest of my life. There's actually three things in my mind where if somebody can solve these three things and these three things are the following.
Measuring engineering productivity.
Brad: Yep.
Scott: measuring the impact of tech debt.
Brad: Mm-hmm.
So that you can make a case one way or the other about addressing it?
Scott: Sure, or just, yeah,
exactly.
Brad: Just measuring.
Scott: Quantitating the impact of it.
Brad: Yep.
Scott: Quantitating the impact of it.
And I've developed some very expensive ways to very labor-intensive ways to do that,
but developing some cheaper ways to do that.
The topic for today is decomposing a monolith, and I feel like we could talk to every...
engineering leader in the history of the world and try to get to the bottom of how to do this and let me set the scene for you I think you've probably been in this scene before you not only have seen this play before you've probably been in this play generally speaking web application web back end of a web application
number of years old in particular the one I'm dealing with is
a decade old although i've dealt with ones that have been five years old eight years old etc using some kind of framework you know could be a node kind of thing could be a you know django i happen to be working on a rails thing right now but it could apply to dot net you know asp whatever
Brad: Yeah, totally.
Scott: they're on these days it could apply to anything you know i've we've got about 100 engineers working on
A monolith, when we say monolith, we're talking about a back-end, a giant back-end service that does many things,
Brad: Mm-hmm.
Scott: that has been adapted and evolved over the years as a business has grown and changed.
And it has a whole host of well-intentioned yet poor technical decisions and just plain old poor technical decisions,
you know, has reached the point where...
engineers say hey it just takes us twice as long to do everything because
Brad: Yeah.
Scott: there is so much tech that there's so many places where there are if else statements because somebody wanted to hack something in or you have unintended consequences where you change something here and it breaks something over there and no one knows why etc etc etc
And so, you know, the general strategy that I've seen is,
hey,
company,
well, what are the two strategies? Strategy one is, well, we're going to write it from scratch. We're going to rewrite this thing.
Brad: Second system.
Scott: Second system could work, but it's really hard because you've got a decade of requirements in there that nobody really understands.
Brad: Right.
Nobody even remembers them.
Scott: None of the people work here anymore,
right?
So it takes a really, really long time to do that.
highly risky etc etc the classic Netscape 4 problem right they
Brad: Yep.
Scott: took you know it's kind of a joke but they took like two years to write Netscape 4 and what Internet Explorer came along and crushed them or something like that Internet Explorer is dead in fact I use Microsoft Edge for the first time last night that's beside the point
the other strategy is well we're going to incrementally rewrite this
And we're going to somehow create some separation of concerns so that we don't have such a ball of mud,
right?
The classic ball of mud paper,
spaghetti code.
And, you know, people try to do that either by enforcing boundaries within the monolith or by separating into different services or.
components that can be imported from things like there's a different bunch of different strategies but they ultimately try to reduce the complexity by reducing by creating some boundaries and
Brad: Mm-hmm.
Scott: it seems like a slog and there's one other thing that i've seen that's common in these systems there's usually at least one but usually a few kind of
I'll call them god objects or octopus objects right
Brad: Mm-hmm.
Scott: they have their tentacles into everything it
Brad: Yep.
Scott: could be the user object or if you have a transaction system it could be the order object like there's a thing which like everything touches and
Brad: Yep.
Scott: that tends to actually be the real bottleneck because nobody can actually figure out how to that's the obvious first piece to take out oh let's extract the order
Brad: Mm
Scott: from
Brad: hmm.
Scott: the system so
Brad: Mm hmm.
Scott: that it's really clean but that's actually the most obvious thing to get out or the user is the most obvious thing to get out those then become the hardest thing to get out because everything touches it
Brad: Yeah,
Scott: so you
Brad: right.
Scott: actually have to take out all the other stuff first and just leave it there but what then happens is you're just peeling off little parts that don't really make a difference okay so i peeled off the address surface well whoop-dee-doo it like nobody ever touches the address surface so it's like you know and so people start to peel off the little parts around the edge and then what you get is you still have the monolith it's slightly smaller but now you got new code in this new system that's nice and fresh and clean
But you still got the monolith and
Brad: Yeah.
Scott: you can, unless your organization has a ton of resources and a ton of discipline.
You put some effort into chiseling some stuff off, but you never really make a ton of progress. You just it's a grind.
I'm on my third company like this,
Brad: Yeah,
Scott: by the way.
Brad: exactly.
Well,
I okay,
so first of all, it's funny because you started by saying or you kind of mentioned to me recently, you said, it's all about decomposing the monolith, right?
Scott: Yeah.
Brad: And that's the phraseology you use and the example you're giving right now of the ball of mud is that it's a monolith ball of mud and it's got like a
like a god object or two in it and and so that's you know the scenario that you've just described is very is is kind of the the state that you're in you start your starting state um
Scott: That's the state.
Brad: but can i i guess i want to double check something or challenge something like i've i've been in that same play where it wasn't a monolith but it was
a bunch of microservices but it was the same ball of mud and it was the same there was a god object or multiple god objects that were over you know they were the areas of concern weren't well defined or and so i guess is the problem actually the monolith or is the problem the 10 years worth of
poorly specified user require poorly specified business requirements that have basically been you know glued and pasted and baling wired and and you know bubble gummed together whether it was a monolith or not
Scott: Yeah,
it isn't necessarily that it's a monolith.
I think the way I would say it is you have a system that is resistant to change.
and it's resistant to change because it has many
Brad: international needs
Scott: many yeah many areas that are tightly coupled um
Brad: yeah
Scott: so even if you just want to go do an ordinary feature oh and and the footprint of the system whether it's a monolith or many services or whatever is large so if you want if you want to go oh and typically
you got a lot of new people right so
Brad: Yeah, sure.
Scott: because any organization that's 10 years old has a lot of new people right so if i'm an engineer and you want me to go make have a simple feature right hey display the user's birth date right well why is it resistant to change first i have to do a ton of investigation just to figure out how to do it or i don't and it maybe has unintended consequences or i do and it still hasn't
I don't think that consequences
Brad: Yeah,
Scott: because I didn't even know about that service over there.
Brad: right,
Scott: Nobody mentioned that to me, right?
Brad: right,
Scott: And so somebody comes along and says,
why is it going to take two months to display? We have a database here which has a birth date in it and I just want to put a birth date on the user's profile.
And why is that going to take two months exactly?
You
Brad: right.
Scott: know?
And it's because, well, there's these 74 services in between we got to change.
I'll give you actually a real,
real concrete example of this one.
I remember from my time at Rover,
we wanted to change the number of days in the future that somebody could schedule.
a dog sit and I thought well that's going to be easy there's just a constant somewhere that's right now it's 120 and we just need to make it say 400 or whatever the number we need to be is right and then the estimate comes back like at two months and I sat down with the engineer and I was like okay he was a he was a really good engineer so I trusted him but I was like I just need you to explain to me what why is this taking so long and he walked me through it
And the answer was, I remember this number specifically, there were 21 places in the system that depended on this thing.
and a lot of them were kind of hard some of them were hard coded and some of them looked at the constant and so on and so forth and it just wasn't as simple as you thought right because
Brad: Yeah, yeah
Scott: of all this time so to me it's the interdependencies or the tight coupling which is the source of the issue and you get tight coupling from poor design intentional poor design or unintentional poor design what I mean by intentional is just like
like somebody did it and they weren't doing a good job or unintentional as they did it and they made a mistake we all make mistakes right
Brad: Yes,
Scott: you
Brad: really.
Scott: know that we come back and look at and look oh that was a bad way to design that right and that's ultimately I think where it comes from and then I think it gets compounded in times of stress you know where a business is asking us to do something quickly and we just think well I'll just jam this in here
And it really doesn't seem like that big of a deal, but, you know,
10 years, everybody does one of those a year and you've got 100 years on your team and you do it for a decade.
That's a thousand of those things.
Brad: Yeah, totally.
Well,
so the question at hand is, how do you solve this?
Like, how do you make it so that you're 10 years into this thing?
And what do you do to make it so that you're not every single small request is a month long project?
Scott: Yeah.
Brad: Okay, so let's, is there,
do we have examples of teams that have done it really well?
Like, or is this just everyone always struggles with this?
Scott: Well, that is a great question. And if we had examples of teams, anybody who is listening to this podcast knows of anybody in
Brad: who doesn't
Scott: the world
Brad: get into this who after
Scott: done
Brad: 10
Scott: well,
Brad: years doesn't get into this totally
Scott: well, okay,
there's two questions, people who don't get into this, but people who've gotten out of it, there's two different things. How do you, okay,
so
The fastest way out of a hole is to stop digging.
Brad: Yep.
Scott: So who's figured out how to stop digging,
right,
and at least get out of it? And that's a different question than how do you never get into it in the first place.
Brad: Yeah,
agreed.
Agreed.
Scott: Now what I've also seen well-intentioned teams draw figuratively speaking into diagrams,
but they want the system to look like.
Brad: Mm
Scott: right
Brad: hmm.
Scott: well we're going to take this order object this god on you we're going to extract it out we're going to extract the user out right that sounds nice but the devil's in the details right you have to build some sort of bridge i guess
Brad: Yeah.
And I think that's usually the right that tends to be the technique that most teams attempt.
Right. Is they pick the.
You know, you did a great job of teeing this up in the introduction where it's like you kind of like pick the the object that's the least constrained object and you,
you know, slice it and cleanly put it to the side,
you know, and you never get to the place where.
The over-dependent object is the one you're attacking.
Scott: That's right, because the most constrained object is the one that people use the most.
There's one other thing I want to throw into this sauce,
a little seasoning in the soup.
I could be wrong about this because you mentioned the situation with having it be.
you know a service-based architecture I've seen this problem when you're using an ORM right it's so easy to just go connect this object to that object with an ORM and
Brad: Yeah.
Scott: it's all behind the scenes and then your whole system uses an ORM but you want to extract one piece of it to say not use an ORM
Brad: Well,
I've seen it in another example I've seen it is somebody starts out where they're like, oh, well,
we're going to like not get into this mess. We're going to keep ourselves from getting into this mess and we're going to we've got this great idea.
We're going to have a I don't know, I'll pick some example. We're going to have a.
message bus and we're
Scott: Mm-hmm.
Brad: going to keep everything all the microservices cleanly separated and they just use this message bus to talk to each other right
But then next thing you know, it's like the message bus is overly type specific and the message bus becomes the thing that's over constrained and you,
oh,
oopsie, I added a new message type,
but I didn't update all the other components to know that this new parameter exists on this message type and boom,
Scott: Yeah.
Brad: crashes the message bus or all the components that didn't know about it start crashing,
right?
Scott: You want to hear a doozy?
Brad: Sure.
Scott: Organization I heard of. They were using an ORM.
They were using the on change event from the ORM to spawn messages into the message bus that then other objects or whatever would listen to other parts of the system would listen to.
So including the analytics system would use those events to.
collect you know telemetry about the app and um so then it's even worse because you're the putting of messages on the bus oh and so by tapping into the unchange event it's getting the raw data and putting it on the message bus so by by it's so now people don't actually even really know
when they're breaking if I have a message and I do it and then something breaks oh I changed that message that must have been me right
But when you're just changing an object and the ORM is saving it and then something is breaking,
nobody knows. It's
Brad: Right.
Scott: like, well, why is this breaking all of a sudden?
Brad: Right.
We're we're our this is a classic tar baby.
Right.
Like we just it starts getting stickier and we start getting more stuck to it. Right. So it's not even just, oh,
hey,
you know, the answer is decompose your monolith because if you use microservices, you'll prevent yourself.
yourself from getting into this mess nope nope you can you can still be in this mess with microservices oh
Scott: Mm
Brad: um
Scott: -hmm.
Brad: let's use like strategies of of um you know decoupling like you know message buses and things like that oh well no you can still get yourself into this mess and so here we are we're still stuck with this problem i the more i think about it um what's the what's the what's the uh um
The thought experiment of the ship that I replace one piece at a time, if I replace every piece of the ship one piece at a time, do I still have the same ship at the end? It's
Scott: Mm
Brad: not Prometheus, but one of the, you know, one of the Iliad, right?
Like one of the Greek or,
you know, epic poems.
The teams that I've seen that I'm not going to say they're going to hit it out of the park. I'm not saying they've solved this problem,
but the ones that I've seen manage this problem the most are the ones that essentially don't try to rewrite the whole system. Don't fall for second system effect, but they do basically rewrite pieces one at a time.
Scott: Yeah.
Brad: try to be as intentional as possible that's the only that's the only path i've seen have
Scott: Yeah.
Brad: a hope
Scott: Yeah.
So, okay.
This would be a great topic to talk to people about. Let me ask you another question.
Okay.
So team I'm working with now recently embarked on something like this. They, the commerce portion of what they're doing, you know, payments and stuff like that,
they extracted.
And one of the engineers on the team that I like, he spent four,
I talked about the octopus problem,
Brad: Yeah.
Scott: he spent four months figuring
Brad: Researching.
Scott: out where all the touch points were.
Brad: Yeah. How many legs this octopus had.
Scott: So, yeah,
that's right.
He wrote actual, he actually wrote some software to profile what was happening so he could understand where this thing.
interactive because one of the things that happened was you know there would be like the virtual sort of dead code and you wouldn't even know if you needed to do it so we spent all this time doing this they did it it succeeded it was a big huge effort they had to have he and he and three other people spent four months figuring out what to after the new service was built four people spent four months figuring out what the touch points were going to be able to fix and then 20 people spent six months doing it
This huge effort,
which gets me to item number two, item number one on my list, maybe it's item number two, the impact of measuring the impact of these things.
Brad: Yeah.
Scott: So all our business partners are saying, well,
that's great.
I now have a cleaner version of what I had before.
Exactly how did that help me out?
Brad: Exactly.
Scott: Well, it's making everything go faster. In fact, the team is already reporting that projects are going faster.
You know, but to my business partners,
stakeholders,
I mean, that doesn't mean anything to them.
They understand the words and they have some trust, but they don't feel it, right?
Brad: Right,
Scott: They don't feel it. They don't feel it like they feel money on the balance sheet.
Brad: right,
Scott: That's for sure.
Brad: right.
Scott: So how do we measure the benefits of these things? Let's say that it really does take a long time and a lot of hard, discipline, patient work.
It's the only way.
It's really hard to get a business to fund that for the long term because they're just saying, I don't understand what I'm getting for this,
given
Brad: Yeah,
right.
Scott: that we can't measure engineering productivity,
even though this is the kind of thing that is improving the productivity.
Brad: Yeah,
I mean, I think that I think that the right.
So we talked a little bit about this in a previous episode where we were talking about the idea that if you're going to measure the.
performance or the value being added by a platform team for example one of the ways to measure that is are the application teams sitting on top of that platform team are they able to move faster are they they're
Scott: Do their, you know, projects take less whatever unit of measure you use to measure productivity?
And so it's really hard to say this will definitely produce this result.
But once you've done the work, you ought to be able to have some metric of projects that touch this now go faster.
Right.
Brad: In theory,
yeah,
that's what we're hearing, you know, go to lay for sure.
Scott: Right.
Right.
Brad: What do you think about, you know, a few years ago, Google came out with the Dora metrics and
Scott: Yeah, yeah, I like the Dora metrics.
Brad: one of those metrics is cycle time,
if I recall.
Scott: Yeah.
Brad: But it called for lead time for changes, what they call the amount of time it takes to get commit to production.
Scott: Yep.
Brad: Let's take a look. What does that mean exactly?
lead time for changes how long does it take to get to go from code committed to code successfully running in production that's really not cycle time that's i mean to go from code committed to code successfully running is nothing it's really like from someone has an idea to when the code is live how
Scott: Yeah, exactly.
That's what I want.
I want time from idea to production.
That's what I want to measure.
Brad: do you think you would control for scope
I mean,
a small idea and a big idea
Scott: Sure.
Brad: are different.
Scott: Totally.
Well, okay.
I had this question for you. I actually, I remember sending this to you between our last two episodes. I was like,
how do you, I'm going to get to your question about Dora and how do you control for scope?
I'm going to get there for a second, in a second.
On a team of 100,
right?
Okay, so you're the CTO, you've been airdropped in,
you're getting to know this team.
There's 120 engineers there.
Do all of your teams.
use the same process like do they use the same metrics do they use the same whatever mechanism you use for say estimates whatever mechanisms you use for tracking progress are all the teams using the same process
Brad: I can't tell you whether they are right now,
but.
given my choice,
they would be.
Scott: Right.
So the reason I ask that question is because.
One of the things I've seen often is the sort of like organic nature of the organization or small d democratic nature of the organization is you somehow said,
yeah,
yeah, let the teams come up with their own processes.
Right.
Brad: Yeah.
Scott: And now you're sitting here at the, you know, at the CTO level and you're like,
well.
I have no idea if Team A is, I can't measure Team A's performance relative to a benchmark because they're not even following the same process as the benchmark.
So
Brad: Right.
Scott: I don't know,
like you're over there using story points, you know, and everyone else is using.
I don't know, T-shirt sizes,
right?
So now we have this problem where we don't even have the same language,
right?
Brad: That's right.
Well,
Scott: How
Brad: what's interesting is that,
so for that problem I found a way to correct for
Scott: do you correct
Brad: that.
Scott: for that?
Brad: To me, I could do 10 episodes on this.
I probably already talked about it in every one we've talked about.
To me, the key measurement is estimation accuracy.
Scott: Right.
Brad: And as a team, I don't really care how you want to estimate.
I'm just going to measure how good you are at it,
which is when did you say you were going to get something done and when did it get done,
right? And if you want to use...
planning poker or you want to make numbers up or you just want to let the boss put it on a schedule it doesn't matter everybody's the boss the team manager put it on a schedule it doesn't matter to me i mean it doesn't matter i prefer the teams i'll do it the same way but ultimately what matters is where they accurate right but usually the way our industry does it is we do despite the effort of story points trying to be units of value you
We do estimate in time.
Let's say you've got this project takes you four weeks.
And now you're going to do some work so that a large project only takes you, you know, you fix your system that's all broken.
So you do some work so that a large project only takes you two weeks.
Scott: That's fantastic.
Brad: Yeah,
it's great. You made a tremendous investment.
But now when the next project that comes along,
you're not going to call it a large project anymore.
You're going to call it a medium project,
right?
Scott: Right.
Brad: Because it only takes you two weeks.
And so...
you'd like to be able to say well look the engineering team did this work and we used to only be able to do 10 large projects in a quarter and now we can do 20 large projects in a quarter Yep. but what happens is we used to be able to do 10 large projects and we can still do 10 large projects they happen to just be bigger projects but we don't really get that signal right because we
Scott: I adjust think that.
Brad: you know it's like we don't really have a way of a unit of engineering work we need that we need to
We the what would we call that a gates now
Scott: A clock,
of course. No, I'm just kidding.
Thousand lines of code.
Well, look,
Brad: I get it. I get it. I was trying to I was trying to think of that made sense,
but it doesn't make any sense
Scott: it's the classic worst possible metric.
Brad: Maybe we should call it a brook didn't wasn't he did the mythical man month was not brook didn't he write that we should call it a brook a unit of engineering work is a brook
Scott: All right.
Well,
OK,
so I think that like, OK,
so let me just like.
Try to rewind and pull on the thread a little bit here.
So, okay.
So remember,
we start out with, we're starting with this challenge of how do we measure that the cleanup of the technical debt,
the investment that we put into making this thing better?
Why?
How do we prove that that was useful? And the answer that I think we all agree with, we agree is it should either fixate a stability.
a stability issue that we've been haunted by,
or it should make us more efficient as an engineering team.
Brad: should reduce that cycle time which
Scott: It should reduce that cycle time from idea to production.
And the only
Brad: is
Scott: way
Brad: crazy
Scott: we,
Brad: because if I have more ideas the time is longer right if you can do if you like if you can do one idea if you can do one idea a month and I can do one idea a month but you've got ten ideas and I've got two ideas
your cycle time will be longer well
Scott: Well,
if I am if I if I can get twice as many ideas done in the same amount of time that you can get an idea done,
then I get twice as much work done. Right.
Like so
Brad: what
Scott: the if same.
Brad: your if your ideas are half the size the reason i have two and you have ten is it's five times bigger
Scott: Well,
this is the thing. Right.
Then now you have to have a way of measuring the value of the idea.
Right?
Brad: Well, value is interesting because that's a subtle choice of word.
You could have chosen size of the idea.
Scott: No,
value of the idea.
I want to deliver more value to the customer more quickly.
So
Brad: that's
Scott: that's what I
Brad: dollar
Scott: would want.
Brad: it's incremental dollars per unit of time oh holy shazal is the metric incremental is it essentially i don't know if this is quite right is it revenue growth per engineer salary dollar
Scott: I mean, that's not a bad way of measuring like, is it worthwhile?
Like that would be the measurement for me,
like, should I build this software myself or should I just buy something off the shelf?
shelf right like
Brad: Right. Basically,
Scott: revenue
Brad: last year we could
Scott: amortization of however i got the software built right well
Brad: generate a million dollars of revenue with 10 people and this year we can generate $2 million of revenue with 10 people.
But it's the tricky one because there are so many different factors that go into what caused that revenue to increase.
Scott: another another one of our problems here that we have to be honest with ourselves about is
What if the ideas coming out of our product team are just dumb ideas and aren't going to drive any revenue?
Brad: that's what i was saying is that there's a lot of things that go into that it could be the dumb ideas it could be the marketing maybe marketing did a better job this year they had a better ad campaign and they drove a bunch of revenue like there's a bunch of different reasons it's really really hard to connect it to revenue or whatever metric it is
Scott: Right.
Right.
Yeah.
I mean, this is this is the fundamental. This is hard.
This is hard.
Well,
so I guess the question here is, given that this is definitely going to be hard,
what are the areas that we OK,
so I want I want to throw this out. Right. I asked you the probative question of are all your teams even following the same process?
Right.
Brad: Right, yeah,
Scott: And
Brad: yeah.
Scott: I and I think that that's like, I mean.
Not only do you, what if you have teams that not only are not following the same process,
but they're actually following a bad process?
Like they're actually like not doing something that if you were to set a break point and look at what they're doing,
you and every other like reasonably experienced engineering leader would say, what are you doing? That's not the way to write software.
Right.
How do we solve for that?
Are you?
Brad: it's just freaking i don't know if we're shaving a yak or pulling the thread on a sweater but we're doing one of them right now i mean it's just all crazy well i think this is yeah i mean what do you do what do you do about that i don't know either um
it's I mean it's all it's all intertwined I guess let's just can I tweak the subject for
Scott: Yeah,
Brad: a second go
Scott: totally,
totally.
Brad: back to where we started a little bit let's just we only reason we went down this productivity path was when you said well the only really way to do this is to just sort of chisel work away at it for a long period of time like I don't think he said Prometheus's boat I don't know whose boat that is he doesn't sound like a sailor seems like a sailing kind of situation but
you don't make it like you know the only way is to be like a sailboat and just carefully just go through the whole thing and replace every part one part at a time before you know it you got a new boat right and the only reason i went down the productivity route there was to say well if you're going to take that strategy
it's going to be it's going to be tough for a business to digest endless investment for no visible return right
But maybe, this is where I wanted to change the subject to,
maybe actually we could come up with an argument or a narrative or something to convince our business stakeholders actually that is the best thing to do is just invest X percent of your dollars every year in building, putting new parts in the boat because that's just the only way to not have your boat fall apart in the middle of the ocean.
Right.
Scott: I mean, that would be true for hard infrastructure,
right?
Like if you own a factory,
you can't not put maintenance into it.
Brad: So, yeah, I mean, it's interesting. A factory is very different in the sense of, I mean, I think one of the things that makes.
software complicated is it's under constant change the
Scott: Yeah.
Brad: factory is not under nearly as much change maybe it is maybe it's getting used i don't know i went through the ballard locks this week they close it for either two or four weeks a year i think two weeks a year for maintenance so
Scott: For maintenance.
Brad: two weeks out of 52 and it's getting used all day every day and it's over 100 years old
Scott: Yeah.
Brad: So let's think about that for a second.
Software typically gets capitalized on a three-year schedule.
Does that mean every year we should be spending a third of our resources to build this to maintain it?
Scott: I mean, isn't that the theory behind what depreciation and capitalization is all about?
Like you're basically saying like the lifetime of this thing is three years.
Brad: but let's think about that for a second if we've got 10 engineers let's just make it nine engineers to make our math easier nine engineers and
Scott: Three of them are constantly working on fixing a third of the system.
Brad: yeah but six of them are making new system right so it actually starts compounding so do you end up in some kind of situation where like after
Five years like 75% of your team is actually working on replacing parts of the boat
Scott: Well,
again,
like this is the this is kind of like what I was trying to say is a it's got nothing to do with what the code architecture was,
whether it was a monolith or microservice.
It has much more to do with the 10 years of product requirements.
Right.
Brad: Well, in deferred maintenance,
Scott: And
Brad: it sounds like.
Scott: years ago I worked with a developer and his his like claim to fame was he always wanted to go look at his Git.
um history and have more deleted lines of code than added lines of code
Brad: Mm
Scott: like-hmm. that was his measure
Brad: Yeah.
Scott: of that was that was what he said like i am a great developer because i have always deleted more code than i've added right so it was funny because like anybody that ever like you know added some module or something like that this guy was the first person to be like i'll i'll take the ticket to delete that module that we decided we don't need anymore
more because you know he would just be he would be deleting code that other people added that was you know easy you know he was the guy that was kept doing appendectomies on on the code you know and um and and it was funny it was like it was comical and also he's not wrong like if you're the person that can if you're if you're actively getting rid of pieces of the boat that we're no longer using
That's very valuable to the organization.
Brad: Right. I mean, effectively,
if you think of your code as your plant and equipment and there's less you have to maintain,
because maybe the strategy here is actually more of a financial argument to get your business people and your finance people to buy into just not to be looking for an ROI,
but to just be like, well, of course, I mean, we just have to maintain our system and that that should be something like a third of your resources.
for any given level of system let's just think of a team with three people let's just think of one person in year one that person writes one year of new code in
Scott: Mm-hmm.
Brad: year two they write a second year of new code in year three they write a third year of new code now in year four they write they have to refactor the first year new code so they didn't write any new code
So that means over a six year, and that happens again for three years,
so over a six year period, it's been 50% of their time has been new code versus.
So is that it? Is it 50%?
Scott: Well,
I think in realistically what actually happens,
right,
is on year one, you write new code,
year two, you write new code, year three, you write new code,
year four, you delete half of the code from year one, you rewrite half the code from year one, and you add another half a year code.
Brad: Got
Scott: Right.
Brad: you. So at that point, at that point, what have you got? You've got three
Scott: Now you've got now you've got three and
Brad: and
Scott: a half
Brad: a half.
Scott: years of fresh code.
Brad: Well,
but half of the half of those three and a half, one half a year is tech debt.
Scott: Exactly.
Brad: is deferred one half year is deferred maintenance and
Scott: Right.
Brad: so let's say you do that again the next year now you've got four years with a year of tech debt and let's say you do that again you've got four
Scott: Four and a half years with it.
Brad: and a half years
Scott: A
Brad: a
Scott: year
Brad: year
Scott: and a half.
Brad: half and so you know one more year you've got five years of two years of tech debt one more year you got five so you're starting to catch up right
But it's probably something like 50%. Now,
Scott: Yeah.
Brad: how do we convince our business partners to invest 50% of their precious engineering resources in maintenance?
Scott: Yeah,
that's a big ask.
That's a pretty big ask.
Brad: It is a big ask.
it's kind of the thing they need right and then if you're in a hole if we're in that situation where you've got the five years of tech debt because you've been doing it that way now you not only have to ask for the 50 to keep up you gotta you have more to dig out it's
Scott: Yeah. Well,
Brad: kind of insane yeah
Scott: and that's why what second system effect is really about is declaring bankruptcy,
right?
Brad: Yeah.
Scott: Like I'm just I'm just picking up and moving away from the factory that's got all the broken down equipment in it.
Brad: It's just like when people tear down an old house, take their car to the junkyard instead of fixing it.
Scott: Yeah, I'm notoriously bad.
I, you know, I try really hard not to ever let this happen at work, but my personal email inbox is just like, you know.
gets gets overloaded i'm not i'm really horrible at zero inbox right and so every once in a while i just declare inbox bankruptcy i just like move all of the entire inbox into a folder called bankrupt bankrupt inbox and now i got a zero inbox nice
Brad: Well, that's interesting.
I am the opposite of you. I'm an inbox zero guy. I was an inbox zero before we got on this call today.
Scott: so how much of your day is spent
keeping your inbox at zero right
Brad: Not much.
Scott: well that's good then maybe that's a is there a clue here what's
Brad: Yeah.
First clue two,
Scott: don't
Brad: don't send emails because
Scott: write when new code
Brad: you send emails,
people
Scott: when you
Brad: respond
Scott: write new code
Brad: to them.
Scott: when you write new code you have code to maintain that's
Brad: Don't respond to emails.
Don't respond to emails.
Because when you respond to emails,
people think it's a way to communicate with you.
And if you don't respond, they'll find other ways.
I'm just joking.
I would say the answer is I don't check email that often.
I don't stay glued to it all day.
And I have found some systems, I would be happy to talk about this at another time,
but I've found some systems for processing quickly through a lot of email.
Scott: Here's my point, by the way. I wasn't actually trying to raise the topic of inbox email inboxes.
Brad: Yeah,
Scott: I'm saying, is there a corollary there to the developer problem that we're facing?
Brad: right.
Scott: Right. Like I just saw this really interesting meme on the Internet. It was like a ADHD tip.
Instead of putting it down,
put it away.
Right.
Like people that have ADHD is super hard.
Like I put it down like my now my office is completely strewn with stuff.
Right.
No, no. Instead of putting it down, put it away.
Right.
So, again,
Brad: Yeah
Scott: like I'm not trying to simplify the notion of strategies for dealing with ADHD,
but I think there's an interesting corollary here between our problem of technical debt.
Brad: Hmm?
Scott: Right.
It's not dissimilar from the problem of inbox debt.
Brad: No.
But I'd say the difference is you can still receive email when you declare bankruptcy versus you don't have to go reconstruct all those messages and all those communications that somebody sent to you.
You know, it seems to him.
Scott: Yeah.
Brad: Well, to me, this is the this is the most interesting topic of our time.
Scott: Right.
Brad: If we can solve this, we can solve climate change.
That's just what I'm saying,
because it's the vexing problem that I see every organization getting into and then they got to get out of it because their system. And that's what I'm seeing now in my third organization in a row.
Scott: actually really the fourth in a row it gets to this point where the resistance the system's resistance to change becomes commercially impractical for the company right essentially
Brad: Right.
Scott: they all essentially they are becoming tech debt bankrupt and they're they're hiring us to get them out of bankruptcy to do a restructuring as one would might say right and the question is how do you do it
Right. And I hear you saying here the only way to do it is to patiently go through bit by bit.
And
Brad: Well,
Scott: then I
Brad: and I think and I think and the other thing I don't know if I specify I didn't put enough emphasis on this particular syllable,
but I think that like being willing to cut things is
Scott: Yeah.
Brad: a really critical part of that.
Scott: Yeah.
People have a really hard time with that.
Brad: Especially
Scott: Right.
Brad: when they don't know what you're asking them to cut.
Right.
Scott: Yo, I would actually say there's even another hard thing in there.
Well,
while you're in there problem.
Brad: Yeah.
Right.
Scott: Oh,
Brad: It's
Scott: well,
Brad: good.
Scott: if you're going to be if you're going to be refactoring that thing,
could we just make this change to it?
Brad: Right.
Right.
Exactly.
Exactly.
Scott: That just makes it go backwards.
Well,
this is we have got to find we got to get some guests on here.
Brad: Yeah, they that have that have tackled this problem.
Scott: either succeeded or failed and help us figure out how to do it i
Brad: I love it. I love it.
Scott: will say one thing a company that i was at haven't been there for several years but they're about six or 12 months ago somewhere that company the cto did call me i was talking to him the new guy and they had been trying to refactor one of their mobile apps
and he called to tell me they were done and it took us we had started together and it took four years to
Brad: Wow.
Scott: do a to do a mobile app so
Brad: Wow.
Yeah.
Scott: it's it takes time yeah
Brad: Yeah,
it's hard.
It's way easier if you basically start with a fresh slate of requirements.
And you go, I don't care about the old requirements.
I'm starting fresh with my requirements.
And that gives you a ton of freedom.
If you if you can do that.
Scott: Yeah,
I feel like that my apprehension there is I've never seen a business,
I've yet to come across a business that is willing to invest the time and money to declare bankruptcy.
Brad: Yeah,
it's
Scott: There's
Brad: hard.
Scott: some sort of six to 12 month time horizon,
which is all that they can really imagine.
Brad: Yeah,
Scott: Maybe it's because they operate in fiscal years and they're like, I can't invest multiple fiscal years in this thing.
Brad: exactly.
Exactly.
Scott: All right. Well,
Brad: All
Scott: that's all I got for
Brad: right, now. man.
Scott: It didn't seem like we got anywhere today.
Brad: We didn't get anywhere.
Well, let's declare bankruptcy on this episode of this podcast.
Scott: All right.
Brad: That's the answer.
All right.
Have a good one. I'll talk to you next week.
This episode of How Many CTOs Does It Take Podcast was edited by Saffron Epigal. Our theme song was AI Generated by Brad.
Please follow us on all the social media platforms at at HowManyCTOsPod.
You can find us on YouTube,
LinkedIn,
Instagram,
Facebook,
Threads, X,
TikTok,
and Patreon or at our website at HowManyCTOsPod.com.
And remember to rate, review,
and subscribe on the podcast platform of your choice.