Transcript: Navigating Technical Debt and Code Quality in Agile Teams
0:00 Brad You had the culture,
0:01 Brad you had the talent,
0:02 Brad you had the intent and effort and everything and it still happened.
0:25 Scott Welcome to another episode of How Many CTOs.
0:27 Brad Thanks for having me,
0:28 Brad Brad.
0:28 Scott I love talking with you. Okay,
0:30 Scott so I've been dying to talk to you about this incident that just happened with this.
0:35 Brad Don't forget to subscribe,
0:37 Brad rate,
0:37 Brad and review all
0:38 Scott Yeah, smash that subscribe button.
0:40 Brad your favorite podcast platforms.
0:42 Scott Yeah,
0:43 Scott on all of them.
0:44 Scott Like we're on all of them. And we're on all the socials.
0:48 Scott Okay, so you know I'm working on this small team. I'm in the trenches.
0:51 Scott which is we're like seven developers it's super fun i'm actually writing code myself and this team has got a lot of like they take they take their code base very seriously and they like code reviews and detailed like code stewards and and this past sprint we had a developer who's going to do a very simple feature
1:14 Scott It turned out what was like there,
1:15 Scott you know, the feature of something like add an admin panel to the client app that only admins should be able to go to.
1:22 Scott And when they started working on this,
1:24 Scott they stumbled on the fact that the code that they were going to have to like integrate with this new feature had dependencies on completely random pieces of code that had nothing to do with their features.
1:40 Scott So this is basically a basic breakdown in, like, areas of concern of the code,
1:44 Scott like bad encapsulation or bad areas of concern.
1:47 Scott Like, it was multiple objects were over constrained and oversubscribed.
1:55 Scott And I was struck by the fact that, like, here it is a team that I'm watching on a daily basis in the trenches with them.
2:04 Scott And they're all really good at their jobs and they're all really intentional and yet completely whiffed on this one.
2:14 Scott They let this classic example of bad anti-patterns of technical debt creep in.
2:23 Scott And I'm struck by like you're sitting there at the, you know, at the 150 employee level,
2:30 Scott right? You're like.
2:31 Scott you know, you're like three layers of management away.
2:36 Scott And how do we trust? How do we verify?
2:40 Scott How do we know that this isn't just happening rampantly across the code base all the time? And what do we do about it?
2:49 Brad Okay, here's my first two reactions.
2:54 Brad They're not exactly on the nose of your question.
2:57 Brad I'll get to that.
2:59 Brad My first thought is like agile principles,
3:03 Brad scrum, agile,
3:04 Brad that kind of approach,
3:05 Brad it doesn't see refactoring as a bug, it sees it as a feature,
3:10 Brad right?
3:11 Scott Yeah.
3:12 Brad Because it allows you, in the balance, in the spectrum or the balance of over-engineering and not building the thing that you want.
3:21 Brad making progress, but then having to go back and refactor because you didn't over-engineer, you made near-term decisions,
3:31 Brad it embraces that refactoring side because that
3:37 Brad philosophy says look we want to have working software that meets customer needs and we spend so much time trying to think of every eventuality that first of all we build stuff we're not going to need second of all it takes us forever to get done and we don't actually get working software in people's hands so that was my first reaction which is like well I'm not sure that refactoring
3:57 Brad is a bad especially at an early stage like yours when you're trying to get fast
4:01 Scott Totally.
4:01 Brad to get your thing launched i'm not sure that's terrible it's it's the feature if you made some other decision fast go
4:08 Scott Okay.
4:08 Brad ahead
4:08 Scott So hold on a second. Let me just let me just sort of pause and echo back and sort of like validate and agree with where where I think I think you're spot on about that. Right.
4:18 Scott Like the idea that the login controller.
4:22 Scott had some functionality that missed functionality and now we're adding this new concept of like rules or something like that part of the refactoring totally agree with you agile it's all it's it's exactly why we do agile so 100 agree this particular example though is the type of thing that some of my senior engineers are like no no we we prevent this kind of stuff from happening
4:49 Scott happening because we do code reviews and we're really uh we're really anal about the code reviews and we don't let bad code get get written in the first place right and like a simple example like this login controller was polluted with details about audio like it had like details about like audio playback like it never no like no way should have been in the login controller right
5:14 Scott So, yeah,
5:16 Scott so agreed on your point about refactoring,
5:18 Brad Yeah.
5:18 Scott and yet mistakes were made.
5:21 Brad Mistakes were made. Well, we are human.
5:25 Brad So I guess the question is, then you're asking the question is, okay, you're asking a different question, which is.
5:30 Brad The code review process apparently is not a sufficient mechanism to prevent those mistakes from being made.
5:36 Scott Yeah,
5:37 Scott totally.
5:38 Brad And so it'd be very interesting.
5:40 Brad I would be curious about something.
5:41 Brad I would want to go I want to so if I was having a retrospective and saying,
5:45 Brad well,
5:45 Brad how did this happen?
5:46 Brad How do we prevent it from happening again?
5:49 Brad And,
5:49 Brad you know, a good open hearted like people were not trying to just assign blame.
5:54 Brad I would.
5:55 Brad ask the question well what was it about the code review process that allowed this to happen and i can think of some potential reasons right one is the human error the code review was not done thoroughly right
6:08 Scott Mm hmm.
6:08 Brad or human error the code review was done thoroughly but somebody missed it right
6:15 Scott Yeah.
6:15 Brad um i could think of another piece which feeds human error
6:21 Brad It was a whatever happened was a the costume is it was a big code review.
6:26 Brad It was a lot of code.
6:27 Scott Yeah, totally,
6:28 Scott totally, totally. Yeah,
6:29 Brad it was.
6:29 Scott that's actually that's one of the things that's been that has shown up in the retros like,
6:34 Scott oh, large, you know, you know, you know, more than 100 lines of code in your PR and that's like way harder to deal with. If it's thousands of lines of code, you're definitely going to put a great tax on top of your.
6:47 Scott the code reviewers and so keep them small and stuff like yeah so so
6:51 Brad I
6:51 Scott for
6:52 Brad will submit that it is impossible to do a good code review with more than 300 lines of code.
7:01 Scott sure okay
7:02 Brad Impossible.
7:03 Brad Not, not humanly possible.
7:04 Scott so hold on a second hold on a second scott i love the direction you're going and this is all this is all 100 aligned with i think the how you fix it at the six person team
7:15 Brad Yes. How do you fix it at the big team?
7:16 Scott Yeah,
7:17 Scott that's really the question here.
7:19 Brad I'm not saying I've done this, but here could be a concrete example.
7:22 Brad There is a system called danger,
7:24 Brad danger rules.
7:25 Brad There are rules you can put in your CI.
7:28 Brad It's a framework for adding rules and checks and things like that to your CI pipeline.
7:32 Brad I really like it. I think the CI pipeline is actually a way that you can really enforce a lot of quality.
7:39 Brad So just as a concrete example.
7:41 Brad One could make a rule that it will reject APR that has more than 300 lines of code.
7:48 Brad Totally. That's one thing.
7:49 Scott Right.
7:50 Scott But here's my question for you, though.
7:52 Scott You've been air dropped in.
7:54 Scott You've got 10, 15 years of culture and code.
8:01 Scott How do you know?
8:05 Brad I have two thoughts. The first is where I always go is trying to quantify something.
8:11 Brad That's where I always go because I don't know.
8:12 Brad I just know anecdotally I came in here and it's not just this job. It's the last job and the one before that.
8:19 Brad My last three jobs have all been about fixing engineering teams that have had tremendous tech debt.
8:26 Brad and where things weren't getting done and people were unhappy and they were good engineers but it was just like it was just a situations in crisis my last three jobs has been kind of crazy and so my inclination is to always and you hear anecdotally everyone's like it's so bad it's so bad so that so my inclination is to try to go to quantify now on the one hand I have a senior individual on our team right now who is trying to go through every
8:51 Brad package of our component of our code that we've developed a rubric for the level of tech debt and he's trying to evaluate with the teams how is this now this doesn't prevent it but at least it gives us a way of starting to keep score on the other hand I just read an article I saw an article I could dig it up
9:12 Brad I can put it in the show notes because
9:14 Scott Yes, please put it in the show notes.
9:15 Brad some engineers and researchers at Google tested like 130 different.
9:21 Brad indicators of tech debt and they could find nothing that correlated to tech debt
9:26 Scott Right
9:26 Brad so maybe i'm going down they were trying to objectively measure tech debt maybe i'm going down a fool's errand to try and measure this but the only way i know how is to just get a number and say like look when you take all this stuff and yet you take all these packages and we scored them all these different ways and we edited them all up and we did all this stuff and like here's our number and we're going to do this once a quarter
9:47 Brad And hopefully it's going to go down, right?
9:49 Scott Yeah.
9:50 Scott What I was struck by in the real time experience of this real example of tech net really slowing down a project and everybody looking at it and going,
10:01 Scott oh, yeah, mistakes were made like it was sort of like it was clearly objective,
10:10 Scott not subjective that.
10:12 Scott The task was harder because of existing code.
10:18 Scott In examining the existing code,
10:21 Scott every single engineer that looked at it said,
10:23 Scott oh, yeah,
10:24 Scott that's wrong.
10:25 Brad It's so crazy. You had the culture,
10:28 Brad you had the talent,
10:29 Brad you had the intent and effort and everything,
10:32 Brad and it still happened.
10:33 Scott And it still happened. Now, here's the other.
10:38 Scott And honestly,
10:39 Scott infuriating thing to me, because this is what really got my hackles up.
10:44 Scott The team constantly adds tech debt to a backlog and says, we'd like to do this. We'd like to do this. We'd like to do this.
10:54 Scott This one wasn't on that.
10:56 Scott When I sort of try to look at the list of what they have versus what.
11:00 Scott What this one was,
11:01 Scott I actually think the stuff they came up with was so much less important than this one.
11:08 Brad What were the differences?
11:09 Brad What were the kinds of things they were coming up with versus this one?
11:13 Brad This one was a tight coupling.
11:16 Scott This was a tight coupling.
11:17 Scott This was basically tight. So this was a coding anti-pattern of tight coupling,
11:22 Scott and it definitely fell into the...
11:26 Scott uh it wasn't broken until it was like it wasn't a problem until it was right it was a ain't broke don't fix it type thing if anybody had come across it and said hey this is weird the login controller knows about the audio right i
11:44 Brad Yeah.
11:44 Scott could i could definitely picture some i could picture myself being like okay that sounds like a legit problem problem
11:52 Scott Let's work on that when it comes around kind of thing,
11:57 Brad Let's
11:57 Scott right?
11:57 Brad add it to the backlog.
11:58 Scott Let's add it to the backlog.
12:00 Scott The kind of stuff that is more like is in there, you know, honestly is stuff that's like go back and add type doc to old files that don't have type doc.
12:14 Scott so that one i could imagine somebody a new developer coming along and going oh there's this whole new set of code i don't understand if there was only type if there was documentation on it would be better you know but i don't know so so anyway so in the spirit of like a nice bite-sized challenge here i like your idea of like can we use tools to enforce this i like that idea
12:38 Brad Well, you know, I know a team that has written automation into their CI pipeline to prevent tight coupling.
12:48 Scott That's interesting.
12:49 Brad Now, specifically the technology they're using.
12:54 Brad Django and Python for
12:56 Scott Mm
12:56 Brad their back
12:56 Scott -hmm.
12:57 Brad -end services and in that framework you have to explicitly declare when you're essentially import you have to declare other packages right so that gives them a mechanism to say oh you are you are importing something from somewhere else you cannot do that right
13:15 Scott Yeah, and this code happens to be like type script.
13:20 Scott And so there's an element of like lazy, you know, lazy importing that's happening that definitely like that's one of the things that one of the developers kind of came up with.
13:30 Scott I think that in the spirit of like,
13:34 Scott we ask these questions,
13:35 Scott we explore them and we often land on man.
13:39 Scott Yet again,
13:40 Scott we've run into another one of these unanswerable questions.
13:43 Scott The thing that is most interesting to me about this is I know this team is talented,
13:50 Scott has a great deal of process and culture where they care about code quality and they care about moving fast and creating quality code.
13:59 Scott And yet such a basic mistake happened.
14:03 Brad So what it did happen and it leads me to another thing when you mention the kinds of things that they're putting in the tech debt backlog.
14:11 Brad Not all tech debt is created equal.
14:14 Scott Exactly.
14:15 Scott That's the crux of my question.
14:17 Scott How do you know that the team is good at evaluating the value of the tech that they're asking to fix?
14:25 Brad Well,
14:25 Brad or in the processes that we have to prevent it.
14:30 Brad We have to ensure that those are actually designed to prevent the high cost tech debt,
14:36 Brad the high interest tech debt,
14:38 Scott Yeah,
14:38 Brad right?
14:38 Scott exactly.
14:39 Brad Not the low interest tech debt.
14:41 Brad So like some documentation slows down onboarding, but you're not hiring rapidly, so it's low interest,
14:48 Brad right?
14:49 Brad And so how do...
14:51 Brad This gets into the culture, I suppose.
14:53 Brad I said a minute ago that culture is right,
14:55 Brad but maybe there is some refinement of the culture, which is like actually getting when we do these code reviews,
15:00 Brad when we do these things,
15:01 Brad here's a specific type of tech debt coupling that we actually need to really be focused on because this is a high interest tech debt versus
15:09 Scott Yeah.
15:10 Brad these other kinds of things.
15:11 Brad And that's a teachable moment.
15:13 Brad I'm not sure it's an answer for an organization more at scale.
15:17 Brad I mean, this is why,
15:18 Brad to get to the organization of skill,
15:19 Brad this is why people love services and microservices, because it,
15:23 Scott At least it keeps your bad code choices closed behind a wall.
15:30 Brad yeah, that's right.
15:31 Brad If your contract first and
15:37 Brad It minimizes tight coupling,
15:39 Brad or I guess I shouldn't say it minimizes it.
15:41 Brad It's a structure that makes it harder to tightly couple.
15:46 Scott Yeah,
15:46 Brad But this is a tough one.
15:48 Brad Tech debt is a tough one.
15:49 Scott it's the toughest of them all.
15:51 Brad All right, well hopefully in the comments on this episode that we receive for future episodes,
15:57 Scott Exactly.
15:57 Brad people will tell us how they've solved this problem and what they think.
16:02 Brad You should be doing to solve this problem.
16:04 Brad It's stealth startup
16:05 Scott I love it.
16:08 Scott This episode of How Many CTOs Does It Take Podcast was edited by Saffron Epigal. Our theme song was AI Generated by Brad.
16:15 Scott Please follow us on all the social media platforms at at HowManyCTOsPod. You can find us on YouTube,
16:23 Scott LinkedIn,
16:23 Scott Instagram,
16:24 Scott Facebook,
16:24 Scott Threads, X,
16:25 Scott TikTok,
16:26 Scott and Patreon,
16:27 Scott or at our website at HowManyCTOsPod.com.
16:31 Scott And remember to rate, review,
16:34 Scott and subscribe on the podcast platform of your choice.
16:37 Scott Thanks for listening.