Navigating Technical Debt and Code Quality in Agile Teams
- 00:00 Introduction and Welcome
- 00:30 Discussing a Recent Incident
- 00:58 Importance of Code Reviews
- 03:00 Challenges in Code Refactoring
- 05:25 Code Review Process and Human Error
- 07:22 Tools and Strategies for Code Quality
- 08:11 Evaluating and Managing Tech Debt
- 15:47 Conclusion and Call to Action
- 16:06 Outro and Credits
Resources:
- How Many CTOs Pod: https://howmanyctospod.com
- Scott Porad: https://www.linkedin.com/in/scottporad/
- Brad Hefta-Gaub: https://www.linkedin.com/in/bradheftagaub/
- Defining, Measuring, and Managing Technical Debt - Google Research https://ieeexplore.ieee.org/document/10109339
Brad: You had the culture,
you had the talent,
you had the intent and effort and everything and it still happened.
Scott: Welcome to another episode of How Many CTOs.
Brad: Thanks for having me,
Brad.
Scott: I love talking with you. Okay,
so I've been dying to talk to you about this incident that just happened with this.
Brad: Don't forget to subscribe,
rate,
and review all
Scott: Yeah, smash that subscribe button.
Brad: your favorite podcast platforms.
Scott: Yeah,
on all of them.
Like we're on all of them. And we're on all the socials.
Okay, so you know I'm working on this small team. I'm in the trenches.
which is we're like seven developers it's super fun i'm actually writing code myself and this team has got a lot of like they take they take their code base very seriously and they like code reviews and detailed like code stewards and and this past sprint we had a developer who's going to do a very simple feature
It turned out what was like there,
you know, the feature of something like add an admin panel to the client app that only admins should be able to go to.
And when they started working on this,
they stumbled on the fact that the code that they were going to have to like integrate with this new feature had dependencies on completely random pieces of code that had nothing to do with their features.
So this is basically a basic breakdown in, like, areas of concern of the code,
like bad encapsulation or bad areas of concern.
Like, it was multiple objects were over constrained and oversubscribed.
And I was struck by the fact that, like, here it is a team that I'm watching on a daily basis in the trenches with them.
And they're all really good at their jobs and they're all really intentional and yet completely whiffed on this one.
They let this classic example of bad anti-patterns of technical debt creep in.
And I'm struck by like you're sitting there at the, you know, at the 150 employee level,
right? You're like.
you know, you're like three layers of management away.
And how do we trust? How do we verify?
How do we know that this isn't just happening rampantly across the code base all the time? And what do we do about it?
Brad: Okay, here's my first two reactions.
They're not exactly on the nose of your question.
I'll get to that.
My first thought is like agile principles,
scrum, agile,
that kind of approach,
it doesn't see refactoring as a bug, it sees it as a feature,
right?
Scott: Yeah.
Brad: Because it allows you, in the balance, in the spectrum or the balance of over-engineering and not building the thing that you want.
making progress, but then having to go back and refactor because you didn't over-engineer, you made near-term decisions,
it embraces that refactoring side because that
philosophy says look we want to have working software that meets customer needs and we spend so much time trying to think of every eventuality that first of all we build stuff we're not going to need second of all it takes us forever to get done and we don't actually get working software in people's hands so that was my first reaction which is like well I'm not sure that refactoring
is a bad especially at an early stage like yours when you're trying to get fast
Scott: Totally.
Brad: to get your thing launched i'm not sure that's terrible it's it's the feature if you made some other decision fast go
Scott: Okay.
Brad: ahead
Scott: So hold on a second. Let me just let me just sort of pause and echo back and sort of like validate and agree with where where I think I think you're spot on about that. Right.
Like the idea that the login controller.
had some functionality that missed functionality and now we're adding this new concept of like rules or something like that part of the refactoring totally agree with you agile it's all it's it's exactly why we do agile so 100 agree this particular example though is the type of thing that some of my senior engineers are like no no we we prevent this kind of stuff from happening
happening because we do code reviews and we're really uh we're really anal about the code reviews and we don't let bad code get get written in the first place right and like a simple example like this login controller was polluted with details about audio like it had like details about like audio playback like it never no like no way should have been in the login controller right
So, yeah,
so agreed on your point about refactoring,
Brad: Yeah.
Scott: and yet mistakes were made.
Brad: Mistakes were made. Well, we are human.
So I guess the question is, then you're asking the question is, okay, you're asking a different question, which is.
The code review process apparently is not a sufficient mechanism to prevent those mistakes from being made.
Scott: Yeah,
totally.
Brad: And so it'd be very interesting.
I would be curious about something.
I would want to go I want to so if I was having a retrospective and saying,
well,
how did this happen?
How do we prevent it from happening again?
And,
you know, a good open hearted like people were not trying to just assign blame.
I would.
ask the question well what was it about the code review process that allowed this to happen and i can think of some potential reasons right one is the human error the code review was not done thoroughly right
Scott: Mm hmm.
Brad: or human error the code review was done thoroughly but somebody missed it right
Scott: Yeah.
Brad: um i could think of another piece which feeds human error
It was a whatever happened was a the costume is it was a big code review.
It was a lot of code.
Scott: Yeah, totally,
totally, totally. Yeah,
Brad: it was.
Scott: that's actually that's one of the things that's been that has shown up in the retros like,
oh, large, you know, you know, you know, more than 100 lines of code in your PR and that's like way harder to deal with. If it's thousands of lines of code, you're definitely going to put a great tax on top of your.
the code reviewers and so keep them small and stuff like yeah so so
Brad: I
Scott: for
Brad: will submit that it is impossible to do a good code review with more than 300 lines of code.
Scott: sure okay
Brad: Impossible.
Not, not humanly possible.
Scott: so hold on a second hold on a second scott i love the direction you're going and this is all this is all 100 aligned with i think the how you fix it at the six person team
Brad: Yes. How do you fix it at the big team?
Scott: Yeah,
that's really the question here.
Brad: I'm not saying I've done this, but here could be a concrete example.
There is a system called danger,
danger rules.
There are rules you can put in your CI.
It's a framework for adding rules and checks and things like that to your CI pipeline.
I really like it. I think the CI pipeline is actually a way that you can really enforce a lot of quality.
So just as a concrete example.
One could make a rule that it will reject APR that has more than 300 lines of code.
Totally. That's one thing.
Scott: Right.
But here's my question for you, though.
You've been air dropped in.
You've got 10, 15 years of culture and code.
How do you know?
Brad: I have two thoughts. The first is where I always go is trying to quantify something.
That's where I always go because I don't know.
I just know anecdotally I came in here and it's not just this job. It's the last job and the one before that.
My last three jobs have all been about fixing engineering teams that have had tremendous tech debt.
and where things weren't getting done and people were unhappy and they were good engineers but it was just like it was just a situations in crisis my last three jobs has been kind of crazy and so my inclination is to always and you hear anecdotally everyone's like it's so bad it's so bad so that so my inclination is to try to go to quantify now on the one hand I have a senior individual on our team right now who is trying to go through every
package of our component of our code that we've developed a rubric for the level of tech debt and he's trying to evaluate with the teams how is this now this doesn't prevent it but at least it gives us a way of starting to keep score on the other hand I just read an article I saw an article I could dig it up
I can put it in the show notes because
Scott: Yes, please put it in the show notes.
Brad: some engineers and researchers at Google tested like 130 different.
indicators of tech debt and they could find nothing that correlated to tech debt
Scott: Right
Brad: so maybe i'm going down they were trying to objectively measure tech debt maybe i'm going down a fool's errand to try and measure this but the only way i know how is to just get a number and say like look when you take all this stuff and yet you take all these packages and we scored them all these different ways and we edited them all up and we did all this stuff and like here's our number and we're going to do this once a quarter
And hopefully it's going to go down, right?
Scott: Yeah.
What I was struck by in the real time experience of this real example of tech net really slowing down a project and everybody looking at it and going,
oh, yeah, mistakes were made like it was sort of like it was clearly objective,
not subjective that.
The task was harder because of existing code.
In examining the existing code,
every single engineer that looked at it said,
oh, yeah,
that's wrong.
Brad: It's so crazy. You had the culture,
you had the talent,
you had the intent and effort and everything,
and it still happened.
Scott: And it still happened. Now, here's the other.
And honestly,
infuriating thing to me, because this is what really got my hackles up.
The team constantly adds tech debt to a backlog and says, we'd like to do this. We'd like to do this. We'd like to do this.
This one wasn't on that.
When I sort of try to look at the list of what they have versus what.
What this one was,
I actually think the stuff they came up with was so much less important than this one.
Brad: What were the differences?
What were the kinds of things they were coming up with versus this one?
This one was a tight coupling.
Scott: This was a tight coupling.
This was basically tight. So this was a coding anti-pattern of tight coupling,
and it definitely fell into the...
uh it wasn't broken until it was like it wasn't a problem until it was right it was a ain't broke don't fix it type thing if anybody had come across it and said hey this is weird the login controller knows about the audio right i
Brad: Yeah.
Scott: could i could definitely picture some i could picture myself being like okay that sounds like a legit problem problem
Let's work on that when it comes around kind of thing,
Brad: Let's
Scott: right?
Brad: add it to the backlog.
Scott: Let's add it to the backlog.
The kind of stuff that is more like is in there, you know, honestly is stuff that's like go back and add type doc to old files that don't have type doc.
so that one i could imagine somebody a new developer coming along and going oh there's this whole new set of code i don't understand if there was only type if there was documentation on it would be better you know but i don't know so so anyway so in the spirit of like a nice bite-sized challenge here i like your idea of like can we use tools to enforce this i like that idea
Brad: Well, you know, I know a team that has written automation into their CI pipeline to prevent tight coupling.
Scott: That's interesting.
Brad: Now, specifically the technology they're using.
Django and Python for
Scott: Mm
Brad: their back
Scott: -hmm.
Brad: -end services and in that framework you have to explicitly declare when you're essentially import you have to declare other packages right so that gives them a mechanism to say oh you are you are importing something from somewhere else you cannot do that right
Scott: Yeah, and this code happens to be like type script.
And so there's an element of like lazy, you know, lazy importing that's happening that definitely like that's one of the things that one of the developers kind of came up with.
I think that in the spirit of like,
we ask these questions,
we explore them and we often land on man.
Yet again,
we've run into another one of these unanswerable questions.
The thing that is most interesting to me about this is I know this team is talented,
has a great deal of process and culture where they care about code quality and they care about moving fast and creating quality code.
And yet such a basic mistake happened.
Brad: So what it did happen and it leads me to another thing when you mention the kinds of things that they're putting in the tech debt backlog.
Not all tech debt is created equal.
Scott: Exactly.
That's the crux of my question.
How do you know that the team is good at evaluating the value of the tech that they're asking to fix?
Brad: Well,
or in the processes that we have to prevent it.
We have to ensure that those are actually designed to prevent the high cost tech debt,
the high interest tech debt,
Scott: Yeah,
Brad: right?
Scott: exactly.
Brad: Not the low interest tech debt.
So like some documentation slows down onboarding, but you're not hiring rapidly, so it's low interest,
right?
And so how do...
This gets into the culture, I suppose.
I said a minute ago that culture is right,
but maybe there is some refinement of the culture, which is like actually getting when we do these code reviews,
when we do these things,
here's a specific type of tech debt coupling that we actually need to really be focused on because this is a high interest tech debt versus
Scott: Yeah.
Brad: these other kinds of things.
And that's a teachable moment.
I'm not sure it's an answer for an organization more at scale.
I mean, this is why,
to get to the organization of skill,
this is why people love services and microservices, because it,
Scott: At least it keeps your bad code choices closed behind a wall.
Brad: yeah, that's right.
If your contract first and
It minimizes tight coupling,
or I guess I shouldn't say it minimizes it.
It's a structure that makes it harder to tightly couple.
Scott: Yeah,
Brad: But this is a tough one.
Tech debt is a tough one.
Scott: it's the toughest of them all.
Brad: All right, well hopefully in the comments on this episode that we receive for future episodes,
Scott: Exactly.
Brad: people will tell us how they've solved this problem and what they think.
You should be doing to solve this problem.
It's stealth startup
Scott: I love it.
This episode of How Many CTOs Does It Take Podcast was edited by Saffron Epigal. Our theme song was AI Generated by Brad.
Please follow us on all the social media platforms at at HowManyCTOsPod. You can find us on YouTube,
LinkedIn,
Instagram,
Facebook,
Threads, X,
TikTok,
and Patreon,
or at our website at HowManyCTOsPod.com.
And remember to rate, review,
and subscribe on the podcast platform of your choice.
Thanks for listening.