Transcript: Culture, Quality, and Tech Debt: Lessons from the Front Lines with Greg Conklin
0:00 Greg Over time are you seeing a pattern here and is there a way for you to look at that as a leadership team?
0:05 Greg Yeah,
0:06 Greg dig into the technical things that are going wrong,
0:08 Greg but maybe your problem is not technical.
0:11 Greg Maybe there's something in the culture that you need to fix.
0:31 Brad All right. Welcome to another episode of How Many CTOs Does It Take Podcast?
0:36 Brad I'm Brad Hefta-Gaub and my co-host is...
0:38 Greg Stop or add.
0:40 Brad And today we have a special guest with us, Greg Conklin.
0:47 Brad Yeah. So, Greg,
0:48 Brad you and I and Scott all worked together at 98.6, but you've done so many other things. Why don't you give us a quick little rundown, introduce yourself and.
0:57 Greg Yeah, for sure.
0:58 Greg So I've been doing engineering leadership for a little over 25 years,
1:03 Greg I think.
1:04 Greg A lot of that has been early stage startups.
1:07 Greg Some of them have turned into things you know and most have not.
1:12 Greg Early in my career it was real networks where I got to work with Brad.
1:15 Greg We really were inventing streaming media.
1:18 Greg Most recently I spent a bunch of time at Dropbox right I was leading product engineering there building both the core product and some of the new products we were launching around video collaboration and other document collaboration products.
1:35 Greg And then the other startup I joined very early that people have heard of is Groupon, right?
1:41 Greg I joined their pre-IPO but post-Super Bowl ad.
1:45 Greg So the craziness was sort of reaching its,
1:49 Greg well,
1:49 Greg what I thought was sort of reaching its peak, but learned a ton.
1:52 Greg I saw a ton of different transitions and transformations there.
1:56 Greg That's me.
1:58 Brad So welcome very much to our little podcast here,
2:01 Brad Greg.
2:01 Brad It's so good to talk to you again.
2:07 Brad Why don't we, I don't know, Scott,
2:09 Brad you want to kind of chat a little bit about our most recent time where the three of us spent time together and sort of things we all collectively learned from that experience?
2:18 Brad Yeah, so we all spent about a year together.
2:22 Brad Roughly speaking,
2:23 Brad give or take a little bit, at 98.6, which was a telemedicine startup.
2:32 Brad It was kind of peaking during the beginning of the pandemic as people needed to do medicine online.
2:37 Brad I think I was there first and then I can't and then Brad you came after me and then Greg we sort of passed like ships in the night. I think you showed up right at the tail end as I was on my way out the door and I think each of us spent about a year there.
2:48 Brad Maybe Brad you might have spent the longest time there, maybe a couple of years there.
2:52 Brad I guess I have that badge of honor.
2:56 Brad Yes.
2:56 Brad Well, it's a great badge.
2:58 Brad You know, it wasn't a great fit for me. I personally didn't enjoy working in health care,
3:03 Brad but I did learn some things there,
3:07 Brad which was really great.
3:09 Brad And I thought I wanted to share what I learned with you guys, but I wanted to hear about what you guys learned as well.
3:14 Brad And I think because I think there's a good lesson in that,
3:17 Brad which is even when jobs don't work out or startups don't work out, there's stuff to learn from them.
3:22 Brad And the two things I learned at 98.6.
3:26 Brad One was I got a whole new appreciation for empowering people.
3:32 Brad And let me give you an example of that.
3:34 Brad There was some database upgrade that needed to happen and there needed to be a planned outage for it.
3:39 Brad And whoever was the engineering manager working on that came to me and said, you know, I need to do this, you know, I need to get approval for this, blah, blah, blah, blah. And I just started working there.
3:49 Brad And so I started going around asking people saying,
3:52 Brad you know, who do you get to sign off on this to do this thing? We can schedule it, etc.
3:57 Brad And the person who I was working for said to me, well,
4:00 Brad nobody.
4:01 Brad If that engineering manager thinks we need to have a site outage, a planned outage, they need to go around to the right stakeholders,
4:07 Brad make sure they're informed,
4:08 Brad schedule it.
4:09 Brad That's their decision.
4:10 Brad They are empowered to make that decision.
4:12 Brad And I had never seen that level of empowerment all the way down to creating a planned outage at an organization before. And that was really influential on me.
4:21 Brad I thought I was a person who tried to empower my.
4:23 Brad I managers and leaders before and I was like, oh, that's a whole new level.
4:26 Brad Cool.
4:26 Brad That was one thing I learned. The other thing I learned was about quality.
4:31 Brad I've mostly worked on consumer internet.
4:33 Brad I've worked on some business to business stuff too,
4:35 Brad but mostly consumer internet.
4:36 Brad And, you know, you get thousands and thousands of errors a day on an ordinary consumer internet website.
4:43 Brad But I saw 98.6 getting.
4:47 Brad single digits and double digits of errors a day or a week even and then investigating every single one of them because they were doing health care and they said we can't have you just can't have the application not work during an online medical visit and that really gave me a whole new appreciation for what operational quality was like when when I went into my first operational review and there were eight errors the previous week and every one of them had had a post-mortem on them
5:15 Brad I was like,
5:15 Brad whoa,
5:16 Brad that totally blew my mind for what it meant to really deliver a high quality product.
5:21 Brad So those were both, even though I was only at that company a short time, they were both really, those experiences were really influential on me.
5:27 Brad I don't know what your guys experience like.
5:29 Brad Start with you, Greg, since you're our guest here today.
5:31 Greg Yeah,
5:31 Greg I feel like when I joined,
5:34 Greg there was.
5:36 Greg The company was doing or starting through this transition.
5:39 Greg I remember when I first joined, I was told they were reaching the point where they needed actual engineering managers,
5:48 Greg right?
5:48 Greg And then maybe even the next layer beyond that.
5:51 Greg I can't remember exactly how large it was.
5:53 Greg It was, it was.
5:55 Greg tens of engineers right um or or thereabouts yeah
5:59 Brad I think it was like about 150 total across
6:02 Greg okay yeah
6:02 Brad the work.
6:02 Greg but it was at the point where cross engineering dependency management needed to be figured out right it was a bunch of teams that um each were organized to own a particular part of the product or part of the technology
6:20 Greg But they were bumping into each other all the time. And so I think what I the thing I sort of learned there is, well,
6:29 Greg AI renewed it is cross team dependency management is hard.
6:33 Greg And what I learned is one thing that tends to work is just like brute force it.
6:40 Greg And we I'm sure we can talk a bit a bit about that. There's no there's no.
6:44 Greg There are some frameworks out there and some ways of thinking that can help like optimize the process a bit,
6:49 Greg but there was nothing like
6:52 Greg Every so often whatever your cadence was getting everybody in the room
6:57 Greg All the team leaders and this walking through the list and being like,
7:01 Greg okay
7:01 Greg We're not gonna maybe work out all the dependencies,
7:04 Greg but like what are the ones we need to get into? So that was one and then the other thing the other thing I learned
7:09 Greg was the and I'm curious to get your perspective on this was there was sort of two centers of gravity within that healthcare company there was the engineering team technology organization and then the medical team and being able to work in an organization where
7:31 Greg both were sort of co-equal and had very different processes and mentalities and a lot of ways cultures and the sort of things I were working on needed to build the tools to support the medical team and that was for me that was a big that was a big struggle trying to figure out the right leverage the pull or the right way to be influential on that
7:49 Brad Yeah, that's so interesting.
7:54 Brad There's a lot, there's a rich vein in both those topics,
7:56 Brad Greg.
7:59 Brad I'll kind of throw out my topics of where I feel like I learned. I learned so much at 98.6. It was, it was a very.
8:10 Brad lesson rich environment let's say um i think that one of the things i would say i would categorize i'm going to say my two big learnings my first big learning was all around culture um i i felt like um 98.6 was one of the most intentional by far uh
8:34 Brad cultures that I had ever been a part of like I had never I had never I'd never worked for a company that that had such a strong list of core values and such intentionality around hiring against those core values and such intentionality around speaking to those core values in in every minute and so and
8:56 Brad There were a lot of good things that came out of that,
8:58 Brad and there were a lot of bad things that came out of that.
9:01 Brad And so I feel like one of the learnings that I got out of that is just going to be in the category called culture has so many downstream effects that you need to really be thoughtful about the culture you put in place.
9:21 Brad Um,
9:21 Brad and to Scott's point,
9:23 Brad like, um,
9:23 Brad that kind of like empowerment thing was an example of that, you know, but another really high level, high order bit that I would say is that, um,
9:30 Brad there,
9:31 Brad a lot of energy was put into the culture around,
9:34 Brad um,
9:35 Brad In the hiring process around filtering out sort of like programmerness or like kind of that kind of toxicity that you've seen in so many engineering companies where it's like, you know, this real like, you know, alpha hard push type thing and that got filtered out really well.
9:56 Brad And a really strange unintended consequence of that is that also happened to filter out bravery and willingness to.
10:09 Brad uh sort of like speak up
10:10 Greg Yeah.
10:11 Brad and uh speak out and so that's that's a that was a thing that I learned from that process now I don't have any I don't know that I got any great answers out of what to do instead but that was definitely like a thing that I learned from that
10:27 Brad The other big category I would say that I learned a ton from, and actually I learned a lot of this once Greg joined the organization because Greg brought a bunch of skills and approaches to this problem that I hadn't personally hadn't seen such strong leadership in before Greg came and kind of did it. And that was sort of like solving the problem of how do we have operational excellence.
10:52 Brad uh,
10:52 Brad while dealing with technical debt. And I think that one of the, so for example, when I joined,
10:59 Brad um,
11:00 Brad they had a, uh, the,
11:02 Brad the product had missed, uh,
11:04 Brad the uptime SLA for five quarters in a row, uh,
11:09 Brad when I got there and, um,
11:11 Brad and
11:13 Brad it was because of technical debt and problems associated with technical debt and sort of like turning that into like just going from the we're investigating these incidents into and now what do we do with that?
11:28 Brad I learned a lot about the processes and skills to solve that problem.
11:34 Brad How many companies do you think actually have?
11:38 Brad and uptime SLA that they track for five quarters in a row.
11:43 Brad I would assume many
11:45 Greg Yeah,
11:45 Brad do.
11:45 Greg I would think many do.
11:47 Greg I think that's
11:51 Greg one of the relatively straightforward metrics to kind of measure.
11:55 Greg I feel like your system's out of the box.
11:57 Greg Like AWS dashboard will tell you your uptime.
12:00 Greg And there's also a bit of a...
12:04 Greg I don't know if meme is the right word,
12:05 Greg but like five nines is a thing.
12:07 Greg And if you can't do five nines, you can do four nines.
12:09 Greg If you can't do four nines, you do three nines.
12:12 Brad Greg and I worked at a company 20 years ago together that was claimed five nines. That was a revenue science.
12:20 Greg Yeah.
12:21 Brad We did the five nines on one of the earliest versions of behavioral ad targeting.
12:25 Brad I think more of what I was trying to make is I don't know that most companies set that standard.
12:30 Brad They kind of react to it.
12:32 Brad If systems are down a lot, they say this is the problem.
12:35 Brad We need to get it back up.
12:36 Brad But I do remember we had contractual obligations at 98
12:40 Greg That's
12:40 Brad .6
12:40 Greg right.
12:40 Brad for certain uptimes. So we paid a lot of attention to it.
12:44 Brad I'm curious to dig in more. Greg,
12:47 Brad you can tell us about how did you handle that situation of.
12:51 Brad dealing with these stability issues through the tech debt and
12:54 Greg Well, four
12:55 Brad why did 98.6 which is only four years old have tech debt
13:00 Greg years old is plenty of time to accumulate a good amount of debt,
13:05 Greg right?
13:07 Greg I think the solution,
13:09 Greg there's layers to this, and I think you have to decide.
13:13 Greg where on the operational or team maturity you are on this scale,
13:19 Greg like do you have these, I mean Scott to your kind of gets to the point of your question,
13:24 Greg A,
13:25 Greg do you have these metrics,
13:26 Greg B,
13:28 Greg have you told your engineering team they matter,
13:30 Greg right,
13:30 Greg and then do you have the sort of observability in place to know how to debug where to debug with something.
13:39 Greg Something like that starts to go from green to yellow to red.
13:43 Brad Yeah, and I think that like.
13:45 Brad It's interesting that Scott,
13:47 Brad you said that you felt like when you got there, there was sort of like, you know, in the operational review,
13:51 Brad it was like any any error that happened, there was a root cause analysis of it.
13:56 Brad One of the things I noticed was that we we had the process that they had there was they had team members were on call and they would rotate on call.
14:11 Brad And when incidents happened,
14:14 Brad what it almost always,
14:16 Brad at least when I first got there in the first couple quarters of being there,
14:20 Brad it was a very much a put out the fire, not a.
14:27 Brad figure out why the fire started and move the kindling and the gasoline away from the open stove.
14:36 Brad So it was all putting out fires and it wasn't I didn't see a lot of progress being made against really coming back and saying, well,
14:44 Brad this is the root cause and therefore we should invest in fixing the root cause.
14:50 Greg Yeah,
14:50 Greg that's what I sort of recall too,
14:51 Greg like the post-mortem process.
14:53 Greg process was it existed but I'm not sure to what extent like engineering teams were putting time into doing the report doing the review either with us or with somebody else understand what was sort of gone it was sort of okay that fires out back to the backlog that the product manager wants me to
15:17 Greg Chase down.
15:18 Brad Yeah.
15:18 Brad And that's kind of the key insight that I feel like I give you credit, Greg,
15:22 Brad for bringing to this. And I was like, after seeing you implement this,
15:27 Brad I was just like, oh, this is I am going to use this from now on in the future. And that is, OK,
15:32 Brad we're going to do the RCA and then what we're going to do is we're actually going to have like a leadership team meeting that's a recurring.
15:39 Brad No-shit-it's-going-to-happen meeting where we're going to walk through the RCAs and we're actually going to ask ourselves,
15:45 Brad okay,
15:45 Brad what is the technical issue that this RCA is calling out and how likely is it to happen again and therefore we are going to put fixing this issue actually on the schedule.
16:00 Brad Right. And when I say it like that, it sounds super simple. Like, of course, why wouldn't you do that?
16:05 Brad But it was literally like scheduling the meeting where director level leaders would come in and we would all like we're all sitting there holding each
16:15 Brad Yep.
16:15 Brad other accountable to be like we read the RCAs and this is what it said. The database is three versions past, you know, the latest version or whatever.
16:26 Greg with the third party off system is is super flaky right and like and then what i mean by looking at the maturity of the team i feel like there's a there's the next couple level ups there the next couple level ups is okay
16:40 Greg over time are you seeing a pattern here and is there a is there a way for you to look at that as a leadership team i know in the previous podcast scott you had talked about like creating like a tech debt backlog right um and understanding that the process for looking at okay what's what's been failing and let's make sure we have a more longitudinal plan for getting that done and then the next one from there is like
17:05 Greg Yeah, dig into the technical things that are going wrong,
17:07 Greg but maybe your problem is not technical.
17:11 Greg Maybe we can go through the list. Maybe it could be something about the skills of the team.
17:20 Greg Maybe it could be about the nature of the business.
17:24 Greg Maybe it can be about, you mentioned culture.
17:27 Greg Maybe there's something in the culture that you need to fix.
17:30 Greg understanding those.
17:32 Brad So can I ask more about this meeting and this process that you set up?
17:35 Greg Sure.
17:36 Brad Who was in that meeting?
17:37 Brad Who were the people? Was it with product management in that meeting?
17:40 Brad I guess I have a few questions.
17:41 Brad My first question was who is in the meeting?
17:42 Brad My second question is you're going to put stuff on a team schedule,
17:46 Brad which is inevitably going to be adversarial with the product backlog,
17:49 Brad right? It's a zero-sum game.
17:51 Brad There's only one team, one set of people on the team that can only do so much.
17:54 Brad How did you navigate that piece?
17:56 Brad I guess the two questions I have.
17:58 Greg Yeah.
17:58 Greg So as an engineering leader,
18:02 Greg you, I set clear expectations.
18:06 Greg Like these are, these are invariants.
18:08 Greg We talked about five
18:10 Brad Mm-hmm.
18:10 Greg nines, like that's an, that's an invariant.
18:14 Greg So it doesn't mean it's, it's a non-starter to, I mean, you're always going to have those conversations, I think when business starts ended up in a rough shape and you need to make trade-offs, but.
18:27 Greg The outcomes and the to-dos from a post-mortem are priority zero,
18:33 Greg and you can put an SLA around that.
18:36 Greg Like these issues need to be resolved in two weeks,
18:40 Greg or else what is what you need to figure out.
18:44 Greg Like everyone gets six demerits or something like that, right?
18:48 Greg But that's sort of how this works.
18:51 Brad Yeah,
18:51 Brad and I think that the...
18:53 Brad The key insight of the let's have a meeting where once a month, I think, was what we scheduled. And it was, so to answer your question directly, Scott,
19:02 Brad my recollection was we had, it was, you know, sort of me as the VP,
19:05 Brad you know, or in this case, I was the senior most engineering leader.
19:12 Brad the next my my staff basically so the directors um and uh the engineering directors and um and i think that we that meeting did include the the product uh like the senior most product people in that case they were i think their their level was director at that point not a vp but um and you know they would come um and
19:39 Brad I think that I don't know that it ever necessarily became that adversarial, although kind of getting engagement with them about the importance of quality,
19:49 Brad that was something that was kind of an ongoing thing that we were working on.
19:54 Brad And to Greg's point,
19:55 Brad it was like the or else demerits.
19:59 Greg where Brad had set the tone that we didn't want to show up next month and not had made progress against these things.
20:07 Brad Correct.
20:08 Brad That's right.
20:09 Greg And so it was just that recurring accountability that we're all going to sit in a room with a guy who has resting serious face.
20:17 Brad What I meant by adversarial is not necessarily adversarial between the people,
20:21 Brad but.
20:23 Brad You know,
20:24 Brad product managers typically want engineering teams to spend as much time as possible working on their product backlogs.
20:31 Brad And that's a zero sum game with then issues like this, which I suppose you could say is in the sort of agile comes out of the agile,
20:39 Brad right? This is just deferred work from some previous product backlog work that didn't happen.
20:45 Brad Did you have an agreement up front with those product managers?
20:48 Brad For sure.
20:49 Brad I feel like there's a lot of engineering judgment that needs to be applied during the postmortem, right? Because you can end up in situations where the action items out of a postmortem can be, you know, in certain cultures it could be, ah.
21:07 Brad well if it's an if it's a if it's a post-mortem action item and it must be done then i'm going to stuff it with stuff i want to get i want to get done right um or the other culture side of the culture can be like i'm going to i'm going to minimize the amount of things i put on that list of things that need to get done because i got i got a product manager that's breathing down my neck so there's just i don't know if i have any magic spell for that other than you have to apply
21:34 Brad The judgment there that, yes, this went down, there should have been a unit, there should have been unit test coverage there.
21:43 Brad Okay,
21:44 Brad that seems pretty straightforward and light and as the engineering leader in that meeting,
21:50 Brad yeah,
21:51 Brad I hereby bless that and stamp that as a postmortem action item that should take priority over getting the product backlog done.
21:58 Brad And then I alluded to, you know, when Brad and I were there, we encountered a situation where we were getting a lot of outages because our third party authentication system was flaky.
22:11 Brad And that's not so simply declared a postmortem action item just to like rip out your auth system. So that one's like,
22:20 Brad okay, cool,
22:20 Brad that needs to, you know.
22:23 Brad be put in the hopper for a bigger conversation at the leadership level including product managers around are we are we going to do this or not
22:30 Greg Yeah, and that was a good example where to Greg's recollection of the story,
22:35 Greg like this was a third party off system.
22:38 Greg It was the source of many of our missed quarterly SLAs and kind of.
22:48 Greg The idea of like instead of just having the RCA that goes,
22:51 Greg yep,
22:52 Greg third party auth system failed.
22:53 Greg That's what happened here to actually like have that catalog and go,
22:57 Greg it has happened,
22:59 Greg you know, dozens of time and look at the actual like impact on our on our contracts because of it.
23:07 Greg And now we were able to like surface that and go, look, this is going to keep happening until we invest in this.
23:14 Greg effort and and you know kind of the the
23:20 Greg Going from the mindset of, oh,
23:23 Greg errors happen,
23:24 Greg we put out the fire, and we document that the fire happened,
23:27 Greg to a mindset of, we're going to look at all the fires that happen, and we're going to start to look for common themes,
23:34 Greg and we're going to start to invest in the tech debt creating those common themes.
23:39 Greg That mindset shift allowed us to then successfully have the conversation with the senior product team to say,
23:47 Greg We need to invest in this.
23:49 Greg This is a huge red flag here that we have to fix this.
23:56 Greg Our product is useless if people can't log into it.
24:00 Greg Right.
24:00 Brad Yeah.
24:01 Greg And it took that evidence and it took that sort of like process, that additional process was the key to unlocking fixing something that then was a major,
24:15 Greg you know, a blessing to the engineering team because like now all of a sudden we didn't regularly have fires on that part.
24:25 Brad One quick Dropbox story that I think about a lot was I was in a series of these postmortems and I was detecting this pattern and the pattern was for some of the Dropbox website that isn't heavily trafficked.
24:44 Brad There would be a code change that would, when deployed,
24:47 Brad the page just didn't work.
24:49 Brad Like it would immediately and entirely 100% of the time give you a 503,
24:53 Brad right?
24:57 Brad I was like, okay, there's something going on here,
25:00 Brad and obviously when you dig in, it's not as simple as you think.
25:04 Brad It's about the different environmental and experimentational setup between the production environment and your test environments.
25:13 Brad So I gathered up the staff engineers,
25:16 Brad and I said, look.
25:18 Brad let's do this like meta postmortem every quarter where I want you to look at all the sabs and all the problems and I want you to come back to me with your top three and I'm going to get those things we're going to get those things done and they did that and when they came back they were like
25:38 Brad Well,
25:38 Brad Greg,
25:38 Brad if you really want to do something that's going to reduce our quality risk,
25:43 Brad it has nothing to do with these salves. It was, and I don't know if I want to mention it on the podcast,
25:49 Brad but it was like some very dangerous activities engineers were doing to get customers back in shape.
25:56 Brad That's the thing you should fix.
25:58 Brad And that's when a light bulb.
26:00 Brad lit up like okay you just need to there's a lot of value in just creating the space to have the conversations around the engineers that know what's going on
26:09 Brad This will grow.
26:09 Greg So what you're saying, Greg,
26:11 Greg just to clarify,
26:12 Greg not to intentionally completely air the dirty laundry,
26:15 Greg but you're saying that like one,
26:18 Greg you implemented a meta-analysis of the RCAs. Like that's essentially the same playbook that we ran,
26:24 Greg you and I ran at 98.6. So you implement a meta-analysis and then it happened to be that the meta-analysis came back and it was essentially a,
26:32 Greg I'm going to put this in the category of a culture.
26:35 Greg culture thing because it was you had essentially cowboy ops or cowboy dev
26:39 Brad Right.
26:39 Greg happening um in in that was
26:42 Brad Well,
26:42 Greg what they were the doing
26:42 Brad eye opening
26:43 Greg yeah
26:43 Brad thing was, it was something I was not even aware that was happening.
26:47 Brad Right.
26:47 Greg right
26:48 Brad And that had me reflecting like,
26:51 Brad okay,
26:51 Brad why did I?
26:53 Brad Why did I not know that was happening?
26:55 Brad Right?
26:55 Brad You think you have these processes in place that will cover these sorts of things,
26:59 Brad but
26:59 Brad Um,
27:00 Brad you gotta, you know, open up the space to have your engineers kind of lift up these risks to you in a way that's, that's, that's structured and you get the visibility in that, even to that sort of stuff.
27:11 Greg You know, it's interesting.
27:12 Greg I don't know if this is, this makes me think about another thing that I learned from.
27:19 Greg my time at 98.6 and again like super talented team and i have a ton of respect for the folks there and they worked really hard um and so uh i don't mean this as a judgment anyway but but one thing i noticed was there was a a strong the language around blameless retro had this um
27:44 Greg I think unintended side effect where and I've stopped in my current team, I don't use the term blameless retro.
27:51 Greg I use the term egoless retro instead.
27:54 Greg And the reason for that is that they they would not put people's names on any of the timeline of what happened.
28:03 Greg They would basically try to anonymize.
28:08 Greg the TikTok around understanding what happened. And what I noticed was that makes it really hard to have a...
28:18 Greg Like, well,
28:19 Greg what really happened here?
28:20 Greg Like, no one's going to get fired over this. Like, it's OK that Brad,
28:23 Greg you know, logged into the terminal and started like, you know, deleting rows out of the database. I need to know that that happened,
28:32 Greg you know, like and I'm just kind of curious,
28:36 Greg Greg,
28:36 Greg do you feel like when you say like creating the space,
28:40 Greg like how do we get?
28:43 Greg How common, I feel like it's pretty common for people to be afraid of kind of really laying out what happened in and it's and how do you create the culture so that people are not afraid of really talking about what happened.
29:00 Brad That's a great question.
29:01 Brad I think that's probably a specific question around the general question of like building culture.
29:06 Brad Like I think.
29:08 Brad I think it's top down and bottom up.
29:12 Brad Like it's easy for top down leaders to completely break a culture in that sort of environment where something like that happens and you start pointing the fingers or you start digging in there.
29:26 Brad And then bottom up in that.
29:29 Brad like you have to make sure that culture and that expectation is set before you get into that post-mortem meeting because at the point at which something has failed and things are a little bit radioactive you're and you don't have those expectations set that it's going to be okay then you're going to have a bad time
29:47 Brad A language I used around that.
29:51 Brad So there's a lot of language I use. I think you talk about it all the time. You use the word blameless. I've never used egoist, but I kind of like it.
29:58 Brad I would continuously emphasize we are here in this room to come up with action items and working agreements to avoid this mistake in the future.
30:10 Brad We've all made mistakes.
30:12 Brad whoever made this mistake they know it they're not happy about they're unhappy with themselves about it like we're not here to pick on greg for fat fingering something in the terminal we're here to figure out how we can make it so that that the opportunity for greg to fat finger something doesn't even exist any longer either because we've got
30:31 Brad We got guardrails around it or automation around it or whatever.
30:34 Brad I came back to that language, action items and working agreements.
30:37 Brad That's the outcome we're looking for,
30:39 Brad not who screwed up, but what can we do to avoid it or minimize the impact of suppose the last radius.
30:50 Brad So.
30:53 Brad You wrote a blog post, which I think is relevant to this part of the conversation about things to do early at a startup to make it easier in the future.
31:07 Brad Yeah.
31:09 Brad I did.
31:12 Brad These were just based on sort of observations I
31:15 Brad I had I've seen the opposite as well where early startups decided early startups populated by engineers that had seen a thing or two at larger companies and decided to kind of over architect the scalability of it ends up slowing them down I think maybe we've all you're laughing and maybe you've seen some of that some of that too but
31:36 Brad Yeah.
31:36 Brad in a couple
31:38 Brad In a couple places I've seen engineering and engineering leadership lay down certain expectations around how certain standards will be put in place.
31:52 Brad One that sort of struck me, I didn't kind of realize it was too late at a place I worked was performance,
32:02 Brad like product latency and speed.
32:05 Brad Even just saying like,
32:06 Brad okay, we expect web pages to respond in two seconds,
32:12 Brad whatever that happens to be in certain percentiles and certain measurements of that, or backend services need to respond in 100 milliseconds or things like this.
32:22 Brad It set the expectation. And what happens if they don't, like you can decide and maybe you crank that down or up over time,
32:30 Brad but just setting those sort of expectations.
32:33 Brad means that, okay, when I am working with my team to decide how I'm going to meet the requirements from my product manager,
32:43 Brad it's non-negotiable that it is this performant, right?
32:52 Brad and then the flip side is you know i would talk to teams that didn't have those sort of standards and they'd say well greg if you if you wanted it to be that fast we never would have built it this way in the first place and that's a right and that's like okay oi that that seemed like a conversation we should have had to
33:10 Greg Yeah.
33:10 Brad had earlier and so that that that's that's one that's one example and there's other examples of like i mean we had talked about reliability and
33:18 Brad setting however many number of nines and
33:21 Brad Mm-hmm.
33:22 Brad some other engineering standards you can put into place performance in my experience has been one where it's always hard to kind of retrofit performance even reliability and uptime like it sucks that you have to go backtrack a little bit but it's it when you talk to your engineering teams about what it would take to get reliability up
33:44 Brad It tends to be just work, right?
33:47 Brad Yeah.
33:48 Brad But then performance seems like, oh shit,
33:51 Brad like, this wasn't, I turned, it turns out I built a minivan,
33:58 Brad right?
34:00 Brad Which is what I thought we wanted.
34:03 Brad That makes it hard.
34:05 Brad If you were starting.
34:09 Brad a new company or new building a new application new web app today new online service and what are the top of those things that you could put in place early that makes life easier down the road what are the top three what's the top one
34:25 Brad I mean, the things I sort of written about were like expectations and standards.
34:28 Brad So like I would, I would.
34:31 Brad I mean, I have a list on the blog.
34:33 Brad Performance is towards the top of the list. I think analytics,
34:36 Brad right,
34:37 Brad understanding customer behavior.
34:39 Brad And by that, I sort of be,
34:40 Brad so a lot of these frameworks you can kind of get off the shelf have like exception logging and, you know, error reporting.
34:48 Brad And depending on how big you are,
34:50 Brad you can start with grep as your tool and work yourself up from there.
34:55 Brad By customer analytics, I mean like who's using your website.
34:58 Brad Like the data you would need for a project manager to be able to make a decision.
35:04 Brad like a Google Analytics or Amplitude or Mixpanel kind of thing.
35:08 Brad Exactly,
35:09 Brad exactly.
35:10 Brad Where does test automation fall in your framework?
35:13 Brad Test automation is on that list.
35:16 Brad And by test automation,
35:18 Brad what I've seen is...
35:23 Brad Engineers have this pre-installed expectation that they're going to write unit tests or they're going to write end-to-end tests.
35:31 Brad And then over time what ends up happening if you're not careful is your build system starts to slow.
35:40 Brad It starts to become flaky.
35:43 Brad right your build time goes from 10 minutes to 30 minutes to hours and then you sort of reach a tipping point where at least I've seen this in a couple places I've worked where the the CICD system takes so much time it actually starts changing engineering behaviors around how they write code right because if it's going to take three hours for the build to run and likely to flake out
36:08 Brad then I'm not going to do the quick iterative development process I'm going to batch up and so it's not just yes you should have unit tests or you should have automated testing but like what what are the what are the boundaries of those things like are you going to set an SLA for your build and if it goes beyond that sort of amount of time or the amount of resources the amount of unreliability that you're going to
36:33 Brad Push the big red button and do something about it.
36:36 Brad Yeah, we did that at Rover where our target was 10 minutes and when it got to 20 minutes, then we took action.
36:44 Brad And I would say about every six months,
36:46 Brad we had to have someone spend some substantial amount of time working on it to figure out what the next iteration of making it go faster was.
36:57 Brad Yeah, and at Dropbox, we had reached a point where our build times were really quite slow and it was starting to change the way sprints were run and the way engineers would calculate,
37:07 Brad you know, the work they could get done in a certain iteration.
37:11 Brad And we ended up doing a little project to sort of analyze our end-to-end testing.
37:20 Brad right um uh like what was going on there why was it flaky and there was a bunch of infrastructures things like you know you at that particular scale in order to do sort of a an end-to-end test you needed automation that would
37:34 Brad build an entire mini version of the Dropbox infrastructure just so you can run a test right and that that took some real infrastructure work to kind of work out but then we just sort of did an audit of these tests like what are what are these tests I feel like engineers they look at the code they're building to build the product and
37:52 Brad there's this notion that you want to refactor and re-architect parts of the product but it's not really natural for you to do that with your test with your
38:01 Greg Yeah.
38:01 Brad test and
38:01 Greg How often do you go back and refactor your tests?
38:04 Brad what we found what we found is like a third of the tests we could just not run like they were they weren't useless but like they were covered by other tests and then like another i don't know 15 of the tests
38:18 Brad were not not critical and so it could be run on a different cadence right not not not for every not for every sort of um code change and that like cut that part of the test infrastructure in half in terms of time which was which was huge
38:33 Greg You know,
38:34 Greg I'm curious about what you think about this, Greg.
38:36 Greg We ran into this, you and I ran into this at 98.6. And it's so what I learned from that, I added to my list of like,
38:44 Greg every new team, I want to solve this problem early.
38:48 Brad Yeah.
38:49 Greg And that is the ability for like,
38:53 Greg Out of the gate, do you have an architecture and a plan that allows individual developers to actually run?
39:04 Greg the system before making a pr you know like do you have your as your like i almost sort of it's like uh what is it like design for manufacturing where people like kind of think about the design of their physical components so that it can be manufactured i feel like uh design for debugging or design for individual developers to be able to
39:29 Greg to run the stack when they're making code changes.
39:33 Greg I feel like get that done early or else you can never fix it later.
39:39 Brad Yeah, for sure.
39:40 Brad And being able to run the test locally.
39:43 Greg Yeah.
39:43 Brad For sure.
39:44 Brad Greg, you talked about how the build system can change engineers'
39:51 Brad behaviors.
39:52 Brad So can the local development environment.
39:55 Brad Something I've seen recently.
39:57 Brad Is engineers having trouble?
39:59 Greg running the what Brad was saying having trouble running the whole system locally having trouble running the tests locally so then they want a staging environment to do all that on and then you've just the cycle time becomes really long all because we're not taking the time to invest in actually just getting it to work on the desktop
40:22 Brad Well, and even if you can't get it, there might be reasons why, you know, some parts of your system can't practically run on every single desktop.
40:31 Brad At least then make sure you have.
40:34 Brad ephemeral environments that are dedicated to the engineer right so that it's not like we're just throwing it into dev before i can do any testing
40:43 Greg Mm
40:44 Brad right
40:44 Greg -hmm.
40:44 Brad like that's that's what i saw i've seen that with teams when it's like there's no practical way for me to test this except for go ahead and commit it and check it into dev and it's in the ci it's been it's it's in the ci cd and it's sitting there in dev and i've you know i'm going to feature flag around it to test it or something
41:01 Brad something
41:01 Brad Yep,
41:01 Brad you
41:01 Brad I
41:01 Brad know but like that code is in there it has polluted your main line at that point and if that's your only way to test your feature like talk about like super long cycle times right that's that's horrible
41:18 Greg Have either of you worked with GitHub Codespaces?
41:22 Brad have not.
41:24 Greg Dying to meet somebody who's actually used them.
41:27 Greg They seem like the answer to so many of these problems,
41:30 Greg and yet I have not come across a single person who's used them.
41:33 Brad You asked me this in episode one.
41:35 Brad I still haven't researched it because I have a system that's working like the local devs can work in the local environment and then they can also push to an ephemeral environment.
41:45 Brad And man, those two things right there are so powerful that.
41:49 Greg When you say ephemeral environment,
41:51 Greg each developer has their own environment.
41:53 Greg Isn't that right?
41:53 Brad yes yeah
41:54 Greg Their own staging
41:55 Brad i've
41:55 Greg environment.
41:55 Brad seen that too yeah
41:56 Greg Well, that works.
41:57 Brad or even like a you could even do it with off a pull request like every pull request gets its own environment right so then a developer could have multiple things in flight if they wanted and actually like
42:07 Brad Go check it out over here. Go check it out over there.
42:09 Greg I think that's how Codespaces works.
42:11 Greg But there's a scale and cost problem there,
42:14 Greg right,
42:15 Greg which is if you're setting up a staging environment for everybody,
42:18 Greg it starts to get very expensive really quick because those things are not in use very frequently.
42:23 Greg I might use it four minutes a day to test something.
42:27 Greg I guess if you shut it down, if you have a way to shut it down and ramp it up, you save money.
42:32 Scott Yeah.
42:33 Scott I mean, places like Dropbox had something similar to that where it would fire up the right number of instance or an instance in AWS,
42:42 Scott stand up a mini environment,
42:45 Scott populate it with mock data, and then it would persist for some period of time and then be torn down when the developer was done with it.
42:57 Greg There must be third-party solutions to do that.
43:01 Greg I can imagine saying that to some sort of cloud DevOps-y type engineer and they go script the whole thing out and build it all their own.
43:07 Greg There must be solutions for that.
43:10 Brad Well, my point is.
43:13 Brad make sure you have it out of the gate like don't don't wait till it's too late and you've got you know 100 devs and they're all sitting there pulling their hair out because the only way for them to test their stuff is to you know
43:24 Scott Push
43:25 Brad deploy
43:25 Scott it.
43:25 Brad to
43:25 Scott Yeah
43:25 Brad dev to push it you know that that's um that's my point is like i'm i'm sure there's we have a solution that's working for us it's not that expensive code spaces might be a great answer for um
43:38 Brad you know other folks but yeah i
43:40 Greg And you build that whole system yourself or did you use some sort of third party solution for it?
43:46 Brad mean it's not it i mean it's just kind of like you know whatever your standard infrastructure as code type stuff like terraform or whatever and you know you can usually just pop up an instance pretty quickly it's not free to build it like you have to build it but it's mostly just like you know standing up a
44:04 Brad Whatever ECS based on a terraform
44:07 Greg That engineer has like a command line tool they can do that from their desktops. They fire up my got
44:13 Brad We have it wired up into github actions,
44:15 Brad so it's just like boom you open a pull request and deploys to your stack and
44:20 Greg it cool sounds
44:22 Brad Yeah
44:22 Greg like you have a product there
44:26 Brad I am I gonna go that far
44:31 Brad This is so much fun, Greg.
44:32 Brad It was great to reconnect with you and kind of go through some of these things.
44:38 Brad What do you, how would you put a bow on this conversation?
44:43 Brad How would you wrap this up?
44:44 Scott Oh my gosh,
44:45 Scott how would I wrap this up? I mean, we talked about a bunch of different things.
44:49 Scott I think overall,
44:51 Scott it's about making sure as an engineering leader,
44:55 Scott you're setting clear.
44:57 Scott expectations around where the where the guardrails are like where where are the guardrails um i think as an engineering leader you want you want to see your team building a bunch of stuff shipping a bunch of stuff getting it in front of customers having the customers be super happy but you you need guard these guardrails at different scales those guardrails could be very different but even at small teams
45:23 Scott early stage it's not it's not terribly hard to just start setting up guardrails to make sure that like okay well when you crash into this guardrail like at the very least i'm going to get a slack message from you um and then we can and then we can decide and then we can decide as a team what we're going to do about it
45:40 Brad Yeah,
45:41 Brad I mean, I think that, you know, my summary is that.
45:45 Brad Sort of like the seeds that you plant early on in the organization,
45:50 Brad whether those seeds are cultural seeds or technology seeds or process seeds,
45:56 Brad they grow and they result in a garden that is in some shape,
46:04 Brad you know, and it's a lot easier to be planful about.
46:11 Brad the cg plant then to come back later and try to weed it you know something's
46:16 Scott yeah something's going to grow something's going to grow there what
46:19 Brad growing
46:21 Greg That's a great way to say it.
46:22 Greg Something's going to grow there.
46:24 Scott are the what are the things i when i do skip levels or i'm talking with staff or principal engineers or even i'm talking to engineering manager about their team i want to know if they're are they proud of are they proud of what they're doing are they are they proud of the work
46:39 Scott that's like a high level because if there's quality problems if they're always dealing with 2am pages they're not they're not proud of that they're not proud of that right yeah
46:49 Brad Yeah.
46:50 Greg Great question to ask people.
46:51 Greg I love it.
46:52 Brad Yeah.
46:54 Scott really great this has been great
46:55 Greg Cool,
46:55 Scott catch up
46:56 Greg man.
46:56 Scott really enjoyed the conversation this is awesome
46:58 Greg Before you go, Greg,
47:00 Greg where is your blog so everyone can find it?
47:02 Scott gregconklin.com easy
47:05 Greg Easy peasy.
47:06 Brad Nice.
47:06 Scott peasy yeah
47:07 Brad We'll add that to the show notes for sure.
47:09 Scott awesome well thank you two really appreciate it
47:11 Greg All right. Thank you.
47:12 Scott yep bye
47:15 Brad This episode of How Many CTOs Does it Take Podcast was edited by Saffron Heppigow.
47:20 Brad Our theme song was AI generated by Brad.
47:23 Brad Please follow us on all the social media platforms at at HowManyCTOsPod.
47:28 Brad You can find us on YouTube,
47:30 Brad LinkedIn,
47:31 Brad Instagram,
47:31 Brad Facebook,
47:32 Brad Threads, X, TikTok,
47:34 Brad and Patreon,
47:34 Brad or at our website at HowManyCTOsPod.com.
47:39 Brad And remember to rate, review, and subscribe on the podcast platform of your choice.
47:45 Brad Thanks for listening.