Culture, Quality, and Tech Debt: Lessons from the Front Lines with Greg Conklin

0:00 Greg: Over time are you seeing a pattern here and is there a way for you to look at that as a leadership team?

0:05 Yeah,

0:06 dig into the technical things that are going wrong,

0:08 but maybe your problem is not technical.

0:11 Maybe there's something in the culture that you need to fix.

0:31 Brad: All right. Welcome to another episode of How Many CTOs Does It Take Podcast?

0:36 I'm Brad Hefta-Gaub and my co-host is...

0:38 Greg: Stop or add.

0:40 Brad: And today we have a special guest with us, Greg Conklin.

0:47 Yeah. So, Greg,

0:48 you and I and Scott all worked together at 98.6, but you've done so many other things. Why don't you give us a quick little rundown, introduce yourself and.

0:57 Greg: Yeah, for sure.

0:58 So I've been doing engineering leadership for a little over 25 years,

1:03 I think.

1:04 A lot of that has been early stage startups.

1:07 Some of them have turned into things you know and most have not.

1:12 Early in my career it was real networks where I got to work with Brad.

1:15 We really were inventing streaming media.

1:18 Most recently I spent a bunch of time at Dropbox right I was leading product engineering there building both the core product and some of the new products we were launching around video collaboration and other document collaboration products.

1:35 And then the other startup I joined very early that people have heard of is Groupon, right?

1:41 I joined their pre-IPO but post-Super Bowl ad.

1:45 So the craziness was sort of reaching its,

1:49 well,

1:49 what I thought was sort of reaching its peak, but learned a ton.

1:52 I saw a ton of different transitions and transformations there.

1:56 That's me.

1:58 Brad: So welcome very much to our little podcast here,

2:01 Greg.

2:01 It's so good to talk to you again.

2:07 Why don't we, I don't know, Scott,

2:09 you want to kind of chat a little bit about our most recent time where the three of us spent time together and sort of things we all collectively learned from that experience?

2:18 Yeah, so we all spent about a year together.

2:22 Roughly speaking,

2:23 give or take a little bit, at 98.6, which was a telemedicine startup.

2:32 It was kind of peaking during the beginning of the pandemic as people needed to do medicine online.

2:37 I think I was there first and then I can't and then Brad you came after me and then Greg we sort of passed like ships in the night. I think you showed up right at the tail end as I was on my way out the door and I think each of us spent about a year there.

2:48 Maybe Brad you might have spent the longest time there, maybe a couple of years there.

2:52 I guess I have that badge of honor.

2:56 Yes.

2:56 Well, it's a great badge.

2:58 You know, it wasn't a great fit for me. I personally didn't enjoy working in health care,

3:03 but I did learn some things there,

3:07 which was really great.

3:09 And I thought I wanted to share what I learned with you guys, but I wanted to hear about what you guys learned as well.

3:14 And I think because I think there's a good lesson in that,

3:17 which is even when jobs don't work out or startups don't work out, there's stuff to learn from them.

3:22 And the two things I learned at 98.6.

3:26 One was I got a whole new appreciation for empowering people.

3:32 And let me give you an example of that.

3:34 There was some database upgrade that needed to happen and there needed to be a planned outage for it.

3:39 And whoever was the engineering manager working on that came to me and said, you know, I need to do this, you know, I need to get approval for this, blah, blah, blah, blah. And I just started working there.

3:49 And so I started going around asking people saying,

3:52 you know, who do you get to sign off on this to do this thing? We can schedule it, etc.

3:57 And the person who I was working for said to me, well,

4:00 nobody.

4:01 If that engineering manager thinks we need to have a site outage, a planned outage, they need to go around to the right stakeholders,

4:07 make sure they're informed,

4:08 schedule it.

4:09 That's their decision.

4:10 They are empowered to make that decision.

4:12 And I had never seen that level of empowerment all the way down to creating a planned outage at an organization before. And that was really influential on me.

4:21 I thought I was a person who tried to empower my.

4:23 I managers and leaders before and I was like, oh, that's a whole new level.

4:26 Cool.

4:26 That was one thing I learned. The other thing I learned was about quality.

4:31 I've mostly worked on consumer internet.

4:33 I've worked on some business to business stuff too,

4:35 but mostly consumer internet.

4:36 And, you know, you get thousands and thousands of errors a day on an ordinary consumer internet website.

4:43 But I saw 98.6 getting.

4:47 single digits and double digits of errors a day or a week even and then investigating every single one of them because they were doing health care and they said we can't have you just can't have the application not work during an online medical visit and that really gave me a whole new appreciation for what operational quality was like when when I went into my first operational review and there were eight errors the previous week and every one of them had had a post-mortem on them

5:15 I was like,

5:15 whoa,

5:16 that totally blew my mind for what it meant to really deliver a high quality product.

5:21 So those were both, even though I was only at that company a short time, they were both really, those experiences were really influential on me.

5:27 I don't know what your guys experience like.

5:29 Start with you, Greg, since you're our guest here today.

5:31 Greg: Yeah,

5:31 I feel like when I joined,

5:34 there was.

5:36 The company was doing or starting through this transition.

5:39 I remember when I first joined, I was told they were reaching the point where they needed actual engineering managers,

5:48 right?

5:48 And then maybe even the next layer beyond that.

5:51 I can't remember exactly how large it was.

5:53 It was, it was.

5:55 tens of engineers right um or or thereabouts yeah

5:59 Brad: I think it was like about 150 total across

6:02 Greg: okay yeah

6:02 Brad: the work.

6:02 Greg: but it was at the point where cross engineering dependency management needed to be figured out right it was a bunch of teams that um each were organized to own a particular part of the product or part of the technology

6:20 But they were bumping into each other all the time. And so I think what I the thing I sort of learned there is, well,

6:29 AI renewed it is cross team dependency management is hard.

6:33 And what I learned is one thing that tends to work is just like brute force it.

6:40 And we I'm sure we can talk a bit a bit about that. There's no there's no.

6:44 There are some frameworks out there and some ways of thinking that can help like optimize the process a bit,

6:49 but there was nothing like

6:52 Every so often whatever your cadence was getting everybody in the room

6:57 All the team leaders and this walking through the list and being like,

7:01 okay

7:01 We're not gonna maybe work out all the dependencies,

7:04 but like what are the ones we need to get into? So that was one and then the other thing the other thing I learned

7:09 was the and I'm curious to get your perspective on this was there was sort of two centers of gravity within that healthcare company there was the engineering team technology organization and then the medical team and being able to work in an organization where

7:31 both were sort of co-equal and had very different processes and mentalities and a lot of ways cultures and the sort of things I were working on needed to build the tools to support the medical team and that was for me that was a big that was a big struggle trying to figure out the right leverage the pull or the right way to be influential on that

7:49 Brad: Yeah, that's so interesting.

7:54 There's a lot, there's a rich vein in both those topics,

7:56 Greg.

7:59 I'll kind of throw out my topics of where I feel like I learned. I learned so much at 98.6. It was, it was a very.

8:10 lesson rich environment let's say um i think that one of the things i would say i would categorize i'm going to say my two big learnings my first big learning was all around culture um i i felt like um 98.6 was one of the most intentional by far uh

8:34 cultures that I had ever been a part of like I had never I had never I'd never worked for a company that that had such a strong list of core values and such intentionality around hiring against those core values and such intentionality around speaking to those core values in in every minute and so and

8:56 There were a lot of good things that came out of that,

8:58 and there were a lot of bad things that came out of that.

9:01 And so I feel like one of the learnings that I got out of that is just going to be in the category called culture has so many downstream effects that you need to really be thoughtful about the culture you put in place.

9:21 Um,

9:21 and to Scott's point,

9:23 like, um,

9:23 that kind of like empowerment thing was an example of that, you know, but another really high level, high order bit that I would say is that, um,

9:30 there,

9:31 a lot of energy was put into the culture around,

9:34 um,

9:35 In the hiring process around filtering out sort of like programmerness or like kind of that kind of toxicity that you've seen in so many engineering companies where it's like, you know, this real like, you know, alpha hard push type thing and that got filtered out really well.

9:56 And a really strange unintended consequence of that is that also happened to filter out bravery and willingness to.

10:09 uh sort of like speak up

10:10 Greg: Yeah.

10:11 Brad: and uh speak out and so that's that's a that was a thing that I learned from that process now I don't have any I don't know that I got any great answers out of what to do instead but that was definitely like a thing that I learned from that

10:27 The other big category I would say that I learned a ton from, and actually I learned a lot of this once Greg joined the organization because Greg brought a bunch of skills and approaches to this problem that I hadn't personally hadn't seen such strong leadership in before Greg came and kind of did it. And that was sort of like solving the problem of how do we have operational excellence.

10:52 uh,

10:52 while dealing with technical debt. And I think that one of the, so for example, when I joined,

10:59 um,

11:00 they had a, uh, the,

11:02 the product had missed, uh,

11:04 the uptime SLA for five quarters in a row, uh,

11:09 when I got there and, um,

11:11 and

11:13 it was because of technical debt and problems associated with technical debt and sort of like turning that into like just going from the we're investigating these incidents into and now what do we do with that?

11:28 I learned a lot about the processes and skills to solve that problem.

11:34 How many companies do you think actually have?

11:38 and uptime SLA that they track for five quarters in a row.

11:43 I would assume many

11:45 Greg: Yeah,

11:45 Brad: do.

11:45 Greg: I would think many do.

11:47 I think that's

11:51 one of the relatively straightforward metrics to kind of measure.

11:55 I feel like your system's out of the box.

11:57 Like AWS dashboard will tell you your uptime.

12:00 And there's also a bit of a...

12:04 I don't know if meme is the right word,

12:05 but like five nines is a thing.

12:07 And if you can't do five nines, you can do four nines.

12:09 If you can't do four nines, you do three nines.

12:12 Brad: Greg and I worked at a company 20 years ago together that was claimed five nines. That was a revenue science.

12:20 Greg: Yeah.

12:21 Brad: We did the five nines on one of the earliest versions of behavioral ad targeting.

12:25 I think more of what I was trying to make is I don't know that most companies set that standard.

12:30 They kind of react to it.

12:32 If systems are down a lot, they say this is the problem.

12:35 We need to get it back up.

12:36 But I do remember we had contractual obligations at 98

12:40 Greg: That's

12:40 Brad: .6

12:40 Greg: right.

12:40 Brad: for certain uptimes. So we paid a lot of attention to it.

12:44 I'm curious to dig in more. Greg,

12:47 you can tell us about how did you handle that situation of.

12:51 dealing with these stability issues through the tech debt and

12:54 Greg: Well, four

12:55 Brad: why did 98.6 which is only four years old have tech debt

13:00 Greg: years old is plenty of time to accumulate a good amount of debt,

13:05 right?

13:07 I think the solution,

13:09 there's layers to this, and I think you have to decide.

13:13 where on the operational or team maturity you are on this scale,

13:19 like do you have these, I mean Scott to your kind of gets to the point of your question,

13:24 A,

13:25 do you have these metrics,

13:26 B,

13:28 have you told your engineering team they matter,

13:30 right,

13:30 and then do you have the sort of observability in place to know how to debug where to debug with something.

13:39 Something like that starts to go from green to yellow to red.

13:43 Brad: Yeah, and I think that like.

13:45 It's interesting that Scott,

13:47 you said that you felt like when you got there, there was sort of like, you know, in the operational review,

13:51 it was like any any error that happened, there was a root cause analysis of it.

13:56 One of the things I noticed was that we we had the process that they had there was they had team members were on call and they would rotate on call.

14:11 And when incidents happened,

14:14 what it almost always,

14:16 at least when I first got there in the first couple quarters of being there,

14:20 it was a very much a put out the fire, not a.

14:27 figure out why the fire started and move the kindling and the gasoline away from the open stove.

14:36 So it was all putting out fires and it wasn't I didn't see a lot of progress being made against really coming back and saying, well,

14:44 this is the root cause and therefore we should invest in fixing the root cause.

14:50 Greg: Yeah,

14:50 that's what I sort of recall too,

14:51 like the post-mortem process.

14:53 process was it existed but I'm not sure to what extent like engineering teams were putting time into doing the report doing the review either with us or with somebody else understand what was sort of gone it was sort of okay that fires out back to the backlog that the product manager wants me to

15:17 Chase down.

15:18 Brad: Yeah.

15:18 And that's kind of the key insight that I feel like I give you credit, Greg,

15:22 for bringing to this. And I was like, after seeing you implement this,

15:27 I was just like, oh, this is I am going to use this from now on in the future. And that is, OK,

15:32 we're going to do the RCA and then what we're going to do is we're actually going to have like a leadership team meeting that's a recurring.

15:39 No-shit-it's-going-to-happen meeting where we're going to walk through the RCAs and we're actually going to ask ourselves,

15:45 okay,

15:45 what is the technical issue that this RCA is calling out and how likely is it to happen again and therefore we are going to put fixing this issue actually on the schedule.

16:00 Right. And when I say it like that, it sounds super simple. Like, of course, why wouldn't you do that?

16:05 But it was literally like scheduling the meeting where director level leaders would come in and we would all like we're all sitting there holding each

16:15 Yep.

16:15 other accountable to be like we read the RCAs and this is what it said. The database is three versions past, you know, the latest version or whatever.

16:26 Greg: with the third party off system is is super flaky right and like and then what i mean by looking at the maturity of the team i feel like there's a there's the next couple level ups there the next couple level ups is okay

16:40 over time are you seeing a pattern here and is there a is there a way for you to look at that as a leadership team i know in the previous podcast scott you had talked about like creating like a tech debt backlog right um and understanding that the process for looking at okay what's what's been failing and let's make sure we have a more longitudinal plan for getting that done and then the next one from there is like

17:05 Yeah, dig into the technical things that are going wrong,

17:07 but maybe your problem is not technical.

17:11 Maybe we can go through the list. Maybe it could be something about the skills of the team.

17:20 Maybe it could be about the nature of the business.

17:24 Maybe it can be about, you mentioned culture.

17:27 Maybe there's something in the culture that you need to fix.

17:30 understanding those.

17:32 Brad: So can I ask more about this meeting and this process that you set up?

17:35 Greg: Sure.

17:36 Brad: Who was in that meeting?

17:37 Who were the people? Was it with product management in that meeting?

17:40 I guess I have a few questions.

17:41 My first question was who is in the meeting?

17:42 My second question is you're going to put stuff on a team schedule,

17:46 which is inevitably going to be adversarial with the product backlog,

17:49 right? It's a zero-sum game.

17:51 There's only one team, one set of people on the team that can only do so much.

17:54 How did you navigate that piece?

17:56 I guess the two questions I have.

17:58 Greg: Yeah.

17:58 So as an engineering leader,

18:02 you, I set clear expectations.

18:06 Like these are, these are invariants.

18:08 We talked about five

18:10 Brad: Mm-hmm.

18:10 Greg: nines, like that's an, that's an invariant.

18:14 So it doesn't mean it's, it's a non-starter to, I mean, you're always going to have those conversations, I think when business starts ended up in a rough shape and you need to make trade-offs, but.

18:27 The outcomes and the to-dos from a post-mortem are priority zero,

18:33 and you can put an SLA around that.

18:36 Like these issues need to be resolved in two weeks,

18:40 or else what is what you need to figure out.

18:44 Like everyone gets six demerits or something like that, right?

18:48 But that's sort of how this works.

18:51 Brad: Yeah,

18:51 and I think that the...

18:53 The key insight of the let's have a meeting where once a month, I think, was what we scheduled. And it was, so to answer your question directly, Scott,

19:02 my recollection was we had, it was, you know, sort of me as the VP,

19:05 you know, or in this case, I was the senior most engineering leader.

19:12 the next my my staff basically so the directors um and uh the engineering directors and um and i think that we that meeting did include the the product uh like the senior most product people in that case they were i think their their level was director at that point not a vp but um and you know they would come um and

19:39 I think that I don't know that it ever necessarily became that adversarial, although kind of getting engagement with them about the importance of quality,

19:49 that was something that was kind of an ongoing thing that we were working on.

19:54 And to Greg's point,

19:55 it was like the or else demerits.

19:59 Greg: where Brad had set the tone that we didn't want to show up next month and not had made progress against these things.

20:07 Brad: Correct.

20:08 That's right.

20:09 Greg: And so it was just that recurring accountability that we're all going to sit in a room with a guy who has resting serious face.

20:17 Brad: What I meant by adversarial is not necessarily adversarial between the people,

20:21 but.

20:23 You know,

20:24 product managers typically want engineering teams to spend as much time as possible working on their product backlogs.

20:31 And that's a zero sum game with then issues like this, which I suppose you could say is in the sort of agile comes out of the agile,

20:39 right? This is just deferred work from some previous product backlog work that didn't happen.

20:45 Did you have an agreement up front with those product managers?

20:48 For sure.

20:49 I feel like there's a lot of engineering judgment that needs to be applied during the postmortem, right? Because you can end up in situations where the action items out of a postmortem can be, you know, in certain cultures it could be, ah.

21:07 well if it's an if it's a if it's a post-mortem action item and it must be done then i'm going to stuff it with stuff i want to get i want to get done right um or the other culture side of the culture can be like i'm going to i'm going to minimize the amount of things i put on that list of things that need to get done because i got i got a product manager that's breathing down my neck so there's just i don't know if i have any magic spell for that other than you have to apply

21:34 The judgment there that, yes, this went down, there should have been a unit, there should have been unit test coverage there.

21:43 Okay,

21:44 that seems pretty straightforward and light and as the engineering leader in that meeting,

21:50 yeah,

21:51 I hereby bless that and stamp that as a postmortem action item that should take priority over getting the product backlog done.

21:58 And then I alluded to, you know, when Brad and I were there, we encountered a situation where we were getting a lot of outages because our third party authentication system was flaky.

22:11 And that's not so simply declared a postmortem action item just to like rip out your auth system. So that one's like,

22:20 okay, cool,

22:20 that needs to, you know.

22:23 be put in the hopper for a bigger conversation at the leadership level including product managers around are we are we going to do this or not

22:30 Greg: Yeah, and that was a good example where to Greg's recollection of the story,

22:35 like this was a third party off system.

22:38 It was the source of many of our missed quarterly SLAs and kind of.

22:48 The idea of like instead of just having the RCA that goes,

22:51 yep,

22:52 third party auth system failed.

22:53 That's what happened here to actually like have that catalog and go,

22:57 it has happened,

22:59 you know, dozens of time and look at the actual like impact on our on our contracts because of it.

23:07 And now we were able to like surface that and go, look, this is going to keep happening until we invest in this.

23:14 effort and and you know kind of the the

23:20 Going from the mindset of, oh,

23:23 errors happen,

23:24 we put out the fire, and we document that the fire happened,

23:27 to a mindset of, we're going to look at all the fires that happen, and we're going to start to look for common themes,

23:34 and we're going to start to invest in the tech debt creating those common themes.

23:39 That mindset shift allowed us to then successfully have the conversation with the senior product team to say,

23:47 We need to invest in this.

23:49 This is a huge red flag here that we have to fix this.

23:56 Our product is useless if people can't log into it.

24:00 Right.

24:00 Brad: Yeah.

24:01 Greg: And it took that evidence and it took that sort of like process, that additional process was the key to unlocking fixing something that then was a major,

24:15 you know, a blessing to the engineering team because like now all of a sudden we didn't regularly have fires on that part.

24:25 Brad: One quick Dropbox story that I think about a lot was I was in a series of these postmortems and I was detecting this pattern and the pattern was for some of the Dropbox website that isn't heavily trafficked.

24:44 There would be a code change that would, when deployed,

24:47 the page just didn't work.

24:49 Like it would immediately and entirely 100% of the time give you a 503,

24:53 right?

24:57 I was like, okay, there's something going on here,

25:00 and obviously when you dig in, it's not as simple as you think.

25:04 It's about the different environmental and experimentational setup between the production environment and your test environments.

25:13 So I gathered up the staff engineers,

25:16 and I said, look.

25:18 let's do this like meta postmortem every quarter where I want you to look at all the sabs and all the problems and I want you to come back to me with your top three and I'm going to get those things we're going to get those things done and they did that and when they came back they were like

25:38 Well,

25:38 Greg,

25:38 if you really want to do something that's going to reduce our quality risk,

25:43 it has nothing to do with these salves. It was, and I don't know if I want to mention it on the podcast,

25:49 but it was like some very dangerous activities engineers were doing to get customers back in shape.

25:56 That's the thing you should fix.

25:58 And that's when a light bulb.

26:00 lit up like okay you just need to there's a lot of value in just creating the space to have the conversations around the engineers that know what's going on

26:09 This will grow.

26:09 Greg: So what you're saying, Greg,

26:11 just to clarify,

26:12 not to intentionally completely air the dirty laundry,

26:15 but you're saying that like one,

26:18 you implemented a meta-analysis of the RCAs. Like that's essentially the same playbook that we ran,

26:24 you and I ran at 98.6. So you implement a meta-analysis and then it happened to be that the meta-analysis came back and it was essentially a,

26:32 I'm going to put this in the category of a culture.

26:35 culture thing because it was you had essentially cowboy ops or cowboy dev

26:39 Brad: Right.

26:39 Greg: happening um in in that was

26:42 Brad: Well,

26:42 Greg: what they were the doing

26:42 Brad: eye opening

26:43 Greg: yeah

26:43 Brad: thing was, it was something I was not even aware that was happening.

26:47 Right.

26:47 Greg: right

26:48 Brad: And that had me reflecting like,

26:51 okay,

26:51 why did I?

26:53 Why did I not know that was happening?

26:55 Right?

26:55 You think you have these processes in place that will cover these sorts of things,

26:59 but

26:59 Um,

27:00 you gotta, you know, open up the space to have your engineers kind of lift up these risks to you in a way that's, that's, that's structured and you get the visibility in that, even to that sort of stuff.

27:11 Greg: You know, it's interesting.

27:12 I don't know if this is, this makes me think about another thing that I learned from.

27:19 my time at 98.6 and again like super talented team and i have a ton of respect for the folks there and they worked really hard um and so uh i don't mean this as a judgment anyway but but one thing i noticed was there was a a strong the language around blameless retro had this um

27:44 I think unintended side effect where and I've stopped in my current team, I don't use the term blameless retro.

27:51 I use the term egoless retro instead.

27:54 And the reason for that is that they they would not put people's names on any of the timeline of what happened.

28:03 They would basically try to anonymize.

28:08 the TikTok around understanding what happened. And what I noticed was that makes it really hard to have a...

28:18 Like, well,

28:19 what really happened here?

28:20 Like, no one's going to get fired over this. Like, it's OK that Brad,

28:23 you know, logged into the terminal and started like, you know, deleting rows out of the database. I need to know that that happened,

28:32 you know, like and I'm just kind of curious,

28:36 Greg,

28:36 do you feel like when you say like creating the space,

28:40 like how do we get?

28:43 How common, I feel like it's pretty common for people to be afraid of kind of really laying out what happened in and it's and how do you create the culture so that people are not afraid of really talking about what happened.

29:00 Brad: That's a great question.

29:01 I think that's probably a specific question around the general question of like building culture.

29:06 Like I think.

29:08 I think it's top down and bottom up.

29:12 Like it's easy for top down leaders to completely break a culture in that sort of environment where something like that happens and you start pointing the fingers or you start digging in there.

29:26 And then bottom up in that.

29:29 like you have to make sure that culture and that expectation is set before you get into that post-mortem meeting because at the point at which something has failed and things are a little bit radioactive you're and you don't have those expectations set that it's going to be okay then you're going to have a bad time

29:47 A language I used around that.

29:51 So there's a lot of language I use. I think you talk about it all the time. You use the word blameless. I've never used egoist, but I kind of like it.

29:58 I would continuously emphasize we are here in this room to come up with action items and working agreements to avoid this mistake in the future.

30:10 We've all made mistakes.

30:12 whoever made this mistake they know it they're not happy about they're unhappy with themselves about it like we're not here to pick on greg for fat fingering something in the terminal we're here to figure out how we can make it so that that the opportunity for greg to fat finger something doesn't even exist any longer either because we've got

30:31 We got guardrails around it or automation around it or whatever.

30:34 I came back to that language, action items and working agreements.

30:37 That's the outcome we're looking for,

30:39 not who screwed up, but what can we do to avoid it or minimize the impact of suppose the last radius.

30:50 So.

30:53 You wrote a blog post, which I think is relevant to this part of the conversation about things to do early at a startup to make it easier in the future.

31:07 Yeah.

31:09 I did.

31:12 These were just based on sort of observations I

31:15 I had I've seen the opposite as well where early startups decided early startups populated by engineers that had seen a thing or two at larger companies and decided to kind of over architect the scalability of it ends up slowing them down I think maybe we've all you're laughing and maybe you've seen some of that some of that too but

31:36 Yeah.

31:36 in a couple

31:38 In a couple places I've seen engineering and engineering leadership lay down certain expectations around how certain standards will be put in place.

31:52 One that sort of struck me, I didn't kind of realize it was too late at a place I worked was performance,

32:02 like product latency and speed.

32:05 Even just saying like,

32:06 okay, we expect web pages to respond in two seconds,

32:12 whatever that happens to be in certain percentiles and certain measurements of that, or backend services need to respond in 100 milliseconds or things like this.

32:22 It set the expectation. And what happens if they don't, like you can decide and maybe you crank that down or up over time,

32:30 but just setting those sort of expectations.

32:33 means that, okay, when I am working with my team to decide how I'm going to meet the requirements from my product manager,

32:43 it's non-negotiable that it is this performant, right?

32:52 and then the flip side is you know i would talk to teams that didn't have those sort of standards and they'd say well greg if you if you wanted it to be that fast we never would have built it this way in the first place and that's a right and that's like okay oi that that seemed like a conversation we should have had to

33:10 Greg: Yeah.

33:10 Brad: had earlier and so that that that's that's one that's one example and there's other examples of like i mean we had talked about reliability and

33:18 setting however many number of nines and

33:21 Mm-hmm.

33:22 some other engineering standards you can put into place performance in my experience has been one where it's always hard to kind of retrofit performance even reliability and uptime like it sucks that you have to go backtrack a little bit but it's it when you talk to your engineering teams about what it would take to get reliability up

33:44 It tends to be just work, right?

33:47 Yeah.

33:48 But then performance seems like, oh shit,

33:51 like, this wasn't, I turned, it turns out I built a minivan,

33:58 right?

34:00 Which is what I thought we wanted.

34:03 That makes it hard.

34:05 If you were starting.

34:09 a new company or new building a new application new web app today new online service and what are the top of those things that you could put in place early that makes life easier down the road what are the top three what's the top one

34:25 I mean, the things I sort of written about were like expectations and standards.

34:28 So like I would, I would.

34:31 I mean, I have a list on the blog.

34:33 Performance is towards the top of the list. I think analytics,

34:36 right,

34:37 understanding customer behavior.

34:39 And by that, I sort of be,

34:40 so a lot of these frameworks you can kind of get off the shelf have like exception logging and, you know, error reporting.

34:48 And depending on how big you are,

34:50 you can start with grep as your tool and work yourself up from there.

34:55 By customer analytics, I mean like who's using your website.

34:58 Like the data you would need for a project manager to be able to make a decision.

35:04 like a Google Analytics or Amplitude or Mixpanel kind of thing.

35:08 Exactly,

35:09 exactly.

35:10 Where does test automation fall in your framework?

35:13 Test automation is on that list.

35:16 And by test automation,

35:18 what I've seen is...

35:23 Engineers have this pre-installed expectation that they're going to write unit tests or they're going to write end-to-end tests.

35:31 And then over time what ends up happening if you're not careful is your build system starts to slow.

35:40 It starts to become flaky.

35:43 right your build time goes from 10 minutes to 30 minutes to hours and then you sort of reach a tipping point where at least I've seen this in a couple places I've worked where the the CICD system takes so much time it actually starts changing engineering behaviors around how they write code right because if it's going to take three hours for the build to run and likely to flake out

36:08 then I'm not going to do the quick iterative development process I'm going to batch up and so it's not just yes you should have unit tests or you should have automated testing but like what what are the what are the boundaries of those things like are you going to set an SLA for your build and if it goes beyond that sort of amount of time or the amount of resources the amount of unreliability that you're going to

36:33 Push the big red button and do something about it.

36:36 Yeah, we did that at Rover where our target was 10 minutes and when it got to 20 minutes, then we took action.

36:44 And I would say about every six months,

36:46 we had to have someone spend some substantial amount of time working on it to figure out what the next iteration of making it go faster was.

36:57 Yeah, and at Dropbox, we had reached a point where our build times were really quite slow and it was starting to change the way sprints were run and the way engineers would calculate,

37:07 you know, the work they could get done in a certain iteration.

37:11 And we ended up doing a little project to sort of analyze our end-to-end testing.

37:20 right um uh like what was going on there why was it flaky and there was a bunch of infrastructures things like you know you at that particular scale in order to do sort of a an end-to-end test you needed automation that would

37:34 build an entire mini version of the Dropbox infrastructure just so you can run a test right and that that took some real infrastructure work to kind of work out but then we just sort of did an audit of these tests like what are what are these tests I feel like engineers they look at the code they're building to build the product and

37:52 there's this notion that you want to refactor and re-architect parts of the product but it's not really natural for you to do that with your test with your

38:01 Greg: Yeah.

38:01 Brad: test and

38:01 Greg: How often do you go back and refactor your tests?

38:04 Brad: what we found what we found is like a third of the tests we could just not run like they were they weren't useless but like they were covered by other tests and then like another i don't know 15 of the tests

38:18 were not not critical and so it could be run on a different cadence right not not not for every not for every sort of um code change and that like cut that part of the test infrastructure in half in terms of time which was which was huge

38:33 Greg: You know,

38:34 I'm curious about what you think about this, Greg.

38:36 We ran into this, you and I ran into this at 98.6. And it's so what I learned from that, I added to my list of like,

38:44 every new team, I want to solve this problem early.

38:48 Brad: Yeah.

38:49 Greg: And that is the ability for like,

38:53 Out of the gate, do you have an architecture and a plan that allows individual developers to actually run?

39:04 the system before making a pr you know like do you have your as your like i almost sort of it's like uh what is it like design for manufacturing where people like kind of think about the design of their physical components so that it can be manufactured i feel like uh design for debugging or design for individual developers to be able to

39:29 to run the stack when they're making code changes.

39:33 I feel like get that done early or else you can never fix it later.

39:39 Brad: Yeah, for sure.

39:40 And being able to run the test locally.

39:43 Greg: Yeah.

39:43 Brad: For sure.

39:44 Greg, you talked about how the build system can change engineers'

39:51 behaviors.

39:52 So can the local development environment.

39:55 Something I've seen recently.

39:57 Is engineers having trouble?

39:59 Greg: running the what Brad was saying having trouble running the whole system locally having trouble running the tests locally so then they want a staging environment to do all that on and then you've just the cycle time becomes really long all because we're not taking the time to invest in actually just getting it to work on the desktop

40:22 Brad: Well, and even if you can't get it, there might be reasons why, you know, some parts of your system can't practically run on every single desktop.

40:31 At least then make sure you have.

40:34 ephemeral environments that are dedicated to the engineer right so that it's not like we're just throwing it into dev before i can do any testing

40:43 Greg: Mm

40:44 Brad: right

40:44 Greg: -hmm.

40:44 Brad: like that's that's what i saw i've seen that with teams when it's like there's no practical way for me to test this except for go ahead and commit it and check it into dev and it's in the ci it's been it's it's in the ci cd and it's sitting there in dev and i've you know i'm going to feature flag around it to test it or something

41:01 something

41:01 Yep,

41:01 you

41:01 I

41:01 know but like that code is in there it has polluted your main line at that point and if that's your only way to test your feature like talk about like super long cycle times right that's that's horrible

41:18 Greg: Have either of you worked with GitHub Codespaces?

41:22 Brad: have not.

41:24 Greg: Dying to meet somebody who's actually used them.

41:27 They seem like the answer to so many of these problems,

41:30 and yet I have not come across a single person who's used them.

41:33 Brad: You asked me this in episode one.

41:35 I still haven't researched it because I have a system that's working like the local devs can work in the local environment and then they can also push to an ephemeral environment.

41:45 And man, those two things right there are so powerful that.

41:49 Greg: When you say ephemeral environment,

41:51 each developer has their own environment.

41:53 Isn't that right?

41:53 Brad: yes yeah

41:54 Greg: Their own staging

41:55 Brad: i've

41:55 Greg: environment.

41:55 Brad: seen that too yeah

41:56 Greg: Well, that works.

41:57 Brad: or even like a you could even do it with off a pull request like every pull request gets its own environment right so then a developer could have multiple things in flight if they wanted and actually like

42:07 Go check it out over here. Go check it out over there.

42:09 Greg: I think that's how Codespaces works.

42:11 But there's a scale and cost problem there,

42:14 right,

42:15 which is if you're setting up a staging environment for everybody,

42:18 it starts to get very expensive really quick because those things are not in use very frequently.

42:23 I might use it four minutes a day to test something.

42:27 I guess if you shut it down, if you have a way to shut it down and ramp it up, you save money.

42:32 Scott: Yeah.

42:33 I mean, places like Dropbox had something similar to that where it would fire up the right number of instance or an instance in AWS,

42:42 stand up a mini environment,

42:45 populate it with mock data, and then it would persist for some period of time and then be torn down when the developer was done with it.

42:57 Greg: There must be third-party solutions to do that.

43:01 I can imagine saying that to some sort of cloud DevOps-y type engineer and they go script the whole thing out and build it all their own.

43:07 There must be solutions for that.

43:10 Brad: Well, my point is.

43:13 make sure you have it out of the gate like don't don't wait till it's too late and you've got you know 100 devs and they're all sitting there pulling their hair out because the only way for them to test their stuff is to you know

43:24 Scott: Push

43:25 Brad: deploy

43:25 Scott: it.

43:25 Brad: to

43:25 Scott: Yeah

43:25 Brad: dev to push it you know that that's um that's my point is like i'm i'm sure there's we have a solution that's working for us it's not that expensive code spaces might be a great answer for um

43:38 you know other folks but yeah i

43:40 Greg: And you build that whole system yourself or did you use some sort of third party solution for it?

43:46 Brad: mean it's not it i mean it's just kind of like you know whatever your standard infrastructure as code type stuff like terraform or whatever and you know you can usually just pop up an instance pretty quickly it's not free to build it like you have to build it but it's mostly just like you know standing up a

44:04 Whatever ECS based on a terraform

44:07 Greg: That engineer has like a command line tool they can do that from their desktops. They fire up my got

44:13 Brad: We have it wired up into github actions,

44:15 so it's just like boom you open a pull request and deploys to your stack and

44:20 Greg: it cool sounds

44:22 Brad: Yeah

44:22 Greg: like you have a product there

44:26 Brad: I am I gonna go that far

44:31 This is so much fun, Greg.

44:32 It was great to reconnect with you and kind of go through some of these things.

44:38 What do you, how would you put a bow on this conversation?

44:43 How would you wrap this up?

44:44 Scott: Oh my gosh,

44:45 how would I wrap this up? I mean, we talked about a bunch of different things.

44:49 I think overall,

44:51 it's about making sure as an engineering leader,

44:55 you're setting clear.

44:57 expectations around where the where the guardrails are like where where are the guardrails um i think as an engineering leader you want you want to see your team building a bunch of stuff shipping a bunch of stuff getting it in front of customers having the customers be super happy but you you need guard these guardrails at different scales those guardrails could be very different but even at small teams

45:23 early stage it's not it's not terribly hard to just start setting up guardrails to make sure that like okay well when you crash into this guardrail like at the very least i'm going to get a slack message from you um and then we can and then we can decide and then we can decide as a team what we're going to do about it

45:40 Brad: Yeah,

45:41 I mean, I think that, you know, my summary is that.

45:45 Sort of like the seeds that you plant early on in the organization,

45:50 whether those seeds are cultural seeds or technology seeds or process seeds,

45:56 they grow and they result in a garden that is in some shape,

46:04 you know, and it's a lot easier to be planful about.

46:11 the cg plant then to come back later and try to weed it you know something's

46:16 Scott: yeah something's going to grow something's going to grow there what

46:19 Brad: growing

46:21 Greg: That's a great way to say it.

46:22 Something's going to grow there.

46:24 Scott: are the what are the things i when i do skip levels or i'm talking with staff or principal engineers or even i'm talking to engineering manager about their team i want to know if they're are they proud of are they proud of what they're doing are they are they proud of the work

46:39 that's like a high level because if there's quality problems if they're always dealing with 2am pages they're not they're not proud of that they're not proud of that right yeah

46:49 Brad: Yeah.

46:50 Greg: Great question to ask people.

46:51 I love it.

46:52 Brad: Yeah.

46:54 Scott: really great this has been great

46:55 Greg: Cool,

46:55 Scott: catch up

46:56 Greg: man.

46:56 Scott: really enjoyed the conversation this is awesome

46:58 Greg: Before you go, Greg,

47:00 where is your blog so everyone can find it?

47:02 Scott: gregconklin.com easy

47:05 Greg: Easy peasy.

47:06 Brad: Nice.

47:06 Scott: peasy yeah

47:07 Brad: We'll add that to the show notes for sure.

47:09 Scott: awesome well thank you two really appreciate it

47:11 Greg: All right. Thank you.

47:12 Scott: yep bye

47:15 Brad: This episode of How Many CTOs Does it Take Podcast was edited by Saffron Heppigow.

47:20 Our theme song was AI generated by Brad.

47:23 Please follow us on all the social media platforms at at HowManyCTOsPod.

47:28 You can find us on YouTube,

47:30 LinkedIn,

47:31 Instagram,

47:31 Facebook,

47:32 Threads, X, TikTok,

47:34 and Patreon,

47:34 or at our website at HowManyCTOsPod.com.

47:39 And remember to rate, review, and subscribe on the podcast platform of your choice.

47:45 Thanks for listening.