Preparing for the Worst: Tabletop Exercises, Unit Tests, and Everything In Between

0:00 Scott: That exercise,

0:01 the hygiene exercise,

0:02 is valuable.

0:19 Hello, Brad.

0:22 Hey,

0:22 Brad: Hey, Scott.

0:22 Scott: welcome to this week's episode of...

0:25 How many CTOs does it take to put in a light bulb?

0:29 My name is Scott Porad.

0:30 This is my trusty co-host.

0:32 Brad: Brad Hefta-Gaub.

0:33 Scott: How are you doing, Brad?

0:34 Brad: I'm doing good. Actually, it's funny you say how many CTOs does it take to put in a light bulb because my light bulbs died earlier this summer and the ceiling is really tall. I got to go get an extra tall ladder to put the light bulbs in and it's starting to get darker.

0:49 You know, it's starting to stay darker later,

0:51 Scott: Mm

0:52 Brad: get

0:52 Scott: -hmm.

0:52 Brad: darker earlier,

0:53 and I'm going to have to fix these light bulbs.

0:55 It's not summer anymore.

0:56 It's not summer anymore.

0:58 Scott: It's not summer anymore.

0:59 Or you can get one of those sticks with the thing on the top that you don't have

1:03 Brad: Fair

1:03 Scott: to get a ladder.

1:03 Brad: enough, because it's got like a glass thing on it. I got to take the glass thing off before I can fix the light bulbs.

1:08 It's going to be a whole thing.

1:09 Scott: It's going to be a whole thing.

1:12 We did a whole thing at TaskRabbit last night.

1:14 Brad: Yeah.

1:15 Scott: If any of the TaskRabbit tiers are working there or working there listening, they'll know about this.

1:22 We did an AWS region move.

1:25 Brad: This is

1:26 Scott: In

1:26 Brad: South Park.

1:26 Scott: US West 1 to US West 2,

1:29 I think it is well known that US West 1 is kind of like a second class citizen in the AWS world.

1:37 It has fewer features. It costs more.

1:40 It's less of everything. So we did this migration.

1:44 It's a big project.

1:45 A big team of people are involved in it.

1:47 They did a really good job. Their attention to detail was really good.

1:49 There's just been a few.

1:50 two little incidents this morning that we've had to clean up one thing didn't get whitelisted the IP properly etc it's overall working really well the reason the reason I'm bringing this up is twofold the first is

2:03 Well, let me start at the end. It made me think we should move regions every year.

2:07 Brad: just to make sure you got stuff buttoned up, right?

2:11 Scott: Yeah,

2:11 it forced us to really get our stuff buttoned up, to understand how our system works.

2:17 And this company, I don't know that this company had ever moved regions since it went on to AWS.

2:21 And so the fact that it just took a ton of work to even just figure out everything that was out there that we had to take care of and how it was all configured and all the little.

2:28 little you know extra stuff and now it's all pristine and clean and neat and tidy right and so it's like this amazing housekeeping exercise for just like getting stuff in order and you'll learn a ton about

2:41 your systems you know as you have systems that are mature you just have turnover and institutional knowledge gets

2:47 Brad: Yeah.

2:47 Scott: lost and so it forces us to go learn a bunch of stuff which was really cool and it's I just thought and it went really smoothly and kudos to the team and I just thought oh we should do this like every other every year every other year just so that we're just cleaning out the house

2:59 Brad: But yeah,

3:00 but I think you mentioned to me before we hit the record button that there was like a 33 page runbook and you had like 25 plus people on the call.

3:06 And so.

3:07 You might get

3:08 Scott: Yeah,

3:08 Brad: a little bit

3:08 Scott: so.

3:08 Brad: of static if you want to do that too frequently.

3:12 Scott: Well, like I said, the company had never done this before.

3:16 So a number of people,

3:17 about seven to 10 people had been working on this for a few months to prepare.

3:21 They had built a very detailed runbook of everything that needed to happen.

3:24 They had practiced it a few times. And then we went through it last night and they had practiced rollback scenarios at each different spot. If there was a failure at this spot, what the rollback scenario would be, et cetera, et cetera.

3:37 And then we had one person from every other engineering team involved.

3:44 And there were a lot of those engineers,

3:45 not a lot, but people were like, well, why do I need to be on this call?

3:48 Like on call, can't you just call me?

3:50 And we were like, no,

3:51 we just want everyone there.

3:52 Like we just want this to go as smoothly as possible.

3:54 And maybe you won't pick up the phone in the middle of the night or something like that. And let's just have everyone on the call.

3:59 You know, you can expense pizza or whatever you want to do at your house. And it went well.

4:03 I just thought, you know, that exercise,

4:07 the hygiene exercise is valuable.

4:10 Brad: Totally. It's kind of like doing the recurring tabletop exercise where you.

4:17 I mean, yours was more, it was real.

4:18 It was like,

4:19 Scott: It

4:19 Brad: you

4:19 Scott: was real.

4:20 Brad: were really doing it. It wasn't a tabletop, but, you know, you still can learn a lot by just recurring, scheduling those recurring,

4:27 even if you do it as role play, you can often like discover something,

4:31 you know, that

4:32 Scott: That's

4:33 Brad: you didn't

4:33 Scott: right.

4:33 Brad: think of.

4:34 Scott: This was very helpful because one of the things we want to do in the

4:37 next 12 months is improve our disaster recovery and business continuity processes that's a big step for us not a big step but an important thing we want to do and this really helped with that because it helped us know our system so that when we go to say let's what would it be like if

4:54 U.S. West 2 disappeared

4:57 How we handle that.

4:58 Brad: Totally, totally.

4:59 And that's obviously like, you know, at the scale of TaskRabbit, that's like a very real scale,

5:04 very real complex systems. And like you said, you hadn't done this in several years. So you had some, you know, kind of like the classic like it's documented somewhere, but we got to figure out where all the documentation is or figure out the as built if it wasn't documented.

5:20 Scott: Yeah, I don't

5:21 Brad: Yeah.

5:21 Scott: think it was documented before.

5:22 I have a question for you about tabletop exercises.

5:25 Brad: Yeah, tell me.

5:26 Scott: I'd like to know two things.

5:27 I'd like to know, well, first, let's explain. I'm going to interview you.

5:30 Brad,

5:31 Brad: Okay.

5:31 Scott: tell us what a tabletop exercise is for the 41 listeners in our audience.

5:35 I bet there's at least 37 of them who don't know.

5:37 And then tell us like your theory of developing a tabletop exercise and then tell us your top three ones that you like to do.

5:44 Brad: Okay.

5:45 Well, OK,

5:46 so tabletop exercise is where you essentially create a contained space, usually a meeting.

5:54 You might give your people a little bit of preparation to so that they know kind of maybe the context the tabletop exercise going to be in.

6:02 But then you basically present them with a scenario and in real time,

6:07 everybody around the room.

6:10 does their part to resolve the scenario.

6:16 Okay.

6:16 Scott: Mm-hmm.

6:17 Brad: And the main thing here is not that you get it right, but typically what you're going to do is you're going to document everything like super easy. If everything going over, you know, video call,

6:27 just click the record button on the thing and say,

6:29 okay,

6:29 here's our scenario.

6:31 And then you just start walking through it and say, what do you do? What are you going to do? What are you going to do? How would you, and then what you're looking for is.

6:40 You're essentially looking for gaps in processes or documentation where the person in the room goes,

6:48 oh,

6:49 I don't know what I would do now, right?

6:51 Or they might say, someone in the room might say,

6:54 well,

6:55 okay,

6:55 you know, the database got deleted.

6:59 I'm going to start to go,

7:00 I'm going to go get the, you know, the last snapshot and someone in the room might go.

7:07 well how did you know to do that he's like you know they might just be like well because i'm a dba i know that's what i'm going to do and then you'd say well great let's make sure we have it clearly documented if you were the only one that would have known to do that then we have a gap right so that's kind of the general idea of the way we do a tabletop and usually

7:25 the person who creates the tabletop comes up with a scenario and then they want to think through like and they'll usually have like a set of questions that they're ready to kind of prompt the team with as you go along to sort of ask you know ask probative questions along the way you know to kind of keep that keep that scenario rolling and then the most important part is after you're done with the whole scenario then you just do a

7:50 you know kind of do a hot wash and a where you're just like okay let's talk through all that happened here let's like debrief how we thought about it and let's come up with a punch list of gaps that we want to then commit to improving and fixing in our in our technology our processes or our culture yes

8:08 Scott: Well, it sounds to me like it is an incident response process drill.

8:13 And

8:13 Brad: exactly just. so so hopefully you've got your incident response

8:16 process that you've already documented and you're just following it.

8:20 Scott: And then it sounds like all good incident response processes, there should be a retrospective at the end to reflect upon both the incident and the process.

8:29 Brad: Yeah, exactly.

8:30 Now, one thing I think is where tabletops can go outside of your incident response process is

8:37 If your incident response process has been written by your technical team and is very maybe it's it's more technical in nature and maybe it included things like, oh,

8:47 this is when we contact legal counsel or this is when we contact the the the public information officer or the comms team,

8:55 you might have written that in your incident in your tip in your traditional tech focused incident response plan.

9:03 But a lot of organizations don't necessarily have that piece. And so sometimes you'll try to do tabletops where it's got nothing to do with technology.

9:10 Like, in fact, when I was at 98.6, I think this was maybe after you left and I was still there, I participated in a tabletop which was a business continuity tabletop around what happens if a key person.

9:28 leaves the organization like what happens if the ceo leaves the organization how do we handle that

9:33 Scott: You did this after I left because everybody was just, what are we going to do without Scott?

9:40 Brad: it may have been it may have been that may have been the impetus of it you know as far as you'd know

9:48 and so that was an that was an example of a tabletop that that leaned a little less technical but still like exposed a bunch of really interesting questions like oh how do we how do we handle that so you asked me what a tabletop was did i get a good answer for you does that make sense or what else did you want to ask me

10:02 Scott: I'll

10:02 Brad: about that

10:02 Scott: have to ask my mom when she listens to this podcast and see that that makes sense to you.

10:06 Let's see.

10:06 Well, then I was like,

10:07 okay, so you have to come up with a scenario when you're doing, you know, you're a fractional CTO,

10:11 you go in and you work with teams, you have to come up with a scenario to help them walk through this.

10:16 Any tips for someone out there on how to come up with a scenario?

10:20 Brad: Well,

10:20 I just did the most recent one that I did.

10:23 I did it because I was sort of like the scenario I came up with just seemed like really in vogue, right?

10:29 Like the scenario I came up with was.

10:31 The company has installed,

10:32 you know, at Claude into Slack and simultaneously hooked up Claude to the MCP server to some important,

10:40 you know, database and then an intern at Claude's to go clean up some data and tells it to make no mistakes and Claude decides to delete the database,

10:52 right?

10:52 Like that just seemed like a very like topical, like that's been in the news. I'll use that one as the scenario.

10:57 scenario and it

10:58 Scott: Mm-hmm.

10:59 Brad: turned out to be a really great exercise like the team that I put this together for they were like they actually did find a couple places where they had some gaps in their data recovery plan and they also it also turned into a really great conversation around oh how do we like we're getting so much power out of these tools we really don't want to slow people down so how do we how do we how do we manage governance of these agents

11:23 Um,

11:23 how do we manage the governance so that people can still move fast, but you know, not,

11:28 not just count on asking the agent to make no mistakes.

11:31 Scott: Yeah, I can think of three or four different scenarios worth coming back to over and over again.

11:39 I'll just rattle them off.

11:40 One is some key data has been deleted,

11:43 whether it's data in a database or media assets or something, something key has been deleted.

11:48 A secret.

11:49 or token has been leaked he you know whatever your hosting situation is ours is aws whatever has gone down and it's going to be down for a while you need to restore your system so a disaster recovery scenario and then the last one coming from personal experience your ceo gets an email and that email says

12:12 Here's a snapshot.

12:13 Here's a screenshot of some pictures of your database.

12:16 Brad: Yeah.

12:17 Scott: So send me some Bitcoin.

12:19 Brad: Yep.

12:19 That's a good one.

12:20 Scott: That was a day.

12:21 Brad: Yeah.

12:22 Scott: Totally. But what happens if that happens if somebody tells you they're inside your system?

12:27 Brad: Yeah, exactly.

12:28 Yeah.

12:29 And that's a really good one. I was going to say some kind of a scenarios that have some kind of a hack or exfiltration or some kind of a like anything that's like anything that's like an outside intruder in your system is a good one because it's going to it's going to push on a lot of different things. It's going to push on like,

12:46 well,

12:47 how do we how do we respond to that?

12:49 How do we handle that? Right. And that your your your Bitcoin ransom.

12:53 ransomware attack is a good example of that but then you also have to start challenging yourself well how would we did we did we miss it if we missed it why did we miss it and more importantly you know what do we what systems do we have in place to make sure that our monitoring and our safeguards are updated as we evolve our system right like like that's that's super common right like if you lock down your you've got a bunch of great monitoring you've got like

13:18 like you know burst monitoring and alarms that are going to go off but then you had some new component and you didn't monitor that one or you know so so how do you make sure that every new component is getting monitored correctly so there's all kinds of stuff that you can learn from these tabletop exercises and it's way better to do it

13:34 You know, not in the heat of battle.

13:35 Scott: For sure.

13:36 So here's something interesting.

13:38 We were talking about external penetration,

13:40 somebody getting into our systems.

13:41 A number of years ago, when I was sort of early in my information security type of experience, I was working with this guy, it's like a security guy,

13:49 and he was really pressing on the role based access control,

13:53 access control is access control. And I was like,

13:56 I can't remember his name, let's just say his name was Jeff.

13:59 I was like, finally,

14:00 I was like, Jeff,

14:01 I don't understand why you're, and this is after, you know, working with him for quite a while. I'm like, so I don't understand why you keep on pushing on the access controls. To me, it feels like the perimeter of the system is so much more important to harden than the access controls.

14:15 He said,

14:15 I totally get why you're thinking about that.

14:17 But most incidents happen from someone internally.

14:24 Or someone gets access to somebody's account internally and then they've

14:27 Brad: Yeah.

14:27 Scott: got unlimited control in there.

14:29 Brad: Yep.

14:29 Scott: And I was like, oh, that was a light bulb that went off. And then later in my career, probably five or 10 years later, I was involved in two,

14:36 not one, but two internal embezzlement incidents.

14:40 Brad: Yeah.

14:41 Yeah.

14:42 Scott: And we actually had an incident recent at TaskRabbit. It ended up being just fine.

14:48 We use,

14:49 we have.

14:50 We're gig marketplace,

14:51 so we collect money from clients who book jobs, then we pay it out to taskers who do the work and sitting in between is Stripe.

14:58 Brad: Mm

14:58 Scott: And

14:58 Brad: -hmm.

14:58 Scott: there was this incident we had where

15:01 A handful,

15:02 like, let's just say, you know, under a thousand people got paid twice.

15:06 Some bug paid people twice.

15:08 And so people are running around trying to fix this. And in my mind, my light bulb went off. I was like,

15:13 oh, that is a signal of an embezzlement kind of thing.

15:18 And this is something I had learned earlier in my career.

15:22 When you think you see something happening internally.

15:25 You have to treat everybody as a suspect,

15:27 right?

15:27 Brad: Yeah, sure.

15:29 Scott: And so you have to understand like,

15:30 okay, who is the one or two people that I'm going to start with and say, we need now I'm going to try who are the one or two people you're going to trust out of the gate,

15:38 right?

15:39 Brad: Right.

15:40 Scott: Yeah, right.

15:40 And say we need to actually quietly go look at some other things here.

15:45 We need the turtles all the way down. We need to investigate the investigators kind of thing.

15:50 So what I'm sharing all this just to say another type of tabletop which would be valuable to do is the internal actor because it forces you as an organization to say, well, wait a second if you're an internal actor.

16:02 I think you have an internal action,

16:03 you have to be much more suspicious,

16:05 but also careful about how you manage that incident.

16:08 Because

16:08 Brad: Yeah.

16:09 Scott: in one of those embezzlement incidents I was involved in,

16:11 the embezzler threw the incident responders off the scent.

16:18 The embezzler got involved in the incident response.

16:22 Brad: Oh, wow.

16:24 Scott: Yeah.

16:24 Brad: That's wild.

16:24 Scott: And so but we did we didn't suspect that that was the issue. So we didn't have any reason that we were we were being clueless about it. And so we didn't have any reason to suspect it was an embezzlement thing.

16:35 And that person, it came to be that that person was involved in the response and was helping both to deal with it, but also.

16:44 Just gently nudging

16:46 Brad: That's definitely

16:46 Scott: people in

16:47 Brad: nudging

16:47 Scott: one direction

16:47 Brad: people

16:47 Scott: in or the other direction to get it to talk the scent.

16:50 Brad: Man sounds a little bit like the the AI is breaking into hugging face

16:56 Scott: You know, I don't know the details of what's happened there. I haven't read about it. What happened at Hugging Face?

17:02 Brad: OK,

17:02 well,

17:03 I'll do a bad job of summarizing this and people that are interested can always go and read the actual like teardowns. There's now been like multiple independent auditors that came in and did the teardown. But basically.

17:15 OpenAI was testing a new model,

17:18 probably Astra.

17:19 This was pre-Astra being released,

17:21 and they had specifically set up the agents on what they believed was a sandbox that had no access to the Internet.

17:31 Turns out that that sandbox wasn't properly.

17:35 locked down and so they were able the agents were actually able to get to the internet basically they were given a they were given a specific cyber security test to take and they were asked to you know to attempt to do this break-in but they thought they they'd set it up in a in a sandbox and what the agents

17:52 The agents basically thought they had passed the test, but they wanted to double check that they'd passed the test.

17:59 So they wanted to find a place where they thought the answers to the test would be.

18:02 They decided that Hugging Face was a place that might have the answers to the tests.

18:07 They found a...

18:10 They found essentially a zero-day exploit in one of the components of the sandbox that was keeping them in,

18:16 used that zero-day exploit to jump out of the sandbox and to access Hugging Face and then did essentially a targeted attack to break into Hugging Face to try to get the answers to the test and bring it back.

18:31 they were you know and and there's all kinds of other stuff in the analyst analysis of it and i think a lot of this is a little bit of the anthropomorphizing and the sort of like describing this as um i think is kind of making it sound a little bit more nefarious than it actually is if we think the way the agents actually work for example um

18:51 The agents were leaving notes.

18:54 It's described as they were leaving notes for each other.

18:59 they were given access to a they were given access to a working directory where they were told that resources and stuff could be shared there so they could spawn they were allowed to spawn up subagents obviously we want agents to spawn up subagents that's another piece of the thing it's like oh they they had an army of agents like that's just how agent like spawning subagents is i just asked one of my code review agents to do a code review and and it was like i'm gonna spin up five subagents to do this code review you know like

19:24 fine great that's make yourself more efficient you know anyway so that's basically the summary of what it was was that and and there was a little bit of a there there is there is a claim that there was a little bit of a like a racing of the logs and a kind of trying to rewrite the logs so that if they're if the if the experimenters came to look at the results you know because the agents have to show their results as they do the test they try there was some there was apparently some effort by the agents to just

19:50 To hide the fact that they had actually gotten the answers or attempted to get the answers from Huggingface.

19:57 Scott: Is that true?

19:58 Brad: Yes.

19:59 Scott: I mean,

20:00 Brad: Yeah.

20:00 Scott: that's what the audit said.

20:02 Brad: So I have two thoughts about this whole thing.

20:06 You said, used the word nefarious.

20:09 My first thought was like,

20:11 well,

20:11 agents aren't nefarious, they're resourceful.

20:14 Scott: exactly

20:14 Brad: Right.

20:14 Scott: yes exactly yeah

20:15 Brad: They're just trying,

20:17 they're like just water,

20:18 they're like electricity just trying to get to the ground,

20:20 right? And they'll just go wherever they need to go to get there.

20:22 But the fact that it tried to cover its tracks makes me start to think about nefarious.

20:29 Scott: yeah exactly and that's the part where i i haven't personally read the entire audit so i i'm i'm a little bit i'm a little bit like playing a game of telephone here where i'm reporting on the

20:39 the sort of like the most extreme version of what's been characterized

20:41 Brad: I mean,

20:42 Scott: by not necessarily the most technical people like whenever they talk about them passing notes to each other i'm kind of like well that's not like that's not secret that's like a thing that that's that's we expect the agents to do that anyway they leave they leave files for each other that's what they do

20:58 Brad: well, look,

20:59 Brad,

20:59 we are on the internet with this podcast and the internet is known to be a source of reliable information.

21:04 information so I think you know telephone reporting is great of

21:09 Scott: exactly exactly right exactly so it's interesting like we we did finally get to ai didn't we we did get to ai yeah

21:18 Brad: course we did what do you think about this thing you sent me this thing the other day about unit tests what

21:26 Scott: there's this meme

21:27 Brad: do you going think about

21:28 Scott: around unit tests are dead that's the meme that's going around

21:32 Brad: So many engineers are going to be so happy. Finally,

21:35 I've been dragging my feet on this bullshit forever.

21:38 Scott: Right. Well, so the steel man goes something like, so if you say, oh, unit tests are dead,

21:43 the steel man goes something like, first of all, there's a little bit of a who watches the watcher. Like if you're asking an agent to both write functional code and then to go write unit tests for that functional code, then they're not really writing unit.

21:57 unit tests for the code they're writing unit tests they're passing the code they wrote right so there there's a little bit of like you maybe want some separation of responsibility there but that argument probably would have been a problem for human developers too because usually we ask the human developers who write the code to also write the unit tests

22:14 Brad: And rarely do they write them first.

22:16 Scott: exactly another one of the steel mans that is being made about why unit tests are not as valuable is because

22:21 Now that we can write.

22:24 more end-to-end tests and it's more practical to do end-to-end testing with agents i was just talking to an engineer who's like his task was to like fix or to port some web software to make it work better make the the web app work better on mobile web screens and he just plugged claude code into his android phone and he hooked it up with the android debug cable and just told claude code to keep running until it

22:49 it worked as good as possible and this thing just went and it was basically doing the whole thing it was like logging into the web page it was like clicking on stuff it was taking screenshots it was coming back and go that doesn't look right and was like fixing that you know so it was like it was doing it was doing the it was doing the bug fixing by very much end-to-end testing the application in that context

23:13 Brad: Mm-hmm.

23:13 Scott: and it did amazing work right so we now know that agent

23:16 Agents can do great end-to-end testing.

23:18 So one argument,

23:19 a steel man argument for like,

23:20 don't waste your time on unit tests would be.

23:23 look end-to-end tests are so much more valuable and it used to be almost impractical to do all the end-to-end tests but now it's not impractical to do that so put your tokens put your energy behind the end-to-end tests that's that's that's where the real value is and then there's some other like more complicated testing frameworks like you know like black box testing or like you know you know golden golden data testing where you like you know you're essentially doing like

23:49 like you get expected you get a fixed results and then if any change gets made you test the test results against the fixed results and if any deltas between the expected fixed results and the and the the you know the new results you you kind of flag that and those are also harder to do right you know end and one of the reasons end-to-end testing is so hard is because you have to create all this synthetic data right and you have to be able to reproduce the synthetic data

24:14 But none of that's hard anymore.

24:16 OK,

24:16 so I think the argument, the steel man argument,

24:19 and I'm going to tell you why I disagree with it, but the steel man argument goes something like unit tests for this poor,

24:26 you know, poor man's thing for trying to, you know, the whole test pyramid of like we write just a whole bunch of unit tests because they're easier to write than integration and end to end tests.

24:38 But if we do enough of them, then that gives us some test confidence.

24:44 And now that we have the ability to really write high quality end-to-end tests,

24:48 we don't really need all those unit tests.

24:51 That's the steel man for why unit tests are dead.

24:54 I think that framing is completely wrong.

24:57 Now, the part that I agree with is absolutely that end-to-end tests are better for a lot of things and they're so much more economical to write.

25:07 We absolutely should be investing and leaning into end-to-end tests.

25:10 Yay,

25:11 go end-to-end tests. I love it. Let's do it.

25:13 I love it. No complaints there. The part.

25:17 that I disagree with is there is this whole space we've implemented all this code and the reality is that the end-to-end test often exercises a fraction of all the code we had to write

25:35 Because we had to write code,

25:37 we do need to write code for all the edge cases, for all the exceptions.

25:42 We need to do, and most of the code that the, this is sort of like the insight of the engineer, right?

25:47 They were given a spec,

25:48 the spec had some user stories, but the spec and the user stories were silent on a lot of weird little corner cases, right?

25:55 And so as good engineers,

25:56 we make decisions about that. Well,

25:58 how do we handle, you know, how do we handle a 500 error from some back?

26:02 back-end service like how does the front-end handle that right or

26:05 Brad: Mm-hmm.

26:06 Scott: what how do we handle like asynchronous code right we're going to have race conditions that are going to happen with multiple parallel asynchronous code right very difficult to make that happen in an end-to-end test it's very difficult to construct the exact environment of your end-to-end test to make all these special cases that need

26:27 need to have robust code behind them happen right and so that's where i feel pretty strongly that what what i want is i want to know that oh another thing that's very difficult to do is it's very difficult to get um

26:43 coverage reports on a true end-to-end test right

26:47 Brad: Mm

26:48 Scott: um

26:48 Brad: -hmm.

26:48 Scott: especially in modern architectures microservices and things like that right because that that end-to-end experience is is is hitting multiple different services and if you really wanted to say like did my end so what i care about is i care about end-to-end testing but i really care i want to be able to answer the question you

27:07 For every line of code that's been written,

27:09 that's running in production,

27:10 every line of code that's running in production,

27:12 has it been tested,

27:14 right,

27:15 automatically?

27:16 Brad: Mm-hmm.

27:17 Scott: Has it been tested automatically?

27:19 And was the test that tested it automatically,

27:22 was it actually testing the proper constraints,

27:27 right? So I don't.

27:30 I mean,

27:30 coverage is great. I love code coverage of unit test code coverage or end-to-end test coverage,

27:36 but I need to know that it didn't just run the code, but it actually asserted

27:40 Brad: Mm-hmm.

27:40 Scott: the things that mattered,

27:42 right?

27:42 Brad: Mm-hmm.

27:44 Scott: I think that we should basically demand raising the bar,

27:48 not lowering it. Yes,

27:49 let's add end-to-end tests.

27:51 Absolutely.

27:52 Let's get coverage on our end-to-end tests,

27:54 right?

27:55 Brad: Mm-hmm.

27:56 Scott: Let's validate that those end-to-end tests are how much of the code,

28:00 how much of the production code is it touching?

28:02 And then the parts of the production code that those end-to-end tests aren't touching,

28:05 let's backfill all that with real high quality.

28:10 You know, mutation tested unit tests and integration tests.

28:14 And

28:14 Brad: Mm-hmm.

28:15 Scott: that's where you just can't, you can't really validate your coverage.

28:21 especially with mutation testing like i've not seen anybody do end-to-end test mutation like that's that sounds like the holy grail like okay

28:30 Brad: What do you mean exactly by mutation testing?

28:33 Scott: what i mean by what i mean by mutation is so and there's and there's a bunch of great tools that you can just put i i in fact just you know in preparing to talk to you about this i was like how hard is it to just randomly throw a mutation test

28:46 platform on top of an arbitrary piece of code.

28:49 So I went and grabbed an arbitrary piece of code that I have that didn't have mutation testing on it,

28:54 and I installed mutation testing on it to see how well it would work. Mutation testing is where you have all your unit tests,

29:00 right?

29:01 Brad: Mm

29:01 Scott: And

29:01 Brad: -hmm.

29:02 Scott: you ask the question,

29:03 actually we talked a little bit to Uncle Bob about this, you ask the question, well,

29:08 is the unit test actually catching mistakes in the code?

29:13 And one way to do that is to intentionally go add a mistake to the code.

29:19 Brad: Mm-hmm.

29:20 Scott: And when you add a mistake to the code,

29:22 does the unit test fail?

29:23 Brad: Mm-hmm.

29:24 Scott: That's what mutation testing is.

29:27 And For there's

29:27 Brad: instance,

29:27 Scott: – exactly,

29:28 Brad: you could have a method and hollow out the entire method and just return zero.

29:34 Scott: exactly.

29:35 And does the unit test fail?

29:36 So there's some of the great tools that are out there to just kind of like throw mutation testing on top of your code.

29:41 They do things like –

29:43 every you know comparison operator just go and change compare just change a change of less than equal to to a less than change a less than equal to a greater than change a less than equal to it to an equal right and that change in the code you just you just introduced a bug in the code by changing the code right if the code was correct if it was supposed to be less than or equal to and that was correct

30:09 and it was being tested by the unit test that it was correct then by changing that code the unit test should fail

30:14 Brad: If it doesn't.

30:15 Scott: what and if it doesn't it means your unit test was not robust right so mutation testing is really hard because what you're doing is every single one of your unit tests that test a function you need to go make

30:30 multiple mutations of that function and rerun that unit test across multiple mutations that and typically the way mutation testing like kind of reports back is like it looks a lot like a unit test or test coverage thing where it's like this many tests but what it does is for each test it it creates mutants they call mutants and then they rerun the test and which mutants survive right is this is this metric so it's like you're kind of looking to see

30:55 if 0% of your mutants survive your unit test,

30:59 your unit test is awesome because basically it failed every time any one of these mutations was done to the function.

31:07 Brad: Brett, I can tell that this topic is sort of dull for you.

31:13 uh i'm

31:13 Scott: I got

31:14 Brad: in touch

31:14 Scott: low energy.

31:14 Brad: hey

31:15 Scott: I'm low energy.

31:15 This is low energy,

31:16 Brad.

31:16 Brad: everybody you should see brad when he gets excited about something can't even get him to sit still we you haven't touched on integration tests so

31:27 Scott: Yeah.

31:28 Brad: first for the mothers in our audience out there what is an integration test

31:33 Scott: Okay,

31:34 so...

31:35 I think it's easiest to talk about integration tests by first saying what unit tests are,

31:40 and then we can talk about integration tests. So unit tests, we take a single object,

31:46 a single object,

31:47 typically a function,

31:49 typically a class,

31:51 and then we...

31:53 Test that object in isolation.

31:55 And since very few objects in code exist in isolation,

32:01 the way you test

32:03 The way you unit test something is you stick a bunch of mock scaffolding of all the other objects that that object depends on around

32:13 Brad: Mm

32:13 Scott: it.

32:13 Brad: -hmm.

32:14 Scott: And you create an idealized version of the other function it was going to call.

32:20 And then you call the unit.

32:23 And you know that anything behind it is behaving as expected because you put a fake one there.

32:29 And so now you're only testing the thing that you wanted. That's a unit test.

32:35 Integration test is when you take multiple real objects together,

32:38 you glue them together,

32:40 and then you, you know, send tests across it.

32:43 So a simple example of that might be.

32:46 An integration test that tests my back-end service that depends on the database behind it, it's an integration test if I'm standing up the database,

32:59 standing up my service.

33:02 touching the you know testing the service and it actually does round trips through the database that's like the simplest version of an integration test yeah

33:09 Brad: There's a form of integration test out there that has piqued my interest lately.

33:13 I've never actually written one of my own.

33:17 I'll tell you about it in a second.

33:18 I have directed two teams towards it, though,

33:20 and I think one might be actually working with it.

33:22 I've heard it referred to as contract testing.

33:25 There's a framework out there called PACT.

33:27 And basically, as we have more and more of these services that have APIs,

33:32 whether it's a client like a mobile client or a web client talking to a back-end system or two back-end systems talking to each other,

33:41 it's a way to ensure that those APIs are contracts between these different parts of the system.

33:50 So it's a way to ensure that the change in your.

33:55 system doesn't change any contract or can still play nicely with I think it does it both ways either I can ensure that it didn't change the contract or the part I'm changing or I can ensure that

34:07 Ensure that my part of the system can still play nicely,

34:11 can still talk effectively, doesn't fail when intergaging with anybody else's contracts.

34:16 Scott: Yeah.

34:17 Brad: And at least this PAC framework does a thing where when you, I guess when you produce an API,

34:22 it produces some sort of artifact. I'm not exactly sure what it is, but this artifact can be used.

34:31 It can be used in two ways.

34:33 One is when you're changing your system,

34:35 let's call it your microservice,

34:37 you can verify you didn't break any of the interface.

34:39 And then the clients of that system

34:42 have an artifact that they essentially I guess it's like building a mock of that of that interface um so that's long-winded and I've probably jumbled it but have you worked with anything like that are you familiar with that

34:54 Scott: Yes, I think that having API tests or interface tests is a really good tool,

34:59 especially when you have microservices or you have a classic like full stack architecture,

35:04 back end, front end.

35:05 Absolutely. In fact.

35:07 That is one of the teams that I'm working with.

35:13 They didn't implement unit tests on one of their primary back-end services.

35:19 They just did API tests with it.

35:23 And so it's an integration test because those APIs only really work if you've got the database running behind it. This is an example of a type of integration test I'm talking about.

35:31 So literally they stand up the database,

35:34 they stand up their back-end service,

35:35 and then they have effectively a client.

35:38 And that client has just a whole list of API calls that it makes.

35:43 It's not really an end-to-end test because this is just the back end. It's only testing the correctness of the back end, but it does full things like.

35:54 You know, I'm going to just describe this generically as like create,

35:58 create,

35:59 create user,

36:00 add,

36:00 Brad: Mm

36:00 Scott: add

36:01 Brad: -hmm.

36:01 Scott: event to user,

36:02 create all the events of the user, add a second user,

36:06 user one and user two become friends,

36:08 you know, and you just run through this, you know, script of all these different things that you you expect that if you call those APIs in a specific order.

36:20 then the next API call where you get state back should match your expected behavior,

36:25 and that's a classic API pack test style thing of an integration test against one service plus another service behind it.

36:33 Brad: Two thoughts. I think what this particular packed framework does is make that a little bit easier,

36:37 but that's the framework. We can talk about that later.

36:40 My Spidey sense,

36:41 this is my second point, my Spidey sense tells me this is a really high leverage.

36:46 test space those testing those apis or those interfaces because yeah if you're if you're expecting to create a user if i guess if any of those parts downstream don't work this i guess this is kind of making the argument of the get rid of the unit test guy which

37:00 Scott: Right.

37:00 Brad: is i'm not i'm not saying i agree with the get rid of the unit tests but uh i think if i had i guess what i'm saying is if i had to pick between getting if i could only have one could i have unit tests or could i have integration tests

37:14 I would probably pick the integration tests.

37:16 Scott: Okay, well,

37:17 so I hear you on

37:17 Brad: It's

37:17 Scott: that.

37:18 Brad: just my guts telling you, but I haven't really thought about this that hard.

37:20 Scott: So the part that's hard about that is,

37:22 okay, so let's talk that through for a second.

37:25 If you want to do just integration testing against this back end, then...

37:31 How do you make sure that it behaves correctly when one of the other integration components is misbehaving?

37:40 Brad: Right.

37:41 That's exactly as soon as you started talking, I was just spent a second thinking about this. I was like, yeah, this only this doesn't test error conditions. It tests bad code,

37:50 Scott: Or

37:51 Brad: but it doesn't test the system having a failure somewhere in it.

37:54 Scott: it tests bad code across the happy path.

37:58 Path and all

37:58 Brad: Mm-hmm.

37:59 Scott: the unhappy paths that you can think of.

38:02 Brad: Right.

38:02 But it doesn't test network problems and doesn't test other systems going down.

38:06 It doesn't. There's a variety of things that that doesn't test.

38:09 Scott: Exactly.

38:10 Brad: Right.

38:10 So this gets back to your point, which I will now agree with you.

38:13 We should just call this podcast two CEOs violently agree.

38:17 Scott: See two CEOs violently agree.

38:19 Exactly.

38:21 Brad: And that's the date we should rebrand this whole thing.

38:24 And yeah,

38:26 I think I think I'm with you. You need those unit tests because.

38:28 it's because you do need to test what happens in this case.

38:32 And I'll give you a concrete example of something that bit us in the behind three times in the last two years. And finally, the other day I was like,

38:40 we have to fix this.

38:41 We have a third party component that it went down, the third party went down for the third time in two years. And when they went down, it caused us to have an outage.

38:52 And I was like, why don't we have some sort of circuit breaker type thing around that?

38:58 It's an important component, but it's not critical. We could still operate without it.

39:03 Scott: Oh, yeah.

39:03 You're saying that you're saying this is a component that is not a mission critical component,

39:07 but when it went down, other mission critical components basically misbehaved in such a way that you were down.

39:12 Brad: yeah i mean uh i'll just be concrete so we have this little third we used we have chat between customers we use a third-party chat service we didn't write our own chat and um they went down they've had and not not long not like long enough it's just they just had blips in their service and when they had a blip in ours it created a blip in our service everyone freaks out and i was okay chat is i mean it's important it's important our customers and our clients and taskers have to be able to talk to each other

39:38 each other but if they can't talk to each other for 10 minutes it'll be okay

39:41 and that's better than having big parts of the system go down and you can't you couldn't test that with an integration test but

39:49 Scott: it'd be you very

39:49 Brad: could

39:49 Scott: difficult

39:50 Brad: test that with

39:50 Scott: it

39:50 Brad: a unit test

39:50 Scott: would you yeah exactly or i mean to you'd have to jump through a lot of hoops to test that with a integration test you'd have to do something like set it up in a sandbox where you somehow

40:00 turned on and off the network on the sandbox or something like that you know

40:03 Brad: or

40:03 Scott: like

40:03 Brad: something like that or or had something where you could turn off that specific integration right i

40:08 Scott: right and this is my point about okay and so this is this is my this is now now now i'll pull on the thread of mutation testing again because you know i got all excited about mutation testing you asked me to calm down so now i've calmed down um

40:21 Brad: did not ask you to calm down

40:22 Scott: but how are you going to do your integration tests

40:27 How are you certain that the integration test tests the correctness of the code as written?

40:36 In other words, when I do this simple integration test where I'm like calling the API.

40:42 And the data comes back and I checked like I put data in,

40:47 I got data out.

40:48 And when I put in exactly this data in,

40:50 I got exactly the data I expected out.

40:52 Okay.

40:52 What I really want to ask you is,

40:54 are you sure you got the data shape correct to test the code as written?

41:00 And that's what mutation testing does,

41:02 right?

41:02 Mutation testing tells us,

41:04 was our test

41:07 validly testing the code as the code as written or the code the correctness of the code or did we just happen to have like the perfect shape test data for the happy path that works in the code

41:20 Brad: Right.

41:21 I mean, you could solve that theoretically by like.

41:24 yeah let's say you're testing the create user method right and you throw some data at it you get a good user back you could run that unit test let's say a half dozen different ways feeding a different data and different shapes some

41:37 Scott: And

41:38 Brad: of what you might expect to fail at me i'd expect to say i don't expect yeah

41:41 Scott: you should do that.

41:42 Like, I definitely

41:43 Brad: 100

41:43 Scott: think you should include those types of tests in your unit and your integration tests.

41:47 But you still have to ask yourself,

41:48 well,

41:48 did I get all the shape of all those dozen things?

41:51 And things did I get them thoroughly enough?

41:53 Did I get them thorough enough?

41:54 Brad: Yeah.

41:55 Okay.

41:55 I want to finish on one question for you. Help me answer a common developer argument against unit tests.

42:03 And here is that argument.

42:04 The amount of time it took us to write all these tests over the years has not outweighed the number of bugs that it has caught.

42:14 Scott: Okay in in 2019

42:18 You got me, Scott.

42:20 Brad: Mm-hmm.

42:20 Scott: That's a good call.

42:22 In 2026, it's no longer a good call because your agents can write all that code for you and they can do it really fast and they can actually write all those test cases for you.

42:36 Brad: Gotcha.

42:37 So the time has changed.

42:40 This is your Einstein moment.

42:41 You're actually saying time is not constant.

42:44 Scott: Exactly. I'm just saying like I'm saying like now is the golden age of validation.

42:50 We like it's bonkers to me that everyone's like,

42:53 oh,

42:54 I'm so afraid about the quality of my agent's code. Well,

42:57 then write, write, make the agents make two different agents.

43:01 One's not allowed to look at the.

43:03 not allowed to look at the implementation they're only allowed to look at the the functions and the expected they're only allowed to look at the type doc of the function and have them go write a bunch of unit tests and then and had the other agent against the other agents code right like go ahead separate it you know and then have a third agent that does mutation testing against that and is validating that that that those unit tests really were valid yeah and and also write all the end-to-end tests

43:28 you know i mean if people are complaining about unit tests they for sure were like i'm not writing playwright tests for end-to-end tests that's that's that's that's crazy making i'm

43:39 Brad: I haven't heard that complaint in a long time.

43:41 Scott: going to hire a team i'm going to hire a team in ukraine to go write my end-to-end tests like literally i've been at companies that did that like oh we're going to offshore the end-to-end test because they're just too hard to do and we don't have time to do that no not anymore you don't

43:54 You don't not anymore.

43:55 You don't have to say that because now you just get your agents to do it.

43:58 Brad: All right,

43:59 Professor Einstein,

44:00 this

44:02 Scott: It's good talking

44:02 Brad: has been

44:02 Scott: to you.

44:02 Brad: a fun one.

44:03 Scott: It's fun talking to Scott.

44:05 Brad: It's been another fun week.

44:06 What do we say?

44:08 Scott: Rate and review,

44:09 smash that subscribe button and don't forget to tell your mom.

44:13 Brad: That's right. Don't forget to tell your mom.

44:14 All right. Talk to you soon, Brad.

44:15 Scott: Talk to you later, Scott. Bye.

44:17 This episode of the How Many CTOs Does It Take podcast was produced and edited by Saffron HeptaCal.

44:23 Our theme song was AI generated by Brad.

44:26 Please follow us on all the social media platforms at HowManyCTOsPod.

44:31 You can find us on YouTube,

44:33 LinkedIn,

44:34 Instagram,

44:34 Facebook,

44:35 Threads, X,

44:36 TikTok,

44:36 and Patreon.

44:37 Or at our website at HowManyCTOsPod.com.

44:41 And remember to rate,

44:43 review,

44:43 and subscribe on the podcast platform of your choice.

44:46 Thanks for listening.