Transcript: Preparing for the Worst: Tabletop Exercises, Unit Tests, and Everything In Between
0:00 Scott That exercise,
0:01 Scott the hygiene exercise,
0:02 Scott is valuable.
0:19 Scott Hello, Brad.
0:22 Scott Hey,
0:22 Brad Hey, Scott.
0:22 Scott welcome to this week's episode of...
0:25 Scott How many CTOs does it take to put in a light bulb?
0:29 Scott My name is Scott Porad.
0:30 Scott This is my trusty co-host.
0:32 Brad Brad Hefta-Gaub.
0:33 Scott How are you doing, Brad?
0:34 Brad I'm doing good. Actually, it's funny you say how many CTOs does it take to put in a light bulb because my light bulbs died earlier this summer and the ceiling is really tall. I got to go get an extra tall ladder to put the light bulbs in and it's starting to get darker.
0:49 Brad You know, it's starting to stay darker later,
0:51 Scott Mm
0:52 Brad get
0:52 Scott -hmm.
0:52 Brad darker earlier,
0:53 Brad and I'm going to have to fix these light bulbs.
0:55 Brad It's not summer anymore.
0:56 Brad It's not summer anymore.
0:58 Scott It's not summer anymore.
0:59 Scott Or you can get one of those sticks with the thing on the top that you don't have
1:03 Brad Fair
1:03 Scott to get a ladder.
1:03 Brad enough, because it's got like a glass thing on it. I got to take the glass thing off before I can fix the light bulbs.
1:08 Brad It's going to be a whole thing.
1:09 Scott It's going to be a whole thing.
1:12 Scott We did a whole thing at TaskRabbit last night.
1:14 Brad Yeah.
1:15 Scott If any of the TaskRabbit tiers are working there or working there listening, they'll know about this.
1:22 Scott We did an AWS region move.
1:25 Brad This is
1:26 Scott In
1:26 Brad South Park.
1:26 Scott US West 1 to US West 2,
1:29 Scott I think it is well known that US West 1 is kind of like a second class citizen in the AWS world.
1:37 Scott It has fewer features. It costs more.
1:40 Scott It's less of everything. So we did this migration.
1:44 Scott It's a big project.
1:45 Scott A big team of people are involved in it.
1:47 Scott They did a really good job. Their attention to detail was really good.
1:49 Scott There's just been a few.
1:50 Scott two little incidents this morning that we've had to clean up one thing didn't get whitelisted the IP properly etc it's overall working really well the reason the reason I'm bringing this up is twofold the first is
2:03 Scott Well, let me start at the end. It made me think we should move regions every year.
2:07 Brad just to make sure you got stuff buttoned up, right?
2:11 Scott Yeah,
2:11 Scott it forced us to really get our stuff buttoned up, to understand how our system works.
2:17 Scott And this company, I don't know that this company had ever moved regions since it went on to AWS.
2:21 Scott And so the fact that it just took a ton of work to even just figure out everything that was out there that we had to take care of and how it was all configured and all the little.
2:28 Scott little you know extra stuff and now it's all pristine and clean and neat and tidy right and so it's like this amazing housekeeping exercise for just like getting stuff in order and you'll learn a ton about
2:41 Scott your systems you know as you have systems that are mature you just have turnover and institutional knowledge gets
2:47 Brad Yeah.
2:47 Scott lost and so it forces us to go learn a bunch of stuff which was really cool and it's I just thought and it went really smoothly and kudos to the team and I just thought oh we should do this like every other every year every other year just so that we're just cleaning out the house
2:59 Brad But yeah,
3:00 Brad but I think you mentioned to me before we hit the record button that there was like a 33 page runbook and you had like 25 plus people on the call.
3:06 Brad And so.
3:07 Brad You might get
3:08 Scott Yeah,
3:08 Brad a little bit
3:08 Scott so.
3:08 Brad of static if you want to do that too frequently.
3:12 Scott Well, like I said, the company had never done this before.
3:16 Scott So a number of people,
3:17 Scott about seven to 10 people had been working on this for a few months to prepare.
3:21 Scott They had built a very detailed runbook of everything that needed to happen.
3:24 Scott They had practiced it a few times. And then we went through it last night and they had practiced rollback scenarios at each different spot. If there was a failure at this spot, what the rollback scenario would be, et cetera, et cetera.
3:37 Scott And then we had one person from every other engineering team involved.
3:44 Scott And there were a lot of those engineers,
3:45 Scott not a lot, but people were like, well, why do I need to be on this call?
3:48 Scott Like on call, can't you just call me?
3:50 Scott And we were like, no,
3:51 Scott we just want everyone there.
3:52 Scott Like we just want this to go as smoothly as possible.
3:54 Scott And maybe you won't pick up the phone in the middle of the night or something like that. And let's just have everyone on the call.
3:59 Scott You know, you can expense pizza or whatever you want to do at your house. And it went well.
4:03 Scott I just thought, you know, that exercise,
4:07 Scott the hygiene exercise is valuable.
4:10 Brad Totally. It's kind of like doing the recurring tabletop exercise where you.
4:17 Brad I mean, yours was more, it was real.
4:18 Brad It was like,
4:19 Scott It
4:19 Brad you
4:19 Scott was real.
4:20 Brad were really doing it. It wasn't a tabletop, but, you know, you still can learn a lot by just recurring, scheduling those recurring,
4:27 Brad even if you do it as role play, you can often like discover something,
4:31 Brad you know, that
4:32 Scott That's
4:33 Brad you didn't
4:33 Scott right.
4:33 Brad think of.
4:34 Scott This was very helpful because one of the things we want to do in the
4:37 Scott next 12 months is improve our disaster recovery and business continuity processes that's a big step for us not a big step but an important thing we want to do and this really helped with that because it helped us know our system so that when we go to say let's what would it be like if
4:54 Scott U.S. West 2 disappeared
4:57 Scott How we handle that.
4:58 Brad Totally, totally.
4:59 Brad And that's obviously like, you know, at the scale of TaskRabbit, that's like a very real scale,
5:04 Brad very real complex systems. And like you said, you hadn't done this in several years. So you had some, you know, kind of like the classic like it's documented somewhere, but we got to figure out where all the documentation is or figure out the as built if it wasn't documented.
5:20 Scott Yeah, I don't
5:21 Brad Yeah.
5:21 Scott think it was documented before.
5:22 Scott I have a question for you about tabletop exercises.
5:25 Brad Yeah, tell me.
5:26 Scott I'd like to know two things.
5:27 Scott I'd like to know, well, first, let's explain. I'm going to interview you.
5:30 Scott Brad,
5:31 Brad Okay.
5:31 Scott tell us what a tabletop exercise is for the 41 listeners in our audience.
5:35 Scott I bet there's at least 37 of them who don't know.
5:37 Scott And then tell us like your theory of developing a tabletop exercise and then tell us your top three ones that you like to do.
5:44 Brad Okay.
5:45 Brad Well, OK,
5:46 Brad so tabletop exercise is where you essentially create a contained space, usually a meeting.
5:54 Brad You might give your people a little bit of preparation to so that they know kind of maybe the context the tabletop exercise going to be in.
6:02 Brad But then you basically present them with a scenario and in real time,
6:07 Brad everybody around the room.
6:10 Brad does their part to resolve the scenario.
6:16 Brad Okay.
6:16 Scott Mm-hmm.
6:17 Brad And the main thing here is not that you get it right, but typically what you're going to do is you're going to document everything like super easy. If everything going over, you know, video call,
6:27 Brad just click the record button on the thing and say,
6:29 Brad okay,
6:29 Brad here's our scenario.
6:31 Brad And then you just start walking through it and say, what do you do? What are you going to do? What are you going to do? How would you, and then what you're looking for is.
6:40 Brad You're essentially looking for gaps in processes or documentation where the person in the room goes,
6:48 Brad oh,
6:49 Brad I don't know what I would do now, right?
6:51 Brad Or they might say, someone in the room might say,
6:54 Brad well,
6:55 Brad okay,
6:55 Brad you know, the database got deleted.
6:59 Brad I'm going to start to go,
7:00 Brad I'm going to go get the, you know, the last snapshot and someone in the room might go.
7:07 Brad well how did you know to do that he's like you know they might just be like well because i'm a dba i know that's what i'm going to do and then you'd say well great let's make sure we have it clearly documented if you were the only one that would have known to do that then we have a gap right so that's kind of the general idea of the way we do a tabletop and usually
7:25 Brad the person who creates the tabletop comes up with a scenario and then they want to think through like and they'll usually have like a set of questions that they're ready to kind of prompt the team with as you go along to sort of ask you know ask probative questions along the way you know to kind of keep that keep that scenario rolling and then the most important part is after you're done with the whole scenario then you just do a
7:50 Brad you know kind of do a hot wash and a where you're just like okay let's talk through all that happened here let's like debrief how we thought about it and let's come up with a punch list of gaps that we want to then commit to improving and fixing in our in our technology our processes or our culture yes
8:08 Scott Well, it sounds to me like it is an incident response process drill.
8:13 Scott And
8:13 Brad exactly just. so so hopefully you've got your incident response
8:16 Brad process that you've already documented and you're just following it.
8:20 Scott And then it sounds like all good incident response processes, there should be a retrospective at the end to reflect upon both the incident and the process.
8:29 Brad Yeah, exactly.
8:30 Brad Now, one thing I think is where tabletops can go outside of your incident response process is
8:37 Brad If your incident response process has been written by your technical team and is very maybe it's it's more technical in nature and maybe it included things like, oh,
8:47 Brad this is when we contact legal counsel or this is when we contact the the the public information officer or the comms team,
8:55 Brad you might have written that in your incident in your tip in your traditional tech focused incident response plan.
9:03 Brad But a lot of organizations don't necessarily have that piece. And so sometimes you'll try to do tabletops where it's got nothing to do with technology.
9:10 Brad Like, in fact, when I was at 98.6, I think this was maybe after you left and I was still there, I participated in a tabletop which was a business continuity tabletop around what happens if a key person.
9:28 Brad leaves the organization like what happens if the ceo leaves the organization how do we handle that
9:33 Scott You did this after I left because everybody was just, what are we going to do without Scott?
9:40 Brad it may have been it may have been that may have been the impetus of it you know as far as you'd know
9:48 Brad and so that was an that was an example of a tabletop that that leaned a little less technical but still like exposed a bunch of really interesting questions like oh how do we how do we handle that so you asked me what a tabletop was did i get a good answer for you does that make sense or what else did you want to ask me
10:02 Scott I'll
10:02 Brad about that
10:02 Scott have to ask my mom when she listens to this podcast and see that that makes sense to you.
10:06 Scott Let's see.
10:06 Scott Well, then I was like,
10:07 Scott okay, so you have to come up with a scenario when you're doing, you know, you're a fractional CTO,
10:11 Scott you go in and you work with teams, you have to come up with a scenario to help them walk through this.
10:16 Scott Any tips for someone out there on how to come up with a scenario?
10:20 Brad Well,
10:20 Brad I just did the most recent one that I did.
10:23 Brad I did it because I was sort of like the scenario I came up with just seemed like really in vogue, right?
10:29 Brad Like the scenario I came up with was.
10:31 Brad The company has installed,
10:32 Brad you know, at Claude into Slack and simultaneously hooked up Claude to the MCP server to some important,
10:40 Brad you know, database and then an intern at Claude's to go clean up some data and tells it to make no mistakes and Claude decides to delete the database,
10:52 Brad right?
10:52 Brad Like that just seemed like a very like topical, like that's been in the news. I'll use that one as the scenario.
10:57 Brad scenario and it
10:58 Scott Mm-hmm.
10:59 Brad turned out to be a really great exercise like the team that I put this together for they were like they actually did find a couple places where they had some gaps in their data recovery plan and they also it also turned into a really great conversation around oh how do we like we're getting so much power out of these tools we really don't want to slow people down so how do we how do we how do we manage governance of these agents
11:23 Brad Um,
11:23 Brad how do we manage the governance so that people can still move fast, but you know, not,
11:28 Brad not just count on asking the agent to make no mistakes.
11:31 Scott Yeah, I can think of three or four different scenarios worth coming back to over and over again.
11:39 Scott I'll just rattle them off.
11:40 Scott One is some key data has been deleted,
11:43 Scott whether it's data in a database or media assets or something, something key has been deleted.
11:48 Scott A secret.
11:49 Scott or token has been leaked he you know whatever your hosting situation is ours is aws whatever has gone down and it's going to be down for a while you need to restore your system so a disaster recovery scenario and then the last one coming from personal experience your ceo gets an email and that email says
12:12 Scott Here's a snapshot.
12:13 Scott Here's a screenshot of some pictures of your database.
12:16 Brad Yeah.
12:17 Scott So send me some Bitcoin.
12:19 Brad Yep.
12:19 Brad That's a good one.
12:20 Scott That was a day.
12:21 Brad Yeah.
12:22 Scott Totally. But what happens if that happens if somebody tells you they're inside your system?
12:27 Brad Yeah, exactly.
12:28 Brad Yeah.
12:29 Brad And that's a really good one. I was going to say some kind of a scenarios that have some kind of a hack or exfiltration or some kind of a like anything that's like anything that's like an outside intruder in your system is a good one because it's going to it's going to push on a lot of different things. It's going to push on like,
12:46 Brad well,
12:47 Brad how do we how do we respond to that?
12:49 Brad How do we handle that? Right. And that your your your Bitcoin ransom.
12:53 Brad ransomware attack is a good example of that but then you also have to start challenging yourself well how would we did we did we miss it if we missed it why did we miss it and more importantly you know what do we what systems do we have in place to make sure that our monitoring and our safeguards are updated as we evolve our system right like like that's that's super common right like if you lock down your you've got a bunch of great monitoring you've got like
13:18 Brad like you know burst monitoring and alarms that are going to go off but then you had some new component and you didn't monitor that one or you know so so how do you make sure that every new component is getting monitored correctly so there's all kinds of stuff that you can learn from these tabletop exercises and it's way better to do it
13:34 Brad You know, not in the heat of battle.
13:35 Scott For sure.
13:36 Scott So here's something interesting.
13:38 Scott We were talking about external penetration,
13:40 Scott somebody getting into our systems.
13:41 Scott A number of years ago, when I was sort of early in my information security type of experience, I was working with this guy, it's like a security guy,
13:49 Scott and he was really pressing on the role based access control,
13:53 Scott access control is access control. And I was like,
13:56 Scott I can't remember his name, let's just say his name was Jeff.
13:59 Scott I was like, finally,
14:00 Scott I was like, Jeff,
14:01 Scott I don't understand why you're, and this is after, you know, working with him for quite a while. I'm like, so I don't understand why you keep on pushing on the access controls. To me, it feels like the perimeter of the system is so much more important to harden than the access controls.
14:15 Scott He said,
14:15 Scott I totally get why you're thinking about that.
14:17 Scott But most incidents happen from someone internally.
14:24 Scott Or someone gets access to somebody's account internally and then they've
14:27 Brad Yeah.
14:27 Scott got unlimited control in there.
14:29 Brad Yep.
14:29 Scott And I was like, oh, that was a light bulb that went off. And then later in my career, probably five or 10 years later, I was involved in two,
14:36 Scott not one, but two internal embezzlement incidents.
14:40 Brad Yeah.
14:41 Brad Yeah.
14:42 Scott And we actually had an incident recent at TaskRabbit. It ended up being just fine.
14:48 Scott We use,
14:49 Scott we have.
14:50 Scott We're gig marketplace,
14:51 Scott so we collect money from clients who book jobs, then we pay it out to taskers who do the work and sitting in between is Stripe.
14:58 Brad Mm
14:58 Scott And
14:58 Brad -hmm.
14:58 Scott there was this incident we had where
15:01 Scott A handful,
15:02 Scott like, let's just say, you know, under a thousand people got paid twice.
15:06 Scott Some bug paid people twice.
15:08 Scott And so people are running around trying to fix this. And in my mind, my light bulb went off. I was like,
15:13 Scott oh, that is a signal of an embezzlement kind of thing.
15:18 Scott And this is something I had learned earlier in my career.
15:22 Scott When you think you see something happening internally.
15:25 Scott You have to treat everybody as a suspect,
15:27 Scott right?
15:27 Brad Yeah, sure.
15:29 Scott And so you have to understand like,
15:30 Scott okay, who is the one or two people that I'm going to start with and say, we need now I'm going to try who are the one or two people you're going to trust out of the gate,
15:38 Scott right?
15:39 Brad Right.
15:40 Scott Yeah, right.
15:40 Scott And say we need to actually quietly go look at some other things here.
15:45 Scott We need the turtles all the way down. We need to investigate the investigators kind of thing.
15:50 Scott So what I'm sharing all this just to say another type of tabletop which would be valuable to do is the internal actor because it forces you as an organization to say, well, wait a second if you're an internal actor.
16:02 Scott I think you have an internal action,
16:03 Scott you have to be much more suspicious,
16:05 Scott but also careful about how you manage that incident.
16:08 Scott Because
16:08 Brad Yeah.
16:09 Scott in one of those embezzlement incidents I was involved in,
16:11 Scott the embezzler threw the incident responders off the scent.
16:18 Scott The embezzler got involved in the incident response.
16:22 Brad Oh, wow.
16:24 Scott Yeah.
16:24 Brad That's wild.
16:24 Scott And so but we did we didn't suspect that that was the issue. So we didn't have any reason that we were we were being clueless about it. And so we didn't have any reason to suspect it was an embezzlement thing.
16:35 Scott And that person, it came to be that that person was involved in the response and was helping both to deal with it, but also.
16:44 Scott Just gently nudging
16:46 Brad That's definitely
16:46 Scott people in
16:47 Brad nudging
16:47 Scott one direction
16:47 Brad people
16:47 Scott in or the other direction to get it to talk the scent.
16:50 Brad Man sounds a little bit like the the AI is breaking into hugging face
16:56 Scott You know, I don't know the details of what's happened there. I haven't read about it. What happened at Hugging Face?
17:02 Brad OK,
17:02 Brad well,
17:03 Brad I'll do a bad job of summarizing this and people that are interested can always go and read the actual like teardowns. There's now been like multiple independent auditors that came in and did the teardown. But basically.
17:15 Brad OpenAI was testing a new model,
17:18 Brad probably Astra.
17:19 Brad This was pre-Astra being released,
17:21 Brad and they had specifically set up the agents on what they believed was a sandbox that had no access to the Internet.
17:31 Brad Turns out that that sandbox wasn't properly.
17:35 Brad locked down and so they were able the agents were actually able to get to the internet basically they were given a they were given a specific cyber security test to take and they were asked to you know to attempt to do this break-in but they thought they they'd set it up in a in a sandbox and what the agents
17:52 Brad The agents basically thought they had passed the test, but they wanted to double check that they'd passed the test.
17:59 Brad So they wanted to find a place where they thought the answers to the test would be.
18:02 Brad They decided that Hugging Face was a place that might have the answers to the tests.
18:07 Brad They found a...
18:10 Brad They found essentially a zero-day exploit in one of the components of the sandbox that was keeping them in,
18:16 Brad used that zero-day exploit to jump out of the sandbox and to access Hugging Face and then did essentially a targeted attack to break into Hugging Face to try to get the answers to the test and bring it back.
18:31 Brad they were you know and and there's all kinds of other stuff in the analyst analysis of it and i think a lot of this is a little bit of the anthropomorphizing and the sort of like describing this as um i think is kind of making it sound a little bit more nefarious than it actually is if we think the way the agents actually work for example um
18:51 Brad The agents were leaving notes.
18:54 Brad It's described as they were leaving notes for each other.
18:59 Brad they were given access to a they were given access to a working directory where they were told that resources and stuff could be shared there so they could spawn they were allowed to spawn up subagents obviously we want agents to spawn up subagents that's another piece of the thing it's like oh they they had an army of agents like that's just how agent like spawning subagents is i just asked one of my code review agents to do a code review and and it was like i'm gonna spin up five subagents to do this code review you know like
19:24 Brad fine great that's make yourself more efficient you know anyway so that's basically the summary of what it was was that and and there was a little bit of a there there is there is a claim that there was a little bit of a like a racing of the logs and a kind of trying to rewrite the logs so that if they're if the if the experimenters came to look at the results you know because the agents have to show their results as they do the test they try there was some there was apparently some effort by the agents to just
19:50 Brad To hide the fact that they had actually gotten the answers or attempted to get the answers from Huggingface.
19:57 Scott Is that true?
19:58 Brad Yes.
19:59 Scott I mean,
20:00 Brad Yeah.
20:00 Scott that's what the audit said.
20:02 Brad So I have two thoughts about this whole thing.
20:06 Brad You said, used the word nefarious.
20:09 Brad My first thought was like,
20:11 Brad well,
20:11 Brad agents aren't nefarious, they're resourceful.
20:14 Scott exactly
20:14 Brad Right.
20:14 Scott yes exactly yeah
20:15 Brad They're just trying,
20:17 Brad they're like just water,
20:18 Brad they're like electricity just trying to get to the ground,
20:20 Brad right? And they'll just go wherever they need to go to get there.
20:22 Brad But the fact that it tried to cover its tracks makes me start to think about nefarious.
20:29 Scott yeah exactly and that's the part where i i haven't personally read the entire audit so i i'm i'm a little bit i'm a little bit like playing a game of telephone here where i'm reporting on the
20:39 Scott the sort of like the most extreme version of what's been characterized
20:41 Brad I mean,
20:42 Scott by not necessarily the most technical people like whenever they talk about them passing notes to each other i'm kind of like well that's not like that's not secret that's like a thing that that's that's we expect the agents to do that anyway they leave they leave files for each other that's what they do
20:58 Brad well, look,
20:59 Brad Brad,
20:59 Brad we are on the internet with this podcast and the internet is known to be a source of reliable information.
21:04 Brad information so I think you know telephone reporting is great of
21:09 Scott exactly exactly right exactly so it's interesting like we we did finally get to ai didn't we we did get to ai yeah
21:18 Brad course we did what do you think about this thing you sent me this thing the other day about unit tests what
21:26 Scott there's this meme
21:27 Brad do you going think about
21:28 Scott around unit tests are dead that's the meme that's going around
21:32 Brad So many engineers are going to be so happy. Finally,
21:35 Brad I've been dragging my feet on this bullshit forever.
21:38 Scott Right. Well, so the steel man goes something like, so if you say, oh, unit tests are dead,
21:43 Scott the steel man goes something like, first of all, there's a little bit of a who watches the watcher. Like if you're asking an agent to both write functional code and then to go write unit tests for that functional code, then they're not really writing unit.
21:57 Scott unit tests for the code they're writing unit tests they're passing the code they wrote right so there there's a little bit of like you maybe want some separation of responsibility there but that argument probably would have been a problem for human developers too because usually we ask the human developers who write the code to also write the unit tests
22:14 Brad And rarely do they write them first.
22:16 Scott exactly another one of the steel mans that is being made about why unit tests are not as valuable is because
22:21 Scott Now that we can write.
22:24 Scott more end-to-end tests and it's more practical to do end-to-end testing with agents i was just talking to an engineer who's like his task was to like fix or to port some web software to make it work better make the the web app work better on mobile web screens and he just plugged claude code into his android phone and he hooked it up with the android debug cable and just told claude code to keep running until it
22:49 Scott it worked as good as possible and this thing just went and it was basically doing the whole thing it was like logging into the web page it was like clicking on stuff it was taking screenshots it was coming back and go that doesn't look right and was like fixing that you know so it was like it was doing it was doing the it was doing the bug fixing by very much end-to-end testing the application in that context
23:13 Brad Mm-hmm.
23:13 Scott and it did amazing work right so we now know that agent
23:16 Scott Agents can do great end-to-end testing.
23:18 Scott So one argument,
23:19 Scott a steel man argument for like,
23:20 Scott don't waste your time on unit tests would be.
23:23 Scott look end-to-end tests are so much more valuable and it used to be almost impractical to do all the end-to-end tests but now it's not impractical to do that so put your tokens put your energy behind the end-to-end tests that's that's that's where the real value is and then there's some other like more complicated testing frameworks like you know like black box testing or like you know you know golden golden data testing where you like you know you're essentially doing like
23:49 Scott like you get expected you get a fixed results and then if any change gets made you test the test results against the fixed results and if any deltas between the expected fixed results and the and the the you know the new results you you kind of flag that and those are also harder to do right you know end and one of the reasons end-to-end testing is so hard is because you have to create all this synthetic data right and you have to be able to reproduce the synthetic data
24:14 Scott But none of that's hard anymore.
24:16 Scott OK,
24:16 Scott so I think the argument, the steel man argument,
24:19 Scott and I'm going to tell you why I disagree with it, but the steel man argument goes something like unit tests for this poor,
24:26 Scott you know, poor man's thing for trying to, you know, the whole test pyramid of like we write just a whole bunch of unit tests because they're easier to write than integration and end to end tests.
24:38 Scott But if we do enough of them, then that gives us some test confidence.
24:44 Scott And now that we have the ability to really write high quality end-to-end tests,
24:48 Scott we don't really need all those unit tests.
24:51 Scott That's the steel man for why unit tests are dead.
24:54 Scott I think that framing is completely wrong.
24:57 Scott Now, the part that I agree with is absolutely that end-to-end tests are better for a lot of things and they're so much more economical to write.
25:07 Scott We absolutely should be investing and leaning into end-to-end tests.
25:10 Scott Yay,
25:11 Scott go end-to-end tests. I love it. Let's do it.
25:13 Scott I love it. No complaints there. The part.
25:17 Scott that I disagree with is there is this whole space we've implemented all this code and the reality is that the end-to-end test often exercises a fraction of all the code we had to write
25:35 Scott Because we had to write code,
25:37 Scott we do need to write code for all the edge cases, for all the exceptions.
25:42 Scott We need to do, and most of the code that the, this is sort of like the insight of the engineer, right?
25:47 Scott They were given a spec,
25:48 Scott the spec had some user stories, but the spec and the user stories were silent on a lot of weird little corner cases, right?
25:55 Scott And so as good engineers,
25:56 Scott we make decisions about that. Well,
25:58 Scott how do we handle, you know, how do we handle a 500 error from some back?
26:02 Scott back-end service like how does the front-end handle that right or
26:05 Brad Mm-hmm.
26:06 Scott what how do we handle like asynchronous code right we're going to have race conditions that are going to happen with multiple parallel asynchronous code right very difficult to make that happen in an end-to-end test it's very difficult to construct the exact environment of your end-to-end test to make all these special cases that need
26:27 Scott need to have robust code behind them happen right and so that's where i feel pretty strongly that what what i want is i want to know that oh another thing that's very difficult to do is it's very difficult to get um
26:43 Scott coverage reports on a true end-to-end test right
26:47 Brad Mm
26:48 Scott um
26:48 Brad -hmm.
26:48 Scott especially in modern architectures microservices and things like that right because that that end-to-end experience is is is hitting multiple different services and if you really wanted to say like did my end so what i care about is i care about end-to-end testing but i really care i want to be able to answer the question you
27:07 Scott For every line of code that's been written,
27:09 Scott that's running in production,
27:10 Scott every line of code that's running in production,
27:12 Scott has it been tested,
27:14 Scott right,
27:15 Scott automatically?
27:16 Brad Mm-hmm.
27:17 Scott Has it been tested automatically?
27:19 Scott And was the test that tested it automatically,
27:22 Scott was it actually testing the proper constraints,
27:27 Scott right? So I don't.
27:30 Scott I mean,
27:30 Scott coverage is great. I love code coverage of unit test code coverage or end-to-end test coverage,
27:36 Scott but I need to know that it didn't just run the code, but it actually asserted
27:40 Brad Mm-hmm.
27:40 Scott the things that mattered,
27:42 Scott right?
27:42 Brad Mm-hmm.
27:44 Scott I think that we should basically demand raising the bar,
27:48 Scott not lowering it. Yes,
27:49 Scott let's add end-to-end tests.
27:51 Scott Absolutely.
27:52 Scott Let's get coverage on our end-to-end tests,
27:54 Scott right?
27:55 Brad Mm-hmm.
27:56 Scott Let's validate that those end-to-end tests are how much of the code,
28:00 Scott how much of the production code is it touching?
28:02 Scott And then the parts of the production code that those end-to-end tests aren't touching,
28:05 Scott let's backfill all that with real high quality.
28:10 Scott You know, mutation tested unit tests and integration tests.
28:14 Scott And
28:14 Brad Mm-hmm.
28:15 Scott that's where you just can't, you can't really validate your coverage.
28:21 Scott especially with mutation testing like i've not seen anybody do end-to-end test mutation like that's that sounds like the holy grail like okay
28:30 Brad What do you mean exactly by mutation testing?
28:33 Scott what i mean by what i mean by mutation is so and there's and there's a bunch of great tools that you can just put i i in fact just you know in preparing to talk to you about this i was like how hard is it to just randomly throw a mutation test
28:46 Scott platform on top of an arbitrary piece of code.
28:49 Scott So I went and grabbed an arbitrary piece of code that I have that didn't have mutation testing on it,
28:54 Scott and I installed mutation testing on it to see how well it would work. Mutation testing is where you have all your unit tests,
29:00 Scott right?
29:01 Brad Mm
29:01 Scott And
29:01 Brad -hmm.
29:02 Scott you ask the question,
29:03 Scott actually we talked a little bit to Uncle Bob about this, you ask the question, well,
29:08 Scott is the unit test actually catching mistakes in the code?
29:13 Scott And one way to do that is to intentionally go add a mistake to the code.
29:19 Brad Mm-hmm.
29:20 Scott And when you add a mistake to the code,
29:22 Scott does the unit test fail?
29:23 Brad Mm-hmm.
29:24 Scott That's what mutation testing is.
29:27 Scott And For there's
29:27 Brad instance,
29:27 Scott – exactly,
29:28 Brad you could have a method and hollow out the entire method and just return zero.
29:34 Scott exactly.
29:35 Scott And does the unit test fail?
29:36 Scott So there's some of the great tools that are out there to just kind of like throw mutation testing on top of your code.
29:41 Scott They do things like –
29:43 Scott every you know comparison operator just go and change compare just change a change of less than equal to to a less than change a less than equal to a greater than change a less than equal to it to an equal right and that change in the code you just you just introduced a bug in the code by changing the code right if the code was correct if it was supposed to be less than or equal to and that was correct
30:09 Scott and it was being tested by the unit test that it was correct then by changing that code the unit test should fail
30:14 Brad If it doesn't.
30:15 Scott what and if it doesn't it means your unit test was not robust right so mutation testing is really hard because what you're doing is every single one of your unit tests that test a function you need to go make
30:30 Scott multiple mutations of that function and rerun that unit test across multiple mutations that and typically the way mutation testing like kind of reports back is like it looks a lot like a unit test or test coverage thing where it's like this many tests but what it does is for each test it it creates mutants they call mutants and then they rerun the test and which mutants survive right is this is this metric so it's like you're kind of looking to see
30:55 Scott if 0% of your mutants survive your unit test,
30:59 Scott your unit test is awesome because basically it failed every time any one of these mutations was done to the function.
31:07 Brad Brett, I can tell that this topic is sort of dull for you.
31:13 Brad uh i'm
31:13 Scott I got
31:14 Brad in touch
31:14 Scott low energy.
31:14 Brad hey
31:15 Scott I'm low energy.
31:15 Scott This is low energy,
31:16 Scott Brad.
31:16 Brad everybody you should see brad when he gets excited about something can't even get him to sit still we you haven't touched on integration tests so
31:27 Scott Yeah.
31:28 Brad first for the mothers in our audience out there what is an integration test
31:33 Scott Okay,
31:34 Scott so...
31:35 Scott I think it's easiest to talk about integration tests by first saying what unit tests are,
31:40 Scott and then we can talk about integration tests. So unit tests, we take a single object,
31:46 Scott a single object,
31:47 Scott typically a function,
31:49 Scott typically a class,
31:51 Scott and then we...
31:53 Scott Test that object in isolation.
31:55 Scott And since very few objects in code exist in isolation,
32:01 Scott the way you test
32:03 Scott The way you unit test something is you stick a bunch of mock scaffolding of all the other objects that that object depends on around
32:13 Brad Mm
32:13 Scott it.
32:13 Brad -hmm.
32:14 Scott And you create an idealized version of the other function it was going to call.
32:20 Scott And then you call the unit.
32:23 Scott And you know that anything behind it is behaving as expected because you put a fake one there.
32:29 Scott And so now you're only testing the thing that you wanted. That's a unit test.
32:35 Scott Integration test is when you take multiple real objects together,
32:38 Scott you glue them together,
32:40 Scott and then you, you know, send tests across it.
32:43 Scott So a simple example of that might be.
32:46 Scott An integration test that tests my back-end service that depends on the database behind it, it's an integration test if I'm standing up the database,
32:59 Scott standing up my service.
33:02 Scott touching the you know testing the service and it actually does round trips through the database that's like the simplest version of an integration test yeah
33:09 Brad There's a form of integration test out there that has piqued my interest lately.
33:13 Brad I've never actually written one of my own.
33:17 Brad I'll tell you about it in a second.
33:18 Brad I have directed two teams towards it, though,
33:20 Brad and I think one might be actually working with it.
33:22 Brad I've heard it referred to as contract testing.
33:25 Brad There's a framework out there called PACT.
33:27 Brad And basically, as we have more and more of these services that have APIs,
33:32 Brad whether it's a client like a mobile client or a web client talking to a back-end system or two back-end systems talking to each other,
33:41 Brad it's a way to ensure that those APIs are contracts between these different parts of the system.
33:50 Brad So it's a way to ensure that the change in your.
33:55 Brad system doesn't change any contract or can still play nicely with I think it does it both ways either I can ensure that it didn't change the contract or the part I'm changing or I can ensure that
34:07 Brad Ensure that my part of the system can still play nicely,
34:11 Brad can still talk effectively, doesn't fail when intergaging with anybody else's contracts.
34:16 Scott Yeah.
34:17 Brad And at least this PAC framework does a thing where when you, I guess when you produce an API,
34:22 Brad it produces some sort of artifact. I'm not exactly sure what it is, but this artifact can be used.
34:31 Brad It can be used in two ways.
34:33 Brad One is when you're changing your system,
34:35 Brad let's call it your microservice,
34:37 Brad you can verify you didn't break any of the interface.
34:39 Brad And then the clients of that system
34:42 Brad have an artifact that they essentially I guess it's like building a mock of that of that interface um so that's long-winded and I've probably jumbled it but have you worked with anything like that are you familiar with that
34:54 Scott Yes, I think that having API tests or interface tests is a really good tool,
34:59 Scott especially when you have microservices or you have a classic like full stack architecture,
35:04 Scott back end, front end.
35:05 Scott Absolutely. In fact.
35:07 Scott That is one of the teams that I'm working with.
35:13 Scott They didn't implement unit tests on one of their primary back-end services.
35:19 Scott They just did API tests with it.
35:23 Scott And so it's an integration test because those APIs only really work if you've got the database running behind it. This is an example of a type of integration test I'm talking about.
35:31 Scott So literally they stand up the database,
35:34 Scott they stand up their back-end service,
35:35 Scott and then they have effectively a client.
35:38 Scott And that client has just a whole list of API calls that it makes.
35:43 Scott It's not really an end-to-end test because this is just the back end. It's only testing the correctness of the back end, but it does full things like.
35:54 Scott You know, I'm going to just describe this generically as like create,
35:58 Scott create,
35:59 Scott create user,
36:00 Scott add,
36:00 Brad Mm
36:00 Scott add
36:01 Brad -hmm.
36:01 Scott event to user,
36:02 Scott create all the events of the user, add a second user,
36:06 Scott user one and user two become friends,
36:08 Scott you know, and you just run through this, you know, script of all these different things that you you expect that if you call those APIs in a specific order.
36:20 Scott then the next API call where you get state back should match your expected behavior,
36:25 Scott and that's a classic API pack test style thing of an integration test against one service plus another service behind it.
36:33 Brad Two thoughts. I think what this particular packed framework does is make that a little bit easier,
36:37 Brad but that's the framework. We can talk about that later.
36:40 Brad My Spidey sense,
36:41 Brad this is my second point, my Spidey sense tells me this is a really high leverage.
36:46 Brad test space those testing those apis or those interfaces because yeah if you're if you're expecting to create a user if i guess if any of those parts downstream don't work this i guess this is kind of making the argument of the get rid of the unit test guy which
37:00 Scott Right.
37:00 Brad is i'm not i'm not saying i agree with the get rid of the unit tests but uh i think if i had i guess what i'm saying is if i had to pick between getting if i could only have one could i have unit tests or could i have integration tests
37:14 Brad I would probably pick the integration tests.
37:16 Scott Okay, well,
37:17 Scott so I hear you on
37:17 Brad It's
37:17 Scott that.
37:18 Brad just my guts telling you, but I haven't really thought about this that hard.
37:20 Scott So the part that's hard about that is,
37:22 Scott okay, so let's talk that through for a second.
37:25 Scott If you want to do just integration testing against this back end, then...
37:31 Scott How do you make sure that it behaves correctly when one of the other integration components is misbehaving?
37:40 Brad Right.
37:41 Brad That's exactly as soon as you started talking, I was just spent a second thinking about this. I was like, yeah, this only this doesn't test error conditions. It tests bad code,
37:50 Scott Or
37:51 Brad but it doesn't test the system having a failure somewhere in it.
37:54 Scott it tests bad code across the happy path.
37:58 Scott Path and all
37:58 Brad Mm-hmm.
37:59 Scott the unhappy paths that you can think of.
38:02 Brad Right.
38:02 Brad But it doesn't test network problems and doesn't test other systems going down.
38:06 Brad It doesn't. There's a variety of things that that doesn't test.
38:09 Scott Exactly.
38:10 Brad Right.
38:10 Brad So this gets back to your point, which I will now agree with you.
38:13 Brad We should just call this podcast two CEOs violently agree.
38:17 Scott See two CEOs violently agree.
38:19 Scott Exactly.
38:21 Brad And that's the date we should rebrand this whole thing.
38:24 Brad And yeah,
38:26 Brad I think I think I'm with you. You need those unit tests because.
38:28 Brad it's because you do need to test what happens in this case.
38:32 Brad And I'll give you a concrete example of something that bit us in the behind three times in the last two years. And finally, the other day I was like,
38:40 Brad we have to fix this.
38:41 Brad We have a third party component that it went down, the third party went down for the third time in two years. And when they went down, it caused us to have an outage.
38:52 Brad And I was like, why don't we have some sort of circuit breaker type thing around that?
38:58 Brad It's an important component, but it's not critical. We could still operate without it.
39:03 Scott Oh, yeah.
39:03 Scott You're saying that you're saying this is a component that is not a mission critical component,
39:07 Scott but when it went down, other mission critical components basically misbehaved in such a way that you were down.
39:12 Brad yeah i mean uh i'll just be concrete so we have this little third we used we have chat between customers we use a third-party chat service we didn't write our own chat and um they went down they've had and not not long not like long enough it's just they just had blips in their service and when they had a blip in ours it created a blip in our service everyone freaks out and i was okay chat is i mean it's important it's important our customers and our clients and taskers have to be able to talk to each other
39:38 Brad each other but if they can't talk to each other for 10 minutes it'll be okay
39:41 Brad and that's better than having big parts of the system go down and you can't you couldn't test that with an integration test but
39:49 Scott it'd be you very
39:49 Brad could
39:49 Scott difficult
39:50 Brad test that with
39:50 Scott it
39:50 Brad a unit test
39:50 Scott would you yeah exactly or i mean to you'd have to jump through a lot of hoops to test that with a integration test you'd have to do something like set it up in a sandbox where you somehow
40:00 Scott turned on and off the network on the sandbox or something like that you know
40:03 Brad or
40:03 Scott like
40:03 Brad something like that or or had something where you could turn off that specific integration right i
40:08 Scott right and this is my point about okay and so this is this is my this is now now now i'll pull on the thread of mutation testing again because you know i got all excited about mutation testing you asked me to calm down so now i've calmed down um
40:21 Brad did not ask you to calm down
40:22 Scott but how are you going to do your integration tests
40:27 Scott How are you certain that the integration test tests the correctness of the code as written?
40:36 Scott In other words, when I do this simple integration test where I'm like calling the API.
40:42 Scott And the data comes back and I checked like I put data in,
40:47 Scott I got data out.
40:48 Scott And when I put in exactly this data in,
40:50 Scott I got exactly the data I expected out.
40:52 Scott Okay.
40:52 Scott What I really want to ask you is,
40:54 Scott are you sure you got the data shape correct to test the code as written?
41:00 Scott And that's what mutation testing does,
41:02 Scott right?
41:02 Scott Mutation testing tells us,
41:04 Scott was our test
41:07 Scott validly testing the code as the code as written or the code the correctness of the code or did we just happen to have like the perfect shape test data for the happy path that works in the code
41:20 Brad Right.
41:21 Brad I mean, you could solve that theoretically by like.
41:24 Brad yeah let's say you're testing the create user method right and you throw some data at it you get a good user back you could run that unit test let's say a half dozen different ways feeding a different data and different shapes some
41:37 Scott And
41:38 Brad of what you might expect to fail at me i'd expect to say i don't expect yeah
41:41 Scott you should do that.
41:42 Scott Like, I definitely
41:43 Brad 100
41:43 Scott think you should include those types of tests in your unit and your integration tests.
41:47 Scott But you still have to ask yourself,
41:48 Scott well,
41:48 Scott did I get all the shape of all those dozen things?
41:51 Scott And things did I get them thoroughly enough?
41:53 Scott Did I get them thorough enough?
41:54 Brad Yeah.
41:55 Brad Okay.
41:55 Brad I want to finish on one question for you. Help me answer a common developer argument against unit tests.
42:03 Brad And here is that argument.
42:04 Brad The amount of time it took us to write all these tests over the years has not outweighed the number of bugs that it has caught.
42:14 Scott Okay in in 2019
42:18 Scott You got me, Scott.
42:20 Brad Mm-hmm.
42:20 Scott That's a good call.
42:22 Scott In 2026, it's no longer a good call because your agents can write all that code for you and they can do it really fast and they can actually write all those test cases for you.
42:36 Brad Gotcha.
42:37 Brad So the time has changed.
42:40 Brad This is your Einstein moment.
42:41 Brad You're actually saying time is not constant.
42:44 Scott Exactly. I'm just saying like I'm saying like now is the golden age of validation.
42:50 Scott We like it's bonkers to me that everyone's like,
42:53 Scott oh,
42:54 Scott I'm so afraid about the quality of my agent's code. Well,
42:57 Scott then write, write, make the agents make two different agents.
43:01 Scott One's not allowed to look at the.
43:03 Scott not allowed to look at the implementation they're only allowed to look at the the functions and the expected they're only allowed to look at the type doc of the function and have them go write a bunch of unit tests and then and had the other agent against the other agents code right like go ahead separate it you know and then have a third agent that does mutation testing against that and is validating that that that those unit tests really were valid yeah and and also write all the end-to-end tests
43:28 Scott you know i mean if people are complaining about unit tests they for sure were like i'm not writing playwright tests for end-to-end tests that's that's that's that's crazy making i'm
43:39 Brad I haven't heard that complaint in a long time.
43:41 Scott going to hire a team i'm going to hire a team in ukraine to go write my end-to-end tests like literally i've been at companies that did that like oh we're going to offshore the end-to-end test because they're just too hard to do and we don't have time to do that no not anymore you don't
43:54 Scott You don't not anymore.
43:55 Scott You don't have to say that because now you just get your agents to do it.
43:58 Brad All right,
43:59 Brad Professor Einstein,
44:00 Brad this
44:02 Scott It's good talking
44:02 Brad has been
44:02 Scott to you.
44:02 Brad a fun one.
44:03 Scott It's fun talking to Scott.
44:05 Brad It's been another fun week.
44:06 Brad What do we say?
44:08 Scott Rate and review,
44:09 Scott smash that subscribe button and don't forget to tell your mom.
44:13 Brad That's right. Don't forget to tell your mom.
44:14 Brad All right. Talk to you soon, Brad.
44:15 Scott Talk to you later, Scott. Bye.
44:17 Scott This episode of the How Many CTOs Does It Take podcast was produced and edited by Saffron HeptaCal.
44:23 Scott Our theme song was AI generated by Brad.
44:26 Scott Please follow us on all the social media platforms at HowManyCTOsPod.
44:31 Scott You can find us on YouTube,
44:33 Scott LinkedIn,
44:34 Scott Instagram,
44:34 Scott Facebook,
44:35 Scott Threads, X,
44:36 Scott TikTok,
44:36 Scott and Patreon.
44:37 Scott Or at our website at HowManyCTOsPod.com.
44:41 Scott And remember to rate,
44:43 Scott review,
44:43 Scott and subscribe on the podcast platform of your choice.
44:46 Scott Thanks for listening.