From Specs to Monitoring: A CTO's Guide to QA Best Practices

0:00 Scott: The part of test automation that's always been the most challenging is that user interaction testing,

0:04 right?

0:04 Those are always the ones that are the most brittle,

0:06 the hardest to get to work well, but in an AI world,

0:10 they actually might be the most important.

0:29 Brad: All right. Welcome, everybody,

0:30 to another episode of the How Many CTOs Does It Take podcast.

0:33 I'm Brad Hefta-Gaub, and this is my co-host.

0:36 Scott: Scott Porad.

0:38 Brad: And Scott,

0:39 today,

0:39 what are we going to talk about today?

0:42 I want to talk about.

0:44 QA and how what what are the gold standard what is the gold standard for QA quality assurance as part of web apps and honestly I kind of want to pick your brain a little bit about this Scott because I have realized as I've looked over the last decade or so of my career I've never really felt like I've gotten it right

1:09 But how about that for a big topic?

1:12 Scott: I don't know that I have gotten it right either,

1:15 and I actually,

1:15 but I do know that I feel like I have some contrarian views about it, which

1:20 Brad: Okay.

1:21 Scott: should be interesting.

1:22 But let me start with a philosophical question,

1:24 not

1:25 Brad: Yeah,

1:25 okay.

1:25 Scott: a question,

1:27 philosophical comment.

1:29 I think when a lot of people talk about QA or quality assurance,

1:33 they're talking about a person who tests something,

1:38 clicks

1:38 Brad: Okay.

1:39 Scott: through an app or does something like that, right?

1:44 And I've always sort of preached QA begins way,

1:48 way, way upstream.

1:51 Did we have the right requirements?

1:53 Did we have the right scope?

1:55 Did we build the right technical design?

1:59 Did we do a good code review?

2:02 Did we write test automation to prevent defects?

2:05 Did we write monitoring to detect defects?

2:08 Brad: Yeah.

2:09 Scott: That's all quality assurance,

2:12 right?

2:12 You can actually even go further than that and say,

2:14 did we do small?

2:18 commits and small pull requests so that it was easy to review.

2:25 There's a lot of different things all along the whole SDLC that leads to quality. That person at the end who clicks through it,

2:37 if they are your safety net.

2:40 Brad: They're your only safety net.

2:41 Scott: If they're your only safety net,

2:43 then, you know, you're probably effed.

2:47 Brad: Hey, first of all, I love that framing that I don't I you're not going to get an argument with me on that.

2:54 But to be fair, I really wanted to talk a little bit about the how do we verify that the code that we've written is behaving the way we wanted it to behave.

3:05 Scott: Yeah

3:06 Brad: And so of the of the chain of stuff that you described.

3:12 I think I'm asking about the middle third of that.

3:17 Scott: Yeah.

3:18 Well,

3:19 where I've had the most success in my career,

3:22 we leaned very heavily on two things.

3:26 A lot of test automation,

3:27 unit tests, all different types of test automation.

3:32 We did not have manual.

3:37 CUA testers doing stuff.

3:38 Brad: Interesting.

3:39 Scott: We felt like engineers should be responsible for what they're shipping.

3:45 We had a very...

3:48 Well,

3:48 I should tell you about a bunch of the practices we have,

3:50 okay,

3:50 because they're actually all part, they're all part of the package.

3:53 But I guess I said there were two things.

3:54 We had a lot of test automation,

3:56 we had a lot of monitoring, but there was a specific type of monitoring.

3:59 Often people monitor for when things go wrong.

4:02 You can also monitor for when things go right and log when things go right and then set up monitors and say,

4:07 the thing that's supposed to be going right isn't going right anymore,

4:10 right?

4:11 And I think that's a...

4:13 different mindset about monitoring that can result in like better detection of problems.

4:24 So I'll dig into that for a second.

4:28 When we were doing that,

4:29 we were doing a lot of logging and monitoring around user behaviors.

4:33 So I'll give you a concrete example.

4:39 We knew that, you know, customers came to our website and our mobile app and they ultimately entered a credit card and to book something.

4:48 And so we started logging how often credit cards got added.

4:52 And then we put that into Datadog and we just I can't remember exactly how we did it, but I actually can't remember. I'll come back to that in a minute. We put it in the Datadog and then we put a monitor on that and an anomaly monitor so that.

5:05 We have enough volume that it was pretty consistent throughout the day. You should be having a relatively steady stream of people adding credit cards to their accounts.

5:14 I mean, it's going to be lower on Christmas Day than it's going to be on, you know, Black Friday or whatever,

5:18 but it is.

5:22 you're going to be able to monitor that and put an anomaly detection on it so if you ship a bug where all of a sudden we broke credit cards you're not seeing the user behavior like something that's tracking our user behavior has changed and i actually remember how we implemented that

5:38 I don't remember the details,

5:39 but it was roughly something like we had everything that inherited from some sort of base object and on the save method of that object, we just put logging in. So we were actually logging the save of every object. And if you think the save of every object is effectively a user behavior,

5:53 somebody did something that needed to be recorded,

5:55 right?

5:56 They added an item to their cart,

5:58 they added a credit card,

5:59 whatever it was.

6:00 And so that became a very powerful tool for us to help.

6:03 detect problems that were happening but let's go a little bit further upstream

6:08 Brad: Hold

6:08 Scott: oh

6:08 Brad: on a second,

6:08 Scott: yeah

6:09 Brad: Scott.

6:09 So I think what you're talking about there,

6:11 I think this sounds like a really great point. And I feel like, you know, again,

6:15 like we kind of talked a little bit about this and in past episodes that we've had guests that are sort of like made this point.

6:21 So what I'm hearing you talk about right now is kind of the idea that production safety nets.

6:28 And

6:28 Scott: Yeah.

6:28 Brad: something like the idea that like your users aren't your testers, but production is ground truth.

6:35 Whatever is happening in production at scale is ground truth about the effectiveness of your product.

6:43 Scott: Yeah.

6:43 Brad: And so this term of RUM or real user monitoring and tracing the idea of like using your telemetry, whether it's open telemetry or Datadog or whatever you said,

6:54 and it's capturing everything that's happening and you can you should be able to say in a steady state.

7:02 you know 100 you know like 100 of transactions end with the person the shopping cart ended you know or whatever and so then if that all of a sudden changes you know something has gone wrong

7:15 Scott: Yes, I learned something.

7:17 You know the book Anna Karenina?

7:20 Brad: yeah

7:22 I mean, I don't have it memorized,

7:23 but.

7:24 Scott: Do you know the first line of that book?

7:26 It's a famous first line.

7:28 It said all happy families are alike.

7:30 Every unhappy family is unhappy in its own way.

7:33 Brad: Yeah.

7:34 Scott: And so apply that to technology.

7:37 It's often easier to.

7:41 verify that something's happening correctly than it is to because things go wrong in all sorts of unexpected ways that's why it's going wrong right so but it's you can under you can understand the pattern of a correct thing more easily and so it's often better to try and look at the things that are going well and eliminate them and then you got all the rest over all the leftovers and say if there are problems here or not

8:06 Brad: So I like that idea a lot and I think that there's value in that and I agree that that's part of the, you know, kind of like the best practices quality guide,

8:18 Scott: Mm-hmm.

8:18 Brad: right?

8:21 But I guess one of the things that I.

8:25 And, you know, maybe this will go back to an episode we've talked about in the past about this notion of tech debt versus product debt.

8:31 But one of the things that I wonder about with quality is you started with are our requirements really well documented?

8:40 And are the and so if you have high quality requirements and you have high quality descriptions of what you want the behavior to be,

8:47 then downstream you can check against that, you know, like acceptance criteria.

8:52 et cetera but

8:52 Scott: Yeah, that's

8:53 Brad: what if you don't have high quality requirements or you've lost your requirements into the ages of of product debt or something like that you know like

9:03 Scott: a good question.

9:04 Brad: early on early on in a lot of ventures you kind of don't know what's going to work right you're kind of doing experimental stuff you're throwing stuff against the wall you're trying to see if it works like i don't like

9:15 What if I don't really know what the expected behavior is or I'm moving?

9:19 I'm not good at describing the expected behavior.

9:23 So one of the earliest things,

9:25 one of the earliest bugs that gets reported is, oh,

9:28 the product team says,

9:29 oh,

9:29 that didn't act the way I thought it was going to act.

9:31 And you're like,

9:32 well,

9:32 but,

9:33 you know,

9:33 the spec was silent on this issue. You're describing a thing that was not covered in your acceptance criteria.

9:40 Scott: Well, yeah,

9:41 I mean, I learned a phrase a long time ago,

9:43 which I haven't used in a long time, but it comes to mind as a spec bug.

9:47 That was a bug in the spec.

9:48 Brad: Yeah,

9:48 I used to always say.

9:50 I used to always say the spec is silent on that issue.

9:52 Scott: Yeah.

9:52 Brad: The spec didn't say anything about that.

9:54 If you don't say it in the spec,

9:57 you've left it up to the whims of the developer.

10:00 Scott: Well,

10:00 that, okay,

10:01 so to answer your question,

10:03 and that's a tough one, but I think there's a couple quality steps that I didn't talk about earlier that I think are useful in this case.

10:12 One of them definitely is the sprint demo or at least just a demo before you release it, like just showing it to the whoever requested it. Like this is what we built. Is this what you wanted?

10:23 I mean, that's what Scrum and Agile always said, right?

10:26 You know, lots of people, I think we've talked about this here, how you can set up different.

10:31 you know, temporal environment,

10:33 formal environments for different builds,

10:34 like even they give the request,

10:36 hey,

10:36 wait, I've made your change,

10:37 I put it in this environment for you, go take a go experiment with it.

10:41 I worked at companies where product management owned QA,

10:46 the

10:46 Brad: By

10:46 Scott: point of view was,

10:47 hey,

10:48 Brad: the way, I support that idea a lot.

10:50 Scott: yeah.

10:50 Brad: That's a really interesting configuration.

10:53 Scott: create create a full loop right we're going to tell you what we want you're going to do it we're going to verify that it was done the way we wanted it done right and they they own the that qa process um so i think those three things right

11:08 sewing the doing it actually doing a demo before or incremental demos along the way but doing a demo before you release it can surface a lot of those things where there's silence i think is really effective for surfacing silence it's also good at surfacing silence not just when it doesn't work right but when the engineer is trying to build the thing to show it's like well you guys didn't say what you wanted me to do here so i need to be clear about that

11:35 Yeah,

11:36 and having minimal having the stakeholder review it themselves,

11:39 potentially having QA be part of that stakeholder team, I think are all key things to do.

11:43 Brad: Yeah, I had a we just literally just had a example of that sort of thing happen on Sprint's on one of the projects in the project I spend almost all my time on and the product team basically described a.

11:58 drag and drop feature for some component but these components have some interesting behavior where they kind of toggle between different states and

12:08 it's like they didn't really say what to do if you dragged something and then toggled it right and the developers at the time were like wait i feel like there's some i feel like we're not describing the whole thing here and you know and

12:20 Scott: Mm-hmm.

12:20 Brad: i encouraged them and kind of got the product team agreement to say yeah go ahead and code what we've said here and let us play with it and if it's and we'll we'll know whether we've missed something when once we can play with it right

12:33 And it became super obvious in this little sprint demo like,

12:36 oh yeah.

12:38 You're right.

12:39 We missed a big, we have a big gap in what this is and you coded it, you coded what we asked for.

12:46 And yeah,

12:47 let's not ship that. It's not, it doesn't, it doesn't work out. It doesn't work well enough for us to ship.

12:52 Scott: Well, I think what you did there, and I've seen this done well a lot of times, but it has to be the team has to have the right mindset,

12:58 is sometimes the requester

13:00 Brad: Yeah, exactly.

13:00 Scott: doesn't quite know what they want,

13:02 right?

13:02 So the engineer gets,

13:04 you know, they get an outline.

13:05 and give it to the engineer and the engineer the mindset is okay i'm just going to build you something and let's see what your reaction is to it like how about we do that some might call it a prototype or a proof of concept right like and that actually if the if the group of people working on this have the right mindset of like this engineer is intentionally going to bring me something that is not totally baked so that i can react to it

13:30 That can be a very good technique. In fact, I've started in many aspects of my professional life to embrace that approach of getting a first draft out, a rough first draft out early.

13:45 often which has bold choices in it simply to elicit reactions like i think i know what you want from me but i don't really know so i'm just going to get this in front of you and i'm going to give the provocative things in there so that i can actually get out of you what you want and you can do that with code too rather

14:02 Brad: Yeah.

14:02 Scott: there's something i have to come back to i talked about the monitoring how that was so important i really need to talk about

14:09 the core of

14:12 I would say the QA process that the monitoring was the tail end.

14:16 We required what I would call then an implementation plan or some people call a technical design doc.

14:22 The thoroughness of how that should be written was varied based on what was building.

14:29 Sometimes you just wrote it in the ticket. But the thing that we required is that was peer reviewed. Somebody else, you at least turned to your neighbor and said.

14:38 Hey,

14:38 Brad,

14:38 I'm thinking of doing this thing.

14:41 This is how I'm thinking of doing it. Do you think that's the right way to do it? You know, and the more complex it was, we would checklist for all these things of like the more things you checked on this checklist,

14:51 the more people you should get involved,

14:53 the more writing you should do. But that was the key part of quality is just have a conversation with another human up front about what you're planning on doing.

15:01 It's kind of like a flavor of rubber duck debugging,

15:04 which is just saying it out loud, just putting it on paper has the qualitative effect of like.

15:11 What's the investment? Just talking to another person.

15:14 Brad: And that's a great point about like baking in quality early on.

15:20 And, you know, there's other parts of that too that I'm assuming were also part of your SDLC like code reviews and post implementation reviews.

15:28 That's another great opportunity for sort of quality gates.

15:32 Scott: Well, let's talk about those two things because code review is a really big part for us.

15:37 We rely very heavily on code review.

15:39 As good as I've seen it anywhere,

15:41 and I have never been able to get the culture as strong at any other place I've worked,

15:45 which is because this company, we did not have manual QA testers at the end.

15:51 The engineers knew they were responsible for their own stuff. They also knew when they were doing a code review for their peers,

15:58 like it wasn't just like I'm going to skim it and make sure it works. We have a cultural expectation. You will.

16:04 Put this branch on your machine and execute it, right?

16:07 Brad: As

16:07 Scott: You will not

16:08 Brad: the code

16:08 Scott: just,

16:08 Brad: review

16:08 Scott: as

16:08 Brad: as the code reviewer.

16:10 Scott: the code reviewer, you will run this, right?

16:13 And you will.

16:14 just trying to get it to build on your own machine is a big piece of QA right and so that was an important cultural piece and that itself had a huge made a huge difference in QA and then you brought up something else oh you were what was the other thing you said about post implementation something

16:30 Brad: Well, it was like a post implementation review,

16:33 not not just a code review like I'm reading the code,

16:36 but like, OK,

16:37 yeah, really walk me through this.

16:39 What was your implementation?

16:42 So I had

16:43 Scott: Gotcha.

16:43 Brad: an implementation I planned.

16:45 Scott: Yeah.

16:45 Brad: Then I had an implementation I actually implemented it, probably variance, it probably had some variance from the plan.

16:52 Scott: Yeah.

16:52 Brad: And then I have the code that manifests the implementation,

16:56 right?

16:57 Scott: Yeah.

16:58 I want to take a tangent for a sec because we cannot do an episode without talking about AI,

17:03 Brad: I was just

17:03 Scott: although we have done it in the past.

17:04 Brad: about to say something about this.

17:06 Scott: I'm starting to get into this, we talk about pair programming,

17:10 I'm starting to get into this thing pair code reviews.

17:13 Because in a world where AI is maybe writing more and more of the code,

17:18 and you still have to get your lesser experienced developers to be more experienced developers.

17:24 I think the way I do that is to get is pair code reviews

17:28 Brad: I like that.

17:29 You know, we used to do, this is so long ago,

17:32 right?

17:33 So it was like.

17:34 You know, a web-based GitHub didn't exist at the time. Right.

17:38 But I remember it used to be that code reviews were literally we would get a bunch of us engineers would go to the conference room and the developer who wrote the system would pull up the code on the on the big screen TV.

17:49 Scott: On screen?

17:50 Brad: And then we'd

17:51 Scott: Absolutely.

17:51 Brad: all maybe have our paper copies of the code or we'd have our local copies on our computers and we'd be like,

17:58 OK,

17:58 Scott,

17:59 walk us through this new.

18:01 This new auth system that you've implemented and we're going to walk

18:04 Scott: Yeah.

18:04 Brad: it through, we're going to walk through it. We get to ask our questions. We get to almost like a doctoral thesis defense or something.

18:12 Scott: Yeah.

18:13 I remember those days.

18:14 Brad: What has happened to that?

18:16 Has the world moved too fast now?

18:18 Is that meaning

18:19 Scott: Social media.

18:20 Brad: no

18:21 Scott: Everyone's just staring at their screens all day

18:25 Brad: attention

18:25 Scott: long.

18:25 Brad: span.

18:26 Scott: People don't interact as much anymore.

18:28 Brad: Okay,

18:28 so I have a couple of quick reactions. I love your peer code reviews.

18:35 I like that idea.

18:37 I kind of want to challenge this in something we haven't talked about yet here in this conversation is how do you decide how much you're going to invest in each of these steps?

18:53 Right.

18:53 And I feel like, you know, I often I'm often working with, you know, startups that are pre MVP startups that are trying to like figure out like get product market fit, you know, or and and I feel like it's pretty hard for me as a CTO or a fractional CTO to tell an entrepreneur like you're going to need to spend.

19:19 20% of your budget on writing automated tests for this feature that you haven't gotten product market fit on yet.

19:27 Scott: It's such high ROI.

19:30 If you think about it as a portfolio, what I was trying to say when I mumbled my words is,

19:34 oh, yeah,

19:35 but it's such high ROI.

19:37 If you think about it as a portfolio,

19:39 like the ROI on the tests of the stuff that you keep.

19:45 around is tremendous and if you end up throwing out a bunch of code because those parts of your product didn't meet like didn't didn't fit with your customer well those were investments that were zeros but overall like

19:59 end up with a system that has a high level of test coverage is just such a high ROI.

20:04 Brad: Let me challenge that and say,

20:06 I think there's some framework that we would need to apply that has something like a.

20:13 kind of roi uh roi of test investment type metric right and for example i think we might want to include things like okay what's the what's the risk of the feature is there security privacy data loss what's the cost of failure like if the thing fails is it a revenue loss is it a trust loss is it you know or demos

20:37 how often are we changing this code is there like a bunch of churny code benefits from tests how complex is it how much runtime criticality is it you know performance media real time and you know do you have some kind of a compliance or contractual requirement right so this is kind of like this risk reward how much risk is there in the future how much cost of what happens if you fail you know what and to me it feels like there's some way to

21:04 way to like multiply all that stuff out and then kind of get yourself a guideline of like okay like how much do we need to invest in the in each of the components of the of the test off automation of this based on that all

21:17 Scott: Okay, so when we just talked about test automation,

21:20 we talked about all these phases

21:21 Brad: these phases

21:21 Scott: along the

21:22 Brad: i

21:22 Scott: way.

21:22 Brad: feel like like for example like if something is relatively low like let's call this the the investment

21:30 heuristic right so

21:31 Scott: Mm-hmm.

21:32 Brad: if we scored like risk cost of failure rate of change complexity criticality compliance if we scored all of those like on some zero to three type thing and then you added up all the points

21:45 Scott: Yeah.

21:45 Brad: then you're going to get a bunch of features that are like low

21:50 low roi heuristic right like

21:53 Scott: Mm-hmm.

21:53 Brad: they're not they don't have any risk to them if they fail that's not a big deal um it's relatively not complex code you know you've got no contractual obligation to it and so that if that score is low on like all these vectors then i feel like very low roi in investing in a lot of automation there and going with

22:15 with and going with just some kind of like a low end like smoke it just hold the hold the developers accountable for smoking it and or end-to-end only type stuff if it's a critical path feels like a reasonable investment but if it's like high risk high cost of failure you know high complexity

22:36 then like then go hardcore on strong unit tests strong integration tests throw all the bells and whistles at it and throw the canary the canary and the rum monitoring of

22:46 Scott: I would say yes in general, but I'll tell you how I think about it, like a little bit finer points.

22:54 Everything, let's just go,

22:56 everything,

22:57 code review for everything.

22:58 Brad: course

22:58 Scott: Everything

22:59 Brad: everything

22:59 Scott: gets code reviewed.

22:59 Brad: gets code reviewed totally agree with that

23:01 Scott: Everything gets code reviewed.

23:03 engineers save me but it's only one line it's so simple why should someone code review it it's like well if it's so simple it won't take them very long Exactly. right

23:09 Brad: I agree there.

23:09 Scott: so it's just let's just take that decision out of it everything gets code review the second thing i would say is everything gets unit tests right

23:19 if it's everything gets unit tests but then the other things

23:22 Brad: unit

23:22 Scott: how much time

23:23 Brad: tests

23:23 Scott: you spend yeah

23:23 Brad: though really are about like logic that are fractionable logic right so like when you say everything get unit tests like the instantiate the button ui the unit test for that tends to be you're not those those are low

23:40 Scott: low value you're saying it's low value but what i would also so to me there's a cultural piece there

23:46 Which is like the thing about the unit tests is in this heuristic of things is it's easy to make a mistake on the.

23:55 on the heuristic and think something is low value when it really should have been high value so like at least with the code review and the unit test which they sort of see as the foundational things like let's just say we're going to let's just we're just going to have a culture of just doing these things and it's going to be it's going to be it's going to set the right culture of having these things buttoned up but then when it comes to more complex tests your integration and end tests and it comes to your monitoring and it comes to your upfront

24:22 technical design planning,

24:23 those are the areas where I've varied it based on the heuristic that you're using.

24:31 But in the center of it, in my mind, is the code review and the unit test is just we're just going to do those because what happens is, oh, this, you know, this button,

24:41 it's, you know, if the unit test doesn't provide a lot of value, I'm not going to put a unit test on it.

24:47 But then somehow one way or the other someone breaks buttons and no buttons don't

24:51 Brad: You

24:52 Scott: work anymore.

24:52 Brad: also include in everything has to have a unit test. Do you have some kind of a code coverage branch coverage requirement as well?

24:58 Well, like must have 100% coverage,

24:59 must have 80% coverage,

25:01 what?

25:02 Scott: We put like a yes,

25:06 sort of, and they had 100% strived for 100%. In one company, we just had an agreement, hey, it's got to be.

25:16 You need to have a certain level of coverage.

25:17 Another company I was at, we actually had a danger rule that wouldn't let you commit if your coverage wasn't.

25:23 Brad: And somebody figured out how to add like 100 lines of comments and get their coverage up.

25:29 Scott: and there was another place that was that we had a different danger rule which is you couldn't commit if the over i think i can't remember how this worked maybe we didn't do this maybe we just thought about it but it was you can't commit if you're making the overall coverage go down

25:43 Brad: Yes.

25:43 Scott: right we just always want to be getting better yeah people game the system all the time and that's i mean i don't love it but you know i think most people don't try to game the system but i think what i was really trying to do with code review

25:56 code reviews and unit test coverage was just set a foundational culture of like these are two things where if we do them right can really save our bacon frequently and let's just take the decision making out of it because the decision it's it's clear when you don't need the tests and it's clear when you do need the tests it's the gray zone in between where you make the mistakes

26:18 Brad: Yeah,

26:18 Scott: so

26:18 Brad: yeah

26:19 Scott: let's just eliminate the gray zone by just saying we're going to do everything

26:22 Brad: Okay, so you sort of hinted to the question of AI in the case of, you know, like, let's start doing pair code reviews of AI so we can teach young developers how to read code and understand code.

26:35 I love that idea.

26:37 But let me ask this other AI question.

26:39 With the trend towards vibe coding.

26:42 I'm skeptical.

26:43 I am skeptical that the vibe coding out there, I feel like it's making every possible wrong choice in your quality chain of quality,

26:56 right?

26:57 Like lack of clear specificity of what the behavior is,

27:01 lack of a design plan up front,

27:07 lack of a review of that.

27:08 But, you know, and didn't you recently have a story with you were asking me, right?

27:13 You sent me this text message about, hey, my my AI code pilot wrote a bunch of unit tests,

27:19 but it's not running them.

27:20 What do I tell it? How do I tell it to run the unit test it wrote?

27:24 Scott: Well, it's funny. I did text you that recently, and it turns out that some configuration got messed up.

27:30 And so that wasn't the only problem I had.

27:33 It wouldn't do anything to my.

27:35 into my Oh, code it

27:35 Brad: man.

27:35 Scott: wouldn't even change it there was there was some setting somewhere i couldn't find it i actually had to uninstall this thing and reinstall it there was some setting somewhere i must have twiddled with that

27:45 you know like safe mode which is like okay well we'll tell you what to do but we're not going to do it to your code right and i couldn't turn it off but all of this talk the instant you mentioned ai made me think something

27:58 It's that final sort of end-to-end test,

28:00 which is becoming the most important,

28:02 because it's like you've mentioned many times the ephemeral app,

28:06 you know, people aren't going to buy apps anymore.

28:09 They're just going to say, AI,

28:10 make me this app that does this thing, right?

28:12 The code is going to be a black box.

28:14 Brad: Yeah.

28:16 Scott: Nobody's going to know what's going on. It could be spaghetti cut in there. It could be beautiful.

28:19 Nobody's going to know. Nobody's going to care as long as they ask, does the things we want

28:23 Brad: But

28:23 Scott: to do.

28:23 Brad: isn't that didn't you just didn't you just basically bring us right back to manual testing,

28:27 which is all ephemeral apps will just be built. They'll be completely bespoke. And the answer to quality is, did it did it meet the needs of the person who used it?

28:37 Scott: Well.

28:38 I mean, you can automate those tests, right?

28:41 When the users click the button,

28:42 does the thing go ding?

28:44 Brad: Yeah.

28:44 Scott: But you don't need to know if the code underneath there,

28:47 you don't need to code review that.

28:48 You don't need to have unit tests on that.

28:49 You don't need to know if it's any good.

28:50 The data could all be stored in one giant.

28:53 All your data could be in one text file that's stuffed into one field in one database.

28:57 It could be crazy sauce,

28:59 but hey,

29:01 I pushed the button, it went ding.

29:02 Brad: Yeah, the database is

29:03 Scott: I

29:04 Brad: literally

29:04 Scott: guess just it works.

29:04 Brad: in a giant string in the JS file of the main.js.

29:09 Scott: Yeah.

29:09 Brad: Yeah.

29:10 Scott: It's not the craziest thing in the world.

29:13 It probably is the craziest

29:14 Brad: I'm

29:14 Scott: thing

29:14 Brad: sure

29:14 Scott: in the world.

29:15 Brad: that worse code has been written.

29:16 Somebody might actually write an ETL job to copy usernames and passwords from one database to another.

29:22 You never know.

29:23 Scott: It's you never know.

29:24 But this is making me think the part of testing that has always been the most challenging test automation that's always been the most challenging is that user interaction

29:33 Brad: Yeah.

29:33 Scott: testing,

29:33 right?

29:33 right selenium

29:35 Brad: Yeah.

29:35 Scott: those kinds of i don't know there's different tools now is it cypress i think is one that's out there i haven't been up those are always the ones that are the most brittle the hardest to get to work well and engineers hate them the most but in an ai world they actually might be the most important degree

29:52 Brad: I agree.

29:52 Scott: Which is pretty interesting.

29:54 Brad: All right. So basically.

29:56 Scott: Okay, wait, there's one other. So you're going to go. So basically there's one other thought I have to put out there about the AI world.

30:01 I've thought about this a little bit,

30:03 but I haven't really formed any concrete thoughts. I'd love yours.

30:06 What's the role of test driven development in the AI world?

30:09 Or it's getting closer actually to the Gherkin stuff,

30:12 the requirements,

30:13 the BDD. What is BDD?

30:17 whatever the requirement-driven development is getting closer to that

30:20 Brad: Yeah.

30:20 Scott: stuff.

30:21 Brad: I mean, look, I.

30:23 I mean, all the AI agent code that I'm writing these days, I do try to tell it like from day one, like,

30:30 hey, I'm going to want you to have unit tests for everything. I'm going to want you to make a Cypress or Selenium end-to-end tests.

30:38 And as we go through each of these steps,

30:41 Mr.

30:41 AI agent,

30:42 like get, you know, get to the end point and.

30:45 I want unit tests that test the expected outcome of the step.

30:51 And so I, to your point about your chain of quality,

30:55 like, yeah,

30:57 absolutely.

30:57 It's way easier for me to write unit tests for everything.

31:01 The AIs are quite good at that and I can even throw it like I can run a coverage report and say,

31:07 hey, your unit tests only had.

31:10 75% coverage and I noticed that it's missing all of these branches from this code that you wrote can you please go write your unit tests that will test those corner cases as well and it's usually pretty darn good at getting those the only times that the only times that it will tend to like have to work extra hard or fail to do it is if you've got like some kind of defensive coding that's like you know gracefully handling null pointers or something like that you just

31:35 Because there's no version of it's like the code paths are almost unexecutable or something like that.

31:42 Scott: Yeah, you're making these you're making you've provoked thoughts in my head, you're making me be thoughtful about something you've pointed out a few times, the upstream stuff, and that the vibe coding is making the upstream stuff like get fuzzier because you're just talking to your computer,

31:58 but it actually makes me think that's in terms of quality like that's.

32:05 More important, you know, recently I mentioned to you, I can't remember if I mentioned it on the podcast or not, but I had seen these posts about people doing prompt engineering with Jason.

32:14 Brad: Yeah, yeah, you and I talked a little bit about that, I think, recently.

32:18 Scott: And I was thinking about that and I'm wondering.

32:21 If you agree with this, I think I thought, well, I think the reason this worked better is actually that JSON is just helping people structure their thoughts more

32:30 Brad: I

32:30 Scott: clearly.

32:31 Brad: agree with that.

32:31 Scott: If people actually wrote as clearly as the JSON is driving them to write.

32:39 Plain text might be fine,

32:40 but people just get there and do sloppy stuff in their words and the computer doesn't know what to do.

32:47 Brad: I was going to say like another quick little mini story here that I recently had a pair programming session with one of my junior engineers where he was working on a bug.

33:03 he was struggling to work on it.

33:05 And I said,

33:05 hey, why don't we get together and we'll just sit down and do some pair programming together on this.

33:10 And I focused a lot on the how are you prompting the AI to help solve this problem.

33:16 and um there was a couple things that i took away from it one thing that was really cool was he and i kind of like prepped ahead of time like i knew the bug i grabbed his copy of the code base i went and threw some prompts at it and watched the ai come up with answers and so i kind of had my own kind of what i thought it was going to be good at um and he did some really cool stuff where he actually

33:38 ran i was super happy to see this he ran the code he ran two different versions of the code he had like the develop branch so he knew like the main code base he had his local build he was inspecting it in chrome and and this was something like a rendering type issue so he was like inspecting the two different objects uh in the chrome dom to see like why is this the tailwind css class not being applied correctly so it was like he did some really good prep debugging that i thought was great and i you know kind of said hey

34:04 i said hey good job buddy um but when he prompted the ai he didn't he didn't basically give it a bunch of context right so he didn't say this code works the old code works i changed the following when i changed the following the bug got introduced he didn't give it the context of the other stuff he changed and i kind of was like hey

34:29 Why don't you try that? Why don't you try telling it what you changed?

34:33 That's going to give him give the AI a better clue about where the possible interactions were happening.

34:40 And it was night and day the difference in the AI's results. Like without that context, the AI just threw up some completely random ideas that weren't were completely wrong.

34:50 Scott: How did he tell the AI what he changed? Like, did he give it a diff, like here's my diff?

34:53 Or did he just say in words like this is what I did?

34:56 Brad: In this particular case, he said.

35:00 it was something like this was drag and drop related so it was like an object that didn't used to be draggable that he was making draggable so simply saying like this object used to not be draggable i started using the draggable action so like that was enough context for it to

35:16 Scott: That

35:16 Brad: that

35:16 Scott: was enough.

35:16 Brad: was enough um

35:17 Scott: Fascinating.

35:17 Brad: that being said i have definitely given it prompts like my branch is brad's feature x branch

35:26 It has this following bug.

35:27 The develop branch has this following bug.

35:32 uh it doesn't have the bug it works correctly here's the description of bug and so i'll give i'll sometimes give the ai and it'll often go and say cool i'm going to go check out those two branches and do a diff between them and that and you'll see it like working away going up checking out old branch doing the diff you know and then come back and go oh i see you made the following changes

35:51 here's where the problem is so i feel like so the reason i mentioned this only because of your json point like i think that's a

35:57 That's a really good point that like if you had a I'm doing a bug fix and the prompt is I'm going to make sure when you do the prompt for bug fix,

36:05 always give it the context of the branch. Where does the bug not happen?

36:12 Where does the bug happen?

36:14 You know, what was the change you were making when the bug started to happen,

36:18 et cetera.

36:20 Scott: Can I leave you with an anecdote today?

36:22 Brad: Yeah,

36:22 I

36:22 Scott: Have you seen the video going around the internet of the school teacher making the peanut butter and jelly sandwiches?

36:29 Brad: have not, but my immediate brain starts to picture what the joke is going to be about this.

36:36 Scott: It's not a joke.

36:37 I mean, this is.

36:39 uh this is an elementary school teacher you don't see the kids in the video you only see the teacher but you hear the kids and i would i would guess they're you know third fourth fifth grade somewhere in that range based on the tones of their voices and the kinds of things they say and they had an assignment there she's an elementary she's doing they have a descriptive writing

37:00 Brad: And she's following their descriptions to how to make a sandwich.

37:03 Scott: yeah and she has been she's asked them

37:06 write instructions for how to make a sandwich and so she has got a jar of peanut butter a jar of jelly and a loaf of bread and the instructions are you know you're supposed to write how to make a peanut butter and jelly sandwich and she says okay i'm gonna she's got the stack of papers right there she's like i'm gonna do exactly what you told me

37:29 and she doesn't in this video and it's adorable because the kids are laughing and they're like no no no so the first one is like get some bread and put peanut butter and jelly on it and so she puts the bread on the desk there and she puts the peanut butter and jelly right on top of the bag of the loaf of bread she's like is that what you meant you know they're all like no no no no then she gets another x one and it says um get some bread

37:54 and spread peanut butter and jelly on it or something like that and so she opens up the jar of jelly and takes her hand in and spreads it right on top of the bag of the wonder bread or whatever it is and spread it and they're all no no but this is a you know and it was a really powerful powerful illustration of what it means to be descriptive Yeah. and when we think about this AI in context

38:19 right context is so important because when someone says we'll take the bread and spread some jelly on it when you hear the word spread as adults or even maybe as a child you kind of like get a knife and use it to spread it on the bread but these computers are so smart yet so dumb and you have to just give them exquisitely detailed context to get them to do the thing that you want them to do

38:45 And how that relates to QA, which is the topic we started with, I have no idea,

38:50 but the video was adorable.

38:52 Brad: Love it. I'm going to have to link that video in the show notes of this episode.

38:57 Scott: I'll have to find it and I'll do that.

38:59 Brad: Great talking to you, Scott.

39:01 This was super helpful.

39:02 I appreciate your insight on the chain of quality.

39:05 Scott: Thank you. I'm glad it could be helpful.

39:07 It's fun talking to you too.

39:08 Brad: Everybody, please rate, review and smash that subscribe button.

39:13 Scott: You could be our one millionth subscriber.

39:15 Brad: He could be.

39:17 Scott: See everybody.

39:18 Brad: This episode of How Many CTOs Does It Take podcast was edited by Saffron Epigal.

39:23 Our theme song was AI Generated by Brad.

39:27 Scott: Please follow us on all the social media platforms at at HowManyCTOsPod.

39:32 You can find us on YouTube,

39:34 LinkedIn,

39:34 Instagram,

39:35 Facebook,

39:36 Threads, X,

39:36 TikTok,

39:37 and Patreon.

39:38 Or at our website at HowManyCTOsPod.com.

39:43 And remember to rate, review,

39:45 and subscribe on the podcast platform of your choice.

39:48 Thanks for listening.