Transcript: From Specs to Monitoring: A CTO's Guide to QA Best Practices
0:00 Scott The part of test automation that's always been the most challenging is that user interaction testing,
0:04 Scott right?
0:04 Scott Those are always the ones that are the most brittle,
0:06 Scott the hardest to get to work well, but in an AI world,
0:10 Scott they actually might be the most important.
0:29 Brad All right. Welcome, everybody,
0:30 Brad to another episode of the How Many CTOs Does It Take podcast.
0:33 Brad I'm Brad Hefta-Gaub, and this is my co-host.
0:36 Scott Scott Porad.
0:38 Brad And Scott,
0:39 Brad today,
0:39 Brad what are we going to talk about today?
0:42 Brad I want to talk about.
0:44 Brad QA and how what what are the gold standard what is the gold standard for QA quality assurance as part of web apps and honestly I kind of want to pick your brain a little bit about this Scott because I have realized as I've looked over the last decade or so of my career I've never really felt like I've gotten it right
1:09 Brad But how about that for a big topic?
1:12 Scott I don't know that I have gotten it right either,
1:15 Scott and I actually,
1:15 Scott but I do know that I feel like I have some contrarian views about it, which
1:20 Brad Okay.
1:21 Scott should be interesting.
1:22 Scott But let me start with a philosophical question,
1:24 Scott not
1:25 Brad Yeah,
1:25 Brad okay.
1:25 Scott a question,
1:27 Scott philosophical comment.
1:29 Scott I think when a lot of people talk about QA or quality assurance,
1:33 Scott they're talking about a person who tests something,
1:38 Scott clicks
1:38 Brad Okay.
1:39 Scott through an app or does something like that, right?
1:44 Scott And I've always sort of preached QA begins way,
1:48 Scott way, way upstream.
1:51 Scott Did we have the right requirements?
1:53 Scott Did we have the right scope?
1:55 Scott Did we build the right technical design?
1:59 Scott Did we do a good code review?
2:02 Scott Did we write test automation to prevent defects?
2:05 Scott Did we write monitoring to detect defects?
2:08 Brad Yeah.
2:09 Scott That's all quality assurance,
2:12 Scott right?
2:12 Scott You can actually even go further than that and say,
2:14 Scott did we do small?
2:18 Scott commits and small pull requests so that it was easy to review.
2:25 Scott There's a lot of different things all along the whole SDLC that leads to quality. That person at the end who clicks through it,
2:37 Scott if they are your safety net.
2:40 Brad They're your only safety net.
2:41 Scott If they're your only safety net,
2:43 Scott then, you know, you're probably effed.
2:47 Brad Hey, first of all, I love that framing that I don't I you're not going to get an argument with me on that.
2:54 Brad But to be fair, I really wanted to talk a little bit about the how do we verify that the code that we've written is behaving the way we wanted it to behave.
3:05 Scott Yeah
3:06 Brad And so of the of the chain of stuff that you described.
3:12 Brad I think I'm asking about the middle third of that.
3:17 Scott Yeah.
3:18 Scott Well,
3:19 Scott where I've had the most success in my career,
3:22 Scott we leaned very heavily on two things.
3:26 Scott A lot of test automation,
3:27 Scott unit tests, all different types of test automation.
3:32 Scott We did not have manual.
3:37 Scott CUA testers doing stuff.
3:38 Brad Interesting.
3:39 Scott We felt like engineers should be responsible for what they're shipping.
3:45 Scott We had a very...
3:48 Scott Well,
3:48 Scott I should tell you about a bunch of the practices we have,
3:50 Scott okay,
3:50 Scott because they're actually all part, they're all part of the package.
3:53 Scott But I guess I said there were two things.
3:54 Scott We had a lot of test automation,
3:56 Scott we had a lot of monitoring, but there was a specific type of monitoring.
3:59 Scott Often people monitor for when things go wrong.
4:02 Scott You can also monitor for when things go right and log when things go right and then set up monitors and say,
4:07 Scott the thing that's supposed to be going right isn't going right anymore,
4:10 Scott right?
4:11 Scott And I think that's a...
4:13 Scott different mindset about monitoring that can result in like better detection of problems.
4:24 Scott So I'll dig into that for a second.
4:28 Scott When we were doing that,
4:29 Scott we were doing a lot of logging and monitoring around user behaviors.
4:33 Scott So I'll give you a concrete example.
4:39 Scott We knew that, you know, customers came to our website and our mobile app and they ultimately entered a credit card and to book something.
4:48 Scott And so we started logging how often credit cards got added.
4:52 Scott And then we put that into Datadog and we just I can't remember exactly how we did it, but I actually can't remember. I'll come back to that in a minute. We put it in the Datadog and then we put a monitor on that and an anomaly monitor so that.
5:05 Scott We have enough volume that it was pretty consistent throughout the day. You should be having a relatively steady stream of people adding credit cards to their accounts.
5:14 Scott I mean, it's going to be lower on Christmas Day than it's going to be on, you know, Black Friday or whatever,
5:18 Scott but it is.
5:22 Scott you're going to be able to monitor that and put an anomaly detection on it so if you ship a bug where all of a sudden we broke credit cards you're not seeing the user behavior like something that's tracking our user behavior has changed and i actually remember how we implemented that
5:38 Scott I don't remember the details,
5:39 Scott but it was roughly something like we had everything that inherited from some sort of base object and on the save method of that object, we just put logging in. So we were actually logging the save of every object. And if you think the save of every object is effectively a user behavior,
5:53 Scott somebody did something that needed to be recorded,
5:55 Scott right?
5:56 Scott They added an item to their cart,
5:58 Scott they added a credit card,
5:59 Scott whatever it was.
6:00 Scott And so that became a very powerful tool for us to help.
6:03 Scott detect problems that were happening but let's go a little bit further upstream
6:08 Brad Hold
6:08 Scott oh
6:08 Brad on a second,
6:08 Scott yeah
6:09 Brad Scott.
6:09 Brad So I think what you're talking about there,
6:11 Brad I think this sounds like a really great point. And I feel like, you know, again,
6:15 Brad like we kind of talked a little bit about this and in past episodes that we've had guests that are sort of like made this point.
6:21 Brad So what I'm hearing you talk about right now is kind of the idea that production safety nets.
6:28 Brad And
6:28 Scott Yeah.
6:28 Brad something like the idea that like your users aren't your testers, but production is ground truth.
6:35 Brad Whatever is happening in production at scale is ground truth about the effectiveness of your product.
6:43 Scott Yeah.
6:43 Brad And so this term of RUM or real user monitoring and tracing the idea of like using your telemetry, whether it's open telemetry or Datadog or whatever you said,
6:54 Brad and it's capturing everything that's happening and you can you should be able to say in a steady state.
7:02 Brad you know 100 you know like 100 of transactions end with the person the shopping cart ended you know or whatever and so then if that all of a sudden changes you know something has gone wrong
7:15 Scott Yes, I learned something.
7:17 Scott You know the book Anna Karenina?
7:20 Brad yeah
7:22 Brad I mean, I don't have it memorized,
7:23 Brad but.
7:24 Scott Do you know the first line of that book?
7:26 Scott It's a famous first line.
7:28 Scott It said all happy families are alike.
7:30 Scott Every unhappy family is unhappy in its own way.
7:33 Brad Yeah.
7:34 Scott And so apply that to technology.
7:37 Scott It's often easier to.
7:41 Scott verify that something's happening correctly than it is to because things go wrong in all sorts of unexpected ways that's why it's going wrong right so but it's you can under you can understand the pattern of a correct thing more easily and so it's often better to try and look at the things that are going well and eliminate them and then you got all the rest over all the leftovers and say if there are problems here or not
8:06 Brad So I like that idea a lot and I think that there's value in that and I agree that that's part of the, you know, kind of like the best practices quality guide,
8:18 Scott Mm-hmm.
8:18 Brad right?
8:21 Brad But I guess one of the things that I.
8:25 Brad And, you know, maybe this will go back to an episode we've talked about in the past about this notion of tech debt versus product debt.
8:31 Brad But one of the things that I wonder about with quality is you started with are our requirements really well documented?
8:40 Brad And are the and so if you have high quality requirements and you have high quality descriptions of what you want the behavior to be,
8:47 Brad then downstream you can check against that, you know, like acceptance criteria.
8:52 Brad et cetera but
8:52 Scott Yeah, that's
8:53 Brad what if you don't have high quality requirements or you've lost your requirements into the ages of of product debt or something like that you know like
9:03 Scott a good question.
9:04 Brad early on early on in a lot of ventures you kind of don't know what's going to work right you're kind of doing experimental stuff you're throwing stuff against the wall you're trying to see if it works like i don't like
9:15 Brad What if I don't really know what the expected behavior is or I'm moving?
9:19 Brad I'm not good at describing the expected behavior.
9:23 Brad So one of the earliest things,
9:25 Brad one of the earliest bugs that gets reported is, oh,
9:28 Brad the product team says,
9:29 Brad oh,
9:29 Brad that didn't act the way I thought it was going to act.
9:31 Brad And you're like,
9:32 Brad well,
9:32 Brad but,
9:33 Brad you know,
9:33 Brad the spec was silent on this issue. You're describing a thing that was not covered in your acceptance criteria.
9:40 Scott Well, yeah,
9:41 Scott I mean, I learned a phrase a long time ago,
9:43 Scott which I haven't used in a long time, but it comes to mind as a spec bug.
9:47 Scott That was a bug in the spec.
9:48 Brad Yeah,
9:48 Brad I used to always say.
9:50 Brad I used to always say the spec is silent on that issue.
9:52 Scott Yeah.
9:52 Brad The spec didn't say anything about that.
9:54 Brad If you don't say it in the spec,
9:57 Brad you've left it up to the whims of the developer.
10:00 Scott Well,
10:00 Scott that, okay,
10:01 Scott so to answer your question,
10:03 Scott and that's a tough one, but I think there's a couple quality steps that I didn't talk about earlier that I think are useful in this case.
10:12 Scott One of them definitely is the sprint demo or at least just a demo before you release it, like just showing it to the whoever requested it. Like this is what we built. Is this what you wanted?
10:23 Scott I mean, that's what Scrum and Agile always said, right?
10:26 Scott You know, lots of people, I think we've talked about this here, how you can set up different.
10:31 Scott you know, temporal environment,
10:33 Scott formal environments for different builds,
10:34 Scott like even they give the request,
10:36 Scott hey,
10:36 Scott wait, I've made your change,
10:37 Scott I put it in this environment for you, go take a go experiment with it.
10:41 Scott I worked at companies where product management owned QA,
10:46 Scott the
10:46 Brad By
10:46 Scott point of view was,
10:47 Scott hey,
10:48 Brad the way, I support that idea a lot.
10:50 Scott yeah.
10:50 Brad That's a really interesting configuration.
10:53 Scott create create a full loop right we're going to tell you what we want you're going to do it we're going to verify that it was done the way we wanted it done right and they they own the that qa process um so i think those three things right
11:08 Scott sewing the doing it actually doing a demo before or incremental demos along the way but doing a demo before you release it can surface a lot of those things where there's silence i think is really effective for surfacing silence it's also good at surfacing silence not just when it doesn't work right but when the engineer is trying to build the thing to show it's like well you guys didn't say what you wanted me to do here so i need to be clear about that
11:35 Scott Yeah,
11:36 Scott and having minimal having the stakeholder review it themselves,
11:39 Scott potentially having QA be part of that stakeholder team, I think are all key things to do.
11:43 Brad Yeah, I had a we just literally just had a example of that sort of thing happen on Sprint's on one of the projects in the project I spend almost all my time on and the product team basically described a.
11:58 Brad drag and drop feature for some component but these components have some interesting behavior where they kind of toggle between different states and
12:08 Brad it's like they didn't really say what to do if you dragged something and then toggled it right and the developers at the time were like wait i feel like there's some i feel like we're not describing the whole thing here and you know and
12:20 Scott Mm-hmm.
12:20 Brad i encouraged them and kind of got the product team agreement to say yeah go ahead and code what we've said here and let us play with it and if it's and we'll we'll know whether we've missed something when once we can play with it right
12:33 Brad And it became super obvious in this little sprint demo like,
12:36 Brad oh yeah.
12:38 Brad You're right.
12:39 Brad We missed a big, we have a big gap in what this is and you coded it, you coded what we asked for.
12:46 Brad And yeah,
12:47 Brad let's not ship that. It's not, it doesn't, it doesn't work out. It doesn't work well enough for us to ship.
12:52 Scott Well, I think what you did there, and I've seen this done well a lot of times, but it has to be the team has to have the right mindset,
12:58 Scott is sometimes the requester
13:00 Brad Yeah, exactly.
13:00 Scott doesn't quite know what they want,
13:02 Scott right?
13:02 Scott So the engineer gets,
13:04 Scott you know, they get an outline.
13:05 Scott and give it to the engineer and the engineer the mindset is okay i'm just going to build you something and let's see what your reaction is to it like how about we do that some might call it a prototype or a proof of concept right like and that actually if the if the group of people working on this have the right mindset of like this engineer is intentionally going to bring me something that is not totally baked so that i can react to it
13:30 Scott That can be a very good technique. In fact, I've started in many aspects of my professional life to embrace that approach of getting a first draft out, a rough first draft out early.
13:45 Scott often which has bold choices in it simply to elicit reactions like i think i know what you want from me but i don't really know so i'm just going to get this in front of you and i'm going to give the provocative things in there so that i can actually get out of you what you want and you can do that with code too rather
14:02 Brad Yeah.
14:02 Scott there's something i have to come back to i talked about the monitoring how that was so important i really need to talk about
14:09 Scott the core of
14:12 Scott I would say the QA process that the monitoring was the tail end.
14:16 Scott We required what I would call then an implementation plan or some people call a technical design doc.
14:22 Scott The thoroughness of how that should be written was varied based on what was building.
14:29 Scott Sometimes you just wrote it in the ticket. But the thing that we required is that was peer reviewed. Somebody else, you at least turned to your neighbor and said.
14:38 Scott Hey,
14:38 Scott Brad,
14:38 Scott I'm thinking of doing this thing.
14:41 Scott This is how I'm thinking of doing it. Do you think that's the right way to do it? You know, and the more complex it was, we would checklist for all these things of like the more things you checked on this checklist,
14:51 Scott the more people you should get involved,
14:53 Scott the more writing you should do. But that was the key part of quality is just have a conversation with another human up front about what you're planning on doing.
15:01 Scott It's kind of like a flavor of rubber duck debugging,
15:04 Scott which is just saying it out loud, just putting it on paper has the qualitative effect of like.
15:11 Scott What's the investment? Just talking to another person.
15:14 Brad And that's a great point about like baking in quality early on.
15:20 Brad And, you know, there's other parts of that too that I'm assuming were also part of your SDLC like code reviews and post implementation reviews.
15:28 Brad That's another great opportunity for sort of quality gates.
15:32 Scott Well, let's talk about those two things because code review is a really big part for us.
15:37 Scott We rely very heavily on code review.
15:39 Scott As good as I've seen it anywhere,
15:41 Scott and I have never been able to get the culture as strong at any other place I've worked,
15:45 Scott which is because this company, we did not have manual QA testers at the end.
15:51 Scott The engineers knew they were responsible for their own stuff. They also knew when they were doing a code review for their peers,
15:58 Scott like it wasn't just like I'm going to skim it and make sure it works. We have a cultural expectation. You will.
16:04 Scott Put this branch on your machine and execute it, right?
16:07 Brad As
16:07 Scott You will not
16:08 Brad the code
16:08 Scott just,
16:08 Brad review
16:08 Scott as
16:08 Brad as the code reviewer.
16:10 Scott the code reviewer, you will run this, right?
16:13 Scott And you will.
16:14 Scott just trying to get it to build on your own machine is a big piece of QA right and so that was an important cultural piece and that itself had a huge made a huge difference in QA and then you brought up something else oh you were what was the other thing you said about post implementation something
16:30 Brad Well, it was like a post implementation review,
16:33 Brad not not just a code review like I'm reading the code,
16:36 Brad but like, OK,
16:37 Brad yeah, really walk me through this.
16:39 Brad What was your implementation?
16:42 Brad So I had
16:43 Scott Gotcha.
16:43 Brad an implementation I planned.
16:45 Scott Yeah.
16:45 Brad Then I had an implementation I actually implemented it, probably variance, it probably had some variance from the plan.
16:52 Scott Yeah.
16:52 Brad And then I have the code that manifests the implementation,
16:56 Brad right?
16:57 Scott Yeah.
16:58 Scott I want to take a tangent for a sec because we cannot do an episode without talking about AI,
17:03 Brad I was just
17:03 Scott although we have done it in the past.
17:04 Brad about to say something about this.
17:06 Scott I'm starting to get into this, we talk about pair programming,
17:10 Scott I'm starting to get into this thing pair code reviews.
17:13 Scott Because in a world where AI is maybe writing more and more of the code,
17:18 Scott and you still have to get your lesser experienced developers to be more experienced developers.
17:24 Scott I think the way I do that is to get is pair code reviews
17:28 Brad I like that.
17:29 Brad You know, we used to do, this is so long ago,
17:32 Brad right?
17:33 Brad So it was like.
17:34 Brad You know, a web-based GitHub didn't exist at the time. Right.
17:38 Brad But I remember it used to be that code reviews were literally we would get a bunch of us engineers would go to the conference room and the developer who wrote the system would pull up the code on the on the big screen TV.
17:49 Scott On screen?
17:50 Brad And then we'd
17:51 Scott Absolutely.
17:51 Brad all maybe have our paper copies of the code or we'd have our local copies on our computers and we'd be like,
17:58 Brad OK,
17:58 Brad Scott,
17:59 Brad walk us through this new.
18:01 Brad This new auth system that you've implemented and we're going to walk
18:04 Scott Yeah.
18:04 Brad it through, we're going to walk through it. We get to ask our questions. We get to almost like a doctoral thesis defense or something.
18:12 Scott Yeah.
18:13 Scott I remember those days.
18:14 Brad What has happened to that?
18:16 Brad Has the world moved too fast now?
18:18 Brad Is that meaning
18:19 Scott Social media.
18:20 Brad no
18:21 Scott Everyone's just staring at their screens all day
18:25 Brad attention
18:25 Scott long.
18:25 Brad span.
18:26 Scott People don't interact as much anymore.
18:28 Brad Okay,
18:28 Brad so I have a couple of quick reactions. I love your peer code reviews.
18:35 Brad I like that idea.
18:37 Brad I kind of want to challenge this in something we haven't talked about yet here in this conversation is how do you decide how much you're going to invest in each of these steps?
18:53 Brad Right.
18:53 Brad And I feel like, you know, I often I'm often working with, you know, startups that are pre MVP startups that are trying to like figure out like get product market fit, you know, or and and I feel like it's pretty hard for me as a CTO or a fractional CTO to tell an entrepreneur like you're going to need to spend.
19:19 Brad 20% of your budget on writing automated tests for this feature that you haven't gotten product market fit on yet.
19:27 Scott It's such high ROI.
19:30 Scott If you think about it as a portfolio, what I was trying to say when I mumbled my words is,
19:34 Scott oh, yeah,
19:35 Scott but it's such high ROI.
19:37 Scott If you think about it as a portfolio,
19:39 Scott like the ROI on the tests of the stuff that you keep.
19:45 Scott around is tremendous and if you end up throwing out a bunch of code because those parts of your product didn't meet like didn't didn't fit with your customer well those were investments that were zeros but overall like
19:59 Scott end up with a system that has a high level of test coverage is just such a high ROI.
20:04 Brad Let me challenge that and say,
20:06 Brad I think there's some framework that we would need to apply that has something like a.
20:13 Brad kind of roi uh roi of test investment type metric right and for example i think we might want to include things like okay what's the what's the risk of the feature is there security privacy data loss what's the cost of failure like if the thing fails is it a revenue loss is it a trust loss is it you know or demos
20:37 Brad how often are we changing this code is there like a bunch of churny code benefits from tests how complex is it how much runtime criticality is it you know performance media real time and you know do you have some kind of a compliance or contractual requirement right so this is kind of like this risk reward how much risk is there in the future how much cost of what happens if you fail you know what and to me it feels like there's some way to
21:04 Brad way to like multiply all that stuff out and then kind of get yourself a guideline of like okay like how much do we need to invest in the in each of the components of the of the test off automation of this based on that all
21:17 Scott Okay, so when we just talked about test automation,
21:20 Scott we talked about all these phases
21:21 Brad these phases
21:21 Scott along the
21:22 Brad i
21:22 Scott way.
21:22 Brad feel like like for example like if something is relatively low like let's call this the the investment
21:30 Brad heuristic right so
21:31 Scott Mm-hmm.
21:32 Brad if we scored like risk cost of failure rate of change complexity criticality compliance if we scored all of those like on some zero to three type thing and then you added up all the points
21:45 Scott Yeah.
21:45 Brad then you're going to get a bunch of features that are like low
21:50 Brad low roi heuristic right like
21:53 Scott Mm-hmm.
21:53 Brad they're not they don't have any risk to them if they fail that's not a big deal um it's relatively not complex code you know you've got no contractual obligation to it and so that if that score is low on like all these vectors then i feel like very low roi in investing in a lot of automation there and going with
22:15 Brad with and going with just some kind of like a low end like smoke it just hold the hold the developers accountable for smoking it and or end-to-end only type stuff if it's a critical path feels like a reasonable investment but if it's like high risk high cost of failure you know high complexity
22:36 Brad then like then go hardcore on strong unit tests strong integration tests throw all the bells and whistles at it and throw the canary the canary and the rum monitoring of
22:46 Scott I would say yes in general, but I'll tell you how I think about it, like a little bit finer points.
22:54 Scott Everything, let's just go,
22:56 Scott everything,
22:57 Scott code review for everything.
22:58 Brad course
22:58 Scott Everything
22:59 Brad everything
22:59 Scott gets code reviewed.
22:59 Brad gets code reviewed totally agree with that
23:01 Scott Everything gets code reviewed.
23:03 Scott engineers save me but it's only one line it's so simple why should someone code review it it's like well if it's so simple it won't take them very long Exactly. right
23:09 Brad I agree there.
23:09 Scott so it's just let's just take that decision out of it everything gets code review the second thing i would say is everything gets unit tests right
23:19 Scott if it's everything gets unit tests but then the other things
23:22 Brad unit
23:22 Scott how much time
23:23 Brad tests
23:23 Scott you spend yeah
23:23 Brad though really are about like logic that are fractionable logic right so like when you say everything get unit tests like the instantiate the button ui the unit test for that tends to be you're not those those are low
23:40 Scott low value you're saying it's low value but what i would also so to me there's a cultural piece there
23:46 Scott Which is like the thing about the unit tests is in this heuristic of things is it's easy to make a mistake on the.
23:55 Scott on the heuristic and think something is low value when it really should have been high value so like at least with the code review and the unit test which they sort of see as the foundational things like let's just say we're going to let's just we're just going to have a culture of just doing these things and it's going to be it's going to be it's going to set the right culture of having these things buttoned up but then when it comes to more complex tests your integration and end tests and it comes to your monitoring and it comes to your upfront
24:22 Scott technical design planning,
24:23 Scott those are the areas where I've varied it based on the heuristic that you're using.
24:31 Scott But in the center of it, in my mind, is the code review and the unit test is just we're just going to do those because what happens is, oh, this, you know, this button,
24:41 Scott it's, you know, if the unit test doesn't provide a lot of value, I'm not going to put a unit test on it.
24:47 Scott But then somehow one way or the other someone breaks buttons and no buttons don't
24:51 Brad You
24:52 Scott work anymore.
24:52 Brad also include in everything has to have a unit test. Do you have some kind of a code coverage branch coverage requirement as well?
24:58 Brad Well, like must have 100% coverage,
24:59 Brad must have 80% coverage,
25:01 Brad what?
25:02 Scott We put like a yes,
25:06 Scott sort of, and they had 100% strived for 100%. In one company, we just had an agreement, hey, it's got to be.
25:16 Scott You need to have a certain level of coverage.
25:17 Scott Another company I was at, we actually had a danger rule that wouldn't let you commit if your coverage wasn't.
25:23 Brad And somebody figured out how to add like 100 lines of comments and get their coverage up.
25:29 Scott and there was another place that was that we had a different danger rule which is you couldn't commit if the over i think i can't remember how this worked maybe we didn't do this maybe we just thought about it but it was you can't commit if you're making the overall coverage go down
25:43 Brad Yes.
25:43 Scott right we just always want to be getting better yeah people game the system all the time and that's i mean i don't love it but you know i think most people don't try to game the system but i think what i was really trying to do with code review
25:56 Scott code reviews and unit test coverage was just set a foundational culture of like these are two things where if we do them right can really save our bacon frequently and let's just take the decision making out of it because the decision it's it's clear when you don't need the tests and it's clear when you do need the tests it's the gray zone in between where you make the mistakes
26:18 Brad Yeah,
26:18 Scott so
26:18 Brad yeah
26:19 Scott let's just eliminate the gray zone by just saying we're going to do everything
26:22 Brad Okay, so you sort of hinted to the question of AI in the case of, you know, like, let's start doing pair code reviews of AI so we can teach young developers how to read code and understand code.
26:35 Brad I love that idea.
26:37 Brad But let me ask this other AI question.
26:39 Brad With the trend towards vibe coding.
26:42 Brad I'm skeptical.
26:43 Brad I am skeptical that the vibe coding out there, I feel like it's making every possible wrong choice in your quality chain of quality,
26:56 Brad right?
26:57 Brad Like lack of clear specificity of what the behavior is,
27:01 Brad lack of a design plan up front,
27:07 Brad lack of a review of that.
27:08 Brad But, you know, and didn't you recently have a story with you were asking me, right?
27:13 Brad You sent me this text message about, hey, my my AI code pilot wrote a bunch of unit tests,
27:19 Brad but it's not running them.
27:20 Brad What do I tell it? How do I tell it to run the unit test it wrote?
27:24 Scott Well, it's funny. I did text you that recently, and it turns out that some configuration got messed up.
27:30 Scott And so that wasn't the only problem I had.
27:33 Scott It wouldn't do anything to my.
27:35 Scott into my Oh, code it
27:35 Brad man.
27:35 Scott wouldn't even change it there was there was some setting somewhere i couldn't find it i actually had to uninstall this thing and reinstall it there was some setting somewhere i must have twiddled with that
27:45 Scott you know like safe mode which is like okay well we'll tell you what to do but we're not going to do it to your code right and i couldn't turn it off but all of this talk the instant you mentioned ai made me think something
27:58 Scott It's that final sort of end-to-end test,
28:00 Scott which is becoming the most important,
28:02 Scott because it's like you've mentioned many times the ephemeral app,
28:06 Scott you know, people aren't going to buy apps anymore.
28:09 Scott They're just going to say, AI,
28:10 Scott make me this app that does this thing, right?
28:12 Scott The code is going to be a black box.
28:14 Brad Yeah.
28:16 Scott Nobody's going to know what's going on. It could be spaghetti cut in there. It could be beautiful.
28:19 Scott Nobody's going to know. Nobody's going to care as long as they ask, does the things we want
28:23 Brad But
28:23 Scott to do.
28:23 Brad isn't that didn't you just didn't you just basically bring us right back to manual testing,
28:27 Brad which is all ephemeral apps will just be built. They'll be completely bespoke. And the answer to quality is, did it did it meet the needs of the person who used it?
28:37 Scott Well.
28:38 Scott I mean, you can automate those tests, right?
28:41 Scott When the users click the button,
28:42 Scott does the thing go ding?
28:44 Brad Yeah.
28:44 Scott But you don't need to know if the code underneath there,
28:47 Scott you don't need to code review that.
28:48 Scott You don't need to have unit tests on that.
28:49 Scott You don't need to know if it's any good.
28:50 Scott The data could all be stored in one giant.
28:53 Scott All your data could be in one text file that's stuffed into one field in one database.
28:57 Scott It could be crazy sauce,
28:59 Scott but hey,
29:01 Scott I pushed the button, it went ding.
29:02 Brad Yeah, the database is
29:03 Scott I
29:04 Brad literally
29:04 Scott guess just it works.
29:04 Brad in a giant string in the JS file of the main.js.
29:09 Scott Yeah.
29:09 Brad Yeah.
29:10 Scott It's not the craziest thing in the world.
29:13 Scott It probably is the craziest
29:14 Brad I'm
29:14 Scott thing
29:14 Brad sure
29:14 Scott in the world.
29:15 Brad that worse code has been written.
29:16 Brad Somebody might actually write an ETL job to copy usernames and passwords from one database to another.
29:22 Brad You never know.
29:23 Scott It's you never know.
29:24 Scott But this is making me think the part of testing that has always been the most challenging test automation that's always been the most challenging is that user interaction
29:33 Brad Yeah.
29:33 Scott testing,
29:33 Scott right?
29:33 Scott right selenium
29:35 Brad Yeah.
29:35 Scott those kinds of i don't know there's different tools now is it cypress i think is one that's out there i haven't been up those are always the ones that are the most brittle the hardest to get to work well and engineers hate them the most but in an ai world they actually might be the most important degree
29:52 Brad I agree.
29:52 Scott Which is pretty interesting.
29:54 Brad All right. So basically.
29:56 Scott Okay, wait, there's one other. So you're going to go. So basically there's one other thought I have to put out there about the AI world.
30:01 Scott I've thought about this a little bit,
30:03 Scott but I haven't really formed any concrete thoughts. I'd love yours.
30:06 Scott What's the role of test driven development in the AI world?
30:09 Scott Or it's getting closer actually to the Gherkin stuff,
30:12 Scott the requirements,
30:13 Scott the BDD. What is BDD?
30:17 Scott whatever the requirement-driven development is getting closer to that
30:20 Brad Yeah.
30:20 Scott stuff.
30:21 Brad I mean, look, I.
30:23 Brad I mean, all the AI agent code that I'm writing these days, I do try to tell it like from day one, like,
30:30 Brad hey, I'm going to want you to have unit tests for everything. I'm going to want you to make a Cypress or Selenium end-to-end tests.
30:38 Brad And as we go through each of these steps,
30:41 Brad Mr.
30:41 Brad AI agent,
30:42 Brad like get, you know, get to the end point and.
30:45 Brad I want unit tests that test the expected outcome of the step.
30:51 Brad And so I, to your point about your chain of quality,
30:55 Brad like, yeah,
30:57 Brad absolutely.
30:57 Brad It's way easier for me to write unit tests for everything.
31:01 Brad The AIs are quite good at that and I can even throw it like I can run a coverage report and say,
31:07 Brad hey, your unit tests only had.
31:10 Brad 75% coverage and I noticed that it's missing all of these branches from this code that you wrote can you please go write your unit tests that will test those corner cases as well and it's usually pretty darn good at getting those the only times that the only times that it will tend to like have to work extra hard or fail to do it is if you've got like some kind of defensive coding that's like you know gracefully handling null pointers or something like that you just
31:35 Brad Because there's no version of it's like the code paths are almost unexecutable or something like that.
31:42 Scott Yeah, you're making these you're making you've provoked thoughts in my head, you're making me be thoughtful about something you've pointed out a few times, the upstream stuff, and that the vibe coding is making the upstream stuff like get fuzzier because you're just talking to your computer,
31:58 Scott but it actually makes me think that's in terms of quality like that's.
32:05 Scott More important, you know, recently I mentioned to you, I can't remember if I mentioned it on the podcast or not, but I had seen these posts about people doing prompt engineering with Jason.
32:14 Brad Yeah, yeah, you and I talked a little bit about that, I think, recently.
32:18 Scott And I was thinking about that and I'm wondering.
32:21 Scott If you agree with this, I think I thought, well, I think the reason this worked better is actually that JSON is just helping people structure their thoughts more
32:30 Brad I
32:30 Scott clearly.
32:31 Brad agree with that.
32:31 Scott If people actually wrote as clearly as the JSON is driving them to write.
32:39 Scott Plain text might be fine,
32:40 Scott but people just get there and do sloppy stuff in their words and the computer doesn't know what to do.
32:47 Brad I was going to say like another quick little mini story here that I recently had a pair programming session with one of my junior engineers where he was working on a bug.
33:03 Brad he was struggling to work on it.
33:05 Brad And I said,
33:05 Brad hey, why don't we get together and we'll just sit down and do some pair programming together on this.
33:10 Brad And I focused a lot on the how are you prompting the AI to help solve this problem.
33:16 Brad and um there was a couple things that i took away from it one thing that was really cool was he and i kind of like prepped ahead of time like i knew the bug i grabbed his copy of the code base i went and threw some prompts at it and watched the ai come up with answers and so i kind of had my own kind of what i thought it was going to be good at um and he did some really cool stuff where he actually
33:38 Brad ran i was super happy to see this he ran the code he ran two different versions of the code he had like the develop branch so he knew like the main code base he had his local build he was inspecting it in chrome and and this was something like a rendering type issue so he was like inspecting the two different objects uh in the chrome dom to see like why is this the tailwind css class not being applied correctly so it was like he did some really good prep debugging that i thought was great and i you know kind of said hey
34:04 Brad i said hey good job buddy um but when he prompted the ai he didn't he didn't basically give it a bunch of context right so he didn't say this code works the old code works i changed the following when i changed the following the bug got introduced he didn't give it the context of the other stuff he changed and i kind of was like hey
34:29 Brad Why don't you try that? Why don't you try telling it what you changed?
34:33 Brad That's going to give him give the AI a better clue about where the possible interactions were happening.
34:40 Brad And it was night and day the difference in the AI's results. Like without that context, the AI just threw up some completely random ideas that weren't were completely wrong.
34:50 Scott How did he tell the AI what he changed? Like, did he give it a diff, like here's my diff?
34:53 Scott Or did he just say in words like this is what I did?
34:56 Brad In this particular case, he said.
35:00 Brad it was something like this was drag and drop related so it was like an object that didn't used to be draggable that he was making draggable so simply saying like this object used to not be draggable i started using the draggable action so like that was enough context for it to
35:16 Scott That
35:16 Brad that
35:16 Scott was enough.
35:16 Brad was enough um
35:17 Scott Fascinating.
35:17 Brad that being said i have definitely given it prompts like my branch is brad's feature x branch
35:26 Brad It has this following bug.
35:27 Brad The develop branch has this following bug.
35:32 Brad uh it doesn't have the bug it works correctly here's the description of bug and so i'll give i'll sometimes give the ai and it'll often go and say cool i'm going to go check out those two branches and do a diff between them and that and you'll see it like working away going up checking out old branch doing the diff you know and then come back and go oh i see you made the following changes
35:51 Brad here's where the problem is so i feel like so the reason i mentioned this only because of your json point like i think that's a
35:57 Brad That's a really good point that like if you had a I'm doing a bug fix and the prompt is I'm going to make sure when you do the prompt for bug fix,
36:05 Brad always give it the context of the branch. Where does the bug not happen?
36:12 Brad Where does the bug happen?
36:14 Brad You know, what was the change you were making when the bug started to happen,
36:18 Brad et cetera.
36:20 Scott Can I leave you with an anecdote today?
36:22 Brad Yeah,
36:22 Brad I
36:22 Scott Have you seen the video going around the internet of the school teacher making the peanut butter and jelly sandwiches?
36:29 Brad have not, but my immediate brain starts to picture what the joke is going to be about this.
36:36 Scott It's not a joke.
36:37 Scott I mean, this is.
36:39 Scott uh this is an elementary school teacher you don't see the kids in the video you only see the teacher but you hear the kids and i would i would guess they're you know third fourth fifth grade somewhere in that range based on the tones of their voices and the kinds of things they say and they had an assignment there she's an elementary she's doing they have a descriptive writing
37:00 Brad And she's following their descriptions to how to make a sandwich.
37:03 Scott yeah and she has been she's asked them
37:06 Scott write instructions for how to make a sandwich and so she has got a jar of peanut butter a jar of jelly and a loaf of bread and the instructions are you know you're supposed to write how to make a peanut butter and jelly sandwich and she says okay i'm gonna she's got the stack of papers right there she's like i'm gonna do exactly what you told me
37:29 Scott and she doesn't in this video and it's adorable because the kids are laughing and they're like no no no so the first one is like get some bread and put peanut butter and jelly on it and so she puts the bread on the desk there and she puts the peanut butter and jelly right on top of the bag of the loaf of bread she's like is that what you meant you know they're all like no no no no then she gets another x one and it says um get some bread
37:54 Scott and spread peanut butter and jelly on it or something like that and so she opens up the jar of jelly and takes her hand in and spreads it right on top of the bag of the wonder bread or whatever it is and spread it and they're all no no but this is a you know and it was a really powerful powerful illustration of what it means to be descriptive Yeah. and when we think about this AI in context
38:19 Scott right context is so important because when someone says we'll take the bread and spread some jelly on it when you hear the word spread as adults or even maybe as a child you kind of like get a knife and use it to spread it on the bread but these computers are so smart yet so dumb and you have to just give them exquisitely detailed context to get them to do the thing that you want them to do
38:45 Scott And how that relates to QA, which is the topic we started with, I have no idea,
38:50 Scott but the video was adorable.
38:52 Brad Love it. I'm going to have to link that video in the show notes of this episode.
38:57 Scott I'll have to find it and I'll do that.
38:59 Brad Great talking to you, Scott.
39:01 Brad This was super helpful.
39:02 Brad I appreciate your insight on the chain of quality.
39:05 Scott Thank you. I'm glad it could be helpful.
39:07 Scott It's fun talking to you too.
39:08 Brad Everybody, please rate, review and smash that subscribe button.
39:13 Scott You could be our one millionth subscriber.
39:15 Brad He could be.
39:17 Scott See everybody.
39:18 Brad This episode of How Many CTOs Does It Take podcast was edited by Saffron Epigal.
39:23 Brad Our theme song was AI Generated by Brad.
39:27 Scott Please follow us on all the social media platforms at at HowManyCTOsPod.
39:32 Scott You can find us on YouTube,
39:34 Scott LinkedIn,
39:34 Scott Instagram,
39:35 Scott Facebook,
39:36 Scott Threads, X,
39:36 Scott TikTok,
39:37 Scott and Patreon.
39:38 Scott Or at our website at HowManyCTOsPod.com.
39:43 Scott And remember to rate, review,
39:45 Scott and subscribe on the podcast platform of your choice.
39:48 Scott Thanks for listening.