From Chaos to Control: Bringing Specification Driven Design to Vibe Coding with David Bauer

0:00 David: Our philosophy,

0:01 so yes, we're still doing unit tests,

0:03 but we care much, much more about the integration tests.

0:06 The integration tests for us are the gold standard.

0:25 Brad: Welcome, everybody, to another episode of the How Many CTOs Does It Take podcast?

0:29 I'm Brad Hefta-Gaub-Galb, and our regular co-host Scott Porat is not with us today because he's traveling on business.

0:34 But we have a very interesting guest with us, Dave Bauer.

0:38 And Dave,

0:39 why don't you introduce yourself?

0:40 David: Yeah, thanks,

0:41 Brad.

0:41 Thank you for having me.

0:43 So I am Dr. David Bauer.

0:46 I've got a PhD in computer science.

0:49 My background is primarily in distributed computing.

0:51 computing.

0:52 And for the last 26,

0:55 28 years, I've been working primarily in the U.S. intelligence community.

1:00 Did some startups when I was in graduate school, did three or four startups,

1:04 had phenomenal timing,

1:06 2000,

1:06 2001.

1:08 So I definitely hit the dot-com bubble and blew up a bunch of companies that way.

1:11 Brad: That's always a fun time.

1:14 David: Yeah, and then I decided to get back into it in 2018 and everything was cooking with gas and doing really well and we have learned a lot of things in the intelligence community about AI and I wanted to take a commercial and started a company in 2018 and then COVID hit.

1:34 So we actually did fairly well through COVID but then once COVID was over.

1:39 then we went in a different direction.

1:42 And what I got really interested in while I was working on that AI startup was federated learning and how to, you know, kind of move the model training process and the model serving process down to the data as opposed to the other way around.

1:56 And so that is what Exonis AI is all about.

1:59 It's actually not just federated learning for machine learning.

2:04 But it's actually leveraging and so what we've done is we've created a federated data layer and then we use that for many different things. We use it for data fusion,

2:13 semantic ontologies and a bunch of decision intelligence.

2:17 And so that's what we're doing today.

2:20 Brad: That sounds like some pretty interesting stuff for sure.

2:24 David: Absolutely. Absolutely.

2:26 Brad: Tell me a little bit about how not only what you've built, but I would love to hear about how you're building it.

2:32 You know, obviously nowadays it's like the big conversation is,

2:36 can we safely write software?

2:39 using AI agents.

2:42 And I think it's interesting how some of the big frontier models, they're like,

2:47 oh yeah,

2:48 we use AI to do all of our,

2:49 build all of our stuff.

2:51 So

2:51 David: Yes.

2:51 Brad: I'm just kind of curious what your take on that is.

2:53 David: We're a startup.

2:55 I think we've been around for about two years now.

2:59 kind of coming out of stealth in the last six months or so.

3:02 And we have to use it, right?

3:05 We feel like we don't have a choice, right?

3:08 If we don't use it to accelerate what we're doing and to maximize our people and our experience and our skills, then we're going to be at a disadvantage.

3:15 We're going to fall behind and we won't be successful.

3:18 So not only do we have to use it, it's not a begrudging use it,

3:22 we want to use it.

3:24 because we want to get more done.

3:26 We want to get higher quality work done.

3:29 And we want to, you know, also,

3:33 you know, improve all the things that engineers don't like to do, right? Documentation and QA and, you know, diagrams and other things, right?

3:46 So for us, it has really been kind of a crazy.

3:51 crazy whirlwind six months this last six months and knowing that we have to go in this direction we have spent a lot of time actually recreating our entire internal process and and re-engineered the entire thing and it really kind of started about four months ago everybody's moving to MCP services right everybody's building MCP tools on top of their existing capabilities

4:16 buddies and you know we're no exception and then we very quickly hit on the whole notion of hey we need an mcp gateway so now we're going in the direction of now we have to build an mcp gateway you know along the way we're kind of internally encouraging everybody just use it right it doesn't matter cloud gemini you know chat gpt we don't care use your favorite we'll pay for it let's figure it out

4:42 Brad: Yeah.

4:42 David: And it was good at first, and then it got crazy very quickly.

4:48 Right.

4:50 And, you know, suddenly I'm getting all sorts of crazy commits.

4:54 You know, I'm getting all sorts of crazy PRs that I'm trying to, you know, make sense out of.

5:00 And, you know, just we're losing traction,

5:04 right?

5:04 Because PRs aren't getting more.

5:06 Brad: Wait,

5:06 so hold on a second. Hold on a second,

5:08 Dave. Let me rewind there and just make

5:10 David: Yeah.

5:10 Brad: sure I'm catching what you're laying down.

5:12 So you did the mandate of like,

5:15 not a mandate,

5:16 but you said,

5:16 hey,

5:16 you're free to use this stuff. We think this is important for us, for our velocity.

5:22 We are in a figure out what works for you mode.

5:25 So we're tool agnostic and we're kind of like,

5:28 please go for it.

5:30 And you had a side effect that was unexpected of that.

5:33 bit of that of too much velocity or too low quality or what when you said it's sort of like you had that moment what what was that moment oh

5:43 David: That moment was every PR that we were reviewing.

5:48 It just wasn't, it was not correct.

5:52 Brad: okay correctness

5:54 David: You would get.

5:57 150 changes on a PR,

6:00 right?

6:00 Brad: Mm hmm.

6:01 David: So how can a human read through 150 changes in a single PR and then you have, I don't know, 10 of them a day,

6:09 right?

6:11 They're coming at you faster, right?

6:12 Everybody's vibe coding and they're coming in really quick.

6:14 Brad: Yeah,

6:15 totally.

6:15 David: You have to do them in the right order and make sure everything merges and get rid of the conflicts and oh, by the way,

6:21 here's 200 changes.

6:22 your SWIFTY changes, your 75 changes.

6:24 Brad: Hmm.

6:25 David: And so there are definitely like kind of core components in the code that we really care about a lot more than others.

6:32 And, you know, for example, security related aspects, authentication,

6:37 that kind of stuff,

6:38 data level security and ABAC controls.

6:42 And so, you know, we kind of would go and scrutinize those first and we realized.

6:47 this is way off track and you know with any large software project I think we've probably got you know well over a million lines of code you

6:56 Brad: Sure.

6:56 David: know there's going to be these side effects right you make a change here and it affects something else over there and in the beginning we're like okay

7:06 I'll roll the dice and

7:07 Brad: Yeah.

7:08 David: I'll merge that PR,

7:10 even though I didn't really look at all 250 changes and things went south from there,

7:16 right? So then we couldn't get through testing,

7:18 we couldn't get releases out,

7:21 right?

7:21 Just we started to, you know, have too much broken code, right? Too many breaking changes.

7:27 And then

7:29 Brad: Can you give

7:29 David: Yeah.

7:30 Brad: the listeners a little bit more framing about scale of the team and kind of scope of the types of projects you're dealing with?

7:42 David: So we have about 30 people on the team.

7:46 Probably 20 of those are engineering minded.

7:49 My founder and partner,

7:51 Chris Yonklis,

7:53 he is a mad scientist, right?

7:55 Right. He's big brain.

7:57 He's been in the field for about 30 years,

7:59 35 years in the intelligence community.

8:02 And, you know.

8:04 Between him and I,

8:05 we've had every experience imaginable when it comes to working with data,

8:08 when working with systems,

8:10 large-scale systems.

8:12 By coding and unlocked for us the ability to get a lot more done with a smaller team much more quickly.

8:20 Brad: Yeah, sure.

8:20 David: But putting it all together was really where things started to fall apart.

8:25 Brad: How did you tackle that?

8:26 David: Yeah,

8:27 so I will get to the solution.

8:29 You know, we started building MCP servers and, you know, suddenly we kind of broke like 100 tools and 150 tools,

8:35 things started to fall apart, right?

8:38 Models could no longer make sense out of which tool to use.

8:41 And it didn't matter how well we described them, you know, it would just pick the wrong tool, you know, so then we realized we need an MCP gateway,

8:50 we need a composable architecture,

8:51 we need to decompose things more.

8:53 But then...

8:54 And now you've got this, you know, many different components.

8:57 I think we're up to about a dozen now,

8:59 you know, and they all have to be cross-functional, right?

9:03 Brad: Yeah.

9:03 David: So we've encapsulated functionality fairly well,

9:07 but they do have to be cross-functional. And so really now the challenge is how do we actually build a complete agentic architecture just for our software development process,

9:18 for our product development process.

9:20 that enables us to have all the speed, but have speed with control is

9:28 Brad: Mm-hmm.

9:28 David: really the key and not have crazy things like, you know, merges with, you know, PRs with hundreds of changes.

9:39 And, you know, one of the other things that we have is.

9:44 You know, I would have developers tell me, well, you have to do these four PRs in this order in order to get my change.

9:50 Brad: Right.

9:51 David: So it was nuts.

9:52 So what we did was we actually spent months,

9:56 there was months as a CTO where my

9:59 You know, CEO,

10:01 my CMO,

10:01 my sales folks,

10:03 my customer success folks would ask me, what are we doing? How are we going to do this? When are things going to come out? Why are things delayed?

10:09 Why didn't we get a release this week?

10:12 I would just tell them, I don't know.

10:15 I don't know.

10:17 Stuff's broken.

10:19 We're figuring it out.

10:20 Brad: Yeah.

10:20 David: So it took about three months.

10:22 And then we hit on a.

10:25 essentially a systems engineering based approach to product development called specification driven design and

10:32 Brad: Sure.

10:32 David: I'm not going to say this is something new it's not it's not you know even necessarily our idea really what I started to do was just read everything I could right and you know the vast majority of articles are like hey here's all the reasons why vibe coding is bad I'm in a position where I can't just say it's bad and walk away right

10:50 Right.

10:51 I have to figure out how to make it work, which we've done.

10:55 Specification driven design is all about specifying the functionality and the requirements of the platform that you're building or the product that you're building.

11:05 Brad: Yeah.

11:06 David: And so once you've got that correct,

11:09 then what we've done is we've built an enormous number of tools around it.

11:14 to automate the process.

11:16 So we have maybe 120 different components in our, you know, in our SVD.

11:24 We have an automated process for going through the development, an automated development process for those specifications. The LLMs are tuned,

11:37 they have skills,

11:38 they have goals, they have

11:40 You know, we have get commit hooks,

11:42 pre-commit hooks,

11:43 post

11:43 Brad: Yeah.

11:43 David: -commit hooks, and scripts that get called from those hooks.

11:48 We built an entire STD-based package that gets applied to the code at different times so that we can do gap analysis. So every ticket that we do now,

11:59 every issue that we resolve now,

12:02 it gets a gap analysis from the software or specification.

12:07 Driven

12:07 Brad: Yep.

12:07 David: design.

12:08 It gets a review for compliance with the specification.

12:13 We actually do the implementation.

12:15 So there are guidelines for how to do the implementation.

12:19 We have a whole agentic framework that kind of steps each issue through this process.

12:24 Brad: Yeah.

12:24 David: Once it's done,

12:25 there's a validation process,

12:29 a review process. So, for example,

12:31 SDD Validate.

12:32 That tool ensures that everything, all of the tests are working,

12:36 unit

12:36 Brad: Yeah.

12:37 David: tests, but then also integration tests before it can, you know, create a merge.

12:42 And then what I'm really excited about is we've integrated that with our agile methodology.

12:50 So I'll kind of get to the punchline here.

12:52 Integrating it with our GitLab,

12:54 we use GitLab.

12:56 How everybody else uses GitLab,

12:58 but we're not using them for technical issues.

13:00 We're using them for spec related

13:01 Brad: Yeah.

13:01 David: issues.

13:02 And the punchline is what I've seen on our team, on any team, you have people of varying levels of experience in.

13:13 coding and other aspects of software development.

13:15 What I've seen is we are no longer heads down focused on code,

13:20 focused on bug fixes,

13:22 focused on, you know, minor issues.

13:25 Those things are distractions.

13:27 Coding.

13:28 is the scarce skill that used to be the bottleneck. For

13:32 Brad: Yeah.

13:33 David: us, it's no longer the bottleneck. Now, the scarce skill that is the bottleneck is your experience,

13:40 your knowledge,

13:41 your ability to describe the system,

13:43 how to build the system.

13:45 So the specs do go down into implementation,

13:48 right?

13:49 They go into implementation

13:49 Brad: Totally.

13:49 David: details and other things.

13:51 And what it's done is it's unlocked our team. We have team.

13:55 team members who only maybe understood their one component.

13:57 Now they're understanding all of the system.

14:02 From a day-to-day perspective, we're thinking about and we're talking about the product, right?

14:08 We do our daily stand-ups. We don't talk about, oh, I fix bug 62.

14:12 You know, we talk about, you know, I'm going to implement this part of the system.

14:17 And I don't think the specification is exactly correct.

14:20 So we all talk together about how it should work.

14:23 Brad: I love it.

14:24 David: And it is just a complete mental change.

14:26 Brad: So this is great,

14:28 obviously.

14:28 um i know i you know scott's going to be disappointed he wasn't here because he's a big fan of sdd have you are you so just to clarify are you did you build your own sdd tooling or

14:40 David: Yes.

14:40 Brad: are you using something like spec kit or one of the other sort of

14:43 David: We

14:43 Brad: like interesting

14:44 David: built our own.

14:45 We built

14:45 Brad: interesting

14:45 David: our own.

14:46 Brad: yeah

14:46 David: Because what we're finding, Brad, is that the tools are not suited for vibe coding.

14:52 Brad: Yeah.

14:52 David: All right.

14:53 I'll give you an example.

14:55 GitLab. GitLab is per example.

14:57 GitLab is designed to implement an agile process,

15:01 an agile methodology that is based on humans taking time to

15:08 work iterate on issues tickets

15:11 Brad: Yeah,

15:11 right.

15:11 David: and you know the thought process is if they you know what have we always said if the issue is too small and it only takes you 10 minutes it probably wasn't defined well if the issue takes three months it's definitely not defined well right so we try to get these things down into an hour a day a week right because that's the human pace of coding

15:35 And that's fantastic.

15:37 It's also really slow when

15:39 Brad: Right.

15:39 David: you're in start vibe coding and suddenly you're closing 20 tickets a day and some of those tickets,

15:45 you know, they maybe have lots of points associated with them.

15:49 You know, maybe it's a week long task and then you got it done in four hours and then you knocked off a couple day long tasks as well.

15:57 Now your velocity is,

15:58 Brad: Right.

15:59 David: is, is no longer suited.

16:02 to that paradigm.

16:03 Brad: Yeah, totally.

16:04 So let me let me just interject there for a second, because in the start of your story, you started talking about how you were talking about sort of like the.

16:12 no more than 150 lines of code uh changed or something like that and that's a common like you know the idea of like when we were humans all writing the code the idea that someone would give you a pr that was hundreds of lines of code you'd be like i can't reason about this and so people would even build like you know danger danger rules or something like that that would like automatically reject prs that were more than that amount of code and one of the things you

16:37 things that I've certainly noticed, and I think that you're scratching at this, so I would really love to hear about how you guys have addressed this,

16:44 is that, again, because the agents can write so much code so quickly, that the unit of work product, and you sort of referenced this earlier,

16:56 like as a developer, if I'm like trying to implement an epic,

16:59 and I have to break it up into five or six different tasks.

17:04 Then the first four tasks aren't aren't they're testable in unit unit testable, but they aren't really like integration testable until you get to the fifth task. Right.

17:15 And so like the unit of work product kind of only sees its only sees its real value or its testability or its or its you know, its usability after five or six different PRs are merged. And and I felt like you were kind of like scratching.

17:30 scratching at that problem and then

17:32 David: Yes.

17:32 Brad: now that we give these to the agents the agent can completely implement a full epic in a in a at a single you know at a single pass and there's a benefit to that that you get you get an object at the end which is a fully criticizable fully realized fully testable fully you know critiquable thing as opposed to little pieces that are are only

17:57 uh interrogatable in abstract until the final kind of everything gets glued together so like that's a benefit of the of the spits out an epic the downside is it's you know 30 000 lines of code or 3 000 lines of code either way it's

18:13 David: Whatever

18:13 Brad: just too

18:14 David: it is,

18:14 Brad: much

18:14 David: yeah.

18:14 Brad: too much to reason about so so one

18:19 I'm assuming you are you felt that same trajectory and is in how and is your case the case you're making is,

18:28 hey,

18:28 SDD is what solves that or help me understand,

18:32 like, like, how did you Well, bridge that gap?

18:34 David: STD is a better human process for by coding.

18:40 But you still need multiple tools around it.

18:43 And it does impact how you work, which I mentioned.

18:49 The one part I didn't mention is that our integration testing,

18:52 our end-to-end platform testing,

18:54 that fifth task now kind of becomes the zeroth task,

18:58 right?

18:58 Brad: Yeah.

18:59 David: Because without that,

19:01 it's almost impossible to validate.

19:04 And so what we have is essentially a ongoing staging environment where things are constantly being tested end to end across the platform.

19:15 We have a composable architecture,

19:17 so we're able to kind of recreate the staging environment in pieces clean.

19:22 And then, you know, you mentioned automated CICD. We've got a complete end to end automation for CICD. So that's not really the issue.

19:30 issue it's not an issue to do what we're doing but the the the integration testing is the key that and

19:37 Brad: Yeah.

19:38 David: i'm not going to tell you that we don't vibe code portions of it because we do but what we really don't do is let it vibe code the tests themselves right

19:48 Brad: Oh, interesting.

19:48 David: what we what we will let it do is we'll let it vibe code the test cases

19:54 Brad: Mm

19:54 David: Right.

19:54 Brad: -hmm.

19:54 David: So the data for the tests,

19:57 we will let it test.

19:59 So, you know, negative tests, fuzzy test solids, all that, you know, the values and everything that it's testing,

20:06 those can come from an LLM and that's fine.

20:09 But the integration tests themselves.

20:13 We keep that kind of close to best and really,

20:16 you know, things can't pass until they have that.

20:20 And so for us, that becomes the primary focus in terms of like knowing what we have is working correctly.

20:28 And it's and it just becomes that much more important. So maybe it was a gate for a release that it passed a certain percentage of integration tests.

20:38 you know, based on priority and whatnot.

20:40 But now it's integrated into the front end of the development process.

20:46 And so by the time you get to that release gate,

20:49 right,

20:50 it's a non-issue.

20:52 Brad: Yeah, this is one of the things that I'd see in the teams that I work with that are the higher performance teams.

20:59 Well,

20:59 one, I think they started.

21:02 kind of with a strong SDLC and philosophy around,

21:06 you know, what, what their quality gates,

21:10 what quality gates mattered,

21:11 right?

21:11 And that's what I kind of hear you talking about,

21:13 like define your quality gates,

21:14 be really strict about your quality gates.

21:16 And

21:18 And then once you can give agents very clear instructions about the quality gates,

21:22 you shall not pass until this quality gate has been met at each step of the way.

21:28 I think that's for sure one of the best practices that I'm seeing.

21:33 That to me strikes me as like the most replicable skill,

21:39 right?

21:40 Like we can replicate that in any of our software projects.

21:44 David: We're kind of at a point now where we're feeling like unit testing may be a thing of the past.

21:50 Brad: Oh, interesting.

21:51 What do you mean by that?

21:52 David: So if you vibe code your unit tests,

21:56 I mean, the models are phenomenal at working around problems.

22:00 And they and that shows up mostly for us, at least that shows up most in the unit testing.

22:08 So I'll give an example.

22:09 Brad: Yeah, yeah, give me an example.

22:10 David: We had a graph analysis to our projects, to all of our projects, because we had a model writing unit tests like mad said, yeah, 100 unit tests pass on this class.

22:25 class is never referenced right

22:27 Brad: it's Hehehehe

22:27 David: just dead code right just dead code somewhere along the line it got vibed and then it got vibed away but the code's still there right so now you got to have oh I got to have a pruning right but but how do you know

22:41 It isn't used in some other project,

22:43 Brad: All

22:43 David: right?

22:43 Brad: right. So here.

22:43 All right. So I'm going to challenge you on that one.

22:46 I'm going to say I actually have taken the opposite stance and I feel pretty confident that this is a powerful way.

22:53 So first of all, I totally agree with you that the agents will happily like follow your instructions,

22:58 you know, and if you ask for a, you know, module to have like 100% code coverage, then it will. I had the agent just do this to me a couple of weeks ago where.

23:07 I had it basically like the way it found to get 100% code coverage was to ignore all the files that didn't have 100% coverage.

23:16 Like this package had, you know, 10 files in it and five of them had 100% coverage. So it just ignored the other five.

23:23 So certainly you cannot, you have to watch for stuff like that.

23:26 You know, when you said earlier,

23:27 like, you know, AI lets, you know, the things that developers hate to do is the thing that you should give to the AI.

23:33 And every developer I've ever known is heated writing unit tests.

23:37 And so what

23:38 David: All

23:38 Brad: I

23:38 David: right.

23:38 Brad: advise...

23:38 David: Hold on.

23:39 Let's pause right there for a second, because one of the tests I have for new hires is I always ask them, what does your process look like for software development?

23:47 And if they don't tell me,

23:49 I start by developing the unit tests and figure out all the edge cases and corner cases and everything else.

23:57 And then I write the code.

23:59 I don't view them as a mature developer,

24:03 right?

24:03 Brad: All right, fair enough.

24:04 David: So I agree with you.

24:05 Nobody enjoys writing unit tests,

24:07 but once you kind of get into that mode and you see the value of it,

24:11 then it really improves your coding.

24:16 dramatically.

24:17 So if I'm not writing the code anymore,

24:19 do I still need them? Do I still need to take that approach,

24:23 right?

24:24 And we did consider having the models generate the unit tests in a way where it would do all those same considerations and then develop the code to pass the test.

24:34 And that may still be the viable way to do it, just like we're doing with the integration testing. Have it start building all the unit tests first.

24:43 freeze those or lock those and then make the code match but we're afraid that it's just going to flop the problem instead of faking out unit tests now it's going to fake out code i mean you've seen the defensive programming and code right where it'll put a you know try accept or you know block around everything and always return something positive

25:05 Brad: I guess the thing that I would say,

25:06 I mean, I.

25:07 I think this is a healthy debate,

25:09 and I think that it could be a situational thing where depending on the type of software that you're building,

25:15 it might change what my answer is.

25:18 But where I've kind of landed right now that I feel pretty good about for kind of your,

25:24 let's say it's the class of sort of like full stack web apps,

25:28 which is a lot of software out there, right?

25:31 David: Mm-hmm.

25:32 Brad: I feel like those are.

25:34 I feel like we should absolutely have the agents write unit tests for every line of code they write and we should have quality gates on the unit tests. So to your point, like,

25:47 is the unit test valid,

25:48 right?

25:50 I think that the reasonable quality gates that you can run is you can demand 100% code coverage and 100% like 100% branch coverage,

26:00 right?

26:01 Like every line of this code must be covered and must be justifiable.

26:05 And then you can put quality standards on the tests. So, for example, you can...

26:12 Like you can set up linting rules that validate that every test has an actual assertion in it, has a reasonable assertion in it. Right.

26:21 David: Yeah.

26:21 Brad: And then, of course, you can certainly have the models.

26:25 essentially do adversarial code reviews of the unit tests right and so it's straight and then of course you could do something like you could set up mutations right so if you if you run a mutation pass where you modify the code does the unit test fail right and so you could have like essentially like these validation passes that you run against the all of your tests whether they're unit tests or integration tests or end-to-end tests

26:51 And the sheer volume of the ability for you to write 100% code coverage and 100% mutation coverage is something that was just like a typical PR is could end up being 10 times the code for testing.

27:14 compared to the one line

27:15 David: Yes.

27:16 Brad: of code of actual work product.

27:18 And it used to be very difficult to justify paying a human engineer to do that. But now I feel like just make the robots do that.

27:27 David: That is a very valid point.

27:29 And I don't want to make anyone think that we're not doing unit testing.

27:32 We are absolutely doing unit testing.

27:35 And we do have, you know, many of those kind of gate checks in there for it.

27:41 It's something worth thinking about.

27:43 Brad: Yeah

27:43 David: And here's why.

27:45 Now you have so and you make a very valid point.

27:50 It's something engineers don't enjoy.

27:52 It's something the robots can do really well.

27:54 It's something they can do quickly.

27:55 It's something they can do to really add quality to a project very rapidly.

28:01 I agree with all of those things.

28:03 I feel like you have a new tradeoff.

28:06 How much time am I going to spend babysitting these

28:11 Brad: Yeah.

28:12 David: robots in the development of unit tests?

28:16 Right.

28:16 And how many tokens am I going to burn?

28:19 And how much time am I going to wait for these agents to complete on those tasks as

28:25 Brad: Yeah.

28:25 David: opposed to net positive tasks?

28:30 Our philosophy,

28:31 so yes,

28:32 we're still doing unit tests,

28:33 but we care much, much more about the integration test.

28:36 Brad: Well, that's fantastic.

28:38 I agree with you that I'd much

28:40 David: Integration

28:40 Brad: rather have.

28:41 David: tests for us are the gold standard.

28:42 Brad: I totally agree with that.

28:44 David: Having 100% coverage on that,

28:46 on our APIs and on our services and everything else is really key for us to have confidence that we're delivering.

28:55 quality software.

28:57 And still some things can slip through the cracks.

29:00 The question is, if you can get the robots to have your integration testing be that high,

29:04 do you really need the unit test?

29:06 Brad: Yeah.

29:07 David: This is what I would really say.

29:08 So I'm going to bend your brain a little bit more.

29:10 Brad: All right, tell me.

29:11 David: One of the things that we are moving away from as an engineering team is managing anything in GitLab.

29:21 Brad: Okay.

29:22 Yeah,

29:22 David: We're

29:22 Brad: tell me.

29:22 David: not doing issues and PRs and other things any longer.

29:28 We've automated our process to a point where developers don't need to do that.

29:34 So what does that look like? What it looks like for an engineer is when they sit down,

29:40 they can actually just what so they're not vibe coding code,

29:45 right?

29:46 they're in our system right they're not saying hey go grab an issue and let's go work on that issue right they're not moving it on a board and saying okay I'm going to work on that issue today what they're doing because we have the the STD

30:01 they're actually when they talk to their model they're talking to it about the spec so

30:06 Brad: Yeah, totally.

30:07 David: I'm starting out my day and the first thing that I will ask my model is

30:13 What is the highest priority issue that I need to be working on in X,

30:18 Y, or Z component that I care about?

30:20 In fact, RSDD is aware of the capabilities of the developers on our team based on previous commits that they've made.

30:31 So it kind of knows what's well suited to one person or another,

30:34 and it will offer suggestions. It'll say, here are the top three things I think you,

30:40 the developer, should be working on right now.

30:43 Brad: Right.

30:43 David: The

30:43 Brad: I love it.

30:43 David: developer can ignore it and say, I want to work on something else. Okay.

30:46 But they will be forced to pick something from the specification.

30:51 And then they can begin that process that we discussed for the SSDD after that.

30:58 But what they're really doing is they're scrutinizing, right?

31:03 So there's gap analysis. That's kind of the first step. Here's the gap in the actual code.

31:07 You know, this is what's not working or not implemented or whatever the case may be.

31:13 And they are trained now to question the spec.

31:18 Is the spec correct?

31:19 Is this the right way to implement it?

31:22 So when we are vibe coding,

31:24 when we're interacting with an agent,

31:26 we are asking it those types of questions.

31:29 Brad: You're vibe specking.

31:30 David: Yeah,

31:31 we're vibe specking,

31:32 right?

31:32 Brad: Yeah.

31:33 David: Our process includes the steps of,

31:36 hey,

31:36 you know, we're going to update the ticket,

31:38 I'm going to move it to the in-progress state,

31:41 I'm going to update that up in GitLab.

31:43 We love GitLab.

31:44 So what we've done is,

31:46 because we do love GitLab, is we've modified how we use GitLab.

31:51 um to to implement the std process um

31:55 Brad: Yeah.

31:55 David: a little bit more cleanly and so developers engineers are no longer in get lab at all everything all of that part of it is managed for them we have some of our homegrown tooling involves managing the sprint process

32:12 um as well as managing all the agents that were better doing the vibe coding and so um it knows how to talk to gitlab for the user you know we use the glab cli so it's doing a lot of that maintenance stuff for them um and so all we're doing and get like people like me and gitlab will do the prs and things like that to make sure that things are on track and

32:35 Brad: Yeah.

32:36 David: like i said our days are now spent

32:39 talking about the product talking about you know we're not pulling up an issue board and saying this is what i did yesterday this is what i'm going to do today you

32:47 Brad: Yeah.

32:47 David: know we're we're

32:49 We're talking about the problem

32:51 Brad: Yeah, I think that's great.

32:52 The idea that you spend most of your time working with the agent to understand the problem that you're trying to solve, not the way you're going to solve the problem.

33:04 I think that is 100% the right shift.

33:08 That's essentially what I see in the high performance teams I'm working with. Now, each of them, they don't all necessarily use SDD.

33:15 proper or spec kit proper a lot of them you know but they're effectively doing the same practices right so the engineers are essentially getting a description from the product team of the thing that the product owner wants and then they spend their time with the llm basically refining that

33:34 refining that spec adding their um you know their opinionated um worldview of like this is how this component should should work in terror going back and forth like how you know what code exists to do that i like to say the engineers act like engineering managers to

33:53 David: Yeah.

33:53 Brad: the agents who are their engineers and

33:56 David: Product

33:56 Brad: in the yeah

33:57 David: managers,

33:57 Brad: technical

33:57 David: right?

33:58 Technical product managers.

33:59 Brad: product managers who are basically like

34:01 Like, and so they still need to have that technical skill to be able to.

34:07 A judge,

34:09 hey,

34:09 is this thing that's coming back to me from the LLM, whether it be a specification or a plan or ultimately the code,

34:18 they need to basically be able to interrogate each of those steps to be like, is this actually going to meet the business need?

34:25 Is this practicable to be implemented the way that the thing is saying it wants to implement it? Is its assessment?

34:33 Is the agent's assessment of the gap analysis,

34:36 is that believable?

34:38 Do I agree with its gap analysis?

34:41 And when all of that is, you know, when the engineer is ready to put their name on that package,

34:49 then they can say start coding.

34:52 David: Yes.

34:52 So I'm going to take it further.

34:55 We have a product we call Exonus Decision Intelligence.

34:59 And the purpose of the tool is to provide a data-based approach to decision-making within an organization.

35:09 And this is most commonly used in organizations that have high compliance requirements.

35:15 So you could think finance,

35:17 you could think healthcare.

35:18 You could think intelligence community with classified data,

35:21 it doesn't matter,

35:22 right?

35:22 Supply chain logistics.

35:24 In fact, we have one customer and they deploy rain sensors.

35:30 And they told us the other day, they said, yeah, one of the things we love about your tool is there's actually a legal compliance requirement for how we do firmware updates to the sensors in the field.

35:42 And I'm like, what do you, I never heard of such a thing, right?

35:46 It's basically out there measuring rainfall,

35:48 right?

35:48 But doing a firmware upgrade has a legal repercussion that they have to watch out for.

35:55 We built the tool for.

35:58 industry you know industries that need to maintain compliance across their decision making process and so it's an evidence-based process we create hashes for the data so you could tell if the data has been tampered with we implement a complete you know set of steps that implement an intelligence process everything from collecting data throughout the federated environment all the way down to you know

36:25 creating what we call effects or basically actions or tasks that come out of the decision making process and so what we've done is we've started actually implementing that tool for our STD process right so now when we talk about doing a PR right oh we always like to have a two-person sign off on a PR right have two people looked at it have two people signed off on it so we actually do attestation on our PRs in

36:50 Brad: Sure.

36:51 David: our own tool

36:52 Right.

36:52 So what does that mean?

36:55 What it what it really means and what I'm getting to is because you hit on it.

37:01 Other parts of our company are really, truly integrating into the engineering process.

37:07 So our go-to-market team,

37:09 our customer success team,

37:11 they will now become involved in what we're doing because they can attest, is there a real business value here?

37:20 Is this something useful for a customer?

37:23 And we don't put the attestation just at the end, right? That actually comes into

37:27 to, you know, prioritization of tasks,

37:30 right,

37:31 which scheme

37:33 which schemas which specifications do we want to work on next which ones are most important and as you can imagine it changes right I have some customer related things I need to do I

37:42 Brad: Totally.

37:42 David: have bug fixes I have RFEs I have stuff I want to do in the product I have stuff the marketing team wants me to do if you're doing user-driven design right that that be is it so you have all these things competing for your attention on the engineering task

37:58 And so we leverage these capabilities that we deliver,

38:04 you know, in these other industries,

38:05 we actually leverage them in our process. So when I say, you know, we're moving away from GitLab, it's like, yeah,

38:13 we're moving,

38:14 we still leverage GitLab, don't get me wrong.

38:17 But we're not, that's not the primary user interface.

38:20 Brad: Yeah, sure.

38:21 David: Like,

38:21 Brad: That makes sense.

38:21 David: it's more the record,

38:23 it becomes a system of record.

38:24 record for everything that we're doing and and less of control a human-centered control now we use our own tools for the human-centered process so we're excited about that and then you know that leverages our federated data layer which means that you know wherever we have data right you say well what what Federation do you have in your development environment well I have multiple development environments

38:50 Right. I have test beds for customers.

38:52 I have my staging environments. I have integration test results going into different systems.

38:57 GitLab is a system.

38:58 Right.

38:59 So all of those sources.

39:03 are federated, right?

39:04 Brad: Yeah.

39:04 David: You might say, well,

39:06 why most people, they just use a model and it can poke into all those things. Okay,

39:10 great. A model can poke into all those things, but a model can't really make sense of all those things and they can't do it at scale.

39:17 And so what we're doing with our federated data layer is we're making that information accessible to the agents in a way that they can digest it in a way that they could.

39:28 consume it and make sense out of it but we in our tooling can put um you know controls around it and

39:35 Brad: Yeah.

39:35 David: so that's where we build um evidence blocks with hashes on the data so that once you've made a decision i always kind of joke with my cpo because he's always complaining when he designs something that we we we don't implement it in production he's like oh you forgot that feature people really like that feature

39:55 We will know when we get this right,

39:57 when I...

39:59 point back to the decision and say you attested to change in the spec and that's why it's not in the system which

40:07 Brad: Yeah,

40:07 David: is a piece

40:07 Brad: right.

40:08 David: that GitLab doesn't none of these systems cover today and and that's that's where we're getting to now

40:15 Brad: That's pretty cool.

40:16 That's pretty cool.

40:17 So I have a couple of questions.

40:19 The first question is around the area of you learned so much in this process.

40:25 And do you think that you would.

40:29 Like, how would you summarize your advice?

40:33 What advice would you give to other CTOs or other engineering leaders who are trying to figure out how to solve the problem of we need to move fast, but we need quality and, you know, there's all this noise and noise in the zeitgeist about you should be using agents to write code and you shouldn't be using agents.

40:56 agents to write code,

40:57 what would your advice be to engineering leaders at the beginning of this path?

41:02 David: That's easy.

41:03 Good leaders know that it's all about the people in an organization.

41:08 If you can empower them,

41:10 if you can enable them,

41:11 if you understand what makes us as humans happy to go to work every day, excited to go to work, excited to work on something.

41:21 that comes down to understanding what you're building knowing what you're creating no understanding how have you ever worked on a project and you're like yeah i don't really understand how users you

41:31 Brad: Right.

41:31 David: know my customers use this thing now i'm just doing the ticket they told me to work on right i think we've all had that experience at one point or another um you know i don't know why we're building this thing this way right um so when you

41:46 enable your people with these tools and I don't mean just give them access to cloud code right or cloud co-worker you know whatever open cloud whatever it is you want to give them get lab do um that's not what I'm talking about what I'm talking about is when you really center around the people in the organization

42:08 how they can be happier as individuals,

42:11 how they can be better at working together,

42:13 then you can ask yourself, how can I use these tools to accomplish those things?

42:18 And what we hit on in the very beginning was,

42:22 wow,

42:24 we really enjoy these tools because I'm not getting distracted with the coding.

42:28 I'm able to think about what I'm coding more,

42:32 and that's more fun.

42:34 Brad: Yeah, totally.

42:35 David: So that's what we chose to focus on first and foremost is how do we get more of that?

42:41 Do we want to move faster?

42:43 Yes.

42:43 Do I want to hire 100 people to do these tasks?

42:47 No.

42:47 Right.

42:48 I want to remain profitable. I want to do things quick.

42:51 I want to do them with high quality.

42:52 I want to do them with high level of security and compliance and documentation and testing and all the other things. And I want to communicate it to my marketing.

43:01 marketing team and and and so on and so forth and and I want them to communicate with me

43:06 all of those things make a company successful but

43:09 Brad: Yeah.

43:10 David: it's the people first and foremost and when you focus on that I can't stress it enough if you focus on that and that would kind of lead me to the next thing that we're doing with SDD is we're adapting into those other parts of the company we we are developing marketing

43:27 specs, right?

43:28 Brad: Yeah.

43:29 David: Our marketing team is developing specs.

43:32 They're marketing focused.

43:34 We think the market needs X,

43:35 Y, or Z.

43:36 Brad: Yeah, totally.

43:36 David: We say, fine,

43:38 you can use these tools,

43:39 make us a spec.

43:41 Here are the guidelines.

43:42 Here are the tools for making a spec,

43:44 planning a spec.

43:45 We have tools for planning specs and reviewing specs and other things.

43:50 So we give them access to all those things so the marketing team could do it.

43:53 do it.

43:54 Our customer success team,

43:55 customers want these features.

43:57 They want these things. Okay,

43:59 let's make it a spec,

44:00 right?

44:01 So what that does is it ensures that service folks,

44:08 marketing folks,

44:09 sales folks,

44:10 sales engineers.

44:12 Everybody is understanding what we're doing.

44:15 Brad: Yeah, totally.

44:16 David: That is such a much stronger company to build.

44:20 Brad: Yeah, it's funny.

44:21 It's so interesting you say this.

44:23 I feel like one of the earliest things I learned as an engineer, you know, some 35 plus years ago was that.

44:35 I was always more effective if I understood the business need.

44:40 Right.

44:40 David: Yeah

44:41 Brad: Like I can I can build anything if I understand the business need.

44:44 And then when I first became a manager of other engineers,

44:48 like that became my mantra.

44:50 Like, let's let's make sure the engineers on the team really understand the business need.

44:57 I mean, sure, there's certainly some engineers who say, well,

45:00 the real joy I get is from, you know, placing the curly brackets exactly in the space that makes the magic happen.

45:05 happen you know there are people that like the the process of the coding but i find that i always found that the engineers that were most valuable to an organization most effective you know like you want to call them the 10x engineers they were the ones that really got the business need and and loved understanding the business need

45:27 David: And can communicate.

45:28 Brad: And can communicate that.

45:30 So it sounds like it sounds like it sounds like I guess if I was to repeat your advice,

45:36 I think what I'm saying or reflect back your advice, I think what I hear you saying is great companies are made by.

45:44 teams that are enjoying what they're doing and motivated to solve real problems for customers.

45:49 And so anything that you can do to facilitate that is what's going to make you more successful.

45:56 David: I can give a concrete example.

45:59 Our CPO there for a couple months throughout this process, this maturation process that we've been going through,

46:04 you know, he grabbed on to chat GBT because he found it to be.

46:10 have it to be an incredible planning tool and he is a phenomenal communicator and he understands the value of communication right not just verbally but also through documentation and other means visually you know diagrams and other things and we always used to make fun of me over we've worked together for decades you know because

46:34 He would always come to the table with a PowerPoint deck.

46:37 Hey, I have an idea.

46:39 Here's the PowerPoint,

46:40 right?

46:40 He's a very visual person.

46:41 He understands how people learn.

46:43 And so he tries to hit all the bases so that people can understand what he's talking about.

46:48 When he started vibe coding,

46:50 he was throwing over these 20 page, 30 page, 50 page screeds, right?

46:55 It's all, you know, chat threads.

46:58 Here it is. This is the description. This is

47:01 This is the solution,

47:01 right?

47:02 And he kept throwing them over the fence at us.

47:05 And I would say to him, nobody's reading them.

47:09 Right.

47:09 Brad: Right.

47:09 David: Nobody wants to read the output of your chat. Right. Nobody wants to read your chat thread.

47:15 I don't know. Maybe maybe you enjoy it. I don't enjoy it. You know, you have to have the thought process that goes along with it. Right. There's there's there's something more dynamic happening there than just words being printed on a page.

47:28 And that is how we got into this direction in the first place.

47:33 As

47:33 Brad: Yeah.

47:33 David: we said, we need to be focused on those things,

47:37 right?

47:38 So how are we going to get focused on those things?

47:41 And engineers don't want to sit there and read for four hours straight and then go write code or vibe code,

47:47 Brad: Right,

47:47 David: you

47:47 Brad: right.

47:48 David: know?

47:48 So the question became all these things that we're talking about.

47:54 How do we make it enjoyable?

47:55 Well,

47:55 I need to understand what we're doing.

47:57 Our answer six months ago would have been then read the docs,

48:00 right,

48:00 RTFD.

48:02 But people don't want to do that.

48:04 Brad: Right.

48:05 David: What do we do, right?

48:06 We Google the problem,

48:07 look it up in Stacktrace. Oh, okay,

48:09 that's my,

48:09 okay,

48:10 you know, that's what we've been trained to do, right?

48:13 You know, find a similar solution and then apply it in your context.

48:17 So,

48:18 no,

48:18 nobody wants to read the docs.

48:20 But having a spec-driven process,

48:24 the specs then become,

48:26 and they're broken down and decomposed in a way where they're consumable.

48:30 Brad: Yeah.

48:30 David: Right.

48:30 I don't need to read 300 pages.

48:33 I can read the section that I care about. I can read a subsection.

48:36 I can have a conversation about it. We're going to talk about that today.

48:40 Okay.

48:40 I can spend 20 minutes reading a page or two and then I can come with thoughts.

48:45 And that makes our CPO very happy.

48:49 It makes our engineers very happy.

48:50 Brad: Yeah.

48:50 David: It makes our marketing team very happy. It makes our customer success team very happy. It makes our CEO happy because now it's kind of clicking,

48:59 right?

48:59 It's kind of clicking and everybody's working together and it's facilitating those interactions that you're supposed to have between all these different groups in order to have a successful product,

49:11 a successful company,

49:13 a successful team.

49:14 Brad: And the key,

49:15 it sounds like, is you're focusing on what really matters, which is what is the actual problem that you're solving?

49:21 What's the business need?

49:22 How's that manifest as the product?

49:25 David: That's right.

49:26 Brad: This is super interesting.

49:27 I really appreciate you kind of describing this,

49:30 and it's a pretty...

49:33 It's a pretty, it's a pretty, I don't want to say miraculous turnaround,

49:37 but it's a great, it's a great arc of sort of starting with the chaos and ending with a process that is working for you and your team.

49:47 And I love it. I think, I think you're taking this to exactly the right next level.

49:55 And so.

49:56 Kudos to you and the team.

49:57 David: Thank you. Kind of glad Scott wasn't here

50:00 Brad: No,

50:01 I actually think Scott's super disappointed. He's going to hear this episode. He's going to be disappointed because he is I mean, I think this is exactly what he is working on trying to implement at Tash Grab It.

50:14 And it's it's it's it sounds like it's you know, it's a great proof point.

50:19 And so the naysayers.

50:21 The naysayers can naysay

50:23 David: I

50:23 Brad: all they want.

50:23 David: don't want them to steal all my secrets.

50:27 Brad: Well, awesome.

50:29 It was great talking to you, Dr.

50:31 David Bauer,

50:32 David: Yeah,

50:32 Brad: and I appreciate the time that you spent.

50:35 And yeah,

50:37 I look forward to hearing more about your team's success in the future.

50:41 Keep us posted.

50:43 David: thank you so much. I really appreciate having me on.

50:45 Thank you.

50:46 Brad: Yeah, you're welcome.

50:46 And to the audience,

50:48 we will see you guys next week on another episode of the How Many CTOs Does It Take podcast.

50:53 This episode of the How Many CTOs Does It Take podcast was produced and edited by Saffron Heptacow.

50:59 Our theme song was AI generated by Brad.

51:02 Please follow us on all the social media platforms at HowManyCTOsPod.

51:07 You can find us on YouTube,

51:09 LinkedIn,

51:10 Instagram,

51:11 Facebook.

51:11 Look Reds X TikTok and Patreon or at our website at HowManyCTOsPod.com and remember to rate,

51:19 review and subscribe on the podcast platform of your choice.

51:22 Thanks for listening.