Transcript: From Chaos to Control: Bringing Specification Driven Design to Vibe Coding with David Bauer
0:00 David Our philosophy,
0:01 David so yes, we're still doing unit tests,
0:03 David but we care much, much more about the integration tests.
0:06 David The integration tests for us are the gold standard.
0:25 Brad Welcome, everybody, to another episode of the How Many CTOs Does It Take podcast?
0:29 Brad I'm Brad Hefta-Gaub-Galb, and our regular co-host Scott Porat is not with us today because he's traveling on business.
0:34 Brad But we have a very interesting guest with us, Dave Bauer.
0:38 Brad And Dave,
0:39 Brad why don't you introduce yourself?
0:40 David Yeah, thanks,
0:41 David Brad.
0:41 David Thank you for having me.
0:43 David So I am Dr. David Bauer.
0:46 David I've got a PhD in computer science.
0:49 David My background is primarily in distributed computing.
0:51 David computing.
0:52 David And for the last 26,
0:55 David 28 years, I've been working primarily in the U.S. intelligence community.
1:00 David Did some startups when I was in graduate school, did three or four startups,
1:04 David had phenomenal timing,
1:06 David 2000,
1:06 David 2001.
1:08 David So I definitely hit the dot-com bubble and blew up a bunch of companies that way.
1:11 Brad That's always a fun time.
1:14 David Yeah, and then I decided to get back into it in 2018 and everything was cooking with gas and doing really well and we have learned a lot of things in the intelligence community about AI and I wanted to take a commercial and started a company in 2018 and then COVID hit.
1:34 David So we actually did fairly well through COVID but then once COVID was over.
1:39 David then we went in a different direction.
1:42 David And what I got really interested in while I was working on that AI startup was federated learning and how to, you know, kind of move the model training process and the model serving process down to the data as opposed to the other way around.
1:56 David And so that is what Exonis AI is all about.
1:59 David It's actually not just federated learning for machine learning.
2:04 David But it's actually leveraging and so what we've done is we've created a federated data layer and then we use that for many different things. We use it for data fusion,
2:13 David semantic ontologies and a bunch of decision intelligence.
2:17 David And so that's what we're doing today.
2:20 Brad That sounds like some pretty interesting stuff for sure.
2:24 David Absolutely. Absolutely.
2:26 Brad Tell me a little bit about how not only what you've built, but I would love to hear about how you're building it.
2:32 Brad You know, obviously nowadays it's like the big conversation is,
2:36 Brad can we safely write software?
2:39 Brad using AI agents.
2:42 Brad And I think it's interesting how some of the big frontier models, they're like,
2:47 Brad oh yeah,
2:48 Brad we use AI to do all of our,
2:49 Brad build all of our stuff.
2:51 Brad So
2:51 David Yes.
2:51 Brad I'm just kind of curious what your take on that is.
2:53 David We're a startup.
2:55 David I think we've been around for about two years now.
2:59 David kind of coming out of stealth in the last six months or so.
3:02 David And we have to use it, right?
3:05 David We feel like we don't have a choice, right?
3:08 David If we don't use it to accelerate what we're doing and to maximize our people and our experience and our skills, then we're going to be at a disadvantage.
3:15 David We're going to fall behind and we won't be successful.
3:18 David So not only do we have to use it, it's not a begrudging use it,
3:22 David we want to use it.
3:24 David because we want to get more done.
3:26 David We want to get higher quality work done.
3:29 David And we want to, you know, also,
3:33 David you know, improve all the things that engineers don't like to do, right? Documentation and QA and, you know, diagrams and other things, right?
3:46 David So for us, it has really been kind of a crazy.
3:51 David crazy whirlwind six months this last six months and knowing that we have to go in this direction we have spent a lot of time actually recreating our entire internal process and and re-engineered the entire thing and it really kind of started about four months ago everybody's moving to MCP services right everybody's building MCP tools on top of their existing capabilities
4:16 David buddies and you know we're no exception and then we very quickly hit on the whole notion of hey we need an mcp gateway so now we're going in the direction of now we have to build an mcp gateway you know along the way we're kind of internally encouraging everybody just use it right it doesn't matter cloud gemini you know chat gpt we don't care use your favorite we'll pay for it let's figure it out
4:42 Brad Yeah.
4:42 David And it was good at first, and then it got crazy very quickly.
4:48 David Right.
4:50 David And, you know, suddenly I'm getting all sorts of crazy commits.
4:54 David You know, I'm getting all sorts of crazy PRs that I'm trying to, you know, make sense out of.
5:00 David And, you know, just we're losing traction,
5:04 David right?
5:04 David Because PRs aren't getting more.
5:06 Brad Wait,
5:06 Brad so hold on a second. Hold on a second,
5:08 Brad Dave. Let me rewind there and just make
5:10 David Yeah.
5:10 Brad sure I'm catching what you're laying down.
5:12 Brad So you did the mandate of like,
5:15 Brad not a mandate,
5:16 Brad but you said,
5:16 Brad hey,
5:16 Brad you're free to use this stuff. We think this is important for us, for our velocity.
5:22 Brad We are in a figure out what works for you mode.
5:25 Brad So we're tool agnostic and we're kind of like,
5:28 Brad please go for it.
5:30 Brad And you had a side effect that was unexpected of that.
5:33 Brad bit of that of too much velocity or too low quality or what when you said it's sort of like you had that moment what what was that moment oh
5:43 David That moment was every PR that we were reviewing.
5:48 David It just wasn't, it was not correct.
5:52 Brad okay correctness
5:54 David You would get.
5:57 David 150 changes on a PR,
6:00 David right?
6:00 Brad Mm hmm.
6:01 David So how can a human read through 150 changes in a single PR and then you have, I don't know, 10 of them a day,
6:09 David right?
6:11 David They're coming at you faster, right?
6:12 David Everybody's vibe coding and they're coming in really quick.
6:14 Brad Yeah,
6:15 Brad totally.
6:15 David You have to do them in the right order and make sure everything merges and get rid of the conflicts and oh, by the way,
6:21 David here's 200 changes.
6:22 David your SWIFTY changes, your 75 changes.
6:24 Brad Hmm.
6:25 David And so there are definitely like kind of core components in the code that we really care about a lot more than others.
6:32 David And, you know, for example, security related aspects, authentication,
6:37 David that kind of stuff,
6:38 David data level security and ABAC controls.
6:42 David And so, you know, we kind of would go and scrutinize those first and we realized.
6:47 David this is way off track and you know with any large software project I think we've probably got you know well over a million lines of code you
6:56 Brad Sure.
6:56 David know there's going to be these side effects right you make a change here and it affects something else over there and in the beginning we're like okay
7:06 David I'll roll the dice and
7:07 Brad Yeah.
7:08 David I'll merge that PR,
7:10 David even though I didn't really look at all 250 changes and things went south from there,
7:16 David right? So then we couldn't get through testing,
7:18 David we couldn't get releases out,
7:21 David right?
7:21 David Just we started to, you know, have too much broken code, right? Too many breaking changes.
7:27 David And then
7:29 Brad Can you give
7:29 David Yeah.
7:30 Brad the listeners a little bit more framing about scale of the team and kind of scope of the types of projects you're dealing with?
7:42 David So we have about 30 people on the team.
7:46 David Probably 20 of those are engineering minded.
7:49 David My founder and partner,
7:51 David Chris Yonklis,
7:53 David he is a mad scientist, right?
7:55 David Right. He's big brain.
7:57 David He's been in the field for about 30 years,
7:59 David 35 years in the intelligence community.
8:02 David And, you know.
8:04 David Between him and I,
8:05 David we've had every experience imaginable when it comes to working with data,
8:08 David when working with systems,
8:10 David large-scale systems.
8:12 David By coding and unlocked for us the ability to get a lot more done with a smaller team much more quickly.
8:20 Brad Yeah, sure.
8:20 David But putting it all together was really where things started to fall apart.
8:25 Brad How did you tackle that?
8:26 David Yeah,
8:27 David so I will get to the solution.
8:29 David You know, we started building MCP servers and, you know, suddenly we kind of broke like 100 tools and 150 tools,
8:35 David things started to fall apart, right?
8:38 David Models could no longer make sense out of which tool to use.
8:41 David And it didn't matter how well we described them, you know, it would just pick the wrong tool, you know, so then we realized we need an MCP gateway,
8:50 David we need a composable architecture,
8:51 David we need to decompose things more.
8:53 David But then...
8:54 David And now you've got this, you know, many different components.
8:57 David I think we're up to about a dozen now,
8:59 David you know, and they all have to be cross-functional, right?
9:03 Brad Yeah.
9:03 David So we've encapsulated functionality fairly well,
9:07 David but they do have to be cross-functional. And so really now the challenge is how do we actually build a complete agentic architecture just for our software development process,
9:18 David for our product development process.
9:20 David that enables us to have all the speed, but have speed with control is
9:28 Brad Mm-hmm.
9:28 David really the key and not have crazy things like, you know, merges with, you know, PRs with hundreds of changes.
9:39 David And, you know, one of the other things that we have is.
9:44 David You know, I would have developers tell me, well, you have to do these four PRs in this order in order to get my change.
9:50 Brad Right.
9:51 David So it was nuts.
9:52 David So what we did was we actually spent months,
9:56 David there was months as a CTO where my
9:59 David You know, CEO,
10:01 David my CMO,
10:01 David my sales folks,
10:03 David my customer success folks would ask me, what are we doing? How are we going to do this? When are things going to come out? Why are things delayed?
10:09 David Why didn't we get a release this week?
10:12 David I would just tell them, I don't know.
10:15 David I don't know.
10:17 David Stuff's broken.
10:19 David We're figuring it out.
10:20 Brad Yeah.
10:20 David So it took about three months.
10:22 David And then we hit on a.
10:25 David essentially a systems engineering based approach to product development called specification driven design and
10:32 Brad Sure.
10:32 David I'm not going to say this is something new it's not it's not you know even necessarily our idea really what I started to do was just read everything I could right and you know the vast majority of articles are like hey here's all the reasons why vibe coding is bad I'm in a position where I can't just say it's bad and walk away right
10:50 David Right.
10:51 David I have to figure out how to make it work, which we've done.
10:55 David Specification driven design is all about specifying the functionality and the requirements of the platform that you're building or the product that you're building.
11:05 Brad Yeah.
11:06 David And so once you've got that correct,
11:09 David then what we've done is we've built an enormous number of tools around it.
11:14 David to automate the process.
11:16 David So we have maybe 120 different components in our, you know, in our SVD.
11:24 David We have an automated process for going through the development, an automated development process for those specifications. The LLMs are tuned,
11:37 David they have skills,
11:38 David they have goals, they have
11:40 David You know, we have get commit hooks,
11:42 David pre-commit hooks,
11:43 David post
11:43 Brad Yeah.
11:43 David -commit hooks, and scripts that get called from those hooks.
11:48 David We built an entire STD-based package that gets applied to the code at different times so that we can do gap analysis. So every ticket that we do now,
11:59 David every issue that we resolve now,
12:02 David it gets a gap analysis from the software or specification.
12:07 David Driven
12:07 Brad Yep.
12:07 David design.
12:08 David It gets a review for compliance with the specification.
12:13 David We actually do the implementation.
12:15 David So there are guidelines for how to do the implementation.
12:19 David We have a whole agentic framework that kind of steps each issue through this process.
12:24 Brad Yeah.
12:24 David Once it's done,
12:25 David there's a validation process,
12:29 David a review process. So, for example,
12:31 David SDD Validate.
12:32 David That tool ensures that everything, all of the tests are working,
12:36 David unit
12:36 Brad Yeah.
12:37 David tests, but then also integration tests before it can, you know, create a merge.
12:42 David And then what I'm really excited about is we've integrated that with our agile methodology.
12:50 David So I'll kind of get to the punchline here.
12:52 David Integrating it with our GitLab,
12:54 David we use GitLab.
12:56 David How everybody else uses GitLab,
12:58 David but we're not using them for technical issues.
13:00 David We're using them for spec related
13:01 Brad Yeah.
13:01 David issues.
13:02 David And the punchline is what I've seen on our team, on any team, you have people of varying levels of experience in.
13:13 David coding and other aspects of software development.
13:15 David What I've seen is we are no longer heads down focused on code,
13:20 David focused on bug fixes,
13:22 David focused on, you know, minor issues.
13:25 David Those things are distractions.
13:27 David Coding.
13:28 David is the scarce skill that used to be the bottleneck. For
13:32 Brad Yeah.
13:33 David us, it's no longer the bottleneck. Now, the scarce skill that is the bottleneck is your experience,
13:40 David your knowledge,
13:41 David your ability to describe the system,
13:43 David how to build the system.
13:45 David So the specs do go down into implementation,
13:48 David right?
13:49 David They go into implementation
13:49 Brad Totally.
13:49 David details and other things.
13:51 David And what it's done is it's unlocked our team. We have team.
13:55 David team members who only maybe understood their one component.
13:57 David Now they're understanding all of the system.
14:02 David From a day-to-day perspective, we're thinking about and we're talking about the product, right?
14:08 David We do our daily stand-ups. We don't talk about, oh, I fix bug 62.
14:12 David You know, we talk about, you know, I'm going to implement this part of the system.
14:17 David And I don't think the specification is exactly correct.
14:20 David So we all talk together about how it should work.
14:23 Brad I love it.
14:24 David And it is just a complete mental change.
14:26 Brad So this is great,
14:28 Brad obviously.
14:28 Brad um i know i you know scott's going to be disappointed he wasn't here because he's a big fan of sdd have you are you so just to clarify are you did you build your own sdd tooling or
14:40 David Yes.
14:40 Brad are you using something like spec kit or one of the other sort of
14:43 David We
14:43 Brad like interesting
14:44 David built our own.
14:45 David We built
14:45 Brad interesting
14:45 David our own.
14:46 Brad yeah
14:46 David Because what we're finding, Brad, is that the tools are not suited for vibe coding.
14:52 Brad Yeah.
14:52 David All right.
14:53 David I'll give you an example.
14:55 David GitLab. GitLab is per example.
14:57 David GitLab is designed to implement an agile process,
15:01 David an agile methodology that is based on humans taking time to
15:08 David work iterate on issues tickets
15:11 Brad Yeah,
15:11 Brad right.
15:11 David and you know the thought process is if they you know what have we always said if the issue is too small and it only takes you 10 minutes it probably wasn't defined well if the issue takes three months it's definitely not defined well right so we try to get these things down into an hour a day a week right because that's the human pace of coding
15:35 David And that's fantastic.
15:37 David It's also really slow when
15:39 Brad Right.
15:39 David you're in start vibe coding and suddenly you're closing 20 tickets a day and some of those tickets,
15:45 David you know, they maybe have lots of points associated with them.
15:49 David You know, maybe it's a week long task and then you got it done in four hours and then you knocked off a couple day long tasks as well.
15:57 David Now your velocity is,
15:58 Brad Right.
15:59 David is, is no longer suited.
16:02 David to that paradigm.
16:03 Brad Yeah, totally.
16:04 Brad So let me let me just interject there for a second, because in the start of your story, you started talking about how you were talking about sort of like the.
16:12 Brad no more than 150 lines of code uh changed or something like that and that's a common like you know the idea of like when we were humans all writing the code the idea that someone would give you a pr that was hundreds of lines of code you'd be like i can't reason about this and so people would even build like you know danger danger rules or something like that that would like automatically reject prs that were more than that amount of code and one of the things you
16:37 Brad things that I've certainly noticed, and I think that you're scratching at this, so I would really love to hear about how you guys have addressed this,
16:44 Brad is that, again, because the agents can write so much code so quickly, that the unit of work product, and you sort of referenced this earlier,
16:56 Brad like as a developer, if I'm like trying to implement an epic,
16:59 Brad and I have to break it up into five or six different tasks.
17:04 Brad Then the first four tasks aren't aren't they're testable in unit unit testable, but they aren't really like integration testable until you get to the fifth task. Right.
17:15 Brad And so like the unit of work product kind of only sees its only sees its real value or its testability or its or its you know, its usability after five or six different PRs are merged. And and I felt like you were kind of like scratching.
17:30 Brad scratching at that problem and then
17:32 David Yes.
17:32 Brad now that we give these to the agents the agent can completely implement a full epic in a in a at a single you know at a single pass and there's a benefit to that that you get you get an object at the end which is a fully criticizable fully realized fully testable fully you know critiquable thing as opposed to little pieces that are are only
17:57 Brad uh interrogatable in abstract until the final kind of everything gets glued together so like that's a benefit of the of the spits out an epic the downside is it's you know 30 000 lines of code or 3 000 lines of code either way it's
18:13 David Whatever
18:13 Brad just too
18:14 David it is,
18:14 Brad much
18:14 David yeah.
18:14 Brad too much to reason about so so one
18:19 Brad I'm assuming you are you felt that same trajectory and is in how and is your case the case you're making is,
18:28 Brad hey,
18:28 Brad SDD is what solves that or help me understand,
18:32 Brad like, like, how did you Well, bridge that gap?
18:34 David STD is a better human process for by coding.
18:40 David But you still need multiple tools around it.
18:43 David And it does impact how you work, which I mentioned.
18:49 David The one part I didn't mention is that our integration testing,
18:52 David our end-to-end platform testing,
18:54 David that fifth task now kind of becomes the zeroth task,
18:58 David right?
18:58 Brad Yeah.
18:59 David Because without that,
19:01 David it's almost impossible to validate.
19:04 David And so what we have is essentially a ongoing staging environment where things are constantly being tested end to end across the platform.
19:15 David We have a composable architecture,
19:17 David so we're able to kind of recreate the staging environment in pieces clean.
19:22 David And then, you know, you mentioned automated CICD. We've got a complete end to end automation for CICD. So that's not really the issue.
19:30 David issue it's not an issue to do what we're doing but the the the integration testing is the key that and
19:37 Brad Yeah.
19:38 David i'm not going to tell you that we don't vibe code portions of it because we do but what we really don't do is let it vibe code the tests themselves right
19:48 Brad Oh, interesting.
19:48 David what we what we will let it do is we'll let it vibe code the test cases
19:54 Brad Mm
19:54 David Right.
19:54 Brad -hmm.
19:54 David So the data for the tests,
19:57 David we will let it test.
19:59 David So, you know, negative tests, fuzzy test solids, all that, you know, the values and everything that it's testing,
20:06 David those can come from an LLM and that's fine.
20:09 David But the integration tests themselves.
20:13 David We keep that kind of close to best and really,
20:16 David you know, things can't pass until they have that.
20:20 David And so for us, that becomes the primary focus in terms of like knowing what we have is working correctly.
20:28 David And it's and it just becomes that much more important. So maybe it was a gate for a release that it passed a certain percentage of integration tests.
20:38 David you know, based on priority and whatnot.
20:40 David But now it's integrated into the front end of the development process.
20:46 David And so by the time you get to that release gate,
20:49 David right,
20:50 David it's a non-issue.
20:52 Brad Yeah, this is one of the things that I'd see in the teams that I work with that are the higher performance teams.
20:59 Brad Well,
20:59 Brad one, I think they started.
21:02 Brad kind of with a strong SDLC and philosophy around,
21:06 Brad you know, what, what their quality gates,
21:10 Brad what quality gates mattered,
21:11 Brad right?
21:11 Brad And that's what I kind of hear you talking about,
21:13 Brad like define your quality gates,
21:14 Brad be really strict about your quality gates.
21:16 Brad And
21:18 Brad And then once you can give agents very clear instructions about the quality gates,
21:22 Brad you shall not pass until this quality gate has been met at each step of the way.
21:28 Brad I think that's for sure one of the best practices that I'm seeing.
21:33 Brad That to me strikes me as like the most replicable skill,
21:39 Brad right?
21:40 Brad Like we can replicate that in any of our software projects.
21:44 David We're kind of at a point now where we're feeling like unit testing may be a thing of the past.
21:50 Brad Oh, interesting.
21:51 Brad What do you mean by that?
21:52 David So if you vibe code your unit tests,
21:56 David I mean, the models are phenomenal at working around problems.
22:00 David And they and that shows up mostly for us, at least that shows up most in the unit testing.
22:08 David So I'll give an example.
22:09 Brad Yeah, yeah, give me an example.
22:10 David We had a graph analysis to our projects, to all of our projects, because we had a model writing unit tests like mad said, yeah, 100 unit tests pass on this class.
22:25 David class is never referenced right
22:27 Brad it's Hehehehe
22:27 David just dead code right just dead code somewhere along the line it got vibed and then it got vibed away but the code's still there right so now you got to have oh I got to have a pruning right but but how do you know
22:41 David It isn't used in some other project,
22:43 Brad All
22:43 David right?
22:43 Brad right. So here.
22:43 Brad All right. So I'm going to challenge you on that one.
22:46 Brad I'm going to say I actually have taken the opposite stance and I feel pretty confident that this is a powerful way.
22:53 Brad So first of all, I totally agree with you that the agents will happily like follow your instructions,
22:58 Brad you know, and if you ask for a, you know, module to have like 100% code coverage, then it will. I had the agent just do this to me a couple of weeks ago where.
23:07 Brad I had it basically like the way it found to get 100% code coverage was to ignore all the files that didn't have 100% coverage.
23:16 Brad Like this package had, you know, 10 files in it and five of them had 100% coverage. So it just ignored the other five.
23:23 Brad So certainly you cannot, you have to watch for stuff like that.
23:26 Brad You know, when you said earlier,
23:27 Brad like, you know, AI lets, you know, the things that developers hate to do is the thing that you should give to the AI.
23:33 Brad And every developer I've ever known is heated writing unit tests.
23:37 Brad And so what
23:38 David All
23:38 Brad I
23:38 David right.
23:38 Brad advise...
23:38 David Hold on.
23:39 David Let's pause right there for a second, because one of the tests I have for new hires is I always ask them, what does your process look like for software development?
23:47 David And if they don't tell me,
23:49 David I start by developing the unit tests and figure out all the edge cases and corner cases and everything else.
23:57 David And then I write the code.
23:59 David I don't view them as a mature developer,
24:03 David right?
24:03 Brad All right, fair enough.
24:04 David So I agree with you.
24:05 David Nobody enjoys writing unit tests,
24:07 David but once you kind of get into that mode and you see the value of it,
24:11 David then it really improves your coding.
24:16 David dramatically.
24:17 David So if I'm not writing the code anymore,
24:19 David do I still need them? Do I still need to take that approach,
24:23 David right?
24:24 David And we did consider having the models generate the unit tests in a way where it would do all those same considerations and then develop the code to pass the test.
24:34 David And that may still be the viable way to do it, just like we're doing with the integration testing. Have it start building all the unit tests first.
24:43 David freeze those or lock those and then make the code match but we're afraid that it's just going to flop the problem instead of faking out unit tests now it's going to fake out code i mean you've seen the defensive programming and code right where it'll put a you know try accept or you know block around everything and always return something positive
25:05 Brad I guess the thing that I would say,
25:06 Brad I mean, I.
25:07 Brad I think this is a healthy debate,
25:09 Brad and I think that it could be a situational thing where depending on the type of software that you're building,
25:15 Brad it might change what my answer is.
25:18 Brad But where I've kind of landed right now that I feel pretty good about for kind of your,
25:24 Brad let's say it's the class of sort of like full stack web apps,
25:28 Brad which is a lot of software out there, right?
25:31 David Mm-hmm.
25:32 Brad I feel like those are.
25:34 Brad I feel like we should absolutely have the agents write unit tests for every line of code they write and we should have quality gates on the unit tests. So to your point, like,
25:47 Brad is the unit test valid,
25:48 Brad right?
25:50 Brad I think that the reasonable quality gates that you can run is you can demand 100% code coverage and 100% like 100% branch coverage,
26:00 Brad right?
26:01 Brad Like every line of this code must be covered and must be justifiable.
26:05 Brad And then you can put quality standards on the tests. So, for example, you can...
26:12 Brad Like you can set up linting rules that validate that every test has an actual assertion in it, has a reasonable assertion in it. Right.
26:21 David Yeah.
26:21 Brad And then, of course, you can certainly have the models.
26:25 Brad essentially do adversarial code reviews of the unit tests right and so it's straight and then of course you could do something like you could set up mutations right so if you if you run a mutation pass where you modify the code does the unit test fail right and so you could have like essentially like these validation passes that you run against the all of your tests whether they're unit tests or integration tests or end-to-end tests
26:51 Brad And the sheer volume of the ability for you to write 100% code coverage and 100% mutation coverage is something that was just like a typical PR is could end up being 10 times the code for testing.
27:14 Brad compared to the one line
27:15 David Yes.
27:16 Brad of code of actual work product.
27:18 Brad And it used to be very difficult to justify paying a human engineer to do that. But now I feel like just make the robots do that.
27:27 David That is a very valid point.
27:29 David And I don't want to make anyone think that we're not doing unit testing.
27:32 David We are absolutely doing unit testing.
27:35 David And we do have, you know, many of those kind of gate checks in there for it.
27:41 David It's something worth thinking about.
27:43 Brad Yeah
27:43 David And here's why.
27:45 David Now you have so and you make a very valid point.
27:50 David It's something engineers don't enjoy.
27:52 David It's something the robots can do really well.
27:54 David It's something they can do quickly.
27:55 David It's something they can do to really add quality to a project very rapidly.
28:01 David I agree with all of those things.
28:03 David I feel like you have a new tradeoff.
28:06 David How much time am I going to spend babysitting these
28:11 Brad Yeah.
28:12 David robots in the development of unit tests?
28:16 David Right.
28:16 David And how many tokens am I going to burn?
28:19 David And how much time am I going to wait for these agents to complete on those tasks as
28:25 Brad Yeah.
28:25 David opposed to net positive tasks?
28:30 David Our philosophy,
28:31 David so yes,
28:32 David we're still doing unit tests,
28:33 David but we care much, much more about the integration test.
28:36 Brad Well, that's fantastic.
28:38 Brad I agree with you that I'd much
28:40 David Integration
28:40 Brad rather have.
28:41 David tests for us are the gold standard.
28:42 Brad I totally agree with that.
28:44 David Having 100% coverage on that,
28:46 David on our APIs and on our services and everything else is really key for us to have confidence that we're delivering.
28:55 David quality software.
28:57 David And still some things can slip through the cracks.
29:00 David The question is, if you can get the robots to have your integration testing be that high,
29:04 David do you really need the unit test?
29:06 Brad Yeah.
29:07 David This is what I would really say.
29:08 David So I'm going to bend your brain a little bit more.
29:10 Brad All right, tell me.
29:11 David One of the things that we are moving away from as an engineering team is managing anything in GitLab.
29:21 Brad Okay.
29:22 Brad Yeah,
29:22 David We're
29:22 Brad tell me.
29:22 David not doing issues and PRs and other things any longer.
29:28 David We've automated our process to a point where developers don't need to do that.
29:34 David So what does that look like? What it looks like for an engineer is when they sit down,
29:40 David they can actually just what so they're not vibe coding code,
29:45 David right?
29:46 David they're in our system right they're not saying hey go grab an issue and let's go work on that issue right they're not moving it on a board and saying okay I'm going to work on that issue today what they're doing because we have the the STD
30:01 David they're actually when they talk to their model they're talking to it about the spec so
30:06 Brad Yeah, totally.
30:07 David I'm starting out my day and the first thing that I will ask my model is
30:13 David What is the highest priority issue that I need to be working on in X,
30:18 David Y, or Z component that I care about?
30:20 David In fact, RSDD is aware of the capabilities of the developers on our team based on previous commits that they've made.
30:31 David So it kind of knows what's well suited to one person or another,
30:34 David and it will offer suggestions. It'll say, here are the top three things I think you,
30:40 David the developer, should be working on right now.
30:43 Brad Right.
30:43 David The
30:43 Brad I love it.
30:43 David developer can ignore it and say, I want to work on something else. Okay.
30:46 David But they will be forced to pick something from the specification.
30:51 David And then they can begin that process that we discussed for the SSDD after that.
30:58 David But what they're really doing is they're scrutinizing, right?
31:03 David So there's gap analysis. That's kind of the first step. Here's the gap in the actual code.
31:07 David You know, this is what's not working or not implemented or whatever the case may be.
31:13 David And they are trained now to question the spec.
31:18 David Is the spec correct?
31:19 David Is this the right way to implement it?
31:22 David So when we are vibe coding,
31:24 David when we're interacting with an agent,
31:26 David we are asking it those types of questions.
31:29 Brad You're vibe specking.
31:30 David Yeah,
31:31 David we're vibe specking,
31:32 David right?
31:32 Brad Yeah.
31:33 David Our process includes the steps of,
31:36 David hey,
31:36 David you know, we're going to update the ticket,
31:38 David I'm going to move it to the in-progress state,
31:41 David I'm going to update that up in GitLab.
31:43 David We love GitLab.
31:44 David So what we've done is,
31:46 David because we do love GitLab, is we've modified how we use GitLab.
31:51 David um to to implement the std process um
31:55 Brad Yeah.
31:55 David a little bit more cleanly and so developers engineers are no longer in get lab at all everything all of that part of it is managed for them we have some of our homegrown tooling involves managing the sprint process
32:12 David um as well as managing all the agents that were better doing the vibe coding and so um it knows how to talk to gitlab for the user you know we use the glab cli so it's doing a lot of that maintenance stuff for them um and so all we're doing and get like people like me and gitlab will do the prs and things like that to make sure that things are on track and
32:35 Brad Yeah.
32:36 David like i said our days are now spent
32:39 David talking about the product talking about you know we're not pulling up an issue board and saying this is what i did yesterday this is what i'm going to do today you
32:47 Brad Yeah.
32:47 David know we're we're
32:49 David We're talking about the problem
32:51 Brad Yeah, I think that's great.
32:52 Brad The idea that you spend most of your time working with the agent to understand the problem that you're trying to solve, not the way you're going to solve the problem.
33:04 Brad I think that is 100% the right shift.
33:08 Brad That's essentially what I see in the high performance teams I'm working with. Now, each of them, they don't all necessarily use SDD.
33:15 Brad proper or spec kit proper a lot of them you know but they're effectively doing the same practices right so the engineers are essentially getting a description from the product team of the thing that the product owner wants and then they spend their time with the llm basically refining that
33:34 Brad refining that spec adding their um you know their opinionated um worldview of like this is how this component should should work in terror going back and forth like how you know what code exists to do that i like to say the engineers act like engineering managers to
33:53 David Yeah.
33:53 Brad the agents who are their engineers and
33:56 David Product
33:56 Brad in the yeah
33:57 David managers,
33:57 Brad technical
33:57 David right?
33:58 David Technical product managers.
33:59 Brad product managers who are basically like
34:01 Brad Like, and so they still need to have that technical skill to be able to.
34:07 Brad A judge,
34:09 Brad hey,
34:09 Brad is this thing that's coming back to me from the LLM, whether it be a specification or a plan or ultimately the code,
34:18 Brad they need to basically be able to interrogate each of those steps to be like, is this actually going to meet the business need?
34:25 Brad Is this practicable to be implemented the way that the thing is saying it wants to implement it? Is its assessment?
34:33 Brad Is the agent's assessment of the gap analysis,
34:36 Brad is that believable?
34:38 Brad Do I agree with its gap analysis?
34:41 Brad And when all of that is, you know, when the engineer is ready to put their name on that package,
34:49 Brad then they can say start coding.
34:52 David Yes.
34:52 David So I'm going to take it further.
34:55 David We have a product we call Exonus Decision Intelligence.
34:59 David And the purpose of the tool is to provide a data-based approach to decision-making within an organization.
35:09 David And this is most commonly used in organizations that have high compliance requirements.
35:15 David So you could think finance,
35:17 David you could think healthcare.
35:18 David You could think intelligence community with classified data,
35:21 David it doesn't matter,
35:22 David right?
35:22 David Supply chain logistics.
35:24 David In fact, we have one customer and they deploy rain sensors.
35:30 David And they told us the other day, they said, yeah, one of the things we love about your tool is there's actually a legal compliance requirement for how we do firmware updates to the sensors in the field.
35:42 David And I'm like, what do you, I never heard of such a thing, right?
35:46 David It's basically out there measuring rainfall,
35:48 David right?
35:48 David But doing a firmware upgrade has a legal repercussion that they have to watch out for.
35:55 David We built the tool for.
35:58 David industry you know industries that need to maintain compliance across their decision making process and so it's an evidence-based process we create hashes for the data so you could tell if the data has been tampered with we implement a complete you know set of steps that implement an intelligence process everything from collecting data throughout the federated environment all the way down to you know
36:25 David creating what we call effects or basically actions or tasks that come out of the decision making process and so what we've done is we've started actually implementing that tool for our STD process right so now when we talk about doing a PR right oh we always like to have a two-person sign off on a PR right have two people looked at it have two people signed off on it so we actually do attestation on our PRs in
36:50 Brad Sure.
36:51 David our own tool
36:52 David Right.
36:52 David So what does that mean?
36:55 David What it what it really means and what I'm getting to is because you hit on it.
37:01 David Other parts of our company are really, truly integrating into the engineering process.
37:07 David So our go-to-market team,
37:09 David our customer success team,
37:11 David they will now become involved in what we're doing because they can attest, is there a real business value here?
37:20 David Is this something useful for a customer?
37:23 David And we don't put the attestation just at the end, right? That actually comes into
37:27 David to, you know, prioritization of tasks,
37:30 David right,
37:31 David which scheme
37:33 David which schemas which specifications do we want to work on next which ones are most important and as you can imagine it changes right I have some customer related things I need to do I
37:42 Brad Totally.
37:42 David have bug fixes I have RFEs I have stuff I want to do in the product I have stuff the marketing team wants me to do if you're doing user-driven design right that that be is it so you have all these things competing for your attention on the engineering task
37:58 David And so we leverage these capabilities that we deliver,
38:04 David you know, in these other industries,
38:05 David we actually leverage them in our process. So when I say, you know, we're moving away from GitLab, it's like, yeah,
38:13 David we're moving,
38:14 David we still leverage GitLab, don't get me wrong.
38:17 David But we're not, that's not the primary user interface.
38:20 Brad Yeah, sure.
38:21 David Like,
38:21 Brad That makes sense.
38:21 David it's more the record,
38:23 David it becomes a system of record.
38:24 David record for everything that we're doing and and less of control a human-centered control now we use our own tools for the human-centered process so we're excited about that and then you know that leverages our federated data layer which means that you know wherever we have data right you say well what what Federation do you have in your development environment well I have multiple development environments
38:50 David Right. I have test beds for customers.
38:52 David I have my staging environments. I have integration test results going into different systems.
38:57 David GitLab is a system.
38:58 David Right.
38:59 David So all of those sources.
39:03 David are federated, right?
39:04 Brad Yeah.
39:04 David You might say, well,
39:06 David why most people, they just use a model and it can poke into all those things. Okay,
39:10 David great. A model can poke into all those things, but a model can't really make sense of all those things and they can't do it at scale.
39:17 David And so what we're doing with our federated data layer is we're making that information accessible to the agents in a way that they can digest it in a way that they could.
39:28 David consume it and make sense out of it but we in our tooling can put um you know controls around it and
39:35 Brad Yeah.
39:35 David so that's where we build um evidence blocks with hashes on the data so that once you've made a decision i always kind of joke with my cpo because he's always complaining when he designs something that we we we don't implement it in production he's like oh you forgot that feature people really like that feature
39:55 David We will know when we get this right,
39:57 David when I...
39:59 David point back to the decision and say you attested to change in the spec and that's why it's not in the system which
40:07 Brad Yeah,
40:07 David is a piece
40:07 Brad right.
40:08 David that GitLab doesn't none of these systems cover today and and that's that's where we're getting to now
40:15 Brad That's pretty cool.
40:16 Brad That's pretty cool.
40:17 Brad So I have a couple of questions.
40:19 Brad The first question is around the area of you learned so much in this process.
40:25 Brad And do you think that you would.
40:29 Brad Like, how would you summarize your advice?
40:33 Brad What advice would you give to other CTOs or other engineering leaders who are trying to figure out how to solve the problem of we need to move fast, but we need quality and, you know, there's all this noise and noise in the zeitgeist about you should be using agents to write code and you shouldn't be using agents.
40:56 Brad agents to write code,
40:57 Brad what would your advice be to engineering leaders at the beginning of this path?
41:02 David That's easy.
41:03 David Good leaders know that it's all about the people in an organization.
41:08 David If you can empower them,
41:10 David if you can enable them,
41:11 David if you understand what makes us as humans happy to go to work every day, excited to go to work, excited to work on something.
41:21 David that comes down to understanding what you're building knowing what you're creating no understanding how have you ever worked on a project and you're like yeah i don't really understand how users you
41:31 Brad Right.
41:31 David know my customers use this thing now i'm just doing the ticket they told me to work on right i think we've all had that experience at one point or another um you know i don't know why we're building this thing this way right um so when you
41:46 David enable your people with these tools and I don't mean just give them access to cloud code right or cloud co-worker you know whatever open cloud whatever it is you want to give them get lab do um that's not what I'm talking about what I'm talking about is when you really center around the people in the organization
42:08 David how they can be happier as individuals,
42:11 David how they can be better at working together,
42:13 David then you can ask yourself, how can I use these tools to accomplish those things?
42:18 David And what we hit on in the very beginning was,
42:22 David wow,
42:24 David we really enjoy these tools because I'm not getting distracted with the coding.
42:28 David I'm able to think about what I'm coding more,
42:32 David and that's more fun.
42:34 Brad Yeah, totally.
42:35 David So that's what we chose to focus on first and foremost is how do we get more of that?
42:41 David Do we want to move faster?
42:43 David Yes.
42:43 David Do I want to hire 100 people to do these tasks?
42:47 David No.
42:47 David Right.
42:48 David I want to remain profitable. I want to do things quick.
42:51 David I want to do them with high quality.
42:52 David I want to do them with high level of security and compliance and documentation and testing and all the other things. And I want to communicate it to my marketing.
43:01 David marketing team and and and so on and so forth and and I want them to communicate with me
43:06 David all of those things make a company successful but
43:09 Brad Yeah.
43:10 David it's the people first and foremost and when you focus on that I can't stress it enough if you focus on that and that would kind of lead me to the next thing that we're doing with SDD is we're adapting into those other parts of the company we we are developing marketing
43:27 David specs, right?
43:28 Brad Yeah.
43:29 David Our marketing team is developing specs.
43:32 David They're marketing focused.
43:34 David We think the market needs X,
43:35 David Y, or Z.
43:36 Brad Yeah, totally.
43:36 David We say, fine,
43:38 David you can use these tools,
43:39 David make us a spec.
43:41 David Here are the guidelines.
43:42 David Here are the tools for making a spec,
43:44 David planning a spec.
43:45 David We have tools for planning specs and reviewing specs and other things.
43:50 David So we give them access to all those things so the marketing team could do it.
43:53 David do it.
43:54 David Our customer success team,
43:55 David customers want these features.
43:57 David They want these things. Okay,
43:59 David let's make it a spec,
44:00 David right?
44:01 David So what that does is it ensures that service folks,
44:08 David marketing folks,
44:09 David sales folks,
44:10 David sales engineers.
44:12 David Everybody is understanding what we're doing.
44:15 Brad Yeah, totally.
44:16 David That is such a much stronger company to build.
44:20 Brad Yeah, it's funny.
44:21 Brad It's so interesting you say this.
44:23 Brad I feel like one of the earliest things I learned as an engineer, you know, some 35 plus years ago was that.
44:35 Brad I was always more effective if I understood the business need.
44:40 Brad Right.
44:40 David Yeah
44:41 Brad Like I can I can build anything if I understand the business need.
44:44 Brad And then when I first became a manager of other engineers,
44:48 Brad like that became my mantra.
44:50 Brad Like, let's let's make sure the engineers on the team really understand the business need.
44:57 Brad I mean, sure, there's certainly some engineers who say, well,
45:00 Brad the real joy I get is from, you know, placing the curly brackets exactly in the space that makes the magic happen.
45:05 Brad happen you know there are people that like the the process of the coding but i find that i always found that the engineers that were most valuable to an organization most effective you know like you want to call them the 10x engineers they were the ones that really got the business need and and loved understanding the business need
45:27 David And can communicate.
45:28 Brad And can communicate that.
45:30 Brad So it sounds like it sounds like it sounds like I guess if I was to repeat your advice,
45:36 Brad I think what I'm saying or reflect back your advice, I think what I hear you saying is great companies are made by.
45:44 Brad teams that are enjoying what they're doing and motivated to solve real problems for customers.
45:49 Brad And so anything that you can do to facilitate that is what's going to make you more successful.
45:56 David I can give a concrete example.
45:59 David Our CPO there for a couple months throughout this process, this maturation process that we've been going through,
46:04 David you know, he grabbed on to chat GBT because he found it to be.
46:10 David have it to be an incredible planning tool and he is a phenomenal communicator and he understands the value of communication right not just verbally but also through documentation and other means visually you know diagrams and other things and we always used to make fun of me over we've worked together for decades you know because
46:34 David He would always come to the table with a PowerPoint deck.
46:37 David Hey, I have an idea.
46:39 David Here's the PowerPoint,
46:40 David right?
46:40 David He's a very visual person.
46:41 David He understands how people learn.
46:43 David And so he tries to hit all the bases so that people can understand what he's talking about.
46:48 David When he started vibe coding,
46:50 David he was throwing over these 20 page, 30 page, 50 page screeds, right?
46:55 David It's all, you know, chat threads.
46:58 David Here it is. This is the description. This is
47:01 David This is the solution,
47:01 David right?
47:02 David And he kept throwing them over the fence at us.
47:05 David And I would say to him, nobody's reading them.
47:09 David Right.
47:09 Brad Right.
47:09 David Nobody wants to read the output of your chat. Right. Nobody wants to read your chat thread.
47:15 David I don't know. Maybe maybe you enjoy it. I don't enjoy it. You know, you have to have the thought process that goes along with it. Right. There's there's there's something more dynamic happening there than just words being printed on a page.
47:28 David And that is how we got into this direction in the first place.
47:33 David As
47:33 Brad Yeah.
47:33 David we said, we need to be focused on those things,
47:37 David right?
47:38 David So how are we going to get focused on those things?
47:41 David And engineers don't want to sit there and read for four hours straight and then go write code or vibe code,
47:47 Brad Right,
47:47 David you
47:47 Brad right.
47:48 David know?
47:48 David So the question became all these things that we're talking about.
47:54 David How do we make it enjoyable?
47:55 David Well,
47:55 David I need to understand what we're doing.
47:57 David Our answer six months ago would have been then read the docs,
48:00 David right,
48:00 David RTFD.
48:02 David But people don't want to do that.
48:04 Brad Right.
48:05 David What do we do, right?
48:06 David We Google the problem,
48:07 David look it up in Stacktrace. Oh, okay,
48:09 David that's my,
48:09 David okay,
48:10 David you know, that's what we've been trained to do, right?
48:13 David You know, find a similar solution and then apply it in your context.
48:17 David So,
48:18 David no,
48:18 David nobody wants to read the docs.
48:20 David But having a spec-driven process,
48:24 David the specs then become,
48:26 David and they're broken down and decomposed in a way where they're consumable.
48:30 Brad Yeah.
48:30 David Right.
48:30 David I don't need to read 300 pages.
48:33 David I can read the section that I care about. I can read a subsection.
48:36 David I can have a conversation about it. We're going to talk about that today.
48:40 David Okay.
48:40 David I can spend 20 minutes reading a page or two and then I can come with thoughts.
48:45 David And that makes our CPO very happy.
48:49 David It makes our engineers very happy.
48:50 Brad Yeah.
48:50 David It makes our marketing team very happy. It makes our customer success team very happy. It makes our CEO happy because now it's kind of clicking,
48:59 David right?
48:59 David It's kind of clicking and everybody's working together and it's facilitating those interactions that you're supposed to have between all these different groups in order to have a successful product,
49:11 David a successful company,
49:13 David a successful team.
49:14 Brad And the key,
49:15 Brad it sounds like, is you're focusing on what really matters, which is what is the actual problem that you're solving?
49:21 Brad What's the business need?
49:22 Brad How's that manifest as the product?
49:25 David That's right.
49:26 Brad This is super interesting.
49:27 Brad I really appreciate you kind of describing this,
49:30 Brad and it's a pretty...
49:33 Brad It's a pretty, it's a pretty, I don't want to say miraculous turnaround,
49:37 Brad but it's a great, it's a great arc of sort of starting with the chaos and ending with a process that is working for you and your team.
49:47 Brad And I love it. I think, I think you're taking this to exactly the right next level.
49:55 Brad And so.
49:56 Brad Kudos to you and the team.
49:57 David Thank you. Kind of glad Scott wasn't here
50:00 Brad No,
50:01 Brad I actually think Scott's super disappointed. He's going to hear this episode. He's going to be disappointed because he is I mean, I think this is exactly what he is working on trying to implement at Tash Grab It.
50:14 Brad And it's it's it's it sounds like it's you know, it's a great proof point.
50:19 Brad And so the naysayers.
50:21 Brad The naysayers can naysay
50:23 David I
50:23 Brad all they want.
50:23 David don't want them to steal all my secrets.
50:27 Brad Well, awesome.
50:29 Brad It was great talking to you, Dr.
50:31 Brad David Bauer,
50:32 David Yeah,
50:32 Brad and I appreciate the time that you spent.
50:35 Brad And yeah,
50:37 Brad I look forward to hearing more about your team's success in the future.
50:41 Brad Keep us posted.
50:43 David thank you so much. I really appreciate having me on.
50:45 David Thank you.
50:46 Brad Yeah, you're welcome.
50:46 Brad And to the audience,
50:48 Brad we will see you guys next week on another episode of the How Many CTOs Does It Take podcast.
50:53 Brad This episode of the How Many CTOs Does It Take podcast was produced and edited by Saffron Heptacow.
50:59 Brad Our theme song was AI generated by Brad.
51:02 Brad Please follow us on all the social media platforms at HowManyCTOsPod.
51:07 Brad You can find us on YouTube,
51:09 Brad LinkedIn,
51:10 Brad Instagram,
51:11 Brad Facebook.
51:11 Brad Look Reds X TikTok and Patreon or at our website at HowManyCTOsPod.com and remember to rate,
51:19 Brad review and subscribe on the podcast platform of your choice.
51:22 Brad Thanks for listening.