AI and the Data Swamp In Your Backyard with Ben Wilcox

0:00 Ben: We have a big data problem.

0:01 We need more attention to get the most value out of at least these AI systems.

0:25 Brad: Welcome, everybody,

0:25 to another episode of the How Many CTOs Does It Take podcast?

0:29 My name's Brad Hefta-Gaub, and this is my trusty co-host.

0:33 Scott: Scott Porad,

0:34 how are you doing today,

0:35 Brad?

0:35 Brad: I'm doing great,

0:36 Scott.

0:36 We have a guest with us today,

0:38 Ben Wilcox,

0:40 and he's got a pretty interesting origin story, and looking forward to chatting with him.

0:44 Ben, why don't you introduce yourself and kind of tell us what kind of brought you here.

0:49 Ben: Well, thanks for having me. I'm Brad Scott.

0:51 So Ben Wilcox, I'm the CTO and CISO of ProArch.

0:56 We're a data and AI company.

0:58 We help organizations get the most out of their data and really bring it towards intelligence.

1:04 Very big Microsoft partner.

1:06 My origin story actually started as I was a teenager.

1:09 I started on the entrepreneurial side of things and I built a web hosting business when I was 15 years old.

1:17 So, I

1:18 Brad: Love it.

1:18 Ben: instilled that until just after the dot-com bubble burst and things were kind of very challenging in the telecom industry.

1:28 I sold that off and then I moved into software development for about three years and realized I'm not a good coder.

1:36 That's not my forte.

1:39 But I am really good at organization.

1:41 I'm really good at system integrations.

1:43 and I really liked data and so I started working for a system integrator in the Microsoft realm and I thought it was just going to be like hey this is going to be you know a couple more years on the board move on do something new turned out that I love the system integration side

2:01 And I love building things.

2:03 I like being in the forefront of stuff.

2:04 And I love the fact that like if you're a Microsoft partner,

2:08 you're kind of pushing your boundaries at all times and always learning something new, right,

2:13 as Microsoft did it. And so I got a lot of opportunity early on to work in early cloud stuff,

2:19 some of the terrible days of early SharePoint.

2:22 which I'm glad to have put behind me now.

2:25 I've actually also done security practices. I built our managed security service offerings,

2:31 brought that to scale.

2:33 And today I kind of sit at the intersection of what we do from a platform perspective, how we enable our customers to meet the Microsoft.

2:43 You know, investment that they're making,

2:45 helping words understand how to get the most out of AI and doing it securely.

2:49 So that's a little bit about me.

2:51 Brad: Well, look that we didn't even didn't even take us five minutes before I got uttered

2:55 Scott: It's true.

2:56 We really only talk about two things on this podcast,

2:58 Ben.

2:59 AI and tech debt.

3:00 That's pretty much our whole.

3:02 I have a question for you about system integrator.

3:05 I have never been an assistant integrator. I've never worked with a system integrator. It's kind of.

3:09 That's not been my space in the tech world.

3:12 I've been more of a startup application software,

3:15 online application type of guy. So maybe for me and really for our 39 listeners out there,

3:21 what does system integrators do?

3:23 Ben: You have someone who has a product, right, like Microsoft, and there's a bunch of configuration,

3:28 best practices,

3:29 standardization,

3:30 things you want to do.

3:32 data migrations etc the system integrators really can bring that all together so in the context of the microsoft stack right it could be any of the products around there it could be helping organizations like so say you're a sass company right do you really want to run your own infrastructure in azure like you probably have the devops side of it but what about the security pieces and maybe some other areas out there right

3:59 Probably not. You really want to just do,

4:01 you want to run your business.

4:02 So system integrators can be a partner that can do MSP type of services,

4:09 manage security types of services, or be that partner that can help you integrate and bring those pieces together.

4:14 Scott: And so what are the, now I've got a lot of questions at this point, and we won't go too far down this road, but what are the kinds of challenges that you face doing that?

4:24 I would imagine if I'm a company and I want to say take something that I'm currently operating and have Microsoft operated, it seems like.

4:34 I guess maybe I'm just thinking out loud,

4:36 adapting how I do stuff or adapting my processes to work the Microsoft way or I don't even know. Maybe you could.

4:42 Ben: Oh, it could be, right?

4:43 You could be moving platforms.

4:44 How do you move from AWS to Azure?

4:49 It could be how do I upgrade and do it safely and maybe I don't have that expertise in-house.

4:56 The system integrator arena certainly evolved.

5:00 I mean, when I started in this space,

5:03 it was, hey, let's upgrade your servers,

5:05 we'll upgrade your applications that run on top of it,

5:08 we'll put this into a refresh cycle.

5:11 Then things moved to cloud,

5:13 made things a lot easier.

5:14 We don't have to necessarily do as much hands-on configuration of things like productivity apps like Exchange and so forth.

5:21 In some areas, Microsoft has taken that responsibility of what you do from an operations perspective and simplified it through their SaaS offerings and automation.

5:28 Scott: Mm-hmm.

5:29 Ben: The same side,

5:30 what they've also done is, you know, we have all of these platforms now,

5:33 right?

5:34 You buy Microsoft 365 and you have access to 100 different platforms, it turns out, but most of it's not secure by default.

5:41 So how do we go about enabling the best security from the get-go?

5:47 It's hard to be an expert at everything and have a subject matter expertise and work with a system integrator who can kind of pull those pieces as you need them.

5:57 Brad: Is your typical customer base going,

6:00 I mean, you started by leading with, let's say you're an application developer,

6:04 like building your own SaaS platform.

6:06 form is that a typical customer for you or is it i i would admit i naively assumed that system integrators were traditionally working with the businesses that were installing other people's software and then trying to basically get all that software to work for their specific business not necessarily delivering software to somebody else i i sort of imagined it being the you know the

6:29 the supply chain, you know, pub or the law firm or the, you know, people that were using software.

6:36 Is that just a misunderstanding on my part or is that also a typical client?

6:43 Ben: That's also a typical client.

6:44 So,

6:44 I mean, for us, we're probably a little different than most system integrators. We're a full stack system integrator, meaning like we, you know, pro arc actually.

6:55 Actually, that started as a app building company,

6:59 built ERP software.

7:01 So roots of pro-ARC are probably a little different than most system integrators.

7:05 Most system integrators started kind of on the server hardware side of it and kind of scaled up from there into adjacent services.

7:12 We went the path of application development,

7:16 data platforms.

7:18 then ProArc purchased a company that I came from, which was the cloud transformation services and cybersecurity.

7:27 And so, you know, and now that's evolved into,

7:30 hey,

7:30 guess what?

7:31 All of those practice areas are what are required for AI.

7:35 Brad: Yeah, totally.

7:36 Ben: So it's kind of evolving,

7:39 I think, you know, where we've gone and where the new.

7:44 let's say the new breed of you know system integrators are going right really have to bring a whole bunch of different capabilities it can't just be one single area

7:51 You look at what it takes to build AI,

7:54 you've got infrastructure in there, you've got platform services in there,

7:57 you've got building code on there, right?

8:00 If you're building true AI agents on there,

8:02 there's security, right?

8:04 How do you handle the governance and all those other pieces?

8:06 Those are all the things that a system integrator can bring and maybe add to a business, right?

8:12 The business may only have...

8:14 bespoke expertise in certain areas so oftentimes like in the mid market you know even in you know software development companies bring us in for security expertise bring us in for you know maybe a data expertise area where you know you want to do something new or maybe you're you want to know hey what else has everyone else been doing out here when it comes to AI like what are some of the best practices you've seen because we're able to bring some of that scale

8:41 you know best practices down quickly and ideally we come with our own accelerators and that's something like we do which is that accelerator has the big best building practices already there right it's something we're keeping up to date right so

8:56 We want to get to the part of value much quicker than worrying about all this architecture and everything else. That stuff has to get done and it takes us a lot of time to do it up front.

9:06 But once it's there, we want to get to the customer getting the value as much possible and solve that problem.

9:12 Scott: Interesting.

9:12 So, okay, well,

9:13 we got to AI quick today.

9:15 That never happens.

9:18 Brad: What did you say?

9:19 I think you said at least in the

9:22 I think the episode that just came out this week, you said something like,

9:25 gosh,

9:26 I posted on LinkedIn.

9:28 I'm sure glad things aren't moving that fast or else it would be hard to keep up.

9:32 Scott: Yeah, it's a good thing technology isn't moving fast right now.

9:38 I can't keep up. Okay,

9:39 you touched on something which I think is interesting,

9:42 which I'd love to hear about. Within the enterprise,

9:44 people want to use AI to know all about their whole organization,

9:47 right?

9:48 And all of a sudden that means like all your different data sources,

9:51 your knowledge bases,

9:53 your emails,

9:53 your chats, your whatever,

9:55 all have to suddenly become available somehow

9:57 Brad: Yeah.

9:57 Scott: and pulled together.

10:00 and you brought up and there's there's the information security piece which is interesting it seems like every everybody every SAS service or I don't even know if it's SaaS service every application every SaaS service is trying to make it so you can plug every everything's getting plugged into everything else or

10:16 Ben: Right.

10:16 Scott: it's just going to be a big giant web of

10:19 everything talking to everything what have you learned that we can because right now it seems like a mess to me so i don't even maybe you've got some knowledge here that like give us all a point in the right direction

10:30 Ben: Oh, I completely agree.

10:32 We've, as companies,

10:35 have created this data swamp.

10:37 That's my best

10:38 Scott: Data

10:38 Ben: way to put

10:38 Scott: swamp.

10:38 Ben: it.

10:39 Scott: Yeah.

10:39 Ben: Like, it's...

10:40 Brad: Not a data lake, a data swamp.

10:43 Ben: There's like alligators coming around this thing.

10:46 Like, it's, you know.

10:48 maybe hold that thing's floating around like that stuff is is what every environment is and I don't care if you're turning on copilot or using cloud or you know whatever it is right these all are coming into these unstructured data sources

11:02 Yeah, knowledge can be key,

11:04 but guess what? Old knowledge really isn't that useful anymore.

11:07 Brad: I literally just one of the companies I do fractional CTO work for that we were in a meeting with the chief data officer and he said, I love this line. He goes, I've never seen a database that was well structured once real humans started using it.

11:24 I was just like, there you go.

11:26 That's how you get the data swamps.

11:27 Ben: You know, if you remember like when we start putting stuff up in the cloud, right?

11:32 Hey, it's unlimited,

11:33 right?

11:34 Like just keep storing stuff and like, it's great.

11:36 You know, we turn on logging and, you know, hey, yeah, we'll use this data sometime.

11:42 Well,

11:43 guess what? Probably didn't.

11:44 Sitting there,

11:45 you know, you have sensitive data, you have old data, right?

11:51 I think that

11:53 What we haven't anticipated is maybe some of the things that surface when you least expect it and things that get tied together like sensitive information and non-sensitive information and a response and you're like,

12:05 how did this come up with this? Why is it talking about someone's payroll while somehow it thinks that this is right?

12:11 You know, in some context that I've read in this other document, they talk about payroll,

12:15 right?

12:16 Someone had a note or a company name or something else, and it just pulled something it thought was good,

12:22 right?

12:22 It's just guessing.

12:23 So I think we have a big data problem and probably classification problem.

12:29 Those are the two things that I think are going to...

12:34 We need more attention to get the most value out of at least these AI systems,

12:40 co-pilot cloud or whatever else.

12:42 Scott: Yeah, it's interesting.

12:43 You know, previously in the pre in the before times, we maybe didn't have the best controls around our data.

12:53 However.

12:55 We had security by inaccessibility, right?

12:58 Ben: Mm-hmm.

13:00 Brad: Bye.

13:00 Scott: Like maybe anybody who could get their hands into the data warehouse or into the production database or into something could get to a lot of data that they shouldn't have,

13:12 but they had to be able to get in there and they had to know where to look.

13:18 And now we're making it easier to get in there with all these different connectors and you don't have to know where to look anymore.

13:25 Brad: Yeah.

13:25 Scott: You can just ask and it'll find it for you and it'll find stale information.

13:32 You know, which is tricky.

13:34 Does any of you seen anybody like getting close to any answers for this or we just way too early for any answers to solve this stuff.

13:41 Brad: Wait,

13:42 and Scott,

13:42 actually,

13:43 wait.

13:43 So it's interesting.

13:44 Like you said we had security through difficulty of access,

13:49 but I actually think that what you just said is we had like.

13:55 data provenance by inaccessibility it was like yeah there's

13:59 Scott: Yes.

14:00 Brad: probably bad data in there but so few people know how to access it that we can then put the burden on them to you better only present stuff that you you have validated

14:13 Scott: Yes, that's right. Okay.

14:14 Recently,

14:15 somebody showed off, they had plugged in, I don't know, let's just say cloud.

14:20 I don't know what they plugged in. They plugged in something to our data warehouse and they asked the question,

14:25 how much revenue did we do last month?

14:27 And it said, you know, $459 million,

14:30 which is not true.

14:32 We wish it was true,

14:34 but it's not even close.

14:36 Probably because it was saying last month was April.

14:39 April and it was looking at all the Aprils in the history of the company and summing those up right yes you nailed it there Brad which is we put a burden of validation on the user and

14:52 Brad: Yeah.

14:53 Scott: I go around all the time saying to people you cannot believe anything that AI tells you

14:58 Brad: Yeah.

14:58 Well, so Ben,

14:59 how do you solve this problem or what are your, what are, or what maybe,

15:02 maybe is that.

15:04 Is that the real, is that one of the very real high nails that we're seeing?

15:08 Ben: I don't think that there's a solid, like, solid all answer here,

15:12 but it does come into getting back to, like, governance and how you are storing data, how you're going to keep it for,

15:20 you know, what is your policy of where you're going to keep it now.

15:25 This information, right? Are you going to let, you know, your employees keep, like for me,

15:30 I've been with the company for 20 years.

15:31 Do you want me to keep 20 years of email in

15:34 Scott: Mm

15:34 Ben: there,

15:34 Scott: -hmm.

15:34 Ben: right? Is that good stuff? And frankly,

15:37 for me, like as I use these different systems,

15:40 right,

15:40 I have ability to get into stuff.

15:43 Like I end up with a bunch of junky information at the time out of, for example,

15:48 Copilot.

15:48 It's not Copilot's fault. It's actually a data problem because.

15:52 You know, we have, for example,

15:54 like might have access to a customer information set,

15:58 pro-ARC information set.

16:00 I'm referencing the pro-ARC one,

16:01 but I get customer information back in mind.

16:03 I'm like, oh, this is useless.

16:05 That's segmentation and being able to really identify and label your information. So from a data loss prevention perspective,

16:12 you can do labeling,

16:13 right?

16:14 And that's really about classification.

16:16 And you can tell your AI tool,

16:17 say you're not going to be able to use this or.

16:20 It's going to be encrypted,

16:21 right? This isn't something that we're going to allow.

16:23 That's one way to fix that.

16:25 The other side of it's going.

16:27 hey if something does come up there right surfaces up into a sensitive label in the response so that I know that this is sensitive and you know if I see that in my response I'm like what is this going on here this isn't the right context so at least that it's another flag for me in there or the third is like we got to just go clean up the stuff we got to go purge some of this old information having a bunch of developers

16:52 developers in our company having lots of research and development people don't like to delete things they like to keep things around hey i'm going to get this in you know one month i'm going to need this for another project and you know it turns out you never need it ever again right we all have those long history folders in one drive or wherever you're keeping your information but it's the archive folder right you

17:16 Brad: Yeah.

17:18 Ben: So we just got to be able to tag

17:20 Brad: Don't

17:20 Ben: that.

17:20 Brad: tell anybody they have to delete that stuff, man. Watch how quickly they'll slap your hand.

17:26 Scott: You know, my personal laptop ran out of disk space this week, so I had to go through and start to delete stuff.

17:33 And it's crazy, like all the old stuff I had in there I had.

17:37 copies of disk drives from previous computers saved onto this computer all ridiculous okay but i want to get back to the main event here which is ben i think i think you have some experience from your career that might be relevant i mean i'm sure you have many that's relevant but for a particular part which is as cto's

17:59 I don't know if the right word is persuading or convincing or whatever,

18:02 convincing our businesses to pay for this stuff.

18:06 Right.

18:06 Because we see these problems,

18:08 we see these information security problems, but we have business stakeholders in general who are going to say, so what kind of revenue is that going to drive or what type of cost is that going to save us? And the answer to that question is, well, neither.

18:21 Right.

18:22 It's actually going to cost you money and it's not going to do anything for your revenue right away.

18:25 It's going to be a bank shot where now your employees may be more efficient in your experience doing all this system integration.

18:32 Or otherwise, how do you think we get the buy-in, the investment to actually do this stuff before we have a crisis and then, of course, whenever there's a crisis, a company wants to pay to fix it, but like free crisis.

18:46 Ben: So I think there's a couple of things.

18:47 So one,

18:48 right,

18:49 if you.

18:52 are talking to the operational leadership,

18:54 right?

18:54 They

18:54 Scott: Mm

18:54 Ben: may have

18:55 Scott: -hmm.

18:55 Ben: this idea of this use case of, you know, I want to use AI to do X, Y, Z.

19:01 And,

19:01 you know, I want this agent to be able to do these things.

19:03 And then we start talking about,

19:05 well,

19:05 let's talk about what sort of data this is going to be feeding,

19:10 right,

19:10 that agents can have access to.

19:12 um we start looking at that data and say hey this data is not very good right you got all these you know whether it's validation errors or you know it's not well structured it's kind of messy etc um we can't really do too much right your accuracy when we start you know using this tool we're going to measure it and you're going to have 70 accuracy

19:34 Right. Now we've got to start tuning it and getting closer and closer to try to tune that down.

19:38 But what if we cleaned your data set first and really started looking at existing data warehouse where we bring in some place new and we kind of start fresh with this, you know, exact set that's just needed for this specific problem.

19:52 Turns out we can probably get about 85% accuracy once we clean that up. And right now we've automatically instead of spending two months.

19:59 Trying to...

20:01 Brad: you know clean and you know get the AI to better filter on that we've got a much cleaner data set and you can get to actually solving your problem much better

20:09 Scott: So are you saying, Ben, that one of the – with the sort of change that we're experiencing right now with people thinking about AI,

20:18 thinking about how to automate,

20:20 use AI to automate parts of their business, that essentially is a wedge or an opportunity for us to say.

20:27 say hey this is going to go much better for you if you deal with this thing that you've been ignoring the the data swamp in your backyard like is that so that's is that part of the pitch here like

20:41 Brad: It is and start with just that use case, right?

20:43 We're not talking about cleaning up all your data.

20:45 Let's just start with that one use case,

20:46 right?

20:48 Create a new pipeline bring it in someplace else.

20:50 Um, you know get the filters going on that side so that

20:55 you start working on cleaner data sets that AI can can do and that's true I think with any of these tools right if we look at like co-pilot right who wants to spend a bunch of time you know having a problem where they have bad data out of there right the AI adoption is going to go down right these these companies are putting money

21:15 And, you know, any of these tools are all expensive,

21:18 right?

21:18 They're all $20, $30 a user at least.

21:21 Scott: I want to float an idea to the two of you and see if this resonates.

21:25 I think a lot of people are blaming hallucinations when there's some other thing that's causing the bad result.

21:38 And I think one of those examples of a thing that causes bad results is...

21:43 bad data and maybe the ai didn't hallucinate it just literally pulled up some concrete data and it was just there it was in there it was bad data it wasn't it wasn't hallucinated it was just bad data to begin with had a human looked at it and believed it they also would have hallucinated

22:03 When I see people, you know, because my main use of AI these days is, you know, speeding up developers writing code.

22:11 Brad: Mm-hmm.

22:12 Scott: And the naysayers who are like, oh, I can't trust it because it hallucinates. It's like, no,

22:18 it didn't hallucinate. It just borrowed a paradigm from your code base that was already a bad paradigm.

22:25 Look,

22:26 I can go look at your code right now and show you the exact same thing that you're complaining that it.

22:29 that it came up with you you wrote that like six months ago right so there you go i want to float that idea like people are blaming hallucination in cases where it's actually just pulling out real data that happens to be wrong or bad

22:45 I'll buy that.

22:46 Brad: I think that there's a good percentage of that.

22:48 I also think that there's a good percentage of your prompt is poor.

22:53 And so your results are poor,

22:56 right?

22:56 Scott: Mm

22:56 Brad: So you're

22:56 Scott: -hmm.

22:57 Brad: not getting the tightness, you're not getting the security,

23:00 you're not getting whatever it is, like your code's not,

23:04 you know, extra long because you chose not to go and provide a very tight description. And so AI then had to guess, right?

23:14 That's all it's doing is guessing what the next piece is.

23:17 Scott: Yeah. And when we ask developers to write code and we don't give them a specification,

23:22 guess what they do?

23:23 They guess they fill it in. And I guess like to put my, you know, because we're talking about data and the data swamp in your backyard,

23:31 like that data analyst,

23:34 that data engineer,

23:35 right?

23:37 they're having to either trust the data that they have or they're having to like you know like not necessarily guess but at least work around it and think about and whether that was a human or an or an AI agent it still slows both of them down if if they've got bad working data yeah

23:55 Brad: Ben, you said something a minute ago, which.

23:59 Scott: It slightly goes against at least my own personal conventional wisdom,

24:02 but I think you might be onto something.

24:05 You talked about taking that data and moving it into its own data source and cleaning it there or having it be clean there.

24:12 So much of at least, I don't know if it's total conventional wisdom,

24:16 but it's like been having a single source of truth for data.

24:19 And now we're talking about maybe the way to get this CMOS accurate is actually to have multiple copies of the same data.

24:27 but so that we can have clean versions of it now that I say that out loud it's a little bit like the sort of data mark model we used to follow I suppose where you had raw data and then you transformed it into something else maybe there's a pattern there we can learn from what we already did

24:42 Brad: You know a lot of the work that we do is is taking

24:46 All of these business functions,

24:48 right? Let's talk about like Salesforce,

24:51 right?

24:52 There might be an Oracle ERP or some other things, right? And the business wants to get some sort of intelligence out of the value of combining these data sources.

25:02 Maybe my approach is a little different because we're always thinking about how do we generate these insights for the business value side of it.

25:10 And those are typically bespoke systems that have their own set of parameters.

25:17 And those are generally much harder to change.

25:21 They're more impactful to change.

25:24 And so when we're trying to get to an answer of something.

25:27 we typically go through what that atom or type of processing of let's just extract the pieces that we need well i guess we'll bring it in as a bronze layer right we'll clean it there but you know as we move to silver and gold we'll clean it up and and get it to a point where gold's the one that we're ultimately leveraging on the on the ai side

25:48 I don't know. So we're just trying to pick the cleanest, easiest route and sometimes also the one that requires the least business resistance.

25:57 Scott: Mm

25:57 Brad: It's

25:57 Scott: -hmm.

25:58 Brad: not going to solve the problem that, hey, your Salesforce data still sucks and needs to be cleaned or whatever it is,

26:04 but we're at least solving the other piece,

26:07 which is, hey,

26:09 let's start working on something that actually is good.

26:11 Maybe we actually can put that data lineage in place and start seeing what we're cleaning up.

26:17 So you can go back to your business stakeholders and say,

26:20 hey, you know what?

26:20 I know how we got to this revenue number.

26:22 It came from this system.

26:24 the system you know we cleaned it up because you know this parameter and we can kind of see what changed in there businesses are complex like the revenue number that my CFO wants to see and looks at is different than what someone that's in operations is looking at on a day-to-day basis at the Yeah. end of the day the board wants to see the CFO number but the operations person needs to make decisions on it so

26:51 They have some other parameters that they're layering in.

26:53 Scott: Scott asked this question,

26:54 and I think this is kind of one of the universal questions that, you know, CTOs or CISOs or.

27:01 um cios or cd chief data officers it's like this this this mix of us that are dealing with our fingers in the technology um and obviously by the time we get to the c-suite we have to be good at communicating with our non-technical stakeholders um and you know to you know i think scott's question earlier around the thing of like so often the question comes down to like

27:26 how's it affect my top line or how's it affect my bottom line and you know obviously every time you have a new project that you're doing you've got a wedge to try to like clean things up and clean up technical effectively like this is this is like technical debt this is like technical debt in a code base this is technical debt in your

27:45 In your data lake,

27:46 you know, and it's

27:48 Data debt.

27:49 data debt.

27:50 Yeah,

27:51 exactly.

27:51 And do you?

27:54 I mean, have you found like a magic incantation to help convince the non-technical stakeholders that this is real and they have to deal with it?

28:05 Brad: it's partially about education it's also about showing business value right like if we can show value and return that that's generating some money for someone right getting investments to do stuff is cheap

28:18 So that's why I was trying to take, especially early on, take the least path of resistance to try to get to a point where we can show that, hey, this is the process you guys should be doing,

28:29 right?

28:30 um if we for example i'll take it back to one of our um consultants works with companies to help them with their co-pilot adoption right and looking at it from a perspective of you have all this unstructured data let's you know use it in more productive ways um if we start you know as one of those exercises we take a scope actually or point and we like point it to a SharePoint library and then we you know build an

28:56 Build an agent on there that can use just that subset of information.

29:01 for that business outcome people are so much happier with that input in there and they recognize and then you go through that process of explaining hey this is you know because it's really scoped narrow right we can we can you know ground the responses in your data sets make sure it's not going out to the web and adding other pieces and so forth you know they can start understanding that and I think so it's kind of a little bit of a show and tell

29:26 It's also showing that,

29:27 hey,

29:28 you can get value out of this at the end, right?

29:30 Here's an output and we can actually measure some of those outputs,

29:33 right?

29:34 Hey, look,

29:35 once we can teach people how to fish essentially,

29:37 right?

29:38 Their productivity goes like this or hey, let's, you know, once we get your agent grounded with much better data, you know, instead of a 60%, you know, quality rate or whatever metrics you want to track.

29:53 important on your responses there maybe we are at 85 you know guess what your end customers are going to be a lot happier because they're expecting you know that level of response or whatever it is so it's kind of an interesting thing so i think it's if you can show the value and you know make sure you know what's important to the audience that's consuming that then you'll be able to get and secure that extra thing right now all of a sudden you know we have all these things around sharepoint governance that have started coming

30:19 coming up customers are asking for that because they realize that people are starting to see that hey you can clean up this data you know and get rid of some of the stuff that we have or label our sensitive information so that it doesn't show up in prompts or maybe we have contractual obligations that ensure that we can't show this stuff or you can't use it in AI

30:43 you know we're meeting those obligations contractual obligations are always a great wedge but that certainly opens the door if you can't meet your customer obligations or your contractor obligations then you know it's funny money opens up there

31:00 Scott: Yeah, that is definitely true.

31:03 So

31:05 How much has changed in this space?

31:08 Like, at least in the data space,

31:10 these aren't new challenges. They're just being highlighted like never before.

31:17 Is that right,

31:18 Brad: Ben? Yeah, because you're letting something else reason over it, right?

31:21 Like, we were consuming this data before with filters.

31:26 Analysts would go write your RDI report or whatever it is.

31:29 you know they knew what to filter now we're just asking a question and expecting the filter to to you know be in place and that's not happening so you know it's getting better I mean look at the tools they ask questions now but you know it's still it's still not the same

31:50 Scott: You know,

31:50 Brad sent me something recently that someone had put out there. I can't remember what it was called. It was like an AI manifesto or something like that.

31:59 Yeah.

31:59 It was sort of modeled on the agile manifesto. I don't remember every detail of it, but if I recall,

32:04 one of the first bullet points it had there was.

32:08 taste and judgment are valuable,

32:10 something like that, right?

32:12 Yeah.

32:12 And I think what you're saying when you talk about that person building the report in PowerBI,

32:17 they were using their taste and judgment to understand,

32:21 oh,

32:22 I should filter this data out of this report or, oh, this.

32:26 table doesn't have good information in it good data in it so i shouldn't rely upon it or whatever it had knowledge the person the person had knowledge and experience that the ai i mean it isn't really intelligent right so the ai doesn't really have that taste and judgment i guess it's just making the garbage in garbage out problem worse than ever

32:49 Brad: 100%.

32:50 Scott: And I guess as CTOs or as data leaders,

32:54 as information security leaders,

32:56 we're going to have to do something with our business leaders to bring this to the surface in a different way than we ever have before.

33:06 Maybe like some of the guardrails that help us get away with not doing it are kind of melting away.

33:11 Brad: From what I see, our teams are completely collapsing.

33:18 We are doing projects, I mean, I'll go back four or five years ago, right?

33:24 Security may or may not have been in the room as someone from the infrastructure team in Azure or whoever would be maybe consulted for a little bit and brought in.

33:38 you know the data engineer would be working on his kind of piece and you know there'd be some collaboration but we're what we're seeing now and you know the developer the full stack developer would be building front end or whatever else with everything middleware but now what we end up with is all these people are working together and they're trying to solve one problem and do all

34:04 Do all of their jobs in one solution.

34:07 Scott: Mm

34:08 Brad: And

34:08 Scott: -hmm.

34:08 Brad: very much before it was,

34:10 hey, we're rolling this out,

34:11 security,

34:12 go bless it.

34:13 Now it's, you know, security has to create their security guardrails.

34:19 You know, the data team has to create their data guardrails, you know, the team that's writing the agents,

34:25 right, they need to have their solid framework and being able to make sure that you have the right agent framework in place or customization so that the agents respond in the right ways and giving the best outcome.

34:41 We are really seeing a complete collapse of the teams and how they work and really becoming a fusion team of sorts,

34:52 which security teams are learning to develop and developing teams are learning to secure.

34:59 It's really kind of interesting time I don't think I've seen ever before.

35:04 Scott: No, the whole,

35:06 my whole career at least.

35:08 has been the specialization right which is like you know you know i've always done online stuff that started out you kind of did everything and then we started developing all these different little specialties the different parts you just name some of them the data engineer i mean the data engineer even exists that phrase 10 years ago maybe 10 years ago it did but like you know it's all been a matter of specialization as the industry has matured

35:33 And now this is kind of like flipping everything up.

35:36 Brad: Yeah.

35:36 Scott: I wonder if, do you think we'll start to see a different kind of specialization just into different roles?

35:42 Because it isn't practical. Like if you look at just like human society in general, and like the evolution of our like, say, political or economies,

35:53 right? It's all about specialization,

35:55 right? From the

35:58 subsystem farmer right all the way down to now you've got a guy that's a data engineer right and so like it can't it can't be the case that we all meld into everybody does everything there has to be some sort of new specialization that's the only way this stuff scales right

36:15 Brad: Yeah, I think you just have to have adjacent skills,

36:18 right?

36:18 So you have to have

36:21 And understanding, for example, as a developer of what, you know, the SDLC process is or, you know, how to secure your stuff.

36:29 And those,

36:31 I think it's really coming up to the architects really have to collaborate a lot more in setting those frameworks in the beginning.

36:38 And then once you kind of have that set of frameworks.

36:42 Hey,

36:42 go at it, right?

36:43 Now you are sticking within the guardrails of the system and you can operate within that and that's in your guiding side and maybe that's where all the specialties are.

36:55 Architects are starting to have to have adjacent skills and starting to learn about what's important to each other.

37:00 Scott: The thing that keeps coming to my mind is, especially when you talk about, Ben,

37:04 the idea of these guardrails and stuff like that,

37:06 one of the things that I...

37:07 I'm pretty convinced of is that like we always needed those guardrails it's just that so many companies got you know got either didn't build the guardrails early or they didn't they didn't build them until they it was too late and and they needed them it was too late or it's not so much that

37:29 kind of the ai agents force us to have those guardrails we should add those guardrails for the humans right like the the thing that you know just what like a couple weeks ago somebody uh some startup had a ai agent that basically deleted their entire production database right and and it's like hold on a second like that

37:46 Okay, sure,

37:47 we can blame maybe it was cursor or cloud code and maybe we can blame for cell or whatever back end it was, you know, super base or for cell or AWS or whatever.

37:57 But the reality is that human that that agent was doing was working on behalf of that human apparently had the credentials to delete all the production databases.

38:08 Like, what was that all about?

38:10 Brad: Yeah,

38:10 like they weren't calling the fundamentals,

38:13 right?

38:13 you know robust access controls you shouldn't have had all that so that would that could have happened in a hundred different ways right you could have had a cron job could delete that database but like i guess the difference is like you know we're getting into a point where agents are taking actions right cron job might take an action but you know we're asking to actually go do things and make decisions right so essentially like you know agent go out and buy me a bunch of food like

38:40 I'd be curious, like once that becomes a commonplace of like how many people are left with this to cart,

38:45 you know, groceries that they didn't order would be a lot of people.

38:48 Scott: Yeah. Why do I have so many bananas?

38:53 Well, you said you wanted a bunch of bananas.

38:59 right but the the eager intern could have just as easily deleted the production database and in fact how many of them have

39:06 Brad: Or the person that's rushed, right?

39:08 I've had that before.

39:11 I did one of my big mistakes was I migrated the

39:16 sharepoint data the wrong direction um one day when i was overwrote the production system with like month old information because

39:25 Scott: Oh,

39:25 Brad: i was in a rush

39:25 Scott: wow.

39:26 Brad: and that that was a that

39:27 Scott: That was a bad day.

39:28 Brad: was a bad that was a bad week um but like those things can happen right like it doesn't you're just being rushed you're thinking about it um those types of things happen on a regular basis but now

39:41 You know, the ramification is, hey, this AI agent's doing it on behalf,

39:45 right, where they probably have less logic in there,

39:48 right?

39:49 They're not thinking logically, they're just guessing.

39:50 Scott: Maybe the higher order bit is that the hype around how fast everybody can move with AI is.

39:59 Ben: turning some leaders into saying we need to be moving as fast as the other guy and sure the ai agent is the tool that lets you move fast but maybe the root cause or the or the you know sort of the original sin is are you moving faster than you can reason about

40:19 Brad: I disagree a little bit.

40:21 Ben: Okay, fair enough.

40:22 Brad: I think that the agents should,

40:26 you know, if given the permission,

40:28 right, if it's not in the like, you know, ask me before you take action type mode,

40:31 they can choose to make this their judgment, it comes back what I was saying a minute ago,

40:37 the agent's judgment and taste is not the same as the humans.

40:40 And so they are making decisions where even the intern

40:44 Atture might be like, wait a second, this doesn't seem right.

40:47 And yes,

40:48 databases get deleted,

40:49 data gets copied the wrong way, but there's something about the agents that they actually lack intelligence,

40:55 they lack.

40:57 judgment and just intuitiveness that I think it's creating a slippery slope than what we had before.

41:03 Scott: They lack the business context,

41:05 right?

41:06 So it goes back to that piece of understanding what's important to the business or maybe it's what's important to me,

41:12 right?

41:13 They don't have that context.

41:15 And I think some of the tools have seen better context,

41:18 but when we start getting into like enterprise data,

41:21 that's...

41:22 a whacking there.

41:23 Brad: One of the first systems I worked on a way, way long time ago,

41:26 when you logged into the production system back in the days when you logged into a production system,

41:31 we got this message and it popped up and it said, hi, this is your friendly system administrator.

41:36 One,

41:37 play nicely with others.

41:39 Two,

41:39 think before you type.

41:41 Right.

41:41 And so it was giving you this warning. Right.

41:43 And so you knew as a person and I was, you know, 23 at the time or whatever.

41:47 And if I was about to type RM,

41:49 I was thinking before I typed. Right.

41:52 But the agent isn't thinking.

41:55 It's just doing.

41:57 Because it saw somewhere on the internet the data that it was trained on said to type RM at this moment,

42:02 or probably to type RM at this moment.

42:06 So it's slippery. It's slippery.

42:09 Now maybe we can train these agents to be skeptical,

42:11 just like we train the interns to be skeptical.

42:14 Ben: You know, the system prompt lets you do,

42:17 lets you create some.

42:19 You know, skeptical personalities, if that's the direction that you're trying to go, but but you still I still think that Ben I still think that the point Ben's making or at least the point I'm taking away from what you said, Ben, is like guardrails matter.

42:37 Scott: Yeah, they do.

42:38 Ben: guardrails matter and i just feel like it's not i'm not happy with the idea i i would be i feel dissatisfied with the idea that and and there's and i can't count the number of times in my career where this has been the fallback right is the well at least we have a human and their taste and their judgment

42:57 will prevent them from making the mistake.

42:59 It's maybe a last line of defense,

43:01 but it's not a very good last line of defense.

43:04 Brad: We need

43:05 Scott: Titan

43:05 Brad: some.

43:05 Scott: who controls like that.

43:06 That's not that's not a defensible position.

43:09 I don't think right like that's that's minimal effort on their right Titan who controls prevent that

43:15 Brad: Well,

43:16 right.

43:16 And we spent a generation building up technical controls. Now these agents have come along. We basically threw them all out. We're gonna have to rebuild.

43:22 Go them again.

43:23 Ben: oh yeah what is it like uh i don't use cloud code as much as i use some of the other tools but cloud code as a literally it's like asks you can i do this can i do this can i do this and and then they have this they basically have a you know an option called dangerously accept any tools and it's like dash dash dangerously accept any tools or something like that it's like

43:47 no i i'm sorry i cannot bring myself to to run claude code that way you know so it's it's this it's i i feel like claude code is a great example where they've got this like

43:58 they've they basically are like teetering on the fence and and falling hard off of either side either you have to give it permission for everything it does or just like yolo

44:10 Scott: I am confident that most people yell up in there.

44:15 That's what I think.

44:16 Brad: You know,

44:17 I don't know what most people do.

44:19 I actually have been accepting,

44:21 you know, I choose to do it every time. I don't want to call it choose to issue the community's time,

44:27 but I do find myself getting annoyed.

44:29 Like, it'd be better if I could just say do it.

44:32 I thought that was the promise of this thing.

44:34 Scott: Well,

44:34 it's done some of those things too on the productivity side of using Cloud a lot in PowerPoint.

44:40 And I live a little bit more dangerously in PowerPoint.

44:44 I am allowed the dangerous execution deletion of information because I'm like, hey, you know what?

44:52 OneDrive has a backup version of this,

44:55 so that's my take on it.

44:57 But other areas, you got to go back and look at it and understand what it is.

45:04 We used to make conscious decisions around all these things and what was coming next.

45:08 Now,

45:09 default's probably okay,

45:11 but maybe it's not in certain instances.

45:15 Maybe it's going outside of the parameters of what you really want it to do.

45:18 Ben: Yeah,

45:19 there you go.

45:20 Default's probably okay,

45:22 but maybe it isn't. Maybe that's, so we started with the data swamp and we've ended with the defaults might be okay,

45:31 but maybe it isn't.

45:33 There you go.

45:34 That seems like the sum of it.

45:36 Brad: I think the lesson I took from this in addition to it's worth investing to clean up your data is buyer beware. That's what the defaults are probably okay kind of means to me is like.

45:47 You can't just accept all this stuff at face value,

45:50 whether it's the data you're getting out.

45:52 or the way the tools are working you really got to know what's going on yeah

45:56 Scott: yeah maybe you have to invest in different areas right if you are going to accept those defaults right maybe you have to go back and have more rigorous accountability and code reviews and that applies to just about anything right anything that using AI it doesn't have to be just code but make sure your quality is there I mean

46:15 Brad: output reviews yeah awesome well it was a real pleasure

46:19 Ben: chatting with you ben needless to say like this is a deep topic and i think you got both my wheels and scott's wheels spinning a bit on trying to think through this quagmire of the data swamp in the backyard thanks

46:31 Brad: Thanks for coming on today,

46:33 Ben.

46:33 Scott: Just everyone remember you're not alone in your data swamp.

46:36 You're

46:37 Ben: you're

46:37 Scott: not alone.

46:37 Ben: not alone you're not alone in your data swamp there you go all right well what do we say scott

46:44 Brad: Spanish rate review.

46:46 Ben: There you go.

46:47 Brad: Tell all your friends,

46:48 blast it out of the car radio.

46:49 Everybody wants to hear this podcast.

46:53 Ben: How many CTOs podcast on your radio? Awesome.

46:56 Well, it was great talking to you, Ben, and we'll see you next week, Scott.

46:58 This episode of the How Many CTOs Does It Take podcast was produced and edited by Saffron Heptakow. Our theme song was AI Generated by Brad.

47:08 Please follow us on all the social media platforms.

47:11 at how many CTOs pod.

47:13 You can find us on YouTube,

47:15 LinkedIn,

47:16 Instagram,

47:16 Facebook,

47:17 Threads, X,

47:18 TikTok,

47:18 and Patreon,

47:19 or at our website at howmanyctospod.com.

47:23 And remember to rate, review,

47:25 and subscribe on the podcast platform of your choice.

47:28 Thanks for listening.