Transcript: AI and the Data Swamp In Your Backyard with Ben Wilcox
0:00 Ben We have a big data problem.
0:01 Ben We need more attention to get the most value out of at least these AI systems.
0:25 Brad Welcome, everybody,
0:25 Brad to another episode of the How Many CTOs Does It Take podcast?
0:29 Brad My name's Brad Hefta-Gaub, and this is my trusty co-host.
0:33 Scott Scott Porad,
0:34 Scott how are you doing today,
0:35 Scott Brad?
0:35 Brad I'm doing great,
0:36 Brad Scott.
0:36 Brad We have a guest with us today,
0:38 Brad Ben Wilcox,
0:40 Brad and he's got a pretty interesting origin story, and looking forward to chatting with him.
0:44 Brad Ben, why don't you introduce yourself and kind of tell us what kind of brought you here.
0:49 Ben Well, thanks for having me. I'm Brad Scott.
0:51 Ben So Ben Wilcox, I'm the CTO and CISO of ProArch.
0:56 Ben We're a data and AI company.
0:58 Ben We help organizations get the most out of their data and really bring it towards intelligence.
1:04 Ben Very big Microsoft partner.
1:06 Ben My origin story actually started as I was a teenager.
1:09 Ben I started on the entrepreneurial side of things and I built a web hosting business when I was 15 years old.
1:17 Ben So, I
1:18 Brad Love it.
1:18 Ben instilled that until just after the dot-com bubble burst and things were kind of very challenging in the telecom industry.
1:28 Ben I sold that off and then I moved into software development for about three years and realized I'm not a good coder.
1:36 Ben That's not my forte.
1:39 Ben But I am really good at organization.
1:41 Ben I'm really good at system integrations.
1:43 Ben and I really liked data and so I started working for a system integrator in the Microsoft realm and I thought it was just going to be like hey this is going to be you know a couple more years on the board move on do something new turned out that I love the system integration side
2:01 Ben And I love building things.
2:03 Ben I like being in the forefront of stuff.
2:04 Ben And I love the fact that like if you're a Microsoft partner,
2:08 Ben you're kind of pushing your boundaries at all times and always learning something new, right,
2:13 Ben as Microsoft did it. And so I got a lot of opportunity early on to work in early cloud stuff,
2:19 Ben some of the terrible days of early SharePoint.
2:22 Ben which I'm glad to have put behind me now.
2:25 Ben I've actually also done security practices. I built our managed security service offerings,
2:31 Ben brought that to scale.
2:33 Ben And today I kind of sit at the intersection of what we do from a platform perspective, how we enable our customers to meet the Microsoft.
2:43 Ben You know, investment that they're making,
2:45 Ben helping words understand how to get the most out of AI and doing it securely.
2:49 Ben So that's a little bit about me.
2:51 Brad Well, look that we didn't even didn't even take us five minutes before I got uttered
2:55 Scott It's true.
2:56 Scott We really only talk about two things on this podcast,
2:58 Scott Ben.
2:59 Scott AI and tech debt.
3:00 Scott That's pretty much our whole.
3:02 Scott I have a question for you about system integrator.
3:05 Scott I have never been an assistant integrator. I've never worked with a system integrator. It's kind of.
3:09 Scott That's not been my space in the tech world.
3:12 Scott I've been more of a startup application software,
3:15 Scott online application type of guy. So maybe for me and really for our 39 listeners out there,
3:21 Scott what does system integrators do?
3:23 Ben You have someone who has a product, right, like Microsoft, and there's a bunch of configuration,
3:28 Ben best practices,
3:29 Ben standardization,
3:30 Ben things you want to do.
3:32 Ben data migrations etc the system integrators really can bring that all together so in the context of the microsoft stack right it could be any of the products around there it could be helping organizations like so say you're a sass company right do you really want to run your own infrastructure in azure like you probably have the devops side of it but what about the security pieces and maybe some other areas out there right
3:59 Ben Probably not. You really want to just do,
4:01 Ben you want to run your business.
4:02 Ben So system integrators can be a partner that can do MSP type of services,
4:09 Ben manage security types of services, or be that partner that can help you integrate and bring those pieces together.
4:14 Scott And so what are the, now I've got a lot of questions at this point, and we won't go too far down this road, but what are the kinds of challenges that you face doing that?
4:24 Scott I would imagine if I'm a company and I want to say take something that I'm currently operating and have Microsoft operated, it seems like.
4:34 Scott I guess maybe I'm just thinking out loud,
4:36 Scott adapting how I do stuff or adapting my processes to work the Microsoft way or I don't even know. Maybe you could.
4:42 Ben Oh, it could be, right?
4:43 Ben You could be moving platforms.
4:44 Ben How do you move from AWS to Azure?
4:49 Ben It could be how do I upgrade and do it safely and maybe I don't have that expertise in-house.
4:56 Ben The system integrator arena certainly evolved.
5:00 Ben I mean, when I started in this space,
5:03 Ben it was, hey, let's upgrade your servers,
5:05 Ben we'll upgrade your applications that run on top of it,
5:08 Ben we'll put this into a refresh cycle.
5:11 Ben Then things moved to cloud,
5:13 Ben made things a lot easier.
5:14 Ben We don't have to necessarily do as much hands-on configuration of things like productivity apps like Exchange and so forth.
5:21 Ben In some areas, Microsoft has taken that responsibility of what you do from an operations perspective and simplified it through their SaaS offerings and automation.
5:28 Scott Mm-hmm.
5:29 Ben The same side,
5:30 Ben what they've also done is, you know, we have all of these platforms now,
5:33 Ben right?
5:34 Ben You buy Microsoft 365 and you have access to 100 different platforms, it turns out, but most of it's not secure by default.
5:41 Ben So how do we go about enabling the best security from the get-go?
5:47 Ben It's hard to be an expert at everything and have a subject matter expertise and work with a system integrator who can kind of pull those pieces as you need them.
5:57 Brad Is your typical customer base going,
6:00 Brad I mean, you started by leading with, let's say you're an application developer,
6:04 Brad like building your own SaaS platform.
6:06 Brad form is that a typical customer for you or is it i i would admit i naively assumed that system integrators were traditionally working with the businesses that were installing other people's software and then trying to basically get all that software to work for their specific business not necessarily delivering software to somebody else i i sort of imagined it being the you know the
6:29 Brad the supply chain, you know, pub or the law firm or the, you know, people that were using software.
6:36 Brad Is that just a misunderstanding on my part or is that also a typical client?
6:43 Ben That's also a typical client.
6:44 Ben So,
6:44 Ben I mean, for us, we're probably a little different than most system integrators. We're a full stack system integrator, meaning like we, you know, pro arc actually.
6:55 Ben Actually, that started as a app building company,
6:59 Ben built ERP software.
7:01 Ben So roots of pro-ARC are probably a little different than most system integrators.
7:05 Ben Most system integrators started kind of on the server hardware side of it and kind of scaled up from there into adjacent services.
7:12 Ben We went the path of application development,
7:16 Ben data platforms.
7:18 Ben then ProArc purchased a company that I came from, which was the cloud transformation services and cybersecurity.
7:27 Ben And so, you know, and now that's evolved into,
7:30 Ben hey,
7:30 Ben guess what?
7:31 Ben All of those practice areas are what are required for AI.
7:35 Brad Yeah, totally.
7:36 Ben So it's kind of evolving,
7:39 Ben I think, you know, where we've gone and where the new.
7:44 Ben let's say the new breed of you know system integrators are going right really have to bring a whole bunch of different capabilities it can't just be one single area
7:51 Ben You look at what it takes to build AI,
7:54 Ben you've got infrastructure in there, you've got platform services in there,
7:57 Ben you've got building code on there, right?
8:00 Ben If you're building true AI agents on there,
8:02 Ben there's security, right?
8:04 Ben How do you handle the governance and all those other pieces?
8:06 Ben Those are all the things that a system integrator can bring and maybe add to a business, right?
8:12 Ben The business may only have...
8:14 Ben bespoke expertise in certain areas so oftentimes like in the mid market you know even in you know software development companies bring us in for security expertise bring us in for you know maybe a data expertise area where you know you want to do something new or maybe you're you want to know hey what else has everyone else been doing out here when it comes to AI like what are some of the best practices you've seen because we're able to bring some of that scale
8:41 Ben you know best practices down quickly and ideally we come with our own accelerators and that's something like we do which is that accelerator has the big best building practices already there right it's something we're keeping up to date right so
8:56 Ben We want to get to the part of value much quicker than worrying about all this architecture and everything else. That stuff has to get done and it takes us a lot of time to do it up front.
9:06 Ben But once it's there, we want to get to the customer getting the value as much possible and solve that problem.
9:12 Scott Interesting.
9:12 Scott So, okay, well,
9:13 Scott we got to AI quick today.
9:15 Scott That never happens.
9:18 Brad What did you say?
9:19 Brad I think you said at least in the
9:22 Brad I think the episode that just came out this week, you said something like,
9:25 Brad gosh,
9:26 Brad I posted on LinkedIn.
9:28 Brad I'm sure glad things aren't moving that fast or else it would be hard to keep up.
9:32 Scott Yeah, it's a good thing technology isn't moving fast right now.
9:38 Scott I can't keep up. Okay,
9:39 Scott you touched on something which I think is interesting,
9:42 Scott which I'd love to hear about. Within the enterprise,
9:44 Scott people want to use AI to know all about their whole organization,
9:47 Scott right?
9:48 Scott And all of a sudden that means like all your different data sources,
9:51 Scott your knowledge bases,
9:53 Scott your emails,
9:53 Scott your chats, your whatever,
9:55 Scott all have to suddenly become available somehow
9:57 Brad Yeah.
9:57 Scott and pulled together.
10:00 Scott and you brought up and there's there's the information security piece which is interesting it seems like every everybody every SAS service or I don't even know if it's SaaS service every application every SaaS service is trying to make it so you can plug every everything's getting plugged into everything else or
10:16 Ben Right.
10:16 Scott it's just going to be a big giant web of
10:19 Scott everything talking to everything what have you learned that we can because right now it seems like a mess to me so i don't even maybe you've got some knowledge here that like give us all a point in the right direction
10:30 Ben Oh, I completely agree.
10:32 Ben We've, as companies,
10:35 Ben have created this data swamp.
10:37 Ben That's my best
10:38 Scott Data
10:38 Ben way to put
10:38 Scott swamp.
10:38 Ben it.
10:39 Scott Yeah.
10:39 Ben Like, it's...
10:40 Brad Not a data lake, a data swamp.
10:43 Ben There's like alligators coming around this thing.
10:46 Ben Like, it's, you know.
10:48 Ben maybe hold that thing's floating around like that stuff is is what every environment is and I don't care if you're turning on copilot or using cloud or you know whatever it is right these all are coming into these unstructured data sources
11:02 Ben Yeah, knowledge can be key,
11:04 Ben but guess what? Old knowledge really isn't that useful anymore.
11:07 Brad I literally just one of the companies I do fractional CTO work for that we were in a meeting with the chief data officer and he said, I love this line. He goes, I've never seen a database that was well structured once real humans started using it.
11:24 Brad I was just like, there you go.
11:26 Brad That's how you get the data swamps.
11:27 Ben You know, if you remember like when we start putting stuff up in the cloud, right?
11:32 Ben Hey, it's unlimited,
11:33 Ben right?
11:34 Ben Like just keep storing stuff and like, it's great.
11:36 Ben You know, we turn on logging and, you know, hey, yeah, we'll use this data sometime.
11:42 Ben Well,
11:43 Ben guess what? Probably didn't.
11:44 Ben Sitting there,
11:45 Ben you know, you have sensitive data, you have old data, right?
11:51 Ben I think that
11:53 Ben What we haven't anticipated is maybe some of the things that surface when you least expect it and things that get tied together like sensitive information and non-sensitive information and a response and you're like,
12:05 Ben how did this come up with this? Why is it talking about someone's payroll while somehow it thinks that this is right?
12:11 Ben You know, in some context that I've read in this other document, they talk about payroll,
12:15 Ben right?
12:16 Ben Someone had a note or a company name or something else, and it just pulled something it thought was good,
12:22 Ben right?
12:22 Ben It's just guessing.
12:23 Ben So I think we have a big data problem and probably classification problem.
12:29 Ben Those are the two things that I think are going to...
12:34 Ben We need more attention to get the most value out of at least these AI systems,
12:40 Ben co-pilot cloud or whatever else.
12:42 Scott Yeah, it's interesting.
12:43 Scott You know, previously in the pre in the before times, we maybe didn't have the best controls around our data.
12:53 Scott However.
12:55 Scott We had security by inaccessibility, right?
12:58 Ben Mm-hmm.
13:00 Brad Bye.
13:00 Scott Like maybe anybody who could get their hands into the data warehouse or into the production database or into something could get to a lot of data that they shouldn't have,
13:12 Scott but they had to be able to get in there and they had to know where to look.
13:18 Scott And now we're making it easier to get in there with all these different connectors and you don't have to know where to look anymore.
13:25 Brad Yeah.
13:25 Scott You can just ask and it'll find it for you and it'll find stale information.
13:32 Scott You know, which is tricky.
13:34 Scott Does any of you seen anybody like getting close to any answers for this or we just way too early for any answers to solve this stuff.
13:41 Brad Wait,
13:42 Brad and Scott,
13:42 Brad actually,
13:43 Brad wait.
13:43 Brad So it's interesting.
13:44 Brad Like you said we had security through difficulty of access,
13:49 Brad but I actually think that what you just said is we had like.
13:55 Brad data provenance by inaccessibility it was like yeah there's
13:59 Scott Yes.
14:00 Brad probably bad data in there but so few people know how to access it that we can then put the burden on them to you better only present stuff that you you have validated
14:13 Scott Yes, that's right. Okay.
14:14 Scott Recently,
14:15 Scott somebody showed off, they had plugged in, I don't know, let's just say cloud.
14:20 Scott I don't know what they plugged in. They plugged in something to our data warehouse and they asked the question,
14:25 Scott how much revenue did we do last month?
14:27 Scott And it said, you know, $459 million,
14:30 Scott which is not true.
14:32 Scott We wish it was true,
14:34 Scott but it's not even close.
14:36 Scott Probably because it was saying last month was April.
14:39 Scott April and it was looking at all the Aprils in the history of the company and summing those up right yes you nailed it there Brad which is we put a burden of validation on the user and
14:52 Brad Yeah.
14:53 Scott I go around all the time saying to people you cannot believe anything that AI tells you
14:58 Brad Yeah.
14:58 Brad Well, so Ben,
14:59 Brad how do you solve this problem or what are your, what are, or what maybe,
15:02 Brad maybe is that.
15:04 Brad Is that the real, is that one of the very real high nails that we're seeing?
15:08 Ben I don't think that there's a solid, like, solid all answer here,
15:12 Ben but it does come into getting back to, like, governance and how you are storing data, how you're going to keep it for,
15:20 Ben you know, what is your policy of where you're going to keep it now.
15:25 Ben This information, right? Are you going to let, you know, your employees keep, like for me,
15:30 Ben I've been with the company for 20 years.
15:31 Ben Do you want me to keep 20 years of email in
15:34 Scott Mm
15:34 Ben there,
15:34 Scott -hmm.
15:34 Ben right? Is that good stuff? And frankly,
15:37 Ben for me, like as I use these different systems,
15:40 Ben right,
15:40 Ben I have ability to get into stuff.
15:43 Ben Like I end up with a bunch of junky information at the time out of, for example,
15:48 Ben Copilot.
15:48 Ben It's not Copilot's fault. It's actually a data problem because.
15:52 Ben You know, we have, for example,
15:54 Ben like might have access to a customer information set,
15:58 Ben pro-ARC information set.
16:00 Ben I'm referencing the pro-ARC one,
16:01 Ben but I get customer information back in mind.
16:03 Ben I'm like, oh, this is useless.
16:05 Ben That's segmentation and being able to really identify and label your information. So from a data loss prevention perspective,
16:12 Ben you can do labeling,
16:13 Ben right?
16:14 Ben And that's really about classification.
16:16 Ben And you can tell your AI tool,
16:17 Ben say you're not going to be able to use this or.
16:20 Ben It's going to be encrypted,
16:21 Ben right? This isn't something that we're going to allow.
16:23 Ben That's one way to fix that.
16:25 Ben The other side of it's going.
16:27 Ben hey if something does come up there right surfaces up into a sensitive label in the response so that I know that this is sensitive and you know if I see that in my response I'm like what is this going on here this isn't the right context so at least that it's another flag for me in there or the third is like we got to just go clean up the stuff we got to go purge some of this old information having a bunch of developers
16:52 Ben developers in our company having lots of research and development people don't like to delete things they like to keep things around hey i'm going to get this in you know one month i'm going to need this for another project and you know it turns out you never need it ever again right we all have those long history folders in one drive or wherever you're keeping your information but it's the archive folder right you
17:16 Brad Yeah.
17:18 Ben So we just got to be able to tag
17:20 Brad Don't
17:20 Ben that.
17:20 Brad tell anybody they have to delete that stuff, man. Watch how quickly they'll slap your hand.
17:26 Scott You know, my personal laptop ran out of disk space this week, so I had to go through and start to delete stuff.
17:33 Scott And it's crazy, like all the old stuff I had in there I had.
17:37 Scott copies of disk drives from previous computers saved onto this computer all ridiculous okay but i want to get back to the main event here which is ben i think i think you have some experience from your career that might be relevant i mean i'm sure you have many that's relevant but for a particular part which is as cto's
17:59 Scott I don't know if the right word is persuading or convincing or whatever,
18:02 Scott convincing our businesses to pay for this stuff.
18:06 Scott Right.
18:06 Scott Because we see these problems,
18:08 Scott we see these information security problems, but we have business stakeholders in general who are going to say, so what kind of revenue is that going to drive or what type of cost is that going to save us? And the answer to that question is, well, neither.
18:21 Scott Right.
18:22 Scott It's actually going to cost you money and it's not going to do anything for your revenue right away.
18:25 Scott It's going to be a bank shot where now your employees may be more efficient in your experience doing all this system integration.
18:32 Scott Or otherwise, how do you think we get the buy-in, the investment to actually do this stuff before we have a crisis and then, of course, whenever there's a crisis, a company wants to pay to fix it, but like free crisis.
18:46 Ben So I think there's a couple of things.
18:47 Ben So one,
18:48 Ben right,
18:49 Ben if you.
18:52 Ben are talking to the operational leadership,
18:54 Ben right?
18:54 Ben They
18:54 Scott Mm
18:54 Ben may have
18:55 Scott -hmm.
18:55 Ben this idea of this use case of, you know, I want to use AI to do X, Y, Z.
19:01 Ben And,
19:01 Ben you know, I want this agent to be able to do these things.
19:03 Ben And then we start talking about,
19:05 Ben well,
19:05 Ben let's talk about what sort of data this is going to be feeding,
19:10 Ben right,
19:10 Ben that agents can have access to.
19:12 Ben um we start looking at that data and say hey this data is not very good right you got all these you know whether it's validation errors or you know it's not well structured it's kind of messy etc um we can't really do too much right your accuracy when we start you know using this tool we're going to measure it and you're going to have 70 accuracy
19:34 Ben Right. Now we've got to start tuning it and getting closer and closer to try to tune that down.
19:38 Ben But what if we cleaned your data set first and really started looking at existing data warehouse where we bring in some place new and we kind of start fresh with this, you know, exact set that's just needed for this specific problem.
19:52 Ben Turns out we can probably get about 85% accuracy once we clean that up. And right now we've automatically instead of spending two months.
19:59 Ben Trying to...
20:01 Brad you know clean and you know get the AI to better filter on that we've got a much cleaner data set and you can get to actually solving your problem much better
20:09 Scott So are you saying, Ben, that one of the – with the sort of change that we're experiencing right now with people thinking about AI,
20:18 Scott thinking about how to automate,
20:20 Scott use AI to automate parts of their business, that essentially is a wedge or an opportunity for us to say.
20:27 Scott say hey this is going to go much better for you if you deal with this thing that you've been ignoring the the data swamp in your backyard like is that so that's is that part of the pitch here like
20:41 Brad It is and start with just that use case, right?
20:43 Brad We're not talking about cleaning up all your data.
20:45 Brad Let's just start with that one use case,
20:46 Brad right?
20:48 Brad Create a new pipeline bring it in someplace else.
20:50 Brad Um, you know get the filters going on that side so that
20:55 Brad you start working on cleaner data sets that AI can can do and that's true I think with any of these tools right if we look at like co-pilot right who wants to spend a bunch of time you know having a problem where they have bad data out of there right the AI adoption is going to go down right these these companies are putting money
21:15 Brad And, you know, any of these tools are all expensive,
21:18 Brad right?
21:18 Brad They're all $20, $30 a user at least.
21:21 Scott I want to float an idea to the two of you and see if this resonates.
21:25 Scott I think a lot of people are blaming hallucinations when there's some other thing that's causing the bad result.
21:38 Scott And I think one of those examples of a thing that causes bad results is...
21:43 Scott bad data and maybe the ai didn't hallucinate it just literally pulled up some concrete data and it was just there it was in there it was bad data it wasn't it wasn't hallucinated it was just bad data to begin with had a human looked at it and believed it they also would have hallucinated
22:03 Scott When I see people, you know, because my main use of AI these days is, you know, speeding up developers writing code.
22:11 Brad Mm-hmm.
22:12 Scott And the naysayers who are like, oh, I can't trust it because it hallucinates. It's like, no,
22:18 Scott it didn't hallucinate. It just borrowed a paradigm from your code base that was already a bad paradigm.
22:25 Scott Look,
22:26 Scott I can go look at your code right now and show you the exact same thing that you're complaining that it.
22:29 Scott that it came up with you you wrote that like six months ago right so there you go i want to float that idea like people are blaming hallucination in cases where it's actually just pulling out real data that happens to be wrong or bad
22:45 Scott I'll buy that.
22:46 Brad I think that there's a good percentage of that.
22:48 Brad I also think that there's a good percentage of your prompt is poor.
22:53 Brad And so your results are poor,
22:56 Brad right?
22:56 Scott Mm
22:56 Brad So you're
22:56 Scott -hmm.
22:57 Brad not getting the tightness, you're not getting the security,
23:00 Brad you're not getting whatever it is, like your code's not,
23:04 Brad you know, extra long because you chose not to go and provide a very tight description. And so AI then had to guess, right?
23:14 Brad That's all it's doing is guessing what the next piece is.
23:17 Scott Yeah. And when we ask developers to write code and we don't give them a specification,
23:22 Scott guess what they do?
23:23 Scott They guess they fill it in. And I guess like to put my, you know, because we're talking about data and the data swamp in your backyard,
23:31 Scott like that data analyst,
23:34 Scott that data engineer,
23:35 Scott right?
23:37 Scott they're having to either trust the data that they have or they're having to like you know like not necessarily guess but at least work around it and think about and whether that was a human or an or an AI agent it still slows both of them down if if they've got bad working data yeah
23:55 Brad Ben, you said something a minute ago, which.
23:59 Scott It slightly goes against at least my own personal conventional wisdom,
24:02 Scott but I think you might be onto something.
24:05 Scott You talked about taking that data and moving it into its own data source and cleaning it there or having it be clean there.
24:12 Scott So much of at least, I don't know if it's total conventional wisdom,
24:16 Scott but it's like been having a single source of truth for data.
24:19 Scott And now we're talking about maybe the way to get this CMOS accurate is actually to have multiple copies of the same data.
24:27 Scott but so that we can have clean versions of it now that I say that out loud it's a little bit like the sort of data mark model we used to follow I suppose where you had raw data and then you transformed it into something else maybe there's a pattern there we can learn from what we already did
24:42 Brad You know a lot of the work that we do is is taking
24:46 Brad All of these business functions,
24:48 Brad right? Let's talk about like Salesforce,
24:51 Brad right?
24:52 Brad There might be an Oracle ERP or some other things, right? And the business wants to get some sort of intelligence out of the value of combining these data sources.
25:02 Brad Maybe my approach is a little different because we're always thinking about how do we generate these insights for the business value side of it.
25:10 Brad And those are typically bespoke systems that have their own set of parameters.
25:17 Brad And those are generally much harder to change.
25:21 Brad They're more impactful to change.
25:24 Brad And so when we're trying to get to an answer of something.
25:27 Brad we typically go through what that atom or type of processing of let's just extract the pieces that we need well i guess we'll bring it in as a bronze layer right we'll clean it there but you know as we move to silver and gold we'll clean it up and and get it to a point where gold's the one that we're ultimately leveraging on the on the ai side
25:48 Brad I don't know. So we're just trying to pick the cleanest, easiest route and sometimes also the one that requires the least business resistance.
25:57 Scott Mm
25:57 Brad It's
25:57 Scott -hmm.
25:58 Brad not going to solve the problem that, hey, your Salesforce data still sucks and needs to be cleaned or whatever it is,
26:04 Brad but we're at least solving the other piece,
26:07 Brad which is, hey,
26:09 Brad let's start working on something that actually is good.
26:11 Brad Maybe we actually can put that data lineage in place and start seeing what we're cleaning up.
26:17 Brad So you can go back to your business stakeholders and say,
26:20 Brad hey, you know what?
26:20 Brad I know how we got to this revenue number.
26:22 Brad It came from this system.
26:24 Brad the system you know we cleaned it up because you know this parameter and we can kind of see what changed in there businesses are complex like the revenue number that my CFO wants to see and looks at is different than what someone that's in operations is looking at on a day-to-day basis at the Yeah. end of the day the board wants to see the CFO number but the operations person needs to make decisions on it so
26:51 Brad They have some other parameters that they're layering in.
26:53 Scott Scott asked this question,
26:54 Scott and I think this is kind of one of the universal questions that, you know, CTOs or CISOs or.
27:01 Scott um cios or cd chief data officers it's like this this this mix of us that are dealing with our fingers in the technology um and obviously by the time we get to the c-suite we have to be good at communicating with our non-technical stakeholders um and you know to you know i think scott's question earlier around the thing of like so often the question comes down to like
27:26 Scott how's it affect my top line or how's it affect my bottom line and you know obviously every time you have a new project that you're doing you've got a wedge to try to like clean things up and clean up technical effectively like this is this is like technical debt this is like technical debt in a code base this is technical debt in your
27:45 Scott In your data lake,
27:46 Scott you know, and it's
27:48 Scott Data debt.
27:49 Scott data debt.
27:50 Scott Yeah,
27:51 Scott exactly.
27:51 Scott And do you?
27:54 Scott I mean, have you found like a magic incantation to help convince the non-technical stakeholders that this is real and they have to deal with it?
28:05 Brad it's partially about education it's also about showing business value right like if we can show value and return that that's generating some money for someone right getting investments to do stuff is cheap
28:18 Brad So that's why I was trying to take, especially early on, take the least path of resistance to try to get to a point where we can show that, hey, this is the process you guys should be doing,
28:29 Brad right?
28:30 Brad um if we for example i'll take it back to one of our um consultants works with companies to help them with their co-pilot adoption right and looking at it from a perspective of you have all this unstructured data let's you know use it in more productive ways um if we start you know as one of those exercises we take a scope actually or point and we like point it to a SharePoint library and then we you know build an
28:56 Brad Build an agent on there that can use just that subset of information.
29:01 Brad for that business outcome people are so much happier with that input in there and they recognize and then you go through that process of explaining hey this is you know because it's really scoped narrow right we can we can you know ground the responses in your data sets make sure it's not going out to the web and adding other pieces and so forth you know they can start understanding that and I think so it's kind of a little bit of a show and tell
29:26 Brad It's also showing that,
29:27 Brad hey,
29:28 Brad you can get value out of this at the end, right?
29:30 Brad Here's an output and we can actually measure some of those outputs,
29:33 Brad right?
29:34 Brad Hey, look,
29:35 Brad once we can teach people how to fish essentially,
29:37 Brad right?
29:38 Brad Their productivity goes like this or hey, let's, you know, once we get your agent grounded with much better data, you know, instead of a 60%, you know, quality rate or whatever metrics you want to track.
29:53 Brad important on your responses there maybe we are at 85 you know guess what your end customers are going to be a lot happier because they're expecting you know that level of response or whatever it is so it's kind of an interesting thing so i think it's if you can show the value and you know make sure you know what's important to the audience that's consuming that then you'll be able to get and secure that extra thing right now all of a sudden you know we have all these things around sharepoint governance that have started coming
30:19 Brad coming up customers are asking for that because they realize that people are starting to see that hey you can clean up this data you know and get rid of some of the stuff that we have or label our sensitive information so that it doesn't show up in prompts or maybe we have contractual obligations that ensure that we can't show this stuff or you can't use it in AI
30:43 Brad you know we're meeting those obligations contractual obligations are always a great wedge but that certainly opens the door if you can't meet your customer obligations or your contractor obligations then you know it's funny money opens up there
31:00 Scott Yeah, that is definitely true.
31:03 Scott So
31:05 Scott How much has changed in this space?
31:08 Scott Like, at least in the data space,
31:10 Scott these aren't new challenges. They're just being highlighted like never before.
31:17 Scott Is that right,
31:18 Brad Ben? Yeah, because you're letting something else reason over it, right?
31:21 Brad Like, we were consuming this data before with filters.
31:26 Brad Analysts would go write your RDI report or whatever it is.
31:29 Brad you know they knew what to filter now we're just asking a question and expecting the filter to to you know be in place and that's not happening so you know it's getting better I mean look at the tools they ask questions now but you know it's still it's still not the same
31:50 Scott You know,
31:50 Scott Brad sent me something recently that someone had put out there. I can't remember what it was called. It was like an AI manifesto or something like that.
31:59 Scott Yeah.
31:59 Scott It was sort of modeled on the agile manifesto. I don't remember every detail of it, but if I recall,
32:04 Scott one of the first bullet points it had there was.
32:08 Scott taste and judgment are valuable,
32:10 Scott something like that, right?
32:12 Scott Yeah.
32:12 Scott And I think what you're saying when you talk about that person building the report in PowerBI,
32:17 Scott they were using their taste and judgment to understand,
32:21 Scott oh,
32:22 Scott I should filter this data out of this report or, oh, this.
32:26 Scott table doesn't have good information in it good data in it so i shouldn't rely upon it or whatever it had knowledge the person the person had knowledge and experience that the ai i mean it isn't really intelligent right so the ai doesn't really have that taste and judgment i guess it's just making the garbage in garbage out problem worse than ever
32:49 Brad 100%.
32:50 Scott And I guess as CTOs or as data leaders,
32:54 Scott as information security leaders,
32:56 Scott we're going to have to do something with our business leaders to bring this to the surface in a different way than we ever have before.
33:06 Scott Maybe like some of the guardrails that help us get away with not doing it are kind of melting away.
33:11 Brad From what I see, our teams are completely collapsing.
33:18 Brad We are doing projects, I mean, I'll go back four or five years ago, right?
33:24 Brad Security may or may not have been in the room as someone from the infrastructure team in Azure or whoever would be maybe consulted for a little bit and brought in.
33:38 Brad you know the data engineer would be working on his kind of piece and you know there'd be some collaboration but we're what we're seeing now and you know the developer the full stack developer would be building front end or whatever else with everything middleware but now what we end up with is all these people are working together and they're trying to solve one problem and do all
34:04 Brad Do all of their jobs in one solution.
34:07 Scott Mm
34:08 Brad And
34:08 Scott -hmm.
34:08 Brad very much before it was,
34:10 Brad hey, we're rolling this out,
34:11 Brad security,
34:12 Brad go bless it.
34:13 Brad Now it's, you know, security has to create their security guardrails.
34:19 Brad You know, the data team has to create their data guardrails, you know, the team that's writing the agents,
34:25 Brad right, they need to have their solid framework and being able to make sure that you have the right agent framework in place or customization so that the agents respond in the right ways and giving the best outcome.
34:41 Brad We are really seeing a complete collapse of the teams and how they work and really becoming a fusion team of sorts,
34:52 Brad which security teams are learning to develop and developing teams are learning to secure.
34:59 Brad It's really kind of interesting time I don't think I've seen ever before.
35:04 Scott No, the whole,
35:06 Scott my whole career at least.
35:08 Scott has been the specialization right which is like you know you know i've always done online stuff that started out you kind of did everything and then we started developing all these different little specialties the different parts you just name some of them the data engineer i mean the data engineer even exists that phrase 10 years ago maybe 10 years ago it did but like you know it's all been a matter of specialization as the industry has matured
35:33 Scott And now this is kind of like flipping everything up.
35:36 Brad Yeah.
35:36 Scott I wonder if, do you think we'll start to see a different kind of specialization just into different roles?
35:42 Scott Because it isn't practical. Like if you look at just like human society in general, and like the evolution of our like, say, political or economies,
35:53 Scott right? It's all about specialization,
35:55 Scott right? From the
35:58 Scott subsystem farmer right all the way down to now you've got a guy that's a data engineer right and so like it can't it can't be the case that we all meld into everybody does everything there has to be some sort of new specialization that's the only way this stuff scales right
36:15 Brad Yeah, I think you just have to have adjacent skills,
36:18 Brad right?
36:18 Brad So you have to have
36:21 Brad And understanding, for example, as a developer of what, you know, the SDLC process is or, you know, how to secure your stuff.
36:29 Brad And those,
36:31 Brad I think it's really coming up to the architects really have to collaborate a lot more in setting those frameworks in the beginning.
36:38 Brad And then once you kind of have that set of frameworks.
36:42 Brad Hey,
36:42 Brad go at it, right?
36:43 Brad Now you are sticking within the guardrails of the system and you can operate within that and that's in your guiding side and maybe that's where all the specialties are.
36:55 Brad Architects are starting to have to have adjacent skills and starting to learn about what's important to each other.
37:00 Scott The thing that keeps coming to my mind is, especially when you talk about, Ben,
37:04 Scott the idea of these guardrails and stuff like that,
37:06 Scott one of the things that I...
37:07 Scott I'm pretty convinced of is that like we always needed those guardrails it's just that so many companies got you know got either didn't build the guardrails early or they didn't they didn't build them until they it was too late and and they needed them it was too late or it's not so much that
37:29 Scott kind of the ai agents force us to have those guardrails we should add those guardrails for the humans right like the the thing that you know just what like a couple weeks ago somebody uh some startup had a ai agent that basically deleted their entire production database right and and it's like hold on a second like that
37:46 Scott Okay, sure,
37:47 Scott we can blame maybe it was cursor or cloud code and maybe we can blame for cell or whatever back end it was, you know, super base or for cell or AWS or whatever.
37:57 Scott But the reality is that human that that agent was doing was working on behalf of that human apparently had the credentials to delete all the production databases.
38:08 Scott Like, what was that all about?
38:10 Brad Yeah,
38:10 Brad like they weren't calling the fundamentals,
38:13 Brad right?
38:13 Brad you know robust access controls you shouldn't have had all that so that would that could have happened in a hundred different ways right you could have had a cron job could delete that database but like i guess the difference is like you know we're getting into a point where agents are taking actions right cron job might take an action but you know we're asking to actually go do things and make decisions right so essentially like you know agent go out and buy me a bunch of food like
38:40 Brad I'd be curious, like once that becomes a commonplace of like how many people are left with this to cart,
38:45 Brad you know, groceries that they didn't order would be a lot of people.
38:48 Scott Yeah. Why do I have so many bananas?
38:53 Scott Well, you said you wanted a bunch of bananas.
38:59 Scott right but the the eager intern could have just as easily deleted the production database and in fact how many of them have
39:06 Brad Or the person that's rushed, right?
39:08 Brad I've had that before.
39:11 Brad I did one of my big mistakes was I migrated the
39:16 Brad sharepoint data the wrong direction um one day when i was overwrote the production system with like month old information because
39:25 Scott Oh,
39:25 Brad i was in a rush
39:25 Scott wow.
39:26 Brad and that that was a that
39:27 Scott That was a bad day.
39:28 Brad was a bad that was a bad week um but like those things can happen right like it doesn't you're just being rushed you're thinking about it um those types of things happen on a regular basis but now
39:41 Brad You know, the ramification is, hey, this AI agent's doing it on behalf,
39:45 Brad right, where they probably have less logic in there,
39:48 Brad right?
39:49 Brad They're not thinking logically, they're just guessing.
39:50 Scott Maybe the higher order bit is that the hype around how fast everybody can move with AI is.
39:59 Ben turning some leaders into saying we need to be moving as fast as the other guy and sure the ai agent is the tool that lets you move fast but maybe the root cause or the or the you know sort of the original sin is are you moving faster than you can reason about
40:19 Brad I disagree a little bit.
40:21 Ben Okay, fair enough.
40:22 Brad I think that the agents should,
40:26 Brad you know, if given the permission,
40:28 Brad right, if it's not in the like, you know, ask me before you take action type mode,
40:31 Brad they can choose to make this their judgment, it comes back what I was saying a minute ago,
40:37 Brad the agent's judgment and taste is not the same as the humans.
40:40 Brad And so they are making decisions where even the intern
40:44 Brad Atture might be like, wait a second, this doesn't seem right.
40:47 Brad And yes,
40:48 Brad databases get deleted,
40:49 Brad data gets copied the wrong way, but there's something about the agents that they actually lack intelligence,
40:55 Brad they lack.
40:57 Brad judgment and just intuitiveness that I think it's creating a slippery slope than what we had before.
41:03 Scott They lack the business context,
41:05 Scott right?
41:06 Scott So it goes back to that piece of understanding what's important to the business or maybe it's what's important to me,
41:12 Scott right?
41:13 Scott They don't have that context.
41:15 Scott And I think some of the tools have seen better context,
41:18 Scott but when we start getting into like enterprise data,
41:21 Scott that's...
41:22 Scott a whacking there.
41:23 Brad One of the first systems I worked on a way, way long time ago,
41:26 Brad when you logged into the production system back in the days when you logged into a production system,
41:31 Brad we got this message and it popped up and it said, hi, this is your friendly system administrator.
41:36 Brad One,
41:37 Brad play nicely with others.
41:39 Brad Two,
41:39 Brad think before you type.
41:41 Brad Right.
41:41 Brad And so it was giving you this warning. Right.
41:43 Brad And so you knew as a person and I was, you know, 23 at the time or whatever.
41:47 Brad And if I was about to type RM,
41:49 Brad I was thinking before I typed. Right.
41:52 Brad But the agent isn't thinking.
41:55 Brad It's just doing.
41:57 Brad Because it saw somewhere on the internet the data that it was trained on said to type RM at this moment,
42:02 Brad or probably to type RM at this moment.
42:06 Brad So it's slippery. It's slippery.
42:09 Brad Now maybe we can train these agents to be skeptical,
42:11 Brad just like we train the interns to be skeptical.
42:14 Ben You know, the system prompt lets you do,
42:17 Ben lets you create some.
42:19 Ben You know, skeptical personalities, if that's the direction that you're trying to go, but but you still I still think that Ben I still think that the point Ben's making or at least the point I'm taking away from what you said, Ben, is like guardrails matter.
42:37 Scott Yeah, they do.
42:38 Ben guardrails matter and i just feel like it's not i'm not happy with the idea i i would be i feel dissatisfied with the idea that and and there's and i can't count the number of times in my career where this has been the fallback right is the well at least we have a human and their taste and their judgment
42:57 Ben will prevent them from making the mistake.
42:59 Ben It's maybe a last line of defense,
43:01 Ben but it's not a very good last line of defense.
43:04 Brad We need
43:05 Scott Titan
43:05 Brad some.
43:05 Scott who controls like that.
43:06 Scott That's not that's not a defensible position.
43:09 Scott I don't think right like that's that's minimal effort on their right Titan who controls prevent that
43:15 Brad Well,
43:16 Brad right.
43:16 Brad And we spent a generation building up technical controls. Now these agents have come along. We basically threw them all out. We're gonna have to rebuild.
43:22 Brad Go them again.
43:23 Ben oh yeah what is it like uh i don't use cloud code as much as i use some of the other tools but cloud code as a literally it's like asks you can i do this can i do this can i do this and and then they have this they basically have a you know an option called dangerously accept any tools and it's like dash dash dangerously accept any tools or something like that it's like
43:47 Ben no i i'm sorry i cannot bring myself to to run claude code that way you know so it's it's this it's i i feel like claude code is a great example where they've got this like
43:58 Ben they've they basically are like teetering on the fence and and falling hard off of either side either you have to give it permission for everything it does or just like yolo
44:10 Scott I am confident that most people yell up in there.
44:15 Scott That's what I think.
44:16 Brad You know,
44:17 Brad I don't know what most people do.
44:19 Brad I actually have been accepting,
44:21 Brad you know, I choose to do it every time. I don't want to call it choose to issue the community's time,
44:27 Brad but I do find myself getting annoyed.
44:29 Brad Like, it'd be better if I could just say do it.
44:32 Brad I thought that was the promise of this thing.
44:34 Scott Well,
44:34 Scott it's done some of those things too on the productivity side of using Cloud a lot in PowerPoint.
44:40 Scott And I live a little bit more dangerously in PowerPoint.
44:44 Scott I am allowed the dangerous execution deletion of information because I'm like, hey, you know what?
44:52 Scott OneDrive has a backup version of this,
44:55 Scott so that's my take on it.
44:57 Scott But other areas, you got to go back and look at it and understand what it is.
45:04 Scott We used to make conscious decisions around all these things and what was coming next.
45:08 Scott Now,
45:09 Scott default's probably okay,
45:11 Scott but maybe it's not in certain instances.
45:15 Scott Maybe it's going outside of the parameters of what you really want it to do.
45:18 Ben Yeah,
45:19 Ben there you go.
45:20 Ben Default's probably okay,
45:22 Ben but maybe it isn't. Maybe that's, so we started with the data swamp and we've ended with the defaults might be okay,
45:31 Ben but maybe it isn't.
45:33 Ben There you go.
45:34 Ben That seems like the sum of it.
45:36 Brad I think the lesson I took from this in addition to it's worth investing to clean up your data is buyer beware. That's what the defaults are probably okay kind of means to me is like.
45:47 Brad You can't just accept all this stuff at face value,
45:50 Brad whether it's the data you're getting out.
45:52 Brad or the way the tools are working you really got to know what's going on yeah
45:56 Scott yeah maybe you have to invest in different areas right if you are going to accept those defaults right maybe you have to go back and have more rigorous accountability and code reviews and that applies to just about anything right anything that using AI it doesn't have to be just code but make sure your quality is there I mean
46:15 Brad output reviews yeah awesome well it was a real pleasure
46:19 Ben chatting with you ben needless to say like this is a deep topic and i think you got both my wheels and scott's wheels spinning a bit on trying to think through this quagmire of the data swamp in the backyard thanks
46:31 Brad Thanks for coming on today,
46:33 Brad Ben.
46:33 Scott Just everyone remember you're not alone in your data swamp.
46:36 Scott You're
46:37 Ben you're
46:37 Scott not alone.
46:37 Ben not alone you're not alone in your data swamp there you go all right well what do we say scott
46:44 Brad Spanish rate review.
46:46 Ben There you go.
46:47 Brad Tell all your friends,
46:48 Brad blast it out of the car radio.
46:49 Brad Everybody wants to hear this podcast.
46:53 Ben How many CTOs podcast on your radio? Awesome.
46:56 Ben Well, it was great talking to you, Ben, and we'll see you next week, Scott.
46:58 Ben This episode of the How Many CTOs Does It Take podcast was produced and edited by Saffron Heptakow. Our theme song was AI Generated by Brad.
47:08 Ben Please follow us on all the social media platforms.
47:11 Ben at how many CTOs pod.
47:13 Ben You can find us on YouTube,
47:15 Ben LinkedIn,
47:16 Ben Instagram,
47:16 Ben Facebook,
47:17 Ben Threads, X,
47:18 Ben TikTok,
47:18 Ben and Patreon,
47:19 Ben or at our website at howmanyctospod.com.
47:23 Ben And remember to rate, review,
47:25 Ben and subscribe on the podcast platform of your choice.
47:28 Ben Thanks for listening.