Transcript: Setting the Tone: Building Trust Through Security
0:00 Scott We don't put it on our balance sheets as businesses,
0:02 Scott but security is an asset or liability on that balance sheet. And all those vulnerabilities that aren't fixed are really, you can convert them to dollars and put them as a liability.
0:33 Scott Welcome everybody to yet another episode of how many CTOs does it take today?
0:40 Scott Uh,
0:41 Scott my name is Scott Poirier. It's always Scott Poirier, and I'm with my trusty co-host.
0:46 Brad Brad Hefta-Gaub, who sometimes is Brad Hefta-Gaub.
0:49 Scott Correct.
0:50 Scott And today, the topic is security.
0:53 Scott Brad sent me a text earlier this week. He said,
0:56 Scott hey,
0:56 Scott let's talk about security on the podcast this week.
0:59 Brad Yeah.
0:59 Scott I said, sure.
1:01 Scott So, Brad,
1:02 Scott what do you want to talk about?
1:04 Brad Well,
1:05 Brad I want to talk about...
1:07 Brad Not just let's not talk about security best practices, although that will certainly come up as things we talk about.
1:12 Brad But I'm just kind of curious about your experience in as a CTO, what have you seen as sort of the various cultures around security and technologies around securities and ways leaders we can make sure we're doing the right thing? And, you know, how.
1:31 Brad What's the role of the CTO in security?
1:34 Scott That is a great question.
1:37 Scott Well,
1:37 Scott I think the first thing that one needs to ask is,
1:41 Scott does the security function like the information security function roll up into the CTO? Well, let's for the purposes of this conversation,
1:48 Scott assume that it does.
1:52 Scott And some companies it's separate because some people see that's a little bit like the
1:57 Scott Fox guarding the hen
1:58 Brad Yeah.
1:58 Scott house, if that makes sense.
2:01 Scott And so I would say the first role of the CTO in security is mindset or attitude or culture,
2:12 Scott the tone at the top.
2:14 Scott What is the tone that the head of engineering is setting?
2:20 Scott And let me, before I go any further,
2:22 Scott let me say this.
2:23 Scott I am not a security expert.
2:25 Scott I don't have any of the security certifications or anything like that.
2:28 Scott But I have worked at three or four different companies where security for different reasons was important,
2:34 Scott either because we had scale where it mattered to be secure.
2:39 Scott We had external partners who wanted some sort of security.
2:41 Scott sort of trust that their data was being handled properly or we were working where you and I work together in health care and we had confidential patient data so I'm not an expert but I've been around it and I've had to lead a security function on a few different occasions
2:59 Scott So, step number one, tone at the top. And I would say tone at the top goes both ways. And what I mean by both ways is toward one's own engineering team setting the tone for the engineering team, but also out and up towards the senior leadership.
3:17 Scott That it has to, as a CTO, it's your job to set that tone and educate and train and guide in both directions.
3:24 Scott because you know I often say most people don't know what we do inside of engineering and they really don't know what we do inside of engineering when it comes to security but
3:34 Brad Yeah.
3:35 Scott but they will know they will know when there's a problem
3:39 Brad Yeah,
3:39 Brad exactly.
3:40 Brad It's funny.
3:41 Brad I just as I realized as I asked you that question and you started answering it,
3:44 Brad I was like, maybe the better maybe the better place for us to start. First of all, love your point about tone.
3:49 Brad 100% agree with that.
3:51 Brad Because I've seen places where, you know, to your point,
3:55 Brad they don't care about it until all of a sudden it matters.
4:00 Brad Right.
4:01 Brad Like if you have a breach,
4:02 Brad everyone's going to care about security.
4:05 Brad But it probably was it's very it's it's it's probably the breach probably occurred because.
4:15 Brad of the tone like it's very likely that we had a lax security posture because we had a unclear tone or a clear tone that was security is not important that we
4:31 Brad led to the choices that ended up resulting in a breach.
4:35 Brad So does
4:35 Scott Yes,
4:36 Brad that ring true to you?
4:38 Scott that rings true.
4:40 Scott Let me tell you, I've been involved in what I would call three real security incidents.
4:49 Scott Okay,
4:49 Scott but one was only one was really a breach. So let me tell you about these three because I won't be revealing anything.
4:55 Scott One of them was when I was working at Rover.
4:57 Scott This was not a breach.
4:59 Scott It was a situation.
5:02 Scott Rover was a two-sided marketplace.
5:04 Scott So you had buyers and sellers who were communicating via messages.
5:09 Scott And we had, I can't remember if it was fake buyers defrauding.
5:14 Scott Sellers are fake sellers defrauding buyers,
5:17 Scott but there was people were getting defrauded through the system. So there was nothing about our technology that was broken,
5:23 Scott but it was to the end user,
5:26 Scott they felt like something was wrong.
5:30 Scott That was one thing.
5:31 Scott The second thing was at another company,
5:34 Scott not Rover,
5:35 Scott this was when I was at DALI, which DALI got acquired by TaskRabbit.
5:38 Scott We had an account credential stuffing attack.
5:42 Scott So this is where some rogue actor has found
5:46 Scott a database of usernames and passwords that's been leaked on the internet and they come and try them on your website and people who have used the same username and password on multiple websites,
5:56 Scott then they're able to get into that account and they're able to do stuff, right?
6:03 Scott Again,
6:03 Scott our system was working. Someone used a legitimate username and password.
6:08 Brad Yeah.
6:09 Scott To access the system. Now, you could say, well,
6:12 Scott we didn't have two-factor authentication,
6:14 Scott which was true,
6:15 Scott and we then thereafter introduced two-factor authentication,
6:18 Scott and that solves that problem.
6:20 Scott The third actually was also at Dolly, and it was a real breach. Now, the breach predated my time there.
6:32 Scott by a couple of years that but the consequence was thereafter so what happened was we received an email from someone one day that said hey we have access to your systems and we want a bunch of bitcoin right
6:45 Brad Oh, no.
6:45 Scott and yeah that's right oh no and we were able to diagnose that
6:55 Scott two key things here.
6:57 Scott They had gotten access to our systems through an insecure password.
7:02 Scott Somebody had a password like password123 and that let them get in.
7:07 Scott And we were also able to diagnose that they,
7:11 Scott whoever this person was,
7:12 Scott had actually not been inside,
7:15 Scott to whatever extent they were able to show us proof that they really had been inside our system.
7:20 Scott But they had not been in our system for at least a year or two based on the proof that they were showing us. They were showing us database schemas and we were able to say, oh, we know when certain tables were added or deleted.
7:35 Scott And so this is the time frame when they were in the vector that they got in.
7:39 Scott That was a server that had this crappy password that had been out of commission for a long time.
7:45 Scott So those are kind of, that was the real incident that I was involved in.
7:49 Scott And I will tell you, you get real heart palpitations until.
7:57 Scott You find out, oh,
7:58 Scott okay,
7:59 Scott they got in a while ago and they're not in right now.
8:02 Scott So at least the door was unlocked, but now it's locked.
8:06 Scott So that was a little scary.
8:08 Brad And by the way, you just mentioned something, you used the I word incident.
8:12 Brad And one of the things that definitely is on my mind about security is one of the things about having an attitude of security and an attitude and a sort of like a.
8:23 Brad culture and systems that help you solve for security is do you have a robust incident response process do you have a documented
8:31 Scott Mm-hmm.
8:31 Brad incident response process do you have you practiced it have you had a chance to you know work those muscles and you know do you have a culture that's good at incident response because certainly any incident that's a security incident
8:47 Brad It's going to be the type of incidents it's going to really lean on your culture of excellence and your high performance team dynamic because it's going to be stressful and there's going to be a lot of opportunity for finger pointing.
9:05 Brad And have you established a great quality of ego-less,
9:11 Brad retrospective incident management type team coordination? Do you have systems in place? Do you have systems in place for incident management in general?
9:23 Scott Yes, now I'm able to describe these events succinctly, but at the time they're happening.
9:29 Scott I don't want to make it seem overly dramatic.
9:30 Scott There's bullets flying in every direction.
9:32 Scott I mean, people are shooting each other. There's just a lot going on. You don't understand the situation.
9:37 Scott You're like, what's happening?
9:38 Scott Who's in our system?
9:39 Scott And then you're talking to one of your people and they're like, wait, this CC2 instance,
9:43 Scott we don't even have this instance anymore.
9:45 Scott Like, you know, you're slowly unpeeling and like trying, you know, getting an understanding.
9:51 Scott I forgot a fourth incident that I've been involved with.
9:55 Scott this was an internal embezzlement yeah
10:00 Scott this was a situation where again
10:03 Brad No one will notice if we just take a penny per transaction.
10:05 Scott kind of it wasn't exactly like that this was again at one of the market I've worked at three marketplaces so this is at one of the marketplaces I was at and somebody
10:15 Scott There was a cadence,
10:17 Scott I don't remember exactly the rhythm,
10:18 Scott but let's just say it was once a week where workers got paid and there was like a cron job that paid these workers.
10:26 Scott And somebody inside the company who was figured out that they could insert rows into this table that did the payouts.
10:38 Scott And so it would pay
10:39 Brad Hmm.
10:39 Scott them.
10:40 Scott But there was no check and balance on that.
10:43 Scott The only way we discovered it was at the end of the year, you have to issue 1099s for all these people. And the person who was doing that was just kind of spot checking and just noticed like unusual payments, like either they were kind of large.
11:03 Scott or they didn't have they would usually be there was a piece of the the idea of the person who you know there's like a a user id and they're like but this isn't a real user id these are fake like what's so what's going on they so if that person hadn't been double checking
11:22 Brad You
11:22 Scott we
11:22 Brad wouldn't do it.
11:22 Scott might have we wouldn't have noticed and then through quite a bit of investigation we were able to figure out it was somebody internally um and we were you know we got variety lawyers involved in that stuff and uh they not to say it made it better but this is what this is what the lawyer told us actually this will be good this will be a fun story to say look
11:47 Scott In my experience, we were able to get ourselves in touch with this person who dealt with this kind of problem.
11:51 Scott They were a former federal prosecutor who is now in private practice and we were just like, what should we do?
11:57 Scott Like, do we call the cops? Like,
11:59 Scott you know, what is, how do you handle this situation?
12:02 Scott And he said, look, in my experience, when somebody is stealing from their company,
12:07 Scott they're having money problems.
12:11 Scott And those money problems are coming from a divorce,
12:14 Scott gambling or drugs.
12:17 Brad Yeah,
12:18 Brad yeah
12:18 Scott OK, that's probably most likely that's what you have going on here.
12:22 Scott And that's what was going on. We had somebody who had gambling losses and was trying to cover it up from his spouse.
12:29 Scott And this is kind of how he.
12:32 Brad Yeah.
12:33 Scott doing that anyway
12:34 Brad So back to security.
12:36 Brad So yeah,
12:37 Scott okay so tone at the top I
12:39 Brad tone at the top, obviously. What else?
12:42 Scott would I'll offer two frameworks to our listeners and I'll say one thing overarching both of these frameworks
12:49 Scott And I say this to people all the time.
12:52 Scott Lots of times people,
12:53 Scott when they go to address these frameworks,
12:56 Scott they're just trying to check boxes.
12:57 Brad Yeah.
12:57 Scott And I try to, this is a tone at the top thing, I always try to say, look, what's actually important?
13:04 Scott I don't care what this framework says.
13:06 Scott I don't care what, you know, the control that we're supposed to put in place is.
13:10 Scott What actually matters is that we're more secure.
13:13 Scott That's what matters.
13:14 Brad Right.
13:14 Scott And in fact, I just had this conversation yesterday with a general counsel at TaskRavit.
13:19 Scott He asked me the question we were talking about some security thing where there's a little bit of risk and he's like, he was trying to understand, Scott, are we just trying to check the box here or is this a real thing to make us more secure?
13:29 Scott I was like,
13:30 Scott no, this is, I want to do this because it will make us more secure. Like I'm not doing this to just check the box.
13:35 Scott If you're doing it to check the box, you might as well.
13:37 Scott not do it okay so what are my two frameworks one is a framework i invented and then another one is a is real world so here what's the what's the one i invented very simple three stages it's like it's like a castle you're the king of your castle brad secure the perimeter make sure nobody can get in is number one number two make sure everybody who does get in you know who they are
14:03 Scott And number three, make sure the people who do get in can only go to the rooms of the castle that you let them go into.
14:08 Brad Yeah, minimum exposure,
14:11 Brad minimum.
14:12 Scott Yeah,
14:12 Scott so number one is,
14:14 Scott you know, have yourself inside some sort of.
14:16 Scott private network and have all the outside of that network locked down real tight number two is you know don't have shared user accounts and stuff like that in your network everyone you know have all that stuff on the surface and number three is role-based access control and if you do those things and part of securing the perimeter stuff like
14:36 Scott you know all the different types of security things you can use ssh keys and two-factor authentication and all that kind of stuff and to me like if you follow that framework you're going to be pretty solid right
14:46 Brad So I love it. That's all great stuff.
14:50 Brad I feel like that's really speaking to,
14:53 Brad we haven't really talked about the code.
14:57 Brad So you're
14:57 Scott Oh, yeah.
14:58 Brad in the zone of access to systems,
15:04 Brad access to information,
15:05 Brad controlling the humans,
15:06 Brad controlling the humans that you know about.
15:09 Brad Right.
15:09 Scott Yeah.
15:11 Brad And I think you're right, like you're you're enumerating like all the best practices that we all hear.
15:16 Brad Right.
15:16 Brad You know, like, you know, actual authentication of the person who they are, who they say they are. And that's like quality passwords and two factor authentication and appropriately secure systems.
15:26 Brad Don't ever write your own cryptography. Right.
15:28 Brad Like and and then, you know, role based accesses, that's like clear.
15:36 Brad Like we really need to know that it's you,
15:38 Brad Brad,
15:38 Brad and we need to know that Brad has access to do these things. Right.
15:42 Brad As opposed to all of dev uses.
15:45 Brad dev and just logged in with root and the root password is password um
15:49 Scott Right. Yeah,
15:50 Scott exactly.
15:52 Brad and of course you know modern systems like all of our cloud providers uh all of them have you know user account management systems that allow us to do really fine grain controls there um and i think that i'll just say like uh
16:07 Brad Because I've worked so often with startups and growth stage companies,
16:12 Brad right?
16:12 Brad Like one of the things that often you find is like in the early days, you're just go,
16:18 Brad go, go and you're taking shortcuts all over the place, right?
16:21 Brad And,
16:22 Brad you know, you and I like we've show up after the fact, right?
16:25 Brad And maybe shortcuts have already been taken. Maybe it's been years of shortcuts have been taken.
16:29 Brad And so you've got a little bit of a.
16:33 Brad figuring out what you what your exposures are can be you kind of come to it late right um and i feel like that mistake of maybe
16:46 Brad i just turned on a bunch of privileges you know maybe i've just i've got these agents that do this work for me you know
16:54 Brad that i've got keys stored all over the place right especially in today's now this is going to pivot into the software conversation right because a lot of times our software is built on privileged components talking to other components and negotiating privilege at the software layer right i kind of threw out a couple things there one we've got this like often we're confronted with this legacy of choices that were made that we may need to come and clean up
17:22 Brad And then I'm also sort of starting to pull at the thread of what about software security?
17:27 Brad So what are your thoughts on those two topics?
17:30 Scott I got thoughts on those two topics. I want to say one thing before I finish a previous thought and then get into this.
17:36 Scott I talked about two frameworks.
17:38 Brad Oh yeah, you said your framework,
17:39 Brad which is the...
17:40 Scott That's my personal little framework,
17:42 Scott and it didn't talk about software.
17:45 Scott I'm going to come back to that.
17:46 Scott The other is these standard frameworks like SOC 2,
17:49 Scott ISO 9000 or whatever.
17:51 Scott And the one that I've found is the best to start with, and by best I mean just easiest to start with,
17:58 Scott is called CIS,
18:00 Scott Center for Information Security.
18:02 Scott already um they have uh what they call uh implementation groups so they kind of give you there's a lot of there's like 200 controls in this thing and they sort of say okay we'll do these 50 first
18:15 Brad Yeah.
18:15 Scott right so it's just a good it's a good starter kit and um the re so that's one of the reasons i like it and i also like it because it's a little more prescriptive it sort of just says do this thing right a lot of the
18:28 Scott frameworks that are out there are more like guidance like you shall have some way to ensure that people log in securely right and so when you're getting this gets to your question about you're coming in a little bit later i tend to start with that ig1 i'm sorry cis and then you've got the implementation group one it just provides a roadmap
18:51 Scott Because you're overwhelmed. You got so much. You walk into a scenario and you're like,
18:55 Scott oh, my God, there's no security here at all.
18:57 Scott What do we even do?
18:59 Scott And it just provides a, you know, I hate to reuse the word framework, but a framework or a roadmap for like,
19:03 Scott OK,
19:04 Scott well,
19:04 Scott we just work our way down this checklist and one by one do all these things.
19:10 Scott And I have a specific philosophy there,
19:11 Scott which is it's better to go shallow and broad.
19:18 Scott and keep on doing that as opposed to you can take one of these controls and you can go really really deep on it right it's it's better it's better to get a simple little lock on every door and then come back and put a better lock on each door and then a better lock on each door as opposed to putting spending all your time to making sure door number one is rock solid but doors two through 20 are wide open
19:38 Brad So for people's benefit,
19:40 Brad I just pulled up the CIS IG1 framework and I'll just like read through.
19:45 Brad through kind of their various controls they have just as an example to your point about how you could go deep on these or you could go broad so like control one is inventory and control of enterprise assets you can imagine how like yeah do i have
19:59 Scott inventory of all my control of my enterprise assets.
20:03 Scott Inventory and control of software assets.
20:06 Scott Data protection.
20:08 Scott That's control three.
20:09 Scott Control four,
20:10 Scott secure configuration of enterprise assets and software.
20:14 Scott Control five,
20:15 Scott account management.
20:16 Scott Right, do I have accounts?
20:19 Scott Control six,
20:19 Scott account control management,
20:21 Scott access control management.
20:22 Scott That's the roles-based control,
20:23 Scott right? Control seven,
20:25 Scott continuous vulnerability management.
20:28 Scott control eight audit log management control nine email and web browser protections and there's 18 of these I'm going to stop here at nine but malware data recovery network infrastructure network monitoring security awareness
20:44 Scott service provider management application software security incident response management and pen testing so yeah like you can imagine that all of those you can go each of those each of those you could go super deep on but you're going to way more benefit if you do if you do something for everyone
21:01 Brad And those are actually called control families,
21:04 Brad and within each of those that they have.
21:07 Brad many controls underneath there i think the total
21:09 Scott safeguards maybe underneath or something like that
21:11 Brad i think they do call them safeguards yeah uh i believe there are 100 about 150 total controls or safeguards in that framework in those 18 families and then they've broken those up into those three layers implementation group or ig123 yeah and so my strategy is to go shallow and get a little everything just like painting the golden gate bridge you paint one coat
21:32 Brad and then just go back over and paint another coat you keep on painting coats endlessly because by the time you get to the one end of the golden gate bridge the other end is already faded you need to start over
21:40 Scott Yeah.
21:41 Brad okay so when we talked about we wanted to get into software yeah so two thoughts about this i first went to humans and it's because i was influenced once
21:52 Brad The first job I had where I had to do security,
21:55 Brad we hired someone,
21:56 Brad and this was a guy who came out of some bigger companies.
21:59 Brad I think he'd been at Walmart and some other places, and my inclination as an engineer was to start with the software.
22:06 Brad And he was telling me, you've got to start with user access,
22:09 Brad and I didn't understand why.
22:11 Brad And he's like,
22:11 Brad well,
22:11 Brad look,
22:12 Brad I've been doing this a while,
22:13 Brad and the reality is most of your problems.
22:17 Brad come because a human made an error Yeah, right
22:19 Scott right
22:20 Brad and you know it's not as frequent that someone finds some clever way to exploit the software that you wrote it's much more the social engineering that people do to get someone's password out of them or that someone used a weak password and then when they used a weak password and somebody got in you didn't have the roles and set set the right way so
22:43 Brad I learned that from him and then that sort of changed my mind. My mindset was always like,
22:48 Brad oh, protect against SQL injection attacks and stuff like that. That's where I sort of started out in this journey.
22:53 Brad And then I learned like,
22:54 Brad oh, it turns out most of the problems are human.
22:57 Brad And then those four incidents that I told you
22:59 Scott All
23:00 Brad about,
23:00 Scott human.
23:00 Brad all human, right?
23:02 Scott Yeah.
23:03 Brad And so, and in fact, I've been at some companies where we have done, we have done.
23:09 Brad You know, we phished our own employees. You know, that's when you do something to try and get an employee to give you their password.
23:15 Brad We phished our own employees as part of our training.
23:19 Brad And one place I had 26% of the employees gave up their password.
23:23 Scott Yeah.
23:24 Brad Right.
23:25 Brad I mean, this is, you know, this is crazy.
23:28 Brad And this was the classic, you know.
23:32 Brad hey we need you to go take this survey or something click this link and the person clicks the link and we set this whole thing it was a page that looked like you had to log into google but you know the url was like google spelled wrong or something like
23:45 Scott Right.
23:45 Brad that and then they typed in their username and password to log into google and boom you know we have it so um it again comes back to that tone at the top and setting the culture okay but then you come back to the software
23:58 Brad Well, let's talk about the software stuff.
24:01 Brad My experience, and this is not again,
24:04 Brad not my biggest area of expertise. And by the way, it is all changing with AI.
24:09 Brad I mean, it's like a whole new game.
24:12 Brad The things I lean on are one is coming back to that perimeter is your network setup.
24:19 Brad securely basically if you're running a web app or you just locked down to port 80 right do you have all these other port 443 but like is that 443 is that your sso port i can't remember other people remember that stuff these days better than i do it uh do you just have a lockdown remember i told you about that incident we had where there was this one ec2 instance with a lousy password that was at a company that was not using a network and all of their instances were on
24:47 Brad the public internet
24:47 Scott I
24:48 Brad you
24:48 Scott see.
24:48 Scott They didn't have any kind of an application load balance or any kind of a...
24:53 Brad nothing protecting it so they had this one container that someone had set up probably to experiment with something and then that person set a weak password and then it was out there and someone found it who knows how they found it so um that's one piece and then the other piece is uh standards-based like you know
25:13 Brad I've been at companies that have had their own homegrown authentication systems and it's just like what are you doing these days man
25:20 Scott Yeah.
25:20 Brad like we should not be you unless you're
25:25 Brad off gov or off zero like you're not getting paid to have good security to make your own security system so like you know rely on a third party for this kind of stuff and whether it's all the stuff that matters which is databases and money and users like that's all stuff where you can find whether it's you know
25:45 Brad open source or commercial systems to use where you got a community of people to help you there.
25:51 Scott So I love that point, and I think that's a great segue actually into one of the other topics that we've become a little bit more aware of, and that is supply chain attacks.
26:04 Brad Yeah.
26:04 Scott Right.
26:04 Scott And in fact, we just had one recently,
26:07 Scott React had a supply chain attack with Next.js and React.
26:13 Scott So it's like pretty big, common,
26:15 Scott large,
26:16 Scott large open source products that end up with a, you know, significant vulnerability in a supply chain that a lot of people.
26:27 Scott You know, so there are best practices for making sure you're ready for that kind of a thing,
26:32 Scott right?
26:32 Scott Like software inventory,
26:34 Scott you know, like do you really know what software is being used to build your software?
26:38 Scott With NPM packages these days, it's just like so easy to be like,
26:42 Scott oh, I need a new, like what was the thing that was like, there was one that was not even an attack,
26:47 Scott but there was like a famous one like a couple years ago where this guy had this open source package that was for.
26:52 Scott for colors and everyone
26:54 Brad Yep.
26:54 Scott like used that color package and then one day it was just like he took it down it's like no one's software could build anymore because you know he got rid of the package
27:07 Brad Well, you talk about NPM,
27:09 Brad you know, you type in NPM, I can't remember the words for NPM at the moment,
27:13 Brad but like NPM install color package and you watch it sit there and install 100 dependencies and you're just thinking to yourself, what is this stuff?
27:21 Scott Right.
27:22 Brad So, okay,
27:23 Brad I said there were two things on the software side,
27:25 Brad but I fibbed and I actually have a third thing on the software side.
27:27 Scott Yeah.
27:29 Brad the system you know npm audit in the npm space will tell you uh where your packages are at in terms of security vulnerability stuff like that make that a part of your build and if you've got a critical or if you've got a high don't let it deploy you know set in a set an sla for your team um it says look any high vulnerability this is how i like to set these slas i like to
27:55 Brad I have, I'm going to go deep here for a second, when I say deep, a little bit of a tangent,
27:59 Brad whether it's a bug or a vulnerability or a performance issue,
28:06 Brad any of these kinds of things.
28:08 Brad Here's how I think about it. You've got critical,
28:10 Brad high,
28:11 Brad and medium luck.
28:12 Brad Critical is, actually you have one on top of that.
28:14 Brad You've got the top one, which is above critical. Let's call it red alert.
28:17 Brad Red alert is stop, drop,
28:20 Brad and roll.
28:20 Brad Stop what you are doing now and work on this thing until it is fixed day or night.
28:25 Brad Okay.
28:26 Brad Critical is, hey,
28:28 Brad break your current sprint to work on this.
28:30 Brad You had some stuff in your sprint,
28:32 Brad pull this thing into your sprint,
28:34 Brad and disrupt your sprint to make it happen.
28:37 Brad High is...
28:39 Brad this needs to go in the next sprint.
28:43 Brad Medium is this needs to go in the next sprint or the sprint thereafter.
28:47 Brad And low is you need to do it within the next 90 days.
28:51 Brad And when it comes to low,
28:53 Brad if you don't do it in 90 days, we're just going to close this ticket.
28:57 Brad We've decided that if it's not important for us to do it in 90 days, we're just going to get rid of this ticket.
29:01 Brad So those are priorities.
29:03 Brad You can use those priorities for bugs.
29:06 Brad for performance for vulnerabilities.
29:08 Brad So step two there is for each of those categories.
29:12 Brad you decide on criteria now vulnerabilities is the easiest because something like NPM audit is telling you
29:19 Scott Yeah.
29:19 Brad and you can just map it right oh we got a high high high severity vulnerability that's going to be put into our bug system our ticketing system JIRA be marked as a high priority and that'll just go in the next sprint and then it's really easy to build a report in JIRA which shows you things that are out of SLA
29:38 Brad And as a manager,
29:40 Brad you can just manage against that.
29:41 Scott By the way, I love what you're saying here.
29:44 Scott I want to pull on a thread, though, that I think you're ending up hinting at,
29:50 Scott which is.
29:51 Scott And you mentioned this at the beginning where you said, hey,
29:55 Scott you're not just setting the tone to the engineers and the teams below you, but you're also setting the tone up and out to your peers and to the other leadership team.
30:05 Scott And I think that that is hinting at a really common thing where.
30:12 Scott How do you change the philosophy or how do you help manage your stakeholders into thinking of security as more than just a cost center?
30:23 Scott Because certainly when you say,
30:24 Scott hey, I've got these tickets and you just use some great language like critical, we stop what we're doing to it breaks our sprint.
30:32 Scott We work on it.
30:33 Scott Hi,
30:33 Scott we got to get it into this next sprint.
30:36 Scott Right.
30:36 Scott Like you're making statements that the average product owner,
30:39 Scott if they don't have a security mindset, don't believe in security mindset,
30:42 Scott they're going to they might bristle at the idea that you're saying these things trump their feature ideas.
30:49 Brad They do bristle. And let me tell you, they bristle in two ways.
30:56 Brad I mentioned that you can apply this also to bugs and to...
31:03 Scott performance
31:04 Brad So, yes,
31:05 Brad product managers can really bristle because when you start to say,
31:08 Brad hey, any endpoint that's slower than, let's say,
31:11 Brad 500 milliseconds,
31:12 Brad that's going to be a high priority bug. And anything that's slower than a thousand milliseconds is high priority and anything that's slower than 500 is medium priority,
31:20 Brad then they're going to say that's filling up their sprint. And when you say,
31:23 Brad hey, anything that's, you know, we're going to set some standards for what's a high priority bug and a medium priority.
31:28 Brad And they say, well, gosh, if we set those standards,
31:31 Brad my sprints are going to get filled up with my sprints. Of course, the sprints are going to get filled up with all these bugs.
31:37 Brad And do you know what I'm going to say?
31:38 Brad I'm going to say, well, then we need to do better QA, don't we?
31:42 Brad Because we're letting all these things out.
31:43 Brad But coming back to the security point,
31:46 Brad I would say I try to impress upon them a few things.
31:52 Brad One is that.
31:54 Brad We don't want to have a breach.
31:56 Brad A breach is incredibly expensive.
32:00 Brad I've done some cost analysis on some of these breaches I talked to you about before.
32:04 Brad And when you start to get the lawyers involved,
32:07 Brad and I've never been in one of those cases where we've had to disclose to our customers,
32:11 Brad but you've received a letter in the mail just like I have,
32:14 Brad which says so and so company was breached. I mean, these are hundreds of thousands, if not millions of dollars in expense.
32:20 Scott They've
32:21 Brad And
32:21 Scott gotten hundreds so of
32:21 Brad that's
32:21 Scott millions of
32:22 Brad real.
32:22 Scott dollars.
32:23 Brad Right.
32:23 Brad That's real.
32:26 Brad The other thing I try to impress upon them is we have to have a culture of maintenance because when we get behind on these things,
32:34 Brad it's not a hole we can really ever dig out of.
32:37 Scott Yep.
32:38 Brad So, I mean, OK,
32:40 Brad do you want to blow off this high?
32:42 Brad severity vulnerability a sprint okay blow it off a sprint but if you blow them off six months then you're going to have 20 of them and it's going to take us six sprints to get it figured out right so we have to just always stay up with it it's just a cost of doing business and the other thing i try to impress upon people is we don't put it on our balance sheets as businesses
33:03 Brad But security is an asset or a liability on that balance sheet.
33:08 Brad And all those vulnerabilities that aren't fixed are really, you can convert them to dollars and put them as a liability on.
33:16 Scott Yeah, they're almost like what Jon Helm talked about.
33:19 Scott They're like.
33:20 Scott Naked calls,
33:21 Scott right?
33:22 Brad Yeah.
33:22 Brad Let's assume if one of those high severity vulnerabilities is exploited, it's worth half a million dollars.
33:30 Brad Let's assume that one out of a hundred of those is going to happen.
33:35 Brad So that means that.
33:38 Brad It's $50,000. That's a $50,000 liability on average on your sheet, on your balance sheet.
33:43 Brad And so we can decide as a business if we want to carry those risks.
33:48 Brad And lots and lots of business people will say yes.
33:52 Brad But the first time they go through a real incident,
33:55 Brad then they will regret it.
33:56 Brad So I try to do that.
33:59 Brad And then the other piece, and this is the last thing I'll say is.
34:03 Brad is when you set it up is just create that culture of oh it's a high and we put it in the next sprint then it just becomes a non-topic any longer it's a little rough at first to get people to buy into it but once you get them to buy into it then nobody thinks about it anymore it's just a
34:16 Brad cut course matter doing
34:17 Scott Yeah,
34:17 Brad things.
34:18 Scott I think that's fair.
34:19 Scott I also think and this I kind of opened the door a little bit to the question of like, what do you do when you kind of arrive and it's and you've got like a legacy of problems,
34:27 Scott but I'll just take a moment here. If you have if you are lucky enough to be starting from scratch.
34:35 Scott Then this is something that you can build into your processes to be completely automated.
34:39 Scott Like you mentioned,
34:40 Scott you know, you just make a CICD rule that like requires that there are no vulnerabilities.
34:44 Scott And, you know, the current state of the art in static analysis is really quite good.
34:50 Scott And so, you know, you have all these different things. It's like, you know, DependBot can tell you if you've got vulnerabilities in your dependencies,
34:57 Scott you can run vulnerability scans.
34:59 Scott You can be running static analysis against your code base to see that you're accidentally doing stuff like being vulnerable to SQL injection or being vulnerable to API injection of various sorts. It's like.
35:15 Scott The static analysis tools are pretty damn good for that stuff right now.
35:18 Scott And so set it up as part of your build process to just not let you check in code that has the problem.
35:24 Brad Exactly right.
35:25 Brad The build process,
35:27 Brad and I'm a big fan of something called danger system,
35:29 Brad danger system, something like that,
35:31 Brad which is a way to build rules into your build process and your deploy process.
35:36 Brad It is really where you can put your guardrails, right?
35:41 Brad Whether it's everything from like don't deploy a critical kind of thing or no PR is bigger than 300 lines or no PRs after 430,
35:50 Brad no pushes after 430 on Friday.
35:53 Brad You know, you can put all those rules in your build system.
35:56 Brad And when you build a culture that does that,
35:59 Brad then it's basically a straight line from incident retrospective to let's put a rule in the build to prevent that from happening.
36:06 Brad I'll put it again.
36:07 Scott Yeah, totally.
36:08 Scott Now, you also mentioned the notion of checking boxes and things like that. And I'll just make a quick, I'm not going to give a sales pitch for this product, but I will tell you, I just recently started working with a product called Vanta.
36:21 Scott And Vanta is essentially an automated system for compliance regimes.
36:26 Scott And so it actually,
36:27 Scott so I'm using it with one of my clients to do SOC 2 compliance.
36:33 Scott And it's so far so good. It's a pretty powerful system.
36:37 Scott So you you actually hook it up to your GitHub and you hook it up to your JIRA ticketing system and you can hook it up to your various cloud providers. And it does analysis for you to see if you're meeting all the controls.
36:50 Brad Mm-hmm.
36:50 Scott And I think that and so,
36:51 Scott for example, a good example of this is, you know, it will analyze your controls around your logins and your credentials.
36:57 Scott Do you have do you have appropriate does everybody who has.
37:00 Scott has an account in github are they a mapped person on your accounts in your you know your google workspace are they are is there a correlation there and and do you have you know checks there these are examples of the types of controls that you know would be in your cis you know ig1 but
37:23 Scott A tool like Vanta,
37:25 Scott and there's probably others out there that do the same thing. I actually, in fact, know there are others out there that do the same thing.
37:29 Scott They actually automate that control checking,
37:31 Scott and I think that makes it possible for, makes it
37:35 Brad Yeah,
37:35 Scott it a lot easier.
37:38 Brad let me offer two other tools that are out there.
37:40 Brad Three other tools that are out there, actually.
37:43 Brad We use a tool called Wiz,
37:44 Brad which does a lot of that.
37:46 Brad We use a tool called Drata,
37:48 Brad which is a competitor of Aventa, and then built into AWS for any of you who are using AWS,
37:53 Brad and I would assume that Azure and Google Cloud Platform have the same thing.
37:58 Brad Within AWS, it's called AWS Security Hub.
38:03 Brad You just go in there and you say,
38:04 Brad run a check, use this framework to run a check for me, and they've got dozens of, they've got, you know, SOC 2 and HIPAA and you name it, they've got them all in there,
38:14 Brad and it'll just run through whatever AWS setup you have and tell you how you're doing against that.
38:20 Brad Oh, here you've got this bucket exposed publicly.
38:23 Brad Oh, here you've got this container that's not part of the VPCs, etc. And so...
38:28 Brad It gives you then that's your kind of your roadmap for getting started on what to fix.
38:34 Brad Let me offer one other framing and I'm going to say this is what I'm about to say is a little bit in evolution in my mind.
38:43 Brad I used to frame it like this. In most of our companies, you have to think about we actually have two technology environments.
38:50 Brad We have what we call our production environment,
38:53 Brad which is often an AWS or Azure or Google Cloud or something like that,
38:57 Brad which is like our web service that we.
38:59 Brad serve people with and then we have our corporate environment which is like our work in our laptops our workstations our google workspace or whatever you use all those kinds of things and so you have to cover both and i tend to when i look at these control frameworks i tend to apply it separately to both of them because it's easy to say okay we'll do user access control oh sweet we got aws sort it out i was like okay well that's only half
39:27 Brad That's it. You got this stuff over here.
39:29 Brad Where I've started to evolve in my mind of that framing a little bit lately is I actually think there's a third environment.
39:35 Brad It's a portion which bridges the two and it's the data environment.
39:40 Brad Right. Is that we often have data coming from our production environment or data coming from our corporate environment that is going into, you know, some sort of data warehouse kind of thing, which is a third space. And sometimes you own that third space and it lives like in your cloud place,
39:57 Scott Yeah
39:57 Brad your cloud place. That's funny.
39:59 Scott your cloud environment or sometimes there's a third party like a snowflake or something like that but it's kind of this third piece that you have to think about that stuff as well CIS does have a control family about data but again I tend to think about the control families or the rows of the different controls and then you have three columns your corporate your production your data and the only reason I do that is to help people think
40:25 Scott think more clearly about all the areas we have to
40:28 Brad Or
40:28 Scott cover
40:28 Brad the gaps.
40:28 Scott so that we're all yes we're covering all the ground and that's kind of the three main the three main areas that tend to cover it all we've
40:39 Brad Totally.
40:39 Scott gone on a huge tangent from software i don't know how we got from software to talking about
40:44 Scott a control environment matrix.
40:47 Brad Here's how I tie those things together in my mind is that typically as CTOs, right,
40:55 Brad you're typically working for an organization that is building software.
41:00 Brad And I think that you're right that it's easy to get hyper focused on that part of it.
41:07 Brad And to leave out, you know, like your your colleague said,
41:11 Brad you know, the human factor,
41:12 Brad right,
41:13 Brad or to leave out the corporate factor or to leave out the, you know, like a, you know, like a good example of this would be like, oh.
41:22 Brad You know, if I have let's say I'm doing a good job of not checking in secrets into my code base. Yay me.
41:30 Brad But my developers have, you know, secrets on their development machines that, you know, are used for their development environment.
41:40 Brad Right. Well,
41:41 Brad now that means that that that machine and that individual corporate user is an attack vector.
41:47 Brad Right.
41:48 Scott Yeah,
41:48 Brad And so you could say, oh,
41:50 Brad I'm doing such a great job of doing the software part securely.
41:53 Brad But then if you've missed your if you miss coverage on the attack vector of humans that work for your company,
41:58 Brad then you're not really doesn't matter that you did all that work to secure the software.
42:03 Scott that's right.
42:04 Scott You know, we were talking about the software side and I didn't think of it, but I should have now that I hear what you said.
42:12 Scott One of the most basic things on the software side is a security, a secrets manager.
42:16 Brad Yeah.
42:16 Scott You're not using, you know, one password and AWS secrets manager or whatever you want to use. We use a system called Doppler at our company.
42:28 Scott Like you're just missing one of the low hanging fruits for keeping things more secure.
42:34 Scott And there's a lot of different choices out there you can use.
42:38 Scott One of the things I did run into,
42:40 Scott I was at a company where we had a lot of smart engineers,
42:45 Scott and they often wanted to use the best tool for the job.
42:49 Scott And so there was some secret manager they were using.
42:52 Scott When they started to put in a secrets manager,
42:55 Scott they started to put in this really good one, right?
42:58 Scott But it didn't have the ease of use and developer tooling around it.
43:03 Scott So then it just took developers a long time.
43:07 Scott And so they
43:07 Brad So I know what happened.
43:08 Scott just skipped it.
43:09 Brad They skipped it and they put the secrets right
43:10 Scott Yeah.
43:10 Brad in the code.
43:12 Scott Yeah.
43:12 Scott So this is one of the cases where I think you really do have to think about what is your engineering or your developer workflow,
43:21 Scott because you want it to be such that it needs to be.
43:28 Scott The cost-benefit equation,
43:29 Scott it has to be easier to use the secret manager than to not,
43:34 Scott because if it isn't, people won't do You it.
43:36 Brad know, it's so funny you say that. I literally just heard a podcast with a company that's basically working on using AI to basically do dynamic.
43:48 Brad pen testing against all these different you know surface areas that you're talking about and this is you know the founder of that company it's very much like a longtime security expert and was making that was telling a story about a time that he was working consulting with a CISO that basically said oh I want my secret manager to be really really hard to use and he was like what why would you do that and he's like
44:14 Brad Well,
44:14 Brad because I would like to try to encourage the developers to simply not use secrets, to come up with solutions that don't require secrets.
44:22 Brad And he was like, that's not going to work,
44:24 Brad buddy.
44:25 Brad Like, it's not going to work.
44:27 Brad Like your developers are going to, I mean, you can't avoid secrets like all of the tooling that we use nowadays,
44:34 Brad all the API packages that we have to use, they're always going to have secrets involved in them. So to your point, if you.
44:39 Brad And I think,
44:40 Brad by the way, I think we could rewind this to the very top first comment that you made,
44:45 Brad which is it's all about the tone that you set,
44:47 Brad right?
44:47 Brad And so if you set a tone that says,
44:50 Brad like, look.
44:51 Brad security matters and of course you're going to have to have software that talks to software and therefore you're going to have to have credentials for software and that means you're going to have to have secrets and that matters and we're not going to not do that we're going to do that right and that matters and then and then you have developers who will at least voice the concern like hey I'm trying to use the secrets manager and it's really hard to use can we find a better secrets manager that has better
45:16 Brad dev tooling as opposed to like fuck it i'm just going to go around the policy agreed
45:22 Scott Yeah.
45:23 Scott Here's the last thought I'll share.
45:24 Scott I mean, maybe we'll have more to discuss, but it seems like for me, this is the last thought that I have.
45:30 Scott If you follow good security practices,
45:32 Scott you follow the CIS framework or the SOC2 framework or whatever,
45:35 Scott you do these things,
45:36 Scott you're just following good software engineering practices.
45:40 Scott And if you follow them.
45:43 Scott It is work to set them up. That matrix I was just describing of your control families going down the rows in your environments.
45:51 Scott So let's say you've got 18 control families and you've got three environments.
45:54 Scott You've got 56.
45:56 Scott Did I get that right?
45:56 Scott Whatever,
45:57 Scott you've got a lot.
45:58 Scott That's like 56.
45:59 Scott It isn't 56.
46:00 Scott Three times 18.
46:01 Scott I don't know what it is, but whatever it is,
46:03 Scott let's just say it's 56 for the purposes of my illustration.
46:08 Scott That's 56 projects you have to do.
46:10 Scott And if there's three implementation groups,
46:15 Scott that's actually 160.
46:19 Scott big something projects
46:20 Brad Big
46:20 Scott you
46:20 Brad number.
46:21 Scott got to do it's a big so it's a lot of work it's a lot of work to set it up but when you do get it going it actually greases the rails of your software engineer um it just makes things go more smoothly because a lot of getting it set up is having um you know procedures and processes for how something get done and so you're just streamlining the work that you're doing
46:45 Scott And so I see a lot of people see security in opposition to software engineering. I see security as, you know, accelerant or catalyst to software engineering.
47:01 Scott And actually, I said that was less than, but I didn't say the last last thing.
47:06 Scott This is going to be maybe the nerdiest thing I ever say on this podcast in
47:11 Brad I'm I'm I'm
47:12 Scott all of our episodes.
47:14 Brad on edge
47:15 Scott It's almost spiritual.
47:18 Scott Information security and specifically information security frameworks is the grease that makes capitalism go.
47:27 Scott It's what allows individuals and businesses to trust each other.
47:32 Scott And if we didn't trust each other,
47:34 Scott we would have a harder time transacting with each other.
47:37 Scott And, you know, I trust that this dollar I'm going to give you is worth a dollar,
47:41 Scott right?
47:41 Scott I trust that this data I'm going to give you, you're going to keep secure.
47:45 Scott And if all that stuff didn't happen, if all those trust frameworks weren't there.
47:52 Scott You would only do business with people who you could see and shake their hand and think of how different our world would be. And so in a certain way,
48:03 Scott I have almost like it's to me, it's almost like.
48:07 Scott I'm a little bit in awe of it, whether it's our security frameworks we're talking about,
48:11 Scott or you got the, what was that one having to do with public companies that came out after Enron?
48:16 Brad Yeah,
48:17 Brad Sorbain's
48:18 Scott Sox, Sarbanes-Oxley.
48:19 Brad -Oxley.
48:20 Scott Yep. Or you got HIPAA with your health information.
48:23 Scott You got PCI with your credit card information, all this stuff.
48:26 Scott All these are the things that allow all of the actors,
48:29 Scott meaning all the people in businesses in the economy,
48:31 Scott to trust each other.
48:33 Scott And trust is what makes the whole thing go.
48:37 Brad I think the way to sum this up, actually,
48:39 Brad Scott,
48:41 Brad rewinding to the whole, we basically like summarize the whole podcast as trust is fundamental to being able to be in business.
48:55 Brad And the tone that we need to set as leaders.
48:59 Brad is that establishing good engineering practices will establish good security practices,
49:07 Brad which therefore allow us to implement the trust and to demonstrate the trust that makes us an effective business.
49:14 Scott Yeah.
49:16 Brad And I don't think that you could say it any better than that.
49:21 Scott No.
49:22 Scott You said it well.
49:23 Brad There you go.
49:25 Brad Well,
49:26 Brad this was fun. I think we covered a lot of ground.
49:28 Brad I think we talked about some important things.
49:30 Brad Obviously, there's way more that we could talk about it. Like,
49:32 Brad I think that like this is right there in the in the mix of like, what do you do when you show up and there's been, you know, a fair amount of technical debt or security debt,
49:40 Brad and then you kind of try to clean it up and you touched on many of those topics.
49:46 Brad But I love the idea of if you can start from if you get to start with a clean.
49:50 Brad clean slate then for sure put security in place right away just make it part of your just make it your culture make it part of your engineering practices and your life's going to be so much easier yeah
50:02 Scott agreed and brad i'm going to go a little bit out on a limb here i don't think you'll object to what i'm about to say but for any of the 40 people who are listening to this because our listenership increased by one person this week brad
50:15 Brad there we go nice
50:18 Scott I would be happy to, and I think you would be happy to also take a call with anybody if you're struggling with this at your company and need some guidance.
50:25 Brad Yeah.
50:26 Scott I'd be happy to help out in part because I think good security serves us all as an industry.
50:34 Scott You know, if all of our websites and all of our mobile apps and all of our software systems were flaky, nobody would trust this industry and we'd all be out of jobs and so on and so forth.
50:43 Scott And so I think it's good for us as an industry.
50:47 Brad And I think that's a great way to sum it up. But gosh darn it,
50:49 Brad you made me think of one last thing. And I'm going to say this and then we're going to sum this up.
50:53 Brad Just because a breach happens,
50:55 Brad that's not the end of the story,
50:57 Brad right?
50:58 Brad Like we said earlier,
50:59 Brad like having high quality teams that understand how to respond to incidents,
51:04 Brad that's what that's that's like such an important part of this.
51:09 Brad And so, you know, a part of that tone is like security matters. And also we are professionals such that when something goes wrong, we deal with it in a professional manner.
51:20 Scott Absolutely.
51:21 Scott Now, there's a whole topic we didn't discuss,
51:23 Scott and there's another podcast from this.
51:26 Scott And we actually should find the right guest for it because I'm not the right person.
51:31 Scott AI is changing everything.
51:33 Brad AI is changing everything.
51:34 Scott You've got data getting exposed in ways that we never had it exposed before.
51:38 Scott You've got agents crawling around your environment.
51:42 Scott I was just talking to someone this week.
51:44 Scott He was telling me how he basically gave.
51:50 Scott some sort of AI agent root access to it was a problem in their environment he basically gave this agent root access and said can you help me figure this out and it figured it out and he was like it was amazing and so scary at the same time right
52:03 Brad Totally.
52:03 Scott and so there's this whole new world out there that to be honest I don't even have a concept of how to cope with but there are going to be agents that
52:15 Scott Do the wrong like I've heard of them already agents deleting databases right and all this stuff and It comes back to that second plank of actually I've known it comes back to but the whole that's a whole other podcast We're gonna have to talk about
52:27 Brad All right.
52:27 Brad We've beaten this one. This one's a good one.
52:30 Brad We covered this one.
52:32 Brad Let's wrap it up with.
52:35 Brad Tone is what matters.
52:37 Brad Trust is what matters. And if you set the right tone for security and trust,
52:41 Brad it's just the same as good engineering practices.
52:46 Brad All right.
52:47 Brad Well,
52:48 Brad what do we say?
52:48 Scott smash that subscribe button tell all of your friends and leave us a great review say this is these are the best the best podcast ever
53:00 Scott Better than Joe Rogan or Mark Maron or any of those podcasters out there.
53:04 Scott And also you can tell all your friends that you knew about it first.
53:09 Brad You knew about it first.
53:10 Brad That's right.
53:10 Brad All right. Thanks,
53:11 Brad Scott.
53:12 Brad It was great talking to you. We'll talk to you next week.
53:14 Scott See you, Brad.
53:17 Brad This episode of How Many CTOs Does It Take Podcast was edited by Safran Epbegaard.
53:22 Brad Our theme song was AI Generated by Brad.
53:25 Brad Please follow us on all the social media platforms at at HowManyCTOsPod. You can find us on YouTube,
53:32 Brad LinkedIn,
53:33 Brad Instagram,
53:33 Brad Facebook,
53:34 Brad Threads, X,
53:35 Brad TikTok,
53:35 Brad and Patreon,
53:36 Brad or at our website at HowManyCTOsPod.com.
53:41 Brad And remember to rate, review,
53:43 Brad and subscribe on the podcast platform of your choice.
53:46 Brad How many CTOs Thank you for listening.