Transcript: Tackling Tech Culture and Interviewing: Insights from Ted Neward
0:00 Brad I fully maintain that most companies do not interview at all well.
0:22 Scott Welcome everybody to another edition of the How Many CTOs Does It Take podcast.
0:27 Scott Today we have as a special guest,
0:29 Scott Ted Newart.
0:30 Scott Ted,
0:31 Brad Yeah.
0:31 Scott welcome.
0:32 Brad Thank you.
0:34 Scott Can you please take a moment,
0:35 Scott introduce yourself to our...
0:38 Scott 22 listeners that listen to this podcast.
0:40 Brad Well,
0:41 Brad you know, hi,
0:43 Brad my name is Ted Neward. I currently work as a senior distinguished engineer over Capital One.
0:49 Brad In prior incarnations, I was the director of technology culture over at Rocket Mortgage. I was the director of the developer relations team and a simultaneously director of
1:03 Brad of a development team over at Smartsheet.
1:06 Brad And prior to that, I was the CTO of a startup consulting company here in Seattle.
1:12 Brad And prior to that, I did a bunch of stuff, spoke at some conferences, wrote some books,
1:18 Brad the usual, you know, shenanigans that people get up to doing when they don't have anything better to do with their life.
1:27 Brad I spent a lot of time
1:30 Brad technologically in both the .NET and the Java worlds and I've probably played around with more programming languages than most people know exist including many esoteric programming languages that you would never wish on your worst enemy
1:48 Scott PL1.
1:50 Brad That's not technically an esoteric language. That's just really old such that it feels esoteric. No, I'm talking about the esoteric languages like Rockstar.
2:01 Brad It's an actual programming language so that now you can truthfully claim to be a Rockstar developer on your resume.
2:07 Brad It was created, it was born of a tweet by Dylan Beattie,
2:12 Brad who is a very popular speaker over in Europe,
2:14 Brad and somebody said, you know, that sounds really cool.
2:17 Brad I'm going to put together an actual implementation of the specification,
2:20 Brad and now it's a real thing, and so inspired by the creation of the language,
2:27 Brad Dylan is also now the headliner for a rock band that does parody songs.
2:34 Brad I think one of their biggest chart topping hits was You Give Rest a Bad Name,
2:41 Brad sung to the Bon Jovi song that sounds somewhat similar to that.
2:45 Scott I love it.
2:46 Brad I've spent a lot of time doing a lot of different things, consulting,
2:50 Brad CTO,
2:52 Brad engineering management and general,
2:56 Brad you know, making an annoyance of myself anywhere that I'm.
3:00 Brad not immediately kicked out.
3:02 Scott Well,
3:03 Scott welcome to the pod,
3:05 Scott Ted.
3:06 Brad Thank
3:06 Scott It's
3:06 Brad you.
3:07 Scott great to have you.
3:08 Scott I can't wait to hear some of these war stories.
3:11 Scott And it's interesting, too,
3:13 Scott because you just kind of mentioned most recently one of the things that you were responsible for, I believe,
3:19 Scott at Rocket Money was sort of like the culture of the development team,
3:22 Scott Yeah,
3:22 Scott right?
3:23 Scott I want to hear about that.
3:24 Scott Yeah.
3:25 Brad Yeah, so tech culture was an interesting experiment.
3:28 Brad The CIO, basically this was just before COVID hit and the CIO had kind of this grand vision that Rocket Mortgage needed to transition from being a mortgage company to becoming a technology company.
3:46 Brad And that involves doing a lot of things that really aren't technical in nature,
3:51 Brad but had everything to do with, as the name implies,
3:54 Brad the culture of the engineering group and that rocket mortgage.
3:58 Brad And so he asked me to put together a group that would focus on that for some length of time. We were thinking probably two years.
4:06 Brad Turned out to be less simply because other parts of the company said,
4:11 Brad you know, that actually belongs.
4:13 Brad lungs over here rather than over there in engineering so we focus for example on job descriptions which on the one hand you think that sounds really boring and pedestrian
4:23 Brad But, you know, what we discovered, we went through Workday, which is the bane of my existence.
4:30 Brad I hate it. It's absolutely worse.
4:32 Brad It's the worst tool ever, except for all of the other HR tools out there.
4:37 Brad So I don't know what that's saying.
4:39 Brad But we went through and we looked at all of the different job descriptions that were in the engineering sphere at Rock Mortgage. And it turned out that if you wanted to be an associate scientist,
4:50 Brad software engineer at Rocket Mortgage. You needed to have two to three years of experience,
4:54 Brad which considering this was an entry-level position kind of begged the question,
4:58 Brad how are you ever supposed to get two to three years experience,
5:01 Brad you know, unless we're just saying we don't want people who are brand new right out of college.
5:06 Brad So that was an interesting discovery. The even more interesting discovery was that if you wanted to be a vice president of technology at this company.
5:14 Brad The only thing that was required was a four-year degree.
5:19 Brad that was
5:19 Scott no
5:20 Brad it no
5:20 Scott experience necessary
5:21 Brad experience
5:21 Scott no
5:21 Brad necessary whatsoever you didn't have to have any technical skills you didn't have to have anything you just needed a four-year degree so clearly there was some major calibration issues and frankly you know it may sound like I'm throwing shade at Rocket Mortgage I'm really not because they at least have a
5:36 Scott this
5:36 Brad gut
5:37 Scott this
5:37 Brad who
5:37 Scott is this
5:37 Brad like yeah
5:38 Scott is the story i've i've heard it at so many places i've
5:41 Brad yeah
5:41 Scott been or yeah
5:42 Brad and
5:42 Scott totally
5:42 Brad so we formed a group we called it tech
5:45 Brad tech talent and that group literally brought sort of an agile development methodology to working with the folks over in HR to clean up all of the different job descriptions inside of Workday.
6:00 Brad As another part of it,
6:01 Brad we focused on interviewing,
6:03 Brad right?
6:04 Brad You know, how do we calibrate interviews across the company?
6:07 Brad And I fully maintain that interviewing at most, even the FANGs, most companies do not interview at all well.
6:17 Brad Despite the fact that, you know, we've studied how they do interviews,
6:21 Brad I mean, 20 years ago it was, you know, how do you move Mount Fuji?
6:25 Brad Why are manhole covers round? It was all about these really interesting questions designed to show us how you think.
6:32 Brad When in truth, what they were really doing was, you know, for a certain definition of the term,
6:39 Brad interview torture.
6:40 Brad It's literally,
6:41 Brad can I throw something at you that completely floors you that you can't, you know, reason your way out of in any way, shape or form?
6:50 Brad And what does make for effective interviews and how do we normalize this?
6:57 Brad Having an interviewer having a VP say,
7:00 Brad well, I know what I, you know, when I see the right candidate,
7:04 Brad I'll know it sort of that Potter Stewart quote,
7:06 Brad right? I can't define it for you, but I'll know it when I see it.
7:10 Brad That quote, by the way, has done more damage to Western civilization than anything else in our modern history.
7:16 Brad Because, great,
7:18 Brad if that VP is going to be the only one who does interviews for the entire organization,
7:23 Brad you know, for a startup of 30 people, that's reasonable.
7:26 Brad For a technology organization of 2,500 people inside of a company of 35,000,
7:32 Brad that is completely untenable.
7:35 Brad But now
7:36 Scott Yeah
7:36 Brad it becomes what exactly, how exactly do we scale that and keep it consistent,
7:42 Brad right? These are the kinds of problems that tech culture was wrestling with.
7:47 Scott Yeah, this is so fascinating.
7:49 Scott I am being,
7:51 Scott I don't want to say triggered, but activated by your stories here.
7:57 Scott I have a couple questions for
7:58 Brad Yeah.
7:59 Scott you.
8:00 Scott I have to, I'm curious,
8:01 Scott I'm assuming you've seen the, let's start with the leveling conversation,
8:06 Scott because that's, I love this conversation.
8:08 Scott Have you seen the Dropbox public source?
8:12 Scott open source version of their leveling guidelines yeah
8:15 Brad I've seen several.
8:16 Brad I know Dropbox had one at one point. I think Honeycomb posted one at one point was another.
8:24 Brad There were a couple of companies, particularly the smaller the company,
8:27 Brad the more willing they were to share what they had in terms of their leveling guidelines.
8:33 Brad And a lot of this was part of the whole push to be more transparent about salaries.
8:39 Brad values as well.
8:39 Brad So I've seen several of them and truthfully they all start to blur.
8:45 Brad You know, the fact that, to be very blunt,
8:50 Brad I don't really care what your leveling guidelines look like.
8:53 Brad I care that they exist.
8:55 Brad I care that they're written down and that they're targetable and they're actionable.
9:00 Brad Right.
9:00 Scott Yeah.
9:00 Brad Because now it actually makes it much easier for myself as an engineering manager,
9:07 Brad for somebody else as somebody who works for me, to have a better sense of expectations and I can go and compare that against, you know, the leveling guidelines,
9:16 Brad the job descriptions in lack of a leveling guideline to say, well, I've done these things on this list.
9:23 Brad And I've done them at this particular level. So therefore,
9:26 Brad I should be, you know, I should be set to receive a midpoint grain or a above average grade or, you know, whatever,
9:36 Brad however you want to do that a five point scale or a or a terminology scale.
9:40 Brad But it gives me the opportunity to be able to look and see what I'm doing on a day to day basis to make sure that it lines up with that.
9:47 Brad Right.
9:47 Brad And if I don't know.
9:50 Brad Then what happens is I fall back to what my boss tells me to do.
9:54 Brad And then if we change bosses, I'm screwed.
9:57 Brad I mean, the new boss says,
9:59 Brad well,
9:59 Brad that's not what I would have asked you to do.
10:01 Brad But that's not the point.
10:03 Brad This is what my boss at the time asked me to do. And you start getting into a very much, you know, the corporate version of a he said,
10:09 Brad she said kind of scenario.
10:11 Brad And frankly,
10:12 Brad nobody wins in that scenario.
10:14 Brad So the fact that you've published something right there just already puts you head and shoulders above the rest of the crowd.
10:21 Scott How do you think about this question?
10:24 Scott Sometimes these leveling documents are written very like, I call it practical or performance based and sometimes they're very much competency based and sometimes they should be very much values based.
10:38 Scott You know, like an engineer,
10:41 Scott a frontline engineer working on a team, you know, has features and projects coming down the pipeline,
10:46 Scott etc.
10:47 Scott often isn't necessarily in control of the.
10:51 Scott what they're working on as opposed to you know doing it and so I've just seen these leveling guides written differently like sometimes I've seen it's like it's a set of competencies and so if you are doing all these things okay but it doesn't actually say that you ever delivered any software that seems like a key part of the job here is delivering software sometimes it's very much delivery based sometimes it's very much value based I've never kind of thought what's your take on the mix of those things
11:17 Brad So to me, the biggest thing is that there's sort of twofold there.
11:22 Brad One is, are they in any way, shape or form measurable,
11:28 Brad right?
11:28 Scott Mm-hmm.
11:29 Brad And I'll take as an example, frequently when you see these job descriptions or these performance expectations,
11:35 Brad you'll see the line, write good code.
11:40 Brad Okay, great.
11:42 Brad What is good code?
11:43 Scott Well, it's not evil.
11:45 Brad Pick your favorite programming language,
11:47 Brad right?
11:47 Brad And by the way, if my code is used on the death stars that make it good code or bad code,
11:51 Scott Exactly.
11:52 Brad you know, good enough to pass Emperor Palpatine's quality check so I didn't get force choked.
11:57 Brad But on the other hand, it's used to destroy planets.
11:59 Brad So good code or bad code? Seriously,
12:02 Brad what does good code look like?
12:03 Brad And you ask this in a room of a dozen engineers and what you will quickly find is they will segregate into three or four camps.
12:11 Brad camps, each of which bitterly hates the other.
12:15 Brad Right.
12:15 Brad Because, you know, on the one hand, good code takes full advantage of all of the capabilities of a programming, you know, of the programming language available to you.
12:24 Brad And so now you start seeing,
12:26 Brad you know, like the ternary operator instead of using a traditional if else.
12:32 Brad And there are a lot of people who are like,
12:33 Brad oh,
12:33 Brad my God, no,
12:34 Brad if I see the ternary, that's just too subtle, too hard for people to reason about.
12:40 Brad Okay,
12:41 Brad both of these are reasonable positions,
12:43 Brad but which defines good code?
12:47 Brad And what we find is, this is why I really hate that Potter Stewart quote,
12:51 Brad because for most people, good code is when I look at it and it speaks to me personally.
12:59 Brad But unless you can articulate it, unless you can articulate what makes good code over bad code.
13:05 Brad then you can't use that as part of any sort of decision-centric criteria because we can't decide,
13:15 Brad even the three of us on this call,
13:18 Brad if we can't decide what good code is,
13:19 Brad how is anybody else supposed to decide what good code is so that they can decide to write good code?
13:25 Scott How do you solve for that?
13:26 Brad Well, in that particular case,
13:28 Brad if you really want to have something...
13:30 Brad describing the code then bring it to something that's more concrete right so for example writes code that adheres to our coding standard as documented here
13:43 Brad Right.
13:43 Brad And so that coding standard might say things like always prefer if else unless is an expression is immediately,
13:53 Brad you know, the result of this, you know, if the else is going to yield a particular expression,
13:57 Brad then you can use a ternary operator.
13:58 Brad however you want to phrase that right um you know prefers not to use chained method calls beyond two or three levels deep which by the way blows away almost all of your builder style fluent interfaces or whatever that is right if it's written down if there is a coding standard out there right and i don't mean chooses smart variable names that's not really what i'm thinking about but i'm
14:25 Brad What I'm thinking about is something that somebody can look at code and decide whether or not this adheres to that code standard or gets close to that code standard.
14:34 Scott Well, and hopefully with most of those things you just mentioned, we've got linters running that are going to not let us check in code that doesn't match that coding standard.
14:43 Brad That's useful for when I have full control over the build process,
14:47 Brad less useful for when we're talking about doing interview styles.
14:50 Brad Right.
14:50 Brad But on top of that, there's a lot of things that a good Linter won't be able to pick up on.
14:55 Brad And truthfully, it's actually quite surprising how few places of business actually use those static analysis tools.
15:03 Scott Yeah.
15:04 Brad There's just, I mean, I just run across all these different places where the idea of a static tool that would go in and, you know, evaluate the source code as opposed to we're going to do pull requests,
15:17 Brad we're going to do code reviews,
15:19 Brad we're going to do a human centric process.
15:20 Brad Whatever reason, the industry still feels, it feels to me like our industry is still very uncomfortable with that kind of automated judgment,
15:32 Brad if you will, about my code.
15:35 Scott Yeah, that's so interesting.
15:37 Scott I know that one of the things that, you know, I'm currently my primary gig is with a pretty small team. You know, there's less than a dozen of us.
15:46 Scott And early days, it was like blank hard drive and we were starting from scratch.
15:51 Scott And I kind of talked about this in our kind of intro episode where early on when there was only a couple of us, we put in place a bunch of these.
16:00 Scott kind of best practices like having linters running having you know having just a light a little bit of process you know even doing like sprints with retros even though there's only three of us you know and it is interesting I I kind of like feel like
16:19 Scott I've got so many scars from when it was a 500 engineering,
16:24 Scott a 500 engineer size team and people were arguing about, you know, brace placement on code reviews. And it's like that, like that is a waste of everybody's time.
16:34 Brad I
16:35 Scott Let's
16:35 Brad agree with
16:35 Scott just
16:35 Brad you and
16:35 Scott agree.
16:36 Scott Let's agree on what we're going to code to.
16:38 Scott Let's put static linters in place and it doesn't, you can't check it in if it doesn't pass lint, you know.
16:44 Brad I'll even go so far as to say
16:46 Brad Say, look,
16:46 Brad there is a canonical representation when it sits inside of source control,
16:50 Brad but when you check the code out and you start working with it.
16:54 Brad Feel free to reformat it according to your local preferences because particularly for code formatting, you know, all of these IDEs and so forth are so good at parsing it and being able to say,
17:06 Brad oh, you want the braces here,
17:08 Brad you want them on the start of the next line,
17:10 Brad you actually want them like over on column 80 so it looks like you're programming in Python,
17:16 Brad whatever you want to do.
17:17 Brad I don't care.
17:17 Brad We can reformat it however you want it.
17:20 Brad The thing that I think, you know, and you actually, you said a trigger phrase for me, which is the word best practice.
17:28 Scott Fair enough.
17:28 Brad There are no such things. I don't know if you knew this. There is no such thing as a best practice.
17:33 Scott My practice.
17:34 Brad Well,
17:35 Brad okay,
17:35 Brad that you can you can free to say my practice and kind of like my dad used to say, so long as you live under my roof, you will cut your hair the way I want it.
17:45 Brad wanted to be cut because I was living under his roof when it's under your roof you can call it your practice all day long but that doesn't necessarily make it a best practice because
17:56 Brad In some respects,
17:58 Brad what works for a team of a dozen people may not work for a team of 500 or for an organization of 500. And there you actually might want to allow each of the teams a certain amount of localized control in terms of how they want to do it.
18:13 Brad Maybe all of the ANSI K&R developers are on that team over there and they all want their braces on the next line.
18:20 Brad And frankly, at the end of the day, I really don't care if I'm not writing code on that team.
18:24 Brad What I care about is that the team is able to.
18:27 Brad jump in look at source code and ingest it you know wholesale rather than having to parse through piece by piece because that's one of the things psychologically that we see that's different between a junior and a senior developer the senior can just like literally the brain pattern matches the whole thing you slurp the whole thing in at once and you understand what's going on the junior has to kind of read it line by line to be able to track what's going on
18:53 Brad And frankly,
18:54 Brad the code being formatted a particular way, if it makes it easier for the senior to slurp it in, great,
18:59 Brad reformat it however you want it.
19:01 Scott Yeah.
19:01 Scott So we kind of have ended up on a tributary of the river,
19:05 Scott which is we were talking about culture and you were talking about changing a culture and you started and you caught my attention at least by starting with the leveling guide and the interviewing,
19:19 Scott which my interpretation.
19:20 Scott The hesitation after thinking about that was,
19:22 Scott okay,
19:22 Scott so get the right people in the room and the culture presumably starts to take care of itself.
19:29 Brad The problem is getting the right people in the room is hard at scale.
19:35 Brad Right.
19:35 Scott Mm
19:36 Brad Because
19:36 Scott -hmm.
19:36 Brad each of us as an interviewer will bring our own biases and our own perspectives into the conversation.
19:45 Brad And we have to be very,
19:47 Brad very careful that as an organization, whether or not somebody gets hired is more a factor of which of the three of us they interviewed with rather than their own skill set.
19:59 Brad Right.
20:00 Brad Because that's actually something that we have seen happen at a number of organizations.
20:05 Brad If you've got like all these different interview pods,
20:08 Brad again,
20:08 Brad I'm talking about a large scale organization,
20:10 Brad 2,500 engineering,
20:12 Brad 35,000 total.
20:15 Brad You're not going to have five people that do all the interviews for all of engineering.
20:19 Brad That's just not feasible,
20:21 Brad you know, and they're not going to do it for in addition to the thirty five thousand of rocket mortgage.
20:27 Brad They're not going to do the interviews for rocket homes,
20:30 Brad rocket money,
20:30 Brad rocket auto,
20:31 Brad rock central.
20:33 Brad So we have to have a way of somehow structurally making sure that these pods are all coming to within some acceptable degree of difference,
20:45 Brad right?
20:46 Brad Because otherwise,
20:47 Brad if this pod is really,
20:49 Brad really, really liberal about who it is they hire and this other pod is really, really strict about what they hire.
20:57 Brad We could have the same person interview twice at the company,
21:00 Brad once get a yes,
21:01 Brad the other get a no,
21:02 Brad and all of a sudden we don't really know what we're hiring anymore.
21:05 Brad We really don't know the skill level of the people that we hire.
21:09 Brad So this is partly where it's not just about get the right people in the room.
21:13 Brad We also have to give them some structural guidance about what
21:18 Scott Yeah
21:18 Brad it means to be an associate level engineer versus a senior level engineer versus an architect versus an engineering manager.
21:27 Brad We have to give them those guidelines and we have to hold them accountable to those guidelines.
21:31 Brad Because, again,
21:32 Scott Hold the interviewer as well as the interviewee.
21:35 Brad exactly.
21:36 Brad One of the things I've always wanted to do at a company and I've never found a company that was brave enough to do it, which is unit test our interview process.
21:46 Brad Right?
21:46 Scott What would that look like?
21:47 Brad Well,
21:48 Brad I mean, just as with any sort of unit testing scenario,
21:51 Brad right, you take known quantities and put it through your code to see if it comes up with the right answer.
21:56 Brad So take a known quantity, that is to take somebody whose skill set you are comfortable with and understand and put them through the interview process and see what result the interviewers come back with.
22:09 Brad Now, the ideal scenario to do this is where the people doing the interviewing don't know that this is a test, right?
22:17 Brad We have to kind of mock it out a little bit.
22:19 Scott Double-blind it somehow.
22:20 Brad Yeah,
22:21 Brad exactly.
22:22 Brad And so this is where, you know, you take some of your colleagues,
22:26 Brad people that, you know, you or I know from the local community,
22:29 Brad somebody who is currently a,
22:31 Brad you know, you grab somebody at Microsoft, for example, since we're up here, I live in Redmond.
22:35 Brad Redmond grab a buddy at Microsoft and say okay you are a senior software engineer Microsoft I'm gonna you know I'm gonna buy you dinner in exchange for going through an interview cycle over here at my company
22:46 Brad And let's see where my interview process pegs them out,
22:50 Brad because I've watched companies interview somebody and say, you know, oh,
22:56 Brad we don't think this person is a senior software engineer.
22:58 Brad And then you follow them on LinkedIn and discover that they just got hired at Microsoft as a senior software engineer.
23:05 Brad And it's like, wait,
23:06 Brad do we think somehow that our notion of a senior software engineer is actually somebody that's stronger?
23:13 Brad Then somebody at Microsoft,
23:15 Brad I mean, how exactly does this relate?
23:18 Brad And without that kind of sanity checking,
23:21 Brad right?
23:21 Brad I mean, we spend so much time trying to get feedback on so many other elements of engineering,
23:27 Brad yet we trust our gut and our intuition when it comes to interviewing because trusting our gut and our intuition, that was always a great way to write software too,
23:38 Brad right?
23:39 Scott Right.
23:41 Scott Ted,
23:41 Scott so it's interesting.
23:43 Scott One simple question.
23:44 Scott Did you actually ever unit test this? Did you try this?
23:48 Brad I have gotten so close to actually
23:52 Scott So
23:52 Brad pulling it off.
23:55 Brad We did do, we did do a little bit of this. I mean, the HR people had a fit when I told them what I wanted to do for a variety of reasons,
24:03 Brad some of which may be legit.
24:05 Brad I just don't know many of which I think were imagined,
24:09 Brad right?
24:10 Brad You know, one was,
24:11 Brad well, you know, how are we going to compensate these people for their time, blah,
24:15 Brad blah, blah. It's like, look.
24:16 Brad Let me worry about that, right?
24:17 Brad Many of these people that I would bring in would be doing it primarily just on a lark. You
24:22 Scott Yeah
24:22 Brad know, it'd be just kind of fun to see what would happen,
24:25 Brad right?
24:26 Brad To do that sustainably, obviously, you'd have to figure out a little bit more sustainable model.
24:31 Brad But we did actually do,
24:35 Brad we started looking at a tool designed to try to help us do some of this interviewing stuff. One of the
24:43 Brad It claimed it was AI-based.
24:45 Brad I don't know how much AI was actually in it, but that it would help with some of the interviewing process.
24:50 Brad And we actually solicited some volunteers from inside the company to sanity check,
24:55 Brad right?
24:56 Brad It wasn't necessarily a full unit test.
24:59 Brad But at least it was a, oh, yeah,
25:01 Brad you know, my role is currently this, and I did these things inside of your interview process, and,
25:07 Brad you know, it felt about,
25:10 Brad you know, it felt similar to what I was doing.
25:12 Brad If we took an associate and put them through the senior,
25:15 Brad the associate was just lost, they struggled,
25:18 Brad which is exactly the right result.
25:20 Brad They should struggle if they're interviewing for a senior position.
25:23 Brad So we did a little bit of that.
25:25 Brad We didn't we didn't do it nearly as comprehensively as I would have liked
25:29 Scott Another thing that this comes to mind for me or kind of brings up for me is and Scott might remember some of this and be somewhat triggered by this as well. He and I spent a short stint at a company that will be remain nameless and they had a very,
25:45 Scott very structured interview process that was focused on the company's core values.
25:52 Brad Mm hmm.
25:53 Scott And they used, you know, star technique and they had they used probably it was greenhouse or something to like that where they had scorecards and they had to produce, you know, the interview. Everybody was trained before they could become an interview all about soliciting data and using that data in their scoring rubric against all these different things.
26:16 Scott What's interesting to me is.
26:19 Scott In my sense, they.
26:21 Scott They spent a lot of time and energy. We spent a lot of time and energy on training people to the tool,
26:27 Scott the interview tooling process.
26:30 Scott And at least when I arrived and Scott was already there and one of the first conversations we had when we sort of like what I'm like,
26:39 Scott hey,
26:39 Scott I'm new here.
26:40 Scott What do I got to work with? And Scott kind of let me in on some of the challenges that they were facing.
26:45 Scott And one of the interesting side effects was that.
26:48 Scott was that they had leaned into the culture question and
26:52 Brad Yeah
26:53 Scott one of the goals of the culture that they had developed was very much to absolutely filter out any toxic programmerness.
27:04 Scott Right. Which
27:05 Brad Yep.
27:05 Scott is a pretty reasonable thing to try to filter out.
27:08 Brad Yep.
27:09 Scott But they were then kind of realizing that one of the things that they had inadvertently filtered out in this process was courage or sort of like willingness to sort of try to take a risk in solving a problem.
27:27 Brad Yep.
27:27 Scott Right.
27:27 Brad Yep.
27:28 Scott And it was so interesting to me. That was one of the lessons I took away from that was that, you know, it's like,
27:36 Scott I don't think anybody intentionally gamed the system, but anything that you can measure can be gamed.
27:42 Brad Yep.
27:42 Scott And this is sort of like a corollary of that where we weren't, no one was trying to game it, but the framework resulted in.
27:52 Scott Filtering out what we wanted to filter out and had a side effect of filtering out something we definitively did not want to filter
27:58 Brad This
27:58 Scott out.
27:59 Brad is where we start getting into more of the systems thinking,
28:02 Brad right?
28:02 Brad The realization that the levers and the knobs and so forth that we can slide and turn,
28:10 Brad they don't always correspond linearly to the results that we get,
28:16 Brad right?
28:17 Brad And frankly,
28:18 Brad you know, you're going to make mistakes like that.
28:22 Brad I mean, when you start, you know, when you start plucking around inside a system and you start trying to create some of these structural things,
28:29 Brad you're going to absolutely run into some of that. And kudos to them for actually doing the diagnostic work to figure out that they were filtering out this quality that they did like as they were trying to filter out qualities that they don't.
28:44 Brad And frankly,
28:45 Brad there's, you know, as with most things, it's not one or zero, it's on a continuum.
28:53 Brad Right. Having the courage to speak up in a meeting and say, I think we are headed down a bad path.
29:00 Brad Well,
29:01 Brad that same courage is what enables me to say, never trust anybody over the age of 35.
29:07 Brad Right. There's a question of degree here.
29:10 Brad We don't want to filter out courage.
29:13 Brad We want you to have just the right amount of courage.
29:16 Brad And frankly,
29:17 Brad that is a very, very hard thing to calibrate.
29:20 Brad In humans,
29:21 Brad until psychologists are able to map personalities to a point-based scale,
29:29 Brad And which will be never,
29:30 Brad by the way, it's certainly not in our lifetimes.
29:34 Brad We're going to struggle with this, but we see the same thing has happened anytime companies have,
29:39 Brad for example, changed CEOs,
29:41 Brad right? Look at the stark difference that occurred when Microsoft got rid of Steve Ballmer and brought in Satya Nadella. The company transformed itself in a two year span of time from nobody wants to be here because it's fiercely cutthroat competitive.
29:55 Brad to everybody wants to be here because they are really focused on churning out some cool stuff and that's that's just from the structural change of having a new ceo some of this is just going to happen right so it's it's it's almost heisenberg in nature right the fact that you're observing it means that intrinsically you're influencing the outcome i don't think we can avoid that and let me be really clear right all those rubrics and
30:22 Brad And all of that structural stuff,
30:24 Brad all of that is there to combat the unconscious bias that each of us brings into an interview.
30:29 Brad Let's be very, very painfully transparent about it.
30:33 Brad I am most comfortable when I am looking across the table at somebody who looks like me,
30:38 Scott That's human nature.
30:40 Brad right?
30:40 Brad For any definition of I,
30:42 Brad right,
30:43 Brad whether it's Brad or Scott or Ted or my wife Charlotte or any of my.
30:47 Brad any of my kids or any of my friends,
30:49 Brad they are always more comfortable looking across the table at somebody just like them.
30:55 Brad Again,
30:56 Brad like you said,
30:57 Brad it's human nature.
30:58 Brad So how do we try to combat that so that we are,
31:03 Brad in fact, trying to find people who are qualified and could do the job, not just somebody that, yeah, sort of can do it and besides which we both like the Seahawks so they can't be a bad person.
31:15 Scott That's sort of the origin of the sample coding project.
31:20 Scott Which I personally found a lot of success with that in my career, but it seems to have fallen out of favor.
31:26 Scott I wonder what your take is on that.
31:28 Brad So there's a couple of things to realize about a sample coding project. And before I go down this path,
31:33 Brad let me be very clear.
31:34 Brad I like it as well.
31:35 Brad Personally,
31:36 Brad I think the best way to the best way to test a band at your wedding is to ask them to play the funky chicken.
31:45 Brad Right.
31:45 Brad Because you know you're going to ask them to do that at your wedding.
31:47 Brad Give them something reasonable to do and see how they do it.
31:51 Brad That seems to me to be by far and away the best way to test anybody for anything.
31:57 Brad That said,
31:58 Brad the coding project,
32:00 Brad a lot of this will start to get into some of the implementation details,
32:03 Brad right?
32:04 Brad So first of all, if I say, let me take you to the whiteboard.
32:07 Brad And let me ask you to scoped out a breadth first tree search algorithm using nothing more than the whiteboard and the whiteboard marker.
32:17 Brad Those of us who went to university and studied computer science and did so recently will intrinsically have a better chance of passing that than those of us who never went to university or never went to computer science.
32:34 Brad My personally,
32:35 Brad my degree was international relations. You want to talk about when nations go to war,
32:40 Brad you want to talk about the difference between macro and microeconomics and its influence on foreign policy.
32:44 Brad See, I'm your chicken.
32:45 Brad I'll go.
32:46 Brad Let's go all day long.
32:48 Brad But I didn't study all of that classic big O notation and whatnot.
32:52 Brad And every time I got into a conversation about some of the formalized aspects of computer science,
32:57 Brad I was at a disadvantage.
33:00 Brad Now, the other question is, is what I'm going to be doing on a daily basis actually reflective in the code sample?
33:09 Brad Right.
33:09 Brad If you're asking me to balance a tree or if you're asking me to search or sort a tree,
33:14 Brad is that actually what I'm doing on the job or am I moving a button three pixels to the left?
33:20 Brad And frankly,
33:21 Brad if it's more the latter,
33:22 Brad then it's not the fact that you have a code sample that's the problem. It's the fact that your code is sampling the wrong thing.
33:29 Brad You're not asking the band to play the Funky Chicken,
33:32 Brad you're asking them to play Beethoven's Fifth.
33:34 Brad Which for a wedding really just doesn't make a whole lot of sense,
33:37 Brad right?
33:38 Scott Right.
33:39 Scott We put a lot of energy into getting that coding project to mimic as best as we could the kinds of things that you would be working on in if you came to work at our company because we were doing a consumer Internet website to really break it down.
33:55 Scott you're basically building forms I mean
33:57 Brad Forms
33:57 Scott that's you
33:58 Brad over data.
33:58 Scott weren't really just building for but you were doing crud right you were getting data in and out and some other things related to that so we tried to create something which just asked the question can this person actually do the thing that we kind of have to do a lot and
34:16 Scott Our recruiters really didn't like it because it slowed the process down and people would drop out and things like that.
34:22 Brad Well, I was just going to say recruiters don't like it because they want to get as many candidates as possible in front of the hiring manager, because in some cases.
34:31 Brad You know, again,
34:32 Brad going back to maximizing what you are measured on,
34:37 Brad right,
34:37 Brad Campbell's law and all that,
34:38 Brad recruiters are often compensated based on the number of people that they can put in front of a hiring manager.
34:45 Brad So their goal is to maximize throughput,
34:47 Brad you know, through the process.
34:49 Brad If they were actually compensated for the minimum number of candidates required before we got a successful one,
34:56 Brad they would actually be more interested.
34:58 Brad more interested in
34:58 Scott Oh,
34:58 Brad filtering out
34:58 Scott yeah.
34:59 Brad more upfront.
34:59 Brad This is where, again,
35:01 Brad just playing around with the numbers just a little bit can really change how people's behavior carries forward.
35:08 Brad And frankly,
35:09 Brad I think you did the right thing in terms of you structured your interview process because...
35:16 Brad And it's absolutely a signal to you as to how well this person will perform on the job. It's also a signal to the person taking this that this is what this is going to be like, right?
35:29 Brad You can absolutely lean in to say,
35:31 Brad your job here will be doing lots of things like this and in this kind of environment. So, you know.
35:39 Brad This is the job, right?
35:41 Brad It works as a two-way street there.
35:44 Scott By the way, Ted and Scott,
35:45 Scott this kind of dovetails nicely into sort of a little mini controversial question,
35:49 Scott which is, you know, nowadays with the rise of AI coding tools and agents,
35:56 Scott I personally ask the question,
36:00 Scott well, how do you use AI in your day-to-day work?
36:03 Scott And the answer to that question seems pretty important to me.
36:09 Scott You may guess where I land on what I want to hear from that person or what I don't want to hear,
36:15 Scott but what do you think about that?
36:16 Brad So as a question, I think it's reasonable to ask,
36:19 Brad right?
36:19 Brad Now,
36:20 Brad the part that concerns me a little bit is you know what you want to hear.
36:25 Brad Which now leads me to, okay,
36:27 Brad so this is basically a checkbox, right?
36:30 Brad You know, whichever way you want to go. Do you use AI?
36:33 Brad Yes, no, right?
36:34 Brad And it doesn't, those kinds of things often don't allow for subtlety, right?
36:41 Scott Well, actually,
36:42 Scott Ted,
36:42 Scott what I wanted to hear is let's have a conversation about.
36:47 Scott Do you or don't you and how did you make that choice?
36:50 Brad So then I think when you phrase it that way, you're putting the question in a different light, right?
36:57 Brad We sometimes get a little bit over clever with our questions because we think that people can understand where we're coming from,
37:03 Brad right?
37:04 Brad It goes back to the why are manhole covers round,
37:06 Brad right?
37:07 Brad And people at Microsoft were asking that question because they wanted people to kind of reason through it.
37:14 Brad But because that particular nuance wasn't always conveyed as a part of the question itself,
37:20 Brad there were a lot of people at Microsoft that were interviewing people saying,
37:24 Brad why are manhole covers round?
37:25 Brad And the person would say they're round because,
37:28 Brad well, that makes it easier for them to roll down the street in order to be able to get them into place.
37:34 Brad And the interviewer would be like,
37:37 Brad OK,
37:37 Brad well,
37:38 Brad I don't really know the right answer here says that they won't.
37:40 Brad they won't fall in
37:41 Scott Ha!
37:41 Brad but i actually had a conversation with a road crew not too long ago who said no absolutely they could still fall in depending upon you know if you like wedged it one guy said oh yeah i've had a man called fall in on me before yeah i don't know how i don't really care
37:58 Brad The bigger thing is, if you want to ask a question about, do I think AI is important?
38:04 Brad Do I use it as a part of my daily life?
38:06 Brad Just ask that.
38:07 Scott Hmm.
38:08 Brad Let's get away from the era of trick questions where we're trying to elicit a particular,
38:13 Brad you know, I want to see if they'll be able to follow my way of thinking around here.
38:18 Brad You know, personally,
38:19 Brad I don't use any AI tools.
38:22 Brad I don't, partly because I don't code on a daily basis as it stands,
38:25 Brad partly because I don't think we've fully figured out what our relationship to these AI tools is or should be,
38:33 Brad because I'm hearing all kinds of stories of people saying,
38:36 Brad oh, yeah, I had AI write this whole thing for me,
38:39 Brad and now I'm finding that there are all kinds of things buried inside this code that, you know, there are landmines waiting to go off.
38:47 Brad and I didn't know it was there so you know I think it's very reasonable if you say do you use AI there are certain people who are listening to this podcast who thought aha Brad wants them to say yes or other people listening to this podcast said aha Brad wants them to say no and other people still were saying aha
39:08 Brad bro doesn't know what the right answer is and so he's trying to get some free consulting out of it you know
39:14 Scott We all know that the real right answer is it depends.
39:17 Scott Of course, that's the right answer.
39:21 Brad spoken like a true consultant scott and a man after my own heart yeah yeah this is what i mean when you ask an interview question right the candidates are already under so much stress and they're under so much pressure
39:35 Brad don't be around the bush just ask them your damn question what do you think about AI you know I believe AI
39:42 Scott Yeah.
39:42 Brad is transformative but we have no idea what the actual transformation will look like I'm curious to hear your thoughts right and now on the rubric and this is one way to make sure that your question is fairly approachable is to be able to write out a rubric
39:59 Brad In some cases, it's not whether or not somebody follows the rubric real closely. It's the fact that you were able to articulate what you're looking for, right?
40:07 Brad The act of actually writing the rubric is itself the real win there.
40:11 Brad Candidate has some experience with AI and or has done some reading.
40:16 Brad That's a thumbs up answer.
40:18 Brad Candidate blindly plugs everything into chat GPT, that's a thumbs down.
40:24 Brad That in of itself,
40:26 Brad what are you looking for out of that question can help feedback into how do I phrase the question so that the candidate has a very reasonable chance to answer it fairly.
40:37 Scott Yeah,
40:37 Scott I love it.
40:38 Brad What are you looking for,
40:39 Brad Brad?
40:40 Scott I think Ted just did a fantastic job of summarizing what I was looking for.
40:44 Scott And I think it's fantastic feedback that he said,
40:47 Scott hey,
40:48 Scott you got to be careful about how you ask the question.
40:49 Scott I think that I mean, I I really appreciate you calling me on that,
40:52 Scott Ted,
40:52 Scott because I agree like you can you can ask the question in such a way that I'm like you said, interviewing is hard, you know, it's
41:01 Brad It's
41:01 Scott very hard.
41:01 Brad very hard.
41:02 Brad It's very hard.
41:03 Brad And this is part of the reason why people should be trained to interview.
41:09 Brad Right.
41:10 Brad I mean, forget for a moment the legal liabilities around interviewing,
41:15 Brad right?
41:15 Brad All the things you are not allowed to ask a candidate as part of an interview process,
41:19 Brad which comes as a surprise to anybody who's never been through formal interview training.
41:24 Brad No,
41:24 Brad I don't get to ask how old you are.
41:26 Brad No,
41:26 Brad I don't get to ask if you're married.
41:28 Brad No,
41:28 Brad I don't get to ask if you have kids.
41:30 Brad You know, in many respects,
41:32 Brad I don't get to ask any questions that would.
41:34 Brad in, you know, indirectly give me the answer to that information,
41:38 Brad even if I'm just trying to be friendly,
41:41 Brad right?
41:41 Brad Because all of that can open you up to a discrimination lawsuit.
41:45 Brad So it becomes important to train people on what we're looking for in an interview and, you know, how we use the rubric and how we phrase these questions and so forth.
41:57 Brad Um,
41:58 Brad I,
41:59 Brad the first time I ever interviewed a candidate,
42:01 Brad my boss walked past my cubicle and tossed some papers on my desk and said,
42:06 Brad yeah,
42:07 Brad we've got three candidates coming in this afternoon.
42:09 Brad I want you to interview them.
42:11 Brad Okay,
42:11 Brad if you're doing that, you're doing it wrong,
42:14 Brad period, full stop.
42:15 Brad Those candidates were screwed even before they set foot in the building.
42:19 Brad Right. Because in the absence of any more formal training,
42:22 Brad I didn't have an idea of what questions I was going to ask. I didn't even know what position we were interviewing them for.
42:28 Brad I was just getting a general sense of their technical skills, as my boss put it.
42:34 Brad Okay,
42:35 Brad I'm going to ask you questions about the Windows API and C++ and, you know.
42:41 Brad I have no idea.
42:42 Brad Are we actually hiring you to do Pearl?
42:43 Brad I
42:44 Scott Yeah.
42:44 Brad have no idea.
42:45 Brad That was a fail.
42:46 Brad That was absolutely a fail on the part of the company.
42:49 Brad It had nothing to do with the candidates.
42:51 Scott So,
42:52 Scott Ted,
42:52 Scott first of all, this has been a fantastic conversation. I had no idea where it was going to go.
43:02 Scott How can we try to wrap this up? Like what's the best advice? I mean, amazing stories about how hard you work to try to put these rubrics and these standards in place.
43:13 Scott Is there a simple version of this that CTOs or engineering managers listening to this can take away?
43:20 Scott Like what's the,
43:20 Scott obviously you can't give us the answer,
43:22 Scott but what's at least the process to get the answer that's right for the organization?
43:28 Brad So it's it's so a recognize that everything we're talking about here is essentially philosophy,
43:35 Brad right? We're talking about a particular organization's philosophy about interviewing.
43:43 Brad And it begins in many respects with when we when we think about interviewing,
43:48 Brad what are we actually doing? And the short answer is we're making a bet.
43:53 Brad Me hiring you,
43:54 Brad Brad,
43:55 Brad is a bet that you're going to deliver as much or more value to the company than it will cost us to have you here.
44:03 Brad And in the nature of all things bet,
44:06 Brad there is a degree of vagueness and opacity to it. I can't see into the future.
44:15 Brad But what I can do is I can make it very clear to myself what it is exactly I'm looking for.
44:21 Brad This is where we get back to job description. This is where we get back to performance expectations.
44:27 Brad And by the way, the job description and the performance expectations had better line up on top of each other like a Venn diagram.
44:33 Brad They'd better be so close they're almost identical because otherwise you're setting people up to fail.
44:41 Brad That's the first part is getting that to be, you know, items that are actionable and measurable,
44:47 Brad you know. Now, not everything is going to be measurable.
44:50 Brad We're not trying to reduce this to an algorithm, but we at the very least want to create a heuristic out of it, right?
44:56 Brad The second part of it then becomes how do I gather evidence that suggests whether or not this is a good bet.
45:03 Brad The interview is one such piece of evidence.
45:07 Brad My GitHub profile is another such piece of evidence.
45:11 Brad And any articles that I have published is a piece of evidence.
45:14 Brad Any podcasts that I've done, anything,
45:17 Brad any college projects that I did when I was recently graduated,
45:21 Brad the elite code problem is just one of thousands of potential pieces of evidence. And
45:28 Brad And as an organization, we should really sit down and ask ourselves,
45:32 Brad what evidence would we accept?
45:34 Brad If Guido von Rossum or Anders Halsberg showed up at your office one day and said,
45:41 Brad I want to work for you,
45:43 Brad would you make them take your leak code problem?
45:47 Brad Probably not,
45:49 Brad right? Even though HR might suggest that we have to have a standardized process for everybody.
45:55 Brad What we really need to do is we need to establish that there is evidence out there that supports the idea that this person has technical skill.
46:02 Brad Well,
46:03 Brad Xander's invented C-sharp. Guido's been the benevolent dictator of Python for life up until he stepped down, which begs the question of that whole for life part.
46:13 Brad But clearly they both have contributions that are very easily recognizable.
46:18 Brad They've already established that they have the technical skills.
46:22 Brad Here's somebody who comes to us from working on open source on GBE or, you know, Jakarta Tommy or one of these other open source projects that's out there.
46:33 Brad Can I go look at their poll requests?
46:35 Brad Can I go follow some of their flow?
46:38 Brad What evidence do they have out there in the public eye that I can look at and say they have the skills?
46:44 Brad So now we can actually start thinking more about human personalities.
46:49 Brad Are they going to be a difficult person to work with, which is always hard to elucidate in an interview,
46:54 Brad but we've already established their technical bona fides.
46:58 Brad So now we can move on to some of these other questions.
47:02 Brad Thinking about your process,
47:04 Brad thinking about what it is you're trying to get out of it, and then remembering that feedback,
47:09 Brad right?
47:10 Brad Feedback always has to, there has to be a loop there.
47:13 Brad How do we get feedback about our interview process,
47:16 Brad both from the interviewers as well as from the interviewee? How do we do a retro on interviewing,
47:23 Brad right? What does that look like? How do we do that in such a way that is?
47:27 Brad that is reasonably objective these are not easy things these are hard but when you do them and do them well now all of a sudden you'll find that the quality of candidates coming into the company is much stronger and you have much less concerns that somebody just chat gpt'd
47:43 Scott Yep.
47:43 Brad their way into a job
47:44 Scott Yep.
47:46 Scott Okay, thanks for joining us today.
47:47 Brad You're very welcome.
47:49 Brad You know, it's interesting.
47:50 Brad I'm reading a book right now that actually suggests that having objectives is not the...
47:56 Brad best way to accomplish the thing you're trying to accomplish it's written by a pair of AI researchers and the book they write is not the book that they thought that they would be writing it's literally about how they they they are finding repeatedly in a lot of the research that they're doing that going for that objective going straight towards it doesn't always get you there the best analogy I can think about is
48:22 Brad is the Chinese finger trap you know you put your fingers in both end of it and more you try to pull them apart the tighter they get stuck most objectives will have measurements that tell you how closer you are to getting your objective which means I need to try to measure how far apart my fingers are but the way to get out of a Chinese finger trap is not to keep pulling hard but to push them closer together
48:46 Brad which by any objective measurement means I'm actually failing.
48:50 Brad And that's a really,
48:51 Brad I'm about four chapters into this book and I'm fascinated because they're really touching on something that the major successes in life don't come from single-mindedly pursuing an objective.
49:05 Brad And it's like, wow,
49:06 Brad that really, really opens up a whole new world of thinking about how we go about doing this.
49:14 Brad And you have to try and iterate and play and experiment before you can actually accomplish the thing that you're setting out to do.
49:23 Brad I don't know. I just, I find this whole space fascinating and the older I get and the less I know,
49:30 Brad the more I find out that I'm discovering all sorts of interesting things,
49:34 Brad way more than when I was 25 and I thought the world revolved around C++ and Java.
49:39 Brad So great
49:40 Scott Love it.
49:41 Scott All right. With that, we'll close this episode of How Many CTOs Does It Take?
49:46 Scott Thank you, Ted.
49:47 Scott Thank you, Ted.
49:48 Scott It was great talking to you.
49:49 Brad talking to you guys too.
49:52 Scott This episode of How Many CTOs Does It Take podcast was edited by Saffron Epigau.
49:56 Scott Our theme song was AI Generated by Brad.
50:00 Scott Please follow us on all the social media platforms at at howmanyctospod. You can find us on YouTube,
50:07 Scott LinkedIn,
50:08 Scott Instagram,
50:08 Scott Facebook,
50:09 Scott Threads, X,
50:10 Scott TikTok,
50:10 Scott and Patreon, or at our website at howmanyctospod.com. And remember to rate, review,
50:18 Scott and subscribe on the podcast platform of your choice.
50:22 Scott Thanks for listening.