Tackling Tech Culture and Interviewing: Insights from Ted Neward
In this episode of "How Many CTOs Does It Take?" podcast, hosts Scott Porad and Brad Hefta-Gaub welcome special guest Ted Neward, a senior distinguished engineer at Capital One, joins the hosts to discuss the challenges and intricacies of engineering leadership and technology culture. Ted shares his diverse experiences, from his previous roles at Rocket Mortgage and Smartsheet to his current position. The discussion delves into the often overlooked aspects of job descriptions, interview processes, and the importance of measurable and actionable performance expectations. Ted also reflects on the use of coding projects in interviews, the role of AI tools in programming, and the necessity of training interviewers to avoid unconscious bias. This episode provides valuable insights into creating effective and inclusive engineering cultures.
- 00:00 Introduction and Guest Welcome
- 00:34 Ted's Professional Journey
- 01:27 Exploring Esoteric Programming Languages
- 03:11 Tech Culture at Rocket Mortgage
- 04:16 Challenges in Job Descriptions and Interviewing
- 08:02 The Importance of Leveling Guidelines
- 11:36 Defining Good Code
- 19:02 Interview Process and Structural Guidance
- 25:53 Interview Training and Culture Challenges
- 27:09 Unintended Consequences of Filtering
- 27:59 Systems Thinking in Hiring
- 30:21 Bias and the Interview Process
- 31:28 Sample Coding Projects in Interviews
- 35:44 AI in the Hiring Process
- 43:29 The Philosophy of Interviewing
- 47:50 Concluding Thoughts and Reflections
- 49:50 Outro and Credits
Resources:
- How Many CTOs Pod: https://howmanyctospod.com
- Scott Porad: https://www.linkedin.com/in/scottporad/
- Brad Hefta-Gaub: https://www.linkedin.com/in/bradheftagaub/
- Ted Neward: https://www.linkedin.com/in/tedneward/
#podcast #technology #leadership #ctos #technicaldebt #management #leadership #problemsolvingtechniques #problemsolvingstrategies #engineering #entrepreneurship #engineeringmanagement #hiring #engineeringtalent
Brad: I fully maintain that most companies do not interview at all well.
Scott: Welcome everybody to another edition of the How Many CTOs Does It Take podcast.
Today we have as a special guest,
Ted Newart.
Ted,
Brad: Yeah.
Scott: welcome.
Brad: Thank you.
Scott: Can you please take a moment,
introduce yourself to our...
22 listeners that listen to this podcast.
Brad: Well,
you know, hi,
my name is Ted Neward. I currently work as a senior distinguished engineer over Capital One.
In prior incarnations, I was the director of technology culture over at Rocket Mortgage. I was the director of the developer relations team and a simultaneously director of
of a development team over at Smartsheet.
And prior to that, I was the CTO of a startup consulting company here in Seattle.
And prior to that, I did a bunch of stuff, spoke at some conferences, wrote some books,
the usual, you know, shenanigans that people get up to doing when they don't have anything better to do with their life.
I spent a lot of time
technologically in both the .NET and the Java worlds and I've probably played around with more programming languages than most people know exist including many esoteric programming languages that you would never wish on your worst enemy
Scott: PL1.
Brad: That's not technically an esoteric language. That's just really old such that it feels esoteric. No, I'm talking about the esoteric languages like Rockstar.
It's an actual programming language so that now you can truthfully claim to be a Rockstar developer on your resume.
It was created, it was born of a tweet by Dylan Beattie,
who is a very popular speaker over in Europe,
and somebody said, you know, that sounds really cool.
I'm going to put together an actual implementation of the specification,
and now it's a real thing, and so inspired by the creation of the language,
Dylan is also now the headliner for a rock band that does parody songs.
I think one of their biggest chart topping hits was You Give Rest a Bad Name,
sung to the Bon Jovi song that sounds somewhat similar to that.
Scott: I love it.
Brad: I've spent a lot of time doing a lot of different things, consulting,
CTO,
engineering management and general,
you know, making an annoyance of myself anywhere that I'm.
not immediately kicked out.
Scott: Well,
welcome to the pod,
Ted.
Brad: Thank
Scott: It's
Brad: you.
Scott: great to have you.
I can't wait to hear some of these war stories.
And it's interesting, too,
because you just kind of mentioned most recently one of the things that you were responsible for, I believe,
at Rocket Money was sort of like the culture of the development team,
Yeah,
right?
I want to hear about that.
Yeah.
Brad: Yeah, so tech culture was an interesting experiment.
The CIO, basically this was just before COVID hit and the CIO had kind of this grand vision that Rocket Mortgage needed to transition from being a mortgage company to becoming a technology company.
And that involves doing a lot of things that really aren't technical in nature,
but had everything to do with, as the name implies,
the culture of the engineering group and that rocket mortgage.
And so he asked me to put together a group that would focus on that for some length of time. We were thinking probably two years.
Turned out to be less simply because other parts of the company said,
you know, that actually belongs.
lungs over here rather than over there in engineering so we focus for example on job descriptions which on the one hand you think that sounds really boring and pedestrian
But, you know, what we discovered, we went through Workday, which is the bane of my existence.
I hate it. It's absolutely worse.
It's the worst tool ever, except for all of the other HR tools out there.
So I don't know what that's saying.
But we went through and we looked at all of the different job descriptions that were in the engineering sphere at Rock Mortgage. And it turned out that if you wanted to be an associate scientist,
software engineer at Rocket Mortgage. You needed to have two to three years of experience,
which considering this was an entry-level position kind of begged the question,
how are you ever supposed to get two to three years experience,
you know, unless we're just saying we don't want people who are brand new right out of college.
So that was an interesting discovery. The even more interesting discovery was that if you wanted to be a vice president of technology at this company.
The only thing that was required was a four-year degree.
that was
Scott: no
Brad: it no
Scott: experience necessary
Brad: experience
Scott: no
Brad: necessary whatsoever you didn't have to have any technical skills you didn't have to have anything you just needed a four-year degree so clearly there was some major calibration issues and frankly you know it may sound like I'm throwing shade at Rocket Mortgage I'm really not because they at least have a
Scott: this
Brad: gut
Scott: this
Brad: who
Scott: is this
Brad: like yeah
Scott: is the story i've i've heard it at so many places i've
Brad: yeah
Scott: been or yeah
Brad: and
Scott: totally
Brad: so we formed a group we called it tech
tech talent and that group literally brought sort of an agile development methodology to working with the folks over in HR to clean up all of the different job descriptions inside of Workday.
As another part of it,
we focused on interviewing,
right?
You know, how do we calibrate interviews across the company?
And I fully maintain that interviewing at most, even the FANGs, most companies do not interview at all well.
Despite the fact that, you know, we've studied how they do interviews,
I mean, 20 years ago it was, you know, how do you move Mount Fuji?
Why are manhole covers round? It was all about these really interesting questions designed to show us how you think.
When in truth, what they were really doing was, you know, for a certain definition of the term,
interview torture.
It's literally,
can I throw something at you that completely floors you that you can't, you know, reason your way out of in any way, shape or form?
And what does make for effective interviews and how do we normalize this?
Having an interviewer having a VP say,
well, I know what I, you know, when I see the right candidate,
I'll know it sort of that Potter Stewart quote,
right? I can't define it for you, but I'll know it when I see it.
That quote, by the way, has done more damage to Western civilization than anything else in our modern history.
Because, great,
if that VP is going to be the only one who does interviews for the entire organization,
you know, for a startup of 30 people, that's reasonable.
For a technology organization of 2,500 people inside of a company of 35,000,
that is completely untenable.
But now
Scott: Yeah
Brad: it becomes what exactly, how exactly do we scale that and keep it consistent,
right? These are the kinds of problems that tech culture was wrestling with.
Scott: Yeah, this is so fascinating.
I am being,
I don't want to say triggered, but activated by your stories here.
I have a couple questions for
Brad: Yeah.
Scott: you.
I have to, I'm curious,
I'm assuming you've seen the, let's start with the leveling conversation,
because that's, I love this conversation.
Have you seen the Dropbox public source?
open source version of their leveling guidelines yeah
Brad: I've seen several.
I know Dropbox had one at one point. I think Honeycomb posted one at one point was another.
There were a couple of companies, particularly the smaller the company,
the more willing they were to share what they had in terms of their leveling guidelines.
And a lot of this was part of the whole push to be more transparent about salaries.
values as well.
So I've seen several of them and truthfully they all start to blur.
You know, the fact that, to be very blunt,
I don't really care what your leveling guidelines look like.
I care that they exist.
I care that they're written down and that they're targetable and they're actionable.
Right.
Scott: Yeah.
Brad: Because now it actually makes it much easier for myself as an engineering manager,
for somebody else as somebody who works for me, to have a better sense of expectations and I can go and compare that against, you know, the leveling guidelines,
the job descriptions in lack of a leveling guideline to say, well, I've done these things on this list.
And I've done them at this particular level. So therefore,
I should be, you know, I should be set to receive a midpoint grain or a above average grade or, you know, whatever,
however you want to do that a five point scale or a or a terminology scale.
But it gives me the opportunity to be able to look and see what I'm doing on a day to day basis to make sure that it lines up with that.
Right.
And if I don't know.
Then what happens is I fall back to what my boss tells me to do.
And then if we change bosses, I'm screwed.
I mean, the new boss says,
well,
that's not what I would have asked you to do.
But that's not the point.
This is what my boss at the time asked me to do. And you start getting into a very much, you know, the corporate version of a he said,
she said kind of scenario.
And frankly,
nobody wins in that scenario.
So the fact that you've published something right there just already puts you head and shoulders above the rest of the crowd.
Scott: How do you think about this question?
Sometimes these leveling documents are written very like, I call it practical or performance based and sometimes they're very much competency based and sometimes they should be very much values based.
You know, like an engineer,
a frontline engineer working on a team, you know, has features and projects coming down the pipeline,
etc.
often isn't necessarily in control of the.
what they're working on as opposed to you know doing it and so I've just seen these leveling guides written differently like sometimes I've seen it's like it's a set of competencies and so if you are doing all these things okay but it doesn't actually say that you ever delivered any software that seems like a key part of the job here is delivering software sometimes it's very much delivery based sometimes it's very much value based I've never kind of thought what's your take on the mix of those things
Brad: So to me, the biggest thing is that there's sort of twofold there.
One is, are they in any way, shape or form measurable,
right?
Scott: Mm-hmm.
Brad: And I'll take as an example, frequently when you see these job descriptions or these performance expectations,
you'll see the line, write good code.
Okay, great.
What is good code?
Scott: Well, it's not evil.
Brad: Pick your favorite programming language,
right?
And by the way, if my code is used on the death stars that make it good code or bad code,
Scott: Exactly.
Brad: you know, good enough to pass Emperor Palpatine's quality check so I didn't get force choked.
But on the other hand, it's used to destroy planets.
So good code or bad code? Seriously,
what does good code look like?
And you ask this in a room of a dozen engineers and what you will quickly find is they will segregate into three or four camps.
camps, each of which bitterly hates the other.
Right.
Because, you know, on the one hand, good code takes full advantage of all of the capabilities of a programming, you know, of the programming language available to you.
And so now you start seeing,
you know, like the ternary operator instead of using a traditional if else.
And there are a lot of people who are like,
oh,
my God, no,
if I see the ternary, that's just too subtle, too hard for people to reason about.
Okay,
both of these are reasonable positions,
but which defines good code?
And what we find is, this is why I really hate that Potter Stewart quote,
because for most people, good code is when I look at it and it speaks to me personally.
But unless you can articulate it, unless you can articulate what makes good code over bad code.
then you can't use that as part of any sort of decision-centric criteria because we can't decide,
even the three of us on this call,
if we can't decide what good code is,
how is anybody else supposed to decide what good code is so that they can decide to write good code?
Scott: How do you solve for that?
Brad: Well, in that particular case,
if you really want to have something...
describing the code then bring it to something that's more concrete right so for example writes code that adheres to our coding standard as documented here
Right.
And so that coding standard might say things like always prefer if else unless is an expression is immediately,
you know, the result of this, you know, if the else is going to yield a particular expression,
then you can use a ternary operator.
however you want to phrase that right um you know prefers not to use chained method calls beyond two or three levels deep which by the way blows away almost all of your builder style fluent interfaces or whatever that is right if it's written down if there is a coding standard out there right and i don't mean chooses smart variable names that's not really what i'm thinking about but i'm
What I'm thinking about is something that somebody can look at code and decide whether or not this adheres to that code standard or gets close to that code standard.
Scott: Well, and hopefully with most of those things you just mentioned, we've got linters running that are going to not let us check in code that doesn't match that coding standard.
Brad: That's useful for when I have full control over the build process,
less useful for when we're talking about doing interview styles.
Right.
But on top of that, there's a lot of things that a good Linter won't be able to pick up on.
And truthfully, it's actually quite surprising how few places of business actually use those static analysis tools.
Scott: Yeah.
Brad: There's just, I mean, I just run across all these different places where the idea of a static tool that would go in and, you know, evaluate the source code as opposed to we're going to do pull requests,
we're going to do code reviews,
we're going to do a human centric process.
Whatever reason, the industry still feels, it feels to me like our industry is still very uncomfortable with that kind of automated judgment,
if you will, about my code.
Scott: Yeah, that's so interesting.
I know that one of the things that, you know, I'm currently my primary gig is with a pretty small team. You know, there's less than a dozen of us.
And early days, it was like blank hard drive and we were starting from scratch.
And I kind of talked about this in our kind of intro episode where early on when there was only a couple of us, we put in place a bunch of these.
kind of best practices like having linters running having you know having just a light a little bit of process you know even doing like sprints with retros even though there's only three of us you know and it is interesting I I kind of like feel like
I've got so many scars from when it was a 500 engineering,
a 500 engineer size team and people were arguing about, you know, brace placement on code reviews. And it's like that, like that is a waste of everybody's time.
Brad: I
Scott: Let's
Brad: agree with
Scott: just
Brad: you and
Scott: agree.
Let's agree on what we're going to code to.
Let's put static linters in place and it doesn't, you can't check it in if it doesn't pass lint, you know.
Brad: I'll even go so far as to say
Say, look,
there is a canonical representation when it sits inside of source control,
but when you check the code out and you start working with it.
Feel free to reformat it according to your local preferences because particularly for code formatting, you know, all of these IDEs and so forth are so good at parsing it and being able to say,
oh, you want the braces here,
you want them on the start of the next line,
you actually want them like over on column 80 so it looks like you're programming in Python,
whatever you want to do.
I don't care.
We can reformat it however you want it.
The thing that I think, you know, and you actually, you said a trigger phrase for me, which is the word best practice.
Scott: Fair enough.
Brad: There are no such things. I don't know if you knew this. There is no such thing as a best practice.
Scott: My practice.
Brad: Well,
okay,
that you can you can free to say my practice and kind of like my dad used to say, so long as you live under my roof, you will cut your hair the way I want it.
wanted to be cut because I was living under his roof when it's under your roof you can call it your practice all day long but that doesn't necessarily make it a best practice because
In some respects,
what works for a team of a dozen people may not work for a team of 500 or for an organization of 500. And there you actually might want to allow each of the teams a certain amount of localized control in terms of how they want to do it.
Maybe all of the ANSI K&R developers are on that team over there and they all want their braces on the next line.
And frankly, at the end of the day, I really don't care if I'm not writing code on that team.
What I care about is that the team is able to.
jump in look at source code and ingest it you know wholesale rather than having to parse through piece by piece because that's one of the things psychologically that we see that's different between a junior and a senior developer the senior can just like literally the brain pattern matches the whole thing you slurp the whole thing in at once and you understand what's going on the junior has to kind of read it line by line to be able to track what's going on
And frankly,
the code being formatted a particular way, if it makes it easier for the senior to slurp it in, great,
reformat it however you want it.
Scott: Yeah.
So we kind of have ended up on a tributary of the river,
which is we were talking about culture and you were talking about changing a culture and you started and you caught my attention at least by starting with the leveling guide and the interviewing,
which my interpretation.
The hesitation after thinking about that was,
okay,
so get the right people in the room and the culture presumably starts to take care of itself.
Brad: The problem is getting the right people in the room is hard at scale.
Right.
Scott: Mm
Brad: Because
Scott: -hmm.
Brad: each of us as an interviewer will bring our own biases and our own perspectives into the conversation.
And we have to be very,
very careful that as an organization, whether or not somebody gets hired is more a factor of which of the three of us they interviewed with rather than their own skill set.
Right.
Because that's actually something that we have seen happen at a number of organizations.
If you've got like all these different interview pods,
again,
I'm talking about a large scale organization,
2,500 engineering,
35,000 total.
You're not going to have five people that do all the interviews for all of engineering.
That's just not feasible,
you know, and they're not going to do it for in addition to the thirty five thousand of rocket mortgage.
They're not going to do the interviews for rocket homes,
rocket money,
rocket auto,
rock central.
So we have to have a way of somehow structurally making sure that these pods are all coming to within some acceptable degree of difference,
right?
Because otherwise,
if this pod is really,
really, really liberal about who it is they hire and this other pod is really, really strict about what they hire.
We could have the same person interview twice at the company,
once get a yes,
the other get a no,
and all of a sudden we don't really know what we're hiring anymore.
We really don't know the skill level of the people that we hire.
So this is partly where it's not just about get the right people in the room.
We also have to give them some structural guidance about what
Scott: Yeah
Brad: it means to be an associate level engineer versus a senior level engineer versus an architect versus an engineering manager.
We have to give them those guidelines and we have to hold them accountable to those guidelines.
Because, again,
Scott: Hold the interviewer as well as the interviewee.
Brad: exactly.
One of the things I've always wanted to do at a company and I've never found a company that was brave enough to do it, which is unit test our interview process.
Right?
Scott: What would that look like?
Brad: Well,
I mean, just as with any sort of unit testing scenario,
right, you take known quantities and put it through your code to see if it comes up with the right answer.
So take a known quantity, that is to take somebody whose skill set you are comfortable with and understand and put them through the interview process and see what result the interviewers come back with.
Now, the ideal scenario to do this is where the people doing the interviewing don't know that this is a test, right?
We have to kind of mock it out a little bit.
Scott: Double-blind it somehow.
Brad: Yeah,
exactly.
And so this is where, you know, you take some of your colleagues,
people that, you know, you or I know from the local community,
somebody who is currently a,
you know, you grab somebody at Microsoft, for example, since we're up here, I live in Redmond.
Redmond grab a buddy at Microsoft and say okay you are a senior software engineer Microsoft I'm gonna you know I'm gonna buy you dinner in exchange for going through an interview cycle over here at my company
And let's see where my interview process pegs them out,
because I've watched companies interview somebody and say, you know, oh,
we don't think this person is a senior software engineer.
And then you follow them on LinkedIn and discover that they just got hired at Microsoft as a senior software engineer.
And it's like, wait,
do we think somehow that our notion of a senior software engineer is actually somebody that's stronger?
Then somebody at Microsoft,
I mean, how exactly does this relate?
And without that kind of sanity checking,
right?
I mean, we spend so much time trying to get feedback on so many other elements of engineering,
yet we trust our gut and our intuition when it comes to interviewing because trusting our gut and our intuition, that was always a great way to write software too,
right?
Scott: Right.
Ted,
so it's interesting.
One simple question.
Did you actually ever unit test this? Did you try this?
Brad: I have gotten so close to actually
Scott: So
Brad: pulling it off.
We did do, we did do a little bit of this. I mean, the HR people had a fit when I told them what I wanted to do for a variety of reasons,
some of which may be legit.
I just don't know many of which I think were imagined,
right?
You know, one was,
well, you know, how are we going to compensate these people for their time, blah,
blah, blah. It's like, look.
Let me worry about that, right?
Many of these people that I would bring in would be doing it primarily just on a lark. You
Scott: Yeah
Brad: know, it'd be just kind of fun to see what would happen,
right?
To do that sustainably, obviously, you'd have to figure out a little bit more sustainable model.
But we did actually do,
we started looking at a tool designed to try to help us do some of this interviewing stuff. One of the
It claimed it was AI-based.
I don't know how much AI was actually in it, but that it would help with some of the interviewing process.
And we actually solicited some volunteers from inside the company to sanity check,
right?
It wasn't necessarily a full unit test.
But at least it was a, oh, yeah,
you know, my role is currently this, and I did these things inside of your interview process, and,
you know, it felt about,
you know, it felt similar to what I was doing.
If we took an associate and put them through the senior,
the associate was just lost, they struggled,
which is exactly the right result.
They should struggle if they're interviewing for a senior position.
So we did a little bit of that.
We didn't we didn't do it nearly as comprehensively as I would have liked
Scott: Another thing that this comes to mind for me or kind of brings up for me is and Scott might remember some of this and be somewhat triggered by this as well. He and I spent a short stint at a company that will be remain nameless and they had a very,
very structured interview process that was focused on the company's core values.
Brad: Mm hmm.
Scott: And they used, you know, star technique and they had they used probably it was greenhouse or something to like that where they had scorecards and they had to produce, you know, the interview. Everybody was trained before they could become an interview all about soliciting data and using that data in their scoring rubric against all these different things.
What's interesting to me is.
In my sense, they.
They spent a lot of time and energy. We spent a lot of time and energy on training people to the tool,
the interview tooling process.
And at least when I arrived and Scott was already there and one of the first conversations we had when we sort of like what I'm like,
hey,
I'm new here.
What do I got to work with? And Scott kind of let me in on some of the challenges that they were facing.
And one of the interesting side effects was that.
was that they had leaned into the culture question and
Brad: Yeah
Scott: one of the goals of the culture that they had developed was very much to absolutely filter out any toxic programmerness.
Right. Which
Brad: Yep.
Scott: is a pretty reasonable thing to try to filter out.
Brad: Yep.
Scott: But they were then kind of realizing that one of the things that they had inadvertently filtered out in this process was courage or sort of like willingness to sort of try to take a risk in solving a problem.
Brad: Yep.
Scott: Right.
Brad: Yep.
Scott: And it was so interesting to me. That was one of the lessons I took away from that was that, you know, it's like,
I don't think anybody intentionally gamed the system, but anything that you can measure can be gamed.
Brad: Yep.
Scott: And this is sort of like a corollary of that where we weren't, no one was trying to game it, but the framework resulted in.
Filtering out what we wanted to filter out and had a side effect of filtering out something we definitively did not want to filter
Brad: This
Scott: out.
Brad: is where we start getting into more of the systems thinking,
right?
The realization that the levers and the knobs and so forth that we can slide and turn,
they don't always correspond linearly to the results that we get,
right?
And frankly,
you know, you're going to make mistakes like that.
I mean, when you start, you know, when you start plucking around inside a system and you start trying to create some of these structural things,
you're going to absolutely run into some of that. And kudos to them for actually doing the diagnostic work to figure out that they were filtering out this quality that they did like as they were trying to filter out qualities that they don't.
And frankly,
there's, you know, as with most things, it's not one or zero, it's on a continuum.
Right. Having the courage to speak up in a meeting and say, I think we are headed down a bad path.
Well,
that same courage is what enables me to say, never trust anybody over the age of 35.
Right. There's a question of degree here.
We don't want to filter out courage.
We want you to have just the right amount of courage.
And frankly,
that is a very, very hard thing to calibrate.
In humans,
until psychologists are able to map personalities to a point-based scale,
And which will be never,
by the way, it's certainly not in our lifetimes.
We're going to struggle with this, but we see the same thing has happened anytime companies have,
for example, changed CEOs,
right? Look at the stark difference that occurred when Microsoft got rid of Steve Ballmer and brought in Satya Nadella. The company transformed itself in a two year span of time from nobody wants to be here because it's fiercely cutthroat competitive.
to everybody wants to be here because they are really focused on churning out some cool stuff and that's that's just from the structural change of having a new ceo some of this is just going to happen right so it's it's it's almost heisenberg in nature right the fact that you're observing it means that intrinsically you're influencing the outcome i don't think we can avoid that and let me be really clear right all those rubrics and
And all of that structural stuff,
all of that is there to combat the unconscious bias that each of us brings into an interview.
Let's be very, very painfully transparent about it.
I am most comfortable when I am looking across the table at somebody who looks like me,
Scott: That's human nature.
Brad: right?
For any definition of I,
right,
whether it's Brad or Scott or Ted or my wife Charlotte or any of my.
any of my kids or any of my friends,
they are always more comfortable looking across the table at somebody just like them.
Again,
like you said,
it's human nature.
So how do we try to combat that so that we are,
in fact, trying to find people who are qualified and could do the job, not just somebody that, yeah, sort of can do it and besides which we both like the Seahawks so they can't be a bad person.
Scott: That's sort of the origin of the sample coding project.
Which I personally found a lot of success with that in my career, but it seems to have fallen out of favor.
I wonder what your take is on that.
Brad: So there's a couple of things to realize about a sample coding project. And before I go down this path,
let me be very clear.
I like it as well.
Personally,
I think the best way to the best way to test a band at your wedding is to ask them to play the funky chicken.
Right.
Because you know you're going to ask them to do that at your wedding.
Give them something reasonable to do and see how they do it.
That seems to me to be by far and away the best way to test anybody for anything.
That said,
the coding project,
a lot of this will start to get into some of the implementation details,
right?
So first of all, if I say, let me take you to the whiteboard.
And let me ask you to scoped out a breadth first tree search algorithm using nothing more than the whiteboard and the whiteboard marker.
Those of us who went to university and studied computer science and did so recently will intrinsically have a better chance of passing that than those of us who never went to university or never went to computer science.
My personally,
my degree was international relations. You want to talk about when nations go to war,
you want to talk about the difference between macro and microeconomics and its influence on foreign policy.
See, I'm your chicken.
I'll go.
Let's go all day long.
But I didn't study all of that classic big O notation and whatnot.
And every time I got into a conversation about some of the formalized aspects of computer science,
I was at a disadvantage.
Now, the other question is, is what I'm going to be doing on a daily basis actually reflective in the code sample?
Right.
If you're asking me to balance a tree or if you're asking me to search or sort a tree,
is that actually what I'm doing on the job or am I moving a button three pixels to the left?
And frankly,
if it's more the latter,
then it's not the fact that you have a code sample that's the problem. It's the fact that your code is sampling the wrong thing.
You're not asking the band to play the Funky Chicken,
you're asking them to play Beethoven's Fifth.
Which for a wedding really just doesn't make a whole lot of sense,
right?
Scott: Right.
We put a lot of energy into getting that coding project to mimic as best as we could the kinds of things that you would be working on in if you came to work at our company because we were doing a consumer Internet website to really break it down.
you're basically building forms I mean
Brad: Forms
Scott: that's you
Brad: over data.
Scott: weren't really just building for but you were doing crud right you were getting data in and out and some other things related to that so we tried to create something which just asked the question can this person actually do the thing that we kind of have to do a lot and
Our recruiters really didn't like it because it slowed the process down and people would drop out and things like that.
Brad: Well, I was just going to say recruiters don't like it because they want to get as many candidates as possible in front of the hiring manager, because in some cases.
You know, again,
going back to maximizing what you are measured on,
right,
Campbell's law and all that,
recruiters are often compensated based on the number of people that they can put in front of a hiring manager.
So their goal is to maximize throughput,
you know, through the process.
If they were actually compensated for the minimum number of candidates required before we got a successful one,
they would actually be more interested.
more interested in
Scott: Oh,
Brad: filtering out
Scott: yeah.
Brad: more upfront.
This is where, again,
just playing around with the numbers just a little bit can really change how people's behavior carries forward.
And frankly,
I think you did the right thing in terms of you structured your interview process because...
And it's absolutely a signal to you as to how well this person will perform on the job. It's also a signal to the person taking this that this is what this is going to be like, right?
You can absolutely lean in to say,
your job here will be doing lots of things like this and in this kind of environment. So, you know.
This is the job, right?
It works as a two-way street there.
Scott: By the way, Ted and Scott,
this kind of dovetails nicely into sort of a little mini controversial question,
which is, you know, nowadays with the rise of AI coding tools and agents,
I personally ask the question,
well, how do you use AI in your day-to-day work?
And the answer to that question seems pretty important to me.
You may guess where I land on what I want to hear from that person or what I don't want to hear,
but what do you think about that?
Brad: So as a question, I think it's reasonable to ask,
right?
Now,
the part that concerns me a little bit is you know what you want to hear.
Which now leads me to, okay,
so this is basically a checkbox, right?
You know, whichever way you want to go. Do you use AI?
Yes, no, right?
And it doesn't, those kinds of things often don't allow for subtlety, right?
Scott: Well, actually,
Ted,
what I wanted to hear is let's have a conversation about.
Do you or don't you and how did you make that choice?
Brad: So then I think when you phrase it that way, you're putting the question in a different light, right?
We sometimes get a little bit over clever with our questions because we think that people can understand where we're coming from,
right?
It goes back to the why are manhole covers round,
right?
And people at Microsoft were asking that question because they wanted people to kind of reason through it.
But because that particular nuance wasn't always conveyed as a part of the question itself,
there were a lot of people at Microsoft that were interviewing people saying,
why are manhole covers round?
And the person would say they're round because,
well, that makes it easier for them to roll down the street in order to be able to get them into place.
And the interviewer would be like,
OK,
well,
I don't really know the right answer here says that they won't.
they won't fall in
Scott: Ha!
Brad: but i actually had a conversation with a road crew not too long ago who said no absolutely they could still fall in depending upon you know if you like wedged it one guy said oh yeah i've had a man called fall in on me before yeah i don't know how i don't really care
The bigger thing is, if you want to ask a question about, do I think AI is important?
Do I use it as a part of my daily life?
Just ask that.
Scott: Hmm.
Brad: Let's get away from the era of trick questions where we're trying to elicit a particular,
you know, I want to see if they'll be able to follow my way of thinking around here.
You know, personally,
I don't use any AI tools.
I don't, partly because I don't code on a daily basis as it stands,
partly because I don't think we've fully figured out what our relationship to these AI tools is or should be,
because I'm hearing all kinds of stories of people saying,
oh, yeah, I had AI write this whole thing for me,
and now I'm finding that there are all kinds of things buried inside this code that, you know, there are landmines waiting to go off.
and I didn't know it was there so you know I think it's very reasonable if you say do you use AI there are certain people who are listening to this podcast who thought aha Brad wants them to say yes or other people listening to this podcast said aha Brad wants them to say no and other people still were saying aha
bro doesn't know what the right answer is and so he's trying to get some free consulting out of it you know
Scott: We all know that the real right answer is it depends.
Of course, that's the right answer.
Brad: spoken like a true consultant scott and a man after my own heart yeah yeah this is what i mean when you ask an interview question right the candidates are already under so much stress and they're under so much pressure
don't be around the bush just ask them your damn question what do you think about AI you know I believe AI
Scott: Yeah.
Brad: is transformative but we have no idea what the actual transformation will look like I'm curious to hear your thoughts right and now on the rubric and this is one way to make sure that your question is fairly approachable is to be able to write out a rubric
In some cases, it's not whether or not somebody follows the rubric real closely. It's the fact that you were able to articulate what you're looking for, right?
The act of actually writing the rubric is itself the real win there.
Candidate has some experience with AI and or has done some reading.
That's a thumbs up answer.
Candidate blindly plugs everything into chat GPT, that's a thumbs down.
That in of itself,
what are you looking for out of that question can help feedback into how do I phrase the question so that the candidate has a very reasonable chance to answer it fairly.
Scott: Yeah,
I love it.
Brad: What are you looking for,
Brad?
Scott: I think Ted just did a fantastic job of summarizing what I was looking for.
And I think it's fantastic feedback that he said,
hey,
you got to be careful about how you ask the question.
I think that I mean, I I really appreciate you calling me on that,
Ted,
because I agree like you can you can ask the question in such a way that I'm like you said, interviewing is hard, you know, it's
Brad: It's
Scott: very hard.
Brad: very hard.
It's very hard.
And this is part of the reason why people should be trained to interview.
Right.
I mean, forget for a moment the legal liabilities around interviewing,
right?
All the things you are not allowed to ask a candidate as part of an interview process,
which comes as a surprise to anybody who's never been through formal interview training.
No,
I don't get to ask how old you are.
No,
I don't get to ask if you're married.
No,
I don't get to ask if you have kids.
You know, in many respects,
I don't get to ask any questions that would.
in, you know, indirectly give me the answer to that information,
even if I'm just trying to be friendly,
right?
Because all of that can open you up to a discrimination lawsuit.
So it becomes important to train people on what we're looking for in an interview and, you know, how we use the rubric and how we phrase these questions and so forth.
Um,
I,
the first time I ever interviewed a candidate,
my boss walked past my cubicle and tossed some papers on my desk and said,
yeah,
we've got three candidates coming in this afternoon.
I want you to interview them.
Okay,
if you're doing that, you're doing it wrong,
period, full stop.
Those candidates were screwed even before they set foot in the building.
Right. Because in the absence of any more formal training,
I didn't have an idea of what questions I was going to ask. I didn't even know what position we were interviewing them for.
I was just getting a general sense of their technical skills, as my boss put it.
Okay,
I'm going to ask you questions about the Windows API and C++ and, you know.
I have no idea.
Are we actually hiring you to do Pearl?
I
Scott: Yeah.
Brad: have no idea.
That was a fail.
That was absolutely a fail on the part of the company.
It had nothing to do with the candidates.
Scott: So,
Ted,
first of all, this has been a fantastic conversation. I had no idea where it was going to go.
How can we try to wrap this up? Like what's the best advice? I mean, amazing stories about how hard you work to try to put these rubrics and these standards in place.
Is there a simple version of this that CTOs or engineering managers listening to this can take away?
Like what's the,
obviously you can't give us the answer,
but what's at least the process to get the answer that's right for the organization?
Brad: So it's it's so a recognize that everything we're talking about here is essentially philosophy,
right? We're talking about a particular organization's philosophy about interviewing.
And it begins in many respects with when we when we think about interviewing,
what are we actually doing? And the short answer is we're making a bet.
Me hiring you,
Brad,
is a bet that you're going to deliver as much or more value to the company than it will cost us to have you here.
And in the nature of all things bet,
there is a degree of vagueness and opacity to it. I can't see into the future.
But what I can do is I can make it very clear to myself what it is exactly I'm looking for.
This is where we get back to job description. This is where we get back to performance expectations.
And by the way, the job description and the performance expectations had better line up on top of each other like a Venn diagram.
They'd better be so close they're almost identical because otherwise you're setting people up to fail.
That's the first part is getting that to be, you know, items that are actionable and measurable,
you know. Now, not everything is going to be measurable.
We're not trying to reduce this to an algorithm, but we at the very least want to create a heuristic out of it, right?
The second part of it then becomes how do I gather evidence that suggests whether or not this is a good bet.
The interview is one such piece of evidence.
My GitHub profile is another such piece of evidence.
And any articles that I have published is a piece of evidence.
Any podcasts that I've done, anything,
any college projects that I did when I was recently graduated,
the elite code problem is just one of thousands of potential pieces of evidence. And
And as an organization, we should really sit down and ask ourselves,
what evidence would we accept?
If Guido von Rossum or Anders Halsberg showed up at your office one day and said,
I want to work for you,
would you make them take your leak code problem?
Probably not,
right? Even though HR might suggest that we have to have a standardized process for everybody.
What we really need to do is we need to establish that there is evidence out there that supports the idea that this person has technical skill.
Well,
Xander's invented C-sharp. Guido's been the benevolent dictator of Python for life up until he stepped down, which begs the question of that whole for life part.
But clearly they both have contributions that are very easily recognizable.
They've already established that they have the technical skills.
Here's somebody who comes to us from working on open source on GBE or, you know, Jakarta Tommy or one of these other open source projects that's out there.
Can I go look at their poll requests?
Can I go follow some of their flow?
What evidence do they have out there in the public eye that I can look at and say they have the skills?
So now we can actually start thinking more about human personalities.
Are they going to be a difficult person to work with, which is always hard to elucidate in an interview,
but we've already established their technical bona fides.
So now we can move on to some of these other questions.
Thinking about your process,
thinking about what it is you're trying to get out of it, and then remembering that feedback,
right?
Feedback always has to, there has to be a loop there.
How do we get feedback about our interview process,
both from the interviewers as well as from the interviewee? How do we do a retro on interviewing,
right? What does that look like? How do we do that in such a way that is?
that is reasonably objective these are not easy things these are hard but when you do them and do them well now all of a sudden you'll find that the quality of candidates coming into the company is much stronger and you have much less concerns that somebody just chat gpt'd
Scott: Yep.
Brad: their way into a job
Scott: Yep.
Okay, thanks for joining us today.
Brad: You're very welcome.
You know, it's interesting.
I'm reading a book right now that actually suggests that having objectives is not the...
best way to accomplish the thing you're trying to accomplish it's written by a pair of AI researchers and the book they write is not the book that they thought that they would be writing it's literally about how they they they are finding repeatedly in a lot of the research that they're doing that going for that objective going straight towards it doesn't always get you there the best analogy I can think about is
is the Chinese finger trap you know you put your fingers in both end of it and more you try to pull them apart the tighter they get stuck most objectives will have measurements that tell you how closer you are to getting your objective which means I need to try to measure how far apart my fingers are but the way to get out of a Chinese finger trap is not to keep pulling hard but to push them closer together
which by any objective measurement means I'm actually failing.
And that's a really,
I'm about four chapters into this book and I'm fascinated because they're really touching on something that the major successes in life don't come from single-mindedly pursuing an objective.
And it's like, wow,
that really, really opens up a whole new world of thinking about how we go about doing this.
And you have to try and iterate and play and experiment before you can actually accomplish the thing that you're setting out to do.
I don't know. I just, I find this whole space fascinating and the older I get and the less I know,
the more I find out that I'm discovering all sorts of interesting things,
way more than when I was 25 and I thought the world revolved around C++ and Java.
So great
Scott: Love it.
All right. With that, we'll close this episode of How Many CTOs Does It Take?
Thank you, Ted.
Thank you, Ted.
It was great talking to you.
Brad: talking to you guys too.
Scott: This episode of How Many CTOs Does It Take podcast was edited by Saffron Epigau.
Our theme song was AI Generated by Brad.
Please follow us on all the social media platforms at at howmanyctospod. You can find us on YouTube,
LinkedIn,
Instagram,
Facebook,
Threads, X,
TikTok,
and Patreon, or at our website at howmanyctospod.com. And remember to rate, review,
and subscribe on the podcast platform of your choice.
Thanks for listening.