Structuring hiring processes
Part 3 of a 3-part series on learning how to hire as an early startup employee: the nuts and bolts of setting up a hiring process.
There isn't a whole lot of magic involved in being able to hire well. When I first started hiring at our startup with 5 employees, we didn't have any established hiring processes and every search was its own special snowflake. Because we didn’t really know what we were looking for or how to evaluate candidates, few people would make it through our whole process. We also used to wait to figure out the next step after each call, so it’d take months to get to an offer.
But by the time we'd grown our company to 150+ folks, we'd figured out a process that consistently gave us high-quality, successful hires. (Huge thanks to the colleagues and advisors who actually knew how to hire that taught me most of this, and an amazing team partnering with me as we honed our process.)
In previous posts in this series, I went over my general strategies and principles for hiring. In this post, I'll share more of the nuts and bolts of my hiring process. Of course, nothing in this blog post is unique or ground-breaking. But hopefully having our process laid out will help you with yours!
Basic hiring process
My basic hiring process has the following components:
One or two phone screens, to gauge technical fit and tell the candidate about the role.
An exercise, to see the candidate's skillset in action and seed subsequent conversations.
Panels, to dive deeper on the candidate's skillset and possible team fit.
Closing call and references, if needed.
I'm a pretty strong believer that this hiring process can apply to both technical and non-technical roles, and that creativity and intentionality are key to any successful hiring process. That said, this model is certainly much easier to adapt to traditional technical roles like software engineering or data science. If you're thinking of applying this to a non-technical role and struggling to come up with ideas for the exercise or panel, please reach out - I'd love to brainstorm!
This post is written from my perspective, and will likely be most useful to someone new-ish to hiring at an early startup, with a lot of flexibility and control over how they hire and likely little HR support.
Phone screens
There are two types of phone screens: one that I call the "HR" phone screen and the other a technical screen. The goals of the HR screen are to assess high-level fit, share about the role and company, and get baseline compensation and timing expectations. High-level fit isn't anything more detailed than "are they a decent human who can communicate with other humans?" and "is this person a jerk?" The goal of the technical screen is to gauge baseline technical fit: does the candidate actually know the things they say they know on their resume and do they have the minimum skillset to move on to next steps?
Overall, the goal of the screens is to filter out non-viable candidates: are the compensation expectations grossly mismatched? Are they unavailable to start for months? Do they not actually have experience with your team's primary coding language? Are they way too junior to contribute at the level you need?
The initial screen is also for the candidates to filter you out if it's not the right role for them, so it's important to speak candidly about the role in these early conversations. (Remember our principles: respect the candidate!) For example, I once had a great candidate for a role who had a strong machine learning background. We spoke candidly about the lack of machine learning in the role, and even though they were still interested at the end of our call, they followed up the next day to say that the lack of machine learning actually was a deal breaker. If I hadn't been direct and honest that we weren't planning on doing machine learning any time soon, they would have continued in the process and wasted both their own time and my team's.
That said, do be on the lookout for folks self-selecting out erroneously, for example if they feel they are underqualified even though you don’t. It can be worth a follow-up to make sure they're not backing out due to a misunderstanding or, worse, biases that lead to underrepresented folks self-selecting out. I once had a couple of follow-up calls after the initial screen stage with a candidate who wasn't totally sure if she was a good fit for our role - and she's who we ended up hiring to great success!
Phone screen format
Ideally, HR does the HR screen and the hiring manager (you) does the technical screen. If you're short on staff or don't have HR, it's also totally fine to combine the two screens. If it's your first time doing the HR screen, I recommend prepping your language for the compensation expectation discussion, since it can be scary to do for the first time. Also make sure that you know the salary band for the role! Even if your company hasn’t set formal salary bands, your boss should have a sense of how high would be too high to pay someone, so make sure you ask.
For some less technical or more junior roles, you might be able to get a non-technical person like HR to do both screens. In that case, it's important to give your colleague a set of specific questions to ask candidates that are straightforward enough for your colleague to be able to transmit the candidates' answers back to you via their notes.
I've found that 30 minute phone calls are the best for phone screens. In order to keep it to 30 minutes, I usually end up (1) writing down my spiel about the role and (2) writing out 2-3 questions to ask every candidate I speak to. By writing these down, I essentially give myself a mini-script that keeps me from rambling and taking up precious time. It also helps me compare candidates more consistently. Three questions doesn’t feel like a lot, but remember: the goal of the screen is to assess baseline technical fit. For me, I’d usually ask one question about coding technologies (“what languages do you have experience with”), a data science project (“tell me about a complex analysis you did”), and a cross-functional question (“tell me about a time you automated a process in the real world”).
A really important skill to develop as a screener is to learn to cut candidates off: if they're going on too long, going down a path answering a question you didn't ask, or you just need to move on to the next question. It's definitely uncomfortable at first, but being on the phone rather than a video call makes it easier, which is also why I prefer phone calls over video.
During an active interview process, expect to do many phone screens. For a senior role, I plan for up to 10 or 15 screens per week, with 4-5 moving on to panels (for junior roles, I expect about double that). That's why it's so important to keep them to 30 minutes!
Exercises
After the phone screen(s), I like to have candidates do some sort of exercise. Here is where I encourage you to break out of the mold and think very critically about what signal you're trying to get out of the exercise and what it's like for the candidate to go through your exercise. (Remember our principles: focus on signal and respect the candidate!)
Exercises are great for a few reasons:
They give great signal: if you set up your exercise well, then it will have a very high signal-to-noise ratio and teach you more about the candidate than just asking them questions.
They are a way to bring in the rest of the team: if you're hiring for a role that impacts a lot of people, you can't reasonably have the candidate meet everyone in interviews - that would take way too much of your and their time. But take-homes and presentations can be shared among a much larger group than individual conversations, and so you can give more of the team exposure to the candidate without necessarily adding a lot of burden to the interview process.
They are a good crutch for less experienced interviewers: I once saw an impassioned Twitter thread (that I can no longer find!) saying that truly good interviewers don't need an exercise to probe someone's skills. While that may or may not be true, I know that I'm not skilled enough at interviewing (yet) to get all the signal I need from just talking. Seeing someone's work really helps ground conversations.
They let candidates fail: with a good exercise, some candidates will fail spectacularly - this sucks for the candidate but is great for the process because it means you can cancel the rest of the interviews and save your panel and the candidate’s time.
There are three types of exercises: take-homes, pairing exercises, and presentations.
Take-home exercises
These are small assignments that you give to candidates to do at home on their own time. Take-homes are appropriate for junior or mid-level roles, and their goal is primarily as a sanity check: does this person actually know how to code? Do they actually have the minimal skills to do the job?
Respecting the candidate and keeping it real with exercises are critical: if you're asking candidates to do work on their own time, it's extremely important that it's a good use of their time.
That means that the exercise should:
NOT be just boilerplate, copied-from-the-internet questions
reflect some aspect of what the candidate can expect to do in their day-to-day on the job
teach the candidate something about what your company does
be short and doable by someone with zero context
Some design considerations when writing a take-home:
Keep it small: you should assume it will take them at least 2x the time it would take you.
Focus on signal: is this truly giving you signal about their fit for the role, or is it just a generic coding interview? Also make sure that the candidate is also getting signal about the role and/or your company's work from the exercise. If you are using standard questions from the internet, make sure put them in the context of your company's work rather than some vague unnamed organization.
Give them everything they need: don't ask them to do busy work or onerous data wrangling just for the sake of it. Make sure that whatever they need to do reflects signal that you care about, and avoid "gotcha" questions. For example, if they should use a specific tool, don't make them guess, just tell them!
Punt things that require a lot of context or aren’t straightforward to the panel. Candidates should be able to do the exercise with zero context about your work. They also shouldn't have to write an essay to describe their work; if you want to have a conversation with them, then wait until the panels to do so!
I've found that 2-3 straightforward tasks with very explicit instructions plus one open-ended question is a winning format for take-homes. For example, for data science exercises we've given candidates a small, sanitized, and anonymized version of a real dataset that we would use in our day-to-day work. We ask them to do a few basic things like make a plot or calculate some averages and correlations and we close with one open-ended question that is often along the lines of "pick a question that you could answer with this dataset and explore it."
Pairing exercises
Asking more senior candidates to do a take-home may not be feasible or feel respectful to the candidates. If you still want to gauge their coding skills, a good alternative is a live-pairing exercise. It's also a reasonable strategy to prepare both a take-home and pairing exercise, and let candidates choose. That way candidates who aren't comfortable with live interviews can do the work on their own time, and candidates who don't want to do homework can just show up to a call and bang out the work.
The guidelines for a pairing exercise are basically the same as for a take home: focus on signal for both you and the candidate, make it realistic, and give them everything they need.
When you're hosting a pairing session:
Let the candidate use the internet to answer their questions. This will be a huge part of their job, and you can actually learn a lot about how they work by seeing how they search for answers online.
Help the candidate out if they get stuck on anything. It can be nerve-racking to be put on the spot, or they may be more familiar with a different coding language. Details that you're confident they'd be able to easily figure out with a search or their colleagues are fine to just give them. It's not a test; it's a conversation.
Acknowledge that it's an interview setting. When candidates ask questions, don't hesitate to give them the simple "for the purposes of this interview" answer. You can also mention the true, more complex answer, but don't let the details of reality distract from the simplified exercise you've given them.
Presentations
Presentations are a great exercise for scientist, manager, or tech lead roles. Their format is really simple, with slight variations depending on the role:
Any PhD-level+ science role: just give a standard research/job talk.
Senior technical folks: talk us through a technical project in which you had considerable ownership.
Managers: given a hypothetical situation, what would you do?
My favorite example of the hypothetical situation was the prompt we gave our head of data science candidates:
Our executive team has said that launching a customer data dashboard is a top priority for next quarter. What role do you think the Data Science Team will need to, or should, play to support this effort? What specific actions would you take to ensure our Data Science Team’s success?
My favorite type of hypothetical situation is one that isn't hypothetical at all, because it gives the candidates real signal about what their job will be and it helps the panel know how to engage and ask questions regardless of how experienced they are at interviewing.
Presentation prompts can be more vague than take-home exercises, because you’re giving them to more senior folks who should know how to prepare and will need to handle ambiguity in their role.
Exercises for non-technical roles
I strongly believe that this hiring process can be applied to any kind of role, even if it's not traditional software or data science. In those cases, coming up with the exercise just requires a little bit of creativity: what will be this person's actual job? What are the things they need to know how to do with their eyes closed?
One great example of a take-home exercise for a non-technical role is for an executive assistant: given a set of conference dates, book travel and lodging for a busy professor. Apparently, candidates often fail this spectacularly (for example, booking two red eye flights). Another great example for a program manager role is: draft an email to an academic collaborator who is late on getting back to you with a set of results. Here, you get the signal about their work and you can seed a follow-up conversation later where you ask them what additional steps they’d take before sending the email.
Interview panels
For many roles, most candidates will pass the exercise stage. After that, it's on to panels. I try to limit myself to two panels per role, except for more senior or leadership roles where I can add a third if needed. Each panel can have 1-3 interviewers, though I find that pairs of interviewers works really great.
The goal of each panel should be intentional and clear to your panel interviewers. I like to have one within-team panel and one cross-team panel. For example, many of our data science roles had one panel with the role's closest data science colleagues and one panel with representatives from the teams they'd interact most closely with, usually software engineering or customer success.
When you get to this stage of the hiring process, it's a good idea to assemble the whole panel for a briefing meeting where you tell them about the role, tell them about your ideas for their panels, and answer any questions they have. (Remember: respect the panel!)
Panel format
After being on many panels myself, I landed on a standard format that served me well no matter what role I was interviewing for.
Small talk: Start with some small talk — this breaks the ice and helps candidates loosen up a bit.
Overview: Give a brief overview of what they can expect from this call.
Your intros: Introduce yourself and have your fellow interviewer(s) introduce themselves. Talk about your background, mention your current role and how you'd work with this person.
Their intro: Ask the candidate to introduce themselves: ask for a short summary of their background and what brings them to this role. Make sure to engage with this - ask follow up questions about anything in their background you find interesting or relevant.
Exercise debrief: Briefly touch base on the exercise they did (take-home or presentation). Thank them for taking the time to do it. If they got something wrong or did something weird, ask them about it. Otherwise, tell them you don't have specific questions but just want to hear their reflections on the exercise and how it went for them.
Technical content: this is the bulk of the interview, where you work through a problem together. More on this below.
Behavioral questions: Close with some non-technical or behavioral questions. If you're building a team with an intentional culture, make sure to ask questions to get signal on how they'd vibe with that culture.
Q&A: Finally, make sure to leave at least 5 minutes (ideally 10) for their questions.
Technical content
I really like to do some sort of exercise together during the bulk of the technical panel. I find that working through a problem together gives me infinitely better than signal than just peppering a candidate with questions, and it also helps them get a taste of the job.
Ideally, the technical exercise is a watered down version of literally what their job will be. For example:
For a data quality analyst role, teach them to look at some data and associated diagnostic plots and tell them what QC metric they should be looking for. Then go through a few examples of real data and see what they find.
For an epidemiologist role, ask them to talk you through a weird spike in Covid-19 case data. Bonus points, ask them to think through how they'd respond to a customer angrily asking them about it.
For more software-y roles, start with a system partially mapped out in something like Miro. Tell them what you need the overall system to do, and ask them to talk through filling in the missing piece.
This part can be really daunting to the candidate, so I try to give a preamble to make it less scary. My favorite way to do so is to say: “The goal of this conversation is to simulate what it’d be like to work together.” (Because it's true!) I also make it clear they can ask follow-up questions – in fact, we're often more interested in the questions they ask than the answers they give.
For scientific roles, I also really love asking one pie-in-the-sky question — something that's stumped our team or something that's existentially concerning about the field. When I give this question, I make it clear that it's a real question our team is mulling over, that there is no answer let alone a right answer, and that (again) they should ask questions. This is usually the best part of the interview, and engaging with their ideas and creativity can be an amazing way to get high-quality signal on what it’d be like to work together. That said, remember that they have zero context so don't be afraid to redirect them if they're going down the wrong path and celebrate any time they get close to a correct answer: you're the experts, not them!
Meta-signal is the best signal
Meta-signal from exercises and panels can sometimes be more useful than the signal from the interview content itself.
The way that candidates interact with your interviewers and panels is extremely telling: do they ask questions or jump straight to problem-solving? Do they get lost in scientific jargon or are they able to abstract away things they don't understand? Do they argue with you and tell you you're wrong? Are they able to answer junior folks' questions in a way that they understand?
I also love it when candidates break the fourth wall and acknowledge that they're in an interview, talking to real humans on the other side who may one day be their colleagues. This is especially pertinent in presentations, and especially so for high-level roles: did they tailor their talk to the expected audience or is it at the completely wrong level? It’s also relevant during technical exercises: are they asking you questions directly, or just vaguely wondering out loud without leveraging the subject matter experts (you) who are right there?
Failing is also signal
If your hiring process isn't working, that's signal too! Processes can fail in many ways. Sometimes they feel like a slog, or they're going way too slowly. Other times the process is going fine, but the feedback from your interview panel is always mixed and conflicting. And sometimes you're not even getting through to the panel stage, because the candidates just don't seem like the right fit from the start.
No matter what's happening, it's ok! It just means that you may need to revisit your hiring process.
Is the JD not reflective of what you actually need? If you're not getting the right set of skillset into the funnel, revisit your job description. Compare how you talk about the role to your panel with what's written down: is it actually what you're looking for?
Is the panel aligned on the role? Do they know what you, the hiring manager, are looking for?
Is this actually the role your team needs? If it's the right panel and you've re-aligned them, but their feedback stays off the mark, is what they're evaluating the role your team really needs?
Are you looking for a unicorn? If so, what do you really need from this candidate? Could you split this up into two roles?
Making the hire
After each batch of candidates makes it through panels, it’s helpful to reconvene your hiring panel for a debrief to hear everyone's feedback and potentially decide to move to an offer.
Even if it's a clear yes from the written feedback, make sure to meet and get everyone's thoughts - this helps get buy-in and builds excitement within the panel. (Remember: they've put in a bunch of time to the interviews, so if you found someone amazing it's nice to celebrate together!)
If it's a clear no, discuss what was missing and use the conversation as an opportunity to refine your process and expectations.
Finally, if the panel has mixed feedback on a candidate that usually means no — the exception is if part of your panel isn't fully aligned on the role. If the panel isn't totally sure about a candidate, it can also be worth doing one more interview with the hiring manager or a new interviewer for more clarity. Tell the interviewer the panel's concerns and ask them to probe those concerns directly with the candidate. You might also be able to reach out to references to get your questions answered.
Importantly, hiring decisions are not a democracy. Not every interview panel has equal weight: sometimes the intra-team panel is much more important than the cross-team one. Some panels or individuals might even have veto power over candidates. For example, if you're hiring someone's boss and they really didn't like the candidate, it's probably better to skip them and continue with the process. Finally, as the hiring manager you're in charge: it's up to you to gather the panel's feedback, build alignment within the panel, and ultimately make the hiring decision that will best serve your team.
Thanks for making it this far! As I've written in this series, hiring is something that I really enjoyed doing and miss now that I'm no longer in my previous startup role.
If you're embarking on a new hiring process and interested in collaborating, please reach out!