How To Design a Successful Interview Process for Hiring Top Talent

Most companies don’t have a serious, repeatable interviewing process for hiring.

Instead they wing it, bringing people in for interviews, asking a few questions, turning it into a fireside chat, and then hoping for the best. In my experience that’s not good enough.

At Standout Jobs (2007-2010) we hired some great people. I still consider them friends today, but more importantly they’ve all gone on to bigger and better things. And many of them still say that their experience at Standout Jobs was a fantastic one, even though the company wasn’t a success. They learned a lot, grew as people, and found teammates that they genuinely respect and appreciate. A few of the guys went on to work together in other projects and startups, and I’m sure some of them will re-connect in the future.

That feedback is a testament -in part- to the effort we put into hiring and interviewing people. I’ve seen a lot of different hiring strategies, from the nonchalant to the super intense (even more than our approach, which I’ve shared below.) I can’t say there’s one approach that works for every company, or that our approach is even “the right one” for you, but I do know that everyone we hired that went through the process was successful and stayed with us; those that we bent the rules for (often to expedite the hiring process) didn’t pan out quite as well. So for us, the interview process we implemented was a very good filter.

Note: Most of the hiring we did was in 2007-2008. That’s a few years ago, and my memory of every specific detail is a touch fuzzy. I wish I had kept more notes on the specifics, questions we asked, etc. There are those out there that may be able to fill in some details…

1. Reviewing Resumes

The process typically started with a resume. The resume isn’t dead, but the format should change, especially for developers that have plenty of resources (like Github) for sharing their work and demonstrating their capabilities. Resumes still contain some nuggets of information that are helpful, and we went through them all. Here are some of the things we looked for:

  • Spelling errors (almost an automatic “thanks but no thanks”)
  • Lots of short hops between companies (a potentially bad sign)
  • Open source contributions, side projects or other work shared in a portfolio
  • Twitter account and/or blog (so we could take a look at what people were interested in, talking about, etc.
  • Previous startup work and the quality of those startups

Both my co-founder (Fred Ngo) and I reviewed resumes. In some cases we shared resumes with other team members to get their opinions as well. If we were rejecting someone, I’d email them. Every single one. I did my best to make sure everyone that applied heard back from us at least once. This is a tough process, but I encourage every company to do it. After reviewing and filtering resumes, we’d screen the remaining candidates over the phone.

2. Screening Candidates

We put a lot of effort into screening candidates. Given that the next step (in-person interviews) lasted almost a day, we didn’t want to go through that with too many people.

Screening candidates was mostly about:

  1. Verifying the validity of their resume (i.e. how much did they exaggerate?)
  2. Testing for role fit (i.e. do they understand the role, and is that a good fit?)
  3. Testing for cultural fit (i.e. do we think we’ll get along well?)

I tried keeping screening calls to 30 minutes, but they’d often go longer. It was always interesting to see what people talked about (specific work they’d done, accomplishments, interests.) I also gave people an opportunity to ask questions, which was also instructive on their personality and what they cared about.

We had a standard set of fairly open-ended questions, but I wasn’t too concerned if we got through all of them. The goal was to have a conversation and see how it went. Some of the questions I asked included:

  • Tell me about [pick something on the resume]…
  • What did you learn at [pick company name]?
  • What are you looking for in a new company?
  • Tell me about the [pick a side project s/he's worked on]… (or: Have you ever thought of doing a side project of some kind? If so, why haven’t you? What would you do?)
  • What’s an ideal work environment for you?
  • Do you read Hacker News? (i.e. Are you genuinely interested in startups and new technology?)

Some of the gotchas I came across:

  • Lying. A lot of people lie on their resume. They drastically overstate their contributions to things that were successful. You expect some exaggeration (or even lack of clarity because it’s hard to measure how valuable a specific person’s code was to a project’s success, for example) but lying is a no-no.
  • Dissing past / existing employers. There are a lot of crappy employers out there, and a lot of crappy jobs, but spending 30 minutes complaining about them isn’t the right approach. It’s better to be more politically correct, tease out the positives (you must have learned something, met someone, had your eyes opened up to different ways of doing things) and move on. Badmouthing people isn’t going to win you points with potential new employers.
  • Not having any questions. A candidate without any questions is most likely a disinterested candidate.
  • Not having done any research on the company. Unfortunately a lot of candidates barely looked at our website when we screened them, so they really didn’t know what we were about. When you’d ask, “What interests you about Standout Jobs?” and they didn’t have an answer, it was definitely a bad sign.

Once screening was complete, we’d decide if the candidate should come in for the in-person interviews. If not, we’d let them know. If so, we’d ask them to block off half a day or more and we’d explain the process to them (verbally or via email) so they’d be somewhat prepared.

3. First In-Person Interview (with me)

I typically did the first in-person interview. Similar to the screening process, I’d prepare a list of fairly standard questions (with some deviation based on the specific person I was speaking to) and we’d spend 1-1.5 hours talking. Admittedly, these conversations got too casual at times, and I caught myself talking more than listening, but generally I found them constructive. You could probably be more rigorous here than I was, and make this interview shorter.

Ultimately, I was really looking for a personal connection. Did I want to bring this person onto my startup roller coaster and ride with them? It wasn’t about being friends (while I’m friendly with many ex-employees, we didn’t spend tons of time outside of work socializing), it was about finding a mutual connection and respect. You are going into battle with them, and a lot of it will get ugly, so there better be a real sense of camaraderie and understanding.

4. Second In-Person Interview (with Fred)

Next, Fred took over and did a more technical interview with candidates (assuming we were hiring programmers). If we were hiring for other positions, Fred still did an interview but didn’t ask technical questions.

Fred had his own style for interviews. He asked theoretical questions, math questions, and worked through different problems with candidates. He also spent a good chunk of time getting to know the candidates, how they communicated, what interested them technically, etc. So it was a combination of practical (does this person know what they’re talking about) and personal. These interviews typically lasted 1 hour.

5. Third In-Person Interview (with the team)

The next interview for a candidate was with a few members of our team. We wanted to give our existing employees a chance to meet prospective employees. This usually lasted 30-60 minutes. The discussions were broad ranging, because the team loved to talk technology and they often found candidates eager to discuss technology as well. I didn’t participate in these meetings (neither did my co-founder, Fred) so I can’t really go into a lot of detail, but I do like the idea of having candidates meet the team. That’s who they’ll be working with every single day. And the truth is that a lot of employees stay at a particular job because of their co-workers, even if they don’t like the job, the company or their boss. That’s how important it is for people to really bond with their co-workers.

6. The Practical

After three in-person interviews (now 2.5-3.5 hours in), we’d move to the practical portion of the interview. We used to call this a “technical test” but it really wasn’t about testing technical capabilities. Candidates coming in for an interview knew there’d be a practical component, but prior to the interview we didn’t provide any additional detail, except perhaps, “we’ll get you to code something.”

Note: This is for developers and technical people, but we did run others (like customer support people) through a practical exercise as well. I’ll share some details on that later.

Here’s how we designed the practical “test”:

(i) The Setup. Fred and I would tell the candidate that we were going to roleplay a scenario, and ask them to participate. At some points during the practical we’d be roleplaying, at other times we’d step out of that and review things. We’d explain the general steps to them – roleplay the initial scenario (where we’re a client asking the candidate to build an app), they’d design the app on paper and present it (partially roleplayed), they’d go build the app (not roleplayed), and they’d present the app after (partially roleplayed.)

(ii) The Scenario. Now it was time for a bit of roleplay. Fred and I would pretend we want the candidate to build a photo sharing application for our university dorm. We picked this application for a number of reasons:

  1. Everyone understood what a photo sharing app was and generally how it would work (we weren’t trying to invent something radically new here)
  2. It was a simple enough application, but it had a bunch of important “web” elements (login, user accounts, content management, etc.)
  3. People felt like they could build it very easily, which proved interesting later on

Fred and I would tell the candidate, “We want a photo sharing app for our friends in our university dorm,” but wouldn’t provide a lot of details. It was up to the candidate to ask questions, get information from us and propose ideas. Imagine a scenario where you’re doing work-for-hire with a “less than brilliant or clear” customer. That was the scenario.

Ultimately we were testing communication skills and critical thinking. What questions did the candidate ask? What assumptions did they make? How did they approach communication with us as a client? Did they embrace the experience or not?

We learned a lot during this phase.

It was amazing to see how some people dove into the scenario and started designing UIs on paper with us, while others didn’t ask very much at all and said, “OK, I’ll go build a photo sharing app.” In some cases we had to prompt candidates with specific cues: “Um, you didn’t ask us about privacy.” Or Fred and I would start talking to each other, “Do we want everyone to have access? Should people be able to delete photos? What happens if someone starts posting too much junk?” Good point, good point…then we’d wait to see if the candidate jumped in.

We were also looking to see how people prioritized the steps and things they could do, and how they communicated that to us. Some people were great at negotiating: “So is A or B more important? A is easier, but B might do more of what you want.” Some were great at brainstorming new ideas and proposing stuff, which we loved (as “clients” and employers!)

Once we were finished with this process (~30 minutes), we’d break out of roleplaying and move to the next step.

(iii) The Design. At this point we’d ask the candidate to design the system on paper. They’d often ask, “What do you want to see?” And we’d respond, “It’s up to you. What do you think would be helpful for explaining how you’re going to build this for us [as the roleplayed customer], and what do you think would be useful for you as the developer?”

We saw all kinds of things here, from UML to UI mockups. Some people had reams of paper and they’d walk us through the entire application they were proposing to build (soliciting feedback the entire way), others had one or two sheets. It was very interesting to see how people think, structure things, and again, how well they communicate.

Candidates had 30 minutes. At the end we’d jump back into roleplaying so they could present their work; to help us (as “clients”) understand what they were going to do. We’d come to a conclusion with them on what the deliverable should be. Then we’d move out of our roleplayed roles and ask them about their approach, why they did certain things, etc.

Before they could jump into coding, we’d ask the following question:

“OK, so given the scope of the project, how much of it can you get done in 2 hours?”

(Sometimes we only did 1.5 hours depending on time constraints.)

This was one of the most interesting parts of the interview.

Universally, candidates would say that they could get all of it, or nearly all of it done. We’d dive into specifics and they’d give us a list of deliverables they were prepared to commit to. We knew they couldn’t complete all the work, but it was a “just small enough project” to give people confidence that they could. It set the tone: This is going to be hard, stressful and intense, but go for it! Most candidates were eager for the challenge.

Three points:

  • Initially we were giving candidates a clean computer. It was almost completely empty, and they could set it up as they saw fit. The idea was to see how well they could stay focused on the task at hand versus setting up the machine to their perfect parameters. It made the situation a bit more challenging. After a couple interviews we stopped doing this. I don’t think it really provided us with a lot of valuable information about candidates, and the task was hard enough as it is.
  • We always offered people more time if they wanted it, usually 30-60 minutes. Most people took us up on that offer.
  • We always allowed people to ask us questions (either as the clients in the roleplay scenario) or not. If they had technical questions, or got stuck somewhere, they could come to us and talk. Very few did.

No one ever finished.

We knew this would be the case; candidates did not. And what came out of this experience with candidates was always interesting. Occasionally, a candidate would get almost nothing done, spending all their time setting up the clean computer, or just staring at the screen paralyzed. Others quit halfway through. In one particular case, the candidate admitted to us that he just wasn’t ready for a job at an early stage startup (there were personal things going on his life that were just too distracting) and so he bowed out. He was a very good developer, but the timing just wasn’t right. And that’s OK. He’s gone on to be very successful.

7. The Post-Practical

Once the practical was finished, Fred and I would get back together with the candidate and review his or her work. They’d show us a demo of what they’d built (if anything) and talk through their approach, decisions they made, and the places where they got stuck. It was always fascinating to see where people put their energy and limited time: some made sure it looked good even if the functionality was shallow, some wanted to get all the plumbing in place, and others got caught on specific (often minor) details and couldn’t adapt.

We always asked them, “Now that you’ve gone through the process, where did you underestimate the effort / work required?” Again, it was fascinating and instructive to hear people’s answers. We didn’t go back into roleplaying mode at this point, but it was important to see how candidates reflected on the conversations we had in those scenarios. How were they trying to address client needs? How did they shift on-the-fly, while keeping in mind how they would explain their decisions to the client? What assumptions did they make that they could then properly communicate?

We were always looking for clarity and adaptability. How they worked under pressure was certainly part of it, but not the only thing that mattered to us; after all, this was a fake scenario. Their ability to work in this scenario wasn’t completely indicative of their ability to work on a deadline, although I believe it was a decent proxy. You also got a sense for who had that special extra gear; the few people with the determination to plough through, take chances and be capable of clearly and confidently explain what they did after the fact.

Once the post-practical was done, the interview was over. All told, the candidate had now spent 5.5-7.5 hours with us. It was a gruelling, but effective process.

Conclusions

Hiring the right people is extremely hard. If you don’t take it seriously and put the effort into it, you’ll end up hiring sub-par people that don’t have the necessary talent or cultural fit. And as we discovered, timing is important as well; if someone isn’t ready to make the commitment needed, it’s better to find out beforehand.

The fact is you will never be 100% successful at hiring – some hires just won’t work out. There could be any number of reasons (most of which are out of your control), but you can reduce your risk significantly with a thorough interview process.

Here are some final thoughts about our process:

  • Very few candidates disliked the process. I remember a few of them thought the roleplaying was silly, and some of them couldn’t understand why we were bothering with the whole thing, but most of them genuinely appreciated the effort. Even those that fell apart during the process understood why we were going through it in the first place. And many of the candidates were more interested in working for us because we put in so much effort.
  • The interview process was the best way of presenting our company’s culture. Candidates got to meet most of the team, talk to them, learn about them, and then they were put through the gauntlet, and came out the other side with an appreciation for the intensity of the company.
  • Technical ability is fairly easy to measure. Communication skills, adaptiveness, eagerness, heart, intensity and culture are much harder to measure and understand. We already knew (or were fairly certain) about a candidate’s technical abilities before going through the practical, but we didn’t know if they’d be the right coder for us.
  • As I’ve said above, our process is just one way of doing things. I think it was effective, but it won’t work for every company. Those that performed the best throughout the interview were definitely some of our best employees. I suppose that could be a fluke, but I don’t think so. We filtered a lot of people out, and I know (now looking back with the power of hindsight) that we made the right decisions.

We also used practical tests when hiring for other positions other than developers. For example, we ran customer support candidates through a test (although a simpler one than above.) Candidates applying for a customer support role went through the same in-person interviews (with me, Fred and other employees). They would then spend 5-10 minutes with one of our customer support people. Our employee would walk candidates through our website and talk through our core benefits, as well as some of the frequently asked questions we received from customers. Candidates then had 5-10 minutes to prepare and were asked to present what they’d learned back to us. Watching a candidate present on the benefits of Standout Jobs was very helpful in assessing their communication skills, personality and ability to adapt quickly.

Some candidates had done some research in advance, and they often performed better because they were prepared. That was just fine with me. Customer support candidates were extremely surprised when we told them we’d go through a practical assessment, but it was definitely helpful for us.

Recruiting successfully is crazy hard.

The pool of talent is always smaller than you’d like, there are always competing interests, and it takes a significant time commitment to recruit consistently and well. When interesting candidates do come knocking, there’s a temptation to hire them quickly and see how it goes. I’ve done that before, and it usually didn’t work out. You get lucky sometimes, but you’re taking a big risk; not just with the one individual you’re hiring, but with the effectiveness and culture of the entire company. There’s merit in the saying, “hire slow, fire fast.” Don’t be too slow about it (or you’ll miss people), but you do need to be extremely deliberate and thoughtful about the process. Put the time into a recruitment and interview process as early as possible in your company’s existence, measure the results, learn and iterate as you go along.

April 2, 2012 Posted in Recruiting by

  • http://twitter.com/alainwong Alain Wong

    Recruiting top talent is probably the biggest challenge a company faces. That’s why VCs invest in people, not just ideas. From picking your co-founder to picking your teammates, these decisions have a huge impact on culture and how a company comes together moving forward.

    Great blog post, Ben.

  • http://www.addvalue.com.au/ conference gifts

    when it comes to recruiting and hiring new employee a lot of things should be considered. You need to make a point of who you are looking for. At the same time this acts as a filter of those applicants who will just apply and waste your time in interviewing. You must be specific on the requirements you are needing. And also a person who as a lot of experience and great portfolio would be the best candidate to get hired. 

  • Pingback: A Great Interview Process for Hiring Great Talent | SGEye Eyes the World

  • http://twitter.com/jbondc Jonathan

    Great insight. The “Scenario” is most definitely a great way to see if the candidate will fit in your team. HR in my opinion is more of an art than a science, your “best candidate formula” keeps changing and so should your methodology. As far as filtering technical talent goes, “side projects” seem real important to me, I’m not so sure about spelling mistakes. Another great post Ben!

  • http://www.venturegrit.com/ Andy Jagoe

    Ben, I came across your blog through Ash Maurya’s Running Lean and am enjoying your writing. Having spent 15 years starting businesses, I could not agree more with your conclusions. While it is great that there are so many more tools now to help startups (Lean Startup, Lean Canvas, Customer Development, AngelList), it seems like the people side of a startup has changed little since I did my first startup in 1996. If anything, new developments in hiring/recruiting seem like incremental improvements to the old way of doing things.

    I’m intrigued by your process–especially The Practical.

    Do you think it might have been valuable if a high quality third party service did a set of standard interviews with the candidate (like 30-45 min resume walk-through, 45 min case, 45 min cognitive/analytical reasoning, 45 min communications/leadership/team), captured them all on video and provided you with easy to parse transcriptions and a “candidate report” so you and your team could “send out the deck” in advance of doing your first round of interviews? The benefit would be reducing the time your team needs to spend on standard/generic elements so you can start your interview with advance knowledge of where a candidate is strong/weak and can focus specifically on the performance metrics required for your specific job. I would think you probably still would want to do The Practical, but maybe spend less time phone screening and increase the hit rate on the people you do bring in?

    Not sure if it would be better to have a high quality interview service available on demand at the company’s request (a la usertesting.com) or for candidates to have a trusted third party do these interviews and provide them to potential employers in advance to reduce the friction and information asymmetries currently in matching employees and employers.

  • http://www.instigatorblog.com/ Benjamin Yoskovitz

    Thanks for the comment. I’m not sure re: the service you’re proposing. On the one hand I like the idea of having candidates pre-filtered, on the other hand, I like to be as close to candidates as possible (just as I like to be close to customers, and don’t want to outsource the responsibility of acquiring them, at least early on in the process.)

    I’m not sure we ever received enough applications at Standout to really justify paying in advance for any form of screening, but I wouldn’t be opposed to trying something to see how it works. If I felt the quality of candidates coming through was great, and I knew there weren’t any “false positives” (or very few) in the group that I didn’t get to see…

    But outsourcing (even early parts) of recruiting is tantamount to outsourcing your brand, so it has to be done very very carefully.

  • http://www.instigatorblog.com/ Benjamin Yoskovitz

    Jonathan – people may be offended by filtering on spelling mistakes (and I can’t say it was an absolute filter) but if a candidate can’t put in a bit of time to make sure the quality of their application is high, it’s a bad sign. And the reality is that most companies are getting a lot of resumes (even small companies) and most are bad. So you need those super early filters. Sure, you might lose out on some people, but without those fairly cut and dry early filters you’ll waste too much time.

  • http://www.venturegrit.com/ Andy Jagoe

    Appreciate your perspective and agree you have to be very careful with your recruiting process. Maybe outsourcing is the wrong word.

    How did you typically source candidates? I’ve found that early stage companies most often rely on the personal networks of the management team. This works well when the management team is experienced and less well when the founders are very young or just out of college.

    At some point, all personal networks reach their limit and something else must be done. Once a company gets product/market fit and goes into growth mode and is trying to hire 3 people a week things get much more complicated.

    It feels like there are some major inefficiencies in the job market. For example, it’s expensive for job seekers to interview with more than a few companies (in terms of time and opportunity cost) and the early stage of the hiring process (resume review, functional assessment, etc) needs to be repeated at each company. Same issue in reverse for companies…they have to assess each candidate from the ground up.

    I wonder if increasing the amount of up front perspective a company has on a candidate before starting its recruiting process doesn’t significantly benefit everyone.

  • http://www.instigatorblog.com/ Benjamin Yoskovitz

    Andy – it might. And there are more companies out there trying to establish “scores” of various kinds for people (including developers) that may provide some form of benchmark.

    Recruiters are also in a position to do some vetting up-front. That’s a big part of their job, and I don’t see them going away any time soon (if at all).

    Personal networks do run out, there’s no question about it. And driving a high quality funnel is always one of the hardest tasks, but I don’t think there are easy ways of doing it. It’s the same like saying, “Generating leads for my company isn’t easy, I want a magic way of just having tons of *quality* leads show up at my door.” Doesn’t work that way.

    I’m not sure we should even think about the top of the funnel as inefficient, as much as it’s a differentiator for those companies that put the proper effort in to attract the highest quality people (just like some companies do a better, more comprehensive job of generating leads.)

  • http://www.venturegrit.com/ Andy Jagoe

    I value your thoughts and engagement on this, Ben. Thanks.

    I’ve been spending a lot of time lately talking to startups about how hard it is to recruit and retain great people and everyone seems to agree it is a major problem.

    I think all steps of the typical process could use improvement–and more education/resources on best practices wouldn’t hurt either. Far too many startups seem to be “winging it” and getting results more like a coin toss on quality.

    I wonder if there might be some analog to what has happened with “social networking”…

    It’s something people have always done, but it used to be via email, phone, etc (ostensibly “push”) before Facebook and Twitter (“pull”). There was limited serendipity and abundance mentality at work in the old model.

    Is today’s job searching / recruiting process like “push” and should it be re-conceived as “pull”?

  • http://www.printingfairy.com/carbonless-forms-printing carbonless forms

    Nowadays the most difficult job is recruiting in any company or office because competition is very high to seek a job therefore, you have receive hundreds of CVs. First challenge is call interview to right persons. However, your all techniques are very effective to evaluate any person for any job.

  • mdr

    Great blog!!! Thanks
    ———–
    Fassaaditood ja fassaadide remont. 
    Fassaaditood ja renoveerimine.
    Fassaaditööd, Maalritööd, Ehitustööd, Trepikodade remont.

  • http://web-promo.com.ua/ webpromo

    I really enjoyed this article.

  • Pingback: » Baseline Scenario: What I Didn’t Get To Say On Bill Maher Last Night

  • Pingback: Managing Startups: Best Posts of 2012 |

  • Pingback: Managing Startups: Best Posts of 2012 (guest post)

  • Pingback: Managing Startups: Best Posts of 2012 | andrewbruce01

  • Pingback: Managing Startups: Best Posts of 2012 | njdrmtrade.com

Ben Yoskovitz
I'm VP Product at GoInstant (acq. by Salesforce).

I'm also a Founding Partner at Year One Labs, an early stage accelerator in Montreal. Previously I founded Standout Jobs (and sold it).

My bio »

or

Follow me on Twitter

Get updates and special content
When I publish new content, get it directly in your inbox. Subscribers will get special stuff as well not available on the blog (but I promise it will be infrequent + high quality.)
Get the Lean Analytics Book!
Awesome Jobs
Check out the job opportunities at my portfolio companies.
Startup & Investor Resources
Find Stuff
My Photos