21st February 2010

The Joel Test no.11

The application process is there for two reasons. One, for the employer to get to know if the employee is a good fit. And second for the employee to get to know if the employer is a good fit.

With any role a new hire needs to address a number of 'transferable skills' questions such as 'can this person communicate clearly' and 'can this person work with others'. However, on top of this each role has specific skillset requirements. It is these requirements that need bespoke methods of identification. The question is, what methods should we use?

The developer application process

Developer application processes can vary greatly. However their focus must be 'is this person able to program at the level we require?'.

Here's just a few examples of the methods I have encountered myself:

---

The Joel Test #11

In his article on interviewing for new developersJoel Spolsky identifies two key aspects to look for in new hires. That they are smart and that they can get things done.

The Joel test (a means of measuring the quality of a software team) provides a pretty compelling argument as to why candidates should write code as part of the application process.

11. Do new candidates write code during their interview?

    Would you hire a magician without asking them to show you some magic tricks? Of course not...

    ...Yet, every day, programmers are hired on the basis of an impressive resumé or because the interviewer enjoyed chatting with them. Or they are asked trivia questions ("what's the difference between CreateDialog() and DialogBox()?") which could be answered by looking at the documentation. You don't care if they have memorized thousands of trivia about programming, you care if they are able to produce code. Or, even worse, they are asked "AHA!" questions: the kind of questions that seem easy when you know the answer, but if you don't know the answer, they are impossible.

    Please, just stop doing this. Do whatever you want during interviews, but make the candidate write some code.

Joel Spolsky - http://www.joelonsoftware.com/

The big question

So how do each of the approaches above answer the big question "Can this person actually develop software with us / the way we do / the way we intend to in the foreseeable future."

If we take a look back at some of the variants only a few actually ask for code:

Can you develop?
Old school	A technical representative will quiz you	As part of an interview I was once asked "So are you comfortable writing Javascript functions?". I replied "Yes". Needless to say the only party that learnt something here was me. And I only learnt that the technical interviewer either wasn't very good at interviewing, or wasn't very good in general. The problem is, what can you ask verbally that actually proves competence?
Mini Contractor	At the end of the day the employer sees your code and assesses how much help you needed from others	In this case the company needs some work for you to do that you can get up to speed with quickly (rather than wasting time teaching you) and is not under NDA. This approach can be useful to see how the applicant works in your environment but has extreme time costs for both parties, particularly as their presence may disrupt 'non-interviewing' members of staff.
Blue Chip (legendary logic tests)	Answer a thought experiment/riddle	I've not encountered this myself during interview - the concept is almost mythical. Although logic questions may, like apptitude tests, give an insight into how 'smart' someone is it is nowhere near as directly related to programming as getting actual code samples to be done/provided.
Right here right now	Produce a code sample to a particular problem in a short time frame while being monitored	Requires a relatively 'trivial' task that can be completed in a short amount of time. Being watched adds abnormal pressure but maintains control over the entire process and removes the ability to cheat the system. If the situation doesn't involve a computer (code on white board) it is quite different to the expected role.
Show us the code	Produce a code sample to a specific set task prior to coming in for interview	Can be 'gamed' (completed by someone else and/or with abnormal effort - 'yeh it only took 2 hours *cough*every minute since you gave me the task to now*cough*'). Doesn't give you a direct insight into the process only the process as viewed by the participant retrospectively.

If we agree that completing real code in some way is the only way to identify the programming ability of a prospective candidate we rule out all but the last two approaches (with 'right here right now' and the 'mini contractor' being roughly equivalent but for the probable time frame involved). And then it becomes a case of which is better?

What do we use?

At Headscape we use a process similar to 'Show us the code'. After a CV/Covering letter and phone interview sift we ask applicants to complete a mini project (based on a spec provided) and submit it to us as the next stage. This is for a number of reasons:

• Asking programming questions verbally is inherently difficult. It can be tricky to avoid trivia style questions and is not particularly representative of day to day work.
• Asking someone to code on a white board without the luxury of intellisense or google will filter out a chunk of applicants however it doesn't necessarily provide you with a full picture of them. Such testing relies on being able to generalise the success of creating a small algorithm to overall success on the job and doesn't account for 'off days'.
• The actual output we receive is not so much 'marked' with a pass or fail (although it can be used to assess whether an applicant is at the level we require). The main benefit from our point of view is twofold... A useful effect of asking for new code is that the commitment required acts as a 'desire filter' on the applicants behalf. Applicants sending mailshot CV's to multiple jobs may be put off continuing down a path that requires more commitment from them - and this is a good thing. As an employer we want employees who really want to be with us rather than just someone who needs a job.
• By asking them to complete new work in their own time we do not force them to have an extended stay at our offices for testing nor do we 'suprise' them with an algorithm question on the day. The other benefit therefore is that the technical part of the interview can revolve around the task. "How did you go about it?", "What did you find difficult?", "What did you need to research?", "What would you improve?", "How does this solution account for extensibility?", "Did you reuse some of your code/third parties?", "Did you satisfy the spec". All of these questions are only available if we have an extended code sample. And by setting each candidate the same mini-project it is more comparable than discussing their previous most recent project. It also gives the candidate scope to impress us by going the extra mile. For example I have previously been asked to write an IsPalindrome function on a white board. Comparing that to being given a mini project there is clearly more scope for me to show my desire (by adding extra functionality), show my planning skills (by building in extensibility), show my decision making (by using/not using third parties or previous code). All things that would be relevant to a developer role but are difficult to assess when reviewing single algorithms.

The argument is essentially the same as the coursework vs exams debate from education. Although coursework is more open to cheating and less able to be tightly controlled it is actually more like working life.

Discussing the project can easily identify if the candidate has had some 'assistance' from a third party. The main shortcoming then is not knowing exactly how much of the time between phone interview and in-person interview the candidate spent creating the output they provided. In my view this shortcoming is balanced by being able to have an in-depth discussion of the project. The speed of response and passion in their discussion surrounding this should provide pointers to how 'smart' they are.

Any suspicions or a large amount of applicants can be filtered by doing an initial technical interview over the phone regarding the project if necessary.

As an applicant

So thats all well and good as an employer. Just sit back and fire out homework for your applicants. But as an applicant is it merely another hoop to jump through?

When I've been an applicant myself I actually prefer to do code in some way. Not because I like to be tested or even just because I like to write code but because it allows me scope to get a feel for the employer.

• Do I get a glazed look on the face of the technical interviewer when I discuss some aspect of the project?
• Did they generally agree with my approach or are they entirely different?
• Did they notice the bit I rushed?
• Did they offer a different approach I hadn't considered? (can I learn from them?)
• Did they smugly point out a minute flaw to establish their dominance
• Did the back and forth about my code at times slip out of interview mode and become more like a discussion with colleagues?

All of these points allow me to guage my half of the application bargain "Is the employer a good fit for me?".

Reservations?

While agreeing with Joel that Smart does not mean 'knows the answer to trivia questions' I also understand that by asking for a mini-project to be completed we are asking for a reasonable amount of commitment from the applicant. However we both benefit from this in the richness of the interview process that follows.

Something we have also considered at Headscape is changing the 'test' to a situation where we provide a solution to the applicant and ask them to extend it in some way. This would be more realistic (most projects are continuations of old ones, or built on a base codebase rather than clean slates) and would also allow us to ask the applicant their thoughts on the project we provided. Would they have done it this way?

Allowing them to critique us not only assists their desire to judge if we are a good fit for them but allows us to see their analytical and critical skills.

In Closing

Our interviews involve key personal (representatives from each team PMs, Directors, Developers) and we must assess the standard job interview issues. However when it comes to development some form of coding is essential. In our case we favour the 'coursework' style. It may be that a mix of both is appropriate. With all things 'it depends'. If you expect 100 applicants to get to the testing phase then assessing those projects will be a huge undertaking. But relying on only a small in-person algorithm test can be difficult to generalise potential success from. Mixing the two, by bringing in a candidate and asking them to code all day may seem like an ideal compromise but requires a single fixed large timeframe commitment from both parties. We also all know how different it is to code with someone watching over your shoulder.

At all times we need to consider the benefit for both us as employers and applicants as potential employees through the process. We are not the only ones making a big decision. If we can somehow forge the process to allow the applicant to test us we will inevitably have a better new hire in the long run.