Tests For Programmers

A solution that's gaining traction is not to interview at all without seeing some code that the candidate claims they wrote, and then as the first stage of an interview, to get the candidate to write some code.

Some interviewers even require that the candidate write code on paper, or on a whiteboard, without using an editor, or having the web to look up details of library routines, etc.

Others claim that this is grossly unfair and unrealistic. In any job, they say, access to such facilities is the norm, and preventing such access is giving a skewed view of the candidate's abilities.

I'm going to leave that to another time. Instead I'll give here the results of an ad hoc, informally specified, unscientific and statistically invalid test.

When the question came up on Hacker News in February 2010 someone said:

So in response I ran a small test. People asked by email, and I sent back a "specification" for a routine. Although not exactly the sort of thing that would turn up every day in a working programmer's life, it was intended to be a small task of the type that I expect a programmer to be able to do without thinking too hard.

Hamming in "You and Your Research" wrote this: I notice that if you have the door to your office closed, you get more work done today and tomorrow, and you are more productive than most. But 10 years later somehow you don't quite know what problems are worth working on; all the hard work you do is sort of tangential in importance. He who works with the door open gets all kinds of interruptions, but he also occasionally gets clues as to what the world is and what might be important. Now I cannot prove the cause and effect sequence because you might say, "The closed door is symbolic of a closed mind." I don't know. But I can say there is a pretty good correlation between those who work with the doors open and those who ultimately do important things, although people who work with doors closed often work harder. Somehow they seem to work on slightly the wrong thing ­ not much, but enough that they miss fame. A similar thing happens in groups of programmers. Those who can work in splendid isolation can do fantastic work, but it sometimes seems not quite to be the right thing ...

Let me say here that when I do this for real I don't expect fully debugged, flawless, efficient, working code. What I want is to see some code that looks vaguely right. It's the start of the interview process. What the candidate writes is the beginning of the discussion about coding, and if it's not right then we can work on it together, and in that way I discover not only if they have a clue, but also whether I think we can work together. In some places a programmer can work in splendid isolation, given detailed specs, producing polished code for others to test, and never having to interact with others. Our shop isn't like that, and "fitting in" is an important aspect.

We also program in C++, Python, PHP, AWK, C, and a smattering of other, less mainstream languages. Someone's specific knowledge of a language is less important than their ability to work through a problem.

So here's what I asked people:

Write a C routine with the following prototype: void condense_by_removing( char *z_terminated , char char_to_remove ); Your routine should modify the given zero-terminated string in place, removing all instances of the given char.

I wasn't asking for a critique of the problem, although some discussion of it was almost expected. I know some languages have this sort of primitive built in, but I was looking for someone who could write a loop, check termination conditions, move data around, and have confidence that their code did what it was supposed to do.

I certainly expected more thinking time than coding time, and not that much of either. The programmers who work for me did it in about 2 minutes, one of them making a small error that was quickly corrected when the code was reviewed.

So how did people get on?

I received 12 requests for the test, which I duly sent out. Of those:

• Three people never replied.
• That's OK, there are several perfectly valid reasons.

• One gave an invalid return address.
• If you're reading this and never got a reply, email me again and I'll try harder to work out your return address.

• One person doesn't know any C, but sent a routine in Python that tried to stick with the intent of the question.
  • That would earn an interview on its own.
  • Subsequent emails showed that they could do it "Pythonically" as well.
• One person didn't really understand the spec and doesn't write in C. They sent what they thought the spec meant, and provided a PHP function call that did the right thing, but didn't demonstrate the skills I was looking for. That would also have earned an interview - the effort was enough to be worth keeping to the next stage
• One routine in C was sub-optimal. It worked, but it had complexity O(n^2) and used a library routine to lump bits of memory around. To be fair, they were a self-proclaimed C++ programmer, not a C programmer, but subsequent email exchanges showed that they weren't aware of what they did.
• This doesn't of itself mean they're a bad programmer, and if there hadn't been other excellent candidates they would have been called for interview.
• It does mean they don't yet have the skills we would need. It suggests inexperience, and a lack of awareness of what goes on "under the hood."

• The remaining five were fine. Some were supplied with "main" routines for testing, some included comments, all were pretty clear, and apart from one, they all found the same solution with minor variations.