If you have a committee of 6 choosing the 4 or so biologists to interview for a single position, what would a really accurate way to do it be? I think you would look hard at the error factor. If smart, unbiased professors are doing the choosing, shouldn’t they arrive at broadly similar conclusions? What are the odds that even the first sorting of the long list into a shorter list for everyone to look at will be scientific?
There are several kinds of errors to consider. One I really worry about is bias. Are we reading women and minorities differently? Being aware of bias helps, I think.
Another error comes from reading files too quickly. You might miss something important because you have so many files to read and there is a deadline. It is important to divide up files so people don’t read too many too quickly.
A third kind of error comes from weighting anything besides scientific excellence and promise of future success on the first pass. We need to cut down the pool by considering this crucial factor alone.
The fourth kind of error comes in especially in really broad searches. It can be difficult to compare very different kinds of people. I was dumbfounded at how little our top neuroscience candidates publish, but that is apparently standard for their field. If they were in our search, we would not choose them, possibly missing the best person. Even in a more narrow search like ours for an ecologist, different fields are hard to compare. Theoretical vs. empirical foci, for example, result in different levels of publishing. Women are generally known to publish fewer but more substantive papers.
We could search much more narrowly of course, for an ecologist that studies the mites on buffalo, for example. We would have a better chance of picking the best person that does exactly this, but we would miss brilliance in nearby fields.
How do we overcome all these potential errors in choosing? We don’t. We can’t be fair. In fact, one might think it is not our job to be fair since it is so impossible. Our job is to hire an excellent scientist, colleague, and teacher. There are likely to be others even better in the pool, but not discoverable by our imperfect techniques.
Wow! That sure is sobering. How can I be so sure? Well, for starters, say we took our 200 candidates and had each person read 66 or so, choosing the top 8 from their pool. Further say two people read each folder. Shouldn’t there be broad consensus across readers? Has anyone ever studied this? I’m speaking hypothetically here, but I bet it is common for there to be very little overlap between the two readers. If there were no overlap, we would end up with 48 people to take to the next level if everyone kept to choosing just 8. I bet in most searches the number after a first cut like this is over 35, a surprising lack of consensus.
If this were science, we would go back to the beginning and say our process was not discovering what we want to discover. But we do not do so. There are a couple of reasons. First, we do not have the time. Second, maybe it is likely that the very best people each person reads will make it to the top. It will be interesting to see if the final candidates were among the few chosen by both readers. By the way, the two readers are typically randomly paired in all combinations in the best processes.
Who are we likely to miss at this stage? I think it is the people with big ideas who do not publish prolifically. Or it might be the people with big ideas and excellent work who focus, as so many do, on what they are asking rather than what they have figured out. But probably most of the people we miss are just random errors, no good reason to have chosen Hannah over Isabel.
For better or for worse, this process will narrow the field to 48 or fewer people chosen by someone and the whole committee will read those files and discuss them. Will this lessen arbitrariness? We can again see how much overlap there is when everyone is reading everything. The goal of the next step is to cut this list by half or more in a process that begins to consider fit, collegiality, and teaching.
This is the process I most often see. But there are no rules from my university as to the best way to do this. I wish we could be informed more by social scientists who might have studied this and can tell me how best to optimize outcome and fairness while not sacrificing undue faculty time.