One of the things we do at meetings and at this time of year in universities is we judge posters and sometimes talks or papers. The winning poster gets a ribbon and maybe a hundred dollars. It is another line on the resumé.
This is very different from the kind of judging that goes on at NSF or NIH where proposals are discussed in detail and judged carefully on scientific merits after group discussion of the strengths and weaknesses. In poster sessions there are a lot of judges. Usually only two people see any one poster. How can this kind of process possibly end up with a winner?
It gets worse. Often the forms for judging are broken down into a lot of categories. You are supposed to give points for each, perhaps presentation, importance of project, use of statistics, or other such categories. You struggle to divide up the points. You struggle to be fair.
At the end, there is usually a simple averaging of points from different judges and the posters with the most points wins. This means two things. First, pay attention to the total points, more than to the different sub-categories. Second and most important is to use the full range of point possibilities. Give your best poster the full points on everything. Bring it down from there only slightly if you think your next few posters are also highly worthy.
If you think of yourself as a hard grader and therefore stay shy of the top marks even for the best posters, then your marks simply won’t count. They will be swamped out by other judges who used more of the range. This could be fixed if each judge’s scores were Z transformed or something like that. More simple than this would be if each judge’s top poster were noted. But these things usually don’t happen when votes are compiled, just as there is usually no or little group discussion of the science of each poster. There just isn’t time.
Does this mean you should also go down to the low end and mark some down a lot? No, because no decisions get made there. There will be very few winners, so marking down the weaker posters even a little will suffice to keep them from winning. There is no point in being discouraging to the lower scorers, though often the students do not even see the scores.
If students do get feedback, written comments are best. I think each student should get a substantive positive comment and a substantive negative (but gentle) comment, so they can feel both rewarded and see where they can improve.
There are other circumstances where a larger group judges papers or proposals with little room for substantive discussion. These suggestions hold there too. Just remember to make your top score full points, with very high points to those following close behind. This way your vote will count.