There are only four things that you should consider when you review a proposal submitted to the National Science Foundation. You should weight them more or less equally. By that, I mean that weakness in any one of these elements should sink the proposal, at least in these times of very low funding rates. Furthermore, it means that your written evaluation should devote equal space to these four. If you are on the panel, you should devote equal discussion time to each of these four parts. These elements are: 1) the importance and creativity of the question; 2) the scientific productivity of the investigators; 3) the methodological feasibility and design of the project; and 4) the broader impacts of the project.
All too often it is the case that only the third element receives significant attention. I have sat through panel discussions more than once where attention was so focused on the minutiae of the methods that the importance of the question was entirely left out. How much should we care about a beautiful experimental design when the question is banal, already answered, a repeat of something in a neighboring system, or completely unrelated to significant theory? Don’t get me wrong. Methods matter. They matter more in some areas than others. New methods allow us to address new questions. Statistical analyses must be appropriate and carefully thought through. There must be sufficient preliminary data to convince us that the methods are possible. But the other elements are equally important. Why are they so often brushed over?
Maybe it is easier to just focus on the methods. We can judge them without too much difficulty. We don’t have to put ourselves on the line, so to speak, to judge methods. But judging NSF proposals is important. It should be done carefully. It should be done comprehensively. Perhaps it is easier to reach consensus on good methods as compared to the importance of the questions. That is exactly why it takes time to discuss the importance of the question. Perhaps it requires more background in the field to address where the question fits in the fabric of knowledge. I think science would be better off if we all spent more time thinking about the importance of our questions and if we spent more time seeking the most important questions, not the me-too questions, or the questions that only someone in love with the system could care about. So try hard to figure out how important the questions of the proposal are, and write about it in your review.
If it is challenging to judge the importance of the questions, it might be even more awkward to judge the productivity of the investigators. But it is crucially important. In this case, past performance is a good predictor of future performance. Where do these investigators publish? How prolific are they? Excellent researchers will publish some papers in the flagship journals of the field. They will write clearly in ways that get their papers into top journals, because they will make the connection between their work and big questions clear. They will be able to put their work in context by citing appropriate work, because they read broadly and deeply. Their papers will be a joy to read. Hypotheses will be clearly laid out and motivated. Data that addresses them will be clear in figures. The ramifications of the work will be discussed. The investigators will not hesitate to point out shortcomings and future directions. Look at some of the 10 papers cited in the CV of the researcher. How good are they? Where are they published? The introductions of these papers can also help with judging the importance of the questions in the proposal, if you can’t tell from the proposal itself. Yes, NSF is asking you to judge the investigators.
Don’t expect the researchers to publish only in top journals. Some studies just don’t pan out as well as they might, but are still worth publishing, for scientific and personal reasons. They may present interesting natural history that doesn’t tie easily to theory. The study may be the project of a student for whom publication is important.
You may adjust your publication expectations to career stage, though it might be true that postdocs have more time for productivity than professors. Long prior productivity is not as relevant as productivity in the few years immediately before proposal submission.
There was a time when the fourth element of an NSF proposal was so new, people could only state that their work itself was broadly relevant or that they had a female or even a member of an underrepresented minority in the lab. Now you will find energetic investigators are designing museum exhibits, setting up websites, finding teaching, and mentoring K-12 students and teachers, and many other things. It is refreshing to see the energy put into these broader impacts. After all, it is the public in general that funds our research. Why not spend a little time on education and contact at this level?
There, that’s it. Four things, not one. Think about it and plunge in.
October is when the NSF grants that made the preproposal cut-off last January will be reviewed in panels. This generally means that three panel members will read and review each proposal. If they are lucky they will have three or more reviews by experts close to the field to consult. If you are on a panel, or receive a proposal for review, please pay attention to all the elements, and judge carefully. It will be a distracting time, an odd discussion where most of the people in the room are working on other things, waiting for their proposals to come up. It is a wonderful opportunity to talk about science and scientists.
You may wonder how much this advice applies to other kinds of proposals. I would say it is generally important to judge the question, the scientist, and the methods. Some focused programs may equalize at one of those levels. Some may not have a broader impacts section. The point is to be brave and judge the whole thing, not just the balance of the methods. Have fun!