By now everyone has heard whether or not their preproposal at NSF in DEB or IOS has been invited for a full proposal. About three quarters of us are disappointed, perhaps fewer if people wrote more than one, so had better odds. What should you do if you were not invited to go ahead? What I have to say is not specific to a narrow area or the USA but is generally relevant, using this system as an example.
First of all, remind yourself of what a great scientist, teacher, and mentor you are. Do not fall into the trap of low expectations for yourself. This is really hard to do when you have had a punch in the gut from NSF, particularly as you face the real consequences for funding your lab, getting tenure, and retaining respect at your institution. Don’t let them take away your self esteem. Don’t let them make you feel terrible. Figure out somewhere to get satisfaction anyway. This often comes best from remembering to give. Find something to do for someone, in science, teaching, or otherwise.
But ultimately you do have to look at that proposal and the comments. Ideally these comments will help you write a more effective preproposal next January, or for another program. They might like some of your plans but not others. They might have discovered a methodological flaw. They might think your sample size is too small, or your approach inadequate. Read these comments really carefully. Try to see your proposal through someone else’s eyes. Get someone to read it outside your group. Pretend it isn’t your proposal but someone else’s whom you are mentoring. Set defensiveness aside.
But, you say, you did not get a real review. There were no substantive comments. The referees did not like the proposal, clearly, but only said things like it wasn’t cohesive, there was too much information on the system, or too little, too many hypotheses, or too few, the aims were too tightly connected, or unconnected. Maybe you didn’t even get the full three reivews that NSF says preproposals should get. You got scientific word salad. The reviewers did not want to bother to tell you exactly what was wrong with your proposal. They didn’t like it. They liked others better. They didn’t bother to do the hard job of careful comments, but it is clear your proposal did not excite them.
Do they owe you a real review? Are they required to actually grapple with the ideas in your proposal and point out where they are strong and where weak? Do they have to find a flaw in your thinking to deny you? Are they supposed in any way to help you do better science?
The sad truth is that they are not. There is nothing in the whole process that says they have to actually discuss your ideas. They may be very distant from your field. All they are required to do is pick a quarter of the proposals to go forward and write some word salad for the rest. You do not even get to have three reviews necessarily. But remember, part of the reason they went to preproposals is the difficulty in getting careful reviews.
At NIH the system is different. They do not have preproposals but they triage as many as half the proposals, sending them away with no review at all. Michael Eisen just posted on Twitter that the work that got his HHMI renewed was not discussed at NIH, , @mbeisen. So, no, you do not get your work discussed there necessarily either.
Is this fair? Does the system work? I would have to say that no, it is not fair, but the system works. There is no time to be fair. The system has checks at NSF in the form of Committees of Visitors, who oversee the process after the fact. Does that make it fair? What does fair even mean with such low funding rates? This question and the general one of fairness should be addressed most explicitly with respect to systematic bias towards those underrepresented in science. This has been addressed and discussed in places like the DEB blog.
What should you do? First, remember to be the careful reviewer when it is your turn to do that. On your own proposal, call your program officer. See if she has any more substantive advice. Volunteer to serve on a panel if you have not done this, so you can see how the process works. Get colleagues to read your proposal, colleagues willing to do the hard work of being very frank and critical. Get examples of successful preproposals from friends.
One thing the preproposal process has introduced is a stage with no expert ad hoc reviews, just a panel that has to get through a lot of proposals. So the sad truth is they did not like yours and they are not the ones to tell you why.