Preproposals came to DEB and IOS about three years ago to solve the problem of too many proposals and too few willing reviewers. They were also thought to solve the problem of too much investigator time wasted on writing unfunded proposals. There are probably other reasons for beginning them at that time. After all, many big programs with many investigators and institutions or other complexities have preproposals, so it wasn’t a new idea, even for NSF.
Preproposals have generated tons of controversy. Some people like them, but most seem to loathe them. I know NSF DEB and IOS have done their own careful study and that it is going to come out soon. Here are my thoughts developed from talking to many people and from personal experience. I bet you can think of additional points.
1. Preproposals are not less work, for the writer or the reader.
The idea was that a preproposal was only 5 pages and so they would not take so much time. But trying to squish your ideas into 5 pages can be more work than explaining yourself fully as anyone who has tried to write a paper for Science knows. You don’t know what to include and what to leave out. You agonize over balance between ideas, background, and experimental detail, not to mention broader impacts.
There is another way in which preproposals are more work. Two get funding you have to write two proposals with different target audiences, different lengths, and different times of year. The second time may be exciting, but about half those proposals are also declined.
2. Referees have to keep two different standards in mind, 5 page and 15 page. In many countries the whole proposal is about 5 pages and all judgements are made at that level. Then everyone gets used to that standard and has a clear idea of what level of detail to expect. Here we have two standards making it hard not to expect something from 5 pages that cannot be there.
3. Preproposals are only judged by a committee, so you are less likely to have an expert in your field read it. Sometimes I feel like no one who really knows the field of the proposal read it. This is inevitable when the preproposals are only read by a committee. The comments that come back on the preproposals can seem shallow and off the mark. You may have had a weakness in your proposal that you addressed as well as possible, yet no one noticed it. Instead the panel went after something you did not detail because it is routine for your field. Doesn’t everyone deserve a team of experts to read their proposal, or at least a level playing field?
4. Preproposal reviewers still focus on methods. The idea originally was that you would be judged just by your big ideas and you don’t need to go into detail on the methods. Taken to an extreme you could propose to do amazing and impossible things that anyone should want to fund. Of course you don’t do this. I have been discouraged by how much NSF panels at all levels focus on methods, no matter how much program officers entreat panels to do otherwise. People may feel more comfortable with challenging methods. They may feel less able to really judge what is a big idea and what is not. In a way the biggest ideas will have no audience of experts. This can be particularly true when there is no one on the panel that really knows your field.
5. If the top preproposals translate into the top proposals, we don’t need two steps. I do not know if this is true, but I bet it is. Excellence tends to be correlated across different arenas. Taking away the chance to ever get fully evaluated can really hurt struggling investigators, perhaps keeping them from performing to their optimum and only rewarding the people that already know how to do this.
6. Preproposals do not solve many problems once a year deadlines would not also solve. A once a year deadline with full proposals and a mix of panel and ad hoc reviews would keep researchers from churning proposals. It would allow early career investigators to submit twice, once regular, once CAREER. It would give everyone the thoughtful advice in comments on a full proposal. It would require greater support from the community to get all the ad hoc reviews, but it would be worth it.
7. Preproposals create another status ranking among investigators, which can be very demoralizing. The countries with the longest lifespans are those with less hierarchy, like Sweden, many studies show. While it will help your longevity to be high status (see study of Nobel laureates), isn’t what we really want longer, fulfilled lives for everyone? You get that from less ranking. We can’t get rid of the most fundamental rank, funded and unfunded (but see a soon-to-come post), but we can get rid of the shameful not asked for a full proposal category.
8. Preproposals are terrible for researcher morale. I suppose this one could have been predicted, but I did not think of it before. Somehow having your precious ideas not even be considered worthy of a full proposal is really discouraging. Any time that writing a shorter proposal might have saved is lost to gloominess about the whole scientific enterprise. It is bad enough not to get funded. It is simply horrible to have this be the result of something less than a full chance to explain yourself. There is tons of evidence of low morale. This is the biggest problem with preproposals. Say what you will about whether it should have happened. It did happen. We need to fix it.
9. What to do next? I recommend that we drop the preproposals, go to a single full proposal a year, and as a community commit to careful review of the adhoc proposals we receive. If you are not getting any, contact your program officer and let her know you are available. Change is complicated. I think this was a good idea, but it did not work. It is time to drop it.