Preproposals came to DEB and IOS about three years ago to solve the problem of too many proposals and too few willing reviewers. They were also thought to solve the problem of too much investigator time wasted on writing unfunded proposals. There are probably other reasons for beginning them at that time. After all, many big programs with many investigators and institutions or other complexities have preproposals, so it wasn’t a new idea, even for NSF.
Preproposals have generated tons of controversy. Some people like them, but most seem to loathe them. I know NSF DEB and IOS have done their own careful study and that it is going to come out soon. Here are my thoughts developed from talking to many people and from personal experience. I bet you can think of additional points.
1. Preproposals are not less work, for the writer or the reader.
The idea was that a preproposal was only 5 pages and so they would not take so much time. But trying to squish your ideas into 5 pages can be more work than explaining yourself fully as anyone who has tried to write a paper for Science knows. You don’t know what to include and what to leave out. You agonize over balance between ideas, background, and experimental detail, not to mention broader impacts.
There is another way in which preproposals are more work. Two get funding you have to write two proposals with different target audiences, different lengths, and different times of year. The second time may be exciting, but about half those proposals are also declined.
2. Referees have to keep two different standards in mind, 5 page and 15 page. In many countries the whole proposal is about 5 pages and all judgements are made at that level. Then everyone gets used to that standard and has a clear idea of what level of detail to expect. Here we have two standards making it hard not to expect something from 5 pages that cannot be there.
3. Preproposals are only judged by a committee, so you are less likely to have an expert in your field read it. Sometimes I feel like no one who really knows the field of the proposal read it. This is inevitable when the preproposals are only read by a committee. The comments that come back on the preproposals can seem shallow and off the mark. You may have had a weakness in your proposal that you addressed as well as possible, yet no one noticed it. Instead the panel went after something you did not detail because it is routine for your field. Doesn’t everyone deserve a team of experts to read their proposal, or at least a level playing field?
4. Preproposal reviewers still focus on methods. The idea originally was that you would be judged just by your big ideas and you don’t need to go into detail on the methods. Taken to an extreme you could propose to do amazing and impossible things that anyone should want to fund. Of course you don’t do this. I have been discouraged by how much NSF panels at all levels focus on methods, no matter how much program officers entreat panels to do otherwise. People may feel more comfortable with challenging methods. They may feel less able to really judge what is a big idea and what is not. In a way the biggest ideas will have no audience of experts. This can be particularly true when there is no one on the panel that really knows your field.
5. If the top preproposals translate into the top proposals, we don’t need two steps. I do not know if this is true, but I bet it is. Excellence tends to be correlated across different arenas. Taking away the chance to ever get fully evaluated can really hurt struggling investigators, perhaps keeping them from performing to their optimum and only rewarding the people that already know how to do this.
6. Preproposals do not solve many problems once a year deadlines would not also solve. A once a year deadline with full proposals and a mix of panel and ad hoc reviews would keep researchers from churning proposals. It would allow early career investigators to submit twice, once regular, once CAREER. It would give everyone the thoughtful advice in comments on a full proposal. It would require greater support from the community to get all the ad hoc reviews, but it would be worth it.
7. Preproposals create another status ranking among investigators, which can be very demoralizing. The countries with the longest lifespans are those with less hierarchy, like Sweden, many studies show. While it will help your longevity to be high status (see study of Nobel laureates), isn’t what we really want longer, fulfilled lives for everyone? You get that from less ranking. We can’t get rid of the most fundamental rank, funded and unfunded (but see a soon-to-come post), but we can get rid of the shameful not asked for a full proposal category.
8. Preproposals are terrible for researcher morale. I suppose this one could have been predicted, but I did not think of it before. Somehow having your precious ideas not even be considered worthy of a full proposal is really discouraging. Any time that writing a shorter proposal might have saved is lost to gloominess about the whole scientific enterprise. It is bad enough not to get funded. It is simply horrible to have this be the result of something less than a full chance to explain yourself. There is tons of evidence of low morale. This is the biggest problem with preproposals. Say what you will about whether it should have happened. It did happen. We need to fix it.
9. What to do next? I recommend that we drop the preproposals, go to a single full proposal a year, and as a community commit to careful review of the adhoc proposals we receive. If you are not getting any, contact your program officer and let her know you are available. Change is complicated. I think this was a good idea, but it did not work. It is time to drop it.
Sociobiology 
I am curious if there was any discussion of limiting how many NSF grants an awardee can hold concurrently?
Actually, we did not talk about anything that concrete. You can see from the tweets it was very much about broader impacts. There were 11 newcomers I think, as well as a new committee head, so I think we were figuring out what we should be doing. I think we need to concentrate on both process and content. Is the process fair and efficient? Are new areas getting the funding they deserve?
I thought one point of the preproposals was to increase the chance to get qualified external reviewers to take a look at a full proposal. in previous times of 6-month submission cycles, it was often difficult for NSF to get qualified external reviewers, and the hope was that external reviewers are more likely to take the time for review if full proposals are sent out for review only once every year. Does NSF have any stats on that now, that it takes them less time/tries to commit external reviewers?
I figure that two preproposals equals about the same work as one full proposal, so with everyone trying to submit at least two preproposals, work load was probably not reduced overall (and snowboarding over Christmas break is as ruined as it always was).
If workload on faculty was really what this was about, then why not just make all proposals 5-pagers, as they are in many other countries? It would change the bifurcation of reviewer culture cited above, and would actually provide a benefit to all involved. Your point #5 is indeed well taken!
I agree with this. Do away with the 15 page proposals. I’ve reviewed countless proposals for foreign granting agencies, served on a German DFG panel (where proposals were much shorter), and have served several times on NSF DDIG panels. In all cases, it is easy to separate the cream from the skim. Why go to the effort on both PI and reviewers’ parts to write/read long proposals? Hopefully the NSF has data now on the correlation between ratings for preproposals and on the subsequent full proposals that might show whether 5-page proposals are good predictors of success in the traditional 15-page proposal.
I have found it quite difficult to separate the cream from the skim even in 15 page proposals, particularly for innovating ideas that are so far ahead of everyone else such that noonone on the panel fully understood the landscape. I think NSF aims to fund such innovating ideas, rather than routine cream that can be separated easily from routine skim.
Perhaps also relevant to the discussion: paper just out in BioScience on results from a survey by NSF gauging response to the change -> http://bioscience.oxfordjournals.org/content/64/9/837.short
Another issue the author didn’t point out is how this new system affects post-doctoral researchers. Many post-docs don’t have PI status and so are unable to submit proposals as the principal investigator. This is problematic because post-docs have a hard time getting hired at research institutions without proven experience garnering grant funding. Furthermore, a lot of literature shows that many of our best and most productive ideas come relatively early in the career process, before mentoring students and administrative duties take a larger share of our time and attention. So its a bit of a catch 22, you can’t apply for grants without PI status, but its hard to get PI status without having received grants.
Historically, a post-doc could go to a PI and ask them to act as PI on a proposal that the post-doc writes. In essence, the PI is acting as a “backer”, offering institutional support to allow the young researcher to float their big idea to the larger academic community. Before the pre-proposal process this usually came with little cost for the PI. The post-doc pretty much did most of the work.
Now, with a limit on the number of proposals that PI’s can be on as PI, I’ve noticed that folks are much more reluctant to “host” a post-doc’s idea. I’ve had to really shuffle and juggle to find a sponsor for my ideas. The common response is “That sounds like a cool idea, unfortunately I’m already committed to 2 projects as a PI, why don’t you check with so and so….”
I think this is an unintended consequence of the new system. While NSF wanted to prevent PI’s from shotgun blasting the system with lots of proposals, I don’t think they fully appreciated the percentage of those extra proposals that were actually the work of young investigators without PI status.
I wish there was a vehicle to allow post-doc’s to apply for funding directly, perhaps with Universities co-signing on a proposal by proposal basis. Certainly if the post-doc’s mentor is wiling to host the project, the University should be happy to these things funded for the prestige and revenue it generates.