Nobody likes change, at least not if the system is working for them. Arguments for keeping things the way they are usually win over arguments for change, until the current system is badly broken. One reason to avoid change is that it always has complex, unanticipated consequences, as conservative Edmund Burke famously argued in 1790 in Reflections on the revolution in France.
The recent change requiring pre-proposals by the Divisions of Environmental Biology (DEB) and Integrative Organismal Systems (IOS) at the US National Science Foundation (NSF) really has three components. First, a four page preliminary proposal is required before a full-length 15-page proposal can be submitted. Second, there is only one time per year that the preliminary and subsequent full proposals can be submitted, January for preliminary proposals, and August for the lucky few chosen to take the next step. Third, any given principal investigator can only be on two proposals from a given division. Each of these components is likely to have different consequences, so I will treat each separately.
Is the system badly broken? I think most DEB or IOS program officers would say that it is. Before the change, they were inundated with proposals twice a year. For each batch, they convened review panels of 15 or so people, then pleaded with a few hundred more people to provide external reviews and send them in on time. When things worked well, the panel met for three days and reviewed proposals that had at least six reviews, three from the panel and three from specialists more closely familiar with the proposed research. Thoughtful discussion provided everyone with helpful advice for improving their research and funding to the top proposals. But as funding rates plummeted, and eager scientists increased in number, disappointment spread. Some unfunded scientists are reluctant to take the time to review proposals. Funded scientists are few enough in number to make up the difference, even if they were all entirely cooperative. When funding rates become very low, small details can make inappropriate differences. Arguments about whether we are funding science or scientists and other concerns about the details of the process, proliferate. After one panel, I saw an eminent senior scientist put his head in his hands, lift it only for another sip of beer, and swear he would not continue to participate in the depressing process of denying funding to so many excellent studies. Furthermore, it has become increasingly difficult to lure active scientists into taking a rotation as program officer at NSF. These people are the real heroes of the whole system, working unbelievable hours for their scientific community. So, what are we supposed to do?
Can we evaluate the best science in four pages? I would say yes, for the most part. I think this is enough space to put forth your big ideas, show how they advance the field, and that they are feasible. There are plenty of funding agencies around the world that work entirely with short proposals. This past summer when I had to organize the judging of nearly a hundred posters, I did a little experiment. I read all the abstracts and rated them before looking at the posters. Then the teams of judges, not including me, evaluated all the posters. The bottom line is the abstract I judged to be the best was also judged by others to be the best poster overall. All four winning posters had abstracts I put in the top category. With a preliminary proposal, you have a lot more space than an abstract. NSF has used preliminary proposals in other contexts and my experience with such a panel was a positive one. I felt that the proposals we chose as finalists would truly become the best proposals. But that is just my impression. It would be good to have some real analysis of the issue.
Why don’t the biographical sketches count for more? A mutual fund prospectus always claims that past performance is no guarantee for the future, but for scientists this is absolutely not the case. I agree with the limit of ten published papers on the vita of principal investigators, for that lets us focus on their best work. But I do not feel that enough weight is given to this part of the package. A scientist that publishes outstanding, creative work in high-profile journals is likely to continue to do so. I believe studies show this is our best predictor of what will be done with additional funding, so why isn’t it counted more heavily, at least for investigators past the very beginning? Maybe the problem is there isn’t much to discuss. Maybe each of the ten papers should be followed with a couple of sentences describing the work. Accomplishment based renewals acknowledge this issue, but I have always been warned against trying for one.
What about the limit of only two preliminary proposals per investigator? I suppose for most of us this is not a limit that impacts us very much, since not many of us are writing more than two proposals at any given time. It forestalls the strategy of throwing in lots of preliminary proposals, like arrows at a target.
Is a single annual deadline for preliminary proposals a good idea? This is the part of the new system that most gives me pause. If we find out we did not make the cut, what do we do from May to January? What keeps us optimistic, working hard in the field we love? I know our colleagues at NSF hope that this hiatus will free us to explore, to do unfunded research, plan for future proposals more carefully, rather than frantically writing proposals that fail to be funded. But what if we just get discouraged? This is the big danger. One might argue that this part of the change should have been omitted, or postponed. What if there were two deadlines, but a pre-proposal could only go in once a year and the limit of two proposals per PI held for both times, so you could submit two in January, or one in January and one in July? Would having two deadlines cost too much?
Why do only DEB and IOS have pre-proposals? I suppose a certain amount of autonomy in NSF programs is a good thing. I do not have a clear idea of how competitive funding is across the directorates at NSF. Molecular and Cellular Biosciences (MCB) has come up with a different solution to a crunch: deadlines every eight months. This is a smaller change in many ways. I wonder if their issues are the same as those of DEB and IOS.
What can we do to increase funding to DEB and IOS? Neal Lane, former director of NSF, Rice University professor, friend, and one of the nicest people on the planet, argues persuasively that we all need to become better civic scientists. The English call what we do “blue skies research,” driven by curiosity, not a specific agenda. There is plenty of evidence that this is where true innovations of enormous practical significance originally come from. As a civic scientist, you can help the public, school children, elected officials, everyone, know this. The trouble is, I think we are doing an ever-improving job at this, with TED talks, lectures online, podcasts, Science Cafes, and the many excellent popular behavior, ecology, and evolution books coming out every year. Still, we need to do what we can to help others understand why what we love to do is also vitally important for our nation and our planet.
Will DEB and IOS analyze this experiment? I hope there will be a careful analysis of the impact of this experiment, done by social scientists who know how to evaluate such things. The best plan for doling out funding is one that is fair, transparent, and results in excellent science, and happy scientists.