Are NSF preliminary proposals a good idea?

Nobody likes change, at least not if the system is working for them. Arguments for keeping things the way they are usually win over arguments for change, until the current system is badly broken. One reason to avoid change is that it always has complex, unanticipated consequences, as conservative Edmund Burke famously argued in 1790 in Reflections on the revolution in France.

The recent change requiring pre-proposals by the Divisions of Environmental Biology (DEB) and Integrative Organismal Systems (IOS) at the US National Science Foundation (NSF) really has three components. First, a four page preliminary proposal is required before a full-length 15-page proposal can be submitted. Second, there is only one time per year that the preliminary and subsequent full proposals can be submitted, January for preliminary proposals, and August for the lucky few chosen to take the next step. Third, any given principal investigator can only be on two proposals from a given division. Each of these components is likely to have different consequences, so I will treat each separately.

Is the system badly broken? I think most DEB or IOS program officers would say that it is. Before the change, they were inundated with proposals twice a year. For each batch, they convened review panels of 15 or so people, then pleaded with a few hundred more people to provide external reviews and send them in on time. When things worked well, the panel met for three days and reviewed proposals that had at least six reviews, three from the panel and three from specialists more closely familiar with the proposed research. Thoughtful discussion provided everyone with helpful advice for improving their research and funding to the top proposals. But as funding rates plummeted, and eager scientists increased in number, disappointment spread. Some unfunded scientists are reluctant to take the time to review proposals. Funded scientists are few enough in number to make up the difference, even if they were all entirely cooperative. When funding rates become very low, small details can make inappropriate differences. Arguments about whether we are funding science or scientists and other concerns about the details of the process, proliferate. After one panel, I saw an eminent senior scientist put his head in his hands, lift it only for another sip of beer, and swear he would not continue to participate in the depressing process of denying funding to so many excellent studies. Furthermore, it has become increasingly difficult to lure active scientists into taking a rotation as program officer at NSF. These people are the real heroes of the whole system, working unbelievable hours for their scientific community. So, what are we supposed to do?

Can we evaluate the best science in four pages? I would say yes, for the most part. I think this is enough space to put forth your big ideas, show how they advance the field, and that they are feasible. There are plenty of funding agencies around the world that work entirely with short proposals. This past summer when I had to organize the judging of nearly a hundred posters, I did a little experiment. I read all the abstracts and rated them before looking at the posters. Then the teams of judges, not including me, evaluated all the posters. The bottom line is the abstract I judged to be the best was also judged by others to be the best poster overall. All four winning posters had abstracts I put in the top category. With a preliminary proposal, you have a lot more space than an abstract. NSF has used preliminary proposals in other contexts and my experience with such a panel was a positive one. I felt that the proposals we chose as finalists would truly become the best proposals. But that is just my impression. It would be good to have some real analysis of the issue.

Why don’t the biographical sketches count for more? A mutual fund prospectus always claims that past performance is no guarantee for the future, but for scientists this is absolutely not the case. I agree with the limit of ten published papers on the vita of principal investigators, for that lets us focus on their best work. But I do not feel that enough weight is given to this part of the package. A scientist that publishes outstanding, creative work in high-profile journals is likely to continue to do so. I believe studies show this is our best predictor of what will be done with additional funding, so why isn’t it counted more heavily, at least for investigators past the very beginning? Maybe the problem is there isn’t much to discuss. Maybe each of the ten papers should be followed with a couple of sentences describing the work. Accomplishment based renewals acknowledge this issue, but I have always been warned against trying for one.

What about the limit of only two preliminary proposals per investigator? I suppose for most of us this is not a limit that impacts us very much, since not many of us are writing more than two proposals at any given time. It forestalls the strategy of throwing in lots of preliminary proposals, like arrows at a target.

Is a single annual deadline for preliminary proposals a good idea? This is the part of the new system that most gives me pause. If we find out we did not make the cut, what do we do from May to January? What keeps us optimistic, working hard in the field we love? I know our colleagues at NSF hope that this hiatus will free us to explore, to do unfunded research, plan for future proposals more carefully, rather than frantically writing proposals that fail to be funded. But what if we just get discouraged? This is the big danger. One might argue that this part of the change should have been omitted, or postponed. What if there were two deadlines, but a pre-proposal could only go in once a year and the limit of two proposals per PI held for both times, so you could submit two in January, or one in January and one in July? Would having two deadlines cost too much?

Why do only DEB and IOS have pre-proposals? I suppose a certain amount of autonomy in NSF programs is a good thing. I do not have a clear idea of how competitive funding is across the directorates at NSF. Molecular and Cellular Biosciences (MCB) has come up with a different solution to a crunch: deadlines every eight months. This is a smaller change in many ways. I wonder if their issues are the same as those of DEB and IOS.

What can we do to increase funding to DEB and IOS? Neal Lane, former director of NSF, Rice University professor, friend, and one of the nicest people on the planet, argues persuasively that we all need to become better civic scientists. The English call what we do “blue skies research,” driven by curiosity, not a specific agenda. There is plenty of evidence that this is where true innovations of enormous practical significance originally come from. As a civic scientist, you can help the public, school children, elected officials, everyone, know this. The trouble is, I think we are doing an ever-improving job at this, with TED talks, lectures online, podcasts, Science Cafes, and the many excellent popular behavior, ecology, and evolution books coming out every year. Still, we need to do what we can to help others understand why what we love to do is also vitally important for our nation and our planet.

Will DEB and IOS analyze this experiment? I hope there will be a careful analysis of the impact of this experiment, done by social scientists who know how to evaluate such things. The best plan for doling out funding is one that is fair, transparent, and results in excellent science, and happy scientists.

About these ads

About Joan E. Strassmann

Evolutionary biologist, studies social behavior in insects & microbes, interested in education, travel, birds, tropics, nature, food; biology professor at Washington University in St. Louis
This entry was posted in Grant proposals. Bookmark the permalink.

46 Responses to Are NSF preliminary proposals a good idea?

  1. oikosjeremy says:

    Excellent post–like all of them! Just found your blog and it’s wonderful stuff, I’ve put in a plug for you over at the Oikos Blog (http://oikosjournal.wordpress.com/2012/01/05/advice-how-to-succeed-in-academia/)

    Given your concerns about placing inappropriate weight on tiny differences between proposals, not funding many very good scientists, and not putting sufficient weight on the biographical sketch, I’m very curious about your views on the Canadian NSERC system (specifically, their Discovery Grant program). NSERC DGs are very different from NSF grants, and the DG program doesn’t really suffer at all from the concerns you raise, although of course it has its own drawbacks. My own thoughts on this issue are here (briefly, I much prefer the Canadian system):
    http://oikosjournal.wordpress.com/2011/05/16/should-granting-agencies-fund-projects-or-people/

  2. patlorch says:

    You make some good points Joan. I would add that the decrease in number of full proposals per year can have devastating effects on pre-tenure faculty. I feel the new approach also gives further advantages to funded faculty with experience at getting grants. They are better able to do what is essentially a sales pitch in 4 pages. They can also probably get by for an extra 6 months to a year, depending on how their pre-proposal does.

    I have heard some program officers describe one motivation for this new approach as an attempt to save us time from writing unsuccessful proposals. You rightly identify the real motivation as an attempt to reduce NSF proposal loads. Unfortunately, the only way NSF can reduce the pressure to write proposals is to find a way to fund more of them. The new approach will require us to write more proposals to try to get the funding we need, not to be more thoughtful. I agree with you that we may just get discouraged.

    One radical way to increase funding rates, while simultaneously reducing the pressure from college and university administrators to get more and bigger grants, would be to remove overhead from grants. It is a radical idea which would require restructuring how we fund research and universities, but I believe overhead is one of the ultimate causes of the high proposal load. The effect of dwindling funding rates and of overhead has been highlighted in a recent issue of the AAUP magazine Academe (http://www.aaup.org/AAUP/pubsres/academe/2011/ND/default.htm).

  3. Say the government wanted to give some money directly to universities for infrastructure, and wanted to find a way to do it that was fair and flexible. They could go by rankings, giving the most to Harvard, for example, because it might be the best. Or they could have a way of doing it that just mirrors the people doing research judged by their peers to be good enough to be funded. I think this is a great system, because it allows obscure schools to rise more quickly in their funding. The problem comes because we think of overhead as ours, not theirs. The NIH model is better in this one respect, where the investigators only really see direct costs. So, if the government wants a flexible way of funding universities directly, doing it as a percentage of federal funds coming in is not the worst way. There are lots of other issues, but it is important to keep this one in mind.

    Another issue where I think universities are not paying their full share is summer salary. We work all year long. We have undergrads in the lab or field in the summer. We plan courses. We have meetings. Why are we supposedly not paid for three months in the summer? I think universities should pay our full salary as the default. I have my 9 month salary paid out over 12 months and do not think about summer salary generally. This is not something that has changed much over my career, though it is harder at the beginning. (I remember when I first started at Rice in 1980, my salary was outside, below, the range of what they said they paid faculty.)

    • oikosjeremy says:

      The Canadian system is again a useful comparison here. Overhead in Canada isn’t part of the grant budget. Instead, NSERC just totals up the amount of NSERC grant funding that all the PI’s at a given university receive in a given year, and then pays the university a fixed percentage of that (20%, I think?) as overhead. And Canadian academic appointments are all 12 month appointments, so there’s no need to budget for summer salary (and in fact, NSERC has a rule against it). I wish I knew more about the history of the Canadian system, how it ended up the way it is rather than as an American-type system.

      I’ve heard it claimed that too much of the way in which US universities operate depends on the current structure of federal research funding for NSF to make any substantive moves in the direction of an NSERC-type system. I’ve even heard it claimed that, if NSF were to make such moves, getting an NSF grant would lose prestige compared to getting an NIH grant and all Biology depts. would end up being staffed exclusively by biomedical researchers. Without wanting to question the challenges of substantial reform, this sounds like a rather extreme view to me, like claiming in 1903 that motor cars cannot and will not be widely adopted because the whole transport system is structured around horses.

  4. It is no surprise that the Canadian system sounds more equitable and wiser. But how do we balance a certain soft complacency if funding is guaranteed against the edge that struggling (but not too much) for funding can give research? How does Canada fund biomedical research? In the US, is the best health related research from people who struggle to get NIH grants or people who don’t have to struggle at the national institutes? Are people naturally sufficiently achievement oriented that the additional struggle for funding is actually not helpful? I am not a social scientist and do not know the answers to these questions. But it is great to bring them up. Maybe someone knows of studies on these issues.

    • oikosjeremy says:

      Good questions, Joan. Re: avoiding soft complacency, a recent study of HHMI-funded investigators vs. matched NIH-funded investigators is relevant; I’ve blogged it here:
      http://oikosjournal.wordpress.com/2011/06/28/evidence-that-granting-agencies-should-fund-people-not-projects/

      NSERC is of course different than HHMI in that it gives many more people much smaller amounts of money. Which could promote risk-taking rather than complacency on the part of PI’s. If you know you have a low-but-pretty-reliable baseline level of funding for your lab, maybe you’re more willing to take risks on other, bigger grant applications.

      NSERC also recently changed their system so that, while renewal rates are still high and the average grant size hasn’t changed, the amount of money that you receive when you successfully renew can vary a lot from one renewal to the next, depending on how productive you are and how good your proposed research is. There used to be a lot of inertia in the system–the size of your grant couldn’t really go down, or up, very much from one renewal to the next, which probably did breed too much complacency. That’s no longer the case. Does this way of doing things hit the “sweet spot”, imposing an “optimal level of struggling” on PI’s? I think so, but as you say you’d need to do some really good social science to prove it.

      Also worth considering that, as an NSERC DG-funded investigator, I only have to write one 5-page grant every 5 years to get that baseline level of funding for my lab. I write other grants, but overall I spend far less time chasing money (or reviewing grant applications from others) than my US colleagues. I can use that time to actually do science.

  5. Certainly NSERC funding to individual investigators is not guaranteed, and have seen no signs that my colleagues working in the NSERC system are complacent about it.

    I think where the 2 proposal limit will be a serious constraint is on collaborative research — which has been increasingly important and encouraged. That should be considered seriously and assessed as the data on how this affects outcomes accrues. Maybe it would be worth considering a limit of 2 proposals as PI but allowing people to be a co-PI on additional proposals.

  6. The system in Canada is as bad, if not worse, than the system in the US. In the 1990s the “Canadian Association for Responsible Research Funding” (CARRF) made not a dent on the status quo. Numerous papers and my book “Tomorrow’s Cures Today? How to Reform the Health Research System” (Harwood Academic, 2000), spelled out a modification of peer review – “bicameral review” – but, again, not a dent!

    Johannes Wheeldon and Richard Gordan have recently reopened the matter in the Canadian version of the Huffington Post:
    http://www.huffingtonpost.ca/johannes-wheeldon-phd/peer-review-academia-canada_b_1114049.html
    For more, please go to my peer review web-pages at http://post.queensu.ca/~forsdyke/peerrev.htm

    • oikosjeremy says:

      It sounds like you are referring to biomedical research in Canada, about which I know little. My arguments solely concern NSERC, the Canadian NSF equivalent. NSERC recently underwent an international review which strongly praised its approach, while also suggesting tweaks like reducing the inertia in funding levels.

  7. Kim says:

    As Dutch citizen working in the US, I would applaud if NSF would reduce the page lengths for the proposals to four pages and eliminate the full proposals all together. The long proposals are motivated by the wrong idea, and that is that we actually should judge details of experiments at the level of grant proposals. Unless we make the experimental designs of the proposal mandatory with penalties when we are not doing it that way, nobody is required to follow them as soon as the money is awarded. What we need is ideas, not protocols. And if you have a great idea, 4 pages is more than enough to do that. Many European grants are based on just several pages or relative small word counts (2000 or 4000). And that is not a problem.

    • oikosjeremy says:

      That’s what NSERC does–proposals are 5 pages long, not counting budget, references, etc. But NSERC is funding research programs, not individual projects (i.e. an investigator can only hold one NSERC Discovery Grant at a time, which can be used to support any or all of the investigator’s projects). Not sure such short grants would work as well for individual projects, where you probably do want to know more about the methodological details.

  8. elitism is a plague of humanity since centuries says:

    You are a partisan of the elitism and so you give a partial point of view. Elitism is not a universal truth but a belief or an opinion.
    Your judging a hundreds of poster is just a joke: you can judge only on criterion/a. In research, what are your criteria? The cost? The novelty? The (financial, cultural, patrimonial, etc…) benefit it can bring? How you measure each criterion?
    A human cannot be objective. You will try to imagine you are but your brain is not whatever your effort. So, the real questions are: What’s a good poster? What’s a good project?
    Publishing high as a good criterion? Great! See that: http://the-scientist.com/2011/12/19/top-science-scandals-of-2011/ and Nature (2005) and all analyses that demonstrate the harder a competition, the higher probabilities that winners are cheaters.
    Moreover, what you consider as the best proposals is just dependent of your conscious or unconscious cognition. In this article, you think that “all is for the best in the best of all possible worlds” but you will never see what your rejecting judging process had missed (as an beta error).
    Why not considering this question using the scientific approach?
    Let’s begin to formulate all the possibilities and, as in epistemologic/scientific approach, let’s settle a criterion (that we assume as subjective because there’s not universal criterion) and evaluate them (mean and variance components …all moments in fact of the distribution of our criterion values) compared to score of the “neutral” random and a rotative granting.

    Please preach for rational and truth-spelling (as subjectivity) approaches in all aspects of our life, even more in scientific domain. Thanks.

  9. discouraged says:

    I agree that overhead is one of the fundamental problems with the current system. At least half of my colleagues on a recent panel had never received an NSF grant, in spite of trying for years with good projects. It struck me as ironic that these folks from teaching-intensive programs were being asked to choose among high six-figure projects when many of them would give their eye-teeth for $30k/year for three years.

    At the end of the panel, the NSF mucky-mucks came in and asked for our reactions and advice. When I mentioned the unfairness of overhead cutting deeply into an already-limited pot, I was patronizingly and quickly shut down. “Can’t and won’t change,” was the message. In other words, “Thanks for coming and doing all this work for us, but we have no intention of creating a system that gives YOU a realistic chance of funding.” Very discouraging.

    What efforts exist to reform NSF? Are there watchdog groups lobbying for reform and increased NSF budgets? If these groups don’t exist, what should they look like if we want to create them?

    I’ve spoken with senior researchers who have enjoyed decades of funding from NSF. They are discouraged and pessimistic. They are not getting funded on some of their best ideas ever. If they feel this way, how should the little folks feel?

  10. juliebyrd says:

    I think you have done a good job of summarizing both the problem and why this is an imperfect, but understandable solution. I do think it is good to not have to be working on proposals constantly, but a problem arises if the preproposal is not accepted and the lab is no longer funded. Unfortunately, the PI can’t be working on more research to support better proposals, and this will most likely result in discouragement.
    I also have a small note, which is not a big deal in the grand scheme of things, but would definitely improve the stress level and sanity of those working on pre-proposals: is there any way the deadline could be moved to, say, February? The end of December is one of the few times that people make time to spend with friends and family and take a little break from the rigors of academia. Now we cannot even enjoy that time because of the stress associated with the preproposal.

  11. Joan, thanks for the perspective. I think there is a lot of valuable stuff here and I just want to make two points in response, first to the comments and then to the post.

    1) You can’t drop the NSF O/H rate without squeezing “basic” science out of major research institutions (See http://scientopia.org/blogs/proflikesubstance/2010/10/28/random-final-nsf-panel-thoughts/) and the Canadian argument also doesn’t work (See http://scientopia.org/blogs/proflikesubstance/2011/08/19/why-we-cant-all-move-to-canada/)

    2) My biggest concern about these changes is for the new PIs. The shift to an annual cycle has a far greater effect on those who are trying to score their first grant than those trying to get their 7th. There will be far less feedback on full proposals to new PIs in this system and fewer kicks at the can, which is not a good thing. Also, your statement “I suppose for most of us this is not a limit that impacts us very much, since not many of us are writing more than two proposals at any given time.” is steeped in the thinking of an establish investigator. Obviously funding rates are way down and new PIs are struggling to get projects to stick in a way that wasn’t happening when this generation of curently established PIs cut their teeth. Pre-tenure people right now need to be working on more than two proposals.

    I have heard all the arguments that “two good proposals are better than 4 average ones”, but the reality is that PIs who are starting out now need to quickly ramp up to three or four good proposals that they can submit, because the simple odds that you can “win” with just two proposals don’t work out. It’s a different game now.

    The net result is that new PIs get pushed back in two ways by this change: they get less feedback with fewer chances AND they now have limits on the number of projects they can submit. People will get creative and look for ways to send proposals to IOS if they normal submit to DEB, and the reverse, but it is still an issue.

    Just my $0.02

    • oikosjeremy says:

      Hi Prof-like Substance,

      I know we’ll probably have to agree to disagree on some things. But in the interests of moving things forward, I’m wondering how far you think NSF could, or should, go in terms of funding more investigators as lower average levels (a possibility Joan raises in her comment below). This would be a move in an NSERC-ish direction. Presumably NSF does have some scope to move at least a little bit in this direction? I mean, surely raising success rates by, say, 1%, with a correspondingly small drop in average grant size, and thus a correspondingly small drop in the amount of overhead paid out per grant, wouldn’t result in any significant effect on the support of universities for basic research? So given that a small move in this direction would surely be feasible, how far do you think NSF could go? Can you put a rough number on it?

      • I would be more in favor of adopting an NIH-like policy as I describe below (comment threading makes this hard to follow), similar to the R01 vs. R15 and R21. I think there is a clear model already in place that would satisfy a lot of people.

  12. Great thoughts, all, or mostly all. How about this for a radical solution to several of these problems: fund based on 4 page proposals alone, adding a budget and a budget justification to what we submit now. Make the budget justification a little longer so we could sneak in more methods via their cost. Second, have two deadlines a year, in February and August. The thorny issue is whether to fund more at lower levels, and whether people with one funded proposal can have two. Currently funding is at fairly high levels for reasons NSF has made fairly clear. There is disagreement among the programs as to how many grants one person should have. I think there should be transparency and uniformity. Before agreeing that one person should be top PI on more than one NSF or NIH grant, I would have to be convinced that none of the proposals that would leave investigators totally unfunded are worthwhile. As it is, some argue one per PI and others don’t, leaving us confused as to what to do.

    • DrugMonkey says:

      Sure sounds like prioritizing democratic funding over quality and performance. “worth funding”? Incredibly low bar. Most proposals I read in NIH-land are “worth” funding in the sense that good stuff will result compared with, say, going to war. I can’t imagine NSF-land differs much.

      So yes, this is a one-grant-for-all proposal. And it is dangerously short sighted.

    • Kim says:

      Like it. Although I would not use the budget to squeeze in methods. In the end, nobody is doing exactly what they propose, so why bother with unnecessary detail which in the end is only page fill. Really, most of us know how to do experiments. If I indicate the general line of experiments, you know whether I overreach or stray to much out of what I am reasonably be expected to know about.

  13. The “a grant for all” approach is akin to saying everyone should get the same playing time on every sports team. It’s unrealistic and too idealistic. We ignore differences in achievement and ability at our own risk. Not everyone is going to be equally productive with equal resources.

    • oikosjeremy says:

      With all due respect, I do think this is a distortion of the NSERC DG approach. Not everyone gets a grant, and as I noted above, grant sizes actually vary by almost an order of magnitude (about 15,000-140,000/year). Surely we can grant that, on the continuua of success rates from 0-100%, both extremes are suboptimal, and that on the continuua of grant size distributions from “everybody gets the same amount” to “the best person gets all the money”, both extremes are suboptimal, and that the optimum is somewhere in the middle. And surely we can also grant that there are constraints that would prevent any given agency from adopting policies at any arbitrary point on these continuua. But granting all that still leaves open the important substantive questions of exactly what range of policies on these continuua are feasible, and the location of the optimal policy within the range of feasible policies. I would sincerely welcome your comments on these open, substantive questions.

      • Dude, put down the brandy and use English.

      • marilynmann says:

        I’m getting vertigo just reading this.

      • oikosjeremy says:

        Ok, wiseacres, short version of my comment: drop the irrelevant rhetoric about how “one person one grant” is a terrible system. Instead, please state exactly how much you think NSF could shift their success rates and average grant size. Please also state exactly how much they ought to. Or, if you think a hybrid system, as discussed below, is the way to go, please specify your preferred version.

        You may not like my writing style. But I don’t like it when I ask a perfectly reasonable question and you use my writing style as an excuse not to answer it. Prof-like Substance, you’ve repeatedly dodged or ignored perfectly reasonable questions I’ve posed, here and on your own blog. I remain sincerely interested in your answer to the two questions I just posed, and I hope you’ll finally decide to answer them.

        Here and on your own blog, you’ve explained at great length why the US can’t adopt the Canadian system. You have also emphasized what you say are massive problems with many other proposed reforms to the US system. And you’ve emphasized all the problems with the reforms NSF actually chose. Yet you’ve also said that the old NSF system was broken and needed to change. So how about just coming out and saying what changes you’d like to see? You clearly care a lot about these issues and have thought a lot about them. Why won’t you share your preferred solutions? Just saying “I don’t have any answers, all I know is that everybody else’s answers are totally wrong” is not a very attractive stance. Especially when accompanied by snark about writing style. I can promise you that I would consider your suggestions more respectfully than you’ve considered mine.

      • discouraged says:

        I like these ideas, OJ, and I like your writing style. (I don’t have much patience with smug jackasses, btw.)

      • oikosjeremy says:

        Ok Proflikesubstance, I’ve found a comment where you say you favor the NIH system of different grant types, aimed at different classes of investigator. Apologies for missing it, though I do wish you’d come out and said so earlier. You had ample opportunity, here and on your own blog.

        NSF does have the CAREER program, and so isn’t totally unlike NIH. But yes, NSF could move further towards an NIH-type system.

        NIH is far from immune from complaints about success rates, though, including from new investigators who are going for pots of money reserved specifically for them. Also, NIH new investigators often run into trouble once they are no longer eligible for the grants reserved specifically for them and have to compete for R01′s. So does the NIH system solve the problems created by low success rates, or just postpone them?

      • oikosjeremy says:

        @discouraged: Thanks!

    • cackleofradness says:

      Waaiiit a minute. One commenter chides another by saying “Dude, put down the brandy and use English.” And the other says “I like these ideas, OJ, and I like your writing style. (I don’t have much patience with smug jackasses, btw.)” in response….

      And the brandy comment is the one deleted?

      FSP moderates comments as they come in, and has developed a reasoning behind the ones that are allowed–many comments on women’s blogs especially can be derogatory and for whatever other reason a blogger might not allow some to be published–but generally transparency in these decisions as well as continuity between them is appreciated and helps to build trust amongst your readership.

  14. Some novelists have one great book in them, say Catcher in the rye, or To kill a mockingbird. Others keep amazing me, Salman Rushdie, Alice Hoffman, Naguib Mafouz, Pat Barker. I suppose more have several great books than have only one. If each of us have a unique perspective, and all have been doled the same number of hours in the day, isn’t there a limit as to how much research one person can do or manage? At the extreme, if we look at some of the labs that have ten or 20 postdocs, wouldn’t some of those postdocs be better off with their own independent funding? In the US, junior professors are not beholden to their senior colleagues in developing their research programs, as they are in some European systems. Striking the balance and funding excellence is tricky. Of course I do not agree that all should be funded equally. The goal is to get the newest ideas worked on, for we are doing science in the public interest, even if it is blue skies research. How we do this is neither fund everyone equally, nor give some big prof at insert-important-university millions of dollars and a stable of postdocs. I do not know the social science literature that looks at these questions, but I bet it exists.

    • oikosjeremy says:

      NIH actually did some research on these issues a year or two ago (sorry, no time to search for a link right now). They did indeed find that, dollar-for-dollar, their most productive investigators were those with an intermediate level of funding, not the ones with armies of postdocs and grad students. I suspect the same would be true in non-biomedical fields, but that the funding level that maximizes “bang for the buck” would vary greatly between fields. I too would be very interested to know about any similar research that’s been undertaken in non-biomedical fields, and if it doesn’t exist NSF needs to fund someone to do it.

  15. Anna says:

    Joan, I always like your blog, thanks for taking the time to publish your thoughts.

    I’d just like to say that as a (fellow) sociobiologist, I don’t understand why more people don’t understand that this is a classic tragedy of the commons issue. Given (approx) stable funding in $$ to NSF, we are all locked in an inevitable arms race to write more and more proposals to receive the same or even decreasing levels of funding. This is a logical consequence and also what NSF numbers show. This is an enormous waste not just of our time but also of taxpayer money. Remember that your salary while writing grants, and all the NSF, panel, and reviewer man-(or woman-)hours are an investment that we taxpayers pay for no return other than finding out who should get a paltry (by industry or even NIH standards) sum of money.
    Limiting the number of proposals submitted per PI is a necessary and the only step to prevent this.

    Time will tell if beginning investigators really suffer from this, but right now their complaints seem to be whining to me. (1) It is really hard to have two proposals that are good enough to be funded per year, and it is not NSFs job to read through all your early ideas to find out which ones are. (2) You can really be on more than 2 preproposals. First, CAREER don’t count towards this number. Second, you can be on 2 for DEB and IOS each. Third, you can be (paid) senior investigator on an unlimited number of other grants if you are the type that collaborates much (am I right in this?). (3) You do get feedback on preproposals, so you really have more time than before to actually revise your ‘sales pitch’. Assuming you use that time rather than waiting until after August.

    If you are a beginning investigator reading this: get feedback from your colleagues (in your department, your friends, fellow new investigators, whomever) on your preproposals before submitting. Be candid with yourself and others. A half-baked idea won’t get funded, so save it for another year.

    • drugmonkey says:

      If I am not mistaken the “tragedy of the commons” rests in large part on an ethos that all are equally entitled to, say, the village well water. Are all scientists equally entitled to public grant funding? What about those postdocs that don’t even have professorial appointments yet? Where’s the love for their piece of the pie?

  16. Jason Munshi-South says:

    Why not a hybrid approach? There could be two tiers of grants:

    Tier I. Four-page proposals only for 2-3 year grants that top out at $100k or some other arbitrary level much lower than current funded grants. The target acceptance rate could be something like 50%. I do most of my research with one PhD student and a few undergrads…we can complete good research with this amount of funding. I received a NSF RIG in my 2nd year as TT faculty at about these levels…would love to be able to apply continuously for something similar rather than only trying for the CAREER and regular cycle full grants.

    Tier II. Full 15 pg proposals as they have been. Funding rates remain very low. The idea would be that people apply for Tier I to keep doing good science, but go for a Tier II when they have their best ideas supported by robust results from previous studies.

    I floated this idea to NSF POs and one of the higher-ups last year when I was on a panel. At least one of the POs claimed that the majority of PIs don’t want small grants because they can’t do their best science. He was particularly firm that nobody wants the NSERC system. These claims were anecdotal and not supported by any kind of proper study. Plus, there is probably a strong ascertainment bias from talking only to PIs that have recently received large grants.

    • oikosjeremy says:

      Anecdotally, I have numerous US colleagues who would much prefer the NSERC system; I also have US colleagues who prefer the US system. So while I can’t put numbers on the range of opinion that’s out there, I agree that the NSF PO you spoke to is totally wrong when he says “nobody” in the US wants an NSERC-type system. That PO needs to talk to a wider range of people (and if that PO just thinks people who prefer the NSERC system are whining because they’re weak scientists who didn’t get funded or because they’re too wimpy to compete for money, he or she is *really* out of touch!)

      I also have a Canadian colleague who felt like he needed NSF-type grants to do the science he wanted to do, and so suggested that NSERC move to a hybrid system somewhat like the one you suggest. He suggested cutting everyone’s DG by 10%, and using the money to fund a program of large, NSF-style project-based grants, without worrying about how low the success rate would be (because after all, the DG program would still be there as a backstop, albeit at a reduced funding level). My colleague thought this would be a really popular idea, because everyone would be cocky enough to think they’d be one of the lucky few to get one of these big new project grants. He was at least slightly wrong about that–personally, I’m sufficiently risk-averse, and sufficiently confident in my ability to do good science cheaply, that I’m not willing to pay 10% of my DG in exchange for what’s effectively a lottery ticket. But I could imagine that many people might find such a hybrid system attractive (and I personally would much prefer such a hybrid system to the current NSF system). Much would depend on details, of course, particularly the average grant size and success rate of the reduced DG program.

      • I wonder whether we don’t have the model you’re thinking of right here in the US. NIH has it’s R01s that are the go to, but the R15 and R21 mechanisms are options along the lines you are referring to, with high paylines than the R01s. R15s have a budget limited to $300k in direct costs/3yrs and are for schools that take in less than a certain threshold of NIH dollars a year. R21s are similarly limited in budget, $250K/3yrs I think, and are for preliminary ideas.

      • Odyssey says:

        And nothing is stopping people from asking for less money in an NSF proposal…

      • Jason Munshi-South says:

        Yes, Prof-like Substance, the NIH R15 is a decent model for something that NSF could offer. Some of us can find study sections at NIH to apply to, but not all ecologists / evol biologists will be able to. The NSF RIG grants were like the R15 as well, but they were available to only a very small group of PIs.

        Odyssey, it is true that one could ask for less money, but the marginal benefit of doing so will be low unless everybody reduces their budget. If you are the only one to submit a small budget, then that will only help if you are just on the bubble for funding (say, #16 on the list of 15 grants the POs want to fund). In the last round of applications, there were over 280 applications in the study section I apply to. That should give you an idea of how unlikely it is that one’s smaller budget will make any difference if everyone else is going for the typical amount.

  17. Time will tell if beginning investigators really suffer from this, but right now their complaints seem to be whining to me. (1) It is really hard to have two proposals that are good enough to be funded per year, and it is not NSFs job to read through all your early ideas to find out which ones are. (2) You can really be on more than 2 preproposals. First, CAREER don’t count towards this number. Second, you can be on 2 for DEB and IOS each. Third, you can be (paid) senior investigator on an unlimited number of other grants if you are the type that collaborates much (am I right in this?). (3) You do get feedback on preproposals, so you really have more time than before to actually revise your ‘sales pitch’. Assuming you use that time rather than waiting until after August.

    Riiiiiight.

    Re#1, You are operating under the idea that there is a direct correlation between “good science” and funding. If you have ever been on a panel there is no question that many good proposals go unfunded and that it almost exclusively depends on the people in the room as to what gets the highest ratings.

    Re#2, you’re right that you can pitch two each to IOS and DEB, but I do not believe you can be listed at all on the Personnel front page for more than two each. Some people will be able to pitch ideas to both, others are more specific.

    Re#3, Feedback on preproposals is a completely different beast than feedback on full proposals. Having now written two of the former, they are remarkably different documents than the latter. If a new PI doesn’t get to the full proposal stage in year 1 and does in year 2 but does not get funded, they will the middle of their third year before they are submitting a full proposal with panel feedback. That is a radical change from the previous system.

    Half-baked ideas won’t get funded no matter what the situation, and if you weren’t already getting feedback from your colleagues then you were doing it wrong. Time will tell how this hits the younger set, but having a bunch of establish folks say “you’ll be fine! Look, I am.” is not exactly helpful.

    I am completely open to trying something new because it is clear that the previous system was in trouble. I understand how we got here, I’m simply pointing out some of the potential problems that could pop up. We ought to be keeping an eye on these things before we realize we’ve crippled a cohort.

  18. patlorch says:

    I would have to agree with the response of Prof-like substance to Anna, especially Re#3.

  19. Since comments seem to be disappearing around here, I have expanded on my thoughts on how we could change the system for the better in depth here: http://scientopia.org/blogs/proflikesubstance/2012/01/07/should-nsf-become-nih-light/

  20. About deleting comments: yes I delete them after a day or so if they are only useful, in my opinion, at the time. I deleted two one sentence comments,”I’m getting vertigo just reading this.” and “Dude, put down the brandy and use English.” I did not delete the one that calls me an elitist that begins: “You are a partisan of the elitism…” because I thought some people might be interested.

    I am new to handling comments, and am OK with leaving them all in if that is what you want, so I put those two short ones back in. I imagine it would be frustrating to bother to write something and have it deleted. But the other kind of frustration is that of people trying to learn from all these thoughtful comments and getting derailed by chatter. Thoughts?

  21. WRT comment deletion, my feeling is simple. If you delete comments based simply on whether you judge them to have worth, it is a form of censoring. While I can suport comment removal for vicious personal attacks or anything involving bigotry, making a call on the value of non-offensive comments is a slippery slope. It sends the message to commenters that they could be censored at any time and stunts conversation. The bloggosphere is awash in posts on the topic of comment moderation policies and their effect and I don’t think a discussion of it here does much for the topic at hand. I think if you want an honest conversation, regardless of how people’s (valid) opinions are stated, you’re better off letting readers decide what they want to digest.

    That said, it’s your space to do with as you please.

  22. OK, fair enough. I won’t delete anything that isn’t ad hominem etc. and leave you-all to figure out what’s worth reading. Thanks for the advice. It also makes it easier.

    • Thumbs up to the new commenting policy. Comments serve sort of as a running story and when you delete the ones that don’t necessitate it, you leave holes in the story and make it harder for readers to follow the evolving story.

      Digging the blog by the way.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s