NSF Preproposals – a failed idea at DEB and IOS?

DSC00855Preproposals came to DEB and IOS about three years ago to solve the problem of too many proposals and too few willing reviewers. They were also thought to solve the problem of too much investigator time wasted on writing unfunded proposals. There are probably other reasons for beginning them at that time. After all, many big programs with many investigators and institutions or other complexities have preproposals, so it wasn’t a new idea, even for NSF.

Preproposals have generated tons of controversy. Some people like them, but most seem to loathe them. I know NSF DEB and IOS have done their own careful study and that it is going to come out soon. Here are my thoughts developed from talking to many people and from personal experience. I bet you can think of additional points.

1. Preproposals are not less work, for the writer or the reader.
The idea was that a preproposal was only 5 pages and so they would not take so much time. But trying to squish your ideas into 5 pages can be more work than explaining yourself fully as anyone who has tried to write a paper for Science knows. You don’t know what to include and what to leave out. You agonize over balance between ideas, background, and experimental detail, not to mention broader impacts.

There is another way in which preproposals are more work. Two get funding you have to write two proposals with different target audiences, different lengths, and different times of year. The second time may be exciting, but about half those proposals are also declined.

2. Referees have to keep two different standards in mind, 5 page and 15 page. In many countries the whole proposal is about 5 pages and all judgements are made at that level. Then everyone gets used to that standard and has a clear idea of what level of detail to expect. Here we have two standards making it hard not to expect something from 5 pages that cannot be there.

3. Preproposals are only judged by a committee, so you are less likely to have an expert in your field read it. Sometimes I feel like no one who really knows the field of the proposal read it. This is inevitable when the preproposals are only read by a committee. The comments that come back on the preproposals can seem shallow and off the mark. You may have had a weakness in your proposal that you addressed as well as possible, yet no one noticed it. Instead the panel went after something you did not detail because it is routine for your field. Doesn’t everyone deserve a team of experts to read their proposal, or at least a level playing field?

4. Preproposal reviewers still focus on methods. The idea originally was that you would be judged just by your big ideas and you don’t need to go into detail on the methods. Taken to an extreme you could propose to do amazing and impossible things that anyone should want to fund. Of course you don’t do this. I have been discouraged by how much NSF panels at all levels focus on methods, no matter how much program officers entreat panels to do otherwise. People may feel more comfortable with challenging methods. They may feel less able to really judge what is a big idea and what is not. In a way the biggest ideas will have no audience of experts. This can be particularly true when there is no one on the panel that really knows your field.

5. If the top preproposals translate into the top proposals, we don’t need two steps. I do not know if this is true, but I bet it is. Excellence tends to be correlated across different arenas. Taking away the chance to ever get fully evaluated can really hurt struggling investigators, perhaps keeping them from performing to their optimum and only rewarding the people that already know how to do this.

6. Preproposals do not solve many problems once a year deadlines would not also solve. A once a year deadline with full proposals and a mix of panel and ad hoc reviews would keep researchers from churning proposals. It would allow early career investigators to submit twice, once regular, once CAREER. It would give everyone the thoughtful advice in comments on a full proposal. It would require greater support from the community to get all the ad hoc reviews, but it would be worth it.

7. Preproposals create another status ranking among investigators, which can be very demoralizing. The countries with the longest lifespans are those with less hierarchy, like Sweden, many studies show. While it will help your longevity to be high status (see study of Nobel laureates), isn’t what we really want longer, fulfilled lives for everyone? You get that from less ranking. We can’t get rid of the most fundamental rank, funded and unfunded (but see a soon-to-come post), but we can get rid of the shameful not asked for a full proposal category.

8. Preproposals are terrible for researcher morale. I suppose this one could have been predicted, but I did not think of it before. Somehow having your precious ideas not even be considered worthy of a full proposal is really discouraging. Any time that writing a shorter proposal might have saved is lost to gloominess about the whole scientific enterprise. It is bad enough not to get funded. It is simply horrible to have this be the result of something less than a full chance to explain yourself. There is tons of evidence of low morale. This is the biggest problem with preproposals. Say what you will about whether it should have happened. It did happen. We need to fix it.

9. What to do next? I recommend that we drop the preproposals, go to a single full proposal a year, and as a community commit to careful review of the adhoc proposals we receive. If you are not getting any, contact your program officer and let her know you are available. Change is complicated. I think this was a good idea, but it did not work. It is time to drop it.

Posted in Creativity, Grant proposals, Grants, New ideas, NSF | Tagged , , , , , | 5 Comments

What I learned at the National Science Foundation

Here are some things I learned last week at NSF while serving on the Advisory Committee for Biological Sciences. I expect every time I go I’ll learn something new, but in some ways what a beginner learns first can be important.

France Córdova, NSF director, and John Wingfield, AD Biological Sciences

France Córdova, NSF director, and John Wingfield, AD Biological Sciences

1. Anything to do with the federal government has a lot of rules. It would be impossible to try to teach us all those rules at once. Rather, the permanent people at NSF guide us away from areas we cannot change. This makes sense even if we strongly disagree with something, because there is no point in tilting at something that is not in our purview. I’m only beginning to learn what this applies to, but here are a few examples. Our committee is an advisory committee. These committees have rules under FACA, the Federal Advisory Committee Act. We don’t get to mess with these rules, so get over it. Another example is we might have ideas on making the description of Broader Impacts more clear, but we don’t get to do that. They are NSF wide, and come also from evaluations of the National Science Board. I think there are tons more things like this, so when a permanent person at NSF tells you something is off limits, listen to them. We have plenty to do within our limits.

2. The mix of temporary rotators and permanent staff at NSF is very useful,

Our leader, Kay Gross

Our leader, Kay Gross

but very confusing. I imagine the rotators come in full of energy, determined to make a difference, with big plans for change, but then settle in to understanding what the nature of possible change is. The permanent people seem to welcome the rotators and help them mold their enthusiasm to things that are feasible. The wise rotator figures out how to be mentored by the permanent people. The wise permanent person figures out how to use the new perspective of the rotators.

DSC008513. The people at NSF see more science than I’ll ever see. These people see the earliest forms of new ideas. They see so many different ideas from different perspectives, research institutions, and disciplines, that they develop a sense for what is really innovative in a way that is much more profound that I can do from my relatively narrow corner. This is not because I don’t try to be broad, but simply because with all the things I do, I have no time. If we want insight into what areas might be new and needing more funding, as was the case for molecular evolution a couple decades ago, the program officers are the people most likely to know.

4. When funding rates are so low, decisions are complicated. Proposals at IOS and DEB require pre-proposals, once a year. Proposals at MCB have a single annual deadline. A lot of thought has gone into both these processes. People who don’t like the preproposals for example, generally also want to submit twice a year. They think that if they can do this they will have more chances to get funding, therefore getting tenure, a promotion, whatever. But what they forget is the NSF is not giving out any less money with the pre-proposals. Isn’t writing a careful pre-proposal once a year a better use of your time than writing twice a year, often bouncing back from comments you don’t have time to digest? Isn’t the real question about how we can convince our universities to understand that some excellent researchers may simply not have NSF funding?

5. The workload at NSF is horrendous. Our program officers have to be really efficient with their time. They have to justify every decision they make. There are all kinds of reports they do that we don’t necessarily see. The workload is also not even with some divisions handling many more proposals and dollars than others with the same staff. I guess the rotators do this for all they learn in a short time.

6. NSF is exciting. It is a really great feeling to be part of the heart of curiosity driven science in America. It is really fun to be with so many smart, caring people, all trying to help make this nation a better place.

 

Posted in Grant proposals, Grants, NSF, Outreach, Politics | Tagged , , , , | Leave a comment

What is NSF’s Advisory Committee for the Biological Sciences supposed to do?

When I got asked to be on the Advisory Committee (AC) for the Biological Sciences at NSF, I said yes. I supposed this was an important thing to do. I have been trying to say no and preserve time for my own interests and those of my group, but am vigilant to opportunities where I feel I might make a difference.

I didn’t know exactly what being on the AC would entail, and will not look up the email that enticed me. I’m not sure how far this open meeting thing extends. But what NSF does is give out money for research, education, and for broadening the reach of science in America, all things that are important. Would we be asked to weigh in on what gets what percentage of the money?

One might think so because the top of page one in the orientation materials tells about that money, all seven billion dollars of it.photo There if you can read this, you can see what we have to play with. Keep reading through this report, likely available on the page for the AC.

There is a lot of other interesting stuff in the orientation booklet. There is also some other stuff we were given. I tried to read it before I came, but I kept looking for what we were supposed to do with the information and did not find that, so stopped reading. After all, I have plenty of other things I have to take action on every day. I figured we would learn this at the meeting.

We got a schedule which basically consisted of talks followed by discussion. To talk, our leader, Kay Gross, had us turn our name cards on end. She did an amazing job of calling on people in order and it meant that while you were waiting for your turn you did not have to fret or waive your hand, or interrupt. But the problem was that we couldn’t really have a discussion with 30 people at the table. For example, one person said that he had a hard time figuring out how to get minority undergraduates in his research laboratory. That is something I know how to do, but I didn’t tilt my card. After all, it would be 5 or 10 people before I would be called on, and the discussion would have moved on.

The whole day went like this, very interesting talks about important subjects, like is there a shortage in STEM (science, technology, engineering, math) or not? Michael Teitelbaum thought not, at least not overall. I’m going to believe him because he was Science’s Person of the Year in 2013 and published this in the Atlantic. He also wrote an important book on whether the US is Falling behind in science. It was exciting to hear him, to have him seated only a few seats away, someone I had only before talked to on the phone, back when I was involved on the NSF/Sloan molecular evolution panels. It makes me really happy when such amazing, famous people are so nice.

Still, I’m confused. I guess I expected a really clear set of concrete tasks. I expected maybe we could ourselves decide what we do. They gave us the AC Charter. It says we are established under the Federal Advisory Committee Act, FACA. We are “To provide advice and recommendations to the National Science Foundation concerning support for research, education, and human resources in the biological sciences.” There is some more detail that includes advising on program management, overall program balance, and some other stuff. We cost about $100,000 a year, of which 40K is for NSF staff support cost. We meet twice a year. We report to the Assistant Director of the Biological Sciences Directorate.

All that makes sense and seems consistent with what I figured was our charge. But I was still puzzled that this did not seem to be what we were doing. But then there is Item 8 on our list, the Designated Federal Officer (DFO). This person is an NSF employee who will be the DFO. This person can approve or call the meetings, prepare and approve meeting agendas, attend the meetings, and “adjourn any meeting when the DFO determines adjournment to be in the public interest.” This seems bizarre to me. I guess I just assumed that part of an advisory committee would be a certain degree of independence. After all, they are free to ignore what we say, so why can’t we organize ourselves?

It isn’t like there aren’t other committees at NSF that are free to act quite independently. The Committee of Visitors, for example can ask for anything they want and try hard to understand the operations and fairness of a program. I was on one once for DEB (can’t get away from acronyms if it is the federal government, I give up). I tried so hard to find bias, asked for all kinds of things, and was really delighted with the depth of fairness I found.

It isn’t that the DFO isn’t a nice person, or doesn’t want the same things we want, the best for the research and education future of the country. It’s just it seems an odd way to do things. Maybe I’ll understand it better tomorrow. Or maybe they will decide I’m not the kind of person likely to be effective on this committee.

Another way to understand how this committee works is to look at the minutes from the last meeting, before I joined. It met in March, in this same room. Someone took minutes, which we approved today. These minutes give the flow of the meeting, introductions, items. The first item is the Bio budget. It was presented. The committee discussed it, according to the notes, nothing substantive. Then they went on to a CoV report. The committee discussed it and approved it. Basically all the rest of the minutes go on this way, with presentations followed by discussions. Occasionally the AC had a more meaty discussion, as when they were enthusiastic about GoLife. Well, there was really a few more substantive comments, but I’m not going to go into them here.

This is unlike any other advisory committee I’ve been on, but it is late, so I’ll save my thoughts on what an advisory committee should do for another time.

Posted in Grant proposals, Managing an academic career, NSF, Outreach | Tagged , , , , , | 2 Comments

Why it is OK to have kids in grad school or as a post-doc

There was a post by Leonard Cassuto on having kids in graduate school and what a difference it made to one student when she heard that it was explicitly all right. The point that Leonard made was that our role as faculty is to support and educate our students and help them grow into the next generation’s leaders. This means listening to them.  It means making it clear that there are boundaries. One of these is on their decision to have children.

DSC00780

Adorable Jacorri, born when his mom was in grad school.

I suppose there are people out there that secretly hope their grad students and postdocs will not have kids. Maybe these people think if they don’t talk about it, it will never occur to their group that having kids is an option. Isn’t that silly?

Obviously whether or not to have kids and when to have them is a personal decision. It is also one worth mulling over. Part of that may be discussing it informally with your lab friends. It should not be taboo. We talk about everything else at lunch, so why not this? I bring it up first. I think people should have kids when that feels like the most wonderful thing to do. I also think it can be a relief to have the first baby if you want to have kids. It puts all the anxiety of when it is a good time behind you. It turns out right now is a wonderful time.

Anna and Danny survived being born before tenure!

Anna and Danny survived being born before tenure!

Obviously a family takes time. But can also give you balance and happiness, perhaps more than you ever thought one heart could hold. A number of my graduate students have had children in graduate school. Never has it been any kind of problem. In fact, it can really focus a student.

Me and my youngest, Philip, the only one born after tenure.

Me and my youngest Philip Queller, the only one born after tenure.

What I tell my friends and students is that I think having kids and having a career is completely possible. One can even enrich the other. But kids take time, so something has to change. That is most likely to be in the hobbies, free time, hanging out department. Instead you will find ways to exercise, socialize, and cook dinner that involve those sticky little curious hands.

My own story is that my daughter was born the month I started as assistant professor at Rice. Her brothers came 3 and 10 years later, each a welcome, joyous addition to our family. If a paper or two was missed, or a seminar trip didn’t get made, it mattered not at all. Perhaps thinking harder about what work was worth doing made me a better scientist! So have those babies if it suits you. We will welcome them and help you fit your new life together!

Posted in Graduate school, Managing an academic career, Postdocs, Professors with families | Tagged , , , , , | 1 Comment

What to wear when you go to the National Science Foundation?

You just got chosen for an NSF panel. You’ve read the proposals, submitted your reviews and are flying into DC tomorrow. But there is one last thing. What do you wear? I suppose no one talks about this because after all we are scientists, interested in ideas, not something so shallow as clothes. Above all we want to be comfortable and to have our clothes not stand out.

Usually the panels I serve on are confidential, so I feel constrained from commenting. This one, the NSF Biology Advisory Council, is public. I suppose I could have anonymously noted clothing before, but didn’t. So yesterday I took the time to tally some aspects of what people view as appropriate attire for serving on this important committee.

There were about 20 people in the room, evenly divided between women and men. Your top is what shows the most around the table, so if you want to fit in, pay attention to the top. Most people had shirts and some kind of informal jacket on, but lots had no jacket. Two men had on ties and two women had scarves. A third had only short sleeves, but there was only one nice t-shirt. Four of ten women wore black, so no need to dig out the NYC black. Besides black, the tops were gray, blue, turquoise, melon, mauve, brown, and khaki. Bottom line is you can where whatever you like on top. The room was not too cold for short sleeves.

I had to wait for a break to look around at what people wore besides their tops. No one had shorts on. Two women wore skirts, while the rest had pants on. Seven had khakis, three had blue jeans, and the rest had some form of dark pants that you would have to be better at clothes than me to classify. Nearly everyone had socks on. The men mostly wore leather shoes and the women mostly wore sandals. There were three pairs of running shoes. One woman only had low heels.

Overall, the people from NSF were more formally dressed than were the people from outside. I suppose they have to be ready at all times to deal with the public or Congress.

I know you are just dying to know what fashion conscious me wore. I was one of the minority in blue jeans and tennis shoes. But on top I had a nice gray silk Eileen Fisher short sleeved shirt. You see, I travel very light and didn’t want to pack a spare pair of shoes.

So now you know what to wear to NSF, get yourself on the list to serve on a panel, even if you haven’t been funded yet. Just let the program officer in your area know!

 

Posted in Business meetings, Grant proposals, NSF, Travel | Tagged , , , , | 4 Comments

How to stop grading unfairly: 9 ways

It may be only the second week of class, but I have a stack of 45 tests to grade. The students had to answer 10 of 20 questions that generally could be answered in 3 to 5 sentences. Now I have to grade them. How can I possibly do this fairly? What I want is for the points I give a test to reflect what the student knows about the material. What could get in the way?

A lot of problems could arise, I am reminded from the workshop on grading that I went to right before classes began. Beth Fisher had some wise answers: have clear questions; have a grading rubric; make sure your test matches what you have taught. But in a way what most impressed me were the colleagues in attendance who were struggling with fair grading. This came out most strongly in the courses that were so large they had different teaching assistants grading different people. One young professor who was determined to improve mentioned that the teaching assistants had inadvertently both graded some students. Sounds fine, but the problem was that they both wrote their scores on the top of the page, somehow not noticing the other’s score. And the grades each gave were quite divergent, according to the professor. I wonder what kind of pandemonium in the class this led to.

photoArbitrariness in the grade is horrifying to the students, for it might affect their whole career. I suppose to us faculty it is our dirty little secret, something few are as open to addressing as my forthright colleague. What can we do? Here are a few thoughts.

1. Have students put their student ID or course assigned number on the test and not their name. It has been very well documented that once we know our students, we give the ones that are generally talkative in class, or have a record of good performance better marks. We see things in their answers that are not there. Conversely, we are more critical of those we think less highly of. If we don’t know who they are, we will grade the questions more objectively. I don’t know many in the class at this early point, but I still had to make myself go back over the answers of the few I know to make sure I wasn’t being too stingy or generous.

2. Grade in clear-cut categories as much as possible. My students have 10 questions, each worth 10 points, with the whole test being worth about 6.7% of the course grade. Instead of grading each question on the scale of 1 to 10, I generally only use 3 numbers, 0, 5, and 10. They get a 0 if nothing is correct about their answer, a 5 if they have some correct information, and some wrong information, or are incomplete. If they are entirely correct and complete, they get 10 points. I’m not a total stickler, so many of them get 10 for most of their questions. A very few situations will give a student a 3 or a 7. This kind of grading helps reduce bias because the judgements are easier.

3. Have the same person grade all of the test, or of a section of the test. If this is not done, then there will be great inconsistencies, no matter how carefully the other parts are attended to. Some people get together after a test to grade together, each person taking a question or a page. This will not always be possible, though, in the large classes we have today. Lab notebooks in particular were mentioned as needing multiple graders. What to do?

4. If there must be multiple graders, standardize them on every assignment. Make a few copies of a few papers and have all the graders grade them. Compare scores, discuss, try it with a new set of papers and repeat until the graders are very uniform. Does this sound like too much trouble? Just think for a moment how important fair grades are for the students.

5. Have a clear rubric and a key. A rubric is just a list of things the assignment calls for and points assigned. If it is too detailed, it will make things harder. The more separate sections you have, the better. For my test, the rubric is quite simple. For the Wikipedia articles the students write, the rubric will be more complicated. Here is my rubric for the first things the students do on Wikipedia, evaluate articles already posted, below. You can see that most of the points are for completion of the category.

 

Grading rubric: 70 points in all, 14 points per organism, 5 organisms
For each organism:
5 points: What are the strengths of this entry? What have you learned that is most interesting?
5 points: Name 3 general categories in the outline that are missing and could be included. Explain why for each.
4 points: Look at the talk page. Comment on the details here, including the ranking and importance of the article.
Full points will be given to entries in each category that are thorough, exhibit careful thinking, and tie to the material of the course. Your writing should be intelligible without going back to the original Wikipedia page.

6. Grade a given section or question all in one sitting. We change how we grade according to mood. Just look at the decisions from an even more important arena: our justice system. Ed Yong reports on a study of parole granting and finds judges that have eaten recently or are at the beginning of a session are more lenient, to large effect. I purposely did not continue grading after the 5K I ran yesterday when I was feeling extremely mellow.

7. Grade the test or project question by question, piece by piece. You will be more consistent if you grade all of question 5 before grading any other question. Likewise, with a lab report, grade all of a certain section before grading any other section.

8. Mix up the order of the papers. If you are grading question by question, you can easily shuffle the papers a bit when you finish a question. That way each student will get the benefit and cost of position. You may be differently lenient at the beginning when you haven’t seen all the possible answers. You may be desperate for a break towards the end.

9. Be aware of inadvertent bias and try to avoid it. All of these things assume you want to be a fair grader and are trying hard. They address things inherent to human nature. Following these, and I’m sure there are other good tips, will simply make the learning process and its evaluation a more accurate reflection of what a given student is demonstrating on a given assignment.

Posted in Teaching, Undergraduates | Tagged , , , , , | 3 Comments

Thoughts on easy and hard test questions for undergraduate students.

No this is not going to be a post on Bloom’s taxonomy of questions memorized or conceptual, useful as that is. Here I’m talking about the kinds of material students struggle with.

If ever there was a clear book surely it is Richard Dawkins‘s The Selfish Gene. I use it to begin my course in behavioral ecology and just gave a test of short essay questions, only two weeks into the semester. The class had read the whole book, guided by study questions I wrote that you can find here. They answered 10 of 20 possible questions. Here are some things they had the most trouble with. I am guessing these are general patterns, not specific to this assignment.DSC06017

1. If part of the question has a numerical answer, they will get it correct. Worker relatedness to sisters, hymenopteran sex ratios, sister to self relatedness, they get it all. For a more specific example, the main cost of sex is you pass on only half your genes. I’ll count that as right for that half of the question because they forget about the costs of hunting for a mate and other such things and they are less essential. But they are quite confused about how to phrase the benefit and have a hard time deciding if it is in the interest of the gene or the individual.

2. They are not strong on considering the framing of the question and can slip over to considering the interests of a different party. The answer to why female mice eat their babies when a new male takes over should be about why this might be advantageous for the female, not the male. A female should behave according to the interests of an unrelated male only if it is her interests also, or she is forced into it. Eating babies may be the best of a bad job if the male is going to kill them anyway, for example. If a question asks about plasmids and nuclear genes, bacteria are unlikely to be the correct context. I could go on.

3. Students assume they know more about the organism than they do. They assume the organisms are more like people than they are. I wish they would all start by assuming plankton, drifting in time and space, more or less. Or bacteria. Or amoebae and only add on other attributes when they as specifically mentioned. They always seem to assume attributes about an organism that they do not know. This ties to the general point that true wisdom comes from understanding your own ignorance. In the mouse example above, many students assume the male will help mom with the babies, in total absence of evidence.

4. After reading Dawkins, if you ask students directly what the unit of selection is, they will say the gene, but their thinking on subsequent questions will clearly show they don’t get the consequences of this answer. They will mention species helping if you ask about some kinds of cooperation. They will consider the group for advantages to sex. It reminds me of what Ross Nehm said about their minds being like a room full of furniture. What we add has to fit with what is there before. Most of them will read Selfish Gene, get direct questions correct, yet not change their thinking about natural selection and evolution very much at all. How to encourage this to happen is my challenge for the rest of the semester. I’m guessing that what will be most effective is helping them learn how to nudge furniture around in their beautiful minds, maybe occasionally adding a lovely table or new chair.

But it is only the second week of class. In some ways what they have done is impressive. After another 13 weeks of discussion, quizzes, writing, and teaching, I’m betting some will never forget why a slug might chew off its hermaphrodite mate’s penis as being about the female, not the male function.

Posted in Teaching, Undergraduates | Tagged , , , , , , | 4 Comments