It is kind of side information as to exactly how a panel works because it doesn’t tell you to write clear, excellent proposals with careful methods that address important questions with sufficient preliminary data to convince the panel as to the feasibility. Still, panels are so important, it is interesting to learn how they work. This is about NSF only. Furthermore, I’m leaving out the details of what the program officers have to do, like choose the panelists and referees, write the internal summaries, and then make the final choices, mostly according to panel rankings. There are two main kinds of panels, those with outside reviewers who will be true experts in your sub discipline and those reviewed only by three panelists.
You might wonder what value is added to the process from the panel meetings. Why can’t the program officers just get the reviews and decide? It is because discussion is valuable. With it you can learn whether written complaints are viewed as serious by everyone. You can get a broader opinion of what is exciting. You can learn to understand the proposal much more thoroughly.
First, the proposals are reviewed by professors in the field. They write down what they consider to be the strengths and weaknesses of the science and of the broader impacts. Then they write a summary. If you are on the panel, after you submit your evaluation, you can see what the other people said about the proposal. You do all your reviews and read all the comments by others before the panel meets. Usually you review about 15 proposals if you are on the panel. If someone found a fatal flaw, or a problem of any kind, you can look back at the proposal and decide for yourself.
Once the panel starts, things get hectic. It is basically a room with 15 to 25 people in it, but the main conversation is only between three of them and a couple of program officers. If you are at NSF, there are likely to be other NSF people watching the process from around the room. Or the room could be a virtual room, with phone and video connection. That’s how the DDIG panel I served on recently was. (There are lots of DDIG panels, so no, I did not review your proposal.) The process is basically entirely the same virtual or physical. Only the chit chat at lunch and in the hall is missing.
One person of the three assigned to the proposal presents the topic and the reviews. Then the other two speak in turn. There can be discussion. It lasts about 10 minutes, at least on DDIG panels, and probably on preproposal panels. In this way, we did 60 proposals in three days. After the discussion, the proposal gets put somewhere on the board, under Not Competitive, Low Priority, Medium Priority, or High Priority. One person summarizes the conversation and electronically submits it to the other 2 panelists on that proposal and to a program officer for approval. They often have to be revised according to comments panel members put in a special comments section.
Two things are taken very seriously: conflicts of interest and confidentiality. If you have a conflict of interest, you have to leave the room. In the virtual case, it can mean you have to leave your own office if you don’t want to re-sign in. Usually we are glad of the break. Conflicts of interest are specifically defined. Furthermore, you can declare a conflict of interest if you feel one even if it doesn’t meet the official category. You have a lifetime conflict of interest with your own former graduate students, for example. I have a conflict of interest with the University of Memphis because my daughter teaches there, in sociology. All my husband’s conflicts of interest are my own.
Confidentiality is also important. At NSF now, who is on the panel is not open information. What is in the proposals is private, at least until they are funded. It should not be shared with anyone. After all, people are sharing their best ideas and plans. Only they should be able to take advantage of this. So you should not be hearing who did well or poorly at any given panel.
If there is a lot of disagreement, or confusion, often someone else on the panel not assigned to the proposal will take a quick look at it and contribute. This can be very useful. Sometimes the proposal is rediscussed later when that person has had time to look at it more carefully. Another thing that a panel member not assigned to the proposal can do is summarize what she’s hearing in a way that can bring a discussion to a close. This is also very useful.
As proposals get put on the board, there are sometimes adjustments to position initiated by people who were on a cluster and have a feeling as to their relative ranking. It is complicated because no one is on all of them. At the end of the meeting, there is much more adjusting, though generally nothing moves very much. Clusters tend to get spread a little. We want to give the program officers the clearest possible advice as to what we think the ranking is. We also generally are told how many will be funded, so it is unlikely more than that will go in the top category.
So, that’s the process. I’m comfortable with it. I’ve served on a Committee of Visitors which evaluates the outcomes of the process over a series of years and can look for various kinds of bias. I was comfortable with that too, though it is discouraging how little money the US wants to put into all-important basic research, wonderfully called blue-skies research in the UK.