How to do an external review, particularly of a university department

Yes, dear reader, I got to do a review in Uppsala in glorious May!

There is a sameness to human organizations. This means that if you are conducting an external review, you can probably do a surprising amount of it without knowing anything about the particular group at hand. Knowing this, then looking for the particulars of the group will both save a lot of time and keep you from missing something important. Another thing that is important from the start is to not leave everything about the structure of the review to the organizers. For example, I would like plenty of time for brainstorming. I would like time for one-on-one informal conversations among the people in the department and the committee. No one says anything very controversial in front of a group of peers to an entire committee. The most interesting news comes from leaks for a reason. As a reviewing committee, we want those leaks! And we want time to follow up on them.

Here are some thoughts on reviewing. These lists are overlapping, beginning with an overview in case you don’t have time to read the more detailed list of questions. The point is to do this carefully and critically. You will learn as much about yourself and your own institution as you do about the one you review. This is another reason why it is worth doing.

1. What is the goal of the review from the perspective of the party that is paying for it?
The more clearly articulated review goals are, the more straightforward they are to conduct. If the goals are too vague, the review will be difficult to do.
2. What are all the functions of the entity you are evaluating?
3. Who are the participants?
4. Follow the money.
5. Be sure to have time for one on one interactions.

Likely issues:
1. Procedures are not clear.
2. There is a lack of transparency.
3. Communication problems – expectations unclear, information hard to get
4. Unfairness, unevenness, erratic behavior
5. Weak leadership
6. Poor balance of top down and bottom up.
7. Lack of democracy
8. Poor mentoring
9. Unprofessional, even abusive behavior
10. Bias, decision making conversations outside of formal structure.

An external review committee can help a program or a department by simply focusing on goals, structure, procedures, communication, transitions, and leadership. First, help the group articulate what their goals are. Until this is established, it is hard to do anything else. What is the structure of the group? What procedures ensure smooth functioning? Communication is important so everyone knows and understands goals and procedures and shares information of all kinds. Any human enterprise will have transitions. Handling transitions requires clear procedures and careful mentoring so people take the next step easily and gracefully. Finally, a strong and caring leader can keep the whole group focused, happy, and can keep the members on task and within guidelines. Here are more detailed questions.

1. Make the review match what was asked of you to review.
2. Get information from different groups separately, so they can talk more freely.
3. Make frequent use of 3 x 5 cards or sticky notes on which people write their opinions and turn them in for openness, independency, and anonymity.
4. Have time for one on one conversations between review panel members and constituents.
5. Try to get a clear idea of what level controls or limits function so you don’t ask for change from a party that cannot provide it.
6. In the review have only a handful of main points, carefully documented and articulated.
7. Follow the main points with a longer list of smaller points.
8. The oral summary given at the end of the review may be more powerful than the later written review, so work hard to make that strong.
9. Separate issues you identify as problems from your suggested solutions because the group may arrive at different, better solutions from their better self-knowledge.
10. Be timely.
11. Ask for a follow-up to you or two another body, perhaps a new outside review committee.

Questions a discerning review committee should ask:


1. What is the goal of the review? Who asked for the review and who might benefit from a careful review?
2. What issues do the stakeholders think need to be addressed? If there were no issues, it is unlikely there would be a review. The constituents are most likely to know their own issues. Be sure that all stakeholders are talked to and not only in big groups, but also in mingling scenes so there can be one on one conversation.
3. What kinds and quantities of resources will be available to address concerns raised by this review? If there are no funds for change, for example, the reviewers should know that.
4. When was the last review, what concerns were raised, and how were they addressed? Prior reviews should be given to the committee. Reviews that raise concerns that are not then addressed do not bode well for future reviews.
5. Who will see the review?


6. What are the goals of this unit? If it is a university department, the goals may seem obvious. They are likely to put excellent research and teaching as major goals. More specific goals can be really useful. Does the department
7. What is the structure of the unit? This should include information on the decision-making power, levels of autonomy, and how the money flows.
8. What are the formal procedures? Clear, written procedures are essential to the smooth functioning of a unit. They allow for fairness and for both rewarding excellence and dealing with challenges.
9. How much power over their own courses, research, schedule do individuals have while meeting programmatic needs? We know people work best for themselves, but we also know that core needs need to be met.
10. How is information communicated within the group and to others? There is no point in having excellent programs and opportunities if the right people don’t know about them. Redundant communication is essential.
11. How are transitions handled? Few are in the same position for very long. Consider undergrads, graduate students, postdocs, longer term non-permanent people, professors, and program or department chairs. A clear trajectory and careful mentoring make transitions at all levels go more smoothly. When new people are hired, the process should be open and fair.
12. What is the group leadership structure? A good leader can make or break a group, but effective goals, structures, and procedures will limit the harm of an undesirable leader.


13. How are members educated about scientific integrity, implicit bias, respectful climate, and fairness? This should be understood to be important. There are some good materials out there that take the general form of scenarios that can be discussed. A strong leader will make it clear that bias and abuse are not ever tolerated.
14. What is the make-up of the group with respect to under-represented categories? If your junior people are more diverse than your senior people, you have a problem. What procedures are hurting diversity? Bad history is no longer an excuse.
15. How is hiring and promoting kept fair and open? This is a huge topic I have written about elsewhere, as have many others.


16. How is excellence judged and rewarded?
17. How are individuals or groups that are struggling helped?
18. How are big new initiatives encouraged and supported?
19. How is collaboration fostered?


20. How is the building maintained, shared, and kept functioning as a safe workplace?
21. How is new equipment obtained, maintained, and shared?
22. How are teaching needs identified, provided, and publicized?
23. What is the outreach situation? What do you do? How do you support it?
24. How is funding handled? What are the plans for replacing term-limited external funding?
25. If there is a sub-group structure, is it supported by actual collaborations within (network diagram) and is it useful?
26. How are students and more advanced people mentored?
27. How are newcomers helped with housing, visas, and settling in?
28. How often do you meet in various constellations (different chairs, students etc.)
29. What does each stakeholder identify as the biggest strength and weakness?
30. How are difficult people handled?
31. Do you have ways of facilitating large changes when desired?
32. What else do you want us to know?

Another perk of a review is getting to know the others on the committee. They will be amazing and you will learn from them so much.

Posted in Department Evaluation, Department politics | Tagged , , | Leave a comment

Radical thoughts on awards, or who should get into the National Academy of Sciences

The annual meeting of the National Academy of Sciences ended with a President’s Dinner at a museum, and the business meeting, with announcement of new members. There is intense silence as we stare at the three screens of names. The first two are the 84 who made it in. The third screen shows those who nearly made it, but did not, beginning with the most unlucky number 85. We cannot photograph the screens, but that third screen is what has our most attention.
I look at those names and think of how many great scientists all three screens represent. I wonder how joining this prestigious body will change these people. I hope that the change is for the better and that it will mean they will bring a broader perspective to their home community and their home perspective to national issues.
And those thoughts bring me to my more radical thought. It is based on two things. The first is about the point of the National Academy, to be a pool of experts to advise the government. Does that not mean that it would be good if they come from diverse areas in all respects? And won’t they be good for their home community with the broader perspective I like to think Academy membership brings? They will get that perspective from the duties of members, whether attending meetings, editing for our journal, or serving on National Research Council committees. If this is the case, then it is also a good thing for as many communities as possible to benefit from NAS members. It is not good for our nation to have regions that lack NAS members. It is not good for our nation to have members concentrated in a few places. This is my first point, that the nation’s interests are served best from diverse perspectives. I do not think this is a radical thought.
My second point is the radical half of the idea. This opinion I am about to present is verifiable, though I am not social scientist enough to know the literature in this area. It is that we are very poor in making fine judgements about excellence in people. We try to do this for many things. We try to be fair. First, we divide knowledge into disciplines and evaluate what we know. Someone who studies stars will not know the nuances of discovery in microbes. We divide things up more finely, often conceptually. Biology is divided into several sections My section is 27, evolution, and my secondary section is 26, genetics. But even within evolution there are areas I know better and areas I know less well, though the experience of being an academy member broadens one.
Once we have divided science and scientific approaches into sections, we can make lists of excellent people more easily since we can read their papers and understand their scientific advances. When we do this, it is easy to see that researchers are not all equal. Some simply have better ideas, test them more rigorously, advance the field more than others. So don’t think I would ever claim everyone is the same when it comes to science. I do not think that.
But what I do think is that truly phenomenal advances are rare. If we relied on them alone for giving recognition, in most years no one would be recognized, perhaps not even in most decades. What is more common is that someone takes a certain area and explores it thoroughly, making excellent advances in both understanding that area and in using

Mike Donoghue and Jane Lubchenco calling a brand new member.

understanding of that area for more general conceptual advances. But these things occur on many fronts, so how do we choose between people advancing in similar ways but different systems? How do we choose between the snake person and the monkey flower person?
I would argue that we cannot really use excellence alone as the final criterion. If we already have 5 monkey flower people and no snake people, take the snake person. If we have people from a given place in excess, take the person from elsewhere. The same of course goes for diversity in gender, race, and ethnicity.
Would it be true that if you make it to the final ballot, you are indistinguishable from those at the top of that ballot? Maybe, maybe not. How about top half of that ballot? Surely at some point you would say that you cannot distinguish people, so then use other criteria. Pick someone from a non-coastal state. Pick a woman. We are only human and cannot make these fine decisions easily. Why do we pretend we can?
How about other honors, awards, and the like? I would argue there too that we cannot make the final judgements very clearly very often, so use other criteria once the field is narrowed down. How many more women might NSF’s Waterman award go to if they decided they could not distinguish in excellence among the top group?

Posted in Awards and prizes, equity, Judging | Tagged , , , | 1 Comment

Why expanding the requirements for the Waterman Award won’t fix it

The National Science Foundation gives out an annual prize to their choice for the top scientist, the Waterman Award. Actually, it is a committee that does the choosing. You can see for yourself who is on it, here. The award has not gone to a woman in a long time. This year it even went to two people and neither were women. I do not seem to be alone in thinking there is a problem with this; look here, here, and here.

Yet NSF is one of the fairest places around. I was at an Advisory Committee meeting for the Biology Directorate earlier this week and women were clearly well represented in leadership positions. The head of NSF entirely is a woman, France Córdova. There is other evidence of fairness at NSF I can’t go into here. So I’m guessing they are as frustrated as anyone.

Does this mean we have to take some people’s advice and throw up our hands and simply have a boy’s pile and a girl’s pile? Is it really so impossible to avoid bias even when it is recognized? I do not know the answer to that.

But I am a bit discouraged by the NSF response to the problem. They have decided that the problem is not with bias in the nominations or bias in the committee, but simply that there are not enough good young women. So they have lengthened the time window to 10 years since Ph.D. or 40 years old. This will certainly increase the pool, but I submit that it is narrow thinking by the committee as to what is outstanding and perhaps narrowness in nominations that is the problem, not the dearth of excellent women in the category. In fact, I know some personally that were nominated, some even by me, but I’ll keep this private.

Bias is a huge problem, as we heard about in a gripping talk today here at the National Academy of Sciences from Jennifer Eberhardt. I can hardly do justice to her compelling talk, but I’ll try on a couple of points backed up with the papers. In the first that she talked about she flashed photos of men of different races or random photos so quickly one could not know if one actually saw them. She then gradually filled in an image of a gun. The people tested, including policemen, were quicker to see a gun from the incomplete drawing if they had subliminally seen pictures of black men. The other study flashed visible pictures of people of different races in different frequencies and in that one if there was a larger percentage of blacks, people were more likely to support a punitive form of three strikes and lifetime jail. She has more. The point is we are biased. All of us. Better to understand it and try to make conscious amends.

Studying Jennifer’s work should be required for all. There are others in her field and in behavioral economics that treat these issues. Only with understanding can we have any hope of advancing to get rid of things that hurt people’s lives far more profoundly than the inequity in the Waterman does.

Posted in Uncategorized | Leave a comment

Can we fix inequity in awards for women scientists?


Our wonderful undergraduates at their poster session look towards a world with less bias.

Unfairness to women in academic recognition is common. Anne Lincoln and co-authors, for example published on the Matilda Effect on awards and prizes in science. Women achieve much more for much less recognition, if they get credit for their own ideas and inventions at all. Shelley Correll has written about biases against women, as have many others. I am not going to review this work today. This is an action plan and some principles.

  1. My most basic principle is that no one wants to be biased and no one, not men nor women, consciously favor men, male-led projects, or erroneously attribute ideas to men. I start from this perspective that scientists are good, well-meaning people who try hard to be fair. Of course there are the others, but if all the good people formulated an action plan of their own, the problem would largely go away.
  2. We do not understand our own motivations very well, or even at all. We are evolved creatures, evolved to behave in certain ways and to make judgements quickly that were good enough sometime in our evolutionary past. Assuming these judgements are unbiased is silly. This is true for the legal system where one study found that judges gave much harsher sentences before lunch (blog here), to others that show the shortcomings of the simultaneous line-up instead of a sequential one for identifying ill doers by eye witnesses. I think it is true for pretty much everything. I get a lot of insight from reading a lot of social science literature. Daniel Ariely comes to mind, but that could be a false attribution because he spoke twice at Wash U. These two principles motivate the action plan in the next three points.


    Ursula Goodenough, mentor to many, wonderful colleague, and scrupulously fair, at her retirement festivities.

  3. Take the nomination pledge. With this you agree to nominate worthy women and under-represented minorities for awards broadly and in your discipline. These can be society prizes, university prizes, or others. Educate yourself as to what is out there and nominate.  You decide what is a good number for you. It could be one a year, one a month, or one for some particular awards. Make it a normal part of your academic responsibilities, like letters of recommendation are.  Encourage your colleagues to do likewise.
  4. Don’t use recall to choose whom to nominate. It gives our brains too much leeway for bias. I suppose we would all think spontaneously of the same handful of people. Instead use recognition. Get lists of society members. Look at editorial boards. See who on those lists is worthy and unrecognized. Keep a list of people to nominate and fill them in when something comes up. Don’t just stick to people from your own institution or own sub sub discipline. Remember, recognition, not recall.
  5. Finally, if you find yourself on a judging committee, remember how bad we are at evaluating quality or predicting who will shine in the future. Remember to be broad about discipline and to avoid cliques and narrow windows of prestige. Think broadly. Choose women and under-represented minorities. Look at ideas and outcomes. Don’t count citations or publications.

I think we can fix this problem with these two perspectives and these three simple steps. After all, there are plenty of outstanding scientists in the target communities. Help get them the recognition they deserve!

Posted in Uncategorized | 3 Comments

Grant review: best proposal or most needy scientist?

Budding scientists hard at work on their posters.

In a few short hours I read nine proposals from graduate students interested in funding for their research. Because this was a focused call, they were largely similar. They all involved field work, natural history, and queries into the relationship of humans and their environment. But they varied in all the details, some including more anthropology and archaeology, others more systematics, or ecology, others behavior and economics. Some had simple techniques and others more complex ones. The questions asked varied and were all more empirical than conceptual. I wanted to fund all of them, but we could fund only one or two.

How could I rank them? Why should it be so hard? After all, I had their resumés, their letters of recommendation, and detailed proposals that included budgets. Some were clearly more polished than others. Some students had already published or attended scientific meetings. The letters of recommendation were not from people from the USA and so had a different tone, one I found harder to parse.

But my real problem was not any of these. It was that what I really wanted to know was where the funding dollars would do the most good. If I could answer that question, then whom to fund should be clear. But this brought up another question, how to define the most good. Assuming I could judge these things, it could be the best person, the best proposal, or the most needy person. Ideally, it would have to be a mix of these to make the most difference to science. A stellar person who also had other funds might not be the best choice. Likewise, a very needy researcher whose proposal was very weak might not be a good choice either. What I wanted to identify was the person for whom these funds could really make a difference, someone for whom these modest funds could launch a career.

Isn’t where the funds can do the most good a worthy goal for all funding decisions? I think it should be. Now I will digress from my recent experience to that with which I am most familiar, the US National Science Foundation. My sense is that what we do on panel there is judge the best science and the best broader impacts. But this does not mean that the other things are not considered. It just means that it is more efficient and accurate to separate judging the science from judging other things. That is why the panels do not make final decisions. I totally agree with this. After all, what if there were no proposals from say 10 states (you pick) that were scientifically as high as the ones from Harvard? Would it make sense for us as a nation to just fund Harvard and never fund those other states? I say no. For that would mean a promising students from, say Alabama, would have no opportunity to be exposed to teaching and research from someone with NSF funding. It would reinforce our already too strong class structure and be bad for a nation that wants excellence from all regions.
Continuing along the line of considering our national funding agencies, I do not think it is best for our nation to put NSF and NIH funds into the hands of a few. I think it is horrible that one person can receive many NIH grants, for example. It is bad for discovering the best science, bad for encouraging the most scientists, bad for discovering the best ways of dealing with our human predicament. What the limit should be in these days of collaborative science might be complex, but that is a solvable question once the principle is accepted.I think the earlier people are in the education spectrum, the less we should judge and the more we should encourage. I think science fairs for kids that pick nationwide winners are a joke. They reward access to labs as much as anything. At the grad student level, I am also in favor of smaller grants for more people.So I ranked the nine proposals and the nine proposers since I had to. I hope they all shine.

Posted in Awards and prizes, Graduate school, Grant proposals, Grants, NSF | Tagged , , | 3 Comments

Can you answer this crucial scientific question?

“Where did you see that,” may not seem like the most crucial question, but for natural science, history, geography, archaeology, and many other fields, provenance is crucial information. The volunteers that keep eBird useful will challenge you if you claim to have seen a bird at a time or place it was rare. Fossils are not nearly as useful and are useless for many questions if they have been dug out of the rock layer they come from since that takes away age information. Some of my systematics friends are unwilling to identify samples that lack their GPS coordinates.

Every field biologist knows that putting the date and the location on notes and samples is an essential first step. We use GPS for our soil collections for our social amoebae, but even back in the salad days of research for me, I mapped things. I could tell you where each wasp nest was in a field, and where that field was on a map. I knew counties of states and countries. For some nests, I knew paces and compass directions.

Even researchers that never get outside need to know the provenance of their cell lines. How many studies have claimed one line but actually been on the overpowerful HeLa lines? The careful researcher verifies the provenance of their cell lines, assures themselves that the knockout is in the gene they think it is.

Why is provenance so important? It is because nothing is alone and everything happens in context. Provenance helps us understand that context. Provenance helps link the studies I do with the studies others do. Appreciating provenance is a characteristic of a careful researcher.

This is not just true for natural sciences. It is true for archaeology where relics without their context are lost of their meaning. It is why theft at archaeological sites is so terrible. It is particularly true for history. How can we know the why and what of events without clear documentation of when and where? Provenance is what allowed my husband’s father, Donald Queller, to make the links he made in his books on Venice and the Crusades.

So imagine my horror when I went downstairs and discovered my 90-year-old father, not senile at all, thought it would be a good idea to separate from their envelopes the letters my grandmother who was still in Germany wrote to her husband who had made it to the USA in 1936? The family soon followed, and those letters are history. The envelopes are their provenance. I sure do hope I can convince him to put them back together, but I’m not optimistic. Those of you who know me, will now know where I got my attitude.

For you, just be sure you can answer the crucial questions about your data and your samples and can tell the world exactly where and when you got them.

Posted in Communication, Field work, Research | Tagged , , , , , | 2 Comments

Will your paper meet the reporting checklist requirements for Nature Ecology and Evolution?

Perhaps one of the biggest shocks to students new to research is how slow and painstaking it is. More than one has decided it is no fun at all, nothing like reading cool articles or watching nature videos. And I’m not here to tell you otherwise. If anything, you should start out even more slowly and carefully. Worry about everything. Record everything. But what does that even mean?

Well, the new journal from the for-profit line from Nature (actually Springer Nature and don’t get me started), this one being Nature Ecology and Evolution has a very useful checklist. I suppose a lot of other journals have this too, but here is the one I came across. For every figure, you have to have in the caption (or methods if too long)

sample size as a number,

sample collection methods and if technical or biological replicates (what these mean with microbes can be challenging)

how many times the experiment was replicated

definitions of statistical methods and measures

for sample sizes under 5 each data point has to be plotted

clear information on tests, whether one or two sided, indication of centrality and error bars.

Then they have several other sections where you have to report exactly where in the text something was done, like sample size selection, inclusion criteria, randomization and blinding procedures, normality of data and more.

Then you have to make it clear where your data will be publicly available and how your code will be published. They encourage Data Descriptors “to maximize data reuse.” but that link did not work for me.

Plos Biology also has some lists for good standards for meta analyses and the like, here.


Chandra Jack and Neil Buttery know all about careful science

So pay attention to these standards when you are designing your experiments. I’ve always said when students have asked me to consider how something will look in the Methods section. This is more specific. I particularly like the requirement that if there are fewer than 5 points, to show them, not make a misleading bar graph.

I also feel like our statistical analyses are stuck in the pre-computer days and we should probably ditch them and start over, beginning with randomization tests as the standard, but that is a post for another time.

Posted in Uncategorized | Leave a comment