There is a sameness to human organizations. This means that if you are conducting an external review, you can probably do a surprising amount of it without knowing anything about the particular group at hand. Knowing this, then looking for the particulars of the group will both save a lot of time and keep you from missing something important. Another thing that is important from the start is to not leave everything about the structure of the review to the organizers. For example, I would like plenty of time for brainstorming. I would like time for one-on-one informal conversations among the people in the department and the committee. No one says anything very controversial in front of a group of peers to an entire committee. The most interesting news comes from leaks for a reason. As a reviewing committee, we want those leaks! And we want time to follow up on them.
Here are some thoughts on reviewing. These lists are overlapping, beginning with an overview in case you don’t have time to read the more detailed list of questions. The point is to do this carefully and critically. You will learn as much about yourself and your own institution as you do about the one you review. This is another reason why it is worth doing.
1. What is the goal of the review from the perspective of the party that is paying for it?
The more clearly articulated review goals are, the more straightforward they are to conduct. If the goals are too vague, the review will be difficult to do.
2. What are all the functions of the entity you are evaluating?
3. Who are the participants?
4. Follow the money.
5. Be sure to have time for one on one interactions.
1. Procedures are not clear.
2. There is a lack of transparency.
3. Communication problems – expectations unclear, information hard to get
4. Unfairness, unevenness, erratic behavior
5. Weak leadership
6. Poor balance of top down and bottom up.
7. Lack of democracy
8. Poor mentoring
9. Unprofessional, even abusive behavior
10. Bias, decision making conversations outside of formal structure.
An external review committee can help a program or a department by simply focusing on goals, structure, procedures, communication, transitions, and leadership. First, help the group articulate what their goals are. Until this is established, it is hard to do anything else. What is the structure of the group? What procedures ensure smooth functioning? Communication is important so everyone knows and understands goals and procedures and shares information of all kinds. Any human enterprise will have transitions. Handling transitions requires clear procedures and careful mentoring so people take the next step easily and gracefully. Finally, a strong and caring leader can keep the whole group focused, happy, and can keep the members on task and within guidelines. Here are more detailed questions.
ADVICE FOR PROVIDING AN EFFECTIVE REVIEW:
1. Make the review match what was asked of you to review.
2. Get information from different groups separately, so they can talk more freely.
3. Make frequent use of 3 x 5 cards or sticky notes on which people write their opinions and turn them in for openness, independency, and anonymity.
4. Have time for one on one conversations between review panel members and constituents.
5. Try to get a clear idea of what level controls or limits function so you don’t ask for change from a party that cannot provide it.
6. In the review have only a handful of main points, carefully documented and articulated.
7. Follow the main points with a longer list of smaller points.
8. The oral summary given at the end of the review may be more powerful than the later written review, so work hard to make that strong.
9. Separate issues you identify as problems from your suggested solutions because the group may arrive at different, better solutions from their better self-knowledge.
10. Be timely.
11. Ask for a follow-up to you or two another body, perhaps a new outside review committee.
Questions a discerning review committee should ask:
1. What is the goal of the review? Who asked for the review and who might benefit from a careful review?
2. What issues do the stakeholders think need to be addressed? If there were no issues, it is unlikely there would be a review. The constituents are most likely to know their own issues. Be sure that all stakeholders are talked to and not only in big groups, but also in mingling scenes so there can be one on one conversation.
3. What kinds and quantities of resources will be available to address concerns raised by this review? If there are no funds for change, for example, the reviewers should know that.
4. When was the last review, what concerns were raised, and how were they addressed? Prior reviews should be given to the committee. Reviews that raise concerns that are not then addressed do not bode well for future reviews.
5. Who will see the review?
6. What are the goals of this unit? If it is a university department, the goals may seem obvious. They are likely to put excellent research and teaching as major goals. More specific goals can be really useful. Does the department
7. What is the structure of the unit? This should include information on the decision-making power, levels of autonomy, and how the money flows.
8. What are the formal procedures? Clear, written procedures are essential to the smooth functioning of a unit. They allow for fairness and for both rewarding excellence and dealing with challenges.
9. How much power over their own courses, research, schedule do individuals have while meeting programmatic needs? We know people work best for themselves, but we also know that core needs need to be met.
10. How is information communicated within the group and to others? There is no point in having excellent programs and opportunities if the right people don’t know about them. Redundant communication is essential.
11. How are transitions handled? Few are in the same position for very long. Consider undergrads, graduate students, postdocs, longer term non-permanent people, professors, and program or department chairs. A clear trajectory and careful mentoring make transitions at all levels go more smoothly. When new people are hired, the process should be open and fair.
12. What is the group leadership structure? A good leader can make or break a group, but effective goals, structures, and procedures will limit the harm of an undesirable leader.
ETHICS, CLIMATE, AND BIAS QUESTIONS:
13. How are members educated about scientific integrity, implicit bias, respectful climate, and fairness? This should be understood to be important. There are some good materials out there that take the general form of scenarios that can be discussed. A strong leader will make it clear that bias and abuse are not ever tolerated.
14. What is the make-up of the group with respect to under-represented categories? If your junior people are more diverse than your senior people, you have a problem. What procedures are hurting diversity? Bad history is no longer an excuse.
15. How is hiring and promoting kept fair and open? This is a huge topic I have written about elsewhere, as have many others.
16. How is excellence judged and rewarded?
17. How are individuals or groups that are struggling helped?
18. How are big new initiatives encouraged and supported?
19. How is collaboration fostered?
20. How is the building maintained, shared, and kept functioning as a safe workplace?
21. How is new equipment obtained, maintained, and shared?
22. How are teaching needs identified, provided, and publicized?
23. What is the outreach situation? What do you do? How do you support it?
24. How is funding handled? What are the plans for replacing term-limited external funding?
25. If there is a sub-group structure, is it supported by actual collaborations within (network diagram) and is it useful?
26. How are students and more advanced people mentored?
27. How are newcomers helped with housing, visas, and settling in?
28. How often do you meet in various constellations (different chairs, students etc.)
29. What does each stakeholder identify as the biggest strength and weakness?
30. How are difficult people handled?
31. Do you have ways of facilitating large changes when desired?
32. What else do you want us to know?
Another perk of a review is getting to know the others on the committee. They will be amazing and you will learn from them so much.