Trust your collaborators?

How many wasps are on this nest? What are their unique identifying marks? How many eggs, larvae, and pupae are in the nest? How many times does a given wasp dominate another? These are the questions that gave the numbers for my earliest graduate school research projects. I worried about every single one of these numbers. Is there a wasp hiding on the back of the nest? Have I recognized the painted marks correctly? Did I miss a young larva, calling it an egg (actually unlikely because eggs are pearly white and young larvae, though tiny, turn pinkish)? Was that really a domination, or just a wasp climbing on another as she flew off the nest? I worried about my data and did my best.

We learn what blind means in science. If there is an experiment, we make sure when we are scoring resulting actions we do not know which treatment a case received. There are lots of ways to do this. But the point is we don’t really even trust ourselves to be unbiased because we could inadvertently favor a hypothesis. How to avoid bias is something worth spending time on.

We keep careful data notebooks. In my lab these are still mostly on paper. The pages are numbered and dated. But increasingly data are collected directly onto loggers of various sorts. There we also preserve the details of data collection.

Once we have collected our data, we examine it for obvious errors. We graph it and look hard at the outliers. Are they real, or was there a data entry or other kind of mistake? Often we enter data twice and then compare as a way of checking that step. Of course, if the outliers are real, we keep them.

But what about our collaborators? What if they have not been as careful as we have? How can we tell that we have a sloppy or fraudulent collaborator? What checks should we do? These questions are timely because of the Jonathan Pruitt case, where collaborators who trusted his data and trusted him are now retracting papers. I am not going to summarize that case here but here are links to what Kate Laskowski, Dan Bolnick, and Science have said. Perhaps a link to the Dynamic Ecology blog  and my own previous post are also warranted.

I know you want an answer. Perhaps a great R package to run your collaborators through. Or a tutorial from Elizabeth Bik on how to recognize fraud in images of biological samples or gels. Maybe you want a personality test, or to learn of the traits common to those who cheat.

I have to disappoint you. I have racked my brain for what we might do but for anything I thought of I ran across two stumbling blocks. One was wondering what else the collaborators in the Pruitt case might have done. The other was thinking of my own collaborators and how I might behave differently in the future.

My conclusion was that, no, we cannot be constantly checking our collaborators’ data. No, we cannot ask that they show us their raw data. No, we cannot identify a flawed personality type that cheats. We are stuck. These techniques might work occasionally, but basically they will not work. They did not break open the Pruitt case. That, apparently, was thanks to an internal whistle blower with inside knowledge of the problem (see previous links).

Or are we stuck when it comes to our collaborators? I think there are only two possible solutions. The first is to stop collaborating. Collect all your data yourself. Then you will be sure of its accuracy. But what would that do to science? How greatly that would slow down progress?

What is the other solution? It is easy, but flawed. But it is the best option, by far.  It will not avoid any of the pain the Pruitt collaborators are currently suffering. But it is best for science. It is to simply trust your collaborators.

Odds are they are trustworthy. In most cases they are either replicating something you are doing but somewhere else, as with the big ecology experiments where plant communities are studied in similar ways all over the globe. Or they are providing an expertise in something you are not skilled in and have no hope of ever learning on top of everything else you do. In neither of these cases can you check their data in any meaningful way.

And no, trustworthiness does not increase with the number of beers you have shared with a collaborator. Cooperation, new ideas, and scientific fun may increase, but not necessarily good data.

For some people some of those collaborators will provide flawed data.  Should we limit the potential impact of this possibility by not collaborating too much with any one person? Here again, I would say no. I have spent much of my career collaborating mostly with one person and with a lot of others in addition. I know of other very productive long-term completely trustworthy collaborations.

So I suggest a simple personal solution. Trust your collaborators. But what does that mean for science when someone turns out to be fraudulent?  Where is the protection there? Here again I have an answer. It is that no important idea should be validated by work from just one lab or one set of collaborators.

We should think hard about this and keep track of what truths have come from just one group. If they are close to your field, spend some time redoing, or doing similar experiments so that science progresses on a firm foundation. And of course, remember to be absolutely trustworthy yourself. I hope someone will let us know what to now believe about spider social personalities.

About Joan E. Strassmann

Evolutionary biologist, studies social behavior in insects & microbes, interested in education, travel, birds, tropics, nature, food; biology professor at Washington University in St. Louis
This entry was posted in Collaboration, Data and analysis, Ethics. Bookmark the permalink.

15 Responses to Trust your collaborators?

  1. «…no important idea should be validated by work from just one lab or one set of collaborators.»
    Yes, this! I have come to value *ground breaking* studies only when published back-to-back with the competing study by another lab/set of collaborators, or reproduced by competitors within a reasonable time frame. Grant agencies and journal editors/publishers could really help to divert scientists from this *be the first* mantra.

  2. Hi Joan,
    I am afraid I don’t agree that collaborators should be trusted in general. Early in my career I tested the hypothesis that men would be superior to women at assessing family resemblances between parents and infants because women know who their children are, but men are always a bit uncertain and it matters. The student who gathered the data brought back results showing nice confirmation. But the results seemed a bit too good. I insisted on seeing the raw data and the distribution was a bit off. I held off on publication and got another student to do it all over again the next summer; this larger data set showed no difference at all. The first student seemed to have selectively not included some results that did not support the hypothesis. I don’t know if the biased data resulted from conscious fraud or just lack of care combined and wanting to be helpful.
    Ever since I have been suspicious of data that supports a hypothesis too nicely, have insisted on seeing raw data, and have tried hard to independently replicate findings. Those seem to me to be useful alternatives to global trust or mistrust of collaborators.
    But this is a good topic to discuss, esp as news about Eysenck’s frauds has just surfaced, along with a history of reluctance to investigate.
    Randy Nesse

    • Hi Randy,
      Yes you were right to question results that seem too perfect and come from a student working with you. I would certainly do exactly the same. It is not so hard when it is in your group and the kind of data you are expert in. And from any collaboration I would certainly call out anything that seemed too perfect. But so many collaborations these days the other party is collecting a kind of data totally outside one’s own wheelhouse. In my case, for example, chemical structures of small molecules. Within my lab group I routinely look at lab notebooks, but outside it is not going to work. I guess the situation is more nuanced than perhaps I indicated. Students always need to be taught. In your case you might even have explained possible issues to that first student and sent him or her out to try again.

  3. Perhaps readers would be interested in the final publication from that project.
    Sex differences in ability to recognize family resemblance
    January 1990Ethology and Sociobiology 11(1):11-21
    DOI: 10.1016/0162-3095(90)90003-O
    https://www.researchgate.net/publication/222467478_Sex_differences_in_ability_to_recognize_family_resemblance

  4. Pingback: Friday links: Jonathan Pruitt retraction fallout, AE “malpractice” at JTB, and more (UPDATEDx4) | Dynamic Ecology

  5. µ says:

    Joan, you write “So I suggest a simple personal solution. Trust your collaborators.”

    I am thinking: how naive.

    There are collaborators, and then there are collaborators.

    A few years ago, we had a chance to start a collaboration with Pruitt, but because our BS-meters had gone off the scale while listening to his seminar in my department, we decided not to. Earlier, a researcher in my lab had been unable to replicate some of the early work on spider personalities, and we decided to wait, observe.

    So now we have a chance to observe. Our BS-meters have been right all along.

    I have listened to too many talks when my BS-meter went off the scale.
    I have had collaborations when my BS-meter went off during collaborative work, and you are telling me that I should have continued those collaborations and not ask hard questions?

    I am with Randy Nesse on this one.

    • I guess I should clarify. I didn’t mean turn off all judgement and collaborate with anyone. I just meant that we cannot actually check on all our collaborators. I bet most collaborations fail because the other party does not come through with their share and not issues such as this case. Of course you should still use your judgement and ask hard questions.

  6. Pingback: Who can we trust? | Small Pond Science

  7. sgsterrett says:

    I think your reply saying that you should have been more nuanced is important. I am wondering if you would update your post. I don’t know if you checked principles of research ethics before writing your blog post, but the statement that “My conclusion was that, no, we cannot be constantly checking our collaborators’ data. No, we cannot ask that they show us their raw data.” is very worrisome.
    Yes, you absolutely should check the raw data of your collaborator if you are going to be the main author of a paper. Even other co-authors have some responsibilities concerning the data in a paper they are the co-author of. Did you review the research ethics principles that govern your field of research regarding co-authorship, before writing this blog post? I think it might be helpful to your readers if you did so, and discussed them the topic in that context.

    • The points you make on how much you should check your collaborators’ data are fine in a mythical world. In the real world, the best collaborators collect data that oneself would never be able to check. How could I check the way that a chemistry collaborator arrived at the structure of a small molecule? It is totally out of my wheelhouse. I suppose the real answer is for everyone to be open with their data and their analyses so anyone desiring can check any part of a paper they are expert in. Of course one should only collaborate with people one trusts, but as the #pruittdata case shows, even very smart, charming, collaborative people might have problems. Trust and extreme openness are the answers. I am learning ways to make our data even more open and I think we’ll learn a lot from following this case as to how to do this best.

      • sgsterrett says:

        There has already been a lot of work by professional groups about the role of coauthors. It is recognized that the responsibilities of co-authors might vary from discipline/field to discipline/field. (Council of Science Editors) That is why I asked what research ethics principles govern the cases you had in mind.

        There are various models, from the American Psychological Association using the principle that “”The primary author assumes responsibility for the publication, making sure that the data are accurate ” to the biomedical field using a “contributorship” model that isn’t as demanding of every co-author, but requires that which person is to be held accountable for which part of the work be clearly identified. So clearly a primary author on a paper in psychology is expected to look at the raw data, whereas that might not be the case in the biomedical sciences. However, there are still responsibilities for co-authors that do align with your remarks. They are more specific, though. If a co-author isn’t able to verify the data, then they would be required to specify what their more limited role in the work was. And sometimes someone should be acknowledged for their contributions but not be named as a co-author.

        I think the current research ethics principles already developed are in line with what you are saying about trust being indispensable, and transparency being needed, too. However, I think this has all been discussed already and should have been clear to the authors and co-authors in relatively recent publications. If not, then I would say something has gone wrong in disseminating these research ethics principles. I suggest that that is where some effort might profitably be spent, which is something that can be done immediately.

        Thus, I think it would be dismissive to say making co-authors accountable for looking at data only applies in an ideal world. The professionals and academics who worked so hard on guidelines for authorship on scientific papers did mean the guidelines to apply in the world we live in and publish in.

  8. µ says:

    This is in reply to the comment by sgsterrett February 9, 2020, 1:22 PM.

    In the area of Behavior/Evolution/Ecology (Pruitt’s research area), I know that the issues you raise have not been discussed adequately. I know, because this is also my research area.

    I have seen too much heterogeneity in how PIs and even heads of major research institutions conduct research.

    I have seen reluctance to discuss these issues by important researchers in the field, because discussion would highlight weaknesses in specific research programs.

    • sgsterrett says:

      Thanks for your comment. The discussion in the original blog was general. What I was saying is that the issues of responsible co-authorship in science have been discussed and there are many guidelines written. I quoted the one for the APA, and Pruitt is in a Dept of Psychology. Then I gave one at the other end of the spectrum in terms of co-author responsibility, from biomedicine. My point is that the blogpost was starting from scratch, with no reference to any existing guideline. I wanted to point out that in fact a lot of effort has been put into writing guidelines for responsible authorship. Perhaps they are not referred to as often as they should be.

      I do not doubt that what you report is accurate, if you mean discussions among actual practitioners. If there have not been discussions among practitioners in the field, such as among the authors of these papers, I wrote that: “then I would say something has gone wrong in disseminating these research ethics principles. I suggest that that is where some effort might profitably be spent, which is something that can be done immediately.”

      So, I hope we agree.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.