‘The PubPeer conundrum:’ One view of how universities can grapple with a ‘waterfall of data integrity concerns’

As Retraction Watch readers no doubt know, PubPeer has played a key role in a growing number of cases of misconduct, allowing sleuths to publicly shine light in shadowy corners and prompting action by many universities. (Disclosure: Our Ivan Oransky is a volunteer member of the PubPeer Foundation’s board of directors.) But that has also meant that universities can feel overwhelmed by a deluge of PubPeer comments.

In a new article, three attorneys from Ropes & Gray in Boston who advise universities on such cases, along with Barbara Bierer, a researcher and former research integrity officer at the Brigham and Women’s Hospital, also in Boston, examine “the origins of PubPeer and its central role in the modern era of online-based scouring of scientific publications for potential problems and outlines the challenges that institutions must manage in addressing issues identified on PubPeer.” Attorneys Mark Barnes, Minal Caron and Carolyn Lye, and the Brigham’s Barbara Bierer, also recommend ways federal regulations could change to make the investigation process more efficient. We asked them to answer some questions about the article.

What prompted you to write this piece? 

As acting or designated research integrity officers (“RIOs”), and as legal counselors to other institutions and their RIOs, we have fielded innumerable questions about how to deal responsibly with the waterfall of data integrity concerns posted on PubPeer and other websites.  The RIO community is, in general, under-resourced to be able to investigate every concern that is posted, and we have been asked many times:  Are there best practices here?  Do the federal regulations at 42 C.F.R. Part 93 (“Part 93”) require us to review these websites and assess all concerns posted about research that has been conducted at our institution or by our researchers’ work?  We wrote this piece to try to respond to these community concerns in an organized way, to trigger a larger community discussion on these issues. 

Additionally, we find that many members of the data integrity community may be unaware of the genesis of PubPeer, and this lack of knowledge can lead to wild theories about who began and who operates PubPeer.  But in fact, there is a significant amount of information in the public domain about PubPeer, if you dig for it (albeit, mostly from 2016 and earlier).  That is why, as a secondary objective, we wanted to provide a detailed overview of PubPeer, its animating rationales, and origins.

As you note, some of you have been acting or designated RIOs at various institutions. How did that experience inform your thinking about PubPeer? 

As designated RIOs, we have been directed by ORI and other government entities to regard specific PubPeer postings as formal allegations, and either to conduct an assessment of those allegations or to take them directly to inquiry.  ORI also sometimes voices a requirement that in the course of an inquiry or investigation, RIOs must review online sources of public comments to ensure that all concerns related to a respondent’s past published research have been reviewed and considered (particularly if an inquiry committee has decided to dismiss a set of allegations at the inquiry stage, instead of proceeding to full investigation).  Our experience is similar to the reported experiences of many RIOs.

You write that “the influx of PubPeer comments received by institutions and triaged by RIOs and other research compliance personnel has placed a significant burden on research institutions already stretched thin by ongoing research misconduct proceedings and other research integrity and research compliance obligations.” For the benefit of Retraction Watch readers, what are these other obligations? Are they required by federal law or regulation? 

First and foremost, RIOs are required under Part 93, other federal agency regulations, and their institutional policies, to assess all complaints made to them about research integrity concerns; they then must shepherd many cases through inquiry and investigation. Although rigorous due process protections are needed to ensure fairness for respondents, they can cause processes to be slow and cumbersome.  Further, in many cases, the allegations can be so numerous and so complex that speedily completing the misconduct process is just not possible.  In addition, the role of a RIO is not always a full-time position, and many RIOs serve in other roles as well (e.g., Vice Provost/Associate Dean for Research) or have many other intra-institutional responsibilities (e.g., NIH- and NSF-required responsible conduct of research (“RCR”) training, other research-related compliance tasks).  RIOs often simply do not have sufficient capacity to be able to scour these public science websites, identify all relevant postings, and consider their implications.

What is the role of ORI in responding to PubPeer comments? Is there a mechanism for that? 

ORI and analogous offices in other federal agencies (e.g., DoD, NSF, DoE) can be effective in screening publicly posted concerns and bringing to a RIO’s attention those concerns that merit attention and assessment.  ORI often refers allegations posted to PubPeer to the corresponding author’s institution, instructing the institution to conduct an assessment or inquiry of the allegations.  Yet ORI typically does not indicate how it became aware of allegations referred to an institution.  It may be that ORI does not proactively monitor PubPeer, but instead conducts a screening review of any PubPeer-sourced allegations that it receives from complainants and then passes on the relevant institution and allegations that seem cognizable.

ORI is in the midst of reviewing the regulations that guide it, which have not been updated since 2005. Several of you have written a response to their call for feedback. Do you think there is an opportunity to address PubPeer in new regulations? You argue for a narrowing of the scope of what’s known as the “subsequent use exception,” including limiting any investigations to work in the last six years. What is that exception, and how would this help?

As we say in our published paper, it would be very useful if RIOs could be afforded, under Part 93, greater discretion to review concerns and allegations and try to resolve them with the authors and any cognizant journals at assessment, but this, as we argue, should apply only to situations in which original, accurate research data are available and the problem is reasonably identified by the RIO as “honest error.”  This procedural reform could speed the process in a fault-free and constructive way and, at the same time, would free up time for other, less readily resolvable cases. 

Making the matter even more complicated, while the ORI research misconduct regulations contain a statute of limitations (a six-year lookback period) and the subsequent use exception to the six-year lookback period, other federal agencies (such as NSF) have no statute of limitations embedded in their research misconduct regulations. Another possible reform we suggest is narrowing the exceptions to that statute of limitations, which would reduce the burden on all institutional research misconduct processes, thus allowing more sustained focus on more recently published research.  We recognize, however, that this reform would allow some truly egregious behavior to evade scrutiny under the research misconduct process, which is a serious consideration militating against such a reform. 

In a blog post about your paper, Dorothy Bishop says that “what is needed is hard data on how many reported cases of misconduct proceed to a full investigation, and how many subsequently are found to be justified.” Are data available on how often PubPeer comments turn out to be unfounded? If not, are there best guesses? 

We are not aware of any available data nor any reliable way in which this could be assessed, particularly because of the confidentiality obligations in research misconduct proceedings imposed by the applicable federal regulations and most institutional policies.  We agree entirely with Dr. Bishop that the collection and analysis of this information – if it were possible – would be very useful.

Are you concerned that raising the bar for investigations will mean many cases that should be investigated aren’t, and that significant parts of the scientific literature will remain, or become more, corrupted? 

We’re not sure what you mean by “raising the bar,” but we can say this:  the best deterrent to falsification and fabrication of data and plagiarism is to identify serious concerns; address them quickly, thoroughly and fairly; correct the scientific record at the earliest possible time; and impose sanctions on those who have committed misconduct.  No system can detect and prosecute all compliance violations.  As Cesare Beccaria said in regard to the deterrent effect of criminal law, the severity of punishment is less important than its swiftness and its certainty.  Here, we need credible, reliable, swift deterrence to re-order the system and reinforce sound ethics in research, and we believe that significant time delays do hinder the deterrent effect of the current approach.

Sleuths have over the years said that many institutions already find reasons to dismiss cases that have merit, and you acknowledge in your paper that “if ORI were to give institutions greater latitude in their institutional decision-making at the early stages of a research misconduct proceeding, there would probably be a wide variety of outcomes,” some of them likely negative. How would your proposal guard against that? Is there an argument for adding more resources and improving institutional culture rather than, or in addition to, limiting what gets investigated? 

Of course, it would be useful for every research institution to examine its research misconduct workload (as well as the workload that has been delayed, deferred and/or avoided) and compare that workload to the available resources.  But we believe that the ability to resolve “honest error” cases quickly and to document the reasons why each case was resolved, if accompanied by robust auditing of this process by ORI and other cognizant agencies, could speed the process in many cases, allow prompt correction of the scientific record, and free up RIOs to focus on those researchers whose actions are clearly not due to “honest error” but instead may have been intentional, knowing or reckless in their dissemination of false data or plagiarized work.

Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].