As Retraction Watch readers no doubt know, PubPeer has played a key role in a growing number of cases of misconduct, allowing sleuths to publicly shine light in shadowy corners and prompting action by many universities. (Disclosure: Our Ivan Oransky is a volunteer member of the PubPeer Foundation’s board of directors.) But that has also meant that universities can feel overwhelmed by a deluge of PubPeer comments.
In a new article, three attorneys from Ropes & Gray in Boston who advise universities on such cases, along with Barbara Bierer, a researcher and former research integrity officer at the Brigham and Women’s Hospital, also in Boston, examine “the origins of PubPeer and its central role in the modern era of online-based scouring of scientific publications for potential problems and outlines the challenges that institutions must manage in addressing issues identified on PubPeer.” Attorneys Mark Barnes, Minal Caron and Carolyn Lye, and the Brigham’s Barbara Bierer, also recommend ways federal regulations could change to make the investigation process more efficient. We asked them to answer some questions about the article.
What prompted you to write this piece?
As acting or designated research integrity officers (“RIOs”), and as legal counselors to other institutions and their RIOs, we have fielded innumerable questions about how to deal responsibly with the waterfall of data integrity concerns posted on PubPeer and other websites. The RIO community is, in general, under-resourced to be able to investigate every concern that is posted, and we have been asked many times: Are there best practices here? Do the federal regulations at 42 C.F.R. Part 93 (“Part 93”) require us to review these websites and assess all concerns posted about research that has been conducted at our institution or by our researchers’ work? We wrote this piece to try to respond to these community concerns in an organized way, to trigger a larger community discussion on these issues.
Additionally, we find that many members of the data integrity community may be unaware of the genesis of PubPeer, and this lack of knowledge can lead to wild theories about who began and who operates PubPeer. But in fact, there is a significant amount of information in the public domain about PubPeer, if you dig for it (albeit, mostly from 2016 and earlier). That is why, as a secondary objective, we wanted to provide a detailed overview of PubPeer, its animating rationales, and origins.
As you note, some of you have been acting or designated RIOs at various institutions. How did that experience inform your thinking about PubPeer?
As designated RIOs, we have been directed by ORI and other government entities to regard specific PubPeer postings as formal allegations, and either to conduct an assessment of those allegations or to take them directly to inquiry. ORI also sometimes voices a requirement that in the course of an inquiry or investigation, RIOs must review online sources of public comments to ensure that all concerns related to a respondent’s past published research have been reviewed and considered (particularly if an inquiry committee has decided to dismiss a set of allegations at the inquiry stage, instead of proceeding to full investigation). Our experience is similar to the reported experiences of many RIOs.
You write that “the influx of PubPeer comments received by institutions and triaged by RIOs and other research compliance personnel has placed a significant burden on research institutions already stretched thin by ongoing research misconduct proceedings and other research integrity and research compliance obligations.” For the benefit of Retraction Watch readers, what are these other obligations? Are they required by federal law or regulation?
First and foremost, RIOs are required under Part 93, other federal agency regulations, and their institutional policies, to assess all complaints made to them about research integrity concerns; they then must shepherd many cases through inquiry and investigation. Although rigorous due process protections are needed to ensure fairness for respondents, they can cause processes to be slow and cumbersome. Further, in many cases, the allegations can be so numerous and so complex that speedily completing the misconduct process is just not possible. In addition, the role of a RIO is not always a full-time position, and many RIOs serve in other roles as well (e.g., Vice Provost/Associate Dean for Research) or have many other intra-institutional responsibilities (e.g., NIH- and NSF-required responsible conduct of research (“RCR”) training, other research-related compliance tasks). RIOs often simply do not have sufficient capacity to be able to scour these public science websites, identify all relevant postings, and consider their implications.
What is the role of ORI in responding to PubPeer comments? Is there a mechanism for that?
ORI and analogous offices in other federal agencies (e.g., DoD, NSF, DoE) can be effective in screening publicly posted concerns and bringing to a RIO’s attention those concerns that merit attention and assessment. ORI often refers allegations posted to PubPeer to the corresponding author’s institution, instructing the institution to conduct an assessment or inquiry of the allegations. Yet ORI typically does not indicate how it became aware of allegations referred to an institution. It may be that ORI does not proactively monitor PubPeer, but instead conducts a screening review of any PubPeer-sourced allegations that it receives from complainants and then passes on the relevant institution and allegations that seem cognizable.
ORI is in the midst of reviewing the regulations that guide it, which have not been updated since 2005. Several of you have written a response to their call for feedback. Do you think there is an opportunity to address PubPeer in new regulations? You argue for a narrowing of the scope of what’s known as the “subsequent use exception,” including limiting any investigations to work in the last six years. What is that exception, and how would this help?
As we say in our published paper, it would be very useful if RIOs could be afforded, under Part 93, greater discretion to review concerns and allegations and try to resolve them with the authors and any cognizant journals at assessment, but this, as we argue, should apply only to situations in which original, accurate research data are available and the problem is reasonably identified by the RIO as “honest error.” This procedural reform could speed the process in a fault-free and constructive way and, at the same time, would free up time for other, less readily resolvable cases.
Making the matter even more complicated, while the ORI research misconduct regulations contain a statute of limitations (a six-year lookback period) and the subsequent use exception to the six-year lookback period, other federal agencies (such as NSF) have no statute of limitations embedded in their research misconduct regulations. Another possible reform we suggest is narrowing the exceptions to that statute of limitations, which would reduce the burden on all institutional research misconduct processes, thus allowing more sustained focus on more recently published research. We recognize, however, that this reform would allow some truly egregious behavior to evade scrutiny under the research misconduct process, which is a serious consideration militating against such a reform.
In a blog post about your paper, Dorothy Bishop says that “what is needed is hard data on how many reported cases of misconduct proceed to a full investigation, and how many subsequently are found to be justified.” Are data available on how often PubPeer comments turn out to be unfounded? If not, are there best guesses?
We are not aware of any available data nor any reliable way in which this could be assessed, particularly because of the confidentiality obligations in research misconduct proceedings imposed by the applicable federal regulations and most institutional policies. We agree entirely with Dr. Bishop that the collection and analysis of this information – if it were possible – would be very useful.
Are you concerned that raising the bar for investigations will mean many cases that should be investigated aren’t, and that significant parts of the scientific literature will remain, or become more, corrupted?
We’re not sure what you mean by “raising the bar,” but we can say this: the best deterrent to falsification and fabrication of data and plagiarism is to identify serious concerns; address them quickly, thoroughly and fairly; correct the scientific record at the earliest possible time; and impose sanctions on those who have committed misconduct. No system can detect and prosecute all compliance violations. As Cesare Beccaria said in regard to the deterrent effect of criminal law, the severity of punishment is less important than its swiftness and its certainty. Here, we need credible, reliable, swift deterrence to re-order the system and reinforce sound ethics in research, and we believe that significant time delays do hinder the deterrent effect of the current approach.
Sleuths have over the years said that many institutions already find reasons to dismiss cases that have merit, and you acknowledge in your paper that “if ORI were to give institutions greater latitude in their institutional decision-making at the early stages of a research misconduct proceeding, there would probably be a wide variety of outcomes,” some of them likely negative. How would your proposal guard against that? Is there an argument for adding more resources and improving institutional culture rather than, or in addition to, limiting what gets investigated?
Of course, it would be useful for every research institution to examine its research misconduct workload (as well as the workload that has been delayed, deferred and/or avoided) and compare that workload to the available resources. But we believe that the ability to resolve “honest error” cases quickly and to document the reasons why each case was resolved, if accompanied by robust auditing of this process by ORI and other cognizant agencies, could speed the process in many cases, allow prompt correction of the scientific record, and free up RIOs to focus on those researchers whose actions are clearly not due to “honest error” but instead may have been intentional, knowing or reckless in their dissemination of false data or plagiarized work.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
I’m on author on a paper that is going through resubmission. New data recently revealed that the conclusions of the paper may be flawed and we need to collect more data… The corresponding author is pushing to resubmit without that data stating that we will correct the record in follow up papers. Is this how these things start?
Yep. You’ve got a bad apple.
My advice: You should get a lawyer (not affiliated with the institution) to determine what to do immediately.
If you ignore this and let the author convince the rest of you to continue, you risk being co-opted into a scheme from which you can never escape: you are in on it now.
Better to take some risks now when the stakes are at the smallest they will be, than let this grow into something worse.
I don’t know if Eugenie Reich can advise you directly, but you can reach out to her anonymously (get a protonmail account and do not use institutional resources to access the account) and check with her. You can Google her and there’s a RW piece about her too.
Good luck.
I think she (Mandy) can avoid this problem via much easier ways too. She can simply refuse (in WRITTEN — EMAIL) to submit the paper in its current form to any journal, accentuating that it is illegal and unethical to submit a paper we know is incorrect.
The corresponding author is not legally allowed to submit a paper unless all authors read the final version and agree to its submission in its final form. If the corresponding author still submits the paper despite her refusal, she can easily defend herself later or even immediately after the submission, by contacting the journal.
I think she is then at risk of retaliation by the corresponding author, her institution, and maybe other coauthors. I think she needs independent advice before deciding.
I agree about the risk and the necessity of getting legal advice before anything, especially given the extremely vague question asked by Mandy.
But honestly if my institution is so fraudulent that it backs up the fraudulent corresponding author and retaliates me –instead of punishing him and supporting me, I would resign.
If Mandy is a student, then fighting wouldn’t be an option. Perhaps the best practice would be to ask nicely to be excluded from the list of authors; I know many times, this is not technically possible.
Another good practice would be to secretly consult the university dean of research and secretly tell on her professor about this, without the corresponding author knowing. Perhaps, their dean can force the corresponding author to update and correct the paper before submission.
It depends. You should give more details. What exactly do you mean by new data? Data within this same research? Or from other projects? How and Why the new data is not incorporated within the article? What do you mean by “correct the record”? As if knowingly publishing an “incorrect” paper first?
If the current conclusions are decisively incorrect based on updated analyses, knowingly hiding the important new data and submitting false conclusions with old data is definitely a misconduct.
Correct the record later doesn’t even make sense. Don’t disrupt the record to correct it later! You should incorporate the new data into the dataset, redo the analyses, and update the whole discussion and conclusions parts. Then the paper will be legal and ethical.
I faced similar problems 30 y ago. I had collected all data for submission of a research paper after presenting partly in a conference. A person who was forced into my work as a co-author, was pushing to break down data among 4 papers, not one, without understanding the entire work. He, as a rookie academic, desperately needed MORE papers for own business. Because breaking data into pieces was harming its integrity, I submitted it in one paper. The paper was very successful with 374 citations, appearing in several patents and numerous in research papers and theses (plus few plagiarism cases).
Well done and congratulations! As a side note, we all have seen people who are forced into our papers, perhaps tens or hundreds of them.
Thanks!
When fraudster bully is supported and protected by institution, what a poor researcher as target can do, especially in an institution that dirty politics play major role.
At that time, it was so risky and high chance of failure challenging academic misconduct, but now is very very different. Even people with no formal affiliation no power kick out most powerful academics like Stanford president for research misconduct. And next day (or next hour) all the world get informed in breaking news.
RW: thank you for conducting the interview. I see little difference between the authors’ comments here and the paper.
I highly commend reading Dr. Bishop’s thoughtful reply, and there are PubPeer comments on their site as well; how could there not be.
https://pubpeer.com/publications/3A38E9CB44BDC47AC5BB32B0D4185C
A pubpeer promotion?
This situation must be what factory owners were faced with when the EPA came along and told them not to throw all their waste in the river, when being able to throw all the waste in the river represented most of the profit margin.
“We don’t have enough resource to handle all of the integrity claims” really means “We don’t want to hire enough people, partly because it would cost money and partly because it would require us to admit that there is a huge problem”.
If even 10% of the complaints on PubPeer were vexatious, that might just have some validity. And maybe one day we might get to that. But for the moment, the institutions probably need to take a good long look into the abyss.
The fact that universities feel overwhelmed by PubPeer comments just indicates the real scale of the problem.
The fact that they turn to lawyers to address it points to the reason why the problem got to this scale to begin with.
You hit the nail on the head!
Universities just don’t want to do the work, don’t want to investigate and for sure don’t want to accept/acknowledge they have more rotten apples working for them than they like. On top of this universities in general whitewash most of the times. So now they will just have more work to say ‘oh no, this is not fraud, this is an honest mistake’. Oh no this is not a paper mill author, this author has a huge network and that is how he/she manages to publish >100 papers a year.
I think the authors reach the exact opposite of the correct conclusion in their article regarding the statute of limitations. They argue that the statute of limitations should be applied more strictly than it currently is. This is of course a pure legal consideration, which has no bearing whatsoever on scientific issues. If a fraud happened a century ago, it is still a fraud, and the scientific record should be still be corrected by a retraction.
Was thinking the same- a lot of things I see go through PubPeer aren’t necessarily articles receiving comments within 6 years. Some of them are from the early 2000s, 1990s… but the record still needs to be corrected.
The solution is simple, the authors of the papers state it here: “The RIO community is, in general, under-resourced to be able to investigate every concern that is posted”
Then simply hire more people!
You don’t solve the problem by trying to limit the number of cases you should investigate based on ‘expiration dates’.
“free up RIOs to focus on those researchers whose actions are clearly not due to “honest error” but instead may have been intentional, knowing or reckless in their dissemination of false data or plagiarized work.”
Does it matter if the problematic data are “honest”. Still needs to be dealt with.
Without paying attention to the pattern and frequency of errors it is difficult to determine intent. Careless, reckless, intentional…a gradient.
What a sad and problematic take on the research integrity issue, which has been increasingly scrutinized by researchers and the general public! The tip of the problematic iceberg of research misconduct uncovered by “sleuths,” image detection tools, etc., raises even more significant concerns about the potential massive scope of the problem. Trust in medical and scientific institutions is on the decline.
The brilliant solution to these mounting problems? Relax the rules and make it harder to investigate research misconduct because it’s too hard, takes too much time, and the “Research Integrity Officer” has more important cases to investigate… Indeed a cynical solution that could only emanate from lawyers. Raise the bar for triggering investigations? Narrow the statute of limitations?
None of these solutions are aimed at protecting research integrity. They are designed to protect the status quo of rich and powerful organizations that don’t want the flow of grant money to slow down. And I say this as a grant-dependent researcher at one of these institutions.
Disappointing article. Makes me worry even more for the erosion of scientific integrity.
The elephant in the room is it is clear that the peer review process as is, is not fit for purpose if things can be found to be wrong with papers post publication. So rather than try to treat the symptoms it would be better to address the cause and prevent bad papers from being published in the first place. One can’t just employ loads of RIOs to sift the backlog without changing the peer review process to fix the the problem. It’s like trying to clean up a flood before fixing the broken pipe. You will never finish cleaning up
I bet there is an army of retired scientists who, if given university library, etc. access, would be more than willing to investigate PubPeer claims for universities. Part time work, likely not much salary needed or required, and it could be done from their homes via telework. I know a few people who might do this. I will soon be retired and I would consider it.
I have worked in a research integrity office grappling with the numerous cases on pubpeer and I agree that the universities need to hire more integrity officers to investigate these dodgy people.
But we also need better processes – think of process as a force multiplier. Look at COPE flowcharts that allow things to stall out if authors don’t respond – things aren’t much better at universities and journals are free to just ignore university inquiries or requests for retraction. Our current systems are designed to fail – to make it take longer to clean up literature than to corrupt it and get promoted or move on. Improve process, make it easy to find misconduct and retract papers and you will suddenly find institutions actually doing things. But this takes thought and work from universities, funders and unions so will never happen.
I believe it is a forgone conclusion where a University’s loyalties lie. Bad press and public outcry notwithstanding, a University will attempt to sweep allegations of misconduct under the rug wherever possible, particularly where it concerns prominent researchers that bring in millions of dollars in funding and intellectual property. Universities are, in every sense, a business and as such, protect their investments in the public and legal spheres accordingly.
This is exactly why public fora such as Pubpeer are so utterly essential. Grift is rife, all the way to the top. Administrators are interested in damage control and loss minimization, not lofty principles of scientific integrity or taxpayer benefit.
They key in this case is to make disclosure of all raw, pre-processed, un-normalized data a precondition of publication without exception so that data manipulation and misrepresentation can be identified by the scientific public on sites such as Pubpeer, as peer review is apparently insufficient to meet this need.
Excellent comment. Just let me add one point about the last paragraph: Many students and profs I know fabricate the whole data and methods from scratch, all from their imagination. So even if we ask for raw data, they’ll give us an intact and consistent one. I know many of such fake raw datasets will confess if tortured enough, but many others look genuine.
One way to fight this is to ask not just for raw data but also to mandate that all researchers must take (anonymized) photos and films from all their cases one by one. If such a rule is in place, no researcher will dare to fabricate data because he knows if he claims he has studied 36 humans, he needs to give the journal photos and movies for different steps of 36 different humans. If he claims to have 120 in vitro tests, he needs to show the journal 120 photos of 120 samples as well as 120 reports outputted by the lab machine in use. I know it is a little bit hard on the researcher, but that’s the only good way to fight data fabrication.
One way I advocate for this is is similar to the tracing methods that police forensic scientists use, namely that the output from the apparatus used to generate the data itself be furnished, including metadata from the software, as proof that it is unaltered. This would include files such as FACS (cell sorting) (*.FCS) and original files from software used to run PCR experiments (*.EDS). Of course, there are ways to manipulate even these data formats, such as swapping samples to fit a desired outcome. Nonetheless, it makes it more difficult to do so. As the vogue is now to include RNA-seq data with papers in the biological sciences (my particular domain of familiarity), I have taken to analyzing these wonderful, unbiased data sets to compare with other results in manuscripts to look for obvious inconsistencies. I, like you, am surrounded by fraudsters, and very powerful ones at that.
Yes that would be very good especially in combination with other methods like the ones I suggested. Almost all fraudsters I know are not tech savvy enough to manipulate metadata or even know what it is. But even if they know, manipulating a large amount of metadata would be very stressing for them, deterring many of them. Imagine using more than one forensic method at once (like adding the methods I suggested). No one will ever dare to fake data anymore.
Exactly as you said, we need forensics. Currently journals blindly trust authors. This used to work fine before but never anymore. That should change. Journals should become ultra paranoid forensic experts that presume everybody as a fraud unless the author can prove otherwise.
And that is very very easy (no need for extra staff or technology). Just once a journal’s screening checklist necessitates the submission of raw data + metadata + supplementary data + photographs / videos from steps of the work, that alone suffices to deter about 99.99% of fraudsters from submitting their fabricated paper to that journal. And once this is mandated for all journals by COPE and ICMJE and governments, no author will even dare to fake data.
Some raw data, metadata, and supplementary materials are confidential and can’t be published. But the journal can demand the authors to submit them for internal assessments, and assure the authors that their data are safe with the journal and not to be published; they are just for forensic purposes and catching potential fraudsters! That last sentence is more than enough to scare off almost 100% of cheaters!! 🙂
The journal doesn’t even need to investigate the submitted materials. The very fact that the journal CAN investigate them in the future at will will make any fraudulent authors extremely insecure. 😀
@RW, I had two questions and I don’t want an answer. Just please think about them:
1. Can you as an a well-known party investigating and fighting scientific crimes encourage journals and more importantly COPE and ICMJE and other governing bodies (like governments, NIH, NHS, etc) to mandate the submission of meta-data, raw data, supplementary data, and photographs (and possibly even videos) taken from the steps of the work? I am not talking about open data. The files can be confidential and visible only to editors and reviewers at the journal. Or if the authors choose, the files can be openly available to the public too.
2. How can someone who has no place in the pyramid of power spread this idea and request COPE and others to incorporate such rules in their flowcharts?
The only saving grace is that it seems difficult to produce fake raw data that are fully consistent, if one judges from the number that get caught.
as someone who is an administrator (not a RIO)- my work can be really frustrating because I put the case together and we see the researcher clearly violating ethical standards, though not quite research misconduct… what it feels like the conversation becomes is how we can just put the BS back on them if it blows through the institution. Idk- the whole system is broken and it’s a huge issue and only getting larger with universities doubling down on troubling metrics and predatory journals accepting any damn type of paper. It is hard to keep up with and we are totally understaffed. Some of us do our best….but still need to feed our families.
I believe there is a simple procedural change that would have multiple relevant benefits:
– accelerate the resolution of allegations
– bypass expensive lawyers (who need to be brought up to speed on PubPeer)
– reduce the workload of research integrity officers (RIOs)
– increase transparency
– improve the quality of investigations
The procedural change would be to make public the relevant original data as the *first* step of any investigation. The data need to be gathered in any case. The authors’ own data cannot be defamatory so there would be no grounds for legal resistance. In most cases (except some research on humans), there is no reason for the data to remain secret, indeed there is a growing expectation that they will be made public as a matter of course. If the authors claim the data are not problematic, call their bluff. If in reality the data are problematic, the risk of exposure will sometimes precipitate a rapid resolution of the allegation. Once the data are public, interested experts could help with an in-depth forensic analysis. There is now a community with deeper and broader expertise, and greater motivation, than most RIOs or editors. In the best cases, the RIO would need to do little more than gather the analysis once the dust has settled, draw the necessary conclusions regarding the data, whose early communication to the public would be an additional benefit, and then proceed to the internal blame-storming sessions.
As an aside, once the above benefits of early public access to the data have been understood, the utility of requiring public data sharing at the time of publication should be clear.
Responding to the specific idea of a “statute of limitations”, it would need to be implemented much more carefully than suggested to have a chance of working. The current situation arose because everybody could see that some researchers had been cutting corners and cheating with impunity. An administrative decree making official that de facto impunity would just take institutions back to square one, but this time they would be facing a much more organised, alert and expert community.
Of course, the real solution is to make it so that fewer researchers cheat! For that to happen, cheating shouldn’t pay. And for that to happen, funding for researchers and institutions should create effective incentives. It seems that is (slowly) beginning to happen, at least at an institutional level. Instead of trying to neuter those incentives, they should be passed on to researchers.
Disclosure: I’m a PubPeer organiser