Since it launched in 2012, PubPeer has grown to become a standard part of the scientific lexicon, and its numerous post-publication discussions have led to more editorial notices than we can count. But it’s also faced its share of critics, including a scientist who took the site to court to unmask commenters he alleged had cost him a job offer. The site won that case on appeal, but is today launching new features that will make it impossible for the site to reveal users’ identities, as well as easier to read and format comments. We spoke with PubPeer co-founder Brandon Stell about what to expect from the new site.
Retraction Watch: What changes have you introduced to the site?
Brandon Stell: There are many things that we have wanted to do to extend PubPeer but unfortunately all development ground to a halt a few years ago when the original developer, George Smith (who was working on PubPeer in his spare time) stopped working on the site after we decided to transition to a nonprofit. Recently, thanks to a generous grant from The Laura and John Arnold Foundation, we were able to hire a very talented new programmer to work on the site full-time and completely rewrite it to to make it easier to add some of the ideas we have for new features in the future. [Editor’s note: Retraction Watch is also funded by the Arnold Foundation.]
There are many changes behind the scenes in this current rewrite but the two biggest changes for users are the improvement of readability/formatting of comments and the security of user anonymity. Users will now have options to format their comments to be more academic and they will be able to easily insert and format images and equations. We feel that strong anonymity is very important for effective post-publication peer review and to address the security of user anonymity on PubPeer we decided to do our best to not retain any identifying information for any users that wish to remain anonymous.
RW: You’ve added some identity management features. What are the most notable changes, and what problems are they designed to address?
BS: There are now two types of accounts: anonymous and signed. Anonymous accounts with random user names from the tree of life (e.g. drosophila melanogaster) can now be generated when anonymous comments are submitted and users can sign into those accounts again later (using only a key that we provide them) in order to leave subsequent comments, receive feedback about why comments were moderated, check for replies to all their comments left with that account, etc. (for the PubPeer aficionados out there: it is as if Peer X accounts and Unregs have been merged into one)
RW: Will the identity changes you’ve installed make it more difficult for scientists to unmask (and thereby seek recourse from) anonymous commenters?
BS: Yes, that is one of the main motivations for that change. Once the transition to the new site is complete our goal is to not be able to reveal any user information if we receive another subpoena or if the site is hacked.
RW: You’re adding links to retraction notices (as well as EoCs, errata, etc.) directly in the commenting threads. Often users post those on their own, so what’s the advantage of the site taking that over?
BS: It’s true that some users occasionally insert expressions of concern and retraction notices into PubPeer comments but it is far from systematic and not easily searchable. Richard Smith and Steph Smith-Unna recently created a great API to easily access publication events and we are using it to more systematically pull information into PubPeer, make it searchable, and insert it into the appropriate timepoints of the PubPeer commenting timelines. Richard is using the data provided by Crossref and PubMed but if you know of anyone with a more complete API we would love to use that as well 🙂 [Editor’s note: We’re working on that for retractions, stay tuned!]
RW: What prompted these changes?
BS: As you know, we feel that most of the problems with scientific reproducibility stem from the fact that scientific evaluation committees around the world rely so heavily on journal names to identify promising science, and scientists. To help alleviate that problem we want to continue to build a timeline for each publication that includes all comments from the community, references, and other publication events (i.e. retractions, expressions of concern, acceptance in an overlay journal, etc). We believe that if all that information becomes more centralized and easily searchable, evaluation committees will use it to help them make more informed decisions. That will hopefully loosen the grip that journals have on deciding the future of science, put evaluation back in the hands of community, and change some of the publishing incentives to ultimately strengthen science.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.
Great! Congratulations to PubPeer.
It’s notable that nowhere in his answer to “what prompted these changes” does he mention anything that justifies anonymous commenting. It’s disappointing that the most controversial feature of the site, which comprises the overwhelming majority of public activity on PubPeer, and which they insist on defending to the point of performing amazing mental gymnastics, isn’t at all something that addresses what “most of the problems with scientific reproducibility stem from.” You can do all those things signed, without affecting the de-centralization and openness of the forum.
It’s also interesting that they freely admit making wholesale backend changes specifically for the purpose of preventing themselves from being held to the same standards of transparency they demand from others. I suspect that their actions might not legally protect them from liability completely, based on the precedents they’ve already set for themselves.
Perhaps when reviewers of papers and grant are not anonymous, pubpeer may change.
“You can do those things signed”. Perhaps you can. Not everyone can.
He’s not exactly sticking his neck out there by identifying himself as “Dr. J. Smith”
But why is Dr Smith not willing to identify himself?
Disagree with just about everything in this comment. For the umpteenth time, identity does not matter if content of allegations is sound. Focus on the message not the messenger. In addition, note that anonymity is allowed on the site but not mandated – anyone who wants to use their real ID is free to do so. If you feel comfortable doing that, go for it, but recognize that others may not share the same comfort level.
Re: being held to a standard of transparency, it’s really about the legal excuse of plausible deniability. The ability to say “I’d love to tell you, but I can’t because I don’t know” is a very powerful legal defense. But, if questioned on this, it requires you to actually not know. All PubPeer is doing, is constructing the site so they can say this in good faith.
It’s no different than LavaBit’s closure – they had operated on a defense of plausible deniability (“we can’t reveal your data to lawmakers because it’s encrypted and we don’t have the key”). When it became clear thanks to Snowden that the feds had a back-door, LavaBit could no longer confidently say this, so chose to shut down. Compare to the other major tech’ companies who for several months continued to claim your data was safe from the government, even though they knew otherwise.
I agree with this position but it should apply to reviewers in journals themselves. More generally, the whole journal process is anachronistic and very harmful. In the medical sciences there appears so much speculative, questionable and adversarial material that gets published but frequently remains inadequately reviewed, frequently by de facto parties. There is, for example, strong opposition to recommendations from the American Diabetes Association but they own the journals and appoint editors.
PubPeer should be incorporated into the review process where there should be levels of publication. Publications should not be considered permanently accepted until they’ve gone through a certain number of months of open review. This idea has been proposed by others in different forms and might be considered important because of the importance of medicine for our health.
Many PubPeer postings are images of western blots with boxes superimposed to identify duplications/splicings. Nobody needs to know the identity of the person posting the images to ascertain the veracity of such claims.
1) There are many examples of people who had suffered a lot for criticizing the wrong scientists (even if they ‘won’ in the end). 2) People can (and in some fields most likely) will get back at you in grant and paper reviews. 3) In many cases supervisors or administrators discourage open criticisms and PPPR – as a distraction from direct work responsibilities and as a potential source of troubles. 4) In many institutions it is explicitly forbidden to openly/publicly criticize your colleagues (you should do it though an appropriate office).
I understand that platforms like PubPeer the Online Journal Club offer a venue for post-publication peer review, it is important to highlight that comments made on such platforms are not subject to the rigorous peer review process that is the cornerstone of academic publishing. Therefore, while these comments can at times provide valuable insights, they should not be held to the same standard as formal academic review.
There are some concerns and limitations regarding the reliability and utility of PubPeer the Online Journal Club:
1. Anonymity and ethical concerns
• Unaccountable critics; the anonymity can encourage open dialogue, it also means that individuals posting critiques are not held to any accountability. This can encourage behavior ranging from nitpicking to targeted harassment.
• Bias and vendettas: the anonymity aspect can also allow individuals to pursue personal vendettas or biases without immediate identification, potentially leading to misleading or unfair criticisms.
2. Quality and expertise of comments
• Comments can range from highly insightful critiques by experts in the field to uninformed opinions; unlike traditional peer review, there is no quality control. In the worst cases, incorrect or misleading criticisms could be posted, which might unjustly damage the reputation of researchers or misinform the scientific community.
• I would like to bring to your attention the qualifications of the individual (s) who has raised these concerns/comments, which do not possess formal qualifications in legal or academic law, nor are members of the Ethics Committee on Publication Ethics (COPE).
3. Institutional and career impact
• Articles or authors that receive negative comments may experience unwarranted stigmatization, which could affect their career prospects or funding opportunities. In academia, the perceived quality of a paper can influences an author’s citation rates and journal impact factors. PubPeer the Online Journal Club comments, regardless of their accuracy, can potentially skew this perception.
4. Lack of comprehensive coverage
• PubPeer the Online Journal Club discussions often focus on high-impact papers or controversial topics, leaving a vast majority of published literature without any post-publication peer review. Not every researcher is aware of or has the time to engage with PubPeer, the Online Journal Club, making it a somewhat exclusive platform that does not necessarily represent the broader scientific community.
5. No formal editorial oversight
• Without an editorial body to fact-check or moderate comments, there’s a risk of spreading unverified or inaccurate information. Traditional peer-review processes often adhere to strict ethical guidelines, such as COPE guidelines; PubPeer the Online Journal Club lacks a formal structure for ensuring that ethical standards are met.
6. Legal risks
• Comments on PubPeer, the Online Journal Club, have, in some cases, led to legal action for defamation; the lack of formal oversight and the possibility of anonymous posting raises concerns about legal liabilities.