PubPeer Selections: Good behavior earns praise; questions about HIV vaccine research

pubpeerPubPeer continues to make its mark on the scientific literature. Here’s another installment of PubPeer Selections:

24 thoughts on “PubPeer Selections: Good behavior earns praise; questions about HIV vaccine research”

    1. It is curious to note the background of the individual that hosts the opinions of Kamoun (1st link above), Benjamin Schwessinger:
      https://blushgreengrassatafridayafternoon.wordpress.com/bio-2/
      “For my PhD I decided to move on. After several rejections from Universities across the Atlantic I ended up doing a rotation PhD at the John Innes Center (JIC) and The Sainsbury Laboratory (TSL).”
      “Within my rotation year I first worked with Jonathan Jones/ Cyril Zipfel”
      “In my final rotation, I joined Sophien Kamoun’s Lab and worked with the vanguard Joe Win”
      “I still remember when a senior post-doc in David Bauclombe‘s lab congratulated me on the map-based cloning of my first gene”

      All linked by a common thread: Voinnet.

      Reply

    1. There was no “heated discussion”, just a single comment that noticed an undisclosed Gel splicing (in a paper from 1999). And it wasn’t even from the same group, there was only a single shared co-author.

      Could you stop this witch-hunt, please? You’re doing science a disfavor.

      Reply

      1. Indeed, almost all discussion was removed… I am amazed! Well pity, for it was quite interesting. Sorry if you did not read that, I saw no witch-hunt on it, to say any witches that I may have bernt!..

        Reply

          1. Hard to discuss this openly since almost all discussion on that link was removed from public view. I find it curious that SK’s response on this thread was unregistered and that PubPeer removed all other comments (mostly from registered users) which were neither libellous or against rules, without notice. According with Twitter, more content was removed concerning this general topic, e.g. refer to https://twitter.com/KamounLab/status/558573564673392640
            Maybe try asking PubPeer?

  4. I wish to provide some anonymous commentary about the first of the blog posts that appeared on January 22. I am commenting here at RW because they have no DOI, but pertain to interpretations of PubPeer and post-publication peer review (PPPR). Given the leadership role of Kamoun in plant science, close scrutiny of these public statements, which may or may not be his, is essential. One will notice that the subject of the blog post is often confusing and it is sometimes unclear whether the blog is referring to a collective voice of the bloggers Ksenia Krasileva [1] or Benjamin Schwessinger [2], or the actual voice of Kamoun himself (see the description about the Tunisian background). The comments below will thus a subject-neutral response to what has been written. The comments about the comments on that blog post are meant to understand the responses that have taken place at PubPeer as a follow-up to the apparent correction of duplicated figures in a 2012 PLOS Pathogens paper [3]. All words in quotation marks are from the blog post and the comment following the blog comment are my response, as a general member of the plant science community.

    [1] https://blushgreengrassatafridayafternoon.wordpress.com/bio-1/
    [2] https://blushgreengrassatafridayafternoon.wordpress.com/bio-2/
    [3] https://storify.com/KamounLab/post-publication-review-of-van-damme-et-al-plos-pa

    https://blushgreengrassatafridayafternoon.wordpress.com/2015/01/22/be-positive-from-witch-hunts-to-the-new-reward-culture/
    “Be positive! From witch hunts to the new reward culture”

    1) “This is a guest blog post by Sophien Kamoun, a highly respected group leader in Plant Biology at The Sainsbury Laboratory, Norwich.”

    2) “We decided to publish his post as a valuable contribution to the debate on ‘open science’.” So, who is “we” exactly?

    3) “However the represented opinion does not necessarily reflect our both/both opinions.” If this is not his/her/their opinion, then whose is it?

    4) “Anyone can access the data and comment on it.” Correct. That is why PubPeer exists.

    5) “We move from elitist old boy clubs to an open door party.” Most of the lower and middle class of plant science do not have such elite “boy club” experience. So, perhaps this “we” could explain the concept in more detail?

    6) “As a native of Tunisia open science matches well with my personality.” Is the cultural element relevant? Is this a description of Kamoun? A search of the web indicates that Kamoun is a name of Tunisian origin. The bloggers should confirm what information refers to whom, for clarity.

    7) “I grew up in the typical Mediterranean culture of vibrant discussion, constant arguing and yes the occasional bickering.” Who is this referring to?

    8) “These traits are embedded in me. I know they can be irritating to others.” Who is this referring to? As equally as others respect these opinions, so too, should the voice of this blog respect the voices of PubPeer commentators.

    9) “A paucity of critical thinking and engagement among the citizen of modern and, presumably, well-educated societies is one of the drama of our age.” At PubPeer, it could be safe to assume that commentators and moderators all share this noble view.

    10) “It is therefore natural for me to support all efforts of post-publication review. Platforms such as PubPeer aim at extending the discussion and analysis beyond publication in peer-reviewed literature. They are perfect for this era of journal proliferation and internet communication. They also address flaws in pre-publication peer-review that have been well documented.” Perfectly stated but who is this referring to?

    11) “But PubPeer has evolved, most certainly against the wishes of its anonymous founders, into a modern day “witch hunt” platform. Many comments seem valid. But pointless and frivolous comments are being posted, and have certainly increased in frequency in recent weeks (at least in plant biology).” Whose ideas are these? The mood suddenly swings here: PubPeer is suddenly labelled as a witch-hunting platform. Perhaps what the person who wrote this blog needs to understand is that perhaps plant science has been in a state of denial for decades in terms of seriously correcting the literature and that debate has been equally stifled by the elite that run plant science globally, and the editor boards of most elite journals. Perhaps PPPR has finally woken up the elite of that structure and given that structure a little jolt? Incidentally, how does this person know “the wishes of its anonymous founders”? If they are anonymous, then has that person had special avenues of communication with the PubPeer founders?

    13) “How to address this? How to buffer or eliminate such posts while maintaining the original goal of PubPeer as a vibrant journal club platform? My suggestions are two-fold.” PubPeer will remain vibrant with or without this opinion because of freedom of speech.

    14) “First, PubPeer should encourage and promote the posting of positive reviews.” This viewpoint completely contradicts the statement made above: “They also address flaws in pre-publication peer-review”

    15) “Peer review is not about “Gotcha! I found a flaw!” It is primarily to endorse excellent science.” Actually, this is really surprising to hear for two reasons: 1) excellent science is not excellent because of the peer review; 2) the key purpose of peer review is precisely to detect flaws. It is exactly because traditional peer review has considerable flaws that PPPR evolved and that some journals/publishers now apply open peer review policies, to encourage openness and transparency about the evaluation of work over time, in a time-independent manner, i.e., indefinitely. Many consider that true peer review (i.e., scrutiny, criticism and analysis) only begin after a paper is published.

    16) “It does shock me that the great majority of the posts I read only list negative comments. This is not what peer review or journal clubs are about. More often than not, we find positives in the literature we read and discuss.” Indeed, this is a good point. One expects, by analogy, that the traditional peer review to have provided a critical and positive evaluation. However, it can be porous and those pores are being shown at PubPeer. Why is this such a negative thing?

    17) “There is plenty of great science out there. Why shouldn’t we acknowledge it and promote it?” But you are wrong. Every time a paper gets cited, science is promoted. And the claim that positive science is not promoted at PubPeer is incorrect. Please read PubPeer entries more carefully to appreciate that positive aspects are promoted. Just yesterday, PubPeer was used to promote a useful product from the Donald Danforth Center:
    https://pubpeer.com/publications/79E724F6F5249BEF55F48705BB6A59#fb22441

    18) “Why are posters rushing to reveal “vertical lines” in a blot but failing to highlight a flawless figure?” By poster, you probably mean “commentator”. Given Kamoun’s expertise, perhaps he could come forward to help the literature become perfected, help correct it through PPPR and by identifying such flawed figures or gels. Kamoun’s skills are needed, for plant science’s posterity. And of course, you or he should feel free to compliment any good science along the way. That is a personal choice.

    19) “PubPeer and related platforms have a role to play here. They could help build the confidence and reputation of young scientists, strengthen their CVs – further shifting the obsession with impact factors and publishing in glam magazines to a focus on the quality of an individual’s work.” Again, perfectly worded. Nothing to add.

    20) “Second, PubPeer may consider recruiting an Editorial Board to help moderate the questionable posts. I expect many reputable scientists, junior and senior, to be willing to volunteer just as we do for scientific journals. An Editorial Board that reflects diversity in gender, geography, career stage, and research topic would improve transparency and credibility. It will also serve to temper criticism and cynicism about PubPeer that is prevalent among many in the scientific community.” What an excellent idea. Being a leader in plant science, would Kamoun and his expansive network like to be the first to establish a PPPR editor board for the plant sciences at PubPeer? This could be a new and much needed task force to reevaluate the published literature. Think of it as a duty to the history of plant science. Leaders should lead, not simply comment.

    21) “The reality is that post-publication peer review is here to stay.” That’s right. And that’s not a bad thing.

    22) “The recent episode that my colleagues and I faced was a timely teaching moment. It reminded us of the importance of record keeping, archiving old data, ensuring that pictures integrate visible labels and so on. Several members of my lab told me that this sorry episode has prompted them to document and store their data more rigorously. Nobody wants to find out 10 years later that they cannot respond to an allegation about their paper. Mistakes do happen so we should be prepared to respond and revise.” In other words, you are offering your words of thanks to PubPeer for creating a platform that allowed the mistakes in Kamoun’s paper to be detected and hopefully corrected? Are you also thankful that more attention was drawn to the PLOS Pathogens paper, and that the PubPeer posts drew more attention than the actual journal web-site? Are you thankful to PubPeer for this incredible and revolutionary academically positive experience that seems to have enriched the policies of your (whoever you are) colleagues?

    23) “At my host institution, The Sainsbury Lab, which is currently led by Head of Lab Cyril Zipfel, recent episodes have further justified the misconduct training initiatives that were already undertaken way before the current brouhaha.” Are you perchance referring to Voinnet and the link to TSL? It is delightful to hear that TSL had a pro-PPPR-leaning approach before the “brouhaha”.

    24) “We need to raise awareness of these important issues. Scrutiny and discussion of the science post-publication should become part of the culture. A shift to a new reward culture is happening. It’s not only where you published but also what you published. Quality indicators other than the journal impact factor are becoming recognized. It’s you, the next generation of scientific leaders, who can ensure that the cultural shift takes hold. And PubPeer and other post-publication peer review platforms have a role to play in this new reward culture.” Again, brilliantly stated. Nothing to comment on there.

    This dissection of that blog’s comments is precisely to invoke further discussion about this suddenly core and fundamental topic: post-publication peer review in plant science.

    Reply

    1. The blog posts state, in one of the replies: ““To the Kamoun lab: I promise to make a start by now placing my honest, positive and fair opinion on several of your great manuscripts!” edit: In hope they allow positive comments!” Expect a wave of Phythopthora infestans papers being linked at PubPeer with a non-arbitrary set of forcefully positive comments… indeed, if this continues, PubPeer will become a circus, those who post to critique the literature, and then followed by a wave of those who aim to flood the comments to save their own image. Indeed, PPPR is in a state of evolution.

      Reply

  5. Dear SK,
    Thanks very much for your valuable and thoughtful comments on Kamoun’s blog post on our, K. Krasileva and B. Schwessinger, blog. I encourage you to re-post your comments on Kamoun’s post in the blogs comments section. We hoped to encourage an open debate about PPPR in plant science by publishing his post. For now I would like to clarify only points concerning the authorship of the post.

    2) ‘We’ indicates the owner of the blog K. Krasileva and B. Schwessinger. This was highlighted int the following sentence by ‘both/both’ with links to our ‘biographies’.

    3) The post was written solely by Sophien Kamoun and reflects his personal opinion and not necessarily the opinions of the owners of the blog, K. Krasileva and B. Schwessinger.

    Reply

  6. jessi
    Pubpeer by now has 24 papers by Walumbwa flagged for pretty serious errors but there appears to be absolutely no action from either his employer (FIU), his numerous co-authors, or many of the journals. So far I can see that only three journals have retracted some of his work. Not a pretty picture of the “science” of management/business. https://pubpeer.com/search?q=walumbwa&sessionid=BCC18A28EEA23E21784A&adv=none

    I read most of the comments. Some of those were very good and some less so.

    There was a story in RW some time ago about why so few retractions have appeared in management. The Walumbwa and Lichtenthaler cases indicate one thing: management is far from being some unique snowflake field. At this point it seems that the low amount of retractions is mostly explained by the small or nonexistent post-publication management-oriented review community pushing for more scrutinity. Another explanation could be that the editors and other bigwigs have less experience in dealing with retractions, at least when compared to life sciences (or social psychology…).

    So thumbs up! Walumbwa-Lichtenthaler are only the start.

    Reply

    1. A valid analysis. Walumbwa forms part of the editor board of the journal in which one of his paers is being heavily criticized at PubPeer. Plant science has a similar crisis, I believe. There is all this superficial talk that everyone is going to do PPPR, but in fact, who is actually doing it? Try using the analogy that you used for management with the bigwigs in plant science at PubPeer. You will get hit hard with the grapes of wrath. The resistance to true, down-to-earth and gritty PPPR in plant science is massive. Nobody actually wants to get their hands dirty, for fear of entering into conflict with their colleagues, their neighbours, or with unknowns. If there is so much reticence, then how do the very same bigwigs expect the errors in the literature to be corrected? With magic? No, PPPR is not a noble task. It is an exhaustive, constand plod. It is a frustrating mission that has no rewards (not should it ever). But it has a noble silver lining: the outcome, despite anger, frustration and occasional tragedies along the way, will be (hopefully) a literature that is a but more reflective of a cleaner picture (not necessarily perfect, but cleaner). As I see it, from experience, we are still in phase 1 of PPPR in plant science: mentalizing the plant science community that it is essential.

      Reply

  7. On August 22, 2014, Paul S. Brookes stated, at PubPeer:
    https://pubpeer.com/topics/1/A9C1A3DBF6649CC62871AA7DC87B17#fb22683
    “Bottom line (IMO) – go ahead and comment on everything, because doing so might just cause someone else to contribute a vital part of the underlying story, which provides context and helps build an overall picture. If people don’t comment because they think some event or other is not important enough, then we lose this context.”

    Absolutely true. However, it appears to be an issue now.

    There is some discussion at PubPeer, about how to improve the PubPeer diamond.
    https://pubpeer.com/topics/1/1A0BDA96C3E7BD263364577E50E08F#fb22828
    May I suggest the following:
    a) Make sure that all new comments must be under a registered name, like PubMed Commons.
    b) Those registered names can be anonymous, but registered nonetheless, to preserve the importance of anonymity. If there are concerning comments, then these can be directed, by PubPeer moderators, to registered commentators.
    c) There must be a review panel for comments, which screens the veracity before they are posted. That implies topical editor boards. Comments which need to be verified need to appear with some sort of a different category, like an “in debate” sign or status, to indicate that what has been stated is not necessarily true, but it remains an unknown question requiring the verification of the authors.
    d) PubPeer will then assume responsibility that it approves and any comment that is ultimately published. Blaming the messengers every time is simply not an alternative. PubPeer must be structured in such a way that there is correct, and sufficient, checks and balances to every post. It is not the commentators’ fault if the system has pores or is weak.
    e) Post-publication peer review. Past comments posted on PubPeer should be reviewed, again, by open panel “judges” or a team of actual professional reviewers and not just some hidden team. As equally as those who come to PubPeer in a bid to correct the literature, or ask questions about it, so too should PubPeer also evolve to ask questions about its own postings, which its moderators approved, in a bid to clear up errors, doubts, or concerns.
    f) Unless this happens, there is going to be much more than just heated debate at PubPeer. There are going to be serious problems, and clashes, as each one interprets differently what the purpose of PubPeer is.

    Reply

    1. It pays to familiarise oneself with the existing PPPR landscape.

      a) and b) Some commenters need industrial-strength anonymity, as shown by legal threats and actual court cases involving several PPPR sites, including PubPeer. Only PubPeer offers the anonymity that enables users to protect themselves fully, i.e. against a subpoena of the site for user information. I hope that doesn’t change.

      c) Just consider everything to have an “in debate” status permanently. Problem solved. Oh and most critical comments will never be “verified by the authors”. You’ll need to get used to the idea that there are actually some bad guys out there who do cheat and have no plans to admit it. More generally, you should consider PubPeer comments as a potential source of information, not as a definitive judgement; no such thing exists anyway. Note also that some erroneous comments arise because something is confusing; it can therefore be beneficial to other users if the explanation remains.

      d) For legal reasons, I suspect that it would be impossible for PubPeer to “assume responsibility” for all comments and survive.

      e) Don’t wait to be invited – go ahead, be a judge and verify some comments.

      f) Coming back to the fact that there really are some bad guys out there, if problems are discovered in a big shot’s work (happens all the time), the stakes can be incredibly high. There is no way that differences will be resolved amicably. Usually somebody’s career is terminated; in the past it would often have been the commenter’s.

      Offering a few actual examples of gratuitously bad comments on PubPeer (or even some statistics on a random sample) would be good. Have you reported them?

      Reply

  8. Witch hunt: an investigation ostensibly carried out to uncover subversive activities but actually used to harass and undermine those with differing views. This is the American Heritage Dictionary’s definition. Anyone familiar with this definition cannot seriously liken it to the on-goings at Pubpeer. Pubpeer has unearthed numerous cases of invalid science. As opposed to valid science, that is scientific data that actually show what is being claimed, not a stitched-up version.

    I am writing this from the perspective of a senior scientist, whose every move is determined by decisions made in complete anonymity. Grant committees, peer reviewers etc. And here comes Dr Bastian and denounces anonymity when it comes to uncover the unbelievable amount of invalid data that can be found in the papers of seemingly respectable “colleagues”. Voinnet (20-30 papers discussed at Pubpeer), Latchman, Stephanou (20-30 papers), Baulcombe (~10 papers), Hanna (several papers and phd thesis), Maedler (~10) to just name a few of the current cases. We would not know anything about this if it was not for Pubpeer. No editor ever sounded an alarm. In each case not merely a single incident, but a pattern becomes obvious. These are powerful “colleagues” with important roles including as peer reviewers, on grant committee boards, on journal editorial boards, on ethics (!) committees, you name it. In each and every of these roles they act in complete anonymity. If, however, unacceptable practices are unearthed in the works of those thought to be beyond reproach, there are outcries (“witch hunt” without a single example) and calls for the end of anonymity. Is there a limit to hypocrisy? I for one want the playing field leveled in this regard at least. In most other regards it is badly warped, as we have to compete with “colleagues” who submit manipulated manuscripts to publishing sooner and obtain grants (based on the same invalid data) more easily. I think everyone is well advised to stay out of the radar of such “colleagues” and their multiple means of retribution in anonymity. Particularly since those “colleagues” and their apologists do not appear to be particularly rare, or coy, for that matter.

    Reply

    1. “I am writing this from the perspective of a senior scientist.” This is really important. We need the greater participation of seniors, as well as from junior researchers, post-docs, etc entering the conversation. The more opinions we get, whether of support, or of frustration, the clearer the picture becomes. Without clarity, science cannot advance. Retractions, insecurity, distrust, accusations, doubt and fear are all fundamental aspects of science discovery, in fact, so why do so many act so shocked to see a public display of such emotions on PubPeer, for example? Ironically, alot of the advancement takes place, as you correctly point out, under hidden anonymous doors. There is one word which is salient in your statement: “hypocrisy”. As cases get more complex, and as more and more voices cry out “foul play”, or simply just point out what they perceive to be errors, the conflicts will increase. Because images need to be defended, pride must be sustained, and this always involves a cost. And most definately, these “radars” will expose themselves if enough questions are asked. There will be costs, and there will be victims, but this seems to me to be the natural course of PPPR.

      Reply

  10. Lorraine B. Johnson*, on Twitter, states: “Anonymity encourages predatory bullying. Let the sunshine in!” Could Dr. Johnson provide concrete proof to support that claim? https://twitter.com/lorraine_lbj
    *(JD/MBA) Co-Chair of Consumers United for Evidence Based Healthcare; Patient Engagement Panel PCORI; Exec. Com. PCORnet; Exec. Dir.

    Reply

  11. Again, on the first mentioned case of PubPeer, I would also like to recommend reading the interesting ongoing discussion at:
    https://www.pubpeer.com/publications/D3DF6B4B19C383804BB3748B60604D

    For the record (in case it also almost completely disappears like the other thread above): right now it contains 22 comments, with 4 registered Peers, and several unregistered peers. Curiously someone claiming being an author is posting via unregistered comments, in spite of this author having a registered account, as demonstrated by his other thread: https://pubpeer.com/publications/877A42B12A198A44306771E0D80AB8

    Why? The discussion is interesting.

    Reply

  12. Alright, in fact some comments disappeared from the thread, which came down to 18 comments now, originally 22. Have a look.

    https://www.pubpeer.com/publications/D3DF6B4B19C383804BB3748B60604D

    i) one of the comments removed was suggesting that the authors should post their answers to readers as registered comments instead of unregistered comments claiming to be the authors;

    ii) there are several unregistered comments on the thread praising the authors and defending explanations for any issues found.

    Why complimentary comments are OK and increasing, and a suggestion of posting a registered comment (along with some other 3 comments) were simply removed?

    Reply

  13. An important development.

    Peer 1 states:
    https://pubpeer.com/topics/1/1A0BDA96C3E7BD263364577E50E08F#fb25288
    (February 18th, 2015 1:58pm UTC)
    “I was following an interesting debate on the thread https://pubpeer.com/publications/8FCB37BCB82D58B3AF19B13AF256A9 and suddenly it got stripped of almost all its comments. I saw no misbehaviour on it — to summarise, there was an initial discussion on the veracity of the observation made by the original comment, followed by a discussion on the relevance of pointing out and correcting undisclosed splicing. I had at least a couple of comments there that seemed relevant and polite that got flushed with all the rest. Another interesting discussion in which comments were erased was https://pubpeer.com/publications/D3DF6B4B19C383804BB3748B60604D where some 4 comments were removed, including at least one of mine (if remember well) in which I praised the authors for engaging in the discussion and suggested they posted their comments as identified registered comments, for better clarity and record. Why should such be removed?”

    Unregistered Submission ( February 18th, 2015 2:07pm UTC ) states “Sorry, but you are profoundly incorrect. There were more comments deleted than the ones you mention. So, not only was a comment deleted for no good reason, but your attempt to reinvestigate was apparently too cursory to verify it, and insteady you have dismissed the complaint. So, it is not true that “readers can judge for themselves”, as you allege, because (1) you deleted the information from the discussion thread, and then (2) you have inaccurately represented your actions in this thread.”

    Peer 0 (February 18th, 2015 9:48pm UTC) responds “- Comments not focused on data may be removed (even if they are polite and relevant). – A huge stream of comments over the last few months overwhelmed moderators. As a result, we made major changes to our back-end that makes moderation much more efficient. If reported comments still appear on the site please report them again. [Oops… It seems you are right: When we disabled a comment on that thread a comment nested below it was also disabled. Since this is a borderline case, we have now re-enabled both. Please accept our apologies.]”

    Peer 1: ( February 19th, 2015 3:33pm UTC ) responds: “Thanks for a reply. I take form this discussion that not only relevant polite comments can be removed — not quite sure if exactly at random –, but also actually comments directly discussing data (such as the borderlined case above and some others I mentioned) were deleted, while clearly inappropriate — reported — comments can persist. I think there is no point in investing thought/time in writing an elaborate comment or following any deeper discussion where its contents can be silently edited or disappear without clear reason.”

    PS: disclaimer. I am none of the individuals who has commented in any of the comments posted above (Peer 0, Peer 1, Unreg).

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.