PubMed today launches a pilot version of PubMed Commons,
a system that enables researchers to share their opinions about scientific publications. Researchers can comment on any publication indexed by PubMed, and read the comments of others.
In general, we’re big fans of post-publication peer review, as Retraction Watch readers know. Once it’s out of its pilot phase — and we hope that’s quite soon — PubMed Commons comments will be publicly available. So this is a step forward — but only a tentative one. That’s because of the first bullet point in the terms of service commenters agree to:
- Only those individuals listed as an author on a PubMed citation may make comments in PubMed;
- If possible, provide detailed references to papers (eg. page numbers, figure pointers) and unpublished data; refer to external websites if a longer comment or reference to other work is necessary;
- Do not describe or share unpublished work by others;
- Comments should not contain discriminatory, racist, offensive, inflammatory, or unlawful language;
- Comments should not contain partisan political views;
- Comments should not have explicit commercial endorsements.
So PubMed Commons isn’t exactly a town commons, unless you happen to live in a town whose residents are all scientists.
Stanford’s Rob Tibshirani, who was involved in organizing PubMed Commons, wrote in a blog post prepared in advance of the launch:
One would like the system to be inclusive as possible but many scientists would not be interested in posting comments in a system with a high proportion of irrelevant or uninformed comments. NIH also needed a rule for who could post that would be pretty clear cut and not based on e.g. some judgment of the experience or knowledge of the participants. The decision was made that comments could only be posted by authors of papers in PubMed. This would make the situation symmetric in that all people who comment can have their own work commented on. It would also include a large number of potential participants and would meet NIH’s need for something unambiguous. Unfortunately it would leave out many people who could add valuable input, including many graduate students, patient advocates, and science journalists. I’m a little worried about this restriction, as I want to make the system open to as many users as possible. But hopefully that is a pretty wide net, and it may be widened further in the future.
We’re a bit worried about this restriction, too. David Lipman, director of the National Center for Biotechnology Information, which runs PubMed, tells Retraction Watch:
We wanted to have it be inclusive enough that it would be a wide range of people commenting. On the other hand, if we open it up to anybody initially, we were afraid we’d see a lot of the the kind of comments you see in Popular Science and on newspaper sites, and scientists don’t want to deal with that. This felt inclusive but filtered enough, to people with a stake in it.
Popular Science, of course, recently shut down their comments after tiring of “trolls and spambots.” The magazine cited research of comments’ so-called “nasty effect” by University of Wisconsin professors Dominique Brossard and Dietram Scheufele. We asked them for their take. Scheufele told us:
If the same effects hold that we found for online news articles, we could see a similar effect where just the tone of (unmoderated) discourse reshapes how we interpret the scientific articles immediately preceding them. In this sense, separating the outlet for publication from the place where we might engage in – sometimes heated – debates about what these articles mean or how much we agree or disagree with conclusions, might lead to a better debate overall.
We’re actually among those who can comment on PubMed Commons, because we’ve published a paper indexed in PubMed — all about post-publication peer review, as it turns out. But we really hope this net widens, and soon. Of course many scientists find it beneath them to read or respond to comments from the great unwashed masses — you know, the great unwashed masses whose taxes pay for their grants. What a terrible inconvenience! Sure cuts down on the number of people who might criticize your paper, though.
We think this is a particular problem because PubMed won’t allow anonymous commenters, either. Tibshirani writes:
After much discussion, the group remained deeply split on this issue. Those wanting anonymous posts were concerned that many scientists, especially junior researchers, would be reluctant to make critical comments. But those opposed to anonymous comments believed that the quality of interchange would be higher if commenters were required to identify themselves. In the end, these differences weren’t really resolved and the decision was to start without anonymous comments and reevaluate after the system had been fully public for a while. While debating this issue various proposals were put on the table for ways to allow participants to review and essentially sponsor the anonymous post of another participant.
That last bit is interesting, and could be a bit of a stopgap measure. But it’s not enough. By one measure, those opposed to anonymity are correct to say that “the quality of interchange would be higher if commenters were required to identify themselves.” Comments such as “This was a great contribution to the literature” are more likely. But the kind of respectful but critical questioning that helps correct the scientific record will be much less likely, because of realistic fears that senior scientists won’t like it very much when you criticize them. We’re once again struck that many journals and scientists seem to believe fervently in anonymous peer review to prevent this sort of revenge, but won’t allow anonymous comments because they make them uncomfortable.
Lipman:
I absolutely think that is an issue, and that is the argument that went on internally among people who were involved. There are some people who won’t be afraid to make a collegial, constructive, but highly negative critique. But I do think that the advantage of having your name show is, you’re more careful about how you write it.
We already have the web as a whole where you can say whatever you want. Then we have letter to the editor. I’m imagining that this is something in between, more efficient, more fluid, and more unrestricted than a letter to the editor.
Fair enough, but we really hope the net is widened to include anonymous comments, too. Or PubMed could leave that space to PubPeer, which has been quite successful so far and has managed to moderate effectively. But that would be a shame. PubMed has a great opportunity to be inclusive, and its traffic would mean high visibility for post-publication peer review, along with more pressure on journals and others to correct the scientific record.
Scheufele has a different take on that, though. He worries about “the crowdsourcing of academic peer-review:”
…We spent decades or centuries coming up with a system that would allow us to fund the best and most promising research. The system is far from perfect, but it is far superior to putting scientific expertise on the same level as public opinion when it comes to judging the scientific merits of an article. I could very much see an instance, for example, where public discourse questions if we need research on girls’ body image. The scientific importance of this research is obvious to me, but I think it would just be a matter of time before we’re seeing a variant of campaign ads like this on this topic.
He continues:
As I said elsewhere, more conversation about issues, including science, is always a good thing. What the best format is for those conversations and how we can structure it in order to make sure we get the best outcomes, however, are empirical questions. and I am always wary of efforts – no matter how well meaning they might be – that rely on someone’s intuition about what might work rather than empirical data about what will work and what will not. And I am not seeing a lot of empirical data being used to justify this new mode of online commenting by PubMed. Until we have that research available, it may make sense to maintain at least some of the weight-bearing walls of the ivory tower.
What PubMed Commons will end up looking like will be informed by this sort of discussion. Lipman seems open to change:
This is not an NIH project, except that NIH is allowing them to do it. We need to get the drive and direction from people who are going to be active in it.
In a FAQ, the NLM calls PubMed Commons
…a forum for open and constructive criticism and discussion of scientific issues. It will thrive with high quality interchange from the scientific community.
Let’s make it truly open, and end that second sentence with “high quality interchange,” full stop.
“Of course many scientists find it beneath them to read or respond to comments from the great unwashed masses — you know, the great unwashed masses whose taxes pay for their grants. What a terrible inconvenience! ”
Which just makes their point that it’s easy to make snotty and condescending comments that contribute nothing to a discussion, other than to turn people off. This is exactly the sort of “drive-by-commenting” that risks rendering the system uninteresting.
Hmm. Maybe some entrepreneur could set up a PubMirror site that offers comments from the masses based on the original PubMed record. That would actually be an interesting experiment.
You wouldn’t have to mirror the whole thing. You could just use the IDs to pull the titles, and if you wanted to set up a new discussion just seed it with that ID/title.
Changed my mind. It should be PubPub. Could be structured just like PubPeer, but without the author notices and such.
I just went to look at an article I know was a hot-hot-hot discussion item around science last year and the public in many social media forurms (10.1016/j.fct.2012.08.005), and there’s not a smidge of discussion over at PubPeer on it. So it needs to be broader if you really want the conversation to include the public.
Something like this was created by Bosco Ho and Mark Reid; it’s called annotatr.
I found that sentence unnecessarily snarky. Keep the sense of humor, but please, drop the snark.
That comment also demonstrates a gross misunderstanding of the job that scientists are hired to do — researchers get paid to do research, not to do unlimited public education. Expecting researchers to engage with the public in fora like this is equivalent to requiring every professor to always have his door open so that any person off the street can drop in to question him about whatever they are curious about at the moment (which does happen at my university).
> the job that scientists are hired to do —
> researchers get paid to do research,
An honest non-snarky question: what obligation do they have to produce work of a certain quality? Who gets to “cry tripe where tripe is served,” as a playwright said?
I’m no career scientist but I’ve heard plenty of anecdotes of inside baseball, friends “peer reviewing” each other’s work, etc, all resulting in scientists in some cases NOT achieving “do research” in any scientifically valid sense.
Honestly, what’s the remedy for this?
Communicating to someone who does not already know the “technical language” in science can get very tricky. There is a reason for all this terminology- it’s so we can be precise with each other.
I think part of the solution is for us scientists to work on recalling how to communicate with folks who do not have degrees in our exact subjects- and for the public to help us by educating themselves, with the resources we put up on the internet!
For individual scientists, they are obligated to their peers. These are the people who decide whether they get grants and whether they keep their jobs. Politics (such as “likability”) surely plays a role in these decisions, but I know plenty of situations where a well-liked person was rejected for grants and tenure. I personally have no problem being blunt while reviewing papers and grants, even if I like and respect the authors.
This system may be imperfect, but it’s the best system that I’m aware of. These issues (e.g. conflicts of interests and personal feelings) play a role in all jobs. The alternative seems to be excessive reliance on formula (e.g. seniority and “objective” performance measures), which are no better at rewarding merit.
Outside of peer-evaluation, evaluation applies to scientific institutions (not individuals). This evaluation is done constantly by the people who apply science (e.g. engineers) and those who fund it (politicians and the public). My understanding is that the USA’s scientific institutions are very effective.
And then there are independent fora such as science magazines and websites (“Science 2.0” comes to mind as one site where there is heavy skepticism of the scientific establishment). If there is going to be some forum where the public provides feedback on the effectiveness of scientific institutions, I don’t think that PubMed is it. PubMed is full of primary publications, and they are inevitably messy and incomplete.
I disagree with the basic premise of implementing commenting on PubMed abstracts right now on two counts:
1) Time is a limited resource, and all scientists I know are looking for ways to better navigate the deepening sea of email, texts, posts, databases, pdfs, meetings, calls, panels, etc. that can either enhance or diminish creativity, productivity, and joy in trying to do good science. Implementing commenting on PubMed Abstracts will lead to both useful and useless activity, and it’s time that we start holding new information outlets (in science) accountable for value beyond just a few good anecdotes. I suggest holding off implementation until a plan for allowing it to evolve to achieve optimal value to the scientific community is in place. In the current web pages describing PubMed Commons and its accompanying FAQ, it states that a user can send a comment to the help desk, but beyond that there is no description of how or if the value of PubMed Commons will be measured or improved once it is running.
2) There is ample evidence to show that what gets into the published biomedical literature is generally poorly reported in a number of important ways (search Google for “pubmed papers on reporting quality”), so it makes far more sense to focus on quality of what gets published, rather than commenting on abstracts of poorly reported publications. In 2009, we (a large group of collaborators) submitted an R01 proposal to NIH/NLM to develop a nonprofit system that would work with publishers and PubMed called Xiopub to make it easy for authors to implement standards directly in their manuscripts (abstract is pasted below), but unfortunately one reviewer loved it, one was lukewarm, and one nuked it, so it did not get funded. We subsequently have advanced the concept, as shown here: http://www.youtube.com/watch?v=Z0UFBHnn1aA but we still have not been able to raise the approximately $3 million needed to establish the community, build, test, and do pilot implementations of a scalable prototype of what we now call Symeme. In addition to reducing omissions, the envisioned system would allow comments to be tied to specific parts of the article, which, if well-implemented, would be far more powerful than allowing general comments on the summary data of an abstract.
Here is the abstract from our 2009 R01 proposal which was not funded (this was its third submission):
“The long term objective of this project is to significantly improve the usability of new bioscience publications, and by doing so, to improve the quality and rate of progress of bioscience research itself. Progress in bioscience research is proportional to the quality of the publications upon which a researcher can formulate critical next questions. Our research has shown that the current bioscience publication system performs poorly for many of its intended purposes, including functions we have termed Grazing, Repeating Experiments, Reviewing or Aggregating Literature, Evidence-Based Health Practice Support, and False or Erroneous Publication Detection. We have also demonstrated that a number of prominent researchers have similarly identified problems with the current publication system, and have taken action to improve the usability of publications in their own domains by creating and implementing standards checklists and other types of data reporting standards. These standardization efforts are promising, but are held back by their lack of convenient integration with the existing publication system. We performed eWorkshops with a broad spectrum of stakeholders in the current bioscience publication system, and developed the concept for XioPub Aqua (XPA): a bioscience publication builder and integrator. XPA will provide a convenient way for publication authors to learn about existing standards and to tag the manuscript content pertaining to specific standard items in an easy step by step process. These tags will become part of the manuscript, and will add significant value to all bioscience publication stakeholders as follows: Authors who do a good job of tagging their articles will have their articles cited more often; Authors of “Xiopublications” will publish faster, because the downstream review process by Editors and reviewers will be more efficient; after a sufficient number of Xiopublications exist, authors will benefit because finding articles pertinent to a specific question will take less time; Reviewers and Editors will benefit by being able to more easily check adherence to standards and add comments to specific parts of manuscript text; Publishers will benefit by being able to provide a higher-value product to their readers; Publication Users in general will benefit by being able to extract and view tagged standards-specific data from groups of Xiopublications, substantially increasing the ability to compare and contrast study results. Standards Creators and Curators will benefit by having a forum in which they can manage, analyze, and improve the quality of their standards based on use statistics and user feedback. We propose to perform the research and development needed to deploy an open-source, community-curated, performance proven XPA through three specific Aims: Specific Aim 1: Complete functioning prototype XioPub Aqua system. Specific Aim 2: Implement XioPub Aqua system in three Vertically Integrated Test Teams including prominent publishers and the NLM. Specific Aim 3: Move XioPub Aqua production systems to a well-prepared nonprofit XioPub foundation for continued support and development.
G. Steven Bova
[email protected]
Dear Steven Boa, at the end of your explanation, the only thing that matters to scientists is, would your service be FREE for users and viewers alike? Your venture sounds more like a business venture than a free, altruistic service to the science community.
This depends on whether PubMed is a tool to help scientists get their work done or if it’s a tool for public education. I think of it as a tool for scientists — in which case limiting comments to scientists is totally appropriate. Scientists lack good public (global) fora in which to discuss research.
Patient advocates and science journalists already have prominent places to publish their opinions about research. The only (high profile outlet available to scientists is peer-reviewed journals, which is an incredibly slow and tedious process.
In all honesty, it also is a very ill-informed comment even if we ignore the snark. Let’s take it at face value: respond to the people who pay your grant money!
Well, that means I can ignore any questions from people from just about anywhere in the world, as they are unlikely to have contributed a single cent to anything I have done. When they don’t pay taxes in the country where the scientist works, they have no rights, because they did not contribute financially.
Also, take the papers coming out of industry. Does that not mean then that no one can ask questions? Those great unwashed masses definitely did not pay for that through taxes.
We ought to remind ourselves that we are (largely) funded via the public purse and as such Joe public has every right to comment on anything he funds. That includes science and science research. It would be refreshing.
David Lipman, director of the National Center for Biotechnology Information, which runs PubMed, tells Retraction Watch “……we were afraid we’d see a lot of the the kind of comments you see in Popular Science and on newspaper sites, and scientists don’t want to deal with that.” I respect but vehemently disagree with David Lipmans stance.
Actually, as a scientist I do want to deal with public engagement, school trips and public comments. This is either 100% or not at all. I would strongly encourage any biomedical scientists reading to be very eager to be asked questions or be given suggestions by engineers, which may of the public that have been blocked from commentating are.
The science community may be made a laughing stock if we hide behind anonymous peer review for publications and grant applications and will not allow anonymous post peer review or restrict public engagement on the science they fund.
We have nothing to hide and everything to share withy the public.
I suspect politicians will defend the public right of full access and judgement sooner or later.
Of course everyone has the right to comment on research. It’s simple free speech.
It’s also reasonable for the public to want to see what they are paying for — transparency is an important part of democracy, and any discussion on PubMed commons would be publicly visible (as I understand it).
However, it’s also reasonable for professionals to develop resources that support their professional activities; for scientists, the organization and evaluation research publications is an essential resource. To do this, we need to rely on each other’s opinions. When we’re deciding what opinions to pay attention to, the credibility of the speaker is important, and PubMed Commons seems to have settled on a reasonable gauge of credibility for providing opinions on scientific articles. It may not be perfect, but it is immensely better than what we have now, and it’s much more important to get something that has a good chance of just working, rather than something that might be “perfect” but also is more likely to be a complete failure (whether due to technical of social limitations)
There is no reason that PubMed Commons needs to be a forum for engaging with the general public. Scientists have already developed plenty of systems for communicating our results to the public (e.g. popular science publications, university education), and I see no reason that PubMed commons would be remotely as effective as the existing alternatives.
If “joe taxpayer” wants scientists to spend more time engaging with the public, then they should pay scientists to engage with the public. Even if we were to consider “the public” as our employers, it’s not as though our employers own us and can arbitrarily demand extra services from us on a whim.
It would seem that today’s science communication is quite unidirectional: You the scientist tells Joe Taxpayer a condensed and often-times watered-down story around your scientific research and Joe Taxpayer has no appropriate means of asking questions.
In my opinion, anything that gets people to comment on scientific output directly — as opposed to friends’ Facebook statuses etc. — would be a benefit for us all.
Apart from all this, don’t you think that today’s model is extremely biased to Joe Taxpayer’s disadvantage?
How many decades have researchers now spent burning through taxpayer money with almost zero direct or indirect feedback from Joe T?
Think of all the money that’s been wasted on playing the journal lottery (Nature, Science, Cell). Think of all the $ wasted on grants for people that published sloppy or falsified data just because the buddy system works so well in pre-publication peer reviewing.
Infusing the system with a little sincere scrutiny can’t hurt — let’s go from zero to just a tiny bit and see what happens.
It’s free speech until it affects affects a grant review and/or a publication decision. Then it becomes restricted speech or false speech. Only a few favored researchers are able to speak their minds today. The junior scientists/faculty can’t risk upsetting anyone and so are forced to either give false praise or remain silent.
Adamretchless said “Scientists have already developed plenty of systems for communicating our results to the public (e.g. popular science publications, university education)”
How are recent scientific results communicated to the public through university education? Maybe knowledge gained in the 1970s has finally made it into the education system. However, the more recent knowledge from last year may not appear in academics till the end of the decade.
“How are recent scientific results communicated to the public through university education? Maybe knowledge gained in the 1970s has finally made it into the education system. However, the more recent knowledge from last year may not appear in academics till the end of the decade.”
At any good university, science majors are studying recent primary research reports in their own field (by the time they are seniors). After receiving this training, they are capable of reading the scientific literature on other topics that interest them (assuming that they are willing to dedicate the time to studying background information). An increasing amount of scientific literature is publicly available, which I think is a good development.
For people without that training, it is fine and appropriate that they are being taught about discoveries that are >10 years old. It takes that long for researchers to figure out the meaning of major discoveries. The “latest research” is not useful to the general public (even engineers and doctors, usually) — it is only useful to the people who are participating in the cutting-edge scientific discussion (and curious onlookers). The fact that something is old does not mean that it is incorrect — if a new discovery does indicate that established theory is incorrect, then responsible teachers and reporters will mention that caveat when they describe the older theory.
If someone (like an engineer) does need clarification of something described in a research report, he can always email the authors. That doesn’t guarantee that he’ll get a response, but if an engineering firm needs the attention of a researcher, they should hire the researcher as a consultant.
Finally, there are tons of academic programs to transfer recent research knowledge to the people who apply that knowledge. Medical schools have continuing education for doctors (as do professional societies like the Americas Society of Microbiology). State-run Agriculture schools employ farm advisers who regularly collaborate with the full-time researchers.
“It’s free speech until it affects affects a grant review and/or a publication decision. Then it becomes restricted speech or false speech. Only a few favored researchers are able to speak their minds today. The junior scientists/faculty can’t risk upsetting anyone and so are forced to either give false praise or remain silent.”
I don’t know what country you live in, but I’ll try to interpret this in an American context. First, I have to note that you’ve changed the topic — I was talking about the general public’s right to comment on research, you’re talking about the ability of researchers to frankly criticize each other’s work in public fora.
Your assertion that junior scientists can’t risk upsetting anyone is completely at odds with my own experience. My field is big/diverse enough that I am unlikely to have my work reviewed by any particular person, so even if I make an enemy or two it’s no big deal. The papers that I would criticize are often written by foreigners who have less influence on my career than other scientists. Also, sometime an “outsider” will publish a high-profile paper that is relevant to my field, but they will demonstrate their ignorance of established theory and I will need to set the record straight. In this case, the reason that I need to criticize their paper is tightly linked to the reason that I don’t have to fear retaliation — they are not experts in my field of research.
Even within the core community of experts, most people welcome a good debate. Such debates often drive publications and grant funding. Participating in a high-profile debate is a good way to raise one’s own profile. The worst thing is to have your work ignored.
I’m sure that some scientists abuse their power, but turning PubMed into a public bulletin board is not going to stop them.
@adamretchless
“Your assertion that junior scientists can’t risk upsetting anyone is completely at odds with my own experience.”
This can happen a lot in other nations, due to how the money is distributed for science. (I saw it happen in Germany a few times.) In the USA, junior biomedical people can be put under pressure to “tow the party line”, so to speak; again because of the way the funding system works and the way they try and rate the quality or impact of science. But, this very, very much depends on the field- and how crowded the field is.
Adamretchless,
I am an American so you don’t have to interpret anything for me. I am glad your field is big enough that you can upset people without retaliation. This is not always the case.
You wrote “Also, sometime an “outsider” will publish a high-profile paper that is relevant to my field, but they will demonstrate their ignorance of established theory and I will need to set the record straight.”
So you strictly enforce the established theory? I thought the whole purpose of science was to revise the theory based on new evidence…
I sense some hostility in your writing.
Mark: “you don’t have to interpret anything for me”
I was interpreting for myself. I was describing the basis of my interpretation so that you understood why I wasn’t considering libel laws and other legally enforceable restrictions on speech. Your use of the terms “false speech” and “restricted speech” seemed to be references to such restrictions.
Mark: ” I thought the whole purpose of science was to revise the theory based on new evidence…I sense some hostility in your writing.”
A theory cannot be revised or disproven if it is not being addressed in an analysis. That is the problem I was talking about. While I am very frustrated by the high-profile publishing of bad analysis, I’m not hostile towards the people who do it — they tried, and they made a mistake due to their ignorance.
But to get back to the original issue (leaving my emotional state behind), I expect that the vast majority of the potential useful comments on PubMed Commons will not be subject to self-censorship. Online anonymity has its own problems, and the benefits of anonymity in this situation are not great enough to compensate for those problems.
If some scientific institutions are so dysfunctional that researchers do not feel free to respectfully criticize each other’s work, then those institutions need serious reform. I don’t think that anyone has suggested that PubMed Commons would contribute to meaningful reform of those institutions. I am not interested in designing our institutions around the lowest common denominator. Healthy professional communities will find PubMed Commons very useful with signed comments, and that should not be put at risk for the sake of allowing unhealthy communities to mask their problems.
It’ll be interesting to see which fields have a culture that encourages young scientists to openly criticize established scientists.
This is exactly correct. If you want this to be productive, then you need informed opinions. I would say the net is wider than need be with those requirements. Not to sound elitist, but the average layperson has little contribute to the average Nature article. Consider the average junior professor upon first appointment has 8-10 years post college training.
whoops, this got misplaced somehow. I meant to reply to the first commenter.
“Only those individuals listed as an author on a PubMed citation may make comments in PubMed; ”
That is not the only criteria you have to meet – at least at present.
http://www.ncbi.nlm.nih.gov/pubmedcommons/faq/#joincommons
In addition to being listed as an author you must either
1. Be a NIH or Wellcome Trust grant recipient or
2. Get an invite from either a grant recipient or someone already participating in Pubmed Commons or
3. Belong to a group of at least 50 authors with email addresses ending .ac or .edu who wish to join en masse.
While it’s a nice idea in theory to open up commenting to *everybody*, scientist or not, I can see at least one major problem. Let’s assume that PubMed has lots of articles on various types of vaccines for human disease. There’s an unfortunately large contingent of people out there, both alleged “scientists” and the “great unwashed masses”, who firmly believe that vaccines “cause” autism. It’s bad enough when the so-called scientists want to flamewar each other. But when you open up commenting to every parent of an autistic child who’s convinced their child would be “normal” but for this vaccine or that vaccine, instead of having an honest scientific debate, now you have to deal with a comments section that’s bogged down by unnecessary, irrelevant, uninformed vitriol from people who quite literally don’t know what they’re talking about. In short, I understand wanting to open up the comments to everybody, scientist or not, who’d like to contribute to a discussion. But I also understand wanting to limit the audience in an attempt to also limit the irrelevant flamewars. PopSci, as you noted, got rid of their comments because they got tired of dealing with “trolls and spambots”. In order to properly moderate a site, someone’s got to be watching it 24/7. PopSci decided it was more cost effective to just get rid of the comments than to pay someone to sit there watching the screen, waiting for someone to say something they shouldn’t.
So, PubMed Commons for when you want to use your real name, and PubPeer for when you want to remain anonymous. As someone who has experienced first hand the legal shit-storm that results from using one’s real identity in such situations, I think I’ll be sticking with PubPeer thanks.
One possible downside is that many of the comments will end up as brown nosing about Prof Bigshot’s latest paper. Of course there will be exceptions, but in regard to allowing anonymous comments, you’re damned if you do and damned if you don’t. Depending on how popular the system becomes, moderating it probably would not scale well. Perhaps one could apply some of the techniques that commercial “review” operations such as Yelp use, but even they obviously struggle with balancing openness and fairness – and have a lot more resources at their disposal.
Time will tell, but nevertheless this is a start that hopefully can be built on in the future.
Agreed. I am warming to pubpeer these days.
It is still akin to eating a raw onion sandwich rather than a meat-feast deli special, with all the toppings, when compared to a certain site that is no more.
This sounds interesting to try and watch how it develops. I can imagine a system like here where answers or comments get a rating – that may help to get some of the most interesting points out of many commenters.
While I in theory approve of opening Pubmed articles to public commenting, I’m concerned about the overall quality of commenting and the commenting culture that will arise on the site. According to the Pubmed website, they have over 2.8 million articles archived in the database. With that many articles to moderate comments on, their seems to be a large potential for abuse of the commenting system (both by “unwashed-masses” and researchers alike.) I would hope commenting would allow for a more open discourse both among scientists and the public, but the reality is such a large database will be difficult to manage. Anyway, I’ll be interested to see how the current plan to open abstracts to commenting from a select few transpires.
More than 23 million, actually.
Well I guess it had to happen sooner rather than later, but within hours of the site opening I managed to get a comment banned/removed. Yay first!
It was essentially a re-hash of this comment at PubPeer (https://pubpeer.com/publications/D2A46528724F9B59FD58693CA41560) with an update thrown in about the response of the journal (i.e. 13 weeks passed , no explanation for why this relatively simple matter has not been dealt with, and now they’re just ignoring my emails).
So this means 2 things….
1) Someone (NIH staff) is watching, and there’s editorializing going on here.
2) The policy (under usage guidelines) about what is acceptable is apparently not complete or well defined. I have no idea what category my comment fell into that got it banned, but apparently someone at NIH thought it was not good, but didn’t have the wherewithal to actually tell me the reason.
I will report back if I can get any information on the specific reason, but I suspect it would fall into the umbrella of “could be misconstrued as alleging misconduct” The problem is, there’s NOTHING in the policy that says anything about such comments not being allowed! If such comments are not allowed (i.e. we’re not allowed to say when we think the data looks fishy), then this site just lost a lot of functionality in my opinion. So, that’s PubPeer 2, PubMedCommons 0.
Ack – I was stupid. In between being on PubPeer and trying to paste my comment across, coupled with logging in through eRA commons ID, I managed to click on the article with a VERY similar title, and paste the comments into the wrong article. (Bangs head on desk and promises not to make silly mistakes again). So, it seems an apology is in order for PubMedCommons staff, who were correct in removing my comment. The comment has now been appended to the correct article.
Ooops yeah I know how it feel when this kind of crap happens! It is OK
In my opinion, it is entirely appropriate that commenting at PubMed be limited. Can you imagine visiting PubMed and finding something that looked like a YouTube comment thread?
“First”
Why not allow the authors of the paper to decide what kind of commenting they want? The open-access movement has thrived because some scientists were willing to take a hit in impact factor to open research to a wider audience. Maybe anonymous and completely open commenting won’t be popular at first, but may become more mainstream.
One issue is the “backlog”. The demise of Abnormal Science and then Science Fraud, followed by the growth of PubPeer means that there may be more comments to deal with in the first yeas.
Another issue is the growth of commenting. Abnormal Science and Science Fraud were an outlet for a relatively small number of dedicated people. PubPeer is being evangelised and so comments are coming from far more quarters. So moderation load will increase.
However, there are two reasons why this will fail.
(1) The lack of anonymity is incredibly disappointing. If pre publication peer review is anonymous, why not pos publication. Is anyone scared? Generally the more eyes the better.
(2) The restrictions on who can comment are also disappointing. Sure, there are issues with tolls and people with an agenda, but no supporting data, e.g., creationism, anti vaccine lobbies. However, good dose of moderation, so that critiques require data, e.g, blot A looks the same as blot B, but A and B describe different experiments on different proteins, plus a ban on metaphysics, and we are good to go.
Scientists do science and are too busy to respond? I work in a University, and yes, my door is open to all and I respond to queries from undergrads, professors and the public. It may surprise some, but I consider myself to be a member of the public. Indeed, one day I may wish for advice or information from an expert in a discipline I know nothing about. So this is part of the job and to consider that it is not is dereliction of duty.
This is an excellent step forward and very good news. Actually, I agree with the first bullet for one simple reason: it ensures that only peer review takes place and not comments by biased individuals who may not be suitably qualified and who add “noise” to the critique system. The name Bohannon immediately comes to mind as the type of “scientist” that PubMed and the NIH do NOT want making comments on PubMed Commons because they are journalists and not scientists. Now, Elsevier has to follow suit with such a service on sciencedirect/scopus, Springer with SpringerLink, Wiley with Wiley Online, Taylor and Francis and Informa with their platforms and then step by step, publisher by publisher. Maybe mass pressure on the publishers I list could ensure that such a new system could be implemented quite soon. If PubMed makes this move, and if they could get the “ethical” endorsement of COPE, then it will be difficult for the main-stream publishers NOT to implement such a system. These publishers are about profit, so any hint of dissatisfaction by their “base”, i.e., the scientists, because a suitable “peer” and “justice” system are not in place, could rock the profits boat as scientists flee elsewhere where there is a form of justice and true peer review, including post-publication peer review. Even great portals like Wikipedia or DOAJ could have similar systems…
“If pre publication peer review is anonymous, why not pos publication”
There are a variety of differences between the two:
1) pre-publication review can block your publication.
2) pre-publication review is mediated by an editor, who is not anonymous… and the reviewers are not anonymous to him.
3) pre-publicaiton reviewers are a small group of people who have been asked to provide a service. The post-publication system involves a large group of people who are making comments by their own initiative.
These facts don’t clearly indicate whether either of these forms of feedback should be anonymous or not, but they do indicate that a simple analogy cannot be drawn between anonymous comments on PubMed Commons and “anonymous” pre-publishing peer-review (which is only semi-anonymous).
” It may surprise some, but I consider myself to be a member of the public.”
Given that the definition of “public” is that in includes everyone, there’s nothing surprising here.
Pre peer review is anonymous, not semi anonymous – that an editor “knows” does not change the fact that to the authors the reviewer is unknown.
Similarly, if you register and have to l login on a post publication review site, then someone (indeed likely many) in the system know who the anonymous reviewer is.
In pre publication a reviewer can block publication of allow complete garbage to be published. In post publication, a reviewer can demonstrate that a paper is garbage.
So are the two really so different?
People behave differently when their identity is known, and as a result they require less external moderation; that’s why it matters that the reviewer is known to the editor, and it’s why anonymous online comment systems have such a troublesome history.
Likewise, there is a big difference between being involved in a conversation where someone that you respect (the editor) is paying attention to you, and a situation where some random bureaucrat has the ability to look up your name if you manage to catch their attention (assuming that the “anonymous” commenting system requires you to register with your real name).
“In pre publication a reviewer can block publication of allow complete garbage to be published. In post publication, a reviewer can demonstrate that a paper is garbage.”
My papers are not garbage, so the difference is huge.
just to be clear on that last point:
I am much more likely to get annoyed at a pre-publication reviewer who offers pointless/misguided criticisms than I am to get annoyed at a post-publication reviewer who does the same. In the later case, I would try to respond, and then move on with the assumption that other readers would see the errors in this criticisms. In the former case, I may have to satisfy the reviewer before anyone would have a chance of seeing my report. In the worst case, I might start to wonder if the reviewer is trying to sabotage me (it’s never happened to me, but I’ve heard of it).
This could be used as an argument for or against reviewer anonymity, but either way it only applies to pre-publication review, where the reviewer’s power is much greater than in post-publication review.
Adamretchless
1. “pre-publication review can block your publication”. It can also be open to abuse where editors and reviewers are chosen by the authors. The review may, therefore, be at least in part, predetermined and the data not scrutinised appropriately. Post publication review soon publicises this as we have all seen.
2. “pre-publication review is mediated by an editor, who is not anonymous… and the reviewers are not anonymous to him”. Or her, of course. But some pre publication review is anonymous. There are no hard and fast rules followed by all journals and this is a problem where well-known, well funded experts may allegedly use their influence to interfere with due process of rigorous review.
3) “pre-publicaiton reviewers are a small group of people who have been asked to provide a service. The post-publication system involves a large group of people who are making comments by their own initiative”. The “service” asked of a small group of people may be tainted with a bit of pressure to allow a less than rigorous review of the data and indeed maybe, albeit inadverdently, a disservice to science as a whole if data is not rigourously reviewed.
There is then the possibility of the old boys network where a reviewer may be chosen by authors as they have grant applications on each others desks. Even if problems are seen in a review it may not be flagged for fear or reprisals.
We should have anonymous post publication review so no-one is exposed to the storm that Paul Brookes has been.
Adamretchless’ point seemed to be that pre and post publication reviews do not take place under the same conditions. He did not argue in favour or against either, as I read it. In all your three points you seem to give more arguments that they are not the same. Next, Adamretchless concluded that because there are differences, one cannot generalise any prefered (non)anonimity for one type of review to the other type. That seems valid.
I agree with Stewart that anonymous post publication review is preferable. I know researchers that are, let’s say, less inclined to see criticisms as constructive. As a personal anecdote, I published in Frontiers once, which discloses (prepublication) reviewers, and noticed that in the final round of comments I was amically called by my first name and congratulated with another nice paper (this has never occurred to me in anonymous reviews; I suspect that the reviewer needlessly felt that he should make clear that the reviews had been well-intended). This made me realise that reviewers simply do not behave the same when there is a risk that authors might hold a grudge because of received critical comments (whether that fear would be justified or not). Since that innocent episode, I never review any paper unless it is anonymous: for prepublication, I would simply reject the invitation; for post-publication, I would simple not post.
I realise that that is no argument that anonymous review is “better”, but it is the only type that I choose to participate in.
Aside, note that any reviewer can voluntarily choose to disclose their name in an anonymous review (unless an editor edits that away, which I haven’t knowingly encountered yet), but, reversed, no reviewer can un-disclose their identity once it is revealed. That alone would be an argument for me already to advocate anonymous reviews: they are more inclusive. I would advocate some form of registration or qualification though, to avoid trolls from taking over.
I agree that post-publication review is very important. That is actually why I am defending the PubMed Commons proposal against insinuations of corruption (elitism, ego protection, etc). The way that the system is described, it will gather all of the low-hanging fruit of post-publication review. As I see it, this system, will generate 90% of the benefit that can be gained from post-publication review, while requiring only 10% of the effort that would be required to generate the full 100%.
Making the system open and anonymous will require a lot more work while providing only a little bit more benefit. I’d hate for the administrators to get bogged down in the complexity of implementing an open and anonymous system, such that they never establish anything. I’d also hate for government bureaucrats to be responsible for moderating (censoring) a broad public discussion about scientific research. It’s much better to keep this system limited to a smaller community where professional etiquette and concerns about one’s reputation results in more self-regulation.
I’m sure that some good comments will be withheld due to fears of reprisal, but I’d bet that they will be a small minority. Even if fear of reprisal means that the comments will be limited to established researchers, the value of these comments will still be immense relative to what’s available today. In my own case, I can think of two publications that I would have loved to provide critical public reviews for, even as a graduate student. One such publication claimed to have rejected a model that I proposed in my own paper, yet their analysis completely failed to address my model. In another case, a high-profile publication reached some absurd conclusions because they failed to consider the dynamics on which my model was founded. In both cases, I think I could only benefit from criticizing these papers. We would have submitted these criticisms as letters to the journals if we thought that they would get published without undue effort. If I did feel too threatened to publish these criticisms on my own, I could have brought them to my adviser or another senior research whom I knew, and perhaps they would have published the criticism or somehow provided me with support (perhaps by co-authoring or publicly endorsing my criticism).
PubMed started as part of the solution. Then they understood what “change” means and became part of the problem. They don’t like my computer: my browser is no good, they want to set cookies, want java script, want subscription or something else, otherwise – not showing any Pub papers. Those who want to show papers, like arXiv, simply give me the papers. With their New Initiative, PubMed shows Leadership, but it should be clear that choosing the participants, they are choosing the contents of the comments. To keep scientific discussion going, all you need is to expand Letters to the Editor into the e-format, for each journal (no anonymity of course).
I’d suggest that seeing how this works and then broadening it is a good strategy, unless they have a really terrific moderation and reputational system to efficiently weed out wastes of time or put them elsewhere.
If offer an analogy from past history.
Once upon a time there was USENET, which had many online discussion groups of various kinds, starting in the mid-1980s. One was comp.arch, for computer architecture, and the discussions were open, but were sensible enough that a reasonable number of serious experts participated, as happened with some of the other technical groups.
Then AOL opened up USENET, an event later called Eternal September, and there would be waves, and that was survivable at first, but it got worse with the WWW … to the point that serious experts mostly quit posting, because it got be too much work to sort through the nonsense, even with KILLFILEs to help.
Maybe PubMed is technical enough not to suffer this fate, but from this experience, it is far safer to be restrictive at first, and then progressively open it up. Alternatively, one needs really good moderation tools, better than I’ve seen, i.e., one click to send a silly comment off to al alternate thread and explain why. if relevant experts mostly decide there is too much junk to be worth the time, they go away. This is why some blogs have a “Borehole” or equivalent that keep the comments, but keep them from being distractions.
If this service became open to anyone who would like to comment, I (and I’d imagine most of my colleagues) would simply not use it and wouldn’t even bother to look at the comments. Any US scientist who has attempted to “engage with the public” over Thanksgiving dinner with his/her family knows just how quickly non-expert input into a scientific discussion can devolve. (One of my favorites is “You’re just a PhD, but a real doctor posted xyz on this website that I read.”) Academic scientists have enough to do as it is. For example, we educate the children of the taxpayers and we figure out what causes the diseases from which the taxpayers suffer. Isn’t that the return on investment that the taxpayers want? Or would they prefer that we waste time debating with them in online flame wars? I think the former. So personally, I’d simply ignore an open service like this.
One of the things I miss about Germany was how seriously they took my PhD. Including the general public. Compared to Long Island, it was wonderful- people saw science education as a basic necessity for living in a modern society, not something that they are too scared to even try since it might be “hard”. Perhaps if the USA educated its population better…..
In any case, commenting is invite-only, and I suspect will be heavily moderated for a while. I’ve posted a few comments thus far. It’s nice just for keeping my article summaries online, in public, where people can see if they desire.
Yes well, Germans also obey pedestrian traffic light signals religiously, they call MDs who get a PhD “Dr Dr” – and then don’t follow it up with “I feel like a pack of cards” – just take a seat I’ll deal with you later. One hates to dredge up past history, the Germans I have known were all lovely people who would hardly ever invade Poland, but you can’t feel that the instinctive and abiding respect of Germans for authority goes some way towards understanding the Sonderweg.
All I am saying is the respect that Germans have for people holding doctorates can’t be separated with a whole other number of other cultural norms and mores that are not so desirable. For example, while I would not recommending taking a complaint of harrassment or scientific misconduct against a person with a PhD anywhere in the world, I would particularly not recommend it in Germany.
Yep. I’m more than aware Germany ain’t perfect (there are some interesting cultural diffs b/w East & West, for example, when it comes to funding). I’m more referring to the fact that I could have a conversation in a pub about what I do, and the “general public” (non-science people) followed it better than American non-scientists seemed to. Same can be said for many EU countries.
Good moderation takes time.
The Internet equivalent of Gresham’s Law for money tends to mean bad commentary displaces good, especially on any contentious topic. Right now:
EIther a blog (or equivalent, such as the commentary after new articles) is well-moderated, or noise rapidly outweighs signal, because all it takes is a few people, even ignoring the paid sock puppets as per Fox News or things like Wiki-PR. I have seen far too many blogs whose comments have been rendered useless by being unmoderated.
Unfortunately, the general state of tools is not even back to what we had in USENET in the late 1980s, where a) People were identified by email addresses, so tended to have single virtual identities.
b) With on command, you could stick someone in your KILLFILE and your newsreader wouldn’t show you any more posts by that identity in any newsgroup.
Given better tools, one might open comments up, but so far, I haven’t seen a good set in any one place.
1) Reputational systems (thumbs-up/down) are useful but often get gamed. I’ve seen blogs where total scientific nonsense (on a topic I know) get hugely liked, whereas anyone who tries to introduce real facts gets hammered. Amazon had a problem (maybe still does, haven’t checked, where disliked comments got (mostly) hidden,unless you clicked on them. A person would post a negative review on a book (iin this case, one of the sort in which a non-technical person claims that climate science is a hoax by a cabal of scientists), then someone who loved the book would ask a question. The original poster would try to answer, but get so many thumbs-down their answers disappeared. I saw a sequence where every answer got disappeared.
2) A good moderation team needs to be able to {accept, reject, or “hide” comments, in the latter case, with a code that explains why, such as off-topic, ad hominem, etc.} After all, newspapers select letters-to-editor to be printed. In this case, “hide” either means to suppress the text of a comment, while leaving a one line header, and letting a reader click to see it, or follow it to a separate thread that collects such. That is actually useful, since it allows one to calibrate commenters.
3) Readers need better tools for lessening noise. Identify on WWW is much trickier, and there is nothing that works very well to offer a KILLFILE equivalent. At the very least, one would like to be able to say:
a) Show me everything, even hidden messages.
b) Show me the comment headers, but I’ll click if I want to see more.
c) I trust the moderators, so don’t show me anything they’ve hidden.
But it would be nice to get the KILLFILE equivalent back, at least individually, especially where blogs actually require user registration so identities are a little better referenced.
“Anonymous” comments are a plague, although “pseudonymous” ones have a role, as sometimes there are legitimate reasons for hiding a real-world identity. If someone uses a consistent pseudonym, at least readers can build up a reputational model for them over time. Some blog systems allow “Anonymous”, which is simply useless, especially with multiple”Anonymous” arguing with each other.
In the real world, for better or worse, over time people assess the credibility of others, but in the virtual world, we haven’t really caught up.
In Singapore, @ Republic Polytechnic (16-19-year-olds), the first week of class for all students has trainign in how to search the Web and assess the credibility of what they find. Good idea.
Big deal. Post-publication “peer review” has been a part of many journals for a long time, especially the OA ones. The problem is that nobody really cares about it. If by some miracle a comment or question is raised, the authors rarely, if ever, respond. Citations rule and that, in my opinion, is all the “peer review” one needs post-publication.
I’d be open to two tracks of comments / thumbs-up-down … one for the people who meet the current criteria, and another for everyone.
Two thoughts.
1. Seems to me that for the PubMed Commons to stay clear of defensiveness, it needs to offer the exact same thumbs-up-down feature that this blog has. If citizen scientists aren’t allowed to comment, they should at least be allowed to boo and hiss.
2. When I learned of this policy I sniffed a familiar rotted-core odor, and quickly recognized it: Crowd trumps credentials: Medpedia’s Dead.
I know this is terribly disturbing to some scientists. Well, I remember a couple of things that have stuck with me since childhood:
1. “Any scientist must always be open to new information coming to light that means their previous view must change.” Time after time I’ve seen (and heard of) stories of scientists vigorously resisting open discussion. These people need to be reminded of the professional standards they seem to be blocking.
2. “Any scientist worth their salt can explain what they work on to a grade schooler.” Certainly not in full detail, but through the years time after time I’ve seen great scientists who do make sense.
I don’t expect anyone (including myself) to donate limitless time to limitless questions. At the same time, I feel strongly that everyone needs to be open to scrutiny of their work.
IMO real scientists can’t be allowed to put a higher priority on their comfort than on the effectiveness of the scientific method at producing knowledge that others can reproduce. Since most papers are never replicated by other labs, it seems to me that it’s dicey to put limits on who’s allowed to comment.
e-Patient Dave
Well said. At the moment science is largely a closed shop. Journals, funders and the like are trying to open it up, and are facing REAL resistence from the establishment. We need action, not just words.
It will happen, we just have to be a little bit erm, patient 🙂
Ultimately it will be the public that determine how our politicians act on this, and not scientists. We must also remember that science fraud is not supported by the general public and will have a huge impact upon how they decide to fund science in the future.
I expect full open access with the ability to comment on open forums will be the way to go.
The point is not to protect authors from the oh-so-dangerous comments of laypeople. The point is to keep the system useful to readers and would-be commenters by keeping it from being overwhelmed by noise.
“it needs to offer the exact same thumbs-up-down feature that this blog has”
This blog demonstrates exactly how useless the “thumbs-up-down” system is.
The core problem is that there are two possible interpretations for the up/down:
1) That the author has made a valuable contribution to the conversation (or conversely, has posted something distracting or inflammatory)
2) That the author has expressed a popular opinion (or conversely, an unpopular opinion)
My impression is that in most fora, the second usage dominates. In this case, the thumbs-up-down are totally useless (except as an exercise for developing a thick skin). Science should not depend on popularity. Even worse, these ratings are not a meaningful reflection of public opinion — they are more akin to a mob of fanatics marching down the street chanting slogans while reasonable people are at home with their families or at work. When I say that I wouldn’t care one whit for such evaluation, I’m not speaking as an arrogant scientist, I’m speaking as a person who is capable of thinking for myself.
As a democrat, I think that such systems only provide the superficial semblance of participation. At best they are a waste of time, at worst they are a distraction from real participation.
I’d guess enough PubMed topics are technical enough to have restricted audiences.
I wonder about contentious topics.
In the light of well-known effects like Fox News sock puppets, one wonders what would happen to a paper that found some brand of e-cigarettes had tin particles and other metals in them?
This will be an issue, but these organizations are perfectly able to hire published scientists (many of whom may not have done any research in 20 years).
Why should appearing in PubMed’s author database be a prerequisite for being allowed to comment? This is clearly pure nonsense and shows the persistent superiority complex you find in large number of scientists.
Should someone who works in the field of sleepwalking be allowed to comment on a paper discussing lung cancer therapies? Why? Because they are a scientist? Yet a lung cancer patient, who is not an author can’t?
A farmer can’t ask questions in relation to papers discussing the type of soil on his land, yet Oscar Winner Colin Firth can, because he’s named on a paper.
To live in such an ivory tower…
“Should someone who works in the field of sleepwalking be allowed to comment on a paper discussing lung cancer therapies?”
It is pretty much irrelevant if they are allowed to comment, because they will have no interest in commenting (unless they are a cancer patient, perhaps).
It is entirely relevant. The point is that two people who are entirely unqualified to discuss highly technical cancer therapy papers are treated differently, purely because one has a name on a paper somewhere in PubMed.
In this case, we have to ask “what is the purpose of pubmed commons”?
If the purpose is to make everyone feel included in the discussion (i.e. establish democratic legitimacy), then it matters if two people are treated differently. However, if the purpose is simply to create a useful resource for navigating the literature, then all that matters is what actually gets written.
Also, a lot of medical doctors will not be able to comment.
If PubMed does want more stringent selection of comments, they could leverage the “related articles” feature of the database. I think the best use would be to prioritize comments from those authors.
Aside from that, I don’t think that PubMed includes much material related to agriculture (though it is constantly expanding it’s reach). I think this gets us back to the fundamental question — why would PubMed be the appropriate forum for a broad public discussion? The PubMed database can be mirrored elsewhere (as will PubMed Commons), and other systems (e.g. Google Scholar) are more comprehensive. Google surely is more capable of managing a system that is fully inclusive.
I ought to remind everyone, though, that even PubPeer was not open to everyone in principle, as you needed an institutional email for signing up and commenting. With time they decided putting up unregistered comments, and even so at much reluctance.
I used my gmail account to register with PubPeer & with PubMed Commons.
Of course, it is a published email account; I use it as my “correspondence email” for my papers. (Don’t really like institutional email addresses, they are a bit clunky.)
I see, I thought they only took official inst emails!.. Maybe I should have tried otherwise for safety.
The main idea is that you have have an email already published in the literature to comment as a “peer” in PubPeer; and to be able to sign up for PubMed Commons. These are *usually* university email addresses, but not always (as in my case).
Also: I know not everyone agrees with limiting the conversation like this. I’m a bit conflicted about it myself…..
I would like to bring up a situation that is an inherent problem with PubPeer and now PubMed commenting systems. Indeed some authors can exploit the possibility of commenting on published papers to bring up false concerns, especially on competitor groups. Please see the examples below, where I am convinced some group is picking at another group by raising questionable concerns:
https://pubpeer.com/publications/F7FF3B67DC0FFB4BDFAF034EBD0A48
https://pubpeer.com/publications/B533DEFB8F771B02423B839CA7B0B6
I caught these two accidentally. Maybe there are much more cases like this, where a passing reader might just think “Argh, more WB manipulation, what rascals” or even “This Peer 2 is clearly one of the authors trying to make a fool of us, once again”.
Thus I would indeed recommend care with taking comments on papers too seriously and quickly — there are already guys trying to use them the wrong way. I am sure this will be a constant.
Just clicked on the links- looks like the comments have been taken down…..?
Oh OK!.. If I were in PubPeer I would have left the comments with reprimands to set an example. But the case was, apparently some competitor group made comments in (at least) two papers of a set of authors from Switzerland, claiming there were serious western blots issues and problems with statistical analysis. That “uneven background and streak between lanes” sort of thing. For some reason I got curious and went to check on the papers, and the blots seemed perfectly OK to me, and I could say little of the analysis. The comments came from Peer1 and another unregistered user. I made there my suspicions on the comments and my impressions on the blots, and reported the comments. Now they were wholly removed.
Still, I am not sure how frequent this is. I have checked on those by chance. I am not sure if many passing readers go to the papers to check the veracity of comments made, and many times authors do not respond for many reasons. This is probably an intrinsic problem of the post-review system, and I think can only be fought back with further review, like I did. But how many more could there be?
It wouldn’t be out of the question that the strip Nav1.7 of the panels Nav1.7 + B2 and Nav1.7 + B4 were the same blot but a different exposure time
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3757325/figure/F6/
The configuration of the bands and relative intensities look very similar, but there are some artifact spots that differ. This could be simply a factor of taking two different exposures or tidied up by photoshop.
I not saying it is true, just I could see it was possible.
Otherwise I think it was someone expressing genuine doubts, rather than a plot.
As a “still working at the bench” scientist, I would appreciate PubPeer comments that indicate what others have had trouble reproducing, or if yields, accuracy, etc. were over-represented. The same goes for issues with specific reagents or anti-bodies in a paper. After spending a year of a PhD working on something that I found out others had massive trouble reproducing but hadn’t been formally shot down, this kind of venue could be a timesaver.
I don’t know that everyone is going to be honest. As for the specific reagents or anti-bodies, if I come out say that a particular source’s antibodies are not any good, I have opened the door to libel and/or slander. Given the amount of international purchasing that goes on, I have to be cognizant of which nation’s law prevails.
NCBI has provided an update on how PubMed commons is being used.
http://ncbiinsights.ncbi.nlm.nih.gov/2013/11/01/early-developments-in-the-pubmed-commons-pilot/
“Perhaps a third of the first 200 comments included critique or pointed to other studies or reviews with the potential to change people’s interpretations or conclusions. Some authors posted corrections or changed their own conclusions in the light of others’ subsequent work. Authors are also using PubMed Commons to update people on their work – including links to databases that have moved, providing contextual information and backstories as well as new, relevant work”
They also have a sort of “thumbs up/down” system, but they are more explicit about the meaning of this rating. They ask “was this comment helpful yes/no”.