By now, Retraction Watch readers may have heard about new Nobel laureate Randy Schekman’s pledge to boycott Cell, Nature, and Science — sometimes referred to the “glamour journals” — because they damage and distort science. Schekman has used the bully pulpit of the Nobels to spark a conversation that science dearly needs to have about the cult of the impact factor.
The argument isn’t airtight. Schekman — now editor of eLife, an open access journal — says that open access journals are a better way to go, although he doesn’t really connect mode of publishing with the quality of what’s published. Others have pointed out that the move will punish junior members of his lab while likely having no effect on the career of someone who has published dozens of studies in the three journals he’s criticizing, and has, well, won a Nobel.
All that aside, it was Schekman’s reference to retractions that, not surprisingly, caught our eye:
In extreme cases, the lure of the luxury journal can encourage the cutting of corners, and contribute to the escalating number of papers that are retracted as flawed or fraudulent. Science alone has recently retracted high-profile papers reporting cloned human embryos, links between littering and violence, and the genetic profiles of centenarians. Perhaps worse, it has not retracted claims that a microbe is able to use arsenic in its DNA instead of phosphorus, despite overwhelming scientific criticism.
Although the first sentence is an inference — one that others have also argued — everything else Schekman says in this paragraph is true. But just how many retractions have these journals had? And how does that compare to the number in the Proceedings of the National Academy of Sciences (PNAS) while one Randy Schekman was editor? Here’s the 2006-2011 data, analyzed according to Ferric Fang and Arturo Casadevall’s Retraction Index, which calculates the rate of retraction per 1,000 papers published:
|
Journal |
Retractions |
Articles |
Retraction Index |
Impact Factor |
|
Science |
10 |
5702 |
1.754 |
32.452 |
|
Nature |
7 |
5403 |
1.296 |
36.235 |
|
Cell |
11 |
2899 |
3.794 |
34.774 |
|
PNAS |
23 |
21614 |
1.064 |
10.472 |
So yes, PNAS had a lower Retraction Index than the other journals, but not really that much lower than Nature. Put another way, however, PNAS retracted 23 papers from 2006 to 2011, while Cell, Nature, and Science retracted 28. And perhaps even more important, there were 1,300 retractions in journals other than those four.
“Wait,” you’re saying, “are more retractions really a bad thing? Didn’t you just publish a post about a study that said the opposite?” Well yes, yes we did. But Schekman is suggesting retractions are a mark against a journal, which we think makes PNAS’s record of retractions fair game.
When Schekman refers to the fact that Science hasn’t retracted the arsenic life paper, he invites readers to review the history of requests for retractions at PNAS. And that doesn’t look all that different from the journals he’s criticizing.
We covered this retraction of a cheetah fossil find, for example, which took four years, three of which were during Schekman’s tenure. Along the way, PNAS wouldn’t even publish a letter of critique. Compare that with the multiple letters — and two replication attempts — Science published about the arsenic life paper.
Or take this example, chronicled by UCLA researcher Andrew Diener. Diener raised concerns about a paper in late 2006, and after an exchange of emails with the authors and the journal, managing editor Daniel Salsbury told him:
We have now received the authors’ response to your comments. A member of the Editorial Board has re-read the paper and the comments from you and the authors. I have included the authors’ reply and encourage you to contact them directly regarding any additional concerns you may have with their work. The Editorial Board has concluded that no additional action is required by the authors at this time.
Diener wasn’t satisfied with that, so he wrote the journal again. Then two years went by, and he found the paper retracted. So PNAS eventually did the right thing. But in those two years, the paper picked up ten citations.
None of this proves that PNAS under Schekman had a worse record than did Cell, Science, or Nature. It does, however, suggest the picture may be a bit more complicated than his Guardian piece let on.
In the end, though, it’s hard to disagree with Schekman’s conclusion:
Funders and universities, too, have a role to play. They must tell the committees that decide on grants and positions not to judge papers by where they are published. It is the quality of the science, not the journal’s brand, that matters.
In fact, even Nature editor in chief Philip Campbell agrees.
He’s right, but there is the stink of hypocrisy since it is no doubt his successful history of publishing in exactly those journals that has put him where he is today.
The problem as I see it (as I posted in a previous thread) is that those journals (particularly Science and Nature) are really magazines that consider themselves gatekeepers of the Hot New Thing. They are discriminators of fashion and I compared them to Vogue. The issue is not one of quality science vs poor science, but one of self-validation through publication in so-called elite journals. In other words, something is ‘important’ simply because it is published in Science/Nature/Vogue.
Science should not be a fashion show, nor a popularity contest, and there should not be such gatekeeper journals.
These journals aren’t gatekeepers, there are literally thousands of credible alternatives. But they may be fashion gurus, to better fit your analogy.
you ever try to get a faculy position without a glamour article?
Nature Cell and Science look at publications as products based on acceptable risk/fame and glory theory: High impact factor journals are willing to publish riskier, but potentially higher-impact claims ASAP – more retractions are the price for getting high-impact science out early. The more negative version of this theory is that high impact factor journals CARE more about a high impact factor than about the integrity of what they publish. Go on feed them garbage and you will get garbage.
I am curious about the consequences of having ‘luxury’ funding agencies like the HHMI and Welcome trust in charge of a journal. Have the funders run a journal seems to present a conflict of interest. Will the editors of Elife be under any pressure to review HHMI funded research? Or will they refuse to consider research that contradicts a member of their prestigious body? Will people without HHMI funding feel under pressure to publish in Elife to help get HHMI funding?
This is an excellent analysis of Schekman’s views. It’s a complex issue. On one hand, we sorely need to have this discussion, as the glamor rags of science really do distort the whole process of impact and assessment, and its only going to get worse as the stakes get higher and the makeup of what constitutes a minimal publishable unit in these journals increases. On the other hand, and with the greatest respect to Dr. Schekman, we have to seriously ask whether he has the credibility to lead this discourse. As someone who has more than 40 Cell, Science, and Nature publications to his name, has built a career with such high impact credits, and who has a vested interest in the success of a new journal, I don’t think he fits the bill.
But if his papers had been rejected by those journals, he would also be accused of a conflict of interest; and had he not been an editor at PNAS, it would be said that he had no experience with editing a relatively high-visibility journal.
Another aspect of the problem (just because no one else has mentioned it) is that the glamor rags have a constituency far beyond the pool of potential authors: e.g., scientific administrators, educators, policy-makers, and a slice of the general public. The interests of these non-authors are diverse. However, seeing that deserving researchers get a fair (measured how?) amount of public exposure for their work is probably not often a critical consideration for non-authors. It *is* a factor. I don’t necessarily disagree with you. But, integrating over the whole constituency of those journals, fairness to researchers and rewarding merit (assuming we can measure them) are not always the dominant variables — and probably shouldn’t be.
Unfortunately, there are at least a few directors of big research institutions who never really did in my view important science, but only politics. They are even proud of choosing new laboratory heads just by the impact factors the applicants got, combined with the citations. Of course, nobody can really be able to understand and evaluate all the research capabilities an applicant for some prestigious full professorships has, but some institutions appear to be proud of having an administrator to get a lot of money and to make the decision on who will get the new positions. It is clear for me that there is no possibility to fill such positions with real basic scientists, but more with those in currently popular medical fields. Not many people really understood my (unpublished) work.
One might add to the dataset and put in a column for “corrections”, where data have been used “inappropriately”, including use of the same data for multiple experiments describing different conditions. My own experience at PNAS was now entirely wonderful, detailed here http://ferniglab.wordpress.com/2013/02/06/responses-to-evidence-of-self-plagiarism/
An interesting read and the point is taken. Perhaps this is the tip of the iceberg.
It is not only retractions, its all the dodgy data in the glamour journals or shall I say “errors”
Of course, as we all know too well, dodgy data is not limited to the glamour journals, but if glamour journals could take a lead, perhaps others will follow.
A simple solution is to make science-fraud-linked grant awards a criminal offence. Not only should monies be repaid to the grant bodies in question, but fair and just punishments dispensed.
Publishing papers in “elite” journals surely brought Schekman to the career point where he is now (and the Nobel Prize of course). However, whatever are his personal or professional reasons for boycott, he is right. And also, if a scientist who still needs to prove himself would publicly declare something like this, his/hers future career would be certainly damaged and ridiculed because of failure to publish in this journals. So the only way for things to be corrected in a deep-rooted system, is for someone who finally gains the power of being untouchable, to speak out! And to gain such a position you firstly need to be uplifted by the same system you are opposing. System can only be destroyed by the inside, especially the wrong one. So, Schekman got it right. And of course, there is always a way to make accomplishments relative, by using appropriate rhetorical and logical skills. But it any case – he got it right. Criticism should not be reserved only for the “week” and “pour” (scientific journals)! Good job, congrats to him!!!
” System can only be destroyed by the inside, especially the wrong one. ”
One problem is that Schekman did not propose a plan for overturning the system. He merely said he was not going to publish in high-impact journals. I don’t think even he will succeed in this. His post-docs and junior collaborators will insist on submitting to Cell, Science, and Nature as much as they can.
It’s hard to see what to do about all this. Department Chairs, Tenure Committees, and the all-important funding agencies all look mainly at one thing: your publication record. How many papers? And Where? That’s the beginning and the end right there. And as far as I can tell, the only alternatives to this are favoritism and nepotism. I can foresee no mechanism for the objective evaluation of scientific effort, skill, or quality.
“How many papers? And Where? That’s the beginning and the end right there.”
Yes, heaven forbid they actually “read the papers”. (Or bring in an outside expert with no COIs to read/evaluate. Easier said than done in narrow fields, I suppose.)
Double-blind peer review would not eliminate publication bias, but it would reduce it. At minimum it would force reviewers to read the papers so that they can guess who the authors are. eLife missed an opportunity by not having double-blind review.
A couple weeks ago, I reviewed a manuscript for a journal with a double-blind review process. A quote from the first page: “We previously reported… [citation].” Boom. Anonymity gone. Double-blind peer review is a nice idea, but rarely seems to work in practice.
Awesome!
I agree. However, I cannot escape the feeling that raising the voice can be good for a start. At least, to promote public awareness about the issue. In practice, it will probably not make any significant change.
In other words, Schekman is a Gorbachev.
This is RW. Isn’t there a statistician in the house? I calculate that PNAS had a significantly lower retraction rate than the other three combined (p < 0.05), but not significantly lower than Science or Nature, taken individually. There's also something to be said for using the impact factor as a very rough proxy for resulting damage to science (just as it is sometimes used as a proxy for the scientific importance of a non-retracted article). In that case, PNAS is significantly better than any of the others, individually or in combination.
What those numbers mean — assuming I haven't botched the calculation — is ultimately up to y'all, but there's no sense in tossing retraction statistics around unless we're willing to accept that they might mean something.
in some ways, would the even better metric be citations of retracted papers?
say fake journal Natural Cell Science retracts ten papers, but all within a week of publication with no citations… little harm, little foul… but The Science of Nature’s Cells retracts only 1 paper, but only after 5 years and 1987629 citations.
Then again, a paper with a splashy finding that leaves mechanisms wide open is the best way to generate citations… but not the best way to do science…
I think it’s worth following the link to the Fang/Casadevall paper, which demonstrates the correlation between impact factor and retraction index.
People keep confidently asserting that Schekman would not be where he is today without having published in glamour journals. There is plenty of reason to believe that this can help, but where is the evidence for his particular case? Who is to say that he would not have made the same accomplishments and won the Nobel prize if he had published his papers in other journals?
will eLife be turned into another elite journal category – similar to Nature, Science, Cell. Looking at the editorial board – it appears to be in that direction. This is true for other subsidary journals from the above magazines/journals. Cancer Discovery is not far behind becoming one of those. So ordinary scientists will not have any chance there. It would be interesting to check how many of these elite journals have the same names as members of editorial boards?
That’s it! I’m joining Schekman! I simply WON’T publish all of my awesome papers in Cell, Science or Nature! I refuse out of principle! (which of course, is the only reason why I won’t have any when I’m on the job market)
On a more serious note… why not boycott PNAS(S) too? While not quite as high impact, unlike SNC, it grants some scientists different peer review options than it allows for other scientists. Selective for the masses and fodder for making unworthy post-docs from big name labs have prettier resumes.
Finally, he calls on funding agencies to not judge CVs by their impact factor publications… but this whole thing could be ended rather quickly: If the NIH were to only allow funded researches to publish in journals owned and published by not-for-profit companies or government agencies. Boom, SCN would be done in a flash and, while some publishers would likely rise up as the high impact replacements, it would be better having the $$$ out of it…
no need to add an extra S, the name is already embarassing enough.
“If the NIH were to only allow funded researches to publish in journals owned and published by not-for-profit companies or government agencies… it would be better having the $$$ out of it…”
right. So it would be infinitely better for it to be all politics, instead. Have you seen politicians worldwide? I feel like I would prefer the most money-grubbing capitalist making decisions strictly on profit motive versus the capricious dingbats who run governments. At least you know where the one is coming from.
Schekman’s comment “It is the quality of the science, not the journal’s brand, that matters” is noble, and he’s not first to say it, but no-one to my knowledge has posited a workable and fair approach for doing so at large scale, hence popularity of the Impact Factor.
It seems Schekman is reinventing the proverbial wheel. Inevitably, eLife would end up the same as the very journals he deems to be the problem, I suspect. Couldn’t he use his stature as a Nobelist/scientist/editor to try to improve the process for publication and peer review? Open access or conventional publication are just the formats and a reflection of technology, I would think.
Nature had a perfect summary of the problem in their press release:
“The research community tends towards an over-reliance in assessing research by the journal in which it appears, or the impact factor of that journal.”
It seems a bit hypocritical because they could do something about it if the wanted to:
blog.pubpeer.com/?p=77
If he were sincere about fixing the scientific publishing he would have stopped the PNAS contributed by articles, where the editor is the last author as well and the paper is reviewed by their friends and former students or postdocs.
Sheckman can put his money where his mouth is: he can crucify his own post-docs by having them publish their great work (that they thought of, not Scheckman–does an advisor ever help?) in eLife.
LOL!
not quite. the recommendation letters the postdocs from his lab (or any big name PI) will get will likely compensate for the lack of NCS papers. however, it would be interesting to see if Schekman will now find it harder to attract good postdocs. my guess is he won’t, for the reason stated above.
I find Schekman’s comment “It is the quality of the science, not the journal’s brand, that matters” interesting. If he truly believes in this, I wonder if he would allow any of his students/Post docs to publish in any of the “predatory” journals.
I have found that glamor journals like Nature have become advocacy journals that promote questionable research solely because it fits their prevailing bias while refusing to publish more valid research when it contradicts their advocacy. As a conservationist I found the most egregious example to be their coverage of the spreading chytrid fungus that was causing amphibian extinctions. Education would have helped to prevent the spread of that novel diseases by researchers and the pet trade but Nature only published papers that blamed climate change and the epidemiologists had to publish elsewhere. Nature’s detrimental actions are well documented in the essay “Contrasting Good and Bad Science: Disease, Climate Change and the Case of the Golden Toad” available at http://landscapesandcycles.net/contrasting-good-and-bad-science–disease–climate.html
Another example of how the power of the mighty corporation has control over research these days. Over the last 30 years big business has had a strategy to supress research that is not in their economic interests. They have done this by subverting good science in the interests of the bottom-line. All that counts is the profit motive and if science does not fit into their strategy to achieve this or reduces it, they simply get the scientific magazines to kill it off to a great extent. Therefore the corporate hold on pure scientific research and innovation is now vastly stifled on the altar of corporate profits. Indeed many scientists (profs et al) are now in the service of the giant pharmaceuticals who do their bidding. Unfortunately when you are dealing with an industry that turns over $1 trillion annually, integrity and what was once a conscience turns to a subservient medium. I just do not know how we can now change this as the corporates have basically taken control of R&D and how it is published and what is published, destroying science not creating it.
Dr David Hill
CEO, World Innovation Foundation
He’s a little late on his boycott. I’ve been boycotting Science, Nature, and “Post-Science and Nature” for years now. Not by choice of course.
I am starting to do the same. Science is not for sale anymore
Well, it is about time somebody speaks up. These are magnazines and not journals. They feed off teh impact factor and want you to think that thhey are top journals. Over the past two years, I do not cite any works of from these ‘magazines’ . We should all do the same. If they are good, then they do not have to use teh impact factor metric to push the work.
You are correct. Here are two articles published by Nature last year.
Martin Schultze, et al. “ Controlling Dielectrics with the Electric Field of Light” Nature 493, 75-78 (2013) and
Agustin Schiffrin et al. “Optical Field-Induced Current in Dielectrics”, Nature 493, 70-74 (2013)
Jacob B Khurgin replied to these articles in his “Optically induced currents in dielectrics as a nonlinear optical effect) published in http://arxiv.org/abs/1303.3994.
A gain, the physics is wrong as you can see by reading the reply. Expect retraction anytime!
I agree, this was a hot topic by the nanoplasmonics community in the SPIE SanDiego conference. Simply Nature refused to retract the work despite strong evidence from researchers in Germany (Prof. R. Claus).
Nature Cell and Science look at publications as products based on acceptable risk/fame and glory theory: High impact factor journals are willing to publish riskier, but potentially higher-impact claims ASAP – more retractions are the price for getting high-impact science out early. The more negative version of this theory is that high impact factor journals CARE more about a high impact factor than about the integrity of what they publish. Go on feed them garbage and you will get garbage.
see facebook: Boycott Nature Photonics
Everyone says he’s a nice guy, but I reported problems with papers to PNAS during his tenure and problems with eLife papers during his time there already. I have an academic email address, they are papers in my field, yet I receive not a single reply. He was cc’ed on all second attempts. Nothing.
Dr Randy Schekman is 100% right with his reasoning that addresses how magazines such as Nature Magazine undertake their business with regard to highly important work which in many ways they suppress. In this respect Nature Magazine we have found is basically a pawn in the game of big business. They do their biding when the bottom-line is threatened was our finding. The fallacy that a vaccine will come in time to prevent the world’s future most deadly pandemic in terms of Bird Flu et al is just a single example of their power over such magazines such as ‘Nature’. They stop the truth emerging in terms of alternative strategies and scientific solutions when that work affects n particular the big pharma’s vast profit making machine.
In this respect we have first hand experience of how Nature Magazine operates behind closed doors. A few of countless articles that may be of interest and mind opening are as follows –
http://worldinnovationfoundation.blogspot.co.uk/2013/12/possibly-most-important-keynote-speech.html
http://worldinnovationfoundation.blogspot.co.uk/2013/12/vaccines-will-never-save-us-from-deadly.html
http://worldinnovationfoundation.blogspot.co.uk/2013/12/global-pharmaceutical-giants-have-made.html
Dr David Hill
World Innovation Foundation
Science Nature and Cell have already boycotted me and I don’t have to boycott them. That explains why I don’t publish there. Any chance of me winning a Nobel?
Why do you want to publish in these magazines?