This past April, Amparo Acker-Palmer and colleagues published a study in Nature, “Ephrin Bs are essential components of the Reelin pathway to regulate neuronal migration.” Within a day of its publication, Nature readers were raising questions about many of its figures. They started like this:
Andy Gu said:
Looks like Fig 1a, the two middle figures are actually the same with little move from desired regions. I don’t trust their data now…..
After several such comments, Nature senior editor Noah Gray weighed in:
Many thanks to all who have pointed out potential irregularities in the figures of this manuscript. The issue is currently under investigation by Nature, with the full cooperation of the authors, and we will move quickly to bring this issue to conclusion. Both editors and authors are committed to ensuring that the scientific record is accurate and are thus working diligently to restore confidence in the results. Thank you for your patience.
A few Retraction Watch readers have brought this exchange to our attention, and suggested that a retraction was in order, so we’ve been keeping an eye out. Nature has acted, but rather than a retraction, the journal has issued a correction. It’s a whopper:
In this Letter we made errors in representative image choice, including mislabelling of images or choosing an image from the inappropriate genotype. In all cases, choice of images was completely independent of the data analysis and so none of the conclusions in our original Letter are affected. We apologise for any confusion these errors may have caused.
Figure 1a depicts a Tbr1 staining of the adult mouse cortex for four different genotypes. In the process of choosing representative pictures that reflect the results of our analysis shown in Fig. 1b, cropped images from original pictures were inadvertently mislabelled and used incorrectly. We provide below a corrected version of Fig. 1a with new representative images for the following genotypes: WT and Reln1/1;Efnb32/2. A new high-magnification picture for WT is also shown in the two rightmost panels. Original images for every genotype and additional examples are shown in the Supplementary Information of this Corrigendum.
Figure 1c depicts a Brn1 staining of the E17.5 mouse cortex for five different genotypes. In the process of figure assembly cropped images from original pictures were inadvertently mislabelled and used incorrectly. We provide below a corrected Fig. 1c with a new image for Reln1/1; Efnb3–/–. In the ephrinB3 compound mice (Reln1/2; Efnb32/2) Brn11 cells aberrantly accumulate in the lower layers of the cortex and do not migrate to the upper layers, resembling the Reeler (Reln2/2) phenotype. Original pictures and additional examples are shown in the Supplementary Information of this Corrigendum, where arrows indicate the distribution of Brn11 cells. We have also included results from a new, reproduced experiment recently performed with an additional cohort of animals that shows exactly the same results.
In Fig. 1d, the second panel, labelled ‘Reln1/1;Efnb3–/–’ should instead be labelled ‘Reln1/2’. In the Methods summary section ‘Stimulation of neurons’, ‘‘Cortical neurons from E14.5 were grown….’’ should instead read ‘‘Cortical neurons from E15.5 were grown….’’.
There were mistakes in the supplementary online material, too:
Further errors in the Supplementary Information of the original Letter are described and corrected in the Supplementary Information of this Corrigendum.
We thought this was quite an extensive collection of errors, so we wanted to know why Nature thought a correction was a better way to deal with them than a retraction. The journal responded (emphasis theirs):
Please see the definitions for corrigenda and retractions in the Nature journals. In this case a corrigenda was appropriate since the finding in the paper remains unchanged.
A Corrigendum is a notification of an important error made by the author(s) that affects the publication record or the scientific integrity of the paper, or the reputation of the authors or the journal. All authors must sign corrigenda submitted for publication. In cases where coauthors disagree, the editors will take advice from independent peer-reviewers and impose the appropriate amendment, noting the dissenting author(s) in the text of the published version.
A Retraction is a notification of invalid results. All coauthors must sign a retraction specifying the error and stating briefly how the conclusions are affected, and submit it for publication. In cases where coauthors disagree, the editors will seek advice from independent peer-reviewers and impose the type of amendment that seems most appropriate, noting the dissenting author(s) in the text of the published version.
We were also curious about the new results presented in the Corrigendum:
We have also included results from a new, reproduced experiment recently performed with an additional cohort of animals that shows exactly the same results.
We’ve seen claims like this before, in other journals, and at least one of those journals said it hadn’t even looked at the new data. Here, at least Nature is making it available. We wondered if Nature typically peer-review statements made in corrections and retractions, and whether they’d peer-reviewed this particular set of new results.
Nature would only sat that peer review is confidential, and that it would be best if we spoke to the authors. We tried to do that, but the corresponding author hasn’t responded to our requests for comment.
We’ll update with anything we find out.
Foe someone who wants to see what’s wrong with Figures.
http://blog.m3.com/Retraction/20110904/1
http://blog.m3.com/Retraction/20110905/1
http://blog.m3.com/Retraction/20110906/1
http://blog.m3.com/Retraction/20110907/1
i had the distinct privilege of watching this scientific trainwreck of a paper progress from the very beginning.
This wasn’t mentioned in the post, but we noticed that after the online Comments section for the article had initially been active, when postings expressing the exact same concerns began to amass, it was mysteriously made unavailable for any additional comments, and stayed that way for several months.
i’m a young graduate student with a manuscript that has gone through multiple rounds of review at two of the top tier journals. i have spent ungodly amounts of hours readjusting, reformatting, tweaking and scrutinizing each one of the figures over, and over, and over again to ensure that everything is where it should be, and is what it should be. it’s inconceivable to me how these kinds of errors can be made.
if the explanation of the MULTIPLE concerns being the result of plain and honest error are to be believed, then the authors truly should not be doing science. this was not a single problem in some buried supplemental figure, there were majore “errors” in 50% of the main figures, including the very first figure!!!
it took myself and my colleagues less than 30s to identify the “errors”.
it’s so ridiculously insulting that this is allowed to faulted to error.
things like this affect science as whole, and very much not in a good way.
Archived versions of pages:
https://web.archive.org/web/20111006165905/http://blog.m3.com/Retraction/20110904/1
https://web.archive.org/web/20111006165850/http://blog.m3.com/Retraction/20110905/1
https://web.archive.org/web/20111006165901/http://blog.m3.com/Retraction/20110906/1
https://web.archive.org/web/20111006165841/http://blog.m3.com/Retraction/20110907/1
wow, pretty bad Photoshopping there..
“A *Retraction* is a notification of invalid results”.
OK. Why then why are papers lacking IRB approval retracted?
Good question, which we covered in the LabTimes column we link to in the post. But keep in mind this is Nature’s definition; Nature publishes few clinical trials.
Those papers contain data that have been obtained unethically (e.g. without patients’ informed consent) and are thus invalid.
But, results are still valid and correct.
COPE Guidelines: Journal editors should consider retracting a publication if:
– they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error)
– the findings have previously been published elsewhere without proper crossreferencing, permission or justification (i.e. cases of redundant publication)
– it constitutes plagiarism
– it reports unethical research
I don’t think data obtained after fraudulent concealment are valid. (Then why have ethics committees in the first place?) The data might be correct though.
If a retraction is required when the results are invalid, then why do papers get retracted that have falsified and/or fabricated results that ended up to be true?
I don’t necessarily buy the editor’s explanation, and I don’t necessarily buy the authors’ (lack of an) explanation. On the face of it, I can see where if one selects random files that it is possible to choose one photo that may overlap with another group.
But, these photos were (1) cropped (meaning that someone must have cropped a single photo multiple times and used those crops from a single photo for multiple, different experimental groups) and (2) parts of unique, separate transgenic mice. In both cases, one would surmise that since these are different transgenic mice (and, presumably, different slides), one would put different groups into separate folders.
Overall, I find it rather odd–both the correction and the authors’ explanations.
Both the correction and the authors’ explanations are just hogwash. They should retract it or this is gonna start new trend!
I agree quite strongly.
Multiple consecutive errors, each building on the previous one would have had to have been made to end up with final product used in several of the figures. the simplicity of their explanation is most definitely insufficient…
Compounded with the fact that at there’s two of these instances in one of the figures (figure 1), in addition to more “errors” in another main figure, makes it really, really difficult to believe that these were simple errors.
And if the were only simple errors, then wow. just wow.
I believe this practice is endemic to the Nature family of journals, not just the mother-ship…
In 2007, Nature Cell Biol published a paper claiming that uncoupling proteins (UCPs) are an essential component of the mitochondrial calcium uptake machinery (PMID: 17351641). I work in this field, and immediately noticed several problems with the paper, including mislabeling of figures. Major issues surrounded their siRNAs which appeared to be not specific for different UCP isoforms, and the fact that they chose to study isolated mitochondria from liver – a tissue that does not actually express UCPs at all. These issues are so fundamental, they should have been picked up in peer review by anyone proficient in the field.
I drafted a letter to the editor (termed “correspondence” by Nature Publishing Group) highlighting the issues. I got a couple of colleagues to co-sign it. After several months of review, Nature responded that they would not accept the correspondence because it failed to repeat any of the original work and show that it was flawed. They simply would not accept any criticism that shed a bad light on their peer review system. By this time, the original authors had already published a review article in another, building on their “seminal” discovery. I also began to find my grants and other manuscript submissions being criticized for failing to take account of this novel work.
I went away and recruited what is arguably the finest group of co-authors I have ever had the pleasure to work with… a veritable who’s who in mitochondria, with a combined ~200 years of research experience on this topic. We repeated some of the experiments in UCP knockout mice, using tissues that express the proteins… 2 labs, opposte sides of the Atlantic, 2 different knockout mice, 4 different tissues, 2 different calcium uptake assay methods. As we suspected, calcium uptake in mitochondria from UCP knockout mice was identical to that in wild types.
It took 3 rounds of review and more than a year, just to get a 2 page correspondence accepted for publication. The page limit was strictly enforced for us (but not for the original authors’ response to our letter). During the review process for the 2nd revision, some 3 months after receiving the comments from 4 separate reviewers, and almost ready to resubmit, the editor forwarded a set of late reviews from a 5th reviewer and asked us to address them. Throughout the review process, the original authors were given the last word on every count, and we essentally did not know their final response until we saw it in print.
In the final proofing stages, the editors demanded that we remove comments about the siRNAs not being specific, because this was apparently a labelling mistake, and was to be addressed in an erratum notice to accompany the letters. The journal did not want to admit such a simple mistake had slipped through the peer review process, making it appear this was noticed by the editors or by the authors, rather than the fact it was pointed out to them in our letter. The erratum notice contains no information about how this mistake came to light.
Our letter was finally published 18 months after the original paper (PMID 18978830).
As an interesting follow up, the genes/proteins responsible for mitochondrial calcium uptake were finally cloned this year (2 excellent publications in Nature, PMIDs 21685886 and 21685888). In one of these papers, the negative control used by the authors to show that the effect was due to a specific protein (and not just a non-specific mitochondrial membrane protein interaction), was UCP2. Vindication at last.
Bottom line… you can have alll the heavy hitters on your side, but if you challenge something in a NPG journal, you will have a fight to even get in the door, followed by a pitched battle to get something published, with every possible curve-ball thrown at you during the review and revision process. NPG does not like it when you find mistakes that should have been found in peer review. The phrase “it’s in Nature so it must be true” was never more appropriate.
I admire your perseverance, and I congratulate you for succeeding at publishing your commentary within 18 months of the flawed report at Nature; this actually seems fast for a correction. Terrible for your field that you and your colleagues had to spend valuable time cleaning up the mess of others, and a mess that a careful reviewer of the original Nature report should have recognized by spending perhaps only half an hour longer on the review.
I agree, science will be better off if we cease adulating Nature and Science. We need to stop confusing quality with the difficulty of getting published in such journals.
Hearing these accounts of such gross negligence on the part of reviewers and editors with no consequence to all parties involves really disheartens me (a young PI). You very much diminished my disheartenment with your relentless approach to calling out the journal on its article – thank-you.
I really worry that what we are seeing is only the tip of the iceberg and what goes on behind the scenes – especially in the grant review process. Unfortunately I was faced with such an issue recently with regards to a grant that I was asked to review. The PI is a very successful scientist with numerous awards, papers and grants. I identified some data (a gel) that was re-used several times (manipulated in photoshop) in the grants preliminary data as well as the same gel being published in different articles and labelled as corresponding to different genes. I dug a little deeper and discovered that this lab was very active in re-cycling data as representative for different genes in different papers. I guess this lab has very low research costs since they need to only run one or two gels to produce several papers. I brought this to the attention of the chair of my grant review panel (the grant was triaged) and we brought it to the attention to the granting agency. I was told thank-you and that a full investigation will be undertaken and to protect you we will no longer communicate with you regarding this matter. I replied that I would be watching and if nothing was done in 12 months (ie retraction or major amendments of the papers in question) I will write to each of the journals to identify the articles in question. Nothing has happened yet and we are about to reach the 12 month mark – I feel very obliged and ethically compelled to contact the journals but also very fearful since the PI is very prominent and powerful. What should I do? I feel that this is being swept under the carpet.
Why aren’t reviewers held accountable? Maybe it is time that for the sake of full disclosure that reviewers should be identified once an article is accepted for publication. Reviewers should have nothing to worry about – especially if they did an honest and detailed assessment of the work and have no conflicts.
To deal with wrongdoing in science, here is some advise:
http://www.ethicsresearch.com/images/RRW_7-17-10.pdf
Also here: NATURE Vol 466:438-440 (22 July 2010).
I’m in a similar situation – noticed some dodgy data in a grant, followed it up and found the same problem in numerous papers. It is currently under investigation and I have no recourse other than wait-and-see. It has only been a few months so far, but if I don’t see retractions within a year, I will be “flipping tables”.
If you’re in the US, contact the Office of Research Integrity. If the PI is a recipient of federal funds (NIH etc.), they will investigate, and the consequences will not be pretty! If this is for a small granting agency, then you may be out of luck because that’s outside their jurisdiction.
Another important thing is to contact your own University’s research integrity officer (or whatever they’re called) and get all your suspicions and other observations documented.
From fraud to chaos. Nature will “impose” parts of the paper? So, Nature is actually a co-author? But everybody will agree that chaos is better than fraud. Justifications: 1) authors had 1 hr. 24 min. to write the paper, 2) Nature had to be faster: they concocted Nature’s definitions in 14 min. 3) at the time, Nature’s definitionist was on route to Singapore.
The new initiatives will follow.
The figures in the original paper that initially raised question go way above and beyond a simple mix up of raw images. Not only were side-by-side panels in the same figure overlapping images from the same brain, they were also contrasted differently and rotated slightly. You can actually align columns 2+3 from Fig. 1A to form a continuous section of brain. That’s not all!!! IN THE SAME EXACT FIGURE (Fig 1C columns 2+3) there is another instance of “mistaken choice of representative image” that goes beyond simple mistaken use of images. Still, that’s not all. There are two additional instances of blatant fraud in supplementary figures 4 (1+2) and supplementary figure 5 (c4+e4).
To counter the phrase “it’s in Nature so it must be true”, a phrase I commonly hear is “50% of all Nature publications are not reproducible”. Personally, instances such as this only reinforce this belief. I haven’t been able to come up with any sort of plausible reasoning behind the ultimate publication of the article. It’s beyond beyond and extremely disheartening to witness as a young scientist.
I really hate it when, after clearly fake or fraudulent data is found in a prestigious journal, the authors repeat the experiment and verify their original conclusion, and then say, “it’s all OK, it was reproducible!” The point is not correctness, but rather the integrity of the scientific method.
If I made up a paper saying I’ve found the Higgs boson, making up data from a pretend particle collider, I might well be correct that the particle exists, but it’s not science if I just write it up like a story.
Maybe we all should take just enough data to think we might have some result, make up the rest of the evidence, and then publish. Only if we are caught in fraud will we have to actually do the rest of the work. What a time and money saver that will be.
See my comment below for a repeat performance of this exact same scenario, this time in Science. It seems you really can get away with it, although it might help if the fraudulent data has already accumulated nearly a 1000 citations (according to google scholar) as in the example I give.
These are corrupt practices, nothing more, nothing less.
Clearly fraudulent, this paper should have been retracted in full. Could this be an emerging trend? Big journals refusing to fully retract articles, instead favoring large corrections? Science has recently done the same with the XMRV paper which, arguably, is less dodgy than this one.
This could be a worrying trend.
This “worrying trend” is certainly re-enforced by the fact that these journals are part of a bigger media outlet defining brands under the umbrella such as “Nature” with all the diversified journals bringing in revenue. These journals are business models which like newspapers gain only from having a lot of articles cited which draws company ads into the publication. Therefore, it even pays to delay corrections.
My personal opinion is that approaching and convincing journals to correct or retract manuscripts is going to remain a cumbersome and nerve-wracking task.
What will develop sooner than later will be an expansion of the “retraction watch” model. Why not mirroring the Pubmed site and having a tag on each publication. The tag can be filled with comments about problems encountered while reading published papers or experimenting on the same problem. Many labs see publications and make connections to their own work. How often does somebody fail to reproduce, we will never know. Producing negative data cannot be published easily. Getting notified about potential problems with any scientific problem connected to a publication would save a lot of ill-spent time and also save resources.
Why aren’t reviewers held accountable? Maybe it is time that for the sake of full disclosure that reviewers should be identified once an article is accepted for publication.
“Representative examples” – rarely representative, in this case, not even examples at all!
Paul Brookes (September 28 comment) is correct to say the problem is endemic to the family of NPG journals. In my experience, the culture within NPG is We are Nature, we are the arbiters of quality, and we make the rules. They have an obtuse way of responding to criticism with arrogant dismissiveness.
Back in 2003, Robert Rubin and I notified Nature Neuroscience about significant undisclosed conflicts of interest in a review article authored by Charles Nemeroff (PMID: 12403988). The editors failed to see the ethical forest for the legal trees. For cover, they fell back on in-house policies that, at that time, called for disclosure in reports of original research but not in review articles (where disclosures are even more necessary). Their tone was similar to the hair splitting in this case about Corrigendum versus Retraction. Rather than respond forthrightly, the NPG editors gave Dr. Rubin and me the runaround for months until we finally took it to the New York Times. The notoriety that followed led to a swift change of heart, publication of our letter, and public declaration of a new policy on disclosures for all of NPG. The aggravation to us was similar to what Paul Brookes described. The original authors were given the last word to respond, they laced their reply with ad hominem digressions that a competent editor should have disallowed, and they were not required to observe the space limit that applied to us.
This year, I have just been through a similarly exasperating experience with Nature. As I read an article in Nature published February 24 of this year concerning the peptide hormone PACAP in posttraumatic stress disorder, a number of serious methodologic concerns leapt off the pages (PMID: 21350482). . Following the directions of the journal, I first communicated these concerns to the authors. They responded evasively at first and later with unprofessional abuse and threats. The matter then went to the journal editor who, once again, fell back on pedantically idiosyncratic in-house rules and refused to publish my letter, which called for a retraction. Instead, he required the authors to publish an Addendum, which duly appeared September 1 (DOI: 10.1038/nature10396). This published Addendum is itself a model of evasiveness and obfuscation, and it deals with only two of the multiple issues I raised in criticism. Throughout this 4-5 month process, the tone of communications from Nature was defensive rather than open. The in-house rules of NPG appear designed to facilitate glossing over problems that are reported in the design of studies and in the integrity of data reporting.
I also second the suggestion that incompetent or lazy reviewers should experience consequences. In the case of the PACAP-PTSD article, one of the presumed reviewers who waved the article through to publication penned an accompanying commentary that appeared in the same issue of Nature (PMID: 21350472). Do journals maintain a blacklist of incompetent reviewers, do you think?
Thanks blog like this, we get to know the harrowing ordeals of our peers and the reality behind these fancy journals!!
Yeh, I honestly think the way forward here is the publication of the review files, just like EMBO J is now doing. Although having read many of these files online, I think the practice reveals many more problems with the review process, chief among them is the endless request for more and more and more and more needless experiments. Going all the way and revealing the reviewer names is also necessary. It will happen sooner rather than later.
Sooner, rather than later you will not have any reviewers.
I agree with Pyshnov, if reviewers are going to be punished by their mistakes or flaws, soon there will be no reviewers left. At the end, their task is not remunerated despite its importance and the consequences it may have in the scientific job. Inmy opinion, the difficulty to find willing reviewers is already a trend. And yes, journals try to score the job of their peer-reviewers in order to keep a track on them and be able to select the “best performers”, although in my opinion, weighing more speediness than thoroughness (another trend).
Here’s a good example of exactly this practice, in Science this time, for a change. First, the paper: (DOI: 10.1126/science.1071914)
Figures 3B and 4 contain shocking and blatant forged data. Bands are copied, cut and pasted freely and there is abosultely no way this can happen by accident. It is open and naked forgery.
Copies of the figures (contrast adjusted to emphasize fraud) can be found at: http://www.scribd.com/doc/67627419 and http://www.scribd.com/doc/67627401
This was of course reported to the journal, and 5 years later the following appeared: (DOI: 10.1126/science.318.5856.1550b)
Complete rubbish, the correction doesn’t even attempt to explain the original deception. At which point does the corrrection “make it all better after all”. Phew, what a relief, and I thought for a minute that the copied and pasted gel bands represented fraud, whereas all they represented were results that the authors could obtain at a later date, but didn’t have at the time of publication. So, move along please, nothing to see.
Same group has plenty of previous form in similar matters. (http://en.wikipedia.org/wiki/David_Baltimore). It seems if you are a nobel prize winner, you are untouchable.
Do all corrections need to be investigated? Here’s a good example of what I am taking about, in J Neurosci.
http://www.jneurosci.org/content/28/44/np.short
I know of a Nature papers in which the authors managed to do ChIP followed by a qPCR with a set of primers 2000bp apart ( yes, a qPCR), that was supposed to validate the mechanistic part of the papers.
I was interested in using those primers for my own experiment so my advisor and I contacted the authors to ask them if there was a copy/paste sequence mistake (it happened) because surely they couldn’t use primers which are 2000bp apart for a qPCR…the corresponding authors asked the first authors to confirm the sequences, not only did she confirm the primers sequence but she ‘nicely’ gave me the target gene sequence she used to design the primers, that was a cDNA sequence!! Except she performed a DNA ChIP! So, on the cDNA the primers were 200bp apart but on the DNA they are 2000bp! Yet she claimed that her. ChIP qPCR product is 200 bp.
We told them that there was an inconsistency, but my advisor decided that he wouldn’t report this to Nature, even though that ChIP was basically showing that the effect of transcription factor X on target gene Y was direct.
And of course no referees picked it up because unless you need the sequences, no one would ever verified. So, does it mean everything is wronh in the paper, maybe not, but certainly they didn’t demonstrate what they claimed, and they never answered when we told them they used a cDNA sequence to design primers meant for a DNA qPCR! Floppy postdoc with a very ambitious PI= floppy science
Dr Amparo Acker-Palmer has now been elected EMBO Member: http://www.embo.org/news/press-releases/press-releases-2015/embo-announces-new-members-for-2015
And got an advanced ERC grant…..just announced.
http://brain.mpg.de/news-events/news/news/archive/2015/may/article/amparo-acker-palmer-receives-an-erc-advanced-grant.html
A recent update came out in Nature. It takes a great deal of effort and a very long time to correct the literature.